Information vs Misinformation: A Theorycrafters Lament
by, 10-07-2012 at 11:36 PM (5452 Views)
Over the last 5 or so years I’ve been doing Elemental theorycrafting one of the things I’ve had to contend with is other people posting misinformation via guides, videos or forum posts. It’s one of those niggly downsides to theorycrafting work that varies depending on how you handle the “There’s People Wrong On The Internet!” problem.
For me, I don’t mind if people have their own opinions, or if they differ from my own. I don’t even mind being proven wrong when it comes to math, but it’s that proof bit that often people fail on. It’s not just enough to go “I think X > Y”, you have to come up with a supporting argument, spell out your reasoning, and in most cases have math as well. In a simple case, you’d want to do something like this blog post where I compared Elemental Mastery and Ancestral Swiftness while considering the impact of Ascendance (and since Ascendance is more of a 2.5 multiplier less crit chance and your normal Lava Burst frequency, we can pretty safely say that Elemental Mastery comes out on top).
Need More Data
So that’s the “theory” part, but what about data collection? If you’re doing data collection, you have to know what you’re looking for, but in most cases you need multiple replications of each test in order to reduce variation in your data sample. When I’m looking for spell coefficients for Fire Elemental spells, I’ll run 3-4 casts of each Fire Elemental with & without gear so I’ve got a large enough data set that I can be reasonably sure I have the minimum and maximum spell values from which I can work out the coefficients.
If you are doing something more complicated and doing dummy dry runs to look at rotation mechanics and/or talent/glyph selection then you will need many many more runs. Ideally you’d want 10 to 15 runs of about 6 minutes per setup, and try to execute each as perfectly as possible which may require an additional warm up run or two to get used to the rotation change.
Once you have your “dummy data”, your task won’t stop there. You’ll need to load it into something like World Of Logs to start doing some analysis of the rotations. You’ll need to total the damage/hits/crits/etc from your runs, average them, and then compare them between your different test sets. The trick will be to figure out what differences are from human error, statistical variability (ie: you ended up with more crits or overloads in one data set than the other) or actually comes from the changes you intentionally made. Once you have this information together, then you can start presenting an argument, but remember, don’t adjust the data to match your hypothesis.
There’s more than one way to skin a...
Data collection is all well and good, but how does that differ from what I do? I do some data collection, but it’s primarily focused on individual spell mechanics. Once I know those, I can add them into my model of the Elemental DPS rotation. The advantage of building a model (or a simulation, but more on that later) is that you can take chance rates and average them out. For example, a 10% crit rate with a 200% critical can be said to be, on average, 110% damage per hit. This way if you increase your critical strike chance by 1%, your average damage increases to 111%. If you were testing this via dummy testing you’d have to spend hours to get enough data to even notice a slight difference in outcomes.
On the flip side, models require knowing how each individual little bit works. Usually this isn’t an issue, as most of these little bits don’t change and we are often assisted by patch notes when they do. There’s also a number of people involved in knowing and working out these mechanics, updating their systems, and informing others of what they’ve found. One case of this was discovered in December 2011 between myself and the guys at Simulationcraft where it turned out that Lava Burst overloads were being affected by the Lava Burst glyph, but Lightning Bolt overloads weren’t.
Speaking of Simulationcraft, that’s a prime example of the third method of analysis: simluation. Again, this requires knowing all the mechanics, but instead of averaging out chances and estimating uptimes that models do, simulations run multiple itterations of the rotation in very short periods of time. This gives dummy-like test data results, but in a much faster time frame. Ideally this is the preferred way to do cross class comparisons & do more thorough analysis of talents/glyphs/stat weights, but it takes time to run and requires programming knowledge to adjust further than the program allows (which I do not have, which is why I stick with my spreadsheet models).
Why Misinformation Makes Me Sad
There’s two reasons why I try to combat misinformation when I see it. The first is that information, once it’s out there, stays out there. Most of the “pro” guide sites out there have out of date or incomplete information for Elemental Shaman at the moment, most of which was based on work I originally wrote back when Cataclysm was first released.
Usually this is because once something gets out there, it gets copied to a few sites, and then it’s accepted as “fact”. If something contradictory is posted elsewhere, it’s one post against three or four, so it’s either a matter of contention or the one is wrong, right? As an aside, this is also why I’m a bit dubious about “Class Guide Sites” that cover everything, verses the class guide threads on forums or posts on class or spec specific sites. At times they feel like they are fancy skins and copied content with the intent of driving ad revenue, but maybe that’s just me being cynical.
The second reason is that I put a lot of effort into researching this stuff, building these spreadsheets, writing these blogs & guides, all so the Elemental players out there have a resource to go to should they need it. I’m also a red head & can get a little irritated at times, so when I see someone go “hey guys, I just did X and found that Y > Z” I think “That took maybe an hour, has no statistical validity and whyyyyyy???”. It’s just so... frustrating... at times...
Replying to such posts can be a bit dangerous. Most of the times I get it wrong because my language use tends to be based around how I’m thinking, and while it’s usually always verbose, when directed at a misinformed post it can get a little... snarky. I should work on that a bit more.
What Can You Do?
The best thing you can do if you have a question about whether one choice is better than another is to ask a well thought out question about it, with as much supporting information as you can. Don’t just ask which is better though, seek to understand the mechanics behind the choices, so that you can understand why one is better than the other, and hopefully answer your own question in the process.
It’s for reasons like this that I’m building the Totemspot Wiki to store all those details, so that everyone asking the right questions can find answers for themselves. It’s a bit patchy at the moment, and will need to be updated for Mists, but hopefully with the expansion we’ll get a few more people on board and slowly pick up speed.
It’s an Almost Thankless Job
From a recent example of the misinformation post, a question popped up in my mind: Should the average player know who their class theorycrafter/guide writers are?
It’s stuff like that this that would make me want to throw the towel in, if it weren’t for the fact that I know what I do is known and is appreciated. There are probably somewhere up to a few dozen serious active theorycrafters in the WoW community at the moment. I know of four of them off the top of my head, counting myself. All bar one are Shaman. Most of the guide writers I know are Shaman, but odds are if you ask your average Shaman player they wouldn’t be able to name more than one, if they could name any at all.No one outside your tiny world knows who you are or what you do
At the End of the Day
All in all, I’m just trying to make sure that the information out there is the right information. Yes, I know that currently we’re in the middle of a beta, and that things will change, but there’s a difference between “currently in beta” and “wrong approach altogether”.
Total Trackbacks 0