Signal to Noise ratio, data and explanation models in WoW subscription analysis

There is a recent buzz around WoW subscription and activity evolution. The first to put this out where WoWInsider showing data and asking “Why are people leaving WoW?” It then spread through the blog space that I read like it does, popping up on Raph Koster’s, Joystiq, Tobold’s blog and more.

I think there are a few things to be said about this as it stands right now, just from a pure data and analysis perspective. But why? Well one of the more interesting reactions to the graph is Raph Koster’s graphical analysis. He claims that a Bumb curve with an exponentially declining asymptotic tail (sorry for the math lingua, this basically just means that you have a smooth increase to a maximum, then it curves back down, but leaves an ever-diminishing tail that never hits zero). Raph proceeds to explain the WoW census data using this curve, overlaying new bumps at what he claims to be expansions of the market or other bumping factors (like a chrismas rush).

I found this analysis intriguing, but above all… just plain speculative. Mostly because the method is flawed. Let me explain why. Or rather let me credit the kinds of sources that have tought me why. The most accessible is the lovely book called “How to lie with statistics” by Darrel Huff which came out in the mid 1950 (old crap so to speak). It also contains a passage on how to make graphs represent just about anything. I found a short summary of the basic idea that unfortunately uses straight lines and rescaling, but there are many more tricks.

Let me try to explain what the pitfall with Raph’s argument is. The short version is that sums bump function is a very nice candidate to describe almost any data. The tail doesn’t make a big difference here because it’s drowned in the noice floor. The flexibility of the bumb function is that you can approximate rapid peak data with a steep hump that will go away quickly, you can approximate slow increases and decreases with a wider bump.

A longer version would show how you can, having essentially two variables to play with per bump function (height, width) one can match the piecewise slope of almost any function, one would go along to show that in the limit of vanishing width you end up with standard sampling, which of course can represent anything up to sampling frequency and that by keeping the height bounded you can always keep the error in the tail bounded. But basically that’s just the mathy and rigorous way of getting to how broad a set of functions one can approximate with these bumps.

A flaw in Raph’s explanation is that he actually doesn’t have ways to explain why he chose particular heights and widths in approximating the function. Take the second to last bump in his graph. This one has a very shallow height, but is rather wide. Why would one see this? There isn’t a real explanation hence one could as well assume that the curve landed there to help approximate the finer structure of the slope to get to the next bump.

But don’t get me wrong. This doesn’t mean that a bump-with-tail model isn’t helpful, or even justifyable. It just means that, because it’s so universal, you really have to take it with a big grain of salt.

Raph for example claims that:

Assuming that the title is equally available everywhere, you can predict the peak from literally three data points, which you can get literally in the first few hours of launch.

But to the best of my knowledge that is really just a hypothesis. And there really are good questions to be raised about it. So can a title, not screw up later? I think it was Tobold who pointed out that Vanguard had a rather uncharacteristic progression (can’t find the link right now, so I may have to correct this), which mostly meant that stuff didn’t quite look like the bump model.

Finally we don’t get an underlaying model explanation for the bump itself. One could try to see it as a probability distribution of some stochastically modeled situation (i.e. it’s complex, so lets pick a randomizing assumption that is valid for our setup at least in an approximate way). It could be Poisson which has a skewed gaussian distribution like Raph’s bumps, but the model there talks about onset probabilities of events happening (people being enticed to buy and subscribe to the game) or a Weibull distribution (which models things failing over time with some probability. i.e. people losing interest in the game for some factor). Weibull is more attractive I think but we really don’t know. It could well be both. But a Weibull model would not justify Raph’s claim that the first couple of hours determine everything of the bump, because failure reasons actually depend on the game’s desirability as it operates. The advantage of these models would be that one could give tangible reasons for the bump shape. Why was it flat, why was it high etc. We are lacking that.

But anyways, bump model or not. The WoW decline is an interesting story and the blogsphere has brought up numerous arguments: bump shapes, people holding off till after the expansion, the nature of the expansion itself, releases of other games like LOTRo. But as with any data analysis we’ll really have to wait and see more evidence. Does the increase in LOTRo qualitatively match a decline in WoW. Have there been questionaires explaining the cancelation of subscriptions (Blizzard actually does have exit questionaires) etc etc. For now it’s a happy place because we can speculate on one graph. MMOGData for example doesn’t yet carry any information of a decline of subscriptions for WoW, so it really is just one source.

So there really is a lot of noise and not that much signal yet.


%d bloggers like this: