More on: Data and explanation models in MMO subscription analysis

Well there is a lively debate in the commentary to Raph’s MMO subscription analysis blog entry. I wrote another answer and submitted, but since then it’s down. In lieu of saving it for later I thought I’d put it up here for now. Basically the gist of the last two comments were from a poster nicked Harvey who indicated that the actual game content is in his view the reason for the TBC downturn. And Neil who clarifies that of course Bass is just a first order model and really we are looking at a second order model here which accounts for both new and current users.

I’ve been scratching my head over this and mostly have been trying to find relevant sources on the web (and maybe I’ll have to hit a library soon too). So far I’ve been marginally successful. I still try to find subscription retention models (maybe from studying traditional subscription businesses, like newspapers, magazine and subscription TV), but not yet. If you know good sources, just post on Raph’s comments when they are back up :D.

OK so here’s the thing that I wrote up for raph’s blog comments section:

————————————————————————————

I’ve been trying to find a reference for the “second order” but failed so far. Basically it’s seems to me that if you want to model this you want an acquisition/diffusion model (say Bass) and you want a subscription retention model (for which I yet lack any references).

The question that Harvey is basically asking is simply this: Isn’t subscription retention what caused the downturn in TBC (or say Vanguard?) The problem with these curves, as I said initially is though that they too easily match a lot of data. Of course one could make up a Bass-only model that matches the Vanguard shape. Fairly good number of early adopters, but negative diffusion leads to the immediate drop from early adoption like we see it. So why not just use Bass? Well, my point is simply just because data fits a curve doesn’t mean that the curve actually describes the actual dynamics that’s going on there. Bringing this back to TBC, many people who played it would say that the downturn was induced by the state of TBC endgame, not by launch, hence questioning the Bass only idea (note that the peak coincides with many players having reached level cap and entering endgame, and before the quoted summer holidays). Yet again the curve fits Bass-type shapes.

While trying to find MMO relevant subscription retention models I found actually sources that discuss the validity of Bass and other marketing models that are treated as “lawlike” by many. Here is a preprint of J. N. Sheth & R. S. Sisodia “Revisiting Marketing’s Lawlike Generalizations” Journal of the Academy of Marketing Science (27) 1, 71-87 (1999)

Let me quote interesting passages:

The second, and we believe more serious shortcoming, is that these models tend to view the rate of adoption of an innovation as an intrinsic characteristic of a market and the innovation itself, with inadequate consideration of factors such as: “How affordable is the innovation to the market at large? How rapidly does the price-performance ratio improve? How widely available is the product over time?” In other words, many of the areas where managerial action is crucial are ignored by the models. They adopt a static, almost fatalistic view of dynamic, evolving markets.

I agree with the basic sentiment here and hence why I reacted to Raph stating that the first few hours of data determine the whole curve. This indeed is static and fatalistic because it means that little can be done after the first few hours.

They also quote another author who expresses this view on Bass diffusion:

Simon (1994) concurs in this view, suggesting that while the mathematical formulations underlying diffusion models tend to be good at ex post (i.e. after the fact) “predictions” based on historical data, their performance with ex ante data is far less impressive. “Even the most fundamental question of why we should actually be able to predict the further diffusion of a durable product from the first few observations is totally unanswered… there is no reason why the diffusion should follow any lawlike pattern. Are we barking up the wrong tree with these efforts?”

This is analogous to my criticism that this is post mortem curve fitting and not faithful prediction, or at least it hasn’t justified itself as such beyond fitting data post mortem.

The attitude towards these curves is actually important because it can define how evolutions are seen. It could be “this was just the market for this expansion and we see the predictable decline after the model” or it could be “we have to improve our end-game to retain subscribers and the decline was preventable”.

Bass indeed seems to suggest the more fatalistic first stance, while trying to provide longevity in subscription services probably should worry about the second. Have subscription services with longevity like the New York Times or the subscription cable TV followed Bump curves? That’d be an interesting question. And as people have argued about EVE or Runescape, these show longevity (which again one can argue to fit Bass micro-bumpage, but isn’t that curve fitting?)

Holding on to the bump shape tightly also defines how data itself is seen. Data is prone to all sorts of trouble and inaccuracies. However Data remains the main link to a ground truth. Distinguishing inaccuracies from fact can be tricky. Yet on top of that Raph postulates that data that doesn’t match the bump curves is wrong outright. Frankly I’m not convinced that such a stance is justified. In fact it’s contrary to Popperian method of science, because it denies the “falsification” (Popper’s term for disproving) of the bump model hypothesis by denying the validity of any contrary data (sorry for this heady sentense).

Again this really don’t mean that there could not exist a justification for a bump model, just that this justification is lacking. For the same reason if subscription retention plays a vanishing role in MMO life-cycles, it would be good to at least have some explanation for this. For these reasons I’m just skeptical.

Advertisements


%d bloggers like this: