Don't just read the text, pore over the data!

I am often asked what I think of an article that appeared about 18 months ago in the Journal of Marketing regarding the link between revenue growth and NPS.  My simple answer:  the data and analysis in that article support our own findings that Net Promoter is statistically just about as good as anything else out there.

But that is not the whole story for three reasons: 

• Their analysis may or may not be statistically sound, but it is managerially and strategically unsophisticated
• It is based on data of questionable value for this purpose
• They miss the whole point of NPS, which is that while it is “good enough” as a predictor and explainer of relative competitive growth, its real strength lies in how powerful it can be as a tool for organizational change

Because the article’s data analysis, based on Norwegian consumer surveys, falls far short of what we can do with our clients' data, it fails to support (or challenge appropriately) what the most sophisticated NPS users all know:  done properly, Net Promoter Scores explain a lot of the variation in competitive organic revenue growth rates.  Read that last sentence carefully, because the language is quite specific.  In fact, if you are going to reduce your customer metrics down to a single question, the Net Promoter methodology is most often the one that does the best job.

Our analysis on behalf of clients typically uses a careful measure of organic growth.  In retail, it is typically same store sales growth within regions that face similar competitors.  In other businesses, we will take care to reverse out the effects of M&A activity over time.  And when we're working with our clients, we'll make sure we aren't looking at the top line growth rate of a large multi-industry conglomerate and comparing it to a single-industry competitor.  Finally, many of our clients have followed the methodology described in depth in The Ultimate Question, in which we validate which specific single question ("recommend" or something else) does the best job of sorting customers into Promoters, Passives and Detractors. 

Unfortunately, the text of the Journal of Marketing article is written without regard to the underlying data in its own data tables.  The authors make extreme criticisms of Fred Reichheld's work by reducing it to a single, out of context quote which they attack relentlessly.

"rather than establish a set of theory-based hypotheses, as is common in most scientific investigations, we test the overarching claim regarding Net Promoter—namely, that Net Promoter is the “single most reliable indicator of a company’s ability to grow”

Two big issues with the Norwegian satisfaction study data and analysis: 

• Of the five industries examined, three are retail businesses in which same store sales is the most reliable measure of organic growth.  But the study made no attempt to use that relatively readily-available data.  In spite of their poor analytic methodology, the data in the article shows that for these industries NPS proves just about as reliable, in the authors' analysis, as anything else they examine.
• One of the Norwegian industries, transportation, consists of a heterogeneous collection of random companies, many of which are regulated monopolies (where the impact of customer loyalty is dampened dramatically by market structure).  In these businesses, no measure of customer loyalty or satisfaction correlates with revenue growth, even by the authors' own analysis.  NPS is not alone.

In sum, the data and analysis in the article actually support Reichheld's claims, even though the text attacks Reichheld ad hominem along with his conclusions.

The attached couple of pages include an excerpt of the primary data table from the paper.  If you read the paper, please pore over the data table and see for yourself whether you agree with the authors' claims.

Summary of issuesData table