It seems to be open season on Fred Reichheld. For many years, his concept of Net Promoter Score as a critical predictor of business success has been questioned by marketers. The Internets are now buzzing with a recent academic study “A Longitudinal Examination of Net Promoter and Firm Revenue Growth” (Timothy L. Keiningham, Bruce Cooil, Tor Wallin Andreassen, & Lerzan Aksoy, Journal of Marketing, July 2007) that duplicated Reichheld’s research but “fails to replicate his assertions regarding the ‘clear superiority’ of Net Promoter compared with other measures in those industries.” See, for example, comments by Adelino de Almeida, Alan Mitchell, and Walter Carl. I didn’t see an immediate rebuttal on Reichheld’s own blog, although the blog does contain responses to other criticisms.
There’s a significant contrast between the Net Promoter approach – focusing on a single outcome measure – and the Balanced Scorecard approach of viewing multiple predictive metrics. I think the Balanced Scorecard approach, particularly if cascaded down to individuals see the strategic measures they can directly affect, makes a lot more sense.
Thank you very much for writing about the research I conducted with Lerzan Aksoy (Koc University), Bruce Cooil (Vanderbilt University), and Tor Wallin Andreassen (Norwegian School of Management) that appears in the current issue of the Journal of Marketing. I would like to provide a little additional insight regarding our results.
ReplyDeleteThere are two key findings from the research:
1) We did not find Net Promoter to be a good predictor of growth at all.
2) We found very strong evidence of research bias in the research reported by Reichheld in support of Net Promoter. In particular, we were able to replicate a subset of Reichheld's reported data for his best case scenarios and compare it to a metric he claimed was examined and found to have a 0.00 correlation to growth, the ACSI. Our findings clearly show that when using Reichheld’s own data, Net Promoter wasn’t superior to the ACSI. It is virtually impossible to imagine a scenario other than research bias as the cause of this finding.
This is a very serious problem. We expect research published in our most prestigious journals to be free of bias in management science, just as we do in all other fields of study. We would not consider this kind of problem acceptable had the research been conducted in medical, psychological, or physics research; the same standards apply in management science.
Managers have adopted Net Promoter based upon the belief that solid science underpinned the claims attributed to the metric. In fact, there would have been no Harvard Business Review paper introducing Net Promoter without the research. This also has serious implications regarding the credibility of Reichheld's book, The Ultimate Question. Additionally, biased "research" that is published in a prestigious management journal contaminates not only management practice, but also management science, as it will be used by scientists as a basis for future research.
Therefore, it is vital that we not be apologists or revisionists when it comes to issues of research bias. Our credibility must never be in question; the truth matters.
While I know that there are firms who claim Net Promoter is doing amazing things for them, testimonials aren't proof. The truth is, as one anonymous blogger wrote on the subject with regard to Net Promoter: “The worst management fads always have testimonials from well-known companies (reengineering, CVA, etc.); that’s how these bad ideas became management fads in the first place.”
It wasn't that far back that CVA (customer value analysis) held the same position that Net Promoter currently holds. The testimonials from leading firms were no less impressive than those currently espoused for Net Promoter. Now almost no firms use CVA since it was shown not to work as promised. And the science underpinning CVA far exceeds that associated with Net Promoter; nonetheless, it was not the promised Holy Grail.
Regardless of whether or not one chooses to believe in Net Promoter, however, we all must insist that the evidence used to support the metric be unbiased. Research bias is totally unacceptable and inexcusable.
Tim Keiningham
Hi Tim,
ReplyDeleteThanks for the clarification. I'm a bit puzzled by your reference to CVA, however. I don't recall a specific claim for how CVA would "work as promised". What did you have in mind?
David
Sorry for the obscure CVA reference. CVA was designed to use attitudinal information collected in a survey to show a company’s relative position vis-à-vis competition (what was referred to as a Market Perceived Quality Ratio). Bradley Gale’s book, Managing Customer Value, has numerous reported linkages between CVA and firm performance (e.g., market share…“Going to School on Customer Value as the Key to Market Share,” p. 83+).
ReplyDeleteCVA was widely used and advocated by some of the most highly regarded firms of the time (e.g., AT&T). Unfortunately, there were several problems with the fundamentals underlying the CVA calculations. Terry Vavra and I point some of these out in our book, The Customer Delight Principle (pp. 41-44). Ultimately, CVA fell out of favor, as its linkage to market performance did not hold.
My comments should not be taken as a slam against Gale’s work on CVA. It represents seminal work in the areas of service quality and customer loyalty. Being first, however, seldom results in these ideas being the last word on the subject.
Oh, that makes more sense. Thanks.
ReplyDeleteDavid,
ReplyDeleteI have read many of the blog postings regarding the claims that the Net Promoter research was biased and subsequently read in detail the Journal of Marketing paper. I would like to add the following to the conversation.
1. It appears that the Journal of Marketing research referred to shows that Net Promoter is, at a minimum, equivalent to ACSI in correlating to growth. (Figure 1). That says that a simple metric driven by a single question is at least as accurate at predicting growth than a more complex algorithm driven by multiple questions.
2. After discussing the claims with Satmetrix, they stand behind the research. Claiming a research bias is a pretty serious allegation. Both Fred and Satmetrix have spent over 10 years studying the correlation between loyalty and growth and helping companies implement programs that improve loyalty and drive growth. The result of this effort is apparent in reviewing the case studies presented at the Net Promoter conferences: http://netpromoter.typepad.com/npc_london_2007/
3. This conversation is missing the point that is attracting business leaders. The value of Net Promoter is its simplicity. Unlike complex satisfaction indexes, Net Promoter is easy to understand and take action on. Simplified surveys drive higher response rates, a better reflection of the customers that matter, rather than random sampling. Using real-time reporting, leaders can get information in the hands of employees that can address detractors, move the passives and nurture the promoters.
At the end of the day, it’s what companies do to improve loyalty that drives growth. Net Promoter offers an approach that is understandable by everyone in the business, not just the statisticians. This gives an organization a rallying cry for building customer centric organizations.
Full Disclosure: During this conversation I was contemplating joining the Satmetrix team. Having experience deploying Net Promoter programs, and researching the market further, I decided to join Satmetrix as CMO: http://www.satmetrix.com/news/pressrelease_2007-07-17.htm
Let's continue the conversation.
Thanks for disclosing your affiliation, Deborah.
ReplyDeleteI'm sure Tim and Fred can defend themselves. But I do have some observations on your comments:
point 1: According to Killingham, Reichheld said ACSI had "0.00 correlation to growth". This is different from your comment that "Net Promoter is, at a minimum, equivalent to ACSI in correlating to growth" (unless you're saying there is no correlation for either, which I don't take to be the case.) So you have not contradicted Killingham's critique.
point 2. that Fred and Satmetrix have "spent over 10 years studying the correlation between loyalty and growth" doesn't address the issue unless we know what they found. I haven't looked at the cases presented at the conferences but would guess they are not academic peer-reviewed studies of the correlation in question.
point 3. Although Net Promoter may or may not be easier to calculate than ASCI, the resulting values are equally simple. You have a stronger point that Net Promoter can be calculated for individuals and acted upon. But it makes sense to "address detractors, move the passives and nurture the promoters" whether or not there is a long-term correlation between Net Promoter and growth.
In fact, I think the last point is really the key one from the Satmetrix point of view: you allow companies to capture the attitudes of specific individuals and react appropriately. That provides value regardless of what Net Promoter does or doesn't do.
- David
Thanks for your blog. While, as you may note, that "Net Promoter Score as a critical predictor of business success has been questioned by marketers," it's not being questioned by numerous companies who are benefiting from their Net Promoter deployments.
ReplyDeleteFor any of your readers interested in learning about Net Promoter in the real world, please check out the Net Promoter Conference blogs (from the London and New York events). Multiple bloggers have written up key takeaways from ~ a dozen sessions on the successes of companies who presented such as T-Mobile, LEGO, HSBC, Schwab, Intuit, Philips, GE, T-Mobile, IBM, and more.
Here's the link to the most recent conference blog:
http://netpromoter.typepad.com/npc_london_2007/
It's also interesting to see the emergency of hiring managers looking for people to manage Net Promoter programs. We have recently launched the Net Promoter Customer Loyalty job board, which has a mix of NP-related jobs, and others that are customer loyalty related. The site can be found off of www.netpromoter.com.
Ms. Eastman’s statement that research bias is a serious allegation is absolutely correct. So we are waiting for a good answer. The argument that Net Promoter is “at least” as good as the ASCI is irrelevant. Reichheld claims his research reveals no relationship between the ACSI and growth.
ReplyDeleteIn the Harvard Business Review article that introduced Net Promoter, in the book, The Ultimate Question, and in presentations regarding Net Promoter, the American Customer Satisfaction Index has been specifically mentioned as not linking to firm growth by Reichheld. The book, The Ultimate Question, argues that the ACSI does not yield much insight into loyalty or growth, noting that “investors rarely waste money on standard satisfaction surveys” as a result (The Ultimate Question, p. 86).
Similarly, an article in the Harvard Business Review states (p. 49): “Our research indicates that satisfaction lacks a consistently demonstrable connection to actual customer behavior and growth. This finding is borne out by the short shrift that investors give to such reports as the American Customer Satisfaction Index. The ACSI, published quarterly in the Wall Street Journal, reflects customer satisfaction ratings of some 200 U.S. companies. In general, it is difficult to discern a strong correlation between high customer satisfaction scores and outstanding sales growth.”
Furthermore, in a web-based presentation, Mr. Reichheld states that a “Bain team looked at the correlation between growth and customer satisfaction, and found there is none.” A scatter diagram was shown with the X-axis labeled “American consumer satisfaction index annual growth” and the Y-axis labeled “Sales annual growth.” The R-square reported was 0.00, indicating no correlation whatsoever.
Given that our findings show that Net Promoter was not superior to the ACSI when using Reichheld’s best-case scenarios, it is virtually impossible to imagine a scenario other than research bias as the cause. This is a VERY SERIOUS problem. We expect research published in our most prestigious journals to be free of bias in management science, just as we do in all other fields of study. We would not consider this kind of problem acceptable had the research been conducted in medical, psychological, or physics research; the same standards apply in management science.
Managers have adopted Net Promoter based upon the belief that solid science underpinned the claims attributed to the metric. In fact, there would have been no Harvard Business Review paper introducing Net Promoter without the research. This also has serious implications regarding the credibility of Reichheld’s book, The Ultimate Question. Additionally, biased “research” that is published in a prestigious management journal contaminates not only management practice, but also management science, as it will be used by scientists as a basis for future research.
It is vital that we not be apologists or revisionists when it comes to issues of research bias. Our credibility must never be in question regarding the research we publish in prestigious journals; the truth matters. Therefore, discussions about Net Promoter by researchers (practitioner and academic) must first adequately address this issue. If not, then why do any research at all, as we can simply present the answers we want to believe as supporting evidence and be done with it? Given the evidence we uncovered, however, we seriously doubt that there will be an acceptable answer to the issue of bias in Reichheld's reported research. [Ironically, in The Ultimate Question, Reichheld emphasizes the importance of eliminating bias from research (pp. 106-111).]
Sincerely,
Tim Keiningham
The Net Promoter score does identify the level of Loyalty that might exist in your customer base. But it does not give you the tools to improve your Customer Loyalty over the long run. Brookeside can do that
ReplyDelete