An oddity of consumer behavior: The more confident you are, the less people believe you?

Several of our clients and members of the NPS Loyalty Forum are loyalty leaders in their respective industries.  With relative Net Promoter Scores ranging from +10% to +30% (if you don't know what a relative Net Promoter Score is, please see this prior post), these companies are looking to get even more business impact from their loyalty leadership.  So we have been doing a bit of research into how recommendations work, how recommendations impact a brand's relative positioning, and the impact of relative NPS on advertising effectiveness.

In the course of that research, I've been trying to read some of the academic literature on the topic, and ran across a few interesting studies.  The most interesting of these is an article published in the April 2010 Journal of Consumer Research.  In this article, Uma R. Karmarkar and Zakary L. Tormala of Stanford's Graduate School of Business reach a counterintuitive conclusion.  In their words:

One potentially compelling prediction is that source certainty generally enhances the persuasive impact of a message.  The logic would be that expressing certainty increases one’s perceived credibility.

...

Click to read more ...

The dangers of tying compensation to customer feedback

If you are really serious about creating a culture of customer advocacy, then you will certainly tie incentive compensation to customer feedback.  Right?

Wrong.

Inherent risks and unintended consequences

Anyone who has ever experimented with incentive compensation, knows it can be a powerful way to create organizational focus.  Done right, incentive compensation can reinforce the company's values and motivate specific actions designed to meet important goals.  Done wrong, the unintended consequences can be devastating.  Some of the most important risks: 

• Extreme focus on the score:  With incentive compensation, the saying goes, you get exactly (and only) what you pay for.  If you give your team goals based on scores, they will focus on those scores.  How will this manifest itself?
  • They may criticize the data collection methodology and score calculation.  
  • They will probably spend way too much time explaining and rationalizing the scores.  You will hear things like "problems with our vendors impacted our customers and depressed our scores" or "our score was low last month because it rained a lot and made the customers depressed."
  • They will complain that the scores measure things outside of their control, and therefore the feedback collection methodology should be modified to focus on things only under their direct control
• Gaming:  The logical extension of extreme focus on the scores is active gaming -- cheating.  This takes several forms.  The two most common:
  • Selective surveying.  In some companies, front line employees can "bury" customers who they know will give negative feedback by recording inaccurate contact info or flagging them as having opted out of surveys.  Similarly, they may explicitly ask only happy customers to answer the survey they will get.  At one retailer, I noticed a clerk would use a Sharpie pen to circle the survey invitation at the bottom of the receipt, look the customer in the eye, and say "It would really help me out if you would fill out our survey.  If you mention me by name, I'll get an award."  She only did this for a subset of the customers, however.  Guess which ones.
  • Misguided attempt to influence the SCORE instead of earning a customer's loyalty (click to enlarge)Begging for scores.  The classic example of this is the car dealership service department that posts a huge sign next to the cashier showing the satisfaction survey with a big bold circle around the top score.  Or the car salesman who, after what was an otherwise good experience, pulls out the survey and tells you that if you don't give him a 10 on these two key questions, he will lose his top status and his kids will starve.  (For a great example running all throughout a company, see the related post: Begging for scores at Hilton.)
• Disaffection and disengagement:  If you don't have a stable, reliable metric before you begin working it into your compensation system, get ready for a real firestorm.  Two major issues can create this sort of situation:
  • Score instability.  Problems with sample selection, changes in the mix of customers by segment or touchpoint or product, differences in response rates caused by survey methodology changes, or even inadequate sample sizes can create instability or even plain unreliable scores. 
  • Inadequate or immature understanding of loyalty drivers.  Early in a typical closed-loop feedback process, front line employees, supervisors and the management team struggle to understand what creates positive and negative changes in scores.  Frankly, it's pretty unusual for employees to trust the scores early in the life of a new metric system.  Until your team has some sense of control and influence over the outcomes, a compensation system based on a new score may seem unfair or arbitrary.  Result:  anger, finger-pointing, defensiveness.  Anything except the intended focus on the fundamental goal of creating more Promoters and fewer Detractors.

Conclusions

Please don't misinterpret my objective or the implications here.  I believe most companies should ultimately work customer feedback scores into incentive compensation systems.  Based on our experience at Bain and with the NPS Loyalty Forum, we have found that if the senior-most executives are explicitly and publicly on the hook for improvement, they pay more attention to the issues.  It also sends a great message to the organization about what the company values.  If not, lots of important opportunities will be missed and the company's efforts may even be at real risk of failure.

Nevertheless, Bain clients and others have had experiences that demonstrate the need for caution.  Before including any form of customer feedback scores in compensation, especially for front line employees -- the ones interacting daily with your customers -- you need a stable, reliable metric that people understand and trust.  Moreover, gaming is serious stuff, and so you need protections against it.  It erodes customer trust and ultimately undermines the entire intent of your efforts to improve customer loyalty using closed-loop customer feedback.

Next time: Some constructive advice about HOW to use feedback scores in incentive compensation

Good try, ScanCafe, but... not quite

When my father passed away, my stepmother sent me six boxes full of 35mm slides.  The majority of the photos were from my childhood, and while not all have significant sentimental value, I figured it would be nice to look through them.

First, I went on eBay and purchased a used slide projector.  Great idea.  It was inexpensive, arrived in just a few days, and it worked quite well.  But a few impediments quickly became apparent.  Number one, I don't have a projection screen.  Two, setting up a projector, hanging a sheet (that was the quickest solution), darkening the room and then getting the presentation going took a long time.  Third, the kids just didn't want to sit through the lengthy process of going through a bunch of slides one at a time at my pace.  Finally, even I didn't enjoy needing to go through dozens of slides to find the one or two I really liked.

So next, I read up on ways to get slides scanned.  Scanning them myself was quickly ruled out.  I would have to buy special scanning equipment, and it became apparent from my online research that quality could be an issue.  Most important, I simply don't have time to babysit a slide scanner and do the necessary edits and adjustments.  Offshore scanning services were the way to go, and a few companies seemed like they had consistently good reviews, with one of them -- ScanCafe -- standing out as the clear best value.

The ScanCafe website makes it very easy to set up your order and even creates a mailing label for you.  When you create the order, you tell them how many slides you think you have (I had nearly 5,000), how many slide trays they are in (for me, it was around 25), and then they calculate an estimated cost.  After you enter some info and they bill your credit card for 1/2 of the estimated price, the mailing label can be printed and you're good to go.

Click to read more ...

Closed loop feedback videos - by popular demand

Several people have asked me to make the videos we created to go along with our Harvard Business Review article easily accessible, so I'll embed them both right here.

Closing the customer feedback loop at Charles Schwab

Link to KnowledgeVision Player version (including accompanying PowerPoint slides)

Net Promoter® Score: A Primer

Link to KnowledgeVision Player version (including accompanying PowerPoint slides)

Relative NPS: How to address the most common NPS misconception

Ron Hollis is the CEO of Quickparts, a company that makes custom-designed parts.  I first learned about them in this Fortune Small Business article last year, highlighting their use of NPS. (The full article is here.)  In addition to founding Quickparts, Ron also wrote a book on design called "Better be Running."

The most common NPS misconception

I recently ran across Ron's blog.  In it, he describes the company's use of the Net Promoter approach to closed loop feedback.  He introduces the topic in a blog post (which you can read in full here). I suspect it is from about a year ago and describes mostly how they calculate the score.  In itself, this isn't all that interesting.  In fact, it appears they use a 1-10 scale rather than a 0-10 scale, and he doesn't make any reference to how they collect the data, from which people in the customer organization they collect the feedback, or what else they ask beyond the simple likelihood to recommend.  So it comes across, if you read only that blog post, as if they just calculate a score and that's all.

He also makes the common mistake of crowing about what we would call "Bottom-up" Net Promoter Score -- collected only from Quickparts customers with a sample and methodology that can't be verified.  Such scores are notoriously unreliable as indicators of a company's overall performance because the sampling methodology relies on existing customers, introducing a bias because it excludes customers who chose not to do business with Quickparts -- or who did business once or twice but were dissatisfied and switched to a competitor.  In some cases, this sample bias is also compounded by how and when the survey invitations are triggered (say, immediately after some sorts of interactions, but not others).

Click to enlargeFar more reliable for comparing scores is what we call a Relative NPS, collected through a "Top-down" process.  You survey a representative sample of all the customers in the market -- whether they do business with you or with one of your direct competitors -- and compare Net Promoter Scores of each relevant competitor.  The "rNPS" for your company is your own absolute NPS minus the NPS of your best competitor.  If your competitor has a higher NPS than you, then your rNPS will be negative.

It is crucial not to stop there.  You MUST also ask why.  You must understand the drivers of difference for your company's performance versus the competition.  The rNPS approach allows you to understand which factors contribute to differences versus competitors and how much.

Calculating Relative Net Promoter Scores requires a little more cost and time, but it's important.  It requires you to be thoughtful about defining the relevant competitive set.  You need an unbiased sample and the sample size needs to be adequate for the cuts you want to make.  You need to survey customers "blind" (without identifying your company as the source of the survey, which can introduce additional response bias), and you need to use a common methodology to compare your company to the competition.

Closing the feedback loop to unlock action

Ron wrote a more recent post, however, demonstrating that they do more than just calculate a score. Click to enlarge This one's a lot more interesting -- you can read it in full here -- because he shares the verbatim feedback from the customer and then describes his company's response.  While this single anecdote doesn't illustrate the full power of a closed-loop feedback like NPS, it at least gives a hint that Quickparts is doing far more with it than just calculating a number and keeping score.  An excerpt from his blog post:

True closed-loop feedback unlocks opportunities to improve individual customer relationships in addition to measuring versus the competition.

QuickParts may be making the mistake of reporting bottom-up NPS externally.  But at least they are creating a true closed loop with customers.

Update on Friday, April 2, 2010 at 7:39PM by Rob Markey

Another company -- one I respect a great deal -- recently fell into this same trap.  1-800-GOT-JUNK's CEO was quoted by Fox Business reporter Nancy Colasurdo as saying:

“Our score right now is 80, which is greater than Apple (AAPL), Amazon (AMZN), and greater than some other companies that we dwarf in comparison to,” Scudamore said. “And we’re proud of that.”

Of course, he's comparing "Apples" to "Junk" -- top-down Net Promoter Scores collected by Satmetrix in a piece of double-blind market research versus bottom-up scores collected from his own customers.  Oh well.

For the full text, see the article:  No 'Junk' in This CEO's Philosophy