Survey With Integrity

Avoiding Inflated Satisfaction Scores:
Three Common Pitfalls

Has a service provider ever handed you a customer satisfaction questionnaire when you were clearly satisfied, but not on other occasions when you thought the transaction did not go so well?  If so, alarms should go off in your head:  Warning!  Satisfaction scores are being inflated!

In all organizations, forces are at work to artificially inflate satisfaction scores:

• Service and support want to be able to report record high scores for quarterly performance reviews.

• Marketing wants to claim their company is "first in customer satisfaction"

• Executives may want sound bites such as "99% of our customers are satisfied" to publicize to investors and market analysts.

Incentive compensation, management pressure and peer pressure can result in flaws in survey methodology that lead to inaccurate measures of satisfaction and loyalty and to faulty customer intelligence.

Temptations and opportunities to artificially raise customer satisfaction scores exist even in the healthiest organizations.  In this article, we highlight three such pitfalls:  biased sampling, biased scores, and biased conclusions.  We show how managers can address these pitfalls and help create organizational cultures that welcome honest, objective feedback from customers.

Pitfall #1: Biased Sampling

As a general rule, once a customer group has been targeted for feedback, every customer in that group should have an equal opportunity of being sampled.  Customers should be sampled randomly, either from the overall population or from defined subsets (stratified sampling).

In reality, though, developing a survey database generally requires collaboration with colleagues who may have a vested interest in favorable scores. Suppose you decide to use a stratified sample. You elect to survey 100 respondents from your largest ten accounts, 100 respondents from your middle 50 accounts, and 100 respondents from your smallest 1000 accounts. If the teams that manage these accounts can handpick which customers will be surveyed from each group, the temptation to select their most satisfied customers may be irresistible.

Even when all customers are invited to participate (a “census”), account teams can simply decline to invite dissatisfied customers, or invite without reminding or following up. This behavior may take the form of not providing email addresses or phone numbers for invitation lists, not providing customers with survey web addresses, or not delivering paper questionnaires.

Without doubt, coordinating client communication with account managers is legitimate and important, especially in high-touch  environments where client communications is often best channeled through a single individual.  Financial services firms are properly sensitive about inundating high net worth clients, in particular, with too many communications.  Nonetheless, to ensure that clients are being well served, an independent party should be responsible for gathering feedback on the relationship.  Deferring to the account manager on the exact timing of the survey can help alleviate his or her concerns about surveying during a difficult period in the relationship.  But ultimately, the process should ensure that the voice of every customer has an opportunity to be heard.

To ensure integrity of the sample:

• Assign the task of participant selection to a manager outside of normal reward channels, an independent third party, or combination of the two.

• Aim for a representative sample of the customer population. That means randomly selecting participants from each of the major customer categories relevant to your survey objectives.

• If a census is used, be sure all invitees receive the same degree and type of encouragement to participate.

Pitfall #2: Biased Scores

A common way to bias mean scores and box percentages is to eliminate “outliers” – respondents whose scores are extreme. Unfortunately, pressure to exclude extreme scores is usually concentrated at the low end of the scale. If extreme low scores are excluded, why not exclude them at the high end also? For academic statistics, excluding outliers may be justified, but where corporate interests are at stake, today’s outlier may be the harbinger of a major problem in the making. Dismissing the concerns of a few outliers also leaves the company vulnerable to word-of-mouth effects of disgruntled customers who broadcast their dissatisfaction to friends and co-workers (referred to as “Antagonists” in CustomerSat Positioning Charts).

Managers may find other reasons to exclude specific unfavorable responses:

• We sent the survey to the wrong contact.

• This is not my customer.

• The customer had a technical crisis that day, so the survey is invalid.

• The customer is up for renewal and they are using low ratings as a bargaining chip.

We have heard all these “explanations,” and sometimes they are valid.  As always, the reasons for unfavorable ratings must be understood on the customer’s terms, not in terms of what the individual or group being rated thinks the ratings mean.

In general, we advocate retaining all surveys in the response database and including their ratings in indexes and other calculations.  When the sample size is large, a few extreme scores will not dramatically affect the overall picture.  When the sample size is small, even a few extreme scores merit close attention. In either case, characteristics of low-scoring respondents should be analyzed to determine whether specific customer segments are being underserved or whether their responses signal a trend in the making.

Pitfall #3: Biased Conclusions

Under pressure to make the best of whatever results are available, managers and executives may make seemingly legitimate but ultimately harmful decisions about how to interpret and report customer feedback.

For example, a senior executive may exert pressure to define a score of 5 or higher on a 10-point scale as "satisfied."  In fact, a 5 or 6 usually suggests indifference.  5 is actually below the midpoint of a 1-to-10 scale, and many CustomerSat clients use a rating of 5 or even 6 to trigger and "alert" indicating a problem in the relationship.  Research indicates that customers who respond in this range are at risk of becoming former customers.  Lumping these respondents with truly satisfied and potentially highly loyal customers is almost certainly misleading.

In one case, the CEO mandated use of a three-point scale, “Dissatisfied,” “Satisfied,” and “Very Satisfied.”  With this scale, only the most disgruntled customers gave the lowest rating, so the CEO could legitimately report that 95% of his company’s customers were “satisfied or very satisfied.”  An even more extreme case is the Yes/No question:

Overall, are you satisfied with the service you have received? (choose one)

[   ] Yes

[   ] No

While top-box percentages using these scales will be very high, the information value and actionablility of the results are very low.  Inflated, "feel-good" reports further generate complacency and undermine the credibility of the survey process.  Employees who know that a company does not distinguish between "Satisfied" and "Very Satisfied" responses have less motivation to go the extra mile to achieve excellence.

To address these concerns, ask how customers, prospects, press, analysts, and investors would likely respond if the survey methods or means of computing box percentages were publicly disclosed.  Would the company look less than honest or lose face?  Exactly that happened to CRM supplier Siebel Systems, a factor credited with contributing to that company's decline in recent years.

When a particular customer segment, product line, or region genuinely excels, it is great to publicize the company’s success. For example, a CEO may well boast when survey results show that among her company’s Fortune 500 clients, 85% give ratings of 8 or higher on a 10-point scale.  Such specificity in survey results builds stakeholder credibility and confidence.

Send Employees the Right Message

It is possible to link customer satisfaction scores too directly with individual performance evaluation and compensation.  Although well intended, closely linking satisfaction scores with individual compensation challenges everyone to obtain, use, and report their feedback honestly.  Instead, emphasis should be on organizational learning -- what customers really think and feel about your products and services-- and what employees can do as teams to address concerns.

The Client Loyalty Program at Metavante is specifically designed to fall outside of the company's normal reward channels.  Program manager Pamela Lager tells clients,

"If you give us high scores, I won't get a raise;  If you give us low scores, I won't get fired.  All I care about is your honest opinions of us."

Reinforcing her message are an incentive program for Metavante employees based on response rates as well as loyalty index scores; and clear, consistent support of the Client Loyalty Program from the highest executive levels.

In education, putting excessive pressure on students to get good grades or schools to achieve high test scores can be counter-productive.  The primary objective, learning, can be undermined by a focus on the grade as the end in itself.

Similarly, in customer-serving organizations, the primary objective is service and satisfaction. “Raves” from your happiest customers should be widely shared to encourage and reward employees.  At the same time, customer satisfaction professionals need to ensure that these benefits are not at the expense of direct, honest communication about what your company does well and where it must do better.

Rigorous survey sampling, distribution, analysis, and conclusions go a long way toward ensuring integrity of data and maximum learning from results.

We'd like to hear your thoughts.  Please send them to Newsletter@CustomerSat.com.  For more information, send email to info@CustomerSat.com.