When composing surveys, should the “overall satisfaction” question go before or after the specific performance metrics (attributes)?  

What the Experts Say

First, let’s define terms.Examples of specific performance attributes include product and service quality, speed, ease of use, courtesy, technical skills, and reliability.  Survey questions such as “overall satisfaction”, “willingness to recommend” and “likelihood to re-purchase” are variously referred to as business outcomes, business objectives, or dependent variables.  We’ll call them outcome questions. 

Generally, outcome questions are not by themselves directly actionable.  Scores on outcome questions reflect and are composites of customer attitudes towards and scores of specific performance attributes.  Outcome questions nonetheless serve three important purposes: 

1)  They are high-level indicators that may be trended and monitored, either individually or as composite indices, over time; 

2)  When used with open-ended questions such as “Why or why not would you recommend?” they may be probed to help guide responsive action (more on this below); and

3)  When used with correlation analytics, they allow the importance of specific attributes to be gauged (more on this below).

CustomerSat Professional Services experts recommend placing outcome question(s) after the performance attributes if any of the following conditions apply:

• Customers deliberately evaluate attributes when making purchase selections or re-purchase decisions

• Some customer experiences covered by the survey are not recent

• Correlations with one or more outcome question(s) will be used to derive attribute importance.

If none of the above conditions apply, outcome questions may be placed before the performance attributes.  Let’s examine the three conditions:

Customers Deliberately Evaluate Attributes when (Re-) Purchasing

In these cases, by placing outcome questions after attributes in these cases, surveys better reflect the thought processes customers use when making purchase selections and re-purchase decisions, and the outcome questions will better reflect the conclusions of these thought processes.  This guideline applies to most business-to-business (B2B) investments, as well as high-ticket consumer purchases such as cars, cameras, and electronics.  It applies least to B2C “impulse” or other purchases, such as cigarettes, film, and shampoo, where brand is often the primary differentiator.

Some Customer Experiences are not Recent

Customer recollections of product and service experiences may be hazy if they occurred some time ago.  Asking the attribute questions before the outcome questions helps refresh the customer’s memory.  Ideally, the attributes should appear in the survey in the same order in which the customer experienced the attributes, helping the customer re-live the experience when completing the survey. 

For automated transactional surveys that gather customer experience feedback soon after transactions have taken place, the customer experience is usually very recent.  For relationship surveys, which are conducted periodically and which cover experiences over a longer time interval than transactional surveys, part or all of the experience may have occurred some time ago.  In these cases, we particularly recommend putting the outcome questions after the attributes. 

Outcome Questions are used to Derive Attribute Importance

If the attributes accurately, fully, and with minimum overlap capture what is important to the customer about the product or service, the responses to outcome questions (“overall satisfaction,” in particular) can be correlated with responses to the specific performance attributes to derive the importance of each attribute. Changes in the scores of the attributes whose responses are highly correlated with scores on business outcomes generally have greater impact on those outcomes and their scores than changes in scores of attributes whose responses are less correlated with those outcomes. The former attributes have high derived importance; the latter have low derived importance. Usually, an organization’s most critical performance attributes are those with both high importance and low performance (e.g., satisfaction) scores, as commonly displayed in quadrant charts using derived importance.

Correlation analytics are most effective when respondents have reviewed and evaluated each of the attributes before registering their overall satisfaction.  If a customer rates all attributes highly except for one, for example, the customer’s rating of overall satisfaction will better reflect his or her perceived importance of that low-scored attribute after the customer has considered each attribute.

Incidentally, correlations between attributes and outcome questions will typically be stronger with “overall satisfaction” than with either “willingness to recommend” or “likelihood of re-purchase” as the outcome.  “Overall satisfaction” depends directly upon the performance attributes, while other factors unrelated to the product or service experience, such as corporate policy or personal budget, may influence “willingness to recommend” and “likelihood of re-purchase.”  Using open-ended probing questions (e.g., “Why or why not would you recommend?”) with outcome questions can help identify these factors.

As summarized in the chart below, we generally recommend placing the overall outcome questions after specific attribute questions.  If the product or service is an "impulse" purchase, the customer experience is very recent, and you either do not wish, or feel it is not meaningful to correlate attributes and outcomes, then outcome questions may be placed before the attributes.

For more information please contact us at expert@CustomerSat.com or call 650-237-3300.  

Honeywell Global Business Services (GBS), a $500 million worldwide business, started surveying customers in 1994.  Over the following decade, the surveys matured from paper to the web and measured over ten thousand customers’ satisfaction with more than a hundred different GBS services.  Despite these advances and long experience, by the year 2001 the surveys presented some major challenges.

Four Survey Challenges

First, the surveys were labor intensive, requiring the manual processing of 30,000 customer names to determine which customers should be invited to complete which surveys.  Second, analysis of the results had to be done in-house by the service owners, who often lacked the necessary time and training.  Third, results became available and action planning could begin only weeks after responses were gathered, resulting in lost productivity and delayed responses to pressing customer issues.  Fourth, surveys were neither customer-friendly nor global: they were limited to English and four other languages. Customers with primary languages different from these five often could not respond, so GBS had no way of knowing how satisfied they were.

Partner Requirements

As a result, GBS launched a search for a customer satisfaction partner that could address these and other challenges.  GBS needed fully integrated and automated survey processes.  In particular, the partner had to be able to automate the labor-intensive process of assigning customers to the right surveys; manage complex survey processes in multiple languages; and deliver self-service solutions with real-time reporting and analytics so that action planning could begin even while survey were underway. The GBS team also needed the flexibility to conduct ad hoc (“pulse”) surveys to measure on-going progress throughout the year. Above all, the team wanted to build a relationship that leveraged a partner’s survey management expertise and solutions that were readily adaptable to GBS’ unique business needs. GBS chose CustomerSat, Inc., who designed a system for GBS with 124 different survey versions and translations in ten languages.  The CustomerSat system for GBS was fully automated from sample selection and e-mail invitations to on-line analytics and real-time results.

Business Rules to Respect Customers’ Time

The CustomerSat system allows quotas to be set for each service group based on its total customer population and the number of respondents required for statistically significant results.  Since a random sample and a sufficient number of responses are necessary to achieve statistical confidence, the survey system has to be dynamic. When an individual receives an emailed survey invitation and clicks on the URL link embedded in the message, a web survey is presented based on that individual, the quota required, and the numbers of responses that have already been received at the time the link is activated.  A customer eligible to participate in ten surveys is typically presented with just one—that survey for which his or her response is most valuable, based on the quota and number of responses received for each survey as of that time. This system allows GBS to spend much less time generating and managing sample lists.  Most importantly, it allows GBS to respect customers’ time and to minimize the number of surveys any one customer is asked to complete in a particular time interval. GBS gets the detailed feedback it requires while asking fewer than ten percent of all customers to complete more than one survey.

Closing the Loop with Customers

In addition to gathering feedback online, CustomerSat provides GBS service owners with web-based dashboards, enabling the owners to easily analyze feedback and identify problem areas earlier.  CustomerSat professional services staff further provides detailed analyses of survey results.  Together, these tools and services make action planning highly efficient and effective. They also enable GBS to build closed-loop processes with customers, enabling service owners to identify and prioritize key actions and share those actions with customers. 

• New White Paper:  A detailed, step-by-step guide for designing systems that yield satisfaction and loyalty benefits quickly, reliably, and economically

• Free webinar: Tuesday, March 25, 1 - 1:45 pm PST

The new CustomerSat white paper, Designing Enterprise-wide Real-time Feedback Systems, describes the CustomerSat Design Methodology, a proven approach to designing enterprise-wide feedback systems that CustomerSat professionals have developed and refined over nearly a decade.  Adhering to this methodology will help customer satisfaction professionals deliver satisfaction and loyalty benefits of real-time feedback systems rapidly, reliably, and economically.    

Optimal design of enterprise-wide feedback systems, unique among information systems, involves considerations not just of IT and corporate strategy, but also of statistics, sampling, market research, and human behavior.  The CustomerSat Methodology encompasses these, and includes such steps as: 

• Identifying and prioritizing customer segments and touchpoints that affect the enterprise’s success

• Determining the right performance metrics and composing surveys tailored to each key customer segment and touchpoint

• Mapping feedback to and designing interactive dashboards that allow employees throughout the enterprise to analyze survey results online

• Determining touch rules that make optimal use of customers’ willingness to provide feedback and protect customers from being over-surveyed 

• Designing action management processes that efficiently respond to customer concerns raised and market opportunities revealed by feedback.  These processes include alerts, case assignments, escalations, and reaching closure with customers.

Common problems avoided by the CustomerSat Methodology, when coupled with CustomerSat solutions and professional services, include:

• Inadequate or no feedback gathered about major elements of an enterprise’s value proposition

• Customers surveyed more frequently than desired, or on issues about which they lack the information or experience to respond

• Customer confidentiality not respected; confidential comments about personnel performance inappropriately distributed 

• Feedback hoarded by a single department, or too broad or too detailed to be meaningful and actionable 

• Results delivered long after feedback is received, after customer loyalty has been damaged.

CustomerSat release 5.0 automates enterprise-wide customer touch rules

CustomerSat 5.0 touch rules conserve, and maximize the benefit enterprises derive from, customers’ time and attention in answering surveys.  In the course of a month, an enterprise may invite customers to complete surveys from headquarters, technical support, web operations, market research, field service, and other divisions and departments.  These surveys provide crucial information to the enterprise, but in aggregate, they can overwhelm customers.  CustomerSat 5.0:

• Protects customers from being over-surveyed

• Ensures that customers are asked to complete the surveys for which their feedback is most valuable

• Presents a consistent look-and-feel to customers for all surveys across the enterprise

• Ensures that sufficient customers are invited to make results statistically significant 

• Stops surveying customers, if desired, after statistically significant numbers of responses have been gathered 

• Controls and automates the acknowledgement and follow-up with the customer, if applicable, after feedback is provided.  

Jonathan Clay joins CustomerSat as VP, Technology

Upcoming Events: 

March 19 - 21
Microsoft Convergence 2003
Gaylord Palms Convention Center
Orlando, FL
Booth #38

March 20 - 21, 2003
PeopleSoft CRM Meeting
Atlanta, GA

March 31 - Apr. 2
SSPA Conference: 
Delivering World-Class Service in Challenging Times
San Diego, CA
Session:  Survey Right: Innovative Survey Technology Controls Customer Touch Rules for Effective Enterprise-Wide Survey Deployments, Jonathan Clay, VP Technology, CustomerSat

For more information please contact us at expert@CustomerSat.com or call 650-237-3300.