Designing Surveys

Topics

• Determining Objectives
• Action Orientation
• Order of Questions
• Which Performance Metrics?
• Length of Questionnaire
• Question Types and Analytics by Type
• Scales
• Specifying Respondent-Descriptive Variables
• Other Guidelines

Determining Objectives

The goal of any survey of customers, employees, or other stakeholders is the generation of actionable results. Consequently, a number of business questions need to be addressed before composing the questionnaire:

• For which products, services, and processes do we need feedback?
• What kinds of feedback will help us achieve corporate objectives? For example, if our goal is to migrate technical support from phone to the Web, we may want to understand customers’ support channel preferences; their experiences with each channel; and their suggestions to improve the effectiveness of Web-based support.
• What are the ultimate business outcomes for which we want to understand the underlying drivers and which we want to enhance over time? Examples are overall customer satisfaction, employee retention, revenues, and profitability.
• What customer segments do we want to hear from? Examples include those defined by product, longevity of employment, industry, role, and region.
• How will we use and reconcile different surveys across the enterprise?
• Are we interested in competitive comparisons? If so, how do we solicit that input? Do we ask our own customers who have had experiences with our competitors, our competitors’ customers, or both?
• Do we want to include measures of perceptions of our brand? This can be done through open-ended questions, closed-ended questions, or both. For example, we might ask what image our brand conveys (open-ended), or whether respondents agree or disagree that our brand conveys specific attributes (open-ended).
• What kinds of analyses do we want to undertake on the results, and actions do we want to drive, and for which segments? Are we interest in performance by metric, trends over time, prioritizing investment areas, grouping customers in meaningful ways, projecting profits, or a combination of the above? Are we interested in being able to take action by region, distributor, product, business unit, call center, technical representative, or sales representative?

Back to Top

Action Orientation and Accountability

Because the goal is actionable feedback, serious thought must be given to each question included in the survey. Feedback can drive two types of action: tactical and strategic. Tactical actions typically address individual respondent concerns, such as low ratings on specific questions.The responsible individuals or managers should be immediately notified of such responses so they can take appropriate action.

Strategic actions typically address entire business processes or the concerns of entire stakeholder segments. These actions may be triggered by the combination of low performance on an attribute, such as customer support technical skills, and high correlation between that attribute and a key business objective, such as customers’ likelihood of contract renewal. Addressing these actions may require significant investments over time.

Accountability is a key element of actionability. Who is responsible for tactical and strategic actions? Assigning ownership of each question when the survey is designed, and getting “buy in” of that ownership, can ensure that the right individuals, work groups, functional areas, or cross-functional teams will be accountable.

Back to Top

Order of Questions

Surveys of stakeholders typically have the following types of closed-ended questions:

• Performance metrics (attributes) related to the different business processes or stakeholder touchpoints (e.g., customer support or employee benefits service)
• Business objective or desired outcome questions (dependent variables), such as overall satisfaction or likelihood to recommend
• Categorical questions for segmenting results.

In addition are open-ended questions, which most commonly seek to probe, clarify, or expand on responses to closed-ended questions.

We recommend putting performance metrics in the order that stakeholders experience them, if applicable. In a typical customer lifecycle, for example, Sales touches customers first. For certain types of products, Sales is followed by Installation or Implementation. After that, customers experience the product or service directly. In some cases, there may be ongoing Account Management. Customers may have to call customer or technical support if they have questions or problems, or a customer service center or web site for billing questions. Putting the metrics in this order helps respondents “re-live” the experience.

If correlation analytics will be used between performance metrics and desired outcome questions to derive the importance of each metric, we recommend putting the metrics before the outcome questions. Having just “re-lived” the experience, the respondents’ implicit weightings of the importance of the metrics will be better reflected in their responses to the outcome questions, thus making for a more effective correlation analysis.

Generally, outcome questions are not by themselves directly actionable. Scores on these outcome questions reflect and are composites of stakeholder attitudes based on their cumulative experiences.

These outcome questions do serve three important purposes:

• They are convenient, high-level indicators that may be tracked and monitored, either individually or as composite indices, over time
• When used with open-ended questions such as, “Why are you unlikely to continue doing business with Company X?” they provide helpful detail to help guide responsive action
• When used as the foundation for correlation analytics, actions to improve performance by metric can be better prioritized.

In the survey, we also want to present attributes in the order they were experienced within each business process. For example, for Technical Support, first the customer makes the call, then interacts with the representative, and ultimately (we hope) experiences problem resolution. If this does not occur with the first call, the problem may be escalated and several more contacts may occur before final resolution.

Performance measures should be presented in that order, e.g.:

• Time to reach representative
• Responsiveness of representative
• Rep’s understanding of business requirements
• Rep’s technical competence
• Rep’s courtesy and professionalism
• Management of escalation process
• Proactive communication of problem status
• Time to problem resolution
• Quality of problem resolution.

Finally, the survey asks categorical questions for segmenting results. Examples are:

• Demographics, either corporate or personal (e.g., geographical location, vertical market, gender)
• Technographics (e.g., use of Windows vs. Macintosh, wireless service provider)
• Psychographics (e.g., attitudes towards politics, religion, race).

Back to Top

Which Performance Metrics?

Since questionnaire “real estate” is limited, we may need to prioritize metrics to ensure that most important ones are included. These priorities can be based on:

• Past studies
• Marketing/sales input
• Direct input from customers
• Expertise and experience of your survey partner.

If you are uncertain about which metrics are should be included, qualitative research can help.

Different approaches include:

• Interviews of senior executives and managers
• Sessions with groups of stakeholder-touching employees
• WebFocus Groups.

A refined list of performance attributes will emerge from these various qualitative investigations.

As surveys are conducted over time, questionnaires should be modified to reflect changes in your:

• Stakeholders’ requirements
• Products, services, and processes
• Organization’s strategy
• Competitive arena.

Over time, you may wish to probe certain areas in more detail or to remove those metrics that are no longer relevant. This results in a dynamic survey that always reflect current conditions. At the same time, we want to keep the core set of attributes intact so that after the baseline set of data has been established, trends in performance by attribute and segment may be tracked.

Back to Top

Length of Questionnaire

How long should a survey be? The question is as impossible to answer as asking how long a book should be. (“Long enough to cover the subject,” perhaps?) Very straightforward transaction surveys covering a single point of contact, such as a phone center, may have ten to 15 questions. More complex transactional surveys in which transactions involve multiple points of contact – such as phone, e-mail, and Web – might have 25-30 questions. Relationship surveys, either for internal or external customers, which cover all aspects of the relationship between an enterprise and its customers or workforce – often have as many as 50-80 questions.

Ways to Reduce Length

Response rate measures the percentage of stakeholders invited that responds to a survey. Completion rate, in some cases an even more important measure, is the percentage that complete the survey. A major cause of non-completion is survey length: respondents lose interest or get impatient and drop out. Three techniques allow both actual and perceived survey length to be reduced: use of respondent descriptors, skipping and branching, and randomized subsets.

Use of Respondent Descriptors (uploaded variables). Any survey question to which we already know the answer should be treated as a respondent-descriptive variable. Such variables as year of first purchase, service contract level, and gender can be imported into the survey system, typically from a CRM system, HR or marketing database, for all potential respondents before contacting them. Doing so, in effect, “pre-answers” those questions, allowing them to be omitted from the questionnaire as seen by respondents, and effectively reducing the length of the questionnaire.

Skipping and Branching. Online technology makes it easy to ensure that respondents see only those questions relevant to their experiences. Skipping hides questions that do not apply to certain respondents; branching routes different respondents to different sets of questions that apply to them. From a survey logic standpoint, skipping is simply a special case of branching, and the terms are often used interchangeably.

Skipping and branching are based on either:

• Respondent-descriptive (uploaded) variables. If such descriptors indicate that a respondent uses product X and not Y or Z, for example, questions about Y and Z will be skipped over for that respondent. Or, if all customers use either X or Y but not both, users of product X will branch to questions about X and users of product Y will branch to questions about Y.
• Survey responses. If a respondent replies “No” to “Have you tried Product Y?” then questions about Product Y will be skipped over. Or, respondents who rate overall satisfaction a 9 or lower on a 10-point scale may branch to “What single thing could we do to most improve your satisfaction?” but those who rate overall satisfaction a 10 may branch to “May we use your name as a reference?”

Random Subsets. For certain choose-one or choose-all-that-apply questions with many choices, random subsets can be used to select and present fewer choices for each respondent. Random subsets reduce the set of choices to a number that respondents can reasonably complete, while ensuring that every choice gets representative consideration from respondents.

For example, if a computer equipment manufacturer competes or partners with several hundred suppliers, many of whom customers would know, it would be unreasonable to ask respondents to mark every brand with which they were familiar and ask them to complete a battery of questions about each brand. Random subsets can be used to select a smaller number of suppliers – perhaps 20 or 30 – from which list it is reasonable to ask respondents to complete the battery of questions about those suppliers with which they are familiar.

Random subsets may be used at the question level as well as at the choice level. Rather than ask respondents to rate ten competitors on each of 20 detailed performance attributes – a total of 200 questions – we might select a random subset of three or four attributes for each competitor and ask respondents to rate just those.

Incidentally, another form of randomization, random ordering, is used eliminate respondents’ bias towards selecting the first choice they hear or, to a lesser extent, read. Random ordering and random subsets may be used together or separately. Random ordering by itself has no impact on the length of a questionnaire. From the early, pre-Internet days of phone interviewing, random ordering is sometimes referred to as rotation: phone interviewers would read aloud a list of choices to respondents starting each time with the next choice in the list, thus rotating through the list with different respondents.

Together, these techniques make the survey experience both shorter and more enjoyable.

Other tips: In online surveys, if you group common questions together in a table or grid, introductory text can be used to describe the common questions, thereby reducing the length of each question in the table. For complex concepts or longer lists of products, passing the cursor over the question can generate a text box containing detailed definitions or descriptions.

Back to Top

Question Types and Analytics by Type

Each question type and group of questions allow for certain analyses and actions, consideration of which is critical in survey design. For more detail, see Analysis and Reporting and Managing Action. Below, we examine five commonly used question types.

Rating questions are the bread-and-butter of customer satisfaction and loyalty surveys. The most common analyses and statistics for rating questions are:

• Frequency distribution
• Mean and standard deviation
• Top and bottom box percentages
• Segmenting (filtering)
• Cross-tabs
• Trendlining over a time interval.

Frequency distributions show the shape of distribution including number of peaks. Multiple peaks may well indicate different stakeholder segments, and should be investigated by applying filters to pinpoint those segments. Online survey systems allow users to click on values in frequency distribution charts (usually bars in a bar chart) and drill down to the detailed survey responses that contained those values, including associated open-ended responses.

Cross-tabulations segment responses to rating, choose-one and choose-all-that-apply questions by the responses to other, usually categorical, questions. Cross-tabs of closed-ended with open-ended questions are also possible; an example would be to see the comments about how service can be improved “sorted” by the ratings given for overall satisfaction with service. Significance testing shows whether the percentages and mean scores by segment are statistically significantly different from each other.

Rating questions are also used in correlation analysis, which identifies and prioritizes actions based on satisfaction score and derived importance. Individual metrics are identified as independent variables and correlated with a rating question designated as a business objective or desired outcome (dependent variable), such as overall satisfaction. Correlations identify likely drivers of the objective.

Choose-one and Choose-all-that-apply. Respondents choose from specified lists. “Choose-one” questions are categorical rather than rated, for example, “In which region are you located? Choose one: North, South, East, West.” Mean scores and other statistics are not defined for them, unlike rating questions. An “Other, please specify” choice ensures that no significant option has been omitted. For “Choose all that apply”, responses can be presented as a percent of all responses, of the number of respondents, or both ways.

Open-ended questions are used to uncover reasons for the ratings, suggestions for improvement, and wishes (e.g., for future offerings). They are typically categorized (“coded”) to quantify the number of mentions of keywords or concepts. They can also be filtered or sorted by the responses to closed-ended questions. Online survey systems allow users to click on open-ended responses and drill down to the complete survey responses that contained those open-ended responses.

Ranking questions force respondents to make trade-offs among or prioritize different choices. A respondent may rate each of three attributes a 10 in importance on a 10-point rating scale, where 10 is “Very Important” and 1 is “Very Unimportant”, but ranking can force the respondent to indicate which of those three attributes is most important. Conversely, if the respondent rates each of three attributes a 10, 9, and 1 on the same scale, their order of importance to the respondent is revealed, but not the large gap in importance between the second and third attributes.

There are many types of ranking scales. One type allows each rank to be used once; another allows one or more choices to be “tied” in ranking. Ranking questions can ask respondents to rank all items on a list; to rank only the top three or some other number; or to select the top three or some other number from the list without regard to ranking within the selected group.

Fixed-sum questions ask respondents to distribute a fixed number of points, dollars, or other units, such as 100 percentage points, among different choices. Allocations may indicate the percent of time spent on different activities; number of dollars to be budgeted to different programs; or relative importance of various attributes.

Back to Top

Scales

There are more rating scales in common usage than can practically be listed here. They include 10-point, 5-point, 4-point, and 3-point scales; scales in which all values are anchored, that is, labeled (e.g., 5 = “Slightly satisfied”); scales in which only end values are anchored; scales without numeric values but text descriptions only; and so on. While it is very important to choose the right scale for your application, it is just as important to stick with a particular scale once it is chosen to be able to track trends and compare performance across business processes.

How a scale will be used dictates in large part the right scale. For most rating questions in satisfaction, loyalty, and commitment surveys, CustomerSat recommends a 10-point scale anchored in three places: 10=”Outstanding”; 5=”Average”; 1=”Unacceptable”.

After decades of research, CustomerSat and its partner and investor J.D. Power and Associates have found this scale to have the following advantages:

• Sensitivity and discrimination. Sensitivity includes the ability to discern small changes over time; discrimination includes the ability to differentiate between groups of respondents with slightly different attitudes and perceptions. The 10-point scale’s sensitivity and discrimination are attributable to its greater granularity than, say, a 5-point or 7-point scale. Among other advantages, a discriminating scale accurately distinguishes between those delighted by their experience and those who are merely pleased. For good discrimination, as a general rule, fewer than 20% of the respondents should fall in the ‘top box’ of a scale. The 10-point scale generally exceeds this requirement. A scale’s qualities of sensitivity and discrimination allow you to effectively compare the performance of different customer service centers, for example, and better track improvements over time.
• Clarity and comprehensibility, especially across language and cultures, for two reasons. First, perhaps because we humans all have ten fingers, regardless of language and culture, respondents around the world readily comprehend the 10-point scale and are least likely to misinterpret it. Second, the scale has only three anchors that require translation across languages, and having to translate anchors introduces complexity and lack of comparability over and above cultural differences. Imagine, for example, the challenge of accurately translating such phrases as “Very satisfied”, “Somewhat satisfied”, “Slightly satisfied”, “Neither satisfied nor dissatisfied”, and so on into a dozen languages. In contrast, translating “Outstanding”, “Unacceptable”, and “Average” is relatively straightforward.
• Suitability for correlation and regression. Because of their granularity, sensitivity and discrimination, 10-point scales lend themselves better to correlation and regression analysis than scales with fewer values.

For the overall satisfaction question, we recommend the 10-point satisfaction scale, where 10 is “Very Satisfied” and 1 is “Very Dissatisfied.” Other scales may also be used, such as Agree/Disagree.

Back to Top

Specifying Respondent-Descriptive Variables

Well-designed feedback systems use variables called descriptors, often provided from a corporate database or CRM system, which identify and categorize the stakeholders being surveyed. Descriptors fall into two groups. Categorical descriptors drive questionnaire logic, allow users to segment and analyze feedback, and provide stakeholder or transaction context for smart alerts and action management. Personal descriptors are used to personalize surveys, invitations, reminders, acknowledgements, alerts, and action management cases. The table below shows examples of both types of descriptors.

Respondent descriptors are used in any of four ways:

• To tailor the questionnaire to the respondent, e.g., "Our records indicate that you purchased our product through reseller XYZ. Please tell us about your experience with XYZ." In this example, “reseller” is both a categorical and personal descriptor: categorical, because it enables users to analyze results by reseller; personal, because it personalizes the questionnaire.
• To personalize the survey invitation and increase response rates, e.g., "Thank you for being our customer since 1993." In this example, again, “year of purchase” could be both a categorical and personal descriptor. The international language in which survey and email invitation appear to the respondent may also be driven by a respondent-descriptive variable.
• To allow filtering, cross-tabulation and other analyses and reporting on survey results. Size of customer, geographical region, vertical market, customer service rep, site, department, and revenues generated are all examples of categorical descriptors.
• To provide context, background, and contact information to recipients within your organization of action alerts and action management. These are both categorical and personal. Action alerts immediately notify the right individuals that customers are dissatisfied for immediate follow-up action. Action management creates cases for selected alerts.

Back to Top

Other Guidelines

Following are some additional pointers to guide your survey-design process:

• Avoid compound questions. Include only one attribute in each rating question. An example of a compound question for rating a customer support representative is: “Proactively identifies problems and takes timely action to resolve them.” Should the rating indicate less than stellar performance and this metric is shown to be important, it would be difficult to put an action plan in place. Is the problem proactive communication on problem status or timeliness of problem resolution?
• Ensure that all performance attributes 1) are mutually exclusive with minimal overlap, and 2) together fully capture what is important to stakeholders about the product, service or experience.
• Be assiduous in ensuring that you have not introduced bias into the questions. Even subtle variations in the way questions are worded can have an effect on the way people respond.
• Ask about symptoms, not root causes. Respondents know what they experience; they may not know causes of unacceptable performance. For example, a respondent can recognize a lack of a support rep’s technical knowledge in the rep’s inability to solve a problem. But the respondent will not generally be able to tell you whether this lacking is due to faulty hiring criteria, inadequate training, or lack of a complete and current knowledge database. Your knowledge of your organization, coupled with your survey partner’s assistance in analyzing survey results, are needed to identify and address root causes.
• Seek clarity and conciseness. To achieve both can be a challenge. The language must be clear, so that all respondents have a common understanding of what the survey intended. If feedback is sought on technical products, for example, be certain that customers understand the terminology used, model designations, and product categories. At the same time, each question should be stated as concisely as possible, so that they are quickly and easily read, engaging the respondent.