By CustomerSat Research Consultant William J. Herald, Ph.D.

Where should your organization allocate its limited resources to maximize improvement in customer satisfaction? Is there a tool that will point you in the right direction?

One proven approach is to plot your survey results on action priority charts - quadrant charts are the most common type - then use the results to determine which mix of attributes (e.g., courtesy/respect, product knowledge, documentation) should receive priority for follow-up action.

A typical quadrant chart is divided into four sections (hence, the term "quadrant") and includes two dimensions: a summary measure of each attribute (e.g., mean satisfaction with responsiveness) and a measure of the attribute’s importance or impact as related to the priority variable (e.g., correlation between each attribute and overall satisfaction). An effective quad chart is one which shows where to focus improvement efforts (e.g., attributes with low satisfaction scores and high importance).

However, there are variations to the typical chart which may allow the resulting chart to better meet your particular needs. Consider these four basic factors when deciding the best type of chart for your organization:

1. Type of importance (or impact) variable There are three main choices: stated, derived, both.
2. Importance measure Your choice depends on the type of importance variable employed.
   Stated: scale, rank, constant sum allocation.
   Derived: correlation, estimated model coefficient (e.g., ), other.
3. Attribute summary statistic The main choices are mean, top box, other.
4. Division Six choices can be made: mean, median, organizational standard, benchmark, relative weightings, none.

While these four factors do not cover all possibilities, they are a good starting point to determine the best match between chart characteristics and your company's needs. Let’s look at each factor in some detail.

Type of Importance (or Impact) Variable

Stated importance asks the respondent to directly specify the importance of various attributes, e.g., quality, timeliness, price, etc. This approach offers a distinct advantage. It eliminates the need to use attribute satisfaction scores to calculate derived importance. But it has a few disadvantages, too.

• It increases survey length.
• Respondents sometimes give high importance scores to every attribute.
• Respondents sometimes give answers they feel may be expected or more acceptable. For example, a respondent may claim an automobile's quality of workmanship is most important, when in reality brand prestige may be the real deciding factor for him or her.

Derived importance, on the other hand, doesn’t depend on a respondent’s statements. It calculates each attribute's importance by looking at the relationship between the attribute's satisfaction score and the priority variable’s score (e.g., overall satisfaction). Derived importance offers several advantages:

• It usually results in shorter surveys.
• It may provide better differentiation between attributes.
• It may uncover importance more accurately than stated importance.

There is at least one disadvantage to using derived importance. An attribute with little variability in scores may appear to be relatively unimportant (because the importance measure depends on variability) when in reality that attribute might be very important.

Which approach should you choose? When deciding between stated importance and derived importance, it’s sometimes prudent to obtain both measures, as long as that doesn’t result in surveys that are too long and complicated. Stated importance gives what the respondent says is most important, while derived importance yields what the data and analysis technique say are most important. These two approaches do not necessarily yield the same result, but both perspectives can be valuable.

Charts of various types can be constructed to show this (inter)relationship. For example, stated importance can be plotted on one axis and derived importance on another. The four quadrants could then depict high stated/high derived, high stated/low derived, low stated/high derived, and low stated/low derived.

A three-dimensional chart could also be constructed to depict derived importance, stated importance, and attribute summary. The chart could have derived importance on the x axis (horizontal), stated importance on the y axis (vertical), and the attribute summary measure (e.g., mean attribute scores) on the z axis. This could be useful when you wish to concentrate resources on improving the high derived importance/high stated importance/low satisfaction attributes.

Importance Measure

Your choice of importance measure depends on the importance type (stated or derived) you employ.

Stated Importance: If stated importance is used, you need to provide a mechanism for the respondent to indicate the relative importance of each attribute. There are three common methods:

• Rating The respondent can be asked to rate the importance of each attribute using a scale response set, e.g., a scale of 1 to 10. A respondent could rate the importance of a software product’s documentation, for example, from 1 ("not at all important") to 10 ("extremely important"). One potential problem with this approach: a respondent may give every attribute a similar high score.
• Ranking In this approach, the respondent ranks the importance of a series of attributes by arranging them in the appropriate sequence. Unfortunately, the resulting data will be ordinal, meaning the difference in importance between a 3 and 4 ranking may not be the same as the difference between a 7 and 8.
• Allocation Finally, the respondent can be asked to allocate 100 points among a list of attributes - the more important the attribute, the more points are allocated to it. Internet-based surveys manage this approach in a fairly straightforward manner; while in phone- and paper-based surveys, it can sometimes be time-consuming.

Derived Importance: If you opt for derived importance as your importance type, a number of statistical techniques can be used to estimate an attribute’s importance.

1. Correlation can be computed between each attribute's score and the priority variable’s score, e.g., the correlation between satisfaction with a product's documentation and overall satisfaction. Pearson's correlation, r, offers several advantages, including:
   
   
   • Understanding and usage are fairly common.
   • Application of calculation is straightforward.
   • Missing values are easily tolerated.
   • Collinearity between attributes does not hinder calculation.

There are a few disadvantages to using Pearson’s correlation. It does not account for interrelationships between attributes, it measures linear association, and it may not be appropriate for certain response scales (e.g., scales with a few choices). If your organization is geared toward percent top-box scores instead of means, Kendall’s tau-b value can be used to measure the association between an attribute’s top-box score and a priority variable’s top-box score. (Depending on the situation, the results using Kendall’s tau-b may not differ much from the results obtained using Pearson’s r.)

2. Modeling is another statistical technique to consider. For example, in a multiple linear regression model, the dependent variable would involve the priority variable (e.g., overall satisfaction), while the independent variables would include each attribute’s score. Although useful, this technique has several disadvantages:
   
   
   • It is not appropriate for small sample sizes, extensive missing values, collinearity, and/or nonlinear relationships.
   • The technique should be utilized only by analysts who are trained in its use.

Although more complex than multiple linear regression analysis, other statistical models may be a better fit for your data and organizational goals. For example, if your sample sizes are large, consider using Structural Equation Modeling (SEM). Also, Latent Class Analysis (LCA) may be useful. If your organization is geared toward percent top-box scores instead of means, Logistic Regression might be a better fit. Other techniques have been mentioned in the literature, including R² (square of correlation), regression beta coefficient times correlation, coefficient of partial correlation, and Impact!Analysis.

Attribute Summary Statistic

Usually, the attribute summary statistic is the mean score, although in certain situations, an argument can be made for using percent top box score. Ultimately, the choice depends on your organization’s emphasis and the scale you employ. If your attribute satisfaction score is on a 1-10 scale, either a mean or top-box percent can be used. On a three-point scale, a top-box percent may be more appropriate. If the scale is appropriate and your organization concentrates on individual or mean scores, then the mean would be a more suitable choice.

Some organizations concentrate on top-box percents, top-two-box percents, or top-three-box percents. An organization’s standards may require that top-three-box scores (e.g., 8-10) must be at certain levels. For these organizations, the top-three-box score might be a better attribute summary statistic. The choice is important because this will be one of the dimensions used in your action priority chart.

Another attribute summary statistic could be the difference between an attribute mean (or top-box percent) and a comparison figure, e.g., an organizational standard or industry benchmark. These values would pinpoint the disparity between attribute values and comparison figures. This approach can be particularly helpful if the standards vary by attribute. For example, the standard for courtesy and professionalism may be higher than the standard for documentation.

Division

As mentioned previously, the most common term for "action priority" charts is quadrant (or quad) charts, a name which suggests that the chart is divided into four areas. Actually, charts can contain more or fewer than four segments. How should your chart be divided? How many areas should it contain?

A number of options exist. For discussion purposes, let’s assume that the attribute summary score is plotted along the horizontal axis, and the attribute’s importance is plotted along the vertical axis.

One option is dividing the chart by a vertical line from the mean attribute summary score (e.g., mean of attribute means or top-box scores) and by a horizontal line from the mean importance measure (e.g., mean of attribute correlations). A second option is dividing the chart using medians instead of means. Either method should result in somewhat equal numbers of points on each side of the vertical and horizontal lines. One disadvantage: It’s possible that a majority of scores are low, but the chart shows half to the right of the line, giving the appearance of higher scores.

Another option is using an internal metric or standard as a dividing line. For example, a company’s goal might be satisfaction mean scores at or above 8.0, so a vertical line could be drawn there. Adopting an industry benchmark is another option. Relative weightings of importance and attribute scores could also be used. That could result in a series of positively sloped lines that separate various segments. Another option is to have no lines drawn. Instead, groups of variables (e.g., those in the upper-left portion of the chart) could be identified. Still another choice is identifying more than four areas, e.g., low-medium-high for attribute scores.

Typically, CustomerSat quadrant charts utilize derived importance with a correlation importance measure, a mean attribute summary statistic, and a mean quadrant division. Specialized charts using other combinations, of course, can also be produced for our clients as needed.

Summary

Determining where to allocate your company’s resources to improve customer satisfaction is a very important decision. Action priority (quad) charts can be valuable tools in your decision-making. Hopefully, the options outlined above will help you determine the best match between the analysis instrument and your organization’s goals and needs. Ideally, there should be a dialogue between the research analyst and your organizational representative(s), preferably early in the research process. This discussion will help uncover the options that are best suited to your organization.

Selected References

CustomerSat, “Top Box, Bottom Box, Mean Score - What’s Important?” CustomerSat.com Connections (October 2000).

CustomerSat, "Prioritizing Action Using Quadrant Charts," A CustomerSat White Paper (October 2005).

David, Monica, "Derived Importance vs. Stated Importance," CustomerSat Insights Newsletter (Q3 2006).

DiPaula, Adam and Barb Justason, "Integrating Explicit and Implicit Approaches," Quirk’s Marketing Research Review (October 2003).

Fitzgerald, Albert and Chad Johnson, "Data Use: Uncovering Customer Loyalty Drivers Using Structural Equation Modeling," Quirk’s Marketing Research Review(October 2002).

Hanson, Randy, “Determining Attribute Importance," Quirk’s Marketing Research Review (October 1992).

Hokanson, Steven, "Data Use: Another Way to Assess Satisfaction Drivers," Quirk’s Marketing Research Review (March 2005).

Pinnell, Jon, "All Customers are Not Created Equal," Quirk’s Marketing Research Review (December 2001).

Quirk’s Staff, "Measuring Customer Satisfaction: Drive Your Action with Derived Importance Analysis," Quirk’s Marketing Research Review (October 1998).

Weisberg, Herbert F., Jon A. Krosnick, and Bruce D. Bowen, An Introduction to Survey Research and Data Analysis, Second Edition (Glenview, Illinois: Scott, Foresman and Company, 1989).