A reassuring score can hide a difficult journey

Your survey shows 4.4 out of 5, yet the same requests keep arriving by phone. Before declaring everything fine or dismissing the questionnaire, examine who responds, when they respond and what they were asked. Satisfaction after a friendly conversation does not necessarily measure how easy it was to receive the right delivery.

First choose the business problem: reducing follow-ups, understanding dropouts or making onboarding easier. That question determines whom to approach and which operational information to connect with their answers.

Describe respondents before interpreting their score

Report how many people were invited, how many responded, the survey period and the groups represented. AAPOR explains that response rate alone does not establish whether nonresponse bias exists or how large it is. A large response count does not automatically compensate for a missing group.

Compare respondents with the relevant customer population: new or established customers, small or large orders, standard journeys or incidents. Identify missing groups explicitly. Do not treat a few answers as a portrait of the entire customer base or publish highly precise percentages based on a handful of people.

Fictional example: the average changes with the mix

Imagine 500 customers invited and 50 responses. Forty established customers average 4.8 out of 5; ten new customers average 2.8. The overall average is 4.4: (40 × 4.8 + 10 × 2.8) ÷ 50. It hides a difficult start for the new respondents. This example is entirely fictional.

If more new customers answer the next survey, the overall average may fall without either group’s experience getting worse. Compare results within consistently defined journeys. In this example, five focused interviews and a review of onboarding requests could help develop a hypothesis, without constituting statistical proof.

Turn comments into a testable decision

Create a few observable categories: promised timing, offer clarity, service access, follow-up and resolution. Allow multiple themes where appropriate. Record how many comments were analysed and keep an ‘other’ category; automated classification must be correctable when it misreads negation or irony.

Prioritise problems through three questions: how many customers appear affected, what consequence do they face, and can an improvement be checked quickly? An uncommon but blocking problem may take priority over a frequent minor irritation. In a working file restricted to authorised people, connect themes with relevant tickets or orders, then present aggregated results to the team.

Your task for next week

AAPOR recommends simple, non-leading questions. Prefer ‘What made getting started difficult?’ to wording that already suggests the service was excellent.

  • Choose one customer journey and read twenty comments, removing unnecessary personal information.
  • Compare respondent profiles with customers on that journey to identify less-heard groups.
  • Ask two people to classify ten comments and discuss disagreements about categories.
  • Test one change, such as a clearer welcome message, during the next period.
  • Track repeated requests and related feedback using the same questionnaire and observation window.

The limits of your conclusion

Silent customers are not automatically happy or unhappy. Comments explain experiences but do not by themselves measure how common those experiences are. Do not attribute improvement to your action if the audience or season changed. Keep these limitations in the report: they make the decision more robust, not less useful.