A single score cannot say which step in the process failed

What it is

“How easy was it overall” and “rate this experience” crush a whole process into one number. A low score says someone is unhappy or found it hard. It does not say whether the hard part was finding, filling, confirming, waiting, or recovering from an error. A single rating is therefore not a diagnostic: it cannot point to which step to change. Localizing a step takes step-level observation, a decomposed success code, or items tied to named stages—not slicing one global score more finely and pretending that is localization.

Why it happens

A global evaluation integrates a whole episode, and people integrate by different rules: some are dominated by a last-step failure, some by total time, some by one sentence of copy. The same low score can come from entirely different steps, and a low average mixes those sources. The number also has no spatial index: no step field, no timestamp, no object. Using it as if it were a funnel gap is inferring location from data that contain none. An open box can help, but completion rates and coding cost already leave the economy of “one item,” and must not be assumed to restore localization automatically.

Studying it

If the question is which step failed, the primary data are step completion, error type, dwell, and backtracking; a questionnaire is at most a supplement. If a global score is still collected, declare in advance that it monitors trend and does not attribute. To demonstrate non-diagnosticity, cross the same people’s global scores with step-level failures: do low scores scatter across different steps. If they scatter, the global score has no locating authority. A diagnostic questionnaire should bind items to named steps or objects and score them separately, rather than clustering a total after the fact and imagining locations.

Where it stops holding

When the process has one step, or every failure is already known to sit on one object, a global score and a step failure are almost the same thing and the locating demand disappears. Monitoring can use a sudden drop in a global score as a signal that should start a localization study; the error is using it as localization. Changing the item to “pick the most troublesome step” is no longer a rating but a locating choice, whose validity and option coverage need their own check and should not be nicknamed “the same experience item.”

Applying it

• Keep the global single item on the dashboard’s trend layer. Any discussion of “which page to change” must bring step data or task observation.
• Send back review materials that contain only a mean score and the sentence “the experience got worse.”
• When a questionnaire must diagnose, write items by step or object, report them separately, and do not roll them into one “experience score” that is then used to narrate steps.
• Check: take the latest low-score sample and see whether a shared failing step can be named. If not, that score does not set the scope of a redesign.

Related

• Same group: Q3.16.1 Single-item error cannot be checked by internal consistency · Q3.16.2 Short scales trade dimensional resolution for lower burden · Q3.16.3 Short-scale psychometrics must be revalidated in the target population
• Adjacent: Q3.06 Task success rate · Q3.12 Funnel and retention analysis
• Search terms: non-diagnostic rating · process localization · global satisfaction