Success rate does not capture the cost of completion
Aliases: effort-blind success · completion cost · success without effort
What it is
Success rate records only whether an endpoint was reached. Two attempts can both count as success while one ended after a single click and the other crawled through errors, help requests, and a remembered back door. Those are not the same evidence for a decision. The rate is silent about time, errors, workload, emotional cost, and whether the person could do it again.
Why it happens
A binary arrival flattens every trajectory that crosses the end state into one count. An interface can convert failure into success by raising tolerance, stretching waits, or hiding the critical action in expert muscle memory, while leaving the cost with the user. Teams that watch only arrival treat “still completable” as “shippable,” and miss completions that make people switch channels next time or refuse to retry under real risk. Cost is the integral of the trace; success is its endpoint. Substituting the endpoint for the integral is a gap in the metric, not a statistical accident.
Studying it
On the same tasks, collect arrival and cost in parallel: time, errors and recovery, help requests, subjective workload or frustration, and use of workarounds. Describe them separately, then inspect whether high success rides on high cost, especially successful but costly cases. When comparing designs, success that stays similar while cost falls supports “still completable, and cheaper”; a rising success rate alone cannot rule out more painful success. Follow-up contact can test whether costly success turns into avoidance.
Where it stops holding
On very short, low-risk tasks, success or failure may already suffice, and cost differences are too small to carry their own metric. Safety-critical work is the opposite: a costly success can be more dangerous than a failure because it hides a near miss. Experts will pay costs that novices will not; a mixed-sample mean cost conceals the split. Field settings add social and time pressure, so a laboratory willingness to grind to success does not mean the same cost will be paid in use.
Applying it
- Report at least one cost measure beside the success rate—time, errors, help, or workload—and mark any that were not collected.
- List “successful but costly” sessions as a priority fix, not as a win.
- Set a guardrail: if success holds while cost rises, do not ship on arrival alone.
- On retest, check whether formerly costly successes became unassisted and cheap; if people still arrive in pain, treat the issue as open.