Support records are a low-cost source of error evidence
Aliases: ticket mining · customer-service logs · unobtrusive error data
What it is
Support tickets, chat transcripts, and refund notes are error archives whose collection cost has already been paid: people describe, under real consequences, where they could not get through. They are cheaper and wider than a lab, and closer to actual harm. They are not a random-sample error rate. They are a list filtered by willingness to complain, channel access, and agent coding rules. Read as a census they over-weight problems that attract complaints and under-weight problems that make people leave quietly.
Why it happens
A ticket passes three filters. The user must judge the problem worth the time, find an entry point, and leave a trace. Agents then compress the story into operational tags built for response times and department routing, not for interface failure types. Repeat contacts, several issues in one ticket, and bot triage change the counting unit. Tag frequency therefore measures how the organization takes intake, with real failures mixed in. The low cost comes from data that already exist, not from an automatic equivalence to error measurement. Failures with money, account, or legal consequences enter the archive more readily; one-off confusion disappears.
Studying it
Map how tickets are created, what the tag dictionary means, and whether the unit is a person, a ticket, or a thread. Start thematic analysis from the raw narrative, not from agent tags, then map frequent themes onto interface states. Align the same window with product logs and look for logged failures with no ticket, and tickets with no matching event. Estimate who never reaches support: new users, low-privilege users, off-hours users. Tickets are for discovery and ranking; incidence still needs task tests or log rates rather than treating tag shares as an error rate.
Where it stops holding
Products with no support channel, or with outsourced channels that withhold original text, cannot use this source. Highly sensitive failures may move to regulators or social media, making tickets fall. If agents are scored on fast close, tags systematically widen and lighten. After a bot layer, human agents see a filtered tail. Staff-filed tickets should not be pooled with customer tickets.
Applying it
- Each month recode a sample of unprocessed originals by interface step, rather than ranking department tags alone.
- Mark features with failure logs but no tickets as silent failures and send them to observation or interviews; do not close them because support is quiet.
- Do not use ticket volume as a usability KPI by itself: entry-point changes, bot intercepts, and opening hours move volume while errors stay put.
- After a fix, watch both the tag and the narrative theme drop, and pair them with a log rate; tags falling while stories remain often means agents recoded.