Human speed and consistency outrun real systems and must be corrected

What it is

Operators understand vague requests, produce a fitting reply in a few hundred milliseconds, and almost never repeat the same foolish error—better than most real systems. That Wizard of Oz human advantage makes the interaction look smoother, smarter, and more stable than it will be after launch. Without correcting delay, error rate, and consistency across turns, enactment tests a sharp colleague, not the target model. Correction is not to make participants suffer; it is to make failure modes appear where they will later appear.

Why it happens

People far outperform specialized models on context, reference, and common sense, so they complete ellipses, forgive slips, and disambiguate unasked. Social pressure also speeds them: operators hate dead air and will speak sooner than a server. On consistency, a person remembers a detail from ten minutes ago and holds character, while a real system often drops state mid-session. Behavior that would need confirmation, repetition, and repair is eaten by the advantage, so the design never reserves a place for it. When the model later shows its true error distribution, the turn structure collapses. The advantage is a systematic bias, not one operator having a good day.

Studying it

Treat operator performance as a variable that must be measured and manipulated. Log delay from recognition to output, error types, and consistency on repeated requests. Experiments often use throttles: a mandatory minimum delay, injected recognition errors from a table, a ban on using memory not written into state. Compare an unconstrained operator with one held to production budgets, and watch repair turns, abandonment, and trust. Train operators on rule compliance, not on task success, or they will get smarter in order to help participants succeed.

Where it stops holding

When the only question is “is this interaction even intelligible,” you can leave the advantage in place as an upper bound on capability. Once the question becomes “is it still usable at the expected error rate,” uncorrected enactment yields false positives. Multiple operators on shifts introduce inconsistency closer to an unstable system, but also personality differences. Generative errors are an open set; a fixed error table is only an approximation. For already-slow human service flows, speed advantage may be small and the bias comes from over-cooperation rather than latency.

Applying it

• Take delay and error budgets from the target system and write them as floors and ceilings the operator may not break.
• Provide failure cards (did not catch that, out of scope, state lost) and require the operator to play the card rather than improvise a clever save.
• In review, mark turns that were faster or more accurate than the rules as invalid evidence of capability.
• What you hand to engineering is not “it was smooth in enactment” but the interaction structures that still held under the constraints.

Related

• Same group: Q5.04.1 A human performs capabilities the system has not yet implemented · Q5.04.2 Interaction hypotheses can be tested before engineering
• Adjacent: Q5.03 Interactive prototypes · Q5.07 How prototypes mislead
• Search terms: Wizard of Oz human advantage · wizard speed bias · error injection