Interaction hypotheses can be tested before engineering

What it is

The research value of Wizard of Oz is that interaction hypotheses can be confirmed or killed before engineering capability is ready. The hypotheses usually look like this: people will complete a booking in one utterance; they will continue after an uncertain recommendation; a generated draft will reduce rather than increase editing load. They are about how people live with a capability, not about model accuracy as such. Testing by enactment before writing recognizers, training data, or backends means it is the interaction idea that fails, not already-sunk development cost.

Why it happens

Once an interaction hypothesis enters engineering, implementation detail protects it: teams explain failure as “the model is not trained yet,” not “this dialogue structure should not exist.” Enactment up front treats capability as a switchable service, so turn design, confirmation, failure visibility, and control allocation can be manipulated on their own. If the hypothesis still fails under enactment—people will not say that sentence, do not trust that recommendation, cannot edit that draft—building the capability will not automatically repair the interaction. Success under enactment only shows the interaction holds given ideal capability; it still needs retesting on a real model. What it rules out is “the interaction cannot work,” not “the model will suffice.”

Studying it

Write hypotheses as observable human-side behavior: whether a given utterance form appears, after which prompt, and whether people switch strategy after failure. The Wizard of Oz condition supplies an upper bound on capability; a contrast then degrades it (forced recognition errors, refused request types) to see whether the hypothesis still holds. Kelley’s early speech work tested dialogue structure first, then decided where to put engineering effort. The unit of analysis is the interaction fragment, not a model metric. A development milestone should read “if the hypothesis is killed, stop building this interaction,” or enactment never enters decisions.

Where it stops holding

Some hypotheses are model hypotheses (“can word error rate fall below a threshold in our acoustics”); Wizard of Oz cannot answer them—corpus and offline evaluation can. If the organization has already promised the capability, an early negative will be ignored and the method loses its decision function. Work practices that take weeks to stabilize cannot be tested as first encounters in a single enacted session. Ethically, hypotheses involving deception or high-stakes advice need review for real harm potential even when enacted.

Applying it

• In the proposal, split interaction hypotheses from model hypotheses; send the former to Wizard of Oz and the latter to offline evaluation.
• For each interaction hypothesis, freeze “what observation counts as a kill” before enactment, so the criterion cannot be rewritten afterward.
• Success under enactment does not schedule a development wrap-up; it schedules a retest of the same hypothesis with real capability.
• After a kill, the output is to change the interaction or drop the capability, not to “build it and see.”

Related

• Same group: Q5.04.1 A human performs capabilities the system has not yet implemented · Q5.04.3 Human speed and consistency outrun real systems and must be corrected
• Adjacent: Q1.02 Exploratory and confirmatory research · Q5.03 Interactive prototypes
• Search terms: pre-development Wizard of Oz · interaction hypothesis · WOZ evaluation