Open questions raise NLU failure

What it is

An open-ended prompt hands the reply space to the user. A gallery guide robot asking “what are you interested in?” sounds like a human attendant; the next utterance may be “the blue one,” “is there a bench,” or “what period is this.” The recogniser faces an unbounded vocabulary; the intent classifier faces an unbounded set of acts. Failure rises not because open questions sound rude, but because the system invited answers it cannot cover.

Why it happens

An open question stacks two uncertainties. Acoustically, language-model perplexity rises: people may produce names, deixis, small talk, complaint; the decoder’s path widens and substitutions and insertions grow. Semantically, even a correct word string may miss every labelled intent or slot — “the blue one” needs gaze, a work list, and reference resolution, and the question itself never pulled candidates into a computable set.

There is also a cooperation problem. In conversation an open question means “you set the topic.” People organise a narrative the way they would for a person, rather than handing over a slot. Longer narratives stress endpointing, recognition, and intent at once. The system made a social commitment to open collaboration while only preparing a closed backend. Failure sits in that seam.

Studying it

Elicit the same slot two ways: open (“what would you like to know”) versus bounded (“artist, date, or continue to the next work”). Log dependent measures separately: word error, intent error (words right, act wrong), out-of-coverage replies, and utterance length. A further independent variable is whether visible objects are present — deixis spikes when they are.

Do not code the corpus as “recognised or not.” Split failures into acoustic error, intent error, uncovered slot, and unbound reference. If the open condition mainly lifts the last two, the microphone is not the problem; the question walked people out of the parsable set. A Wizard-of-Oz that lets a human understand the open reply will systematically undercount live intent error.

Where it stops holding

When the backend really is open-domain and retrieval or a human agent can take the utterance, open-question failure is no longer task failure; the understanding load moved a layer down. Expert users who already know which slot to fill treat the open question as “please speak,” and failure rates approach the closed case. With shared visual attention, “the blue one” may resolve; without a screen, a location, or a work list, the same sentence is a dead end. Defaulting to open questions because they “sound more natural” trades social naturalness for parsability.

Applying it

• Use an open question only when the next layer can digest free text (retrieval, handoff to a person, already scoped to the current object). Otherwise switch to an enumerable choice or a single concrete slot.
• Before shipping, collect real replies to the open wording and tabulate coverage gaps by intent and slot. Frequent uncovered replies mean add coverage or drop the open question.
• An open question needs an exit: after a miss, do not ask the same open question again.
• How to check: compare intent error and uncovered-reply rate between open and bounded wording. If uncovered is the main driver, change the question; do not start by retuning the acoustic model.

Related

• Same group: M2.02.2 Closed questions constrain expression but are reliable · M2.02.3 Questions should narrow after failures
• Nearby: C7.14 Command grammar and free expression · M2.04 Error-recovery wording · M1.04 Context retention
• Search terms: open-prompt NLU failure · open-ended prompt · intent error rate