After fallback, show which interpretation is active

What it is

After falling back to one modality, the system must show which source is active, how far the command progressed, and what is missing. The notice is part of recovery, not decoration: it realigns “what I said” with “what the system did” and offers completion, undo, or a different route.

Why it happens

Fallback changes the expected interpretation hierarchy. Speech may have supplied the action and gesture the object, but now only one is used; without saying so, users read the result as full fusion. A visible interpretation does three things: marks the evidence used, identifies missing parameters, and bounds what has executed. If the system is merely waiting, state also prevents repeated input or the impression of a hang. Granularity should match consequence: lightweight state for low risk, explicit candidates and confirmation for high risk or irreversible action.

Studying it

Compare conditions after failure: no notice, error only, modality shown, and modality plus missing part plus next actions. Variables include failure type, available modalities, task pressure, and notice channel; outcomes include interpretation accuracy, completion time, repeated input, wrong undo, and unnecessary interruption. Eye tracking or logs show whether the notice was seen, but the key evidence is whether users can restate what the system used and why execution has not finished.

Where it stops holding

Not every lightweight failure needs a full explanation. Autocomplete candidates, temporary filters, or low-risk previews can use in-place marks with automatic expiry; strong notices on every frequent failure become noise. Multimodal contexts also require a perceivable channel: describe the fallback by audio or haptics when vision is constrained, and keep visible state as a backup in noise. A notice must not merely shift blame; “unclear recognition” must lead to selectable next steps.

Applying it

• Use a stable template: active modality, determined content, missing or conflicting part, and next action.
• Give executed parts undo or confirmation, and retain candidate objects plus completion methods for unexecuted parts.
• Choose channel and persistence by consequence, and let the notice switch directly to an available modality.
• Verification: after failure, ask users what interpretation was active and what comes next; measure comprehension, repeats, and unnecessary undo.

Related

• Within the group: D5.08.1 When fusion fails, fall back to the literal reading of one modality · D5.08.2 Silent guessing causes harder-to-notice errors than explicit fallback
• Adjacent: D4.07.4 Adjudication should let users perceive that a conflict occurred · D5.09.2 Hidden automatic switching makes users misjudge the active modality
• Search terms: fallback transparency · active interpretation · recovery prompt