When recognition fails, the system should follow a predefined fallback rather than going silent

What it is

When the current mid-air recognizer is already unreliable in this session—a run of low confidence, repeated tracking loss, a sudden environmental drop—the system should enter a predefined fallback, not fall from “still works” into silence. Fallback is a runtime emergency script: still let the user finish the step in front of them. It is not the design inventory that “every function must always have a non-gesture path”; that is a different claim. This one asks what the UI does in the seconds while recognition is collapsing.

Why it happens

Failure is rarely a clean switch. Scores dirty first, tracking fragments first, then comes full silence. If the product only implemented the happy path, that middle stretch has no state to go to, and it looks like random breakage: sometimes works, sometimes doesn’t, no next step. A predefined fallback promotes “recognition is unreliable” to a first-class state: shrink the vocabulary being listened for, demand a coarser confirm, or hand focus to another channel that is already present. The script has to be written before shipping, because there is no product manager to ask “what now” on the failure site. Going silent hands the decision to the user’s guess, which is usually that the device is broken.

Studying it

Halfway through a task, inject repeatable failure: degrade keypoint quality, add occlusion, force confidence into the mid-band. Compare “no script (natural breakage)” with “enter the predefined fallback.” Dependent measures are task completion, time from failure to the next successful action, and whether users report a crash. Coders should mark the first response (repeat the gesture, tap the screen, hunt for a button, quit). Do not fall back before the task starts; that measures everyday availability of a spare channel.

Where it stops holding

Short demo tasks never last long enough to fail, so the lab underestimates the cost of having no script. A fully offline device with no second channel may only be able to pause and explain, not finish the task—and pause-plus-explain is still better than silent death. A safety interlock should enter a safe state when recognition fails; that is a special predefined fallback, not “keep driving the hoist on a looser recognizer.” Occasional single low-confidence frames belong to threshold policy and should not enter degraded mode every time, or the mode will chatter.

Applying it

• Write a failure script: which run of signals counts as “recognition unreliable,” what happens on entry, how to leave. Treat it as a state on a par with the happy path, not a line in an error log.
• Forbid “no response” as the only thing outside the happy path. At least pause and name a reason, even if a second input does not exist yet.
• Accept by injecting failure mid-task: the system should enter the script within seconds, not let the user wave at empty air until they quit.

Related

• Same group: C4.25.2 Fallback can mean shrinking to a simpler gesture set or switching to another modality · C4.25.3 Fallback should not happen silently; users need to know they are in a degraded state · C4.25.4 Without a predefined fallback, edge cases present as random system failure
• Adjacent: C4.24 Recognition confidence and bias direction · C4.28 Necessity of non-gesture paths
• Search: fallback · degraded mode · recognition failure