Visual information is gone when gaze is off the screen

What it is

Three restaurants sit on the car display; TTS says “as shown.” The driver’s gaze is on the road, so those three lines were not drawn for this turn. The smart display across the hob lights a step while someone is looking down to chop; the highlight is equally absent. Glance-away nullifies visual means: hardware still being there does not mean the visual channel is available on this beat. If a multimodal design parks lists, exceptions, and buttons on “the glass people ought to be looking at,” those items drop to zero the moment gaze leaves.

Why it happens

Visual information enters a decision only if it lands in fixation or at least usable periphery. Driving, cooking, carrying lock fixation on the world; the display is lit and encoding never happens. Systems often treat “has a screen” as “the user owns look-back external memory,” so speech reports the conclusion and the screen holds the detail — true when gaze is co-present, false the moment it leaves. The layer delegated to the screen vanishes from the dialogue; people assume there was no detail, or assume speech already said everything.

Nullification is temporal, not a product type. The same panel is suddenly usable in the three seconds of a red light and gone again on the green. Designing “screened device” as one fixed multimodal policy treats the visual layer as still there at occupancy peaks, and leaks constraints that the ear should have carried (allergen, irreversible confirm, next turn). That is not the same claim as “screenless products must be designed as screenless”: the device here has a screen; the failure is this beat’s gaze, not the hardware inventory.

Studying it

Dual-task plus eye tracking or first-person video: primary task locks gaze (driving sim, chopping), secondary task uses “spoken conclusion + on-screen list” to choose. Dependent measures: whether gaze hit the display inside the option window, difference in spoken follow-ups between people who looked and people who did not, and whether “as shown” is followed by “which ones.” Independent variables: list on-screen only / critical items also spoken, hardness of occupancy.

A field diary of where gaze was at each session start. If most “looked at the screen” happens as a catch-up after the session, the visual channel for that turn was retrofitted and complementarity did not succeed. A lab that lets people park the car to look has broken occupancy.

Where it stops holding

Head-up glass with large type is sometimes usable in periphery, so total nullification fails — still measure whether periphery can actually read that line. When people are asked to glance and the scene allows it (standing at a kiosk, on a sofa), the visual channel is open. In a listen-only scene, “did they look” is undefined. Treating nullification as “therefore speech should read the screen verbatim” duplicates the two channels, rather than moving the layer this decision needs onto the ear while gaze is away.

Applying it

• List the stretches where gaze leaves the display (in motion, knife down, child in arms). In those stretches, decision-critical constraints and candidates must not live only on screen.
• Ban “as shown / please look” as the sole information outlet during occupancy. The screen may still update; speech must finish the beat without looking, or explicitly wait until looking is possible.
• Split “has a screen” from “is looking on this beat.” Only the latter turns on “detail on screen, speech for the conclusion.”
• How to check: car or kitchen walkthrough, first-person video. If gaze never hits the display while a critical item appears only there, score visual information for that beat as zero.

Related

• Same group: M3.10.2 On-screen options need names voice can refer to · M3.10.3 The two channels have to line up in time
• Nearby: M3.05 Voice and Screen Complementarity · C7.06 Hands-Free and Eyes-Free Use · M1.01 When Voice-First Is Appropriate
• Search terms: glance-away nullifies visual · eyes-off display · gaze-contingent multimodality