The two channels have to line up in time

What it is

TTS is still on the first restaurant; the display has already highlighted the third card. The user says “this one.” “This” binds to the third under the eyes and the first in the ear, and the system cannot tell which to take. A nav map that has already cut to the next turn while speech is finishing the previous one is the same split. Temporal alignment of voice and screen requires that the two channels be co-present in the seconds they refer to the same object, not that each refresh on its own clock.

Why it happens

Cross-modal binding needs coincidence in time: the auditory label and the visual mark of the same object have to fall inside a window that can be treated as “this beat.” The window may have a small fixed lag (highlight slightly before the word, like captions); it must not drift. TTS duration moves with rate, barge-in, and network jitter; UI animation follows its own curve. Once the two timelines uncouple, demonstratives, ordinals, and the focus row point at different entities. People interpret “this” on whichever channel is salient — usually the card they are looking at — and the system, counting by speech progress, binds wrong.

Misalignment has a direction. Screen ahead: people think speech is naming the current highlight when it is still naming an item already scrolled off. Speech ahead: people hunt the screen for an object not yet drawn and think the system is talking about something off-screen. Alignment is not “don’t write the same sentence twice”; it is making the two streams (even when they divide labour) point at the same referent in time.

Studying it

Offset the highlight relative to the target TTS word: synch, 300 / 800 ms early, 300 / 800 ms late. Elicit “this / this one.” Gold standard is gaze or a later point-to “which card did you mean.” Dependent measures: cross-item misbinding, hesitation, and whether people switch to ordinals to escape the offset. Independent variables include speech rate (the same animation curve against faster TTS enlarges lead).

In the wild, timestamp display-focus changes against TTS word boundaries. Focus changes that land mid-word or on the previous word are systematic misalignment. A lab that lets people tap instead of saying “this” will not see deictic misbinding.

Where it stops holding

Listen-only, no screen, there is no second timeline to align. If the user is not looking, visual focus has no object — what must align is the state at the next gaze return, not a highlight marching with speech for someone who is not watching. Decorative motion (a breathing backlight) is not in the referring system and need not track words. A tiny earcon with an instantaneous highlight has a window too short to matter; there is no interruptible deictic phrase. Treating timing as “make the animation fancier” lengthens the visual timeline and widens the split.

Applying it

• Clock the visual side from the word being spoken: highlights, map cuts, and card focus change only inside that word’s window. Do not let list animation run ahead on its own beat.
• When rate or barge-in changes TTS duration, re-layout the visual timeline; do not play a pre-rendered animation against live speech.
• Do not silently resolve “this / this one” inside a misaligned window. Ask “the one lit on screen, or the one just named,” or pull both channels back into synch first.
• How to check: synch versus ±800 ms highlight offset, elicit “this.” If misbinding rises with offset, the clocks are off. Then check production: focus changes landing mid-word mean fix the clock before the recogniser.

Related

• Same group: M3.10.1 Visual information is gone when gaze is off the screen · M3.10.2 On-screen options need names voice can refer to
• Nearby: M3.05 Voice and Screen Complementarity · M1.09 Context Retention and Reference Resolution · C7.16 Visible Feedback for Voice Input
• Search terms: temporal alignment of voice and screen · temporal alignment · cross-modal binding