Switching modalities costs attention and limb reconfiguration

What it is

Switching modalities is not free. Moving from touch to speech requires taking a hand off the screen, shifting attention from visual to auditory, and reorganizing how the user expresses the request. Each adjustment takes time, and in sub-second interaction rhythms that can be a large share of the total.

Why it happens

The cost comes from three reconfigurations: attention transfer, limb repositioning, and change of expression. Attention transfer takes time and accumulates with frequency; limb repositioning is expensive when it means raising a hand, moving the device, or changing grip; expression change forces users from "point and say" to "describe," a different organizational task. All three depend on the preceding state—if the last action left the hand or attention in a particular place, switching costs more. This is why consecutive actions in one modality are usually faster than alternating between two.

Studying it

Measure the switching cost itself: build tasks that require a mid-task modality change, log the time increment and error rate around the switch, and compare against an all-single-modality version. Variables include switch count, modality pairing, whether limb repositioning is required, and experience. Outcomes include the per-switch time increment and error rate, decomposable into attention, limb, and expression components.

Where it stops holding

When a switch coincides with the natural rhythm of the action—glancing then speaking—its cost is low because the user would make the same movement anyway. When the switch avoids a higher cost, such as moving from speech to vision to avoid misrecognition in noise, it is worth paying. Long-term users of two modalities reduce but do not eliminate switching cost.

Applying it

• Reduce the number of modality switches within a task and keep consecutive actions in one modality.
• Avoid requiring limb repositioning across a switch, such as combining touch with mid-air gesture.
• When a switch is unavoidable, indicate clearly which modality is now expected.
• Verification: measure the time increment around switches against a single-modality version; a clear increment argues for reorganizing the task to reduce switching.

Related

• Within the group: D5.03.2 Frequent switching cancels the benefit of multimodality · D5.03.3 Switching should be user-initiated, not forced
• Adjacent: D5.09.1 Users need to choose the active input or output modality · C5.01 Complementarity and cost across modalities
• Search terms: modality switching · attention shift · reconfiguration cost