Screen-interface knowledge does not transfer unchanged
Aliases: screen-to-robot transfer · cross-medium transfer · embodied transfer gap
What it is
The screen-to-robot transfer gap describes why feedback, undo, and testing practices that work in graphical interfaces may fail once an interface controls a body coupled to the environment. Screen objects rarely injure someone by crossing the display boundary, and their state is readily copied or restored. Robot actions have trajectories, momentum, occlusion, and sometimes irreversible physical consequences.
Why it happens
A graphical interface compresses input, state, and output onto a view that a user can pause to inspect. A robot unfolds the loop across time and space: a person must notice a cue, estimate a path, and coordinate their own body, while a command passes through communication, planning, control, and braking. On screen, cancel is a state transition. For a robot, cancellation is constrained by remaining distance, payload, and whether contact has begun. Identical labels therefore do not imply equivalent control authority.
Studying it
Transfer can be studied by assigning the same task through a screen simulation and a physical robot, then comparing recovery time, intervention timing, path judgement, workload, and separation. The simulation must match viewpoint, latency, and control constraints. A bird's-eye view or instantaneous simulated stop creates an information advantage that cannot be attributed to embodiment. Physical trials should capture near misses and bodily avoidance as well as task completion.
Where it stops holding
Screen knowledge is not uniformly invalid. Consistent terminology, hierarchy, state coding, and control mappings can still reduce learning cost. The gap matters most when actions alter the environment, feedback spans several senses, communication is delayed, or dynamics limit reversal. A digital twin approximates the control problem only when it remains calibrated to real latency, sensor blind spots, and braking behaviour.
Applying it
- Inventory the screen prototype's assumptions of instant response, unlimited undo, and complete visibility, then test each assumption on hardware.
- For every motion-triggering control, identify when safe reversal ceases to be possible. Distinguish command received, braking in progress, and physically stopped.
- Move from hardware-in-the-loop tests to controlled trials of worst-case latency, maximum payload, and occlusion. Compare predicted and measured trajectories instead of checking task completion alone.