Without contact, input must be carved out of continuous motion

What it is

A mid-air gesture is a hand or body movement treated as a command without touching an input surface. Touch has down and up; a key has make and break. Mid-air motion has no such physical edge. Sensors observe a continuous stream, and the system must cut a command out of it. That cut is gesture segmentation (also called gesture spotting). When the cut is wrong, people think they issued a command and nothing happens, or a pass-by, a hair-adjust, or a point-while-talking is harvested as input.

Why it happens

Cameras, depth sensors, or infrared arrays report joint or fingertip positions at a fixed frame rate: a trajectory, not a discrete click. The recognizer must decide which stretch is the gesture and which is preparation, recovery, or unrelated motion. Typical cues are a speed valley, a pose crossing a threshold, the hand entering a volume, or an explicit delimiter. Segmentation and classification are separate stages. A classifier can correctly label a wave while gluing two waves together or splitting one wave in half. Without a surface, people also lose the proprioceptive sense of having pressed; they guess from visuals or sound whether that stretch of motion became an event.

Studying it

Vision work on gesture spotting often runs sliding windows or connectionist temporal classification over unsegmented video, locating boundaries and classes together. HCI evaluations that train and test on pre-sliced samples exclude segmentation error and overestimate in-the-wild performance. Useful independent variables are the segmentation policy (speed valley, pose gate, explicit delimiter), inter-gesture gap, and movement speed; dependent measures are hit, merge, split, and whether people report the system “not keeping up.” Acceptance material should be unsegmented continuous demonstrations, not isolated clips.

Where it stops holding

A cuff covering the wrist, backlight, or a brief exit from the field of view can break a trajectory so the segmenter reports two events or none. Children, older adults, and fatigued users move more slowly and smaller; a laboratory speed-valley threshold may never fire. Importing a touch down/up mental model sends people looking for an edge that is not there. Close-range hand tracking and far-range full-body skeletons have different noise sources and cannot share one speed parameter.

Applying it

• Specify an observable start and end for every gesture; do not ship only an end-pose picture.
• Count segmentation errors separately from classification errors so a single “recognition rate” cannot hide merges and splits.
• Accept against unsegmented continuous recordings: tally commands people thought they issued that were not cut, and cuts the system made that people did not intend.

Related

• Same group: C4.01.2 Fit: contact is undesirable, distant control, or direct spatial manipulation · C4.01.3 Cost vs non-gesture paths must include discovery, entry, correction, and exit · C4.01.4 Gestures are a poor fit for precise numbers and long text
• Adjacent: C4.03 Gesture onset and offset detection · C3.01 Tap
• Search: mid-air gesture · gesture segmentation · gesture spotting