Placement depends on recognizing environment geometry
Aliases: scene understanding · hit-test · environment geometry · surface detection
What it is
“Put the lamp on that table” sounds like a user intention. Execution depends on whether the system has recognized the table. Plane detection and environment meshes turn real surfaces into hittable geometry. Without that geometry, placement can only live in the headset’s own frame: the lamp hangs, follows the wearer, or jumps at the next tracking correction.
Placement here is not choosing a coordinate. It is choosing a surface that has already been recognized. Intention, however clear, does not stand in for that recognition.
Why it happens
Visual SLAM first pulls stable features from the image, fits planes or a mesh, then hit-tests the user’s ray or hand landing. What is hit is the detection result, not the physical table — if detection lags, the landing is in mid-air; if two adjacent tables collapse into one plane, the lamp rides the seam. Semantic labels (floor, wall, desktop) only constrain geometry that already exists; they cannot invent a surface from nothing.
So a successful placement is a chain: enough features → geometry converges → the hit-test lands where the user thought. Any link that stalls leaves the user seeing “I already put it down” and the system holding an unattached pose.
Studying it
In a room with a known furniture layout, have people seat the same virtual object on a named surface. Record time from first observation to the object actually sticking, and the normal error against the real surface once stuck. Contrast a pre-reconstructed mesh with live detection. Plane extents and update events from ARCore or ARKit are process data; the dependent measure should be the user’s “did it sit,” not how many planes the API reported.
Independent variables: surface type (horizontal / vertical / irregular), whether detection has converged, presence of semantic labels. Dependent variables: time to first stick, normal error, counts of “I thought I placed it” while it still hovered.
Where it stops holding
QR codes, magnetic docks, and other pre-laid attach points do not go through environment recognition, so the dependency does not apply. Content that is openly mid-air — a body-following panel, a surrounding menu — does not need a plane. A pre-scanned gallery can have geometry before anyone walks in, so the wait moves to install time; the dependency remains, it just does not occupy the first seconds of this session. Outdoor ground detection at large scale does not share an error model with an indoor tabletop.
Applying it
- Do not draw placement as complete before geometry has converged. A preview may follow the hand; commit waits on a hit-test.
- Show recognized surfaces with a light outline so people aim at what the system sees, not at the table they assume.
- Require planes only on objects that must sit. Surrounding tools should not wait on floor detection to open.
- How to check: name “put it on that coffee table” and record time from the point to the object ceasing to float. If the hand has already let go and the object is still drifting on the tabletop, placement did not wait for recognition.