Environment-depth accuracy sets how bad real–virtual clipping becomes

What it is

Whether a virtual caption stops in front of a real table or drills into the tabletop depends on how accurately the device estimates that table’s depth. Environment depth accuracy is the ceiling on real–virtual clipping: depth biased near, and virtual objects are cut too early by the table; depth biased far, and they sink into the slab before they are cut. Interface logic can be clean and clipping still happens if the depth is wrong. This is not “whether a depth-order rule exists”; it is what geometry that rule is executed on.

Why it happens

Video see-through and mixed reality give each pixel a z from a depth sensor, stereo matching, or a scene mesh. Virtual depth tests treat that z as the real world. Error comes from quantisation, flying pixels, dropouts on glass and black, temporal lag, and simplified meshes that drift off the true surface. Lag hurts in particular: a hand waves, depth is still last frame, and the hand’s volume and a virtual button cut each other. Accuracy worsens with distance; near tables and hands need centimetre-level precision, yet near field moves faster and makes error more visible. On optical see-through without an occluder, even perfect environment depth cannot stop real photons; accuracy only governs cuts among virtual objects and between virtual objects and the estimated real geometry — it will not cut the window itself.

Studying it

In a lab with known geometry (calibrated planes, mocap hands), compare device depth to ground truth, then measure clipping of the same virtual objects.

Independent variables: depth source (ToF / stereo / pre-built mesh), motion speed, surface material (Lambertian / glass / black flock), distance. Dependent variables: depth RMSE or absolute relative error, fraction of clipped pixels at the real–virtual boundary, perceived delay of the cut, a rating of “on the surface” versus “in the surface”.

A static room understates the problem; acceptance must include waving and walking. Report near and far separately — a global RMSE will average a near-field disaster away.

Where it stops holding

Closed VR with no real geometry to respect has no environment depth; clipping is virtual-to-virtual only. A pre-scanned static room can be very accurate until someone moves a chair, at which point accuracy becomes “stale” rather than “noisy”. Direct sun and transparent surfaces are classic ToF and stereo failures; clipping there cannot be repaired by “tweaking the UI a bit” — avoid those regions or fall back to manual placement. When several people share a room, each depth map must also sync; a precision gap makes the same table cut for A and embed for B.

Applying it

• Restrict content that rides real surfaces to planes whose depth is already stable. Do not auto-stick onto noisy regions (glass, windows, black appliances).
• Accept near-field interaction against the worst hand depth, not against empty-room wall RMSE.
• On short depth loss, freeze last occlusion or fall back to a snap that does not depend on environment depth. Do not recut every frame on flying pixels.
• How to check: on the target device, stick a virtual card on a table, then sweep a hand in front of it. A card embedded in the slab, or a hand and card cutting each other, is environment depth falling short — inspect depth before rewriting the UI.

Related

• Same group: N3.10.1 A hand clipping through a virtual object is the most glaring class of interpenetration · N3.10.2 Disabling the depth test to force a panel on top wrecks the whole scene · N3.10.4 Snapping a panel to a surface removes several clipping classes at once
• Nearby: N3.03 Depth Conflict · N5.02 Virtual–Real Occlusion · N5.03 Plane Detection and Environment Understanding
• Search terms: environment depth · scene mesh · depth RMSE