A real object occluding a virtual one needs environment depth
Aliases: environment-depth occlusion · occlusion mapping · scene-mesh occlusion
What it is
Seat a virtual cube behind a real chair and the legs should hide it. Without the chair’s depth, the cube is drawn entirely in front or entirely behind — there is no “hidden by” state. Depth-based occlusion is that requirement: for a real object to occlude a virtual one, the pipeline needs an environment depth — a mesh, a depth map, or a hittable plane — that the depth test can cut the virtual geometry against.
What is missing is not a more fused material. It is real-world geometry that can enter the comparison.
Why it happens
Rasterization keeps a pixel by comparing depths. Virtual geometry brings its own; the real world does not, unless a second depth is built. That depth comes from stereo, time-of-flight, structured light, or a SLAM scene mesh. Resolution, refresh, and coverage all have holes: a table edge becomes a jaggy polyline, chair legs fill in as a slab, a newly moved box is still empty space. Virtual content leaks through holes and sparkles on jagged rims. Depth is also timestamped: the mesh is last second’s room, and a cup has already entered the frame. The test is then run on stale geometry.
Optical see-through devices with no depth API can often only add light, so the real-occludes-virtual direction never enters the pipeline. That is not a drawing miss. There is no second depth to compare.
Studying it
In a lab with known geometry — a table, a chair — place virtual objects where they should be hidden and where they should hide the real surface. Score occlusion accuracy per pixel from an external camera or an eye-position grab, and measure absolute error of the depth map against ground truth (a laser scan or a calibrated model). Independent variables: depth source, mesh resolution, object distance. Dependent variables: correct-occlusion pixel rate, edge flicker counts, the share of trials judged “it is behind.”
Interface work uses this to accept whether, once scene-understanding permission is on, a placement task actually has a “hide behind furniture” tier, rather than only a “stick in front of everything” tier.
Where it stops holding
In the far field against low-frequency backgrounds (a blank wall), coarse depth errors barely show. A pre-scanned, nearly static room can live on a static mesh; people walking and doors swinging turn that mesh into wrong occlusion immediately. Marker-only placement pins content to a QR code and needs no environment depth — and also cannot let unrelated furniture participate. With passthrough off and a fully virtual level, there is no real object in the comparison, and the constraint does not apply.
Applying it
- Treat environment depth or a scene mesh as a prerequisite for any content that must sit behind furniture, not as a post-launch patch.
- Where depth confidence is low, skip precise occlusion: a translucent outline or a temporary near placement beats a hole that leaks.
- In rooms that change (reception, retail), let people trigger a rescan. Do not assume the mesh from boot stays true.
- How to check: park a virtual box behind a real chair back and walk a full circle. If at any angle the box comes through whole between the legs, depth never entered the test.