Video see-through re-digitizes the real world, so a virtual object can occlude a real one

What it is

The same virtual panel, on a video see-through path, can paint the window out. The world is sampled by cameras into an image, then composited with the virtual layer in a framebuffer. Pixels can be overwritten, so a virtual object occluding a real one is the compositor’s default, not an optical trick.

This is virtual-on-real. Real-on-virtual still needs environment depth; digitizing does not give it away.

Why it happens

The camera output is an addressable texture. Depth test, alpha blend, deferred shading — any of them can decide whether a pixel ends as virtual colour or passthrough colour. If the virtual surface is nearer, the real texture in that direction is simply not written. Windows, walls, people walking past can vanish from the picture. The “hole” additive optics cannot cut is, here, an ordinary framebuffer overwrite.

The cost travels with the power. Once hidden, the real information exists only in the compositor’s choice. A wrong rule, a hole in the depth map, or a contour that lags the head turns the occlusion into a misplaced patch. Digitizing makes occlusion possible; it also makes occlusion errors fake at pixel scale.

Studying it

Compositing against ground truth: record the passthrough with no virtual layer, then overlay, and compare the occlusion mask pixel-wise with hand labels or a laser scan. Place a hand and a cup in the near field and watch whether the virtual-on-real contour hugs the object.

Manipulations: compositing order, reconstructed depth on or off, virtual-object distance. Measures: mask IoU, contour offset in pixels, counts of reported “wrong block,” reaching error when grasping through the overlay.

Asking only “does it look good” conflates occlusion correctness with image quality. Score the mask on its own, or you will not know whether this capability is actually working.

Where it stops holding

Without usable depth, virtual-on-real still works (the virtual layer can cover globally) but real-on-virtual does not, and hands sink into the virtual object. Phone passthrough can overwrite pixels too; near-field parallax then hides a patch that is not the patch the eye would have used. Overexposed or motion-blurred real pixels, once covered, strip the highlights and texture people were using for distance. Maintenance and surgical guidance that must see through the virtual object should restrain this capability, or the occluded thing is the task object itself.

Applying it

• When the task needs a solid board, a privacy wall, or a virtual part covering a real remnant, take the video-see-through composite. Do not wait for additive optics to catch up.
• Write the default rule as nearer-covers, and test virtual-on-real and real-on-virtual as two paths. One screenshot cannot accept both.
• Whitelist real objects that must remain (hands, tools, skin) and forbid the virtual layer from writing those pixels.
• How to check: park a virtual panel in front of a window and record passthrough. Highlights of the window should vanish inside the panel contour; a hand in front of the panel should cut the virtual layer the other way. Either side failing means compositing order or the depth source is still unhooked.

Related

• Same group：N5.08.1 In optical see-through, virtual content can only add light and cannot truly occlude real objects · N5.08.3 Camera position is offset from the eye and needs extra correction · N5.08.4 Video see-through still undershoots the eye in dynamic range and resolution; highlights and shadows both lose · N5.08.5 Optical see-through still shows the real world on power loss; video see-through blinds
• Nearby：N5.01 See-through Modes · N5.02 Virtual–Real Occlusion
• Search terms：video see-through occlusion · pass-through compositing · virtual-on-real occlusion