Reality Promises: Virtual-Physical Decoupling Illusions in Mixed Reality via Invisible Mobile Robots

DOI

Humans incessantly manipulate objects in their environment. Yet, mixed reality systems fall short of enabling seamless manipulations of the physical scene, constraining experiences to virtual effects. In this paper, we present the concept of Reality Promises, the mixed-reality illusion of manipulating the scene in ways only virtuality affords while secretly propagating virtual manipulations to physical reality. By decoupling virtual modes of manipulations from the physical mode of manipulation, Reality Promises create the illusion of manipulating the physical scene instantaneously, using magical forces or fantastical creatures. Concealed from the user, our system directs a mobile robot that manipulates physical objects between dynamic virtual-physical decoupling and recoupling points without revealing itself to the user. To render the robot invisible and physical objects interactable, we introduce a robot-aware 3D Gaussian splat rasterization, shading, and animation system that renders splats co-aligned with the local space into the user's passthrough view where needed. We systematically derive interaction protocols that provide cohesive end-to-end user experiences, such as materializing objects out of thin air or applying user or character-induced virtual forces to physical objects.