Scene Responsiveness for Visuotactile Illusions in Mixed Reality

Title of the Paper

Scene Responsiveness for Visuotactile Illusions in Mixed Reality

Paper Information

• Subject Area: Visuotactile illusions and scene responsiveness in Mixed Reality (MR)
• Keywords: Mixed Reality, visuotactile consistency, spatial computing, scene interaction, compressed reality, diminished reality, virtual-physical fusion, scene editing, virtual interaction

Research Background and Problem

• Problems or challenges identified by the authors:

  1. Current mixed reality systems can integrate virtual content into physical scenes but lack the capability to extend visual illusions for realistic environmental manipulation.
  2. There is a lack of high-fidelity visual illusions that simulate the impact of virtual actions on physical scenes.
  3. How to provide an interaction experience with consistency between tactile and visual feedback without compromising physical realism.

• Motivation and significance of the research:

  1. To achieve scene responsiveness in virtual-physical fusion, allowing users or virtual characters to manipulate physical objects in mixed reality scenarios.
  2. To provide more immersive and interactive experiences for applications such as gaming, remote interaction, and entertainment.
  3. To overcome the limitations of existing solutions in terms of visual and interaction consistency between physical and virtual content.

• Related work:

  1. Scene consistency: Techniques for occlusion, collision, and lighting consistency in augmented reality [AR, MR rendering technologies].
  2. Scene editing: Research on replacing physical objects with fictional ones in diminished reality or augmented reality editing.
  3. Scene interaction in MR environments: Exploring the behavior of virtual characters and the adaptation of virtual content in physical scenes under semantic consistency.

Solution

• Methods or solutions proposed by the authors:

  1. Scene Responsiveness: Creating visual illusions through video see-through mixed reality devices, making virtual actions appear to influence the physical scene.
  2. Object Virtualization Technology: Real-time switching of the physical state of objects, hiding physical objects and replacing them with virtual replicas.
  3. Development of two types of illusion experiences:
     • Daydreaming Type: Virtualizing objects visually and preventing tactile inconsistency through "object evasion" and "re-physicalization."
     • Copperfield Type: Illusions of disappearance and reappearance akin to a magician’s tricks, enhancing visuotactile consistency through cross-object re-physicalization.

• Innovations:

  1. Proposing a new visual illusion framework that enables continuous interaction between virtual content and physical objects.
  2. Achieving high-fidelity visuotactile consistency, ensuring users find it difficult to detect the disconnection between visual and physical reality.
  3. Implementing dynamic occlusion and lighting consistency through multi-layer spatial computing and occlusion architecture.

• Implementation steps and techniques:

  1. System Architecture:
     • TwinBuilder Component: Generates digital representations by scanning and semantically annotating the space and objects to create twin counterparts.
     • RealityToggle Component: Handles object occlusion, depth-consistent rendering, and lighting processing.
     • Spielberg Component: Controls the actions of virtual characters and manages user-environment interactions.
  2. Integration of object models using the Unity engine, optimizing character and scene interaction through navigation meshes and animation systems.

Research Outcomes

• Specific achievements:

  1. Successfully implemented high-fidelity visual illusions in video see-through mixed reality devices, ensuring depth and lighting consistency between virtual objects and physical scenes.
  2. Demonstrated the potential for complex interactions and narrative scenarios through the design of two illusion types (Daydreaming and Copperfield).
  3. User studies revealed that the framework can create convincing "visuotactile consistency."

• Comparison with existing solutions and advantages:

  1. Enhanced the immersion of scene responsiveness, allowing virtual characters to interact semantically with physical scenes.
  2. Overcame issues of visual segmentation and tactile inconsistency in traditional mixed reality.

• Experimental or evaluation results:

  1. Among 20 experimental participants, 18 were successfully deceived into believing that the visual illusions could realistically manipulate physical objects.
  2. Users rated the visual consistency of the virtualization and re-physicalization processes highly, though there is still room for improvement.

• Limitations and future directions:

  1. Current 3D model scanning technologies suffer from geometric distortion and insufficient photogrammetry precision.
  2. Visual artifacts may occur when hidden objects move within the user’s line of sight, requiring optimization of dynamic occlusion techniques.
  3. Future work could explore higher-precision scene modeling methods based on Neural Radiance Fields (NeRF) and object tracking.

Summary: This study proposes a new visual illusion framework for the mixed reality field, demonstrating its potential applications while identifying areas for improvement. It lays a technical foundation for future research.