Volumetric Mixed Reality Telepresence for Real-time Cross Modality Collaboration

Document Title

Volumetric Mixed Reality Telepresence for Real-time Cross Modality Collaboration

Document Information

• Subject Area: Mixed Reality (MR) technology and remote collaboration
• Keywords: Extended Reality, Augmented Reality, Virtual Reality, Volumetric Capture, Telepresence, Avatar, Collaboration, Conversational Grounding

Research Background and Issues

• Problems and Challenges:

  • Most current telepresence systems only support two users and lack research on cross-modality collaboration in multi-user environments.
  • How does low-fidelity volumetric capture affect user experience? Current literature supports high fidelity, but low fidelity may also be effective.

• Significance:

  • Cross-modality mixed reality enhances the sense of immersion and realism in remote collaboration.
  • Mixed reality telepresence can facilitate applications in real-world work scenarios, effectively supporting team collaboration in contexts such as teaching.

• Research Motivation and Related Work:

  • Inspired by existing volumetric capture technologies (e.g., Holoportation), the authors designed a solution more suitable for consumer-grade hardware to reduce costs and improve accessibility.
  • The authors proposed a technical platform emphasizing real-time performance, ease of deployment, and multi-user support.

Solution

• Methods and Innovations:

  • The authors proposed a mixed reality system called Virtual Co-Presence (VCP), which supports multi-user real-time collaboration through volumetric rendering and full-body virtual avatars.
  • Innovations include:
    1. Support for full-mode cross-modality collaboration (integrating physical environments, augmented reality, and virtual reality).
    2. No special environmental requirements (e.g., green screens), making it easy to configure.
    3. Optimization of data transmission using volumetric capture technology to reduce bandwidth usage.

• Implementation Steps and Techniques:

  • Volumetric data capture was achieved using an array of Azure Kinect depth cameras, combined with Unity and Microsoft Hololens to provide cross-modality support.
  • Real-time data transmission with low latency was implemented using an optimized network protocol (eNet, LZ4 compression).
  • The system allows users to communicate and interact via volumetric rendering or 3D virtual avatars, integrating spatial audio to enhance depth perception.

Research Outcomes

• Specific Results:

  • Successfully developed and implemented the VCP system, supporting cross-modality remote collaboration.
  • System evaluation was conducted in real teaching scenarios, demonstrating its effectiveness in shared workspace awareness, co-presence, and cross-modality spatial distance.

• Advantages and Comparisons:

  • Compared to existing technologies, VCP offers a more flexible and easily deployable multi-user volumetric capture solution.
  • Supports multi-user remote collaboration and adapts to dynamic environments without requiring expensive hardware.

• Experiments and Evaluation Results:

  • The experimental design involved real training scenarios (tool usage training), revealing:
    • Significant differences in collaboration experiences between local and remote users (remote users perceived a stronger sense of "presence").
    • Full-body volumetric rendering effectively conveyed non-verbal communication characteristics of personal space.
    • Users expressed diverse preferences for different types of avatars (e.g., 360° volumetric capture) and communication methods.

• Limitations and Future Directions:

  • The current system has limitations in visual detail fidelity and camera coverage, which need improvement in future iterations.
  • Future plans include exploring bidirectional volumetric rendering technology, enabling both remote and local users to experience volumetric scenes.
  • Better support for spatial awareness, including providing local users with visual/audio feedback on the remote users' positions.

Conclusion

This study developed an economical, practical, and innovative mixed reality volumetric rendering system. User research demonstrated its potential to support cross-modality remote collaboration in dynamic environments, while also identifying design limitations and directions for future optimization.