Volumetric Hybrid Workspaces: Interactions with Objects in Remote and Co-located Telepresence

Literature Title

Volumetric Hybrid Workspaces: Interactions with Objects in Remote and Co-located Telepresence

Literature Information

• Subject Area: Human-Computer Interaction, Mixed Reality, Remote Collaboration
• Keywords: Augmented Reality, Mixed Reality, Volumetric Capture, Remote Collaboration, Partially Distributed Teams, Workspace Awareness

Research Background and Issues

• Problems or Challenges:

  1. Traditional remote presence technologies often rely on screen-based mediums (e.g., video conferencing), which limit the expression of non-verbal communication (e.g., gestures, pointing).
  2. Existing remote collaboration technologies often lack support for spatial coordination and interaction with multi-format objects (physical, virtual, volumetric).
  3. Two-way volumetric capture is typically applied in dual-user scenarios, with limited exploration of complex, multi-user environments.

• Significance:

  • With the increasing prevalence of remote work, there is a growing demand for remote collaboration technologies that support natural interaction, enhance spatial awareness, and improve non-verbal communication.
  • Breaking the limitations of traditional mixed reality technologies confined to single workspaces, introducing remote users into fully reconstructed shared virtual environments can enhance collaboration efficiency and experience.

• Research Motivation and Related Work:

  • Existing technologies such as Room2Room and Holoportation provide a foundation for remote collaboration but primarily focus on single workspaces or virtualized interactions.
  • This study aims to explore a mixed reality platform supporting multi-user environments, enabling natural interactions between remote and local users through physical, volumetric, and virtual objects.

Solution

• Method or Solution: The authors propose a mixed reality-based volumetric telepresence platform (Volumetric Hybrid Workspaces) that can capture and reconstruct multi-user physical environments and objects in real time. Key features include:

  • Real-time Two-way Communication: Creating shared workspaces using physical objects and human volumetric renderings.
  • Multi-format Object Classification: Supporting interactions with physical objects, remotely rendered volumetric objects, and purely virtual objects.

• Innovations:

  • Supports real-time two-way interactions among multiple users, overcoming the limitations of previous dual-user scenarios.
  • Integrates local and remote users' physical representations to create a unified shared space experience.
  • Provides interaction support for different categories of objects (physical, volumetric, virtual).

• Implementation Steps and Techniques:

  1. Volumetric Capture Device Deployment: Azure Kinect cameras are installed in two physical spaces for real-time environment capture.
  2. Data Stream Compression and Processing: Captured RGB-D data is optimized for low-latency transmission using compression techniques such as H.264 encoding and RVL algorithms.
  3. User Presentation: Microsoft Hololens 2 is used to present local and remote scenes to users via optical see-through mixed reality devices.
  4. Environment Calibration: CV methods are employed to establish spatial anchors between the two spaces, ensuring accurate alignment of remotely captured objects within the shared space.

Research Outcomes

• Specific Results:

  1. Successfully developed a real-time two-way volumetric mixed reality workspace, providing users with a natural and seamless collaborative experience.
  2. Defined three object formats (physical, volumetric, virtual) and explored their usage in collaborative tasks.
  3. User studies demonstrated that the system supports spatial coordination, enhances remote users' sense of participation, and optimizes non-verbal communication for collaboration.

• Advantages:

  • Overcomes the limitations of traditional screen-based remote collaboration, which lacks non-verbal communication methods.
  • Supports interaction with physical objects, naturally integrating them with remote environments.
  • Two-way capture provides accurate and visually real-time user renderings, enhancing the sense of presence in collaboration.

• Experiment and Evaluation Results:

  • 18 participants completed 7 tasks across 6 experiments, reporting that the system improved workspace understanding and collaboration.
  • Virtual objects were perceived as highly visible, but volumetric objects were often mistaken for physical objects due to visual similarity.
  • Tasks involving writing on physical cards enhanced psychological ownership of content creation and promoted tightly coupled interactions.

• Limitations and Future Directions:

  • Limitations: The experiment scale was small and limited to scenarios involving two shared spaces; the dynamic potential of virtual content was not fully utilized.
  • Future Directions:
    • Explore designs for multi-space asymmetric volumetric remote collaboration.
    • Add capabilities for dynamic generation of virtual content, such as virtual private spaces.
    • Expand system support for non-adjacent spaces or integrate different user representations (e.g., virtual avatars and volumetric avatars).