HoloBots: Augmenting Holographic Telepresence with Mobile Robots for Tangible Remote Collaboration in Mixed Reality

Title of the Paper

HoloBots: Augmenting Holographic Telepresence with Mobile Robots for Tangible Remote Collaboration in Mixed Reality

Paper Information

• Domain: Mixed Reality, Remote Collaboration, Human-Computer Interaction
• Keywords: Mixed Reality, Remote Collaboration, Physical Presence, Mobile Robots, Actuated Tangible UI

Research Background and Problem Statement

• Identified Problems or Challenges:

  • Current mixed reality remote presence systems fail to meet the complex tactile interaction needs of real-world environments, such as grasping, manipulating objects, or physical collaboration.
  • The lack of physical embodiment diminishes the sense of presence for virtual users.
  • Existing robotic or physical representation systems face limitations in scalability, number of devices, and lack of user experience evaluation.

• Significance and Importance of the Research:

  • Rich remote physical interactions (e.g., touch, manipulation) are crucial for enhancing collaborative experiences.
  • Technological breakthroughs could provide users with more immersive and participatory remote collaboration scenarios.

• Motivation and Related Work:

  • Current technologies like Holoportation and traditional robotic telepresence systems have improved visual or spatial interactions but lack physical object manipulation capabilities.
  • Existing uses of mobile robots in mixed reality (e.g., PhyShare) have rarely explored the full design space or conducted comprehensive user studies.

Proposed Solution

• Proposed Method or Solution:

  • The HoloBots system: Combines holographic telepresence with desktop mobile robots to provide a tangible and manipulable remote collaboration interface.
  • Core technologies: Utilizes Hololens 2 and Azure Kinect for holographic user capture and rendering; employs Sony Toio robots for interaction with the physical environment.

• Innovations:

  • A comprehensive exploration of the design space for holographic remote collaboration, including object actuation, virtual embodiment, miniature scene interaction, shared virtual tangible interfaces, embodied guidance, and haptic communication.
  • Development of specific applications based on the design space, such as physical storytelling, remote gaming, and hands-on guidance.
  • Quantitative and qualitative user studies to validate the effectiveness of the new system in various collaborative scenarios.

• Implementation Steps and Key Technologies:

  1. System Architecture:
     • Azure Kinect RGB-D cameras capture the remote user's body movements.
     • Hololens 2 performs holographic rendering and hand tracking of the remote user.
     • Sony Toio robots enable desktop mobility and physical interaction.
  2. Core Interaction Design:
     • Object Actuation: Remote users manipulate local objects for movement and expression.
     • Shared Tangible UI: Local and remote users collaboratively manipulate virtual or physical user interfaces.
     • Miniature Body Interaction: Mobile robots represent a miniature embodiment of the remote user, interacting with the local environment.
     • Haptic Communication: Robots provide haptic feedback, such as guidance or notifications.
  3. User Experience Optimization:
     • Design of various robot add-ons for different scenarios (horizontal/vertical surfaces) and application needs (storytelling, UI adjustments, etc.).
     • Synchronization of multiple robots to ensure data consistency between local and remote environments.

Research Outcomes

• Specific Achievements:

  • Successfully developed a deployable HoloBots system supporting innovative remote collaboration interactions and scenario demonstrations.
  • User studies validated that the system significantly enhanced the sense of presence and shared experience in remote mixed reality collaboration.

• Experimental and Evaluation Results:

  • Social Presence:
    • The social presence score under the HoloBots condition (combining holograms and robots) was significantly higher than under "hologram-only" and "robot-only" conditions.
    • Users reported that the combination of holograms and robots made remote user interactions more realistic and present.
  • Cognitive Load and System Usability:
    • While there was no significant difference in cognitive load across conditions, the HoloBots condition scored higher in system usability.
  • User Preferences:
    • 75% of users preferred the fully integrated HoloBots hybrid system.

• Advantages:

  • Created a vital connection between virtual and physical realms, enabling users to interact in real-time using both virtual visuals and physical objects.
  • The system demonstrated high flexibility and task scalability, suitable for various surfaces and communication environments.

• Limitations and Future Directions:

  • Issues with synchronization delays between robots and holographic user movements.
  • Noise interference affecting user immersion.
  • Current implementation supports only two-person collaboration and a limited number of robots.
  • Recommendations include introducing more efficient robots, expanding the interaction area, and incorporating multimodal interaction methods.

Conclusion and Future Work

• Conclusion:

  • HoloBots significantly enhanced the quality and immersion of remote collaboration through holographic and physical robot interactions.
  • User study results demonstrated that the system has strong application potential compared to existing methods, suitable for scenarios such as storytelling, physical gaming, and precise guidance.

• Future Work Directions:

  • Expand the scale of the interaction area to support larger physical scenarios.
  • Integrate a wider variety of robots and IoT devices to enhance physical interaction capabilities.
  • Explore system dynamics in multi-user and multi-robot collaborative scenarios.