Understanding Context to Capture when Reconstructing Meaningful Spaces for Remote Instruction and Connecting in XR

Title of the Paper

Understanding Context to Capture When Reconstructing Meaningful Spaces for Remote Instruction and Connecting in XR

Paper Information

• Subject Area: Human-Computer Interaction (HCI), Extended Reality (XR), Multi-user Remote Collaboration, and Contextual Environmental Reconstruction
• Keywords: Extended Reality, 3D Reconstruction, Cross-generational Study, Context Capture, Remote Instruction, Hobbies, Gardening, Metaverse

Research Background and Issues

• Identified Problems or Challenges:

  1. Current XR technologies for remote collaboration heavily rely on spatial 3D reconstruction but often lack user participation, neglecting the contexts related to informal social experiences.
  2. It remains unclear how to capture social and emotional contexts in informal environments (e.g., hobbies or family activities).
  3. Privacy and content sharing in 3D reconstruction lack a clear operational framework, potentially affecting user experience.

• Significance: Interactions in informal environments (e.g., family gardening instruction or social sharing) carry rich cultural and emotional contexts. Improved context capture and XR tool design can reimagine these interactions, fostering social connections.

• Research Motivation and Related Work: The authors point out that current XR systems primarily target professional environments (e.g., surgical training or remote services), while informal activities and user customization needs remain underexplored. The potential of XR lies not only in its technical presentation but also in its ability to reconstruct user interaction experiences in a more human-centered manner.

Solution

• Methods or Solutions:

  1. Develop a multi-user collaborative XR prototype that supports remote guidance based on 3D models of gardening activities.
  2. Generate 3D models of garden spaces through field video recordings and use them in AR and VR remote interaction scenarios.
  3. Conduct in-depth testing of two prototype scenarios: simulating rainfall to guide water flow planning and visualizing sunlight changes to teach garden layout.

• Innovations:

  1. Systematically explore, for the first time, how XR can leverage 3D reconstruction to promote cross-generational remote instruction and social connection around hobbies like gardening.
  2. Propose new design principles for privacy and user participation to guide context capture in informal activities.

• Implementation Steps and Technologies:

  1. Create 3D models of gardening spaces: use video capture to create models and employ Agisoft Metashape for automated modeling.
  2. Build an XR web application driven by HTML and JavaScript, supporting operation on mobile devices, tablets, and Oculus VR headsets.
  3. Combine two interactive scenarios (rainfall and sunlight changes) to explore the environmental simulation capabilities of virtual objects.

Research Outcomes

• Specific Findings:

  1. Users indicated that XR tools are more suitable than videos for understanding the overall relationships in larger systems (e.g., water flow and sunlight distribution in gardening spaces).
  2. Users emphasized the importance of capturing "life-related" content, such as plant growth stages or animal activities.
  3. Reconstructed models and XR tools performed well in promoting cross-generational teaching and helping users perceive environmental contexts.

• Advantages Compared to Existing Solutions:

  1. User-participatory context capture provides higher personalization for social interactions in XR.
  2. Explored the emotional and educational relevance of privacy features and content sharing, contrasting with existing technology-driven models.

• Experimental or Evaluation Results:

  • A study involving approximately 18 participants showed that cross-generational users could reflect more deeply on the environmental impact and teaching of gardening through XR tools.
  • Although the visual quality of the models was limited, functional models still helped users establish connections with the context in the scenes.

• Limitations and Future Directions:

  1. Limitations:
     • The visual quality of the models remains suboptimal, and details may be lost in unknown contexts.
     • Some users were reserved about integrating technology into natural experiences, with some preferring videos and photos as more intuitive options.
  2. Future Directions:
     • Explore the possibility of reconstructing dynamic objects or life, especially non-static animals and natural behaviors.
     • Provide stronger privacy protection mechanisms to help users define the sensitivity and scope of shared objects.
     • Extend this approach to other informal activity spaces (e.g., kitchens or craft studios) to further validate its application potential.

Conclusion

This study pioneers the application of XR in informal, hobby-oriented spaces, promoting remote teaching and social connection through context capture and cross-generational interaction. It also offers recommendations for improving the design of future intelligent interactive environments.