SpatialProto: Exploring Real-World Motion Captures for Rapid Prototyping of Interactive Mixed Reality

Title of the Paper

SpatialProto: Exploring Real-World Motion Captures for Rapid Prototyping of Interactive Mixed Reality

Paper Information

• Research Area: Human-Computer Interaction Design, Virtual Reality/Augmented Reality System Development, Low-Barrier Prototyping Tools
• Keywords: Spatial Prototyping, Mixed Reality, Interaction, Animation, No Programming Skills, Depth Sensing, MR Headsets, User Study

Research Background and Problem

• Problems and Challenges:

  1. With the growing popularity of Augmented Reality (AR), Virtual Reality (VR), and Mixed Reality (MR), traditional 2D interface-centric interaction paradigms are undergoing a transformation, with spatial computing emerging as a new form of interaction.
  2. Designing experiences for spatial computing faces significant challenges, particularly in the early stages of rapid prototyping. Dynamic interaction design and 3D animation requirements often demand advanced programming and 3D modeling skills, which are difficult for ordinary users and non-experts to handle.
  3. Although a few tools have explored rapid prototyping, they are primarily limited to static content design or VR-specific environments. Furthermore, most tools require device switching or programming, resulting in inefficient user experiences.

• Significance of the Research: This study focuses on lowering the barrier to entry for spatial system prototyping, making it more accessible to users without technical backgrounds. This not only supports the rapid development of MR experiences but also helps explore the potential and broader applications of spatial prototyping.

• Motivation and Related Work: The authors argue for the need for more efficient spatial prototyping tools that can complete all design stages directly within the MR environment, emphasizing simplicity, flexibility, and usability for non-technical users. Previous research, such as ProtoAR and RoomAlive, has not fully addressed the comprehensive challenges of dynamic interactive MR experiences.

Solution

• Method or Solution: The authors propose the SpatialProto system—a rapid spatial prototyping tool designed for depth-sensing MR headsets.

  • Core functionalities include capturing dynamic objects in real-world scenes using depth-sensing cameras, real-time editing, and designing and testing interaction prototypes directly within the MR environment.
  • A pipeline with three key stages was developed: Recording, Editing, and Prototyping.

• Innovations:

  1. Support for Dynamic Animations: Enables recording real-world motions and transforming them into interactive prototype animations.
  2. Accessibility for Non-Experts: No programming skills or specialized setups are required, with a simple and user-friendly workflow.
  3. Immersive Design Experience: Users can capture, edit, and test directly within the MR environment without leaving the MR device.

• Implementation Steps and Key Technologies:

  1. Recording Stage: Utilizes an RGB-D camera to capture 3D scenes, recording depth information, RGB textures, and related metadata. Supports single-frame capture, continuous video recording, and frame-by-frame stop-motion recording.
  2. Editing Stage: Provides various cropping and masking techniques (e.g., depth plane masking, chroma keying, object detection algorithms) to modify colors or image effects for content adjustment.
  3. Prototyping Stage: Users can adjust the spatial position, rotation, and scaling of animations, as well as set triggers and interaction logic (e.g., triggered by head movement, gestures, or gaze).

Research Outcomes

• Specific Results:

  1. The SpatialProto system was successfully designed and implemented, enabling users to create dynamic interactive MR prototypes (e.g., game interactions, interior design simulations) embedded in real-world spaces.
  2. Experiments demonstrated that users without technical backgrounds could complete a full animation design within 20 minutes, significantly reducing the learning curve.

• Advantages:

  1. Compared to existing methods (e.g., ProtoAR), it supports more complex dynamic animation and interaction logic design, with low hardware costs (achievable with commercially available VR devices).
  2. The prototyping process does not require users to leave the MR device, enhancing design efficiency and immersive experience.

• Experimental or Evaluation Results:

  1. A user study involving 9 participants from various disciplines (e.g., computer science, architectural design) found the tool to be user-friendly with a short learning curve.
  2. Successfully designed various animation prototypes, including a jumping duck animation and a clay-modeled explosion. Users creatively suggested additional use cases for the system beyond animation design.

• Limitations and Future Directions:

  1. Visual Limitations: The current system only supports user-perspective capture and cannot record the full 3D shape of objects. Future work could integrate machine learning techniques to complete 3D data.
  2. Performance Optimization: The system experiences significant frame rate drops in complex scenes, which could be addressed by compressing depth data or optimizing algorithms.
  3. Object Detection Algorithms: Current ML detection methods have low accuracy and long inference times. Future versions could further optimize algorithm selection and enable GPU acceleration.

Conclusion

SpatialProto is an innovative system that integrates dynamic animation capture, real-time editing, and immersive design, pioneering low-barrier prototyping for non-expert users on MR devices. Through user studies and application examples, the system significantly enhances the accessibility and immersive experience of spatial design, marking a new direction for future MR creation tools.