Teachable Reality: Prototyping Tangible Augmented Reality with Everyday Objects by Leveraging Interactive Machine Teaching

Title of the Paper

Teachable Reality: Prototyping Tangible Augmented Reality with Everyday Objects by Leveraging Interactive Machine Teaching

Paper Information

• Domain: Augmented Reality (AR), Interaction Design, Machine Learning, and Prototyping
• Keywords: Augmented Reality, Mixed Reality, Prototyping Tools, Tangible Interaction, Everyday Objects, Interactive Machine Teaching, Human-Computer Interaction, Artificial Intelligence

Research Background and Issues

• Identified Problems or Challenges:

  • Existing AR prototyping tools lack the capability to support functional tangible AR applications, especially in integrating everyday objects and real-world interactions.
  • Marker-based tracking methods have limited flexibility and are unsuitable for complex interaction needs (e.g., object deformation, body movements).
  • Custom machine learning models, while flexible, require high technical expertise and are time-consuming.
  • There is a lack of effective tools that integrate computer vision with augmented reality.
  • Current workflows for prototyping demand extensive programming and hardware, reducing the convenience of rapid iteration.

• Significance:

  • Streamlining the rapid prototyping of functional AR applications can inspire creativity and foster effective communication within teams.

• Research Motivation and Related Work:

  • The authors aim to overcome the limitations of the complex processes in current prototyping methods by proposing an interaction-friendly tool that lowers technical barriers and enhances iteration efficiency.
  • The paper reviews existing AR prototyping tools, highlighting the gaps between low-fidelity and high-fidelity technologies, emphasizing the importance of tangible AR applications, and proposing a vision for defining interactions through user demonstrations.

Proposed Solution

• Proposed Approach:

  • Teachable Reality is a mobile AR prototyping tool that combines interactive machine teaching with in-situ AR creation.
  • It allows users to use everyday objects as inputs, capturing visual data and building machine learning models in real-time to define interactions.
  • The tool integrates a "trigger-action" interface to bind user-defined interaction states with AR scenes.

• Innovations:

  • Utilizes interactive machine teaching techniques to enable users to define AR prototype interactions in real-time, avoiding programming.
  • Offers an end-to-end solution from interaction detection to model training, deployment, and real-time testing.
  • Combines physical-virtual interaction with dynamic scene animations, enabling flexible interactions for object deformation, gestures, and other context-driven scenarios.

• Implementation Steps and Key Techniques:

  1. Interaction Detection: Uses mobile-based computer vision models to capture user-demonstrated interaction states.
  2. Scene Creation: Users directly manipulate AR virtual content to set its position, size, and behavior.
  3. Real-Time Testing and Deployment: Stored state scenes dynamically detect and automatically transition specific content, supporting rapid iteration.

Research Outcomes

• Specific Results:

  • Proposed an AR prototyping framework supporting real-time interaction detection, no-code creation, and scene deployment.
  • Demonstrated various application scenarios, including tangible and deformable interfaces, context-aware assistants, and gesture-driven AR experiences.
  • User studies and expert interviews confirmed the tool's effectiveness in lowering the barrier to functional AR prototyping.

• Advantages Over Existing Solutions:

  • More flexible than marker-based methods, without the need for printed markers.
  • Compared to machine learning methods, Teachable Reality significantly reduces programming and hardware costs.
  • Supports more diverse interaction inputs (appearance, position, relationships, etc.) and outputs (3D objects, sound effects, animations).

• Experimental or Evaluation Results:

  • Usability Study: 13 participants completed prototyping tasks in an average of 163 seconds, agreeing that the system reduced creation barriers for non-technical users.
  • Expert Interviews: Experts praised its fast, intuitive workflow and broad flexibility, suggesting the tool as a complement to existing methods.

• Limitations and Future Directions:

  • Limitations:
    • Relies on input detection from computer vision models, with limited accuracy in complex environments.
    • Current object tracking methods are highly color-dependent, making them less adaptable to complex scenes.
  • Future Directions:
    • Enhance the robustness of detection algorithms and explore multimodal inputs such as depth cameras.
    • Introduce complex narrative structures (e.g., state machines or multi-branch logic).
    • Provide designers with interpretable AI interfaces to optimize the model training experience.