Rapido: Prototyping Interactive AR Experiences through Programming by Demonstration

Title of the Paper

Rapido: Prototyping Interactive AR Experiences through Programming by Demonstration

Paper Information

• Domain: Rapid prototyping of augmented reality (AR) interactive experiences
• Keywords: AR interaction design, rapid prototyping, video prototyping, programming by demonstration, state machine, user interface design

Research Background and Problem

• Identified Problems or Challenges:
  Current augmented reality application design tools require users to have advanced programming skills, making it difficult for non-technical designers to quickly create interactive prototypes. Existing tools fail to balance flexibility in the early design stages with the need for rapid iteration. While video prototypes are suitable for early design, their output is typically non-interactive.

• Significance of the Problem:
  Augmented reality applications are becoming increasingly popular, with rapid growth in downloads and usage. However, the complexity of existing design tools hinders broader creative dissemination and does not effectively support interaction design based on common input methods such as touch gestures.

• Research Motivation and Related Work:
  By leveraging the "Programming by Demonstration" (PbD) approach, the authors aim to design a tool that eliminates the need for coding, enabling non-interactive video prototypes to be transformed into executable state machines. This approach lowers the technical barrier and enhances the ability for rapid iteration in the early stages of design.

Solution

• Proposed Solution:
  The authors developed a prototyping system called Rapido, which allows designers to record video scenes, sketch, and demonstrate interaction patterns using mobile devices, enabling the rapid creation of interactive augmented reality prototypes. These interactions are ultimately converted into real-time interactions represented by state machines.

• Innovations:

  1. A workflow combining video prototyping with state machine representation.
  2. Introduction of PbD technology to enhance the rapid prototyping process.
  3. An iterative workflow comprising three stages: video recording, programming by demonstration, and prototype testing.
  4. Empowering designers to create interactive prototypes without coding, with support for inference based on visual and behavioral examples.

• Implementation Steps and Key Techniques:

  1. In the video prototyping stage, designers can record video backdrops, sketch, and demonstrate user inputs and actions, such as defining interactions via touch screens or drag gestures.
  2. In the programming by demonstration stage, the system infers initial state machine logic by defining states and timed operations.
  3. In the testing stage, designers can validate interaction effects and debug through real-time or video playback.

Research Outcomes

• Specific Outcomes:
  Rapido successfully converts non-interactive video prototypes into interactive state machines and effectively supports touch gesture-based mobile augmented reality design. Experiments demonstrated interactive prototypes for scenarios such as furniture placement applications and extended capabilities for other use cases like planetary tracking.

• Comparison with Existing Solutions:

  1. Compared to traditional video prototyping tools, Rapido processes spatial information and generates interactive prototypes rather than merely producing non-interactive video outputs.
  2. Compared to commercial tools (e.g., Adobe Aero), Rapido better supports continuous inputs (e.g., dragging) and includes testing functionality without requiring manual re-entry of interactions.

• Experimental or Evaluation Results:
  Example applications demonstrate that the system supports common AR scenarios (e.g., furniture placement, measurement tools) and provides designers with a simplified method for interaction design. System performance was validated in real design scenarios, with compatibility and functionality further illustrated.

• Limitations and Future Directions:

  • Limitations:

    1. The current state machine model does not support complex computational behaviors, such as conditional or counter variable operations.
    2. It lacks support for collision detection, physics simulation, or advanced visual tracking features, such as facial or object tracking.
    3. The entire framework is dependent on specific platforms (iOS and mobile AR).

  • Future Directions:

    1. Adding support for collision and physical interactions to expand existing behavioral rules.
    2. Supporting multi-user collaboration and URL-based testing functionality.
    3. Exploring the possibility of prototyping directly on AR/VR head-mounted devices to eliminate spatial constraints.
    4. Evaluating and optimizing designers' acceptance of the system and the usability of state machines.

Conclusion

Rapido is an innovative augmented reality design tool that integrates video prototyping, PbD, and state machine technologies, providing designers with a powerful and user-friendly method for generating interactive prototypes. Despite its functional limitations, its minimalist design language and rapid iteration capabilities offer new possibilities for future AR design tools.