Sketched Reality: Sketching Bi-Directional Interactions Between Virtual and Physical Worlds with AR and Actuated Tangible UI

Document Title

Sketched Reality: Sketching Bi-Directional Interactions Between Virtual and Physical Worlds with AR and Actuated Tangible UI

Document Information

• Subject Area: Human-Computer Interaction (HCI), Augmented Reality (AR), Tangible User Interface (TUI)
• Keywords: Augmented Reality, Mixed Reality, Actuated Tangible Interface, Group User Interface, Interaction Design

Research Background and Problem

• Identified Issues or Challenges: Existing augmented reality (AR) drawing tools feature one-way interactions where the physical world influences virtual graphics, but virtual objects (e.g., drawn lines or shapes) do not exert corresponding reactive forces on the physical environment.

• Research Importance: The current one-way interaction mechanism limits the high coupling and collaboration between virtual and physical environments. Achieving bi-directional interaction would expand AR beyond mere visual demonstrations, enhancing physical interaction realism and engagement through feedback. Potential applications include education, design, and entertainment.

• Research Motivation and Related Work: The authors explored the potential of combining AR drawing with actuated tangible interfaces. Early studies, such as Reactile, attempted limited bi-directional interactions but were constrained in scope. These studies indicate a vast design space for bi-directional integration but lack systematic exploration and implementation.

Solution

• Method Overview: A novel approach called "Sketched Reality" is proposed, combining an iPad-based AR drawing interface with small robots (Sony Toio) to establish a framework for bi-directional interaction between virtual and physical worlds.

• Innovations:

  1. Defined and implemented a comprehensive bi-directional interaction design space, encompassing four categories: constraints, geometric relationships, external force application, and dynamic collisions.
  2. Introduced a real-time embedded interactive drawing and actuated device-based bi-directional physical model.
  3. Enabled real-time modifiable virtual graphics and interaction rules, allowing custom interaction effects without programming.

• Implementation Steps and Core Technologies:

  1. Utilized Sony Toio robots as small physical actuated interfaces, leveraging their built-in cameras and Bluetooth communication for precise tracking.
  2. Built a dynamic AR canvas using WebXR technology (primarily A-Frame and 8th Wall) to synchronize virtual graphics with the physical robots' coordinates.
  3. Developed refined mechanisms for physical and virtual constraints (e.g., geometric relationship locking, springs) to control virtual graphics and robots for bi-directional operation.

Research Outcomes

• Specific Results:

  1. Developed a proof-of-concept system supporting AR drawings that directly influence physical robots, while robot movements also impact virtual objects.
  2. Summarized eight distinct types of virtual-physical interaction modes, showcasing a rich design space.

• Advantages Over Existing Methods:

  • Introduced an unprecedented bi-directional interaction model, overcoming the limitations of traditional one-way interactions.
  • Allowed users to spontaneously combine AR and physical devices to construct complex interaction models without prior programming.

• Experiments or Evaluation Results: Demonstrated various application scenarios, including physical education, mechanical exploration, game design, and on-site robot programming.

  • Physical Education: Enabled interactive learning of physics concepts by constructing teaching models such as Newton's cradle or Rube Goldberg machines.
  • Mechanical Exploration: Allowed users to design and experiment with dynamic mechanical systems, such as simulating piston mechanics via robot control.
  • Tangible Games: Supported drawing slingshot or pinball games, enhancing gameplay experience through robot integration.
  • On-Site Programming: Users could define interaction rules between robots via sketching, creating complex procedural behaviors.

• Limitations and Future Directions:

  1. Current implementation is limited by the precision and immersion of mobile AR devices (e.g., iPad); future work could integrate head-mounted displays (HMDs) or projection technologies to optimize user experience.
  2. The system is currently confined to desktop robots; future expansions could target larger-scale environments (e.g., home spaces) or other devices (e.g., IoT).
  3. Formal user studies have not yet been conducted; subsequent research could explore usability, educational benefits, and immersive experience outcomes.