Augmented Reality and Robotics: A Survey and Taxonomy for AR-enhanced Human-Robot Interaction and Robotic Interfaces

Title of the Paper

Augmented Reality and Robotics: A Review and Classification for Enhanced Human-Robot Interaction and Robotic Interfaces

Paper Information

• Subject Area: Augmented Reality (AR), Human-Robot Interaction (HRI), and Robotic Design
• Keywords: Augmented Reality, Mixed Reality, Robotics, Human-Robot Interaction, Dynamic Physical Interfaces, Shape-Changing Interfaces, AR-HRI, VAM-HRI

Research Background and Issues

• Identified Problems or Challenges:

  1. Traditional robotic interaction methods are limited to the robot's physical or visual feedback capabilities, such as conveying information through movements, gaze, or light displays. However, these feedback methods often lack expressive power.
  2. Many existing studies in robotics focus on single-case design explorations, lacking systematic classification and analysis.
  3. The potential of AR in HRI remains underutilized to meet the rapidly growing interaction demands.

• Significance:

  • As robots enter more everyday environments, designing more efficient interfaces for human-robot interaction becomes critical.
  • AR can overcome physical constraints, offering new interaction possibilities between humans and robots while reducing the cognitive burden of shifting visual focus.

• Research Motivation:

  • Extract design practices and strategies from 460 past studies to construct a comprehensive classification system for AR and robotics, advancing academic and industrial research in the field.
  • Provide key issues and guidance for future research through synthesis and discussion.

Solutions

• Proposed Methods or Solutions:

  • This paper proposes a design space classification framework encompassing eight dimensions:
    1. Application methods of augmented reality.
    2. Characteristics of augmented robots.
    3. Purposes and advantages of visual augmentation.
    4. Types of information presented.
    5. Design components and strategies for visual augmentation.
    6. Interaction technologies and modes.
    7. Application domains.
    8. Evaluation strategies.
  • Emphasizes how AR can integrate human and machine interaction needs, creating new interfaces and interactions between the real and virtual.

• Innovative Aspects:

  • Provides a systematic and comprehensive classification framework, redefining the research field from the perspectives of interaction design and visual augmentation, addressing gaps in existing systematic studies.
  • Expands the application scope of AR/HRI by integrating research on robotic user interfaces (e.g., dynamic tactile and shape-changing interfaces).
  • Proposes new open research directions, including improving system usability, expanding AR interface design, and exploring interaction possibilities.

• Implementation Steps and Key Technologies:

  • Data Collection: Gathered 925 papers from academic databases, ultimately selecting 460 for the research corpus.
  • Open Coding: Extracted design space dimensions through preliminary classification, refined through discussions among multiple authors.
  • Systematic Organization: Extracted key references and design cases from the corpus, created a visual classification summary, and categorized them into the identified dimensions.

Research Outcomes

• Specific Outcomes:

  • Developed a comprehensive classification framework covering 460 studies, summarizing eight key dimensions for applying AR in robotic interaction design.
  • Provided numerous design frameworks and examples to help researchers quickly locate relevant literature and position their work.
  • Proposed eight major directions for future research, including:
    1. Enhancing reliability and effectiveness in real-world deployments.
    2. Bringing systems into real-world environments for "in-the-wild" validation.
    3. Breaking physical design limitations to create more flexible and dynamic robotic appearances.
    4. Developing immersive prototyping tools for direct manipulation and iterative optimization.
    5. Investigating the use of embedded visualizations to support real-time decision-making.
    6. Exploring how AR can make robots more interpretable and explorable.
    7. Advancing natural, multimodal interaction designs.
    8. Better integrating virtual and physical worlds.

• Improvements to Existing Solutions:

  • Provided real-time visual feedback in human-robot collaboration, addressing the expressive limitations of traditional interaction methods.
  • Enhanced the interpretability of robots in complex decision-making environments by visualizing and clarifying their decision processes.

• Experimental or Evaluation Results:

  • The classification summary revealed that "labels and annotations" and "paths and trajectories" are the most commonly used design methods in AR-HRI systems.
  • Most practical studies focus on industrial and domestic application domains, with potential for further development in areas like remote collaboration and healthcare.
  • Evaluations were primarily conducted through user studies, technical tests, and proof-of-concept demonstrations, with limited deployment in real-world environments.

• Limitations and Future Directions:

  • Limitations:
    • Some experiments were confined to laboratory settings, requiring validation in real-world scenarios.
    • Current AR devices face limitations in display resolution, battery life, comfort, and tracking accuracy.
  • Future Directions:
    • Promote the widespread practical use of AR systems, particularly by improving technical stability to address potential risks in safety-critical fields.
    • Explore novel human-robot interaction design approaches, especially immersive scenario design tools for complex interaction tasks.
    • Integrate more diverse interaction modalities (e.g., gestures, eye-tracking, voice) to achieve natural multimodal interactions.
    • Combine embedded data visualization with optimized robotic navigation to provide enhanced support for real-time decision-making.