InstruMentAR: Auto-Generation of Augmented Reality Tutorials for Operating Digital Instruments Through Recording Embodied Demonstration

Paper Title

InstruMentAR: Auto-Generation of Augmented Reality Tutorials for Operating Digital Instruments Through Recording Embodied Demonstration

Paper Information

• Topic Area: Application of augmented reality technology in digital instrument operation and automated tutorial generation
• Keywords: Augmented reality, immersive creation, embodied demonstration, wearable devices, haptic feedback, interactive tutorials

Research Background and Problem

• Identified Problems or Challenges: The current process of creating augmented reality (AR) tutorials is not seamless, requiring users to frequently switch between operating instruments and interacting with virtual elements. This manual adjustment of virtual arrow positions, sizes, and orientations is not only tedious but also prone to errors, especially when task steps are complex, increasing users' cognitive load.

• Why This Problem is Important: With the proliferation of digital instruments, there is a growing demand for precise and intuitive operation guides. Compared to traditional tutorials (images or videos), AR tutorials provide real-time, direct visual guidance, reducing cognitive load and improving operational efficiency.

• Research Motivation and Related Work: The authors aim to automatically generate AR tutorials by recording users' demonstration operations, reducing user burden and improving tutorial creation efficiency. They were inspired by related work, such as using cameras to record object trajectories in assembly tasks, but these approaches face challenges like hand occlusion and device complexity. They also focus on developing subsequent support features like real-time feedback.

Solution

• Proposed Method: The authors designed and implemented the InstruMentAR system, which automatically generates AR tutorials based on users' embodied demonstrations. The solution involves hardware and software collaboration:

  • A custom hand-worn wearable device equipped with pressure sensors and haptic modules.
  • A multimodal detection algorithm based on gesture tracking and pressure sensors to capture instrument operations.
  • A tutorial creation and access interface on the Hololens 2 platform.

• Innovative Aspects of the Solution:

  • Automated tutorial generation without manual adjustment of virtual arrows.
  • A warning mechanism that uses haptic and visual feedback to prevent user errors.
  • Seamless integration with instrument interaction workflows.

• Implementation Steps and Key Technologies:

  1. Creation Mode: Users operate instruments while wearing the hand device. The system records pressure sensor inputs and gesture trajectories, automatically generating corresponding virtual arrows, supplementary text, and image information.
  2. Access Mode: When novice users access the tutorial, the system tracks their operations in real time, automatically advancing the tutorial according to correct steps and providing feedback before errors occur.
  3. Hardware Design: The hand-worn wearable device minimizes interference with gesture tracking, featuring pressure sensors and haptic feedback modules designed to fit finger contours.
  4. Multimodal Tracking: Pressure sensors detect the start and end of operations, while gesture data determines the operation's position and scale.

Research Outcomes

• Specific Results:

  • Developed a system for automatically generating AR tutorials, supporting five types of instrument interfaces (buttons, toggles, knobs, sliders, touchscreens) and a detection algorithm for various gesture combinations, along with a real-time feedback mechanism.
  • User studies showed the system significantly reduced tutorial creation time (by approximately 46%) and lowered users' cognitive load during operations.

• Advantages Compared to Existing Solutions:

  • High degree of automation: InstruMentAR greatly simplifies the tutorial creation process compared to traditional immersive AR creation tools that require manual arrow adjustments.
  • Optimized occlusion and device compatibility issues: Direct tracking via the hand reduces occlusion problems compared to tracking the instrument interface.
  • Enhanced error tolerance: Real-time warnings prevent frustration caused by operational errors.

• Experimental or Evaluation Results:

  • In performance tests of the detection model, the system achieved an operational accuracy of approximately 88%, with most errors stemming from gesture tracking failures.
  • Users expressed high satisfaction with both the tutorial creation and access modes. The system's usability scores were 81/100 (creation mode) and 68/100 (access mode), indicating a positive user experience.

• Limitations and Future Directions:

  • Limitations:

    • The hand-worn wearable device caused some discomfort, with some users perceiving it as adding operational complexity.
    • The current system only supports predefined gestures, limiting personalization.
    • The tutorial generation and access processes only support linear paths, with limited support for recovering from user errors.

  • Future Directions:

    • Expand gesture recognition capabilities to support user-defined gestures and multi-touch interactions.
    • Introduce instrument screen state modeling to further optimize tutorial generation and feedback mechanisms by analyzing real-time instrument data.
    • Improve wearable device design to enhance comfort and optimize sensing capabilities (e.g., adding multi-sensor support).
    • Enhance tutorial error-handling features (e.g., adding support for branching paths) to improve tutorial fault tolerance and flexibility.