ARnnotate: An Augmented Reality Interface for Collecting Custom Dataset of 3D Hand-Object Interaction Pose Estimation

Title of the Paper

ARnnotate: An Augmented Reality Interface for Collecting Custom Dataset of 3D Hand-Object Interaction Pose Estimation

Paper Information

• Subject Area: Augmented Reality, 3D Hand-Object Interaction Pose Estimation, Dataset Collection
• Keywords: Augmented Reality, Hand-Object Interaction, 3D Pose Estimation, Dataset Collection, Human-Computer Interaction

Research Background and Problems

• Identified Problems or Challenges:

  • Current 3D pose estimation datasets often fail to cover diverse real-world scenarios, limiting practical applications.
  • Occlusion in hand-object interactions makes data annotation in 2D images challenging, especially for labeling hidden finger joints.
  • Dependence on laboratory-grade equipment restricts the generalizability of dataset collection and user operational convenience.

• Importance of the Problem:

  • 3D hand-object interaction pose estimation has significant application prospects in fields such as intelligent surveillance, education, industrial guidance, and mixed reality interface design.
  • Dataset performance directly impacts the reliability and adaptability of deep learning networks, which is critical for enabling user-customized scenarios on-site.

• Research Motivation and Related Work:

  • Previous studies primarily used multi-camera and sensor setups for 3D data collection but lacked user-friendly operational methods.
  • AR technology, with its spatial awareness capabilities, can address hand-object occlusion issues while offering simple bare-hand tracking and dataset creation functionalities.
  • The authors propose leveraging AR to enhance hand-object interaction dataset collection, enabling immediacy and continuity.

Solution

• Proposed Method/Solution:

  • Designed an interactive AR-based system, "ARnnotate," to allow users to easily create high-quality hand-object interaction datasets.
  • The system consists of two main steps:
    1. Create virtual boundary contours matching the physical object's shape and record interaction labels through bare-hand operations.
    2. Users interact with physical objects based on animations, while the system captures first-person view images and corresponding labels.

• Innovations:

  • Utilizes AR technology to mitigate hand-object occlusion issues, separating image collection from label recording to ensure annotation accuracy.
  • Generates boundary contours adaptable to object geometries using various creation tools (e.g., freehand drawing and primitive tools).
  • Provides visual guidance (e.g., gesture indicators and directional spheres) to assist users in accurately completing annotations and data creation.

• Implementation Steps and Key Technologies:

  1. Boundary Contour Creation: Users create virtual boundary contours using virtual primitives or freehand drawing.
  2. Label Recording: Users manipulate virtual objects and complete interactions, with the system recording 3D poses of objects and hands.
  3. Image Recording: Users interact with physical objects following AR animations, generating associated image data.
  4. Data Processing: Includes trajectory smoothing, time offset adjustment, and hand label correction to ensure data alignment.

Research Outcomes

• Specific Results:

  • Proposed a complete AR system workflow supporting users in collecting hand-object interaction datasets.
  • Successfully conducted experimental dataset collection for four objects using the system, achieving high-quality training results.
  • The system's user interface design was optimized to intuitively and conveniently support ordinary users.

• Advantages Compared to Existing Solutions:

  • Compared to traditional methods relying on complex hardware setups, this system is more cost-effective and suitable for ad-hoc dataset collection by ordinary users.
  • Effectively resolves hand-object occlusion issues while significantly improving annotation efficiency and quality.
  • Deep learning training results are comparable to benchmark datasets, demonstrating the practicality of custom datasets.

• Experimental or Evaluation Results:

  • Dataset annotation accuracy evaluation:
    • Average object translation error: 0.69 cm; rotation error: 2.73°.
    • Average hand annotation error: 0.85 cm.
  • Deep learning model performance:
    • 3D object pose estimation average accuracy: 0.7073 (IoU ≥ 0.5).
    • Hand pose estimation accuracy reached the level of existing benchmark datasets.
  • System usability evaluation:
    • User satisfaction score (SUS): 90.67, with positive user feedback.

• Limitations and Future Directions:

  • Limitations:
    • The current system primarily supports single-hand and fixed-gesture interaction scenarios, with complex gestures and bimanual interactions not yet addressed.
    • Efficiency in creating boundary contours for complex geometric objects needs improvement.
  • Future Directions:
    • Introduce physical contact point modeling to support complex gestures and bimanual interactions.
    • Optimize interaction trajectories with personalized recommendations to suit user preferences.
    • Expand to smartphone platforms to increase data collection scale and support the construction of general benchmark datasets.

Through the ARnnotate system, ordinary users can easily create customized hand-object interaction datasets, addressing many annotation challenges faced by traditional methods. This research contributes to advancing the field of 3D hand pose estimation and fosters more diverse applications across multiple domains.