AI-generated AR Reassembly Guidance from Disassembly Videos to Scaffold Everyday Repair

Paper Title

AI-generated AR Reassembly Guidance from Disassembly Videos to Scaffold Everyday Repair

Publication Info

• Topic area: Augmented Reality (AR) and AI-based repair guidance for household appliances.
• Keywords: Augmented Reality, repair guidance, multimodal large language models, disassembly videos, reassembly planning, user study, self-efficacy, mixed-media visualization, task segmentation, component extraction.

Background and Problem

• Problem / challenge: Repairing household appliances is challenging due to the lack of accessible, up-to-date product manuals and the high costs of creating product-specific 3D assets for AR systems. Current AR solutions often depend on CAD models and pre-constructed assets, limiting scalability and usability.
• Significance: Addressing repair challenges can empower users, extend product lifespans, and contribute to sustainability goals. Providing accessible, scalable repair guidance can lower barriers to engagement in repair activities.
• Motivation and related work: Prior research has explored AR for repair tasks and multimodal large language models (MLLMs) for instruction generation. However, these approaches often rely on structured, pre-existing content like CAD models or manuals, which are not always available. This paper seeks to fill this gap by leveraging user-recorded disassembly videos to generate reassembly guidance.

Solution

• Proposed approach: RePairAR, an AR system that uses MLLMs to generate reassembly guidance directly from user-recorded egocentric disassembly videos, without requiring CAD models or manuals.
• Novelty:
  1. Development of an MLLM-driven pipeline to extract task structure and component information from disassembly videos.
  2. Mixed-media AR visualizations to provide in-situ reassembly guidance.
  3. User study demonstrating the system’s impact on reducing temporal demand and improving repair self-efficacy.
• Procedure and key techniques:
  1. Users record disassembly videos using an AR headset.
  2. The system processes the video through an MLLM pipeline, extracting step-part-relation structures.
  3. Reassembly guidance is generated by reversing the disassembly sequence and visualized through AR overlays, including task flow overviews, component identification, and alignment guidance.

Results

• Concrete findings:
  • Mean over Frames accuracy for step segmentation: 30.6% (range: 1.9%–68.0%).
  • Object extraction F1 score: 0.74 (range: 0.37–1.00).
  • RePairAR reduced perceived temporal demand compared to how-to videos (t = –2.76, p = .015).
  • Self-efficacy scores increased significantly after using RePairAR (Δ = 11.38, p < .001).
• Advantage over baselines:
  • RePairAR provided greater gains in self-efficacy compared to how-to videos (Δ = 2.63, p = 0.030).
  • Participants reported lower temporal demand and appreciated the system’s alignment with their pace.
• Experiments / evaluation:
  • Technical validation on 17 appliance videos with 182 components.
  • User study with 16 participants (mean repair skill: 2.06/5), comparing RePairAR with how-to videos in a within-subjects design.
  • Metrics: System Usability Scale (SUS = 63.91), NASA-TLX workload, and self-efficacy questionnaires.
• Limitations and future work:
  • Usability challenges due to AR headset limitations (e.g., visual blur, discomfort).
  • Imperfect step segmentation and object extraction accuracy.
  • Privacy concerns related to recording and processing user interactions.
  • Future work includes refining the pipeline, extending support to diagnostics, and integrating with other tools and platforms.

Summary

RePairAR is an AR system that uses MLLMs to transform user-recorded disassembly videos into interactive reassembly guidance, addressing the lack of accessible repair documentation. The system demonstrated reliable component extraction and improved repair self-efficacy in a user study, though challenges remain in usability and technical accuracy. By reducing temporal demand and aligning guidance with user pace, RePairAR makes repair tasks more approachable for novices. Future work aims to enhance the pipeline, expand support to other repair phases, and integrate with broader repair ecosystems.