Not Seeing the Whole Picture: Challenges and Opportunities in Using AI for Co-Making Physical, DIY-AT for People with Visual Impairments

Paper Title

Not Seeing the Whole Picture: Challenges and Opportunities in Using AI for Co-Making Physical, DIY-AT for People with Visual Impairments

Publication Info

• Topic area: AI-assisted co-making of physical DIY assistive technologies for people with visual impairments.
• Keywords: Assistive technology, DIY-AT, visual impairments, large language models, tangible toolkits, AI-assisted design, accessibility, co-making, multimodal AI, spatial guidance.

Background and Problem

• Problem / challenge: Existing assistive technologies (AT) are standardized and fail to address the diverse, context-specific needs of people with visual impairments (PVI). DIY-AT offers customization but remains inaccessible due to technical barriers, lack of confidence, and inaccessible tools. Prior toolkits are limited in scope and require technical expertise.
• Significance: Addressing these gaps can empower PVI to create tailored solutions, reduce AT abandonment rates, and foster independence and creativity.
• Motivation and related work: Previous research has explored DIY-AT and AI-assisted tools but has focused on co-design with experts or software-based solutions. There is a lack of work on how AI can support PVI in creating physical, tangible DIY-AT solutions independently.

Solution

• Proposed approach: Development of an AI-assisted tangible DIY-AT toolkit, including an LLM-based AI assistant (A11yMaker AI) and tangible sensing and feedback modules, to support PVI in brainstorming, configuring, and assembling custom AT solutions.
• Novelty:
  1. First integration of an LLM-based AI assistant in a tangible toolkit for PVI to create physical DIY-AT solutions.
  2. Empirical insights from a study with nine PVI participants on their strategies, challenges, and use cases.
  3. Design recommendations for improving AI-assisted DIY-ATs, including spatial/visual support and error mitigation strategies.
• Procedure and key techniques:
  • The toolkit includes six sensing modules (e.g., Camera, Distance, Motion) and four feedback modules (e.g., Vibration, Sound).
  • A11yMaker AI abstracts technical details, supports natural language interaction, and provides guidance on module selection, configuration, and debugging.
  • A user study with nine PVI participants evaluated the toolkit’s usability and effectiveness in creating 14 custom DIY-AT solutions.

Results

• Concrete findings:
  • Participants created 14 DIY-AT solutions addressing navigation, object search, and environmental awareness.
  • The AI assistant acted as a tutor, toolkit expert, and brainstorming partner, lowering technical barriers.
  • Challenges included AI hallucinations, lack of proactive clarification, and insufficient spatial/visual guidance.
• Advantage over baselines:
  • The toolkit enabled PVI to independently brainstorm and implement solutions, which was previously limited to co-design with experts or software-based approaches.
• Experiments / evaluation:
  • A 150-minute lab-based study with nine PVI participants (average age 49.6 years) using think-aloud protocols, interviews, and task-based evaluations.
  • Participants used the toolkit to address personal accessibility challenges, with researchers observing interactions and analyzing transcripts and logs.
• Limitations and future work:
  • Lab-based setting limited real-world applicability; participants suggested testing the toolkit in home environments.
  • Toolkit functionality was constrained by predefined event types; future work could explore AI-driven code generation and integration with IoT devices.
  • The AI assistant lacked multimodal input for spatial awareness; future designs could incorporate AR or camera-based feedback.

Summary

This paper introduces an AI-assisted tangible DIY-AT toolkit that empowers people with visual impairments to create custom assistive technologies. Through a study with nine participants, the toolkit demonstrated its potential to lower technical barriers and foster independence, enabling the creation of 14 diverse solutions. However, challenges such as AI hallucinations, lack of spatial guidance, and limited toolkit capabilities highlight areas for improvement. Future work should focus on integrating multimodal AI for spatial awareness, expanding toolkit functionality, and testing in real-world settings. This research contributes to making DIY-AT more accessible and adaptable for PVI.