A11yExtensions: Accessibility Extensions to Augment Mobile AI Assistive Technology In-Situ

Paper Title

A11yExtensions: Accessibility Extensions to Augment Mobile AI Assistive Technology In-Situ

Publication Info

• Topic area: Enhancing mobile AI assistive technologies for blind users through in-situ extensions.
• Keywords: Accessibility, assistive technology, AI, mobile automation, co-design, extensions, in-situ interventions, blind users, usability, customization.

Background and Problem

• Problem / challenge: Existing mobile AI assistive technologies are limited in functionality, lack customization, and do not address specific user needs effectively. Research advancements in accessibility often remain siloed from commercial applications, leaving gaps in usability and functionality.
• Significance: Addressing these gaps can improve the daily lives of blind users by making assistive technologies more adaptable, efficient, and reliable for diverse tasks.
• Motivation and related work: Previous research has developed features like blind photography guidance and AI verification, but these are not integrated into widely used commercial apps. Extensions and automation tools in HCI have shown promise for customizing existing systems, but their potential in accessibility remains underexplored. This work seeks to bridge these gaps by introducing a framework for integrating new features into existing workflows.

Solution

• Proposed approach: A11yExtensions, a set of in-situ add-ons leveraging mobile automation tools (e.g., iOS Shortcuts) to augment existing assistive technologies with new features.
• Novelty:
  1. Introduction of a design space for mobile accessibility extensions.
  2. Implementation of three co-designed add-ons: Camera Aiming, Cross Checking, and Image Quality.
  3. Findings from a longitudinal co-design process with blind accessibility professionals.
  4. Demonstration of mobile automation as a new paradigm for assistive technology customization.
• Procedure and key techniques:
  • Co-design process with two blind accessibility consultants over four sessions.
  • Development of three add-ons using iOS Shortcuts and App Intents:
    • Camera Aiming: Provides verbal guidance for centering objects in the camera frame.
    • Cross Checking: Verifies AI-generated results by comparing outputs from multiple models.
    • Image Quality: Identifies issues like blurriness or poor lighting in photos.
  • Evaluation of add-ons through user testing and iterative refinement.

Results

• Concrete findings:
  • Add-ons improved task efficiency and usability, e.g., faster photo-taking and error verification.
  • Co-designers appreciated the flexibility and customization offered by the add-ons.
  • Privacy and onboarding challenges were identified, requiring further refinement.
• Advantage over baselines:
  • Enabled features like multi-model verification and camera aiming, which are unavailable in current commercial apps.
  • Reduced manual effort for tasks like cross-checking AI outputs.
• Experiments / evaluation:
  • Conducted four co-design sessions with two blind professionals.
  • Tested add-ons individually and in combined workflows (e.g., preparing a social media post).
  • Evaluated usability, efficiency, and user preferences.
• Limitations and future work:
  • Limited to two co-designers; broader validation is needed.
  • Setup and onboarding remain complex for novice users.
  • Current implementation is constrained by iOS platform limitations.
  • Future work includes expanding to other platforms, improving onboarding, and exploring end-user programming for customization.

Summary

A11yExtensions introduces a novel approach to augmenting mobile AI assistive technologies with in-situ add-ons, addressing gaps in usability, accuracy, and customization. Developed through a co-design process with blind accessibility professionals, the system includes three implemented features: Camera Aiming, Cross Checking, and Image Quality. These add-ons demonstrated improved efficiency and adaptability in real-world tasks, though challenges remain in onboarding and scalability. This work highlights the potential of mobile automation for enhancing accessibility and serves as a foundation for future research and development in assistive technology.