Assistive-Technology Aided Manual Accessibility Testing in Mobile Apps, Powered by Record-and-Replay

Title of the Paper

Assistive-Technology Aided Manual Accessibility Testing in Mobile Apps, Powered by Record-and-Replay

Paper Information

• Field of Study: Accessibility testing in mobile apps supported by assistive technology
• Keywords: Android, Accessibility, Software Testing, TalkBack, Assistive Technology

Research Background and Problem

• Problem or Challenge:
  • Mobile app developers often test touch gesture functionality but neglect accessibility evaluations using assistive technologies like screen readers.
  • Automated accessibility testing tools generally do not account for assistive services and fail to detect many complex issues, such as non-focusable screen elements or interaction sequence problems.
• Significance: Approximately 15% of the global population has disabilities, making it impossible for these users to interact using traditional touch gestures. Legal and social responsibilities require developers to design applications accessible to everyone.
• Research Motivation and Related Work:
  • While user studies are the most reliable way to evaluate accessibility, finding suitable users with disabilities for evaluation is time-consuming and costly.
  • Existing tools like Latte and ATARI have limitations in accessibility testing, struggling to cover different types of interactions and complex interaction scenarios.

Solution

• Proposed Approach:
  • The authors developed a new semi-automated accessibility analysis technique called “A11yPuppetry,” which combines Record-and-Replay (RaR) technology. It records developers’ touch gestures and replays these interactions using a screen reader to generate visual reports.
• Innovations:
  • Converts recorded touch actions into corresponding screen reader behaviors, evaluated through multiple navigation modes.
  • Generates comprehensive reports for each interaction, including unfocused elements and invalid operations.
  • Provides various user studies and experiments to validate the effectiveness of the approach.
• Implementation Steps and Techniques:
  1. Recording Phase: Uses a transparent overlay to record users’ touch gestures while capturing the UI hierarchy and screenshots.
  2. Action Translation: Converts recorded gestures into screen reader navigation and interaction actions.
  3. Replay Phase: Replays interactions on different devices using a screen reader and various navigation modes, generating detailed execution reports.
  4. Report Generation: Aggregates information from the execution process to create a summary report of accessibility issues, presented to developers via a web interface.

Research Outcomes

• Specific Results:
  • A11yPuppetry was used to detect issues such as UI elements that blind users could not focus on, invalid interaction operations, and missing clear text descriptions.
  • Compared to existing solutions, A11yPuppetry covered more types of issues and provided more detailed report information.
  • User studies demonstrated its significant advantages in identifying critical user experience issues.
• Advantages:
  • Capable of detecting assistive technology-related issues in dynamic interactions, which existing tools cannot achieve.
  • Provides visualized reports and user behavior simulations, enabling developers to manually analyze and resolve potential issues.
  • Does not require developers to have advanced knowledge of accessibility technologies to operate.
• Experimental or Evaluation Results:
  • Experiments conducted on five Android applications showed that it effectively helped developers identify issues confirmed by blind users.
  • Automatically detected over 70% of user-confirmed issues, significantly outperforming existing tools like Latte and Accessibility Scanner.
• Limitations and Future Directions:
  • Not suitable for detecting semantic understanding issues, such as unclear text descriptions or contextual design problems.
  • Certain technical limitations (e.g., TalkBack freezing issues and dynamic ad displays) may lead to false reports.
  • Future directions include supporting more assistive services, platform expansion, and using it as an educational tool to help developers understand accessibility design standards.

This paper systematically addresses the tooling challenges of accessibility testing for mobile applications, enabling developers to better meet the needs of users with disabilities. Through experimental validation, it demonstrates potential for industrial application and provides a solid foundation for future research in accessibility testing.