ScreenAudit: Detecting Screen Reader Accessibility Errors in Mobile Apps Using Large Language Models

Research Background and Issues

• What problems or challenges did the authors identify?
  The authors pointed out that many mobile applications fail to meet accessibility standards, making it difficult for individuals with disabilities to use their functionalities. This issue has persisted without significant improvement, even though developer tools like Google's Accessibility Scanner are widely adopted. These automated rule-checking tools still have limited coverage of accessibility errors and fail to identify critical screen reader issues.

• Why is this issue important?
  The inaccessibility of mobile applications can prevent individuals with disabilities from benefiting from digital products, exacerbating the digital divide. Additionally, the lack of clear feedback tools for developers hinders the widespread implementation of accessibility design.

• Research Motivation and Related Work
  Existing automated rule-checking tools can only detect a limited range of errors, such as color contrast and missing labels, while neglecting the validation of dynamic content and the quality of screen reader responses. The authors aim to leverage the latest large language model (LLM) technology to expand the scope of automated detection, providing developers with more meaningful accessibility feedback.

Solution

• What methods or solutions did the authors propose?
  The authors developed a system called "ScreenAudit," which uses LLMs (e.g., GPT-4) to automatically detect screen reader accessibility issues, collect TalkBack's voice output, and generate detailed accessibility reports.

• What are the innovative aspects of this solution?

  • Expanded error detection scope: Compared to existing tools, ScreenAudit can detect more complex and harder-to-identify screen reader accessibility errors.
  • Integration of screen reader output: The system directly analyzes screen reader feedback, simulating real user interaction experiences.
  • Detailed explanations and improvement suggestions: By analyzing TalkBack output, the LLM generates actionable feedback and repair recommendations.

• What are the implementation steps and key technologies used?

  1. Recorder Module: Traverses screen elements, captures TalkBack's voice output, and takes dynamic view snapshots.
  2. Auditor Module: Uses GPT-4 with customized prompts to analyze TalkBack output and detect potential accessibility issues.
  3. Report Viewer Module: Displays detailed reports to developers via a web interface, including issue descriptions, explanations, and repair suggestions.

Research Outcomes

• What specific results were achieved?

  • ScreenAudit achieved a screen accessibility issue detection coverage of 69.2%, significantly higher than Google's Accessibility Scanner (31.3%).
  • Accuracy reached 71.3%, and the system identified additional issues initially overlooked by experts.

• What advantages does it have compared to existing solutions?

  • Broader coverage of screen reader interactions, capable of detecting complex issues such as label quality problems and structural grouping errors that existing tools miss.
  • Provides actionable suggestions to help developers efficiently address issues.
  • Offers intuitive report visualization with higher integration potential with development tools.

• What were the experimental or evaluation results?

  • Experiments demonstrated that ScreenAudit generates high-quality reports strongly correlated with expert evaluations. Experts found the tool effective and user-friendly.
  • Among different LLM prompt structures, the combination of general accessibility guidance with contextual prompts performed best, improving detection accuracy (F1 score of 0.704).

• Limitations and Future Directions

  1. Unable to detect elements not focused by the screen reader: Future work will explore integrating screenshot parsing technologies to expand detection scope.
  2. Need for better contextual understanding: Future research can investigate combining existing crawling technologies to enhance awareness of interaction sequences and content organization.
  3. Lack of functional testing support: Further development could introduce agent-driven interaction testing mechanisms to simulate more complex user scenarios.
  4. No direct code analysis tools: Future iterations could incorporate modules for code-level issue identification and refactoring suggestions, along with support for post-repair re-evaluation.

Conclusion

ScreenAudit demonstrates significant potential for mobile application accessibility evaluation by combining screen reader output with large language model technology. It not only expands the error detection scope of current automated tools but also provides developers with detailed explanations and improvement suggestions, effectively promoting the prioritization of accessibility design in the development process. This work lays the foundation for developing more comprehensive, user-simulation-driven accessibility evaluation tools in the future and points to new directions for addressing accessibility challenges.