Bridging the Gap between Automated Intervention and Actual User Experience: A Mixed-Methods Study on Mobile Accessibility Issues for Screen Reader Users

Paper Title

Bridging the Gap between Automated Intervention and Actual User Experience: A Mixed-Methods Study on Mobile Accessibility Issues for Screen Reader Users

Publication Info

• Topic area: Accessibility in mobile applications for screen reader users
• Keywords: Mobile accessibility, screen readers, automated interventions, user experience, systematic literature review, user study, accessibility guidelines, WCAG, accessibility testing, blind users

Background and Problem

• Problem / challenge: Existing automated tools for detecting and fixing mobile accessibility issues are inconsistent in addressing real user experiences. They often fail to detect subjective and feedback-related issues, and their coverage of accessibility guidelines is limited.
• Significance: With 2.2 billion people experiencing visual disabilities globally, improving mobile accessibility is crucial for equitable access to technology and services.
• Motivation and related work: Prior studies have developed automated tools and conducted systematic reviews, but these tools cover only a fraction of accessibility issues and lack integration with user feedback. This paper seeks to bridge the gap by combining computational methods with insights from screen reader users.

Solution

• Proposed approach: Development of a user-aware categorization of mobile accessibility issues, called Mobile Content Accessibility Guidelines (MCAG), integrating findings from a systematic literature review (SLR) and user studies.
• Novelty:
  1. Conducted a systematic literature review of 31 papers on automated interventions for mobile accessibility.
  2. Performed 20 user studies with blind participants to analyze real-world accessibility issues in four Android apps.
  3. Synthesized findings into MCAG, a guideline based on WCAG principles tailored for screen reader users.
  4. Identified new issue types and a feedback-related category not previously addressed in automated tools.
• Procedure and key techniques:
  1. Conducted an SLR across five databases, identifying 31 relevant papers on automated interventions.
  2. Categorized issues into four types: labeling, navigation, activation, and dynamic change.
  3. Conducted user studies on four Android apps with 20 blind participants, analyzing their experiences with accessibility issues.
  4. Synthesized computational and experiential findings into MCAG, structured around WCAG principles.

Results

• Concrete findings:
  • Identified 22 issue types across four categories: labeling, navigation, activation, and dynamic change.
  • Discovered a new category of feedback-related issues, including "no action feedback," "inadequate instruction," and "inadequate progress indication."
  • Found that subjective issues (e.g., intuitive navigation, label quality) and feedback-related issues are the most impactful but least addressed by automated tools.
• Advantage over baselines:
  • MCAG integrates computational and user-centered perspectives, addressing gaps in prior automated tools.
  • Highlights the limitations of programmatic heuristics and the potential of high-fidelity automation and LLMs for subjective issues.
• Experiments / evaluation:
  • SLR: Analyzed 31 papers, categorizing automated interventions into four techniques (automated crawlers, automation support, label generators, UI annotators).
  • User study: Conducted 20 tests on four Android apps, identifying 6 issue categories and 37 unique issue instances.
  • Evaluated the severity of issues, with activation and feedback-related issues being the most impactful.
• Limitations and future work:
  • Geographic scope limited to North America; apps tested only on Android.
  • Not all accessibility issue types were covered.
  • Future work includes expanding MCAG with more user studies, addressing feedback-related issues, and leveraging LLMs for subjective challenges.

Summary

This paper addresses the gap between automated accessibility tools and real user experiences for screen reader users on mobile apps. Through a systematic literature review and 20 user studies, the authors identified 22 issue types and introduced a new category of feedback-related issues. The findings were synthesized into Mobile Content Accessibility Guidelines (MCAG), structured around WCAG principles. MCAG provides a user-centered framework for addressing accessibility issues, emphasizing the need for high-fidelity automation and solutions for subjective and feedback-related challenges. The study highlights the importance of integrating computational and experiential insights to improve mobile accessibility.