Exploring Mobile Device Accessibility: Challenges, Insights, and Recommendations for Evaluation Methodologies

Title of the Paper

Exploring Accessibility of Mobile Devices: Challenges, Insights, and Recommendations for Improving Evaluation Methods

Bibliographic Information

• Subject Area: Human-Computer Interaction, Accessibility Evaluation for Mobile Devices
• Keywords: Accessibility, Mobile App Accessibility Evaluation, Accessibility Evaluation Methods, User Testing, Manual Evaluation, Mobile Applications

Research Background and Issues

• Mobile devices have become an indispensable part of daily life, but their unique features, such as limited screen size, distinct interaction methods, diverse usage environments, and platform inconsistencies, pose challenges for accessibility evaluation.
• The authors emphasize that while web accessibility evaluation methods have been extensively studied and standardized, there are still many open questions regarding evaluation methods for mobile applications, and authoritative guidelines and standards specifically for mobile devices are lacking.
• The importance of accessibility evaluation lies in its ability not only to identify barriers but also to promote product or service improvement and monitor accessibility trends. However, existing methods show significant shortcomings in terms of comparability of results and effectiveness in monitoring.
• Research Motivation: The authors aim to investigate the current state of mobile app accessibility evaluation methods, identify challenges and directions for improvement, and propose effective recommendations for enhancement.

Solution

• Proposed Approach: The authors adopted a multi-method research approach, conducting the following four studies: analysis of accessibility monitoring reports from EU member states; semi-structured interviews with accessibility evaluation experts; manual evaluation of four mobile applications; and usability testing with users with disabilities.
• Innovations:
  • Synthesized perspectives from literature, practical experience, and user feedback to form a more comprehensive analysis of current evaluation methods.
  • Proposed actionable recommendations to improve evaluation methods for mobile applications, including optimizing automated tools, refining sample selection steps, and enhancing the execution standards of user testing.
• Implementation Steps/Techniques:
  1. Monitoring Report Analysis:
     • Used content analysis to extract data from EU member state reports, focusing on evaluation methods, the number of applications involved, and challenges or observations highlighted in the reports.
  2. Expert Interviews:
     • Engaged with five accessibility evaluation experts to understand their evaluation methods, tools used, and major difficulties encountered.
     • Applied thematic analysis to summarize key challenges: lack of mobile-specific guidelines, limitations of automated tools, and difficulties in communicating findings to developers.
  3. Manual Evaluation:
     • Selected eight mobile applications and adjusted existing WCAG-EM and EN 301 549 standards to manually evaluate success criteria.
  4. User Testing:
     • Recruited six users with disabilities and designed tests involving diverse interaction tasks. By observing actual user behavior and combining results with manual testing, the study identified potential issues that other methods might overlook.

Research Findings

• Specific Findings:
  • Identified significant challenges in existing evaluation methods: lack of authoritative guidance for mobile devices, limited coverage by automated tools, and insufficient standardization of manual and user testing.
  • Accessibility issues showed consistency across different systems (e.g., iOS and Android). Common problems included: elements not recognized as clickable by screen readers, insufficient text and image contrast, and difficulties in adjusting text size or line spacing.
• Comparison with Existing Solutions:
  • The proposed recommendations emphasize the importance of standardization and user involvement, addressing gaps in the interpretation of specific success criteria in existing methods, while also increasing attention to AAA-level standards, such as target size (SC 2.5.5).
• Experimental and Evaluation Results:
  • User testing revealed barriers that manual testing failed to detect. For instance, AAA-level standards, often overlooked due to being non-mandatory, are crucial for mobile device interactions.
  • Highlighted the subjectivity and operational complexity in current mobile app evaluations, which complicates consistency and reliability. The study suggests using clearer, standardized guidelines to optimize the evaluation process.
• Limitations and Future Directions:
  • Limitations:
    • Current evaluation tools and techniques do not fully address accessibility issues in mobile applications.
    • The sample size for user testing was relatively small, requiring more participants and broader testing to further validate findings.
  • Future Directions:
    • Develop more efficient automated tools to enhance detection capabilities across multiple mobile platforms and more success criteria.
    • Revise existing accessibility standards and methods to better accommodate the specific needs of mobile devices and promote the standardization of user participation methods and processes.