TaskAudit: Detecting Functiona11ity Errors in Mobile Apps via Agentic Task Execution

Paper Title

TaskAudit: Detecting Functiona11ity Errors in Mobile Apps via Agentic Task Execution

Publication Info

• Topic area: Automated accessibility evaluation for mobile apps
• Keywords: Accessibility, mobile apps, functiona11ity errors, screen readers, task execution, multi-agent systems, large language models, WCAG, automated testing, user interaction

Background and Problem

• Problem / challenge: Existing accessibility checkers primarily focus on static or heuristic assessments and fail to detect "functiona11ity errors," which are accessibility barriers that manifest only during user interaction.
• Significance: Functiona11ity errors, such as label-functionality mismatches and inappropriate feedback, significantly impact the usability of mobile apps for screen reader users, yet remain undetected by current tools.
• Motivation and related work: Prior tools like Groundhog and AXNav have attempted to address functionality issues but are limited to heuristic-based or mechanical crawling approaches. These methods often generate false positives and fail to capture nuanced interaction-dependent errors, leaving a gap for systems that can semantically interpret interface behavior.

Solution

• Proposed approach: TaskAudit, a system that detects functiona11ity errors in mobile apps through agentic task execution using a screen reader proxy.
• Novelty:
  1. Introduction of the concept of "functiona11ity errors" and their categorization into five types: Locatability, Actionability, Label, Feedback, and Navigation errors.
  2. Development of a multi-agent system that simulates screen reader interactions to detect errors.
  3. Integration of large language models (LLMs) to generate tasks, execute them, and analyze interaction traces for accessibility issues.
  4. Demonstration of TaskAudit’s ability to detect errors missed by existing tools, such as label-functionality mismatches and cluttered navigation.
• Procedure and key techniques:
  1. Task Generator: Uses OCR and LLMs to identify interactive elements from app screenshots and generate task specifications.
  2. Task Executor: Employs a multi-agent system to perform tasks via a screen reader proxy, simulating user interactions.
  3. Accessibility Analyzer: Analyzes execution traces to detect and report functiona11ity errors using a two-stage evaluation process.

Results

• Concrete findings:
  • TaskAudit detected 48 out of 78 functiona11ity errors across 54 app screens, achieving a precision of 66.2% and recall of 61.5%.
  • The Task Generator identified 69.4% of actionable elements in a dataset, with 93.4% of generated captions consistent with human annotations.
  • The Task Executor achieved a 96.0% success rate in executing tasks on accessible screens.
• Advantage over baselines:
  • TaskAudit outperformed Groundhog in precision (66.2% vs. 14.2%) and detected additional error categories (Label, Feedback, and Navigation) that Groundhog could not.
  • Compared to Accessibility Scanner and ScreenAudit, TaskAudit detected significantly more functiona11ity errors.
• Experiments / evaluation:
  • Task identification was evaluated against human-annotated datasets (e.g., RICO).
  • Task execution was tested on 299 tasks from 36 screens of accessible apps.
  • Functiona11ity error detection was benchmarked against three state-of-the-art tools (Groundhog, Accessibility Scanner, ScreenAudit) on 54 screens with 78 known errors.
• Limitations and future work:
  • Limitations include inability to perform touch exploration, reliance on LLMs for task generation, and high computational costs.
  • Future work could explore hybrid approaches combining rule-based and agentic methods, fine-tuning smaller models for cost efficiency, and adapting the system for other interaction modalities like voice-based interfaces.

Summary

TaskAudit introduces a novel approach to detecting functiona11ity errors in mobile apps by simulating screen reader interactions through multi-agent task execution. It addresses limitations of existing tools by leveraging LLMs for semantic analysis and interaction trace evaluation, achieving higher precision and broader error coverage. The system detected 48 out of 78 errors across five categories, including errors missed by prior methods. While computationally intensive, TaskAudit demonstrates the potential for integration into development workflows to enhance accessibility testing and improve mobile app inclusivity. Future work aims to optimize performance and expand applicability to other assistive technologies.