From Struggle to Success: Context-Aware Guidance for Screen Reader Users in Computer Use

Paper Title

From Struggle to Success: Context-Aware Guidance for Screen Reader Users in Computer Use

Publication Info

• Topic area: Assistive technologies for screen reader users in digital system navigation.
• Keywords: Screen reader users, accessibility, assistive technology, large language models, context-aware systems, task guidance, inclusive computing, user study, NVDA, adaptive support.

Background and Problem

• Problem / challenge: Screen reader (SR) users face significant challenges in navigating visually-oriented digital interfaces, requiring them to memorize shortcuts and interpret auditory output. Existing support systems, such as tutorials and human assistance, are either inaccessible or lack real-time availability. Current AI assistants fail to address SR-specific needs, relying on visual descriptions and requiring extensive user input.
• Significance: Addressing these challenges is critical for enabling equal access to education, employment, and social participation for visually impaired individuals.
• Motivation and related work: Prior research has explored accessible tutorials, human-powered Q&A services, and AI-based accessibility overlays, but these approaches are limited in scalability, real-time support, or SR-specific usability. Advances in Large Language Models (LLMs) offer potential for real-time, context-aware assistance, but existing implementations are not tailored to SR users.

Solution

• Proposed approach: AskEase, an LLM-powered, on-demand assistant designed to provide step-by-step, SR-friendly guidance for computer use by leveraging multiple sources of context.
• Novelty:
  1. Automatic collection and management of rich contextual information (e.g., screen states, screen reader traces, and software documentation).
  2. Context-aware, SR-specific guidance that aligns with user preferences and minimizes workflow disruption.
  3. Integration of adaptive support and seamless interaction features for efficient help-seeking.
• Procedure and key techniques:
  • Context engineering: Captures environment (e.g., screenshots, focus highlights), knowledge (e.g., software documentation via retrieval-augmented generation), and conversational context (e.g., chat history).
  • Interaction design: Features include Contextual Q&A, Adaptive Support, and Screen Description, all accessible via keyboard shortcuts.
  • Implementation: Developed as an NVDA add-on using OpenAI GPT-5 for reasoning and planning.

Results

• Concrete findings:
  • Achieved a 96.6% task success rate across 45 tasks from 12 applications, with an average latency of 10.06 seconds per query.
  • In a user study with 12 SR users, participants completed 1.5 tasks on average with AskEase, compared to 0.5 tasks using baseline tools.
  • AskEase reduced perceived workload, including physical demand (−1.25), effort (−1.42), and frustration (−1.58), while improving perceived performance (+1.50).
• Advantage over baselines:
  • Provided more accessible, step-by-step guidance compared to verbose or visually-oriented responses from baseline tools.
  • Minimized workflow disruption through seamless, in-place assistance.
• Experiments / evaluation:
  • Robustness tests across 12 applications and 45 tasks.
  • Within-subject user study with 12 participants, using Microsoft Word and Excel for task evaluation.
  • Metrics: task completion rates, NASA-TLX workload scores, and qualitative feedback.
• Limitations and future work:
  • Limited to NVDA on Windows; future work could extend to other platforms and assistive technologies.
  • Challenges with incomplete environmental context and hallucinations in guidance.
  • Need for personalization, proactive clarification mechanisms, and privacy-preserving designs.

Summary

AskEase is an LLM-powered, context-aware assistant designed to support screen reader users in navigating digital systems. By integrating multiple sources of context and providing SR-specific, step-by-step guidance, it significantly improves task completion rates and reduces perceived workload. A user study demonstrated its effectiveness in enhancing accessibility and usability for visually impaired users. Future work will focus on expanding platform support, improving environmental awareness, and addressing privacy concerns to further empower SR users in education, employment, and everyday computing.