The Sky is the Limit: Understanding How Generative AI can Enhance Screen Reader Users' Experience with Productivity Applications

Research Background and Issues

• Issues and Challenges:

  • Blind users face significant accessibility and usability challenges when using productivity applications such as word processors, spreadsheets, and presentation tools.
  • Screen reader (SR) users often struggle to manage visual information (e.g., charts and graphics), obtain content overviews, locate specific information within documents, and collaborate with others.
  • Numerous functionalities, such as navigation, control access, and document formatting, present accessibility barriers for blind users, resulting in lower efficiency in completing productivity tasks compared to sighted colleagues.

• Importance:

  • These tools have become indispensable in modern life across work, education, and personal contexts. Improving their accessibility is crucial for equal employment opportunities, enhanced education levels, and greater independence.
  • With advancements in Generative AI (GenAI) technology, there is potential to provide blind users with natural language interaction and contextual task understanding to address these challenges.

• Research Motivation and Related Work:

  • While existing research has explored how blind users utilize virtual assistants (e.g., Siri, Alexa) for simple tasks, the focus has been on descriptive tasks and basic operations, with limited investigation into productivity-related improvements.
  • Although some studies have examined specific issues in productivity tools, there is still insufficient understanding of the comprehensive challenges faced by blind users in these applications and how to prioritize solutions.

Solution

• Methods and Innovations:

  • Conducted a survey of 99 blind users who use screen readers and in-depth interviews with 16 users to explore how Generative AI can enhance user experience.
  • Investigated blind users' attitudes toward Generative AI, potential use cases (e.g., accessing graphical content, extracting information, providing task suggestions), and analyzed issues related to trust, privacy, and workflow integration introduced by Generative AI.
  • Proposed design principles for Generative AI to complement existing SR workflows rather than replace them.

• Key Technologies:

  1. Task Assistance: Generate natural language descriptions of content (e.g., chart explanations) or directly execute specific tasks upon request (e.g., spreadsheet sorting).
  2. Information Extraction: Support users in summarizing information from lengthy documents and extracting structured data.
  3. Personalized Integration: Explore methods to integrate Generative AI with screen readers, including system-level or in-app adaptations.

• Implementation Steps:

  1. Investigate and identify the primary pain points of blind users and the most pressing issues in existing productivity applications.
  2. Conduct user testing and gather feedback on potential functionalities of Generative AI to define how it can support existing user workflows.
  3. Explore a mixed-initiative approach to meet user needs, allowing users to decide when to intervene and when to delegate tasks to AI.

Research Findings

• Key Conclusions:

  • Generative AI is perceived as a powerful assistive tool for blind users, enhancing task efficiency and enabling them to complete complex tasks independently without relying on others.
  • Users believe AI can significantly improve accessibility to content, editing and formatting, navigation efficiency, and understanding changes during collaboration.
  • AI can also support users in learning advanced SR operations through keyboard shortcuts, fostering greater independence.

• Comparison with Existing Solutions:

  • Compared to traditional SR and other assistive tools, Generative AI further enhances blind users' productivity and information acquisition capabilities. It is particularly advantageous in handling inaccessible content (e.g., converting it into more usable formats).

• Experimental or Evaluation Results:

  • Survey results indicate widespread interest and positive attitudes among respondents toward the application of Generative AI, especially in reducing task completion time and enabling blind users to operate productivity applications more independently.
  • Interviews further clarified users' diverse needs for interaction styles with Generative AI (e.g., combining voice and text input).

• Limitations and Future Directions:

  1. Limitations:
     • Blind users' awareness and experience with Generative AI are still in the early stages, and future developments may evolve as the technology matures and application scenarios expand.
     • Survey participants were primarily from English-speaking countries with relatively high educational levels, which may not fully represent the needs of the global blind population.
  2. Future Directions:
     • Further validate the long-term impact of Generative AI on blind users, particularly in professional environments.
     • Incorporate feedback from blind users and address the needs of non-English-speaking regions to improve AI model training data and design specialized personalized features.
     • Develop mechanisms for generating highly interpretable AI task outputs and tools for verifying information accuracy to enhance user trust.

This research provides valuable guidance for productivity software and assistive technology designers to develop more inclusive solutions, fundamentally improving the digital participation experience for blind users.