I'm Always a Little Skeptical of It: Verification Practices of Blind Users When Working with Generative AI in Spreadsheets

Paper Title

"I'm Always a Little Skeptical of It: Verification Practices of Blind Users When Working with Generative AI in Spreadsheets"

Publication Info

• Topic area: Verification practices of blind users interacting with Generative AI in spreadsheet tasks.
• Keywords: Generative AI, screen readers, accessibility, blind users, spreadsheets, verification strategies, assistive technology, data analysis, AI hallucinations, error detection.

Background and Problem

• Problem / challenge: Blind users face significant challenges in verifying the accuracy of Generative AI (GenAI) outputs in spreadsheet tasks, particularly for visually intensive tasks such as chart generation and formatting. Existing tools often fail to provide accessible and reliable verification workflows.
• Significance: Accurate verification is critical in spreadsheet tasks for professional and educational contexts. Addressing these challenges can improve accessibility and independence for blind users while ensuring the reliability of AI-assisted workflows.
• Motivation and related work: Previous research has explored spreadsheet accessibility for blind users and the potential of GenAI tools to enhance productivity. However, there is limited understanding of how blind users verify GenAI outputs, particularly in accuracy-critical spreadsheet tasks involving numeric data and visual elements.

Solution

• Proposed approach: The study investigates the verification practices of blind users when working with GenAI in spreadsheets, focusing on error detection and response strategies.
• Novelty:
  1. Empirical characterization of verification methods employed by blind users in spreadsheet tasks.
  2. Analysis of error-response workflows reflecting a skeptical stance toward GenAI outputs.
  3. Design recommendations for improving GenAI-assisted spreadsheet tools with features like multi-model validation and enhanced explainability.
• Procedure and key techniques:
  • Conducted a remote study with 12 blind participants using screen readers.
  • Participants completed spreadsheet tasks involving data analysis and modification with GenAI assistance.
  • Verification strategies were analyzed through screen recordings and qualitative feedback.
  • Tasks included identifying trends, generating charts, applying formulas, and formatting spreadsheets.

Results

• Concrete findings:
  • Participants employed diverse verification methods, including manual checks (e.g., screen readers, Excel functions), same AI-assisted verification (e.g., follow-up questions, reasoning features), cross-AI validation, prior knowledge, and sighted assistance.
  • Errors were most frequent in visual tasks (15 of 18 errors) such as chart generation and formatting, with half of these errors going unnoticed.
  • Verification workflows were multi-step and complex, often involving iterative interactions with AI tools and external assistance.
• Advantage over baselines:
  • The study highlights unique verification challenges faced by blind users, which are qualitatively different from those of sighted users. It identifies strategies and gaps that can inform the design of more accessible GenAI tools.
• Experiments / evaluation:
  • Tasks were performed using real-world datasets (e.g., inflation data, student marksheets).
  • Participants used a variety of GenAI tools (e.g., ChatGPT, Copilot, Gemini) and screen readers (e.g., JAWS, NVDA).
  • Verification methods were analyzed across six tasks, with errors and workflows documented in detail.
• Limitations and future work:
  • Limited generalizability due to the small sample size and participants' intermediate spreadsheet proficiency.
  • Participants had limited prior experience with GenAI-assisted spreadsheet workflows.
  • Future research should include longitudinal studies and highly proficient spreadsheet users to explore evolving strategies and additional challenges.

Summary

This study explores how blind users verify the accuracy of Generative AI outputs in spreadsheet tasks, revealing diverse strategies such as manual checks, AI-assisted verification, and sighted assistance. Errors were frequent in visually intensive tasks, and verification workflows were complex and multi-step. The findings highlight the need for improved error-detection features, multi-model validation, and accessible reasoning capabilities in GenAI tools. These insights can guide the design of more reliable and inclusive AI-assisted spreadsheet workflows, addressing critical accessibility barriers for blind users in professional and educational contexts.