Toward Independent Online Shopping of the Visually Impaired Through Voice-based Computer-Using Agent

Paper Title

Toward Independent Online Shopping of the Visually Impaired Through Voice-based Computer-Using Agent

Publication Info

• Topic area: Accessibility in online shopping for visually impaired users via voice-based Computer-Using Agents (CUA).
• Keywords: Visually impaired users, online shopping, voice-based interaction, Computer-Using Agent (CUA), Large Multimodal Models (LMM), accessibility, Wizard-of-Oz study, cognitive load, disability-centered design, assistive technology.

Background and Problem

• Problem / challenge: Visually impaired users face significant barriers in online shopping due to reliance on visual content, inadequate alternative text, and complex graphical user interfaces (GUIs). Existing assistive technologies do not fully support independent navigation and decision-making in such environments.
• Significance: Online shopping offers visually impaired users a more accessible alternative to physical stores, but current limitations hinder their autonomy and increase cognitive and psychological burdens.
• Motivation and related work: Prior research has focused on assistive technologies like screen readers, OCR, and voice assistants, which improve functional accessibility but fail to address complex GUI navigation and dynamic interactions. Recent advances in Large Multimodal Models (LMM) and Computer-Using Agents (CUA) offer new opportunities for independent online shopping, but their application in this context remains underexplored.

Solution

• Proposed approach: A Semi-Automatic Wizard-of-Oz study using a voice-based CUA system (Operator) to explore how visually impaired users navigate online shopping tasks, followed by debriefing interviews to gather qualitative insights.
• Novelty:
  1. First empirical study on LMM-based CUA for visually impaired users in online shopping.
  2. Identification of user strategies and needs, including trend-seeking, aesthetic impressions, and cognitive load management.
  3. Design implications for disability-centered online shopping environments.
• Procedure and key techniques:
  1. Semi-Automatic Wizard-of-Oz setup where a researcher mediates voice-to-text input for the CUA.
  2. Participants interact solely through voice to explore, compare, and select products.
  3. Data collection through communication logs and debriefing interviews, followed by thematic analysis.

Results

• Concrete findings:
  • Users sought trend information, aesthetic impressions, and detailed color descriptions to compensate for the absence of visual cues.
  • Preferred concise summaries, limited options (3–5 items), and proactive guidance to manage cognitive load.
  • Relied heavily on social proof (e.g., reviews, popularity) to reduce uncertainty.
  • Valued double-checking of selections to prevent errors.
• Advantage over baselines: Enabled independent navigation and decision-making in complex online shopping tasks, moving beyond traditional assistive technologies that require external support.
• Experiments / evaluation:
  • 12 participants with acquired visual impairments completed 2-hour 20-minute sessions, including a 50-minute shopping task and 1-hour debriefing interview.
  • Data included 226 communication logs and 13 hours of audio-recorded interviews, analyzed with thematic coding (Fleiss’s Kappa = 0.865).
• Limitations and future work:
  • Limited to participants with acquired visual impairments and prior online shopping experience.
  • Conducted in a controlled laboratory setting, not reflecting real-world environmental factors.
  • Future work should include diverse impairment types, varying digital literacy levels, and real-world contexts.

Summary

This study explored how visually impaired users navigate online shopping through voice-based interaction with a Computer-Using Agent (CUA). Findings revealed user strategies to compensate for visual and social cue absence, manage cognitive load, and reduce uncertainty. The research proposed design implications for inclusive, disability-centered online shopping environments, emphasizing trend information, aesthetic impressions, concise summaries, and proactive guidance. By enabling independent navigation and decision-making, the study highlights the potential of LMM-based CUAs to transform accessibility in visually intensive domains. Future work should address broader user demographics and real-world conditions.