Deaf and Hard of Hearing Access to Intelligent Personal Assistants: Comparison of Voice-Based Options with an LLM-Powered Touch Interface

Paper Title

Deaf and Hard of Hearing Access to Intelligent Personal Assistants: Comparison of Voice-Based Options with an LLM-Powered Touch Interface

Publication Info

• Topic area: Accessibility of intelligent personal assistants for Deaf and Hard of Hearing individuals.
• Keywords: Deaf speech, dysarthric speech, intelligent personal assistants, accessibility, large language models, touch interface, Wizard-of-Oz, automatic speech recognition, usability, multimodal interaction.

Background and Problem

• Problem / challenge: Intelligent personal assistants (IPAs) struggle to recognize deaf-accented speech, limiting accessibility for Deaf and Hard of Hearing (DHH) individuals who use spoken English. Existing alternatives, such as touch interfaces, are less efficient and lack contextual awareness.
• Significance: Improving IPA accessibility for DHH individuals is crucial for equitable access to technology, enabling independent interaction with smart environments and daily tasks.
• Motivation and related work: Prior studies have explored sign language input and touch-based interfaces but have not adequately addressed the usability of spoken English for DHH users. Current ASR systems fail to accommodate non-standard speech patterns, and touch interfaces lack efficiency and adaptability. This paper investigates viable alternatives, including a novel LLM-powered touch interface.

Solution

• Proposed approach: A mixed-method study comparing three input methods for IPAs: Natural Deaf Speech, Wizard-of-Oz Facilitated English speech, and an LLM-assisted touch interface.
• Novelty:
  1. Evaluation of Alexa’s ability to understand natural deaf-accented speech without prior training.
  2. Introduction of an LLM-powered touch interface that leverages situational context and user history for improved usability.
  3. Comparison of usability metrics across voice-based and touch-based input methods for DHH users.
  4. Identification of user preferences and limitations for multimodal IPA interaction.
• Procedure and key techniques:
  • Participants interacted with IPAs using three methods: direct speech, facilitated speech (via human re-speaking), and touch interface.
  • Quantitative measures included System Usability Scale (SUS), Adjective Scale, Net Promoter Score (NPS), and Word Error Rate (WER).
  • Qualitative insights were gathered through semi-structured interviews.
  • The LLM-powered touch interface used GPT-4o to generate context-aware task suggestions.

Results

• Concrete findings:
  • SUS scores: Touch (63.5), Facilitated English (62.5), Natural Deaf Speech (59.6); differences were not statistically significant.
  • Adjective Scale: Touch (5.15), Natural Deaf Speech (5.0), Facilitated English (4.8); all methods rated between "ok" and "good."
  • NPS: Observed scores were -10 (Touch), -5 (Natural Deaf Speech), and -40 (Facilitated English).
  • WER: Half of participants had no recognition errors; others ranged from 0.61% to 30.91%, with two participants at 100%.
• Advantage over baselines:
  • LLM-assisted touch interface showed higher SUS and Adjective Scale scores compared to prior studies on Tap-to-Alexa and smart home apps.
  • Natural Deaf Speech demonstrated surprising recognition accuracy, motivating further exploration of native IPA support for deaf-accented speech.
• Experiments / evaluation:
  • 20 participants aged 20–75, recruited from DHH communities, tested three input methods in a simulated smart home environment.
  • Tasks included controlling lights, setting timers, and querying information.
  • Mixed-method design combined quantitative metrics and qualitative interviews.
• Limitations and future work:
  • Wizard-of-Oz methodology may have overrepresented the effectiveness of facilitated speech.
  • Sampling focused on DHH individuals who use spoken English, limiting generalizability.
  • LLM touch interface exhibited latency and variability in option ordering.
  • Future work should explore hands-free options like ASL recognition and refine LLM-based interaction.

Summary

This study compared three input methods for IPAs—Natural Deaf Speech, Facilitated English, and LLM-assisted touch—to assess accessibility for DHH users. Quantitative results showed comparable usability scores across methods, with qualitative feedback highlighting mixed experiences with speech recognition and positive attitudes toward touch interfaces. Alexa demonstrated surprising accuracy with deaf-accented speech, but significant limitations remain for users with atypical speech. The LLM-powered touch interface showed promise but requires improvements in latency and UI/UX design. Findings emphasize the need for multimodal IPA interaction options, including native support for deaf-accented speech and hands-free alternatives like ASL recognition.