Assessment of Sign Language-Based versus Touch-Based Input for Deaf Users Interacting with Intelligent Personal Assistants

Document Title

Assessment of Sign Language-Based versus Touch-Based Input for Deaf Users Interacting with Intelligent Personal Assistants

Document Information

• Subject Area: Accessible Interaction Design and Intelligent Personal Assistants
• Keywords: Deaf users, accessible design, intelligent personal assistants, American Sign Language, human-computer interaction, user experience, gesture input, speech recognition technology, multimodal interaction, user studies

Research Background and Issues

• Research Questions and Challenges:
  • Intelligent Personal Assistants (IPAs) primarily interact with users through Automatic Speech Recognition (ASR). However, this voice input mode is not suitable for deaf users.
  • Key issues encountered include:
    • Poor accuracy of speech recognition for atypical speech (e.g., deaf speech).
    • Most IPAs lack support for sign language command input.
    • Limited research on the use of IPAs by deaf users in home environments, with insufficient accessibility design guidelines for related user interfaces.
• Research Motivation:
  • There are approximately 70 million deaf individuals worldwide, with about 500,000 using American Sign Language (ASL) daily. Optimizing IPA input methods for this group is of significant importance.
  • Enhancing the feasibility of sign language in intelligent assistants could pave the way for future universal sign language recognition technologies.
  • The authors respond to two research calls from other scholars: “focusing on real-world applications” and “developing user interface standards for sign language interaction.”

Solution

• Methods and Experimental Design:

  • This study employs the Wizard-of-Oz (WoZ) method to simulate IPA automatic recognition of American Sign Language (ASL). It investigates user experiences with ASL, Tap to Alexa, and smart home application interaction methods in limited scenarios within a smart home environment.
  • In the WoZ method, a researcher proficient in ASL acts as a hidden "wizard," translating ASL into spoken commands in real-time and sending them to an Amazon Echo Show device.
  • The experiment also collected data on users’ language habits, including the distribution of sign language vocabulary and structured observations.

• Key Technologies and Steps:

  1. Input Mode Comparison:
     • Testing three input modes: ASL (Wizard-of-Oz), Tap to Alexa (touchscreen input), and smart applications (operated via phone or tablet).
  2. Study Tasks:
     • Designed three parallel task lists covering common intelligent assistant functions such as lighting control, video playback, and timer tasks.
  3. Data Collection and Analysis:
     • Measured user experience using the System Usability Scale (SUS) and post-task questionnaires.
     • Annotated and analyzed sign language videos using the ELAN tool for vocabulary and non-gesture signals.

• Research Innovations:

  • Systematically compared ASL and touch-based input in smart home environments for the first time.
  • Examined linguistic phenomena in sign language users’ interactions with IPAs, providing data to support the design of future automated sign language input systems.
  • Integrated sign language linguistics, HCI, and accessible design to propose culturally sensitive new IPA wake-up methods.

Research Results

• Specific Findings:

  1. User Experience Results:
     • The ASL mode achieved a SUS score of 71.6, approaching the “acceptable” range and slightly higher than the average score of other systems (70).
     • Tap to Alexa and application-based input methods scored 61.4 and 56.3, respectively, indicating medium to low levels of user experience.
  2. Linguistic Analysis Conclusions:
     • Participants used an average of 47 sign language words, plus 10 fingerspelled words, totaling 246 distinct annotations.
     • Approximately 117 vocabulary items were critical for IPA semantic understanding (covering core command-related terms).
     • Users frequently employed gestures (e.g., waving and pointing) and phrase indexing for interaction, suggesting that IPA support for referential terms requires further exploration.

• Comparison with Existing Solutions:

  • The ASL mode’s user experience was close to traditional voice interaction methods for general populations (SUS 63.7) and outperformed existing text or touch input methods.
  • ASL interaction demonstrated cultural sensitivity, such as gesture-based wake-up methods aligning better with communication habits of the deaf community.

• Research Limitations:

  • The Wizard-of-Oz method introduces potential biases in ASL translation, unable to fully reflect delays and accuracy in future automated systems.
  • Due to experimental constraints, task design focused on limited smart home scenarios, not fully covering the broad interaction needs of deaf users.
  • English wake-up words were culturally mismatched, affecting user perceptions in certain tasks.

• Future Directions:

  1. Explore diverse wake-up methods (e.g., waving, eye contact, and naming actions) for adaptation in true ASL intelligent assistants.
  2. Expand research scope to daily, immersive scenarios such as kitchen or in-car environments, testing ASL functionality in complex dynamic operations.
  3. Advance interdisciplinary research to improve sign language recognition accuracy, particularly for single-handed numbers, fingerspelling, and context-dependent meanings.

Main Output Format and Clarity

• ASL can support limited smart scenario interactions but requires overcoming cultural and technical challenges.
• The authors call for further exploration of sign language recognition technology and accessible intelligent design.