Analyzing Deaf and Hard-of-Hearing Users' Behavior, Usage, and Interaction with a Personal Assistant Device that Understands Sign-Language Input

Document Title

Analyzing Deaf and Hard-of-Hearing Users’ Behavior, Usage, and Interaction with a Personal Assistant Device that Understands Sign-Language Input

Document Information

• Subject Area: Assistive technology and human-computer interaction, specifically the design of personal assistant devices for deaf and hard-of-hearing users
• Keywords: Deaf and hard-of-hearing users, assistive technology, artificial intelligence, personal assistant devices, sign language recognition, accessible design

Research Background and Issues

• Identified Problems or Challenges:

  • Current voice-based personal assistant devices like Amazon Alexa pose accessibility challenges for deaf and hard-of-hearing users, as these devices typically require voice input or rely on voice output.
  • Despite media coverage, there is currently no effective sign language recognition system capable of accurately understanding user commands.
  • Existing datasets for sign language recognition are highly limited, making it difficult to support the training of modern machine learning models effectively.

• Importance of the Issues:

  • As voice control technology becomes more prevalent, failure to address accessibility issues will further exacerbate the digital and technological inequality faced by deaf and hard-of-hearing users.
  • The COVID-19 pandemic has highlighted the importance of home technology, making the development of sign language recognition interaction technology more urgent.

• Research Motivation and Related Work:

  • Existing studies have explored the interest of deaf and hard-of-hearing users in sign language interaction and investigated the types of commands they might want to use with devices. However, these studies are primarily based on user assumptions rather than real interaction scenarios.
  • No research has been conducted to observe actual user behavior to uncover how users naturally interact with devices capable of understanding sign language.

Solution

• Proposed Method or Solution:

  • The authors employed the "Wizard-of-Oz" experimental method (using a hidden human sign language translator to simulate the device's sign language recognition functionality), allowing participants to interact with a personal assistant device using sign language.
  • Detailed documentation of participants' commands, device activation methods, and user behaviors when the device made errors was conducted, resulting in a video dataset of over 1,400 interaction segments with annotations.

• Innovative Aspects:

  • This is the first study to observe actual interaction behaviors of deaf and hard-of-hearing users with a personal assistant device capable of understanding sign language.
  • Practical design guidelines were provided, including preferred activation methods, common ASL (American Sign Language) command structures, and strategies for handling interaction errors.
  • The first annotated dataset of sign language interactions with personal assistant devices was made publicly available.

• Implementation Steps and Key Techniques:

  1. Using a "Wizard-of-Oz" setup, participants interacted with an Alexa device (simulated to understand sign language) via a video conferencing platform.
  2. High-quality video recordings captured each participant's interaction process.
  3. Three team members annotated and categorized the videos in detail, including activation methods, command types, and user responses to errors.
  4. Tools like BERTopic were used for topic modeling of commands to identify broad command categories and patterns.

Research Outcomes

• Specific Outcomes:

  • Identified 11 common activation methods used by deaf and hard-of-hearing users, including spelling "Alexa," waving, and ASL-specific gestures (e.g., the "Curious" gesture).
  • Extracted 15 categories of command topics used by participants, including device control (352 instances), entertainment (162 instances), shopping (126 instances), and unique categories specific to deaf and hard-of-hearing users (e.g., subtitle requests).
  • Defined five major user coping strategies for device errors, including ignoring the error, repeating the command with the same or different expressions.
  • Observed code-switching behavior among participants, transitioning from fluent ASL to more English-like expressions, particularly in scenarios involving repeated device errors.

• Advantages Compared to Existing Solutions:

  • Surpassed studies based on assumptions and survey data by observing actual user-device interactions.
  • The dataset provides a valuable practical foundation for research on sign language recognition and accessible interaction technologies.

• Experimental or Evaluation Results:

  • Using the "Wizard-of-Oz" prototype, participants' interest in the device did not decline after a series of interaction experiences; pre- and post-experiment interest levels were statistically equivalent.
  • The collected interaction data covered extensive content not explored in fragmented datasets, offering practical value for machine learning models and device optimization.

• Limitations and Future Directions:

  • The experiment was conducted online due to the COVID-19 pandemic, preventing testing of physical touch-based activation methods and the use of high-resolution cameras to capture detailed user behaviors.
  • Future research could explore longer-term, more natural device usage behaviors in real home environments.
  • Investigate device forms beyond screens and interactions with screenless devices for DHH users.
  • Extend application areas, such as notifying users of surrounding sound changes (e.g., smoke alarms, doorbells) or providing translation services.