Understanding and Enhancing the Role of Speechreading in Online d/DHH Communication Accessibility

Title of the Paper

Understanding and Enhancing The Role of Speechreading in Online d/DHH Communication Accessibility

Paper Information

• Subject Area: Accessibility in online communication for d/Deaf and Hard-of-Hearing (d/DHH) individuals, with a focus on the role of speechreading in video conferencing.
• Keywords: Speechreading, d/Deaf and Hard-of-Hearing, online communication, video conferencing, accessibility technology.

Research Background and Issues

• What problems or challenges did the authors identify?

  1. Speechreading, an important communication method, is underrepresented in HCI literature.
  2. Technical support for speechreading in online environments is insufficient, with current solutions failing to leverage rich social and environmental cues.
  3. Research on the interaction between speechreading and captions is still in its infancy, with most studies focusing on single technologies rather than the complexities of their combined use.

• Why is this issue important? Speechreading provides d/DHH individuals with an alternative way to "hear" language after auditory loss. As video conferencing and online conversations become increasingly prevalent, understanding how to optimize these technologies to support diverse communication strategies is critical.

• Motivation and Related Work

  1. Previous research has primarily focused on improving word-level speechreading accuracy or training algorithms, neglecting the complex socio-cultural and linguistic contexts of natural conversations.
  2. Current captioning technologies only provide semantic information, overlooking non-verbal and paralinguistic details such as tone and facial expressions.

Solutions

• What methods or solutions did the authors propose?

  1. Designed and tested a series of "design probes" to explore how speechreading and captioning technologies interact:
     • Bottom-up design: Resolving viseme ambiguities.
     • Top-down design: Providing contextual information (keywords, language switching, etc.).
  2. Conducted in-depth interviews and co-design workshops with 12 self-identified speechreading d/DHH participants to explore the social, environmental, and technical factors affecting communication accessibility.

• What is innovative about this solution? The innovation lies in expanding the study of speechreading from technical accuracy to exploring the complex communication dynamics in real-world usage scenarios, shifting the design focus toward more intuitive assistive technologies and user-driven design.

• What are the implementation steps? What key technologies were used?

  1. Three-phase research design:
     • Interviews: Gathered participants' background information and experiences with speechreading in both online and offline scenarios.
     • Design probe testing: Evaluated the effectiveness of various visual design schemes in supporting speech interpretation.
     • Co-design: Redesigned more effective and practical support solutions based on participant feedback.
  2. Key technologies included video overlays, dynamic caption generation, and artificial intelligence for recognizing speech and non-verbal information.

Research Outcomes

• What specific results were achieved?

  1. Proposed strategies for combining speechreading and captioning, along with their advantages and disadvantages in practical scenarios.
  2. Found that most existing technologies overlook the socio-cultural and environmental impacts on speechreading.
  3. Suggested a series of innovative design recommendations, such as dynamic captions, virtual "non-variant speech heads," and keyword extraction.

• How does it compare to existing solutions?

  1. Places greater emphasis on speechreading as a "multi-stream information integration" experience rather than solely enhancing accuracy.
  2. Introduces design standards that consider linguistic context, environmental factors, and users' social roles, enabling more inclusive communication technologies.

• What were the experimental or evaluation results?

  • Most participants found the existing bottom-up design probes overly complex and challenging to learn, preferring more intuitive top-down designs (e.g., keyword extraction and contextual switching indicators).
  • The proposed new design concepts were widely welcomed by participants, though further validation is needed for their technical feasibility.

• Limitations and Future Directions

  1. Limitations:
     • Small sample size (12 participants).
     • Did not include users of Cued Speech.
     • The sentence settings in the design probes were relatively simple and did not fully simulate the complexity of real-time conversations.
  2. Future directions:
     • Explore new features such as video synchronization, language detection, and multilingual captions.
     • Expand the participant pool to an international scope to study the impact of other languages and cultures on speechreading.
     • Develop real-time visual support systems and validate their long-term effectiveness.

Conclusion

This paper opens a new avenue for studying communication accessibility for d/DHH individuals in video conferencing, offering significant insights from the diverse application scenarios of speechreading to innovative design support. The study suggests that future assistive technologies should focus on a comprehensive consideration of social, technical, and environmental impacts while enhancing support for non-verbal speech information, achieving truly "rich and holistic" communication.