"We Speak Visually" : User-generated Icons for Better Video-Mediated Mixed Group Communications Between Deaf and Hearing Participants

Title of the Paper

“We Speak Visually”: User-Generated Icons for Better Video-Mediated Mixed-Group Communications Between Deaf and Hearing Participants

Paper Information

• Research Domain: Human-Computer Interaction and Accessibility Design
• Keywords: Deaf and Hard of Hearing (DHH), Accessibility, Video Conferencing, User-Generated Icons, Mixed-Group Communication, Sign Language, Nonverbal Communication, Participatory Design, Emojis, Cultural Differences

Research Background and Problem Statement

• Identified Problems or Challenges:

  1. Video conferencing platforms are primarily designed for auditory users, offering insufficient support for Deaf and Hard of Hearing (DHH) users.
  2. Significant linguistic and cultural differences exist between Deaf and hearing users, making inclusive communication in mixed groups difficult.
  3. While DHH users benefit from features like real-time captions, traditional video conferencing reliant on text and auditory cues fails to meet their communication needs, especially in fast-paced or informal meeting environments.
  4. The loss of nonverbal signals (e.g., body language and facial expressions) and issues with low video quality exacerbate cognitive load and communication barriers for DHH users during video conferences.

• Importance of the Problem:
  As remote work becomes normalized due to the COVID-19 pandemic, developing remote communication technologies that accommodate users with disabilities has become a critical issue for social equity and workplace diversity. Insufficient support for mixed-group communication risks isolating DHH users and reducing their workplace efficiency.

• Research Motivation and Related Work:
  Previous studies have demonstrated the effectiveness of icons and emojis in conveying emotions and intentions in digital communication, but there is limited research on their potential to improve communication between mixed groups of DHH and hearing users. This study explores the use of user-generated icons to bridge linguistic gaps through participatory design.

Proposed Solution

• Proposed Method or Solution:
  The authors propose using user-generated icons to address communication barriers between DHH and hearing users. By combining participatory design and focus group interviews (FGI), the study investigates how icons can enhance the efficiency and accuracy of expression and understanding in video conferencing.

• Innovative Aspects of the Solution:

  1. Introduction of user-generated icon-based nonverbal visual tools to establish a shared "language" for DHH and hearing users.
  2. Use of participatory design to uncover implicit user needs, positioning icon design as a central element in optimizing mixed-group communication during video conferences.
  3. Integration of DHH cultural characteristics (e.g., sign language, non-manual signals) with the limitations of existing video conferencing platforms to propose future directions for ICON system design.

• Implementation Steps and Key Techniques:

  1. Conduct focus group interviews to understand communication challenges and needs in mixed groups.
  2. Design two-part experiments: user-generated icon creation and real-time conference simulation using icons.
  3. Categorize user-generated icons based on feedback and summarize their usage patterns and functions, including five main categories: reactions, participation, explanations, status indications, and requests for help.

Research Outcomes

• Specific Findings:

  1. Icons were shown to lower the linguistic comprehension barrier, facilitating quick communication between DHH and hearing users.
  2. The inclusion of icons enhanced DHH users' sense of participation, making mixed-group communication more equitable and engaging.
  3. User-generated icons supported communication through various functions such as reactions, status indications, and direct explanations, not only conveying information but also fostering harmonious interaction.

• Advantages Over Existing Solutions:
  Compared to relying solely on automatic captions or sign language interpreters, user-generated icons provide a more intuitive way to convey user intentions, avoiding lengthy and complex textual or signed explanations. They are also adaptable to various real-time meeting environments.

• Experimental or Evaluation Results:

  1. Icons improved communication efficiency, interaction quality, and emotional expression in mixed groups.
  2. Icons demonstrated potential for cross-cultural and cross-group use, with users generally agreeing that they helped establish consensus and reduce misunderstandings.

• Limitations and Future Directions:

  1. The study involved a small participant group, and hearing participants had prior exposure to Deaf culture, which may not represent broader workplace scenarios.
  2. Icons have limitations in expressing lengthy content and may distract users when searching for appropriate icons.
  3. Icon usage may be constrained by meeting topics (e.g., less frequent use in formal or unfamiliar group settings).

  Future Research Directions:

  • Expand the study to include more diverse user groups (e.g., individuals with varying degrees of hearing loss).
  • Extend experiments to broader contexts such as education and family interactions.
  • Develop dynamic icons and multimodal systems combining text and gestures to enhance real-time functionality and expressive capabilities.