“Easier or Harder, Depending on Who the Hearing Person Is”: Codesigning Videoconferencing Tools for Small Groups with Mixed Hearing Status

Title of the Paper

“Easier or Harder, Depending on Who the Hearing Person Is”: Codesigning Videoconferencing Tools for Small Groups with Mixed Hearing Status

Paper Information

• Field of Study: Human-Computer Interaction, Accessibility Design
• Keywords: Accessibility design, Deaf and Hard of Hearing (DHH) communities, videoconferencing, real-time captions, codesign, behavioral feedback

Research Background and Issues

• Identified Problems or Challenges:

  1. Real-time captions generated by Automatic Speech Recognition (ASR) are widely used in videoconferencing but remain limited in accuracy and usability across different hearing scenarios.
  2. The communication behaviors of hearing individuals significantly impact the caption accessibility for Deaf and Hard of Hearing (DHH) users, yet existing research primarily focuses on individual DHH users or simple dyadic interactions between DHH and hearing individuals.
  3. Videoconferencing environments have not adequately addressed the needs of DHH communities, particularly in small mixed-hearing groups, where communication dynamics and social interactions critically influence accessibility outcomes.

• Significance of the Research:

  1. The widespread adoption of videoconferencing during the COVID-19 pandemic has accelerated the demand for accessibility technologies; optimizing real-time captions for groups rather than individual users is crucial.
  2. Mixed-hearing groups are becoming mainstream in educational and professional contexts, necessitating deeper design exploration of their unique communication needs.
  3. Design must go beyond technical capabilities to support changes in social behavior and foster accessibility norms through software infrastructure.

• Motivation and Related Work:
  The authors adopt a decentralized hearing culture approach long advocated by the Deaf community, incorporating the principle of “collective accessibility” from the disability rights movement. By collaborating with mixed groups of DHH and hearing individuals, they aim to co-create more accessible solutions for online interaction and collaboration.

Solution

• Proposed Solution:
  Through three codesign sessions with four participant groups (17 members, including 10 DHH and 7 hearing individuals), the authors explored directions for improving real-time captions in videoconferencing environments. The research design included questioning and observing current practices, sketching future features, and reviewing video prototypes.

• Innovative Contributions:

  1. Emphasizing a “collective accessibility” design approach, shifting the responsibility for accessibility from individuals to the entire group.
  2. Extending existing research by focusing on the social behaviors and communication norms that emerge in small mixed-hearing groups during long-term caption use.
  3. Proposing a series of feature designs centered on group behavioral feedback, including speaker identification and overlap marking, caption error tagging, camera view adjustments, and speech rate control.

• Implementation Steps and Key Techniques:

  1. Phase 1: Observing communication practices and challenges in each group through games and interviews.
  2. Phase 2: Encouraging participants to sketch feature ideas and prioritize them through discussion.
  3. Phase 3: Creating high-fidelity video prototypes incorporating each group’s design concepts to evaluate the features’ effectiveness and potential issues.
  4. Conducting experiments using Zoom and refining speech recognition, video design, and interaction interfaces.

Research Outcomes

• Specific Findings:

  1. In current practices, mixed groups establish specific accessibility norms through adjustments in communication behaviors, such as clear paraphrasing and appropriate turn-taking.
  2. Designed features include speaker identification, speech rate adaptation reminders, and camera adjustment prompts.
  3. Exploring methods to enhance videoconferencing infrastructure, such as creating group accessibility settings and profiles to share behavioral norms.

• Advantages Compared to Existing Solutions:

  1. This research extends the functionality of existing captioning technologies by shifting the focus from individual users to supporting group behaviors.
  2. Integrating accessibility design into the foundational infrastructure of videoconferencing technology provides structured support for customizing group standards.

• Experimental or Evaluation Results:

  1. Most features (e.g., speaker identification, overlap detection, personalized captions) were perceived by participants as significantly improving communication quality.
  2. Certain designs (e.g., contextual prompt buttons or volume display) were considered impractical or added unnecessary cognitive load.
  3. Preferences and usage of features varied based on social and individual factors, as well as group dynamics.

• Limitations and Future Directions:

  • Limitations:
    1. The small number of participants, predominantly white individuals from the U.S., limits cross-cultural applicability.
    2. Using Zoom as the sole platform may have restricted exploration of other tools.
    3. Video prototypes simulated but did not provide fully operational features, leaving real-world adaptability unverified.
  • Future Directions:
    1. Further research on the social impact of automated versus manual options in behavioral feedback tools.
    2. Testing feature applicability in multicultural settings and expanding the study to include other languages and communication styles.
    3. Developing digital “DeafSpace” system standards based on collective accessibility principles.