Remotely Co-Designing Features for Communication Applications using Automatic Captioning with Deaf and Hearing Pairs

Title of the Paper

Remote Collaborative Design: Deaf and Hearing Individuals Co-designing Features to Optimize Communication Applications with Automatic Subtitles

Paper Information

• Field of Study: Human-Computer Interaction (HCI) and Accessibility Design
• Keywords: Deaf and Hard of Hearing (DHH) users, accessibility design, participatory design, automatic speech recognition (ASR), video conferencing, user experience, mixed-ability groups, collaborative design, technological accessibility, remote research

Research Background and Issues

• Problems or Challenges Identified by the Authors:

  • Deaf and hard of hearing (DHH) users face communication challenges in both face-to-face and remote meetings, while current automatic speech recognition (ASR)-based solutions have limitations in accuracy and user interface design.
  • Effective communication with hearing users in group meetings is crucial for the professional success of DHH users. However, traditional solutions such as professional sign language interpreters or real-time subtitles often require prior arrangements, making them unsuitable for impromptu needs.

• Significance of the Research:

  • The shift to remote meeting formats due to the COVID-19 pandemic has made traditional communication methods (e.g., written notes and gestures) more difficult, highlighting the importance of improving accessibility technologies.
  • There is a lack of user experience research in HCI regarding ASR-based communication applications, and few studies have integrated the collaborative participation of both DHH and hearing users in technology design.

• Motivation and Related Work:

  • The pandemic disrupted traditional face-to-face collaborative design methods. Remote collaborative design, however, can address challenges such as participant recruitment and scheduling, while increasing geographical diversity among participants.
  • This study proposes a new approach by leveraging remote collaborative design with DHH and hearing users to explore the effectiveness of ASR feature design, addressing gaps in existing research.

Solution

• Research Methods:

  • The authors designed and conducted 18 fully virtual collaborative design workshops, each involving one DHH participant and one hearing participant, held via Zoom.
  • The workshops focused on two key directions:
    1. How to correct transcription errors in ASR outputs;
    2. How to design notification systems to influence the speaking behavior of hearing users (e.g., reminding them to slow down their speech).
  • The method involved a series of pre-planned collaborative design activities, including individual brainstorming, icebreaker activities using design tools, individual sketching, and team-based collaborative prototyping.

• Innovative Contributions:

  • This study uniquely combines remote design methods, mixed-ability group participation, and collaborative prototyping, offering a novel participatory design model.
  • The authors developed collaborative dialogue methods using real-time online tools (e.g., Miro whiteboards) and flexibly integrated sign language interpretation and text chat to meet diverse user needs.

• Implementation Steps and Key Technologies:

  1. Optimizing communication methods using text chat and sign language interpretation;
  2. Utilizing online tools (Miro) for collaborative prototyping, supplemented by text boxes to provide explanations for design elements that could not be expressed visually;
  3. Participants collaboratively designed ASR features through discussions, sketching, and prototype iterations.

Research Findings

• Specific Outcomes:

  • Methodological Contribution: Validated that remote collaborative design methods effectively facilitate cooperation between DHH and hearing users, producing high-quality design prototypes.
  • Identified three main types of solutions for addressing ASR errors and designing notification systems: icons, pop-ups, and overlays, categorized by visual interference levels into low, medium, and high detectability.
  • Proposed practical recommendations for accessibility design, including workflow setup for remote design, workshop structure, and specific strategies to encourage participant engagement.

• Advantages:

  • Compared to traditional participatory design methods, remote design overcomes geographical limitations and leverages online tools to support user collaboration.
  • Integrating perspectives from mixed-ability groups makes design prototypes more universal and adaptable to diverse accessibility needs.

• Experimental or Evaluation Results:

  • The workshops produced a series of high-quality design prototypes, including mechanisms for correcting text errors and notification features that influence the speaking behavior of hearing users.
  • The research method effectively enhanced collaborative engagement between DHH and hearing users, ensuring smooth communication and information exchange.

• Limitations and Future Directions:

  • Limitations: The study was conducted exclusively via Zoom, which may be constrained by the platform's specific functionalities. Additionally, the small number of participants, predominantly from certain educational backgrounds, may limit the generalizability of the results.
  • Future Research Suggestions: Extend workshop durations to broaden the scope of discussions; diversify participant demographics; further test design prototypes in real-world communication scenarios; explore more open-ended design topics.