Towards Inclusive Video Commenting: Introducing Signmaku for the Deaf and Hard-of-Hearing

Title of the Paper

Towards Inclusive Video Commenting: Introducing Signmaku for the Deaf and Hard-of-Hearing

Paper Information

• Research Area: Inclusive design for video learning and technology applications for Deaf and Hard-of-Hearing (DHH) users
• Keywords: Deaf and Hard-of-Hearing (DHH), social interaction, Danmaku, Signmaku, video learning, generative AI, privacy protection, educational technology

Research Background and Problem

• Identified Problems or Challenges:

  • Existing Danmaku, a form of video annotation through text-based comments, is not user-friendly for many DHH users who primarily communicate through sign language.
  • Sign language users face privacy risks when sharing and viewing comments, while text-based comments fail to provide the equivalent visual and semantic richness of sign language.
  • There is a lack of interactive mechanisms supporting sign language in video learning, which creates additional barriers for DHH users in social interaction and learning participation.

• Significance of the Problem:

  • Over 500,000 people in the United States use American Sign Language (ASL) as their primary communication language, yet video learning platforms are predominantly designed for hearing users.
  • Video learning platforms can enhance learning outcomes through interactivity, but the lack of support for DHH users may exacerbate educational and social inequalities.

• Research Motivation and Related Work:

  • Existing research primarily focuses on improving video accessibility for DHH students through captioning technologies, which do not fully address their information access needs.
  • Social media and Danmaku, as emerging forms of student engagement and interaction, often overlook the needs of sign language users.
  • To explore the potential of Danmaku, research must accommodate the preferences of DHH users, enabling them to communicate in more natural ways.

Proposed Solution

• Proposed Method or Solution:

  • Introduced a sign language-based commenting feature called “Signmaku,” a new interaction model that allows DHH users to comment in ASL.
  • Explored three distinct styles of Signmaku design: Realistic (real human face videos), Cartoon (cartoon-style videos), and Robotic (robot-style videos), addressing the trade-off between privacy and comprehensibility.

• Innovative Contributions:

  • Proposed a novel interaction model combining sign language with Danmaku, enhancing the social connection between video content and learners in video learning.
  • Leveraged generative AI technologies to process sign language videos, exploring the balance between privacy protection and content comprehensibility.

• Implementation Steps and Key Technologies:

  • Phase 1: Conducted a needs assessment with 12 DHH users to gather feedback and preferences regarding Signmaku, including viewing and commenting experiences.
  • Phase 2: Conducted comparative experiments with 20 DHH users to evaluate perceptions of the three Signmaku styles and their acceptance of commenting and sharing Signmaku.
  • Key Technologies:
    • Used VToonify to apply cartoon-style facial and background filters.
    • Used DeepMotion to generate robotic-style facial and body movements.

Research Outcomes

• Specific Findings:

  • The Cartoon style of Signmaku was the most popular due to its combination of entertainment value, low cognitive load, and moderate privacy protection.
  • The Realistic style of Signmaku had the highest comprehensibility and lowest cognitive load but lacked privacy protection.
  • The Robotic style of Signmaku resulted in the highest cognitive load, primarily due to unnatural gestures and facial expressions.

• Comparison with Existing Solutions and Advantages:

  • Compared to text-based Danmaku, Signmaku offers a more user-friendly interaction model for DHH users, addressing the limitations of captions by providing visualized sign language information.
  • The Cartoon style achieved a better balance between sharing and privacy protection, with higher comment and sharing rates than the Robotic style.

• Experimental and Evaluation Results:

  • Users reported significantly enhanced emotional engagement and entertainment during training sessions when viewing Cartoon Signmaku.
  • The process of sharing Signmaku revealed that DHH users demonstrated greater fluency in sign language expression compared to text expression and were more willing to communicate through Signmaku.

• Limitations and Future Directions:

  • Limitations:
    • The study was limited to ASL, and other sign language systems may have different requirements.
    • Participants were primarily from universities focused on DHH education, limiting sample diversity.
    • The current Signmaku design does not support real-time personalization, requiring future development of customization tools.
  • Future Research Directions:
    • Validate the adaptability of Signmaku in mainstream educational environments and expand to DHH users from diverse sign language and cultural backgrounds.
    • Explore ways to involve hearing users in the Signmaku platform to foster broader interaction and collaboration across user groups.
    • Optimize generative AI models to more accurately represent sign language expressions and support a wider range of filter options.