Caption Royale: Exploring the Design Space of Affective Captions from the Perspective of Deaf and Hard-of-Hearing Individuals

Title of the Paper

Caption Royale: Exploring the Design Space of Affective Captions from the Perspective of Deaf and Hard-of-Hearing Individuals

Paper Information

• Research Area: Accessibility Design and Human-Computer Interaction (HCI)
• Keywords: Accessibility, Caption Design, Emotional Expression, Deaf and Hard-of-Hearing, Human-Computer Interaction, Typography Design, Emotion Recognition Models, Diverse Experimental Methods

Research Background and Problem Statement

• Research Questions or Challenges:

  • Current captioning technologies have made significant progress in accurately conveying linguistic content but primarily focus on verbal information, lacking the expression of emotion or tone.
  • Deaf and Hard-of-Hearing (DHH) audiences miss out on emotional cues conveyed through non-verbal vocal information, impacting their content experience.
  • Existing studies have preliminarily explored conveying emotions through caption styles (e.g., using color, font adjustments, etc.), but there has been no systematic investigation of DHH individuals' preferences and the effectiveness of these design styles.

• Significance:

  • Providing accurate and emotionally rich captions for DHH individuals can significantly enhance their video content experience.
  • Systematic exploration of emotional caption design can not only enrich design approaches in the HCI field but also advance accessibility technologies.

• Research Motivation:

  • Limited existing work has preliminarily shown that "emotional captions" can enhance the understanding of emotional contexts for DHH individuals, but broad empirical validation from the user perspective is lacking.
  • The importance of scrolling captions (word-by-word display) for emotional descriptions among deaf users has not been thoroughly studied, representing a significant gap.

Proposed Solution

• Methods and Solutions:

  1. Propose nine distinct caption styles that use attributes such as font color, font weight, size, and shadow to express two dimensions of emotion: valence (positive or negative emotions) and arousal (level of excitement).
  2. Employ a three-phase research design:
     • Study 1: Evaluate the performance of these nine styles in conveying valence and arousal independently.
     • Study 2: Combine the best-performing styles from Study 1 to explore their effectiveness in conveying both valence and arousal together.
     • Study 3: Compare the optimal styles identified in the first two studies with neutral styles in terms of subjective and objective effects.

• Innovations:

  • Conduct the first systematic and comprehensive analysis of the visual design space for "emotional captions."
  • Combine user preferences with emotional conveyance effectiveness, directly comparing which styles better serve DHH individuals.
  • Utilize unique experimental methods (e.g., EmojiGrid for two-dimensional emotion rating, Best-Worst Scaling) to validate style effectiveness from both preference and emotional interpretation perspectives.

• Implementation Steps and Key Techniques:

  1. Data Preparation: Extract transcripts, timestamps, and vocal emotions (based on emotion recognition networks) from emotionally rich video materials.
  2. Style Design: Select font parameters most relevant to emotions from the literature and design experimental styles.
  3. Data Collection: Recruit DHH participants to conduct multi-round tests evaluating style preferences and interpretation effectiveness.
  4. Data Analysis: Quantify style performance using ranking algorithms (e.g., TrueSkill) and correlation tests (e.g., distance correlation).

Research Outcomes

• Specific Findings:

  • Study 1 Results:
    • For valence: Font color and shadow color were the most effective.
    • For arousal: Font color, font weight, font size, and shadow color performed well.
  • Study 2 Results:
    • Optimal combinations included: font color with font weight, font color with font size, font color with shadow color, and shadow color with font size.
  • Study 3 Results:
    • Font color combined with font weight and font color combined with font size were the most effective, demonstrating distinct advantages in cognitive load and emotional conveyance efficiency.

• Advantages:

  • Performance: Subjective preference rankings and results from emotion recognition tasks showed that the above two combination styles significantly outperformed traditional neutral captions.
  • Flexibility: Participant feedback supported further personalization through parameter adjustments (e.g., color intensity, size, weight range).

• Experimental and Evaluation Results:

  • Subjective Perception: Participants found the combination of font color and font size to be the most intuitive for conveying emotions, while the font color and font weight combination offered lower visual interference and cognitive load.
  • Objective Testing: The two optimal styles (Font-color with Font-size and Font-color with Font-weight) showed significantly higher correlation with reference emotional coordinates in emotion recognition tasks compared to other styles.
  • Limitations:
    • Emotional captions may introduce visual fatigue for prolonged video viewing.
    • The cultural differences in color-emotion associations require further validation.

• Future Directions:

  1. Test additional conversational scenarios (e.g., multi-speaker dialogues, real-time captions).
  2. Investigate the impact of cultural differences on the interpretation of emotional captions.
  3. Optimize the real-time processing capabilities of emotional caption systems for dynamic and natural environments.
  4. Explore broader user personalization options to enhance flexibility.