How Users Experience Closed Captions on Live Television: Quality Metrics Remain a Challenge

Title of the Paper

How Users Experience Closed Captions on Live Television: Quality Metrics Remain a Challenge

Bibliographic Information

• Subject Area: Human-Computer Interaction and Accessibility Design
• Keywords: Closed captions, quality metrics, caption usability, live television captions, caption accuracy, user experience evaluation, caption appearance, accessibility design

Research Background and Problem Statement

• What issues or challenges did the authors identify?

  • Closed captions are increasingly important for deaf and hard-of-hearing individuals, but existing quality evaluation metrics (e.g., word error rate) do not fully reflect user experience.
  • Even high-quality captions may still be perceived as problematic by users.
  • The weak correlation between objective quality metrics (e.g., accuracy) and subjective user evaluations indicates that accuracy is not the sole factor influencing user experience.
  • Live captions (especially those used in live television) are more prone to usability issues, such as poor synchronization and frequent errors.

• Why is this issue important?

  • Closed captions are crucial for education, entertainment, and information access, significantly impacting social participation and comprehension.
  • Current standards for evaluating caption quality fail to adequately capture user needs, potentially leading to inappropriate regulatory policies and suboptimal captioning experiences.

• Research motivation and related work:

  • This study aims to uncover the relationship between user experience and caption quality metrics.
  • Through quantitative and qualitative analysis, the study evaluates several existing caption quality metrics (e.g., Word Error Rate (WER), Weighted Word Error Rate (WWER), Automatic Caption Evaluation (ACE), and ACE2).
  • In connection with existing literature, the study explores factors such as hearing level, caption style, caption speed, and visual attention distribution that influence user experience.

Proposed Solution

What methods or solutions did the authors propose?

• Designed two user studies to evaluate the relationship between closed caption quality and user experience:
  • Study 1 (Pilot Study): Compared live television captions and offline television captions to investigate Word Error Rate (WER) and Weighted Word Error Rate (WWER).
  • Study 2: Expanded the research by increasing the sample size, examining more quality metrics (WER, WWER, ACE, ACE2), and incorporating qualitative analysis of user experience.

What is innovative about this solution?

• The study spans multiple quality metrics, deeply exploring their applicability in evaluating user experience with captions.
• The second study introduces caption styles and user-customized tests, exploring the multidimensional factors influencing user perception.
• The mixed-methods approach (quantitative + qualitative analysis) uncovers deeper factors affecting user experience, such as synchronization issues, readability, and caption appearance.

What are the implementation steps? What key technologies were used?

1. Experimental Design:
   • Created video clips (20 to 57 seconds) with captioned content.
   • Designed and debugged visualizations for different caption styles (live scrolling captions and offline pop-up captions).
   • Employed professional captioning services (Rev.com and 3PlayMedia) to test the quality of offline captions.
2. Data Collection:
   • Participants watched captioned videos and evaluated caption quality.
   • Collected participants' quality ratings and related perceived issues for each video.
3. Quality Metric Calculation:
   • Used tools (e.g., SCLite and ACE/ACE2 software) to generate error-type metrics for live captions.
4. Results Analysis:
   • Conducted correlation analysis to examine the relationship between subjective user ratings and objective caption quality.
   • Performed qualitative analysis to identify potential themes and key factors influencing user experience.

Research Findings

What specific findings were achieved?

1. Relationship Between Quality Metrics and User Ratings:
   • The tested metrics (WER, WWER, ACE, ACE2) showed weak correlations with user ratings, capturing only part of the user experience.
   • ACE2 had the highest correlation but still failed to fully reflect users' actual perceptions of caption quality.
2. Qualitative Analysis:
   • Users valued caption accuracy and synchronization but expressed significant frustration with issues such as difficulty following captions, lack of speaker identification, and captions obscuring content.
   • Caption style (scrolling vs. pop-up) and visual design (size, color, background transparency, etc.) did not yield consistent user preferences.

How does this compare to existing solutions? What are its advantages?

• This study explores caption quality evaluation from several new perspectives, such as the impact of hearing ability, caption style, and caption design configurations on user experience.
• Extends beyond purely quantitative data analysis to include qualitative user feedback, revealing a comprehensive set of influencing factors.

What were the experimental or evaluation results?

• Caption source significantly influenced user ratings, with live television captions scoring significantly lower than offline captions.
• Caption style had no significant impact on quality ratings, but issues with visual design and caption obstruction were widely mentioned.
• Linked caption errors (e.g., spelling mistakes, delays) to the feeling of difficulty in following captions, highlighting pain points in user experience.

Limitations and Future Directions

Limitations:

• The sample was biased toward individuals with high English proficiency, failing to capture the experiences of users with lower English proficiency.
• The short video clips provided limited contextual information, making it difficult to fully address error corrections.
• Automatic Speech Recognition (ASR)-based captions, which are increasingly used in live captioning, were not evaluated.

Future Directions:

• Recruit a more diverse user sample, encompassing multilingual and multicultural backgrounds.
• Explore more customized caption generation and display systems to meet user preferences.
• Introduce more accurate caption scoring methods, redesigning quality metrics by integrating user experience data.