Cocomix: Utilizing Comments to Improve Non-Visual Webtoon Accessibility

Title of the Paper

Cocomix: Utilizing Comments to Improve Non-Visual Webtoon Accessibility

Paper Information

• Field of Study: Accessibility Design, Digital Comics (Webtoon), Human-Computer Interaction
• Keywords: Accessibility Design, Digital Comics, Webtoon, Image Descriptions, Comment Data, Low Vision Users
• Conference: CHI Conference on Human Factors in Computing Systems (CHI '22)
• Publication Date and Location: April 29 - May 5, 2022, New Orleans

Research Background and Problem

• Research Problem:

  • Webtoons, as a popular form of digital comics, are highly visual, making it difficult for Blind or Low Vision (BLV) users to achieve an equivalent user experience.
  • Traditional image description methods often lead to information overload, redundant descriptions, loss of narrative coherence, and lack of user control over detail and reading pace in Webtoons.
  • While existing comment data can highlight key plot points and provide background information, there is no clear research on how to leverage this data to enhance Webtoon descriptions.

• Significance of the Research:

  • With a massive global user base, Webtoons often exclude BLV users from mainstream cultural content, creating significant accessibility challenges.
  • By integrating reader-contributed comment data, it is possible to develop more efficient tools for accessible Webtoon reading for BLV users, enhancing cultural inclusion and content experience.

• Motivation and Related Work:

  • Previous studies have shown that reader comments are highly contextually relevant and can serve as a supplementary knowledge source.
  • While accessibility design has explored image and multimedia content, the unique format of Webtoons (vertical scrolling and social interaction) requires novel accessibility design approaches.

Solution

• Method and Solution:

  • Introduced an interactive Webtoon reading tool, Cocomix, which enhances the BLV user experience through comment-driven description generation and interactive design.
  • Highlighted two core features:
    1. Comment-Based Adaptive Descriptions: Dynamically adjust the level of description for each Webtoon panel, providing detailed descriptions for key panels and concise descriptions for less critical ones.
    2. Panel-Anchored Comments: Extract comments related to specific panels, allowing users to explore comments about different plot points on demand.

• Innovations:

  1. First attempt to use reader comment data as a basis for scoring the importance of panel descriptions.
  2. Provides fine-grained interactive controls (users can double-tap to trigger detailed descriptions or swipe to view comments).
  3. Implements an automated pipeline for comment extraction, panel importance scoring, and comment association.

• Implementation Steps and Key Technologies:

  1. Description Design Standards: Developed a set of standards specifically for Webtoon descriptions, streamlining redundant information and grouping panels to reduce repetitive descriptions.
  2. Data Processing: Employed natural language processing techniques, such as noun segmentation, grammar correction, and context matching, to extract high-quality, semantically clear comments.
  3. Automated Pipeline:
     • Panel Importance Scoring: Scored each panel based on comment frequency, dynamically setting the level of detail for key panels.
     • Comment Extraction: Filtered descriptive comments based on content relevance and grammatical features.
     • Comment Association and Presentation: Anchored extracted comments to corresponding panels using semantic similarity and named entity matching.

Research Outcomes

• Specific Outcomes:

  1. Developed a prototype system, Cocomix, which implements dynamic description adjustments and comment anchoring features.
  2. Conducted qualitative analysis on 1,000 Webtoon comments, creating a taxonomy of annotations (e.g., descriptive vs. non-descriptive comments) to support system optimization.
  3. Proposed and validated a new set of accessibility description standards for Webtoons.

• Advantages:

  1. Provides users with greater control over reading pace and information detail.
  2. Offers more efficient and concise audio descriptions for Webtoons compared to existing methods.
  3. Enriches narrative descriptions with comments, enhancing users' understanding and engagement with the content.

• Experiments and Evaluation:

  • Conducted user studies and system validation to evaluate system functionality, technical feasibility, and user experience, including the following two components:
    1. Pipeline Evaluation: Tested the accuracy of comment extraction and panel-anchored comments, showing that the system-extracted descriptive comments were more relevant and useful than the most-liked comments.
    2. User Study: Trial tests with 12 BLV participants indicated that Cocomix was easier to use than baseline systems, and its comment-driven interactive features helped users better understand Webtoon content.

• Limitations and Future Directions:

  1. Dependency on Comment Strategies: Effectiveness may be limited for newly released Webtoons or those with few comments.
  2. Automation of Description Generation: Currently requires manual creation of descriptions; future work could explore automated generation techniques.
  3. Diverse User Needs: Further optimization is needed to meet the varied accessibility needs of users (e.g., based on cognitive load and reading preferences).