More Human or More AI? Visualizing Human-AI Collaboration Disclosures in Journalistic News Production

Paper Title

More Human or More AI? Visualizing Human-AI Collaboration Disclosures in Journalistic News Production

Publication Info

• Topic area: Human-AI collaboration transparency in journalism.
• Keywords: Human-AI collaboration, AI disclosure, journalism, information visualization, transparency, generative AI, user study, co-design, eye tracking, news production.

Background and Problem

• Problem / challenge: Current AI usage disclosures in journalism are simplistic and fail to capture the nuanced collaboration between humans and AI during news production. This limits transparency and may lead to misperceptions about AI's role.
• Significance: Transparency in AI usage is critical for maintaining trust in journalism, mitigating misinformation, and adhering to emerging regulatory frameworks. Effective disclosures can help readers understand the contributions of both humans and AI.
• Motivation and related work: Prior studies have explored static labels and metadata for AI disclosure, but these approaches lack depth and fail to represent the dynamic interplay of human and AI contributions. This paper builds on existing work by designing and evaluating visual disclosures that communicate collaboration ratios and editorial steps.

Solution

• Proposed approach: Development and evaluation of four disclosure visualization prototypes—Textual Disclosure, Role-based Timeline, Task-based Timeline, and Chatbot—to represent human-AI collaboration in journalism.
• Novelty:
  1. Introduction of four distinct prototypes for visually disclosing human-AI collaboration in news production.
  2. Empirical evaluation of these prototypes using a mixed-methods approach, including eye tracking, questionnaires, and interviews.
  3. Insights into how disclosure designs influence perceptions of human and AI contributions, gaze patterns, and user preferences.
  4. Design considerations for implementing nuanced AI disclosures in journalism and beyond.
• Procedure and key techniques:
  1. Co-design sessions with 10 designers and HCI experts generated 69 disclosure concepts.
  2. Selection and development of four prototypes based on usability and information visualization principles.
  3. A controlled lab study with 32 participants evaluated the prototypes using eye tracking, questionnaires, and semi-structured interviews.
  4. Analysis of quantitative (e.g., gaze duration, perceived clarity) and qualitative (e.g., user preferences) data to assess effectiveness.

Results

• Concrete findings:
  • Textual Disclosure was least effective, often overlooked or perceived as too general.
  • Role-based Timeline provided clear overviews but sometimes misrepresented collaboration ratios.
  • Task-based Timeline offered clarity and insight into editorial steps but suffered from undiscovered hover interactions.
  • Chatbot was the most informative but also the hardest to understand, with longer engagement times.
• Advantage over baselines:
  • All prototypes effectively communicated human-AI collaboration ratios, outperforming simplistic textual labels.
  • Role-based and Task-based Timelines provided clearer overviews compared to Textual Disclosure.
  • Chatbot enabled in-depth exploration, surpassing other designs in informativeness.
• Experiments / evaluation:
  • Within-subjects lab study (N=32) using two news articles manipulated for human-AI collaboration ratios.
  • Measures included perceived clarity, informativeness, gaze patterns, and user preferences.
  • Eye tracking revealed differences in engagement across prototypes, with Chatbot requiring the most interaction.
• Limitations and future work:
  • Limited to four prototypes and low-stakes news articles; future studies should explore high-stakes contexts and multimodal disclosures.
  • Need for collaboration with journalists to assess feasibility and integration into editorial workflows.
  • Exploration of long-term effects of repeated exposure to AI disclosures and their impact on trust.

Summary

This study introduces and evaluates four visual disclosure prototypes to communicate human-AI collaboration in journalism. Through co-design sessions and a lab-based user study, the research demonstrates that while all prototypes effectively conveyed collaboration ratios, their effectiveness varied by purpose. Role-based and Task-based Timelines provided clear overviews, whereas the Chatbot offered detailed information but was harder to navigate. Textual Disclosure was the least effective. The findings highlight the potential of nuanced visualizations to enhance transparency in journalism and offer design considerations for broader applications in other domains requiring AI disclosures.