ReelFramer: Human-AI Co-Creation for News-to-Video Translation

Document Title

ReelFramer: Human-AI Co-Creation for News-to-Video Translation

Document Information

• Subject Area: Integration of artificial intelligence and news creation, a content redirection tool for news video production
• Keywords: AI co-creation, creative support tools, short videos, scriptwriting, storyboard creation, narrative frameworks

Research Background and Problem

• Identified Problem or Challenge: Younger audiences increasingly prefer consuming news through short videos on social media, yet news organizations face challenges in converting traditional news texts into engaging short video content. They struggle to balance the accuracy of news information with the entertainment value of short videos; the generated content may lack coherence or fail to align with the style of social platforms.
• Importance of the Problem: Short videos have become a crucial medium for reaching younger audiences. Converting news into this format can more effectively capture public attention, promote message dissemination, and support social education. However, incorrect or incomplete information may mislead audiences, posing a risk to the credibility of news organizations.
• Research Motivation and Related Work: Narrative frameworks are a core concept in traditional news creation, used to organize and optimize content presentation, but their application in short video news remains underexplored. Additionally, generative AI has been applied in text generation and storytelling, but its use in news production still requires addressing issues of information accuracy and control.

Solution

• Proposed Solution: ReelFramer is a human-AI co-creation system that supports news creators in transforming text-based news into social media short video scripts and storyboards. The system introduces three narrative frameworks (descriptive dialogue, on-site reenactment, humorous analogy), allowing users to explore different balances between informational and entertainment value.
• Innovations:
  • Leveraging generative AI to explore narrative frameworks and visual design.
  • Emphasizing the intermediate stage before script generation—establishing narrative details (characters, plot, scenes, and key information).
  • Through interactive design, enabling users to participate in the decision-making process at any time, ensuring content accuracy, coherence, and feasibility.
• Implementation Steps and Key Technologies:
  1. Script Creation:
     • Extracting key information ("who," "what," and "where") from text-based news.
     • Automatically suggesting narrative details, including characters, plot, scenes, and information points.
     • Generating draft scripts based on the user-selected narrative framework, with options for editing and regeneration.
     • Using a highlighter for generated content to evaluate and optimize the script's information coverage.
  2. Visual Design:
     • Generating character profiles based on the script (including costumes, props, and personality design).
     • Proposing scene backgrounds and using DALL-E 2 to generate visual content consistent with textual details.
     • Producing black-and-white sketch-style storyboards, describing actions and expressions in the frames.
  3. Interactive Design:
     • Offering multi-round concept exploration and iterative generation until user satisfaction is achieved.
     • Allowing users to edit, rewrite, or save scripts and visual content throughout the process.

Research Outcomes

• Specific Outcomes:
  • Proposed three narrative frameworks (descriptive dialogue, on-site reenactment, humorous analogy) to provide diverse structures for content presentation.
  • Developed the ReelFramer system, significantly simplifying the news video creation process.
  • Found that establishing narrative details (characters, scenes, information points) as an intermediate step significantly enhances script informativeness and coherence; user satisfaction with generated scripts improved markedly.
• Advantages:
  • Enables users to efficiently explore different narrative frameworks, significantly reducing script creation time.
  • The provided tools lower the entry barrier for novice news video creators and enhance decision-making autonomy in professional news production.
• Experimental Results:
  • Compared to conditions without narrative details, scripts generated with narrative details showed significantly better performance in information coverage, framework consistency, and coherence.
  • User experience studies demonstrated that the system effectively supports the visualization of news content without compromising accuracy or style.
• Limitations and Future Directions:
  • Technical Limitations: Generative AI may produce "hallucinated" incorrect information in some scenarios, relying on users' journalistic expertise for editing and review.
  • Content Scope: The current system primarily supports role-playing-style short videos and struggles to cover all news topics (e.g., particularly serious news).
  • Bias Issues: Generative models may reinforce gender or cultural stereotypes in character and scene design.
  • Future Directions: Incorporating support for social media trends (e.g., popular songs or visual effects), improving intelligent verification mechanisms for content generation, and expanding to domain-specific expert users.

This study provides a successful case for applying generative AI in content creation and proposes scalable methodologies, contributing to addressing the challenges faced by news organizations in the era of short videos.