Envisioning Narrative Intelligence: A Creative Visual Storytelling Anthology

Document Title

Envisioning Narrative Intelligence: A Creative Visual Storytelling Anthology

Document Information

• Authors: Brett A. Halperin, Stephanie M. Lukin
• Affiliations: University of Washington, DEVCOM Army Research Laboratory
• Publication Date and Conference: April 23-28, 2023, CHI '23, Hamburg, Germany
• DOI: 10.1145/3544548.3580744
• Research Areas: Human-Computer Interaction (HCI), Visual Storytelling Generation, Creative Artificial Intelligence
• Keywords: Bias, Creativity, Crowdsourcing, Narrative Intelligence, Narrative Systems, Story Generation, Visual Storytelling

Research Background and Problem

Background and Research Domain

The authors define "narrative intelligence" as the ability of computer systems to imitate and generate human narratives. In recent years, artificial intelligence (AI) has made progress in visual storytelling (generating stories based on a set of images). However, existing visual story generation systems primarily focus on the objectivity of image descriptions, lacking attention to the creative and diverse narrative processes of humans.

Core Issues and Challenges

1. Complexity of Human Creative Processes: Unlike the traditional "image-to-sentence mapping" model, humans incorporate information beyond visuals—such as emotions, biases, and imagination—when constructing stories.
2. Bias and Ethical Concerns: Narrative processes are influenced by the author's cultural background, linguistic biases, and perspective choices, which may pose ethical risks in real-world applications.
3. Lack of Systematic Dataset Analysis: Although datasets like VIST exist, there has been insufficient in-depth research into the cognitive dynamics and semantic layers underlying human creativity.

Research Objectives

• Develop an anthology of human-created stories to design more "creative" and generalizable AI narrative generation systems.
• Investigate ethical and bias-related issues in data collection, analysis, and generation models.

Solutions

Methods and Workflow

1. Data Collection: Crowdsourced Visual Storytelling Anthology

   • Experiment Design: Each story is generated based on three sequentially displayed images. Authors complete four structured writing modules (entity recognition, scene description, cross-image narrative, and title creation).
   • Image Sources:
     1. High-quality Flickr images.
     2. Low-resolution robotic images simulating search-and-rescue (SAR) scenarios.
   • Creation Tools: HIT tasks on Amazon Mechanical Turk, requiring participants to complete stories step-by-step without revising previous sections.

2. Qualitative Analysis Methods

   • Close Reading and Narrative Knowledge Engineering from the humanities.
   • Iterative thematic analysis to annotate and extract common patterns across stories.

Innovations

• Step-by-Step Creation Process: By dividing the process into entities, scenes, narratives, and titles, the mapping from images to stories is refined, balancing logic and creativity.
• Improvisational Storytelling: Controlling the sequence of image displays simulates the real-time dynamics of creative storytelling development.
• Diversity and Bias Analysis: Exploring the impact of biases (linguistic, cultural, gender roles, social class) on narrative practices provides a foundation for analyzing and improving narrative generation models.

Research Outcomes

Key Findings: Five Major Themes

1. Information Within the Narrative Horizon vs. Imagination

   • Three narrative approaches in human creation: image-based description (caption), speculation (comment), and fictional deviation from the original content (contrive).

2. Dynamic Roles of Objects/Entities

   • Transforming objects (e.g., books or plants) from "static props" to "emotionally engaging narrative characters."

3. Experiential Information in Scenarios

   • Differences between single-sensory (primarily visual) and multi-sensory (combining smell, touch, and sound) storytelling forms.

4. Emotional Modulation and Narrative Arcs

   • Reinforcing initial emotions (e.g., "from gloom to doom") or reversing emotions (e.g., "from gloom to optimism").

5. Narrative Bias

   • Evident in cultural and linguistic interpretations (e.g., biases in non-native language markers), perspective choices, and specific character portrayals.

Experimental Analysis Results

• The anthology includes 100 diverse stories created by 73 independent authors.
• Thematic analysis proposed five evaluation criteria for AI narrative generation: "creativity, reliability, expressiveness, adequacy, and accountability."

Comparative Advantages Over Existing Research

• Compared to existing visual storytelling datasets (e.g., VIST), this study emphasizes the narrative generation process rather than isolated textual descriptions.
• Introduces deeper semantic understanding of scenes, addressing explicit biases (e.g., gender stereotypes) in AI-generated narratives.

Limitations and Future Work

1. Insufficient Data Diversity: Initial samples focus on highly proficient language users; broader participant groups are needed to reduce bias.
2. Potential for Automated Evaluation: Current evaluation methods rely on qualitative human analysis, necessitating the integration of automated text analysis tools.

Conclusion and Implications

This study proposes a pioneering, human-centered perspective on narrative intelligence, emphasizing the ethical and diversity dimensions of storytelling. Future directions include making the story anthology publicly available and developing narrative AI tools that incorporate public interests (e.g., transparency and bias control). These findings not only advance HCI methodologies but also open up possibilities in interdisciplinary fields such as education, game design, and digital humanities.