PlayWrite: A Multimodal System for AI Supported Narrative Co-Authoring Through Play in XR

Paper Title

PlayWrite: A Multimodal System for AI Supported Narrative Co-Authoring Through Play in XR

Publication Info

• Topic area: Mixed-reality (XR) systems for AI-assisted storytelling.
• Keywords: AI storytelling, mixed reality, co-creative systems, narrative generation, embodied interaction, play-based authoring, multimodal input, improvisation, Intent Frames, story assembly.

Background and Problem

• Problem / challenge: Existing AI writing tools rely heavily on text prompts, which poorly support the spatial, interactive, and improvisational nature of storytelling. Current systems lack mechanisms to integrate embodied interaction and iterative editing of AI-generated narratives.
• Significance: Addressing this gap enables more intuitive, playful, and expressive storytelling processes, particularly for creators who think visually and spatially, such as hobbyists, educators, and performers.
• Motivation and related work: Prior work has explored text-based narrative generation and visual authoring tools but has largely ignored embodied interaction and play as central modalities for storytelling. Systems like Toyteller and Narratron have shown the potential of symbolic play and multimodal inputs but remain limited in contextualizing space and interaction.

Solution

• Proposed approach: PlayWrite, a mixed-reality system where users create stories by manipulating virtual characters and props, staging scenes, and engaging in dialogue. The system interprets these interactions into structured narrative units (Intent Frames) and visualizes them as Story Marbles for assembly and refinement.
• Novelty:
  1. A mixed-reality system that centers storytelling in play, direct manipulation, and speech interaction.
  2. Introduction of Intent Frames, abstractions that translate user actions into semantic, AI-readable narrative elements.
  3. A multi-agent architecture (Environment, Social, Narrator, and Intent Frame Agents) for interpreting embodied performance and generating coherent narratives.
• Procedure and key techniques:
  1. Users interact with characters and props in an XR environment through gestures, movement, and speech.
  2. A multi-agent pipeline observes and interprets these interactions into Intent Frames.
  3. Intent Frames are visualized as Story Marbles, which users can rearrange on a timeline to refine the narrative.
  4. The system generates a story summary or screenplay based on the assembled marbles.

Results

• Concrete findings:
  • High ratings for expressiveness (M=6.35) and enjoyment (M=6.15) in the Creative Support Index (CSI).
  • Participants valued the system’s ability to foster improvisation and playfulness, with strong agreement that it encouraged experimentation (M=6.46).
  • Mixed ratings for control (M=3.92), reflecting challenges in achieving precise narrative direction.
• Advantage over baselines:
  • PlayWrite enables embodied, multimodal storytelling, moving beyond text-based systems by integrating spatial manipulation and real-time dialogue.
  • Participants reported that the system supported divergent thinking, reduced creative blocks, and encouraged improvisation.
• Experiments / evaluation:
  • A user study with 13 participants (N=13) from diverse writing backgrounds, involving closed-ended and open-ended storytelling tasks in XR.
  • Participants interacted with pre-authored scenes (e.g., Aladdin, Robin Hood), reviewed and rearranged Story Marbles, and evaluated the system through questionnaires and interviews.
  • Quantitative and qualitative data were collected, including CSI scores, post-task surveys, and thematic analysis of participant feedback.
• Limitations and future work:
  • Limited support for precise narrative control and global coherence in story assembly.
  • Study tasks used familiar story worlds, potentially biasing AI responses toward canonical behaviors.
  • Future work could explore customizable assets, fine-tuned AI models, and adaptive initiative levels for different user preferences.

Summary

PlayWrite introduces a novel approach to AI-assisted storytelling by centering the creative process in play, direct manipulation, and multimodal interaction within a mixed-reality environment. The system’s multi-agent pipeline interprets user actions into structured narrative units (Intent Frames), which are visualized as Story Marbles for assembly and refinement. A user study demonstrated that PlayWrite fosters an enjoyable, expressive, and improvisational storytelling experience, though challenges remain in balancing playful exploration with precise narrative control. The work validates play as a design lens for co-creative systems, offering new pathways for intuitive and collaborative human-AI creativity.