Storycaster: An AI System for Immersive Room-based Storytelling

Paper Title

Storycaster: An AI System for Immersive Room-based Storytelling

Publication Info

• Topic area: Generative AI for immersive storytelling in physical spaces
• Keywords: immersive storytelling, generative AI, spatial augmented reality, object editing, room-scale VR, projection mapping, narrator agent, multimodal interaction, personalization, user study

Background and Problem

• Problem / challenge: Existing Cave Automatic Virtual Environment (CAVE) systems and immersive storytelling platforms rely on static, pre-authored content, lack adaptability to users' physical environments, and often require head-mounted displays or complex setups.
• Significance: There is a need for accessible, dynamic storytelling tools that allow everyday users to create personalized narratives in their own physical spaces without technical expertise or external devices.
• Motivation and related work: Prior systems like Room Alive and KidsRoom demonstrated immersive experiences but were limited to predetermined narratives. Advances in generative AI now enable dynamic content creation, but these capabilities have not been fully integrated into room-scale storytelling systems. Storycaster addresses this gap by combining generative AI with spatial augmented reality to create adaptive, user-driven storytelling experiences.

Solution

• Proposed approach: Storycaster, a generative AI-driven CAVE system, transforms physical rooms into interactive storytelling environments through projection mapping, voice-controlled narrative co-creation, and object-level editing.
• Novelty:
  1. Room-Scale Audio-Visual Generation: Combines generative AI techniques (SDXL, Depth ControlNet, cylindrical LoRA) to create projection-mapped visuals and spatial audio tailored to room geometry.
  2. Narrator Agent: Guides users through a three-act narrative structure, responding to voice commands and generating visuals, dialogue, and soundscapes in real-time.
  3. Object Editing: Enables users to transform physical objects into virtual counterparts within the story using Grounded SAM and SD 1.5 inpainting.
  4. User Study: Evaluates Storycaster's impact on immersion, creativity, and personalization through qualitative and quantitative methods.
• Procedure and key techniques:
  1. Users interact with the system via voice commands, initiating a tutorial and then co-creating a three-act story.
  2. Live camera feeds and depth sensors capture room geometry, enabling projection-mapped visuals and object-level transformations.
  3. A narrator agent orchestrates the experience, generating narrative text, visuals, and ambient audio.
  4. Object editing allows users to reimagine physical items as story elements, enhancing personalization.

Results

• Concrete findings:
  • Narrator guidance and spatial audio were the most positively rated features, with 92.3% and 69.3% positive responses, respectively.
  • Object-level editing received mixed feedback, with 38.5% agreeing it enhanced the experience, but 62.5% enjoying personal object integration.
  • Latency for image generation averaged 7.3 seconds, and ambient audio generation took 12 seconds.
• Advantage over baselines:
  • Unlike prior CAVE systems, Storycaster dynamically generates content in response to user input, enabling personalized, co-created narratives.
  • It eliminates the need for head-mounted displays, preserving users' spatial awareness and physical engagement.
• Experiments / evaluation:
  • Conducted with 13 participants (ages ~34.1, diverse backgrounds) in a calibrated projection-mapped room.
  • Mixed-methods analysis revealed high immersion (84.7% positive), ease of use (84.7% positive), and adoption intent (92.3% positive).
  • Participants highlighted the system's ability to foster creativity, confidence, and emotional engagement.
• Limitations and future work:
  • Visual fidelity and latency issues disrupted immersion for some users.
  • Object editing lacked consistency and clarity in some cases.
  • Future work includes improving resolution, reducing latency, enhancing object editing, and exploring portable setups for broader accessibility.

Summary

Storycaster introduces a novel approach to immersive storytelling by integrating generative AI with spatial augmented reality to transform physical rooms into interactive narrative spaces. The system enables users to co-create stories through voice commands, dynamic visuals, spatial audio, and object-level transformations. A user study demonstrated high levels of immersion, creativity, and personalization, though challenges with visual fidelity and latency remain. Storycaster represents a significant step toward accessible, embodied storytelling, with potential applications in education, entertainment, and well-being.