PrevizWhiz: Combining Rough 3D Scenes and 2D Video to Guide Generative Video Previsualization

Paper Title

PrevizWhiz: Combining Rough 3D Scenes and 2D Video to Guide Generative Video Previsualization

Publication Info

• Topic area: AI-assisted previsualization for filmmaking
• Keywords: Previsualization, generative AI, filmmaking, 3D scene blocking, video stylization, motion control, creative workflows, generative video models, AI in pre-production, cinematic tools

Background and Problem

• Problem / challenge: Existing previsualization tools force filmmakers to choose between speed, fidelity, and control. Storyboards lack spatial and temporal precision, while 3D tools require expertise and high-fidelity assets. Generative AI models, while promising, struggle with temporal consistency and spatial grounding.
• Significance: Addressing these limitations can accelerate creative iteration, lower technical barriers, and improve communication among filmmaking teams, especially for resource-constrained productions.
• Motivation and related work: Prior tools like storyboards, 3D previz software, and generative AI models have limitations in fidelity, accessibility, and control. Recent advances in generative AI for style transfer and video generation offer new opportunities but lack integration into lightweight, structured filmmaking workflows.

Solution

• Proposed approach: PrevizWhiz—a system that combines rough 3D scene blocking, generative AI-based frame stylization, and detailed motion control using 2D video references to create flexible, expressive previsualizations.
• Novelty:
  1. Integration of rough 3D environments with generative video models for lightweight previsualization.
  2. Multi-level motion fidelity control, from coarse 3D blocking to fine-grained motion using external video references.
  3. Adjustable resemblance levels for balancing adherence to 3D scaffolds with creative freedom in generative outputs.
  4. A unified workflow supporting rapid iteration and cross-disciplinary collaboration.
• Procedure and key techniques:
  • Scene setup with rough 3D blocking for spatial and temporal structure.
  • Frame stylization using generative AI with adjustable resemblance levels (Strict, Faithful, Flexible, Loose).
  • Motion control through 3D blocking, stylized animations, and external video references.
  • Video generation using multimodal inputs (e.g., skeletons, depth maps) and tools like FlowEdit and ControlNet.

Results

• Concrete findings:
  • Participants rated the system highly for usability (System Usability Scale median = 4/5 for most features).
  • Generated outputs were seen as professional and effective for communication, though alignment with user intent was mixed (median = 3/5 for matching imagination).
  • Video generation latency was approximately 1 minute per clip.
• Advantage over baselines: Faster and more accessible than traditional 3D tools like Cine Tracer, while offering more structure and control than text-to-video approaches like MidJourney or Runway.
• Experiments / evaluation:
  • Study with 10 participants (filmmakers and 3D experts, 1–15 years of experience).
  • Tasks included creating previsualizations for interior and exterior scenes with varying levels of fidelity and creative control.
  • Data collected through surveys, interviews, and logged outputs.
• Limitations and future work:
  • Challenges with temporal consistency, cross-shot continuity, and fine-grained control in generative outputs.
  • Limited study duration (90–120 minutes) and model latency constrained broader exploration.
  • Future work includes improving model reliability, reducing latency, and addressing ethical concerns around authorship and labor displacement.

Summary

PrevizWhiz introduces a novel approach to previsualization by integrating rough 3D scene blocking, generative AI-based stylization, and motion control using 2D video references. The system enables filmmakers to rapidly iterate on visual ideas, balancing structural precision with creative flexibility. A user study demonstrated its effectiveness for lightweight pre-production, though challenges remain in ensuring alignment with creative intent and addressing ethical implications of AI in filmmaking. PrevizWhiz has the potential to lower barriers for independent creators and enhance collaboration across filmmaking teams.