CollageVis: Rapid Previsualization Tool for Indie Filmmaking using Video Collages

Title of the Paper

CollageVis: Rapid Previsualization Tool for Indie Filmmaking using Video Collages

Paper Information

• Domain: Human-Computer Interaction (HCI), filmmaking previsualization tools
• Keywords: previsualization, storyboard, indie filmmaking, video collage, virtual camera, real-time video composition

Research Background and Problem Statement

• Problems and Challenges:

  • Traditional previsualization methods in filmmaking, such as 2D sketch storyboards or 3D animations, are time-consuming and labor-intensive, requiring significant technical expertise and resources, which can be challenging for indie filmmakers.
  • Automated text-to-image/animation previsualization tools are efficient but rely on preset datasets and lack creative freedom.
  • Virtual reality (VR) approaches offer immersion but are costly and have a steep learning curve for non-technical users.
  • Simple and quick test video methods face challenges such as difficulties in coordinating personnel/time, weather constraints during on-site shooting, and other physical limitations.

• Research Motivation:

  • Indie filmmakers often prefer rough test videos but desire a tool that can overcome physical and editing constraints, enabling rapid, flexible, and intuitive scene previsualization.

Solution

• Method or Solution:

  • A real-time video collage tool named CollageVis is proposed, featuring two user interfaces:
    1. Collage Board: Used for actor segmentation, video background replacement, and character tagging (e.g., name labels, color filters, face swapping, and voice modification).
    2. Virtual Stage: Video layers can be moved within a 2.5D virtual space, allowing users to use mobile devices to simulate virtual camera shots for scene layout and shot planning.

• Innovations:

  • Employing video collage technology to replace traditional 2D drawing and 3D animation, lowering the technical and financial barriers for previsualization.
  • Introducing a virtual camera proxy mechanism, enabling users to simulate various camera movements (e.g., handheld, dolly shots) using mobile devices.
  • Supporting dynamic scene changes (time, weather, lighting settings, etc.).
  • Providing 2.5D spatial layout design, facilitating multi-angle view planning of shooting details for film teams.

• Implementation Steps and Techniques:

  1. Using MediaPipe's image segmentation model to extract actor foregrounds in real-time and generate transparent background video layers.
  2. Offering real-time face-swapping functionality (based on SimSwap technology) and utilizing Google Cloud API for speech-to-text conversion.
  3. Using mobile devices as virtual cameras, simulating camera movement through pose sensor data.
  4. Converting video layers and scenes into a virtual environment, supporting layout design and shot capture via mouse or mobile devices.

Research Outcomes

• Specific Results:

  • CollageVis was used to simulate seven classic movie scenes, demonstrating its application capabilities in actor simulation, scene environment adjustments, and shot movement design.
  • User studies revealed that film directors widely appreciated the tool's flexibility and rapid generation capabilities, finding it effective in enhancing collaboration and creative efficiency during previsualization.

• Experimental or Evaluation Results:

  • System usability testing (SUS) scored 80.41/100, with users expressing high approval of its learning curve and user experience.
  • NASA-TLX task load score averaged 16.81, reflecting the tool's ability to reduce cognitive and operational burden during the previsualization process.
  • Users highlighted CollageVis's advantages in saving time, supporting visual expression, and reducing trial-and-error costs.

• Comparative Advantages over Existing Solutions:

  • Compared to 2D drawing tools: Simpler operations, no drawing skills required, intuitive video generation with strong real-time capabilities.
  • Compared to 3D animation tools: Overcomes the barriers of complex modeling and animation techniques, eliminating the need for large-scale equipment like VR devices.
  • Enables quick and flexible modifications, supporting instant camera movements and scene environment switches.

• Limitations and Future Directions:

  • Limitations:
    • In the 2.5D environment, video layers cannot perfectly represent actor postures from different camera angles, requiring multi-angle shooting.
    • Modifications to actors and props require re-recording videos, as adjustments cannot be made directly to existing footage.
  • Future Work:
    • Enhance input diversity, including support for static images, hand-drawn sketches, and rapid integration of 3D models.
    • Improve edge quality in video segmentation.
    • Explore the possibility of converting 2D videos into 3D animations quickly.
    • Extend applications to other creative fields such as music videos and choreography.