Neural Canvas: Supporting Scenic Design Prototyping by Integrating 3D Sketching and Generative AI

Title of the Paper

Neural Canvas: Supporting Scenic Design Prototyping by Integrating 3D Sketching and Generative AI

Paper Information

• Research Domain: The application of computer-aided design combined with generative AI, focusing on the development of tools for 3D scenic design prototyping.
• Keywords: Generative AI, 3D Sketching, Scenic Design, Prototyping, Visualization Tools, Human-Computer Interaction, Image Processing, User Study

Research Background and Problem

Identified Issues and Challenges

• Scenic design often involves creating 3D environments for media such as film, games, and theater. Due to its conceptual and abstract nature, this process is challenging, requiring significant time, budget, and technical expertise.
• Existing tools for scenic design (e.g., traditional 3D modeling and sketching tools) have limitations, such as:
  • Sketching: While highly flexible, it struggles to accurately convey 3D spatial relationships and material appearances.
  • Traditional 3D software: Suitable for final product modeling but not ideal for rapid prototyping.
  • Physical models: Though realistic, they are difficult to create and lack flexibility.
• The rapid development of generative AI enables the creation of high-quality 2D images from text or sketches, but comprehensive tools integrating 3D sketching with generative AI are still lacking.

Importance

• Scenic design is a critical component for achieving immersive user experiences, enhancing storytelling, atmosphere, and spatial organization. However, the lack of efficient tools limits designers' ability to quickly explore and validate ideas.
• To address the limitations of existing tools, innovative design tools are essential for improving creative efficiency and supporting idea development.

Research Motivation and Related Work

• Related fields have made progress in image modeling and rendering, 3D sketching, and generative AI (e.g., multi-view image generation, multiplane images, and interactive segmentation techniques). However, these approaches still fail to fully meet the demands of scenic design for spatial explicitness, appearance expressiveness, and rapid iteration.
• The authors envision a new design tool that combines generative AI and 3D sketching to provide enhanced creative support.

Solution

Method and Approach Overview

The authors propose Neural Canvas, a lightweight web-based 3D platform that integrates generative AI and 3D sketching, enabling:

• Rapid creation of 3D scenic designs with detailed appearances and spatial relationships.
• Unified functionality for model generation, segmentation, projection, and editing within a single platform, reducing the complexity of switching between multiple tools.
• Four unique projection methods to flexibly integrate generated 2D content into 3D environments.

Innovations

1. Introduction of novel projection methods for scenic design:
   • Stroke Projection: Precisely projects 2D sketches onto 3D planes, embedding spatial information.
   • Image Projection: Applies AI-generated images as textures to specified planes in 3D space.
   • Stroke-Controlled Image Projection: Automatically segments AI-generated images based on user sketches across multiple planes and projects them onto corresponding surfaces.
   • Segmentation-Based Image Projection: Uses deep learning methods to segment images, automatically identifying and projecting them onto 3D environments.
2. Integration of 13 generative AI functionalities (including text-to-image, sketch-to-image, image inpainting, and extension), enhancing creative flexibility.
3. Design of an innovative architecture for tight interaction between 3D sketches and AI tools, improving the quality of 3D sketch-generated content and providing robust viewpoint control.

Implementation Steps and Key Technologies

1. System Architecture Design:
   • Separation of front-end and back-end, with the front-end handling user interaction and 3D rendering, and the back-end running AI models and providing high-performance support.
   • Integration of open-source AI models (e.g., Stable Diffusion, Segment Anything) and cloud services.
2. Generative AI Features:
   • Enhanced user control over generative models using technologies like ControlNet.
   • Flexible input methods (e.g., full-screen rendering, reference backgrounds) to specify the style and layout of AI-generated content.
3. User Interface Design:
   • Multifunctional editing modules, including layer management, brush operations, and image segmentation.
   • Interface support for quick import, editing, and storage of designs.

Research Outcomes

Specific Results

• Developed the Neural Canvas platform, significantly improving the efficiency of early-stage scenic design and enabling users to rapidly explore ideas and refine 3D sketches.
• Validated the platform's effectiveness through user studies involving 12 participants with varying levels of experience in design tasks. Results demonstrated:
  • Intuitive 3D design, rapid generation of visual information, and enhanced creative flexibility.
  • User case studies showcased in videos confirmed that Neural Canvas helps creators transform high-quality sketches into 3D prototypes within hours.

Advantages and Comparisons

• The advantages of Neural Canvas over traditional tools include:
  • Detailed appearance generation and robust spatial explicitness.
  • Simplified operations, higher efficiency compared to traditional 3D modeling tools, and suitability for rapid prototyping.
  • Integration of multiple generative AI models, avoiding the complexity of frequent platform switching.
• Compared to existing innovative tools (e.g., Mental Canvas), Neural Canvas additionally offers:
  • Direct support for multi-view operations of AI-generated content.
  • New projection methods that significantly enhance the efficiency of integrating 2D and 3D content.

Insights from User Studies

• Expert and novice users exhibited different behaviors: experts preferred sketch-based content generation and extensively utilized sketch-to-image functionalities throughout the process, while novices relied more on text input for content generation.
• Using 3D sketches significantly improved the input quality for generative AI models, allowing users to control viewpoints for more expected generation outcomes.

Limitations and Future Directions

• User experience in viewpoint switching and projection operations can be improved, such as by introducing depth estimation algorithms or multi-view assistance.
• Layer management complexity increases with scene complexity, which could be optimized through AI-driven automated naming or integration of physical gesture controls.
• The platform currently supports a limited range of generative AI models. Future integration with richer model search platforms (e.g., Modelverse) could expand stylistic diversity.

Conclusion

Neural Canvas redefines the prototyping process for scenic design by combining 3D sketching and generative AI, significantly lowering technical and creative barriers while providing a reference for future design tool development. The platform will continue to evolve through openness and community support, driving the synergy between generative AI and 3D creation.