DeepTreeSketch: Neural Graph Prediction for Faithful 3D Tree Modeling from Sketches

Document Title

DeepTreeSketch: Neural Graph Prediction for Faithful 3D Tree Modeling from Sketches

Document Information

• Research Area: Human-Computer Interaction (HCI), Computer Graphics (CG)
• Keywords: 3D modeling interface, sketch-based system, neural networks, creative support, plant modeling, deep learning, graph generation networks, user interaction, 3D tree reconstruction, procedural generation

Research Background and Problem Statement

• Identified Problems or Challenges:

  1. Manually creating 3D tree models with specified structures is time-consuming and complex, especially for novice users.
  2. Current sketch-based modeling methods fail to accurately capture the complex branching structures and diversity of tree species.
  3. Users lack interactive suggestion support when drawing trees, often requiring extensive manual adjustments, leading to tedious interaction experiences.

• Significance of the Problem:
  High-quality 3D tree models are crucial for visual effects in natural scenes such as games, movies, and landscape design. However, existing 3D modeling tools are complex and have high entry barriers, necessitating simple, efficient methods to meet user demands for customized tree models.

• Research Motivation and Related Work:
  Inspired by learning-based 3D modeling technologies, this study proposes leveraging deep neural networks to learn the structural expressiveness of tree species, addressing the challenges of manual model creation. Existing methods, such as Markov random fields, rule-based procedural generation, and graph convolution networks, exhibit certain capabilities but remain insufficient in terms of interactivity and accuracy.

Solution

• Method or Solution:
  A novel AI-assisted sketch system, DeepTreeSketch, is proposed to convert simple 2D hand-drawn sketches into realistic and detail-rich 3D tree models. The solution consists of the following two components:

  1. Tree Graph Prediction Network (TGP-Net): Simulates the growth process of real plants, predicts 3D tree branching structures, and achieves sketch-to-3D structure conversion by learning latent representations from large-scale 3D tree model datasets.
  2. Procedural Leaf and Fine Branch Generation Module: Utilizes space occupation algorithms to generate detailed leaves by extending branches, enhancing the complexity and visual details of trees.

• Innovative Contributions:

  1. Introduces the first neural network combining graph learning with real tree growth simulation to directly infer 3D tree structures.
  2. Provides precise and coarse control drawing modes (branch strokes and leaf strokes), achieving a balance between user interaction and automatic model generation.
  3. Adds AI-assisted suggestion functionality, offering visualized next-step drawing guidance for novice users, significantly reducing operational burden.

• Implementation Steps and Key Technologies:

  1. TGP-Net Architecture: Employs graph convolution networks (GCN) and multilayer perceptrons (MLP) to iteratively predict node positions and progressively construct 3D tree skeletons.
  2. Space Occupation Algorithm: Simulates competitive spatial growth of tree branches to generate fine branches and leaves that conform to the given morphology.
  3. Interactive Interface Design: Includes a sketch drawing panel, parameter control module, and next-step suggestion functionality to support convenient user interaction.

Research Outcomes

• Specific Outcomes:

  1. Developed the DeepTreeSketch system, capable of generating realistic 3D tree models in real-time from simple 2D sketches.
  2. Implemented an intuitive user interaction interface that enables precise-to-coarse tree design through branch strokes and leaf strokes.
  3. Provided assisted suggestion functionality, significantly improving the efficiency and accuracy of novice users' drawing processes.

• Comparative Advantages:
  Compared to existing methods, DeepTreeSketch excels in precise control of tree morphology and reducing user interaction burdens. Unlike probabilistic or rule-based generation algorithms, this system is more flexible, highly automated, and supports artistic tree design.

• Experimental and Evaluation Results:

  1. TGP-Net significantly outperformed traditional GCN and ResNet-50 baseline methods in predicting 3D tree structures, as evidenced by lower Hausdorff distance and Chamfer distance metrics.
  2. User study results indicated the system is easy to learn and use, with creative support ratings significantly higher than comparison systems.
  3. During tree sketching, the AI-assisted functionality reduced average drawing time by nearly 50% and improved the rationality and balance of the created models.

• Limitations and Future Directions:

  1. The current system only supports static 3D tree models; future plans include extending to dynamic simulations such as wind animations and seasonal changes.
  2. The system may struggle to accurately infer tree species information from incomplete sketches or potentially generate locally uneven tree structures.
  3. Plans to incorporate modeling support for more plant types (e.g., flowers) and further explore applications for reconstructing tree models from real photos or point cloud data.

Appendix

• The tree model dataset was generated using methods such as parameterized L-System algorithms, multi-view image reconstruction, and point cloud reconstruction, covering 12 common tree species.
• The average time to generate a complete 3D tree model from a sketch is approximately 1 second.