PlantoGraphy: Incorporating Iterative Design Process into Generative Artificial Intelligence for Landscape Rendering

Document Title

PlantoGraphy: Incorporating Iterative Design Process into Generative Artificial Intelligence for Landscape Rendering

Document Information

• Subject Area: Integration of artificial intelligence and landscape design, specifically the application of generative AI in landscape visualization
• Keywords: Landscape rendering, large language models, scene graphs, generative artificial intelligence, iterative design, human-computer interaction, low-rank adaptation, graphical interface, conditional generation, design process

Research Background and Issues

• Identified Challenges and Issues:
  1. Current generative AI primarily employs end-to-end generation methods, lacking the flexibility of iterative design, and fails to adequately support the common design processes of landscape designers.
  2. Existing tools demonstrate insufficient sensitivity in understanding the spatial arrangement of design elements and plant selection, such as inaccurate comprehension of directional vocabulary or plant species.
  3. Current systems lack sufficient interactive functionalities to meet the needs of designers for incremental adjustments during the iterative design process.
• Research Importance: Landscape design is a field that combines art and technology. High-quality rendering-based designs enable stakeholders to better understand and evaluate design proposals, thereby promoting efficient and accurate design.
• Research Motivation and Related Work:
  • Traditional landscape design operations are cumbersome, involving multiple stages from conceptualization to 3D modeling to final rendering, often requiring multiple iterations.
  • Generative AI (e.g., Stable Diffusion, DALL-E) has demonstrated potential in image generation but provides insufficient support for the description and generation of landscape scenes.
  • There is a need to develop a system that supports interactive editing and incremental design optimization.

Solution

• Core Methodology and Framework:
  • A system called PlantoGraphy is proposed, which supports an iterative design process from text to scene layout to detailed landscape rendering.
  • The system adopts a two-stage pipeline:
    1. Specification Module: Transforms the user's design concept (text description) into a specific scene layout, using scene graphs as an intermediary representation between text and layout to enhance the model's understanding of design descriptions.
    2. Illustration Module: Utilizes a custom plant dataset and fine-tunes a diffusion model with Low-Rank Adaptation (LoRA) to generate high-fidelity landscape renderings.
• Innovations:
  • Introduces scene graphs as an intermediate representation to help large language models better understand spatial semantics in landscape descriptions.
  • Provides a graphical interactive interface, allowing designers to express design intentions through text input, graphical editing, and layout adjustments.
  • Employs constrained layouts and instance-level latent composition to handle temporal relationships in generation, improving the consistency of generated results.
• Implementation Steps and Key Techniques:
  • The large language model generates scene graphs (nodes represent plants, edges represent plant relationships).
  • Realistic renderings are generated based on scene layouts and diffusion models.
  • Interactive features (node addition, position adjustments) support iterative design.

Research Outcomes

• Main Results:
  1. Functional Evaluation: PlantoGraphy excels in supporting iterative functionalities and generating consistent results.
  2. Model Improvement: Fine-tuning the model with a small amount of domain-specific data significantly enhances generation quality.
  3. Interactive Design Support: User studies indicate that the system significantly improves efficiency and engagement in the creative process.
• Advantages Compared to Existing Solutions:
  • Compared to traditional AI tools, it provides designers with greater control during the generation process, especially for iterative modifications.
  • Demonstrates superior performance in understanding landscape descriptions and generating accurate layouts (achieved through optimized prompt design and domain knowledge injection).
  • Significantly improves the accuracy of customized plant generation, surpassing the capabilities of general-purpose generative models.
• Experiments and Evaluation:
  1. Specification Module Experiments: Quantitative evaluations on object size allocation, spatial reasoning, and perspective reasoning show that models with domain knowledge injection significantly outperform those without.
  2. Illustration Module Experiments: Multi-generation consistency was evaluated using the Structural Similarity Index (SSIM), proving that layout guidance and LoRA-enhanced models achieve higher performance.
  3. User Studies: In experiments comparing traditional tools, PlantoGraphy improved design efficiency, especially in confirming client requirements or during rapid exploration phases.
• Limitations and Future Directions:
  • The current dataset limits the variety of supported plant species, and accuracy decreases as the complexity of generated objects increases.
  • Fine-tuned LoRA models may lead to conceptual bias (e.g., generated results leaning toward specific styles or species).
  • Support for more complex and diverse requirements remains to be improved.
  • Future plans include enhancing system controllability, supporting more input methods (e.g., hand-drawn sketches), and improving user experience.