DreamGarden: A Designer Assistant for Growing Games from a Single Prompt

Research Background and Problem Statement

• Identified Problems and Challenges: Existing generative AI tools in game design primarily operate on a simple "prompt-generation" model, lacking the capability to effectively handle complex tasks. Designers often need to exert fine-grained control over generated content in a symbolic manner, while purely black-box generation has limitations for complex creative applications. Additionally, these tools struggle under the burden of complex design tasks, leading to frequent iterative cycles that increase user fatigue.

• Significance: The game design process is inherently complex, encompassing asset creation, code development, scene layout, and more. There is a growing demand for AI-assisted tools in this domain. Designing a semi-automated, highly interactive tool could significantly enhance development efficiency, lower entry barriers, and allow designers to retain control over key design decisions.

• Research Motivation: By integrating large language models (LLMs) with multi-submodule design, complex game design tasks can be decomposed into more manageable subtasks. Through recursive and hierarchical planning, the aim is to create a user-friendly design assistant.

Proposed Solution

• Proposed Solution:
  The authors propose a game design assistant system called "DreamGarden," which can automatically generate a comprehensive implementation plan—from game layouts to specific assets and code—based on a simple user prompt. It visualizes the hierarchical planning structure as a "growing garden," allowing users to intervene in the plan through pruning, extending, and providing feedback.

• Innovative Features:

  1. Hierarchical Planning: DreamGarden uses recursive LLM calls to break down simple user prompts into a series of organized, actionable subtasks.
  2. Multi-Submodule Integration: Each subtask is handled by independent submodules, such as a code generation module, asset generation module, and more.
  3. User Interactivity: Through a graphical interface, users can freely adjust the planning hierarchy (e.g., expanding or pruning tasks) and provide real-time feedback to ensure high-quality outputs.
  4. Automated Feedback Handling: The system evaluates intermediate results, such as assets and code, using compiler error logs or visual evaluation models, enabling autonomous iterative improvements.

• Implementation Steps and Key Technologies:

  1. Planning Module:
     • High-level prompts are transformed into an overarching game plan outline (e.g., game background, objectives, major design steps).
     • Using a recursive subplanning model, each design step is further refined into leaf nodes, which correspond to actual implementation tasks.
  2. Implementation Module:
     • Includes several submodules, such as code generation (handling Unreal Engine C++ code), asset generation (based on generative diffusion models), and 3D model retrieval (e.g., CLIP-based search).
     • Generated code and assets are automatically evaluated through engine compilation or visual models.
  3. User Interface:
     • A tree-structured node view presents the planning structure and implementation progress.
     • Users can modify the plan by adjusting node states (e.g., marking leaf nodes as non-leaf nodes or vice versa).
     • Users can validate and interact with intermediate outputs as they are generated.

Research Outcomes

• Specific Results:

  1. DreamGarden can transform open-ended user prompts into complex hierarchical plans and execute a series of modular tasks accordingly.
  2. The system outputs reasonable intermediate results (e.g., 3D assets and code) that can be independently compiled and executed, generating preliminary simulation environments.
  3. Users can effectively intervene in the generation process by organizing and adjusting the planning tree, ultimately achieving results that better meet their needs.

• Comparison with Existing Solutions:

  1. DreamGarden overcomes the limitations of existing black-box models that rely solely on "prompt-generation" by offering an open planning and interaction framework with greater flexibility and transparency.
  2. It autonomously handles feedback (e.g., code compilation errors) and iteratively optimizes results without requiring user intervention at every step.
  3. In generating 3D assets and functional game code, it is more automated and modular than traditional methods.

• Experimental or Evaluation Results:

  • The authors conducted a user study with feedback from 10 participants:
    1. Most users acknowledged the system's value in the early prototyping phase of design.
    2. Some users highlighted the potential of the planning module as a standalone tool, even without relying on the system to generate specific content.
    3. Users particularly appreciated the graphical planning and task visualization interface but suggested improvements in how intermediate results are presented, such as enhancing the intuitiveness of color-coded indicators.
    4. Users noted potential performance challenges in more complex scenarios and emphasized the need to improve generation quality and efficiency across multiple iterations.

• Limitations and Future Directions:

  1. Limitations:
     • Limited support for complex game mechanics.
     • Reliability of generated outputs heavily depends on LLM capabilities.
     • Planning decomposition may result in over-fragmentation or incomplete coverage.
  2. Future Directions:
     • Incorporate multimodal inputs (e.g., user-uploaded sketches or initial materials) to assist in plan generation.
     • Enable real-time interaction and modification of generated content within the Unreal Engine to enhance design flexibility.
     • Optimize the graphical interface, such as more intuitive node annotations and simplifying complex feedback into concise summaries.
     • Improve the performance and interoperability of submodules, such as integrating more efficient 2D-to-3D generation models or text-to-code generation models.
     • Expand applications to broader domains, such as robotics control and automated complex software development.