GenPara: Enhancing the 3D Design Editing Process by Inferring Users' Regions of Interest with Text-Conditional Shape Parameters

Research Background and Problem

• Problem or Challenge: In 3D design, specifying design goals and visualizing complex shapes solely through text poses significant challenges. This complexity arises from the spatial and structural intricacies of 3D design, making textual descriptions difficult to standardize and prone to subjective interpretation. Although 3D generative artificial intelligence (GenAI) has made progress in component assembly and high-quality 3D design generation, it still lacks robust support for dynamic generation and editing of shape parameters. Furthermore, current GenAI systems often overlook the iterative and nuanced characteristics essential for design exploration and refinement.
• Significance: In fields like 3D design, which demand precise visual expression, the inability to effectively link abstract goals with concrete implementations can significantly constrain designers' creativity and efficiency. Systems that support the generation of shape parameters and interactive engagement with the design space have the potential to enhance creativity, efficiency, and comprehension in design exploration.
• Research Motivation and Related Work: Existing literature focuses on the decomposition and manipulation of 3D shapes, as well as design generation or optimization through GenAI, but fails to adequately support iterative and nuanced exploration processes. Building on this foundation, this study proposes improving the efficiency and depth of design exploration by generating shape parameters conditioned on rich textual descriptions.

Solution

• Method and Solution: We propose GenPara, an interactive 3D design editing system that supports user design through shape parameters conditioned on rich text descriptions and component-oriented inference methods (e.g., Bayesian inference). The system also introduces an Exploration Map (to visualize the design space) and a Design Versioning Tree (to capture the hierarchical structure of design changes).
• Innovations: The system integrates the following innovations:
  • Utilizing fine-tuned large language models (LLMs) to parse complex 3D design shape parameters and align them with textual descriptions.
  • Employing Bayesian inference to identify Regions of Interest (ROI) and generate design alternatives consistent with user goals.
  • Providing visualization tools (Exploration Map and Design Versioning Tree) to enable designers to efficiently interact with, explore, and track changes in complex shape parameters.
• Implementation Steps and Techniques:
  1. Implement 3D component decomposition and generation using the SALAD and SPAGHETTI models.
  2. Use UMAP for dimensionality reduction to visualize the spatial distribution of design parameters, optimizing user ROIs through Bayesian inference.
  3. Generate shape parameters conditioned on textual descriptions via LLMs and further refine target designs.
  4. Provide a user interface with text input fields, an exploration map, and a design versioning tree to support interactivity and summarize design changes.

Research Outcomes

• Specific Outcomes:
  • Trained LLMs to extract complex shape parameters from textual descriptions and generate alternative design shapes.
  • Developed an interactive system to help designers understand and utilize shape parameters to express design goals.
  • Achieved significant improvements in task concretization and shape exploration.
• Comparative Advantages:
  • Compared to traditional text-based generation systems (e.g., chatbots or image generation models), GenPara more effectively helps designers understand and iteratively refine design shapes through a more intuitive, visualized, and structured approach.
  • The design versioning tree enables global tracking of the design process, systematically organizing iterations and changes.
• Experimental or Evaluation Results:
  • A user study (N=16) showed that compared to baseline systems, GenPara reduced the difficulty of user prompts and enhanced the visualization of relationships between text and shape parameters.
  • Using the NASA-TLX workload scale and Creativity Support Index, users reported higher design satisfaction, less frustration, and greater creativity support.
  • Interaction time with the system significantly increased, while the number of prompt generations decreased, indicating that users spent more time deeply exploring and concretizing designs.
• Limitations and Future Directions:
  • Currently limited to generating 3D models in specific categories such as chairs. Future research should expand to other domains (e.g., automobiles, architecture) and improve LLM adaptability to complex, multi-category designs.
  • The Gaussian blob representation method is limited in tasks requiring highly irregular or fine-tuned adjustments. Future work could explore better integration with semantic labels.
  • There is room for improvement in supporting collaborative design, such as multi-designer interactions or multi-attribute integration.

By combining controllable shape parameter generation with comprehensive visualization tools, GenPara provides an innovative approach for 3D designers, while also paving the way for new applications of 3D GenAI in the design process.