The Impact of Sketch-guided vs. Prompt-guided 3D Generative AIs on the Design Exploration Process

Literature Title

The Impact of Sketch-guided vs. Prompt-guided 3D Generative AIs on the Design Exploration Process

Literature Information

• Domain: Human-Computer Interaction (HCI) and Generative Artificial Intelligence (GenAI) applications in the design exploration process
• Keywords: Generative AI, 3D reconstruction, AI in design processes, design exploration, human-computer interaction, ideation phase, creativity evaluation, self-assessment design methods

Research Background and Problem Statement

• What problems or challenges did the authors identify?

  • Current research primarily focuses on improving the output quality of generative AI, with limited exploration of how different input modalities (e.g., sketches and prompts) impact the design process.
  • Although previous studies have shown that generative AI can rapidly explore the design space, its influence on diversity, creativity, and behavioral differences during the ideation phase has not been systematically analyzed.
  • Existing research lacks a comprehensive understanding of how designers collaborate with sketch-guided or prompt-guided 3D generative systems and how these systems influence design decisions.

• Why is this problem important?

  • Understanding the distinct roles of sketch-guided and prompt-guided generative AI in the design process is crucial for improving tool design, supporting designers' creative thinking, and enhancing design efficiency.
  • Generative AI is becoming an essential tool for designers; systematically analyzing its effects can improve human-AI collaboration and advance generative AI technology.

• Research Motivation and Related Work

  • While there is existing research on 2D image generation and text generation, the specific role of 3D generative models in the design process remains unclear.
  • Workflow analysis and Linkography methods have been used in HCI to understand designer behavior but have not been applied for in-depth analysis in the context of generative AI.

Solution

• What methods or solutions did the authors propose?

  • Developed sketch-guided and prompt-guided 3D generative systems and evaluated their impact on the design process of 12 participants using Linkography and workflow graph analysis.
  • Divided the design process into "ideation phase," "selection phase," and "final design phase," analyzing the effectiveness and limitations of the systems at each stage.

• What is innovative about this solution?

  • The study systematically compared the effects of different input modalities (sketches and text prompts) on the design process, from tool effectiveness to designer behavior, offering forward-looking recommendations for optimizing HCI frameworks.
  • Proposed combining the two modalities to enhance creativity and collaborative efficiency throughout the design process.

• Implementation Steps:

  • Developed and optimized sketch-guided and prompt-guided 3D generative systems based on SALAD and LAS Diffusion models.
  • Designed user experiments to apply these systems, measuring behavioral patterns such as idea diversity and novelty.
  • Used Linkography to analyze the density and entropy of links between design actions (e.g., forward links, backward links) to evaluate creative emotions.
  • Constructed workflow graphs to reveal action sequences and differences in tool usage by designers.

Research Findings

• What specific results were achieved?

  • Sketch-guided systems demonstrated strong support in the later stages of design (selection and refinement phases), aiding convergent thinking and design detailing.
  • Prompt-guided systems were more suitable for the early stages of design (ideation phase), supporting divergent thinking and broad creative exploration.
  • Designer behavior showed that sketch-guided systems tended to promote creative decision-making through self-assessment, while prompt-guided systems emphasized generating inspirational references.

• How does it compare to existing solutions?

  • Provided in-depth insights from design behavior analysis to tool optimization recommendations, surpassing the limitations of existing studies that primarily focus on content quality.
  • Clarified the advantages of both generative AI modalities and proposed a feasible integration strategy to support designers in both broad exploration and precise refinement throughout the process.

• Experimental or Evaluation Results:

  • Linkography analysis showed an average link index of 8.497 and entropy value of 76.002 for the sketch-guided system; the prompt-guided system had a link index of 4.021 and entropy value of 59.301.
  • Workflow graphs revealed dense and directed nodes for the sketch-guided system, while the prompt-guided system exhibited more dispersed nodes, representing diversity in the design process.
  • User surveys and in-depth interviews indicated that designers generally found the sketch-guided system more effective in later stages, while the prompt-guided system enhanced inspiration generation in early stages.

• Limitations and Future Directions

  • Limitations:
    • The experimental environment was a laboratory setting rather than real-world scenarios, which may lead to discrepancies with professional design practices.
    • Some designers experienced difficulties when generative AI outputs did not meet expectations, reflecting limitations in the systems' ability to align with user intent.
  • Future Directions:
    • Investigate how to integrate sketch and prompt modalities to dynamically support different stages of the design process.
    • Explore model optimization techniques based on user feedback (e.g., RLHF) for generative AI applications.
    • Expand the scope of research to real-world design project environments to validate the practical effectiveness of generative AI.