Prototyping with Prompts: Emerging Approaches and Challenges in Generative AI Design for Collaborative Software Teams

Research Background and Issues

• Identified Problems or Challenges:
  The paper shifts focus from traditional AI design to generative AI application design, emphasizing the unpredictability of generative models and the complexity of their interaction with users. Multidisciplinary teams need to rapidly iterate using pre-trained models under constraints, but existing tools and processes fail to effectively support the design and prototyping of generative AI.

• Significance of the Problem:
  Generative AI is being widely integrated into various task workflows, such as content creation and interaction design. Designing generative capabilities requires a deep understanding of how teams establish design principles, iteratively develop prompts, and evaluate outcomes. This will impact the efficiency, safety, and user experience of AI products.

• Research Motivation and Related Work:
  Traditional task-specific AI model design follows clear specifications, while generative AI offers greater flexibility but demands new exploration of model behavior and interaction mechanisms. The paper integrates related studies, such as AI experience prototyping and design team collaboration, to investigate the unique needs and challenges of generative AI design.

Solution

• Method or Solution:

  • The authors conducted a design study in collaboration with 39 industry professionals, including UX designers, product managers, and AI engineers, to explore team dynamics and strategies in generative AI prototyping.
  • They proposed a content-centric prototyping process, emphasizing the central role of generated content features in rapid iteration.

• Innovations:

  • The study delves into how teams adopt prompt engineering methods in generative AI design.
  • It demonstrates how teams use ambiguous content and generative models during the design phase to explore and validate requirements.
  • A "structured approach" to prompt prototyping is proposed, including prompt context, system instructions, output constraints, and few-shot examples.

• Implementation Steps and Key Techniques:

  1. Based on design goals, conduct team discussions to clarify success metrics.
  2. Use tools like AI Studio for iterative prompt engineering, exploring different abstraction levels and output controls.
  3. Integrate high-quality example content (Gold Examples) and few-shot techniques to optimize prompt performance.
  4. Validate prototype interaction and output quality by combining UI design with model behavior.
  5. Test model responses to sensitive content by adjusting generation parameters (e.g., temperature randomness) and crafting warning instructions.

Research Outcomes

• Specific Outcomes:

  • Identified content-driven strategies in generative AI prototyping and proposed effective mechanisms from example content to few-shot prompts.
  • Revealed dynamics of role-sharing and blurred boundaries in team collaboration, highlighting how designers and engineers co-iterate on generative AI prompts.
  • Proposed four core components of prompt prototyping: input context, system instructions, output constraints, and few-shot examples.

• Advantages Over Existing Solutions:

  • Efficiently supports multidisciplinary team collaboration and introduces a new structured framework for prompt engineering.
  • Tightly integrates design with generative model behavior through prompt iteration, significantly improving alignment between design and user needs.
  • Encourages collaboration among diverse team roles through shared representations, such as prompts.

• Experimental or Evaluation Results:

  • Teams demonstrated how dynamic iteration in experimental prompt design improves the quality of generated content.
  • Tested model behavior across different abstraction levels, exploring prompt sensitivity and its impact on generated outputs.
  • Case studies reported potential risks in generated content, such as stereotypes or overfitting in outputs.

• Limitations and Future Directions:

  • Limitations include insufficient representation of real-world team dynamics and lack of domain expert participation in the design study.
  • The transparency of generative models remains unresolved, and the predictability of output behavior needs further improvement.
  • Future research could explore enhanced tool support to address more complex task requirements, such as incorporating user researchers to expand the design process's scope.

In summary, the paper provides an in-depth analysis of the challenges in generative AI design and proposes a novel content-feedback-based prompt engineering prototyping method. It offers theoretical and practical guidance on leveraging team roles and the flexibility of generative models. This research contributes to improving the design efficiency of generative AI in practical applications.