AI-Augmented Brainwriting: Investigating the use of LLMs in group ideation

Title of the Paper

AI-Augmented Brainwriting: Investigating the use of LLMs in Group Ideation

Paper Information

• Research Area: Human-Computer Collaboration, Group Ideation, Educational Technology
• Keywords: LLM, Brainwriting, Human-Computer Collaboration, Group Ideation, Educational Technology, Creative Thinking, Large Language Models, Student Design Practices

Research Background and Problem

• Identified Issues or Challenges:

  1. Traditional group brainstorming methods are limited in generating high-quality ideas due to factors such as peer evaluation, free-riding, and production blocking.
  2. How large language models (LLMs), as generative AI tools, can enhance group collaboration efficiency and effectiveness during the "divergent phase" (idea generation) and the "convergent phase" (idea evaluation and selection of the best solutions).

• Significance of the Problem: Creative practices are central to design education and practice. LLMs have the potential to introduce new collaborative methods for group design. For instance, the effective use of generative AI in design education can expand students' creative thinking and enable teams to explore innovative solutions more deeply.

• Research Motivation and Related Work:

  1. Motivation: The widespread adoption of LLMs offers significant inspiration for creative work, necessitating an evaluation of their potential integration into group ideation processes.
  2. Related Work: Existing literature explores how LLMs support interaction design, prototyping, and programming, but systematic studies on integrating LLMs into group ideation processes are still lacking.

Proposed Solution

• Proposed Methods and Solutions: A group brainwriting framework integrated with LLMs, divided into two main phases: the divergent phase for idea generation and the convergent phase for idea evaluation. During the divergent phase, the framework uses LLMs to generate new ideas and expand the group's creative space. During the convergent phase, an LLM-based evaluation tool is designed to score the ideas.

• Innovative Aspects of the Solution:

  • Divergent Phase: Introduces an enhanced brainwriting method that leverages LLMs to generate a broader and deeper range of ideas, expanding the creative space after the initial human-generated ideas.
  • Convergent Phase: Proposes an LLM-driven evaluation engine that assesses ideas based on three dimensions: (1) relevance, (2) novelty, and (3) depth of insight, addressing the efficiency issues in early-stage idea screening.

• Implementation Steps and Technologies Used:

  1. Divergent Phase:
     • Use an online brainwriting tool (Conceptboard) to allow team members to generate initial ideas.
     • Provide LLM (GPT-3) support to generate new ideas through user-AI interaction and integrate them with the initial ideas.
  2. Convergent Phase:
     • Organize team discussions to select the best ideas.
     • Implement a GPT-4 evaluation engine to score ideas based on well-defined criteria and compare the scores with evaluations from human experts and novice evaluators.

Research Findings

• Specific Outcomes:

  1. Developed a group brainwriting framework integrated with LLMs, demonstrating its effectiveness in enhancing idea generation and evaluation during both the divergent and convergent phases.
  2. Experimental results show that LLMs can provide complementary perspectives and generate more specific and innovative ideas.
  3. Demonstrated the feasibility of designing a GPT-4-driven evaluation engine to support idea assessment.

• Advantages Over Existing Solutions:

  • Enhances traditional brainwriting methods with LLMs, reducing bottlenecks in group creativity.
  • Provides a more consistent and less biased early-stage idea screening tool, saving time and optimizing resource allocation.

• Experimental or Evaluation Results:

  1. Divergent Phase: In a user study involving 16 university students, the majority of final project ideas selected by students incorporated LLM-generated suggestions.
  2. Convergent Phase: A comparison of expert and GPT-4 evaluations revealed a moderate positive correlation between LLM scores and evaluations by human experts and novices.
  3. LLM-generated ideas were more specific in detail, while human-generated ideas tended to be more abstract.

• Limitations and Future Directions:

  1. Limitations:
     • The study was limited to novice designers in a university education setting, lacking research on more professional teams or cross-disciplinary contexts.
     • LLMs may exhibit biases due to limitations in training data, necessitating further refinement of tools to filter out potentially biased ideas.
  2. Future Directions:
     • Develop customized LLM interfaces to enhance the ideation process, particularly by improving AI-generated creativity through more advanced prompt engineering techniques.
     • Explore the use of LLMs for focused ideation on multi-domain problems to further enhance the effectiveness of the divergent phase.
     • Conduct long-term studies to observe the impact of integrating LLMs into design education on students' creativity.

Conclusion

This study demonstrates the potential of LLMs in supporting group ideation and evaluation processes, paving the way for future directions in HCI design practices. Through an innovative brainwriting framework and evaluation tool, LLMs not only effectively complement the creative capacity of design teams but also improve the precision of idea screening. These findings lay the foundation for the broader application of generative AI in design while showcasing new ways for teams to collaborate with AI.