No Evidence for LLMs Being Useful in Problem Reframing

Research Background and Issues

What problems or challenges did the authors identify?

• Problem Reframing is a challenge that designers must face, requiring them to create new and more effective perspectives to redefine design problems. However, this process is complex and labor-intensive, even for experienced designers.
• The authors propose that generative large language models (LLMs) can serve as tools to assist designers in reframing problems, but there is no clear empirical evidence showing that they significantly improve the quality of problem framing.
• Additional challenges include the skill gap between experienced and inexperienced designers when using LLMs, as well as the potential impact of LLM usage on designers' perceived agency and creative ownership.

Why is this issue important?

• Problem reframing is a critical component of design practice, and ineffective problem framing can lead to inefficient or even useless design solutions. Therefore, improving the quality of problem reframing is essential in the design field.
• The rapid proliferation and adoption of LLMs have not been accompanied by a full understanding of their impact on creative potential, which may lead to misuse or inefficient utilization.
• Investigating whether LLMs benefit designers is significant for advancing human-computer collaboration and enhancing designers' capabilities.

Research Motivation and Related Work

• Current studies suggest that generative AI can support creative activities by providing inspiration or expanding ideas. However, there is limited research on the specific impact of LLMs in problem reframing.
• Related research highlights the potential of LLMs in generating inspiration but also notes issues such as low-quality or overly repetitive content generation.
• The authors propose studying the collaboration between designers and LLMs to fill this knowledge gap and explore whether LLMs can effectively improve the quality of problem reframing.

Solutions

What methods or solutions did the authors propose?

• The authors designed and compared three methods for using LLMs in problem reframing:
  1. Free-form Interaction: Designers interact with LLMs through open-ended dialogue.
  2. Direct Generation: LLMs directly generate problem frames, which designers can use to create solutions.
  3. Structured Process: Based on Kees Dorst's nine-step problem reframing process, intermediate content is generated step-by-step to help designers deeply understand the problem.

What is innovative about this solution?

• The study uniquely compares the effectiveness of free-form interaction, direct generation, and structured methods while evaluating the impact of LLMs on designers' varying levels of expertise and perceived agency.
• It introduces a structured method that integrates Dorst's nine-step problem reframing process with LLMs to support deeper problem analysis by designers.
• The research combines expert evaluations with large-scale experiments to comprehensively assess the quality of problem frames and designers' perceived experiences.

What are the implementation steps and key technologies used?

1. Experiment Design: Participants were randomly assigned to one of four conditions (manual, direct generation, free-form interaction, and structured process) to ensure methodological rigor.
2. Design Problem Generation: Three design problems with appropriate reframing complexity were prepared to allow participants to generate both low-quality and high-quality frames.
3. Interaction Interface Design: User interfaces for the direct and structured methods were designed using LLMs, with content generation triggered by buttons to avoid bias caused by differences in participants' prompting skills.
4. Two-Phase Experimental Process: In the first phase, problem frames were collected; in the second phase, experts evaluated the quality of the frames.
5. Data Analysis: Metrics included novelty and practicality scores for the frames, the Creativity Support Index (CSI) of the LLM, and designers' perceived agency and ownership.

Research Findings

What specific results were achieved?

• Using LLMs did not significantly improve the novelty or practicality of problem frames; the quality of frames generated by LLMs was not significantly different from those created manually.
• The use of LLMs widened the skill gap between experienced designers and inexperienced designers: the former generated more novel problem frames and had higher perceived agency.
• The reliability of free-form interaction and structured methods was comparable, with neither showing significant advantages.
• All methods were rated as providing "good support" in the Creativity Support Index (CSI) by participants.

How does it compare to existing solutions?

• This study provides the first systematic evaluation of the impact of LLMs on complex design tasks like problem reframing.
• It offers a detailed analysis of designers' perceived experiences when interacting with LLMs, paving the way for future development of transparent and reliable human-computer collaboration frameworks.

What were the experimental or evaluation results?

• In terms of novelty and practicality scores, LLM-based methods did not outperform manual methods.
• Expert evaluations revealed that experienced designers using LLMs produced more novel problem frames, while less experienced designers were more susceptible to low-quality frames.
• Designers using the structured method reported increased perceived agency, particularly among design experts.

Limitations and Future Directions

Limitations:

1. The experiment was limited to GPT-4o, which may not fully represent the performance of other LLM models.
2. The research methods did not cover all possible LLM usage scenarios, such as combining free-form interaction with structured processes.
3. Evaluation metrics focused on the novelty and practicality of frames, neglecting other potential values (e.g., operability or consensus-building).

Future Directions:

1. Investigate how designers specifically utilize LLM-generated frames to further optimize interaction methods.
2. Explore the potential of LLMs to impact other quality attributes of problem frames, such as persuasiveness or applicability.
3. Conduct longitudinal studies to observe the impact of LLMs on the full lifecycle of problem reframing in real-world design practices.
4. Expand the research sample to other professional fields to validate the findings' generalizability across different contexts.

Conclusion

This study provides the design field with the first in-depth empirical research on the role of generative AI in problem reframing, clarifying its advantages and limitations. By analyzing various problem reframing methods and designers' perceived experiences, the research highlights both the potential and risks of using LLMs, offering new directions for effectively integrating generative AI into practical applications.