Decoupling of Usefulness and Novelty: Evaluating the Impact of Generative AI on Design Outputs and Novice Designers' Creative Thinking

Paper Title

Decoupling of Usefulness and Novelty: Evaluating the Impact of Generative AI on Design Outputs and Novice Designers' Creative Thinking

Publication Info

• Topic area: The impact of Generative AI (GenAI) on design outputs and novice designers' creativity.
• Keywords: Generative AI, creativity, design evaluation, divergent thinking, novelty, usefulness, human-AI collaboration, text-to-image models, design tools, creative thinking.

Background and Problem

• Problem / challenge: While GenAI tools are increasingly used in design, their effects on design outputs and users' creative abilities remain underexplored. Specifically, the relationship between novelty and usefulness in AI-generated designs is unclear.
• Significance: Understanding the impact of GenAI on creativity and design outcomes is crucial as these tools become more integrated into professional workflows, potentially reshaping creative industries.
• Motivation and related work: Prior research highlights creativity's dual components (novelty and usefulness) and the role of creativity support tools (CSTs) in enhancing design processes. However, studies on GenAI's influence on real-world design tasks and its broader implications for creativity are limited.

Solution

• Proposed approach: A two-phase study involving a within-subjects experiment and design evaluation to assess the impact of GenAI on novice designers and their outputs.
• Novelty:
  1. Empirical evidence of a decoupling between novelty and usefulness in GenAI-supported designs.
  2. Analysis of how prior GenAI experience influences users' creative thinking.
  3. Evaluation of GenAI's effects on both design outputs and users' experiences.
  4. Identification of potential drawbacks, such as overcomplexity and misalignment with social norms, in GenAI-generated designs.
• Procedure and key techniques:
  • Study One: 36 participants designed social media ads with and without GenAI support, followed by assessments of mood, self-efficacy, divergent thinking (Alternative Uses Test), and convergent thinking (Anagram Test).
  • Study Two: 105 designs from Study One were evaluated by five clients and 155 online volunteers on creativity, unconventionality, usefulness, visual appeal, and brand alignment.

Results

• Concrete findings:
  • GenAI-supported designs were rated as significantly more creative (clients: β = 0.81, p < .001; volunteers: β = 0.90, p < .001) and unconventional (clients: β = 1.30, p < .001; volunteers: β = 0.66, p < .001).
  • No significant differences in perceived usefulness (clients: β = 0.02, p = .905; volunteers: β = 0.00, p = .995) or brand alignment (clients: β = -0.19, p = .359).
  • Participants new to GenAI showed increased divergent thinking (fluency: p < .05; flexibility: p < .05) after using GenAI, while experienced users showed no such improvement.
• Advantage over baselines:
  • GenAI-supported designs were more visually appealing (volunteers: β = 0.45, p < .01) but not more useful compared to human-only designs.
  • GenAI broadened inclusivity in designs but sometimes introduced socially inappropriate elements.
• Experiments / evaluation:
  • Evaluations were conducted using Likert scales and qualitative feedback from clients and online volunteers.
  • Designs were assessed for creativity, unconventionality, usefulness, visual appeal, and brand alignment.
• Limitations and future work:
  • Reliance on subjective evaluations; real-world metrics like click-through rates were not measured.
  • Use of a single GenAI tool (ChatGPT with DALL-E3) limits generalizability.
  • Focus on novice designers; findings may not apply to experienced professionals.
  • Future research should explore other GenAI tools, real-world design contexts, and longitudinal effects on creativity.

Summary

This study investigates the impact of Generative AI on novice designers and their outputs, revealing a decoupling between novelty and usefulness in GenAI-supported designs. While these designs were rated as more creative and unconventional, they were not perceived as more useful or better aligned with brand identity. Participants new to GenAI experienced short-term gains in divergent thinking, but these effects diminished with prior experience. The findings highlight the need for GenAI tools to balance novelty with practical design goals and suggest future directions for improving human-AI collaboration in creative workflows.