Productive vs. Reflective: How Different Ways of Integrating AI into Design Workflows Affect Cognition and Motivation

Research Background and Problem

• What problems or challenges did the authors identify?
  In human-computer collaboration during the design process, the various roles of artificial intelligence (AI) can significantly impact individual cognitive processes, motivation, and creativity. However, research on these effects remains limited. Additionally, AI-generated content often leads to idea convergence among users or may diminish users' creativity and sense of control over the process, resulting in solutions that lack diversity or depth. Existing studies have not sufficiently explored the specific effects of AI's role in design thinking on users' cognitive load, creativity, and sense of agency.

• Why is this problem important?
  AI has the potential to significantly enhance users' capabilities in design and creative tasks, especially for beginners. Properly positioning AI's role can optimize the creative process. Exploring AI's role not only promotes productivity but also enhances reflective thinking and user agency, which is crucial for the development of future design tools in the era of AI.

• Research Motivation and Related Work
  This study is inspired by existing research that explores how AI can stimulate users' creative thinking or foster self-reflection by generating inspirational content. At the same time, some studies have pointed out the potential for AI-generated content to cause convergence effects or negatively impact users' sense of agency. This study aims to fill the gap in the literature by investigating the multidimensional effects of AI's specific positioning in the design process on creativity.

Solution

• What methods or solutions did the authors propose?
  The authors developed a FigJam plugin integrated with a large language model (LLM) that supports three modes of AI participation: No-AI mode (no AI support), Co-led mode (step-by-step AI support through Q&A), and AI-led mode (one-time generation of complete AI support). By embedding these modes into two common design templates ("Five Whys" problem analysis and "Competitive Analysis"), the authors studied how different AI roles affect users' cognitive processes and creative outputs.

• What is innovative about this solution?

  1. The study systematically compared the effects of three different AI role assignments (No-AI, Co-led, and AI-led) on users' cognitive distribution, creative outcomes, and subjective perceptions of AI's value.
  2. It combined reflective questions with AI-generated content, dynamically generating subsequent support based on each user's input to guide deeper thinking.
  3. It explored how AI can act not only as an information generator but also as a trigger for reflection and critical thinking (e.g., through generating multidimensional competitive analyses).

• What are the implementation steps and key technologies used?

  1. Created an interactive FigJam-based plugin integrated with a large language model (GPT-4), supporting the "Five Whys" and "Competitive Analysis" design templates.
  2. Conducted a controlled experiment with 47 university students, randomly assigning participants to one of the three modes to complete design-related tasks.
  3. Evaluated the effects of AI positioning on participants' creative outputs, cognitive resource allocation, and subjective experiences through video recordings, quantitative analysis (e.g., counting idea units), and qualitative analysis (including interviews and questionnaires).

Research Outcomes

• What specific outcomes were achieved?

  1. Creative Outcomes: AI support (especially AI-led) expanded problem framing and promoted solution diversity. The AI-led group demonstrated significantly higher thematic diversity in their final solutions compared to other groups, but users in the No-AI and Co-led groups exhibited greater confidence in their initial solutions.
  2. Cognitive Activity Distribution:
     • The AI-led group spent more time on information comprehension and synthesis, while the No-AI and Co-led groups spent more time filling in and modifying content.
     • AI support reduced users' preoccupation with sentence formulation, allowing more time for deeper content reflection.
  3. User Perception: Participants in the Co-led group demonstrated deeper thinking about the problem and perceived their solutions as more practical. Users in the No-AI and Co-led groups reported a stronger sense of ownership and trust in the process, while AI-led users felt their solutions were more thematically diverse.

• What advantages does it have compared to existing solutions?
  This study is the first to systematically compare the multifaceted effects of different AI role assignments on design tasks, aiming to identify the optimal balance between enhancing creativity and preserving user agency. Additionally, by designing AI to generate reflective questions, the study goes beyond the limitations of existing research that focuses solely on AI-generated content.

• What were the experimental or evaluation results?

  • The Co-led mode significantly fostered reflective thinking and gave users greater confidence and satisfaction in problem-solving.
  • The AI-led mode enhanced thematic divergence but reduced user autonomy.
  • The No-AI mode, while more time-consuming, adhered more closely to conventional design processes and encouraged spontaneous exploration.

• Limitations and Future Directions

  1. The participants were all design novices. Future research could involve a broader range of participants, including professional designers, to validate the generalizability of the findings.
  2. The current AI-generated content was based on predefined templates, which might limit users' freedom to explore. Future studies could investigate tools that allow users to customize templates and AI queries.
  3. Further exploration is needed into AI support for other types of design tasks (e.g., creative thinking, systems thinking) to enhance cross-task applicability.

Conclusion

This study reveals the complex multidimensional effects of AI's positioning on cognition, behavior, and perception in design tasks. Through an in-depth comparison of the No-AI, Co-led, and AI-led modes, the research provides valuable insights into balancing the enhancement of creativity with the preservation of user agency. These findings offer a solid experimental foundation and theoretical framework for the design and optimization of future AI-supported tools.