CreAItive Collaboration? Users' Misjudgment of AI-Creativity Affects Their Collaborative Performance

Research Background and Issues

• Issues and Challenges:
  The authors investigated the performance of generative AI (e.g., ChatGPT-4) in supporting human collaborative creativity activities, particularly whether it can enhance human performance in creativity tests (e.g., Alternate Uses Test, AUT). The main challenges include: users may misjudge the creative capabilities of AI, thereby affecting collaboration efficiency; long-term use of AI may negatively impact humans' ability to independently complete creative tasks.

• Significance:
  As generative AI technology advances and is widely applied in various fields (e.g., programming, artistic creation), understanding how AI influences human creativity and its potential impact on human skills and learning is crucial. This concerns not only the effective utilization of AI tools but also the long-term understanding of human creativity.

• Research Motivation and Related Work:
  Although existing studies have shown that generative AI can produce high-quality content in certain creative tasks, there is a lack of systematic empirical research on AI's specific role in human-AI collaboration, how users perceive AI's capabilities, and the short- and long-term effects of AI usage. Previous literature also highlights issues such as originality, ownership, and plagiarism in human-AI collaboration.

Solution

• Proposed Methods or Solutions:
  The authors designed an experiment comparing the performance of an AI-supported experimental group and a non-AI-supported control group in collaborative creativity tasks (AUT). They also examined the impact of AI support on subsequent tasks without AI (immediate and delayed tests). Additionally, by analyzing how users interacted with AI during collaboration, the study explored participants' strategies and perceptions of AI.

• Innovations:

  1. The experiment not only investigated short-term task performance but also considered the persistence of learning effects after task completion.
  2. The study proposed that users' misjudgment of AI's creative capabilities might undermine collaboration efficiency and revealed the practical impact of such misjudgment through experimental observations.
  3. It systematically analyzed how users adapt and modify their strategies when collaborating with generative AI.

• Implementation Steps and Key Techniques:

  1. Participants were divided into two groups: the AI-supported group used ChatGPT-4 to complete tasks, while the control group did not use AI.
  2. Tasks were based on the standard creativity test (Alternate Uses Test, AUT), evaluating performance across four dimensions: fluency, flexibility, originality, and elaboration.
  3. Statistical analysis was conducted on experimental data, including performance comparisons across different task stages, changes in user prompt design characteristics, and analysis of the acceptance rate of AI outputs.
  4. Observations and questionnaires were supplemented to understand users' perceptions of AI and their usage strategies.

Research Findings

• Specific Findings:

  1. Task Performance: The AI-supported group showed a slight advantage in the quantity of ideas (fluency) but performed significantly worse in elaboration. Overall, AI support did not significantly improve task performance.
  2. Sustainability of Learning Effects: In immediate tests conducted after removing AI support, the AI group's performance trend persisted, with elaboration still weaker than the control group. These differences disappeared in delayed tests conducted three weeks later.
  3. Misjudgment of AI: Users significantly underestimated the value of AI-generated content, adopting only 40.9% of AI suggestions. This selective behavior contributed to the AI group's potential disadvantage in idea quantity.

• Advantages and Contributions:
  Compared to existing research, this study found:

  • The positive impact of AI on human collaborative creative tasks may be overestimated.
  • Users' misunderstanding of AI capabilities not only affects immediate task performance but may also hinder task learning, though these effects diminish over time.
    This study is the first to explore, from a human-AI collaboration perspective, how users selectively utilize AI outputs and their specific strategies.

• Experimental or Evaluation Results:

  • There were no significant differences between the AI and control groups in certain dimensions (e.g., flexibility and originality), but the AI group showed a significant disadvantage in elaboration.
  • A positive correlation was observed between the proportion of AI output utilized and the quantity of creative ideas (fluency).

• Limitations and Future Directions:

  • Sample bias (computer science students) and the predominantly male demographic limit the generalizability of the results.
  • The study only investigated ChatGPT-4; future research should expand to other generative AI systems.
  • The experimental design restricted direct copying of AI outputs, which may have increased task pressure on users. Further exploration is needed on the effects of copying versus non-copying usage methods.
  • Future research is recommended to conduct longer-term studies to examine the impact of generative AI on task learning transfer.

Conclusion

This study provides a new perspective on the role of generative AI in collaborative creative tasks, revealing potential negative impacts of AI support and the selective usage patterns of humans when interacting with AI outputs. Through an in-depth analysis of task performance, learning effects, and collaboration processes, the study emphasizes the importance of improving understanding of AI capabilities and user skills. It also highlights key directions for future research, such as long-term effects, cross-tool practices, and larger study samples.