Timing Matters: How Using LLMs at Different Timings Influences Writers' Perceptions and Ideation Outcomes in AI-Assisted Ideation

Research Background and Problem

• What problems or challenges did the authors identify?
  The authors investigated two opposing effects of using large language models (LLMs) in creative generation tasks: the expansion of creative thinking and the fixation of creative thinking. It remains unclear how the timing of LLM usage (before or after the start of creative generation) influences human creative outcomes and psychological perceptions.

• Why is this issue important?
  As an assistive tool, LLMs are widely used in various creative tasks. Improper timing of usage may lead to dependency, reduced originality, or suppression of divergent thinking, thereby undermining human creativity. Understanding the optimal timing for LLM usage can help optimize human-AI collaboration, enhance user experience, and support creative tasks.

• Research Motivation and Related Work
  Previous studies have shown that the timing of introducing external examples significantly affects creative outcomes: early exposure may lead to fixation, while later introduction may promote divergent thinking. Additionally, perceived autonomy is closely related to the success of creative tasks. The authors were motivated to experimentally examine the mechanisms by which LLM usage timing affects creative outcomes, autonomy, sense of ownership, and creative self-efficacy.

Solution

• What methods or solutions did the authors propose?
  The authors designed experiments to compare the effects of introducing LLMs early (at the start of creative generation) versus later (after users independently generated ideas). They also studied the impact on user perceptions (e.g., autonomy, sense of ownership, credit attribution) and actual creative outcomes.

• What are the innovative aspects of this solution?

  1. Proposed direct and indirect pathways of LLM usage timing effects, validating the roles of perceived autonomy and sense of ownership as key mediating variables.
  2. Developed an interactive experimental platform to strictly control the timing of participants' access to LLM assistance.
  3. Combined quantitative and qualitative methods to explore the deep mechanisms of human psychological perceptions and LLM-generated thinking.

• What are the implementation steps and key technologies used?

  1. Experimental Design: Two independent conditions (early LLM vs. delayed LLM) were used, with a task to generate helpful ideas on health-related topics.
  2. Participant Allocation: A total of 60 participants were recruited and randomly assigned to the two conditions.
  3. Experimental Procedure: Included pre-task surveys, training (simulating LLM usage for prompt design), the main task, and post-task questionnaires.
  4. Platform Development and Tools:
     • Frontend: Vue.js; Backend: Django.
     • Content generation used OpenAI GPT-4 API, enabling dynamic interaction between participants and AI.
  5. Data Analysis:
     • TF-IDF and cosine similarity were used to measure the similarity between participant-generated ideas and LLM-generated ideas.
     • Structural equation modeling (SEM) was employed to analyze the multiple mediating effects of LLM usage timing on user perceptions and creative outcomes.

Research Findings

• What specific findings were obtained?

  1. Creative Generation Outcomes:
     • Using LLMs after independently generating ideas (delayed LLM group) resulted in more original ideas with lower similarity to LLM-generated ideas.
     • Using LLMs from the start (early LLM group) was more likely to induce creative thinking fixation.
  2. Impact on Psychological Perceptions:
     • The delayed LLM group exhibited significantly higher autonomy, sense of ownership, and creative self-efficacy.
     • The early LLM group showed lower autonomy and tended to attribute more credit to the LLM.
  3. Mediating Mechanisms:
     • Autonomy influenced the sense of ownership, which in turn enhanced creative self-efficacy.
     • Delayed LLM usage indirectly influenced creative outcomes by enhancing autonomy while reducing the influence of LLM-generated content.
  4. User Feedback:
     • Some users reported that early LLM introduction made them feel "lazy" or focus solely on rephrasing prompts.
     • Delayed LLM usage was described as a supportive tool that supplemented and refined independently generated initial ideas.

• What advantages does this solution have compared to existing ones?

  • Provides systematic evidence that the timing of LLM usage significantly affects human creative outcomes, offering new insights into the optimal application timing of AI-generated tools.
  • Goes beyond creative outcomes to deeply analyze changes in user psychological perceptions.

• What were the experimental or evaluation results?

  1. The delayed LLM group produced more independent and original creative outputs.
  2. Differences in the quantity of creative ideas between the two groups approached statistical significance (𝑝=0.063), with the delayed LLM group generating slightly more ideas on average.
  3. SEM analysis confirmed that autonomy and sense of ownership are core driving factors of user behavior patterns.

• Limitations and Future Directions:

  1. The simplified experimental scenario may not fully reflect the role of LLM usage in more complex creative tasks; future work could expand to more diverse tasks.
  2. Participant differences in AI experience and domain knowledge were not thoroughly controlled, which may have potential impacts on creative outcomes and perceptions.
  3. The current study focused on text generation; further validation is needed in other domains (e.g., image generation, music composition).

Through this study, the authors revealed the significant benefits of delayed LLM usage in creative generation, which not only enhances user originality but also strengthens autonomy and confidence. These findings have broad implications for designing more human-centered AI collaboration tools.