How AI Processing Delays Foster Creativity: Exploring Research Question Co-Creation with an LLM-based Agent

Document Title

CoQuest: Exploring Research Question Co-Creation with an LLM-based Agent

Document Information

• Domain: System design and application of large language model (LLM)-based systems in Human-Computer Interaction (HCI)
• Keywords: Scientific discovery, large language models (LLMs), co-creation systems, mixed-initiative design, human-AI collaboration

Research Background and Problem

• Problem and Challenges: Developing innovative research questions requires extensive literature review, especially in interdisciplinary fields. This process is often time-consuming and involves iterative refinement of initial ideas. However, current research methods face issues such as inefficiency and limited creativity.
• Importance of Research: Generating research questions is a critical step in academic creation, helping researchers explore new directions and improve the quality of academic outcomes. While using large language models to generate content, challenges such as generation quality (e.g., hallucinated content and reliability) arise, necessitating the integration of researchers' knowledge with model capabilities.
• Research Motivation: Existing studies suggest that small-scale language models have potential in generating new research questions, but there is a lack of empirical research to understand how researchers evaluate AI-generated questions. Addressing these issues holds theoretical and practical significance, advancing the development of human-AI collaborative creative tools.

Solution

Methodology and System Design

• System Overview: A large language model-based AI system, CoQuest, is proposed to support users in co-creating research questions (RQ) with AI. The system consists of three core panels:

  1. RQ Flow Editor: Assists users in generating, providing feedback, and editing research questions.
  2. Paper Graph Visualizer: Displays the literature network related to research questions.
  3. AI Thoughts: Provides logical explanations for the generated research questions.

• Interaction Design:

  • Offers two interaction modes to control AI initiative:
    1. Breadth-first generation: Generates multiple topic-related research questions for parallel exploration.
    2. Depth-first generation: Sequentially generates a series of related questions to deeply explore a specific direction.

• Technical Architecture:

  • The system is built on the AutoGPT framework, organizing generation logic through Chain-of-Thought prompting techniques.
  • The literature retrieval module utilizes a publicly available HCI domain paper dataset, combining semantic embedding and edge diversity optimization methods for relevant paper recommendations.

• Implementation Steps:

  1. Users provide initial research ideas in the editor.
  2. AI generates a series of research questions based on preset logic, along with reasoning for the generation.
  3. Users explore potential research directions through feedback and iteration.

Research Outcomes

• Experimental Design:

  • Twenty HCI researchers were invited to evaluate the system design through two experimental tasks, with themes "Applications of AR/VR in Education" and "AI and Crowdsourcing Research."
  • Both breadth-first and depth-first conditions were provided, employing a balanced experimental design to control for task order effects on results.

• Key Findings:

  1. User Experience: Users generally preferred the breadth-first mode, considering it more creative and trustworthy.
  2. Generated Outcomes: Research questions generated in depth-first mode were rated as more novel and in-depth, but user experience was less favorable.
  3. User Behavior: During generation wait times, users tended to explore existing research questions and provide feedback. Intentionally extending AI generation time stimulated user reflection and multitasking exploration.
  4. Personalized Needs: Users' academic background and task familiarity significantly influenced their evaluation of system results and interaction preferences. For instance, users familiar with the task focused more on technical details of system outputs.

• Advantages:

  • Provides a model for integrating LLMs with human feedback in research co-creation.
  • System design enhances user engagement while fostering research creativity.

• Limitations and Future Directions:

  1. Limited dataset: The system relies on a fixed literature repository, making it difficult to cover a wide range of academic fields. Future work could integrate larger-scale online databases.
  2. Long-term impact: The study did not explore the long-term effects of system usage on users' research habits and creative thinking.
  3. Insufficient personalization: Future work should further investigate users' personalized needs and optimize system outputs with more adaptive dynamic designs.

Conclusion

This study proposes and validates a novel human-AI co-creation system for generating research questions through theoretical modeling and practical evaluation. It provides foundational design principles and empirical references for exploring generative AI tools in academic domains. The research also highlights potential issues such as academic ethics, bias, and AI dependency, pointing toward the development of more ethically responsible systems.