Learning Agent-based Modeling with LLM Companions: Experiences of Novices and Experts Using ChatGPT & NetLogo Chat

Title of the Paper

Learning Agent-based Modeling with LLM Companions: Experiences of Novices and Experts Using ChatGPT & NetLogo Chat

Paper Information

• Research Domain: Human-Computer Interaction, AI-assisted Programming Education, Computer Modeling
• Keywords: Agent-based Modeling, NetLogo Chat, ChatGPT, Programming Assistant, LLM Companion, Learning with LLMs

Research Background and Problems

• Identified Issues or Challenges:

  • Large Language Models (LLMs) with self-coding capabilities could fundamentally change the way computer programming is approached, but their actual impact on learning programming remains underexplored.
  • While many studies have examined LLM performance in general-purpose programming languages (e.g., Python and JavaScript), there is limited research on their application in specialized domains like Agent-based Modeling (ABM).
  • Existing research suggests that experienced programmers benefit more from LLMs, but the reasons behind this phenomenon remain unclear.

• Research Significance:

  • ABM has been widely applied in natural and social sciences, particularly for simulating emergent phenomena in complex systems.
  • NetLogo, as a widely used ABM programming language, bridges scientific research and education, yet its complexity and steep learning curve pose challenges for many users.
  • LLMs could provide more accessible tools for learning and practicing scientific modeling.

• Research Motivation and Related Work:

  • The research team designed a new system (NetLogo Chat) that integrates Constructionist learning theory and best practices in ABM to support learning and practice in NetLogo through an LLM interface.
  • This study aims to understand user perceptions, behaviors, and needs when using LLMs to facilitate ABM learning, addressing gaps in the existing literature.

Solution

• Proposed Method or Solution:

  • Designed and developed an LLM-based system—NetLogo Chat—integrated into the NetLogo IDE to support users in learning and practicing NetLogo.
  • The design of NetLogo Chat includes several core features: providing authoritative documentation support, clarifying user needs, and assisting with code debugging.

• Innovative Aspects:

  • The system aims to address the issue of LLM hallucinations by emphasizing user control and transparency.
  • It incorporates authoritative information based on official NetLogo documentation to enhance LLM performance and build user trust.
  • It creates a highly integrated learning experience by embedding the conversational assistant directly into the NetLogo development environment.

• Implementation Steps and Key Technologies:

  • Optimized interactions with the LLM through Prompt Engineering and implemented the ReAct framework to reduce hallucinations and improve interpretability.
  • Combined official NetLogo documentation and code examples in the experimental version to enhance AI performance, providing authoritative resources for users to consult.
  • Enabled users to run small code snippets in the chat interface, debug problematic code, and quickly create working models.

Research Findings

• Specific Results:

  • NetLogo Chat was designed to support users in incrementally building code and was seamlessly integrated into the NetLogo IDE.
  • Through empirical interviews with 30 experts and novices, differences in user perceptions, behaviors, and needs when collaborating with LLMs were revealed.
  • Proposed a knowledge gap theory to explain the factors influencing human-LLM collaboration efficiency and offered potential design recommendations.

• Advantages Compared to Existing Solutions:

  • NetLogo Chat provides a more personalized learning experience, accommodating users with varying levels of knowledge and learning paths.
  • Its features enhance user control over code debugging and reduce LLM over-assumptions about user needs.
  • Specifically designed for NetLogo and its academic and educational users, it makes ABM language learning and modeling practices more efficient and accessible.

• Experimental or Evaluation Results:

  • Expert users demonstrated a marked tendency to benefit more from LLMs, while novices were more susceptible to the negative effects of AI hallucinations.
  • Experts critically evaluated and adjusted AI-generated code, whereas novices relied more heavily on AI guidance, lacking experience in code reading and debugging.
  • The experiments revealed knowledge gaps among novices in collaborating with LLMs, including insufficient understanding of modeling, basic syntax, and debugging.

• Limitations and Future Directions:

  • The study sample primarily consisted of participants from North America and Europe, with a low proportion of K-12 educators and students, necessitating more representative samples to validate the findings' applicability.
  • Future research could conduct more rigorous quantitative and controlled experiments to further test and expand on the current findings.
  • Iterative design goals for the future include enhancing personalization features, broader integration, and better adaptation to diverse cultural and knowledge backgrounds.