Development of an LLM-Based Chatbot to Support Learnability in Stardew Valley: A Diary Study Approach

Research Background and Issues

• What problems or challenges did the authors identify?
  As the video game industry becomes more complex and game mechanics grow richer, players face increasing difficulty in learning games. Existing external resources (such as wikis and forums) may lead to information overload or spoilers, prompting players to seek new ways to enhance game learnability. Additionally, in narrative-rich games, helping players understand complex game mechanics while maintaining immersion is a pressing challenge that needs to be addressed.

• Why is this issue important?
  Learnability directly impacts player engagement and gaming experience. Excessive learning difficulty may cause players to lose interest and abandon the game. Therefore, researching how to provide players with immediate information in a natural and immersive way can improve their experience and lower learning barriers.

• Research Motivation and Related Work
  The authors propose integrating large language model (LLM)-based chatbots (such as GPT-4) as in-game assistants to help players quickly access information in complex scenarios while maintaining dialogue tones consistent with the game world. Although existing studies have utilized LLMs for generating quests and NPC dialogues, research on LLMs as learning assistants in narrative-rich role-playing games remains in its early stages.

Solution

• What methods or solutions did the authors propose?
  The authors developed a GPT-4-based game chatbot named "Daisy" and explored its feasibility and impact in the game Stardew Valley through in-depth diary studies and interview surveys.

• What is innovative about this solution?
  "Daisy" is designed as a chatbot with an NPC personality, combining natural dialogue with game context to provide immediate information. Additionally, it adopts a friendly tone and indirect information delivery to avoid spoilers while encouraging players to solve problems independently. This design not only enhances learnability but also improves immersion and emotional engagement.

• What are the implementation steps and key technologies used?

1. Develop the chatbot using the GPT-4 model, set up an NPC personality, and design it to fit the atmosphere of Stardew Valley.
2. Conduct two rounds of testing to refine the chatbot's tone and interaction style.
3. Design the chatbot as a standalone web application that runs alongside the game, enabling seamless access to in-game information.
4. Conduct a three-week diary study user test, collecting gameplay recordings, chat logs, and interview data to evaluate its performance.

Research Findings

• What specific outcomes were achieved?

1. Immediate Information Access: Participants were able to quickly obtain information not provided by the game through natural dialogue, improving game fluidity.
2. Immersion and Emotional Engagement: The chatbot interacted with players using an NPC tone, making them feel embedded in the game environment, thereby enhancing immersion and engagement.
3. Learnability Optimization: Daisy met information needs while encouraging problem-solving, avoiding spoilers, and guiding players to independently discover the fun of the game.

• What advantages does it have compared to existing solutions?
  Compared to external information sources (such as wikis), "Daisy" provides faster immediate information while avoiding information overload and spoilers. Compared to traditional tutorials or in-game hint systems, Daisy enhances integration and enjoyment through natural dialogue.

• What were the experimental or evaluation results?

1. Data: The study collected 247 diary entries, 8,771 chat logs, and 450 hours of gameplay recordings. Daisy's accuracy rate was 81.6% (novice group) and 69.6% (experienced group).
2. Usage Trends: Novice users gradually reduced usage frequency, reflecting a decline in information needs as familiarity increased.
3. Feedback: Most participants expressed satisfaction with Daisy's immediate information and NPC interaction style, especially during the first few weeks, helping them quickly adapt to the game. Experienced players were more sensitive to incorrect answers, emphasizing the importance of accuracy.

• Limitations and Future Directions

1. As an external application, Daisy failed to account for players' specific game states, resulting in limited personalized responses.
2. The primary issue is the inherent "hallucination phenomenon" (incorrect information) of LLMs, which caused significant negative reactions among highly familiar users.
3. Future implementations should incorporate short- and long-term memory functions and real-time integration with player states to improve interaction and information accuracy.
4. Investigating its applicability in multiplayer game environments and addressing potential interference with player communication is recommended.

The research conclusions provide design recommendations for integrating LLM technology into games to enhance learnability, immersion, and player satisfaction. These insights can offer theoretical and practical guidance for developing in-game assistants and companion NPCs in the future.