Private Yet Social: How LLM Chatbots Support and Challenge Eating Disorder Recovery

Research Background and Problem Statement

1. Identified Problems or Challenges:

   • The global population suffering from Eating Disorders (ED) continues to grow, posing significant threats not only to physical health but also to psychological well-being and social life.
   • Managing eating disorders requires long-term support and care. However, the lack of clinical care resources or patients' inability to sustain long-term treatment has led to a shortage of support resources in this field.
   • Chatbots based on large language models (LLMs) are considered a potential solution to fill the gap in providing immediate support in non-clinical settings. However, existing studies suggest that such chatbots may provide misleading or harmful advice due to insufficient understanding of sensitive health issues, such as an excessive focus on weight loss goals.

2. Significance:

   • Managing eating disorders is not only a matter of individual health but also has broader social implications, such as addictive behaviors and high social stigma.
   • Compared to traditional chatbots based on rule-based or early machine learning methods, LLMs offer greater flexibility and complexity, making them a key area for driving innovation in digital health technologies.

3. Research Motivation and Related Work:

   • Currently, there are no LLM-based chatbots specifically designed to meet the needs of individuals with eating disorders.
   • Although LLMs have shown potential value in healthcare applications, they still pose significant safety risks and may provide misleading advice, especially in the domain of mental health.
   • The authors developed a chatbot named "WellnessBot" to explore how patients interact with LLM technology, particularly in managing eating disorders in non-clinical settings.

Solution

1. Approach or Solution:

   • Develop "WellnessBot" as a technological probe to explore the interaction experiences between patients and LLM-based chatbots.
   • Design WellnessBot with specific roles and core functionalities (e.g., emotional support, informational support, and personalized support) to enhance user experience.
   • Conduct a 10-day study involving 26 participants with eating disorders, collecting their chat logs with WellnessBot, questionnaire responses, and interview feedback for comprehensive analysis.

2. Innovative Features:

   • Integrate users' personalized "Wellness Plans" into the LLM chatbot, enabling it to detect user preferences and early crisis signals related to eating disorders, and generate contextually appropriate responses based on users' historical behaviors.
   • Employ a combined approach of "emotional support + informational strategies + personalized memory" in the chatbot design to enable more complex and open-ended conversations.
   • Introduce a "mentor-like role" (as opposed to a professional doctor role) to enhance the chatbot's sense of "social companionship" while avoiding potential medical risks from over-reliance.

3. Implementation Steps and Key Technologies:

   • Design and Implementation: Use the GPT-4 API as the foundation for chatbot dialogue generation, while incorporating user-uploaded information (e.g., wellness plans and user profiles) to tailor the contextual responses.
   • Personalization Settings: Implement short-term and long-term memory structures to enable context tracking and situationally dependent responses during conversations.
   • Data Collection and Monitoring: The research team reviewed the chatbot's interactions with users daily to ensure no major errors or harmful suggestions occurred.
   • Technical Optimization: Combine GPT-3.5 and GPT-4 models to balance user response time with dialogue quality.

Research Outcomes

1. Specific Results:

   • User Engagement: Over the 10-day trial period, 1,477 user messages and 1,668 WellnessBot responses were collected. Most users expressed willingness to use such a tool long-term to support their psychological recovery.
   • User Feedback: Users widely recognized the chatbot as a safe space combining "privacy and social interaction," allowing them to freely express sensitive topics.
   • Psychological Relief: Conversations with the chatbot helped users alleviate anxiety related to eating disorders and gain a greater sense of control over managing their condition.

2. Advantages Over Existing Solutions:

   • More dynamically responsive to users' emotions and personalized needs compared to traditional rule-based chatbots.
   • Provides a private interactive space that combines "personal privacy" and "social support," which is highly scarce in current online support communities or traditional psychological counseling.
   • Capable of detecting and responding to early warning signals pre-set in users' wellness plans, such as dietary triggers and psychological stress.

3. Experimental or Evaluation Results:

   • Users' self-assessed ability to manage their condition significantly improved (treatment control scores) after the experiment, while their overall anxiety about their condition decreased (concern scores).
   • The majority of users (20 participants) found the chatbot to be "helpful" or "very helpful" in their recovery process.

4. Limitations and Future Directions:

   • Limitations:
     • Trust Issues: The experiment revealed that users lacked critical thinking when evaluating the chatbot's responses, making them prone to blind trust.
     • Content Risks: In some scenarios, WellnessBot's emphasis on "low-calorie diets" or "weight loss" could mislead users into reinforcing unhealthy behaviors.
     • Gender and Time Constraints: Almost all participants were female, and the 10-day experiment duration was insufficient to observe the long-term effects of interaction.
   • Future Directions:
     • Explore ways to enhance users' critical thinking through real-time transparency mechanisms and self-reflection prompts during conversations.
     • Integrate with professional clinical care, such as guiding users to seek professional treatment upon detecting critical signals.
     • Develop specialized LLMs tailored for complex mental health scenarios (e.g., eating disorders) to reduce error rates and improve contextual response accuracy.

This study represents a significant step forward in exploring how LLM-based chatbots can support complex mental health challenges. Despite some potential shortcomings, its innovative design and findings provide a foundational basis for further advancements in this field.