Understanding the Benefits and Challenges of Deploying Conversational AI Leveraging Large Language Models for Public Health Intervention

Title of the Paper

Understanding the Benefits and Challenges of Deploying Conversational AI Leveraging Large Language Models for Public Health Intervention

Paper Information

• Subject Areas: Human-Computer Interaction (HCI), Natural Language Processing (NLP), Public Health
• Keywords: Chatbot, Large Language Models, Open-Domain Dialogue Systems, Public Health, Teleconsultation, Social Isolation

Research Background and Problem Statement

• Problem or Challenge: While chatbots powered by large language models (LLMs) have the potential to provide open-ended conversations and emotional support, their application in real-world public health interventions remains in the exploratory stage, particularly for large-scale population health monitoring.
• Significance: Chatbots driven by large language models can alleviate the workload of public health workers by automating repetitive tasks. Exploring their advantages and disadvantages further can help optimize public resources and better meet the health needs of target populations.
• Research Motivation: To investigate the use of the CareCall system (an open-domain chatbot) in real-world health interventions, particularly in addressing social isolation.

Solution

• Proposed Approach: The authors conducted a study on the CareCall system, analyzing its performance in public health interventions compared to traditional chatbots.
• Innovations:
  1. Supporting open-domain emotional conversations through large language models.
  2. Providing emotional support while collecting user health data to alleviate feelings of loneliness.
  3. Analyzing system applications from the perspectives of multiple stakeholders to form comprehensive insights.
• Implementation Steps and Techniques:
  • The CareCall system utilizes a large language model (HyperCLOVA) to support open-domain dialogue.
  • The system generates conversations using an example corpus and displays user health status on a dashboard for monitoring by social workers.
  • Focus group observations and multi-stakeholder interviews were conducted, involving a total of 34 participants.

Research Findings

• Specific Findings:
  1. Advantages: The CareCall system helps social workers gain a comprehensive understanding of users while reducing their workload; users reported that the calls alleviated loneliness and provided emotional support.
  2. Limitations or Challenges:
     • Technical challenges: The system lacks the ability to provide personalized conversations based on individual health histories and has limited long-term memory capabilities.
     • Misalignment between public health needs and chatbot adaptability, making it difficult to answer specific healthcare questions or handle service requests.
     • Occasional generation of inappropriate or problematic responses by the chatbot, posing challenges in maintaining conversation quality.
  3. Experimental and Evaluation Results:
     • Open-domain dialogue systems effectively handle non-health-related topics (e.g., hobbies and cultural life), improving user mood.
     • The system has some potential for health monitoring in socially isolated groups, but users have misconceptions about its cognitive abilities and service scope.
• Limitations and Future Directions:
  1. Research on long-term memory functionality could enhance the system’s performance in providing personalized emotional support.
  2. Resources and processes need to be developed to help stakeholders understand the strengths and weaknesses of open-domain versus task-oriented chatbots, facilitating negotiations between needs and reality.
  3. Mechanisms should be explored to enable target populations or healthcare experts to contribute to the creation of dialogue datasets, addressing the public health needs of specific groups.

Conclusion

Through observations and interviews with multiple stakeholders, this study reveals the advantages and shortcomings of LLM-driven chatbots in public health interventions. It proposes methods to optimize emotional support, address multi-stakeholder collaboration, and expand system adaptability, offering valuable insights for advancing interdisciplinary research in HCI and NLP.