Beyond the Waiting Room: Patient's Perspectives on the Conversational Nuances of Pre-Consultation Chatbots

Literature Title

Beyond the Waiting Room: Patient’s Perspectives on the Conversational Nuances of Pre-Consultation Chatbots

Literature Information

• Domain: Human-Computer Interaction (HCI), Medical Informatics
• Keywords: LLMs, Chatbots, Primary Care, Information Gathering, Patient Registration

Research Background and Issues

• Problems or Challenges:

  • Traditional human-led pre-consultation methods provide personalized interactions but demand significant time and effort from medical staff.
  • Patients often experience "survey fatigue" with conventional questionnaires, leading to lower-quality information and perceptions of being "too simplistic" or "not tailored to individual circumstances."
  • Chatbots powered by large language models (LLMs) can collect information in a conversational manner similar to human interactions, but their performance in clinical settings remains underexplored.

• Significance of the Study:

  • Pre-consultation can enhance patient preparedness for appointments and help physicians make more targeted diagnoses.
  • Implementing pre-consultation via chatbots can reduce labor costs while improving system efficiency.

• Research Motivation:

  • Existing studies have explored feasible chatbot models but have not focused on patient acceptance of these chatbots in clinical environments.
  • This study aims to fill this gap and provide guidance for designing chatbot-based pre-consultation systems in the future.

Solution

• Methods and Implementation:

  • Designed two different pre-consultation chatbots:
    1. An AI agent powered by GPT-4.
    2. A "Wizard-of-Oz Agent" operated by medical professionals.
  • Provided fixed scripts, allowing the chatbots to dynamically adjust questions based on the conversation with patients.
  • Evaluated 33 participants in a real-world walk-in clinic setting, observing their reactions under both conditions.

• Innovations:

  • Compared patient acceptance, interaction effectiveness, and experiential differences between the AI agent and the medical professional-simulated chatbot.
  • The study goes beyond technical performance, delving into patients' psychological experiences, feedback, and the socio-technical context.

Research Findings

• Specific Findings:

  • Patients generally held a positive attitude toward chatbot-based pre-consultation, with similar performance observed under both conditions.
  • While the AI agent could moderately adjust question order and provide background explanations, it underperformed compared to the Wizard agent in terms of probing depth.
  • Patients felt the conversations helped them prepare for face-to-face consultations with doctors and enhanced their ability to articulate their conditions.

• Advantages:

  • Compared to traditional questionnaires, chatbots simulate real-time interactive conversations, reducing patient fatigue.
  • The AI agent provided more explanations about the pre-consultation process, helping patients better understand the overall procedure.

• Experiment and Evaluation Results:

  • Patients rated the dialogue quality of both agents highly, particularly appreciating the logical flow of questions and language fluency.
  • The Wizard agent excelled in depth of questioning and follow-up, while the AI agent tended to offer more empathetic language, though sometimes appearing repetitive and less authentic.
  • Unlike the static design of traditional questionnaires, chatbots demonstrated greater flexibility and content-driven focus during information gathering.

• Limitations and Future Directions:

  • Patients suggested that chatbots should provide a summary at the end of the conversation and allow for modifications or additions to correct potential errors.
  • The AI agent was less effective than the simulated professional agent in certain details; future improvements should focus on enhancing the chatbot's ability to probe further and handle multiple conditions logically.
  • Proposed further research on integrating chatbots with electronic health record (EHR) systems to optimize healthcare resources while ensuring data privacy and accuracy.

Conclusion

This study enriches the understanding of pre-consultation chatbot design through empirical research in walk-in clinic scenarios. The findings indicate significant potential for using chatbots in medical pre-consultation, though optimization of language models is needed to improve information depth and interaction authenticity. Additionally, incorporating user expectation management and privacy safeguards into the pre-consultation process will be critical for driving future adoption of this technology.