Towards Better Health Conversations: The Benefits of Context-seeking

Paper Title

Towards Better Health Conversations: The Benefits of Context-seeking

Publication Info

• Topic area: Improving conversational AI for health information seeking.
• Keywords: Context-seeking, conversational AI, health information, large language models, user experience, wayfinding AI, health chatbots, interaction design, context-first architecture, trust in AI.

Background and Problem

• Problem / challenge: Current large language models (LLMs) struggle to effectively support laypeople in health information seeking due to difficulties in eliciting necessary context, leading to generic or irrelevant responses. Existing benchmarks often overestimate LLM performance by assuming users provide sufficient context upfront.
• Significance: Improving conversational AI for health topics could enhance access to tailored, accurate medical information, empowering users to make informed decisions and reduce anxiety in navigating complex health concerns.
• Motivation and related work: Prior research highlights the importance of conversational grounding and context-seeking in human-expert interactions, but these behaviors are underexplored in AI systems. Existing health chatbots often fail to establish mutual understanding, leading to user dissatisfaction and mistrust. This study builds on these insights to explore how proactive context-seeking can improve AI interactions.

Solution

• Proposed approach: Development of a "Wayfinding AI" that proactively seeks context from users to provide more relevant and tailored health information.
• Novelty:
  1. Demonstrates the value of proactive context-seeking in health-related AI conversations through qualitative and quantitative studies.
  2. Iteratively designs and evaluates a Wayfinding AI that balances immediate answers with clarifying questions.
  3. Provides evidence that Wayfinding AI is preferred over baseline AI in terms of helpfulness, relevance, and tailoring.
  4. Proposes design considerations for integrating context-seeking into conversational AI systems.
• Procedure and key techniques:
  • Conducted five studies (three qualitative, two quantitative) with 261 participants to evaluate user interactions with AI prototypes.
  • Developed Wayfinding AI using prompt tuning and reinforcement learning to optimize context-seeking behavior.
  • Compared Wayfinding AI to baseline AI in randomized, blinded studies, analyzing user satisfaction, conversational dynamics, and preferences.

Results

• Concrete findings:
  • Wayfinding AI was significantly preferred over baseline AI on dimensions such as helpfulness, relevance, tailoring, and goal understanding.
  • Conversations with Wayfinding AI were longer (4.51 vs. 3.69 turns on average) and included more clarifying questions (87.7% vs. 8.0% in first responses).
  • Users engaged more actively with Wayfinding AI, providing specific details they might not have volunteered otherwise.
• Advantage over baselines:
  • Wayfinding AI elicited more context from users, leading to more tailored and relevant responses.
  • Participants rated Wayfinding AI higher in satisfaction across multiple dimensions compared to baseline AI.
• Experiments / evaluation:
  • Studies included interviews, usability sessions, and randomized blinded surveys.
  • Participants tested both Wayfinding AI and baseline AI on their own health questions, with preferences measured across six dimensions.
  • Analysis included thematic coding of qualitative data and statistical tests on quantitative survey responses.
• Limitations and future work:
  • Focused on text-only interactions; future work could explore multi-modal interfaces.
  • Limited to US participants; findings may not generalize globally.
  • Did not explicitly measure clinical accuracy or trust; future studies should assess these aspects and their impact on health outcomes.

Summary

This study demonstrates that proactive context-seeking significantly improves user satisfaction and perceived relevance in health-related AI conversations. The Wayfinding AI, designed to elicit user context through clarifying questions, was preferred over baseline AI in both qualitative and quantitative evaluations. These findings highlight the importance of conversational grounding and suggest design principles for enhancing AI interactions in complex domains like health. Future work should explore multi-modal capabilities, global applicability, and the impact on clinical outcomes and user trust.