The Last JITAI? Exploring Large Language Models for Issuing Just-in-Time Adaptive Interventions: Fostering Physical Activity in a Prospective Cardiac Rehabilitation Setting

Research Background and Issues

What problems or challenges did the authors identify?

• Many patients struggle to maintain the healthy behaviors (e.g., physical activity) developed during cardiac rehabilitation, especially as face-to-face support diminishes over time, often reverting to sedentary habits.
• Just-in-Time Adaptive Interventions (JITAIs) can deliver personalized health interventions at appropriate times and contexts, but traditional implementations (rule-based or machine learning models) face limitations in flexibility, adaptability, and scalability.
• Existing JITAIs implementations struggle to effectively handle sparse data, lack multimodal information integration capabilities, and often fail to generate personalized and dynamically adjustable content.

Why is this issue important?

• Sustained physical activity post-cardiac rehabilitation is critical for improving long-term health outcomes, yet effective support mechanisms are lacking.
• Digital health intervention tools have significant potential to reduce healthcare resource costs and provide long-term support, but methodological and technological limitations hinder their effectiveness.
• Generative language models (LLMs) have recently demonstrated promising capabilities in processing complex data spaces and generating natural language content, warranting exploration of their application in JITAIs.

Research Motivation and Related Work

• LLMs, such as GPT-4, possess the ability to process high-dimensional contextual data and generate dynamic, personalized responses through "zero-shot learning," addressing the shortcomings of traditional JITAIs implementations.
• Previous studies have shown that LLMs exhibit potential in recommendation systems, healthcare domains (e.g., passing USMLE exams), and generating customized intervention measures, but their efficacy in specific scenarios like cardiac rehabilitation JITAIs remains to be validated.
• The core question of this study is: Can LLMs (e.g., GPT-4) serve as decision-making mechanisms to trigger and generate context-specific content to support patients' physical activity goals?

Solutions

What methods or solutions did the authors propose?

• Proposing a GPT-4-based LLM to trigger and generate JITAIs in the context of cardiac rehabilitation.
• Creating structured input scenarios based on three patient personas (generated from real cardiac rehabilitation patient characteristics) and contextual data to provide a foundation for LLM-generated content.
• Comparing GPT-4-generated interventions with two groups of human-generated interventions (non-experts and healthcare professionals) to comprehensively evaluate the quality and usability of the content.

What are the innovative aspects of this solution?

1. Context-aware generation: Unlike traditional rule-based or machine learning JITAIs, GPT-4 dynamically understands complex and sparse contextual data to generate more personalized intervention content.
2. Dual-task capability: Simultaneously generates decisions (whether to trigger an intervention) and textual content (user prompts tailored to specific contexts).
3. Performance comparison and validation: Evaluates GPT-4-generated interventions against human-generated ones using multidimensional scales (e.g., appropriateness, professionalism) and qualitative participant feedback.

What are the implementation steps? What key technologies were used?

1. Creating patient personas and contextual variables: Integrating health parameters and user behavioral contexts (e.g., physical state, time, location).
2. Model execution: Using GPT-4 to generate two types of intervention texts for each scenario.
3. Comparative analysis:
   • Outputs from three generators (GPT-4, non-experts, healthcare professionals) were evaluated across 150 samples per group.
   • Comprehensive Likert scoring and emotional impact analysis covering four dimensions (appropriateness, engagement, effectiveness, professionalism).
4. Qualitative feedback collection: Allowing evaluators to provide detailed opinions, extracting patterns and trends.
5. Statistical analysis: Using linear mixed models and thematic content analysis to identify generation patterns and outcome differences.

Research Outcomes

What specific results were achieved?

• GPT-4-generated JITAIs scored significantly higher in appropriateness, engagement, effectiveness, and professionalism compared to content generated by non-experts and healthcare professionals.
• Contextual evaluations (e.g., positive/negative impacts on participant experiences) indicated that GPT-4-generated interventions had higher emotional acceptability (fewer negative emotional triggers).
• Participants found it difficult to distinguish GPT-4-generated content from that created by healthcare professionals, further highlighting GPT-4's efficacy.

What advantages does it have compared to existing solutions?

• Greater adaptability: GPT-4 dynamically understands complex contexts, offering greater flexibility than traditional rule-based JITAIs.
• Scalability: LLMs excel in handling high-dimensional data streams and automatically generating context-relevant content, enabling scalable interventions.
• Reliability and consistency: GPT-4 outputs exhibit higher consistency than human-generated content, especially for long-term health interventions.

What were the experimental or evaluation results?

• Quantitative metrics:
  • GPT-4-generated interventions scored significantly higher across all four dimensions (e.g., appropriateness mean score increased by approximately 1 point on a 7-point scale).
  • Quantitative consistency validation (Cohen’s Kappa) showed that GPT-4-generated content achieved the highest scoring consistency among evaluators.
• Qualitative feedback:
  • GPT-4 content was widely perceived as more personalized and better aligned with contextual needs.
  • The proportion of negative feedback triggered by GPT-4 content was significantly lower than the other two groups.

Limitations and Future Directions

• Limitations:
  • The study design was based on simulated scenarios, and results have not been validated in real-world deployment.
  • Real-time generation performance may vary due to technological changes in GPT-4 over time.
  • The study did not compare results with traditional JITAIs (e.g., rule-based systems or machine learning models).
• Future Directions:
  • Further testing of LLMs for real-time generation of JITAIs content in practical applications (e.g., bridging the intention-behavior gap).
  • Integrating reinforcement learning mechanisms in health domains to provide higher-quality user feedback loops for LLMs.
  • Investigating hybrid models that combine LLMs with knowledge graphs, such as retrieval-augmented generation (RAG), to enhance professionalism.
  • Exploring the effectiveness and acceptance of dynamic personalized interventions through continuous deployment in real-life settings.

This study provides preliminary evidence of the potential advantages of LLMs in health intervention scenarios (e.g., dynamic generation, multimodal data processing), offering a feasible and promising improvement pathway for Just-in-Time Adaptive Interventions.