Towards Efficient Annotations for a Human-AI Collaborative, Clinical Decision Support System: A Case Study on Physical Stroke Rehabilitation Assessment

Document Title

Towards Efficient Annotations for a Human-AI Collaborative Clinical Decision Support System: A Case Study on Physical Stroke Rehabilitation Assessment

Document Information

• Topic Area: Application of Artificial Intelligence and Machine Learning in Clinical Decision Support Systems, specifically in stroke rehabilitation assessment
• Keywords: Human-centered AI, Human-AI collaboration, Human-in-the-loop systems, Clinical Decision Support Systems, Stroke rehabilitation assessment

Research Background and Issues

• Identified Problems or Challenges:

  • Current clinical decision support systems are predominantly rule-driven or based on machine learning models. Traditional rule-based models are interpretable and flexible but struggle to comprehensively capture complex clinical decision-making rules.
  • Machine learning models can automatically extract patterns from data, but their development requires large amounts of labeled data, which is expensive and time-consuming to collect and annotate.
  • Machine learning models face challenges in clinical applications due to the lack of human-centered design and their "black-box" nature, making their predictions difficult to interpret.

• Significance of the Research:

  • Stroke rehabilitation assessment is a critical component of tailoring treatment plans for patients. However, due to limited therapist time and the complexity of assessments, this process is often overlooked.
  • Developing collaborative clinical decision support systems to assist therapists in making efficient decisions is crucial for improving the quality and efficiency of rehabilitation assessments.

• Motivation and Related Work:

  • Leveraging domain expert knowledge to develop rule-driven systems combined with machine learning can enhance system efficiency.
  • Exploring human-AI collaboration methods can enable more efficient data annotation and complex decision support in high-risk domains such as healthcare systems.

Solution

• Proposed Solution:

  • Develop a hybrid model combining machine learning and domain expert rule-driven models for collaborative clinical decision support systems to assess stroke rehabilitation quality.

• Innovative Contributions:

  • Propose a workflow where rule-based models generate predictions and efficiently annotate low-confidence samples to collect labeled data, thereby reducing annotation workload.
  • Introduce a hybrid model that combines the flexibility of machine learning models for automatic pattern extraction with domain experts' dynamic decision-making through rule-based models.

• Implementation Steps/Key Techniques:

  1. Data Collection: Use Kinect v2 sensors to record joint data of stroke patients performing three upper limb rehabilitation movements.
  2. Rule Definition: Conduct semi-structured interviews with therapists to obtain specific rules for assessing movement quality.
  3. Rule-Based Model: Build an initial scoring model based on the rules to predict rehabilitation movement quality and identify low-confidence samples.
  4. Machine Learning Model: Train a neural network using labeled data to evaluate rehabilitation quality.
  5. Hybrid Model: Integrate the rule-based and machine learning models, assigning weights based on their performance.
  6. System Interface: Develop visualization tools to display patient movement quality assessments, related confidence scores, and movement trajectory trends.

Research Outcomes

• Specific Results:

  • The system can identify low-confidence samples and annotate critical ones using the rule-based model, requiring only 22% to 33% of expert annotations to train a machine learning model with performance comparable to fully labeled training data.
  • The hybrid model achieved competitive prediction performance in stroke rehabilitation assessment, with an average F1 score of 0.7931, consistent with expert annotations.

• Advantages Over Existing Solutions:

  • Compared to traditional methods relying solely on machine learning or rule-based models, this system combines the strengths of both, improving prediction accuracy while enabling dynamic expert participation to enhance flexibility and interpretability.
  • Significantly reduces expert annotation time and workload, making complex tasks more efficient in high-risk environments.

• Experimental or Evaluation Results:

  • Model validation was conducted using assessment data from stroke patients and healthy participants (dataset of 15 patients and 11 healthy participants).
  • The hybrid model achieved higher prediction consistency than standalone rule-based or machine learning models while retaining the interpretability of the rule-based model.

• Limitations and Future Directions:

  • Limitations:
    • The study focuses solely on stroke rehabilitation assessment, lacking application expansion to other clinical decision-making tasks (e.g., cancer diagnosis).
    • The hybrid model's performance may still be constrained by the quality of the rule-based model, and further optimization is needed for low-confidence sample identification.
    • The system interface design has not fully addressed how to build user trust and mitigate potential biases.
  • Future Directions:
    • Explore the application of this human-AI collaboration method in other tasks and data modalities, such as computer vision or other medical scenarios.
    • Optimize confidence score calculation methods to further improve low-confidence sample identification.
    • Enhance interface interactivity to support trust-building in complex decision-making processes.

This study provides valuable insights into the design of human-AI collaboration workflows and data annotation processes in high-risk domains.