Tempo: Helping Data Scientists and Domain Experts Collaboratively Specify Predictive Modeling Tasks

Research Background and Problem Statement

• Identified Problems or Challenges:

  • While time series prediction models hold promise in healthcare and other domains, model specifications (i.e., what to predict, when to predict, and for whom to predict) often fail to align with the expectations of non-technical domain experts. This misalignment leads to decision support tools (DSTs) being difficult to adopt in high-risk domains such as healthcare or public welfare.
  • Traditional model development processes are overly technical, making it challenging for non-technical domain experts to understand and participate in the model improvement process.

• Significance:

  • In high-risk domains such as healthcare and public welfare, the misuse or misalignment of models can result in systemic decision-making errors.
  • Without appropriate early feedback mechanisms, issues with model specifications often surface only after significant development and deployment efforts, which can be costly for teams.

• Research Motivation and Related Work:

  • Many existing tools focus on data cleaning and model selection but lack support for handling complex time-series data and fail to effectively facilitate early collaboration among interdisciplinary teams.
  • Related work suggests that involving domain experts in the early stages of model development can help identify specification issues more quickly, reduce directional biases, and improve the adaptability of the final model.

Solution

• Approach or Solution:

  • A novel interactive system called "Tempo" is proposed to assist data scientists and domain experts in collaboratively and iteratively specifying and refining model specifications.
  • Tempo employs a user-friendly temporal query language to enable rapid experimentation in model design and enhance transparency in data preprocessing.
  • The system also provides subgroup analysis functionality, allowing domain experts to evaluate and provide feedback on model design based on the characteristics and behavior of data subgroups.

• Innovative Features:

  • A readable yet rigorous temporal query language simplifies the aggregation and sequencing of complex data.
  • Subgroup discovery and visualization features directly showcase model behavior under specific rules, enabling domain experts to participate more efficiently.
  • The system supports not only model specification selection but also rapid exploration of alternative modeling directions in the early stages, accelerating iterative development.

• Implementation Method:

  1. Data Import and Query:
     • Define basic field types (e.g., attributes, events, intervals) and simplify data preprocessing tasks through the query language.
  2. Model Specification Design:
     • Use an interactive interface to define time steps, feature inputs, and target variables, supporting rapid iteration and modification.
  3. Subgroup Analysis:
     • Automatically discover subgroups defined by rules, evaluate model performance within these subgroups, and support rule editing.
  4. Model Training and Evaluation:
     • Train specifications using default XGBoost or custom neural network models, providing top-level performance metrics and error detection alerts.

Research Outcomes

• Specific Outcomes:

  • Three case studies demonstrate how the Tempo system helps teams quickly eliminate infeasible model specifications and identify more promising directions:
    1. Web Browsing Data Prediction: Rapidly adjust model time windows to optimize product features.
    2. Sepsis Prediction in Intensive Care: Improve model usability by filtering out unnecessary patient subsets through rule-based exclusion.
    3. Rehospitalization Prediction in Home Health Care: Quickly identify insufficient data specifications and pivot to more insightful modeling targets.

• Comparison with Existing Solutions and Advantages:

  • Compared to traditional tools, Tempo significantly lowers the technical barriers to collaboration in interdisciplinary teams, enabling domain experts to participate more directly in model development.
  • The system's query language and model visualization tools accelerate prototype development and enhance transparency and interpretability through subgroup analysis tools.

• Experimental or Evaluation Results:

  • Across the three case studies, the Tempo system consistently accelerated the iteration from concept to evaluation and facilitated team discussions on non-intuitive problems.
  • User feedback indicates that Tempo offers significant advantages in time-series data processing and model behavior interpretation, while its interactive interface and feature design are highly beneficial for both data scientists and domain experts.

• Limitations and Future Directions:

  • Limitations:
    • The current version is primarily suited for lightweight models and prototype development, which may not be ideal for scenarios requiring large-scale deep learning models.
    • The system's input data format requires manual conversion, potentially increasing the burden of data import.
  • Future Directions:
    • Enhance direct integration with SQL data sources and support more advanced editing functionalities.
    • Explore the use of language models (LLMs) to automatically generate queries and features, further lowering barriers to entry.
    • Deepen subgroup editing functionalities and investigate how subgroup discovery can be further translated into practical model improvements and user decision support.