rTisane: Externalizing conceptual models for data analysis increases engagement with domain knowledge and improves statistical model quality

Title of the Paper

rTisane: Externalizing Conceptual Models for Data Analysis Prompts Reconsideration of Domain Assumptions and Facilitates Statistical Modeling

Paper Information

• Research Area: Human-Computer Interaction, Statistical Modeling, Design and Implementation of Domain-Specific Languages (DSL)
• Keywords: Statistical Analysis, Conceptual Modeling, User Modeling, DSL, Concept Externalization, Data Analysis, Hypothesis Formalization, Mixed-Effects Modeling, User Interface

Research Background and Problem

• Identified Problems or Challenges:
  • Statistical models in data analysis should accurately reflect the analyst's domain knowledge, but current tools have limitations in supporting analysts to externalize implicit assumptions and domain relationships.
  • Few tools explicitly help domain experts (rather than statistical experts) externalize implicit assumptions and translate them into statistical models.
  • Existing tools (e.g., Tisane, Dagitty, DoWhy) optimize the statistical modeling process only on certain technical aspects, without sufficiently guiding analysts to consider the domain relevance of their models.
• Significance:
  • Skipping the externalization of domain assumptions can lead to models failing to capture critical relationships, resulting in erroneous conclusions.
  • Optimizing tool design to unlock the potential of non-statistical experts can improve analysis quality and reduce errors.
• Research Motivation and Related Work:
  • Building on previous research based on Tisane (which provides modeling and statistical analysis capabilities for variable relationships), this study explores how to more directly support conceptual modeling.
  • The authors aim to balance usability and theoretical rigor in the design of domain-specific languages (DSL).

Solution

• Proposed Method or Solution:
  • Develop the rTisane tool, which combines a new DSL with an interactive conceptual clarification interface to support analysts in externalizing and enhancing conceptual models, then translating them into statistical models.
  • Includes:
    1. DSL language: Enables users to declare variables, construct conceptual relationships, and distinguish between known assumptions and testable hypotheses.
    2. Two-stage interactive process: Uses visualization tools to help users eliminate ambiguities in models, refine conceptual models, and generate statistical models.
• Innovations:
  • Introduces a novel language construct that supports externalizing domain assumptions while maintaining abstraction to intuitively reflect domain knowledge.
  • Integrates a two-stage interactive process to help analysts validate and adjust conceptual relationships, ensuring statistical models closely align with their understanding.
• Implementation Steps and Key Techniques:
  1. Variable Definition: DSL supports the declaration of continuous and categorical variables.
  2. Conceptual Model Construction: Relationship types include "causes" (causal relationships) and "relates" (asymmetric associations), with users marking known relationships using "assume" and testable hypotheses using "hypothesize."
  3. Interactive Clarification: GUI assists users in resolving uncertain relationships or loops in structures.
  4. Statistical Model Derivation: Automatically selects covariates, interaction terms, appropriate family models, and link functions from the generated causal graph.
  5. Model Script Output: Generates GLM (Generalized Linear Model) scripts for use in R.

Research Outcomes

• Specific Outcomes:
  • rTisane significantly simplifies the expression and connection between conceptual models and statistical models.
  • Enhances users' depth of thinking about implicit assumptions, with most users reporting that it helped them more accurately express domain knowledge.
  • The quality of statistical models (evaluated via AIC/BIC) was often superior to manually constructed models by users.
  • rTisane successfully enabled users who were unable to independently construct statistical models to complete modeling tasks.
• Advantages Compared to Existing Solutions:
  • Compared to other tools (Tisane, Dagitty), rTisane is more suitable for non-statistical experts, guiding the entire modeling process starting from conceptual models.
  • Provides a bridge from domain assumptions to statistical implementation, making the analysis process transparent and more reproducible.
• Experimental or Evaluation Results:
  1. User Experience:
     • Most users found it easier to express hypotheses using rTisane and were satisfied with and confident in the tool's output models.
     • The tool shifted users from low-level programming tasks to higher-level analytical goals.
  2. Model Quality:
     • On AIC and BIC metrics, most user-generated models were as good as or better than models constructed without the tool.
     • For complex domain problems, rTisane's models better configured variables and controlled relationships.
  3. Limitations:
     • The tool currently provides insufficient intuitive guidance for selecting family functions and link functions.
  4. Acceptance:
     • Users expressed interest in the tool generating "boundary object" documentation for broader collaboration or publication, facilitating academic communication.
• Limitations and Future Directions:
  • The tool needs improvement in result presentation to help users more intuitively understand statistical outcomes.
  • Develop more iterative interactive features, such as optimization based on model outputs.
  • Expand the tool's application across interdisciplinary domains and explore further integration with existing analytical tools.