PriorWeaver: Prior Elicitation via Iterative Dataset Construction

Paper Title

PriorWeaver: Prior Elicitation via Iterative Dataset Construction

Publication Info

• Topic area: Interactive tools for Bayesian prior elicitation
• Keywords: Bayesian analysis, prior elicitation, interactive visualization, dataset construction, observable space, prior predictive checks, user study, statistical modeling, HCI, belief elicitation

Background and Problem

• Problem / challenge: Prior elicitation in Bayesian analysis is difficult, requiring analysts to translate domain knowledge into abstract probability distributions over model parameters. Existing tools often rely on parameter-space inputs, provide limited feedback, and are not user-friendly for Bayesian novices.
• Significance: Effective prior elicitation is critical for accurate Bayesian modeling, especially in small-sample settings. Simplifying this process can lower barriers to Bayesian analysis and improve its adoption across disciplines.
• Motivation and related work: Prior tools have focused on parameter-space elicitation, requiring probabilistic inputs and offering limited support for expressing relationships across variables. Observable-space tools exist but lack comprehensive support for multivariate relationships and iterative refinement. This paper addresses these gaps by introducing a novel approach.

Solution

• Proposed approach: PriorWeaver, an interactive visualization system that reframes prior elicitation as an iterative dataset construction process.
• Novelty:
  1. Reframes prior elicitation as constructing a dataset that represents analysts’ domain knowledge.
  2. Introduces coordinated visualizations (e.g., histograms, scatterplots, parallel coordinates) for expressing distributional and relational knowledge.
  3. Provides actionable feedback via prior predictive checks to support iterative refinement.
• Procedure and key techniques:
  • Analysts externalize domain knowledge through interactive visualizations, constructing a dataset that reflects variable distributions and relationships.
  • PriorWeaver derives statistical priors by bootstrapping the constructed dataset and fitting a predefined model.
  • Prior predictive checks generate feedback by comparing predictive distributions to analysts’ expectations, enabling iterative refinement.

Results

• Concrete findings:
  • Participants using PriorWeaver produced priors that aligned more closely with their expectations from the first iteration and required fewer refinement steps (median of 3 iterations vs. 8 in the baseline).
  • Survey results showed significantly higher ratings for PriorWeaver in terms of comfort, clarity, ease of expression, and alignment with knowledge (p < 0.01).
• Advantage over baselines:
  • PriorWeaver reduced cognitive burden by allowing direct expression of knowledge in the observable space, avoiding abstract parameter translation.
  • Iterative refinement was more purposeful and actionable compared to trial-and-error adjustments in the baseline.
• Experiments / evaluation:
  • A within-subjects user study with 17 participants (Bayesian novices) compared PriorWeaver to a parameter-space baseline.
  • Tasks involved predicting student exam scores and gym member weights using predefined statistical models.
  • Metrics included survey responses, interaction logs, and qualitative feedback.
• Limitations and future work:
  • Current scope is limited to generalized linear models with continuous variables.
  • Future work includes supporting categorical variables, nonlinear relationships, and mixed-effects models, as well as exploring alternative elicitation methods and broader user groups.

Summary

PriorWeaver introduces a novel approach to Bayesian prior elicitation by framing it as an iterative dataset construction process. Through interactive visualizations, analysts can directly express their domain knowledge about variable distributions and relationships. A user study demonstrated that PriorWeaver significantly improved the clarity, comfort, and alignment of priors compared to a parameter-space baseline, while making Bayesian analysis more approachable for novices. Future work aims to expand its applicability to more complex models and variable types, further lowering barriers to Bayesian analysis.