Examining Interpretation Strategies for Multiple Forecast Visualizations with Two and Four Forecasts

Paper Title

Examining Interpretation Strategies for Multiple Forecast Visualizations with Two and Four Forecasts

Publication Info

• Topic area: Visualizations for interpreting multiple forecasts under uncertainty.
• Keywords: Forecast visualization, uncertainty, mental averaging, winner-takes-all, heuristic strategies, density plots, confidence intervals, hypothetical outcome plots, decision-making.

Background and Problem

• Problem / challenge: Limited understanding of how people interpret and integrate competing forecasts presented visually; lack of foundational knowledge to guide effective visualization design.
• Significance: Understanding interpretation strategies is critical for designing visualizations that support accurate decision-making in contexts involving uncertainty, such as policymaking and risk assessment.
• Motivation and related work: Prior studies have explored individual visualization types and strategies but often in domain-specific contexts, limiting generalization. This paper seeks to establish baseline perceptual and cognitive tendencies in a controlled, context-free setting.

Solution

• Proposed approach: Two experiments testing five visualization types (median plots, 95% confidence intervals, standard deviation bands, density plots, and hypothetical outcome plots) across varying forecast agreement levels and numbers of forecasts.
• Novelty:
  1. Identification of 18 distinct interpretation strategies across three categories: mental averaging, winner-takes-all, and heuristic visual artifacts.
  2. Documentation of strategy shifts based on visualization type and forecast properties.
  3. Empirical evidence showing reliance on visual artifacts (e.g., intersection points, end caps) in judgment formation.
  4. Analysis of strategy switching behavior under different forecast agreement conditions.
• Procedure and key techniques:
  • Experiment 1: Tested two forecasts under high and low agreement conditions using normal and skewed distributions.
  • Experiment 2: Extended findings to four forecasts, focusing on mental averaging strategies.
  • Bayesian multilevel modeling to analyze alignment between participant responses and idealized strategies.
  • Participants annotated visualizations directly and provided open-ended explanations of their strategies.

Results

• Concrete findings:
  • Mental averaging strategies were most common for 95% confidence intervals, median plots, and hypothetical outcome plots.
  • Winner-takes-all strategies were prevalent for density plots and standard deviation bands under low forecast agreement.
  • Visual artifacts like intersection points and end caps influenced judgments, introducing variability.
  • Increasing the number of forecasts (from two to four) encouraged mental averaging, particularly under low agreement conditions.
• Advantage over baselines:
  • HOPs and 95% CIs more consistently elicited mathematically ideal strategies (e.g., linear opinion pool) compared to density plots and SD bands.
  • Median plots promoted mental averaging even without explicit uncertainty information.
• Experiments / evaluation:
  • Experiment 1: 500 participants tested across five visualization types, varying forecast agreement, distribution shape, and standard deviation.
  • Experiment 2: 500 participants tested with two and four forecasts under high and low agreement conditions.
  • Metrics included absolute deviation from ideal strategies (Δ), qualitative coding of self-reported strategies, and posterior Bayesian analysis.
• Limitations and future work:
  • Lack of ecological validity due to context-free design; future studies should test domain-specific scenarios.
  • Limited exploration of distributed visualizations (e.g., small multiples).
  • Challenges in eliciting confidence intervals from lay participants; further research needed to improve public understanding of uncertainty.

Summary

This study systematically examines how visualization types and forecast properties influence interpretation strategies for multiple forecasts. Findings reveal that visualization design strongly shapes reasoning, with HOPs and 95% CIs promoting mental averaging, while density plots and SD bands often elicit winner-takes-all approaches under low agreement. Participants frequently relied on visual artifacts, highlighting the need for careful design to minimize unintended biases. Increasing the number of forecasts encouraged mental averaging, suggesting design strategies for improving forecast integration. These insights provide empirical guidance for designing visualizations that align with human reasoning and support accurate decision-making under uncertainty.