More Forecasts, More (Decision) Problems: How Uncertainty Representations for Multiple Forecasts Impact Decision Making

Research Background and Problem

• Problem or Challenge: Currently, users often face multiple predictions regarding the same event during decision-making. These predictions may be based on different models and assumptions, thus containing uncertainty (quantifiable "probabilistic uncertainty" and epistemic "incertitude"). When dealing with complex uncertainty, there are challenges in how people integrate these predictions into their decisions, and there are no clear guidelines for doing so.
• Importance: With the increasing prevalence of multiple predictions in areas such as weather forecasting, pandemics, and elections, helping users make better decisions under multi-prediction uncertainty is particularly crucial. This not only impacts the effectiveness of actual decisions but also involves design issues in the field of information visualization regarding the representation of uncertainty.
• Research Motivation and Related Work: Although previous studies have extensively discussed the representation of uncertainty in single predictions, research on multiple predictions is relatively scarce. This paper explores how two different types of charts (ensembles and p-boxes) can represent uncertainty and measures the impact of these representations on decision-making strategies.

Solution

• Proposed Method: The paper compares two methods of visualizing uncertainty: 1) Ensemble charts directly represent the distribution of multiple predictions; 2) P-box charts only display the upper and lower bounds of the mean distribution. Both methods are based on cumulative distribution functions (CDFs).
• Innovations:
  • Investigates how these two visualization methods influence the conservativeness of users' decision-making strategies, such as whether they lean toward "worst-case" decision-making.
  • Provides a detailed analysis of how the distribution and clustering of predictions affect user decision-making behavior.
• Implementation Steps:
  1. Experiment Setup: Three experiments were conducted to examine the impact of p-boxes and ensembles on users' decision-making strategies (Experiment 1), the effect of prediction clustering on ensembles' visualization (Experiment 2), and the robustness of decision outcomes under unified phrasing (Experiment 3).
  2. Modeling and Analysis: A logistic regression model was used to estimate participants' weight parameters for probabilities, capturing decision-makers' tendencies (e.g., emphasis on "worst-case" probabilities).
  3. Qualitative Analysis: Users' self-reported strategies were collected and coded to analyze which type of information they preferred (e.g., boundaries, medians).

Research Results

• Specific Findings:
  1. Experiment 1: When predictions are uniformly distributed, the effects of p-boxes and ensembles are similar, with users tending to assign higher weight to the worst-case (lower bound) predictions.
  2. Experiment 2: If predictions cluster on one side of the distribution, users' decisions tend to focus on the clustered region rather than solely considering the "worst-case."
  3. Experiment 3: Modifications in wording (reliable vs. equally reliable) had no significant impact on users' decisions in Experiment 1, confirming the robustness of the results.
• Advantages Compared to Existing Solutions:
  • This paper provides the first empirical evaluation of the practical decision-making impact of p-boxes and ensembles, along with more detailed strategy classifications and effect analyses.
  • Offers preliminary design recommendations, such as using p-boxes when aiming to emphasize extreme predictions, while ensembles are more suitable for presenting detailed transparent information.
• Experimental Results and Insights:
  • Users' focus on the worst-case scenario significantly exceeds rational expectations (optimal threshold at probability 0.2, while experimental estimates suggest users' psychological threshold is higher, around 0.3).
  • Regardless of the representation method, most users did not fully adhere to rational decision points.
• Limitations and Future Directions:
  • Limitations:
    1. Participants were only exposed to binary decision tasks, which cannot simulate more complex choice scenarios.
    2. In some self-reported strategies, users' motivations may have been influenced by external factors (e.g., personal empathy), which the experimental analysis did not fully quantify or explain.
  • Future Directions:
    1. Explore how to design more efficient visualizations to guide user attention (e.g., highlighting key curves, adding interactive features).
    2. Extend visualization designs to multi-choice or continuous tasks.
    3. Apply visualization designs to complex uncertainty problems in other fields, such as the representation of multi-branch hypotheses in scientific analysis (multiverse analysis).