Putting scientific results in perspective: Improving the communication of standardized effect sizes

Title of the Paper

Putting Scientific Results in Perspective: Improving the Communication of Standardized Effect Sizes

Paper Information

• Field: User Interface and Statistical Communication
• Keywords: Statistics, Effect Size, Visualization, Cognition, HCI, Standardized Effects, Misunderstanding Correction, Experiment, Information Communication, Science Dissemination

Research Background and Problem

• Background: In a data-driven society, people are increasingly exposed to statistical information about uncertain outcomes. However, scientific results are often summarized solely by differences in mean values, which frequently obscure critical information about individual variability. This summarization approach not only requires domain-specific knowledge (e.g., familiarity with certain statistical units) but may also lead to an overestimation of treatment effects.
• Problem:
  1. How does the current popular method (e.g., presenting only mean differences between groups) affect people's perception of treatment effects?
  2. What effective approaches can be used to reduce potential misunderstandings caused by the way statistical results are reported?
• Motivation and Related Work: In the HCI field and other academic communities, there have been calls for scientists to reduce reliance on p-values and focus more on effect sizes. However, reporting standardized effect sizes remains uncommon in HCI and other scientific domains. Additionally, research on how different formats of presenting effect sizes influence public understanding is relatively limited.

Solution

• Main Methods:
  • Designed and conducted four large-scale, pre-registered randomized experiments involving nearly 5,000 participants.
  • Compared various formats of presenting statistical information (e.g., mean differences, confidence intervals, and descriptions of effect sizes) to assess their impact on misunderstanding.
  • Explored the potential of using highly intuitive analogies (e.g., the relationship between age and height) to explain effect sizes.
• Innovations:
  • Systematically evaluated the shortcomings of commonly used statistical information formats in communication.
  • Proposed a "probability of superiority" presentation format as an alternative to standardized effect sizes (e.g., Cohen's d).
  • Introduced innovative approaches using relatable analogies to help non-expert readers understand statistical results.
• Implementation Steps:
  • Used a fictional "Rolling Stone Competition" scenario in the experiments, where participants paid for tool rentals to improve performance, and measured their perception of the effects.
  • Presented statistical information in different formats under controlled conditions (simple descriptions, mean differences, confidence intervals, visualizations, etc.).
  • Added supplementary information (e.g., analogies for effect sizes, individual result variance) and measured changes in participants' perceptions.
  • Synthesized experimental results to evaluate the effectiveness of each method in correcting misunderstandings.

Research Findings

• Specific Results:
  • Reporting treatment effects using mean differences tends to cause significant misunderstandings.
  • In Experiment 1, visualizing 95% confidence intervals, while adding layers of information, paradoxically exacerbated overestimation of treatment effects.
  • Displaying information about individual result variability significantly reduced misunderstandings. The experiments showed:
    • The "probability of superiority" format effectively reduced misunderstandings, with error reduction by approximately 50%.
    • Highly relatable analogies (e.g., comparing the heights of 16-year-old and 15-year-old females) achieved a bias correction effect similar to "probability of superiority."
  • In conditions without confidence interval visualizations, "probability of superiority" consistently demonstrated robust corrective effects across all original reporting formats.
• Comparison with Existing Solutions: Compared to presenting only mean differences or confidence intervals, this solution supplemented individual-level variability information and modularized and visualized statistical metrics, making them more accessible to non-expert readers.
• Experimental or Evaluation Results:
  • Experiments involving nearly 5,000 participants showed that new information formats (especially "probability of superiority") could reduce willingness-to-pay bias by over 50%.
  • Core analyses and sensitivity analyses of random sample results demonstrated the robustness of the findings.
• Limitations and Future Directions:
  • Results are limited to virtual settings (Rolling Stone Competition) and single-scale scenarios (e.g., winner-takes-all competition settings).
  • Did not test differences in reactions across cultures or between expert/non-expert populations.
  • Further exploration is recommended on better communicating uncertainty in effect size inference and applying effect size analogies in other specific decision-making contexts.

In summary, this study highlights the importance of enhancing transparency and accuracy in the context of scientific communication and provides a series of practical strategies for improving statistical reporting for non-expert audiences.