Visualization Guardrails: Designing Interventions Against Cherry-Picking in Interactive Data Explorers

Research Background and Problem

• Problem Description: With the growing popularity of interactive time-series data exploration platforms (e.g., OurWorldInData or Yahoo! Finance), users can easily create and share data visualizations. However, this freedom has also led to widespread selective information presentation (i.e., cherry-picking), which is often exploited to disseminate false or misleading information. For instance, during the COVID-19 pandemic, 40% of misleading charts employed cherry-picking techniques.
• Significance: Selective data visualization can have severe consequences for decision-makers and the public, such as misconceptions about vaccines or misjudgments in public health policies. Moreover, these charts, which are ostensibly based on real data and appear credible, make subsequent fact-checking extremely challenging.
• Research Motivation: Existing measures to combat misleading information are predominantly post hoc, such as fact-checking, but these methods are often delayed and ineffective in curbing the spread of misinformation. Additionally, much of the data selection is not "wrong" but partially true, making it difficult to identify using traditional fact-checking methods. Therefore, the authors aim to prevent cherry-picking during the data visualization creation stage through "ante hoc" visualization design interventions.

Solution

• Method and Innovation: This paper proposes an intervention called "visualization guardrails." Similar to highway guardrails, these guardrails expose contextual data (e.g., deliberately omitted data or statistical summaries) when necessary, revealing cherry-picking behaviors without interfering with normal operations when no issues are present. This design focuses on addressing cherry-picking in time-series data exploration platforms.
• Design Space:
  1. Layout: Guardrails can be displayed as overlays on the main chart (Superimposition) or presented side-by-side (Juxtaposition).
  2. Information Context: The data displayed by the guardrails can be the same type as the "primary data" or aggregated/transformed "summary data."
  3. Design Characteristics:
     • Non-intrusive: Does not hinder users' freedom in data exploration.
     • Tamper-resistant: Difficult to remove guardrails through screenshots or cropping.
     • Easy to understand: Provides sufficient context to prompt users to reassess the representativeness of the data.
• Key Technologies and Implementation:
  • A prototype data exploration platform was developed, featuring four guardrail examples designed based on known misleading data, such as displaying related countries (based on primary data) or adding industry average lines (based on summary data).
  • Incorporated visual diversity, such as overlaid "track charts" and small sparklines adjacent to the main chart.

Research Outcomes

• Summary of Results:

  1. Proposed a novel framework for analyzing potential issues in data visualization through threat modeling.
  2. Systematically designed the visualization guardrails design space, providing a framework and examples for constructing interventions against cherry-picking.
  3. Validated the impact of visualization guardrails through three crowdsourced experiments (production experiment, reaction experiment, control experiment), finding that specific types of guardrails effectively increase public skepticism.

• Advantages and Experimental Results:

  • In the first experiment, most participants found that overlay guardrails made cherry-picking more challenging, while side-by-side guardrails noticeably reduced task difficulty.
  • The second experiment showed that participants were more likely to question chart content when guardrails were present, with overlay guardrails being the most effective.
  • The third experiment highlighted that the severity of cherry-picking influenced the effectiveness of guardrails, with overlay guardrails consistently performing the best.

• Limitations and Future Directions:

  1. Limitations:
     • Certain guardrails (e.g., summary data) may be visually complex and difficult for general users to understand.
     • Side-by-side designs can be easily overlooked or removed through screenshot cropping.
  2. Future Directions:
     • Further enhance the visibility and interpretability of guardrails, such as integrating tutorials or automated prompts.
     • Explore algorithms for automatically generating contextual data for guardrails to adapt to different domains.
     • Investigate the long-term impact of guardrails in real-world, dynamic adversarial environments (e.g., social media platforms).

Conclusion

This paper introduces an innovative visualization design intervention to reduce the misleading impact of cherry-picking behaviors at their root. Experiments have demonstrated its effectiveness, particularly the significant role of overlay guardrails. However, the key to building effective guardrails lies in designing more intuitive and hard-to-ignore visual cues while dynamically adapting to evolving misleading strategies.