CALVI: Critical Thinking Assessment for Literacy in Visualizations

Document Title

CALVI: Critical Thinking Assessment for Literacy in Visualizations

Document Information

• Research Area: Information Visualization, Visual Literacy, and Evaluation of Misleading Visual Content
• Keywords: Information Visualization, Visualization Literacy, Misleading Visualizations, Measurement, Psychometrics

Research Background and Problem

• Problem/Challenges:

  • Misleading visualizations frequently appear in media, such as incorrect axis orientation or erroneous time axis sequencing, which can easily mislead the public's understanding of data.
  • Current research lacks a systematic approach to assess the public's ability to identify and interpret misleading visualizations. Existing visualization literacy tests are limited to correctly constructed visualizations and cannot cover misleading content.

• Research Importance:

  • As information visualization becomes increasingly prevalent in public life, the ability to identify misleading visualizations and engage in critical thinking has become a crucial aspect of civic data literacy, especially in combating misinformation.

• Research Motivation:

  • The authors aim to address the current research gap, namely the absence of tools to measure the ability to critically interpret misleading visualizations.
  • The authors cite multiple real-world cases to demonstrate how errors or intentional misdirection in visualization construction can lead to misinterpretation.

Solution

The authors developed a testing system called CALVI (Critical Thinking Assessment for Literacy in Visualizations) and systematically constructed the evaluation model through the following steps:

1. Definition and Classification:

   • Introduced the definition of "misleaders," referring to decisions made during visualization construction that result in the chart conveying conclusions inconsistent with the actual data.
   • Built a design space for related misleaders, including 11 types of misleading practices (e.g., "Cherry Picking" and "Manipulation of Scales") and 9 common visualization chart types (e.g., bar charts, line charts, scatter plots).

2. Test Design and Optimization:

   • Leveraged psychometric theory (Item Response Theory, IRT) to design and evaluate various test items, including tasks such as identifying trends and comparing data.
   • Conducted testing with 497 participants, using qualitative and quantitative analyses to validate the discriminative power and content validity of the test items.

3. Test Implementation and Revision:

   • Optimized the diversity and realism of the question pool, ultimately finalizing a high-reliability question bank containing 45 test items.

4. Multi-Application:

   • Provided flexible customization methods, recommending specific question sets and scoring approaches tailored to different educational backgrounds (e.g., general public, university students, or experts).

• Innovations:
  • Task Mapping: Established connections between detailed misleading types and specific visualization tasks (e.g., prediction, value retrieval) to link misleader descriptions with user responses.
  • Psychometric Tools: For the first time, introduced psychometric theory-based modeling for visualization testing, ensuring reliability and discriminative power.
  • Real-World Examples: Incorporated multiple misleading visualization cases from news and policy contexts to highlight real-world significance.

Research Outcomes

• Specific Results:

  • Developed the CALVI assessment module for evaluating the ability to interpret misleading visualizations, including a finalized question bank with 45 items covering a broad range of skills.
  • Provided easiness and discrimination parameters for the 45 test items, enabling future researchers to flexibly utilize the questions.
  • Validated the suitability of the test through error source rates (e.g., whether participants chose incorrect answers due to "misleaders") and expert-determined content validity indices.
  • Achieved an overall reliability coefficient of ??=0.81, indicating high reliability.

• Advantages:

  • Compared to traditional visualization literacy tests (which focus only on correct visualizations), this study's test design adds the ability to interpret, discover, and reason about erroneous and potentially misleading information.
  • The systematic design effectively distinguishes participants who are misled from those who correctly interpret visualizations.

• Experiments and Evaluation:

  • Preliminary experiments indicate that the general public often struggles to clearly identify true data in common misleading scenarios, such as incorrect axis orientation or data omission.
  • Enhanced the public's ability to address misleading visualizations in real-world scenarios.

• Limitations and Future Directions:

  • Limitations:
    • Misleading effects may vary significantly under different tasks or question formulations.
    • The test primarily targets general contexts, and its applicability in specific domains (e.g., finance or biomedical fields) requires further validation.
  • Future Directions:
    • Expand and refine the connections between misleading types and user tasks.
    • Develop adaptive or interactive tests to improve evaluation efficiency.
    • Investigate the long-term impact of misleading visualizations on decision-making behavior.