Graphical Perception of Saliency-based Model Explanations

Title of the Paper

Graphical Perception of Saliency-based Model Explanations

Paper Information

• Domain: Human-AI Collaboration, Visualization Design, Model Explainability
• Keywords: Graphical perception, saliency maps, model explanation, neural networks, user study

Research Background and Problem

• Problem and Challenges: In recent years, the explainability of deep learning models has garnered significant attention, with saliency maps serving as a critical visualization tool for interpreting the predictions of visual recognition models. However, there is still a lack of in-depth understanding of how these visual explanations influence human perception and decision-making in practice, particularly regarding the specific factors of visualization design that affect user perception.

• Significance: In high-stakes application scenarios (e.g., medical image analysis), saliency maps not only reveal the decision logic of models but also assist users in making more reliable decisions. Understanding the impact of visual explanations on human perception can facilitate human-AI collaboration and address inconsistencies between model and human decisions.

• Motivation and Related Work: Existing studies have evaluated the quality of explanations primarily from machine learning methods or automated metrics, often neglecting direct user perception. Unlike prior meta-works such as HIVE and some human-AI collaboration task studies, this paper focuses on the critical role of visualization design in shaping human perception of saliency maps, aiming to fill this research gap.

Proposed Solution

• Method or Solution: An experimental framework was designed to investigate how different visualization designs of saliency maps (e.g., heatmaps, binary masks, and contour lines) influence user perception in alignment tasks. The experiment also considered other characteristics of saliency maps, such as alignment types (overestimating or underestimating target objects) and the distribution of saliency values.

• Innovations:

  1. Investigated the impact of visual encoding types of saliency maps on perception (e.g., comparing heatmaps and contour lines).
  2. Quantified and analyzed the significant effects of alignment types (overestimating, underestimating, or partially matching target objects) on human perception.
  3. Developed a human perception model to analyze how visualization parameters (e.g., range selection) influence decision consistency.

• Implementation Steps and Key Techniques:

  1. Data Preparation: Selected objects from ImageNet and related segmentation datasets, using GradCAM to generate saliency maps.
  2. Visualization Design: Implemented three types of visual encodings for saliency maps (heatmaps, contour lines, binary masks) and explored the effects of different parameters such as thresholds or color ranges on perception.
  3. User Study: Designed various experimental conditions to collect human judgment results (e.g., selecting suitable visualizations) and recorded metrics such as response times.
  4. Quantitative Analysis: Used statistical models (e.g., linear mixed-effects models) to explore the relationships between perception and design variables.

Research Findings

• Key Results:

  1. Visualization Encoding: Users perceived quantitative encodings like heatmaps and contour lines more effectively than binary masks, especially as image alignment (IoU) levels increased.
  2. Alignment Types: Saliency maps that overestimated target objects were more acceptable to users than those that underestimated or partially aligned with the objects.
  3. Saliency Map Characteristics: Saliency maps with "binarized" distributions were more likely to be judged as aligned with objects by users. Additionally, saliency maps with higher entropy (indicating more complex mappings) reduced user consistency in responses.
  4. Human Perception Model: Proposed a perception model based on user responses, accurately predicting human reactions to saliency maps under different visualization parameters.

• Advantages Over Existing Solutions: This study takes a human perception perspective and, for the first time, incorporates factors like alignment types and entropy into the evaluation of saliency maps. Compared to automated metrics (e.g., IoU scores), the perception model provides an evaluation of explanation quality that aligns more closely with user preferences.

• Experimental or Evaluation Results:

  • Experiments showed that heatmaps and contour lines more effectively supported user tasks.
  • In experimental design and interactive prototypes, the model successfully predicted how various parameter settings influenced user judgments (e.g., within applicable visual threshold ranges).

• Limitations and Future Directions:

  1. This study is limited to GradCAM; future work could extend perception analysis to other saliency methods.
  2. The user study focused on participants with limited AI expertise; future studies could include more professional machine learning users.
  3. In terms of scalability, the modeling approach could be extended to more machine learning application domains or non-visual tasks.
  4. Enhancing interactive prototype designs, such as validating whether user feedback in real-world environments improves the interpretability of model explanations.