Shared Interest: Measuring Human-AI Alignment to Identify Recurring Patterns in Model Behavior

Title of the Paper

Shared Interest: Measuring Human-AI Alignment to Identify Recurring Patterns in Model Behavior

Paper Information

• Domain: Human-Computer Interaction and Explainability in Artificial Intelligence
• Keywords: Human-AI Interaction, Explainability, Machine Learning, Model Behavior, Visualization Analysis, Model Alignment

Research Background and Problem Statement

• Identified Problems or Challenges:

  • Using saliency methods to explain deep learning model decisions requires significant manual effort to inspect and aggregate patterns, leading to potential selection bias or arbitrary analysis.
  • Saliency methods primarily focus on individual instances, making it difficult to conduct large-scale analyses to uncover recurring patterns in model behavior.
  • Existing tools lack structured, high-level visual abstractions to help users effectively understand model behavior.

• Significance:

  • As machine learning models are increasingly deployed in real-world applications, understanding the reasoning behind model decisions is crucial for assessing their reliability, especially in high-risk tasks such as cancer diagnosis.
  • Explaining model behavior helps identify potential biases or unreliable decision bases, preventing mistrust or erroneous deployment of results.

• Research Motivation and Related Work:

  • Provide a systematic approach to quantify and aggregate the alignment between saliency results and human decision-making.
  • While prior research has addressed limitations of saliency methods, such as evaluating the faithfulness of their explanations or proposing higher-level concepts to interpret model behavior, these efforts have not effectively addressed the challenge of large-scale dataset analysis.

Solution

• Proposed Method: The authors introduce the "Shared Interest" method, which designs three metrics (IoU Coverage, Ground Truth Coverage, Saliency Coverage) to quantify the alignment between saliency methods and human-generated labels.

  • IoU Coverage: Measures the similarity between the saliency feature set and the ground truth feature set.
  • Ground Truth Coverage (GTC): Evaluates the proportion of ground truth features captured by the saliency method.
  • Saliency Coverage (SC): Assesses the proportion of saliency features that correspond exclusively to ground truth features.

• Innovations:

  • Enables ranking, filtering, and aggregating input instances to support systematic large-scale model behavior analysis.
  • Independent of model architecture, input modality, and saliency methods, allowing application across diverse environments.
  • Designed a suite of tools and interactive interfaces to facilitate rapid analysis of model behavior by domain experts.
  • Extends saliency method outputs into actionable high-level patterns, revealing eight recurring model behavior patterns.

• Implementation Steps and Key Techniques:

  1. Extract saliency feature sets from the model (e.g., feature maps generated by saliency methods).
  2. Compare saliency feature sets with human annotations (ground truth feature sets) and compute the three metrics.
  3. Classify instances into predefined behavior patterns based on metric scores.
  4. Use specialized visualization tools to analyze these instances, enabling interactive exploration and instance aggregation.

Research Outcomes

• Specific Results:

  • Identified eight model behavior patterns, including human-aligned, sufficient subset, sufficient context, context-dependent, among others.
  • Supported large-scale analysis of model behavior through quantitative metrics, uncovering potential issues or behavioral characteristics under different scenarios.

• Advantages:

  • Quantitative metrics for model-human decision alignment eliminate the tedious and arbitrary nature of manual analysis in saliency method explanations.
  • Rapidly identifies errors or reliability issues in model behavior, inspiring further research directions.
  • The new interactive analysis workflow helps users explore model behavior details, such as studying the relationship between input features and specific predictions through "what-if analysis."

• Experimental or Evaluation Results:

  • Demonstrated the effectiveness of Shared Interest across different saliency methods (e.g., LIME, Integrated Gradients) in image classification (ImageNet) and text sentiment analysis (BeerAdvocate) tasks.
  • Domain experts (e.g., dermatologists) quickly assessed model trustworthiness, while machine learning researchers identified hidden issues such as dataset annotation errors.

• Limitations and Future Directions:

  • Shared Interest relies on human-annotated ground truth labels, which may be costly or unavailable in real-world scenarios.

  • Existing labels may not fully capture all information required for human decision-making, reflecting the insufficiency of ground truth annotations.

  • As a tool based on saliency methods, Shared Interest may inherit the limitations of these methods (e.g., inability to fully reflect the model's actual decision-making process).

  • Future Directions:

    • Extend to tabular data (e.g., medical data) to study more complex semantic relationships.
    • Compare the fidelity of different saliency methods to improve the accuracy of model explanations.
    • Use Shared Interest as a dynamic analysis tool during model training to understand how the model optimizes its decisions over time.

Additional Notes

Shared Interest has released its source code and online demonstration, further promoting practical applications and advancements in model explainability research.