How can Explainability Methods be Used to Support Bug Identification in Computer Vision Models?

Document Title

How can Explainability Methods be Used to Support Bug Identification in Computer Vision Models?

Document Information

• Subject Area: Debugging deep learning models in human-computer interaction and computer vision
• Keywords: Computer vision, machine learning model debugging, machine learning explainability, user interface, deep learning, bug identification, interaction design

Research Background and Problem

• Identified Problems or Challenges: Deep learning models, particularly image classification models, often reveal critical issues after deployment. The process of diagnosing these issues and identifying underlying bugs remains insufficient and underexplored. Although many studies highlight the potential of explainability methods in bug identification, there is currently a lack of clear understanding regarding which explainability methods best support the various steps of bug identification, as well as a lack of human-computer interaction research in this area.
• Research Importance: Errors in computer vision models can lead to incorrect predictions, posing safety risks or societal harm. Therefore, it is crucial to identify and resolve errors as early as possible during model development.
• Research Motivation and Related Work:
  • The machine learning community has developed numerous explainability methods, but their practical utility and impact remain underexplored.
  • Traditional software debugging has established methods (e.g., context gathering, hypothesis validation), but their applicability to computer vision models has not been investigated.
  • Currently, only a few studies attempt to integrate user interface design to support model debugging.

Solution

• Methods or Solution:
  • The authors propose a design probe that showcases various potentially relevant explainability functionalities through an interactive interface.
  • An interactive tool was developed using literature review, formative research (18 semi-structured interviews), and iterative co-creation processes.
• Innovations:
  • Integration of multiple types of explanations (e.g., local, global, textual, visual) within the interactive interface.
  • Design of an interface that supports flexible debugging workflows rather than imposing a fixed sequence.
  • Validation of explainability methods' specific support for bug identification through real user studies.
• Implementation Process and Techniques:
  • Tool functionalities designed include performance evaluation, local and global explanations, comparative analysis, importance ranking, etc.
  • The tool is organized into multiple independent functional modules (e.g., "Query" tab, "Confusion Matrix" tab).
  • A case study was designed and implemented to debug a bird classification model.
  • Data exploration features were created to combine domain expertise, supporting model performance analysis and error diagnosis.

Research Outcomes

• Specific Outcomes:
  • Over two-thirds of participants successfully used the tool to identify model errors.
  • Participants utilized various types of explainability features to explore error causes and propose potential corrective actions.
  • Local and global explanations were found to be complementary, helping developers formulate and validate hypotheses.
• Advantages Over Existing Solutions:
  • Supports richer error identification and debugging functionalities, beyond reliance on local saliency maps.
  • Enables identification of implicit errors (e.g., models predicting correctly but relying on inappropriate features).
• Experimental or Evaluation Results:
  • Participants identified an average of 3.5 bugs, with a maximum of 7 bugs identified.
  • Machine learning developers with varying levels of experience showed different levels of acceptance and responses to the tool's functionalities.
  • Feature diversity and interactivity were considered key factors in reducing debugging time and improving efficiency.
• Limitations and Future Directions:
  • The study is limited to explainability research for specific deep learning tasks; future research should expand to other computer vision applications.
  • For developers unfamiliar with domain knowledge, the tool needs to provide better guidance and support.
  • The study highlights the urgent need to enhance explainability methods and explore the integration of automated debugging methods with manual debugging.
  • Providing more guided pathways and design-driven experiences within the tool could further improve debugging efficiency and accuracy.

Summary

This study presents a prototype tool integrating user research findings and multiple explainability features, offering robust support for error identification in computer vision models. The research not only provides an innovative case for debugging practices but also offers significant insights for future studies in explainable AI and human-computer interaction.