When to Explain: Modeling User Need for Explanations in Real-World Autonomous Driving

Paper Title

When to Explain: Modeling User Need for Explanations in Real-World Autonomous Driving

Publication Info

• Topic area: User-centered explainability in autonomous vehicles (AVs)
• Keywords: Explainable AI, autonomous vehicles, user need modeling, driving scenarios, human-AI interaction, dataset creation, predictive modeling, SHAP analysis, adaptive interfaces, AV transparency

Background and Problem

• Problem / challenge: Existing research on Explainable AI (XAI) often assumes users always need explanations, which can lead to inefficiencies and cognitive overload. Current models predicting explanation needs are limited in scope, rely on single modalities, and lack interpretability, making them unsuitable for practical system design.
• Significance: Understanding when users need explanations in AV contexts is critical for improving user trust, acceptance, and experience, while avoiding unnecessary cognitive load.
• Motivation and related work: Prior studies have explored explanation formats, delivery modalities, and factors influencing explanation needs (e.g., user traits, driving scenarios). However, these studies are limited by narrow datasets, lack of comprehensive annotations, and non-interpretable models. This paper seeks to address these gaps by creating a holistic dataset and predictive model.

Solution

• Proposed approach: Development of DriveScene, a large-scale annotated dataset of driving scenarios, and a predictive model leveraging user- and scenario-related factors to determine explanation needs.
• Novelty:
  1. Creation of DriveScene, a dataset with 3,327 real-world driving scenarios annotated with a What–Why–How–Where–When schema and explanation-ready assets.
  2. Development of an interpretable predictive model (Random Forest) for explanation needs, achieving strong performance (F1 = 0.7188, AUC = 0.8221).
  3. Comprehensive analysis of user- and scenario-related factors influencing explanation needs, using SHAP for interpretability.
• Procedure and key techniques:
  1. Curated and annotated driving scenarios from existing datasets (e.g., BDD100K, Argoverse) with detailed labels (e.g., AV actions, event causes, weather).
  2. Conducted a user study with 408 participants to collect explanation need ratings across scenarios.
  3. Trained and evaluated machine learning models (e.g., Random Forest, Gradient Boosting) with feature selection methods (e.g., SelectKBest, Boruta).
  4. Used SHAP analysis to interpret feature contributions to explanation need predictions.

Results

• Concrete findings:
  • Best-performing model: Random Forest with SelectKBest, achieving F1 = 0.7188, accuracy = 0.7432, and AUC = 0.8221.
  • Key predictors: Perceived AV Driving Style, Event Cause, AV Action, Annual Mileage, and Human Driving Style were the most influential features.
  • Dynamic objects (e.g., pedestrians, cars) and high-risk maneuvers (e.g., lane changes) increased explanation needs, while routine actions (e.g., moving forward) decreased them.
• Advantage over baselines:
  • Improved interpretability and predictive power compared to prior models relying on single modalities or narrow datasets.
  • Demonstrated the importance of combining user- and scenario-related factors for accurate predictions.
• Experiments / evaluation:
  • Dataset: 8,038 video samples from 402 participants.
  • Models: Compared logistic regression, SVM, and tree-based methods with feature selection techniques.
  • Metrics: F1 score, accuracy, AUC.
• Limitations and future work:
  • Dataset lacks multi-modal inputs (e.g., LiDAR, depth) and longer temporal context.
  • Models do not capture temporal dynamics or evolving user states.
  • Future work includes incorporating richer telemetry, testing adaptive explanation policies, and expanding dataset diversity.

Summary

This paper introduces DriveScene, a comprehensive dataset of 3,327 annotated driving scenarios, and a predictive model for determining when users need explanations in AV contexts. The Random Forest model achieved strong performance (F1 = 0.7188, AUC = 0.8221) and revealed that factors like perceived AV driving style, event cause, and AV action are critical predictors. The findings highlight the importance of tailoring explanations to user profiles and scenario contexts, particularly for high-risk maneuvers and interactions with dynamic objects. Future work will focus on enhancing dataset richness, incorporating temporal dynamics, and refining adaptive explanation systems.