With the tremendous success of machine learning (ML), concerns about their black-box nature have grown. The issue of interpretability affects trust in ML systems and raises ethical concerns such as algorithmic bias. In recent years, the feature attribution explanation method based on Shapley value has become the mainstream explainable artificial intelligence approach for explaining ML models. This paper provides a comprehensive overview of Shapley value-based attribution methods. We begin by outlining the foundational theory of Shapley value rooted in cooperative game theory and discussing its desirable properties. To enhance comprehension and aid in identifying relevant algorithms, we propose a comprehensive classification framework for existing Shapley value-based feature attribution methods from three dimensions: Shapley value type, feature replacement method, and approximation method. Furthermore, we emphasize the practical application of the Shapley value at different stages of ML model development, encompassing pre-modeling, modeling, and post-modeling phases. Finally, this work summarizes the limitations associated with the Shapley value and discusses potential directions for future research.

Artificial intelligence (AI) techniques have been widely implemented in the domain of autonomous vehicles (AVs). However, existing AI techniques, such as deep learning and ensemble learning, have been criticized for their black-box nature. Explainable AI is an effective methodology to understand the black box and build public trust in AVs. In this paper, a maximum entropy-based Shapley Additive exPlanation (SHAP) is proposed for explaining lane change (LC) decision. Specifically, we first build an LC decision model with high accuracy using eXtreme Gradient Boosting. Then, to explain the model, a modified SHAP method is proposed by introducing a maximum entropy base value. The core of this method is to determine the base value of the LC decision model using the maximum entropy principle, which provides an explanation more consistent with the human intuition. This is because it brings two properties: 1) maximum entropy has a clear physical meaning that quantifies a decision from chaos to certainty, and 2) the sum of the explanations is always isotropic and positive. Furthermore, we develop exhaustive statistical analysis and visualization to present intuitive explanations of the LC decision model. Based on the explanation results, we attribute the causes of predictions with wrong results to model defects or sample sparsity, which provides guidance to users for model optimization.

The explainability of complex machine-learning models is becoming increasingly significant in safety-critical domains such as autonomous driving. In this context, counterfactual explanation (CE), as an effective explainability method in explainable artificial intelligence, plays an important role. It aims to identify minimal alterations to input that can change the model’s output, thereby revealing key factors influencing model decisions. However, generating counterfactual samples might involve manually selecting input features, potentially leading to suboptimal and biased explanations. This study introduces a feature contribution guided CE generation framework to address this issue. Our method utilizes feature contributions based on Shapley values to guide the model’s focus on the most influential features. This enables end-users to quickly pinpoint the search direction in generating CEs (e.g., prioritizing the most critical features) and producing representative CEs. To comprehensively evaluate our method, we conducted experimental validation on two representative machine learning models: autonomous driving decision-making using Deep Q-Network and lane-changing prediction using deep learning. In addition, we conducted a user-centered study to evaluate the practical applicability of the SVCE in autonomous driving scenarios, which serves as a crucial validation of the presented SVCE. The results show that SVCE can help users understand and diagnose the model.

Various artificial intelligence (AI) algorithms have been developed for autonomous vehicles (AVs) to support environmental perception, decision making and automated driving in real-world scenarios. Existing AI methods, such as deep learning and deep reinforcement learning, have been criticized due to their black box nature. Explainable AI technologies are important for assisting users in understanding vehicle behaviors to ensure that users trust, accept, and rely on AI devices. In this paper, an explainable $Q$ -learning method for AV longitudinal control is proposed. First, AI control of AVs is realized by constructing a deep $Q$ -network (DQN) with an intelligent driver model, with the control objective maximizing vehicle speed while preventing collisions. Then, a deep explainer for humans is developed via a Shapley additive explanation (SHAP), and a novel positive SHAP method that defines new base values is proposed to explain how individual state features contribute to decisions. Finally, statistical analyses and intuitive explanations are quantified based on SHAP tools to improve clarity. Elaborate numerical simulations are conducted to demonstrate the effectiveness of the proposed algorithm. The code is available at https://github.com/limeng-1234/Pos $\_$ Shap.

When design, test and validate an intelligent agent, assessing its intelligence is essential. While autonomous vehicles (AVs) are deployed to a certain degree, it is still hard to assess their intelligence because it highly depends on tested scenarios but in real world tested scenarios are limited and far away from edges. Therefore, this paper attempts to propose an intelligence assessment approach for automated driving systems (ADS) based on behavior index (BI) and scenario complexity (SC). The main contributions of the scheme consist of three aspects: 1) proposing an intelligence assessment framework by following the idea of Turing test, 2) presenting a scenario bank for scenario complexity (SC) and behavior metrics for behavior index (BI), and 3) constructing a definition of driving intelligence quotient (DIQ) by the product of SC and BI. Finally, we present a lane-change scenario bank in Monte Carlo simulations to demonstrate the proposed assessment approach.