23 February 2023 in Nature Machine Intelligence
6 April 2021 in IEEE Transactions on Intelligent Transportation Systems
Reinforcement learning (RL) can be used to design smart driving policies in complex situations where traditional methods cannot. However, they are frequently black-box in nature, and the resulting policy may perform poorly, including in scenarios where few training cases are available. In this paper, we propose a method to use RL under two conditions: (i) RL works together with a baseline rule-based driving policy; and (ii) the RL intervenes only when the rule-based method seems to have difficulty handling and when the confidence of the RL policy is high. Our motivation is to use a not-well trained RL policy to reliably improve AV performance. The confidence of the policy is evaluated by Lindeberg-Levy Theorem using the recorded data distribution in the training process. The overall framework is named “confidence-aware reinforcement learning” (CARL). The condition to switch between the RL policy and the baseline policy is analyzed and presented. Driving in a two-lane roundabout scenario is used as the application case study. Simulation results show the proposed method outperforms the pure RL policy and the baseline rule-based policy.
6 January 2020 in IEEE Transactions on Intelligent Transportation Systems
Exiting from highways in crowded dynamic traffic is an important path planning task for autonomous vehicles (AVs). This task can be challenging because of the uncertain motion of surrounding vehicles and limited sensing/observing window. Conventional path planning methods usually compute a mandatory lane change (MLC) command, but the lane change behavior (e.g., vehicle speed and gap acceptance) should also adapt to traffic conditions and the urgency for exiting. In this paper, we propose a reinforcement learning-enhanced highway-exit planner. The learning-based strategy learns from past failures and adjusts the vehicle motion when the AV fails to exit. The reinforcement learning is based on the Monte Carlo tree search (MCTS) approach. The proposed learning-enhanced highway-exit planner is tested 6000 times in stochastic simulations. The results indicate that the proposed planner achieves a higher probability of successful highway exiting than a benchmark MLC planner.
17 August 2022 in IEEE Transactions on Intelligent Transportation Systems
Most environment perception methods in autonomous vehicles rely on deep neural networks because of their impressive performance. However, neural networks have black-box characteristics in nature, which may lead to perception uncertainty and untrustworthy autonomous vehicles. Thus, this work proposes a decision-making method to adapt the potential perception uncertainty due to the sensor noises, fuzzy features, and unfamiliar inputs. The whole method is named as Perception Neural Networks Uncertainty Aware Decision-Making (PNNUAD) method. PNNUAD first uses the Monte Carlo dropout method to estimate the perception neural network uncertainty into a distribution around the original output. Then, the perception uncertainty will be considered in a designed reinforcement learning-based planner using a distributed value function. Finally, a backup policy will maintain the vehicle’s performance to avoid disastrous perception uncertainty. The evaluation section uses an augmented reality urban driving scenario; namely, the scenario builds in the CARLA simulator while the perception uncertainty comes from the real dataset. This case study focuses on the object class uncertainty of a widely used neural network, i.e., YOLO-V3. The results indicate that the proposed method can maintain AV safety even with poor perception performance. Meanwhile, the AV has not become too conservative by defending the perception uncertainty. This work is necessary for applying the statistics neural networks to safety-critical autonomous vehicles, and the source code will be open-source in this work.
10 December 2021 in IEEE Transactions on Systems, Man, and Cybernetics: Systems
This article presents a model-based motion planning and control system for autonomous vehicles and its experimental validation. The system consists of four modules: 1) global routing; 2) behavior planner; 3) local trajectory generation; and 4) trajectory tracking. The algorithm and software of each module are detailed, including a behavior planner with unified models to handle typical scenarios in both highway and urban driving, a deterministic sampling algorithm for robust responsive trajectory generation, and a dynamics-and-delay-aware preview algorithm to achieve accurate trajectory tracking. The developed system is implemented and tested at the Mcity test facility with a full-size automated car and a dozen of challenging traffic scenarios.
6 December 2022 in IEEE Transactions on Intelligent Transportation Systems
Self-driving vehicles (SDVs) are becoming reality but still suffer from “long-tail” challenges during natural driving: the SDVs will continually encounter rare, safety-critical cases that may not be included in the dataset they were trained. Some safety-assurance planners solve this problem by being conservative in all possible cases, which may significantly affect driving mobility. To this end, this work proposes a method to automatically adjust the conservative level according to each case’s “long-tail” rate, named dynamically conservative planner (DCP). We first define the “long-tail” rate as an SDV’s confidence to pass a driving case. The rate indicates the probability of safe-critical events and is estimated using the statistics bootstrapped method with historical data. Then, a reinforcement learning-based planner is designed to contain candidate policies with different conservative levels. The final policy is optimized based on the estimated “long-tail” rate. In this way, the DCP is designed to automatically adjust to be more conservative in low-confidence “long-tail” cases while keeping efficient otherwise. The DCP is evaluated in the CARLA simulator using driving cases with “long-tail” distributed training data. The results show that the DCP can accurately estimate the “long-tail” rate to identify potential risks. Based on the rate, the DCP automatically avoids potential collisions in “long-tail” cases using conservative decisions while not affecting the average velocity in other typical cases. Thus, the DCP is safer and more efficient than the baselines with fixed conservative levels, e.g., an always conservative planner. This work provides a technique to guarantee SDV’s performance in unexpected driving cases without resorting to a global conservative setting, which contributes to solving the “long-tail” problem practically.