2 February 2021 in Nature Communications
Driving intelligence tests are critical to the development and deployment of autonomous vehicles. The prevailing approach tests autonomous vehicles in life-like simulations of the naturalistic driving environment. However, due to the high dimensionality of the environment and the rareness of safety-critical events, hundreds of millions of miles would be required to demonstrate the safety performance of autonomous vehicles, which is severely inefficient. We discover that sparse but adversarial adjustments to the naturalistic driving environment, resulting in the naturalistic and adversarial driving environment, can significantly reduce the required test miles without loss of evaluation unbiasedness. By training the background vehicles to learn when to execute what adversarial maneuver, the proposed environment becomes an intelligent environment for driving intelligence testing. We demonstrate the effectiveness of the proposed environment in a highway-driving simulation. Comparing with the naturalistic driving environment, the proposed environment can accelerate the evaluation process by multiple orders of magnitude.
11 April 2023 in Nature Communications
For simulation to be an effective tool for the development and testing of autonomous vehicles, the simulator must be able to produce realistic safety-critical scenarios with distribution-level accuracy. However, due to the high dimensionality of real-world driving environments and the rarity of long-tail safety-critical events, how to achieve statistical realism in simulation is a long-standing problem. In this paper, we develop NeuralNDE, a deep learning-based framework to learn multi-agent interaction behavior from vehicle trajectory data, and propose a conflict critic model and a safety mapping network to refine the generation process of safety-critical events, following real-world occurring frequencies and patterns. The results show that NeuralNDE can achieve both accurate safety-critical driving statistics (e.g., crash rate/type/severity and near-miss statistics, etc.) and normal driving statistics (e.g., vehicle speed/distance/yielding behavior distributions, etc.), as demonstrated in the simulation of urban driving environments. To the best of our knowledge, this is the first time that a simulation model can reproduce the real-world driving environment with statistical realism, particularly for safety-critical situations.
2 July 2021 in Transportation Research Record: Journal of the Transportation Research Board
Testing and evaluation is a crucial step in the development and deployment of connected and automated vehicles (CAVs). To comprehensively evaluate the performance of CAVs, it is necessary to test the CAVs in safety-critical scenarios, which rarely happen in a naturalistic driving environment. Therefore, how to purposely and systematically generate these corner cases becomes an important problem. Most existing studies focus on generating adversarial examples for perception systems of CAVs, whereas limited efforts have been put into decision-making systems, which is the highlight of this paper. As the CAVs need to interact with numerous background vehicles (BVs) for a long duration, variables that define the corner cases are usually high-dimensional, which makes the generation a challenging problem. In this paper, a unified framework is proposed to generate corner cases for decision-making systems. To address the challenge brought by high dimensionality, the driving environment is formulated based on the Markov decision process, and the deep reinforcement learning techniques are applied to learn the behavior policy of BVs. With the learned policy, BVs behave and interact with the CAVs more aggressively, resulting in more corner cases. To further analyze the generated corner cases, the techniques of feature extraction and clustering are utilized. By selecting representative cases of each cluster and outliers, the valuable corner cases can be identified from all generated corner cases. Simulation results of a highway driving environment show that the proposed methods can effectively generate and identify the valuable corner cases.
12 May 2019 in Transportation Research Record: Journal of the Transportation Research Board
Signal phase and timing (SPaT) information is critical for many in-vehicle applications. However, it is challenging and time-consuming to acquire city-wide SPaT information from local traffic management agencies directly. A significant limitation of existing SPaT information estimation methods in the literature is that they can only be applied to a specified time-of-day (TOD) period. In the real-world, however, different TOD timing plans are used to accommodate fluctuations in traffic demands. In this paper, we propose a novel method for traffic light parameter estimation based on floating car data, which features recognizing TOD breakpoints and can thus be applied to intersections with multi-TOD timing plans. Also, good estimation of TOD breakpoints leads to more data availability for estimation of other parameters. The proposed method is tested with real-world data collected from the DiDi on-line hailing platform in China. The filed test results show promising accuracy. The absolute error of green duration is within 3 s in daytime and the estimation error of TOD breakpoints is within 15 min.
10 May 2024 in IEEE Transactions on Intelligent Transportation Systems
Real-time safety metrics are important for automated driving systems (ADS) to assess the risk of driving situations and assist in decision-making. Although a number of real-time safety metrics have been proposed in the literature, there is a lack of systematic performance evaluations of these metrics. As different behavioral assumptions are adopted in different safety metrics, it is difficult to compare the safety metrics and evaluate their performance. To overcome this challenge, in this study, we propose an evaluation framework utilizing logged vehicle trajectory data so that vehicle trajectories for both the subject vehicle (SV) and background vehicles (BVs) are obtained and the prediction errors caused by behavioral assumptions can be eliminated. Specifically, we examine whether the SV is in a collision unavoidable situation at each moment, given all near-future trajectories of BVs. In this way, we level the ground for a fair comparison of different safety metrics, as a good safety metric should always alarm in advance to the collision unavoidable moment. When trajectory data from a large number of trips are available, we can systematically evaluate and compare different metrics’ statistical performance. In the case study, three representative real-time safety metrics, including the time-to-collision (TTC), the PEGASUS Criticality Metric (PCM) and the Model Predictive Instantaneous Safety Metric (MPrISM), are evaluated using a large-scale simulated trajectory dataset. The results demonstrate that the MPrISM achieves the highest recall and the PCM has the best accuracy. The proposed evaluation framework is important for researchers, practitioners, and regulators to characterize different metrics, and to select appropriate metrics for different applications. Moreover, by conducting failure analysis on moments when a safety metric fails, we can identify its potential weaknesses, which can be valuable for potential refinements and improvements.