28 December 2021 in IEEE Transactions on Intelligent Transportation Systems
Highly Automated Vehicles (HAVs) must undergo strict safety testing before being released to the public. Mileage-based on-road testing suffers from unaffordable time costs and high safety risks. Simulated scenario-based testing has been found to be a trustworthy alternative for testing HAVs’ built-in algorithms and functionalities. Test automation is typically used to generate target scenarios. This approach facilitates customized testing and avoids wasting time on simple and redundant scenarios. This study aims to review test automation methods and discuss how to accentuate their strengths rather than be trapped in their weaknesses under certain applicable conditions. According to their main purposes, we classify test automation methods into coverage-oriented, unsafe-scenario-oriented, and naturalistic-assessment-oriented categories. To further demonstrate the differences of these methods, we then design numerical experiment to compare the capabilities of seven test automation methods. Finally, we compile our observations to form a comprehensive guide for selecting test automation methods with different test requirements in mind.
3 December 2021 in IEEE Transactions on Intelligent Transportation Systems
Automated Vehicles (AVs) need to be thoroughly evaluated in order to ensure their driving capabilities. However, comprehensive evaluations are intractable due to both time and monetary costs. To address this problem, we propose an Adaptive Design of Experiments (ADOE) method to evaluate the safety of AVs. Using this method, a Surrogate Model (SM) is established and updated iteratively. SM in ADOE is used to approximate the results of AV testing and help to select the next concrete scenario to be tested in each iteration. Two different ADOE approaches are proposed in this study for different testing purposes. Since the choice of the surrogate model has a profound impact on the performance of the ADOE method, 6 surrogate models were compared with two logical scenarios at different scales – a car following logical scenario and a cut-in logical scenario. Results show that Extreme Gradient Boosting (XGB) is suitable for both ADOE approaches. And both proposed ADOE approaches achieved desired performance. Scenario-oriented ADOE made full use of each concrete scenario, capturing one collision case for every 1.12 test runs in the car following logical scenario, while SM-oriented ADOE successfully depicted the boundary between safety and danger. Using 0.46% test resources compared to enumeration, the SM-oriented ADOE found 93.9% dangerous scenarios with 90.9% precision. ADOE approaches have great potential in accelerating the evaluation of AV safety.
26 September 2023 in IEEE Transactions on Intelligent Vehicles
With the gradual perfection of Highly Automated Vehicles (HAVs), it is obligatory to assess their safety performance in simulation that mirrors the real-world driving environment. However, the minimal likelihood of exposure to risky events can result in an extremely time-consuming testing process. To address this issue, we applied a surrogate-based method to expedite scenario-based simulated safety testing for HAVs. Model-based surrogates can quickly approximate the results of untested scenarios, thereby facilitating the search for risky scenarios. Car-following and Cut-in scenarios were chosen as two representative Operational Design Domains (ODDs) with different dimensions for case study. Thus, the capabilities of various Surrogate Models (SMs) can be examined in depth. Utilizing the HighD data, two testing ODDs were constructed to be consistent with naturalistic distribution. We demonstrated that the performances of six mainstream SMs differ significantly as the frequency of risky scenarios decreases. Additionally, we conducted multiple rounds of tests to compare the stability of SMs. We also presented a proposal on SMs selection according to the complexity of ODDs and the rarity of risky scenarios. Compared with random testing, the surrogate-based method can search for 4 times as many high-risk Car-following scenarios with only 4% of the test resources, showing great potential in accelerating the testing process. Notably, when the targeted scenarios are not rare in high-dimensional ODD, the calculation simplicity of SMs is the most important factor. Even random testing can be a viable option in such circumstances.
1 December 2023 in IEEE Transactions on Intelligent Transportation Systems
To validate whether Highly Automated Vehicles (HAVs) can live up to human expectations, it is essential to estimate their risk rate within the context of the naturalistic driving environment. Due to the low probability of exposure to risky events, the testing process is exceedingly time-consuming. To tackle this issue, we proposed a Surrogate-based Monte Carlo Method to accelerate the risk assessment for HAVs in scenario-based simulation. Surrogate Models (SMs) were utilized to approximate the outcomes of untested scenarios, hence facilitating the identification of risky scenarios. Therefore, the large number of samples required by Monte Carlo did not need to be tested entirely. Naturalistic distributions fitted from HighD data was used to generate samples. The Car-following and Cut-in scenarios were selected for the case study, as they represented two distinct testing spaces. As such, the capabilities of different SMs can be further analyzed. We proved that the performances of six mainstream SMs were greatly distinguished from each other. Inverse Distance Weighted (IDW), the proven most capable SM, was combined with Monte Carlo. Compared with Monte Carlo, up to 95% of tested samples can be saved. Compared with the Importance Sampling method, another popular improvement in Monte Carlo simulation, the proposed method can further save 58.1% of CPU execution time.
7 May 2025 in IEEE Transactions on Vehicular Technology
The advancement of Highly Automated Vehicles (HAVs) safety testing plays a crucial role in the large-scale deployment of Automated Driving Systems (ADS). Scenario-based testing has become increasingly popular due to its customizability and testing efficiency. Unfortunately, the scarcity of collected realworld scenario data constrains the construction of a high-coverage scenario library. It is of great value to make small data into big data by generating fresh new scenarios out of existing ones. A common approach is to use Data-driven Scenario Generation (DSG) methods. However, these methods suffer from some drawbacks, including the low quality of the generated scenarios (i.e., the scenario fidelity) and the lack of diversity and criticality compared to the original scenarios (i.e., the scenario directivity). These problems are mainly caused by the complex and dynamic nature of scenarios and the rarity of critical situations in naturalistic driving environment. To address those challenges, we propose a data-driven method named the Multi-vehicle Joint Trajectory Generator (MJTG), devised to selectively generate potentially feasible scenarios based on limited collected scenarios. The MJTG framework has demonstrated significant effectiveness in terms of fidelity and directivity. Generated scenarios exhibit over 85% compliance with functional check standards. Additionally, the framework excels in addressing directivity, resulting in a reduction of 0.66 seconds in the modified time-tocollision. Overall, the MJTG framework holds promising prospects in constructing high-coverage test-worthy scenario libraries and advancing the safety testing of HAVs.
31 October 2023 in Transportation Research Record: Journal of the Transportation Research Board
Microscopic traffic simulation is vital to assess the performances of various traffic operation and management schemes. Microscopic traffic simulation is usually not parameter-free, and it relies on independent parameters to predict traffic evolution. Thus, parameter calibration is indispensable to conveying trustworthy simulation results. Heuristic algorithms are widely used for parameter calibration. Its logic is for achieving iterative optimization through continuous trial-and-error simulations. This process is time-consuming and usually takes several hours, making the calibration unable to meet the requirements of speed and efficiency. In recent years, parallel computing technology has been gradually applied in the engineering realm, which makes rapid calibration possible. Following the three steps of parallel framework selection, algorithm bottleneck identification, and subtask load balancing, this paper designs and implements the parallelization of genetic algorithm and particle swarm optimization (PSO) calibration algorithms. Finally, the proposed parallel framework is applied to simulation parameter calibration of a section of a 5 km long highway in Australia, and the effectiveness of parallel computing is evaluated from the two dimensions of reduction in calibration computational time and scalability. The results show that the proposed parallel calibration algorithm can shorten the 5 h calibration process to less than 1 h, reducing the calibration time by 80%. The parallel PSO calibration algorithm has better scalability, and its acceleration effect is better when more processors are used.