In this article, we address the difficulty of controlling unmanned surface vehicles (USVs) under unforeseeable and unobservable external disturbances using model-based reinforcement learning (MBRL) without human’s prior knowledge. A novel MBRL approach, filtered probabilistic model predictive control (FPMPC) is proposed to iteratively learn the USV model and an MPC-based policy in a probabilistic way through trial-and-error interactions. Compared with existing MBRL approaches that model the unobservable disturbances as system noise, FPMPC introduces a Bayesian filter process to implicitly translate the system dynamics to a partially-observed Markov decision process to present those disturbances as hidden states. An adaptive sample selection is proposed to remove the redundant learning samples based on the filter belief. Equipped with bias compensation and parallel computation, an FPMPC system, specific for USV, is developed. Evaluated by both position holding and target reaching tasks in a real USV data-driven simulation, FPMPC shows its significant superiority in control performances, generalization capability, and sample efficiency under large disturbances compared with the baseline approaches.
11 January 2024 in IEEE Transactions on Vehicular Technology
Autonomous vehicles have received considerable attention, yet high-speed path following control remains a critical and challenging issue. At high speeds, achieving perfect control based on an accurate dynamic model of autonomous vehicles is difficult due to the nonlinear model parameters that change at the driving limit. Moreover, the strong coupling relationship between longitudinal control and lateral control also affects control performance. To address these challenges, a reinforcement learning-based control approach (RL-controller) is developed in this paper and applied to the high-speed path following task. The proposed RL-controller overcomes the challenges of explicit modeling and tight coupling control. Firstly, to obtain efficient learning features, our RL-controller is developed using a deep soft actor-critic method that incorporates multi-layer dense connection networks with skip connections and feature reuse. Secondly, an estimation method of the lane curvature is developed, which is chosen as prior knowledge to augment the input state of reinforcement learning. We model the lane using a transformer-based network structure that utilizes a third-order polynomial and train the model on our self-collected dataset. Thirdly, intuitive reward functions are built from the cross-track error, the angle error and the sideslip angle. At last, we verify our proposed RL-controller utilizing an open autonomous driving simulator. Compared with imitation learning and model-based optimal control methods, our RL-controller outperforms them in terms of the driving speed and compound error.
Collective perception is a new paradigm to extend the limited horizon of individual vehicles. Incorporating with the recent vehicle-2-x (V2X) technology, connected and autonomous vehicles (CAVs) can periodically share their sensory information, given that traffic management authorities and other road participants can benefit from these information enormously. Apart from the benefits, employing collective perception could result in a certain level of transmission redundancy, because the same object might fall in the visible region of multiple CAVs, hence wasting the already scarce network resources. In this paper, we analytically study the data redundancy issue in highway scenarios, showing that the redundant transmissions could result in heavy loads on the network under medium to dense traffic. We then propose a probabilistic data selection scheme to suppress redundant transmissions. The scheme allows CAVs adaptively adjust the transmission probability of each tracked objects based on the position, vehicular density and road geometry information. Simulation results confirm that our approach can reduce at most 60% communication overhead in the meanwhile maintain the system reliability at desired levels.
10 October 2022 in IEEE Transactions on Intelligent Vehicles
Test the safety of Autonomous Driving Systems (ADS) with realistic traffic conditions is important to the insurance industry, legislators, and third-party technical services. However, the scarcity of risky driving events distributed in real-world driving often makes sampling inefficient. In this paper, we propose a unified and hierarchical testing framework for efficient and unbiased safety tests of ADS. We first extract the risk subspace from the Naturalistic Driving Data (NDD) with defined safety measures. Subsequently, we use the Gaussian Copula method to accurately model the subspace joint probability density function (PDF). We further formulate a Kriging model-based optimization problem for finding the appropriate Importance Sampling (IS) parameters, where the Kriging model is trained iteratively within limited computational resources. Eventually, the risky concrete scenarios library is generated using the obtained Importance Sampling parameters. With the simulation results skewered back to the original risk subspace, the ADS probability of failure (e.g., crash rate) can thus be accurately and unbiasedly estimated. Experiment results show that using 0.94% computational resources, the Kriging surrogate model captures 96.04% of ADS crashes with at least 90.25% precision. On crash rate estimation, our proposed method achieves a consistent result with that of Monte Carlo simulation, provided that the relative error does not exceed 10% while improving the testing efficiency by up to 6,539 times.