6 March 2023 in IEEE Transactions on Intelligent Transportation Systems
This paper aims to solve the trajectory tracking control problem for an autonomous vehicle based on reinforcement learning methods. Existing reinforcement learning approaches have found limited successful applications on safety-critical tasks in the real world mainly due to two challenges: 1) sim-to-real transfer; 2) closed-loop stability and safety concern. In this paper, we propose an actor-critic-style framework SRL-TR $^2$ , in which the RL-based TRajectory TRackers are trained under the safety constraints and then deployed to a full-size vehicle as the lateral controller. To improve the generalization ability, we adopt a light-weight adapter State and Action Space Alignment (SASA) to establish mapping relations between the simulation and reality. To address the safety concern, we leverage an expert strategy to take over the control when the safety constraints are not satisfied. Hence, we conduct safe explorations during the training process and improve the stability of the policy. The experiments show that our agents can achieve one-shot transfer across simulation scenarios and unseen realistic scenarios, finishing the field tests with average running time less than 10 ms/step and average lateral error less than 0.1 m under the speed ranging from 12 km/h to 18 km/h. A video of the field tests is available at https://youtu.be/pjWcN_fV24g.
11 June 2024 in IEEE Transactions on Intelligent Vehicles
Recently, the autonomous driving technology is at a critical phase evolving from typical, closed scenarios to largescale, open driving scenarios, which is challenged by the diversity and complexity of the driving environment. Reinforcement learning has the potential to address this challenge, but its application has been hindered by safety issues. To address the safety concern, we propose a Lagrangian framework called Knowledge-Enhanced Safe Reinforcement Learning (KESRL), which integrates prior knowledge of model-based methods into the model-free safe RL framework. Firstly, a model-based constraint based on high-order control barrier function is derived to deduce the safety criteria for the control system. Secondly, we propose a model-free constraint which estimates the distribution of accumulated safety cost and restricts the worst-case safety cost value. Furthermore, the above constraints are unified into the proposed Lagrangian framework to equip the end-to-end autonomous driving system with safety constraints. Experimental results conducted in the CARLA simulator demonstrate the effectiveness of the proposed framework KESRL in decreasing the collision rate and enhancing the control safety compared with several state-of-the-art RL methods.
9 April 2025 in IEEE Internet of Things Journal
Developing trustworthy reinforcement learning (RL) agents for safety-critical control tasks, such as end-to-end autonomous driving, has been a longstanding challenge due to low sample efficiency. Prior works have attempted to address this challenge by performing self-supervised auxiliary tasks like self-reconstruction or predicting long-term future states. However, there still remain unexplored sequential features and causality relationships inherent in sequential state, action, and reward signals in the frequency domain. To fully exploit the temporal and frequential features, we propose a contrastive RL framework called brain-inspired causality-aware contrastive learning (BRYANT) to achieve efficient representation learning and human-like autonomous driving. Different from existing temporal predictive methods, we transform the sequential latent representations, reward, and action signals into the frequency domain, followed by the symmetric temporal prediction pattern for real and imaginary parts of the frequential signals. To capture the temporal causality for the latent representations, we introduce a brain-inspired network structure called Closed-form Continuous-time (CfC) network to parameterize the derivative of the latent representations and establish the neural dynamic model. Experimental results conducted in the CARLA simulator demonstrate the effectiveness of BRYANT in efficient representation learning, enabling agents to concentrate on potential risks and decrease the collision rate compared to several state-of-the-art RL methods. Furthermore, through the visualization of the latent representation prediction process, we reveal the causal relationships between the critic Q values and the latent representation vectors in the frequency domain, and demonstrate the effectiveness of the frequency-domain prediction.