Reinforcement learning has shown success in solving complex control problems, yet safety remains paramount in engineering applications like energy management systems (EMS), particularly in hybrid electric vehicles (HEVs). An effective EMS is crucial for coordinating power flow while ensuring safety, such as maintaining the battery state of charge within safe limits, which presents a challenging task. Traditional reinforcement learning struggles with safety constraints, and the penalty method often leads to suboptimal performance. This study introduces Lagrangian-based parameterized soft actor–critic (PASACLag), a novel safe hybrid-action reinforcement learning algorithm for HEV energy management. PASACLag utilizes a unique composite action representation to handle continuous actions (e.g., engine torque) and discrete actions (e.g., gear shift and clutch engagement) concurrently. It integrates a Lagrangian method to separately address control objectives and constraints, simplifying the reward function and enhancing safety. We evaluate PASACLag’s performance using the World Harmonized Vehicle Cycle (901 s), with a generalization analysis of four different cycles. The results indicate that PASACLag achieves a less than 10% increase in fuel consumption compared to dynamic programming. Moreover, PASACLag surpasses PASAC, an unsafe counterpart using penalty methods, in fuel economy and constraint satisfaction metrics during generalization. These findings highlight PASACLag’s effectiveness in acquiring complex EMS for control within a hybrid action space while prioritizing safety.

Autonomous lane-change, a key feature of advanced driver-assistance systems, can enhance traffic efficiency and reduce the incidence of accidents. However, safe driving of autonomous vehicles remains challenging in complex environments. How to perform safe and appropriate lane change is a popular topic of research in the field of autonomous driving. Currently, few papers consider the safety of reinforcement learning in discretionary lane-change scenarios. We introduce safe hybrid-action reinforcement learning into discretionary lane change for the first time and propose the Parameterized Soft Actor–Critic with PID Lagrangian (PASAC-PIDLag) algorithm. Furthermore, we conduct a comparative analysis with Parameterized Soft Actor–Critic (PASAC), which is an unsafe version of PASAC-PIDLag. Both algorithms are employed to train the lane-change strategy to output both discrete lane-change decisions and continuous longitudinal vehicle acceleration. Our simulation results indicate that at a traffic density of 15 vehicles per kilometer (15 veh/km), the PASAC-PIDLag algorithm exhibits superior safety with a collision rate of 0%, outperforming the PASAC algorithm, which has a collision rate of 1%. The generalization assessments reveal that at low traffic density levels, both the PASAC-PIDLag and PASAC algorithms are proficient in attaining zero collision rates. However, at high traffic density levels, although both algorithms result in collisions, PASAC-PIDLag has a much lower collision rate than PASAC.

The co-optimization of adaptive cruise control (ACC) and energy management strategy (EMS) can improve the energy-saving capability of a hybrid electric vehicle (HEV). Co-optimization for a series-parallel HEV involves continuous control variables, such as the engine speed and torque, and discrete control variables, such as the clutch engagement decision. To solve the mixed-integer optimal control problem, we propose a hybrid-action reinforcement learning (HARL) algorithm called parameterized proximal policy optimization (PA-PPO) for series-parallel HEV co-optimization of ACC and EMS to minimize the clutch engagement/disengagement times, energy consumption, and guarantee the car-following performance. Firstly, the ACC model and the control-oriented HEV EMS model considering the clutch engagement decision control are constructed and combined to form the co-optimization model. Secondly, the PA-PPO algorithm is proposed and trained to solve the co-optimization problem. Compared to model predictive control (MPC), the proposed PA-PPO algorithm achieves similar fuel consumption while significantly reducing clutch engagement/disengagement times and jerk values. In addition, the computation time of PA-PPO is five-orders-of-magnitude smaller than MPC. Finally, the PA-PPO control performance is validated using a Simulink high-fidelity model (HFM).