Reinforcement learning (RL) based energy management is one of the current hot-spots of hybrid electric vehicles. Recent advances in RL based energy management focus on energy-saving performance but less considers the constrained setting for training safety. This paper proposes an RL framework named Coach-Actor-Double Critic (CADC) for the optimization of energy management considered as the constrained Markov Decision Process (CMDP). A bi-level onboard controller includes a neural network (NN) based strategy Actor and rule-based strategy Coach for online energy management. Once the output of Actor exceeds the constrained range of feasible solutions, Coach would take charge of energy management to ensure safety. By using Lagrangian relaxation, the optimization for constrained Markov decision process transforms into an unconstrained dual problem to minimize the energy consumption while minimizing the Coach participation. The parameters of Actor are updated in a manner of policy gradient through RL training with the Lagrangian value function. Double Critic with the same structure synchronously estimate the value function to avoid overestimate bias. Several experiments with the bus trajectories data demonstrate the optimality, self-learning ability, and adaptability of CADC. The results indicate that CADC outperforms the existing RL based strategies and reaches above 95% energy-saving rate of the offline global optimum.
Coordinating a platoon of connected hybrid electric vehicles (HEVs) poses challenges due to the intricacy of their powertrains and the diverse driving scenarios encountered. The existing mainstream framework uses a hierarchical control scheme, simplifying the unified optimization problem into two separate series control processes: the powertrain level and the vehicle level. However, this approach overlooks the inherent interdependence between the vehicle and powertrain systems, which can hinder effective optimization and collaboration in terms of energy management across multiple vehicles. To address this problem, a multi-agent reinforcement learning-based energy control framework is proposed, aiming to unleash the energy-saving potential through an integrated collaborative optimization of velocity optimization and energy management strategy for HEV platoon. The proposed strategy constructs a joint-goals value function based on Markov games for HEV platooning and utilizes long short-term memory networks to capture temporal associations of the platoon dynamics. In addition, an asynchronous reinforcement learning method is introduced for knowledge sharing among HEVs in the platoon. The simulation results demonstrate that the proposed approach effectively improves driving behavior and powertrain energy efficiency through multi-vehicle coordination. Compared to the rule-based baseline, the fuel consumption of the platoon is reduced by 19.2% through the coordination of connected HEVs.
26 October 2020 in IEEE Transactions on Intelligent Transportation Systems
Understanding the underlying patterns of the urban mobility dynamics is essential for both the traffic state estimation and management of urban facilities and services. Due to the coupling relationship of generative factors in spatial-temporal domain, it is challenging to model the citywide traffic dynamics under a structural pattern of critical features such as hours of days, days of weeks and weather conditions. To address this challenge, this article develops a disentangled representation learning framework to learn an interpretable factorized representation of the independent data generative factors. In order to make full use of the knowledge on generative factors, this article proposes spatial-temporal generative adversarial network (ST-GAN) to assign the generative factors of traffic flow to the feature vector in latent space and reconstructs the high-dimensional citywide traffic flow from the given factors. With the help of the disentangled representations, the decomposed feature vector in latent space discloses the relationship between underlying patterns and citywide traffic dynamics. Several comprehensively experiments show that ST-GAN not only effectively improves the prediction accuracy but also promisingly characterize structural properties of the traffic evolution process.
23 December 2021 in IEEE Transactions on Transportation Electrification
Cooperative control of connected and automated electric vehicles (CAEVs) offers a great potential for a safe, high-efficient, sustainable transportation system. Among them, coordinated on-ramp merging based on an on-ramp merging vehicle and a mainline facilitating is a hot spot to lighten the impact of shockwave on highway junctions. Reinforcement learning (RL) is a promising solution to address this problem for its strong adaptiveness and self-learning ability. The existing methods are dedicatedly designed to circumvent the sparse reward problem of long-term merging safety but restrict the optimization space and neglect individual costs, especially irreversible loss, such as battery aging. In this article, a hindsight experience replay RL coordinated freeway on-ramp merging frame is proposed to face the sparse reward problem in ramp merging problem and, meanwhile, co-optimize traffic efficiency, energy-saving, and battery health performance. Compared with the existing optimization-based strategy, it fundamentally avoids abundant expert knowledge and elaborate design for reward shaping and model construction while ensuring the nature of merging tasks without excessive simplification. Numerical experiments are conducted to verify the optimality, adaptability, and self-learning ability of the proposed strategy. With comparison experiment, the proposed strategy surpasses state-of-the-art RL methods more than 13% in overall index throughput with 12% energy consumption while easing individual battery aging during the merging process.
Mileage anxiety is one of the most important factors that affect the driving experience due to the limitation of battery capacity. Robust and accurate prediction of the energy consumption of the journey of the electric vehicle can guide the driver to allocate the power rationally and relieve the anxiety of the mileage. Since vehicle sharing is the biggest application scenario of electric vehicles, it is a critical challenge in share mobility research area. In this paper, a travel energy consumption prediction model of electric vehicles is proposed in order to improve the mobility of shared cars and reduce the anxiety of drivers because they are worried about insufficient power. A recurrent neural network with attention mechanism and deep neural network is used to build the model. To validate the proposed model, a simulation is demonstrated based on both traffic and vehicle information. After the simulation, experimental results show that the proposed model has high prediction accuracy, and we also show through visualization how the model finds high relevant road segments of the road network while dealing with corresponding traffic state input.