This paper proposes a multi-objective energy management strategy (EMS) based on multi-agent reinforcement learning (MARL) for a hybrid electric vehicle equipped with an engine-generator set (EGS) and a hybrid energy storage system (HESS, consisting of a battery and ultracapacitor). First, besides improving fuel economy, maintaining battery state of charge (SOC), reducing battery degradation and constraint on ultracapacitor SOC are also taken into consideration, formulating multi-objective energy management. Then, the problem is solved using MARL which combines game theory and reinforcement learning. In this framework, EGS and HESS are viewed as two intelligent agents respectively, and their interactions are described as a general-sum stochastic game. Following the principle of MARL, the two agents can learn the optimal control policy which guarantees the Nash equilibrium of multiple objectives, thus achieving a satisfactory balance among them. In the simulation, the MARL-based EMS is compared with single-agent reinforcement learning (SARL) which ignores the relations of different agents, and dynamic programming (DP) which integrates various targets into a single cost function with weight coefficients. The simulation results verify the superiority of the proposed EMS in optimizing multiple objectives.

As a fundamental task of hybrid electric vehicles (HEVs), energy management strategies are critical for improving the performance. This paper proposes a new queue-Dyna reinforcement learning (RL) algorithm based energy management strategy (EMS), which can substantially reduce the online learning time while guarantees the control performance compared with widely used Q-learning. To solve the existing problems of direct and indirect RL based EMSs, a blended RL algorithm, Dyna, is introduced first. By reusing the actual experience to construct a model online, Dyna integrates direct and indirect RL, and thus has the advantages of both. Furthermore, two novel strategies of backward focusing and prioritized sweeping are incorporated in the Dyna framework, developing the queue-Dyna algorithm. To the best of our knowledge, it's the first attempt of adopting queue-Dyna in the EMS of a HEV. A comparative simulation of direct RL, indirect RL, Dyna and queue-Dyna is implemented, and the results demonstrate the proposed algorithm achieves a great improvement in fast learning and maintains satisfied fuel consumption. At last, a hardware-in-the-loop experiment verified the real-time performance of the proposed EMS.

The stability of brake control is an important guarantee for the safety of heavy-duty vehicles (HDVs) at high speeds. However, the electro-hydraulic actuation braking systems often exhibit a significant delay in seconds, which makes braking performance forecasting and control difficult. To address the torque tracking control problem with time delay, a deep inference and control method is proposed. First, a theoretical delay time under different rotating speeds is identified with a data-driven model. Then, a fast end-to-end prediction model is established to estimate the torque performance of the next step with delay information. The deep Q-network (DQN) learning approach is proposed to learn the experimental data by exploring and seeking the optimal control strategy in the time delay environment. A comparative simulation of the proposed DQN-based controller with or without considering time delay, and the rule-based method with or without considering time delay is implemented, and an online processor-in-the-loop(PIL) test with the edge computing device NVIDIA Jetson Xavier NX is performed on the robustness condition. The simulation results and PIL test results demonstrate that the proposed control framework achieves a great improvement in the torque tracking effect with time efficiency.

To solve air pollution problems and reduce energy costs, hybrid electric vehicles (HEV) are applied for larger scale as a superior solution and their energy management strategies (EMS) are essential for achieving higher fuel efficiency. This paper proposes a reinforcement learning (RL) based real-time EMS for a HEV, which requires no prior knowledge of the future driving conditions. First, considering that the practical driving conditions are mutative, an online recursive Markov Chain model is constructed, which can depict the stochastic environment more accurately. Then, a novel RL framework, heuristic search, is introduced, which means using extra knowledge in the search process. It aims to find an optimal action for a specific state rather than learn a policy for an entire driving cycle like traditional RL algorithms. Consequently, the learning process is significantly efficient and real-time optimization can be realized. Moreover, a Q-table is learned offline, and is used online as a heuristic function to direct the search process, which can accelerate the convergence rate and improve the real-time performance further. In the simulation, the effectiveness of the recursive Markov Chain model and the superiority of the heuristic function are verified. Then, a comparative simulation of heuristic search, Q-learning, dynamic programming (DP) and equivalent consumption minimization strategy is conducted and evaluated. The proposed method consumes 1.63% more fuel compared with DP in a totally unknown cycle, which demonstrates its control performance and real-time capability.

The practical driving scenarios have a decisive influence on the performance of energy management strategies (EMSs) for hybrid electric vehicles (HEVs). How to adapt to various scenarios is still a challenging task for reinforcement learning (RL) based EMSs. To tackle this problem, the paper proposes an online-learning adaptive EMS based on a RL framework, Dyna. In the framework, besides directly improving the policy, the real experience from the vehicle is reused to establish an interactive model. Through the model, simulated experience is generated to meanwhile improve the policy and most high-priced real-world exploration is substituted by model-based exploration, which substantially reduces the learning time and lowers the learning price. Thus, Dyna-based EMS achieves fast and low-cost online learning when the HEV enters a new scenario, significantly enhancing the strategy’s adaptability. In the simulation, three typical driving scenarios are first selected for testing: city, suburb and highway. For all three scenarios, Dyna only cumulates about 60% driving cost of DQN in the first 100 cycles. In an off-road scenario, Dyna also achieves about 70% cumulative costs compared with DDPG and TD3 considering 100 cycles, which demonstrates the superiority in the adaptability of the proposed EMS.