Energy management strategy is an important factor in determining the fuel economy of hybrid electric vehicles; thus, much research on how to distribute the required power to engines and motors of hybrid vehicles is required. Recently, various studies have been conducted based on reinforcement learning to optimally control the hybrid electric vehicle. In fact, the fundamental control approach of reinforcement learning shares many control frameworks with the control approach by using deterministic dynamic programming or stochastic dynamic programming. In this study, we compare the reinforcement learning based strategy by using these dynamic programming-based control approaches. For optimal control of hybrid electric vehicle, each control method was compared in terms of fuel efficiency by performing simulation by using various driving cycles. Based on our simulations, we showed the reinforcement learning-based strategy can obtain global optimality in the optimal control problem with an infinite horizon, which can also be obtained by stochastic dynamic programming. We also showed that the reinforcement learning-based strategy can present a solution close to the optimal one using deterministic dynamic programming, while a reinforcement learning-based strategy is more appropriate for a time variant controller with boundary value constraints. In addition, we verified the convergence characteristics of the control strategy based on reinforcement learning, when transfer learning was performed through value initialization using stochastic dynamic programming.

With the development of autonomous vehicles, research on energy-efficient eco-driving is becoming increasingly important. The optimal control problem of determining the speed profile of the vehicle for minimizing energy consumption is a challenging problem that necessitates the consideration of various aspects, such as the vehicle energy consumption, slope of the road, and driving environment, e.g., the traffic and other vehicles on the road. In this study, an approach using reinforcement learning was applied to the eco-driving problem for electric vehicles considering road slopes. A novel model-based reinforcement learning algorithm for eco-driving was developed, which separates the vehicle’s energy consumption approximation model and driving environment model. Thus, the domain knowledge of vehicle dynamics and the powertrain system is utilized for the reinforcement learning process, while model-free characteristics are maintained by updating the approximation model using experience replay. The proposed algorithm was tested via a vehicle simulation and compared with a solution obtained using dynamic programming (DP), and as well as conventional cruise control driving with constant speed. The simulation results indicated that the speed profile optimized using model-based reinforcement learning had similar behavior to the global solution obtained via DP and energy saving performance compared with cruise control.

Fuel cell electric vehicles use fuel cells as their main power source; the vehicle is driven by an electric motor, and have an electric battery as a secondary power source that stores regenerative braking energy and assists driving. To reduce the hydrogen fuel consumption by using these fuel cells and electric batteries efficiently, an energy management strategy is needed for the proper distribution of power among them. In this study, model-based reinforcement learning was utilized for energy management. For the optimal control of a fuel-cell electric vehicle, reinforcement learning is conducted using an internal vehicle powertrain model in the learning algorithm; initially, the model is completely unknown, but the model is learned with data from experiences as the learning process progresses. Then, reinforcement learning is conducted for the environment of the driving cycle profile to optimize the control policy. In this study, vehicle simulation was conducted using standard driving cycles, and the results showed that the learning process converged steadily and that the powertrain model was well learned. The simulated fuel consumption values show that the proposed algorithm reduces fuel consumption compared to the rule-based strategy by an average of 5.7%.

The energy management strategy of a hybrid electric vehicle directly determines the fuel economy of the vehicle. As a supervisory control strategy to divide the required power into its multiple power sources, engines and batteries, many studies have been conducting using rule-based and optimization-based approaches for energy management strategy so far. Recently, studies using various machine learning techniques have been conducted. In this paper, a novel control framework implementing Model-based Q-learning is developed for the optimal control problem of hybrid electric vehicles. As an online energy management strategy, a new approach could learn the characteristics of a current given driving environment and adaptively change the control policy through learning. Especially, for the proposed algorithm, the internal powertrain environment and external driving environment are separated so they can be learned via the reinforcement learning framework, which results in a simpler and more intuitive control strategy that can be explained using the vehicle state approximation model. The proposed algorithm is tested and verified through simulations, and the simulation results present near optimal solution. The simulation results are compared with conventional rule-based strategies and optimal control solutions acquired from Dynamic Programming.

Hybrid electric vehicles, operated by engines and motors, require an energy management strategy to achieve competitive fuel economy performance. The equivalent consumption minimization strategy is a well-known algorithm that can be employed for the energy management of hybrid electric vehicles, based on the concept of the equivalent cost of fossil fuels and electric battery energy. However, in the equivalent consumption minimization strategy approach, a parameter called the equivalent factor should be determined to obtain the optimal control policy. In this study, reinforcement learning based approaches are proposed to determine the equivalent factor. First, we show that the equivalent factor can be indirectly extracted from the reinforcement learning results, using the control action from reinforcement learning for the specific driving cycle. In addition, a novel approach that combines reinforcement learning and the equivalent consumption minimization strategy is proposed, where the equivalent factor is determined based on the interaction between the reinforcement learning agent and driving environment, while the control input is decided by the equivalent consumption minimization strategy based on the determined equivalent factor. A model-based reinforcement learning method is used, and the proposed method is validated for vehicle simulation using a parallel hybrid electric vehicle. The simulation results show that the proposed method can achieve a near-optimal solution, which is close to the global solution obtained with the dynamic programming approach (96.7% compared to dynamic programming result in average), and improved performance of 4.3% in average compared with the existing adaptive equivalent consumption minimization strategy.

Equivalent Consumption Management Strategy (ECMS), a representative energy management strategy for hybrid electric vehicles (HEVs) derived from Pontryagin's minimum principle, is known to produce a near-optimal solution if the costate or equivalent factor of electric use is appropriately determined according to the driving conditions. One problem when applying the control concept to real-world scenarios is that it is difficult to precisely evaluate the performance of the control parameter before driving is complete, so the costate cannot be determined properly. To address this issue, this study proposes a practical method for estimating an appropriate costate based on Deep Q-Networks (DQNs), which is a reinforcement learning algorithm that uses a Deep Neural Network to evaluate the performances and determine the best control parameter or costate. The control concept benefits vehicle energy management by selecting the control parameter most related to stochastic conditions or future driving information based on artificial intelligence (AI), while optimal control is deterministically conducted by ECMS if the control parameter is given. Simply, only the implicit part of the optimal controller is solved via artificial intelligence. In the simulation results, not only does the proposed control concept outperform an existing ECMS that uses an adaptive technique for determining the costate, but the concept is also very feasible, in that it does not need a model for evaluating the performances.