With the development of artificial intelligence, there has been a growing interest in machine learning-based control strategy, among which reinforcement learning (RL) has opened up a new direction in the field of hybrid electric vehicle (HEV) energy management. However, the issues of the current RL setting ranging from inappropriate battery state-of-charge (SOC) constraint to ineffective and risky exploration make it inapplicable to many industrial energy management strategy (EMS) tasks. To address this, an adaptive hierarchical EMS combining heuristic equivalent consumption minimization strategy (ECMS) knowledge and deep deterministic policy gradient (DDPG), which is a state-of-the-art data-driven RL algorithm, is proposed in this work. For comparison purposes, the proposed strategy is contrasted with dynamic programming (DP), proportion integration differentiation (PID)-based adaptive ECMS, and rule-based and standard RL-based counterparts, and the results show that the fuel consumption after SOC correction for the proposed strategy is very close to that of the DP-based control and lower than that of the other three benchmark strategies. Considering that the proposed strategy can make better use of the RL techniques while realizing an effective, efficient, and safe exploration in a data-driven manner, it may become a strong foothold for future RL-based EMS to build on, especially when the controller has to be trained directly and from scratch in a real-world environment.

With the development of recent artificial intelligence technology, especially after the great success of AlphaGo, there has been a growing interest in applying reinforcement learning (RL) to solve energy management strategy (EMS) problems for hybrid electric vehicles. However, the issues of current RL algorithms including deployment inefficiency, safety constraint, and simulation-to-real gap make it inapplicable to many industrial EMS tasks. With these in mind and considering the fact that there exists many suboptimal EMS controllers which can generate plentiful amounts of interactive data containing informative behaviors, an offline RL training framework that tries to extract policies with the maximum possible utility out of the available offline data is proposed. Furthermore, with connected vehicle technology standard in many new cars, rather than bringing all the data to the storage and analytics, a scheduled training framework is put forward. This cloud-based approach not only alleviates the computational burden of edge devices, but also more importantly provides a deployment-efficient solution to EMS tasks that have to adapt to changes of driving cycle. To evaluate the effectiveness of the proposed algorithm on real controllers, a hardware-in-the-loop (HIL) test is performed and the superiority of the proposed algorithm in contrast to dynamic programming, behavior cloning, rule-based, and vanilla off-policy RL algorithms is given.

For the ongoing revolution in developing Intelligent and Connected Vehicles (ICVs), there is a lack of research for powertrain control system using the latest artificial intelligence and V2X (Vehicle-to-everything) technology that have already been widely adopted in the autonomous driving systems. In this context, a recent development of deep reinforcement learning (DRL) and one of the latest computing framework are coupled to facilitate an onboard-based intelligent powertrain control. Taking the boost control of a diesel engine equipped with variable geometry turbocharger as an example, the results show that the final control behavior indicated by the cumulated rewards is improved by 50.43% and the learning efficiency is improved by 74.29% for the proposed curiosity-driven DRL algorithm, compared to the same structure DRL algorithm with classic random exploration policy. In addition, unlike most of DRL-based powertrain optimization algorithms which have only been applied to single-machine architecture, this work manages the proposed DRL algorithm in parallel and more importantly from an edge computing perspective. This, in addition to greatly speeding up the algorithm training, can also realize a good balance of control accuracy and generality depending on the selected training scenario.

Deep reinforcement learning (DRL) is an area of machine learning that combines a deep learning approach and reinforcement learning (RL). However, there seem to be few studies that analyze the latest DRL algorithms on real-world powertrain control problems. Meanwhile, the boost control of a variable geometry turbocharger (VGT)-equipped diesel engine is difficult mainly due to its strong coupling with an exhaust gas recirculation (EGR) system and large lag, resulting from time delay and hysteresis between the input and output dynamics of the engine’s gas exchange system. In this context, one of the latest model-free DRL algorithms, the deep deterministic policy gradient (DDPG) algorithm, was built in this paper to develop and finally form a strategy to track the target boost pressure under transient driving cycles. Using a fine-tuned proportion integration differentiation (PID) controller as a benchmark, the results show that the control performance based on the proposed DDPG algorithm can achieve a good transient control performance from scratch by autonomously learning the interaction with the environment, without relying on model supervision or complete environment models. In addition, the proposed strategy is able to adapt to the changing environment and hardware aging over time by adaptively tuning the algorithm in a self-learning manner on-line, making it attractive to real plant control problems whose system consistency may not be strictly guaranteed and whose environment may change over time.

Deep reinforcement learning (DRL) excels at playing a wide variety of simulated games and allows for a generic learning process that does not consider a specific knowledge of the task. However, due to the fact that a large prohibitively number of interactions with the environment are required and that the initial policy behavior is almost random, such an algorithm cannot be trained directly in a real-world environment while satisfying given safety constraints. In this article, a control framework based on DRL that shifts toward training directly in the transient real-world environment is proposed. This research is working on the assumption that some demonstration knowledge that operates under previous controllers and an abstract of the agent environment dynamics are available. By encoding this prior knowledge into a sophisticated learning architecture, a warm-starting DRL algorithm with a safe exploration guarantee can be anticipated. Taking the boost control problem for a variable geometry turbocharger equipped diesel engine as an example, the proposed algorithm improves the initial performance by 74.6% and the learning efficiency by an order of magnitude in contrast to its vanilla counterpart. Compared with other existing DRL-based powertrain control methods, the proposed algorithm can realize the “model-free” concept in the strict sense, making it attractive for future DRL-based powertrain control algorithms to build on.

Reinforcement learning (RL) based techniques have been employed for the tracking and adaptive cruise control of a small-scale vehicle with the aim to transfer the obtained knowledge to a full-scale intelligent vehicle in the near future. Unlike most other control techniques, the purpose of this study is to seek a practical method that enables the vehicle, in the real environment and in real time, to learn the control behavior on its own while adapting to the changing circumstances. In this context, it is necessary to design an algorithm that symmetrically considers both time efficiency and accuracy. Meanwhile, in order to realize adaptive cruise control specifically, a set of symmetrical control actions consisting of steering angle and vehicle speed needs to be optimized simultaneously. In this paper, firstly, the experimental setup of the small-scale intelligent vehicle is introduced. Subsequently, three model-free RL algorithm are conducted to develop and finally form the strategy to keep the vehicle within its lanes at constant and top velocity. Furthermore, a model-based RL strategy is compared that incorporates learning from real experience and planning from simulated experience. Finally, a Q-learning based adaptive cruise control strategy is intermixed to the existing tracking control architecture to allow the vehicle slow-down in the curve and accelerate on straightaways. The experimental results show that the Q-learning and Sarsa (λ) algorithms can achieve a better tracking behavior than the conventional Sarsa, and Q-learning outperform Sarsa (λ) in terms of computational complexity. The Dyna-Q method performs similarly with the Sarsa (λ) algorithms, but with a significant reduction of computational time. Compared with a fine-tuned proportion integration differentiation (PID) controller, the good-balanced Q-learning is seen to perform better and it can also be easily applied to control problems with over one control actions.