With the development of artificial intelligence, there has been a growing interest in machine learning-based control strategy, among which reinforcement learning (RL) has opened up a new direction in the field of hybrid electric vehicle (HEV) energy management. However, the issues of the current RL setting ranging from inappropriate battery state-of-charge (SOC) constraint to ineffective and risky exploration make it inapplicable to many industrial energy management strategy (EMS) tasks. To address this, an adaptive hierarchical EMS combining heuristic equivalent consumption minimization strategy (ECMS) knowledge and deep deterministic policy gradient (DDPG), which is a state-of-the-art data-driven RL algorithm, is proposed in this work. For comparison purposes, the proposed strategy is contrasted with dynamic programming (DP), proportion integration differentiation (PID)-based adaptive ECMS, and rule-based and standard RL-based counterparts, and the results show that the fuel consumption after SOC correction for the proposed strategy is very close to that of the DP-based control and lower than that of the other three benchmark strategies. Considering that the proposed strategy can make better use of the RL techniques while realizing an effective, efficient, and safe exploration in a data-driven manner, it may become a strong foothold for future RL-based EMS to build on, especially when the controller has to be trained directly and from scratch in a real-world environment.

With the development of recent artificial intelligence technology, especially after the great success of AlphaGo, there has been a growing interest in applying reinforcement learning (RL) to solve energy management strategy (EMS) problems for hybrid electric vehicles. However, the issues of current RL algorithms including deployment inefficiency, safety constraint, and simulation-to-real gap make it inapplicable to many industrial EMS tasks. With these in mind and considering the fact that there exists many suboptimal EMS controllers which can generate plentiful amounts of interactive data containing informative behaviors, an offline RL training framework that tries to extract policies with the maximum possible utility out of the available offline data is proposed. Furthermore, with connected vehicle technology standard in many new cars, rather than bringing all the data to the storage and analytics, a scheduled training framework is put forward. This cloud-based approach not only alleviates the computational burden of edge devices, but also more importantly provides a deployment-efficient solution to EMS tasks that have to adapt to changes of driving cycle. To evaluate the effectiveness of the proposed algorithm on real controllers, a hardware-in-the-loop (HIL) test is performed and the superiority of the proposed algorithm in contrast to dynamic programming, behavior cloning, rule-based, and vanilla off-policy RL algorithms is given.

Energy management strategy (EMS) is the key technology to improving the fuel efficiency of hybrid electric vehicles (HEV). In recent years, the development of artificial intelligence has enabled tremendous advances by utilizing reinforcement learning (RL) for training and deploying deep neural network-based EMS. However, in contrast to the fields of deep learning, such as computer vision and natural language processing, which mainly rely on large-scale offline datasets, most RL-based policies must be trained online by trial-and-error with the initial performance being almost arbitrary. Such a paradigm is considered inefficient and unsafe for industrial automation and can only be used to tackle the EMS problems in the simulation world. Considering that large historical interactive datasets are readily available in the EMS domain, if an RL algorithm can be used to extract a policy purely offline from the prior collected dataset and improve upon data logging policy, the current issues, including sample inefficiency, unsafe exploration, and simulation-to-real gap that prevent the widespread use of RL methods, could be mitigated to a great extent. To this end, this article presents a feasible algorithmic framework for model-based offline RL. Unlike vanilla RL approaches without any consideration against distributional shift, a data-driven dynamic model is built before the policy training using RL. After that, two techniques, namely, conservative MDP and state regularization, are augmented, which are proved to be effective against model overexploitation. By incorporating the guidance of uncertainty awareness, a near optimal policy can be obtained by using only the dataset from a suboptimal controller.

For the ongoing revolution in developing Intelligent and Connected Vehicles (ICVs), there is a lack of research for powertrain control system using the latest artificial intelligence and V2X (Vehicle-to-everything) technology that have already been widely adopted in the autonomous driving systems. In this context, a recent development of deep reinforcement learning (DRL) and one of the latest computing framework are coupled to facilitate an onboard-based intelligent powertrain control. Taking the boost control of a diesel engine equipped with variable geometry turbocharger as an example, the results show that the final control behavior indicated by the cumulated rewards is improved by 50.43% and the learning efficiency is improved by 74.29% for the proposed curiosity-driven DRL algorithm, compared to the same structure DRL algorithm with classic random exploration policy. In addition, unlike most of DRL-based powertrain optimization algorithms which have only been applied to single-machine architecture, this work manages the proposed DRL algorithm in parallel and more importantly from an edge computing perspective. This, in addition to greatly speeding up the algorithm training, can also realize a good balance of control accuracy and generality depending on the selected training scenario.

Deep reinforcement learning (DRL) is an area of machine learning that combines a deep learning approach and reinforcement learning (RL). However, there seem to be few studies that analyze the latest DRL algorithms on real-world powertrain control problems. Meanwhile, the boost control of a variable geometry turbocharger (VGT)-equipped diesel engine is difficult mainly due to its strong coupling with an exhaust gas recirculation (EGR) system and large lag, resulting from time delay and hysteresis between the input and output dynamics of the engine’s gas exchange system. In this context, one of the latest model-free DRL algorithms, the deep deterministic policy gradient (DDPG) algorithm, was built in this paper to develop and finally form a strategy to track the target boost pressure under transient driving cycles. Using a fine-tuned proportion integration differentiation (PID) controller as a benchmark, the results show that the control performance based on the proposed DDPG algorithm can achieve a good transient control performance from scratch by autonomously learning the interaction with the environment, without relying on model supervision or complete environment models. In addition, the proposed strategy is able to adapt to the changing environment and hardware aging over time by adaptively tuning the algorithm in a self-learning manner on-line, making it attractive to real plant control problems whose system consistency may not be strictly guaranteed and whose environment may change over time.

Energy management strategy (EMS) is one of the key technologies that improves the fuel efficiency of hybrid electric vehicles (HEVs) by governing the energy flow between the fuel tank and the electric energy storage. With the rapid development of artificial intelligence especially after the great success of AlphaGo, reinforcement learning (RL) has opened up a new window for EMS. Although many RL-based solutions have been successfully applied to EMS tasks, most current RL-based approaches only consider RL as an offline optimization tool, i.e., RL is used to solve optimization problems given a simulation model. This method may be utilized to realize optimal control in simulation, but there is no guarantee that a well-behaved performance can also be achieved in the real-physical world due to the simulation-to-real gap. Meanwhile, for many industrial EMS tasks especially when optimizing an existing controller, a high-fidelity simulator may be unavailable, but some logging data generated by some suboptimal existing controller and simple models of the principal powertrain components neglecting detailed component dynamics could exist. In this context, a hybrid algorithm combining data-driven and simulation-based RL approach is proposed that tries to learn experience from both the real logging data and simulated simple model. Based on the hardware-in-the-loop (HIL) results, a near optimal policy can be obtained (fuel consumption is decreased by approximately 6.10%) by this hybrid algorithm from a small batch logging data and a simple simulated model when compared with the dynamic programming (DP) method using a high-fidelity simulation model mimicking the real-physical world.