24 April 2024 in IEEE Transactions on Transportation Electrification
The intricate interactions with other road users and the diversity of traffic environments create a challenging decision-making task for autonomous driving systems. While offline learning solutions are renowned for their high execution efficiency and the ability to approximate the optimal policy across the entire state space, they are often unsafe and fragile when encountering untrained states. Conversely, online planning methods possess the capacity to thoroughly assess how current decisions influence future outcomes online, and therefore have better generalization. Nevertheless, these approaches face difficulties in terms of execution efficiency and are susceptible to becoming stuck in locally optimal solutions. In this context, this paper proposes an Integrated Planning and Learning (IPL) algorithm based on the reinforcement learning framework for speed and lane change decision-making on highways. Specifically, at each decision time step, this method utilizes the offline learned model to guide an online Monte Carlo Tree Search (MCTS) algorithm for heuristic search, aiming to formulate a forward-looking policy. The experimental results show that the IPL algorithm performs better generalization when faced with unknown scenarios, and its asymptotic performance is better than other benchmark algorithms. In addition, in contrast to the MCTS-based online planning method, the IPL algorithm enhances execution efficiency and comes closer to achieving global optimality.
17 April 2023 in IEEE Transactions on Transportation Electrification
Energy management strategy (EMS) is one of the key technologies that improves the fuel efficiency of hybrid electric vehicles (HEVs) by governing the energy flow between the fuel tank and the electric energy storage. With the rapid development of artificial intelligence especially after the great success of AlphaGo, reinforcement learning (RL) has opened up a new window for EMS. Although many RL-based solutions have been successfully applied to EMS tasks, most current RL-based approaches only consider RL as an offline optimization tool, i.e., RL is used to solve optimization problems given a simulation model. This method may be utilized to realize optimal control in simulation, but there is no guarantee that a well-behaved performance can also be achieved in the real-physical world due to the simulation-to-real gap. Meanwhile, for many industrial EMS tasks especially when optimizing an existing controller, a high-fidelity simulator may be unavailable, but some logging data generated by some suboptimal existing controller and simple models of the principal powertrain components neglecting detailed component dynamics could exist. In this context, a hybrid algorithm combining data-driven and simulation-based RL approach is proposed that tries to learn experience from both the real logging data and simulated simple model. Based on the hardware-in-the-loop (HIL) results, a near optimal policy can be obtained (fuel consumption is decreased by approximately 6.10%) by this hybrid algorithm from a small batch logging data and a simple simulated model when compared with the dynamic programming (DP) method using a high-fidelity simulation model mimicking the real-physical world.
25 December 2023 in IEEE Transactions on Transportation Electrification
Reinforcement learning (RL) has the capability to discover optimal interactions with the surrounding environment, with the advantage that nearly all required computations can be performed offline. Nevertheless, the lack of explainability for RL-based solutions may prevent their large-scale application in industrial autonomous vehicle tasks. Furthermore, the RL method tends to be unsafe and brittle to scenarios not encountered in training. Conversely, the optimization-based method offers a substantial level of explainability, and through the explicit inclusion of safety constraints, it can guarantee the system’s safety. In this context, building upon RL framework, a fusion algorithm that combines the advantage of the RL-based scheme and optimization-based scheme is proposed. Specifically, unlike traditional RL-based solutions which directly executes from perception to control using only neural network maps, this work introduces a mechanism of uncertainty-aware interval prediction to compute the set of states that can be reached over the planning time horizon. On this basis, a robust control framework is presented, which guarantees system safety while considering the worst-case performance scenarios. To validate the proposed algorithm, the task of an autonomous vehicle merging on to a highway from an on-ramp is simulated in SUMO. The results show that the proposed motion planning and control method combines the advantages of RL and optimization-based methods and achieves balanced performance in smoothness, computational efficiency, explainability and robustness.
6 February 2024 in IEEE Transactions on Industrial Electronics
Considering the simulation-to-real gap and the fact that the data-driven learning methods are often suboptimal, an effective offline-to-online learning paradigm that can further improve the initialized offline policy in a real environment is necessary. However, for most industrial applications, this two-stage learning paradigm is often challenging to implement, as the initialized policy frequently deviates from the optimal improvement during fine-tuning. To address this, an effective offline-to-online reinforcement learning (RL) based training framework that bridges offline initialization and online fine-tuning will be proposed in this work. In contrast to most of RL algorithms, which evaluate unseen actions from the latest policy and extract policies only corresponding to the maximum value function, the proposed method uses two implicit constraints, namely, quantile regression and advantage weighted regression. In order to assess the efficacy of the proposed algorithm with real controllers, a hardware-in-the-loop test is conducted. The findings demonstrate that by leveraging the generalization capability of the quantile regression to estimate the value of the best available state-action pair and derive the policy using the advantage-weighted form of behavioral cloning to favor actions that receive higher advantage, the proposed data-driven RL can not only learn effectively from a static offline dataset, but also exhibit a robust policy improvement during the subsequent fine-tuning process.
2 September 2025 in Engineering Applications of Artificial Intelligence
2 November 2024 in Engineering Applications of Artificial Intelligence