21 April 2021 in IEEE Transactions on Neural Networks and Learning Systems
The vehicle platoon will be the most dominant driving mode on future roads. To the best of our knowledge, few reinforcement learning (RL) algorithms have been applied in vehicle platoon control, which has large-scale action and state spaces. Some RL-based methods were applied to solve single-agent problems. If we need to tackle multiagent problems, we will use multiagent RL algorithms since the parameters space grows exponentially with the increasing number of agents involved. Previous multiagent RL algorithms generally may provide redundant information to agents, indicating a large amount of useless or unrelated information, which may cause to be difficult for convergence training and pattern extractions from shared information. Also, random actions usually contribute to crashes, especially at the beginning of training. In this study, a communication proximal policy optimization (CommPPO) algorithm was proposed to tackle the above issues. In specific, the CommPPO model adopts a parameter-sharing structure to allow the dynamic variation of agent numbers, which can well handle various platoon dynamics, including splitting and merging. The communication protocol of the CommPPO consists of two parts. In the state part, the widely used predecessor–leader follower typology in the platoon is adopted to transmit global and local state information to agents. In the reward part, a new reward communication channel is proposed to solve the spurious reward and “lazy agent” problems in some existing multiagent RLs. Moreover, a curriculum learning approach is adopted to reduce crashes and speed up training. To validate the proposed strategy for platoon control, two existing multiagent RLs and a traditional platoon control strategy were applied in the same scenarios for comparison. Results showed that the CommPPO algorithm gained more rewards and achieved the largest fuel consumption reduction (11.6%).
1 February 2020 in Accident Analysis & Prevention
13 October 2023 in IEEE Transactions on Intelligent Transportation Systems
Connected and automated vehicle (CAV) platooning exhibits significant potential in enhancing traffic efficiency and sustainability. In unsteady traffic conditions, CAV platoons frequently require splitting and merging maneuvers to avoid obstacles. This study introduces a hybrid automaton architecture for collision avoidance motion planning during CAV platoon merging and splitting. A velocity obstacle algorithm based on potential fields is developed to detect collision risks and calculate collision-free velocity solutions. Two predictive control-based optimization models are developed for collision-avoidance path planning, catering to both single-cruising vehicles and vehicle platoons. A synergetic architecture based on hybrid automaton is developed to coordinate vehicle motions during platoon splitting and merging. Numerical experiments are performed to evaluate the performance of the proposed hybrid automaton architecture under various obstacle scenarios. The results demonstrate that the proposed algorithms effectively identify collision risks within CAV platoons and determine optimal vehicle velocities. The proposed architecture demonstrates excellent performance in adjusting vehicle maneuvers and adapting CAV platoon formations to changing driving environments.
17 November 2022 in IEEE Transactions on Intelligent Transportation Systems
A promising way to improve efficiency in highway on-ramp regions is to control connected and automated vehicles (CAVs) to pass the merging section sequentially. The primary objective of this paper is to incorporate the reinforcement learning (RL) technique into vehicle merging control to achieve global cooperation. A communication protocol among RL agents is integrated with the Soft Actor-Critic (CSAC) algorithm. The parallel SAC agents with a parameter-sharing structure cooperate to optimize the common reward function. It enables CAVs to know the actions of each other so that they proactively negotiate to adjust speeds. We designed a parsimonious state representation containing crucial merging information to speed up the RL training. The effects of the merging control strategy were investigated with the simulation model in scenarios with different CAV penetration rates and traffic flow compositions. Results showed that the CSAC-based merging strategy generated collision-free merging trajectories with a short travel time while guaranteeing traffic safety in various traffic conditions. Four baselines (including two rule-based and two RL-based merging strategies) were applied in the same scenarios for comparison. It was found that the proposed merging strategy took the safest merging behaviors (zero traffic conflicts) and reduced the most significant travel time (56.9%).
27 June 2023 in Journal of Advanced Transportation
Most of the current variable speed limit (VSL) strategies are designed to alleviate congestion in relatively short freeway segments with a single bottleneck. However, in reality, consecutive bottlenecks can occur simultaneously due to the merging flow from multiple ramps. In such situations, the existing strategies use multiple VSL controllers that operate independently, without considering the traffic flow interactions and speed limit differences. In this research, we introduced a multiagent reinforcement learning-based VSL (MARL-VSL) approach to enhance collaboration among VSL controllers. The MARL-VSL approach employed a centralized training with decentralized execution structure to achieve a joint optimal solution for a series of VSL controllers. The consecutive bottleneck scenarios were simulated in the modified cell transmission model to validate the effectiveness of the proposed strategy. An independent single-agent reinforcement learning-based VSL (ISARL-VSL) and a feedback-based VSL (feedback-VSL) were also applied for comparison. Time-varying heterogeneous traffic flow stemming from the mainline and ramps was loaded into the freeway network. The results demonstrated that the proposed MARL-VSL achieved superior performance compared to the baseline methods. The proposed approach reduced the total time spent by the vehicles by 18.01% and 17.07% in static and dynamic traffic scenarios, respectively. The control actions of the MARL-VSL were more appropriate in maintaining a smooth freeway traffic flow due to its superior collaboration performance. More specifically, the MARL-VSL significantly improved the average driving speed and speed homogeneity across the entire freeway.
18 March 2024 in IEEE Transactions on Intelligent Transportation Systems
Deep reinforcement learning (DRL) algorithms often face challenges in achieving stability and efficiency due to significant policy gradient variance and inaccurate reward function estimation in complex scenarios. This study addresses these issues in the context of multi-objective car-following control tasks with time lag in traffic oscillations. We propose an expert demonstration reinforcement learning (EDRL) approach that aims to stabilize training, accelerate learning, and enhance car-following performance. The key idea is to leverage expert demonstrations, which represent superior car-following control experiences, to improve the DRL policy. Our method involves two sequential steps. In the first step, expert demonstrations are obtained during offline pretraining by utilizing prior traffic knowledge, including car-following trajectories from an empirical database and classic car-following models. In the second step, expert demonstrations are obtained during online training, where the agent interacts with the car-following environment. The EDRL agents are trained through supervised regression on the expert demonstrations using the behavioral cloning technique. Experimental results conducted in various traffic oscillation scenarios demonstrate that our proposed method significantly enhances training stability, learning speed, and rewards compared to baseline algorithms.