In urban areas, the problem of recurring daily congestion is constantly increasing. A possible solution is seen in the application of adaptive traffic signal control (ATSC) systems for the control of signalized intersections. While available ATSC systems can achieve an increase in the Level of Service, the focus of ATSC research has shifted towards the application of reinforcement learning (RL) techniques, which allow the controller to learn the optimal control policy by direct interaction with the environment. This paper describes the fundamentals of traffic signal control, RL algorithms and approaches and their application to ATSC, with a discussion on the impact of connected and autonomous vehicles on future traffic signal control. In conclusion, a summary of open research questions and possible directions for future research in the domain of RL‐based traffic signal control is given.

The trend of increasing traffic demand is causing congestion on existing urban roads, including urban motorways, resulting in a decrease in Level of Service (LoS) and safety, and an increase in fuel consumption. Lack of space and non-compliance with cities’ sustainable urban plans prevent the expansion of new transport infrastructure in some urban areas. To alleviate the aforementioned problems, appropriate solutions come from the domain of Intelligent Transportation Systems by implementing traffic control services. Those services include Variable Speed Limit (VSL) and Ramp Metering (RM) for urban motorways. VSL reduces the speed of incoming vehicles to a bottleneck area, and RM limits the inflow through on-ramps. In addition, with the increasing development of Autonomous Vehicles (AVs) and Connected AVs (CAVs), new opportunities for traffic control are emerging. VSL and RM can reduce traffic congestion on urban motorways, especially so in the case of mixed traffic flows where AVs and CAVs can fully comply with the control system output. Currently, there is no existing overview of control algorithms and applications for VSL and RM in mixed traffic flows. Therefore, we present a comprehensive survey of VSL and RM control algorithms including the most recent reinforcement learning-based approaches. Best practices for mixed traffic flow control are summarized and new viewpoints and future research directions are presented, including an overview of the currently open research questions.

Variable Speed Limit (VSL) control systems are widely studied as solutions for improving safety and throughput on urban motorways. Machine learning techniques, specifically Reinforcement Learning (RL) methods, are a promising alternative for setting up VSL since they can learn and react to different traffic situations without knowing the explicit model of the motorway dynamics. However, the efficiency of combined RL-VSL is highly related to the class of the used RL algorithm, and description of the managed motorway section in which the RL-VSL agent sets the appropriate speed limits. Currently, there is no existing overview of RL algorithm applications in the domain of VSL. Therefore, a comprehensive survey on the state of the art of RL-VSL is presented. Best practices are summarized, and new viewpoints and future research directions, including an overview of current open research questions are presented.

The prevailing variable speed limit (VSL) systems as an effective strategy for traffic control on motorways have the disadvantage that they only work with static VSL zones. Under changing traffic conditions, VSL systems with static VSL zones may perform suboptimally. Therefore, the adaptive design of VSL zones is required in traffic scenarios where congestion characteristics vary widely over space and time. To address this problem, we propose a novel distributed spatial-temporal multi-agent VSL (DWL-ST-VSL) approach capable of dynamically adjusting the length and position of VSL zones to complement the adjustment of speed limits in current VSL control systems. To model DWL-ST-VSL, distributed W-learning (DWL), a reinforcement learning (RL)-based algorithm for collaborative agent-based self-optimization toward multiple policies, is used. Each agent uses RL to learn local policies, thereby maximizing travel speed and eliminating congestion. In addition to local policies, through the concept of remote policies, agents learn how their actions affect their immediate neighbours and which policy or action is preferred in a given situation. To assess the impact of deploying additional agents in the control loop and the different cooperation levels on the control process, DWL-ST-VSL is evaluated in a four-agent configuration (DWL4-ST-VSL). This evaluation is done via SUMO microscopic simulations using collaborative agents controlling four segments upstream of the congestion in traffic scenarios with medium and high traffic loads. DWL also allows for heterogeneity in agents’ policies; cooperating agents in DWL4-ST-VSL implement two speed limit sets with different granularity. DWL4-ST-VSL outperforms all baselines (W-learning-based VSL and simple proportional speed control), which use static VSL zones. Finally, our experiments yield insights into the new concept of VSL control. This may trigger further research on using advanced learning-based technology to design a new generation of adaptive traffic control systems to meet the requirements of operating in a nonstationary environment and at the leading edge of emerging connected and autonomous vehicles in general.

Variable Speed Limit (VSL) is a traffic control approach that optimises the mainstream traffic on motorways. Reinforcement Learning approach to VSL has been shown to achieve improvements in controlling the mainstream traffic bottleneck on motorways. However, single-agent VSL, applied to a shorter motorway segment, can produce a discontinuity in traffic flow by causing the significant differences in speeds between the uncontrolled upstream flow and the flow affected by VSL. A multi-agent control strategy can be used to overcome these problems by assigning speed limits in multiple upstream motorway sections enabling smoother speed transition. In this paper, we proposed a novel approach to set up multi-agent RLbased VSL by using the W-Learning algorithm (WL-VSL), in which two agents control two segments in the lead up to the congested area. The reward function for each agent is based on the agent’s local performance as well as the downstream bottleneck. WL-VSL is evaluated in a microscopic simulation on two traffic scenarios using dynamic and static traffic demand. We show that WL-VSL outperforms base cases (no control, single agent, and two independent agents) with the improvement of traffic parameters up to 18 %.