Persistent congestions which are varying in strength and duration in the dense traffic networks are the most prominent obstacle towards sustainable mobility. Those types of congestions cannot be adequately resolved by the traditional Adaptive Traffic Signal Control (ATSC). The introduction of Reinforcement Learning (RL) in ATSC as tackled those types of congestions by using on-line learning, which is based on the trial and error approach. Furthermore, RL is prone to the dimensionality curse related to the state–action space size based on which a non-linear quality function is derived. The Deep Reinforcement Learning (DRL) framework uses Deep Neural Networks (DNN) to digest raw traffic data to approximate the quality function of RL. This paper provides a comprehensive analysis of the most recent DRL approaches used for the ATSC algorithm design. Special emphasis is set to overview of the traffic state representation and multi-agent DRL frameworks applied for the large traffic networks. Best practices are provided for choosing the adequate DRL model, hyper-parameters tuning, and model architecture design. Finally, this paper provides a discussion about the importance of the open traffic data concept for the extensive application of DRL in the real world ATSC.

In urban areas, the problem of recurring daily congestion is constantly increasing. A possible solution is seen in the application of adaptive traffic signal control (ATSC) systems for the control of signalized intersections. While available ATSC systems can achieve an increase in the Level of Service, the focus of ATSC research has shifted towards the application of reinforcement learning (RL) techniques, which allow the controller to learn the optimal control policy by direct interaction with the environment. This paper describes the fundamentals of traffic signal control, RL algorithms and approaches and their application to ATSC, with a discussion on the impact of connected and autonomous vehicles on future traffic signal control. In conclusion, a summary of open research questions and possible directions for future research in the domain of RL‐based traffic signal control is given.

Variable Speed Limit (VSL) control systems are widely studied as solutions for improving safety and throughput on urban motorways. Machine learning techniques, specifically Reinforcement Learning (RL) methods, are a promising alternative for setting up VSL since they can learn and react to different traffic situations without knowing the explicit model of the motorway dynamics. However, the efficiency of combined RL-VSL is highly related to the class of the used RL algorithm, and description of the managed motorway section in which the RL-VSL agent sets the appropriate speed limits. Currently, there is no existing overview of RL algorithm applications in the domain of VSL. Therefore, a comprehensive survey on the state of the art of RL-VSL is presented. Best practices are summarized, and new viewpoints and future research directions, including an overview of current open research questions are presented.

Modern urban mobility needs new solutions to resolve high-complexity demands on urban traffic-control systems, including reducing congestion, fuel and energy consumption, and exhaust gas emissions. One example is urban motorways as key segments of the urban traffic network that do not achieve a satisfactory level of service to serve the increasing traffic demand. Another complex need arises by introducing the connected and autonomous vehicles (CAVs) and accompanying additional challenges that modern control systems must cope with. This study addresses the problem of decreasing the negative environmental aspects of traffic, which includes reducing congestion, fuel and energy consumption, and exhaust gas emissions. We applied a variable speed limit (VSL) based on Q-Learning that utilizes electric CAVs as speed-limit actuators in the control loop. The Q-Learning algorithm was combined with the two-step temporal difference target to increase the algorithm’s effectiveness for learning the VSL control policy for mixed traffic flows. We analyzed two different optimization criteria: total time spent on all vehicles in the traffic network and total energy consumption. Various mixed traffic flow scenarios were addressed with varying CAV penetration rates, and the obtained results were compared with a baseline no-control scenario and a rule-based VSL. The data about vehicle-emission class and the share of gasoline and diesel human-driven vehicles were taken from the actual data from the Croatian Bureau of Statistics. The obtained results show that Q-Learning-based VSL can learn the control policy and improve the macroscopic traffic parameters and total energy consumption and can reduce exhaust gas emissions for different electric CAV penetration rates. The results are most apparent in cases with low CAV penetration rates. Additionally, the results indicate that for the analyzed traffic demand, the increase in the CAV penetration rate alleviates the need to impose VSL control on an urban motorway.

Existing transportation infrastructure and traffic control systems face increasing strain as a result of rising demand, resulting in frequent congestion. Expanding infrastructure is not a feasible solution for enhancing the capacity of the road. Hence, Intelligent Transportation Systems are often employed to enhance the Level of Service (LoS). One such approach is Variable Speed Limit (VSL) control. VSL increases the LoS and safety on motorways by optimizing the speed limit according to the traffic conditions. The proliferation of Connected and Autonomous Vehicles (CAVs) presents fresh prospects for improving the operation and measurement of traffic states for the operation of the VSL control system. This paper introduces a method for the detection of multiple congested areas that is used for state estimation for a dynamically positioned VSL control system for urban motorways. The method utilizes Q-Learning (QL) and CAVs as mobile sensors and actuators. The proposed control approach, named Congestion Detection QL Dynamic Position VSL (CD-QL-DPVSL), dynamically detects all of the congested areas and applies two sets of actions, involving the dynamic positioning of speed limit zones and imposed speed limits for all detected congested areas simultaneously. The proposed CD-QL-DPVSL control approach underwent an evaluation across six distinct traffic scenarios, encompassing CAV penetration rates spanning from $10\%$ to $100\%$ and demonstrated a significantly better performance compared to other control approaches, including no control, rule-based VSL, two Speed-Transition-Matrices-based QL-VSL configurations with fixed speed limit zone positions, and a Speed-Transition-Matrices-based QL-DVSL with a dynamic speed limit zone position. It achieved enhancements in macroscopic traffic parameters such as the Mean Travel Time and Total Time Spent by adapting its control policy to every simulated scenario.