The traffic signal control problem is an extensively researched area providing different approaches, from classic methods to machine learning based ones. Different aspects can be considered to find an optima, from which this paper emphasises emission reduction. The core of our solution is a novel rewarding concept for deep reinforcement learning (DRL) which does not utilize any reward shaping, hence exposes new insights into the traffic signal control (TSC) problem. Despite the omission of the standard measures in the rewarding scheme, the proposed approach can outperform a modern actuated control method in classic performance measures such as waiting time and queue length. Moreover, the sustainability of the realized controls is also placed under investigation to evaluate their environmental impacts. Our results show that the proposed solution goes beyond the actuated control not just in the classic measures but in emission-related measures too.
23 February 2022 in IEEE Transactions on Intelligent Transportation Systems
Reinforcement learning applications are spreading among different domains, including autonomous vehicle control. The diverse situations that can happen during, for instance, at a highway commute are infinite, and with labeled data, the perfect coverage of all use-cases sounds ambitious. However, with the complex tasks and complicated scenarios faced during an autonomous vehicle system design, the credit assignment problem arises. How to construct appropriate objectives for the artificial intelligence to learn and the preferences between the different goals also matter of the designer’s choice. This work attempts to tackle the problem by utilizing successor features and providing a possible decomposition of the reward functions, guiding the agent’s actions. This method makes the training easier for the agent and enables immediate, profound performance on new combined tasks. Furthermore, with the optimal composition, the desired behavior can be fine-tuned, and as an auxiliary gain, the decomposition empowers different driving styles and makes driving preferences rapidly changeable. We introduce the adaptation of FastRL algorithm to autonomous vehicle domain, meanwhile developing a stabilizing way of using Successor Features, namely DoubleFastRL. We compare our solution for a highway driving scenario with basic agents such as Q-learning having multi-objective training.
In this paper, we study the problem of traffic signal control in general intersections by applying a recent reinforcement learning technique. Nowadays, traffic congestion and road usage are increasing significantly as more and more vehicles enter the same infrastructures. New solutions are needed to minimize travel times or maximize the network capacity (throughput). Recent studies embrace machine learning approaches that have the power to aid and optimize the increasing demands. However, most reinforcement learning algorithms fail to be adaptive regarding goal functions. To this end, we provide a novel successor feature-based solution to control a single intersection to optimize the traffic flow, reduce the environmental impact, and promote sustainability. Our method allows for flexibility and adaptability to changing circumstances and goals. It supports changes in preferences during inference, so the behavior of the trained agent (traffic signal controller) can be changed rapidly during the inference time. By introducing the successor features to the domain, we define the basics of successor features, the base reward functions, and the goal preferences of the traffic signal control system. As our main direction, we tackle environmental impact reduction and support prioritized vehicles’ commutes. We include an evaluation of how our method achieves a more effective operation considering the environmental impact and how adaptive it is compared to a general Deep-Q-Network solution. Aside from this, standard rule-based and adaptive signal-controlling technologies are compared to our method to show its advances. Furthermore, we perform an ablation analysis on the adaptivity of the agent and demonstrate a consistent level of performance under similar circumstances.
10 April 2024 in IEEE Robotics and Automation Letters
This work addresses the challenge of reinforcement learning with reward functions that feature highly imbalanced components in terms of importance and scale. Reinforcement learning algorithms generally struggle to handle such imbalanced reward functions effectively. Consequently, they often converge to suboptimal policies that favor only the dominant reward component. For example, agents might adopt passive strategies, avoiding any action to evade potentially unsafe outcomes entirely. To mitigate the adverse effects of imbalanced reward functions, we introduce a curriculum learning approach based on the successor features representation. This novel approach enables our learning system to acquire policies that take into account all reward components, allowing for a more balanced and versatile decision-making process.