The interaction of multiple autonomous agents gives rise to highly dynamic and nondeterministic environments, contributing to the complexity in applications such as automated financial markets, smart grids, or robotics. Due to the sheer number of situations that may arise, it is not possible to foresee and program the optimal behaviour for all agents beforehand. Consequently, it becomes essential for the success of the system that the agents can learn their optimal behaviour and adapt to new situations or circumstances. The past two decades have seen the emergence of reinforcement learning, both in single and multi-agent settings, as a strong, robust and adaptive learning paradigm. Progress has been substantial, and a wide range of algorithms are now available. An important challenge in the domain of multi-agent learning is to gain qualitative insights into the resulting system dynamics. In the past decade, tools and methods from evolutionary game theory have been successfully employed to study multi-agent learning dynamics formally in strategic interactions. This article surveys the dynamical models that have been derived for various multi-agent reinforcement learning algorithms, making it possible to study and compare them qualitatively. Furthermore, new learning algorithms that have been introduced using these evolutionary game theoretic tools are reviewed. The evolutionary models can be used to study complex strategic interactions. Examples of such analysis are given for the domains of automated trading in stock markets and collision avoidance in multi-robot systems. The paper provides a roadmap on the progress that has been achieved in analysing the evolutionary dynamics of multi-agent learning by highlighting the main results and accomplishments.

We analyse active space debris removal efforts from a strategic, game-theoretical perspective. Space debris is non-manoeuvrable, human-made objects orbiting Earth, which pose a significant threat to operational spacecraft. Active debris removal missions have been considered and investigated by different space agencies with the goal to protect valuable assets present in strategic orbital environments. An active debris removal mission is costly, but has a positive effect for all satellites in the same orbital band. This leads to a dilemma: each agency is faced with the choice between the individually costly action of debris removal, which has a positive impact on all players; or wait and hope that others jump in and do the ‘dirty’ work. The risk of the latter action is that, if everyone waits, the joint outcome will be catastrophic, leading to what in game theory is referred to as the ‘tragedy of the commons’. We introduce and thoroughly analyse this dilemma using empirical game theory and a space debris simulator. We consider two- and three-player settings, investigate the strategic properties and equilibria of the game and find that the cost/benefit ratio of debris removal strongly affects the game dynamics.

MIT Press Journals is a mission-driven, not-for-profit scholarly publisher devoted to the widest dissemination of its content.