Multi-Armed Bandit (MAB) techniques have been successfully applied to many classes of sequential decision problems in the past decades. However, non-stationary settings -- very common in real-world applications -- received little attention so far, and theoretical guarantees on the regret are known only for some frequentist algorithms. In this paper, we propose an algorithm, namely Sliding-Window Thompson Sampling (SW-TS), for nonstationary stochastic MAB settings. Our algorithm is based on Thompson Sampling and exploits a sliding-window approach to tackle, in a unified fashion, two different forms of non-stationarity studied separately so far: abruptly changing and smoothly changing. In the former, the reward distributions are constant during sequences of rounds, and their change may be arbitrary and happen at unknown rounds, while, in the latter, the reward distributions smoothly evolve over rounds according to unknown dynamics. Under mild assumptions, we provide regret upper bounds on the dynamic pseudo-regret of SW-TS for the abruptly changing environment, for the smoothly changing one, and for the setting in which both the non-stationarity forms are present. Furthermore, we empirically show that SW-TS dramatically outperforms state-of-the-art algorithms even when the forms of non-stationarity are taken separately, as previously studied in the literature.

Wireless access networks are often characterized by the interaction of different end users, communication technologies, and network operators. This paper analyzes the dynamics among these "actorsâ by focusing on the processes of wireless network selection, where end users may choose among multiple available access networks to get connectivity, and resource allocation, where network operators may set their radio resources to provide connectivity. The interaction among end users is modeled as a noncooperative congestion game, where players (end users) selfishly select the access network that minimizes their perceived selection cost. A method based on mathematical programming is proposed to find Nash equilibria and characterize their optimality under three cost functions, which are representative of different technological scenarios. System level simulations are then used to evaluate the actual throughput and fairness of the equilibrium points. The interaction among end users and network operators is then assessed through a two-stage multileader/multifollower game, where network operators (leaders) play in the first stage by properly setting the radio resources to maximize their users, and end users (followers) play in the second stage the aforementioned network selection game. The existence of exact and approximated subgame perfect Nash equilibria of the two-stage game is thoroughly assessed and numerical results are provided on the "qualityâ of such equilibria.

The existence of simple uncoupled no-regret learning dynamics that converge to correlated equilibria in normal-form games is a celebrated result in the theory of multi-agent systems. Specifically, it has been known for more than 20 years that when all players seek to minimize their internal regret in a repeated normal-form game, the empirical frequency of play converges to a normal-form correlated equilibrium. Extensive-form (that is, tree-form) games generalize normal-form games by modeling both sequential and simultaneous moves, as well as imperfect information. Because of the sequential nature and presence of private information in the game, correlation in extensive-form games possesses significantly different properties than in normal-form games, many of which are still open research directions. Extensive-form correlated equilibrium (EFCE) has been proposed as the natural extensive-form counterpart to the classical notion of correlated equilibrium in normal-form games. Compared to the latter, the constraints that define the set of EFCEs are significantly more complex, as the correlation device (a.k.a. mediator) must take into account the evolution of beliefs of each player as they make observations throughout the game. Due to that significant added complexity, the existence of uncoupled learning dynamics leading to an EFCE has remained a challenging open research question for a long time. In this article, we settle that question by giving the first uncoupled no-regret dynamics that converge to the set of EFCEs in n -player general-sum extensive-form games with perfect recall. We show that each iterate can be computed in time polynomial in the size of the game tree, and that, when all players play repeatedly according to our learning dynamics, the empirical frequency of play after T game repetitions is proven to be a \( O(1/\sqrt {T}) \) -approximate EFCE with high probability, and an EFCE almost surely in the limit.

This paper addresses the problem of network selection, which arises when wireless mobile users can choose among multiple available wireless access networks to connect to. In particular, we are interested in studying the dynamics of the competition among different selfish mobile users which operate the network selection with the goal of minimizing their own selection cost. To this extent, we formalize the problem as a non-cooperative game, and we study the quality of the corresponding Nash equilibria under three expressions for the users' selection cost by deriving tight analytical bounds on the price-of-anarchy and the price-of-stability for the games. Finally, we introduce a mathematical programming formulation for the games which allows one to compute the equilibria in realistic wireless access network topologies