Personalization of playlists is a common feature in music streaming services, but conventional techniques, such as collaborative filtering, rely on explicit assumptions regarding content quality to learn how to make recommendations. Such assumptions often result in misalignment between offline model objectives and online user satisfaction metrics. In this paper, we present a reinforcement learning framework that solves for such limitations by directly optimizing for user satisfaction metrics via the use of a simulated playlist-generation environment. Using this simulator we develop and train a modified Deep Q-Network, the action head DQN (AH-DQN), in a manner that addresses the challenges imposed by the large state and action space of our RL formulation. The resulting policy is capable of making recommendations from large and dynamic sets of candidate items with the expectation of maximizing consumption metrics. We analyze and evaluate agents offline via simulations that use environment models trained on both public and proprietary streaming datasets. We show how these agents lead to better user-satisfaction metrics compared to baseline methods during online A/B tests. Finally, we demonstrate that performance assessments produced from our simulator are strongly correlated with observed online metric results.
Users in music streaming platforms typically consume tracks sequentially in sessions by interacting with personalised playlists. To satisfy users, music platforms usually rely on recommender systems that learn users' preferences over individual tracks and rank the tracks within each playlist according to the learned preferences. However, such rankings often do not fully exploit the sequential nature of the users' consumption, which may result in a lower within-a-session consumption. In this paper, we model the sequential within-a-session preferences of users and propose an optimisation-based sequencing approach that allows for optimally incorporating such preferences into the rankings. To this end, we rely on interaction data of a major music streaming service to identify two most common aspects of the users' sequential preferences: (1) Position-Aware preferences, and (2) Local-Sequential preferences. We propose a sequencing model that can leverage each of these aspects optimally to maximise the expected total consumption from the session. We further perform an extensive offline and off-policy evaluation of our model, and carry out a large scale online randomised control trial with 7M users across 80 countries. Our findings confirm that we can effectively incorporate sequential preferences of users into our sequencer to make users complete more and skip less tracks within their listening sessions.