Recommender systems have been widely applied in different real-life scenarios to help us find useful information. In particular, reinforcement learning (RL)-based recommender systems have become an emerging research topic in recent years, owing to the interactive nature and autonomous learning ability. Empirical results show that RL-based recommendation methods often surpass supervised learning methods. Nevertheless, there are various challenges in applying RL in recommender systems. To understand the challenges and relevant solutions, there should be a reference for researchers and practitioners working on RL-based recommender systems. To this end, we first provide a thorough overview, comparisons, and summarization of RL approaches applied in four typical recommendation scenarios, including interactive recommendation, conversational recommendation, sequential recommendation, and explainable recommendation. Furthermore, we systematically analyze the challenges and relevant solutions on the basis of existing literature. Finally, under discussion for open issues of RL and its limitations of recommender systems, we highlight some potential research directions in this field.
15 March 2025 in ACM Transactions on Information Systems
Multimodal recommendation, which utilizes rich multimodal information to learn user preferences, has attracted significant attention. Most works focus on designing powerful encoders for extracting multimodal features, and simply aggregate the learned features together to make prediction. Consequently, they have a limited capacity to learn the inter-modality knowledge including the modality-shared and modality-unique knowledge. In fact, learning the modality-shared knowledge enables us to align cross-modality data for fusing heterogeneous modality features. Learning the modality-unique knowledge is equally important when recommendation tasks only involve a small amount of shared features and the necessary information is contained within specific modality. In this article, we propose Contrastive Modality-Disentangled Learning (CMDL) to overcome this critical limitation. CMDL exactly captures the inter-modality knowledge by achieving modality disentanglement. Specifically, CMDL first disentangles the initial representation into the modality-invariant and modality-specific representations. Afterwards, CMDL introduces a novel manner of contrastive learning to approximate the MI upper bounds for achieving disentanglement regularization. Building upon the proposed regularization, CMDL encourages the modality-invariant and modality-specific representations to capture the modality-shared and modality-unique knowledge respectively and to be statistically independent to each other. Empirically, extensive experiments are conducted on benchmark datasets, demonstrating the superior performance of CMDL compared with strong multimodal recommenders.
10 July 2025 in ACM Transactions on Information Systems
Sequential recommendation systems aim to predict users’ next likely interaction based on their history. However, these systems face data sparsity and cold-start problems. Utilizing data from other domains, known as multi-domain methods, is useful for alleviating these problems. However, traditional multi-domain methods rely on meaningless ID-based item representation, which makes it difficult to align items with similar meanings from different domains, yielding sup-optimal knowledge transfer. This article introduces LLM-Rec , a framework that utilizes pre-trained Large Language Models (LLMs) for domain-agnostic recommendation. Specifically, we mix user’s behaviors from multiple domains and concatenate item titles into a sentence, then use LLMs for generating user and item representations. By mixing behaviors across different domains, we can exploit the knowledge encoded in LLMs to bridge the semantic across over multi-domain behaviors, thus obtaining semantically rich representations and improving performance in all domains. Furthermore, we explore the underlying reasons why LLMs are effective and investigate whether LLMs can understand the semantic correlations as the recommendation model, and if advanced techniques like scaling laws in NLP also work in recommendations. We conduct extensive experiments with LLMs ranging from 40 M to 6.7 B to answer the above questions and to verify the effectiveness of LLM-Rec in multi-domain recommendation. The source code is available at https://github.com/WHUIR/LLMRec .
With the rapid evolution of transformer architectures, researchers are exploring their application in sequential recommender systems (SRSs) and presenting promising performance on SRS tasks compared with former SRS models. However, most existing transformer-based SRS frameworks retain the vanilla attention mechanism, which calculates the attention scores between all item-item pairs. With this setting, redundant item interactions can harm the model performance and consume much computation time and memory. In this paper, we identify the sparse attention phenomenon in transformer-based SRS models and propose Sparse Transformer for sequential Recommendation tasks (STRec) to achieve the efficient computation and improved performance. Specifically, we replace self-attention with cross-attention, making the model concentrate on the most relevant item interactions. To determine these necessary interactions, we design a novel sampling strategy to detect relevant items based on temporal information. Extensive experimental results validate the effectiveness of STRec, which achieves the state-of-the-art accuracy while reducing 54% inference time and 70% memory cost. We also provide massive extended experiments to further investigate the property of our framework.
7 August 2023 in IEEE Transactions on Knowledge and Data Engineering
Offline reinforcement learning (RL) makes it possible to train the agents entirely from a previously collected dataset. However, constrained by the quality of the offline dataset, offline RL agents typically have limited performance and cannot be directly deployed. Thus, it is desirable to further finetune the pretrained offline RL agents via online interactions with the environment. Existing offline-to-online RL algorithms suffer from the low sample efficiency issue, due to two inherent challenges, i.e., exploration limitation and distribution shift. To this end, we propose a sample-efficient offline-to-online RL algorithm via Optimistic Exploration and Meta Adaptation (OEMA). Specifically, we first propose an optimistic exploration strategy according to the principle of optimism in the face of uncertainty. This allows agents to sufficiently explore the environment in a stable manner. Moreover, we propose a meta learning based adaptation method, which can reduce the distribution shift and accelerate the offline-to-online adaptation process. We empirically demonstrate that OEMA improves the sample efficiency on D4RL benchmark. Besides, we provide in-depth analyses to verify the effectiveness of both optimistic exploration and meta adaptation.
Improving user retention with reinforcement learning (RL) has attracted increasing attention due to its significant importance in boosting user engagement. However, training the RL policy from scratch without hurting users’ experience is unavoidable due to the requirement of trial-and-error searches. Furthermore, the offline methods, which aim to optimize the policy without online interactions, suffer from the notorious stability problem in value estimation or unbounded variance in counterfactual policy evaluation. To this end, we propose optimizing user retention with Decision Transformer (DT), which avoids the offline difficulty by translating the RL as an autoregressive problem. However, deploying the DT in recommendation is a non-trivial problem because of the following challenges: (1) deficiency in modeling the numerical reward value; (2) data discrepancy between the policy learning and recommendation generation; (3) unreliable offline performance evaluation. In this work, we, therefore, contribute a series of strategies for tackling the exposed issues. We first articulate an efficient reward prompt by weighted aggregation of meta embeddings for informative reward embedding. Then, we endow a weighted contrastive learning method to solve the discrepancy between training and inference. Furthermore, we design two robust offline metrics to measure user retention. Finally, the significant improvement in the benchmark datasets demonstrates the superiority of the proposed method. The implementation code is available at https://github.com/kesenzhao/DT4Rec.git.