Recommender systems are becoming a salient part of many e-commerce websites. Much research has focused on advancing recommendation technologies to improve accuracy of predictions, although behavioral aspects of using recommender systems are often overlooked. In our studies, we explore how consumer preferences at the time of consumption are impacted by predictions generated by recommender systems. We conducted three controlled laboratory experiments to explore the effects of system recommendations on preferences. Studies 1 and 2 investigated user preferences for television programs across a variety of conditions, which were surveyed immediately following program viewing. Study 3 investigated the granularity of the observed effects within individual participants. Results provide strong evidence that the rating presented by a recommender system serves as an anchor for the consumer's constructed preference. Viewers' preference ratings are malleable and can be significantly influenced by the recommendation received. The effect is sensitive to the perceived reliability of a recommender system and, thus, not a purely numerical or priming-based effect. Finally, the effect of anchoring is continuous and linear, operating over a range of perturbations of the system. These general findings have a number of important implications (e.g., on recommender systems performance metrics and design, preference bias, potential strategic behavior, and trust), which are discussed.

This article investigates the impact of rating data characteristics on the performance of several popular recommendation algorithms, including user-based and item-based collaborative filtering, as well as matrix factorization. We focus on three groups of data characteristics: rating space, rating frequency distribution, and rating value distribution. A sampling procedure was employed to obtain different rating data subsamples with varying characteristics; recommendation algorithms were used to estimate the predictive accuracy for each sample; and linear regression-based models were used to uncover the relationships between data characteristics and recommendation accuracy. Experimental results on multiple rating datasets show the consistent and significant effects of several data characteristics on recommendation accuracy.

Recommender systems are an integral part of the online retail environment. Prior research has focused largely on computational approaches to improving recommendation accuracy, and only recently researchers have started to study their behavioral implications and potential side effects. We used three controlled experiments, in the context of purchasing digital songs, to explore the willingness-to-pay judgments of individual consumers after being shown personalized recommendations. In Study 1, we found strong evidence that randomly assigned song recommendations affected participants’ willingness to pay, even when controlling for participants’ preferences and demographics. In Study 2, participants viewed actual system-generated recommendations that were intentionally perturbed (introducing recommendation error), and we observed similar effects. In Study 3, we showed that the influence of personalized recommendations on willingness-to-pay judgments was obtained even when preference uncertainty was reduced through immediate and mandatory song sampling prior to pricing. The results demonstrate the existence of important economic side effects of personalized recommender systems and inform our understanding of how system recommendations can influence our everyday preference judgments. The findings have significant implications for the design and application of recommender systems as well as for online retail practices. The online appendix is available at https://doi.org/10.1287/isre.2017.0703 .

We develop a general-purpose agent-based simulation and modeling approach to analyze how user–recommender interactions affect recommender systems in the long run. Our explorations show that, over time, user–recommender interactions consistently lead to the longitudinal performance paradox of recommender systems. In particular, users’ reliance on recommendations, while helping users discover relevant items, actually hurts the future diversity of items that are recommended and consumed as well as slows down the system’s learning pace (i.e., the rate of predictive accuracy improvement). We also demonstrate unique benefits of certain hybrid consumption strategies—that is, that take advantage of both popularity- and personalization-based recommendations—in facilitating improvements in consumption relevance over time. Because users’ consumption strategies can significantly influence the longitudinal performance of recommender systems, it is important for designers to analyze the histories of a system’s recommendations and users’ choices to infer and understand users’ consumption strategies. This would enable the system to anticipate users’ consumption behavior and strategically adjust the system’s parameters according to its long-term performance objectives.

The article explores stability as a new measure of recommender systems performance. Stability is defined to measure the extent to which a recommendation algorithm provides predictions that are consistent with each other. Specifically, for a stable algorithm, adding some of the algorithm’s own predictions to the algorithm’s training data (for example, if these predictions were confirmed as accurate by users) would not invalidate or change the other predictions. While stability is an interesting theoretical property that can provide additional understanding about recommendation algorithms, we believe stability to be a desired practical property for recommender systems designers as well, because unstable recommendations can potentially decrease users’ trust in recommender systems and, as a result, reduce users’ acceptance of recommendations. In this article, we also provide an extensive empirical evaluation of stability for six popular recommendation algorithms on four real-world datasets. Our results suggest that stability performance of individual recommendation algorithms is consistent across a variety of datasets and settings. In particular, we find that model-based recommendation algorithms consistently demonstrate higher stability than neighborhood-based collaborative filtering techniques. In addition, we perform a comprehensive empirical analysis of many important factors (e.g., the sparsity of original rating data, normalization of input data, the number of new incoming ratings, the distribution of incoming ratings, the distribution of evaluation data, etc.) and report the impact they have on recommendation stability.