This is the author's version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record was published in RecSys '11 Proceedings of the fifth ACM conference on Recommender systems, http://dx.doi.org/10.1145/2043932.2044016

This is the author's version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record was published in RecSys '10 Proceedings of the fourth ACM conference on Recommender systems, http://dx.doi.org/10.1145/1864708.1864796.There is considerable methodological divergence in the way precision-oriented metrics are being applied in the Recommender Systems field, and as a consequence, the results reported in different studies are difficult to put in context and compare. We aim to identify the involved methodological design alternatives, and their effect on the resulting measurements, with a view to assessing their suitability, advantages, and potential shortcomings. We compare five experimental methodologies, broadly covering the variants reported in the literature. In our experiments with three state-of-the-art recommenders, four of the evaluation methodologies are consistent with each other and differ from error metrics, in terms of the comparative recommenders' performance measurements. The other procedure aligns with RMSE, but shows a heavy bias towards known relevant items, considerably overestimating performance.This work was supported by the Spanish Ministry of Science and Innovation (TIN2008-06566-C04-02) and the Community of Madrid (CCG10-UAM/TIC-5877