Machine learning (ML) methods are proliferating in scientific research. However, the adoption of these methods has been accompanied by failures of validity, reproducibility, and generalizability. These failures can hinder scientific progress, lead to false consensus around invalid claims, and undermine the credibility of ML-based science. ML methods are often applied and fail in similar ways across disciplines. Motivated by this observation, our goal is to provide clear recommendations for conducting and reporting ML-based science. Drawing from an extensive review of past literature, we present the REFORMS checklist (recommendations for machine-learning-based science). It consists of 32 questions and a paired set of guidelines. REFORMS was developed on the basis of a consensus of 19 researchers across computer science, data science, mathematics, social sciences, and biomedical sciences. REFORMS can serve as a resource for researchers when designing and implementing a study, for referees when reviewing papers, and for journals when enforcing standards for transparency and reproducibility.

When making recommendations, there is an apparent trade-off between the goals of accuracy (to recommend items a user is most likely to want) and diversity (to recommend items representing a range of categories). As such, real-world recommender systems often explicitly incorporate diversity into recommendations, at the cost of accuracy. We study the accuracy-diversity trade-off by bringing in a third concept: user utility. We argue that accuracy is misaligned with user utility because it fails to incorporate a user's consumption constraints: at any given time, users can typically only use at most a few recommended items (e.g., dine at one restaurant, or watch a couple of movies). In a theoretical model, we show that utility-maximizing recommendations---when accounting for consumption constraints---are naturally diverse due to diminishing returns of recommending similar items. Therefore, while increasing diversity may come at the cost of accuracy, it can also help align accuracy-based recommendations toward the more fundamental objective of user utility. Our theoretical results yield practical guidance into how recommendations should incorporate diversity to serve user ends.

In two-sided matching---such as between firms and workers, hospitals and residents, or colleges and students---noise is inevitable. For example, firms evaluate job applicants using limited and imperfect information from resumes and interviews. Given this noise, do the "correct" matches still form? This article addresses this question, in a setting in which (subsets of) colleges share true preferences based on a measure of student value. In a noise-free setting, the highest value students match with their most preferred colleges. However, suppose each college makes offers based on independent noisy estimates of student value. Do the highest-value students still match? In essence, we are interested in how "local" noise introduced in the evaluation processes of individual colleges is aggregated by the market to produce the "global" outcome. Perhaps nothing remarkable happens---individual colleges make noisy decisions, and the set of matched students is "just as noisy." But one can also imagine two types of market-level effects that are more interesting: an attenuating effect, in which noise cancels out, or an amplifying effect, in which noise compounds. We show that in large markets the most extreme versions of both effects can occur. In the basic model we analyze, students each have a true value υ ∈ ℝ. Each college c ranks students according to an estimated value υ + X_c, where X_c is drawn i.i.d. from some distribution D. Given these noisy college preferences and any distribution of student preferences, we analyze the probability that a student with true value υ is matched in the corresponding stable matching. When there is a continuum of students and the number of colleges grows large, we find that: • (Theorem 1) If D is light-tailed (i.e., when the maximum order statistic concentrates), the probability a student matches approaches a step function; all students with true value above a certain cutoff are guaranteed to match, and all students with value below the cutoff are guaranteed not to match, as in the noiseless case. Noise is fully attenuated. • (Theorem 2) If D is long-tailed (i.e., when the hazard rate vanishes), the probability a student matches approaches the same constant independent of their true value. Noise is fully amplified. These results reveal that matching markets can both strongly attenuate and amplify noise, suggesting that either a "wisdom of crowds" or "foolishness of crowds" can occur in matching markets. While we first consider effects when colleges share the same true preferences over students, the results extend to a much broader setting. We show that analogues of Theorems 1 and 2 hold for any large coalition of colleges with the same true preferences, even when this coalition exists within a broader economy where students and other colleges have otherwise arbitrary preferences. Full paper: https://arxiv.org/abs/2402.16771