4 March 2019 in Nature Human Behaviour
1 November 2018 in Research Policy
12 October 2021 in Journal of the Association for Information Science and Technology
Peer review is essential for advancing scientific research, but there are long‐standing concerns that authors' prestige or other characteristics can bias reviewers. Double‐blind peer review has been proposed as a way to reduce reviewer bias, but the evidence for its effectiveness is limited and mixed. Here, we examine the effects of double‐blind peer review by analyzing the review files of 5,027 papers submitted to a top computer science conference that changed its reviewing format from single‐ to double‐blind in 2018. First, we find that the scores given to the most prestigious authors significantly decreased after switching to double‐blind review. However, because many of these papers were above the threshold for acceptance, the change did not affect paper acceptance significantly. Second, the inter‐reviewer disagreement increased significantly in the double‐blind format. Third, papers rejected in the single‐blind format are cited more than those rejected under double‐blind, suggesting that double‐blind review better excludes poorer quality papers. Lastly, an apparently unrelated change in the rating scale from 10 to 4 points likely reduced prestige bias significantly such that papers' acceptance was affected. These results support the effectiveness of double‐blind review in reducing biases, while opening new research directions on the impact of peer‐review formats.
17 November 2022 in Proceedings of the National Academy of Sciences
There are long-standing concerns that peer review, which is foundational to scientific institutions like journals and funding agencies, favors conservative ideas over novel ones. We investigate the association between novelty and the acceptance of manuscripts submitted to a large sample of scientific journals. The data cover 20,538 manuscripts submitted between 2013 and 2018 to the journals Cell and Cell Reports and 6,785 manuscripts submitted in 2018 to 47 journals published by the Institute of Physics Publishing . Following previous work that found that a balance of novel and conventional ideas predicts citation impact, we measure the novelty and conventionality of manuscripts by the atypicality of combinations of journals in their reference lists, taking the 90th percentile most atypical combination as “novelty” and the 50th percentile as “conventionality.” We find that higher novelty is consistently associated with higher acceptance; submissions in the top novelty quintile are 6.5 percentage points more likely than bottom quintile ones to get accepted. Higher conventionality is also associated with acceptance (+16.3% top–bottom quintile difference). Disagreement among peer reviewers was not systematically related to submission novelty or conventionality, and editors select strongly for novelty even conditional on reviewers' recommendations (+7.0% top–bottom quintile difference). Manuscripts exhibiting higher novelty were more highly cited. Overall, the findings suggest that journal peer review favors novel research that is well situated in the existing literature, incentivizing exploration in science and challenging the view that peer review is inherently antinovelty.
28 October 2021 in Management Science
The evaluation and selection of novel projects lies at the heart of scientific and technological innovation, and yet there are persistent concerns about bias, such as conservatism. This paper investigates the role that the format of evaluation, specifically information sharing among expert evaluators, plays in generating conservative decisions. We executed two field experiments in two separate grant-funding opportunities at a leading research university, mobilizing 369 evaluators from seven universities to evaluate 97 projects, resulting in 761 proposal-evaluation pairs and more than $250,000 in awards. We exogenously varied the relative valence (positive and negative) of others’ scores and measured how exposures to higher and lower scores affect the focal evaluator’s propensity to change their initial score. We found causal evidence of a negativity bias, where evaluators lower their scores by more points after seeing scores more critical than their own rather than raise them after seeing more favorable scores. Qualitative coding of the evaluators’ justifications for score changes reveals that exposures to lower scores were associated with greater attention to uncovering weaknesses, whereas exposures to neutral or higher scores were associated with increased emphasis on nonevaluation criteria, such as confidence in one’s judgment. The greater power of negative information suggests that information sharing among expert evaluators can lead to more conservative allocation decisions that favor protecting against failure rather than maximizing success. This paper was accepted by Alfonso Gambardella, business strategy.
1 July 2025 in Nature Communications
Self-promotion in science is ubiquitous but may not be exercised equally by everyone. Research on self-promotion in other domains suggests that, partly due to adverse reactions to non-gender-conforming career-enhancing behaviors, women tend to self-promote less often than men. We test whether this pattern extends to online spaces by examining scholarly self-promotion over six years using 23M tweets about 2.8M research papers authored by 3.5M scientists. We find that, overall, women are about 28% less likely than men to self-promote their papers on Twitter (now X) despite accounting for important confounds. The differential adoption of Twitter does not fully explain the gender gap in self-promotion, which is large even in relatively gender-balanced research areas, where adversity is expected to be smaller. Moreover, we find that the gender gap increases with higher performance and academic status, being most pronounced for research-prolific women from top-ranked institutions who publish papers in high-impact journals. We also find differential returns with respect to gender: while self-promotion is associated with increased tweets of papers compared to no self-promotion, the increase is slightly smaller for women than for men. Our findings reveal that scholarly self-promotion online varies meaningfully by gender and can contribute to a measurable gender gap in the visibility of scientific ideas.