Progress in science relies in part on generating hypotheses with existing observations and testing hypotheses with new observations. This distinction between postdiction and prediction is appreciated conceptually but is not respected in practice. Mistaking generation of postdictions with testing of predictions reduces the credibility of research findings. However, ordinary biases in human reasoning, such as hindsight bias, make it hard to avoid this mistake. An effective solution is to define the research questions and analysis plan before observing the research outcomes—a process called preregistration. Preregistration distinguishes analyses and outcomes that result from predictions from those that result from postdictions. A variety of practical strategies are available to make the best possible use of preregistration in circumstances that fall short of the ideal application, such as when the data are preexisting. Services are now available for preregistration across all disciplines, facilitating a rapid increase in the practice. Widespread adoption of preregistration will increase distinctiveness between hypothesis generation and hypothesis testing and will improve the credibility of research findings.

Replication studies in psychological science sometimes fail to reproduce prior findings. If these studies use methods that are unfaithful to the original study or ineffective in eliciting the phenomenon of interest, then a failure to replicate may be a failure of the protocol rather than a challenge to the original finding. Formal pre-data-collection peer review by experts may address shortcomings and increase replicability rates. We selected 10 replication studies from the Reproducibility Project: Psychology (RP:P; Open Science Collaboration, 2015) for which the original authors had expressed concerns about the replication designs before data collection; only one of these studies had yielded a statistically significant effect (p < .05). Commenters suggested that lack of adherence to expert review and low-powered tests were the reasons that most of these RP:P studies failed to replicate the original effects. We revised the replication protocols and received formal peer review prior to conducting new replication studies. We administered the RP:P and revised protocols in multiple laboratories (median number of laboratories per original study = 6.5, range = 3–9; median total sample = 1,279.5, range = 276–3,512) for high-powered tests of each original finding with both protocols. Overall, following the preregistered analysis plan, we found that the revised protocols produced effect sizes similar to those of the RP:P protocols (Δr = .002 or .014, depending on analytic approach). The median effect size for the revised protocols (r = .05) was similar to that of the RP:P protocols (r = .04) and the original RP:P replications (r = .11), and smaller than that of the original studies (r = .37). Analysis of the cumulative evidence across the original studies and the corresponding three replication attempts provided very precise estimates of the 10 tested effects and indicated that their effect sizes (median r = .07, range = .00–.15) were 78% smaller, on average, than the original effect sizes (median r = .37, range = .19–.50).

Replication is vital for increasing precision and accuracy of scientific claims. However, when replications “succeed” or “fail,” they could have reputational consequences for the claim’s originators. Surveys of United States adults (N = 4,786), undergraduates (N = 428), and researchers (N = 313) showed that reputational assessments of scientists were based more on how they pursue knowledge and respond to replication evidence, not whether the initial results were true. When comparing one scientist that produced boring but certain results with another that produced exciting but uncertain results, opinion favored the former despite researchers’ belief in more rewards for the latter. Considering idealized views of scientific practices offers an opportunity to address incentives to reward both innovation and verification.

Across three studies, LoBue and DeLoache (2008) provided evidence suggesting that both young children and adults exhibit enhanced visual detection of evolutionarily relevant threat stimuli (as compared with nonthreatening stimuli). A replication of their Experiment 3, conducted by Cramblet Alvarez and Pipitone (2015) as part of the Reproducibility Project: Psychology (RP:P), demonstrated trends similar to those of the original study, but the effect sizes were smaller and not statistically significant. There were, however, some methodological differences (e.g., screen size) and sampling differences (the age of recruited children) between the original study and the RP:P replication study. Additionally, LoBue and DeLoache expressed concern over the choice of stimuli used in the RP:P replication. We sought to explore the possible moderating effects of these factors by conducting two new replications—one using the protocol from the RP:P and the other using a revised protocol. We collected data at four sites, three in Serbia and one in the United States (total N = 553). Overall, participants were not significantly faster at detecting threatening stimuli. Thus, results were not supportive of the hypothesis that visual detection of evolutionarily relevant threat stimuli is enhanced in young children. The effect from the RP:P protocol (d = −0.10, 95% confidence interval = [−1.02, 0.82]) was similar to the effect from the revised protocol (d = −0.09, 95% confidence interval = [−0.33, 0.15]), and the results from both the RP:P and the revised protocols were more similar to those found by Cramblet Alvarez and Pipitone than to those found by LoBue and DeLoache.

Ledgerwood (1) argues that there are two independent uses of preregistration that are conflated in Nosek et al. (2) and elsewhere: “Preregistering theoretical predictions enables theory falsifiability. Preregistering analysis plans enables type I error control.” We appreciate that the comment elevates the complementary roles of prediction and analysis plans in preregistration. We disagree that they are conflated in the sense of being “two types of preregistration.” To enable theory falsification, we agree that a preregistration should offer a prediction derived from theory and provide the theoretical context. However, a prediction without an analysis plan is inert for falsification. An analysis plan is necessary to specify how the prediction will be tested with … [↵][1]1To whom correspondence should be addressed. Email: nosek{at}virginia.edu. [1]: #xref-corresp-1-1