Another social science looks at itself: Experimental economists have joined the reproducibility discussion by replicating selected published experiments from two top-tier journals in economics. Camerer et al. found that two-thirds of the 18 studies examined yielded replicable estimates of effect size and direction. This proportion is somewhat lower than unaffiliated experts were willing to bet in an associated prediction market, but roughly in line with expectations from sample sizes and P values. Science , this issue p. 1433
We measure how accurately replication of experimental results can be predicted by black-box statistical models. With data from four large-scale replication projects in experimental psychology and economics, and techniques from machine learning, we train predictive models and study which variables drive predictable replication. The models predicts binary replication with a cross-validated accuracy rate of 70% (AUC of 0.77) and estimates of relative effect sizes with a Spearman ρ of 0.38. The accuracy level is similar to market-aggregated beliefs of peer scientists [1, 2]. The predictive power is validated in a pre-registered out of sample test of the outcome of [3], where 71% (AUC of 0.73) of replications are predicted correctly and effect size correlations amount to ρ = 0.25. Basic features such as the sample and effect sizes in original papers, and whether reported effects are single-variable main effects or two-variable interactions, are predictive of successful replication. The models presented in this paper are simple tools to produce cheap, prognostic replicability metrics. These models could be useful in institutionalizing the process of evaluation of new findings and guiding resources to those direct replications that are likely to be most informative.
9 November 2023 in Proceedings of the National Academy of Sciences
The COVID-19 pandemic struck societies directly and indirectly, not just challenging population health but disrupting many aspects of life. Different effects of the spreading virus—and the measures to fight it—are reported and discussed in different scientific fora, with hard-to-compare methods and metrics from different traditions. While the pandemic struck some groups more than others, it is difficult to assess the comprehensive impact on social inequalities. This paper gauges social inequalities using individual-level administrative data for Sweden’s entire population. We describe and analyze the relative risks for different social groups in four dimensions—gender, education, income, and world region of birth—to experience three types of COVID-19 incidence, as well as six additional negative life outcomes that reflect general health, access to medical care, and economic strain. During the pandemic, the overall population faced severe morbidity and mortality from COVID-19 and saw higher all-cause mortality, income losses and unemployment risks, as well as reduced access to medical care. These burdens fell more heavily on individuals with low income or education and on immigrants. Although these vulnerable groups experienced larger absolute risks of suffering the direct and indirect consequences of the pandemic, the relative risks in pandemic years (2020 and 2021) were conspicuously similar to those in prepandemic years (2016 to 2019).
The COVID-19 pandemic has demonstrated the importance of unbiased, real-time statistics of trends in disease events in order to achieve an effective response. Because of reporting delays, real-time statistics frequently underestimate the total number of infections, hospitalizations and deaths. When studied by event date, such delays also risk creating an illusion of a downward trend. Here, we describe a statistical methodology for predicting true daily quantities and their uncertainty, estimated using historical reporting delays. The methodology takes into account the observed distribution pattern of the lag. It is derived from the “removal method”—a well-established estimation framework in the field of ecology.
The 2014 Research Excellence Framework (REF2014) was conducted to assess the quality of research carried out at higher education institutions in the UK over a 6 year period. However, the process was criticized for being expensive and bureaucratic, and it was argued that similar information could be obtained more simply from various existing metrics. We were interested in whether a prediction market on the outcome of REF2014 for 33 chemistry departments in the UK would provide information similar to that obtained during the REF2014 process. Prediction markets have become increasingly popular as a means of capturing what is colloquially known as the ‘wisdom of crowds’, and enable individuals to trade ‘bets’ on whether a specific outcome will occur or not. These have been shown to be successful at predicting various outcomes in a number of domains (e.g. sport, entertainment and politics), but have rarely been tested against outcomes based on expert judgements such as those that formed the basis of REF2014.