9 April 2025 in Royal Society Open Science
Various open science practices have been proposed to improve the reproducibility and replicability of scientific research, but not for all practices, there may be evidence they are indeed effective. Therefore, we conducted a scoping review of the literature on interventions to improve reproducibility. We systematically searched Medline , Embase , Web of Science , PsycINFO , Scopus and Eric , on 18 August 2023. Any study empirically evaluating the effectiveness of interventions aimed at improving the reproducibility or replicability of scientific methods and findings was included. We summarized the retrieved evidence narratively and in evidence gap maps. Of the 105 distinct studies we included, 15 directly measured the effect of an intervention on reproducibility or replicability, while the remainder addressed a proxy outcome that might be expected to increase reproducibility or replicability, such as data sharing, methods transparency or pre-registration. Thirty studies were non-comparative and 27 were comparative but cross-sectional observational designs, precluding any causal inference. Despite studies investigating a range of interventions and addressing various outcomes, our findings indicate that in general the evidence base for which various interventions to improve reproducibility of research remains remarkably limited in many respects.
10 July 2024 in Open Research Europe
Background Many interventions, especially those linked to open science, have been proposed to improve reproducibility in science. To what extent these propositions are based on scientific evidence from empirical evaluations is not clear. Aims The primary objective is to identify Open Science interventions that have been formally investigated regarding their influence on reproducibility and replicability. A secondary objective is to list any facilitators or barriers reported and to identify gaps in the evidence. Methods We will search broadly by using electronic bibliographic databases, broad internet search, and contacting experts in the field of reproducibility, replicability, and open science. Any study investigating interventions for their influence on the reproducibility and replicability of research will be selected, including those studies additionally investigating drivers and barriers to the implementation and effectiveness of interventions. Studies will first be selected by title and abstract (if available) and then by reading the full text by at least two independent reviewers. We will analyze existing scientific evidence using scoping review and evidence gap mapping methodologies. Results The results will be presented in interactive evidence maps, summarized in a narrative synthesis, and serve as input for subsequent research. Review registration This protocol has been pre-registered on OSF under doi https://doi.org/10.17605/OSF.IO/D65YS
18 October 2023 in Open Research Europe
Background Many interventions, especially those linked to open science, have been proposed to combat the reproducibility crisis. To what extent these propositions are based on scientific evidence from empirical evaluations is not clear. Aims The primary objective is to identify interventions that have been formally investigated regarding their influence on reproducibility and replicability. A secondary objective is to list any facilitators or barriers reported and to identify gaps in the evidence. Methods We will search broadly by using electronic bibliographic databases, broad internet search, and contacting experts in the field of reproducibility, replicability, and open science. Any study investigating interventions for their influence on the reproducibility and replicability of research will be selected, including those studies additionally investigating drivers and barriers to the implementation and effectiveness of interventions. Studies will first be selected by title and abstract (if available), semi-automated, and then by reading the full text by at least two independent reviewers. We will analyze existing scientific evidence using scoping review and evidence gap mapping methodologies. Results The results will be presented in interactive evidence maps, summarized in a narrative synthesis, and serve as input for subsequent research. Review registration This protocol has been pre-registered on OSF under doi https://doi.org/10.17605/OSF.IO/D65YS
17 June 2024 in MetaArXiv
Various interventions – especially those related to open science – have been proposed to improve the reproducibility and replicability of scientific research. To assess whether and which interventions have been formally tested for their effectiveness in improving reproducibility and replicability, we conducted a scoping review of the literature on interventions to improve reproducibility. We systematically searched Medline, Embase, Web of Science, PsycINFO, Scopus and Eric, on August 18, 2023. Grey literature was requested from experts in the fields of reproducibility and open science. Any study empirically evaluating the effectiveness of interventions aimed at improving the reproducibility or replicability of scientific methods and findings was included. An intervention could be any action taken by either individual researchers or scientific institutions (e.g., research institutes, publishers and funders). We summarized the retrieved evidence narratively and in an evidence gap map. Of the 104 distinct studies we included, 15 directly measured the effect of an intervention on reproducibility or replicability, while the other research questions addressed a proxy outcome that might be expected to increase reproducibility or replicability, such as data sharing, methods transparency or preregistration. Thirty research questions within included studies were non-comparative and 27 were comparative but cross-sectional, precluding any causal inference. Possible limitations of our review may be the search and selection strategy, which was done by a large team including researchers from different disciplines and different expertise levels. Despite studies investigating a range of interventions and addressing various outcomes, our findings indicate that in general the evidence-base for which various interventions to improve reproducibility of research remains remarkably limited in many respects.
8 October 2024 in MetaArXiv
Objectives: To test whether improving a Christmas tree meringue recipe using reporting guidelines yields more appealing and sweeter meringues.Design: A prospective, superiority, single-blind, cluster-randomised (1:1) controlled trial Setting: A public participatory event in a large cultural facility in France Participants: Budding chefs with basic culinary skills, possessing the utensils necessary for baking Christmas tree meringues, and not having burned pasta in the past month (for safety reasons).Interventions: Each budding chef was randomised to a standard recipe for making Christmas tree meringues or to the same recipe written in consultation with a professional baker using the TIDieR checklist - a reporting guideline for description of complex interventions - plus a short video tutorial.Main outcome measures: The primary outcome was reproducibility in terms of visual aspect as rated by an independent jury which compared the cooked Christmas tree meringues with the recipe picture on a scale from 1 to 10. Secondary outcomes included colour, size, taste and survival time in the course of a sale organised as part of the public event. Analyses were performed in intention-to-eat (randomization unit: budding chefs / analysis unit: Christmas trees).Results: 60 budding chefs (30 in each group) baked a total of 845 Christmas tree meringues. There was no significant difference between the groups (mean difference=-0.1; [95%CI -0.99; 0.80]; p-value=0.84; intra-cluster correlation, ICC=0.77) on visual aspect. No difference was found for reproducibility in terms of colour (mean difference=-0.31; [95%CI -0.97; 0.35]; p-value=0.35; ICC=0.67) or size (mean difference=-0.17; [95%CI -1.07; 0.73]; p-value=0.71; ICC=0.74). There was no significant difference in terms of taste between the groups (mean difference= -0.55; [95%CI -1.62; 0.52]; p-value=0.31). 400 meringues were sold during the public event with no difference in survival time between groups (hazard ratio= 1.26 [95% CI 0.75–2.09], p-value=0.38) in favor of the control group.Conclusions: Our study failed to demonstrate that an improved recipe using the TIDieR reporting guideline with a video tutorial improved the reproducibility in terms of visual aspect, colour, size, taste and sales for Christmas tree meringues. The best way to succeed in baking Christmas tree meringues - and thereby to improve reproducibility of experiments - remains a mystery still to be solved by further explorations. Trial registration: https://osf.io/dnhbx