6 April 2022 in Current Directions in Psychological Science
Psychological science’s “credibility revolution” has produced an explosion of metascientific work on improving research practices. Although much attention has been paid to replicability (reducing false positives), improving credibility depends on addressing a wide range of problems afflicting psychological science, beyond simply making psychology research more replicable. Here we focus on the “four validities” and highlight recent developments—many of which have been led by early-career researchers—aimed at improving these four validities in psychology research. We propose that the credibility revolution in psychology, which has its roots in replicability, can be harnessed to improve psychology’s validity more broadly.
16 February 2024 in Research Integrity and Peer Review
Journal editors have a large amount of power to advance open science in their respective fields by incentivising and mandating open policies and practices at their journals. The Data PASS Journal Editors Discussion Interface (JEDI, an online community for social science journal editors:www.dpjedi.org) has collated several resources on embedding open science in journal editing (www.dpjedi.org/resources). However, it can be overwhelming as an editor new to open science practices to know where to start. For this reason, we created a guide for journal editors on how to get started with open science. The guide outlines steps that editors can take to implement open policies and practices within their journal, and goes through the what, why, how, and worries of each policy and practice. This manuscript introduces and summarizes the guide (full guide:https://doi.org/10.31219/osf.io/hstcx).
30 September 2020 in Group Processes & Intergroup Relations
Public reactions to protests are often divided, with some viewing the protest as a legitimate response to injustice and others perceiving the protest as illegitimate. We examine how online news sources oriented to different audiences frame protest, potentially encouraging these divergent reactions. We focus on online news coverage following the 2014 police shooting of a Black teenager, Michael Brown. Preregistered analyses of headlines and images and their captions showed that sources oriented toward African Americans were more likely to include content conveying racial injustice and legitimacy of the subsequent protests than sources oriented toward a general audience. In contrast, general audience sources emphasized conflict between protesters and police, making fewer references to the protesters’ cause. Whereas much work on media segregation addresses the propensity of audiences to consume different sources, our work suggests that news sources may also contribute to information fragmentation by differentially framing the same events.
16 November 2020 in MetaArXiv
In Registered Reports (RRs), initial peer review and in-principle acceptance occurs before knowing the research outcomes. This combats publication bias and distinguishes planned and unplanned research. How RRs could improve the credibility of research findings is straightforward, but there is little empirical evidence. Also, there could be unintended costs such as reducing novelty. 353 researchers peer reviewed a pair of papers from 29 published RRs from psychology and neuroscience and 57 non-RR comparison papers. RRs outperformed comparison papers on all 19 criteria (mean difference=0.46; Scale range -4 to +4) with effects ranging from little improvement in novelty (0.13, 95% credible interval [-0.24, 0.49]) and creativity (0.22, [-0.14, 0.58]) to larger improvements in rigor of methodology (0.99, [0.62, 1.35]) and analysis (0.97, [0.60, 1.34]) and overall paper quality (0.66, [0.30, 1.02]). RRs could improve research quality while reducing publication bias and ultimately improve the credibility of the published literature.
7 October 2020 in PsyArXiv
Psychological science’s "credibility revolution" has produced an explosion of metascientific work on improving research practices. While much attention has been paid to replicability (reducing false positives), improving credibility depends on addressing a wide range of problems afflicting psychological science, beyond simply making psychology more replicable. Here we focus on the "four validities" (Shadish et al., 2002) and highlight recent developments aimed at improving these four validities in psychology research, many of which have been led by early career researchers. We propose that the credibility revolution in psychology, which has its roots in replicability, can be harnessed to help psychology improve its validity more broadly.
13 April 2022 in Royal Society Open Science
What research practices should be considered acceptable? Historically, scientists have set the standards for what constitutes acceptable research practices. However, there is value in considering non-scientists’ perspectives, including research participants'. 1873 participants from MTurk and university subject pools were surveyed after their participation in one of eight minimal-risk studies. We asked participants how they would feel if (mostly) common research practices were applied to their data: p -hacking/cherry-picking results, selective reporting of studies, Hypothesizing After Results are Known (HARKing), committing fraud, conducting direct replications, sharing data, sharing methods, and open access publishing. An overwhelming majority of psychology research participants think questionable research practices (e.g. p -hacking, HARKing) are unacceptable (68.3–81.3%), and were supportive of practices to increase transparency and replicability (71.4–80.1%). A surprising number of participants expressed positive or neutral views toward scientific fraud (18.7%), raising concerns about data quality. We grapple with this concern and interpret our results in light of the limitations of our study. Despite the ambiguity in our results, we argue that there is evidence (from our study and others’) that researchers may be violating participants' expectations and should be transparent with participants about how their data will be used.