29 April 2023 in Systematic Reviews
In this commentary, we discuss ChatGPT and our perspectives on its utility to systematic reviews (SRs) through the appropriateness and applicability of its responses to SR related prompts. The advancement of artificial intelligence (AI)-assisted technologies leave many wondering about the current capabilities, limitations, and opportunities for integration AI into scientific endeavors. Large language models (LLM)—such as ChatGPT, designed by OpenAI—have recently gained widespread attention with their ability to respond to various prompts in a natural-sounding way. Systematic reviews (SRs) utilize secondary data and often require many months and substantial financial resources to complete, making them attractive grounds for developing AI-assistive technologies. On February 6, 2023, PICO Portal developers hosted a webinar to explore ChatGPT’s responses to tasks related to SR methodology. Our experience from exploring the responses of ChatGPT suggest that while ChatGPT and LLMs show some promise for aiding in SR-related tasks, the technology is in its infancy and needs much development for such applications. Furthermore, we advise that great caution should be taken by non-content experts in using these tools due to much of the output appearing, at a high level, to be valid, while much is erroneous and in need of active vetting.
3 November 2021 in Journal of Clinical Epidemiology
Most systematic reviews of interventions focus on potential benefits. Common methods and assumptions that are appropriate for assessing benefits can be inappropriate for harms. This paper provides a primer on researching harms, particularly in systematic reviews. Commentary describing challenges with assessing harm. Investigators should be familiar with various terminologies used to describe, classify, and group harms. Published reports of clinical trials include limited information about harms, so systematic reviewers should not depend on these studies and journal articles to reach conclusions about harms. Visualizations might improve communication of multiple dimensions of harms such as severity, relatedness, and timing. The terminology, classification, detection, collection, and reporting of harms create unique challenges that take time, expertise, and resources to navigate in both primary studies and evidence syntheses. Systematic reviewers might reach incorrect conclusions if they focus on evidence about harms found in published reports of randomized trials of a particular health problem. Systematic reviews could be improved through better identification and reporting of harms in primary studies and through better training and uptake of appropriate methods for synthesizing evidence about harms.
17 April 2019 in American Journal of Epidemiology
Women comprise about half of senior epidemiologists, but little is known about whether they are also viewed as leaders (i.e., authorities) in the field. We believe editorial roles are markers of leadership in a field. Our objective was to describe the distribution of gender across authorship of editorials published in 5 high-impact epidemiology journals over the past 8 years. We included editorials and commentaries published in American Journal of Epidemiology, European Journal of Epidemiology, Epidemiology, International Journal of Epidemiology, and Journal of Clinical Epidemiology between 2010 and 2017. We classified genders of all authors as woman, man, or unknown and computed the proportions of women editorial authors over all journals and according to position (e.g., first author). Only 31% (682/2,228) of all editorial authors and 36% (524/1,477) of unique editorial authors (i.e., counting each editorial author name only once) were women. We identified 1,180 editorials; 594 had sole authors, 24% (141/594) of whom were women, and 586 had 2 or more authors, 31% (184/586) of which had women as first authors. If women are underrepresented as editorial authors across epidemiology journals (e.g., as a marker of epidemiology leadership), the situation merits immediate correction.
1 March 2022 in Journal of Clinical Epidemiology
In this methodologic study (Part 2 of 2), we examined the overlap in sources of evidence and the corresponding results for harms in systematic reviews for gabapentin. We extracted all citations referenced as sources of evidence for harms of gabapentin from 70 systematic reviews, as well as the harms assessed and numerical results. We assessed consistency of harms between pairs of reviews with a high degree of overlap in sources of evidence (>50%) as determined by corrected covered area (CCA). We found 514 reports cited across 70 included reviews. Most reports (244/514, 48%) were not cited in more than one review. Among 18 pairs of reviews, we found reviews had differences in which harms were assessed and their choice to meta-analyze estimates or present descriptive summaries. When a specific harm was meta-analyzed in a pair of reviews, we found similar effect estimates. Differences in harms results across reviews can occur because the choice of harms is driven by reviewer preferences, rather than standardized approaches to selecting harms for assessment. A paradigm shift is needed in the current approach to synthesizing harms.
1 March 2022 in Journal of Clinical Epidemiology
We compared methods used with current recommendations for synthesizing harms in systematic reviews and meta-analyses (SRMAs) of gabapentin. We followed recommended systematic review practices. We selected reliable SRMAs of gabapentin (i.e., met a pre-defined list of methodological criteria) that assessed at least one harm. We extracted and compared methods in four areas: pre-specification, searching, analysis, and reporting. Whereas our focus in this paper is on the methods used, Part 2 examines the results for harms across reviews. We screened 4320 records and identified 157 SRMAs of gabapentin, 70 of which were reliable. Most reliable reviews (51/70; 73%) reported following a general guideline for SRMA conduct or reporting, but none reported following recommendations specifically for synthesizing harms. Across all domains assessed, review methods were designed to address questions of benefit and rarely included the additional methods that are recommended for evaluating harms. Approaches to assessing harms in SRMAs we examined are tokenistic and unlikely to produce valid summaries of harms to guide decisions. A paradigm shift is needed. At a minimal, reviewers should describe any limitations to their assessment of harms and provide clearer descriptions of methods for synthesizing harms.