OBJECTIVES: ChatGPT is an artificial intelligence model that can interpret free-text prompts and return detailed, human-like responses across a wide domain of subjects. This study evaluated the extent of the threat posed by ChatGPT to the validity of short-answer assessment problems used to examine pre-clerkship medical students in our undergraduate medical education program. METHODS: Forty problems used in prior student assessments were retrieved and stratified by levels of Bloom's Taxonomy. Thirty of these problems were submitted to ChatGPT-3.5. For the remaining 10 problems, we retrieved past minimally passing student responses. Six tutors graded each of the 40 responses. Comparison of performance between student-generated and ChatGPT-generated answers aggregated as a whole and grouped by Bloom's levels of cognitive reasoning, was done using t-tests, ANOVA, Cronbach's alpha, and Cohen's d. Scores for ChatGPT-generated responses were also compared to historical class average performance. RESULTS: ChatGPT-generated responses received a mean score of 3.29 out of 5 (n = 30, 95% CI 2.93-3.65) compared to 2.38 for a group of students meeting minimum passing marks (n = 10, 95% CI 1.94-2.82), representing higher performance ( P = .008, η2 = 0.169), but was outperformed by historical class average scores on the same 30 problems (mean 3.67, P = .018) when including all past responses regardless of student performance level. There was no statistically significant trend in performance across domains of Bloom's Taxonomy. CONCLUSION: While ChatGPT was able to pass short answer assessment problems spanning the pre-clerkship curriculum, it outperformed only underperforming students. We remark that tutors in several cases were convinced that ChatGPT-produced responses were produced by students. Risks to assessment validity include uncertainty in identifying struggling students and inability to intervene in a timely manner. The performance of ChatGPT on problems requiring increasing demands of cognitive reasoning warrants further research.
Traditional approaches to marking short-answer questions face limitations in timeliness, scalability, inter-rater reliability, and faculty time costs. Harnessing generative artificial intelligence (AI) to address some of these shortcomings is attractive. This study aims to validate the use of ChatGPT for evaluating short-answer assessments in an undergraduate medical program. Ten questions from the pre-clerkship medical curriculum were randomly chosen, and for each, six previously marked student answers were collected. These sixty answers were evaluated by ChatGPT in July 2023 under four conditions: with both a rubric and standard, with only a standard, with only a rubric, and with neither. ChatGPT displayed good Spearman correlations with a single human assessor (r = 0.6–0.7, p < 0.001) across all conditions, with the absence of a standard or rubric yielding the best correlation. Scoring differences were common (65–80%), but score adjustments of more than one point were less frequent (20–38%). Notably, the absence of a rubric resulted in systematically higher scores (p < 0.001, partial η2 = 0.33). Our findings demonstrate that ChatGPT is a viable, though imperfect, assistant to human assessment, performing comparably to a single expert assessor. This study serves as a foundation for future research on AI-based assessment techniques with potential for further optimization and increased reliability.
There is an urgent need to prevent excessive pregnancy weight gain, a contributor to both maternal and child obesity. However, the majority of women had reported not being counseled to gain an appropriate amount of gestational weight by their health care providers. We developed a knowledge translation (KT) tool designed to facilitate the clinical interaction between pregnant women and their health care providers (HCPs). We piloted the tool on the impact on women’s knowledge of gestational weight gain (GWG) goals, and evaluated its potential in promoting appropriate knowledge about GWG within the 2009 Institute of Medicine guidelines. We conducted a prospective cohort study, comparing women’s knowledge about GWG after the KT tool to women from the same clinics and care providers the year prior. Our primary outcome was the proportion of women who reported receiving an appropriate GWG recommendation from their care provider. We evaluated knowledge on a survey conducted at enrollment in the cohort at ≤ 20 weeks gestation and evaluated participant satisfaction with the KT tool in the third trimester. We performed univariate and multivariable logistic regression analyses for differences in outcomes with historical controls from the same clinics. Our a priori sample size calculation required 130 participants to demonstrate a 15% increase in reported counseling about gestational weight gain. One hundred and forty-six women were recruited and 131 (90%) completed the enrollment survey. Women who received the KT tool were more likely to report receiving a specific GWG recommendation from their HCP (adjusted odds ratio [AOR] 3.45, 95% confidence interval [CI] 2.22-5.37) and discussing GWG topics with their HCP (AOR 7.96, 95% CI 4.41-14.37), and believing that there were risks to their infants with inadequate GWG (AOR 2.48, 95% CI 1.14-5.37). Half of women (49.5%) indicated that they would recommend the tool to a friend. Women who received the KT tool reported receiving more counseling on GWG from their HCPs and were more aware of the risks of gaining outside appropriate GWG recommendations. The association between GWG education and GWG requires further research.