Traditional approaches to marking short-answer questions face limitations in timeliness, scalability, inter-rater reliability, and faculty time costs. Harnessing generative artificial intelligence (AI) to address some of these shortcomings is attractive. This study aims to validate the use of ChatGPT for evaluating short-answer assessments in an undergraduate medical program. Ten questions from the pre-clerkship medical curriculum were randomly chosen, and for each, six previously marked student answers were collected. These sixty answers were evaluated by ChatGPT in July 2023 under four conditions: with both a rubric and standard, with only a standard, with only a rubric, and with neither. ChatGPT displayed good Spearman correlations with a single human assessor (r = 0.6–0.7, p < 0.001) across all conditions, with the absence of a standard or rubric yielding the best correlation. Scoring differences were common (65–80%), but score adjustments of more than one point were less frequent (20–38%). Notably, the absence of a rubric resulted in systematically higher scores (p < 0.001, partial η2 = 0.33). Our findings demonstrate that ChatGPT is a viable, though imperfect, assistant to human assessment, performing comparably to a single expert assessor. This study serves as a foundation for future research on AI-based assessment techniques with potential for further optimization and increased reliability.
At the Michael G. DeGroote School of Medicine, a significant component of the MD curriculum involves written narrative reflections on topics related to professional identity in medicine, with written feedback provided by their in-person longitudinal facilitators (LFs). However, it remains to be understood how generative artificial intelligence chatbots, such as ChatGPT (GPT-4), augment the feedback process and how MD students perceive feedback provided by ChatGPT versus the feedback provided by their LFs. In this study, 15 MD students provided their written narrative reflections along with the feedback they received from their LFs. Their reflections were input into ChatGPT (GPT-4) to generate instantaneous personalized feedback. MD students rated both modalities of feedback using a Likert-scale survey, in addition to providing open-ended textual responses. Quantitative analysis involved mean comparisons and t-tests, while qualitative responses were coded for themes and representational quotations. The results showed that while the LF-provided feedback was rated slightly higher in six out of eight survey items, these differences were not statistically significant. In contrast, ChatGPT scored significantly higher in helping to identify strengths and areas for improvement, as well as in providing actionable steps for improvement. Criticisms of ChatGPT included a discernible “AI tone” and paraphrasing or misuse of quotations from the reflections. In addition, MD students valued LF feedback for being more personal and reflective of the real, in-person relationships formed with LFs. Overall, findings suggest that although skepticism regarding ChatGPT’s feedback exists amongst MD students, it represents a viable avenue for deepening reflective practice and easing some of the burden on LFs.
Electronic Medical Records (EMRs) are central to the modern healthcare system. Recent advances in artificial intelligence (AI), particularly generative artificial intelligence (GenAI), have opened new opportunities for the advancement of EMRs. This scoping review aims to explore the current real-world applications of GenAI within EMRs to support an understanding of AI applications in healthcare. A literature search was conducted following PRISMA-ScR guidelines. The search was conducted using Ovid MEDLINE, up to 28 October 2024, using a peer-reviewed search strategy. Overall, 55 studies were included. A list of five themes was generated by human reviewers based on the literature review: data manipulation (24), patient communication (9), clinical decision making (8), clinical prediction (8), summarization (4), and other (2). The majority of studies originated from the United States (35). Both proprietary and commercially available models were tested, with ChatGPT being the most commonly referenced LLM. As these models continue to be developed, their diverse use cases within EMRs have the potential to improve patient outcomes, enhance access to medical data, streamline hospital workflows, and reduce physician workload. However, continued problems surrounding data privacy, trust, bias, model hallucinations, and the need for robust evaluation remain. Further research considering the ethical, medical, and societal implications of GenAI applications in EMRs is essential to validate these findings and address existing limitations to support healthcare advancement.
Large language model-based virtual patients (LLM-VPs) are an emerging simulation tool for health professions education, but their design and integration into curricula are not well characterized. This scoping review mapped how LLM-VPs are being used for simulated clinical learning across health professions. Following a protocol registered on OSF, we searched MEDLINE, EMBASE, CENTRAL, Scopus, and Web of Science to 11 April 2025, per PRISMA-ScR guidelines, and included 21 studies that used LLMs to generate virtual patients for simulated clinical encounters. Data were extracted on technical design, fidelity domains, curricular integration, human factors, and Technology Acceptance Model constructs, and synthesized narratively. Most studies (n = 11) were pilot or feasibility evaluations with small samples (median 21) and used GPT-based models with dynamic text chat. Integration was limited to 10 studies that operated as pilots, 7 as electives, and 3 as core curricular components. The outcomes focused on Level 2 learning (clinical reasoning and preclinical OSCE performance), with predominantly self-report assessments. No studies reported Level 3 or 4 outcomes. Fidelity was strongest in cognitive, socio-cultural, and emotional domains, and 11 studies reported hallucinations or inaccurate outputs. LLM-VPs appear feasible and well-received but remain early-stage, underscoring the need for fidelity-aligned design and more rigorous, longitudinal evaluations.
Background: Generative artificial intelligence (AI) tools including ChatGPT have the potential to augment the process of designing examinations and assessments for medical learners, leading to time and resource savings, and the ability to produce large volumes of practice problems tailored to learner-specific strengths and weaknesses. Methods: This study compares the quality of free-text assessment problems and answer keys generated by ChatGPT to those produced by faculty educators for a renal and hematology curriculum subunit. Five expert reviewers reviewed a collection of 21 free-text assessment problems, 9 from a collection of historical assessment problems used in an undergraduate medical program and 12 produced with ChatGPT. Reviewers assigned a score from 1 to 5, reflecting the overall quality. Results: The average quality of problems generated by ChatGPT was greater than that of human-generated problems (4.00 vs. 2.71, p < 0.001). Using ordinal mixed-effect modeling, human-generated problems had significantly lower odds of receiving higher ratings than ChatGPT-generated problems (β = −2.43, 95% confidence interval −3.34 to −1.51, p < 0.001). Conclusions: It is suggested that ChatGPT can assist expert faculty educators in producing assessment tools, with direct benefits to medical learners, although it cannot entirely replace this role in its current state.