Keyna Bracken

Dr. Keyna Bracken

orcid

Research Focus & Metrics

Academic Career Length

12Years (2015-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
1 January 2023 in Journal of Medical Education and Curricular Development
Verified
Journal Article
Open access
LMLeo Morjaria
Leo Morjaria
LBLevi Burns
Levi Burns
KBKeyna Bracken
Keyna Bracken
QNQuang N. Ngo
Quang N. Ngo
MLMark Lee
Mark Lee
MSMatthew Sibbald
Matthew Sibbald
OBJECTIVES: ChatGPT is an artificial intelligence model that can interpret free-text prompts and return detailed, human-like responses across a wide domain of subjects. This study evaluated the extent of the threat posed by ChatGPT to the validity of short-answer assessment problems used to examine pre-clerkship medical students in our undergraduate medical education program. METHODS: Forty problems used in prior student assessments were retrieved and stratified by levels of Bloom's Taxonomy. Thirty of these problems were submitted to ChatGPT-3.5. For the remaining 10 problems, we retrieved past minimally passing student responses. Six tutors graded each of the 40 responses. Comparison of performance between student-generated and ChatGPT-generated answers aggregated as a whole and grouped by Bloom's levels of cognitive reasoning, was done using t-tests, ANOVA, Cronbach's alpha, and Cohen's d. Scores for ChatGPT-generated responses were also compared to historical class average performance. RESULTS: ChatGPT-generated responses received a mean score of 3.29 out of 5 (n = 30, 95% CI 2.93-3.65) compared to 2.38 for a group of students meeting minimum passing marks (n = 10, 95% CI 1.94-2.82), representing higher performance ( P = .008, η2 = 0.169), but was outperformed by historical class average scores on the same 30 problems (mean 3.67, P = .018) when including all past responses regardless of student performance level. There was no statistically significant trend in performance across domains of Bloom's Taxonomy. CONCLUSION: While ChatGPT was able to pass short answer assessment problems spanning the pre-clerkship curriculum, it outperformed only underperforming students. We remark that tutors in several cases were convinced that ChatGPT-produced responses were produced by students. Risks to assessment validity include uncertainty in identifying struggling students and inability to intervene in a timely manner. The performance of ChatGPT on problems requiring increasing demands of cognitive reasoning warrants further research.

19 January 2024 in International Medical Education
Verified
Journal Article
Open access
PDF file
LMLeo Morjaria
Leo Morjaria
LBLevi Burns
Levi Burns
KBKeyna Bracken
Keyna Bracken
Anthony J LevinsonAnthony J Levinson
Anthony J Levinson
QNQuang N. Ngo
Quang N. Ngo
MSMatthew Sibbald
Matthew Sibbald
Traditional approaches to marking short-answer questions face limitations in timeliness, scalability, inter-rater reliability, and faculty time costs. Harnessing generative artificial intelligence (AI) to address some of these shortcomings is attractive. This study aims to validate the use of ChatGPT for evaluating short-answer assessments in an undergraduate medical program. Ten questions from the pre-clerkship medical curriculum were randomly chosen, and for each, six previously marked student answers were collected. These sixty answers were evaluated by ChatGPT in July 2023 under four conditions: with both a rubric and standard, with only a standard, with only a rubric, and with neither. ChatGPT displayed good Spearman correlations with a single human assessor (r = 0.6–0.7, p < 0.001) across all conditions, with the absence of a standard or rubric yielding the best correlation. Scoring differences were common (65–80%), but score adjustments of more than one point were less frequent (20–38%). Notably, the absence of a rubric resulted in systematically higher scores (p < 0.001, partial η2 = 0.33). Our findings demonstrate that ChatGPT is a viable, though imperfect, assistant to human assessment, performing comparably to a single expert assessor. This study serves as a foundation for future research on AI-based assessment techniques with potential for further optimization and increased reliability.

30 April 2015 in BMC Pregnancy and Childbirth
Verified
Journal Article
Open access
PDF file
SMSarah D McDonald
Sarah D McDonald
CPChristina K Park
Christina K Park
EPEleanor Pullenayegum
Eleanor Pullenayegum
KBKeyna Bracken
Keyna Bracken
WSWendy Sword
Wendy Sword
MBMelissa Brouwers
Melissa Brouwers
There is an urgent need to prevent excessive pregnancy weight gain, a contributor to both maternal and child obesity. However, the majority of women had reported not being counseled to gain an appropriate amount of gestational weight by their health care providers. We developed a knowledge translation (KT) tool designed to facilitate the clinical interaction between pregnant women and their health care providers (HCPs). We piloted the tool on the impact on women’s knowledge of gestational weight gain (GWG) goals, and evaluated its potential in promoting appropriate knowledge about GWG within the 2009 Institute of Medicine guidelines. We conducted a prospective cohort study, comparing women’s knowledge about GWG after the KT tool to women from the same clinics and care providers the year prior. Our primary outcome was the proportion of women who reported receiving an appropriate GWG recommendation from their care provider. We evaluated knowledge on a survey conducted at enrollment in the cohort at ≤ 20 weeks gestation and evaluated participant satisfaction with the KT tool in the third trimester. We performed univariate and multivariable logistic regression analyses for differences in outcomes with historical controls from the same clinics. Our a priori sample size calculation required 130 participants to demonstrate a 15% increase in reported counseling about gestational weight gain. One hundred and forty-six women were recruited and 131 (90%) completed the enrollment survey. Women who received the KT tool were more likely to report receiving a specific GWG recommendation from their HCP (adjusted odds ratio [AOR] 3.45, 95% confidence interval [CI] 2.22-5.37) and discussing GWG topics with their HCP (AOR 7.96, 95% CI 4.41-14.37), and believing that there were risks to their infants with inadequate GWG (AOR 2.48, 95% CI 1.14-5.37). Half of women (49.5%) indicated that they would recommend the tool to a friend. Women who received the KT tool reported receiving more counseling on GWG from their HCPs and were more aware of the risks of gaining outside appropriate GWG recommendations. The association between GWG education and GWG requires further research.

22 June 2021 in Medical Science Educator
Verified
Journal Article
Closed access
PDF file
KBKeyna Bracken
Keyna Bracken
ALAnthony J. Levinson
Anthony J. Levinson
MMMeera Mahmud
Meera Mahmud
IAIlana Allice
Ilana Allice
MVMeredith Vanstone
Meredith Vanstone
LGLawrence Grierson
Lawrence Grierson

4 December 2024 in Medical Science Educator
Verified
Journal Article
Closed access
PDF file
KBKeyna Bracken
Keyna Bracken
ASAmr Saleh
Amr Saleh
JSJeremy Sandor
Jeremy Sandor
MSMatthew Sibbald
Matthew Sibbald
MLMicheal Lee-Poy
Micheal Lee-Poy
QNQuang Ngo
Quang Ngo