Background: The systematic review of clinical research papers is a labor-intensive and time-consuming process that often involves the screening of thousands of titles and abstracts. The accuracy and efficiency of this process are critical for the quality of the review and subsequent health care decisions. Traditional methods rely heavily on human reviewers, often requiring a significant investment of time and resources. Objective: This study aims to assess the performance of the OpenAI generative pretrained transformer (GPT) and GPT-4 application programming interfaces (APIs) in accurately and efficiently identifying relevant titles and abstracts from real-world clinical review data sets and comparing their performance against ground truth labeling by 2 independent human reviewers. Methods: We introduce a novel workflow using the Chat GPT and GPT-4 APIs for screening titles and abstracts in clinical reviews. A Python script was created to make calls to the API with the screening criteria in natural language and a corpus of title and abstract data sets filtered by a minimum of 2 human reviewers. We compared the performance of our model against human-reviewed papers across 6 review papers, screening over 24,000 titles and abstracts. Results: Our results show an accuracy of 0.91, a macro F1-score of 0.60, a sensitivity of excluded papers of 0.91, and a sensitivity of included papers of 0.76. The interrater variability between 2 independent human screeners was κ=0.46, and the prevalence and bias-adjusted κ between our proposed methods and the consensus-based human decisions was κ=0.96. On a randomly selected subset of papers, the GPT models demonstrated the ability to provide reasoning for their decisions and corrected their initial decisions upon being asked to explain their reasoning for incorrect classifications. Conclusions: Large language models have the potential to streamline the clinical review process, save valuable time and effort for researchers, and contribute to the overall quality of clinical reviews. By prioritizing the workflow and acting as an aid rather than a replacement for researchers and reviewers, models such as GPT-4 can enhance efficiency and lead to more accurate and reliable conclusions in medical research.

Access to practice opportunities for history-taking in undergraduate medical education can be resource-limited. Large language models are a potential avenue to address this. This study sought to characterize changes in learner self-reported confidence with history-taking before and after a simulation with an LLM-based patient and understand learner experience with and the acceptability of virtual LLM-based patients. This was a multi-method study conducted at McMaster University. Simulations were facilitated with the OSCEai tool. Data was collected through surveys with a Likert scale and open-ended questions and semi-structured interviews. A total of 24 participants generated 93 survey responses and 17 interviews. Overall, participants reported a 14.6% increase in comfort with history-taking. Strengths included its flexibility, accessibility, detailed feedback, and ability to provide a judgement-free space to practice. Limitations included its lower fidelity compared to standardized patients and at times repetitive and less clinically relevant feedback as compared to preceptors. It was overall viewed best as a supplement rather than a replacement for standardized patients. In conclusion, LLM-based virtual patients were feasible and valued as an adjunct tool. They can support scalable, personalized practice. Future work is needed to understand objective metrics of improvement and to design curricular strategies for integration.

Background: This study aims to evaluate the effectiveness of the OSCEai, a large language model-based platform that simulates clinical encounters, in enhancing undergraduate medical education. Methods: A web-based application, OSCEai, was developed to bridge theoretical and practical learning. Following use, medical students from the University of Calgary Class of 2026 completed an anonymized survey on the usability, utility, and overall experience of OSCEai. Results: A total of 37 respondents answered the anonymized survey. The OSCEai platform was highly valued for its ability to provide data on demand (33/37), support self-paced learning (30/37), and offer realistic patient interactions (29/37). The ease of use and medical content quality were rated at 4.73 (95% CI: 4.58 to 4.88) and 4.70 (95% CI: 4.55 to 4.86) out of 5, respectively. Some participants (8/37) commented that few cases were not representative and needed clarification about app functionality. Despite these limitations, OSCEai was favorably compared to lecture-based teaching methods, with an overall reception rating of 4.62 (95% CI: 4.46 to 4.79) out of 5. Interpretation: The OSCEai platform fills a gap in medical training through its scalable, interactive, and personalized design. The findings suggest that integrating technologies, like OSCEai, into medical curricula can enhance the quality and efficacy of medical education.

Summary: Background: The global launch of ChatGPT on November 30, 2022 has sparked widespread public interest in large language models (LLMs), and interest in the medical community is growing. Indeed, recent preprints on medRxiv have examined ChatGPT and GPT-3 in the context of standardized exams, such as the United States Medical Licensing Examination. These studies demonstrate modest performance relative to national averages. In this work, we enhance OpenAI’s GPT-3 model through zero-shot learning, anticipating that it outperforms experienced neurosurgeons in written question-answer tasks for common clinical and surgical questions on vestibular schwannoma. We aimed to address LLM accountability by including in-text citations and references to the responses provided by GPT-3. Methods: The analysis involved (i) creating a dataset through web scraping, (ii) developing a chat-based platform called neuroGPT-X, (iii) enlisting expert neurosurgeons across international centers to create and answer questions and evaluate responses, and (iv) analyzing the evaluation results on the management of vestibular schwannoma. The survey had a blinded and unblinded phase. In the blinded phase, a neurosurgeon with 30+ years of experience curated 15 questions regarding common clinical and surgical contexts of vestibular schwannoma. Then, four neurosurgeons, ChatGPT (January 30, 2023 model, aka naive GPT), and a context-enriched GPT model independently provided their responses. Three experienced neurosurgeons blindly evaluated the responses for accuracy, coherence, relevance, thoroughness, speed, and overall rating. Then, all seven neurosurgeons were unblinded to all responses and provided their thoughts on the potential of expert LLMs in the clinical setting. Findings: Both the naive and content-enriched GPT models provided faster responses to the standardized question set (p0.999) or performance across the aforementioned domains (p>0.999). Of interest, all expert surgeons expressed concerns about the reliability of GPT in accurately addressing the nuances and controversies surrounding the management of vestibular schwannoma. Further, we developed neuroGPT-X, a chat-based platform designed to provide point-of-care clinical support and mitigate limitations of human memory. neuroGPT-X incorporates features such as in-text citations and references to enable accurate, relevant, and reliable information in real-time. Interpretation: A context-enriched GPT model provided non-inferior responses compared to experienced neurosurgeons in generating written responses to a complex neurosurgical problem for which evidence-based consensus for management is lacking. We show that context enrichment of LLMs is well-suited to transform clinical practice by providing subspecialty-level answers to clinical questions in an accountable manner. Research in Context: Evidence before this study: We searched PubMed for “(vestibular schwannoma OR acoustic schwannoma) AND (GPT-3 OR Generative Pretrained Transformer OR large language model)” with no filters and identified no relevant articles. We then searched PubMed using the string “(subspecialty OR neurosurgery OR physician) AND (GPT-3 OR Generative Pretrained Transformer OR large language model) AND (fine-tuning OR context enrichment)” with no filters and identified three studies. One study noted that domain-specific knowledge enhanced pre-trained language models. Added value of this study: To our knowledge, this is the first study to show the non-inferiority of a context-enriched LLM in a question-answer task on common clinical and surgical questions compared to experienced neurosurgeons worldwide, determined by their neurosurgical colleagues. Furthermore, we developed the first online platform incorporating an LLM, chat memory, in-text citations, and references regarding comprehensive vestibular schwannoma management. To assess the model’s performance, a neurosurgeon with 30+ years of experience managing patients with vestibular schwannoma curated 15 questions to the model, ChatGPT, and four international expert neurosurgeons. A separate, blinded group of three expert neurosurgeons assessed these answers for accuracy, coherence, relevance, thoroughness, speed, and overall rating. This study demonstrated the capability of context-enriched LLMs as point-of-care informational aids. Importantly, all expert surgeons raised questions regarding the nuances and role of human experience and intuition that GPT may not capture in generating opinions or recommendations. Implications of all the available evidence: The present study, with its subspecialist-level performance and interpretable results, suggests that context-enriched LLMs show promise as a point-of-care medical resource. Evaluations from experienced neurosurgeons showed that a context-enriched GPT model was rated similarly to neurosurgeon responses across evaluation domains in this study. This work serves as a springboard for expanding this tool into more medical specialties, incorporating evidence-based clinical information, and developing expert-level dialogue surrounding LLMs in healthcare.

Background: The financial implications of central nervous system (CNS) cancers are substantial, not only for the healthcare service and payers, but also for the patients who bear the brunt of direct, indirect, and intangible costs. This study sought to investigate the impact of healthcare spending on CNS cancer survival using recent US data.Methods: This study used public data from the Disease Expenditure Project 2016 and the Global Burden of Disease Study 2019. The primary outcome was the annual healthcare spending trend from 1996 and 2016 on CNS tumors adjusted for disease prevalence, alongside morbidity and mortality. Secondary outcomes included drivers of change in healthcare expenditures for CNS cancers. Subgroup analysis was performed stratified by age group, expenditure type, and care type provided.Results: There was a significant increase in total healthcare spending on CNS cancers from $2.72 billion (95% CI: $2.47B to $2.97B) in 1996 to $6.85 billion (95% CI: $5.98B to $7.57B) in 2016. Despite the spending increase, the mortality rate per 100 000 people increased, with 5.30 ± 0.47 in 1996 and 7.02 ± 0.47 in 2016, with an average of 5.78 ± 0.47 deaths per 100 000 over the period. The subgroups with the highest expenditure included patients aged 45 to 64, those with private insurance, and those receiving inpatient care.Conclusions: This study highlights a significant rise in healthcare costs for CNS cancers without corresponding improvements in mortality rate, indicating a mismatch of healthcare spending, contemporary advances, and patient outcomes as it relates to mortality.

BACKGROUND The systematic review of clinical research papers is a labor-intensive and time-consuming process that often involves the screening of thousands of titles and abstracts. The accuracy and efficiency of this process are critical for the quality of the review and subsequent health care decisions. Traditional methods rely heavily on human reviewers, often requiring a significant investment of time and resources. OBJECTIVE This study aims to assess the performance of the OpenAI generative pretrained transformer (GPT) and GPT-4 application programming interfaces (APIs) in accurately and efficiently identifying relevant titles and abstracts from real-world clinical review data sets and comparing their performance against ground truth labeling by 2 independent human reviewers. METHODS We introduce a novel workflow using the Chat GPT and GPT-4 APIs for screening titles and abstracts in clinical reviews. A Python script was created to make calls to the API with the screening criteria in natural language and a corpus of title and abstract data sets filtered by a minimum of 2 human reviewers. We compared the performance of our model against human-reviewed papers across 6 review papers, screening over 24,000 titles and abstracts. RESULTS Our results show an accuracy of 0.91, a macro F1-score of 0.60, a sensitivity of excluded papers of 0.91, and a sensitivity of included papers of 0.76. The interrater variability between 2 independent human screeners was κ=0.46, and the prevalence and bias-adjusted κ between our proposed methods and the consensus-based human decisions was κ=0.96. On a randomly selected subset of papers, the GPT models demonstrated the ability to provide reasoning for their decisions and corrected their initial decisions upon being asked to explain their reasoning for incorrect classifications. CONCLUSIONS Large language models have the potential to streamline the clinical review process, save valuable time and effort for researchers, and contribute to the overall quality of clinical reviews. By prioritizing the workflow and acting as an aid rather than a replacement for researchers and reviewers, models such as GPT-4 can enhance efficiency and lead to more accurate and reliable conclusions in medical research.