Background: ChatGPT is among the most popular large language models (LLMs), exhibiting proficiency in various standardized tests, including multiple-choice medical board examinations. However, its performance on otolaryngology–head and neck surgery (OHNS) certification examinations and open-ended medical board certification examinations has not been reported. Objective: We aimed to evaluate the performance of ChatGPT on OHNS board examinations and propose a novel method to assess an AI model’s performance on open-ended medical board examination questions. Methods: Twenty-one open-ended questions were adopted from the Royal College of Physicians and Surgeons of Canada’s sample examination to query ChatGPT on April 11, 2023, with and without prompts. A new model, named Concordance, Validity, Safety, Competency (CVSC), was developed to evaluate its performance. Results: In an open-ended question assessment, ChatGPT achieved a passing mark (an average of 75% across 3 trials) in the attempts and demonstrated higher accuracy with prompts. The model demonstrated high concordance (92.06%) and satisfactory validity. While demonstrating considerable consistency in regenerating answers, it often provided only partially correct responses. Notably, concerning features such as hallucinations and self-conflicting answers were observed. Conclusions: ChatGPT achieved a passing score in the sample examination and demonstrated the potential to pass the OHNS certification examination of the Royal College of Physicians and Surgeons of Canada. Some concerns remain due to its hallucinations, which could pose risks to patient safety. Further adjustments are necessary to yield safer and more accurate answers for clinical implementation.
19 June 2024 in Otolaryngology–Head and Neck Surgery
Objective: The recent surge in popularity of large language models (LLMs), such as ChatGPT, has showcased their proficiency in medical examinations and potential applications in health care. However, LLMs possess inherent limitations, including inconsistent accuracy, specific prompting requirements, and the risk of generating harmful hallucinations. A domain‐specific model might address these limitations effectively.Study Design: Developmental design.Setting: Virtual.Methods: Otolaryngology–head and neck surgery (OHNS) relevant data were systematically gathered from open‐access Internet sources and indexed into a knowledge database. We leveraged Retrieval‐Augmented Language Modeling to recall this information and utilized it for pretraining, which was then integrated into ChatGPT4.0, creating an OHNS‐specific knowledge question & answer platform known as ChatENT. The model is further tested on different types of questions.Results: ChatENT showed enhanced performance in the analysis and interpretation of OHNS information, outperforming ChatGPT4.0 in both the Canadian Royal College OHNS sample examination questions challenge and the US board practice questions challenge, with a 58.4% and 26.0% error reduction, respectively. ChatENT generated fewer hallucinations and demonstrated greater consistency.Conclusion: To the best of our knowledge, ChatENT is the first specialty‐specific knowledge retrieval artificial intelligence in the medical field that utilizes the latest LLM. It appears to have considerable promise in areas such as medical education, patient education, and clinical decision support. The model has demonstrated the capacity to overcome the limitations of existing LLMs, thereby signaling a future of more precise, safe, and user‐friendly applications in the realm of OHNS and other medical fields.
Background: There is a pressing need for healthcare to respond to the climate crisis. Family physicians, given their central role in community healthcare provision, are strategically placed to lead, support, and promote sustainable healthcare, yet guidance on how to do this is fragmented. Objective: To identify and evaluate toolkits and aids on sustainable healthcare to act as a curated resource for family physicians and their care teams interested in delivering evidence-based sustainable healthcare in their clinical practices. Methods: A scoping review was completed of the published and grey literature across 4 databases and 2 search engines to identify articles and aids/toolkits from 1990 to present. Toolkits were subsequently evaluated for purpose, evidence-base, implementation process, adaptability to family medicine, and outcome measures. Results: The search identified 17,751 articles. Screening resulted in 20 published articles and 11 toolkits. Most articles presented simple checklists to support greening clinic initiatives, 3 studies focussed on partial carbon footprint analyses, and 4 on educational initiatives. Toolkits ranged in sustainability topics and degree of depth covered, and adaptability and outcome measures. None of the resources identified have been formally evaluated for effectiveness. Conclusions: A range of aids exist to support greening of clinic operations; however, there is a significant gap in the literature for greening clinical care. Two toolkits were found to be comprehensive, one requiring tracking and reporting of sustainability initiatives. This scoping review provides a starting point for motivated family doctors and community clinics to initiate change and support more sustainable healthcare.
BACKGROUND ChatGPT is among the most popular large language models (LLMs), exhibiting proficiency in various standardized tests, including multiple-choice medical board examinations. However, its performance on otolaryngology–head and neck surgery (OHNS) certification examinations and open-ended medical board certification examinations has not been reported. OBJECTIVE We aimed to evaluate the performance of ChatGPT on OHNS board examinations and propose a novel method to assess an AI model’s performance on open-ended medical board examination questions. METHODS Twenty-one open-ended questions were adopted from the Royal College of Physicians and Surgeons of Canada’s sample examination to query ChatGPT on April 11, 2023, with and without prompts. A new model, named Concordance, Validity, Safety, Competency (CVSC), was developed to evaluate its performance. RESULTS In an open-ended question assessment, ChatGPT achieved a passing mark (an average of 75% across 3 trials) in the attempts and demonstrated higher accuracy with prompts. The model demonstrated high concordance (92.06%) and satisfactory validity. While demonstrating considerable consistency in regenerating answers, it often provided only partially correct responses. Notably, concerning features such as hallucinations and self-conflicting answers were observed. CONCLUSIONS ChatGPT achieved a passing score in the sample examination and demonstrated the potential to pass the OHNS certification examination of the Royal College of Physicians and Surgeons of Canada. Some concerns remain due to its hallucinations, which could pose risks to patient safety. Further adjustments are necessary to yield safer and more accurate answers for clinical implementation.