Purpose: This study aimed to compare the effectiveness of 3 artificial intelligence language models—GPT-3.5, GPT-4o, and Gemini, in delivering patient-centered information about thyroid eye disease (TED). The authors evaluated their performance based on the accuracy and comprehensiveness of their responses to common patient inquiries regarding TED. The study did not assess the repeatability of artificial intelligence responses, focusing on single-session evaluations per model. Methods: Five experienced oculoplastic surgeons assessed the responses generated by the artificial intelligence models to 12 key questions frequently asked by TED patients. These questions addressed TED pathophysiology, risk factors, clinical presentation, diagnostic testing, and treatment options. Each response was rated for correctness and reliability on a 7-point Likert scale, where 1 indicated incorrect or unreliable information and 7 indicated highly accurate and reliable information. Correctness referred to factual accuracy, while reliability assessed trustworthiness for patient use. The evaluations were anonymized, and the final scores were averaged across the surgeons to facilitate model comparisons. Results: GPT-3.5 emerged as the top performer, achieving an average correctness score of 5.75 and a reliability score of 5.68, excelling in delivering detailed information on complex topics such as TED treatment and surgical interventions. GPT-4o followed with scores of 5.32 for correctness and 5.25 for reliability, generally providing accurate but less detailed information. Gemini trailed with scores of 5.10 for correctness and 4.70 for reliability, often providing sufficient responses for simpler questions but lacking detail in complex areas like second-line immunosuppressive treatments. Statistical analysis using the Friedman test showed significant differences between models (p < 0.05) for key topics, with GPT-3.5 consistently leading. Conclusions: GPT-3.5 was the most effective model for delivering reliable and comprehensive patient information, particularly for complex treatment and surgical topics. GPT-4o provided reliable general information but lacked the necessary depth for specialized topics, while Gemini was suitable for addressing basic patient inquiries but insufficient for detailed medical information. This study highlights the role of artificial intelligence in patient education, suggesting that models like GPT-3.5 can be valuable tools for clinicians in enhancing patient understanding of TED.

Background: This study aimed to evaluate the performance of Large Language Models (LLMs) in answering ophthalmology residency exam questions, comparing their accuracy to human residents, and determine current models’ potential for use in medical education and clinical work. OpenAI’s latest model, GPT-4o, is of particular interest due to being significantly faster than its predecessor; however, it is yet unknown how it compares in terms of knowledge and accuracy. Therefore, our objective was to assess GPT-4o’s performance in various metrics against other LLMs (GPT-4 and Gemini Advanced) and the human group, specifically in ophthalmology, a specialized topic with limited training data, to explore how well these artificial intelligence (AI) models handle subspecialty-specific content. Methods: A dataset of 600 questions, derived from Israeli ophthalmology residency exams conducted between 2020 and 2023, and based on content from the Basic and Clinical Science Course (BCSC) textbooks, was utilized, covering 12 ophthalmologic subspecialties, in text-only (379 questions) and image-based (221 questions) formats. GPT-4o, GPT-4, and Gemini Advanced were tested on this dataset to measure overall accuracy, with additional analyses by exam year, subspecialty topics, and image-processing capabilities based on categorization of images according to their content, including labeled images, pathology, fundus photographs, and complex visuals like graphs and multi-image questions. Chi-squared tests were used to determine statistical significance across different metrics. Results: GPT-4o achieved the highest overall accuracy (74.2%), significantly outperforming GPT-4 (61.9%) and Gemini Advanced (65.8%). While variations in year-to-year exam performance were statistically insignificant, GPT-4o’s accuracy was comparable to human residents, even exceeding the national average in 2021. In text-only questions, GPT-4o demonstrated superior performance, with an accuracy of 82.3% compared to GPT-4’s 73.8% and Gemini Advanced’s 72.3%. Image-based performance was also higher for GPT-4o (60.5%), surpassing GPT-4 (41.8%) and Gemini Advanced (39.3%), surpassing them in every metric, and particularly in labeled images, pathology, and fundus images. However, the model’s ability to process complex visuals, such as abstract drawings and multi-image prompts, remained limited. Performance across subspecialty topics also varied, with GPT-4o still excelling in most categories; statistical significance was lower in topics with fewer questions, like uveitis and refractive surgery, making for less notable differences when compared to the overall comparison. Conclusions: The study anticipates further advancements in this field, with improvements in all metrics expected as the technology is refined to meet individual needs in the medical industry, particularly in ophthalmology. GPT-4o’s improvements in clinical reasoning and image analysis mark a substantial leap in performance. The model approached human-level performance in our testing, suggesting increased potential for integration into ophthalmology clinical settings compared to previous generations. Future research should focus on assessing and evaluating image interpretation capabilities, enhanced accuracy, and addressing ethical considerations to ensure safe, effective AI integration into clinical practice.

This study aimed to evaluate the performance of seven advanced AI Large Language Models (LLMs)—ChatGPT 4o, ChatGPT O3 Mini, ChatGPT O1, DeepSeek V3, DeepSeek R1, Gemini 2.0 Flash, and Grok-3—in answering multiple-choice questions (MCQs) in optics and refractive surgery, to assess their role in medical education for residents. The AI models were tested using 134 publicly available MCQs from national ophthalmology certification exams, categorized by the need to perform calculations, the relevant subspecialty, and the use of images. Accuracy was analyzed and compared statistically. ChatGPT O1 achieved the highest overall accuracy (83.5%), excelling in complex optical calculations (84.1%) and optics questions (82.4%). DeepSeek V3 displayed superior accuracy in refractive surgery-related questions (89.7%), followed by ChatGPT O3 Mini (88.4%). ChatGPT O3 Mini significantly outperformed others in image analysis, with 88.2% accuracy. Moreover, ChatGPT O1 demonstrated comparable accuracy rates for both calculated and non-calculated questions (84.1% vs. 83.3%). This is in stark contrast to other models, which exhibited significant discrepancies in accuracy for calculated and non-calculated questions. The findings highlight the ability of LLMs to achieve high accuracy in ophthalmology MCQs, particularly in complex optical calculations and visual items. These results suggest potential applications in exam preparation and medical training contexts, while underscoring the need for future studies designed to directly evaluate their role and impact in medical education. The findings highlight the significant potential of AI models in ophthalmology education, particularly in performing complex optical calculations and visual stem questions. Future studies should utilize larger, multilingual datasets to confirm and extend these preliminary findings.