Background/aims: This study assesses the proficiency of Generative Pre-trained Transformer (GPT)-4 in answering questions about complex clinical ophthalmology cases.Methods: We tested GPT-4 on 422Journal of the American Medical AssociationOphthalmology Clinical Challenges, and prompted the model to determine the diagnosis (open-ended question) and identify the next-step (multiple-choice question). We generated responses using two zero-shot prompting strategies, including zero-shot plan-and-solve+ (PS+), to improve the reasoning of the model. We compared the best-performing model to human graders in a benchmarking effort.Results: Using PS+ prompting, GPT-4 achieved mean accuracies of 48.0% (95% CI (43.1% to 52.9%)) and 63.0% (95% CI (58.2% to 67.6%)) in diagnosis and next step, respectively. Next-step accuracy did not significantly differ by subspecialty (p=0.44). However, diagnostic accuracy in pathology and tumours was significantly higher than in uveitis (p=0.027). When the diagnosis was accurate, 75.2% (95% CI (68.6% to 80.9%)) of the next steps were correct. Conversely, when the diagnosis was incorrect, 50.2% (95% CI (43.8% to 56.6%)) of the next steps were accurate. The next step was three times more likely to be accurate when the initial diagnosis was correct (p<0.001). No significant differences were observed in diagnostic accuracy and decision-making between board-certified ophthalmologists and GPT-4. Among trainees, senior residents outperformed GPT-4 in diagnostic accuracy (p≤0.001 and 0.049) and in accuracy of next step (p=0.002 and 0.020).Conclusion: Improved prompting enhances GPT-4’s performance in complex clinical situations, although it does not surpass ophthalmology trainees in our context. Specialised large language models hold promise for future assistance in medical decision-making and diagnosis.

Background/aims: To compare the performance and cost-effectiveness of DeepSeek-R1 with OpenAI o1 in diagnosing and managing ophthalmology clinical cases. Methods: In this cross-sectional study, a total of 300 clinical cases spanning 10 ophthalmology subspecialties were collected from StatPearls, each with a multiple-choice question on diagnosis or management. DeepSeek-R1 was accessed through its public chat interface, while OpenAI o1 was queried via its Application Programming Interface with a standardised temperature of 0.3. Both models were prompted using plan-and-solve+. Performance was calculated as the proportion of correct answers. McNemar’s test was employed to compare the two models’ performance on paired data. Intermodel agreement for correct diagnoses was evaluated via Cohen’s kappa. Token-based cost analyses were performed to estimate the comparative expenditures of running each model at scale, including input prompts and model-generated output. Results: DeepSeek-R1 and OpenAI o1 achieved an identical overall performance of 82.0% (n=246/300; 95% CI: 77.3 to 85.9). Subspecialty-specific analysis revealed numerical variation in performance, though none of these comparisons reached statistical significance (p>0.05). Agreement in performance between the models was moderate overall (κ=0.503, p<0.001), with substantial agreement in refractive management/intervention (κ=0.698, p<0.001) and moderate agreement in retina/vitreous (κ=0.561, p<0.001) and ocular pathology/oncology (κ=0.495, p<0.01) cases. Cost analysis indicated an approximately 15-fold reduction in per-query, token-related expenses when using DeepSeek-R1 vs OpenAI o1 for the same workload. Conclusions: DeepSeek-R1 shows strong diagnostic and management performance comparable to OpenAI o1 across ophthalmic subspecialties, while significantly reducing costs. These results support its use as a cost-effective, open-weight alternative to proprietary models.

Objectives: The integration of multimodal capabilities into GPT-4 represents a transformative leap for artificial intelligence in ophthalmology, yet its utility in scenarios requiring advanced reasoning remains underexplored. This study evaluates GPT-4’s multimodal performance on open-ended diagnostic and next-step reasoning tasks in complex ophthalmology cases, comparing it against human expertise. Methods: GPT-4 was assessed across three study arms: (1) text-based case details with figure descriptions, (2) cases with text and accompanying ophthalmic figures, and (3) cases with figures only (no figure descriptions). We compared GPT-4’s diagnostic and next-step accuracy across arms and benchmarked its performance against three board-certified ophthalmologists. Results: GPT-4 achieved 38.4% (95% CI [33.9%, 43.1%]) diagnostic accuracy and 57.8% (95% CI [52.8%, 62.2%]) next-step accuracy when prompted with figures without descriptions. Diagnostic accuracy declined significantly compared to text-only prompts (p = 0.007), though the next-step performance was similar (p = 0.140). Adding figure descriptions restored diagnostic accuracy (49.3%) to near parity with text-only prompts (p = 0.684). Using figures without descriptions, GPT-4’s diagnostic accuracy was comparable to two ophthalmologists (p = 0.30, p = 0.41) but fell short of the highest-performing ophthalmologist (p = 0.0004). For next-step accuracy, GPT-4 was similar to one ophthalmologist (p = 0.22) but underperformed relative to the other two (p = 0.0015, p = 0.0017). Conclusions: GPT-4’s diagnostic performance diminishes when relying solely on ophthalmic images without textual context, highlighting limitations in its current multimodal capabilities. Despite this, GPT-4 demonstrated comparable performance to at least one ophthalmologist on both diagnostic and next-step reasoning tasks, emphasizing its potential as an assistive tool. Future research should refine multimodal prompts and explore iterative or sequential prompting strategies to optimize AI-driven interpretation of complex ophthalmic datasets.

Code-free deep learning (CFDL) allows clinicians with no coding experience to build their own artificial intelligence models. This study assesses the performance of CFDL in glaucoma detection from fundus images in comparison to expert-designed models. Deep learning model development, testing, and validation. A total of 101 442 labeled fundus images from the Rotterdam EyePACS Artificial Intelligence for Robust Glaucoma Screening (AIROGS) dataset were included. Ophthalmology trainees without coding experience designed a CFDL binary model using the Rotterdam EyePACS AIROGS dataset of fundus images (101 442 labeled images) to differentiate glaucoma from normal optic nerves. We compared our results with bespoke models from the literature. We then proceeded to externally validate our model using 2 datasets, the Retinal Fundus Glaucoma Challenge (REFUGE) and the Glaucoma grading from Multi-Modality imAges (GAMMA) at 0.1, 0.3, and 0.5 confidence thresholds. Area under the precision-recall curve (AuPRC), sensitivity at 95% specificity (SE@95SP), accuracy, area under the receiver operating curve (AUC), and positive predictive value (PPV). The CFDL model showed high performance metrics that were comparable to the bespoke deep learning models. Our single-label classification model had an AuPRC of 0.988, an SE@95SP of 95%, and an accuracy of 91% (compared with 85% SE@95SP for the top bespoke models). Using the REFUGE dataset for external validation, our model had an SE@95SP, AUC, PPV, and accuracy of 83%, 0.960%, 73% to 94%, and 95% to 98%, respectively, at the 0.1, 0.3, and 0.5 confidence threshold cutoffs. Using the GAMMA dataset for external validation at the same confidence threshold cutoffs, our model had an SE@95SP, AUC, PPV, and accuracy of 98%, 0.994%, 94% to 96%, and 94% to 97%, respectively. The capacity of CFDL models to perform glaucoma screening using fundus images presents a compelling proof of concept, empowering clinicians to explore innovative model designs for broad glaucoma screening in the near future. Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.

Purpose: To compare the performance and cost-effectiveness of DeepSeek-R1 with OpenAI o1 in diagnosing and managing ophthalmology clinical cases. Study Design: Cross-sectional evaluation. Methods: A total of 300 clinical cases spanning 10 different ophthalmology subspecialties were collected from StatPearls. Each case presented a multiple-choice question regarding the diagnosis or management of the clinical case. DeepSeek-R1 was accessed through its public chat-based interface, while OpenAI o1 was queried via an Application Program Interface (API) with a standardized temperature setting of 0.3. Both models were prompted using the Plan-and-Solve+ (PS+) prompt engineering method, instructing them to answer multiple choice questions for each case. Performance was calculated as the proportion of correctly answered multiple choice questions. McNemar’s test was employed to compare the two models’ performance on paired data. Inter-model agreement for correct diagnoses was evaluated via Cohen’s kappa. A token-based cost analysis was performed to estimate the comparative expenditures of running each model at scale, accounting for both input prompts and model-generated output. Results: DeepSeek-R1 and OpenAI o1 both achieved an identical overall performance of 82.0% (n=246/300; 95% CI: 77.3-85.9). Subspeciality-specific analysis revealed numerical variation in performance, though none of these comparisons reached statistical significance (p>0.05). Agreement in performance between the models was moderate overall (κ=0.503, p<0.001), with substantial agreement in Refractive Management/Intervention (κ=0.698, p<0.001) and moderate agreement in Retina/Vitreous (κ=0.561, p<0.001) and Ocular Pathology/Oncology (κ=0.495, p<0.01) cases. Cost analysis indicated an approximately 15-fold reduction in per-query, token-related expenses when using DeepSeek-R1 compared with OpenAI o1 for the same workload. Conclusions: DeepSeek-R1 demonstrates robust diagnostic reasoning and management decision-making capabilities, performing comparably to OpenAI o1 across a range of ophthalmic subspecialty cases, while also offering a substantial reduction in usage costs. These findings highlight the feasibility of utilizing open-weight, reinforcement learning-augmented LLMs as an accessible, cost-effective alternative to proprietary models.

Objectives: To develop a proof-of-concept machine learning (ML) neural network model to predict post-operative visual outcomes in children with congenital cataracts undergoing intraocular lens (IOL) implantation, thereby guiding the optimal timing for IOL insertion. Determining the ideal timing and predicting outcomes for IOL implantation in children remains clinically complex due to variability in eye development and measurement accuracy. Methods: Retrospective analysis using a publicly available dataset from 110 children diagnosed with congenital cataracts, who underwent IOL implantation at the Eye and ENT Hospital of Fudan University. A neural network model with a hidden layer of 10 nodes was developed in MATLAB 2024a using the scaled conjugate gradient algorithm. Input variables included demographic and clinical features; the target was achieving visual acuity greater than 20/40. Performance metrics were evaluated using cross-entropy loss, sensitivity, specificity, and accuracy. Results: Training completed after 14 epochs with the test set reaching the highest performance metrics: 88.2% accuracy, 88.9% sensitivity, and 87.5% specificity. ROC curve analysis showed AUC values of 0.942 (training), 0.920 (validation), 0.885 (test), and 0.917 (overall). Conclusions: The neural network effectively predicted post-operative visual outcomes, offering potential clinical utility in guiding IOL implantation decisions. Despite limitations in dataset diversity, this study lays the foundation for future development of personalized strategies in pediatric cataract care.