BACKGROUND Large language models (LLMs) have demonstrated remarkable performance on medical licensing examinations, particularly in English-language contexts. However, little research has systematically evaluated their capabilities in non-English languages, especially morphologically complex languages like Turkish. Furthermore, the risk of data contamination in studies using publicly available exam questions remains inadequately addressed, and comparative analyses across multiple state-of-the-art models from both closed and open-source ecosystems are lacking in the ophthalmology literature. OBJECTIVE This study aimed to comprehensively evaluate the performance of seven leading LLMs from the 2024-2025 period on Turkish Medical Specialty Examination (TUS) ophthalmology questions, analyzing factors such as question type (case-based vs. knowledge-based), chronological question age (2015-2019 vs. 2020-2024), and clinical subspecialty, while critically assessing the risk of data contamination and the technology's realistic potential in medical education. METHODS A set of 210 TUS ophthalmology questions (2015-2024) was presented to seven LLMs (OpenAI o1-preview, Claude 4.5 Sonnet, GPT-4o, Gemini 2.5 Pro, DeepSeek-R1, Llama 3 70B, Command R+) using a standardized Turkish prompt template between November-December 2024. All responses underwent blinded, expert qualitative coding by two experienced ophthalmologists for accuracy, explanation quality (1-3 Likert), confidence level (1-3 Likert), and error type (conceptual error, hallucination, inattentiveness). Statistical analyses included chi-square tests for inter-model comparisons, independent samples t-tests for closed versus open-source model groups, odds ratio calculations, and Spearman correlation for confidence-accuracy relationships. RESULTS OpenAI o1-preview achieved the highest overall accuracy (95.2%, 200/210), followed by Claude 4.5 Sonnet (93.8%) and GPT-4o (91.9%). Notably, all three models demonstrated 100% accuracy on clinical case-based questions (n=105). A significant performance gap existed between closed-source (average 91.0%) and open-source models (average 66.1%) (p<0.001, OR=5.14, 95% CI: 2.58-10.22). Chronological analysis revealed substantial performance declines on new questions (2020-2024) for models with older knowledge cutoff dates: Command R+ (-22.9%), DeepSeek-R1 (-14.3%), and Gemini 2.5 Pro (-8.6%), suggesting potential data contamination. Qualitative analysis showed error concentration (over 65%) in complex clinical areas including glaucoma (28.7%) and retina (23.2%). Hallucination rates were substantially higher in open-source models (42.5% vs. 12.8% in closed-source models). A weak, non-significant correlation was found between model confidence and answer accuracy (Spearman ρ=0.21, p=0.09), indicating a "confident incorrectness" paradox. CONCLUSIONS Advanced closed-source LLMs demonstrate near-expert-level textual performance on Turkish TUS ophthalmology questions, highlighting their potential as complementary educational tools for exam preparation and case-based learning. However, significant performance variations between model types, vulnerabilities in nuanced clinical judgment (particularly in glaucoma and retina subspecialties), suspected data contamination in older models, high hallucination rates in open-source models, and the confidence-accuracy disconnect necessitate a critical approach with continuous human oversight. This text-based performance must not be equated with real-world, visually-intensive clinical diagnostic competence. Implementation requires rigorous validation protocols, AI literacy education for clinicians and students, and clear ethical frameworks, with ultimate authority remaining with the human expert. Future research should evaluate multimodal LLMs' visual diagnostic capabilities and conduct fine-tuning studies with Turkish ophthalmology datasets.