Large language models (LLMs) are increasingly used in professional examinations, but their relative performance on dental board-style questions remains unclear. This study compared two reasoning-optimized models, GPT-o3 and GPT-5T, with a general-purpose multimodal model, GPT-4o, using 399 Japanese dental board-style multiple-choice questions from 2018 to 2022. All questions were presented in Japanese, and items originally accompanied by charts, photographs, or other figures were analyzed separately from items without visual materials. Accuracy and item-level agreement were assessed using pairwise McNemar tests, stratified analyses according to the original presence of visual materials, the Breslow–Day test for homogeneity of odds ratios, and two-proportion z-tests. GPT-5T achieved the highest overall accuracy (294/399, 73.7%), followed by GPT-o3 (257/399, 64.4%) and GPT-4o (255/399, 63.9%). Pairwise McNemar tests showed that GPT-5T outperformed both GPT-4o (Holm-adjusted p = 0.00098) and GPT-o3 (Holm-adjusted p = 0.00072), whereas GPT-o3 and GPT-4o did not differ significantly (Holm-adjusted p = 0.920). Accuracy was lower for questions originally containing visual materials than for questions without such materials across all three models (GPT-4o: 49.7% vs. 72.2%; GPT-o3: 55.1% vs. 69.8%; GPT-5T: 59.9% vs. 81.8%). The advantage of GPT-5T was more evident in questions without visual materials, and heterogeneity across question formats was observed for GPT-5T versus GPT-o3. GPT-5T showed the strongest performance in this dataset. Questions originally containing visual materials were associated with lower accuracy across all models. Because the comparison was based on distinct item groups rather than experimentally manipulated visual conditions, this result should be interpreted as a difference across question formats and may also reflect differences in item composition and difficulty between the two groups.
1 January 2020 in The Journal of the Kyushu Dental Society
Large language models (LLMs) show varying accuracy on professional examinations, particularly for questions involving visual materials. This study compared GPT-4o, o3, GPT-5 Thinking, and Gemini 3 on 400 Japanese National Dental Technician Examination questions from 2018–2022 and investigated whether between-model differences varied by question format. The dataset comprised 252 text-only questions and 148 questions accompanied by visual materials. Accuracy was compared using Cochran’s Q test and Holm-adjusted exact McNemar tests. A question-level mixed-effects logistic regression assessed model-by-item-type interactions while accounting for examination year. Overall accuracy was 63.8% for GPT-4o, 70.8% for o3, 74.8% for GPT-5 Thinking, and 80.5% for Gemini 3 (Cochran’s Q, p < 0.001). Gemini 3 significantly outperformed the other models, whereas o3 and GPT-5 Thinking did not differ significantly after adjust-ment. All models showed lower accuracy on visual-material questions than on text-only questions. The overall model-by-item-type interaction was significant (likelihood-ratio χ² = 11.756, p = 0.008), indicating that between-model differences varied by question format. The relative advantages of GPT-5 Thinking and Gemini 3 over GPT-4o were at-tenuated for visual-material questions. These findings highlight format-dependent dif-ferences in LLM performance and the limitations of aggregate accuracy as a bench-marking metric.