This study aimed to evaluate the performance of large language models (LLMs) and multimodal LLMs in interpreting the Breast Imaging Reporting and Data System (BI-RADS) categories and providing clinical management recommendations for breast radiology in text-based and visual questions. This cross-sectional observational study involved two steps. In the first step, we compared ten LLMs (namely ChatGPT 4o, ChatGPT 4, ChatGPT 3.5, Google Gemini 1.5 Pro, Google Gemini 1.0, Microsoft Copilot, Perplexity, Claude 3.5 Sonnet, Claude 3 Opus, and Claude 3 Opus 200K), general radiologists, and a breast radiologist using 100 text-based multiple-choice questions (MCQs) related to the BI-RADS Atlas 5^th edition. In the second step, we assessed the performance of five multimodal LLMs (ChatGPT 4o, ChatGPT 4V, Claude 3.5 Sonnet, Claude 3 Opus, and Google Gemini 1.5 Pro) in assigning BI-RADS categories and providing clinical management recommendations on 100 breast ultrasound images. The comparison of correct answers and accuracy by question types was analyzed using McNemar's and chi-squared tests. Management scores were analyzed using the Kruskal- Wallis and Wilcoxon tests. Claude 3.5 Sonnet achieved the highest accuracy in text-based MCQs (90%), followed by ChatGPT 4o (89%), outperforming all other LLMs and general radiologists (78% and 76%) (P < 0.05), except for the Claude 3 Opus models and the breast radiologist (82%) (P > 0.05). Lower-performing LLMs included Google Gemini 1.0 (61%) and ChatGPT 3.5 (60%). Performance across different categories of showed no significant variation among LLMs or radiologists (P > 0.05). For breast ultrasound images, Claude 3.5 Sonnet achieved 59% accuracy, significantly higher than other multimodal LLMs (P < 0.05). Management recommendations were evaluated using a 3-point Likert scale, with Claude 3.5 Sonnet scoring the highest (mean: 2.12 ± 0.97) (P < 0.05). Accuracy varied significantly across BI-RADS categories, except Claude 3 Opus (P < 0.05). Gemini 1.5 Pro failed to answer any BI-RADS 5 questions correctly. Similarly, ChatGPT 4V failed to answer any BI-RADS 1 questions correctly, making them the least accurate in these categories (P < 0.05). Although LLMs such as Claude 3.5 Sonnet and ChatGPT 4o show promise in text-based BI-RADS assessments, their limitations in visual diagnostics suggest they should be used cautiously and under radiologists' supervision to avoid misdiagnoses. This study demonstrates that while LLMs exhibit strong capabilities in text-based BI-RADS assessments, their visual diagnostic abilities are currently limited, necessitating further development and cautious application in clinical practice.

Evaluating the reference accuracy of large language models in radiology: a comparative study across subspecialties - Diagnostic and Interventional Radiology

Aim: The aim of this study was to assess the relationship between mammographic breast density and biochemical indices of calcium metabolism, as well as femoral neck bone mineral density (BMD) in women aged 40-80 years. Material and Methods: In this retrospective, single-center study, 598 women undergoing screening mammography between November 2020 and March 2025 were included. After applying exclusion criteria, 200 age-matched women were randomly selected and evenly stratified into four BI-RADS breast composition categories (A, B, C, and D), 50 women in each. Serum vitamin D3, parathyroid hormone (PTH), calcium, and femoral neck t-scores were retrieved from institutional records and compared between these four categories. Results: Vitamin D3 levels (p=0.021) and femoral neck t-scores (p=0.041) were found to be significantly different between four categories. Post hoc analysis demonstrated significantly different vitamin D3 concentrations in the Category D compared with categories A (p=0.021), B (p=0.032), and C (p=0.048). Femoral neck t-scores progressively decreased in Category D compared to categories A (p=0.013), B (p=0.025), and C (p=0.041). No significant differences were observed for PTH (p=0.589) or serum calcium (p=0.441). Conclusion: Higher mammographic breast density was associated with lower vitamin D3 concentrations and reduced femoral neck BMD. Breast composition may serve as a readily available imaging biomarker to identify women at increased risk of vitamin D deficiency and osteoporosis, and has potential integration into individualized risk stratification strategies.