Purpose: To investigate the feasibility of Retrieval-augmented Generation (RAG)-enhanced Large Language Models (LLMs) in answering questions about two German neurovascular guidelines. Methods: Four LLMs (GPT-4o-mini, Llama 3.1 405B Instruct Turbo, Mixtral 8 × 22B Instruct, and Claude 3.5 Sonnet) with RAG as well as GPT-4o-mini without RAG were evaluated for generating answers about two German neurovascular guidelines (“S3 Guideline for Diagnosis, Treatment, and Follow-up of Extracranial Carotid Stenosis” and “S2e Guideline for Acute Therapy of Ischemic Stroke”). The answers were classified as “correct”, “inaccurate”, or “incorrect” by two neurovascular experts in consensus. Additionally, retrieval performance of five retrieval strategies was analyzed on a synthetic dataset of 384 questions. Results: Claude Sonnet 3.5 achieved the highest answer correctness (70.6% correct, 10.6% wrong), followed by Llama 3.1 (64.7%, 15.3% wrong), GPT-4o-mini with RAG (57.6%, 15.3% wrong), and Mixtral (56.6%, 17.6% wrong). GPT-4o-mini without RAG performed significantly worse (20.0%, 32.9% wrong). Retrieval errors were the primary cause of incorrect answers (80%). For retrieval, BM25 achieved the highest accuracy (82.0%), outperforming vector-based methods like “BAAI/bge-m3” (78.4%). Conclusion: RAG significantly improves LLM accuracy for medical guideline question answering compared to the inherent knowledge of pretrained LLMs alone while still showing significant error rates. Improved accuracy and confidence metrics are needed for safer implementation in clinical routine. Additionally, our results demonstrate the strong performance of general LLMs in medical question answering for non-English languages, such as German, even without specific training.
Introduction: Ventriculoperitoneal shunts (VPS) are an essential part of the treatment of hydrocephalus, with numerous valve models available with different ways of indicating pressure levels. The model types often need to be identified on X‑rays to assess pressure levels using a matching template. Artificial intelligence (AI), in particular deep learning, is ideally suited to automate repetitive tasks such as identifying different VPS valve models. The aim of this work was to investigate whether AI, in particular deep learning, allows the identification of VPS models in cranial X‑rays. Methods: 959 cranial X‑rays of patients with a VPS were included and reviewed for image quality and complete visualization of VPS valves. The images included four VPS model types: Codman Hakim (n = 774, 81%), Codman Certas Plus (n = 117, 12%), Sophysa Sophy Mini SM8 (n = 35, 4%) and proGAV 2.0 (n = 33, 3%). A Convolutional Neural Network (CNN) was trained using stratified five-fold cross-validation to classify the four VPS model types in the dataset. A finetuned CNN pretrained on the ImageNet dataset as well as a model trained from scratch were compared. The averaged performance and uncertainty metrics were evaluated across the cross-validation splits. Results: The fine-tuned model identified VPS valve models with a mean accuracy of 0.98 ± 0.01, macro-averaged F1 score of 0.93 ± 0.04, a recall of 0.94 ± 0.03 and a precision of 0.95 ± 0.08 across the five cross-validation splits. Conclusion: Automatic classification of VPS valve models in skull X‑rays, using fully automatable preprocessing steps and a CNN, is feasible. This is an encouraging finding to further explore the possibility of automating VPS valve model identification and pressure level reading in skull X‑rays.
Introduction: This study investigates the influence of carotid artery elongation on neurovascular intervention and outcome in acute stroke treatments proposing an easily assessable imaging marker for carotid elongation. Methods: 118 patients who underwent mechanical thrombectomy for middle cerebral artery occlusions were included. The carotid elongation ratio (CER), center-line artery length to scan’s Z-axis, was measured on the affected side in CT-angiographies. Full and partial correlations of CER with periprocedural times, complications and outcome were computed. Multivariate logistic regression, including comorbidities, for prediction of dichotomized mRS outcome after 3 months was performed. Results: CER showed no significant correlation with recanalization success. Weak, outlier-driven correlation was found with recanalization time (p = 0.021, cor = 0.2). Weak correlations were found with improvement of NIHSS score at discharge and mRS score after 3 months (p = 0.023 and p = 0.031, each rho=-0.2). There was moderate correlation with NIHSS score at discharge (p = 0.001, rho = 0.3). Patients with favorable outcomes (mRS 0–2) exhibited lower CER (p = 0.012). Partial correlations of CER with favorable outcomes were observed after correcting for age, sex and cardiovascular risk factors (cor = 0.2, p = 0.048). Multivariate analysis (Nagelkerke’s R2 = 0.42) identified NIHSS score at admission, diabetes, hypertension and intervention time as significant factors for predicting outcome at 3 month, while CER showed the highest log Odd’s (2.97). Conclusion: Correlations between CER and clinical improvement suggest that carotid elongation might be a risk factor for poorer outcome without relevant effect on endovascular treatment and should not guide treatment decisions. Further studies should consider carotid elongation as an individual neurovascular risk factor, independent of hypertension.
Purpose: To evaluate the feasibility of a locally deployable large language model (LLM) system for automated MRI protocol selection addressing data privacy, annotation burden, and scalability limitations. Methods: This retrospective study included 598 German-language MRI order entries from three neuroradiology domains (brain, head/neck, spine) between June 2018 and January 2023. A radiologist labeled entries for 27 protocol classes based on institutional standard operating procedures (SOP). An SOP-grounded AI system using MedGemma 27B was developed to predict the MRI protocol from the order entry. The system was optimized using Stochastic Introspective Mini-Batch Ascent (SIMBA), a self-reflective prompt optimization algorithm, and compared with a hierarchical system that first classified the body region and then the MRI protocol. Data efficiency was evaluated using training subsets of 10–119 examples across 3 optimization runs per subset size. Results: The flat zero-shot model achieved 73.07% accuracy in the three-domain setting on the held-out dataset ( n = 479). In the hierarchical model, prompt optimization yielded 73.90% ± 3.24% with 30 labeled examples and a maximum of 74.46% ± 3.82% but did not outperform the flat approach. Prompt optimization benefited the hierarchical system, whereas the flat model already performed strongly without labeled examples for optimization. Performance on brain and head/neck cases remained broadly stable after expansion from 22 to 27 protocol classes, i.e., including spine. Conclusion: A locally deployable SOP-grounded open-weight LLM can support MRI protocol selection while preserving data privacy and needing minimal labeled data. In this dataset, hierarchical routing and prompt optimization did not improve overall performance over the flat baseline, although they altered optimization behavior and error profile. These findings support prospective evaluation in human-in-the-loop clinical workflows.