Background/Objectives: Multimodal large language models (MLLMs) offer potential for automated fracture detection, yet their diagnostic stability under repeated inference remains underexplored. This study evaluates the diagnostic accuracy, stability, and intra-model consistency of four MLLMs in detecting hand fractures on plain radiographs. Methods: In total, images of hand radiographs of 65 adult patients with confirmed hand fractures (30 phalangeal, 30 metacarpal, 5 scaphoid) were evaluated by four models: GPT-5 Pro, Gemini 2.5 Pro, Claude Sonnet 4.5, and Mistral Medium 3.1. Each image was independently analyzed five times per model using identical zero-shot prompts (1300 total inferences). Diagnostic accuracy, inter-run reliability (Fleiss’ κ), case-level agreement profiles, subgroup performance, and exploratory demographic inference (age, sex) were assessed. Results: GPT-5 Pro achieved the highest accuracy (64.3%) and consistency (κ = 0.71), followed by Gemini 2.5 Pro (56.9%, κ = 0.57). Mistral Medium 3.1 exhibited high agreement (κ = 0.88) despite low accuracy (38.5%), indicating systematic error (“confident hallucination”). Claude Sonnet 4.5 showed low accuracy (33.8%) and consistency (κ = 0.33), reflecting instability. While phalangeal fractures were reliably detected by top models, scaphoid fractures remained challenging. Demographic analysis revealed poor capabilities, with age estimation errors exceeding 12 years and sex prediction accuracy near random chance. Conclusions: Diagnostic accuracy and consistency are distinct performance dimensions; high intra-model agreement does not imply correctness. While GPT-5 Pro demonstrated the most favorable balance of accuracy and stability, other models exhibited critical failure modes ranging from systematic bias to random instability. At present, MLLMs should be regarded as experimental diagnostic reasoning systems rather than reliable standalone tools for clinical fracture detection.
Background/Objectives: Accurate differentiation of benign melanocytic nevi from invasive melanoma in dermato-oncology directly informs biopsy decisions and oncological management. Vision–language models (VLMs) are increasingly explored for image-based skin cancer assessment, but their diagnostic reliability and robustness to adversarial input manipulation remain insufficiently characterized. We evaluated three contemporary VLMs for diagnostic performance and susceptibility to single-word adversarial input manipulation (prompt injection) on dermoscopic images of histopathologically confirmed lesions. Methods: Fifty-two dermoscopic images (26 benign melanocytic nevi, 26 invasive melanomas) were analyzed using Claude Opus 4.7, Gemini 3.1 Pro, and GPT-5.4 under four conditions: an unmodified baseline and three adversarial conditions with a single opposite-of-ground-truth label embedded as a visual overlay, in image metadata, or both. Three independent rounds per image × model × condition yielded 1872 classifications across 52 lesions (independent diagnostic units) and 16,848 structured-output observations in total. Results: Baseline diagnostic accuracy ranged from 58.3% to 62.2%, with asymmetric sensitivity and specificity, including a pronounced benign-labeling bias in one model that missed 22 of 26 invasive melanomas. All adversarial conditions reduced accuracy to near-zero levels (0.0–1.9%; all p < 10−7 after Bonferroni correction). Repeated queries produced identical incorrect outputs in 98–100% of cases (Fleiss’ κ 0.97–1.00). Non-diagnostic outputs remained largely unchanged, and self-reported confidence did not decrease. Conclusions: Contemporary VLMs show limited baseline performance and marked vulnerability to minimal adversarial input in dermoscopic skin cancer assessment. The failure selectively alters the malignancy decision while preserving surrounding outputs and confidence, indicating that, within the conditions evaluated here, these systems do not currently appear suitable for unsupervised clinical use in dermato-oncology in the absence of input-integrity safeguards and qualified human oversight.
3 June 2026 in Journal of Plastic, Reconstructive & Aesthetic Surgery
Accurate assessment of burn depth and total body surface area (TBSA) is critical for clinical decision-making; however, it remains subjective and prone to interobserver variability. Multimodal large language models (MLLMs) are increasingly encountered in clinical contexts, but whether these systems can reliably assess burn images remains unclear. We evaluated four MLLMs (GPT-5.4 Pro, Grok 4.1, Gemini 3.1 Pro, and Claude Opus 4.6) on 50 clinical burn photographs using a repeated-inference design with five independent runs per model. Burn depth classification was assessed in numeric and text-based formats, alongside ordinal TBSA estimation. Performance varied across the models, with burn depth accuracy ranging from 34.0 ± 6.5% to 76.4 ± 6.8% and TBSA accuracy from 32.8 ± 9.4% to 68.4 ± 3.3%. Inter-run reliability (Fleiss' κ) ranged from slight (κ = 0.171) to almost perfect (κ = 0.916), demonstrating response variability not captured by single-query evaluations. Notably, no model combined high accuracy and high reliability, indicating a dissociation between performance and consistency. All models showed a tendency toward overestimation of burn depth, including assignment of fourth-degree burns despite their absence in the dataset. Error direction analysis revealed model-specific and task-dependent biases, including opposing patterns within the same model. Internal consistency between numeric and text classifications was near-perfect (99.6-100%), indicating format-invariant but systematically biased outputs. These findings demonstrate that MLLM performance is characterized by stochastic response instability invisible to single-query evaluations. Such inconsistency for identical inputs represents a fundamental limitation for workflows requiring consistent outputs across repeated evaluations.
Background: Large language models (LLMs) have demonstrated strong performance on general medical examinations. Whether this performance translates to highly specialized, subspecialty-level board examinations remains unclear. This study evaluates the accuracy and inter-run stability of contemporary LLMs using authentic European Board of Nuclear Medicine (EBNM) Fellowship Examination material. Methods: Ten LLMs (five proprietary, five open-source) completed 50 EBNM multiple-choice questions across five independent zero-shot runs, resulting in 2500 total inferences. Accuracy was calculated per model across runs. Inter-run reliability was assessed using pairwise Cohen’s kappa coefficients. Pairwise model differences were analyzed using McNemar’s test with Bonferroni correction (α = 0.0011). Results: Mean accuracy ranged from 53.6% to 100.0%, with all models exceeding an illustrative 50% pass threshold. Inter-run reliability varied substantially (κ = 0.370–1.000; mean κ = 0.716). High accuracy did not consistently correspond to high reproducibility. Gemini 2.5 Pro achieved high accuracy (93.6%) but showed the lowest reliability (κ = 0.370), whereas DeepSeek V3.2 demonstrated perfect accuracy and agreement across all runs. No significant correlation between accuracy and reliability was observed (Spearman ρ = 0.394, p = 0.26). Conclusions: LLMs demonstrate strong but heterogeneous performance on high-stakes medical knowledge assessments. Differences in reproducibility highlight the need for multi-run evaluation when considering LLMs for educational or clinical knowledge-support applications and for continued validation using non-disclosed examination material.
Background and Objectives: There is a global rise of public interest in integrative medicine. The principles of integrative medicine combining conventional medicine with evidence-based complementary therapies have been implemented in many medical areas, including plastic surgery, to improve patient’s outcome. The aim of the present study was to systematically analyze the application and use of additional non-pharmacological interventions (NPIs) of patients of a German department of plastic surgery. Materials and Methods: The present real-world data study utilized data from the Network Oncology registry between 2016 and 2021. Patients included in this study were at the age of 18 or above, stayed at the department of plastic surgery and received at least one plastic surgical procedure. Adjusted multivariable logistic regression analyses were performed to detect associations between the acceptance of NPIs and predicting factors such as age, gender, year of admission, or length of hospital stay. Results: In total, 265 patients were enrolled in the study between January 2016 and December 2021 with a median age of 65 years (IQR: 52–80) and a male/female ratio of 0.77. Most of the patients received reconstructive surgery (90.19%), followed by hand surgery (5.68%) and aesthetic surgery (2.64%). In total, 42.5% of the enrolled patients accepted and applied NPIs. Physiotherapy, rhythmical embrocations, and compresses were the most often administered NPIs. Conclusions: This exploratory analysis provides a descriptive overview of the application and acceptance of NPIs in plastic surgery patients within a German integrative care setting. While NPIs appear to be well accepted by a subset of patients, further prospective studies are needed to evaluate their impact on clinical outcomes such as postoperative recovery, pain management, patient-reported quality of life, and overall satisfaction with care.