Background/Objectives: Accurate differentiation of benign melanocytic nevi from invasive melanoma in dermato-oncology directly informs biopsy decisions and oncological management. Vision–language models (VLMs) are increasingly explored for image-based skin cancer assessment, but their diagnostic reliability and robustness to adversarial input manipulation remain insufficiently characterized. We evaluated three contemporary VLMs for diagnostic performance and susceptibility to single-word adversarial input manipulation (prompt injection) on dermoscopic images of histopathologically confirmed lesions. Methods: Fifty-two dermoscopic images (26 benign melanocytic nevi, 26 invasive melanomas) were analyzed using Claude Opus 4.7, Gemini 3.1 Pro, and GPT-5.4 under four conditions: an unmodified baseline and three adversarial conditions with a single opposite-of-ground-truth label embedded as a visual overlay, in image metadata, or both. Three independent rounds per image × model × condition yielded 1872 classifications across 52 lesions (independent diagnostic units) and 16,848 structured-output observations in total. Results: Baseline diagnostic accuracy ranged from 58.3% to 62.2%, with asymmetric sensitivity and specificity, including a pronounced benign-labeling bias in one model that missed 22 of 26 invasive melanomas. All adversarial conditions reduced accuracy to near-zero levels (0.0–1.9%; all p < 10−7 after Bonferroni correction). Repeated queries produced identical incorrect outputs in 98–100% of cases (Fleiss’ κ 0.97–1.00). Non-diagnostic outputs remained largely unchanged, and self-reported confidence did not decrease. Conclusions: Contemporary VLMs show limited baseline performance and marked vulnerability to minimal adversarial input in dermoscopic skin cancer assessment. The failure selectively alters the malignancy decision while preserving surrounding outputs and confidence, indicating that, within the conditions evaluated here, these systems do not currently appear suitable for unsupervised clinical use in dermato-oncology in the absence of input-integrity safeguards and qualified human oversight.
Background: Burn extent guides triage, transfer and fluid resuscitation, yet its clinical estimation is imprecise and observer-dependent. Multimodal large language models (MLLMs) process clinical photographs without task-specific training, but their error has rarely been separated into systematic and random components or their performance across skin tones characterized. Methods: Three state-of-the-art MLLMs (Gemini 3.1 Pro, GPT-5.6 Sol, and Fable 5) each assessed 153 burn photographs five times under an identical prompt. The tasks were as follows: burned proportion of the imaged field, against an expert-guided pixel-wise segmentation (tolerance ± 10 percentage points, pp); burned percentage of total body surface area (TBSA), against physician consensus (±2 pp); and binary Fitzpatrick skin tone (FST; light I–III versus dark IV–VI). The first of these was the primary endpoint. Results: The primary endpoint was in the range of 32.5–70.2%, with TBSA at 69.5–77.5%. All models compressed the estimation range (slopes 0.58–0.76, intercepts +10.0 to +22.2 pp); one multiplicative constant per model brought errors differing more than twofold into a 1.4 pp range. Across repeated queries, the median within-image range was 5.0–25.0 pp; averaging the five answers reduced error by only 0.24–2.22 pp. FST accuracy was 83.8–91.9% against a majority-class baseline of 81.0%. Conclusions: Averaging repeated answers removes only the smaller, random component; the larger, systematic one persists and requires calibration against reference data before clinical use can be considered.
Background: Burn photography is a stringent testbed for visual artificial intelligence (AI) in medical imaging, since it permits quantitative and qualitative clinically relevant parameters to be elicited. It is therefore a useful proxy for current vision AI. Reducing a photograph to a semantic segmentation removes color, texture and anatomy while providing pixel-exact boundaries. How spatial assessment responds, and how far instance-level structure can be recovered from a map that does not encode it, is unknown. Methods: Three state-of-the-art multimodal large language models (MLLMs) each assessed 153 burn photographs five times under three input conditions: the photograph alone (IMAGE), its segmentation mask alone (MASK), or both combined (BOTH). Four items were elicited per image: burn presence, the burned body region, per-cell classification of a 3 × 3 grid, and the number of separate burns, a surrogate for instance-level recognition. Each model was compared across conditions, paired on the same images. Results: Body-region accuracy was 97.1–100.0% under IMAGE, 47.5–54.0% under MASK and 50.2–99.9% under BOTH. Grid-cell accuracy was 77.7–82.3%, 77.9–96.3% and 90.9–98.4%. On the burn count, exact-match was 60.5–66.3%, 61.3–96.6% and 84.7–97.4%, and no model significantly beat the trivial classifier under IMAGE, two of three did under MASK and all three under BOTH. Across 18 paired comparisons of BOTH against a single channel, BOTH was better in 11, worse in three and indistinguishable in four. Conclusions: Combining the two inputs improved spatial assessment in most comparisons and degraded it in others, consistently across neither models nor tasks. Additional visual information therefore does not deterministically improve model performance, and disruption remains possible.
Background: Large language models (LLMs) increasingly pass medical board examinations and aid clinical knowledge retrieval and decision support; validating their specialist knowledge is a prerequisite for safe use. Two limitations weaken existing evidence: many studies reuse public questions that may be in training data, and most evaluations use only English-language examinations. Plastic surgery serves here as a representative, highly specialized subfield of medicine. Methods: We used the non-public board examination of the Turkish Society of Plastic, Reconstructive, and Aesthetic Surgeons (TSPRAS): 100 single-best-answer (SBA) items in official parallel Turkish and English versions. Six contemporary LLMs from three developers, in matched free and paid tiers, completed six runs per language (7200 responses) under consistent English instruction. Accuracy, inter-run reliability, and cross-language and tier differences were analyzed. Results: All models exceeded the 60% passing threshold (71.5–86.4% aggregate), with almost perfect inter-run agreement (Cohen’s κ ≥ 0.843). Question language produced no significant difference after correction, and item difficulty correlated strongly across languages (Pearson r = 0.928); divergent items reflected content, not language. Paid variants offered only incremental advantages. Conclusions: Current LLMs showed no significant accuracy difference between the Turkish and English examination versions under a constant English prompt, supporting supervised study-aid use on this MCQ format.