Accurate assessment of colorectal lesion morphology during colonoscopy is essential for guiding treatment and estimating cancer risk. The Paris classification is widely adopted for this purpose but suffers from substantial inter-observer variability, while Vision Transformers (ViTs) can base their decisions on diffuse, off-lesion attention patterns that are hard to interpret. This study investigates whether directly supervising ViT attention maps with expert lesion annotations can concurrently improve Paris classification performance and model explainability. We propose a Lesion-Focused Attention Loss (LLFA), an attention-supervised pretraining objective that uses expert polyp bounding boxes to focus last-layer [CLS] attention on annotated lesion regions, followed by standard cross-entropy fine-tuning. LLFA is applied to six ViT architectures and evaluated on the public SUN dataset for binary (0-I vs. 0-II) and three-class (0-Ip, 0-Is, 0-IIa) Paris classification. Performance is assessed using frame-wise accuracy and the AttIn, we additionally perform an ablation study against a Grad-CAM consistency baseline. Attention-supervised pretraining yields consistent gains in both accuracy and lesion-focused attention. Across the six ViTs, adding LLFA improves three-class accuracy by up to 7 percentage points. In a detailed ablation on ViT-B/16, LLFA outperforms a Grad-CAM consistency baseline by about 5-13 percentage points across the 2-class and 3-class tasks, and χ2 tests confirm a significant association between high AttIn and correct predictions. Direct supervision of ViT attention with LLFA leverages expert knowledge to jointly boost Paris classification accuracy and spatial interpretability, and compares favourably with Grad-CAM-based explanation regularisation. The source code and dataset splits are publicly available at https://github.com/LucaCarlini/SUNDatasetPretraining.
22 May 2026 in International Journal of Computer Assisted Radiology and Surgery
Purpose: Video question answering (VideoQA) in the surgical domain aims to enhance intraoperative understanding by enabling AI models to reason over temporally coherent events rather than isolated frames. Current approaches are limited to static image features, and available datasets often lack temporal annotations, ignoring the dynamics critical for accurate procedural interpretation. Methods: We propose SurgViVQA, a surgical VideoQA model that extends visual reasoning from static images to dynamic surgical scenes. It uses a Masked Video–Text Encoder to fuse video and question features, capturing temporal cues like motion and tool–tissue interactions, which a fine-tuned LLM then decodes into coherent answers. To evaluate its performance, we curate REAL-Colon-VQA, a colonoscopic video dataset including motion questions and diagnostic attributes, including out-of-template questions with rephrased or semantically altered formulations to evaluate model robustness. Results: Experimental validation on REAL-Colon-VQA and the public EndoVis18-VQA dataset shows that SurgViVQA outperforms existing image-based VQA benchmark models and remains competitive with a fine-tuned video VLM baseline. In particular, SurgViVQA improves over PitVQA by +9% on REAL-Colon-VQA and +9% on EndoVis18-VQA in Keyword Accuracy, while achieving the strongest overall lexical and semantic generation performance. A perturbation study on the questions further confirms improved generalizability and robustness to variations in question phrasing. Conclusion: SurgViVQA and the REAL-Colon-VQA dataset provide a framework for temporally aware understanding in surgical VideoQA, enabling AI models to interpret dynamic procedural contexts.
17 July 2026 in International Journal of Computer Assisted Radiology and Surgery
Purpose: Future human-in-the-loop surgical visual question answering (VQA) requires detecting unreliable answers before use. In this preclinical methodological study, we distinguish answer consistency from question validity: A model may generate mutually consistent answers that still fail to address the clinical question. Methods: We propose question-aligned semantic nearest neighbor entropy (QA-SNNE), a black-box failure-detection score that adds question-answer alignment to semantic entropy through bilateral gating. QA-SNNE modulates pairwise similarities among sampled answers by their question relevance, using embedding, entailment, or cross-encoder alignment. To assess robustness under language variation, we construct an out-of-template rephrased version of a benchmark surgical VQA dataset, in which only the question wording is modified while images and ground-truth answers remain unchanged. Results: We evaluate QA-SNNE on five VQA models across two benchmark surgical VQA datasets, considering both zero-shot and parameter-efficient fine-tuned settings, including out-of-template questions. QA-SNNE improves AUROC on EndoVis18-VQA for two of three zero-shot models in-template (e.g., +15% for Llama3.2 and +21% for Qwen2.5) and achieves up to +8% AUROC relative improvement among zero-shot models under out-of-template rephrasing, with mixed results on external validation. Conclusion: By making uncertainty estimation explicitly question-aware, QA-SNNE provides a practical and model-agnostic safeguard for preclinical surgical VQA evaluation. Linking semantic uncertainty to question relevance enables more reliable failure detection under variation in question phrasing.