Foundation models have revolutionized computer vision by achieving vastly superior performance across diverse tasks through large-scale pretraining on extensive datasets. However, their application in surgical computer vision has been limited. This study addresses this gap by introducing SurgeNetXL, a novel surgical foundation model that sets a new benchmark in surgical computer vision. Trained on the largest reported surgical dataset to date, comprising over 4.7 million video frames, SurgeNetXL achieves consistent top-tier performance across six datasets spanning four surgical procedures and three tasks, including semantic segmentation, surgical phase recognition, and critical view of safety (CVS) classification. Compared with the best-performing surgical foundation model, SurgeNetXL shows mean improvements of 4.0 %, 8.9 %, and 11.4 % for semantic segmentation, phase recognition, and CVS classification, respectively. Additionally, SurgeNetXL outperforms ImageNet1k by 16.1 %, 8.0 %, and 4.3 % for the respective tasks. In addition to advancing model performance, this study provides key insights into scaling pretraining datasets, extending training durations, and optimizing model architectures specifically for surgical computer vision. These findings pave the way for improved generalization and robustness in data-scarce scenarios, offering a comprehensive framework for future research in this domain. All models and a subset of the SurgeNetXL dataset, including over 2 million video frames, are publicly available at: https://github.com/TimJaspers0801/SurgeNet.

Reliable recognition and localization of surgical instruments in endoscopic video recordings are foundational for a wide range of applications in computer- and robot-assisted minimally invasive surgery (RAMIS), including surgical training, skill assessment, and autonomous assistance. However, robust performance under real-world conditions remains a significant challenge. Incorporating surgical context - such as the current procedural phase - has emerged as a promising strategy to improve robustness and interpretability. To address these challenges, we organized the Surgical Procedure Phase, Keypoint, and Instrument Recognition (PhaKIR) sub-challenge as part of the Endoscopic Vision (EndoVis) challenge at MICCAI 2024. We introduced a novel, multi-center dataset comprising thirteen full-length laparoscopic cholecystectomy videos collected from three distinct medical institutions, with unified annotations for three interrelated tasks: surgical phase recognition, instrument keypoint estimation, and instrument instance segmentation. Unlike existing datasets, ours enables joint investigation of instrument localization and procedural context within the same data while supporting the integration of temporal information across entire procedures. We report results and findings in accordance with the BIAS guidelines for biomedical image analysis challenges. The PhaKIR sub-challenge advances the field by providing a unique benchmark for developing temporally aware, context-driven methods in RAMIS and offers a high-quality resource to support future research in surgical scene understanding.

Deep learning-based surgical vision systems achieve strong performance in semantic segmentation and phase recognition, but their black-box nature limits traceability in safety-critical clinical settings. Prototype-based networks offer an interpretable alternative by grounding predictions in learned visual exemplars, yet their suitability for surgical video understanding remains insufficiently characterized. We adapted a prototype-based architecture to two surgical datasets, laparoscopic cholecystectomy and robot-assisted minimally invasive esophagectomy (RAMIE), and benchmarked it against conventional baselines. We evaluated a segmentation-only setting, in which prototype size and capacity were ablated, and a multitask setting, in which three strategies for coupling prototype learning to semantic segmentation and surgical phase recognition were compared. Prototype-based models underperformed the conventional baselines across both tasks and datasets. In the segmentation-only setting, the selected prototype configurations achieved Dice scores of 72.23% on Cholecystectomy and 72.07% on RAMIE, compared with 74.35% and 74.02% for the corresponding conventional baselines, and showed weaker boundary agreement. In the multitask setting, the best prototype strategy recovered competitive segmentation performance but remained 6–12 F1 points below the conventional baseline for phase recognition. Qualitatively, prototype activation maps exposed intra-structure decompositions and contextual cues that are not directly available from black-box baselines. Prototype networks provide spatially traceable evidence for surgical scene understanding, but currently trade interpretability for reduced boundary precision and phase-recognition performance. These findings motivate future work on scene-level and temporally aware prototypes for explainable surgical AI.