6 February 2025 in Communications Medicine
Background: Over the next 5 years, new breast cancer screening guidelines recommending magnetic resonance imaging (MRI) for certain patients will significantly increase the volume of imaging data to be analyzed. While this increase poses challenges for radiologists, artificial intelligence (AI) offers potential solutions to manage this workload. However, the development of AI models is often hindered by manual annotation requirements and strict data-sharing regulations between institutions. Methods: In this study, we present an integrated pipeline combining weakly supervised learning—reducing the need for detailed annotations—with local AI model training via swarm learning (SL), which circumvents centralized data sharing. We utilized three datasets comprising 1372 female bilateral breast MRI exams from institutions in three countries: the United States (US), Switzerland, and the United Kingdom (UK) to train models. These models were then validated on two external datasets consisting of 649 bilateral breast MRI exams from Germany and Greece. Results: Upon systematically benchmarking various weakly supervised two-dimensional (2D) and three-dimensional (3D) deep learning (DL) methods, we find that the 3D-ResNet-101 demonstrates superior performance. By implementing a real-world SL setup across three international centers, we observe that these collaboratively trained models outperform those trained locally. Even with a smaller dataset, we demonstrate the practical feasibility of deploying SL internationally with on-site data processing, addressing challenges such as data privacy and annotation variability. Conclusions: Combining weakly supervised learning with SL enhances inter-institutional collaboration, improving the utility of distributed datasets for medical AI training without requiring detailed annotations or centralized data sharing.
8 February 2025 in Journal of Surgical Education
Recent studies investigated the potential of large language models (LLMs) for clinical decision making and answering exam questions based on text input. Recent developments of LLMs have extended these models with vision capabilities. These image processing LLMs are called vision-language models (VLMs). However, there is limited investigation on the applicability of VLMs and their capabilities of answering exam questions with image content. Therefore, the aim of this study was to examine the performance of publicly accessible LLMs in 2 different surgical question sets consisting of text and image questions. Original text and image exam questions from 2 different surgical question subsets from the German Medical Licensing Examination (GMLE) and United States Medical Licensing Examination (USMLE) were collected and answered by publicly available LLMs (GPT-4, Claude-3 Sonnet, Gemini-1.5). LLM outputs were benchmarked for their accuracy in answering text and image questions. Additionally, the LLMs' performance was compared to students' performance based on their average historical performance (AHP) in these exams. Moreover, variations of LLM performance were analyzed in relation to question difficulty and respective image type. Overall, all LLMs achieved scores equivalent to passing grades (≥60%) on surgical text questions across both datasets. On image-based questions, only GPT-4 exceeded the score required to pass, significantly outperforming Claude-3 and Gemini-1.5 (GPT: 78% vs. Claude-3: 58% vs. Gemini-1.5: 57.3%; p < 0.001). Additionally, GPT-4 outperformed students on both text (GPT: 83.7% vs. AHP students: 67.8%; p < 0.001) and image questions (GPT: 78% vs. AHP students: 67.4%; p < 0.001). GPT-4 demonstrated substantial capabilities in answering surgical text and image exam questions. Therefore, it holds considerable potential for the use in surgical decision making and education of students and trainee surgeons.
12 June 2026 in Scientific Data
The limited availability of diverse and representative training data poses a critical barrier to the development of clinically relevant computational tools for intraoperative surgical decision support. Surgical procedures are not routinely recorded, and data annotation requires domain expertise, resulting in a scarcity of open-access surgical video datasets with high-quality annotations. Existing datasets are typically limited to single institutions and specific procedures, such as cholecystectomy, and rarely comprise patient-level metadata like demographic characteristics, disease history, or laboratory parameters. The Appendix300 dataset comprises 330 laparoscopic surgery recordings, including 325 full-length laparoscopic appendectomies and 5 control recordings from non-appendectomy procedures in pediatric and adult patients treated at five German centers. The dataset includes patient-level clinical metadata (demographics, medical history, clinical symptoms, preoperative laboratory parameters, and histopathologic findings, as well as standardized expert annotations of the laparoscopic grade of appendicitis). This dataset enables novel validation tasks for computer vision in laparoscopic surgery and facilitates simulation of decentralized learning approaches, overall enhancing the breadth and translational relevance of AI-based surgical video analysis.
7 September 2026 in Medical Image Analysis
3 October 2025 in medRxiv
Background Progress in artificial intelligence-based analysis of surgical videos has been constrained by reliance on manual frame-level annotations rather than patient-level outcomes. In addition, concerns about data privacy restrict the exchange of laparoscopic video data and, thereby, multicenter collaboration. Methods To address these limitations, we developed a pipeline that integrates weakly supervised deep learning with Swarm Learning, a decentralized machine learning approach that enables collaborative model training without data centralization. We evaluate our pipeline using a newly curated dataset of 397 laparoscopic appendectomy recordings from six international surgical centers. We identified optimal modelling configurations (frame sampling rates and model architectures) and subsequently compared Swarm Learning to single-center and centralized learning across three novel patient-level disease staging tasks: (i) binary detection of perforated appendicitis, (ii) laparoscopic grading of appendicitis, and (iii) histopathologic inflammation grading. In addition, we surveyed participating centers to identify real-world barriers to the clinical implementation of our decentralized learning pipeline for surgical video analysis. Results For appendicitis grading tasks, frame sampling at 1.0 frames per second and use of the SurgTempoNet architecture resulted in reliable classification performance, outperforming SurgFrameNet and Multiple Instance Learning. Across all three disease staging tasks, Swarm Learning consistently outperformed single-center training and achieved performance comparable to centralized learning, with stable generalization in external validation. The user survey identified hardware failure and limited integration of the decentralized learning pipeline with electronic patient records as key barriers to the clinical implementation of our decentralized learning pipeline for collaborative surgical video analysis. Conclusions Weakly supervised deep learning enables the prediction of patient-level endpoints directly from surgical video data. Swarm Learning facilitates privacy-preserving multicenter collaboration and achieves performance on par with centralized learning, highlighting its potential for advancing clinically relevant, collaborative AI development in surgical video analysis, especially when integrated with patients’ electronic health records. Article Description This study introduces a decentralized, privacy-preserving pipeline that combines weakly supervised deep learning with Swarm Learning to predict patient-level outcomes from laparoscopic appendectomy videos. Using data from six international surgical centers, the approach demonstrated performance comparable to centralized learning across three disease staging tasks while preserving data confidentiality by design.