This paper presents a novel augmented reality (AR)-based neurosurgical training simulator which provides a very natural way for surgeons to learn neurosurgical skills. Surgical simulation with bimanual haptic interaction is integrated in this work to provide a simulated environment for users to achieve holographic guidance for pre-operative training. To achieve the AR guidance, the simulator should precisely overlay the 3D anatomical information of the hidden target organs in the patients in real surgery. In this regard, the patient-specific anatomy structures are reconstructed from segmented brain magnetic resonance imaging. We propose a registration method for precise mapping of the virtual and real information. In addition, the simulator provides bimanual haptic interaction in a holographic environment to mimic real brain tumor resection. In this study, we conduct AR-based guidance validation and a user study on the developed simulator, which demonstrate the high accuracy of our AR-based neurosurgery simulator, as well as the AR guidance mode’s potential to improve neurosurgery by simplifying the operation, reducing the difficulty of the operation, shortening the operation time, and increasing the precision of the operation.

Background: Indocyanine green (ICG) fluorescence imaging is a new technology that can improve the real-time location of tumor edges and small nodules during surgery. However, no study has investigated its application in laparoscopic insulinoma enucleation. This study aimed to evaluate the feasibility and accuracy of this method for intraoperative localization of insulinomas and margin assessment during laparoscopic insulinoma enucleation. Materials and Methods: Eight patients who underwent laparoscopic insulinoma enucleation from October 2016 to June 2022 were enrolled. Two methods of ICG administration, ICG dynamic perfusion and three-dimensional (3D) demarcation staining, were utilized in the laparoscopic insulinoma enucleation. Tumor-to-background ratio (TBR) and histopathologic analysis were used to evaluate the feasibility and accuracy of these novel navigation methods in laparoscopic insulinoma enucleation. Results: All eight enrolled patients underwent both ICG dynamic perfusion and 3D demarcation staining. ICG dynamic perfusion images were available for six of them, among which five tumors could be recognized by TBR (largest TBR in each case 4.42±2.76), while the other could be distinguished by the disordered blood vessels in the tumor area. Seven out of eight specimens had successful 3D demarcation staining (TBR 7.62±2.62). All wound bed margins had negative frozen sections and final histopathologic diagnoses. Conclusions: ICG dynamic perfusion may be helpful in observing the abnormal vascular perfusion of tumors, providing similar functionality to intraoperative real-time angiography. ICG injection under the tumor pseudocapsule may be a useful method for acquiring real-time, 3D demarcation for the resection of insulinoma.

Compared with open liver resection, laparoscopic hepatectomy (LH) aims to deliver equivalent oncological outcomes with the benefits of reduced trauma, faster postoperative recovery, and improved postoperative quality of life¹. Because of the complex anatomical structure of the liver, accurate identification of key landmarks is crucial for safe implementation of LH². Next-generation computer vision approaches based on artificial intelligence (AI) technology can perform pixel-level semantic segmentation of medical images, achieving a human-like image recognition ability, and have the potential to provide real-time guidance during complex operations^3,4. However, few studies have constructed AI models that can recognize anatomical structures and landmarks during LH^5–8. The aim of this study was to train AI models for the real-time automatic recognition of critical anatomical landmarks during LH and to assess their performance. Intraoperative videos of 64 standardized laparoscopic left lateral sectionectomy procedures (LLLS) from four centres were used to train a preliminary AI model for LH. The surgical approach to LLLS in these videos was the standardized two-step approach, which involves dissecting and dissociating the left Glissonean pedicle (LGP) first, and then dissecting and dissociating the left hepatic vein (LHV)^9,10. DeepLab v3+ and nnU-Net were chosen for semantic segmentation model training based on the labelled data (Fig. S1)^11,12. Some 51 videos from 3 centres were labelled and used as the internal training cohort. Data from 13 videos from another institution were labelled and used as an independent external cohort. Intersection over union (IoU), F1 score (Dice coefficient), and recognition speed (first recognition by surgeon versus AI model) were the main indicators used to evaluate the performance of the AI models. For ease of understanding, factors used in the calculation formulas for IoU, F1 score, sensitivity, specificity, and accuracy were true-negative, false-negative, true-positive, and false-positive (Fig. S2). The Committee of the Ethical Board of the four hospitals reviewed this study and approved the use of video data and medical records (2024-KY-064-01).

Accurate retinal vessel segmentation is a crucial step in the clinical diagnosis and treatment of fundus diseases. Although many efforts have been presented to address the task, the segmentation performance in challenging regions (e.g., collateral vessels) is still not satisfactory, due to their thin morphology or the low contrast between foreground and background. In this work, we observe that an intrinsic appearance exists in the retinal image: among the dendritic vessels there are generous similar structures, e.g., the main and collateral vessels are all curvilinear, but they have noticeable scale differences. Based on this observation, we propose a novel cross-scale attention transformer (CAT) to encourage the segmentation effects in challenging regions. Specifically, CAT consumes features with different scales to produce their shared attention matrix, and then fully integrates the beneficial information between them. Such new attention architecture could explore the multi-scale idea more efficiently, thus realizing mutual learning of similar structures. In addition, a progressive edge refinement module (ERM) is designed to refine the edges of foreground and background in the segmentation results. Through the idea of edge decoupling, ERM could suppress the background feature near the blood vessels while enhancing the foreground feature, so as to segment vessels accurately. We conduct extensive experiments and discussions on DRIVE and CHASE_DB1 datasets to verify the proposed framework. Experimental results show that our method has great advantages in the Se metric, which are 0.88–7.26% and 0.81–7.11% higher than the state-of-the-art methods on DRIVE and CHASE_DB1, respectively. In addition, the proposed method also outperforms other methods with 0.17–2.06% in terms of the Dice metric on DRIVE.