When developing context-aware systems, automatic surgical phase recognition and tool presence detection are two essential tasks. There exist previous attempts to develop methods for both tasks but majority of the existing methods utilize a frame-level loss function (e.g., cross-entropy) which does not fully leverage the underlying semantic structure of a surgery, leading to sub-optimal results. In this paper, we propose multi-task learning-based, LAtent Space-constrained Transformers, referred as LAST, for automatic surgical phase recognition and tool presence detection. Our design features a two-branch transformer architecture with a novel and generic way to leverage video-level semantic information during network training. This is done by learning a non-linear compact presentation of the underlying semantic structure information of surgical videos through a transformer variational autoencoder (VAE) and by encouraging models to follow the learned statistical distributions. In other words, LAST is of structure-aware and favors predictions that lie on the extracted low dimensional data manifold. Validated on two public datasets of the cholecystectomy surgery, i.e., the Cholec80 dataset and the M2cai16 dataset, our method achieves better results than other state-of-the-art methods. Specifically, on the Cholec80 dataset, our method achieves an average accuracy of 93.12±4.71%, an average precision of 89.25±5.49%, an average recall of 90.10±5.45% and an average Jaccard of 81.11 ±7.62% for phase recognition, and an average mAP of 95.15±3.87% for tool presence detection. Similar superior performance is also observed when LAST is applied to the M2cai16 dataset.
18 January 2023 in IEEE Transactions on Medical Robotics and Bionics
Dense depth estimation plays a crucial role in developing context-aware computer-assisted intervention systems. However, it is a challenging task due to low image quality and highly dynamic surgical environment. The task is further complicated by the difficulty in acquiring per-pixel ground truth depth data in a surgical setting. Recent works on self-supervised depth estimation use image reconstruction (i.e., the warped images) as supervisory signal, which helps to eliminate the requirement of ground truth depth annotations but also causes over-smoothed depth predictions. Additionally, most existing depth estimation methods are built upon static laparoscopic images, ignoring rich temporal information. To address these challenges, we propose a novel spatiotemporal vision transformers-based self-supervised depth estimation method, referred as SVT-SDE. Unlike previous works, SVT-SDE features a novel spatiotemporal vision transformers (SVT) architecture, which can learn complementary visual and temporal information from the input stereoscopic video clips. We further introduce high-frequency-based supervisory signal, which helps to preserve fine-grained details of depth estimation. Results from experiments conducted on two publicly available datasets demonstrate the superior performance of SVT-SDE over the state-of-the-art self-supervised depth estimation methods.
Inflammatory bowel disease (IBD) includes Crohn’s disease (CD) and ulcerative colitis (UC). The accurate classification of IBD from colonoscopy images is critical for diagnosis and treatment. However, the lack of labeled data poses a major challenge for developing deep learning-based IBD classification approaches. Recently, pseudo-labeling-based semi-supervised learning methods offer a promising solution in leveraging both labeled and unlabeled data to improve classification performance. Nevertheless, due to significant intra-class variability and the subtle inter-class differences in IBD colonoscopy images, pseudo-labels are often inaccurate, which results in confirmation bias and suboptimal performance. To address this challenge, a Subclass-Aware Contrastive Semi-Supervised Learning method, referred to as SACSSL, is proposed for accurate IBD classification by integrating a subclass-aware contrastive module into a pseudo-labeling-based semi-supervised framework, e.g., FixMatch. Specifically, unlabeled samples are first partitioned into confident and uncertain samples according to the confidence of pseudo-labels. An instance-level contrastive loss is then applied to uncertain samples, aiming to mitigate confirmation bias. Furthermore, intra-class heterogeneity is captured by introducing a set of prototypes for each subclass and assigning confident samples to these prototypes to form fine-grained subclasses, and supervised contrastive loss is applied to promote intra-subclass clustering, thereby enhancing inter-class separability while preserving intra-class diversity. Our method is evaluated on two datasets, i.e., an in-house collected Daping dataset for IBD classification and a publicly available LIMUC dataset for UC severity grading. On both datasets, our method achieves state-of-the-art performance under the semi-supervised setting. Specifically, with only 20% labeled data, the proposed method reaches an overall accuracy of 93.2% and an F1-score of 80.1% on the Daping dataset, which is close to the fully supervised upper bound (94.0% accuracy and 80.8% F1-score), and it achieves an overall accuracy of 76.4% and an F1-score of 68.9% on the LIMUC dataset. Comprehensive experimental results demonstrate the effectiveness of our method for semi-supervised colonoscopy image classification.
3 June 2026 in IEEE Transactions on Instrumentation and Measurement
Robot-Assisted Laminectomy (RAL) has become an emerging approach for treatment of lumbar spinal stenosis. However, the spine is an articulated structure that can be deformed under contact force during grinding. Patient-image registration performed at the start of RAL cannot account for intraoperative intervertebral motion, resulting in degraded surgical accuracy and compromised safety. To tackle this issue, we propose a novel stereo tracking technology for intraoperative intervertebral motion compensation in RAL. Our tracking technology is a marker-based method, consisting of customized markers, a coding system, and a labeling and position tracking framework. Specifically, we first propose a new type of markers featured by binary-coded spatial patterns, referred to as SurgTag marker. Each marker in the codebook has a unique digital IDentification (ID) code and a fixed set of keypoints. By rigidly attaching a marker with a unique ID to each vertebra during RAL, we can simultaneously track multiple vertebrae. We then introduce a two-stage SurgTag-marker-based tracking framework consisting of a marker labeling stage followed by a keypoint detection stage. In theory, we formulate both marker labeling and keypoint detection as one-to-one set prediction problems and further propose respectively two different types of DEtection-TRansformer (DETR)-based detector: Marker-DETR and Keypoint-DETR. Marker poses are then estimated based on detected keypoints, enabling tracking and compensation for intervertebral motion. Comprehensive experiments were conducted for evaluation. The proposed method achieved on average a submillimeter translation error and a sub-degree orientation error on both a synthetic dataset and an ex-vivo dataset. Additionally, surgical robot with the proposed technology achieved a submillimeter grinding error, demonstrating its effectiveness for improving surgical accuracy. The code will be publicly available at https://github.com/Wenyuan-Sun/StereoTracking.
3 April 2024 in International Journal of Computer Assisted Radiology and Surgery