The purpose of this work was to develop a clinically viable laparoscopic augmented reality (AR) system employing stereoscopic (3-D) vision, laparoscopic ultrasound (LUS), and electromagnetic (EM) tracking to achieve image registration. We investigated clinically feasible solutions to mount the EM sensors on the 3-D laparoscope and the LUS probe. This led to a solution of integrating an externally attached EM sensor near the imaging tip of the LUS probe, only slightly increasing the overall diameter of the probe. Likewise, a solution for mounting an EM sensor on the handle of the 3-D laparoscope was proposed. The spatial image-to-video registration accuracy of the AR system was measured to be $2.59±0.58mm$ and $2.43±0.48mm$ for the left- and right-eye channels, respectively. The AR system contributed 58-ms latency to stereoscopic visualization. We further performed an animal experiment to demonstrate the use of the system as a visualization approach for laparoscopic procedures. In conclusion, we have developed an integrated, compact, and EM tracking-based stereoscopic AR visualization system, which has the potential for clinical use. The system has been demonstrated to achieve clinically acceptable accuracy and latency. This work is a critical step toward clinical translation of AR visualization for laparoscopic procedures.

Surgical tool tracking has a variety of applications in different surgical scenarios. Electromagnetic (EM) tracking can be utilised for tool tracking, but the accuracy is often limited by magnetic interference. Vision‐based methods have also been suggested; however, tracking robustness is limited by specular reflection, occlusions, and blurriness observed in the endoscopic image. Recently, deep learning‐based methods have shown competitive performance on segmentation and tracking of surgical tools. The main bottleneck of these methods lies in acquiring a sufficient amount of pixel‐wise, annotated training data, which demands substantial labour costs. To tackle this issue, the authors propose a weakly supervised method for surgical tool segmentation and tracking based on hybrid sensor systems. They first generate semantic labellings using EM tracking and laparoscopic image processing concurrently. They then train a light‐weight deep segmentation network to obtain a binary segmentation mask that enables tool tracking. To the authors’ knowledge, the proposed method is the first to integrate EM tracking and laparoscopic image processing for generation of training labels. They demonstrate that their framework achieves accurate, automatic tool segmentation (i.e. without any manual labelling of the surgical tool to be tracked) and robust tool tracking in laparoscopic image sequences.

Digital stethoscopes can enable the development of integrated artificial intelligence (AI) systems that can remove the subjectivity of manual auscultation, improve diagnostic accuracy, and compensate for diminishing auscultatory skills. Developing scalable AI systems can be challenging, especially when acquisition devices differ and thus introduce sensor bias. To address this issue, a precise knowledge of these differences, i.e., frequency responses of these devices, is needed, but the manufacturers often do not provide complete device specifications. In this study, we reported an effective methodology for determining the frequency response of a digital stethoscope and used it to characterise three common digital stethoscopes: Littmann 3200, Eko Core, and Thinklabs One. Our results show significant inter-device variability in that the frequency responses of the three studied stethoscopes were distinctly different. A moderate intra-device variability was seen when comparing two separate units of Littmann 3200. The study highlights the need for normalisation across devices for developing successful AI-assisted auscultation and provides a technical characterisation approach as a first step to accomplish it.

Heart sound segmentation has been shown to improve the performance of artificial intelligence (AI)-based auscultation decision support systems increasingly viewed as a solution to compensate for eroding auscultatory skills and the associated subjectivity. Various segmentation approaches with demonstrated performance can be utilized for this task, but their robustness can suffer in the presence of noise. A noise-robust heart sound segmentation algorithm was developed and its accuracy was tested using two datasets: the CirCor DigiScope Phonocardiogram dataset and an in-house dataset – a heart murmur library collected at the Children’s National Hospital (CNH). On the CirCor dataset, our segmentation algorithm marked the boundaries of the primary heart sounds S1 and S2 with an accuracy of 0.28 ms and 0.29 ms, respectively, and correctly identified the actual positive segments with a sensitivity of 97.44%. The algorithm also executed four times faster than a logistic regression hidden semi-Markov model. On the CNH dataset, the algorithm succeeded in 87.4% cases, achieving a 6% increase in segmentation success rate demonstrated by our original Shannon energy-based algorithm. Accurate heart sound segmentation is critical to supporting and accelerating AI research in cardiovascular diseases. The proposed algorithm increases the robustness of heart sound segmentation to noise and viability for clinical use.