In endovascular and cardiovascular surgery, real-time guidewire morphological and positional analysis is an important pre-requisite for robot-assisted intervention, which can aid in reducing radiation dose, contrast agent and procedure time. Nevertheless, this task often comes with the challenge of the deformable elongated structure with low contrast in noisy X-ray fluoroscopy. In this paper, a real-time multi-functional framework is proposed for fully automatic guidewire morphological and positional analysis, namely guidewire segmentation, endpoint localization and angle measurement. In the first stage, the proposed Fast Attention Recurrent Network (FAR-Net) achieves real-time and accurate guidewire segmentation. In the second stage, the endpoint localization and angle measurement algorithm robustly obtain sub-pixel-level endpoint and angle of the guidewire tip. Quantitative and qualitative evaluations on MSGSeg dataset consisting of 180 X-ray sequences from 30 patients demonstrate that the proposed framework significantly outperforms simpler baselines as well as the best previously-published result for this task. The proposed approach reached F1-Score of 0.938, mean distance error of 0.596 pixels, endpoint localization & angle measurement accuracy of 97.8% & 95.3%, and inference rate of approximately 13 FPS. The proposed framework not only addresses the issues of extreme class imbalance and misclassified examples, but also meets the real-time requirements, achieving the state-of-the-art performance. The proposed approach is promising for integration into robotic navigation frameworks to various intravascular applications, enabling robotic-assisted intervention.

Magnetic Resonance Imaging (MRI) has been proven to be an efficient way to diagnose Alzheimer’s disease (AD). Recent dramatic progress on deep learning greatly promotes the MRI analysis based on data-driven CNN methods using a large-scale longitudinal MRI dataset. However, most of the existing MRI datasets are fragmented due to unexpected quits of volunteers. To tackle this problem, we propose a novel Temporal Recurrent Generative Adversarial Network (TR-GAN) to complete missing sessions of MRI datasets. Unlike existing GAN-based methods, which either fail to generate future sessions or only generate fixed-length sessions, TR-GAN takes all past sessions to recurrently and smoothly generate future ones with variant length. Specifically, TR-GAN adopts recurrent connection to deal with variant input sequence length and flexibly generate future variant sessions. Besides, we also design a multiple scale & location (MSL) module and a SWAP module to encourage the model to better focus on detailed information, which helps to generate high-quality MRI data. Compared with other popular GAN architectures, TR-GAN achieved the best performance in all evaluation metrics of two datasets. After expanding the Whole MRI dataset, the balanced accuracy of AD vs. cognitively normal (CN) vs. mild cognitive impairment (MCI) and stable MCI vs. progressive MCI classification can be increased by 3.61% and 4.00%, respectively.

The real-time localization of the guidewire endpoints is a stepping stone to computer-assisted percutaneous coronary intervention (PCI). However, methods for multi-guidewire endpoint localization in fluoroscopy images are still scarce. In this paper, we introduce a framework for real-time multi-guidewire endpoint localization in fluoroscopy images. The framework consists of two stages, first detecting all guidewire instances in the fluoroscopy image, and then locating the endpoints of each single guidewire instance. In the first stage, a YOLOv3 detector is used for guidewire detection, and a post-processing algorithm is proposed to refine the guidewire detection results. In the second stage, a Segmentation Attention-hourglass (SA-hourglass) network is proposed to predict the endpoint locations of each single guidewire instance. The SA-hourglass network can be generalized to the keypoint localization of other surgical instruments. In our experiments, the SA-hourglass network is applied not only on a guidewire dataset but also on a retinal microsurgery dataset, reaching the mean pixel error (MPE) of 2.20 pixels on the guidewire dataset and the MPE of 5.30 pixels on the retinal microsurgery dataset, both achieving the state-of-the-art localization results. Besides, the inference rate of our framework is at least 20FPS, which meets the real-time requirement of fluoroscopy images (6-12FPS).

Objective: Technical skills assessment plays an important role in the professional development of an interventionalist in percutaneous coronary intervention (PCI), however, most of traditional assessment methods are time-consuming and subjective. This paper aims to develop objective assessment techniques. Methods: In this study, a natural-behavior-based analysis framework is proposed to qualitatively and quantitatively assess technical skills in PCI. In vivo porcine studies were conducted to deliver a medical guidewire to two target coronaries of left circumflex arteries by six novice and four expert interventionalists. Simultaneously, four types of natural behaviors (i.e., hand motion, proximal force, muscle activity and finger motion) were acquired from the subjects' dominant hand and arm. The features extracted from behaviors of different skill-level groups were compared using Mann-Whitney U-test for effective behavior selection. The effective ones were further applied in Gaussian-mixture-model-based qualitative assessment and Mahalanobis-distance-based quantitative assessment. Results: The qualitative assessment achieves 92% accuracy to distinguish the novice and expert attempts, which is significantly higher than that of using single guidewire motions. Furthermore, the quantitative assessment can assign objective and effective scores for all attempts, indicating high correlation (R=0.9225) to those obtained by traditional methods. Conclusion: The objective, effective and comprehensive assessment of technical skills can be provided by qualitatively and quantitatively analyzing interventionalists' natural behaviors in PCI. Significance: This paper suggests a novel approach for technical skills assessment and the promising results demonstrate the great importance and effectiveness of the proposed method for promoting the development of objective assessment techniques.

Percutaneous coronary intervention (PCI) has increasingly become the main treatment for coronary artery disease. The procedure requires high experienced skills and dexterous manipulations. However, there are few techniques to model PCI skill so far. In this study, a learning framework with local and ensemble learning is proposed to learn skill characteristics of different skill-level subjects from their PCI manipulations. Ten interventional cardiologists (four experts and six novices) were recruited to deliver a medical guidewire to two target arteries on a porcine model for in vivo studies. Simultaneously, translation and twist manipulations of thumb, forefinger, and wrist are acquired with electromagnetic (EM) and fiber-optic bend (FOB) sensors, respectively. These behavior data are then processed with wavelet packet decomposition (WPD) under 1-10 levels for feature extraction. The feature vectors are further fed into three candidate individual classifiers in the local learning layer. Furthermore, the local learning results from different manipulation behaviors are fused in the ensemble learning layer with three rule-based ensemble learning algorithms. In subject-dependent skill characteristics learning, the ensemble learning can achieve 100% accuracy, significantly outperforming the best local result (90%). Furthermore, ensemble learning can also maintain 73% accuracy in subject-independent schemes. These promising results demonstrate the great potential of the proposed method to facilitate skill learning in surgical robotics and skill assessment in clinical practice.