Recently a number of studies demonstrated impressive performance on diverse vision-language multi-modal tasks such as image captioning and visual question answering by extending the BERT architecture with multi-modal pre-training objectives. In this work we explore a broad set of multi-modal representation learning tasks in the medical domain, specifically using radiology images and the unstructured report. We propose Medical Vision Language Learner (MedViLL), which adopts a BERT-based architecture combined with a novel multi-modal attention masking scheme to maximize generalization performance for both vision-language understanding tasks (diagnosis classification, medical image-report retrieval, medical visual question answering) and vision-language generation task (radiology report generation). By statistically and rigorously evaluating the proposed model on four downstream tasks with three radiographic image-report datasets (MIMIC-CXR, Open-I, and VQA-RAD), we empirically demonstrate the superior downstream task performance of MedViLL against various baselines, including task-specific architectures. The source code is publicly available at: https://github.com/SuperSupermoon/MedViLL

X-ray coronary angiography is a primary imaging technique for diagnosing coronary diseases. Although quantitative coronary angiography (QCA) provides morphological information of coronary arteries with objective quantitative measures, considerable training is required to identify the target vessels and understand the tree structure of coronary arteries. Despite the use of computer-aided tools, such as the edge-detection method, manual correction is necessary for accurate segmentation of coronary vessels. In the present study, we proposed a robust method for major vessel segmentation using deep learning models with fully convolutional networks. When angiographic images of 3302 diseased major vessels from 2042 patients were tested, deep learning networks accurately identified and segmented the major vessels in X-ray coronary angiography. The average F1 score reached 0.917, and 93.7% of the images exhibited a high F1 score > 0.8. The most narrowed region at the stenosis was distinctly captured with high connectivity. Robust predictability was validated for the external dataset with different image characteristics. For major vessel segmentation, our approach demonstrated that prediction could be completed in real time with minimal image preprocessing. By applying deep learning segmentation, QCA analysis could be further automated, thereby facilitating the use of QCA-based diagnostic methods.

To compare the diagnostic accuracy of on-site computed tomography (CT)-derived fractional flow reserve (FFR) and stress CT myocardial perfusion (CTP) in patients with coronary artery disease. Using a prospective CTP registry, 72 patients with invasive FFR were enrolled. CT-derived FFR was computed on-site using rest-phase CTP data. The diagnostic accuracies of coronary CT angiography (CCTA), CT-derived FFR, and stress CTP were evaluated using an area under the receiver-operating characteristic curve (AUC) with invasive FFR as a reference standard. Logistic regression and the net reclassification index (NRI) were used to evaluate incremental differences in CT-derived FFR or CTP compared with CCTA alone. The per-vessel prevalence of haemodynamically significant stenosis (FFR ≤ 0.80) was 39% (54/138). Per-vessel sensitivity and specificity were 94 and 66% for CCTA, 87 and 77% for CT-derived FFR, and 79 and 91% for CTP, respectively. There was no significant difference in the AUC values of CT-derived FFR and CTP (P = 0.845). The diagnostic performance of CCTA (AUC = 0.856) was improved by combining it with CT-derived FFR (AUC = 0.919, P = 0.004, NRI = 1.01) or CTP (AUC = 0.913, P = 0.004, NRI = 0.66). CT-derived FFR values had a moderate correlation with invasive FFR (r = 0.671, P < 0.001). On-site CT-derived FFR combined with CCTA provides an incremental diagnostic improvement over CCTA alone in identifying haemodynamically significant stenosis defined by invasive FFR, with a diagnostic accuracy comparable with CTP.

Background: Transformer-based language models have shown great potential to revolutionize health care by advancing clinical decision support, patient interaction, and disease prediction. However, despite their rapid development, the implementation of transformer-based language models in health care settings remains limited. This is partly due to the lack of a comprehensive review, which hinders a systematic understanding of their applications and limitations. Without clear guidelines and consolidated information, both researchers and physicians face difficulties in using these models effectively, resulting in inefficient research efforts and slow integration into clinical workflows. Objective: This scoping review addresses this gap by examining studies on medical transformer-based language models and categorizing them into 6 tasks: dialogue generation, question answering, summarization, text classification, sentiment analysis, and named entity recognition. Methods: We conducted a scoping review following the Cochrane scoping review protocol. A comprehensive literature search was performed across databases, including Google Scholar and PubMed, covering publications from January 2017 to September 2024. Studies involving transformer-derived models in medical tasks were included. Data were categorized into 6 key tasks. Results: Our key findings revealed both advancements and critical challenges in applying transformer-based models to health care tasks. For example, models like MedPIR involving dialogue generation show promise but face privacy and ethical concerns, while question-answering models like BioBERT improve accuracy but struggle with the complexity of medical terminology. The BioBERTSum summarization model aids clinicians by condensing medical texts but needs better handling of long sequences. Conclusions: This review attempted to provide a consolidated understanding of the role of transformer-based language models in health care and to guide future research directions. By addressing current challenges and exploring the potential for real-world applications, we envision significant improvements in health care informatics. Addressing the identified challenges and implementing proposed solutions can enable transformer-based language models to significantly improve health care delivery and patient outcomes. Our review provides valuable insights for future research and practical applications, setting the stage for transformative advancements in medical informatics.

Lee JG, et al. Korean J Radiol. 2021 Nov;22(11):1764-1776. https://doi.org/10.3348/kjr.2021.0148

In percutaneous intervention for treatment of coronary plaques, guidewire navigation is a primary procedure for stent delivery. Steering a flexible guidewire within coronary arteries requires considerable training, and the non-linearity between the control operation and the movement of the guidewire makes precise manipulation difficult. Here, we introduce a deep reinforcement learning (RL) framework for autonomous guidewire navigation in a robot-assisted coronary intervention. Using Rainbow, a segment-wise learning approach is applied to determine how best to accelerate training using human demonstrations, transfer learning, and weight initialization. ‘State’ for RL is customized as a focus window near the guidewire tip, and subgoals are placed to mitigate a sparse reward problem. The RL agent improves performance, eventually enabling the guidewire to reach all valid targets in ‘stable’ phase. For the last 300 out of 1000 episodes, the success rates of the guidewire navigation to the distal-main and side targets were 98% and 99% in 2D and 3D phantoms, respectively. Our framework opens a new direction in the automation of robot-assisted intervention, providing guidance on RL in physical spaces involving mechanical fatigue.