Purpose: To develop deep learning-based networks for the diagnosis of diabetic retinopathy (DR) with cataracts based on infrared fundus images. Setting: Shanghai General Hospital, Shanghai Eye Disease Prevention & Treatment Center, Shanghai, China. Design: Development and evaluation of an artificial intelligence (AI) diagnostic method. Methods: A total of 10 665 infrared fundus images from 4553 patients with diabetes were used to train and test the model. For image quality assessment, left and right eye classification, DR diagnosis and grading, and segmentation of 3 DR lesions, an end-to-end software using EfficientNet and UNet was developed. The accuracy and performance of the software in comparison to human experts was evaluated. Results: The model achieved an accuracy of 75.31% for left and right eye classification, 100% for DR grading and diagnosis tasks, and 73.67% for internal test set, with corresponding areas under the curve (AUCs) of 0.88, 1.00, and 0.89, respectively. For DR lesion segmentation, the AUCs of hemorrhagic, microangioma, and exudative lesions were 0.86, 0.66, and 0.84, respectively. In addition, a contrast test of human-machine film reading confirmed the software's high sensitivity (96.3%) and specificity (90.0%) and consistency with the manual film reading group (κ = 0.869, P < .001). This easily deployable software generated reports quickly and promoted efficient DR screening with cataracts in clinical and community settings. Conclusions: AI-assisted software can perform automatic analysis of infrared fundus images and has substantial application value for the diagnosis of DR patients with cataracts.

Owing to a large amount of multi-modal data in modern medical systems, such as medical images and reports, Medical Vision-Language Pre-training (Med-VLP) has demonstrated incredible achievements in coarse-grained downstream tasks (i.e., medical classification, retrieval, and visual question answering). However, the problem of transferring knowledge learned from Med-VLP to fine-grained multi-organ segmentation tasks has barely been investigated. Multi-organ segmentation is challenging mainly due to the lack of large-scale fully annotated datasets and the wide variation in the shape and size of the same organ between individuals with different diseases. In this paper, we propose a novel pre-training & fine-tuning framework for Multi-Organ Segmentation by harnessing Medical repOrt Supervision (MOSMOS). Specifically, we first introduce global contrastive learning to maximally align the medical image-report pairs in the pre-training stage. To remedy the granularity discrepancy, we further leverage multi-label recognition to implicitly learn the semantic correspondence between image pixels and organ tags. More importantly, our pre-trained models can be transferred to any segmentation model by introducing the pixel-tag attention maps. Different network settings, i.e., 2D U-Net and 3D UNETR, are utilized to validate the generalization. We have extensively evaluated our approach using different diseases and modalities on BTCV, AMOS, MMWHS, and BRATS datasets. Experimental results in various settings demonstrate the effectiveness of our framework. This framework can serve as the foundation to facilitate future research on automatic annotation tasks under the supervision of medical reports.

Breast cancer is the most commonly diagnosed cancer and the leading cause of cancer-related mortality in women globally. Mammography is essential for the early detection and diagnosis of breast lesions. Despite recent progress in foundation models (FMs) for mammogram analysis, their clinical translation remains constrained by several fundamental limitations, including insufficient diversity in training data, limited model generalizability, and a lack of comprehensive evaluation across clinically relevant tasks. Here, we introduce VersaMammo, a versatile foundation model for mammograms, designed to overcome these limitations. We curated the largest multi-institutional mammogram dataset to date, comprising 706,239 images from 21 sources. To improve generalization, we propose a two-stage pre-training strategy to develop VersaMammo, a mammogram foundation model. First, a teacher model is trained via self-supervised learning to extract transferable features from unlabeled mammograms. Then, supervised learning combined with knowledge distillation transfers both features and clinical knowledge into VersaMammo. To ensure a comprehensive evaluation, we established a benchmark comprising 92 specific tasks, including 68 internal tasks and 24 external validation tasks, spanning 5 major clinical task categories: lesion detection, segmentation, classification, image retrieval, and visual question answering. VersaMammo achieves state-of-the-art performance, ranking first in 50 out of 68 specific internal tasks and 20 out of 24 external validation tasks, with average ranks of 1.5 and 1.2, respectively. These results demonstrate its superior generalization and clinical utility, offering a substantial advancement toward reliable and scalable breast cancer screening and diagnosis.