Detecting in open-world scenarios poses a formidable challenge for models intended for real-world deployment. The advanced closed set object detectors achieve impressive performance under the closed set setting, but often produce overconfident misprediction on unknown objects due to the lack of supervision. In this paper, we propose a novel Evidential Object Detector (EOD) to formulate the Open Set Object Detection (OSOD) problem from the perspective of Evidential Deep Learning (EDL) theory, which quantifies classification uncertainty by placing the Dirichlet Prior over the categorical distribution parameters. The task-specific customized evidential framework, equipped with meticulously designed model architecture and loss function, effectively bridges the gap between EDL theory and detection tasks. Moreover, we utilize contrastive learning as an implicit means of evidential regularization and to encourage the class separation in the latent space. Alongside, we innovatively model the background uncertainty to further improve the unknown discovery ability. Extensive experiments on benchmark datasets demonstrate the outperformance of the proposed method over existing ones.
1 October 2023 in Journal of Cataract & Refractive Surgery
Purpose: To develop deep learning-based networks for the diagnosis of diabetic retinopathy (DR) with cataracts based on infrared fundus images. Setting: Shanghai General Hospital, Shanghai Eye Disease Prevention & Treatment Center, Shanghai, China. Design: Development and evaluation of an artificial intelligence (AI) diagnostic method. Methods: A total of 10 665 infrared fundus images from 4553 patients with diabetes were used to train and test the model. For image quality assessment, left and right eye classification, DR diagnosis and grading, and segmentation of 3 DR lesions, an end-to-end software using EfficientNet and UNet was developed. The accuracy and performance of the software in comparison to human experts was evaluated. Results: The model achieved an accuracy of 75.31% for left and right eye classification, 100% for DR grading and diagnosis tasks, and 73.67% for internal test set, with corresponding areas under the curve (AUCs) of 0.88, 1.00, and 0.89, respectively. For DR lesion segmentation, the AUCs of hemorrhagic, microangioma, and exudative lesions were 0.86, 0.66, and 0.84, respectively. In addition, a contrast test of human-machine film reading confirmed the software's high sensitivity (96.3%) and specificity (90.0%) and consistency with the manual film reading group (κ = 0.869, P < .001). This easily deployable software generated reports quickly and promoted efficient DR screening with cataracts in clinical and community settings. Conclusions: AI-assisted software can perform automatic analysis of infrared fundus images and has substantial application value for the diagnosis of DR patients with cataracts.
Open set recognition (OSR) for medical images is vital to ensure practical safeguards in clinical applications. It demands establishing model awareness of rare and unknown conditions, rejecting what the model does not know, and thus preventing these conditions from being dangerously misclassified into any false known classes. Meanwhile, it often further requires training the OSR model based on a mixture of severely class-imbalanced and partially labeled data due to the prevalence of long-tailed data distribution and the massive cost of medical image annotations. Unfortunately, most existing OSR methods perform poorly under such complex conditions, largely because they neglect to thoroughly exploit the knowledge from the imbalanced and unlabeled data to help better distinguish the known and unknown lesion patterns. This paper proposes a novel Semi-supervised and Class-Imbalanced Open Set medical image Recognition (SCI-OSR) algorithm as a joint solution to address these issues. The major contributions are as follows. Firstly, we propose a modified open mixup approach boasting selective fine-grained lesion foreground fusion based on the learned weighted attentive masks to augment the unlabeled, small-sized, and even pseudo-unknown class data, where the generated attentive masks serve as effective candidate lesion area estimators. Secondly, our approach can more effectively extract the disentangled and discriminative feature representations based on our tailored consistency and contrastive losses, which helps more properly exploit the augmented data and aids in pushing the learned features of known and unknown classes to be well separated by large margins. Finally, a prototype-based open set classifier is established based on the extracted discriminative features of the original and augmented data. It favors producing equally low confidence scores over all non-positive classes and thus more reliably rejects the unknown query data. Through comparative experiments and ablation studies conducted on several popular medical benchmarks, we demonstrate that our proposed SCI-OSR can successfully boost OSR performances, leading to improved known class classification and unknown class rejection over other state-of-the-art methods.
Digital X-ray radiography is widely used in clinical diagnosis. High quality chest X-ray is conducive to the accurate diagnosis of diseases by clinicians. However, the quality assessment of the chest X-ray images mainly depends on the subjective evaluation of doctors, and the results are influenced by the skill level and experience of the evaluators, involving many issues such as heavy workload and various uncertain factors in such subjective judgment. In this paper, we propose a chest X-ray quality assessment method that combines image-text contrastive learning with medical domain knowledge fusion. Based on pretraining the model from contrastive text-image pairs, large-scale real clinical chest X-ray and diagnostic report text information are fused, and the model is fine-tuned to achieve cross domain transfer learning. While improving the prediction accuracy of the algorithm, the cost of massive sample data annotation is avoided. The local visual patch features of the X-ray images are aligned with multiple text features to ensure that the visual features contain more fine-grained image information. Theoretical analysis and experimental results show that the contrastive learning algorithm based on the fusion of triplet information in medical knowledge graph and chest X-ray multi-modal data has achieved good performance in terms of accuracy. In addition, the method proposed in this paper can be easily extended to complete other tasks such as medical image multi-lesion segmentation and disease progression prediction.
Previous works have shown that explicit snippets relationship modeling can be helpful for feature learning on untrimmed action videos. However, the snippets relationship learning in these methods are far from optimal in that they failed to consider the valuable temporally coarse-grained features, learnable soft relationship weights, and separate relationship learning in different temporal orders. To address this issue, we proposed a novel SGC-Block for improved snippet relationship learning, which enables the temporally coarse-to-fine soft valued snippet-wise relationship learning in different temporal directions. The SGC-Block constructs the snippets graph and explicitly models the (1) temporal relations (TPR); (2) coarsegrained snippet-wise relations (CSR); (3) fine-grained snippet-wise relations (FSR); and an additional (4) adaptive relations (ADR). Especially, the novel CSR is inspired by the feature pyramid pooling structure to obtain the coarse feature presentations in the temporal dimension. Experimental results showed that our proposed approach outperforms most state-of-the-art methods on the THUMOS14 and ActivityNet-1.3 benchmarks.