The detection of lesions in various diseases remains a challenging task in medical image processing, given the diverse morphologies, sizes, and boundaries of lesions associated with different illnesses. In this paper, we propose an advanced lesion detection model named DHC-YOLO, which integrates Multi-Scale Dilated attention (MSDA) and multi-head self-attention (MHSA) within the YOLOv8 network. The method also introduces an enhanced feature fusion through the Concatenation (Concat) operation in the Feature Pyramid Networks (FPN) structure of YOLOv8. The DHC-YOLO model achieves superior performance in lesion detection by effectively aggregating semantic information across various scales in the attended receptive field, reducing redundancy in self-attention mechanisms without the need for complex operations or additional computational costs. The incorporation of MHSA enhances the network’s ability to extract diverse features, and the Concat operation in FPN improves multi-scale feature fusion. Our evaluations on brain tumor, colonic polyp, and esophageal cancer datasets demonstrate the superiority of our method over baseline YOLOv8 and several state-of-the-art object detection models. Specifically, on the brain tumor dataset, DHC-YOLO achieves mAP50 and mAP50:95 scores of 88.3% and 73.5%, respectively; on the colonic polyp dataset, the scores are 88.8% and 67.2%; and on the esophageal cancer dataset, the scores are 51.3% and 20.7%. These compelling results underscore the robust performance of DHC-YOLO in lesion detection tasks.

Esophageal cancer is a globally significant but understudied type of cancer with high mortality rates. The staging and differentiation of esophageal cancer are crucial factors in determining the prognosis and surgical treatment plan for patients, as well as improving their chances of survival. Endoscopy and histopathological examination are considered as the gold standard for esophageal cancer diagnosis. However, some previous studies have employed deep learning-based methods for esophageal cancer analysis, which are limited to single-modal features, resulting in inadequate classification results. In response to these limitations, multi-modal learning has emerged as a promising alternative for medical image analysis tasks. In this paper, we propose a hierarchical feature fusion network, MM-HiFuse, for multi-modal multitask learning to improve the classification accuracy of esophageal cancer staging and differentiation level. The proposed architecture combines low-level to deep-level features of both pathological and endoscopic images to achieve accurate classification results. The key characteristics of MM-HiFuse include: (i) a parallel hierarchy of convolution and self-attention layers specifically designed for pathological and endoscopic image features; (ii) a multi-modal hierarchical feature fusion module (MHF) and a new multitask weighted combination loss function. The benefits of these features are the effective extraction of multi-modal representations at different semantic scales and the mutual complementarity of the multitask learning, leading to improved classification performance. Experimental results demonstrate that MM-HiFuse outperforms single-modal methods in esophageal cancer staging and differentiation classification. Our findings provide evidence for the early diagnosis and accurate staging of esophageal cancer and serve as a new inspiration for the application of multi-modal multitask learning in medical image analysis. Code is available at https://github.com/huoxiangzuo/MM-HiFuse.