21 June 2022 in Computers & Operations Research
27 December 2024 in IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing
Currently, RGB-thermal (RGB-T) object detection algorithms have demonstrated excellent performance, but issues such as modality failure caused by fog, strong light, sensor damage, and other conditions can significantly impact the detector's performance. This paper proposes a multimodal object detection method named diffusion enhanced object detection network (DENet), aiming to address modality failure problems caused by non-routine environments, sensor anomalies, and other factors, while suppressing redundant information in multimodal data to improve model accuracy. Firstly, we design a multidimensional incremental information generation module based on a diffusion model, which reconstructs the unstable information of RGB-T images through the reverse diffusion process using the original fusion feature map. To further address the issue of redundant information in existing RGB-T object detection models, a redundant information suppression module is introduced, minimizing cross-modal redundant information based on mutual information and contrastive loss. Finally, a kernel similarity-aware illumination module (KSIM) is introduced to dynamically adjusts the weighting of RGB and thermal features by incorporating both illumination intensity and the similarity between modalities. KSIM can fine-tunes the contribution of each modality during fusion, ensuring a more precise balance that improves recognition performance across diverse conditions. Experimental results on the DroneVehicle and VEDAI datasets show that DENet performs outstandingly in multimodal object detection tasks, effectively improving detection accuracy and reducing the impact of modality failure on performance.
22 December 2025 in IEEE Transactions on Geoscience and Remote Sensing
The current mainstream research on color-thermal (i.e., RGB-T) object detection assumes that the RGB and thermal images are strictly aligned. However, in practical situations, due to the insufficient spatiotemporal synchronization, stereo disparity of the camera in the installation position, and the errors in the image pairs registration process, the position of the same objects in RGB and thermal images are not completely overlapped. The position shift can cause distortion, trailing, and blurring issues during the image fusion process, leading to a decrease in model detection accuracy. To address this challenge, we propose a novel multiscale graph spatial alignment network (MGSANet), which can effectively alleviate the negative effects of cross-modal image misalignment. Specifically, we represent the feature maps extracted from RGB and thermal images by backbone network as a graph structure, and use graph attention network (GAT) to model the spatial position deviation relationship. Furthermore, considering the multiscale characteristics of the objects, we represent the feature maps with multiscale graphs. We then align RGB and thermal feature maps in a potential feature space according to the learned deviation relationship for object detection. In addition, considering the scarcity of RGB-T datasets from the perspective of unmanned aerial vehicle (UAV), and to verify the object detection performance on different platforms, we construct an RGB-T object detection dataset collected by the UAV platform, named KUSTDrone. We conducted experiments on datasets collected by vehicle and UAV platforms respectively. Experimental results demonstrate that MGSANet outperforms the competitive methods for weakly aligned RGB-T object detection. The dataset will be accessible at https://github.com/KustTeamWQW/KUSTDrone with a license, and the code will also be accessible at https://github.com/KustTeamWQW/MGSANet.