17 September 2024 in Engineering Applications of Artificial Intelligence
7 March 2025 in Expert Systems with Applications
24 November 2023 in Measurement
6 February 2026 in IEEE Robotics and Automation Letters
Infrared (IR) and visible (VIS) image pairs suffer from position offset of the same object across different modalities in practice. Related work focus on improving detection performance through feature alignment and fusion, ignoring object accurate localization in different modalities, while complementary information at the object level can help to further explain and judge the object. To address this, we propose a multi-spectral pedestrian detection method based on YOLO, featuring explicit offset learning for both feature alignment and object offset prediction. First, an Adaptive Multi-scale Mask Fusion (AMMF) module is designed to enhance features by learning to dynamically fuse mask predictions from Feature Pyramid Network (FPN) in both modalities. Then, a Region-Aware Supervised Feature Alignment (RASFA) module is proposed with a symmetric design. This module simultaneously predicts both IR-to-VIS and VIS-to-IR offset fields within one efficient framework. The former enables robust feature alignment supervised on target regions, while the latter directly provides object-level offsets. As a result, the detection head can efficiently output IR detections by applying these readily available offsets to the VIS detections, eliminating the need for a separate offset prediction branch. Experiments on three public multispectral pedestrian datasets demonstrate that our method not only improves detection performance but also achieves accurate localization of different modalities, outperforming previous state-of-the-art methods.
12 June 2026 in IEEE Sensors Journal
Autonomous train operation requires vehicles to be equipped with reliable perception systems for foreign object detection to ensure operational safety. However, the substantial heterogeneity of foreign objects, together with the difficulty of collecting intrusion data samples, poses a major challenge to achieving accurate and robust intrusion perception. To overcome this challenge, we propose a camera-LiDAR fusion framework capable of achieving accurate segmentation of abnormal intruding objects by relying solely on real normal data and simulated anomalous data for training. Specifically, we propose a multi-level fusion module based on normalizing flows that maps image and point cloud features into the shallow latent space, enhancing the consistency of their feature representations. Then, features from different sensors are fused to accurately model the feature distribution of normal scenes, thereby enabling the preliminary localization of intrusive foreign objects and the extraction of their corresponding features when foreign objects are present in the scene. Further, an asymmetric cross-modal transformer block is designed to compute attention scores over fused multimodal features, which can guide image feature extraction to achieve iterative aggregation of intrusion target features. The final aggregated features are fed into a segmentation predictor to obtain perception results. Moreover, we build RailMFOD, the first multimodal dataset designed for foreign object perception in open railway environments. Extensive experimental results demonstrate that the proposed method can accurately segment intruding foreign objects, achieving an IoU of 63.91% while maintaining real-time inference efficiency. These results validate the practical applicability of our method for ensuring safe autonomous railway operations. The RailMFOD dataset can be obtained from https://github.com/sunnytt98/RailMFOD.