To address the challenges in existing multi-sensor data fusion methods for rail surface defect diagnosis, particularly their limitations in fully exploiting potential synergistic information among multimodal data and effectively bridging the semantic gap between heterogeneous multi-source data, this paper proposes a diagnostic approach based on a Progressive Joint Representation Graph Attention Fusion Network (PJR-GAFN). The methodology comprises five principal phases: Firstly, shared and specific autoencoders are used to extract joint representations of multimodal features through shared and modality-specific representations. Secondly, a squeeze-and-excitation module is implemented to amplify defect-related features while suppressing non-essential characteristics. Thirdly, a progressive fusion module is introduced to comprehensively utilize cross-modal synergistic information during feature extraction. Fourthly, a source domain classifier and domain discriminator are employed to capture modality-invariant features across different modalities. Finally, the spatial attention aggregation properties of graph attention networks are leveraged to fuse multimodal features, thereby fully exploiting contextual semantic information. Experimental results on real-world rail surface defect datasets from domestic railway lines demonstrate that the proposed method achieves 95% diagnostic accuracy, confirming its effectiveness in rail surface defect detection.

An image–text retrieval method that integrates intramodal fine-grained local semantic information and intermodal global semantic information is proposed to address the weak fine-grained discrimination capabilities for the semantic features located between image and text modalities in cross-modal retrieval tasks. First, the original features of images and texts are extracted, and a graph attention network is employed for region relationship reasoning to obtain relation-enhanced local features. Then, an attention mechanism is used for different semantically interacting samples within the same modality, enabling comprehensive intramodal relationship learning and producing semantically enhanced image and text embeddings. Finally, a triplet loss function is used to train the entire model, and it is enhanced with an angular constraint. Through extensive comparative experiments conducted on the Flickr30K and MS-COCO benchmark datasets, the effectiveness and superiority of the proposed method were verified. It outperformed the current method by 6.4% relatively for image retrieval and 1.3% relatively for caption retrieval on MS-COCO (Recall@1 using the 1K test set).