22 September 2025 in IEEE Transactions on Affective Computing
Multimodal sentiment analysis (MSA) has exhibited superior performance compared to unimodal sentiment analysis. However, in real-world scenarios, multimodal data are prone to being incomplete due to various interferences. To address this issue, this paper combines cross-modal adaptive knowledge information with adversarial reconstruction of complete-incomplete multimodal representations, proposing a novel neural network model called Dynamic Information Transfer-Adversarial Learning (DIT-AL) for robust MSA of unaligned modal sequences with missing data. First, the model employs a symmetric structurebased attention feature fusion to facilitate local and global feature interactions across modalities, learning multimodal representations for complete-incomplete data pairs. Then, two mechanisms are utilized simultaneously to recover the semantic information of the missing data. The first mechanism dynamically learns the relative relationships between complete and incomplete modal representations through an adaptive information distillation module, bridging the semantic gaps caused by missing modal data and enhancing the representation capacity of incomplete multimodal representations. The second mechanism introduces adversarial learning and feature reconstruction based on paired multimodal representations to generate semantic information for incomplete data, thereby achieving consistency in the feature distribution between complete and incomplete data. Experimental results on three public benchmarks (CMU-MOSI, CMU-MOSEI, and CHSIMS) demonstrate that the proposed method can effectively improve the robustness of MSA under random data missing conditions by using adaptive information distillation combined with adversarial reconstruction. Compared with state-of-the-art methods, it achieves higher recognition accuracy for robust MSA
25 February 2026 in Digital Signal Processing
25 June 2024 in IEEE MultiMedia
Scene recognition is a challenging task in computer vision because of the diversity of objects in scene images and the ambiguity of object layouts. In recent years, the emergence of multimodal scene data has provided new solutions for scene recognition, but it has also brought new problems. To address these challenges, the Self-Attention and Distillation based Multimodal Scene Recognition Network (SAD-MSR) is proposed in this paper. The backbone of the model adopts the pure transformer structure of self-attention, which can extract local and global spatial features of multimodal scene images. A multi-stage fusion mechanism was developed for this model, in which the concatenated tokens of two modalities are fused based on self-attention in the early stage, while the high-level features extracted from the two modalities are fused based on cross-attention in the late stage. Furthermore, a distillation mechanism is introduced to alleviate the problem of a limited number of training samples. Finally, we conducted extensive experiments on two multimodal scene recognition databases, SUN RGB-D and NYUD2, to show the effectiveness of SAD-MSR. Compared with other state-of-the-art multimodal scene recognition methods, our method can achieve better experimental results.