3 November 2022 in IEEE Access
Multimodal Sentiment Analysis (MSA) is an emerging research field that aims to identify the sentiment of speakers through Audio (A), Video (V), and Text (T) modalities. The major challenge is to capture joint representations that can associate and integrate information from various modalities. Most of the existing methods are prone to acquiring joint representation through the concatenation of input features. However, these methods are short of exploiting interactions fully to ensure consistency and complementarity among modalities. To solve this problem, we design a novel multimodal sentiment analysis framework named Cross-Modal Joint Representation Transformer (CMJRT), which exploits hierarchical interactions among modalities by passing joint representations from bimodality to unimodality. Specifically, we adopt cyclic translation to obtain joint representations of bimodality, where one modality is translated to the other modality forward and backward by encoder-decoders. The translation process ensures consistency between modalities. In addition, to explore complementarity among modalities, the cross-modal transformer is used to reinforce each unimodality with common information from bimodality. Extensive experiments on CMU-MOSI and CMU-MOSEI datasets demonstrate that our proposed method outperforms existing approaches.
22 February 2023 in IEEE Journal on Miniaturization for Air and Space Systems
Unmanned aerial vehicle (UAV) synthetic aperture radar (SAR) plays an important role in modern remote sensing for its characteristics of all weather, all day-and-night, zero casualty, flying flexibility, and low cost. However, the atmospheric turbulence will cause motion errors to UAV SAR, resulting in unmodeled phase errors. The phase errors will degrade the focusing quality of the image and bring difficulties to the recognition task. Meanwhile, it is difficult for a convolution neural network (CNN) to extract and utilize the back-scattering information for target recognition. To this end, a novel defocusing adaptive complex CNN (DA-CCNN) is proposed, which can realize the overall computation of the network in the complex-valued data domain and effectively extract the phase history information of the complex-valued data. Furthermore, it is the first time that the image entropy metric is introduced into the fully complex deep neural network to improve the focusing quality of the image and the interpretability of the network. The experiment is carried out using the benchmark dataset of MSTAR 10. In order to simulate the defocused images generated by UAV SAR and certify the robustness to phase errors, datasets with the contamination are also applied. The results show that on the benchmark data, the recognition accuracy of DA-CCNN is comparable to that of the existing methods. On the data with phase errors, DA-CCNN shows stronger robustness and higher accuracy in terms of recognition than the reported networks.