26 July 2021 in IEEE Transactions on Multimedia
The development of social media enables fake news to be expressed in a multi-modal form, which is disseminated on various social platforms and brings harmful social impacts. To handle this challenge, the fake news detection task was proposed to examine whether false information is contained in multi-modal news. Existing methods exploit various approaches with cross-modal interaction and fusion, which have proven to be effective in detecting common fake news. However, although the description of multi-modal news is narrated around entities, the previously developed methods pay less attention to this characteristic. They do not explore its benefits to the detection task and underperform with respect to the detection of fake news that requires entity-centric comparisons. To make up for this omission, we explore a novel paradigm to detect fake news by aligning and fusing multi-modal entities and propose the Entity-oriented Multi-modal Alignment and Fusion network (EMAF). Our work adopts entity-centric cross-modal interaction, which can reserve semantic integrity and capture the details of multi-modal entities. Specifically, we design an Alignment module with the improved dynamic routing algorithm and introduce a Fusion module based on the comparison, the former aligns and captures the important entities and the latter compares and aggregates entity-centric features. Comparative experiments conducted on multiple public datasets, including Weibo, Twitter, and Reddit, reveal the superiority of the proposed EMAF method, and extensive analytical experiments demonstrate the effectiveness of our proposed modules.
26 December 2022 in IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing
The rapid development of Remote Sensing (RS) technology has produced massive images, which makes it difficult to obtain interpretation results by manual screening. Therefore, researchers began to develop automatic retrieval method of RS images. In recent years, cross-modal RS image retrieval based on query text has attracted many researchers because of its flexible and has become a new research trend. However, the primary problem faced is that the information of query text and RS image is not aligned. For example, RS images often have the attributes of multi-scale and multi-objective, and the amount of information is rich, while the query text contains only a few words, and the information is scarce. Recently, Graph Neural Network (GNN) has shown its potential in many tasks with its powerful feature representation ability. Therefore, based on GNN, this paper proposes a new cross-modal RS feature matching network, which can avoid the degradation of retrieval performance caused by information misalignment by learning the feature interaction in query text and RS image respectively, and modeling the feature association between the two modes. Specifically, to fuse the within-modal features, the text and RS image graph modules are designed based on GNN. In addition, in order to effectively match the query text and RS image, combined with the multi-head attention mechanism, an image-text association module is constructed to focus on the parts related to RS image in the text. The experiments on two public standard datasets verify the competitive performance of the proposed model.
1 September 2020 in Neurocomputing