Multimedia-based recommendation is a challenging task that requires not only learning collaborative signals from user-item interaction, but also capturing modality-specific user interest clues from complex multimedia content. Though significant progress on this challenge has been made, we argue that current solutions remain limited by multimodal noise contamination. Specifically, a considerable proportion of multimedia content is irrelevant to the user preference, such as the background, overall layout, and brightness of images; the word order and semantic-free words in titles; etc . We take this irrelevant information as noise contamination to discover user preferences. Moreover, most recent research has been conducted by graph learning. This means that noise is diffused into the user and item representations with the message propagation; the contamination influence is further amplified. To tackle this problem, we develop a novel framework named Multimodal Graph Contrastive Learning (MGCL), which captures collaborative signals from interactions and uses visual and textual modalities to respectively extract modality-specific user preference clues. The key idea of MGCL involves two aspects: First, to alleviate noise contamination during graph learning, we construct three parallel graph convolution networks to independently generate three types of user and item representations, containing collaborative signals, visual preference clues, and textual preference clues. Second, to eliminate as much preference-independent noisy information as possible from the generated representations, we incorporate sufficient self-supervised signals into the model optimization with the help of contrastive learning, thus enhancing the expressiveness of the user and item representations. Note that MGCL is not limited to graph learning schema, but also can be applied to most matrix factorization methods. We conduct extensive experiments on three public datasets to validate the effectiveness and scalability of MGCL ^{1 1}We release the codes of MGCL at https://github.com/hfutmars/MGCL. .

Recently, user cold-start recommendations have attracted a lot of attention from industry and academia. In user cold-start recommendation systems, the user attribute information is often used by existing approaches to learn user preferences due to the unavailability of user action data. However, most existing recommendation methods often ignore the sparsity of user attributes in cold-start recommendation systems. To tackle this limitation, this article proposes a novel Inductive Heterogeneous Graph Neural Network (IHGNN) model, which utilizes the relational information in user cold-start recommendation systems to alleviate the sparsity of user attributes. Our model converts new users, items, and associated multimodal information into a Modality-aware Heterogeneous Graph (M-HG) that preserves the rich and heterogeneous relationship information among them. Specifically, to utilize rich and heterogeneous relational information in an M-HG for enriching the sparse attribute information of new users, we design a strategy based on random walk operations to collect associated neighbors of new users by multiple times sampling operation. Then, a well-designed multiple hierarchical attention aggregation model consisting of the intra- and inter-type attention aggregating module is proposed, focusing on useful connected neighbors and neglecting meaningless and noisy connected neighbors to generate high-quality representations for user cold-start recommendations. Experimental results on three real datasets demonstrate that the IHGNN outperforms the state-of-the-art baselines.

Micro-video recommendation has attracted extensive research attention with the increasing popularity of micro-video sharing platforms. Traditional approaches consider micro-video recommendation as a matching task and ignore the rich relationships among users and micro-videos from various modalities (e.g., visual, acoustic, and textual). Recently, GNN-based approaches show promising performance for the micro-video recommendation task. However, they mainly focus on the homogeneous graph which includes only one type of nodes or relations, and cannot be applied to the heterogeneous graph which consists of users, micro-videos, and related multi-modal information. In this paper, a novel Heterogeneous Hierarchical Feature Aggregation Network (HHFAN) is proposed for personalized micro-video recommendation. Our goal is to explore the highly complicated relationship information among users, micro-videos and related multi-modal information from a modality-aware Heterogeneous Information Graph (M-HIG), and thus generate high-quality user and micro-video embeddings for recommendation. The proposed model consists of two key components: (1) In data structure level, we build a heterogeneous graph and utilize a random walk based sampling strategy to sample neighbors for users and micro-videos. (2) In representation learning level, we design a hierarchical feature aggregation network including the intra- and inter-type feature aggregation networks to better capture the complex structure and rich semantic information in the heterogeneous graph. We evaluate our method on two real-world datasets and the results demonstrate that the proposed model outperforms the baseline methods.

Yifei Wang, Dizhan Xue, Shengjie Zhang, Shengsheng Qian. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

Personalized micro-video recommendation has attracted a lot of research attention with the growing popularity of micro-video sharing platforms. Many efforts have been made to consider micro-video recommendation as a matching task and shown promising performance, while they only focus on simple features or multi-modal attribute information. Recently, Graph Neural Networks (GNNs) have been employed in many recommendation tasks and achieved impressive success. However, these GNN-based methods may suffer from the following limitations: (1) fail to capture the heterogeneity of nodes in uservideo bipartite graphs; (2) ignore the non-local (global) semantic correlation information remained in heterogeneous graphs. In this paper, we present a novel approach, Heterogeneous Graph Contrastive Learning Network (HGCL), for personalized microvideo recommendation. To consider heterogeneity in user-video bipartite graphs, we first introduce a heterogeneous graph encoder network for a high-quality representation learning of users and micro-videos. Specifically, we design a random surfing model to generate node-type specific homogeneous graphs to preserve the heterogeneity. Then we propose a graph contrastive learning framework to achieve representation learning on each node-type specific homogeneous graph by maximizing the mutual information between local patches of a graph and the global representation of the entire graph. Finally, a type-crossing objective function is proposed to jointly integrate the node embeddings from different node types to facilitate high-quality representation learning. Experimental results on real-world datasets in the micro-video recommendation task validate the performance of our method, compared with state-of-the-art baseline algorithms.

With the rapid development of social media platforms and the increasing scale of the social media data, the rumor detection task has become vitally important since the authenticity of posts cannot be guaranteed. To date, Many approaches have been proposed to facilitate the rumor detection process by utilizing the multi-task learning mechanism, which aims to improve the performance of rumor detection task by leveraging the useful information in the stance detection task. However, most of the existing approaches suffer from three limitations: (1) only focus on the textual content and ignore the multi-modal information which is key component contained in social media data; (2) ignore the difference of feature space between the stance detection task and rumor detection task, resulting in the unsatisfactory usage of stance information; (3) largely neglect the semantic information hidden in the fine-grained stance labels. Therefore, in this paper, we design a Multi-modal Meta Multi-Task Learning (MM-MTL) framework for social media rumor detection. To make use of multiple modalities, we design a multi-modal post embedding layer which considers both textual and visual content. To overcome the feature-sharing problem of the stance detection task and rumor detection task, we propose a meta knowledge-sharing scheme to share some higher meta network-layers and capture the meta knowledge behind the multi-modal post. To better utilize the semantic information hidden in the fine-grained stance labels, we employ the attention mechanism to estimate the weight of each reply. Extensive experiments on two Twitter benchmark datasets demonstrate that our proposed method achieves state-of-the-art performance.