Xiaolin Chen

Xiaolin Chen

Auto-generated profile, based on publicly available information

Xiaolin Chen

School of Software, Joint SDU-NTU Centre for Artificial Intelligence Research, Shandong University, Jinan, China

Research Focus & Metrics

Academic Career Length

7Years (2020-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
10 July 2024
Verified
Conference Paper
Open access
HWHaokun Wen
Haokun Wen
XSXuemeng Song
Xuemeng Song
XCXiaolin Chen
Xiaolin Chen
YWYinwei Wei
Yinwei Wei
LNLiqiang Nie
Liqiang Nie
TCTat-Seng Chua
Tat-Seng Chua
Composed image retrieval (CIR) aims to retrieve the target image based on a multimodal query, i.e., a reference image paired with corresponding modification text. Recent CIR studies leverage vision-language pre-trained (VLP) methods as the feature extraction backbone and perform nonlinear feature-level multimodal query fusion to retrieve the target image. Despite the promising performance, we argue that their nonlinear feature-level multimodal fusion may lead to the fused feature deviating from the original embedding space, potentially hurting the retrieval performance. To address this issue, in this work, we propose shifting the multimodal fusion from the feature level to the raw-data level to fully exploit the VLP model's multimodal encoding and cross-modal alignment abilities. In particular, we introduce a Dual Query Unification-based Composed Image Retrieval framework (DQU-CIR), whose backbone simply involves a VLP model's image encoder and a text encoder. Specifically, DQU-CIR first employs two training-free query unification components to derive a unified textual and visual query based on the raw data of the multimodal query, respectively. The unified textual query is derived by concatenating the modification text with the extracted reference image's textual description, while the unified visual query is created by writing the key modification words onto the reference image. Ultimately, to address diverse search intentions, DQU-CIR linearly combines the features of the two unified queries encoded by the VLP model to retrieve the target image. Extensive experiments on four real-world datasets validate the effectiveness of our proposed method.

7 November 2023 in ACM Transactions on Information Systems
Verified
Journal Article
Open access
XCXiaolin Chen
Xiaolin Chen
XSXuemeng Song
Xuemeng Song
LJLiqiang Jing
Liqiang Jing
SLShuo Li
Shuo Li
Linmei HuLinmei Hu
Linmei Hu
LNLiqiang Nie
Liqiang Nie
Text response generation for multimodal task-oriented dialog systems, which aims to generate the proper text response given the multimodal context, is an essential yet challenging task. Although existing efforts have achieved compelling success, they still suffer from two pivotal limitations: (1) overlook the benefit of generative pretraining and (2) ignore the textual context-related knowledge . To address these limitations, we propose a novel dual knowledge-enhanced generative pretrained language mode for multimodal task-oriented dialog systems (DKMD), consisting of three key components: dual knowledge selection , dual knowledge-enhanced context learning , and knowledge-enhanced response generation . To be specific, the dual knowledge selection component aims to select the related knowledge according to both textual and visual modalities of the given context. Thereafter, the dual knowledge-enhanced context learning component targets seamlessly, integrating the selected knowledge into the multimodal context learning from both global and local perspectives, where the cross-modal semantic relation is also explored. Moreover, the knowledge-enhanced response generation component comprises a revised BART decoder, where an additional dot-product knowledge-decoder attention sub-layer is introduced for explicitly utilizing the knowledge to advance the text response generation. Extensive experiments on a public dataset verify the superiority of the proposed DKMD over state-of-the-art competitors.

2 November 2020 in IEEE Transactions on Multimedia
Verified
Journal Article
Closed access
XCXiaolin Chen
Xiaolin Chen
XSXuemeng Song
Xuemeng Song
SCSiwei Cui
Siwei Cui
TGTian Gan
Tian Gan
ZCZhiyong Cheng
Zhiyong Cheng
LNLiqiang Nie
Liqiang Nie
In this paper, we work towards linking users' identities on different social media platforms by exploring the user-generated contents (UGCs). This task is non-trivial due to the following challenges. 1) As UGCs involve multiple modalities (e.g., text and image), how to accurately characterize the user account based on their heterogeneous multi-modal UGCs poses the main challenge. 2) As people tend to post similar UGCs on different social media platforms during the same period, how to effectively model the temporal post correlation is a crucial challenge. And 3) no public benchmark dataset is available to support our user identity linkage based on heterogeneous UGCs. Towards this end, we present an attentive time-aware user identity linkage scheme, which seamlessly integrates the temporal post correlation modeling and attentive user similarity modeling. To facilitate the evaluation, we create a comprehensive large-scale user identity linkage dataset consisting of 5,765 users with 1,729,500 UGCs on Twitter and Instagram. Extensive experiments have been conducted on our dataset and the results verify the effectiveness of the proposed scheme.

9 December 2021 in IEEE Transactions on Multimedia
Verified
Journal Article
Closed access
XSXuemeng Song
Xuemeng Song
SFShi-Ting Fang
Shi-Ting Fang
XCXiaolin Chen
Xiaolin Chen
YWYinwei Wei
Yinwei Wei
ZZZhongzhou Zhao
Zhongzhou Zhao
LNLiqiang Nie
Liqiang Nie
Outfit compatibility modeling, which aims to automatically evaluate the matching degree of an outfit, has drawn great research attention. Regarding the comprehensive evaluation, several previous studies have attempted to solve the task of the outfit compatibility modeling by integrating the multi-modal information of fashion items. However, these methods primarily focus on fusing the visual and textual modalities, but seldom consider the category modality as an essential modality. In addition, they mainly focus on the exploration of the intra-modal compatibility relation among fashion items in an outfit, but ignore the importance of the inter-modal compatibility relation, i.e., the compatibility across different modalities between fashion items. Due to the fact that each modality of the item could deliver the same characteristics of the item as other modalities [1], [2], as well as certain exclusive features of the item, overlooking the inter-modal compatibility could yield sub-optimal performance. To address these issues, a multi-modal outfit compatibility modeling scheme with modality-oriented graph learning is proposed, dubbed as MOCM-MGL, which takes both the visual, textual and category modalities as input and jointly propagates the intra-modal and inter-modal compatibilities among fashion items. Experimental results on the real-world Polyvore Outfits-ND and Polyvore Outfits-D datasets have demonstrated the superiority of our proposed model over existing methods.

26 October 2023
Verified
Conference Paper
Open access
ZLZhenyang Li
Zhenyang Li
YGYangyang Guo
Yangyang Guo
KWKejie Wang
Kejie Wang
XCXiaolin Chen
Xiaolin Chen
LNLiqiang Nie
Liqiang Nie
MKMohan Kankanhalli
Mohan Kankanhalli

18 July 2023
Verified
Conference Paper
Open access
XCXiaolin Chen
Xiaolin Chen
XSXuemeng Song
Xuemeng Song
YWYinwei Wei
Yinwei Wei
LNLiqiang Nie
Liqiang Nie
TCTat-Seng Chua
Tat-Seng Chua
Textual response generation is an essential task for multimodal task-oriented dialog systems. Although existing studies have achieved fruitful progress, they still suffer from two critical limitations: 1) focusing on the attribute knowledge but ignoring the relation knowledge that can reveal the correlations between different entities and hence promote the response generation, and 2)only conducting the cross-entropy loss based output-level supervision but lacking the representation-level regularization. To address these limitations, we devise a novel multimodal task-oriented dialog system (named MDS-S2). Specifically, MDS-S2 first simultaneously acquires the context related attribute and relation knowledge from the knowledge base, whereby the non-intuitive relation knowledge is extracted by the n-hop graph walk. Thereafter, considering that the attribute knowledge and relation knowledge can benefit the responding to different levels of questions, we design a multi-level knowledge composition module in MDS-S^2 to obtain the latent composed response representation. Moreover, we devise a set of latent query variables to distill the semantic information from the composed response representation and the ground truth response representation, respectively, and thus conduct the representation-level semantic regularization. Extensive experiments on a public dataset have verified the superiority of our proposed MDS-S2. We have released the codes and parameters to facilitate the research community.