Qingyi Si

Qingyi Si

Auto-generated profile, based on publicly available information

Qingyi Si

Institute of Information Engineering Chinese Academy of Sciences Beijing China

Research Focus & Metrics

Academic Career Length

6Years (2021-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
1 January 2022
Verified
Conference Paper
Open access
PDF file
QSQingyi Si
Qingyi Si
FMFandong Meng
Fandong Meng
MZMingyu Zheng
Mingyu Zheng
ZLZheng Lin
Zheng Lin
YLYuanxin Liu
Yuanxin Liu
JZJie Zhou
Jie Zhou
Qingyi Si, Fandong Meng, Mingyu Zheng, Zheng Lin, Yuanxin Liu, Peng Fu, Yanan Cao, Weiping Wang, Jie Zhou. Findings of the Association for Computational Linguistics: EMNLP 2022. 2022.

1 January 2023
Verified
Conference Paper
Open access
PDF file
QSQingyi Si
Qingyi Si
TWTong Wang
Tong Wang
ZLZheng Lin
Zheng Lin
XZXu Zhang
Xu Zhang
YCYanan Cao
Yanan Cao
WWWeiping Wang
Weiping Wang
Qingyi Si, Tong Wang, Zheng Lin, Xu Zhang, Yanan Cao, Weiping Wang. Findings of the Association for Computational Linguistics: EMNLP 2023. 2023.

24 March 2024 in Proceedings of the AAAI Conference on Artificial Intelligence
Verified
Journal Article
Open access
PDF file
PLPeize Li
Peize Li
QSQingyi Si
Qingyi Si
PFPeng Fu
Peng Fu
ZLZheng Lin
Zheng Lin
YWYan Wang
Yan Wang
Visual question answering is a multimodal task that requires the joint comprehension of visual and textual information. However, integrating visual and textual semantics solely through attention layers is insufficient to comprehensively understand and align information from both modalities. Intuitively, object attributes can naturally serve as a bridge to unify them, which has been overlooked in previous research. In this paper, we propose a novel VQA approach from the perspective of utilizing object attribute, aiming to achieve better object-level visual-language alignment and multimodal scene understanding. Specifically, we design an attribute fusion module and a contrastive knowledge distillation module. The attribute fusion module constructs a multimodal graph neural network to fuse attributes and visual features through message passing. The enhanced object-level visual features contribute to solving fine-grained problem like counting-question. The better object-level visual-language alignment aids in understanding multimodal scenes, thereby improving the model's robustness. Furthermore, to augment scene understanding and the out-of-distribution performance, the contrastive knowledge distillation module introduces a series of implicit knowledge. We distill knowledge into attributes through contrastive loss, which further strengthens the representation learning of attribute features and facilitates visual-linguistic alignment. Intensive experiments on six datasets, COCO-QA, VQAv2, VQA-CPv2, VQA-CPv1, VQAvs and TDIUC, show the superiority of the proposed method.

23 October 2023 in ACM Transactions on Multimedia Computing, Communications, and Applications
Verified
Journal Article
Open access
YWYan Wang
Yan Wang
PLPeize Li
Peize Li
QSQingyi Si
Qingyi Si
HZHanwen Zhang
Hanwen Zhang
WZWenyu Zang
Wenyu Zang
PFPeng Fu
Peng Fu
Knowledge-based visual question answering not only needs to answer the questions based on images but also incorporates external knowledge to study reasoning in the joint space of vision and language. To bridge the gap between visual content and semantic cues, it is important to capture the question-related and semantics-rich vision-language connections. Most existing solutions model simple intra-modality relation or represent cross-modality relation using a single vector, which makes it difficult to effectively model complex connections between visual features and question features. Thus, we propose a cross-modality multiple relations learning model, aiming to better enrich cross-modality representations and construct advanced multi-modality knowledge triplets. First, we design a simple yet effective method to generate multiple relations that represent the rich cross-modality relations. The various cross-modality relations link the textual question to the related visual objects. These multi-modality triplets efficiently align the visual objects and corresponding textual answers. Second, to encourage multiple relations to better align with different semantic relations, we further formulate a novel global-local loss. The global loss enables the visual objects and corresponding textual answers close to each other through cross-modality relations in the vision-language space, and the local loss better preserves semantic diversity among multiple relations. Experimental results on the Outside Knowledge VQA and Knowledge-Routed Visual Question Reasoning datasets demonstrate that our model outperforms the state-of-the-art methods.

1 August 2021
Verified
Conference Paper
Open access
PDF file
QSQingyi Si
Qingyi Si
YLYuanxin Liu
Yuanxin Liu
PFPeng Fu
Peng Fu
ZLZheng Lin
Zheng Lin
JLJiangnan Li
Jiangnan Li
WWWeiping Wang
Weiping Wang
Electronic proceedings of IJCAI 2021

28 October 2024
Verified
Conference Paper
Open access
HJHuishan Ji
Huishan Ji
QSQingyi Si
Qingyi Si
ZLZheng Lin
Zheng Lin
WWWeiping Wang
Weiping Wang
Throughout rapid development of multimodal large language models, a crucial ingredient is a fair and accurate evaluation of their multimodal comprehension abilities. Although Visual Question Answering (VQA) could serve as a developed test field, limitations of VQA evaluation, like the inflexible pattern of Exact Match, have hindered MLLMs from demonstrating their real capability and discourage rich responses. Therefore, this paper proposes the use of semantics-based evaluators for assessing unconstrained open-ended responses on VQA datasets. As characteristics of VQA have made such evaluation significantly different from the traditional Semantic Textual Similarity (STS) task, to systematically analyze the behaviour and compare the performance of various evaluators including LLM-based ones, we propose three key properties, i.e., Alignment, Consistency and Generalization, and a corresponding dataset Assessing VQA Evaluators (AVE) to facilitate analysis. In addition, this paper proposes a Semantically Flexible VQA Evaluator (SFVE) with meticulous design based on the unique features of VQA evaluation.Experimental results verify the feasibility of model-based VQA evaluation and effectiveness of the proposed evaluator that surpasses existing semantic evaluators by a large margin. The proposed training scheme generalizes to both the BERT-like encoders and decoder-only LLM. Relaed codes and data available at https://github.com/jihuishan/flexible_evaluation_for_vqa_mm24.