Yifei Gao

Yifei Gao

Auto-generated profile, based on publicly available information

Yifei Gao

Tianjin University, Tianjin, China

Research Focus & Metrics

Academic Career Length

3Years (2024-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
12 April 2024 in IEEE Transactions on Knowledge and Data Engineering
Verified
Journal Article
Closed access
NXNing Xu
Ning Xu
YGYifei Gao
Yifei Gao
ALAnan Liu
Anan Liu
HTHongshuo Tian
Hongshuo Tian
YZYongdong Zhang
Yongdong Zhang
Knowledge-based Visual Question Answering (KB-VQA) aims to answer the image-aware question via the external knowledge, which requires an agent to not only understand images but also explicitly retrieve and integrate knowledge facts. Intuitively, to accurately answer the question, we humans can validate the retrieved knowledge based on our memory, and then align the knowledge facts with the image regions to infer answers. However, most existing methods ignore the process of knowledge validation and alignment. In this paper, we propose the Multi-Modal Validation and Domain Interaction Learning method, which consists of two components: 1) Multi-modal validation for knowledge retrieval. We propose the multi-modal validation module (MMV) to evaluate the confidence of each retrieved knowledge fact via images and questions, which preserves knowledge candidates effective for inferring answers. 2) Domain interaction for knowledge integration. We propose the Domain Interaction TRansformer module (DI-TR) to align visual regions with knowledge facts by the interaction learning in the improved transformer. Specifically, the inter-domain and intra-domain masks are injected into each self-attention layer to control the integration scope. The proposed method outperforms several strong baselines on three widely-used knowledge-based datasets: KRVQA, OK-VQA and VQA2.0. Extensive experiments and ablation studies demonstrate the effectiveness of multi-modal knowledge validation and domain interaction learning.

28 October 2024
Verified
Conference Paper
Open access
NXNing Xu
Ning Xu
YGYifei Gao
Yifei Gao
TZTing-Ting Zhang
Ting-Ting Zhang
HTHongshuo Tian
Hongshuo Tian
ALAnan Liu
Anan Liu
News Captioning involves generating the descriptions for news images based on the detailed content of related news articles. Given that these articles often contain extensive information not directly related to the image, captions may end up misaligned with the visual content. To mitigate this issue, we propose the novel cross-modal coherence-enhanced feedback prompting method to clarify the crucial elements that align closely with the visual content for news captioning. Specifically, we first adapt CLIP to develop a news-specific image-text matching module, enriched with insights from language model MPNet using a matching-score comparative loss, which facilitates effective cross-modal knowledge distillation. This module enhance the coherence between images and each news sentences via rating confidence. Then, we design confidence-aware prompts to fine-tune LLaVA model with by LoRa strategy, focusing on essential details in extensive articles. Lastly, we evaluate the generated news caption with refined CLIP, constructing confidence-feedback prompts to further enhance LLaVA through feedback learning, which iteratively refine captions to improve its accuracy. Extensive experiments conduct on two public datasets, GoodNews and NYTimes800k, have validated the effectiveness of our method.


1 January 2026 in IEEE Transactions on Circuits and Systems for Video Technology
Verified
Journal Article
Closed access
YGYifei Gao
Yifei Gao
NXNing Xu
Ning Xu
HTHongshuo Tian
Hongshuo Tian
SWSha Wei
Sha Wei
ALAnan Liu
Anan Liu