Zijia Zhao

Zijia Zhao

Auto-generated profile, based on publicly available information

Zijia Zhao

Institute of Automation, Chinese Academy of Sciences & School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China

Research Focus & Metrics

Academic Career Length

4Years (2023-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
18 July 2023
Verified
Conference Paper
Open access
ZZZijia Zhao
Zijia Zhao
LGLongteng Guo
Longteng Guo
XHXingjian He
Xingjian He
SSShuai Shao
Shuai Shao
ZYZehuan Yuan
Zehuan Yuan
JLJing Liu
Jing Liu
Multimodal representation learning has shown promising improvements on various vision-language tasks (e.g., image-text retrieval, visual question answering, etc) and has significantly advanced the development of multimedia information systems. Most existing methods excel at building global-level alignment between vision and language while lacking effective fine-grained image-text interaction. In this paper, we propose a jointly masked multimodal modeling method to learn fine-grained multimodal representations. Our method performs joint masking on image-text input and integrates both implicit and explicit targets for the masked signals to recover. The implicit target provides a unified and debiased objective for vision and language, where the model predicts latent multimodal representations of the unmasked input. The explicit target further enriches the multimodal representations by recovering high-level and semantically meaningful information: momentum visual features of image patches and concepts of word tokens. Through such a masked modeling process, our model not only learns fine-grained multimodal interaction, but also avoids the semantic gap between high-level representations and low-or mid-level prediction targets (e.g., image pixels, discrete vision tokens), thus producing semantically rich multimodal representations that perform well on both zero-shot and fine-tuned settings. Our pre-trained model (named MAMO) achieves state-of-the-art performance on various downstream vision-language tasks, including image-text retrieval, visual question answering, visual reasoning, and weakly-supervised visual grounding.

23 April 2025 in Pattern Recognition
Verified
Journal Article
Closed access
ZZZijia Zhao
Zijia Zhao
LGLongteng Guo
Longteng Guo
TYTongtian Yue
Tongtian Yue
EHErdong Hu
Erdong Hu
SSShuai Shao
Shuai Shao
JLJing Liu
Jing Liu

28 October 2024
Verified
Conference Paper
Open access
SLShichen Lu
Shichen Lu
LGLongteng Guo
Longteng Guo
WWWenxuan Wang
Wenxuan Wang
ZZZijia Zhao
Zijia Zhao
TYTongtian Yue
Tongtian Yue
SLSi Liu
Si Liu
Recently, large vision language models (LVLMs) have advanced AI by integrating visual and linguistic data for tasks like visual conversation, image captioning, and visual question answering. Current LVLM research either scales up model size for performance or reduces parameters for limited computational resources. We believe both large and tiny models have unique strengths and that collaborative training yields better results than independent training. We propose Collaborative Training of Tiny-Large Vision Language Models (CTVLMs), a framework connecting large and tiny models via a projection layer and leveraging a synergistic training strategy. Our framework improves training efficiency by strengthening the interconnection between large and tiny models. Using the parameter efficiency of tiny models, we effectively align image-text features, then apply knowledge distillation to help large models better align cross-modal information. During fine-tuning, the large model's extensive knowledge enhances tiny model's performance. This collaborative approach allows models to adapt to various computational resources and outperforms existing methods in vision-language tasks.