Fang Peng

Fang Peng

Auto-generated profile, based on publicly available information

Fang Peng

State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences, Beijing, China

Research Focus & Metrics

Academic Career Length

9Years (2018-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
7 September 2023 in IEEE Transactions on Multimedia
Verified
Journal Article
Closed access
FPFang Peng
Fang Peng
XYXiaoshan Yang
Xiaoshan Yang
LXLinhui Xiao
Linhui Xiao
YWYaowei Wang
Yaowei Wang
CXChangsheng Xu
Changsheng Xu
Although significant progress has been made in few-shot learning, most of existing few-shot image classification methods require supervised pre-training on a large amount of samples of base classes, which limits their generalization ability in real world application. Recently, large-scale Vision-Language Pre-trained models (VLPs) have been gaining increasing attention in few-shot learning because they can provide a new paradigm for transferable visual representation learning with easily available text on the Web. However, the VLPs may neglect detailed visual information that is difficult to describe by language sentences, but important for learning an effective classifier to distinguish different images. To address the above problem, we propose a new framework, named Semantic-guided Visual Adapting (SgVA), which can effectively extend vision-language pre-trained models to produce discriminative adapted visual features by comprehensively using an implicit knowledge distillation, a vision-specific contrastive loss, and a cross-modal contrastive loss. The implicit knowledge distillation is designed to transfer the fine-grained cross-modal knowledge to guide the updating of the vision adapter. State-of-the-art results on 13 datasets demonstrate that the adapted visual features can well complement the cross-modal features to improve few-shot image classification.

2 October 2023 in IEEE Transactions on Multimedia
Verified
Journal Article
Closed access
LXLinhui Xiao
Linhui Xiao
XYXiaoshan Yang
Xiaoshan Yang
FPFang Peng
Fang Peng
MYMing Yan
Ming Yan
YWYaowei Wang
Yaowei Wang
CXChangsheng Xu
Changsheng Xu
Visual Grounding (VG) is a crucial topic in the field of vision and language, which involves locating a specific region described by expressions within an image. To reduce the reliance on manually labeled data, unsupervised methods have been developed to locate regions using pseudo-labels. However, the performance of existing unsupervised methods is highly dependent on the quality of pseudo-labels and these methods always encounter issues with limited diversity. In order to utilize vision and language pre-trained models to address the grounding problem, and reasonably take advantage of pseudo-labels, we propose CLIP-VG, a novel method that can conduct self-paced curriculum adapting of CLIP with pseudo-language labels. We propose a simple yet efficient end-to-end network architecture to realize the transfer of CLIP to the visual grounding. Based on the CLIP-based architecture, we further propose single-source and multi-source curriculum adapting algorithms, which can progressively find more reliable pseudo-labels to learn an optimal model, thereby achieving a balance between reliability and diversity for the pseudo-language labels. Our method outperforms the current state-of-the-art unsupervised method by a significant margin on RefCOCO/+/g datasets in both single-source and multi-source scenarios, with improvements ranging from 6.78% to 10.67% and 11.39% to 14.87%, respectively. Furthermore, our approach even outperforms existing weakly supervised methods. The code and models are available at https://github.com/linhuixiao/CLIP-VG .

1 January 2018 in IFAC-PapersOnLine
Verified
Journal Article
Closed access
ZLZhenbo Li
Zhenbo Li
FPFang Peng
Fang Peng
BNBingshan Niu
Bingshan Niu
GLGuangyao Li
Guangyao Li
JWJing Wu
Jing Wu
ZMZheng Miao
Zheng Miao

28 October 2024
Verified
Conference Paper
Open access
LXLinhui Xiao
Linhui Xiao
XYXiaoshan Yang
Xiaoshan Yang
FPFang Peng
Fang Peng
YWYaowei Wang
Yaowei Wang
CXChangsheng Xu
Changsheng Xu
Visual grounding, which aims to ground a visual region via natural language, is a task that heavily relies on cross-modal alignment. Existing works utilized uni-modal pre-trained models to transfer visual or linguistic knowledge separately while ignoring the multimodal corresponding information. Motivated by recent advancements in contrastive language-image pre-training and low-rank adaptation (LoRA) methods, we aim to solve the grounding task based on multimodal pre-training. However, there exists significant task gaps between pre-training and grounding. Therefore, to address these gaps, we propose a concise and efficient hierarchical multimodal fine-grained modulation framework, namely HiVG. Specifically, HiVG consists of a multi-layer adaptive cross-modal bridge and a hierarchical multimodal low-rank adaptation (HiLoRA) paradigm. The cross-modal bridge can address the inconsistency between visual features and those required for grounding, and establish a connection between multi-level visual and text features. HiLoRA prevents the accumulation of perceptual errors by adapting the cross-modal features from shallow to deep layers in a hierarchical manner. Experimental results on five datasets demonstrate the effectiveness of our approach and showcase the significant grounding capabilities as well as promising energy efficiency advantages. The project page: https://github.com/linhuixiao/HiVG.

26 January 2025 in Materials Science and Engineering: A
Verified
Journal Article
Closed access
FPFang Chao Peng
Fang Chao Peng
CGchunhuan Guo
chunhuan Guo
FJFengchun Jiang
Fengchun Jiang
DYDing Yuan
Ding Yuan
HYHanyu Yin
Hanyu Yin
Sergey KonovalovSergey Konovalov
Sergey Konovalov

14 November 2025 in IEEE Transactions on Multimedia
Verified
Journal Article
Closed access
FPFang Peng
Fang Peng
XYXiaoshan Yang
Xiaoshan Yang
YWYaowei Wang
Yaowei Wang
CXChangsheng Xu
Changsheng Xu
Few-shot action recognition is a crucial task for mitigating the challenges of data scarcity in video understanding. Recent advancements in large-scale pre-trained models have introduced the potential of incorporating semantic knowledge from multi-modal pre-trained models, such as CLIP, to alleviate these challenges. Although some progress have been made, existing methods still rely on class-level text embeddings that are inherently low in diversity, limiting their ability to generalize to unseen actions. To overcome this limitation, we propose a novel framework called Progressive Learning of Instance-Level Proxy Semantics (ProLIPS). ProLIPS integrates Proxy Semantic Diffusion (PSD) to generate rich, instance-level proxy semantic features with diverse semantic contents and temporal dynamics, utilizing a multi-step CLIP-guidance mechanism and a time-conditioned reverse diffusion process. Our approach preserves the diversity of semantic-aligned visual features, significantly improving the generalization and robustness of few-shot action recognition. Extensive experiments on five challenging benchmarks demonstrate the effectiveness of ProLIPS.