Xiu Yan

Xiu Yan

Auto-generated profile, based on publicly available information

Xiu Yan

Meituan Group, Beijing, China

Research Focus & Metrics

Academic Career Length

4Years (2023-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
26 August 2024 in IEEE Transactions on Neural Networks and Learning Systems
Verified
Journal Article
Closed access
YZYao Zhu
Yao Zhu
YCYuefeng Chen
Yuefeng Chen
XMXiaofeng Mao
Xiaofeng Mao
XYXiu Yan
Xiu Yan
YWYue Wang
Yue Wang
XJXiangyang Ji
Xiangyang Ji
Learning generalized representations from limited training samples is crucial for applying deep neural networks in low-resource scenarios. Recently, methods based on contrastive language-image pretraining (CLIP) have exhibited promising performance in few-shot adaptation tasks. To avoid catastrophic forgetting and overfitting caused by few-shot fine-tuning, existing works usually freeze the parameters of CLIP pretrained on large-scale datasets, overlooking the possibility that some parameters might not be suitable for downstream tasks. To this end, we revisit CLIP’s visual encoder with a specific focus on its distinctive attention pooling layer, which performs a spatial weighted-sum of the dense feature maps. Given that dense feature maps contain meaningful semantic information, and different semantics hold varying importance for diverse downstream tasks (such as prioritizing semantics like ears and eyes in pet classification tasks rather than side mirrors), using the same weighted-sum operation for dense features across different few-shot tasks might not be appropriate. Hence, we propose fine-tuning the parameters of the attention pooling layer during the training process to encourage the model to focus on task-specific semantics. In the inference process, we perform residual blending between the features pooled by the fine-tuned and the original attention pooling layers to incorporate both the few-shot knowledge and the pretrained CLIP’s prior knowledge. We term this method as semantic-aware fine-tuning (). is effective in enhancing the conventional few-shot CLIP and is compatible with the existing adapter approach (termed ). Extensive experiments on 11 benchmarks demonstrate that both and significantly outperform the second-best method by $+$ 1.51 $\%$ and $+$ 2.38 $\%$ in the one-shot setting and by $+$ 0.48 $\%$ and $+$ 1.37 $\%$ in the four-shot setting, respectively.

1 January 2024
Verified
Conference Paper
Open access
PDF file
PSPeng Sun
Peng Sun
YZYao Zhu
Yao Zhu
YZYunjian Zhang
Yunjian Zhang
XYXiu Yan
Xiu Yan
ZWZizhe Wang
Zizhe Wang
XJXiangyang Ji
Xiangyang Ji
Peng Sun, Yao Zhu, Yunjian Zhang, Xiu Yan, Zizhe Wang, Xiangyang Ji. Findings of the Association for Computational Linguistics: EMNLP 2024. 2024.

5 February 2025 in International Journal of Computer Vision
Verified
Journal Article
Closed access
PDF file
YZYao Zhu
Yao Zhu
XYXiu Yan
Xiu Yan
CXChuanlong Xie
Chuanlong Xie

7 December 2023 in arXiv
Verified
Preprint
Open access
PDF file
YZYao Zhu
Yao Zhu
YCYuefeng Chen
Yuefeng Chen
WWWei Wang
Wei Wang
XMXiaofeng Mao
Xiaofeng Mao
XYXiu Yan
Xiu Yan
XJXiangyang Ji
Xiangyang Ji
Learning generalized representations from limited training samples is crucial for applying deep neural networks in low-resource scenarios. Recently, methods based on Contrastive Language-Image Pre-training (CLIP) have exhibited promising performance in few-shot adaptation tasks. To avoid catastrophic forgetting and overfitting caused by few-shot fine-tuning, existing works usually freeze the parameters of CLIP pre-trained on large-scale datasets, overlooking the possibility that some parameters might not be suitable for downstream tasks. To this end, we revisit CLIP's visual encoder with a specific focus on its distinctive attention pooling layer, which performs a spatial weighted-sum of the dense feature maps. Given that dense feature maps contain meaningful semantic information, and different semantics hold varying importance for diverse downstream tasks (such as prioritizing semantics like ears and eyes in pet classification tasks rather than side mirrors), using the same weighted-sum operation for dense features across different few-shot tasks might not be appropriate. Hence, we propose fine-tuning the parameters of the attention pooling layer during the training process to encourage the model to focus on task-specific semantics. In the inference process, we perform residual blending between the features pooled by the fine-tuned and the original attention pooling layers to incorporate both the few-shot knowledge and the pre-trained CLIP's prior knowledge. We term this method as Semantic-Aware FinE-tuning (SAFE). SAFE is effective in enhancing the conventional few-shot CLIP and is compatible with the existing adapter approach (termed SAFE-A).