Mengchen Liu

Mengchen Liu

Auto-generated profile, based on publicly available information

Mengchen Liu

Microsoft Corporation

Research Focus & Metrics

Academic Career Length

7Years (2020-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
13 November 2020
Verified
Book Chapter
Closed access
PDF file
YCYinpeng Chen
Yinpeng Chen
XDXiyang Dai
Xiyang Dai
MLMengchen Liu
Mengchen Liu
DCDongdong Chen
Dongdong Chen
LYLu Yuan
Lu Yuan
ZLZicheng Liu
Zicheng Liu

21 July 2022 in arXiv
Verified
Preprint
Open access
PDF file
KWKan Wu
Kan Wu
JZJinnian Zhang
Jinnian Zhang
HPHouwen Peng
Houwen Peng
MLMengchen Liu
Mengchen Liu
BXBin Xiao
Bin Xiao
LYLu Yuan
Lu Yuan
Vision transformer (ViT) recently has drawn great attention in computer vision due to its remarkable model capability. However, most prevailing ViT models suffer from huge number of parameters, restricting their applicability on devices with limited resources. To alleviate this issue, we propose TinyViT, a new family of tiny and efficient small vision transformers pretrained on large-scale datasets with our proposed fast distillation framework. The central idea is to transfer knowledge from large pretrained models to small ones, while enabling small models to get the dividends of massive pretraining data. More specifically, we apply distillation during pretraining for knowledge transfer. The logits of large teacher models are sparsified and stored in disk in advance to save the memory cost and computation overheads. The tiny student transformers are automatically scaled down from a large pretrained model with computation and parameter constraints. Comprehensive experiments demonstrate the efficacy of TinyViT. It achieves a top-1 accuracy of 84.8% on ImageNet-1k with only 21M parameters, being comparable to Swin-B pretrained on ImageNet-21k while using 4.2 times fewer parameters. Moreover, increasing image resolutions, TinyViT can reach 86.5% accuracy, being slightly better than Swin-L while using only 11% parameters. Last but not the least, we demonstrate a good transfer ability of TinyViT on various downstream tasks. Code and models are available at https://github.com/microsoft/Cream/tree/main/TinyViT.