Pan Zhou

Pan Zhou

Auto-generated profile, based on publicly available information

Pan Zhou

Singapore Management University

Research Focus & Metrics

Academic Career Length

10Years (2017-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
1 November 2023 in IEEE Transactions on Pattern Analysis and Machine Intelligence
Verified
Journal Article
Closed access
WYWeihao Yu
Weihao Yu
CSChenyang Si
Chenyang Si
PZPan Zhou
Pan Zhou
MLMi Luo
Mi Luo
YZYichen Zhou
Yichen Zhou
XWXinchao Wang
Xinchao Wang
MetaFormer, the abstracted architecture of Transformer, has been found to play a significant role in achieving competitive performance. In this paper, we further explore the capacity of MetaFormer, again, by migrating our focus away from the token mixer design: we introduce several baseline models under MetaFormer using the most basic or common mixers, and demonstrate their gratifying performance. We summarize our observations as follows: 1)MetaFormer ensures solid lower bound of performance: By merely adopting identity mapping as the token mixer, the MetaFormer model, termed IdentityFormer, achieves $> $>80% accuracy on ImageNet-1 K. 2)MetaFormer works well with arbitrary token mixers: When specifying the token mixer as even a random matrix to mix tokens, the resulting model RandFormer yields an accuracy of $> $>81%, outperforming IdentityFormer. Rest assured of MetaFormer's results when new token mixers are adopted. 3)MetaFormer effortlessly offers state-of-the-art results: With just conventional token mixers dated back five years ago, the models instantiated from MetaFormer already beat state of the art. a)ConvFormer outperforms ConvNeXt: Taking the common depthwise separable convolutions as the token mixer, the model termed ConvFormer, which can be regarded as pure CNNs, outperforms the strong CNN model ConvNeXt. b)CAFormer sets new record on ImageNet-1 K: By simply applying depthwise separable convolutions as token mixer in the bottom stages and vanilla self-attention in the top stages, the resulting model CAFormer sets a new record on ImageNet-1 K: it achieves an accuracy of 85.5% at $224 \times 224$224×224 resolution, under normal supervised training without external data or distillation. In our expedition to probe MetaFormer, we also find that a new activation, StarReLU , reduces 71% FLOPs of activation compared with commonly-used GELU yet achieves better performance. Specifically, StarReLU is a variant of Squared ReLU dedicated to alleviating distribution shift. We expect StarReLU to find great potential in MetaFormer-like models alongside other neural networks.

27 March 2024 in IEEE Transactions on Pattern Analysis and Machine Intelligence
Verified
Journal Article
Closed access
PZPan Zhou
Pan Zhou
XXXingyu Xie
Xingyu Xie
ZLZhouchen Lin
Zhouchen Lin
SYShuicheng Yan
Shuicheng Yan
AdamW modifies Adam by adding a decoupled weight decay to decay network weights per training iteration. For adaptive algorithms, this decoupled weight decay does not affect specific optimization steps, and differs from the widely used $\ell _{2}$ -regularizer which changes optimization steps via changing the first- and second-order gradient moments. Despite its great practical success, for AdamW, its convergence behavior and generalization improvement over Adam and $\ell _{2}$ -regularized Adam ( $\ell _{2}$ -Adam) remain absent yet. To solve this issue, we prove the convergence of AdamW and justify its generalization advantages over Adam and $\ell _{2}$ -Adam. Specifically, AdamW provably converges but minimizes a dynamically regularized loss that combines vanilla loss and a dynamical regularization induced by decoupled weight decay, thus yielding different behaviors with Adam and $\ell _{2}$ -Adam. Moreover, on both general nonconvex problems and PŁ-conditioned problems, we establish stochastic gradient complexity of AdamW to find a stationary point. Such complexity is also applicable to Adam and $\ell _{2}$ -Adam, and improves their previously known complexity, especially for over-parametrized networks. Besides, we prove that AdamW enjoys smaller generalization errors than Adam and $\ell _{2}$ -Adam from the Bayesian posterior aspect. This result, for the first time, explicitly reveals the benefits of decoupled weight decay in AdamW. Experimental results validate our theory.

4 July 2024 in IEEE Transactions on Pattern Analysis and Machine Intelligence
Verified
Journal Article
Closed access
XXXingyu Xie
Xingyu Xie
PZPan Zhou
Pan Zhou
HLHuan Li
Huan Li
ZLZhouchen Lin
Zhouchen Lin
SYShuicheng Yan
Shuicheng Yan
In deep learning, different kinds of deep networks typically need different optimizers, which have to be chosen after multiple trials, making the training process inefficient. To relieve this issue and consistently improve the model training speed across deep networks, we propose the ADAptive Nesterov momentum algorithm, Adan for short. Adan first reformulates the vanilla Nesterov acceleration to develop a new Nesterov momentum estimation (NME) method, which avoids the extra overhead of computing gradient at the extrapolation point. Then Adan adopts NME to estimate the gradient's first- and second-order moments in adaptive gradient algorithms for convergence acceleration. Besides, we prove that Adan finds an $\epsilon$ -approximate first-order stationary point within $\mathcal {O}(\epsilon ^{-3.5})$ stochastic gradient complexity on the non-convex stochastic problems (e.g., deep learning problems), matching the best-known lower bound. Extensive experimental results show that Adan consistently surpasses the corresponding SoTA optimizers on vision, language, and RL tasks and sets new SoTAs for many popular networks and frameworks, e.g., ResNet, ConvNext, ViT, Swin, MAE, DETR, GPT-2, Transformer-XL, and BERT. More surprisingly, Adan can use half of the training cost (epochs) of SoTA optimizers to achieve higher or comparable performance on ViT, GPT-2, MAE, etc , and also shows great tolerance to a large range of minibatch size, e.g., from 1 k to 32 k.

21 November 2019 in IEEE Transactions on Pattern Analysis and Machine Intelligence
Verified
Journal Article
Closed access
PZPan Zhou
Pan Zhou
CLCanyi Lu
Canyi Lu
JFJiashi Feng
Jiashi Feng
ZLZhouchen Lin
Zhouchen Lin
SYShuicheng Yan
Shuicheng Yan
Multi-way or tensor data analysis has attracted increasing attention recently, with many important applications in practice. This paper develops a tensor low-rank representation (TLRR) method, which is the first approach that can exactly recover the clean data of intrinsic low-rank structure and accurately cluster them as well, with provable performance guarantees. In particular, for tensor data with arbitrary sparse corruptions, TLRR can exactly recover the clean data under mild conditions; meanwhile TLRR can exactly verify their true origin tensor subspaces and hence cluster them accurately. Moreover, the objective function of TLRR can be optimized via efficient convex programing with convergence guarantees. Besides, we provide two simple yet effective dictionary construction methods, namely the simple TLRR (S-TLRR) and robust TLRR (R-TLRR), to handle slightly and severely corrupted data respectively. Experimental results on two computer vision data analysis tasks, image/video recovery and face clustering, clearly demonstrate the superior performance, efficiency and robustness of our developed method over state-of-the-arts including the popular LRR[1] and SSC[2] methods.

29 October 2022
Verified
Book Chapter
Closed access
PDF file
JXJunbin Xiao
Junbin Xiao
PZPan Zhou
Pan Zhou
TCTat-Seng Chua
Tat-Seng Chua
SYShuicheng Yan
Shuicheng Yan

27 July 2022 in IEEE Transactions on Neural Networks and Learning Systems
Verified
Journal Article
Closed access
SLShuai Lin
Shuai Lin
CLChen Liu
Chen Liu
PZPan Zhou
Pan Zhou
ZHZi-Yuan Hu
Zi-Yuan Hu
SWShuojia Wang
Shuojia Wang
XLXiaodan Liang
Xiaodan Liang
Graph-level representations are critical in various real-world applications, such as predicting the properties of molecules. However, in practice, precise graph annotations are generally very expensive and time-consuming. To address this issue, graph contrastive learning constructs an instance discrimination task, which pulls together positive pairs (augmentation pairs of the same graph) and pushes away negative pairs (augmentation pairs of different graphs) for unsupervised representation learning. However, since for a query, its negatives are uniformly sampled from all graphs, existing methods suffer from the critical sampling bias issue, i.e., the negatives likely having the same semantic structure with the query, leading to performance degradation. To mitigate this sampling bias issue, in this article, we propose a prototypical graph contrastive learning (PGCL) approach. Specifically, PGCL models the underlying semantic structure of the graph data via clustering semantically similar graphs into the same group and simultaneously encourages the clustering consistency for different augmentations of the same graph. Then, given a query, it performs negative sampling via drawing the graphs from those clusters that differ from the cluster of query, which ensures the semantic difference between query and its negative samples. Moreover, for a query, PGCL further reweights its negative samples based on the distance between their prototypes (cluster centroids) and the query prototype such that those negatives having moderate prototype distance enjoy relatively large weights. This reweighting strategy is proven to be more effective than uniform sampling. Experimental results on various graph benchmarks testify the advantages of our PGCL over state-of-the-art methods. The code is publicly available at https://github.com/ha-lins/PGCL.