Shunyu Liu

Shunyu Liu

Auto-generated profile, based on publicly available information

Shunyu Liu

School of Computer Science and Technology, East China Normal University, Shanghai, China

Research Focus & Metrics

Academic Career Length

2Years (2025-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
9 January 2025 in ACM Computing Surveys
Verified
Journal Article
Open access
YYYutao Yang
Yutao Yang
Jie ZhouJie Zhou
Jie Zhou
XDXuanwen Ding
Xuanwen Ding
THTianyu Huai
Tianyu Huai
SLShunyu Liu
Shunyu Liu
Liang HeLiang He
Liang He
Recently, foundation language models (LMs) have marked significant achievements in the domains of natural language processing and computer vision. Unlike traditional neural network models, foundation LMs obtain a great ability for transfer learning by acquiring rich common sense knowledge through pre-training on extensive unsupervised datasets with a vast number of parameters. Despite these capabilities, LMs still struggle with catastrophic forgetting, hindering their ability to learn continuously like humans. To address this, continual learning (CL) methodologies have been introduced, allowing LMs to adapt to new tasks while retaining learned knowledge. However, a systematic taxonomy of existing approaches and a comparison of their performance are still lacking. In this article, we delve into a comprehensive review, summarization, and classification of the existing literature on CL-based approaches applied to foundation language models, such as pre-trained language models, large language models, and vision-language models. We divide these studies into offline and online CL, which consist of traditional methods, parameter-efficient-based methods, instruction tuning-based methods and continual pre-training methods. Additionally, we outline the typical datasets and metrics employed in CL research and provide a detailed analysis of the challenges and future work for LMs-based continual learning.