Ning Ding

Ning Ding

Auto-generated profile, based on publicly available information

Ning Ding

Computer Science, Tsinghua University, Beijing, China

Research Focus & Metrics

Academic Career Length

15Years (2012-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
2 March 2023 in Nature Machine Intelligence
Verified
Journal Article
Open access
PDF file
NDNing Ding
Ning Ding
YQYujia Qin
Yujia Qin
GYGuang Yang
Guang Yang
FWFuchao Wei
Fuchao Wei
ZYZonghan Yang
Zonghan Yang
MSMaosong Sun
Maosong Sun
With the prevalence of pre-trained language models (PLMs) and the pre-training–fine-tuning paradigm, it has been continuously shown that larger models tend to yield better performance. However, as PLMs scale up, fine-tuning and storing all the parameters is prohibitively costly and eventually becomes practically infeasible. This necessitates a new branch of research focusing on the parameter-efficient adaptation of PLMs, which optimizes a small portion of the model parameters while keeping the rest fixed, drastically cutting down computation and storage costs. In general, it demonstrates that large-scale models could be effectively stimulated by the optimization of a few parameters. Despite the various designs, here we discuss and analyse the approaches under a more consistent and accessible term ‘delta-tuning’, where ‘delta’ a mathematical notation often used to denote changes, is borrowed to refer to the portion of parameters that are ‘changed’ during training. We formally describe the problem and propose a unified categorization criterion for existing delta-tuning methods to explore their correlations and differences. We also discuss the theoretical principles underlying the effectiveness of delta-tuning and interpret them from the perspectives of optimization and optimal control. Furthermore, we provide a holistic empirical study on over 100 natural language processing tasks and investigate various aspects of delta-tuning. With comprehensive study and analysis, our research demonstrates the theoretical and practical properties of delta-tuning in the adaptation of PLMs.

24 December 2024 in ACM Computing Surveys
Verified
Journal Article
Open access
YQYujia Qin
Yujia Qin
SHShengding Hu
Shengding Hu
YLYankai Lin
Yankai Lin
WCWeize Chen
Weize Chen
NDNing Ding
Ning Ding
MSMaosong Sun
Maosong Sun
Humans possess an extraordinary ability to create and utilize tools. With the advent of foundation models, artificial intelligence systems have the potential to be equally adept in tool use as humans. This paradigm, which is dubbed as tool learning with foundation models , combines the strengths of tools and foundation models to achieve enhanced accuracy, efficiency, and automation in problem-solving. This article presents a systematic investigation and comprehensive review of tool learning. We first introduce the background of tool learning, including its cognitive origins, the paradigm shift of foundation models, and the complementary roles of tools and models. Then we recapitulate existing tool learning research and formulate a general framework: starting from understanding the user instruction, models should learn to decompose a complex task into several subtasks, dynamically adjust their plan through reasoning, and effectively conquer each sub-task by selecting appropriate tools. We also discuss how to train models for improved tool-use capabilities and facilitate generalization in tool learning. Finally, we discuss several open problems that require further investigation, such as ensuring trustworthy tool use, enabling tool creation with foundation models, and addressing personalization challenges. Overall, we hope this article could inspire future research in integrating tools with foundation models.

22 June 2022 in Research Square
Verified
Preprint
Open access
PDF file
NDNing Ding
Ning Ding
YQYujia Qin
Yujia Qin
GYGuang Yang
Guang Yang
FWFuchao Wei
Fuchao Wei
ZYZonghan Yang
Zonghan Yang
MSMaosong Sun
Maosong Sun
As pre-trained language models (PLMs) have become the fundamental infrastructure for various NLP tasks and researchers have readily enjoyed themselves in the pretraining-finetuning paradigm, evidence from emerging research has continuously proven that larger models tend to yield better performance. However, despite the welcome outcome, the process of fine-tuning large-scale PLMs brings prohibitive adaptation costs. In fact, fine- tuning all the parameters of a colossal model and retaining separate instances for different tasks are practically infeasible. This necessitates a new branch of research focusing on the parameter-efficient adaptation of PLMs. In order to unleash the imagination of the possible advantages of such methods, not limited to parameter efficiency, we coined a new term delta tuning from a morphological point of view to refer to the original “parameter efficient tuning”. In contrast with the standard fine-tuning, delta tuning only fine-tunes a small portion of the model parameters while keeping the rest untouched, largely reducing both the computation and storage costs. Recent studies have demonstrated that a series of delta tuning methods with distinct tuned parameter selection could achieve performance on a par with full-parameter fine-tuning, suggesting a new promising way of stimulating large-scale PLMs. In this paper, we first formally describe the problem of delta tuning and then comprehensively review recent delta tuning approaches. We also propose a unified categorization criterion that divides existing delta tuning methods into three groups: addition-based, specification-based, and reparameterization-based methods. Though initially proposed as an efficient method to steer large models, we believe that some of the fascinating evidence discovered along with delta tuning could help further reveal the mechanisms of PLMs and even deep neural networks. To this end, we discuss the theoretical principles underlying the effectiveness of delta tuning and propose frameworks to interpret delta tuning from the perspective of optimization and optimal control, respectively. Furthermore, we provide a holistic empirical study of representative methods, where results on over 100 NLP tasks demonstrate a comprehensive performance comparison of different approaches. The experimental results also cover the analysis of combinatorial, scaling and transferable properties of delta tuning. To facilitate the research of delta tuning, we are also developing an open-source toolkit, OpenDelta2, that enables practitioners to efficiently and flexibly implement delta tuning on PLMs. At last, we discuss a series of real-world applications of delta tuning.

21 October 2024
Verified
Conference Paper
Open access
YCYulin Chen
Yulin Chen
NDNing Ding
Ning Ding
Hai-Tao ZhengHai-Tao Zheng
Hai-Tao Zheng
ZLZhiyuan Liu
Zhiyuan Liu
MSMaosong Sun
Maosong Sun
BZBowen Zhou
Bowen Zhou
Artificial intelligence has been applied in various aspects of online education to facilitate teaching and learning. However, few approaches have been made towards a complete AI-powered tutoring system. In this work, we explore the development of a full-fledged intelligent tutoring system based on large language models (LLMs). The proposed system ChatTutor, powered by state-of-the-art LLMs, is equipped with automatic course planning and adjusting, informative instruction, and adaptive quiz offering and evaluation. ChatTutor is decomposed into three inter-connected core processes: interaction, reflection, and reaction. Each process is implemented by chaining LLM-powered tools along with dynamically updated memory modules. To demonstrate the mechanism of each working module and the benefits of structured memory control and adaptive reflection, we conduct a wide range of analysis based on statistical results and user study. The analysis shows the designed processes boost system consistency and stability under long-term interaction and intentional disruptions, with up to 5% and 20% increase in performance respectively. Meanwhile, we also compare the system with scripts from real-world online learning platform and discuss the potential issues unique to LLM-based systems.

16 February 2012 in Acta Oto-Laryngologica
Verified
Journal Article
Closed access
ZYZhimin Yan
Zhimin Yan
NDNing Ding
Ning Ding
XLXiaosong Liu
Xiaosong Liu
HHHong Hua
Hong Hua
Congenital agenesis of the salivary glands is an extremely rare congenital condition, which may cause severe xerostomia, progressive dental caries, and oropharyngeal candidiasis in children. To date, there have been few documented cases of aplasia of the major salivary glands. Congenital agenesis of the salivary glands accompanied by absence of the lacrimal puncta is even more rare. We report a case of a 5-year-old boy with xerostomia and extensive dental caries. Salivary gland imaging with sodium pertechnetate 99mTcO4 showed bilateral aplasia of the parotid, submandibular, and sublingual glands. The patient was found to lack the left lacrimal puncta through physical examination. Here we describe the clinical presentation, diagnostic essentials, and medical and dental management of the patient.

18 July 2024 in IEEE/ACM Transactions on Audio, Speech, and Language Processing
Verified
Journal Article
Open access
YQYujia Qin
Yujia Qin
XWXiaozhi Wang
Xiaozhi Wang
YSYusheng Su
Yusheng Su
YLYankai Lin
Yankai Lin
NDNing Ding
Ning Ding
JZJie Zhou
Jie Zhou
Why can pre-trained language models (PLMs) learn universal representations and effectively adapt to broad NLP tasks differing a lot superficially? In this work, we empirically find evidence indicating that the adaptations of PLMs to various few-shot tasks can be reparameterized as optimizing only a few free parameters in a unified low-dimensional intrinsic task subspace , which may help us understand why PLMs could easily adapt to various NLP tasks with small-scale data. To find such a subspace and examine its universality, we propose an analysis pipeline called intrinsic prompt tuning (IPT). Specifically, we resort to the recent success of prompt tuning and decompose the soft prompts of multiple NLP tasks into the same low-dimensional nonlinear subspace, then we learn to adapt the PLM to unseen data or tasks by only tuning parameters in this subspace. In the experiments, we study diverse few-shot NLP tasks and surprisingly find that in a 250-dimensional subspace found with 100 tasks, by only tuning 250 free parameters, we can recover 97% and 83% of the full prompt tuning performance for 100 seen tasks (using different training data) and 20 unseen tasks, respectively, showing great generalization ability of the found intrinsic task subspace. Besides being an analysis tool, IPTcould further help us improve the prompt tuning stability.