Zijian Gao

Mr. Zijian Gao

NUDT(National University of Defense Technology)
orcid

Research Focus & Metrics

Academic Career Length

8Years (2019-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
13 August 2021 in Entropy
Verified
Journal Article
Open access
PDF file
Zijian GaoZijian Gao
Zijian Gao
KXKele Xu
Kele Xu
BDBo Ding
Bo Ding
HWHuaimin Wang
Huaimin Wang
Recently, deep reinforcement learning (RL) algorithms have achieved significant progress in the multi-agent domain. However, training for increasingly complex tasks would be time-consuming and resource intensive. To alleviate this problem, efficient leveraging of historical experience is essential, which is under-explored in previous studies because most existing methods fail to achieve this goal in a continuously dynamic system owing to their complicated design. In this paper, we propose a method for knowledge reuse called “KnowRU”, which can be easily deployed in the majority of multi-agent reinforcement learning (MARL) algorithms without requiring complicated hand-coded design. We employ the knowledge distillation paradigm to transfer knowledge among agents to shorten the training phase for new tasks while improving the asymptotic performance of agents. To empirically demonstrate the robustness and effectiveness of KnowRU, we perform extensive experiments on state-of-the-art MARL algorithms in collaborative and competitive scenarios. The results show that KnowRU outperforms recently reported methods and not only successfully accelerates the training phase, but also improves the training performance, emphasizing the importance of the proposed knowledge reuse for MARL.

6 July 2024 in Neural Networks
Verified
Journal Article
Closed access
Zijian GaoZijian Gao
Zijian Gao
KXKele Xu
Kele Xu
HZHuiping Zhuang
Huiping Zhuang
LLLi Liu
Li Liu
XMXinjun Mao
Xinjun Mao
HWHuaimin Wang
Huaimin Wang

11 April 2025 in Proceedings of the AAAI Conference on Artificial Intelligence
Verified
Journal Article
Open access
PDF file
Zijian GaoZijian Gao
Zijian Gao
SHShanhao Han
Shanhao Han
XZXingxing Zhang
Xingxing Zhang
KXKele Xu
Kele Xu
DZDulan Zhou
Dulan Zhou
HWHuaimin Wang
Huaimin Wang
Logit-based knowledge distillation (KD) is commonly used to mitigate catastrophic forgetting in class-incremental learning (CIL) caused by data distribution shifts. However, the strict match of logit values between student and teacher models conflicts with the cross-entropy (CE) loss objective of learning new classes, leading to significant recency bias (i.e. unfairness). To address this issue, we rethink the overlooked limitations of KD-based methods through empirical analysis. Inspired by our findings, we introduce a plug-and-play pre-process method that normalizes the logits of both the student and teacher across all classes, rather than just the old classes, before distillation. This approach allows the student to focus on both old and new classes, capturing intrinsic inter-class relations from the teacher. By doing so, our method avoids the inherent conflict between KD and CE, maintaining fairness between old and new classes. Additionally, recognizing that overconfident teacher predictions can hinder the transfer of inter-class relations (i.e., dark knowledge), we extend our method to capture intra-class relations among different instances, ensuring fairness within old classes. Our method integrates seamlessly with existing logit-based KD approaches, consistently enhancing their performance across multiple CIL benchmarks without incurring additional training costs.

11 October 2023 in IEEE Transactions on Artificial Intelligence
Verified
Journal Article
Closed access
CCChao Chen
Chao Chen
YZYuanzhao Zhai
Yuanzhao Zhai
Zijian GaoZijian Gao
Zijian Gao
KXKele Xu
Kele Xu
SYSen Yang
Sen Yang
HWHuaimin Wang
Huaimin Wang
Reinforcement learning (RL) has achieved promising results in solving numerous challenging sequential decision problems. To address the issue of sparse extrinsic rewards, researchers have proposed intrinsic rewards, enabling the agent to acquire skills that may prove valuable in the pursuit of future rewards. One representative approach for generating intrinsic rewards involves constructing a predictive model to assess the novelty of states. However, due to the stochastic nature of complex environments, intrinsic rewards can be noisy. Directly employing noisy forward predictions to supervise policies can be detrimental to learning performance and efficiency. Many recent studies utilize the $\ell _{2}$ norm or variance to measure novelty, which further amplifies the noise through squaring operations. In this paper, we aim to tackle these challenges by leveraging Nuclear Norm Maximization (NNM). Specifically, we propose a novel curiosity reward that accurately quantifies the novelty of the exploration environment while exhibiting a high tolerance for noise and outliers. Our extensive experiments in various benchmark environments demonstrate that NNM achieves state-of-the-art performance compared to previous curiosity-based methods. When trained solely with intrinsic rewards, NNM achieves a human-normalized score of 1.09 on a subset of 26 Atari games, twice the performance of methods based on competitive intrinsic rewards.

7 December 2023 in IEEE Transactions on Emerging Topics in Computational Intelligence
Verified
Journal Article
Closed access
Zijian GaoZijian Gao
Zijian Gao
YLYiying Li
Yiying Li
KXKele Xu
Kele Xu
YZYuanzhao Zhai
Yuanzhao Zhai
BDBo Ding
Bo Ding
HWHuaimin Wang
Huaimin Wang
The sparsity of extrinsic rewards presents a significant challenge for deep reinforcement learning (DRL). As an alternative, researchers have focused on intrinsic rewards to improve exploration efficiency. One of the most representative approaches is utilizing curiosity. However, the challenge of designing effective intrinsic rewards remains, as artificial curiosity differs significantly from human curiosity. In this article, we introduce a novel curiosity approach for DRL, named DyMeCu, which stands for Dynamic Memory-based Curiosity. Inspired by human curiosity and information theory, DyMeCu constructs a dynamic memory using the online learner following the bootstrap paradigm. Additionally, we design a two-learner architecture inspired by ensemble techniques to access curiosity better. The information gap between the two learners serves as the intrinsic reward for agents, and the state information is consistently consolidated into the dynamic memory. Compared with previous curiosity methods, DyMeCu can better mimic human curiosity by using a dynamic memory that can be dynamically grown based on a bootstrap paradigm with two learners. Large-scale empirical experiments on multiple benchmarks, including DeepMind Control Suite and Atari Suite, demonstrate that DyMeCu outperforms competitive curiosity-based methods with or without extrinsic rewards. In the Atari Suite, DyMeCu achieves a mean human-normalized score of 5.076 on a subset of 26 Atari games, achieving a 77.4% relative improvement over the best other baselines. In the DeepMind Control Suite, DyMeCu presents new state-of-the-art results across 11 tasks of all 12 when compared to curiosity-based methods and other pre-training strategies.

11 November 2025 in IEEE Transactions on Circuits and Systems for Video Technology
Verified
Journal Article
Closed access
Zijian GaoZijian Gao
Zijian Gao
KXKele Xu
Kele Xu
XZXingxing Zhang
Xingxing Zhang
HZHuiping Zhuang
Huiping Zhuang
TWTianjiao Wan
Tianjiao Wan
HWHuaimin Wang
Huaimin Wang
Exemplar-free Class-Incremental Learning (EFCIL) poses a significant challenge in mitigating catastrophic forgetting, due to the absence of exemplars. Recently, analytic learning-based methods propose a recursive alignment procedure to execute EFCIL in a phase-invariant manner and show state-of-the-art performance. However, they heavily rely on a frozen feature extractor trained with the initial dataset to avoid the misalignment between feature and label spaces, ignoring the importance of acquiring generalizable features across incremental tasks for performance improvement. To tackle this, we rethink the obscured sub-optimality of analytic learning-based methods, particularly through empirical reevaluation, and then introduce the Multi-head analytic learning (Muheal) approach. Muheal forms the multi-head model with a delicate feature extractor, thereby introducing a feature optimization procedure and a forgetting compensation module to balance the learning and forgetting. Specifically, within the feature optimization procedure, the feature extractor seeks to learn more generalizable features in a self-supervised manner using the fully-connected classification head. An analytic learning-based classification head follows to align the feature-label space. Additionally, we employ the compensation module to generate and align pseudo-features with a replicated analytic head, thus preventing overfitting and testing. Comprehensive experiments on several benchmark datasets have demonstrated that Muheal significantly outperforms existing state-of-the-art EFCIL methods and is comparable, if not superior, to methods that use replay techniques.