9 August 2023 in IEEE Access
Continual learning aims to enable neural networks to learn new tasks without catastrophic forgetting of previously learned knowledge. Orthogonal Gradient Descent algorithms have been proposed as an effective solution to mitigate catastrophic forgetting. However, these algorithms often rely on the Neural Tangent Kernel regime, which imposes limitations on network architecture. In this study, we propose a novel method to construct an orthonormal basis set for orthogonal projection by leveraging Catastrophic Forgetting Loss. In contrast to the conventional gradient-based basis that reflects an update of model within an infinitesimal range, our loss-based basis can account for the variance within two distinct points in the model parameter space, thus overcoming the limitations of the Neural Tangent Kernel regime. We provide both quantitative and qualitative analysis of the proposed method, discussing its advantages over conventional gradient-based baselines. Our approach is extensively evaluated on various model architectures and datasets, demonstrating a significant performance advantage, especially for deep or narrow networks where the Neural Tangent Kernel regime is violated. Furthermore, we offer a mathematical analysis based on higher-order Taylor series to provide theoretical justification. This study introduces a novel theoretical framework and a practical algorithm, potentially inspiring further research in areas such as continual learning, network debugging, and one-pass learning.
27 August 2025 in IEEE Access
This paper introduces a diffusion model that replaces the target prior distribution from a standard Gaussian to non-zero-mean Gaussian priors, with shifted latent trajectories determined by non-zero-mean Gaussian noise, thereby maintaining the closed-form of conventional diffusion models. Unlike conventional conditional models such as conditional denoising diffusion probabilistic models or classifier guidance, the proposed model implicitly aligns each class with a unique component in the Gaussian priors. Carefully devised positioning strategies uniformly distribute Gaussian components without introducing additional learnable parameters, which is essential to implicit-only conditioning on our model. Qualitative and quantitative experiments demonstrate that, even without additional conditioning layers or a classifier, the proposed framework achieves performance comparable to explicit methods in terms of Fréchet Inception Distance, precision, and recall. The framework is further validated under unsupervised settings by replacing class labels with pseudo-labels generated from K-means clustering on feature representations.