28 March 2024 in IEEE Transactions on Pattern Analysis and Machine Intelligence
Given data with noisy labels, over-parameterized deep networks suffer overfitting mislabeled data, resulting in poor generalization. The memorization effect of deep networks shows that although the networks have the ability to memorize all noisy data, they would first memorize clean training data, and then gradually memorize mislabeled training data. A simple and effective method that exploits the memorization effect to combat noisy labels is early stopping. However, early stopping cannot distinguish the memorization of clean data and mislabeled data, resulting in the network still inevitably overfitting mislabeled data in the early training stage. In this paper, to decouple the memorization of clean data and mislabeled data, and further reduce the side effect of mislabeled data, we perform additive decomposition on network parameters. Namely, all parameters are additively decomposed into two groups, i.e. , parameters $\mathbf {w}$ are decomposed as $\mathbf {w}=\bm {\sigma }+\bm {\gamma }$ . Afterward, the parameters $\bm {\sigma }$ are considered to memorize clean data, while the parameters $\bm {\gamma }$ are considered to memorize mislabeled data. Benefiting from the memorization effect, the updates of the parameters $\bm {\sigma }$ are encouraged to fully memorize clean data in early training, and then discouraged with the increase of training epochs to reduce interference of mislabeled data. The updates of the parameters $\bm {\gamma }$ are the opposite. In testing, only the parameters $\bm {\sigma }$ are employed to enhance generalization. Extensive experiments on both simulated and real-world benchmarks confirm the superior performance of our method.
29 December 2025 in IEEE Transactions on Pattern Analysis and Machine Intelligence
Deep neural networks possess remarkable learning capabilities but are vulnerable to overfitting in the presence of mislabeled data. A well-known memorization effect causes networks to first fit clean samples and later memorize noisy labels. Although early stopping can partially alleviate this issue, it cannot prevent the accumulation of incorrect knowledge or recover information lost due to mislabeled inputs. In this paper, we introduce an innovative mechanism for continuous review and timely correction of learned knowledge. Our approach allows the network to repeatedly revisit and reinforce correct information while promptly addressing any inaccuracies stemming from mislabeled data. We present a novel method called self-not-true-distillation (SNTD). This technique employs self-distillation, where the network from previous training iterations acts as a teacher, guiding the current network to review and solidify its understanding of accurate labels. Crucially, SNTD masks the true class label in the logits during this process, concentrating on the non-true classes to correct any erroneous knowledge that may have been acquired. We also recognize that different data classes follow distinct learning trajectories. A single teacher network might struggle to effectively guide the learning of all classes at once, which necessitates selecting different teacher networks for each specific class. Additionally, the influence of the teacher network's guidance varies throughout the training process. To address these challenges, we propose SNTD+, which integrates a class-wise distillation strategy along with a dynamic weight adjustment mechanism. Together, these enhancements significantly bolster SNTD's robustness in tackling complex scenarios characterized by label noise.