28 March 2024 in IEEE Transactions on Pattern Analysis and Machine Intelligence
Given data with noisy labels, over-parameterized deep networks suffer overfitting mislabeled data, resulting in poor generalization. The memorization effect of deep networks shows that although the networks have the ability to memorize all noisy data, they would first memorize clean training data, and then gradually memorize mislabeled training data. A simple and effective method that exploits the memorization effect to combat noisy labels is early stopping. However, early stopping cannot distinguish the memorization of clean data and mislabeled data, resulting in the network still inevitably overfitting mislabeled data in the early training stage. In this paper, to decouple the memorization of clean data and mislabeled data, and further reduce the side effect of mislabeled data, we perform additive decomposition on network parameters. Namely, all parameters are additively decomposed into two groups, i.e. , parameters $\mathbf {w}$ are decomposed as $\mathbf {w}=\bm {\sigma }+\bm {\gamma }$ . Afterward, the parameters $\bm {\sigma }$ are considered to memorize clean data, while the parameters $\bm {\gamma }$ are considered to memorize mislabeled data. Benefiting from the memorization effect, the updates of the parameters $\bm {\sigma }$ are encouraged to fully memorize clean data in early training, and then discouraged with the increase of training epochs to reduce interference of mislabeled data. The updates of the parameters $\bm {\gamma }$ are the opposite. In testing, only the parameters $\bm {\sigma }$ are employed to enhance generalization. Extensive experiments on both simulated and real-world benchmarks confirm the superior performance of our method.
1 October 2025 in IEEE Transactions on Neural Networks and Learning Systems
Personalized federated learning (PFL) allows clients to collaboratively train their personalized models to handle situations where data from different clients are not independent and identically distributed (non-IID). Previous PFL research implicitly assumes that clients benefit most from those with similar data distributions. Correspondingly, methods such as personalized weight aggregation assign higher weights to similar clients during aggregation. We pose a question: can a client benefit from other clients with dissimilar data distributions, and if so, how? This question is particularly relevant in scenarios with a high degree of non-IID, where clients have widely different distributions, and learning from only similar clients will result in a loss of knowledge from many other clients. We note that when dealing with clients with similar distributions, current methods tend to enforce their models to be close in the parameter space. It is reasonable to conjecture that a client can benefit from dissimilar clients if we allow their models to depart from each other. Based on this idea, we propose DiversiFed, which allows each client to learn from clients with diversified distribution. DiversiFed pushes personalized models of clients with dissimilar distributions apart in the parameter space while pulling together those with similar distributions. In addition, to achieve the above effect without using prior knowledge of distribution, we design a loss function that leverages model similarity to determine the degree of attraction and repulsion between any two models. Experiments on benchmark and medical datasets show that DiversiFed can outperform the state-of-the-art (SOTA) methods by up to 3.19%.