10 October 2024 in IEEE Transactions on Computers
Previous state-of-the-art studies have demonstrated that adversaries can access sensitive user data by membership inference attacks (MIAs) in Federated Learning (FL). Introducing differential privacy (DP) into the FL framework is an effective way to enhance the privacy of FL. Nevertheless, in differentially private federated learning (DP-FL), local gradients become excessively sparse in certain training rounds. Especially when training with low privacy budgets, there is a risk of introducing excessive noise into clients’ gradients. This issue can lead to a significant degradation in the accuracy of the global model. Thus, how to balance the user's privacy and global model accuracy becomes a challenge in DP-FL. To this end, we propose an approach, known as differential privacy federated aggregation, based on significant gradient protection (DP-FedASGP). DP-FedASGP can mitigate excessive noises by protecting significant gradients and accelerate the convergence of the global model by calculating dynamic aggregation weights for gradients. Experimental results show that DP-FedASGP achieves comparable privacy protection effects to DP-FedAvg and cpSGD (communication-private SGD based on gradient quantization) but outperforms DP-FedSNLC (sparse noise based on clipping losses and privacy budget costs) and FedSMP (sparsified model perturbation). Furthermore, the average global test accuracy of DP-FedASGP across four datasets and three models is about $2.62$ %, $4.71$ %, $0.45$ %, and $0.19$ % higher than the above methods, respectively. These improvements indicate that DP-FedASGP is a promising approach for balancing the privacy and accuracy of DP-FL.
28 February 2025 in IEEE Transactions on Network Science and Engineering
In Differentially Private Federated Learning (DP-FL), gradient clipping can prevent excessive noise from being added to the gradient and ensure that the impact of noise is within a controllable range. However, state-of-the-art methods adopt fixed or imprecise clipping thresholds for gradient clipping, which is not adaptive to the changes in the gradients. This issue can lead to a significant degradation in accuracy while training the global model. To this end, we propose Differential Privacy Federated Adaptive gradient Clipping based on gradient Norm (DP-FedACN). DP-FedACN can calculate the decay rate of the clipping threshold by considering the overall changing trend of the gradient norm. Furthermore, DP-FedACN can accurately adjust the clipping threshold for each training round according to the actual changes in gradient norm, clipping loss, and decay rate. Experimental results demonstrate that DP-FedACN can maintain privacy protection performance similar to that of DP-FedAvg under member inference attacks and model inversion attacks. DP-FedACN significantly outperforms DP-FedAGNC and DP-FedDDC in privacy protection metrics. Additionally, the test accuracy of DP-FedACN is approximately 2.61%, 1.01%, and 1.03% higher than the other three baseline methods, respectively. DP-FedACN can improve the global model training accuracy while ensuring the privacy protection of the model. All experimental results demonstrate that the proposed DP-FedACN can help find a fine-grained privacy-accuracy trade-off in DP-FL.
28 February 2025 in IEEE Internet of Things Journal
In real-world Industrial Internet of Things (IIoT) scenarios, due to the limited storage capacity of IIoT devices, fresh data continuously received by diverse devices will overwrite the outdated data and change the local data distribution. However, state-of-the-art studies have demonstrated that federated learning tends to focus on training with fresh data, and the latest global model may forget the historical update directions (i.e., catastrophic forgetting). This issue can significantly degrade the global model accuracy. Existing methods primarily focus on integrating outdated data characteristics into fresh data but overlook the large parameter update gap between global and local models during global aggregation. This gap can cause the global model updates to deviate from the optimal direction. To this end, we propose a federated adaptive weighted aggregation method based on model consistency (FedAWAC). Specifically, FedAWAC measures the model consistency on devices and dynamically adjusts the aggregation weights of each local model, thereby guiding the global model toward optimal updates. Furthermore, FedAWAC integrates $\mathcal {M}$ historical global models most correlated to the latest global model on the cloud server to overcome catastrophic forgetting. Experiments on four different datasets (nonidentically and independently distributed settings) indicate that compared to five baselines, FedAWAC can improve global model accuracy by an average of 1.86%, reduce the forgetting rate by an average of 3.91%, and save average memory usage by up to 2.57 GB.
15 July 2025 in IEEE Internet of Things Journal
Cloud-centric deployment of large models brings privacy concerns and communication burdens to IoT devices. Federated Learning (FL) can help numerous IoT devices collaboratively train a large model in a distributed manner to provide intelligent services and applications (AIoT). However, due to the limited storage capacity of IoT devices, fresh data collected by IoT devices often overwrites outdated data and establishes Non-IID data distributions. Moreover, state-of-the-art studies have indicated that FL tends to use fresh data for model training. This causes the global model to forget outdated data’s characteristics (i.e., catastrophic forgetting). Some methods utilize Knowledge Distillation (KD) to extract and integrate characteristics from both fresh data and outdated data. However, existing KD-based methods use fixed distillation temperatures for different IoT devices, which overlooks that fixed distillation temperatures cannot match the data distributions on different IoT devices and may degrade global model accuracy. To this end, we propose a Federated Dynamic Decoupled Distillation method based on Logits distribution (Fed3DL). Specifically, Fed3DL utilizes decoupled distillation to mitigate catastrophic forgetting. To dynamically adjust the distillation temperature of each IoT device, Fed3DL novelly builds an adaptive temperature-aware mechanism based on the Logits distribution. Furthermore, Fed3DL introduces a regularization term into the local distillation loss to improve global model accuracy. Experiments on four datasets show that Fed3DL can improve the global model accuracy by an average of 3.19%, reduce the forgetting rate by an average of 4.46%, and achieve the lowest inter-class accuracy disparity.
11 December 2025 in IEEE Transactions on Consumer Electronics
Large Language Models (LLMs) can stimulate intelligent functions on mobile devices in the applications of consumer electronics. However, it is challenging to deploy LLMs on mobile devices due to limited memory and computational resources that fluctuate as devices execute multiple AI tasks concurrently. Quantization is a widely adopted compression technique, but faces two major difficulties in practical deployment: 1) Existing quantization methods are unable to select suitable models for deployment according to the resource limitations of mobile devices; 2) Mixed-precision quantization-based methods utilize reinforcement learning and element-level quantization, which incurs substantial computational and memory costs on mobile devices. To this end, we propose Resource Awareness-Based Salience-Weighted Mixed-Precision Quantization (RAS-MQ). Specifically, RAS-MQ constructs a memory and latency computation incorporating model parameters and device constraints, then RAS-MQ employs integer linear programming for optimal per-layer bit-width allocation. Weight matrices are partitioned into groups with bit-widths assigned based on average saliency ranking. Furthermore, RAS-MQ applies training-free quantization range clipping to salient weights, which can improve representational capacity while reducing quantization error. Experiments on five zero-shot inference datasets, for the 2-bit quantization of LLaMA2-13B, RAS-MQ outperforms the state-of-the-art baseline (SqueezeLLM) by 0.85% in accuracy and reduces weight memory consumption from 24.5 GB to 4.1 GB, achieving a 5.98× compression ratio.
1 January 2023 in International Journal of Sensor Networks