Federated learning is a popular framework designed to perform the distributed machine learning while protecting client privacy. However, the heterogeneous data distribution in real-world environments makes it difficult to converge when performing model training. In this article, we propose the federated gradient scheduling (FedGS), an improved historical gradient sampling utilization method for optimizers that utilize historical gradients in the federated learning to alleviate the instability problem of historical gradient information due to non-IID. FedGS consists of two main steps to improve federated learning performance. First, clustering uses clients’ label distributions, which relabel clients and their submitting gradients. Second, sampling gradient clusters to generate an IID gradient set, which feeds to optimizers to derive valid momentum information. Besides, we introduce differential privacy to collaborate with FedGS to enhance clients’ privacy protection strength. Compared to previous non-IID federated learning solutions, our method can achieve greater resistance to temporal non-IID. Moreover, experiments show that FedGS can achieve faster convergence and performance gain of up to 10% over existing state-to-art methods in some scenarios. FedGS can combine with existing methods easily to achieve better performance. We further verify that our method has robust performance gains in different non-IID scenarios, demonstrating the adaptability of FedGS for different scenarios.

Centralized learning now faces data mapping and security constraints that make it difficult to carry out. Federated learning with a distributed learning architecture has changed this situation. By restricting the training process to participants’ local, federated learning addresses the model training needs of multiple data sources while better protecting data privacy. However, in real-world application scenarios, federated learning faces the need to achieve fairness in addition to privacy protection. In practice, it could happen that some federated learning participants with specific motives may short join the training process to obtain the current global model with only limited contribution to the federated learning whole, resulting in unfairness to the participants who previously participated in federated learning. We propose the FedACC framework with a server-initiated global model accuracy control method to address this issue. Besides measuring the accumulative contributions of newly joined participants and providing participants with a model with an accuracy that matches their contributions, the FedACC still guarantees the validity of participant gradients based on the accuracy-decayed model. Under the FedACC framework, users do not have access to the full version of the current global model early in their training participation. However, they must produce a certain amount of contributions before seeing the full-accuracy model. We introduce an additional differential privacy mechanism to protect clients’ privacy further. Experiments demonstrate that the FedACC could obtain about 10% to 20% accuracy gain compared to the state-of-art methods while balancing the fairness, performance, and security of federated learning.

Federated learning (FL) provides a learning framework without participants sharing local raw data, but individual privacy is still at risk of disclosure through attacking the trained models. Due to the strong privacy guarantee, differential privacy (DP) is widely applied to FL to avoid privacy leakage. Traditional private learning adds noise directly to the gradients. The continuous accumulated noise on parameter models severely impairs learning effectiveness. To solve this problem, we introduce the idea of differentially private continuous data release (DPCR) into FL and propose an FL framework based on DPCR (FL-DPCR). Meanwhile, our proposed Equivalent Aggregation Theorem demonstrates that DPCR effectively reduces the overall error added to parameter models and improves FL's accuracy. To improve FL-DPCR's learning effectiveness, we introduce Matrix Mechanism to construct a release strategy and design a binary-indexed-tree (BIT) based DPCR model for Gaussian mechanism (BCRG). By solving a complex nonlinear programming problem with negative exponents, BCRG achieves optimal release accuracy efficiently. Besides, we exploit the residual privacy budget to boost the accuracy further and propose an advanced BCRG version (ABCRG). Our experiments show that, compared to traditional FL with DP, our achievements improve the accuracy with gains ranging from $3.4\%$ on FMNIST to $65.7\%$ on PAMAP2.

Integrating data from multiple parties to achieve cross-institutional machine learning is an important trend in Industry 4.0 era. However, the privacy risks from sharing data pose a significant challenge to data integration. To integrate data without sharing data and meet large-scale samples' modeling needs, we propose two vertical federation learning algorithms for ridge regression via least-squares solution for two-party and multi-party scenarios, respectively. Compared with the state-of-the-art algorithms, our algorithms only need one round of calculation for the optimization instead of iteration. Furthermore, our algorithms can effectively handle large-scale samples due to the number of cryptographic operations in our algorithms being independent of the number of samples. Through our proposed the matrix secure agent computing theory and $\delta$ -data indistinguishability theory, we provide quantitative theoretical guarantees for the security of our algorithms. Our algorithms satisfy complete data indistinguishability under the “semi-honest” assumption and the quantitative security under the “malicious” assumption. The experiments show that our proposed algorithm takes only about 400 seconds to handle up to 9.6 million large-scale samples, while the state-of-the-art algorithms take close to 1000 seconds to handle every 1000 samples, which embodies the advantage of our algorithms in handling large-scale samples.