This article presents and characterizes an Open Application Repository for Federated Learning (OARF), a benchmark suite for federated machine learning systems. Previously available benchmarks for federated learning (FL) have focused mainly on synthetic datasets and use a limited number of applications. OARF mimics more realistic application scenarios with publicly available datasets as different data silos in image, text, and structured data. Our characterization shows that the benchmark suite is diverse in data size, distribution, feature distribution, and learning task complexity. The extensive evaluations with reference implementations show the future research opportunities for important aspects of FL systems. We have developed reference implementations, and evaluated the important aspects of FL, including model accuracy, communication cost, throughput, and convergence time. Through these evaluations, we discovered some interesting findings such as FL can effectively increase end-to-end throughput. The code of OARF is publicly available on GitHub. ¹

Federated Learning (FL) is a popular distributed machine learning paradigm that allows multiple participants to collaboratively train a model without sharing the raw data. In each round of FL, all participants train local models in parallel, and a server aggregates the local models to create the global model. Neuron matching is a promising aggregation method that utilizes permutational invariance to improve the quality of the global model and reduces communication costs. However, existing neural matching approaches have two strict limitations: 1) They are only applicable to models with simple sequential networks and cannot support advanced models such as ResNet. 2) Their approaches significantly increase the model size after matching, which causes massive memory and communication costs as the training proceeds. In this paper, we propose a novel neuron matching algorithm called federated generalized matched averaging (FedGMA) to mitigate these limitations. Our experiments show that the proposed method is applicable to complex network structures such as ResNet and InceptionNet, and achieves accuracy comparable to algorithms that directly average the entire model. It also reduces communication costs and demonstrates robustness in data heterogeneity scenarios.