The development of high-quality deep learning models demands the transfer of user data from edge devices, where it originates, to centralized servers. This central training approach has scalability limitations and poses privacy risks to private data. Federated Learning (FL) is a distributed training framework that empowers physical smart systems devices to collaboratively learn a task without sharing private training data with a central server. However, FL introduces new challenges to Beyond 5G (B5G) networks, such as communication overhead, system heterogeneity, and privacy concerns, as the exchange of model updates may still lead to data leakage. This paper explores the communication overhead and privacy risks facing the implementation of FL and presents an algorithm that encompasses Knowledge Distillation (KD) and Differential Privacy (DP) techniques to address these challenges in FL. We compare the operational flow and network model of model-based and model-agnostic (KD-based) FL algorithms that enable customizing per-client model architecture to accommodate heterogeneous and constrained system resources. Our experiments show that KD-based FL algorithms are able to exceed local accuracy and achieve comparable accuracy to central training. Additionally, we show that applying DP to KDbased FL significantly damages its utility, leading to up to 70% accuracy loss for a privacy budget $\epsilon \leq 10$ .
Deep learning-based Human Activity Recognition (HAR) systems received a lot of interest for health monitoring and activity tracking on wearable devices. The availability of large and representative datasets is often a requirement for training accurate deep learning models. To keep private data on users’ devices while utilizing them to train deep learning models on huge datasets, Federated Learning (FL) was introduced as an inherently private distributed training paradigm. However, standard FL (FedAvg) lacks the capability to train heterogeneous model architectures. In this paper, we propose Federated Learning via Augmented Knowledge Distillation (FedAKD) for distributed training of heterogeneous models. FedAKD is evaluated on two HAR datasets: A waist-mounted tabular HAR dataset and a wrist-mounted time-series HAR dataset. FedAKD is more flexible than standard federated learning (FedAvg) as it enables collaborative heterogeneous deep learning models with various learning capacities. In the considered FL experiments, the communication overhead under FedAKD is 200X less compared with FL methods that communicate models’ gradients/weights. Relative to other model-agnostic FL methods, results show that FedAKD boosts performance gains of clients by up to 20 percent. Furthermore, FedAKD is shown to be relatively more robust under statistical heterogeneous scenarios.
The adoption of IoT and monitoring devices in 5G and beyond networks has been widespread. Unmanned Aerial vehicles (UAVs) have shown success in connecting rural and remote areas due to the high cost of deploying infrastructures like cellular network base stations and optical fiber connections in vast landscapes with sparse populations. The constrained energy of UAVs results in limited coverage area and flight time, which in turn reduces the potential of UAVs to provide task-oriented wireless communication links. In this paper, we explore path optimization and transmission organization algorithms to minimize flight time and extend the range of UAVs performing collaborative Federated Learning (FL) among geographically dispersed nodes communicating through wireless connections offered by UAVs coupled with Device-to-Device (D2D) networks. The UAV orchestrates federated learning (FL) between spatially scattered homes via LoRa wireless communication. We formulate the drone path optimization as a Traveling Salesman Problem (TSP) and employ Self-Organizing Maps (SOM) for path planning. Additionally, Knowledge Distillation (KD)-based FL is used to reduce communication overhead for the resource-constrained UAV-IoT system. Experimental results demonstrate SOM’s ability to represent the topological structure of nodes and produce a cost-efficient Hamiltonian cycle, from which the drone path is derived. Our results demonstrate the communication efficiency and utility of KD-based FL compared to model-based FL methods. The proposed hybrid solution enables energy-constrained UAVs to perform FL over large areas leveraging a shared dataset for KD and a SOM-based path optimization algorithm.
Medical image segmentation aims to identify important or suspicious regions within medical images. However, many challenges are usually faced while developing networks for this type of analysis. First, preserving the original image resolution is of utmost importance for this task where identifying subtle features or abnormalities can significantly impact the accuracy of diagnosis. The introduction of the dilated convolution module helped preserve resolution in deep convolutional neural networks, but it has a drawback: loss of local spatial resolution due to increased kernel sparsity in checkboard patterns. To address this, in this work, a double-dilated convolution module is proposed to maintain local spatial resolution while achieving a large receptive field. This approach is applied to tumor segmentation in breast cancer mammograms as a proof-of-concept. Additionally, this study tackles the issue of pixel-level class imbalance in mammogram screenings by comparing various loss functions to find the best one for mass segmentation. Our work also addresses the "black-box" nature of the models by performing quantitative and qualitative evaluations of their interpretability using the Gradient weighted Class Activation Map (Grad-CAM) and comparing it with other explainable models for image segmentation. An experimental analysis on lesion segmentation networks is performed on mammogram screenings from the INBreast dataset, both before and after integrating the proposed dilation module into a state-of-the-art deep convolutional neural network. The results demonstrate the effectiveness of the proposed module in terms of Dice similarity and Miss Detection rate for mass segmentation. Our analysis also promotes using the Tversky Loss function in training pixel-imbalanced data and integrating Grad-CAM for explaining image segmentation results.
Integrating machine learning with the Internet of Things (IoT) enables many useful applications. For IoT applications that incorporate video content analysis (VCA), deep learning models are usually used due to their capacity to encode the high-dimensional spatial and temporal representations of videos. However, limited energy and computation resources present a major challenge. Video captioning is one type of VCA that describes a video with a sentence or a set of sentences. This work proposes an IoT-based deep learning-based framework for video captioning that can (1) Mine large open-domain video-to-text datasets to extract video-caption pairs that belong to a particular domain. (2) Preprocess the selected video-caption pairs including reducing the complexity of the captions’ language model to improve performance. (3) Propose two deep learning models: A transformer-based model and an LSTM-based model. Hyperparameter tuning is performed to select the best hyperparameters. Models are evaluated in terms of accuracy and inference time on different platforms. The presented framework generates captions in standard sentence templates to facilitate extracting information in later stages of the analysis. The two developed deep learning models offer a trade-off between accuracy and speed. While the transformer-based model yields a high accuracy of 97%, the LSTM-based model achieves near real-time inference.
Emerging Internet of Things (IoT) applications, such as sensor-based Human Activity Recognition (HAR) systems, require efficient machine learning solutions due to their resource- constrained nature which raises the need to design heterogeneous model architectures. Federated Learning (FL) has been used to train distributed deep learning models. However, standard federated learning (fedAvg) does not allow the training of het-erogeneous models. Our work addresses the model and statistical heterogeneities of distributed HAR systems. We propose a Federated Learning via Augmented Knowledge Distillation (FedAKD) algorithm for heterogeneous HAR systems and evaluate it on a self-collected sensor-based HAR dataset. Then, Kullback-Leibler (KL) divergence loss is compared with Mean Squared Error (MSE) loss for the Knowledge Distillation (KD) mechanism. Our experiments demonstrate that MSE contributes to a better KD loss than KL. Experiments show that FedAKD is communication-efficient compared with model-dependent FL algorithms and outperforms other KD-based FL methods under the i.i.d. and non-i.i.d. scenarios.