Large Language Models (LLMs), such as LLaMA and GPT-4, have transformed the paradigm of natural language comprehension and generation. Despite their impressive performance, these models still face certain challenges, including the need for extensive data, high computational resources, and privacy concerns related to their data sources. Recently, Federated Learning (FL) has surfaced as a cooperative AI methodology that enables AI training across distributed computation entities while maintaining decentralized data. Integrating FL with LLMs presents an encouraging solution for privacy-preserving and collaborative LLM learning across multiple end-users, thus addressing the aforementioned challenges. In this paper, we provide an exhaustive review of federated Large Language Models, starting from an overview of the latest progress in FL and LLMs, and proceeding to a discourse on their motivation and challenges for integration. We then conduct a thorough review of the existing federated LLM research from the perspective of the entire lifespan, from pre-training to fine-tuning and practical applications. Moreover, we address the threats and issues arising from this integration, shedding light on the delicate balance between privacy and robustness, and introduce existing approaches and potential strategies for enhancing federated LLM privacy and resilience. Finally, we conclude this survey by outlining promising avenues for future research in this emerging field.

Federated Learning (FL) has emerged as a promising framework to address data privacy concerns associated with mobile devices, in contrast to conventional Machine Learning (ML). However, traditional FL encounters significant challenges due to the heterogeneities among different clients. Clustered Federated Learning (CFL) has demonstrated effectiveness in mitigating the data heterogeneity challenge, which significantly limits a broader application of FL. Nevertheless, existing CFL approaches often tightly couple the clustering process with the main FL process, affecting the flexibility and performance of CFL. In this paper, we propose a pre-clustering-based CFL approach, named SnapCFL, which decouples the CFL process into pre-clustering and main FL stages, considering both the impact of heterogeneity on CFL accuracy and the framework's flexibility. The pre-clustering stage models the measurement of data similarity as a two-sample hypothesis testing problem to more accurately group clients and alleviate data heterogeneity. In the main FL stage, a constraint-based client selection method is employed to address the system heterogeneity problem. We conduct extensive experiments using popular datasets with various heterogeneity settings. The results demonstrate that SnapCFL achieves excellent performance in terms of accuracy and efficiency. Compared to five other state-of-the-art approaches, SnapCFL can improve model accuracy by 0.7%$\sim$36.4%, and achieve the same level of accuracy with at least 0.08× the convergence time.

In 2016, the IEEE task group ah (TGah) released a new standard called IEEE 802.11ah, and industrial Internet of Things (IoT) is one of its typical use cases. The restricted access window (RAW) is one of the core MAC mechanisms of IEEE 802.11ah, which aims to address the collision problem in the dense wireless networks. However, in each RAW period, stations still need to contend for the channel by Distributed Coordination Function and Enhanced Distributed Channel Access (DCF/EDCA), which cannot meet the real-time requirements of most industrial applications. In this paper, we propose a channel-aware contention window adaption (CA-CWA) algorithm. The algorithm dynamically adapts the contention window based on the channel status with an external interference discrimination ability, and improves the real-time performance of the IEEE 802.11ah. To validate the real-time performance of CA-CWA, we compared CA-CWA with two other backoff algorithms with an NS-3 simulator. The results illustrate that CA-CWA has better performance than the other two algorithms in terms of packet loss rate and average delay. Compared with the other two algorithms, CA-CWA is able to support industrial applications with higher deadline constraints under the same channel conditions in IEEE 802.11ah.

Human whole-body pose estimation is a challenging task since the model needs to learn more keypoints than the body-only case. To meet the needs of real-time performance while maintaining accuracy is also a hard issue in whole-body pose estimation due to the learning capability of lightweight networks. In order to solve the above problems to a large extent, we propose a light whole-body pose estimation method with an optimized training strategy. The model is designed based on bottom-up architecture as a base network followed by a refinement network. We propose a two-stage training process, which learns rough features in the first stage and then improves estimation precision in the second stage. An online data augmentation procedure is proposed in the second stage to improve refinement performance. We also introduce a separate learning refinement structure that fine-tunes for body, foot, and hand part independently. Experimental results show that our method improves over 8%–10% average precision compared with other lightweight state-of-the-art approaches in the whole-body pose estimation task, with nearly a quarter (25%) size of model parameters saved.

With the rapid growth of the number of devices generating and collecting data, there has been a surge in the large-scale emergence of artificial intelligence (AI) applications predicated on Internet of Things (IoT) networks and terminals. Collaborative Inference is a prospective paradigm for accelerating Deep Neural Network (DNN) inference by harnessing the computational resources of multiple IoT devices. However, in highly lossy network environments, such as those encountered in wireless communication systems, the transmission loss of intermediate feature maps between devices can result in significant degradation of co-inference accuracy. In this paper, we first conduct a comprehensive investigation into the impact of intermediate feature map loss in real-world wireless scenarios and provide an in-depth analysis of loss patterns under UDP transmission. Motivated by these observations, we introduce Robust Co-inference Framework (RCIF), a novel framework that employs a hierarchical mask strategy to selectively drop activations at two different scales of feature maps. This approach enhances the robustness of DNN co-inference in the presence of network losses. Our evaluation on a variety of datasets and network architectures demonstrates that RCIF significantly enhances the accuracy and robustness of distributed DNN co-inference under highly lossy network conditions. Specifically, our results show that RCIF can achieve up to a 659% increase in accuracy compared to the original model under particularly poor network conditions.

As sixth-generation (6G) networks evolve towards AI-native architectures, Federated Learning (FL) is becoming a cornerstone for enabling intelligent services by leveraging distributed data from diverse sources such as Integrated Sensing and Communication (ISAC) devices and edge clients. However, a critical challenge lies in efficiently handling data removal requests, mandated by regulations like the “right to be forgotten”. This problem is significantly exacerbated by the extreme data heterogeneity (non-IID) inherent across diverse 6G devices and the communication constraints of wireless networks. To address these challenges, this paper proposes SeFUL, a novel two-stage federated unlearning framework tailored for the security and privacy demands of 6G systems. SeFUL first proactively mitigates data heterogeneity by partitioning clients into clusters based on their data distribution similarity. Subsequently, a lightweight, information-theoretic unlearning strategy is deployed. This method surgically erases information by optimizing a composite loss function which, in the latent space, pushes the feature representations of forgotten data away from their original class cluster and towards samples from other classes, while reinforcing knowledge from the retain set. Comprehensive experiments on benchmark datasets demonstrate that SeFUL achieves unlearning performance on par with the gold standard of complete model retraining. It successfully reduces forget-set accuracy to nearly random guess levels while preserving high retain-set accuracy, significantly outperforming existing state-of-the-art methods. Furthermore, Membership Inference Attacks (MIAs) confirm that SeFUL effectively reduces privacy risks to a level statistically indistinguishable from a fully retrained model, validating its efficacy as a robust privacy-preserving mechanism.