Large Language Models (LLMs), such as LLaMA and GPT-4, have transformed the paradigm of natural language comprehension and generation. Despite their impressive performance, these models still face certain challenges, including the need for extensive data, high computational resources, and privacy concerns related to their data sources. Recently, Federated Learning (FL) has surfaced as a cooperative AI methodology that enables AI training across distributed computation entities while maintaining decentralized data. Integrating FL with LLMs presents an encouraging solution for privacy-preserving and collaborative LLM learning across multiple end-users, thus addressing the aforementioned challenges. In this paper, we provide an exhaustive review of federated Large Language Models, starting from an overview of the latest progress in FL and LLMs, and proceeding to a discourse on their motivation and challenges for integration. We then conduct a thorough review of the existing federated LLM research from the perspective of the entire lifespan, from pre-training to fine-tuning and practical applications. Moreover, we address the threats and issues arising from this integration, shedding light on the delicate balance between privacy and robustness, and introduce existing approaches and potential strategies for enhancing federated LLM privacy and resilience. Finally, we conclude this survey by outlining promising avenues for future research in this emerging field.

Federated Learning (FL) has emerged as a promising framework to address data privacy concerns associated with mobile devices, in contrast to conventional Machine Learning (ML). However, traditional FL encounters significant challenges due to the heterogeneities among different clients. Clustered Federated Learning (CFL) has demonstrated effectiveness in mitigating the data heterogeneity challenge, which significantly limits a broader application of FL. Nevertheless, existing CFL approaches often tightly couple the clustering process with the main FL process, affecting the flexibility and performance of CFL. In this paper, we propose a pre-clustering-based CFL approach, named SnapCFL, which decouples the CFL process into pre-clustering and main FL stages, considering both the impact of heterogeneity on CFL accuracy and the framework's flexibility. The pre-clustering stage models the measurement of data similarity as a two-sample hypothesis testing problem to more accurately group clients and alleviate data heterogeneity. In the main FL stage, a constraint-based client selection method is employed to address the system heterogeneity problem. We conduct extensive experiments using popular datasets with various heterogeneity settings. The results demonstrate that SnapCFL achieves excellent performance in terms of accuracy and efficiency. Compared to five other state-of-the-art approaches, SnapCFL can improve model accuracy by 0.7%$\sim$36.4%, and achieve the same level of accuracy with at least 0.08× the convergence time.

It is significant to study power device package fatigue failure as it seriously affects the reliability of power system. Nevertheless, the research of power device failure process is insufficient. In this paper, an accurate and intelligent approach is proposed to predict the power device fatigue failure process with multiple fatigue sampling method (MFSM) and minimal component unit method (MCUM). MFSM is proposed to accurately build the power device lifetime model. It is accomplished through multiple sampling fatigue morphology evolution process of solder layers combined with the fatigue parameter. Morphology evolution is detected by scanning acoustic microscope (SAM) technology under accelerated lifetime test (ALT). The fatigue parameter is got through finite element analysis (FEA) by establishing each sampling geometry model. Then, the lifetime model is determined by their same failure area fraction ( F _s ). In particular, digital image processing (DIP) is applied to detailly describe solder layer shapes which is also the key to building a real FEA geometry model. MCUM is utilized to complete the prediction of failure process, where solder layers are divided into minimal units and the FEA solution and location information of each unit are known. Based on lifetime model, the failure area can be got and the fatigue failure process can be finished intelligently by cosimulation. The proposed method is accurate and intelligent enough in predicting the failure of solder layers which is more helpful for planned device management.

Studies over the past decade have generated a wealth of molecular data that can be leveraged to better understand cancer risk, progression, and outcomes. However, understanding the progression risk and differentiating long- and short-term survivors cannot be achieved by analyzing data from a single modality due to the heterogeneity of disease. Using a scientifically developed and tested deep-learning approach that leverages aggregate information collected from multiple repositories with multiple modalities (e.g., mRNA, DNA Methylation, miRNA) could lead to a more accurate and robust prediction of disease progression. Here, we propose an autoencoder based multimodal data fusion system, in which a fusion encoder flexibly integrates collective information available through multiple studies with partially coupled data. Our results on a fully controlled simulation-based study have shown that inferring the missing data through the proposed data fusion pipeline allows a predictor that is superior to other baseline predictors with missing modalities. Results have further shown that short- and long-term survivors of glioblastoma multiforme, acute myeloid leukemia, and pancreatic adenocarcinoma can be successfully differentiated with an AUC of 0.94, 0.75, and 0.96, respectively.

Human whole-body pose estimation is a challenging task since the model needs to learn more keypoints than the body-only case. To meet the needs of real-time performance while maintaining accuracy is also a hard issue in whole-body pose estimation due to the learning capability of lightweight networks. In order to solve the above problems to a large extent, we propose a light whole-body pose estimation method with an optimized training strategy. The model is designed based on bottom-up architecture as a base network followed by a refinement network. We propose a two-stage training process, which learns rough features in the first stage and then improves estimation precision in the second stage. An online data augmentation procedure is proposed in the second stage to improve refinement performance. We also introduce a separate learning refinement structure that fine-tunes for body, foot, and hand part independently. Experimental results show that our method improves over 8%–10% average precision compared with other lightweight state-of-the-art approaches in the whole-body pose estimation task, with nearly a quarter (25%) size of model parameters saved.

With the rapid growth of the number of devices generating and collecting data, there has been a surge in the large-scale emergence of artificial intelligence (AI) applications predicated on Internet of Things (IoT) networks and terminals. Collaborative Inference is a prospective paradigm for accelerating Deep Neural Network (DNN) inference by harnessing the computational resources of multiple IoT devices. However, in highly lossy network environments, such as those encountered in wireless communication systems, the transmission loss of intermediate feature maps between devices can result in significant degradation of co-inference accuracy. In this paper, we first conduct a comprehensive investigation into the impact of intermediate feature map loss in real-world wireless scenarios and provide an in-depth analysis of loss patterns under UDP transmission. Motivated by these observations, we introduce Robust Co-inference Framework (RCIF), a novel framework that employs a hierarchical mask strategy to selectively drop activations at two different scales of feature maps. This approach enhances the robustness of DNN co-inference in the presence of network losses. Our evaluation on a variety of datasets and network architectures demonstrates that RCIF significantly enhances the accuracy and robustness of distributed DNN co-inference under highly lossy network conditions. Specifically, our results show that RCIF can achieve up to a 659% increase in accuracy compared to the original model under particularly poor network conditions.