Large Language Models (LLMs) have showcased exceptional capabilities in various domains, attracting significant interest from both academia and industry. Despite their impressive performance, the substantial size and computational demands of LLMs pose considerable challenges for practical deployment, particularly in environments with limited resources. The endeavor to compress language models while maintaining their accuracy has become a focal point of research. Among the various methods, knowledge distillation has emerged as an effective technique to enhance inference speed without greatly compromising performance. This article presents a thorough survey from three aspects: method, evaluation, and application, exploring knowledge distillation techniques tailored specifically for LLMs. Specifically, we divide the methods into white-box KD and black-box KD to better illustrate their differences. Furthermore, we also explored the evaluation tasks and distillation effects between different distillation methods and proposed directions for future research. Through in-depth understanding of the latest advancements and practical applications, this survey provides valuable resources for researchers, paving the way for sustained progress in this field.

Federated learning (FL) has attracted increasing attention to building models without accessing raw user data, especially in healthcare. In real applications, different federations can seldom work together due to possible reasons such as data heterogeneity and distrust/inexistence of the central server. In this article, we propose a novel framework called MetaFed to facilitate trustworthy FL between different federations. MetaFed obtains a personalized model for each federation without a central server via the proposed cyclic knowledge distillation. Specifically, MetaFed treats each federation as a meta distribution and aggregates knowledge of each federation in a cyclic manner. The training is split into two parts: common knowledge accumulation and personalization. Comprehensive experiments on seven benchmarks demonstrate that MetaFed without a server achieves better accuracy compared with state-of-the-art methods [e.g., 10%+ accuracy improvement compared with the baseline for physical activity monitoring dataset (PAMAP2)] with fewer communication costs. More importantly, MetaFed shows remarkable performance in real-healthcare-related applications.

There is a growing interest in applying machine learning techniques to healthcare. Recently, federated machine learning (FL) is gaining popularity since it allows researchers to train powerful models without compromising data privacy and security. However, the performance of existing FL approaches often deteriorates when encountering non-iid situations where there exist distribution gaps among clients, and few previous efforts focus on personalization in healthcare. In this article, we propose FedAP to tackle domain shifts and obtain personalized models for local clients. FedAP learns the similarity between clients via the statistics of the batch normalization layers while preserving the specificity of each client with different local batch normalization. Comprehensive experiments on five healthcare benchmarks demonstrate that FedAP achieves better accuracy compared to state-of-the-art methods (e.g., 10%+ accuracy improvement for PAMAP2) with faster convergence speed.

Surface electromyography (sEMG) armband-based gesture recognition is an active research topic that aims to identify hand gestures with a single row of sEMG electrodes. As a typical type of biological signal, sEMG on one channel is nonstationary temporally and related to multiple adjacent muscles spatially, which hinders the effective representation in gesture recognition. To tackle these aspects, we propose a spatial–temporal features-based gesture recognition method (STF-GR) in this article. Specifically, STF-GR first decomposes the nonstationary multichannel sEMG by multivariate empirical mode decomposition, which jointly transforms each channel into a series of stationary subsignals. It can keep the temporal stationarity within-channel as well as the spatial independence across-channel. Then, by the convolutional recurrent neural network, STF-GR extracts and merges spatial–temporal features of decomposed sEMG signal. Finally, a negative log-likelihood-based cost function is used to make the final gesture decision. To evaluate the performance of STF-GR, we conduct experiments on three data sets, noninvasive adaptive hand prosthetic (NinaPro), CapgMyo, and BandMyo. The first two are publicly available, and BandMyo is collected by ourselves. Experimental evaluations with within-subject tests show that STF-GR exceeds the performance of other state-of-the-art methods, including deep learning algorithms that are not focused on spatial–temporal features and traditional machine learning algorithms that use handcrafted features.

Human activity recognition requires the efforts to build a generalizable model using the training datasets with the hope to achieve good performance in test datasets. However, in real applications, the training and testing datasets may have totally different distributions due to various reasons such as different body shapes, acting styles, and habits, damaging the model’s generalization performance. While such a distribution gap can be reduced by existing domain adaptation approaches, they typically assume that the test data can be accessed in the training stage, which is not realistic. In this article, we consider a more practical and challenging scenario: domain-generalized activity recognition (DGAR) where the test dataset cannot be accessed during training. To this end, we propose Adaptive Feature Fusion for Activity Recognition (AFFAR) , a domain generalization approach that learns to fuse the domain-invariant and domain-specific representations to improve the model’s generalization performance. AFFAR takes the best of both worlds where domain-invariant representations enhance the transferability across domains and domain-specific representations leverage the model discrimination power from each domain. Extensive experiments on three public HAR datasets show its effectiveness. Furthermore, we apply AFFAR to a real application, i.e., the diagnosis of Children’s Attention Deficit Hyperactivity Disorder (ADHD), which also demonstrates the superiority of our approach.

Time series remains one of the most challenging modalities in machine learning research. Out-of-distribution (OOD) detection and generalization on time series often face difficulties due to their non-stationary nature, wherein the distribution changes over time. The dynamic distributions within time series present significant challenges for existing algorithms, especially in identifying invariant distributions, as most focus on scenarios where domain information is provided as prior knowledge. This paper aims to address the issues induced by non-stationarity in time series through the exploration of subdomains within a complete dataset for generalized representation learning. We propose Diversify , a general framework, for OOD detection and generalization on dynamic distributions of time series. Diversify operates through an iterative process: first identifying the ’worst-case’ latent distribution scenario, then working to minimize the gaps between these latent distributions. We implement Diversify by combining existing OOD detection methods according to either extracted features or outputs of models for detection while we also directly utilize outputs for classification. Theoretical insights support the framework's validity. Extensive experiments are conducted on seven datasets with different OOD settings across gesture recognition, speech commands recognition, wearable stress and affect detection, and sensor-based human activity recognition. Qualitative and quantitative results demonstrate that Diversify learns more generalized features and significantly outperforms other baselines.