Knowledge distillation (KD) is crucial for remote sensing image (RSI) classification, particularly as the operating environment in remote sensing is often constrained by hardware limitations. However, prior research has not fully addressed the challenge of leveraging KD to develop lightweight, high-accuracy models for RSI classification. A key issue is the sparse distribution of training data, which often results in asymmetry within the data. This asymmetry impedes the transfer of prior knowledge during the distillation process, diminishing the overall efficacy of KD techniques. To overcome this challenge, we propose a novel, symmetry-enhanced approach that augments the logit-based KD process, improving its effectiveness and efficiency for RSI classification. Our method is distinguished by three core innovations: a symmetrically generative algorithm to enhance the symmetry of the training data, an efficient algorithm for constructing a robust teacher ensemble model, and a quantitative technique for feature alignment. Rigorous evaluations on three benchmark datasets demonstrate that our method outperforms 14 existing KD-based approaches and 30 other state-of-the-art methods. Specifically, the student model trained with our approach achieves accuracy improvements of up to 22.5% while reducing the model size and inference time by as much as 96% and 88%, respectively. In conclusion, this research makes a significant contribution to RSI classification by introducing an efficient and effective data symmetry-driven method to enhance the knowledge transferring efficiency of the logit-based KD process.
In various fields, knowledge distillation (KD) techniques that combine vision transformers (ViTs) and convolutional neural networks (CNNs) as a hybrid teacher have shown remarkable results in classification. However, in the realm of remote sensing images (RSIs), existing KD research studies are not only scarce but also lack competitiveness. This issue significantly impedes the deployment of the notable advantages of ViTs and CNNs. To tackle this, the authors introduce a novel hybrid‐model KD approach named HMKD‐Net, which comprises a CNN‐ViT ensemble teacher and a CNN student. Contrary to popular opinion, the authors posit that the sparsity in RSI data distribution limits the effectiveness and efficiency of hybrid‐model knowledge transfer. As a solution, a simple yet innovative method to handle variances during the KD phase is suggested, leading to substantial enhancements in the effectiveness and efficiency of hybrid knowledge transfer. The authors assessed the performance of HMKD‐Net on three RSI datasets. The findings indicate that HMKD‐Net significantly outperforms other cutting‐edge methods while maintaining a significantly smaller size. Specifically, HMKD‐Net exceeds other KD‐based methods with a maximum accuracy improvement of 22.8% across various datasets. As ablation experiments indicated, HMKD‐Net has cut down on time expenses by about 80% in the KD process. This research study validates that the hybrid‐model KD technique can be more effective and efficient if the data distribution sparsity in RSIs is well handled.