1 November 2022 in IEEE Access
Compared with traditional knowledge distillation, which relies on a large amount of data, few-shot knowledge distillation can distill student networks with good performance using only a small number of samples. Some recent studies treat the network as a combination of a series of network blocks, adopt a progressive graft strategy, and use the output of the teacher network to distill the student network. However, this strategy ignores the importance of the local feature information generated by the teacher block, which indicates what features should be learned by the corresponding student block. In this paper, we argue that using the features output from the teacher block can guide the student block to further learn more useful information from the teacher block. Therefore, we propose a joint learning framework for few-shot knowledge distillation that exploits both the output of the teacher network and the local features generated by the teacher block to optimize the student network. The local features will guide the student block to learn the output of the teacher block, and the output of the teacher network will allow the student network to take its learned local features to better contribute to the classification. In addition, further model compression was carried out to design a series of student networks with fewer number of parameters by reducing the number of network channels. Finally, extensive experiments using the model on CIFAR10 and CIFAR100 datasets show that our method outperforms SOTA, and our method has considerable advantages even with a very small number of parameters in further model compression experiments.
23 January 2025 in IEEE Access
Deploying deep learning models for industrial defect detection is challenging, as smaller models lack precision and larger models require significant computational resources. To address this, knowledge distillation has been used to transfer knowledge from a larger teacher model to a smaller student model, though its effectiveness is limited by inaccurate teacher predictions and inadequate handling of difficult samples. In this paper, a novel method, Distribution-Corrected Knowledge Distillation (DCKD), is proposed to overcome these issues. A Logits Distribution Correction (LDC) Module is introduced to correct inaccuracies in the teacher model’s predictions, and an Iterative Hard Sample Aggregation (IHA) Module is used to enhance the student model’s ability to handle challenging cases. This approach significantly improves detection accuracy across multiple industrial datasets, surpassing state-of-the-art distillation techniques. Notably, the student model outperforms the teacher model on the NEU and CG-DF datasets, with mAP scores improving from 0.766 and 0.716 to 0.794 and 0.747, respectively.