The complexity of deep neural network models (DNNs) severely limits their application on devices with limited computing and storage resources. Knowledge distillation (KD) is an attractive model compression technology that can effectively alleviate this problem. Multi-teacher knowledge distillation (MKD) aims to leverage the valuable and diverse knowledge distilled by multiple teacher networks to improve the performance of the student network. Existing approaches typically rely on simple methods such as averaging the prediction logits or using sub-optimal weighting strategies to fuse distilled knowledge from multiple teachers. However, employing these techniques cannot fully reflect the importance of teachers and may even mislead student’s learning. To address this issue, we propose a novel Decoupled Multi-Teacher Knowledge Distillation based on Entropy (DE-MKD). DE-MKD decouples the vanilla knowledge distillation loss and assigns adaptive weights to each teacher to reflect its importance based on the entropy of their predictions. Furthermore, we extend the proposed approach to distill the intermediate features from multiple powerful but cumbersome teachers to improve the performance of the lightweight student network. Extensive experiments on the publicly available CIFAR-100 image classification benchmark dataset with various teacher-student network pairs demonstrated the effectiveness and flexibility of our approach. For instance, the VGG8|ShuffleNetV2 model trained by DE-MKD reached 75.25%|78.86% top-one accuracy when choosing VGG13|WRN40-2 as the teacher, setting new performance records. In addition, surprisingly, the distilled student model outperformed the teacher in both teacher-student network pairs.

Transferring the extensive and varied knowledge contained within multiple complex models into a more compact student model poses significant challenges in multi-teacher knowledge distillation. Traditional distillation approaches often fall short in this context, as they struggle to fully capture and integrate the wide range of valuable information from each teacher. The variation in the knowledge offered by various teacher models complicates the student model’s ability to learn effectively and generalize well, ultimately resulting in subpar results. To overcome these constraints, We introduce an innovative method that integrates both localized and globalized frequency attention techniques, aiming to substantially enhance the distillation process. By simultaneously focusing on fine-grained local details and broad global patterns, our approach allows the student model to more effectively grasp the complex and diverse information provided by each teacher, therefore enhancing its learning capability. This dual-attention mechanism allows for a more balanced assimilation of specific details and generalized concepts, resulting in a more robust and accurate student model. Extensive experimental evaluations on standard benchmarks demonstrate that our methodology reliably exceeds the performance of current multi-teacher distillation methods, yielding outstanding outcomes regarding both performance and robustness. Specifically, our approach achieves an average performance improvement of 0.55% over CA-MKD, with a 1.05% gain in optimal conditions. These findings suggest that frequency-based attention mechanisms can unlock new potential in knowledge distillation, model compression, and transfer learning.

Recently, learning-based image inpainting has gained much attention. It widely utilizes an auto-encoder structure and can obtain compact feature representation in the encoder to achieve high-quality image inpainting. Although this approach has achieved encouraging inpainting results, it inevitably reduces the high-resolution representation due to interval downsampling. In order to solve this problem and achieve an excellent image inpainting effect, this paper proposes a brand-new generative network, Interactive Separation Network, which retains the high-resolution information and extracts the semantic features from corrupted images. Furthermore, this paper also discusses network designs with different complexity in different application scenarios. Finally, to improve the effectiveness and robustness of our proposal to large corrupted regions in the inpainting image, we further propose a flexible and highly reusable reconstruction scheme to complete the inpainting in the prediction process gradually. Experiments show that our proposed generation network and reconstruction scheme can significantly improve the quality of repaired images. The proposed method significantly outperforms the state-of-the-art image inpainting approaches in image quality.

The main content of Text-guided Image Manipulation (TGIM) research is the use of textual information to modify the corresponding content in the input image. Based on generative adversarial networks (GAN), this research has achieved impressive manipulation performance. Nevertheless, the quality of image manipulation still needs to be further improved. In this paper, an attention-driven TGIM method is proposed to further improve the quality of image manipulation. Specifically, the proposed method uses an attention mechanism to fine-tune the whole process of image manipulation at the word level. Through attentional fine-tuning, the quality of image manipulation can be continuously improved to realize high-quality image manipulation effects. The proposed method is experimentally validated on a public Caltech-UCSD birds-200-2011 (CUB) dataset, and the qualitative and quantitative comparison results demonstrate the superior performance of the proposed method on TGIM. Compared to the existing TGIM methods, the proposed method improves the Inception Score (IS) by 22.6% and reduces Fréchet Inception Distance (FID) by 13.4%.