Knowledge distillation has proven to be an effective model compression method that exploits the knowledge from a teacher model for supervising a student model by minimizing the distribution difference between the knowledge and the prediction produced by the student model. In this work, we focus on improving its performance from the perspective of divergence measures. A general form that represents a family of divergence measures is introduced and a novel learning paradigm that jointly optimizes multiple measures is proposed by formalizing it as a multi-objective learning problem. Conditioned on Pareto optimality, the weights of different divergences are tuned in an automated way during training. Extensive experiments on multiple datasets show the proposed method can significantly improve the performance of student network compared to the state-of-the-art methods for knowledge distillation.
The realization of the third-generation artificial intelligence (AI) requires the evolution from perceptual intelligence to cognitive intelligence, where knowledge graphs may not meet the practical needs anymore. Based on the dual channel theory, cognitive graphs are established and developed through coordinating the implicit extraction module and the explicit reasoning module as well as integrating knowledge graphs, cognitive reasoning and logical expressions, which have achieved successes in multi-hop question answering. It is desired for cognitive graphs to be widely used in advanced AI applications such as large-scale knowledge representations and intelligent responses, promoting the development of Al dramatically. This review discusses cognitive graphs systematically and elaborately, including basic concepts, generations, theories and technologies. Moreover, we try to predict the development of cognitive intelligence in the short-term future and further enlighten more researches and studies.