Progressive Blockwise Knowledge Distillation for Neural Network Acceleration

Progressive Blockwise Knowledge Distillation for Neural Network Acceleration

Hui Wang, Hanbin Zhao, Xi Li, Xu Tan

Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence
Main track. Pages 2769-2775. https://doi.org/10.24963/ijcai.2018/384

As an important and challenging problem in machine learning and computer vision, neural network acceleration essentially aims to enhance the computational efficiency without sacrificing the model accuracy too much. In this paper, we propose a progressive blockwise learning scheme for teacher-student model distillation at the subnetwork block level. The proposed scheme is able to distill the knowledge of the entire teacher network by locally extracting the knowledge of each block in terms of progressive blockwise function approximation. Furthermore, we propose a structure design criterion for the student subnetwork block, which is able to effectively preserve the original receptive field from the teacher network. Experimental results demonstrate the effectiveness of the proposed scheme against the state-of-the-art approaches.
Keywords:
Machine Learning: Deep Learning
Computer Vision: Computer Vision