Knowledge Amalgamation from Heterogeneous Networks by Common Feature Learning

Knowledge Amalgamation from Heterogeneous Networks by Common Feature Learning

Sihui Luo, Xinchao Wang, Gongfan Fang, Yao Hu, Dapeng Tao, Mingli Song

Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence
Main track. Pages 3087-3093. https://doi.org/10.24963/ijcai.2019/428

An increasing number of well-trained deep networks have been released online by researchers and developers, enabling the community to reuse them in a plug-and-play way without accessing the training annotations. However, due to the large number of network variants, such public-available trained models are often of different architectures, each of which being tailored for a specific task or dataset. In this paper, we study a deep-model reusing task, where we are given as input pre-trained networks of heterogeneous architectures specializing in distinct tasks, as teacher models. We aim to learn a multitalented and light-weight student model that is able to grasp the integrated knowledge from all such heterogeneous-structure teachers, again without accessing any human annotation. To this end, we propose a common feature learning scheme, in which the features of all teachers are transformed into a common space and the student is enforced to imitate them all so as to amalgamate the intact knowledge. We test the proposed approach on a list of benchmarks and demonstrate that the learned student is able to achieve very promising performance, superior to those of the teachers in their specialized tasks.
Keywords:
Machine Learning: Developmental Learning
Machine Learning: Transfer, Adaptation, Multi-task Learning
Machine Learning: Multi-instance;Multi-label;Multi-view learning
Machine Learning: Feature Selection ; Learning Sparse Models
Machine Learning: Deep Learning