Amalgamating Filtered Knowledge: Learning Task-customized Student from Multi-task Teachers

1 Introduction

In recent years, deep learning has achieved state-of-the-art performances in most if not all of the artificial intelligent applications. The success of deep learning, however, heavily relies on a massive number of human annotations, sometimes even up to the scales of tens of millions such as those of ImageNet. In many cases, the training data or annotations are confidential and therefore not available to the public. To alleviate the re-training and reproducing effort, many developers have now released their trained models online, so that users may use them directly or finetune them on a new dataset.

In this paper, we study how to take advantage of such pretrained networks to produce a new and versatile network, without having access to any human annotation. Our problem setup is as follows. We assume that we are given a pool of pre-trained networks, like those downloaded from the model zoo, as teacher models. Each such teacher may be either a single- or multi-task model, where the former can be treated as a degenerated case of the latter. We aim to train a customized multi-task student model, termed TargetNet, which masters an arbitrary set of the teachers' expertise, using only the provided teacher models and without human annotations. In other words, the student amalgamates the multi-domain knowledge, relevant to its pre-designated customized task, from the multiple teachers that specialize in different tasks.

The proposed task therefore finds its important application in CNN reusing, as it requires no human annotation yet aggregates the knowledge and consequently handles the tasks of multiple teachers of distinct domains, each of which, again, may be single- or multi-task. It is more generalized than Knowledge Distilling (KD) that learns a student working on the same task as a single teacher does, and more flexible than multi-task learning since we can customize the tasks in need. As the initial attempt towards this aspiring goal, for this work we focus on classification and utilize the state-of-the-art ResNet [He et al., 2016] and DenseNet [Huang et al., 2017] backbone for the student and the teachers. Nevertheless, the proposed approach is not restricted to these two backbones and others can be applied as well.

An example of the learned multi-task student network is depicted in Fig. 1, where the first several layers shown in black correspond to shared features and each particular task branches out at its own individual location. Specifically, the TargetNet is learned in a layer-wise manner, where we go through each layer and amalgamate the filtered knowledge from those teachers relevant to the customized tasks. This is achieved via entangling the layer of the student with the teacher network, in other words, by replacing the corresponding layer in the teacher with the one to be learned in the student, followed by computing the loss and eventually putting the layer back in the student. The branch-out location for each individual task, on the other hand, is obtained by searching for the layer that yields the overall minimum training loss.

With the proposed training strategy, we are able to derive a TargetNet that, not only masters a combination of various teachers' skills but also outperforms the teachers in their own domains. Furthermore, the size of the TargetNet is much smaller than the ensemble of the relevant teachers.

Our contribution is therefore an effective approach to training a student model termed TargetNet, without human annotations, that amalgamates knowledge from a pool of multi- or single-task teachers working on different tasks. From these teachers we filter the relevant knowledge to train the TargetNet by entangling them all. The TargetNet after training, at a reasonably compact size, handles multiple customized tasks and leads to results superior to the corresponding teachers.

2 RelatedWork

In this section, we first briefly review the recent approaches in multi-task learning, and then look at a related task, knowledge distillation, which focuses on training a student model that handles the same task as the teachers do.

Multi-task Learning

Multi-task learning has been widely studied in computer vision, natural language processing, and machine learning communitiesTo describe the hierarchical relations among tasks and to learn model parameters under the regularization frameworks, several lines of work have been proposed.

One line is to utilize the given tree structure. For example, [Han et al., 2014] proposes a probabilistic tree sparsity model that utilizes the tree structure to obtain the sparse solution. More recently, ML-Forest [Wu et al., 2016] is proposed to learn an ensemble of hierarchical multi-label classifier trees to reveal the intrinsic label dependencies.

Another line has been focused on fusing MTL with CNN to learn the shared features and the task-specific models. For example, [Zhang et al., 2014] proposes a deep CNN for joint face detection, pose estimation, and landmark localization. Misra et. al. [Misra et al., 2016] propose a cross-stitch network for MTL to learn an optimal combination of shared and task-specific representations. In [Zhang and Zhang, 2014], a task-constrained deep network is developed for landmark detection with facial attribute classifications as the side tasks. [Zhao et al., 2018] proposes a multi-task learning system to jointly train the task of image captioning and two other related auxiliary tasks which help to enhance the CNN encoder and the RNN decoder in the image captioning model.

Considering a large number of possible label sets, most multi-task learning methods require sufficient labeled training samples. Multi-Label Co-Training [Yuying et al., 2018] introduces a semi-supervised method which leverages information concerning the co-occurrence of pairwise labels.

Unlike traditional multi-task learning, our customized multi-task learns the tasks' hierarchy in CNN without the new demand for dataset with groundtruth annotations.

Knowledge Distillation

First proposed in [Hinton et al., 2015], knowledge distillation aims at training a student model of a compact size by learning from a larger teacher model or a set of teachers handling the same task, and thus finds its important application in deep model compression [Yu et al., 2017]. More recently, the work of [Gao et al., 2017b] introduces a multi-teacher and single-student knowledge concentration approach. The work of [Shen et al., 2019], on the other hand, trains a student classifier by learning from multiple teachers working on different classes. Each teacher here, however, is restricted to be single-task.

There are also attempts focusing on tasks other than classification. For example, [Chen et al., 2017] handles object detection and learns a student model with improved accuracy; [Huang et al., 2018] specializes in speech applications by conducting knowledge distillation on sequence data.

The proposed work here, unlike any prior ones we are aware of, trains a student model by learning from a pool of teachers, each of which may be either single- or multi-task. The student model, under the specification of users, and amalgamates only the filtered knowledge.

3 Method

In this section, we provide the technical details of the proposed approach.

3.1 ProblemDefinition

Our goal is to train a student model, termed as TargetNet, of which the tasks are customized by the users. As depicted in Fig. 2, we learn TargetNet using pre-trained teacher models and unlabelled images. Since each teacher may handle multiple tasks, of which only a subset is relevant to TargetNet, we would have to first filter the knowledge from a teacher and then amalgamate such filtered knowledge from all the teachers so as to produce TargetNet.

Let us use I to denote an input image, and use A = {A1,A2, ...,AN} to denote a set of teachers, each of which may be multi-task. Let Tn denote the number of tasks handled by a teacher An, and thus we have An = {a1n , a2n , ..., aTn n }, forming the teacher tasks' union as A = {A1, ...,AN}. To make a customized task union for target, tasks {apn n , ..., aqn n } are selected in teacher n. Then for TargetNet, the customized tasks can be defined as:

where Cn ⊆ An, C ⊆ A, which means the customized tasks can either be the whole union or a subset of the teachers' task ensemble.

In our approach, we take a block to be the atomic unit of a network. Specifically, we use F k n to denote the feature maps in the k-th block of the n-th pre-trained teacher, which are to be filtered first before being further amalgamated.

{ } 1 1 2 1 1 1 ,..., , T a a a 3.2 CustomizedMulti-task TargetNet

⎪⎪⎭ ⎪⎪⎬ ⎫ ⎪⎪⎩ ⎪⎪⎨ ⎧ N N q N p N q p a a a a , ... , , , ... , 1 1 1 1 { } N N 2 N 1 N ,..., , T a a a In this section, we describe the proposed approach to learning a compact student network, TargetNet. The training is achieved via a strategy, which learns the parameters of the student entangled with those of the teachers. In essence, our approach is a block-wise learning method, as depicted in Fig. 2, that learns the parameters of the student network through transforming the student knowledge into each of the teachers' expertise domains for computing the loss and thus updating the parameters.

The regions with high activations from a neuron may share some task related similarities, even though these similarities may not be intuitive for human interpretation. In order to capture these similarities, there should be also neurons mimic these activation patterns in student networks.

Here, to clarify the proposed method, we describe the details in the multi-label task, which is a specific field of multitask problem. Despite the TargetNet eventually has multiple output streams for each task, it is initialized with the same architecture as those of the teachers except for the last few fully connected layers(fc), where we set the number of fc the same as that of tasks. We then train the TargetNet and finally branch out the streams for each task, after which the initial blocks following the branch-out are removed.

3.3 Pretrained Teacher Networks

In the set of N pretrained teacher networks with the same architecture A, each teacher An specializes in Tn different tasks, where Tn ≥ 1 which means the teachers can be either in single or multiple tasks' architecture.

For teacher network An with Tn tasks, when Tn = 1 with task An = {a1n }, then the parameters for network An can be easily updated with the loss ℓ(a1n , a1∗ n ). In the situation that Tn > 1, and teacher An is the expert for task An = {a1n , a2n , ..., aTn n }. Then the loss function can be written as:

where ℓ is the cross entropy loss for each task in network An,

{ } 1 1 1 1 ,..., q p a a ain and ai∗ n are the groudtruths and predict labels for i-th task respectively.

{ } N N ,..., q N p N a a { } 1 1 1 1 ,..., q p a a { } N N ,..., q N p N a a We use this loss function to pre-train the network to get parameters θ. Given an input image I, the outputs of Tn tasks can be defined as O(Tn) n = An(I, θn). For each teacher network, we define the block to be the minimum unit, then the network can be denoted as a stack of B blocks An = {A1n ,A2n , ...,ABn } and the last few fully connected layers F (Tn) n . The feature maps output from each block can be defined as:

where F 1 = A1 (I, θ1 ), which takes the original image as input.

In this paper, F k contains the knowledge we need for guiding the target network. When the tasks in the teacher are all selected for the target, the intermediate features {F 1 , F 2 , ..., F B } contain all the knowledge to be amalgamated for TargetNet. When only a part of the tasks are chosen, F (B) can't be directly used for guidance for the reason that the knowledge of the un-selected tasks is entangled with no difference and will pollute the customized tasks.

3.4 Filtered-Knowledge Amalgamation

For training the customized multi-task TargetNet T for the selected task set C, we transfer the problem into learning the features F k u at the k-th block of TargetNet. We feed the unlabelled samples into both teachers and the target, in which way the features of the teachers and the initial ones of the Target are both obtained. To amalgamate the useful part of them, we introduce a two-step filtering strategy to acquire the exactly needed knowledge for the chosen customized tasks, one is the teacher-level filtering f, the other is the task-level one g.

For teacher-level filtering, considering the learned knowledge Fu in TargetNet works as a container having the knowledge of all the teachers, we need to decompose it into N

streams as Fun for each teacher. Then the process can be defined as

which is applied in every block k, with 1 ≤ n ≤ N and 1 ≤ k ≤ B. This filtering function is learnable and realized by a light channel coding module consisting of a global pooling layer and two fully connected layers.

To deal with the task-customized demand, a task-level filtering is constructed. Let F k n denotes the obtained features at the k-th block of the n-th teacher network. Obviously, this intermediate features are learned in pre-trained networks for all the tasks {a1n , a2n , ..., aTn n }. So for the customized task set Cn, we use g to filter the features specifying for the selected tasks, then g(F k n ) outputs the customized knowledge, where g(F k n ) = F k n when Cn = An, And the loss function to update block k in TargetNet can be defined as:

where λn is a set of weights that balances each network, Dist(·) is the distance function measuring the difference between the features.

With the loss function Eq. (5), parameter updating takes place within block k, in T k and the connecting teacher-level filters {f k n }Nn=1 , and blocks {T 1 , ..., T k-1 } keep unchanged.

However, since the knowledge of the teachers and the target are in the different feature space, the distance function is hard to determine for directly aligning the features between teachers and student. Also, for training each block in TargetNet, we have to adjust the weights λn and the termination conditions, as the magnitudes of the feature maps and convergence rates vary block by block. Moreover, without groundtruth annotations, the filtering function g(·) in Eq. (5) seems impossible to define and train.

To this end, we learn the features of the student by entangling them with those of the teachers. In block n, we compute F k un by sending F k u , denoting the amalgamated features, to f k n , denoting the filter at the corresponding level of the teacher. Next, we substitute the features F k n at the k-th block of n-th teacher with F k un , and derive the predictions {Oiun }qn i=pn from the n-th teacher with F k un as its features. We write,

where k < l ≤ B, pn ≤ i ≤ qn. In the final output, we only select the needed predictions for customized tasks Cn: {Opn un , ..., Oqn un } ⊆ O(Tn) un .

Then the losses with corresponding tasks are calculated and the loss Lu in Eq. (7) can be rewritten as:

Algorithm 1 Training TargetNet with n-th teacher

Input: I: original input; An(θn): n-th pre-trained teacher specifies in tasks {ain }Tn i=1 ; B: the total number of blocks; {ain }qn i=pn : customized tasks.

Output: {Ti}qn i=pn : task-specific TargetNet.

• 1: Initial TargetNet T (θ0) and task-level filter fn;
• 2: Feed I into An and T , O(Tn) n = An(I)
• 3: Initial F 0 u = F 0 n = I
• 4: for block k = 1 : B do
• 5: Get F k u = T k ([F 1 u , ..., F k-1 u ], θk 0 );
• 6: Transform F k u to F k un = f k n (F k u );
• 7: Compute O(Tn) un using Eq. (6);
• 8: Compute the loss of {Oiun }qn i=pn and {Oin }qn i=pn ;
• 9: Update T k (θku ) and f k n ;
• 10: Get the convergence loss value Lin .
• 11: end for
• 12: Find branch-out point Sin = argminn Lin ;
• 13: Get Ti by connecting {T 1 (θ1u ), ..., T Sin(θSin u )} with C Sin n , {ASin+1 n (θSin+1
• n ), ...,ABn (θB n )} and F in . 14: Fine-tune {Ti}qn i=pn and return it.

where ℓ in the same as the loss function in Eq. (2). |C| is the total number of tasks in the customized task set C, which is:

3.5 Branch Out

As we are handling multiple tasks here, it is non-trivial to decide where to branch out the TargetNet into separate taskspecific streams to achieve the optimal performances on all tasks simultaneously. Here, we allow different tasks to branch out at different spots. Specifically, once the B blocks of TargetNet have been trained using Eq. (7), we obtain the trained blocks {T 1 (θ1u ), T 2 (θ2u ), ..., T B (θB u )} for TargetNet T and the final loss values for each block and each task Lin . We then take the branch-out blocks to be:

Then for each task in the customized task set C, we can get a branch-out block spot.

We take out the initial blocks following the last branchout block, once the branch-out spot is determined, meaning that all the initial blocks after max(Sin ) are removed. Then for each task i selected from teacher An, we choose to replicate the corresponding following blocks in the teacher to that branch of task. That is, we regroup the branched task-specific TargetNet for task i as:

where we also keep the corresponding teacher-wise filter module for connecting the TargetNet and the corresponding teacher.

Each task-specific Ti shares the first few blocks of the TargetNet, and branches out at different blocks, which is the way to make them hierarchical. As a whole, the process of training TargetNet from one teacher is concluded in Alg. 1.

4 Experiments and Results

Here we provide our experimental settings and results. We test the performance on the multi-label problem.

4.1 Experimental Settings

Datasets. We evaluate the proposed method on the PASCAL Visual Object Classes (VOC) dataset [Everingham et al., 2010]. In this paper, both PASCAL VOC 2007 and VOC 2012 are employed for experiments. These two datasets contain 9,963 and 22,531 images respectively. We use the training set for pre-training the teachers. For TargetNet, we use the unlabeled images and the knowledge amalgamated from the teachers. The evaluation metric is Average Precision (AP) and mean of AP (mAP) complying with the PASCAL challenge protocols. As another popular benchmark for multilabel learning, we utilize MS-COCO dataset [Lin et al., 2014] to validate the effectiveness of the proposed method, which contains 123,287 images for training and validation with 80 object concepts annotated. As the ground truth labels of the

test set are not available, we evaluate the performance of all the methods on the validation set instead.

Implementation details. We implemented our model using TensorFlow with a NVIDIA M6000 of 24G memory. We adopt the poly learning rate policy. We set the base learning rate to 0.01, the power to 0.9, and the weight decay to 5e-3. Due to limited physical memory on GPU cards and to assure effectiveness, we set the batch size to be 16 during training. The TargetNet which is in the same architecture as the teachers (ResNet50 and DenseNet), is initialized with parameters pre-trained on ImageNet.

4.2 Experimental Results

Customized Labels

There are a total 20 labels in Pascal VOC 2007, we divide them randomly into two groups which are learned separately in two teacher networks. From each teacher, we choose one task to customize TargetNet dealing with two-label classification. The teachers are pre-trained in ResNet-50 with 10 labels. Here we conduct two sets of experiments: one is the two visual-irrelated tasks ('Bus' from teacher1, 'Diningtable' from teacher2), the other one is the two visual-related tasks ('Dog' from teacher1, 'Horse' from teacher2).

In Fig. 3, we show the accuracies of each task branching out at each block of ResNet-50. The upper one shows the trend of AP of two visual-irrelated labels, where we also show the initial AP of the label pre-trained in the teachers ('teacher1' and 'teacher2'). For 'bus', it achieves the highest AP of 96.2% in block b3 6 surpassing the baseline in teacher1 with 94.2%, and TargetNet branches out at this block according to Eq. (9), which conforms to the demand for branching out to get the highest accuracy for the chosen task. For 'Dingingtable', TargetNet branches out at block b4 3 acquiring AP of 95.6% in contrast with 95.4% in teacher2.

In the lower chart of Fig. 3, the results of TargetNet customized with two visual-related labels are depicted. TargetNet branches out at b3 5 for 'dog', b3 2 for 'horse' with AP of {90.5%, 96.0%} vs {88.2%, 94.7%} in teachers. As can be seen in the results, classifications of 'bus' and 'Diningtable' share 13 blocks in TargetNet more than 9 blocks in classifications of 'dog' and 'horse', which shows the 'visualirrelated' set is more entwined than the 'visual-related' one. This observation reveals that the relationship of multiple tasks is hard to determine and obtain, and the proposed method learns their hierarchy during the block-wise training.

Multiple Labels

To the best of our knowledge, we are the first to study the customized-label classification. So we set the customized task set C = A, which makes the TargetNet deal with a normal multi-label task for comparison with other methods. We compare the performance of the proposed method against the following state-of-the-art approaches: HCP-2000 [Wei et al., 2014], HCP-VGG [Wei et al., 2016], RLSD+ftRPN [Zhang et al., 2018], CNN+RNN [Wang et al., 2016], CNN+LSTM [Wang et al., 2017], PF-DLDL [Gao et al., 2017a] and the method of [Zhou et al., 2018].

Tab. 1 displays the results on Pascal VOC 2007 dataset. 'TargetRes50' indicates the TargetNet in ResNet-50 and branches out at different blocks for each task to obtain the best accuracy, outperforming the teachers 'TeacherRes50', which consists of two teachers each specifying in a 10-label classification task, which are distinguished by different background colors in the tale. Also we show the APs in TargetNet of 4 main blocks during the block-wise training as 'TargetRes50 b1∼b 4', where the most of the labels' AP scores are increased. Also, we divide the labels into 3 groups, each learned in one teacher network. Then 'TargetRes50-3' is the TargetNet guided by these 3 teachers, which increases the accuracies further than that guided by two teachers. For the generalization test, we change the architecture to DenseNet 'TargetDenseNet' for both the target and teachers, and acquires better results thanks to better teachers' performance.

Tab. 2 reports the results on Pascal VOC 2012 dataset. Our work, TargetNets amalgamated from 2 teachers ('TargetRes50' and 'TargetDenseNet') and amalgamated from 3 teachers ('TargetRes50-3') all outperform the other methods with the highest mAP of 93.3% in 'TargetDenseNet', note that we do not use the image region information.

Tab. 3 presents the results on MS-COCO dataset. We follow the work of [Wang et al., 2016] to select the top k = 3 labels for each image. In tab. 3, we compare the overall precision ('O-P'), recall ('O-R'), F1 ('O-F1'), and per-class precision ('C-P'), recall ('C-R'), F1 ('C-F1'). Our proposed TargetNets learned from two teachers ('TargetRes50' and 'TargetDenseNet') outperform the exiting methods and achieve the best on all the measurements. And 'TargetDenseNet' works better than 'TargetRes50' in 5 indicators.

5 Conclusion and FutureWork

In this paper, we focus on training a customized multi-task student model, by learning from a pool of multi- or singletask teacher models and without any human-labelled annotation. The student amalgamates the relevant knowledge filtered from the teachers, and masters the complete or a subset of expertise of them. Specifically, we adopt a block-wise scheme that trains the features of the student entangled with the teachers. Our experimental results demonstrate that, the derived student model achieves very promising results without any groundtruth annotations, even outperforms those of the teachers in their own domain.

The current work, admittedly, focuses on the classification task only. In the future, our goal is to conduct knowledge amalgamation from multiple heterogeneous networks handling other tasks, and make it generalized to an even broader domain. This includes not only the higher-level vision tasks including object detection [Wang et al., 2018], tracking [Wang et al., 2014], and image search [Wang et al., 2011], but also the lower-level ones like image rectification [Yin et al., 2018].

Acknowledgements

This work is supported by National Key Research and Development Program (2016YFB1200203) , National Natural Science Foundation of China (61572428,U1509206), Key Research and Development Program of Zhejiang Province (2018C01004), the Program of International Science and Technology Cooperation (2013DFG12840), the Startup Funding of Stevens Institute of Technology, and the Fundamental Research Funds for the Central Universities.