Boosting Multi-Label Image Classification with Complementary Parallel Self-Distillation

1 Introduction

Natural images often contain multiple visual objects, which can be characterized by a set of image labels. Multi-label image classification (MLIC) task is to recognize all these objects, which is highly relevant to other vision tasks such as object detection, image retrieval, and semantic segmentation.

Most existing MLIC research works focus on exploiting the label correlation property, which distinguishes it from the single-label image classification problem. Label correlation exploitation strategies, such as pair-wise and high-order label correlation have been extensively studied. Deep learningbased approaches, such as RNN [Wang et al., 2016], graph model [Chen et al., 2019a; Chen et al., 2019c; Nguyen et Decomposing correlation al., 2021] and attention mechanism [Gao and Zhou, 2021] are widely employed to encode the image label correlation, yielding decent performance.

∗Corresponding Author

Figure 1: Class Activation Map (CAM) of class mouse and truck. Both target labels are marked in red. Overemphasizing the correlation by specifically putting the mouse and laptop categories together to train an MLIC model leads to model overfitting because the MLIC model infers the label mouse only based on the features of a laptop, e.g. upper left. On the contrary, if we train individual models parallelly for those two labels, the model will capture the category-specific features, e.g., upper right. However, when the object is not well exposed, the model needs to rely on the labeldependence knowledge to mine the category-specific feature. Training together succeeds in recognizing label truck with the help of co-occurring car and boat, e.g., lower left. It is quite challenging to reconcile the model overfitting and the label correlation exploitation.

Although label correlation is a useful feature for the MLIC problem, it can also be misleading due to model overfitting. As illustrated by Figure 1, emphasizing co-occurrence leads to an inference of targets primarily from co-occurring objects, which may not represent all cases. On the contrary, decomposing this correlation leads to learning discriminative features of the target itself, which may also fail in classification due to the lack of context. Therefore, in addition to capturing label co-occurrence, our proposed framework also includes the discriminative features of individual labels, represented by dis-occurrence. Furthermore, because of its multi-class nature, the problem complexity of an MLIC task, in terms of the prediction space grows exponentially as the category number increases, e.g., the prediction result has 2c possibilities for a c-class MLIC problem. When the image category number is higher, the model learning task becomes more challenging and issues with model overfitting are more likely to occur. One prominent algorithm branch to reduce the complexity is to decompose the original problem into a set of binary problems with common strategies like one-vs-one and one-vs-all [Galar et al., 2011], or advanced strategies like DChooser [Chen et al., 2021].

In this paper, we adopt this divide-and-conquer thought and propose a generic MLIC framework that addresses both model overfitting and label dependency modelling. We first decompose an MLIC task into several simpler sub-tasks, each with fewer object categories. Individual models are trained in parallel for tackling these sub-tasks. After that, knowledge distillation, an effective way to learn a compact model with generalization capability from model ensemble [Gou et al., 2021], is conducted to learn a global model containing all object categories. The label decomposition reduces the complexity of each sub-task, which helps each individual model learn more representative features. In the model distillation process, those sub-models serve as teachers whose logit outputs are utilized as the soft targets to supervise the learning of a model which contains all categories with the same architecture as those teachers (i.e., self-distillation). These soft targets function as a label smoothing regularizer [Yuan et al., 2020] for a better optimization. The contributions of our method are summarized as follows:

• A generic MLIC framework, called Parallel Self-Distillation (PSD) is proposed. With proper task-decomposition strategy, the original MLIC model training task is formulated as a multiple sub-model parallelly training task, then these sub-models are distilled into a global model.We demonstrate that our framework can be flexibly applied to existing MLIC models and improve their performances.
• Two strategies, namely Co-occurrence Graph Partition (CGP) and Dis-occurrence Graph Partition (DGP), are elaborated, for decomposing the MLIC task via label partition. They model the label correlation by two complementary graphs. The co-occurrence graph models the label correlation, based on which the spectral clustering result tends to assign co-occurring labels into the same cluster. This induces the individually trained sub-model to learn the joint pattern of these co-occurring classes. While the dis-occurrence graph assigns labels without co-occurrence into one task for learning the category-specific patterns. These two complementary strategies are simultaneously leveraged in PSD to reconcile the model overfitting and label correlation exploitation.

We conduct extensive experiments on two widely-used MLIC datasets, MS-COCO and NUS-WIDE. Experimental results demonstrate that our framework can be plugged into different approaches to boost the performance without increasing complexity. We also visualize the implicit attention of our framework to expose the overfitting of co-occurrence and demonstrate the effectiveness of our approach.

2 RelatedWorks

Multi-label image classification (MLIC) is different from single-label image classification in that it relies on the label correlation property. Many MLIC approaches have been devoted to exploiting this property. For example, the methods proposed in [Chen et al., 2019a; Chen et al., 2019c] build label correlation graphs and adopt GNN for label feature learning. Sample imbalance is another issue of MLIC. In [Wu et al., 2020], a distribution-balanced loss is proposed. It re-balances the training sample weights and designs a negative-tolerant regularization which can avoid oversuppression caused by the dominance of negative classes. An asymmetric loss is proposed in [Ridnik et al., 2021], where the contribution of positive samples is maintained. Vision Transformer [Dosovitskiy et al., 2020] has recently been introduced to MLIC not only because of its strong feature extraction capability, but also because the self-attention mechanism can capture rich patterns between visual features and class label tokens [Lanchantin et al., 2021; Liu et al., 2021]. In this work, we test the performance of our proposed framework within Transformer with a naive Transformer encoder.

Knowledge distillation was initially proposed to transfer knowledge from large complex networks to slimmer networks in order to retain the performance of the large network with less computation and model size [Hinton et al., 2015]. [Zhang et al., 2019] find that distilling a pretrained model with the same architecture can boost the model performance. The technique is called self-distillation. Zhou et al. investigate the bias-variance tradeoff brought by distillation and propose to use weighted soft labels that enable a sample-wise bias-variance tradeoff [Zhou et al., 2021c]. In [Xiang et al., 2020], a multiple experts distillation method is proposed to handle the long-tailed distribution in the image classification task. The application of KD in MLIC can be found in [Song et al., 2021]. In that work, model distillation is adopted to alleviate the model bias toward difficult categories.

3 Methodology

3.1 Preliminary and Overview

Given an MLIC Task T := {(X, Y )} where X is the image set and Y is its corresponding label set, the goal is to establish a visual learning model F(·), which is able to predict the labels of a given image x ∈ X, y ← F(x). y = [y(1), y(2), · · · , y(m)] ∈ Y is a m-dimensional binary label vector and m is the number of categories. The binary element y(j) ∈ {0, 1} indicates the existence of the corresponding category in an image. Let C = {c1, c2, · · · , cm} be the category set where |C| = m. With regard to the multi-class classification issue, a larger m implies a highdimensional label space and thus a more difficult MLIC task. MLIC often suffers from more complexity in comparison to ordinary single-label multi-class image classification even in the same label space because of the label correlation. The divide-and-conquer strategy is an intuitive and common way for tackling such complex tasks. The basic idea of this strategy is to decompose the complex task into a set of simpler sub-tasks, and then assemble the sub-solutions to yield the final solution of the original task.

In this paper, we follow such a strategy and propose a Parallel Self-Distillation (PSD) framework to address MLIC issue. The architecture of PSD is shown in Figure 2. In PSD, the first step is to decompose the original MLIC task T into simpler MLIC sub-tasks by dividing the category set into several smaller subsets with a decomposition strategy {Tt}kt=1 = Ψ(T , C) where k is the number of sub-tasks and Ψ(·, ·) is a task decomposition strategy. Tt := {(XCt , YCt )|Ct ∈ C} is the t-th sub-task where Ct is a subset of C and is the label space of Tt. XCt is the images, which contain categories in Ct, and YCt is the label set of these corresponding images. The second step is to train an MLIC model FCt (·) for each sub-task individually. Finally, these trainedMLICmodels are assembled to yield a compact model which considers them as teachers for knowledge distillation. Moreover, we elaborate two task decomposition strategies named Co-occurrence Graph Partition (CGP) and Dis-occurrence Graph Partition (DGP) for PSD. The decomposition issue is regarded as a spectral clustering problem. In CGP, a label co-occurrence graph is constructed to assign co-occurring labels into the same cluster to induce the model to learn the joint patterns. On the contrary, DGP constructs a label dis-occurrence graph for spectral clustering, and tends to assign labels without co-occurrence into the same sub-task to learn the categoryspecific patterns. These two strategies are intrinsically complementary with each other and will be applied together to PSD , namely Complementary PSD (CPSD). Notably, the proposed approach can be flexibly plugged into any deep learning-based MLICmodels for further improvement.

3.2 Complementary Task Decomposition

Task decomposition is the key of the PSD framework. In the design of the MLIC task composition strategy, two important aspects should be paid attention to. One is the simplification of the task complexity by reducing the dimension of label space. The other is the label correlation exploitation.

In model optimization, these two aspects are somewhat conflicted. The label correlation complicates the MLIC task and easily triggers model overfitting to mislead the feature learning to learn features of the co-occurring object instead of the original ones.

In order to avoid this, we design two task decomposition strategies named Co-occurrence Graph Partition (CGP) and Dis-occurrence Graph Partition (DGP) based on spectral clustering. Finally, these two aspects of knowledge will be distilled into a unified MLIC model. CGP and DGP translate the task decomposition issue as the unsupervised category graph partition problem for solution. We construct a co-occurrence graph G+ and a dis-occurrence graph G- to encode the joint and specific patterns among categories respectively. By considering categories C as vertices and measuring the cooccurrence probabilities of categories as their similarities, we can define a m×m-dimensional similarity matrix S. Its ij-th element is Sij = eij/ni ∈ [0, 1] where eij is the amount of images containing both ci and cj. And ni is the amount of images containing ci. Mathematically speaking, let P be the affinity matrix of category graph G. The affinity matrices of G+ and G- can be denoted as follows,

where τ is a positive hyper-parameter for smoothing the cooccurrence probabilities which follow the long-tail distribution. A higher value of τ is able to alleviate more suppression of the head to the tail. Since G+ and G- are the undirected graphs, we also symmetrize the affinity matrices. The affinity matrices P + and P - respectively encode the degrees of co-occurrence and dis-occurrence among categories. Therefore, we can use them to produce graph Laplacians, and then conduct spectral clustering to partition the label space as a normalized graph cut problem,

where L = D - P is the Laplacian matrix and D is the degree matrix of G. D is a diagonal matrix whose ii-th element Dii = ∑j Pij. F is the learned graph embedding of vertices (categories), which encode the co-occurrence or dis-occurrence information of each category. Each of its columns encodes a graph cut operation represented by a vector. The aforementioned optimization problem can be efficiently solved by eigenvalue decomposition. The optimal ˆ F is the eigenvectors corresponding to the top-k minimum eigenvalues. By employing k-means to ˆ F, the categories can be clustered into k category subsets {Ct}kt=1 where ⋃kt=1 Ct = C and ⋂kt=1 Ct = ⊘. Then the task can be decomposed into k sub-tasks via data sampling according to the clustered category subset, {Tt}kt=1 . The sub-tasks generated by CGP {T + t }kt=1 are used to induce the model to learn the features of co-occurring objects. The sub-tasks generated by DGP {T - t }kt=1 exhibit lower complexity because of the neglect of the label correlation and encourage the model to focus on the extraction of category-specific features. All the teacher models trained by these sub-tasks will finally be used in the Parallel Self-Distillation (PSD), y ← (x,y)∈Tt σ(FCt (x))

where σ means sigmoid activation, Tt ∈ {T - t }kt=1 ∪{T + t }kt=1 since we expect to reconcile both aspects of knowledge and exploit their complementarity natures for better supervising the training of the student model. Moreover, the teacher models trained by these sub-tasks and generated from both strategies are highly different, which can further benefit PSD from the perspective of ensemble learning.

3.3 Parallel Self-Distillation

For each sub-task Tt, we train a teacher model,

with the Asymmetric Loss (ASL) [Ridnik et al., 2021] as the classification loss to suppress the negative effects from superabundant negative samples,

where yi the ground truth label of xi and ˆ yCt i = σ(FCt (xi)) is its predictions with respect to the category subset Ct. Its jth element ˆ yCt i (j) ∈ [0, 1] is the predicted label (probability) of the sample xi with respect to the category cj , and ˆ yCt i (j) = max(ˆ yCt i (j) - µ, 0). µ ≥ 0 is a threshold to filter out the negative samples with low predictions. γ+ ≥ 0 and γ- ≥ 0 are respectively the positive and negative focusing hyperparameters defined in ASL.

Once all teacher models have been obtained, we can merge their prediction results to yield a complete logit prediction according to the category order,

where ρ(·) is a label merging and reshuffling operation. Here, let ˆ y+ i and ˆ yi be the logit predictions produced by the teacher models based on CGP and DGP respectively. We establish a student model F(·) with the same architecture as teachers to produce the logit predictions of a sample, ˆ yi s = F(xi). The Mean Square Error (MSE) is adopted to measure the discrepancy between the logit predictions of the student model and the teacher models,

as the knowledge distillation loss for supervising the student model training.

Finally, the optimal student model can be obtained by addressing the following optimization problem,

where LCcls is the ASL loss of the original task T . LCcls can be constructed with Equation 4 based on the full data X. By the above manner, the learned optimal MLIC model ˆ F will finally incorporate the knowledge acquired by the CGP and DGP-based teacher models, and then we can use it to infer the label of a multi-label image in the testing stage, y = σ( ˆ F (x)).

4 Experiments

4.1 Experimental Setup

Datasets. Two widely used MLIC datasets, named MSCOCO and NUS-WIDE, are used for the evaluation of our method. MS-COCO contains 122,218 images with 80 categories of objects in natural scenes, including 82,081 images for training and 40,137 images for validation. In the official partition of NUS-WIDE dataset, it contains 125,449 labeled training pictures and 83,898 labeled test pictures from Flickr, which share 81 labels in total.

Following the conventions, mAP (mean average precision) is deemed as the main evaluation metric. We also report overall precision (OP), recall (OR), F1-measure (OF1) and percategory precision (CP), recall (CR), F1-measure (CF1).

Implementation details. We conduct experiments on three popular backbones, namely ResNet101 [He et al., 2016], TResNetL1 [Ridnik et al., 2021] and ResNeXt50SWSL [Yalniz et al., 2019], which are all pretrained on ImageNet-1K. A naive Vision Transformer encoder [Dosovitskiy et al., 2020] named ResNet101-TF is implemented with visual tokens extracted fromResNet101. In ResNet101TF, m class tokens are extracted from GloVe [Pennington et al., 2014] for class predictions, where the depth, number of multi-heads attention, and hidden dimensions are set to be 3, 4, and 1024 respectively. Q2L [Liu et al., 2021] is also adopted to verify the effect of our approach on well-designed methods. All experiments follow a training pipeline where Adam optimizer is used with weight decay of 10-4 under a batch size of 32. ASL is applied as the default classification loss function, and the hyper-parameters of ASL are simply left as their default settings. τ in Equation 1 is set to be 3. We set the training epoch to be 20 and 80 for sub-models and the compact global model individually. The number of clusters k will be discussed in our ablation study.

4.2 Comparison with State-of-The-Art Methods

Tables 1 and 2 report the MLIC performances of several methods evaluated on MS-COCO and NUS-WIDE datasets respectively. We apply our proposed framework on recent benchmarks to evaluate the effectiveness.

The observations show that our method generally boosts all baselines, and performs the best on both datasets. For example, the enhanced ResNet101, ResNet101-TF, Q2L-R101, TResL and ResNeXt50 get 1.5%, 0.9%, 0.9%, 1.1% and 1.0% gains respectively in mAP over their original ones on MS-COCO dataset. Such gains of ResNet101+TF and TResL are 1.7% and 1.8% on NUS-WIDE, which is a larger scale dataset. These experimental results also imply that our method performs much better on a larger-scale dataset.

In addition, the observations also imply that, based on our framework, the naive model is able to achieve state-of-the-art performances without involving additional costs in parameter scale or more complicated architectures. For example, our method gets 0.7% gains in mAP over MGTN using only its backbone on MS-COCO dataset. Another interesting phenomenon is that we achieve smaller performance gains on the more advanced models. For example, CPSD improves more on ResNet101 in comparison with ResNet101-TF and Q2LR101. We attribute this to the fact that the more powerful approaches are much harder to trap in model overfitting. Even so, our method still introduces a considerable improvement.

4.3 Ablation Study

Discussion on task decomposition strategy. We plot the performances of PSD under different k with different task decomposition strategies Ψ on both MS-COCO and NUSWIDE in Figure 3. The results indicate that the performances of PSD increased along with the increase of k with all strategies on both datasets. Moreover, our proposed strategies consistently perform much better than the random ones when k ≥ 2. DGPD performs better than CGPD when k is small while the best performances of the two strategies are highly similar. We attribute this to the fact that overemphasizing the label correlation causes model overfitting more easily when the label space is in high-dimensional, while the label correlation is still able to benefit MLIC when the category-specific features are well learned. Actually, reducing the size of the cluster (increasing k) can also be deemed as a natural way to break down label correlation. A larger k means more teacher models are needed to be trained, which leads to higher time cost. To achieve the tradeoff between the performance and the model training time, we set k = 5.

Ablation study on components. Table 3 shows the ablation study results of our method with different baselines on MS-COCO. The results show that our PSD framework boosts all the baselines and outperforms the common Self-Distillation (SD) with considerable advantages using CGP, DGP or their combination. For example, CPSD further improves SD by 1.4%, 0.8%, 0.8%, 0.7% and 0.8% on ResNet101, TRestL, ResNeXt50, Q2L-R101 and ResNext101-TF respectively. Moreover, CPSD performs much better than models using only one decomposition strategy, i.e. CGPD and DGPD. The improvements under different baselines are around 0.5%. These observations confirm the effectiveness of our method.

4.4 Explainable Visualization Study

We apply Class Activation Map (CAM) on the validation set of MS-COCO to visualize the implicit attentions of different images with respect to a specific category using different task decomposition strategies as shown in Figure 4. In this figure, the columns from left to right respectively show the original images and the images highlighted by the activations maps of the baseline ResNet101 models enhanced with PSD under CGP, DGP and their combination (CPSD). The first two cases (the top two rows) show that the category-specific features have been ignored by the model based on CGP due to the model overfitting caused by the overemphasis of the label correlation, while they are well learned by our DGP models. The last two cases show that the label co-occurrence information has not been exploited by the model based on DGP due to the occlusion, however the co-occurring categories are able to assist the model in discovering these features with the help of the label correlation information. The visualizations demonstrate that our proposed method can perform well in each of these situations and also reflect that our method can better exploit and reconcile the category-specific knowledge and label correlation knowledge.

5 Conclusion

In this paper, we proposed a simple yet effective Parallel SelfDistillation (PSD) framework in which the original complex MLIC task is decomposed into a set of simpler sub-tasks via label partition. Then multiple teacher models are trained in parallel to address these sub-tasks individually. The final model is obtained through the ensemble of these teacher models with knowledge distillation. For better boosting PSD, we introduce two task decomposition strategies, which address the task decomposition issue through conducting two complementary co-occurrence graph partitions. These two strategies, which respectively induce the models to learn the category-specific and category-correlated knowledge, are applied to PSD to set up the sub-tasks. Extensive experimental results on MS-COCO and NUS-WIDE demonstrate that our framework is effective and can be plugged into different approaches to boost the performances.

10

Acknowledgements

Reported research is partly supported by the National Natural Science Foundation of China under Grant 62176030 and 71621002, the Natural Science Foundation of Chongqing under Grant cstc2021jcyj-msxmX0568, and the Strategic Priority Research Program of Chinese Academy of Sciences Grant XDA27030100.