Fusion Label Enhancement for Multi-Label Learning

1 Introduction

Learning with ambiguity has become one of the hot topics among the machine learning communities [Geng, 2016; Gao et al., 2017]. Multi-label learning (MLL) is a common choice to deal with the label ambiguity problem, because an instance can be annotated by multiple labels simultaneously in MLL [Zhang and Zhou, 2014; Liu et al., 2021]. In MLL,

∗Corresponding Author

• (a) IRI: sand ≻ sky ≻ tree
• (b) IRI: tree ≻ sky ≻ sand

Figure 1: Exemplar images which have been annotated with multiple labels sand, sky and tree. The implicit relative importance (IRI) information is marked in the images.

each class label is treated as a logical indicator of whether the corresponding label is relevant or irrelevant to the instance, i.e., +1 represents relevant to the instance and -1 represents irrelevant to the instance. Such label represented by -1 or +1 is called logical label. During the past years, multi-label learning techniques have been widely applied to various fields such as document classification [Liu et al., 2017], video concept detection [Lo et al., 2011], image classification [Lanchantin et al., 2021], fraud detection [Wang et al., 2020] etc.

In real-world MLL problems, the implicit relative importance of each possible label is generally different. For example, as shown in Fig.1 which has two images annotated with three positive labels sand, sky and tree. For the relevant label variance in Fig.1(a), the importance of sand should be greater than that of sky and tree, because sand can describe the image more apparently. Meanwhile, in Fig.1(b), the importance of tree should be greater than that of other labels. Therefore, the implicit relative importance information of different labels provides more accurate information for MLL and should be fully mined and leveraged.

Recently a novel technology named label distribution which can model the implicit relative importance of different labels has been proposed [Geng, 2016]. Formally speaking, given an instance x, label distribution assigns each y ∈ Y a real value dyx (label description degree), which indicates the importance of y to x. Since label distribution extends the supervision from binary to distribution, it is more applicable for real-world scenarios. Unfortunately, many training sets only contain simple logical labels rather than label distributions due to the difficulty of obtaining the label distributions directly. Therefore, the technology of label enhancement (LE) [Xu et al., 2021a] has been proposed. The aim of LE is to recover the label distributions from the logical labels in the training set via leveraging the topological information of the feature space and the correlation among the labels. Recently LE has attracted increasing research attention due to its potential to address the label ambiguity problem in machine learning and success in many real-world multi-label applications [Xu et al., 2020; Xu et al., 2021b].

Most of the existing LE-based MLL methods have two stages: 1) the numerical labels are first reconstructed, and 2) the predictive model is trained by levering the reconstructed labels. In these approaches, LE is independent of the training of the predictive model. When solving specific tasks, LE may not meet the needs of the training process on the predictive model. Other LE methods are one-stage approaches. These methods implement LE and the training process on the predictive model iteratively. However, these methods lack control over the interaction and matching between LE and the training process, i.e., the label enhancement process is decoupled from the training process, which finally affects the whole learning system. Therefore, it is essential to effectively integrate the label enhancement process and the training process to avoid the mismatch between the two processes.

In light of the above observations, we propose Fusion Label Enhancement for Multi-label learning (FLEM), in which label enhancement and the training process on the predictive model are performed simultaneously and interactively. Specifically, we first design a matching and interaction mechanismwhich leverages a novel interaction label enhancement loss to avoid the mismatch between the label enhancement process and the training process. Meanwhile, we establish the relationship between LE and the training process by employing a novel unified label distribution loss. With the proposed loss functions, the label enhancement information can be can effectively utilized to guide the training process on the predictive model. By simultaneously performing LE and the training process, the problem of mismatch between the two processes can be well addressed, and the performance of the model can be effectively improved.

2 RelatedWork

Multi-label learning (MLL) has attracted increasing interest recently. Traditional MLL approaches can be roughly grouped into three types based on the thought of the order of label correlations: first-order, second-order and high-order [Zhang and Zhou, 2014]. The first-order approaches decompose MLL into a series of binary classification and neglect the fact that the information of one label may be helpful for the learning of another label [Boutell et al., 2004]. The secondorder approaches consider the correlations between pairs of class labels, but they only focus on the difference between the relevant label and irrelevant label [F¨urnkranz et al., 2008]. The high-order approaches consider the correlations among label subsets or all the class labels [Wang et al., 2019]. But these approaches take the equal label importance assumption.

Some researchers try to assign each label different importance through class frequencies. [Wu et al., 2020] proposes a distribution-balanced loss for multi-label tasks by re-balancing weights and mitigating the over suppression of negative labels. [Ridnik et al., 2021] designs an asymmetric loss that uses different γ values to weight positive and negative samples in focal loss. [Huang et al., 2021] combines negative-tolerant regularization (NTR) [Wu et al., 2020] and class-balanced focal loss (CB) [Cui et al., 2019] and presents a novel loss function named CB-NTR. Balanced softmax method [Zhang et al., 2021] turns the multi-label learning loss into comparisons of relevant label scores and irrelevant label scores, and balances the importance of each label automatically. However, these methods are based on elaborately designed rules that can not dynamically adjust the importance of labels based on the given instances. Meanwhile, these approaches ignore the correlations between labels.

Label enhancement (LE) is a recent method proposed for mining the implicit relative importance among different labels and the correlation between them. Many novel label enhancement methods have been proposed in recent years [Xu et al., 2021a; Xu et al., 2020]. These methods can be broadly categorized into two directions: two-stage approaches and one-stage approaches. GLLE [Xu et al., 2021a] and LEVI [Xu et al., 2020] are representative algorithms of two-stage approaches. They assume that the label distribution space should share a similar topological structure with the feature space. The main concern of these approaches is they separate the label enhancement process from the model training process, which leads to label enhancement may not meet the needs of model training. LEMLL [Shao et al., 2018] and VALEN [Xu et al., 2021b] are representative algorithms of one-stage approaches. They perform label enhancement and the training process on the predictive model iteratively. However, these approaches lack control over the interaction and matching between label enhancement and model training, which may cause the mismatch between the model training and label enhancement.

In the next section, a novel LE approach for multi-label learning will be introduced. Different from existing label enhancement approaches, a novel matching and interaction mechanism is designed to perform the label enhancement process and the training process on the predictive model simultaneously and avoid the mismatch between them.

3 The FLEM Approach

First of all, the main notations used in this paper are listed as follows. The instance variable is denoted by x, the particular i-th instance is denoted by xi, the label variable is denoted by y, the particular j-th label value is denoted by yj, the logical label vector of li is denoted by li = (ly1 xi , ly2 xi , ..., lyc xi )T, where c is the number of possible labels. The description degree of y to x is denoted by dyx, and the label distribution of xi is denoted by di = (dy1 xi , dy2 xi , ..., dyc xi )T.

3.1 Overview

In FLEM, the label enhancement process and the training process on the predictive model are performed simultaneously. Specifically, FLEM adopts a predictive model f (·) for performing the MLL task and a label distribution estimator g (·) for label enhancement. The predictive model f (·) and the label distribution estimator g (·) are trained simultaneously during the whole training stage. For training the label distribution estimator, FLEM adopts three strategies to guide it to generate label distributions effectively. For avoiding inconsistency between LE and the training process on the predictive model, FLEM employs a matching and interaction mechanismwhich leverages a novel interaction label enhancement loss from the perspective of instances. For training the predictive model, FLEM integrates a novel unified label distribution loss to build a corresponding relationship between label distribution and the training of the predictive model. The algorithmic description of FLEM is given in Algorithm 1.

3.2 Label Enhancement

The existing MLL methods simply divide the label set into relevant labels and irrelevant labels, which can not be extended to the form of label distribution. Therefore, we need to recover the label distributions from the given logical label vectors, so as to guide the training of the predictive model.

In FLEM, the label distribution estimator g (·) is used to recover the label distributions. For generating the recovered label distribution, g (·) needs to use the topological information in both feature space and label space, so as to better obtain the correlation between labels according to the feature information. Therefore, we adopt two modules gF and gL for feature embedding and label embedding, which transform feature information and label information into continuous embedding values respectively. For the instance xi, we define the predicted values for label enhancement as

where gD is the module of g for decoding, C (·, ·) is the concatenation operation. Then the recovered label distribution vector di can be obtained by the softmax operation on d∗i .

Training Strategies

In order to enable the label distribution estimator to generate a reasonable estimate of label distribution, we propose the following three training strategies. Each of them can lead to effective optimization.

1) The Similarity Strategy. The first strategy trains g(·) by matching the similarities of instances in feature space and label space. Specifically, the similarity matrix A of a batch of instances can be obtained by:

where S is cosine similarity, xm and xn are the m-th and nth instances. Meanwhile, by estimating the label distribution vector of these instances, we can obtain the similarity matrix Z of the predicted values for label enhancement:

where d∗m and d∗n are the predicted values for label enhancement of the m-th and n-th instances. Then the distance between A and Z is minimized to train the label distribution estimator. In FLEM, we adopt the following loss:

where M is the number of instances.

2) The Threshold Strategy. The second strategy is based on the criterion that the minimum description degree of relevant labels is not less than the maximum description degree of irrelevant labels. Therefore, for a batch of instances, the loss function is:

where Ωpos and Ωneg are the sets of relevant and irrelevant labels respectively, dy∗n xi and dy∗ p xi are predicted values for label enhancement and ϵ is the threshold.

3) The Label Distribution Strategy. The third strategy considers that the output of the predictive model is useful for training the label distribution estimator. Specifically, it optimizes the Kullback-Leibler (KL) divergence between the output of the predictive model and the label distribution estimator:

where P is the j-th element of the output of the predictive model for instance i through the softmax operation.

Matching and Interaction Mechanism

Using label distribution to guide multi-label model training faces an urgent problem, i.e., how to ensure that the generated label distribution is the recovered label distribution required for predictive model training. The existing label enhancement approaches adopt two-stage or iterative methods, which leads to the separation of the training processes of the predictive model and the label distribution estimator. In fact, the label distribution estimator can use the information of the predictive model to guide its own training, so as to realize matching and interaction.

The most important problem is how the label distribution estimator interacts with the predictive model. An efficient way is to align their training processes to achieve training matching. Inspired by asymmetric focal loss [Ridnik et al., 2021] which is a powerful approach to balance the importance of different instances, we design the interaction label enhancement loss for each training instance xi:

where pj is the prediction of the predictive model through the sigmoid operation, p∗j is the prediction of the label distribution estimator through the sigmoid operation, γ+ and γ- are two hyper-parameters to control each part of the loss function.

Algorithm 1 FLEM algorithm

Input: The MLL training set D = {(xi, li)}ni=1 , the number of epoch T, the number of iteration I, hyper-parameters α and β;

• 1: Initialize the parameters of the label distribution estimator g(·) and the predictive model f(·).
• 2: for t = 1, ..., T do
• 3: Shuffle training set D = {(xi, li)}ni=1 into I minibatches;
• 4: for k = 1, ..., I do
• 5: Obtain label distribution di for each example xi by Eq. (1) and the softmax operation;
• 6: Update f and g by forward computation and backpropagation by fusing Eq. (8) and Eq. (10);
• 7: end for
• 8: end for

Output: The predictive model f(·).

The interaction loss matches the label distribution estimator with the training process of the predictive model from the perspective of instances. Specifically, it can distinguish "easy" instances and "hard" instances, and place a higher weight of loss on "hard-to-distinguish", which may have more inter-class information. With the knowledge from both models, the label distribution estimator can effectively generate label distributions that meet the needs of the training process on the predictive model. Then the whole training loss for the label distribution estimator can be given:

3.3 Classifier Training

Unified Label Distribution Loss

With the recovered label distribution vector di, the implicit relative importance information can be can utilized to guide the training of the predictive model. Since the label distribution is introduced, we can optimize the predictive model from a global perspective. Specifically, we minimize the following unified label distribution loss to guide the training of f (·):

where n is the number of instances, s(j) i is the output logit of the predictive model for the i-th instance and Zi = ∑ck=1 s(k) i . The unified label distribution loss LLD establishes a corresponding relationship between the recovered label distribution vector and the output of the model. Compared with the losses of existing MLL approaches, it is more effective by using the distribution information of labels. In particular, we have the following theorem.

Theorem 1. LLD gives an upper bound for cross-entropy loss.

Since logical labels are effectively extended to label distribution, our unified label distribution loss is a generalization form of cross-entropy loss. It also indicates that if accurate label distribution information can be obtained, it can effectively guide the training of the model.

Training Process

As mentioned in Section 3.1, in FLEM, the predictive model f (·) and the label distribution estimator g (·) are trained simultaneously. For the training of the predictive model, we not only concern the distance between the recovered distribution and the expected label distribution, but also the sign consistency of them. It leads to the minimization of

where LSC can be any loss function that can lead to sign consistency. For generality, we adopt the vanilla binary crossentropy loss:

where pj is the prediction of the predictive model through the sigmoid operation. The gradient-based optimization method can be used to optimize the model parameters. It should be noticed that when using Eq.(10) to update f(·), it will not update g(·), i.e. the stop-gradient operation is performed on each term about g(·) for Eq.(10). Meanwhile, using Eq.(8) to update g(·) will not update f(·) either.

In FLEM, f is set as linear classifier, gF , gL and gD are set as three-layer-neural networks with a residual structure. In the prediction stage, given a new instance x∗ , the prediction of the model can be obtained by f (x∗ ).

4 Experiments

In this section, the efficiency and the performance of FLEM are evaluated in multiple MLL datasets. All methods are implemented by PyTorch. All the computations are performed on a GPU server with NVIDIA Tesla V100, Intel Xeon Gold 6240 CPU 2.60 GHz processor and 32 GB GPU memory. Our code is available at: https://github.com/ailearn-ml/FLEM.

4.1 Datasets and Preprocessing

To evaluate the proposed approach, we conduct experiments on several real-world datasets, including AAPD [Yang et al., 2018], Reuters [Debole and Sebastiani, 2005], VOC07, VOC12 [Everingham et al., 2015], COCO14, COCO17 [Lin et al., 2014], CUB [Wah et al., 2011] and NUS [Chua et al.,

2009]. For AAPD and Reuters, we use the feature extracted by pre-trained XML-CNN [Liu et al., 2017]. For the other datasets, ResNet-50 pre-trained on ImageNet is used to extract the feature of instances. Following common practices [Liu et al., 2017; Lanchantin et al., 2021], these datasets are split into training set, validation set and testing set. Statistics of these real-world datasets are given in Table 1.

4.2 Comparing Algorithms and Evaluation Metrics

We compare our methods with several traditional and stateof-the-art MLL and LE technologies. The compared methods include: 1) ML-KNN [Zhang and Zhou, 2007]: a classical approach, which adopts kNN into MLL. 2) Random k-labelsets (RAKEL) [Tsoumakas et al., 2011]: a high-order approach which transforms MLL into an ensemble of multi-class learning problems. 3) Focal loss (FL) [Lin et al., 2020]: a simple but widely used strategy for classification. 4) Class-balanced loss (CB) [Cui et al., 2019]: a class-wise re-weighting guided by the effective number of each class En = (1-βn )/(1-β). 5) Rebalanced focal loss (R-FL) [Wu et al., 2020]: a combination of re-balanced weighting and focal loss. 6) Distribution balance loss (DB) [Wu et al., 2020]: a recently proposed distribution-balanced loss for MLL. 7) Balanced softmax method (BS) [Zhang et al., 2021]: a balanced loss function for MLL. 8) Label enhancement via variational inference (LEVI) [Xu et al., 2020]: an advanced technology for LEbased MLL, which has been proved to be effectively applied to MLL. 9) VAriational Label ENhancement (VALEN) [Xu et al., 2021b]: a recent LE technology, whose training strategy for the predictive model is set to the same strategy as LEVI

in our experiments. Among them, ML-KNN and RAKEL are representative traditional MLL algorithms, LEVI and VALEN are advanced LE-based MLL technologies, and the others are recent methods proposed for MLL. For all of these state-ofthe-art algorithms, we train linear classifiers for classification tasks. The optimization process spans over 30 epochs using the AMSGrad variant [Reddi et al., 2018] of AdamW [Loshchilov and Hutter, 2017] with a weight decay of 0.0001. The learning rate is set to 0.001 for all algorithms. For FLEM, hyperparameters α and β are are selected by grid search from the set {0.0001, 0.001, 0.01, 0.1}.

Four widely-used MLL evaluation metrics are selected in this experiment, i.e., Hamming loss, Ranking loss, Coverage and Average precision. For each algorithm, we select the model that obtains the optimal Hamming loss on the validation set for testing.

4.3 Performance Comparison

Tables 2, 3, 4 and 5 tabulate the experimental results of different algorithms on the eight MLL datasets evaluated by four evaluation metrics, and the best performance on each dataset is highlighted by boldface. In Tables 2, 3, 4 and 5, "FLEM-S", "FLEM-T" and "FLEM-D" refer to the FLEM algorithm using the three strategies proposed in section 3.2 respectively. For each evaluation metric, "↓" indicates the smaller the better while "↑" indicates the larger the better. The ranks are given in the parentheses right after the performance values. The average rank of each algorithm over all the datasets is also calculated and given in the last row of each table.

From Tables 2, 3, 4 and 5, it can be observed that FLEM is superior to existing MLL algorithms in almost all cases.

This result indicates that the label enhancement method in FLEM is effective and can help the model significantly improve classification performance. Meanwhile, compared with the advanced LE methods, FLEM has achieved better results in almost all metrics. This observation demonstrates that the matching and interaction mechanism proposed in this paper is effective and can well match the label enhancement with the model training to improve the performance of the model.

When looking at the average ranks over all the eight realworld datasets, FLEM achieves rather competitive performance over other algorithms. All three methods proposed in this paper can exceed existing methods. When compared with these state-of-the-art algorithms, FLEM achieves 1st in 96.9% cases. Thus, FLEM possesses rather superior performance over the state-of-the-art algorithms across all the evaluation measures.

4.4 Ablation Study

We further conduct an ablation study to verify the effectiveness of the unified label distribution loss and the interaction label enhancement loss. Based on the Label Distribution strategy proposed in section 3.2, we evaluate three cases of our method: 1) FLEM-I: without the unified label distribution loss, 2) FLEM-II: without the interaction label enhancement loss and 3) Ours: the proposed FLEM. Table 6 tabulates the experimental results of different versions on the VOC07 dataset, and the best performance on each comparison is highlighted by boldface. From Table 6, we can find that both the interaction label enhancement loss and the unified label distribution loss are beneficial to improve the performance, which proves the validity of the proposed matching and interaction mechanism and the unified label distribution loss.

5 Conclusions

This paper proposes a novel label enhancement method for multi-label learning. By utilizing a novel matching and interaction mechanism, the mismatch between label enhancement and classifier training is effectively avoided. In the meantime, a novel unified label distribution loss is proposed for guiding MLL models to learn with the recovered label distributions. Extensive experimental results demonstrate that FLEM can bring significant performance improvements over the existing MLL methods. In future work, we will further explore the theory of FLEM, promote the versatility of FLEM and design more effective methods.

Acknowledgments

This research was supported by the National Key Research & Development Plan of China (No. 2018AAA0100104, No. 2018AAA0100100), the National Science Foundation of China (62125602, 62076063), China Postdoctoral Science Foundation (2021M700023), Jiangsu Province Science Foundation for Youths (BK20210220).