Contrastive Label Enhancement

1 Introduction

In recent years, Label Distribution Learning (LDL) [Geng, 2016] has drawn much attention in machine learning, with its effectiveness demonstrated in various applications [Geng et al., 2013; Zhang et al., 2015; Qi et al., 2022]. Unlike single-label learning (SLL) and multi-label learning (MLL) [Gibaja and Ventura, 2014; Moyano et al., 2019; Zhao et al., 2022], LDL can provide information on how much each la-

lem of label ambiguity [Geng, 2016]. However, Obtaining label distributions is more challenging than logical labels, as it requires many annotators to manually indicate the degree to

∗Corresponding author

which each label describes an instance and accurately quantifying this degree remains difficult. Thus, [Xu et al., 2019] proposed Label Enhancement (LE), leveraging the topological information in the feature space and the correlation among the labels to recover label distributions from logical labels.

More specifically, LE can be seen as a preprocessing of LDL [Zheng et al., 2021], which takes the logically labeled datasets as inputs and outputs label distributions. As shown in Figure 1, this image reflects the complete information of the sample including some details. Meanwhile, its corresponding logical labels only highlight the most salient features, such as the sky, lake, mountain, and forest. Features contain comprehensive information about samples with many redundancies, while logical labels hold arresting information but are not allsided. Therefore, it is reasonable to assume that features and logical labels can be regarded as two descriptions of instances from different views, possessing complete and salient information of samples. The purpose of LE tasks can be simplified as enhancing the significant knowledge in logical labels by utilizing detailed features. Subsequently, each label is allocated a descriptive degree according to its importance.

Most existing LE methods concentrate on establishing the mapping relationship between features and label distributions under the guidance of logical labels. Although these previous works have achieved good performance for LE problem, they neglect that features and labels are descriptions of two different dimensions related to the same samples. Furthermore, logical labels can only indicate the conspicuous information of each sample without obtaining the label description ranking. The label distributions may appear to be quite different even if the logical labels present the same results.

To address these issues, we propose the ConLE method which fuses features and logic labels to generate the highlevel features of samples by contrastive learning strategy. More specifically, we elaborately train a representation learning model, which forces the features and logical labels of the same instance to be close in projection space, while those of different instances are farther away. By concatenating the representations of features and logical labels in projection space, we get high-level features including knowledge of logic labels and features. Accordingly, label distributions can be recovered from high-level features by the feature mapping network. Since it is expected that the properties of labels in the recovered label distributions should be consistent with those in the logical labels, we design a training strategy with label-level consistency to guide the learning of the feature mapping network.

Our contributions can be delivered as follows:

• Based on our analysis of label enhancement, we recognize that features and logical labels offer distinct perspectives on instances, with features providing comprehensive information and logical labels highlighting salient information. In order to leverage the intrinsic relevance between these two views, we propose the Contrastive Label Enhancement (ConLE) method, which unifies features and logical labels in a projection space to generate high-level features for label enhancement.
• Since all possible labels should have similar properties in logical labels and label distributions, we design a training strategy to keep the consistency of label properties for the generation of label distributions. This strategy not only maintains the attributes of relevant and irrelevant labels but also minimizes the distance between logical labels and label distributions.
• Extensive experiments are conducted on 13 benchmark datasets, experimental results validate the effectiveness and superiority of our ConLE compared with several state-of-the-art LE methods.

2 RelatedWork

In this section, we mainly introduce the related work of this paper from two research directions: label enhancement and contrastive learning.

Label Enhancement. Label enhancement is proposed to recover label distributions from logical labels and provide data preparation for LDL. For example, the Graph Laplacian LE (GLLE) method proposed by [Xu et al., 2021] makes the learned label distributions close to logical labels while accounting for learning label correlations, making similar samples have similar label distributions. The method LESC proposed by [Tang et al., 2020] uses low-rank representations to excavate the underlying information contained in the feature space. [Xu et al., 2022] proposed LEVI to infer label distributions from logical labels via variational inference. The method RLLE formulates label enhancement as a dynamic decision process and uses prior knowledge to define the target for LE [Gao et al., 2021]. The kernel-based label enhancement (KM) algorithm maps each instance to a highdimensional space and uses a kernel function to calculate the distance between samples and the center of the group, in order to obtain the label description. [Jiang et al., 2006]. The LE algorithm based on label propagation (LP) recovers label distributions from logical labels by using the iterative label propagation technique [Li et al., 2015]. Sequential label enhancement (Seq LE) formulates the LE task as a sequential decision procedure, which is more consistent with the process of annotating the label distributions in human brains [Gao et al., 2022]. However, these works neglect the essential connection between features and logical labels. In this paper, we regard features and logical labels as sample descriptions from different views, where we can create faithful high-level features for label enhancement by integrating them into the unified projection space.

Contrastive Learning. The basic idea of contrastive learning, an excellent representation learning method, is to map the original data to a feature space. Within this space, the objective is to maximize the similarities among positive pairs while minimizing those among negative pairs. [Grill et al., 2020; Li et al., 2020]. Currently, contrastive learning has achieved good results in many machine learning domains [Li et al., 2021; Dai and Lin, 2017]. Here we primarily introduce several contrastive learning methods applied to multi-label learning. [Wang et al., 2022] designed a multi-label contrastive learning objective in the multi-label text classification task, which improves the retrieval process of their KNN-based method. [Zhang et al., 2022] present a hierarchical multilabel representation learning framework that can leverage all available labels and preserve the hierarchical relationship between classes. [Qian et al., 2022] propose two novel models to learn discriminative and modality-invariant representations for cross-modal retrieval. [Bai et al., 2022] propose a novel contrastive learning boosted multi-label prediction model based on a Gaussian mixture variational autoencoder (C-GMVAE), which learns a multimodal prior space and employs a contrastive loss. For ConLE, the descriptions of one identical sample are regarded as positive pairs and those of different samples are negative pairs. We pull positive pairs close and negative pairs farther away in projection space by contrastive learning to obtain good highlevel features, which is really beneficial for the LE process.

3 The ConLE Approach

In this paper, we use the following notations. The set of instances is denoted by X = {x1, x2, ..., xn} ∈ Rdim1×n , where dim1 is the dimensionality of each instance and n is the number of instances. Y = {y1, y2, ..., yc} denotes the complete set of labels, where c is the number of classes. For an instance xi, its logical label is represented by Li = (ly1 xi , ly2 xi , . . . , lyc xi )T , where lyj xi can only take values of 0 or 1. The label distribution for xi is denoted by Di = (dy1 xi , dy2 xi , . . . , dyc xi )T , where dyj xi depicts the degree to which xi belongs to label yj. It is worth noting that the sum of all label description degrees for xi is equal to 1. The purpose of LE tasks is to recover the label distribution Di of xi from the logical label Li and transform the logically labeled dataset S = {(xi,Li)|1 ≤ i ≤ n} into the LDL training set E = {(xi,Di)|1 ≤ i ≤ n}. The proposed Contrastive Label Enhancement (ConLE) in this paper contains two important components: the generation of high-level features by contrastive learning and the training strategy with label-level consistency for LE. Overall, the loss function of ConLE can be formulated as follows:

where lcon denotes the contrastive loss for high-level features, latt indicates the loss of training strategy with label-level consistency. The framework of ConLE and the detailed procedure of these two parts is shown in Figure 2.

3.1 The Generation of High-Level Features by Contrastive Learning

The first section provides a detailed analysis of the essence of LE tasks. We regard features and logic labels as two descriptions of samples. Features contain complete information, while logic labels capture prominent details. Label distributions show the description degree of each label. We can't simply focus on the salient information in logical labels, but make good use of salient information and supplement the detailed information according to the original features. To effectively excavate the knowledge of features and logical labels, we adopt the contrastive learning of sample-level consistency.

To reduce the information loss induced by contrastive loss, we do not directly conduct contrastive learning on the feature matrix [Li et al., 2021]. Instead, we project the features (X) and logical labels (L) of all samples into a unified projection space via two mapping networks (F1(·; θ),F2(·; ϕ)), and then get the representations Z and Q. Specifically, the representations of features and logic labels in the projection space can be obtained by the following formula:

where xm and Lm represent the features and logical labels of the m-th sample, Zm and Qm denote their embedded representations in the dim2-dimensional space. θ and ϕ refer to the corresponding network parameters.

Contrastive learning aims to maximize the similarities of positive pairs while minimizing those of negative ones. In this paper, we construct positive and negative pairs at the instance level with Z and Q where {Zm, Qm} is positive pair and leave other (n - 1) pairs to be negative. The cosine similarity is utilized to measure the closeness degree between pairs:

To optimize pairwise similarities without losing their generality, the form of instance-level contrastive loss between Zm and Qm is defined as:

where lZm denotes the contrastive loss for Zm and lQm indicates loss of Qm. Specifically, the item lZm is defined as:

and the item lQm is formulated as:

where τI is the instance-level temperature parameter to control the softness. Further, the instance-level contrastive loss is computed across all samples as:

The expressions Z and Q updated by contrastive learning strategy will be concatenated as high-level features H, which are taken as inputs of the feature mapping network to learn the label distributions:H = concat(Z, Q). (9)

3.2 The Training StrategyWith Label-Level Consistency for LE

Based on the obtained high-level features, we introduce a feature mapping network F3 to generate label distributions. In other words, we have the following formula:

where Dm is the recovered label distribution of the m-th sample and Hm is the high-level feature, and φ denote the parameter of feature mapping network F3 .

In ConLE, we consider the consistency of label attributes in logical labels and label distributions. Firstly, because of recovered label distributions should be close to existing logical labels, we expect to minimize the distance between logical labels and the recovered label distributions, which is normalized by the softmax normalization form. This criterion can be defined as:

where Dm and Lm represents the recovered label distribution and logic label of the m-th sample. Moreover, logical labels divide all possible labels into relevant labels marked 1 and irrelevant labels marked 0 for each sample. We hope to ensure that the attributes of relevant and irrelevant labels are consistent in label distributions and logical labels. This idea is considered in many multi-label learning methods [Kanehira and Harada, 2016; Yan et al., 2016]. Under their inspiration, we apply a threshold strategy to ensure that the description

Algorithm 1 The optimization of ConLE

Input: Training instances X = {x1, x2, ..., xn}; Logical labels L = {L1, L2, ..., Ln}; Temperature parameter τI Output: label distributions D = {D1,D2, ...,Dn}

• 1: Random Initialize θ, ϕ and φ;
• 2: while not converged do
• 3: Obtain {Zm,Qm}nm=1 by Eq. (2) and Eq. (3);
• 4: Obtain the high-level features H by Eq. (9);
• 5: Obtain label distributions D by Eq. (10);
• 6: Optimize θ, ϕ, φ through Eq. (1);
• 7: end while
• 8: return D

degree of relevant labels should be greater than that of irrelevant labels in the recovered label distributions. This strategy can be written as follows:

where Pm is used to indicate the set of relevant labels in xm, Nm represents the set of irrelevant labels in xm, dy+ xm and dyxm are the prediction results of LE process.

In this way, we can get the loss function of threshold strategy:

where ϵ is a hyperparameter that determines the threshold. The formula can be simplified to:

Finally, the loss function of training strategy for label-level consistency can be formulated as follows:

where λ1 and λ1 are two trade-off parameters.

This designed training strategy can guarantee that label attributes are the same in the logical labels and label distributions, thus obtaining a better feature mapping network to recover label distributions. The full optimization process of ConLE is summarized in Algorithm 1.

4 Experiments

4.1 Datasets

We conduct comprehensive experiments on 13 real-world datasets to verify the effectiveness of our method. To be specific, SJAFFE dataset [Lyons et al., 1998] and SBU-3DFE dataset [Yin et al., 2006] are obtained from the two facial expression databases, JAFFE and BU-3DFE. Each image in datasets is rated for six different emotions (i.e., happiness, sadness, surprise, fear, anger, and disgust) using 5-level scale. The Natural Scene dataset is collected from 2000 natural scene images. Dataset Movie is about the user rating for 7755 movies. Yeast datasets are derived from biological experiments on gene expression levels of budding yeast at different time points [Eisen et al., 1998]. The basic statistics of these datasets are shown in Table 1.

4.2 EvaluationMeasures

The performance of the LE algorithm is usually calculated by distance or similarity between the recovered label distributions and the real label distributions. According to [Geng, 2016], we select six measures to evaluate the recovery performance, i.e., Kullback-Leibler divergence (K-L)↓, Chebyshev distance (Cheb)↓, Clark distance (Clark)↓, Canberra metric (Canber)↓, Cosine coefficient (Cosine)↑ and Intersection similarity (Intersec)↑. The first four are distance measures and the last two are similarity measures. The formulae for these six measures are summarized in Table 2.

4.3 ComparisonMethods

We compare ConLE with six advanced LE methods, including FCM [Gayar et al., 2006], KM [Jiang et al., 2006], LP [Li et al., 2015], GLLE [Xu et al., 2021], LEVI-MLP [Xu et al., 2022] and LESC [Tang et al., 2020]. The following are the datails of comparison algorithms used in our experiments:

1) FCM: This method makes use of membership degree to determine which cluster each instance belongs to according to fuzzy C-means clustering.

1. KM: It is a kernel-based algorithm that uses the fuzzy SVM to get the radius and center, obtaining the membership degree as the final label distribution.
2. LP: This approach applies label propagation (LP) in semi-supervised learning to label enhancement, employing graph models to construct a label propagation matrix and generate label distributions.
3. GLLE: The algorithm recovers label distributions in the feature space guided by the topological information.
4. LEVI-MLP: It regards label distributions as potential vectors and infers them from the logical labels in the training datasets by using variational inference.
5. LESC: This method utilizes the low-rank representation to capture the global relationship of samples and predict implicit label correlation to achieve label enhancement.

4.4 Experimental Results

Implementation Details. In ConLE, we adopt the SGD optimizer [Ruder, 2016] for optimization and utilize the LeakyReLU activation function [Maas et al., 2013] to implement the networks. The code of this method is implemented by PyTorch [Paszke et al., 2019] on one NVIDIA Geforce GTX 2080ti GPU with 11GB memory. All experiments for our selected comparison algorithms follow the optimal settings mentioned in their papers and we run the programs using the code provided by their relevant authors. All algorithms are evaluated by ten times ten-fold cross-validation for fairness. When comparing with other algorithms, the hyperparameters of ConLE are set as follows: λ1 is set to 0.5, λ2 is set to 1 and the temperature parameter τI is 0.5.

Recovery Performance. The detailed comparison results are presented in Table 3, with the best performance on each dataset highlighted in bold. For each evaluation metric, ↓ shows the smaller the better while ↑ shows the larger the better. The average rankings of each algorithm across all the datasets are shown in the last row of each table.

The experimental results clearly indicate that our ConLE method exhibits superior recovery performance compared to the other six advanced LE algorithms. Specifically, ConLE can achieve the ranking of 1.00, 1.23, 1.00, 1.07, 1.15 and 1.00 respectively for the six evaluation metrics. ConLE obtains excellent performance both on large-scale datasets such as movie and small-scale datasets such as SJAFFE. ConLE can attain significant improvements both in comparison with algorithm adaption and specialized algorithms by exploring the description consistency of features and logical labels in the same sample. We integrate features and logical labels into the unified projection space to generate high-level features and keep the consistency of label attributes in the process of label enhancement.

Ablation Studies. Our ConLEmethod consists of twomain components: generating high-level features by contrastive learning and a training strategy with label-level consistency for LE. Ablation studies are conducted to verify the effectiveness of the two modules in our method.

Therefore, we first remove the part of ConLE that generates high-level features and get a comparison algorithm ConLEh, whose loss function can be written as:

In ConLEh, we only explore the consistency information of label attributes without considering the description consistency of features and labels.

Secondly, we need to remove the strategy that ensures the consistency of label attributes. To ensure the normal training process, we still keep the strategy of minimizing the distance between label distributions and logical labels. The loss function of the comparison function ConLEl:

Table 4 provides the recovery results of ConLEh, ConLEl and ConLE. Due to the limitation of space, only the representative results measured on Kullback-Leibler, Clark, Canberra and Intersection are shown in the table. From the experimental results, we can observe that ConLE is superior to ConLEh and ConLEl in all cases. Compared with ConLEh, ConLE considers the inherent relationship between features and logical labels. It grasps the description consistency of samples and constructs high-level features for training. Compared with ConLEl, ConLE considers label-level consistency of logical labels and label distributions. It makes that each relevant label in the logical labels has a greater description degree in the label distributions. Therefore, our experimental results have verified that both modules of ConLE play essential roles in achieving excellent recovery performance. The integration of these modules in the complete ConLE method has been demonstrated to be highly effective.

Parameters Sensitivity. To investigate the sensitivity of ConLE to hyperparameters, we performed experiments on SBU-3DFE with different values of the two trade-off hyperparameters λ1 and λ2. In this experiment, we fix one hyperparameter and choose another hyperparameter from {0.1, 0.3, 0.5, 0.8 ,1, 5, 10}. As shown in Figure 3, we can observe that the ConLE method can obtain satisfactory recovery results and our model is insensitive to λ1 and λ2.

Convergence Analysis. To illustrate the convergence of ConLE, we present an experiment conducted on Movie dataset by Canberra↓ as an example, with the corresponding convergence curve depicted in Figure 4. The value of the objective function decreases and the performance increases with more iterations. Finally, they tend to be stable. The properties remain the same for all datasets.

5 Conclusion

In this work, we propose Contrastive Label Enhancement (ConLE), a novel method to cope with the (Label Enhancement) LE problem. ConLE regards features and logic labels as descriptions from different views, and then elegantly integrates them to generate high-level features by contrastive learning. Additionally, ConLE employs a training strategy that considers the consistency of label attributes to estimate the label distributions from high-level features. Experimental results on 13 datasets demonstrate its superior performance over other state-of-the-art methods.

Acknowledgments

This work was supported by the National Key R&D Program of China under Grant 2020AAA0109602.