Decoupled Imbalanced Label Distribution Learning

1 Introduction

Label Distribution Learning (LDL) [Geng, 2016] aims to establish a mapping from an instance to a label distribution, which encompasses a specific set of labels and indicates the degree to which each label describes the instance. LDL has been shown to be an effective approach to address the issue of label ambiguity [Wang and Geng, 2019; Kou et al., 2024b;

∗Corresponding Author

Wang et al., 2024]. The underlying mechanism of label distribution mapping has also attracted significant research attention [Shen et al., 2017; Lu and Jia, 2024; Li et al., 2024]. LDL has been successfully applied to numerous practical scenarios and learning paradigms, including expression analysis [Le et al., 2023], video parsing [Gao et al., 2021; Zhang et al., 2023], age estimation [Smith-Miles and Geng, 2020; Zhang et al., 2021], image captioning [Yang et al., 2023], beauty sense [Xie et al., 2015; Ren and Geng, 2017], fewshot learning [An et al., 2024], multi-label learning [Wang and Geng, 2024; Kou et al., 2024b], and partial label learning [Xu et al., 2023], big model [Peng et al., 2025], etc.

LDL directly addresses the deeper and more ambiguous question: "Howmuch does each label describe the instance?"

In other words, it considers the relative importance of each label in describing the instance. The effectiveness of LDL hinges on the relatively balanced nature of the description degrees, with generally small differences among them, enabling the use of more balanced supervision information to train the LDL model. However, achieving highly balanced description degrees is quite challenging in practical applications due to the significant subjectivity involved in annotating the label distribution. For example, when training a score distribution model for movies [Geng and Hou, 2015], obtaining an ideally balanced emotional distribution like Fig. 1(a) requires hundreds or even thousands of annotators to label the same movie. Yet, due to limitations such as insufficient numbers of annotators, varying backgrounds, ages, subjective opinions, annotation noise, etc., it is common for the score distribution to become imbalanced, as shown in Fig. 1(b). The dominant scores occupy an excessively large descriptive space, leaving very little space for the non-dominant labels. The excessive variance among these description degrees can severely impact the performance of label distribution learning in solving practical problems [He and Garcia, 2009; Wu et al., 2020; Oh et al., 2022; Fu et al., 2024]. This newly emerging and challenging scenario is defined as Imbalanced Label Distribution Learning (ILDL) [Zhao et al., 2023b]. RDA [Zhao et al., 2023b] uncovers the underlying reason behind the performance degradation of imbalanced distribution learning from an alignment perspective. RDA asserts that existing LDL methods incorrectly assume that the consistency between the feature distributions of the training set and the test set is invalid. Consequently, RDA introduces a two-phase alignment approach.

Both LDL-HR [Wang and Geng, 2021b] and DKD [Zhao et al., 2022] methods demonstrate that non-dominant labels can either enhance model generalization or transfer implicit knowledge. However, RDA solely focuses on enhancing the representation capabilities of the feature space and label space from a representation alignment perspective, without addressing the issue of excessive attenuation of non-dominant labels in LDL model learning due to the over-representation of dominant labels.

To solve the above issue, we propose a novel method named Decoupled Imbalanced Label Distribution Learning (DILDL). This method decouples the imbalanced label distribution into dominant and non-dominant label distributions. Specifically, we divide the label distribution learning process into two levels: (1) prediction of description degrees for dominant labels and (2) prediction of description degrees for nondominant labels. Based on this division, we reformulate the LDL loss into two components, as illustrated in Figure 1(c). Despite the inherent imbalance in the initial label distribution, our decoupling methodology successfully establishes an equilibrium in the description degrees between dominant and non-dominant label distributions. After decoupling, the description degrees of non-dominant labels become independent of those of dominant labels, eliminating the need for their concurrent learning. From the perspective of gradient analysis, we demonstrate that decoupling can enhance the learning of gradient information related to non-dominant labels.

The overall framework of the proposed decoupled imbalanced label distribution learning (DILDL) is shown in Figure 2. We utilize the DILDL loss to learn the mapping from instances to label distributions in the first branch of the decoder Fθ. More importantly, DILDL also decouples the label distribution during the representation distribution alignment stage. Our contributions can be summarized as follows:

• We propose a decoupled method to address the issue of excessive emphasis on dominant labels, which leads to excessive attenuation of non-dominant labels in imbalanced label distribution learning.
• We prove that the decoupled approach in ILDL can further enhance the implicit knowledge of non-dominant labels from a gradient analysis perspective.
• We conduct extensive experiments to demonstrate the effectiveness of the proposed Decoupled Imbalanced Label Distribution Learning (DILDL), and our method achieves state-of-the-art performance.

2 RelatedWork

Label Distribution Learning. Label distribution learning (LDL) was first proposed by [Geng, 2016], and it has been successfully applied to ambiguous tasks. For example, [Geng and Hou, 2015] formulated movie scores from multiple annotators as a score distribution and simultaneously fit a sigmoid function to each component of the score distribution using a multi-output support vector machine. LDL-ALSG [Chen et al., 2020] was proposed to address the facial expression recognition using the topological information of labels from related but more distinct tasks. The underlying assumption of LDL-ALSG is that facial images should have similar expression distributions to their neighbors in the label space of action unit recognition and facial landmark detection. More recently, [Le et al., 2023] leveraged neighborhood information in the valence-arousal space to adaptively construct emotion distributions for training samples, taking into account the uncertainty of provided labels when incorporating them into the label distributions to solve the facial expression recognition problem. Additionally, LDL has been applied to age estimation [Zhang et al., 2021], image sentiment analysis [Yang et al., 2017a; Yang et al., 2017b], text classification [Zhao et al., 2023a], and other areas. Beyond its contributions across various application areas, LDL can further enhance the performance of other machine learning paradigms. For example, [Kou et al., 2024b] introduced an auxiliary multi-label learning (MLL) process in LDL to capture low-rank label correlation on that MLL rather than LDL, justifying the advantages of exploiting low-rank label correlation in the auxiliary MLL through LDL. LDL-FCC [Wang et al., 2024] was designed to explore fuzzy membership-induced correlation and to jointly realize fuzzy clustering and label correlation learning via LDL. However, as analyzed in [Zhao et al., 2023b], previous LDL methods ignored the imbalance problem in label distribution and did not consider the distribution gap between the training set and test set.

Imbalance Label Distribution Learning. Imbalanced learning is a pressing topic that arises from the long-tail distribution of data. Under-sampling and over-sampling are two of the most popular approaches to tackle the long-tail recognition problem [Byrd and Lipton, 2019; Buda et al., 2018; Ghosh et al., 2024]. [Soltanzadeh et al., 2023] addresses the imbalance issue by presenting an under-sampling approach based on a metaheuristic method, where the under-sampling problem is formulated as an optimization problem. The proposed method aims to select an optimal subset of majority samples to handle both the imbalance and class-overlap problems simultaneously, while avoiding excessive elimination of majority samples. Apart from imbalanced classification, imbalanced regression has also garnered significant attention in recent years [Yang et al., 2021; Wang and Wang, 2024; Liu et al., 2023]. VIR [Wang and Wang, 2024] borrows data with similar regression labels to compute the variational distribution of latent representations, predicts the entire normalinverse-gamma distributions, and modulates the associated conjugate distributions to probabilistically re-weight the imbalanced data. However, both imbalanced classification and imbalanced regression focus on identifying imbalances at the label end. The uniqueness of imbalanced label distribution learning (ILDL) lies in the fact that each label is accompanied by a continuous description degree [Geng, 2016; Kou et al., 2024a; Wang and Geng, 2019]. Consequently, traditional imbalanced classification and imbalanced regression methods cannot be directly applied to ILDL. To address this, RDA [Zhao et al., 2023b] introduced the first specialized ILDL algorithm, which aligns the distributions of feature representations and label representations to bridge the gap between the training set and test set caused by imbalance. However, RDA does not effectively address the issue of excessive attenuation of non-dominant labels in ILDL model learning.

3 Approach

Inspired by LDL-HR [Wang and Geng, 2021b], AEKT [Park and Lee, 2024], and DKD [Zhao et al., 2022], we reformulate the ILDL loss as a weighted sum of two components: one representing the distribution of dominant labels and the other representing the distribution of non-dominant labels. Additionally, we integrate our decoupled method into the RDA [Zhao et al., 2023b] framework. Decoupling is implemented for dominant and non-dominant labels during both the label distribution prediction and representation alignment stages.

3.1 Decoupled the Label Distribution

Assume fθ(·) is the mapping function from the instance space X to the label distribution space Y. The objective function of LDL is to minimize the difference of the ground truth and predicted label distribution. The Kullback-Leibler (KL) divergence is the most common used loss function. Therefore, the objective function can be written as,

where dyj xi is the groundtruth of the description degree. To start, we decouple the distribution of dominant label. Specifically, for each instance x, the distribution of dominant label is defined as ¯Dx = [ ¯ dy1 x , ¯ dy2 x , ..., ¯ dyc x ]T , where ¯ dyj x is defined by

where yx is the label with the highest description degree, the decoupled distribution assigns a description degree of 1 to the dominant label and 0 to all other labels. According to the definition of LDL, the label distribution satisfies two constraints, i.e. dyj xi ∈ [0, 1] and ∑cj=1 dyj xi = 1, the distribution of non-dominant labels is defined as ˆDx = [ ˆ dy1 x , ˆ dy2 x , ..., ˆ dyc x ]T , where ˆ dyj x is defined by

Therefore, Eq. (1) can be re-written as,

where α is the trade-off parameter to balance the decoupled label distributions. ˆ f(·) is the decoupled prediction of nondominant labels. d′ is the predicted description degree.

The first term of Eq.(4) for the distribution of the dominant label can be written as,

Follow the gradient calculation method in AEKT [Park and Lee, 2024], the gradient of LDDL with the logit zk i of the dominant label yx can be calculated as follows,

The gradient of LDDL with the logit zj i of the non-dominant label yj can be calculated as follows,

The second term of Eq. (3) can be re-formulated as,

The gradient of LNDDL with respect to the logits zj i of non-dominant labels yj is calculated by,

In summary, multiplying the decoupled gradients by the balanced parameter α and β can yield the gradients for dominant labels and non-dominant labels,

Based on the gradient analysis of the distributions of dominant and non-dominant labels, we can draw the follow the two conclusions:

• When the label distribution is very imbalance, i.e. the margin between ¯ dyk xi and ˆ dyk xi is very large, as shown in Eq. (11), the imbalance in label distribution can cause the LDL model to overly focus on dominant labels, leading to a small description degree ¯ dyj xi for non-dominant labels (the second term in Eq. (11)), the gradient of the non-dominant labels ∂LLDL/∂zj i can be still delivered during the learning process since the β gives an independent balance for the distribution of non-dominant labels.
• As shown in the first term in Eq. (11), when the predicted label distribution ˆ d′ yk xi is large from the ground truth ˆ dyk xi , the weight for the predicted label distribution of the non-dominant labels ˆ d′ yj xi will be reduced, which will further address the issue of excessive losses resulting from inaccurate label distribution prediction during the learning process.

3.2 Decoupled Representation Distribution Alignment

RDA [Zhao et al., 2023b] asserts that aligning the distributions of feature representations and label representations can narrow the distribution gap between the training set and test set, which is often exacerbated by the imbalance issue. Inspired by RDA [Zhao et al., 2023b], we introduce the decoupled representation alignment approach to carry out distinct representation learning for both dominant and non-dominant labels.

As depicted in Figure 2, we design feature and label encoders to learn the representations of instances and label distributions, respectively. Specifically, the label distribution encoder consists of three branches, with two branches specifically dedicated to learning the representations of dominant and non-dominant labels.

The Central Limit Theorem as well as the informationtheoretic state that the sum/mean of many independent random variables approximates a Gaussian distribution under suitable conditions and the Gaussian distribution has maximum entropy, making it the most "uncertain" (least assuming) distribution for data. Consequently, without specific distribution info, Gaussian is the default for label-feature alignment. Assuming that the distributions of features and labels adhere to Gaussian distributions, we utilize the KL divergence to align the decoupled label information with the feature representation of the model.

resents the i-th element of the latent space. Similarity,

To further align the distributions of feature representations and label representations, following RDA [Zhao et al., 2023b], we use the reparameterization trick [Rezende et al., 2014] to calculate the similarity of features and labels. For the labels: rlabelDis = µwhole + σwholeδwhole, where µwhole and σwhole are calculated from the whole label encoder. δwhole ∼ N(0, I). For the features: rfeature = µfeature + σfeatureδfeature, where µfeature and σfeature are calculated from the feature encoder. δfeature ∼ N(0, I). The alignment-3 is defined as,

where Amn and Zmn are cosine similarity matrix of feature representations and label representations,

where m and n are m-th and n-th instances. In Figure 2, each element of the similarity matrix is abbreviated as S(rmn feature ) or S(rmn labelDis )

To reduce the likelihood of significant disparities in the label distribution representation, rlabelDis derived using the reparameterization technique from the Gaussian distribution of label representation are fed into the decoder Fde, enabling an alignment between the predicted label distribution and the ground truth distribution.

where D indicates the ground truth, rlabelDis is the reparameterization result from the Gaussian distribution of the label representation. Fde is the decoder network, which is represented by three green cubes in Figure 2.

3.3 Objective

During the training phase, the optimization objective of our proposed DILDL consists of six components, namely, two decoupled imbalance label distribution learning losses and four alignment losses.

where α, β, γ and λ are balance parameters to balance the weight during the training process in the total loss Ltotal. In the inference stage, the predicted label distribution can be obtained from Fdecoder(Fencoder(x∗ )).

4 Experiment

In this section, we conduct extensive experiments on six ILDL datasets, which are sampled from standard LDL datasets, to assess the effectiveness of our proposed decoupled imbalance label distribution learning approach. In the following subsections, we will report on the datasets used, the evaluation metrics, the experiment setup, the results, and further analysis. All experiments were implemented using the PyTorch framework and executed on one NVIDIA GeForce RTX 4060 GPU. The code of the paper has been opensourced.

4.1 Datasets and Evaluation

The datasets encompass a diverse range of sources, including SCUT-FBP [Xie et al., 2015], Flicker-LDL [Yang et al., 2017a; Yang et al., 2017b], Movie [Geng and Hou, 2015], Emotion6 [Peng et al., 2015], Natural Scene [Geng, 2016], and RAF-ML [Li and Deng, 2019], each providing unique insights and challenges for our research.

To ensure the robustness and generalizability of our model, we adopt a rigorous experimental design. Specifically, we randomly split each dataset 10 times, allocating a substantial portion of 90% of the data to the combined training and validation sets. Within this 90%, we typically further subdivide the data into separate training and validation subsets to finetune our models and prevent overfitting. The remaining 10% of the data is reserved for the test set, serving as an unbiased evaluation of our model's performance on unseen data.

To better verify the proposed DILDL, following ILDL [Zhao et al., 2023b], four distance metrics (Chebyshev ↓, Clark↓, Canberra↓, Kullback-Leibler↓) and two similarity metrics (Cosine↑, Intersection↑) are adopted to evaluate the performance of all the methods. The "↓" after the distance metrics indicates "the smaller the better", and the "↑" after the similarity metrics indicates "the larger the better".

4.2 Implementation Details

We select 10 comparison methods, which fall into three major categories: LDL algorithms, adaptation ILDL algorithms, and specially designed ILDL algorithm. SA-BFGS [Geng, 2016], EDL-LRL [Jia et al., 2019b], LDLSF [Ren et al., 2019a], LDL-LCLR [Ren et al., 2019b], Adam-LDLSCL [Jia et al., 2019a] and LDL-LDM [Wang and Geng, 2021a] are six state-of-the-art LDL algorithms. Following the objective function reshaping method [Zhao et al., 2023b], OFR-FL, OFR-CB, OFR-DB techniques are used to reshape the LDL algorithms into three adaptation ILDL approaches. In addition, the latest specially designed ILDL algorithm RDA [Zhao et al., 2023b] is also selected as the comparison method.

The learning rate is set 0.001. The batch size is 50. The trade-off parameter α in Eq. (17) is 0.6, which is selected from parameter sensitivity analysis. The trade-off parameters λ, β and γ for alignment are set 0.1. The maximum epoch is 300. During the inference stage, the predicted label distribution is obtained from the decoder network after the feature encoder.

4.3 Results

Table 1 and 2 show the main results of different methods on six dataset measured by Chebyshev and KL divergence. The best performances are highlighted in bold. And ·/◦ represents whether our proposed method is statistically superior/inferior to the comparing methods, which is calculated by two-tailed t-test under 0.05 significance level. From the results, our proposed algorithm consistently achieves the lowest Chebyshev distance across all datasets, indicating superior performance compared to the other algorithms. Specifically, our proposed method outperforms all others with a distance of 0.1752±0.0091, significantly lower than the next best (RAD with 0.1962±0.0068) on Movie dataset. Similarly, our method achieves 0.2684±0.0182, outperforming RAD (0.2849±0.0157) and other competitors on SCUT-FBP dataset. Our proposed algorithm also demonstrates the best performance in terms of KL divergence across all datasets, consistent with the Chebyshev distance results. For example, on Emotion6 dataset, our method gains the best performance with a divergence of 0.7393±0.0254. Our method's divergence of 1.5011±0.1165 is notably lower than RAD's 1.6071±0.1107 on Flickr LDL, underlining its efficiency. Table 5 also verifies the effectiveness of our method.

In summary, the experimental results clearly demonstrate that our proposed algorithm significantly outperforms the other algorithms in terms of both Chebyshev distance, KL divergence and Euclidean distance across multiple ILDL datasets. These findings underscore the robustness and effectiveness of our method in tackling imbalance label distribution learning tasks. The consistent lower distances and divergences achieved by our algorithm across diverse datasets suggest its generalizability and potential for practical applications where accurate imbalance label distribution learning is paramount importance.

4.4 Ablation Study

The decoupled section of our proposed method seamlessly integrates both the decoupled label distribution and the decoupled representation distribution alignment. To evaluate the impact of each component within our proposed method, we conducted a rigorous ablation study by defining variations of DILDL. Specifically, "DRDA" denotes the decoupled representation distribution alignment component, while "DLD" signifies the decoupled label distribution component.

Tables 3 and 4 present the results of the ablation studies conducted on both the Movie dataset and the Natural Scene dataset, offering insights into the influence of the DRDA and DLD components across various evaluation metrics. The table for the Movie dataset (Table 4) illustrates the performance of different configurations with and without DRDA and DLD. When both DRDA and DLD are active (indicated by checkmarks), we observe significant improvements across all metrics compared to when either or both are deactivated. Similarly, the results for the Natural Scene dataset (Table 5) demonstrate the advantages of combining DRDA and DLD. For instance, the result decreases from 1.118 to 1.063 when both DRDA and DLD are enabled, indicating better alignment of distributions. Without DRDA, the result increases from 1.063 to 1.096 under the KL divergence metric, which underscores the effectiveness of the proposed DRDA. The combined results from both datasets consistently reveal that enabling both DRDA and DLD leads to enhanced performance across all metrics. This underscores the positive contribution of both components in improving the quality of imbalance label distribution learning.

4.5 Parameter Sensitivity Analysis

To select the optimal balance parameters, we compare the performances of our proposed DILDL with various values of hyperparameters across six datasets, evaluating the performance using the Chebyshev distance. Figure 3 illustrates the results of the parameter sensitivity analysis. From the figure, we can conclude that the parameter α exhibits a higher degree of sensitivity compared to the other three parameters, suggesting that meticulous tuning of this parameter can further enhance the model's performance. When α = 0.6, the model's performance is optimal in most cases. Additionally, DILDL exhibits better performances with 0.1 for the other three hyperparameters.

5 Conclusion

LDL has been proven to be an efficient learning paradigm for solving the label ambiguity problems. However, it may encounter the challenge posed by ILDL. In this paper, we propose a novel ILDL method named Decoupled Imbalanced Label Distribution Learning (DILDL). DILDL decomposes imbalanced label distributions into dominant and non-dominant components. By employing the decoupling approach, we independently balance the description degrees of both dominant and non-dominant labels. Furthermore, we separately align feature representations with those of dominant and nondominant labels, thereby significantly mitigating the issue of gradient information attenuation for non-dominant labels. Experimental results demonstrate that our proposed method outperforms other methods. In the future, we aim to explore the application of feature decoupling methods to further enhance the performance of the DILDL approach.

Acknowledgements

This work was supported by the National Natural Science Foundation of China under Grant No. 62406155, the project No. ZR2024QF115 supported by Shandong Provincial Natural Science Foundation, the National Natural Science Foundation of China under No. 62471202 and No. 62476135, the Innovation Capability Enhancement Project for Technologybased Small and Medium-sized Enterprises of Shandong Province under Grant No. 2024TSGC0777, the Opening Fund of Shandong Provincial Key Laboratory of Ubiquitous Intelligent Computing, the Shandong Province Youth Innovation Team Project under Grant No. 2024KJH032), and the Development Program Project of Youth Innovation Team of Institutions of Higher Learning in Shandong Province.