Label Enhancement via Joint Implicit Representation Clustering

1 Introduction

Label polysemy, i.e., the cases where an instance can be described by multiple labels simultaneously, is common in practical machine learning tasks. Label distribution is an effective label form to describe label polysemy, where each label is assigned a real value to indicate how much this label describes the instance. Since label distribution provides fine-grained information about label polysemy, it has been widely applied in many practical tasks, such as emotion analysis [Jia et al., 2019; He and Jin, 2019; Peng et al., 2015], age estimation [Gao et al., 2018; Wen et al., 2020; Shen et al., 2021], and so on.

A popular topic regarding label distribution is how to predict label distributions for unseen instances. LDL (label distribution learning) [Geng, 2016] is an effective learning paradigm for solving this problem. However, LDL requires the training instances annotated with label distributions, yet quantifying label distributions is costly and even impractical.

∗Corresponding author is Weiwei Li.

Therefore, a technique called LE (label enhancement) [Xu et al., 2018] was proposed to solve this problem. In general, existing works about LE mostly assume that instances annotated with simple labels (such as logical labels [Xu et al., 2021] or multi-label rankings [Lu and Jia, 2022]) are available, then label distributions can be recovered by mining the instance relation or the label relation, and finally any LDL algorithm can be trained to predict label distributions for unseen instances. For example, in terms of the instance relation, some algorithms [Hou et al., 2016; Zheng et al., 2023; Wen et al., 2021] represent the feature vector of each instance as a linear combination of the feature vectors of its neighbors and then make the instances retain this combination in the label distribution space. There are also some algorithms [Xu et al., 2018; Zhang et al., 2021; Xu et al., 2023] that construct an instance graph (whose edges encode the similarities of instances in the feature space), and use the Laplacian matrix of this graph to regularize the label distributions. In terms of label relation, N-LDL [Luo et al., 2021] exploits a global label correlation, i.e., all instances share a common label correlation pattern. Besides, more algorithms [Lv et al., 2019; Xu et al., 2021; Jia et al., 2023] leverage local label correlations, i.e., instances from different clusters are regularized by different label correlation patterns.

Overall, above algorithms mostly rely on two processes: mining the instance relation, and clustering instances (for capturing more accurate local label correlations). In terms of these two processes, existing algorithms suffer from the following three problems: 1) These algorithms merely mine the instance relation in the feature space; however, due to the semantic differences between features and labels, regularizing label distributions with the instance relation entirely dependent on features can lead to considerable errors. Moreover, simple labels are more semantically consistent with label distributions, while their potential instance relation is ignored by most algorithms. As shown in Figure 1, the overall error of using only features (blue part) significantly exceeds that of using both features and simple labels (orange part). 2) Existing approaches for mining instance relation are mainly tailored to tabular features; however, real-world features are often diverse (e.g., images, texts, or graphs), and it is unreliable to naively apply these approaches to diverse features. 3) Although some algorithms try to extract richer representations by the kernel method, they typically treat instance clustering as independent of learning representations, which may lead to suboptimal results.

Therefore, we propose a deep generative model called JRC to simultaneously learn and cluster the joint implicit representations of both features and simple labels, which can be used to improve any existing LE algorithm that involves mining the instance relation or local label correlations. As shown in Figure 2(a), we treat the joint implicit representations as lowdimensional latent vectors Z = {zi}ni=1 , which can probabilistically generate the feature matrix X and the simplelabel-based instance graph A, then model the variational posterior of zi by the A-based graph convolutional network, and finally employ the variational Bayes to infer the implicit representation. To encourage the model to adaptively explore instance clusters, inspired by VaDE [Jiang et al., 2017], we assume that zi is generated from a Gaussian mixture prior, i.e., the cluster to which zi belongs is dominated by a categorical indicator c. Besides, we propose a novel label distribution recovery module which is then integrated with JRC, thus constituting a novel generative label enhancement model called LEIC (i.e., Label Enhancement model via joint Implicit representation Clustering). As shown in Figure 2(b), in order to utilize the joint implicit representations, we additionally assume that the Gaussian priors of the implicit representations in JRC are conditionally controlled by the label distributions D; in order to utilize local label correlations, we use the cluster posteriors to reweight the instances and construct multiple cluster-specific label correlation graphs which can be then incorporated into the variational posteriors of label distributions via graph convolutional networks. Besides, following LEVI [Xu et al., 2020] and GLEMR [Lu et al., 2023], we also assume that label distributions can generate simple labels Y due to their strong consistency. Finally, we conduct extensive experiments to show the effectiveness of our proposal.

Our main contributions can be summarized as:

• We propose a deep generative model to learn the joint implicit representations of both features and simple la-

bels, which facilitates the acquisition of more accurate instance relations and enables dealing with various forms of features.

• We incorporate Gaussian mixture clustering and implicit representation learning into a unified framework, which encourages instance clustering and representation learning to accommodate each other and thus facilitates the discovery of more accurate local label correlations.
• We propose a novel generative LE model to recover label distributions, which utilizes the instance relation and local label correlations in a principled way and performs the processes of learning and clustering joint representations and recovering label distributions end-to-end.

2 RelatedWork

Traditional label enhancement considers how to recover label distributions from logical labels. Recently, there was also work that considers recovering label distributions frommultilabel rankings. Therefore, label enhancement can be defined in a general sense as the process of enhancing simple labels into label distributions. Simple labels, either logical labels or multi-label rankings, are mostly enhanced into label distributions by the following ways. On the one hand, the most widely studied is estimating label distributions by mining instance relations. Specifically, some algorithms [El Gayar et al., 2006; Jiang et al., 2006; Wang et al., 2023] find the representative points of each label in the feature space and then estimate the relative importance among the labels based on the distance from each instance point to the representative points. Some algorithms [Xu et al., 2018; Zhang et al., 2021; Xu et al., 2019; Liu et al., 2021b] directly compute pairwise instance distances (or affinities) according to the feature vectors and use them to regularize the label distribution vectors. Some algorithms use the technique of manifold learning [Izenman, 2012; Bengio et al., 2013]; they re-express the feature vector of each instance as a linear combination of the feature vectors of its neighbors and then make the instances retain this combination in the label distribution space. Specifically, some of these algorithms [Hou et al., 2016; Shao et al., 2018; Liu et al., 2021a] do not impose additional constraints on the re-expression coefficients; some algorithms [Wen et al., 2021; Tang et al., 2020; Zheng et al., 2023] assume that the re-expression coefficients matrix are of low rank; some algorithms [Zhang et al., 2018; Lv et al., 2019; Xu et al., 2023] consider these coefficients to be sparse.

On the other hand, label relations were also studied in many label enhancement works. For example, N-LDL [Luo et al., 2021] mines global label correlation, i.e., all instances share a label correlation pattern. Some algorithms [Jia et al., 2023; Xu et al., 2021] mines local label correlation, i.e., instances belonging to the same cluster share a label correlation pattern.

3 Methodology

3.1 ProblemFormulation

We deal with the datasets that appear as pairs {(xi, yi)}ni=1 , where xi and yi denote the i-th vector of features and the ith vector of simple labels (either logical labels or multi-label rankings are possible), respectively. X = [x1; · · · ; xn] and Y = [y1; · · · ; yn] denote the feature matrix and simple label matrix, respectively. The goal of LE is to infer latent label distributions D = [d1; · · · ; dn] based on {(xi, yi)}ni=1 .

3.2 JRCModel

Generation Process

Since we use a deep generative model to infer the joint implicit representations of instances, we first describe the process of generating data points.

1. Specify the index of the Gaussian prior from a categorical distribution: p(c) = ∏ni=1 Cat(ci|k-1 · 1k), where 1k is a k-dimensional all-ones vector.
2. Generate joint implicit representations from the chosen Gaussian: p(Z|c) = ∏ni=1 N (zi|µci , diag(σ2 ci )), where µci and σci are learnable parameters.
3. Generate the observations of feature vectors: p(X|Z) = ∏ni=1 N (µzi , diag(σ2 zi )), [µzi ; ζ -1 (σzi )] = f(zi; θz), where f(zi; θz) is given by a neural network with parameters θz; ζ -1 (·) is the inverse of softplus function.
4. Generate the adjacency matrix of the instance graph based on Y: p(A|Z) = ∏ni=1 ∏nj=1 Ber(Aij |σ(z⊤i zj )), where the adjacency matrix is obtained by

where I(·) is the indicator function, knn(x) denotes the set of instances near x, and in this paper we consider 20 nearest neighbors, i.e., |knn(x)| = 20.

The above generation process will derive the following factorization of the joint density:

Variational Inference

Since obtaining the exact posteriors of the latent variables in the generative model is intractable, we approximate them by the variational distributions. As shown in Figure 2(a), we assume that the variational posterior of latent variables can be factorized as:

Here, to fully exploit the instance relation from simple labels, we use the graph convolutional network to parameterize q(Z|A,X), i.e.,

where g(X,A;φx) denotes a graph convolutional network with adjacency matrix of A and learnable parameter of φx; gi(X,A;φx) is defined as the i-th vector of g(X,A;φx). Besides, the cluster posterior can be obtained by:

We aim to minimize the KL divergence between the variational posterior and the true posterior, which is equivalent to maximizing the ELBO (evidence lower bound). According to the factorizations Equation (1) and Equation (2), the ELBO can be rewritten as:

In Equation (5), the first term measures the likelihood that the variational posteriors reconstruct the observed data; The second and third terms measure the similarity between the learned variational posteriors and the prior beliefs. In order to optimize Equation (5) using gradient ascent techniques, we turn to SGVB estimator [Kingma and Welling, 2014]. Since q(zi|A,X) is Gaussian, it can be reparameterized by:

Then the first term in Equation (5) can be estimated by:

where L is the number of Monte Carlo samples, ϵ(l) i is a sample from the standard normal distribution. Since c is discrete, the third term has a closed form, i.e.,

and the second term can be rewritten as:

where the KL divergence between two Gaussian distributions, i.e., KL(q(zi|A,X)‖p(zi|ci = j)), has a closed form [Kingma and Welling, 2014]. Substituting Equation (7), Equation (8) and Equation (9) into Equation (5) yields the optimization objective of JRC.

3.3 LEICModel

Generation Process

Based on the JRC model, the generation process of LEIC makes two additional assumptions as follows: 1) Parameters of the Gaussian mixture prior of z are conditionally dominated by label distributions, which follows a Dirichlet prior. 2) Label distributions can probabilistically generate simple labels. According, the joint density can be factorized as:

where p(c), p(X|Z), and p(A|Z) are the same as in JRC; the prior of label distributions is p(D) = ∏ni=1 Dir(di|1m), where m denotes the number of labels; the conditional distribution p(Z|c,D) is defined as:

where [µdici ; ζ -1 (σdici )] = f(di; θci ); f(di; θci ) is given by a neural network with parameters θci . If the simple labels are logical labels, p(Y|D) can be a Bernoulli distribution; if the simple labels are multi-label rankings, p(Y|D) can be a Plackett-Luce distribution [Plackett, 1975].

Variational Inference

Compared to JRC, LEIC additionally needs to infer the posterior of label distributions. We assume the variational posterior of D as q(D|X, Y) = ∏ni=1 Dir(di| ˆαi), where

In Equation (12), the first term (i.e., ˜Gi · ζ ◦ f(xi;φd)) aims to mine local label correlations. ζ ◦ f(·) denotes a neural network whose output is then transformed by the softplus function; ˜Gi is the symmetric Laplacian matrix of Gi:

where π denotes the softmax function, g is given by a graph convolutional network with adjacency matrix of A and learnable parameters of φc. ˜Gi ·ζ◦f(xi;φd) is essentially a graph convolution operation in which the message passing process works on the labels rather than on the instances. To facilitate understanding Equation (13), we consider the correlation between the i-th and j-th labels for instance xt (i.e., the (i, j) entry of Gt):

Equation (14) can be decomposed into the processes of obtaining cluster-specific label correlations and instancespecific label correlations: 1) We first use the linear kernel (i.e., ysiysj) to quantify the correlation between the ith and j-th labels, and then reweight instances by the cluster memberships to obtain the cluster-specific label correlations. 2) We then integrate cluster-specific label correlations by the cluster memberships of a specific instance to obtain the instance-specific label correlation. Note that a potential downside of graph convolutional networks is over-smooth [Li et al., 2019], which can easily derive a label distribution with close description degrees for all labels. Therefore, we additionally incorporate simple labels into the variational posterior, where λ controls the strength of simple labels.

In order to encourage the inference of c and D to influence each other through local label correlations, we assume that their variational posteriors share the parameter γ, i.e., q(c|X,A) = ∏ni=1Cat(ci|[γi1; · · · ; γik]). Finally, we give the factorization of variational posterior:

According to Equation (10) and Equation (15), we can rewrite the ELBO as:

Intuitively, the first term of Equation (16) is the same as in JRC, i.e., Equation (7); The second term of Equation (16) measures the likelihood of reconstructing the simple labels from the label distribution posteriors. This term involves the expectation w.r.t. the Dirichlet posterior q(D|X,Y) whose direct reparameterization is difficult unlike the Gaussian posterior [Kingma and Welling, 2014]. Therefore, we decompose the Dirichlet distribution into multiple Gamma distributions which can be reparameterized by their approximated inverse cumulative density function [Joo et al., 2020]:

where uij ∼ Uni(0, 1) is the (i, j) entry of U, ˆαij is the j-th element in ˆαi, Γ(·) is the Gamma function. Then, we have

Besides, the third and fourth terms of Equation (16) encourage the learned posteriors to incorporate the prior beliefs; the third term has a closed form similar to Equation (8). The forth term is the KL divergence between two Dirichlet distributions, which has a closed form. The fifth term promotes the label distribution to better generate the joint implicit representation, which can be estimated by the reparameterization shown in Equation (17):

Substituting these equations into Equation (16) yields the optimization objective of LEIC.

4 Experiments

4.1 Experimental Configurations

Datasets and evaluation metrics. Due to page limits, we select six representative real-world LDL datasets from different tasks respectively, and their brief descriptions are shown in Table 1. For Emotion6 and Twitter-LDL, we extract a 168-dimensional feature vector for each instance [Ren et al., 2019]. Besides, we use min-max normalization to preprocess the feature vectors for all datasets to accelerate the convergence. We use four commonly used LDL metrics to measure the similarity or distance between the estimated label distributions and the ground-truth. They are Chebyshev distance (Cheb), KL divergence (KL), cosine coefficient (Cosine), and intersection similarity (Intersec). The first two are distance metrics (i.e., the lower value indicates the better performance), and the last two are similarity metrics (i.e., the higher value indicates the better performance).

Comparison algorithms. We compare our algorithm with five LE algorithms, which are GLLE [Xu et al., 2021], LELR [Jia et al., 2023], VLEG [Xu et al., 2023], FLE [Wang et al., 2023], and GLEMR [Lu et al., 2023]. The hyperparameter ranges of GLLE are λ1 ∈ {10-4 , 10-3 , · · · , 103 } and λ2 ∈ {10-4 , 10-3 , · · · , 103 }; The hyperparameters for LELR, VLEG, FLE and GLEMR follow their respective literature. For our JRC and LEIC, k is set to m+1, the dimension of the joint implicit representation is set to 64, λ is selected from {1, 2, · · · , 10}, neural networks f are modeled as linear functions for simplicity, and Adam [Kingma and Ba, 2015] is adopted as the optimizer.

Experimental method. Recovery and predictive experiments [Xu et al., 2021] are the two basic methods we use to evaluate the performance of LE algorithms. Due to page limits, we only consider the case where the simple label is logical label in the experimental section. In the recovery experiment, we first reduce the ground-truth label distributions in the LDL dataset to logical labels, and then recover label distributions from these logical labels using LE algorithms, and finally compute the distances or similarities between the recovered label distributions and the ground-truth label distributions. In the predictive experiment, we first randomly dividing dataset (70% for training and 30% for testing), and then use LE algorithms to recover the label distributions of training instances which is then used to train an SABFGS [Geng, 2016] model, and finally we repeat the above proess ten times and report the mean predictive performance of SABFGS on test instances.

4.2 Empirical Validation of JRC

Joint Implicit Representations

Here we test the instance representations obtained by JRC.

1) On the one hand, the instance representations obtained by JRC can improve most existing LE algorithms. We run all comparison algorithms based on the original feature vectors and the instance representations obtained by JRC, respectively, and show the recovery performance and predictive performance in Table 3 and Figure 4, respectively. It can be seen that JRC significantly improves the performance of the LE algorithm in most cases. The average improvements in recovery performance ranking are 21.12% (for GLLE), 21.21% (for LELR), 21.01% (for VLEG), 14.01% (for FLE), and 21.95% (for GLEMR), respectively; the average improvements in predictive performance ranking are 15.96% (for GLLE), -1.33% (for LELR), 4.91% (for VLEG), 15.76% (for FLE), and 19.66% (for GLEMR), respectively.

2) On the other hand, the instance representations obtained by JRC can derive better instance relations. In Table 2, we compute the neighborhood divergences of different representations on multiple datasets to measure the quality (w.r.t. the label distribution) of the instance neighborhoods (or instance relations). The neighborhood divergence of a specific instance x is quantified as the mean of the KL divergences from the neighborhood instances to x itself w.r.t. the ground-truth label distribution. The "Original features" column and "JRC" column indicate that the instance neighborhoods are determined by the original feature vectors and the instance representations obtained by JRC, respectively. It can be found that JRC considerably reduces the neighborhood divergence; in other words, the JRC-based instance representations can derive better instance relations compared to original features.

Adaptive Clustering

Here we test the instance clustering obtained by JRC.

1) On the one hand, the instance clustering obtained by JRC can improve most existing LE algorithms that involve local label correlations. Since both GLLE and LELR involve local label correlations, we test the effectiveness of the JRCbased instance clusters for GLLE and LELR. Figure 3 shows the recovery performance variations arising from JRC-based instance clusters. It can be seen that for GLLE and LELR, the performance bars on KL are mostly lower than the horizontal dashed line (which corresponds to 0), and the performance bars on Cosine are mostly higher than the horizontal dashed line, which indicates that JRC-based clustering can improve the recovery performance of GLLE and LELR. Besides, we note that the performance bars of GLLE+ and LELR+ on KL and Cosine are mostly equal to the dashed line, which indicates that JRC-based clustering has almost no effect on GLLE+ and LELR+. This is because GLLE+ and LELR+ are built on top of the JRC-based representations which have clear clustering structures, so that traditional clustering methods also obtain similar results to the JRC-based clustering.

2) On the other hand, the instance clustering obtained by JRC has smaller intra-cluster divergence. Table 4 shows the average intra-cluster divergence (i.e., the average of KL divergences from the label distributions of all examples within a specific cluster to their center) of different clustering methods. It can be seen that the average intra-cluster divergence of JRC-based clustering is significantly smaller than others.

4.3 Empirical Validation of LEIC

Recovery and predictive experiments. Table 3 and Figure 4 show the recovery performance and predictive performance of LEIC, respectively. Figure 5 shows the statistical test results of a pairwise two-tailed t-test with 0.05 significance level. It can be seen that LEIC has competitive advantages both in recovering the label distributions and in serving for the subsequent LDL task.

Parameter analysis and ablation study. To observe the impact of the hyperparameter λ on the recovery performance, we take λ from 0 to 10, and the results are shown in Figure 6. It can be seen that an appropriate addition of λ (e.g., λ ∈ (0, 4)) improves the performance on all datasets, which indicates that the simple label information can alleviate the over-smoothing problem of GCN to some extent. To validate the module for mining the local label correlations, we replace Equation (12) with ˆαi = f([xi; yi];φd), and report the results in Table 5. It can be seen that using Equation (12) in LEIC has better KL performance and Cosine performance on all datasets than using ˆαi = f([xi; yi]; φd).

Recovery experiment on the image dataset. To test the performance of our models on non-tabular features, we perform recovery experiments on the original Emotion6 dataset. Specifically, we set the backbone neural networks in JRC and LEIC to ResNet-18 [He et al., 2016], and train comparison algorithms by JRC-based instance representations. The performance is shown in Table 6. Combined with Table 3, it can be seen that the performance on original Emotion6 is better than on the pre-processed tabular data.

5 Conclusion

In this paper, we propose a deep generative model called JRC which has following merits: 1) JRC facilitates the acquisition of more accurate instance relations and instance clustering, which can improve most existing LE algorithms; 2) JRC can handdle various forms of features, thus compensating for the shortcomings of the LE algorithms oriented to tabular features. Besides, we also propose a generative LE model called LEIC based on the JRC model. LEIC model mines instance relations and local label correlations by JRC and utilize both relations in a principled way to recover more accurate label distributions. Extensive experiments show that JRC model can improve the performance of most LE algorithms involving instance relations or local label correlations, and LEIC outperforms the state-of-the-art LE algorithms remarkably.

Acknowledgments

This work is partially supported by the National Natural Science Foundation of China (62176123, 61906090), the Found of Nanjing University of Aeronautics and Astronautics Research Base Innovation (Science and Technology) Project under Grant NJ2020022, and the Fund of Prospective Layout of Scientific Research for Nanjing University of Aeronautics and Astronautics.