Label Enhancement via Cross-View Fusion and Mixed Graph Propagation

1 Introduction

Label ambiguity has become an important challenge in machine learning [Liu et al., 2021], especially when dealing with complex real-world data that cannot be effectively represented by single or multiple labels. Traditional supervised learning methods, such as single-label learning (SLL) and multi-label learning (MLL) [Gibaja and Ventura, 2014; Moyano et al., 2019; Zhang et al., 2022], assign fixed labels to instances, but fail to capture the intricate relationships between labels, which are crucial in tasks like emotion recognition, medical diagnosis, and image captioning [Jia et al., 2019]. Label Distribution Learning (LDL) addresses this by assigning a distribution of labels to each instance, enabling a more detailed representation of the data [Geng, 2016].

∗ First corresponding author

†Second corresponding author

LDL requires instances with exact label distributions for the training process. However, obtaining label distributions is often costly and impractical, especially when dealing with large datasets, as most real-world datasets are labeled with logical labels [Xu et al., 2019]. To solve this problem, label enhancement (LE) emerged as a promising solution. LE recovers label distributions exactly from existing logical labels. This process is vital for enabling LDL in practical applications, providing the necessary supervision for training models that can handle label distributions.

Despite the promising performance of previous LE methods, they still exhibit several limitations. First, most existing LE methods treat logical labels and features independently, failing to effectively integrate their complementary information. They overlook the fact that features and categories provide different perspectives on the same instance. Specifically, the more correlated two labels are, the closer the corresponding description degrees should be. Second, categories within the label typically exhibit semantic or co-occurrence-based relationships, which can provide valuable additional information for recovering label distributions. Specifically, the higher the correlation between two categories, the closer their corresponding description degrees should be. However, many methods fail to fully consider the interdependencies between categories during label distribution recovery, resulting in suboptimal accuracy in the recovered distributions.

To address the aforementioned challenges, we propose a novel label enhancement method-CVMG (Cross-View Fusion and Mixed Graph-based Label Enhancement). This method integrates logical labels and feature information using a cross-attention mechanism, which generates high-level feature representations. By combining complementary information from both perspectives, CVMG enhances feature representations, thereby improving the accuracy of label distribution recovery. Furthermore, CVMG constructs a mixed dependency graph that captures both instance-level and category-level dependencies. This graph structure enables efficient propagation of label information across instances and categories, addressing the issue of inadequate consideration of category dependencies in existing methods.

Our contributions are summarized as follows:

• We propose a novel label enhancement method, termed

CVMG (Cross-View Fusion and Mixed Graph-based Label Enhancement), which leverages a cross-attention mechanism to fuse logical labels and feature information from two complementary views, thereby generating high-level feature representations.

• CVMG further constructs a mixed dependency graph that integrates instance-level dependencies induced by high-level features with category-level dependencies mined from logical labels, and performs label propagation on this mixed graph to improve the accuracy of label distribution recovery.
• Extensive experiments conducted on 13 benchmark datasets demonstrate the effectiveness and superiority of CVMG over existing state-of-the-art label enhancement methods.

2 RelatedWork

2.1 Label Enhancement

Label Enhancement is used to recover label distributions from logical labels, thereby facilitating more accurate Label Distribution Learning. In recent years, many LE algorithms have been proposed to address the problem of label distribution recovery. Fuzzy-based LE methods, such as fuzzy clusteringbased LE [El Gayar et al., 2006] and kernel-based LE [Jiang et al., 2006], handle label uncertainty by constructing fuzzy membership degrees for each label class. However, it remains unclear how the fuzziness introduced by these methods can effectively enhance logical labels into accurate label distributions. Graph-based LE methods leverage the topological structure of graphs to capture relationships between instances and propagate label information. Label Propagation (LP)- based LE [Li et al., 2015] and Manifold Learning (ML)-based LE [Hou et al., 2016] model instance similarities through graph structures and propagate label information across similar instances, thereby enhancing logical labels and generating label distributions. Graph Laplacian-based LE (GLLE) [Xu et al., 2019] further improves label distribution recovery by considering both instance-level and category-level dependencies but fails to fully exploit the category-related information embedded in logical labels. Label Enhancement with Sample Correlations (LESC) [Tang et al., 2020] considers the global graph information based on the low-rank representation (LRR) assumption.

Recent research has shifted towards more advanced paradigms, aiming to model the complex relationship between features and logical labels more effectively. PLEML [Zhu et al., 2021] generates privileged information via multilabel learning to recover label distributions more effectively. LEVI [Xu et al., 2020] models label distributions as latent vectors and infers them through variational approximation. ConLE [Wang et al., 2023] aligns features and logical labels in a shared embedding space using contrastive learning. LIB [Zheng et al., 2023] further improves recovery by exploiting both label assignment and label gap information. In semisupervised settings, SMLE [Ye et al., 2025] refines label distributions for labeled and unlabeled data via kNN aggregation and label propagation, while leveraging local label correlations. Similarly, FLEM [Zhao et al., 2025] jointly optimizes LE and multi-label learning in an end-to-end manner, reducingmismatches withmodel training. Together, thesemethods strengthen the connection between features and logical labels to advance label distribution recovery. While these methods have advanced the field of label distribution enhancement, they heavily depend on feature correlations and overlook the richness of information inherent in the logical labels themselves.

2.2 AttentionMechanism

The attention mechanism [Vaswani et al., 2017] has become a central technique in machine learning, enabling models to focus on relevant parts of the input and thus improve overall performance. Self-attention-based models like the Transformer excel in capturing long-range dependencies [Li et al., 2024], making them highly effective for sequence modeling. Given its powerful representation capabilities, the attention mechanism has also been widely applied to multi-label classification tasks [Lanchantin et al., 2021; Dosovitskiy et al., 2020]. [Hua et al., 2020] proposed an attentional region extraction network to localize and re-coordinate attentional regions for consequent label inference. [Tan et al., 2022] introduced a Transformer-driven semantic relation inference method using class activation mapping to locate regions of interest. Recent advancements in cross-attention include multimodal feature fusion, such as in the CAGRN method, where label embeddings are used as queries to interact with feature representations, thus explicitly modeling feature-label dependencies and enabling category-level relational modeling [Bi et al., 2024]. Similarly, CAFF-DINO demonstrates the use of cross-attention in multi-spectral object detection, where it aligns and fuses features from different modalities using cross-attention, showing its utility in complementary information fusion [Helvig et al., 2024]. Building on these ideas, we introduce a cross-attention mechanism for Label Enhancement (LE), integrating complementary information from logical labels and features, optimizing feature representation. This approach allows for the optimization of feature representations, thereby laying the foundation for more accurate label distribution recovery.

3 Methodology

3.1 Formulation of Label Enhancement

Let X = [x1, x2, · · · , xn ] ∈ Rq×n be the feature set of instances, where n is the number of instances and q is the feature dimension. The logical label vector of instance xi is given by yi = (yi,1, yi,2, · · · , yi,c)T ∈ {0, 1}c , where c is the number of labels. If the label yi,k is relevant to xi, then yi,k = 1; otherwise yi,k = 0. Let dyx denote the description degree of label y to feature x. Then di = (dyi,1 xi , dyi,2 xi , · · · , dyi,c xi )T represents the ground-truth label distribution of xi. Given a training set S = {(xi, yi), 1 ≤ i ≤ n}.

LE recovers the label distribution ̂ di of xi from the logical label yi, converting S to the label distribution dataset ̂D = {(xi, ̂ di ), 1 ≤ i ≤ n}, satisfying ̂ dyi,j xi ∈ [0, 1] and ∑cj=1 ̂ dyi,j xi = 1, while making the prediction label distribution dataset ̂D as close as possible to the (unknown) true label distribution dataset D = {(xi, di)|1 ≤ i ≤ n}.

The proposed CVMG is illustrated in Figure 1.

3.2 Cross-View Fusion via AttentionMechanism

Logical labels and features essentially describe the same instance from complementary perspectives. Logical labels provide significant information about the relevance of the label to the instance, while features offer a comprehensive description of the instance. In label enhancement, by integrating information from both perspectives, the unique characteristics of each can be better utilized, leading to more accurate label distributions. Therefore, we introduce a cross-attention mechanism to fuse their complementary information.

The core idea behind attention mechanisms in deep learning is to dynamically assign different weights to different parts of the input, allowing the model to determine which parts are most crucial. In our model, the cross-attention mechanism is used to combine the distinct perspectives of logical labels and features. Specifically, the features are treated as queries, while the logical labels are used to generate keys and values:

where xi and yi represent the original feature and logical label of each instance, whileW Q, W K , andW V are the trainable weight matrices. hi is the high-level feature representation generated by the cross-attention mechanism through the calculation of the correlation between features and labels, leading to the final high-level feature representation.

We train the cross-attention module using contrastive loss. Contrastive loss works by maximizing the similarity between samples of the same class and minimizing the similarity between samples of different classes, thereby encouraging the cross-attention module to learn more discriminative feature representations. We calculate the Jaccard similarity between the logical labels and set a threshold to determine whether a pair of samples belongs to the same class. The construction of the contrastive loss is as follows:

where n denotes the number of instances, P(i) is the set of positive samples for hi, and A(i) is the set of all contrastive samples for hi, which includes both positive samples and negative samples from different classes. τ is the temperature parameter to control the softness.

The cross-attention module minimizes the contrastive loss to extract information from the original features that aligns with the logical labels. In this way, a high-level feature representation is generated for each instance, incorporating both logical label and feature information, enabling a more precise description of the instance. The resulting high-level features are then used for subsequent label enhancement tasks, further improving the model's performance.

3.3 Label Propagation Based onMixed Dependency Graph

In the previous section, we introduced how to obtain the highlevel feature representation H = [h1, h2, · · · , hn ] ∈ Rq×n through the attention module. In this section, our goal is to recover the label distribution using the high-level features and logical labels. Two matrices can be constructed: the feature matrix H and the logical label matrix Y = [y1, y2, . . . , yn] ∈ {0, 1}c×n . Using these two matrices as inputs, our objective is to obtain the label distribution matrix D = [d1, d2, . . . , dn] ∈ [0, 1]c×n .

In the label enhancement task, the matrix D should satisfy the following two criteria:

Instance-level label distribution similarity assumption: Specifically, if two instances have similar feature representations, their corresponding label distributions (i.e., the i-th and j-th columns of D) should also be similar. This assumption is derived from the smoothness assumption.

Category-level label distribution similarity assumption: Specifically, if two categories are highly correlated, such as having high co-occurrence rates, they are likely to co-occur in many instances. This means that the corresponding row vectors in D should be similar.

We model the constraint information in these criteria by constructing a mixed dependency graph and performing label propagation (LP) on this graph. In the following, we will provide a detailed explanation of how these criteria are integrated into a unified optimization framework.

Instance-level Label Distribution Dependency

At the instance level, we model label distribution dependencies by exploiting the similarity among instances in the highlevel feature space. Under the smoothness assumption, the manifold structure of the feature space is expected to be preserved in the label space, such that instances with similar features tend to exhibit similar label distributions.

Traditional Label Propagation-based LE methods construct fully connected graphs, allowing labels to propagate among all instances. However, instances with low feature similarity often lack meaningful semantic relationships, and such weak connections may introduce noise. Therefore, we sparsify the instance-level similarity matrix WH by preserving only truly similar instance connections, improving both robustness and computational efficiency. In this sparsification strategy, if two instances i and j have k-nearest neighbors, their similarity value (wH)ij is computed based on their Gaussian RBF kernel similarity as follows, with NNkH referring to the k nearest neighbor search within nodes of the high-level feature set.

where σ > 0 is the width parameter for similarity calculation which is fixed to be 1 in this paper.

Category-level Label Distribution Dependency

At the category level, we focus on the correlation between categories. Specifically, the higher the correlation between two categories, the closer their semantic or categorical descriptions, and their corresponding label distributions should exhibit higher similarity. For instance, in facial expression datasets, happiness and surprise often appear together on the same face, especially when a person is extremely surprised, typically accompanied by a smile. This category-level similarity can be mined from known logical labels and serves as an effective guide in the label propagation process.

In category-level similarity modeling, we use Jaccard similarity to measure the correlation between categories. We first compute the intersection matrix I = (iij) ∈ Nc×c and the union matrix U = (uij) ∈ Nc×c from the label matrix Y , where I represents the co-occurrence of categories and U represents their total occurrence, as defined by the following formulas, where ui denotes the total count of category i across all instances.

Based on these matrices, the Jaccard similarity matrix J = (jij) ∈ Rc×c for each pair of categories is computed as:

Considering that the similarity between categories may be influenced by noise, especially when the correlation between categories is weak, we retain only the TopK most relevant category pairs. This allows us to focus on the most semantically related pairs, thereby avoiding interference from weakly correlated category pairs. This approach not only reduces the impact of noise but also lowers computational complexity. The final category-level correlation matrixWY is shown below:

LE Using Mixed Dependency Graph

By combining the instance-level similarity matrix WH and the category-level correlation matrix WY , we construct a mixed dependency graph that incorporates both instancelevel and category-level similarities. Label propagation is then performed on this mixed dependency graph. During the label propagation process, we construct the label propagation matrix PH based on WH: PH = (∆H)-1/2WH(∆H)-1/2 . Here, ∆H = diag(δH(1), . . . , δH(n)) is a diagonal matrix, with each diagonal element di representing the sum of the elements in the i-th row of matrix WH, specifically: δH(1) = ∑nj=1 (wH)ij. Similarly, the label propagation matrix PY constructed based on the category-level correlation matrix WY , is given by PY = (∆Y )-1/2WY (∆Y )-1/2 , where ∆Y = diag(δY (1), . . . , δY (c)).

Let F = (fi,j) ∈ Rc×n be a non-negative matrix, where each element fi,j ≥ 0 is proportional to the importance of the j-th label in the i-th instance. Based on the logical labels, the initial matrix F (0) is instantiated as the normalized logical label matrix Y , which serves as the initial "soft labels". The label propagation update rule combines both instance-level and label-level propagation. Specifically, in the t-th iteration, the matrix F is updated using the label propagation matrices PH and PY as follows:

where α and β are hyperparameters that control the labellevel and instance-level propagation. For stability, weights are normalized to form a convex combination. After completing the iterative propagation, the matrix F ⋆ is obtained, and each column of F ⋆ is normalized to estimate the label distribution, as follows:

4 Experiments

4.1 Datasets

We evaluate our method using 13 real-world multi-label datasets, covering a variety of domains. Facial expression datasets SJAFFE and SBU-3DFE rate images on six basic emotions using a 5-level scale. The Natural Scene dataset consists of 2,000 images of natural scenes, and the Movie dataset contains ratings for 7,755 movies, collected from 478,656 unique users. Additionally, the Yeast datasets (from Yeast-alpha to Yeast-spoem) include data from biological experiments involving gene expression levels of budding yeast at different time points. A summary of these datasets is provided in Table 1.

All datasets offer ground-truth label distributions, enabling label enhancement evaluations. For consistency with previous studies, we apply a binarization strategy to these datasets to ensure uniformity in the evaluation process.

4.2 EvaluationMetrics

We evaluate the recovery performance using six widely used metrics: Chebyshev distance (Cheb), Clark distance (Clark), Canberra metric (Canber), Kullback-Leibler divergence (KL), Cosine coefficient (Cosine) and Intersection similarity (Intersec). The first four are distance metrics, where smaller values indicate better performance, while the last two are similarity metrics, where larger values are preferred. The formulas for these metrics are summarized in Table 4.

4.3 AlgorithmComparison

We compare CVMG with six state-of-the-art algorithms in the field of label enhancement: LP [Li et al., 2015], GLLE [Xu et al., 2019], LESC [Tang et al., 2020], LIB [Zheng et al., 2023], LEVI [Xu et al., 2020], ConLE [Wang et al., 2023].

• LP adapts semi-supervised label propagation to LE, employing graph models to construct the label propagation matrix and generate label distributions.
• GLLE enhances label distributions in the feature space guided by the topological information.
• LESC uses low-rank representation to capture global sample relationships, improving label distribution recovery by predicting implicit correlations.
• LIB infers label distributions from logical labels using variational inference, focusing on relevant label information and ignoring irrelevant data.
• LEVI recovers label distributions through variational inference, incorporating multi-output regression for more accurate label prediction.
• ConLE integrates features and logical labels into a unified space via contrastive learning, generating high-level features for label distribution recovery.

4.4 Recovery Performance

Implementation Details. In the experiments, the crossattention fusion module uses 8 attention heads to generate enhanced features, with batch size 256 and 50 epochs. Positive pairs are formed by Jaccard similarity (threshold 0.3) on logical labels. The contrastive learning temperature coefficient is set to 0.1, and the optimizer learning rate is 1 × 10-3 . Label propagation is performed on the enhanced features: the instance-level KNN graph connects each instance to its 20 nearest neighbors, and the category-level top-K graph connects each category to two-thirds of the total categories. During label propagation, α is selected from [0.8, 0.85, . . . , 1.0], and β is selected from [0.1, 0.12, . . . , 0.2]. To ensure stability and convergence, when the sum of α and β exceeds 1, they are scaled to slightly below 1, maintaining positive weight for the initial label term and enabling shrinkage updates. The iterative propagation converges and terminates when ∥F (t+1) - F (t) ∥∞ < 10-5 , where F (t) denotes the predicted label distribution matrix at iteration t. For baseline methods, results are generated using their original settings.

Experimental Results. We evaluate our method on 13 realworld datasets and compare it with six benchmark LE algorithms, with detailed results in Table 2. The best performance for each dataset is highlighted in bold, and the last row shows the average ranking across all datasets. Our method consistently outperforms the benchmarks, achieving rankings of 1.38, 1.00, 1.08, 1.27, 1.35, and 1.00 across six metrics, demonstrating superior label distribution recovery. Moreover, the results are largely insensitive to the choice of trade-off hyperparameters, indicating stable performance without precise fine-tuning.

Ablation Studies. The proposed method consists of two main components: cross-view fusion via an attention mechanism and label propagation based on a mixed dependency graph. These two components are designed to address complementary aspects of the label enhancement problem. To better understand the contribution of each component and their respective roles in the overall framework, we conducted an ablation study by selectively removing or modifying individual modules.

Therefore, we first removed the part of CVMG responsible for cross-view fusion, resulting in a comparative algorithm MGLE. Secondly, we removed the label propagation based on the mixed dependency graph and reduced it to label propagation on a fully connected graph based solely on feature similarity, resulting in the algorithm CVLP.

The results are shown in Table IV, comparing the complete CVMG framework (top row) with two ablation versions: MGLE (middle row) and CVLP (bottom row). This behavior is expected, as label enhancement is inherently influenced by both relationships among instances and dependencies among categories, and removing either component inevitably weakens the model's ability to recover accurate label distributions. The results show that removing the crossattention feature generation leads to a significant performance drop. This degradation can be attributed to the absence of high-level features, causing the model to struggle with capturing the complex relationships between logical labels and features. High-level features play a crucial role in improving the accuracy of label distribution recovery. Similarly, removing the mixed dependency graph-based label propagation causes a sharp decline in the efficiency of label information propagation. Without this mechanism, label propagation becomes limited and fails to fully exploit the relationships between instances and categories. For datasets with very few labels (e.g., Yeast-spoem, c = 2), the category-level Top-K graph is simple and contributes modestly, yet still provides complementary guidance as a lightweight correlation regularizer, while the instance-level KNN graph dominates propagation. Overall, the ablation study highlights the critical role of cross-view fusion via attention mechanism and mixed dependency graph-based label propagation in label enhancement.

5 Conclusion

In this work, we propose CVMG (Cross-View Fusion and Mixed Graph-based Label Enhancement), a novel method for the label enhancement problem. By integrating logical labels and features through a cross-attention mechanism, CVMG is able to capture complementary information from different views and generate more informative high-level features. Based on these representations, the proposed method further constructs a mixed dependency graph that jointly models instance-level similarities and category-level dependencies, enabling effective label propagation. Experimental results on 13 real-world datasets show that CVMG consistently outperforms existing label enhancement methods.

Acknowledgments

This work was supported by the National Natural Science Foundation of China under Grants 62476135, 62406145, and the Natural Science Foundation of the Higher Education Institutions of Jiangsu Province under Grant 24KJB520015.