Unreliable Partial Label Learning with Recursive Separation

1 Introduction

Partial label learning (PLL) is a typical weakly supervised learning problem where the candidate label set is given for each instance but among which only one is true. Compared with the ordinary supervised learning problem where each instance is associated with only one ground-truth label, partial label learning induces predictive model from ambiguous labels, hence considerably reduces the cost of data annotations. Nowadays, PLL has been extensively employed in the field of web mining [Luo and Orabona, 2010], multimedia content analysis [Zeng et al., 2013], automatic image annotations [Chen et al., 2018], ecoinformatics [Liu and Dietterich, 2012; Tang and Zhang, 2017], etc.

∗Corresponding authors.

A variety of methods have been proposed for addressing the PLL problem. The most common strategy to learn from partial labels is disambiguation, where Identification-Based Strategy (IBS) and Average-Based Strategy (ABS) are two main disambiguation strategies. For IBS, iterative optimization is employed to predict true label treated as latent variable. While ABS treats all labels in the candidate label set in an equal manner where probabilities of modeling outputs are averaged to get the final prediction. It memorizes all candidate labels, since it avoids identifying the latent ground truth label. Recently, deep neural network based IBS method have achieved promising performance on PLL. PiCO [Wang et al., 2022] achieves a significant improvement in performance by adopting contrastive learning strategy in PLL, which is able to learn high quality representation. CR-DPLL [Wu et al., 2022] is a novel consistency regularization method which achieved state-of-the-art performance on PLL, almost nearing supervised learning. Even so, whether IBS or ABS, both assumed that the true label is present within the candidate label set.

However, the assumption that the true labels are consistently present within the candidate label sets would be unrealistic. In existing PLL setting, the annotation for each instance is the partial labels (i.e. candidate label set) rather than the true label directly, thus significantly reduces the difficulty and cost. Against this backdrop, it poses a challenge for annotators to ensure that the true labels are present within the candidate label sets. Therefore, Unreliable Partial Label Learning (UPLL) [Lv et al., 2023] is proposed in response, which is a more general problem than existing PLL. In UPLL, it is acknowledged that the true label may not be present within the candidate label set for each instance. Since that, it significantly reduces the difficulty and cost associated with data annotation. Above all, UPLL addresses the issue of labeling instances that are difficult to distinguish. Hence, UPLL could be deemed a more prevalent and valuable problem.

Despite existing PLL methods achieved promising performance, suffering from unreliable partial labeling, current PLL methods encounter numerous challenges when applied to UPLL. It will exhibit a significant decline in performance on UPLL datasets, particularly for high unreliable rates. RABS

[Lv et al., 2023] has demonstrated that bounded loss functions have the ability to fit the ground-truth label against the interference of unreliability and other candidates. However, it would fail on high unreliable levels or high partial levels. This urges us to design an efficient method to manipulate high unreliability problem.

Motivated by this consideration, a framework named Unreliable Partial Label Learning with Recursive Separation (UPLLRS) is devised to issue this puzzle. In this paper, a novel separation method named Recursive Separation (RS) is proposed for known unreliable rate scenes to separate unreliable samples and reliable samples. However, it is limited in the real world because the real unreliable rate is difficult to know. In order to tackle this problem, more generally, we design a self-adaptive strategy for RS algorithm which could fit unknown unreliable rate. Pilot experiments have demonstrated the effectiveness of self-adaptable RS algorithm. After that, we combine a label disambiguation strategy with semisupervised learning techniques in the second stage of UPLLRS. Experiments show that our method achieve state-ofthe-art results on the UPLL datasets. Our contributions can be summarized as follows:

• A self-adaptive recursive separation algorithm is proposed for effectively separating raw dataset into a reliable subset and an unreliable subset.
• A two-stage framework is proposed for inducing the predictive model, based on the self-adaptive RS strategy. Upon obtaining both the reliable subset and unreliable subset, the disambiguation strategy utilizes the reliable subset for learning while incorporating information from the unreliable subset by the semi-supervised technique.
• The UPLLRS framework is versatile, capable of handling both image and non-image datasets. Utilizing data augmentation techniques on image datasets, the performance will be further enhanced.

The rest of this paper is organized as follows. First, we briefly review related works on partial label learning. Second, the details of the proposed UPLLRS are introduced. Third, we present the results of the comparative experiments, followed by the final conclusion.

2 RelatedWork

Partial label learning deals with the problem that the true label of each instance resides in the candidate label set. Many algorithms have been proposed to tackle this problem, with existing PLL methods broadly classified into classical and deep learning approaches.

In classical PLL, label disambiguation is based on averaging or identification. In averaging-based methods, the candidate label set and non-candidate label set are treated the same[H¨ullermeier and Beringer, 2006; Cour et al., 2011; Zhang and Yu, 2015]. For example, [Cour et al., 2011] discriminated candidate labels and non-candidate labels with a convex loss. But identification-based methods progressively refine labels in the candidate set during the model training[Chen et al., 2013; Yu and Zhang, 2016]. [Yu and Zhang, 2016] optimized the constraint on the maximum margin between maximum modeling output of candidate labels and that of other labels.

However, the model output of averaging-based methods often overwhelms the true label, resulting in low accuracy. As a result, many identification-based algorithms have been devised in recent years[Feng and An, 2019; Gong et al., 2017; Lyu et al., 2019; Tang and Zhang, 2017; Xu et al., 2019]. Nevertheless, these classical methods often have a bottleneck due to the restriction of the linear model.

Given the success of deep neural network-based methods in classification tasks, a proliferation of PLL approaches incorporating deep neural networks have emerged. [Yao et al., 2020a] designed two regularization techniques in the training with ResNet representing the first exploration of deep PLL. [Yao et al., 2020b], referring to the idea of co-training, trained two networks to interact with each other for label disambiguation. Concurrently, a progressive method proposed by [Lv et al., 2020] progressively identified true label adopting the memorization effect of deep network. [Feng et al., 2020] formalized the partial label generation process and proposed two provably consistent algorithms, risk consistent (RC) classifier and classification consistent (CC) classifier. Then, a leveraged weighted loss, which balances the contributions of candidate labels and non-candidate labels, was proposed by [Wen et al., 2021]. With the development of contrastive learning, [Wang et al., 2022] applied contrastive learning to PLL for effective feature representation. Recently, [Wu et al., 2022] designed a consistency regularization framework in deep PLL, which gives very small performance drop compared with fully supervised learning.

However, it is common for false positive labels to be inadvertently chosen from label set, rather than being selected randomly. More specifically, it is acknowledged that each instance may not possess a uniform prior label distribution, but rather a latent label distribution which encompasses vital labeling information. Thus, [Xu et al., 2021b] proposed an instance-dependent approach named VALEN, which aims to recover the latent label distribution via label enhancement [Xu et al., 2023; Xu et al., 2021a], leveraging it to further improve performance in real-world settings. VALEN first generates a label distribution through label enhancement, then utilizes variational inference to approximate that distribution.

In practice, despite the demonstrated empirical success of the aforementioned algorithms in the PLL task, their effectiveness is limited when the ground-truth label may not be present within the candidate label set. Therefore, the unreliable partial label learning is proposed in [Lv et al., 2023], which is more general in comparison to current PLL. Furthermore, [Lv et al., 2023] has proved that bounded loss, such as the Mean Absolute Error (MAE) loss and the Generalized Cross Entropy (GCE) loss, is robust against unreliability. However, the performance of RABS remains limited in the presence of high levels of unreliability.

3 Preliminaries

Let X and Y be feature space and label space respectively, and p(x, y) be the distribution on X × Y. Moreover, D =

{(xi, yi)}ni=1 is the training set in which xi is i-th instance and yi is the corresponding ground-truth label, and V = {(xi, yi)}ki=1 is the validation set including k pairs of instance xi with ground-truth label yi.

In PLL problem, the distribution p(x, y) is corrupted to p(x, s) in which s is candidate label set satisfied p(yi ∈ si) = 1, ∀yi ∈ Y, and the training set is corrupted to ¯D = {(xi, si)}ni=1 . The goal of PLL task is to induce a classifier from ambiguous dataset ¯D.

However, in UPLL, µ called unreliable rate is the probability of true label yi not in the candidate label set si, it can be expressed formally as:

Such that the candidate label set si is corrupted to ˜ s that is the unreliable candidate label set. Then, the UPLL training set can be denoted as ˜D = {(xi, ˜ si)}ni=1 .

4 ProposedMethod

In this section, we firstly propose the self-adaptive Recursive Separation (RS) algorithm which aims to differentiate reliable samples and unreliable samples effectively. After that, numerous pilot experiments were conducted which demonstrate that self-adaptive RS algorithm is effective. Finally, a framework entitled Unreliable Partial Label Learning with Recursive Separation (UPLLRS) is proposed. There are two key stages in this framework. At the beginning, the self-adaptive Recursive Separation (RS) effectively split training dataset into a reliable subset and an unreliable subset. Subsequently, in order to induce a predictive model, a disambiguation strategy is employed to progressively identify ground-truth labels while utilizing semi-supervised learning techniques in combination.

4.1 Recursive Separation

The memorization effect can be interpreted as the deep network firstly fit correct labels and then gradually fit wrong labels through the learning phase [Bai et al., 2021]. In recent years, small loss trick has been demonstrated to be an effective method for addressing label noise. That inspires us to identify the reliability of samples and pay more attention to the reliable samples. Furthermore, it is discovered that the top-10% large loss samples contain more unreliable samples than any other parts after a few epochs of training, as Figure 1 shows. This motivated us to progressively take unreliable partial samples away from the training set by iteratively excluding top-γ large loss samples. Based on this idea, we introduce a multi-class classifier f(·; θ) with parameters θ. More specifically, the training phase of recursive separation task is optimizing the following classical multi-class classification objective function:

where LRS is the loss function for the Recursive Separation (RS) stage. According to [Liu et al., 2020; Bai et al., 2021], in the early-learning stage, the gradient direction of crossentropy loss is close to the correct optimization direction. It

Algorithm 1 Self-adaptive RS Algorithm

Input: Separation network f(·; θ) with trainable parameters θ; Unreliable partial label training set ˜D = {(xi, ˜ si)}ni=1 and validation set V = {(xi, yi)}ki=1 ; Small epochs β for each separation step; Separation rate γ; RS patience ϕ and max separation step λ.

Output: Reliable subset ˜DλR = {(xi, ˜ si)}mi=1 and unreliable

subset ˜DλU = {(xi)}n-m i=1 . 1: Let ϕcurr ← 0 and AccV ← 0; 2: for i ← 1 to λ do 3: Randomly initialize θi0 ; 4: for j ← 1 to β do 5: Train f(·; θij-1 ) using dataset ˜DiR; 6: Calculate loss l according Eq. 3; 7: Update parameters from θij-1 to θij ; 8: if j = β then 9: Sort l by value in descending order; 10: Exclude top-γ instances from ˜DiR and add excluded instances to ˜DiU without labels; 11: end if 12: end for 13: Evaluate f(·; θij ) on dataset V and calculate accuracy Acccurr ; 14: if Acccurr < AccV then 15: ϕcurr ←ϕcurr +1; 16: if ϕcurr ≥ ϕ then 17: break; 18: end if 19: else 20: AccV ←Acccurr, ϕcurr ← 0; 21: end if 22: end for 23: return Reliable subset ˜DλR and unreliable subset ˜DλU .

inspires us to choose Categorical Cross Entropy (CCE) [Lv et al., 2023] loss as LRS under the UPLL setting, such that the objective function can be rewritten as:

Following [Lv et al., 2023], if the dataset has C classes, the pj(f(xi; θ)) can be specified as:

where f(·)j is the output for j-th class. More specifically, pj (f (·)) is the j-th class probability of classifier f (·)'s output. Since the classifier will fit more unreliable labels after the early learning stage, the classifier f(·; θ) should only be trained for several epochs β. Formally speaking, let ˜DiR be the reliable subset of i-th separation step. At the very beginning, set ˜D0R = ˜D. Then let ˜DU = {xi}mi=1 denotes instances excluded from the reliable subset. θij is the parameters for i-th step separation's j-th training epoch. Note that θi0

denotes the randomly initialized parameters in i-th separation step.

The RS algorithm can be described as follows: for the ith step, the parameters θi0 are randomly initialized. Then we train f(·; θ) for several epochs β, get parameters θiβ . After that, we retrieve the final epoch (i.e. β-th epoch) training losses for each sample and sorted them by loss value in descending order. Simultaneously, instances that are correlated with the top-γ (0 < γ < 1) maximum loss values are shifted to the ˜DiU . Following λ steps of separation, we will acquire a reliable subset ˜DλR and an unreliable subset ˜DλU .

If the unreliable rate µ on dataset is known, the λ can be estimated by µ directly. However, it is generally not feasible in real-world settings. Given this reality, a self-adaptive strategy has been devised to accommodate various levels of unreliability. Intuitively, as the count of unreliable samples descending in ˜DiR, the accuracy on validation or test set will increase. But at the latter phase, most of the unreliable samples have been removed and samples in ˜DiR are totally reliable. As the value of | ˜DλR| goes down, the accuracy on the validation or test set diminishes. Given the need for addressing the limitations of unknown µ, we propose a self-adaptive RS algorithm that incorporates an early-stopping technique to terminate the process of separation at an appropriate time. The self-adaptive strategy dictates that, should the accuracy on the validation set cease to improve over ϕ consecutive epochs, the separation process will be terminated. This leads to the final determination of ˜DλR and ˜DλU , with ϕ representing the separation patience. The details of self-adaptive RS algorithm is exhibited in Algorithm 1.

4.2 Pilot Experiments

In order to validate the efficacy of the self-adaptive RS method put forth, a series of pilot experiments were conducted to assess the ability of the method to identify and exclude unreliable samples, resulting in the formation of a reliable subset.

To explore this idea, we generate UPLL dataset on CIFAR10 [Krizhevsky et al., 2009] at first with four different unreliable rates 0.1, 0.2, 0.3 and 0.4. Then annotate reliable or unreliable for each sample by the true label in or not in the candidate label set respectively. The partial rate is fixed as 0.1 in the pilot experiments. MultiLayer Perceptron (MLP) is used as backbone since complex networks will overfit unreliable samples faster than plain networks. The Categorical Cross Entropy (CCE) loss [Lv et al., 2023] is utilized to train the classifier. More generation process of the dataset see the section 5 below for a detailed description.

At first, we train the network for 5 epochs. In the 5-th epoch, the samples in training set are sorted by loss value in descending order. The experimental results are reported in Figure 1. The orange bars represent the number of unreliable samples and the blue bars represent the number of reliable samples. The 40K samples are divided into 10 sections and each section contains 4K samples. As is shown, the unreliable samples in the first section occupy a larger proportion than the reliable samples. But in the last section, it's exactly the opposite of that.

It is concluded that the section with higher loss value will contain more unreliable samples than the one with lower loss value. Motivated by this finding, we try to exclude top-3% samples every 5-epoch as a separation step with four different unreliable rates {0.1, 0.2, 0.3, 0.4}. Then record the variation of real reliable rate on the training set. As shown in Figure 2, the proportion of reliable samples increases as the number of separation steps increases. That is to say, our self-adaptive RS method can effectively exclude unreliable samples and then get a highly reliable subset. With relatively low unreliability, the model is able to achieve a higher level of accuracy. In the following, a framework is designed to learn from these two subsets.

Algorithm 2 UPLLRS Algorithm with General Solution

Input: Network g(·; ω) with parameters ω. Unreliable partial label dataset ˜D and validation set V . Max training epochs T . Output: Parameters ω for g(·).

• 1: Obtain reliable subset ˜DλR and unreliable partial dataset ˜DλU by executing Algorithm 1;
• 2: Randomly initialize ω.
• 3: for i ← 1 to T do
• 4: Train g(·; ω) from ˜DλR;
• 5: Calculate loss according Eq. 6;
• 6: Update wij according Eq. 7;
• 7: Use g(·; ω) and get pseudo labels Rwhich over threshold τ on dataset ˜DλU ;
• 8: Add pseudo labels R and corresponding instances to reliable dataset ˜DλR and remove it from ˜DλU ;

• 9: end for
• 10: return ω.

4.3 The Overall Framework

The overall framework of UPLLRS consists of two stages. Firstly, the self-adaptive RS component recursively separates ˜D, obtaining a reliable subset ˜DλR and an unreliable subset ˜DλU as a result. Then, the disambiguation strategy, in conjunction with a semi-supervised learning approach, induces the model from both ˜DλR and ˜DλU . Ultimately, we arrive at a well-trained classifier g(·; Θ).

General Solution

After the recursive separation stage, the dataset is split into the reliable subset ˜DλR and unreliable subset ˜DλU . As the unreliable rate µ goes up, ˜DR will remain fewer and fewer samples while ˜DU will collect more and more unreliable samples. In order to fully leverage the information contained within the ˜DU, we employ pseudo-labeling technique for instances in ˜DλU . After the completion of each epoch, the model is evaluated on ˜DλU and high-confidence samples are added to ˜DλR. More specifically, the pseudo label for each instance xi is given by:

where pg denotes the model's predicted class distribution. We only retain the pseudo labels which satisfy max(pg(xi)) ≥ τ, where τ is a threshold. It is fixed as 0.95 in our experiments.

Although the ˜DλR is reliable, the labels therein are ambiguous. Adopting the disambiguation method PRODEN [Lv et al., 2020], the weighted loss can be written as:

in which wij is the confidence of the j-th class being consistent with the concealed true class for the i-th instance. It is estimated by the output of classifier g(·; ω), which is defined as:

where gj(·) is the j-th coordinate of g(·). For initialization, the weights are uniform, i.e. if j ∈ si, wij = 1/|si|, otherwise wij = 0. The algorithm of the overall framework is illustrated in Algorithm 2.

Augmented Solution for Image Datasets

The general solution is able to handle both image and nonimage datasets. As for image datasets, augmentation is an important procedure in classification tasks [Shorten and Khoshgoftaar, 2019]. Both ˜DλR and ˜DλU can employ image augmentation strategies to further enhance their performance. For the ˜DλR, the consistency regularization based method CRDPLL [Wu et al., 2022] is capable of being employed which takes advantage of image augmentation and achieves promising performance on PLL. To be specific, the LPLL can be written as:

where LSup (x, s) = - ∑k /∈s log (1 - gk(x)) and Ψ(x, s) = ∑z∈A(x) KL(s‖g(z)). KL(·) denotes the Kullback-Leibler divergence and A(x) denotes the set of random augmented versions of instance x. π(t) = min{tπ/T ′ , π} is a dynamic balancing factor, where t is the current epoch and T ′ is a constant. More specifically, the factor is increased to π at the T ′ - th epoch, and thereafter maintained at a constant value of π until the end of the training. Meanwhile, the label weights are iteratively updated every epoch, more details are presented in Appendix A.1.

As for ˜DλU , a semi-supervised learning method [Sohn et al., 2020] can be leveraged to extract potential valuable information in the unreliable instances. Specifically, for the images in the unreliable subset ˜DλU , the pseudo labels generated by the model's prediction where the images is weakly augmented. Next, we selectively preserve the samples whose pseudo labels satisfy the condition of max(pg(xwi )) ≥ τ. Then the model is trained to predict the pseudo labels when fed with a strongly-augmented version of the same image. Thus, the loss function for unreliable subset ˜DλU takes the following form:

where xwi and xsi are the weak and strong augmentation of xi respectively. 1(·) is an indicator function and LCE represents corss-entropy loss. Consequently, the objective of UPLLRS is as follows:

where ξ is a scalar hyperparameter.

5 Experiments

A comprehensive set of experiments were conducted to evaluate the performance of our method under varying levels of partial and unreliable labeling. The results demonstrate that our approach achieves state-of-the-art accuracy on tasks involving UPLL.

5.1 Datasets and Implementation Details Datasets

We utilize two commonly employed image datasets, CIFAR10 and CIFAR-100 [Krizhevsky et al., 2009], as the basis for synthesizing our UPLL dataset. Besides, we also utilize two additional datasets Dermatology and 20Newsgroups from UCI machine learning Repository [Dua and Graff, 2017] to further validate the effectiveness of our proposed method. In our experiments, the datasets are partitioned into training, validation, test set in a 4:1:1 ratio. Further elaboration can be found in Appendix A.2.

Following the confusing strategy in [Lv et al., 2023], the ground-truth labels in the raw dataset are corrupted initially and then generate partial labels by the flipping process. That is to say, for an instance x with the ground-truth label y = i, i ∈ Y, it has a fixed probability 1 - µ do not make any operation. But it has a probability κ to flip into j, where j ∈ Y, j ̸= i, κ = µ/(C - 1). The unreliable label is called ˜ yi . Subsequently, ˜ yi is considered as the true label for generating the candidate label set, employing a uniform partial labeling with probability η in accordance with the approach presented in [Lv et al., 2020], where η denotes the partial rate.

5.2 Baselines

In order to demonstrate the efficacy of our proposed method and to gain insight into its underlying characteristics, we conduct comparisons with seven benchmark methods including one UPLL method and six state-of-the-art PLL methods: 1) RABS [Lv et al., 2023]: An unreliable PLL method that proved the robustness of Average-Based Strategy (ABS) with bounded loss function in mitigating the impact of unreliability. In our experiment, the Mean Average Error (MAE) loss is chosen as the baseline. 2) PiCO [Wang et al., 2022]: A PLL method combines the idea of contrastive learning and class prototype-based label disambiguation method. 3) CR-DPLL [Wu et al., 2022]: A deep PLL method based on consistency regularization. 4) PRODEN [Lv et al., 2020]: A PLL method which progressively identifies true labels in candidate label sets. 5) RC [Feng et al., 2020]: A risk-consistent method for PLL which employs importance re-weighting strategy. 6) CC [Feng et al., 2020]: A classifier-consistent method for PLL using transition matrix to form an empirical risk estimator. 7) LWS [Wen et al., 2021]: A PLL method utilizing Leveraged weighted (LW) loss which balances the trade-off between losses on partial labels and others.

Note that two partial label learning methods PiCO [Wang et al., 2022] and CR-DPLL [Wu et al., 2022] are not suitable on the Dermatology and 20Newsgroups. More details can be found in Appendix A.2.

Implementation Details

For the first stage (i.e. self-adaptive RS), a 5-layer perceptron (MLP) is utilized to separate samples with CCE [Lv et al., 2023] loss. The learning rate is 0.1, 0.18, 0.1; small epochs β = 5, 6, 5; separation rate γ = 0.03, 0.005, 0.03; on the CIFAR-10, CIFAR-100 and UCI datasets respectively. Max separation step λ = ⌊log1-γ 0.3⌋. As for the second stage, we employ different backbones for different datasets. On CIFAR-10 dataset and CIFAR-100 dataset, we use WideResNet28 × 2 [Zagoruyko and Komodakis, 2016] as the predictive model, and we employ the Augmented Solution. On the UCI datasets, a 5-layer perceptron (MLP) is employed, and we utilize the General Solution. The learning rate is 5e - 2 and the weight decay is 1e - 3; ξ is set as 2 on CIFAR-10 and 0.3 on CIFAR-100. We implement the data augmentation technique following the "strong augmentation" in CR-DPLL [Wu et al., 2022]. This processing is applied on Ours, PRODEN, RC, CC, LWS. As for PiCO and CR-DPLL, the augmentation setups followed recommended setting in the previous works.

The optimizer in our experiment is Stochastic Gradient Descent (SGD) [Robbins and Monro, 1951] in which momentum is set as 0.9. For the learning rate scheduler, we use a cosine learning rate decay [Loshchilov and Hutter, 2016]. Otherwise, each model is trained with maximum epochs T = 500 and employs early stopping strategy with patience 25. In other words, if the accuracy does not rise in validation set V for 25 epochs, the training process will be stopped. All experiments are conducted on NVIDIA RTX 3090. What's more, the implementation of our method is based on PyTorch [Paszke et al., 2019] framework. We report final performance using the test accuracy corresponding to the best accuracy on validation set for each run. Finally, we report the mean and standard deviation based on five independent runs with different random seeds.

5.3 Experiment Results

Table 1 reports the experimental results on CIFAR-10 and CIFAR-100 synthesized datasets. As is shown, our UPLLRS method outperforms all compared methods. The improvements are particularly pronounced in scenarios with high levels of unreliability. Take η = {0.1, 0.3, 0.5}, µ = 0.5 on CIFAR-10 dataset as an instance, our method improves by 7.47%, 9.1%, 11.25% respectively compared with the second-best methods. It is worth noting that our method exhibits a minimal decline in accuracy as the unreliable rate µ increases. For example, for η = 0.5, the accuracy for µ = 0.1 is 92.47% and µ = 0.5 is 89.56%, only 2.91% accuracy drop. In contrast, the second-best method's accuracy drop up to 13.57%. UPLLRS also achieves the best performance and significantly outperforms other compared methods on the CIFAR-100 synthesized dataset. For the settings with η = 0.1, µ = 0.5, our UPLLRS also achieves 60.66% outperforming second-best 8.32%. Conversely, other methods either exhibit poor performance or fail to converge.

We further evaluate the performance of UPLLRS on nonimage datasets Dermatology and 20Newsgroups. Table 2 reports the experimental results on it. Our method demonstrates a clear advantage and surpasses all the methods evaluated in comparison. For instance, in the condition of η = 0.3, µ = {0.3, 0.5} on Dermatology, our method exhibited a 9.83%,

23.94% over the second-best method respectively. Furthermore, during experimentation with η = 0.1, our method exhibited a drop of 11.29% when varying the µ from 0.3 to 0.5. Hence, our method has noticeable resistance to unreliability. In contrast, other methods exhibit a significant decline.

5.4 Ablation Study

In this subsection, we present the results of our ablation study which serve to demonstrate the efficacy of the components of our UPLLRS method: RS and Unreliable Subset ˜DλU . The experiments are conducted on CIFAR-10 dataset with η = 0.3, µ = {0.1, 0.3, 0.5} and CIFAR-100 dataset with η = 0.05, µ = {0.1, 0.3, 0.5}. Other hyperparameter settings are consistent with those utilized in the primary experiments. Besides, analysis on hyperparameter ξ and γ are elaborated in detail in the Appendix A.3.

Here, we conduct ablation studies on the individual components to investigate their contributions. Two variants are selected: 1) Without RS. That implies that the corrupted dataset will not be partitioned into subsets, but rather utilized directly to induce the final classifier. 2) Without unreliable subset ˜DλU . It can be stated that the final classifier is directly trained on the reliable subset. All other parameters are held constant as in the primary experiment. As shown in Table 3, It is apparent that the contribution of RS surpasses that of ˜DλU . Take CIFAR-10 with µ = 0.5 as an instance, it was observed that the variant without the utilization of the ˜DλU experienced a mere 1.35% decline in performance compared to the full UPLLRS model. However, when comparing the variant without the self-adaptive RS to the variant without the ˜DλU , a significant decline of 7.75% was observed. Furthermore, while utilizing the ˜DλU on the CIFAR-100 dataset resulted in a slight increase, this can likely be attributed to the lower accuracy of pseudo-label generation from ˜DλU , as CIFAR-100 has a substantially larger number of classes than CIFAR-10.

6 Conclusion

In this work, we propose a novel two-stage framework named Unreliable Partial Label Learning with Recursive Separation (UPLLRS). First, the self-adaptive recursive separation strategy is proposed to separate the training set into a reliable subset and an unreliable subset. Second, a disambiguation strategy progressively identifies ground-truth labels in the reliable subset. Meanwhile, the semi-supervised learning techniques are employed for the unreliable subset. Experimental results demonstrate that our method attains state-of-the-art performance, particularly exhibiting robustness in scenarios with high levels of unreliability.

Acknowledgments

This work is supported by National Key R&D Program of China (2018AAA0100104), the National Science Foundation of China (62206050, 62125602, and 62076063), China Postdoctoral Science Foundation (2021M700023), Jiangsu Province Science Foundation for Youths (BK20210220), Young Elite Scientists Sponsorship Program of Jiangsu Association for Science and Technology (TJ-2022-078), and the Big Data Computing Center of Southeast University.