Boosting Single Positive Multi-label Classification with Generalized Robust Loss
Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence. Main Track. Pages 3825-3833.
https://doi.org/10.24963/ijcai.2024/423
Keywords
- Machine Learning: ML: Weakly supervised learning
- Machine Learning: ML: Classification
- Machine Learning: ML: Multi-label learning
Abstract
Multi-label learning (MLL) requires comprehensive multi-semantic annotations that is hard to fully obtain, thus often resulting in missing labels scenarios. In this paper, we investigate Single Positive Multi-label Learning (SPML), where each image is associated with merely one positive label. Existing SPML methods only focus on designing losses using mechanisms such as hard pseudo-labeling and robust losses, mostly leading to unacceptable false negatives. To address this issue, we first propose a generalized loss framework based on expected risk minimization to provide soft pseudo labels, and point out that the former losses can be seamlessly converted into our framework. In particular, we design a novel robust loss based on our framework, which enjoys flexible coordination between false positives and false negatives, and can additionally deal with the imbalance between positive and negative samples. Extensive experiments show that our approach can significantly improve SPML performance and outperform the vast majority of state-of-the-art methods on all the four benchmarks. Our code is available at https://github.com/yan4xi1/GRLoss.
- [Bekker and Davis, 2020] Jessa Bekker and Jesse Davis. Learning from positive and unlabeled data: A survey. Ma- chine Learning, 109:719-760, 2020.
- [Chen et al., 2023] Hao Chen, Ran Tao, Yue Fan, Yidong Wang, Jindong Wang, Bernt Schiele, Xing Xie, Bhiksha Raj, and Marios Savvides. SoftMatch: Addressing the quantity-quality tradeoff in semi-supervised learning. In ICLR, 2023.
- [Chua et al., 2009] Tat-Seng Chua, Jinhui Tang, Richang Hong, Haojie Li, Zhiping Luo, and Yantao Zheng. Nus- wide: a real-world web image database from national uni- versity of singapore. In Proceedings of the ACM CIVR, pages 1-9, 2009.
- [Cole et al., 2021] Elijah Cole, Oisin Mac Aodha, Titouan Lorieul, Pietro Perona, Dan Morris, and Nebojsa Jojic. Multi-label learning from single positive labels. In CVPR, pages 933-942, 2021.
- [Durand et al., 2019] Thibaut Durand, Nazanin Mehrasa, and Greg Mori. Learning a deep ConvNet for multi-label classification with partial labels. In CVPR, pages 647-657, 2019.
- [Everingham and Winn, 2012] Mark Everingham and John Winn. The PASCAL visual object classes challenge 2012 (VOC2012) development kit. Pattern Anal. Stat. Model. Comput. Learn., Tech. Rep, 2007(1-45):5, 2012.
- [Feng et al., 2020] Lei Feng, Jun Huang, Senlin Shu, and Bo An. Regularized matrix factorization for multilabel learning with missing labels. IEEE Transactions on Cy- bernetics, 52(5):3710-3721, 2020.
- [Feng et al., 2021] Lei Feng, Senlin Shu, Zhuoyi Lin, Feng- mao Lv, Li Li, and Bo An. Can cross entropy loss be ro- bust to label noise? In Proceedings of the Twenty-Ninth In- ternational Conference on International Joint Conferences on Artificial Intelligence, pages 2206-2212, 2021. IJCAI
- [Ghiassi et al., 2023a] Amirmasoud Ghiassi, Robert Birke, and Lydia Y Chen. Multi label loss correction against missing and corrupted labels. In Asian Conference on Ma- chine Learning, pages 359-374. PMLR, 2023.
- [Ghiassi et al., 2023b] Amirmasoud Ghiassi, Cosmin Octa- vian Pene, Robert Birke, and Lydia Y Chen. Trusted loss correction for noisy multi-label learning. In Asian Confer- ence on Machine Learning, pages 343-358. PMLR, 2023.
- [Ghosh et al., 2017] Aritra Ghosh, Himanshu Kumar, and P Shanti Sastry. Robust loss functions under label noise for deep neural networks. In Proceedings of the AAAI con- ference on artificial intelligence, volume 31, 2017.
- [He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of CVPR, pages 770-778, 2016.
- [Higashimoto et al., 2024] Ryota Higashimoto, Soh Yoshida, Takashi Horihata, and Mitsuji Muneyasu. Unbiased pseudo-labeling for learning with noisy la- bels. IEICE Transactions on Information and Systems, 107(1):44-48, 2024.
- [Hu et al., 2021] Zijian Hu, Zhengyu Yang, Xuefeng Hu, and Ram Nevatia. Simple: Similar pseudo label exploita- tion for semi-supervised classification. In CVPR, pages 15099-15108, 2021.
- [Huynh and Elhamifar, 2020] Dat Huynh and Ehsan Elhami- far. Interactive multi-label cnn learning with partial labels. In CVPR, pages 9423-9432, 2020.
- [Jiang et al., 2023] Yangbangyan Jiang, Qianqian Xu, Yun- rui Zhao, Zhiyong Yang, Peisong Wen, Xiaochun Cao, and Qingming Huang. Positive-unlabeled learning with la- bel distribution alignment. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
- [Kim et al., 2022] Youngwook Kim, Jae Myung Kim, Zeynep Akata, and Jungwoo Lee. Large loss matters in weakly supervised multi-label classification. In Proceed- ings of CVPR, pages 14156-14165, 2022.
- [Kiryo et al., 2017] Ryuichi Kiryo, Gang Niu, Marthinus C Du Plessis, and Masashi Sugiyama. Positive-unlabeled learning with non-negative risk estimator. Advances in Neural Information Processing Systems, 30, 2017.
- [Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Be- longie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Com- mon objects in context. In The 13th European Conference on Computer Vision, pages 740-755. Springer, 2014.
- [Lin et al., 2017] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense ob- ject detection. In Proceedings of the IEEE international Conference on Computer Vision, pages 2980-2988, 2017.
- [Liu et al., 2021] Weiwei Liu, Haobo Wang, Xiaobo Shen, and Ivor W Tsang. The emerging trends of multi-label learning. IEEE Transactions on Pattern Analysis and Ma- chine Intelligence, 44(11):7955-7974, 2021.
- [Liu et al., 2022] Fengbei Liu, Yu Tian, Yuanhong Chen, Yuyuan Liu, Vasileios Belagiannis, and Gustavo Carneiro. ACPL: Anti-curriculum pseudo-labelling for semi- supervised medical image classification. In CVPR, pages 20697-20706, 2022.
- [Liu et al., 2023a] Biao Liu, Jie Wang, Ning Xu, and Xin Geng. Can class-priors help single-positive multi-label learning? arXiv preprint arXiv:2309.13886, 2023.
- [Liu et al., 2023b] Biao Liu, Ning Xu, Jiaqi Lv, and Xin Geng. Revisiting pseudo-label for single-positive multi- label learning. In International Conference on Machine Learning, pages 22249-22265. PMLR, 2023.
- [Lv et al., 2019] Jiaqi Lv, Ning Xu, RenYi Zheng, and Xin Geng. Weakly supervised multi-label learning via label enhancement. In IJCAI, pages 3101-3107, 2019. IJCAI
- [Ma et al., 2020] Xingjun Ma, Hanxun Huang, Yisen Wang, Simone Romano, Sarah Erfani, and James Bailey. Nor- malized loss functions for deep learning with noisy labels. In ICML, pages 6543-6553. PMLR, 2020.
- [Rastogi and Mortaza, 2021] Reshma Rastogi and Sayed Mortaza. Multi-label classification with missing labels using label correlation and robust structural learning. Knowledge-based Systems, 229:107336, 2021.
- [Ridnik et al., 2021] Tal Ridnik, Emanuel Ben-Baruch, Na- dav Zamir, Asaf Noy, Itamar Friedman, Matan Protter, and Lihi Zelnik-Manor. Asymmetric loss for multi-label clas- sification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 82-91, 2021.
- [Russakovsky et al., 2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. ImageNet large scale visual recogni- tion challenge. International Journal of Computer Vision, 115:211-252, 2015.
- [Sun et al., 2022] Ximeng Sun, Ping Hu, and Kate Saenko. Dualcoop: Fast adaptation to multi-label recognition with limited annotations. NeurIPS, 35:30569-30582, 2022.
- [Tarekegn et al., 2021] Adane Nega Tarekegn, Mario Gia- cobini, and Krzysztof Michalak. A review of methods for imbalanced multi-label classification. Pattern Recog- nition, 118:107965, 2021.
- [Verelst et al., 2023] Thomas Verelst, Paul K Rubenstein, Marcin Eichner, Tinne Tuytelaars, and Maxim Berman. Spatial consistency loss for training multi-label classi- fiers from single-label annotations. In Proceedings of the IEEE/CVF Winter Conference on Applications of Com- puter Vision, pages 3879-3889, 2023.
- [Wah et al., 2011] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The Caltech- UCSD birds-200-2011 dataset. 2011.
- [Wang et al., 2019] Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In Proceed- ings of ICCV, pages 322-330, 2019.
- [Wang et al., 2022] Haobo Wang, Ruixuan Xiao, Yixuan Li, Lei Feng, Gang Niu, Gang Chen, and Junbo Zhao. Pico: Contrastive label disambiguation for partial label learning. In ICLR, 2022.
- [Wang et al., 2023] Ao Wang, Hui Chen, Zijia Lin, Zixuan Ding, Pengzhang Liu, Yongjun Bao, Weipeng Yan, and Guiguang Ding. Hierarchical prompt learning using clip for multi-label classification with single positive labels. In Proceedings of the 31st ACM International Conference on Multimedia, pages 5594-5604, 2023.
- [Wu et al., 2014] Baoyuan Wu, Zhilei Liu, Shangfei Wang, Bao-Gang Hu, and Qiang Ji. Multi-label learning with missing labels. In the 22nd International Conference on Pattern Recognition, pages 1964-1968. IEEE, 2014.
- [Xia et al., 2023a] Shiyu Xia, Jiaqi Lv, Ning Xu, Gang Niu, and Xin Geng. Towards effective visual representations for partial-label learning. In Proceedings of CVPR, pages 15589-15598, 2023.
- [Xia et al., 2023b] Xiaobo Xia, Jiankang Deng, Wei Bao, Yuxuan Du, Bo Han, Shiguang Shan, and Tongliang Liu. Holistic label correction for noisy multi-label classifica- tion. In ICCV, pages 1483-1493, 2023.
- [Xie and Huang, 2018] Ming-Kun Xie and Sheng-Jun Huang. Partial multi-label learning. In Proceedings of AAAI, volume 32, 2018.
- [Xie and Huang, 2021] Ming-Kun Xie and Sheng-Jun Huang. Partial multi-label learning with noisy label identification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(7):3676-3687, 2021.
- [Xie et al., 2022] Ming-Kun Xie, Jiahao Xiao, and Sheng- Jun Huang. Label-aware global consistency for multi-label learning with single positive labels. Advances in Neural Information Processing Systems, 35:18430-18441, 2022.
- [Xu et al., 2022] Ning Xu, Congyu Qiao, Jiaqi Lv, Xin Geng, and Min-Ling Zhang. One positive label is suffi- cient: Single-positive multi-label learning with label en- hancement. NeurIPS, 35:21765-21776, 2022.
- [Zhang and Sabuncu, 2018] Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. NeurIPS, 31, 2018.
- [Zhang et al., 2021] Youcai Zhang, Yuhao Cheng, Xinyu Huang, Fei Wen, Rui Feng, Yaqian Li, and Yandong Guo. Simple and robust loss design for multi-label learning with missing labels. arXiv preprint arXiv:2112.07368, 2021.
- [Zhang et al., 2023] Wenqiao Zhang, Changshuo Liu, Lingze Zeng, Bengchin Ooi, Siliang Tang, and Yueting Zhuang. Learning in imperfect environment: Multi-label classification with long-tailed distribution and partial labels. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1423-1432, 2023.
- [Zhao et al., 2022] Xingyu Zhao, Yuexuan An, Ning Xu, and Xin Geng. Fusion label enhancement for multi-label learn- ing. In IJCAI, 2022. IJCAI
- [Zhou et al., 2022a] Donghao Zhou, Pengfei Chen, Qiong Wang, Guangyong Chen, and Pheng-Ann Heng. Acknowl- edging the unknown for multi-label learning with single positive labels. In ECCV, pages 423-440. Springer, 2022.
- [Zhou et al., 2022b] Jianan Zhou, Jianing Zhu, Jingfeng Zhang, Tongliang Liu, Gang Niu, Bo Han, and Masashi Sugiyama. Adversarial training with complementary la- bels: On the benefit of gradually informative attacks. NeurIPS, 35:23621-23633, 2022.
References were extracted automatically from the PDF and may contain errors.
Full text extracted automatically from the PDF; figures, tables and equations may differ. The PDF is the version of record.
1 Introduction
Multi-label learning (MLL) constructs predictive models to provide multi-label assignments to unseen images [Zhao et al., 2022]. Conventionally, MLL assumes that each training instance is fully and accurately labeled with all relevant classes. However, obtaining such comprehensive labels is often expensive and even impractical in many realworld scenarios [Lv et al., 2019]. Due to these limitations of vanilla MLL, Multi-label Learning with Missing Labels (MLML) [Wu et al., 2014] received broad attentions, and was usually considered as a typical weakly supervised learning problem [Kim et al., 2022].
In this paper, we focus on an extreme variant of MLML, i.e., Single Positive Multi-label Learning (SPML), where only one label is verified as positive for a given image, leaving all the other labels as the unknown ones [Cole et al., 2021]. Compared to a fully labeled setting, SPML poses a practically more relevant yet substantially more challenging scenario. Since only one positive label is known, existing approaches for general MLML based on label relations, such as learning positive label correlations [Huynh and Elhamifar, 2020], creating label matrices [Feng et al., 2020], or learning to infer missing labels [Durand et al., 2019], are obstructed in the context of single labeled positive [Cole et al., 2021].
∗ Full version can be found in https://arxiv.org/abs/2405.03501.
†Corresponding Author
Nevertheless, in the realm of SPML, the priority still lies in effectively dealing with missing labels. Generally speaking, two strategies are commonly leveraged. The first strategy is to treat missing labels as unknown variables needed to be predicted [Rastogi and Mortaza, 2021]. As discussed above, predicting such unseen labels based on only one positive label is indeed challenging. The second one is to assume the missing labels are All Negative (AN), so that to transform SPML into a fully supervised MLL problem [Cole et al., 2021]. To our knowledge, AN assumption is still of the most popularity for MLML, including SPML (as a baseline) [Liu et al., 2023b], and normally trained with Binary Cross-entropy (BCE) loss. Following the relevant study, in this paper, AN with BCE is also considered as the baseline in our experimental part.
Unfortunately, AN assumption is oversimplified and inevitably results in a large number of false negative labels [Zhang et al., 2023]. Ignoring false negatives will introduce label noise, which substantially impairs SPML performance [Ghiassi et al., 2023b]. To circumvent the pitfalls of mislabeling in AN, pseudo-labeling has been explored [Hu et al., 2021]. In this work, we adapt the ideas of soft pseudo labeling to SPML. In addition, the remaining false negatives in pseudo-labeling are regarded as noise [Liu et al., 2023b], and properly handling these noisy labels can consistently improve SPML performance [Xia et al., 2023b]. Therefore, we specifically design a novel robust denoising loss to reduce the negative impact of noise in pseudo labels.
Another challenge in SPML is the extreme intra-class imbalance between positive and negative labels within one category, which is much severer than vanilla MLL and MLML [Ridnik et al., 2021; Tarekegn et al., 2021]. In order to tackle intra-class imbalance in SPML, a novel approach using both entropy maximization and asymmetric pseudolabeling has been introduced [Zhou et al., 2022a], leading to promising results. Nevertheless, the mechanism to address the issue of positive-negative label imbalance in SPML remains relatively unexplored. Furthermore, the imbalance across classes, a.k.a inter-class imbalance, easily causes the Long-tailed effect [Zhang et al., 2023], which is even aggravated in SPML. In this paper, we effectively mitigate both intra-class and inter-class imbalance issue by instance and class sensitive reweighting.
In this paper, aiming at solving the SPML problem, we propose a new loss function framework. Specifically, we first derive an empirical risk estimation for SPML based on singleclass unbiased risk estimation (URE), and accordingly propose a novel loss function framework for SPML problems. To integrate Pseudo-labeling, Loss reweighting and Robust loss into our framework, we build Generalized Robust Loss (GR Loss) for SPML, allowing previous SPML methods to be naturally incorporated. With two mild assumptions, we offer the specific forms of the components in our GR Loss. Extensive experimental results on four benchmark datasets demonstrate that our GR Loss can achieve state-of-the-art performance. Our contributions can be summarized in four-fold:
- Framework level: We propose a novel SPML loss function framework to estimate the posterior probability of missing labels more accurately. In particular, our framework can unify existing SPML methods, enabling a comprehensive and holistic understanding of these former work.
- Formulation level: We design a tailored robust loss function for both positive and negative labels, as a novel surrogate for cross-entropy loss. The proposed robust loss is offthe-shell and can be seamlessly plug-in for our proposed framework to build the full Generalized Robust Loss.
- Methodology level: We elaborate the false negative and imbalance issues in SPML by gradient analysis and empirical study, and first propose a soft pseudo-labeling mechanism to address these issues.
- Experimental level: We demonstrate the superiority of our GR Loss by conducting extensive empirical analysis with performance comparison against state-of-the-art SPML methods across four benchmarks.
2 RelatedWork
Due to its highly incomplete labels, SPML presents unique challenge for MLL [Liu et al., 2023a], especially resulting in oversimplified prediction to falsely identify all labels as positive [Verelst et al., 2023]. In addition to SPML, incomplete supervision widely occurs in the context of weakly supervised learning, where pseudo-labeling has been widely employed [Liu et al., 2022; Chen et al., 2023; Wang et al., 2022; Zhou et al., 2022b]. The pseudo-labeling was initially applied to SPML to waive the issue of false negatives caused by AN assumption [Cole et al., 2021], where the pseudo labels were incorporated with regularization [Zhou et al., 2022a; Liu et al., 2023b]. In particular, label-aware global consistency regularization leverages contrastive learning to extract flow structure information, so that the latent soft labels can be accurately recovered [Xie et al., 2022].
Pseudo-labeling may introduce label noise [Xia et al., 2023a; Higashimoto et al., 2024]. In order to effectively cope with noisy labels, denoising robust losses [Ma et al., 2020], such as Generalized Cross Entropy (GCE) [Zhang and Sabuncu, 2018], Symmetric Cross Entropy (SCE)) [Wang et al., 2019], and Taylor Cross Entropy [Feng et al., 2021], have been successively developed. In particular, robust Mean Absolute Error (MAE) loss was combined with Cross Entropy (CE) loss to simultaneously consider the robustness and generalization performance [Ghosh et al., 2017]. It has been shown that robust losses, typically reserved for multi-class scenario, are also effective in MLL, especially in the presence of incomplete labels [Zhang et al., 2021; Xu et al., 2022], where both false positive and false negative labels should be properly addressed [Ghiassi et al., 2023a]. Our method is under the same umbrella, and specifically tailors robust loss to SPML.
3 Method
3.1 ProblemDefinition
In partial multi-label learning [Xie and Huang, 2018; Xie and Huang, 2021], the full data can be represented as a triplet (x, y, s), where x∈X represents the input feature, y∈Y = {0, 1}C denotes the ground-truth label, and s ∈S ={0, 1}C is the observed label. In addition, yc denotes the c-th entry of y: yc = 1 indicates the relevance to the c-th class, while yc =0 means the irrelevance. Besides, sc is the c-th entry of s: sc=1 indicates the relevance to the c-th class, while sc=0 means either irrelevance or relevance, i.e., the unknown.
Let D={(xn , sn )}Nn=1 be a partially labeled MLL training set with N instances. For a given sample xn , we only have access to the observed label sn , while the ground-truth label yn remains unknown. In SPML, each instance xn has only one observed positive label, leaving the observed labels for all other categories being missing, represented as ∑Ci=1 sni =1.
3.2 Expected Risk Estimation
The goal of SPML is to find a function f : X →Y from training set D to predict true labels for each x∈X. In multi-label classification, the standard approach is to independently train C binary classifiers f=[f1, f2, · · · , fC], using a shared backbone. For a certain class, it can be viewed as a PU-learning problem [Kiryo et al., 2017; Jiang et al., 2023], where y and s degenerate to scalars. Let s = 1 if x is labeled, and s = 0 if it is unlabeled. Accordingly, the surrogate expected risk function w.r.t. loss L is:
Then, the empirical risk is derived from Eq.(1), and the details are provided in Appendix A of our full version:
where k(x)=P(y =1 | x, s=0) represents the probability that an unlabeled sample is false negative under AN assumption. Consequently, for the classifier f, the empirical risk is:
3.3 Novel SPML Loss Framework
According to Eq.(3), we propose a novel loss function framework for SPML based on the following two considerations.
Pseudo-labeling. Given the difficulty in evaluating accurate ki(x), we estimate it in an online manner: Let pi(x) = fi(x) be the current model output, and we use ˆ k(pi(x); β) to esitimate ki(x), where β is the parameters.
Loss reweighting. In order to deal with the class imbalance and false negatives, we propose the class-and-instancespecific weight v(pni ; α) to reweight different samples w.r.t. the category, where α is the parameters. Typically, for any certain class i, positive and negative losses are in a unified form for balanced classification, such as cross entropy loss L+i =-log(pi), L-i =-log(1-pi). However, there is a severe intra-class and inter-class imbalance in SPML, together with noisy labels in the pseudo labels and negative samples, i.e., false negatives. Therefore, we add v(pni ; αi) to the loss of each instance for each class, and use Lni,1 , Lni,2 , Lni,3 to decouple the loss in Eq.(3) into three surrogate losses. Accordingly, the total loss can be expressed as:
and the class-and-instance-specific loss Lni is decoupled as:
The Analysis and Formulation of ˆ k(p; β)
In this section, we present two assumptions on ˆ k(p; β). The superscript n and subscript i are properly omitted for simplicity. We defer to verify their validity in Appendix E.3 (see the full version). Based on these assumptions, we introduce the explicit form of the pseudo-labeling function ˆ k(p; β).
Assumption 1. In the initial training phase, ˆ k(p; β) is nearly a constant function.
Remark 1. In the initial phase of training, the model is unable to provide accurate prediction, thus the initial output can be approximately considered as random one. As a result, the probability ˆ k(p; β) is independent of the output and performs as a constant function. Furthermore, this constant is approximately equal to the number of false-negative labels divided by the number of missing labels, i.e., ˆ k(p; β)= #{y=1,s=0} #{s=0} .
Assumption 2. In the final training stage, ˆ k(p; β) gradually becomes a monotonically increasing function.
Remark 2. In the final stage of training, the output of the well-trained model should perfectly fit the posterior probability P(y=1 | x). Therefore, we have:
where we use the noise-free fact of SPML, i.e., P(s = 1 | y = 0, x) = 0. Moreover, under the Selected Completely At Random (SCAR) assumption [Bekker and Davis, 2020], a= P(s = 1 | y = 1) = P(s = 1 | y = 1, x) is a constant independent of the sample and ranges between 0 and 1. As a result, ˆ k(p; β) is a monotonically increasing function.
The formulation. According to the above two assumptions, we define the form of ˆ k(p; β) as Logistic function:
where the parameters β = [w, b]. We point out that there are other choices for the form of ˆ k(p; β), and we use Logistic is because of its simplicity and it can easily satisfy the requirements in the assumptions. In detail, when w= 0, ˆ k(p; β) is a constant, corresponding to the early-stage training. On the other hand, if increasing w> 0 and decreasing b < 0, while maintaining their ratio within a certain range, the Logistic can reflect the characteristics of late-stage training. In particular, as w→+∞, bw =-τ, ˆ k(p; β) turns into the following threshold based hard pseudo-labeling strategy:
The update of β. Since ˆ k(p; β) is only used as a calibration for the model output p to better estimate k(x), we stop the gradient backpropagation of p in ˆ k(p; β), and instead consider β as a trainable parameter. However, the performance is not satisfactory if we treat β as parameter (see the details in the last row of Table 3). Consequently, we introduce linear regularization to deterministically update β along with the training process. In other words, we assume that both w, b linearly increase with the training epochs in the following manner:
Here, t=0, 1, 2, · · · , T denotes the current training epoch, T represents the total number of training epochs, w(0) , b(0) are initially fixed according to Assumption 1, and w(T) , b(T) are two hyperparameters.
The Analysis and Formulation of v(p; α)
When the model performs well, the output confidence p can be largely trusted. In this circumstance, when the corresponding observed label s = 0, if p is close to 1, it is probably that the corresponding instance is false negative or an outlier, and in either case, its weight should be reduced.
On the other hand, inspired by focal loss [Lin et al., 2017], in the case of imbalance between positive and negative samples, the weight of simple samples should be reasonably reduced. Hence, when s=0, if p is close to 0, the corresponding instance is probably a simple one, and its weight should be small. Conversely, if p is close to 0.5, its weight should be larger. Furthermore, because positive label is too limited and definitely correct, we set its weight to 1.
The formulation. Based on the above discussions, we define v(p; α) as:
The update of α. Similar to β, the parameters α = [σ, µ] are instead set as linearly updated. The value of α(t) evolve over the training epochs in the following manner:
where µ(0) , σ(0) are initially fixed, and µ(T ) , σ(T ) are two hyperparameters to be determined.
The Analysis and Formulation of L1, L2, L3
In order to obtain more accurate supervision information, for missing labels, we employ ˆ k(p; β) as a soft pseudo label to indicate the probability of its presence. However, in the training process, we still need to deal with a large amount of noise in pseudo labels. Training with robust loss can mitigate the noise issue. As a contrast, binary cross-entropy loss is good at fitting but does not enjoy robustness. In this work, we propose to combine MAE with BCE to build a novel robust loss.
Particularly, inspired by the Generalized Cross Entropy (GCE) loss, we define the loss functions as follows:
Here, q1, q2, q3 are hyperparameters to adjust the robustness of L1, L2, L3, which can be seen as a trade-off between MAE loss and BCE loss: When qj =1, Lj is the MAE loss; when qj → 0, Lj asymptotically becomes BCE. The closer qj is to 0, the lower robustness the loss exhibits, allowing faster learning convergence. Conversely, the closer qj is to 1, the stronger robustness the loss equipped, making it more effective against incorrect supervision. Thus, controlling qj can balance the convergence speed and robustness. Detailed explanations can be found in Appendix D (see the full version). Figure (1a) shows the loss curves of L3 for different values of q3: When q3 =0.001, the curve approximates BCE loss, and when q3=1, it represents MAE loss.
3.4 Relation to Other SPML Losses
In this section, we summarize the existing MLML and SPML losses and show how to integrate them into our unified loss function framework.
The core of our framework is to estimate the posterior probabilities of missing labels by adopting the label confidence p. As discussed above, v(p; α) and ˆ k(p; β) are both functions of p, which fundamentally aligns with the essence of the pseudo-labeling method. Transforming the pseudolabeling method into the proposed framework, v(p; α) and ˆ k(p; β) can be represented as:
Here, τ1, τ2 are adaptively changed with the training epoch t, acting as adaptive thresholds. Besides, in existing methods, L1, L2, L3 typically employ cross-entropy loss:
L1 = -log(p), L2 = -log(p), L3 = -log(1 -p). (17) We present specific forms of components in our framework corresponding to existing MLML/SPML methods in Table 1. More explanations are in Appendix B of full version.
3.5 Gradient Analysis
To better understand the performance of the proposed GR Loss, we conduct a gradient-based analysis, which is commonly used for in-depth study of loss functions [Ridnik et al., 2021]. Observing gradients is beneficial as, in practice, network weights are updated according to the gradient of loss. For convenience, let z=zc denote the output logit of the c-th class for x, and L∅=ˆ k · 1-pq2 q2 +(1-ˆ k) · 1-(1-p)q3 q3 represent the loss for the unannotated label. The gradients of L∅ w.r.t. z are given by:
where p=σ(z)=(1 + e-z )-1 and ˆ k is defined in Eq.(7).
An intuitive interpretation of the gradient is: When g(p) > 0, a decrease of p leads to reduction in loss; conversely, when g(p) < 0, an increase of p results in reduction in loss. Accordingly, we analyzed the gradient of GR Loss at different epochs and the loss gradients of other SPML methods, including EM loss and Hill loss. Figure (1b) illustrates the gradients of different losses, where β(0) and β(T) represent the gradients at the beginning and the end of training, respectively. Eventually, we reach the following conclusions:
- The imbalance between positive and negative labels is a key factor to affect SPML performance. By properly adjusting q3, we can rebalance the supervision information.
- The issue of false negatives is also a key factor to impact the gradient (see Eq.(18)), which can be effectively addressed by adjusting ˆ k(p; β).
A detailed derivation process and specific explanations with evidence are provided in Appendix C (see the full version).
| Methods | EN Loss [Verelst et al., 2023] | EM Loss [Zhou et al., 2022a] | Hill Loss [Zhang et al., 2021] | Focal Margin+ SPLC [Zhang et al., 2021] | GR Loss (Ours) |
|---|---|---|---|---|---|
| ˆ k(p) | {0 p ≤ τ1 undefined otherwise | {0 p≤τ1 1 otherwise | 0 | {0 p≤τ1 1 otherwise | 1 1+exp{-(w·p+b)} |
| v(p) | {1 otherwise 0 s=0 and p>τ1 | 1 otherwise α s=0 and p>τ1 β s=0 and p≤τ1 | 1 | 1 | {1 s = 1, exp{- (p-µ)2 2σ2 } s = 0. |
| L1 | - log(p) | - log(p) | - log(p) | (1-pm)γ log(pm) | 1-pq1 q1 |
| L2 | - log(p) | p log(p)+(1 - p) log(1 - p) | undefined | (1-pm)γ log(pm) | 1-pq2 q2 |
| L3 | - log(1 - p) | -ˆ p log(p)-(1 - ˆ p) log(1 -p) | -(λ-p)p2 | -(λ - p)p2 | 1-(1-p)q3 q3 |

Figure 1: The gradient analysis. Better viewed in color.
4 Experiments
We provide the main empirical results of the proposed GR Loss in this section, including the comparison with state-ofthe-art methods, the ablation study of our framework and robust loss, and the hyperparameter analysis. More experimental results can be found in Appendix E of the full version.
4.1 Experimental Setup
Dataset. We evaluate our proposed GR Loss on four benchmark datasets: Pascal VOC-2012 (VOC) [Everingham and Winn, 2012], MS-COCO-2014(COCO) [Lin et al., 2014], NUS-WIDE(NUS) [Chua et al., 2009], and CUB200-2011(CUB) [Wah et al., 2011]. We first simulate the single-positive label training environments commonly used in SPML [Cole et al., 2021], and replicate their training, validation and testing samples. In these datasets, only one positive label is randomly selected for each training instance, while the validation and test sets remain fully labeled. More dataset descriptions are provided in Appendix E.1 of full version.
Implementation details and hyperparameters. For fair comparison, we follow the mainstream SPML implementation [Cole et al., 2021]. In detail, we employ ResNet-50 [He et al., 2016] architecture, which was pre-trained on ImageNet dataset [Russakovsky et al., 2015]. Each image is resized into 448×448, and performed data augmentation by randomly flipping an image horizontally. We initially conduct a search to determine and fix the hyperparameters q2 and q3 in Eq.(14), typically 0.01 and 1, respectively. Because the robust loss has a significant impact on training, which is also reflected in Table 3. Therefore, we only need to adjust four hyperparameters in (β(T) , α(T) ). More details about hyperparameter settings are described in Appendix E.2 of our full version.
Comparing methods. In our empirical study, we compare our method to the following state-of-the-art methods: AN loss (assuming-negative loss) [Cole et al., 2021], AN-LS (AN loss combined with Label Smoothing) [Cole et al., 2021], Focal loss [Lin et al., 2017], ROLE (Regularised Online Label Estimation) [Cole et al., 2021], Hill loss [Zhang et al., 2021], SPLC (Self-Paced Loss Correction) [Zhang et al., 2021], EM (Entropy-Maximization Loss) [Zhou et al., 2022a], EM+APL (Asymmetric Pseudo-Labeling) [Zhou et al., 2022a], SMILE (Single-positive MultI-label learning with Label Enhancement) [Xu et al., 2022], Large Loss (LL-R, LL-Ct, LLCp) [Kim et al., 2022], and MIME [Liu et al., 2023b]. The goal of all the above methods is to propose new SPML loss function, which is consistent with ours. Indeed, we are also concerned that there are methods that adopt pre-training strategies or use different backbones, such as: LAGC [Xie et al., 2022], DualCoop [Sun et al., 2022], HSPNet [Wang et al., 2023], CRISP [Liu et al., 2023a]. Due to the length limitation and in order to conduct a fair comparison, we only consider the former loss function modification approach in this study.
4.2 Results and Discussion
The experimental results of most existing MLML and SPML methods on four SPMLbenchmarks are reported inTable 2. It is observed that AN loss performs the worst almost on all four datasets, indicating that the false negative noise introduced by AN assumption has a significant negative impact on SPML. Meanwhile, Focal Loss, as a baseline method, is ineffective for imbalance in SPML, especially the inter-class imbalance.
Notably, our GR Loss outperforms existing methods in the first three SPML benchmarks, i.e., VOC, COCO, NUS, and achieves the second-highest result on CUB, slightly lower than MIME. The main reason for the second best on CUB is, our method treats SPML as C multiple binary classification tasks and the more categories means the more positive labels for each image, thus the stronger correlation between labels, leading to inferior performance of our method. In detail, for VOC (20 classes), the mAP is 0.63% higher than the secondbest, and on COCO (80 classes) and NUS (81 classes), the improvements are 0.25% and 0.34%, respectively. However, there are 312 classes in CUB and an average of 31.5 positive classes per image. As a result, our method is slightly worse than MIME, which considers label correlation and inter-class differences.
| Methods | VOC | COCO | NUS | CUB | |
|---|---|---|---|---|---|
| Baselines | AN [Cole et al., 2021] et | 85.89±0.38 | 64.92±0.19 | 42.49±0.34 | 18.66±0.09 |
| AN-LS [Cole al., 2021] | 87.55±0.14 | 67.07±0.20 | 43.62±0.34 | 16.45±0.27 | |
| Focal loss [Lin et al., 2017] | 87.59±0.58 | 68.79±0.14 | 47.00±0.14 | 19.80±0.30 | |
| Hill loss [Zhang et al., 2021] | 87.71±0.26 | 71.43±0.18 | - | - | |
| SPLC [Zhang et al., 2021] | 88.43±0.34 | 71.56±0.12 | 47.24±0.17 | 18.61±0.14 | |
| MLML Methods LL-R [Kim et al., 2022] | 89.15±0.16 | 71.77±0.15 | 48.05±0.07 | 19.00±0.02 | |
| LL-Ct [Kim et al., 2022] | 88.97±0.15 | 71.20±0.29 | 48.00±0.08 | 19.31±0.16 | |
| LL-Cp [Kim et al., 2022] | 88.62±0.23 | 70.84±0.33 | 47.93±0.01 | 19.01±0.10 | |
| ROLE [Cole et al., 2021] | 88.26±0.21 | 69.12±0.13 | 41.95±0.21 | 14.80±0.61 | |
| EM [Zhou et al., 2022a] | 88.67±0.08 | 70.64±0.09 | 47.25±0.30 | 20.69±0.53 | |
| SPML Methods EM+APL [Zhou et al., 2022a] | 89.19±0.31 | 70.87±0.23 | 47.78±0.18 | 21.20±0.79 | |
| SMILE [Xu et al., 2022] | 87.31±0.15 | 70.43±0.21 | 47.24±0.17 | 18.61±0.14 | |
| MIME [Liu et al., 2023b] | 89.20±0.16 | 72.92±0.26 | 48.74±0.43 | 21.89±0.35 | |
| GR Loss (ours) | 89.83±0.12 | 73.17±0.36 | 49.08±0.04 | 21.64±0.32 |
4.3 Ablation Study
In order to present an in-depth analysis on how the proposed method improves SPML performance, we conduct thorough ablation study on VOC and COCO and report the results in Table 3. It is observed that using only ˆ k(p; β) results in the greatest improvement, almost a 1% increase compared to using the other two items alone. Utilizing both ˆ k(p; β) and L1, L2, L3 leads to a significant enhancement, with mAP reaching 89.35 and 72.86. Moreover, adding v(p; α) can further elevate the model's performance, with mAP improving to 89.83 and 73.17, respectively. The last row indicates that treating β and α as trainable parameters obtains inferior results, which verifies our analysis.
4.4 Hyperparameter and Distinguishability
The impact of β. We present the results of GR Loss with different β(T) in Figure (2a). To facilitate the analysis, we adjust the threshold τ =-b/w while keeping b(T) fixed at -2, -5, and -8, respectively. The results indicates that our model consistently achieves the optimal when τ =1, achieving a best mAP at 89.83 with b=-2. As the threshold decreasing, the performance gradually declines. However, it is still superior to the baseline, i.e., vanilla AN loss (the red dotted line).
The impact of α. We present GR Loss performance with different α(T ) in Figure (2b). By fixing σ(T ) at 0.1, 0.5, and 1, and varying the mean µ from 0.2 to 1, all the three curves peak at 0.8. Moreover, the mAPs for all parameter combinations are above 89 and mostly surpass both EM loss and AN loss, which indicating our method enjoys promising robustness.
| Methods | VOC | COCO | ||
|---|---|---|---|---|
| AN loss (baseline) | 85.89 | 64.92 | ||
| ˆ k(p; β) | v(p; α) | L1,L2,L3 | mAP (%) | |
| ✓ | ✗ | ✗ | 88.31 | 70.41 |
| ✗ | ✓ | ✗ | 87.32 | 69.17 |
| ✗ | ✗ | ✓ | 87.48 | 69.96 |
| ✓ | ✓ | ✗ | 88.11 | 70.48 |
| ✓ | ✗ | ✓ | 89.35 | 72.86 |
| ✗ | ✓ | ✓ | 87.83 | 70.20 |
| ✓ | ✓ | ✓ | 89.83 | 73.17 |
| ◦ | ◦ | ✓ | 89.03 | 71.84 |
The impact of q. Due to the rare correctness of the label with s = 1, we set q1 =0.01 to enhance convergence speed and instead adjust q2 and q3. The result in Figure (2c) indicates that our model achieves the best mAP per epoch on the validation set when q2 =0.01, q3 =1. However, if we increase the value of q2 or decrease the value of q3, the model's performance deteriorates. This suggests that for missing labels, GR Loss has a higher tolerance for false positive noise. Setting q2=0.01 will reduce robustness but can accelerate the learn-


- (a) The Wasserstein distance for GR Loss and AN loss at each epoch on VOC.
- (b) The confidence distribution for the predicted missing labels at the epochs where AN loss performs best.
- (c) The confidence distribution for the predicted missing labels at the epochs where our GR Loss performs best.


Figure 3: Distinguishing ability of model predictions on VOC. Better viewed on screen and in color.
ing by offering large gradient. In contrast, the model has less tolerance for false negative noise, hence setting q3 =1 where L3 becomes the MAE loss, allows greater accommodation of false negative noise. Additionally, this setting reduces the learning effectiveness on negative supervision due to small gradient, thereby addressing the issue of imbalance between positive and negative samples.
Distinguishability of model predictions. A model with promising generalization should be capable of producing informative predictions for unannotated labels, i.e., the predicted probabilities for positive and negative labels should be clearly distinguishable for unannotated labels. Hence, we assess the confidence of model's predictions on validation set for all classes and all unannotated labels at each epoch, and quantitatively measure the confidence distribution difference for positive and negative labels with Wasserstein distance. As shown in Figure (3a), the Wasserstein distances for GR Loss are much greater than those for AN loss at each epoch. Besides, as shown in Figure (3c), compared with AN loss in Figure (3b), for GR Loss, the confidence of negative labels is concentrated around 0.2, while that of positive labels is evenly distributed and becomes extremely low when smaller than 0.9, mainly clustering around 1.
5 Conclusion
In this paper, we proposed a novel loss function called Generalized Robust Loss (GR Loss) for SPML. We employ a soft pseudo-labeling mechanism to compensate the lack of labels. Meanwhile, we specifically design a robust loss to cope with the noise introduced in pseudo labels. In addition, we introduce two tunable functions ˆ k(·; β) and v(·; α) to calibrate model output p, so that to simultaneously deal with the intraclass and inter-class imbalance. Furthermore, we demonstrate the validity of GR Loss from both theoretical and experimental perspectives. From the theoretical perspective, we derive empirical risk estimates for SPML and perform a gradient analysis. Besides, in experiments, our method mostly achieves state-of-the-art results on all four benchmarks.
Nevertheless, there are three limitations in our study. First, q1, q2, q3 in the robust loss Eq.(14) are all set as static hyperparameters. However, it is better to set the robust loss to be dynamically varying so that to balance the fitting ability and robustness. Second, as mentioned earlier, we did not evaluate our GR Loss on different backbones to fully validate its effectiveness. Third, our method does not consider the correlation among classes, resulting in an inferior performance compared with MIME on CUB. To overcome these limitations will shed light on the promising improvement of our current work.
Acknowledgements
This work was supported by National Natural Science Foundation of China (No.61702099), UIBE Excellent Young Scholar Project (21YQ10), and Scientific Research Laboratory of AI Technology and Applications, UIBE.
Contribution Statement
The joint first authors Yanxi Chen and Chunxiao Li are equally contributed to formulation and experiments. In addition, Xinyang Dai, Jinhuan Li, Weiyu Sun, Yiming Wang, Renyuan Zhang, and Tinghe Zhang are equally contributed to writing and experiments. Authors are sorted alphabetically.
@inproceedings{ijcai2024p423,
title = {Boosting Single Positive Multi-label Classification with Generalized Robust Loss},
author = {Chen, Yanxi and Li, Chunxiao and Dai, Xinyang and Li, Jinhuan and Sun, Weiyu and Wang, Yiming and Zhang, Renyuan and Zhang, Tinghe and Wang, Bo},
booktitle = {Proceedings of the Thirty-Third International Joint Conference on
Artificial Intelligence, {IJCAI-24}},
publisher = {International Joint Conferences on Artificial Intelligence Organization},
editor = {Kate Larson},
pages = {3825--3833},
year = {2024},
month = {8},
note = {Main Track},
doi = {10.24963/ijcai.2024/423},
url = {https://doi.org/10.24963/ijcai.2024/423},
}
TY - CPAPER TI - Boosting Single Positive Multi-label Classification with Generalized Robust Loss AU - Chen, Yanxi AU - Li, Chunxiao AU - Dai, Xinyang AU - Li, Jinhuan AU - Sun, Weiyu AU - Wang, Yiming AU - Zhang, Renyuan AU - Zhang, Tinghe AU - Wang, Bo T2 - Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI-24 PB - International Joint Conferences on Artificial Intelligence Organization PY - 2024 SP - 3825 EP - 3833 DO - 10.24963/ijcai.2024/423 UR - https://doi.org/10.24963/ijcai.2024/423 ER -Copied
Machine-readable versions of this paper, for tools and AI agents: