CARD: Semi-supervised Semantic Segmentation via Class-agnostic Relation based Denoising

1 Introduction

The semantic segmentation model learns to predict pixel-wise categorical labels given an unseen image [Chen et al., 2017]. However, its performance relies on massive pixel-level annotations, where the labeling process can be laborious and timeconsuming. To tackle this limitation, semi-supervised semantic segmentation is studied to facilitate learning in a low data regime by leveraging a fraction of fully labeled images and massive unlabeled samples to achieve comparable results to the fully-supervised setting.

The key in semi-supervised semantic segmentation lies in mining extra supervision from unlabeled data. Motivated by advances in semi-supervised classification [Berthelot et al., 2019; Sohn et al., 2020; Zhang et al., 2021], recent semi-supervised segmentation favors pseudo labeling [Chen et al., 2020] and consistency regularization [Mittal et al., 2019]. Their hybrid methods have achieved state-of-theart performance [Chen et al., 2021]. Recent works mainly enforce prediction consistency on differently perturbed unlabeled data [French et al., 2020] and meanwhile conduct self-supervision by generating pseudo labels. However, few works have paid attention on the problem of label noise in this process. In the early stage of training, the weak classifier cannot guarantee high quality pseudo supervision and the misleading signals can hardly be corrected by the knowledge of the model itself, which leads to the confirmation bias [Arazo et al., 2020]

∗Corresponding author

In this work, we propose an effective pseudo label denoising approach to achieve robust semi-supervised semantic segmentation. The motivation comes from the smoothness assumption [Luo et al., 2018] that two near samples should have close corresponding predictions. Since the weak classifier trained on limited data can produce a decent amount of high-quality predictions, we assume this knowledge can be transferred to those noisy regions to re-calibrate their predictions, based on a reasonable "closeness" estimation between samples. Given a wrongly predicted feature embedding, once its semantic connections to plenty of features with correct labels are known, it is possible to conduct weighted prediction ensemble to re-estimate its prediction using its semantic neighbors. The key to such an approach relies on reliable feature relations. One straightforward way is to leverage features from the segmentation encoder to estimate their relations based on a pre-defined distance metric (e.g., L2 distance). However, such direct relation estimations are inaccurate, as shown in Fig. 1(c) where the wrongly predicted pixel (blue star) only activates its geometric but not semantic neighbors, which prevents effective label correction.

To obtain reliable feature relations, we propose a classagnostic relation network (CARNet) to enhance the relation estimation. Given features from the segmentation encoder, the proposed module maps these features into a new metric space where the feature relation is supervised in a classagnostic way. Reliable relation estimation can be obtained through CARNet as shown in Fig. 1(d), where strong connections are built between the target feature to all its semantic neighbors. The noisy label can be corrected by transferring knowledge from plenty of reliable features with such connections. Class-agnostic supervision is the key for the CARNet, which is extracted from categorical annotations with no cost. Such supervision only focuses on the class equivalence between features, which primarily alleviates the bias caused by unbalanced classes. Therefore, it enables CARNet to provide reliable feature relation estimations. With refined feature relations, effective label denoising in self-training can be achieved. Furthermore, a progressive denoising strategy is proposed to control the pace of denoising during training in order to achieve stable and efficient label correction.

Our approach achieves state-of-the-art performances on Pascal VOC and Cityscapes under various semi-supervised settings. Note that the proposed framework is solely based on label denoising and does not contain any form of strong data augmentation, while previous works highly relied on this. Our contributions are summarized as follows:

• We propose a pseudo label denoising routine by transferring knowledge from reliable predictions to noisy labels through feature relations.
• We propose a module named CARNet to online refine feature relation estimation by enhancing their intra-class connections.
• We propose a progressive denoising strategy to conduct stable and efficient self-training. The framework achieves state-of-the-art performances.

2 RelatedWorks

2.1 Semi-supervised Semantic Segmentation

Early works on semi-supervised semantic segmentation apply generative models and adversarial training to generate high-quality pseudo labels [Hung et al., 2018; Mittal et al.,

2019]. Recent works pay attention to consistency regularization between predictions on different views of unlabeled data. French et al. [French et al., 2020] introduces strong data augmentations of patch-level CutMix on unlabeled data to create randomly mixed versions of images. GCT [Ke et al., 2020] introduces a multi-encoder system to enforce consistency among predictions of different encoders under featurelevel perturbations. C 3 -SemiSeg [Zhou et al., 2021] creates region-level data augmentation to minimize the feature discrepancy between labeled and unlabeled data. CPS [Chen et al., 2021] applies two differently initialized models in selftraining to conduct cross pseudo supervision. Note that the noisy pseudo label problem is not well studied in previous works, which is the main focus of this paper.

2.2 Learning with Noisy Labels

Our approach is closely related to the topic of learning from noisy labels. Recent works mainly focus on mining reliable supervisions according to model disagreement. Coteaching [Han et al., 2018] trains two models simultaneously, and each model selects its small-loss samples which are used to train another model. Dual-student [Ke et al., 2019] follows a similar path and adopts the idea to the field of semisupervised classification. DMT [Feng et al., 2020] leverages two differently initialized models to let each model generate offline pseudo labels for the other one. Inter-model disagreement is used to re-weight training loss to achieve noise-robust training. Unlike the above methods that try to bypass noisy supervision, our approach aims to directly correct those noisy labels based on the knowledge of feature relations.

3 Methodology

3.1 Overview

The proposed method aims to re-calibrate noisy pseudo labels by aggregating knowledge from their corresponding semantic neighbors. After conducting a warm-up on the annotated set, a student-teacher-CARNet framework is built to conduct selftraining on unlabeled images. The teacher model is the exponential moving average (EMA) of the student network. The whole process of the self-training stage is shown in Fig. 2, which can be divided into the following parts: (1) Given predictions from the teacher model, a class-wise confidence bank is built and updated to achieve online label reliability estimation. It dynamically identifies both potentially noisy regions and high-quality predictions. (2) The CARNet after warmup is continuously supervised by relation labels from reliable teacher predictions to further boost accuracy of feature relation estimation. (3) Based on the relation estimation, label denoising is conducted progressively to provide enhanced supervision for the student model.

3.2 Class-balanced Label Reliability Estimation

Since our goal is to correct noisy labels with reliable predictions, the first thing to consider is how label quality is measured. In this work, to set the criterion, we follow [Sohn et al., 2020] to simply adopt prediction confidence as reliability indicator. Unlike previous work that sets a single fixed threshold for all classes, we design dynamic categorical thresholds, considering the evolved model knowledge and the varied learning difficulties among classes. Class-wise confidence banks are built to online track the model performance on each class. For class k, a bank N k collects the softmax probability from instances that are predicted as k on unlabeled data. The bank is updated with latest predictions during training as a queue and maintains a sufficient size to estimate the confidence distribution on class k. Then ηkα, the αth quantile of N k , can be used as the confidence threshold to determine whether a prediction is reliable. The per class threshold can guarantee class-balanced reliability estimation. By setting a low threshold ηk low and a high threshold ηkhigh , we can determine both confident and uncertain regions to conduct knowledge transfer as shown in Fig. 3.

3.3 Class Agnostic Feature RelationMining

CARNet is proposed to measure the closeness of any two feature vectors, which serves for denoising. We exploit relation supervisions from the provided annotations and predicted pseudo labels to obtain accurate feature relation estimation. Reliable relation estimations can be obtained by performing distance calculations from its output feature maps.

Training CARNet from Annotations. In warm-up training, both segmentation model and CARNet are trained on provided annotations. To provide class-agnostic supervision, we directly transform a one-hot coded K class annotation map of size H × W × K into a HW × HW relation matrix G as in Eq. (1).

where each element gij ∈ G indicates class equivalence between features of pixel i and j. Value 1 is given if they share the same class annotation and 0 otherwise. Under the binary supervision G, we leverage cosine similarity between CARNet feature pair fi and fj to estimate feature relations:

Then aij is processed by a sigmoid function, outputting ˆaij, which is used in loss calculation. We gather positive feature pairs in set Zp and negative pairs in Zn as

The CARNet learns to minimize the cross-entropy loss Lcar between predicted semantic relation ˆaij and relation labels gij, as shown in Eq. (4):

Training CARNet from Mined Reliable Continuous Supervision. Supervision fromannotations has brought CARNet decent performance. However, it is sub-optimal to stop training CARNet during training the segmentation model on unlabeled data. Since the CARNet is trained on features from segmentation encoder, it is beneficial to be continuously adapted to the segmentation encoder which evolves during training. More importantly, knowledge from segmentation model on unlabeled data can be transferred into relation supervision to expand training samples for CARNet. To mine reliable relations, we adopt class-wise confidence bank from section 3.2 and set the threshold η(t,k) high for class k at time t as the median value of current confidence bank N (t,k) :

During training, a categorical prediction on unlabeled pixel is considered reliable only if its class probability is over the current threshold η(t,k) high . With filtered predictions, two sets Z u p and Z u n are built the same way as in Eq. (3) to gather reliable prediction pairs on unlabeled pixels. The dynamically updated relation supervision is used to continuously supervise CARNet by minimizing Lucar :

3.4 Progressive Pseudo Label Online Denoising

We conduct online label corrections during training on unlabeled data. To perform stable and efficient denoising, we propose a strategy to progressively correct noisy regions in a class-balanced way while keeping high-quality labels as anchor supervision. By adopting prediction confidence as a noise indicator, the denoising starts from the least confident pseudo labels since they are most likely to be noise and mislead the self-training. The denoising is conducted in a reverse-confidence manner to firstly target on the most noisy labels and then expand target regions progressively towards those with higher confidence.

Class-balanced Denoising Curriculum. Based on the categorical confidence estimation from section 3.2, we design a class-balanced denoising curriculum for label correction. We map current training iteration t to a low threshold η(t,k) low ← η(t,k) g(t) to locate noisy regions. g(t) denotes the mapping, which is calculated as:

where ⌈.⌉ indicates ceil operation and T is total training iterations. b is the upper bound for quantile selection and ranges from 10 to 100. As training proceeds, the noise threshold η(t,k) low starts from η(t,k) 10 and finally reaches the up limit η(t,k) b .

Noisy Targets and Denoising Candidates. We define noisy regions in pseudo labels at step t as Rt in Eq. (8). Label of pixel i whose confidence is below current threshold η(t,k) low needs to be rectified. Meanwhile, we mine confident predictions from current teacher model as shown in Fig. 2(i). The confident prediction set is defined as Qt in Eq. (9). Qt serves as references for label re-estimation of Rt .

Relation based Online Prediction Ensemble. As shown in Fig. 2(ii), labels of Rt are rectified with Qt and CARNet. To correct a label at location i ∈ Rt , we draw interfeature similarities atij fromCARNet to search for the semantic neighbors of the target. Then we turn similarities into normalized weights wt ij to determine contribution of each prediction at location j to correct pseudo labels at i:

Then the pseudo label prediction at location i is corrected by weighted fusion of all its updated semantic neighbours ptj :

The corrected hard labels ˆ yt i are converted from the soft predictions as:

The student model is then trained on the updated pseudo labels ˆ yt i . From the view of consistency regularization, we enforce the student prediction to be consistent with both its temporal ensemble (teacher model EMA) and spatial ensemble (weighted fusion of teacher predictions).

3.5 Loss Functions

Warm up stage. Pixel-wise cross-entropy loss in Eq. (13) is applied on segmentation model, where ya is segmentation label from annotation. Lcar is used to supervise CARNet. The total loss Lw in warm-up stage is shown in Eq. (14):

Self-training stage. We adopt a dynamic cross-entropy loss on outputs si from student model to further alleviate the effect from potential noise. The loss is formulated as

where wt i is the predicted probability to weight the sample loss. More attention is paid on confident samples while less weights are given on uncertain predictions. In this way, the model is updated in a conservative way, which leads to noise robust training. Combined with continuous supervision on CARNet, the overall loss function for unlabeled samples is

4 Experiments

4.1 Model Implementation

Backbone Network. Since the majority of works report results on DeepLabv2 [Chen et al., 2017], we adopt the same model as the base model. We follow the implementation from [Hung et al., 2018] and use ResNet-101 as the backbone encoder.

CARNet Architecture. The architecture of CARNet is shown in Fig. 4. It takes multi-level feature maps from the segmentation encoder to capture comprehensive representations. The first level features are downsampled to half the original size to align with other levels. A 1 × 1 convolutional layer is applied, followed by a batchnorm and ReLU layer, to process level-wise features in parallel. Feature channels are all reduced to 128 before being fed into self-attention, where the feature representations are further enhanced. The enhanced multi-level feature is then concatenated on channel dimension and processed by a final 1 × 1 convolution layer.

4.2 Experimental Setup

Datasets. PASCAL VOC 2012 [Everingham et al., 2010] with SBD annotations [Hariharan et al., 2011] are commonly used benchmark for semantic segmentation. It has 20 foreground classes and a background class. The training set and validation set consist of 10,582 and 1,449 images. The images are randomly cropped to 321 × 321 for training. Cityscapes [Cordts et al., 2016] is a dataset for urban scene understanding. It consists of 30 categories, while only 19 are used in the evaluation. The training set and validation set contain 2,975 and 500 finely annotated images. The original image resolution is 2048 × 1024. We resize it as 1024 × 512 and train the model on cropped 512 × 256 patches following [Hung et al., 2018; French et al., 2020].

Evaluation Protocols. For PASCAL VOC, 1/100, 1/50, 1/20, and 1/8 data splits are created. For Cityscapes, 1/30, 1/8, 1/4 images are randomly sampled from the dataset. The selected images are used as labeled data, while the rest are as an unlabeled set. Three folds are created for each data partition, and the final performance is the average on the three folds on the validation set. Mean intersection over union (mIoU) is adopted as the evaluation metric. We conduct single-scale evaluation without CRF post-processing.

Implementation Details. For both PASCAL VOC and Cityscapes datasets, we use the stochastic gradient descent (SGD) optimizer with learning rate 0.001, weight decay 0.0005 and momentum 0.9. Following the common practice, we use 'poly' learning rate policy where the initial learning rate is scaled by (1 - iter/max iter)0.9 . Among all semisupervised protocols, the warm-up training on labeled sets lasts 10k iterations. We conduct single-stage self-training and train the model on unlabeled images for 40 epochs. The batch size is set to 8 for both training stages. The weight for updating the EMA teacher model is 0.999. For data augmentation, only random horizontal flip is applied. Upper bound b in Eq. (7) for denoising region is set as 60th class-wise quantile.

4.3 Comparisons to State-of-the-art Methods

We compare the proposed method with different semisupervised semantic segmentation methods on PASCAL VOC and Cityscapes, including AdvSemiSeg [Hung et al., 2018], S4GAN+MLMT [Mittal et al., 2019], DMT [Feng et al., 2020], ECS [Mendel et al., 2020], ClassMix [Olsson et al., 2021], Contra-SemiSeg [Alonso et al., 2021], CutMix [French et al., 2020], C 3 -SemiSeg [Zhou et al., 2021] and DARS [He et al., 2021]. For a fair comparison, we mainly show the results on the same backbone, i.e., DeepLabv2 with ResNet-101. All results are reported by their original papers under the same settings.

Results on PASCAL VOC. Table 1 presents the mIoU results on PASCAL VOC validation dataset with COCO [Lin et al., 2014] pretrained weights. The proposed method improves over purely supervised baseline by 16.6%, 15.6%, 10.2%, and 6.1% under 1/100, 1/50, 1/20, and 1/8 data proportions, respectively. On 1/8 data partition, our method achieves comparable performance to fully supervised settings, with a small gap of 0.4%. Our method achieves state-of-the-art performances under all settings compared to the previous works. In particular, our method surpasses the prior best method Contra-SemiSeg by 2.7%, 2.8%, and 2.3% under 1/50, 1/20, and 1/8 protocols, respectively. With ImageNet [Deng et al., 2009] pretrained weights, our method achieves similar performances and outperforms Contra-SemiSeg by 0.4%, 2.4%, and 2.2% under the same data partitions.

Results on Cityscapes. Table 2 shows the quantitative comparison of Cityscapes. Our method outperforms the supervised baseline by a large margin, with gains of 8.8%, 6.0%, and 4.4% under 1/30, 1/8, and 1/4 data proportions. In particular, our method nearly closes the performance gap between 1/4 data partition (66.45%) and oracle (66.82%).

4.4 Ablation Studies

To investigate the contribution of each component, we conduct ablation studies based on 1/50 data split in the PASCAL VOC dataset, and mIoU results are obtained on the validation set. We set two baselines to compare with: the model supervised only by labeled data (denoted as "Supervised" in Table 3) and the model trained in a plain self-training approach on initial pseudo labels (denoted as "ST" in Table 3).

Feature Relation based Denoising. We ablate the framework to manifest the effectiveness of the proposed label denoising approach. In Table 3, dynamic self-weighted loss Luseg in experiment I brings 3.2% gain over plain selftraining. Upon that, in Experiment II, plain feature relation based denoising is introduced, with feature relations calculated from the segmentation encoder. The denoising approach fails, leading to a significant performance drop. The main reason is that inaccurate relation estimation fails to capture semantic connections. In Experiment III, the feature relation estimation from CARNet guides the denoising process to the right path, leading to a performance gain of 7.3% compared to Experiment I, which indicates that the CARNet plays a vital Qt role in the relation-based denoising approach. Our progressive denoising strategy further improves the framework by 0.5% as shown in Experiment IV. During self-training, our method produces more accurate pseudo labels compared to the approach, as shown in Fig. 5.

Figure 6: Ablation study on confidence thresholds. (a) Effect of denoising region Rt . (b) Effect of Qt to mine CARNet supervision.

Denoising Regions. Fig. 6(a) presents the effect of denoising region Rt . When the denoising region increases to 60%, the performance peaks at 70.95% but sees a slight decrease when the denoising region expands further. By observation, the regions that demand correction most are the predictions below the 60% class-wise confidence threshold.

Mining Continuous CARNet Supervision. We mine relation supervision from filtered teacher outputs Qt controlled by ηthigh . Fig. 6(b) shows the effect of threshold choices when generating supervision. The model performance increases when filtered teacher predictions increase from 10% to 50%. Top performance 70.95% is achieved when 50% of teacher predictions are leveraged in CARNet supervision. Overall, our method is insensitive to different confidence thresholds.

5 Conclusion

This paper has proposed a novel semi-supervised semantic segmentation framework based on effective pseudo label denoising. We present a label denoising routine by transferring confident label predictions to rectify the label for noisy regions, through reliable feature semantic connections. Classagnostic relation network (CARNet) is designed to capture reliable feature connections, which is supervised by class equivalence. Extensive experiments demonstrate that our method outperforms the state-of-the-art methods by a significant margin, indicating the effectiveness of our method.

Acknowledgments

This work was suppoted by National Key R&D Program of China under grant SQ2021YFE020328 and National Science Foundation of China under grant 61972323.