SCMT: Self-Correction Mean Teacher for Semi-supervised Object Detection

1 Introduction

Supervised object detection requires annotations with accurate class labels and bounding box coordinates, which is expensive and time-consuming. Semi-supervised learning is one of the most dominant approaches for solving this issue, which leverages the large-scale unlabeled data to improve the performance of the detector trained on a small amount of labeled data. In this work, we focus on semi-supervised object detection to reduce the efforts of box-level annotations while improving the performance of supervised object detection.

Recently, most semi-supervised object detection methods are based on pseudo-labeling strategy [Sohn et al., 2020c; Liu et al., 2021], where a teacher-student framework are utilized. Specifically, a teacher model is firstly trained on the labeled data and then used to generate pseudo-labels for the unlabeled data. After that, the unlabeled data with pseudo-labels is combined with labeled data to train the student model. In order to obtain stable and reliable pseudo-labels, the exponential moving average (EMA) [Liu et al., 2021; Xu et al., 2021; Tang et al., 2021] is adopted to gradually update the teacher model. However, the final performance is still limited since noisy pseudo-labels inherently exist in the learning process.

To improve the performance of semi-supervised object detection, it is natural to ask: what causes the inherently noisy pseudo-labels in the teacher-student framework? Note that in this framework, the pseudo-labels are selected from the prediction results of teacher based on their corresponding classification score with a single threshold. In semisupervised classification, a higher filtering threshold is commonly adopted to ensure high-quality pseudo-labels. However, it is not applicable in SSOD for two reasons: (i) The single threshold cannot guarantee both precision and recall for foreground pseudo-labels. In object detection, a training image usually contains multiple foreground instances, and the detector needs to distinguish the foreground from the background and classify the foreground into specific categories. A higher threshold can guarantee the high precision of foreground pseudo-labels, but introduce missing annotations for some foreground on the same image. Conversely, a lower threshold will alleviate the issue of missing annotations, but more wrong annotations will be introduced simultaneously. Therefore, no matter how the single threshold is set, the noise problem cannot be avoided in pseudo-labeling. (ii) The classification score cannot reflect the quality of bounding box coordinates. Object detection is formulated as a multitask learning problem involving both classification and localization, but the objectives of these two tasks are different. Specifically, the proposals are treated equally in classification learning if their IoUs with any ground-truth box are larger than a lower bound threshold, while the localization learning seeks for the proposal that can best fit the ground-truth bounding box. This phenomenon is referred to as misalignment between classification and localization in [Jiang et al., 2018], where the bounding boxes with higher classification scores may have poor localization quality, leading to the inaccurate bounding box coordinates in pseudo-labels. As a result, only relying on a high threshold to select pseudo-labels will inevitably cause the existence of noise in SSOD, as shown in Figure 1a.

In this paper, we propose a novel Self-Correction Mean Teacher (SCMT) framework to alleviate the noisy pseudolabels problem in SSOD. The motivation is intuitive, which is to dynamically adjust the loss weights of box candidates in student's classification and regression learning, and assign smaller loss weights to possible noisy box candidates. The dynamic weights are illustrated in Figure 1b. We observe that the network can easily learn simple box candidates with high confidence, and tend to make uncertain predictions on difficult and noisy box candidates. In light of this, the network outputs are used as proxy indicators to estimate confidence scores for box candidates. Specifically, for each student-generated box candidate, we define the confidence score as a combination of its corresponding localization accuracy and classification score from teacher's network outputs. The confidence score is served as the dynamic loss weight for student-generated box candidates, where the box candidate with a higher confidence score will have a greater loss weight. As the confidence score is generated by teacher and can be regarded as the self-correction value of teacher-generated pseudo-labels, we name our method as Self-Correction Mean Teacher.

We highlight the contributions of this paper as follows:

• For semi-supervised object detection, we first investigate the limitations of existing methods based on teacherstudent framework, where the noise inherently exist in pseudo-labels, and then we propose a simple yet effective Self-Correction Mean Teacher (SCMT) framework.
• In SCMT, the losses of student-generated box candidates are dynamically re-weighted with confidence scores estimated by the teacher, which aims to reduce the adverse effects of noisy box candidates.
• Our SCMT achieves state-of-the-art performance on the MS-COCO dataset under both the COCO-standard and COCO-additional settings, which demonstrates the effectiveness of our proposed framework.

2 RelatedWork

Semi-supervised Learning in Classification. Significant progress has been made in semi-supervised image classification. The majority of the recent work can be roughly categorized into pseudo-labeling and consistency regularization. The main objective of pseudo-labeling methods is to generate pseudo-labels for unlabeled data with a model trained on labeled data. [Lee and others, 2013] firstly introduces pseudolabeling in semi-supervised learning, the pseudo-labels are generated from predictions of a trained neural network. Recently, Noisy Student [Xie et al., 2020] demonstrates that pseudo-labeling can further improve the performance on ImageNet with an additional large amount of unlabeled data. Consistency regularization methods [Sajjadi et al., 2016; Tarvainen and Valpola, 2017] utilize the assumption that no matter what perturbations are applied, the model should keep the consistent output for the same unlabeled data. [Sajjadi et al., 2016] introduces random dropout and random data augmentation as input perturbations. Several approaches combine ideas from both pseudo-labeling and consistency regularization [Berthelot et al., 2019b; Berthelot et al., 2019a; Sohn et al., 2020a]. In [Sohn et al., 2020a], pseudo-labels are generated by the teacher training on weakly augmented images, and the student trains on strongly augmented images to learn consistent predictions over pseudo-labels.

Semi-supervised Learning in Object Detection. Currently, SSOD has developed rapidly, inspired by semi-supervised image classification. State-of-the-art results of SSOD have been established by combining pseudo-labeling and consistency regularization. STAC [Sohn et al., 2020c] proposes to use weakly augmented images for pseudo-labeling and strongly augmented images for model consistency training. However, the pseudo-labels are generated only once and are fixed throughout the rest of training, which seriously limits the performance of SSOD. To alleviate this issue, [Liu et al., 2021; Xu et al., 2021; Tang et al., 2021] introduce mean teacher to generate pseudo-labels from teacher and train the student simultaneously, while the teacher is gradually updated by the student. The quality of pseudo-labels is crucial to pseudo-labeling methods. Unbiased-Teacher [Liu et al., 2021] introduces a class-balance loss to address the pseudolabeling bias problem caused by class-imbalance. The current state-of-the-art method Softer-Teacher [Xu et al., 2021] adopts a high filtering threshold to guarantee a high precision of the positive pseudo-labels, and proposes using a reliability measure to weight the loss of each background box candidate to address the problem of missing annotations. When compared with Softer Teacher, we propose to re-weight losses for both positive and negative samples from both RPN and RCNN stages, where the loss weight is dynamically obtained based on the classification score and localization accuracy.

3 Method

For semi-supervised object detection, we are given a set of labeled data Ds = {xsi , ys i }Ns i=1 and a set of unlabeled data Du = {xui }Nu i=1 , where xi and yi denote the i-th sample and its corresponding label, Ns and Nu represent the numbers of supervised and unsupervised samples. Our goal is to leverage the unlabeled data to exceed the performance of models trained only using labeled data. For the rest of this section, we first review a widely used teacher-student framework, then introduce our proposed improvement based on this framework.

3.1 Teacher-Student Framework

Teacher-student is a typical semi-supervised object detection framework, which includes two networks: teacher network and student network. Both networks use the same architecture, but are in charge for different mission.

Teacher network is firstly obtained by training on labeled data Ds using supervised loss Ls as in Faster RCNN [Ren et al., 2015], which is constructed by four components: the RPN classification loss Lrpn cls , the RPN regression loss Lrpn reg , the RCNN classification loss Lroi cls and the RCNN regression loss Lroi reg ,

Then, the parameters of student network are initialed by the parameters of the teacher network. To address the lack of annotations for unsupervised data, motivated by FixMatch [Sohn et al., 2020b], a weak-strong data augmentation strategy and pseudo-labeling method are adopted. Specifically, given an unlabeled sample xui , the prediction result of teacher model are first obtained based on a weakly augmented version of xui , then after applying non-maximum suppression (NMS) and confidence-based box filtering, we acquire the pseudo annotation ˆyu i for xui . Using ˆyu i as pseudo-labels, we compute the classification loss and regression loss based on the student's output for a strongly augmented version of xui . The unsupervised loss in this step is computed as follows:

Finally, the objective function of the student network is composed of a supervised loss Ls applied on labeled data and an unsupervised loss Lu applied on unlabeled data, which can be written as follows:

where λu is a fixed scalar to control the influence of the unsupervised loss, and we set λu = 4 for all experiments setting.

In teacher-student framework, the parameters of teacher is obtained using the supervised loss as Eq. (1) and fixed for the training of student.To obtain more stable pseudo-labels, [Liu et al., 2021] introduces the mean teacher framework to further gradually refine the parameters of teacher network θt as an exponential moving average(EMA) of the parameters of student network θs,θt ← αθt + (1 - α)θs, (4)

where α is a smoothing coefficient hyper-parameter, which is usually set to 0.999.

3.2 Self-CorrectionMean Teacher Framework

The quality of pseudo-labels is important for mean teacher based semi-supervised object detection. However, as discussed in Section 1, noise inherently exists in pseudo-labels, which may not only lead to poor classification generalization capability, but also harm the bounding box regression ability. In this paper, we propose SCMT to deal with the above issues from both classification and regression learning, the overall framework is illustrated in Figure 2.

Self-Correction Weight for Noise-aware Classification. For two-stage object detector, the classification learning is involved in both RPN and RCNN stages, where the box candidates are assigned labels according to their overlaps with ground-truth labels. As discussed in Section 1, the noisy pseudo-labels are inevitable in SSOD, which will cause the box candidates being assigned with wrong labels and harm the classification learning.

Therefore, we propose to reduce the impact of box candidates with wrong labels by dynamically re-weighting their classification losses, where we introduce a confidence score to quantify the importance of candidate boxes by taking into account the classification score as well as the localization accuracy. In specific, for student-generated box candidates classification, the prediction results with high confidence of these candidates are likely to be correct, and those with low confidence are usually confused and uncertain ones. While for student-generated box candidate localization, the localization accuracy can be represented as IoU between the region proposal and its corresponding pseudo bounding box, where the box candidate with higher IoU is more accurate. Thus, we propose to define the confidence score of each box candidate as the product of its classification score and localization accuracy. Motivated by this observation, for each studentgenerated box candidate, we use its corresponding confidence score from teacher as the dynamic loss weight, and greater loss weights will be assigned to box candidates with high confidence. Here we use teacher to estimate confidence scores of box candidates, because the prediction results of teacher are more stable than those of student. Contrary to the widely used hard sample mining approaches, e.g. Focal Loss [Lin et al., 2017b], where the loss is designed to down-weight the contributions of the well-classified samples. The proposed self-correction weight pays more attention to the learning of convincing samples, as we argue that the convincing samples are usually the important samples in classification. The selfcorrection weight can be easily obtained based on the mean teacher framework. However, in the RPN and RCNN stage, the processes of generating weights are different, and the details are described below.

In the RPN stage, the strongly augmented unlabeled image used in student is simultaneously put through the teacher's RPN head to obtain the classification score and box coordinates of each anchor's corresponding region proposal, as shown in Figure 2. Note that the anchor set is the same for the teacher and the student. Therefore, the RPN classification self-correction weight of each student's anchor can be formulated as:

Here, ai and pi denote the i-th candidate anchor and its logit output generated by teacher, respectively. Apos and Aneg represent positive and negative candidate anchor sets, separately. Besides, IoU( ˆ ai, gi) denotes the IoU between the region proposal ˆ ai from the teacher with respect to anchor ai and its corresponding pseudo bounding box gi.

In the RCNN stage, each student-generated region proposal is also input into the teacher's RCNN head to obtain the classification score and box candidates of the refined region proposal. Since the RCNN needs to classify region proposals into specific categories, which is a multi-class classification task. For each region proposal, the classification score under the corresponding category based on the assigned pseudo-labels is used to calculate the self-correcting weight. We formulate the RCNN classification self-correction weight as:

Here bcj is the j-th candidate proposal assigned to the c-th category based on pseudo-labels, pcj denotes the logit output of the proposal with respect to the c-th category, which is generated by teacher. Bpos and Bneg represent positive and negative candidate proposal sets, separately. Besides, C represents the total number of foreground categories, and C + 1-th class denotes the background category. In addition, IoU( ˆ bcj , gj ) denotes the IoU between the refined region proposal ˆ bcj from the teacher with respect to bcj and its corresponding pseudo bounding box gj .

Self-Correction Weight for Classification-aware Regression. As same in classification, regression learning is also needed in both RPN and RCNN stages, but the regression learning in RCNN stage takes a more important role than in RPN stage. The difference from classification is that the regression learning is performed only for positive box candidates with foreground categories. We argue that wrong labels of box candidates will also damage the localization ability, because they may lead to unstable regression loss. For example, given an actual background box candidate, which is assigned as foreground class by incorrect pseudo-labels, the regression branch will continue to learn offsets of this candidate mistakenly.

To tackle the problem of unstable regression, we introduce self-correction weight to adjust loss weights of the noisy box candidates in the regression learning. The self-correction weight is only adopted in the RCNN stage, since the fine regression learning in the RCNN stage is more important for accurate localization. The RCNN regression self-correction weight is the same as the RCNN classification self-correction weight, and can be formulated as:

The proposed RCNN regression self-correction weight can not only alleviate the impact of noisy pseudo-labels in the regression learning, but also promote the consistency of regression and classification learning, as the regression branch allocates more attention for the candidate proposals with high classification scores.

Overall Loss Optimization. In the self-correction mean teacher framework, the overall loss is composed of supervised loss from the labeled images and unsupervised loss from the unlabeled images. The supervised loss is the same as which in the teacher-student framework. The unsupervised loss is modified with the self-correction weight, and can be formulated as:

4 Experiments

4.1 Dataset and Evaluation

We evaluate our approach on the large-scale dataset MSCOCO [Lin et al., 2014]. For MS-COCO, we use version 2017 in all experiments, including train2017, unlabeled2017 and val2017. train2017 set has a total of 118k labeled images, unlabeled2017 set contains 123k unlabeled images and val2017 set has 5k images in total. For a fair comparison, we follow STAC to validate the performance on COCO-standard and COCO-additional settings. For COCO-standard setting, we set up three different proportions: 1%, 5%, and 10% to sample images from train2017 as the labeled training data, and the remaining unsampled images are used as the unlabeled data. For COCO-additional setting, we use the fully train2017 as the labeled data and the additional unlabeled2017 as the unlabeled data. Besides, val2017 set is used to evaluate our approach on both settings.

4.2 Implementation Details

We implement our SCMT framework based on the MMDetection toolbox [Chen et al., 2019]. To get a fair comparison, we follow STAC to use Faster R-CNN with ResNet-50 backbone and FPN [Lin et al., 2017a] as the default base model. The filtering threshold is one of the main hyper-parameters in our SCMT framework, and is set to 0.7 throughout our experiments. Since the amount of training data of COCO-standard setting and COCO-additional setting has large differences, the training schedules are slightly different. For both settings, the initial learning rate is set to 0.01. For COCO-standard setting, the learning rate decays by 10× at 120k and 160k, with a total training step of 180k. For COCO-additional setting, the learning rate decays by 10× at 240k and 320k, with a total training step of 360k. All models are trained with batch size of 32, and the other training hyper-parameters are the same as standard Faster R-CNN. As for the data augmentation, we follow Softer-Teacher to use different data augmentation for labeled image training and unlabeled image training. Specifically, we apply scale jitter and color jitter for weak augmentation and additional geometric augmentations and cutout patches for strong augmentations.

4.3 Overall Performance

In this section, we compare our approach with previous stateof-the-art SSOD methods. As shown in Table 1, our SCMT

performs favorably against other methods under all settings, including various COCO-standard setting and COCOadditional setting. Specifically, for the 1% protocol, SCMT improves the mAP of Softer-Teacher from 20.46% to 23.09%, which achieves 2.63% mAP improvement; for the 10% protocol, SCMT improves the mAP of Softer-Teacher from 34.04% to 35.42%, which achieves 1.38% mAP improvement. It is worth noting that our SCMT trained on 5% labeled data achieves 32.14% mAP, which is even higher than most of other SSOD methods trained on 10% labeled data. SCMT consistently outperforms Supervised Baseline by more than 11% absolute mAP under all 1%, 5% and 10% protocols, and as there is less labeled data, the improvement becomes greater. COCO-additional setting is more challenging. Our SCMT still achieves 4.93% improvement in mAP over Supervised Baseline and about 1.16% mAP improvement under the high benchmark (Softer-Teacher) of 41.40% mAP. The result demonstrates that our proposed SCMT can further improve the performance of a well-trained object detector on a largescale labeled dataset by using more unlabeled data.

4.4 Ablation Studies

We validate the effectiveness of our key designs in this section. All the ablation experiments are performed on COCOstandard setting with 10% labeled data. Due to the limitation of machine resources, we train the model with a total training step of 90k, and the learning rate decays by 10× at 60k and 80k, which is different from the setting used in Table 1.

Effectiveness of different components: We ablate each component of the proposed SCMT step by step. Table 2 reports the results. The basic framework Mean Teacher described in Section 3.1 is adopted as our semi-supervised baseline, which achieves 31.9% mAP. As shown in Table 2, the RPN classification self-correction weight and RCNN classification self-correction weight improve the mAP of baseline by 0.4% and 1.0%, respectively. Furthermore, the combination of these two weights greatly outperforms the baseline by 1.5% mAP. This gain mainly comes from reducing the classification problems caused by noisy pseudo-labels, including missing detection and background false detection. Besides, when applying the RCNN regression self-correction weight, the performance reaches 33.8% mAP, which improves the baseline by 1.9%. We further investigate the effects of RPN classification self-correction weight. From Table 3, we observe that the RPN classification self-correction weight makes an improvement of 0.8%, 0.3% and 0.2% for ARs, ARm andARl , respectively. Note that larger improvements are achieved for ARs, because the poor features of small objects cause more noisy pseudo-labels, and the selfcorrection weight can avoid the impact of this issue.

Effects of other hyper-parameters: We study the effects of different filtering thresholds for generating pseudo-labels. Table 4 reports the results. The best performance is achieved when the filtering threshold is set to 0.7. And we find that a too high threshold is not the optimal choice, which may introduce more missing annotations in pseudo-labels, resulting in hindering the training and limiting the performance of the object detector.

4.5 Further Investigation

We further investigate the superiority of SCMT over other methods by comparing the error type analysis generated by the TIDE toolbox [Bolya et al., 2020] in Figure 3, and the qualitative results of GradCAM [Selvaraju et al., 2017] in Figure 4. The analysis reveals more granular information about where SCMT improves mAP relative to other methods. We observe that the Supervised-Baseline has a very serious miss error problem, and the learned features cannot focus on the foreground area, which indicates the difficulty of object detection with a small amount of labeled data. Surprisingly, SCMT significantly closes the gap between semisupervised object detection and fully supervised object detection. Both the classification error and localization error can be well solved, thus leading to better mAP. Besides, we find that SCMT concentrates more on the foreground area, which can encourage the network to extract informative and meaningful features as shown in Figure 4.

5 Conclusion

In this paper, we propose a simple yet effective training framework for semi-supervised object detection, which can dynamically re-weight the unsupervised loss to mitigate the effects of noisy pseudo-labels in both classification and regression. Based on the teacher-student framework in SSOD, we first generate the pseudo-labels by teacher model, then obtain the confidence score combining classification score and localization accuracy from teacher model for each student's proposal as the dynamic loss weight, which is referred to as self-correction weight. Namely, we use the loss weights provided by the teacher model to alleviate the side effect of the noisy pseudo-labels from the teacher model. The proposed method outperforms state-of-the-art methods by a large margin on MS-COCO benchmark under both partially labeled data and fully labeled data settings.