Double-Check Soft Teacher for Semi-Supervised Object Detection

1 Introduction

With the development in computing power and the availability of large-scale labeled datasets, deep learning has achieved great breakthroughs in various tasks. However, these advances are accompanied by a large amount of timeconsuming and labor-intensive data annotation efforts, especially for object detection tasks. To make better use of unlabeled data, semi-supervised learning has received increasing attention. However, so far the research progress of semisupervised learning is mainly concentrated in the field of image classification [Berthelot et al., 2019; Sohn et al., 2020a], and there are still many unsolved problems in its application to object detection where the data labeling work is more tedious and is thus more worth investigating. In this paper, a simple but effective solution is proposed by deeply revisiting the essential reasons that the pseudo-label-based semisupervised object detection framework may have excessive noise and low recall rate that seriously restrict the model performance (refer to Figure 1).

∗Corresponding author.

(b) The low recall rate of the pseudo-labeled harms the model training, e.g., small objects (ball and person in the image) will be ignored and misclassified as background.

Figure 1: Low-quality pseudo-labels and their possible detrimental analysis. The solid boxes are the pseudo-labels adopted by the model during training, and the dashed boxes are the candidate object bounding box generated during training.

Pseudo-label-based semi-supervised methods [Liu et al., 2021; Xu et al., 2021] consist the mainstream solution for semi-supervised object detection. Among them, the quality of pseudo-labels is a key factor affecting performance. Current works have carried out a lot of exploration on "generating high-quality pseudo-labels" [Yang et al., 2021; Tang et al., 2021b; Zhou et al., 2021]. However, for object detection tasks, high quality pseudo-label requires both accurate classification, precise location, and high recall rate, which makes "generating high quality pseudo-labels" extremely formidable. In contrast to existing works, we identify the essential reasons for the adverse effects of low-quality pseudo-labels and propose a method to "combat low-quality pseudo-labels". Specifically, conventional pseudo-labels are tailored for classification tasks, and applying them directly to object detection leads to the following two critical problems:

1. The bounding box position corresponding to the pseudolabel is not accurate enough, which may lead to inconsistent semantic content in the model training although the candidate bounding boxes and pseudo-labels have enough overlap, as shown in Figure 1(a).
2. ii) The low recall rate of pseudo-labels will cause a large number of candidate boxes to be mistakenly regarded as background category due to insufficient matching of pseudo-labels to mislead model training, as shown in Figure 1(b).

To solve the above issues, we propose Double-Check Soft Teacher (DCST), an approach that corrects the candidate box labeling errors caused by the low quality pseudo labels, basing on the mainstream Teacher-Student architecture [Sohn et al., 2020a], where the Teacher generates pseudo labels on weakly augmented images to train the Student with strongly augmented images. In particular, our method sends the candidate boxes generated by the student model with reference to the pseudo-labels into the teacher model for further verification, and then assigns probabilistic soft labels (including background) to each candidate box based on features extracted from weakly augmented images. We call this process as "Double Check". This mechanism enables the model to identify the mislabeled candidate boxes and re-label them correctly, which can be seen as the teacher checking the students' homework, correcting the errors while retaining the correct results. Accompanying the "Double Check", candidate boxes with low confidence may appear, which are not suitable to be assigned one-hot hard label, and the perfect soft label hence comes into being. In this way, incorrect background boxes still have a chance to be corrected as foreground (certain prediction distribution of foreground categories) to participate in training, although they are not sure of their exact class.

On the other hand, considering the negative impact of the low recall rate of pseudo-labels on the model performance, we introduce "TOP-K pseudo labeling", which leverages the sum of the cumulative Top-K probability predictions greater than a certain threshold as the selection basis for pseudolabels. In fact, many objects of different categories share similar appearances, such as "sofa" and "chair", "TV" and "laptop". Under the condition of semi-supervised learning with limited labels, it is arduous for the model to generate sufficiently high-confident predictions for a specific category, which makes it easy for the model to miss a large number of high-quality hard foreground objects in the process of selecting pseudo-labels, resulting in an excessively low recall rate. TOP-K pseudo labeling enables the model to detect similar objects and provide more candidate object boxes for the double check processing, which would further improve the low classification accuracy.

To sum up, we highlight the contributions of this paper as follows:

• We identify the unreliable positions and the low recall rate consists of the essential reasons for the adverse effects of low-quality pseudo-labels.
• The novel Double-Check Soft Teacher with TOP-K pseudo labeling approach is proposed to prevent the detrimental effects due to noisy pseudo-labels.
• On the MS-COCO semi-supervised object detection benchmark, our model consistently performs favorably against the state-of-the-art methods by significant margins.

2 RelatedWork

2.1 Semi-supervised Learning

Semi-supervised learning (SSL) aims to make full use of unlabeled data to facilitate the model learning process with limited labeled data. Most of the semi-supervised learning methods aim at image classification tasks. There are two mainstream paradigms of these works, consistency based methods and pseudo label based methods. Consistency based SSL methods introduce a regularization loss to make the model produce similar predictions for the same image under different disturbance. There are many different ways to implement the disturbance, such as applying different kinds of data augmentations for the same image [Laine and Aila, 2017; Tarvainen and Valpola, 2017], or following the idea of adversarial training to perturb the input images along the adversarial direction [Miyato et al., 2018]. Pseudo-labeling based (self-training based) methods [Xie et al., 2020b; Li et al., 2019] refer to generating pseudo labels for unlabeled images and then using them for supervised training. Recently, data augmentations have proven to be an effective paradigm for boosting SSL on image classification [Berthelot et al., 2019; Xie et al., 2020a; Sohn et al., 2020a]. Our work also adopts different data augmentations to achieve consistent regularization, and follows the idea of pseudo labeling to deal with unlabeled data, but we focus on the consistency regularization of bounding-box level prediction.

2.2 Semi-supervised Object Detection

Similar to semi-supervised image classification, semisupervised object detection also follows the two aforementioned algorithm paradigms, including consistency [Jeong et al., 2019; Tang et al., 2021a] based method and pseudo labeling based models [Sohn et al., 2020b; Wang et al., 2021; Xu et al., 2021; Liu et al., 2021; Yang et al., 2021; Tang et al., 2021b; Zhou et al., 2021]. Recently, pseudo labeling based methods have developed rapidly. Among them, STAC [Sohn et al., 2020b] first proposed a paradigm of generating pseudo labels using weakly augmented images and training with strongly augmented images. This method has become a classic benchmark recently. For pseudo labeling based method, the quality of pseudo labels is the key factor affecting the performance. The improvements achieved by the works after STAC [Wang et al., 2021; Xu et al., 2021; Liu et al., 2021; Yang et al., 2021; Tang et al., 2021b; Zhou et al., 2021] are basically to improve the quality of

Unsupervised Branch

Supervised Branch

pseudo labels. However, most of their proposed methods can not well address the inaccurate location and low recall rate of pseudo-labels. Note that [Wang et al., 2021] proposed a data uncertainty based method to detect uncertain candidate boxes and reduce their weights. [Xu et al., 2021] used the background class score generated by the Teacher model as the weight of the background candidate boxes to reduce the ill effects of incorrect background predictions. However, those methods failed to fundamentally solve the mislabeling problem caused by the noisy labels. Our proposed DCST can detect and correct errors in all candidate boxes, fundamentally addressed the harmful effects of insufficient pseudolabel quality.

3 Method

Problem Definition

In the semi-supervised object detection task, a set of labeled images Ds = {xsi , ysi }Ns i=1 and a set of unlabeled images Du = {xui }Nu i=1 are available for training. Ns and Nu denote the number of labeled data and unlabeled data, respectively. For each image xi, the corresponding label yi contains the position and category information of object instances contained in the image. The goal is to make use of both labeled and unlabeled data to learn the object detection model.

Pseudo labeling is one of the most widely adopted algorithms to involve unlabeled data during training. Traditional pseudo-label learning mechanisms designed for image classification cannot ensure both the accuracy of the generated bounding box localization and the high recall rate of box-level pseudo-labels. In order to overcome the hazards caused by low-quality pseudo labels, we propose Double-Check SoftTeacher (DCST), an approach that corrects the candidate box labeling errors caused by the low quality pseudo labels, base on the mainstream Teacher-Student architecture. Furthermore, a new sample filtering scheme called "TOP-K pseudo labeling" which leverages the sum of the cumulative Top-K

probability predictions greater than a certain threshold as the selection basis for pseudo-labels is introduced for improving the recall rate of pseudo labels. The overall pipeline of our method is shown in Figure 2.

3.1 End2End Teacher-Student

The Teacher-Student Mutual Learning framework [Liu et al., 2021] is employed to generate pseudo labels for unlabeled images. During the network optimization procedure, every training batch is composed of several randomly sampled labeled and unlabeled images. Labeled images can be directly used for training in the supervised manner. The weakly and strongly augmented counterparts of unlabeled images are regarded as inputs of the teacher and student model respectively. The outputs of the teacher model are leveraged to construct pseudo labels for unlabeled images, which are subsequently applied for training the student model. We build up the student model with the two-stage object detection network Faster-RCNN, consisting a region proposal subnetwork (RPN) for inferring object proposals and a ROI head for predicting class probabilities and adjusting bounding boxes positions. Denote the parameters of the student model be θs. The teacher model shares the same architecture with the student model. Suppose its parameters be θt, then it is updated via the exponential moving average (EMA),

where α is a constant indicating the ensemble ratio between the teacher and the student.

The student model is trained with the supervised loss on labeled images and the unsupervised loss on unlabeled images respectively. For labeled images, the supervised loss Lsup is consisting of classification and regression losses calculated on outputs of the RPN and ROI head,

For the unlabeled image xui , the detected objects having inference confidence larger than a threshold σ in the teacher model are regarded as pseudo labels ̂ yui . Then the unsupervised loss is calculated with the pseudo labels ̂ yui ,

Finally, the overall objective function for optimizing parameters of the student model is defined as the weighted sum of the supervised loss and the unsupervised loss:

where λ is a constant. The SGD optimizer is adopted to update parameters of the student model. After every optimization step, the teacher model is updated according to Eq. 1.

3.2 Double-CheckMechanism

In the training process of the object detection task, the twostage object detector first generates candidate boxes with RPN. Then, the candidate boxes are assigned with labels according to the provided annotations. For each candidate box, the best matched annotation is regarded as the ground-truth, if the IoU between the candidate box and the annotation is larger than a certain threshold; otherwise, the candidate is marked as background. In the semi-supervised scenario, due to the inaccurate location and low recall rate of pseudo labels, the assigned labels of candidate boxes usually contain a lot of errors, as shown in Figure 1. In order to correct these errors, we propose the double-check mechanism.

The double-check mechanism is ingenious in that the teacher model reloads the position of filtered boxes to extract features from the weakly augmented images through ROI Pooling. For each filtered box, a new reliable category label is generated from the prediction result inferred by the ROI head of the teacher model, as shown by the path constituted by yellow arrows in Figure 2.

If the original category label of the filtered box is correct, the prediction distribution tends to be consistent with the original category label; otherwise, the original category label is corrected by the new reliable category probabilities.

3.3 Soft Teacher

Assuming that there are n types of objects in the data set, the Teacher will generate a prediction distribution of length n + 1 for each filtered box in the double-check mechanism. This prediction distribution represents the probability that the filtered box belongs to each class, including the background. We directly take this distribution as the category soft label to train the student model.

There are two benefits to this approach. First, soft label enhances the double-check correction capability. This enables false background boxes to be corrected as foreground boxes, even if the model cannot determine the specific foreground class. Secondly, soft labels can be utilized for exploring the underlying similarity cues among classes. For example, as shown in Figure 3, the model may not be able to confirm whether the object in the detected box is a TV or a laptop, it is sure that the object does not belong to any other categories. Such kind of boxes can be leveraged in the form of soft labels for training the student model.

With the help of soft labels, the classification loss of ROI in unsupervised loss is calculated with the soft cross-entropy, as shown below:

where Ni is the number of filtered boxes for training in image xui , pj is the prediction logits of the j-th filtered box, and ̂ ycls j is its corresponding soft label.

3.4 TOP-K Pseudo Labeling

The employment of the double-check mechanism relaxes the requirement for pseudo labels, which makes high precision on category no longer important. Meanwhile, the soft label mechanism enhances the error correction ability and enables the model to handle similar-looking objects. Based on these two characteristics, we further propose a TOP-K pseudolabeling strategy to promote the recall rate of pseudo labels.

The conventional pseudo-label screening scheme uses the top-1 score of the foreground classes in the prediction result as the confidence of the detection box. This approach easily overlooks hard samples as shown in Figure 1, leading to low recall rate in pseudo labels. To tackle this problem, we propose a TOP-K pseudo labeling scheme, through thresholding the accumulated value of the top-k scores. Such pseudo labeling strategy is beneficial for preserving similar-looking objects as pseudo-labels, thereby improving the recall rate of pseudo-labels. TOP-K pseudo labeling aggravates the class uncertainty of pseudo-labels, but the double-check mechanism eliminates the negative influence of unreliable pseudo category labels. Meanwhile, the additional look-alike objects brought by TOP-K pseudo labeling can further benefit the learning of the student model under the soft label mechanism.

4 Experiments

Dataset. Following existingworks, we test the performance of our method on MS-COCO benchmark [Lin et al., 2014]. Subsets train2017 and unlabeled2017 are used for training, and subset val2017 is used for testing. Our experiments are divided into two protocols: partially labeled data and fully labeled data. In partially labeled data protocol, we respectively sample 1%, 5% and 10% of train2017 as labeled data, and the remaining is used as unlabeled data. For every setting, five trials are conducted using different random seeds for random data sampling, and we report the averaged evaluation metrics. In the fully labeled data protocol, all images in train2017 are regarded as labeled data, and all images in unlabelled2017 are regarded as unlabeled data. The standard mean average precision (mAP) on val2017 is used as the evaluation metric.

Implementation Details. For fair comparisons, we use Fast-RCNN [Ren et al., 2015] built with ResNet50 [He et al., 2016] and feature pyramid networks [Lin et al., 2017] as the object detector. The code is implemented under the framework of MMdetection [Chen et al., 2019]. Top-3 classes are considered for generating pseudo labels from predictions of the teacher model, and the threshold σ is set to 0.7. Following existing works [Liu et al., 2021; Xu et al., 2021], we set α = 0.999 in the EMA procedure and λ = 4 for the weight of unsupervised loss in Eq. 4. The same weak and strong data augmentation schemes in [Xu et al., 2021] are chosen. For partially labeled data, the model is trained for 180,000 iterations on 4 GPUs. Each GPU processes ten images. The learning rate is initialized to 0.1 and is divided by 10 at the 120,000-th and 160,000-th iteration. For fully labeled data, the model is trained for 720,000 iterations on 8 GPUs. Each GPU processes eight images. λ is set to 2. The learning rate is initialized as 0.1 and is divided by 10 at the 480,000-th and 640,000-th iteration. We follow the default settings in MMdetection to set other hyper-parameters.

4.1 Experimental Results

We compare our method against a series of existing methods including STAC [Sohn et al., 2020b], ISMT [Yang et al., 2021], Instant-Teaching [Zhou et al., 2021], HumbleTeacher [Tang et al., 2021b], Unbiased-Teacher [Liu et al., 2021], and Soft-Teacher [Xu et al., 2021], when partially or fully labeled data is leveraged for network optimization. Under scenarios of partially labeled data, our proposed method

(DCST) consistently outperforms existing methods by large margins, as shown in Table 1. Owing to the capacity in refining the bounding boxes and the re-weighting mechanism for eliminating the influence of incorrect background boxes, Soft-Teacher achieves the best performance among existing methods. However, the mislabeling problem can not be fundamentally overcome by the re-weighting mechanism. Meanwhile, the impact of misclassified foreground objects is not taken into consideration. In contrast, our proposed DCST assigns reliable soft labels to the filtered candidate boxes, which simultaneously rectified the misclassified boxes both in foreground and background. Compared to Soft-Teacher, our method derives mAP gains of 2.56, 1.36 and 1.16 on 1%, 5% and 10% labeled data, respectively. The improvement brought by our method increases as the percent of labeled data decreases. Considering less labeled data usually leads to pseudo labels with lower quality, the above phenomenon supports the argument that "combating low-quality pseudo labels" is a more effective solution in the protocol of limited labeled data.

The adoption of fully labeled data generates pseudo labels with relatively higher quality. Under this scenario, our method can still achieve the best performance, as shown by the last column of Table 1. This indicates that our method is capable of exploring unlabeled data more effectively, even when sufficient labeled data is provided.

The experimental results illustrate that the quality of pseudo labels is the key factor for promoting semi-supervised object detection. Compared to the strategy of discovering high-quality pseudo labels which is widely adopted by previous methods, "combating low quality pseudo labels" is a simpler while more effective solution. Moreover, further research is suggested to combine the two kinds of techniques for extracting reliable pseudo labels from unlabeled images more thoroughly.

4.2 Ablation Study

In this subsection, we conduct experiments to verify the efficacy of critical components in our method. Without specification, 10% labeled data is adopted for network optimization.

Efficacy of Double-Check Soft Teacher. Variants of our method are implemented via applying different double-check strategies. The experimental results are shown in Table 2. Employing the double-check strategy to post-processing foreground or background boxes improves the mAP metric by 0.5 or 0.8, respectively. Processing both kinds of boxes with the double-check strategy can bring more performance gain, leading to 2.2 higher mAP compared to the baseline without using the double-check strategy. We also attempt to reimplement the double-check module through directly allocating hard labels to object proposals. As shown by the second last row of Table 2, it produces results with 0.7 lower mAP, compared to the final version which assigns soft labels to object proposals. The reason is that soft labels can help to explore the underlying similarity cues among different categories and model the class probabilities more reasonably.

Efficacy of TOP-KPseudo Labeling. Table 3 presents the performance of using different number of top classes for the pseudo labeling mechanism. The precision, recall, and Fmeasure evaluates the efficacy in harvesting class-agnostic annotations. A pseudo annotation is regarded as a true positive, once it hits certain ground-truth annotation with an over 0.5 IoU. As can be observed, higher recall and F-measure can be obtained after setting the number of top classes larger than 1. This indicates that the top-k pseudo labeling is capable of discovering more objects without sacrificing much precision. Consequently, the final object detection performance can also be improved. Meanwhile, we can observe that using different numbers of top classes larger than 1 has little influence on the pseudo labeling and object detection performance.

Analysis about Quality of Pseudo Labels. The distribution of the maximum IoU between pseudo labels and groundtruth labels is presented in Figure 4. The variation of precision/recall metrics against the "K" value for pseudo labeling is illustrated in Table 3. We can observe that, there exist enormous number of pseudo labels which severely deviate from any ground-truth labels. Rectifying those labels is critical to explore unlabeled images in semi-supervised object detection. Our proposed double-check mechanism can achieve this goal via re-allocating the class probabilities, hence producing promising object detection performance with small amount of labeled data.

Choice for PseudoLabelingThreshold. Table 4 shows the performance of setting different values for the pseudo labeling threshold σ. The best performance is achieved when σ is set to 0.6. Varying σ to 0.7 causes little interference to the detection performance.

5 Conclusion

In this paper, we revisit the semi-supervised object detection task and point out two crucial problems of conventional pseudo labeling techniques: inaccurate position and low recall rate, which lead to category labeling errors during the training process. To overcome these issues, we propose Double-Check Soft Teacher, a novel algorithm to correct the errors among pseudo labels. Experiments on the COCO benchmark illustrate that our method outperforms the stateof-the-art methods under various percents of labeled data. Based on the experimental results, we conclude that "combating low-quality pseudo labels" is crucial for semi-supervised object detection.

Acknowledgments

This work was supported in part by the Key-Area Research and Development Program of Guangdong Province under Grant No.2021B0101410003, in part by the Guangdong Basic and Applied Basic Research Foundation under Grant No.2020B1515020048, in part by the National Natural Science Foundation of China under Grant No.61976250 and No.U1811463, and in part by the Guangzhou Science and technology project under Grant No.202102020633. This work was also supported by the Guangdong Provincial Key Laboratory of Big Data Computing, the Chinese University of Hong Kong, Shenzhen.