One Weird Trick to Improve Your Semi-Weakly Supervised Semantic Segmentation Model

1 Introduction

Recent semantic segmentation algorithms have successfully solved challenging benchmark datasets for semantic segmentation tasks like PASCAL VOC [Everingham et al., 2015] and MS COCO [Lin et al., 2014]. To do so, however, they use a large number of pixel-level annotations, which require extensive human labor to obtain. Great attention in computer vision research has thus turned to weakly-supervised learning [Jiang et al., 2019; Wang et al., 2020; Sun et al., 2020]. Weakly-supervised semantic segmentation aims to classify each pixel of test images, trained only on image-level labels (whether a class is present in the image, but not its location). Although weakly-supervised approaches have seen success in both semantic segmentation and object localization tasks, Choe et al. [2020] cast significant doubt on their validity and practicality. They argue that although weakly-supervised learning algorithms are designed to be trained only on imagelevel labels, they inevitably use explicit pixel-level labels (or, equivalently, manual judgement of outputs) in hyperparameter tuning. Since at least some fully-supervised inputs are necessary, Choe et al. point out that simply using a small number of these, e.g. five per class, to train a fully-supervised localization model substantially outperforms a weakly-supervised counterpart. To still take advantage of less-expensive weaklysupervised data points, though, perhaps the most natural change is to the semi-weakly supervised semantic segmentation (SWSSS) task: here only a small number of pixel-level labels are provided, as well as a large number of image-level labels.

Segmentation networks trained on a small number of pixellevel labels often confuse similar classes, e.g. cat and horse, as they architecturally tend to focus on local features rather than farther-away distinguishing areas. Thus, the additional supervision from image-level labels can be potentially quite helpful. Most existing SWSSS methods generate pseudo-labels from a classifier using class activation maps (CAMs) [Zhou et al., 2016], then train a segmentation network using both pseudo-labels and true pixel labels. These pseudo-labels, however, are difficult to extract: they tend to focus on small discriminative regions of objects, ignoring less-distinctive bulks of objects and often including nearby pixels that are not part of objects. Our analysis shows that as the baseline segmentation model improves with more training data, the pseudo-labels quickly provide more incorrect than correct supervision to what the model already would have predicted. Previous methods have thus employed additional information, such as saliency maps, or additional processing methods, adding complexity and many more hyperparameters to tune on a fully-supervised validation set. We use weak supervision data differently, without requiring any side information and introducing far fewer new hyperparameters.

To motivate our method, consider Figure 1. Baseline models with small training sets predict many classes which are not present in the image. If we ignore the predictions for the class of any pixel which are not present in the image at all, which we term oracle filtering, then the segmentation performance improves dramatically. Inspired by this, we propose a simple algorithm we call prediction filtering. Prediction filtering uses a multi-label classifier trained on only image-level labels to filter out segmentation predictions deemed very unlikely by the classifier, replacing predictions for those pixels with the next-most-likely class allowed through the filter. It is compatible with any segmentation model, and the threshold for "very unlikely" is the only new hyperparameter introduced.

Although the classifier is not perfect, because it is trained on a large weakly-supervised set, its predictions tend to be quite accurate. Moreover, it is trying to solve an easier problem than the segmentation network, using a different architecture. As we will see in the experiments, even without any additional weakly-supervised data, prediction filtering tends to improve the segmentation performance. When applied to baseline segmentation models, the performance significantly improves; adding it to the baseline model variant from Luo and Yang [2020] achieves (to our knowledge) the new highest performance on PASCAL VOC in the SWSSS regime. As prediction filtering is so general, it can even be easily applied to models which already exploit weakly-supervised data via pseudo-labels; doing so on the state-of-the-art SWSSS algorithms [Ouali et al., 2020; Luo and Yang, 2020] yields a new model with significantly higher performance, with more improvement for models trained on fewer fully-labeled images.

2 Approaches to Semantic Segmentation

Fully-supervised semantic segmentation. In this task, we have a training set of images with pixel-level class labels: Dpixel = {(xi, yi)}Mi=1 , where xi ∈ R3×Hi×Wi are images and yi ∈ {0, 1}K×Hi×Wi are pixel labels, with K the number of classes. Our goal is to find a model that can predict pixel labels y given a new image x.

Current approaches are mostly based on convolutional networks. One important factor to their success is using larger receptive fields via dilated convolutional layers [Yu and Koltun, 2016; Chen et al., 2017]. Even so, state-of-the-art algorithms still misclassify many pixels when multi-label classifiers on the same data obtain near-perfect accuracy. We conjecture this is because segmentation models still miss global structure of the image when looking at an individual pixel. We will exploit that a classifier "looks at images differently."

Weakly-supervised semantic segmentation. To avoid extensive human labor for pixel-level annotation, there have been many attempts to replace pixel-level labels with imagelevel labels: Dimage = {(xi, zi)}Ni=1 , with zi ∈ {0, 1}K a "logical or" of each channel of the unknown yi. We still want a model to produce y. The most common pipeline for weakly-supervised semantic segmentation is to generate a class activation map (CAM) [Zhou et al., 2016], refine it with various post-processing methods, then use it as a pseudo-label to train a semantic segmentation network. However, CAMs tend to focus only on discriminative regions of an object. Prior work has attempted to expand the CAM

to entire objects by masking out parts of an image [Singh and Lee, 2017] or intermediate feature [Zhang et al., 2018; Choe and Shim, 2019]; these methods do indeed expand the resulting CAM, but often too much, in very unstable ways. Another popular approach to grow the CAM regions, via methods proposed by Kr¨ahenb¨uhl and Koltun [2011], Huang et al. [2018], or Ahn and Kwak [2018], until it converges to the object region [Chen et al., 2020].

Semi-weakly supervised semantic segmentation. Choe et al. [2020] point out fundamental problems with weaklysupervised learning, as discussed in Section 1. We thus consider combining a small number of pixel-annotated images Dpixel with many weakly-supervised images Dimage, which we refer to as semi-weakly supervised semantic segmentation (SWSSS). Although they have not used exactly this name, many papers have already addressed this setting. Broadly speaking, the most common approach is to generate pseudolabels from a CAM, and use these in combination with true labels to train a segmentation network [Papandreou et al., 2015; Wei et al., 2018; Li et al., 2018; Ouali et al., 2020; Luo and Yang, 2020; Lee et al., 2021; Dang et al., 2022; Lai et al., 2021]. (For lack of space, we unfortunately elide all details of these approaches.) Because image-level labels have no spatial information, however, it is fundamentally difficult to make accurate pseudo-labels. As we will now argue, as the number of pixel labels increases and base models improve, the benefit of pseudo-labels drastically diminishes.

To demonstrate this, we train a DeeplabV1 segmentation network [Chen et al., 2015] on Dpixel consisting of 1,464 images from PASCAL VOC training set, and a VGG16 classifier on Dimage containing all 10,582 images in the full training set. To examine what part of the image causes this prediction, we extract CAMs for cat and horse classes using GradCAM [Selvaraju et al., 2017]. Although the classifier confidently predicts that only the cat class is present in the given image (Pr(cat) = 0.97, Pr(horse) = 0.03), Figure 2(a) shows that the segmentation model predicts most of the cat's body as horse (pink region). The CAM shows that the classifier makes this decision based on the most discriminative region of the object, i.e. the cat's head. The segmentation model does the same at the green (top) location, correctly predicted as cat; at the yellow (middle) location, however, the horse prediction is based mostly on the more ambiguous body area. As Dpixel grows, this phenomenon is largely alleviated; it seems 1,464 images were not enough for the segmentation model to learn "where to look."

Supervision added by previous models from classifier CAMs, then, will also tend to focus on discriminative regions of an object, and can therefore be misleading. To estimate the effectiveness of the pseudo-labels, we define a measure called mNet, the mean over classes c of

Here the subscript ·i,c refers to the set of pixels of i-th training image whose label is c; GT refers to the ground truth labels y, Pred to the predicted labels from a baseline segmentation model, and Pseudo to the CAM-based pseudo-labels. The first (blue) term in the numerator measures how much correct supervision the pseudo-label adds (see Figure 2(b)); the second (red) term, howmuch incorrect supervision is added. The denominator combines these two regions. mNet does not exactly measure how much the pseudo-labels help predictions, but it gives a rough sense of how correct their information is.

Figure 2(c) shows that "plain" CAM (in green, lowest) indeed helps when Dpixel is very small, but as it grows, mNet even becomes negative. It is possible to improve these predictions by, for instance, post-processing with a CRF (blue line, top). This, however, requires a complicated structure with several additional hyperparameters to tune on a fullysupervised validation set; setting these parameters correctly significantly affects performance, as shown e.g. by the substantially worse mNet when changing the threshold for a foreground region θ from 0.3 to 0.2 (orange line, middle).

3 Prediction Filtering

Motivation. Given a segmentation network f, whose output on the i-th image is f(xi) ∈ RK×H×W , the final prediction at each pixel (h, w) is normally

where K = {1, 2, . . . , K}. Oracle filtering (Figure 1) instead only considers classes actually present in the image, maximizing over ˜Ki = {c : zi,c = 1}. This improves segmentation performance substantially; the mIoU (mean intersection over union, the standard segmentation performance metric) of a DeeplabV1 segmentation network trained on Dpixel with 1,464 images improves from 61.8 to 70.6. We conjecture this is because the segmentation network has not learned to appropriately consider global features when predicting at each pixel of the image, while the classifier, solving an easier problem with more data, can immediately identify relevant areas.

Prediction filtering. Inspired by this phenomenon, we propose a simple post-processing method, prediction filtering. Given Dimage with a large number of images, we can train a highly accurate multi-label classifier g; a ResNet50 achieves 99% accuracy and 97.5% average precision on PASCAL VOC. Hence, we constrain predictions to come from the classifier's predictions instead of ground truth classes, ˆKi = {c : g(xi)c > τ}, where g(xi)c is the output logit of g for class c, and τ is a threshold to determine the presence of a class in an image. We provide full pseudocode in the appendix.

Compared to other SWSSS algorithms, prediction filtering has several advantages. First, the architecture is simple. It only requires an additional classifier, which can be trained in parallel with the segmentation network; most existing methods require training a classifier first. Second, it requires only a single additional hyperparameter, the threshold τ, far fewer than required by other SWSSS algorithms. For instance, MDC [Wei et al., 2018] requires two thresholds to determine the foreground and background regions, in addition to selecting the number of dilation layers with different rate for each layer. (We provide a comprehensive comparison of hyperparameter counts in the appendix). Prediction filtering thus minimizes the requirements on the additional fully-supervised validation set. Third, it can be independently added to any segmentation algorithm, including existing SWSSS algorithms; we do so in our experiments.

Effect on performance. Prediction filtering helps performance when an incorrect prediction is filtered out and the "backup" prediction is correct; it hurts when a correctlypredicted object is incorrectly filtered. It can also change an incorrect prediction to a different incorrect prediction; this can in fact either increase or decrease the mIoU score.

For a non-perfect classifier and reasonable setting of τ , it is conceivable for prediction filtering to hurt segmentation performance - although we did not observe this in our experiments. There is always a value of the threshold τ, however, for which prediction filtering at least does not hurt: just take τ → -∞, in which case no predictions are changed. As τ increases, it likely (though not certainly) removes incorrect predictions before it begins removing any correct predictions. In another extreme, for a perfect classifier, prediction filtering approaches oracle filtering; clearly, oracle filtering may not achieve perfect performance, but it can only help.

4 Experimental Evaluation

Dataset. We evaluate prediction filtering on PASCALVOC 2012 [Everingham et al., 2015] which contains 10,582 training, 1,449 validation, and 1,456 test images. For SWSSS, we follow the training splits of Ouali et al. [2020], where 1,464 images are used for Dpixel . As with previous work, we evaluate segmentation performance by mean Intersection over Union (mIoU), generally on the validation set. Test set performance is obtained from the PASCAL VOC evaluation server, without any tricks such as multi-scale or flipping.

Implementation. To verify the robustness of our method, we experiment with five semantic segmentation baselines: DeeplabV1 (based on VGG16 [Simonyan and Zisserman, 2015]), DeeplabV3 (based on ResNet-101 [He et al., 2016]), their deeper variants with wider receptive fields used by Luo and Yang [2020] which we call DeeplabV1-W and DeeplabV3-W, and a Transformer model called OCRNet [Strudel et al., 2021] (based on HRNetV2-W48 [Wang et al., 2021]). For prediction filtering, we use a ResNet50

classifier. We also apply prediction filtering to several existing SWSSS models. Although CRF post-processing is no longer commonly used in semantic segmentation tasks, in our experiments it still significantly improves the performance of models trained on a small number of pixel-level labels. We thus apply CRFs as default except when otherwise specified.

Comparison with state-of-the-art. In Table 1, we compare the performance of prediction filtering to existing methods when |Dpixel | is 1,464. We reproduce results for Deeplab, OCRNet, WSSL, CCT, DualNet, and AdvCAM. 1 Among VGG-based methods, DeeplabV1 with prediction filtering outperforms the other methods without using any additional information. Similarly, filtering also improves models with stronger backbones, though the margin of improvement is less dramatic since the baseline is better. Prediction filtering can help even when it does not involve adding any new training data: it significantly helps WSSL, CCT, AdvCAM, and DualNet, which already use weakly-labeled data. It is worth highlighting that simply adding prediction filtering to the DeeplabV3-W baseline achieves the new highest performance, slightly higher than DualNet (with a ResNet-W backbone) with prediction filtering, both of which are notably better than the previous state-of-the-art (DualNet with a ResNetW backbone without filtering). Prediction filtering a DualNet also sets the new state-of-the-art for VGG-based models.

Results on the test set. We further evaluate prediction filtering on the test set of VOC2012. In Table 2, we provide the performance of DeeplabV1 and DeeplabV3-W on the test set, as well as with prediction filtering applied. We can observe that prediction filtering improves the intersection-over-union

1 Our AdvCAM result is worse than reported by Lee et al. [2021], because we did not apply inference-time tricks, such as multi-scale and flipping, for fair comparison with other methods.

(IoU) scores for most of the 21 classes, leading to significant improvements in terms of mIoU (as on the validation set).

Various levels of supervision. Figure 3 shows the segmentation performance of DeeplabV1, DualNet with a VGG backbone, and DeeplabV3-W trained on 200, 500, 800 and 1,464 images with pixel labels, with and without prediction filtering. The blue (bottom) line shows performance of the base model; orange (middle) shows with prediction filtering; green (top) is with oracle filtering, which upper-bounds the possibility of improvement of prediction filtering with a better classifier. For smaller numbers of pixel labels, the performance gain from prediction filtering is drastically larger. For example, at 800 images with pixel labels, DeeplabV1 goes from 56.8 → 64.4, DeeplabV3-W from 72.3 → 74.3. The improvements for 200 pixel labels are 12.4, 1.8, and 4.8.

Relationship with CRF. CRFs adjust the prediction for each pixel by encouraging nearby, similar-colored pixels to have the same label. This usually improves the segmentation performance by refining detailed boundaries and making class predictions across an object more consistent. The latter role overlaps to some extent with prediction filtering. If the size of the wrong prediction is large, though, a CRF might expand the area of the wrong prediction, rather than remove it. Table 3, as well as the qualitative results to come shortly, shows that the methods complement one another.

Without image-level labels. Although image-level labels are easier to obtain than pixel-level labels, annotation effort is still nontrivial. Our hypothesis about why prediction filtering works, however, is largely that classifiers "looks at images differently" than segmentation networks do. It thus might help even if it does not use any additional data: Dpixel and Dimage contain the same images. Table 3 (right) shows this is the case. Even without introducing any actual new information, prediction filtering still improves mIoU significantly. Changes between classes. In Section 2, we showed some qualitative evidence that the segmentation network with low pixel labels tends to be confused between similar classes, and that prediction filtering can help to compensate it by looking at other parts of the image. To further demonstrate this, Figure 4 shows the pixel-level confusion matrix for a DeeplabV1 model with CRF before and after prediction filtering. Figure 4(a) shows a strong block structure where pixels from one animal class are often confused for another animal class, or vehicles for vehicles. In Figure 4(b), prediction filtering has dramatically reduced these types of mistakes.

Qualitative results. Figure 5 shows a success (top) and failure (bottom) for prediction filtering. At top, an object is mostly incorrectly predicted - bike as motor bike - and CRF only makes this worse. The classifier in filtering, however, correctly identifies there is no motor bike, and the model's "second choice" prediction is largely correct. In the failure case, an object (table) is mostly occluded; the segmentation model still identifies it, but the classifier misses it.

Further experiments. See the supplementary material.

5 Conclusion

Most existing semi-weakly supervised semantic segmentation algorithms exploit the pseudo-labels extracted from a classifier. Doing so, however, requires a complicated architecture and extensive hyperparameter tuning on fullysupervised validation sets. We propose prediction filtering, a simple post-processing method that only considers the classes for segmentation a classifier is confident are present. Our experiments demonstrated adding this method to baselines achieves the new highest performance on PASCAL VOC in SWSSS regimes, and adding it to existing SWSSS algorithms uniformly improves their performance. We expect prediction filtering can become a standard post-processing method for segmentation, along with CRFs, at least when a relatively large number of weakly-labeled images are available and the portion of class labels present in most images is low.

Acknowledgements

Support provided by the Canada CIFAR AI Chairs program, the Natural Sciences and Engineering Research Council of Canada, WestGrid, Compute Ontario, and Compute Canada.