Hierarchical Semantic Contrast for Weakly Supervised Semantic Segmentation

1 Introduction

Semantic segmentation is a fundamental task in computer vision, aiming at delineating target objects on the pixel level. With the recent advance of Deep Learning, fully supervised semantic segmentation (FSSS) models have made significant progress [Chen et al., 2017; Yuan et al., 2020; Xie et al., 2021]. However, compared to other vision tasks, such as classification and object detection, acquiring sufficient pixel-level annotations for segmentation is exceedingly costly and time-consuming. Recently, many efforts have been devoted to weakly supervised semantic segmentation (WSSS) to lower the reliance on pixel-level annotations by utilizing "weak" labels (e.g., image-level labels [Wang et al., 2020; Lee et al., 2021c; Du et al., 2022], scribbles [Lin et al., 2016; Liang et al., 2022; Vernaza and Chandraker, 2017], and bounding boxes [Lee et al., 2021b; Oh et al., 2021]). Thereinto, since image-level annotations can be obtained effortlessly, image-level WSSS is currently the popular direction, and we also adopt this form for WSSS.

∗Corresponding Author. This work was supported in part by Shanghai Science and Technology Committee under grant No. 21511100600 and No. 22511106000. We appreciate Shanghai Engineering Research Center of Intelligent Computing System for providing the computing resources.

In absence of pixel-wise annotations, the supervision signal only involves the existence of object classes without any locations and contours, leaving a huge supervision gap between WSSS and FSSS. One main solution is using Class Activation Map (CAM) [Zhou et al., 2016], which reveals object regions through the internal activations of the classifier to generate pseudo pixel-level supervision. However, the CAMs are prone to cover the most discriminative regions of objects (under-activation) or activate irrelevant regions (overactivation), affecting the segmentation performance. Previous studies design various methods to improve CAM in different ways, e.g., region growing [Kolesnikov and Lampert, 2016; Huang et al., 2018; Wang et al., 2018], adversarial erasing [Kumar Singh and Jae Lee, 2017; Wei et al., 2017; Hou et al., 2018], and auxiliary saliency supervision [Zeng et al., 2019; Oh et al., 2021; Wu et al., 2021].

Recently, driven by potent contrastive learning paradigms, several representation-based WSSS algorithms have been proposed [Du et al., 2022; Zhou et al., 2022]. They design different learning tasks on a single semantic level, such as pixel and region level, and manifest noticeable performance improvement. However, one prior knowledge is that complex scenes are composed of multiple hierarchies of semantics [Lu et al., 2021]. For instance, from the coarse-grained level, Figure 1a can be separated into region-of-interests (ROI) and background (Figure 1b). Then, ROI can be decomposed to "person" and "motorbike" (Figure 1d), which further consist of more fine-grained components like "face" and "body", and "handlebar" and "wheel" (Figure 1e). For precise CAMs, the following semantics should be distinguishable from each other: 1) ROI and background (ROI level); 2) the objects of each class (class level); and 3) the disjoint pixels of objects across classes (pixel level). Only implementing single-level contrastive learning may hinder the model from learning a more comprehensive semantic structure where the semantics are self-consistent simultaneously on the ROI, class, and pixel levels. To mitigate the above problem, we propose hierarchical semantic contrast (HSC) in lieu of single-level semantic contrastive learning to exploit different levels of semantic relations, aiming at making the model learn better object pattern understanding and more precise CAM inference.

Specifically, in the training stage, HSC infers coarse CAMs to generate pixel-wise pseudo-labels. Since pixel-level semantic contrast is susceptible to noises (i.e., false pseudolabels), we apply dense Conditional Random Field (CRF) to refine these pseudo-labels. Then, based on the activations of the CAMs and refined pseudo-labels, HSC implements semantic contrast in the embedding space from the prospects of three semantic hierarchies, i.e., ROI level, class level, and pixel level. Considering the semantic consistency under different views (transformations) of each image, HSC establishes a siamese network structure that adopts cross supervision on hierarchical semantic contrast, where one view of representations serves as the additional semantic supervision for the other view. Moreover, to utilize abundant semantics in the same hierarchy (e.g., class-level representations of person with various characteristics across different images), we establish momentum prototypes to complement more holistic and accurate representations for class-level and pixel-level semantic contrast. As the training goes, these prototypes merge representations that belong to the same hierarchy but across different images, and participate in contrastive learning as additional supervision. The results of extensive experiments demonstrate the superiority of our proposed approach.

Collectively, the main contributions of this paper can be summarized as follows:

• We propose hierarchical semantic contrast (HSC) for WSSS, which is motivated by the prior knowledge of
• semantic hierarchies harbored in each image. It involves three levels of semantic contrast and aims to learn more abundant semantic relations, enabling the model to earn better object pattern understanding and generate more precise CAMs. Compared to the prior art with singlelevel contrast, PPC [Du et al., 2022], the CAMs of HSC have more complete objects and precise boundaries without any saliency supervision.
• The siamese architecture with cross supervision and momentum prototype learning effectively couples hierarchical semantic contrast and further improves the quality of CAM inference. Moreover, we prove that CRF in the training stage can refine pseudo labels from coarse CAMs for better representation learning, especially in the noise-sensitive pixel-level semantic contrast.
• Our approach supports plug-and-play in existing WSSS models. We demonstrate its effectiveness on both nonsaliency-guided and saliency-guided baseline (SEAM [Wang et al., 2020] and EPS [Lee et al., 2021c]). The performance of their variants surpasses corresponding vanilla versions by large margins. With EPS, we record new state-of-the-art on PASCAL VOC 2012.

2 RelatedWork

2.1 WSSS with Image-level Labels

WSSS with image-label is a promising task that significantly alleviates the reliance on large numbers of pixel-wise annotations for training segmentation models. The convention of WSSS can be divided into two steps. Step 1 is to train a classification model and use its CAMs to generate pseudolabels. Step 2 is to use these labels to train a segmentation model. Since CAM can only highlight the discriminative regions and thus hardly cover complete object regions, SEC [Kolesnikov and Lampert, 2016] proposes three principles to refine CAM, i.e., seed, expand, and constrain, which are followed by many subsequent works. They can be summarized as region growing [Kolesnikov and Lampert, 2016; Huang et al., 2018; Wang et al., 2018], incorporating selfsupervised learning [Shimoda and Yanai, 2019; Wang et al., 2020], adversarial erasing [Kumar Singh and Jae Lee, 2017; Wei et al., 2017; Hou et al., 2018], and auxiliary supervision [Zeng et al., 2019; Oh et al., 2021; Wu et al., 2021]. These past efforts mainly focus on images individually, ignoring the abundant semantics in the images.

2.2 WSSS with Semantic-level Supervision

To utilize the semantics in the images, early works dedicate to exploit the semantic affinity to refine the pseudolabels [Ahn and Kwak, 2018; Ahn et al., 2019] or compute the semantic co-attention between pairs of images to obtain more consistent and integral CAM regions [Fan et al., 2020; Sun et al., 2020]. With the advance in contrastive learning, several current studies attempt to perform dense contrastive learning to improve object localization ability and obtain better CAMs. RCA [Zhou et al., 2022] establishes memory banks for each class and applies class-level semantic contrast and aggregation on the dataset level. PPC [Du et al., 2022]

introduces consistency regularization and proposes pixel-toprototype learning based on [Wang et al., 2021], constraining intra(inter)-class compactness(dispersion) in the feature space. In contrast, our work proposes hierarchical semantic contrast paradigm to build semantic relations simultaneously on ROI, class, and pixel levels, learning a comprehensive semantic structure for CAM inference.

3 Methodology

Our proposed HSC is implemented in Step 1 of WSSS to make the model generate more accurate CAMs. Building upon classification task (Lcls), HSC can be interpreted as an auxiliary learning task, whose overall loss function for training is the linear combination of the ROI-level contrastive loss Lrsc, class-level contrastive loss Lcsc, and pixel-level contrastive loss Lpsc:

where γ is a positive weight to rescale the loss of HSC. After generating our pseudo-masks, Step 2 is to use them to train DeepLab-ASPP [Chen et al., 2017] segmentation model following common practice in WSSS. The framework overview of HSC is illustrated in Figure 2.

3.1 Preliminary

Class Activation Map

To generate CAM, the first step is to train a multi-label classification CNN (e.g., ResNet-38 [Wu et al., 2019]). Given a batch of N images as input, its output of the last convolutional layer is f ∈ RN×D×HW , where D is the channel dimension, and HW is the spatial size of f. Then, f is aggregated to C channels by a 1 × 1 convolutional layer Fcam:

Here, C is the number of total classes. Finally, global average pooling (GAP) is connected after Fcam to retrieve the final classification scores. In the above process, f ′ followed by ReLU function is theoretically equivalent to the CAMs of C classes [Zhang et al., 2018]. As it is a more efficient way to compute CAM during forward propagation, we follow this manner in our paper.

Feature Representations

We set projectors to map original f to the embedding space for semantic contrast. Different from pixel-level semantics, ROI and class levels are more abstract semantic hierarchies. Therefore, we set a projector Fproj1 for ROI-level and classlevel semantic contrast and a projector Fproj2 for pixel-level semantic contrast, respectively. Fproj1 comprises two projection layers with convolution operation and activation to map f to representation u1, and Fproj2 comprises one simple projection layer, mapping f to representation u2. It is worth noticing that the channel dimension of u1 is larger than u2, aiming at accommodating more ROI-level and class-level representation information. The details of this part can be viewed in Section 4.2.

Momentum Prototype Learning

Considering the semantic information of the same hierarchy is unidentical in different images (e.g., cats with diversified characteristics), we set momentum prototype learning to complement the holistic semantics for class-level and pixellevel contrast inspired by [Zhang et al., 2021]. These momentum prototypes update via Exponential Moving Average (EMA) as training goes. For both class-level and pixel-level hierarchies, we set one momentum prototype for each class, which is updated by representations with high activations. Their momentum prototype learning and updating criteria will be detailed in Section 3.3 and Section 3.4, respectively.

Cross Supervision

Considering that the semantics of an image under different views (transformations) should be consistent [Wang et al., 2020], we impose semantic consistency on the two views in our siamese architecture, where the representations of each hierarchy from one view can act as a supervisory signal for the other view, and vice versa. This design implicitly achieves consistency regularization to improve the robustness of semantic structure against various image scales. In HSC, each hierarchy's representations and momentum prototypes are involved in cross supervision. For simplicity, Section 3.2-3.4 only describe HSC from one view.

3.2 ROI-level Semantic Contrast (RSC)

Given an image, if one CAM can cover a more complete and precise area of ROI, its ROI and background representations should contain sufficiently different semantics, and thus having a large distance in the embedding space. Further, in a batch of images, their pairs of ROI and background representations should also be distinct. Based on this insight, we propose to push apart the ROI and background representations across images. Note that in the current hierarchy, we do not encourage the model to pull representations with the same property together (as adopted in many self-supervised contrastive learning paradigms), because the semantics of ROI and background in each image are unique in most cases. The pulling behavior will cause the collapse of the embedding space, deteriorating the quality of the CAM.

Representation of ROI and Background

For the i-th image I in one batch and its representation ui1 , we first conduct pixel-wise argmax function on its CAM to assign the category for each pixel, and generate the pseudo binary mask of its ROI and background. Thereinto, if one pixel p belongs to any non-background class, it will be marked in its ROI mask mio ∈ RHW , otherwise it will be marked in its background mask mib ∈ RHW . Guided by mio and mib , we select their corresponding dense representations from ui1 , and then aggregate them into a pair of ROI and background representations as follows:

where rio and rib is the i-th image's representation of the ROI ro and that of background rb, respectively.

ROI-level Semantic Contrast

Given a batch of N images, the ROI-level semantic contrast applies the following formulation:

where sim(·, ·) is the cosine similarity between pair of representations. Finally, the ROI-level semantic contrast with cross supervision is calculated by:

where ˜ rb is the background representations of the other view.

3.3 Class-level Semantic Contrast (CSC)

Due to lacking pixel-wise annotations for the model, the CAMs of some classes probably suffer from over-activation or under-activation. In this case, some class-level representations are ambiguous because they are coupled with the representations of other classes. Therefore, the goal of class-level semantic contrast is to mine high-quality representations of each class from the entire dataset, utilizing them to calibrate the model to learn more precise representations of each class while pushing them apart from other classes' representations.

Class-level Prototype Mining

The class-level momentum prototype will be updated by the representations of pixels with high CAM values. Given a batch of N images, we first aggregate the representations of each class. SupposeMc is the collection of the indices of the images with class c in the current batch, then the class-level representation of c will be:

where | · | represents the cardinality, Aic is the collection of pixels with top kc activation values of class c at the i-th image, and each pixel p in the i-th image has its activation value of wi (p) and representation ui1 (p). Then, if c appears in the current batch, its momentum prototype Pc will be updated by:

where λ is the momentum for the prototype update and C is the collection of the classes involved in the current batch.

Class-level Semantic Contrast

Given class c involved in the current batch and the prototpyes P = {Pc}Cc=1 , we pull rc and its corresponding momentum prototype together, and meanwhile push apart other momentum prototypes via Info-NCE loss [Oord et al., 2018]:

where τ = 0.1 is a sharpening temperature used to reconcile the distribution of distances. Finally, the class-level semantic contrast is calculated with cross supervision as:

where ˜ P denotes the class-level momentum prototpyes of the other view.

3.4 Pixel-level Semantic Contrast (PSC)

A recent study, PPC [Du et al., 2022], has proposed pixel-toprototype contrast for WSSS. Its training objective is to build pixel-level supervision from the reliable prototype of each class to narrow the gap between WSSS and FSSS. Based on its three contrast learning paradigms (i.e., cross-view contrast, cross-CAM contrast, and intra-view contrast), we further improve this way by adding CRF refinement and Momentum prototype learning as conducted in CSC. The brief introduction of the above contrast learning paradigms and their corresponding improvements are described as follows.

CRF Refinement

Since each pixel will be involved in pixel-level semantic contrast, the validity of pseudo-labels generated by CAMs is important. However, we observe that the quality of pseudolabels is not ideal in many cases, such as complex scenes and overlapping objects. If one pixel's pseudo-label is misclassified, its corresponding representation will be conducted PSC with undesirable prototypes, which degrades the quality of PSC. Consequently, before implementing PSC, we apply CRF to refine generated pseudo-labels: y′ = CRF(y, I), where I denotes the whole image and y denotes its pseudolabel. The pseudo-labels and corresponding pixel representations are down-sampled before PSC, and thus CRF does not bring much computation load.

Pixel-level Prototype Mining

For each class c, its pixel-level prototype Qc is produced from the representations of highly activated pixels that are assigned to c in the current batch:

where Bc is the collection of top kp activated pixels in class c for the current batch, and each pixel p has activation value of w(p) and representation u2(p). Then, class c's momentum prototype Q′c will be:

Pixel-level Semantic Contrast

Followed by PPC, given a pixel p and the prototypes Q = {Qc}Cc=1 , the pixel-level semantic contrast Fpsc(·) holds the following formulation:

where y′ (p) denotes the refined pseudo-label of p and Qy′ (p) is its corresponding prototype.

For the intra-view contrast, it is not involved supervision signals from the other view, where pseudo-labels y′ , representation u2, and prototypes Q are derived from the same view. For the cross-view contrast, the prototypes of the current view are replaced with the prototypes ˜Q from the other view. For the cross-CAM contrast, the pseudo-labels of the current view are replaced with the pseudo-labels ˜y′ from the other view. For our proposed momentum prototype learning, we replace the original prototypes Q with the momentum prototypes Q′ to implement intra-view contrast. Finally, given one whole image I, the pixel-level semantic contrast is calculated as:

4 Experiment

4.1 Dataset and EvaluationMetric

We evaluate our approach on the standard WSSS benchmark, PASCAL VOC 2012 (20 object classes and one background class). Following the convention in semantic segmentation, we adopted its augmented training set (SBD) [Hariharan et al., 2011] that consists of 10582 images to train our classification model in step 1 and the segmentation model in step 2. Mean intersection over union (mIoU) is used to evaluate initial seeds, pseudo-labels, and final segmentation performance. The mIoU of segmentation performance on the test set is evaluated from the official evaluation server.

4.2 Implementation Details

Following the baseline, SEAM and EPS, ResNet38 [Wu et al., 2019] with output stride = 8 is adopted as the backbone network. For our siamese network architecture, the backbone and the projectors share weights between the two views. The projector Fproj1 is equipped with two Conv-ReLU blocks, mapping 4096-dimensional features to 256-dimensional embedding space, and the projector Fproj2 is equipped with one Conv-ReLU block, mapping 4096-dimensional features to 128-dimensional embedding space. For the training images of View #1, they are first randomly rescaled with the range of [448, 768] by the longest edge and then randomly cropped by 448 × 448. Based on the images of View #1, they are downsampled to 128 × 128 for View #2. When integrating our approach to SEAM and EPS, we follow the same training and inference protocol in SEAM and EPS, including the training epoch, optimizer, learning rate, learning rate decay strategy, and weight decay. At the training stage, we set γ = 0.1 to balance the loss of Lhsc and their supervision loss. At the inference stage, we adopt multi-scale inference with flipping transformation as conducted in previous works.

CRF is implemented on the initial seeds as a post-processing procedure to refine the pseudo-labels. For the segmentation network, we train DeepLab-ASPP [Chen et al., 2017] with ResNet101 backbone using the pseudo-labels generated from our approach with EPS. More detailed settings are available in our Appendix.

4.3 Initial Seed and Pseudo Label Evaluation

Table 1 reports the segmentation performance of initial seeds, seeds after CRF, and final pseudo-labels on PASCAL VOC 2012 train set. Following SEAM [Wang et al., 2020], our initial seeds are generated by setting a range of thresholds to separate the objects and backgrounds in the original CAM inferences. Pseudo-labels are refined by PSA [Ahn and Kwak, 2018] from seeds after CRF. As can be seen, for SEAM with non-saliency guidance, HSC dramatically enhances its performance, resulting in an 8.9% and 9.7% increase in initial seeds and seeds after CRF, respectively. Furthermore, on EPS with saliency guidance, HSC can also boost its performance on initial seeds (+2.3%) and seeds after CRF (+3.2%), achieving new state-of-the-art performance. In comparison to the methods equipped with semantic contrast paradigms (i.e., PPC and RCA), our approach achieves noticeable improvement when integrating with EPS. In terms of our refined pseudo-labels, we note that our approach with SEAM obtain slight improvement compared to seeds after CRF. Considering previous studies [Lee et al., 2021c; Du et al., 2022] have manifested that PSA is limited to yield benefits when seeds after CRF are already exquisite enough, we reckon that this is mainly caused by the performance bottleneck of PSA, and thus we directly use seeds after CRF as pseudo-labels when equipping HSC to stronger baseline EPS.

To investigate how HSC improves the quality of initial seeds and seeds after CRF, we present the qualitative results of CAMs generated by SEAM, PPC, and HSC, respectively.

As shown in Figure 3, we observe that our approach achieves more superior performance than SEAM and PPC in terms of object completeness and boundary precision. Especially in the boundaries, the CAMs of our method cover more accurately without any saliency supervision compared to PPC with single-level semantic contrast.

4.4 Final Segmentation Performance

Table 2 provides the comparison of HSC against representative methods in terms of final segmentation results on PASCAL VOC 2012 val and test set. As seen, building upon DeepLab-ASPP, our proposed approach brings significant gains over EPS. With equipping HSC, we increase the segmentation mIoU of EPS by 2.7% on val set and 3.7% on test set, setting a new state-of-the-art on WSSS. Figure 4 illustrates some qualitative segmentation results of HSC, from which we can observe that our method works well in both simple and complex scenes.

Furthermore, compared to PPC, the prior art with singlelevel semantic contrast, we explore which categories have performance improvement brought by hierarchical semantic contrast. Table 3 tabulates the categorical mIoU performance comparison on test set. We can find that our proposed HSC achieves significant performance improvements (1.45% ∼ 4.54%) in eight categories. The objects in most of these categories are relatively challenging to segment, as they often appear in complex scenes (such as multiple instances, occlusions, and objects with different sizes or various characteristics). This result demonstrates that hierarchical semantic contrast can better guide the segmentation model to handle these cases.

4.5 Ablation Study

To investigate how each component in our proposed approach contributes to WSSS, we conduct extensive ablation studies in this section. Here, all ablation analyses are implemented with SEAM baseline on PASCAL VOC 2012 train set. The results are presented in Table 4.

Hierarchical Semantic Contrast. We investigate the necessity of learning hierarchical semantic contrast for WSSS. Based on cross supervision, each level's semantic contrast paradigm provides a performance boost, improving the baseline performance from 55.4% to 62.6% (+7.2%). This proves that our approach fulfills the goal of obtaining a better object pattern understanding by learning a holistic semantic structure across different semantic hierarchies.

Cross Supervision. It is an essential component in semantic contrast. When integrating it into HSC, we witness a noticeable performance increment from 60.7% to 62.6% (+1.9%). This indicates that the implicit consistency regularization can allow the model to learn the semantic consistency across different views, which effectively bridges the supervision gap between pixel-level and image-level annotations.

CRFRefinement. We note that CRF refinement on pseudolabels works well with PSC, increasing from 63.5% to 64.2% (+0.7%). By contrast, it shows marginal improvement, from 64.2% to 64.3% (+0.1%), for CSC and RSC. This is in line with our assumption that PSC is more susceptible to false pseudo-labels. After CRF refinement, HSC can obtain better pseudo-labels to perform semantic contrast. In comparison to PSC, the representations of RSC and CSC are estimated from numerous representations, which are more robust against false pseudo-labels.

MomentumPrototype Learning. InCSC, we usemomentum prototypes to calibrate the model to learn high-quality representations at class level. Further, we also introduce this paradigm to improve PSC. When MPL is integrated into PSC, the initial seeds become more precise, from 62.6% to 63.5% (+0.9%), indicating that the model learns holistic semantics across different images from momentum prototypes. We empirically choose the momentum λ and the prototype number kc and kp to implement this paradigm. In this paper, we set λ = 0.9, kc = 16, and kp = 32 for the optimal performance.

5 Conclusion

In this work, we propose a novel approach, HSC, to conduct semantic contrast at different levels as the auxiliary learning task to learn semantic segmentation using imagelevel supervision only. HSC conducts semantic contrast from fine-grained hierarchy to coarse-grained hierarchy, building a more comprehensive semantic structure in the embedding space. In comparison to representative approaches, this enables the model to have significant improvements in the object completeness and boundary precision of CAMs, which further boosts the final segmentation performance. Extensive experiments validate the effectiveness of HSC and manifest its leading performance of WSSS on PSCAL VOC 2012.