Complete Instances Mining for Weakly Supervised Instance Segmentation

1 Introduction

Instance segmentation involves the simultaneous estimation of object location and masking segmentation, and it has made significant progress with the assistance of large datasets and instance-level annotations. However, process of obtaining instance-level annotations can be costly and time-consuming. As a solution, weak annotations such as box-level and imagelevel annotations have been utilized in instance segmentation. Among these weak annotations, the use of image-level annotations is the most cost-effective, but also the most challenging due to the difficulty of aligning coarse annotations with the finer task.

∗Corresponding Author

Weakly supervised instance segmentation (WSIS), which involves using only image-level annotations, has experienced widespread growth in recent years. There are two main paradigms for addressing WSIS: proposal-based paradigm and proposal-free paradigm. The proposal-based paradigm [Zhou et al., 2018; Liu et al., 2020; Shen et al., 2021; Ou et al., 2021] assumes that an instance can be represented by a proposal, thus simplifying WSIS as a classification task. However, these approaches will result in impaired segmentation performance due to redundant segmentation , as illustrated in Figure 1. Typically, if we simply select a few results as output, the segmentation results will be the most discriminative parts of instances. In contrast, the proposal-free paradigm [Ahn et al., 2019; Kim et al., 2022] generates segmentation results online without proposals. During training, this paradigm relies on confident precomputed pseudo labels to achieve better performance. To leverage these pre-computed pseudo labels, IRNet [Ahn et al., 2019] utilizes Class Attention Maps (CAM) [Zhou et al., 2016] while BESTIE [Kim et al., 2022] uses weakly supervised semantic segmentation (WSSS) maps. The proposalfree paradigm is heavily dependent on pre-computed pseudo labels, which can limit its potential for further improvement.

To address the limitations mentioned above, we propose a novel proposal-based method that operates in an online refinement manner and consists of three key components: MaskIoU heads, a Complete Instances Mining (CIM) strategy, and an Anti-noise strategy. Our method also utilizes pre-computed pseudo labels to warm up the model. Instead of using techniques such as random walk (RW) [Ahn et al., 2019] or conditional random fields (CRF) [Ge et al., 2019; Shen et al., 2019; Zhu et al., 2019] to mine complete instances, which can significantly slow down training process, our method employs MaskIoU heads to predict the integrity scores of proposals and the CIM strategy to mine complete instances. We then generate refined pseudo labels based on spatial relationships using the mined complete instances. Finally, the Anti-noise strategy ensures that our method does not suffer from significant performance degradation when pre-computed/refined pseudo labels are noisy.

The main contributions of our work are as follows:

• We introduce the MaskIoU head to WSIS for the first time and propose an Anti-noise strategy to filter out noise caused by pre-computed pseudo labels and refined pseudo labels, improving the robustness of our method.
• We explicitly address the problem called redundant segmentation and present an effective Complete Instances Mining (CIM) strategy to guide the network to pay more attention to complete instances.
• Despite its simplicity, our method achieves state-of-theart performance on the PASCAL VOC 2012 and MS COCO datasets with a notable margin.

2 RelatedWork

2.1 Weakly Supervised Object Detection

Multiple Instance Learning (MIL) is a popular paradigm for addressing weakly supervised object detection (WSOD), as it utilizes proposals generated by selective search [Uijlings et al., 2013] as a bag of instances.

WSDDN [Bilen and Vedaldi, 2016] proposed a classification branch and a detection branch to output confident proposals as results, but this approach tends to predict the most discriminative parts of objects. To handle this issue, contextlocnet [Kantorov et al., 2016] introduced contextual information to a contrastive model. OICR [Tang et al., 2017] proposed an online instance classifier refinement strategy, which utilizes the proposals of the highest confidence as a pseudo label to supervise the next branch for improved performance. MIST [Ren et al., 2020a] considered the presence of multiple instances and employed Concrete DropBlock to mine complete instances. C-MIL [Wan et al., 2019] designed a novel MIL loss to avoid getting stuck into local minima, while PCL [Tang et al., 2020] transformed the single MIL problem into multiple MIL subproblems through clustering proposals. WSOD2 [Zeng et al., 2019] combined bottom-up and topdown objectness knowledge as evidence to discover complete instances in the candidates. NDI-WSOD [Wang et al., 2022] and OD-WSCL [Seo et al., 2022] approached WSOD through contrastive learning.

Although WSIS is distinct from WSOD, it is worth noting that our approach was inspired by WSOD approaches and WSOD approaches can be easily transferred to WSIS.

2.2 Weakly Supervised Instance Segmentation

WSIS with only image-level annotations can be divided into two paradigms: proposal-based paradigm and proposal-free paradigm.

Following the proposal-based paradigm, PRM [Zhou et al., 2018] used peaks in Peak Response Maps (PRM) as instance cues and combined them with CAM to predict instances. IAM [Zhu et al., 2019] generated Instance Activation Maps (IAM) through an extent filling module to identify complete instances. Label-PEnet [Ge et al., 2019] employed a cascaded pipeline in a coarse-to-fine manner to simultaneously perform classification, object detection, and instance segmentation. Arun et al. [Arun et al., 2020] used a conditional distribution to explicitly model the uncertainty, which enables pseudo labels to become more reliable. Fan et al. [Fan et al., 2018] and LIID [Liu et al., 2020] used instance saliency labels to support the generation of pseudo labels and considered the relationships in the training set. WS-RCNN [Ou et al., 2021] proposed an Attention-Guided Pseudo Labeling (AGPL) strategy and an Entropic OpenSet Loss to improve WSIS. PDSL [Shen et al., 2021] treated proposals as segmentation cues and proposed a framework for learning detection and segmentation in parallel. Our proposed method is similar to PDSL, but does not require additional segmentation branches due to the ability of proposals to effectively segment instances.

Following the proposal-free paradigm, IRNet [Ahn et al., 2019] used displacement field to indicate instances and generate complete instances via Class Boundary Maps and random walk. BESTIE [Kim et al., 2022] transferred the knowledge of WSSS to WSIS by a strengthening constraint. While our method also leverages pre-computed pseudo labels, we design our approach in an online refinement manner and propose an Anti-noise strategy to reduce the dependence on these pre-computed labels.

Thorough comparisons of representative proposal-based and proposal-free methods show that outputs from these two paradigms focus on different aspects. Proposal-based methods tend to have higher recall and lower precision, while proposal-free methods exhibit the opposite. Despite these differences, there is not a significant gap in performance between the two paradigms. Previous proposal-based methods have effectively utilized the power of proposals, which not only consider bottom-up information of the image but also simplify the task. However, a convolutional neural network (CNN) trained for image classification typically results in redundant segmentation. To alleviate this problem, some researchers [Zhu et al., 2019; Arun et al., 2020; Ou et al., 2021] have made some attempts. In contrast, our method explicitly models this problem and addresses it through the Complete Instances Mining (CIM) strategy.

3 ProposedMethod

3.1 Preliminary and Overview

Given an image I sized by HI ×WI , and its image-level label Y ∈ RC , where C is the number of categories and Yc = 1 indicates that image I contains the cth category, and otherwise, Yc = 0. To simplify the task, proposals R ∈ RN×HI×WI are obtained using off-the-shelf proposal techniques [Pont-Tuset et al., 2017; Maninis et al., 2018], where Rn is a binary segmentation mask and N indicates the number of proposals. In this work, we modify the image-level label Y ∈ RC+1 to include the background category, denoted as Y0 = 1.

Our method follows the proposal-based paradigm and consists of an Anti-noise branch and several Refinement branches, as illustrated in Figure 2. To begin, we obtain pre-computed pseudo labels using the AGPL strategy proposed by WS-RCNN [Ou et al., 2021]. Then, we use MaskFuse to extract proposal features and feed them into multiple branches. Finally, the CIM strategy explicitly models the redundant segmentation problem and generates refined pseudo labels. To further enhance the robustness of our network, we apply an Anti-noise strategy.

3.2 Reviewing AGPL

The AGPL strategy is a method for generating pseudo labels for a set of proposals in the WS-RCNN model. The method starts by training a classifier and producing CAM. For each target category c, confident peaks from CAM are selected as instance cues, denoted as pc = {pc1 , pc2 , · · · , pci }, where pci means the ith peak in the cth category. For each peak, AGPL averages and thresholds proposals containing it to leverage support mask Sc i , denoted the number of proposals as nci .

The AGPL strategy sorts the support mask set in descending order according to scores of peaks and assigns labels to proposals based on the IoU between the proposal and the support mask. The pre-computed pseudo labels of proposals are represented as ˆ y0 ∈ RN×{C+1} .

During the labeling process, each proposal is assigned to only one category. Proposals that overlap with support masks but are not assigned to any categories are assigned as background. Finally, proposal clusters are generated based on whether proposals are assigned by the same support mask, with each background proposal being assigned as a single cluster. The proposal clusters set is denoted asH = {(C1,M1, s1), (C2,M2, s2), · · · , (CN 0,MN 0, sN 0)}, where Cn, Mn, and sn represent the set of proposals, the number of proposals, and the label of the nth proposal cluster, respectively.

3.3 MaskFuse

In contrast to WSOD, WSIS utilizes binary masks as proposals rather than bounding boxes. As a result, RoIPool [Girshick, 2015] and RoIAlign [He et al., 2017] operations are not available for WSIS.

To address the lack of feature extraction operation in WSIS, we propose a lightweight operation called MaskFuse. The MaskFuse leverages the bounding box feature B ∈ Rh×w×D obtained through the RoIAlign operation and Algorithm 1 Complete Instances Mining (CIM) strategy Input: Image label Y , proposals R, classification scores yk-1 , integrity scores tk-1

Parameter: NMS threshold τnms, containment threshold τcon, percentage of seeds pseed Output: Pseudo ground truth P k

1: P k = ∅ 2: for c = 1 to C do 3: if Yc = 1 then 4: // Step 1: selecting seeds 5: Rcls ←Sort(yk-1 ∗,c ) 6: Rkeep ←keep top pseed percent of Rcls 7: indseed = NMS(Rkeep, τnms) 8: // Step 2: mining pseudo ground truth 9: Calculate matrixM ∈ RN×N ,Mi,j = Ri∩Rj Rj 10: Mcon =M[:, indseed] > τcon 11: indgt = argmax(Mcontk-1 ∗,c , dim= 0) 12: P k c .append(R[indgt]) 13: end if 14: end for

extracts the corresponding proposal Rcrop ∈ Rh×w using the RoICrop operation. Then, we concatenate Rcrop ⊙ B with B and utilize a convolutional layer and two fully connected layers to fuse features from mask-level and box-level. The MaskFuse allows each proposal to be represented by a feature with contextual information, enabling WSOD methods to be transferred to WSIS.

3.4 Refinement Branch

The MaskIoU head, proposed by MS R-CNN [Huang et al., 2019], and the center-ness head, proposed by FCOS [Tian et al., 2019], are designed to learn the quality of predicted results (i.e., masks and bounding boxes) in their respective methods. However, this powerful head is missing in WSIS. To fill this gap, we implement the MaskIoU head into our framework.

Given proposal features, the kth Refinement branch produces two matrixes yk , tk ∈ RN×{C+1} from classification and MaskIoU heads, which represent the classification and integrity scores, respectively. Furthermore, we employ the CIM strategy to produce refined pseudo labels from preceding branch, denoted as ˆ yk , ˆ tk . Following OICR [Tang et al., 2017], we also propose loss weights wk ∈ RN to mitigate the degradation caused by noisy refined pseudo labels.

where Nfg and Nbg represent number of foreground and background. LSmooth-L1 indicates smooth-L1 loss [Girshick, 2015].

3.5 Complete Instances Mining (CIM)

Motivated by redundant segmentation, we propose a novel Complete Instances Mining (CIM) strategy. CIM can be divided into two steps: selecting seeds and mining pseudo ground truth. The first step is similar to the operation of MIST [Ren et al., 2020a] strategy.

Let's start with a simple modeling of redundant segmentation. As we expected, partial segmentations typically have high classification scores and low integrity scores, while complete segmentations have the opposite. Additionally, partial segmentations are often spatially contained within complete segmentations. Hence, we utilize classification and integrity scores as metrics for CIM, illustrated in Algorithm 1. CIM awares multiple instances through selecting seeds, which maintains a high recall of pseudo ground truth. For each seed, the proposal that contains it and has the highest integrity score is considered as its corresponding pseudo ground truth. Mining pseudo ground truth enables our method to identify complete instances by aggregating multiple seeds.

To collect more pseudo labels, we assign refined pseudo labels to all proposals. For each proposal, if it highly overlaps with pseudo ground truth, we consider its classification target to be the category of pseudo ground truth. τcls is a hyperparameter.

The integrity target is determined in the same way. τiou is also a hyper-parameter.

In which, τiou is typically greater than τcls. The loss weight for each proposal can be calculated using Equation 6, where c and j indicate the category and index of pseudo ground truth that has the largest overlap with the Ri, respectively.

We follow the rule mentioned in Section 3.2 to assign background labels. To further improve performance, we implement a cascaded threshold τcas into our framework inspired by Cascade R-CNN [Cai and Vasconcelos, 2018]. This involves modifying τcls and τiou in the kth Refinement branch to τcls + (k - 1) τcas and τiou + (k - 1) τcas, respectively.

3.6 Anti-Noise Strategy

High-quality pre-computed pseudo labels facilitate the network in achieving faster convergence and better performance. However, noise in these labels can confuse the network and lead to overfitting. Similarly, noisy refined pseudo labels also degrade segmentation performance. To address these issues, we propose an Anti-noise strategy comprising an Anti-noise branch and an Anti-noise sampling strategy.

Anti-noise branch. We adopt the WSDDN branch [Bilen and Vedaldi, 2016] as the Anti-noise branch. Similar to the Refinement branches, it also produces two matrixes y0 , t0 ∈ RN×{C+1} from classification head and Anti-noise head. Then, image-level score can be obtained by yI =

∑Nn=1 (y0n,∗ ⊙ t0n,∗ ). We combine Binary Cross-Entropy and PCL [Tang et al., 2020] losses as the objective function.

Here, the hyper-parameter α is set to 12 by default. Especially, the Anti-noise head is not directly guided, which enables it to filter out noise in pre-computed pseudo labels.

Anti-noise sampling. Refined pseudo labels also contain noise, as constraints of the CIM strategy are relaxed. Our observations suggest that noisy labels tend to have lower loss weights. Based on this insight, we adopt an Anti-noise sampling strategy that treats loss weights as sampling probabilities to sample pseudo ground truth P k . Specifically, we implement sampling with replacement to filter out noise in refined pseudo labels.

In summary, the objective function can be written as:

where K is set to 3 by default.

4 Experiments

4.1 Datasets and EvaluationMetrics

Following previous methods, we also evaluate our method on PASCAL VOC 2012 [Everingham et al., 2010] and MS COCO [Lin et al., 2014] datasets. The VOC 2012 dataset includes 10,582 images for training and 1,449 images for validation, comprising 20 object categories. The COCO dataset comprises 115K training, 5K validation, and 20K testing images across 80 object categories. Following previous methods, we report the mean average precision (mAP) with IoU thresholds of 0.25, 0.5, 0.7, and 0.75 for VOC 2012 and report mAP with IoU thresholds from 0.5 to 0.95 for COCO.

4.2 Implementation Details

Our method is implemented in PyTorch and experiments are conducted on an Nvidia RTX 3090. We use COB [Maninis et al., 2018] method to generate proposals for all experiments and utilize ResNet50 [He et al., 2016] as the backbone. As we using mAP25 and mAP50 as evaluation metrics, we set classification τcls and integrity τiou thresholds to 0.25 and 0.5, respectively. The cascaded threshold τcas is set to 0.1. τnms, and pseed in Algorithm 1 are set to τcls, and 0.1, respectively. Containment threshold τcon is set to 0.85 following SoS [Sui et al., 2021]. Although there are many hyper-parameters, we only tune values of τcas and pseed.

During training, we use the SGD optimization algorithm with an initial learning rate of 2.5 × 10-4 and a weight decay of 5 × 10-4 . We adopt a step learning rate decay schema with a decay weight of 0.1 and set the mini-batch size to 4.

The total number of training iterations is 4.5 × 104 for the VOC 2012 dataset and 24 × 104 iterations for the COCO dataset. For data augmentation, we apply five image scales {480, 576, 688, 864, 1200} with random horizontal flips for both training and testing. During testing, we employ the product of classification and integrity scores as the output of each Refinement branch and average these outputs as the final scores. Following previous methods, we also generate pseudo labels from our method for training Mask R-CNN.

4.3 ComparisonWith State-of-the-Art

We compare the performance of our method with previous state-of-the-art WSIS methods, as shown in Table 1 and Table 2. For a fair comparison, we also report the results obtained with different backbones, i.e., VGG-16 [Simonyan and Zisserman, 2015] and HRNet-W48 [Wang et al., 2021]. Note that MIST is originally an object detection method, and we adapt it to the WSIS task for comparison.

Our proposed method outperforms all previous methods on both datasets, achieving higher performance than LIID [Liu et al., 2020] even without the use of instance saliency labels, demonstrating that such labels are not necessary for further advancement in WSIS. Additionally, our method demonstrates a 4.9% and 1.7% improvement in terms of mAP50 on the VOC 2012 and COCO, respectively, when compared with BESTIE [Kim et al., 2022]. Thanks to impressive semantic segmentation maps produced by WSSS, BESTIE achieves excellent performance on simple images, i.e., single instance, multiple non-adjacent instances, and multiple instances of different categories. However, BESTIE tends to produce grouping instances results caused by its strengthening constraints, illustrated in the second row of Figure 3. In contrast, we follow the proposal-based paradigm which always results in redundant segmentation rather than grouping instances. Hence, we propose CIM to mine complete instances, illustrated in the third row of Figure 3.

Although my approach may appear overly complex, it is implemented using only one linear layer per head, and its training can be completed in a few hours, as shown in Table 3.

Meanwhile, the code implementation of CIM is simple.

4.4 Ablation Study

We conduct several ablation studies on the VOC 2012 dataset to evaluate the effectiveness of each component. In these studies, we use ResNet-50 as the backbone and do not apply Mask R-CNN to save time. When the CIM strategy is unavailable, we adopt MIST [Ren et al., 2020a] strategy to replace it and adapt τcls and τnms to 0.5.

Impact of AGPL

As shown in Table 4, the use of AGPL results in an improvement in segmentation performance, with an increase of 49.2% and 20.6% in terms of mAP50 and mAP75, respectively. Although AGPL produces reliable pre-computed pseudo labels, it only provides coarse complete instance cues, leading to a significant improvement in mAP50 but little gain in mAP75.

Impact of MaskIoU and CIM

As shown in Table 4, the performance is slightly improved when the MaskIoU heads are applied alone because the distribution of refined pseudo labels does not change. By employing the CIM strategy, our method achieves a 1.3% and 2.4% improvement in terms of mAP50 and mAP75, respectively. This demonstrates that our method can effectively mine complete instances, resulting in improved performance for stricter metrics, i.e., mAP75.

Impact of Cascaded Threshold

Furthermore, the cascaded threshold design allows our method to operate in a coarse-to-fine manner. As shown in the last row of Table 4, the cascaded threshold results in a 1.0% and 2.3% improvement in terms of mAP50 and mAP75, respectively, as it guides deeper Refinement branches to focus on more complete instances.

We also evaluate the seeds and pseudo ground truth generated by CIM, as shown in Figure 4. It is obvious that pseudo ground truth outperforms seeds and deeper Refinement branch performs better. As reported in the Table 5, the effect of different cascade thresholds is small.

Impact of Anti-Noise Strategy

Following UFO2 [Ren et al., 2020b], we treat the precomputed pseudo labels as ground truth and apply a KLdivergence loss on the Anti-noise head, which reduces its performance to 50.4% in terms of mAP50. This result suggests that redundant parameters in the Anti-noise head effectively filter out noise in pre-computed pseudo labels.

To further evaluate the impact of Anti-noise sampling, we introduce noise by increasing the number of seeds in the CIM process. As shown in Table 6, the use of Anti-noise sampling increases the robustness of our method.

5 Conclusion

In this paper, we propose a proposal-based approach in an online refinement manner to address the redundant segmentation problem. Our method incorporates the MaskIoU head and utilizes the CIM strategy to mine complete instances without resorting to RW or CRF. Additionally, we propose the Anti-noise strategy to filter out noise in pseudo labels. Our approach demonstrates state-of-the-art performance on the VOC 2012 and COCO datasets. Moving forward, we expect to investigate methods to enhance the robustness of the model without resorting to Anti-noise sampling strategy that may complicate the analysis.

Acknowledgments

This work was funded by Natural Science Foundation of China under Grants No. 62076099 and No. 61703166.