MISA: MIning Saliency-Aware Semantic Prior for Box Supervised Instance Segmentation

1 Introduction

Instance segmentation, which aims to locate different objects and assign them pixel-wise masks, is a fundamental task in vision. The efficacy of instance segmentation methods operating in various paradigms (e.g, query-based [Wang et al., 2020b], top-down [Chen et al., 2019a], and bottom-up [Gao et al., 2019]) has experienced a swift evolution in the vision community. Nonetheless, these methods heavily rely on labour intensive manual mask annotations, which incur substantial time costs. In contrast, coarse object annotations significantly reduce annotation costs, e.g, the average time cost of point-level and box-level annotations for each instance is

∗Corresponding author.

9s and 7s respectively, which is much lower than that of masklevel annotation (79.2s) [Cheng et al., 2022b].

For a trade-off between annotation costs and segmentation performance, our work focuses on box-supervised instance segmentation (BSIS). Given the limited information from bounding boxes, most current methods model the pairwise affinity at the image level to obtain extra prior knowledge [Hsu et al., 2019; Tian et al., 2021], and propagate the pairwise affinity globally and locally [Li et al., 2022b; Li et al., 2023b]. Although these methods mitigate the challenges arising from the lack of mask supervision, there still exists a significant gap compared to fully supervised approaches. The main reason is that the objective of instance segmentation task is inherently more complicated, the net-

form instance discrimination. Consequently, the model may sacrifice the perceptual ability for object shape and contour in order to recognize multiple instances, leading to suboptimal segmentation results. This issue becomes even more serious due to the lack of accurate mask supervision. An intuitive comparison is shown in Figure 1, the performance of BSIS network is obviously inferior to box-supervised semantic segmentation (BSSS) when neglecting instance discrimination. By straightforwardly focusing on the segmentation task, the model demonstrates better segmentation capability and may provide valuable guidance for addressing complex segmentation tasks (i.e, BSIS) in weakly supervised setting.

In this work, we present MISA to address box supervised instance segmentation task. We MIne the Saliency-Aware semantic prior knowledge within dense feature of a welloptimized BSSS model, and perform cross-model guidance onto the query-based BSIS model to facilitate its representation learning process, thereby enhancing model's segmentation capability. The proposed MISA constitutes of two components: (1) a Frequency-Space Distillation (FSD) module, and (2) Semantic Enhanced Pairwise Affinity (SEPA).

As for the Frequency-Space Distillation (FSD) module, we extract assorted salient semantic prior knowledge from BSSS model and perform cross-model alignment aiming to transfer the salient prior to BSIS model. Specifically, we decompose the dense feature representations of segmentation networks into high-frequency and low-frequency components respectively. Then we acquire the high-frequency features of object regions and the low-frequency features of background regions according to the ground-truth bounding boxes. Next, we align the aforementioned features between BSIS and BSSS networks to facilitate the BSIS model in capturing essential semantic prior by minimizing feature level Mean Squared (L2) loss. Rich textural knowledge in the high-frequency component of the object features drives the network towards accurately segmenting edges, while the low-frequency component of background features implies global structural information, which prevents false positive predictions.

As for the Semantic Enhanced Pairwise Affinity (SEPA), we borrow the object perceptual ability of the BSSS network to emphasize the contribution of salient foreground objects for pairwise affinity, providing more accurate guidance for the BSIS network. In particular, inspired by class activation maps (CAMs) [Zhou et al., 2016], we extract the classifier weights corresponding to different object categories and utilize them to reweight the semantic features, highlighting the contributions of different channels in discriminating foreground objects. The modulated saliency-aware features then cooperated with the low-level cues to model the pairwise affinity, which will propagate on the predicted mask to effectively guide the BSIS model.

Overall, our contributions can be summarized as follows:

• We propose MISA to guide the query-based box supervised instance segmentation during learning process via MIning Saliency-Aware semantic prior from BSSS model. To the best of our knowledge, this is the first work to introduce cross-model guidance to boost the performance of BSIS.
• Two novel technologies are presented: (1) a FrequencySpace Distillation (FSD) module, and (2) Semantic Enhanced Pairwise Affinity (SEPA). Both technologies explore the salient semantic prior within the dense feature of the BSSS model from different perspectives, so as to provide valuable guidance that enhances the segmentation capability of BSIS model.
• Extensive experimental results on various benchmarks

demonstrate that our proposed method remarkably outperforms existing state-of-the-art BSIS methods. Our work narrows down the gap in performance between full mask and box supervised instance segmentation.

2 RelatedWorks

2.1 Instance Segmentation

Instance segmentation aims to segment and recognize each foreground object from an image. Existing instance segmentation methods can be roughly categorized into query-based, top-down and bottom-up paradigms [Wang et al., 2022]. Topdown approaches [He et al., 2017; Chen et al., 2019a] detect target objects and further utilize semantic segmentation algorithms to achieve pixel-level results. Bottom-up methods [Gao et al., 2019] accomplish segmentation by clustering similar pixels and grouping them into distinct objects. Recently, query-based methods [Wang et al., 2020b; Cheng et al., 2022a] have emerged as state-of-the-art paradigm, which introduce a set of learnable query vectors to decode predicted masks from dense feature representations. Our work builds upon the query-based paradigm.

2.2 Box Supervised Segmentation

Box supervised segmentation aims to accomplish semantic or instance segmentation tasks using only bounding box annotations.It has gained growing attention due to its balance between segmentation performance and annotation costs [Shen et al., 2023]. For box supervised semantic segmentation (BSSS), Box2Seg [Kulharia et al., 2020] utilizes GrabCut [Rother et al., 2004] to refine CAMs [Zhou et al., 2016] and combines those with attention maps to jointly guide the segmentation network. BAP [Oh et al., 2021] aims to acquire high-quality pseudo masks by excluding background regions from the bounding boxes. For box supervised instance segmentation (BSIS), BoxInst [Tian et al., 2021] assumes that adjacent pixels with similar colors are likely belong to the same instance. Recently, several approaches [Li et al., 2023b; Li et al., 2022b] utilize low-level image cues to model pairwise affinity, and propagate it on the predicted masks to generate pseudo labels. In our work, we focus on mining salient semantic prior from a well-optimized BSSS network to boost the segmentation ability of BSIS network.

2.3 Knowledge Distillation

Knowledge distillation [Hinton et al., 2015] aims to compress the complexity of the model by aligning soft labels between a cumbersome model and a compact model. This paradigm has been widely adopted in object detection and semantic segmentation. SSTKD [Ji et al., 2022] transfers the low-level structural and statistical texture information from teacher to student model individually. SKD [Liu et al., 2019] proposes to capture structured information between pixels by constraining both pairwise similarity and holistic correlations. DeFeat [Guo et al., 2021] highlights the significance of both foreground and background during the learning process, and distilling them separately to yield more prominent results. In contrast, our method focuses on performing a cross-model distillation to boost the segmentation performance of BSIS.

3 Methodology

In this section, we introduce our proposed MIning SaliencyAware semantic prior (MISA) for box supervised instance segmentation. The overall architecture of MISA is shown in Figure 2. In Section 3.1, we review the semantic segmentation and query-based instance segmentation methods. In Section 3.2, we present the Frequency-Space Distillation (FSD) module to effectively extract and transfer the salient prior knowledge within dense feature of BSSS model. In Section 3.3, we introduce the Semantic Enhanced Pairwise Affinity (SEPA) to emphasize the contribution of salient objects for modeling more accurate pairwise affinity.

3.1 Preliminary

Semantic Segmentation

Most semantic segmentation methods employ an encoderdecoder architecture. A training image is fed into an encoder to obtain low-resolution feature maps, then the feature are continuously upsampled in decoder, generating the semantic dense feature representation S ∈ RH×W×D to obtain prediction mask:

where Msem ∈ [0, 1]H×W×C denotes the prediction semantic probability map, D refers to the channels of semantic feature map, C denotes the number of categories. '∗' represents a convolution operation and K = {k1 , ..., kC } ∈ RC×D refers to a 1 × 1 conv with C convolutional kernels.

Query-based Instance Segmentation

Query-based instance segmentation methods often feed the extracted image feature into both the query branch and mask feature branch, yielding corresponding instance-aware embeddings Q = {q1 , ..., qN } ∈ RN×D′ and dense feature representations I ∈ RH×W×D′ for generating predicted masks:

where N denotes the number of positive sample grid points in the training image (i.e, SOLOv2), Mn ∈ [0, 1]H×W×1 is the prediction mask corresponding to the nth grid point.

3.2 Frequency-Space Distillation

The purpose of proposed Frequency-Space Distillation module is to explore the salient prior knowledge within dense feature of a well-optimized BSSS model, and imbue the BSIS network with extracted valuable salient prior. As shown in Figure 3, to capture texture knowledge and global structural information within the dense feature representations, we first apply spectral modulation in Fourier domain [Si et al., 2023] to extract the high-frequency and low-frequency components of the dense feature separately:

where F is the dense feature presentations (i.e, S and I), FFT(·) and IFFT(·) denote Fourier transform and inverse Fourier transform respectively. α ∈ [0, 1]H×W is the Fourier mask, such as for high-pass filtering:

here θ is the frequency-dependent scaling factor and t denotes the threshold frequency radius.

Texture knowledge in the high-frequency component of the foreground regions is beneficial for recognizing object contours, while global structural information in the lowfrequency component of the background regions helps reduce false positive predictions. Based on this, we decompose the high-frequency foreground regions and the low-frequency background regions of the dense feature representations separately. Let B ∈ {0, 1}H×W be a binary mask, where B(i, j) = 1 if the location (i, j) belongs to any bounding boxes in a training image and B(i, j) = 0 otherwise. The decomposed dense feature representations can be calculated as follows:

Finally, the decomposed feature representations between BSIS and BSSS network will be aligned via minimizing the Mean Squared (L2) loss to accomplish salient prior transfer:

where λbg and λfg denote the loss coefficients in background and foreground regions, Nbg and Nfg refer to the number of pixels in background and foreground regions. We adopt adaptation and projector layers [Yang et al., 2022] to adjust the dense feature of BSIS before frequency-space decomposing.

3.3 Semantic Enhanced Pairwise Affinity

Since ground-truth masks are not available, most previous research leverage the color consistency assumption that pixels with similar colors correspond to shared labels [Lin et al., 2016; Tian et al., 2021]. In this context, some studies [Araslanov and Roth, 2020; Li et al., 2023b] propagate lowlevel pixel color affinity on predicted masks M to generate pseudo labelsMˆ , formulated as below:

where Aij,kl is the low-level pairwise affinity between pixels at locations (i, j) and (k, l), V (i, j) and V (k, l) represent RGB vectors, σV denotes the standard deviation of V , N represents the scope of the various receptive fields (e.g, 8way local neighbors [Ru et al., 2022] and global pixels [Li et al., 2023b]). wij is the normalization coefficient:

However, in cases where the object and background have similar color intensities, the color consistency assumption become inapplicable [Li et al., 2023a]. To acquire more robust prior knowledge, we employ dense feature S, obtained from a well-optimized BSSS model (from Equation 1), to establish high-level feature affinity, denoted as Af :

Nevertheless, the formulation of Eq. (10) indicates that each feature vector contributes equally to the pairwise affinity. This lacks concerns on the impact of salient foreground objects, which can incorporate undesired noise from the background regions. To provide more accurate cross-model guidance, we further mine deep semantic prior from the BSSS network and modulate the deep feature to strengthen the awareness of target categories. To specify, inspired by CAMs [Zhou et al., 2016], we first extract the classifier parameters K ∈ RC×D , which map D-dimensional dense feature to C-dimensional score map, encapsulating the contributions of different feature channels to different categories.

We view K as the saliency factors and then use it to reweight the channel of dense feature for each foreground category c to effectively reduce noise and amplify the distance between foreground and background vectors:

where Kc represents the cth entry of the K. Consequently, the high-level feature pairwise affinity for each foreground category c can be calculated as below:

The ultimate Semantic Enhanced Pairwise Affinity Ac ˆ , corresponded to category c, is defined as:

Therefore, the propagated pseudo labels Mn ˆ in Equation 7 with ground-truth category c can be adjusted as:

Finally, we adopt the Mean Absolute (L1) loss between predicted masks M and propagated pseudo labels Mˆ to guide the BSIS network effectively.

4 Experiments

4.1 Datasets

PASCAL VOC 2012

The PASCAL VOC 2012 dataset [Everingham et al., 2010] includes 20 object categories. This dataset is divided into training and validation subsets, with 10,582 images for training and 1,449 images for validation.

COCO

The COCO dataset [Lin et al., 2014] is widely used in image segmentation task. It comprises 80 different object categories and contains a training set of 110k images, a validation set of 5k images, and a testing set of 20k images.

4.2 EvaluationMetrics

We use the standard COCO style metrics to evaluate the proposed method. It contains mask AP (averaged over IoU thresholds), AP50, AP75 (mask AP at different IoUs), and APS, APM, APL (mask AP at different scales). All AP is calculated using mask IoU [He et al., 2017].

4.3 Implementation Details

Our proposed method is implemented in Pytorch [Paszke et al., 2017] with mmcv/mmdet [Chen et al., 2019b] repository. We choose SOLOv2 [Wang et al., 2020b] and Mask2Former [Cheng et al., 2022a] as the query-based instance segmentation frameworks, and we default to using SOLOv2 unless specified. The different backbones of the model (i.e, ResNet [He et al., 2016] and Swin Transformer [Liu et al., 2021]) are pretrained on ImageNet [Russakovsky et al., 2015]. We train our model on 8 GPUs with a batch size of 16, and adopt AdamW as the optimizer with the initial learning rate set to 1.2 × 10-4 and weight decay set to 0.05.

We also apply the projection loss term [Tian et al., 2021; Li et al., 2022a] to constrain the predicted mask. Following Apro[Li et al., 2023b], we propagate the Semantic Enhanced Pairwise Affinity (SEPA) locally and globally to obtain pseudo labels. We set θ = 0.2 and t = 10 in Equation 4. The affinity balanced coefficient in Equation 13 is set to 0.7. In the objective function of Equation 6, we set λbg = 6 and λfg = 10 by default. The data augmentation strategies follow the default settings recommended in mmdet. We adopt the SeMask [Jain et al., 2023] as the segmentation framework of BSSS, and we align the backbone of the BSSS and BSIS models to better comprehend the impact of our approach in comparative experiment. The BSSS model is used solely for auxiliary training and is not utilized during inference.

4.4 Main Results

Results on COCO

We compare MISA with the state-of-the-art BSIS methods on the COCO test-dev dataset in Table 1. Our proposed method demonstrates remarkable superiority over other methods across various backbones. Specifically, MISA outperforms BoxInst [Tian et al., 2021] and DiscoBox [Lan et al., 2021] by 2.8% and 2.9% mask AP with SOLOv2 framework and ResNet-50 backbone. It obtains 36.0% mask AP, which is higher than BoxLevelSet and Box2Mask by 2.6% and 1.8% mask AP with ResNet-101 backbone. With more powerful backbone and query-based framework (i.eSwin Transformer [Liu et al., 2021] and Mask2Former [Cheng et al., 2022a]), MISA exhibits outstanding performance with 42.0% mask AP, surpassing existing state-of-the-art models such as SIM [Li et al., 2023a], BoxTeacher [Cheng et al., 2023], and Apro [Li et al., 2023b]. This is attributed to MISA's ability to mine and leverage salient semantic prior knowledge as guidance, effectively mitigating the influence of noise and boosting the the perceptual ability for object shape and contour.

Results on PASCAL VOC 2012

We also conduct experiments on PASCAL VOC 2012 val dataset to further validate the effectiveness and generalization capability of our method. As shown in Table 2, our proposed MISA demonstrates superior performance compared to recent BSIS methods based on different architectures.

Qualitative Results

We show the visualization results of the different pairwise affinity in Figure 4. The examples provide an intuitive illustration of the limitations of color affinity. Specifically, the color affinity in Figure 4(a) erroneously propagates from the snowboard to background due to their similar appearances. In Figure 4(b), the propagation of color affinity is incorrectly interrupted at the black-and-white boundary of the dog's fur. In contrast, our proposed SEPA mitigates the issues arising from misleading color prior and yields more robust propagation results.

Figure 5 presents visualization results that demonstrate the performance of our proposed MISA on the COCO val split. It should be emphasized that our method consistently demonstrates robust object segmentation capabilities even in intricate scenarios and multiple-instance images.

4.5 Ablation Study

We conduct ablation studies on PASCAL VOC 2012 val split to verify the effectiveness of each module in MISA. We adopt ResNet-50 and SOLOv2 as the baseline, and the mask AP at different IoU thresholds are reported. Additionally, we discuss the sensitivity of hyperparameters and report the relevant experiments in the supplementary materials.

Contributions of FSD and SEPA

Table 3 shows the contributions of the two components introduced in our work (i.e, FSD and SEPA). Relying solely on low-level cues to model pairwise affinity [Li et al., 2023b], the method achieves mask AP of only 38.3%. After incorporating the FSD module, the performance is improved to 39.5% mask AP. When propagating the SEPA to obtain pseudo labels, our method reaches 39.3% mask AP. When both the FSD and SEPA modules are employed, the performance is significantly boosted to 39.9% mask AP. This is due to the combination of FSD and SEPA provides a valuable prior guidance for BSIS network.

Distillation Schemes for FSD

We investigate the impact of frequency domain decomposition and foreground-background decoupling schemes on the performance. As shown in Table 4, using only feature level distillation yields a limited 0.4% mask AP improvement. After incorporating the frequency domain decomposition and foreground-background decoupling modules, there are 0.1% and 0.3% mask AP improvements respectively. When the aforementioned modules are combined, the performance is further enhanced to 39.5% mask AP. This suggests that FSD enables BSIS to capture salient prior knowledge from BSSS.

Different Pairwise Affinities for Propagation

We validate the efficacy of SEPA using two propagation modes (i.e, PAMR [Araslanov and Roth, 2020] and Apro [Li et al., 2023b]. PAMR employs the 8-way neighbors propagation scope, whereas Apro performs affinity propagation in both global and local contexts. As show in Table 5, the use of semantic feature improves the performance by 0.2% and 0.4% mask AP for PAMR and Apro respectively. After using the saliency factors to modulate the semantic feature, the results are largely improved by 0.6% and 1.0% mask AP. This indicates that SEPA can generate more accurate pseudo labels in different propagation modes.

5 Conclusion

In this work, we indicate that owing to the intricate objective and limited guidance, the box supervised instance segmentation (BSIS) model often seeks a trade-off by significantly sacrificing its segmentation capability to recognize multiple instances. To this end, we present MIning Saliency-Aware semantic prior (MISA) from a well-optimized box supervised semantic segmentation (BSSS) network, and incorporating cross-model guidance into the training process of BSIS, so as to boost the segmentation ability of BSIS model. The proposed MISA consists of two technologies, (1) a FrequencySpace Distillation (FSD) module, and (2) Semantic Enhanced Pairwise Affinity (SEPA). Both technologies make efforts to mine the salient semantic prior within the dense feature of the BSSS model from different perspectives and provide valuable guidance for BSIS model. The experimental results on different benchmarks demonstrate the superior performance of our proposed method. Taking a broader view, our proposed MISA shows a novel perspective for exploring cross-model guidance to boost the model capability of BSIS.

Acknowledgments

This work is supported by National Key R&D Program of China (2023YFD2000303) and National Natural Science Foundation of China (62372433, 62072438).

Contribution Statement

Hao Zhu and Yan Zhu contributed equally to this paper.