Beyond the Prototype: Divide-and-conquer Proxies for Few-shot Segmentation

1 Introduction

Benefiting from the superiority of deep convolutional neural networks, various computer vision tasks have made tremendous progress over the past few years, including image classification [He et al., 2016], object detection [Ren et al., 2015], and semantic segmentation [Long et al., 2015], to name a few. However, such an effective technique also has its inherent limitation: the strong demand for a considerable number of annotated samples from well-established datasets to achieve satisfactory performance [Deng et al., 2009]. For the tasks such as semantic segmentation that performs dense predictions on given images, the acquisition of training sets with sufficient data costs even more, substantially hindering the development of deep learning systems. Few-shot learning (FSL) has emerged as a promising research direction to address this issue, which intends to learn a generic model that can recognize new concepts with very limited information available [Vinyals et al., 2016] [Lang et al., 2022].

* Equal contribution.

† Corresponding author.

In this paper, we undertake the few-shot segmentation (FSS) task that can be regarded as a natural application of FSL technology in the field of image segmentation [Shaban et al., 2017]. Specifically, FSS models aim to retrieve the foreground regions of a specific category in the query image while only providing a handful of support images with corresponding annotated masks. Prevalent FSS approaches [Zhang et al., 2019; Tian et al., 2022] follow the prototype learning paradigm to perform meta-reasoning, where single or multiple prototypes are generated by the average pooling operation [Zhang et al., 2020], furnishing query object inference with essential foreground cues. However, their segmentation capabilities could be fragile when confronted with daunting few-shot tasks with factors such as blurred object boundaries and irrelevant object interference. We argue that one possible reason for these failures lies in the underexplored information from support image-mask pairs, and it is far from enough to rely solely on squeezed object-related features to guide segmentation. Taking the 1-shot segmentation task in Figure 1 as an example, conventional approaches tend to confuse several distractor objects (e.g., cat and sofa) and yield inaccurate segmentation boundaries (e.g., dog's paw and head), revealing the limitations of the scheme based on such descriptors.

To alleviate the abovementioned problems, we propose a simple yet versatile framework in the spirit of divide-andconquer (refer to the right part of Figure 1), where the descriptors for guidance are derived from a broader episodic perspective, rather than just the objects themselves. Concretely, we first implement a novel self-reasoning scheme on the annotated support image, and then divide the coarse segmentation mask into multiple regions with different properties. Using effective masked average pooling operations, a series of support-induced proxies are thus generated, each playing a specific role in conquering the typical challenges of daunting FSS tasks. Furthermore, we devise a unique parallel decoder structure in which proxies with similar attributes are integrated to boost the discrimination power. Compared with conventional prototype-based approaches, our proposed framework, named divide-and-conquer proxies (DCP), allows for the development of appropriate and reliable information as a guide at the "episode" level, not just about the object cues.

Our primary contributions can be summarized as follows:

• We present a simple yet versatile FSS framework in the spirit of divide-and-conquer, where the descriptors for guidance are derived from a broader episodic perspective to tackle the typical challenges of daunting tasks.
• Compared with conventional prototype-based approaches, the proposed framework allows for the development of appropriate and reliable information, rather than merely the object cues themselves, providing a fresh insight for future works.
• Extensive experiments on two standard benchmarks demonstrate the superiority of our framework over the baseline approach (up to 5∼10% on average), establishing new state-of-the-arts in few-shot literature.

2 RelatedWorks

Semantic Segmentation. Semantic segmentation is a fundamental and essential computer vision task, the goal of which is to classify each pixel in the image according to a predefined set of semantic categories. Most advanced approaches follow fully convolutional networks (FCNs) [Long et al., 2015] to perform dense predictions, vastly boosting the segmentation performance [Zhao et al., 2017]. On top of that, the rapid development of this field has also brought some impressive techniques, such as dilated convolutions [Chen et al., 2017] and attention mechanism [Huang et al., 2019]. In this work, we adopt the Atrous Spatial Pyramid Pooling (ASPP) module [Chen et al., 2017] based on dilated convolution to build the decoder, enjoying a large receptive field and multi-scale feature aggregation.

Few-shot Learning. Few-shot learning (FSL) is proposed to tackle the generalization problem of conventional deep networks, where meta-knowledge is shared across tasks to facilitate the recognition of new concepts with very little support information. Representative FSL approaches can be grouped into three categories: optimization-based method [Finn et al., 2017], hallucination-based method [Chen et al., 2019], and metric-based method [Vinyals et al., 2016]. Our work closely relates to the prototypical network (PN) [Snell et al., 2017] assigned to the third category. PN directly leverages the feature representations (i.e., prototypes) computed through global average pooling operations to perform non-parametric nearest neighbor classification. Our DCP, on the other hand, utilizes foreground/background prototypes for dense feature comparison [Zhang et al., 2019], providing essential information for subsequent parametric query image segmentation.

Few-shot Segmentation. Few-shot segmentation (FSS) has emerged as a promising research direction to tackle the dense prediction problem in a low-data regime. Existing approaches typically employ a two-branch structure, in which the annotation information travels from the support branch to the query branch [Zhang et al., 2019] [Siam et al., 2019]. Generally speaking, how to achieve more effective information interaction between the two branches is the primary concern in this field. OSLSM [Shaban et al., 2017], the pioneering work of FSS, proposed to generate classifier weights for query image segmentation in the conditioning (support) branch. Whereafter, the prototype learning paradigm [Snell et al., 2017] was introduced into state-of-the-art approaches for better information exchange. SG-One [Zhang et al., 2020] computed the affinity scores between the prototype and query features, yielding the spatial similarity map as a guide. CANet [Zhang et al., 2019] proposed to conduct a novel dense comparison operation based on the prototype, which exhibited decent segmentation performance under few-shot settings. However, these methods are limited to the utilization of features of foreground objects (i.e., prototype) to facilitate segmentation and fail to provide pertinent and reliable guidance from a broader episodic perspective. Our method draws on the idea of prototype learning while proposing a set of support-induced proxies that go beyond it.

3 Problem Setup

Few-shot segmentation (FSS) aims to segment the objects of a specific category in the query image using only a few labeled samples. To improve the generalization ability for unseen classes, existing approaches widely adopt the metalearning strategy for training, also known as episodic learning [Vinyals et al., 2016]. Specifically, given a training set Dtrain and a testing set Dtest that are disjoint with regard to object classes, a series of episodes are sampled from them to simulate the few-shot scenario. For a K-shot segmentation task, each episode is composed of 1) a support set S = {(Iks ,Mks,c )}k=1,...,K c∈Cepisode , where Iks is the k-th support image, Mks,c is the corresponding mask for class c, and Cepisode represents the class set of the given episode; and 2) a query set Q = {Iq,Mq,c} where Iq is the query image andMq,c is the ground-truth mask available during training while unknown during testing.

4 Method

In this section, we propose a simple yet effective few-shot segmentation framework, termed divide-and-conquer proxies (DCP). The unique feature of our approach, as also the source of improvements, lies in the spirit of divide-and-conquer. The divide module generates a set of proxies according to the prediction and ground-truth mask of the support image; the conquer module utilizes these distinguishing proxies to provide essential segmentation cues for the query branch, as depicted in Figure 2.

4.1 Divide

Self-reasoning Scheme. Following the paradigm of advanced FSS approaches [Zhang et al., 2019], we freeze the backbone network to boost generalization capabilities. Taking ResNet [He et al., 2016] for example, the intermediate feature maps after each convolutional block of the support branch are represented as {Fbs }Bb=1 *. We use the high-level features generated by the last block, i.e., block4, to perform the self-reasoning scheme:

where fseg(·) is a lightweight segmentation decoder with residual connections, including three convolutions with 256 filters, one convolution with 2 filters, and an argmax(·) operation for producing coarse prediction masks ˆMaux s ∈{0, 1}H×W . H and W denote the height and weight respectively, and their product H×W is the minimum resolution of all feature maps. Unlike the widely adopted prototypeguided segmentation framework, such a mask-agnostic selfreasoning scheme is more efficient and can also provide reliable auxiliary information.

Proxy Acquisition. Given the coarse prediction ˆMaux s and down-sampled ground-truth mask Ms of the support image, we first evaluate their differences to derive the corresponding mask (i.e., valid region) Mm∈P1 of each proxy where P1={α, β, γ, δ} denotes the index set of proxies:

*B denotes the total number of convolutional blocks contained in the backbone network, which is equal to 4 for ResNet.

where (x, y) indexes the spatial location of the mask, 1(·) is an indicator function that outputs 1 if the condition is satisfied or 0 otherwise, and G∈RH×W represents an all 1's matrix with the same shape as Mm. Mα and Mβ aggregate the main and auxiliary foreground information respectively, generally corresponding to the body and boundary regions of objects; while Mγ and Mδ gather the primary and secondary background information respectively, which typically correspond to the regions of the generalized background and distractor objects. As described above, these masks are generated according to the segmentation results of the model, therefore the main (primary) and auxiliary (secondary) status may change when encountering complex tasks. The extreme case is Mβ=Ms, but even so, the guidance information is still complete. To better understand the effect of each mask, we present several examples in Figure 3.

Then, we conduct masked average pooling operations [Zhang et al., 2020] on the support features Fs∈RC×H×W and masks Mm computed above to encode the set of proxies {pm}:

where pm∈RC represents a specific proxy in the set, playing some roles in conquering the typical challenges of FSS tasks.

Prototype Acquisition. Similar to the acquisition method of proxies (Eq. (3)), the set of prototypes {pn}n∈P2 are also generated using the masked average pooling operation where P2={f, b} represents the index set of prototypes:

where j=1 if the foreground prototype pf is evaluated and j=0 if the background one pb is calculated.

4.2 Conquer

Feature Matching. Dense feature comparison is a common approach in the FSS literature that compares the query feature maps Fq with the global feature vector (i.e., prototype) at all spatial locations. In view of its effectiveness, we follow the same technical route and extend it to a dual-prototype setting (see Figure 3(b)), which can be defined as:

where ζh×w(·) expands the given vector to a spatial size h×w and ⊕ represents the concatenation operation along channel dimensions. The superscript "tmp" denotes temporary, and the subscript n∈P2 indicates the index of the prototype.

Feature Activation. Given the proxies {pm}m∈P1 and prototypes {pn}n∈P2 computed by Eqs. (3-4), we evaluate the cosine similarity between each of them and query features at each spatial location respectively:

where Al∈RH×W denotes the activation map of the l-th item. Figure 4 presents some examples to illustrate the region of interest for each feature vector pl. Actually, there is an alternative approach to propagate information about feature vectors {pl}l∈P1∪P2 to the query branch, that is, to perform the same operation as Eq. (5). Such a scheme, however, introduces additional computational overhead and performance degradation, as discussed in § 5.3.

Parallel Decoder Structure. Most previous works abandon the prototype pb due to the complexity of background regions in support images. We argue that specifically designing the network to capture foreground and background features separately could have a positive effect on performance, whereas a simple fusion strategy would be counterproductive. To this end, a unique parallel decoder structure (PDS) is devised in which proxies/prototypes with similar attributes are integrated to boost the discrimination power:

where Ptmp is a temporary set of vector indexes, representing {α, β} when foreground features Frefined q;f are evaluated and {γ, δ} when background ones Frefined q;b are computed. These two refined query features are then fed into the corresponding decoder network respectively, and the output single-channel probability maps are concatenated to generate the final segmentation result.

4.3 Training Loss

To guarantee the richness of the information provided by proxies while preventing extreme situations (§ 4.1), we introduce additional constraints on the prediction results of selfreasoning schemes for both support and query branches. The total loss for each episode can be written as:

where L represents the binary cross-entropy (BCE) loss function. λ1 and λ2, the balancing coefficients, are set to 1.0.

5 Experiments

5.1 Setup

Datasets. We evaluate the proposed approach on two standard FSS benchmarks: PASCAL-5i [Shaban et al., 2017] and COCO-20i [Nguyen and Todorovic, 2019]. The former is created from PASCAL VOC 2012 [Everingham et al., 2010] with additional mask annotations from SDS [Hariharan et al., 2011], consisting of 20 semantic categories evenly divided into 4 folds: {5i }3i=0 ; while the latter, built fromMS COCO [Lin et al., 2014], is composed of 80 semantic categories divided into 4 folds: {20i }3i=0 . Models are trained on 3 folds and tested on the remaining one in a cross-validation manner. We randomly sample 1, 000 episodes for evaluation.

Evaluation Metrics. Following previous FSS approaches [Li et al., 2021; Tian et al., 2022], we adopt mean intersectionover-union (mIoU) and foreground-background IoU (FBIoU) as our evaluation metrics, in which the mIoU metric is primarily used since it better reflects the overall performance of different categories.

Implementation Details. Two different backbone networks (i.e., VGG16 [Simonyan and Zisserman, 2014] and ResNet50 [He et al., 2016]) are adopted for a fair comparison with existing FSS methods. Following [Tian et al., 2022], we pre-train these backbone networks on ILSVRC [Russakovsky et al., 2015] and freeze them during training to boost the generalization capability. The setup of pre-processing technology is the same as [Tian et al., 2022]. The SGD optimizer is utilized with a learning rate of 0.005 for 200 epochs on PASCAL-5i and 50 epochs on COCO-20i . All experiments are performed on NVIDIA RTX 2080Ti GPUs with the PyTorch framework. We report the average results of 5-runs with different random seeds to reduce the influence of selected support-query image pairs on performance.

5.2 Comparison with State-of-the-arts

We evaluate the proposed DCP framework with state-of-theart methods on two standard FSS datasets. Table 1 presents the 1-shot and 5-shot results assessed under the mIoU metric on PASCAL-5i . It can be observed that our models outperform existing approaches by a sizeable margin, especially under the 5-shot setting, indicating that more appropriate and reliable information is supplied for query image segmentation. Table 2 summarizes the performance of different methods with ResNet50 backbone on COCO-20i , in which the proposed approach sets new state-of-the-arts with a small number of learnable parameters. Specifically, our DCP achieves 2.89% (1-shot) and 3.78% (5-shot) of mIoU improvements over the previous best approach (i.e., PFENet), respectively.

For qualitative analysis, we first visualize the segmentation results of DCP and baseline approach, as illustrated in Figure 5. It can be found that the false positives caused by irrelevant objects are significantly reduced (see the first two rows). Meanwhile, more accurate segmentation boundaries are obtained, benefiting from the superiority of pβ (see the last row). Note that for more details of the baseline approach, please refer to Table 3 in the ablation studies (§ 5.3).

5.3 Ablation Studies

The following ablation studies are performed with VGG16 backbone on PASCAL-5i unless otherwise stated.

Proxy & Prototype. As discussed in § 4.2, the derived proxies, along with foreground/background prototypes, tend to propagate pertinent information for the query branch in the form of activation maps. In Table 3, we evaluate the impact of each of them on segmentation performance under 1-shot settings. Overall, the following observations could be made: (i) By introducing the foreground activation map Mf , the performance of baseline approach is greatly improved from 58.52% to 59.64%+1.12, demonstrating the importance of foreground cues for guidance. Furthermore, we argue that such a simple yet powerful variant could serve as a new baseline approach for future FSS works. (ii) The background information provided by mask annotations needs to be used with caution since the performance might even deteriorate without PDS (see the second row). (iii) Our proposed proxies show a superior capability compared to the prototypes in facilitating query image segmentation (60.12% vs. 60.63% mIoU), which is the so-called "beyond the prototype". (iv) There is a complementary relationship between the prototype and proxy, the combination of which could further boost performance. (v) Surprisingly, background-related proxies could be better integrated with prototypes (see rows 6-7). One possible reason for this phenomenon is that the interference of irrelevant objects may have a greater influence on the FSS model among many factors, while Mδ can effectively reduce false positives. Parallel Decoder Structure. After obtaining the activation maps (Eq. (6)), we first attempt to leverage a simple fusion scheme to guide the query branch, i.e., concatenating Ftmp q;f and {Al}l∈P1∪P2 and feeding the refined features to a single decoder network. Unfortunately, such a fusion scheme does not offer performance gains, which we attribute to the information confusion between the expanded foreground prototype ζH×W (pf) and background-related activation maps (Mγ and Mδ). To this end, we devise a unique parallel decoder structure (PDS) where proxies/prototypes with similar attributes are integrated, yielding a superior result compared to the previous scheme (see rows 3-4 of Table 3).

Efficiency. We also evaluate the efficiency of different fusion strategies between derived proxies {pm}m∈P1 and query features Fq, as presented in Table 4. Please note that the first row represents the previously discussed scheme (see Eq. (7)), while the second row denotes the scheme that concatenates query features with the expanded proxies, similar to Eq. (6). The experiment results indicate that the first fusion strategy exhibits remarkable segmentation performance with fewer parameters (0.26M) and faster speed (16+FPS), demonstrating the effectiveness of activation maps for guidance.

6 Conclusion

We have proposed an efficient framework with the spirit of divide-and-conquer, which performs a self-reasoning scheme to derive support-induced proxies for tackling the typical challenges of FSS tasks. Qualitative and quantitative results show that the proposed proxies could provide appropriate and reliable information to facilitate query image segmentation, surpassing conventional prototype-based approaches by a considerable margin, i.e., the so-called "beyond prototype".

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grants 62136007 and U20B2065, in part by the Shaanxi Science Foundation for Distinguished Young Scholars under Grant 2021JC-16, and in part by the Fundamental Research Funds for the Central Universities.