FGNet: Towards Filling the Intra-class and Inter-class Gaps for Few-shot Segmentation

1 Introduction

Brilliant efforts have been made in image semantic segmentation [Long et al., 2015; Szegedy et al., 2017; Badrinarayanan et al., 2017; Chen et al., 2017; Yu et al., 2020], achieving excellent performance in several large-scale labeled datasets [Silberman et al., 2012; Zhou et al., 2017; Cordts et al., 2016]. However, current top-performing approaches rely heavily on extensive pixel-wise annotations, which is time-consuming and labour-intensive. To handle this issue, few-shot segmentation (FSS) [Shaban et al., 2017] has received lots of attention in recent years.

FSS is an extension task of few-shot learning, aiming to learn the generalization ability from the given classes and adapt it to the arbitrary novel classes with only a handful of support samples. The mainstream strategy of FSS follows the pattern of metric learning [Dong and Xing, 2018; Wang et al., 2019; Liu et al., 2020] based on a global descriptor, named prototype. In particular, the prototype denotes a representation vector of a specific category, and the prototype-based methods generate a representative prototype for each category from the limited support samples. Then the prototype is leveraged to activate the query feature for predicting the mask of the query image.

* Corresponding author

Code is available at: github.com/YXZhang979/FGNet

However, FSS suffers from a dilemma, which lies in two aspects, i.e., intra-class gap and inter-class gap. On the one hand, the given support images are limited while the query images are various, resulting in the intra-class appearance gap between support data and query data. On the other hand, the feature space of support data is also sparse and in low coverage, leading to the problem of the inter-class classification gap. As shown in Figure 1, such an issue results in an ambiguity to the distinction between the target class and the nontarget class with the similar representation.

Current FSS approaches mainly focus on refining the prototype quality, enhancing query features [Yang et al., 2020; Li et al., 2021] or seeking for appropriate matching mechanisms [Wang et al., 2020; Siam et al., 2021]. In spite of their high performances, those methods fail to eliminate the intra-class appearance gap and inter-class classification gap essentially. Especially, no matter how the prototype is refined, the intersection between support images and query images remains inadequate. Moreover, the issue of the inter-class gap is rarely discussed, which makes the generation of prototypes very difficult to distinguish the different classes with similar representations [Okazawa, 2022]. In this paper, we concentrate on tackling the above two problems in one single uniform framework.

Due to the scarcity of the support data, the pattern in the query image may not be contained in the prior knowledge. Accordingly, we propose a Self-Adaptive Module (SAM) to reduce the intra-class gap, which is motivated by [Liu and Qin, 2020; Fan et al., 2022]. We get around the issue and propose a self-adaptive mechanism to establish an enhanced prototype for further prediction. Such a prototype contains the underlying information of the query sample itself, which caters to each query data by self-alignment, as the intrainstance similarity is higher than the cross-instance similarity. Moreover, we propose an inter-class loss to increase the similarity of support prototype and query prototype, aiming to guide the network to extract the intrinsic feature of each specific category.

As to the inter-class gap, the instance of the target class may share a similar feature space with the non-target class, due to the inadequacy of the support data. Therefore, we propose an Inter-class Feature Separation Module (IFSM) to distance the inter-class representations. Specifically, we reduce the prototype similarity between different categories to make the prototype discriminating. Moreover, due to the setting of FSS, the background area may contain the latent non-target classes. To distinguish the foreground with the latent instance [Yang et al., 2021] of the non-target class in the background region, we leverage superpixel-guided clustering [Li et al., 2021] and propose a background SLIC (B-SLIC) method to divide the background into several sub-areas. Then we present a novel loss to enlarge the distance between the support prototype and the background prototypes of each subarea. In this way, the separation performance is improved to secernate the different categories, especially those with highly analogous representations.

Combining the above building blocks, we propose a uniform network to fill both the intra-class and inter-class gaps, named FGNet. To evaluate the performance of FGNet, we conduct extensive experiments and ablation studies. Experimental results show that FGNet surpasses previous SOTAs on both PASCAL-5i [Everingham et al., 2010] and COCO-20i [Lin et al., 2014] datasets.

In summary, our main contributions are as follows:

• We propose FGNet, a uniform prototypical learning network to fill both the intra-class and inter-class gaps for few-shot segmentation.
• We introduce two modules, i.e., SAM and IFSM, to get around the intra-class appearance discrepancy and separate the prototype of different classes, respectively. We also present several new losses and B-SLIC to further improve the separation performance of FGNet.
• Extensive experiments show that FGNet surpasses other prevalent FSS approaches and achieves state-of-the-art performances on both PASCAL-5i and COCO-20i on the metric of mean intersection over union (MIoU).

2 RelatedWork

Semantic segmentation Semantic segmentation is a fundamental task in computer vision, which classifies each pixel into a pre-defined category. The mainstream paradigm is based on the fully convolutional network (FCN) [Long et al., 2015], which replaces all the linear layers with the convolutional layers. Recent breakthroughs in semantic segmentation leverage the encoder-decoder structure for better feature extraction [Chen et al., 2018], the dilated convolution to enlarge the receptive field [Mehta et al., 2018] and the attention mechanism to model long-range dependency [Strudel et al., 2021; Xie et al., 2021a]. However, these approaches rely heavily on large-scale annotated datasets, resulting in a poor adaptation ability to the unseen classes with only a handful of the annotated samples.

Few-shot learning Few-shot learning aims to learn the generalization ability to conduct classification on unseen categories with only a handful of training samples available. Existing methods can be roughly divided into two branches, i.e., metalearning based approaches [Baik et al., 2021; Xu et al., 2021; Ding et al., 2021] and metric-learning based approaches [Doersch et al., 2020; Chen et al., 2022]. The main idea of the former is to improve the capability of fast adaptation to the novel classes. In the latter approaches, the distance of similarity measurement is employed to seek for the relevance of the support-query pair. Different from few-shot classification, few-shot segmentation predicts the mask in the pixel-level, which is different and more challenging than the classification task.

Few-shot segmentation FSS is a challenging task that extends semantic segmentation to the few-shot scenario. It requires conducting pixel-wise prediction of the unseen categories with only a small number of annotated samples. OSLSM [Shaban et al., 2017] first introduced the task of FSS, which proposes a two-branch network based on metalearning strategy. Recently, metric-learning based approaches [Wang et al., 2019; Liu et al., 2020] are proposed for FSS, which constrcuts a global descriptor for each specific category, named prototype. The later works mainly focus on improving the quality of prototypes [Li et al., 2021; Yang et al., 2020; Liu et al., 2022], enhancing the matching mechanism [Wang et al., 2020; Zhuge and Shen, 2021], leveraging background information [Dong et al., 2021; Tang et al., 2021] and introducing memory networks [Xie et al., 2021b; Wu et al., 2021].

Despite their success, existing approaches can hardly eliminate the intra-class and inter-class gaps. Although the prototype is refined to be comprehensive, the intra-class appearance difference remains inevitable. Such a gap causes an obstacle for predicting the mask of query images whose features are not overlapped with the limited support data. Moreover, there is little attention to the inter-class classification gap, leading to the ambiguous decision boundary to classify the non-target data with a similar representation to the target class. Therefore, this gives the motivation of this paper: can we fill both the gaps of FSS in a uniform framework to enhance the performance?

3 Method

3.1 ProblemDefinition

Different from the classic semantic segmentation, FSS aims at learning the generalization ability to adapt to unseen categories. Specifically, models are trained on the set of categories Ctrain and tested on the set of novel categories Ctest, where Ctrain ∩ Ctest = φ. Both the training set Dtrain = {IS/Q,MS/Q} and test set Dtest = {IS/Q,MS/Q} are composed of several episodes, where I ∈ RH×W×3 denotes the RGB image and M ∈ RH×W represents the binary mask. The subscripts S and Q stand for support and query, respectively. We follow the episodic fashion [Shaban et al., 2017] to train and test our model. Each episode is composed of k support samples {(I iS ,M iS )}ki=1 and a query sample (IQ,MQ), which share the same category c. With a batch size of B, the model predicts the query mask ˜MQ to approximate the corresponding ground-truth mask MQ.

3.2 Overview of FGNet

The overall architecture of FGNet is illustrated in Figure 2. Such a network is composed of two core modules, i.e., a Self-Adaptive Module (SAM) and an Inter-class Feature Separation Module (IFSM), focusing on handling the intra-class variation and separating the target class with the classes that share similar features, respectively.

The overview data flow of FGNet is as follows. First, the support image and the query image are fed into a shared convolutional neural network (CNN) [LeCun et al., 1989] for feature extraction. Through IFSM, we obtain an interclass loss Linter for distancing the representations of different classes. As shown in Figure 2, SAM consists of three parts, i.e., SAM (pre), feature enhancement module (FEM) and SAM (post). To narrow the intra-class gap, SAM exploits the query feature and calculates a query prototype for self-alignment. Subsequently, features and similarity maps are concatenated and fed into an FPN-like network [Tian et al., 2020] for feature enhancement. This module aims to rectify the scale inconsistency and refine the feature in a multiscale manner. Leveraging the enhanced feature, we generate an enhanced prototype, which activates the query feature to predict the mask. Moreover, the enhanced feature passes through three 1 × 1 convolutional blocks, followed by a softmax operation to predict another mask. The average fusion of the two predicted masks forms the final prediction ˜MQ. Therefore, the segmentation loss Lseg is calculated by the binary cross entropy loss of predicted mask ˜MQ and groundtruthMQ. In addition, we calculate the intra-class loss Lintra based on these prototypes to improve the descriptor similarity and compact the feature space of the same class. Accordingly, the total loss function L is formulated as:

where α1, α2 and α3 are balanced factors, and we empirically set α1 = 0.25, α2 = 0.25 and α3 = 0.5, respectively. We dive into the details of SAM and IFSM below.

3.3 Self-Adaptive Module

Despite great efforts to refine the prototype [Li et al., 2021; Liu et al., 2020], the huge intra-class variation remains inevitable due to the scarcity of support data and the diversity of query data. Therefore, we design SAM to exploit the query feature and establish a query prototype to match the query feature itself. The query prototype is more effective to predict the query mask, since the intra-instance similarity is much higher than the traditional cross-instance similarity. Such a query prototype is homologous to the corresponding query feature. That is, it is suitable to fill the intra-class gap and resolve the issue from the intra-instance perspective. Moreover, we propose an intra-class loss to improve the similarity of the support prototype and the query prototype, which guides the network to extract more intrinsic features of each specific category.

Given the query feature Fq and the support prototype ps, we first calculate the similarity map M1 through pixel-wise cosine similarity, and then generate a query prototype pq by a masked average pooling operation, formulated as:

where (x, y) denotes the coordinate, µ represents a threshold to activate M1, and 1 is an indicator function. Each pixel M (x,y) 1 ∈ [0, 1] stands for the confidence of the foreground. The selection of µ is significant for establishing a query prototype for self-alignment. Here we set µ = 0.7 empirically. As the foreground region is of high similarity and sensitive to the noise, we need to select high-confidence features to obtain the prototypes, which are further used for self-alignment. Note that, each activation threshold of the similarity map in this paper is 0.5 unless explicitly stated.

Subsequently, we establish a fused prototype pf through the aggregation of ps and pq, computed by:

where β1 and β2 represent the weights for prototype fusion and we set β1 = β2 = 0.5 empirically. Then we activate the query feature by pf . Such a self-matching process generates a high-quality similarity map M2, as the intra-instance information is fully exploited for self-adaptation.

Motivated by [Tian et al., 2020], we employ an FPN-like network (FEM) for feature enhancement. The concatenation of Fs, Fp, M2 and the expansion of pf forms the input of FEM. Such a network outputs the enhanced feature Fe with comprehensive and multi-scale information of the specific class. Then we activate Fe by pf to compute the similarity map M3 that is further utilized to generate the enhanced prototype pe, which is formulated by:

where τ is a threshold of confidence to establish the enhanced prototype pe, and 1 is an indicator function. Here we set τ = 0.5 empirically.

Employing the above four prototypes, we define the intraclass loss function Lintra as:

where cos is the cosine similarity and SP denotes the permutation of the prototype set P = {ps, pq, pf , pe}. We compute the cosine similarity between prototype pairs to narrow their discrepancy. This intra-class loss aims to guide the network to extract more intrinsic features of a specific category, despite the appearance gap between the support image and the query image.

Following the procedure described in Section 3.1, we obtain the final mask ˜ MQ. Thus the segmentation loss Lseg is formulated as:

where BCE stands for the binary cross entropy loss of the predicted query mask ˜MQ and its ground-truth MQ.

3.4 Inter-class Feature SeparationModule

IFSM is divided into two branches, as shown in Figure 3. On the one hand, reducing the similarity of prototypes with different classes helps to improve the separation performance [Okazawa, 2022] on differnet category. On the other hand, background regions usually contain latent classes [Yang et al., 2021], while they are ignored due to the irrelevance of the target class. Therefore, we take both scenarios into account to enlarge the representation distance of the target class and non-target class.

Given a batch size B, we first calculate the support prototype for each episode and obtain the support prototype set Ps = {pc1 s , pc2 s , ..., pcB s }, where the superscript ci represents the category of the i-th prototype. Accordingly, the crossclass loss Lcross is formulated by:

where 1 and cos are defined the same as those in Eqs. (4) and (5), respectively. The average similarity of the pairwise prototypes with different classes forms the cross-class loss. Such a loss is introduced to separate the representation space of distinct categories, resulting in the improvement of predicting the ambiguous data near the decision boundary.

Furthermore, we fully exploit the background information, as the ignored latent-class instance may hide in the background region due to the special setting of FSS. Compared with the foreground instance, the information of the background is extremely complicated. Specifically, the background area contains not only the noncontinuous stuff, e.g., sky, but also other continuous things of the non-target classes. Hence, we divide the background region into Nsub sub-areas, motivated by the superpixel method SLIC [Achanta et al., 2012]. Such a strategy, which is called B-SLIC by us, aims to cluster the pixels with similar representations to form several small sub-regions in the background area. Our B-SLIC operation takes the pixel-level feature and the coordinate into account to calculate the distance for clustering, inspired by [Irving, 2016]. Thus the distance D between two different pixels is calculated by:

where df and dc denote the Euclidean distance of feature and coordinate spaces of the two pixels. The balanced factor m is set to be m = √Nbg/Nsub [Achanta et al., 2012], where Nbg represents the total number of background pixels. Note that, the setting of Nsub follows the strategy in [Li et al., 2021]. Accordingly, we obtain Nsub background regions and establish the background prototype set Pbg = {P 1 bg , P 2 bg , ..., P Nsub bg }. Therefore, the background-class loss Lbg is formulated by:

Finally, according to Eqs. (7) and (9), the total inter-class loss Linter is calculated by:

where γ1 and γ2 are the balanced factors and we set γ1 = γ2 = 0.5 empirically. Such a loss targets to reduces the representation similarity of different categories, which enlarges the distance between classes and refines the ambiguous decision boundary.

4 Experiments

Datasets and Metric. To evaluate the performance of FGNet, we conduct experiments on two widely-used FSS datasets, i.e., PASCAL-5i [Shaban et al., 2017] and COCO-20i [Lin et al., 2014]. PASCAL-5i and COCO-20i are derived from the traditional segmentation datasets Pascal VOC 2012 [Everingham et al., 2010] and MS COCO [Lin et al., 2014], with the extra annotations in SDS [Hariharan et al., 2014] and FWB [Nguyen and Todorovic, 2019], respectively. The categories are partitioned into four equal splits for crossvalidation. Specifically, three splits are selected for training, while the rest is for evaluation. During inference, 1k supportquery pairs in PASCAL-5i and 20k support-query pairs in COCO-20i are randomly selected for evaluation. Besides, we use MIoU as our primary metric to evaluate FGNet under both 1-shot and 5-shot settings.

Implementation Details. The prevalent backbone ResNet [He et al., 2016] pretrained on ImageNet [Deng et al., 2009] is employed as our feature extractor. The features in block2 and block3 are concatenated to produce the feature map. We use SGD optimizer to train FGNet, with 0.9 momentum and 5e-3 initial learning rate. To separate different classes, we set a large batch size of 16. All images are cropped to 473 × 473 resolution and augmented by random horizontal flipping. Moreover, we remove the last ReLU for better generalization [Yang et al., 2021].

4.1 Comparison with State-of-the-art

To evaluate the effectiveness of FGNet, we report our main results on PASCAL-5i and COCO-20i datasets. Employing the commonly-used backbone ResNet-101, our method achieves the best mean performances in both 1-shot and 5shot scenarios on both datasets, compared with several previous state-of-the-art approaches.

PASCAL-5i We list our results of PASCAL-5i in Table 1. Our method is superior to other top-performing approaches with respect to MIoU under both 1-shot and 5-shot settings. Specifically, in the 1-shot task, FGNet reaches 68.6%MIoU, which improves the previous SOTA [Okazawa, 2022] by 1.1% MIoU. In the 5-shot task, our method achieves 73.3% MIoU, outperforming the previous best performance [Fan et al., 2022] by 0.8%. Moreover, FGNet obtains high performances on split-0 and split-2 in both 1-shot and 5-shot scenarios. Note that, the improvements of the 1-shot scenario are higher than the 5-shot scenario for most models.

COCO-20i We present our results of the challenging COCO20i dataset in Table 2. As shown in the table, our method outperforms the previous methods by a large margin and achieves separately 48.1% MIoU and 54.1% MIoU under 1shot and 5-shot settings. In particular, FGNet exceeds the previous best performance [Okazawa, 2022] by 1.2%MIoU and 0.8%MIoU in 1-shot and 5-shot scenarios, respectively. Interestingly, the improvement of the 1-shot task is also greater than the 5-shot task for most models. We believe that the situation is not a coincidence and we will further discuss the possible reason in Section 4.3.

4.2 Ablation Study

An ablation experiment is conducted to verify the necessity of SAM and IFSM, which are the core modules of FGNet. The results are presented in Table 3. The performance of the vanilla model (using similarity map M1 as the final prediction similar to [Wang et al., 2019]) without SAM and IFSM is 59.7% MIoU. With the incorporation of SAM, the model obtains an improvement of 10.5%MIoU. Besides, the introduction of IFSM increases the MIoU by 6.7%. Each module gains a significant improvement, compared with the vanilla approach. Furthermore, combined with both SAM and IFSM, our method achieves 73.3% MIoU, which is 13.6% MIoU higher than the vanilla network. Therefore, we dive into investigating how SAM and IFSM narrow the intra-class and inter-class gaps separately.

4.3 Intra-class Gap Reduction

To make our self-adaptive method more easily understood, we conduct experiments and analyses to demonstrate how SAM narrows the intra-class appearance gap.

Self-Adaptive process. The visualization results of the selfadaptive procedure of SAM are illustrated in Figure 4. First, we employ the support prototype to activate the query feature to obtain the activation of the similarity map M1. Notice that, M1 is unsatisfactory due to the inadequate intersection of support data and query data. Taking the first row in Figure 4 for an example, the support image contains only the pattern of cat head and claws, which brings difficulty in recognizing cat body in the query image. To fill the intra-class gap, we calculate the query prototype (by Eq. (2) and Eq. (3)) and leverage it for self-alignment. Specifically, we generate a fused prototype by Eq. (4). Such a fused prototype contains the intra-instance information of the query sample, which benefits the activation of the query feature itself. With the activation of the fused prototype, we obtain a relatively high-quality mask M2. After the enrichment of FEM, the enhanced feature contains more comprehensive information of the discriminating category, contributing to generating the enhanced prototype, and this process assists to match with the query feature for the final prediction.

Thresholds of similarity maps. The thresholds of µ in M1 and τ in M3 are significant for feature selection and prototype establishment. We explore the best combination of the two thresholds and the results are shown in Figure 5. The combination of µ = 0.7 and τ = 0.5 achieves the best MIoU performance. We think that the high-confidence feature in M1 is important to capture the underlying characteristics of the query sample. Thus µ = 0.7 is suitable to activate the query feature itself and generate a reasonable query prototype. Moreover, an intermediate threshold τ = 0.5 in M3 is appropriate, as the fused prototype and the enhanced feature require taking more acceptable representations into account, contributing to generating a comprehensive prototype for the final prediction.

Intra-class loss. Besides the self-adaptive process, SAM also employs an extra intra-class loss Lintra to reduce the intraclass gap. Such a loss aims to guide the backbone network to extract more intrinsic features of the discriminating category rather than the superficial appearance features. As shown in Table 4, the removal of Lintra decreases the performance by 1.2% MIoU. Therefore, we think that minimizing the intraclass loss is beneficial to the target class. Despite the apparent difference between the support data and query data, the network digs out the underlying and discriminating representations of each category. The intra-class loss results in the compaction of the intra-class feature space of each specific category, which eliminates the variation from another perspective.

Advantages of SAM. We summarize the advantages of SAM to narrow the intra-class appearance gap in two aspects: 1) Leveraging the self-adaptive process, we generate a prototype that caters to the query sample for self-alignment; 2) With the intra-class loss Lintra, the backbone tends to extract underlying representations of each specific class, which compacts the intra-class prototype space for an accurate prediction. Furthermore, as mentioned earlier, the improvement of the 1-shot task is higher than the 5-shot task. We think that the 1-shot task benefits more from the self-adaptation process, as the prototype under the 1-shot setting is more unreliable. With the feature enhancement and self-adaptive mechanism, the 1-shot task owns ample room for improvement compared with the 5-shot scenario.

4.4 Inter-class Gap Reduction

We carry out experiments and analyses to demonstrate how IFSM overcomes the inter-class classification gap.

Inter-class loss. We investigate the influence of the crossclass loss Lcross and the background separation loss Lbg, which are the two parts of the inter-class loss Linter. As shown in Table 5, the removal of Lcross and Lbg decreases the result by 2.7% MIoU and 1.4% MIoU, respectively. Therefore, both Lcross and Lbg are significant to the total inter-class loss, since Lcross distances the feature spaces between different categories and Lbg differentiates the foreground with the latent instances in the background region, which reduces the similarity of the foreground prototype with the latent nontarget prototypes and rectifies the decision boundary.

Performance on similar classes. To evaluate the effectiveness of different classes that are difficult to distinguish, we conduct sufficient experiments to obtain the improvement of each category. For a fair comparison, we select 1k supportquery pairs for each category in split-0 and split-2 (the significantly improved splits) under the 5-shot setting. As shown in Table 6, the improvement of 10 dining-table and 14 person are 22.0% MIoU and 23.8% MIoU, respectively, which are much higher than other classes. The features of such categories are likely to be ambiguous with other classes since they usually appear in complex scenarios. Specifically, 10 dining-table is usually covered with cluttered tablewares and 14 person appears in diverse scenes with various poses and decorations. Moreover, the performance of 1 bicycle and 13 motorbike are increased by 17.3% MIoU and 15.9% MIoU, respectively. The representations of these two classes are similar to each other, leading to the challenge of accurate prediction without the consideration of the inter-class relation. Therefore, with the incorporation of IFSM, the network is guided to extract intrinsic representation for each specific category, resulting in distancing the feature space of similar classes that are difficult to classify.

Advantages of IFSM. IFSM closes the inter-class gap by separating the prototype spaces of the different categories. As shown in Figure 6, the prototypes (visualized by t-SNE [Van der Maaten and Hinton, 2008]) of each category are adjacent and mixed with respect to the vanilla model. However, the prototype distance is enlarged with IFSM, which reduces the ambiguity of classification and rectifies the decision boundary. On the one hand, with the cross-class loss, the network distances the representations of similar classes. On the other hand, the background separation loss further reduces the prototype similarity of the target class with other latent non-target classes. Therefore, IFSM enlarges the margin of prototypes of different categories to improve the separation performance, which fills the inter-class gap.

5 Conclusion

In this paper, we proposed FGNet to fill the intra-class and inter-class gaps for few-shot segmentation. To narrow the intra-class gap, we introduced a Self-Adaptive Module (SAM) to fully exploit the query representations for selfalignment. Moreover, we proposed an Inter-class Feature Separation Module (IFSM) to separate the prototype spaces of different classes, which bridges the inter-class gap. In addition, we put forward B-SLIC to take the latent classes in the background region into account, and designed several new losses to further improve the separation performance of FGNet. Experimental results show that FGNet effectively fills both the gaps, and meanwhile achieves SOTA performances on multiple datasets.

Acknowledgements

This work was supported by the National Natural Science Foundation of China (No. 62172385), the Anhui Initiative in Quantum Information Technologies (No. AHY150300), and the Innovation Program for Quantum Science and Technology (No. 2021ZD0302900).