Cross-Domain Few-Shot Semantic Segmentation via Doubly Matching Transformation

1 Introduction

Relying on large-scale labeled datasets [Ros et al., 2016; Richter et al., 2016; Silberman et al., 2012], semantic segmentation [Long et al., 2015; Zhao et al., 2016; Chen et al., 2017; Xie et al., 2021] has achieved rapid development in recent years. However, it is difficult to collect such a large amount of training data which requires massive time and expensive annotation costs in practical scenarios. Fewshot Semantic Segmentation (FSS) [Shaban et al., 2017] has been proposed to reduce the heavy dependence of traditional semantic segmentation models on a large number of labeled images. FSS aims to achieve accurate segmentation of a query image only using a few annotated support images. Existing FSS methods [Rakelly et al., 2018; Zhang et al., 2018; Wang et al., 2019; Liu et al., 2020a; Okazawa, 2022] usually adopt meta-learning [Vinyals et al., 2016; Snell et al., 2017], which consists of two stages: metatraining and meta-testing. In the meta-training stage, an FSS model is trained on many meta-tasks using base classes. The trained model can then perform accurate segmentation on novel classes in the meta-testing stage.

* Corresponding Author

However, in practical applications, there always exists a large domain gap between source and target datasets due to different label spaces and feature distributions, causing inferior generalization of FSS models to unseen domains. To solve the problem of significant performance degradation of FSS models under cross-domain scenarios, CrossDomain Few-shot Semantic Segmentation (CD-FSS) [Lei et al., 2022] is proposed to simultaneously solve the problems of few shot and domain gaps. The primary CD-FSS method, PATNet [Lei et al., 2022], eliminates the domain gap by transforming domain-specific features into domain-agnostic ones and conducting segmentation in the domain-agnostic feature space. It combines the prototype set of support images with some learned anchor layers to construct transformation matrices, based on which domain-specific features are transformed into domain-agnostic ones. However, during the meta-testing stage, generating domain-agnostic features for hundreds or thousands of query images just based on the transformation matrices obtained from one or several support images can easily lead to overfitting. In addition, we find that most existing CD-FSS methods [Min et al., 2021; Lei et al., 2022] only concentrate on foreground object regions and ignore background regions during the segmentation process. They directly filter out the background of support images and only construct dense correlations with foreground objects. Considering that objects belonging to the same class mostly lie in similar environments, segmentation based on the similarities between not only the foreground objects but also the backgrounds is very likely to result in better performance.

Based on the above considerations, we propose a novel Doubly Matching Transformation-based Network (DMTNet) for cross-domain few-shot semantic segmentation. DMTNet first exploits a Self-Matching Transformation (SMT) module to construct a unique transformation matrix for each image based on its own prototype. Then, domain-specific features of each query and support image are transformed into domain-agnostic ones self-adaptively, which can avoid overfitting during the meta-testing stage. After obtaining domainagnostic features, we propose a Dual Hypercorrelation Construction (DHC) module to learn the hypercorrelation between the query image with both the foreground and background of the support image, and generate foreground and background predictions, correspondingly. Supervising both object-wise and background-wise segmentation can enhance the training performance considering the possible similarities between the backgrounds of similar objects. In addition, in the meta-testing stage, we design a Test-time Self-Finetuning (TSF) strategy to further improve query predictions in unseen domains by self-tuning a handful of parameters in the network with support images.

To evaluate the performance of DMTNet, we conduct extensive experiments and ablation studies on four benchmark datasets, including ISIC2018 [Codella et al., 2019], Chest Xray [Candemir et al., 2014], Deepglobe [Demir et al., 2018], and FSS-1000 [Wei et al., 2019]. Experimental results show that DMTNet achieves remarkable improvement, surpassing state-of-the-art approaches on all four datasets.

In summary, our main contributions are as follows:

• We propose a novel Doubly Matching Transformationbased Network (DMTNet) for CD-FSS. Instead of completely relying on support images, we propose a SelfMatching Transformation (SMT) module to transform domain-specific query features into domain-agnostic ones in a self-adaptive manner. Compared with transformation completely based on one or several support images, DMTNet can avoid overfitting to a large extent.
• We propose a Dual Hypercorrelation Construction (DHC) module to learn the hypercorrelations between the query image and both the foreground and background of the support image. For the first time, we execute segmentation based on both foreground object similarities and background similarities.
• We design a Test-time Self-Finetuning (TSF) strategy for meta-testing. Only self-tuning a handful of parameters in the network can significantly improve query predictions in target domains while maintaining minimal complexity.
• Extensive experimental results show that DMTNet achieves state-of-the-art performance on four CD-FSS benchmarks, i.e., ISIC2018, Chest X-ray, Deepglobe, and FSS-1000.

2 RelatedWork

Few-shot Semantic Segmentation. FSS aims to generate a pixel-wise prediction of the novel class with only a few labeled support images. Existing FSS methods can be divided into two categories: metric-based methods and relationbased ones. Metric-based methods [Wang et al., 2019; Liu et al., 2020b] represent support images as several class prototypes by utilizing masked average pooling, use nonparametric measurement tools such as cosine similarity to measure the similarity between these prototypes and query features, and segment the query image based on the similarities. However, metric-based methods lose much spatial information when compressing the global feature maps into a prototype vector [Li et al., 2021; Zhang et al., 2021]. Since the prototype has a limited ability to express a target category, some researchers propose relation-based methods [Zhang et al., 2019b; Yang et al., 2020; Tian et al., 2020; Min et al., 2021] to construct dense correspondences between support-query pairs by calculating their similarities. However, the segmentation performance of these FSS methods will decrease rapidly when facing a large domain gap between the source and the target domain and cannot generalize well to the target domain.

Cross-domain Semantic Segmentation. Cross-domain semantic segmentation can be categorized into domain adaptive semantic segmentation (DASS) and domain generalized semantic segmentation (DGSS). DASS trains the model by jointly using source domain data and some labeled or unlabeled target domain data so that the model can quickly generalize well to the target domain. Recent works can be grouped into adversarial training and self-training approaches. The former [Hoffman et al., 2017; Long et al., 2017] aims to align the distributions of the source domain and the target domain via generative adversarial networks or Fourier transforms. The latter [Zou et al., 2018a; Zou et al., 2018b] is trained with pseudo-labels for the target domain. DGSS tries to bridge the domain gap through two main approaches, including Normalization and Whitening (NW), and Domain Randomization (DR). NW [Pan et al., 2019; Peng et al., 2022] normalizes the mean and standard deviation of the source data and whitens the covariance of the source data. DR [Peng et al., 2021; Huang et al., 2021] transforms source images into randomly stylized images and trains the network with them together. Unlike the setting of cross-domain semantic segmentation, CD-FSS not only has no access to target domain data during training, but also has disjoint label space between the source and target domains.

Cross-domain Few-shot Semantic Segmentation. Different from FSS, there are domain gaps between the source dataset and the target dataset, i.e., both data feature distributions and label spaces in the meta-testing stage are different from the meta-training stage. Recently, some works have been proposed to address this task. For example, PixDA [Tavera et al., 2021] proposes a pixel-by-pixel adversarial training strategy that uses a novel pixel-wise loss and discriminator to bridge the domain shift. To improve the generalization of the segmentation model, RTD [Wang et al., 2022] designs a novel meta-memory module that transfers the intra-domain style information from the source domain images into target domain images. [Lu et al., 2022] introduces a transductive fine-tuning method, which addresses the domain gap by using support labels to implicitly supervise query segmentation. PATNet [Lei et al., 2022] establishes a new evaluation benchmark for CD-FSS and converts the domain-specific features to domain-agnostic ones to enhance generalization ability. Our proposed method tackles several key issues, including overfitting by only utilizing support information for feature transformation, intra-class appearance variances, and under-utilized information, which are overlooked by previous works.

3 Method

3.1 Problem Setting

The problem setting of CD-FSS can be formulated as follows. There is a source domain (Xs, Ys) and a target domain (Xt, Yt). Xs and Xt represent the input data distributions while Ys and Yt represent the label spaces. In CDFSS, Xs ̸= Xt and Ys ∩ Yt = ∅, i.e., the input data distribution of the source domain is different from that of the target domain, and the label spaces of the domains do not intersect. Based on the data distributions and the label spaces, we construct the training set Dtrain and testing set Dtest. In term of N-way K-shot segmentation, both Dtrain and Dtest consist of a large number of episodes. Each episode contains a support set S = {(I s i ,Msi )}N×K i=1 and a query set Q = {(I q i ,Mqi )}Qi=1 , where I ∈ RH×W×3 denotes the RGB

image and M ∈ RH×W is the binary mask. So Dtrain = {IS/Q,MS/Q}source and Dtest = {IS/Q,MS/Q}target . In meta-training stage, the model is trained on Dtrain, without exposure to the target domain. After completing episodes training, the model segmentation performance is evaluated using Dtest in the meta-testing stage.

3.2 Overview of DMTNet

The overall architecture of DMTNet is illustrated in Figure 1. DMTNet consists of two major functional modules to bridge the domain gap: the Self-Matching Transformation (SMT) module and the Dual Hypercorrelation Construction (DHC) module.

During the meta-training stage, the support set and query set are first fed to a shared convolutional neural network to extract multi-level pyramid features. Then, we exploit SMT to learn each support and query image a transformation matrix, and transform the domain-specific features into domainagnostic ones. Subsequently, DHC constructs the support foreground-query and support background-query hypercorrelations based on the domain-agnostic features, respectively. Finally, two available modules, 4D convolutional pyramid encoder and 2D convolutional pyramid decoder [Min et al., 2021] are adopted to obtain the predicted query mask based on the dual hypercorrelations.

During the meta-testing stage, two steps are involved in obtaining the final prediction mask. We design a novel Test-time Self-Finetuning (TSF) strategy in the first step, where only fine-tuning a few parameters of the network can significantly refine the coarse mask in the second step and encourage the model to adapt quickly to the target domain.

3.3 Self-Matching Transformation

In cross-domain scenarios, domain style information in features hinders the model from accurately segmenting foreground objects. Therefore, ensuring the invariance of foreground objects while generalizing the domain style information can improve the generalization of representations. One useful strategy is to perform a feature transformation to transform the domain-relevant features into domain-irrelevant features. However, existing methods [Lei et al., 2022] transform the support and query image features only based on the prototypes of the support image, which may pose potential issues for meta-testing. As the support set during the metatesting stage has only a few images for each class, repeatedly utilizing the same support images may cause overfitting. In addition, according to [Fan et al., 2022], there may exist a huge appearance difference between the support and query images, even if they belong to the same class. In this case, the transformation matrix derived only from the support image may not be useful for the query images. This motivates us to reduce the dependence on the support features and mine information from the query image itself during feature transformation. So we propose a Self-Matching Transformation (SMT) module. As shown in Figure 1, SMT consists of two stages. In the first stage, a Similarity-based Self-matching module is used to generate a rough segmentation mask, based on which the prototype of the query image is obtained. In the second stage, some learnable anchor layers are used to transform the support and query images' domain-specific features into domain-agnostic ones.

Similarity-based Self-matching. Inspired by [Fan et al., 2022], we first generate coarse segmentation masks for the query images via similarity-based self-matching between the query features with the foreground and background prototypes of the support image. Since compressing a global feature map into a prototype vector will lose much detailed information, we propose to divide the support features into several local features and generate a more fine-grained predicted query mask by measuring the similarity between support local prototypes and query global features.

Specifically, for a 1-way 1-shot task, we first obtain the Llevel pyramid features of the support and query images, i.e., {F s l , F q l }Ll=1 , where Fl ∈ RCl×Hl×Wl . Then we calculate the support global foreground prototype csfl ∈ RCl at intermediate layer l with the support maskMs ∈ {0, 1}H×W via Masked Average Pooling (MAP), as follows:

where (x, y) are spatial positions, and δl [·] denotes the bilinear interpolation. For simplicity, we represent δl [Ms ] as Msl . Similarly, the support global background prototype csbl can be calculated in the same way. Then we divide the support feature map F s l into P support local feature maps, i.e.,

we set it to 0.25. We also obtain the local support masks {Mslp}P p=1 following the same division. Then we calculate the p-th support local foreground prototype csflp ∈ RCl by MAP:

The p-th support local background prototype csblp is similar to this. We then calculate the confidence matching correlation maps between query features and support local prototypes to obtain the naive query mask:

where η (·) denotes the softmax function. ξ (·) denotes a similarity measure function. In this work, we use the cosine similarity. WithM1l, the query foreground and background prototype cqfl , cqbl can be calculated.

To make the predicted rough query mask as accurate as possible, and thus provide a most accurate transformation matrix for later adaptive feature transformation, we propose to use a binary cross-entropy (BCE) loss function as supervision here, formulated as:

Adaptive Feature Transformation. Similar to [Seo et al., 2020; Lei et al., 2022], we use a linear transformation as the transformation mapper. The difference is that we construct specialized transformation matrices for support and query features, respectively, to ensure the invariance of the foreground objects during the adaptive transformation process. We construct the support prototype matrix and the

weight matrices are defined as Asl = [ asfl ∥asfl∥ , asbl ∥asbl∥ ], Aql =

PATNet [Lei et al., 2022], we only set three anchor layers for support and query features, respectively. They correspond to the feature maps of three dimensions, i.e., low, medium and high-level features. So we construct two transformation matrices [Ws l ,Wq l ] by solvingWs l Csl =Asl andWq l Cql =Aql . Since Csl and Cql are non-square matrices, we calculate their generalized inverse by Cs+ l = {CsT l Csl }-1 CsT l , Cq+ l =

{CqT

l Cql

}-1

CqT

l . In particular, we propose to refine Cq+

by integrating Cs+ l into Cq+ l :

where β mainly controls the integration ratio, which is set to 0.5. Finally, the support and query transformation matrices at layer l can be calculated as Ws l = Asl Cs+ l , Wq l = Aql Cq+ l respectively, where Ws l ,Wq l ∈ RCl×Cl .

l

3.4 Dual Hypercorrelation Construction

Considering that objects from the same category are very likely to lie in similar environments, the background information correlations between the query and support images can also be used in CD-FSS. However, the existing method [Min et al., 2021] directly filters out the support background region using the support mask. Differently, we propose a Dual Hypercorrelation Construction module (DHC) to explore the dense correlations between the query features with the foreground and background features of the support images in domain-agnostic space.

Firstly, we construct the 4D correlation tensor Corrfl ∈ RHl×Wl×Hl×Wl based on the support foreground features F s fl and query features F q l by cosine similarity:

where (x1, y1) and (x2, y2) denotes 2D spatial positions of support and query feature maps, respectively.

Secondly, we construct Corrbl based on the support background features F s bl and query features F q l :

.

Then, the dense correlation maps are fed to two available modules, 4D convolutional pyramid encoder and 2D convolutional pyramid decoder [Min et al., 2021] to generate the predicted query foreground mask Mf and background mask Mb. The training supervision on the two predicted masks is:

where α1 is a tuning weight. Finally, we train the model in an end-to-end manner by jointly optimizing all the losses:

where α2 is a balancing hyperparameter. α1 and α2 are set as 1.0 and 0.5, respectively.

3.5 Test-time Self-Finetuning

In the meta-testing stage, we design a Test-time SelfFinetuning (TSF) strategy to refine query predictions in unseen domains. PATNet[Lei et al., 2022] proposes to finetune the anchor layers by reducing the distribution distance between support foreground prototypes and the query foreground prototypes which are obtained by the predicted query mask. However, they assume images belonging to the same class have similar appearances, which is not always true in few-shot scenarios. We contend that there may exist significant appearance variances even within the same class. Therefore, aligning the foreground prototypes of the support and query images may cause overfitting to support images and distortion of query foreground features. To this end, we propose TSF to self-tune the network by trying to predict the ground-truth masks of the support images. By finetuning the network on support images, our model can learn style information of the target domain, leading to the generation of more accurate masks for the query images.

As shown in the Figure 1, TSF has two steps. In the first step, the model outputs the predicted support masksMs and updates the network with the loss:

In the second step, we freeze the entire network and execute the final prediction for the query image.

Similar to PATNet, we do not finetune the whole network. Instead, we only finetune a few parameters of the encoder, which is validated by the quantitative experiments. The details can be found in Section 4.4.

4 Experiments

4.1 Experimental Setup

To fairly compare the cross-domain segmentation performance of DMTNet with PATNet [Lei et al., 2022], we choose PASCAL VOC 2012 with SBD augmentation as the source domain, and ISIC2018, Chest X-ray, Deepglobe, and FSS1000 as the target domains.

ISIC2018 is a skin cancer screening dataset consisting of lesion images. The dataset contains three types of skin lesions and a total of 2,596 images, each with one primary lesion region. The initial spatial resolution is approximately 1022 × 767 and we uniformly reduce it to 512 × 512.

Chest X-ray is a Tuberculosis X-ray image dataset with 566 annotated images, which is collected from 58 abnormal cases with a manifestation of Tuberculosis and 80 normal cases. The initial spatial resolution is 4020 × 4892 and we downsize it to 1024 × 1024.

Deepglobe is a satellite image dataset. Each image is densely annotated at pixel level with 7 categories: urban land, agricultural land, rangeland, forestland, water, barren land, and unknown. Following PATNet, we partition each image into 6 pieces and filter the single class images and the 'unknown' class, leading to a dataset of 5,666 images with a spatial resolution of 408 × 408.

FSS-1000 is a natural image dataset specialized for FSS, which contains 1,000 categories and each category has 10 annotated images. Each image contains only one segmentation target and has a resolution of 224 × 224.

We use the standard Intersection over Union (IoU) metric by averaging the prediction results across 5 runs with different random seeds. Each run is composed of 1,200 tasks for ISIC2018, Chest X-ray, and Deepglobe. And for FSS-1000, each run contains 2,400 tasks. We evaluate our model under 1-way 1-shot and 1-way 5-shot settings.

4.2 Implementation Details

Following previous works, we employ ResNet-50 and VGG16 pre-trained on ILSVRC as our backbones. For ResNet50, the features from the conv3 x, conv4 x, and conv5 x are extracted to produce feature maps. The channel dimensions of the three anchor layers are set to 512, 1024, and 2048, respectively. For the VGG-16 backbone, the features from the conv4 x to conv5 x are extracted to produce feature maps. The channel dimensions of the three anchor layers all are set to 512. In the meta-training stage, we use Adam optimizer to train DMTNet for 19 epochs with a learning rate of 1e3. In the self-finetuning of the meta-testing stage, we use Adam optimizer with a learning rate of 1e-6 for ISIC2018, Deepglobe and FSS-1000, 1e-1 for Chest X-ray. All input images are resized to 400 × 400 resolution.

4.3 Comparison with State-of-the-Art Methods

We compare the performance of our model with several stateof-the-art methods. These methods are categorized into three groups: transfer learning, few-shot semantic segmentation, and cross-domain few-shot semantic segmentation methods. All methods are trained on PASCAL VOC and tested on the four datasets. The experimental results are shown in Table 1. Because the whole ISIC is seen as one class in DAM [Chen et al., 2023], we do not compare their report results on ISIC here (under the same ISIC dataset setting, our model exceeds DAM by 2.25% in the 5-shot setting). We can see that under both (1-way) 1-shot and (1-way) 5-shot settings, the performance of our model ranks at the top on the average results of the four datasets, reaching 59.74%MIoU in the 1-shot setting and 66.01% MIoU in the 5-shot setting. We show the 1way segmentation visualization results of our model on four datasets in Figure 2.

When compared with the state-of-the-art PATNet [Lei et al., 2022], our model exceeds PATNet by 3.68% and 4.02% in the 1-shot and 5-shot settings respectively. This suggests that using query-specific transformation matrices and the dual hypercorrelation construction can avoid overfitting to base classes, leading to further enhancement of the segmentation results. In addition, when using VGG-16 as the backbone, our model significantly improves performance, leading to the second-best results on Chest X-ray and Deepglobe.

4.4 Ablation Study

Component Analysis. We conduct several ablation experiments to verify the effectiveness of the three key modules of DMTNet, i.e. SMT, DHC and TSF. Table 2 shows the impact of each module on the model performance. We can see that using all three modules proposed in this paper achieves the best results, and removing any of them would lead to a drop in the average performance across four datasets. Figure 3 further shows the segmentation visualization results of removing the TSF and the TSF+DHC modules, demonstrating the impact of our proposed module in a more intuitive way.

Fine-tuning Parameters for TSF. We conduct quantitative experiments to select the fine-tuning parameters for the test-time self-finetuning strategy. Following PATNet [Lei et al., 2022], we choose the low-, medium- and high-level anchor layers. Additionally, we believe that the encoder/decoder should have an important impact on the cross-domain segmentation performance as it hierarchically fuses/reconstructs the 4D correlation maps, therefore we also choose the two fusing/de-fusing layers in the encoder/decoder. The results are presented in Table 3. We can observe that fine-tuning all but the decoder layers can lead to performance improvement, demonstrating the effectiveness of the TSF. Fine-tuning the encoder layers achieves the best performance of 59.45%, indicating the effectiveness of bringing target domain information into the encoder for better fusion of correlation maps.

Visualization w.r.t. SMT. We visualize the image feature distributions before and after SMT to more clearly illustrate the effect of SMT in bridging the domain gaps, as shown in Figure 4. We show the feature distributions of ISIC and FSS1000 datasets for comparison with PASCAL VOC. We can see that the feature distributions of ISIC and FSS-1000 are closer to the PSCAL VOC after using SMT, which proves the effectiveness of our module in bridging domain distances and better generalization to the target domains.

5 Conclusion

We propose DMTNet for cross-domain few-shot semantic segmentation. DMTNet first exploits an SMT module to calculate each support and query image a transformation matrix based on its own prototype, and transform their domainspecific features into domain-agnostic ones self-adaptively. Then, a DHC module is used to explore the dual hypercorrelation between the query image with the foreground and background of the support image in the domain-agnostic space, based on which a foreground and background prediction mask are obtained and supervised during meta-training, respectively. During meta-testing, a TSF strategy is used to further improve the segmentation performance by only refining a few parameters of DMTNet. Extensive experiments show that DMTNet is effective and achieves state-of-the-art performance on four datasets with different domain gaps.

Acknowledgments

This work was partially supported by the National Natural Science Foundation of China (No. 62276129 & No. 62206127), and the Natural Science Foundation of Jiangsu Province (No. BK20220890).