Dichotomous Image Segmentation with Frequency Priors

1 Introduction

Dichotomous image segmentation (DIS) [Qin et al., 2022] aims to segment fine-grained objects from various natural scenes. As an emerging image segmentation task, DIS has great potential in a large number of applications, such as image editing [Kawar et al., 2023], 3D reconstruction [Geiger et al., 2011], remote sensing [Zhang et al., 2022a], medical image analysis [Dong et al., 2023b; Ji et al., 2022], virtual reality [Singh et al., 2020], and so on. Different from existing segmentation tasks, DIS focuses on challenging fine-grained object segmentation. Therefore, conventional segmentation models show unsatisfactory performance in DIS, raising significant demands for designing models dedicated to DIS.

∗Corresponding author.

Image segmentation is a long-standing topic in computer vision and has made significant progress with the development of deep learning techniques [Qin et al., 2020]. Deep learning has also been employed for DIS. A typical method is IS-Net [Qin et al., 2020], which employs an intermediate supervision strategy to improve network training for DIS. Despite the progress of existing works, many factors restrict the accuracy of DIS. Existing works usually rely on features extracted from images without the consideration of valuable frequency priors. In practice, a number of works have demonstrated that prior knowledge from the frequency domain can effectively improve the performance of various computer vision tasks. For instance, [Qian et al., 2020] proposed a novel face forgery detection network, called F3Net, which utilizes frequency information to perceive forgery cues. [Zhong et al., 2022] designed a frequency enhancement module to introduce the frequency domain as an additional clue to make up for the lack of a single RGB domain to better detect camouflaged objects. These studies demonstrate that knowledge in the frequency domain can assist in the identification of finegrained object boundaries, which play a vital role in DIS.

However, incorporating frequency priors into DIS faces a number of challenges, including (i) how to extract informative frequency priors from images, (ii) how to embed the frequency priors into multi-scale side-out features of the backbone, and (iii) how to address the heterogeneity of multi-scale features before frequency priors embedding.

To this end, we propose to improve DIS with frequency priors and design a novel deep learning model, called FPDIS, to address the aforementioned challenges. Specifically, we propose a frequency prior generator to extract informative frequency priors with both fixed and learnable filters. The resulting frequency priors are then embedded into harmonized multi-scale features for accurate DIS, as shown in Figure 1. Here, we propose a feature harmonization module (FHM) to harmonize the multi-scale side-out features from a backbone based on our pyramid feature extractor. Our FHM harmonizes the grouped features to reduce the heterogeneity of multi-scale features. To embed these frequency priors efficiently, we propose a frequency prior embedding module (FPEM), where two feature embedding components are integrated in a cascaded manner. Our FP-DIS makes full use of frequency priors to improve DIS and addresses the challenges associated with frequency priors extraction and embedding with specially-designed modules. Extensive experiments on the benchmark dataset, DIS5K [Qin et al., 2022], show that our FP-DIS achieves promising performance in comparison with cutting-edge methods with superior finegrained segmentation capabilities. To put this in perspective, our FP-DIS outperforms the state-of-the-art model, IS-Net, by a large margin in terms of maximal F-measure on different sub-datasets, i.e., 4.4% on DIS-TE1, 2.8% on DIS-TE2, 3.8% on DIS-TE3, 1.9% on DIS-TE4, and 3.2% on DIS-VD. Our code is available at https://github.com/dongbo811/FP-DIS.

2 RelatedWork

2.1 Image Segmentation

Recent works have made great progress on image segmentation tasks [Pang et al., 2022]. However, compared with the general segmentation models and the specific scene models, the core difficulty of the DIS task lies in the mining of object details. For general segmentation task models, the architecture design tends to be robust to each class in various scenes. Due to the relatively rough labeling of the dataset, it cannot be directly applied to the DIS task. There are also models designed for specific tasks, such as visually salient object detection [Dong et al., 2021], camouflaged object detection [Fan et al., 2021a], marine animal segmentation [Li et al., 2021b], and medical image segmentation [Fan et al., 2020; Liu et al., 2021; Lin et al., 2022]. However, these methods are unsuitable for the DIS task. Specifically, since objects in salient detection tasks always have prominent contours and distinct colors, current methods mainly use boundary enhancement [Qin et al., 2019], semantic enhancement [Wu et al., 2019] or design attention mechanism [Woo et al., 2018] to improve segmentation performance. Camouflaged object detection aims to segment camouflaged objects from complex natural scenes. The difficulty is that the object to be segmented is indistinguishable from the background with very similar features such as color, outline, texture, etc. In recent studies, some construct multi-task learning frameworks [Zhai et al., 2021], some propose uncertainty reasoning methods [Li et al., 2021a] or multi-scale feature fusion [Chen et al., 2022] for enhancing camouflaged object detection performance. According to IS-Net [Qin et al., 2022], DIS is formulated as a category-agnostic task defined on non-conflicting annotations for accurately segmenting objects with different structure complexities, regardless of their characteristics. Following this principle, the DIS5K dataset is constructed, which uses three commonly used metrics, namely isoperimetric inequality quotient, number of object contours, and number of dominant points, to measure the fine-grained level of objects in various scenes. However, due to the elusive target objects with complex detailed structures in DIS, there is a bottleneck for feature representation, which limits the segmentation performance. Unlike current techniques that only focus on semantic enhancement, we add frequency priors to help the model capture more detailed information.

2.2 Frequency Priors in Computer Vision

Frequency domain signals have been widely used in computer vision tasks, such as image classification [Stuchi et al., 2017], super-resolution [Huang et al., 2017], and fake face detection [Li et al., 2018]. Some current studies use highpass filters [Pandey et al., 2016] to extract useful detailed features, and some models use discrete Fourier transforms [Durall et al., 2019] to convert images into the frequency domain to explore bottom-layer information. To improve the performance, many models propose to enhance the feature representation [Pang et al., 2022]. Some methods artificially synthesize features to further enrich the contained information [De Carvalho et al., 2013]. Some recent studies propose extracting important frequency components and filtering frequencies [Dong et al., 2023a], which are beneficial to enhance semantic information. [Zhang et al., 2022b] proposed to process different video bands separately to promote similarity within objects. In addition, some methods [Liu et al., 2023] employ knowledge distillation to reduce the domain gap between the frequency domain and the image domain. Different from these methods, our model obtains the frequency priors of the image by DCT transform and inverse transform, combined with frequency filters. The frequency priors contain detailed information that is difficult to detect in the color space of RGB images. Furthermore, there exists a large gap between image and frequency domain features. Therefore, we design a frequency prior embedding module to eliminate the semantic gap between frequency priors and image features by modulating their distribution to obtain a finer segmentation effect.

3 Method

In this section, we explain details about the proposed FP-DIS. The overall architecture is shown in Figure 2.

3.1 Overall Architecture

The core sight of FP-DIS is that, with the guidance of the frequency priors, the multi-scale image features can capture more details of the input image. Motivated by this, FP-DIS mainly consists of four parts: a pyramid feature extractor, a frequency prior generator, a feature harmonization module, and a frequency prior embedding module. As shown in Figure 2, we first capture multi-scale features of the input image with a pyramid feature extractor consisting of a CNN-based backbone and a transformer-based component. The feature harmonization module is adopted to harmonize features at adjacent semantics in different scales. Meanwhile, the input image is fed into the frequency prior generator to calculate frequency priors. Finally, the frequency priors are embedded into the harmonized multi-scale features with the frequency prior embedding module. The details of each module are described in the following subsections.

3.2 Pyramid Feature Extractor

Convolution neural networks are widely adopted for common vision tasks and get a satisfactory performance. However, in the DIS task, the size of the input image is always large and the target objects own abundant details. It is difficult for shallow networks to learn rich semantics and refine the features fed with large input since they concentrate more on local information. To obtain more semantic information on multi-scale, we use the vision transformer with long-distance modeling capability to deepen the network in addition to the convolution layers in the pyramid feature extractor.

Specifically, the pyramid feature extractor first uses ResNet-50 as the CNN-based backbone to extract multiscale features {Xi}4i=1 ∈ R H 2i × W2i ×Ci from the input image I ∈ RH×W×3 , where H and W denote the height and width, and Ci ∈ {256, 512, 1024, 2048} is the number of channels. Then the feature X4 is downsampled by a 3 × 3 convolution layer with a stride of 2, fed into a transformer block [Ren et al., 2022] to get X5. Finally, we use the same operations on X5 to obtain X6. Notably, the channel of X5 and X6 are 256. To facilitate subsequent processing, we convert the channel of all these features {Xi}6i=1 to 96.

3.3 Frequency Prior Generator

Inspired by [Qian et al., 2020], to generate frequency priors, the discrete cosine transform (DCT) is first used to transform the image I into the frequency domain to generate the frequency distribution map M, i.e.,

Next, the fixed filter and the learnable filters extract different and valid frequency components. Especially, the fixed filter divides the frequency components into different bands (low frequency, medium frequency, high frequency, and all frequency), while the learnable filters provide more abundant information, and σ = 1-exp(-x) 1+exp(-x) used to normalize x to the range between -1 and +1. Finally, the frequency priors XFP are generated using the inverse discrete cosine transform (iDCT):

where iDCT(·) denotes inverse discrete cosine transform, ⊗ is Hadamard product, Ff and Fl are the fixed filter and the learnable filters, respectively.

3.4 Feature HarmonizationModule

The multi-scale features generated by the pyramid structure contain different structural and semantic information, inducing considerable heterogeneity. The shallow layers capture abundant details, while the deeper layers extract features with more semantics. The fusion of these multi-scale features would drive the model to focus on both detailed and abstract information. Therefore, we design a feature harmonization module to harmonize features at different scales. As shown in Figure 3, the FHM consists of two major parts, namely the grouping component and the harmonization component, each of which is detailed as follows.

Grouping Component. In this component, given features Xi and Xi+1 from the pyramid feature extractor, we expand the channel by N times and then split them into N groups. In this way, the intra-layer groups from the same input contain similar features, while the inter-layer groups from different inputs contain features with a large variation. Finally, we obtain 2N groups of features. Each grouped feature is further split into three sub-groups along the channel dimension.

Harmonization Component. After the grouping of the given inputs Xi and Xi+1, the semantics within each group of layers are tight, and the semantic difference between each group of different scales is obvious. Therefore, we need to achieve feature harmonization, and the core of the harmonization mechanism is to use a gate unit for filtering. There are two harmonization mechanisms namely the inter-group harmonization and intra-group harmonization. For inter-group harmonization, as shown in Figure 3, it (green block) collects a splitter of all groups. So it has 2N inputs with different scales. For intra-group harmonization, it collects splitters with the same scale as shown in Figure 3 (blue block). So, for each scale, it has two harmonization components.

Using inter-group harmonization as an example, we denote the grouped features as {Gn}Nn=1 , set N = 4. As shown in the far right of Figure 3, we first obtain the aggregated feature by adding all input features. Note that, for inter-group harmonization, we interpolate the splitters fromXi to the same scale as Xi+1 since these inputs are with different scales. These splitters are then fed into a gate unit, which computes the modulation weight matrix A. Mathematically, the gate unit is defined as follows:

where Avg(·) is an adaptive average pooling operation acting on the spatial dimensions of features, MLP(·) stands for a multilayer perception, ReLU(·) denotes a ReLU activation function, and Softmax(·) represents the softmax function.

The modulation weights are multiplied with the original features to obtain a harmonized feature map, i.e.,

Finally, the output of inter-group harmonization is interpolated into the same size of each branch. We concatenate the harmonized features from the same branch, pass them through a convolution layer for smoothness, and add them with the input features to obtain the output features (X′i ,X′i+1 ) of the feature harmonization module.

3.5 Frequency Prior EmbeddingModule

Incorporating frequency priors into image features makes the output contain more detailed information. However, direct fusion leads to inferior performance due to the semantic gap between the frequency and image domains. To address this problem, in the frequency prior embedding module (FPEM), we design the feature embedding component which consists of K groups of dynamic filters followed by an adaptive modulation. To make full use of the frequency priors, we adopt the cascaded feature propagation mechanism.

Feature Embedding Component. For clarity, we denote the inputs of the feature embedding component as paired features (Xu,Xd) at upper and lower branches, and the output as feature Xo. As shown in Figure 4, the features at two branches are added up to obtain the aggregated feature first. Then, the aggregated feature is passed through K branches of dynamic filters to learn different information from different representation domains. Each branch consists of a convolution operation to reduce the channel dimension to a quarter of the original for information compression, a BatchNorm layer and a ReLU layer to select the important part and another convolution layer and BatchNorm layer to restore the channel dimension. The output Xdf(k) of each branch is calculated with:

where DynFilterk (·) denotes the kth dynamic filter. {Xdf(k)}Kk=1 is then fed into the adaptive modulation unit to compute the attention coefficient matrix Aam:

where Sigmoid(·) denotes the sigmoid operation, and ω is 2. Finally, the output feature Xo is defined as follows:

where Reverse(·) is a reverse operation [Chen et al., 2018b] . Cascaded Feature Propagation. Benefiting from the advantages of the frequency prior feature, we first embed frequency priors into the reconciled features. Since the feature information embedded in the frequency priors has a strong representation ability, in order to further make full use of this information, we adopt a cascade method to propagate the information from the low-resolution deep semantics to the shallow highresolution feature space. Therefore, in the FPEM, there are two cascaded frequency embedding components. The first frequency embedding component uses the frequency priors embedded into the coordinated feature as input to obtain the frequency-embedded coordinated feature. Then the second frequency embedding component takes the output of the first frequency embedding component and the output of the upper layer FPEM as input to realize the transfer from deep semantics to shallow semantics. Considering that the deepest features are missing the output of the previous level, the E6 is obtained from X6 by using a transformer block to enrich the global semantics.

3.6 Loss Function

Ground truth supervises the predictions of six FPEMs and E6 with the same loss function L, which consists of a weighted Intersection over Union (IoU) loss LωIoU [Wei et al., 2020] and a weighted Binary Cross Entropy (BCE) loss LωBCE [Wei et al., 2020] defined as follows:

where P and G denote the prediction and the ground truth.

4 Experiments

4.1 Experimental Settings

Dataset. We performed extensive experiments on a largescale benchmark dataset, DIS5K [Qin et al., 2022], which contains a total of 5,470 images from 225 categories. The entire dataset is divided into three subsets: DIS-TR, DIS-VD and DIS-TE. DIS-TR and DIS-VD contain 3,000 training images and 470 validation images, respectively. DIS-TE is further split into four subsets (DIS-TE1, 2, 3, 4) with ascending shape complexities, each containing 500 images. The DIS5K dataset covers diverse objects with different geometric structures and appearances. Meanwhile, it provides images with higher resolution, more complex structural details, and higher annotation accuracy than existing object segmentation datasets. Therefore, segmentation on DIS5K is challenging and demands models with solid capabilities in identifying structural details.

Implementation Details. The model is implemented with the Pytorch framework on an A100 GPU. In the training stage, the ResNet-50 [He et al., 2016] is pre-trained on ImageNet-1K [Deng et al., 2009], the rest of the modules are initialized randomly. And we use the Adam optimizer with an initial learning rate of 1e-4, decaying by 10 every 50 epochs. The number of training epochs is set to 200. The resampled images with a size of 1024 × 1024 are fed into the proposed FP-DIS to get segmentation results in an end-to-end manner. No post-processing is needed throughout the inference process.

Evaluation Metrics. To make a comprehensive and fair comparison, we adopt five widely-used metrics to evaluate the performance, including maximal F-measure F m β [Achanta et al., 2009], weighted F-measure F w β [Margolin et al., 2014], Mean Absolute Error M, Structure Measure Sα [Fan et al., 2017] and Mean Enhanced Alignment Measure Em φ [Fan et al., 2018]. F m β and F w β consider both precision and recall in the binary classification of all the pixels. M is a measure of element-wise difference between the prediction and the paired ground truth mask. Sα is an effective indicator for evaluating structural similarity at region and object level. Em φ is widely used for evaluating pixel-level and image-level matching between the prediction and ground truth.

4.2 Comparison with State-of-the-arts

Quantitative Evaluation. Tables 1 and 2 show the comparison results of FP-DIS and 10 other methods, including UNet [Ronneberger et al., 2015], BASNet [Qin et al., 2019], GCPANet [Chen et al., 2020], U2Net [Qin et al., 2020], SINetV2 [Fan et al., 2021a], PSPNet [Zhao et al., 2017], DLV3+ [Chen et al., 2018a], HRNet [Wang et al., 2020], STDC [Fan et al., 2021b], and IS-Net [Qin et al., 2022]. The experiment results demonstrate that our model significantly outperforms other models across all five evaluation metrics. Specifically, taking all four test subsets into consideration (as shown in Table 2, using dataset DIS-TE(1-4)), the proposed method obtains remarkable advancements even compared with the second-best model, improving 3.2%, 4.4%, 1.3%, 2.8% and 3.7% of the F m β , F w β , M, Sα and Em φ , respectively. These comparison results demonstrate the effectiveness of our model.

Qualitative evaluation. Figure 5 shows the visual results of ours and typical methods. Compared with other methods, our model can capture more fine details from complex backgrounds, and accurately segment objects with complex structures. Furthermore, it can be seen from Figure 6 that the feature embedding frequency priors are more delicate, which can significantly improve the segmentation performance.

4.3 Ablation Study

In this section, we perform comprehensive ablation experiments on DIS-TE1. First, we investigate the frequency priors generated using different filters. Then we make a comparison of models with the feature harmonization module and frequency prior embedding module, respectively.

Frequency Prior Generator. The frequency prior generator contains discrete cosine transform and inverse discrete cosine transform using different filters. Therefore, we conduct three sets of experiments to verify the effectiveness of these filters with various settings. Actually, in the "w/o FPG" setting, we replace the FPG module with a convolutional layer, which means that the input image is processed by a convolutional layer before being fed into the FPEM module. As shown in Table 3, there is no significant difference in the performance using fixed or learnable filters in the FPG module, but the combination of these two kinds of filters contributed to better performance, indicating that the two filters are mutually beneficial and can effectively improve model's ability to detect detailed information. We also investigate the impact of changing the number of dynamic filter groups (1, 2, 3, 4, default is 2) in the feature embedding component. The results are shown in Table 4. Figure 6 shows the heatmaps of the backbone are relatively rough, and the heatmaps generated by the feature harmonization module and feature prior embedding module have more refined details.

Feature Harmonization Module. The feature harmonization module aims to reduce the variability between multiscale information and help the model to extract rich information from different scale features. From the experiment results shown in Table 5, we can see that the feature harmonization module contributes significantly to the performance of the model. Considering that the structure of the feature harmonization module may have an impact on the model, we list seven different grouping and separation schemes, and the experiments confirm that our proposed method performs better and has a more rational design. We also conduct experiments on removing the intra-layer and inter-layer harmonization components, as well as the gating mechanisms in the feature harmonization component, to study their contributions. The results shown in Table 6 demonstrate that these changes always lead to performance degradation compared to the proposed method, which verifies the benefits of each component. Comparing the output features from the pyramid feature extractor and harmonization features in Figure 6, we can see that the FHM effectively reduces ambiguity, and the harmonization features have higher confidence.

Frequency Prior Embedding Module. To verify the effectiveness of the frequency prior embedding module, we replace the FEC in different positions with addition operations. The results, shown in Table 7, reveal that FPEM is beneficial to improve the segmentation performance. The results of w/o FPEM prove that there is indeed a semantic gap between frequency prior feature and image features. Additionally, we replace the adaptive modulation with an addition operation to evaluate its effectiveness (i.e., w/o adapt). The adaptive modulation strategy can reduce this gap, improving the segmentation accuracy of the model. According to the heatmap from the frequency prior embedding module in Figure 6, the detected objects have better refine textures.

5 Conclusions

In this paper, we have proposed a novel DIS model, FP-DIS, which can generate frequency priors to guide fine-grained segmentation. We innovatively embed frequency priors into the image features to achieve accurate DIS. For this purpose, we adapt the frequency prior feature through dynamic filters to extract accurate frequency information. Meanwhile, we reduce the heterogeneity in multi-scale image features by harmonizing the features. Finally, we propose a frequency prior embedding module, which provides efficient frequency priors embedding for predicting DIS maps. Extensive experiments demonstrate the advantages of the proposed model over the existing models. Further ablation experiments sufficiently verify the effectiveness of the proposed modules.

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grants 62201465 and 62001425, and the Fundamental Research Funds for the Central Universities under Grant D5000220213.

Contribution Statement

Yan Zhou and Bo Dong contribute equally to this work.