Mixture-of-Queries Transformer: Camouflaged Instance Segmentation via Queries Cooperation and Frequency Enhancement

1 Introduction

Camouflage is a naturally evolved strategy for animals to hide themselves via adapting their body's coloring to match the surroundings, which is used for hunting prey or avoiding detection by natural enemies, as shown in Figure 1(a). Since there is a lot of demand for understanding the widespread camouflage-like scenarios, (e.g., polyp segmentation [Fan et al., 2020b], lung infection segmentation [Fan et al., 2020c], search-and-rescue work [Turi´c et al., 2010], manipulated image/video detection and segmentation [Xu and Feng, 2023; Zhang et al., 2024]), the task of predicting the location and instance-level masks of camouflaged objects (i.e., Camouflaged Instance Segmentation, CIS) has been proposed. Therefore, CIS is worth studying and has gradually received more attention in recent years. However, it also has challenges due to high intrinsic similarities between the target objects and the background.

∗Corresponding author

Compared to the tremendous development in generic instance segmentation [Bolya et al., 2019; Wang et al., 2020a; Wang et al., 2020b; Ren et al., 2015; He et al., 2017; Cai and Vasconcelos, 2019; Chen et al., 2019], camouflaged instance segmentation remains an under-explored issue, and only a few efforts have been made to study it in the past three years [Pei et al., 2022; Luo et al., 2023; Dong et al., 2023; Li et al., 2024; Le et al., 2023]. CFL [Le et al., 2021] is a first attempt. It is a two-stage method that fuses general instance segmentation methods for camouflaged instance segmentation but has limited performance. Subsequently, OSFormer [Pei et al., 2022] is proposed as the first one-stage method for CIS. It takes advantage of a transformer network, which achieves a flexible framework that can be trained endto-end for camouflaged instance segmentation. Recently, DCNet [Luo et al., 2023] has been proposed to segment camouflaged instances via explicit de-camouflaging and achieves CIS by jointly modeling pixel-level camouflage decoupling and instance-level camouflage suppression. In the same period, UQFormer [Dong et al., 2023] adopts a typical DETRlike architecture [Carion et al., 2020] and exploits the information on object edges.

Although the aforementioned works have made some progress, two fundamental inspirations have not been taken into account: the priori of camouflage principles and the human habit of segmenting camouflaged instances. (1) The priori of camouflage principles: Only when you know how to camouflage can you see through camouflage. Many years ago, zoologists discovered that animals can camouflage themselves by matching their colors or patterns with the background. To look deeper into camouflage, we sample some images of camouflaged animals and analyze them thoroughly. Since it is hard to spot camouflaged objects in the surroundings, we perform the Fourier transform on these images to discover some clues in the frequency domain. We first decompose these camouflaged images into phrase and amplitude components and reconstruct images from only phrase component and amplitude component, respectively, presented in Figure 1. It is easy to find that the phase component of the Fourier spectrum preserves high-level semantics (contours and semantics) of original images, while the amplitude component contains low-level statistics (colors and styles). Therefore, enhancing the influence of contours and eliminating the interference of colors would certainly benefit the performance on CIS. (2) The human habit of segmenting camouflaged instances: When humans segment a camouflaged image, their visual system instinctively sweeps across the scene and determines some candidates. Then, the visual system gradually searches for valuable clues throughout the scene to obtain accurate segmentation masks. For some heavily camouflaged scenes with highly accurate segmentation like some medical image datasets [Fu et al., 2019], it may even refine the masks labeled by multiple experts. Gradually refining and integrating the decisions of multiple experts are potentially effective for CIS. Therefore, it makes sense to take full advantage of both inspirations of (1) and (2) for improving the performance of the CIS task.

Motivated by the above discussions, we proposed a Mixture-of-Queries Transformer (MoQT) trained in an endto-end manner for CIS, which includes a Frequency Enhancement Feature Extractor (FEFE) based on modeling the colors and contours of camouflaged instances and a Mixtureof-Queries Decoder (MoQ Decoder) in transformer architecture referring to the segmentation process of multi-experts collaboration. First, inspired by the camouflage principles discussed above, we design a Frequency Enhancement Feature Extractor to capture more clues of camouflaged instances in the frequency domain. Specifically, we propose to adopt Fourier spectrum amplitude and phase to model image color information and contour information, respectively, as shown in Figure 1. With the help of color and contour information, we design a contour enhancement module and a color removal module, which can increase the contour effect while eliminating color interference. This mechanism in the frequency domain is suitable for debunking the principle of animal camouflaging, which is reasonable for providing gains on CIS. Second, for the Mixture-of-Queries Decoder, which is different from the standard DETR framework, we design multiple expert groups of queries inspired by the success of Mixture-of-Experts (MoE) mechanism. In DETRs, object queries are wonderful designs in decoders, which have two roles: a) candidates for objects and b) interaction with encoded features. And there are only a group of queries regarded as candidates initialized before feeding into the standard transformer decoder, as shown in Figure 2(a). These candidates are refined for segmentation by learning the parameters of each decoder layer, where each decoder layer determines whether the refined candidates are suitable for final prediction. Differently, we design multiple groups of queries (a group of queries considered an expert) in each decoder layer for explicitly refining the candidates, which can be illustrated in Figure 2(b). In each layer, there is an expert from the last layer and several new initialized experts. And we propose a gating network in each decoder layer to mix these experts, deciding which newly initialized experts are selected to refine candidates explicitly for the next layer forwarding. The gating network accepts the encoded features as input, and the output is the weights of recombination of the various experts. This mechanism can refine outputs hierarchically to a fine-grained level via a mixture of experts, which can generate more accurate instance masks, as shown in Figure 2(b). In favor of these two designs, our method can utilize multiple expert queries to integrate effective clues of camouflaged objects in both spatial and frequency domains, which can achieve outstanding segmentation performance.

In summary, our main contributions are three-folds. (1) We propose a Mixture-of-Queries Transformer (MoQT) trained in an end-to-end manner for CIS, which takes advantage of the priori of camouflage principles and refers to the human habit of segmenting camouflaged instances. To the best of our knowledge, this is the first attempt to introduce MoE into DETR-like frameworks for segmentation. (2) We proposed a Frequency Enhancement Feature Extractor (FEFE) and a Mixture-of-Queries Decoder (MoQ Decoder) for our MoQT, where FEFE is used for color removal and contour enhancement. The MoQ Decoder aims to mix multiple groups of queries hierarchically to provide more accurate predictions. (3) Extensive experimental results on COD10K and NC4K show consistent performance gains compared with 19 baseline methods and verify the superiority of our method.

2 RelatedWork

2.1 Camouflaged Object Detection

Camouflaged Object Detection is usually considered as one of the most important origins of CIS and aims to identify the camouflaged objects from the background and has witnessed the development of art and biology [Fan et al., 2020a; Le et al., 2019]. Early research [Pan et al., 2011; Sengottuvelan et al., 2008] in COD mainly uses handcrafted features (e.g., gradient, texture, and intensity features) to tell the camouflaged objects from their surroundings. Later, deep learning (DL) improves COD's performance in an end-to-end manner, and plenty of DL-based methods [Pang et al., 2022; Yang et al., 2021; Xu et al., 2021; Zhong et al., 2022; Mei et al., 2021; Ren et al., 2021] have been proposed. For example, ZoomNet [Pang et al., 2022] discusses how to capture camouflaged objects in complex surroundings in a multi-scale manner. Moreover, UGTR [Yang et al., 2021] combines the benefits of both Bayesian learning and transformer-based reasoning to handle camouflaged object detection with probabilistic and deterministic information. Some works [Xu et al., 2021; Zhong et al., 2022] even go beyond the RGB domain and explore frequency clues for better performance. In this paper, a Frequency Enhancement Feature Extractor, which refines frequency clues with contour enhancement and color removal, is adopted and allows full rein to both the camouflaged characteristics and the surrounding textures.

2.2 Camouflaged Instance Segmentation

Camouflage Instance Segmentation (CIS) learns most lessons from traditional instance segmentation. The purpose of instance segmentation is to assign pixel-level mask prediction for various instances. Nowadays, instance segmentation methods can be roughly divided into two parts: One-stage approaches [Bolya et al., 2019; Wang et al., 2020a; Wang et al., 2020b] and two-stage approaches [Ren et al., 2015; He et al., 2017; Cai and Vasconcelos, 2019; Chen et al., 2019]. Two-stage methods apply mask segmentation after proposal region detection, such as Faster R-CNN [Ren et al., 2015], Mask R-CNN [He et al., 2017], Cascade R-CNN [Cai and Vasconcelos, 2019], and HTC [Chen et al., 2019]. CFL [Le et al., 2021], the first attempt in CIS, also applies twostage instance segmentation methods. However, one-stage methods show faster inference than two-stage methods and achieve comparable performance. For example, YOLACT [Bolya et al., 2019] adopts two parallel tasks to produce nonlocal prototype masks with adaptive coefficients. Furthermore, SOLO [Wang et al., 2020a] and SOLO-v2 [Wang et al., 2020b] predict the instances' center and then decouple the instance masks with kernel feature learning. Recently, researchers have found transformers [Cheng et al., 2021; Cheng et al., 2022] show excellent performance on instance segmentation with the assistance of attention mechanisms and instance-specific prototypes. Therefore, transformer-based methods like OSFormer [Pei et al., 2022], DCNet [Luo et al., 2023] and UQFormer [Dong et al., 2023] utilize transformers in CIS and achieve great progress. Inspired by [Pei et al., 2022; Luo et al., 2023; Dong et al., 2023], our Mixtureof-Queries Transformer (MoQT) introduces a Mixture-ofQueries Decoder (MoQ Decoder) in the transformer decoder to combine the capabilities of multi-experts hierarchically, which enhances camouflage semantics and refines details of instance masks.

3 Method

3.1 Architecture Overview

The overall framework of our proposed model is presented in Figure 3. The whole architecture of our method is a typical MaskFormer-like [Cheng et al., 2021] model, composed of a Frequency Enhancement Feature Extractor (FEFE), a Pixel Decoder, and a Mixture-of-Queries decoder (MoQ Decoder). In the FEFE, we get valuable multi-scale features enhanced by the Fourier transform for revealing the camouflaged clues, where the phase component and amplitude can be used for modeling the information of contours and colors, respectively. We use a contour Enhancement Module (CEM) and a Color Remove Module (CRM) to mine the potential information of contours and eliminate the interference of colors for capturing clues of camouflaged instances. Then, the Pixel Decoder (based on FPN [Lin et al., 2017]) gradually upsamples low-resolution features from the output of the backbone to generate high-resolution per-pixel embeddings. The MoQ Decoder computes from per-pixel embeddings and some initialized experts (a series of queries) to get the output prediction. Specifically, in MoQ Decoder, we propose a Mixtureof-Queries Layer (MoQ Layer) after each decoder layer and transform M experts (each expert includes N queries) via self and cross attention mechanisms, where the MoQ Layer is used to combine the M experts of queries hierarchically. Finally, following previous work, we use a mask head and a matching algorithm to output the CIS prediction.

3.2 Frequency Enhancement Feature Extractor

As mentioned in our Introduction, the camouflage clues are mainly comprised of high-level semantics (e.g., contours and semantics) and low-level statistics (e.g., colors and styles), which can be reflected by the phrase and amplitude components of Fourier spectrum, respectively. As shown in Figure 1, it is believed that enhancing the influence of contours and eliminating the interference of colors would certainly benefit the performance of segmenting the camouflaged instances. Thus, to explore the camouflaged clues, we design FEFE (Frequency Enhancement Feature Extractor) to model the colors and contours of camouflaged objects, where the phrase and amplitude components are applied to identify the camouflaged semantics from surroundings in FEFE. Specifically, suppose H and W are the height and width of the input, and the Fourier transformation F(x) performed on each channel with a given camouflaged image x ∈ R3×H×W can be denoted as:

where J represents the imaginary unit, A(x) (modeling colors) and P(x) (modeling contours) are the amplitude and phrase components.

Then, we can get multi-scale image features Fk ∈ RH×W×C , k ∈ {2, 3, 4, 5} extracted from a backbone network with the origin image x. Besides, we feed A(x) into a lightweight CNN and a 1×1 convolution to obtain the global camouflaged color information Fcolor ∈ RC , and eliminate its interference via Color Remove Module (CRM). While for the phrase component P(x), due to that P(x) includes some information on contours and textures, extracting multi-scale features of P(x) can present unique advantages to mining camouflaged clues. Thus, we feed P(x) into the backbone and obtain hierarchical features Fkcontour , k ∈ {2, 3, 4, 5} to explore the effects of contours as much as possible by the Contour Enhancement Module (CEM). Formally, the process of FEFE, including CRM and CEM for each scale feature F ∈ {F2 , F3 , F4 , F5 }, can be expressed as:

where avg c and avg s indicate average pooling along spatial and channel axis, and δ is an activation function. CRM and CEM are designd to generate Mcolor and Mcontour, respectively. With the above module, we can get multi-scale refined features Fkenhance , k ∈ {2, 3, 4, 5}. Further, to acquire more fine-grained features for more accurate segmentation, we fuse Fkenhance , k ∈ {2, 3, 4, 5} by feeding these features into the pixel decoder based on FPN [Lin et al., 2017] architecture, which is used to gradually upsample low-resolution features from the output of the FEFE to generate high-resolution perpixel embeddings X ∈ R H4 × W4 ×C .

3.3 Mixture-of-Queries Decoder (MoQDecoder)

In order to capture camouflaged instances, the popular transformer-based architecture like MaskFormer [Cheng et al., 2021] and Mask2Former [Cheng et al., 2022], proposes a set of queries to identify whether each pixel belongs to a camouflaged instance. Meanwhile, as discussed in our Introduction, humans may segment camouflaged instances by gradually searching and multi-person collaboration. Inspired by the discussion, we propose a Mixture-of-Queries Decoder (MoQ Decoder) for hierarchically segmenting camouflaged instances.

Mixture-of-Queries Mechanism: Different from the standard Transformer decoder architecture, in each layer, we introduce a Mixture-of-Quries Layer (MoQ Layer) after the original decoder layer, and initialize M experts Ei, i ∈ [1,M], where each expert contains N queries qi, i ∈ [0, N - 1], qi ∈ Rd . Each query is responsible for an object candidate. So, we have Ei = {q0, q1, · · · , qN-1}, qi ∈ Rd . Further, the detailed architecture of the designed MoQ Layer is illustrated in Figure 4, where the gated network G outputs a sparse (M + 1)-dimensional vector G(x) ∈ RM+1 to indicate the weights of each expert. Therefore, given the input of each decoder layer E and the M initialized experts [E1,E2, · · · ,EM], the output y of the MoQ Layer can be written as follows:y = E · softmax(G(X)), (3)

where X is the output of pixel decoder, and G(X) indicates the output of the gated network and E =

[E′ ,E1,E2, · · · ,EM]. E′ is the output of the standard transformer decoder layer fed with E. Besides, the forward process of each MoQ Decoder Layer (including a original decoder layer and a MoQ layer) can be formulated as:

where LN is layer normalization and MLP denotes the multilayer perception network. During the training process, to provide deep supervision, we followMask2Former to adopt auxiliary losses with additional mask prediction heads and Hungarian match loss after each MoQ Layer.

3.4 Objective Function

As shown in Figure 3, with the fused feature X ∈ R H4 × W4 ×C output by Pixel Decoder and the instance candidates ˆE ∈ RN×C generated by the MoQ Decoder, we can finally obtain the segmentation map, which can be formulated as:

To train the whole network, following DETR [Carion et al., 2020], we adopt a Hungarian matching algorithm to match a ground truth label with each predicted segment instance. If no suitable label exists, a special label ("no object") is assigned. Therefore, including the instances and mask supervision, the objective function contains three terms: Cross-entropy Loss LCE for the instance score, Focal Loss Lfocal and Dice Loss Ldice for the mask predictions after each MoQ Layer, written as:

where L means the amount of decoder layers. By default, we set α = 20 and β = 1.

4 Experiments

4.1 Experimental Setups

Following the mainstream works of CIS [Dong et al., 2023; Luo et al., 2023], we evaluate our method in two datasets: COD10K and NC4K. COD10K includes 3040 training images and 2026 testing images, while NC4K contains 4121 test images for evaluating the generalization of proposed models. To provide a fair comparison, we train models in the training set in COD10K, and meanwhile test models in both test sets of COD10K and NC4K, which is a standard setting proposed in previous works [Luo et al., 2023; Pei et al., 2022]. In order to comprehensively evaluate the models, We use AP50, AP75, and AP scores as evaluation metrics to quantify the performance of our method and baselines [Luo et al., 2023; Dong et al., 2023; Pei et al., 2022]. Besides, it is free to check the supplementary material for more details of the method and experiments.

4.2 Comparison with State-of-the-Art Methods

The CIS task is a relatively novel task that has been proposed in recent years, and only a few previous works are involved in this task, such as OSFormer [Pei et al., 2022], DCNet [Luo et al., 2023], and UQFormer [Dong et al., 2023]. Consequently, we also adopt several popular generic instance segmentation methods as baselines on the CIS task for a more comprehensive test. And for a fair comparison, the backbone of these methods is configured as ResNet-50. The performance comparison results are shown in Section 3.2. It is easy to observe that our proposed model can consistently outperform the state-of-the-art methods by a large margin on both COD10K and NC4K test sets.

(1) Results on COD10K. As shown in Section 3.2, we compare our proposed model with 6 CIS models (i.e., OSFormer [Pei et al., 2022], DCNet [Luo et al., 2023], etc.), 13 generic instance segmentation models. Our model can achieve 51.8% in AP75, which outperforms the second best method DCNet [Luo et al., 2023] by 4.3% in AP75. In AP, our model also gets a performance improvement of 2.6%. Notice that our method does not achieve the highest value in AP50, instead of a comparable performance of 73.0% in AP50. These results indicate that our method can acquire more accurate segmentation masks of camouflaged objects.

(2) Results on NC4K. Likewise, we evaluate these methods on NC4K dataset, and the results on this test set reflect the generalization ability of these models. Our model yields 59.0% in AP75, while the previous best method DCNet is 56.5%, which demonstrates that our method gets an obvious gain of 2.5% in AP75, suggesting a great generalization ability of our model as well. In AP, our model achieves the highest performance metrics of 54.7%, surpassing the second best method (DCNet) by 1.9%. Besides, our model also obtains a 1.35% improvement in AP50. The overall metrics of various AP values reflect our method's obvious superiority over other baselines.

4.3 Ablation Studies and Visualizations

To look deeper into our proposed method, in this section, we present a series of ablation studies to demonstrate the effectiveness of each proposed module.

Effectiveness of proposed modules. To explore the effectiveness of the proposed FEFE and MoQ Decoder, we validate the importance of each component by removing them one at a time. As shown in Section 3.4, the performance without MoQ Decoder drops by 2.2 % in AP, 1.7% in APAP50 and 2.5% in APAP75 on COD10K-Test. On NC4K-Test, the metrics of AP, AP50 and AP75 are also reduced by 1.5%, 0.7% and 1.7%, respectively. Similarly, if the components of FEFE are ablated, there is a drop in segmentation performance as well. For example, on COD10K-Test, the performance just achieves 47.1% in AP , 72.8% in AP50 and 50.6% in AP75, which are consistently lower than that without any modules ablated (as shown in the last row of Section 3.4). The reduced performance demonstrates that these two proposed modules can capture clues of camouflaged instances and provide accurate segmentation. With both modules, our method can lead to huge performance gains in evaluation metrics.

Various backbones. To further explore the potential of our model, we equip it with different feature extractor backbones, such as ResNet-50 [He et al., 2016], ResNet-101 [He et al., 2016], SwinTransformer-Tiny (Swin-Tiny) [Liu et al., 2021], and SwinTransformer-Small (Swin-Small) [Liu et al., 2021]. For a fair comparison with baselines, all these models are pretrained on ImageNet-1k. The results are presented in Section 3.4. With the same backbone, our method shows the best performance among compared baselines, which indicates our method outperforms the state-of-the-art methods. For example, when ResNet-101 is the backbone, the metrics of AP of our method are 48.6% and 55.4% on COD10K and NC4K datasets, respectively, while the second best method just reaches 46.8% and 53.5%. With a larger backbone, the results also prove that our method has the potential for further improvement.

Analysis of the number of decoder layers. We apply auxiliary losses after each decoding layer, as formulated in Equation (6). Hence, the number of decoder layers L is important for the segmentation performance. As presented in Section 3.4, we vary the number of decoder layers, picked from the set {2, 4, 6, 8, 10, 12}. We find that the overall performance of the model improves with the increase of L. And when L = 6, the model can get the best performance. There is no additional performance gain when the L continues to increase, which may be caused by limited data to train the model for further improvement.

Ablation on the number of queries. Object queries are essential in the transformer architecture for prediction. Therefore, we study the performance with different numbers of queries in each expert group. As shown in Figure 5, we change the number of queries from 10 to 40 and evaluate the performance metrics of AP and AP75 in both COD10K and NC4K test sets. In fact, the number of queries in each group should be larger than the actual count of objects to avoid instance fusion, which is determined by the dataset distribution. Moreover, it can be seen that when the number is set as 10, our model obtains the best performance on both datasets. For example, when the number is 10, AP and AP75 is 47.99% and 51.77% on COD10K, respectively. Meanwhile, AP and AP75 reach 54.73% and 58.97%.

Impacts about Hyper-parameters. We study the impacts of the Hyper-parameters α and β in Equation (6). On both COD10K and NC4K datasets, when α = 20, the best performance is achieved, proved by the metrics of AP = 47.99% and 54.73%, respectively, shown in Figure 6(a). Therefore, we choose α = 20 in our method by default. For the hyperparameter β, we change the value of β from 0.1 to 2, and the results as presented in Figure 6 (b). It can be seen that the model gets the best performance when β = 1. Therefore, to get the best performance, we set α as 20, and β as 1.

Figure 7: Visualizations of Various Methods. Different colored masks indicate different instances.

Visualization Results. To comprehensively evaluate our method, we also present some qualitative analysis, as shown in Figure 7. We visualize the segmentation masks of various methods, including OSFormer [Pei et al., 2022], DCNet [Luo et al., 2023], and our method, to demonstrate the performance wtih qualitative results. It can be seen that our method performs better than previous methods, which can be proved by the clear boundaries and accurate masks of our method (shown in the last row of Figure 7). In short, our method not only improves the evaluation metrics on two datasets but also gains in visual results of segmentation masks.

5 Conclusion

In this paper, we propose a novel Mixture-of-Queries Transformer (MoQT) for camouflaged instance segmentation. MoQT applies a Frequency Enhancement Feature Extractor for feature extraction in the frequency domain, with the assistance of a contour enhancement module and a color removal module. Besides, a Mixture-of-Queries Decoder uses multiple expert groups of queries as candidates and shares semantic information with transformer encoder features. Multiscale features enable MoQT to refine prediction hierarchically and get fine-grained instance masks with collaboration of multiple groups of queries. Compared with plenty of stateof-the-art baselines, our proposed MoQT shows outstanding performance on two benchmark datasets, demonstrating the proposed method's effectiveness.

Acknowledgments

This work was supported by Ant Group Postdoctoral Programme. The authors from Ant Group are supported by the Leading Innovative and Entrepreneur Team Introduction Program of Hangzhou (Grant No.TD2022005).