SCOUT: Semi-supervised Camouflaged Object Detection by Utilizing Text and Adaptive Data Selection

1 Introduction

Camouflaged object detection (COD) [Fan et al., 2020; Fan et al., 2022] aims at segmenting objects that are visually concealed in their surroundings, which has important applications in several fields, such as military [Cannaday et al., 2023], environmental monitoring [Yadav et al., 2018], urban security [Li et al., 2024] etc. Existing COD methods always require large amounts of labeled data to segment camouflaged objects precisely [Mei et al., 2021].

However, camouflaged objects often employ complex camouflage strategies to blend deeply with their background. Such camouflage strategies make pixel-level camouflaged object annotations difficult to obtain, and annotating the entire dataset requires a significantly greater cost. To reduce the annotation costs, semi-supervised COD methods utilize only a small amount of labeled data alongside a large amount of unlabelled data. However, as shown in Figure 1, we identified the following two main issues with existing methods [Lai et al., 2024; Zhang et al., 2024]: 1). Most existing methods rely on random sampling to select a portion of the data as the labeled set, without thoroughly considering the quality of the selected data. This often results in the annotation of some meaningless data. 2). For the selected small amount of labeled data, existing models struggle to fully learn camouflage-related knowledge, resulting in poor segmentation performance.

∗Corresponding author

Therefore, we argue that there is still significant room for improvement in the effectiveness of unlabeled data utilization. As shown in Figure 1, we propose a SCOUT model that introduces an adaptive selection strategy and a text guidance strategy. Specifically, the adaptive selection strategy is implemented by the Adaptive Data Augment and Selection(ADAS) module. The ADAS module adaptively selects valuable training data through the Adaptive Data Augment (ADA) and the Adaptive Data Selection (ADS) component. The text guidance strategy is implemented by the Text Fusion Module (TFM). To help the model fully learn camouflagerelated knowledge, TFM introduces referring text assistance and enhances representation ability, thereby improving the model's performance across various camouflage scenarios.

By annotating precise image-level referring text for existing mainstream COD datasets, we develop a new dataset called RefTextCOD. Our comprehensive evaluations on this dataset demonstrate substantial enhancements over existing semi-supervised COD models. Especially, our method improves the Mean Absolute Error (MAE) by 52.0% and the S-Measure by 19.1% compared to previous SOTA semisupervised COD methods. Additionally, our approach outperforms some supervised COD methods, highlighting its greater practical applicability.

Our main contributions are summarized as follows:

• We proposed an innovative semi-supervised COD model SCOUT, and extensive experiments have demonstrated its high performance and effectiveness.
• To avoid meaningless data annotations, we proposed an ADAS module, which selects valuable data through an adversarial augment and sampling strategy.
• To fully utilize the referring text, we proposed a TFM module, which leverages the selected valuable data by combining camouflage-related knowledge and textvisual interaction.
• We build a new dataset, namely RefTextCOD. We performed image-level referring text annotations on existing mainstream camouflaged object detection (COD) datasets, providing a data foundation for this paper and future explorations of other COD tasks.

2 RelatedWorks

2.1 Camouflaged Object Detection

Camouflaged object detection (COD) has a long-standing history, and deep learning-based COD methods have seen rapid development in recent years. Existing fully-supervised COD methods have already achieved great performance by employing multiple strategies. For example, [Pang et al., 2022; Pang et al., 2023] extract multi-scale features from the backbone and design strategies for fusion. [Fan et al., 2022; Jia et al., 2022; Zhang et al., 2022] further use multi-stage refine. Some methods introduce additional information, e.g. boundary guidance [Sun et al., 2022; Ji et al., 2023; Zhai et al., 2021], texture clues [Ji et al., 2023; Zhu et al., 2021; Ren et al., 2023], and other information such as frequency domain and depth [Zhong et al., 2022; Lin et al., 2023].

The fully-supervised COD methods heavily rely on a large amount of labeled data, while pixel-level annotation incurs significant costs. As a result, some unsupervised, weaklysupervised, and semi-supervised methods have emerged. UCOS-DA [Zhang and Wu, 2023] is the first unsupervised COD method that addresses the task as a domain adaptation problem. SCOD [Zhang et al., 2024] is the first weakly supervised COD method, which introduces a novel feature-guided loss and consistency loss with a new scribble learning approach. Semi-supervised COD methods have also received considerable attention from researchers. This paper focuses on investigating the rationality of data selection and the sufficiency of data utilization in semi-supervised COD methods.

2.2 Semi-Supervised Camouflaged Object Detection

In traditional fully-supervised learning, models require extensive labeled data for training to achieve optimal performance. However, obtaining labeled data in practical applications is often costly and time-consuming. Semi-supervised learning (SSL) enhances the model's generalization capability by combining labeled data and unlabeled data [Chen et al., 2023; Grandvalet and Bengio, 2004; Mi et al., 2022], which effectively addresses the challenges of acquiring labeled data. Some previous works [Chen et al., 2021; Sohn et al., 2020; Wang et al., 2022] introduce the pseudo-labeling mechanism, where a teacher model is trained using a small amount of labeled data, and then the teacher model is used to produce the pseudo labels of the unlabeled data for the subsequent training of the student model. Semi-supervised learning has shown significant potential in various fields.

In semi-supervised COD methods, CamoTeacher [Lai et al., 2024] applies different augmentation strategies to teacherstudent networks and introduces a dual-rotation consistency loss. SCOD-ND [Fu et al., 2024] proposes a window-based voting strategy and an ensemble learning algorithm to eliminate noise from labels. However, these methods typically split the dataset into labeled sets and unlabeled sets by random sampling, without considering the value of the selected data. Moreover, they do not make full use of the labeled data. In this paper, We adaptively select valuable samples for training and introduce referring text to assist the model in learning camouflage-related knowledge from the labeled data.

2.3 Referring Camouflaged Object Detection

The concept of Referring Camouflaged Object Detection (Ref-COD) was first proposed by [Zhang et al., 2023], which leverages a batch of images as the referring information to guide the identification of the specified camouflaged objects. With the development of MLLMs, the rich intrinsic knowledge that MLLMs learned from massive amounts of data can be used to augment a variety of downstream tasks. Recently works [Cheng et al., 2023; Hu et al., 2025] have extended this concept by utilizing MLLMs and designing a series of prompts to assist the COD task. In this paper, we focus on how to use precise referring text to assist the model in learning camouflage-related knowledge from labeled data.

2.4 Active Learning

Active learning in the context of deep learning is a strategy designed to reduce the labeling cost by allowing the model to selectively query the most informative samples from an unlabeled dataset for annotation. It is particularly useful in scenarios where data labeling is expensive, time-consuming, or requires expert knowledge. Active learning approaches can be divided into membership query synthesis [Angluin, 1988; King et al., 2004], stream-based selective sampling [Argamon-Engelson and Dagan, 1999], and pool-based active learning from application scenarios [Settles, 2009]. In this paper, we use an adversarial augmentation and selection strategy to select valuable data that the model can partially understand but still needs further learning.

3 RefTextCODDataset

To adapt to this work, we build a new dataset, namely RefTextCOD. The motivation behind this proposed dataset is that existing datasets do not have image-level referring text.

To achieve a fair and effective comparison with existing methods, we expect to be able to construct text-based referring COD experiments in settings that are as similar as possible. Referring to [Chen et al., 2022; Fan et al., 2022], we used the mainstream COD datasets: CHAMELEON ([Wu et al., 2019]), CAMO ([Yan et al., 2021; Le et al., 2019]), COD10K ([Fan et al., 2020]), NC4K ([Le et al., 2019]) as the base image data. This allows us to focus more on annotations without collecting camouflaged images from scratch.

It is time-consuming to annotate camouflaged objects with captions, and manual annotating tends to lead to inconsistent quality. To ensure that the camouflaged object captions contain meaningful information for COD tasks (i.e. texture, color, and shape), we used two vision language models (e.g. QwenVL [Bai et al., 2023] and GPT4-Vision [OpenAI, 2024]) to generate a summary first. Specifically, we design a series of prompts with contextual logic: 1). First, the VLM model will be guided to locate the object and justify its classes; 2). Then, the model is directed to characterize the physical properties of foreground objects and background. 3). Finally, we request the model to aggregate and streamline all the features to generate complete referring text.

Finally, we employing manual screening and conducted a thorough review and revision of all annotations to ensure their accuracy and quality. As a result, we annotated 9,487 images from four datasets and organized them into a new dataset called RefTextCOD. The prompt used in the above process, the whole pipeline will be provided in the appendix.

4 Methods

The overview of SCOUT is depicted in Figure 2. In Section 4.1, we introduce the Adaptive Data Augment and Selection (ADAS) module. The module selects valuable data for annotation through an Adaptive Data Augment (ADA) component and an Adaptive Data Selection (ADS) component. In Section 4.2, we present the Text Fusion Module (TFM). This module conducts semantic knowledge transfer and text-visual interaction to make full use of the selected valuable data. We provide the details of the total loss and training process in Section 4.3.

4.1 Adaptive Data Augment and Selection

During this phase, our objective is to select valuable data from an unlabeled set DU = {XIu,j }Mj=1 for semi-supervised training, represented by the ADAS module. The ADAS module contains two sub-components: an Adaptive Data Augmentation (ADA) component and an Adaptive Data Selection (ADS) component. The ADA is a trainable data augmentation module through an adversarial augment strategy, while the ADS finds valuable data via an adversarial sampling strategy.

Formally, given the unlabeled set DU = {XIu,j }Mj=1 , the ADA component adopts a set of parameterizable data augmenter [Suzuki, 2022] {Aθc color , Aθg geo} (i.e. , color augmentation and geometric transformation) to augment each unlabeled data XIu,j ,

where XI,Aug u,j denotes the augmented unlabeled data. We feed XI,Aug u,j into both teacher and student models, parameterized by θT and θS, to obtain segmentation masks ˆYT u,j and ˆYS u,j ,

where F(XI,Aug u,j ; θT ),F(XI,Aug u,j ; θS) denote the predictions of the teacher model and student model, respectively, on augmented unlabeled data XI,Aug u,j .

After obtaining the segmentation masks ˆYT u,j and ˆYS u,j , the ADS calculates the score of each unlabeled data XIu,j and selects valuable data for annotation. Specifically, we first use a scoring function S(·) to calculate the unlabeled data XIu,j scores,

where SSIM(·) denotes the structural similarity metrics, and MAE(·) denotes the mean absolute error. The ADS component uses Kernel Density Estimation (KDE) normalization to normalize all scores and uniformly map them to the [0, 1] interval, then it sorts the unlabeled set DU based on these scores. At this point, data with scores close to 0 are considered hard data, while those close to 1 are considered easy data. Among these ordered data, extremely low-scoring hard data are too complex for the model or contain noise that hinders its convergence. On the other hand, extremely highscoring easy data are of no value to train the model and contribute to overfitting. Therefore, the ADS component selects data with moderate scores that are close to 0.5, which the model can partially understand but still needs further learning. Specifically, the ADS component samples the unlabeled data in unlabeled set DU = {XIu,j }Mj=1 with scores close to 0.5. Those selected unlabeled data XIl,i will undergo referring text annotation XTl,i and segmentation mask annotation Yl,i to construct the labeled set DL = {XIl,i ,XTl,i ,Yl,i}Ni=1 for next training phase. Those remaining unlabeled data will form a new unlabeled set DR = {XIu,o }Oo=1 .

To obtain a pre-trained teacher and student model, we initialize the labeled set DL by considering the diversity of data in DU. Specifically, we perform K-Means clustering using the color, texture, and frequency domain features of data in DU. Then we select the data near the clustering centers to construct the initial labeled set DL.

The ADA component adopts an adversarial manner for training. Specifically, given the labeled set DL, we use the pre-trained augmenter {Aθc color , Aθg geo} to obtain the augmented input XI,Aug l,i by Eq. (1). Then we train the teacher and student models with Binary Cross-Entropy (BCE) loss LBCE. Following [Suzuki, 2022], the training objective of augmenter loss LAug is to minimize the loss of the teacher model and maximize the loss of the student model,

4.2 Text FusionModule

The TFM module utilizes the precise referring text XTl,i in the labeled set DL = {XIl,i ,XTl,i ,Yl,i}Ni=1 to further extract the knowledge of camouflage object. Specifically, for input labeled data XIl,i , with corresponding referring text annotations XTl,i , the TFM utilizes CLIP-Text Encoder [Radford et al., 2021] CLIPT (·) to obtain text feature FTl,i ,

For input labeled data XIl,i , assuming that the hierarchical features obtained by the Image Encoder are {FIl,i,k }, k ∈ [1,NE],where NE is the number of hierarchical feature levels, we use a clue attention mechanism to generate the clue query QCLUE, which highlights the camouflage-related region by the guidance of attention score ACAMO l,i .

where qCAMO is a learnable query, andWQ,WK,WV denote the linear projection weight, respectively. dh represents the attention head dimension. In order to enable the clue attention mechanism to learn how to extract camouflage-related region features, the following loss function is leveraged,

where down(·) denotes a downsampling function. Then we apply QCLUE to calculate the similarity scores W C between the codebook C T that stores class-related knowledge extracted from the labeled data, and then aggregates to refine coarse referring text, the similarity scores is calculated by

where the COS(·) is the consine similarity function. Next, we use these scores W C to perform a weighted aggregation of the vectors in the codebook, generating a clue vector,

We use the class word XTcls l,i extracted from the referring text XTl,i to supervise the clue vector V CLUE,

We combine this clue vector V CLUE with the referring text features FTl,i to enhance the representation ability. Then, we

CAMO (250)

COD10K (2026)

NC4K (4121)

use a hierarchical feature cross-attention mechanism for textvisual information fusion,

where Concat(·) denotes the concatenation operation, and CrossAttn(·) denotes the cross-attention function. During training and inference of unlabeled data in DR, only the attention scores ACAMO l,i are supervised by the pseudo-labels from the teacher model. The total loss function for the TFM module can be formulated as,

where λt = 1 for labeled data in DL and λt = 0 for unlabeled data in DR. We use the class words XTcls l,i extracted from the referring text annotation XTl,i of the labeled data in DL to initialize the codebook C T .

4.3 Total Loss

The total loss function can be represented as,

where Ls denotes the supervised loss, Lu denotes the unsupervised loss, and λu is the weight of unsupervised loss to balance the loss terms. The labeled loss function consists of three parts,

where LsSeg denotes the supervised segmentation loss between the student predictions and groud-truth. The unlabeled loss function consists of two parts,

where LuSeg denotes the semi-supervised loss between the student predictions and pseudo labels (produced by the teacher model). Following [Zheng et al., 2024], we use a combination of Binary Cross Entropy loss LBCE, Intersection Over Union loss LIOU, and Structure Similarity Index Measure loss LSSIM to build the supervised loss LsSeg and semisupervised loss LuSeg . The complete definition of these loss functions can be found in the appendix.

5 Experiments

5.1 Experiment Settings

Training Set. To compare with the existing works, following [Luo et al., 2023; Fan et al., 2020], we use 1000 images from the CAMO trainset and 3040 images from the COD10K trainset as the training set for our experiments. During the training process, we follow the data partition ratios from previous semi-supervised COD results [Lai et al., 2024], training the model with 1%, 5%, and 10% of labeled data. However, diverging from traditional semi-supervised segmentation approaches, we do not employ a random data sampling strategy. Instead, we utilize the proposed ADSA module to actively select the valuable data with annotating labels (i.e. segmentation mask and referring text), while the remaining portion is treated as unlabeled data. For all unlabeled data, the referring text is fixed to a single sentence "Camouflaged objects; hidden objects; concealed objects".

Testing Sets. We test the model's performance on four mainstream COD benchmark testing sets, CHAMELEON with 76 test images, CAMO with 250 test images, COD10K with 2026 test images, and NC4K with 4121 test images. To comprehensively evaluate the model, we test its performance under two different settings: using fixed referring text (i.e. "Camouflaged objects; hidden objects; concealed objects") and using precise image-level referring text during test.

Evaluation Protocol. For a fair and comprehensive evaluation, we employ the S-measure (Sm) [Fan et al., 2017], mean and weighted F-measure (F m β , F ω β ) [Margolin et al., 2014], max and mean E-measure (E x ϵ , E m ϵ ) [Fan et al., 2018], mean absolute error (M) [Perazzi et al., 2012].

Implementation Details. All images are resized to 640×640

for training and testing. We employ the ImageNet pretrained Swin-base [Liu et al., 2021] as our image encoder, use recently developed BiRefBlock [Zheng et al., 2024] from High-Resolution Dichotomous Image Segmentation (HRDIS) fields to build the decoder, and utilize CLIP-ViT-Large as our text encoder. The parameters of the CLIP text encoder are frozen during the training process, while all others are trainable. The batch size is set to 6 for each GPU during training, Adam is used as the optimizer, and the learning rate is initialized to 1e-4 and use multi-step decay strategy with 30 training epochs. All experiments are implemented with PyTorch 2.1 and a machine with Intel(R) Xeon(R) Silver 4214R CPU @ 2.40GHz, 256GiB RAM, and 8 NVIDIA Titan A80080G GPUs.

5.2 Main Results

Qualitative Analysis. We show visualizations of a series of camouflaged object segmentation masks predicted by our method and related methods in some challenging scenarios. As shown in Figure 3, we notice that our method achieves higher segmentation accuracy than existing semi-supervised COD approaches, with clearer edges and a more detailed representation of the camouflaged object.

Quantitative Analysis. We compare the proposed SCOUT's performance with existing semi-supervised models on four COD test datasets. Since these methods are not open-sourced and are difficult to reproduce, we only report the experimental results published in their original papers. As shown in Table 1, SCOUT has already surpassed all previous methods across all metrics on all datasets when tested with fixed text (e.g. "Camouflaged objects; hidden objects; concealed objects; CLUE Token"). Thanks to the assistance of the clue vector in TFM, SCOUT effectively improves fixed text, achieving results nearly identical to those inferences with precise texts. Compared to the previous best results, our method achieves an improvement of 52.0% in M, 19.1% in Sm, 16.3% in E m ϕ , and 29.1% in F m β , effectively demonstrating the high performance of the proposed SCOUT.

5.3 Ablation Study

Module Ablations. To validate the effectiveness of the proposed module, we first perform ablations on the ADA and ADS components in the ADAS module and the TFM module. When the ADS component is not used, the labeled set is obtained through random sampling. When the TFM module is not used, the referring text does not participate in model training. We retrain the model with 5% labeled data setting and test it on COD10K-Test set. The results are shown in Table 2, which indicates a significant performance drop when neither module is used. Compared to the baseline, our method

achieves an improvement of 28.9% in M and 6.21% in Sm.

Effectiveness of Data Augment. We conduct comparative experiments between random augmentation and the trainable augmenters used in this paper. We retrain the model on different settings with 5% labeled data and test on the COD10KTest set. As shown in Table 3, when using the learnable augmentation, we find that color augmentation has a significant effect. This is related to the complex colors typically found in camouflaged objects.

Effectiveness of Data Scoring. We conduct an ablation experiment on the data sampling strategy used in the ADS component in the ADAS module. Specifically, we resample 5% of the data by using Top-K easy, Top-K hard, and different score selection centers, and train the model. We then test it on the COD10K-Test set, and the results are shown in Table 4. When only easy and hard samples are used, the model struggles to achieve optimal performance. However, when selecting data around 0.5, the selected data not only suits the model's learning but also balances difficulty, resulting in the best performance.

Effectiveness of Referring Text. We conduct further exploration into the role of referring text. We compare the performance of using precise and fixed referring text during both training and testing on the unlabeled set. The results are shown in Table 5. We find that precise referring text indeed helps the model learn camouflage-related knowledge more effectively, and the TFM is able to enhance fixed text, achieving performance comparable to that of precise text.

6 Conclusions

In this paper, we address the shortcomings of existing semisupervised COD methods, which fail to adaptively select and utilize high-quality data, resulting in poor performance. We propose an innovative semi-supervised COD model SCOUT. Specifically, we propose the ADAS module to avoid meaningless data annotations by selecting valuable data through an adversarial augment and sampling strategy. Additionally, we propose the TFM to fully utilize the referring text by combining camouflage-related knowledge and text-visual interaction. Furthermore, we propose a RefTextCOD dataset, which contains a large number of image-level referring text annotations. Extensive experiments demonstrate the effectiveness of the proposed framework and modules.

Ethical Statement

There are no ethical issues.

Acknowledgements

This work was supported by the National Science Fund for Distinguished Young Scholars (No.62025603), the National Natural Science Foundation of China (No. U21B2037, No. U22B2051, No. U23A20383, No. 62176222, No. 62176223, No. 62176226, No. 62072386, No. 62072387, No. 62072389, No. 62002305 and No. 62272401), and the Natural Science Foundation of Fujian Province of China (No. 2021J06003, No. 2022J06001).