Unified Unsupervised Salient Object Detection via Knowledge Transfer

1 Introduction

Salient object detection (SOD) aims to identify the most visually significant objects in images. Supervised SOD methods have achieved excellent results, but due to their heavy reliance on pixel-level annotations for salient objects, unsupervised SOD (USOD) has been gaining increasing attention. USOD not only eliminates the need for annotated data but also exhibits strong generalization performance when applied to other tasks [Niu et al., 2021; Wu et al., 2021].

Traditional SOD methods rely heavily on hand-crafted features, such as color and contrast, for saliency extraction. Although these methods prove effective for simple scenes, they encounter difficulties in complex scenes due to the absence of high-level semantic information. Existing deep learning-based USOD methods [Nguyen et al., 2019; Zhang et al., 2018] mostly utilize the predictions generated by traditional SOD methods as saliency cues and incorporate semantic information to generate refined saliency predictions. Recently, based on the observation that CNNs pre-trained on large-scale data usually produce high activations on some primary objects, A2S [Zhou et al., 2023a] have developed a method to distill saliency from the activation maps of deep networks and generate high-quality pseudo-labels. However, we found that during the initial training phase, the presence of hard samples in complex scenes or along object boundaries results in the accumulation of irreparable errors.

∗Corresponding authors

Unsupervised SOD is generally considered to exhibit strong generalization and transferability due to its annotationfree nature. However, prevailing USOD methodologies predominantly focus on the Natural Still Image (NSl) domain, exemplified by RGB, RGB-D, and RGB-T. Consequently, USOD on non-NSI domain, encompassing video SOD and Remote Sensing Image (RSI) SOD, remains largely unexplored, presenting a notable research gap in the field. We believe that different SOD tasks share common knowledge, and exploiting this shared knowledge can benefit transfer performance. On the other hand, compared to NSI SOD, the available datasets for video SOD or RSI SOD are relatively small and burdensome to obtain. As a result, training models from scratch on these tasks to obtain satisfactory performance is currently deemed impractical. Therefore, we advocate for the investigation of a more universally applicable unsupervised saliency knowledge transfer method.

To address the aforementioned issues, we design a unified framework for generic unsupervised SOD tasks. Firstly, we propose the Progressive Curriculum Learning-based Saliency Distilling (PCL-SD) mechanism to guide the extraction of saliency cues. At the early stages of training, we only extract preliminary saliency cues from easy samples. As the training progresses, we progressively incorporate hard samples to mine deeper saliency knowledge. The employment of PCL-SD effectively mitigates the initial accumulation of errors, leading to a more stable and robust training process. Next, we utilize the obtained saliency cues to train a saliency detector and design a Self-rectify Pseudo-label Refinement (SPR) mechanism to improve the quality of pseudo-labels. On one hand, the proposed SPR employs the saliency predictions of the model during training to rectify incorrect predictions within the pseudo-labels. On the other hand, it incorporates the prior knowledge of the input image to prevent the model from becoming complacent. The SPR mechanism demonstrates a strong capability in self-supervised learning, resulting in improved pseudo-label quality. Finally, we devise an adapter-tuning method to transfer the acquired saliency knowledge to non-NSI SOD tasks, such as video SOD and RSI SOD. Specifically, we selectively fine-tune the deep features, ensuring effective adaptation of the model to the target task while mitigating the risk of model degradation.

Our main contributions can be summarized as follows:

• We propose the Progressive Curriculum Learning-based Saliency Distilling (PCL-SD) mechanism to extract saliency cues from easy samples to hard ones.
• We design the Self-rectify Pseudo-label Refinement (SPR) mechanism to gradually improve the quality of pseudo-labels during the training process.
• We devise an adapter-tuning method to transfer saliency knowledge from NSI SOD to non-NSI SOD tasks, achieving impressive transfer performance.

Note that we are the first to consider knowledge transfer from NSI domain to non-NSI domain, and develop a unified framework for generic USOD tasks. Experiments on RGB, RGBD, RGB-T, video SOD and RSI SOD benchmarks confirm the state-of-the-art USOD performance of our method.

2 RelatedWorks

2.1 Unsupervised Salient Object Detection

Traditional SOD methods rely on hand-crafted features to extract saliency cues. For instance, [Perazzi et al., 2012] estimates saliency by evaluating the contrast in uniqueness and spatial distribution within the image. [Jiang et al., 2011] employ a combination of bottom-up salient stimuli and objectlevel shape prior to segment salient objects. Although these approaches perform well in simple scenes, they face challenges in handling complex scenes due to the lack of highlevel semantic information.

Existing deep learning-based methods for USOD typically involve two stages. In the first stage, pseudo-labels are obtained, while in the second stage, a network is trained using these pseudo-labels. For instance, [Zhang et al., 2017] fuses multiple noisy saliency cues to generate supervisory signals for training the deep salient object detector. In [Nguyen et al., 2019], a set of refinement networks are initially trained to enhance the quality of these saliency cues, and the refined pseudo-labels are subsequently utilized to train a saliency detector. A more recent approach, A2S [Zhou et al., 2023a], proposes a method to distill saliency from the activation maps of deep networks, achieving high-quality pseudo-labels.

2.2 Knowledge Transfer in SOD

Knowledge transfer involves applying models or features trained in one task or domain to another related task or domain. A typical example is fine-tuning a deep network that was pre-trained on large-scale data for a specific target task. However, the exploration of knowledge transfer across different SOD tasks remains insufficient. Among the limited studies, [Fu et al., 2022] addresses the RGB-D SOD task as a few-shot learning problem and enhances performance by incorporating knowledge from RGB SOD. [Zhou et al., 2023b] employs data from multiple SOD tasks to train a generalized saliency detector. Nevertheless, when extending to generic SOD tasks, the inherent gap between various SOD tasks can impede effective model training. Consequently, it becomes crucial to devise a knowledge transfer approach that is rooted in shared knowledge.

3 ProposedMethod

Figure 2 illustrates the proposed two-stage framework. In stage 1, we train a saliency cue extractor (SCE) to transfer saliency knowledge from a pre-trained deep network. The proposed Progressive Curriculum Learning-based Saliency Distilling is employed to mitigate the initial accumulation of errors in training and ensure the stability and robustness of the training process. In stage 2, we utilize the obtained saliency cues as initial pseudo-labels to train a saliency detector (SD). CRF [Kr¨ahenb¨uhl and Koltun, 2011] is adopted to enhance the initial pseudo-labels, and we employ the proposed Self-rectify Pseudo-label Refinement mechanism to improve pseudo-labels quality during the training process gradually.

Initially, we train our base model on Natural Still Image (NSI) SOD and subsequently transfer the model to non-NSI SOD tasks. Throughout the training of the base model, we combine all the NSI data for training. However, during the transfer process, we only employ task-specific data for training. For example, when migrating to video SOD, we solely utilize video frames and optical flow as input. The transfer process also follows a two-stage training approach, while we applied the proposed fine-tuning method to optimize the SCE instead of training it from scratch. Besides, ResNet-50 [He et al., 2016] pre-trained by MoCo-v2 [Chen et al., 2020], A2S [Zhou et al., 2023a] and MIDD [Tu et al., 2021] are employed as the pre-trained deep network, SCE, and SD, respectively. A more detailed description and explanation can be found in supplementary materials.

3.1 Progressive CurriculumLearning-based Saliency Distilling

The problem of obtaining saliency cues or extracting salient information from scratch has always been a challenge for unsupervised salient object detection methods. Earlier deep learning-based methods [Zhang et al., 2017] relied on noisy saliency cues generated by traditional SOD methods, while approaches like A2S [Zhou et al., 2023a] employ the activation maps produced by a pre-trained network as saliency cues. This method effectively extracts the saliency information embedded in the pre-trained network. However, at the early stages of training, hard samples in complex scenes may corrupt the fragile saliency patterns in the network, leading to irreparable accumulation errors and the risk of pattern collapse. To address this issue, we introduce the concept of curriculum learning into saliency distilling and propose Progressive Curriculum Learning-based Saliency Distilling (PCL-SD). As can be seen in Figure 3, the proposed PCL-SD rigidly excludes hard samples at the early stages of training and gradually incorporates them as training progresses. As a result, the model progressively extracts saliency knowledge from easy to hard samples, and the entire training process becomes more robust and stable.

The process of saliency distilling can be formulated as:

Here, N represents the number of pixels, and S(i) denotes pixel i in the saliency prediction S output by the saliency cue extractor (SCE). To be intuitively described, Lsal pulls the predicted values of each pixel towards either 0 or 1. However, during the early stages of training, Lsal may pull hard samples with values close to 0.5 in the wrong direction, which we refer to as the problem of error accumulation. The proposed PCL-SD strategy focuses on two essential aspects: (1) how to define hard samples, and (2) how to gradually incorporate them. Firstly, the determination of a pixel in the saliency prediction S as a hard sample is based on its prediction value. Specifically, a pixel S(i) is classified as a hard sample if

Here, p is the threshold for dividing hard samples, with a larger p indicating more hard samples are divided. Secondly, the value of p is initially set as 0.2 and progressively decreased during the training process until all samples are included. This decrease is governed by the formula:

where Ec and Et denote current epoch and total epoch, respectively. Finally, we define PCL-SD as:

where ⊙ denotes the Hadamard product for matrices.

3.2 Self-rectify Pseudo-label Refinement

Obtaining high-quality pseudo-labels is crucial for training a saliency detector (SD). As shown in Figure 4, the saliency prediction S output by SD can partially rectify errors within the pseudo-labels. We define saliency prediction as posterior rectification: Rpost = S. However, while this posterior rectification can rectify errors in initial pseudo-labels, it also introduces the risk of the model becoming overly confident and stagnant. To overcome this, we introduce prior information from the input image to optimize saliency prediction, in order to avoid the model falling into a self-complacent trap.

Previous approaches primarily rely on CRF for prior rectification, which entails significant computational costs. Inspired by [Ru et al., 2022], we employ a real-time pixel refiner to provide efficient prior rectification based on the input image. To start, let I and P represent the input image and position information, while σf and σp denote the standard deviation of feature values and position differences, respectively.

The parameters ω1 and ω2 control the smoothness. We define the feature distance di,j f and position distance di,j p between pixels as follows:

Then, the refiner R(·) is then defined as:

Here, N(·) represents the set of neighboring pixels in an 8way manner. Finally, the prior rectification can be defined as:

where ⊙ denotes the Hadamard product for matrices. At last, the refined pseudo-label is defined as:

Here, Gref refers to the pseudo-labels after refinement, and Gpre refers to the previous pseudo-labels. The introduction of Gref aims to improve the stability of the refinement process. λ1, λ2, λ3 are empirically assigned as 0.2, 0.6, and 0.2, respectively. As shown in Figure 4, prior rectification has effectively compensated for the considerable loss of local details. The proposed SPR mechanism combines prior and posterior rectification, gradually improving the quality of pseudo labels during the training process, demonstrating strong selfsupervised performance.

3.3 Knowledge Transfer to Non-NSI SOD Tasks

We investigate the transferability of the proposed method to video SOD and Remote Sensing Image (RSI) SOD. Figure 5 illustrates the varying degrees of relevance between different SOD tasks. The tasks within NSI SOD benefit from a greater amount of shared knowledge, allowing for the joint training of multiple tasks to achieve a better generalization performance. However, as we broaden our focus to generic SOD

tasks, the inherent gap between tasks becomes the primary influencing factor. Joint training becomes more challenging and poses risks of model degradation. More discussions on this topic can be found in supplementary materials.

We posit that identifying an appropriate fine-tuning method can effectively address the issue of model degradation. Inspired by recent studies on Adapter-tuning [Houlsby et al., 2019], we design a simple but effective fine-tuning method for knowledge transfer from NSI SOD to non-NSI SOD tasks. Specifically, for end-to-end tasks in SOD, the prevailing methods and models employ the U-net [Ronneberger et al., 2015] structure and utilize multi-scale feature aggregation to achieve accurate saliency predictions. We contend that shallow features primarily contribute to local details and possess a degree of cross-task generality, while deep features play a pivotal role in salient object localization and exhibit task-specific characteristics. Hence, we suppose that finetuning solely the network layers or modules responsible for deep feature handling allows the model to adapt to the target task while circumventing degradation. Technically, we define the deep feature handling process in the model as

Here, F represents the deep features extracted by the backbone, ˆ F denotes the processed features, and T signifies the network layer or module performing the processing. In specific end-to-end SOD models, T can comprise a convolutional layer that modifies the number of feature channels or a network module that enhances the features. Our adapter-tuning approach can be defined as:

In this equation, Ta refers to the adapter, which possesses a structure consistent with T. Following the processing, Ta is connected to the original network through a residual connection. During fine-tuning, we exclusively optimize the weights of Ta while keeping the remaining weights of the model frozen. The detailed description can be found in supplementary materials. It is worth mentioning that this fine-tuning method is universal for any kind of SOD method or task.

3.4 Supervision Strategy

We initially train the saliency cue extractor (SCE) in the first stage, followed by training the saliency detector (SD) in the second stage. In the training of the first stage, we also incorporate Boundary-aware Texture Matching (BTM) [Zhou et al., 2023b] to introduce extra structural cues, and is defined as Lbtm. Moreover, a structural consistency loss is employed to achieve transformation-invariant predictions, and is formulated as:

Here, ˆ S denotes saliency prediction after transformation. To ensure training stability, only random scaling is adopted. The total loss for training SCE can be defined as:

γ is empirically assigned as 0.05.

We train the saliency detector (SD) with IoU loss, which is defined as:

G refers to the pseudo-labels, and the total loss for training SD can be defined as:Lsd = LIoU + Lsc. (14)

4 Experiments

4.1 Implementation Details

Training Settings

The batch size is set to 8 and input images are resized to 320×320. Horizontal flipping is employed as our data augmentation. We train the saliency cue extractor for 20 epochs using the SGD optimizer with an initial learning rate of 0.1, which is decayed linearly. We train the saliency detector for 10 epochs using the SGD optimizer with a learning rate of 0.005. All experiments were implemented on a single RTX 3090 GPU.

Datasets

We follow the prevalent settings of SOD and relevant tasks. Here are some details about the datasets we used. (1) RGB SOD: We use the training subsets of DUTS [Wang et al., 2017] to train our method. ECSSD [Yan et al., 2013], PASCALS [Li et al., 2014], HKU-IS [Li and Yu, 2015], DUTSTE [Wang et al., 2017] and DUT-O [Yang et al., 2013] are employed for evaluation. (2) RGB-D SOD: We choose 2185 samples from the training subsets of NLPR [Peng et al., 2014] and NJUD [Ju et al., 2014] as the training set. RGBD135 [Cheng et al., 2014], SIP [Fan et al., 2020] and the testing subsets of NJUD and NLPR are employed for evaluation. (3) RGB-T SOD: 2500 images in VT5000 [Tu et al., 2022a] are for training, while VT1000 [Tu et al., 2019], VT821 [Wang et al., 2018] and the rest 2500 images in VT5000 are for testing. (4) Video SOD: We choose the training splits of DAVIS [Perazzi et al., 2016] and DAVSOD [Fan et al., 2019] to train our method. SegV2 [Li et al., 2013], FBMS [Brox and Malik, 2010] and the testing splits of DAVIS and DAVSOD are employed for evaluation. (5) Remote Sensing Image SOD: We choose the training splits of ORSSD [Li et al., 2019] and EORSSD [Zhang et al., 2020b] to train our method. The testing splits of ORSSD and EORSSD are employed for evaluation.

Metrics

We employ three metrics to evaluate our model and the existing state-of-the-art methods, including Mean Absolute Error M, average F-measure (Fβ) [Achanta et al., 2009] and E-measure (Eξ) [Fan et al., 2018]. Specifically, (M) measures the average pixel-wise difference between the prediction P and the ground truth G, and is calculated as M = 1N ∑Ni=1 |P(i) -G(i)|. Fβ considers both precision and recall values of the prediction map, and can be computed as Fβ = (1+β2 )×Precision×Recall β2×Precision+Recall , with β2 set to 0.3. Eξ takes into account the local pixel values along with the image-level mean value, and is defined as Eξ = 1N ∑Ni=1 ϕξ(i, j), where ϕξ represents the enhanced alignment matrix.

4.2 Comparisons With State-of-the-Art

We report the performance of our method on five representative SOD tasks, and more qualitative results are provided in supplementary materials.

Results on RGB SOD

Table 1 presents a quantitative comparison between the proposed method and recent fully-supervised, weaklysupervised, and unsupervised methods. The fully-supervised methods include MINet [Pang et al., 2020] and VST [Liu et al., 2021], the weakly-supervised methods contain WSSA [Zhang et al., 2020a] and MFNet [Piao et al., 2021], and the unsupervised methods comprise SBF [Zhang et al., 2017], TSD [Zhou et al., 2023b] and STC [Song et al., 2023]. Our results are presented under different settings: (1) Training our method using task-specified data, denoted as "Ourst.s.", for a fair comparison; (2) Training our method using NSI data, including RGB, RGB-D, and RGB-T datasets, referred to as "Ours".

The results presented in Table 1 clearly indicate that the proposed method outperforms existing USOD methods, leading to significant improvements in performance. Additionally, our unsupervised approach demonstrates competitive performance in comparison to recent weakly-supervised and fully-supervised methods. Notably, our method, referred to as "Ours", exhibits a slight superiority over "Ourst.s.". We suppose that this advantage stems from the utilization of a more extensive training dataset, which enhances the model's generalization ability and leads to improved performance when applied to unseen images.

A qualitative comparison is presented in Figure 6. As can be seen, our method has achieved more accurate and complete saliency prediction. Moreover, our approach exhibits remarkable performance in dealing with multiple objects (row 2).

Results on RGB-D and RGB-T SOD

Table 2 and 3 present a comparison between the proposed method and recent methods on RGB-D and RGB-T benchmarks, respectively. For a fair comparison, we also train our method using task-specified data, denoted as "Ourst.s.". VST [Liu et al., 2021], CCFE[Liao et al., 2022], DSU [Ji et al., 2022], TSD, MIDD [Tu et al., 2021] and SRS [Liu et al., 2023] are employed for comparison. Our method has achieved state-of-the-art performance on both RGB-D and RGB-T SOD. Moreover, our proposed approach, referred to as "Ours", exhibits a substantial performance improvement compared to "Ourst.s.". We attribute this improvement to the limited size of the training datasets for these specific tasks. In contrast, "Ours" was trained on a diverse range of datasets encompassing RGB, RGB-D, and RGB-T SOD, effectively utilizing shared common knowledge across different SOD tasks.

Results on Video SOD and RSI SOD

Table 4, 5 present a comparison between the proposed method and recent methods on video SOD and RSI SOD benchmarks, respectively. STVS [Chen et al., 2021], WSVSOD [Zhao et al., 2021], LVNet [Li et al., 2019], MJRB [Tu et al., 2022b] and TSD are employed for comparison. We consider video SOD and RSI SOD as two types of target transfer tasks. Thus, in the table, "Ours" represents zero-shot transfer results, while "Oursf " refers to the outcomes obtained by finetuning the transferred model on the target task using our proposed knowledge transfer approach. Note that the transfer for video SOD and RSI SOD is conducted separately. Our model exhibits excellent adaptability to the target task following fine-tuning, and exhibits remarkable performance.

4.3 Ablation Study

Evaluation on Pseudo-label Quality

We assessed the quality of the pseudo-labels generated by models trained on different datasets. As previously mentioned, "Ourst.s." denotes the model trained using taskspecific data, whereas "Oursf " refers to the model transferred to the target task. The results are presented in Table 6. In comparison to "Ourst.s.", "Ours" exhibits slightly inferior performance on the RGB training set, but displays a notable improvement on the RGB-D and RGB-T training sets, which possess a comparatively limited amount of training data. This indicates that a larger training dataset yields superior model performance and enhanced generalization ability. Furthermore, "Oursf" shows a slight improvement in video SOD, whereas it exhibits a substantial enhancement in RSI SOD. This indicates that video SOD and NSI SOD share more common knowledge, while RSI SOD requires greater fine-tuning and adaptation. More analysis on the adaptation to target tasks is presented in supplementary materials.

Evaluation on SPR

We evaluated the influence of various rectifications on the pseudo-labels, and the results are presented in Table 7. The posterior rectification Rpost effectively corrects the erroneous predictions present in pseudo-labels, while the prior rectification Rpri adequately compensates for the lack of local details in pseudo-labels. As a result, the proposed SPR gradually enhances the quality of pseudo-labels, thereby improving the model's performance.

Evaluation on Supervision Strategy

We evaluated the effectiveness of the proposed supervision strategy, as shown in Table 8. We treat all samples as easy samples to examine the effectiveness of PCL-SD. Upon applying PCL-SD, the model exhibits a slight improvement on the training set. Nonetheless, an impressive enhancement in performance can be observed on the test set. This improvement substantiates the model's heightened generalization capability. Additionally, we explored the training of the saliency detector using different loss functions. The results indicate that the commonly employed binary cross-entropy (bce) in supervised SOD did not lead to effective performance enhancement. We hypothesize that this ineffectiveness may be attributed to the errors and interference stemming from incorrect predictions in pseudo-labels. In contrast, the selfsupervised loss Lms delivered a noteworthy improvement.

5 Conclusion

In this paper, we propose a two-stage unified unsupervised SOD framework for generic SOD tasks, with knowledge transfer as the foundation. Specifically, we introduce two innovative mechanisms: Progressive Curriculum Learningbased Saliency Distilling (PCL-SD) and Self-rectify Pseudolabel Refinement (SPR), which aim to extract saliency cues and optimize pseudo-labels. Additionally, we present a universal fine-tuning method to transfer the acquired saliency knowledge to generic SOD tasks. Extensive experiments on five representative SOD tasks validate the effectiveness and feasibility of our proposed method.

Acknowledgements

This work was partially supported by the National Natural Science Foundation of China (No. 62272227 & No. 62276129), and the Natural Science Foundation of Jiangsu Province (No. BK20220890).