Projection, Interaction and Fusion: A Progressive Difference Fusion Network for Salient Object Detection

1 Introduction

Salient Object Detection (SOD) [Jiang et al., 2013; Zhou et al., 2024], also known as saliency detection, aims to mimic the mechanism of the human visual system by identifying and highlighting the most prominent or important objects and regions in natural images. In recent years, deep learningbased salient object detection methods have made significant progress, but the performance of these methods has gradually reached a saturation point. To delve deeper into this issue, we adopted the CIEDE2000 [Luo et al., 2001] color difference formula to classify five commonly used datasets [Wang et al., 2017; Yang et al., 2013; Yan et al., 2013; Li and Yu, 2015;

∗Corresponding author

Li et al., 2014] in the SOD field and selected the top 30% of the most challenging samples to form a subset named TOP30. TOP-30 can be considered as encapsulating the most challenging cases in the SOD field. Building upon this, we introduced two metrics, Location IoU (IoUL) and Boundary IoU (IoUB) [Cheng et al., 2021], to measure a model's localization accuracy and segmentation quality. As shown in Figure 1(a), starting in 2022, the localization and segmentation capabilities of methods in challenging and complex scenarios have started to plateau. As shown in Figure 1(b), existing SOTA methods perform poorly in complex scenarios, which can be attributed to three main reasons: (i) the scale variability of salient objects (row 1); (ii) the shape variability of salient objects (row 2); and (iii) the confusion between global and detail information during the multi-scale feature fusion stage (row 3).

To address the aforementioned challenges, we propose a novel architecture-Progressive Difference Fusion Network (PDFNet). First, to overcome the scale variability of salient objects, we design the Self-Guided Module (SGM) with dynamic receptive fields using large kernel convolutions. This allows the model to dynamically adjust the receptive field during the multi-scale feature extraction phase, enhancing the model's scale adaptability. Second, to address the shape variability of salient objects, we use cross convolutions to design the Feature Aggregation Module (FAM) with a feedback loop. This module can preliminarily aggregate features from the deepest three layers of the encoder to generate initial global features, thereby improving the model's shape adaptability. Third, to alleviate the confusion between global and detail information during multi-scale feature fusion, we propose a Fine-Grained Feature Fusion Stage (F3 Stage) composed of Progressive Difference Fusion Units (PDFU). This stage models multi-scale features as fine-grained nodes and enhances the feature representations of nodes at different scales through the interaction of difference features between the nodes.

In summary, our main contributions can be summarized as follows:

1. We propose a novel SOD architecture-Progressive Difference Fusion Network (PDFNet). Our method achieves state-of-the-art performance on five commonly used datasets.
2. We designed the Self-Guided Module (SGM) and Feature
3. aggregation module (FAM), which enhance feature representation and improve the model's adaptability to the size and shape variations of salient objects.
4. We propose a Progressive Difference Fusion Unit (PDFU), which projects multi-scale features into finegrained nodes to facilitate fine-grained fusion across multi-scale features.

Additionally, we optimize the traditional post-processing technique, the fully connected CRF [Lafferty et al., 2001], and propose the Conditional Random Field based on Patch (CRFbp) that focuses on handling discrete points, further improving the accuracy of predictions.

2 RelatedWorks

2.1 Vision Transformer

When the Transformer [Vaswani, 2017] was first introduced, it was primarily used for natural language processing tasks. ViT [Dosovitskiy, 2020] was the first to apply the pure Transformer architecture to computer vision, achieving remarkable results in image classification, semantic segmentation, and other tasks. VST [Liu et al., 2021a] proposed the first pure Transformer-based saliency object detection model, proving the effectiveness and potential of Transformer-based models for saliency detection. Swin Transformer [Liu et al., 2021c], a variant of Vision Transformer, innovatively introduced the sliding window mechanism, which efficiently extracts local features while effectively capturing global context information.

Owing to their superior global context modeling capability compared to CNNs, Transformer-based methods often outperform CNN-based methods in SOD.

2.2 Multi-Scale Feature Fusion

The method of integrating multi-scale features has made significant progress in SOD. For example, U-Net [Ronneberger et al., 2015] proposed a U-shaped encoder-decoder architecture, which became the foundational framework for many subsequent SOD works. VST [Liu et al., 2021a] was the first to use a Transformer for saliency object detection, constructing global features from a patch set to obtain a global perspective that guides feature fusion. PAKRN [Xu et al., 2021] proposed the first dual-stream architecture of "localization first, then segmentation," where the localization branch first obtains global context information and then progressively refines the prediction results through feature fusion. EDN [Wu et al., 2022] deepens the encoder layers further to obtain global context information, which is used as an attention map for channel attention to guide feature fusion. GPONet [Yi et al., 2024] uses a gated recurrent network to filter redundant information during feature fusion.

Despite significant progress in existing methods for SOD, confusion between global and detail information remains a key challenge, particularly in complex scenes.

3 Method

3.1 Overall Architecture

As shown in Figure 2, for a given input image I ∈ R3×H×W , we represent its multi-level outputs generated by the backbone encoder as a set FEn = {F (i) En |i ∈ {0, 1, 2, 3, 4}}. Then, we enhance the feature set FEn through the SGM, reducing the number of channels to 64 to obtain the enhanced feature set FSG = {F (i) SG|i ∈ {0, 1, 2, 3, 4}}. Following this, the FAM aggregates F (2) SG, F (3) SG and F (4) SG to generate the initial global feature map. Next, the F3 Stage, composed of multiple PDFUs, performs fine-grained fusion of multi-scale features from adjacent levels. Finally, the predictions are refined using CRFbp to enhance the overall results.

3.2 Self-GuidedModule

Previous research [Luo et al., 2016] has shown that increasing the size of the convolution kernel can enlarge the effective receptive field (ERF). However, larger convolution kernels increase computational costs and hinder the deepening of the model. Therefore, we combine large kernel dilated convolutions with depthwise separable mechanisms [Chollet, 2017] and introduce Scale-Aware Attention (SAA) to further enhance the network's scale adaptability.

As shown in Figure 3(a), suppose the input of SSA is X ∈ RC×H×W . We first use two depthwise separable convolutions with different receptive fields to expand the global receptive field of the input features, resulting in receptive field enhanced features X(1) ds and X(2) ds :

Where X(i) ds ∈ R C2 ×H×W ; and DwC(k,d)(·) denotes a depthwise convolution layer with a kernel size of k × k and a dilation rate of d; and FC(·) represents a fully connected layer, which is technically implemented by a 1 × 1 convolution layer.

Next, channel-wise average pooling and max pooling are applied to obtain Aavg and Amax, which are then used to weight and sum with X(1) ds and X(2) ds to produce the output of SSA:

Where ASSA ∈ RC×H×W , which is consistent with the input feature X; Cat(·) denotes the concatenation operation, ⊕ represents the element-wise addition operation, ⊙ indicates the element-wise multiplication operation, and σ(·) represents the sigmoid activation function.

Assuming the input feature of the encoder is FEn ∈ RC×H×W , the process of SGM can be described as follows:

Where FSG ∈ RC′×H×W , the SGM compresses the feature channels to C ′ in the final FC layer and C ′ is set to 64 in this paper; CBRk×k(·) denotes a sequence of operations consisting of a convolution layer, batch normalization, and ReLU activation; and k represents the size of the convolution kernel as k × k.

3.3 Feature AggregationModule

The structure of the Feature Aggregation Module (FAM) is shown in Figure 3(b). Since the first two stages of the encoder (Stage 0 and 1) contain excessive low-level detail information, we select the features F (2) SG, F (3) SG and F (4) SG from the three deepest stages of the encoder for preliminary aggregation:

Where Ffuse ∈ R64×H×W ; Ui(·) represents the dynamic upsampling operation [Liu et al., 2023], with i denoting the upsampling factor.

Previous works have demonstrated [Ding et al., 2019; Guo et al., 2022] that a cross-shaped asymmetric convolution can serve as a complement to square convolutions, helping capture elongated objects, enhancing the model's shape adaptability. Based on this, we design the Cross-Feedback Attention (CFA). The process can be represented as:

Where DwCk×k(·) represents the depthwise convolution operation, and k denotes the size of the convolution kernel.

CFA captures multi-scale contextual information through multi-path asymmetric depthwise convolutions and aggregates local information from other branches via feedback connections.

Finally, multi-scale dynamic weights are used to enhance the fused features:

Where FFA ∈ R64×H×W and represents the initial aggregated features.

3.4 Progressive Difference Fusion Unit

Fine-grained Node Projection

Typically, researchers assume that the deepest features from the encoder represent global features [Zhao et al., 2017; Zhao et al., 2021; Wu et al., 2022]. However, high-level features may contain some detail information, and low-level features may also carry some global information. As shown in Figure 4(a), to extract feature information at a finer granularity, we design the Fine-grained Node Projection (FGNP), which projects the features into detail nodes and global nodes based on different scales. In general, for an input feature X ∈ RC×H×W , the FGNP process can be expressed as:

Where i denotes the layer of the input feature from the encoder; r is the channel reduction factor used to decrease the computational load during node interactions, which is set to 2 in this paper; ABR(k,d)(·) represents an operation sequence consisting of an atrous convolution with kernel size k × k, dilation rate d, followed by BatchNorm and ReLU layers.

Fine-grained Node Interaction

For convenience in discussion, we define two different nodes to be interacted as x and y. Then, the difference feature from x to y, denoted as f x→y d , is defined as:

As shown in Figure 4(b) and (c), we define the interaction between nodes of the same level but different scales as internal interaction I(·, ·), while the interaction between nodes of different levels but the same scale is defined as external interaction E(·, ·). This process can be expressed as:

Where k denotes the number of iterations; CG(·) represents convGRU [Ballas et al., 2015]; and γ represents the learnable parameter, and its initial value is set to 0 in this paper.

Let the low-level feature be X(i) ∈ RC×H×W and the high-level feature be X(i+1) ∈ RC×H×W . The initial finegrained node set generated from X(i) is denoted as V (i) = {d(i) (0,0) , g(i) (0,0) }. Here, v(i) (t1,t2) represents the fine-grained node obtained by projecting the feature X(i) , which has undergone t1 rounds of internal interactions and t2 rounds of external interactions. For the fine-grained node set V (i) , one round of external interaction can be represented as:

One round of internal interaction can be represented as:

Where K is the total number of iterations, set to 2 in this paper. After K interactions, all nodes in V (i) and V (i+1) undergo fine-grained fusion to obtain the output of the i-th layer PDFU:

Where F (i) PDFU ∈ RC×H×W , which is consistent with the input features.

Fine-grained Feature Fusion Stage

We designed the Fine-Grained Feature Fusion Stage (F3 Stage) based on PDFUs to progressively perform fine-grained fusion of multi-scale features layer by layer and output intermediate prediction results for multi-scale supervision:

Where P (i) M represents the intermediate prediction results, with i indicating the decoder's layer. The model's generated prediction results can be expressed as:

The visualization results of the fine-grained nodes are shown in Figure 5. It can be observed that for the same level of PDFU, global nodes mainly process global localization information, while the weights of detail nodes are concentrated around the edges of the salient objects, processing detailed segmentation information. Specifically, as interactions with other nodes progress, the model enhances its learning of edge information, which is crucial for segmentation in SOD.

3.5 Supervision Strategy

For training, we adopted the widely used multi-level supervision strategy in this field. For the loss function, we use the combination of Lwbce and Lwiou proposed by F3Net [Wei et al., 2020]. The saliency loss and total loss can then be defined

Where GT ∈ RH×W represents the ground truth; Di(·) denotes the downsampling operation implemented by bilinear interpolation, with i indicating the downsampling factor; β is a hyperparameter and in this work, and it is set to 1.

4 Experiment and Analysis

4.1 Datasets

Following the setup of most existing studies, our model will be trained on the DUTS-TR [Wang et al., 2017] dataset and evaluated on five widely recognized benchmark datasets: DUT-OMRON [Yang et al., 2013], DUTS-TE [Wang et al., 2017], ECSSD [Yan et al., 2013], HKU-IS [Li and Yu, 2015], and PASCAL-S [Li et al., 2014].

4.2 Implementation Details

Our PDFNet employs Swin-B as the backbone network. Training and testing were conducted on an NVIDIA 2080 Ti, with input images resized to 384x384. During model training, the Adam optimizer is employed with an initial learning rate set to 1.0 × 10-5 . The learning rate adjustment follows a polynomial decay (PolyLr) strategy, with the entire training process spanning 80 epochs. The batch size is set to 4 to balance memory consumption and training efficiency.

4.3 EvaluationMetric

We will evaluate all methods using the following widely used evaluation metrics in the SOD field: Mean Absolute Error

(M) [Perazzi et al., 2012], S-measure (Sα) [Fan et al., 2017], Weighted F-measure (F ω β ) [Margolin et al., 2014], and Average E-measure (Ea ξ ) [Fan et al., 2018].

4.4 Comparisons with State-of-the-Art

This paper compares the proposed method with the top 16 state-of-the-art models in recent years, including: F3Net [Wei et al., 2020], TSPOANet [Liu et al., 2021b], VST [Liu et al., 2021a], PAKRN [Xu et al., 2021], PoolNet+ [Liu et al., 2022], EDN [Wu et al., 2022], ICON-S [Zhuge et al., 2022], PGNet [Xie et al., 2022], SelfReformer [Yun and Lin, 2023], BBRF [Ma et al., 2023], MENet [Wang et al., 2023], MGuidNet [Hui et al., 2023], DASOD [Asheghi et al., 2024], GLSTR [Ren et al., 2024], ISNet [Zhu et al., 2024], and GPONet [Yi et al., 2024]. Specifically, some models have two implementations, one based on CNNs and the other on Transformers. We have chosen the Transformer-based implementations for comparison.

Quantitative Comparison

As shown in Table 1, our PDFNet achieves the best performance across five classic benchmark datasets, outperforming other state-of-the-art methods. Furthermore, Figure 6 presents the PR curves and F-measure curves for the aforementioned networks. From the figure, it is clearly observed that our method (denoted by the red curve) outperforms other methods.

Visual Comparison

The visual comparison results are shown in Figure 7. Compared to other methods, our predictions perform excellently in complex scenes: for scenes with salient objects of inconsistent scales (row 1), our method achieves better localization. For elongated and irregularly shaped objects (row 2), our method generates more refined boundaries. In scenes where salient objects are difficult to identify (row 3), our method is able to accurately locate and segment.

4.5 Ablation Study

Effectiveness of SGM

As shown in Table 2, to validate the effectiveness of SGM, we removed it frommodel (4) to obtain model (1). It can be observed that on five datasets, the performance metrics have all decreased. Specifically, Ea ξ decreased by 1.1%, 0.9%, 0.6%, 0.7%, and 0.7%, respectively. This indicates that SGM significantly enhances the model's scale adaptability.

Effectiveness of FAM

To validate the effectiveness of FAM, we removed FAM from model (4) to obtain model (2). It can be observed that on five datasets, the performance metrics have all decreased. Specifically, Ea ξ decreased by 0.8%, 0.8%, 0.5%, 0.7%, and 0.6%, respectively. This indicates that FAM enhances the model's shape adaptability.

Effectiveness of PDFU

To further validate the effectiveness of PDFU, we removed PDFU from model (4) to obtain model (3). It can be observed that the performance metrics have all decreased on five datasets. Specifically, F ω β decreased by 1.7%, 1.4%, 1.3%, 1.1%, and 1.1%, respectively. This demonstrates that PDFU effectively alleviates the issue of confusion between global and detail information.

Effectiveness of CRFbp

From (4) and (5) in Table 2, we can observe that after adding the CRFbp post-processing, the results of model (5) generally outperform those of model (4). With the inclusion of CRFbp, the F ω β of (5) increased by 0.6%, 0.6%, 0.4%, 0.6% and 0.4% on five datasets compared to (4). At the same time, we notice that Sα in (5) slightly decreased compared to (4). This may be because CRFbp focuses on processing local discrete points, which may disrupt the structural integrity of the predicted map. Therefore, we recommend using CRFbp as an optional step during the model prediction phase.

5 Conclusion

In this paper, to address the performance bottleneck of SOD models in complex scenes, we propose a new SOD architecture-PDFNet. Specifically, we first design the Self-Guided Module (SGM) to enhance the model's scale adaptability. Then, we design the Feature Aggregation Module (FAM), which enhances the model's shape adaptability. Finally, we design the F3 Stage based on Progressive Difference Fusion Units (PDFU) to alleviate the issue of information confusion within features. Additionally, we optimize the fully connected CRF and propose CRFbp, which further improves the model's performance. Our proposed PDFNet achieves stateof-the-art results on five widely used datasets.

Acknowledgments

This work was supported in part by the National Key Research and Development Plan of China under Grant 2021YFB3600503, in part by the National Natural Science Foundation of China under Grant 61972097 and U21A20472, in part by the Major Scientific Research Project for Technology Promotes Police under Grant 2024YZ040001, in part by the Natural Science Foundation of Fujian Province under Grant 2021J01612 and 2020J01494, in part by the Major Science and Technology Project of Fujian Province under Grant 2021HZ022007, in part by the Fujian Collaborative Innovation Center for Big Data Application in Governments, and in part by the Fujian Engineering Research Center of Big Data Analysis and Processing.