R³Net: Recurrent Residual Refinement Network for Saliency Detection

1 Introduction

Saliency detection seeks to highlight the most visually distinctive objects in an image. Inferring salient objects benefits many applications, such as weakly-supervised object detection [Baisheng Lai, 2017] and visual tracking [Hong et al., 2015]. However, detecting salient objects requires the semantic understanding of the whole image as well as the detailed structures of the objects. Hence, saliency detection is a fundamental yet challenging problem in computer vision.

∗ Joint first authors

†Corresponding author (xuemx@scut.edu.cn)

Traditional methods employed hand-crafted visual features (e.g, color, texture, and contrast) or heuristic priors [Liu et al., 2011; Yang et al., 2013; Zhu et al., 2014; Cheng et al., 2015] to detect salient objects from the input images. These hand-crafted features and priors are incapable of capturing the high-level semantic knowledge, making the saliency prediction unsatisfactory. To improve the detection accuracy, many deep saliency networks [Li and Yu, 2015; Zhao et al., 2015; Li et al., 2016] have been proposed by leveraging the deep semantic features of fully convolutional neural networks (FCNs) [Long et al., 2015]. Although the deep features of FCN-based methods capture more high-level semantic information over hand-crafted features, their results suffer from neglecting many fine saliency details due to the coarseness of feature maps in the deep layers of FCNs [Kong et al., 2016].

More recently, several works [Liu and Han, 2016; Li and Yu, 2016; Hou et al., 2017] utilized short connections to merge multi-level features of FCNs in order to simultaneously exploit high-level semantic information and low-level detailed structures for saliency detection. Although they improve predictions by using complementary information of multi-level features, those methods conduct the prediction at one stage, making the results still unsatisfactory. Wang et al. [Wang et al., 2017] further improve the performance by leveraging a stage-wise refinement network to process the saliency prediction in many stages, which can progressively refine the intermediate saliency maps by absorbing low-level detail information. However, such a refinement procedure tends to introduce non-salient regions as it mainly relies on the low-level features to refine the saliency maps. In addition, it is insufficient to learn salient objects for refinement using a plain network at each stage, because it produces the refined saliency prediction from the scratch (random noise) without preserving previous prediction. On the other hand, the residual learning has exhibited better performance in many vision tasks, such as image classification [He et al., 2016] and face attribute manipulation [Shen and Liu, 2017]. We employ a residual network as the starting point of our refinement network.

In this paper, we propose a novel deep refinement network to more accurately detect salient objects from the input images by leveraging advantages of the residual learning and saliency information encoded in multiple layers of an FCN. To achieve this, we first design a residual refinement block (RRB), which takes the deep features concatenated with the previous predicted saliency map as inputs to learn the difference (residual) between the ground truth and the previous saliency map. Then, we embed a sequence of RRBs in an FCN to construct our recurrent residual refinement network (R3Net), which progressively refines the saliency maps at each recurrent step by alternatively leveraging the high-level semantic features and the low-level detailed features as the inputs, which can enhance the saliency details and suppress non-salient regions of intermediate saliency maps simultaneously. Finally, we take the saliency prediction at the last recurrent step as the final output of our network. The whole network is trained in an end-to-end manner.

To verify the effectiveness of the presented R3Net, we evaluate it on five famous salient object detection benchmarks, and compare our results with those of 16 state-of-the-art methods. The experiments demonstrate that our model quantitatively and qualitatively outperforms other saliency detectors. Overall, we can summary the contributions of this work as follows.

• First, we design a novel residual refinement block (RRB) to learn the residual between the ground truth and the saliency map at each recurrent step. This learning strategy can make the network easy to train and help to learn the complementary saliency information of previous prediction for the refinement.
• We propose a recurrent residual refinement network (R3Net) to progressively refine the saliency maps by building a sequence of RRBs to alternatively use the lowlevel features and high-level features. Such a general refinement strategy has potential to be used in other tasks such as semantic segmentation and object detection.
• Third, we evaluate our network on five famous benchmark datasets and compare it with 16 state-of-the-art saliency detection methods. Overall, our method consistently has the best performance on all the five datasets.

2 Methodology

We illustrate the architecture of our recurrent residual refinement network (R3Net) equipped with residual refinement blocks (RRBs) in Figure 1. It begins by utilizing a feature extraction network to produce a set of feature maps, which contain low-level details and high-level semantic information with different scales. Then, the feature maps at shallow layers are integrated to generate the low-level integrated features (denoted as L), and the feature maps at deep layers are merged together to form the high-level integrated features (denoted as H). After that, we generate an initial saliency map from H, and then develop a set of residual refinement blocks (RRBs) to progressively refine the intermediate saliency maps by harnessing L and H alternatively.

Meanwhile, at each recurrent step, we impose the supervision signal [Xie and Tu, 2015] to compute the loss between the ground truth and the predicted saliency map during the training process. Finally, we take the saliency map at the last recurrent step as the final output of our network. In following subsections, we first elaborate how to build the RRBs, and then introduce the proposed R3Net for saliency detection.

2.1 Residual Refinement Block (RRB)

We develop an RRB at each recurrent step to correct prediction errors in the previous saliency map for refinements. The RRB alternatively takes the low-level integrated features (L) or the high-level integrated features (H) with the saliency map predicted at the previous step as inputs, and outputs a refined saliency map by adding the previous saliency map with a learned residual (difference between the ground truth and the previous saliency map); see orange dash boxes of Figure 1 for the first and second RRBs.

Formally, an RRB is defined as:

where we first obtain the residual (residualj) at j-th recurrent step by sending the concatenation (Cat) of the predicted saliency map Sj-1 at (j -1)-th recurrent step and the feature maps F , to the function Φj , which consists of three convolutional layers (see Figure 1). Then, the residualj is added with Sj-1 in an element-wise way to compute the output Sj of our RRB. Note that the feature map F is alternatively set as L or H in our recurrent network (see Section 2.2 for details), and we find the unshared parameters Φ of our RRBs at different recurrent steps show a superior performance than the shared parameters; see Table 1 for detailed comparisons.

Unlike [Wang et al., 2017], which directly learns the desired underlying saliency mapping for the refinement in a plain network, our RRB explicitly learns to fit a residual that reflects the difference between the ground truth and the previous saliency map, since learning residual is much easier, and usually outperforms learning from plain networks in various tasks, as suggested in [He et al., 2016; Shen and Liu, 2017; Xie et al., 2017]. Figure 2 shows the training loss of our R3Net (see Section 2.2 for details) with/without the residual learning technique in our RRB. Obviously, our residual learning can ease the optimization task with a faster convergence at early stages, and reduce the training error over directly learning underlying saliency mapping; see Table 1 in Sec. 3.3 for the quantitative comparisons of these two learning ways.

2.2 Recurrent Residual Refinement Network

To learn the salient regions in a refinement mechanism, we develop a novel deep network with a sequence of RRBs to gradually refine the saliency predictions. As illustrated in Figure 1, our network first applies ResNeXt [Xie et al., 2017] as the feature extraction network to produce a set of feature maps with different scales. The feature maps at deep layers have large scales and are able to capture the high-level semantic information of salient objects, while the feature maps at shallow layers have small scales and can extract the fine delicate structures of salient regions. Then, we produce the low-level integrated features (denoted as L) by upsampling the feature maps from the first three layers to a quarter of the size of the input image, concatenating them together, and applying a convolution operation to merge those features and reduce the feature dimensions:

where Fi is the upsampled feature maps at the i-th layer; Cat operation is to concatenate feature maps at the first three layers; fconv is known as the feature fusing network, consisting of 3 convolution layers, followed by 3 PReLU activation functions [He et al., 2015]. Similarly, we produce the highlevel integrated features (denoted as H) by using the feature maps (F4 and F5) at the last two layers:

Our network first predicts an initial saliency map (denoted as S0) from the high-level integrated features (H), which tend to capture the locations of the salient objects but neglect a lot of saliency details. Then, taking S0 as a starting point, we develop a sequence of RRBs to progressively refine the saliency predictions. Specifically, since the low-level integrated features L is capable of discovering many saliency details of input images, we build the first RRB by setting F in Eq. 1 as the L to refine the initial saliency map (S0) and obtain the saliency map (S1) with more fine details. However, the low-level integrated features (L) also contain a large number of non-saliency cues, and thus simultaneously introduce nonsalient regions into the S1. Hence, we build the second RRB by using H to replace F in Eq. 1 to remove non-salient regions introduced by L. Since the high-level features H focus on semantic cues of the salient objects, such an operation can eliminate non-saliency details that are not located in semantic salient regions. To further improve the saliency prediction, we construct a sequence of RRBs by alternatively incorporating L and H several times.

In addition, we apply deep supervision mechanism [Xie and Tu, 2015] to impose the supervision signal on the predicted saliency map at each recurrent step during the training process. By adding auxiliary supervisions connected to the intermediate steps, each RRB is capable to learn the residual from the ground truth directly, which makes the network optimization easier [Xie and Tu, 2015]. Finally, we take the saliency map at the last recurrent step to compute the final output of our R3Net network.

3 Experiments

In this section, we describe the training and testing strategies of our R3Net, introduce the benchmark datasets and evaluation metrics, and report the experimental results.

3.1 Training and Testing Strategies

Loss function. As shown in Figure 1, our network can output several saliency maps, including the initial saliency map S0, and a sequence of refined saliency maps (S1, . . . , Sn) after applying RRB n times. During the training process, we apply deep supervision mechanism [Xie and Tu, 2015] into impose a supervision signal (ground truth) for each saliency output, and thus we can compute the cross-entropy loss between each predicted saliency map and the ground truth (supervision). The total loss Θ of our network is defined as the summation of the loss on all predicted saliency maps:

where w0 and Y0 are the weight and loss in our initial saliency prediction; wi and Yi denote the weight and loss of the prediction at i-th recurrent step; n is the number of recurrent steps employed in our method. In our experiment, we empirically set all the weights (including w0 and wi) as 1, and set the hyper-parameter n as 6 by balancing the time performance and the detection accuracy (see Section 3.3 for details).

Training parameters. In order to accelerate the training process and reduce the over-fitting issue, we use the welltrained ResNeXt network on ImageNet [Xie et al., 2017] to initialize parameters of feature extraction network (see Figure 1), while other layers are randomly initialized from a Gaussian distribution. We use the stochastic gradient descent (SGD) to train the network with the momentum of 0.9 and the weight decay of 0.0005, set the basic leaning rate as 0.001, adjust the learning rate by the "poly" policy [Liu et al., 2015] with the power of 0.9, and stop the training procedure after 6k iterations. The R3Net is trained on the MSRA10K dataset [Cheng et al., 2015], which is widely used for training the saliency models [Lee et al., 2016; Zhang et al., 2017a]. Images in this dataset are randomly rotated, cropped and horizontally flipped for data augmentation. Our R3Net is trained on a single GPU with a mini-batch size of 14, and it takes only 80 minutes to train the network.

Inference. In the testing stage, for each input image, our R3Net can predict a saliency map at each recurrent step. Our final result is obtained by upsampling the prediction at the last recurrent step to the size of the input image, and then applying the fully connected conditional random field (CRF) [Kr¨ahenb¨uhl and Koltun, 2011] to enhance the spatial coherence of the saliency maps.

3.2 Datasets and EvaluationMetrics

After training the R3Net on the MSRA10K dataset [Cheng et al., 2015] (containing 10,000 images), we perform various experiments to evaluate the proposed network on five widelyused saliency benchmark datasets, including ECSSD [Yan et al., 2013] with 1, 000 images, HKU-IS [Li and Yu, 2015] with 4, 447 images, PASCAL-S [Zhang et al., 2017a] with 850 images, SOD [Hou et al., 2017] with 300 images, and DUTOMRON [Yang et al., 2013] with 5, 168 images. Please refer to the [Hou et al., 2017] for the detail descriptions of these saliency benchmark datasets.

We use two metrics to quantitatively compare our method with our rivals: F-measure (Fβ) and mean absolute error (MAE) (see [Hou et al., 2017] for their definitions). A better saliency detector shall have a larger F-measure and a smaller MAE. To do fair comparisons, we apply the implementations of [Hou et al., 2017] to compute the F-measure and MAE for all the compared methods.

3.3 Ablation Analysis

We first perform ablation experiments on the 5 benchmarks to evaluate the effectiveness of our R3Net. First, we show the results of our R3Net with different recurrent steps. Second, we perform a comparison with "R3Net w/o r", which has a similar structure with our R3Net but refines the saliency map (six times) without residual learning. Third, we modify our R3Net by using the shared parameters among all the RRBs and denote the new one as "R3Net w s" for a comparison. Then, we also compare with "R3Net-LL" using only L as the features F of Eq. 1 in all the recurrent steps, and "R3Net-HH" using only H as the features F of Eq. 1 in all the recurrent steps to verify the effectiveness of alternatively using L and H during the refinement procedure. Moreover, we compare another two models (denoted as "R3Net-D" and "R3Net-V"), which use the DenseNet [Huang et al., 2017] (161 layers) and the VGG-Net [Simonyan and Zisserman, 2015] (16 layers) respectively, instead of the our ResNeXt.

Table 1 summaries the quantitative results of these different settings. From the results, we has the following observations: (1) R3Net-0 to R3Net-7 in Table 1 are the initial prediction and our refined results from the first to seventh recurrent step. It is observed that our recurrent mechanism significantly outperforms the initial prediction (denoted as "R3Net-0"), and the saliency detection accuracy consistently increases in the first four iterations, and then becomes stable from the fifth to seventh iteration. To the end, we empirically set the total recurrent step as six by balancing the performance and time complexity. (2) Comparing our "R3Net" with "R3Net w/o r", we can find that our model with residual learning is superior to the model without residual learning, demonstrating the effectiveness of the proposed RRB. (3) The comparison between our "R3Net" and "R3Net w s" demonstrates that our method with separated parameters in different RRBs has a superior performance. (4) Our model has a better performance over "R3Net-LL", demonstrating that H can help to suppress non-saliency regions caused by L. Moreover, the superior performance of our method over the "R3Net-HH" indicates that L can complement more saliency details lacked in H. (5) Comparing "R3Net" with "R3Net-D" and "R3Net-V", we can find that our model equipped with ResNeXt has a better performance, showing that the ResNeXt [Xie et al., 2017] extracts more powerful features than VGG-Net and DenseNet.

3.4 Comparison with the State-of-the-arts

We further compare the results of our method with those of 16 state-of-the-art saliency detectors (see the first column of Table 2 for compared saliency detectors). Among them, MR [Yang et al., 2013], wCtr* [Zhu et al., 2014] and BSCA [Qin et al., 2015] use hand-crafted features to differentiate the salient objects from background while others are based on the deep learning framework to learn the features for saliency detection. In order to conduct a fair comparison, we obtain saliency detection results of compared methods by using either the saliency maps provided by the authors, or their implementations with recommended parameter settings.

Quantitative comparison. Table 2 summaries the comparison results in terms of F-measure and MAE. Our method (R3Net) consistently outperforms others on almost all the five datasets with respect to both two metrics, demonstrating the superior performance of our method on detecting salient objects. Especially, our method outperforms the previous stateof-the-arts by a significant margin on the "DUT-OMRON" dataset, where the images are with complicated salient regions. It shows that our method is more powerful to deal with the challenging images, which is further manifested in the following visual comparisons (see Figure 3). Note that our network is trained on the MSRA10K dataset, but it still has a superior performance over others (e.g. RFCN and MDF) that are directly trained on PASCAL-S or HKU-IS, demonstrating a good generalization capability of our R3Net, which is vital for the saliency detection.

For methods based on the deep neural networks, the training set and the feature extract network are important, because a good training set will provide a large number of representative training data and provide more knowledge to the deep neural network, and a strong feature extract network will obtain more powerful features for discovering the salient regions. However, the recent saliency detectors based on the deep neural networks use different kinds of training sets and feature extract networks. For a fair comparison, we retrain the models (NLDF [Luo et al., 2017], DSS [Hou et al., 2017], Amulet [Zhang et al., 2017b], and SRM [Wang et al., 2017]) by using the same basic network (ResNeXt) and the same training set (MSRA10K) as our method. Table 2 also reports the comparison results, where these retrained models are denoted as "XX+". By using the powerful basic network and the MSRA10K dataset, these models show better performances than the original ones, but our method still outperforms these new models by a significant margin.

Visual comparison. We also visually compare our method with our rivals on different input images; see Figure 3. From these results, we can observe that other methods tend to include non-saliency backgrounds (see (d)-(i) of the first two rows) or lost many saliency details (see (d)-(i) of the third and fourth rows), while our R3Net produces more accurate results of detecting salient objects, and our results are more consistent with the ground truths (see (b) and (c)). In addition, for those challenging images with small salient objects (the fifth row in Figure 3) and multiple objects (the last row in Figure 3), our method also predicts more precise saliency maps than others, which further indicates the effectiveness and robustness of the presented R3Net. The code, trained model and more results are publicly available at https://github.com/zijundeng/R3Net.

4 Conclusion

This paper presents a novel refinement network equipped with a sequence of residual refinement blocks (RRBs) for single-image saliency detection. Our key idea is leveraging RRBs to recurrently learn the difference (called "residual" in this work) between the coarse saliency map and the ground truth by alternatively harnessing the low-level features (L) and high-level features (H) of an FCN. Learning the residual counterpart in our RRB module can make the network easy to be optimized, and obtain the complementary saliency information of the intermediate results to refine the saliency prediction maps. Furthermore, our RRBs employ the low-level features and the high-level features to alternatively refine the saliency maps by compensating saliency details and suppressing non-salient regions. We test our R3Net on 5 benchmark datasets, and the experimental results show that our method consistently outperforms the other 16 state-of-the-art methods. The proposed refinement scheme is general enough to be applied to other important computer vision tasks, such as object detection and instance-aware saliency detection.

Acknowledgments

The work is supported by the Shenzhen Science and Technology Program (JCYJ20170413162256793), the Research Grants Council of the Hong Kong Special Administrative Region (Project no. CUHK 14225616), the Hong Kong Innovation and Technology Commission (Project no. ITS/304/16), NSFC (Grant No. 61772206, U1611461, 61472145), Special Fund of Science and Technology Research and Development on Application from Guangdong Province (Grant No. 2016B010124011), Guangdong High-level Personnel of Special Support Program (Grant No. 2016TQ03X319), and the Guangdong Natural Science Foundation (Grant No. 2017A030311027). Xiaowei Hu is funded by the Hong Kong Ph.D. Fellowship.