Structure-Aware Residual Pyramid Network for Monocular Depth Estimation

1 Introduction

Monocular depth estimation, which aims to predict the depth value of each pixel from a given RGB image, is crucial for understanding scene geometry, and can be applied to facilitate other vision tasks, such as semantic segmentation [Park et al., 2017] and hand tracking [Qian et al., 2014]. It is an ill-posed problem because of the inherent ambiguity due to perspective projection. Recently, CNN-based approaches have achieved significant success in monocular depth estimation [Laina et al., 2016; Fu et al., 2018; Xu et al., 2018b; Hao et al., 2018; Hu et al., 2019]. To resolve the ambiguity, they typically employ an encoder-decoder architecture to implicitly fuse features that represents object appearance, geometry, semantics, spatial relations, etc. The encoder gradually extracts multiscale features, and the decoder employs multi-stage upsampling as well as shortcut connections to restore object details in high-resolution predictions.

∗Corresponding author

Though a great improvement on average pixel-wise metrics has been made, the underlying structure of objects and stuff is not well preserved by current CNN-based methods. The problem becomes especially challenging when the size of objects and stuff varies widely in complex scenes. As Figure 1 shows, it is challenging for existing approaches to accurately recover the large-scale geometry (walls) and local details (boundaries and small parts) at the same time. This inaccurate inference at regions of diverse scales motivates us to fully exploit the hierarchical scene structure in depth prediction. Scene structure, depicting the organization and arrangement of multiple interrelated elements in a complex scene, varies widely according to the element type. The global structure represents the spatial arrangement of large-size elements such as walls, floors, and furniture objects. Local structure describes geometric details of objects and their parts. The natural hierarchy of scene structure provides essential constraints between the depth values of pixels in multiple scales. Although previous CNN-based techniques extract multi-scale image features and gradually fuse them to predict a depth map, the underlying hierarchical structure of the scene has not been taken into account.

In this paper, we introduce a Structure-Aware Residual Pyramid Network (SARPN) to fully exploit scene structures in multiple scales for depth prediction. A Residual Pyramid Decoder (RPD) is proposed to predict multi-scale depth maps in a coarse-to-fine manner. Depth maps in upper levels in the pyramid represent the global scene structure, while depth maps in lower levels capture more local structures of objects or parts. To convey the global structure and constrain the generation of finer details, we proposed a residual refinement module to predict residual depth maps, which progressively add details on the scene structure on a larger scale. In order to fuse multi-scale features extracted from the input image for residual prediction, we propose an Adaptive Dense Feature Fusion (ADFF) module to adaptively select more effective features for each scale. Integrating the residual pyramid decoder and adaptive dense feature fusion module, our method simultaneously preserves the hierarchical scene structures and produces accurate depth estimation for both large-size shapes and fine details of small object parts, as Figure 1 shows. Our contributions are summarized as follows:

• We propose a Structure-Aware Residual Pyramid Network (SARPN), which takes the underlying scene structure in multiple scales into account for accurate depth prediction.
• Our Adaptive Dense Feature Fusion (ADFF) module adaptively selects features from all scales to predict residual depths at different structure scales.
• The proposed method achieves state-of-the-art performance on the challenging NYUD v2 dataset. More importantly, the visual quality of recovered depth maps is significantly improved.

2 RelatedWork

In recent years, CNNs have become the most successful techniques for various visual tasks, and were firstly used for monocular depth estimation [Eigen et al., 2014] in a multiple scale scheme. Later on, fully convolutional network (FCN) was proposed for semantic segmentation [Long et al., 2015] and has been widely used in many dense prediction tasks, including depth estimation.

When FCN-based architecture was first adopted for depth estimation, the resolution and accuracy ware largely improved by using ResNet to extract features and up-projection blocks [Laina et al., 2016]. In order to improve the quality of depth estimation for local details, many strategies have been introduced. Applying conditional random field as postprocessing [Li et al., 2015] or integrating it in CNNs [Xu et al., 2017] largely improves the prediction quality for small objects. Later, an attention model is integrated to improve the estimation performance [Xu et al., 2018b]. Multi-scale architecture becomes a common solution to avoid the loss of local details caused by spatial pooling and convolutions [Fu et al., 2018]. Instead of multi-scale network structure, dilated convolution is used to extract multi-scale features for depth estimation [Hao et al., 2018]. Hu et al. [2019] proposed an effective multi-scale feature fusion module to produce clear object boundaries. Although these methods have achieved remarkable results by fusing multi-scale features, they still face the problem of inaccurate prediction for complex scenes of which the structure varies largely in scales, from large room layout to fine object details.

In order to better restore structure details, a few methods design new loss functions to explicitly constrain scene geometry. Zheng et al. [2018] proposed an order-sensitive softmax loss to constrain global layouts. Similarly, Fu et al. [2018] used an ordinary regression loss. With respect to clear boundaries and details, a loss function is designed by combining depth, surface normal and gradient in a local neighborhood of depth maps [Hu et al., 2019].

Due to the strong correlation between many visual tasks, such as depth estimation, semantic segmentation, and normal estimation, many approaches employ a joint task learning framework. A multi-scale CNN was designed to simultaneously perform semantic segmentation, depth estimation, and normal estimation [Eigen and Fergus, 2015]. A set of intermediate auxiliary tasks are utilized to guide the final depth estimation and semantic segmentation [Xu et al., 2018a]. Zhang et al. proposed a novel joint task-recursive learning method to recursively refine the results of depth estimation and semantic segmentation [Zhang et al., 2018]. A synergy network is proposed to automatically learn information sharing strategy between depth estimation and semantic segmentation [Jiao et al., 2018]. Moreover, based on the observed long-tail distribution of depth values, an attention-driven loss is also designed to improve the accuracy [Jiao et al., 2018].

3 Methodology

Our network consists of three main parts: an encoder for multi-scale feature extraction, an adaptive dense feature fusion module, and a residual pyramid decoder, as Figure 2 shows. We first introduce the network architecture in Sec. 3.1. The residual pyramid decoder and adaptive dense feature fusion module are explained in Sec. 3.2 and 3.3, respectively.

3.1 Structure-Aware Residual Pyramid Network

Our approach begins with an encoder which extracts multiscale features {Fiex }Li=1 from the input image, where Fiex indicates the feature maps extracted at the i-th level. L is the number of layers in our network. Following the stateof-the-art approach [Hu et al., 2019], we use SENet [Hu et al., 2018] as the backbone of our encoder. It extracts more effective features by re-weighting features of different channels. Given an input image with size W × H, the size of these feature maps are respectively [ W2i , H 2i ], and they carry both high-level semantic information and low-level detail information. Then, these multi-scale feature maps are simultaneously fed to our dense feature fusion module to produce a Fused Feature Pyramid (FFP). These feature maps in FFP are represented by {Fifs }Li=1 , where Fifs indicates the fused feature maps at the i-th level of the pyramid of fused features.

In the decoder part, different from the previous methods that directly predict a depth map by sequentially upsampling feature maps [Laina et al., 2016; Hu et al., 2019], our residual pyramid progressively predicts multiple depth maps in a coarse-to-fine manner. The depth map at the top level with size W32 × H 32 is predicted first as the initial scene layout. We utilize a 1 × 1 convolution operation to reduce the channel number of the feature maps FLex to the same as the channel number of feature maps FLfs of fused feature pyramid and concatenate them together. A residual block is used to predict a depth map DL in size of [ W 2L , H 2L ] from the concatenated feature maps. Then we gradually refine the depth prediction by our proposed residual pyramid decoder.

3.2 Residual Pyramid Decoder

Our residual pyramid decoder predicts depth maps of multiple scales in order to restore the hierarchical scene structures in a coarse-to-fine manner. As shown in Figure 2, the depth maps in lower resolutions depicts more global scene layout, while the depth maps in higher resolutions contain more structure details. In each level of the pyramid decoder, we predict a residual map instead of a dense depth map from fused image features in FFP. The residual map and the depth map predicted at the upper level are integrated together to produce a refined depth map in the current scale using our Residual Refinement Module (RRM). The components of each RRM are shown in Figure 3. The depth map Di+1 predicted at the upper scale is upsampled to the current scale by bilinear interpolation. A residual depth map Dires is generated by utilizing the fused features Fifs . After adding the residual map and the upsampled depth map, a residual block, which contains three convolutional layers, is employed to refine the prediction and outputs a depth map Di at the i-th scale. This residual architecture induces our network to effectively represent the structure details at each scale and hierarchically refine scene structures. Meanwhile, the global scene layout is well preserved by our residual pyramid decoder.

3.3 Adaptive Dense Feature Fusion

In general, due to pooling operations and convolution operations with strides in CNNs, a large amount of low-level visual features are lost. As a result, it is difficult for the decoder to recover the lost low-level structure details. However, both low-level features and high-level features are critical for predicting residual maps in all layers, because the residual maps convey additional details on a global scene structure, as the residual pyramid illustrates in Figure 2. In order to provide sufficient information for the prediction of a residual map in each level, we propose an Adaptive Dense Feature Fusion (ADFF) module. This dense fusion module consists of L Multi-scale Feature Fusion (MFF) modules to predict L fused feature maps, which compose a fused feature pyramid for residual prediction.

In each layer, the MFF adaptively selects eligible features from all feature scales when predicting the depth map for each individual scale. We follow the detailed implementation of MFF proposed in [Hu et al., 2019]. The L feature maps {Fiex }i=1,...,L are first resized to the resolution of current scale using bilinear interpolation and refined with a residual refine block. The refined feature maps are concatenated and fed into a conv-layer to reduce the number of channels.

3.4 Loss Function

In order to train our residual pyramid network for predicting accurate depth maps while preserving scene structures in various scales, we compute the difference between the predicted depth map Di and the ground-truth Gi at each scale and combine the losses of all scales together. For each scale, we follow the definition of the loss function proposed in [Hu et al., 2019]. It consists of three terms, ldepth considering the pixel-wise difference between the predicted depth Dl and the ground truth Gl , lgrad which penalizes errors round edges, and lnormal to further improve fine details. Combing all the L scales, our loss function for the entire network is formulated as

4 Experiments

To demonstrate the effectiveness of the proposed approach, we evaluate our approach on the challenging NYUD v2 dataset [Silberman et al., 2012]. We compare our approach with a couple of state-of-the-art approaches and show the superiority of the proposed method on both quantitative and qualitative evaluations.

4.1 Experimental Setup

The NYU-Depth v2 dataset [Silberman et al., 2012] contains 464 video sequences of indoor scenes captured with Microsoft Kinect. 654 aligned RGB-Depth pairs are provided for testing depth estimation methods for indoor scenes. All images have a resolution of 640 × 480. To training our network, we use the training dataset which contains 50K RGBD images, select and then augment in the same way as [Hu et al., 2019]. Each image is downsampled to 320 × 240 using bilinear interpolation, and then center-cropped to 304 × 228. The predicted depth maps are in a resolution of 152 × 114. For testing, the predicted depth maps are upsampled to match the size of the corresponding ground truth using bilinear interpolation.

We implement the proposed model using PyTorch [Paszke et al., 2017]. The encoder, SENet, is initialized by a model pretrained on ImageNet [Deng et al., 2009]. The other layers in our network are randomly initialized. We use a step learning rate decay policy with Adam optimizer, and starting from an initial learning rate of linit = 10-4 . It is reduced to 10% every 5 epochs. We use β1 = 0.9, β2 = 0.999, and weight decay as 10-4 . The proposed network was trained for 20 epochs with a batch size of 6.

4.2 Performance Comparison

Quantitative Evaluation

Following previous studies, we adopt four metrics including average relative error (REL), root mean squared error (RMS), mean log 10 error (log 10), and accuracy with three thresholds, to quantitatively evaluate our depth estimation performance. Table 1 shows the results of our SARPN and recent approaches. Among the approaches of single task learning, our approach performs the best on REL, log 10 error, and accuracy with three thresholds. We are in the third position with respect to RMS. We speculate that the methods [Fu et al., 2018; Jiao et al., 2018] pay more attention to the absolute pixel-wise accuracy when designing their networks and loss functions, ignoring fine structures of target scenes. As a result, these methods achieve higher performance in RMS, but performs worse on the REL metric and other metrics.

We also compare our method with four approaches that employ joint task learning [Eigen and Fergus, 2015; Xu et al., 2018a; Zhang et al., 2018; Jiao et al., 2018]. The results demonstrated that our method outperforms three methods and achieves comparative performance with [Jiao et al., 2018], even they use a large number of extra labels for semantic segmentation during the training process. Moreover, the depth maps produced by [Jiao et al., 2018] present very blurry object boundaries and miss geometric details. We compare the predicted depth maps in Figure 4 to demonstrate the capability of our method on restoring clear object boundaries and finer details.

We also analyze the contribution of each component in our proposed network. We use a simple UNet-like architecture as our baseline, where SENet [Hu et al., 2018] is employed as the backbone of our encoder. The decoder in our baseline employs a multi-stage upsampling scheme to recover a depth map. A variant (baseline+RPD) is implemented by adding the proposed RPD on the baseline model. As shown in Table 1, the performance is gradually improved by incorporating RPD and ADFF. More specifically, after adding the proposed RPD, performance among all the metrics are improved by a large margin from the baseline, while REL decreases by 6.5%, RMS decreases by 3.5%, log 10 error decreases by 3.8%. After adding the ADFF module, the performance is further improved, while REL decreases by 3.5%, RMS decreases by 2.7% and log 10 error decreases by 4%.

In order to prove the effectiveness of our method on preserving object details, we also compute edge accuracy to measure the quality of recovered edge details, same as [Hu et al., 2019]. Precision, Recall, and F1 score are computed according to edge pixels in the ground truth map. From Table 2, we can see that our F1 score surpasses all other methods under three different thresholds. This indicates that our method restores the most structure details.

Qualitative Evaluation

We compare a series of depth maps predicted by our method and other state-of-the-art methods [Laina et al., 2016; Xu et al., 2017; Fu et al., 2018; Hu et al., 2019] in Figure 5. It can be seen that the depth maps predicted by our method are visually better than other methods. Scene structures are well preserved in different scales, especially for large planar regions and object details. For example, our method predicts accurate geometric details for the bookshelf in the first row, the chair in the third row, and the sofa in the fifth row. For large planar regions (the upper-left wall in the second row, and the floor of the third), our method also generates better results.

To better illustrate the capability of our method on preserving scene structure of large planar regions, we project the predicted depth maps as 3D point clouds and render them in novel views. As Figure 6 shows, our reprojected results are the closest to ground truth. In particular, the large wall regions recovered by our method are much more flat, while other methods suffer from severe distortions.

Model Generalization

In addition to the NYUD v2 dataset, we further explore the generalization ability of our proposed network on other datasets. We test our network, which is trained on the NYUD v2 dataset only, on ScanNet dataset [Dai et al., 2017] and SUN-RGBD dataset [Song et al., 2015], which contain more diverse RGBD data. As shown in Figure 7, even the data distribution of these two datasets and NYU Depth v2 is greatly different, our method could recover structures in various scales, including smooth large planar regions and object details. Moreover, our method also fills holes in the ground truth depth map automatically while maintains the scene structure.

5 Conclusion

In this paper, we propose a Structure-Aware Residual Pyramid Network for accurate monocular depth estimation. A residual pyramid decoder is introduced to predict multi-scale depth maps, which takes the underlying hierarchical scene structures into account. The residual pyramid induces our network to progressively add finer structures at a specific scale while preserving the coarser layout predicted at the upper level. Meanwhile, by using the proposed adaptive dense feature fusion module, the image features from all scales are adaptively fused when predicting the residual depth map for each scale. Experiment results demonstrate that our method achieves state-of-the-art performance in both quantitative and qualitative evaluation.

Acknowledgements

This work was supported by the National Key Research & Development Plan of China under Grant 2018YFC0307905, the National Natural Science Foundation of China (NSFC) under Grants 61632006, 61622211, and 61620106009, the Priority Research Program of Chinese Academy of Sciences under Grant XDB06040900, as well as the Fundamental Research Funds for the Central Universities under Grant WK3490000003 and WK2100100030.