Depth-Relative Self Attention for Monocular Depth Estimation

1 Introduction

Depth estimation, a task to estimate the distance from the viewpoint, is one of the most important tasks in computer vision having a variety of applications such as autonomous driving [Wang et al., 2019; You et al., 2019], object localization [Kramer and MacKinnon, 1993; Tompson et al., 2015], 3D reconstruction [Geiger et al., 2011; Izadi et al., 2011], to name just a few. Due to the cost and power consumption of depth measuring sensors (e.g., LiDAR, Time-of-Flight), a single RGB image has been used for this task in many realworld applications [Atapour-Abarghouei and Breckon, 2018; Yucel et al., 2021; Wofk et al., 2019]. The major difficulty of this task, monocular depth estimation (MDE), is that the task is ill-posed since there are multiple answers for the given scene. Recently, deep neural networks alleviated this problem by exploiting diverse visual clues such as relative size, brightness, patterns, and vanishing point extracted from an RGB image. It has been shown that these visual clues, collectively called

"visual hints", are useful in predicting the depth [Saxena et al., 2007; Ming et al., 2021].

In order to improve the quality of visual hints, a pre-trained network referred to as 'backbone' has been widely used. Recently, depth estimation performance has been substantially improved [Li et al., 2022a; Yuan et al., 2022] due to the diverse and complex RGB-based visual features obtained from the large-scale backbone networks [Girshick et al., 2014; Kolesnikov et al., 2020; Liu et al., 2021]. However, we observe that some visual information such as painted surfaces, a patterned carpet, and reflected sunlight sometimes provide false signals to the network and degrade the accuracy of the predicted depth (see Figure 1). While the visual hints are useful to some extent, they would do more harm than good when models become overly dependent on such information. In the sequel, we call the visual hints that confuse the model and therefore have an adverse effect on the depth estimation as "visual pits". For example, in Figure 1(a), the dark paint of the truck affects the model such that the truck appears farther away than it really is. Clearly, visual pits can be a potential risk factor for the autonomous driving system.

To reduce the negative effect of visual pits, we should design the system such that the extracted features are more related to depth while less dependent on RGB-based information. In other words, we expect that the features corresponding to pixels of similar depths to be similar. As an enabler to achieve this goal, we exploit the relative depth, a difference between the depth of two pixels. If the relative depth between two pixels is small, the model should generate similar features regardless of their RGB attributes and spatial distances in 2D image 1 . When this property is satisfied, even though the two complementary information (i.e., RGB and depth) make contradictory predictions, we can use the relative depth as guidance in estimating the correct depth. For example, in Figure 1(b), the model is confused by the reflected sunlight, resulting in an incorrect prediction that the upper part of the pillar is farther than its real depth. Even in this case, using the relative depth between the lower and upper parts of the pillar is small, the model can figure out that both parts are actually at the same depth.

1 In order to make a clear distinction between the distance in the 2D image and the real world, we exclusively use the terms 'near/far' for the former, and 'close/distant' for the latter.

In this paper, we propose a novel MDE model referred to as RElative Depth Transformer (RED-T). The key idea of RED-T is to exploit the relative depth as guidance in computing the self-attention weights. To this end, we design the depth-relative attention module on top of the backbone. Using the relative depth information, this module modifies the self-attention weight in two steps. First, we gather the relative depth information to determine which pixels should be similar in the feature domain. Second, we adjust the self-attention weight based on the relative depth; large (small) attention weights for pixels of small (large) relative depths. We expect that features with large attention weights are more or less similar to each other since the self-attention mechanism is basically a weighted sum of features. In fact, through the proposed depth-aware self-attention process, the features corresponding to pixels with small relative depths will be close to each other in the feature domain, even if their RGB values are different. Whereas, if the relative depth is large, the features would be distinct although their corresponding RGB attributes may look alike.

To demonstrate the negative effect of visual pits, we propose a new practical MDE environment termed range-restricted MDE. In the conventional depth estimation benchmarks, the target depth range is the same for both training and evaluation. To make things worse, the annotated depth data has limitations in range (e.g., 10m for NYU-v2 [Silberman et al., 2012] and 80m for KITTI [Geiger et al., 2013] datasets). However, in real-world scenarios, we should estimate the depth of distant objects correctly even if their depths are not specified in the training data. When the model only learns the correlation between RGB attributes and limited depths, the model would inaccurately estimate the unseen depths to seen depths highly dependent on the RGB information, which will intensify the adverse influence of visual pits on such unseen depth range. In the proposed environments, we erase the depth labels of a certain range as if they are not annotated in training data. For example, for the data with a depth range of 0 ∼ 80m, we remove the training labels in 40 ∼ 80m and evaluate the model with the full range (0 ∼ 80m). In our evaluations, we show that RED-T predicts not only the learned depth but also the outof-range depth more accurately than previous state-of-the-art MDE models.

The contributions of this work are as follows:

• We employ relative depth, a difference between the depth of pixels, as guidance to solve the problem of visual pit. To the best of our knowledge, we are the first to tackle the negative effect of visual information in MDE.
• We propose a novel depth-relative attention that adjusts the self-attention weight based on the relative depth. In essence, the proposed mechanism guides feature such that the depth is more considered than RGB information.
• Using two MDE datasets (KITTI & NYU-v2), we evaluate RED-T and show that the proposed RED-T outperforms the recent MDE models that use the same backbone in all metrics for the extremely competitive KITTI dataset.
• To evaluate the performance of MDE models in practical environments, we suggest new depth estimation scenarios that restrict observable depth range during training. We show that the depth-relative attention bias makes the model more robust in estimating unseen depth ranges.

2 RelatedWork

Adversarial Effect of Visual Pits. Visual information extracted from an RGB image such as color, texture, style, and brightness is known to be helpful in object detection, semantic segmentation, and super resolution [Kim et al., 2002; Liang et al., 2021; Zhong and Jain, 2000], etc. However, this perceptual information is not always reliable, especially when the goal of the task is to generate output in a non-RGB domain with RGB input. For example, in the image segmentation task, the pixels corresponding to the same class must produce the same mask even if their RGB values are different. Previous work pointed out that the physical effects of illumination, shadow, shading, and highlights can cause considerable noise in the image segmentation output [Vazquez et al., 2010]. Likewise, in MDE, visual pits such as patterned surfaces, dark screens, and reflections in the mirror can disturb the depth estimation process. To avoid the failures caused by visual pits, we exploit the relative depth information such that the MDE model can focus more on depth-related information.

Relative Depth. Several studies have used relative depth information for depth estimation [Huynh et al., 2020; Lee and Kim, 2019], but their works are very distinct from ours. The main difference to the referred papers is that 1) we investigated the 'visual pit' problem which has not been studied before and 2) we exploited the novel depth-relative attention bias instead of the position-relative bias. We note that studies [Huynh et al., 2020; Lee and Kim, 2019] that introduced relative depth in the feature extraction are difficult to apply when dense depth labels are not given. For example, the former does not show the performance on KITTI, a dataset where only a few pixels are labeled sparsely. Also, the latter mentioned that the training process was less reliable on KITTI as their labels are not annotated. In contrast, as can be observed in restricted label experiments, our method works well on much sparser scenarios than the original KITTI. Please also note that we are the first to exploit relative depth information among the models that use the same Swin backbone [Li et al., 2022a; Li et al., 2022b; Agarwal and Arora, 2023].

3 RED-T: RElative Depth Transformer

In this section, we discuss three components, backbone, neck, and head, of the proposed RED-T. Figure 2 illustrates the overall architecture of the model.

3.1 Monocular Depth Estimation

Let H and W be the height and width of the image, then the MDE model takes a single RGB image I ∈ RH×W×3 as input and returns the estimated depth map D ∈ RH×W×1 . Each element of D represents a distance d from the viewpoint. Because the ground truth depth map D∗ contains only a few annotated pixels, the loss is computed on those pixels in the training stage.

3.2 Backbone: Position-relative Transformer

As a backbone, we use Swin Transformer (Swin) [Liu et al., 2021], a multi-stage Transformer whose self-attention (SA) is computed with non-overlapping local windows. In Swin, SA between n pixels is calculated as:

where h is the attention head index over the total number of heads Nh and dh is the attention head dimension. Qh,Kh, Vh ∈ Rn×dh are the query, key, value matrices for the h-th attention head, respectively, and Ah ∈ Rn×n is the attention weight. To promote the spatial relationship between pixels, Swin adds the relative positional attention bias Bh ∈ Rn×n to the attention weight. Note that Bh is unrelated to the content and depends only on the difference in coordinates (i.e., spatial location) between pixels.

3.3 Neck: Parallel Multi-scale Aggregation

The neck performs parallel processing of the multi-scale backbone features with different scales and then stacks them together at the highest resolution (largest scale). To ensure that the scale of features is the same, features are upsampled to the highest resolution. Let f{0,1,2,3} be the image features extracted from the backbone corresponding to 1/4, 1/8, 1/16, 1/32 scales, then each image feature fi is passed through a convolutional block. The block takes i-th feature fi as an input and then returns the processed feature gi. The generated features g{0,1,2,3} are concatenated and passed through an additional linear layer followed by layer normalization.

Traditional feature pyramid network (FPN) merges multiscale features one after another, from the smallest scale features to the largest scale ones [Lin et al., 2017; Tan et al., 2020; Redmon and Farhadi, 2018]. Since FPN merges the multiscale features sequentially, global information from smallscale features can be blurred during the hierarchical process [Chen et al., 2020; Yu et al., 2020]. This might cause a loss of the global information presumably obtained from low-resolution features in local pixels. Our neck architecture overcomes the potential weakness by combining all scales simultaneously.

3.4 Head: Depth-relative Transformer

The relative depth r between two pixels x1 and x2 is the difference between their depth values d1 and d2, that is r(x1, x2) = d1 - d2. To obtain the relative depth between every pair of pixels, each pixel should have its own depth value; however, such a dense depth map is not available during the training and even the GT map does not contain depth values for all pixels. To deal with the issue, we generate the intermediate dense depth map prediction and use it to compute the relative depth information. The relative depth information is then used to predict the enhanced depth map. This process can be interpreted as self-guided bootstrapping; RED-T repeats this cycle multiple times (K times) to improve the intermediate depth maps progressively.

The detailed process of each cycle is as follows:

Discretization. In the i-th iteration, the model produces an intermediate depth map Di. Since Di is a real-valued dense depth map, every pixel of Di has its own estimated depth value and thus every relative depth can be computed. Then, we discretize depth values by uniformly splitting the min-max depth range, where the number of bins Nb is a hyperparameter. This discretization converts depth map Di into Ddis i , as illustrated in Figure 3. Note that the number of possible relative depths after the discretization is 2Nb - 1, from -Nb + 1 to Nb -1. If we increase Nb in the discretization process, a more fine-grained granularity of relative depth can be obtained. We empirically observed that 128 bins are sufficient.

Parameterization. We parameterize the possible relative depths as embedding parameters θDE ∈ R(2Nb-1)×Nh (see Figure 3). The goal of this parameterization is to map a raw relative depth to a trainable parameter that can be simultaneously trained with other parameters. By doing so, the effect of relative depth on the attention weight can be automatically adopted for performance during training. Note that the parameter size of θDE is quite small (about 2K per each self-attention module) although different attention head uses different embedding parameters.

Pairwise subtraction &Embedding. For every pixel pair, we perform a pairwise subtraction of two discretized depth values fromDdis i and then take the corresponding embedding parameter from θDE to construct the depth-relative attention bias R. For example, in Figure 3, the pairwise subtraction outputs 197 by subtracting 198(A) and 1(B), which are discretized depths. Then, the vector corresponding to the index 197 is taken from θDE to (A, B) point of R. The R represents the relationship between pixels in terms of their depth difference, or relative depth. Note that each entry of R only depends on the relative depth between pixels and not on their visual features.

Depth-relative Self-attention. Instead of using the conventional relative positional attention bias B (see Eq. (1)), we incorporate the relative depth attention bias R as below:

This novel depth-relative SA mechanism encourages pixels of similar depth to focus more on each other. By assigning higher attention weight to features of similar depth (small relative depth), the features can be more correlated to depth. Thus, the model can less affected by visual pits such as patterns or colors.

3.5 Other Details

Relative Depth Computation. From earlier trials, we find that the uniform separation of depth range works well and performs better than the log-uniform partitioning as suggested in DORN [Fu et al., 2018]. Different depth-relative SA blocks equip their own relative depth embedding parameters θDE so that each SA can learn diverse depth relationships.

Training loss. The total loss Ltotal is the scale-invariant loss [Eigen et al., 2014] averaged over all intermediate depth maps Di∈{0,1,...K-1} and the final depth map DK.

where hi,x = log d∗x - log di,x and T is the number of valid GT labels. The loss for each depth map is calculated by the same equation above, reducing the possibility of the wrong prediction being amplified through iterations. We set λ=0.85 and α=10 following previous works [Bhat et al., 2021; Yuan et al., 2022].

4 Experiment

4.1 Dataset

NYU-v2 [Silberman et al., 2012] dataset includes pairs of RGB images and depth maps on 464 indoor scenes, which are separated into 120K training samples from 249 scenes and 654 testing samples from 215 scenes. The range of depth labels is up to 10 meters. We train our model on 50K subset following previous work [Yuan et al., 2022].

KITTI [Geiger et al., 2013] dataset consists of paired RGB images and corresponding depth maps obtained by a 3D laser scanner on 61 outdoor scenes while driving. The range of depth annotations is up to 80 meters. We apply two mainly used training/testing dataset splits. First, following the Eigen split setting [Eigen et al., 2014], we train our model with about 26K samples from 32 scenes and test on 687 samples from 29 scenes. Second, for the online depth prediction configuration [Geiger et al., 2012], we use 72K training samples, 6K validation samples, and 500 testing samples.

4.2 Implementation Details

We employ Swin-Large as a backbone, pre-trained on ImageNet-22K dataset [Deng et al., 2009] with an input image size of 224 and window size of 7. Each stage of the convolutional neck produces 512-channel feature maps, which are then concatenated and projected to 512 channels. The number of depth-relative SA heads is set to 8 and their window size and shift size is set to 8 and 4, respectively. We set K = 3 and Nb = 128 as default. The size of the output depth map is the 1/4 scale of the input image, which is then resized to the full resolution.

We use AdamW optimizer [Kingma and Ba, 2014] with a learning rate of 1e-4, (β1, β2) of (0.9, 0.999), and a weight decay of 0.1. The learning rate starts at 4e-6, increases to the maximum value for 25% of the total iterations, and then decreases to 1e-6. We train our model with a batch size of 16 for 24 epochs on 8× NVIDIA A5000 24GB GPUs. The gradient is accumulated every 2 batches and clipped to the maximum gradient norm of 0.1. Please see the Appendix for details about data pre-processing, augmentation, metrics, and evaluation procedure.

4.3 Depth Estimation Performance

Table 1 shows the MDE performance on the NYU-v2 dataset. Despite the fact that several models employ the same or larger backbones than RED-T or exploit additional data during training [Ranftl et al., 2021], RED-T achieves higher or comparable results in most of the metrics. In particular, RED-T reduces 'Abs Rel' and 'log 10' errors by 4.2% and 4.9% compared to NeWCRFs [Yuan et al., 2022], respectively.

Table 2 presents the performance on KITTI Eigen split dataset. RED-T outperforms previous works in every metric; especially, RED-T achieves lower relative errors ('Abs Rel' and 'Sq Rel') and absolute errors ('RMSE' and 'RMSE log'). We also evaluate RED-T on the KITTI official split which measures the performance on the official server. As shown in Table 3, RED-T surpasses competitors by a large margin, especially in 'Abs Rel' and 'iRMSE' metrics.

The number of parameters of models that use the same SwinL backbone is 270.4M, 273.8M, and 248.3M for NewCRFs, DepthFormer, and RED-T, respectively. Note that the backbone contains 195.0M parameters.

4.4 Qualitative Evaluation

In Figure 1(a), a truck is painted with diverse colors (i.e, black, gray, white) on its surface. Although the depth of the surface continuously changes, in previous work [Bhat et al., 2021], undesired change in depth appears in the estimated output due to the color difference. Another example is a kitchen counter wall decorated with a square pattern (Figure 1(c)). While the depth of the wall should change smoothly, in the previous work, the pattern erroneously stands out in the depth map. Thanks to the relative depth that help distinguish visual pits from visual hints, RED-T is robust to such obstacles. In other words, RED-T can accurately predict the depth of an object while much less affected by its visual appearance in 2D images. Please check the Appendix for more qualitative comparisons.

5 Range-restrictedMDE

5.1 Motivation

As mentioned in Section 1, the harm of visual pits would be amplified when the model only exploits RGB information for depth estimation. Unfortunately, this is an inherent problem for MDE because 1) the model only takes a single RGB image as input, and 2) the range of the annotated depth label is limited. Therefore, for certain depth ranges that the model did not observe during training, the model solely depends on RGB values including visual pits which hurts the performance.

5.2 Task Specification

We propose a new MDE task that only a limited range of GT labels is given during training. Specifically, let the GT labels in test data d∗ ∈ [dmin, dmax], then we remove labels larger than dclip and use only [dmin, dclip] during training phase. As a result, the model should predict both seen and unseen depth ranges during the test phase. Figure 4 shows examples of the restricted GT maps corresponding to different dclip values.

5.3 Experimental Results

We conduct experiments on the KITTI dataset, where dmax = 80m, with two configurations of dclip ∈ {40m, 60m}. Table 4 shows that RED-T achieves much lower performance degradation than previous models. In dclip = 60m setting, RED-T achieves zero performance loss in the 'Abs Rel' metric and 7.3% reduction in 'RMSE' metric, while AdaBins suffers from 32.8% and 13.6% performance loss, respectively. We claim that RED-T is robust to unseen depth range because the model can avoid visual pits by actively incorporating the relative depth information in the model design.

5.4 Effectiveness of Relative Bias

To highlight the importance of the relative depth, we repeat the same experiments without relative bias (i.e., forcing R = 0) on the NYU dataset. In Table 5, RED-T without depth-relative attention guidance shows 1) worse performance and 2) larger relative performance decay compared to the proposed RED-T. The gap between the RED-T with and without relative bias becomes larger as the observable depth range (dclip) decreases.

In addition, we measure the RMSE as a function of distance in various scenarios. In Figure 5(a), models trained by a restricted depth range show much larger error compared to the baseline (i.e., the model trained on full depth range) in unseen (depth) ranges. We observe that relative bias improves the generalization capability of the model in unseen ranges. Furthermore, in Figure 5(b), we show that multiple iterations of depth-relative processing in the head consistently reduce the error. Specifically, the RMSE is reduced by 2.1%, 4.6%, and 10.5% as the number of iterations k increases.

Figure 5: Fine-grained depth estimation results of range-restricted MDE experiments on the KITTI dataset.

One may think that the improvement of the relative bias is not dramatic on conventional depth estimation metrics. We argue that current metrics do not sufficiently express the negative effect of visual pits. First, the metric values are averaged over valid pixels that are sparsely annotated, but visual pits mostly appear within concentrated regions. Second, in terms of the number of pixels, the proportion of visual pits to the entire image is often very small (under 1% over the entire image). Nevertheless, we emphasize that visual pits are risk factors for practical systems; even the danger amplifies when the model attempts to predict unseen depth.

6 Conclusion

In this paper, we proposed RED-T which aims at minimizing the adversarial effect of visual pits. To do so, RED-T utilizes relative depth information as a means to guide the monocular depth estimation process. Specifically, we adopt self-attention bias to encourage each pixel to assign high attention weight to other pixels of close depth. RED-T achieved superior depth estimation performance on NYU-v2, KITTI Eigen/official split datasets compared to the competitors. To demonstrate the effectiveness of relative depth, we introduced a new MDE task that restricts observable depth range during training.

Acknowledgments

This work was supported by the Future-promising Convergence Technology Pioneer Program of the National Research Foundation of Korea (NRF) grant funded by the Korea Ministry of Science and ICT (MSIT) (No. 2022M3C1A3098746) and National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (RS-2023-00208985).