Region Deformer Networks for Unsupervised Depth Estimation from Unconstrained Monocular Videos

1 Introduction

Depth sensing plays an important role in 3D scene understanding. For instance, it is crucial for robots to be aware of how far the surrounding objects are away from themselves, which helps robots keep clear of obstacles and adjust future behaviour. Recently, learning based single-image depth estimation has attracted a lot of attention due to the rapid progress of Convolutional Neural Networks (CNN). Supervised methods [Eigen et al., 2014; Li et al., 2015; Liu et al., 2016] aim at learning a mapping from color image to per-pixel depth by neural networks. However, these methods require a large quantity of color-depth pairs and collecting such dataset is challenging, especially in outdoor scenarios. Unsupervised learning gets rid of the dependence on ground truth depth and shows a promising direction. The key idea of unsupervised learning is to use the warping based image reconstruction loss between adjacent frames to guide the learning process. Several methods have been proposed to use stereo images to estimate depth [Garg et al., 2016; Godard et al., 2017]. Although no ground truth depth is required, the stereo images are still not as common as monocular videos and they need to be carefully synchronized.

∗Corresponding author

In this paper, we consider unsupervised depth estimation from monocular videos. In practice, videos are ubiquitous and unlimited. Many previous works in this line are based on the static scene assumption [Mahjourian et al., 2018; Wang et al., 2018; Zhou et al., 2017], where only camera motion is considered, leading to inaccurate results for moving objects. Several works tried to explicitly model object motion, either with optical flow [Yin and Shi, 2018] or SE(3) transforms [Casser et al., 2019] by assuming rigid motion of objects. However, Yin and Shi [2018] do not report obvious improvement even with the residual flow learning scheme and Casser et al. [2019] only model the motion of rigid objects like cars in driving scenes. For real world scenarios, deformable objects are often present in various forms such as pedestrians and animals.

We observe that the coupling of camera motion and individual object motion often causes ambiguities and may confuse the learning process in dynamic scenes. Therefore we propose to disentangle camera motion and individual object motion between adjacent frames by introducing an additional transformation fθ : R2 → R2 for every independently moving object to deform itself to adjacent frame, as illustrated in Fig. 1. The transformation is modeled as a bicubic deformation and the transformation parameters θ are learned by a CNN, which is fully guided by image appearance dissimilarity (see Sec. 3.1 for detail). To realize the individual transformation, the existing instance segmentation method, Mask R-CNN [He et al., 2017], is used to segment objects out in each frame, which is only needed at training time and helps the motion representation to learn a better depth network.

The paper has made the following contributions:

• We present a learning based method to estimate depth from unconstrained monocular videos. The method consists of DepthNet, PoseNet and Region Deformer Networks, which does not need ground truth supervision.
• We propose a deformation based motion representation to model the non-rigid motion of individual objects between adjacent frames on 2D images. This representation is general and enables our method to be applicable to unconstrained monocular videos.
• We conduct extensive experiments on three datasets across diverse scenes. Our method can not only achieve the state-of-the-art performance on standard benchmarks KITTI and Cityscapes, but also show promising results on a crowded pedestrian tracking dataset, which validates the effectiveness of the proposed deformation based motion representation.

2 RelatedWorks

This section briefly reviews some learning based depth estimation work that is most related to ours.

Supervised Depth Estimation. Many supervised methods have been developed to estimate depth [Eigen et al., 2014; Li et al., 2015; Liu et al., 2016; Xie et al., 2016]. These methods use CNN to learn a mapping from RGB images to depth maps. However, they need a dataset with ground truth depth which is hard to acquire, especially in outdoor scenarios, and hence limits its applicability. Several works try to resolve this limitation by using synthetic data [Mayer et al., 2018; Zheng et al., 2018] or images from Internet [Chen et al., 2016; Li and Snavely, 2018], but special care must be taken to generate high quality data, which can be very time-consuming.

Unsupervised Depth Estimation. Unsupervised approaches use image reconstruction loss between adjacent frames to provide self-supervision. Garg et al. [2016] propose to use calibrated stereo pairs as supervision to train a single view depth CNN. Godard et al. [2017] further improve the performance by imposing left-right consistency constraints. Zhou et al. [2017] propose to learn depth and ego-motion from monocular videos under the static scene assumption, with an additional learned explainability mask to ignore the motion of objects. Yin and Shi [2018] propose to learn a residual flow to handle the motion of objects. Zou et al. [2018] jointly learn depth and optical flow from monocular videos with a cross-task consistency loss in the rigid scene.

Our work focuses on depth estimation from monocular videos as videos are more easily available than rectified stereo pairs. The work most similar to ours is [Casser et al., 2019]. However, there are two important differences: (i) Casser et al. [2019] model object motion in 3D with SE(3) transformation, which is good for rigidly moving objects, like cars in driving scenes. We use a deformation based representation to model object motion in 2D image plane, which is more general to be applicable to diverse real world scenarios. (ii) To handle the common issue that cars moving in front of camera at roughly the same speed are often projected into infinite depth in monocular setting, Casser et al. [2019] propose to impose object size constraints depending on the height of object segmentation mask, which is not suitable for deformable objects as the actual scale can be varied over time. Also, the constraints in [Casser et al., 2019] are learned by a network which can be tricky to find the good hyper-parameters. Instead, we choose to use a simple yet efficient prior inspired from [Ranftl et al., 2016], which is more general for diverse scenes and has no parameters to learn.

Learning Geometric Transformation. Spatial Transformer Networks (STN) [Jaderberg et al., 2015] build the first learnable module in the network architecture to handle geometry variation of input data, which is realized by learning a global parametric transformation. Deformable ConvNets [Dai et al., 2017] further extend STN by learning offsets to regular grid sampling locations in the standard convolution. STN and Deformable ConvNets are both aiming at designing network architectures with geometry invariant for supervised tasks like classification and segmentation. Our deformation based motion representation aims at learning a transformation for each of individual objects to model object motion between adjacent frames.

3 ProposedMethod

This section presents our generic framework for unsupervised depth estimation from unconstrained monocular videos. The input is a sequence of video frames {It}Nt=1 , It ∈ RH×W×C , where H,W,C represent frame It's height, width and number of channels, respectively. Our goal is to estimate the corresponding depth maps {Dt}Nt=1 , Dt ∈ RH×W . For this purpose, we build a DepthNet to learn the mapping from color image to per-pixel depth map, a PoseNet to learn the mapping from two adjacent frames to their relative camera pose transformation, and multiple RDNs in parallel to learn the transformations that model the motion of individual objects from one frame to its adjacent frame. The overall framework is illustrated in Fig. 2. For two adjacent frames It and It+1, we first obtain instance segmentation masks from the existing Mask R-CNN model. It is fed into the DepthNet to predict its depth Dt. The concatenation of It and It+1 is fed into the PoseNet to learn the camera pose transformation Tt+1→t between It and It+1, where objects are masked out to avoid motion clue from possibly moving objects. We further use the RDN to model the motion ∆p of individual objects in parallel. With Dt, Tt+1→t,∆p and It+1, we reconstruct a synthetic frame I ⋆ t+1→t corresponding to It. The appearance dissimilarity between I ⋆ t+1→t and It provides training signal of our framework. During testing, only the DepthNet is needed to predict the depth for an input frame.

Below we first give the basic formulation of the loss function, and then describe the deformation based motion representation that explicitly handles object motion on 2D images.

3.1 Basic Formulation

With Dt, It+1, Tt+1→t, we synthesize image

corresponding to frame It, where W is a warping function. We first construct W based on the static scene assumption, and then improve it for dynamic scenes by adding individual object motions in Sec. 3.2.

For each pixel coordinates p = (x, y) in frame It, we can obtain its projected coordinates ˆ p = (ˆx, ˆ y) in frame It+1 with the estimated depth Dt and camera transformation Tt+1→t

where h(p) = (x, y, 1) denotes the homogeneous coordinates of p and K denotes the camera intrinsic matrix. We use the bilinear mechanism used in [Jaderberg et al., 2015] to sample frame It+1 to create image ˆ It+1→t .

The appearance dissimilarity between reconstructed image ˆ It+1→t and It is defined as

where function ρ is a combination of L1 photometric error and Structure Similarity (SSIM) [Wang et al., 2004]:

To handle occlusion/disocclusion between adjacent frames, per-pixel minimum of the dissimilarities with previous frame and next frame is used as proposed in [Godard et al., 2018]:

where ˆ It-1→t is reconstructed image from It-1.

To regularize the depth, we further impose an image-aware depth smoothness constraint as commonly used in previous works [Godard et al., 2017; Yin and Shi, 2018]:

The total loss function is a combination of Lap and Ls applied on four different image scales, ranging from the model's input resolution, to an image that is 1/8 in height and width:

where λ is a hyper-parameter. We use Eq. 7 as our baseline model, which is similar to that in [Casser et al., 2019].

3.2 Region Deformer Networks

For dynamic scenes, individual objects may have their own motions besides camera motion. We propose to explicitly model individual object motion on 2D images. Specifically, our goal is to learn the displacement vector ∆p = (∆x,∆y) of every pixel belonging to moving objects. Then the corresponding pixel of p can be found by p⋆ = ˆ p +∆p. This process is accomplished by learning a function f : R2 →R2 for each object to map (x, y) to the displacement (∆x,∆y). The basic requirement for f is that it should be flexible enough to model non-rigid object motion. We choose the bicubic function, which is expressed as

The bicubic function is widely used in various applications. It has low computational cost, and meanwhile it contains 32 coefficients, providing sufficient degrees of freedom for modeling freeform deformation (or motion).

To learn the transformation parameters {aij}3i,j=0 and {bij }3i,j=0 , we design the Region Deformer Networks (RDN). The workflow of the RDN is illustrated in Fig. 1. Given two adjacent frames It and It+1, an instance segmentation model [He et al., 2017] is used to segment objects within each frame. Let M i t andM i t+1 denote the i-th binary object segmentation masks in frame It and It+1, respectively. We first compute the reconstructed image ˆ It+1→t andmask ˆM i t+1→t by camera motion using Eq. 1, which eliminates camera motion between adjacent frames. The input of RDN is the concatenation of objects Oi t = It ⊙M i t and ˆOi t+1→t = ˆ It+1→t ⊙ ˆM i t+1→t in It and ˆ It+1→t, respectively, where ⊙ denotes element-wise multiplication, making only the i-th object visible for every independent RDN. When multiple objects exist in a single frame, multiple RDNs are used in parallel to model every independent object motion. The outputs of RDN are the parameters of the transformation. By applying the transformation to its corresponding object, we obtain the displacement of the pixels belonging to that object.

Now we are ready to refine the function W by defining a new warping function W⋆ . If a pixel p belongs to static background, we compute its correspondence by Eq. 2. If p belongs to moving objects, its correspondence is found by p⋆ = ˆ p+∆p, where ˆ p is from camera motion using Eq. 2 and ∆p is from object motion obtained by RDN. In general, for every pixel p in frame It, we can get its correspondence by

where Mt = ∑Si=1 M i t is the binary instance segmentation mask for It: Mt(p) is 1 if p belongs to moving objects; otherwise 0, and S is the total number of objects in It. By modeling individual object motion with RDN, we can get more accurately reconstructed image

where {fi}Si=1 are the individual transformations learned by the RDN. Similarly, we can generate image I ⋆ t-1→t .

3.3 Object Depth Prior

Depth estimation from monocular videos has an issue that objects moving with camera at the roughly same speed are often projected to infinite depth, as this shows very little appearance change, resulting in low reprojection error [Godard et al., 2018]. Casser et al. [2019] propose to impose object size constraints by additionally learning the actual scales of objects. These constraints are internally based on the assumption that object scales are fixed. However, in real world scenarios, deformable objects are often present, like pedestrians and animals, which are not applicable for these constrains. Furthermore, as the actual scales are also learned during the training process, it can be tricky to find the good hyper-parameters.

We propose to use a simple yet efficient prior inspired from [Ranftl et al., 2016]: objects are supported by their surrounding environment, which is often true in most real world scenes. This prior can be used by requiring the depths of moving objects to be smaller or equal to their horizontal neighbors. However, noticing that overlapping objects may exist in the real world, which might violate this depth prior, we thus introduce a soft constraint, which is formulated as

where dobj is the mean depth of an individual object, dneigh is the mean depth of its horizontal neighbors in a small range, and δ is a small positive number to handle exceptions violating our depth prior. The main idea here is to use the depth prior to prevent the degenerated cases of infinite depth. Note that if this prior is satisfied, which happens most of the time, Eq. 11 actually has no use (i.e., the loss becomes 0).

By incorporating the RDN and object depth prior, our final loss function can be expressed as

where µ is a hyper-parameter. The only difference between L(k) ap and L⋆(k) ap is that we replace It+1→t, It-1→t in Eq. 1 with I ⋆ t+1→t , I ⋆ t-1→t . We treat Eq. 12 as our motion model.

4 Experiments

4.1 Datasets

We conduct experiments on several datasets across diverse scenes, including not only standard benchmarks KITTI and Cityscapes, but also a publicly available pedestrian tracking dataset. The details of each dataset are given below.

KITTI. The KITTI dataset [Geiger et al., 2012] is a popular benchmark for scene understanding in outdoor driving scenario. Only the monocular video sequences are used for training and no ground truth depth is needed. The performance is evaluated on Eigen split [Eigen et al., 2014] using the standard evaluation protocol.

Cityscapes. The Cityscapes [Cordts et al., 2016] is an outdoor driving dataset similar to KITTI, but with more moving objects. We use this dataset for training and the evaluation is done on Eigen split.

Pedestrian Tracking Dataset. To validate that our motion representation is general enough to model deformable objects, we collect videos from a publicly available pedestrian tracking dataset [Ess et al., 2009], which was recorded on a crowded pedestrian zone. This dataset is very challenging as large human deformations are frequently observed.

4.2 Implementation Details

Our method is implemented in TensorFlow. The input images are resized to 128 × 416 when training on KITTI and Cityscapes datasets, and to 240 × 320 when training on the pedestrian tracking dataset. The loss weights µ, α are set to 0.5, 0.15, respectively. The smoothness weight λ is set to 0.04 when training on KITTI and the pedestrian tracking dataset, and to 0.008 when training on Cityscapes dataset. The δ in Eq. 11 is set to 0.01 when training on KITTI and the pedestrian tracking dataset, and to 0.5 when training on Cityscapes dataset due to the different data distributions of the datasets. The batch size is chosen to be 4 and Adam [Kingma and Ba, 2014] is used to optimize the network with

β1 = 0.9, β2 = 0.99. When training the baseline model, the learning rate is set to 2 × 10-4 . Our motion model is trained with learning rate of 2 × 10-5 and the network weights are initialized from our trained baseline model. We use the same DepthNet and PoseNet architectures as [Casser et al., 2019]. Our RDN uses the same architecture as PoseNet except for the last output layer.

Evaluation Metrics. We evaluate our method with standard metrics reported in [Eigen et al., 2014], which include 4 error metrics (Abs Rel, Sq Rel, RMSE, RMSE log) and 3 accuracy metrics (δ < 1.25, δ < 1.252 , δ < 1.253 ). The definition of these metrics is referred to [Eigen et al., 2014].

4.3 Results

The KITTI and Cityscapes Datasets. We report our depth estimation results on standard Eigen test split [Eigen et al., 2014] of KITTI raw dataset in Tab. 1. All methods are evaluated on KITTI dataset, no matter whether they are trained on KITTI or Cityscapes dataset. We achieve comparable results with [Casser et al., 2019] when training on KITTI dataset, which is specially designed for outdoor driving scenes. When training on more dynamic dataset Cityscapes, our performance is consistently better than that of [Casser et al., 2019]. More evidence can be seen from the qualitative comparisons in Fig. 3. On the other hand, our motion model is consistently better the baseline no matter training on KITTI or Cityscapes dataset (see Tab. 1 and the visual results in Fig. 4).

Ablation Study. To further evaluate our proposed bicubic motion representation, we create a new baseline named 'Ours (Rigid)', where we replace the bicubic function in Eq. 8 with rigid motion representation and keep all the other settings the same. The results given in Tab. 2 clearly demonstrate the superiority of our bicubic motion representation, winning the majority of the metrics.

To evaluate the contribution from the object depth prior, we add another baseline called 'Ours (w/o prior)', which disables the prior in our full motion model. As shown in Tab. 2, compared with Ours (Motion), the performance of Ours (w/o prior) degrades a lot, which verifies that our object prior is simple yet efficient to handle the infinite depth issue as explained in Sec. 3.3.

The Pedestrian Tracking Dataset. To illustrate the generality of our proposed bicubic motion representation, we conduct experiments on a crowded pedestrian tracking dataset, which is quite different from KITTI and Cityscapes datasets and particularly challenging due to the presence of many deformable pedestrians. Fig. 5 visualizes the depth prediction results of samples from this dataset. Clear improvements can be seen from our motion model. The promising results show the generality of our proposed bicubic motion representation and indicate that our framework is applicable to unconstrained monocular videos.

5 Conclusion

We have presented a learning based approach to estimate depth from unconstrained monocular videos without ground truth supervision. The approach consists of DepthNet, PoseNet and RDN, for which a deformation based bicubic motion representation is proposed to model object motions in diverse scenes. The experimental results on several datasets show the promising performance of the proposed approach and validate the effectiveness of the deformation based motion representation as well.

6 Acknowledgements

The authors are supported by the National Natural Science Foundation of China (No. 61672481) and the Youth Innovation Promotion Association CAS (No. 2018495).