Removing Foreground Occlusions in Light Field using Micro-lens Dynamic Filter

1 Introduction

Removing the dense and complex occlusions is beneficial for many high-level computer vision applications, such as object detection, recognition and tracking, since the accuracy may be highly improved if the background objects are recovered.However, it is difficult to automatically detect and remove dense and irregular occlusions in one single image. On one hand, the depth information is unknown and foreground occlusions are hard to identify. On the other hand, no reliable information about the occluded background is provided. By contrast, the recently developed Light Fields (LFs) contain Sub-Aperture Images (SAIs) that capture scenes in different directions, in which background points that are occluded in the central SAI may be visible in other SAIs. Therefore, LFs show great potential in the foreground occlusion removal task.

In recent years, with the rapid development of deep learning, single image inpainting methods [Liu et al., 2018; Xie et al., 2019; Li et al., 2020] have made significant breakthroughs. By learning plausible structures from a large amount of data, these inpainting methods try to combine surrounding semantical information to 'generate' labeled missing regions. However, for the occlusion removal task in LFs, it is more important to find and further 'seek' occluded regions in different SAIs. Since LFs contain structure information among different SAIs, the foreground occlusions can be identified accordingly and the occluded background is also possible to find in other SAIs.

∗ corresponding author: yflin@bjtu.edu.cn

Recently, [Wang et al., 2020] developed an end-to-end learning-based framework to calculate occlusion-free central SAIs using LFs as input. However, the related results show blurry background and aliasing effects. The main problems for LF occlusion removal are: 1) Although lots of LF depth estimation methods [Tsai et al., 2020] have been proposed, it is still difficult and costly to find accurate depth information, especially for occluded points. 2) How to combine all SAIs in grid-like angular sampling to completely remove occlusions. 3) It is difficult to design one network that has large receptive fields to find all occluded points in other SAIs.

In this paper, we propose a novel end-to-end learning based framework to solve the above problems. The angular and spatial information in LFs is fully considered to simultaneously 'seek' and 'generate' occluded background regions. The LFs are first shifted to extract effective features so that different disparities are possible to estimate. We then develop the micro-lens dynamic filters for the shifted LFs to find correct background points using angular information. The features of shifted LFs are further integrated to learn surrounding spatial information to supplement more details. The occlusion-free central SAIs are finally obtained once the abundant information in LFs is fully explored.

One typical example is shown in Fig 1, in which our method is able to recover the complex scenes without any prior occlusion information. Our main contributions are summarized as follows:

• We design a novel end-to-end learning-based framework based on the shift operation for occlusion removal task, which fully explores the spatial and angular information of LFs and implicitly learns the disparity information.
• We develop the micro-lens dynamic filter, which is effective to find occluded pixels from other SAIs.
• Experiments show that our method can automatically identify and remove occlusions in specified disparities. Comparisons on both synthetic dataset and real-world dataset show that our method achieves superior performances over the state-of-the-art methods.

2 RelatedWork

In this section, we briefly overview the related single image inpainting methods and LF occlusion removal methods.

Single Image Inpainting. If foreground occlusions are accurately masked, the widely developed single image inpainting methods can be applied to recover the background regions. [Liu et al., 2018] proposed Partial Convolution by assigning the missing and non-missing pixels in the convolution operation with different weights so that the model focused more on restoring the missing areas. [Xie et al., 2019] further extended the Partial Convolution from the encoding layers to the decoding layers, in order to further improve the visual effect. [Li et al., 2020] creatively transformed the process of inpainting method into an iterative structure. However, the training of these inpainting methods is quite difficult since the models need a large amount of dataset to learn the plausible semantics. Moreover, when occluded background regions are complex, only using the learned semantic information cannot achieve reliable results.

Light Field Occlusions Removal. In order to remove occlusions in LFs, some traditional methods have been proposed. [Vaish et al., 2004] proposed a refocusing method to warp LFs in spatial domain by a special value and then average all SAIs to remove occlusions. Then, the median cost and entropy cost are further proposed in [Vaish et al., 2006] to improve their method. Because these methods can't distinguish the light from occlusion and background very well, their results are quite blurred. [Pei et al., 2013] used pixel labeling via energy minimization to solve this image blur problem. However, their method cannot get the image clear in all depth levels. Then they came up with an image matting approach in [Pei et al., 2018] to get all-in-focus images. [Yang et al., 2014] presented a depth free all-in-focus method to remove occlusions. Specifically, they divided the scene into different visible layers to directly deal with layer-wise occlusions. [Xiao et al., 2017] used the clustering method to distinguish occlusion layers and background layers. [Pendu et al., 2018] noted that there is redundancy in different views so that they removed occlusions via constructing the lowrank matrix. However, all these methods used handcrafted features and cannot achieve satisfactory results in complex background areas. [Wang et al., 2020] recently first proposed to remove foreground occlusions using LFs throughout an end-to-end network. They constructed their network with the U-shape structure and used Atrous Spatial Pyramid Pooling (ASPP) [Wang et al., 2019] to extract features from input SAIs. They also proposed to use LFs and several occlusion images to construct the training dataset and prove that the trained model can handle both synthetic and real-world occluded scenes. However, since all SAIs are simply concatenated as the input, the angular information cannot be fully extracted and the related regions cannot be well recovered.

3 Motivation

In this paper, we use L(u, v, x, y) ∈ RU×V×X×Y to represent 4D LFs, where (u, v) and (x, y) are the angular and spatial coordinates, respectively. By fixing angular coordinates (u, v), SAI Iu,v(x, y) is obtained. Similarly, by fixing spatial coordinates (x, y), the micro-lens image is represented as Mx,y(u, v). LFs occlusion removal task aims to get a clean center SAI I clean uc ,vc (x, y) from L(u, v, x, y), where uc , vc represent the center angular coordinates.

In order to find occluded pixels in other views, we follow the shift strategy in LF depth estimation [Tsai et al., 2020] to construct shifted LFs. Specifically, the shifted LFs Ld (u, v, x, y) with different disparities d are calculated:

The example shifted LFs with different d, displaying with micro-lens images, are shown in Fig 2. Following the LF digital refocusing in [Ng et al., 2005], the refocused images can be obtained by directly summing up each micro-lens image as one pixel. If we shift LFs with one disparity d∗ , the objects in disparity d∗ are in focus and the pixels in the related micro-lens image M d∗ x,y (u, v) show consistent color. However, in Fig 2, the background regions are occluded by dense and fine objects. The micro-lens image M dback x,y (u, v), which is shifted using background disparity dback, contains not only background colors but also occlusion colors. If we directly average the pixels in the micro-lens image, the result refocused image shows the background with blurry occlusions. In our method, we propose to delete occlusion points and keep background points in the shifted micro-lens images so that images without occlusion can be recovered.

Specifically, we first construct D possible shifted LFs Ld (u, v, x, y). As Fig 2, for one specific point (x, y) in central SAIs, D shifted micro-lens images M d x,y (u, v) are available. In one specific shifted micro-lens image, the occluded background pixels can be found and further extracted. Inspired by the dynamic upsampling filters in video superresolution [Jo et al., 2018], we design novel micro-lens dynamic filters for the LF occlusion removal task. The microlens dynamic filter measures the possibility of each pixel in shifted micro-lens images belonging to background points and is calculated for each point in the central SAI. Then, by summing up the weighted micro-lens images, the pixel is recovered with the correct background color.

4 The Proposed Approach

The overview architecture of our proposed method is shown in Fig 3. The shifted LFs are first fed into the Feature Extraction (FE) module HFE , which explores the latent relationship between angular domain and spatial domain. The extracted features are then separately fed into a novel Microlens Dynamic Filter (MDF) module HMDF and U-shape Reconstructed (UR) module HUR. HMDF is designed to calculate the micro-lens filters so that occluded pixels are effectively found using angular information. HUR aims to reconstruct images with more details based on spatial information.

4.1 Feature ExtractionModule

For each shifted LF, we construct one branch to extract both spatial and angular features. Specifically, we design a spatialangular block, which contains 4 parallel branches, 2 for 2D

convolution with dilation =1, 2 on spatial domain and other 2 for angular domain. The spatial and angular features are then concatenated in the end of each block. In our method, 2D spatial convolution and 4 spatial-angular blocks are built in order in each branch HFE to extract effectively features:

In order to reduce the complexity of the network, the weights are shared in different feature extraction branches.

4.2 Micro-lens Dynamic Filter Module

After feature extraction, we combine all features of different shifted LFs as FFE(d, u, v, x, y). We then design the MDF module HMDF to calculate the micro-lens filters Wx,y(d, u, v) ∈ RD×U×V . HMDF contains several angular convolution in order to fully explore the potential relationship between pixels in shifted micro-lens images. A softmax layer is then implemented on the (d, u, v) dimension to normalize Wx,y :

Wx,y(d, u, v) represents the possibilities of pixels in shifted micro-lens images belonging to the occluded background points. The higher the value, the larger possibility it has. We calculate Wx,y(d, u, v) for each point (x, y) in the central SAI.

As analyzed before, for each point (x, y) in the central SAI, D shifted micro-lens images are available, which can be expressed as Mx,y(d, u, v). By multiplying the shifted microlens images Mx,y(d, u, v) with the corresponding micro-lens filters Wx,y(d, u, v), the pixels belonging to occluded background are extracted and foreground occlusions are deleted:

FMDF indicates the calculated occlusion removal images using local angular information. ⊙ denotes the dot product.

4.3 U-shape ReconstructedModule

In MDF module, angular information from the same point of other SAIs is extracted in order to restore occluded pixels. We further introduce the UR module to integrate spatial information from adjacent regions around the occluded points for more details. Specifically, the 2D convolution is first implemented on angular domain without padding in order to reduce the angular dimension. The widely used U-Shape structures [Wang et al., 2020; Ronneberger et al., 2015] is then introduced to enlarge the receptive field and extract multi-scale spatial features. The U-shape structure includes encoding layers, decoding layers and skip-connections is constructed as shown in Fig 3. In order to effectively extract spatial features in SAIs, the 2D convolution is performed on the spatial domain. The output FUR(x, y) is then computed as:

Finally, we fuse FMDF (x, y) and FUR(x, y) using 2D spatial convolution to get the final output I clean uc ,vc (x, y).

4.4 Loss Functions

Besides the widely used L1 loss lossL1, we introduce one mask loss lossmask to make the model focus more on occlusion areas, which is calculated as:

where I gt uc ,vc denotes the ground truth occlusion-free image. Since the occlusion mask is available in the training dataset, the mask can be accordingly set, i.e 1 for occluded areas and 0 for other regions. We also introduce the perceptual loss lossvgg in [Wei et al., 2019] based on the VGG-19 Network [Simonyan and Zisserman, 2014] to achieve better visual effects. The final loss function is calculated as:

where λ1 and λ2 denote loss weights and are set as 0.5 and 0.1 in the following experiments, respectively.

5 Experiment

We first introduce the datasets and training details. The proposed method is then compared with other methods. Finally, the ablation study is taken for further evaluation.

Datasets. We follow [Wang et al., 2020] to use 61 LF images from [Honauer et al., 2016; Lanman et al., 2011; Wanner et al., 2013; Vaish and Adams, 2008] and 80 occlusion images to synthesize the training dataset. Since only 4 synthetic LFs with ground truth (4-synLFs) are used in [Wang et al., 2020] for testing, we further synthesize another LF dataset (9-synLFs) using 3D MAX and the public 3D models in [Company, 2013]. The new synthetic dataset contains 9 LFs with complex backgrounds and different foreground occlusions. We further resize the 4-synLFs with different scale factors so that images with different disparities can be further evaluated. In our experiment, the original 4-synLFs are used as the validation set and the model is finally tested on the resized 4-synLFs [Wang et al., 2020], new 9-synLFs and one real-world dataset [Wang et al., 2020].

Disparity Setting. Following the training strategy in [Wang et al., 2020], our model is also designed to remove occlusions that have positive disparities and keep background regions that have negative disparities. In this way, multiple occlusions in different disparities can be removed step by step by shifting LFs. In order to find micro-lens images in background disparities, we choose totally D shifted values d, which is evenly spaced in d ∈ {dmin, · · · , 0}. In our experiments, we choose dmin=-9 and D = 10, which is able to handle the above dataset with different disparities. Experiments with different parameters are further analyzed in Sec 5.2.

Training Details. We crop the training images into 5 × 5 × 64 × 64 pixel patches. The RGB color is put into the channel dimension in the input and output. Our model has 2.7M parameters, which is 14 times fewer than DeOccNet, 24 times fewer than LBAM, 11 times fewer than RFR. We perform upsampling, flipping, rotation, color conversion for data augmentation. The proposed model is optimized using the Adam [Kingma and Ba, 2014] algorithm with a batch size of 5. The initial learning rate is set as 1e - 3 in the first 150 epochs and is then divided by 10 every 50 epochs. We implement the model with pyTorch framework and the training process roughly takes 3 days with an Intel(R) Xeon(R) CPU E5-2683 v3 @ 2.00GHz with a Titan XP GPU. Occlusions are removed in one LF with around 10s using our model.

5.1 Comparison with State-of-the-Arts

The only learning-based LF occlusion removal network DeOccNet [Wang et al., 2020] is compared. The state-of-the-art inpainting methods LBAM [Xie et al., 2019] and RFR [Li et al., 2020] are also compared by providing the central SAI and the ground truth occlusion mask. In order to learn plausible semantics information, the inpainting methods need a large amount of images (around 15000 images) for training. LBAM [Xie et al., 2019] and RFR [Li et al., 2020] cannot achieve comparable results if only our 61 training dataset is used for training. Therefore, we choose to carefully fine-tune their pre-trained models on our training dataset.

Synthetic Scenes

We calculate the PSNR and SSIM metrics of the recovered RGB central view image for quantitative comparison. For the image inpainting methods, only occlusion regions are recovered and the other regions are directly copied from the original images. Therefore, we also replace the non-occluded areas using the original images in DeOccNet and ours, labelled with ∗, to give a further evaluation on how these methods perform on occluded areas.

Table 1 and Fig 4 show the quantitative and qualitative results of 4-synLFs and 9-synLFs. Using the occlusion masks, the related results focus on whether the occluded background regions are accurately recovered. Note that by resizing LFs in spatial domain, the disparities also change. Compared with a small disparity range, removing occlusions in a large disparity range is more difficult since we need to find more possible regions in LFs. From Table 1, we can find that the average PSNR and SSIM of ours∗ are far better than other methods on different datasets. Qualitative results in Fig 4 also show that our method can recover the complex background textures which are fully occluded in the central SAI. By comparison, other methods produce significant artifacts and wrong textures. The inpainting methods LBAM [Xie et al., 2019] and RFR [Li et al., 2020] generate background regions according to the plausible semantics. It is difficult for them to recover complex backgrounds. DeOccNet [Wang et al., 2020] is also designed to use the abundant information of LFs, but their network cannot find the occluded background since the angular information is not well explored. Moreover, the performance of DeOccNet decreases dramatically in the 9-synLFs, which indicates that their generalization ability is insufficient.

Fig 5 and Table 1 further show the comparison of our method and DeOccNet, in which no occlusion masks are provided. Different with the inpainting methods, foreground occlusions are automatically recognized and removed through the end-to-end network. Numerical results in Table 1 indicate that our results achieve more than 3 dB higher PSNR than DeOccNet. Fig 5 further shows our method produces less artifacts and less color distortion than DeOccNet.

Real-world Scenes

Fig 1 and Fig 6 show the occlusion removal results using the real-world dataset. In order to compare with the inpainting methods, we manually label the occlusion masks for them. By contrast, the results of DeOccNet and ours are directly

9-synLFs(02) PSNR/SSIM 21.5212/0.7147 26.4362/0.8460

Figure 6: Qualitative results on the real-word image with different occlusion removal tasks. The green and red line indicate the 0 disparity of the input LFs and the objects before the line are needed to remove. Our method outperforms others in different tasks.

produced without occlusion masks. As in Fig 1, our method is the only one that is able to recover the occluded trees with rich textures. In Fig 6, we further compare different tasks in which occlusions in different disparities are removed. As mentioned before, through the shift operation, positions of zero disparity can be adjusted and occlusions in specific disparities are removed. As shown, in this complex example, by shifting the zero disparity to the 'red' or 'green' lines, the objects before the line are successfully removed in our method.

This means our method successfully learns the disparity information and is able to remove occlusions that we do not need. Although DeOccNet can also recognize disparity information, the results are far less accurate than our method and have blurry background textures. Other inpainting methods cannot recover accurate details in this textured background. Note that in this example, the bottom grass regions are close to the camera and are identified as the foreground that should be removed. However, since no background regions behind the grass can be recovered, all the methods choose to blur the foreground grass.

5.2 Ablation Study

We conduct several ablation studies on the 4-synLFs with different hyper-parameters, structures and loss functions as in Table 2. For fair comparisons, the number of parameters of all these models is kept almost the same. By comparing different dmin and D, we find that for the 4-synLFs with (-12, 9) disparities, larger disparity range and more shifted numbers provide more accurate results. We then compare our model with different structures, in which the proposed HMDF is deleted or modified. As shown, without HMDF , only HUR cannot find correct occluded points. Moreover, using dynamic filters whose size in R1×U×V or RD×1×1 cannot fully use the abundant information in different shifted LFs or in different SAIs, respectively. By contrast, without HUR, the spatial information cannot be fully explored and the performance also decreases. Finally, combining the proposed mask loss and the commonly used perceptual loss, the results can be further improved by 1.4 dB in PSNR.

6 Conclusion

In this paper, we proposed a novel end-to-end learning-based method for LF occlusion removal, in which depth information is implicitly learned. The micro-lens dynamic filter is designed to remove foreground occlusions and recover background details. The angular and spatial information in LFs is fully extracted and integrated in our network. Experiments on synthetic and real-world datasets prove that our method achieves obviously higher performances than others.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (No.61902018).