Intrinsic Image Decomposition by Pursuing Reflectance Image

1 Introduction

As one of the fundamentals for problems like surface retexturing and object compositing [Bi et al., 2015], intrinsic image decomposition (IID) has been gaining great attention these years. IID aims at decomposing an image into two separate images: the reflectance image and the shading image. Following the lambertian reflection assumption, the image I is modelled as the pixel-wise product of reflectance (or albedo) R and shading S.I ≈ R ⊙ S. (1)

∗Equal contribution.

Low res. branch attention mask

High res. branch attention mask

Figure 1: Visualization of the learned attention masks. Greener means larger value, bluer means smaller value. Our method better solves the problem of intrinsic image decomposition in the wild by disentangling the learning process of the two components: the flat surfaces and the boundaries in the reflectance images.

While the above assumption is elegant, the acquisition of the groundtruth reflectance and shading images is extremely hard. As a result, MIT Intrinsic [Grosse et al., 2009], MPISintel [Butler et al., 2012], ShapeNet [Shi et al., 2017], CGIntrinsics [Li and Snavely, 2018a] propose to construct datasets with groundtruth reflectance and shading images through rendering objects or scenes with computer softwares. However, these datasets are synthetic and lack realism, thus the algorithms trained on them cannot generalize well to real-world images and limit its usefulness. Subsequently, Intrinsic Image in the Wild (IIW) [Bell et al. ], and Shading Annotations in the Wild (SAW) [Kovacs et al., 2017] dataset is proposed to collect supervision from the real-world images on reflectance and shading images, respectively. In these two datasets, humans are asked to make comparisons (darker color or shading) between two points on an images. And eventually, a sparse set of pairwise comparisons is collected for each dataset. Apparently, compared to the aforementioned synthetic datasets, this kind of data is extremely hard to collect, and thus limits the usage in the practice.

In this paper, we attempt to use only reflectance supervision for training. In this way, we only need to collect label for reflectance images (not both reflectance and shading images). In fact, in the literature, a number of works already tried to solve IID with only reflectance supervision on the IIW dataset [Liu et al., 2019; Wang and Lu, 2019]. However, they usually cannot gain satisfying results. We argue that this is because they did not fully utilize the characteristics of reflectance images. Even with both supervision (reflectance and shading) [Li and Snavely, 2018a; Zhou et al., 2019], the results are also far from satisfactory.

Learning a decomposition model with real-world images with only pairwise supervision on reflectance images is very challenging in the following two aspect. Firstly, it is hard for model to output a reasonable reflectance image with such sparse pairs of comparison groundtruth per image. Secondly, with no supervision on shading, we can only rely on the quality of output reflectance image and use Eqn.(1) to calculate the shading image.

To alleviate the above problems, we decided to pursue reflectance images to solve the problem of IID. We first take a deep investigation on the reflectance images. We observe that there are actually two components in a reflectance image: the flat surfaces with different colors, and the boundaries between these surfaces. The flat surfaces are usually an approximate constant color with low frequency information, and the boundaries are usually sparse yet full of high frequency details.

By leveraging these findings, we then propose RDNet (short for Reflectance Disentanglement Network) to disentangle the learning process of the two components. We argue that through this design, the process of IID can be better modelled. Inspired by OctConv [Chen et al., 2019], we first use a componential branch separator module to separate the networks into two branches: a low resolution branch, and a high resolution branch. We use the low resolution branch to capture the flat surfaces for learning the low frequency information, while the high resolution branch is used to capture the boundaries for learning the high frequency details. We make this design to reduce the spatial redundancy in extracting low frequency information in the low resolution branch, and to maintain the high frequency details in the high resolution branch. This disentangling process can let the networks focus on more specific tasks and thus gain better performance.

To further make the disentanglement more effective, we propose an attentional branch merger module. Two attention masks are learned to attend the spatial location of high or low resolution branches. This makes sure that each branch is actually focusing on learning the component that it is assigned. Eventually, an attention mask regularization term on the attention masks are proposed to limit the sparsity for the mask of the high resolution branch, and the smoothness for the mask of the low resolution branch. This in turn also force each branch to focus on its assigned components. In addition, our model also gains high interpretability through visualization of the attention masks, which have demonstrated the procedure of disentanglement and utilization (cf. Figure 1). In summary, our contribution is fourfold:

• We propose RDNet ‡ , to our knowledge, the first attempt to disentangle the learning process of the two components in the reflectance images: the flat surfaces, and the boundaries between these surfaces. Also, RDNet only requires reflectance supervision during training.
• Through carefully designed architecture and visualization, RDNet is equipped with high interpretability by demonstrating the procedure of disentanglement and utilization.
• Extensive experiments on the challenging real-world IIW dataset show that RDNet outperforms all current state-ofthe-art solutions by a large margin.
• On the densely labeled datasets (MIT and MPI-Sintel), RDNet can also achieve state-of-the-art results on both reflectance and shading images, when only applying supervision on the reflectance image during training.

2 RelatedWork

Intrinsic image decomposition. The methods of IID can generally be classified into two categories, optimization based and learning based methods. Optimization based methods aim at designing effective priors and attempt to solve this problem by optimizing some energy function for each image [Grosse et al., 2009; Barron and Malik, 2014; Chen and Koltun, 2013; Shen and Yeo, 2011; Zhao et al., 2012; Bi et al., 2015]. Learning based methods are becoming the mainstream now with the rapid development of convolutional neural networks [He et al., 2016; Chen et al., 2019; Sun et al., 2019]. They make use of the collected data with supervision, and let the networks learn how to decompose by themselves without the need of human tailored filters and functions [Cheng et al., 2018; Fan et al., 2018; Lettry et al., 2018; Liu et al., 2019; Narihira et al., 2015a; Narihira et al., 2015b; Nestmeyer and Gehler, 2017; Shi et al., 2017; Wang and Lu, 2019; Zhou et al., 2015; Zoran et al., 2015]. More recently, [Zhang et al., 2021; Liu and Lu, 2020; Liu et al., 2020] utilized unsupervised learning in the field of IID.

Reflectance image learning. To solve the IID problem on IIW datatset [Bell et al., 2014], many learning based methods have been proposed. Since the only supervision on the IIW dataset is the pairwise reflectance judgement on reflectance images, most works focus on how to learn and output 'good' reflectance images. A group of works [Narihira et al., 2015b; Zhou et al., 2015; Zoran et al., 2015] start by training a classifier using deep features of the two image patches, then use a globalization method to recover the full reflectance images. [Nestmeyer and Gehler, 2017] develop a CNN approach to output the dense reflectance images directly with the help of a hinge loss function. [Fan et al., 2018] propose to improve [Nestmeyer and Gehler, 2017] by adding a domain filter to the network. More recently, [Wang and Lu,

‡ Code available in https://github.com/lzhbrian/RDNet

2019] make use of the distribution and divergence of features, and achieve current state-of-the-art results. ERDIN [Liu et al., 2019] modifies an architecture borrowing from the task of super resolution and is able to get even better numbers with a post-processing filter. Researchers also achieve IID by joint training with other tasks [Baslamisli et al., 2018] and auxiliary data [Li and Snavely, 2018a; Li and Snavely, 2018b]. While most learning based methods have provided decent performance, we argue that they didn't make full use of the properties in the reflectance images.

3 Method

3.1 Preliminaries

Prior to our work, there are mainly two types of methods to solve the problem of IID.

One-headed network (Figure 2a). Many method worked on the sparsely labeled IIW datasets only focus on learning and outputing reflectance images [Liu et al., 2019; Wang and Lu, 2019; Fan et al., 2018]. The network is usually designed to only output the reflectance images. The shading images are subsequently calculated by S = I/R.

Two-headed network (Figure 2b). For densely labeled datasets, since the dataset are all synthetic, we are able to obtain the groundtruth reflectance and shading images. This kind of networks are usually designed as two-headed. The input image is sent into two separate networks for obtaining reflectance and shading images, respectively. The two networks are often supervised with separate loss functions.

Ours (Figure 2c). In this work, we follow Figure 2a, and only design a dedicated network to output reflectance images. Because the reflectance images are learned well enough, directly calculating shading images by S = I/R can also achieve decent results. In our network, supervision is only applied to the reflectance images during training. No supervision is applied to the shading images.

3.2 RDNet

We now introduce our proposed RDNet (Reflectance Disentanglement Network) for intrinsic image decomposition. The overall architecture is shown in Figure 2c, RDNet composes of three important designs, namely Componential branch separator, Attentional branch merger, and Attention mask regularization.

Componential Branch Separator

As mentioned in Section 1, we observe that there are actually two independent components in an reflectance image: the flat surfaces with different colors, and the boundaries between these surfaces. The flat surfaces component usually consists of constant colors with low frequency information, and the boundaries component consists of sparse edges with high frequency details. In order to disentangle the learning of two components, we propose to separate the input images into two branches through a componential branch separator module.

Different from traditional CNN architecture (cf. Figure 2a) used for IID, the componential branch separator additionally LQSXW

LPDJH

5HV%ON

���

5HV%ON

RXWSXW

UHÁHFWDQFH

6 , � 5

RXWSXW

VKDGLQJ

(a) Existing one-headed networks.

5HV%ON

5HV%ON

VKDGLQJ

(b) Existing two-headed networks.

Figure 2: Network architectures.

generates a low resolution branch from the original feature maps Fh ∈ RH×W×Ch through a strided 3x3 convolution. This procedure is shown in Figure 3. The new low resolution feature maps possess half the size of the original feature maps, and with a channel number of Cl. The original feature maps with higher resolution is responsible for capturing the boundaries component of the reflectance images, while the new low resolution feature maps are responsible for capturing the flat surfaces component. The design philosophy of this module is that the boundaries component is occupied by high frequency details, and needs a higher resolution feature maps to represent. On the other hand, the flat surfaces component in the reflectance images is occupied by low frequency information, and thus can be embedded in lower resolution feature maps and make the model more effective.

Attentional Branch Merger

After being separated by the componential branch separator, the two branches then go through several residual blocks [He et al., 2016] to learn a more meaningful feature representation, respectively. In the end of the two branches, an attentional branch merger is used to merge the attention enhanced features of the two branches, and then use several convolutions to output the final reflectance image. Through this process, we are able to ensure that the high and low resolution branches are actually learning the components they are assigned.

In the attentional branch merger module (cf. Figure 4), two attention masks M h and M l are generated with an instance normalization from the feature maps of the two branches, and followed by a 1x1 conv and a sigmoid function. The two attention masks are then multiplied elementwisely with their respective original feature maps, and added together after an upsample procedure apply to the low resolution output. The upsample procedure is realized by a bilinear upsampling, followed by a 1x1 conv, batch normalization and ReLU.

Attention Mask Regularization

In order to learn reasonable attention, we apply two respective regularization terms to the attention masks of high and low resolution branches. For attention mask on the high resolution branch, we use an L1 regularization term on M h to ensure the sparse characteristic of boundaries component:

Another intuitive choice is to use the gradient maps of images as the supervision signals. However, we argue that the gradient maps shall contain high frequency information brought by shading as well. We only want the ones brought by reflectance. By using L1 regularization, we are able to let the network learn by itself which high frequency information should be captured with proper constraints.

For attention mask on the low resolution branch, a total variation loss on M l is used to ensure the smoothness of flat surfaces component:

Reflectance Supervision

For sparsely labeled dataset (i.e. IIW), we use a WHDR hinge loss [Nestmeyer and Gehler, 2017] on the output reflectance image:

where R is the output reflectance image, ξ is the margin, δ is a hyperparameter quantifies the significant level of the relative difference between two points, and Ji ∈ {1, 2,E} is the ith groundtruth human judgement indicating that if point i1 is darker than (1), lighter than (2), or equal to (E) point i2.

For dense supervision datasets (i.e. MIT and MPI-Sintel), we utilize an MSE loss on the output reflectance images. Additionally, the image gradients in the x and y directions are also supervised:

where R is the output reflectance image, and ˆR is the groundtruth reflectance image.

Overall Loss Function

Finally, the overall loss function can be written as:

where λmaskl and λmaskh is the weight of the regularization terms. Note that our network only outputs the reflectance images and the loss function is only applied on them during training. The shading images are obtained by S = I/R, and no supervision is used.

3.3 Discussion

Here, we summarize some desirable properties of RDNet. Firstly, with the componential branch separator, we disentangle the learning process of the two components in reflectance images: the flat surfaces and the boundaries. By using feature maps with different resolutions, we successfully disentangle the components with distinct frequency information. Secondly, the attentional branch merger can learn the spatial attention of each branch. Through focusing on important spatial location for different components, each of them can be better modelled. Thirdly, the attention mask regularization term can limit the sparsity of the high resolution branch attention masks, and the smoothness of the low resolution branch attention masks. This poses a strong prior on the information to be learned, and thus makes the disentanglement and utilization much more effective. Last but not least, we are able to visualize the learned attention masks and reveal the internal mechanism of the model. This makes the model equipped with strong interpretability and alleviate this undesirable property of deep neural networks.

4 Experiments

In what follows, we design and conduct extensive experiments to answer the following four research questions.

• RQ1: How does our RDNet compare with other stateof-the-art IID methods on the challenging real-world IIW dataset? (Sec.4.2)
• RQ2: How does our RDNet perform on the densely labeled MPI-Sintel and MIT datasets? (Sec.4.2)
• RQ3: How does each module designed contribute to the improvement of RDNet? (Sec.4.3)
• RQ4: How does our RDNet disentangle the two components of reflectance images? (Sec.4.3)

4.1 Datasets andMetrics

Sparsely Labeled Dataset

IIWdataset. The real-world IIWdataset [Bell et al., 2014] contains 872,161 pairwise reflectance comparisons across 5,230 photos. We follow the same settings as [Fan et al., 2018]. For evaluation, the performance is measured on the reflectance images with the Weighted Human Disagreement Rate (WHDR). Since the pairwise judgement of human labels might be subjective, we note that humans actually have a median WHDR of 7.5% across all photos in IIW.

Densely Labeled Datasets

MPI-Sintel dataset. The MPI-Sintel dataset [Butler et al., 2012] contains 8950 images from 18 scene level computer generated images sequences. We follow the same settings as [Cheng et al., 2018]. The performance is measured on siMSE, si-LMSE, and DSSIM.

MIT dataset. The MIT dataset [Grosse et al., 2009] contains 20 object level images, each with 11 different lighting conditions. We follow the same settings as [Cheng et al., 2018]. The performance is measured on si-MSE.

4.2 Qualitative and Quantitative Results

Sparsely Labeled Dataset (RQ1)

Table 1 presents the quantitative results of our proposed method compared with other methods on the IIW dataset. We can see that our method outperforms the current state-of-thearts of [Wang and Lu, 2019] by a large margin. Note that, without a post-processing filter, we already outperform [Liu et al., 2019], which is the current best number with a postprocessing filter. If we further apply a bilateral filter to our output reflectance images, we can further lower our WHDR score to 13.19, which sets the new record for the IIW dataset.

We then compare some qualitative results in Figure 5. [Bi et al., 2015] is the best optimization based method, and [Wang and Lu, 2019] is the current learning based state-ofthe-art method. Firstly, we have observed much clearer and sharper reflectance images compared to [Wang and Lu, 2019], and more accurate reflectance images compared to [Bi et al., 2015]. Notice the picture frame on the wall in the first image, the leftmost person's blue top near the waist in the second image, and the sundries on the cabinet of the right side in the third image. Secondly, we are also able to obtain much more accurate shading images. Notice the floor and sofa in the first image, wall and table in the second image, and the ceramic tiles on the right side and floor in the third image. Other methods usually recognize textures in the above objects as shading. Note that we do not have any supervision on shading during training. But with better modelling for reflectance images, the quality of shading images can also be improved considerably.

Densely Labeled Datasets (RQ2)

The performance of previous methods on densely labeled datasets is already very promising. We still conduct experiments on these datasets for two main reasons. Firstly, we want to confirm that our network architectures are suitable for most IID datasets. Secondly, different from existing methods, our network only output and apply supervision on the reflectance images. No supervision on the shading images is used. We want to know whether this setting is sufficient for networks to learn well. From Table 2 and Table 3, we can see that our proposed method outperforms all baselines by a large margin in all metrics on both MIT Intrinsic dataset and MPI-Sintel dataset. Note that all of our baselines use shading images as supervision, while we only use reflectance images as supervision.

4.3 Analysis

Ablation Study (RQ3)

To further quantify the contribution of each design of our method, we conduct some ablation study in Table 4. We start by implementing a simple network architecture as in Figure 2a. The network contains 10 residual blocks, and is trained with a WHDR hinge loss (Eqn.4) on the IIW dataset. This architecture is basically the same as the direct intrinsic network in [Fan et al., 2018]. Surprisingly, we find that with appropriate training schedule, this network is able to obtain WHDR=14.40, which is already sufficient to outperform most existing baselines. As a reminder, [Fan et al., 2018] only achieves WHDR=15.40 with the same settings. After adding the componential branch separator module, and use a simple sum operation to merge the output feature maps of two branches, we can get WHDR=13.84, which is already the state-of-the-art result. This justifies the effectiveness of the disentanglement. Then, adding the attentional branch merger, and the attention mask regularization can further get a decrease of 0.1 and 0.09, respectively. By making the network deeper (adding more residual blocks), we continue to lower the score to 13.47.

Disentangling Reflectance Image Components (RQ4)

Finally, for better understanding the internal process of networks, we visualize the attention masks learned in Figure 1. We can clearly observe that the high resolution branch attention mask is paying attention to the boundaries, and the low resolution branch attention mask is focusing on the flat surfaces. This justifies that our network has disentangled the two components as we expected.

5 Conclusion

In this paper, we investigate the possibilities of only using reflectance images for supervision when solving the problem of intrinsic image decomposition. In such way, the demand for labeled data can be greatly reduced. We design RDNet to disentangle the learning process of two components of the reflectance images: the flat surfaces and the boundaries between them. RDNet is equipped with strong interpretability through demonstration of the disentanglement and utilization process. We have shed the light on the internal process of IID through pursuing the characteristics of reflectance images. Extensive experiments have shown that with only reflectance supervision, RDNet is also able to achieve state-ofthe-art results on all datasets.

Acknowledgements

This work was supported by MOST-2018AAA0102004, NSFC-62061136001.