Non-Lambertian Multispectral Photometric Stereo via Spectral Reflectance Decomposition

1 Introduction

Photometric stereo methods, originally proposed by Woodham [Woodham, 1980] and Silver [Silver, 1980], recover detailed three-dimensional (3D) surfaces from images captured from a fixed camera under varying lighting directions, which are commonly obtained at different timestamps. Since conventional photometric stereo (CPS) methods stack images with time-multiplexing, the target surface has to be kept static during the multiple shots. With spectral-multiplexing, multispectral photometric stereo (MPS) [Kontsevich et al., 1994] can recover surface normals from a one-shot multispectral image. As shown in Fig. 1, a single multispectral image for MPS encodes observations under varying lighting directions in different spectral bands, conveying the information about surface normals at that timestamp. With input multispectral images recorded by a multispectral camera under spectral light sources, MPS can reconstruct the 3D shapes of dynamic objects 1 .

∗ Corresponding Author

Existing MPS methods [Guo et al., 2021; Chakrabarti and Sunkavalli, 2016; Ozawa et al., 2018; Anderson et al., 2011b] mostly focus on the Lambertian reflectance, which has limited ability to represent the surface appearance in realworld. For non-Lambertian surfaces, the spectral reflectance varies not only with the spectral band, but also the incidentoutgoing lighting directions w.r.t. the surface normal at different scene points. There have been a few MPS methods

specific hardware setup [Rahman et al., 2014] with two-shot capturing or an input image containing three spectral channels with fixed spectral bands [Ju et al., 2018; Ju et al., 2020a; Ju et al., 2020b]. Due to the restrictive capture setting and limited surface normal estimation accuracy of existing methods, non-Lambertian MPS remains an open and challenging problem. Alternatively, non-Lambertian reflectance is well-studied in the CPS context and great progress has been achieved with data-driven approaches [Chen et al., 2018; Santo et al., 2017; Yao et al., 2020; Ikehata, 2018]. However, non-Lambertian CPS methods cannot be directly applied to multispectral image observations without considering the wavelength-dependent reflectance responses.

1 Please check the supplementary video for the full sequence.

In this paper, we show that MPS can be reformulated as a CPS problem by introducing a spectral reflectance decomposition (SRD) model, which allows us to borrow experiences from well-established CPS theory. Specifically, we assume the target surface has dominant diffuse or speculative reflectance, which is commonly seen in realworld materials (e.g. measured material "paper blue" and "ilm l3 37 metallic" in Fig. 3). Under this assumption, our SRD model can decompose the non-Lambertian spectral reflectance into two independent components: geometric and spectral. The geometric component only varies with the incident-outgoing lighting directions, which can be wellmodeled by non-Lambertian CPS methods. The spectral component is the response w.r.t. the spectral wavelength. For surfaces covered by a uniform material, we can entangle it with the input light intensity as an equivalent light intensity. In this way, the MPS problem can be formulated as an semicalibrated CPS problem under unknown equivalent light intensities. Following the above insights, we propose a neural network NeuralMPS to first predict the equivalent light intensity including the spectral component, and then embed existing CPS methods to recover the surface normal map considering the geometric component only.

To summarize, our contributions are as follows:

• We formulate the MPS problem as a well-studied CPS problem with unknown equivalent light intensity by introducing an SRD model.
• We propose a learning-based MPS network NeuralMPS to accurately recover surface normal under diffuse- or specular- dominant non-Lambertian reflectance.

2 RelatedWorks

The surface general reflectance can be modeled by the Bidirectional Reflectance Distribution Function (BRDF) [Szeliski, 2010], which describes how much light at each wavelength arriving at an incident direction is emitted in a reflected direction. With a fixed view direction, the isotropic spectral BRDF is a function R(n, l, λ) of the surface normal n, the lighting direction l and the wavelength λ. Therefore, the BRDF response w.r.t. the three variables can be recorded as a cube, as shown in Fig. 2. If we apply the BRDF response to a surface covered by uniform material and ignore the crosstalk, the surface spectral reflectance can be recorded as the same cube. In this section, we review CPS and MPS approaches under Lambertian reflectance and non-Lambertian reflectance, respectively. These four categories are based on different simplifications on the spectral BRDF cube. Among all the categories, we assume distant lights with calibrated directions.

Lambertian, CPS CPS under Lambertian reflectance assumption is the most classic setup. In such case, the spectral reflectance under varying lighting directions and wavelengths is a constant value, i.e., R(n, l, λ) = c (see Fig. 2 (a)). Without loss of generality, we set c = 1. Given image measurements under more than 3 non-coplanar lighting directions, classical photometric stereo works [Woodham, 1980; Silver, 1980] provided a closed-form solution to surface normal estimation.

Non-Lambertian, CPS The spectral reflectance for nonLambertian CPS is varying with the incident-outgoing lighting directions (geometric component), but omits the variation w.r.t the wavelength (spectral component). Therefore, the spectral BRDF R(n, l, λ) = R(n, l) and the spectral cube shown in Fig. 2 (b) is a Kronecker product of an all-in-one spectral response vector and the geometric response matrix. Existing non-Lambertian CPS methods applied parametric-based [Chen et al., 2019b; Shi et al., 2014] or dictionary-based reflectance model [Enomoto et al., 2020; Hui and Sankaranarayanan, 2016] to represent the geometric component R(n, l). Recently, neural-based photometric stereo methods [Chen et al., 2018; Ikehata, 2018; Yao et al., 2020] achieved high accuracy on surface normal estimation under non-Lambertian reflectance, where the geometric component is learned from data with diverse reflectances. Please refer to the two surveys [Shi et al., 2019; Ju et al., 2022] for comprehensive reviews of non-Lambertian CPS methods.

Lambertian, MPS In MPS, the spectral component has to be taken into account. The Lambertian spectral BRDF keeps constant w.r.t. geometric component but varies over different wavelengths, i.e., R(n, l, λ) = R(λ). Therefore, the spectral cube shown in Fig. 2 (c) is a Kronecker product of a spectral response vector and an all-in-one geometric response matrix. Different from Lambertian CPS, MPS is ill-posed even under Lambertian reflectance. Existing methods require additional shape priors [Anderson et al., 2011b; Anderson et al., 2011a] or reflectance clustering [Ozawa et al., 2018; Chakrabarti and Sunkavalli, 2016] to solve the problem. Recently, Guo et al. [Guo et al., 2021] formulated the Lambertian MPS problem into a well-posed one and provided a unique solution for surface normals with image cues only.

Non-Lambertian, MPS Non-Lambertian MPS problem is the most challenging case among the four categories, as the spectral BRDF R(n, l, λ) is related to both wavelength and incident-outgoing lighting directions. Therefore, the spectral BRDF is recorded as a general cube (see Fig. 2 (d)). Existing non-Lambertian MPS methods assumed analytical BRDF model and two-shot data capturing [Rahman et al., 2014]. However, the analytical BRDF model is limited to a small set of materials, and the two-shot data capturing eliminates the advantage of MPS over CPS on dynamic shape recovery. Ju et al. [Ju et al., 2020a; Ju et al., 2020b] propose learningbased methods to solve non-Lambertian reflectance. However, the spectral band of the input multispectral images in the test phase is required to keep fixed in the training stage, which limits their application to real-world scenarios. Our method regards the spectral BRDF cube as a Kronecker product of the spectral response vector (spectral component) and geometric response matrix (geometric component). As we can see in the following sections, such a decomposition enables the application of off-the-shell non-Lambertian CPS methods on the non-Lambertian MPS task.

3 Spectral Reflectance DecompositionModel

In the context of MPS, the material spectral reflectance varies with both the incident-outgoing lighting directions and spectral bands. Under a fixed view direction, it can be formulated as a function of surface normal n, lighting direction l, and wavelength λ. Following the dichromatic model proposed in [Shafer, 1985], we represent the spectral reflectance as

where Rg∗ and Rs∗ represent the geometric component and the spectral component, the subscript d and s denote diffuse and specular reflectance respectively. We assume the non-Lambertian target surfaces are with dominant diffuse (i.e., R(n, l, λ) = Rgd(n, l)Rsd(λ)) or specular (i.e., R(n, l, λ) = Rgs(n, l)Rss(λ)) reflectance. In this way, we introduce our SRD model that disentangles the nonLambertian spectral reflectance into only two components:

where the spectral component Rs(λ) is related to the wavelength, and the geometric component Rg(n, l) is related to the variation w.r.t. surface normal and lighting direction.

We illustrate and verify our SRD model by measured spectral BRDFs [Dupuy and Jakob, 2018]. As shown in Fig. 3, we plot measured spectral reflectances of two materials: "paper blue" and "ilm l3 37 metallic" in the [360nm, 1000nm] wavelength range, where curves in different colors show the spectral reflectance at varying lighting direction and surface normal pairs: (l, n). If the surface follows Lambertian reflectance [Anderson et al., 2011b; Esteban et al., 2011], the spectral reflectance shown in the figure should contain one single curve, which is not flexible to represent the realworld spectral reflectances. To verify the effectiveness of our SRD model, we sample 195 wavelengths and 200 × 200 (l, n) pairs to build an spectral reflectance response matrix R ∈ R195,40000 . We achieve SRD by conducting SVD on R such that

where rs ∈ R195 and rg ∈ R40000 are the decomposed spectral and geometric components in the vector form obtained from the first column of U and V, respectively. We reshape the rg into matrix form and draw the two components in Fig. 3. To measure the SRD accuracy, we define the fitting energy as the ratio of the largest singular value over the sum of all the singular values. The energy over all the 51 materials is 80.1% for the average, 81.6% for the median, and 96.1% for the maximum, which reveals that we can approximate the spectral reflectance in high accuracy with the decomposed rs and rg. We also reconstruct the spectral reflectance response matrix R from the decomposed spectral and geometric parts. The reconstructed spectral reflectance curves shown in Fig. 3 are shown to be close to the originally measured one.

3.1 SRD-based Image FormationModel

In this section, we show that the MPS problem is equivalent to the CPS under unknown light intensity based on our proposed SRD model.

Given an orthogonal multispectral camera with a linear radiometric response and f spectral directional lights with calibrated lighting directions, we turn on all the spectral light sources and capture a single multispectral image. In this way, the f spectral bands of the multispectral image record measurements under varying lighting directions. Considering a non-Lambertian surface with general isotropic spectral BRDF, the image observation for one scene point under the j-th incoming lighting can be written as follows:

where n ∈ S2 ⊂ R3 represents the unit surface normal vector, lj ∈ S2 ⊂ R3 and ej ∈ R+ are lighting direction and radiance of the j-th light source, respectively, with its spectra defined by Ej(λ) : R+ → R+. The camera spectral sensitivity at the j-th spectral band is represented by Cj(λ), and the attach shadow is modeled by max(·, 0).

Similar to previous MPS works [Guo et al., 2021; Chakrabarti and Sunkavalli, 2016; Ozawa et al., 2018], we assume the crosstalk between spectral bands is negligible, i.e., observations under each spectral light can only be observed in its corresponding camera channel. With negligible crosstalk and the proposed SRD model shown in Eq. (2), the image observation can be re-written as

and e′j is defined by

where Ωj is the j-th spectral band of the corresponding spectral light and camera channel. The camera spectral sensitivity, light spectra, material spectral component, and the light intensity are encoded in e′j . If the material spectral component is uniform for the whole surface (e.g. monochromatic material), e′j keeps constant for all the surface points. Therefore, we name e′j as the equivalent light intensity. According to the image formation model shown in Eq. (5), given the image measurement m and its corresponding lighting direction l, the non-Lambertian MPS task can be reformulated as estimating the surface normal n under unknown geometric reflectance Rg(n, l) and the equivalent light intensity e′j . In this way, non-Lambertian MPS is equivalent to the nonLambertian CPS problem with unknown light intensities.

4 Learning-based Non-LambertianMPS

Guided by the conclusion in the last section, we present the NeuralMPS to solve MPS under non-Lambertian reflectances. As shown in Fig. 4, our NeuralMPS contains two modules with functionalities indicated by their names: Equivalent Light Intensity Estimation Network (ELIE-Net) and Surface Normal Estimation (SNE) Module.

ELIE-Net ELIE-Net aims at predicting the equivalent light intensities from observed multispectral images and the calibrated lighting directions. As shown in Fig. 4, image observation at each spectral channel, the corresponding calibrated spectral lighting direction, and the object mask are fed into a shared encoder separately for extracting the local feature maps. The local features of different spectral observations are then pooled into a global one by a max-pooling layer. From a shared decoder fed with concatenated local and global features, the equivalent light intensity for each image observation is regressed. To supervise the ELIE-Net, we minimize the ℓ2 distance with the ground-truth equivalent lighting intensity extracted from the measured spectral BRDFs:

where s and ˆs correspond to the GT and predicted equivalent lighting intensity.

SNEmodule With estimated equivalent lighting intensities, we then normalize the input multispectral images at the corresponding spectral bands to remove the influence of spectral components in MPS. Given these normalized image observations (I′ in Fig. 4) as well as the calibrated lighting directions, our SNE module recovers the surface normal map by considering the geometric component only. As discussed in Sec. 3, any existing non-Lambertian CPS methods can be used as our SNE module. In this paper, we choose a learning-based CPS method PS-FCN [Chen et al., 2018] as our SNE module. For better fitting the MPS image observations, the PS-FCN as the SNE module are re-trained on our synthetic multispectral dataset "PS-Spectral", as we will show in the next. Following the training strategy of PS-FCN [Chen et al., 2018], the loss function is defined as the cosine similarity between the estimated and the ground-truth surface normals:

where p is the number of valid pixels in the object mask, ni and ˆni denote the ground-truth and estimated surface normal vectors at the i-th pixel.

Difference from SDPS-Net Our method shares a similar pipeline with the SDPS-Net [Chen et al., 2019a]. Here we elaborate on the differences between our work and SDPSNet [Chen et al., 2019a]. Firstly, SDPS [Chen et al., 2019a] focuses on the non-Lambertian uncalibrated CPS problem, which has a completely different image formation from the non-Lambertian MPS problem that we are working on. Specifically, the LC-Net of the SDPS-Net [Chen et al., 2019a] learns to estimate the lighting direction and light source irradiance in the uncalibrated CPS, where no assumption is made on the non-Lambertian BRDF. However, the ELIE-Net of our method has calibrated lighting directions as input and focuses on predicting the equivalent light intensity, which encodes not only the light source irradiance but also the integral of camera spectral sensitivity, light spectra, and the material spectral component. In addition, our method for the MPS task relies on the proposed SRD assumption for non-Lambertian spectral BRDF, which ease the difficulties of the non-Lambertian MPS problem by the well-studied CPS problem.

Secondly, LC-Net [Chen et al., 2019a] recasts the regression-based problems of lighting direction and intensity into a classification-based one, where the continuous lighting space is discretized into pre-defined bins and classes. However, we found equivalent light intensity has a high dynamic range (HDR) 2 , as it encodes more than the light source irradiance. The discretization in LC-Net [Chen et al., 2019a] is sub-optimal for accurately recovering the equivalent lighting intensity. Therefore, we predict equivalent lighting intensity in our MPS task in a regression manner.

"PS-Spectral" Dataset To train our NeuralMPS, we build a synthetic training dataset with shapes coming from the Blobby dataset [Johnson and Adelson, 2011] as well as the Sculpture dataset [Wiles and Zisserman, 2017]. Each shape in the dataset is rendered with 51 measured isotropic spectral BRDFs [Dupuy and Jakob, 2018].

To build the dataset, we first render a multispectral image tensor and its corresponding spectral reflectance tensor with the size of [f, p, t], where f, p, t denote the number of light directions, pixels in the image, and spectral bands, respectively. In the MPS setting, each lighting direction corresponds to one spectral band. Therefore, we randomly select f out of t spectral bands to produce one multispectral image data with the size of [f, p, f]. In our dataset setting, we select f = 39 lighting directions distributed uniformly, and t = 195 spectral bands following the same wavelength range of [Dupuy and Jakob, 2018]. The spatial resolution of the image is set as 128 × 128. To generate the ground-truth equivalent lighting intensity, we conduct the SVD decomposition on the spectral reflectance tensor R with our SRD model following Eq. (3).

We also render a test dataset including two shapes: SPHERE and BUNNY. An example of the rendered multispectral images in our test dataset can be found in the supplementary material.

5 Experiments

In this section, we evaluate our method on both synthetic and real data. In Sec. 5.1, the accuracy of the proposed SRD model is verified on measured spectral BRDF database [Dupuy and Jakob, 2018] firstly. Then, in Sec. 5.2 we compare our NeuralMPS with the state-of-the-art MPS method GO21 [Guo et al., 2021] on surface normal estimation. The ablation study with and without the ELIE-Net is also provided to evaluate the effectiveness of predicting equivalent light intensity. Finally, in Sec. 5.3 we give the qualitative evaluation on real-captured data to verify our application on real scenarios.

5.1 Evaluation of the SRDModel

Our SRDmodel decomposes the spectral reflectance into the geometric and spectral components via SVD decomposition. We verify the accuracy of the SRD model on diverse measured spectral BRDFs [Dupuy and Jakob, 2018]. Specifically, we choose a sphere shape containing diverse surface normals and render image observations under a set of lighting directions following uniform distribution. We conduct this rendering process on 51 spectral BRDFs [Dupuy and Jakob, 2018].

As mentioned in Sec. 4, the spectral BRDF can be concentrated as a matrix R with shape of [f, p, t]. With our SRD model, R is decomposed into geometric component rg of shape [f × p] and spectral component rs of shape [t] via SVD in Eq. (3). To measure reconstruction error Er of our SRD model, we reconstruct spectral BRDF R′ from the spectral and geometric components, and calculate the Frobenius norm of the difference between the ground-truth spectral reflectance and the reconstructed ones, i.e.,

2 Please find more details in the supplementary material.

To show the effectiveness of our SRD model on nonLambertian spectral reflectance, we also evaluate the approximation error of Lambertian assumption for the spectral BRDF. Specifically, we set the geometric component as an all-in-one vector, i.e., rg = 1, and find the spectral component rs in the Lambertian case that best fit rg and R via least-square. After that, we also reconstruct the reflectance matrix from the geometric and spectral components for the Lambertian case.

As shown in Fig. 5 (top), the reconstructed spectral reflectance from our SRD model is more accurate than the Lambertian assumption used in GO21 [Guo et al., 2021], revealing the effectiveness of our SRD model for general nonLambertian spectral reflectance. Besides, we found that the reconstruction errors with t = 12 bands as well as t = 195 bands have little difference. Therefore, our SRD model can be applied to real-world multispectral cameras with limited spectral bands.

From Fig. 5, we observed that our SRD model works well for most materials. However, there are materials (e.g., "cc ibiza sunset") showing mixed specular and diffuse reflectance. The spectral BRDFs for these materials violate our SRD model that assumes the reflectance is either diffuse- or specular- dominant, like materials "ilm l3 37 metallic" and "paper blue". During the training of our NeuralMPS, we drop the last 9 materials whose reconstruction errors are larger than

0.05, which are treated as outliers of our SRD model. However, we keep the 9 materials in the test phase for a fair comparison and present the normal estimation results of all the 51 spectral BRDFs in the next section.

5.2 Evaluation of NeuralMPS

As shown in Fig. 5 (bottom), we evaluate the normal estimation error on the SPHERE test dataset. The state-of-theart MPS method GO21 [Guo et al., 2021] is based on the assumption of Lambertian reflectance, therefore the surface normal estimation has relatively high accuracy on diffuse materials such as the "paper blue". However, for materials with specular reflectance, GO21 [Guo et al., 2021] cannot produce reasonable surface normal recovery, such as the case of "ilm l3 37 metallic" and "cc ibiza sunset". Benefited from our SRD model and designed network for non-Lambertian MPS, the surface normal estimation of our method is more accurate compared with the state-of-the-art MPS method GO21 [Guo et al., 2021] on diverse materials, demonstrating the strength of our MPS method on general spectral reflectance. On the other hand, we found the normal estimations for the last 9 materials have relatively higher error, possibly due to the relative lower accuracy of the SRD model in the corresponding spectral BRDFs.

Ablation study of ELIE-Net As shown in Table 1, to make an ablation study of ELIE-Net, we evaluate the surface normal estimation error of our NeuralMPS with and without the module on test dataset SPHERE and BUNNY. Specifically, "w/o" or "w/" in the first column refer to the results recovered from SNE module where the input images are normalized by a all-one vector or the estimated equivalent light intensities from "ELIE-Net". Based on the mean angular error of the surface normal estimates over the test dataset, the absence of equivalent light intensities leads to a performance drop in normal estimation (by 11% and 2% for SPHERE and BUNNY). Therefore, the CPS method used in our SNE module ("w/o ELIE-Net") cannot be directly used for solving the MPS task. It is necessary for non-Lambertian MPS to predict the equivalent lighting intensity with our proposed ELIE-Net.

It's worth noticing that the ELIE-Net and SNE module are trained separately. Therefore the ELIE-Net can be directly applied to SNE modules that adopt other conventional photometric stereo (CPS) methods for solving multispectral photometric stereo (MPS), without the influence of a specific network structure. This setting would be beneficial for nonlearning-based CPS works. On the other hand, the proposed NeuralMPS can also be trained end-to-end when the SNE module adopts learning-based CPS methods. We have our NeuralMPS which is trained in this way tested on test dataset SPHERE and BUNNY, and the mean angular error in degree is 6.22 for SPHERE and 9.10 for BUNNY.

5.3 Experiments on Real Data

In Fig. 6, we test our method on real multispectral images of non-Lambertian surfaces released by GO21 [Guo et al., 2021]. Taking the spectral images and calibrated lighting directions as input, we compare our NeuralMPS with existing MPS methods CS16 [Chakrabarti and Sunkavalli, 2016] and GO21 [Guo et al., 2021] on surface normal recovery. As CS16 [Chakrabarti and Sunkavalli, 2016] takes only threechannel RGB images as input, we select the image observations with the wavelength 450, 550, and 650 [nm] to mimic the RGB input. For GO21 [Guo et al., 2021] and our method, we test the methods on multispectral images with 12 and 4 spectral bands (labeled as f4 and f12) to evaluate the application on real-world multispectral cameras with varying number of spectral channels. Since the ground-truth surface normal and shape are not available in the real-world MPS dataset, we conduct a qualitative evaluation by comparing the integrated shapes from the estimated surface normal maps using the normal integration method [Xie et al., 2014].

As shown in Fig. 6, we present the estimated surface normals from existing methods and ours. Since CS16 [Chakrabarti and Sunkavalli, 2016] mainly focuses on Lambertian reflectance, the recovered shape is distorted due to the specular highlights, as shown in the closed-up views. GO21 [Guo et al., 2021] is adapted to arbitrarily many spectral bands as the input and removes the specular highlights and shadows as outliers with the position thresholding strategy [Shi et al., 2019]. Therefore, the shape recoveries from GO21 (f12) are reasonable taking 12 spectral bands as input. However, the method fails with limited spectral input (GO21 (f4)), as the position thresholding strategy prefers more data for the outlier removal. Compared with existing methods, our NeuralMPS achieves reasonable results for all the three objects. There are no distortions caused by specular highlights. Since our method learns the nonLambertian spectral reflectance from the measured spectral BRDF dataset, rather than treating the specular highlights as outliers, the recovered surface normal remains in good quality even with limited spectral image inputs, as shown in Ours (f4).

6 Conclusion

In this paper, we propose a multispectral photometric stereo method NeuralMPS for surface normal recovery under diffuse- or specular- dominant non-Lambertian spectral reflectance. The spectral reflectance decomposition model is the key of our method, which represents spectral BRDF as a composition of geometry components and spectral components. In this way, the non-Lambertian MPS problem can be recast into the well-studied non-Lambertian CPS problem with unknown equivalent light intensity in which the spectral component is embedded. Our NeuralMPS therefore design the ELIE-Net to estimate the equivalent light intensity and apply the SNE module to further achieve surface normal recovery under non-Lambertian spectral reflectance.

Acknowledgements

This work was supported by the National Natural Science Foundation of China (Grant No. 62088102, 62136001, 62202409), the China Postdoctoral Science Foundation (Grant No. 2022M720236), and the JSPS KAKENHI (Grant No. JP23H05491).

Contribution Statement

Jipeng Lv and Heng Guo contributed equally to this work.