Efficient Multi-View Inverse Rendering Using a Hybrid Differentiable Rendering Method

1 Introduction

Inverse rendering, which recovers 3D geometry, reflection properties and illumination from 2D images, is a long-standing and challenging problem in computer graphics and vision [Patow and Pueyo, 2003]. Benefiting from the advances in deep neural networks, many deep learning-based methods learn to obtain the materials and plane normals of near-flat objects in a data-driven way [Li et al., 2018b; Aittala et al., 2016; Gao et al., 2019]. However, these methods struggle to deal with complex geometry, resulting in limited application in practice. Some other deep learning-based methods use 3D geometric representations such as signed distance fields [Yariv et al., 2020; Zhang et al., 2021b], tetrahedral meshes [Munkberg et al., 2021], and occupancy functions [Mildenhall et al., 2020; Zhang et al., 2021c; Boss et al., 2021; Srinivasan et al., 2021;

∗Corresponding Author

Zhang et al., 2021a; Zhang et al., 2022], to handle more complex geometries. However, such geometry representations may require post-processing in order to be used in traditional graphics pipelines [Lorensen and Cline, 1987; Remelli et al., 2020], which may cause material information loss and affect the rendering image quality.

The emergence of differentiable rendering techniques [Kato et al., 2020] means that image loss can be back-propagated along the rendering pipeline to solve inverse rendering problems, which promotes the development of multi-view inverse rendering tasks using triangular meshes as the geometry representation. Approximate differentiable rendering methods [Loper and Black, 2014; Kato et al., 2018; Liu et al., 2019; Chen et al., 2019] utilize simplified rendering processes and can solve the inverse rendering problem efficiently, but the simple material models they use usually lead to poor visual effects. On the contrary, physicallybased differentiable rendering methods [Li et al., 2018a; Li et al., 2021] can reconstruct high-quality physically-based rendering (PBR) materials in the path tracing manner, but their realistic results come at a high computational cost. Although some methods have been proposed to accelerate the reconstruction process [Luan et al., 2021], they often impose strong assumptions on the lighting conditions to narrow down the parameter search space, which hinders their wide application.

In this paper, we propose a hybrid differentiable rendering method to reconstruct triangular meshes and PBR materials from multi-view real-world images captured by conventional hand-held cameras (e.g. mobile phones) with uncontrolled lighting conditions. Our method consists of two phases. In the initialization phase, we use traditional methods to reconstruct a rough triangular mesh for the scene. Then in the optimization phase, we take a hybrid approach to optimize the scene geometry and PBR materials, where we first optimize the geometry using an approximate method, followed by the PBR materials using a physically-based method. Our novel formulation benefits from both the efficiency of approximate methods and the high quality of physically-based methods. Extensive experiments show that our method achieves a significantly faster training and rendering speed than state-of-the-art methods, while achieving results of comparable or better quality.

In summary, our contributions include:

• We propose a novel hybrid differentiable rendering optimization method based on triangular meshes, which utilizes an

approximate differentiable rendering method to optimize the geometry, and a physically-based differentiable rendering method to get the PBR materials.

• The proposed pipeline is user-friendly and can work end-toend. Furthermore, the optimized scene parameters can be easily deployed on mainstream commercial graphics engines without the need for conversion, making it applicable to a wide range of applications such as virtual reality and augmented reality.
• We conduct extensive experiments on both synthetic and real-world data, which demonstrate that our method outperforms or achieves comparable results to the state-of-the-art methods while being significantly more efficient.

2 RelatedWork

Shape Reconstruction. Reconstructing object geometry is a long-standing problem in computer vision and graphics. In traditional methods, the Structure-from-Motion (SfM) method [Schonberger and Frahm, 2016] is first applied to generate sparse feature points and find their correspondence to further generate a sparse point cloud and a rough mesh. Then, the Multi-View-Stereo (MVS) method [Schönberger et al., 2016] is leveraged to generate dense pixel-level matching. Finally, a dense mesh with vertex color is generated by the Poisson reconstruction method [Kazhdan and Hoppe, 2013]. A few recent learning-based methods assume that the target object mesh is homeomorphic with the sphere. The method in [Wang et al., 2018] uses an image feature network (2D CNN) to extract perceptual features from the input image, and a cascaded mesh deformation network (GCN) to progressively deform an ellipsoid mesh into the desired 3D model. These methods can only reconstruct rough geometry, which is insufficient for downstream tasks such as VR and AR.

Reflectance Reconstruction. Reflectance model, i.e., spatially-varying bidirectional reflectance distribution function (SVBRDF), describes how light interacts with surfaces in the scene. Traditional SVBRDF reconstruction methods [Matusik, 2003; Lensch et al., 2003; Holroyd et al., 2010; Dong et al., 2010; Chen et al., 2014; Dong et al., 2015; Kang et al., 2018] rely on dense input images measured using auxiliary equipment, e.g., gantry. Some other works focus on

[Zhou et al., 2016; Kim et al., 2017; Park et al., 2018] exploiting the structure of SVBRDF parameter spaces to reduce the requirement for the number of input images. Additionally, some data-driven works [Li et al., 2018b; Aittala et al., 2016; Gao et al., 2019] have been introduced recently to produce plausible SVBRDF estimations for near-flat objects using a small number of input images. Despite their ease of use, these methods struggle to handle more complex objects.

Differentiable Rendering. Differentiable methdos are reviewed in detail in [Kato et al., 2020]. Here we focus on methods closely related to our work. Traditional rendering pipelines, e.g., rasterization and ray-tracing, are not differentiable due to some discrete parts, which means that they cannot work with the gradient descent method just like neural networks. The emergence of differentiable rendering has changed all. Some approximately-based methods have been proposed recently. [Loper and Black, 2014; Kato et al., 2018; Kato and Harada, 2019] compute approximated gradients to optimize scene parameters. Besides, [Liu et al., 2019; Chen et al., 2019] replace the z-buffer-based triangle selection of a vanilla rasterization process with a probabilistic manner. Unfortunately, the result is not so good due to inaccuracies introduced by these methods. On the contrary, Monte Carlo edge sampling based methods [Li et al., 2018a; Zhang et al., 2019] provide unbiased gradient estimates capable of producing plausible results. But these methods are resource-consuming because of their path tracing module, hindering their generalization.

Differentiable Rendering based Multi-view Inverse Rendering. The emergence of differentiable rendering boosts the development of the inverse rendering techniques. Several prior works leverage differentiable rendering methods to solve the inverse problem. Wu et al. [2020] propose a completely unsupervised method for face reconstruction from just one face image using the approximately-based differentiable method, which does not rely on existing 3D morphable face models. Luan et al. [2021] leverage the Monte Carlo edge sampling based methods to reconstruct fine geometry and plausible material. However, they assume that the camera and light are collocated. In addition, a rough geometry scanned by professional equipment is also needed. Although these settings could narrow down the solution space, they also reduce the generalization of the method. The method proposed by Li et al. [2021] reduces the requirements for input. It takes as input multi-view wild scene images, and reconstructs the initial geometry through MVS, then use the general Monte Carlo path tracing differentiable renderer to optimize the material, illumination, and geometry. Although it can achieve good results, it is resource-consuming and time-consuming.

3 Our Method

3.1 Overview

Our inverse rendering pipeline, shown in Figure 2, takes as input a set of RGB images captured by conventional hand-held cameras from multiple viewpoints. Using traditional methods (SfM and MVS), our method first reconstructs a virtual scene (in the form of a triangle mesh with vertex color) that roughly matches the real scene in the initialization phase. Afterward, in the optimization phase, our method first uses an approximate differentiable rendering method to optimize the geometry, then uses a physically-based differentiable rendering method to optimize the reflectance. Both methods iteratively improve the image loss between rendered images and the ground truth for the same viewpoint. Finally, the optimized scene parameters can be used for a variety of applications such as novel view synthesis and relighting. Conceptually, our pipeline can be formulated as an analysis-by-synthesis problem

where Θ represents the scene parameters that need to be estimated, including geometry and reflection parameters; L is a loss function to be minimized; ˜ I is the ground truth image, and I(Θ) is the image rendered by our hybrid differentiable rendering method using the same camera parameters as ˜ I. In the following, we present the details of each phase.

3.2 Initialization Phase

Our reconstruction pipeline is designed to work in complex, unstructured scenes where the input images have no additional information, such as depth. To tackle these challenges, we leverage traditional methods to obtain an initial geometry. Specifically, as shown in Figure 2(a), we use the SfM method from [Schonberger and Frahm, 2016] to generate sparse feature points in each image. These points are then used to reconstruct a sparse point cloud and a rough mesh by matching the feature points across images. Next, we use the MVS reconstruction method from [Schönberger et al., 2016] to generate dense pixel-level matching. Finally, a dense mesh is then obtained from Poisson surface reconstruction [Kazhdan et al., 2006], with vertex colors derived from the input images.

In terms of lighting, we use an environment map as our lighting source. The resolution of the environment map is 512 × 128. The environment map is a learnable parameter that is inferred during the material optimization phase. In the beginning, we assume that the light is white and the three channels of our environment map are set to (0.5, 0.5, 0.5).

3.3 Optimization Phase

In the optimization phase, starting from the initial geometry obtained in the previous step, we adopt a hybrid differentiable rendering method to further optimize the geometry and reflectance. Specifically, we utilize an approximate differentiable rendering method to optimize the geometry, and a physically-based differentiable rendering method to optimize the reflectance. Our hybrid approach is motivated by the following observations in experiments:

• Although the geometry reconstructed by SfM and MVS in the initialization phase provides a good approximation of the scene, there can be some defects in the boundary regions.
• Approximate differentiable rendering methods dedicated to

calculating the gradient of geometry can be very efficient, but the visual effect of their results may be of lower quality due to their simplification of the material model.

• Physically-based methods can accurately reconstruct complex materials, but their high computational costs and resource demands may impede their efficiency.

Specifically, starting from the triangle mesh obtained in the initialization phase, we further optimize its vertex positions θp which define the geometry, as well as two 2D texture maps that contain the diffuse albedo θd and specular albedo θs respectively and define the SVBRDF. In this way, our scene parameters can be represented as Θ = (θp, θd, θs). We first optimize the vertex positions using an approximate differentiable rendering method. Afterward, we optimize the diffuse albedo and specular albedo using a physically-based differentiable rendering method. Details of the optimization are presented below.

Geometry Optimization. To optimize the geometry, we adopt a differentiable model similar to Soft Rasterizer [Liu et al., 2019] to compute a silhouette image for each input image using the same camera parameters (see the supplementary materials for details of the computation), to indicate the occupancy from the same view directions as the input images. Then we minimize the following loss function to derive the vertex positions θp:

where Lsil is a silhouette loss that indicates the consistency between the computed silhouette images Isil(θp) and the groundtruth ones ˜ Isil derived from the input [Liu et al., 2019]:

with ⊗ and ⊕ being the element-wise product and sum operators, respectively. The other terms in Lgeo are regularizers. Among them, Llap = ∥LV∥2 is a mesh Laplacian loss Llap of a mesh with n vertices, V is the n × 3 coordinates matrix, and L ∈ Rn×n is the Laplacian matrix of the mesh (See [Nealen et al., 2006] for details). Lnormal = ∑i,j [1- (ni · nj)]2 is a normal consistency loss to make the normals of adjacent faces to vary slowly, where the sum is over all triangle pairs (i, j) sharing a common edge, and ni and nj are the face normals of the two specific triangles. Ledge = √∑i e2i is an edge length loss to avoid long edges that can cause ill-shaped triangles, where ei denotes the length of the i-th edge.

Reflectance Optimization. Our reflectance optimization is based on the rendering equation proposed in [Kajiya, 1986]. For a surface point x with surface normal n, let Li(ωi; x) be the incident light intensity at location x along the direction ωi, and SVBRDF fr(ωo, ωi; x) be the reflectance coefficient of the material at location x for incident light direction ωi and viewing direction ωo. Then the observed light intensity Lo(ωo; x) is an integral over the upper hemisphere Ω:

We leverage a modified Cook-Torrance (CT) model [Cook and Torrance, 1982] based on [Zeltner et al., 2021] as our reflectance model, which contains a rough surface with diffuse and specular reflection without refraction. Mathematically, our reflectance model can be described as the following,

where θd and θs are the diffuse and specular reflectance:

Specifically, ρd is diffuse albedos, ρs is specular albedos, and h is the halfway vector. D(h, α) is the microfacet distribution function which is GGX [Walter et al., 2007] used in our work. α and n(x) denote the surface's roughness and normal, respectively. G is a shadowing-masking function. Like other works, we also ignore the Fresnel effect which cannot be observed in our scene. To optimize the reflectance parameters θr = (θd, θs), we minimize the following reflectance loss:

where Lrgb = ∥∥∥ ˜ Irgb - Irgb(θr)∥∥∥1 measures the ℓ1 norm of the difference between the rendered color image Irgb(θr) and the ground truth ˜ Irgb. R(θr) is a regularizer for the diffuse albedo θd and specular albedo θs. Similar to [Schmitt et al., 2020], we assume that nearby pixels with similar diffuse albedos have similar specular albedos. So we choose

where kp,q = exp (- ∥p-q∥22 2σ2 1 - (θd[p]-θd[q])2 2σ2 2 ) and p, q are two specific mesh vertices.

4 Experiments

We perform quantitative and qualitative evaluation of our method using extensive experiments on both synthetic and real data, and compare it with state-of-the-art methods.

Implementation Details. We implement the geometry reconstruction module in the initialization phase using COLMAP 1 [Schönberger et al., 2016; Schonberger and Frahm, 2016], a general-purpose Structure-from-Motion and MultiView Stereo pipeline. In the optimization phase, we implement the approximate differentiable rendering based on Pytorch3d [Ravi et al., 2020], and the physically-based differentiable rendering based on Mitsuba 2 [Nimier-David et al., 2019]. The rest of our optimization pipeline is implemented with PyTorch using the Adam optimizer. For geometry optimization, we use the weights (λsil , λlap , λedge , λnormal ) = (1.0, 1.0, 1.0, 0.01) for Eq. 1, and 0.001 for the learning rate. The ground-truth silhouette images required for Eq. 1 are obtained either from rendering (for synthetic data) or using a background removal tool 2 (for real data). For reflectance optimization, we use the weights (λrgb, λreg) = (0.1, 1.0) for Eq. 2, and 0.0001 for the learning rate. To avoid excessive memory consumption in the physically-based rendering module, we set the maximum depth of ray tracing bounce to 3, the downsampling factor of raw images to 4, and the number of sampling per pixel spp to 4 in the iterative optimization phase.

1 https://github.com/colmap/colmap

2 https://www.remove.bg/

We train and test our method on one NVIDIA RTX 3090 GPU. For geometry optimization, our implementation uses 200 iterations per view and takes about 0.5 hours per example. For reflectance optimization phase, our implementation uses 400 iterations per view and takes 1.5∼2.5 hours per example.

Synthetic Data. Our synthetic data is created using meshes and textures from [Zhou et al., 2016] and the Internet, covering different materials and geometries. Concretely, for each object, we render 400 images with colored environmental lighting using graphics engines, 300 for training, and the left 100 for novel view synthesis testing, whose viewpoints are evenly distributed inside the upper hemisphere. In addition, we also render the same object with two other environment lighting maps as the ground truth for the following relighting performance testing. To quantitatively evaluate our method, we use three image quality metrics-LPIPS [Zhang et al., 2018], SSIM and PSNR-to compare the rendering results with the corresponding ground truth images. Figure 3 shows examples of results from our method and their ground truth, as well as their evaluation metric values. Both the quantitative metrics and the qualitative visualization show that our novel views and relighting results match the ground truth closely. Fig. 6 shows the diffuse and specular albedo for the lemon model.

Real Data. We evaluate our method on multiple real-world images from the DTU datasets [Aanæs et al., 2016], where the objects are glossy and the illumination is static across different views. We use two objects from the dataset, shiny scan114 buddha and scan110 ghost, and discard photos with strong shadows. Our inverse rendering results are shown in Figure 4. We can see that our pipeline generates photo-realistic novel views, and results of relighting and material editing.

Comparison with State-of-the-art Methods. As far as we are aware, there is no prior work tackling exactly the same problem as our work: reconstructing triangle meshes and PBR materials from multi-view real-world object images with uncontrolled lighting conditions. The methods closest to our method are [Luan et al., 2021; Li et al., 2021], but no source code or data is released. It is worth noting that our method addresses the shortcomings of [Li et al., 2021] in dealing with geometry, while overcoming the limitation of [Luan et al., 2021] which requires the camera and point lighting to be in the same position and the images to be taken in dark scenes.

We compare our method to the most related neural rendering approaches, including PhySG [Zhang et al., 2021b], NeRF [Mildenhall et al., 2020] and IDR [Yariv et al., 2020], in terms of novel view synthesis. These approaches are different from our method in the model of light transport: NeRF uses the occupancy function to describe geometry and appearance maps and gets the pixel color according to location and viewing direction in a ray-marching way, while PhySG and IDR use SDF to represent geometry and material. In addition, unlike the aforementioned two methods that focus solely on novel view synthesis, PhySG also performs inverse rendering tasks, using Spherical Gaussian functions to describe materials and illumination. Qualitative and quantitative comparisons between different methods on our synthetic data are depicted in Figure 5 and Table 1. The unsatisfactory performance of PhySG may be due to the presence of strong specular highlights in our synthetic data, which are difficult to describe using the Spherical Gaussian functions employed by PhySG. NeRF performs relatively poorly in view synthesis because its volumetric representation does not concentrate colors around surfaces as well as surface-based approaches. IDR does a better job in view synthesis because of its view-dependence model. However, it still struggles to synthesize specular highlights due to the non-physical model of appearance. In contrast, our method models highlight well, benefiting from our PBR materials. We also compare the running time, where our training time is the sum of initialization time and optimization time. Thanks to the combination of efficiency from the approximate method and the accuracy from the physically-based method, our hybrid approach runs significantly faster than the two baselines both in training and testing. In addition, our optimized geometry and material can be deployed directly on mainstream graphics engines.

We also compare our method with the latest related work IRON [Zhang et al., 2022] which adopts neural representations and also leverages a hybrid optimization scheme. Note that IRON assumes a point light source, while our method allows for uncontrolled lighting conditions. As a result, IRON performs poorly in complex lighting situations. Figure 9 com-

150 Inputs

200 Inputs

26.63/0.772/0.093

31.04/0.890/0.063

33.28/0.9160/0.045

300 Inputs

350 Inputs

34.42/0.924/0.041

35.07/0.928/0.036

PSNR/SSIM/LPIPS

Figure 8: Comparison between our method and IRON [Zhang et al., 2022] on the hotdog data.

pares the two methods in different lighting conditions. Our method can achieve similarly good results as IRON on pointlighting data, while being superior with more complex lighting. In particular, IRON cannot handle the highlights on the surface of objects under environment lighting because their light source cameras are in the same position, while our method can handle it well. In addition, for some flat data like the hotdog in Figure 8, IRON struggles to obtain appropriate geometry, while our method can produce an accurate reconstruction.

Ablation Study. Figure 7 shows our ablation study on the number of input images, which depicts how the number of input images affects our reconstruction quality. With too few input images, the optimization may become highly underconstrained, making it difficult to produce accurate synthesis results. In our experiments, 150 images are sufficient to produce a quite good result. With 250 or more input images, our results will closely match the ground truth. More ablation studies can be found in the supplementary materials.

5 Conclusion

We introduce a novel efficient hybrid differentiable rendering method to reconstruct triangular object mesh and PBR material from multi-view real-world images with uncontrolled lighting conditions. Unlike prior works that require massive resource consumption or approximated rendering process, we utilize an approximate method to optimize the geometry and a physically-based method to estimate the reflectance so that we could benefit from both the efficiency of the former and the high quality of the latter. In general, our method can handle a wide range of real-world scenes, providing an attractive and efficient solution and enabling photo-realistic novel view synthesis and relighting applications.

Limitations and future work. Our method can have difficulties with very thin geometry, which is a common problem of mesh-based methods. In addition, our method optimizes geometry and material separately. A potential future work is a unified pipeline to optimize geometry and material simultaneously, which should further improve the result quality.

Acknowledgments

This work was supported by the NSFC under Grant 62072271.