Optimized View and Geometry Distillation from Multi-view Diffuser

1 Introduction

Traditionally, the process of generating a three-dimensional model from a singular image necessitates extensive and meticulous efforts by highly skilled artists. However, recent advancements in neural networks, particularly through the adaptation of 2D diffusion models for 3D synthesis, have rendered the conversion of a single image into a 3D object feasible. The early breakthrough comes from the text to 3D domain, where DreamFusion [Poole et al., 2023] and Score Jacobian Chaining (SJC) [Wang et al., 2023a] proposes a Score Distilling Sampling (SDS) strategy to distill the scores learned by 2D diffusion models from large-scale images to optimize a Neural Radiance Field (NeRF) [Mildenhall et al., 2020], circumventing the need for 3D data. Successive approaches further improve the quality and diversity of generated geometries from textural prompts [Wang et al., 2023b; Lin et al., 2023; Chen et al., 2023]. Particularly, RealFusion [Melas-Kyriazi et al., 2023] migrates the scheme to generate plausible 3D reconstruction matches to a single input image via textual inversion adapted supervision.

†Corresponding author.

More relevantly, 3DiM [Watson , 2023 ] and MVDream [Shi et al., 2023] develop a pose-conditional imageto-image diffusion model, which generates the novel view at a target pose from a source view. Zero-1-to-3 [Liu et al., 2023b] adopts a similar framework and learns control of viewpoints through a synthetic dataset and demonstrates zero-shot generalization to in-the-wild images. Though Zero1-to-3 demonstrates plausible novel views, they are not multiview consistent and the geometry distilled from SDS tends to be oversmoothed. To enhance the multi-view consistency, SyncDreamer [Liu et al., 2024] devises a volume-encoded multi-view noise predictor to share information across different views. Wonder3D [Long et al., 2024] predicts the multiview color images along with their normal maps from a crossdomain diffusion model. Though enhancing the multi-view consistency of image generation, these methods compromise the flexibility of camera positioning and only allow synthesis for a limited number of views.

In this study, we observe that the predicted unconditional noise from the multi-view diffuser, i.e., the Zero-1-to-3 model, appears to be biased. That is, even if we only add very low-level noise to a normal image and use the unconditional noise predicted by a Zero-1-to-3 model for denoising, the result still tends to deviate greatly from the original image. We analyze and rectify the critical bias by using an unconditioned noise from a pre-trained 2D diffusion model and greatly refining the geometry fidelity. Moreover, previous approaches use either 3D volume [Liu et al., 2024] or ray aggregation [Tseng et al., 2023] to share information across views. We consider the radiance field as the consistency prior, and encourage the generated multi-view images to be consistent with the NeRF renderings. We develop a twostep specialization process of a 2D diffusion model, which is adept at conducting target-specific denoising and generating high-quality multi-view images from NeRF renderings. We then further use the refined views to generate the geometry and texture from NeuS [Wang et al., 2021], and in the meanwhile enforce input view consistency using view score distillation. Our approach generates comparable-quality of multi-view images and geometry to the SOTA approaches, including SyncDreamer and Wonder3D, without enforcing any restriction on camera poses. Consequently, we posit that our approach offers superior adaptability and effectiveness in addressing the challenges associated with generating consistent and high-quality multi-view imagery and geometry, affirming its substantial potential for widespread application in relevant fields.

2 RelatedWork

2D Diffusion Models for 3D Generation. Recent advancements in 2D diffusion models [Rombach et al., 2022; Croitoru et al., 2023] and large-scale visual language models, notably the CLIP model [Radford et al., 2021; Song et al., 2024], have catalyzed new approaches for generating 3D assets. Pioneering efforts such as DreamFusion [Poole et al., 2023] and SJC [Wang et al., 2023a] have developed methods for transforming 2D text into images, subsequently facilitating the generation of 3D shapes from text. This approach has inspired a range of subsequent studies that adopt a shapeby-shape optimization scheme. Additionally, the integration of 2D diffusion models with robust vision language models, especially CLIP [Radford et al., 2021], has emerged as a significant exploration in the generation of 3D assets [Xu et al., 2023b]. The typical methodology involves optimizing a 3D representation, such as NeRF, mesh, or SDF, and then utilizing neural rendering to generate 2D images from various viewpoints. These images are processed through 2D diffusion models or the CLIP model to calculate SDS losses, which guide the optimization of the 3D shape. Building on the foundations laid by DreamFusion and SJC, numerous works have enhanced text-to-3D distillation methods in various aspects. Notably, Magic3D [Lin et al., 2023] develops a two-stage coarse-to-fine optimization framework for high-resolution 3D content generation, and ProlificDreamer [Wang et al., 2023b] proposes a Variational Score Distillation (VSD) for generating highly detailed geometry. However, challenges such as low efficiency and the multi-face Janus problem, where optimized geometry tends to produce multiple faces due to the lack of explicit 3D supervision, remain prevalent. Furthermore, some works [Radford et al., 2021] have applied this distillation pipeline in single-view reconstruction tasks. While these methods have achieved impressive results, they often require extensive time for textual inversion and NeRF optimization, without always guaranteeing satisfactory outcomes. In contrast with the 2D diffusion to 3D extension, which is ignorant to multi-view consistency, our method focuses on the multi-view diffuser technique, which predicts the noises for novel views, and inherently avoids the Janus problem.

Multi-view Diffusion Models. In light of the complexities involved in ensuring the integrity of generated 3D content, recent efforts [Watson et al., 2023; Gu et al., 2023; Deng et al., 2023; Tseng et al., 2023; Chan et al., 2023; Yu et al., 2023; Tang et al., 2023; Liu et al., 2023c] have explored the feasibility of directly generating novel views from a single image input. Notably, the Scene Representation Transformer [Sajjadi et al., 2022] extends the vision transformer to image sets, enabling global information integration for 3D reasoning. Similarly, 3DiM [Watson et al., 2023] develops a pose-conditional image-to-image diffusion model, translating a single input view into consistent and sharp completions across multiple views. A seminal work in this area, Zero-1-to-3 [Liu et al., 2023b], utilizes a similar network structure, trained on a large-scale synthetic 3D dataset, demonstrating notable generalizability. Recent advancements, such as One-2-3-45 [Xu et al., 2023a], have leveraged the generalizable neural reconstruction method SparseNeuS [Long et al., 2022] to directly produce 3D geometry from images generated by Zero-1-to-3. While this approach is more efficient and alleviates the Janus (multi-head) problem, it tends to produce lower-quality results with reduced geometric detail. In a different vein, SyncDreamer [Liu et al., 2024] concentrates on object reconstruction, generating images in a single reverse process and utilizing attention to synchronize states among views. This contrasts with Viewset Diffusion [Szymanowicz et al., 2023], which requires predicting a radiance field. SyncDreamer solely relies on attention for synchronization, fixing the viewpoints of generated views to enhance training convergence. A significant trend in recent research has been the design of various functional attention layers. Consistent123 [Weng et al., 2023] employs a shared self-attention layer, where all views query the same key and value from the input view. ConsistNet [Yang et al., 2024] introduces two sub-modules: a view aggregation module and a ray aggregation module, to extract features consistent across multiple views. MVDream [Shi et al., 2023] utilizes 3D self-attention. Wonder3D [Long et al.,

2024] goes a step further by not only outputting multi-view images but also outputting the normal map for each perspective, using cross-domain attention to maintain the consistency of the 3D structure. Both SyncDreamer and Wonder3D, resulting from fixed view outputs, exhibit sensitivity to the camera angle of the input image. In contrast, our method can reconstruct accurate 3D structures from inputs captured at varying camera angles. While existing approaches have focused on developing novel modules for enforcing multi-view consistency, our work demonstrates that an optimized distillation strategy can yield views and geometries comparable to models trained on large-scale datasets. This insight may inspire further exploration in improved strategies for geometry and view extraction.

3 Method

Our objective is to synthesize consistent multi-view images and high-quality geometric representations from a single input image. Notable prior work, such as the Zero-1-to-3, has demonstrated impressive results by utilizing an image and camera pose-conditioned diffusion model. However, this approach encounters limitations, particularly in terms of generating inconsistent multi-view images and a tendency for over-smoothing in the geometric output. The state-of-theart research, including SyncDreamer [Liu et al., 2024] and Wonder3D [Long et al., 2024], addresses these challenges by incorporating additional modules for consistency or employing normal supervision. However, this often compromises the flexibility of positioning the target camera at will. Contrastingly, our work adopts a distinct methodology, showcasing that comparable quality in views and geometry can be achieved through a meticulously crafted distillation strategy. Central to our approach is the insight that the unconditional noise predictions from Zero-1-to-3 are inherently biased. We propose the utilization of unconditional noise from the Stable Diffusion [Takagi and Nishimoto, 2023] model to rectify this issue, as elaborated in Sec. 3.1. Our method, termed Unbiased Score Distillation (USD), significantly enhances the quality of the radiance field relative to previously used SDS/SJC methods. Furthermore, we employ the optimized NeRF [Mildenhall et al., 2020] as a consistency prior, in contrast to previous implicit constraints such as 3D volume or ray aggregation. We ensure that the generated views and geometry align coherently with the distilled NeRF to achieve consistency. We posit that specializing a diffusion model to denoise the target object is crucial. To this end, we implement the DreamBooth [Ruiz et al., 2023] technique and engage in a two-stage fine-tuning process, detailed in Sec. 3.2. The first stage involves using multi-view images from Zero-1-to-3 as positive samples, contrasting them with text-prompt generated images as negative samples for image style learning. In the subsequent stage, the input image serves as the positive sample, with all Zero-1-to-3 generated images as negatives, focusing on learning finer details. Subsequently, we introduce a low level of noise into the NeRF renderings and employ the fine-tuned diffusion model for denoising. The final step involves applying the NeuS technique for mesh reconstruction, while in the meanwhile using View Score Distillation to ensure input view consistency. Fig. 2 illustrates the complete pipeline of our methodology.

3.1 Unbiased Score Distillation

In this section, we highlight the significant bias in Zero-1to-3's unconditional noise due to insufficient unconditional training and object-level dataset bias, affecting geometry quality in SDS-based distillation, both theoretically and empirically. We then propose a rectification method.

Bias in Unconditional Noise. As a multi-view diffuser, Zero1-to-3 predicts noises of a target image latent z given two conditions: the input image cI and relative camera pose cP , i.e., it learns the probability distribution P(z|cI , cP ) for image latent z. We apply Bayes' Rule to decompose the conditional probability:

Diffusion models estimate the score [Hyv¨arinen and Dayan, 2005] of the data distribution, i.e., the derivative of the log probability, giving us the following expression:

This leads to the score estimate with classifier-free guidance (CFG [Ho and Salimans, 2022]):

where ϵϕ is a neural network for predicting noises given conditions, α1 and α2 are guidance scales that enable separately trading off the strength of conditions cI and cP separately. To ensure accurate prediction of ϵϕ(zt, cP , ∅) and ϵϕ(zt, ∅, ∅), one needs to randomly drop cI and cP during training ϵϕ. However, we observed that Zero-1-to-3 did not truly follow this procedure to supervise unconditional noise. As shown in Fig. 4, Zero-1-to-3 only randomly dropped image conditions cI while keeping cP untouched. During inference, it replaces the tensor after fully connected (FC) layer f(·) with zeros (not exactly equal to setting both cI and cP to zeros as f(0) ̸= 0), leads to a bias. Moreover, the dataset used for fine-tuning is majorly object-centric, which may also introduce additional domain bias. Fully addressing this bias problem requires re-training the multi-view diffuser on a broader and more balanced dataset, which requires tremendous effort. Here we propose an effortless fix in the following.

Rectification. To alleviate the inaccurate unconditional noise problem, we first set α1 = α2 = ω to eliminate ϵϕ(zt, cP ,∅). We then and yield a special case of ϵCFG ϕ (zt, cI, cP ) as:

To demonstrate our setting of α1 and α2 is valid, we provide an empirical validation can be found in the Appendix E. Further, we consider ϵϕ(zt,∅,∅) predicts noises from only the noisy latent and is equivalent to the unconditional noise ϵψ (zt,∅) of Stable Diffusion (SD) as it uses the same variational autoencoders (VAE [Kingma and Welling, 2013]) as Zero-1-to-3. As such, the bias has been well rectified. To verify the effect of our rectification, Tab. 1 shows the denoising effect using various unconditional noise settings: (1) use SD unconditional noise ϵψ (zt,∅). (2) use Zero-1-to-3 noise ϵϕ (zt, f(0, cP )) with f(0, cP ) as a condition (same to the training process as in Fig 4). (3) use Zero-1-to-3 noise ϵϕ (zt, 0) with 0 as a condition (referring to the inference process, as shown in Fig 4). The SD noise generates the best result, and setting (2) is slightly better than (3) as it follows the training set up. More details can be found in the Appendix A. Unbiased Score Distillation. One major application of a multi-view diffuser is to distill 3D content, represented as NeRF paramater θ, using the SDS loss for optimization:

where w(t) is a weighting function, ϵ is standard Gaussian noise, and zt refers to the noisy latent as zt = √αtz + √1 - αtϵ, with αt being the noise scheduler. We rewrite the noise difference in Formula 5 by adding an additional weighting factor λ to [ϵψ (zt, ∅) - ϵ]:

where setting λ = 1 we get Formula 5. Inspired by DDS [Hertz et al., 2023] and CSD [Yu et al., 2024], we observed that setting λ = 0 can significantly improve the details of the 3D details generated using SDS. Further details and an in-depth analysis are provided in the Appendix D. We obtain our Unbiased Score Distillation (USD) as:

Our USD generates much better and consistent 3D than the SDS/SJC method used in Zero-1-to-3, the results can be found in the Appendix E.

3.2 Consistent View and Geometry Distillation

Although USD can be used to optimize NeRF for view synthesis, the resulting images still tend to be blurry, and directly extracting geometry from the NeRF density field introduces noise into the mesh. Since our ultimate goal is to extract highquality geometry and consistent multi-views, we propose to utilize the generated NeRF as the view consistency prior, i.e., the final high-quality view images should be consistent with the NeRF rendering. Thus, our problem transformed into a denoising problem.

Two-Stage Specified Diffusion. We leverage the recent advance in diffusion model specialization, i.e., the DreamBooth [Ruiz et al., 2023], to fine-tune a 2D diffusion model for the specific target in the input image. We observe that the novel view images generated by Zero-1-to-3, though not multi-view consistent, tend to have the same style as the input image. We design a two-stage tuning method to gradually let the diffusion model learn the object details. In the first stage, we use the multi-view images generated by Zero-1-to-3 as positive samples, contrasting them with text-prompt generated images as negative samples for learning the visual style of the target. In the subsequent stage, the input image serves as the positive sample, with all Zero-1-to-3 generated images as negatives, focusing on learning finer details. During optimization, we use a unique identifier [V] to capture the visual style of the target. In the second stage, we set only the input image as the positive sample and the images generated by Zero-1-to-3 as negative ones, and use the additional identifier [S] to capture the identity of the specific target.

Geometry and Texture Distillation with Input View Supervision. With the specialization diffusion model, we add a small noise, Stable Diffusion scheduler t = 200, to the NeRF render images and conduct the denoising process. Then, we use the NeuS technique to reconstruct the geometry from the high-quality and clear images (i.e., 100 input images).

We observe that the input view is rarely used for optimization of the geometry and texture in multi-view diffusers, despite the fact that the input image is most faithful to the target object. To exploit the input view information, we further develop a reference view score distillation during the NeuS reconstruction process. Specifically, we consider the image render from NeuS at the input image viewpoint as zt ∗ = R(Θ, p∗ ), where R is the rendering function from NeuS model defined by Θ, p∗ is the camera pose of the input image, can be set to a particular relative pose. We define the following reference view distillation loss as:

where yt is input view image. We add this reference view distillation loss LRV to original photometric loss with optimized images for optimizing the NeuS parameter Θ. This scheme is a better supervision strategy than directly applying MSE loss on the input view. The reason for this is that, compared to computing MSE loss directly at the pixel-level, our patch-aware noise(latent)-level approach places greater emphasis on the perceptual quality of the image.

4 Experiments

We conduct extensive experiments, both qualitatively and quantitatively, to demonstrate the effectiveness of our method.

4.1 Implementation Details

Optimizing the Pipeline. We use exactly the same set of hyperparameters for all experiments and do not perform any per-object hyperparameter optimization. We use the implicitvolume implementation in threestudio [Guo et al., 2023] as our 3D representation (NeRF), which includes a multiresolution hash-grid and a MLP to predict density and RGB. The NeRF is optimized for 10,000 steps with an Adam optimizer at a learning rate of 0.01, weight decay of 0.05, and betas of (0.9, 0.95). For USD, the maximum and minimum time steps are decreased from 0.98 to 0.5 and 0.02, respectively, over the first 5,000 steps. We adopt the Stable Diffusion [Takagi and Nishimoto, 2023] model of V2.1. The classifier-free guidance (CFG) scale of the USD is set to 7.5 following [Wang et al., 2023b]. The DreamBooth backbone is implemented using Stable Diffusion V2.1. In the first stage, we use Stable Diffusion to generate 200 images as negative samples. Additionally, we utilize 6 positive sample images with 360◦ surrounding camera poses (at 60◦ intervals) for training. The USD (NeRF) process takes about 1.5 hours on a NVIDIA Tesla V100 (32GB) GPU. To achieve reduced running time, we provide additional discussions and experimental results in the Appendix C. For DreamBooth fine-tuning, we train the model around for 600 steps with a learning-rate as 2e-6, weight decay as 0.01 and a batch size of 2. More details on the experimental setup are provided in the Appendix B.

Camera Setting. Since the reference image is unposed, we assume its camera parameters [Liu, 2023] are as follows. We set the field of view (FOV) of the camera is 40◦ , and the radial distance is 1.5 meters. Note this camera setting works for images subject to the front-view assumption. For images taken deviating from the front view, a manual change of polar angle or a camera estimation is required.

4.2 Evaluation Protocol

Evaluation Datasets. Following prior research [Liu et al., 2023b; Liu et al., 2024; Long et al., 2024], we adopt the Google Scanned Object dataset [Downs et al., 2022] for our evaluation, which includes a wide variety of common everyday objects. Our evaluation dataset matches that of SyncDreamer [Liu et al., 2024], consisting of 30 objects that span from everyday items to animals. For each object in the evaluation set, we render an image with a size of 256 × 256 and use it as the input. Additionally, to assess the generalization ability of our model, we include images with diverse styles collected from the website in Zero-1-to-3, SyncDreamer and Wonder3D.

Metrics. To evaluate the quality of single view reconstruc-

tion, we used two commonly used metrics: the chamfer distance (CD) between the ground truth shape and the reconstructed shape, and the volume IoU. Due to different methods using different normative systems, before calculating these two metrics, we first align the generated shapes with the basic fact shapes. Moreover, we adopt the metrics PSNR, SSIM and LPIPS for evaluating the generated color images.

4.3 Ablation Study

We validate our design choices by ablating 4 major model variants, that are without Unbiased Score Distillation, without reference view distillation loss, without the two-stage DreamBooth (DB). As shown in Fig. 6, the 3D models generated without USD exhibit biased texture colors, and their shapes are not smooth. Not using reference view supervision will result in the inability to recover the same texture details as the input image, and not using DB will result in blurring of texture details. We also conducted a quantitative analysis on the GSO dataset, presented in Tab. 4. The results demonstrate that USD is crucial for geometric accuracy, and all our submodules collectively enhance overall performance.

4.4 Different Viewing Angle Comparisons

We found that SyncDreamer [Liu et al., 2024] and Wonder3D [Long et al., 2024] are very sensitive to viewing angles. If a relatively high viewing angle is input, SyncDreamer and Wonder3D will predict incorrect multi-view images. The results can be found in the Appendix E.

4.5 Single ViewReconstruction

We evaluate the quality of the reconstructed geometry of different methods. The quantitative results are summarized in Tab. 2, and the qualitative comparisons are presented in Fig. 5. The quality of Wonder3D shape reconstruction depends on the perspective of the input view, such as 'Train' shown in Fig. 5, where Wonder3D generated incorrect prediction results. The shape generated by SyncDreamer undergoes deformation due to the camera pose in the input view. One-23-45 [Liu et al., 2023a] attempts to reconstruct meshes from the multiview-inconsistent outputs of Zero-1-to-3 [Liu et al., 2023b]. While it can capture coarse geometries, it loses important details in the process. In contrast, our method can achieve good reconstruction quality and texture in terms of geometric structure and texture. In the paper, all the surface extraction demonstrated by our method is built on the Instant NGP (iNGP) [M¨uller et al., 2022] based SDF reconstruction method [Guo, 2022], and we use Blender Cycles [Community, 2018] to render the results.

4.6 Novel View Synthesis

We evaluate the quality of novel view synthesis for different methods. The quantitative results are presented in Tab. 3, and the qualitative results can be found in Fig. 3. Zero1-to-3 produces visually reasonable images, but they lack multi-view consistency since it operates on each view independently (we don't show the results of Zero-1-to-3). Although SyncDreamer introduces a volume attention scheme to enhance the consistency of multi-view images, their model is sensitive to the elevation degrees of the input images and tends to produce unreasonable results. Wonder3D ensures 3D consistency by generating normal maps, but may result in incorrect results for some input viewpoints.

4.7 Text-to-Image-to-3D

As a case study, we combine text-to-image models, i.e., the Stable Diffusion or Imagen [Saharia et al., 2022] to generate 3D models from text. We show some examples in the Appendix E. Compared to DreamFusion [Poole et al., 2023], ProlificDreamer [Wang et al., 2023b] and MVDream [Shi et al., 2023], our method shows no multi-face Janus problem and conforms to the text faithfully.

5 Conclusions

In this work, we introduce an optimized approach for distilling geometry and views from a multi-view diffuser, with a specific focus on the Zero-1-to-3 model. We observed that the direct application of the SDS/SJC technique to Zero-1-to3 is often suboptimal, primarily due to bias issues inherent in unconditional noise. To address this, we propose an Unbiased Score Distillation (USD) strategy by leveraging unconditioned noises from a 2D diffusion model to effectively enhance the optimized radiance field. Moreover, we developed a two-stage DreamBooth refinement process to improve the rendering of views. This process ensures consistency across multiple perspectives while simultaneously enhancing image quality. While we have identified and addressed the bias issue in the Zero-1-to-3 model, the underlying causes remain to be fully understood. Future research will delve into the theoretical aspects of this bias problem. Additionally, we aim to explore the potential applications of USD in other fields, such as image translation and view synthesis.

Acknowledgements

This work is supported by the National Natural Science Foundation of China (NSFC No. 62272184 and No. 62402189), the China Postdoctoral Science Foundation under Grant Number GZC20230894, the China Postdoctoral Science Foundation (Certificate Number: 2024M751012). The computation is completed in the HPC Platform of Huazhong University of Science and Technology.