3D Surface Super-resolution from Enhanced 2D Normal Images: A Multimodal-driven Variational AutoEncoder Approach

1 Introduction

With the rapid development of computer hardware and software technology, the requests for 3D surface reconstruction, perception and analysis are becoming more and more popular in various immersive applications [Feng et al., 2019], such as metaverse. High-quality 3D data is one of the fundamental carriers for satisfying these application scenarios. However, on one hand, the production speed of high-resolution (HR) 3D content suffers due to the high cost of 3D data-acquisition sensors. On the other, the utilization of existing legacy 3D objects with low-resolution (LR) needs to be enhanced. Considering these facts, it is necessary to develop a low-cost method for the surface enhancement of 3D objects.

Several methods for 3D surface super-resolution have been developed in the past few years, which can be roughly divided into two categories depending on the computation domain: 1) 3D domain-based methods and 2) 2D domain-based methods. In the 3D domain, there are voxel-based methods [Xie et al., 2020], point-cloud-based methods [Luo et al., 2021], distance-functions-based methods [Chen et al., 2021b] and mesh-based methods [Schult et al., 2020]. In these methods, those traditional ones can only optimize some surface mathematical properties and lead to a smooth result, while learning-based methods face the problem of the lack of data and high computational overhead due to the complexity of 3D data representations [Hanocka et al., 2019].

∗This work was supported in part by the National Natural Science Foundation of China under Grants 61902251, in part by the Natural Science Foundation of Shenzhen City under Grants JCYJ20220809160139001, and in part by the Natural Science Foundation of Guangdong Province under 2021A1515011877 and 2022A1515011245. (Corresponding author: Miaohui Wang)

In the 2D domain, existing studies usually use normal images [Chen et al., 2018], [Zhang et al., 2021] or depth images generated by projecting the surface to 2D domain [Schwarz et al., 2018], [Voynov et al., 2019] to enhance a 3D surface. Surface super-resolution in the 2D domain has two significant advantages: 1) the computational complexity will be greatly reduced, and 2) many well-evaluated approaches in the image processing field can be reused directly, such as image super-resolution (SR). Due to the fact that a normal map carries more micro geometric shapes and details but a depth map only contains range or distance information, it is a better choice to perform 3D surface SR in the 2D normal domain.

However, existing SR methods in the 2D domain usually only focus on a single image modality, which rarely utilizes the complementary information available in different modalities of a 3D object. Recently, some image synthesis methods [Esser et al., 2021], [Huang et al., 2021] and image SR methods [Yang et al., 2020], [Liu et al., 2021] have taken multimodal information into account and improved the performance of network. Besides, [Xie et al., 2022] developed a preliminary multimodal transformer framework for 3D surface super-resolution (MNSRNet), which considered three different modalities including depth, RGB and normal images.

Inspired by MNSRNet, we develop a multimodal-driven variational autoencoder (mmVAE) framework to perform 3D surface super-resolution based on enhanced 2D normal images. There are two significant differences between mmVAE and MNSRNet: Firstly, MNSRNet introduced a depth modality branch to preserve large surface structure correctness. However, since 3D surface SR focuses on recovering high-frequency information, introducing a whole branch is informatively redundant. Therefore, mmVAE discards the depth modality but introduces a cross-scale structure to preserve large surface structures, which can also reduce the computational overhead. Secondly, MNSRNet introduced a multimodal transformer to perform feature alignment and fusion. However, due to the limitations of the transformer structure, the training is usually unsatisfactory when the amount of data is insufficient, and the lack of direct constraints on the alignment module leads to limited performance. To address these problems, mmVAE investigates a VAE-based structure to reduce the requirements and improve the constraints on the modality alignment and fusion process by a feature reparameterization structure. However, since the classic VAE architecture is limited by the standard normal distribution in latent variable reparameterization, resulting in limited generation ability, we further propose a Gaussian mixture model and the corresponding loss for the multimodal alignment to improve its expression ability.

To summarize, this work has the following contributions: (1) establishing a new multimodal 3D surface superresolution framework based on the VAE framework in the 2D normal domain, (2) investigating a Gaussian mixture model in fusing the normal and texture modalities for 3D objects, which is a more comprehensive multimodal fusion approach to exploit auxiliary modality information, and (3) developing a new cross-scale modality VAE network structure, which is able to simultaneously preserve large surface structure as well as fine-grained surface geometry.

2 RelatedWork

Because our mmVAE is based on enhanced 2D normal images, we briefly review some closely representative imagebased SR methods, including single image super-resolution (SISR) and multimodal image super-resolution (MISR).

2.1 Single Image Super-resolution

In the SISR task, following the research by [Dong et al., 2014], a number of CNN-based approaches have been proposed such as VDSR [Kim et al., 2016], EDSR [Lim et al., 2017], and RCAN [Zhang et al., 2018b]. Recently, with the success of the self-attention mechanism, transformer-based network structure has been also studied for SISR [Chen et al., 2021a]. Besides, some other useful modules such as VAE structure [Liu et al., 2020], [Chira et al., 2022], generative adversarial network (GAN) [Chen et al., 2022], and dual regression network [Emad et al., 2021] have been used in SISR.

2.2 Multimodal Image Super-resolution

The combination of multimodal information (e.g., different viewpoints, sensors, or domains), especially homogeneous multimodality that means different types of modality information obtained from the same type of sensor for the same object, is a hotspot for MISR to enhance image reconstruction performance [Yao et al., 2021]. For instance, [Wang et al., 2018] introduced the image segmentation map as the prior information to improve the learning performance of GAN. [Liu et al., 2021] proposed a CVAE structure by using reference images to enhance the restoration performance. [Li et al., 2019] employed a normal image to guide the superresolution of a texture image. [Xie et al., 2022] proposed a transformer-based multimodal network to learn from different modality images for surface enhancement.

Frankly speaking, MISR has been investigated in some preliminary investigations, but the exploration of multimodalbased 3D object surface enhancement is still in its infancy. One of the reasons is the difficulty of constructing descriptors that can effectively represent information between different modalities. And another important reason is how to integrate and fusing these descriptors effectively is also challenging. In the light of the above considerations, exploring mechanisms for more efficient multimodal exploitation of 3D surface modality information through VAE structures is the most direct motivation for our paper.

3 Multimodal Variational AutoEncoder

3.1 Overview

Problem Formulation. Our goal is to enhance the surface of a 3D object via the assistance of RGB modality. To simplify this problem, we propose to represent a 3D surface in the 2D normal domain, which converts 3D surface SR into a normal image super-resolution task. Generally speaking, SR aims to restore the spatial resolution of an input LR normal image Nlr ∈ RH×W×3 by scale s, which can be obtained by minimizing the loss function Loverall between the upscaled normal image Nsr ∈ RsH×sW×3 and the ground-truth (GT) normal image Ngt ∈ RsH×sW×3 .

Architecture. Previous studies have witnessed the positive effect of multimodal data in deep learning [Yao et al., 2021] and the power of VAE structure. In light of this, the proposed multimodal variational autoencoder (mmVAE) superresolution network takes the RGB and normal modalities simultaneously, and adopts a cross-scale multimodal encoderdecoder structure as a backbone. The overview of mmVAE is depicted in Figure 2, which can be formulated as

where Rlr represents multiple LR images obtained under different lighting conditions at the same view, and the LR normal image Nlr is reconstructed by [Xie et al., 2014]. It can be seen that our mmVAE consists of three main components: 1) a modality processor denoted by mP(·), 2) a multimodal Gaussian mixture model denoted by mmGMM(·), and 3) a cross-scale modality encoder-decoder structure denoted by csME(·) and csMD(·), respectively.

The mP(·) module firstly extracts two auxiliary modality guidances: 1) normal angle guidance and 2) RGB frequency guidance. These two modality guidances have a similar distribution, which will be helpful to be fused in the subsequent network structure. Then, these two modalities with the LR normal image are upscaled to generate the corresponding modality inputs Aup, Fup and Nup, respectively. Next, similar to a classical encoder-decoder structure, these upscaled modalities are fed into the encoder module csME(·) to generate an internal cross-scale modality feature Xs, where s denotes a downsampling factor and X indicates their source modalities. After that, the mmGMM(·) module utilizes Xs in the same scale to align and fuse the related modality information. Therefore, mmGMM(·) will output latent probability variables Zs with different scales. Finally, Zs will be fed into the decoder module csMD(·) to restore an auxiliary normal image N′sr and the final normal image Nsr. In Sec. 3.3 and Sec. 3.4, we provide the details of mmGMM and our csME-csMD structure.

Loss Function. In order to train the proposed mmVAE in the manner of multimodal features and cross-scale structure, we devise a loss function combined with the modality loss and scale loss, which can be formulated as Eq. (2).

where Lmodality represents a modality loss, Lscale represents a scale loss, and λ1 is a positive scaling factor.

Inspired by the Kullback-Leibler (KL) divergence [Kullback and Leibler, 1951] and the VAE paradigm [Minnen et al., 2018], we design a two-part modality loss function Lmodality: 1) The first part adopts the paradigm of the VAE architecture converging the probability variables before the GMM module to N (0, 1) to obtain a more generalized Gaussian mixture model for each modality; 2) The second part directly minimizes the probability variables distance between the target normal and the other modalities after the GMM module, thus providing an explicit multimodality alignment. As a result, Lmodality is defined by

where s denotes a feature down-scale factors, and x denotes different modalities. µ and σ represent the mean and variance of a Gaussian distribution model of these probability variables, respectively. The details of them can be found in Sec.3.3. λ2 is a positive scaling factor.

Then, we design a scale loss function for mmVAE as

where the content loss Lcontent(·) is used to constrain Nsr and N′sr simultaneously. λ3 is a positive scaling factor. And the detail of the content loss is defined in Eq. (5).

where (H,W) represents the height and width of a predicted normal image. cos(·, ·) denotes the element-wise cosine operator calculating the difference between two normal images, and ↓ represents a downsampled normal map with the same size as Nlr. This back-projection error Lprj inspired by [Haris et al., 2018] is developed to constrain the downsampled SR normal modality close to LR one , which is used to preserve large surface structures. λ4 denotes a positive scaling factor.

3.2 Modality Processor (mP)

As aforementioned, due to the differences across different modalities, multimodal-based methods usually encounter negative influence in combining the latent features from different modalities. Because the SR task is focusing on reconstructing the high-frequency information of the input data, in order to reduce the data distribution differences and distill the valuable features from the normal image and RGB images, we represent them as a normal angle guidance Alr and an RGB frequency guidance Flr. It is worth noting that both Alr and Flr represent the high-frequency information of an input 3D surface.

Normal Angle. The extraction of the normal angle guidance consists of two steps: 1) generating a low-frequency normal image called a shape normal image Nshape, and 2) calculating the vector angle between the shape normal and the original normal.

Firstly, the generation of Nshape can be formulated as

where conv(·) represents a convolution operation and κave denotes an average filter kernel.

Secondly, we obtain the angle between Nshape and Nlr by

where the normal angle alr ∈ Alr for each pixel is the arccosine result of the dot-product of nlr fromNlr and the corresponding vector nshape fromNshape.

Based on above steps, the original normal image will be converted from Nlr ∈ RH×W×3 to Alr ∈ RH×W×1 representing the pixel-wise relief of a 3D object surface. It is noted that smaller values indicate smoother surface, while larger values represent the high frequency and contour information of a 3D surface. For instance, a toy normal angle map is provided in Figure 2.

RGB frequency. Due to the diversity of the object materials and surface geometry structures, the uncertainty of camera sensors or lighting conditions, the rawmulti-lighting photographs may contain many unfavorable issues, such as exposure errors, shadows from self-obscuring, speculator reflection, and uneven brightness due to different reflection intensities. However, those outliers also contain useful information. To fully utilize the input data, we develop a two-step scheme to denoise the RGB modality images and extract their valuable information: 1) extracting three brightness level images from Rlr , and 2) performing a masked DCT-transform to separate and combine their high-frequency features.

For the first step, we calculate a pixel-wise brightest image Rl, a darkest image Rd, and an average image Ra from the original RGB modality images, respectively. This operation aims to denoise and capture the different levels of surface material detail. For the second step, we convert these images from the spatial domain into the frequency domain. Then we mask the low 1/8 part on the frequency map, and leave the high-frequency part. By performing the invert-DCT, we can reconstruct a feature map with only the high-frequency information. Finally, these three features are concatenated to get the final RGB frequency guidance Flr. The RGB frequency guidance can be formulated as

where Tdct(·) and T ′ dct (·) represent the forward-DCT and invert-DCT respectively. The mask(·) manipulation removes the top-left 1/8 DCT frequency map which represents the low-frequency texture information. As a result, Alr and Flr represent high-frequency information from the surface normal and the texture structure of an input object, respectively. The related complement information will be upscaled and used as the auxiliary modality guidance to improve the subsequent network.

3.3 Multimodal GaussianMixture Model (mmGMM)

Mathematically, a classical variational autoencoder (VAE) network can be formulated as

where X denotes the input of VAE and Y denotes the output of VAE. Based on the VAE structure, a high-dimensional latent variable Z is fed into the decoder to generate a target result. This latent variable Z is a variable that is sampled from a probability model parameterized by the encoded data. This paradigm of encoding the input data and then sampling it from a probability model can reduce the noise of the latent features and further improve the generation capability of a learned model.

Generally, most VAEs adopt a Gaussian model N(µ, σ) [Pu et al., 2016] as the probability modeling of latent variables. Each dimension of the latent variable represents an implicit feature of the target probability distribution, which can finally fit the target distribution in the form of a Gaussian mixture model by decoders. However, due to the fact that the ability of these models to express features is relatively low, and they rarely considers the multimodal information, which makes them increase the number of feature layers but it is still difficult to fit the ideal potential distribution of the decoder. This further can limit their generation performance. Based on these considerations, we propose to use a multimodal Gaussian mixture model (mmGMM) as shown in Figure 3, which is used as the probability model for our mmVAE to generate the encoded feature more accurately. Due to the fact that a Gaussian mixture model can be expressed as a combination of multiple Gaussian models efficiently, we design the mmGMM module, and it can be formulated as a weighted summary of several Gaussian models

where Zxs = ∑Kk=1 ωx k N (µxk , σx k ), x from {A, F, N} denotes the normal angle, RGB frequency, and normal modality, respectively. s denotes a downsampling scalar.

Firstly, we adopt a channel attention block [Zhang et al., 2018a], which extracts the started feature from csME. Then with a ReLU layer followed by a convolution layer, we convert the related features to represent the mean µxk , variation σx k , and weights ωx k of a GMM distribution, where x denotes the corresponding modality and k represents the number of Gaussian components in a combined Gaussian mixture distribution. Each set of parameters will be used to resample a standard Gaussian distribution by a reparameterization trick [Kingma et al., 2015] and to calculate the weighted summary as a modality latent variable Zxs . This part is equivalent to compressing a large number of Gaussian distributions to obtain a lower-dimensional Gaussian mixture feature. Finally, with another convolution block, each Zxs will generate a modality weight ωx and calculate another weighted summary to obtain the corresponding fused latent variable Zs for the following csMD.

3.4 Cross-scale Modality-based Encoder-Decoder

To more effectively utilize the information in different scales of the fused modal features on the mmGMM structure, we propose a cross-scale network structure to drive the whole VAE operation. The cross-scale modality encoder (csME) encodes the input modal data into latent parametric features which control the probability distribution of mmGMM. Then, the latent features are resampled by mmGMM and decoded by the cross-scale modality decoder (csMD).

For the csME module, as shown in Figure 2, the input modality guidance is passed through a three-3×3-convolution layer to extract the related shallow angle feature, shallow frequency feature, and shallow normal feature from Aup, Fup, and Nup, respectively. Then, to allow the network to pay attention to both large structures and small details of a 3D object, we use several scaling blocks, which are composed of 2 convolution layers connected to a ReLU activation layer followed by a pooling layer, to resize the input features by the scale 12 . This conversion will generate several shallow features in different scales. For instance, Scale = 4, it will be the Xup ∈ R4H×4W×C → X2 ∈ R2H×2W×C and X2 ∈ R2H×2W×C → X4 ∈ R4H×4W×C , where X denotes one of the candidate modality features (i.e., X = {A, F, N}). These features will be fed into mmGMM to generate the corresponding fused latent variables Z2 and Z4.

For the csMDmodule, due to the fact that the effectiveness of a gradual refinement based on the hierarchical structure has been proven [Cai et al., 2019], we combine the high scaling and low scaling latent variables which aim to capture different levels of object details together to help the mmVAE focus both geometry shape and detail comprehensively, and thus obtain a better reconstruction result.

Firstly, for high scaling latent variables Z4 in csME, we employ two Conv blocks (consisting of three convolutional layers and one batch-normalization layer with one ReLU activation layer) to decode two latent features as the scalar and bias information. Then with a 3×3 convolution layer, the original input normal image Nlr is scaled and added by these features. After this feature enhancement, using a scale s upsampling block UPs(·) [Shi et al., 2016], an upscaled highfrequency feature is generated F′sr . Followed by a 3×3 convolution layer and added to the upscaled normal image Nup with a vector normalization operation, an SR normal image denoted as N′sr directly from scale ×4 information is generated. This process can be formulated by Eq. (11)

where γ denotes the vector normalization operation. ⊗ and ⊕ refer to the element-wise multiplication and addition, respectively. Cs(·) and Cb(·) denote the convolution module generating two latent features, and Co(·) denotes three output convolutional layers. After that, we adopt larger latent variables Z2 to refine the normal image generated by Z4. This process formulated by Eq. (12) will generate the final output upscaled normal image Nsr. The whole network is trained by optimizing the loss function in Sec. 3.1.

4 Experiments

In this section, we implement our mmVAE and compare it with recent 3D surface SR methods, including mesh-based, point-cloud-based, depth-based, normal SISR, and normal MISR methods.

4.1 Experimental Protocols

Implementation Details. Our mmVAE has been implemented in PyTorch, and the Adam optimizer is used with default parameters (β1 = 0.9 and β2 = 0.999) and K = 4 in mmGMMmodule. The detailed hyper-parameter settings of loss function are λ1 = 5, λ2 = 0.2, λ3 = 0.2, λ4 = 0.1. The initial learning rate is 3e-4 We have trained mmVAE by using a mini-batch size of 4 for 3000 epochs with one Nvidia Tesla P100 GPU, which takes about two days and nights. All the input images for training are adaptively cropped, randomly rotated (90◦ , 180◦ , and 270◦ ), and horizontally flipped. For instance, in the ×4 scale, the HR and LR image patches are 196×196 and 48×48, respectively. All the trained weights are initialized by the Kaiming distribution, and the bias is initialized as a constant. In addition to the above operations, we have not introduced additional training skills.

Datasets. Due to the fact that mmVAE is a data-driven SR scheme, it requires a lot of high-resolution normal image pairs. However, the most widely-used Photometric Stereo datasets, such as the DiLiGenT dataset (10 objects) [Shi et al., 2019] and the Gourd & Apple dataset (3 objects) [Alldrin et al., 2008], do not have enough objects for training. Therefore we use the latest wonderful photometric stereo dataset WPS for training [Xie et al., 2022], which contains 400 highresolution multimodal samples. To fairly evaluate the performance of these methods, we use DiLiGenT, Gourd & Apple and select 1 6 of the WPS dataset as the testing set. The rest of the WPS dataset is used as the training (validation) set. Both the training and testing data are down-sampled with the Bicubic (BI) degradation by × 12 and × 14 to generate the LR images as the network input. To perform a fair evaluation, all the compared methods are trained from scratch using the official settings and the same training iterations as mmVAE.

Evaluation Metrics. For quantitative comparisons, we adopt four quality measurements to evaluate the superresolution performance. In the 2D normal image domain, we take two commonly-used indicators in SISR for evaluation which are Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM). Besides, two widelyused metrics are used to quantitatively measure the 3D reconstruction results [Qu´eau et al., 2018], including Mean Angular Error (MAE) and Mean Relative Depth Error (MRDE). The final object surface is reconstructed by the Surface-fromNormal (SfN) method [Xie et al., 2014], [Cao et al., 2021]. Specifically, we have adopted the public available discrete geometry-based SfN method [Xie et al., 2014] to reconstruct an enhanced 3D surface. The MAE result is calculated by

where ˜ni,j and ni,j denote the predicted normal and the ground-truth normal, respectively. ||N|| represents the total number of the input normal pixels.

In addition, MRDE can be computed by

where ˜pi,j and pi,j represent the vertex position of the reconstructed surface and the ground-truth surface, respectively.

4.2 Baseline Settings

We have compared our mmVAE with 8 representative methods, which can be categorized into five groups: mesh-based method (denoted by "Mesh"), point-cloud-based method (denoted by "Points"), depth-based methods(denoted by "Depth"), SISR-based methods (denoted by "SISR"), and MISR-based methods (denoted by "MISR"). For mesh-based methods, Catmull-Clark subdivision (C-C) [Loop and Schaefer, 2008] is one of the most widely-used mesh subdivision methods, which can efficiently upscale a triangular mesh by a heuristic algorithm. For point-cloud-based methods, we choose the PU-GCN network [Qian et al., 2021] to represent the SR task of point clouds. In the experiments, we convert the related meshes into point clouds for PU-GCN, upscale, and re-convert them into meshes for comparison [Bernardini et al., 1999]. For depth-based methods, we choose the MSG network [Voynov et al., 2019] to represent the SR task in depth images. For normal SISR-based methods, we choose RCAN [Zhang et al., 2018a] to represent a convolutional attention structure and IPT [Chen et al., 2021a] to represent a self-attention structure. For normal MISR-based methods, we choose 3DASR [Li et al., 2019] to represent the hybrid fusion method, RefVAE [Liu et al., 2021] to represent a hybrid fusion method with the VAE structure, and MNSRNet [Xie et al., 2022] to represent the multimodal normal-based method with the transformer structure. The detailed experimental setting is consistent with [Xie et al., 2022].

4.3 Performance Comparisons

Figure 4 demonstrates the visual comparisons of some representative 3D objective surfaces. For SISR, due to the small size of the existing 3D training dataset, the CNN-based structure as RCAN showed better results than the transformer structure such as IPT. For MISR, they may not take full advantage of the additional multimodal information. The simple structure of 3DASR cannot fuse the modality information well and hence suffers from the negative effects of instability and confusion across multimodalities. MNSRNet with the transformer structure suffers from the size of the training set. As seen, RefVAE uses both VAE structure and multimodal information to obtain the best result in the baseline methods.

Table 1 provides the detailed average results on all testing datasets, including DiLiGenT, Gourd & Apple, and WPS. Specifically, our method achieves 11 of the first-best results and 1 of the second-best result in terms of PSNR, SSIM, MAE, and MRDE on both the ×2 and ×4 settings. As seen in Table 1, our method performs better than other methods in upsampling small images.

In addition, we have conducted the time and space comparison experiments on the DiLiGenT dataset. In the ×4 setting, MNSRNet takes a total of 20.9647s and 11445MB, while mmVAE takes only 8.2965s and 4265MB. It validates the effectiveness of the proposed cross-scale structure.

4.4 Ablation Study

mmVAE contains three main modules for multimodal learning, including mP, mmGMM, and cross-scale encoderdecoder structure. To verify the effectiveness of these modules, we further conduct additional experiments on the DiLiGenT dataset with the ×4 setting. Five independent experiments are conducted as shown in Table 2, where the related modules selected (not selected) are denoted by the symbol "√" ("×"). For the mmGMM module, we also performed ablation experiments for different hyperparameter K. It can be seen that with the increase of K, the performance has improved. However, this also has marginal effects, and too large K does not bring significant gains. Meanwhile, it is worth noting that when K = 1, the properties GMM will be lost.

In the ablation experiments, the replacement of mP is to use the lightest RGB images as the auxiliary guidance. The replacement of mmGMM is to directly concatenate all the related modality information, and then use three 3×3 convolution layers and one 1×1 convolution layer to shrink the intermediate channels similar to 3DASR. The replacement of the cross-scale structure is to use the results for scale 14 as the final outputs. In Table 2, the ablation result of mmGMM shows better than MNSRNet. That is because DiLiGenT has smaller image size, so the reconstruction results are more easily influenced by lowfrequency information. However, as the image size increases, the crossmodal information derived from mmGMM plays a more important role. For instance, mmVAE without the mmGMM module is less effective than MNSRNet on the WPS dataset, where the related PSNR/SSIM/MAE/MRDE results are 32.5947/0.9420/3.8024/2.2555, respectively.

5 Conclusion

In this paper, we have established a multimodal-driven variational autoencoder super-resolution framework to enhance 3D surfaces in the 2D normal domain. More specifically, we jointly considered the RGB and normal modalities to restore high-quality surface details as well as preserve fine-grained geometry structures. To effectively utilize the multi-modality information, we extracted two guidance features as auxiliary. To effectively fuse the related auxiliary information, we further investigated a multimodal Gaussian mixture model for 3D surface super-resolution. Moreover, we developed a cross-scale encoder-decoder structure to fully utilize the cross-modality information. Finally, we reconstruct an enhanced 3D object surface from the recovered high-resolution normal image. Experimental results on different benchmark datasets demonstrate the effectiveness of the proposed approach qualitatively and quantitatively.