Hybrid Mesh-Gaussian Representation for Efficient Indoor Scene Reconstruction

1 Introduction

Reconstructing a high-quality 3D representation with multiple unordered images is a critical task in computer vision and graphics. In recent years, an implicit representation approach called the neural radiance field (NeRF) [Mildenhall et al., 2020] shows extraordinary performance. NeRF combines the deep learning and volumetric rendering approach to produce high-fidelity images from novel views. However, the requirement of dense sampling of spatial positions along rays heavily slows down the training and rendering speed. By representing the scene with plenty of anisotropic 3D Gaussian primitives, 3D Gaussian splatting (3DGS) [Mildenhall et al., 2020] has emerged as a charming and efficient scene representation that achieves real-time rendering for scenes with unprecedented fidelity. 3DGS parameterizes the scene as an optimized set of

† Corresponding author.

3D Gaussians obtained from the structure from motion or randomly initialized. It efficiently renders all relevant Gaussian primitives into a 2D plane via splatting-based rasterization to obtain 2D images. [Waczynska et al., 2024; Gao et al., 2024; Fan et al., 2024] utilize the GS to yield accurate reconstructions to extract meshes to simulate physical interactions. Despite the compelling performance achieved by 3D-GS, the number of Gaussian primitives is excessively redundant [Fan et al., 2024; Fang and Wang, 2024]. For tile-wise sorting and pixel-wise alpha-blending in the splatting-based rasterization, massive Gaussian primitives can seriously slow down rendering. Several methods have explored the contributions [Fan et al., 2024; Fan et al., 2023] or important scores [Papantonakis et al., 2024] of each Gaussian to prune the redundant Gaussian for a compact representation. These methods focus on Gaussian attributes to reduce Gaussians. This paper proposes leveraging a GS-mesh hybrid representation to reduce Gaussians. For rich-texture flat areas, vanilla GS requires a large number of Gaussians to offer a precise representation, while textured mesh is capable of expressing high-frequency information through texture. Specifically, we utilize textured mesh (with an opacity set to 1) as the background with certain depths to reduce the number of Gaussians and accelerate the inference rendering speed.

As the distribution of 3D Gaussians shown in Fig. 1, the Gaussians mainly concentrate on complicated geometry and rich texture regions. To precisely represent complex geometries, a substantial number of fine-grained and elaborate meshes are required, which are not obtainable in real-world scene captures. Consequently, we still rely on the Gaussians to represent the complicated regions accurately. For texturerich areas, we use meshes to represent flat geometric structures and texture maps to express high-frequency content information to reduce the number of Gaussian primitives significantly. Nevertheless, an ideal mesh is not available in realworld scenes. Various methods have been designed to extract mesh from multiple images using NeRF [Mildenhall et al., 2020] or 3D-GS [Kerbl et al., 2023]. Owing to the weak supervision over texture-less areas and the challenges in representing geometrically complex regions, the extracted mesh exhibits noticeable geometric errors. We design some metrics to clear inaccurate and fine-grained meshes, preventing them from misleading the 3D representation. Then, we integrate the textured mesh into the 3DGS method to jointly represent the 3D indoor scenes. To reduce the Gaussians in front of the mesh, we employ texture supervision to narrow the visual disparity between the images rendered from the mesh and the ground truth. Additionally, we introduce a transmittanceaware mask to prevent the projection of front objects' colors onto the mesh. Compared to 3DGS, the proposed hybrid approach significantly reduces Gaussian primitives with a comparable quality.

In summary, our contributions are as follows:

• We propose a hybrid representation that leverages the advantages of Gaussian splatting and textured mesh to reduce the Gaussian primitives.
• We devise certain metrics to prune the flawed meshes in geometrically complex areas and retain them in texturerich flat regions to facilitate joint rendering.
• We introduce transmittance-aware supervision to facilitate joint optimization of the hybrid representation.

2 RelatedWorks

2.1 3DGaussian Splatting

Recently, 3DGS [Kerbl et al., 2023] has emerged as an advancement in novel view synthesis, achieving real-time, highfidelity rendering. Different from implicit neural fields with volume rendering based on ray marching, 3DGS represents the 3D scene with explicit Gaussians that can be rendered through tile-based sorting and rasterization processes. A plethora of subsequent studies have surfaced, innovating technology across diverse fields such as human avatar rendering [Luo et al., 2024; Liu et al., 2024], content creation [Li et al., 2024; Zhou et al., 2024], and scene rendering [Wu et al., 2024; Lin et al., 2025; Lin et al., 2024].

However, the performance of 3DGS models is constrained by the large number of Gaussians due to the rasterization process. [Bul`o et al., 2024] leverages a per-pixel error function as the density criterion and introduces a mechanism to control the total number of Gaussians. Taming 3DGS [Mallick et al., 2024] designs a steerable densification method that yields any desired number of Gaussians after training. [Papantonakis et al., 2024] assigns a redundancy score for each Gaussian to measure how necessary a Gaussian is to represent the scene. Then, Gaussians whose redundancy scores are higher than the adaptive threshold are pruned. Mini-splatting [Fang and Wang, 2024] proposes an intersection-preserving technique to retain Gaussians that act as intersection points and adopts an importance-weighted sampling approach to maintain a sparse set of Gaussians. LightGaussian [Fan et al., 2023] computes a global significance score of each Gaussian considering the Gaussian's volume, opacity, transmittance, and hit count of each training view. It then ranks all Gaussians by their global significance scores to quantitatively guide the pruning of the lower-ranked Gaussians. All these methods rely solely on GS to represent the 3D scene and take into account the attributions of Gaussians to prune the unimportant ones. As illustrated in Fig. 1, the Gaussian primitives predominantly concentrate on the texture-rich and geometrically complex regions. Our hybrid representation aims to reduce the Gaussians representing the texture-rich area via textured mesh while behaving the same as 3DGS in regions with intricate geometry. Theoretically, our method is perpendicular to previous approaches.

2.2 Hybrid representation

In order to combine the benefits of multiple 3D expressions (i.e. NeRF, mesh, GS), various hybrid approaches have been proposed [Wen et al., 2024; Dhamo et al., 2024; Wu and Tuytelaars, 2024]. VMesh [Guo et al., 2023] depicts an object with a textured mesh and an auxiliary sparse volume for efficient view synthesis. HybridNeRF [Turki et al., 2024] combines the best of surface and volume-based rendering into a single model to achieve a real-time rendering with high quality. SuGaR [Gu´edon and Lepetit, 2024] regulates the 3D Gaussians to align well with the surface to extract the mesh and jointly optimizes the Gaussians and the mesh. GSDF [Yu et al., 2024a] and NeuSG [Chen et al., 2023] improve the quality of surface reconstruction by combining the benefits of 3DGS with neural implicit fields. MeshGS [Choi et al., 2024] introduces several regularization techniques to precisely bind Gaussian splats with the extracted mesh surface for high-quality rendering. Some methods use a GSmesh hybrid representation to make the GS editable and controllable. GaMeS [Waczynska et al., 2024] parameterizes each Gaussian primitive by the vertices of the mesh face to allow modifying the Gaussians in a mesh manner. SplattingAvatar [Shao et al., 2024] disentangles the motion and appearance of an avatar with explicit mesh faces and implicit appearance modeling with GS. It can directly control the mesh to empower its compatibility with animation techniques. GaussianAvatars [Qian et al., 2024] places Gaussian splats onto a parametric morphable face mode to enable control in terms of expression, pose, and viewpoint of head avatars. HAHA [Svitov et al., 2024] models human avatars utilizing Gaussian splatting and a textured mesh for efficient rendering with much fewer Gaussians. This method assumes the availability of a water-tight SMPL-X parametric mesh and doesn't need to consider the occlusion of objects in the scene. Kim et.al [Kim and Lim, 2024] integrates mesh to represent the room layout and employs GS for other objects with a prior segmentation. Nevertheless, the textured mesh is only used for layouts that demand minimal Gaussians, and the manual acquisition of the mesh from a synthetic dataset makes it impractical for real-world scenes. Our method leverages mesh to dominate the texture-rich area, effectively reducing the number of Gaussian primitives while maintaining a comparable rendering performance.

3 Methodology

Taking multi-view images with calibrated camera poses and sparse point clouds derived from the structure from motion (SFM) algorithm as input, we aim to combine the 3DGS and textured mesh to accurately represent indoor scenes while maintaining rendering quality and improve efficiency.

3.1 Preliminary

3D Gaussian Splatting 3DGS [Kerbl et al., 2023] utilizes anisotropic 3D Gaussian primitives to represent the 3D scene, achieving state-of-the-art visual quality and rendering efficiency. Each explicit Gaussian primitive is characterized by the following contributions: position center u ∈ R3 , opacity α, orthogonal rotation matrix R, diagonal scale matrix S, and spherical harmonics (SH) coefficients. Each Gaussian is defined by u and covariance matrix Σ ∈ R3×3 as:

To guarantee the physical meaning of the covariance matrix during optimization, it is formulated as Σ = RSST RT to keep a positive semi-definite character. To render images, 3D Gaussian primitives are projected onto the 2D image plane with the Jacobian affine approximation J of the project matrix and view transformation W: Σ′ = JWΣW T J T . Then alpha blending is applied from front to back based on the sorted depth to render the color of each pixel as follows:

where N denotes the number of sorted Gaussians related to the pixel p, ci denotes the color of the projected 2D Gaussian G′i . The L1 loss and D-SSIM term between the ground truth image I and the rendered image ˆ I is utilized to optimize 3D Gaussians primitives:

where λ is set to 0.2 as default.

Texturedmesh Amesh is a collection of vertices and faces that provides the geometric description of 3D models. A texture map is an image that maps the detailed visual information of images onto the surface of a mesh. To render the triangle meshes with a texture map, barycentric interpolation is applied to accurately assign texture colors to individual pixels in the rendered image based on UV coordinates. In this way, textured meshes enable fast real-time rendering of flat regions with realistic visual results. However, it struggles to model intricate structures, which we rely on the Gaussians to represent.

3.2 Overview

Our hybrid representation depicts indoor scenes with triangular mesh surfaces and anisotropic Gaussian primitives. In this section, we first clear the flawed meshes at geometrically complex regions to make room for Gaussians and refine the remaining ones at the texture-rich region to optimize the texture map. Then, we propose a transmittance-aware texture loss equipped with a warm-up to optimize the hybrid representation jointly. The overall pipeline of our approach is shown in Fig. 2.

3.3 Mesh Pruning

Recent NeRF and Gaussian Splatting (GS) approaches extract meshes composed of numerous triangles from multi-view images. However, due to weak supervision in texture-less areas and challenges in representing complex geometries, the extracted meshes often exhibit significant geometric errors that adversely impact the hybrid representation. Furthermore, Gaussians outperform textured meshes in modeling geometrically complex and thin regions. To address these issues, we propose a pruning strategy that considers normal maps, adjacent angles, and mesh sizes to reduce the extracted mesh, thereby allocating space for Gaussian primitives to handle intricate regions.

We adopt the Planar-based Gaussian Splatting (PGSR) [Chen et al., 2024] as the baseline for initial mesh extraction. In texture-less regions, Gaussian primitives exhibit large scales S, resulting in numerous mesh bumps with incorrect geometry that occlude surrounding objects. To mitigate this, we first prune the mesh by removing Gaussians exceeding a predefined scale threshold and subsequently extract the mesh using the truncated signed distance function (TSDF). This process yields a coarse mesh with minor bumps and rough surfaces, leading to increased disk storage and slower rendering speeds.

To further streamline the mesh, we apply a quadric errorbased simplification method called QSlim to reduce the number of triangles to a desired count K, while preserving geometric topology. Next, we eliminate mesh segments representing complex geometries by analyzing the total variation of the normal map, angles between adjacent triangles, and triangle sizes. For each training view, we project the mesh onto 2D images to establish a mapping from pixels to triangles, enabling targeted pruning based on specific metrics.

Specifically, regions with high normal variation, indicating complex geometries, are identified using a prior normal map from the pre-trained StableNormal diffusion model. The top αnormal percent of pixels with the highest total variation are marked for triangle removal. Additionally, triangles with angles exceeding 45◦ between adjacent faces are pruned to eliminate boundaries of flat regions or complex geometries. To remove trivial geometries that hinder texture map optimization as occluders, we sort and prune the smallest αarea percent of triangles based on their area sizes.

This comprehensive pruning process may result in floating triangles and tiny holes. To resolve these issues, we remove isolated connected triangles with counts below a minimum threshold αgroup and close any resultant holes in the mesh to prevent floating geometries. Finally, we apply the LS3 subdivision surface algorithm to smooth the mesh further. All pruning and refinement operations are automated with predefined hyperparameters, resulting in a smooth mesh that predominantly represents texture-rich flat areas.

To incorporate high-frequency visual information from multi-view images, we attach texture coordinates (UV maps) to the mesh using Xatlas. We then initialize the texture map using Nviffrast, a differentiable rasterization tool, under image supervision. Consequently, we obtain a smooth and discrete mesh that effectively represents texture-rich flat regions, allowing us to gradually densify Gaussian primitives using the textured mesh as a background with fixed depth values.

3.4 Joint optimization

Joint rasterization As shown in Fig. 2, we simultaneously optimized the 3DGS and the textured mesh in a differentiable way. For each training view, we first rendered the mesh to get a RGB image Im and a depth map Dm and made the texture map optimizable. During the rasterization of GS, if the pixel is equipped with a triangle, we used the alpha-blending to produce an RGB value Igs of Gaussian primitives before the triangle (Dgs<Dm) and got a transmittance coefficient T applied to the mesh. Otherwise, the rasterization performs as the vanilla GS. In short, we regard the mesh as an opaque background with a certain depth value. The final rasterization result is given as follows:

where Ibg denotes the predefined background color. Following the training strategy of vanilla GS, we replaced the ˆ I in eqn. 3 with the Ih to provide the supervision.

Transmittance-aware supervision Under the photometric supervision of the hybrid representation, the optimization process prefers to cover the front of the mesh with numerous Gaussian primitives. It will reduce the mesh's contribution to each pixel. To reduce the Gaussian primitives in front of the mesh, we adopt a ∥˜ I, ˜ Im∥22 norm loss between the masked ground truth ˜ I and the masked texture image ˜ Im. ˜ I equals to I when triangle is available for this pixel, else ˜ I equals to 0.

However, simply enforcing the rendered mesh to be close to the ground truth image will induce some ghosts on the mesh, shown in Fig. 5. When the explicit object in front of the back mesh is absent, in most cases in our settings, since we pruned the mesh with complex geometry. This supervision will project the object onto the back meshes along the ray from the camera center. When the mesh is viewed from other perspectives, we will see a ghost object that should not exist. First, we warm up the training only with the photometric loss for the first N iterations to make up the absent object with Gaussians. Then we proposed a transmittance-aware texture loss Lt to prevent the incorrect supervision hindering the optimization of the texture map as follows:

MT denotes a transmittance-aware mask for each pixel. If the T is larger than 0.5, we prefer to rely on the textured mesh to dominate the pixel to reduce the Gaussian primitives. Otherwise, we hope to use the hybrid representation. Therefore, we used a sigmoid function to get the mask for each pixel:

Since we adopt the L2 to reduce the Gaussian primitives and use hybrid representation to render the final image, we set λ to zero after densification iteration (i.e 15k) of 3DGS training. Combined with the warm-up strategy, some Gaussian primitives will be created at the object's location, making the transmittance for the back mesh less than 0.5. Therefore, the object's color has a negligible impact on the optimization of back meshes. The joint optimization is supervised by:

where λ controls the balance between the hybrid representation and the textured mesh.

4 Experiments

We first present the details of the proposed hybrid representation. We then assess performance for indoor scenes on the Deep blending dataset [Hedman et al., 2018] and the challenging Scannet++ dataset [Yeshwanth et al., 2023].

4.1 Implementation Details

We build our method upon the open-source 3DGS code. Following [Kerbl et al., 2023], we train our models for 30K iterations across all scenes and use the same densification, schedule, and hyperparameters. We follow all the default settings of PGSR [Chen et al., 2024] to extract the initial mesh. We use the Nvdiffrast [Laine et al., 2020] to enable differentiable rendering of the mesh and the optimization of the textured map. Following [Papantonakis et al., 2024], the FPS measurements isolate the runtime of the CUDA rasterizer routine only and exclude any graphics API overheads. The final mesh contains about a hundred thousand triangles with a texture map of size 3 × 2048 × 2048. We follow standard practice and report SSIM, PSNR, and LPIPS for rendering evaluation. All experiments are conducted on a single V100 GPU. More detailed information are provided in supplementary A.

Datasets. We verify the effectiveness of our approach using ten real-world indoor scenes from publicly available datasets: four scenes from the Deep blending [Hedman et al., 2018] and six scenes from ScanNet++ [Yeshwanth et al., 2023].

4.2 Quantitative Results

Rendering Performance. The quantitative results of various methods are presented in Tab. 1. We evaluated the standard metrics such as SSIM, PSNR, LPIPS, Gaussians number, and FPS, comparing against the baseline 3DGS and other Gaussians methods. Tab. 1 demonstrates that, in comparison to the 3DGS baseline, our hybrid representation effectively cuts down around 18% and 35% of Gaussian primitives used in 3DGS while maintaining comparable performance. We then incorporate the textured mesh into [Papantonakis et al.,

2024] with our joint optimization strategy to further decrease the number of the Gaussians by about 20% and 50% with a negligible impact on performance. Additionally, the realtime rendering speed is accelerated as a result of using fewer Gaussian primitives. We provide more detailed experimental results and analysis about integrating the mesh into GS methods in the supplementary C.

Since Mip-splatting performs similarly to 3DGS, except for anti-aliasing, we demonstrate the rendered images and Gaussian distributions of 3DGS and Reduced GS in Fig. 3. the The hybrid representation can effectively reduce the Gaussians of the texture-rich regions.

4.3 Qualitative Results

In addition to reducing the Gaussian primitives, the textured mesh demonstrates a sharper representation compared to the purely Gaussians. This is particularly evident in flat regions that are predominantly represented by the mesh. The proposed transmittance-aware supervision plays a crucial role by directly translating the high-frequency visual details from the images into the texture map. As the visual comparisons illustrated in Fig. 4, the mesh enhanced with a texture map is capable of rendering high-fidelity images, particularly in texture-rich flat regions. In contrast, rendering using Gaussian primitives tends to exhibit blurring and artifacts. This demonstrates the advantage of incorporating textured meshes in our hybrid approach to achieve superior visual quality in areas where detailed texture is essential. More visual results are provided in supplementary C.2.

4.4 Upper Bound

To evaluate the upper bound of the hybrid representation and the proposed joint optimization strategy, we conduct experiments on the Replica dataset [Straub et al., 2019], a synthesis indoor scene dataset with ideal meshes.

We also apply the mesh prune onto the ideal mesh and then jointly optimize the GS and mesh to evaluate novel view renderings. Results in Tab. 2 show that if the mesh possesses an accurate geometry, the hybrid representation can achieve better performance, faster rendering with much fewer Gaussians.

4.5 Ablation

We conduct detailed ablation studies using four indoor scenes from the deep bleending dataset to validate the key components of the proposed approach.

Mesh prune & transmittance-aware supervision. We present the ablation study of our mesh prune strategy and transmittance-aware supervision in Tab. 3. For the raw mesh, we only use QSlim decimation to make it have the same number of faces as the mesh pruned by our method for a fare comparison. The results on deep bleeding demonstrate a marked improvement (around 1 dB) in rendering quality compared to the results of a naive combination of mesh and GS. For raw mesh without the transmittance-sensitive supervision, Gaussians were unable to compensate for errors caused by incorrect geometry in intricate parts (e.g., telephone toy) and missing local objects (e.g., wires) shown in Fig. 5. The object's color is wrongly projected onto the back mesh. The mesh pruning strategy can remove intricate parts, which leads to improved performance. We illustrate several visual comparisons of meshes before and after prune in supplementary B.

We evaluate the effectiveness of the proposed transmittance-aware supervision on the pruned mesh. If MT is set to 1, there is an inability to distinguish between the foreground elements and the background components for each pixel of the training images. All pixel values are projected back to the mesh, resulting in inferior performance. The nonlinear sigmoid function performs better by acting as a soft indicator function.

Texture loss. As illustrated in Tab. 4, we assess the balance between the photometric loss and the texture loss by varying the coefficient λ. Without the supervision between the mesh and ground truth images (λ = 0), it will create more Gaussians in front of the mesh, thus weakening the representation of the textured mesh. Conversely, a high λ will lead to a high transmittance to the mesh, which will make the transmittanceaware mask degrade to a constant MT = 1 mask. Besides, it forces images rendered solely from the textured mesh to overfit the ground truth and potentially impact the rendering of the hybrid representation.

Texture initialization. We compare three methods for initializing the texture map. The first sets all values in the texture map to 0.5. The second uses Xatlas to get UV coordinates and then optimizes the texture map with Nvdiffrast. The third method utilizes texrecon [Waechter et al., 2014] to generate texture maps automatically. The first two methods allow setting the size of the texture map (set to 2024 × 2024 × 3 as default), while texrecon generates multiple texture maps that need to be merged manually. The results are shown in Tab. 5, we use the Xatlas as the default choice.

5 Conclusion

In this paper, we propose a hybrid representation that integrates textured meshes with 3D Gaussian Splatting (3DGS) to reduce the Gaussian primitives for indoor scenes. Based on the observation that texture-rich flat regions demand numerous Gaussian primitives to accurately capture significant color variations, we employ textured meshes to serve as background elements with specific depth values. This strategy substantially reduces the required number of Gaussians and accelerates rendering speeds. Additionally, we developed a mesh pruning method that considers normal maps, adjacent angles, and mesh sizes to eliminate meshes in geometrically intricate regions, thereby allocating space for Gaussian primitives. Our joint optimization technique ensures that both the textured mesh and Gaussian splats contribute effectively to the final representation, leveraging their respective strengths. Experimental results on the Deep Blending and ScanNet++ demonstrate that our hybrid approach maintains comparable rendering quality while significantly reducing the number of Gaussian primitives.

Limitations. While our method excels in scenes with prominently texture-rich flat regions, it is less effective in environments dominated by multiple intricate structures unless a high-quality mesh is provided. Furthermore, our current approach does not optimize vertex coordinates or account for view-dependent effects of meshes. Addressing these limitations will be the focus of our future research, aiming to extend the applicability and robustness of our hybrid representation across a wider range of complex scenes.

Acknowledgments

This work was supported in part by the Theme-based Research Scheme (TRS) project T45-701/22-R of the Research Grants Council (RGC), Hong Kong SAR.