FastScene: Text-Driven Fast Indoor 3D Scene Generation via Panoramic Gaussian Splatting

1 Introduction

3D models have a wide range of applications in video production, gaming, AR/VR, and other fields. However, generating high-quality 3D models typically requires professional designers to utilize specialized software with a considerable amount of time, which is inconvenient for those seeking fast

∗Corresponding author

3D model generation. The development of generative models makes Text-to-3D object generation [Poole et al., 2022], [Lin et al., 2023] possible and impressive. However, the generation of 3D scenes still presents significant challenges, requiring large-scale scene reconstruction, multi-view images, and the assurance of scene realism and consistency.

Recently, some works attempt to tackle the 3D scene generation challenges. Set-the-Scene [Cohen-Bar et al., 2023] applies global-local training from text prompts and 3D object proxies, while generating controllable scenes. However, the quality and resolution of the generated scenes are unsatisfactory due to the lack of corresponding geometry. SceneScape [Fridman et al., 2024] generates long-range views, producing diverse styles. However, its view quality decreases over time due to the inpainting and depth estimation error accumulation. Text2Room [H¨ollein et al., 2023] and Text2NeRF [Zhang et al., 2024] gradually generate perspective novel views. Nevertheless, their incremental local operations hardly ensure scene consistency and coherence. CtrlRoom [Fang et al., 2023] fine-tunes ControlNet [Zhang et al., 2023] for editable panorama generation, and then performs mesh reconstruction. However, Ctrl-Room tends to flatten the 3D model with limited scene quality, since it hardly generates multi-view images.

As one of the 3D representation techniques, the radiance fields methods, exemplified by Neural Radiance Fields (NeRF) [Mildenhall et al., 2020], have made significant breakthroughs. Since most NeRF-based methods suffer from slow rendering speed [Mildenhall et al., 2020], [Barron et al., 2022], rendering process acceleration becomes an important issue. Recently, 3D Gaussian Splatting (3DGS) [Kerbl et al., 2023] has achieved success in the rendering speed with highquality. However, the typical 3DGS only takes regular images as the input. It faces challenges when handling panoramas, which are difficult to handle with existing Structure-fromMotion (SFM) [Snavely et al., 2006] methods.

To address the above issues, we propose a novel Text-to-3D scene framework, called FastScene, which aims at fast generating consistent and authentic scenes with high-quality. As shown in Figure 1, our approach primarily comprises three stages. 1) In the first stage, given a text prompt, we generate a panorama by utilizing the pre-trained Diffusion360 [Feng et al., 2023]. Panorama is selected due to its ability to capture the global information and exhibit explicit geometric constraints, which is advantageous in overcoming the scene inconsistency issue of perspective view. Then, we adopt EGformer [Yun et al., 2023] for panorama depth estimation. 2) In the second stage, we propose Coarse View Synthesis (CVS) to generate novel panoramic views with holes for specific camera poses. Since large-distance novel views generation results in numerous holes, which is not conducive to inpainting, we propose Progressive Novel View Inpainting (PNVI) to gradually fill the holes within a small distance. Nevertheless, we experimentally find that caused by cumulative distortion errors, directly inpainting panorama usually results in edge blurring and distortion. Instead, we propose to perform inpainting in cubemap, and utilize Cube-toEquirectangular (C2E) to obtain the corresponding inpainted panorama. We then replace non-hole pixels in the inpainted panorama with their original values. Furthermore, we synthesize a dataset that aligns with our hole distribution, and retrain AOT-GAN [Zeng et al., 2022] for inpainting. 3) After acquiring multi-view panoramas, we employ 3DGS for fast scene reconstruction. However, the original COLMAP [Schonberger and Frahm, 2016] only supports perspective views, making it challenging to obtain point clouds from panoramas. Therefore, we introduce Multi-View Projection (MVP), which divides the panorama into perspective views, enabling feeding into 3DGS for reconstruction. MVP as a plug-and-play module can be easily applied without requiring additional computational resources. Extensive experiments validate that our method can fast generate high-quality 3D scenes while ensuring scene consistency.

Our contributions can be summarized as follows:

1. We propose a novel Text-to-3D indoor scene framework FastScene, enabling fast and high-quality scene gen-
2. eration, while ensuring scene consistency. Additionally, given the text prompt, there is no need to predesign complex camera parameters or motion trajectories, which makes FastScene a user-friendly scene generation paradigm.
3. We propose a novel panoramic view synthesis method PNVI, which adopts CVS to generate novel views with holes, and performs precision-controllable progressive inpainting to generate refined views. Additionally, to improve the inpainting quality, we synthesize a largescale distribution-based spherical mask dataset.
4. To the best of our knowledge, we are the first to solve panoramic 3DGS from a single panorama, and the proposed FastScene is highly adaptable to existing panoramic data for reconstruction.

The rest of this paper is organized as follows. Section 2 briefly reviews the related works of this paper. Section 3 introduces the design details of the proposed FastScene. Section 4 provides experimental results for comparisons and ablation study. Conclusions are summarized in Section 5.

2 RelatedWorks

2.1 Text-Driven 3D scene Generation

Recently, there has been considerable focus on 3D scene generation. Set-the-Scene [Cohen-Bar et al., 2023] introduces an agent-based global-local framework to synthesize controllable 3D scenes, while enabling diverse scene editing options. However, it suffers from shortcomings in the quality and resolution of generation scenes without corresponding geometry. While SceneScape [Fridman et al., 2024] generates consistent views by introducing a pre-trained text-to-image model

[Rombach et al., 2022], and possesses the capability to generate scenes in various styles. However, the view quality of SceneScape is reliant on geometric priors and diminishes over time due to the inpainting and depth error accumulation. More recently, both Text2Room [H¨ollein et al., 2023] and Text2NeRF [Zhang et al., 2024] rely on incremental frameworks to generate new perspectives on a per-image basis. However, their incremental local operations can hardly guarantee scene consistency and coherence. Later on, Ctrl-Room [Fang et al., 2023] proposes to encode text input and convert scene code into a 3D bounding box for editing. Subsequently, it generates panoramas by fine-tuning ControlNet [Zhang et al., 2023], and reconstructs mesh through Possion reconstruction [Kazhdan et al., 2006] and MVS-texture [Waechter et al., 2014]. However, Ctrl-Room struggles to generate highquality 3D models, and tends to flatten the 3D model due to the limited number of generated views.

2.2 Text-Driven Panorama Generation

Unlike 2D images, panoramas cover the 360° × 180° field of view, which provides more 3D scene information. Text2Light [Chen et al., 2022] synthesizes panorama images from text input via a multi-stage auto-regressive generative model. However, it ignores the boundary continuity of the panorama, resulting in an open-loop content. MVDiffusion [Tang et al., 2023] generates high-resolution panoramas by fine-tuning a pre-trained text-to-image diffusion model. However, artifacts usually appear on the "sky" and "floor" views, which decreases the realism of generated scenes. StitchDiffusion [Wang et al., 2024b] crops the left and right sides of the panorama to maintain the scene continuity. However, the cracks at the seams are still noticeable. Diffusion360 [Feng et al., 2023] proposes a circular blending strategy to maintain the geometry continuity, which generates high-resolution boundary-continuous panoramas.

2.3 Novel View Synthesis

Novel view synthesis is a popular area of significant interest. Early methods rely on multi-view images and attempt to incorporate the knowledge from epipolar geometry to perform smooth interpolation between the different views [Chen and Williams, 1993], [Debevec et al., 1996]. Some methods synthesize novel views by deep networks from a few images [Sajjadi et al., 2022], [Mirzaei et al., 2023]. In contrast, [Gu et al., 2023], [Shen et al., 2023] allow for generating novel views from a single image.

A significant breakthrough in novel view synthesis is NeRF [Mildenhall et al., 2020] and its derivative works [Barron et al., 2021], [Barron et al., 2022], [Chen et al., 2023]. The rendering speed of most radiance-based methods is slow, accelerating rendering becomes an important but challenging problem, with representative works such as Instant-NGP [M¨uller et al., 2022] and 3DGS [Kerbl et al., 2023]. Some NeRFbased works [Wang et al., 2024a], [Chen et al., 2024] attempt to synthesize panoramic novel views. However, since SFM struggles to handle panoramas due to its unique structure [Snavely et al., 2006], it is difficult to utilize original 3DGS for panorama rendering.

3 Method

3.1 Overview

As shown in Figure 1, given a text prompt P, we first use Diffusion360 [Feng et al., 2023] to generate the corresponding panorama S0, and then employ EGformer [Yun et al., 2023] to estimate the depth map D0. Thereafter, given a new camera pose Pn, we perform CVS to obtain the corrupted panorama Sn with holes. To fill these holes, we propose PNVI, which gradually inpaints perspective cubemap views Sni (i = 0, 1, ...5) rather than directly inpaints the panorama. Subsequently, these clean cubemap images are reprojected equidistantly to obtain the clean panorama. We then replace non-hole pixels in the inpainted panorama with their original values to obtain the novel panorama Sn. Similarly, iterating PNVI multiple times results in numerous novel clean panoramic views. As COLMAP [Schonberger and Frahm, 2016] does not support panoramic inputs, we employ MVP to generate the corresponding perspective views, followed by 3DGS to implement the 3D scene reconstruction.

3.2 Text-Driven Panorama Generation and CVS

Compared to perspective views, a key geometric characteristic of panorama is the continuity of the boundaries. Additionally, the panorama encompasses information about the entire scene and exhibits explicit geometric constraints, which is beneficial for our subsequent processing. Thus, we utilize Diffusion360 [Feng et al., 2023] for text-to-panorama generation, which adopts the blending strategy to maintain the geometry continuity. After that, we estimate the depth map using EGformer [Yun et al., 2023] to capture the spatial information of the scene. Then, we propose CVS to obtain a new panoramic view under a given camera pose, as shown in Figure 2. According to the theory of equidistant projection on the spherical panorama, we can project a 2D image of size 1024 × 512 onto a sphere, where the latitude range is 180◦ and the longitude range is 360◦ . The calculation for the latitude angle θa and longitude angle ϕa are as follows:

where xa and ya represent the image coordinates of coordinate system a, while W and H represent the width and height of the panorama, respectively. We then utilize the triangle transformation to obtain the spherical basis coordinates:

afterward, we multiply the depth d by the 3D coordinates ax, ay, az to initial the spherical coordinates Ca:

Given a new camera pose Pn, we take it as the origin of the new spherical coordinate system n, and subtract the original coordinates Ca from the new origin Pn to get the new spherical coordinates:

Then, we reproject the coordinates Cn to the new coordinate system n:

where θn and ϕn denote the latitude and longitude of the novel view, and xn and yn represent the image coordinates of coordinate system n.

We summarize equations (1) to (11) as a mapping F from (xa, ya) to (xn, yn):

Therefore, we only need to determine if the mapped pixels (xn, yn) lie within the panorama. If they are inside, we keep the normal RGB values, otherwise we set them as holes with a value 255:

Correspondingly, we can obtain the mask image Mn (with value 0 for normal regions and 1 for unseen) for inpainting:

3.3 Progressive Novel View Inpainting

After CVS, we obtain multi-view panoramas with holes. To reconstruct the scene using 3DGS, we need to fill these holes. To this end, we propose the PNVI to obtain clean novel views. Due to the lack of indoor panoramic datasets with mask information to retrain the inpainting network, we construct a new dataset, as detailed in Section 4.2. We endeavored to conduct direct panorama inpainting, yet observed that with an increasing distance of movement, a plethora of spurious shadows manifested along the peripheries of the panorama. Therefore, E2C is utilized to obtain six cubemap images from one panorama, and cubemap inpainting is conducted using the retrained AOT-GAN [Zeng et al., 2022]. After that, C2E is utilized to form the panorama. Finally, we replace non-hole pixels in the inpainted panorama with their original values to obtain the novel panorama Sn.

However, when directly moving the camera to large poses, the hole-to-image area ratios become extensive, raising difficulties for inpainting, irrespective of the model training quality. To address the aforementioned issue, we propose a progressive inpainting mode, as shown in Figure 3, which enables inpainting in large camera poses. Specifically, assuming we move the camera along the X-axis by a distance of 0.33 meters, the hole-to-image area ratio of the novel view image increases to 64.3%, which means more than half of the images are with holes, as reported in Table 1. Therefore, we decide to divide the long distance into small moves (e.g., 0.02 meters per move) to relieve the long distance inpainting difficulty. In this way, the hole-to-image ratio is only 15% at each move. By progressively moving from P0 to Pn, we can obtain a clean view at the endpoint.

3.4 Panoramic 3DGaussian Splatting

The original inputs for 3DGS are multiple RGB perspective views. Following the COLMAP [Schonberger and Frahm, 2016] pipeline, sparse point clouds and camera parameters are obtained. Nevertheless, algorithms within COLMAP pertaining on perspective views exclusively exhibit inadequacies when confronted with panoramic perspectives, leading to a disorderly reconstruction outcome. As shown in Figure 4a, assuming the camera moves along x, y, andz axes, the adoption of the original COLMAP fails to produce accurate point clouds and camera poses. This arises from the distinctive distortions and intricacies inherent in panoramas, making the application of conventional SFM arduous in the endeavor to align spatial information across diverse viewpoints.

Therefore, we introduce MVP to solve the aforementioned problem. Specifically, given the panorama S with size W × H and the requirements for n perspective images (V1, V2, ..., Vn) with size R × R. Firstly, we calculate the rotation matrix Ri for each camera. For each perspective view Vi(1 ≤ i ≤ n), we define a projection mapping function P(S, Vi), which maps the pixels of the panorama to the perspective view. By projecting the panoramic pixels (m, q), (0 ≤ m < W, 0 ≤ q < H), new projected coordinates (j, k), (0 ≤ j, k < R) for the perspective view can be obtained. Then, we send the multi-view perspective images to COLMAP to obtain the point clouds required by 3DGS. As shown in Figure 4b, for multi-view panoramic inputs, our method enables the generation of accurate point clouds and camera poses, thereby allowing for seamless processing using 3DGS. The loss function L is defined as the weighted sum of L1 and LD-SSIM [Wang et al., 2004]:

we follow [Kerbl et al., 2023] to set λ = 0.2.

4 Experiments

4.1 Implementations Details

We implement our method on PyTorch. We use the pretrained Diffusion360 [Feng et al., 2023] and EGformer [Yun et al., 2023] for panorama generation and depth estimation, respectively. We retrain the AOT-GAN [Zeng et al., 2022] on our synthesized dataset for inpainting, described in Section 4.2. We choose CLIP Score [Hessel et al., 2021], Natural Image Quality Evaluator (NIQE) [Mittal et al., 2012b], and Blind/Referenceless Image Spatial Quality Evaluator (BRISQUE) [Mittal et al., 2012a] to evaluate the rendering quality in an unsupervised manner. It takes about 15 minutes to generate a complete scene on a single NVIDIA RTX A6000 GPU with 49G memory. Specifically, panorama generation takes 10 seconds, the PNVI process takes approximately 2 minutes, acquiring 3DGS training data requires around 3 minutes, and scene generation takes 10 minutes.

4.2 Panoramic Inpainting Dataset

Due to the absence of panoramic datasets with our mask distribution, it is essential to generate a corresponding dataset.

Specifically, we select the synthetic dataset Structured3D [Zheng et al., 2020], which comprises 21k photorealistic panoramic scenes. We select 14k images with complete scenes that are more realistic. Subsequently, for each panorama, we generate 16 types of masks using equations (12) to (14), corresponding to eight movement directions on the coordinate axis, with two movement units of 0.02m and 0.04m for each direction. Then we perform E2C projection for each panorama and mask image. Finally, there are a total of 84k perspective RGB images and 1344k masks. After obtaining the dataset, we retrain AOT-GAN [Zeng et al., 2022], with all training and testing sizes set as 512 × 512.

4.3 Comparisons with Other Methods

To validate the effectiveness of our method, we conduct quantitative and qualitative comparisons with previous indoor scene generation methods, including Text2Room [H¨ollein et al., 2023], Set-the-Scene [Cohen-Bar et al., 2023], and SceneScape [Fridman et al., 2024]. We render 30 images of each scene for evaluation.

Quantitative Comparison. By giving an identical text prompt input, we test the generative performance of different methods. Since conventional image quality assessment metrics, such as PSNR and SSIM, are not applicable to our task, we adopt unsupervised evaluation metrics.

As reported in Table 2, Text2Room performs modestly due to the lack of global consistency. SceneScape suffers from decreased image quality caused by accumulated errors during long-distance movements. Set-the-Scene exhibits limited perceptual performance due to its lower resolution and texture quality. On the contrary, our method not only achieves superior performance in terms of CLIP Score, NIQE, and BRISQUE metrics, but also demonstrates the fastest generation speed. A fast generation process is important, since it is an obvious advantage for user-friendly tasks.

Qualitative Comparison. Furthermore, to comprehensively validate the performance of our FastScene, we present the qualitative comparison results with other scene generation methods. We provide the same text prompt, such as common indoor scenes: bedroom, living room, dining room, etc., and then obtain the generation results of different methods. As shown in Figure 5, Text2Room [H¨ollein et al., 2023] can generate faithful local views, but it fails to ensure consistency across the entire scene. SceneScape [Fridman et al., 2024] has the ability to generate long-range immersive views. However, as the distance increases, the accumulation of errors results in a detrimental loss of details. Set-the-Scene [CohenBar et al., 2023] possesses the ability to generate editable scenes. However, its rendered images are blurry and texture quality is inadequate to meet perceptual needs. In comparison, our method generates high-quality scenes in a fast way, and ensures the scene consistency as well. More scene generation results can be found in our Supplementary Material.

In conclusion, both quantitative and qualitative comparison experiments confirm that our method can rapidly and effectively generate globally consistent scenes with high-quality.

Table 2: Quantitative comparison with other methods, with all results tested on the same hardware device.

4.4 Extension Experiments on Panoramic Datasets

To validate the adaptability of our PNVI and MVP on existing panoramas for 3DGS, we conduct extension experiments on the Matterport3D 1k, 2k [Chang et al., 2017], and Replica360 4K [Straub et al., 2019] datasets, containing panoramas at resolutions of 1K, 2K, and 4K, respectively. As shown in Figure 6, our method is capable of reconstructing 3D scenes

from panoramas at different resolutions.

(a) Matterport3D, 1K

Figure 6: The reconstruction results of indoor panoramic datasets, validating that our method can effectively transfer to 360° datasets.

Furthermore, to demonstrate the effectiveness of our method, we compare the performance with panoramic novel views synthesis works on Replica360 4K: DS-NeRF [Deng et al., 2022], SinNeRF [Xu et al., 2022], DietNeRF [Jain et al., 2021], 360FusionNeRF [Kulkarni et al., 2023], and PERF [Wang et al., 2024a]. These NeRF-based methods inherently lack the ability to infer occluded content and have insufficient geometric constraints for panoramic structures. As a result, they suffer from varying degrees of blurriness and reduced quality, as shown in Figure 7 and Table 3. Among them, PERF exhibits relatively satisfactory results, but it lacks consideration of panoramic geometric information, and there is a certain degree of quality degradation. On the contrary, we design PNVI and MVP to fully consider the constraints of the panoramic structure, while employing 3DGS rather than NeRF architecture, resulting in higher rendering quality in both quantitative and qualitative performance.

The extension experiments further demonstrate that our method can be extended to existing panoramas and perform high-quality novel view synthesis.

4.5 Ablation Studies

To validate the necessity of our inpainting mode and the effectiveness of the progressive inpainting strategy in PNVI, we design two corresponding ablation studies for different inpainting modes:

Directly inpainting panorama. We first retrain the AOTGAN on our synthesized panoramic dataset, and then directly perform inpainting on panoramas. We find the performance is ideal for small-distance movements, as shown in Figure 8(a).

However, as the movement distance increases, noticeable distortion and edge-blurring artifacts appear, as shown in Figure 8(b). This is due to the accumulated errors in depth estimation and the projection errors in the inpainting process. Additionally, due to discrepancies between the truth depth values in the dataset and our estimations, the distribution of holes is not entirely consistent between the training and inference stages.

Inpainting for a large distance. To validate the effectiveness of our progressive inpainting strategy, we perform inpainting on novel views with large camera poses, rather than incrementally moving. According to Figure 8(c), it is evident that directly inpainting large poses results in serious artifacts, which affects subsequent processing. When there is a large hole-to-image ratio, it becomes challenging to ensure the generation quality, thus affecting the consistency of the overall scene. By progressively inpainting the cubemap images, our PNVI strategy can address distortion and edgeblurring issues, as shown in Figure 8(a).

Table 4 reports the quantitative comparisons of different inpainting modes, where our FastScene achieves the best performance in scene generation. In summary, the ablation studies further demonstrate the effectiveness of our method.

5 Conclusion

We propose a fast Text-to-3D indoor scene generation framework FastScene, exhibiting satisfactory scene quality and consistency. For users, FastScene only requires a text prompt without designing motion parameters, and provide a complete high-quality 3D scene in only 15 minutes. The proposed PNVI with CVS can generate consistent novel panoramic views, while MVP projects them into perspective views, facilitating 3DGS reconstruction. Extensive experiments demonstrate the effectiveness of our method. FastScene provides a user-friendly scene generation paradigm, and we believe it has wide-ranging potential applications. In future work, we will focus on 3D scene editing and multimodal learning.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (No.62071500), and Shenzhen Science and Technology Program (Grant No. JCYJ20230807111107015).