A Comprehensive Survey of Interaction Techniques in 3D Scene Generation

1 Introduction

Recent advancements in generative models have transformed the 3D scene generation from a fully manual design task into a data-driven synthesis process. Instead of explicitly constructing every scene component, users only need to provide partial information and rely on generative models to infer the remaining structure [Lin et al., 2023; Wen et al., 2025a]. This transition fundamentally changes the role of the user, shifting scene generation from a static modeling task to an interactive process that requires continuous user input and feedback.

Interactive 3D scene generation refers to the mechanisms in which users inject their intentions, constraints, or corrective feedback into the scene synthesis process. Unlike the traditional modeling workflows, where users explicitly specify geometry and parameters, interaction in generative systems often operates at different abstraction levels. User input involves encoding high-level semantics, partial structural information, or localized editing operations, while the model is responsible for translating these signals into coherent 3D representations.

Existing studies defines interaction according to the locus of user intervention within the workflow, and formulating interaction as the condition setting during the scene generation[Wen et al., 2025a]. While a dominant paradigm approaches interaction as generative conditioning, distinct methodologies prioritize interactive editing to refine scene geometry and semantics after synthesis. Furthermore, emerging research [Fan et al., 2025] extends the interaction towards action-level modeling in embodied interaction, emphasizing the generation of physically consistent human-scene and human-object interactions in evolving environments.

Despite the rapid proliferation of research across these interaction paradigms, the literature remains fragmented, lacking a unified framework to organize these diverse techniques. As summarized in Table 1, existing surveys predominantly focus on broad generative architectures [Wen et al., 2025a], such as the transition from NeRF to 3D Gaussian Splatting, or are limited to specific subdomains such as 3D representation [Wang, 2024] and human motion [Xue et al., 2025]. Crucially, these reviews often overlook the interaction mechanisms themselves, failing to systematically analyze how user intent is encoded, processed, and integrated into the 3D synthesis pipeline. Consequently, there is a lack of comprehensive guidance on selecting appropriate interaction techniques for different stages of the 3D creation pipeline.

This survey aims to present a comprehensive and unified view of interaction techniques in 3D scene generation. We propose a structured taxonomy that organizes existing methods into the three aforementioned paradigms: (1) Interactive Scene Generation, which unifies semantic-driven generation grounded in multi-modal directives with structure-guided generation constrained by explicit spatial layouts; (2) Interactive Scene Editing, which focuses on post-synthesis refinement through high-level semantic modification or precise geometryaware manipulation; and (3) Embodied Interaction, which extends the scope to dynamic action model, emphasizing physically plausible human-scene and human-object interactions within the synthesized environment. Beyond this taxonomy, we critically analyze the technical foundations and evolutionary trajectories within each category. We also summarize essential datasets and evaluation benchmarks tailored for interactive 3D scene tasks.

Scope of This Survey. In this survey, we provide a comprehensive overview of interaction techniques for 3D scene generation, covering areas of generation, editing, and human-scene interaction. First, we introduce core concepts of common 3D representation methods, generative models, and problem formulation in interaction 3D scene (Section 2). Then, we classify recent research into three main paradigms: Interactive 3D Scene Generation (Section 3.1), Interactive 3D Scene Editing (Section 3.2), and Embodied Interaction (Section 3.3). For each paradigm, we review representative methods and analyze their interaction mechanisms, control characteristics, and technical distinctions. We then summarize commonly used datasets and evaluation metrics for interactive 3D scene generation (Section 4). Finally, we discuss open challenges and future research directions in Section 5.

2 Background

2.1 3D Scene Representations

3D representation fundamentally determines the fidelity of generation and granularity of user interaction. We categorize existing representations into explicit, implicit, and hybrid paradigms, analyzing their modelings and applications in terms of editability and generative consistency.

Explicit Representations. Explicit representations define geometry directly in Euclidean space. Polygonal Meshes [Wang, 2024] allow for precise manipulation of topology and texture, making them the standard for graphics pipelines and downstream editing. However, generating meshes with coherent topology remains challenging for neural networks due to their discrete nature. Point Clouds [Wang, 2024] offer a flexible, unstructured format that is easy to generate via regression but lacks surface connectivity, complicating physical simulation and high-quality rendering. Voxel Grids [Wang, 2024; Jiang et al., 2024] discretize space into regular cubic volumes , simplifying generation into a classification task. Yet, their cubic memory complexity limits resolution, making them unsuitable for large-scale detailed scene interaction.

Implicit Representations. Implicit methods represent scenes as continuous functions parameterized by neural networks. Neural Radiance Fields (NeRF) [Mildenhall et al., 2021] map spatial coordinates and viewing directions (x, d) to density and color (σ, c), enabling photorealistic view synthesis. Signed Distance Fields (SDF) [Park et al., 2019] map coordinates to the distance to the nearest surface s(x), facilitating precise geometric operations and surface extraction. While implicit representations excel in visual fidelity, they entangle scene attributes within network weights, making localized interaction computationally expensive and difficult without explicit decoupling.

Hybrid Representations. To bridge the gap between explicit controllability and implicit rendering quality, 3D Gaussian Splatting (3DGS) [Kerbl et al., 2023] has emerged as a dominant paradigm. 3DGS represents a scene as a set of anisotropic particles with center µ, covariance Σ, color c, and opacity α, allowing for real-time selection and transformation, while the differentiable rasterization pipeline ensures NeRFlevel rendering quality. This duality makes 3DGS particularly advantageous for real-time interactive generation and editing.

2.2 Deep Generative Models

Generative models serve as the computational engine for synthesizing 3D structures from user intents. Early interactive systems predominantly leveraged Generative Adversarial Networks (GANs) [Goodfellow et al., 2014] due to the efficiency. By optimizing a minimax process between a generator and a discriminator, GANs synthesize 3D content in a single forward pass, enabling millisecond-level feedback essential for real-time exploration. However, GANs often struggle with training instability and mode collapse, limiting their ability to capture the complex and multi-modal distributions required for diverse scene generation. Generative models have shifted towards Diffusion Models (DMs) [Ho et al., 2020], which formulate generation as an iterative denoising process. While this iterative nature incurs higher inference computational costs, it unlocks superior generation quality and training stability. Diffusion models also promote progressive interaction models [Rombach et al., 2022]. User signals, whether text prompts, coarse layouts, or sparse strokes, can be injected at different noise levels to improve the global structure or local details. This granularity of control makes diffusion the standard for modern interactive systems, driving parallel research into distillation techniques to alleviate reasoning bottlenecks and approach real-time performance.

2.3 ProblemFormulation

We formally define the problem of Interactive 3D Scene Generation, where S denotes the space of 3D scene representations. The goal is to learn a generative model Φ that synthesizes a scene S ∈ S conditioned on a multi-modal user control signal C. At step t, the user observes the current state St-1 and provides an interaction signal ut. The model updates the scene:

where θ represents the learned parameters. The objective is to maximize the plausibility of the generated scene while strictly adhering to the user's constraints:

where Lalign measures the alignment between the scene and user intent (e.g., CLIP score for text, IoU for layout), and λ balances realism and controllability. This formulation unifies the three paradigms discussed in this survey: generation (S0 from null), editing (updating S via u), and embodied interaction (optimizing S for physical agent dynamics).

3 Categories of Interactive Techniques

We categorize interactive techniques based on the granularity of user control and the stage of intervention within the generative pipeline. As illustrated in Figure 1, we structure existing methodologies into three primary paradigms: (1) Interactive Scene Generation, dealing with the synthesis of 3D scenes driven by high-level semantics and explicit structural constraints; (2) Interactive Scene Editing, focusing on the refinement of geometry and appearance; and (3) Embodied Interaction, which extends generation to dynamic and physicsaware interactions between agents and environments. The following content critically analyzes the technical foundations and representative frameworks within each paradigm.

3.1 Interactive 3D Scene Generation

Within interactive generation, existing interactive generation methods can be broadly divided according to the precision with which user intent constrains the scene structure, as illustrated in Figure 2. Semantic-driven interaction provides a high-level and low-effort control interface, where users express intent through abstract signals such as text, images, or audio. In contrast, structure-guided interaction emphasizes precise control by allowing users to explicitly specify spatial constraints, such as object layouts, scene graphs, or geometric proxies. This contrast reflects a fundamental trade-off in interactive generation between flexible but ambiguous semantic control and precise but effort-intensive structural control, which underlies the design choices of most interactive 3D scene generation systems.

Semantic-driven Generation

Semantic-driven generation allows users to synthesize 3D environments using high-level abstract prompts, such as text descriptions, reference images, or motion trajectories. This paradigm relies on distilling priors from large-scale 2D generative models into consistent 3D representations.

Text-based Generation. Several works focused on effectively mapping to target 3D representations from user-provided text prompts, which usually utilize text-to-image models as the prior information for generating 3D scenes. Early text-to-3D approaches [Poole et al., 2023; Lin et al., 2023] leverage pretrained text-to-image diffusion models as semantic priors, optimizing 3D scene representations to align with text-conditioned visual distributions. This strategy enables data-efficient 3D generation without large-scale 3D-text datasets and has become a dominant paradigm for semantic-driven interaction. However, text-driven interaction also exposes fundamental limitations. Natural language descriptions rarely encode precise spatial layouts or physical constraints, leading to challenges in multi-object arrangement and geometric consistency. While recent methods [Zhu et al., 2025] introduce multi-stage optimization or hierarchical prompting to mitigate these issues, the core ambiguity of text-based control remains an inherent property of this interaction paradigm.

Beyond static 3D scenes, users increasingly expect to model temporal and spatial evolution in dynamic environments, which further amplifies the challenges of semantic-driven interaction. MAV3D [Singer et al., 2023] appears as a pioneering effort that enables direct generation of 4D scenes from text descriptions. It optimizes a dynamic NeRF representation augmented with motion fields and leverages a pretrained textto-video diffusion model to guide scene attributes, producing dynamic scenes observable from arbitrary viewpoints. However, coupling static appearance and motion within a single optimization process often leads to artifacts such as temporal inconsistency and view-dependent distortions. Dream-in4D [Zheng et al., 2024] addresses this by further introducing a two-stage framework that explicitly decouples static scene generation from motion learning. Similarly, TextMesh4D [Dai et al., 2025] proposes a Jacobian Deformation Field combined with local-global semantic regularization, enabling direct generation of dynamic mesh sequences while preserving spatiotemporal consistency. These text-to-4Dmethods expose a fundamental limitation of semantic-driven interaction: maintaining geometric and temporal consistency across space and time under weak semantic constraints.

Image-basedGeneration. Tomitigate the semantic ambiguity inherent in text prompts, Image-based Generation leverages visual exemplars to provide explicit spatial cues. This interaction modality significantly constrains the solution space, making it effective for tasks requiring high fidelity. The evolution of this paradigm has been characterized by a shift from slow optimization-based synthesis to efficient and interactive frameworks. Early approaches primarily utilized GANs to synthesize panoramas from image priors [Hu et al., 2022], however they often lacked flexibility in local control. To enhance interactivity, Follow-Your-Click [Ma et al., 2024] introduced regional selection mechanisms, allowing users to animate specific scene areas via sparse clicks. However, a major bottleneck for interactive applications has been generation latency. Addressing this, Bolt3D [Szymanowicz et al., 2025] proposes a feed-forward latent diffusion model that bypasses lengthy per-scene optimization, capable of outputting 3D representations from unposed images at interactive frame rates. Similarly, to facilitate rapid online creation, WonderWorld [Yu et al., 2025] introduces Fast Layered Gaussian Surfaces, that employs geometric initialization and directed deep diffusion, enabling users to generate consistent scenes from a single image and layout prompts in seconds. These advancements are pivotal, as they transition image-to-3D generation from an offline processing task to a real-time creative workflow.

Multi-modal Conditioned Generation. Standard text or image prompts often struggle to articulate dynamic behaviors, complex temporal evolution, or non-visual attributes. Recent research extends semantic interaction into the spatiotemporal domain by incorporating additional modalities, such as audio cues [Zhang et al., 2024b] and motion trajectories. These signals enrich control by explicitly encoding motion intent and temporal patterns, bridging the gap between static scene generation and dynamic storytelling.

Trajectory-based interaction has emerged as a critical direction for defining spatiotemporal dynamics. Motion Prompting [Geng et al., 2025] enables fine-grained control by conditioning video generation on user-drawn sparse or dense trajectories, permitting the specification of both object-specific paths and global camera motion. Moving towards real-time feedback, MotionStream [Shin et al., 2025] proposes an interactive video world model that synthesizes content in a streaming manner, allowing users to manipulate trajectories or transfer motion styles while viewing results instantaneously. In the 3D domain, Drag4D [Kang et al., 2025] establishes a unified framework that integrates object motion control with background synthesis, enabling users to define 3D trajectories for objects generated from single images. By allowing users to choreograph the movement within generated scenes, these methods unlock significant potential for interactive 3D animation and virtual cinematography.

Structure-guided Generation

Structure-guided interactive generation introduces explicit spatial constraints into the generation process, enabling users to precisely control scene structure, object relationships, and layout configurations. Unlike semantic-driven interaction, which relies on the model to infer spatial organization from abstract cues, structure-guided methods directly constrain where and how scene elements are arranged. This paradigm shifts part of the generation responsibility from the model to the user, resulting in improved determinism and controllability at the cost of increased interaction effort.

Scene Graph-based Generation. Scene graphs provide a symbolic representation of scene structure by encoding objects as nodes and spatial or semantic relationships as edges. Early scene graph-based methods [Ma et al., 2018] primarily relied on data-driven retrieval pipelines, where scene graphs served as blueprints for selecting and placing object assets from databases. While effective for enforcing relational consistency, these approaches often suffer from limited geometric diversity and coarse spatial control. Recent advances aim to bridge symbolic structure and continuous geometry. GraLa3D [Huang et al., 2024] jointly leverages scene graphs and layouts by using large language models to infer non-overlapping spatial arrangements from text, combining relational constraints with geometric guidance. DreamScene [Li et al., 2025] further integrates LLM-based agents to infer hybrid graphs encoding both object semantics and spatial constraints, enabling end-toend scene synthesis. Similarly, Liu et al. [Liu et al., 2025b] transform sparse scene graphs into dense embedding maps that condition diffusion-based generators, improving spatial fidelity while preserving user-specified relationships. These methods demonstrate that scene graphs are most effective when coupled with continuous spatial representations, highlighting the trend toward symbolic-geometric hybrid control.

Semantic Graph-based Generation. Unlike the scene graph, semantic-based layout serves as an intermediate representation for encoding the structure and semantic organization of 3D scenes. It provides users with advanced guidance for 3D scene generation, ensuring the controllability and consistency of scene element placements. One aspect of the research focuses on 2D semantic layouts. BlockFusion [Wu et al., 2024] generates 3D scenes in the form of cubic blocks. It introduced a two-dimensional layout adjustment mechanism, which enables users to precisely determine the placement and arrangement of elements by manipulating the bounding boxes of two-dimensional objects, thereby providing users with more control over the generation process. 3D City [Feng et al., 2025] integrated 2D layout, 3D terrain, and procedural modeling techniques to create a fully editable 3D environment, allowing users to efficiently customize and modify scene components. They utilize the structured data extraction process to separate model and layout features from textual descriptions to create 2D layouts to guide 3D terrain generation.

One the other aspect of research focuses on more complex 3D semantic layouts, which have better interactivity and controllability. SceneCraft [Yang et al., 2024] proposed a detailed method for indoor scenes generating, the core of which lies in converting 3D semantic layout into multi-view 2D proxy maps, and designing a semantic and depth-conditioned diffusion model to generate multi-view images, which are used to learn NeRF as the final scene representation. Layout2Scene [Chen et al., 2025] guides the scene generation through 3D semantic layout based on geometric and appearance diffusion priors. It combines semantic layout with text description, where the text description is used to specify the scene type, and the 3D semantic layout provides fine-grained control to eliminate the ambiguity in the global text description. SceneFactor [Bokhovkin et al., 2025] maps the textual descriptions of the scene area to a 3D semantic layout map, and guide the high-fidelity geometric synthesis of scene geometry corresponding to the proxy semantics, enabling users to achieve more controllable scene generation and editing.

Structure-guided interaction enables precise and deterministic control by explicitly encoding spatial structure, making it particularly suitable for complex multi-object scenes. However, it requires users to invest more effort in specifying layouts or structural proxies, highlighting a fundamental trade-off between interaction burden and controllability.

3.2 Interactive 3D Scene Editing

Interactive 3D scene editing focuses on post-generation refinement, allowing users to modify geometry, appearance, semantics, or spatial layout of an existing scene. Unlike interactive generation, where user input conditions synthesis from scratch, editing-based interaction decouples interaction from initial generation and emphasizes local controllability, consistency preservation, and iterative correction. This paradigm is essential for correcting model errors, enforcing fine-grained constraints, and supporting creative workflows that require incremental adjustments.

Prompt-based Editing. Prompt-based editing enables users to specify editing intent through high-level instructions, such as text or reference images. These methods typically rely on pretrained vision-language models to interpret user commands and propagate semantic changes to 3D representations. Previous approaches [Haque et al., 2023] extend text-to-image diffusion models to 3D domains, enabling instruction-based modification of scene attributes. CustomNeRF [He et al., 2024] introduces a unified text and image-driven editing framework that uses 2D masks to localize edits and adopts a local-global optimization strategy to maintain multi-view consistency. Dialogue-based interaction further enhances controllability by allowing users to iteratively refine instructions. ChatEdit-3D [Fang et al., 2024] slices 3D scenes into 2D texture atlases and employs LLMs to parse user instructions, enabling conversational editing through 2D operations. GaussEdit [Shu et al., 2025] extends this paradigm by allowing users to define regions of interest via bounding boxes and applying categoryaware diffusion regularization to ensure physically plausible edits. Prompt-based editing offers intuitive semantic control but remains sensitive to region localization accuracy and crossview consistency, particularly for complex geometric changes.

Geometry-aware Editing. Geometry-aware editing provides direct and precise control by allowing users to manipulate scene geometry through clicks, strokes, dragging, or selection. These interactions explicitly operate on spatial structures, making them more deterministic than prompt-based approaches. 4D-Editor [Jiang et al., 2023] enables object-level editing in dynamic NeRFs using user strokes on single frames, maintaining spatiotemporal consistency via hybrid semantic features. iControl3D [Li et al., 2024b] allows users to specify semantic segmentations, scribbles, or depth cues, which are fused with diffusion-generated images to construct complete scenes. SceneExpander [Zhang et al., 2024c] supports groupaware editing, where dragging one object triggers automatic re-layout of surrounding objects to preserve spatial coherence. DragScene [Gu et al., 2024] and InterGSEdit [Wen et al., 2025b] further exploit 3D Gaussian Splatting to enable realtime, multi-view consistent geometric manipulation. While geometry-aware editing excels in precision and determinism, it often requires more complex interfaces and careful consistency handling, especially for large-scale or dynamic scenes.

3.3 Embodied Interaction

Embodied interaction extends interactive 3D scene generation beyond manipulation to action execution within generated environments. In this paradigm, interaction is realized through dynamic behaviors of humans and objects, requiring joint reasoning over geometry, motion, contact, and physical constraints. Unlike generation or editing, embodied interaction inherently involves dynamics and physical feasibility.

Human-Scene Interaction (HSI). Early studies explored possibilities of human interaction with static scenes based on simple scene conditions and initial human states [Wang et al., 2021]. Later works incorporated action labels and text descriptions as semantic conditions to enhance user control. Advancements in LLMs have significantly enhanced applications in this area. UniHSI [Xiao et al., 2023] proposed a unified HSI framework, supporting users to control various scene physical interactions uniformly via language commands, characterized by using LLMs to generate interaction plans, making training interaction-annotation-free. Current progress lies in generating realistic and physically plausible human interaction in 3D scenes. InterScene [Pan et al., 2024] proposes dividing character-scene interaction into "interaction" and "navigation" phases, using interaction controllers (e.g., sit/stand actions) and navigation controllers (obstacle avoidance path following) respectively, to enable physics-simulated characters to execute long-term interaction actions in multiobject, complex 3D scenes. PhySIC [Yalandur Muralidhar et al., 2025] reconstructs physically plausible 3D human-scene interactions from a single RGB image. By jointly optimizing human, scene, and contact information, it resolves depth ambiguities and physical inconsistencies in invisible parts.

Human-Object Interaction (HOI). HOI is considered a specialized subset of HSI, emphasizing the accurate execution of approaching, grasping, and manipulating objects. Some studies focus on the interaction generation of whole-body motion. InterDiff [Xu et al., 2023] uses a diffusion network to generate human-object interaction sequences and employs a physics-aware interaction correction module using relative motion patterns around contact points to ensure physical perceptibility. ZeroHSI [Li et al., 2024a] reconstructs realistic human motion by extracting motion priors from video generation models and using differentiable neural rendering, achieving human-scene and human-object interaction in static and dynamic 3D environments without paired motion capture data.

Compared to kinematics-based methods, physics-based methods are more advantageous for HOI because physics simulators can explicitly constrain humanoid and object dynamics, creating more realistic human-machine interactions in 3D scenes. PhysHOI [Wang et al., 2023] is a representative work introducing a physics-based whole-body HOI imitation framework, making HOI imitation feasible in diverse scenes. InteractMove [Cai et al., 2025] first incorporated text-conditioned 3D interaction actions (including grasping, manipulation) into a scene generation framework, emphasizing physical feasibility, enabling users to generate the physical process of humans interacting with movable objects in 3D scenes (including grasping, moving, and collision avoidance) via text control. Uni-Inter [Liu et al., 2025a] attempts to unify multiple interaction types (including object-object, humanscene) beyond single interaction types, proposing a unified framework to model volumetric representations of humans and various interaction entities, and generating physically plausible actions via joint-level probability prediction to support composite interaction scenes.

4 Datasets and Evaluation

In this section, we review commonly used datasets and evaluation metrics from an interaction-centric perspective, highlighting how current dataset support different interaction paradigms and significant gaps in this field.

4.1 Datasets

Existing datasets can be broadly categorized into generation and editing, human-scene interaction, and human-object interaction, each reflecting different assumptions about interaction complexity and physical grounding. In Table 2, generation and editing datasets, such as 3D-FRONT [Fu et al., 2021] and Objaverse-XL [Deitke et al., 2023], emphasize scene diversity and structural annotations, supporting controllable synthesis and post-generation modification. However, they often lack interaction trajectories or physical annotations. Human-scene interaction datasets, including PROX [Hassan et al., 2019] and HUMANISE [Wang et al., 2022], focus on capturing realistic human behaviors within scenes, but are limited in scale due to data collection complexity. Human-object interaction datasets, such as GRAB [Taheri et al., 2020] and BEHAVE [Bhatnagar et al., 2022], provide fine-grained contact and manipulation annotations, enabling physics-aware modeling, but typically cover fewer object categories. However, a dataset that simultaneously supports interactive generation, editing, and embodied interaction is still lacking, motivating future efforts toward unified, interaction-centric benchmarks.

4.2 Evaluation

Controllability is used to evaluate the ability to respond to user input. CLIP Score[Hessel et al., 2021] is utilized to measure the alignment between the generated images and the conditioned text, which can reflect whether the generated scene follows the user-specified prompt. Motion quality focuses on evaluating the authenticity, accuracy, and diversity of the interactive movements in the generated scenarios. The indicators usually include Diversity, Multi-modality, R Precision, and MPJPE. Physical scalability is used to evaluate how the generated interactions comply with the physical constraints of the environment or objects, which is typically measured through Non-Collision, Contact, and Penetration. User research mainly measures the perceived naturalness and overall quality of the generated interaction through the subjective assessment by evaluators. Participants are usually required to rank or rate the generated interaction sequences based on multiple aspects, typically including authenticity, naturalness, and physical acceptability. Platforms such as Prolific and Amazon Mechanical Turk facilitate the recruitment of diverse participants and enable the effective expansion of user research.

5 Challenges and Future Directions

Interactive 3D scene generation has progressed rapidly; however, current systems still fall short of practical requirements for reliable control, consistent feedback, and physically plausible interaction. The core difficulties are not only model capacity, but also the lack of interaction-centric data, evaluation protocols, and system-level design that jointly support generation, editing, and embodied behaviors.

5.1 Challenges

Interaction-centric data and evaluation. A primary bottleneck is the scarcity of datasets that explicitly capture interactive signals and their corresponding target outcomes. Existing resources are often designed for static generation or isolated interaction tasks, and rarely provide unified annotations across semantic intent, structural constraints, editing operations, and embodied trajectories. On the evaluation side, commonly used metrics are still dominated by visual fidelity or geometric reconstruction quality, which are insufficient to assess interactive systems. What remains under-specified is an interaction-aware evaluation protocol that measures (i) controllability under user constraints, (ii) consistency across viewpoints and time, and (iii) response stability under iterative user feedback.

Controllability under heterogeneous user inputs. Modern interactive systems face heterogeneous inputs, such as text, images, layouts, sketches, and direct manipulations. A persistent limitation is the inability to robustly resolve conflicts across modalities and across interaction stages. For example, semantic prompts may imply goals that contradict a structural layout, or geometry-aware edits may break semantic alignment. Existing methods often rely on manual weighting, leading to unstable behavior when user instructions are ambiguous. Achieving predictable controllability therefore requires principled mechanisms for constraint prioritization, uncertainty handling, and iterative correction.

5.2 Future Directions

Physics-grounded interaction and physically consistent feedback. Akey direction is to incorporate stronger physical grounding into interactive generation, not only as correction but as part of the interaction loop [Meng et al., 2025]. This includes learning or integrating physical priors to constrain generation and editing outcomes, and supporting embodied interaction with motion and force consistency. A practical challenge is that high-fidelity simulation can be costly, so future systems may require hybrid designs that combine fast learned dynamics with selective simulation for critical events. Physics-grounded feedback can serve as an additional interaction channel, enabling systems to explain failures and maintain consistency during iterative edits.

Unified multi-modal interaction with conflict-aware control. Another important direction is a unified interaction framework that treats multi-modal inputs as complementary constraints rather than independent conditions. This requires methods that can (i) align semantic and structural signals into a shared control space, (ii) detect and resolve conflicts across modalities, and (iii) support incremental updates without regenerating the entire scene from scratch. A promising system design is to maintain explicit intermediate states, such as structured scene graphs, editable layouts, or factorized representations. Therefore, user actions can be applied locally with predictable global effects. Ultimately, multi-modal interaction should enable more natural and efficient workflows, while providing reliable controllability in complex scenes.

6 Conclusion

This survey provides an interaction-centered perspective on 3D scene generation, systematically reviewing how users intervene in the generation pipeline through different interaction mechanisms. Rather than organizing prior work by representations or model architectures, we frame existing methods according to where user input enters the workflow and what aspects of the scene it directly controls. We categorize interactive techniques into three major paradigms: Interactive Generation, Interactive Editing, and Embodied Interaction, which together cover intent specification during synthesis, post-generation refinement, and action execution within generated environments. This survey also summarizes commonly used datasets and evaluation practices, providing a clear fragmentation across generation, editing, and embodied interaction scenarios. We hope this survey serves as a foundation for understanding interaction mechanisms in 3D scene generation and provides guidance for developing more controllable and physically grounded interactive 3D systems.

Contribution Statement

Yuqi Li and Siwei Meng contributed equally to this work. All authors contributed to the survey design, literature review, manuscript writing, and revision.