Inter3D: A Benchmark and Strong Baseline for Human-Interactive 3D Object Reconstruction

1 Introduction

3D object reconstruction has been a long-standing focus of research in computer vision and graphics [Berger et al., 2017]. Recently, neural radiance fields (NeRFs) [Mildenhall et al., 2021] pioneer volumetric rendering to enable high-fidelity modelling of 3D objects with fine-grained geometric structures, facilitating its widespread applications across various domains [Barron et al., 2022; Fridovich-Keil et al., 2023a; Pumarola et al., 2021; Poole et al., 2022; Liu et al., 2024].

Novel States

Figure 1: Illustration of reconstructing a human-interactive object, Furniture, in our Inter3D. The object consists of n = 3 movable parts, highlighted in red, green, and olive-green. With 2n = 8 discrete states, the task requires significant computational and memory resources, making it impractical for existing methods to train separate models for each state. Furthermore, ensuring consistency in external appearances and internal structures across states poses a significant challenge when states are trained independently. In contrast, our approach efficiently synthesizes novel combination states by observing only the canonical and individual part states.

3D Gaussian splatting [Kerbl et al., 2023] extends the principles of NeRFs by representing an object with a set of 3D Gaussian primitives and synthesizes novel views via rasterization, offering a more expressive and rendering-efficient 3D representations [Luiten et al., 2024].

Despite their advancements, existing methods predominantly focus on modeling static objects or dynamic ones with simple continuous motions, where the states of objects exhibit no abrupt changes. Also, these methods primarily focus on synthesizing novel views of the observed states during training. However, many objects commonly encountered in daily life are interactive and have numerous discrete states. A human-interactive object with n movable parts results in a total of 2n states. Efficiently modeling such objects holds significant potential for applications in virtual reality [Wohlgenannt et al., 2020], gaming [Cameir˜ao et al., 2009], and embedded AI [Machinery, 1950].

As illustrated in Figure 1, the Furniture object consists of three movable parts, highlighted in red, green, and olivegreen. It features one canonical state and three individual states where only one part is open, along with four combination states where two or more parts are open. Existing methods fail to efficiently model all possible states of the Furniture object. A straightforward solution would be to train a separate 3D model for each object state. However, this solution incurs prohibitive computational and memory overhead. Moreover, different states of the same human-interactive object inherently share consistent external appearances and internal structures, which this solution fails to capture.

To address these limitations, we propose Inter3D, a novel benchmark and method for human-interactive 3D object reconstruction. Inter3D introduces a self-collected dataset comprising commonly encountered yet challenging interactive objects in daily life, such as Car, Display Case, Drawer, and Furniture. It also introduces a new evaluation scenario, i.e., novel state synthesis, in which only the canonical and individual part states of an object are observed during training, while all combination states remain unseen and are reserved for evaluation, as illustrated in Figure 1. This setup is particularly challenging as no views of combination states are used during training, yet the synthesized novel states must remain consistent with the observed states.

Our method effectively tackles this challenging scenario by leveraging Space Discrepancy Tensors in combination with multi-resolution hash encoding (MHE) [M¨uller et al., 2022] to efficiently model all states of an interactive object. Specifically, MHE is initially employed to represent the canonical state of the object, where all movable parts remain untouched. The Space Discrepancy Tensors capture the inherent variations between each manipulated part state and the canonical state, providing a compact and expressive representation for the decomposition of individual movable parts. To mitigate rendering quality degradation caused by trajectory perturbations across training states, we introduce a Mutual State Regularization mechanism to ensure consistency of movable parts across states. Additionally, we explore two new occupancy grid sampling schemes to optimize training efficiency. After training on the canonical state and individual part states, our method can synthesize arbitrary combination states by selecting the maximum density difference.

Our key contributions in this work are summarized as follows:

• We introduce a novel benchmark for human-interactive 3D object reconstruction, supported by a self-collected dataset of commonly encountered interactive objects and a novel state synthesis testing scenario. This benchmark provides a foundation for advancing research in this field.
• We propose a novel approach that integrates Space Discrepancy Tensors with multi-resolution hash encoding, enabling efficient modeling of all possible states of interactive objects. This approach provides a strong baseline for future studies.
• We develop a Mutual State Regularization mechanism to mitigate rendering artifacts caused by trajectory inconsistencies and design two occupancy grid sampling strategies to balance memory usage and training speed.

2 RelatedWorks

Static 3D Reconstruction. Recent advances in implicit 3D reconstruction, particularly with neural radiance fields (NeRFs) [Mildenhall et al., 2021], have enabled high-quality novel view generation by predicting volume density and radiance from 3D coordinates and viewpoint directions. Variants like mip-NeRF [Barron et al., 2022] optimize performance in more complex scenarios. 3D Gaussian splatting [Kerbl et al., 2023] uses a set of Gaussian primitives with learnable parameters, i.e., positions, covariance matrices, opacity, and spherical harmonics coefficients, and differentiable tile rasterizer to improve synthesis quality and training speed. However, these methods primarily focus on modeling static objects and can only be applied to a single state of a human-interactive object. To model all possible states, a series of independent 3D models would be required, significantly increasing computational costs and failing to ensure consistency across states.

Dynamic 4D Reconstruction. Dynamic 4D reconstruction methods, such as K-Plane [Fridovich-Keil et al., 2023a], DNeRF [Pumarola et al., 2021], and 4D Gaussians [Wu et al., 2024], extend vanilla NeRFs or Gaussian splatting by incorporating motion factors to model geometric and texture changes of time-varying objects. These methods excel at modeling dynamic objects with continuous states and synthesizing novel views of the training states. However, our approach focuses on modeling human-interactive objects with numerous discrete states and emphasizes novel state synthesis, a task that is infeasible for these methods.

3 Human-Interactive Benchmark

3.1 Data Collection

To build the benchmark, we collect a diverse dataset comprising four objects with multiple movable parts commonly encountered in daily life: Car, Display Case, Drawer, and Furniture. These objects are arranged in increasing order of complexity. The Car represents smooth-surfaced objects with two movable parts, i.e., front and rear doors, providing a straightforward yet realistic interaction scenario. The Display Case, with its two movable glass doors and intricate internal contents, introduces additional complexity, especially for assessing the effectiveness of our approach under lowdensity conditions. The Drawer, featuring a typical threedrawer design commonly seen in daily environments, serves as a representative example of structured objects with discrete states. Finally, the Furniture, consisting of three types of cabinets, each with one movable part, features a challenging sandy background that introduces additional noise and interference, making it the most complex object in our dataset.

We capture object videos using handheld, forward-facing mobile shots and utilize SAM2 [Ravi et al., 2024b] to segment movable parts that remain in place. For each object, 60 images are extracted through uniform sampling at regular intervals to represent the canonical state and each individual state. As illustrated in Figure 3, we collect forwardfacing data for a toy car with two movable parts, i.e., its front and rear doors. Images are captured for the canonical state S0 and for individual part states S1 and S2, where only one door is opened at a time. Camera poses are extracted using Canonical Modelling

Movable Part Decomposition

Arbitrary Combination Rendering

Figure 3: Data collection example of the object Car in our humaninteractive benchmark Inter3D. A sequence of forward-facing images are captured for the canonical state S0 with doors closed, the state S1 with the front door open, and the state S2 with the rear door open.

COLMAP [Fisher et al., 2021], ensuring precise alignment for subsequent learning.

3.2 Novel State Evaluation

In traditional 3D or 4D reconstruction scenarios, objects are either static or exhibit continuous states. Furthermore, models typically observe all possible states during training, with only novel views used for evaluation. In contrast, our benchmark introduces interactive objects with numerous discrete states, where the evaluating states are novel and remain entirely unseen during training.

For an object with n movable parts, there are 2n possible states. Each part is manipulated individually, resulting in an individual state Si, i ∈ {1, . . . , n}, which is observed during training. When multiple parts are manipulated simultaneously, the resulting state becomes a spatial combination of individual states. For instance, Si + Sj, i, j ∈ {1, . . . , n}, represents the simultaneous manipulation of the ith and jth parts. These combinatorial states are entirely unseen during training and reserved for evaluation rendering, significantly increasing the complexity of the task. This challenging setup requires models to achieve a higher level of generalization and adaptability, thereby pushing the boundaries of existing 3D reconstruction techniques.

4 Human-Interactive 3DReconstruction

Inter3D establishes a new benchmark for synthesizing diverse states of human-interactive objects, offering a powerful baseline method. Unlike classical approaches, which require 2n separate models to represent an object with n movable parts, our method efficiently represents the object using a unified model. Figure 2 illustrates the framework of our approach. In this section, we first describe the details of InstantNGP [M¨uller et al., 2022] in Sec. 4.1. Then, we detail the initialization Canonical Modelling in Sec. 4.2, which defines the canonical state of a human-interactive object and provides a solid foundation for the subsequent steps. Sec. 4.3 introduces the Space Discrepancy Tensors, which efficiently models discrepancies between individual part states and canonical state.

Sec. 4.4 presents the pipeline for representing novel combination states, and elaborates how the Mutual State Regularization alleviates the impractical constraints on camera trajectories across training states. Finally, Sec. 4.5 discusses two occupancy grid sampling strategies designed to enhance training efficiency.

4.1 Preliminaries

InstantNGP [M¨uller et al., 2022] boosts up the rendering and training speed of NeRFs. It proposed a multi-resolution voxel grid, implicitly encoded using hash functions, enabling efficient optimization and rendering with compact models. A ray passing through three-dimensional space is defined as r(t) = o + td, where o is the ray's origin, d is its direction, and t represents the distance along the ray. InstantNGP employs a density occupancy grid with step-based indexing to acquire sampling points. Then, based on the 3D coordinates of a sampling point, feature vectors of multiple surrounding voxels are retrieved from the hash table. These voxel feature vectors are then linearly interpolated to obtain the feature vector h for the sampling point. An MLP is used to compute the color c and density σ of the sampling points. The pixel color C is determined by the following volume rendering equation applied to the N sampling points along the ray:

LMSE is employed to minimize the discrepancy between the predicted pixel color and the ground truth C gt :

4.2 Canonical Modelling

Canonical Modelling reconstructs the canonical state of the object, providing a solid foundation for the subsequent movable part decomposition, where a part of the object is manipulated into an individual state. As shown in Figure 2, we leverage InstantNGP to model the canonical state S0.

To enhance the compactness of spatial points and reduce the presence of fuzzy clouds in space, we employ the distortion loss function fromMipNeRF360 [Barron et al., 2022]:

where wi = (1 - e-σiδi )Ti, and s represents the distance from the sample point to the ray origin. Additionally, we incorporate a loss term aimed at minimizing the entropy of the cumulative weights, which helps to concentrate the distribution of spatial sample points:

where O is calculated as the sum of the weights wi of N sample points along the ray, given by: O = N∑i wi .

4.3 Movable Part Decomposition

In this section, we aim to identify the movable components from the interactive object, and learn their individual states. As illustrated in Figure 2, we train the model across all individual states, starting with the initialized canonical state S0. Instead of explicitly identifying the moving parts, we propose Space Discrepancy Tensors to implicitly encode the individual states, enhancing the scalability and flexibility of our method.

Specifically, the Space Discrepancy Tensors are three spatially correlated feature tensors in the Cartesian coordinate system, denoted as Tix , Tiy , and Tiz , each with a shape of 512×32, where 512 represents the spatial resolution and 32 denotes the feature dimension. These tensors are leveraged to capture the discrepancies introduced by the individual state Si. We normalize the 3D coordinates of a random sampling point p and project them onto these three feature tensors. Using nearest-neighbor sampling, we extract the spatially relevant features along each axis, and then compute the final feature vector ti of p by element-wise multiplication:

where πa(p) represents projecting p onto the axis a, and ψ(Tia , ·) denotes the nearest-neighbor sampling of a point feature from the tensor Tia . To ensure the non-interactive part consistency between the canonical state and individual states, similar to K-Planes [Fridovich-Keil et al., 2023b], we employ Lconsis to constrain Ti towards 1:

We also employ the Laplacian (second-order derivative) filter Lsmooth in K-Planes [Fridovich-Keil et al., 2023b] to encourage the smooth variations in the state Si.

4.4 Arbitrary Combination Synthesis

We propose a simple yet effective strategy of synthesizing a novel combination state on top of the training states. For example, the state S1 + S2 in Figure 4 can be generated by spatially merging S1 and S2. Specifically, our goal is to combine the varying components of S1 and S2 relative to S0, along with the non-interactive parts of S0, to form a new state S1 + S2. The sample point density σi associated with the individual state Si are employed to capture the differences w.r.t. the canonical state S0 and the state with the maximum differences dominates the volume rendering:

The corresponding density and color c∆ and σ∆ are then queried to render the new state S1 + S2 via Equation 1. It is straightforward to adapt this synthesis strategy to an interactive object with more movable parts.

Mutual State Regularization. In practice, it is infeasible to maintain absolutely identical camera trajectories across different training states, which degrades the synthesis quality of movable parts. To address this issue, we propose the Mutual State Regularization (MSR) to enhance the spatial consistency of movable parts between training states.

As shown in Figure 5.a, we match the inactive movable part, i.e., the back door, of S1 to that of S0. Specifically, we randomly sample a set of points along a ray passing through the back door of S0 (segmented by SAM2 [Ravi et al., 2024a]) and obtain the pixel value C0[Mask] via integration. We then project the points onto the S1 space and query the pixel value C1[Mask]. The LMSR loss is employed to align them, ensuring the back door consistency:

Reversely, in Figure 5.b, we align the pixel values of the back door of S0 with those of S1. The alignment process can be easily generalized to other individual state Si.

4.5 Training Efficiency

As aforementioned, in Canonical Modelling, we employ InstantNGP to reconstruct the canonical state of an interactive object. In the Movable Part Decomposition stage, we model various individual states, each sharing non-interactive parts but featuring different movable parts. To achieve this, we propose two occupancy grid sampling strategies that balance training memory and speed.

Shared Occupancy Grid. We reuse the occupancy grid from Canonical Modelling by only increasing its learned density values, which represent the manipulated movable parts in each individual state. This strategy saves the memory usage, but experimentally increases the number of sampling points in each ray, which slows down the training process.

Independent Occupancy Grid. In this strategy, we set independent occupancy grids for individual states, initialized with the canonical state, and update them separately. While this approach significantly increases memory usage, it greatly enhances training speed by reducing the number of sampling points.

With these strategies, we can choose the better option depending on computational resources and preferences.

4.6 Overall Objectives

The first two stages of our approach involves training. The learning objective in Canonical Modelling LCM is as follows:

where λ1 and λ2 are the weighting coefficients, both set to 1e-3 in our experiments. In Movable Part Decomposition, an individual state Si is randomly selected for training. We employ both LCM and the learning objective LMPD to optimize the model:

where λ3, λ4 and λ5 are set as 1e-4, 1e-3, and 1e-2, respectively.

5 Experiments

5.1 Implementation Details

We employ the Adam optimizer [Kingma and Ba, 2014] with hyperparameters lr = 1e-2, β1 = 0.9, β2 = 0.99, and ϵ =1e15 to train our model. The image resolution in our dataset is 1280×680. Each epoch consists of 1,000 iterations and 8,192 pixels are randomly selected for each iteration. The Canonical Modelling stage requires two epochs to reconstruct the canonical shape of an interactive object. During the Movable Part Decomposition stage, a random individual state is selected for each iteration to learn the corresponding movable part. This phase takes 13 epochs. All experiments are conducted on a single NVIDIA GeForce RTX 3090 GPU.

5.2 Case Study

Car. As shown in Figure 6, the training states of the Car object consist of the canonical state S0, the front door open state S1, and the rear door open state S2. The combination state S1 + S2, where both doors are open simultaneously, is entirely synthesized. The view visualizations showcase our efficacy.

Display Case. Reconstructing a transparent object is inherently challenging, even without interactive components. The Display Case, illustrated in Figure 7, features two glass doors as movable components and contains numerous small gadgets, further complicating the scenario. Synthesized views from various poses demonstrate the effectiveness of our method in handling this complex setting.

Drawer. The Drawer object, depicted in Figure 8, consists of three vertically arranged drawers, resulting in four novel combination states. The synthesized views across different states demonstrate consistent internal structures and external appearances.

Furniture. Figure 9 illustrates the Furniture scene, which includes three types of cabinets. Each cabinet features a movable part located on the front or the top and is surrounded by a pile of sand. The synthesized combination states demonstrate realistic cabinet appearances and consistent background rendering. Additionally, Figure 10 presents rendered images from varying camera poses of multiple synthesized combination states, demonstrating that our method effectively generates geometrically consistent arbitrary novel states.

5.3 Comparison with State-of-the-Arts

As the first work focusing on the novel state synthesis of a human-interactive object, we compare our method with three typical counterparts for 3D or 4D reconstruction, i.e., InstantNGP [M¨uller et al., 2022], 3D Gaussian [Kerbl et al., 2023], and 4D Gaussian [Wu et al., 2024], as shown in Figure 11. All methods are trained on the same states as ours, i.e., the canonical state and individual states. For our method, we present the synthesized combination state where all movable parts are manipulated. For existing methods, we select the closest view to the combination state for comparison, as they lack the inherent designs of synthesizing novel states.

The highlighted visualizations in Figure 11 reveal that existing methods fail to generate unseen states during training. For instance, in the Car object, InstantNGP simply overlays the movable parts, i.e., the front and rear doors, while 3D Gaussian blends the doors together, failing to "open the doors" and resulting in visible quality degradation and prominent artifacts. Similarly, 4D Gaussian associates each individual state with time but still suffers from state overlap. In contrast, our method successfully renders the combination state where both car doors are open, and produce much higher-quality images. In the Drawer object, InstantNGP reconstructs the background wall but fails to model the open drawers. 3D Gaussian is unable to reconstruct the scene entirely, missing both the environment and the open drawers. 4D Gaussian, on the other hand, produces significant ghosting artifacts in the drawers. In comparison, our method successfully synthesizes the novel state with all drawers open. Furthermore, with the effective Movable Part Decomposition and Mutual State Regularization, our method achieves superior rendering quality on novel combination states.

5.4 Ablation Study

As shown in Figure 12, we conduct ablation experiments on the proposed Mutual State Regularization (MSR) and occupancy grid sampling strategies.

Effect of Mutual State Regularization. The first column shows the camera trajectories used to capture training state images. The next two columns compare the synthesized novel states under the Independent Occupancy Grid. Evidently, trajectory inconsistencies lead to severe modeling failures, especially for objects with multiple movable parts, i.e., Furniture and Drawer. Without MSR, noticeable artifacts appear in the highlighted regions of the Furniture, while missing regions are observed in the Drawer object.

Effect of Occupancy Grid Sampling Strategies. The last two columns of Figure 12 present the ablation results for the Independent and Shared Occupancy Grid sampling strategies. No significant differences in rendering quality are observed, demonstrating that both strategies are well-suited for our task. A detailed comparison of these strategies in terms of Memory Allocation (MA) and Training Time (TT) is provided in Table 1. Across all objects, the Independent strategy significantly reduces training time but requires more memory, while the Shared strategy exhibits the opposite behavior.

6 Conclusion

In this work, we present Inter3D, a comprehensive benchmark and a strong baseline method for novel state synthesis of human-interactive objects with multiple movable parts. To facilitate this research, we construct a diverse dataset comprising commonly encountered but challenging interactive objects from everyday life. On this dataset, we introduce a novel and practical evaluation scenario, i.e., novel state synthesis, where only canonical and individual part states are observed during training, while combination states are withheld for evaluation. To efficiently model the full range of states of an interactive object, we leverage Space Discrepancy Tensors with multi-resolution hash encoding, enabling precise learning of individual states. To ensure movable part consistency across training states, even under trajectory perturbations, we propose a Mutual State Regularization mechanism. Furthermore, we explore two occupancy grid sampling schemes to balance memory efficiency and training speed. Extensive experiments demonstrate that our approach achieves highfidelity novel state synthesis, significantly outperforming existing methods. These results establish a strong foundation for advancing research in interactive 3D object reconstruction and open new avenues for exploring 3D object synthesis.

Acknowledgements

Shenzhen Science and Technology Program under Grant ZDSYS20220527171400002, the National Natural Science Foundation of China (NSFC) under Grants 62406197, 62271324, 62231020 and 62371309.

Contribution Statement

This work was a collaborative effort by all contributing authors. Gan Chen and Ying He made equal contributions to this study and are designated as co-first authors. Gang Xu and Ming Li, serving as the corresponding authors, are responsible for all communications related to this manuscript.