ManiSplat: Manipulation Trajectory Synthesis from Monocular Video via Decoupled 3D Gaussian Splatting

1 Introduction

Reconstructing dynamic and interactive 3D scenes from realworld observations remains a fundamental challenge in computer vision and robotics. Most existing dynamic-Gaussian frameworks treat Gaussians as functions of time, implicitly encoding motion within the representation. Methods such as Deformable 3D Gaussian [Yang et al., 2024], 4D-GS [Wu et al., 2024], MotionGS [Zhu et al., 2024], and Spacetime Gaussian [Li et al., 2024b] enable high-quality dynamic rendering but lack explicit object-level disentanglement. Later works, including Ex4DGS [Lee et al., 2024], HUGS [Zhou et al., 2024], EgoGaussian [Zhang et al., 2025a], SplitGaussian [Li et al., 2025], and B´ezierGS [Ma et al., 2025], explicitly separate static and dynamic components to improve geometric stability and temporal coherence. However, these approaches primarily target non-interactive dynamic scenes-such as driving or human motion-where dynamics arise naturally and are only observed rather than induced. They remain limited in interaction-driven scenarios, where articulated agents (e.g., robotic arms) actively manipulate objects, producing complex contact interactions and pose changes. Although IGFuse [Hu et al., 2026] explores interactive Gaussian reconstruction by fusing multi-state scene scans, it relies on discrete observations with object rearrangements rather than continuous robotic manipulation videos.

∗This work was done during an internship at Horizon Robotics.

†Project lead.

‡Corresponding author.

Meanwhile, the Real2Sim2Real (R2S2R) paradigm has emerged as an effective strategy to bridge real-world perception and simulation, enabling consistent transfer of sensing and control for robotic learning[Chebotar et al., 2019]. Recent studies have incorporated 3DGS into robotic pipelines to enhance photorealism and physical consistency. For instance, Robo-GS [Lou et al., 2025], RL-GSBridge [Wu et al., 2025], SplatSim [Qureshi et al., 2025], and RoboGSim [Li et al., 2024a] couple Gaussian Splatting with physics-aware constraints to achieve differentiable rendering and physically consistent control; DREMA [Barcellona et al., 2024] and RE3SIM [Han et al., 2025] further construct learnable digital twins for embodied policy learning. However, most of these frameworks rely on static scene scans and pre-aligned assets, making it difficult to model robots directly from dynamically captured videos.

To address these limitations, we propose a unified framework for reconstructing interaction-driven dynamic Gaussian scenes directly from monocular ego-view robotic videos. First, we design a Graph-Structured Disentangled Representation that explicitly separates the robot arm, manipulable objects, and static background into independently optimizable Gaussian subfields within a scene graph. Second, we present a Task-Oriented Spatio-Temporal Alignment module. This module leverages the inherent logic of manipulation tasks-alternating between Motion and Skill phases-to construct accurate pseudo-ground-truth pose trajectories through Hybrid Pose Estimation and appearance alignment, bridging the domain gap while ensuring stability.

Finally, leveraging this decoupled and controllable representation, we develop a Topology-Preserving Data Augmentation pipeline. By adopting a "skill-preserving, motionadaptive" strategy, we can apply rigid transformations to identified skill segments while autonomously re-planning motion segments. This mechanism enables the generation of large-scale, physically feasible, and spatially diverse synthetic trajectories from a single real-world demonstration, providing high-fidelity data support for downstream embodied policy learning.

In summary, our main contributions are as follows:

• We propose a unified framework for reconstructing interactive and simulation-ready Gaussian scenes from real-world monocular ego-view robotic videos.
• We design a graph-based disentangled representation system that achieves independent modeling and objectlevel controlled reconstruction of the robot, objects, and background.
• We introduce a task-oriented spatio-temporal alignment mechanism that significantly improves trajectory stability and rendering fidelity during interactions through hybrid pose estimation and appearance constraints.
• We develop a topology-preserving data augmentation method capable of synthesizing physically consistent augmented trajectories from a single demonstration, effectively addressing data sparsity and enhancing policy generalization.

2 RelatedWorks

2.1 Real2Sim2Real

The Real2Sim2Real (R2S2R) paradigm bridges real-world data and simulation for robotic learning, enabling consistent transfer of perception and control. Recent efforts integrate 3D Gaussian Splatting (3DGS) into robotic pipelines to enhance photorealism and physical realism. Robo-GS [Lou et al., 2025] fuses mesh geometry, Gaussian kernels, and physics attributes via a Gaussian-Mesh-Pixel binding mechanism for differentiable, physically grounded rendering. RLGSBridge [Wu et al., 2025] and SplatSim [Qureshi et al., 2025] embed 3DGS within reinforcement learning frameworks, ensuring visually consistent control and reducing Sim2Real gaps. RoboGSim [Li et al., 2024a], DREMA [Barcellona et al., 2024], and RE3SIM [Han et al., 2025] further integrate Gaussian splatting with physics simulation to construct learnable, photorealistic digital twins for policy learning. RoboSplat [Yang et al., 2025] directly edits reconstructed Gaussian scenes to generate diverse robotic demonstrations. Unlike these methods that depend on static scene scans for geometry and pose alignment, our approach performs video-based dynamic disentanglement and fusion, leveraging temporal motion cues from real videos to unify appearance and pose across the real and simulated domains.

2.2 Dynamic Gaussian Reconstruction

Recent advances in dynamic scene modeling extend static 3D Gaussian splatting to the temporal domain. Deformable 3D Gaussian [Yang et al., 2024] reconstructs dynamic scenes in a canonical space via a learned deformation field, enabling realtime, temporally smooth rendering. 4D-GS [Wu et al., 2024] jointly encodes space and time through 3D Gaussians coupled with 4D neural voxels for compact, high-resolution rendering. MotionGS [Zhu et al., 2024] introduces optical-flowbased motion priors to decouple camera and object motion, improving monocular reconstruction accuracy. Spacetime Gaussian [Li et al., 2024b] augments 3D Gaussians with temporal opacity and motion attributes to achieve high-fidelity, view- and time-dependent rendering. MEGA [Zhang et al., 2025b] and FreeTimeGS [Wang et al., 2025] further improve efficiency and flexibility through compact color encoding and learnable motion functions, respectively. However, these approaches generally treat Gaussians as temporal functions and fail to disentangle individual moving objects from the scene. To address this, recent works such as Ex4DGS [Lee et al., 2024], HUGS [Zhou et al., 2024], EgoGaussian [Zhang et al., 2025a], SplitGaussian [Li et al., 2025], and B´ezierGS [Ma et al., 2025] explicitly separate static and dynamic components, enabling more stable and consistent motion reconstruction. Nevertheless, existing methods are primarily designed for passive dynamic scenes (e.g., driving or human motion) and cannot effectively model robotic manipulation scenarios, where object trajectories are actively induced by articulated agents. In contrast, our framework explicitly models interaction-driven dynamics within Gaussian fields, bridging the gap between dynamic reconstruction and physically grounded manipulation understanding.

3 Preliminary

3.1 Gaussian Splatting

Following [Kerbl et al., 2023], a 3D scene is represented as a set of Gaussian primitives G = {x, Σ, α, c}, where x denotes the 3D center position, Σ is the spatial covariance matrix, α is the opacity coefficient, and c is the RGB color vector. During rendering, each Gaussian is projected onto the 2D image plane through a differentiable α-blending process, and the final pixel color C is computed by accumulating Gaussian contributions along the viewing ray as C = ∑i∈N ciα′i ∏j<i (1 - α′j ), where N denotes the ordered set of Gaussians intersected by the ray.

We further define the application of a rigid transformation T = (R, t) ∈ SE(3) to a Gaussian as T ⊗ G = {Rx + t, RΣR⊤, α, c}, where R and t denote the rotation and translation components, respectively. This operation transforms the Gaussian's center and covariance into world coordinates while keeping its opacity and color unchanged.

4 Method

As illustrated in Fig. 2, our framework reconstructs controllable Gaussian scenes from real robotic videos and generates diverse, physically feasible demonstration data. The system consists of three core components: Graph-Structured Disentangled Representation (§4.1) models the robot, objects, and background as independent Gaussian subfields to achieve a structured scene representation. Task-Oriented SpatioTemporal Alignment (§4.2) leverages interaction logic (Motion vs. Skill) to construct pseudo-ground-truth object poses, while performing appearance alignment to ensure the rendered Gaussians are consistent with real-world observations in both pose and visual features. Topology-Preserving Data Augmentation (§4.4) synthesizes new trajectory data by rigidly transforming skills and re-planning motions, significantly scaling up the demonstration data. Finally, we present our unified optimization objective in §4.3.

4.1 Graph-Structured Disentangled Representation

Gaussian Scene Graph. To strictly disentangle the scene for downstream control, we represent the world as a dynamic scene graph comprising three fundamental types of nodes: (1) The Background Node (Gbg), which contains static Gaussians representing the environment; (2) The Robot Node (Grobot), modeled as articulated Gaussian clusters driven by the URDF kinematic chain; (3) The Object Nodes (Gvobj ), representing manipulable tools or rigid entities.

At any time step t, the complete scene is composed via the union of these transformed subfields:

Robot Node. The robot geometry is explicitly driven by its kinematic state. Given joint angles qt ∈ Rn , the Gaussian primitives attached to link l, denoted as ¯G(l) robot , are transformed to world space via forward kinematics Tbase→l(qt):

where ⊗ denotes the application of an SE(3) transformation to the Gaussian centers and covariances.

Object Nodes. Each object node v maintains a canonical Gaussian set ¯Gvobj and a time-varying pose Tv(t) ∈ SE(3). The world-space representation is computed as:

By decoupling object-specific Gaussians from the background, our framework enables high-fidelity rendering of object interactions and supports flexible trajectory manipulation.

4.2 Task-Oriented Spatio-Temporal Alignment

Accurate estimation of object pose Tv(t) is critical for reconstruction. Instead of relying on noisy optical flow or heuristic smoothing, we propose a task-oriented approach that utilizes the inherent structure of manipulation tasks-alternating between free-space motion and contact-rich interaction.

Interaction-Aware Phase Segmentation. Following the semantic decomposition in DemoGen [Xue et al., 2025], we partition the demonstration into two distinct phases based on the proximity between the robot end-effector (EE) and the target object. Let d(t) denote the distance between the EE and the object center. We define a proximity threshold τ to classify the trajectory: the Skill Phase (S) occurs when d(t) < τ , representing critical interactions like grasping or placing; otherwise, the segment is classified as the Motion Phase (M), representing free-space transport or approaching. For a typical pick-and-place task, the sequence is naturally segmented as: Mapproach → Spick → Mtransfer → Splace → Mreturn.

Hybrid Pose Estimation. Based on these phases, we construct a reliable pseudo-ground-truth pose trajectory, denoted as T pseudo v (t), by applying phase-specific geometric constraints:

1. StaticConstraint (Mapproach,Mreturn): When the object is not involved in manipulation, it remains static in the world frame. We fix its pose to the initial or final stable state::

where Tv(tinit) denotes the initial stable pose of the object before the task starts, and Tv(tfinal) denotes the final stable pose of the object after the task is completed.

1. Attached Constraint (Mtransfer): During transport, the object is rigidly attached to the gripper. We assume the relative transformation Trel (captured at the end of the grasp skill) remains constant. The object pose is derived purely from robot kinematics:

where Tee(t) is the accurate end-effector pose from the robot controller.

1. Tracking-Based Constraint (Spick, Splace): During interaction, the object may undergo complex movements that are neither static nor fully rigid with the EE. To capture this, we employ a point tracker to track a set of 3D keypoints P = {pk} on the object. We solve for the optimal pose T(t) that minimizes the registration error:

Appearance Alignment. Lighting and texture differences between the URDF-rendered robot arm and its real counterpart often lead to a significant appearance gap. To bridge this domain gap, we utilize SAM2 [Ravi et al., 2024] to generate dynamic masks Mdyn for the moving components. During training, appearance consistency is enforced via masked L1 and LSSIM losses:

To further disentangle dynamic objects from the static background, we impose a mask-consistency constraint between the rendered alpha map of dynamic Gaussians OG dyn and the SAM2 mask Mdyn:Lmask = ∥OG dyn - Mdyn∥1. (8)

Enforcing these alignment losses not only bridges the visual gap but also promotes coarse pose alignment, as visual consistency implicitly guides the geometric optimization of the scene graph.

4.3 Optimization

We jointly optimize the Gaussian parameters and the scene graph structure. In addition to the standard photometric loss used in vanilla 3D Gaussian Splatting [Kerbl et al., 2023], which ensures pixel-level color and structural consistency, we incorporate strong geometric and appearance constraints to handle the challenges of robotic manipulation scenes.

Specifically, we introduce a pose loss Lpose to supervise the object trajectories based on our hybrid estimation:

where ⊖ denotes the distance metric in SE(3), covering both rotation and translation.

The complete optimization objective is formulated as a weighted sum of the rendering, appearance, mask, and pose terms:

By integrating these losses, our framework ensures that the reconstructed scene is not only visually high-fidelity but also physically consistent with the robot's kinematic constraints, facilitating effective Real2Sim2Real transfer.

4.4 Topology-Preserving Data Augmentation

To scale up demonstration data for policy learning, we generate synthetic trajectories by modifying the scene layout while preserving the underlying task logic. We adopt a "SkillPreserving, Motion-Adaptive" strategy that augments object poses and ensures kinematic feasibility through autonomous re-planning.

Skill Augmentation via Rigid Transformation. Manipulation skills rely on the local relative pose between the gripper and the object. For an identified Skill segment Si, we apply a random rigid transformation Taug ∈ SE(3) to the object's pose. To maintain the integrity of the interaction mechanics, the end-effector trajectory is subjected to the same transformation:

This operation ensures that delicate actions, such as grasping or insertion, remain geometrically invariant and are simply transposed to a new task coordinates defined by Taug.

Motion Generation via Re-planning. Motion segments serve as transitions between consecutive skills. Directly transforming the original motion trajectory often results in collisions or kinematic singularities within the augmented layout. To address this, we treat each Motion segment Mi as a navigation problem between the augmented skills. Let the terminal state of the preceding augmented skill be q′start and the initial state of the subsequent augmented skill be q′goal . We invoke a motion planning framework to generate a new, collision-free trajectory that connects these states. This decoupled approach enables the robot to adaptively navigate the modified environment while strictly adhering to the core manipulation logic.

5 Experiments

5.1 Dataset

To comprehensively evaluate our method, we utilize both synthetic and real-world datasets. The synthetic data is generated using the RoboTwin simulation platform [Chen et al., 2025], which provides high-fidelity rendering and precise ground truth parameters. The real-world data is captured using an AgileX dual-arm manipulator, recording complex interaction sequences in physical environments.

For the reconstruction task, we employ a framesubsampling strategy to partition the data into training and testing sets, evaluating the novel view synthesis quality. For the pose estimation task, we rely on the synthetic ground truth pose trajectories to quantitatively assess the accuracy of our estimations.

5.2 Experimental Setup

Implementation details. The training process is conducted for a total of 30,000 iterations using the Adam optimizer. To ensure robust scene initialization, we obtain the initial object poses via GenPose++ [Zhang et al., 2024]. For the segments requiring dynamic tracking, CoTracker [Karaev et al., 2025] is employed to provide point-based estimation. The distance threshold τ , which distinguishes between the motion and skill phases, is set to 0.2. The loss balancing hyperparameters are assigned as λapp = 1.0, λmask = 1.0, and λpose = 1.0. All experiments and runtime evaluations are performed on a single NVIDIA RTX 4090 GPU.

Baselines. We compare our proposed method against several representative Gaussian Splatting-based frameworks, categorized by their modeling capabilities. First, we include 3DGS [Kerbl et al., 2023] as a baseline for static scene reconstruction, serving as a reference that lacks dynamic modeling capabilities. To evaluate dynamic scene modeling, we compare against 4DGS [Wu et al., 2024] and DeformableGS [Yang et al., 2024]. While these methods effectively handle non-rigid deformations, they do not explicitly decouple the object from the environment. Finally, we examine decoupled dynamic reconstruction methods, including HUGS [Zhou et al., 2024], which employs function fitting for pose curves. To ensure a fair comparison with the latter, we extend the dynamics function within HUGS to support full 3D spatial modeling.

5.3 Dynamic Reconstruction

Qualitative Analysis. The qualitative performance of dynamic reconstruction is illustrated in Fig. 3, while quantitative results are detailed in Tab. 1. Due to the lack of explicit object-environment decoupling, 4DGS and DeformableGS struggle to handle the rapid motion of the robotic arm, often failing to deform correctly and resulting in significant artifacts and motion blur. While decoupled methods address this to some extent, they face distinct challenges in pose estimation. HUGS employs dynamics curve fitting to constrain motion; however, during high-degree-of-freedom interactions-such as the moment of grasping-the strict smoothing function tends to obliterate the necessary abrupt changes in object pose, similarly leading to blurred reconstruction. Our method addresses these limitations through a multi-stage pose handling strategy. In the motion phase, we leverage robot kinematics to maintain relative static attributes, while in the abrupt skill phase, we incorporate tracking supervision. This allows us to maintain stability in both pose and appearance throughout the entire interaction sequence.

Quantitative Analysis. Quantitatively, 3DGS yields the lowest PSNR as it lacks dynamic modeling capabilities. While 4DGS and DeformableGS improve upon this with dynamic reconstruction, their inability to decouple the scene results in poor performance when evaluating the manipulator and object separately. For fair comparison, we supply HUGS with ground-truth robot joint angles, resulting in manipulator reconstruction quality comparable to ours. However, their object reconstruction metrics are significantly lower, as HUGS suffers from over-smoothing in the presence of complex contact. Our method achieves superior performance across all metrics by accurately capturing the complex interaction dynamics.

5.4 Pose Estimation

Pose Estimation Accuracy. We evaluate pose accuracy using RoboTwin ground-truth. As shown in Tab. 2, our method achieves the lowest error, outperforming GenPose++ and HUGS. Specifically, we reduce Et to 0.5864 cm and ER to 5.185◦ , a significant improvement over GenPose++ (0.9965 cm, 7.109◦ ). By leveraging kinematic data for refinement, our approach effectively mitigates estimation drift and ensures high-fidelity reconstruction.

Trajectory Analysis. Fig. 4 visualizes real-world trajectories. Compared to the GenPose++ baseline, our method demonstrates superior temporal stability by suppressing highfrequency noise. Crucially, it accurately captures the abrupt pose transitions during gripper engagement (e.g., near Frame 50), which are typically obscured by noise in single-frame estimation methods. This ensures a stable trajectory during the motion phase while precisely reflecting discrete physical events during the skill phase.

5.5 Data Augmentation

Beyond reconstruction, our decoupled representation enables high-fidelity data augmentation. Figure 5 demonstrates this capability using a scene containing a flower and a vase. The first row displays the original grasping demonstration. The bottom two rows show augmented sequences where both the position and orientation of the flower are altered, while the vase remains stationary. Our method successfully generates photorealistic novel scenarios from a single source video, preserving interaction fidelity across different spatial configurations. This capability is particularly valuable for scaling datasets for downstream tasks, such as Vision-LanguageAction (VLA) model training, where diverse object poses are critical for generalization.

We conduct an ablation study to evaluate the individual contributions of the pose constraint Lpose, appearance alignment loss Lapp, and opacity mask supervision Lmask. As shown in Tab. 3, Lpose provides the most significant performance gain, as it ensures geometric stability and reduces the high-frequency jitter inherent in sparse-view reconstruction. The introduction of Lapp further refines the rendering quality by enforcing photometric consistency across the sequence.

Notably, while the opacity-controlling Lmask yields a relatively marginal improvement in global metrics, it is indispensable for obtaining high-quality decoupled representations. By explicitly supervising the Gaussian opacity, Lmask effectively eliminates floaters and ensures sharp, physically plausible boundaries between the robot manipulator and the manipulated objects. This precise disentanglement is critical for downstream tasks like data augmentation, where clean object-environment separation is required for realistic recomposition.

6 Limitation and FutureWork

Despite its effectiveness in structured scene reconstruction, our framework still has limitations. In particular, rendering quality may degrade when data augmentation introduces large spatial offsets from the original demonstration. This is mainly due to the sparse-view nature of 3D Gaussian Splatting, where Gaussians optimized from limited viewpoints may lack sufficient geometric consistency for large-viewpoint generalization.

Several directions remain for future work. First, replacing 3DGS with more geometrically constrained representations, such as 2D Gaussian Splatting (2DGS), could improve surface consistency and view synthesis quality. Second, augmentation quality could be further enhanced by incorporating multiple reference demonstrations. By fusing multi-state information from diverse interaction sequences of the same object, the framework could build a more robust scene representation and support higher-fidelity synthesis under large spatial perturbations.

7 Conclusion

We present a unified framework for reconstructing interactive, simulation-ready Gaussian digital twins from realworld monocular egocentric robotic videos. Our framework adopts a graph-based disentangled representation to separately model the robot, objects, and background, enabling object-level controllable reconstruction. A task-oriented spatio-temporal alignment mechanism improves trajectory stability and rendering fidelity by separating free-space motion from contact-rich interaction phases. To alleviate data sparsity, we further introduce a topology-preserving augmentation strategy that synthesizes physically consistent trajectories from a single demonstration. Overall, our approach provides a scalable pipeline for converting raw monocular robotic videos into high-fidelity digital assets.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (No. 62576308), Zhejiang Provincial Natural Science Foundation of China (No. LZ24F030005), and Fundamental Research Funds for the Central Universities (No. 226-2025-00167).