Towards Robust Deterministic and Probabilistic Modeling for Predictive Learning

1 Introduction

Predictive learning, a self-supervised learning method, excels in uncovering latent structures within unannotated spatiotemporal data. This topic models temporal evolution by predicting future frames from given ones, offering extensive applications in autonomous driving [Jin et al., 2024], climate modeling [Lam et al., 2022], traffic flow [Nie et al., 2024d], robotics [Gupta et al., 2022], and popular world simulations [Nie et al., 2024b]. However, spatiotemporal sequences, abundant and readily available in nature, typically exhibit intricate spatial correlations, movement trends, and multi-object interaction in practical scenarios.

Struggling with the inherent complexity and randomness of future events, predictive learning has developed into two main approaches: recurrent-based and recurrent-free. Recurrent-based methods consist of recurrent unit variants

* Corresponding Author.

(e.g., LSTM [Hochreiter and others, 1997], ConvLSTM [Shi et al., 2015], and ST-LSTM [Wang et al., 2017]) and state transition connections across timesteps. These methods embed motion information into hidden states to model temporal dynamics, as shown in Fig. 1. Recurrent-free methods employ parallel spatiotemporal translators instead of recurrent units to model spatiotemporal dependencies. SimVP [Gao et al., 2022], TAU [Tan et al., 2023], and TAT [Nie et al., 2024a] propose elaborate spatiotemporal learning modules for implicit temporal evolution capture. These models are required to adeptly acquire sophisticated motion patterns autonomously. DMVFN [Hu et al., 2023], on the other hand, utilize optical flow to gracefully refine explicit motion depiction, effectively diminishing the occurrence of artifacts. Nonetheless, these methods inevitably exhibit limitations over time, as shown in Fig. 2. DMVFN [Hu et al., 2023] captures motion effectively but often fails to maintain visual consistency, yielding "correct" yet not "ideal" results, e.g., more details of clothes or horse heads are lost. Conversely, TAU [Tan et al., 2023] excels in preserving detailed visual features but struggles with accurate motion representation, e.g., the motion of the arm produces deformation.

The observations above reveal that low-level details typically remain stable, whereas high-level motion exhibits dynamic variations. This seems intuitive, for example, an individual's visual features remain relatively constant in the short term, while their behavior exhibits high stochasticity. The prediction can be improved by designing models based on these data properties, which existing methods have not fully explored. To this end, we address two key questions:

• (i) How to represent and model high-level motion dynamics? Explicit modeling is essential for understanding complex motion dynamics. The most common methods like optical flow describe a one-to-one pixel mapping with two channels, but this approach often overlooks interactions between more pixels, as shown in Fig. 1. Therefore, we formulate motion flow by computing inter-frame latent similarities, creating a many-to-many mapping with more channels. Although motion flow captures richer motion patterns, its long-term motion distribution modeling is more challenging. To address this, we introduce a motion diffuser that employs a spatiotemporal state space model with the diffusion structure to learn the transition from Gaussian noise to true motion distribution.
• (ii) How to enhance low-level details and multi-scale dynamics? Most of the leading methods typically extract features in the original latent space. In contrast, we propose a spectral-aware enhancer that models each frame individually in the frequency domain, preserving more visual detail features. To address significant cross-scale motion variations between consecutive frames, we implement motion flow sharing and motion-visual warping strategies. These approaches significantly reduce complexity and enhance long-term prediction capabilities.

In this paper, we present a new perspective through Disentangling Deterministic and Probabilistic (DDP) modeling for robust predictive learning. Extensive experiments demonstrate the effectiveness of DDP for various prediction scenarios. Our key contributions are summarized as follows:

• Introduced DDP, a novel framework that disentangles deterministic visuals from probabilistic motion for spatiotemporal prediction.
• Proposed explicit motion modeling (motion flow) and a spatiotemporal diffusion model to capture its complex, probabilistic nature.
• Developed a spectral-aware enhancer and motion-visual warping techniques to improve low-level detail preser-

vation and handle multi-scale dynamics.

• Extensive experiments show that DDP achieves state-ofthe-art performance across various real-world scenes.

2 RelatedWork

2.1 Predictive LearningModels

Recurrent-based models have historically dominated predictive learning. ConvLSTM [Shi et al., 2015] augments LSTM's spatial learning with convolutional architecture. PredRNN [Wang et al., 2017] introduces spatiotemporal LSTM, capturing spatial and temporal dependencies. PredRNN++[Wang et al., 2018b] addresses gradient vanishing through a gradient highway unit. MCnet [Villegas et al., 2017] decomposes the motion and content modeling with LSTM and CNN. SADM [Bei et al., 2021] fuses the content semantic maps and optical flow motion maps for future frame prediction. E3D-LSTM [Wang et al., 2018c] extends LSTM with 3D convolution. PredRNNv2 [Wang et al., 2022] employs curriculum learning and memory decoupling loss. WaST [Nie et al., 2024d] presents an innovative wavelet-based spatiotemporal framework for modeling spatial frequency and temporal variations. ModeRNN [Yao et al., 2023] uses spatiotemporal slots to extract visual dynamics components, addressing mode collapse.

Recent research has shifted towards recurrent-free models to overcome parallelization limitations. vid2vid [Wang et al., 2018a] decomposes video visuals and motion for frame prediction with spatiotemporal adversarial learning. [Wu et al., 2020] decomposes the background scene and moving objects with instance maps. SimVP [Gao et al., 2022] employs Inception modules with UNet architecture for temporal dependency learning. TAU [Tan et al., 2023] decomposes temporal attention into intra-frame static and inter-frame dynamical components. DMVFN [Hu et al., 2023] proposes a dynamic multi-scale voxel flow network. In contrast, our approach disentangles deterministic visuals and probabilistic motion in latent space to enhance prediction accuracy.

Overall Framework

Framework Details

Figure 3: Illustration of the DDP architecture. Our DDP features two pathways in latent space: the deterministic path and the probabilistic path. The probabilistic path models the distribution of well-defined motion flows through a motion diffuser, while the deterministic path preserves and enhances visual details in the frequency domain via a spectral-aware enhancer.

2.2 State Space Models

State Space Models (SSMs) with efficient hardware-aware designs, e.g., Mamba [Gu and Dao, 2023], have recently demonstrated significant potential for long sequence modeling with linear complexity. ViS4mer [Islam and Bertasius, 2022] employs a 1D Structured State Space Sequence (S4) model for long-range temporal dependencies in video classification. S4ND [Nguyen et al., 2022] extends 1D S4 to multi-dimensional data, including 2D images and 3D videos. TranS4mer [Islam et al., 2023] combines self-attention and S4 for movie scene detection, while S5 [Wang et al., 2023] introduces a selective mechanism to S4, enhancing its performance in long-form video understanding. DiffuSSM [Yan et al., 2023] replaces attention mechanisms with a more scalable SSM-based backbone for high-resolution image generation. ViM [Zhu et al., 2024] demonstrates that self-attention is not essential for visual representation learning by constructing a pure SSM-based model. VMamba [Liu et al., 2024] addresses the direction-sensitive issue by introducing a crossscan module to traverse the spatial domain. Our work explores spatiotemporal diffusion SSMs for probabilistic modeling of motion dynamics.

3 Method

3.1 Framework Overview

The spatiotemporal predictive learning aims to model spatial and temporal dependencies of given past T frames I = {It}Tt=1 to predict the most reasonable future T ′ frames I ′ = {I′t }T +T ′ t=T+1 , where It ∈ RC×H×W denotes the tth frame. Our DDP involves three important components: (i) Probabilistic Motion Modeling, (ii) Deterministic Visual Modeling, and (iii) Motion-Visual Warping. We detail them in the following.

3.2 Probabilistic MotionModeling

We first construct the visual-agnostic motion flow {Ft→t+1}, t ∈ {1, 2, . . . , T - 1} by computing the token similarity across frames. Then the motion diffuser models past motion flow MDiff(Ft→t+1) to estimate future motion flow ˆFT→T+t′ in an iterative manner.

Motion Flow Formulation. Given the latent feature maps Xt ∈ RC×N , t ranges from 1 to T and N = H ×W. As illustrated in Fig. 4(a), we represent the ith feature patch as Xit and compute dot product similarity for two consecutive frames {Xt, Xt+1} to formulate the motion flow:

Unlike optical flow that establishes one-to-one pixel correspondence, motion flow Fi,j t→t+1 captures the many-to-many patch relationship across frames, showing the impact of the ith patch on jth patch in different frames.

Motion Flow Estimation. To effectively capture long-term motion patterns, we explored a new spatiotemporal diffusion state space model, motion diffuser, which is constructed from a sequence of SSMs. They are systems that map a 1D function or sequence x(t) ∈ R ↦→ y(t) ∈ R. It can be expressed as a linear Ordinary Differential Equation (ODE):

where A ∈ RM×M and B,C ∈ RM are its parameters and h(t) ∈ RM denotes the hidden state. The discrete versions [Karras et al., 2022] of ODE include a timescale parameter ∆ to transform the continuous parameters A,B to discrete parameters A,B. One common discrete method is the Zero-Order Hold (ZOH), represented as:

Mamba [Gu and Dao, 2023] further extends the discretization process with a selection mechanism. Based on these, we build the motion diffuser using stacked SSM blocks. Given the motion flow of the past T frames {Ft→t+1} ∈ R(T-1)×N×C , where C = N. These sequences are input into the motion diffuser to perform the forward and backward spatiotemporal selective scan, as shown in Fig. 3. Specifically, the sequences are aggregated after the forward scan and the reverse scan of the flipped sequences. Finally, the estimated motion flow can be formulated as:

where we directly estimate the motion flow from the Tth to the (T +t′ )th frame to reduce error accumulation. Moreover, as shown in Fig. 4(b), the MDiff cross-scale shares the estimated motion flow through convolutional projection to reduce multi-scale iterations. The diffusion process progressively adds noise to the motion flow, as shown in Fig. 5. Let βt represent the noise variance ratio at time t, and αt = 1-βt. With context motion flow as condition c and denoised motion flow as z, these are concatenated as input. The training loss for the motion diffuser ϵϕ (zt ; t) is:

3.3 Deterministic Visual Modeling

While recurrent-based models excel at capturing motion patterns, they often struggle to maintain visual consistency. To address this limitation, we introduce the Spectral-Aware Module (SAM) designed to enhance frequency-domain representations (as shown in Fig. 6(a)). Adopting a MetaFormerlike paradigm [Yu et al., 2022; Nie et al., 2024c], SAM comprises: (i) Dilated Reparam Convolution (DRConv) [Ding et al., 2023] for token mixing, which augments a non-dilated large kernel with parallel reparameterizable dilated small kernels, and (ii) Energy-based Frequency Channel Mixing

𝐅𝐅𝑡𝑡+1→𝑇𝑇

𝐅𝐅𝑡𝑡→𝑇𝑇 =�𝑖𝑖=𝑡𝑡

𝐅𝐅𝑖𝑖→𝑖𝑖+1

Motion Flow Diffusion (single channel case)

Appearance-Motion Warping Figure 5: Illustration of the single-channel diffusion process. The vector is reshaped into a spatial format for visualization.

(EFCM) for channel mixing. EFCM computes the mean ˆµ = 1N ∑Ni=1 xi and variance ˆσ2 = 1N ∑Ni=1 (xi - ˆµ)2 of latent features X, where N = H×W. The energy value [Yang et al., 2021] is determined by minimizing:

where δ represents the hyper-parameter, and ei,j is the energy value of the target token ti,j, where i ranges from 0 to H - 1, and j ranges from 0 to W - 1. The energy feature Xe is formed by grouping all ei,j values. Then, we rescale the energy value to limit excessively large energy values:

For energy features, we pool it into a global vector x ∈ RC×1×1 , and then transform it to Fourier space:

where amplitude component A(x)(z) and phase component P(x)(z) of F(x)(z) represent different information, thus we introduce attention-based operations to enhance A(x)(z) and P(x)(z) respectively:

where Afilter(·) and Pfilter(·) denote 1 × 1 filters for the amplitude and phase components. The symbol ⊙ signifies the Hadamard product for attention weighting. Then we convert the Fourier features to their original space via the inverted Fourier transform F -1 (A(x) (z)′ ,P (x) (z)′ ). Finally, we context broadcast it to the original input.

3.4 Motion-Visual Warping

Inspired by the warping operation in optical flow, which uses the flow Ft→t+1 to map It to It+1 in pixel space. Similarly, the motion-visual warping applies motion flow ˆFT→T+t′ on the observed T visual features Xt to obtain the (T + t′ )th features ˆXT+t′ in latent space:

Unlike optical flow which only maps the current frame, this warping operation integrates all prior visual features Xt via the motion flow Ft→T , as shown in Fig. 6(b). This process takes place at each scale, and then the decoder produces future frames by converting the warped features from latent to pixel space.

4 Experiments

We demonstrate the effectiveness of the DDP model with multi-scenario evaluations. These scenarios are crucial for numerous applications requiring robust spatiotemporal predictive models. We demonstrate that the DDP model performs favorably against the state-of-the-art models on five challenging datasets corresponding to these scenarios.

Implementation Details. Our method uses PyTorch on an NVIDIA A100 GPU, training with 16-sequence minibatches, the Adam optimizer, and the OneCycle scheduler. We apply a weight decay of 5e-2 and select learning rates from {1e-2 , 5e-3 , 1e-3 } for stability. We use the MSE loss to supervise training and stochastic depth for regularization.

4.1 HumanMotions: UCF Sports

Dataset and Setup. UCF Sports [Rodriguez et al., 2008] comprises 150 videos from diverse sports scenes, depicting 10 distinct actions with complex human motion patterns. Following STRPM [Chang et al., 2022], we scale resolution from 480 × 720 to 512 × 512, using 6,288 sequences for training and 752 for testing. The model observes 4 frames and predicts 1 frame (4 → 1) during training and 6 frames (4 → 6) during testing.

Main Results. Tab. 1 presents model performance, reporting PSNR and LPIPS metrics for (T +1)th and (T +6)th frames. DDP demonstrates significant performance gains over other methods. This dataset includes complex scenarios and motion patterns, like camera movement and motion blur. Our design effectively addresses these issues by separating visuals and motion, as shown in Fig. 2, demonstrating the potential for real-world applications and scalability to high-resolution spatiotemporal data.

4.2 Synthetic Motions: MovingMNIST

Dataset and Setup. The Moving MNIST [Srivastava et al., 2015] dataset is constructed by randomly sampling two digits with 64×64 pixels from the MNIST dataset and making them float and bounce at boundaries with a constant direction and velocity. There are 10,000 sequences for training and 10,000 for testing. The model observes the first 10 frames and predicts the next 10 frames.

Main Results. Tab. 2 shows MSE and PSNR metrics of DDP network against the state-of-the-art predictive learning methods. Our model significantly outperforms these methods in both metrics. We also show a prediction example in Fig. 7(b). Notably, SimVP [Gao et al., 2022] improves visual details through introducing IncepU, but missing part of the motion modeling leads to error accumulation over time as shown in the last row of Fig. 7(b). In contrast, DDP using a motion diffuser to model long-term motion patterns explicitly can mitigate this issue. This suggests that the DDP models synthetic motions better than other methods.

4.3 Driving Scenes: KITTI&Caltech

Dataset and Setup. The generalization ability is crucial for real-world driving scenes. The KITTI&Caltech [Geiger et al., 2013; Doll´ar et al., 2009] dataset evaluates generalization ability across different datasets. Following standard practice [Gao et al., 2022], we train the model on the KITTI [Geiger et al., 2013] dataset and evaluate it against the Caltech Pedestrian [Doll´ar et al., 2009] dataset. We resize the resolution to 128×160, and models predict the next frame by previously observed 10 frames.

Main Results. Tab. 3 shows the quantitative results of the proposed model and mainstream methods. The DDP model achieves strong performance under all metrics consistent with previous observations. These empirical results demonstrate the effectiveness of the DDP model for modeling spatiotemporal driving data. Qualitative visualizations Fig. 7(a) shows that our method can better predict lane lines and pinpoint distant, small entities than other methods, which indicates the potential generalization across scenes.

4.4 Traffic Flow: TaxiBJ

Dataset and Setup. TaxiBJ [Zhang and others, 2017] comprises taxi GPS trajectory data in Beijing, with 30-minute intervals and 32×32 spatial granularity. Models predict 4 future frames based on 4 observed frames. Complex road network dependencies and non-linear temporal dynamics have historically challenged traffic forecasting methods.

Main Results. Quantitative results are presented in Tab.3, with qualitative visualizations in Fig.8. TAU [Tan et al., 2023], despite introducing a temporal attention unit and setting benchmarks in several datasets, fails to adequately capture road spatiotemporal dependencies, resulting in prediction inaccuracies (Fig. 8, last two rows). For unstructured data (e.g., traffic, climate), intensity is similar to visual features in structured data. DDP still works well in both intensity and dynamics, consistently outperforming other approaches with minimal intensity differences across most regions. Notably, the optical flow-based DMVFN [Hu et al., 2023] method underperforms in these scenes. In contrast, DDP, making no data-specific assumptions, demonstrates broader applicability across various modalities.

4.5 Global Climate: WeatherBench

Dataset and Setup. WeatherBench [Rasp et al., 2020] contains climatic data from 1979 to 2018, re-gridded to 5.625◦ (32 × 64 grid points) and 1.40625◦ (128 × 256 grid points). We evaluate temperature prediction at 5.625◦ resolution, using 2010-2015 for training, 2016 for validation, and 2017-

2018 for testing. The model forecasts 12-hour temperature based on 12-hour historical global temperature data.

Main Results. For the WeatherBench dataset, Tab. 3 presents quantitative comparisons with state-of-the-art methods. DDP consistently outperforms all other state-of-the-art methods across all reported error metrics (MSE, MAE, and RMSE). It is particularly noteworthy that some methods, such as PhyDNet and DMVFN, exhibit significantly higher error values on this task, indicating substantial deviations in their predictions from the ground truth climate patterns. This comprehensive superiority suggests DDP's architecture is particularly wellsuited for capturing the complex, long-range spatiotemporal dependencies inherent in climate data.

4.6 Ablation Studies

In this section, we further perform extensive ablation studies to study the components' effectiveness in our DDP.

Ablation of the Probabilistic Path. We implemented various spatiotemporal modules as motion diffusers (Tab. 4(a)), utilizing convolution (Conv) and self-attention (SA). Replacing spatiotemporal SSMs with these variants, our experiments demonstrate that SSMs outperform other methods in terms of MSE metric, validating their efficacy in modeling spatiotemporal data. Moreover, We experimented with various backward scanning methods (Tab. 4(b)), where the spatiotemporal flip scanning yielded the lowest MSE.

Ablation of the Deterministic Path. We substituted the deterministic branch of the spectral-aware module (SAM) with various metaformer modules: Vision Transformer (ViT), Swin Transformer, and ConvNext, as shown in Fig. 9. Additionally, we compared different spectral architectures, in-

cluding Fast Fourier Convolution (FFC), Fourier Neural Op-

erator (FNO), and Wavelet Gating Network (WGN). Results

demonstrate that our SAM, synthesizing the strengths of these

frameworks, achieves optimal performance.

Ablation of the Long-Term Prediction. To investigate the components affecting long-term predictions in DDP, we tested long-term input and output scenarios (Fig. 10). We replaced the Motion Diffuser (MDiff) with a deterministic model and restricted Motion-Visual Warping (MVW) to map only current frame latent features. Results indicate that the motion diffuser significantly impacts long-term dynamics, while MVW marginally enhances this capability.

5 Conclusion and FutureWork

This paper presented Disentangling Deterministic and Probabilistic (DDP) modeling, a novel predictive learning framework employing two distinct latent pathways. The probabilistic pathway explicitly models complex, many-to-many motion patterns via a motion diffuser, while the deterministic pathway utilizes a spectral-aware enhancer to preserve and amplify visual details in the frequency domain. This dualarchitecture design effectively balances robust visual consistency with the accurate capture of intricate, long-termmotion dynamics. Comprehensive experimental validation demonstrates DDP's significant outperformance of existing state-ofthe-art methods, yielding qualitatively superior visual predictions. Future research will focus on addressing the scalability to substantially longer sequences (e.g., >100 frames) and higher resolutions (e.g., 4K). These advancements present considerable challenges, requiring optimized resource management and highly efficient inference strategies.

Contribution Statement

Xuesong Nie and Haoyuan Jin made equal contributions.