CDSTraj: Characterized Diffusion and Spatial-Temporal Interaction Network for Trajectory Prediction in Autonomous Driving

1 Introduction

In the domain of autonomous driving (AD), trajectory prediction plays a pivotal role by providing invaluable insights for the subsequent trajectory planning module, thereby enhancing the safety of navigation in complex and dynamic traffic scenarios [Huang et al., 2022]. The continuous presence of mixed traffic flow necessitates a trajectory prediction model that deeply understands the heterogeneity and uncertainty of traffic scenarios [Liao et al., 2024e]. Despite the proliferation of trajectory prediction models, significant gaps remain in the thorough investigation of the impact of heterogeneous and uncertain traffic scenarios on future motion.

The initial gap we pinpoint hinges on the accurate simulation of future traffic scenarios-a cornerstone for enhancing trajectory prediction precision [Wang et al., 2023; Liao et al., 2024d]. The challenge is amplified by the intrinsic uncertainties characterizing traffic dynamics, making the accurate forecast of future scenarios a complex endeavor. Prevailing models have primarily concentrated on uncertainties inherent to the target agent [Zhao et al., 2019; Alahi et al., 2016; Gupta et al., 2018], thereby neglecting the comprehensive uncertainty pervasive in the overall traffic scenarios. This oversight highlights an imperative need for trajectory prediction frameworks to adeptly navigate and mitigate uncertainties, facilitating a more accurate and holistic simulation of future traffic scenarios.

The second identified gap related to the sophisticated mechanisms by which traffic scenarios exert influence on human driving behaviors. The decision-making processes of human drivers are profoundly shaped by their interactions with other traffic agents, with such interactions predicated on a nuanced interplay between spatial and temporal dimensions. Nevertheless, prevailing models have primarily focused on capturing spatial interactions, starkly overlooking the critical temporal dynamics [Wang et al., 2023]. This neglect highlights the critical necessity for models that adeptly incorporate both spatial and temporal interaction.

To address these challenges, we introduce a novel generative model, CDSTraj, which is built on a dual architecture. As shown in Fig. 1. Our model employs an encoder designed to generate spatial-temporal features from past states. while significantly fusing confidence features to ensure the stability and reliability of the representation. This enriched feature set serves as the foundational input for a decoder that generates future trajectory predictions. A key innovation in our approach is the introduction of a characterized diffusion, which is seamlessly integrated with a spatial-temporal interaction network. This synergistic combination allows our model to be aware of the indeterminacy associated with both scene-toagent and agent-to-agent contexts. Consequently, this leads to trajectory predictions that are both more accurate and reliable, even in dynamically changing environments. Overall, our main contributions can be summarized as follows:

• We introduce the Characterized Diffusion Module, a novel approach that enhances trajectory prediction by dynamically simulating future traffic scenarios through iterative uncertainty mitigation. This module significantly augments the predictive accuracy by integrating complex, contextual scenario features, allowing for a more nuanced understanding of potential motion.
• We unveil the Spatial-Temporal Interaction Module, which leverages a spatio-temporal attention mechanism to meticulously model and analyze the intricate interactions characteristic of traffic scenarios. Unique to this module is its three-stage architecture, meticulously designed to efficiently capture and process information across spatial and temporal dimensions.
• Our rigorous empirical investigations underscore the superiority of our model over the existing trajectory prediction model. Through extensive experiments, our model achieves top performance on public datasets such as NGSIM, HighD, and MoCAD. The exceptional performance on the MoCAD dataset is especially significant, offering a fresh perspective for evaluation through its unique right-hand drive configuration and obligatory left-hand traffic regime, thus underscoring our model's adaptability and accuracy in varying driving scenarios.

2 RelatedWork

Trajectory Prediction for Autonomous Driving. Early trajectory prediction methods primarily relied on manual feature engineering and rule-based techniques, including linear regression and Kalman filters [Prevost et al., 2007]. These methods were limited in capturing complex interactions in dynamic environments. The field evolved significantly with the introduction of deep learning, specifically Recurrent Neural Networks (RNNs) [Kim et al., 2017] and Long ShortTerm Memory (LSTM) networks [Altch´e and de La Fortelle,

2017; Alahi et al., 2016; Liao et al., 2024b]. These advancements enabled the capture of temporal dependencies in trajectories. Further innovation came with Graph Neural Networks (GNNs) [Zhou et al., 2021; Liao et al., 2024a; Liao et al., 2024c], which provided a more nuanced approach to modeling interactions among agents in crowded scenes.

Generative Models for Trajectory Prediction. Generative models like Generative Adversarial Networks (GANs) [Gupta et al., 2018] and Variational Auto-Encoders (VAEs) [Lee et al., 2017] have gained prominence in trajectory prediction. GANs involve a generator and a discriminator engaged in mutual learning, while VAEs use a generative model and a variational posterior, the optimization of which can be complex. Diffusion models, on the other hand, offer a simplified training process by focusing on matching forward and inverse diffusion processes. To the best of our knowledge, this work is the first to leverage diffusion models for capturing the confidence features.

Denoising Diffusion Probabilistic Models. Denoising Diffusion Probabilistic Models (DDPM) [Ho et al., 2020], known as diffusion models, have gained prominence as powerful generative models for various applications, including image [Ramesh et al., 2022; Rombach et al., 2022; Liao et al., 2024f], video [Ho et al., 2022], and 3D shape[Poole et al., 2022] generation. Inspired by the diffusion models' enormous representation capacities in numerous generation tasks, our work introduces the novel application of diffusion models to trajectory prediction in autonomous driving, addressing the challenges of modeling uncertainties and complex agent interactions in dynamic environments.

3 ProblemFormulation

The paramount objective of this study is the precise prediction of trajectories for all entities within the proximity of an autonomous vehicle (AV) situated in an environment characterized by mixed autonomy. For this purpose, every entity proximal to the AV is designated as a target agent. At a specific time tc, our model endeavors to utilize the historical states of both the target agent and its neighboring agents to predict the future trajectory of the target agent, represented as Y0, extending to a future time tc + tf . The historical state since the time tc - th is denoted by X0 for the target agent andXi for the neighboring agents.

The novelty of our model lies in its exploitation of anticipated future traffic scenarios, specifically the future trajectories of neighboring agents, to enhance the accuracy of trajectory prediction for the target agent. To this end, we develop an Characterized Diffusion Module, designed to systematically mitigate the uncertainty inherent in the trajectories of neighboring agents, thereby enabling the accurate prediction of their future trajectories Yi. Formally, our prediction model Φ is represented as:

4 Methodology

4.1 Overview of the Model Framework

As illustrated in Figure 2, our model comprises three primary components: the Characterized Diffusion Module, the ST Interaction Module, and a Multi-modal Decoder. The Characterized Diffusion Module employs an inverse diffusion process to generate the future trajectories of neighboring agents. Concurrently, the ST Interaction Module extracts Spatial-Temporal Interaction features through a methodical alternation between spatial and temporal dimensions. Ultimately, the predicted trajectories are generated using a multimodal decoder, which synthesizes the processed information to produce accurate trajectory prediction.

4.2 Characterized Diffusion

In order to predict the future trajectories of neighboring agents, the Characterized Diffusion Module treats trajectory prediction as the reverse process of motion Characterized Diffusion and gradually eliminates the uncertainty of future trajectories by learning a parameterized Markov chain with the observed historical states. More specifically, during the diffusion process, the uncertainty inherent in future trajectories is simulated by iteratively introducing Gaussian noise. Conversely, in the inverse diffusion process, this uncertainty is iteratively mitigated to derive the anticipated future trajectories accurately. The detailed procedure is shown in Fig. 3. Mathematically, let C be the future trajectory of the neighboring agents. Firstly, We initialize the diffused unit C0 :

We use a forward diffusion operation fdiffuse(·) to add uncertainty to Cδ-1 and transition to diffused unit Cδ :

where Cδ is the diffused unit at the δth diffusion step, Γ is the total number of steps.

After n iterations, our model is able to capture a comprehensive spectrum of uncertain traffic scenarios with maximum coverage. Then, the inverse diffusion process is applied to accurately derive the future trajectories of neighboring vehicles. This step-by-step refinement ensures high fidelity in predicting vehicle movements, effectively addressing the inherent complexity of dynamic traffic scenarios. Due to the indeterminacy of future trajectories, in the denoising procedure, it is usually more reliable to capture more than one inverse unit to extract sufficient trajectory information. Therefore, we draw K independent and identically distributed samples to initialize the denoising unit ̂CΓk from a normal distribution.

We formulate the trajectories generation process as a reverse diffusion by iteratively applying a denoising operation fdenoise(·) to obtain the denoised unit ̂Cδk conditioned on historical states X0, Xi and, unit ̂Cδ+1 k :

In a denoising module, two parts are trainable: a transformer-based context encoder fcontext(·) to learn a socialtemporal embedding and an uncertainty estimation module fϵ(·) to estimate the uncertainty to reduce. Mathematically, the δth denoising step works as follows:

where αδ and ¯αδ = ∏δi=1 αi are parameters in the diffusion process and z ∼ N(z; 0, I) is the uncertainty. We leverage a context encoder fcontext (·) on historical states (X, XN ) to obtain the context condition Cencoder, and estimates the uncertainty ϵδθ in the uncertain trajectory ̂Yδ+1 k through uncertainty estimation fϵ(·) implemented by multi-layer perceptions with the context C; Eq. 7 provides a standard denoising step.

The final K extracted units are ̂C = {̂C01 , ̂C02 , ..., ̂C0K }. Finally, we obtain the future trajectories with:

where Ω denotes a multi-layer perception with learnable parameter matrix Wcf .

4.3 Spatial and Temporal Interaction

To enhance the precision of modeling the temporal and spatial dynamics of vehicle interactions within the environment, the Space-Time (ST) Interaction Module is designed with a novel structure that alternates between temporal and spatial dimensions. This module is composed of three key components, as illustrated in Fig. 2: 1. Temporal Encoder: where the temporal dependencies of all the agents are extracted from the historical states by a temporal encoder. 2. Spatial Encoder:, which plays an essential role in extracting the spatial relation between the target agent and neighboring agents. 3. ST Fusion: which aims to deeply capture the spatial-temporal interaction.

1) Temporal Encoder: To begin with, a temporal embedding vector F t is obtained from the historical states xt at the tth timestamp using a fully connected layer with a learnable parameter matrix Wemb as follows:

where δ is the fully connected layer and Φ is the LeakyReLU activation function. It can be defined as follows:

where Winit denotes the learnable parameter matrix of encoder ftem and ht denotes the temporal feature at the tth timestamp, which is updated at each timestamp based on the hidden state at the previous timestamp and the embedding vector at the current timestamp. Here, we apply ftem to every single agent shared with the parameters of encoding ftem to reduce variation across numerous agents at different timestamps. Finally, for the target agent, we obtain H0 = [h-Tp+1 0 , h-Tp+2 0 , ..., h00 ] ∈ RTp×D representing the temporal feature over Tp timestamps, and ¯Hi = [¯h-Tp+1 i , ¯h-Tp+2 i , ..., ¯h0i ] ∈ RTp×D denoting the temporal feature of ith neighboring agents and D denotes the number of hidden dimensions.

2) Spatial Encoder: Apparently, naive use of one temporal feature per agent does not capture agent-to-agent spatial relations. Therefore, capturing the spatial relations between agents in the same scene is highly necessary. Given the success of attention mechanisms in sequence-based prediction [Hu, 2020], we adopt a multi-head attention mechanism to obtain spatial relations between agents, which can be represented as follows:

where fsp is the spatial attention.

In a detailed elaboration, Q = [q-Tp+1 , q-Tp+2 , ..., q0 ], K = [k-Tp+1 , k-Tp+2 , ..., k0 ], V = [v-Tp+1 , v-Tp+2 , ..., v0 ] respectively denote linear projected vectors query, key, value and Wq,Wk,Wv indicate three learnable parameter matrices. We apply normalization Π to query and key to represent the importance of agents influencing each other. Formally,

where ω is the attention score representing the similarity between Q and K, (·) denotes the operation of matrix multiplication. Subsequently, we leverage attention scores to discern the significant connections among agents. Mathematically,

where υ is the output of the single-head attention. Compared to single-head, multi-head attention can comprehensively capture local and global spatial relations. Therefore, we apply the multi-head attention mechanism to obtain Υ = [υ1, υ2, ..., υn], carrying spatial relations. We introduce an innovative gating mechanism Hg to control the importance of different heads and selectively amplify or suppress specific heads. This mechanism acts as a gatekeeper for the Υ by adjusting the activation level consisting of two linear layers as:

where σ denotes the activation function sigmoid and ⊙ denotes the multiplication of the corresponding elements of the matrix, κ denotes the linear layer. Note that the output of the spatial encoder we simplify as follows:

3) ST Fusion: Following the spatial encoder, we introduce a ST Fusion fST to deeply capture spatial-interaction from S produced by the preceding module. Formally,

where ¯Q = [¯ qt1 , ¯ qt2 , ..., ¯ qtM ], ¯K = [¯ kt1 , ¯ kt2 , ..., ¯ ktM ], ¯ V = [¯ vt1 , ¯ vt2 , ..., ¯ vtM ] respectively denote linear projected vectors query, key, value at the tth timestamp, and ¯Wq, ¯Wk, ¯Wv indicate three learnable parameter matrices. In analogy with the spatial interaction module, we also use normalization and gating mechanisms to obtain long-term temporal features simplified as follows:

4.4 Decoder

This study defines the trajectory prediction task as a conditional probabilistic prediction problem. Specifically, the decoder is designed to predict the future trajectory for the target agent based on different lateral and longitudinal maneuver classes:

where P( ˆ Y ) is the conditional probabilistic distribution for the predicted trajectory. In detail, we use the LSTM decoder to implement the final multi-modal trajectory prediction.

where ˆ yt is the predicted 2D spatial coordinate at future tth timestamp, Wdecoder denotes the parameter matrix to be learned in the LSTM. Based on the comprehensive feature vectors, we can only apply the simplest LSTM decoder to predict the accurate trajectory easily with fewer parameters.

5 Experiment

To evaluate the performance of our model, we perform expensive experiments on real-world datasets. This study uses a consistent segmentation framework for all three datasets. Each sample is divided into 8-second segments, with the first 16 timestamps (3 seconds) serving as historical data and the following 25 timestamps (5 seconds) for evaluation.

5.1 Datasets

Next Generation Simulation (NGSIM): This dataset [Deo and Trivedi, 2018b] consists of vehicle trajectory datasets from US-101 and I-80, containing approximately 45 minutes of vehicle trajectory data at 10 Hz. It is critical for the analysis of vehicle behavior in a variety of traffic scenarios and assists in the development of reliable AD models.

Highway Drone (HighD): HighD [Krajewski et al., 2018] is a dataset of vehicle trajectories collected from six locations on German highways. It includes 110,000 vehicles, including cars and trucks, and a total distance traveled of 45,000 km. This dataset provides detailed information about each vehicle, including type, size, and maneuvers, making it invaluable for advanced vehicle trajectory analysis and AD research.

Macau Connected Autonomous Driving (MoCAD): It [Liao et al., 2024b] was collected from the first Level 5 autonomous bus in Macau, which has undergone extensive testing and data collection since its deployment in 2020. The data collection period spans over 300 hours and covers various scenarios, including a 5-kilometer campus road dataset, a 25-kilometer dataset covering city and urban roads, and complex open traffic environments captured under different weather conditions, time periods, and traffic densities.

5.2 Training and Implement Details

We adopt a two-stage training approach to train our model. In the first stage, our model is trained to predict a future trajectory with the Mean Squared Error (MSE) [Pan, 2020] loss function as follows:

where (ˆ ytx , ˆ yty ) is the 2D spatial coordinate predicted and (ytx , yty ) denotes the corresponding ground-truth coordinate.

It helps our model to learn the exact position information that approximates the true trajectory. When the model converges to a point using the MSE loss function, we use a Negative Log-Likelihood (NLL) [Kim et al., 2020] loss function. This transition facilitates a more comprehensive exploration of uncertainty within the trajectory prediction.

where∆tx ,∆ty represents (ytx -ˆ ytx ), (yty -ˆ yty ), and σtx , σty denote the standard deviation of x-coordinate and y-coordinate at the tth timestamp. ρtxy is the correlation coefficient between the x and y coordinates at tth timestamp. P t = ρTx ρty√1- (ρtxy )2 is the standard deviation of the probability density function in x and y coordinate. α is an empirical constant value to simplify the calculation.

5.3 Comparison to State-of-the-arts

Our model's performance is evaluated against more than 15 state-of-the-art (SOTA) methods on each referenced dataset. The experimental results, presented in Table 1, show that our model provides significant improvements in trajectory prediction over the prevailing SOTA baselines. Using Root Mean Square Error (RMSE) as the evaluation metric, our model consistently outperforms most baselines, achieving improvements of 29% and 22% over WSiP and STDAN, respectively, over a 5-second horizon. On the HighD dataset, our model consistently outperforms current SOTA baselines, with average improvements ranging from 43%-70% for short-term forecasts (1-3 seconds) and 62%-78% for long-term forecasts (4-5 seconds). These improvements highlight the importance of integrating spatio-temporal and confidence features. It is noteworthy that the prediction error of the HighD dataset is significantly lower than that of the NGSIM dataset for all algorithms, probably due to the fact that the HighD dataset provides more accurate trajectory data, including detailed information on location and speed. In addition, the HighD dataset contains approximately twelve times more samples than the NGSIM dataset. In addition, on the MoCAD dataset, our model excels on busy urban roads, outperforming SOTA baselines by at least 37% for short-term predictions and reducing long-term prediction errors by at least 0.58 metres. These improvements highlight the importance of incorporating characterised diffusion and spatio-temporal interaction networks. In conclusion, our results confirm the effectiveness and efficiency of our model in predicting AV trajectories.

5.4 Ablation Study

Table ?? analyzes five critical components: characterized diffusion, temporal initializer, spatial interaction, temporal deepening, and confidence feature fusion modules. It shows that we test six models, labeled Model A through Model F. Evaluations against the NGSIM datasets reveal that the stripped-down versions (Models A-E) consistently underperform compared to Model F, which includes all components. Importantly, the integration of the characterized diffusion and confidence feature fusion modules significantly enhances performance, underscoring their vital role in improving prediction accuracy. The inclusion of the confidence feature fusion module, which merges spatial-temporal confidence features with maneuver states, enables more precise predictions of the target agent's trajectory, particularly through the incorporation of characterized diffusion.

5.5 Qualitative Results

To qualitatively gain insight, we visualize the prediction results to assess the effectiveness of our proposed model. Fig. 4. displays the qualitative results of our model tested on the NGSIM dataset. The prediction favorably covers the ground truth trajectory despite different numbers of neighboring agents in different scenes. These results demonstrate the impressive capabilities of the CDSTraj model in feature representation.

6 Conclusion

The intricate dynamics and uncertainties inherent in multiagent interactions and scene contexts present a significant challenge in the development of the fully AVs. To address this challenge, we introduce a novel generative model that employs a dual architecture integrating a characterized diffusion mechanism and a spatial-temporal interaction network. Empirical evaluations on the NGSIM, HighD, and MoCAD datasets demonstrate that our model, CDSTraj, consistently outperforms existing SOTA baselines in prediction accuracy over both short and long terms. Future research will explore the application of this model to pedestrian trajectory predictions and investigate the integration of spatial and temporal information. These endeavours may yield significant advancements in AD technologies.

Acknowledgements

This research is supported by the Science and Technology Development Fund of Macau SAR (File no. 0021/2022/ITP, 0081/2022/A2, 001/2024/SKL), ShenzhenHong Kong-Macau Science and Technology Program Category C (SGDX20230821095159012), and University of Macau (SRG2023-00037-IOTSC). Haicheng Liao and Xuelin Li contributed equally to this work. Please ask Dr. Zhenning Li (zhenningli@um.edu.mo) for correspondence.