Beyond Patterns: Harnessing Causal Logic for Autonomous Driving Trajectory Prediction

1 Introduction

Autonomous driving (AD) technology holds promise for revolutionizing transportation by enhancing traffic efficiency and safety [Liao et al., 2025c; Guan et al., 2024]. A critical component of AD systems is trajectory prediction, which involves forecasting the future positions of vehicles and other traffic participants. Accurate trajectory prediction enables these systems to anticipate and respond to dynamic changes in the environment, providing essential inputs for decision-making, route planning, and collision avoidance [Liao et al., 2024b]. Despite significant advancements in trajectory prediction for autonomous vehicles (AVs), modern data-driven models primarily focus on identifying statistical patterns within large datasets [Chen et al., 2021]. While this approach has been effective in capturing correlations, it often overlooks the deeper causal relationships that underpin driving behaviors. These models [Liao et al., 2024c] tend to treat driving environments as mere data-generating processes without recognizing the causative factors that influence vehicle dynamics.

As illustrated in Figure 1, the model may learn that vehicles tend to accelerate in particular scenarios, such as scenario (a). This insight allows the model to anticipate acceleration maneuvers at analogous points in the future, as demonstrated in scenario (b). However, such predictions are based solely on observed patterns and lack an understanding of the underlying causes, such as the presence of crosswalks or sharp curves in the road. This superficial reliance on data correlations results in models that are tethered to the training data, without the capability to infer causal links, such as the impact of pedestrian movement (temporal agent data) or changes in road layout (spatial map data). These limitations become particularly problematic when the model encounters scenarios that are not well-represented in the training data. Without an understanding of causal mechanisms, the model struggles to generalize to new situations, potentially leading to inaccurate predictions and safety risks [Chen et al., 2021;

Liao et al., 2025b; Ge et al., 2023]. This highlights the need for trajectory prediction methods that incorporate causal reasoning to better interpret and predict complex traffic behaviors. This observation highlights the need for a paradigm shift toward causal inference in trajectory prediction, emphasizing the understanding of cause-and-effect relationships within data. Such an approach is crucial for AD systems for several reasons. First, causal inference enhances the robustness of the model [Bagi et al., 2023]. AVs operate in dynamic scenes and may encounter sensor noise, missing data, or abnormal driving behaviors. By identifying underlying causal mechanisms, causal inference can effectively filter out these noises or random perturbations, maintaining the stability and accuracy of predictions. Second, causal inference improves generalization to unseen scenarios, which is essential for reliable performance in diverse environments [Ge et al., 2023; Chen et al., 2021]. Finally, causal models offer increased interpretability by providing insights into why specific predictions are made, thereby enhancing transparency and trust.

To address these needs, we propose a novel causal inference framework for trajectory prediction in AD. Causal inference is a method that goes beyond merely identifying correlations in data to uncover the actual cause-and-effect relationships that drive the behaviors being observed [Pearl, 2009; Kuang et al., 2020]. This is particularly important in dynamic environments like traffic scenarios, where understanding the underlying causes of actions, rather than just patterns, leads to more accurate trajectory predictions for AVs. Our approach constructs a causal graph to explicitly represent the relationships between key variables, such as spatial map data and temporal agent data. As illustrated in Figure 2, we employ a causal inference paradigm, including backdoor adjustment and counterfactual analysis, to isolate the confounding variables in the traffic environment. These paradigms eliminate spurious correlations, uncovering the true causal relationships in the data. Furthermore, we introduce a crossmodal progressive fusion strategy that mimics the gradual reasoning process of human drivers. This strategy incorporates a multi-stage attention mechanism to generate progressive causal queries to improve prediction accuracy.

In summary, our contributions are as follows:

• We propose a novel causal inference paradigm for trajectory prediction in AD, utilizing a causal graph to model cause-and-effect relationships between spatial map data and temporal agent data. This approach en-
• ables the identification of genuine causal links, improving prediction accuracy and robustness.
• We introduce a cross-modal progressive fusion strategy, using a multi-stage attention mechanism to integrate multimodal information. This enhances real-time inference, allowing the model to adapt to dynamic scenes.
• Our extensive experiments across five real-world datasets consistently demonstrate that both our causal inference model and its plug-in module outperform existing methods in diverse traffic scenarios, highlighting their generalization capabilities and reliability.

2 RelatedWork

2.1 Trajectory Prediction for Autonomous Vehicles

Trajectory prediction has undergone a remarkable evolution, transitioning from traditional physics-based models and classical machine learning approaches to sophisticated deep learning architectures. Early methods, such as the Kinematic bicycle model [Wang et al., 2025a] and Kalman filtering [Wang et al., 2023] are often limited in their adaptability to complex scenarios. As a result, these methods are typically restricted to simpler environments and short-term prediction tasks. The field has seen significant progress with the rapid advancement of deep learning technologies. Recent studies have introduced sophisticated models like gated recurrent unit (GRU) [Li et al., 2024; Liao et al., 2024f], long short-term memory (LSTM) [Deo and Trivedi, 2018; Xie et al., 2021], graph attention network (GAT) [Mo et al., 2022; Liao et al., 2024b], generative models [Liao et al., 2024a], and the Transformers [Liu et al., 2024; Chen et al., 2024; Liao et al., 2025a; Liao et al., 2024e], expanding the potential for accurate long-term prediction in complex scenarios. While contemporary trajectory prediction models have demonstrated remarkable performance, their reliance on purely data-driven approaches presents fundamental limitations. These methods excel at identifying statistical regularities in training data but fail to capture the essential causal relationships that dictate agent behaviors in dynamic scenes.

2.2 Causal Inference in Autonomous Driving

Causal inference is a statistical framework designed to estimate the impact of one variable on another by identifying causal effects and mitigating the influence of confounding factors. In this context, two fundamental methodologies have emerged as particularly powerful for causal identification: backdoor adjustment, which enables unbiased effect estimation through proper conditioning on observed covariates, and counterfactual analysis, which examines hypothetical scenarios under different intervention conditions. The former [Pearl, 2009] involves identifying and controlling for confounding variables to reduce bias, while the latter [Kuang et al., 2020] evaluates causal effects by constructing hypothetical scenarios to explore "what if" questions. In the field of trajectory prediction, previous studies employing causal inference have respectively utilized counterfactual analysis [Chen et al., 2021] and backdoor adjustment [Ge

Token Extraction

Multi-mode Prediction

et al., 2023] to alleviate the impact of environmental confounders in traffic scenarios. However, due to an incomplete decomposition of these environmental factors, these methods often fall short in uncovering the underlying causal relationships. To address this limitation, our work advances beyond this limitation through a novel decomposition that separately models spatial and temporal environmental factors, enabling more precise confounding control. By unifying backdoor adjustment with counterfactual reasoning, we establish a more complete causal representation that preserves genuine relationships while eliminating spurious correlations, ultimately achieving superior robustness compared to existing methods.

3 Methodology

3.1 ProblemFormulation

At each time step t, our model utilizes historical observations from t - th to t, denoted as Xt-th:t 0 , along with comprehensive traffic information-spatial map data S, temporal agent data T and a BEV of the traffic scene to forecast the vehicle's trajectory over the next tf time steps, represented as Y = {Xt+1:t+tf 0 }. Specifically, S encompasses lane identifiers and precise coordinates of lane markings, while T includes the historical and predicted states of n surrounding agents, represented as T = {Xt-th:t+tf 1:n , pt:t+tf 1:n }.

3.2 ACausal View on Trajectory Prediction

Causal Graph. This study constructs a causal graph to effectively map the relationships among the historical observations of the target agent X, its future trajectory Y , the temporal agent data T , and the spatial map data S. As shown in Figure 1 (c), X and T influence Y , represented by the edges X → Y and T → Y . Additionally, S impacts both the historical and future trajectories, creating backdoor paths Y ← S → X and Y ← S → T. In causal inference, variables like S and T that influence multiple other variables are termed confounders. The presence of confounders, particularly S, can bias the model's ability to learn the distributional properties of X and T , skewing it towards more common scenarios. For example, if straight-line trajectories dominate training data, the model may neglect specific traffic features, leading to incorrect predictions for turns. The confounding effect of T further hampers the model's ability to assess the impacts of T and X on Y , causing overfitting in interactions.

Casual Inference. Given that spatial map data S is relatively fixed and observable, backdoor adjustment mitigates bias by categorizing confounding factors into distinct groups and making predictions for each group. Therefore, it is necessary to systematically enumerate S and eliminate interference in the pathways X ← S → Y and T ← S → Y . This enables the model to capture genuine causal relationships.

where gθ is the trajectory model, si denotes the enumerated spatial map data, and P(si) is set to 1/n based on the principle of maximum entropy, with n being the assumed number of environmental categories [Ge et al., 2023]. Although backdoor adjustment eliminates the confounding effects of spatial map data S, it does not address the complexities introduced by temporal agent data T. Causal theory suggests that counterfactual analysis can mitigate these effects by intervening in the historical trajectory data X to isolate the impact of T. This involves using the do(·) operator to replace the factual trajectory X with a hypothetical one, fixing X to a specific value and isolating it from its antecedents. Formally,

where Xc represents counterfactual values. We combine backdoor adjustment and counterfactual analysis together in a comprehensive manner to derive: Y = ˜ Y - ˜ Yc.

3.3 ProposedModel

Figure 3 showcases the overall pipeline of our model-token extraction, and multi-mode prediction-into a cohesive system. This subsection outlines the structure of our model.

Token Extraction

From a causal inference perspective, predicting future trajectories requires isolating the factors influencing the target object's decisions and movements. Due to the heterogeneity of traffic information, we employ specialized encoders-Spatial Encoder, Temporal Encoder, and BEV Encoder-to extract comprehensive features from diverse data sources.

Spatial Encoder. In this encoder, the spatial map data S ∈ RNm×n×Wm is tokenized into a sequence using the GRU layers. Then, GAT is applied to extract the spatial tokens Sh ∈ RNm×D for the following backdoor adjustment process. Here, Nm represents the number of map polylines, n signifies the number of points within each polyline. wm indicates the number of attributes for each point, such as location and road type, D denotes the dimension after encoding.

Temporal Encoder. The GRU layers are used in this encoder to produce target Xh ∈ RD and surrounding T h ∈ RNa×D tokens for the raw observations X ∈ RT×Wa and temporal agent data T ∈ RNa×T×Wa . Here, T represents the number of history frames, Wa is the number of state information, Na signifies the number of surrounding agents.

BEV Encoder. Recent studies have shown that explicitly modelling the heterogeneity of driving scenes by incorporating diverse modalities can substantially improve a model's ability to interpret complex interactions, thereby enhancing its accuracy. Unlike the polyline representation, the BEV format uses rasterized images to represent spatial data. Our proposed encoder hierarchically extracts frame-wise pyramid features and fuses them into BEV tokens to facilitate the interaction between these two modalities. BEV is reorganized into a three-dimensional tensor with a hierarchical structure, comprising three distinct semantic layers: Agent, Map and Raster. To mitigate the risk of occlusion that arises when directly overlaying semantic layers onto images, we employ a series of convolutional kernels with varying sizes to extract key information from these semantic layers. This allows for a comprehensive capture of the interactions among the heterogeneous elements of the scene. Finally, We apply an MLP to refine the feature, resulting in the generation of Bh .

Diffusion-based Backdoor Adjustment. This component aims to mitigate the impact of confounding factors in spatial maps by generating a diverse set of potential traffic scenarios, a process known as backdoor adjustment. Since backdoor adjustment requires the stratification of S, we propose leveraging spatial maps generated by a diffusion model to automatically approximate the stratified structure of S. Through this approach, we effectively sever the directed edges S → X and S → T, thereby eliminating spurious correlations and ensuring more accurate causal relationships [Ge et al., 2023]. Specifically, we utilize the output Sh from the spatial encoder as the input Sh 0 for backdoor adjustment and define a diffusion sequence (Sh 0 , Sh 1 , . . . , S h k ), with the forward process gradually introduces Gaussian noise to systematically disrupt the deterministic structure of the road network, eventually rendering it entirely stochastic. Formally,

Conversely, the backward process employs a transformerbased architecture to iteratively reconstruct the typical road structure from this stochastic state, ultimately generating multiple instances. Mathematically,

By repeating this generative process n times, we obtain a backdoor spatial token set ¯ S = {Sh,1 , Sh,2 , . . . , S h,n }.

Targeted Multi-View Attention Module. To enhance the understanding of environmental influences on the target future trajectory and facilitate causal integration, we employ a multi-view attention mechanism across spatial, BEV, and temporal contexts. Formally,

The extracted tokens are processed through an aggregated MLP, producing the contextual target token Xi attn .

Multi-mode Predictions

Cross-modal Progressive Fusion. This component is designed to integrate cross-modal tokens using progressive fusion. We initialize with an anchor-free query Q0 to forecast the target agent's future trajectories. In the following progressive stages, this initial query operates as an adaptive anchor point. Specifically, an attention mechanism is employed to progressively refine our adaptive anchor, identifying the agent's intended position at the current moment, which then becomes the initial query for the subsequent update. After Trec iterations, the definitive progressive inference anchor Qi is developed. Notably, counterfactual analysis entails envisioning an alternate scenario where actual data is replaced with hypothetical data. To reduce the impact of confounding variables T in the trajectory prediction system, we substitute the historical trajectories with zero vectors and rerun the progressive fusion module to generate counterfactual anchor Qic . Dual-scale Information Fusion. This component employs a CNN-based framework to extract and integrate the target agent and its surrounding agent information, resulting in the feature representation Gi . For counterfactual analysis, the original target values are substituted with counterfactual values, and the module is applied synchronously to obtain the corresponding counterfactual representation Gic .

Causal Decoder. In this decoder, the factual tokens Qi and Gi , along with the corresponding counterfactual tokens Qic and Gic are integrated using MLPs to produce composite token ˜ Y and ˜ Yc. This composite token is then processed by a GRU-based decoder, which generates the predicted probability of maneuver ykman with the corresponding maneuver-based trajectory probability mixture model Y k .

3.4 Learning Strategy

The training process of our model is divided into two steps. The first step involves training the backdoor adjustment module. In the second step, with the parameters of the backdoor adjustment module fixed, we train the entire model.

Step-1: Diffusion-based Backdoor Adjustment Training. In the initial step, we train the backdoor adjustment module using the following loss:

where ϵ is sampled from a normal distribution N(0, I) , and the variable Sh j is defined as: Sh j = √ ¯ αjSh 0 + √1- ¯ αj ϵ. Step-2: Full Model Training. In the second step, we train the entire model by combining two loss functions: the intention loss (Lint) and the trajectory loss (Ltraj). Intention loss captures the driver's operational intent, while trajectory loss ensures the accuracy of the trajectory prediction. Intention loss is calculated as follows:

Here, ˆ ykman and ykman are the true probability and the predicted probability of maneuver k, respectively. Moreover, the trajectory loss Ltraj adapts to different datasets and metrics:

where L0 varies by dataset: minADEk for nuScenes, WSADE for ApolloScape, and RMSE for NGSIM, MoCAD, and HighD. In addition, LNLL represents the negative loglikelihood loss, while the coefficients λ0 and λ1 are learnable weights for L0 and LNLL, respectively. Hence, the overall loss L for the model is the sum of intention loss Lint and trajectory loss Ltraj, i.e. L = λintLint + λtrajLtraj, where λint and λtraj are the weights for intention loss and trajectory loss.

4 Experiments

4.1 Experiment Setup

The robustness and accuracy of our model are evaluated on five prominent real-world datasets, including nuScenes [Caesar et al., 2020], ApolloScape [Huang et al., 2018], MoCAD

Ablated variants

A B C D E

✗ ✔ ✔ ✔ ✔

✔ ✔ ✔ ✗ ✔

ApolloScape

-

2.1900

2.0913

2.2934

1.9174

[Liao et al., 2024c], NGSIM [Deo and Trivedi, 2018] and HighD [Krajewski et al., 2018]. In accordance with the ApolloScape and nuScenes Forecasting Challenge, as well as the seminal work [Liao et al., 2024c; Liao et al., 2025b], five metrics are employed to assess the efficacy of our model: minADE, FDE, WSADE, WSFDE, and RMSE.

4.2 Evaluation Results

Comparison to SOTA Baselines. As shown in Tables 1, 2, and 3, our model outperforms the current SOTA baselines across five real-world datasets. On the ApolloScape dataset, our model surpasses MSTG [Tang et al., 2023] in WSADE and WSFDE by 1.84% and 8.20%, respectively. In the nuScenes benchmark, it achieves improvements of at least 7.91% in minADE1 and 4.00% in FDE, particularly in intersections. For NGSIM and HighD datasets, it improves the long-term prediction (3s-5s) by 13.82% and 6.45% in RMSE, respectively. For the MoCAD dataset, which represents urban roads with right-hand drive, we see an improvement of up to 6.60% in long-term predictions. These results validate the prediction accurancy of our model across challenging scenes, including highways, roundabouts, and congested urban roads. Comparison of Robustness. We evaluate the robustness of our causal inference model using two methodologies: introducing controlled noise [Bagi et al., 2023] and randomly removing frames. To simulate observation noise, we augment the nuScenes dataset by defining a noise variable σt as a function of curvature γt, formulated as follows:

where ˙xt = xt+1 - xt and ˙yt = yt+1 - yt represent agent velocities, and α controls the noise magnitude. We set α ∈ {1, 2} during training and α ∈ {8, 16} during testing to simulate severe perturbations. To assess robustness against missing data, 20% or 40% of test frames are randomly removed and replaced with zeroes. Table 4 shows that under both noise and frame removal conditions, our model still outperforms SOTA baselines. These results demonstrate its ability to maintain high prediction accuracy by effectively learning and leveraging causal relationships.

Comparison of Domain Generalization. We partition the nuScenes dataset into four subsets and verify the statistical distinction in driving behaviors using a Kolmogorov-Smirnov test [Berger and Zhou, 2014], which yields p-values below the standard threshold. This confirm significant behavioral differences across regions. To assess the generalization capability of our causal inference model, we perform cross-training and testing, where the model is trained on one subset and tested on others. As shown in Figure 4, our model achieves robust performance in unseen regions, demonstrating its effectiveness in enhancing domain generalization across diverse scenes.

4.3 Ablation Study

We present detailed ablation variants in Table 5 and Table 6. Method E represents the complete model, which achieves the SOTA performance across all metrics, demonstrating the synergistic effect of its components. Model D, which eliminates all causal analysis modules, including Backdoor Adjustment using diffusion, counterfactual analysis in progressive fusion, and dual-scale information fusion, and replaces the Differential Causal Decoder with a simple GRU, exhibits the poorest performance across most metrics, highlighting the significance of the causal inference paradigm in accurately learning casual relationship. Model B, which substitutes the progressive anchor fusion module with a single-stage anchor, also shows degraded performance, indicating the importance of progressive fusion in driving scenarios for precise trajectory prediction. The results of Method A and Method C show varying degrees of performance degradation, further confirming the necessity of each component within the model.

4.4 Validation for Plug-and-Play Causal Inference

We incorporate the causal paradigm into the PGP model by replacing its GRU-based encoder with a backdoor adaptation mechanism and by integrating counterfactual reasoning into the decoder. On the nuScenes dataset, as shown in Figure 5, our integrated model effectively captures causal relationships in complex lane-forking scenarios, where both the original and enhanced PGP versions fail to predict turning maneuvers. These results establish the role of causal inference in advancing behavior prediction performance.

5 Conclusion

This study advances trajectory prediction for AVs by integrating causal inference and progressive fusion strategies. Addressing the limitations of traditional data-driven methods that often neglect causal relationships, our model improves prediction accuracy and reliability through the use of causal graphs, backdoor adjustment, and counterfactual analysis, enabling it to distinguish true causal effects from correlations in complex traffic scenarios. Evaluation results on five realworld datasets show significant performance improvements across multiple metrics, marking a significant step forward in the field of trajectory prediction for fully AD systems.

Acknowledgements

This work was supported by the Shenzhen-Hong KongMacau Science and Technology Program Category C [SGDX20230821095159012], University of Macau [SRG2023-00037-IOTSC, MYRG-GRG2024-00284IOTSC], the Science and Technology Development Fund of Macau [0021/2022/ITP, 0122/2024/RIB2 and 001/2024/SKL], the State Key Lab of Intelligent Transportation System [2024-B001], and the Jiangsu Provincial Science and Technology Program [BZ2024055]. Bonan Wang and Haicheng Liao contributed equally to this work. Please ask Dr. Zhenning Li (zhenningli@um.edu.mo) for correspondence.