Physics-Informed Trajectory Prediction for Autonomous Driving under Missing Observation

1 Introduction

In the field of AVs, the effective prediction of surrounding vehicles' trajectories in dynamic environments encounters two often overlooked yet interrelated challenges. The first challenge arises from real-world observation constraints, such as sensor limitations and environmental factors including obstructions, adverse weather, or traffic congestion. These constraints frequently result in missing observations [Liao et al., 2024b], a scenario that poses a significant hurdle for traditional deep learning models. Despite their effectiveness with ideal datasets, these models typically struggle to adapt to the unpredictable and variable conditions characteristic of realworld driving [Bhattacharyya et al., 2023], an issue that has not been sufficiently addressed in previous studies.

Second, equally overlooked challenge involves ensuring that these models adhere to physical laws in trajectory prediction. Many current models inadequately consider the kinematic constraints of vehicular motion, leading to predictions that are statistically accurate but kinematic infeasible. This limitation can potentially compromise the safety and reliability of AVs' motion plans [Huang et al., 2022; Shen et al., 2023], yet it has not been given due attention in the existing literature.

To bridge these gaps, our study introduces a novel twostage trajectory prediction approach that synergizes datadriven and physics-based methodologies with a Physics Enhancement Stage, as shown in Figure 1, and a Trajectory Prediction Stage. This approach combines the robustness of deep learning with physics-informed principles, ensuring realistic and robust trajectory predictions even in scenarios with missing observations. Demonstrating its effectiveness, our approach consistently outperforms state-of-the-art (SOTA)

models in comprehensive evaluations, even with up to 75% missing observations. The key contributions of our study can be summarized as follows:

• We introduce a pioneering Physics Enhancement Stage which includes a Wavelet Reconstruction Network and a Kinematic Bicycle Model. This integration fosters a significant advancement in the trajectory prediction domain, markedly enhancing the approach's robustness against missing observations and augmenting the kinematic feasibility of the predicted trajectory.
• We significantly enrich trajectory prediction research by introducing MoCAD-missing, a dataset curated from a fully (Level 5) AV in real-world traffic scenarios, and by enhancing the established NGSIM and HighD datasets with scenarios featuring missing observations. These contributions expand the research scope in this domain, providing robust testing grounds for advanced models.
• Our innovative Wave Fusion Encoder, inspired by quantum mechanics, revolutionizes interaction modeling. By conceptualizing vehicle features as waveforms, this module facilitates a novel approach to model vehicle interactions using the principle of wave superposition.
• We conduct comprehensive experiments on MoCADmissing, NGSIM-missing, and HighD-missing datasets. Results show that i) Our approach can outperform other SOTA models across various scenarios even in the face of 75% missing observations, and ii) The wavelet reconstruction network can integrate seamlessly with other SOTA models to enhance the robustness against missing observations.

2 RelatedWork

Initially, the application of physics-based models played a significant role in the realm of trajectory prediction. Notably, various classical physics models, such as the Constant Velocity model [Miller and Huang, 2002] and the Bicycle model [Br¨annstr¨om et al., 2010], garnered substantial attention due to their computational efficiency. Additionally, Kalman filtering is used in tracking algorithms to predict the vehicle's position [Kaempchen et al., 2004]. Moreover, the application of Monte Carlo simulations, as exemplified in the works of Danielsson et al. [Danielsson et al., 2007] offered a means of simulating future vehicle states.

However, while these physics-based models exhibit commendable performance in physical feasibility, they falter in long-term prediction accuracy, particularly in complex and dynamic traffic environments. This limitation has catalyzed the shift towards learning-based methodologies, aiming to enhance prediction accuracy. The introduction of Recurrent Neural Network (RNN) architectures, notably by the work [Zyner et al., 2019; Zyner et al., 2017; Liao et al., 2024d; Chen et al., 2022b], marked a significant advancement in this field. These models [Cong et al., 2023; Zuo et al., 2023; Liao et al., 2024b], harnessed historical vehicle observations to enhance prediction accuracy. Subsequent enhancements in the trajectory prediction field included the application of Convolutional Neural Networks (CNNs), which significantly revolutionized the processing of contextual features. This was exemplified in the use of social tensors [Deo and Trivedi, 2018] and rasterized maps [Gilles et al., 2021].

The learning-based methods mentioned above are purely data-driven methods that lack interpretability and will lead to some impractical trajectories with issues like excessive acceleration. Therefore, researchers have turned their attention to integrating physics knowledge into trajectory prediction. Drawing inspiration from the wave superposition principle in quantum mechanics [Arndt et al., 1999], Wang et al. introduced Wave-MLP [Tang et al., 2022] into the trajectory prediction model to replace the multi-head attention mechanism to obtain information about interactions between vehicles [Wang et al., 2023]. However, their approach was limited to capturing only spatial interaction at the current moment, neglecting the temporal interaction. In [Cui et al., 2020], Cui et al. used a neural network to learn the parameters required for a biaxial bicycle model, and then the biaxial bicycle model was used as the output layer, and the output was consistent with the vehicle kinematics to learn the trajectory. Zhang et al. [Zhang et al., 2023] used wavelet transform theory to predict the trajectory of the aircraft.

3 ProblemFormulation

The main objective of this study is to predict the trajectory of the target vehicle, referring to the selected vehicle that lies within the sensing range of the AV. At each time tc, the AV aims to predict the target vehicle's trajectory Y0 for future tf time based on the historical observations X. Our proposed method consists of two stages: trajectory repair and trajectory prediction. It uses historical observations from time tc - th to tc, including observations for the target vehicle X0 and all its observed surrounding vehicles X1:n, as input. To be more consistent with real-world, the input has missing observations. Formally, our approach could be formulated as:

where the input observations X = {X0, X1:n} consist of the position coordinates, speed, acceleration, vehicle type, and lane number of the target vehicle and all its surrounding vehicles. Correspondingly, F denotes the trajectory prediction stage, while K denotes the physics enhancement stage.

Additionally, we introduce a multi-modal decoder with a hierarchical hierarchical multi-modal prediction framework to model the uncertainty in the driving behavior of surrounding human drivers. At first, the uncertainty of human drivers is reflected in the variety of potential maneuvers. In the real-world, the longitudinal maneuvers are divided into acceleration, deceleration and constant speed, and the lateral maneuvers are divided into left lane change and right lane change, and keep constant. In this framework, the possibility P (Mi|X) of each maneuver Mi is first estimated based on the historical observations of the target vehicle and surrounding vehicles. And then, the human driver's uncertainty is reflected in the trajectory for each potential maneuver, so the trajectory is modeled as a Bivariate Gaussian distribution. Specifically, at time tc+f the prediction is characterized with means µtc+f x and µtc+f y , variances σtc+f x 2 and σtc+f y 2 , and training of the wavelet coefficient extractor and the detailed training strategy is shown in Figure 3.

Specifically, we use Level-3 DWT and its corresponding IDWT in this network. For clarity and consistency, historical longitudinal coordinates observations xi serving as illustrative examples for subsequent analysis, and the complete longitudinal coordinates observations are denoted as ˆxi. Level-3 DWT decomposes the historical longitudinal coordinates observations into detail ci(j, k) and approximation coefficients di(j, k) at different level j and translation k. As shown in Figure 4, this decomposition is achieved through an iterative process involving high-pass filter h and low-pass filter g, followed by downsampling.

After obtaining the ci(j, k) and di(j, k) of the ˆxi, the IDWT is employed to reconstruct the ˆxi:

the correlation coefficient ρtc+f .

4 Methodology

As shown in Figure 2, our approach is divided into two main stages, including the physics enhancement stage and the trajectory prediction stage. The physics enhancement stage is designed to enhance missing historical observations and provide a robust foundation. Moreover, the trajectory prediction stage makes multimodal predictions based on these observations.

4.1 Stage 1: Physics Enhancement

In this stage, we enhance trajectory prediction accuracy by addressing the challenge of missing observations. We utilize the wavelet reconstruction network to restore the full spectrum of historical observations, thereby providing a robust foundation for predicting future trajectories. Furthermore, we prompt the trajectory prediction model by inputting a kinematic feasible immediate future trajectory, thereby improving the kinematic feasibility of the final output trajectory. To this end, the wavelet reconstruction network is also used to enhance immediate future observations. Then, the kinematic model regularizes the enhanced observations to generate the kinematic feasible trajectory.

Wavelet Reconstruction Network

This network includes a pair of reversible operations: Discrete Wavelet Transform (DWT) decomposes the original observations into wavelet coefficients, and Inverse Discrete Wavelet Transform (IDWT) uses the decomposed wavelet coefficients to reconstruct the original observations. However, in the evaluation stage we only have missing historical observations. In order to reconstruct complete historical observations, IDWT cannot use the wavelet coefficients decomposed by DWT from missing historical observations. To address this, a wavelet coefficient extractor is used to extract the wavelet coefficients corresponding to complete historical observations from missing historical observations. The DWT and complete historical observations are used to supervise the

Theorem 1. Complete preservation of observations in DWT decomposition and IDWT reconstruction.

Theorem 1 is the cornerstone of the wavelet reconstruction network, which ensures that the observations reconstructed by IDWT contain the original observations. Following the procedure in Figure 5, we will give a proof for this theorem.

Proof. Apply the Z-transformation to transform the timedomain observations of ˆxi(t) into its z-domain representation X(z) = ∑ ˆxi(t)z-t . Following [Goel, 2014], Level-1 DWT decomposes X(z) to obtain the approximate X1,L(z) and detailed coefficients X1,H(z) in the z-domain:

where G(z) and H(z) are the low-pass filter and high-pass filter in z-domain.

In the IDWT, the interpolation is first used to restore the original frequency and then passes the low-pass reconstruction filter G1(z) and the high-pass reconstruction filter H1(z). By combining X0,L(z) and X0,H(z), the reconstructed observations in z-domain X0(z) could be obtain as:

By analyzing Eq. 6, we can conclude that the relationship between X0(z) and X(z) is determined by the filters. We use the Haar wavelet and its corresponding filters from PyWavelet [Lee et al., 2019], and in the Z-domain:

Substituting the values from Eq. 7 into Eq. 6 we get:

Eq. 8 indicates that the reconstructed observations is a unitdelay version of the original observations, where z-1 represents one unit of delay. In the Z-domain, z-1 represents a unit delay, which does not lead to observations loss but merely indicates a temporal shift of the observations.

The wavelet reconstruction network needs to provide immediate future observations for kinematic bicycle model to generate the kinematic feasible immediate future trajectory. Therefore, in the wavelet reconstruction network, DWT not only needs to decompose complete historical observations but also decompose immediate future observations to supervise the training of wavelet coefficients extractor, enabling this network could enhance immediate future observations.

Kinematic Bicycle Model

A two-axis kinematic bicycle model, centered on the vehicle's center of mass, is employed to generate the trajectory for the immediate future te based on the observations enhanced by wavelet reconstruction network. As shown in Figure 6, critical variables including speed vt , acceleration at , steering angle γ t , coordinates (xt , yt ), heading ψt , as well as the distances from the center of mass to the front lf and rear lr wheels at time t.

The vehicle's state at the next time t+1 is determined using state transition equations, derived from the kinematic bicycle model detailed by Cui et al. [Cui et al., 2020]:

where ∆t denotes the time difference between consecutive times, while the derivatives [ ˙xt , ˙yt , ˙ψt , ˙vt ] at time t are calculated as per the following expressions:

where βt = tan-1 ( lr lf+lr tanγ t) signifies the angle between the vehicle's velocity vector and its heading direction.

4.2 Stage 2: Trajectory Prediction

Wave Fusion Encoder

Our module employs a conceptual framework from quantum mechanics wherein a vehicle's features are conceptualized as a wave. In this framework, the vehicle features are encoded within the amplitude of the wave, while the vehicle's temporal and spatial dynamics are encapsulated in the phase. Mathematically, this conceptual framework can be defined as:

where ˜Xj is the wave representation of vehicle j. |Xj| is the amplitude learning from the vehicle features. The phase is denoted as θj . And ⊙ is the element-wise multiplication, i is the imaginary unit.

With this conceptual framework, we can use the superposition of waves to model the effects of interactions between vehicles. Leveraging the superposition principle of wave, the interaction between two vehicles, Xj and Xk, can be rigorously expressed by the defined mathematical formula:

where ˜Xr is the interaction features. By employing Euler expansion, interaction features could be expressed as:

For convenience of representation, the final output oj is projected into the real number domain:

where W a and W i are learnable weights.

Risk Aware Module

The Risk Aware Module is architected with two core components: a risk encoder and a probsparse attention module. The former is tasked with the translation of the risk vector into the target vector space, thereby yielding the real-time risk features pertinent to the target vehicle. The risk vector, foundational to this module, is conceptualized as per Kloeden et al. [Kloeden et al., 2001] and is quantified by the equation:

Algorithm 1 Risk Aware Algorithm

Input: Q ∈ RLR×d ,K ∈ RLW×d , V ∈ RLW×d , number of surrounding vehicles n Output: Feature map F

• 1: for Head in P Heads do
• 2: U = LR lnLW.
• 3: Randomly select U query-key pairs [QK⊤]U .
• 4: Compute the importance ¯D = max([QK⊤]U ) - mean([QK⊤]U ) for each query.
• 5: Obtain the dominant score matrix F ∈ Rn×d based on importance.
• 6: Fi = Softmax (F/√d) · V.
• 7: end for
• 8: F = Concat[F1, F2, . . . , FP ].
• 9: return F

where Ir is the injury crash rate, v is the speed of the target vehicle, and ∆v is the difference between the target vehicle speed and average traffic speed.

Next, the probsparse attention module is employed in this module to integrate real-time risk considerations into the dynamics between the target vehicle and its surrounding vehicles. It operates under the premise that the target vehicle's interactions are predominantly influenced by surrounding vehicles within a specific proximity. This notion resonates with the findings of Zhou et al. [Zhou et al., 2021], which articulate that only a limited array of salient query-key pairs substantially contribute to the computation of attention. Based on this finding, probsparse attention module is devised to specifically accentuate the interactions among surrounding vehicles. The operation of this module is detailed in Algorithm 1.

5 Experiments

5.1 Experiment Setup

The performance of our approach is evaluated using three well-regarded real-world trajectory prediction datasets: Macao Connected Autonomous Driving (MoCAD) [Liao et al., 2024b], NGSIM [Deo and Trivedi, 2018], and HighD [Krajewski et al., 2018]. These datasets provide longitudinal and lateral coordinates of traffic agents, which are systematically divided into training, validation, and test sets. The trajectory data from these datasets are segmented into 8-second intervals. The first 3 seconds are used as trajectory history for model input (th = 3), and the following 5 seconds are used as ground truth for model prediction (tf = 5). This segmentation enables a comprehensive evaluation of various traffic scenes, including urban streets, campuses, and highways. To measure prediction accuracy, the Root Mean Square Error (RMSE) is used as the evaluation metric, where lower RMSE values indicate higher prediction accuracy. Additionally, the immediate future te for the kinematic bicycle model in the physics enhancement module is set to 1 second. The model is trained on a single Nvidia A40 48GB GPU to convergence, using a learning rate of 0.0005 and a batch size of 64. We employ Mean Square Error (MSE) and Negative Log Likelihood (NLL) as loss functions.

Acknowledging the prevalence of varying missing rates in real-world scenarios, we meticulously engineer three specialized versions of each dataset: MoCAD, NGSIM, and HighD. These enhanced iterations, designated as MoCAD-missing, NGSIM-missing, and HighD-missing, are specifically tailored to embody missing rates of 25%, 50%, and 75%, where 25%, 50%, and 75% of observations are randomly missing per sample. This deliberate design ensures that each dataset is represented across a spectrum of missing conditions, providing a robust framework for a thorough and nuanced evaluation. By integrating these variants into our experimental setup, we aim to extensively assess and validate the resilience and adaptability of our approach. Correspondingly, our approach's performance is benchmarked as Ours (25%), Ours (50%), and Ours (75%), reflecting tests on the datasets with missing rates of 25%, 50%, and 75%, respectively. To address the randomness in missing datasets, we mitigate potential biases by averaging 10 experimental performances with different random seeds.

5.2 Experimental Results

Our evaluation, as delineated in Table 1, benchmarks our approach against prevailing SOTA models, which are conventionally evaluated on datasets devoid of missing observations. Leveraging the MoCAD dataset, which encapsulates a diverse range of urban and campus scenarios, our approach demonstrates a consistent superiority over the incumbent SOTA model across all prediction horizons. The performance enhancements are pronounced, ranging from 18.7% to 54.8%, underscoring the robustness of our approach in real-world conditions. On the NGSIM dataset, our approach achieves a substantial 21.3% reduction in RMSE at the pivotal 5-second prediction horizon, when juxtaposed with the STDAN model. A similar trend of improved performance is observed in the HighD dataset, where our approach achieves a remarkable 28.8% reduction in RMSE at the 5-second prediction horizon. These findings show that our approach can effectively handle missing observations in various scenarios.

5.3 Ablation Study

In this ablation study, we systematically examine the necessity of each component within our proposed approach on 50% NGSIM-missing dataset. The comparative analysis involves four ablated variants (detailed in Table 2) of the model juxtaposed against the comprehensive Model E. Specifically, Model A does not perform the physics enhancement stage. Model B omits the integration of the wave fusion encoder. Model C eschews the incorporation of risk features in the risk-aware module. Lastly, Model D operates without the multi-modal decoder. The performance of these ablated variants is detailed in Table 3. Our findings demonstrate that the comprehensive Model E consistently outperforms the ablated variants across various temporal horizons. This empirical evidence highlights the contribution of each module to the model's effectiveness, affirming their integral roles in achieving optimal predictive performance. Model A, in particular, exhibits the lowest performance. In the NGSIM dataset, its prediction error at a 5-second horizon is 64.0% higher than that of Model E. The robustness of the physics enhancement stage in mitigating missing observations is demonstrated by this stark contrast. Additionally, the performances of Models B and D validate the contributions of the wave fusion encoder and the multi-modal decoder in improving accuracy.

5.4 Validation of Wavelet Reconstruction Network

This experiment investigates the plug-and-play capability of the wavelet reconstruction network with established models, exemplified by the STDAN model. We conduct the validation with three evaluation datasets: complete NGSIM dataset as a baseline (STDAN), 50% NGSIM-missing dataset (STDAN (50%)), and reconstructed dataset by wavelet reconstruction network (STDAN (Reconstructed)). Detailed performance is presented in Table 4. The outcomes depict a notable decline in STDAN's performance with 50% NGSIM-missing dataset. Remarkably, the reconstructed dataset elevates the model's performance to near-baseline levels. To understand the comparison results more intuitively, we visualize the performance of the model with three evaluation datasets in Figure 7. This finding highlights the effectiveness of wavelet reconstruction networks in reconstructing observations and their potential to improve robustness against missing observations.

5.5 Qualitative Results

Figure 8 (a) and Figure 8 (b) depict a comparative analysis of trajectory predictions, highlighting the superior accuracy and robustness of our approach (50% NGSIM-missing dataset) over the STDAN model in both simple and complex traffic scenarios. Figure 8 (c) and Figure 8 (d) further contrast the kinematic feasibility of predicted trajectories.

6 Conclusion

In this study, we develop a two-stage physics-informed trajectory prediction approach, grounded in the foundation of physical theories including wavelet transforms, kinematic bicycle model, and quantum mechanics. This approach adeptly addresses the challenges of missing observations and enhances kinematic feasibility while effectively modeling interactions. Our experiment demonstrates the robustness and accuracy of our approach, showing its superior performance in dealing with missing observations. Furthermore, the MoCADmissing dataset facilitates the evaluation of the model's robustness in real-world missing observation conditions.

Acknowledgements

This research is supported by the Science and Technology Development Fund of Macau SAR (File no. 0021/2022/ITP, 0081/2022/A2, 001/2024/SKL), ShenzhenHong Kong-Macau Science and Technology Program Category C (SGDX20230821095159012), and University of Macau (SRG2023-00037-IOTSC). Haicheng Liao and Chengyue Wang contributed equally to this work. Please ask Dr. Zhenning Li (zhenningli@um.edu.mo) for correspondence.