A Cognitive-Driven Trajectory Prediction Model for Autonomous Driving in Mixed Autonomy Environments

1 Introduction

In the evolving landscape of autonomous driving (AD) systems, the challenge of accurately predicting vehicle trajectories is paramount, especially in mixed autonomy environments where autonomous vehicles (AVs) and human-driven vehicles (HVs) dynamically interact. This complex interplay, soon to become an everyday reality on our roads, is marked by its unpredictability, as detailed in studies like [Schwarting et al., 2019] and [Liao et al., 2024b]. Despite the advancements in deep learning for AD, typified in research such as [Sun et al., 2023] and [Geng et al., 2023], a critical gap remains: these models often lack a nuanced understanding of human driving behavior and decision-making processes

[Chen et al., 2022a; Liao et al., 2024d; Tian et al., 2024; Guan et al., 2024]. This limitation becomes particularly evident in real-world scenarios where the adaptability of these models to diverse and unforeseen conditions is crucial.

This backdrop prompts us to ask critical questions about the future trajectory of AD: Is the key to advancing AD not just in accumulating more data or refining algorithms, but in gaining a deeper understanding of the driving environment itself? How can we reshape our models to interpret and respond to the intricate human dynamics that underpin driving? Motivated by these questions, our research embarks on an innovative path. We propose a paradigm shift, extending beyond conventional data-driven approaches to embrace a critical yet often-neglected aspect of driving - the concept of perceived safety. This concept, pivotal in shaping driving behaviors and decisions, is deeply rooted in psychological constructs, as detailed in [Rubagotti et al., 2022]. According to the Theory of Planned Behavior, individual actions in driving are influenced by attitudes (driving behaviors towards others), subjective norms (personal evaluation of safety), and perceived behavioral control (confidence in driving ability)

[Ajzen, 1991]. Further depth is added by neuroscientific research, such as studies by [Saadatnejad et al., 2022] and [Kronemer et al., 2022], which unveil that perceived safety is an intricate blend of both conscious and instinctive responses, involving the amygdala's emotional processing and the prefrontal cortex's rational decision-making. Notably, this understanding of perceived safety is exemplified in diverse driving scenarios. For instance, when encountering a close car ahead, different drivers exhibit markedly varied responses, as depicted in Figure 1. An aggressive driver, possibly influenced by sensation-seeking tendencies [Zuckerman, 1990], might quickly swerve, perceiving lower risk. Conversely, a cautious driver, perhaps more risk-averse [Rabin, 2013], might opt for a complete stop. These behaviors, far from being random, are intricately linked to each driver's psychological profile and past experiences, revealing a significant limitation in current AD systems: their inability to account for these complex, cognitive behavioral patterns. In response, our research integrates the concept of perceived safety into trajectory prediction models for AVs. This integration does more than add a new variable; it injects a human-centric perspective into the heart of these systems. By doing so, we aim to enhance the models' ability to interpret driving behaviors, leading to predictions that are not only technically accurate but also richly informed by contextual and psychological insights. Overall, the contributions of this study include:

1. Perceived Safety-Aware Module Development: 1) Quantitative Safety Assessment (QSA): A foundational component that objectively evaluates a driving scenario's safety, leveraging metrics like Time-To-Collision (TTC) and Risk Tendency Index (RTI), thereby offering a perception-neutral safety assessment. 2) Driver Behavior Profiling (DBP): Layering upon QSA's foundation, DBP infuses the human element, differentiating between, for instance, an aggressive driver's risk tolerance and a cautious driver's reservations. This profile captures driving nuances in real-time without the need for manual labeling and selection of time windows.
2. Social Interaction-Aware Module Development: Acknowledging the collective ballet on roads, we have architected a transformer-based framework, named Leanformer, which encapsulates the intricate inter-vehicular interactions, mirroring contemporary shifts in AD research trajectories.
3. Robustness Against Data Inconsistencies: Our model significantly outperforms the SOTA baseline models when tested on the NGSIM, HighD, and MoCAD datasets, respectively. In a significant stride towards practical applicability, it demonstrates unparalleled resilience in scenarios with incomplete or inconsistent data. This adaptability ensures that our trajectory predictions remain accurate and reliable even when faced with real-world data challenges, a feature often overlooked in conventional models.

2 RelatedWork

Motion Prediction For Autonomous Driving. A wealth of recent research has harnessed deep learning architectures to predict future motions of target agents for AVs. Employing a variety of frameworks, these models encompass sequential networks [Chen et al., 2022a; Messaoud et al., 2021], graph neural networks [Rowe et al., 2023], transformer constructs [Chen et al., 2022a; Yin et al., 2021], and generative models [Li et al., 2023]. The crux of these approaches is discerning temporal and spatial dynamics between traffic agents from historical data, aiming to optimize prediction accuracy. Moreover, prior research [Deo and Trivedi, 2018; Messaoud et al., 2019; Chen et al., 2022b] has tapped into the intricate social interplays among traffic agents, unveiling latent insights and enhancing predictive performance.

Perceived Safety Concept. The concept of perceived safety has received extensive attention in psychology and physical human-robot interaction (pHRI) [Guiochet et al., 2017]. In pHRI, perceived safety finds applications in various autonomous physical systems, including mobile robots [Scheunemann et al., 2020], industrial manipulators [Davis, 2023], humanoid robots [Busch et al., 2019], and autonomous driving [Aledhari et al., 2023], to assess and represent people's perception of the level of danger in interactions with robots and their level of comfort during such interactions [Bartneck et al., 2009]. However, the evaluation of perceived safety in trajectory prediction is often subjective and lacks standard metrics. This work introduces a new quantitative criterion for perceived safety, inspired by human decisionmaking processes, enhancing the model's contextual understanding of behaviors and traffic conditions.

Driving Behavior Understanding. Numerous previous studies [Schwarting et al., 2019; Chandra et al., 2020] have proposed various criteria and metrics to explicitly represent and detect driving behavior, often utilizing classic scales such as Social Value Orientation (SVO) [Murphy et al., 2011], Driving Anger Scale (DAS) [Deffenbacher et al., 1994], Driving Anger Expression (DAE), Driving Style Questionnaire (DSQ) [French et al., 1993], among others. In contrast to these traditional methods, our approach models driving behavior in real-time with adaptive, behavior-aware criteria, eliminating the need for manual labeling and offering enhanced learning flexibility. This method addresses challenges related to shifting behaviors and time window selection, reducing data variability associated with discrete classification.

3 ProblemFormulation

Our objective is to predict the trajectories of target vehicle (denoted with subscript 0) within the perceptual boundaries of an autonomous vehicle (AV), referred to as the ego vehicle, in settings with mixed autonomy. Given the present moment as t, the task is to estimate the probable future trajectory, Y t:t+tf 0 , of the target vehicle over the ensuing tf time intervals. This estimation is based on the historical states of the target vehicle, Xt-th:t 0 , and the states of surrounding vehicles (indicated by subscript 1:n), Xt-th:t 1:n , within a defined duration th. Given the inherent uncertainties in trajectory predictions, our methodology adopts a multimodal prediction framework. This approach discerns various potential maneuvers for the target vehicle and calculates the corresponding probabilities from prior states. As a result, we obtain a spectrum of predictions, each paired with its confidence level.

Central to our model is the use of historical states as inputs. These comprise the 2D position coordinates pt-th:t 0:n , velocity vt-th:t 0:n , and acceleration at-th:t i spanning the duration from t - th to t. Formally defined, the input is:

The resulting output is:

where pt0 = {(pt0,1 ; ct0,1 ), (pt0,2 ; ct0,2 ), · · · , (pt0,M ; ct0,M )} encompasses both the potential trajectory and its associated likelihood (ΣM1 ct0,i = 1), with M enoting the total number of potential trajectories predicted.

4 Trajectory PredictionModel

Figure 2 illustrates the architecture of our model. Rooted in the encoder-decoder paradigm, the model seamlessly incorporates three novel modules: the Perceived Safety-Aware, the Priority-Aware, and the Interaction-Aware modules. Collectively, these modules are designed to emulate the human decision-making process during driving.

4.1 Perceived Safety Aware Module

Quantitative Safety Assessment. In traffic safety domain, three metrics have gained prominence for their comprehensive portrayal of on-road risks: TTC, Time Exposed Timeto-Collision (TET), and Time Integrated Time-to-Collision (TIT) [Minderhoud and Bovy, 2001]. To align these with our study's objectives, we made slight modifications. Using the 2D position coordinates pti , ptj and velocity vt i , vt j for vehicles i and j at time t, these metrics are formulated as:

1) Time-to-Collision (TTC): TTC is a widely accepted measure used to evaluate the time available before two vehicles collide if they continue on their current trajectories. It offers insights into imminent collision risks and serves as an early warning indicator. TTC for the ith vehicle is computed as:

where di,j represents the distance between vehicles i and j, and ˙ di,j is its rate of change:

2) Time Exposed Time-to-Collision (TET): It measures the exposure duration to critical TTC values within th. It is the sum product of a switching variable and a time threshold τsc (set at 0.1s):

with the switching variable given by:

In our study, TTC∗ , delineating safe and critical thresholds, is set at 3.0s.

3) Time Integrated Time-to-Collision (TIT): An adaptation of TET, TIT integrates TTC profile to evaluate safety levels, factoring in the evolution of each vehicle's TET temporally:

Elevated values of TTC, TET, and TIT imply sustained exposure to potential collision risks, underscoring a deterioration in perceived safety.

4) Risk Tendency Index (RTI): To further capture congestion patterns in complex traffic environments, we propose an index between the ith and jth vehicles at time t, denoted as subjective risk perception indicator (SPR), i.e. Rt i , and dynamic risk volatility indicator (DRV), i.e. Rt i,j , respectively:

In this context, the vector Rt i,j with larger values indicates an increased risk of collision, while the vector ˙Rt i,j characterizes the dynamic congestion conditions in complex traffic scenarios. In addition, the definitions of SPR and DRV are defined as follows:

where the DRV ˙Rt i,j represents the gradient to evaluate fluctuations in SPR Rt i,j and can be expressed as follows:

The quantities qt i,j and ˙ qt i,j are calculated based on several critical parameters related to the dynamics of two traffic agents. These parameters include the lateral velocity vtx , longitudinal velocity vty , 2D position coordinate ptx and pty , as well as the lateral atx and longitudinal aty . Mathematically, it can be represented as follows:

where the ∆i,j(·) denotes the difference between quantities of the i-th and j-th vehicles. A larger vector Rt i,j indicates a higher risk of collision, while the vector ˙Rt i,j describes the dynamic congestion conditions in complex traffic scene.

The encoder within the Perceived Safety-Aware module combines the GCNs and the scaled dot-product multi-head self-attention mechanism. To capture the dynamic geometric relationships among traffic agents, we employ a convolutional neural network on a fully connected interaction multigraph. This multigraph operation layer incorporates sequential perceived-safety criteria Q as nodes and adjacency matrix A as graph edges. The GCN layer is given by:

where ˜D is the degree matrix. The multi-head self-attention mechanism then transform the feature matrix Zk+1 i output from GCNs into query, key, and value vectors. The output for the ith agent from this mechanism is denoted as α.

For improved training stability, inspired by ResNet [He et al., 2016], we integrate Gate Linear Units (GLUs) and Layer Normalization to efficiently manage features:

In particular, the GLUs layer can be defined as:

where α represents the safe attention coefficient from the multi-head attention mechanism, W1 and W2 are the learnable weight parameters associated with the GLUs layer, b1 and b2 are the corresponding biases, ⊙ denotes element-wise multiplication, ϕsigmoid is the sigmoid activation function, and ϕLN(·) stands for Layer Normalization. Correspondingly, the output of the encoder within the QSA is denoted as ¯Ot-th:t safe . Driver Behavior Profiling. Unlike traditional methods that classify driver behavior into fixed, predefined categories, we propose a novel dynamic geometric graph (DGG) based realtime profiling approach. This continuum solution addresses the problem of fluctuating behavioral categorization in previous studies. At a given time t, the graph Gt is constructed as Gt = {V t , Et }, where V t = {vt0 , vt1 , ..., vtn } signifies the node set, and each node vt i represents an individual agent. Et = {et0 , et1 , ..., etn } is the set of edges illustrating potential interactions between agents. Specifically, an edge eti connects node vt i to other nodes (agents) potentially influencing it.

Interactions are considered when agents, such as vi and vj , are sufficiently close. Specifically, if their distance d(vt i , vt j ) is within a set threshold r, an interaction is inferred. This relationship is represented as eti = {vt i , vt j | (j ∈ N t i )} where the neighborhood set N t i is defined by:

The adjacency matrix At of Gt is symmetric. Formally,

To further evaluate the behavior of individual agents and reveal important agents and overall connectivity in the traffic graph, centrality measures are used, including degree J t i (D), closeness J t i (C), eigenvector J t i (E), betweenness J t i (B), power J t

i (P), and Katz J t i (K) centrality. Formally, (1) Degree Centrality: Reflects an agent's number of connections, showing its influence on and vulnerability to others.

where |Nit | denotes the total elements in N t i . (2) Closeness Centrality: Indicates an agent's reachability, suggesting its potential influence over others:

(3) Eigenvector Centrality: Measures an agent's importance by considering both quantity and quality of connections:

where λ is the eigenvalue, indicating the collective influence exerted by an agent and its network.

(4) Betweenness Centrality: Highlights agents that act as bridges or bottlenecks in traffic, crucial in congested scenes:

(5) Power Centrality: Identifies agents in recurrent interactions, hinting at traffic patterns:

where Akii denotes the i-th diagonal element of the adjacency matrix raised to the k-th power, while k! signifies the factorial of k, shedding light on its contribution to the network's structural integrity and dynamism.

(6) Katz Centrality: Emphasizes both direct and distant interactions of an agent, capturing intricate driving patterns:

where n represents the number of agents in the real-time traffic scenario, αk is the decay factor, βk denotes the weight assigned to the immediate neighboring agents, and Akij is the i,j-th element of the k-th power of the adjacency matrix.

To provide better performance in profiling driving behavior, we draw inspiration from metrics such as speed, acceleration, and jerk. Based on these, we introduce three continuous criteria: Behavior Magnitude Index (BMI) C t i , which measures the influence of driving behaviors by evaluating their centrality; Behavior Tendency Index (BTI) Lti , which quantifies behavior propensity by calculating time series derivatives, suggesting higher probabilities of specific behaviors with larger derivatives; and Behavior Curvature Index (BCI) I t i , which uses the jerk concept to measure the intensity of driving behaviors by calculating the second-order derivatives of continuous centrality measures. Formally,

In combination, these criteria offer a holistic view of driving behaviors without needing manual labeling during training, addressing issues of changing behavior labels and time window selection.

The encoder of Driver Behavior Profiling comprises two main components: the LSTM and multi-head self-attention mechanism. The behavior-aware criteria are first processed by the LSTM, yielding temporal vectors:

Here, the LSTM encoder updates the hidden state of agent vi frame-by-frame with shared weights. Using the multi-head self-attention mechanism and GLUs, we compute attention weights for different agent behaviors, producing precise sequential behavioral features in a similar way in QSA:

Lastly, the behavioral features are integrated with priority features and contextual features, then forwarded to the Interaction-Aware module for further processing.

4.2 Priority-Aware Module

In light of recent advances in cognitive studies, it has become evident that the spatial positioning of vehicles within a scene can variably influence the behavior and decisions of a target vehicle. For instance, vehicles located directly in the anticipated trajectory path tend to exert greater influence relative to those situated behind. Furthermore, during overtaking maneuvers, vehicles positioned on the left may carry augmented significance. Recognizing these spatial intricacies, we introduce the Priority-Aware Module. This sophisticated module adeptly transforms the spatial coordinates of agents, encoding them into high-dimensional positional vectors, thereby yielding detailed positional features. Our pooling mechanism adeptly amalgamates dynamic positional information from the encompassing traffic scenario, effectively capturing both singular and multi-agent positional vectors. This mechanism emphasizes the dynamic nuances of position data, accommodating historical agent states, denoted as Stk i , as well as the intricate spatial interplay symbolized by P tk i,j . Mathematically, these relationships are represented as Stk i = {ptk i -ptk-1 i , vtk i -vtk-1 i , atk i -atk-1 i } and P tk i,j = {ptk i -ptk j , vtk i -vtk j , atk i -atk j }, where tk ∈ [t-th, t]. Within this module, the encoder, which amalgamates both LSTM and multi-head attention mechanisms, meticulously processes the dynamic positional vectors. This processing involves transmuting discrete position vectors into a more continuous spatio-temporal domain, thereby enhancing the representation of temporal and spatial dynamics. At each discrete temporal instance t, the encoder assimilates recent historical position vectors via an LSTM network:

Subsequently, the LSTM's output is channeled through a multi-head attention mechanism, culminating in the synthesis of refined priority features:

4.3 Interaction-Aware Module

To better understand the synergistic influence of surrounding vehicles' risk levels, positions, and one's own behavior on the target vehicle's future trajectory, we introduce the Leanformer in this module, which is a lightweight version of the Transformer, allowing for a favorable trade-off between accuracy and efficiency. Mathematically,

This equation captures the integrated effect of safety perception and behavioral tendencies on the target agent's trajectory.

4.4 Multimodal Decoder

The decoder, rooted in a Gaussian Mixture Model (GMM) with multimodality, employs a dedicated LSTM and a fully connected layer. It processes the composite interactive vectors ¯O to forecast the target vehicle's trajectory. The predicted trajectory, Y t:t+tf 0 , is determined by:

where Fθ(·) = ϕReLU (ϕMLP [ϕGN (ϕLSTM(·))]) Moreover, ϕReLU is the ReLU activation, and ϕGN is Group Normalization, used for improved training stability. The decoder's output comprises multiple future trajectories for target vehicle.

5 Experiments

5.1 Experimental Setup

We carried out experiments using three prominent datasets: NGSIM [Deo and Trivedi, 2018], HighD [Krajewski et al., 2018], and MoCAD [Liao et al., 2024b], adhering to a common dataset division framework. Trajectories were split into 8s segments, where the initial 3s served as historical data, and the succeeding 5s were used for assessment, forming the complete test set. The Root Mean Square Error (RMSE) was employed as the evaluation metric. Recognizing a gap in research concerning data omissions in trajectory prediction, we established the missing test set, which was further categorized into three subsets based on the duration of data omissions: drop 3-frames, drop 5-frames, and drop 8-frames. Omissions were purposefully made around the midpoint of the historical trajectory. For instance, in the drop 5-frames subset, data ranging from the (t-8)th to the (t-12)th frame was excluded. To manage these omissions, we employed simple linear interpolation. Furthermore, we train our model on only 25% of the dataset and evaluate it on complete test sets to evaluate its ability to handle unfamiliar data and manage data omissions. Additionally, our model is trained on four A40 48G GPUs. We employ the Adam optimizer and utilize CosineAnnealingWarmRestarts for the scheduler.

5.2 Experimental Results

Performance Comparison on Complete Test Set. As illustrated in Table 1, our model surpasses the best of SOTA baselines with noteworthy gains of 10.6% and 16.2% for shortterm (1s-3s) and long-term predictions (4s-5s) respectively on the NGSIM dataset. Moreover, Table 2 highlights fewer prediction inaccuracies on the HighD dataset than NGSIM, due to HighD's superior trajectory precision and larger sample size. Notably, our model's long-term forecasting outperforms SOTA baselines, showing RMSE improvements of 27.4% for a 5s prediction horizon. Additionally, as shown in Table 3, our model notably excels on busy urban roads, surpassing SOTA baselines by at least 12.2% for short-term predictions and reducing long-term prediction errors by at least 0.57 meters on MoCAD, potentially decreasing traffic accident risks. Performance Comparison on Missing Test Set. Tables 1, 2, and 3 spotlight our model's robustness, even in the face of incomplete datasets. Consistently, across the drop 3-frames and drop 5-frames datasets, it outperforms all baselines. Notably, within the drop 3-frames context, our model even eclipses several leading SOTA benchmarks that were evaluated using complete test sets. However, no model is without its exceptions. In the drop 5-frames dataset, while our model largely dominates, it trails behind DACR-AMTP on NGSIM and the short-term forecasts of iNATran on HighD. Still, these are isolated instances, and the overarching trend confirms the model's robustness and versatility. A deeper dive reveals a predictable trend: the model's performance is inversely proportional to the omission of input data. Yet, even under a stringent scenario like the drop 8-frames-where half the data is missing-our model stands its ground, frequently matching or surpassing most SOTA benchmarks.

Performance on a Limited 25% Training Set. To challenge our model's adaptability, we trained it using a quarter of the available training set from the NGSIM, HighD, and MoCAD, yet evaluated its performance on the complete test set. Impressively, as shown in Tables 1, 2, and 3, even with this limited training data, our model delivered RMSE values that were notably lower than most baseline models. Such results underscore our model's efficiency and robustness in trajectory prediction. This performance indicates a promising potential: our model might substantially cut down on the data demands typically associated with training autonomous vehicles, particularly in scenarios that are data-scarce or unconventional.

5.3 Qualitative Results

It is an intrinsic understanding that vehicles in closer proximity often pose greater risks. However, an aspect frequently neglected by many models is that even distant vehicles, based on their driving behavior, can significantly influence the ego vehicle's trajectory. In our observations, driving behavior substantially impacts risk evaluations. Specifically, drivers with a more aggressive demeanor tend to escalate the risk levels. Drawing a direct comparison, our model's predictions stand out in their precision and alignment with reality, especially when juxtaposed against its counterparts like Stdan and WSiP. Taking the right merge scenario as a case in point: where Stdan and WSiP show discrepancies, it forecasts with minimal deviation, resonating closely with the ground truth. This differential is not just a testament to our model's accuracy but also its adeptness at capturing the nuanced effects of driving behaviors, a facet that seems muted in other models.

5.4 Ablation Studies

We conduct a comprehensive ablation study to assess each component's contribution to our model, with results summarized in Table 4. Model F, which includes all components, consistently outperforms other variations across all metrics, demonstrating their collective value in achieving optimal performance. Model A, excluding the DBP within the Perceived Safety-Aware module, shows significant performance degradation, particularly in short-term predictions, with drops of at least 13.9% and 24.0% on the NGSIM and HighD, respectively. This underscores the critical role of DBP in enhancing trajectory prediction accuracy. Model B, a reduced version of Model F without the Perceived Safety-Aware module, shows a significant reduction in RMSE, particularly for long-term predictions, with improvements of at least 19.6% and 26.2% on NGSIM and HighD datasets, respectively. This emphasizes the importance of perceived safety factors in trajectory prediction, especially for long-term predictions. Model C, which uses absolute coordinates instead of relative positions, demonstrates notable reductions in prediction metrics, underscoring the importance of spatial relationships for accuracy. Model D, lacking the Interaction-Aware module, shows performance drops of at least 6.5% and 20.0% on the NGSIM and HighD datasets for short-term predictions, and at least 12.6% and 18.2% for long-term predictions, respectively. Finally, Model E, with reduced multimodal prediction in the decoder, experiences at least 4.4% and 8.2% performance degradation on the NGSIM and HighD datasets, respectively.

6 Conclusion

This work presents a novel approach rooted in cognitive insights, emphasizing the crucial role of perceived safety in driving behaviors. Our Perceived Safety-Aware Module harmoniously merges Quantitative Safety Assessment and Driver Behavior Profiling, offering a detailed perspective on safety perceptions in driving. Rigorous evaluations on NGSIM, HighD, and MoCAD highlight our model's robustness and adaptability, even under data constraints and data missing. In conclusion, our findings suggest a promising trajectory for future research, bridging computational strength with human cognition to drive safety and efficiency in AVs.

Acknowledgements

This research is supported by the Science and Technology Development Fund of Macau SAR (File no. 0021/2022/ITP, 0081/2022/A2, 001/2024/SKL), ShenzhenHong Kong-Macau Science and Technology Program Category C (SGDX20230821095159012), and University of Macau (SRG2023-00037-IOTSC). Haicheng Liao and Zhenning Li contributed equally to this work. Please ask Dr. Zhenning Li (zhenningli@um.edu.mo) for correspondence.