MFTraj: Map-Free, Behavior-Driven Trajectory Prediction for Autonomous Driving

1 Introduction

The integration of autonomous vehicles (AVs) with humandriven vehicles and pedestrians necessitates advanced trajectory prediction models. Central to these models is their ability to predict the future trajectories of various road users, leveraging historical data. Despite significant advancements, a pivotal challenge persists: modeling the often unpredictable driving behaviors of road users.

These behaviors, shaped by a blend of traffic dynamics, road layouts, and individual cognitive inclinations, offer a unique window into the real-time decision-making processes of humans in complex traffic settings [Schwarting et al., 2019]. Our research has illuminated the pivotal role of understanding human behavioral patterns in trajectory predictions. Recognizing and predicting human driving behavior is not merely about tracing a vehicle's path; it's about understanding the cognitive processes that dictate those paths. By understanding behaviors, AVs can anticipate sudden changes in human-driven vehicles or pedestrian movements, leading to safer co-navigation. Furthermore, behavior-focused predictions can aid in scenarios where traditional data might be ambiguous or incomplete, relying on human behavioral patterns to fill in the gaps. Through the integration of decisionmaking theories, cognitive psychology, and traffic behavior studies [Yin et al., 2021], trajectory prediction models can be enriched, fostering a harmonious coexistence of AVs and human-driven entities on the road.

High Definition (HD) maps, conventionally considered pivotal for trajectory prediction, pose intrinsic challenges. Their creation is resource-intensive, and in the rapidly changing milieu of urban environments, they can quickly become obsolete [Gao et al., 2020]. This has given rise to mapfree models, a paradigm shift that operates independently of HD map data. However, while these models adeptly handle dynamic environments, they may lack the granularity provided by comprehensive road network data. This gap is aptly addressed by the advent of deep learning techniques, notably Graph Neural Networks (GNNs) [Liang et al., 2020; Gao et al., 2020]. GNNs, adept at assimilating extensive data from road users, offer nuanced insights into their interactions and the overarching socio-cognitive context, thereby compensating for the lack of detailed HD maps.

Our contributions are as follows:

1. An advanced map-free architecture for trajectory prediction that obviates the need for HD maps, resulting in significant computational savings.
2. A novel dynamic geometric graph that captures the essence of continuous driving behavior, circumventing the limitations of manual labeling. We have integrated behavioral metrics and criteria, drawing from traffic psychology, cognitive neuroscience, and decision-making frameworks, to craft a model that offers more than mere predictions-it elucidates.
3. Benchmark assessments underscore MFTraj's superior-

ity. Demonstrating a commendable performance elevation of nearly 5.0% on the Argoverse, NGSIM, HighD, and MoCAD datasets, its robustness is further accentuated by its consistent performance even with a data shortfall of 25%- 62.5%, underscoring its adaptability and profound understanding of diverse traffic dynamics.

2 RelatedWork

Recent years have seen an explosion of research in trajectory prediction for autonomous driving (AD), thanks to the burgeoning field of deep learning. These cutting-edge approaches [Liao et al., 2024a; Messaoud et al., 2021; Tian et al., 2024; Liao et al., 2024e] have demonstrated superior performance in complex traffic scenarios. However, they often encounter challenges in adequately representing spatial relationships, such as graphic inputs of the scene. To address this, HD maps, rich in scene and semantic information, have attracted increasing research attention. Considering that Convolutional Neural Networks (CNNs) excel at extracting spatial features from input data, such as spatial features from inputs like vectorized maps or raster images, several studies [Zhao et al., 2021; Gilles et al., 2021; Khandelwal et al., 2020] have merged sequential networks with CNNs. This hybrid approach effectively captures both temporal and spatial features from HD maps, providing enriched contextual information for motion prediction tasks. Recent research has explored Graph Neural Networks (GNNs) [Liang et al., 2020; Mohamed et al., 2020; Liao et al., 2024c], such as Graph Convolutional Networks (GCNs) and Graph Attention Networks (GATs), Transformers [Zhou et al., 2022; Liao et al., 2024d; Chen et al., 2022a; Ngiam et al., 2022], and generative models such as Adversarial Networks (GANs) [Zhou et al., 2023] and Variational Auto Encoders (VAEs) [Walters et al., 2021] for direct encoding of HD maps. For example, VectorNet [Gao et al., 2020] simplifies maps by extracting key points from lane splines and encoding them using GNNs. Moreover, LaneGCN [Liang et al., 2020] and TPCN [Ye et al., 2021] build lane graphs using central line segments, employing GCN to capture dynamic interaction. In addition, HiVT [Zhou et al., 2022] and SSLLanes [Bhattacharyya et al., 2023] represent map elements by relative positions, improving the transformer model for time series trajectory data.

Despite their effectiveness, the limited availability of HD maps and the extensive resources needed for their creation and maintenance impede the widespread adoption of autonomous driving, particularly in areas lacking current HD map coverage. In response to these challenges, this study introduces a map-free model that utilizes generative models and a VRNN [Chung et al., 2015] to account for the variable nature of traffic scenarios. We propose a novel adaptive GCN to model the complexity of real-time interactions in traffic scenes. To streamline model complexity, we apply the Linformer framework [Wang et al., 2020] for a balance between computational efficiency and prediction accuracy in AD applications.

3 Methodologies

3.1 Inputs and Outputs

This study focuses on predicting the trajectory of the target vehicle in interactive driving scenarios, considering all vehicles within the AV's (termed the target vehicle) sensing range. At time t, the ego vehicle anticipates the target vehicle's trajectory for the upcoming tf steps. Our model, drawing from historical data X, considers past trajectories of both the target vehicle (indexed by 0) and its surrounding agents (indexed from 1 to n) over a predefined horizon th. Formally,

where Xt-th:t 0 = {xt-th 0 , xt-th+1 0 , . . . , xt0 } and Xt-th:t i = {xt-th i , xt-th+1 i , . . . , xti } represent the historical trajectories of the target vehicle and those of the surrounding agents from time t - th to t, respectively.

The output of the model is the future trajectory of the target vehicle during the prediction horizon tf :

where yt0 is the 2D coordinates of the target vehicle at time t.

Our model uniquely operates without relying on maps, using only the historical trajectory data of the target and nearby agents. The model needs an input sequence of length th and remains functional even if the historical data is not perfectly sequential. For sporadic missing data points, due to reasons like occlusions or sensor glitches, we employ simple linear imputation or similar methods. In addition, Figure 1 illustrates our proposed model's hierarchical design. Following the encoder-decoder format, it features four key components: behavior-aware, position-aware, interaction-aware modules, and the residual decoder. We delve into each module's specifics below.

3.2 Behavior-aware Module

Moving away from traditional methods that classify driver behaviors into fixed and discrete categories, we offer a more adaptable and flexible solution with our behavior-aware module, which utilizes a continuous portrayal of behavioral attributes. This approach draws inspiration from the multipolicy decision-making framework [Markkula et al., 2020], integrating elements of traffic psychology and dynamic geometric graphs (DGGs) [Boguna et al., 2021] to effectively capture intricate driving behaviors amid ongoing driving maneuvers and evolving traffic conditions.

Dynamic Geometric Graphs. We first model the interactions of different agents with a DGG. At time t, the graph Gt is defined as:

where V t = {vt0 , vt1 . . . , vtn } is the set of nodes, vt i is the ith node representing the i-th agent, Et = {et0 , et1 . . . , etn } is the set of edges representing potential interactions between agents, and eti is the edge between the node vt i and other agents that have potential influences with it. An interaction is assumed to exist only if two agents, e.g., vi and vj, are in close proximity to each other, i.e., their shortest distance d (vt i , vt j ) is less than or equal to a predefined threshold r. Therefore, we define

where N t i = {vt j ∈ V t \{vt i } | d (vt i , vt j ) ≤ r, i ̸= j}.

Correspondingly, the symmetrical adjacency matrix At of Gt can be given as:

Centrality Measures. Centrality measures are graphtheoretic metrics that are widely used to understand various aspects of network structures. These measures provide valuable insights into the importance, influence, and connectivity of nodes or vertices within a graph. As shown in Table 1, we use six centralities to characterize driving behavior. These measures allow evaluation of individual agent behavior within the DGG and reveal key agents and overall connectivity of the traffic graph.

(1) Degree Centrality: Reflects the number of connections an agent has, correlating with its influence on and susceptibility to others. It's defined as:

where |Nit | denotes the total elements in N t i . (2) Closeness Centrality: Indicates an agent's reachability, suggesting its potential influence over others. Defined by:

(3) Eigenvector Centrality: Measures the importance of an agent by considering both the quantity and quality of connections. Expressed as:

where λ is the eigenvalue, indicating the collective influence exerted by an agent and its network.

(4) Betweenness Centrality: Highlights agents that act as bridges or bottlenecks in traffic, crucial in congested situations. Formulated as:

where V t denotes the set of all agents present in the scene, σj,k signifies the total number of shortest paths between agent vt j and agent vtk , and σj,k(vi) represents the number of those paths traversing the agent vt i .

(5) Power Centrality: Identifies agents in recurrent interactions, hinting at traffic patterns. Defined by:

where Akii denotes the i-th diagonal element of the adjacency matrix raised to the k-th power, while k! signifies the factorial of k, shedding light on its contribution to the network's structural integrity and dynamism.

(6) Katz Centrality: Emphasizes both direct and distant interactions of an agent, capturing intricate driving patterns. Given as:

(11)

where n represents the number of agents in the real-time traffic scenario, αk is the decay factor, βk denotes the weight assigned to the immediate surrounding agents, and Akij is the i,j-th element of the k-th power of the adjacency matrix.

Behavior-aware Criteria. Inspired by the relationship between velocity, acceleration, and jerk, we introduce behavioral criteria. These criteria, consisting of Behavior Magnitude Index (BMI) C t i , Behavior Tendency Index (BTI) Lti , and Behavior Curvature Index (BCI) I t i , evaluate different driving behaviors for the target vehicle and its surroundings. They compute thresholds, gradients, and concavities of centrality measures that reflect behaviors such as lane changes, acceleration, deceleration, and driving style. We find that behaviors with significant, fluctuating centrality values in short time frames are likely to affect nearby agents, consistent with human risk perception and time-sensitive decision-making. They are respectively given as follows:

Behavior Encoder. Incorporating behavior-aware criteria, symbolized as J = {C t-th:t 0:n ,Lt-th:t 0:n , I t-th:t 0:n }, our behavior encoder comprises VRNN and GRU components. This encoder succinctly models relationships between random variables across time, yielding precise sequential behavioral features ¯Ot-th:t behavior . Formally:

where ϕGRU and ϕVRNN denote the GRU and VRNN functions. This encoder captures human driving patterns and their temporal dynamics. Next, behavioral features ¯Ot-th:t behavior , fuse with positional features from the position-aware module, subsequently processed by the interaction-aware module for comprehensive feature extraction.

3.3 Position-aware Module

Contrary to traditional methods that emphasize absolute positions [Wang et al., 2022a; Gao et al., 2020] or fixed grids [Deo and Trivedi, 2018], our model emphasizes relative positions. The position-aware module captures individual and group spatial dynamics, interpreting the scene's geometric nuances. These insights are then encoded to produce positional features.

Pooling Mechanism. Our pooling mechanism captures dynamic position data from the traffic environment around the target vehicle, utilizing individual stk i andmulti-agent stk i,j position vectors. This strategy gleans historical trajectories and spatial relationships without depending on fixed positions or grids. The relationships are formulated as:

Position Encoder. The position encoder employs an LSTM to transform discrete position vectors into continuous spatiotemporal representations, thereby enhancing temporal and spatial interactions between agents and map elements. Given the historical position vectors for the target and surrounding agents, it embeds them temporally:

where ¯Ot-th:t-1 position is the positional features output by the position encoder, and ϕLSTM denotes the two-layer LSTM encoder, and ¯ht-th:t-1 i represents the hidden position state updated by the encoder on a frame-by-frame basis, with the weights of the LSTM shared among all agents.

3.4 Interaction-aware Module

Effective trajectory prediction in complex traffic scenarios hinges upon a system's ability to comprehend and anticipate interactions among vehicles. Classic GCN-based methods, although proficient at encapsulating geometric inter-agent relationships, often exhibit limitations in fluid traffic conditions due to their fixed adjacency matrix configurations. To tackle this, we introduce a novel adaptive structure-aware GCN, taking cues from advancements in crystal graphs and material design. This novel approach stands out by its capability to craft spatial feature matrices dynamically, adjusting to the number of agents observed in real-time, which ensures a more fluid and adaptable response to traffic changes. A graphical illustration of this concept is provided in Figure 2.

Breaking away from conventional models that predominantly lean on distance-based positional features, our design holistically blends continuous behavioral features into its graph structure. This not only addresses the multifaceted spatio-temporal interactions but also considers the intricate physical interplays between agents, offering a noticeable enhancement in prediction precision. Our design blueprint encompasses an adaptive convolutional neural network rooted in a fully connected interaction multigraph. This structure is adept at simultaneously capturing sequential behavioral and dynamic positional interactions among agents. The multigraph's operational layer is distinguished by nodes, which symbolize sequential behavioral features ¯Ot-th:t behavior and edges representing positional features ¯Ot-th:t position , as defined below:

where the variable k denotes the layer within the GCN, k ∈ [1, 3], and the symbols ⊙, ϕsgm, and ϕspu represent the element-wise product, sigmoid activation function, and softplus activation function, respectively. Consequently, Wk-1 g and Wk-1 h are learnable matrices, bk-1 g , and bk-1 h are the bias of the k-th layer. ˜ rk-1 i,j can be represented as follows:

Additionally, the initial feature vector ˜z(0) i is defined as:

Furthermore, the output of the adaptive structure-aware GCN for the target vehicle i is then passed to Linformer, an extension architecture of Transformer,

Furthermore, the output of the adaptive structure-aware GCN for the target vehicle i is subsequently fed into a lightweight transformer-based framework-- Linformer [Wang et al., 2020], to efficiently quantify and compute the dynamic attention weight vectors for the surrounding agents, ultimately output the contextual mapping ¯O. This allows for a favorable trade-off between accuracy and efficiency.

3.5 Residual Decoder

The residual decoder, comprising a linear residual and projection layer, processes node vectors to forecast the target vehicle's future trajectory, producing the prediction Yt:t+tf 0 . This is given by:

such that,

where ϕReLU denotes the ReLU activation function, and ϕGN denotes the Group Normalization (GN) function [Wu and He, 2018], which is applied to improve the training stability of our model. In addition, the ϕLinear corresponds to the fully connected layer, while Fθ denotes the residual decoder function.

4 Experiments

4.1 Experimental Setup

Datasets. We tested model's efficacy on Argoverse [Chang et al., 2019], NGSIM [Deo and Trivedi, 2018], HighD [Krajewski et al., 2018], and MoCAD [Liao et al., 2024b] datasets. Data Segmentation. For Argoverse, we predicted a 3-second trajectory from a 2-second observation, while for NGSIM, HighD, and MoCAD, we use 6-second intervals split into 2 seconds of observation and 4 seconds of prediction. These datasets, referred to as the complete dataset, help assess our model in diverse traffic scenarios. Recognizing that realworld conditions often lead to incomplete data, we further assessed our model's resilience using the Argoverse dataset by introducing four subsets with varying levels of missing data: drop 3-frames, drop 5-frames, drop 8-frames, and drop 10-frames. These missing datasets simulate data loss scenarios. For data gaps, we applied simple linear interpolation. Metrics. Our experimental protocol was aligned with the Argoverse Motion Forecasting Challenge and prior work [Liao et al., 2024b], we evaluated the performance of our model using standard metrics: minADE, minFDE, MR, and RMSE. Implementation Details. We implemented our model using PyTorch and PyTorch-lightning on an NVIDIA DGX-2 with eight V100 GPUs. Using the smooth L1 loss as our loss function, the model was trained with the Adam optimizer, a batch

size of 32, and learning rates of 10-3 and 10-4 .

4.2 Experimental Results

Performance Evaluation on the Complete Dataset. Tables 2 and Table 4 present a comparative evaluation of our trajectory prediction model against 25 baselines from 2016 to 2023. Unlike most approaches that depend on HD maps or vectorized map data, our model omits map-based inputs. Still, it consistently outperforms the baselines across metrics like minADE, minFDE, MR, and RMSE for both Argoverse and MoCAD datasets. Specifically, for the Argoverse dataset, MFTraj outperforms most of the SOTA models by margins of 2.9% in minADE, 2.4% in minFDE, and 3.8% in MR, while being on par with HiVT. It excels particularly in challenging long-term predictions (4s-5s) on NGSIM, HighD, and MoCAD datasets, with reductions in forecast error surpassing at least 11.5%, 29.6%, and 21.9%, respectively. This emphasizes its potential for accurate long-term predictions in highway and urban settings.

Performance Evaluation on the Missing Dataset. Table 2 showcases the resilience of our model when faced with incomplete data sets. Our model consistently outperforms all other baselines on the drop 3-frames and drop 5-frames datasets. Notably, on the drop 3-frames dataset, it surpasses nearly all state-of-the-art (SOTA) models trained on full data, highlighting its remarkable predictive strength even with missing data. While its performance on the drop 5frames dataset excels over most baselines, there are exceptions in specific metrics against models like TNT, WIMP, and mm Transformer. As the number of missing frames increases, as in the drop 8-frames and drop 10-frames datasets, there's an expected decline in performance. Yet, even with half the input data missing, our model still competes strongly against top baselines, emphasizing its potential in environments with data interruptions.

Comparative Analysis of Model Performance and Complexity. In Table 3, we compare our model's performance and complexity with various SOTA baselines. While our model doesn't have the lowest parameter count, it excels in all performance metrics. Impressively, it achieves this while using 90.42% and 87.18% fewer parameters than WIMP and Scene-Transformer, respectively. Compared to top-10 SOTA models, our model not only surpasses them in accuracy but is also as efficient, if not more so, than HiVT-128, SSLLanes, LaneGCN, and HOME+GOHOME. This underlines our model's optimal balance of robustness, efficiency, and trajectory prediction accuracy.

4.3 Ablation Studies

We executed an ablation study to assess the impact of individual components within our trajectory prediction model, with the results summarized in Table 5. Model F, i.e., MFTraj, which integrates all components, stands out in all metrics, signifying the synergy of its parts. When the behavior-aware module is excluded in Model A, there are noticeable drops in minADE, minFDE, and MR by 12.6%, 8.8%, and 8.5% respectively, highlighting its pivotal role. Model B, with absolute coordinates, underperforms, emphasizing the relevance of spatial relationships. Model C, without the interactionaware module and Linformer extension, and Model D, lacking Linformer, both show diminished performance. Similarly, Model E, which uses a standard GCN instead of the adaptive one, also lags, underscoring the latter's efficiency. In essence, this study solidifies the importance of each component in Model F. Every part, from understanding behavioral nuances to updating features effectively, bolsters the model's precision and resilience. In essence, this study solidifies the importance of each component in Model F. Every part, from understanding behavioral nuances to updating features effectively, bolsters the model's precision and resilience.

4.4 Qualitative Results

Figure 3 presents the qualitative results of our model using the Argoverse dataset. We've limited the display to the target vehicle's trajectories for clarity. Interestingly, without the aid of HD maps, our model adeptly discerns road semantics, enabling it to make precise and logical predictions for target vehicles in intricate urban settings. Importantly, Figure 4 illustrates a comparison between the trajectories predicted by MFTraj and the SOTA baselines in the same traffic scenarios. MFTraj outperforms Stdan and WSiP in trajectory prediction, especially in complex scenarios such as lane changes and merging. These results demonstrate the superior adaptability and reliability of MFTraj in complex traffic conditions.

5 Conclusion

This work presents a map-free and behavior-aware trajectory prediction model for AVs, integrating four components: behavior-aware, position-aware, interaction-aware modules, and a residual decoder. These components work in concert to analyze and interpret various inputs, understand humanmachine interactions, and account for the inherent uncertainty and variability in the prediction. Evaluated with the Argoverse, NGSIM, HighD, and MoCAD datasets, MFTraj outperformed SOTA baselines in prediction accuracy and efficiency without additional map information. Furthermore, this approach ensures its robustness and adaptability even in the presence of significant missing data; it achieved impressive performance even with a 50% sequential input data deficit. This underscores the resilience and efficiency of MFTraj in predicting future vehicle trajectories and suggests its potential to drastically reduce the data requirements for training AVs, especially in data-missing and limited data scenes.

Acknowledgements

This research is supported by the Science and Technology Development Fund of Macau SAR (File no. 0021/2022/ITP, 0081/2022/A2, 001/2024/SKL), ShenzhenHong Kong-Macau Science and Technology Program Category C (SGDX20230821095159012), and University of Macau (SRG2023-00037-IOTSC). Haicheng Liao and Zhenning Li contributed equally to this work. Please ask Dr. Zhenning Li (zhenningli@um.edu.mo) for correspondence.