G-VTM: A Multimodal Vision-Trajectory Model for Generalized Vehicle Trajectory Prediction

1 Introduction

Vehicle trajectory prediction at junctions remains a critical task in Cooperative Vehicle-Infrastructure Systems (CVIS). This study focuses on generalized vehicle trajectory prediction across diverse junctions, which has significant application in intelligent transportation systems (ITS) [Zhao et al., 2020; Cai et al., 2021; Schimpf et al., 2023; Huang et al.,

∗Corresponding author.

2025]. For example, a model trained on a signalized fourway intersection can be efficiently transferred to an unseen roundabout with a few training data, thereby reducing data collection and model deployment costs. We analyse trajectories of road users derived from unmanned aerial vehicles (UAVs) recorded videos. These UAVs are employed as roadside perception devices. Existing methods are typically tailored to a specific junction [Liao et al., 2024b; Wei et al., 2024], failing to achieve generalization, because they cannot effectively address two key challenges.

Generalizing across junctions with different map semantics and traffic behaviour patterns is the first challenge. The different map semantics primarily arise from distinct road topologies (e.g., global size, road geometry, and lane connectivity) and traffic regulations (signalized or unsignalized). The traffic behavioral patterns of road users are jointly influenced by road network constraints, traffic density, and traffic regulations, leading to heterogeneous data distributions across different junctions. Fig. 1 presents two different traffic scenarios and corresponding data distributions. At the roundabout, road users follow priority-based rules, and the traffic density exhibits a circular shape. The heading angles exhibit a dispersed distribution due to continuous steering adjustments at roughly constant velocity. In contrast, behavior patterns of road users are strictly regulated at the signalized four-way intersection, leading to a deceleration-stop-restart stage-wise movement. The distribution of heading angles is rather concentrated, as road users always execute left or right-angle turns.

Another challenge lies in modeling scenario-adaptive interactions among road users to facilitate generalization. Existing methods primarily employ Graph Neural Networks (GNNs) to learn interactive motion among road users based on fine-grained relative spatial features such as Euclidean distance and polar angle [Mo et al., 2021; Westny et al., 2023]. However, this single-modality trajectory learning lacks holistic perceptions of the surrounding scenario, which explicitly constrain interaction. Besides, it is limited in intuitively capturing direction-aware relative movement relations between the target and interacting road users, such as co-directional parallel or opposing bidirectional movement.

We observed that vision modalities could provide complementary advantages to the trajectory modality. Specifically, although coarse-grained visual images lack precise positional and movement details, they offer a holistic view that facilitates learning global map semantics with local relative spatial relations. Moreover, they provide intuitive perceptual fields to capture overall behavioral patterns of each trajectory and explicitly reveal relative movement relations between the target and interacting road users.

To address the above challenges, we propose a Generalized Vision-Trajectory Model (G-VTM) that effectively extracts and fuses multimodal information. G-VTM introduces two key components: (1) A Map-aware Vision Learner learns global map semantics with local relative spatial relations, and models scenario- and direction-aware interaction from coarse-grained visual features based on a multiscale strategy; (2) A Spatio-temporal Trajectory Learner captures finegrained behavioral patterns of road users guided by visual priors. Finally, a Transformer encoder effectively fuses multimodal semantics, producing future trajectory predictions through a trajectory predictor. Our contributions are summarized as follows:

• We introduce G-VTM, a model that unifies vision and trajectory modality for generalized prediction of vehicle trajectories across diverse junctions.
• A Map-aware Vision Learner is designed to learn global map semantics for map images of arbitrary resolution, and model scenario- and direction-aware visual interaction between the target and interacting road users. Besides, A Spatio-temporal Trajectory Learner captures fine-grained behavioral patterns of road users guided by visual priors.
• We conduct extensive experiments on three real-world datasets. G-VTM achieves strong generalized performance in few-shot and zero-shot settings.

2 RelatedWork

Vehicle trajectory prediction aims to forecast the future motion of road users. It emphasizes modeling interactions among road users and capturing the surrounding scenario semantics. Traffic scenarios that existing methods focused on can be grouped into two types: (i) urban intersections; (ii) roundabouts and highway on-ramps and off-ramps.

Trajectory Prediction at Urban Intersections Urban intersections are among the most challenging traffic scenarios due to complex road topology, explicit traffic regulations, and multiple classes of road users. Early works primarily focus on trajectory prediction of a single road user based on RNNbased or physics-based models [Kawasaki and Tasaki, 2018; Abdelraouf et al., 2022], but fail to capture the interactions among multiple road users. To address this limitation. GRIP++ [Li et al., 2019] constructs a spatio-temporal graph network to model dynamic interactions between road users. FJMP [Rowe et al., 2023] combines attention mechanisms with sparse interaction graphs based on influencer-reactor relationships to improve future interaction modeling. MTR [Shi et al., 2022a] introduces learnable motion query pairs to model both global overall intention and local fine-grained trajectory dynamics, enabling efficient prediction of multimodal trajectories. To handle problems of non-uniform sampling and long-horizon uncertainty, continuous-time modeling has also been explored. LG-ODE [Huang et al., 2020] models multi-agent trajectories in continuous time using Neural ODEs with a latent interaction graph. MTP-GO [Westny et al., 2023] leverages Neural ODEs together with an Extended Kalman Filter (EKF) to facilitate multi-modal probabilistic trajectory prediction. Moreover, HPNet [Tang et al., 2024] models the dynamic dependencies across continuoustime prediction steps via an attention mechanism, using historical predictions to guide the current prediction. Recognizing the impact of traffic signals, KI-GAN [Wei et al., 2024] captures vehicle motion with signal states to generate accurate predictions under traffic regulations.

Trajectory Prediction at Highway On-Ramps and OffRamps and Roundabouts In highway on-ramps and offramps scenarios, existing studies mainly focus on forecasting vehicle behaviors such as lane changes and acceleration/deceleration. CS-LSTM [Deo and Trivedi, 2018] and PiP [Song et al., 2020] introduce convolutional social pooling to capture fine-grained interactions. WSiP [Wang et al., 2023] advances this concept by modeling interactions as wave superposition through a wave-pooling mechanism, which. C2F-TP [Wang et al., 2025] adopts a diffusion-based framework, achieving strength in long-horizon prediction. BAT [Liao et al., 2024b] models driving behavior using dynamic geometric graphs. HLTP [Liao et al., 2024a] incorporates vision-aware pooling with a knowledge distillation framework to refine interaction representation while reducing inference latency. Roundabouts often have more complex road topologies and priority-based traffic regulations, which result in more complicated interactions between road users. MMH-STA [Sun et al., 2023] introduces a macro-micro hierarchical spatio-temporal attention structure to model multiagent trajectory prediction. FSD-Transformer [Yang et al., 2025] leverages traffic states and driving styles to adapt predictions under varying traffic conditions.

3 Preliminaries

3.1 Definition

Figure 2: Overall framework of G-VTM.

mapped onto the hue wheel. With saturation fixed, the lightness is determined by the ratio of time t in the historical horizon tf , representing the temporal evolution of the trajectory.

Definition 1 (Trajectory Features). Features of a road user ν at time t include the current position τ ν t = (xνt , yν t ) and motion information (e.g., velocity, acceleration, and heading angle). Road users include both the target road user νT and interacting road users νi ∈ VI, such as bicycles, motorcycles, and pedestrians. Each road user has its class attribute represented as a one-hot vector ρv of length Nc. We select six road users geographically closest to the target road user, forming the set VI of interacting road users.

Definition 2 (Global Map Image). For a given junction, we visualize its road topology and traffic elements using the provided OpenStreetMap format map. First, the global map is split into 1 m × 1 m grids. Each grid corresponds to a fixed number of pixels and is further grouped into patches. Each patch includes a fixed number of grids. Thereby, ensuring the resolution of the map image is positively correlated with the physical dimensions of the junction.

Definition 3 (Local Trajectory Image). Since the trajectory of road users is concentrated within a local region of the global map, we extract a fixed-resolution local image for each road user to facilitate generalization. We visualize dynamic behaviour patterns of road users on the static map image based on the grid level. The relative position offset between τt-1 and τt is used to compute the direction angle, which is

3.2 Problem Statement

Trajectory Prediction. Given the observed trajectory Hν = ⟨τ ν t0-th+1 , τ ν t0-th+2 , · · · , τ ν t0 ⟩, ν ∈ {νT} ∪ VI of road users in the past th timesteps, the modeling objective of trajectory prediction is to estimate the future trajectory ⟨τ νT t0+1 , τ νT t0+2 , · · · , τ νT t0+tf ⟩ of the target road user, where tf is the prediction horizon and t0 is the current time.

Generalization. The objective of generalization is to learn a robust model that performs well across diverse scenarios. Once pretrained on the source dataset, this model could effectively transfer to the unseen target dataset, predicting accurate future trajectories based on few-shot inputs.

3.3 Rotary Position Embedding

Positional encoding is essential for capturing order- and spatial-aware relationships. Rotary Position Embedding (RoPE) represents relative positional information by rotating feature vectors in the embedding space.

Taking 1D RoPE as an example. Given embedding em ∈ Rd of position m, we calculate query vector qm = Wqem and key vector km = Wkem. The rotary transformation is calculated by rotation Matrixs R(m) which is defined as:

where θk = 10000-2k/d are frequency weights. The inner product in the attention between qm at position m and kn at position n enbale relative position-aware interactions:

4 Methodology

We proposed G-VTM to integrate the complementary strengths of vision and trajectory modalities. As illustrated in Figure 2, G-VTM comprises two key components: a Mapaware Vision learner and a Spatio-temporal trajectory learner.

4.1 Map-aware Vision Learner

General Global Map Encoder. When processing images with variable resolutions, Vision Transformer (ViT) generally employs linear interpolation for absolute positional embeddings, which is inadequate for capturing relative spatial geometric relationships between map patches. Therefore, we incorporate adaptive-resolution rotary positional encoding into the ViT. Specifically, ViT first projects the global map image into patch embeddings Gm = ⟨e1, e2, · · · , eNg ⟩. We assign two-dimensional positional indices to each patch, facilitating the modeling of spatial topological relationships. To support images of arbitrary resolutions, we transform positional indices to a new coordinate based on a scaling factor, which is calculated by the ratio between the current image resolution and the reference resolution (i.e., image size in the pre-trained setting of ViT). Next, we introduce rotation matrices Rx and Ry in the self-attention as definated in Section 3.3:

Local Map Semantic Extraction. Based on the local trajectory image of each road user, we extract corresponding local patch indices I ν = {i1, i2, · · · , iNs } from the global map. The local map patch embeddings can be retrieved from Gm, denoted as Gνs = ⟨ei1 , ei2 , · · · , eiNs ⟩ .

MultiScale Vision Interaction. This module is designed to model scenario- and direction-aware interaction among road users. We employ a multiscale strategy to extract valuable information from coarse-grained visual features. At the ℓ level, the local trajectory images of the target and interacting road users are hierarchically encoded by convolutional layers. Incorporating Gνs , map semantics fused embeddings {Eν 0 |ν ∈ {νT } ∪ VI} are obtrained by a linear combination.

A decomposed attention mechanism is employed to model vision interaction. First, the spatial self-attention mechanism learn the spatial correlation between patch embeddings from Eν 0 for each road user, which is formulated as follows:

where Qν , Kν and V ν are derived from Eν 0 .

To capture overall behaviour representation for each road user, a self-view convolution is applied in Eν 1 to aggregate multi-channel features into a single-channel representation Eν 2 with a kernel of size 1×1. The cross-view attention aims to model the interaction between the target and interacting road users within the same feature channel. The interaction embedding EνT 3 is obtained as follows:

where QνT fromEνT 1 , KνI and V νI fromEνI 1 .

Multiscale Feature Fusion. Given self-view embeddings E(ℓ) 2 and E(ℓ-1) 2 at adjacent scale level, the high-level embedding E(ℓ) 2 is first upsampled by bilinear interpolation, and then fused to E(ℓ-1) 2 using a one-dimentional convolution P(·). This coarse-to-fine aggregated method is formulated as follows, especially ˜ E(L) 2 = E(L) 2 :

We fuse multiscale cross-view patch embeddings with the same calculation in Eq. (6). Finally, we obtain scenario- and direction-aware vision interaction patch embeddings ˜ E3 with the token number of Ns.

4.2 Spatio-temporal Trajectory Learner

TrajectoryTrendLayer. Given historical trajectories of all road users, we compute each point's position relative to the first point to preserve scale variations of different junctions. In terms of interacting road users appearing at different times, we temporally align the trajectory features by padding all sequences to the same length tf . Continuous features (e.g., position and velocity) are embedded by a linear layer, and discrete features (e.g., road user class) are fed into indexfetching embedding layers. The embedded vector of τ ν t is obtained by summing all feature embeddings and then fed into a Long Short-Term Memory network to capture temporal dependencies, resulting in hνt .

Vision Informed MoE. The behavioral patterns of road users are strongly constrained by the surrounding scenario semantics. Therefore, we introduce a Vision Informed Mixtureof-Experts that integrates gating networks with a visionguided bias compensation mechanism, capturing scenariogeneral and scenario-specific behavioral patterns for the target and interacting road users. Specifically, we apply the multiscale aggregated self-view embedding ˜ Eν 2 as visual prior information, integrating the fine-grained trajectory movement features embedding hνt , the output is defined as follows:

where ρν is a one-hot vector indicating the gating network selects road user class indexed expert network output. gn denote behaviour-awre gating networks where scenario-general patterns are extracted by hνt and scenery-specific patterns are captued by ˜ Eν 2 . f (a) and f (c) denote the output of the expert network, which is implemented by two MLP layers. To promote different experts to learn diverse behaviour patterns and provide scenario-adaptive selection of expert subsets. We apply the Top-Kγ gating strategy in gating networks, where Ψ(·) denotes the Softmax function:

Trajectory Interaction Embeddings. Conditioned on the behavioral representation ˜hνt , we apply a Graph Attention Network to model spatial interactions between the target and interacting road users. Formally, we compute edge weight with polar diameter(relative distance) and polar angle:

where Wϵ is learning paramter, and σ is the LeakyReLU activation function. We explicitly encode polar diameter dt and polar angle θt features as ηνT ,νI t = MLP3(dνT ,νI t , θνT ,νI t ) between νT and νI. Finally, the interaction embeddings O = [oνT t0-tf +1 ∥oνT t0-tf+2 ∥ · · · ∥oνT t0 ] is obtained as the output of this module:

4.3 Multimodal Fusion and Optimization

We first separately project vision and trajectory interaction embeddings ˜ E3 and O into a shared embedding space integrated with position embeddings and corresponding modality type embeddings. Then, the multimodal tokens are concatenated into a combined sequence, which is fed into shared Nddepth Transformer Encoder layers to learn cross-modal interaction semantics. Finally, we obtain outputting embeddings ˆ E and ˆO. Due to the larger number of vision tokens compared to trajectory tokens, this dominates the sparse visual attention distribution and hinders effective intra-trajectory modeling. We introduce a perceiver resampler (trainable cross-attention) to reduce the redundant number of visual tokens while preserving critical context information.

The model is trained using multimodal loss. For the trajectory modality, ˆO is fed into multi-head decoder layers to produce diverse future trajectories with corresponding probability {ˆ Yk, ˆπk}Nf k=1 . We optimize the most probable prediction k∗ = argmaxk∈Nf (ˆπk) using RMSE loss. Besides, the predicted probability distribution is supervised using the crossentropy loss, where the target distribution is calculated based on a softmax over the negative final displacement error:

For the vision modality, we obtain label patch embeddings ¯ E by applying a pre-trained vision encoder to the future local trajectory image of νT . The KL-Divergence is employed to encourage consistency between ¯ E and ˆ E:

5 Experiments

Datasets. We use three real-world datasets: SinDTianjin [Xu et al., 2022], InD-Bendplatz [Bock et al., 2020], and RounD-Neuweiler [Krajewski et al., 2020]. The SinDTianjin dataset is collected at a signalized intersection in Tianjin, China, recorded at 10 Hz. The InD-Bendplatz dataset is recorded at an unsignalized intersection in Germany at 25 Hz. The RounD-Neuweiler dataset contains trajectories collected from a roundabout in Germany with the same sampling frequency as InD. We observe 20 trajectory points (4s) to predict the future 20 positions (4s), where trajectories from different datasets are resampled at adaptive frequencies to enable uniform temporal intervals. We also filter trajectories with long stationary periods or negligible movement, and split each dataset into training, validation, and test sets in a 6:2:2 ratio based on recording time.

Baselines and Metrics. We compare G-VTM with stateof-the-art trajectory prediction models that focus on different traffic scenarios. First, we include junction-oriented trajectory prediction models: MTR [Shi et al., 2022b], HPNet [Tang et al., 2024], BAT [Liao et al., 2024b], and KIGAN [Wei et al., 2024], which explicitly model road topologies and interactions of multitype road users. Second, we evaluate several methods designed for highway scenarios that excel at capturing temporal consistency and behaviourawareness, such as CS-LSTM [Deo and Trivedi, 2018], WSiP [Wang et al., 2023], STDAN [Chen et al., 2022], HLTP [Liao et al., 2024a], and C2F-TP [Wang et al., 2025]; Finally, we also include pedestrian trajectory prediction baselines: SocialCircle [Wong et al., 2024], AdapTraj [Qian et al., 2024], and UniEdge [Li et al., 2025], which emphasize complex interaction modeling and multi-modal(diverse trajectory predictions) behavior representation.

Evaluation Metrics. We use four types of basic metrics, the same as MTP-GO [Westny et al., 2023]: ADE (Average Displacement Error), FDE (Final Displacement Error), MR (Miss Rate), and APDE (Average Path Displacement Error). For multi-modal K predictions, the minimum ADE (MinADE) and minimum FDE (MinFDE) are evaluated, which are equal to the deterministic ADE and FDE when K = 1.

Setting. In generalization experiments, we employ a unified set of model parameters for all datasets. Specifically, we configure the hyperparameter combinations {Nα, Kγ , Nd} as {16, 4, 6}. Our method is trained with the AdamW optimizer (learning rate 10-4), a batch size of 32, an early stopping patience of 8, and a maximum of 100 epochs on the training set. The cosine learning rate scheduler with warmup is employed to stabilize the early-stage training and gradually anneal the learning rate for improved convergence, utilizing the first 5% of training steps for linear warmup. All experiments are conducted on Intel Rapids-SCPCUs and NVIDIA A40 GPUs.

Single-Dataset Learning. We compare G-VTMwith other state-of-the-art baselines separately on three datasets. For fairness, all models are set to optimize and output a single deterministic trajectory. As shown in Table 1, G-VTM consistently outperforms most methods in all datasets. Notably, G-VTM achieves these performance gains without relying on diverse future trajectory sampling, highlighting the effectiveness of multimodal (vision and trajectory) semantic fusion.

Few-shot learning. We evaluate the few-shot generalized ability of G-VTM in cross-dataset groups. All models are pre-trained on the source dataset and then fine-tuned on the target dataset by randomly sampling 5% or 10% training data.

We compare G-VTM with baselines that predict a single deterministic trajectory. For fairness, G-VTM is trained to optimize and output the trajectory with the maximum predicted probability. As shown in Table 2, G-VTM consistently outperforms SOTA models in different groups of cross-dataset transfer, achieving an average improvement of 22.2% in ADE, 15.4% in FDE, and 12.6% in MR compared to second-best models with 5% training data. In the fewshot 10% setting, generalized performance further improves by 23.8%, 15.2%, and 13.1%, respectively. These results demonstrate that G-VTM not only predicts a more accurate future trajectory but also enables its classifier to consistently select the most accurate forecast among diverse predictions.

For baselines producing multi-modal K predictions, we optimize the predicted trajectory that has the minimum endpoint error with the ground-truth trajectory. As shown in Table 3, we use the minADE and minFDE as the evaluation metrics. Our method consistently outperforms SOTA baselines. On 5% training data with K = 5, G-VTM achieves average improvements of 11.2% in MinADE and 8.7% in MinFDE compared to the second-best baselines. Besieds, G-VTM benefits from increasing K, showing that it can effectively utilize additional multi-modal trajectory hypotheses rather than merely generating redundant predictions. For example, on {SinD → RounD} with 10% data, G-VTM improves MinADE by 37.4%, and MinFDE by 25.1% with K increased. This improvement is also evident in other crossdataset groups, which shows that G-VTM exhibits a higher generalization upper bound compared to other methods.

Zero-shot Learning. We evaluate the zero-shot generalized performance of G-VTM with KI-GAN and AdapTraj on several cross-dataset groups. With the similar prioritybased traffic regulation and different road topologies, G-VTM achieves an average improvement of 9.0% and 7.4% on {InD → RounD} and {RounD → InD}. On {InD → SinD} that has similar road topologies and different traffic regulations, KI-GAN, which is designed for the SinD dataset, achieves the best performance by explicitly encoding traffic signal states.

Ablation Study. We evaluate the contribution of eachmodule in G-VTM by systematically adding individual components. Results in Table 5 prove that all modules are critical to G-VTM. First, adding vision-related modules consistently improves performance under both in-dataset and crossdataset learning. In particular, excluding vision modality in G-VTM leads to a significant performance drop (ADE-30% in SinD-Tianjin and ADE-16% in SinD→RounD with 5% training data). Moreover, including the MoE module leads to a significant performance improvement, indicating that MoE is essential for modeling behaviour patterns of road users.

Analysis of the Vision-informed MoE. As shown in Figure 3, we analyze the experts activated in different traffic scenarios, including the signaled intersection and the roundabout. First, the distribution of activated experts varied across different junctions because of distinct traffic behavior patterns. Furthermore, the introduction of visual information enables learning scenario-adaptive behavior patterns. It promotes more experts to be activated, capturing diverse behavioral motions. Besides, rich scenario semantics are provided by the vision modality to correct expert weights, thereby guiding more accurate intention modeling.

Computation Study. G-VTMexhibits competitive computational efficiency with a total of 76.5M parameters, 13190 MiB of memory usage, and 1.268 s/iter of the average inference speed. This efficiency enables scalable deployment across diverse junctions, supporting effective generalization without incurring prohibitive computational overhead.

6 Conclusion

G-VTM is a pioneering exploration of multimodal techniques for generalized trajectory prediction in complex traffic scenarios. Extensive experiments demonstrate its state-of-the-art generalized performance across different junction datasets, especially in few-shot settings. G-VTM supports low-cost deployment and low-latency inference, making it well-suited for practical application in intelligent transportation and vehicle-infrastructure systems.

Ethical Statement

There are no ethical issues.

Acknowledgments

This work was supported by the Beijing Natural Science Foundation (No. L252034), Frontier Technologies R&D Program of Jiangsu (Grant No. BF2024052), and Chengdu Science and Technology Program (Grant No. 2025-YF0800097-GX).

Contribution Statement

The authors Xinyue Zhang and Letian Gong contributed equally to this research.