G2LTraj: A Global-to-Local Generation Approach for Trajectory Prediction

1 Introduction

Accurate trajectory prediction is crucial in numerous applications like self-driving, enabling autonomous vehicles to make safe and reliable decisions. This task involves predicting the future trajectories of moving agents (e.g., vehicles and pedestrians) by learning their interactions within the scenarios. Recently, deep learning methods have come to dominate the field of trajectory prediction. These methods typically predict all future steps of an agent either simultaneously [Shi et al., 2021; Shi et al., 2022b; Jia et al., 2023b; Girgis et al., 2022; Li et al., 2021; Bae et al., 2023] or recursively [Yuan et al., 2021; Chen et al., 2021; Gu et al., 2022; Lee et al., 2022; Zhao andWildes, 2021]. Specifically, the simultaneous methods assume that the future steps of an agent are independent of each other and commonly generate all future steps at once. One branch of these methods [Aydemir et al., 2023; Bae and Jeon, 2023; Gu et al., 2021; Mangalam et al., 2021] leverages trajectory anchors or endpoints to generate the intermediate steps concurrently. While these methods have demonstrated promising performance enhancements, the constraints among future steps are ignored, potentially leading to kinematically infeasible predictions, as shown in Fig. 1(a).

†Corresponding author

On the other hand, to capture the temporal constraints among future steps, the recursive strategy utilizes previously predicted time steps to forecast the subsequent ones. However, this strategy suffers from the accumulated error throughout the recursive process [Bae and Jeon, 2023], with the error propagating from the initial step to the final step. To balance the accumulated error and the constraints among future steps, [Jia et al., 2023a] simultaneously generates all future steps and utilizes a recursive network for trajectory refinement. Nonetheless, during refinement, the accumulated error still propagates from the initial step to the final step.

To tackle the above issues, in this paper, we propose G2LTraj, a plug-and-play global-to-local generation approach for trajectory prediction. Specifically, we generate a series of global key steps that uniformly span the entire future time range, establishing the trajectory outline. We treat any two adjacent key steps as the head and the tail steps of a local section. By partitioning the entire trajectory into multiple sections, the accumulated error would not propagate beyond each section. Subsequently, the intermediate steps within each section are recursively generated to complete the whole trajectory. We illustrate the overview of our G2LTraj in Fig. 1(c) intuitively. During the global-to-local generation process, we identify three specific challenges requiring attention. First, during the global key step generation process, since all key steps are generated at once, there is a lack of constraints among them. To this end, we introduce the spatial constraints among key steps by regulating the positional difference. This enables G2LTraj to generate consistent trajectories that adhere to kinematics, as shown in Fig. 1(b). Second, during the local recursive generation process, if merely relying on the head and tail steps to generate the intermediate step, the model's access to past kinematic information is limited, resulting in fragile temporal constraints with past trajectories. This fragility possibly leads to plausible but kinematically infeasible intermediate steps. To maximize the temporal constraints, we integrate the agent features along with the position information of the head and tail steps when generating the intermediate step. By doing so, G2LTraj can enhance its understanding of local motion patterns. Third, trajectories from different agents can exhibit varying kinematics characteristics like distinct velocities. Thus, using a fixed granularity (i.e., the time interval between two adjacent key steps) might not be suitable for each trajectory, which aligns with previous findings [Bae and Jeon, 2023]. Distinguished from manually adjusting the granularity for all trajectories [Bae and Jeon, 2023], we design a selectable granularity strategy tailored for each trajectory. Specifically, we generate key steps at a fine granularity and then downsample these steps to form multiple groups of coarse-grained key steps. Next, by learning the confidence scores of these groups for each trajectory, we select the optimal group for them. Our contributions can be summarized as follows:

• we propose a global-to-local generation approach for trajectory prediction that mitigates the accumulated error and introduces the constraints among future steps.
• To enhance the kinematical feasibility, we propose the spatial constraints among key steps and reinforce the
• temporal constraints among the intermediate steps.
• To ensure the optimal granularity of key steps for each trajectory, we devise a selectable granularity strategy.
• Our G2LTraj demonstrates notable performance gains over seven existing trajectory predictors when evaluated on ETH [Pellegrini et al., 2009], UCY [Lerner et al., 2007] and nuScenes [Caesar et al., 2020]. Extensive experiment results validate its effectiveness.

2 RelatedWork

Accurate trajectory prediction plays a vital role in various applications such as self-driving. Recent state-of-the-art methods primarily rely on neural networks to forecast all future steps of an agent, either simultaneously or recursively.

Simultaneous Trajectory Predictors assume that the future steps of an agent are independent of each other and typically generate future steps concurrently [Jia et al., 2023b; Shi et al., 2021; Shi et al., 2022b; Girgis et al., 2022]. [Shi et al., 2022a] utilizes tree paths based on the observed trajectories to predict future trajectories in parallel. [Shi et al., 2021] and [Li et al., 2021] achieve simultaneous trajectory prediction by capturing both spatial and temporal clues. [Zhou et al., 2023] endeavors to achieve faster inference while maintaining effectiveness. Another branch [Bae and Jeon, 2023; Aydemir et al., 2023; Gu et al., 2021] initially learns the potential endpoints or anchor points of local trajectories and subsequently generates the intermediate steps simultaneously. For example, [Aydemir et al., 2023] employs the predicted endpoints to jointly predict the trajectories of all agents in the scene. [Ye et al., 2022] considers both spatial and temporal perturbations when predicting the endpoint of future trajectories. However, these methods commonly neglect the constraints among future steps, possibly resulting in predictions that are kinematically infeasible [Jia et al., 2023a].

Recursive Trajectory Predictors utilizes previously predicted time steps to generate the subsequent ones [Gu et al., 2022; Lee et al., 2022; Yuan et al., 2021; Chen et al., 2021; Zhao and Wildes, 2021; Salzmann et al., 2020; Xu et al., 2020]. [Marchetti et al., 2022] and [Zhao and Wildes, 2021] employ supplementary storage space to generate future trajectories sequentially. [Navarro and Oh, 2022] proposes guiding long-term time-series trajectory prediction in complex environments. [Gu et al., 2022] progressively discards area indeterminacy until reaching the desired trajectory. [Lee et al., 2022] leverages RNN-based Conditional Variational AutoEncoder (CVAE) to obtain the full trajectories. Moreover, [Jia et al., 2023a] exploits a recursive network for trajectory refinement while simultaneously generating all future steps. Despite capturing temporal constraints, these methods encounter the accumulated error throughout the recursive process, as the error propagates from the initial step to the final step.

To this end, we propose a global-to-local generation approach that mitigates the accumulated error and incorporates constraints among future steps.

3 Methodology

3.1 ProblemFormulation and Overview

In the field of trajectory prediction, we are provided with the observed trajectory coordinates of an interested agent a, denoted as P = {(xi, yi)}0i=-Tp+1 . Here, Tp represents the length of past steps. Naturally, in the given scenario, agent a interacts with various elements in the environment, including other agents and the map. For brevity, we denote these elements as S. Our work can generally build upon any encoderdecoder trajectory predictor, denoted as G. Following the encoder-decoder structure of G, the agent features of a can typically be derived byA = G (P, S) . (1)

Through A, the objective is to predict a's future trajectory coordinates F = {Fi}Tf i=1 , where Tf is the length of future steps. Fi = (xi, yi) denotes the coordinate of step ti.

The overview framework of our G2LTraj is depicted in Fig. 2. Specifically, we propose a global-to-local generation approach for trajectory prediction. In the example of Fig. 2, initially, we simultaneously generate the global key steps t1, t9 and so on with a granularity of 8. Subsequently, the intermediate step t5 is generated by utilizing t1 and t9. Next, t3 (t7) is generated by utilizing t1 and t5 (t5 and t9). This local recursion continues until all future steps are generated.

3.2 Global Key Step Generation

In this section, we generate key future time steps of agent a at once. Specifically, we set the granularity of key steps as L (i.e., the time interval between two adjacent key steps.) Subsequently, we determine the key steps necessary to cover the length of future steps uniformly. Let GL denote the group of key steps under the granularity L, which can be derived by

Here, we will generate a total of N + 1 key steps. Any two adjacent key steps are considered as the head and the tail steps of a local section. Consequently, we partition the entire trajectory into n sections. In this manner, we ensure that the accumulated error among future steps would not propagate beyond each section. Next, we predict the position coordinates ˆ Z ∈ R(N+1)×2 of these N + 1 key steps by

where ψL is a two-layer Multilayer-Perceptron (MLP). Let ˆ Z1+nL ∈ R2 denotes the corresponding predicted coordinate of step t1+nL, where n ∈ {0, 1, ..., N}.

Since the key steps are generated simultaneously, these steps currently lack interrelated constraints, potentially causing kinematically infeasible predictions, as shown in Fig. 1(a). To address this issue, we introduce spatial constraints among key steps by regulating their positional difference. The detailed spatial constraint loss LLs under the granularity L can be calculated by

where Lr denotes the regression loss, such as Mean Square Error (MSE) loss [Gu et al., 2021] or the Negative LogLikelihood (NLL) loss [Varadarajan et al., 2022]. F1+nL denotes the ground truth coordinate of step t1+nL. Our G2LTraj adopts the MSE loss according to the empirical performance. Notably, if Tf < 1+NL, the value F1+NL is not available. In such cases, we set the upper limit of n in Eq. (4) to N -2. As shown in Eq. (4), LLs endeavors to adjust inconsistent steps to a reasonable relative distance from adjacent steps. In this way, we ensure generate consistent trajectories that adhere to kinematic feasibility principles.

Algorithm 1 Global-to-local Generation Process

Input: The granularity of key steps L, the predicted key coordinates { ˆ Z1+iL}Ni=0 , two-layer MLPs {ϕl}l∈{L,L/2,...,2}, the position embedding {p∗}1+NL ∗=0 , and agent features A.

Output: A predicted future trajectory ˆ FL ∈ R(1+NL)×2 for agent a.

• 1: Let l ←L.
• 2: while l > 1 do
• 3: Calculate { ˆ Z1+(i+ 12 )l} NL l -1 i=0 by Eq. (6) in parallel with ϕl, ˆ Z1+il, p1+il, ˆ Z1+(i+1)l, p1+(i+1)l and A as input;
• 4: l ← l/2.
• 5: ▷ l denotes the current interval of existing adjacent predicted steps. Notably, we set L ∈ {2, 4, 8, ...} to ensure l/2 is an integer value continuously. When l = 1, it indicates that the entire trajectory coordinates of 1 + NL steps have already been generated.
• 6: end while

3.3 Local Recursive Generation

After generating the global key steps of agent a, the next phase is to fill in the intermediate steps within each local section. If merely relying on the head and tail steps to generate the intermediate step, the temporal constraints among them are fragile. To illustrate this issue intuitively, we enumerate three cases, as shown in Fig. 3. In these cases, the ground truths display similar t5 and t9 positions in the local coordinate system while exhibiting distinct positions for the intermediate step t7. Solely depending on t5 and t9 limits the model's access to historical kinematic information, resulting in delicate temporal constraints with past trajectories. Consequently, this manner possibly generates plausible but kinematically infeasible t7 which is inconsistent with past trajectories, as shown in the red oval in Fig. 3.

To alleviate this issue, we propose incorporating agent features in addition to the positional information of the head and tail steps. Specifically, agent features A encompass historical kinematics clues. By incorporating A, we can improve the temporal constraints with past trajectories. The improved constraints aim to reduce the plausible prediction range to the kinematically feasible range, as shown in the black oval in the Fig. 3. Besides, we follow [Zhou et al., 2021; Zeng et al., 2023] to employ the position embedding of each step in the time sequence, as it can provide the local temporal information. Next, given the predicted coordinates of the head step ti and the tail step tj, the coordinate of the intermediate step t(i+j)/2 can be derived by a projection layer

where pi ∈ RD and pj ∈ RD denote the position embeddings of steps ti and tj, respectively. Empirically, we project these elements in a simple yet effective manner, which can be calculated by

where ϕl is a two-layer MLP, l is the interval between ti and tj. Due to the changing interval, we utilize multiple ϕl functions that vary with the subscript l, instead of relying on a single MLP. || denotes the concatenation operation. Since the values of ˆ Zi and ˆ Zj may differ significantly with a large l, we project their dimension to align with that of the position embedding using two different weight matrices (i.e., Wh ∈ R2×D and Wt ∈ R2×D ). After finishing the globalto-local recursive generation process, we obtain a predicted trajectory of agent a, denoted as ˆ FL ∈ R(1+NL)×2 , as shown in Algorithm 1.

3.4 Selectable Granularity Strategy

Future Trajectories of different agents can exhibit diverse kinematics characteristics. Thus, employing a fixed granularity may not be appropriate for each trajectory. For instance, unusual behaviors like abrupt direction changes involve rapid shifts in velocity and trajectory. In such cases, a finer granularity might be more adequate as it can better capture sudden motion changes. Conversely, for usual behaviors such as moving straight uniformly, a coarser granularity might suffice, as it can enhance the temporal constraints from more distant steps. To this end, we propose a selectable granularity strategy that aims to be as suitable as possible for each trajectory. In the following, for the sake of brevity, we focus on discussing the cases where Tf = 2M + 1, M is an integer.

The selectable strategy begins by generating key steps at a fine granularity. In other words, we set L to 2 and obtain the key group GL=2 = {t1, t3, ..., tTf }, Subsequently, we downsample GL=2 to derive M - 1 groups of coarse-grained key steps, i.e., GL=4 = {t1, t5, ..., tTf }, ..., GL=2M = {t1, tTf }. Based on these groups, we follow the above procedure to generate M trajectories, denoted as { ˆ F2, ˆ F4, ..., ˆ F2M}. To select the optimal trajectory from these M trajectories, we learn their confidence scores ˆ C ∈ RM by

where ψc is a two-layer MLP, σ denotes a softmax function. ˆ Z1 and ˆ ZTf are the shared head and tail coordinates of these M trajectories. By incorporating them with A, the model can capture the overall trajectory outlines, thereby enhancing the evaluation of the M trajectories. Subsequently, we establish the ground truth for the confidence scores of { ˆ F2m}Mm=1 by

where ADE denotes the average L2 distance between each coordinate of the predicted trajectory and the ground truth. Let C denote {Cm}Mm=1 . The predicted confidence scores ˆ C can be optimized by

where Lr denotes the regression loss. Notably, during the inference process, we only generate the predicted trajectory corresponding to the highest confidence score through our global-to-local approach. This way can significantly reduce the inference time. The final loss function can be denoted as

where LG denotes the inherent loss of the existing predictor G, L2i s is the spatial constraint loss in Eq. (4), η1 and η2 are trade-off hyper-parameters.

4 Experiments

4.1 Experimental Setup

Datasets. We conducted experiments on three widely-used datasets: ETH [Pellegrini et al., 2009], UCY [Lerner et al., 2007] and nuScenes [Caesar et al., 2020]. ETH and UCY comprise five distinct real-world scenarios, (i.e., ETH and HOTEL from ETH; UNIV, ZARA1 and ZARA2 from UCY), where 1,536 pedestrians were recorded in the world coordinates. We employ the standard leave-one-out strategy [Bae et al., 2023; Alahi et al., 2016] for training and evaluation. The task in ETH and UCY is to predict the future 12 steps (4.8 seconds) based on the observed 8 steps (3.2 seconds). nuScenes is a large-scale trajectory dataset that includes both vehicles and pedestrians. The task in nuScenes is to predict the future 12 steps (6 seconds) given the observed 4 steps (2 seconds) and the High-Definition (HD) maps.

Baseline Predictors. We evaluate the effectiveness of our G2LTraj by incorporating it into seven state-of-the-art predictors: SGCN [Shi et al., 2021], STGCNN [Mohamed et al., 2020], PECNet [Mangalam et al., 2020], AgentFormer [Yuan et al., 2021], GraphTERN [Bae and Jeon, 2023], and DMRGCN [Bae and Jeon, 2021] for the ETH and UCY datasets, and AutoBot [Girgis et al., 2022] for the nuScenes datasets. Evaluation Metrics. Following the standard evaluation strategy [Bae et al., 2023; Zhao and Wildes, 2021], we assess the performance on ETH and UCY using the average displacement error(ADE) and the final displacement error(FDE). minADE5 and minADE10 represent the average pointwise L2 distances between the predicted trajectory and the ground truth for the 5 and 10 most likely predictions, respectively, minFDE1 denotes the L2 distance between the final points of the prediction and the ground truth of the most likely prediction. MR5 ( MR10 ) calculates the ratio of scenarios where none of the predicted 5 (10) trajectories are within 2 meters of the ground truth according to FDE. We use the best-ofK strategy to compute the final ADE and FDE with K=20, following prior works. We evaluate the results on nuScenes

validation set with K=10, following [Girgis et al., 2022].

Implementation Details. We set the position embedding dimension D as 64 in Sec. 3.3. For each trajectory, We select the optimal granularity from three groups, i.e., GL=2, GL=4 and GL=8. In Eq. (10), η1 and η2 are set as 0.1 and 1, respectively. For a fair comparison, we adopt the default configurations of the baseline predictors [Bae et al., 2023; Girgis et al., 2022]. Specifically, for ETH and UCY, we employ the AdamW optimizer [Loshchilov and Hutter, 2018], set the batch size to 128, use a learning rate of 0.001, and train for 256 epochs. For nuScenes, we utilize the Adam optimizer [Kingma and Ba, 2014], set the batch size to 64, use a learning rate of 7.5e-4 , and train for 80 epochs. The model is trained using a single 3090 GPU. Notably, in practice, the predictor G commonly generates K trajectories for each agent, aiming to model the multimodal distribution of future trajectories [Bae and Jeon, 2023; Girgis et al., 2022; Bae et al., 2023]. The global-to-local generation process of these trajectories is the same and in parallel.

4.2 Performance Comparison of Main Results

Performance Comparison on ETH and UCY. As shown in Table 1, our G2LTraj framework consistently outperforms all the baseline models in most cases. Notably, when compared to the simultaneous methods, our approach achieves substantial improvements in ADE and FDE, reaching up to 12.12% and 13.27%. Similarly, when compared to the recursive method AgentFormer, our approach demonstrates a significant enhancement in prediction accuracy. Our G2LTraj also has certain limitations, particularly when evaluating on the UNIV subset, using SGCN and STGCNN as the backbone predictors. Overall, these results highlight the effectiveness of our global-to-local generation approach.

Performance Comparison on nuScenes. As shown in Table 2, our G2LTraj framework continuously surpasses AutoBot in three benchmarks in most cases. For example, when utilizing the ensemble strategy, G2LTraj achieves a significant decrease of 3.65% and 12.6% in terms of minADE5

and minADE10. On the other hand, as for the MR5 metric, the performance of G2LTraj exhibits a slight improvement compared to the raw predictor. Notably, the ensemble strategy demonstrates the best performance across all metrics. whereas the model without map information exhibits the poorest performance.

4.3 Ablation Studies

We conduct several ablation studies based on the AutoBot predictor and evaluated on nuScenes, unless otherwise stated. Effectiveness for Mitigating the Accumulated Error. In this part, we validate the effectiveness for mitigating the accumulated error. Specifically, as shown in Table 3, compared to G2LTraj, G2LTraj-R exhibits inferior performance. We show the comparison results more intuitively in Fig. 4. Notably, G2LTraj-R outperforms our G2LTraj by a margin when the time step is less than 3. This presents the effectiveness of the RNN-based network in successfully incorporating temporal constraints for short-term trajectory prediction. As the time step increases, both approaches experience a growth in the accumulated error. However, G2LTraj exhibits a slower growth trend compared to G2LTraj-R. It demonstrates that G2LTraj effectively mitigates the accumulated error by partitioning the entire trajectory into multiple sections.

Effectiveness Analysis of Spatial Constraints. As shown in Table 4, without any spatial constraint, the model exhibits the lowest performance, resulting in reductions of 3.6% and 7.3% in terms of minADE5 and minADE10, respectively. Besides, the constraints applied at the fine granularity level outweigh those applied at the coarse granularity level. Both constraints at the fine and coarse granularity level contribute to performance gains.

Effectiveness Analysis of Temporal Constraints. As shown in Table 5, without the position embedding, the model performance slightly decreases. Without agent features providing the past kinematics information, the model suffers from a substantial performance reduction of approximately 3.13% in terms of minADE10. Only relying on the head and the tail steps, the model is limited to the fragile temporal constraints, resulting in the lowest performance.

Effectiveness Analysis of the Selectable Granularity Strategy. As shown in Table 6, employing three distinct fixed granularities consistently results in a reduction in performance, while the performance of G2 slightly surpasses that of G2LTraj. Overall, the performance across these three granularities surpasses that of the raw predictor. This finding confirms the efficacy of the global-to-local generation approach. Qualitative Results. As depicted in Fig. 5, the results clearly illustrate that G2LTraj greatly enhances the prediction performance compared to GraphTERN [Bae and Jeon, 2023]. Specifically, when spatial-temporal constraints are not enforced among future steps, the predicted trajectories from GraphTERN exhibit more pronounced fluctuations compared to G2LTraj. In contrast, the predicted trajectories generated by G2LTraj demonstrate greater consistency. This confirms that our G2LTraj tends to produce kinematically feasible predictions. Additionally, the predicted trajectories from our G2LTraj align more closely with the ground truth. Notably, both methods face challenges in accurately modeling unexpected behaviors, such as abrupt changes in direction, as shown in the black oval in Fig. 5. These failure cases generally involve rapid shifts in velocity and trajectory.

5 Conclusion

In this paper, we introduce a global-to-local generation approach for trajectory prediction that alleviates the accumulated error and introduces the constraints among future steps. To enhance the kinematical feasibility, we propose the spatial constraints among the global key steps and boost the temporal constraints among the local intermediate steps. To ensure the optimal granularity of key steps for each trajectory, we introduce a selectable granularity strategy. Our G2LTraj achieves noteworthy performance improvements over seven existing trajectory predictors when evaluated on three widely used datasets. Experiment results verify its effectiveness.

Acknowledgements

This work was supported in part by The National Nature Science Foundation of China (Grant No: 62303406, 62273302, 62273303, 62036009, 61936006), in part by Yongjiang Talent Introduction Programme (Grant No: 2023A-194-G, 2022A240-G).

Contribution Statement

Zhanwei Zhang and Zishuo Hua made equal contributions.