DifTraj: Diffusion Inspired by Intrinsic Intention and Extrinsic Interaction for Multi-Modal Trajectory Prediction

1 Introduction

Human trajectory prediction serves a significant role in vast applications, such as intelligent transportation system [Qiao et al., 2014; Teeti et al., 2022], human-robot system [Cheng et al., 2020], and surveillance system [Valera and Velastin, 2005]. It is a challenge to build an effective model for trajectory prediction since trajectories are inherently stochastic. A reasonable solution to the uncertainty in stochastic trajectories is to generate multiple trajectory distributions. Some previous works [Su et al., 2017; Gupta et al., 2018; Mangalam et al., 2020; Bae et al., 2022] have emerged great potentials from the generative models, such as generative adversarial network (GAN) [Goodfellow et al., 2014] and conditional variational auto-encoder (CVAE) [Sohn et al., 2015]. However, GAN often suffers from unstable training, and the prediction results of CVAE have significant biases from the actual trajectories. Recently, inspired by the success of diffusion model [Ho et al., 2020] in the field of image [Nichol and Dhariwal, 2021] and video [Kong et al., 2020], MID [Gu et al., 2022] utilizes the standard diffusion model to produce multiple trajectory distribution. Compared with other generative models, it obtains a more stable training process and reliable trajectories. To accelerate the inference process, LED [Mao et al., 2023] directly learns sophisticated trajectory distributions as the leapfrog initializer of the diffusion model. However, most existing methods only consider humans' extrinsic interaction, resulting in a limited set of typical trajectories that can only represent part of the range of possible outcomes.

* Corresponding author.

To achieve high-precision multi-modal trajectory prediction, we rethink the decision-making process of pedestrians and divide the factors into two aspects: internal and external. Firstly, pedestrians usually keep a walking goal in the mind, which acts on the entire body to control the direction and speed of movements. Essentially, it can be regarded as a purposeful behavior emitted by the pedestrian internally [Xu et al., 2018]. Moreover, external interactions between different entities in the environment, such as pedestrians avoiding collision or going together, can lead to unpredictable and complex trajectories [Shu et al., 2018]. For example, if a pedestrian falls abruptly, the following may also stop and offer assistance, which affects the followers' future movements. Jointly considering the impacts of intrinsic intention and extrinsic interaction to build a trajectory prediction model is still challenging, especially for modeling intention. Most existing methods do not explicitly model the intention but implicitly predict the entire future trajectory containing the intention estimation.

In this paper, we aim to propose a novel framework of multi-modal trajectory prediction to simulate the decisionmaking process of humans by utilizing intrinsic intentions and extrinsic interactions shown in Figure 1. Our framework contains three key components: interaction feature extracting, intention modeling, and trajectory inference. 1) Interaction feature extracting: We classify the influence of interactions into two aspects: temporal and spatial-wise. In detail, we propose a time-attention module to grasp the temporal dependence of each past timestamp and leverage graph attention to learn spatial interactions among different agents. 2) Intention modeling: Due to the difficulty in detecting intention signals inside the mind, we use the destination point to represent the latent intention. Specifically, we present a novel module based on attention for extracting the intention features from the observed trajectory. 3) Trajectory inference: Two modality features are fused to form an initially noised trajectory distribution. Then, a diffusion model will complete the denoising process from coarse to fine. Notably, our framework can generate multiple socially-acceptable trajectories close to realistic distribution by fully considering the intrinsic intention and extrinsic interaction. Furthermore, to improve the accuracy of noised-trajectory estimation and reduce denoising steps, we propose a new sample consistency loss to constrain multiple predictions by forcing all prediction samples to approach the ground truth. Sufficient experiments have been conducted on two public datasets, ETH-UCY and SDD, demonstrating that our method outperforms the previous SOTA methods by a large margin. For example, our test performance on ETH-UCY is improved by 19.0% and 24.2% in terms of ADE and FDE metrics, respectively.

Our contributions are summarised as follows.

• We propose a novel end-to-end paradigm based on the diffusion model for multiple trajectory prediction. By learning from the intrinsic intention and extrinsic interaction, our framework can reduce indeterminacy and predict socially-acceptable trajectories for humans.
• A novel intention module based on attention is presented to capture intention signals inside the human mind. This is the first work to explicitly build an intention model by destination points in human trajectory prediction.
• We introduce a loss constraint named sample consistency loss for multi-modal trajectory prediction to implicitly supervise the model's ability to learn diverse samples' distribution with low variance, meanwhile improving prediction performance.
• Extensive experiments have demonstrated that our proposed method achieves state-of-the-art performance on several human trajectory benchmarks, i.e., ETH-UCY

and SDD. Compared with the previous SOTA methods, the test performance is improved by 19.0%/24.2% regarding the ADE/FDE metrics on ETH-UCY.

2 Method

2.1 ProblemFormulation

Trajectory prediction constitutes a time-series regression task that forecasts the future movements of one or multiple agents based on observed movements. A commonly employed technique for mathematically representing trajectories involves sampling pedestrian data in video frames at a fixed frequency and obtaining a series of 2D trajectory points through coordinate transformation. Considering the trajectory of an agent i changing continuously with time, (xit , yi t ) indicates the position of the agent i. The set of past trajectory is defined as X = {Xi |Xi = (xit , yi t ), i ∈ (1, . . . ,N)} from time steps t = 1, . . . , tobs and the ground truth can be defined similarly as Y = {Y i |Y i = (xit , yi t ), i ∈ (1, . . . ,N)} from time steps t = tobs + 1, . . . , tpred, N is total number of pedestrian. Denote Ωi = (xit , yi t )|t=tpred as a destination point of pedestrian i, which is also referred to the intention of the current trajectory sequence. The predicted future trajectories of all agents are presented as ̂Y = [ ̂ Y 1 , . . . , ̂ Y N ]. Usually, different social behaviors may appear in different scenes, e.g., walking alone and merging into a group. In this work, we split the number of pedestrians appearing in different scenes to better model their interactions.

2.2 Framework Overview

As shown in Figure 2, DifTraj consists of three main parts: interaction feature extracting, intention modeling, and trajectory inference. Multi-modality features learned from humans' intrinsic intention and extrinsic interaction are utilized to guide the estimation of trajectory mean and variance, and then reparameterization techniques are applied to initialize the noised distribution. Finally, the denoising process from coarse to fine is conducted using the diffusion model.

2.3 Interaction Feature Extracting

Modeling human interaction is one essential technique in the trajectory prediction [Xu et al., 2022a; Makansi et al., 2022]. However, the interaction model in existing diffusion-based methods is inadequate in capturing temporal and spatial interaction features. For example, LED [Mao et al., 2023] employs a transformer-based module to capture social influences. However, this module only focuses on positional information in the sequence and cannot directly handle spatial relationships between agents.

Due to the interaction along the temporal and spatial dimensions, a comprehensive interaction influence should attempt to incorporate the temporal and spatial interaction features. We utilize LSTM [Hochreiter and Schmidhuber, 1997] to explicitly model the temporal correlations between interactions, where all cells in LSTM can share hidden states from diverse pedestrians. To grasp the temporal dependence of each past timestamp, we design a time-attention module corresponding to the channel-attention in the SE layer [Hu et al.,

2018]. This design ensures that the past trajectory of each timestamp is carefully encoded and measured. Subsequently, temporal interaction features are given different weights on time dimension. For more details, please refer to Figure ?? in the supplementary material.

where ϕ is the embedding function, Wemb is a learnable weight matrix. Temporal interaction features f tem ext are composed of the hidden states hit from the LSTM. The operator TA layer assigns different weights to the original position on the time dimension.

We feed the hidden states encoded by LSTM into the GAT [Veliˇckovi´c et al., 2017] module to aggregate information from neighbors by assigning different importance to different nodes. The number of pedestrians in each scene is counted, and we split and pack the number of pedestrians that appeared in the same scenes like seq = [start, end]. Consequently, GAT operates on a dynamic number of pedestrians and outputs the aggregated spatial interaction states from other neighbors in the same scene. Eventually, we complete the fusion of spatial and temporal information to form extrinsic interaction feature fext by concatenating f tem ext and f spa ext :

where spatial interaction features f spa ext are obtained by encoding eseq , hseq lstm and zseq tem with the GAT module.

2.4 IntentionModeling

In reality, humans are goal-driven [Tran et al., 2021], and the intrinsic intention can be represented as the end point of trajectories (i.e., destination). Modeling destinations will continuously provide more accurate and specific messages for future trajectory prediction. However, although the dataset provides the future ground truth (GT) of trajectories, the intention of the destination in the real world is often unavailable and is a part of the prediction. Therefore, the model is required to combine past observations to infer the distribution of pedestrian destinations. During the training phase, we first model past observations and raw destinations. Then, we set an attention-based module to improve the ability to map past observations into future intentions. The model can generate coarse destinations by only inputting past trajectories during the inference phase by training a destination decoder. The ground truth of the destination is utilized to supervise the training process of the decoder in Figure 2. The estimated destinations are obtained as follows:

However, this ambiguous intention information is difficult to interpret concretely. We have devised two encoders to model past observations and their corresponding estimated destinations. To simultaneously learn the coordinate information (x, y) from the past trajectory, the locations are reshaped into N × Tpast · 2 and then encoded by a learnable linear transformation to obtain embedded past features Eobs. The transformer-encoder layer is then utilized to further explore the context information Econ of embedded past features Eobs. Residual connection [He et al., 2016] incorporates above two hidden states and forms observation feature fobs.

The destination points are encoded by an MLP to get the destination feature fdes in Eq.(9), which is regarded as an implicit intention. The destination features of each independent agent are fused by cross combination to simulate the interaction between diverse destinations, as the intention of an agent may also be influenced by surrounding neighbors.

In fact, fobs and fdes are two independent variables, and they fail to reveal the influence of intrinsic intention. Additionally, we leverage the cross-attention mechanism to expose the intention feature fint, setting the destination feature as query Q and the observation feature as key K. Thus, the intention feature can be calculated as follows:

where WQ, WK and WV are three learnable weight matrices, the dimension of the former matrix is set the same as the destination feature fdes, and the dimension of the last two matrices are set the same as the observation feature fobs.

2.5 Trajectory Inference

Noised-distribution estimation. The standard diffusion model has demonstrated a solid ability to learn sophisticated distributions of human trajectory [Gu et al., 2022], but leading time-consuming denoising steps to generate high-quality samples. Inspired by LED [Mao et al., 2023], which proposed an initialized diffusion model to accelerate the inference speed, we build a new noised trajectory distribution as the initiation to accelerate the denoising process. Our method can capture humans' intrinsic and extrinsic motivations to estimate high-quality trajectories. We concatenate the intrinsic intention feature and the extrinsic interaction feature to form the multi-modality fusion feature ffusion, and set three learnable decoding modules to estimate the mean ˜ µθ, standard deviation ˜ σθ and normalized positions ˜ Pθ. K samples are generated as follows:

where MLP is selected as the decoder for small parameters and computational complexity, k indicates the number of samples for each prediction. To avoid learning a raw sophisticated distribution as mentioned in [Mao et al., 2023], reparameterization in Eq.(15) can be utilized to refine predicted trajectories.

Denoising procedure. Once we obtain the noised trajectories ˜Y, we utilize an attention-based module to encode the context of observations X and estimate the noise to reduce by a series of learnable linear-transformation operations. The recovered trajectories are presented as the following formula:

where ατ-1 and ¯ατ-1 = ∏τ-1 i=1 αi are parameters in standard diffusion model, β is a constant calculated by α, and z samples from the standard Gaussian distribution z ∼ N(0, I). ϵτ-1 θ is the noise estimated from noised trajectories ˜ Y τ with context embedding C and step τ .

2.6 Training Constraint

We design an end-to-end training strategy that collaboratively optimizes the intrinsic intention and extrinsic interaction feature encoders, including the denoising module. In LED [Mao et al., 2023], the authors divide the training process into two stages and leverage the pre-trained denoising module to train the leapfrog initializer in the second stage. Unlike LED, we believe encoding multi-modality features to generate noised trajectory distribution will continuously affect denoising. To maintain a balance between the mentioned two processes, we optimize the Kullback-Leibler Divergence (DKL) between ˜ µθ and µθ by calculating:

Moreover, we expect each sample to be close to the ground truth and leverage the marginal loss proposed in [Gupta et al., 2018]. Our method generates multiple predicted trajectories by stochastic sampling from the estimated distribution N (˜ µθ, ˜ σθI). The loss function employs L2 norm to minimize the distance between the prediction and the ground truth.

Inspired by [Weng et al., 2023], we introduce a variation of Lmarginal for joint training by swapping the order of taking the minimum over samples k and taking the accumulation over agents n.

Additionally, to train the destination decoder, we set Ldes with L2 norm as follows:

Sample consistency loss. The datasets of pedestrian trajectories are obtained from the specific scenes in reality. For socially acceptable prediction, the model needs to learn the sophisticated indeterminacy of trajectory distribution from the effect of intrinsic intention and extrinsic interaction.

The common multi-modal trajectory prediction approaches encourage the model to generate k potential samples. However, the consistency among each sample is not constrained, resulting in significant deviations between various samples generated under the same distribution. Increasing k is a solution to address large deviations [Gupta et al., 2018], but it raises computational complexity. Empirically, as the number of k gradually increases, the model generates samples as much as possible, so an excellent trajectory close to the ground truth will be identified in this pile of samples with a high probability. Essentially, the sampling program can be regarded as a process of entropy increment. To emphasize the consistency of multi-modal prediction, we propose sample consistency loss to constrain the deviations among different k samples in the trajectory inference phase for each agent. Training our framework combined with Lsc improves the model's fitting ability without additional computation cost.

where µ and ˜ µ are the mean values of the ground truth Y and predicted trajectories ̂Y, respectively. See more details about how sample consistency loss will affect trajectory prediction in Sec. 3.5.

Total loss. To form our training loss, we compose each part together and set different weight coefficients λ as follows:

3 Experiments

3.1 Datasets

The framework is trained and evaluated on two public benchmarks, including ETH-UCY [Pellegrini et al., 2010], and Stanford Drone Dataset [Robicquet et al., 2016].

ETH-UCY. Each dataset contains several different scenes, such as ETH and HOTEL in ETH, ZARA1, ZARA2 and UNIV in UCY. These subsets map from the real-world scenarios, including individual behaviors and group behaviors. The datasets are captured at 25 fps and annotated at 2.5 fps, and we use 3.2s (8 frames) as past input data to predict 4.8s (12 frames) future trajectories.

Stanford Drone Dataset (SDD). The dataset captures extensive human behaviors in campus from the bird's eye view. It covers no less than 40,000 human-scene interactions and 185,000 human-human interactions. Following previous works [Mangalam et al., 2020; Xu et al., 2022b], we use the standard train-test split and predict the future 4.8s (12 frames) using 3.2s (8 frames) past.

3.2 Implementation Details

In the intrinsic intention encoder, we initially use MLPs to learn the linear transformation of past observations and set the output dimension 256. We subsequently utilize a two-layer transformer-based module with head number 4 to encode past embeddings. In the extrinsic interaction encoder, the LSTM, in which the hidden dimension is set to 256, encodes temporal interaction features, and two graph attention layers with 4 heads are utilized to explore spatial interaction features. Then, the concatenate operation fuses a multi-modality feature with a dimension of 512. The decoders used MLPs to estimate the mean, standard deviation, and normalized positions with the same input dimension 1024. The number of samples K is set to 20 to generate multiple results. The model is trained by an AdamW optimizer [Loshchilov and Hutter, 2017] with a batch size 16 and an initial learning rate of 1 × 10-3 on a single 3090 GPU for 100 epochs. For the loss coefficient weights, we set λ1 = 5.0 and λ2 = 1.0.

3.3 Evaluation Protocol

We adopt two widely reported metrics to evaluate the prediction performance: the Average Displacement Error (ADE)

(a) Ablation experiments of the intrinsic intention and extrinsic interaction to trajectory prediction. minADE20 / minFDE20 (meters) are reported. (b) Ablations to study the influence of marginal metrics and joint metrics with the supervision of corresponding loss Lm and Lj .

Table 4: Ablations to study the influence of sample consistency loss Lsc. To comprehensively analyze how this constraint affects multiple-prediction performance, we include other ablative parts. Average ADE/FDE and JADE/JFDE are reported.

and the Final Displacement Error (FDE) [Pellegrini et al., 2009]. ADE measures the accuracy between the ground truth and predicted trajectory over all time steps. FDE measures the accuracy only at the predicted last point of the trajectory. However, a recent work [Weng et al., 2023] thoughtfully analyzes only marginal metrics (ADE and FDE) during evaluation, which will result in an overly optimistic estimation of performance, and the author proposes joint metrics (JADE and JFDE). We also include the joint loss term as Eq.(20) and report evaluation results of joint metrics in our experiments. See more details in the supplementary material.

3.4 Quantitative Results

In order to further present the effectiveness of our proposed method, experiments have been performed to compare with the previous state-of-the-art methods across different datasets (e.g., Social-GAN [Gupta et al., 2018], SoPhie [Sadeghian et al., 2019], PECNet [Mangalam et al., 2020], Trajectron++ [Salzmann et al., 2020], NMMP [Hu et al., 2020], MemoNet [Xu et al., 2022b], GroupNet [Xu et al., 2022a], MID [Gu et al., 2022] and LED [Mao et al., 2023]). Note that the results of LED are reproduced by us for fair comparison. We report two evaluation metrics, minADEK and minFDEK obtained from the Best-of-N [Gupta et al., 2018], which selects the closest sample from K generations to the ground truth and calculates the ADE/FDE metric on it.

ETH-UCY. Table 1 shows the evaluation results of our method compared with 9 previous prominent methods. We can drop that DifTraj outperforms those previously proposed models by observing the average values of metrics ADE and FDE, and achieves the state-of-the-art performance. Specifically, compared with the current state-of-the-art method LED, our method reduces ADE from 0.21 to 0.17 and FDE from 0.33 to 0.25, achieving a 19.0% and a 24.2% improvement, respectively. An incredible phenomenon is that our method achieves the best performance among all subsets.

SDD. Table 2 shows the evaluation results on SDD datasets of our method compared with 9 previous prominent methods. We set a limitation of the sample number K = 20 spreading around all methods. The results indicate that our method can achieve a superior reduction on ADE of 7.83 and FDE of 12.43, which outperforms LED with a 5.2% improvement on ADE and a 8.3% improvement on FDE.

3.5 Ablation Studies

Ablative experiments are all conducted on ETH-UCY shown in Table 3 and Table 4, results isolate the contribution of each part to the prediction performance. Note that the ablation study is not affected by the sample number K, we set default K = 20.

Effect of the intrinsic intention and extrinsic interaction. We conduct ablation studies to analyze how our method predicts more precise and socially-acceptable trajectories by learning from the intrinsic intention and extrinsic interaction as shown in Table 3a. Note that we only change the input feature types with the same dimension and keep other settings consistent. On the one hand, leveraging the past observations and destinations to encode the intrinsic intention features, the estimated trajectory distributions are seriously disturbed by noises with a high indeterminacy, which confuses the denoising process. The results on five subsets of ETH-UCY are poor. In contrast, by utilizing the extrinsic interaction features to estimate pseudo trajectories, we see that the results after the denoising process achieve good performances on all five subsets. We can drop that intrinsic intention and extrinsic interaction are significant to trajectory prediction.

Effect of training with joint metrics. As the author advocated for the use of joint metrics over marginal metrics in [Weng et al., 2023], we conduct ablations to explicitly study the influence of marginal metrics and joint metrics with the supervision of Lmarginal and/or Ljoint to train the model. Note that we utilize both the intrinsic intention and extrinsic interaction features. Intuitively, the joint loss focuses on multiple agent prediction, while the marginal loss cares for individual agent prediction. An interesting result in Table 3b is that the training incorporated with marginal loss Lmarginal and joint loss Ljoint leads to an improvement on both ADE/FDE and JADE/JFDE metrics. A reasonable explanation is that training with marginal and joint loss encourages the model to produce accurate predictions for each agent in any of its K samples and to predict realistic trajectories from multi-agent interaction. The patterns of future human behavior obtain complete preservation. We have attempted to prove that the latter is more challenging to converge than the former. See more details in the supplementary material.

Effect of the sample consistency loss. To comprehensively analyze how this constraint affects multiple-prediction performance, we perform an ablation experiment and include other ablative parts shown in Table 4. As we have indicated that training with marginal loss and joint loss improves the performance in the last ablation experiment, we set them as default settings in this experiment. We see that i) only inputting intention features with the supervision of sample consistency loss reduces ADE/FDE from 0.33/0.48 to 0.30/0.40, only inputting interaction features with the supervision of sample consistency loss reduces ADE/FDE from 0.21/0.34 to 0.19/0.30, and inputting both of them with the supervision of sample consistency loss reduces ADE/FDE from 0.19/0.31 to 0.17/0.25; ii) JADE and JFDE are both improved under conditions of Lsc supervision; and iii) our novel loss helps model learn the sophisticated indeterminacy of trajectory distribution from the intrinsic intention and extrinsic interaction, and improves the prediction performance.

3.6 Qualitative Results

To have a more intuitive understanding of the performance of our method, we visualize the best-predicted trajectory from

20 samples and compare ours with LED in Figure 3. The results show that ours and LED fit the ground truth well, but the latter performs unsatisfactorily in the subset of UNIV with extensive interactive behaviors. Our method produces more accurate and socially-acceptable predictions that are close to humans' real intentions.

4 Conclusion

In this paper, we analyze the conditions of multi-modal trajectory prediction from two objective perspectives and propose a novel end-to-end framework based on the diffusion model to predict more precise and socially-acceptable trajectories for humans. Firstly, a spatial-temporal aggression module is built to extract the extrinsic interaction features for capturing socially-acceptable behaviors. Secondly, we explicitly construct the intrinsic intention module to obtain intention features for precise prediction. Finally, we estimate a noise trajectory distribution with these two features as the initiation of the diffusion model and leverage the denoising process to obtain the final trajectories. Furthermore, to reduce the noise of the initiative trajectory estimation, we present a novel sample consistency loss to constrain multiple predictions. Extensive results indicate that our method has achieved state-ofthe-art performance on ETH-UCY and SDD datasets. In the future, we will try to extend our method to other sequence prediction tasks, such as visual tracking and video segmentation [Shen et al., 2021; Dong et al., 2023; Wu et al., 2023; Wu et al., 2022].

Acknowledgments

This work is partially funded by the Fundamental Research Funds for the Central Universities (Ref. No.: 2042022kf1198), the Science and Technology Development Fund (SKL-IOTSC(UM)-2024-2026) and the State Key Laboratory of Internet of Things for Smart City (University of Macau) (Ref. No.: SKL-IoTSC(UM)-20242026/ORP/GA04/2023), and Key Research and Development Project of Hubei Province (2022BAD175, 2022BCA009).