SocialMP: Learning Social Aware Motion Patterns via Additive Fusion for Pedestrian Trajectory Prediction

1 Introduction

Perceiving, analyzing, and predicting future motion patterns of pedestrians are crucial for applications such as autonomous driving, intelligent transportation systems, robot navigation, and surveillance systems [Wang et al., 2022; Huang et al., 2023; Chen et al., 2018; Quan et al., 2021]. Given the 2D

temporal coordinates of pedestrians within a scene, the task of trajectory prediction involves forecasting multiple plausible future trajectories that apply with social norms [Alahi et al., 2016]. Traditional methods [Gupta et al., 2018; Sadeghian et al., 2019] directly infer the pedestrian's 2D coordinates based on past motion states which not only introduces uncertainty in the social interactions between pedestrians but also lacks interpretability.

To enhance the explainability of models, recent studies adopt dimensionality reduction algorithms to extract lowdimensional motion patterns [Hug et al., 2020; Jazayeri and Jahangiri, 2022]. The key advantage of dimensionality reduction is its ability to capture interpretable motion patterns from available trajectories. For instance, considering some motion patterns of human, such as gradually slowing down to turn right or making a sharp turn while moving straight, some researchers introduce parametric curve functions [Huang et al., 2019; Hug et al., 2022] to capture explainable motion patterns. These approaches effectively reduce the dimensionality of trajectories by transforming sequential coordinates in the spatial domain into a condensed set of key points. Specifically for the trajectory prediction task, Bae et al. [Bae et al., 2023] employs a low-rank descriptor to approximate motion patterns from real-world trajectories. Nevertheless, this approach adopts singular value decompose algorithm which fails to model the complexity of interactions between pedestrians. Additionally, it overlooks the constraints imposed by the environment on pedestrian movement.

In this paper, departing from the current approaches that directly take dimensionality reduction approaches to represent interpretable motion patterns, we propose a novel framework to enhance the social awareness and environmental perception of motion patterns. The corresponding process of how pedestrians decrease the uncertainty of future trajectories from divergent to convergent is illustrated in Figure 1. The motion patterns of target pedestrian (blue clothes) are represented using dimensionality reduction approaches, which converges the predicted motion patterns in Figure 1 (a) to avoid the neighbor pedestrian. In contrary, Figure 1 (b) illustrates that the target pedestrian adopts visual field rules to model environmental social interaction which converges the predicted trajectory by perceiving the states of neighbor pedestrians and neighbor trees. Our model employs an attention-based additive fusion module integrating environmental social interaction into motion patterns to obtain socialaware motion patterns in Figure 1 (c) and we call that SocialMP. SocialMP can not only improve the prediction accuracy but also prevent the generation of future trajectory located in an unwalkable area. The main contributions are summarized as follows:

• We propose SocialMP representation for pedestrian trajectory prediction task which can effectively enhance the social awareness and environmental perception of motion patterns.
• We devise an attention-based additive fusion mechanism to capture the social awareness of motion patterns by integrating environmental social interaction into motion patterns.
• A map loss function is introduced to generate plausible trajectories by applying additional penalty to the average displacement error.
• Comprehensive experiments on two trajectory prediction datasets demonstrate that our approach achieves competitive results compared to the current state-of-theart methods.

2 RelatedWork

Pedestrian trajectory prediction methods can be broadly categorized into model-based and model-free approaches [Jiangbei et al., 2022].

Model-basedMethods. These approaches use mathematical rules to model pedestrian behavior, often based on assumptions about human motion. The Social Force model [Helbing and Molnar, 1998] applies Newtonian laws to describe pedestrian movement. Other variations [Luber et al., 2010; Pellegrini et al., 2009b] enhance this model to better capture pedestrian interactions. Some models incorporate additional techniques, such as context-aware transfers [Xia et al., 2022] or Neural Differential Equations [Jiangbei et al., 2022], to explain pedestrian behavior. However, these methods struggle to capture the full complexity of real-world scenarios and require additional rules as system scale increases.

Model-free Methods. These methods focus on learning from data to model temporal and spatial relationships between pedestrians. Techniques such as social pooling [Alahi et al., 2016] and generative models like CVAE [Ivanovic et al., 2020; Xu et al., 2024], GAN [Gupta et al., 2018; Sadeghian et al., 2019; Shilun et al., 2021], Attention-based models [Saadatnejad et al., 2024; Cheng et al., 2023; Messaoud et al., 2021; Zhang et al., 2023] and diffusion-based models [Bae et al., 2024; Mao et al., 2023; Gu et al., 2022] have been widely applied to capture spatio-temporal dependencies and pedestrian-environment interactions. While these models excel in data fitting, they often require specialized networks for multi-task prediction. Model-free approaches primarily emphasize fitting data through training deep neural networks, which usually lack the mathematical guidance that model-based methods inherently provide. To solve the above problem, Wong et al. [Wong et al., 2024a] adopt an anglebased rule to model neighbor pedestrians' velocity, distance and direction. In this year, they also take the occupancy map into consideration to avoid the obstacles [Wong et al., 2024b] . However, such methods consider all surrounding pedestrians and environmental information of the target pedestrian, which may lead to overfitting due to the limited field of view of humans. Unlike angle-based rules, to model environmental social interaction, we define visual field rules to simulate the real perspective of humans.

3 Methodology

Our method aims to enhance the social awareness of pedestrian motion patterns and reduce the uncertainty in pedestrian intentions. The corresponding pipeline is shown in Figure 2.

3.1 ProblemDefinition

The task of trajectory prediction is to forecast the future trajectories in a scene based on pedestrians' past trajectories. In this work, we denote the 2D trajectory coordinate as (x, y) and the scene map as M. Formally, we define N as the number of pedestrians in the scene and denote each pedestrian's history trajectory during Th observed timesteps as Hi = ((x1i , y1 i ), · · · , (xTh i , yTh i )). Correspondingly, the future ground truth trajectories are expressed as Fi = ((xTh+1 i , yTh+1 i ), · · · , (xTh+Tf i , yTh+Tf i )) during the future timesteps Tf . The trajectory prediction task can be formulated as follows: given the target pedestrian's past trajectories Hi and neighbor's past trajectories {H1, · · · ,Hj, · · · ,HN} where j ̸= i in the scene map M, the objective is to predict s plausible future trajectories of the target pedestrian, denoted as ˆ Fi .

3.2 Environmental Social Interaction

The extraction process of environmental social interaction is mainly divided into three steps. Firstly, we convert the scene segmentation map into a 100×100 pixel-level map. By determining the infeasible regions based on the pixel values of the segmentation map (0 or 255), we extract the coordinates of the infeasible areas. These coordinates are then transformed into real-world coordinates using a homography matrix, and these coordinates are treated as stationary pedestrians.

Next, we define visual field rules to model environmental social interaction. Treue [Treue, 2003] proposes the concept of visual attention and points out that humans exhibit varying degrees of attention from different perspectives. Inspired by these, we consider the limited field of view and attention span of humans and divide the surrounding area of the target pedestrian into three zones: (1) None Attention. This indicates that the pedestrian or obstacle is behind the target pedestrian and cannot be noticed. (2) Weak Attention. This indicates that the pedestrian or obstacle is in the side-front of the target pedestrian, which can be noticed through peripheral vision, but it has minimal impact on the pedestrian. (3) Strong Attention. This refers to direct attention that the pedestrian can consciously perceive, and it exerts a strong interactive influence. Based on these, We set the interaction impact coefficients α, β, and γ to constrain the influence. Specifically, we calculate the relative angle rel(i, j) between the direction of travel of the target pedestrian i and the surrounding pedestrian j. If the value is within the range from 2 3 π to 43 π, the target pedestrian is defined as None Attention, with a value of α; if the value is within the range from 13 π to 2 3 π or range from 43 π to 5 3 π, the pedestrian is defined as Weak Attention, with a value of β; if the value is between 0 and 13 π or between 53 π and 2π, the pedestrian is defined as Strong Attention, with a value of γ. The values of α, β, and γ are hyperparameters that can be adjusted to adapt to different scenarios. In this case, we set α = 0, β = 0.5, and γ = 1.

Lastly, we use visual field rules to model environmental social interaction. Given the trajectories of N pedestrians (including stationary pedestrians), the interactions between the target pedestrian and its neighbors are extracted by predefined rules. We use an angle-based approach to capture the relative motion of each pedestrian. The angle θij between pedestrian j and target i is computed as:

where θij represents the relative direction of pedestrian j to pedestrian i. The representations of velocity f ivlc , direction f idrt , and distance f idst are defined as:

Since the target pedestrian can not simultaneously focus on every surrounding pedestrian in all directions, we divide the vicinity of the target pedestrian into p angular partitions, and calculate the social interaction representations f p,i nei relative to the target pedestrian i in the direction of p. It is important to note that if no pedestrians are present within a specific directional area, the corresponding factors will be set to zero. Each partition's representation f p,i nei is computed using the following formula:

The social interaction features f i si for the target pedestrian i are computed by concatenating the representations from multiple angular partitions:

3.3 Motion Pattern Descriptor

We extract interpretable motion patterns by following Eigentrajectory [Bae et al., 2023], which uses singular value decomposition to represent motion patterns. The extraction process aligns with that described in the original work.

3.4 Additive Fusion

In the additive fusion module, we first embed the temporal observed trajectories into the original environmental social interaction fsi and motion patterns fm, enabling the fuser to extract spatio-temporal dependencies. Then an additive cross-attention mechanism is adopted to compute attention scores between environmental social interaction and motion patterns and result in social-awared motion patterns.

Temporal Embedding. Environmental social interaction capture the spatial interactive features at time step th, but do not inherently account for the sequence of observed trajectories. To address this, we embed the temporal observed trajectories into the social representation fsr to capture temporal dependencies. Motion patterns consist of r singular vectors obtained from the spatio-temporal data decomposition of a scene, encapsulating spatio-temporal information. We also embed the initial positions of pedestrians into the motion patterns to retain information about their initial positions. Formally:

where Xis represents the temporal environmental social interaction with a shape of (N, T, ds) and Xim denotes the temporal motion patterns with a shape of (N, T, dm). Hi and Hi [0] represent the observed trajectories and the original position of pedestrian i, respectively.

Cross Attention Mechanism. An attention-based mechanism is employed to integrate high-dimensional environmental social interaction f i si into low-dimensional motion patterns f im. We utilize cross attention to compute attention scores between environmental social interaction and motion patterns. The formula for calculating the attention scores of Xm on Xs is as follows:

where Q = XmWQ and K = V = XsWK. Through the multi-head cross-attention mechanism, a fused representation SF i is obtained and concatenated with the original representation. Mathematically,

where k is a hyperparameter and we empirically set k to 6.

3.5 Loss Function

SocialMP uses additional loss functions Lsmp and Lmap to train the baseline models. Specifically, to enhance the plausibility of the predicted endpoint, we first convert the predicted real-world coordinates of pedestrians into pixel-level coordinates on the segmentation map using the homography matrix provided by the dataset. Then we determine whether the predicted coordinate points lie within unwalkable areas by examining the pixel values of the scene segmentation map. This process is expressed as:

while the final predicted endpoint falls within unwalkable area, a penalty is applied to the model's loss function. The map loss function is defined as follows:

here, we use final displacement error for Lmap to provide an additional penalty when the model predicts an unreasonable endpoint.

For the loss function of motion patterns, we use its coefficient c to measure the degree of deviation, which is calculated as shown in the following equation:

The final loss is a combination of two loss functions. Formally,

where we empirically set λ1 and λ2 to 1.

4 Experiments

4.1 Experimental Setup

Datasets. The ETH [Pellegrini et al., 2009a] and UCY [Lerner et al., 2007] (ETH-UCY) dataset covers 1523 pedestrians' trajectories across five scenes: ETH, Hotel, Univ, Zara1 and Zara2. These trajectories are based on real-world coordinates captured by an aerial surveillance camera. Similar to the method [Gupta et al., 2018], we adopt the standard leave-one-out strategy for the training and evaluation. Stanford Drone Dataset (SDD) [Robicquet et al., 2016] has 60 drone videos, including 5243 pedestrians in eight different university campus scenes. Following previous work [Liang et al., 2020], we split 60% videos to train, 20% to validate, and 20% to test in pixel coordinates.

Metrics. We use minADE20/minFDE20 [Alahi et al., 2016; Gupta et al., 2018] metrics that the best Average/Final Displacement Error (ADE/FDE) over 20 generated trajectories to measure the prediction accuracy and endpoint accuracy respectively. Simultaneously, we also introduce the metric TCC/COL [Tao et al., 2020; Liu et al., 2021], therein The Temporal Correlation Coefficient measures the Pearson correlation coefficient of motion patterns, and the Collision Rate calculates the percentage of collision cases between pedestrians on the predicted trajectories.

Baseline models. We evaluate our approach against common baselines such as SGCN [Shi et al., 2021], STGCNN [Mohamed et al., 2020], and Implicit [Mohamed et al., 2022]. Other baseline methods are used to compare with the state-of-the-art (SOTA) performance.

4.2 Quantitative Analyses

Performance Evaluation of Representations. We evaluate some original models and the corresponding models (SMPbased) with the SocialMP on ETH-UCY and SDD. As illustrated in Table 1, SMP-based models decrease the values of ADE, FDE, COL and increase the value of TCC in most cases which demonstrates that the SocialMP representations exhibit superior performance compared to the baseline models. In particular, the results of ADE and FDE decrease by at least 36.2% and 45.2% compared to the model without representations. In terms of reliability measures, except for a slight performance degradation in COL, all other metrics' results are improved on the ETH-UCY and SDD datasets. The experimental results demonstrate that our SocialMP can effectively integrate social awareness into motion patterns to handle different scenarios.

Comparisons with State-of-the-Art Models. We compare some models inserting SocialMP with the state-of-theart methods on SDD and ETH-UCY respectively. Table 2 illustrates that SGCN with SoicalMP (SMP-SGCN) obtains competitive performance on SDD. In detail, SMP-SGCN achieves impressive predictive performance with 3.5% better ADE compared with SMEMO and has a performance decrease of only 1.5% in the FDE compared to the SMEMO model. In Table 3, compared with SMEMO, SMP-SGCN achieves better performance with 4.5% better ADE, 2.9% better FDE, 22.0% better TCC and 13.7% better COL. Specifically, SMP-SGCN achieves state-of-the-art performance on the ADE and FDE metrics and SMP-Implicit reached stateof-the-art results on the TCC metric. The performance on the COL of SMP-based models has slightly declined compared to these later methods, and the reason is that SocialMP emphasizes collision avoidance between pedestrians and obstacles, overlooking potential collisions between pedestrians. These results demonstrate that SocialMP-based models can achieve competitive results, and further reflect that the SocialMP can extract more effective representation of motion patterns.

4.3 Qualitative Analyses

Unimodal trajectories of all pedestrians on the scene of UNIV. In Figure 3, we visualize the effect of predicting all pedestrians simultaneously. Visual Field SMP-SGCN utilizes the hyperparameters α, β, and γ to adjust the intensity of attention to social interactions, controlling the degree of influence through three types of attention: none, weak, and strong. Pedestrians with none attention (gray lines) do not affect the motion patterns of the target pedestrian. Weak attention (orange lines) slightly influences the motion patterns of the target pedestrian, enabling the model to better simulate pedestrian movement without disrupting the target pedestrian's trajectory. Strong attention (red lines) places greater emphasis on social interactions, effectively preventing conflicts between the target pedestrian and neighboring pedestrians, resulting in more coordinated trajectories. By flexibly adjusting the weight of social interactions, Visual Field SMP-SGCN effectively avoids overfitting, captures the dynamic nature of social interactions, and produces predictions that are closer to the ground truth trajectory.

Multi-modal trajectories on ETH-UCY. In the top row of Figure 4, we visualize the predicted multi-modal trajectories from five sub-scenes on ETH-UCY dataset respectively, which involves different scenes with sparse crowds or dense crowds including turning right and going straight. The SMPImplicit predicts reasonable and accurate trajectories. Particularly, in the UNIV scene with high-density crowds, SMPImplicit learns the motion patterns of slowly turning right, while the motion patterns with quickly turning right predicted by Implicit deviate from the ground truth trajectory. In the ZARA1 and ZARA2 scenes, the multi-modal trajectories of target pedestrian do not pass through the building, cars and other unwalkable area. The reason is that the surrounding obstacle information is integrated into the SocialMP representations. These results demonstrated that our SocialMP representations can effectively handle complex scenarios and prevent the pedestrians from passing through the unwalkable area.

Unimodal trajectories on SDD. The unimodal trajectories on SDD presented in Figure 4 intuitively reflects the accuracy of SMP-Implicit. We can see that SMP-Implicit performs better accuracy than the Implicit. Specifically, in the scene of BOOKSTORE0, the predictions of SMP-Implicit initially deviate from the ground truth trajectory in order to avoid the trees. Then, the pedestrian adjusts the motion pattern to walk toward the final position and achieve better accuracy compared to the Implicit method.

4.4 Ablation Studies

Effectiveness of each component on SocialMP. We validate the effectiveness of each component of SocialMP and the quantification results on ETH-UCY are listed in Table 4. Therein, we compare "SI+SO+MP" (SocialMP) with its individual ("MP","SI") and combined components ("SI+MP", "SI+SO", "MP+SO") features on ETH-UCY datasets, where "Gain" represents the performance improvement results between the "SI+SO+MP" and the best of other features. Comparing "MP" with "SI+SO", we can see that using motion patterns as trajectory features generally yields better performance. This comparison also indicates that motion patterns have a superior representational expression compared to environmental social interaction. When comparing "SI+SO+MP" to "MP", SocialMP achieves an average improvement of 2.73% and 1.43% on ADE and FDE compared with MPbased models. Though, such improvements come at the price of a decline in part TCC and COL performances. The reason is that the model's training loss function is primarily depended on ADE and FDE, without incorporating TCC or COL metrics into the loss function. These results demonstrate that although individual environmental social interaction yields worse performance than solely relying on motion patterns for predictions, integrating environmental social interaction into motion patterns can still enhance the model's performance.

Validation of map loss function. To validate the effectiveness of map loss function, we visualized predicted trajectories of SMP-SGCN in the BOOKSTORE scene. As shown in Figure 5, in the first column, IO solely applies pedestrians'

coordinates without map and the predicted trajectories pass through the unwalkable area. In the middle column, IO +IM takes map IM into consideration based on IO model and the predicted trajectory gradually moves away from the unwalkable area. There still has a small portion of the predicted trajectory lies within the unwalkable area, and the reason is that the loss function encourages the model to generate trajectories that are as close as possible to the ground truth trajectory. In the last column, IO +IM +LM incorporates an extra map loss function LM to impose a penalty when the predictions of SMP-SGCN pass through the unwalkable area. The generated motion patterns of turning left are away from the unwalkable area, which demonstrates that the effectiveness of proposed map loss function.

5 Conclusion

In this work, we present SocialMP, an innovative representation for pedestrian trajectory prediction aiming to reduce the uncertainty in pedestrian interactions and enhance environmental perception ability. By incorporating environmental social interactions into motion patterns via an attentionbased additive fusion mechanism, SocialMP effectively models complex social behaviors and accounts for physical constraints. Additionally, the proposed map loss function ensures more plausible predictions by penalizing those trajectories that pass through unwalkable areas. Future research will explore to leverage multi-modal data, such as LiDAR or point clouds, to further enhance prediction accuracy.

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grant 62372415, and Outstanding Youth Science Fund of Henan Province under Grant 242300421050.

Contribution Statement

Tianci Gao and Yuzhen Zhang contributed equally to this work and are designated as co-first authors. Hang Guo provided critical review of the manuscript. Pei Lv served as the corresponding author and is responsible for all communications related to this manuscript.