Vision-based Intention and Trajectory Prediction in Autonomous Vehicles: A Survey

1 Introduction

Prediction is about forecasting future events before they occur, which is beneficial for human-machine interaction systems, including surgical and industrial robots and autonomous vehicles (AVs). This survey will focus on prediction in autonomous vehicles. In this domain, prediction refers to the ability of the AV to forecast either the intention of other road users (e.g., they will turn left, right), including other cars, pedestrians and cyclists, or their future trajectory (when using video as input, their location in the image or map plane over a horizon of N future video frames). The accurate prediction of the intention and/or the trajectory of other road users can enable the 'planning' module of the autonomous (ego) vehicle or assistive system to avoid paths potentially leading to accidents and propose safe and reliable courses of action. Furthermore, it can foster social intelligence, as the first step towards understanding the thinking of other road users. For all these reasons, many companies have recently started to create a dedicated pipeline for AV prediction.

Multiple approaches have been suggested to tackle the prediction problem, including both physics-based models and machine learning-based models. The former depends on dynamic equations that govern the vehicle's motion to create an evolution model that predicts the final point and time of the current motion [Br¨annstr¨om et al., 2010]. Such models do not capture contextual information of the scene; therefore, they cannot capture the high-level intentions of road users, and their ability to model the uncertainty of future trajectories is limited [Lin et al., 2000]. Machine learning approaches, on the other hand, can distil high-level understanding from driving scenes by learning from data. They have arguably proven more successful in handling the above challenges in terms of accuracy and generalisation [Rasouli et al., 2019], as they can model both temporal dynamics and the social features of road users and produce suitably multi-modal predictions [Yuan et al., 2021].

Machine Learning (ML)-based prediction methods have explored different approaches to the problem. Some focus on predicting LiDAR point clouds in the future frames [Weng et al., 2020], while others predict occupancy maps [Hoermann et al., 2018]. This survey will focus on ML-based methods outputting visual predictions for road agents using input camera frames, either from an ego-vehicle perspective or from a bird's eye viewpoint (BEV).

Isolated surveys do exist on both the general prediction problem [Rasouli, 2020] and prediction in AVs [Mozaffari et al., 2020]. The latter classifies prediction models based on input, method used and output, doing a good job highlighting the pros and cons of each class. However, neither papers systematically compares the contributions of the models' constituents blocks in tackling different aspects of the prediction problem. Also, existing surveys only list datasets without a detailed comparison between their attributes, failing to indicate which dataset to use to tackle a particular problem.

To the best of the authors' knowledge, this is the first survey to provide such an in-depth analysis of prediction models and datasets. Its main contributions are: (1) a formulation of a generic AV prediction pipeline; (2) a critical review of the ability of the various models' building blocks to model the social, temporal and generative dimensions of the prediction task, as a guide for researchers to select the correct ML technique for their problem; (3) a comparison of the relevant datasets in terms of size, input modalities, target agent and problem; (4) a summary of the latest methodological trends and a reasoned analysis of future research directions.

The survey is structured as follows. Section 2 introduces the prediction pipeline and formalises the prediction problem for both intention and trajectory. Section 3 highlights the main challenges of the prediction task, while Section 4 compares the most popular prediction datasets and lists the associated performance metrics. State-of-the-art (SOTA) prediction approaches are recalled in Tables in Section 5. Based on that, Section 6 focuses on the current research trends and comments on the efficacy of different ML techniques. Research gaps and future directions are discussed in Section 7.

2 ProblemFormulation

Hereafter, the vehicle on which the sensors are mounted is called the ego-vehicle, while all the other road agents (including vehicles, pedestrians, cyclists and traffic lights) are termed target agents, as the prediction model aims to anticipate their future behaviour.

The input to the task is, given a current time instant, a sequence of features X = {Xa t }t=1:Tobs;a=1:A related to A target agents extracted over the previous Tobs time instants. These features might include the target agent's location (either local, in image plane, or global, from a BEV), velocity, heading angle, or pose, as well as context information I in the form of RGB images, LiDAR point clouds, HD maps, semantic segmentation maps, etc. capturing road structure, traffic conditions or other environmental factors.

The problem is to calculate one or both of the following.

1. Intention. A set of K possible future intentions for the A agents in the next Tpred (future) frames, each with an associated probability. In most cases this is considered a classification problem, since intention is assumed to be annotated, either as a mind state or as a future action.
2. Trajectory. A set of K possible future locations for the A agents in the next Tpred future frames, formally S = {Sa k }k=1:K,a=1:A where each Sa k contains a trajectory s = {st}t=1:Tpred , and a scalar confidence c which captures the probability of this trajectory. Trajectory coordinates are measured either in the image plane or in the map plan (BEV).

To address this problem, a typical prediction pipeline will encode the input related to the agents in the scene, to then model their temporal and social dimensions before decoding their uni-modal or multi-modal intention or trajectory. Figure 1 shows the workflow of a typical AV prediction stack.

3 Challenges

Prediction in the AV domain is a complex problem due to the following characteristics of the driving environment.

1. Dynamic: Both the ego-vehicle (observer) and the target agents are moving. Therefore, the future trajectory of the target agent depends on its motion as well as the ego-vehicle's motion. Furthermore, in a dynamic environment the captured data is available sequentially, requiring that the model makes predictions in an online, rather than batch, fashion.
2. Multi-agent: Being multi-agent environments, roads host agents characterised by different goals and features. In particular, pedestrians possess more diverse and unpredictable trajectories than cars, which generally operate within a finite number of predefined lanes in manners restricted by road rules and geometry. Furthermore, the actions of one agent may influence the actions of other agents and vice versa in a social interaction process.
3. Stochastic: There is inherent uncertainty about a user's future intention or trajectory and the multimodality of agents' behaviours, as one past trajectory can have multiple possible future trajectories.
4. Partially observable: The surrounding environment and target agents are only partially observable by the ego vehicle due to occlusions etc; that is, the location, speed, heading angle, latent beliefs of other agents are not accessible/known by the ego-vehicle at all times.
5. Real-time requirement: In such a critical environment in which agents are moving at relatively high speeds, prediction algorithms need to perform in real-time, allowing time for the ego vehicle to react. This, in turn, increases the computational burden on the on-board PC.

In the light of the challenges mentioned above, an ideal prediction algorithmmust model the social and temporal features of the ego vehicle and its surrounding road users (including cars and pedestrians) in an online processing fashion. Furthermore, it must generate multiple future trajectories (each with an associated probability), while running in real-time and assuming partial observability.

4 Training and Evaluation

Prediction approaches are tested on various standard datasets, and their performance evaluated using suitable metrics.

4.1 Datasets

While older datasets were limited in terms of environments and agent categories, modern benchmarks have significantly boosted progress in the AV prediction field.

For example, NGSIM-180 [Coifman and Li, 2017] and the highD Dataset [Krajewski et al., 2018] used drones and surveillance cameras to capture cars on highways, featuring a single type of agent with a limited set of possible actions (move left/right and keep straight). On the other hand, ETH [Pellegrini et al., 2009] contemplated trajectory prediction for off-road pedestrians only. PIE [Rasouli et al., 2019], and its predecessor JAAD [Kotseruba et al., 2016] also targeted the trajectories of pedestrians only. Unlike ETH, they used an ego-vehicle camera to capture the scenes, and added an intention label for crossing or not crossing, allowing the dataset to be used for intention and trajectory prediction. All such datasets would capture the scene using camera sensor only.

KITTI [Geiger et al., 2012] was one of the first multimodality datasets to provide, in addition to camera frames, LiDAR pointclouds for the input scenes. This, in turn, gave rise to the recent interest in detecting objects using 3D bounding boxes [Chen et al., 2017]. Furthermore, KITTI provides annotations for both cars and pedestrians.

The deeper the AI model, the more images it needs to generalise efficiently. Recent datasets such as Lyft [Houston et al., 2020], Waymo [Ettinger et al., 2021], nuScenes [Caesar et al., 2020] and Argoverse [Wilson et al., 2021] have brought about a significant increase in terms of number of annotated frames, paving the road for the training of deep models. In addition to camera and LiDAR, these datasets provide High Definition (HD) maps that capture the topology of the road [Seif and Hu, 2016]. Adding HD maps has made possible to investigate global navigation abilities, enabling in turn the training of models upon a longer prediction horizon. Furthermore, unlike previous datasets, the datasets listed above contemplate more classes, recorded ego-vehicle odometry data, a variety of different cities, various weather and lighting conditions (including rain and night), and labels for other agents including traffic lights and road rules. Nevertheless, they still lack intention prediction-related labels.

LOKI [Girase et al., 2021] has addressed this problem by providing action labels that can be harnessed by intention prediction models. Furthermore, the creators considered additional attributes that may influence the prediction model (e.g. age and weather). E.g., the probability of a pedestrian crossing the street not on a zebra crossing differs if the pedestrian is a teenager or an elderly. Finally, LOKI has an average of 21.6 agents per frame, useful for multi-agent prediction.

In summary, modern datasets helped addressing most of the prediction challenges by providing a huge amount of diverse, multi-agent, multi-modal data which can be used to train models able to predict the behaviour of different types of interacting agents in different weather conditions. Moreover, they provide annotations useful for high-level understanding of the driving scene including location, action, events. Commonly used datasets are outlined in Table 1.

4.2 EvaluationMetrics

Intention prediction. As a classification task, it is evaluated using traditional classification metrics including accuracy, i.e. the percentage of correctly classified samples over the total number of samples. Since it deals only with true positives, accuracy does not quite reflect the true performance in case of class-imbalance. The model will learn to focus on the more populated class in order to output many true positives, while generating a large number of false predictions for the other classes. Therefore, it is recommended to use other complimentary metrics including precision and recall, which consider the false positives and the false negatives, respectively. Precision and recall can also be utilised to calculate other metrics such as the F1 score, mean Average Precision

(mAP) and the Area Under the Curve (AUC) [Rasouli, 2020].

Trajectory prediction. It is considered a regression problem in which the output is either a sequence of bounding boxes around the object of interest or its x - y locations. The Average Displacement Error (ADE) is widely used to evaluate the output. It is the mean l2 distance between all the locations forming the predicted trajectory and the corresponding ground truth. Another common metric is the Final Displacement Error (FDE), which performs the same calculation but only for the final predicted location in the trajectory and its ground truth at the prediction horizon Tpred.

Models which use probabilistic generative methods to generate multi-modal prediction employ additional metrics. Best of N calculates ADE and FDE for the best N samples out of all the generated trajectories. When N equals 1, only the generated trajectory closest to the ground truth is chosen: this method is called minADE and minFDE, respectively. Other generative model-related measures are the different versions of Negative Log Likelihood (NLL), comparing the distribution of the generated trajectories against the ground truth.

5 Overview of the Recent Literature

Behaviour prediction can be either implicit, in the form of future trajectories, or explicit in terms of predicting future actions or events. An agent's intentions can be affected by: a) the agent's own belief or will (challenging to model, since it is usually not observed); b) its social interactions, which can be modelled using different approaches like social pooling, Graph Neural Networks, Attention, etc.; c) environmental constraints, such as the road layout, which can be encoded by High Definition (HD) maps; d) contextual information, in the form of RGB frames, LiDAR point clouds, optical flow, segmentation maps, etc.

On the other hand trajectory prediction is inherently more challenging - unlike intention (which is typically considered as a classification (discrete) problem), trajectory prediction is a regression (continuous) problem. Table 2 outlines state-ofthe-art models targeting intention prediction (first three rows), trajectory prediction (remainder of the table), and prediction of both intention and trajectory (last three rows). The table analyses the type of input modalities, along with their encoding techniques. It then compares different methods used to model the social, temporal, and generative dimensions of the prediction problem. It also contrasts various loss functions, optimisation methods, and datasets used to train the models, before listing their output type and measured metrics.

The different classes of techniques used by the methods listed in the above tables are discussed and compared next to highlight the emerging trends in AV prediction.

6 Take-home Messages

6.1 Ego-Motion and Dynamics

In order to tackle the aforementioned (Section 3) challenge that accompanies a dynamic environment, methods have included a dedicated processing unit to model the ego-vehicle motion [Kotseruba et al., 2021; Neumann and Vedaldi, 2021; Rasouli et al., 2021; Salzmann et al., 2020] to generate more accurate trajectories. In opposition, some methods model the target agents' motion, using either deep networks or dynamical models. [Cai et al., 2020; Neumann and Vedaldi, 2021; Salzmann et al., 2020] use dynamic equations at different levels of their algorithms to constrain the generated trajectory, making sure that they are dynamically feasible. In a related effort, researchers have leveraged additional quantities computed from the inputs directly provided by the dataset, such as pose, optic flow, semantic maps and heat maps.

6.2 Ego-Camera vs Bird's Eye View

In terms of the view in which predictions are formulated, methods can be categorised into two classes. In BEV methods the algorithm processes data coming from a top, maplike view [Salzmann et al., 2020; Varadarajan et al., 2021]. In ego-camera prediction, the algorithm perceives the world from the ego-vehicle perspective [Neumann and Vedaldi, 2021; Bhattacharyya et al., 2021]. The latter is generally more challenging than the former due to a number of reasons.

Firstly, perceiving the world from a BEV allows for a wider field of view and more informative predictions. By contrast, ego-camera views have a shorter visual range that limits the prediction horizon, because the car cannot plan further than it sees. Also, ego-cameras are occluded more commonly than BEVs. Thus, BEV methods suffer less from the 'partial-observability' challenge than those based on an egocamera. Secondly, unless LiDAR data is available, monocular vision makes it difficult for the algorithm to infer target agents' depth - which is an important clue in predicting their behaviour. Finally, as mentioned above, the ego-camera is moving - which requires processing both target agent's motion and the ego-vehicle motion, unlike the static BEV.

Despite inherent challenges in processing an ego-camera view, it remains more practical than BEV. Cars rarely have access to a camera showing BEVs of the road and target agents' locations. The conclusion is that a prediction system should be capable of viewing the world from the ego-vehicle's perspective. It may be advantageous to include LiDAR and / or stereoscopic camera data to perceive the world in 3D.

Another important related point is that, whenever the target agent's location has to be included for prediction, it is desirable to employ bounding box locations rather than purely centre points, as the former coordinates implicitly capture the changes in relative distance between ego-vehicle and pedestrians as well as the camera ego-motion [Rasouli et al., 2019; Rasouli et al., 2021]. In other words, bounding boxes become larger as the agent approaches the ego-vehicle, providing an additional (albeit rudimentary) estimation of depth.

6.3 Temporal Encoding

Since the driving environment is dynamic with many moving agents, it is critical to encode the temporal dimension of the agents to build a better prediction system connecting what happened in the past to what will happen in the future through the present. Knowing where an agent is coming from can help in guessing where it may go next. Most ego-camera-based models [Fang et al., 2021; Kotseruba et al., 2021] deal with shorter time horizons, and use 3D-Convolutional Neural Networks 3D-CNN models to capture the temporal dimension of video data. 3D-CNNs stack the 2D video frames along the third dimension (time), then apply 3D convolutions. This can capture temporal evolution for short-horizon tasks such as intention, in which the last few frames before the event are the most relevant and important. Other works use Hidden Markov Model (HMM) [Cai et al., 2020]. For longer horizons, however, a more complex structure is arguably required including Transformers [Yuan et al., 2021] and variants of Recurrent Neural Networks (RNNs) [Girase et al., 2021] such as Long Short-term Memory (LSTMs) [Bhattacharyya et al., 2021; Fang et al., 2021; Kosaraju et al., 2019; Rasouli et al., 2021; Salzmann et al., 2020; Weng et al., 2021; Yuan et al., 2021] and Gated Recurrent Units (GRUs) [Gilles et al., 2021]. It is noted that Transformer- and RNN-based models outperformed others in achieving SOTA results.

6.4 Social Encoding

To tackle the 'multi-agent' challenge, most top-performing algorithms use different types of graph neural networks (GNNs) to encode the social interaction between agents. [Gilles et al., 2021; Kosaraju et al., 2019] use a fully connected undirected graph in which all agents in the scene are connected. This method is clearly exponential in the number of connections. In opposition, [Fang et al., 2021; Girase et al., 2021; Salzmann et al., 2020; Weng et al., 2021] use a sparsely connected graph so that only agents that are located within a certain distance to each other are connected together - significantly lowering the number of links. Similarly, [Kosaraju et al., 2019; Yuan et al., 2021] implement sparse graphs, assuming that different types of agents have different perception ranges - and thus making the connections in the graph directional. The unique feature of [Girase et al., 2021] is its use of a heterogeneous graph that does not only include road agents but also significant road elements (e.g. exit, entrance) thus modelling the relationship between agents and environment. As for the optimal number of GNN layers, [Addanki et al., 2021] suggest that the deeper the graph the better the performance - in stark contrast, [Weng et al., 2021] and [Liu et al., 2020] claim that the optimal number is just 2.

[Rasouli et al., 2021; Mangalam et al., 2021] use semantic segmentation to extract the visual features of different classes, then find the relationship between them using attention. Unless panoptic segmentation is used (which distinguishes between semantic instances), all surrounding agents of the same class are treated the same whether they are interacting with the ego-vehicle or not. A less efficient method is social pooling, which was used along with attention in [Pang et al., 2021]. [Bhattacharyya et al., 2021] model the social dimension in the latent space of each agent, conditioned on that of previous agents. The limitation of this method is that it uses a human-biased ordering to decide on which agent the modelled agent depends, which may be incorrect.

Most approaches encode the temporal and social dimension separately - either starting with the temporal aspect to then consider the social one [Gilles et al., 2021; Girase et al., 2021; Kosaraju et al., 2019; Rasouli et al., 2021; Weng et al., 2021], or by doing the opposite [Salzmann et al., 2020]. [Yuan et al., 2021] argues that such systems are bound to be suboptimal, and proposed a transformer-based model that can simultaneously encode both dimensions.

6.5 Goal-Conditioning

Studies in neuroscience [Valentin et al., 2007] and computer vision [Gilles et al., 2021; Mangalam et al., 2021] suggest that humans are goal-directed agents. Furthermore, while making decisions, humans follow a series of successive levels of reasoning that eventually create their short or long-term plans. Based upon this, various algorithms have followed a strategy that appears to be beneficial to prediction performance: breaking down the prediction problem into two sub problems. The first is termed epistemic, and seeks the goals of each agent or answers the question 'where are the agents going?' The second is called aleatoric and solves for the trajectory that will carry the agent to the calculated goal, answering the question 'how is this agent going to reach its goal?' Methods may do that explicitly [Girase et al., 2021; Mangalam et al., 2021; Pang et al., 2021] or implicitly [Gilles et al., 2021], often conditioning the generated trajectories on the calculated goals. Those methods have achieved SOTA results in different driving datasets, reflecting the importance of goal-conditioning in improving prediction performance.

6.6 Multimodality

As the road environment is stochastic, one prior trajectory can unfold different future trajectories. Thus, a practical prediction system, that addresses 'stocasticity' challenge, is one that models the uncertainty of the problem. Conditional variational auto-encoders (CVAEs) achieve this by learning the latent space for the agents, then sampling from them during inference. [Girase et al., 2021; Weng et al., 2021; Yuan et al., 2021] model agents independently and use the variational lower bound of the log-likelihood (ELBO) function, while [Bhattacharyya et al., 2021] employ a CVAE to jointly model agents using a β-VAE loss function, arguing that ELBO suffers from posterior collapse. [Salzmann et al., 2020] also model agents independently using a CVAE, but use an infoVAE loss instead. In their ablation studies, however, none of these authors investigate the effect of using different CVAE loss functions on the final prediction.

[Kosaraju et al., 2019] use a generative adversarial network (GAN)-based method that uses adversarial losses for multimodal prediction. [Pang et al., 2021], instead, use a latent belief energy-based method that is close to generative adversarial imitation learning (GAIL) - arguing that GAIL is preferable to inverse reinforcement learning (which is from the same family of models). The latter calculates an agent's policy through a two loop process (an outer one for the cost and an inner one for the policy, which is highly computationally expensive) while the former directly optimises a policy network [Pang et al., 2021]. Interestingly, other authors have created datasets contemplating multiple future trajectories for a same observed trajectory [Liang et al., 2020].

An important observation is that, in the current literature, multimodality is only applied to trajectories, completely neglecting its potential for intention prediction, despite the existence of ways to model the latent space of a discrete variable.

A final shared trend among prediction systems is the implementation of attention mechanisms to calculate the weighted importance of features or input modalities [Fang et al., 2021; Gilles et al., 2021; Girase et al., 2021; Kosaraju et al., 2019; Kotseruba et al., 2021; Pang et al., 2021; Rasouli et al., 2021].

7 Research Gaps and Future Directions

Most researchers did not address the 'real-time' challenge, and did not touch on the computational demand of their systems. For the minority who did, including [Salzmann et al., 2020], their systemwas not running in real-time. Therefore, a promising future direction is to design online prediction systems.

Several successful multi-modal approaches to trajectory prediction using different generative losses have been brought forward. However, the ideal loss function is still to be identified as none of the existing studies have performed suitable ablation studies in this sense. Further, all existing multimodal systems exclusively deal with trajectory prediction, leaving as a clear future research direction the creation of systems for joint multi-modal intent and trajectory prediction.

Current state of the art algorithms struggle to make accurate predictions in the presence of scenes different from those they were trained upon. This is an inherent limitation of prediction methods that learn from examples. A promising (yet unexplored) approach to modelling the social interaction of road users, and directly generalising goaldirected methods, is theory of mind (ToM) [Singh et al., 2021; Cuzzolin et al., 2020]. A notion from cognitive psychology, ToM is the ability to understand and anticipate other agents by simulating their mental states. Applied to autonomous driving, ToM could allow an ego vehicle to think from the perspective of the target vehicle to produce more accurate predictions of its future behaviour, especially in rare situations or cases in which humans change course without visible cues.

Finally, future prediction methods should not be limited by the availability of annotation. The AV industry is generating huge amount of visual data, which can only be leveraged by moving toward self-supervised learning methods.

8 Conclusions

The ability to accurately predict intentions and trajectories of other road agents can significantly enhance AV capabilities. Recent developments leveraging large, diverse and multi-modal datasets have significantly boosted research in this area. SOTA prediction models have harnessed powerful models such as attention, GNNs, LSTMs and transformers, although many outstanding challenges remain. Prediction in autonomous driving provides the key to a higher level of understanding of road scenes, thereby offering the potential improve the safety of future autonomous driving technology.