GraphAD: Interaction Scene Graph for End-to-end Autonomous Driving

1 Introduction

The conventional Autonomous Driving (AD) system is manually divided into multiple sequential modules, including perception [Huang et al., 2021; Li et al., 2022b; Liu et al., 2022], prediction [Hu et al., 2021a; Jia et al., 2023a], planning [Hu et al., 2021b], and control [Wu et al., 2022]. However, the manual division prevents the system from being optimized jointly and globally, resulting in sub-optimal performance. To address this issue, end-to-end autonomous driving algorithms [Hu et al., 2023; Jiang et al., 2023; Zhang et al., 2021] optimize different modules altogether, making the whole system differentiable. With the potential of reducing accumulated errors and achieving higher performance, endto-end algorithms are drawing increasing attention [Chen et al., 2023]. In end-to-end driving algorithms, both the prediction [Gu et al., 2023] and the planning [Zeng et al., 2019;

∗Corresponding author.

Hu et al., 2021b] modules share the same task of predicting future trajectories of agents (i.e., road agents in the prediction task and the ego vehicle in the planning task). The future trajectories are affected by the interactions among the agents and surrounding environments. Hence, modeling the interactions plays a central role in conventional end-to-end algorithms, which is commonly concreted by the attention mechanism [Hu et al., 2023; Jiang et al., 2023]. However, the attention mechanism, mainly based on correlations of implicit features, lacks the prior knowledge of geometry about which driving elements are more important. As a result, the attention-based interactions inevitably waste their modeling capacities on the unimportant driving elements, while performing worse when being impaired by nuisance elements.

In this paper, we propose the Interaction Scene Graph for end-to-end Autonomous Driving (GraphAD) to enhance the interactions among driving elements. GraphAD encodes strong prior knowledge of the interactions into a graph model, the Interaction Scene Graph (ISG). The ISG is a directed graph model whose nodes represent key driving elements in the environment, including traffic agents and lanes. The driving elements are carefully selected for information aggregation, such that only important driving elements are represented. The directed edges in the graph represent the interactions between the nodes. Each node is linked to only a small number of other nodes, making the edges sparse. As a result, the ISG is a concise and efficient representation of the interactions.

Specifically, the ISG consists of two complementary parts, including the Dynamic Scene Graph (DSG) and the Static Scene Graphs (SSG), as shown in Fig. 1. The DSG focuses on the interactions among agents. Each node of DSG corresponds to an agent. Each edge has a weight, measuring the attention one agent pays to the other. The weights are employed to predict the future trajectories of the agents. Note that the interactions among agents depend on the future trajectories, i.e., an agent would pay more attention to one another if their trajectories would collide in the future. Hence, the weights in DSG and the predicted future trajectories are interdependent. The predicted future trajectories can in turn refine the weights in the DSG. So we optimize the DSG and the predicted future trajectories iteratively. The SSG depicts the interaction between the agent and the surrounding map elements. Each agent is represented as a node in the SSG. In the meanwhile, the surrounding lanes in the map are also represented as nodes. The directed edges come from the agent and go to the lanes, modeling the attention the agents pay to the lanes. We apply graph neural networks [Kipf and Welling, 2016; Velivckovic et al., 2017] on the DSG and SSG. The extracted features are utilized to predict the future trajectories of the ego and all other agents. By such means, we are able to adopt a unified method to accomplish both the prediction and the planning tasks.

We evaluate our method on the nuScenes dataset [Caesar et al., 2020]. Extensive ablation studies are conducted to demonstrate the effectiveness of our design choices. We summarize our main contributions as follows: First, GraphAD is the first end-to-end autonomous algorithm which employs a graph model to describe the complex interactions in traffic scenes. The graph model allows us to introduce strong prior knowledge of the traffic scene into the algorithm effectively and efficiently. Second, we elaborately devise the ISG which concisely presents the heterogeneous interactions among ego vehicle, traffic agents, and map elements. In particular, the DSG is able to iteratively refine the prediction of future trajectories and describe subtle interactive games among agents. Finally, when compared with strong baselines [Hu et al., 2023; Hu et al., 2022; Jiang et al., 2023], our method achieves state-of-the-art performance on multiple tasks.

2 RelatedWork

2.1 End-to-End Autonomous Driving

Instead of adopting a modular paradigm in the traditional AD framework, end-to-end methods, which aim to output future actions based on sensor inputs, have attracted considerable attention. When formulated in an end-to-end manner, the whole framework can be optimized towards the ultimate planning task with high computational efficiency [Chen et al., 2023; Hu et al., 2023]. Some pioneering approaches attempt to directly predict the planned trajectory while lacking explicit supervision of intermediate perception and prediction tasks [Prakash et al., 2021; Wu et al., 2022; Wu et al., 2023; Zhang et al., 2021; Jia et al., 2023b]. Considering the transparency and interpretability for safety, recent works [Casas et al., 2021; Hu et al., 2022; Chen and Krahenbuhl, 2022; Hu et al., 2023; Ye et al., 2023; Jiang et al., 2023] introduce requisite preceding tasks in the end-to-end framework, thus unifying perception, prediction, and planning into a holistic model. For instance, UniAD [Hu et al., 2023], which regards task-specific queries as a powerful tool for message passing throughout the AD pipeline, has achieved remarkable performance in both multi-object tracking, online mapping, motion forecasting, occupancy prediction, and planning. FusionAD [Ye et al., 2023] extends the capacity of UniAD [Hu et al., 2023] with multi-modal input. In the meantime, some researchers focus on the impact of different privileged inputs. VAD [Jiang et al., 2023] contends that end-to-end AD can be performed in a fully vectorized manner with high efficiency, while OccNet [Tong et al., 2023] attempts to perform the planning task based on the predicted occupancy.

Despite previous methods that have gained impressive performance, the interactions between traffic agents and the surrounding environment are not fully explored. In this work, we propose the Interaction Scene Graph to explicitly model the heterogeneous interactions between the dynamic and static driving elements.

2.2 Graph Neural Networks

Thanks to the success of Graph Neural Networks in graph data, GNNs [Kipf and Welling, 2016; Hamilton et al., 2017; Velivckovic et al., 2017] have been widely adopted in various fields, such as object detection [Wang and Solomon, 2021; Shi and Rajkumar, 2020; Xu et al., 2019], skeleton-based action recognition [Yan et al., 2018; Li et al., 2023a], person re-identification [Yang et al., 2020]. Also, GNNrelated advances attract researchers in the autonomous driving community, several studies propose to leverage the ability of GNNs for scene perception and motion prediction. GNN3DMOT [Weng et al., 2020] and PTP [Weng et al., 2021] attempt to model inherit interactions among detected targets for 3D multi-object tracking. For online mapping, LaneGCN [Liang et al., 2020] constructs a lane graph from an HD map, and TopoNet [Li et al., 2023b] introduces relation modeling between lane and traffic elements with a learned scene knowledge graph. In multi-agent motion forecasting, both moving agents and map elements are designed as nodes in graph construction, and the introduction of the relationship among them would benefit trajectory prediction [Salzmann et al., 2020; Mo et al., 2022]. HDGT [Jia et al., 2023a] devises a heterogeneous graph and explicitly models all semantics and relations in the scene. Different from prior works, GraphAD is the first to capture the interactions among dynamic agents and static map elements in the end-to-end AD framework. Also, GraphAD proposes to consider the potential movements of dynamic agents in graph construction by introducing the trajectory proposals.

3 Method

The overall framework of GraphAD is presented in Fig. 2. First, with multi-view video sequences, camera parameters, and ego-poses as input, the image features are extracted by the image encoder and then lifted to the Bird-Eye-View (BEV) features. The multi-frame BEV features are further aggregated to form the spatiotemporal scene representation. Second, GraphAD employs two transformer decoders, i.e. the TrackFormer and the MapFormer, to extract the structured representations for the dynamic and static driving elements. Third, the Interaction Scene Graph is explicitly constructed to model the interactions among the ego vehicle, dynamic elements, and static elements, by considering the potential movements. Finally, the graph-aggregated ego query feature, combined with ego status features and high-level driving command, is processed by the planning head to predict the ego-vehicle trajectory. We elaborate on the designs of these steps in the following sections.

3.1 Spatiotemporal Scene Representation

Image Encoder. The image encoder includes a backbone network for multi-scale feature extraction and a neck for fusing these features. Formally, with the multi-view images I ∈ RN×3×HI×WI as input, the image encoder creates the extracted visual features F2d ∈ RN×CI×H′I ×W ′ I , where N is the number of camera views, CI is the channel number, (HI,WI) and (H′I ,W ′ I ) are the input and downsampled image sizes. The output visual feature can contain the fundamental semantics and geometry of the surrounding environment.

Image-to-BEV Transformation. To build a unified scene representation for temporal aggregation and multi-task inference, we lift the multi-view image features to BEV representations with the Lift-Splat-Shoot paradigm [Philion and Fidler, 2020; Huang et al., 2021; Li et al., 2022a]. Specifically, the image features Ft2d at time t are processed to create the context features Ftcon ∈ RN×C×H′I ×W ′ I and categorical depth distributions Dt ∈ RN×D×H′I ×W ′ I , where C is the channel number and D is the number of depth bins. The outer product Ftcon ⊗Dt is then computed as lifted feature point cloud Pt ∈ RNDH′I W ′ I ×C . Finally, the voxel-pooling is employed to process the feature points and generate the BEV feature FtBEV ∈ RC×H×W at time t.

Temporal Feature Aggregation. The multi-frame BEV features {FtBEV }tcur t=tcur-T+1 , where tcur is the current time and T ∈ N+ is the number of frames, are first warped into the ego-centric coordinate system at the current time so that the ego-motion misalignment is removed. Afterward, the aligned multi-frame BEV features are concatenated along the channel dimension and further processed by a convolutional BEV encoder. The output spatiotemporal BEV feature FBEV ∈ RCo×H×W will serve as the unified spatiotemporal scene representation for downstream tasks.

3.2 Structured Element Learning

Based on the spatiotemporal scene features, the extraction of structured elements, including traffic agents and map elements, is important for safety-critical planning in autonomous driving. Therefore, GraphAD utilizes the TrackFormer and the MapFormer to predict these driving-related instances.

TrackFormer. With the spatiotemporal BEV representation, the TrackFormer aims to perform end-to-end 3D object detection and tracking. Following the design of [Hu et al., 2023], we employ two groups of object queries and the transformer decoder to solve the problem. Specifically, one group of track queries, which corresponds to previously detected objects, is still required to predict the updated 3D bounding boxes of the same object identities. The other group of detection queries is responsible for the objects which are visible for the first time. For each timestamp, the positive queries, including the tracked and newborn, will serve as the track queries for the next timestamp. The transformer decoder layer includes the self-attention between all object queries and the deformable attention for attending to the spatiotemporal BEV features.

MapFormer. To better capture the geometric constraints of map elements, we follow recent practices [Liao et al., 2022; Jiang et al., 2023] to learn vectorized representations of local maps. Specifically, the MapFormer utilizes the instance-level and point-level queries to form the hierarchical map queries, which are processed by a similar transformer decoder as in TrackFormer. Finally, the output map queries are projected to the class scores and a series of BEV coordinates of potential map elements. To fully capture the map information, four kinds of elements are modeled, including the lane centerline, lane divider, road boundary, and pedestrian crossing.

3.3 Interaction Scene Graph

With the extracted driving instances in structured formats, including traffic agents and map elements, the key challenge lies in how the network can perceive heterogeneous interactions. These interactions, including the driving game between dynamic agents, or the simple centerline-following heuristics, are important for forecasting the future of surrounding environment and making driving decisions. To this end, we construct the Interaction Scene Graph to capture these heterogeneous interactions. As an iterative process, the Interaction Scene Graph functions in three steps. First, all dynamic and static elements are formulated as graph node representations, including explicit geometry and implicit features. Second, the Interaction Scene Graph is constructed with strong geometric priors. Third, the graph node features are updated based on the established graph edges, which are further processed to update the geometry. The detailed formulation is elaborated in the following paragraphs.

Graph Node Representation. The Interaction Scene Graph is constructed on the structured nodes of traffic agents and map elements. Each graph node is designed to include both the explicit geometry and the implicit features. Note that the ego-vehicle is treated as one of the traffic agents to participate in graph-based interactions.

Specifically, the graph nodes of traffic agents, i.e. dynamic graph nodes, are organized as one set P d = {pd1 , . . . , pdNd }, where Nd is the number of dynamic graph nodes. Also, pdi = (xdi , f d i ) represents the node representation with its trajectory proposal xdi ∈ RMd×2 as BEV coordinates and its node feature f d i ∈ RCg with Cg channels, where Md is the time horizon of trajectory prediction. The trajectory proposals are the trajectory predictions from the previous layer. For the first layer, the clustering results from k-means are utilized instead. The implicit node features are computed as the combination of previous node features, queries from the Track- Former, embeddings of trajectory proposals, and learnable intention embeddings, following [Hu et al., 2023]. For the unified formulation, we treat different modalities of the same agent as different dynamic graph nodes.

Similarly, the graph nodes of map elements, i.e. static graph nodes, are organized as the other set P s = {ps1 , . . . , psNs }, where Ns is the number of static graph nodes and psi = (xsi , f s i ) represents one map element by a series of BEV coordinates xsi ∈ RMs×2 with Ms points and its node feature f s i ∈ RCg with Cg channels. The structured predictions from the MapFormer, including the BEV coordinates and the output query features, are directly used as the static graph nodes. Since the map elements in the driving scenes usually serve as constant environment constraints, their node features are not updated in the iterative layers.

Graph Connection Construction. To capture the heterogeneous interactions between all graph nodes, the Interaction Scene Graph consists of the Dynamic Scene Graph (DSG) and the Static Scene Graph (SSG). The Dynamic Scene Graph is formulated as Gd = (P d ,Ed ) by using the traffic agents as dynamic graph nodes, which intends to model the driving game between these agents. The Static Scene Graph is formulated as Gs = (P d , P s ,Es ) by incorporating both the dynamic and static graph nodes, which focuses on providing the appropriate map information for the dynamic agents. For both DSG and SSG, we follow the same high-level philosophy for computing the edge connections. Specifically, we compute the pair-wise distances between nodes and connect each node to its K nearest neighbors. Despite the straightforward formulation, the design choices of pair-wise distance functions are still underexplored.

Existing graph-based methods [Wang and Solomon, 2021; Jia et al., 2023a] usually exploit the pair-wise distance in feature or coordinate spaces. However, the heterogeneous and evolutionary interactions in the constructed scene graph, with dynamic agents and map elements, cannot be well processed by existing approaches. To this end, we propose to utilize the geometric distances based on trajectory proposals to measure the correlations between graph nodes. On the Dynamic Scene Graph, the distance Hd (pdi , pdj ) between two dynamic graph nodes is computed as the minimal distance between their trajectory proposals at each time, as in Eq. (1):

where xdi (t) refers to the predicted future position at time t. On the Static Scene Graph, the distance Hs (pdi , psj ) between a dynamic node and a static node is computed as the minimal distance between the dynamic trajectory proposal and the static map coordinates, as in Eq. (2):

where xdi (t) refers to the predicted future position at time t and xsj (k) refers to the k-th coordinate point of the predicted map element. When the pair-wise distances are computed, the nearest K graph nodes with minimal distances are selected as the graph neighbors.

Graph Feature Aggregation. Since the interaction connections have been established, the final part is to refine the node feature by aggregating the information from its connected neighbors. A simple yet effective approach is proposed for the feature aggregation in the Interaction Scene Graph. Specifically, the feature of each neighbor node is concatenated with the target node and then processed by a MultiLayer Perceptron (MLP). Finally, the permutation-invariant max-pooling is used to aggregate the processed neighbor features into the target node. Also, the Dynamic Scene Graph and Static Scene Graph share the same approach for graph feature aggregation. At the end of each iteration layer, the updated features of dynamic agents are utilized to predict their multi-modal trajectories, including the probability score and the trajectory points for each modality. The predicted trajectory points are further used to update the geometric node features into the next iteration layer.

3.4 Planning Head

Planning Head Structure. The input information for the planning head includes the high-level driving command, the ego-status features, and the processed ego-query from the Interaction Scene Graph. The three groups of features are concatenated and processed by a simple MLP for the final planning predictions.

Ego-status Features. The ego-status information, which mainly includes the velocity, acceleration, and angular velocity, is important for the open-loop planning performance. Therefore, we use a small Multi-Layer Perceptron (MLP) to encode the ego-status information, along with the history trajectories of the ego-vehicle, into the ego-status features.

Occupancy-based Post-optimization. To further avoid the collision with other road agents and ensure the driving safety, we follow the implementation of UniAD [Hu et al., 2023] to train an occupancy head, whose predictions can be utilized to post-optimize the predicted planning trajectories.

3.5 Training

Loss Functions. The loss functions include the depth estimation loss Ldepth, the TrackFormer loss Ltrack, the MapFormer loss Lmap, the motion trajectory loss Lmotion, the occupancy loss Locc, and the planning loss Lplan. GraphAD is end-to-end trained with the summation of multi-task losses:

Specifically, we use binary cross-entropy for Ldepth and follow existing methods [Hu et al., 2023; Jiang et al., 2023] for training other tasks.

Multi-stage Training. First, GraphAD is trained to jointly predict the 3D object detection and vectorized map elements. Second, we freeze the image backbone and train GraphAD for tracking, vectorized map, and graph-based motion prediction. Finally, we further add the tasks of occupancy prediction and planning for end-to-end training.

4 Experiments

Our experiments are conducted on the challenging nuScenes dataset [Caesar et al., 2020], where 1000 complex driving scenes are included. For annotations, over 1.4M 3D bounding boxes of 23 categories are provided in total, and the keyframes are annotated at 2 Hz.

4.1 Implementation Details

For benchmark results, GraphAD adopts the input size of 640 and ResNet101-DCN [He et al., 2016] as the image backbone. The image neck generates feature maps with 512 channels and 16× downsampling. For image-to-BEV transformation, GraphAD uses the method in BEVDepth [Li et al., 2022a] to generate the BEV features with 80 channels. Four frames of BEV features are fused to create the spatiotemporal scene representation FBEV ∈ R256×200×200 . The TrackFormer strictly follows the settings of UniAD [Hu et al., 2023], while the MapFormer uses 100 map queries and a six-layer transformer decoder. The Interaction Scene Graph stacks three iterative layers for motion prediction with six modalities. The number of neighbours is set to 24 for the Dynamic Scene Graph and 8 for the Static Scene Graph. For ego-status features in the planning head, we follow the preprocessing of CAN-bus information from VAD [Jiang et al., 2023]. For ablation studies, we adopt the input size of 256 × 704 and ResNet50 as the image backbone.

4.2 Metrics

We follow the same evaluation protocol of previous state-ofthe-art method UniAD [Hu et al., 2023]. For motion prediction, we employ End-to-end Prediction Accuracy (EPA), Average Displacement Error (ADE), Final Displacement Error (FDE), and Miss Rate (MR). In planning task, Displacement Error (DE, L2 distance) and Collision Rate (CR) are employed, where CR is considered as the main metric.

4.3 Benchmark Results

Planning Results. As shown in Tab. 1, GraphAD achieves the state-of-the-art performance for open-loop planning on the nuScenes validation set. When compared to the second best method, Agent-Driver [Mao et al., 2023b], GraphAD achieves a 42.9% reduction of collision rate, which demonstrates the effectiveness of the proposed Interaction Scene Graph for aggregating information from related traffic agents and map elements.

Prediction Results. The benchmark results for motion prediction on the nuScenes validation set are summarized in Tab. 2. GraphAD achieves the best performance with 0.68 minADE and 0.514 EPA, significantly outperforming the previous best method UniAD [Hu et al., 2023]. The improved performance validates the enhanced capacity of Interaction Scene Graph in modeling the map guidance and intention interaction from other driving instances.

4.4 Ablation Studies

To demonstrate the effectiveness of the proposed Interaction Scene Graph, we conduct extensive ablation studies on the nuScenes validation set.

Effectiveness of Interaction Scene Graph. In Tab. 3, we ablate the influence of Dynamic Scene Graph (DSG) and Static Scene Graph (SSG) on the motion prediction of traffic agents. We can observe that both types of scene graphs make significant contribution to the performance boost. Since the DSG can model the driving game between dynamic agents and the SSG is able to provide explicit map constraints, both types of graph-based interactions can provide valuable and complementary information for the trajectory prediction. For comprehensive evaluation, we also implement an attentionbased variant, where the inter-agent and agent-map interactions are entirely realized by the vanilla attention mechanism. However, we find the attention-based variant, without explicit geometric prior, fails to extract valid information and generates inferior performance.

Design choices of graph node distance. In Tab. 4, we analyze the influence of different methods for computing the distance between graph nodes. "Feature Distance" and "Current Distance" denote distances in the feature space and distances between current locations respectively, while "Trajectory Distance" is the distance between potential trajectories. Since the distance function directly determines which neighbour nodes will participate in the feature aggregation, its design choice is of vital importance. From the experimental results, we can find the proposed trajectory distance significantly outperforms the current distance because it explicitly considers the potential interactions in the future, which is crucial for accurate trajectory estimation. On the other hand, the geometric distance on trajectories also outperforms the feature distance. It is possibly because the graph nodes, including both traffic agents and map elements, with different sources and modalities have heterogeneous features.

Design choices of methods for graph feature aggregation. In Tab. 5, we compare different methods for aggregating the neighbour node features to update the vertex. As observed in the table, MLP-based aggregation methods performs better than attention-based methods. Furthermore, the max-pooling operation outperforms the avg-pooling method, reaching 0.665m minADE, 0.989m minFDE and 0.160 MR. Thus, we choose MLP with max-pooling as default setting.

Design choices of planning head. In Tab. 6, we explore the effects of different components for the planning task, where "Graph" refers to the proposed Interaction Scene Graph, "Ego-states" means the utilization of ego-vehicle status, and "Post-optim." represents the optimization strategy with the predicted occupancy. The following effects can be observed: (1) The incorporation of ego-state features can bring a significant improvement on the planning performance, since the information, like velocity and acceleration, makes it much easier to recover the ego-trajectory. (2) Whether or not the ego-state features are utilized, the proposed method of Interaction Scene Graph consistently improves the planning performance. (3) The post-optimization with the predicted occupancy plays an important role in ensuring the driving safety, by avoiding the potential collisions with explicit adjustments. With all above components, GraphAD, with smaller input sizes and image backbone, achieves a remarkable collision rate of 0.15%.

4.5 Qualitative Results

To qualitatively evaluate our method for better understanding, we visualize both the intermediate interactions and the final results of GraphAD. As shown in Fig. 3, the agent of interest has 6 predicted future trajectories for different potential intentions. The dynamic scene graph for each trajectory automatically links the agent to other traffic agents nearby. With these explicit geometry priors, the agent can focus on the interactions with the important agents. From the cases in Fig. 4, GraphAD enables the ego vehicle to maneuver safely in complex situations like road junction and opposite meeting. These planning abilities results from the accurate motion prediction and necessary inter-agent interactions, based on the proposed graph designs.

5 Conclusion

In this paper, we propose a new end-to-end autonomous driving algorithm, GraphAD, which employs an elaborately designed graph to describe heterogeneous interactions in complex traffic scenes. The graph explicitly encodes key driving elements and their relations, allowing us to introduce strong prior knowledge into the algorithm. As a consequence, GraphAD achieves state-of-the-art performance in both the prediction and the planning tasks. The way using graphs to encode more complex interactions among diverse traffic instances, such as traffic lights and routing decisions, needs further exploration.

Acknowledgements

This research is supported by PhiGent Robotics, Beijing Nova Program (20240484562), and Geely Automobile Research Institute.