BEVTrack: A Simple and Strong Baseline for 3D Single Object Tracking in Bird's-Eye View

1 Introduction

3D single object tracking (SOT) is crucial for various applications, such as autonomous driving [Yin et al., 2021; Chen et al., 2023]. It aims to localize a specific target across a sequence of point clouds, given only its initial status. Existing tracking approaches [Qi et al., 2020; Zheng et al., 2022] commonly adopt point-based representations, directly taking raw point clouds as input. For example, P2B [Qi et al., 2020] and its follow-up works [Zheng et al., 2021; Zhou et al., 2022] adopt a point-based network [Qi et al., 2017b; Wang et al., 2019] with the Siamese architecture for feature extraction, followed by a point-based appearance matching module [Qi et al., 2020; Zhou et al., 2022; Xu et al., 2023a] for propagation of target cues, and a 3D Region Proposal Network [Qi et al., 2019; Fang et al., 2021] for target localization, as illustrated in Fig. 1(a). M2-Track [Zheng et al., 2022] proposes a motion-centric paradigm, that first segments the target points from their surroundings with a PointNet [Qi et al., 2017a] segmentation network and then localizes the target through a motion modeling approach followed by a box refinement module, as illustrated in Fig. 1(b).

∗Corresponding author.

Although these approaches have demonstrated strong performance on tracking benchmarks, their reliance on intricate designs and the need to address multiple subtasks significantly increase framework complexity, making both training and deployment challenging. This raises an important question: can we simplify the tracking formulation while maintaining competitive performance?

In this paper, we present BEVTrack, a simple yet strong baseline for 3D SOT, as shown in Fig. 1(c). By estimating the target motion in Bird's-Eye View (BEV) to perform tracking, BEVTrack demonstrates surprising simplicity from various aspects, i.e., network designs, training objectives, and tracking pipeline. Specifically, we first adopt a voxel-based network [Zhou and Tuzel, 2018] with the Siamese architecture for feature extraction. Subsequently, we compress height information into the channel dimension to obtain the BEV features. Given that corresponding objects are spatially adjacent in the BEV features across consecutive frames, we can easily fuse them together with an element-wise operation such as concatenation [Duffhauss and Baur, 2020]. Then we adopt several convolutional and down-sampling layers to capture the object motion with a wide range of patterns. Finally, we discard the complicated region proposal network while using a global max pooling followed by a lightweight multilayer perception (MLP) to regress the relative target motion.

To optimize the regression-based trackers, current approaches typically employ conventional L1 or L2 loss. This kind of supervision actually makes a fixed Laplacian or Gaussian assumption on the data distribution, which is inflexible when handling targets possessing diverse attributes (e.g., sizes, moving patterns, and sparsity degrees). For example, when tracking targets with high sparsity degrees, where the predictions may be uncertain while the annotations are not very reliable, the outputs should conform to a distribution with a large variance. To this end, we introduce a novel distribution-aware regression strategy for tracking, which constructs the likelihood function with the learned underlying distributions adapted to distinct targets, instead of making a fixed assumption. Note that this strategy does not participate in the inference phase, thus boosting the tracking performance without additional computation overhead. Despite no elaborate design, BEVTrack exhibits a substantial performance advantage over the current state-of-the-art (SOTA) methods on challenging tracking datasets, i.e., KITTI [Geiger et al., 2012], NuScenes [Caesar et al., 2020], and Waymo Open Dataset [Sun et al., 2020], while operating at 200 FPS, enabling real-time applicability.

The main contributions of this paper are three-fold:

• We propose BEVTrack, a simple yet strong baseline for 3D SOT. BEVTrack marks the first to perform tracking through motion modeling in BEV, resulting in a simplified tracking pipeline design.
• We present a novel distribution-aware regression strategy for tracking, which constructs the likelihood function with the learned underlying distributions adapted to targets possessing diverse attributes. This strategy provides accurate guidance for tracking, resulting in improved performance while avoiding extra computation overhead.
• BEVTrack achieves SOTA performance on three popular benchmarks while maintaining a high inference speed.

2 RelatedWork

Early 3D SOT approaches [Pieropan et al., 2015; Spinello et al., 2010] predominantly utilize RGB-D data and often adapt 2D Siamese networks by integrating depth maps. However, variations in illumination and appearance can adversely affect the performance of such RGB-D techniques. SC3D [Giancola et al., 2019] is the first 3D Siamese tracker based on shape completion that generates a large number of candidates in the search area and compares them with the cropped template, taking the most similar candidate as the tracking result. The pipeline relies on heuristic sampling and does not learn end-to-end, which is very time-consuming.

To address these issues, P2B [Qi et al., 2020] develops an end-to-end framework by employing a shared point-based backbone for feature extraction, followed by a point-wise appearance-matching technique for target cues propagation. Ultimately, a Region Proposal Network is used to derive 3D proposals, of which the one with the peak score is selected as the final output. P2B reaches a balance between performance and efficiency, and many works [Zheng et al., 2021; Shan et al., 2021; Zhou et al., 2022; Hui et al., 2022] follow the same paradigm. Drawing inspiration from the success of transformers [Vaswani et al., 2017], many studies have incorporated elaborate attention mechanisms to enhance feature extraction and target-specific propagation. For instance, PTT [Shan et al., 2021] introduces the Point Track Transformer to enhance point features. PTTR [Zhou et al., 2022] presents the Point Relation Transformer for target-specific propagation and a Prediction Refinement Module for precision localization. Similarly, ST-Net [Hui et al., 2022] puts forth an iterative correlation network, and CXTrack [Xu et al., 2023a] presents the Target Centric Transformer to harness contextual information from consecutive frames. SyncTrack [Ma et al., 2023] introduces a single-branch and singlestage framework, without Siamese-like forward propagation and a standalone matching network. MBPTrack [Xu et al., 2023b] exploits both spatial and temporal contextual information using a memory mechanism. Unlike the Siamese paradigm, M2-Track [Zheng et al., 2022] proposes a motioncentric paradigm, explicitly modeling the target's motion between two successive frames. This motion-centric paradigm shows robustness to the problems caused by the appearance variation and the sparsity of point clouds. Our study also adopts a motion-centric paradigm but focuses on simplifying the tracking formulation, resulting in a streamlined and efficient solution that maintains strong performance.

3 Method

3.1 Overview

Given the 3D bounding box (BBox) of a specific target at the initial frame, 3D SOT aims to localize the target by predicting its 3D BBoxes in the subsequent frames. A 3D BBox Bt ∈ R7 is parameterized by its center (i.e., x, y, z coordinates), orientation (i.e., heading angle θ around the upaxis), and size (i.e., width, length, and height). Suppose the point clouds in two consecutive frames are denoted as Pt-1 ∈ RNt-1×3 and Pt ∈ RNt×3 , respectively, where Nt-1 and Nt are the numbers of points in the point clouds. Notably, the two point clouds have been transformed to the canonical system w.r.t the target BBox Bt-1 (either given as the initial state or predicted by the tracker). Since the size of the tracking target rarely changes, we follow the common practice [Qi et al., 2020] and assume a constant target size. Consequently, the inter-frame target translation offsets (i.e., ∆x,∆y,∆z) and the yaw angle offset ∆θ are regressed. By applying the translation and rotation to the 3D BBox Bt-1, we can compute the 3D BBox Bt to localize the target in the current frame. The tracking process can be formulated as:

where F is the mapping function learned by the tracker.

Following Eq. (1), we propose BEVTrack, a simple yet strong baseline for 3D SOT. The overall architecture of BEVTrack is presented in Fig. 2. It first employs a shared voxelbased backbone [Zhou and Tuzel, 2018] to extract 3D features, which are then squeezed to derive the BEV features (Sec. 3.2). Subsequently, BEVTrack fuses the BEV features via a concatenation operation and several convolutional layers (Sec. 3.3). Finally, it adopts a simple prediction head which contains a global max pooling layer followed by a lightweight MLP. For prediction, we employ a novel distribution-aware regression strategy in the training phase (Sec. 3.4).

3.2 Feature Extraction

To localize the target from surroundings accurately, we need to learn discriminative features from the point clouds. Instead of using a point-based backbone [Qi et al., 2017a; Qi et al., 2017b; Wang et al., 2019] as in [Qi et al., 2020; Zheng et al., 2022; Xu et al., 2023a], we adopt VoxelNet [Zhou and Tuzel, 2018] as the shared backbone. Specifically, we voxelize the 3D space into regular voxels and extract the voxel features of each non-empty voxel by a stack of sparse convolutions, where the initial feature of each voxel is simply calculated as the mean values of point coordinates within each voxel in the canonical coordinate system. The spatial resolution is downsampled 8 times by three sparse convolutions [Graham et al., 2018] of stride 2, each of which is followed by several submanifold sparse convolutions [Graham and Van der Maaten, 2017]. Afterwards, we squeeze the sparse 3D features along the height dimension to derive the BEV features Xt-1 ∈ RH×W×C and Xt ∈ RH×W×C , where H and W denote the 2D grid dimension and C is the number of feature channels.

3.3 BEV-basedMotionModeling

BEV-based Motion Modeling (BMM) aims to model the motion relations of the target from consecutive frames. Given the BEV features Xt-1 and Xt, the corresponding objects are spatially adjacent across them. Therefore, we can easily fuse them together with an element-wise operator such as concatenation to preserve their spatial proximity. Furthermore, we apply several convolutional blocks to encode their motion relations. It is noteworthy that the receptive field of the convolution is limited due to the small kernel size, while the motion pattern of objects is variable in different scenes. As shown in

Fig. 3, objects with a fast-moving speed are far apart across the BEV feature maps. To deal with this issue, we propose to enlarge the receptive field through spatial down-sampling by inserting convolutions of stride 2 at intervals, allowing it to capture a wide range of motion patterns. The above process can be formulated as:

where Convs denotes the convolutional blocks in BMM and [; ] denotes the concatenation operator. Y ∈ RH′×W ′×C′ , where H′ , W ′ , and C ′ denote the spatial dimension and the number of feature channels, respectively. See the appendix for the ablation experiments on the design choices of BMM.

3.4 Distribution-aware Regression

Different from the existing two-stage point-to-box prediction head [Qi et al., 2020; Zhou et al., 2022; Hui et al., 2022], which contains two parts of proposals generation and proposal-wise scores prediction, we propose a one-stage postprocessing-free prediction head consisting of only a global max-pooling layer and an MLP, i.e.,

where M ∈ R8 denotes the expectation of the target translation and yaw offsets ¯u ∈ R4 and their standard deviation σ ∈ R4 , in contrast to prior methods that solely regress the deterministic target motion u. By applying the predicted translation to the last state of the target, we can localize the target in the current frame. Note that the standard deviation σ enables dynamic optimization during training while being removed in the inference phase, as elaborated below.

The difference in sizes, moving patterns, and sparsity degrees among the tracked targets poses great challenges to existing trackers. To address this issue, we propose a novel distribution-aware regression strategy for tracking, which constructs the likelihood function with the learned underlying distributions adapted to distinct targets. In this way, the model can adaptively handle targets with different attributes, thus improving tracking performance.

Following [Li et al., 2021], we model the distribution of the target motion u ∼ P(u) with reparameterization, which assumes that objects belonging to the same category share the same density function family, but with different mean and variance. Specifically, P(u) can be obtained by scaling and shifting z from a zero-mean distribution z ∼ PZ(z) with transformation function u = ¯u + σ · z, where ¯u denotes the expectation of the target translation offsets and σ denotes the scale factor of the distribution. PZ(z) can be modeled by a normalizing flow model (e.g., real NVP [Dinh et al., 2016]). Given this transformation function, the density function of P(u) can be calculated as:

In this work, we employ residual log-likelihood estimation [Li et al., 2021] to estimate the above parameters, which factorizes the distribution PZ(z) into one prior distribution QZ(z) (e.g., Laplacian distribution or Gaussian distribution) and one learned distribution GZ(z | θ). To maximize the likelihood in Eq. (4), we can minimize the following loss:

Here, ˆ z = (ˆu-¯u)/σ, ˆu is the ground truth translation offsets.

4 Experiment

4.1 Experimental Settings

We validate the effectiveness of BEVTrack on three widelyused challenging datasets: KITTI [Geiger et al., 2012], NuScenes [Caesar et al., 2020], and Waymo Open Dataset (WOD) [Sun et al., 2020]. KITTI contains 21 video sequences for training and 29 video sequences for testing. We follow previous work [Qi et al., 2020] to split the training set into train/val/test splits due to the inaccessibility of the labels of the test set. NuScenes contains 1,000 scenes, which are divided into 700/150/150 scenes for train/val/test. Following the implementation in [Zheng et al., 2021], we compare with the previous methods on five categories including Car, Pedestrian, Truck, Trailer, and Bus. WOD includes 1150 scenes with 798 for training, 202 for validation, and 150 for testing. Following M2-Track [Zheng et al., 2022], We conduct training and testing respectively on the training and validation set. Following [Qi et al., 2020], we adopt Success and Precision defined in one pass evaluation (OPE) [Kristan et al., 2016] as the evaluation metrics.

4.2 Comparison with State-of-the-art Methods

Results on KITTI. We present a comprehensive comparison of BEVTrack with the previous state-of-the-art approaches on KITTI, namely SC3D [Giancola et al., 2019], P2B [Qi et al., 2020], BAT [Zheng et al., 2021], V2B [Hui et al., 2021], PTTR [Zhou et al., 2022], STNet [Hui et al., 2022], M2-Track [Zheng et al., 2022], CXTrack [Xu et al., 2023a], SyncTrack [Ma et al., 2023], and MBPTrack [Xu et al., 2023b]. As shown in Tab. 1, BEVTrack surpasses current tracking methods with a significant improvement across all categories on KITTI in Success and Precision metrics. We also visualize the tracking results on four KITTI categories for qualitative comparisons. As shown in Fig. 4, BEVTrack generates more robust and accurate tracking predictions.

For a more comprehensive evaluation of effectiveness in challenging scenarios, we present results from various methods in cases characterized by sparse scenarios, occlusion, and intra-class distractors. To verify the effectiveness in sparse scenarios, we follow V2B [Hui et al., 2021] to select sparse scenes for evaluation according to the number of points lying in the target bounding boxes in the test set. For analysis of the impact of intra-class distractors, we pick out scenes that contain Pedestrian distractors close to the target from the KITTI test split. For the occlusion situations, we filter out scenes of occlusion scores less than one in the KITTI test set.

As shown in Fig. 5, each column from left to right represents the cases of base setting, sparse scenarios, occlusion, and intra-class distractors respectively on the Pedestrian category of the KITTI dataset. Our proposed BEVTrack is more robust to these challenging scenarios than other methods.

Results on Nuscenes. NuScenes poses a more formidable challenge for the 3D SOT task compared to KITTI, primarily attributable to its larger data volumes and sparser annotations (2Hz for NuScenes versus 10Hz for KITTI and Waymo Open Dataset). Subsequently, following the methodology established by M2-Track, we perform a comparative evaluation on the NuScenes dataset against prior methodologies, namely SC3D [Giancola et al., 2019], P2B [Qi et al., 2020], PTT [Shan et al., 2021], BAT [Zheng et al., 2021], PTTR [Zhou et al., 2022], M2-Track [Zheng et al., 2022], and MBPTrack [Xu et al., 2023b]. As shown in Tab. 2, our method achieves a consistent and large performance gain compared with the previous state-of-the-art method, MBPTrack. BEVTrack exhibits superior performance over methods reliant on appearance matching or segmentation, especially in datasets like NuScenes with sparser point clouds.

Results on WOD. WOD dataset presents greater challenges compared to KITTI and NuScenes due to its larger data volumes and complexities. Following M2-Track, we conduct training and testing on the respective training and validation sets. We consider trackers that report results under this setting as competitors, including P2B [Qi et al., 2020], BAT [Zheng et al., 2021], and M2-Track [Zheng et al., 2022]. As illustrated in Tab. 3, BEVTrack consistently outperforms all competitors on average, particularly excelling in the Vehicle category. Remarkably, BEVTrack, with its simple design, demonstrates even greater potential on this large dataset.

Inference Speed. We analyze the efficiency of BEVTrack in Tab. 4. It can be observed that BEVTrack is lightweight with only 0.76G FLOPs and 13.8M parameters. The simple architecture ensures real-time inference of BEVTrack at an impressive speed of 200 FPS on a single NVIDIA GTX 4090 GPU, which is 2.34 times faster than the previously leading method, MBPTrack [Xu et al., 2023b]. The simplicity of BEVTrack's pipeline facilitates flexible adjustments in model size by incorporating advanced backbones or devising more effective BMMmodules to further enhance performance.

4.3 Ablation Studies

In this section, we analyze and compare each basic design in BEVTrack with other choices used in previous works. For better clarification, we ablate the effects of every design choice by replacing it from the proposed BEVTrack.

Pre-processing of Input. The input consists of LiDARscanned point clouds, with a cropped region where the object may appear. Most existing matching-based methods [Qi et al., 2020; Zheng et al., 2021; Fang et al., 2021] use the cropped target template from the previous frame and the full search area in the current frame, referred to as "Temp" in Tab. 5. M2-Track [Zheng et al., 2022] introduces a motioncentric approach, extracting the target points from their surroundings to feed the motion model, denoted as "Seg". CXTrack [Xu et al., 2023a] highlights the loss of contextual information in the previous methods and instead processes two consecutive frames, transforming them into the canonical previous box coordinates, with the region expanded by a margin relative to the object size, referred to as "Cx".

In Tab. 5, we compare the above three pre-processing methods on KITTI and NuScenes. The motion-based tracker BEVTrack performs poorly with the template and search area input ("Temp"). While "Seg" yields satisfactory results, it still lags behind "Cx". The segmentation network struggles to fully distinguish foreground from background, causing cumulative errors in the motion model. In contrast, "Cx" retains valuable contextual information, leading to the best performance, especially on NuScenes, which uses low-beam LiDAR. As a result, BEVTrack adopts "Cx" as its default preprocessing method.

Feature representation. We aim to learn a discriminative feature representation from point clouds. While existing 3D SOT methods often rely on point-based representation networks [Qi et al., 2017a; Qi et al., 2017b; Wang et al., 2019], LiDAR-scanned point clouds exhibit variable sparsity-dense clustering near objects and sparser distribution at a distance. Effectively learning point features that handle both dense and sparse regions is a significant challenge. To address this, we propose using voxel-based representations [Zhou and Tuzel, 2018] instead of point-based ones. As shown in Fig. 6, voxel features are less sensitive to point count variations, helping mitigate the sparsity issue.

In Tab. 6, we compare the "Point" and "Voxel" representations on the KITTI and NuScenes datasets. The "Voxel" setup follows the same approach as BEVTrack. To ensure a fair comparison, the "Point" method adopts a similar pipeline, including feature extraction, motion modeling, and regression. However, "Point" uses PointNet++ [Qi et al., 2017b] for feature extraction, capturing point-level features. To distinguish points across frames, "Point" adds timestamp encoding to the features, which are then processed by a second PointNet++ for motion modeling. As shown in Tab. 6, the significant performance gap between "Point" and "Voxel" supports our claim above, highlighting the effectiveness of using voxelbased representations in BEVTrack.

Motion Modeling. aims to infer the relative target motion (RTM) from consecutive point cloud frames. M2Track [Zheng et al., 2022], the representative motion-based tracker preceding BEVTrack, uses PointNet [Qi et al., 2017a] to predict RTM. In contrast, we introduce BMM for RTM modeling, as described in Sec. 3.3. BMM offers two key advantages over PointNet: 1) BEV better captures motion features, particularly for horizontal movement typical in autonomous driving; 2) BMM leverages cascaded convolutions to model local features and capture diverse motion patterns.

In Tab. 7, we compare the two motion modeling methods on KITTI and NuScenes. For the "PointNet" setting, we restore voxel features to point features using bilinear interpolation and replace the BMM module in BEVTrack with PointNet. As shown, BMM outperforms PointNet by a clear margin, highlighting the importance of local feature learning and BEV representation for motion modeling. Additionally, we observe that the methods differ in sensitivity to background points. According to Tab. 5, BEVTrack performs better without a segmentation network, while M2-Track benefits significantly from it. This suggests that BMM's local feature learning is more robust to background interference.

Paradigm Choices. Most existing 3D SOT methods follow either a matching-based or motion-based paradigm.

Matching-based trackers [Qi et al., 2020; Zheng et al., 2021; Zhou et al., 2022] locate targets by comparing the search area with template point clouds. However, LiDAR point clouds are often textureless and incomplete, hindering effective appearance matching. In contrast, motion-based tracking benefits from the absolute scale of target movement in point clouds, simplifying the tracking process. In this paper, BEVTrack adopts the motion-based paradigm.

We investigate the effect of paradigm selection on BEVTrack's performance by designing a matching-based variant that uses the same basic modules for fairness. Details of the variant's architecture can be found in the appendix. As shown in Tab. 8, the "motion-based" BEVTrack outperforms the matching-based variant by a significant margin. For the pedestrian class, which has similar intra-class distractors often appearing at closer distances, the performance of the matching-based version drops sharply by 7.7/6.9 in success/precision on KITTI and 13.6/18.0 on NuScenes.

Distribution-aware Regression. To investigate the impact of output distribution assumptions on tracking performance, we compare results using different density functions on the KITTI dataset. The Laplacian and Gaussian distributions reduce to standard l1 and l2 loss when assumed to have constant variances. As shown in Tab. 9, the proposed distributionaware regression approach outperforms regression with l1 or l2 loss, highlighting the importance of modeling the actual distribution of target motion.

5 Conclution

This paper introduces BEVTrack, a simple yet strong baseline for 3D single object tracking (SOT). BEVTrack performs tracking within the Bird's-Eye View representation, thereby effectively exploiting spatial information and capturing motion cues. Additionally, we propose a distribution-aware regression strategy that learns the actual distribution adapted to targets possessing diverse attributes, providing accurate guidance for tracking. Comprehensive experiments conducted on widely recognized benchmarks underscore BEVTrack's efficacy, establishing its superiority over state-of-the-art tracking methods. Furthermore, it achieves a high inference speed of about 200 FPS. We hope this study could provide valuable insights to the tracking community and inspire further research.

Limitation. BEVTrack, as a simple baseline, uses only two successive frames, but the sequence of historical trajectories and point clouds holds rich motion data. Modeling object motion from multiple frames could enhance tracking performance. We plan to explore the use of temporal information in BEVTrack for long-term video tracking in future work.

Acknowledgements

This work was supported by the National Natural Science Foundation of China (62376080), the Fundamental Research Funds for the Provincial Universities of Zhejiang (GK259909299001-005), the Basic Research Projects of China (JCKY2022211C004, JCKY2023415C009), and the Zhejiang Key Laboratory of Optoelectronic Intelligent Imaging and Aerospace Sensing. Thanks Jiahao Nie and Jinlong Fan for their helpful discussions.