SparseTT: Visual Tracking with Sparse Transformers

1 Introduction

Visual tracking aims to predict the future states of a target given its initial state. It is applicable broadly, such as humancomputer interactions, video surveillance, and autonomous driving. Most of the existing methods address the tracking problem with sequence prediction frameworks where they estimate the current state based on the initial and the previous states. Thus, it is important to give accurate states in every time slice otherwise errors accumulate and will lead to tracking failure. Significant efforts have been devoted to improving the tracking accuracy, i.e., the accuracy of the target bounding boxes. However, challenges such as target deformation, partial occlusion, and scale variation are still huge obstacles ahead hindering them from perfect tracking. The reason may be that most of these methods adopt crosscorrelation operation to measure similarities between the target template and the search region, which may trap into local optimums. Recently, TransT [Chen et al., 2021] and DTT [Yu et al., 2021] improve the tracking performance by replacing the correlation with Transformer [Vaswani et al., 2017]. However, building trackers with Transformers will lead to a new problem: the global perspective of self-attention in Transformers causes the primary information (such as targets in search regions) under-focused, but the secondary information (such as background in search regions) over-focused, making the edge region between the foreground and background blurred, and thus degrade the tracking performance.

∗Contact Author

In this paper, we attack this issue by concentrating on the most relevant information of the search region, which is realized with a sparse Transformer. Different from vanilla Transformers used in previous works [Chen et al., 2021; Yu et al., 2021], sparse Transformer is designed to focus on primary information, enabling the targets to be more discriminative and the bounding boxes of targets to be more accurate even under severe target deformation, partial occlusion, scale variation, and so on, as shown in Fig. 1.

Summarily, the main contributions of this work are threebox prediction problem and solve it with an encoder-decoder transformer, in which the encoder models the global spatiotemporal feature dependencies between targets and search regions, and the decoder learns a query embedding to predict the spatial positions of the targets. It achieves excellent performance on visual tracking. TrDiMP [Wang et al., 2021] designs a siamese-like tracking pipeline where the two branches are built with CNN backbones followed by a Transformer encoder and a Transformer decoder, respectively. The Transformers here are used to enhance the target templates and the search regions. Similar to previous siamese trackers, TrDiMP applies cross-correlation to measure similarities between the target templates and the search region, which may impede the tracker from high-performance tracking. Noticing this shortcoming, TransT [Chen et al., 2021] and DTT [Yu et al., 2021] propose to replace cross-correlation with Transformer, thereby generating fused features instead of response scoress. Since fused features contain rich semantic information than response scores, these methods reach much accurate tracking than previous siamese trackers.

fold.

• We present a target focus network that is capable of focusing on the target of interest in the search region and highlighting the features of the most relevant information for better estimating the states of the target.
• We propose a sparse Transformer based siamese tracking framework that has a strong ability to deal with target deformation, partial occlusion, scale variation, and so on.
• Extensive experiments show that our method outperforms the state-of-the-art approaches on LaSOT, GOT10k, TrackingNet, and UAV123, while running at 40 FPS, demonstrating the superiority of our method.

2 RelatedWork

Siamese Trackers. In siamese visual trackers, crosscorrelation, commonly used to measure the similarity between the target template and the search region, has been extensively studied for visual tracking. Such as naive cross-correlation [Bertinetto et al., 2016], depth-wise crosscorrelation [Li et al., 2019; Xu et al., 2020], pixel-wise cross-correlation [Yan et al., 2021b], pixel to global matching cross-correlation [Liao et al., 2020], etc. However, crosscorrelation performs local linear matching processes, which may fall into local optimum easily [Chen et al., 2021]. And furthermore, the cross-correlation captures relationships and thus corrupts semantic information of the inputted features, which is adverse to accurate perception of target boundaries. Most siamese trackers still have difficulties dealing with target deformation, partial occlusion, scale variation, etc.

Transformer in Visual Tracking. Recently, Transformers have been successfully applied to visual tracking field. Borrowing inspiration from DETR [Carion et al., 2020], STARK [Yan et al., 2021a] casts target tracking as a bounding Self-attention in Transformers specializes in modeling long-rang dependencies, making it good at capturing global information, however, suffering from a lack of focusing on the most relevant information in the search regions. To further boost Transformer trackers, we alleviate the aforementioned drawback of self-attention with a sparse attention mechanism. The idea is inspired by [Zhao et al., 2019]. We adapt the sparse Transformer in [Zhao et al., 2019] to suit the visual tracking task and propose a new end-to-end siamese tracker with an encoder-decoder sparse Transformer. Driven by the sparse attention mechanism, the sparse Transformer focuses on the most relevant information in the search regions, thus suppressing distractive background that disturbs the tracking more efficiently.

3 Method

We propose a siamese architecture for visual tracking, which consists of a feature extraction network, a target focus network, and a double-head predictor, as shown in Fig. 2. The feature extraction network is a weight-shared backbone. The target focus network built with a sparse Transformer is used to generate target-focused features. The double-head predictor discriminates foreground from background and outputs bounding boxes of the target. Note that our method runs at a real-time speed as no online updating in the tracking phase.

3.1 Target Focus Network

The target focus network is built with sparse Transformer, and it has an encoder-decoder architecture, as shown in Fig. 3. The encoder is responsible for encoding the target template features. The decoder is responsible for decoding the search region features to generate the target-focused features.

3.2 Encoder

Encoder is an important but not essential component in the proposed target focus network. It is composed of N encoder layers where each encoder layer takes the outputs of its previous encoder layer as input. Note that, in order to enable the network to have the perception of spatial position information, we add a spatial position encoding to the target template features, and input the sum to the encoder. Thus, the first encoder layer takes the target template features with spatial position encoding as input. In short, it can be formally denoted as:

where Z ∈ RHtWt×C represents the target template features, Penc ∈ RHtWt×C represents the spatial position encoding, f i enc represents the i-th encoder layer, Y i-1 enc ∈ RHtWt×C represents the output of the (i - 1)-th encoder layer. Ht and Wt are the height and width of the feature maps of target templates, respectively.

In each encoder layer, we use multi-head self-attention (MSA) to explicitly model the relations between all pixel pairs of target template features. Other operations are the same as the encoder layer of vanilla Transformer [Vaswani et al., 2017].

3.3 Decoder

Decoder is an essential component in the proposed target focus network. Similar to the encoder, the decoder is composed of M decoder layers. However, different from the encoder layer, each decoder layer not only inputs the search region features with spatial position encoding or the output of its previous decoder layer, but also inputs the encoded target template features outputted by the encoder. In short, it can be formally denoted as:

where X ∈ RHsWs×C represents the search region features, Pdec ∈ RHsWs×C represents the spatial position encoding, Y Nenc ∈ RHtWt×C represents the encoded target template features outputted by the encoder, f idec represents the i-th decoder layer, Y i-1 dec ∈ RHsWs×C represents the output of (i - 1)-th decoder layer. Hs and Ws are height and width of the feature maps of search regions, respectively.

Different from the decoder layer of vanilla Transformer [Vaswani et al., 2017], each decoder layer of the proposed sparse Transformer first calculates self-attention on X using sparse multi-head self-attention (SMSA), then calculates cross-attention between Z and X using naive multihead cross-attention (MCA). Other operations are the same as the decoder layer of vanilla Transformer [Vaswani et al., 2017]. Formally, each decoder layer of the proposed sparse Transformer can be denoted as:

3.4 Sparse Multi-Head Self-Attention

Sparse multi-head self-attention is designed to improve the discrimination of foreground-background and to alleviate ambiguity of edge regions of foreground. Concretely, in the naive MSA, each pixel value of attention features is calculated by all pixel values of the input features, which makes the edge regions of foreground blurred. In our proposed SMSA, each pixel value of attention features is only determined by K pixel values that are most similar to it, which makes foreground more focused and the edge regions of foreground more discriminative.

Specifically, as shown in the middle of Fig. 4, given a query ∈ RHW×C , a key ∈ RC×H′W ′ , and a value ∈ RH′W ′×C , we first calculate similarities of all pixel pairs between query and key and mask out unnecessary tokens in the similarity matrix. Then, different from naive scaled dot-product attention that is shown in the left of Fig. 4, we only normalize K largest elements from each row of the similarity matrix using softmax function. For other elements, we replace themwith 0. Finally, we multiply the similarity matrix and value by matrix multiplication to get the final results.

The upper right and the lower right in Fig. 4 show examples of normalizing a row vector of the similarity matrix in naive scaled dot-product attention and sparse scaled dot-product attention, respectively. We can see that naive scaled dot-product attention amplifies relatively smaller similarity weights, which makes the output features susceptible to noises and distractive background. However, this issue can be significantly alleviated by sparse scaled dot-product attention.

3.5 Double-Head Predictor

Most existing trackers adopt fully connected network or convolutional network to classification between foreground and background and regression of target bounding boxes, without indepth analysis or design for the head networks based on the characteristics of the tasks of classification and regression. Inspired by [Wu et al., 2020], we introduce a doublehead predictor to improve the accuracy of classification and regression. Specifically, as shown in Fig. 2, it consists of a fc-head that is composed of two fully connected layers and a conv-head that is composed of L convolutional blocks. Unfocused tasks are added for extra supervision in training. In the inference phase, for the classification task, we fuse the classification scores outputted by the fc-head and the one outputted by the conv-head; for the regression task, we only take the predicted offsets outputted by the conv-head.

3.6 Training Loss

We follow [Xu et al., 2020] to generate training labels of classification scores and regression offsets. In order to train the whole network end-to-end, the objective function is the weighted sum of classification loss and regression loss, as the following:

where ωfc, λfc, ωconv and λconv are hyper-parameters. In practice, we set ωfc = 2.0, λfc = 0.7, ωconv = 2.5, λconv = 0.8. The functions Lclass fc and Lclass conv are both implemented by focal loss [Lin et al., 2017], and the functions Lbox fc and Lbox conv are both implemented by IoU loss [Yu et al., 2016].

4 Experiments

4.1 Implementation Details

Training Dataset. We use the train splits of TrackingNet [Muller et al., 2018], LaSOT [Fan et al., 2019], GOT10k [Huang et al., 2019], ILSVRC VID [Russakovsky et al., 2015], ILSVRC DET [Russakovsky et al., 2015] and COCO [Lin et al., 2014] as the training dataset, in addition to the GOT-10k [Huang et al., 2019] benchmark. We select two frames with a maximum frame index difference of 100 from each video as the target template and the search region. In order to increase the diversity of training samples, we set the range of random scaling to [ 1 1+α , 1 + α] and the range of random translation to [-0.2β, 0.2β], in which α = 0.3, β = √(1.5wt + 0.5ht) × (1.5ht + 0.5wt) for the target template, and β = t·s √(1.5ws+0.5hs)×(1.5hs+0.5ws) for the search region. Here wt and ht are the width and height of the target in the target template, respectively; ws and hs are the width and height of the target in the search region, respectively; t and s are the sizes of the target template and the search region, respectively. We set t = 127 and s = 289 in practice.

Model Settings. We use the tiny version of Swin Transformer [Liu et al., 2021] (Swin-T) as the backbone ϕ. In the MSA, SMSA, and MCA, the number of heads is set to 8, the number of channels in the hidden layers of FFN is set to 2048, and the dropout rate is set to 0.1. The number of encoder layers N and the number of decoder layers M are set to 2, and the sparseness K in SMSA is set to 32. See Sec. 4.2 for more discussions about the hyper parameters in the proposed target focus network. In the conv-head of the double-head predictor, the first convolutional block is set to residual block [He et al., 2016], and other L - 1 ones are set to bottleneck blocks [He et al., 2016], where L = 8.

Optimization. We use AdamW optimizer to train our method for 20 epochs. In each epoch, we sample 600,000 image pairs from all training datasets. Note that we only sample 300,000 image pairs from the train split for the GOT-10k benchmark. The batch size is set to 32, and the learning rate and the weight decay are both set to 1 × 10-4 . After training for 10 epochs and 15 epochs, the learning rate decreases to 1×10-5 and 1×10-6 , respectively. The whole training process takes about 60 hours on 4 NVIDIA RTX 2080 Ti GPUs. Note that the training time of TransT is about 10 days (240 hours), which is 4× that of our method.

4.2 Ablation Study

The Number of Encoder Layers. In our method, the encoder is used to enhance the generalization of target template, thus the number of encoder layers is important to our method. Tab. 1 lists the performance of our method using different numbers of encoder layers. Interestingly, the proposed target focus network can still bring comparable performance without the encoder. As the number increases, the performance gradually improves. However, when the number of encoder layers is greater than 2, the performance drops. We argue that excess encoder layers may lead to overfitting of model training. Therefore, we set the number of encoder layers to 2 in the remaining experiments.

The Number of Decoder Layers. We then explore the best setting for the number of decoder layers M, as shown in Tab. 2. Similar to N, as the number of decoder layers increases, the performance gradually improves when M is not greater than 2. We also notice that when M equals 3, the performance decreases and the running speed slows down by large margin. We speculate that it may be caused by overfitting. Thus, M is set to 2 in the remaining experiments.

The Sparseness K in SMSA. In SMSA, the sparseness K significantly affects the activation degree of foreground. Due to the scale variation of targets, a suitable sparseness K ensures good adaptability and generalization at the same time for SMSA. Tab. 3 shows the impact of different sparseness values on the performance of our method. Note that when K = H′W ′ , SMSA becomes naive MSA. We find that SMSA always brings better performance than MSA in our method, which shows the effectiveness and superiority of SMSA. When K is 32, Our method achieves the best performance. Consequently, we set the sparseness K to 32 in our experiments.

4.3 Comparison with the State-of-the-art

LaSOT is a large-scale long-term dataset with high-quality annotations. Its test split consists of 280 sequences, the average length of which exceeds 2500 frames. We evaluate our method on the test split of LaSOT and compare it with other competitive methods. As shown in Tab. 4, our method achieves the best performance in terms of success, precision, and normalized precision metrics.

We also evaluate our method on the test subsets with attributes of deformation, partial occlusion, and scale variation. The results are shown in Tab. 8. As can be seen, our method performs best in the above challenging scenarios, significantly surpassing other competitive methods. These challenges bring ambiguous of determining accurate boundaries of targets thus making the trackers hard to locate and estimate target bounding boxes. However, our method copes with these challenges well.

GOT-10k contains 9335 sequences for training and 180 sequences for testing. Different from other datasets, GOT10k only allows trackers to be trained using the train split. We follow this protocol to train our method and test it on the test split, then report the performance in Tab. 5. We see that our method surpasses the second-best tracker TransT by a significant margin, which indicates that our method is superior to other methods when annotated training data is limited.

UAV123 is a low altitude aerial dataset taken by drones, including 123 sequences, with an average of 915 frames per sequence. Due to the characteristics of aerial images, many targets in this dataset have low resolution, and are prone to have fast motion and motion blur. In spite of this, our method is still able to cope with these challenges well. Thus, as shown in Tab. 6, our method surpasses other competitive methods and achieves the state-of-the-art performance on UAV123, which demonstrates the generalization and applicability of our method.

OTB2015 is a classical testing dataset in visual tracking. It contains 100 short-term tracking sequences covering 11 common challenges, such as target deformation, occlusion, scale variation, rotation, illumination variation, background clutters, and so on. We report the performance of our method on OTB2015. Although the annotations is not very accurate and it has tended to saturation over recent years, as shown in Tab. 6, however, our method still outperforms the excellent tracker TransT [Chen et al., 2021] and achieves comparable performance.

TrackingNet is a large-scale dataset whose test split includes 511 sequences covering various object classes and tracking scenes. We report the performance of our method on the test split of TrackingNet. As shown in Tab. 7, our method achieves the best performance in terms of success metric.

4.4 Qualitative Comparison of SMSA andMSA

To intuitively explore how SMSA works, we visualize some self-attention maps of search regions in Fig. 5, in which the 1-st column and the 4-th column are the search regions, the 2-nd column and the 5-th column are the attention maps generated by SMSA and naive MSA, respectively. For better visualization, we combine the 1-st column and the 2-nd column in the 3-rd column and combine the 4-th column and the 5-th column in the 6-th column. We can see that, compared with MSA, SMSA pays more attention to primary information.

5 Conclusions

In this work, we boost Transformer based visual tracking with a novel sparse Transformer tracker. The sparse self-attention mechanism in Transformer relieves the issue of concentration on the global context and thus negligence of the most relevant information faced by the vanilla self-attention mechanism, thereby highlighting potential targets in the search regions. In addition, a double-head predictor is introduced to improve the accuracy of classification and regression. Experiments show that our method can significantly outperform the stateof-the-art approaches on multiple datasets while running at a real-time speed, which demonstrates the superiority and applicability of our method. Besides, the training time of our method is only 25% of TransT. Overall, it is a new excellent baseline for further researches.

Acknowledgments

The work was supported by the National Key Research and Development Program of China under Grant 2018YFB1701600, National Natural Science Foundation of China under Grant U20B2069 and 62176017.