Learning Target-aware Representation for Visual Tracking via Informative Interactions

1 Introduction

As one of the most fundamental tasks in computer vision, visual object tracking (VOT) aims to estimate the trajectory of the designated target in a video sequence [Smeulders et al., 2013; Li et al., 2013]. Embracing powerful deep networks for appearance modeling, unprecedented achievement has been witnessed in the past years in tracking community. One representative paradigm, namely two-stream network, has been widely adopted in recent tracking methods, e.g., Siamese and Correlation-Filter based approaches [Bertinetto et al., 2016; Danelljan et al., 2019]. The consensus for constructing a robust two-stream tracker is to learn an informative visual representation, i.e., "feature matters" [Wang and Yeung, 2013].

∗Equal Contribution.

†Corresponding author.

For tracking task, an informative feature requires the target representation clearly distinguishable from background and objects with similar semantics (i.e., distractors). Fortunately, other vision tasks, e.g. classification, detection and segmentation, also enjoy sharp contrast between background and objects. A new paradigm from the basic classification task, i.e., backbone network, can thus be seamlessly transferred to advance other fields. Object tracking also benefits from this bonus, where different backbone networks are introduced to learn strong visual representation [Zhang and Peng, 2019; Li et al., 2019]. But itch of scabies for tracking mostly attributes to surrounding distractors. Besides better feature extractors, a tracking algorithm should also consider how to reinforce target-relevant representation as well as avoiding negative influence from distractors. Previous works, e.g., SiamAttn [Yu et al., 2020], try to alleviate this issue by injecting prior knowledge of the reference image (i.e., the first frame) to enhance features of candidate frame. However, only using the target-guided candidate feature for matching without direct feedback to backbone network is not enough. The potential problem is that the branch corresponding to the candidate image cannot well "sense" information of the reference target in its feature learning. Besides, the accumulation effect of target-irrelevant interference in the shallow stages may degrade the feature quality of deeper layers. In a nutshell, the under utilization of target information during representation learning may compromise the ability of filtering unrelated distractors, which results in inferior performance.

In this paper, we set out to address the aforementioned issues by proposing a mechanism called "interaction inside the backbone network (InBN)". As discussed, the anaemic distractor-aware ability at the early learning stages blames on less exposure to prior information. Therefore, one intuitive but rational solution is to increase the branch-wise interaction frequency inside the backbone network of a twostream paradigm. In our design, we take inspiration from recent arising vision Transformer architecture [Dosovitskiy et al., 2020], which is adept on modeling global visual relations [Luo et al., 2016] and can naturally process features from multi-modalities. The proposed general interaction modeler, dubbed as GIM, strengthens the representation of candidate image by learning its relation with reference representation. The relation learning is realized by the contextaware self-attention (CSA) and target-aware cross-attention (TCA) modules (as described later).

For the first time, we integrate the cross-processing modeler into different stages of a backbone network in visual tracking, which can continuously expose prior information to backbone modeling, as shown in Fig. 1. The multiple interactions pattern improves the distractor-aware ability of the early stages at a backbone network through feature aggregation in CSA and relation learning in TCA. Notably, the proposed method can not only adapt to a CNN network but also work well with a Transformer backbone. When equipping the transformer backbone with the proposed GIM and InBN mechanism, we surprisingly observe that complicated matching networks as in TransT [Chen et al., 2021] and AutoMatch [Zhang et al., 2021] are not necessary, where a simple cross-correlation can show promising results.

A series of experiments are conducted to prove the generality and effectiveness of the proposed LiBN and GIM. Taking SiamCAR [Guo et al., 2020] as the baseline tracker, we directly apply GIM and LiBN to its CNN backbone, achieving 3.2 points gains on LaSOT (53.9 vs 50.7). When employing Transformer backbone, our model further improves the performance to SUC of 65.7 on LaSOT.

The main contributions of this work are as follows:

• We present the first work demonstrating that information flow between reference and candidate frames can be realized inside the backbone network, which is clearly different from previous feature matching way that does not provide direct feedback to backbones in visual tracking.
• We prove the effectiveness of the proposed GIM module and InBN mechanism on both CNN and Transformer backbone networks, which makes it a general modeler to improve representation qualities in tracking.

2 RelatedWork

Visual Object Tracking. The Siamese network has been widely used in visual object tracking in recent years [Henriques et al., 2008; Bertinetto et al., 2016; Li et al., 2018; Li et al., 2019; Zhang et al., 2020; Zhang and Peng, 2019; Chen et al., 2021; Danelljan et al., 2019; Bhat et al., 2019]. SiamFC [Bertinetto et al., 2016] first adopts cross-correlation in the Siamese framework, which tracks a target by template matching. By exploiting the region proposal network (RPN) [Ren et al., 2015], SiamRPN [Li et al., 2018] and its variants [Guo et al., 2020; Yu et al., 2020] achieve more precise target scale estimation and fast speed. Later, further improvements upon Siamese trackers have been made, including enhancing feature representations by deeper convolutional backbones [Li et al., 2019; Zhang and Peng, 2019], designing more practical update mechanism [Zhang et al., 2019], introducing anchor-free regression to target estimation [Guo et al., 2020; Zhang et al., 2020], replacing cross-correlation with the automatically learned matching networks [Zhang et al., 2021], introducing vision Transformer [Chen et al., 2021], and so on. Another important branch, i.e., online trackers, always construct models with discriminative correlation filter [Danelljan et al., 2017; Danelljan et al., 2019; Bhat et al., 2019]. Despite of their success, only relying on a sole backbone to extract image representation without any interactive learning may introduce distractors and limit the tracking performance. In this work, we introduce a general interaction modeler to alleviate this issue.

Vision Transformer. Transformer is originally proposed in [Vaswani et al., 2017] for the task of machine translation. Recently it has been introduced into vision tasks and shows great potential. The core idea of Transformer is selfattention mechanism, which takes a sequence as the input, and builds similarity scores of each two feature vectors as attention weights to reinforce or suppress visual representations of corresponding positions. ViT [Dosovitskiy et al., 2020] and its follow ups [Liu et al., 2021; Chu et al., 2021], adopt a convolution-free transformer architecture for image classification, which processes the input image into a sequence of patch tokens. In addition to classification, Transformer is also used in other vision tasks and achieves comparable performances, such as object detection [Carion et al., 2020; Sun et al., 2020b], semantic segmentation [Strudel et al., 2021], multiple object tracking [Sun et al., 2020a], etc. In visual tracking, TrDiMP [Wang et al., 2021a] uses transformer to learn temporal relations between frames, and TransT [Chen et al., 2021] employs cross-attention in matching process to enhance semantic messages. Transformer is naturally suitable for processing multi-modality features. Surprisingly, no prior works in visual tracking consider building interactions inside the two-stream backbone network. The information in the reference image (or template image named in some works) is crucial to explore more target-relevant clues and track targets in complex scenes, especially in the presence of similar distractors. Our work aims to narrow these gaps by proposing an general interaction modeler which promotes the representation of candidate image at different stages of the backbone network.

3 Tracking with General InteractionModeler

In this section, we elaborate on the proposed General Interaction Modeler (GIM) and its integration to different backbone networks via the proposed InBN mechanism for improving representation learning in visual tracking.

3.1 General InteractionModeler

General Interaction Modeler (GIM) contains three essential components including window process (WP), context-aware self-attention (CSA) module and target-aware cross-attention (TCA) module, as illustrated in Fig. 2. The WP module is designed to decrease computational complexity and meanwhile increase the local perception ability. The CSA module aggregates context information by global attention modeling. The TCA module enhances target-related features and suppresses distractor responses by injecting prior knowledge of the reference image to representation learning. In the following, we will detail each module one by one.

Local Perception via Window Process (WP)

Our GIM is built upon the Transformer-like structure as described above. The Achilles' Heel is its quadratic cost for global attention learning and lack of local perception ability [Liu et al., 2021; Chu et al., 2021]. Motivated by recent vision transformers [Liu et al., 2021], we design the window process to merge feature vectors of non-overlapping local areas before attention modules.

Window process (WP) first partitions the input feature f ∈ RB×H×W×C (B is the batch size) by performing nonoverlapping win × win windows (win = 7) on it. Then each group containing win2 feature vectors is regarded as a modeling unit and forms a new dimension with size of fgroup ∈ RB×win2× H win × W win ×C . To fit the requirement of following attention modules, we flatten the spatial dimension of fgroup to fWP ∈ RB×win2× HW win2 ×C , whose sequence length is HW win2 . We perform attention on dimensions of HW win2 and C instead of win2 and C as in Swin-Trans [Liu et al., 2021], which ensures the feasibility to compute the crossattention weights Wattn ∈ RB×win2× HxWx win2 × HzWz win2 with Einstein Summation Convention [Weisstein, 2014] (z and x represent the reference and candidate images respectively). By regarding each window as a computing unit and then modeling relations between different windows, local clues are used without affecting global receptive fields.

Context-aware Self-attention (CSA)

The proposed CSA aggregates context information by performing multi-head self-attention (MHSA) on the partitioned input feature fWP . In particular, given an input feature sequence fseq ∈ RB×HW×C , the inner single-head attention function first maps it into query Q ∈ RB×NQ×d , key K ∈ RB×NK×d and value V ∈ RB×NV ×d (NQ, NK and NV are sequence lengths) with mapping weights WQ,WK,WV ∈ RC×d , then performs dot-product on them,

where d is the number of channels. In our model, we directly split the features along channel dimension for multiple heads, which is computed as follows,

where Qi ∈ RB×NQ× dn , Ki ∈ RB×NK× dn , Vi ∈ RB×NV × dn are divided parts, and Wmap ∈ Rd×C is mapping weights. We employ n = 4, C = 256, and d = 256 to achieve real-time tracking speed without other specified.

CSAs are applied on features of both reference and candidate images. It computes similarity scores between each two positions and uses the scores as attention weights to aggregate context visual features. By considering fWP as a sequence, the sequence length after WP is HW win2 instead of original HW, which has linear cost and promotes modeling speed. In a nutshell, the mechanism of CSA for the input fWP ∈ RB×win2× HW win2 ×C can be summarized as

where WQ,WK,WV ∈ RC×d are mapping weights, fCSA ∈ RB×win2× HW win2 ×C is the output of CSA.

Target-aware Cross-attention (TCA)

As mentioned before, Siamese tracking is conducted on a two-stream network, which is different from the typical onestream classification model. Therefore, it is reasonable to consider the interaction between these two branches during backbone design. In this work, we propose the target-aware cross-attention (TCA) to inject the reference information to the branch of the candidate image, which guides the backbone to perceive the target and filter irrelevant distractors.

For implementation, TCA takes two window-processed features (fxWP ∈ RB×win2× HxWx win2 ×C for candidate image and fzWP ∈ RB×win2× HzWz win2 ×C for reference image) as inputs and maps them to Q and K,V respectively. Through multi-head attention, fxWP are refined by the prior knowledge in fzWP , where the target-related ones are strengthened and the irrelevant ones are suppressed. TCA is formulated as,

where WQ,WK,WV ∈ RC×d are mapping weights, fTCA ∈ RB×win2× HxWx win2 ×C is the output of TCA.

Here we analyze how WP could make the computation of attention learning linear complexity. Without loss of generality, we assume H%win = 0 and W%win = 0. Taking each window as the computing unit, the sequence length is HW win2 . CSA or TCA first computes relations between feature vectors of the same position in different windows, and the computation cost is O( H2W 2 win4 d). Then the total cost for win2 positions is O( H2W 2 win2 d). If we let k1 = H win and k2 = W win , the cost can be computed as O(k1k2HWd), which is significantly more efficient when k1 ≪ H and k2 ≪ W and grows linearly with HW if k1 and k2 are fixed.

3.2 Building Backbone Network

To demonstrate the generality and effectiveness of the proposed GIM, we follow the InBN mechanism and build two types of backbone networks for Siamese tracking, i.e., CNN and Transformer as shown in Tab. 1. To unleash the targetperception ability of shallow layers, we adopt a hierarchical design to employ the GIM, which is named as InBN mechanism. We arrange the proposed GIM to different stages of a tracking backbone to conduct multiple interactions. Based on the reference presentation of different resolutions, the network knows what target is the one being tracked, which eventually enhances target-related messages and suppresses distractors hierarchically.

For the CNN-based model, we take the ResNet-50 in SiamCAR [Guo et al., 2020] as the basic CNN backbone. Following the InBN paradigm, we modify it by directly applying the GIM after each stage. The mapping layers are attached at each stage to unify the output feature dimensions to 256 (see more details in [Guo et al., 2020]).

For the Transformer-based model, we refer to the structure of Swin-Trans [Liu et al., 2021] to build two-stream backbone by stacking the GIMs. As illustrated in Tab. 1, the backbone network is divided into four stages, which contain 2, 2, 6, 2 GIMs, respectively. Notably, we only use the TCA module in the last GIM of each stage, which avoids bringing high computation costs. The InBN mechanism allows backbone to enhance target-relevant ability gradually by hierarchical interactions. For the output features of each stage, we first map their channel dimensions into the same number of 256 with a 1 × 1 linear layer. We upsample or downsample these features to same resolution to align spatial dimension. Finally, we concatenate all features along channel dimension and use a 1×1 linear transformation to shrink channel number to 256.

3.3 Comparison with Other Tracking Backbones

To our best knowledge, this is the first tracking framework that considers branch-wise interactions inside the backbone network for better feature representation. As shown in Fig. 1, we categorize existing backbone networks in visual tracking into two types. The first one (see the first row in Fig. 1), which is employed in the most trackers, simply uses a backbone to extract feature representation and then performs matching without direct feedback for tracking. The lack of reference information makes it hard to robustly locate the target object. Differently, equipping with GIM with InBN mechanism, our backbone network (see the second row in Fig. 1) is able to not only perform multiple interactions to bridge the target information to candidate frame during feature learning, but also exploit global modeling to enrich target-related representation, resulting in improved feature representations in all layers for better performance, yet with negligible computation.

4 Experiments

4.1 Implementation Details

For CNN-based tracker, we apply the proposed Interaction Modeler to the baseline tracker SiamCAR [Guo et al., 2020] (named as CNNInMo). For transformer-based tracker (named as TransInMo), we build backbone network following the structions in Sec. 3.2. A simple depth-wise correlation layer is used for feature matching. Two three-layer MLPs are respectively conducted for classification and regression heads, as similar in [Chen et al., 2021]. Moreover, we test the influence of complicated matching process by equipping TransInMo with the matching module from TransT [Chen et al., 2021] (named as TransInMo*). The training settings for CNNInMo/TransInMo follow the baseline trackers SiamCAR and TransT, respectively (we recommend the readers to [Guo et al., 2020] and [Chen et al., 2021] for more details). In particular, the training splits of LaSOT [Fan et al., 2019], TrackingNet [Muller et al., 2018] and GOT-10k [Huang et al., 2019] and COCO [Lin et al., 2014] are used during learning.

4.2 State-of-the-art Comparisons

We compare our methods with recent state-of-the-art trackers on six tracking benchmarks. The detailed comparison results on the LaSOT [Fan et al., 2019], TNL2K [Wang et al., 2021b], TrackingNet [Muller et al., 2018], UAV123 [Mueller et al., 2016], NFS [Kiani Galoogahi et al., 2017] and OTB100 [Wu et al., 2015] are reported in Tab. 2 and Tab. 3. Notably, CNNInMo runs at 47 fps on GTX2080Ti GPU, while TransInMo/TransInMo* run at 67/34 fps respectively. LaSOT is a large-scale tracking benchmark containing 1,400 videos. We evaluate different tracking algorithms on its 280

testing videos. As reported in Tab. 2, when applying GIM and InBN to SiamCAR without any other modifications, CNNInMo obtains gains of 3.2/2.9 points on success (SUC) and precision (P) scores, respectively. This evidences the effectiveness of the proposed interaction module and mechanism. Our transformer version with simple cross-correlation matching outperforms recent state of the arts TransT [Chen et al., 2021] and TrDiMP [Wang et al., 2021a].

TNL2K is a recently proposed large-scale benchmark for tracking by natural language and bounding box initialization. We evaluate our tracker on its 700 testing videos. CNNInMo outperforms the baseline tracker SiamCAR for 6.9/3.5 points on SUC and precision, respectively.

UAV123 contains 123 aerial sequences captured from a UAV platform. As shown in Tab. 2, the proposed methods (Transformer versions) perform the best among all compared trackers. The CNN version CNNInMo exceeds 1.5 SUC points than the baseline tracker SiamCAR.

NFS consists of 100 challenging videos with fast-moving objects. We evaluate the proposed trackers on the 30 fps version, as presented in Tab. 2. TransInMo and TransInMo* obtain the best two performances on success and precision scores, which shows that our method can improve the robustness of tracking fast-tracking moving targets.

TrackingNet and OTB100 We further evaluate proposed trackers on TrackingNet and OTB100. As presented in Tab. 3, CNNInMo surpasses the baseline SiamCAR for 6.8 points on TrackingNet. TransInMo and TransInMo* are on par with (slightly better than) TransT on these two benchmarks.

4.3 Ablation Study

Components in GIM. As shown in Tab. 4, we analyze the influence of WP and TCA in GIM based on TransInMo*. CSA is the basic modeling layer in GIM, hence we only ablate the other two modules. Tab. 4 shows that without WP and TCA, the tracker obtains an AUC score of 61.1 on LaSOT. When integrating WP into the model, it brings 3.5 points gains (② vs ①) with faster running speed (37 fps vs 30 fps). The TCA module builds branch-wise interaction between the reference and candidate images inside the backbone. As in Tab. 4, TCA brings gains of 2.1 points on AUC (③). When applying WP and TCA together, further improvement is obtained with an AUC score of 65.7 (④), which proves the rationality of our design.

Matching Process. We ablate different matching process for our trackers as shown in Tab. 5. With simple depth-wise correlation (DW) (①, i.e. TransInMo), the tracker has already achieved compelling performance and speed. We then replace DW with more complex matching modules in TransT with different Transformer encoder numbers. Surprisingly, it (⑥) does not obtain significant gains. As we decrease the encoder layers, the performance is even worse than simple DW (①). Our framework indicates that a tracker with InBN mechanism reduces the demand of fussy matching network to filter target-irrelevant distractors. Matching inside the backbone may be a worth choice in future work.

Activation Map Visualization. To better understand our CSA and TCA, we visualize their activation maps in Fig. 3. As shown, when the target appearance changes heavily or similar distractors exist, the branch-wise interaction of TCA can help the backbone sense the target-relevant areas. Then self-attention of CSA employs the context messages to filter useless messages and focus on the target area.

HanningWindow Influence. In Fig. 4, we present the influence of post-processing (i.e. Hanning window) used in our TransInMo* and TransT on two benchmarks. From Fig. 4, we observe that the proposed TransInMo* is more stable. In the case without post-processing (i.e. weight=0), our tracker still shows compelling performance, which indicates the effectiveness of feature representation learned by our backbone.

Due to limited space, more experiments with in-depth analysis, e.g., tracking result visualization, failure case analysis, are presented in the appendix.

5 Conclusions

This work presents a novel mechanism that conducts branchwise interactions inside the visual tracking backbone network (InBN) via the proposed general interaction modeler (GIM). We prove that both the CNN network and the Transformer backbone can enjoy bonus brought by InBN. The backbone network can build more robust feature representation under complex environments with the improved target-perception ability. Our method achieves compelling tracking performance by applying the backbones to Siamese tracking.

Acknowledgments

This work was supported by the National Key Research and Development Program (2020AAA0106800), the Beijing Natural Science Foundation under Grant (Z180006, L211016), the National Natural Science Foundation of China under Grant (62176020) and CAAI-Huawei MindSpore Open Fund. This work is co-supervised by Prof. Liping Jing and Dr. Zhipeng Zhang.