Unified Single-Stage Transformer Network for Efficient RGB-T Tracking

1 Introduction

Visible-Thermal (RGB-T) tracking greatly expands the application scenarios of visual object tracking by using both RGB

∗Corresponding author: Dianxi Shi, dxshi@nudt.edu.cn

and thermal information, significantly improving the tracking performance under challenging conditions such as illumination variation, occlusion, and extreme weather. Therefore, RGB-T tracking has become a research focus in recent years.

Most existing RGB-T tracking methods follow a threestage fusion tracking paradigm which can be shown in Fig. 1(a). These networks separately employ two shallow CNN [He et al., 2015] or Transformer [Dosovitskiy et al., 2020] subnetworks to extract RGB and thermal features from the template and search region. These features are then fused using additional customized modality feature fusion modules to obtain the fusion features. Subsequently, a relation modeling operation between the fusion features of the template and search region, such as online-training [Nam and Han, 2015], cross-correlation [Bertinetto et al., 2016], discriminative correlation [Bhat et al., 2019] and cross-attention mechanism [Hui et al., 2023], will be performed. After relation modeling, the processed search region fusion features are used for prediction. However, the separate subnetworks lead to the lack of interaction between two modalities during the feature extraction stage. As a consequence, the network can only extract regular features from each modality, rather than the dynamic features based on the state of modalities. As shown in Fig. 2, such pattern is not fit to RGB-T tracking especially in complex environments, because different targets have diverse dual-modality appearances, and the appearances of both modalities can change continuously with the environment. Temporary changing or missing appearances in the corresponding modality frequently happened due to the factors like occlusion, illumination variation, or thermal, which leads to the regions covered by the appearances of both modalities are not always consistent. In addition, three-stage fusion tracking paradigm greatly limits the speed improvement.

We propose a unified single-stage Transformer RGB-T tracking network USTrack to solve the above problems. As shown in Fig. 1(b), the core of USTrack is to unify feature extraction, feature fusion, and relation modeling into a single ViT [Dosovitskiy et al., 2020] backbone for simultaneous execution, efficiently obtaining search region fusion features used for prediction. Specifically, we first map the image patches from two modalities to appropriate latent spaces through a dual embedding layer to align the patterns and mitigate the impact of intrinsic heterogeneity to feature fusion. Then, within the attention layers of the ViT backbone, we directly concatenate the token sequences of the four images from the template and search region, upon which we then apply the self-attention operation to the concatenated features. In this self-attention operation, the attention weights between the features of the same image are responsible for extracting modality features, while the weights between the features of images from different modalities are responsible for fusing complementary modality information. The attention weights between the template images and the search region images are responsible for relation modeling. Therefore, we can conveniently unify the three functional stages of RGB-T tracking through the self-attention mechanism for simultaneous execution. This unification of feature extraction and feature fusion alleviates the lack of modality interaction during the feature extraction phase in traditional three-stage RGB-T tracking frameworks, allowing us to directly extract fused features from the template and search region under the modalities interaction. The further unification of fusion feature extraction and relation modeling helps us achieve joint feature extraction, fusion and relation modeling for the first time in the RGB-T tracking networks without designing any complex customized fusion modules, greatly simplifying the current network architecture of RGB-T tracking. The high parallelism of the self-attention also help USTrack achieve a speed more than twice that of existing SoTA methods.

In challenging scenarios, invalid modalities often provide a large amount of noise information. At present, most Transformer-based networks [Xiao et al., 2022; Hou et al., 2022; Hui et al., 2023] directly concatenate or weighted sum the fusion features from search regions of two modalities for final prediction, which inevitably introduces noise information for the final prediction. In order to reduce the impact of noise information, unlike them, we propose a feature selection mechanism based on modality reliability. This mechanism reduces the impact of noise information on prediction by discarding fusion features from invalid modalities. Our contributions are summarized as follows:

• We propose joint feature extraction, fusion, and relation modeling method. It can extract the fusion features of templates and search regions under the interaction of modalities, and simultaneously perform the relation modeling. For the first time, an efficient and concise single-stage fusion tracking paradigm has been provided for RGB-T tracking without the need for designing any customized and specialized feature fusion modules.
• We propose the feature selection mechanism based on modality reliability, which can discard fusion features of invalid modalities according to the modality reliabilities of different tracking environments, thereby reducing the impact of noise information on final prediction and further improving tracking performance.
• USTrack exhibits new state-of-the-art performance on benchmark GTOT [Li et al., 2016], RGBT234 [Li et al., 2019], and VTUAV [Zhang et al., 2022a] while creating the fastest inference speed at 84.2 FPS. In particular, MPR/MSR on the short-term and long-term subsets of VTUAV increased by 11.1%/11.7% and 11.3%/9.7%.

2 RelatedWork

Due to the lack of global perception ability in CNN networks, complementary information cannot be directly aggregated across modalities. So almost all CNN-Based RGBT tracking networks are designed under the three-stage fusion tracking framework. In this part, we briefly summarize the Transformer-based RGB-T tracking methods, which are the most relevant works to us. With the introduction of Transformer into RGB-T tracking, the attention mechanism was initially only used in the feature fusion stage. DRGCNet [Mei et al., 2023] and MIRNet [Hou et al., 2022] use cross-attention to enhance discriminative features from one modality to another, and assign adaptive weights to features of two modalities through gating mechanism to filter redundant and noise information. APFNet [Xiao et al., 2022] proposes an attribute-based progressive fusion network, which enhances the discriminative information specific to challenging attributes through cross-attention. However, the aforementioned Transformer-based RGB-T tracking methods are designed within a detection-based tracking framework[Nam and Han, 2015]. On one hand, during the feature extraction stage, the modality features lack interaction due to the limited global context modeling capability of convolutional neural networks. On the other hand, although some RGBT tracking networks [Hou et al., 2022] based on RT-MDNet [Jung et al., 2018] have almost achieved real-time inference speed, they still follow a three-stage tracking paradigm, which first extracts modality features separately, then fuses features through various attention mechanism, and finally perform the relation modeling operation between the template and search region through online training and continuous fine-tuning, resulting in significant speed bottlenecks for these RGB-T tracking networks.

The latest works TBSI [Hui et al., 2023] and ViPT [Zhu et al., 2023] adopt the powerful RGB tracking network OSTrack [Ye et al., 2022] as their base network architecture, achieving the unification of feature extraction and relation modeling. However, influenced by the three-stage fusion tracking paradigm, they still design the fusion module as a separate component, which is inserted between two Transformer encoders to obtain the fusion features of templates and search regions. It is worth noting that TBSI [Hui et al., 2023] achieves feature fusion by inserting a complex crossattention fusion module between Transformer encoders. This approach alleviates the lack of interaction between modalities during the feature extraction stage and significantly improves the performance. However, due to the extra complex crossattention fusion module, TBSI has only just achieved realtime performance. Unlike ViPT and TBSI, in order to achieve more efficient and concise interaction between modalities during the feature extraction stage, and to enable the fusion tracking network to achieve faster inference speed, despite being inspired by joint feature extraction and relation modeling method [Ye et al., 2022], we completely freed ourselves from the influence of the three-stage fusion tracking paradigm and proposed joint feature extraction, fusion and relation modeling for the first time. We further attempt to unify feature extraction, feature fusion, and relation modeling directly through the self-attention layer within the ViT. It not only can effectively alleviate the problem of lack of interaction between modalities in the feature extraction stage to improve tracking performance, but also fully utilizes the high parallelism of self attention operations, simplifying and accelerating the RGB-T tracking network.

3 Unified Single-Stage RGB-T Tracking

Overview. As shown in Fig. 3, the overall architecture of USTrack consists of three components: a dual embedding layer, a single ViT backbone and the dual prediction heads with feature selection mechanism based on modality reliability. The dual embedding layer uses two learnable embedding layers to map inputs belonging to different modalities to a latent space that is conducive to fusion, reducing the impact of intrinsic heterogeneity of modalities on feature fusion which is based on attention similarity weights. We choose single ViT as our backbone network to achieve joint feature extraction, fusion, and relationship modeling, unifying the three functional stages of RGB-T tracking and providing an efficient single-stage network for RGB-T tracking. Feature selection mechanism based on modality reliability includes two prediction heads and two reliability evaluation modules. It will help the network select search region fusion features generated by the modalities that are more suitable for the current tracking scene for final prediction, reducing the impact of noise caused by invalid modalities on the final results.

3.1 Dual Embedding Layer

The input of USTrack is a pair of target template images and a pair of search region images, containing four images, namely, the RGB template image zrgb image ∈ RHz×Wz×3 , the RGB search region image xrgb image ∈ RHx×Wx×3 , the thermal template image ztimage ∈ RHz×Wz×3 and the thermal search region image xtimage ∈ RHx×Wx×3 . They are first split and flattened into sequences of patches zrgb, zt ∈ RNz×(3P 2 ) and xrgb, xt ∈ RNx×(3P 2 ) , where P ×P is the resolution of each patch, and Nz = HzWz P 2 , Nx = HxWx P 2 are the number of patches of template and search region respectively. Then, two trainable linear layers with parameters Ergb ∈ R(3P 2 )×D and Et ∈ R(3P 2 )×D are used to project zrgb, xrgb and zt, xt into D dimension latent space. The output of this projection are four patch embeddings ˆzrgb, ˆxrgb and ˆzt, ˆxt. Learnable 1D position embeddings Pz and Px are added to the template patch embeddings ˆzrgb, ˆzt and search region patch embeddings ˆxrgb, ˆxt separately, and Learnable 1D modality embeddings Mrgb and Mt are added to the RGB patch embeddings ˆzrgb, ˆxrgb and thermal patch embeddings ˆzt, ˆxt separately. The patch embeddings after adding position and modality embeddings are final features called token embeddings. The above operations can be represented as follows:

After passing the dual embedding layer, RGB template token embeddings ˆzrgb, thermal template token embeddings ˆzt, RGB search region token embeddings ˆxrgb and thermal search region token embeddings ˆxt will be input into the backbone for subsequent processing.

3.2 Joint Feature Extraction & Fusion &Relation Modeling

The self-attention mechanism is the core component of the ViT, and it is also the key to performing joint feature extraction, feature fusion and relation modeling in a single ViT backbone. From the perspective of the self-attention mechanism, we take the RGB search region token embeddings ˆxrgb as an example to further analyze the intrinsic reasons why the proposed network is able to realize simultaneous feature extraction, feature fusion and relation modeling.

In the attention layer, the token sequences ˆxrgb, ˆxt, ˆzrgb, ˆzt from dual embedding layers are concatenated as H = [̂xrgb; ̂xt; ̂ zrgb; ̂ zt ] ∈ R(2Nx+2Nt)×D . Then Selfattention operation is performed on H as follows:

where M is the output of self-attention operation. A is the attention weight. Q, K, and V are query, key and value matrices separately. The superscripts z and x denote matrix items belonging to the template and search region. The subscripts rgb and t denote matrix items belonging to the RGB modality and thermal modality. The calculation of attention weights in Eq. (6) can be expanded to follows:

where the left part of Eq. (8) represents the calculation and the output of attention weights between the RGB search region tokens and the other inputs. the output of self-attention operation can be further written as follows:

where the left part of Eq. (9) is the output corresponding to the RGB search region tokens after the self-attention operation. W xrgb xrgb V xrgb is responsible for aggregating the RGB search region image feature (RGB modality feature extraction). W xrgb xt V xt is responsible for aggregating the thermal modality-specific information based on semantic similarity between two modalities features (feature fusion and modality features interaction). The attention weights can intuitively measure the semantic similarity between modalities. Network can model modality-sharing information based on this similarity. The aggregation of complementary information enables the network to promptly adjust the subsequent extraction of features in RGB search region image. W xrgb zrgb V zt is responsible for aggregating RGB template image feature to further obtain the relation information between the RGB template and the RGB search region (relation modeling based on modality-specific information). W xrgb zt V zt is responsible for aggregating thermal template image feature to further obtain the relation information between the thermal template and the RGB search region (relation modeling based on modalitysharing information). The RGB search region fusion features, which contains relation information, can be used for prediction. Therefore, with the global perception ability of the selfattention, we seamlessly unify feature extraction, feature fusion, and relation modeling into a single ViT backbone. The network can directly extract fusion features of the template and search region under the mutual interaction of modalities, and simultaneously performs relation modeling between fusion features of the template and search region. This alleviates the lack of interaction and guidance between modalities during the feature extraction stage, as well as the problem of additional fusion modules significantly affecting the inference speed of the RGB-T tracking network. Additionally, by inheriting the advantages of relation modeling which is performed by the self-attention, the network can extract more target-specific search region fusion features for prediction under the guidance of two templates.

3.3 Feature SelectionMechanismBased on Modality Reliability

After passing the ViT backbone, two search region fusion features can be obtained for final prediction: Thermal-assisted RGB fusion features based on the RGB search region image, and RGB-assisted thermal fusion features based on the thermal search region image. Both fusion features contain the fusion information of modalities and the relation information between the template and the search region, which can be directly used for target position prediction. To avoid the impact of interference information from invalid modalities on the final prediction. Unlike other networks that obtain fused features based on attention operations, we do not directly concatenate the two fused features or perform weighted sum operations. Instead, we directly discard invalid modality features that are not suitable for the current tracking scene.

As shown in Fig. 3(b), during the training phase, we equip each fusion feature with a prediction head and a reliability evaluation module. We set the same loss for each prediction head and let each reliability evaluation module output a adaptive weight as the modality reliability for the loss of the corresponding prediction head. Then, they are combined into a final total loss function for end-to-end training. This method allows the modality that is not suitable for the current scene to produce inferior results, which will result in larger losses, and then uses the difference between two predictions to guide the modality reliability evaluation module to assign smaller weights to the larger loss by minimizing the overall loss function. Conversely, for fusion features that are more suitable for the current scene, it assigns larger weights. During the testing phase, the network will simultaneously output two results and evaluate the reliability of both modalities. Based on the reliability RRGB and RT , we select the predicted results with higher reliability scores as the final output.

We adopt the prediction head of OSTrack [Ye et al., 2022] directly as our prediction head. The detailed information and corresponding settings can be found in OSTrack. The loss corresponding to the two prediction heads are set as follows:

where LRGB and LT are the loss for each prediction head, LclsRGB and LclsT are the weighted focal loss for classification, L1RGB and L1T are the L1 loss, LgiouRGB and LgiouT are the generalized IoU loss, and λgiou = 2 and λL1 = 5 are the regularization parameters. On the basis, a modality reliability evaluation module is added to each search region fusion features. The evaluation module is a fully convolutional neural network, which consists of several stacked Conv-BNReLU layers. Two modality reliability evaluation modules will output the reliability scores RRGB, RT ∈ R respectively. In order to prevent the model from directly making the weight RRGB and RT zero to minimize the overall loss during the training process, we softmax the reliability scores to obtain the adaptive weight λRGB and λT and overall loss as follows:

where modality reliabilities λRGB and λT are used as the adaptive weights of the loss of the two prediction heads, and the two losses with adaptive weights are combined together as the overall loss to train the model.

4 Experiment

4.1 Experiment Settings

We compare our method with previous state-of-the-art RGBT tracking methods on three benchmarks including VTUAV, RGBT234, and GTOT. GTOT and RGBT234 use success rate SR and precision rate PR as evaluation metrics. VTUAV use Maximum Precision Rate MPR and Maximum Success Rate MSR as evaluation metrics. SR measures the ratio of tracked frames, determined by the Interaction-over-Union (IoU) between tracking result and ground truth. With different overlap thresholds, a success plot (SP) can be obtained, and SR is calculated as the area under curve of SP. MSR adopts the maximum overlap in frame level as the final score. PR measures the percentage of frames whose distance between the predicted position and the ground truth is less than a certain threshold τ. Similar to MSR, MPR adopt the smaller center distance as the final score. τ is set to 20 in our experiment.

Our model is implemented based on Python 3.8, PyTorch 2.0.0. All experiments are conducted on one NVIDIA RTX3090 GPU. We adopt AdamW as the optimizer with 1e-4 weight decay. The learning rate is set as 4e-5 for the backbone and 4e-4 for other parameters. The search regions are resized to 256×256 and templates are resized to 128×128. Each batch size is set to 24, and each epoch contains 30k image pairs. In order to fairly compare our method with other SoTA methods, we aligned our experimental conditions with other methods. We pretrained the network on RGB tracking datasets such as COCO [Lin et al., 2014], LaSOT [Fan et al., 2018], GOT-10k [Huang et al., 2018], and TrackingNet [Muller et al., 2018]. When testing on the GTOT and RGBT234, we only used LasHeR as the training set. When testing on the short-term and long-term testing sets of VTUAV, we only use the training set of VTUAV for training.

4.2 Comparison with SoTAMethods and Analysis

We test our network USTrack on three popular RGB-T tracking benchmarks, comparing performance and speed with the SoTA trackers, such as FSRPN, mfDimp, DAFNet , DAPNet, MANet, CAT, CMPP, JMMAC, MANet++, ADRNet, SiamCDA, M5LNet, TFNet, DMCNet, MFGNet, APFNet, HMFT, MIRNet, ECMD, ViPT and TBSI, to validate the effectiveness of our method. The test results on three datasets show that our method has achieved significant improvements in both performance and inference speed.

Evaluation on VTUAV Dataset. VTUAV dataset is the latest and largest RGB-T tracking dataset, which is currently the only dataset that provides a test for long-term tracking performance of RGB-T tracking methods. Long-term sequences can effectively demonstrate the differences in the target appearances of different modalities, and the persistent changing relationships of two appearances during the tracking process. As shown in Tab. 1 and Fig. 4, despite USTrack being a shortterm tracking network with no template updating or local-toglobal strategies for long-term tracking, we still conduct tests on the short-term and long-term subsets of VTUAV to verify the performance of USTrack. The results were very satisfactory. Compared to the SoTA methods HMFT and HMFTLT, we achieved 11.1%/11.7% and 11.3%/9.7% increases in MPR/MSR on the VTUAV short-term dataset and VTUAV long-term dataset, respectively. Our speed was also 2.78 times and 10.4 times faster than the SoTA method HMFT and HMFT-LT [Zhang et al., 2022a], significantly surpassing the baseline methods of VTUAV. To our knowledge, HMFTLT is currently the only long-term RGB-T tracking method. In comparison, we have achieved significantly better performance, with a tracking speed that is ten times faster than it.

In order to validate that USTrack can adapt well to the diverse dual-modality appearances of targets and the dynamic relationships between modalities, we analyzed the performance of USTrack across all challenging attributes on both short-term and long-term subsets of VTUAV. As shown in Tab. 2 and Tab. 3, in terms of the evaluation metrics MPR/MSR scores on the short-term and long-term datasets of VTUAV, USTrack achieved the highest performance improvements of 23.8%/22.6%&10.5%/10.1%, 39.8%/33.4%&19.1%/16.3%, 20.1%/18.9%&12.2%/10.8%, 11.1%/11.9%&14.9%/13.5%, 12.3%/12.6%&10.6%/9.2% and 5.7%/6.4%&19.7%/16.4% on the challenge attributes deformation (DEF), scale variation (SV), full occlusion (FO), partial occlusion (PO), thermal crossover (TC), and extreme illumination (EI), respectively. For more experimental results of USTrack on VTUAV attributes, please refer to the Appendix. In particular, the DEF and SV attributes effectively demonstrate the differences in the dual-modality appearances of the targets. The FO, PO, TC, and EI attributes can cause the appearance of the corresponding modality to change or disappear, effectively demonstrating the dynamic relationship between two appearances of the target during the tracking process. The most significant performance improvement achieved by USTrack in these attributes effectively proves that the joint feature extraction, fusion and relation modeling method can adapt to diverse dual-modality appearances of targets and the dynamic relationships between modalities by alleviating the lack of modality interaction in the modality feature extraction stage of the three-stage fusion tracking paradigm. Moreover, with the help of the unified single-stage fusion tracking paradigm, USTrack, through a simple network structure and the high parallelism of self-attention operations, has created the fastest inference speed 84.2FPS for RGB-T tracking to date.

Figure 5: (a) This flowchart illustrates the inference process of the feature selection mechanism based on modality reliability on frame 710 of the VTUAV bus 007 video sequence.(b) This figure shows that the feature reliability evaluation module can provide real-time reliability evaluation for each modality in continuous video frames.

Evaluation on RGBT234 Dataset. RGBT234 is currently the most widely used large-scale RGB-T tracking benchmark dataset, consisting of 234 highly aligned videos with about 234K image pairs in total. As shown in Tab.1, Unlike ViPT and TBSI, which also use pure ViT as the backbone but still design customized complex fusion modules under the threestage fusion tracking paradigm, we provide a novel singlestage fusion tracking paradigm that achieves joint feature extraction, fusion, and relation modeling for the first time without the need for any additional feature fusion modules, greatly simplifying the RGB-T tracking network architecture while bringing speed and performance improvements. Compared to the most advanced trackers ViPT and TBSI. We have improved PR/SR on RGBT234 by 3.9%/4.1% and 0.3%/2.1% respectively. TBSI has a speed of 36FPS, while our speed has reached 84FPS. Both performance and speed can prove the effectiveness and efficiency of our method.

Evaluation on GTOT Dataset. GTOT is the first standard dataset in the field of RGB-T tracking. It contains 50 RGB-T video sequences and 7 challenge attributes. We also conducted testing on this dataset and achieved SoTA performance. The test results are shown in Tab. 1. compared with the SoTA methods CMPP and HMFT, our PR/SR scores improved by 0.8%/4.5% and 2.2%/3.4% respectively, while maintaining the fastest inference speed.

4.3 Ablation Experiment and Analysis

Ablation of Dual Embedding Layer. To verify the effectiveness of the dual embedding layer structure, we conducted ablation experiments on the RGBT234 dataset. As a comparison, we have all inputs use the same embedding layer. The results are shown in Tab. 2. The single embedded layer structure resulted in a performance decrease of 1.8% and 2.6% in PR and SR scores. The results show that the use of two independent embedding layers can map the features of two modalities into the latent space conducive to fusion, which can alleviate the impact of the intrinsic heterogeneity of modalities on feature fusion based on attention weight.

Ablation of Feature Selection Mechanism. In order to verify the effectiveness of the feature selection mechanism based on modality reliability, we conducted comparative experiments between our dual prediction head structure with feature selection mechanism and several common prediction head structures on RGBT234. We set up the following comparative experiments, namely, single prediction head based on the fusion features from a single modality search region, single prediction head based on the weighted concatenated fusion features from two search regions, and single prediction head structure based on weighted summation of fusion features from two search regions. All prediction heads have the same structure. To ensure the fairness of the comparative experiment, the weight module used for weighted summation or concatenation of fusion features has the same structure as the modal reliability evaluation module. The experimental results are shown in Tab. 3. Compared to other prediction heads, our dual prediction head structure with feature selection mechanism based on modality reliability performs better. As shown in Fig. 5, we also visualized the actual test sequence, and the visualization showed that our modality reliability had a good correspondence with the real scene, which intuitively reflects the reliability of each modality in the current tracking scene. USTrack will select the fusion features from the search region with high reliability scores to output better prediction results.

5 Conclusion

In this paper, we propose an efficient unified single-stage transformer RGB-T tracking network, USTrack. The core of USTrack is the introduction of the joint feature extraction, fusion and relation modeling approach to address the lack of modality interaction during the feature extraction phase in traditional three-stage fusion tracking paradigms, thereby enhancing the adaptability to diverse dual-modality appearances of targets and the dynamic relationships between modalities. Furthermore, we introduce the feature selection mechanism based on modality reliability. This mechanism discards fusion features generated from ineffective modalities, thereby reducing the impact of noise information on the final prediction to achieve better performance. USTrack has achieved SoTA performance on three mainstream datasets and set a new record for the fastest RGB-T tracking inference speed at 84.2 FPS. Notably, on the VTUAV dataset, which is currently the largest RGB-T tracking dataset, evaluation metrics MPR/MSR has increased by 11.1%/11.7% and 11.3%/9.7%.

Acknowledgments

This work was supported by the Integrated Program of National Natural Science Foundation of China under Grant No.91948303.