Template-based Uncertainty Multimodal Fusion Network for RGBT Tracking

1 Introduction

In recent years, RGBT tracking has attracted increasing attention from researchers due to its broad and practical application prospects in fields such as assisted driving, humancomputer interaction, and security surveillance [Zhang et al., 2024]. The goal of RGBT tracking is to achieve robust allweather and around-the-clock tracking by fully leveraging the complementary advantages of RGB and TIR modalities.

∗Corresponding author

However, the quality of the two modalities fluctuates dynamically in complex scenes. Thus, it is crucial for the designed model to be aware of this for adaption.

Current research on RGBT tracking primarily focuses on the fusion of modality information. These studies can be roughly divided into three categories. The first type of approach [Liu et al., 2023; Chen et al., 2024; Hui et al., 2023a] typically focuses on designing suitable methods to achieve adaptive modal fusion. The second category is to realize the full utilization of different modal information by designing different types of prompts or adapters based on the frozen RGB tracker [Cao et al., 2024; Hou et al., 2024; Zhu et al., 2023]. For instance, BAT [Cao et al., 2024] designs a simple bidirectional adapter, effectively achieving comprehensive fusion of modality features and enhancing the performance of the tracker. The third type of approach [Li et al., 2019b; Li et al., 2024a; Liu et al., 2024] focuses on mining the discriminative features of each modality through feature interaction or attribute decoupling, thereby improving the performance of RGBT trackers. However, existing methods typically generate the dynamic weights of the two modalities, with poor interpretability and difficulty in measuring the quality of the final multimodal outcome. In contrast, uncertaintybased multimodal fusion can effectively reflect the quality of each individual modality as well as the quality of the fused multimodal result to achieve robust multimodal fusion.

Based on the above discussion, we propose an Templatebased Uncertainty Multimodal Fusion Network (TUMFNet) for RGBT tracking, which dynamically evaluates the quality of different modalities by modeling their uncertainty using the reliability of the initial template and Subjective Logic (SL) [Jsang, 2018]. Although the SL theory has made significant progress in uncertainty modeling, current works focus on multimodal classification tasks and is not suitable for directly applying SL to RGBT tracking. Therefore, we propose to employ the reliability of the initial template to explore the uncertainty of different modalities. To this end, we innovatively propose an Uncertainty-Aware Multimodal Fusion Module (UMFM) that leverages the correlation between the template and search region as well as SL to model modal uncertainty. Specifically, UMFM first fuses the template features from both modalities and then uses the integrated template tokens and the search region tokens from each modality to perform correlation matrix computation separately for each modality, resulting in two distinct correlation matrices. Subsequently, UMFM utilizes these correlation matrices to construct the subjective opinions for each modality. To fully leverage the information from both modalities, we introduce an evidential multimodal fusion method to combine the subjective opinions of the two modalities for reliable token-level weight. Meanwhile, we construct modality-level weights based on the uncertainty of the two modalities and the uncertainty of the multimodal fusion. By leveraging both modality-level and token-level weights, we effectively suppress the impact of low-quality modality noise while enhancing the target features.

Additionally, the template plays a crucial role in tracking tasks. Existing tracking methods rely on the initial template and search regions to track the target, making it difficult to capture changes in the target's appearance. To solve this problem, some methods [He et al., 2023; Wang et al., 2024] update the template after a certain interval but fail to ensure the reliability of the template. Other methods [Zhang et al., 2019; Chen et al., 2022] design different modules to assess the reliability of the template but overlook the potential role of the initial template in the template update process. The initial template is inherently reliable, making it reasonable to rely on it to assess the quality of the new template. Therefore, to mine spatio-temporal contextual information from the target and fully leverage the reliable initial template, we design a simple yet effective contrastive template update module. Specifically, we first use the predicted maximum classification score to select candidate templates for template update. Then, we design a simple template quality perceptor that determines whether to update the template by comparing the quality of the initial and candidate templates.

In summary, our major contributions are as follows.

• We propose a novel template-based uncertainty computation framework, which models the uncertainty of different modalities by leveraging the correlation between the template and search region in the Subjective Logic framework.
• We design a novel uncertainty-aware multimodal fusion module, which evaluates the quality of different modalities by constructing modality-specific uncertainty using the correlation between the template and search region, thereby suppressing the interference from low-quality modalities.
• We develop a simple yet effective contrastive template update module that utilizes the initial template to assess the quality of templates and determine whether the dynamic template should be updated.
• Extensive experiments demonstrate that our method outperforms existing RGBT tracking methods on four popular RGBT tracking datasets, including GTOT [Li et al., 2016], RGBT210 [Li et al., 2017], RGBT234 [Li et al., 2019a] and LasHeR [Li et al., 2021].

2 RelatedWork

2.1 RGBT Tracking

RGBT tracking leverages the complementary strengths of both modalities to achieve robust visual tracking, effectively overcoming the limitations of the data from each modality. Existing RGBT tracking methods can be broadly classified into two categories. The first type of approach [Liu et al., 2023; Chen et al., 2024; Hui et al., 2023a; Zhu et al., 2020] achieves effective fusion of features from both modalities by designing specific fusion strategies. TGTrack [Chen et al., 2024] designs a top-down cross-modal guidance attention mechanism to enable interaction between modality information. The second category of methods [Cao et al., 2024; Hou et al., 2024; Zhu et al., 2023; Hong et al., 2024] typically focuses on transferring the capabilities of an RGB tracker to a new RGBT tracker through prompt learning or adapters. BAT [Cao et al., 2024] and SDSTrack [Hou et al., 2024] introduce a simple adapter for multimodal information interaction. The third type of approach [Li et al., 2024a; Liu et al., 2024; Zhang et al., 2021] focuses on enhancing representation learning across different modalities. However, these methods struggle to handle the dynamic changes in modality quality in complex scenarios. Therefore, we propose UMFM, which addresses the variation in modal quality by leveraging the correlation between the reliable template and search region to model the uncertainty of the modalities.

In addition, existing tracking methods typically rely on the initial template to track the target in the search region, making it difficult to adapt to changes in the target's appearance. To address this issue, some methods [He et al., 2023; Wang et al., 2024] update the template at specific intervals, but they struggle to ensure the reliability of the template. Other methods [Zhang et al., 2019; Chen et al., 2022] achieve template update by designing different modules to assess the reliability of the new template. For example, [Chen et al., 2022] propose a regression model to control the template updating process. Although these methods improve tracking performance, they overlook the potential role of the reliable initial template in the template update process. To mine spatio-temporal contextual information of the target and fully leverage the reliable initial template, we propose a simple yet effective contrastive template update module, which determines whether to update the template by comparing the quality of the initial and new templates.

2.2 Uncertainty-aware Multimodal Learning

Uncertainty-aware multimodal learning has developed rapidly in recent years, researchers from different fields have explored the application of the Subjective Logic (SL) theory in their respective tasks fields [Li et al., 2024b; Xu et al., 2024; Kotelevskii et al., 2024]. For instance, [Han et al., 2022] pioneer the application of the SL framework to tackle the issue of low-quality data in multi-view classification tasks. [Xu et al., 2024] further introduce reliable conflictive multiview learning to resolve potential conflicts in multiview data. In other fields, [Li et al., 2024b] propose an adaptive uncertainty learning framework using SL for cross-modal person re-identification. These methods demonstrate the significant advantages of the SL theory in exploring uncertainty. However, directly transferring it to RGBT tracking is not suitable. To bridge this gap, we innovatively propose an uncertainty-aware multimodal fusion module that leverages the correlation between the template and search regions as well as SL to model modal uncertainty.

3 Method

3.1 Framework Overview

The overall structure of the proposed method is illustrated in Figure 1. This framework mainly consists of two core components: the Uncertainty-aware Multimodal Fusion Module (UMFM) and the Contrastive Template Update Module (CTUM). To dynamically perceive the quality of different modalities, UMFM models uncertainty to assess the quality of each modality by the correlation between the template and search region in the Subjective Logic framework. It is worth noting that UMFM is inserted into the 10th, 11th, and 12th encoders of the ViT. In addition, to mine spatio-temporal contextual information from the target during the tracking process, CTUM performs dynamic template update by comparing the quality of the candidate template with the initial template.

3.2 Uncertainty-aware Multimodal FusionModule

Although most existing RGBT tracking methods [Hui et al., 2023a; Cao et al., 2024] have improved tracking performance by designing various modal fusion strategies, they struggle to achieve robust tracking when faced with dynamic changes in modality quality. To address this issue, we propose an Uncertainty-aware Multimodal Fusion Module that models the uncertainty of modalities to perceive changes in modality quality, thereby suppressing low-quality modality information and fully achieve multimodal fusion. It should be noted that we model the uncertainty of different modalities by employing the reliablility of initial template and Subjective Logic [Jsang, 2018].

Subjective Logic Framework. Subjective Logic (SL) provides a formal representation of the uncertainty allocation principle from Dempster-Shafer theory [Liu and Yager, 2008], which is modeled as a Dirichlet distribution. Thus, it offers a method to quantify uncertainty within a wellestablished theoretical framework using the principles of subjective logic. Specifically, we first need to obtain the evidence vector ei for the i-th element. Then, we model the uncertainty u and the belief mass p = {pk}Nk=1 of each element, with the specific process outlined as follows:

where S = ∑Nk=1 (ek + 1) is the the intensity of Dirichlet distribution, pk denotes the belief probability and the parameters of the corresponding Dirichlet distribution are α = {ek + 1}Nk=1 . We can observe that uncertainty is inversely proportional to the total evidence, meaning that as the total evidence increases, uncertainty decreases. Additionally, the Dirichlet distribution represented by α can be defined as follows:

where B(α) represents the N-dimensional beta function, and SN is the N-dimensional unit simplex.

Template-based Uncertainty Calculation. To extend the SL to the RGBT tracking task, the initial step is to construct the evidence vector ei. In particular, we first concatenate the two template features Tr, Tt from both modalities and pass them through a 1x1 convolutional layer (Conv) for feature fusion to obtain the multimodal template features. Then, we calculate the correlation between the search region features Sr, St of each modality and the multimodal template features Tf , obtaining two correlation matrices, Cr ∈ RNt×Ns and Ct ∈ RNt×Ns . Nt and Ns represent the number of tokens in the template and search region, respectively. Considering that the correlation matrix reflects the correlation between search region tokens and multimodal template tokens, we first compute the mean of the two correlation matrices along the column dimension and process them using the ReLU activation function. In this way, we obtain the overall correlation between the search region tokens and the reliable multimodal template features, which is the evidence ei = {ei}Ns i=1 we need, as shown Figure 1. The specific processing procedure is as follows:

where eri and eti represent the evidence vectors of the RGB modality and the TIR modality, respectively.

Following the subjective logic framework mentioned above, we can obtain the parameters αi of the Dirichlet distribution as well as model the uncertainty u of each modality. Thus, we obtain the subjective opinions for each modality. The specific process is as follows:

where S = ∑Ns i=1 (αi) is the intensity of Dirichlet distribution.

Uncertainty-based Multimodal Fusion. To suppress the noise from low-quality modalities as well as to achieve effective multimodal fusion, we introduce a simple evidential multimodal fusion method. The process is as follows:

where bri , ari , ur represent the belief mass, Dirichlet distribution parameters, and uncertainty corresponding to the RGB modality, respectively. Similarly, bmi , ami , um represent the belief mass, Dirichlet distribution parameters, and uncertainty corresponding to the multimodal fusion. In addition, we obtain modality-level weights wr , wt based on uncertainty to suppress noise from low-quality modalities. Meanwhile, to further enhance the target features, we also introduce a tokenlevel weight wm. The detailed computation process is as follows:

where Sm = ∑Ns i=1 (αmi ) is the intensity of Dirichlet distribution. Finally, to achieve effective uncertainty-aware learning, we propose an novel loss function Lu, which supervises the learning process, as shown below:

where αm represents the Dirichlet distribution parameter, y denotes the mask of the target token, α1 refers to the relevant parameters for the target, and α2 refers to the relevant parameters for the background. We present the uncertainty scores for a sequence, as shown in Figure 2. It can be observed that under strong light or jitter, the quality of the RGB modality deteriorates, resulting in higher uncertainty scores, while the uncertainty scores of the TIR modality remain relatively stable.

3.3 Contrastive Template Update Module

In existing tracking methods [Zhang et al., 2019; Wang et al., 2024], template update is often used to capture the appearance changes of the target, thereby achieving robust tracking. However, current methods typically rely on classification scores and IoU scores to determine whether to update the template, overlooking the potential role of the initial template in the template update process. To explore the spatiotemporal context information and fully leverage the reliable initial template, we design a simple yet effective CTUM including a template quality perceptor and a dual-template contrastive training strategy.

Specifically, the template quality perceptor consists of two linear layers, one ReLU layer and one Softmax layer, as shown in Figure 1. To effectively train the perceptor for perceiving the quality of different templates, we design the dual-template contrastive training strategy. During the training phase, for each sampling from the sequence, we select two distinct templates and a search region. Each of the different templates from the same sequence will interact with the same search region to extract features. The features of the two different templates are then input into the perceptor for quality perception, resulting in the quality scores Qk, Qj for both templates. In addition, to facilitate the learning of the perceptor, we generate pseudo labels based on the IoU derived from the prediction results. For two identical search regions Sk, Sj, we perform interactions with different templates. If the IoU obtained by a search region feature after being input into the predictor is higher, we infer that the template interacting with this search region exhibits superior quality. Therefore, the label generation process is described as follows:

where IoUk and IoUj the IoU of the prediction results for the k-th and j-th search regions, respectively. In this way, we obtain the labels indicating the relative quality of the two templates.

In the tracking phase, our template update strategy is as follows:

• Candidate Template Selection. The classification score output by the tracking model reflects the discriminability between the foreground and background, and many methods [Wang et al., 2024] use it as a criterion to assess the
• reliability of the prediction results. Therefore, we use the maximum classification score to obtain the candidate template. Specifically, if the maximum classification score of the current frame exceeds the average of the past classification scores, we consider the current prediction to be relatively reliable and update the candidate template accordingly.
• Quality-contrast Template Update. Although the maximum classification score enables continuous updating of the candidate template, its actual reliability needs further evaluation. The template quality perceptor assesses the quality of both the initial and candidate templates to determine whether the dynamic template should be genuinely updated. The specific template update process is shown in Figure 1. In the tracking process, we performs candidate template quality assessment every N frames. Specifically, for the N-th frame, both the initial template and the candidate template interact with the same two N2 -th frame. The features Ti, Tc of both templates are then input into the perceptor, which predicts the initial template quality score Qi and candidate template quality score Qc. If Qc > Qi, the dynamic template is updated. We present the template update process for three sequences, as shown in Figure 3. It can be observed that when the target in the template is occluded or tracking fails, the perceptor correctly determines whether the current candidate template should be discarded.

3.4 Loss Function

Our method adopts the traditional center point prediction method, where the regression and classification branches predict the target bounding box, and the model learns using both classification and regression losses. Additionally, we introduce a novel loss function LU for the learning of UMFM. To effectively train the template quality perceptor, we also introduce a binary cross-entropy loss LBCE. The overall loss function is as follows:

where Lcls denotes the weighted focal loss, L1 represents the L1 loss, and the generalized IoU loss is represented by Lgiou. The parameters λ1 and λ2 are kept unchanged according to the settings in previous works [Ye et al., 2022]. The parameter λt = min(0.1, t/T ) ∈ (0, 0.1] is the annealing coefficient, t is the index of the current training epoch, and T is the annealing step. In our experiments, λ1 is set to 5, λ2 to 2 and λ3 is set to 0.01.

4 Experiment

4.1 Implementation Details

We take OSTrack [Ye et al., 2022] as the base tracker, which employs ViT as the backbone network for feature extraction. In addition, we use the parameters from DropMae[Wu et al., 2023] as the pre-trained weights. Our model is implemented using PyTorch and experiments are conducted on one RTX 4090 GPU. We train the overall tracking network end-to-end using the LasHeR training set to evaluate GTOT, RGBT210, RGBT234, and LasHeR test set. The input search region and template sizes of the model are 256×256 and 128×128, respectively. The learning rate for the backbone network is set to 1×10-5 , while the learning rate for the other parameters is set to 1×10-4 . The model is trained for a total of 15 epochs. Additionally, we use the AdamW optimizer with a weight decay of 1 × 10-4 . Note that our UMFM is inserted into the 10th, 11th, and 12th blocks of the ViT. λ3 is set to 0.01 and N is set to 20.

Inference: During the inference phase, at every interval of N frames, the input of the model consists of the initial template, dynamic template, and the search region. When performing quality assessment of the candidate template every N frames, the input of model includes the initial template, dynamic template, candidate template, the search region of the N-th frame, and the search region of the N2 -th frame.

4.2 Evaluation Datasets andMetrics

GTOT [Li et al., 2016] dataset is the earliest publicly available RGBT tracking dataset, containing a total of 50 sequences and approximately 15,000 frames. RGBT210 [Li et al., 2017] dataset consists of 210 pairs of RGBT video sequences, totaling 209.4K frames, and includes annotations across 12 attributes. RGBT234 [Li et al., 2019a] dataset extends RGBT210 dataset, offering more precise annotations. It contains a total of 234 pairs of RGBT video sequences, amounting to approximately 233.4K frames. LasHeR [Li et al., 2021] is a large RGBT tracking dataset, consisting of 1,224 aligned video sequences with approximately 1,469.6K frames. It includes 245 test sequences and 979 training sequences, covering 19 real-world challenge attributes.

Following the previous works [Liu et al., 2023; Hui et al., 2023b], we use Precision (PR) and Success Rate (SR) as the main evaluation metrics in one-pass evaluation for quantitative performance analysis, which are commonly employed in current RGBT tracking tasks. To consider target size, we normalize PR to obtain Normalized Precision (NPR) for evaluating tracking performance on the LasHeR dataset.

4.3 Comparisons With State-of-the-Art Methods

We evaluate our algorithm on four popular RGBT tracking datasets and compare its performance with current state-ofthe-art methods.

Evaluation on GTOT dataset. The comparison of the experimental results in GTOT dataset is shown in Table 1, where our method outperforms the current state-of-the-art methods, with gains over HMFT, QAT and US-Track in PR/SR by 4.3%/5.3%, 4.0%/4.7%, and 2.1%/1.9%, respectively. We further compare our method with the CKD tracker, which eliminates modality gaps. In terms of PR and SR, our method outperforms CKD by 2.3% and 3.0%, respectively.

Evaluation on RGBT210 dataset. We compare our method with 7 existing RGBT trackers on RGBT210 dataset, and the results are shown in Table 1. Specifically, compared to TBSI, QAT, and TATrack, our method achieves significant improvements of 5.4%/3.3%, 3.9%/3.9%, and 5.4%/4.0% in PR/SR, respectively. We further compare our method with the topperforming approach, CKD, and observe a performance improvement of 2.3%/0.6% in PR/SR.

Evaluation on RGBT234 dataset. To further validate the effectiveness of our method, we compare it with other 17 state-of-the-art RGBT trackers on RGBT234 dataset. When comparing our method with QAT, BAT, and US-Track on RGBT234 dataset, it is observed that our method achieves improvements of 2.4%/3.4%, 4.0%/3.7%, and 3.7%/4.1% in PR/SR, respectively. Finally, when comparing with the bestperforming method on RGBT234 dataset, CKD, our method outperforms it by 0.8% in PR and 0.4% in SR.

Evaluation on LasHeR dataset. We compare our method with 13 state-of-the-art RGBT trackers on the most challenging LasHeR dataset, and the evaluation results are shown in Table 1. Our method significantly outperforms existing advanced methods due to its effective perception of modality quality and its ability to fully utilize dynamic templates to extract spatio-temporal information, achieving state-of-theart performance with PR, NPR, and SR scores of 76.4%, 72.7%, and 61.4%, respectively. Compared to TATrack and BAT, our method exceeds them by 5.2% and 5.3% in both PR and SR metrics. We also compare our method with the topperforming tracker on this dataset, CKD. Our method outperforms CKD by 3.2%/3.4%/3.3% in terms of PR/NPR/SR.

Additionally, to further validate the effectiveness of the proposed method, we visualize the tracking results of four sequences with multiple challenges, as shown in Figure 4. We can observe that in the sequence manoncall, when the target is heavily occluded, BAT, ViPT, and SDSTrack fail to track the target, while only our method and TBSI manage to locate the target accurately. When faced with the sequence boyaftertree, includes challenges such as occlusion and low-light conditions, all other methods fail to track the target, while only our method achieves stable tracking.

4.4 Ablation Study

Analysis of Different Components. We first analyze the effectiveness of the designed UMFM and CTUM, with the results presented in Table 2. It can be seen that when UMFM is incorporated into the baseline, the model performance shows significant improvement. With the addition of CTUM, the performance is further enhanced due to the utilization of spatio-temporal information.

Analysis of Hyperparameters. We first evaluate different inserting layers of our proposed UMFM and summarize the experimental results in Table 3. As UMFM is added layer by layer, it can be observed that the overall performance of the model gradually improves. We then explore the impact of the interval N on the performance of the proposed method, and the experimental results are shown in Table 4. It can be observed that when the interval N is set to 20, the performance of our method achieves the best result. Compared to the other two template selection methods, although they also achieve significant performance gains, our method demonstrates a more prominent improvement in performance. For more experimental results and visualizations, please refer to the supplementary materials*.

5 Conclusion

In this work, we propose a novel template-based uncertainty framework consisting of an uncertainty-aware multimodal fusion module and a contrastive template update module, for RGBT tracking. UMFM utilizes the reliablility of initial template and Subjective Logic to model the uncertainty of different modalities, addressing the issue of dynamic modality quality changes in complex scenarios. CTUM explores the role of the initial template in the template update process, facilitating efficient dynamic template update. Extensive experiments on four publicly available RGBT tracking datasets demonstrate the effectiveness of our method compared to other state-of-the-art RGBT trackers. In the future, we will continue to explore novel and effective dynamic template update strategies and integrate the uncertainty-aware multimodal fusion module into other tracking frameworks.

* https://github.com/dongdong2061/IJCAI25-TUMFNet

Acknowledgments

This work was supported by the Natural Science Foundation of China (No.62376004) and the Natural Science Foundation of Anhui Province (No.2208085J18).