Modality-Guided Dynamic Graph Fusion and Temporal Diffusion for Self-Supervised RGB-T Tracking

1 Introduction

The RGB-T tracking task, which leverages the complementary strengths of RGB images (rich texture information) and thermal infrared images (enhanced nighttime perception), has gained increasing attention in recent years [Lai et al., 2024; Hu et al., 2024]. However, the reliance on manual annotation for both modalities is labor-intensive and resourcedemanding [Zhang et al., 2024], and the inherent inconsistency in cross-modal annotation further complicates the creation of high-quality RGB-T datasets. Consequently, the development of robust self-supervised methods for RGB-T tracking [Li et al., 2024] has become crucial and offers significant practical advantages.

∗Corresponding author.

To fully leverage information in images [Xi et al., 2023], a class of mainstream methods relies on pseudo-labels [Zheng et al., 2021; Shen et al., 2022; Zhang et al., 2025] for supervision. USOT [Zheng et al., 2021] relies on unsupervised optical flow to detect moving objects and applies a dynamic programming algorithm to establish interframe associations, thereby generating pseudo-labels to provide supervision. However, a discrepancy between the pseudo-labels and ground truth is inevitable. For example, comparing the accurate red bounding box in Fig.1(a) with the pseudo green bounding box in Fig.1(b), it is evident that the pseudo-label incorrectly includes both the object and a similar moving object, thereby further exacerbating performance degradation. As illustrated in Fig. 1(d), when training with pseudo-labels, a significant performance degradation is observed in TBSI [Hui et al., 2023], ViPT [Zhu et al., 2023], and AFter [Lu et al., 2024]. Obviously, this performance degradation stems from pseudo-label noise, which arises from two primary factors: (1) inaccurate pseudo-labels may fail to capture whole object regions and always introduce excessive background noise, and (2) inaccurate annotation caused by interference from similar objects will significantly degrade the model performance. To address these two problems, we designed a selfsupervised RGB-T tracking method as shown in Fig. 1(c).

In fact, label noise also exists in fully supervised tasks, where misalignment between modality labels can introduce label noise. GMMT [Tang et al., 2024] models noise as Gaussian noise, and uses a generative model [Song et al., 2020; Cao et al., 2025] to enhance its ability to perceive noise. However, when dealing with pseudo-labels, simple Gaussian noise is insufficient for modeling the two types of labeling errors mentioned above. To address the pseudo-label noise issue, we propose our method. Specifically, to tackle the first issue, we introduce the Modality-Guided Dynamic Graph Fusion (MDGF) module, which dynamically adjusts the adjacency matrix based on the similarity between the input RGB and thermal infrared modalities. By leveraging a dynamic adjacency matrix to guide graph attention in mining and focusing on object regions, a fusion that supports tracking is achieved, even in the presence of inaccurate annotations. Moreover, to study the second issue, we draw inspiration from GMMT [Tang et al., 2024] and propose a Temporal Graph-Informed Diffusion (TGID) module, which treats fused information from neighboring frames as noise within a diffusion model. This approach helps the model to identify interference from similar objects and train it to be more robust to such noise. Furthermore, the MDGF module complements the TGID module to prevent information loss and enhance overall model performance. In summary, our main contributions are as follows:

• We propose a novel self-supervised RGB-T tracker GDSTrack, which leverages multi-modal optical flow to extract pseudo-labels for coarse localization, significantly reducing the cost of manual annotation.
• We design the MDGF module to simulate interference from similar objects in pseudo-labels and utilize the generative capability of the TGID module for denoising and fine localization, achieving a balance between performance and cost.
• Ablation studies and comparative experiments on four benchmarks demonstrate the state-of-the-art performance of our method and its ability to enhance selfsupervised learning.

2 RelatedWork

Self-supervised Tracking. RGB-T tracking has advanced significantly owing to the complementary nature of RGB and thermal infrared modalities. [Lai et al., 2024] leverage spatio-temporal contextual modeling through long-range cross-frame integration and short-term historical trajectory prompts. [Hu et al., 2024] introduce a temporal state generator that produces temporal information tokens to guide the localization of the object in the next time state. Self-supervised RGB-T tracking [Li et al., 2024] addresses the challenge of requiring large-scale manual annotations for tracking tasks. However, this topic has not received sufficient research attention. Most self-supervised tracking methods rely primarily on the RGB modality alone. The UDT [Wang et al., 2019] exploits the discriminative power of correlation filtering for tracking and employs cycle consistency as a self-supervised signal. Existing deep trackers can be trained using synthesized data in routine ways, without requiring human annotation. [Sio et al., 2020] exploit the fact that an image and any cropped region of it naturally form a pair for self-training. To replace naive cropping methods such as center cropping, [Zheng et al., 2021] propose a more accurate pseudo-label generation method based on optical flow and dynamic programming techniques. Building on this, [Zhang et al., 2025] learn a prompt representation of an object using a pre-trained diffusion model. However, these methods do not account for interference caused by similar objects in the absence of ground-truth labels. Our method tackles the root cause of the noise by simulating distractor noise using the modal fusion results of similar objects, thereby enhancing the robustness of the model to such interference.

RGB-T Fusion aims to fully utilize the complementary characteristics of RGB and thermal infrared images to enhance feature extraction and improve performance in downstream tasks. In terms of the fusion level, [Zhang et al., 2019] consider several fusion mechanisms at the pixel, feature, and response levels. In terms of fusion methods, [Xu et al., 2021] use a channel attention mechanism to implement the adaptive calibration of feature channels before realizing hierarchical feature fusion. [Tang et al., 2023] propose a novel model for infrared and visible image fusion via a dual attention Transformer to represent long-range context information. [Tang et al., 2024] seek to uncover the potential of generative techniques to address the critical challenge in multi-modal tracking. [Chen et al., 2024] predict bounding boxes through a sequence-to-sequence framework. [Zhu et al., 2023] introduce prompt learning for multi-modal fusion tracking method. In terms of frequency domain fusion, [Zhao et al., 2023b] decompose modality-specific and modality-shared features to better reflect the semantic information contained in the high-frequency and low-frequency features. However, these methods do not account for the fusion process and focus on irrelevant areas because of the lack of ground truth labels. Our method can dynamically generate adjacency matrices, guiding the graph attention model to focus more on object regions.

3 Methodology

3.1 Framework Overview

Our model performs self-supervised tracking by fully leveraging the advantages of both RGB and infrared features through the Modality-Guided Dynamic Graph Fusion and Temporal Graph-Informed Diffusion modules, as shown in Fig. 2. GDSTrack obtains the interaction features through the encoder. The RGB and infrared images share encoder parameters. We first use the AMG module to dynamically adjust the adjacency matrix based on the features of the two modalities, guiding graph attention in MDGF to fuse the information from both modalities. The TGID module then denoises similar object noise simulated by the MDGF module. Finally, the output is fed into the tracking head for tracking. GDSTrack framework takes two RGB images and their corresponding infrared images as input: the visible template frame zv ∈ R3×Wz0 ×Hz0 cropped from the first visible frame xv1 ∈ R3×Wx0 ×Hx0 , and the infrared template frame zi ∈ R1×Wz0 ×Hz0 cropped from the first infrared frame xi1 ∈ R1×Wx0 ×Hx0 . The second visible search frame xv2 ∈ R3×Wx0 ×Hx0 , the second infrared search frame xi2 ∈ R1×Wx0 ×Hx0 , where Wz0 and Hz0 are half of Wx0 and Hx0 , respectively. We obtain the RGB and infrared search frame features, denoted as f v x1 ∈ RWxHx×dmodel and f ix1 ∈ RWxHx×dmodel , respectively, after the interaction between the template and the search frame through the encoder. The MDGF module performs an initial fusion of the inputs f v x1 ∈ RWxHx×dmodel and f ix1 ∈ RWxHx×dmodel , producing two layers of fused results f l1 x1 and f S1 x1 .

where MDGF(·) indicates the Modality-guided Adaptive Graph Fusion Module (see Sect. 3.2). f S1 x1 is processed through the tracking head to obtain the tracking score ScoreS1 and the predicted bounding box BS1 :

where we use a series of Conv-BN-ReLU layers to independently estimate the top-left and bottom-right corners following [Cui et al., 2022]. We employ the Generalized Intersection over Union (GIoU) loss and L1 loss to guide tracker training. The loss function in stage one is defined as follows:

Where ˆB is our pseudo label, and we set the loss weight λ1 to 2 and set λ2 to 5 following [Zhao et al., 2023a]. The third search frame is then introduced to obtain the corresponding visible and thermal infrared features, named as f v x2 ∈ RWxHx×dmodel and f ix2 ∈ RWxHx×dmodel .

We first use the MDGF module to obtain the first-layer and second-layer fusion results, denoted as f l1 x2 and f S1 x2 :

We then utilize the TGID module to further denoise f S1 x2 from the MDGF to obtain more refined fused features f S2 x2 :

where TGID(·) indicates the Temporal Graph-Informed Diffusion Module (see Sect. 3.3). We use f v x2 and f ix2 as the initial conditions for the diffusion model, with f l1 x2 from the first layer of MDGF serving as the conditions for the intermediate layers of the denoising process. f S1 x1 is set as distractor noise, contributing to the noise during the added noise process. f S2 x2 is used to obtain the tracking results through DiffHead:

where DiffHead shares the same structure as Head in MDGF. Then, we employ pseudo-labels ˆB to supervise the training:

We use the predicted bounding boxes ˆBS1 from the MDGF to supervise the training process of the diffusion model:

The final loss function in stage two is defined as follows:

where LDM is the generative loss.

3.2 Modality-guided Dynamic Graph Fusion

The MDGF consists of an Adjacency Matrix Generator and a Graph Attention Network. AMG generates a dynamic adjacency matrix based on the similarity between RGB and infrared modalities to guide the graph attention network. This similarity-based fusion method enables GDSTrack to focus more on the coherent regions of the object, even in the absence of ground truth labels.

Adjacency Matrix Generator. The AMG module employs Cosine Similarity to create a mask that guides the crossattention mechanism in generating the corresponding adjacency matrix according to feature variations (see Fig. 3).

We concatenate f v x and f ix along the sequence dimension:

where we get f vi x ∈ R2WxHx×dmodel . Then, we compute the similarity Sim1 between f v x and f ix using the scaled dot product attention [Vaswani et al., 2017]:

where the projections are the parameter matrices W Q ∈ Rdmodel×dk and W K ∈ Rdmodel×dk . 1 √dk is a scaling factor and dk is 2048. We then calculate the Cosine Similarity Sim2 between f v x and f ix :

We sum the two similarities and then shift their values to the range [0, 1]:

where M ∈ R2WxHx×2WxHx .

We use M as a mask, setting the similarity values to less than the threshold θ to negative infinity, and retaining the similarity values from Sim1 otherwise:

Then we get the final similarity:

We then set the Top-K positions with the largest similarity values in the final similarity matrix to 1, while setting all other positions to zero:

To ensure the symmetry of adjacency A′ :

where A ∈ R2WxHx×2WxHx .

Graph Attention Networks. We utilize a two-layer Graph Attention Network (GAT) [Velickovic et al., 2017] to perform graph-based attention fusion of visible and thermal infrared features under the guidance of the adjacency matrix generated by the AMG module:

GAT takes each pixel in the concatenated modalities f vi x as a graph node, and the adjacency matrix A preserves both intramodal and inter-modal relationships. The input features are processed using weighted learning, where the attention mechanism adaptively assigns weights based on the correlations between features.

In the first layer, a learnable matrix W1 ∈ Rdmodel×nhid is used to perform a linear transformation of features f vi x . Then, a shared attentional mechanism and softmax function is performed following the GAT model. Similarly, a learnable vector W2 ∈ Rnhid×outmodel is used to perform a linear transformation of the output of the first layer, f layer1 x .

To reduce information loss, the output of the second layer f l2 x is added to the visible and infrared features after the dimension reduction using a 1 × 1 convolution as P roj(·):

where split is the inverse of Concatenation.

3.3 Temporal Graph-Informed Diffusion

To address the noise from similar objects during pseudo-label training, we designed the TGID module. It treats the MDGF results of nearby frames as noise using the first-layer MDGF features to improve robustness to noise and prevent information loss. Specifically, after obtaining the fusion results f S1 x2 from MDGF, we introduce a diffusion model DDIM [Song et al., 2020] to denoise f S1 x2 . By leveraging the generative capabilities of the diffusion model, we achieved fusion results that are more conducive to tracking. We use f S1 x2 as the input and train the model with the features of the two modalities f v x2 and f ix2 as conditions following [Tang et al., 2024], where we use a 1 × 1 convolution as a projector for feature dimensionality reduction. We also use the first-layer output of MDGF f l2 x2 as another condition. Additionally, distractor object noise f S1 x1 is introduced during the diffusion process to enhance the robustness of the model against similar distractor.

Diffusion process. Due to the motion in video frames, the fusion result of the neighboring frame f S1 x1 is offset at the object location compared to the fusion result of the current frame f S1 x2 . This natural offset can serve as a similarity distractor for the current frame, enhancing the robustness of the model. Based on this observation, in addition to the original Gaussian noise, we incorporate the first-stage fusion result f S1 x1 as part of the noise in the diffusion process.

In the forward diffusion process, we use the MDGF fusion output f S1 x2 as x0. Then x0 undergoes diffusion through the random Gaussian noise ¯ zt and the distractor noise from the first-stage fusion result of the neighboring frame f S1 x as dt:

Denoising process. Some recent works [Cao et al., 2025] utilize additional information as conditions for generative models to guide the denoising process of the model. We employ a UNet [Ronneberger et al., 2015] network to perform the denoising process in the model following [Tang et al., 2024]. We design a simple module named the Condition Fuse (ConFuse) model to serve as supplementary information for the intermediate layers of the UNet network. We use the firstlayer graph convolution results f l1 x generated in MDGF as middle conditions in the denoising process:

The features are downsampled using Conv2(·) to match the size of the intermediate layers of the UNet network named D, then concatenated with D along the channel dimension. Subsequently, Conv1(·) is used to reduce the dimensionality of the features, and finally, they are added to D to prevent information loss.

We calculate the L2 loss between noise and the output of the denoising process:

4 Experiments

Datasets. GTOT dataset [Li et al., 2016] comprises 50 pairs of visible and thermal infrared video sequences and corresponding ground truth annotations. These sequences are designed to examine diversity and biases across seven specific challenges. Attribute-based annotations allow for a more detailed evaluation of tracker performance. RGBT234 [Li et al., 2019] encompassing 234 sequences, with the longest sequence containing up to 8k frames. This dataset is annotated with 12 attributes, making it a robust benchmark for assessing the effectiveness of different trackers. LasHeR dataset [Li et al., 2021] featuring 1,224 pairs of visible and thermal infrared videos, with a total of 730k frame pairs. It introduces 19 realworld challenge attributes, providing a more extensive evaluation framework than GTOT and RGBT234. VTUAV [Zhang et al., 2022] is a large-scale benchmark specifically designed for visible-thermal UAV tracking, including 500 sequences and 1.7 million high-resolution frame pairs. Its diversity encompasses various categories and scenes, supporting exhaustive evaluations that cover short-term tracking, long-term tracking, and segmentation mask prediction.

Metric. Following [Wang et al., 2023; Lu et al., 2024], we evaluate our tracker using three metrics: Precision rate (PR), Normalized precision rate (NPR), and Success rate (SR). PR measures the proportion of frames where the Euclidean distance between the predicted object's center and the ground truth center is below a predefined threshold. Specifically, we use a threshold of 5 pixels for GTOT and 20 pixels for RGBT234 and LasHeR. PR is further normalized as NPR. SR assesses the intersection ratio between the predicted bounding box and the ground truth bounding box, also referred as the overlap ratio. It quantifies the percentage of frames where the overlap exceeds a certain threshold. For all four datasets, we calculate the area under the success rate curve (AUC) to derive the success score.

Implementation Details. Our model is implemented using the PyTorch platform. We use LasHeR [Li et al., 2021] as our training dataset. Following [Zheng et al., 2021], we generate pseudo labels and confidence scores for both the visible and infrared datasets. The label with the higher confidence score is selected as the final pseudo label. We use ViT-B/16 [Dosovitskiy et al., 2020] as the feature encoder for interaction between the template frame and the infrared frame. The encoder is initialized with the pre-trained parameters proposed in [Zhao et al., 2023a]. The template size and search region size are set to 128 × 128 and 256 × 256. Training is carried out using the AdamW optimizer [Loshchilov, 2017]. The batch size is 16, and the learning rate is set to 0.00003. We first train the MDGF module and its corresponding tracking head. Starting from the 10th epoch, the parameters of the ViT encoder are unfrozen. From the 22nd epoch, the encoder, MDGF, and tracking head parameters are frozen, and only the parameters related to the diffusion model and the tracking head (diffuse-head) are trained. The training is guided by the MDGF tracking results and pseudo-labels. The training concludes at the 50th epoch. The weight decay coefficient is set to 0.0001, and the momentum value is set to 0.9.

4.1 Main Results

We present the main experimental results here, and additional results can be found in the appendix.

Comparisons with the State-of-the-Art. As shown in Tab. 1, we compared our method with other state-of-the-art selfsupervised RGB-T tracking methods, including KCF [Henriques et al., 2014]+ RGBT, MEEM [Zhang et al., 2014] + RGBT, TBSI∗ [Hui et al., 2023], AFter∗ [Lu et al., 2024], ViPT∗ [Zhu et al., 2023], UDT [Wang et al., 2019] + RGBT, UDT-FF [Li et al., 2023], USOT [Zheng et al., 2021] + RGBT and S2OTFormer [Li et al., 2024]. We re-trained AFter [Lu et al., 2024], ViPT [Zhu et al., 2023] and TBSI [Hui et al., 2023] on the LasHeR dataset using the same RGB-T pseudolabels as our method, named as AFter∗ , ViPT∗ and TBSI∗ for comparison. It can be observed that our model achieves state-of-the-art performance on the RGBT234, LasHeR, and VTUAV datasets.

It can be seen that our method exceeds the self-supervised method S2OTFormer (68.4% / 47.7%) by 2.5% on PR and 0.8% on SR on the RGBT234 dataset. It also surpasses the S2OTFormer (39.8% / 35.4% / 29.5%) by 6.1% on PR, 3.9% on NPR, and 5.9% on SR on the LasHeR dataset. It also surpasses the S2OTFormer (56.7% / 44.5%) by 15.6% on the PR score and 15.3% on the SR score on the VTUAV dataset. We hypothesize that the suboptimal performance on the GTOT dataset is due to the smaller number and shorter sequence lengths in this dataset. We compared our model with other state-of-the-art self-supervised RGB-T trackers on the RGBT234 dataset. As shown in Fig. 5, our model achieved the best performance on both the PR plots and the SR plots.

Attribute Analysis. The performance of our method GDSTrack, along with other sota self-supervised methods on different attribute challenges of VTUAV dataset is shown in Fig. 4. There is no OV attribute in the Short-term Evaluation of the VTUAV dataset, so we only draw twelve attributes. As observed, our method achieves the best performance on most attributes of the VTUAV dataset with only 2.1% of PR lower on extreme illumination (EI) attribute than UDT-FF.

Speed Analysis. As shown in Tab. 2, GDSTrack achieves better performance than S2OTFormer while maintaining comparable speed.

Visualization Analysis. We conducted visualization on the RGBT234 dataset. As shown in Fig. 6, for better visualization, we magnify the objects within the dashed boxes. Through comparison, we can observe that, thanks to our MDGF module, our method can locate the object area more accurately. For example, in sequence baginhand, as the object is tracked, the ViPT∗ method gradually focuses on the pedestrian, while the After∗ method simultaneously tracks the shadow region. In contrast, our method precisely tracks the pedestrian's bag in hand. Furthermore, benefiting from our TGID module, our method demonstrates stronger resistance to similar object noise. For instance, in sequence cycle2, when tracking the pedestrian, other methods gradually get distracted by similar objects and begin tracking a different pedestrian with similar features, whereas our method consistently tracks the same pedestrian.

4.2 Ablation Study

Component Analysis. We conducted ablation studies on four datasets to evaluate the effectiveness of the MDGF and TGID modules, with the results presented in Tab. 3. We constructed our baseline model by directly incorporating RGB and infrared features into the RGB tracker, using a model trained with pseudo-labels. The baseline results on the four datasets are presented in the first row of Tab. 3.

The ablation of MDGF. The tracker with MDGF demonstrates a notable improvement over the baseline model, with PR scores of 1.8%, 1.2% on the RGBT234 dataset, 2.1%, 1.5%, and 1.5% on the LasHeR dataset, 1.3% and 1.8% on the VTUAV dataset, and 4.9% and 3.5% on the GTOT dataset. As shown in the first and second rows of this table, MDGF can dynamically adjust the adjacency matrix, thereby better integrating the advantages of modalities for subsequent tracking.

The ablation of TGID. The tracker with both MDGF and TGID achieves further improvement over the MDGF-only model, with SR of 0.2% on the RGBT234 dataset, NPR and SR of 0.2%, 0.2% on the LasHeR dataset, PR and SR of 2% and 0.8% on the VTUAV dataset, and 0.4% and 0.3% on the GTOT dataset. From the second and third rows of this table, we can see that the TGID module further enhances the model's performance by improving robustness to interfering noise, building on the MDGF module.

The ablation of AMG. To evaluate the effectiveness of our proposed adjacency matrix generation method, we compared it with other generation methods, as shown in Tab. 4. We employed the identity matrix, self-attention, and cosine similarity methods to generate adjacency matrices, which serve as comparative methods for our proposed AMG module. We can observe that training with the adjacency matrix generated by the AMG method yields the best performance on the LasHeR, VTUAV, and GTOT datasets. On the RGBT234 dataset, the AMG method showed a little decrease of 0.4% in SR, compared to the identity matrix method. The excellent performance of the AMG module proves that, compared to other adjacency matrix generation methods, the AMG module can dynamically capture the inter-modal similarity, direct the model to focus on the object-related regions through graph attention and obtain fusion results that are favorable for tracking.

The ablation of TopK. Tab. 5 shows the performance of different values of K in the AMG module on two datasets. We observe that the model's performance is positively correlated with the value of TopK. When K is set to 256, the model performs the best on GTOT and RGBT234 datasets.

The ablation of Graph-informed condition. As shown in Tab. 6, compared to [Song et al., 2020], using the first layer of graph convolution from the MDGF module as a condition for the intermediate layers of the UNet in the diffusion model resulted in a clear improvement on three datasets.

The ablation of distractor noise. As shown in Tab. 6, adding similar distractor noise resulted in increases of 0.7% in PR and SR on the RGBT234 dataset, 1% and 0.9% on the VTUAV dataset, and 0.2% and 0.9% on the GTOT dataset.

5 Conclusion

In conclusion, we have designed a Dynamic Graph Diffusion framework to address the issues in self-supervised RGB-T tracking tasks. Specifically, to tackle the problem of incorrect fusion of non-object regions, the MDGF module has been proposed to dynamically adjust the adjacency matrix based on the similarity between modalities, guiding graph attention to focus on the fusion of coherent object regions. To address the noise interference caused by similar objects, the TGID module has incorporated the MDGF fusion results from neighboring frames as noise, training the model to improve its robustness against interference from similar objects. Experiments on four datasets have demonstrated that GDSTrack achieves state-of-the-art results in the field of self-supervised RGB-T tracking, validating the effectiveness of our model.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grant Nos. 62172417, 62272461, 62276266 and 62472424.