SSTrack: Sample-interval Scheduling for Lightweight Visual Object Tracking

1 Introduction

Visual object tracking is a fundamental computer vision problem that aims to locate an arbitrary object in a video sequence given its first frame location. With the advance of deep learning techniques, visual object tracking has seen significant progress in recent years. Most deep-learning object trackers follow a template matching paradigm [Bertinetto et al., 2016] that matches the search region and the first frame template with a deep neural network (DNN). Although this simple paradigm proves to be effective, it also poses a high demand on the representation quality of the DNN, especially when the time interval between the template image and the search image is large. To solve this issue, some methods introduce more complicated matching networks like transformer matching [Chen et al., 2021], learning-based matching [Zhang et al., 2021], one-stream matching [Ye et al., 2022], etc. Another approach is leveraging different kinds of temporal information, including dynamic online template [Yan et al., 2021a], temporal memory readout [Fu et al., 2021b], spatiotemporal prompts [Wei et al., 2023], dense temporal token [Zheng et al., 2024], etc.

∗Corresponding author

Despite the high performance, the complex network and the additional temporal information inevitably bring heavy computational overhead which hinders their application in real-world applications like UAV-tracking. To satisfy realworld applications, many strategies are proposed to improve the efficiency of object trackers, including model design [Yan et al., 2021b; Kang et al., 2023; Cao et al., 2021; Gopal and Amer, 2024], model distillation [Cui et al., 2024], dynamic inference [Li et al., 2023; Li et al., 2024], preprocessing [Kou et al., 2023]. Although the strategies are diverse, most of these methods only use one template from the initial frame and employ a lightweight backbone to ensure high efficiency. Without temporal context, this single template setting poses a high demand for representation quality, especially when the time interval between the template and search image is large. However, lightweight backbones are usually more struggled to capture large appearance variations compared with bigger backbones. This conflict between the hard training setting and weak model capability can be troublesome during training. Thus, it is natural to ask the question: Is it possible to customize a training schedule for lightweight trackers to compensate for their weaker representation ability?

To this end, we proposed a customized training framework, termed Sample-Interval Scheduling Track (SSTrack), for lightweight object trackers that emphasize compensating for the weaker representation ability of lightweight models via easier-to-harder sample schedule. As shown in Fig3, the proposed framework handles the challenge from both data and optimization perspectives. From the data perspective, we identify training samples with longer template-search time intervals as harder samples due to their larger appearance variance. As a result, the training difficulty can be adjusted by changing the sampling frequency of samples with different template-search time intervals. To determine the ideal sampling frequency for every epoch, we proposed a novel success-aware sample scheduler that considers both the model's learning progress and the predefined easier-to-harder behavior. The success-aware sample scheduler only uses the latest epoch's training statistics so no significant overhead is introduced. To keep the training cost unchanged, the number of easier short-interval samples is inevitably reduced. From the optimization perspective, we employed a gradient scaling strategy to compensate for this reduction. By up-scaling the gradient of samples with reduced sampling frequency, the original training objective on easier samples can be retained. The gradient of samples with increased sampling frequency is unchanged to enjoy the benefit of increased harder examples. The collective effort from both data and optimization perspectives enables our method to achieve State-Of-TheArt CPU-real-time tracking performance on 5 general object tracking benchmarks and 5 UAV-tracking benchmarks.

Our contribution can be concluded as follows: (i) We analyze the difficulty of training lightweight trackers on samples with long template-search time intervals at early training stages owing to the weaker representation ability of lightweight backbones; (ii) We designed a success-aware sample scheduler that gradually introduces more hard longinterval samples based on both the model's learning progress and the predefined easier-to-harder behavior. To compensate for the reduced easy samples, we additionally employed a gradient scaling strategy that up-scales the gradient of shortinterval samples to retain their original training objective; (iii) Our method achieves State-of-the-Art CPU real-time tracking performance on 5 general object tracking and 5 UAV-tracking benchmarks.

2 RelatedWork

2.1 Lightweight Object Tracking

In real-world applications like UAV-tracking or robotics, object trackers are deployed to edge devices with limited computational power and memory. Thus, many efforts have been made to boost the efficiency of object trackers. Early attempts focus on finding more efficient architectures for Siamese trackers. LightTrack [Yan et al., 2021b] uses NAS to search for efficient subnets for object tracking. FEAR [Borsuk et al., 2022] designs a dual-template representation and a pixel-wise fusion block to improve both the efficiency and the accuracy of Siamese trackers. With the rise of transformer trackers, some methods attempt to design more efficient transformers for visual tracking. HCAT [Chen et al., 2022] designs a hierarchical cross-attention transformer for lightweight feature interaction, while Kang et al. explores modern lightweight transformers for efficient one-stream tracking. Some methods also employ model compression techniques to reduce model size. MixFormerV2 [Cui et al., 2024] presents a distillationbased model reduction paradigm that sequentially reduces the width and depth of the tracker. Another widely explored approach for lightweight tracking is the dynamic inference of the tracking model. Aba-ViTrack [Li et al., 2023] adaptively discards image tokens based on learned halting probabilities, whereas AVTrack [Li et al., 2024] dynamically activates transformer blocks to balance the efficiency and accuracy. AVTrack [Li et al., 2024] also proposed a mutual information maximization loss to facilitate learning view-invariant representations. These methods mainly focus on improving the model design or the supervision, while our method focuses on improving the training data and the optimization.

2.2 Sampling Strategy in Object Tracking

Many object tracking methods [Bertinetto et al., 2016; Ye et al., 2022; Li et al., 2024; Zheng et al., 2024] follow a random sampling strategy that first randomly samples a template frame as the base frame, then randomly samples search frames after the base frame. The maximum time interval between the initial template frame and the search frame (temporal range) is restricted to be lower than a threshold to avoid unstable training. To increase the temporal range without disturbing training, some methods [Yan et al., 2021a; Cui et al., 2022] introduce online frames only ensuring the maximum time interval between the search image and the latest online frame is within a threshold. The temporal range can be safely increased because the online frames provide sufficient recent appearance information. Following this paradigm, TCTrack++ [Cao et al., 2023] proposed an easy-tohard curriculum strategy that increases the number of online frames at 33% and 50% of total epochs to improve accuracy. However, their strategy cannot be applied to the vast majority of lightweight trackers without online frames and they switch to harder samples at predefined timesteps. In contrast, our method is customized for lightweight trackers without online frames and automatically adjusts difficulty based on the training status every epoch.

3 Method

We present SSTrack, a customized training schedule for lightweight object trackers aiming to compensate for the weaker representation ability of lightweight models from both data and optimization perspective. This section first presents an analysis of the necessity of an easier-to-harder training schedule for lightweight models. Next, we detailed the proposed success-aware sampler scheduler for easier-toharder training, followed by the introduction of the gradient scaling strategy which retains the original training objective on reduced easier samples.

3.1 ProblemAnalysis

Object trackers usually follow an image pair matching paradigm [Bertinetto et al., 2016]. Specifically, a lightweight visual object tracker ϕθ(z, x) takes a template image patch z and a search image patch x as input and predicts the target bounding box. Instead of training from random initialization, object trackers typically fine-tune a pretrained backbone network and incorporate an additional localization head to leverage the robust feature representations from large-scale pretraining. For the sake of tracking speed, lightweight object trackers use lightweight pretrained backbones which often yield less robust representations due to their reduced parameters and computational budget. To quantify this difference in representation quality, we compute the representation similarity of the same object across different times in videos with a lightweight model ViT-Tiny [Wang et al., 2023] and a heavy model ViT-Base [He et al., 2022] using Centered Kernel Alignment(CKA) [Cortes et al., 2012; Nguyen et al., 2020]. The result is shown in Fig. 2. For both models, the representation similarity decreases as the time interval between images increases. Compared with the large model(ViT-Base), the representation extracted by the lightweight model (ViT-Tiny) is less consistent for the same object across time, especially when the time interval is large (top-left corner and bottom-right corner). The messages from the above observations are twofold: (i) The training pairs with larger time intervals are naturally more difficult. (ii) The lightweight pretrain model is more inadequate to capture large appearance variation, especially under longer time intervals, which makes training on long-interval samples difficult in the early epochs.

Following the above analysis, we aim to design an easy-tohard training schedule that gradually increases harder training samples with longer template-search time intervals to compensate for the weaker representation ability of lightweight backbones. Next, we detail our proposed success-aware sample scheduler and gradient scaling strategy.

3.2 Easier-to-harder Training via Success-aware Sample Scheduling

In this section, we demonstrate a success-aware sample scheduler for lightweight trackers. We start with an intervalbased sampler that allows for adjusting training difficulty by changing the sampling frequency of different samples. Then, we elaborate on the success-aware sample schedule that alters the training difficulty based on both the model's learning progress and the predefined easier-to-harder behavior.

Interval-based Sampler

To generate training samples during training, most methods follow a random sampling strategy which first randomly selects a template frame at tz and then selects a search frame at tx ensuring the interval tx - tz ∈ [0,M), where M is the maximum sample interval. Under this random sampling strategy, the distribution of the interval tx-tz is fixed throughout training thus the difficulty level remains the same.

To dynamically alter the difficulty, we proposed to use an interval-based sampler that generates image pairs according to a sampling distribution Q = (q1, · · · , qr), 0 ≤ qi ≤ 1, ∑ri qi = 1. Specifically, given the maximum sample interval M, we part [0,M) into bins with equal spacing m, resulting in r = ⌈M/m⌉ bins such that i-th bin bi correspond to [li, ui), where the lower bound is li = mi and the upper bond is ui = min(m(i + 1),M). With probability qi the interval-based sampler generate a sample with interval tx - tz ∈ [li, ui) by first uniformly samples a training video (denote its length as L), then sample the template image at tz ∈ [0,L - li] ∪ [li,L] and the search image at tx ∈ [max(tz - ui + 1, 0), max(tz - li + 1, 0)) ∪ [min(tz + li, L), min(tz + ui, L)).

Given this interval-based sampler, we are able to adjust the training difficulty by changing the sampling frequency Q. Next, we will demonstrate a success-aware sample scheduler that dynamically adjusts training difficulty according to the training dynamics during training.

Success-aware Sample Scheduler

We begin by outlining the workflow of the sample scheduler. After each training epoch, the scheduler evaluates the model's performance at the current difficulty level and adjusts the difficulty for the next epoch based on this measurement. The key to the sample scheduler's design lies in its performance measurement which should consider both the model's learning progress and the predefined easier-to-harder behavior. Additionally, to minimize computational overhead, it is optimal to perform performance measurement solely using training statistics, eliminating the need for additional validation data.

Inspired by the commonly used object tracking metric: success score, we proposed an expected success measurement to satisfy the above design philosophy. To minimize computational overhead, we use the training samples' IoU to estimate the training progress of the corresponding samples. Specifically, given a success threshold τ, we calculate the success rate of all the latest epoch training samples from bin bi and use the success rate as an estimation of the probability(denoted as pτi ) that the current model successfully tracks any sample within bi. With this estimation, we can treat all the samples within bi as a single "general frame" fi with probability pτi to success. The proposed measurement is the expected success over the "general video" {f1, · · · , fr}. Specifically, the success score of the current model on each "general frame" can be seen as a random variable Xτ i ∈ {0, 1}. The sequence of success score formulates a discrete-time stochastic process {Xτ i } with countable state space E = {0, 1}. As the current frame prediction solely relies on the current frame and the previous frame's prediction, we can assume {Xτ i } follows the Markov property and the transition matrix T τ i from time i to i + 1 can be written as follows

where d ∈ [0, 1] is a hyperparameter that can be intuitively understood as the probability of re-detecting the target after tracking failure.

The state distribution at i can be calculated as πτ i = πτ 0 T1 · · · Ti-1, where πτ 0 = (1, 0) is the initial distribution. With the distribution πτ i , the expected success at i is E{Xτ i } = πτ i (1, 0)T . And the expected success over the "general video" {f1, · · · , fr} is Sτ = ∑ri=1 E{Xτ i }/r.

Next, we adjust the difficulty by generating a new sample distribution Q. Samples that contribute more to Sτ should be sampled more frequently so that they are better trained. Therefore, we use ∂Sτ /∂pτi to determine the sampling frequency for bin bi. The final sampling distribution is obtained as qi = ∂Sτ /∂pτi ∑ri=1 ∂Sτ /∂pτi .

There are two hyperparameters τ and d in the formulation of the expected success. First, we use τ to consider the model's learning progress. The success threshold τ affects the difficulty by changing pτi . Ideally, τ should grow as the model gains better tracking ability. We set τ to the mean IoU of the latest epoch's training samples. The underlying reason is that the mean IoU can provide precise information about the training dynamics of the tracker which facilitates the sample scheduler to determine a more appropriate difficulty.

Secondly, we use d to inject predefined easier-to-harder behavior. When d → 0, ∂Sτ /∂pτi → 1/r, so qi is the same for every i. Consider pi = c for every i, when d → 1, ∂Sτ /∂pτi → ∑r-1 m=i-1 cm/r, so qi significantly decreases as i grows up. Therefore, by adjusting d from 0 to 1, an easier-to-harder behavior can be obtained. The pace of introducing harder samples can be regulated by controlling how quickly d approaches 1. We use d(e) = p√ eE as the schedule function, where e is the current epoch and E is the total epoch, p is the order of the schedule function. The benefit of choosing this schedule function is that we can control the pace of increasing hard samples with a single parameter. Specifically, a large p means introducing harder samples more quickly and vice versa.

The success-aware sample scheduler only uses training samples' IoUs and the current training epoch as the indicators to decide the sample schedule. Since the IoU is a byproduct of the loss calculation, no significant additional computation is introduced during training. The wall clock time for training is almost the same as compared with training using the random sampler.

3.3 Retain Original Training Objective on Easier Samples via Gradient Scaling

To keep the number of total training samples fixed, the easierto-harder sample scheduling inevitably reduces the number of short-interval easy samples to increase harder long-interval samples causing a shifted optimization objective, which can potentially undermine the short-term tracking accuracy. Although carefully designed sample schedulers can partly alleviate this issue, inspired by infoBatch [Qin et al., 2023], we propose a gradient scaling technique to further mitigate this problem.

Denoting the data sampled by the Random Sampler (RS) with sampling distribution qR(i) as DR and data sampled by Interval-based sampler (IS) at epoch t with sampling distribution qI t (i) as DI t . DI t can be seen as a scaled version of DR, where the sampling frequency for bin bi is scaled by st(i) = (qR(i) - qI t (i))/qR(i). For a sample v ∈ bi, it is a pruned sample if st(i) ≥ 0, otherwise an enhanced sample. We define St(v) = st(i), v ∈ bi as the scaling ratio of a sample v. Consequently, Pt(v) = max(St(v), 0) is the pruning ratio and Et(v) = St(v) - Pt(v) is the enhance ratio. The gradient scaling strategy enlarges the gradient of pruned samples and keeps the gradient of enhanced samples unchanged. Specifically, The scaling ratio of each sample v is determined by a scaling function Γt(v) = 1/(1 - Pt(v)). The scaling function is directly multiplied by the original loss L(v, θ) as Γt(v)L(v, θ) to achieve gradient scaling. We next provide a theoretical analysis of how gradient scaling retains the training objective on pruned short-interval samples while maintaining the enhanced training objective on increased longinterval samples.

Theoretical Analysis

Following InfoBatch [Qin et al., 2023], we interpret the training objective as minimizing empirical risk L. We assume the discrete sample distribution for DR as ρR(v). Then the discrete sample distribution for DI t can be derived as:

At each epoch t, DR can be divided into two disjoint subset: DRp t = {v|St(v) ≥ 0} for pruned samples and DRe t = {v|St(v) < 0} for enhanced samples.

The training objective on DR is:

The training objective with gradient scaling on DI t is:

By substituting Eq. 2 into Eq. 4, we have

where ct = ∑v (1 - St(v))ρR(v) is a constant at epoch t, ρE (v) = -Et(v)ρR(v) is the data distribution for enhanced long-interval samples.

As shown in Eq. 5, for the short-interval samples pruned in epoch t, after applying the gradient scaling, the objective becomes a constant-resealed version of the original objective in Eq. 3. For the long-interval samples, the new objective is composed of the original objective and an enhancement objective brought by the extra long-interval samples. So the tracker still benefits from the increased harder samples.

4 Experiments

4.1 Implementation Details

We select OSTrack [Ye et al., 2022] as the baseline model considering its simplicity and publicly available training and inference code. To make it lightweight, we replace the ViTBase backbone with ViT-Tiny [Dosovitskiy et al., 2020] and adopt the D-MAE pre-train weight from [Wang et al., 2023; Gao et al., 2025]. All the training and testing settings, except for the training data sampler, are kept the same as OSTrack256. For the interval-based sampler, we set the maximum sample interval M = 200 following OSTrack. We part M into 7 bins with equal spacing 30, which is the frame rate for the training video datasets. An additional bin [0, 1) is added to hold samples from the image dataset COCO [Lin et al., 2014]. The success-aware scheduler uses the average iou from the latest epoch as the success threshold and uses the schedule function with order 2. The model is trained on 4 NVIDIA V100 GPU and the inference speed is measured on Intel Xeon Gold 6146 CPU.

4.2 State-of-the-Art Comparison

Comparison on General Tracking Benchmarks

We first compared our method with 6 State-of-the-Art (SOTA) CPU-real-time object trackers on 5 general object tracking datasets, namely LaSOT [Fan et al., 2019], LaSOText [Fan et al., 2021], GOT-10k [Huang et al., 2019], TrackingNet [Muller et al., 2018], NFS [Kiani Galoogahi et al., 2017]. We only include CPU real-time object trackers with reported CPU speed over 30 FPS for a fair comparison. The results are shown in Tab. 1. Our method outperforms all the previous real-time SOTA methods on all 5 datasets, demonstrating the powerful performance of our method. textbf LaSOT is a diverse tracking dataset with over 280 long-term video sequences. As shown in Tab. 1, our method outperforms previous SOTA HiT-B [Kang et al., 2023] by 2.3% Success and 3.7% Precision. LaSOText is the extension of LaSOT with an additional 150 videos containing 15 novel classes. In Tab. 1, our method achieves 2.2% gain in Success over the previous SOTA HiT-B [Kang et al., 2023]. GOT-10k requires trackers to be trained solely on the training split to verify generalization ability. We follow this protocol and the result is reported in Tab. 1. According to the results in Tab. 1, our method improves the SOTA performance by 4.5%, 2.6% and 5.1%in AO, SR0.5, SR0.75 respectively. TrackingNet is a large-scale short-term tracking dataset with 511 testing videos. As displayed in Tab. 1, our method lifts the SOTA performance by 1.6% in Success and 2.2% in Precision. NFS is a high frame-rate dataset containing challenging fast-moving objects with 100 video sequences. As reported in Tab. 1, our method achieves 65.7% Success, improving previous SOTA by 2.1%.

We further list 7 non-real-time SOTA trackers in Tab. 1 for a more comprehensive comparison. It is worth noting that our method outperforms STARK [Yan et al., 2021a] using only 28% parameters and 14% computation.

Comparison on UAV-tracking Benchmarks

We next compare our method with 7 State-of-the-Art(SOTA) UAV trackers on 5 UAV-tracking benchmarks, namely DTB70 [Li and Yeung, 2017], UAVDT [Du et al., 2018], UAV123 and UAV123@10fps [Mueller et al., 2016], UAVTrack112L [Fu et al., 2021a]. The results are shown in Tab. 2. Our method outperforms or on par with previous SOTA trackers on these 5 UAV-tracking datasets with similar overhead and speed, showing a strong generalization ability to real-world scenarios. textbf DTB70 is a UAV tracking dataset with 70 videos which primarily addresses the challenge of severe UAV motion. As shown in Tab. 2, our method achieves 66.4% SUC and 85.5% PRE. UAVDT is a challenging dataset mainly focus on vehicle tracking with various weather conditions, flying altitudes and camera views. As reported in Tab. 2, our method outperforms previous SOTA Aba-ViTrack [Li et al., 2023] in SUC by 1.1%. UAV123 is a UAV-tracking dataset containing 123 videos captured by low-altitude UAVs. From the results in Tab. 2, it can be seen that our method improves the previous SOTA by 0.5% SUC and 1.2% precision. UAV123@10fps is constructed from UAV123 by resampling from 30 FPS to 10FPS causing more abrupt motion and scale variation. In Tab. 2, our method boosts the SOTA performance by 0.3% SUC and 0.4% precision. UAV112L is the current largest long-termUAV-tracking dataset containing 45 videos. As displayed in Tab. 2, our method advances the SOTA performance by 1.6% in SUC and 1.9% in Precision, showing that our method is capable of

Efficiency Analysis

We comprehensively evaluate the efficiency of our method via the number of parameters, the computational overhead and the inference speed. The results are reported in both Tab. 1 and Tab. 2. Compared with CPU real-time trackers, as displayed in Tab. 1, our method achieves 47FPS on CPU and only uses 1.9G Multiply-Accumulate Operations (MACs) which is comparable to previous CPU real-time trackers. As for parameters, our method only uses 8.0M parameters, which is significantly less than previous SOTA CPU real-time trackers, making it ideal for deploying on edge devices. Compared with UAV trackers, as presented in Tab. 2, our method has similar efficiency with these trackers customized for deployment on UAVs, while having on-par or better accuracy.

4.3 Ablation Study

We conduct the ablation experiments on two challenging long-term tracking datasets, namely LaSOT [Fan et al., 2019] and UAVTrack112L [Fu et al., 2021a], to demonstrate the effectiveness of our method.

Component-wise Analysis

To evaluate the effect of different components of our method, we conduct a component-wise analysis, starting from the baseline (①). The results are reported in Tab. 3. Directly employing the success-aware sample scheduler on the baseline can improve the Precision by 1.0% on LaSOT and 1.4% on UAVTrack112L (①v.s.②), demonstrating the effectiveness of the proposed easier-to-harder sample schedule. Next, we add the gradient scaling strategy, the Precision is further boosted by 1.1% on both datasets (②v.s.③), proving the effectiveness of the gradient scaling strategy.

Different Sample Scheduler Setting

As shown in Tab. 4, we ablate different settings of the success-aware sample scheduler. In the first set of experiments (①, ② and ③), we fix the success threshold τ as the mean IoU of the latest epoch and change the speed of introducing more harder samples via changing the order of schedule function p. In Tab. 4, selecting p = 2 achieves the best accuracy, demonstrating the importance of gradually introducing more long-interval samples. In the second set of experiments (②, ④ and ⑤), we set the order of the schedule function fixed to p = 2, and explore the necessity of using dynamic success threshold τ. As displayed in Tab. 4, using fixed success threshold (④ and ⑤) performs worse than using mean IoU as dynamic success threshold (②), proving the significance of considering the model's learning progress when designing sample scheduler. In the last experiment (⑥), we only update the difficulty at 33% and 50% of the total epoch following [Cao et al., 2023]. Compared with ②, the success score and precision drop significantly on both datasets, showing the importance of our update-every-epoch design.

Generalization Ability

To demonstrate the generalization ability of our approach, we incorporate our method to other SOTA lightweight trackers: AVTrack [Li et al., 2024] and HiT [Kang et al., 2023]. As displayed in Tab. 5, our method achieves consistent gains on multiple lightweight trackers with different backbones and pretrain methods, showing a strong generalization ability. Note that all the hyperparameters in our method remain consistent across all datasets and trackers, while other hyperparameters shared with the baselines are kept unchanged.

5 Conclusion

In this work, we present SSTrack, a novel training schedule designed to enhance the performance of lightweight object trackers. Aiming to compensate for the weaker representation ability of lightweight backbones, we developed a successaware sample scheduler that progressively increases difficult long-interval samples and reduces easy short-interval samples to keep the training cost unchanged. Furthermore, we incorporated a gradient scaling strategy to maintain the positive influences of easier samples, even as their quantity decreases. This combined approach has enabled us to achieve state-ofthe-art accuracy on 5 UAV-tracking benchmarks and 5 general object tracking benchmarks.

1 Baseline results are obtained by training with the official code from their GitHub repository to ensure a fair comparison.

Acknowledgements

The authors would like to thank the anonymous reviewers for their valuable comments and suggestions. This work was supported in part by the Natural Science Foundation of China (Grant No. 62036011, 62422317, U22B2056, 62192782, U24A20331, U2441241, 62172413), the Beijing Natural Science Foundation (Grant No. JQ22014, L223003), the Key Research and Development Program of Xinjiang Urumgi Autonomous Region under Grant No. 2023B01005, the Project of Beijing Science and Technology Committee (Project No. Z231100005923046). Bing Li is also supported in part by the Youth Innovation Promotion Association, CAS.