Wave-wise Discriminative Tracking by Phase-Amplitude Separation, Augmentation and Mixture

1 Introduction

Visual Object Tracking (VOT) is pivotal in real - time computer vision, accurately tracking target objects within video sequences. At its core, tracking hinges on matching and localization, where feature extraction is a critical step determining a tracker's capacity to represent and distinguish targets. To ensure tracker robustness and accuracy, learning discriminative feature representations is essential. Tracking algorithms have significantly advanced, from handcrafted features to the current deep extraction methods. Using deep neural networks, these techniques learn effective features. However, research on deep discriminative tracking has bottlenecks. Prior pixel - based extraction limits information, so a new approach is urgently needed to enhance tracking performance.

Inspired by classical physics and quantum mechanics, wave-wise feature representation offers a new perspective: Images or image tokens are seen as a series of propagated wave-wise components, formed by fusing amplitude and phase via complex wave functions. This representation holds richer semantics, specific invariances, and strong information-aggregation capabilities, opening up new possibilities in visual tasks. It has been successfully applied in various scenarios. Early research relied on the Fourier transform to separately extract phase and amplitude features for specific tasks. But integrating this multi-step process into end-to-end deep learning was difficult. A recent work [Tang et al., 2021] introduced a plug-and-play method in a Multi-Layer Perceptron architecture. Each token was treated as a visual waveform, and neural networks were used to model phase and amplitude extraction and aggregation across all tokens. Notably, phase mainly captures object-related spatial details like textures, edges, and outlines, while amplitude reflects global brightness, being more sensitive to background changes. To better use waveform features in VOT and distinguish targets from backgrounds, we aim to make the model extract phase and amplitude features distinctly from target and background semantic blocks. This ensures clear encoding of target structure and image brightness, resulting in more discriminative aggregated waveform features, rather than just repeating previous methods. In summary, waveform representation improves object tracking in three ways: 1) Capturing Semantic Details: Waveform representation extracts object shape and boundaries through phase shifts, and captures broader contrast changes through amplitude variations; 2) Global and Local Feature Fusion: Waveform features effectively combine information from various positions, facilitating a more comprehensive feature description for both individual images and image blocks. 3) Enhancing Robustness and Stability: The stability of waveform representation ensures improved tracking performance, particularly in challenging scenes.

Motivated by this, we develop a novel Wave-wise Discriminative Tracking method. For tracking tasks, the target's spatial structure and brightness information should be distinguished, with more emphasis on the former. For instance, in certain scenarios, lighting conditions might change, but the basic target structure remains relatively invariant. As shown in Figure 1, changes in image phase (phase swapping) lead to drastic variations, while amplitude swaps can still be distinguished by the human eye to identify objects.

Therefore, the core of our innovation lies in how to manipulate the phase and amplitude features within the framework of waveform representation to highlight the target structure and mitigate the background influence. First, PhaseAmplitude Feature Separation: Drawing on the findings of the literature [Tang et al., 2021], we introduce an attention module to learn the weights of image tokens. Subsequently, a pair of mutually exclusive token weights are generated for phase and amplitude feature modeling respectively. The aim is to mainly model the phase information in the target region, while the amplitude information is mainly distributed in the background area. Integrating the above-mentioned processes constitutes the Mutual Exclusive Phase-Amplitude Extractor (MEPAE) module. Second, Phase-Amplitude Feature Enhancement: During model training, the separated phaseamplitude features within the same batch are randomly shuffled. This implies that each sample in the same batch contains both the original phase-amplitude features and the wave features from other samples. Third, Phase-Amplitude Feature Mixing: In this stage, the initial phase-amplitude features are incorporated into the original wave, and the enhanced phase-amplitude features are interwoven to generate multiple sets of positive and negative waves. Positive waves have the same phase but different amplitudes, and negative waves have different phases. Finally, Feature Discriminability Enhancement: To better enhance the discriminability of features, we combine self-supervised and tracking-supervised losses to constrain all waveform features from both global and local token perspectives. Specifically, the self-supervised loss is used to constrain the similarity between feature pairs such as the original-negative waves and positive-negative waves.

The tracking-supervised loss constrains the token-level features of all waves. The above steps lead to the final waveform tracker, which can effectively extract feature information, improve the object tracking performance, and enhance the accuracy and robustness.

In summary, the main contributions of this work are threefold: (1) We introduce an innovative waveform-based feature representation, and conduct feature learning through the processes of phase-amplitude feature separation, augmentation, and mixture. (2) By combining self-supervised and supervised losses, we strengthen the overall and local learning of waveform features. The introduction of feature-level selfsupervised learning into the tracking framework represents a novel plug-and-play integrated learning paradigm. (3) We construct a novel waveform discriminative tracking method based on the Transformer network, named Wave - wise Discriminative Tracker (WDT). This method has demonstrated excellent performance in multiple benchmark tests, including LaSOT, LaSOText, TNL2K, UAV123, and GOT-10K.

2 RelatedWork

Transformer-based tracker. With the popularity of Transformer [Vaswani et al., 2017] in computer vision, Transformer-based trackers have become a dominant category. SwinTrack [Lin et al., 2022] and OSTrack [Ye et al., 2022] ditch CNNs and fully embrace the transformer structure, achieving excellent results in multiple benchmarks. Recent studies have explored various aspects. [Cai et al., 2024] uses the Transformer encoder to extract interaction features between the template and search region. [Xie et al., 2024] employs the Transformer module for spatial image feature extraction and spatiotemporal video learning. [Zhao et al., 2024] investigates the impact of different data augmentation on Transformer tracker performance. [Wei et al., 2023] takes an autoregressive approach, using Transformer - encoded features to model and predict target trajectories. [Chen et al., 2023] uses a ViT-based encoder for video frame feature extraction and a causal Transformer decoder for tracking results, simplifying the network and enhancing performance. [Li et al., 2023] uses a Transformer backbone to extract joint features of the template and search image, facilitating interaction with text features for prediction.

Wave-wise features in Computer Vision. Phase and amplitude information are crucial features for images and have been successfully applied in several visual tasks [Wang et al., 2025; Wang et al., 2024b; Wang et al., 2024c]. [Zhang et al., 2023b] uses Semantic Frequency Prompt to interact with the feature map in the frequency domain, guiding the student model to learn valuable pixels. [Gu et al., 2023] proposed AdaFuse, which employs the spatial-frequential fusion module on multi-scale features for adaptive multimodal image feature fusion in both frequency and spatial domains. [Cheng et al., 2023] proposed the FGFL framework, generating frequency masks via an improved Grad-CAM algorithm in the frequency domain to filter key category information.

Self-supervised Learning. Self-supervised learning is one of the main types of machine learning, alongside unsupervised [Wang et al., 2024d] and semi-supervised learning

[Wang et al., 2024a]. Self-supervised learning is highly effective in video tasks. For instance, [Qian et al., 2024] trains a correlation network via object consistency in videos. [Ding et al., 2025] improves object segmentation accuracy with spatiotemporal consistency. [Hamilton et al., 2022] mines semantic consistency in feature maps for semantic segmentation.

3 Method

3.1 Framework of WDT

As the Figure 2 depicted, the framework of our WDT can be divided into five parts successively:

Backbone. We adopt the backbone of One-Stream OneStage pipeline for image feature extraction and relation modeling. In this stage, the input template image z and search image x are split and flattened into a chain of patches Nz and Nx. And after a linear projection they will turn into a series of patch embeddings Ez ∈ RNz×D and Ex ∈ RNx×D , where D = P 2 is the feature dimension of each token. Each of token will be added a position embedding and then the template embeddings Ez and search embeddings Ex will be concatenated as Ezx = [Ez,Ex]. And the Ezx will be processed by sequences of Transformer encoder layers. During this process, we finished the initial feature extraction, and at the same time, the relation between the search image and the exemplar image have been calculated.

Mutually Exclusive Phase-Amplitude Extractor (MEPAE). After the backbone network has fully learnt the relationship between the target and the search feature, it will output the transformed feature ˜Ezx ∈ R(Nz+Nx)×P×P . In the original framework, the corresponding search region ˜Ex ∈ RNx×P×P in ˜Ezx is fed into the prediction head to predict the tracking results. In between, MEPAE is embedded as a plug-and-play module. ˜Ex is taken as inputs and a new attention network is designed, which will acquire the labelled attention used to guide the phase and amplitude feature generation. Details will be presented in Sec. 3.2.

Phase-Amplitude Feature Augmentation (PAFA). To further enhance the learning of waveform features, we perform feature enhancement by shuffling the separated phase and amplitude features within a batch. This step is referred to as Phase-Amplitude Feature Augmentation (PAFA). We will discuss this in Section 3.3.

Phase-Amplitude Feature Mixture (PAFM). We insert a PAFM unit between the PAFA and the prediction head. In this process, the free combination of amplitude and phase terms will form a new set of feature samples, including the original waveform sample, the positive waveform samples with constant phase but only varying amplitude, and two groups of negative waveform samples with varying phase. Further we will impose supervised and self-supervised constraints on them to aid waveform feature learning. We will discuss this in later sections Sec. 3.3 and Sec. 3.4.

Prediction Head. We use a center prediction head for target classification and box regression. In addition, we added self-supervised loss and supervised loss for all waveform features. Both the Score map and the predicted box coordinates will be obtained after this structure.

In particular, features are extracted by MEPAE for both training and inference, except that for training there is feature augmentation and mixture, and all wave samples are output, whereas for inference only the predictions of the original waveform samples are output.

3.2 Mutually Exclusive Phase-Amplitude Extractor

Before the introduction of the structure of MEPAE, we present the representation of wave function first.

Wave-based representation. Denote one token feature output by the backbone as a wave ˜tj :

where i2 = -1 and |·| is to calculate the absolute value and ⊙ represents element-wise multiplication. |tj| and the ϕj are the amplitude and the phase we are concerned for. And the amplitude usually carries the content information of the token while the phase in periodic function eiϕj represents the current state of the token in its period.

Network for Mutually Exclusive Tokens Attention. In order to better separate phase and amplitude and reflect different semantic properties, we introduce a simple attention module for learning tokens weights. The tokens attention map Mp and its mutually exclusive attention Ma = 1 - Mp are directly learned based on the output feature ˜Ex, corresponding to phase and amplitude features modelling, respectively. Specifically, in two steps, first the initial features are pooled into ˜Epx ∈ RNx×1×1 . And it is then fed into a token attention module for transformation learning consisting of multiple convolutional layers, Batch Normalization, ReLU and sigmiod functions, ultimately outputting a token attention map Mp ∈ RNx×1×1 .

Network for Phase-Amplitude Extraction. To extract the waveform features, phase and amplitude features need to be extracted separately first. In [Tang et al., 2021], the neural network is innovatively used to express them separately. First we multiply the above attention maps Mp with the initial token features ˜Ex for phase extraction, and then go through a classical Conv-BN-ReLU module for feature transformation to obtain the final phase features P = {p1, p2, · · · , pNx }. In another branch, we multiply ˜Ex by Ma for amplitude extraction, and then obtain the amplitude feature A = {a1, a2, · · · , aNx } through a Conv layer.

The above feature extraction process is jointly integrated into the mutually exclusive phase-amplitude extraction module, the structure of which is shown in Figure 3. The specific extraction process can be expressed as follows:

3.3 Wave-wise Feature Augmentation

To increase the diversity of waveform features, we designed a feature augmentation stage, which includes two steps: phaseamplitude feature augmentation and phase-amplitude feature mixture. In summary, in the prior step, we extracted phase and amplitude features separately using MEPAE. Subsequently, we separately expanded these features, before combining and rearranging them to produce a multitude of distinct waveform samples. The whole process is shown in Figure 4.

Phase-Amplitude Feature Augmentation. We expand phase and amplitude features by performing a Swap in Batch operation, as shown in Figure 4, which randomly disrupt the order of phase and amplitude features within a batch, resulting in new batch-sized groups of phase and amplitude features, respectively. Thus after this session, we already have two pairs of phase-amplitude features.

Phase-Amplitude Feature Mixture. Phase and amplitude features can be aggregated based on the wave function. By cross aggregating two pairs of phase-amplitude features two by two, we can obtain four samples, including the original wave ˜ to , a positive wave ˜ tp with the same phase as the original wave but different amplitudes, and two negative waves ˜ tn1 and ˜ tn2 with different phases. Note that in this process, each aggregation with phase and amplitude follows the mechanism of Eq. 4. According to Euler's formula, the process of wave aggregation is

By substituting the extracted phase and amplitude features separately and drawing on the properties of [Tang et al., 2021], we can directly obtain the final fusion process:

where W t , W i are both learnable weights. j and k denote the j-th row and k-th column of matrix W. ak and pk are amplitude and phase feature, respectively, and ˜ tj denotes the waveform feature of the j-th token.

3.4 Self-supervised Loss and Tracking-supervised Loss

To ensure the robustness and accuracy of the wive-wise features. We design two kinds of constraints for the above enhanced waves as shown in Figure 5. Note that the selfsupervised loss deals with the overall features after all token waves are stretched, whereas the tracking supervised loss is computed with individual token waves as the unit of computation. Combining the two losses from the perspective of global and local constraints can be more effective in improving feature discrimination.

Self-supervised loss. Observing the success of contrastive learning and relative loss fuctions, we adopt the similar selfsupervised loss as [Radford et al., 2021] for our feature augmentation. Specifically, we construct two new samples for learning self-supervised loss, i.e. ˜ Tr and ˜ Ts, one sample is concatenated from the original wave ˜ to and one negative wave ˜ tn1 , and the other sample is concatenated from the positive wave ˜ tp and another negative wave ˜ tn2 . Then we perform normalization on them, respectively:

where the N(·) means the normalization operation.

We conduct tensor multiplication on the concatenated feature ˜t and its transposed feature to compute a similarity logits.

where T means to calculate the transposed value.

Finally we perform a cross-entropy loss on the logits and groundtruth label:

where CE(·) represents the CrossEntropy loss function.

Tracking-supervised loss. Following [Ye et al., 2022], the supervised loss function for the original image features is:

where Oori represents the output of WDT while it takes the original feature as input; gt means the ground-truth label given by the dataset; Lcls is the weighted focal loss [Law and Deng, 2018], Liou is the generalized IoU loss [Rezatofighi et al., 2019] and LL1 means the common l1 loss.

Additionally, since in our method we generate a lot of positive samples with same phase as the original image. Then we put the same constraints on these features as below:

where the Opos means the output of WDT while it takes the positive features as input.

And thus the complete supervised loss is:

where the λSpos = 1 in our method.

To sum up, the total loss for our WDT is:

where the λself = 0.5 in our method.

4 Experiments

4.1 Implementation

Model. Our tracker WDT is built upon ViT [Dosovitskiy et al., 2020] and HiViT [Zhang et al., 2023a], which adopts the template of 192 × 192 pixels and the search of 384 × 384 pixels, respectively abbreviated as WDT-ViT and WDTHiViT. And all of our design modules are inserted after search tokens recovery and before prediction head.

Training. We implement our model in Python using PyTorch and train it with 8 NVIDIA A100 GPUs. And the test are conducted on a single NVIDIA RTX3070 GPU. For WDT-ViT, we set the batch size to 24, the weight decay to 10-4 , the learning rate for the backbone to 4 × 10-5 and the rest parameters to 4 × 10-4 , respectively. The learning rate decreases by a factor of 10 after 240 epochs. For WDT-HiViT, we set the batch size to 4, the initial learning rate of the backbone network to 2 × 10-5 , the learning rate of other parameters to 2 × 10-4 , and the weight decay to 10-4 . The total number of training epochs is 150, and the learning rate decreases by a factor of 10 after 120 epochs. The training datasets are COCO [Lin et al., 2014], LaSOT [Fan et al., 2018], GOT-10k [Huang et al., 2018] and TrackingNet [M¨uller et al., 2018]. The entire tracker is trained using both self-supervised loss and tracking-supervised loss, and the original waves, both the positive and the negatives, are tracked for prediction.

Inference. When tracking inference, we only output the tracking results of the original wave and test on five popular benchmarks: LaSOT [Fan et al., 2018], LaSOText [Fan et al., 2020], GOT-10k [Huang et al., 2018] and UAV123 [Mueller et al., 2016] and TNL2K [Wang et al., 2021b].

4.2 Comparison with State-of-the-Arts

We compare our method with 15 state-of-the-art trackers, which include most representative methods in recent years. The comprehensive results are listed in Table 1.

LaSOT. LaSOT is a widely-used benchmark for long-term tracking. It contains 1,400 videos with more than 3.5M frames in total. And its evaluation metrics are the normalized precision (PNorm), the precision (P) and the area under curve (AUC) of the success plot. Our trackers WDT-ViT and WDT-HiViT rank among the popular comparison methods. WDT-ViT is 0.3% higher than baseline OSTrack in AUC, while WDT-HiViT achieves 73.0% in AUC, 83.3% in PNorm score and 80.7% in P score.

LaSOText. LaSOText is an extension version of LaSOT. It involves 150 videos of 15 classes. The evaluation metrics are the same as the LaSOT. As an extended version of LaSOT, LaSOText contains 150 additional sequences of 15 object classes. Since it is released in recent two years, results on it are relatively fewer but we still conducted new research on it, with AUCs of 52.3% and 53.3%, which is superior to other comparison methods.

TNL2K. TNL2K is also a new dataset for natural language guided tracking. Its testing dataset contains 700 high diversity sequences. To improve the generality of tracking evaluation it introduces several adversarial samples and thermal images. Therefore TNL2K is a challenging benchmark currently. And we boost the performance in each metric and outperform other powerful counterparts like SwinTrack-B and CIA by a substantial margin. WDT-ViT achieves 57.5% in AUC, which surpass OSTrack by 1.6%. WDT-HiViT achieves 59.7% in AUC, and outperform other powerful counterparts like AQAT and STCFormer by a substantial margin.

UAV123. UAV123 is an important dataset from a lowaltitude aerial perspective. We takes the AUC and the precision (P) to evaluate trackers' performance on it. On the UAV123, we are still performing well with 71.1% and 71.2% in AUC, compared to other trackers and are up significantly.

GOT-10k. GOT-10k is consist of 10k sequences for training and 180 videos for testing. It takes the average overlap (AO) and the success rate (SR) at overlap thresholds 0.5 and 0.75 as the evaluation metrics. We evaluated our WDTViT and WDT-HiViT on their official website, and the results showed that the improvements in AO, SR0.5, and SR0.75 were 0.8%, 0.6%, 1.0%, respectively, compared with OSTrack. WDT-HiViT obtains 76.1% in AO, 85.4% in SR0.5 score and 75.3% in SR0.75. Our tracker is also better compared to other trackers.

4.3 Ablation Study

The contributions of Each Module. To judge the exact impact of each part, we test the models equipped with the different component on GOT-10K and LaSOT. When extracting phase and amplitude features, we introduce a mutually exclusive note of semantic separation of phase and amplitude features. As shown in Table 2, the PAE improves tracking performance based on the waveform representation, and the MEPAE improvement is more pronounced with the added mutually exclusive attention. To evaluate the impact of supervised loss and self-supervised loss, we tested the tracking performance of PAE and MEPAE separately with different losses, where WA-T indicates that only tracking supervised loss is included and WA-S indicates that only self-supervised loss is included, As shown in Table 2, the supervised loss is essential for the whole training, and the tracking performance can be further improved with the addition of the selfsupervised loss aid.

Parametric Analysis in Self-supervised Loss. We try different weights for λ for each loss we design. The range of testing weights of each loss is approximately determined by the scale of their values. According to Table 3, results show that 1 and 0.5 are best for Self-Supervised Loss and TrackingSupervised Loss.

Speed and parameters. Our WDT tracker runs at a reduced speed compared to the baseline and its main computational effort is concentrated in the MEPAE module. The parameters of our WDT-ViT are 92M. As for multiplyaccumulate computations (MACs), ours are 57.9G, about 5.4G larger than OSTrack. However, experiments show that there is not too much computational burden of our method. In the same operating environment, our tracker at 53.5 fps, which is slightly slower but still meets the real-time standard, shown in Table 4.

Complex environments. Figure 6 shows the accuracy of LaSOT in different attributes. We can see that our WDT is significantly better than AQAT in IV (illumination variation), FM (fast motion), FOC (full occlusion) , OV (out-of-view), and POC (partial occlusion) situations, demonstrating that WDT can maintain its advantages in more complex environments, especially scenes with significant dynamic changes.

Visualization of tracking. In Figure 7 we display some representative results of our WDT. Obviously, WDT achieves quite accurate results in these scenarios. The results show that WDT can handle the deformation of the target well.

5 Conclusion

In this paper, we propose a novel approach to feature learning in visual object tracking using wave representation. Treating each image patch as a wave function, we extract its phase and amplitude components separately. To enhance feature robustness, we perform augmentation by randomly recombining these components to form positive and negative pairs. A self-supervised constraint regulates this augmentation process, which is combined with tracking-supervised losses to guide wave feature learning at both global and local levels. By embedding these methods into ViT and HiViT, we develop trackers WDT-ViT and WDT-HiViT. Extensive experiments on benchmarks like LaSOT, TNL2K, UAV123, LaSOText and GOT-10K show their superior performance.

Acknowledgments

This work is supported by the Young Scientists Fund of the Hunan Natural Science Foundation (Grant No.2024JJ6474), the Youth Independent Innovation Science Fund Project of NUDT (Grant No.ZK24-08), the National Natural Science Foundation of China (Grant No.62406100), Tianjin Natural Science Foundation (Grant No.24JCQNJC00320) and Beijing Postdoctoral Research Foundation.

Contribution Statement

Huibin Tan and Mingyu Cao contribute equally and are regarded as co-first authors, noted by * . Long Lan and Mengzhu Wang contribute equally and are regarded as cocorresponding authors, noted by † .