MoTRa: Motion-Aware Target Representation Learning for End-to-End Multi-Object Tracking
Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence. Main Track. Pages 1224-1232.
https://doi.org/10.24963/ijcai.2026/137
Keywords
- Computer Vision: Video analysis and understanding
- Computer Vision: Motion and tracking
- Computer Vision: Representation learning
Abstract
Multi-object tracking (MOT) has long faced challenges with identity switches, especially for targets with low appearance discriminability and complex motion. Existing end-to-end trackers typically enhance robustness by modeling long-term temporal information across target-level representations, yet these representations remain insufficiently discriminative for spatially and visually similar targets. In this paper, we present MoTRa, a Motion-aware Target Representation learning framework for end-to-end MOT that enriches target-level representations with adaptive motion cues. As a result, each representation adaptively balances motion cues and appearance-related content cues under varying conditions, enhancing its discriminability for tracking. To achieve this, we propose a feature fusion and alignment module that extracts motion cues and adaptively fuses them with content features into target representations. To further regularize the dynamically fused representations, we propose a target-specific contrastive learning strategy that promotes intra-trajectory consistency and inter-target separability. Experimental results demonstrate the effectiveness of our method, achieving competitive performance on multiple benchmarks.
- [Bernardin and Stiefelhagen, 2008] Keni Bernardin and Rainer Stiefelhagen. Evaluating multiple object tracking performance: the clear mot metrics. EURASIP Journal on Image and Video Processing, 2008:1-10, 2008.
- [Bewley et al., 2016] Alex Bewley, Zongyuan Ge, Lionel Ott, Fabio Ramos, and Ben Upcroft. Simple online and realtime tracking. In 2016 IEEE International Conference on Image Processing, pages 3464-3468. IEEE, 2016.
- [Carion et al., 2020] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, pages 213-229. Springer, 2020.
- [Cui et al., 2023] Yutao Cui, Chenkai Zeng, Xiaoyu Zhao, Yichun Yang, Gangshan Wu, and Limin Wang. Sportsmot: A large multi-object tracking dataset in multiple sports scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9921-9931, 2023.
- [Gao and Wang, 2023] Ruopeng Gao and Limin Wang. Memotr: Long-term memory-augmented transformer for multi-object tracking. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision, pages 9901- 9910, 2023.
- [Gao et al., 2025] Ruopeng Gao, Ji Qi, and Limin Wang. Multiple object tracking as id prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 27883-27893, 2025.
- [Ge et al., 2021] Zheng Ge, Songtao Liu, Feng Wang, Zem- ing Li, and Jian Sun. Yolox: Exceeding yolo series in 2021. arXiv preprint arXiv:2107.08430, 2021.
- [He et al., 2020] Kaiming He, Haoqi Fan, Yuxin Wu, Sain- ing Xie, and Ross Girshick. Momentum contrast for un- supervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 9729-9738, 2020.
- [Huang et al., 2025a] Cheng Huang, Shoudong Han, Mengyu He, Wenbo Zheng, and Yuhao Wei. Dftrack: Deconfused data association framework for multi-object tracking. IEEE Transactions on Circuits and Systems for Video Technology, 35(10):10367-10381, 2025.
- [Huang et al., 2025b] Yuanzhou Huang, Songwei Pei, and Rui Zeng. Dqformer: Transformer with decoupled query augmentations for end-to-end multi-object tracking. ACM Transactions on Multimedia Computing, Communications and Applications, 21(6):1-23, 2025.
- [Li et al., 2024] Haijun Li, Zhuye Xu, Changxi Ma, and Xiao Tang. Multi-object tracking algorithm for unmanned vehicle autonomous driving scene based on online spa- tiotemporal feature correlation. IEEE Access, 12:116489- 116497, 2024.
- [Liu et al., 2022] Shilong Liu, Feng Li, Hao Zhang, Xiao Yang, Xianbiao Qi, Hang Su, Jun Zhu, and Lei Zhang. Dab-detr: Dynamic anchor boxes are better queries for detr. arXiv preprint arXiv:2201.12329, 2022.
- [Liu et al., 2025] Yue Liu, Shijie Zhang, Huayi Li, and Chao Zhang. Satsort: Online multi-spacecraft visual tracking algorithm. Acta Astronautica, 232:103-113, 2025.
- [Luiten et al., 2021] Jonathon Luiten, Aljosa Osep, Patrick Dendorfer, Philip Torr, Andreas Geiger, Laura Leal-Taixé, and Bastian Leibe. Hota: A higher order metric for evalu- ating multi-object tracking. International journal of com- puter vision, 129:548-578, 2021.
- [Meinhardt et al., 2022] Tim Meinhardt, Alexander Kirillov, Laura Leal-Taixe, and Christoph Feichtenhofer. Track- former: Multi-object tracking with transformers. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8844-8854, 2022.
- [Mu and Lin, 2025] Siyuan Mu and Sen Lin. A comprehen- sive survey of mixture-of-experts: Algorithms, theory, and applications. arXiv preprint arXiv:2503.07137, 2025.
- [Qin et al., 2024] Zheng Qin, Le Wang, Sanping Zhou, Pan- pan Fu, Gang Hua, and Wei Tang. Towards generalizable multi-object tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18995-19004, 2024.
- [Ristani et al., 2016] Ergys Ristani, Francesco Solera, Roger Zou, Rita Cucchiara, and Carlo Tomasi. Performance mea- sures and a data set for multi-target, multi-camera track- ing. In Computer Vision -ECCV 2016 Workshops, pages 17-35, Cham, 2016. Springer International Publishing.
- [Segu et al., 2024] Mattia Segu, Luigi Piccinelli, Siyuan Li, Yung-Hsu Yang, Bernt Schiele, and Luc Van Gool. Samba: Synchronized set-of-sequences modeling for multiple ob- ject tracking. arXiv preprint arXiv:2410.01806, 2024.
- [Seidenschwarz et al., 2023] Jenny Seidenschwarz, Guillem Brasó, Victor Castro Serrano, Ismail Elezi, and Laura Leal-Taixé. Simple cues lead to a strong multi-object tracker. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13813- 13823, 2023.
- [Shao et al., 2018] Shuai Shao, Zijian Zhao, Boxun Li, Tete Xiao, Gang Yu, Xiangyu Zhang, and Jian Sun. Crowdhu- man: A benchmark for detecting human in a crowd. arXiv preprint arXiv:1805.00123, 2018.
- [Shim et al., 2025] Kyujin Shim, Kangwook Ko, Yujin Yang, and Changick Kim. Focusing on tracks for online multi-object tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11687-11696, 2025.
- [Sun et al., 2022] Peize Sun, Jinkun Cao, Yi Jiang, Zehuan Yuan, Song Bai, Kris Kitani, and Ping Luo. Dancetrack: Multi-object tracking in uniform appearance and diverse motion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20993- 21002, 2022.
- [Wojke et al., 2017] Nicolai Wojke, Alex Bewley, and Diet- rich Paulus. Simple online and realtime tracking with a deep association metric. In 2017 IEEE International Con- ference on Image Processing, pages 3645-3649. IEEE, 2017.
- [Yang et al., 2023] Fan Yang, Shigeyuki Odashima, Shoichi Masui, and Shan Jiang. Hard to track objects with ir- regular motions and similar appearances? make it easier by buffering the matching space. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 4799-4808, 2023.
- [Yang et al., 2024] Mingzhan Yang, Guangxin Han, Bin Yan, Wenhua Zhang, Jinqing Qi, Huchuan Lu, and Dong Wang. Hybrid-sort: Weak cues matter for online multi- object tracking. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pages 6504-6512, 2024.
- [Yu et al., 2023] En Yu, Tiancai Wang, Zhuoling Li, Yuang Zhang, Xiangyu Zhang, and Wenbing Tao. Motrv3: Release-fetch supervision for end-to-end multi-object tracking. arXiv preprint arXiv:2305.14298, 2023.
- [Zeng et al., 2022] Fangao Zeng, Bin Dong, Yuang Zhang, Tiancai Wang, Xiangyu Zhang, and Yichen Wei. Motr: End-to-end multiple-object tracking with transformer. In European Conference on Computer Vision, pages 659- 675. Springer, 2022.
- [Zeng et al., 2025] Rui Zeng, Yuanzhou Huang, and Song- wei Pei. Tgformer: Transformer with track query group for multi-object tracking. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 39, pages 9824- 9832, 2025.
- [Zhang et al., 2023] Yuang Zhang, Tiancai Wang, and Xi- angyu Zhang. Motrv2: Bootstrapping end-to-end multi- object tracking by pretrained object detectors. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22056-22065, 2023.
- [Zhang et al., 2024] Yang Zhang, Xiaoyan Cui, and Yu Bai. Research on an improved motr-gru multi-target tracking and behavior analysis method. In 2024 4th International Symposium on Artificial Intelligence and Intelligent Man- ufacturing, pages 623-626, 2024.
- [Zhang et al., 2025] Bozhou Zhang, Nan Song, Xin Jin, and Li Zhang. Bridging past and future: End-to-end au- tonomous driving with historical prediction and planning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6854-6863, 2025.
- [Zheng et al., 2024] Guangze Zheng, Shijie Lin, Haobo Zuo, Changhong Fu, and Jia Pan. Nettrack: Tracking highly dy- namic objects with a net. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19145-19155, 2024.
- [Zhu et al., 2020] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020.
References were extracted automatically from the PDF and may contain errors.
Full text extracted automatically from the PDF; figures, tables and equations may differ. The PDF is the version of record.
1 Introduction
Multi-object tracking (MOT) aims to localize all relevant targets in each video frame and assign consistent identities (IDs) over time, enabling applications such as pedestrian analytics [Zhang et al., 2024], autonomous driving [Li et al., 2024], and situational awareness [Liu et al., 2025]. However, achieving reliable identity association remains challenging due to highly similar targets and frequent occlusions.
Most existing MOT methods follow the tracking-bydetection paradigm [Shim et al., 2025], typically coupling an off-the-shelf detector with motion- or appearance-based association strategies. While these approaches achieve impressive performance, their decoupled design and separate optimization make it difficult to jointly model unified target representations and motion dynamics, thereby limiting longterm tracking robustness in challenging scenarios [Zeng et al., 2022]. Recently, end-to-end MOT methods have emerged. By unifying detection and association within Transformerbased architectures, these methods can better learn target representations and capture long-term temporal dependencies for robust tracking. Most prior methods adopt the tracking-bypropagation paradigm [Zeng et al., 2025], where tracked targets are represented by autoregressive track queries propagated across frames to localize their counterparts in the current frame. In contrast, recent work such as MOTIP [Gao et al., 2025] formulates MOT as an in-context ID prediction task, further simplifying the tracking pipeline by binding ID information to each target representation, enabling direct ID decoding for data association. Overall, these advances in endto-end methods demonstrate that better modeling and use of target representations has strong potential for robust tracking in complex scenarios. Nevertheless, current end-to-end methods still struggle to produce sufficiently discriminative target representations, particularly in challenging situations involving severe occlusions and complex interactions.
* Corresponding author.

Building upon DETR-like object detection frameworks
[Carion et al., 2020], recent end-to-end MOT methods typically adopt DETR output embeddings as target-level representations for association, since these embeddings inherently encode positional and appearance information. However, as illustrated in Figure 1, when targets experience complex interactions and occlusions, their output embeddings can become highly similar due to visual and positional similarity, increasing the risk of ID switches and compromising tracking robustness. To address this problem, recent solutions enhance these representations with temporal information [Gao and Wang, 2023], yet the target-level representations remain detectionoriented and limited in discriminability, thereby constraining overall tracking performance. Although introducing additional tracking cues is a natural way to strengthen target representations, naively fusing heterogeneous cues can dilute discriminative signals and hinder reliable association. This leads to a pivotal question: Is it possible to adaptively fuse multiple tracking cues so that the model can emphasize the most discriminative cues under varying tracking conditions?
Motivated by the above perspective, we propose MoTRa, an end-to-end MOT method towards learning more discriminative target representations for robust tracking, as shown in Figure 1. Since DETR output embeddings implicitly encode distinct cues, explicitly enhancing specific cues under different tracking conditions becomes challenging. We therefore decouple the output embeddings into content embeddings and positional information, and further augment the latter with motion trends to form motion embeddings. Here, the content embedding encodes semantic information of the target's appearance and identity, whereas the motion embedding captures inter-frame variations in spatial position. Afterwards, we introduce a motion-aware feature fusion and alignment module (MFA), in which an adaptive gating mechanism fuses motion and content embeddings to emphasize discriminative cues under different tracking conditions. To alleviate the feature inconsistency induced by such dynamic fusion, we further design a target-specific contrastive learning strategy (TSCL), which promotes intra-trajectory consistency and inter-target separability for more stable and discriminative representations. Overall, MoTRa preserves the simplicity of end-to-end architectures while explicitly leveraging readily available motion and content cues to learn tracking-oriented target representations, thereby enhancing target discriminability and achieving impressive performance on DanceTrack [Sun et al., 2022], SportsMOT [Cui et al., 2023], and BFT [Zheng et al., 2024]. Extensive ablation studies validate the effectiveness of the proposed modules. The major contributions of this work are as follows:
- We propose MoTRa, an end-to-end MOT framework that learns discriminative, tracking-oriented target representations for robust tracking in complex scenarios.
- We design an MFA module for adaptive fusion of motion and content cues under varying tracking conditions, and introduce a TSCL strategy to enhance intra-trajectory consistency and inter-target separability.
- Extensive experiments demonstrate that MoTRa enhances tracking performance and achieves competitive results on DanceTrack, SportsMOT, and BFT.
2 RelatedWorks
Tracking-by-Detection. Most existing MOT methods follow the tracking-by-detection (TbD) paradigm, which consists of a detection stage followed by an independent association stage. In the association stage, early TbDmethods [Bewley et al., 2016] primarily leverage motion cues through position prediction and IoU-based matching [Yang et al., 2023]. Later works [Wojke et al., 2017] further refine this by incorporating appearance cues with ReID models for similarity matching. To improve matching robustness, recent approaches refine this process by integrating motion and appearance [Seidenschwarz et al., 2023] and by incorporating additional tracking cues [Yang et al., 2024]. However, these attempts still rely on hand-crafted, similarity-based matching heuristics instead of learning a unified target representation, making it difficult to exploit temporal context for robust tracking, especially under high motion complexity and low appearance discriminability [Qin et al., 2024].
End-to-End Tracking. Most prior methods adopt the Tracking-by-Propagation (TbP) framework [Zeng et al., 2022], which extends DETR-like detectors [Zhu et al., 2020] to tracking by using learnable detect queries and propagated track queries to identify new and existing targets, respectively. Specifically, track queries are derived from previousframe output embeddings [Meinhardt et al., 2022], which are optimized for detection and encode target positional and appearance information. To improve long-term tracking, follow-up works enrich track queries with long-term motion patterns via temporal aggregation modules [Gao and Wang, 2023]. However, TbP still suffers from conflicts caused by the unified processing of the two query types [Huang et al., 2025b]. The recent ID-prediction framework [Gao et al., 2025] alleviates this by decoupling detection and association, where a DETR-like detector first detects objects and obtains their output embeddings, which are then combined with learnable ID tokens for ID prediction. Compared with TbP, this design simplifies the pipeline and allows longerterm historical information to be exploited for association. In this paper, MoTRa follows the ID-prediction paradigm, but learns more discriminative, tracking-oriented representations for association. Rather than directly reusing detection embeddings, it explicitly integrates motion and content cues and emphasizes the most informative cues according to each target's evolving tracking condition, thereby improving tracking performance in complex scenarios.
3 Methods
Most previous Transformer-based end-to-end MOT methods leverage detection output embeddings to represent targets and perform association [Huang et al., 2025b; Gao et al., 2025]. In contrast, our aim is to learn tracking-oriented representations by explicitly integrating motion and content cues, enabling the model to adaptively emphasize discriminative cues based on each target's evolving tracking condition, thereby improving performance in scenarios with complex motion and occlusion. To this end, we present MoTRa, as shown in Figure 2.

3.1 Overview
We implement MoTRa based on an ID prediction framework [Gao et al., 2025], which detects targets and predicts their tracking identities within a single model, thereby enabling end-to-end optimization. Given a sequence of video frames It-T:t, a DETR-like detector is employed to detect targets in each frame and extract their output embeddings Ft-T:t. For association, tracked historical targets Tt-T:t-1 are represented by combining their target-level features Ft-T:t-1 with correlated learnable ID tokens I = {i1 , . . . , iK } as incontext prompts, while newly detected targets in the current frame Dt are represented by Ft and a special ID token ispec . Finally, an ID decoder facilitates interactions between Dt and Tt-T:t-1, allowing the model to predict the ID labels of Dt from their ID token ispec via an ID prediction head. However, as Ft-T:t are optimized for detection, these features cannot reliably distinguish targets under complex occlusions.
In this work, rather than relying on detection-oriented Ft-T:t, we focus on learning more discriminative, trackingoriented features ˜ Ft-T:t for target representation in both Tt-T:t-1 and Dt, thereby enhancing association performance. Specifically, we introduce MFA to adaptively integrate additional motion cues and obtain ˜ Ft-T:t (Section 3.2). In addition, we design a TSCL strategy to ensure the temporal consistency and discriminability of ˜ Ft-T:t (Section 3.3).
3.2 Motion-aware Feature Fusion and Alignment
In real-world tracking scenarios, cue reliability varies across tracking conditions. Positional information is often sufficient when targets are well separated, whereas appearance cues are effective when visual differences are clear. However, when targets experience severe occlusion, the relative reliability of positional and appearance cues becomes uncertain, and static positional cues can be less discriminative than motion cues. Motivated by this, we introduce motion cues into representation learning and propose MFA to derive target-level representations ˜ Ft-T:t that adaptively emphasize discriminative cues under varying tracking conditions.
Motion Cues. Compared with positional cues, motion provides more effective discrimination under occlusion. Unlike cues such as pose, which typically require additional models, motion features are easily derived and can be seamlessly integrated into end-to-end frameworks. Therefore, we exploit motion as an additional tracking cue to enhance association performance. Given the k-th target at frame j (t - T ≤ j ≤ t), we formulate its motion cue as mkj = (xkj , yk j ,wk j , hkj , ∆xkj , ∆yk j ,∆wk j ,∆hkj ) ∈ R8 , where ∆ denotes the offsets from frame j to j + 1, and and all terms are normalized by the image resolution. Specifically, the bounding box coordinates of the target are obtained from its corresponding DETR output embedding F k j using the Bbox head. For targets in the historical trajectories Tt-T:t-1, the motion offsets are computed as coordinate differences between consecutive frames. Since there is no future frame for historical targets at frame t-1, we duplicate the offsets at frame t-2 to pad those at frame t - 1. For targets from the new detections in Dt, since the ID decoder enables interactions between Dt and Tt-T:t-1, we directly predict the motion offsets of these targets from the ID decoder output embeddings using dedicated offset heads, thereby compensating for their unknown motion offsets. Inspired by [Zhu et al., 2020], we refine the predicted offsets layer by layer, enabling the decoder to progressively adjust the predicted offsets across layers.
Feature Fusion andAlignment. Althoughmotion cues are incorporated, they may not reliably distinguish targets under all tracking conditions. To adaptively exploit complementary information, inspired by gating mechanisms [Mu and Lin, 2025], we employ a generator that predicts a Feature Assignment Matrix (FAM) to dynamically balance motion and content cues. Given the DETR output embeddings for all Nj targets in the j-th frame (t - T ≤ j ≤ t), denoted as Fj ∈ RNj×C , we first extract both content and motion embeddings for these targets. Specifically, content embeddings F ′ j ∈ RNj×C are obtained by refining Fj with a feed-forward network (FFN) as a content adapter that emphasizes semantic information relevant to appearance and identity. In parallel, the motion cues Mj ∈ RNj×8 , which encode inter-frame variations in spatial position, are projected into the same Cdimensional space via a dedicated FFN motion adapter, yielding motion embeddings M ′ j ∈ RNj×C . Then, by concatenating F ′ j andM ′ j , we feed them into the FAM generator, which consists of an FFN followed by a softmax operation, to predict the feature assignment matrix:
where Aj = [Acont j ,Amoti j ] ∈ RNj×2 stores the normalized weights assigned to the content and motion embeddings, respectively. Finally, we adaptively fuse the motion and content branches, while keeping a residual connection from Fj :
where F fuse j ∈ RNj×C and ⊙ denotes element-wise multiplication. After the fusion stage, an intra-frame attention module is applied to further refine the fused features F fuse j by explicitly modeling interactions among targets within the same frame, which enhances contextual alignment and intertarget separability, yielding the final tracking-oriented features ˜ Fj for representing targets.
MFA for Current-Frame Detection Refinement. Accurate identity prediction requires effective interactions between historical trajectories and current detections in the ID decoder. To this end, we apply MFA before each crossattention layer to refine target-level representations for both historical trajectories and current detections. As MFA already contains intra-frame attention, which serves the same role as self-attention, we replace the original self-attention module with MFA to refine current-frame features. Together with the offset refinement in MFA, this design allows the decoder to progressively update motion cues and adaptively adjust the fusion patterns between motion and content features.
3.3 Target-Specific Contrastive Learning
While MFA enables feature fusion adaptive to each target's evolving tracking condition, such dynamic fusion may cause the representations of the same target to become temporally inconsistent and reduce the separability between different targets. To address this, we propose a TSCL strategy to supervise the MFA-enhanced target representations, encouraging them to be temporally coherent and identity-discriminative.
Following the current-to-history interaction in the ID decoder cross-attention, TSCL constructs contrastive samples over current-history target pairs, making the supervision consistent with the ID prediction process. For the MFA-enhanced representations, we denote the representation of target identity k at historical frame j, where t - T ≤ j ≤ t - 1, as ˜ F k j , and its current-frame representation from the l-th decoder layer as ˜ F k t,l . TSCL constructs contrastive pairs as ( ˜ F k j , ˜ F k t,l ), treating same-identity pairs as positives and different-identity pairs as negatives. This encourages ˜ F k j to be closer to ˜ F k t,l while separating it from current-frame representations of other identities in layer l. Accordingly, the InfoNCE loss [He et al., 2020] is formulated as:
where τ ∈ (0, 1] is a temperature hyper-parameter and Nt is the total number of targets in the frame t. To fully leverage historical information and enforce consistency across time and decoder depth, we extend this loss by aggregating over all historical frames and all decoder layers:
where Na is the number of targets that appear in both the historical frames and the current frame t, and Nl is the total number of decoder layers.
3.4 Training
In this paper, we introduce additional losses to learn more robust motion-aware representations. Specifically, we adopt a standard L1 loss for offset regression [Zhang et al., 2025], denoted as Loff , and use Eq. 4 for contrastive learning. The tracking loss is defined as:
where Lid is a cross-entropy loss for ID classification [Gao et al., 2025], and each λ is a hyper-parameter for the corresponding loss term. In addition, a detection loss is adopted to supervise the DETR-like detection module [Zhu et al., 2020]:
where Lcls is used for object classification, and LL1 and Lgiou are used for bounding box regression. Finally, we train MoTRa with a unified loss L = Ldet + Ltrk.
4 Experiments
4.1 Datasets andMetrics
We evaluate our method on the DanceTrack [Sun et al., 2022], SportsMOT [Cui et al., 2023], and BFT [Zheng et al., 2024] datasets, which feature high appearance similarity and complex motion. We adopt Higher Order Tracking Accuracy (HOTA), Detection Accuracy (DetA), and Association Accuracy (AssA) from the HOTA metric family [Luiten et al., 2021], as well as MOTA [Bernardin and Stiefelhagen, 2008] and IDF1 [Ristani et al., 2016]. IDF1 and AssAmainly reflect association quality, while MOTA and DetA focus on detection performance. HOTA integrates DetA and AssA to provide a balanced assessment of both detection and association.
| Dataset | Methods | E2E | HOTA↑ | IDF1↑ | MOTA↑ | AssA↑ | DetA↑ |
|---|---|---|---|---|---|---|---|
| DanceTrack [Sun et al., 2022] | Hybrid-SORT* [Yang et al., 2024] DfTrack* [Huang et al., 2025a] TrackTrack* [Shim et al., 2025] MeMOTR [Gao andWang, 2023] MOTRv2† [Zhang et al., 2023] MOTRv3† [Yu et al., 2023] SambaMOTR [Segu et al., 2024] TGFormer* [Zeng et al., 2025] MOTIP [Gao et al., 2025] | ✓ ✓ ✓ ✓ ✓ ✓ ✓ | 62.2 65.5 66.5 68.5 69.9 70.4 67.2 69.1 69.6 | 63.0 68.2 67.8 71.2 71.7 72.3 70.5 71.9 74.7 | 91.6 92.7 93.6 89.9 91.9 92.9 88.1 91.3 90.6 | / 52.4 52.9 58.4 59.0 59.3 57.5 58.3 | / / 80.5 83.0 78.8 81.9 |
| SportsMOT [Cui et al., 2023] | 82.1 | ||||||
| 83.8 | |||||||
| 60.4 | 80.4 | ||||||
| MoTRa (ours) | 71.1 | 76.1 | 90.9 | 62.7 | 80.8 | ||
| MeMOTR [Gao andWang, 2023] | ✓ | 57.8 | 82.0 | ||||
| SambaMOTR [Segu et al., 2024] | ✓ | 68.8 69.8 | 69.9 | 90.2 90.3 | 59.4 | 82.2 | |
| MOTIP [Gao et al., 2025] | ✓ | 72.6 | 71.9 77.1 | 92.4 | 63.2 | 83.5 | |
| MoTRa (ours) | ✓ | 73.8 | 77.9 | 93.0 | 64.8 | 84.1 | |
| NetTrack* [Zheng et al., 2024] | 68.7 | 77.0 | 78.9 | 65.2 | 72.6 | ||
| SambaMOTR [Segu et al., 2024] et | ✓ | 69.6 | 81.9 | 72.0 | 73.6 | 66.0 | |
| BFT [Zheng et al., 2024] | MOTIP [Gao al., 2025] | ✓ | 70.5 | 79.3 | |||
| 77.1 | 71.8 | 69.6 | |||||
| MoTRa (ours) | ✓ | 71.3 | 82.9 | 77.8 | 72.7 | 70.3 |
4.2 Implementation Details
MoTRa is built upon the MOTIP baseline [Gao et al., 2025]. For a fair comparison, we use the same Deformable DETR [Zhu et al., 2020] as the DETR-like detector and keep the pretrained weights and parameter settings identical to MOTIP. Specifically, we store up to K = 50 ID tokens for labeling target identities, based on the maximum number of targets in a frame. The maximum length of historical frames T is set to 29, 59, and 19 for DanceTrack [Sun et al., 2022], SportsMOT [Cui et al., 2023], and BFT [Zheng et al., 2024], respectively. During training, we set λcls, λL1, λgiou, and λid to 2.0, 5.0, 2.0, and 1.0, respectively. We additionally set λoff = 5.0, identical to λL1, and λcl = 1.0 for representation learning. In TSCL, we set the temperature parameter τ to 0.02. We use the AdamW optimizer with an initial learning rate of 0.0001 and a batch size of 1, and multiply the learning rate by 0.1 at scheduled epochs. For DanceTrack, training lasts 10 epochs, with the learning rate reduced at epochs 5 and 9. For SportsMOT, training lasts 13 epochs, with reductions at epochs 8 and 12. For BFT, training lasts 22 epochs, with reductions at epochs 16 and 20. During inference, we set the thresholds for DETR output filtering, newborn objects, and ID prediction to 0.3, 0.6, and 0.2, respectively. All experiments are conducted on a single NVIDIA RTX 4090 GPU.
4.3 Comparison with State-of-the-art Methods
We compare MoTRa with previous methods in Table 1. For Transformer-based end-to-end methods, different DETR-like detectors [Liu et al., 2022] and extra detection datasets [Shao et al., 2018] can significantly affect detection performance and robustness [Gao and Wang, 2023; Segu et al., 2024]. Following MOTIP [Gao et al., 2025], we adopt Deformable DETR [Zhu et al., 2020] and train MoTRa without extra detection datasets for a fair comparison. Despite these constraints, MoTRa still achieves superior association performance, particularly in terms of HOTA, IDF1, and AssA.
DanceTrack and SportsMOT. DanceTrack [Sun et al., 2022] presents group dancer tracking scenarios characterized by uniform appearance and diverse motion, while SportsMOT [Cui et al., 2023] focuses on athlete tracking with high-speed movements and complex interactions. Both datasets pose significant challenges for association, especially in distinguishing targets under occlusion. Therefore, we select DanceTrack and SportsMOT as our primary benchmarks for comparative evaluation. For non-end-to-end methods such as Hybrid-SORT [Yang et al., 2024], which leverage powerful detectors [Ge et al., 2021] and incorporate additional tracking cues to enhance performance, the reliance on manual design makes it challenging to learn unified feature representations for targets, resulting in association performance that is still significantly outperformed by end-to-end methods. For end-to-end methods, MOTIP [Gao et al., 2025] simplifies MOT as an ID prediction task and represents targets with additional identity tokens, achieving superior performance compared to previous tracking-by-propagation approaches such as MeMOTR [Gao and Wang, 2023]. Building on the ID prediction framework, MoTRa further enhances target representations by adaptively integrating motion cues and employing a TSCL training strategy. This yields improvements over the baseline, particularly in HOTA, IDF1, and AssA, demonstrating the effectiveness of our method.
BFT. Unlike human tracking, BFT [Zheng et al., 2024] focuses on open-world, highly dynamic bird tracking, which also poses significant challenges in distinguishing targets due to low discriminability. Compared to the MOTIP baseline, MoTRa still demonstrates improvements in overall performance, with a particularly notable gain of +3.6 IDF1, further demonstrating the generalization ability of our approach.

4.4 Visualization Analysis
We provide a visual analysis of MoTRa's effectiveness in representing targets in challenging scenarios on DanceTrack [Sun et al., 2022], as shown in Figure 3. In Figure 3(a), we visualize how the fusion of motion and content features adapts across frames and scenarios. Since targets across scenarios naturally rely differently on motion and content cues, we focus on temporal changes in the fusion weights (curves) rather than their absolute magnitudes. For the same target, the visualization shows that changing tracking conditions drive our model to rebalance content and motion. Besides, targets with different occlusion patterns also induce different fusion strategies. In DanceTrack0019, the occluded dancer gradually exhibits poses and motions similar to the occluder, leading to a decreased motion weight at frame #277, whereas in DanceTrack0001, larger motion and position differences result in the opposite trend.
In Figure 3(b), we illustrate the effectiveness of the proposed TSCL strategy. Despite the distinct fusion patterns of motion and content features across frames, representations of the same targets in both history frames (circle markers) and the current frame (star markers) remain well clustered, demonstrating strong temporal consistency. Moreover, targets with different identities are more clearly separated than in the original DETR output embeddings, further highlighting the improved discriminability of our learned representations.
4.5 Ablation Study
We conduct ablation experiments on the DanceTrack validation set, using MOTIP [Gao et al., 2025] as the baseline. The hyperparameter T is fixed to 19 during ablation.
| Row | Extra Cue | Gating | TSCL | HOTA↑ | IDF1↑ | MOTA↑ |
|---|---|---|---|---|---|---|
| #1 | N/A (Baseline) | 57.8 | 60.8 | 80.8 | ||
| #2 | Position | ✓ | 58.9 | 62.1 | 80.5 | |
| #3 | Position | ✓ | ✓ | 59.7 | 64.0 | 81.4 |
| #4 | Motion | ✓ | 59.8 | 63.3 | 80.1 | |
| #5 | Motion* | ✓ | 60.5 | 64.4 | 81.1 | |
| #6 | Motion* | ✓ | 59.9 | 64.2 | 80.5 | |
| #7 | Motion* | ✓ | ✓ | 61.2 | 65.8 | 81.1 |
Target Representation Learning. Compared with previous end-to-end MOT methods [Gao et al., 2025], MoTRa introduces additional motion cues into target representation, leverages a gating mechanism to adaptively fuse them, and adopts a contrastive learning strategy to enhance representation discriminability. We therefore conduct an overall ablation to validate these components, as shown in Table 2.
For additional tracking cues, position and motion can be derived from detection results without extra models. The position cue encodes absolute coordinates, and the motion cue captures their temporal changes. Although the original DETR output embeddings F already contain positional information, explicitly decoupling and adaptively fusing these cues improves tracking, yielding +1.1 HOTA (Row #2 vs. Row #1). On top of position cues, incorporating motion offsets as motion cues brings an additional +0.9 HOTA (Row #4 vs. Row #2), confirming the benefit of motion information. Since no future frame is available to compute motion offsets for new detections, we predict their motion offsets to compensate for this missing information, further improving their representations for association (Row #5 vs. Row #4).
For different fusion strategies, we adopt a gating mechanism that allows the model to adaptively fuse multiple cues (Row #7), rather than simply adding them (Row #6). This design enables target representations to emphasize different cues under varying conditions (Figure 3(a)). The results show that the adaptive fusion strategy achieves better performance, bringing a gain of +1.3 HOTA and further validating the effectiveness of our gating-based fusion.
For the contrastive learning strategy, we adopt TSCL during training to enforce temporal consistency for the same targets and enhance discriminability between different targets. As shown in Table 2, applying TSCL further improves tracking performance for both position (Row #3 vs. Row #2) and motion (Row #7 vs. Row #5) modeling. These improvements, together with the visualization in Figure 3(b), further demonstrate the effectiveness of the TSCL strategy.
Architectural Design. To facilitatemore effective information interaction between historical trajectories and detections in the current frame, we apply MFA to both historical targets T and new detections D. We therefore conduct ablations on these structural designs, as shown in Table 3. Notably, since TSCL cannot be adopted when MFA is applied only to T , we disable TSCL in all ablations in Table 3 for a fair comparison.
Building on the baseline model, we first apply MFA only to T . Despite the semantic misalignment between T and D, incorporating motion cues already helps distinguish different trajectories, yielding a gain of +1.6 HOTA. To further align T and D, we then apply a single MFA module to D directly after detection. However, at this stage it is difficult to predict motion offsets for D, leading to ineffective fusion and only negligible gains. To obtain predicted motion offsets and update fusion strategies at each decoder layer, we instead apply MFA across layers, which brings notable improvements over using MFA only on T . Finally, inspired by the bounding box refinement in [Zhu et al., 2020], we refine motion offsets iteratively across layers rather than predicting them independently, yielding a slight additional improvement.
| Target Set | History | Current | Layers | HOTA↑ | IDF1↑ | MOTA↑ |
|---|---|---|---|---|---|---|
| N/A (Baseline) | 57.8 | 60.8 | 80.8 | |||
| T | ✓ | 59.4 | 63.3 | 80.1 | ||
| T &D | ✓ | ✓ | Single | 59.6 | 63.9 | 79.8 |
| T &D* | ✓ | ✓ | Multi | 60.2 | 64.2 | 80.7 |
| T &D | ✓ | ✓ | Multi | 60.5 | 64.4 | 81.1 |
| FAM predict | Residual | HOTA↑ | IDF1↑ | MOTA↑ |
|---|---|---|---|---|
| F | ✓ | 58.3 | 62.2 | 80.4 |
| ADD(F ′ +M ′ ) | ✓ | 60.0 | 63.2 | 81.7 |
| Cat(F ′ , M ′ ) | 59.7 | 63.1 | 80.1 | |
| Cat(F ′ , M ′ ) | ✓ | 61.2 | 65.8 | 81.1 |
Gating and Fusion Strategies. To adaptively fuse distinct cues, we predict an assignment matrix (FAM) conditioned on the concatenation of the content and motion embeddings, F ′ and M ′ . As shown in Table 4, directly using the DETR output embeddings F for FAM prediction brings only negligible gains over the baseline. Although F already encodes appearance and positional information, it lacks temporal motion cues and is therefore suboptimal for learning effective FAMs. In contrast, incorporating motion via ADD or Cat yields clear improvements in tracking performance, with the concatenation design performing best. We further introduce a residual connection during feature fusion to stabilize optimization, which proves crucial for obtaining notable gains.
Intra-frame Attention. The self-attention mechanism in the ID decoder enables information exchange among detected targets within the current frame, which facilitates identity discrimination for association [Gao et al., 2025]. Motivated by this, we extend self-attention to historical targets as an intraframe attention module and integrate it into our MFA. As shown in Table 5, applying the attention to historical targets after feature fusion yields a gain of +0.7 HOTA, indicating that historical targets within the same frame also benefit from information exchange for better feature alignment. We further investigate the placement of the attention relative to feature fusion. Placing attention before fusion allows information exchange only among the raw DETR output embeddings, which lack temporal motion cues. In contrast, placing attention after fusion operates on already fused representations, helping align their semantic information and better enhancing discriminability, thereby achieving better performance.
| Attention | Placement | HOTA↑ | IDF1↑ | MOTA↑ |
|---|---|---|---|---|
| D | After Fusion | 60.5 | 64.0 | 80.6 |
| D & T | Before Fusion | 60.3 | 63.7 | 81.5 |
| D & T | After Fusion | 61.2 | 65.8 | 81.1 |

Effectiveness under occlusion. We further evaluate association performance under different occlusion levels, as shown in Figure 4. Occlusion severity for each video is defined as the ratio between severe overlaps (IoU > 0.6) and all overlaps (IoU > 0.1) [Shim et al., 2025]. MoTRa shows overall gains in IDF1 and AssA over the baseline, especially under higher occlusion severity, indicating improved robustness.
Inference Speed and Parameters. Compared to the baseline, which runs at 27.1 FPS with 59M parameters, our method achieves 25.8 FPS with 63M parameters on the DanceTrack validation set. Despite the extra computational cost, our model remains feasible for practical applications.
4.6 Discussion
Although we introduce a motion-aware target representation learning method, its detection gain remains limited since ID prediction is decoupled from detection. Building on motion cues, we plan to more effectively exploit predicted motion to assist detection in subsequent frames, which we believe can further improve overall tracking performance.
5 Conclusion
This paper presents MoTRa, a motion-aware target representation learning framework that enhances detection-oriented features with motion-aware tracking cues to improve target discriminability through an MFA module and a TSCL strategy. MFA adaptively fuses motion cues with appearancerelated content cues to emphasize the most informative cues under varying tracking conditions, while TSCL regularizes the fused representations by promoting intra-trajectory consistency and inter-target separability. Experiments demonstrate the effectiveness of MoTRa.
Ethical Statement
There are no ethical issues.
Acknowledgements
This work was supported by the Beijing Municipal Science and Technology Program (Z251100003625011), the Beijing Natural Science Foundation (L252018), the NSFC through grant No.62402054, the China Postdoctoral Science Foundation through grant 2024M760279, and the Postdoctoral Fellowship Program and China Postdoctoral Science Foundation under grant BX20250390.
@inproceedings{ijcai2026p137,
title = {MoTRa: Motion-Aware Target Representation Learning for End-to-End Multi-Object Tracking},
author = {Huang, Yuanzhou and Pei, Songwei and Wang, Shuhuai and Liu, Bingfeng and Li, Qian and Wang, Shangguang},
booktitle = {Proceedings of the Thirty-Fifth International Joint Conference on
Artificial Intelligence, {IJCAI-26}},
publisher = {International Joint Conferences on Artificial Intelligence Organization},
editor = {Diego Calvanese},
pages = {1224--1232},
year = {2026},
month = {8},
note = {Main Track},
doi = {10.24963/ijcai.2026/137},
url = {https://doi.org/10.24963/ijcai.2026/137},
}
TY - CPAPER TI - MoTRa: Motion-Aware Target Representation Learning for End-to-End Multi-Object Tracking AU - Huang, Yuanzhou AU - Pei, Songwei AU - Wang, Shuhuai AU - Liu, Bingfeng AU - Li, Qian AU - Wang, Shangguang T2 - Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence, IJCAI-26 PB - International Joint Conferences on Artificial Intelligence Organization PY - 2026 SP - 1224 EP - 1232 DO - 10.24963/ijcai.2026/137 UR - https://doi.org/10.24963/ijcai.2026/137 ER -Copied
Machine-readable versions of this paper, for tools and AI agents: