Unsupervised Embedding and Association Network for Multi-Object Tracking

1 Introduction

Multi-object tracking (MOT), which aims at estimating reliable trajectories of multiple targets in a video sequence, has a wide range of applications, for example, modern autonomous driving [Milan et al., 2016] and intelligent robot vision analysis [Dendorfer et al., 2020]

mainly adopt the tracking-by-detection paradigm, i.e., they continuously detect multiple targets with detectors and then associate those detection predictions into trajectories with different data association algorithms [Zhang et al., 2021b; Wu et al., 2021].

Despite different association algorithms, some MOT methods [Zou, 2020; Wu et al., 2021; Zhang et al., 2021b; Wang et al., 2021b; Guo et al., 2021; Sun et al., 2021; Xu et al., 2020] achieve the impressive performance by addressing the online MOT task with the following two steps: (a) detection and identity embedding (or motion predicting), and (b) embedding feature/motion prediction-based data association. The above two-step tracking procedure inspires three feasible ways to improve tracking performance. For example, some methods [Zhang et al., 2021b; Wang et al., 2021b] adopt the anchor-free detector [Zhou et al., 2019] to discover occluded targets. Some other methods [Zou, 2020; Wu et al., 2021; Zhang et al., 2021b; Wang et al., 2021b; Guo et al., 2021] use an extra identity embedding (or motion predicting) branch for simultaneously generating detection predictions and embedding features (or motion predictions) in their unified networks with the manual annotations of targets. In terms of data association, some deep data association algorithms [Sun et al., 2021; Xu et al., 2020] bring lower IDS and higher Multi-Object Tracking Accuracy (MOTA [Bernardin and Stiefelhagen, 2008]) than hand-crafted constraints.

However, those previous methods mainly focus on the hand-crafted association algorithms, which suppresses the ability to effectively exploit discriminative embedding features to generate robust trajectories. Moreover, learning embedding features with the manual identity annotations brings a high economic cost for supervised MOT methods.

In this paper, we present a novel unsupervised transformerbased embedding and association network (UEANet, as shown in Figure 2). To address the problems of highcost manual annotations and full use of embedding features, UEANet adopts pseudo identity labels to generate discriminative embedding features with a transformer-based discriminative feature enhancer (DFE) module, and it performs an end-to-end data association process in an unsupervised way. We summarize the main contributions as follows:

• We present a novel unsupervised embedding and association network (UEANet) to learn discriminative embedding features and perform an end-to-end data association process for the unsupervised MOT task.
• We train the transformer-based identity embedding branch and MLP-based association branch of UEANet with pseudo identity labels, which go beyond the limitation of high-cost manual identity annotations used in the supervised MOT task.
• UEANet achieves comparable performance compared with previous advanced methods on three MOT datasets, and it ranks second in terms of MOTA.

2 RelatedWork

Recent MOT methods. Tracking by detection is the most popular tracking paradigm, as mostly adopted in recent MOT methods [Zou, 2020; Wu et al., 2021; Zhang et al., 2021b; Wang et al., 2021b; Guo et al., 2021; Sun et al., 2021; Xu et al., 2020]. These methods focus on reliable detection predictions and embedding features (or motion predictions) for subsequently associating targets into trajectories with different data association algorithms. For example, TraDes [Wu et al., 2021] extends the advanced detector [Zhou et al., 2019] with the extra motion prediction branch and uses the motion predictions to perform data association based on the boundingbox IoU constraint. Some other methods [Sun et al., 2021; Xu et al., 2020] adopt fully-connected (FC) network-based association algorithms, which further improve their tracking performance. For example, SST [Sun et al., 2021] employs the deep affinity network to use the pairs of object appearance features between different frames for generating trajectories in the end-to-end way, which highly improves the performance on IDS. However, the hand-craft constraints or simple deep association network need discriminative embedding features (or motion predictions) learned in the supervised way. The manual identity annotations are essential in previous supervised methods. In contrast, recent unsupervised method [Liu et al., 2022] trains the re-ID branch without any manual identity label in the unsupervised way. Moreover, it is adversely affected by confusing embedding features caused by unsupervised training.

We follow the unsupervised learning fashion and present a transformer-based embedding and association network to adopt the pseudo identity labels instead of manual annotations for the unsupervised MOT task.

Our unsupervised method. In this paper, we try to generate discriminative embedding features with the proposed discriminative feature enhancer module, and suppress the competition between detection and identity embedding branches. Moreover, the MLP-based data association branch takes the embedding features of candidate trajectories as input and generates final trajectories in an end-to-end way. Note that we train the identity embedding and data association branches without using any manual identity label. The proposed unsupervised embedding and association network (UEANet) breaks through the limitation of high-cost manual annotations, and achieves comparable performance against previous advanced methods.

3 ProposedMethod

As the overview of UEANet shown in Figure 2, Our method consists of the detection and identity embedding branches integrated with the transformer-based discriminative feature enhancer (DFE) module, and the subsequent MLP-based data association branch. We train the identity embedding and data association branches with the pseudo identity labels generated by the Kalman filter tracker of ByteTrack [Zhang et al., 2021a]. Next, we introduce the motivation of UEANet to extract discriminative embedding features with the DFE module in the unsupervised way.

3.1 Motivation of Our UnsupervisedMethod

Due to the high economic cost of manual identity annotations in the supervised methods, we train the proposed UEANet with pseudo identity labels. As the pseudo identity labels are obtained with relatively poor detection predictions, they lack the interfering identity labels of the near-completely occluded targets. However, the manual detection labels and the pseudo identity labels are not one-to-one correspondences, and bring the competition of network optimization between the detection and identity embedding branches, resulting in confusing embedding features for data association. Unlike the previous unsupervised method [Liu et al., 2022] suffering from the confusing embedding features ect (formulated as Eq.(1)), we develop the DFE module to concurrently enhance the embedding and detection features for generating reliable detection predictions dt and discriminative embedding features et, which are formulated as

wheref P and f U denote the previous and the proposed unsupervised detection and embedding branches. Ft denotes the current-frame image. Then, we perform the end-to-end data association, mathematically defined as

where ge2e denotes the proposed end-to-end unsupervised association branch that calculates the association matrix A between the detection predictions from the frames t and t - 1.

The final trajectories are commonly obtained with the association matrix A, which is learned with pairs of embedding features corresponding to candidate trajectories between the current frame and the associated trajectories, like SST [Sun et al., 2021]. In addition, we jointly train the identity embedding and data association branches to perform MOT in the end-to-end unsupervised way.

3.2 Unsupervised Identity Embedding

We employ the proposed transformer-based discriminative feature enhancer (DFE) in the detection and identity embedding branches of UEANet, as shown in Figure 2. The DFE module aims to suppress the branch competition and generate reliable detection predictions and discriminative embedding features. It contains two components: attention and MLP.

Attention variants. The attention component of DFE is selected from multiple variants, such as self-attention [Vaswani et al., 2017], SGE-attention [Li et al., 2019], shuffle-attention [Zhang and Yang, 2021], and SPSelf-attention [Liu et al., 2021]. It aims to suppress the branch competition by generating branch-dependent features for the detection and identity embedding branches. For example, SGE-attention [Li et al., 2019] improves the learning of different semantic subfeatures of each branch, and intentionally enhances the respective spatial distribution of features for the two branches. The above processes can be formulated as

where xt is the input features of the current frame t. xct and xpt are the detection-dependent features and identitydependent features for the detection and identity embedding processes, respectively. MLPMixer denotes the MLP component.

MLP variants. SomeMLP variants, such as FFN [Vaswani et al., 2017], ResMLP [Touvron et al., 2021], RepMLP [Ding et al., 2021] and MLPMixer [Tolstikhin et al., 2021], are taken into consideration for the MLP component of DFE. For example, MLPMixer [Tolstikhin et al., 2021] is to clearly separate the per-location operations and cross-location operations in multi-layer perceptrons for feature extraction, which enhances the fitting ability of the attention component of DFE, as shown in Eq.(4)&(5). With the cooperation of the two components, the proposed DFE module effectively extracts branch-dependent features to generate reliable detection predictions and discriminative embedding features for the subsequent data association process.

3.3 Unsupervised Data Association

The MLP-based data association branch is integrated into the detection and identity embedding network to replace the hand-craft association algorithms. It is jointly optimized with the detection and identity embedding branches in the unsupervised way. Note that we employ the same MLP component as the one in the DFE module.

As shown in Figure 2, each detection prediction has the corresponding embedding feature. We build embedding feature pairs to mark each candidate trajectory between detection predictions in the past frame t - 1 and the current frame t, similar to SST [Sun et al., 2021]. Moreover, for each detection prediction of t, we only select top-n embedding feature pairs to reduce computational cost. The MLP-based data association branch outputs probabilities of candidate trajectories included in the ground truth, i.e., association confidence scores ct, which are formulated as

where (et, et-1) are pairs of embedding features for candidate trajectories. We keep the trajectories whose association confidence scores are larger than a confidence threshold β.

Compared with the hand-craft association algorithm in [Zhang et al., 2021b] and the FC-based deep association network [Sun et al., 2021], our MLP-based association branch performs the unsupervised end-to-end data association process and achieves robust trajectories (see Figure 3 and Table 2).

3.4 Training Steps and Loss

To adopt unsupervised training of the proposed UEANet, before the epoch T, we firstly generate detection predictions with the detection branch equipped with the network parameters of the past epoch T - 1. Then, we construct trajectories with the Kalman filter tracker of ByteTrack [Zhang et al., 2021a] for generating the pseudo identity labels. At the epoch T, we adopt the pseudo identity labels pGt to train the identity embedding and association branches while training the detection branch with the original manual detection labels. Moreover, embedding features et are mapped to the corresponding identities pt with a FC layer and a softmax function. We split the long trajectories into tracklets to generate the data association labels cGt . Therefore, we add the extra identity embedding loss Lie and data association loss Lda to the original detection loss Ldet utilized in the detector [Zhou et al., 2019] for training the whole tracking network with Eq.7.

where Lie and Lda are the common K-class and 2-class cross-entropy losses. K is the number of trajectories. For Lda, we set the association label to 1 when the candidate trajectory belongs to the pseudo trajectories; otherwise, we set it to 0.

4 Experiments

4.1 Implementation Details

We evaluate the proposed UEANet on the MOT2016 [Milan et al., 2016], MOT2017 [Milan et al., 2016] and MOT2020 [Dendorfer et al., 2020] datasets. For ablation studies in Section 4.2, we follow the previous methods [Zhang et al., 2021b; Wang et al., 2021b; Wu et al., 2021] to use the first half of each video sequence of the MOT2017 training set for training while using the second half for validation. We follow the unsupervised tracking fashion to train the backbone [Zhang et al., 2021b], and the detection, identity embedding and data association branches of UEANet for 30 epochs with a learning rate of 1 × 10-4 and a mini-batch size of 24 on 4 RTX2080 Ti GPUs (using 2 RTX2080 Ti GPUs for ablation studies). We set m to 1 for both detection and identity embedding branches, and experimentally set n to 3. The association confidence threshold β is 0.4 during inference.

Following the prior methods [Zou, 2020; Wu et al., 2021; Zhang et al., 2021b], we use the common evaluation metrics: HOTA, IDF1, and the CLEAR metrics [Bernardin and Stiefelhagen, 2008], etc..

4.2 Ablation Studies

Supervised vs. unsupervised tracking. We evaluate the baseline [Zhang et al., 2021b] and the proposed UEANet based on both supervised and unsupervised training. The results are shown in the first and sixth rows of Table 1. UEANet achieves comparable performance on MOTA, IDF1 and IDS compared with the supervised UEANet, while the supervised baseline outdistances the unsupervised one. These results indicate that UEANet effectively suppresses the adverse influence of the branch competition caused by the pseudo labels and extracts discriminative branch-dependent features by the DFE module for the unsupervised MOT task.

Attention variants. To validate the contribution of the attention component, we adopt the different attention variants in the DFE module and observe their influences on performance. The comparison results are shown in the third, fifth and sixth rows of Table 1. Compared to the unsupervised baseline, the UEANet variants only equipped with the attention component, achieve the different and better tracking results on MOTA, IDF1 and IDS. Moreover, the UEANet variants equipped with both attention and MLPMixer components, further improve the performance on MOTA. For example, UEANet (SGE-Attention + MLPMixer) achieves the improvement of 1.0%/0.7% on MOTA/IDF1 compared with UEANet (Self-Attention + MLPMixer). These observations confirm the important role of the attention component in the DFE module for generating branch-dependent features.

MLP variants. As shown in Table 1, we use four kinds of MLP variants in the detection, identity embedding and data association branches to evaluate their contributions to UEANet. The UEANet variants achieve better tracking performance than the unsupervised baseline while performing the end-to-end MLP-based data association process. Moreover, different MLP variants bring different performance in terms of the cooperation between the attention and MLP components. For example, UEANet (SGE-Attention + MLPMixer) has better performance on MOTA than UEANet (SGE-Attention + FFN), which verifies the different but effective fitting abilities of MLP variants to generate discriminative embedding features and reliable trajectories.

Visualization of trajectories. As shown in Figure 3, we present the visualization of the predicted trajectories obtained by the UEANet variants. The False Negatives (FN [Bernardin and Stiefelhagen, 2008]) of targets are displayed in the figure. we visually evaluate the performance of these variants according to the completeness of trajectories. The proposed UEANet obtains the best trajectories against other variants, which visually proves the superiority of UEANet to suppress the adverse influence of the pseudo identity labels and generate robust trajectories for the unsupervised MOT task.

4.3 Experiments on the MOT Test Sets

MOT2016 and MOT2017. For MOT2016 and MOT2017 [Milan et al., 2016], the official MOTChallenge evaluation server obtains the tracking results for the proposed UEANet. As shown in the first and second rows of Table 2, UEANet ranks top-1 or top-2 among all the published methods. Compared with the second-performance method [Wang et al., 2021b], it achieves an improvement of 1.3%/3.3%/2.2% on MOTA/IDF1/HOTA for MOT2016. Note that UEANet achieves the desirable performance with less computational cost than the top-1 method, ByteTrack [Zhang et al., 2021a] (4 RTX2080 Ti GPUs vs. 8 V100 GPUs) and meanwhile decreases IDS by 30.2% from 2196 to 1533 on MOT2017. Moreover, it outperforms the unsupervised method, UTracK [Liu et al., 2022] on MOTA, IDF1 and IDS. These results indicate that UEANet is a strong end-to-end baseline for the unsupervised tracking task. It exploits the transformer-based DFE module to learn discriminative embedding features and generate reliable trajectories based on the pseudo identity labels.

MOT2020. The MOT2020 [Dendorfer et al., 2020] dataset includes much more persons per frame and heavier occlusion cases than the MOT2017 dataset. As shown in the third row of Table 2, the proposed UEANet also ranks top-2 among all the published methods. However, it only achieves the relatively lower tracking performance (on MOTA, HOTA and IDS) on MOT2020 compared to MOT2017. The reason is most likely the heavier occlusion cases of MOT2020, which have the adverse influence on the pseudo identity labels and the unsupervised training of UEANet. Note that UEANet surpasses UTracK [Liu et al., 2022] and achieves the excellent IDF1 on all three datasets, which directly demonstrates the strong ability to extract the discriminative embedding features for identifying similar targets. The results further indicate that UEANet effectively suppresses the branch competition caused by the pseudo identity labels. It effectively extracts the branch-dependent features with the DFE module for the unsupervised MOT task, even in crowded scenarios.

Visualization of test results. We show the tracking results obtained on the MOT2017 and MOT2020 test sets in Figure 1. These tracking results confirm the outstanding ability of UEANet to deal with the unsupervised MOT task.

Failure examples. As the predicted trajectories shown in Figure 3, the FN of the UEANet variants are visually displayed in the No. 0013 sequence of the MOT2017 validation set. The proposed UEANet achieves much better trajectories than the unsupervised baseline. However, there are still many FN compared with the ground-truth trajectories, which means there is still room for improvement in our method.

5 Conclusion

In this paper, we present a novel tracking network integrated with a transformer-based identity embedding branch and a MLP-based data association branch to perform the unsupervised MOT task with pseudo identity labels instead of highcost manual identity annotations. To alleviate the adverse influence of these pseudo labels, we develop a discriminative feature enhancer (DFE) module to suppress the branch competition and extract branch-dependent features for the unsupervised identity embedding and end-to-end data association processes. We evaluate many UEANet variants in the experiments and make the proposed UEANet a strong baseline for the subsequent researches. Extensive evaluations demonstrate the superiority of our unsupervised method on the three MOT datasets.