D3ETR: Decoder Distillation for Detection Transformer

1 Introduction

The concept of Knowledge distillation (KD) [Hinton et al., 2015] involves transferring knowledge from a large teacher model to a small student model, to enhance the student's performance without incurring costs during model inference. There has been significant progress in the development of KD

methods in recent years, with encouraging results in many vision tasks, such as image classification [Romero et al., 2014; Zagoruyko and Komodakis, 2016; Cho and Hariharan, 2019; Zhang et al., 2018; Tian et al., 2019; Zhou et al., 2021; Yang et al., 2021; Chen et al., 2021; Zhao et al., 2022] and object detection [Chen et al., 2017; Zhang and Ma, 2020; Hao et al., 2020; Dai et al., 2021; Guo et al., 2021; Zhang et al., 2022c; Yang et al., 2022a; Yang et al., 2022b]. However, these methods are predominantly focused on CNNbased models [Simonyan and Zisserman, 2015; He et al., 2016; Sandler et al., 2018] and are related to model structures, particularly in object detection [Chen et al., 2017; Hao et al., 2020; Yang et al., 2022a; Zhang et al., 2022c]. Applying existing KD techniques to novel detectors, such as DETR-based detectors [Carion et al., 2020; Zhu et al., 2020; Meng et al., 2021; Chen et al., 2022c; Liu et al., 2022; Li et al., 2022], poses some challenges and may result in unsatisfactory improvements. Hence, this paper seeks to address this gap by exploring KD strategies for DETR-based detectors.

DETR [Carion et al., 2020] is an end-to-end detector that employs transformer layers [Vaswani et al., 2017]. The pipeline of DETR and its variants involves (i) extracting image features with a backbone, (ii) modeling global context with a transformer encoder, and (iii) predicting objects with a transformer decoder, given the image features and object queries. To develop KD baselines and recipes for DETRbased detectors, we analyze the impact of each component and find that the transformer decoder is crucial for maintaining good performance. Therefore, we focus on exploring KD techniques in the transformer decoder.

However, there is a challenge that the DETR decoder produces outputs in a random order 1 , resulting in no direct correspondence between the decoder outputs of the teacher model and the student model. To address this issue, we introduce MixMatcher, which aims to align the decoder outputs of the teacher and student models. MixMatcher combines two strategies, Adaptive Matching and Fixed Matching, to establish teacher-student correspondence. Adaptive Matching computes the optimal bipartite matching [Kuhn, 1955] between predictions in each decoder layer of teacher and student models. To alleviate the instability issue of bipartite matching in teacher-student Adaptive Matching, we introduce Fixed Matching. We feed the teacher's fixed object queries to the student's decoder as an auxiliary group and apply Fixed Matching to align the outputs of the teacher and student models.

MixMatcher enables us to establish correspondence between the decoder outputs of the teacher and student models. We develop Decoder Distillation for DETR-Based Methods (D3ETR) based on MixMatcher. In addition to predictions, we also consider attention modules (consisting of selfattention and cross-attention) in the decoder layers during distillation. For attention modules, we distill the knowledge contained in the attention maps. The experiments conducted on COCO [Lin et al., 2014a] demonstrate the effectiveness of our proposed D3ETR. It achieves significant improvements in various DETR-based student models (Figure 1). For instance, D3ETR improves Conditional DETR-R50-C5 [Meng et al., 2021] by 8.3 mAP under 12 epochs training setting, with Conditional DETR-R101-C5 [Meng et al., 2021] acting as the teacher.

To summarize, our contributions are in three folds:

• We explore knowledge distillation for DETR-based detectors, and we attempt to address the challenges of distilling knowledge from the transformer decoder.
• We introduce MixMatcher, which combines Adaptive Matching and Fixed Matching to establish the relationship between the DETR-based teacher and student models. Based on MixMatcher, we propose a simple and effective distillation method named D3ETR.
• Extensive experiments reveal that our proposed method, D3ETR, improves the performance of DETR-based detectors significantly.

2 RelatedWork

Knowledge distillation in object detection. Knowledge distillation (KD) is a technique used for model compression and transfer learning. Originally proposed for distilling knowledge from a large teacher model to a compact student model for classification tasks [Yim et al., 2017], KD has since been improved to perform distillation over intermediate features [Romero et al., 2014; Tian et al., 2019], relation representation [Park et al., 2019; Tung and Mori, 2019], attention [Zagoruyko and Komodakis, 2016], and other aspects. Some recent works have successfully applied KD to object detection [Li et al., 2017; Guo et al., 2021; Yang et al., 2022a; Yang et al., 2022b]. ICD [Zheng et al., 2021] proposes an instance-based conditional distillation framework and finds that initializing the student model with the teacher's parameters leads to faster convergence. DeFeat [Guo et al., 2021] decouples foreground and background in the feature maps and distills them separately. FGD [Yang et al., 2022a] uses focal and global distillation to guide the student model, achieving remarkable results. MGD [Yang et al., 2022b] transforms distillation into a feature-generation task that uses the masked student features to generate the full teacher features. These efforts focus on distillation of ordered outputs in CNN-based detectors.

1 In contrast to traditional object detectors based on CNNs, DETR treats object detection as a set prediction problem. We conduct an ablation in Table 3 to verify the importance of constructing the corresponding between the teacher and student decoder outputs.

ViDT [Song et al., 2022] proposes a variation of a transformer-based detector and applies KD to it, performing distillation directly on patch tokens and detection queries between teacher and student. However, in DETRs, the decoder outputs are unordered, creating a lack of direct correspondence between teacher and student queries. IncrementalDETR [Dong et al., 2022] and DETRDistill [Jiahao et al., 2022] propose to construct correspondence between teacher and student predictions through bipartite matching but ignore that bipartite matching between teacher and student may be unstable in the early training stage [Li et al., 2022]. In this paper, we propose MixMatcher, which helps alleviate these issues.

DETR-based object detection. The pioneering work, DETR [Carion et al., 2020], introduces transformers [Vaswani et al., 2017] to object detection, eliminating the need for hand-designed components like non-maximum suppression or initial anchor boxes generation. Since then, several follow-up works [Zhu et al., 2020; Meng et al., 2021; Liu et al., 2022; Li et al., 2022; Zhang et al., 2022b; Zhang et al., 2022a] have built various advanced extensions based on DETR. Deformable-DETR [Zhu et al., 2020] introduced the multi-scale deformable attention scheme, which attends to a small set of points around a reference and achieves better performance than DETR. Other works such as Conditional DETR [Meng et al., 2021] rebuild positional queries based on reference points for extreme region discrimination. DAB-DETR [Liu et al., 2022] extends the query to a 4D anchor box for improved performance. Follow-up work DN-DETR [Li et al., 2022] and DINO-DETR [Zhang et al., 2022b] introduced a novel query denoising algorithm that accelerates decoder training. Group DETR [Chen et al., 2022a; Chen et al., 2022b] and H-DETR [Jia et al., 2022] claim that multiple positive queries are key to fast convergence. These works emphasize that the decoder design plays a pivotal role in DETR. Different from the existing work on designing novel schemes in the decoder, our proposal starts from another orthogonal point of view and transfers the knowledge in the decoder from a large model to a smaller model.

016

017

018

019

020

021

022

023

024

025

026

027

028

029

030

031

032

033

034

035

036

037

038

039

040

041

042

043

044

045

046

047

048

049

050

051

052

053

3 Preliminary

In this section, we first review the architecture of DETR and the attention mechanism. Then we conduct an analysis on the DETR structure to investigate which part has the most significant impact on the performance.

3.1 DETRArchitecture

The DETR architecture is composed of several components: a backbone, such as ResNet [He et al., 2016], a transformer encoder, a transformer decoder, and object class and box position predictors. The backbone is responsible for extracting image features, and the transformer encoder layers model the global context. The transformer decoder takes N object queries, denoted by Q = {q1, . . . , qN}, as input. Each query is responsible for predicting the class and bounding box of either a ground-truth object or a "no object" class. Different forms of queries can be used, such as high-dimensional feature vectors [Carion et al., 2020; Meng et al., 2021], anchor point coordinates [Wang et al., 2022], and box coordinates [Liu et al., 2022]. The object queries are combined into decoder embeddings, which form the queries of the self-attention and cross-attention layers in the decoder. The embeddings are fed into the detection heads to produce N object predictions.

3.2 Analysis on the DETR Structure

DETR-like methods consist of the backbone, the transformer encoder, and the transformer decoder. We conduct experiments to investigate which part has the most significant effect on the detection performance. Figure 2 depicts the results. We find that reducing the number of decoder layers from 6 to 1 leads to a decrease in mAP of 17.8/18.3 for R50/R101 backbones, respectively. Based on this observation, we propose to distill the knowledge in the decoder.

4 Methodology

In this section, we present our proposed method, MixMatcher (Figure 3), which includes two teacher-student matching strategies, namely Adaptative Matching and Fixed Matching.

1

Additionally, we introduce D3ETR, which distills knowledge from the teacher model in decoder predictions, self-attention, and cross-attention.

4.1 MixMatcher

Adaptative Matching. As the output of the DETR decoder is sparse and unordered, direct one-to-one correspondence between the teacher's and student's outputs is not achievable. To address this problem, we propose to view the correspondence between the teacher's and student's outputs as a bipartite matching problem, inspired by DETR's strategy of performing bipartite matching between predicted and groundtruth objects.

Given the prediction yt = (pt , bt ) and ys = (ps , bs ) of the teacher and the student, where p represents the soft logits for category prediction and b represents the 4-D vector for box prediction. The pairwise matching cost is:

Here Ns is the number of predictions made by students. ξ(·) denotes a permutation of Nt teacher predictions. ℓbce is the binary cross-entropy loss and µcls = 20 is the tradeoff coefficient. ℓbox is a combination of ℓ1 loss and GIoU loss [Rezatofighi et al., 2019], with loss weights of 10 and 2, respectively.

To determine a bipartite matching between teacher and student outputs, we perform a search for the permutation of Nt elements ˆ ξ ∈ ΦNt with the lowest cost:

To facilitate the training process, DETR adopts the auxiliary decoding losses such that each decoder layer would make detection predictions, refining the previous stage's predictions. Therefore, we adaptatively match the outputs of teacher and student models at each decoder layer. If there are L decoder layers, we can apply the adaptative matching algorithm to each decoder layer and obtain L matching results: {ˆ ξ1, . . . ˆ ξL}.

Fixed Matching. The instability of bipartite graph matching may cause inconsistent optimization goals in early training stages [Li et al., 2022]. To alleviate this problem in teacher-student adaptative matching, we introduce an auxiliary group for the student model. Within this group, we feed the fixed teacher queries into the student decoder 2 . By giving both models the same input queries, we hope to achieve wellaligned outputs between the auxiliary group and the teacher model.

Unfortunately, instability within the bipartite graph matching process can also arise between the decoder prediction and the ground truth. This may result in a situation where two outputs, generated from the same object query, are supervised by different ground truths. To resolve this issue, we use the label assignment results of the teacher model to replace those of the auxiliary group in the final decoder layer, such that:

2 We assume teacher and student share the same query format.

where ˆσt is the permutation of Nt predictions of the teacher and ˆσs is the permutation of Nt predictions of the student in the auxiliary group. Under these constraints, the teacher model and the auxiliary group in the student model are supervised under the same ground truth (or "no object" [Carion et al., 2020]), enhancing the one-to-one correspondence.

We employ an ingenious approach to integrate the two matching strategies. During training, we provide the student decoder with both the student group and the auxiliary group. These groups share the same decoder parameters, while not interacting with each other in the decoder self-attention. During inference, we exclusively use the student group, while dropping the auxiliary group. The model inference process is the same as the student trained normally.

4.2 D3ETR

Upon establishing query correspondence between the teacher and student models, the teacher's knowledge can be well distilled into the student model. With the decoder's structure in mind, we define three distillation objectives that comprise prediction distillation, self-attention distillation, and crossattention distillation 3 .

Self-attention distillation. Decoder self-attention models the relations between object queries and potentially aids in eliminating duplicate predictions [Meng et al., 2021]. Given a set of N object queries as input, we can obtain the multihead self-attention weight map of the k-th decoder layer, Aks ∈ RM×N×N . Similarly, we obtain the multi-head selfattention weight map of the teacher model, denoted by ˜Aks . Note that, even though the number of teacher queries may be larger than that of the student queries, queries can be selected based on the teacher-student correspondence. Further, we define the decoder self-attention distillation loss as follows:

3 Please note that the following distillations are carried out between matched outputs of teacher and student models.

where L denotes the number of decoder layers, λsa is the loss weight and set to 10, 000 as default.

Cross-attention distillation. The decoder's cross-attention mechanism utilizes the encoder's output as the keys and values, and the output of the self-attention layer is used as the queries. It identifies the relevant regions of the object in the encoder output and aggregates their features. By employing a set of N queries and an encoder output X ∈ RC×HW , the student and teacher models produce the multi-head crossattention weight maps, namely Akc ∈ RM×N×HW and ˜Akc ∈ RM×N×HW , respectively. Then the decoder cross-attention distillation loss function is formulated as:

where the loss weight λca defaults to 10, 000.

Prediction distillation. After establishing the teacherstudent correspondence, we align the student's prediction to that of the teacher. The prediction distillation loss of the k-th layer is defined similarly to Eq. 1:

where ysk i (ytk ξ(i) ) represents the i-th prediction made by the student (teacher) in the k-th decoder layer.

Overall distillation loss function. The aforementioned distillation loss functions are applied to the student group and the auxiliary group. The overall loss function is expressed as below:

where Laux denotes the loss of the auxiliary group.

4.3 Discussion

DETRDistill [Jiahao et al., 2022] is the most related work to ours. They also explore KD in DETR-like frameworks. Our work diverges from them in a couple of ways. (i) We concentrate on distilling the knowledge contained in decoder attention and prediction, while their focus is on the distillation of query feature knowledge. (ii) Although they also consider how to establish teacher-student correspondence, they ignore the instability issue in teacher-student matching. We propose Fixed Matching to address the issue and experimental results demonstrate the significance of considering this problem.

5 Experiments

5.1 Setting

Dataset. We perform the experiments on the COCO 2017 [Lin et al., 2014b] detection dataset, which contains about 118K training (train) images and 5K validation (val) images.

Training. We follow the training setting of DETR [Carion et al., 2020] and Conditional DETR [Meng et al., 2021] that use ImageNet pre-trained backbone from TORCHVISION with Batch Normalisation (BN) layers fixed. The transformer parameters are initialized using the Xavier initialization scheme [Glorot and Bengio, 2010]. We train our models for 12/50 epochs utilizing the AdamW [Loshchilov and Hutter, 2017] optimizer. The learning rate is reduced by a factor of 10 after 11/40 epochs, respectively. The data augmentation scheme is identical to DETR [Carion et al., 2020]: the input image is resized such that the short side is at least 480 pixels and at most 800 pixels and the long side is at most 1333 pixels. The training image is then randomly cropped with a probability of 0.5 to a random rectangular patch.

Teacher models. For DETR [Carion et al., 2020], we utilize the officially released models, trained for 500 epochs with ResNet-101-C5 or ResNet-101-DC5 backbone as the teacher model. For Conditional DETR [Meng et al., 2021], we utilize the official code to train the model for 50 epochs, utilizing ResNet-101-C5 or ResNet-101-DC5 backbone.

Student models. Student models are trained with the AdamW [Loshchilov and Hutter, 2017] optimizer for 12/50 epochs. The student models are trained based on four different backbones: ResNet-50-C5, ResNet-50-DC5, ResNet-18C5, ResNet-18-DC5.

Evaluation. We use the standard COCO evaluation. We report the average precision (AP), and the AP scores at 0.50, 0.75 and for the small, medium, and large objects.

5.2 Main Results

Our method is versatile and can be applied to various DETRlike frameworks. We perform experiments on two prevalent detectors (namely, DETR and Conditional DETR) with 12epoch (1×) and 50-epoch training schedules. Due to the space limit, we put the results of 50-epoch in the appendix. [Zheng et al., 2021] proposes an inheriting strategy to initialize the student model with the teacher's neck and head parameters, leading to improved performance. We employ this strategy to initialize the transformer encoder and decoder of the student model with the teacher's parameters 4 .

Results with a standard 1× schedule. The results are presented in Table 1. All the student detectors obtain significant mAP improvements with the knowledge transferred from teacher detectors. For instance, D3ETR boosts detection performance when applied to Conditional DETR: +8.3 mAP for R50-C5, +9.8 mAP for R18-C5, +6.3 mAP for R50-DC5, and +7.6 mAP for R18-DC5. Rresults of more detectors and training schedules could be found in appendix.

5.3 Ablation Study

In this section, we first compare the proposed decoder distillation method to other CNN-based distillation methods in object detection. Subsequently, we conduct ablation studies to verify each component in our decoder distillation strategies. We adopt Conditional DETR-R101-C5 as the teacher and Conditional DETR-R50-C5 as the student. The student model is trained for 12 epochs without utilizing the inheriting strategy.

4 The inheriting strategy resulted in a performance gain of 2.1/0.4 mAP for Conditional DETR-R50-C5 under 12/50 epochs setting.

The effectiveness of decoder distillation. We compare D3ETR with other state-of-the-art KD approaches for object detection, DeFeat, FitNet, FGD, and MGD, as shown in Table 2. These works focus on distilling ordered outputs, and we employ them to the output feature of the transformer encoder. We utilize identical teacher and student models, as well as training settings in each instance. For competing distillation approaches, we tune the hyper-parameters based on those in the corresponding papers or open-sourced code repositories, and take the best performance. As can be seen from the table, the proposed approach outperforms its counterparts. This highlights that it is more effective to perform distillation on the DETR decoder layers. We also compared the proposed method with DETRDistill, which performs distillation on both the encoder and decoder. The superior result validates the effectiveness and importance of distilling the attention map of the decoder and addressing the instability issue in teacher-student matching. Besides, we integrate the proposed method with MGD or DETRDistill, resulting in enhanced performance. This exemplifies the potential of further improving the performance of our proposed approach. However, it is beyond the scope of this paper and is left for future work.

The effect of each component in our method. We gradually incorporate the proposed strategies into the baseline, and the results are presented in Table 3. If we do not use any teacherstudent matching strategy, the result is 33.9. This shows that it is important to establish a reasonable teacher-student query correspondence. By adopting the Adaptive Matching strategy with distillation in prediction, self-attention, and crossattention, the results improve significantly, and the combination of these three approaches provides the best result (37.2). Subsequently, adding Fixed Matching improves the result to 38.6. Finally, the inheritance strategy boosts the result by 2.1, obtaining 40.7 mAP, which is comparable to Conditional DETR-R50-C5 trained with 50 epochs (40.9).

Teacher-student matching strategy. The proposed MixMatcher comprises two teacher-student matching strategies, namely, Adaptive Matching and Fixed Matching. To validate our design, we perform ablations on the matching strategy and report the results in Table 4. Our findings indicate that either using Adaptive Matching or Fixed Matching can improve the baseline. Adaptative Matching is slightly more efficient and achieves 37.2 mAP. We further find that we can improve the performance when we use two groups of queries but adopt the same matching strategy during training. For instance, the result obtained by two groups with Adaptive Matching is 0.5 higher than that obtained using a single group. We postulate that multiple query groups enable each ground truth to match more positive queries, thereby easing training [Chen et al., 2022a]. Moreover, using these two strategies simultaneously generates the best result of 38.6 mAP. This illustrates that the hybrid design is helpful.

Constraint in Fixed Matching. We introduce a constraint in Fixed Matching to strengthen the teacher-student fixed correspondence as illustrated in Eq. 3. Without this constraint, the output of the auxiliary group may be supervised by different ground truths from the corresponding output of the teacher model. Since the two outputs are generated from the same object query, the model may become confused. According to Table 5, the performance drops from 38.6 to 37.3 without constraints. This also indicates that directly using the teacher's query as an auxiliary group only leads to marginal improvements (+0.1). We additionally test adding constraints on all decoder layers and find the performance to be slightly worse than adding a constraint on the last layer.

5.4 Visualization

To validate whether the student model learns meaningful information from the teacher model, we visualize the spatial attention map [Meng et al., 2021]. Figure 4 shows the results. According to [Meng et al., 2021], the spatial attention maps correspond to the extremities of objects or a small region within the object box. The former helps in locating the object while the latter aids in recognizing its category. Our observation indicates that the student model struggles with identifying the object extremities precisely. However, by employing our proposed distillation framework, D3ETR, the knowledge in the teacher model is well transferred to the student model. The student model learns similar patterns to the teacher model, thus improving the detection performance.

6 Conclusion

In this work, we explore the effectiveness of knowledge distillation for detectors based on DETR architecture. Our approach, MixMatcher, facilitates learning the correspondence between DETR-based teacher and student models, enabling an efficient knowledge transfer. Based on MixMatcher, we propose D3ETR, a straightforward yet effective distillation framework, and demonstrate its effectiveness through extensive experiments.

Acknowledgments

This work is supported by the Sichuan Science and Technology Program (2023YFSY0008), China Tower-Peking University Joint Laboratory of Intelligent Society and Space Governance, National Natural Science Foundation of China (61632003, 61375022, 61403005), Grant SCITLAB-20017 of Intelligent Terminal Key Laboratory of SiChuan Province, Beijing Advanced Innovation Center for Intelligent Robots and Systems (2018IRS11), and PEKSenseTime Joint Laboratory of Machine Vision.

Contribution Statement

Xiaokang Chen and Jiahui Chen make core contributions. Xiaokang formalized the idea and led the project.