AQT: Adversarial Query Transformers for Domain Adaptive Object Detection

1 Introduction

Object detection is active in computer vision and artificial intelligence researches because it is essential to a broad range of real-world applications, such as surveillance and self-driving cars. While the latest object detection methods [Ren et al., 2015; Liu et al., 2016; Tian et al., 2019] have shown great success in several challenging benchmarks [Lin et al., 2014], their capabilities largely rely on massive labeled data. More importantly, applying pretrained object detectors to new environments would result in performance degradation due to the distribution mismatch between training data and deployed environments.

Unsupervised domain adaptation (UDA) [Ganin and Lempitsky, 2015; Tzeng et al., 2017] has been developed to mitigate these issues. In UDA, a model is usually trained with data from a source domain and a similar but different target domain, while the data labels are only available in the source domain. Under the covariate shift assumption, domain adaptation is typically carried out by minimizing cross-domain discrepancy so that the model supervised on the source domain can learn knowledge and representations generalizable to the target domain. Following the established practices for classification [Tzeng et al., 2017] and segmentation [Hoffman et al., 2018] tasks, many domain adaptive object detectors [Saito et al., 2019; Chen et al., 2020; Rezaeianaran et al., 2021] adopt adversarial feature alignment. By introducing a domain classifier, a learnable measure of domain shifts can be used to impose minimax objectives and to force domain invariance. The resultant models tend to be free from biases towards the source domain.

∗Work done outside of Sony

†Work done outside of Amazon

Despite its effectiveness on CNN-based detectors, adversarial feature alignment is less studied on transformer-based detectors [Carion et al., 2020; Zhu et al., 2021]. CNNs and transformers are intrinsically different. CNNs capture visual characteristics in features maps via convolutions, while transformers model token-wise relationships via the attention mechanism. The recent studies [Wang et al., 2021a] also show that adversarial feature alignment on the CNN backbone of detection transformers brings only limited improvements for domain adaptation.

To address this issue, we present AQT (adversarial query transformer) that combines adversarial learning and transformers for domain adaptive object detection. Since tokenwise operations are implemented in transformers, it is logical to conduct token-wise feature alignment and to focus on the domain-specific tokens. Based on this observation, we develop a strategy for the proposed AQT. We illustrate its conceptual workflow in Figure 1. The generator is the original detection transformer (denoted by self-attention layers for simplicity). It extracts a sequence of feature tokens from the input image and further detects the objects from it. The discriminator is a learnable adversarial token and a stack of cross-attention layers. In each cross-attention layer, the adversarial token serves as the query and the intermediate feature tokens from the generator as the key-value pair. To minimize the adversarial loss, the adversarial token tends to attend to the domain-specific feature tokens. On the contrary, the generator is forced to generate domain-invariant features, especially for the attended key-value pairs, to maximize the loss. As constrained by the detection loss Ldet, the generator also has to prevent semantic collapse in the feature tokens. Adversarial feature alignment is thus carried out.

The proposed mechanism is flexible. First, it is a plug-andadapt module and can work with many existing transformerbased detectors [Carion et al., 2020; Zhu et al., 2021]. Second, it can adversarially align features of different levels. Feature tokens in Figure 1 can correspond to patches, feature maps, or detected objects. Namely, our AQT can realize space-, channel-, and instance-level feature alignment.

The main contribution of this work is three-fold. First, we propose a novel approach AQT, which integrates adversarial feature alignment into a detection transformer via an adversarial token to identify the feature tokens hard to align at that moment. Second, the proposed AQT is simple and flexible. It can work with many existing transformer-based detectors and align features of diverse levels in a unified way. Third, the proposed AQT performs favorably against state-of-the-art methods on the benchmarks, including Cityscapes [Cordts et al., 2016] to Foggy Cityscapes [Sakaridis et al., 2018] and Sim10k [Johnson-Roberson et al., 2017] to Cityscapes.

2 RelatedWork

2.1 Object Detection

As a fundamental topic in computer vision, object detection has been actively studied for decades. Recent advances in object detection can be mainly attributed to CNNs, and categorized by whether region-of-interest proposals are extracted (two-stage) or not (one-stage). While two-stage detectors [Ren et al., 2015] are considered to be more accurate, one-stage detectors [Liu et al., 2016] are benefited from their simple structure and can perform faster.

Different from CNN-based detectors, transformer-based detectors [Carion et al., 2020; Zhu et al., 2021] explore tokenwise dependencies for context modeling and eliminate the need for many hand-crafted components, such as anchor generation and non-maximum suppression. DETR [Carion et al., 2020] first introduces transformers to object detection and yields competitive performance. To mitigate the slow convergence and prohibitive memory usage of DETR, Deformable DETR [Zhu et al., 2021] adopts a learnable sparse attention to speed up convergence and to process multi-scale feature maps efficiently. While these methods focus on supervised learning, we aim at generalizing a detection transformer to the unlabeled target domain by utilizing cross-domain data.

2.2 Domain Adaptive Object Detection

To circumvent performance degradation caused by distribution shifts, the research of domain adaptive object detection has drawn attention recently. The seminal work [Chen et al., 2018] investigates adversarial domain adaptation [Ganin and Lempitsky, 2015] for Faster R-CNN [Ren et al., 2015]. Inspired by it, many following works [Saito et al., 2019; Xu et al., 2020a; Hsu et al., 2020; VS et al., 2021; Wang et al., 2021b] employ domain classifiers on different aspects of cross-domain features. Other groups of works utilize selftraining [Kim et al., 2019; Munir et al., 2021], mean teacher framework [Cai et al., 2019; Deng et al., 2021], and imageto-image translation [Chen et al., 2020]. Although these methods are not categorized in adversarial domain adaptation, some of them also adopt adversarial learning as a part of their algorithms.

The aforementioned methods are usually constrained to CNN-based detectors, such as Faster R-CNN [Ren et al., 2015], SSD [Liu et al., 2016], and FCOS [Tian et al., 2019]. Due to the inherent differences between CNNs and transformers, they may be inapplicable to or suboptimal for transformer-based detectors. While detection transformers [Carion et al., 2020; Zhu et al., 2021] have revealed their potentials, less efforts have been made for their adaptation. A recent study [Wang et al., 2021a] empirically finds that existing approaches bring only limited improvements for detection transformers. The authors attribute this finding to the lack of domain invariance in sequential features, and thus propose domain query-based feature alignment. As their domain query belongs to the generator, it can aggregate global features, but not necessarily domain-specific ones. In contrast, our adversarial token works for the discriminator and attends to the hard-to-align feature tokens. Thus, the generator is encouraged to eliminate domain-specific features and produce more domain-invariant ones. In addition, thanks to the flexible adversarial token, our AQT can carry out space-, channel-, and instance-level feature alignment in a unified manner.

3 ProposedMethod

This section describes our proposed approach. We first give a method overview and then elaborate the space-, channel-, and instance-level feature alignment, respectively.

• (b) Space-level feature alignment module

(c) Channel-level feature alignment module

• (d) Instance-level feature alignment module

Figure 2: Overview of our proposed Adversarial Query Transformers (AQT). (a) Our AQT framework with (b) space-level, (c) channel-level, and (d) instance-level feature alignment.

3.1 Overview

Given a source dataset S = {xis , yis }Ns i=1 and an unlabeled target dataset T = {xit }Nt i=1 where x denotes an image and y represents the ground truth for object detection, we train an adaptive detection transformer F with both S and T , and evaluate it on data in the target domain. To this end, the proposed AQT integrates adversarial feature alignment into detection transformer, and carries out domain invariance in the space, channel, and instance levels.

An overview of our framework is shown in Figure 2a. The framework consists of a generator which is an object detector and three discriminators which perform feature alignment in the space, channel, and instance levels, respectively. The detector includes a CNN backbone GB for feature extraction and a detection transformer with an encoder GE and a decoder GD. Given an input image x, we apply the backbone GB and get its feature maps z ∈ RC×H×W , where C is the number of channels, and H and W denote the map height and width, respectively. The feature maps z are then flattened into patch tokens zp ∈ RC×L, where L = H ×W is the number of tokens. The encoder GE aggregates features for each token in zp via the self-attention mechanism. The decoder GD carries out object detection by taking the object queries as input. A detection loss Ldet is applied to derive the network.

For adversarial feature alignment, each layer of the encoder GE consists of not only a self-attention module and a FFN but also a space-level alignment module (green-shaded region in Figure 2a) and a channel-level alignment module (blue-shaded region in Figure 2a). Similarly, each layer in the decoder GD is associated with an additional instance-level alignment module (yellow-shaded region in Figure 2a). The three modules act as the discriminators, and perform space-, channel-, and instance-level feature alignment, respectively.

Figure 2b illustrates the space-level alignment module, which includes a cross-attention layer. A space-level adversarial query, or space query for short, iterates through the layers in GE. The pixels in the feature maps yield the feature tokens and serve as the key-value pairs. A domain classifier DS is adopted in the space-level alignment module of the last encoder layer, and is learned to predict the domain of the space query. By changing the feature tokens from pixels to channels and detected objects, Figure 2c and Figure 2d show the channel-level and instance-level alignment modules, respectively. We elaborate the three modules as follows.

3.2 Space-level Feature Alignment

As mentioned in [Wang et al., 2021a], direct feature alignment on the CNN backbone results in suboptimal performance on detection transformers. To address this problem, we introduce the discriminator, i.e. the adversarial token and cross-attention layers, for adversarial feature alignment.

The adversarial token, similar to the class token [Dosovitskiy et al., 2021], is trainable and derived to fulfill some objective. For the class token, the objective is to find discriminative features for object recognition, while for adversarial token, the objective is to identify hard-to-align feature tokens for domain classification. In attention mechanisms, a query Q attends to a set of keys K and maps itself into a linear combination of the corresponding values V depending on attention weights. As a result, whichever tokens the query gives a high attention weight on, these tokens are likely to be discriminative for domain classification, i.e. being domain-specific.

In each layer i of the transformer encoder GE, we embed a space-level alignment module after the original self-attention layer. This module consists of a cross-attention layer and a linear mapping layer (simplified from a FFN). As shown in Figure 2b, in the cross-attention layer, the adversarial token acts as a query, while the patch tokens zp, which can be viewed as the output of the generator, act as the keyvalue pairs. Since this process is to align space-level features, we term the adversarial token as the space-level adversarial query, or space query qs. Let qis and zip denote the space query and patch tokens which the alignment module in the i-th encoder layer takes as input, where q1 s is a randomly initialized C-dimensional vector and z1 p = zp. The module in the i-th encoder layer maps the space query qis to its successor qi+1 s w.r.t. zip by

where the multi-head attention function MultiHeadAttn defined in [Zhu et al., 2021] takes two inputs, query and key. Note that we omit positional embeddings of the keys, normalization layers, and residual connections for simplicity. In practice, the key-value pairs pass by a gradient reversal layer [Ganin and Lempitsky, 2015] first to reverse the gradients backpropagated to the generator. This process continues until the end of GE, where i is the number of encoder layers N. The space-level domain classifier DS then identifies the domain of the image given the output of the final-layer space query qN+1 s . This domain classification task is optimized by minimizing the following binary cross-entropy loss

where d denotes the domain label. It takes value 0 for source images, and 1 otherwise. Derived to minimize this adversarial loss Lsp adv , the space query qs and the space-level alignment module manage to identify domain-specific local patches. The generator, i.e. the original layers in detection transformer, with an aim to maximize Lsp adv , is forced to generate more domain-invariant features. As adversarial learning progresses, space-level cross-domain features are adapted.

3.3 Channel-level Feature Alignment

Although space-level feature alignment effectively suppresses domain-specific local patches, it alone does not eliminate the global biases in the encoder GE. The main reason is that attention maps are relatively sparse. Only the keys of the most discriminative patches are assigned high attention weights. To solve this problem, we also adopt channel-level feature alignment in GE.

We define channel tokens zc ∈ R ˆL×C as a different view of patch tokens zp. Each channel token comes from a feature map. ˆL is a hyperparameter, and ˆL = C ≪ L empirically. The reason of resizing is to handle images of different sizes and reduce parametrization overloads. In our implementation, we reshape zp back into z ∈ RC×H×W , and pool z to yield ˆ z ∈ RC×P×P , where P 2 = ˆL. Thus, we obtain zc by flattening and transposing ˆ z.

To enable channel-level alignment, we leverage a channellevel adversarial query, or channel query qc. Similar to the space-level alignment module, we embed a channel-level alignment module in each layer of GE. Let qic and zic denote the channel query and channel tokens which the i-th alignment module takes as input, where q1 s is a randomly initialized ˆL-dimensional vector and zic comes from zip . The module in the i-th encoder layer maps the channel query qic to its successor qi+1 c w.r.t. zic via

Again, this process continues until the end of GE. The channel-level domain classifier DC then identifies the domain of the image given the final-layer channel query qN+1 c . The channel-level adversarial loss thus yields

3.4 Instance-level Feature Alignment

Although we make the encoder GE domain-adaptive using both space-level and channel-level alignment, the decoder GD remains biased towards the source domain. Thus, we remedy this issue and show the flexibility of our method by extending it to instance-level feature alignment.

In a detection transformer, object queries zo ∈ RC×Lo , are Lo learned embeddings that decode object representations from the encoder output. To align object representations in zo, we introduce the instance-level adversarial query, or instance query qi. Similar to the alignment modules built in GE, we embed an instance-level module in each layer of GD. Let qj i and zjo denote the instance query and object queries which the j-th alignment module takes as input, where q1 i is a randomly initialized C-dimensional vector and z1 o = zo. The module in the j-th decoder layer maps the instance query qj i to its successor qj+1 i w.r.t. zjo by

This process continues until the end of GD, where j is the number of decoder layers M. The instance-level adversarial loss is thus yielded

Overall Objective. We summarize the overall objective of the proposed Adversarial Query Transformer. The overall adversarial loss function is formulated as

where λsp, λch, and λins denote the balancing weights for the corresponding terms in the loss function. Ldet denotes the detection loss of the adopted detection transformer. The overall loss function of the proposed AQT is

and the optimization objective for our domain adaptive detection transformer F is

4 Experimental Results

4.1 Datasets and Experimental Settings

Cityscapes to Foggy Cityscapes. Cityscapes [Cordts et al., 2016] is an urban scene dataset containing 2,975 training images and 500 validation images. Foggy Cityscapes [Sakaridis et al., 2018] is synthesized from and shared annotations with Cityscapes. We take the highest fog density images following [Rezaeianaran et al., 2021]. In this setting, Cityscapes is used as the source domain, while Foggy Cityscapes is used as the target domain. 8 categories are considered.

Cityscapes to BDD100k daytime. BDD100k [Yu et al., 2020] is a large-scale driving dataset with diverse scenarios. In this setting, Cityscapes is used as the source domain, while the daytime subset of BDD100k is selected as the target domain. Following [Xu et al., 2020a], the common 7 categories are considered.

Sim10k to Cityscapes. Sim10k [Johnson-Roberson et al., 2017] is a synthetic driving dataset containing 10,000 images. In this setting, Sim10k is used as the source domain, while Cityscapes is used as the target domain. Following [Chen et al., 2018], only the category car is considered.

4.2 Implementation Details

We select Deformable DETR [Zhu et al., 2021] as our object detector with a ResNet-50 backbone pre-trained on ImageNet [Deng et al., 2009]. We inherit most hyperparameters and training settings from Zhu et al., including the detection loss Ldet and Xavier initialization [Glorot and Bengio, 2010]. In Cityscapes to Foggy Cityscapes, all λsp, λch, and λins are set to 10-1 . In the other settings, following [Saito et al., 2019], we adopt local alignment on the backbone and weak alignment using the focal loss [Lin et al., 2017]. The λsp, λch and λins are set to 10-1 , 10-5 , and 10-4 , respectively. The batch size is set to 8 in all experiments.

4.3 Comparing with State-of-the-arts

From Table 1 to Table 3, we compare the proposed AQT with the existing methods based on Faster R-CNN [Ren et al., 2015], FCOS [Tian et al., 2019], or Deformable DETR [Zhu et al., 2021] on three adaptation settings. "Source Only" indicates the baselines trained with source data only; "V-16" and "R-50" indicate the backbone is VGG-16 [Simonyan and Zisserman, 2015] and ResNet-50 [He et al., 2016]. † indicates iterative bounding box refinement [Zhu et al., 2021].

Cityscapes to Foggy Cityscapes. After adaptation, our method improves the baseline by 18.6% and outperforms all

• (a) An input image and its corresponding attention map.
• (b) Feature distribution visualizations using t-SNE.

Table 3: Results of Sim10k to Cityscapes.

the other methods. We observe that our method performs worse on the categories "truck", "bus", and "train". We hypothesize this results from fewer instances in these categories, given that transformers rely on sufficient data more than CNNs.

Cityscapes to BDD100k daytime. Our method outperforms all the other methods in terms of mAP. All the methods reported in this experiment is developed upon ResNet50. Again, our method performs worse on the categories with fewer instances.

Sim10k to Cityscapes. Due to the larger domain gap, our method outperforms the state-of-the-arts by a relatively small margin. However, our method still outperforms SFA [Wang et al., 2021a].

Qualitative Result. We evaluate the detection quality of our method by comparing it with three existing methods, EPM [Hsu et al., 2020], ViSGA [Rezaeianaran et al., 2021], and SFA [Wang et al., 2021a]. The results in Figure 4 show the superior performance of our method.

4.4 Ablation Study and Analysis

In this section, we provide detailed ablation study and analysis of AQT. Following [VS et al., 2021], all these studies are conducted on Cityscapes to Foggy Cityscapes.

Where the Adversarial Token Looks at. To better understand how the adversarial token works, we provide an attention map of a space query on an input image, as shown in Figure 3a. The regions with strong responses in the attention map correspond to the foggiest regions in the input image. This demonstrates that the adversarial token tends to attend to the domain-specific features.

Visualizing Feature Distribution Alignment. We use tSNE [van der Maaten and Hinton, 2008] to visualize the activations of different detection transformers, including the baseline (Source only), SFA [Wang et al., 2021a], and ours. Figure 3b shows the results. Different colors represent the features in different domains. Empirically the aligned features via both AQT and SFA are more indistinguishable than the baseline. Interestingly, we notice the aligned feature distribution via AQT is more perceptually similar to the baseline.

Quantitative Ablation Study. We analyze the effect of the proposed three levels of alignment in Table 4, where Box-Refine and Two-Stage are inherited from Deformable DETR [Zhu et al., 2021]. Based on this experiment, we conclude three major observations. First, each of the three levels of alignment leads to reasonable improvement. This demonstrates the effectiveness and flexibility of the proposed mechanism. Second, when a certain level of alignment is augmented with another one, a steady growth occurs. We can observe a rather considerable improvement when the three alignments are adopted, compared to the settings where only any two are adopted. Lastly, our framework can also benefit from other techniques. When using Box-Refine, our method achieves the best result. Thus, we take it as our default setting when compared with the state-of-the-arts.

Qualitative Ablation Study. In Figure 5, we show several detection results from the baseline (Source only), the adapted models, and the ground truth bounding boxes. In the first row, the baseline barely discerns any object, while the adapted model with channel-level alignment ("Ch.") recognizes some objects. With the assistance of instance-level alignment, the adapted model ("Ch.+Ins.") can further localize the truck at the center of the image. It shows that channel-level alignment effectively eliminates cross-domain discrepancy, while the categories with few instances are still hard to detect. This drawback is compensated when adopting instance-level alignment due to reducing biases in the object queries. In the second row, the models in the left three columns fail to recognize the truck at the left of the image, while this false negative is resolved with space-level alignment ("Ch.+Ins.+Sp."). This study empirically shows different levels of alignment can be complementary.

5 Conclusion

In this paper, we present AQT (adversarial query transformers), an adaptation framework to integrate adversarial feature alignment into a detection transformer. It employs a novel adversarial token and a stack of cross-attention layers as the discriminator. As the query in each cross-attention layer, the adversarial token attends the feature tokens from the generator that are hard to align at that moment. Constrained by both the adversarial loss and the detection loss, the generator is forced to eliminate the domain-specific features while maintaining semantics in the feature tokens, hence realizing adversarial feature alignment on detection transformers. The proposed AQT demonstrates the flexibility of the proposed mechanism, combines the merits from the space-, channel-, and instance-level alignment, and yields a new state-of-theart on several domain adaptive object detection benchmarks.

Acknowledgements

This work was supported in part by the Ministry of Science and Technology (MOST) under grants 109-2221-E-009-113MY3, 110-2628-E-A49-008, 111-2634-F-007-002, and 1102634-F-006-022. This work was funded in part by Qualcomm and MediaTek. We thank the National Center for High-performance Computing (NCHC) of National Applied Research Laboratories (NARLabs) in Taiwan for providing computational and storage resources.