Conflict-Alleviated Gradient Descent for Adaptive Object Detection

1 Introduction

The rapid expansion of image data has driven significant advancements in machine learning, particularly in the field of computer vision [Redmon and Farhadi, 2018; Ren et al., 2015; Cai and Vasconcelos, 2018; Zhu et al., 2020; Carion et al., 2020; Liu et al., 2023]. However, a major limitation of these advancements is the assumption that the distribution of the training dataset perfectly matches that of practical application scenarios, an alignment rarely achieved in real-world settings. To tackle the challenges arising from this mismatch in domain distributions, Unsupervised Domain Adaptation (UDA) [Pan and Yang, 2009; Cui et al., 2020; Ganin and Lempitsky, 2015; Kang et al., 2019] has emerged as a practical solution. UDA aims to reduce the performance decline caused by the disparity in domain distributions, removing the need for annotated data in the target domain.

∗Corresponding author

In recent years, UDA algorithms have rapidly developed and achieved significant performance in object detection, known as domain adaptive object detection (DAOD) [Chen et al., 2018; He and Zhang, 2019; Xu et al., 2020; Wang et al., 2021a; Wu et al., 2022]. Specifically, [Chen et al., 2018] is the first method to achieve domain alignment through the game between feature detector and domain classifier. [Wang et al., 2021a] adjusts adversarial domain adaptation models on detection transformers to reduce distribution shift between different domains through aligning sequence features. These approaches are characteristically designed to concurrently optimize both domain alignment and detection tasks throughout the training phase. However, due to the functional differences between various tasks, the optimal gradient descent directions for these tasks may be uncoordinated or imbalanced. Directly sharing network parameters, such as the feature generator, can lead to optimization conflicts and negatively impact the learning of domain-invariant features. Figure 1 (a) illustrates a scenario of suboptimal direction updates. The orange arrow represents the optimal gradient direction for domain alignment, and the blue arrow for detection. In joint optimization, the update gradient (red arrow) is mostly dominated by detection task, owing to existing conflicts, which diminishes the effectiveness of both tasks. In this work, we define two gradients to be conflicting if they point away from one another, i.e., have a negative cosine similarity.

To validate our observations, we conducted experiments in cross-domain object detection scenarios. Figure 2 shows the gradient inner product distribution between two tasks (alignment and detection) during training, using DAF [Chen et al., 2018] and SFA [Wang et al., 2021a] on tasks transitioning from Cityscapes [Cordts et al., 2016a] to Foggy Cityscapes [Sakaridis et al., 2018a]. The results revealed a mix of acute and obtuse angles in gradient distributions, with obtuse angles being common. Conflicting gradients cause the optimizer to favor certain tasks, leading to a poor update direction for a subset or even all tasks. If left untouched, this will degrade adaptation performance. Addressing these conflicts while preserving task functionality is a crucial yet underexplored challenge.

In this paper, we introduce a novel and effective module called Conflict-Alleviated Gradient descent (CAGrad), designed to address the optimization conflicts between domain alignment and detection tasks. When gradients from these two tasks conflict, we modify them by projecting each onto the normal plane of the other, ensuring coordinated development of both tasks during joint training while preserving their specific functions. As depicted in Figure 1 (b), the strategic integration of gradient magnitude and directionhomogenization plays a pivotal role in maintaining the stability of the overall learning process. Moreover, CAGrad is model-agnostic and requires only a single modification in gradient application, making it easy to integrate with existing state-of-the-art DAOD methods for enhanced performance. We theoretically prove the local conditions under which CAGrad improves upon standard multi-task gradient descent and empirically evaluate it on various challenging DAOD tasks, including cross-camera, weather, scene, and synthetic-to-real-world adaptation. Overall, CAGrad proves to be a powerful tool, capable of not only alleviating gradient conflicts but also enhancing detection performance. The main contributions of this paper are summarized as follows:

• We discover the gradient conflict exists in the DAOD methods,which can cause the optimizer to prioritize certain tasks over others, leading to difficulties in converging to a desirable solution.
• We propose an efficient optimization strategy named Conflict-Alleviated Gradient descent (CAGrad), which alters the gradients by projecting each onto the normal plane of the other. Figure 3 is a representative and illustrative example that depicts the usage of CAGrad.
• Extensive experiments validate the effectiveness and universality of CAGrad on various benchmark databases on several DAOD baselines. More insights and analyses of our model are also provided to justify the reasonability of CAGrad and demonstrate its superiority.

CAGrad is a simple and effective tool that easily integrates with existing adversarial learning methods for Domain Adaptive Object Detection (DAOD), without requiring changes to the network architecture. This attribute renders CAGrad highly compatible and user-friendly with various adversarial domain adaptation approaches.

2 RelatedWork

Object detection. Object detection, a key area in computer vision, has seen significant progress over the years. Models like Faster-RCNN [Ren et al., 2015] and Mask-RCNN [He et al., 2017] represent successful two-stage approaches, while one-stage models such as YOLO [Redmon et al., 2016], SSD [Liu et al., 2016], and FCOS [Tian et al., 2019] are known for their real-time capabilities. Unlike these CNN-based detectors, DETR [Carion et al., 2020] introduces transformers to object detection and shows promising results. Deformable DETR [Zhu et al., 2020] further improves this by speeding up convergence and efficiently handling multi-scale features. These developments provide a strong foundation for domain adaptive object detection research.

Domain Adaptive Object Detection. To circumvent performance degradation caused by distribution shifts, the research of domain adaptive object detection has drawn attention recently. As a pioneering work, [Chen et al., 2018] propose the domain adaptive Faster-RCNN method (DAF),which achieves image-level and instance-level feature alignment. Inspired by it, MAF [He and Zhang, 2019] introduces a hierarchical adversarial feature alignment strategy that reduces domain disparity at different scales. HTCN [Chen et al., 2020] employ CycleGAN for data augmentation, generating intermediate domain images to facilitate model alignment between source and target domains. VDD [Wu et al., 2021] tackles the problem by disentangling domain-invariant and domain-specific representations using vector decomposition, while also exploring the extraction of instance-invariant features [Wu et al., 2022]. IDF [Lang et al., 2023] propose a non-adversarial domain discriminator to extract domainspecific features. Additionally, PTMAF[He et al., 2023] and PAATF[He et al., 2022] introduce additional constraints during the adversarial learning stage. Recently, regarding the Transformer object detector, existing adaptation techniques for DETR predominantly rely on model-based approaches SFA [Wang et al., 2021b], aiming to reduce the distribution shift between different domains through sequence feature alignment. AQT [Huang et al., 2022] employs a novel adversarial token and a stack of cross-attention layers as the discriminator.

Previous methodologies have focused on acquiring domain-invariant and class-discriminative feature representations through the concurrent optimization of domain alignment and object detection tasks. However, these approaches often overlook the issue of imbalance and uncoordinated optimization between tasks, a phenomenon we identify as 'gradient conflict' in this study. To address this issue, we introduce the Conflict-Alleviated Gradient descent (CAGrad), a strategy designed to independently manage the gradients of domain alignment and detection tasks. This approach aims to harmonize the two tasks throughout the training process, thereby mitigating the gradient conflict.

3 Proposed Approach

3.1 ProblemDefinition

In domain adaptive object detection, the labeled source domain Ds = {xsi , ys i }ns i=1 comprises ns samples across C classes. In contrast, the unlabeled target domain Dt = {xti }nt i=1 contains nt samples of the same classes. Both Ds and Dt share the same feature and category spaces but differ in data distributions. Our goal is to use the labeled data from Ds and the unlabeled data fromDt to train a deep model that accurately predicts class labels and identifies bounding boxes in the target domain.

3.2 AGeneral Framework of DAOD

Adversarial learning, pioneered by the Domain Adversarial Neural Network (DANN) [Ganin et al., 2016], aligns domains effectively. It uses a feature generator G to deceive the domain discriminator D, which predicts the origin domain of the generated features. Training involves an adversarial game between G and D, optimizing parameters θg and θd using a domain alignment objective function as follows:

To enhance detection performance in the target domain, it is crucial to first ensure that the detector C accurately identifies samples from the source domain. Consequently, we describe the supervised detection loss as follows:

where Lce represents the standard cross-entropy loss function. During the training phase, conventional methods typically optimize both the adversarial (Ladv) and detection (Ldet) objective functions concurrently to achieve domaininvariant and class-discriminant feature representations. The overall minimax objective function is given by:

where θg, θd, and θc represent the parameters of the feature generator, domain discriminator, and category detector, respectively.

3.3 Conflict-Alleviated Gradient Descent

In the realm of DAOD, domain alignment and detection stand as distinct tasks, each possessing its own set of optimal gradient descent directions, which may exhibit inconsistency. This inconsistency can engender conflicts in the optimization of two loss functions during training, thereby exerting an impact on the ultimate performance of domain adaptation. To enable seamless coordination in optimizing these two tasks, we introduce the CAGrad technique.

To streamline our approach, we define the overall loss function as L = L1 + L2, where L1 = Ladv pertains to domain alignment and L2 = Ldet relates to detection. Here, g1 = ∇L1(θ) and g2 = ∇L2(θ) signify the gradients for each task, while g denotes the gradient of the total loss L. Our analysis delineates two scenarios based on the correlation between g1 and g2:

In Scenario I, a positive correlation between g1 and g2 (i.e., cos(g1, g2) > 0 or gT 1 g2 > 0) suggests an acute angle between the gradients, indicating that they are effectively aligned. Consequently, no further modification of g1 and g2 is required.

In Scenario II, a negative correlation between g1 and g2 (i.e., cos(g1, g2) < 0 or gT 1 g2 < 0) indicates an obtuse angle between the gradients, suggesting a conflict in their optimization directions. To address this effectively, projecting each gradient onto the normal plane of the other's gradient can minimize this conflict and enhance the optimization efficiency. This is achieved by:

The Conflict-Alleviated Gradient Descent process is illustrated in Figure 3, leading to a combined gradient expression:

Considering both positive and negative correlations (gT 1 g2 > 0 and gT 1 g2 < 0), we derive the following theorem.

Theorem 1.In the context of DAOD, conflict resolution in optimization can be facilitated by directly altering the gradients to prevent potential conflicts. The combined gradient expression, g, post-modification, is given by:

g = ˜ g1 + ˜ g2

Here, δ(·) denotes an indicator function, which returns a value of 1 if the condition is true and 0 if false, defined as:

The proposed CAGrad methodology effectively coordinates two distinct tasks during optimization.

3.4 Efficient Equivalent Model of CAGrad+DAOD

Using the generalized DAOD model from Eq. 3 in Section 3.2, along with the CAGrad method from Eq. 7, improves training and helps achieve equilibrium in the DAOD framework. We introduce a more proficient loss function that integrates CAGrad principles into the DAOD model, improving convergence and performance. In alignment with CAGrad principles, we reformulate Eq. 7 as follows:

To streamline the presentation, let us define τ1 and τ2 as:

These constants τ1 and τ2 are derived from the gradients of

Algorithm 1 CAGrad+DAOD Optimization Algorithm

Input: Source data {xsi , ys i }i=ns i=1 , Target data {xti }i=nt i=1 , Optimal parameters ϕ = {θg, θc, θd}, learning rate η, max iteration;

Output: Optimal parameters θg, θc, θd;

• 1: Initialization Optimal parameters θg, θc, θd;

2: repeat

• 3: Compute domain alignment loss Ladv (Eq. 1), i.e., L1 and detection loss Ldet (Eq. 2), i.e., L2;
• 4: Compute original gradients g1 and g2 and judge the inner product of two gradients gT 1 g2;
• 5: Compute coefficients τ1 and τ2 by Eq. 10;
• 6: Compute updated total loss ˜ L (Eq. 12);
• 7: Update model parameters: ϕt+1 ←ϕt - η∇ϕt ˜ L
• 8: until max iteration is reached;

the original loss functions L1(θ) and L2(θ). With τ1 and τ2 established, we can rewrite Eq. 9 as follows:

Performing an integral operation on Eq. 11 yields a balanced DAOD model:

The training objective then becomes:

The CAGrad module, as proposed, essentially recalibrates the original loss functions L1 and L2 through the computation of weights τ1 and τ2, which rely on the gradients of the original loss functions. This methodology culminates in a balanced and efficient DAOD model. In scenarios where gT 1 g2>0, we note τ1 = 1 and τ2 = 1, leading to a simplification of the overall loss function to L = L1 + L2, corresponding to the general DAOD model. Figure 4 serves as an exemplary illustration of the CAGrad+DAOD application, with the complete update process delineated in Algorithm 1.

4 Experiments and Results

4.1 Experimental Setup

Datasets and Settings. Our methodology is rigorously evaluated on a variety of datasets, including KITTI [Geiger et al.,

2012], Cityscapes [Cordts et al., 2016b], Foggy Cityscapes [Sakaridis et al., 2018b], Sim10k [Johnson-Roberson et al., 2016], and BDD100k [Yu et al., 2020], which present a broad spectrum of challenges for domain adaptive object detection:

• Cross Camera Adaptation. We evaluate the domain drift between the real domain and the similar domain caused by different cameras, which is an important factor leading to domain differences in the real-world. We employ the training set of KITTI and the Cityscapes as the source and target domains, respectively.
• Weather Adaptation. In this scenario, we use Cityscapes as the source dataset, consisting of 2,975 training images and 500 evaluation images. The counterpart, known as Foggy Cityscapes, is derived from Cityscapes through a fog synthesis algorithm. These datasets enable us to assess the effectiveness of our method in adapting object detection models from clear weather to foggy conditions.
• Scene Adaptation. Cityscapes is again the source dataset in this instance, with the daytime subset of BDD100k acting as the target. The BDD100k subset, encompassing 36,728 training and 5,258 validation images, is rich in diverse daylight scenes, each annotated with bounding boxes.
• Synthetic to Real Adaptation. In this scenario, the source domain is Sim10k, generated through the Grand Theft Auto game engine, containing 10,000 training images with 58,701 bounding box annotations. Cityscapes is employed as the target domain, specifically focusing on car instances for both training and evaluation.

Implementation Details. We benchmark our approach against cutting-edge domain adaptation methods across two categories: (1) The Faster RCNN series, including DAF [Chen et al., 2018], MAF [He and Zhang, 2019], ATF [He and Zhang, 2020], HTCN [Chen et al., 2020], UMT [Deng et al., 2021], PAATF [He et al., 2022], PTMAF [He et al., 2023], and IDF [Lang et al., 2023]. (2) The Deformable DETR series, encompassing SFA [Wang et al., 2021b].

We validate the effectiveness and versatility of our method against various baselines, including DAF, MAF, IDF, and SFA. By default, we employ ImageNet pre-trained ResNet50 [He et al., 2016] and VGG-16 [Simonyan and Zisserman, 2014] as CNN backbones in all experiments. Aligning with the Faster RCNN series, we train the network using the SGD optimizer with a momentum of 0.9 and a weight decay of 5 × 10-4 . The initial learning rate is set to 1 × 10-3 and is reduced to 1 × 10-4 after 5 epochs. A total of 15 epochs are conducted, with a batch size of 2 maintained throughout. In line with the Deformable DETR series, we utilize the Adam optimizer [Kingma and Ba, 2015] for training over 50 epochs. The learning rate is initialized at 2 × 10-4 and reduced by a factor of 0.1 after 40 epochs. A batch size of 4 is employed consistently in all experiments. All these experiments are conducted using RTX3090 NVIDIA GPUs.

4.2 Comparisons with SOTAMethods

Cross Camera Adaptation. In our investigation into crosscamera adaptation, we focused on the transfer between KITTI

(K) and Cityscapes (C), with detailed results outlined in Table 1. Notably, CAGrad facilitated enhancements in detection accuracy across various models DAF, MAF, IDF, and SFA. Specifically, IDF combined with CAGrad surpassed the baseline IDF by significant margins of 1.4% and 1.3% in the K→C and C→K tasks, respectively. SFA augmented with CAGrad exhibited an increment of 1.4% in K→C and an impressive 8.6% in comparison to the source-only Deformable DETR model. This underscores CAGrad's capacity to seamlessly enhance existing alignment-based DAOD methods, thereby elevating their efficacy in image detection tasks.

Weather Adaptation. In assessing the robustness of our target detector under varied weather conditions, we executed a cross-domain model transition from Cityscapes to Foggy Cityscapes. The outcomes, as tabulated in Table 2, reveal that upon integrating CAGrad, models such as DAF, MAF, IDF, and SFA reported mAP values of 33.6%, 37.0%, 43.7%, and 43.5% respectively. Remarkably, the implementation of CAGrad significantly bolstered the cross-domain efficacy of the Deformable DETR, achieving a noteworthy absolute mAP increment of 2.2% (from 41.3% to 43.5%). This improvement attests to CAGrad's ability to rectify the discord and imbalance between domain alignment and detection tasks in the optimization phase, thereby effectively enhancing detection accuracy.

Scene Adaptation. The adaptability of models to varying scenes is crucial. Our proposed method, CAGrad, demonstrates its effectiveness in scene adaptation, as shown in Table 3. It can be observed that after applying CAGrad, the mAP of DAF, SWAD and SFA reaches 26.5%, 27.5%, and 31.6%, respectively. Notably, performance improvements are observed across all categories within the target dataset.

Synthetic to Real Adaptation. Within the synthetic-to-real adaptation domain, the efficacy of CAGrad was rigorously evaluated, as delineated in Table 4. Impressively, CAGrad surpassed the baseline models DAF, MAF, IDF, and SFA by margins of 2.1%, 2.4%, 1.6%, and 2.0%, respectively.

4.3 Analysis and Discussions

Convergence Analysis. Figure 5 illustrates the test accuracy's convergence trajectories relative to iteration counts in tasks transitioning from Cityscape to Foggy Cityscape. Here, the test accuracy of the baseline models is depicted by the blue curve, while the red curve shows the test error for baseline models enhanced with CAGrad. It is evident that, when compared to the baseline alone, the combination of baseline and CAGrad achieves quicker convergence and a reduced test error. This effectively demonstrates CAGrad's significant role in facilitating the optimization process, steering both domain alignment and detection tasks towards more favorable outcomes.

Detection Results. In Figure 6 (a), we show some visual results by SFA (baseline) and our SFA+CAGrad, accompanied with the ground-truth. As can be seen, in all three scenarios, SFA+CAGrad improves the detection performance. It successfully mitigates the false positives generated by SFA and detects challenging objects overlooked by SFA.

Visualization of Feature Distribution. Figure 6 (b) presents the t-SNE visualizations illustrating the feature distributions generated by IDF (baseline) and IDF+CAGrad in the context of weather adaptation. The features processed using CAGrad demonstrate a more pronounced clustering effect, with a diminished presence of samples straddling class boundaries. This enhances the discriminative capacity of the features. Furthermore, these visualizations corroborate the efficacy of the CAGrad mechanism in learning enhancement.

Effectiveness Analysis. Figure 7 presents the inner product distributions of gradients between domain alignment and detection tasks after applying CAGrad to DAF and SFA during the entire training and updating process. Compared to Figure 2, after applying CAGrad, as shown in Figure 7, the inner products of both gradients are positive. This means that the gradient angles of the two tasks have been coordinated into acute angles. The proposed CAGrad avoids optimization conflicts by separately adjusting the gradients of the two tasks to achieve optimal coordination. Experimental results fully illustrate the effectiveness of the proposed CAGrad.

5 Conclusion

This research addresses the challenge of optimization conflict in unsupervised domain adaptive object detection models, specifically between the alignment and detection tasks. To address this challenge, we introduce an innovative yet straightforward technique, the Conflict-Alleviated Gradient descent (CAGrad), which effectively resolves gradient conflicts for each task. Furthermore, we propose a rapidly optimizable equivalent model, CAGrad+DAOD, implementing swift integration of CAGrad. This approach ensures that both the alignment and detection tasks are harmoniously balanced during DAOD optimization. A comprehensive array of experimental evaluations and model analyses substantiate that CAGrad markedly enhances the performance of existing alignment-based DAOD models, contributing to state-of-theart outcomes. Additionally, CAGrad demonstrates scalability and is underpinned by a solid theoretical framework.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (Grant No.62176217).