Dynamic Domain Generalization

1 Introduction

Deep neural networks have achieved great success on various vision tasks, such as image recognition and object detection [He et al., 2016]. Most of deep models are learnt in a closed perception environment where training and testing data are supposed under i.i.d (independent and identical distribution) assumption. However, the reality is open, compound, uncontrolled and incompatible with i.i.d assumption. Such complex scenarios can make conventional models suffer tremendous performance degradation, due to the absence of generalization ability to handle domain shift, which undoubtedly will discount the reliability and security of intelligent systems, e.g., autonomous driving system.

† Corresponding authors: Luojun Lin and Yuanlong Yu

Extensive studies aim to tackle this problem through domain generalization (DG), whose objective is to obtain a robust static model by learning domain-invariant representations via exploiting multiple source domains [Ganin and Lempitsky, 2015; Zhou et al., 2021]. Nevertheless, a severe bottleneck from the static model constrains its freedom to generalize to agnostic target domains with different distribution. In contrast, domain adaptation (DA) [Ganin et al., 2016] and test-time DG [Iwasawa and Matsuo, 2021] are explored to leverage the unlabeled data sampled from the target domain for further unsupervised fine-tuning so as to enhance the target-oriented generalization ability. However, extra training effort and latency are required during testing, while the manufactured chips and edge devices mostly do not possess training resources. Considering these two perspectives, there is a lack of training-free mechanism to adjust the model when encountering the testing data from novel target domains.

To complement this blank, we propose a new DG variant, Dynamic Domain Generalization (DDG). As shown in Figure 1, different from DA, DG, as well as test-time DG methods, the proposed DDG is attached with a meta-adjuster, which learns to adapt to novel domains during pre-training and twists the model during testing without any fine-tuning. To drive the process of learning to adapt, we need to collect training data from numerous novel domains, while the reality is that we are usually hindered by limited source domains. A simple yet friendly solution is that we simulate novel samples from the limited source domains via a random convex interpolation among different source domains, which is termed as DomainMix in this paper. In this way, each simulated sample is restricted to a unique domain. As a consequence, the domain-aware meta-adjuster can be relaxed into an instanceaware meta-adjuster, which modulates the network parameters per instance. Besides, it makes the model easier adapt from source domains to agnostic target domains with undefinable domain distinction, by discarding the utilization of domain label during optimizing meta-adjuster.

Beyond the optimization pipeline of DDG, it is also a critical factor on how to design the meta-adjuster. In order to disentangle the domain-shared features and domain-specific features, the network parameters are decoupled into a static part and a dynamic part where the latter one is generated by the meta-adjuster per instance. To ease the optimization of meta-adjuster, the dynamic parameters are decomposed into a linear combination of several elaborately-designed kernel templates with low-dimensional dynamic coefficients, in which the coefficients are generated dynamically by the metaadjuster via taking the instance embedding as input. As shown in Figure 3, the kernel templates are designed as four asymmetric kernels for the sake of strengthening the kernel skeletons in spatial as well as channel dimensions, which explicitly regularize the meta-adjuster to generate diverse dynamic parameters to adapt to a large variety of novel samples.

Extensive experiments are conducted on three popular DG benchmarks, including PACS [Li et al., 2017], Office-Home [Venkateswara et al., 2017] and DomainNet [Peng et al., 2019]. These datasets contain various situations varying from small to large scales, from simple to complex scenarios, from small to large domain shifts, which are sufficient to demonstrate the effectiveness of our proposed method. To summarize, DDG opens up a new paradigm to study DG problem.

2 RelatedWork

DomainAdaptation. Conventional DA aims to transfer the knowledge learnt from labeled source domains to target domains which may contain a few labeled samples or none at all. The discrepancy-based methods are early developed, which mitigate the distribution discrepancy between two domains by minimizing the well-defined distance loss [Haeusser et al., 2017]. Another line of DA methods is based on adversarial learning, where a domain discriminator is frequently-used to train with the feature extractor adversarially, in order to enforce the feature extractor to learn domain-invariant features by confusing the domain discriminator [Ganin et al., 2016]. Compared with DA methods that adjust models to adapt to target domains explicitly, our method can be viewed as a kind of training-free DA via pre-training merely in source domains.

Domain Generalization. Early DG-related research focus on learning domain-invariant representation, which is believed to be robust if it can suppress domain shift among multiple source domains [Motiian et al., 2017]. Another line of DG methods is based on data augmentation that enhances the diversity of image styles for optimizing the generalization capability of models [Zhou et al., 2020a; Zhou et al., 2021]. Self-supervised learning is also applied to DG that learns task-independent representation with less probability of over-fitting [Carlucci et al., 2019; Kim et al., 2021]. Metalearning is also involved with solving DG [Balaji et al., 2018], where deep model is trained by simulating the meta-train and meta-test tasks in source domains to enhance its generalization ability. Our method can be regarded as a very simple meta-learning-like paradigm, in which meta-adjuster learns to learn parameter generalization for each individual "domain". Besides, a few emerging works show a new insight for DG that deep model can be adapted to target domain at test time [Iwasawa and Matsuo, 2021]. However, the computational cost is inescapable in such methods. In contrast, our method can directly update parameters to adapt to agnostic target domain without any backward propagation.

3 Methods

3.1 Overview

DG aims to train a general model that can be well generalized to any agnostic target domain. In particular, we only consider K-way classification task in this paper. For a vanilla DG task, we are given ns labeled samples {xis , yis , dis }ns i=1 drawn from multi-source domains XS = {XS1 ,XS2 , ...,XSM }, where xis ∈ XS, yis ∈ YS and domain label dis ∈ [1,M]. The objective of DG is to learn a mapping fΘ : {XS1 , ..., XSM } → YS that can also predict precisely on any agnostic target domain.

Vanilla Domain Generalization. For vanilla DG, the learnt mapping fΘ obtained in training stage will be kept static in inference. It usually causes the performance degradation on agnostic target domains due to the inevitable data distribution distinction between the source and target domains.

DynamicDomainGeneralization. To overcome theweakness of vanilla DG, we propose a new variant of DG, namely Dynamic Domain Generalization (DDG). DDG mainly focuses on learning to twist the network parameters to adapt to the instances from diverse novel domains. In this paper, each instance is assumed to be an independent latent domain. Under this assumption, the model parameters Θ can be modeled as a function of input instance x: Θ = Θ(x). To this end, a meta-adjuster is designed to generate network parameters to achieve diverse networks so as to adapt to diverse novel samples. In this case, the mapping fΘ(x) is more likely Train Time to align different instances into the same feature space under the reflection of instance-aware parameters Θ(x). Compared with the static mapping that is difficult to align instances from different data distributions, DDG is easier to achieve feature alignment among diverse novel domains.

The pipeline of our method is illustrated in Figure 2, where DDG framework is composed of a static component and a dynamic component updated by the meta-adjuster based on each input instance. The former one is responsible to learn domain-shared features while the latter one is designed to learn domain-specific features. In order to teach the metaadjuster to adapt to diverse novel domains, we simulate numerous novel training samples via a random convex interpolation among the limited source domains.

3.2 Novel Samples Simulation

Numerous novel samples are required to drive the optimization of learning to adjust. By exploiting the limited source domains, we develop a DomainMix method from MixUp [Zhang et al., 2018] to synthesize novel domains with diverse styles. Given two instances (xis , yis ) and (xjs , yjs ) randomly sampled from two different source domains, the process of novel samples simulation is formulated as:

where α is randomly sampled from Beta distribution in each feed-forward. Each DomainMix sample (¯xis , ¯ yis ) can be viewed as an unique domain and be harnessed to drive the following process of learning to adjust.

3.3 Learn to Adjust

In this section, we decouple the network parameters into a static part and a dynamic part which implicitly disentangles the domain-shared and domain-specific features. Here the dynamic parameters are generated by a shallow generator, termed as meta-adjuster, by taking the instance embedding as input. For a better illustration, we take one of the most popular convolutional neural networks, namely ResNet [He et al., 2016], as the backbone of our dynamic network. As shown in Figure 3, the meta-adjuster is embedded in each middle convolution layer in the residual block (note that each residual block is composed by a sequential of 1 × 1, 3 × 3 and 1 × 1 convolution layers). In the following, we will introduce how to design meta-adjuster effectively.

Meta-Adjuster. Meta-adjuster takes the instance embedding Fin(x) (i.e., the output of last block) as input, so that the meta-adjuster practically can be considered as the function of input instance ∆Θ(x). Therefore, the network parameters adjusting process can be formulated as:

where Θs and Θ(x) represent the static parameters and its corresponding up-to-date parameters, and ∆Θ(x) denotes their dynamic term. Theoretically, the dynamic term∆Θ(x) should be the same dimension with the static parameters Θs. However, it makes the amount of network parameters grow quadratically and then apparently increases the risk of overfitting. Thus, according to matrix factorization, we propose to decompose the dynamic term into a learnable linear combination of several kernel templates {V n}Nn=1 which can be formulated as:

where N is a hyper-parameter that represents the number of kernel templates. {λn(x)}Nn=1 denotes the set of dynamic coefficients, which is a group of instance-aware scalars generated by a learnable shallow meta-adjuster λ(·) via taking instance x (practically the instance embedding Fin(x)) as input. In this way, the dynamic component are restricted into a limited space, which benefits the optimization of whole model.

Asymmetric Kernel Templates. An apparent side effect to decompose meta-adjuster into a linear combination of kernel templates and a few dynamic coefficients is that the modulation brought by dynamic parameters are prone to be tiny and rigid. To avoid this tendency, we design four asymmetric kernel templates to explore the relationship between dynamic parameters from different dimensions, as well as to enforce the meta-adjuster to generate more diverse dynamic parameters for a large variety of novel samples. Another discovery is that the learned knowledge of convolutional kernels is not distributed uniformly, since the skeleton of kernels (i.e. the central criss-cross positions) usually has larger weight values [Ding et al., 2019]. For this reason, we propose to enhance the skeleton of kernels by generating skeleton-based dynamic parameters that is derived from a linear combination of these asymmetric skeleton-based kernel templates.

Suppose that the dimension of static kernels Θs is denoted as cin × cout × k × k, where k, cin and cout represent the kernel size, the input channel number as well as the output channel number. As shown in Figure 3, the asymmetric skeleton-based kernel templates are designed as four matrices with different shapes as cin × 1 × k × k, cin × cout × 1 × 1, cin × cout × k × 1 and cin × cout × 1 × k, respectively. Note that the kernel template with size of cin×1×k×k are implemented as depth-wise convolution, because it merely focuses on spatial relationship. In this way, the dynamic parameters are encouraged to be varied for different input samples. In addition, the skeleton of network parameters can be amplified in spatial as well as channel dimensions.

4 Experiments

4.1 Datasets and Experimental Setup

Datasets. 1) PACS [Li et al., 2017] contains more than 9k images with seven categories collected from four domains, which includes: Photo, Sketch, Cartoon and Art Painting. 2) Office-Home [Venkateswara et al., 2017] contains 15, 500 images with 65 categories with a few images per category. All images are sampled from four domains, including: Art, Clipart, Product and Real-World. 3) DomainNet [Peng et al., 2019] is a large-scale benchmark containing 600k images with 345 categories. The images are collected from six different domains which are Clipart, Inforgraph, Painting, Quickdraw, Real and Sketch.

Experimental Setup. For each dataset, we conduct leaveone-out strategy that randomly selects one domain for evaluation and leaves the remaining ones as source domains for training. Each experiment is repeated three times with different random seeds, and the mean and variance of these results are reported in this paper.

4.2 Implementation Details

All the experiments are conducted on RTX 3090 GPU with PyTorch 1.10.0. We use ResNet-50 as the backbone of our dynamic network, by attaching the meta-adjusters to the original residual blocks (as shown in Figure 3), where the spatial size of kernel templates is set as k = 3. Note that in each block, the meta-adjuster is implemented by a sequential of a global average pooling layer, a ReLU layer sandwiched between two fully-connected layers, and a SoftMax layer to output instance-aware coefficients.

Before training DG tasks, we pre-train our network on ImageNet so as to compare with the existing works fairly. Then, we train DG tasks by fine-tuning the pre-trained model with SGD. The training objective is a cross-entropy loss function, exactly the same with vanilla DG. The asymmetric kernel templates and meta-adjuster are optimized with the main backbone simultaneously during training. For PACS and Office-Home, the network optimization is set with batch size of 64, training epochs of 50, and the initial learning rate of 1e-3 decayed by cosine scheduler. While training on DomainNet, most of the hyper-parameters keep the same with that of PACS, except that the initial learning rate and max epoch are 2e-3 and 15, and the mini-batches are fetched with random domain sampler strategy [Zhou et al., 2021], in order to ensure that each domain is uniformly sampled.

4.3 Comparisons with the State-of-the-Art

We compare our method with the state-of-the-art methods on PACS, Office-Home and DomainNet datasets, where the results are reported in Table 1, Table 2 and Table 3, respectively. From these tables, we can observe that:

1. Our method achieves an average accuracy of 86.21, 72.13 and 46.89 percent on PACS, Office-Home and DomainNet benchmark datasets, respectively. Compared to other DG methods, our method achieves the state-of-the-art results on most of DG benchmarks, which shows the superiority of our DDG.
2. Our method is flexible to combine with other methods to further improve the performance. Taking Mixstyle [Zhou et al., 2021] as an example, the performance of our model can be further improved if we insert MixStyle layers into our model. The results can reach 87.87, 72.31 and 46.93 percent on PACS, Office-Home and DomainNet benchmarks, respectively. It shows that our method is rather robust even on the very strong baselines, i.e. MixStyle.

4.4 Ablation Studies

We conduct ablation studies on PACS dataset to evaluate the effectiveness of asymmetric kernel templates and DomainMix respectively, where the results are shown in Table 4.

Effectiveness of Asymmetric Kernel Templates. We study the effectiveness of asymmetric design of kernel templates, by employing common kernel templates that consists of four identical kernels with spatial dimensions of 1 × 1 or 3 × 3. Besides, we also compare with a channel-wise attention method, namely SENet [Hu et al., 2018], which is also a popular and powerful architecture in CNN community. As shown in Table 4, we can see that:

1. Our method achieves superior performance to the vanilla ResNet and SENet by a large margin, which suggests the effectiveness of meta-adjusters on DG tasks.
2. The result of asymmetric kernel templates is obviously better than that of four identical 1×1 kernel templates and 3×3 kernel templates, which confirms the effectiveness of the asymmetric design of kernel templates. In-depth analysis of asymmetric kernel templates will be introduced in the visualization section.

Effectiveness of Novel Sample Simulation. In order to verify the effectiveness of the novel sample simulation method, we also conduct ablation studies on DomainMix. Instead of using DomainMix, we train the networks with limited source domains. Table 4 shows the comparison results, from which we can observe that our method suffers obvious performance degradation when training without DomainMix strategy. This result demonstrates the importance of novel sample simulation in teaching the meta-adjuster to adapt novel samples from agnostic target domains.

4.5 Visualization

For a better understanding of our method, we visualize the kernel templates and dynamic coefficients in different ways. Note that we use the model trained on the multi-source domains (i.e., Art Painting, Cartoon and Sketch) of PACS.

Visualization of Asymmetric Kernel Templates. In order to further explore the mechanism of asymmetric kernel templates, we have a comparison between static kernels in vanilla ResNet and dynamic kernels in our dynamic network, by the means of visualizing kernel magnitude matrix following the literature [Ding et al., 2019]. For ResNet, we sum up the

3 × 3 kernels along the channel and layer dimensions to obtain a 3 × 3 kernel magnitude matrix. For dynamic network, we randomly select an image from validation set to feed into the model and aggregate all the kernel templates with corresponding dynamic coefficients, summing up along the channel and layer dimensions. The comparison result is shown in Figure 4, from which we can see that:

1. The central criss-cross positions have larger values than the corners of kernel, no matter whether static network or dynamic network. It confirms the kernel skeletons are more crucial than the corners when learning knowledge.
2. The kernel skeletons are further strengthened in dynamic network (as shown in Figure 4(b)). The reason is that the design of dynamic combination among kernel templates can adjust or enlarge the weights of kernel skeletons in instance-aware manner. Consequently, it can promote the model for learning advanced knowledge, which further improves training-free adaptation ability in inference.

t-SNE Visualization of Dynamic Coefficients and Features. In order to investigate the effect of dynamic network on different distribution data, we select a few blocks to visualize corresponding dynamic coefficients and the features. Figure 5(a)-(e) show the distributions of dynamic coefficients from different blocks of our model, by taking the whole dataset as input, including source (i.e., Art Painting, Cartoon, and Sketch) and target domains (i.e., Photo). We can observe that the dynamic coefficients tend to separate based on domains, and this phenomenon is decelerated from shallow to deep layers. It proves that the instance-aware coefficients are domain-aware practically, even without the explicit supervision of domain labels. Thus, the domain-aware coefficients can realize domain alignment in feature space, as shown in Figure 5(f). It stresses the superiority of dynamic coefficients on learning domain-invariant features for DG tasks.

5 Conclusion

In this paper, we develop a brand-new DG variant termed as dynamic domain generalization, which aims to explore a training-free mechanism to adjust the model to adapt to agnostic target domains. To this end, we decouple network parameters into static and dynamic parts to disentangle domainshared and domain-specific features, where the latter ones are dynamically modulated by meta-adjusters with respect to varied novel samples from different domains. To enable this process, DomainMix and Asymmetric Kernel Templates are utilized to simulate novel samples and impose the meta-adjuster to modulate the models diversely for the samples from a large variety of novel domains. Extensive experiments demonstrate the superiority of our proposed DDG to vanilla DG approaches. Our work provides a strong baseline for DDG, and it shows possible to further improve DDG from an optimization perspective in the future.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (No.: 83321016, 83418022, U21A20471); the Fujian Provincial Youth Education and Scientific Research Project (No. 650722).