Stabilizing Adversarial Invariance Induction from Divergence Minimization Perspective

1 Introduction

Invariance to nuisance attributes is a desirable property of many representation learning tasks. A domain-invariant representation is key to building classifiers robust to domain changes [Muandet et al., 2013]. When practitioners apply DNN to data that includes a large amount of user information, the desired representations should be invariant to it [Edwards and Storkey, 2016; Iwasawa et al., 2017]. For legal and ethical reasons, machine learning algorithms must make fair decisions that are independent of sensitive variables such as gender, age, or race [Louizos et al., 2016].

Adversarial invariance induction (AII) is a generic and powerful framework for enforcing invariance to nuisance attributes a into neural networks representations z [Xie et al., 2017]. At the core of AII is the idea of using an external attribute classifier qφ to measure the level of invariance, more specifically to approximate the conditional entropy H(a|z).

This approximated conditional entropy is then used to update the feature extractor fθ, which corresponds to updating fθ to deceive the external classifier. [Xie et al., 2017] proved that, under the assumption of optimal qφ, alternatively optimizing the attribute classifier and the feature extractor converges to the equilibrium where the feature extractor maximizes the true conditional entropy. As the true conditional entropy is maximized if and only if the representations are invariant against nuisance attributes, the procedure naturally ensures the invariance of the representations. A similar approach was extensively used in domain generalization, fairness-aware, and privacy-protection contexts [Edwards and Storkey, 2016; Motiian et al., 2017; Xie et al., 2017; Iwasawa et al., 2017].

Despite the theoretical justification, the practical behavior of AII is still unclear and optimization of AII is often unstable. For example, consider the example of learning lighting invariant human identification tasks used by [Xie et al., 2017]. They showed that adversary learned representations still contain significant information on lighting conditions (specifically, 0.53% prediction accuracy for five scenarios). [Moyer et al., 2018] reported that AII is often overfitted; even if the feature extractor utterly defeats an attribute classifier, a post-hoc classifier can predict the attributes from the representations. Section 2.2 further highlights the instability issue using a custom designed toy dataset.

This paper approaches the instability of AII by rethinking it from a divergence minimization perspective. Here, this divergence minimization perspective refers to the approach of ensuring the invariance by aligning the representations associated with different attributes [Zemel et al., 2013; Louizos et al., 2016]. By formally connecting the goal of AII (maximizing the conditional entropy) and the divergence minimization perspective, we identify a fundamental misconception of the AII formulation; i.e.,it does not ensure proper divergence minimization, which is a requirement of the invariant representations. The lack of divergence minimization considerations explains several practical instabilities of AII. While this paper primarily focuses on the original version of AII, our findings are applicable to several extensions of it [Wang et al., 2018; Jaiswal et al., 2019].

We then present a modification to ensure divergence minimization under the adversarial invariance induction framework. As with AII, the proposed method leverages the power of the adversarial game between a feature extractor and an external attribute classifier, but it deceives the external classifier differently; while AII directly maximizes the approximated conditional entropy, the proposed method implicitly maximizes it by forcing the representations with different attributes to be recognized similarly by the external classifier. We refer to the proposed method as Invariance Induction by Discriminator Matching as it induces invariance by matching the discriminator's output between different attributes. While the modification is simple and easy to implement, it attains better properties from the divergence minimization perspective and therefore gives significant performance gains. For example, our method consistently achieves near-optimal invariance in toy datasets with various configurations, in which AII catastrophically fails. Experiments on four real-world datasets (Opp and USC for user anonymization, and MNISTR and PACS for domain generalization) also support the superior performance of this proposal.

2 Instability Issue of Adversarial Invariance

2.1 Preliminary: Adversarial Invariance Induction

Assume we have a training dataset made of the tuples of (x, y, a), where x ∈ X is an observation, y ∈ Y is a target of x, a ∈ A is a nuisance attribute associated with x, and x, y, z are drawn from the true data distribution p(x, y, a). For example, in the context of privacy-preserving activity recognition using data from wearables, the attribute a corresponds to some sensitive user information, and y is the activity label. In the context of domain generalization, learning representations invariant to domain shifts is a popular approach (the domain label is the attribute in that case).

Assume fθ is a feature extractor parameterized by deep neural networks θ. Adversarial invariance induction (AII) optimizes the following min-max game:

where qψ(y|.) and qφ(a|.) is a conditional probability distribution approximated by deep neural networks parameterized by ψ (for y) and φ (for a) respectively. The categorical attribute classifier qφ(a|.) is often called a discriminator or adversary in this context. λ is the weighting parameter.

The above min-max game is closely related to the maximization of the conditional entropy of attributes given representations, H(a|z). This is a good property of AII because H(a|z) is a natural measurements of the invariance, as it is maximized if and only if z is invariant to a. Formally,

where Hp,q(a|z) is the cross entropy between p and q. The bound is tight when p(a|z) = qφ(a|z). Since Hp,q(a|z) is equal to the second term of eq. 1 excluding the sign, minimizing eq. 1 with respect to θ is equivalent to maximizing the variational upper bound of the conditional entropy. By alternately (or jointly with the gradient reversal layer [Gan et al., 2016]) optimizing θ and φ, this framework ensures the removal of nuisance information from the representations. In the remainder of this paper, we assume the use of alternating optimization to solve the adversarial game. At each iteration, AII first updates the attribute classifier κ times by maximizing eq. 1 while fixing fθ. AII then updates the feature extractor and label classifier by minimizing the eq.1 while fixing qφ.

The relationship (eq. 2) is also used to empirically evaluate the level of invariance [Xie et al., 2017]; i.e., Hp,qeval is used as an estimate of H(a|z) where qeval is a post-hoc classifier that is trained to predict a over the learned representations as correctly as possible. We uses same procedure to empirically evaluate the conditional entropy.

2.2 Instability of AII: Case Study

Before the main analysis, here we highlight the unstable behavior of AII using a toy dataset. The dataset comprises samples from K pieces of Gaussian distributions with different means ([sin( iK π), cos( iK π)], and i ∈ 1, 2, · · ·K) and the same variance, assuming that each distribution corresponds to different attributes. We apply AII on this dataset; qφ, which predicts the distribution ID from the representations, is first updated 100 times with a batch size of 128, and qφ and fθ are alternately updated using stochastic gradient descent with a learning rate of 0.1. Figure 1 compares three values: (1) Hp,qeval (a|z) using a post-hoc classifier qeval, (2) Hp,q(a|z) using qφ which is used during the training, and (3) the theoretical maximum value of the conditional entropy (green line). The post-hoc classifier qeval is parameterized by a neural network with the same architecture as that of qφ. For reference, the learned representations after several iterations (1, 35, 40, and 200 steps respectively) are also visualized. For simplicity, here we only show the results of K = 3. We later compare AII and our proposed method on various configurations.

The results indicate a significant mismatch between Hp,qeval (a|z) (red line) and Hp,q(a|z) (dashed blue line). For example, around 40 iterations, the estimates using qφ become significantly large but the estimates with a post-hoc classifier remain constant or even decrease. As Hp,qeval (a|z) is a more reliable estimate of the conditional entropy, the mismatch means that AII uses inaccurate estimates to update fθ. Moreover, the estimates of conditional entropy using qφ are often larger than the theoretical maximum value of H(a|z). As such, the optimization of AII is unstable and does not maximize the conditional entropy H(a|z) as expected.

(b) AII

(c) Proposal

Figure 2: Visualizing the gradient vector fields (the arrow in b and c) of AII and proposed method. (a) Both methods utilize qφ(a|z), which represented as counterplot in the figure. (b) AII has an incentive to move the distribution away from decision boundary. (c) IIDM prevent the problem by considering both the decision boundary and information from pθ(z|aj) of different attributes.

2.3 Divergence Minimization Perspective

This paper examines the above instability issue from a divergence minimization perspective. Specifically, we consider the pairwise divergence among the conditional distribution of representations given different attributes. In the toy dataset used in Sec. 2.2, it corresponds to the distributional difference among red, green, and blue data points. We first formally link the divergence minimization perspective and the conditional entropy maximization problem, which is the goal of the formulation of AII. Assume D is the divergence over a space of probability distributions.

Proposition 1. We assume that a is a uniform categorical random variable, and z is a random variable of the representations. If fθ gives a representation that maximizes the conditional entropy H(a|z), then D(pθ(z|ai)||pθ(z|aj)) = 0 for all ai, aj and vice versa. Here, the subscription denotes that the distribution depends on the feature extractor.

Proof. The proposition can be derived by the following property of the conditional entropy: H(a|z) is maximized if and only if p(z|ai) = p(z|aj) for all ai ̸= aj ∈ A and z ∈ Z. Using the Lagrange multiplier, the derivative of

is equal to zero for the maximum entropy H(a|z). Solving both equations, we can say p(a1|z) = p(a2|z) = · · · = p(aK|z) = 1K for all z ∈ Z when the conditional entropy is maximized, and based on the definition, the conditional entropy becomes -log 1K .

From Bayes' law and the assumption of uniformity of p(a), p(z|ai) = p(z|aj) holds ∀ai ̸= aj ∈ A and z ∈ Z.

This proposition means that maximizing conditional entropy is asymptotically equal to minimizing pairwise divergence. Unfortunately, the connection challenges AII, because it only pushes the distribution away from the decision boundary, without considering the divergence minimization requirement. Figure 2-(a, b) visualizes how AII updates the feature extractor using the toy dataset used in the previous section. The arrows in the figure represent the direction of the gradient when updating the fθ using the data from pθ(z|a = red)

(updating the distribution pθ(z|a = red)). The gradient vector suggests that AII has an incentive to move the distribution far away, regardless of whether it aligns marginal distributions of different attributes. In other words, AII keeps the pθ(z|a = red) away from the non-desired point where a discriminator correctly predicts the attribute, but does not ensure that it approaches some target distribution, such as pθ(z|a = blue) or pθ(z|a = green).

The lack of divergence minimization explains the undesired behavior of AII described in the previous section. For example, Hp,q(a|z) reaches values larger than the theoretical maximum of the conditional entropy when AII moves the distribution without minimizing the divergence. Also, a significant mismatch between the approximated entropy Hp,q(a|z) and Hp,qeval (a|z) can happen when the update of the feature extractor deceives the current discriminator without minimizing the divergence, or even increasing it.

3 ProposedMethod

Based on this analysis, we hypothesize the lack of the divergence minimization perspective is a major cause of unstable behavior. Here, we describe a way to effectively incorporate the divergence minimization requirement into the adversarial invariance framework. Similar to the original AII, our method employs adversarial training between a feature extractor fθ and a discriminator qφ, but we deceive the discriminator differently. Specifically, we update fθ by minimizing the expectation of the following discriminator matching loss:

where qiθ,φ (a) = ∫ pθ(z|ai)qφ(a|z)dz, which represents the average output of the discriminator given representations associated with attribute ai. DKL is the KL-divergence defined over the output of the discriminator:

We refer to this method as invariance minimization by discriminator matching (IIDM).

While the modification is simple, the objective has better interpretation from the perspective of divergence minimization. In contrast to AII, which updates the feature extractor by considering only the decision boundary, the discriminator matching loss also considers the location of the representations of different attributes. Figure 2 compares the gradient vector fields between AII and IIDM. The gradient vector fields indicate that our proposed method has no incentive to move the distribution far away from the decision boundary, as it violates the divergence minimization constraint. In other words, IIDM prevents the feature extractor to move the distribution regardless of whether it aligns marginal distributions of different attributes or not.

More formally, the objective of IIDM is related to the divergence DKL(pθ(z|ai)||pθ(z|aj)) through the divergence between DKL(qiθ,φ (a)||qjθ,φ (a)). Firstly, we can derive

using the Jensen's inequality. Also, based on the data processing inequality of the f-divergence [Gerchinovitz et al., 2017; Barber et al., 2018], the following inequality holds:

The equality holds if the attribute classifier is invertible [Barber et al., 2018]. In the special case, minimizing the discriminator matching loss ensures the divergence minimization as it is the upper bound of DKL(pθ(z|ai)||pθ(z|ai)). Note that, the invertibility is similar to the optimality of the attribute classifier, which is often assumed in the analysis of the adversarial training. Ensuring inevitability is difficult in general and an open research areas in neural networks community [Behrmann et al., 2018]. Instead, we empirically validate that the proposed method reliably learns invariant representations even without such a regularization.

One implementation issue is how to calculate qiφ (a) in Eq. 3. The straightforward approach is through Monte Carlo approximation: qiφ (a) = Epθ(z|ai)[qφ(a|z)]. Although it is an unbiased estimation, the variance is large if the number of samples is small. The average can be calculated from all samples (or a sufficiently large number of samples from each K attributes) at every iteration. However, it requires additional computation other than standard mini-batch estimation. Moreover, the computation becomes intractable (O(K2 )) as the number of attribute values grows.

We address these issues by a moving centroid mechanism. Instead of estimating qiθ,φ (a) every time with sufficiently large samples, IIDM uses the moving average of qiθ,φ (a):

where Qjt-1 is a previous centroid, qi t (a) is the new estimation of the centroid based on a single batch, and γ is the decay parameter for controlling the speed at which the centroids change. We initialized Qj0 by using all training data points.

Then we can use the standard mini-batch method to calculate the discriminator matching loss. As with AII, IIDM incorporates alternating optimization. Specifically, IIDMfirstly updates the attribute classifier qφ by eq. 1, and updates the feature extractor and the classifier by

where λ is a weighting parameter. Figure 3 shows a pictorial illustration of the proposed method.

4 RelatedWork

As briefly mentioned in the introduction of the paper, the formulation of AII (eq. 1) has theoretical groundings as a way to maximize the conditional entropy [Xie et al., 2017]. While our analysis reveals the unstable behavior of AII, we do not intend to challenge the previous theoretical analysis. Building upon the theoretical grounding under ideal conditions, we examine the practical problem of AII by rethinking it from a divergence minimization perspective of invariance induction. We admit that the perspective itself is not brand new in these fields [Zemel et al., 2013; Louizos et al., 2016]; nevertheless, none of the studies has been linked AII with the divergence minimization perspective.

Similar to our work, several studies have proposed extensions of AII. For example, [Jaiswal et al., 2019] proposes an adversarial forgetting mechanism to ensure the invariance, by introducing another network to produce forgetting masks over the representations. [Wang et al., 2018] combines the adversarial invariance objective and a variational autoencoder to further enforce the invariance. As we have not introduced any architectural modifications, our method could be incorporated to the other extensions. Besides, our findings regarding the original formulation are applicable to several extensions that have the same objective.

As a extention of AII, several studies focus on the semantic alignment problem, i.e., how to align only the pair of samples that have the same semantics (the target label) [Li et al., 2018b; Li et al., 2018c]. Our method can be extended to consider semantic alignment without additional computational costs; semantic alignment can be carried out by computing the centroids for each (attribute, label) tuple and aligning the qφ(a|z) of {x, y, a} between only centroids of the same label y′ = y but different attributes a′ ̸= a. We test this variant in experiments below and denote it as IIDM+.

It is noteworthy that the above formulation resembles the original formulation of GAN [Goodfellow et al., 2014] and domain adversarial networks (DAN) [Ajakan et al., 2014; Gan et al., 2016]. However, it is never used practically as it is known to be unstable and hard to optimize. This fact motivates us to replace the min-max game of the adversarial invariance induction problem. Although no prior work in the invariance induction community has been explicitly considered yet, one can transfer the non-saturating (NS) heuristic used in the GAN via label flipping. The NS-objective is similar to our method in the sense that it uses an asymmetric objective. However, NS has same issue with AII as it only considers the current decision boundary and does not ensure the divergence minimization. We refer to this version as the non-saturating version and denote it by NS.

5 Experiments

5.1 Experimental Settings

Datasets

We provide experimental results on the synthesized dataset and two real world tasks (four datasets) relevant to invariant feature learning: (1) user anonymization (Opportunity and USC datasets), and (2) domain generalization (MNISTR and PACS datasets). All experiments were implemented in PyTorch and were run on either a GTX 1080 or Tesla V100.

For user anonymization tasks, Opportunity and USC datasets were used. This task is to learn anonymized representations (z that do not contain user-identifiable information) while maintaining classification performance. The Opp dataset [Sagha et al., 2011] consists of sensory data regarding human activity in a breakfast scenario. Each record consists of 113 real-value sensory readings. We considered the task of recognizing 18 classes 1 . Following previous studies [Yang et al., 2015; Iwasawa et al., 2017], we use a sliding window procedure with 30 frames and a 50% overlap. The number of samples was 57,790 in total. The feature extractor is parameterized by a convolutional neural network (CNN) with three convolution-ReLU-pooling repetitions followed by one fully connected layer and classification by logistic regression. The discriminator is a simple feedforward network with 800-400 hidden units. The USC-HAD dataset is another activity recognition dataset that consists of 14 subjects [Zhang and Sawchuk, 2012]. The data include 12 activity classes 2 that correspond to people's most essential and daily activities. MotionNode, which is a 6 DOF inertial measurement unit, is used to record the output from accelerometers that record six real sensory values. The same sliding window procedure produced 172,169 samples.

The MNISTR and PACS are two typical datasets of domain generalization tasks. The MNISTR dataset, derived from MNIST, was introduced by [Ghifary et al., 2015]. Its labels comprise the ten digits; domains are created by rotating the images in multiples of 15 degrees: 0, 15, 30, 45, 60, and 75. The domains are labeled with the angle by which they are rotated, e.g., M15 and M30. Each image is cropped to 16 × 16 pixel in accordance with a previous study [Ghifary et al., 2015] 3 . Similar to [Ghifary et al., 2015], we used two convolution layers with 32 and 48 filters of 5 × 5 kernels, followed by a max-pooling layer and two fully connected layers with 100 hidden units. A discriminator with 100 hidden units is connected to the output of the first fully connected layer. The PACS dataset is a relatively new benchmark dataset designed for cross-domain recognition [Li et al., 2017]. It comprises 9991 images in total across seven categories (dog, elephant, giraffe, guitar, house, horse, and person) and four domains of different stylistic depictions (photo, painting, cartoon, and sketch). The diverse depiction styles provide a significant domain gap. We use the ImageNet pre-trained AlexNet CNN [Krizhevsky et al., 2012] as a base network, following previous studies[Li et al., 2017; Li et al., 2018a]. A discriminator with 1024 hidden units is connected to the output of the last fully connected layer.

1 open door 1, open door 2, close door 1, close door 2, open fridge, close fridge, open dishwasher, close dishwasher, open drawer 1, close drawer 1, open drawer 2, close drawer 2, open drawer 3, close drawer 3, clean table, drink from cup, toggle switch, and null

2 walking forward, walking left, walking right, walking upstairs, walking downstairs, running forward, jumping, sitting, standing, sleeping, elevator up, and elevator down

3 We used the dataset distributed at https://github.com/ghif/mtae.

Baselines

CNN trained on the aggregation of data from all source domains. Although there are special treatments for domain generalization, [Li et al., 2017] reports that CNN outperforms many domain generalization methods on the PACS dataset. AII [Xie et al., 2017] is the main baseline. AII+GP uses a variant of AII with an additional gradient penalty regularization used in GAN [Mescheder et al., 2018]. RevGrad is a slightly modified version of AII, which uses the gradient reversal layer [Ganin et al., 2016] to train all the networks (feature extractor, classifier, and discriminator) at the same time. NS is the non-saturating version of AII described in section 4 of this paper. CrossGrad [Shankar et al., 2018], regarded as a state-of-the-art method in domain generalization tasks. Note that it does not intend to learn invariant representation, so we use CrossGrad only for comparing domain generalization performance. IIDM is our proposal. We used the gradient penalty as well. We also tested the semantic alignment version and denoted it as IIDM+.

Optimization

For all datasets and methods, we used RMSprop for optimization. For all datasets except PACS, we set the learning rate to 0.001 and the batch size to 128. For PACS, we set the learning rate to 5e - 5 and the batch size to 64. The number of iterations was 10k, 5k, 20k, and 30k for MNISTR, PACS, Opp, and USC, respectively. For a fair comparison, hyperparameters were tuned on a validation set for each baseline. For the adversarial-trainingbased method, we optimized weighting parameter λ from {0.001, 0.01, 0.1, 1.0}, except for MNISTR, for which it was optimized from {0.01, 0.1, 1.0, 10.0}. The value of α for CrossGrad was selected from {0.1, 0.25, 0.5, 0.75, 0.9}. Unless mentioned otherwise, we set the decay rate γ to 0.7.

Evaluation

In all the experiments, we selected the data of one or several domains for the test set and used the data of a disjoint domain as the training/validation data. We split the data of the disjoint domain into groupings of 80% and 20%. We denote the test domain by a suffix (e.g., MNISTR-M0 denotes that the model is trained with the data from M15, M30, M45, M60, and M75 and evaluated on M0). We conducted 20 validations during training at equal intervals. In each validation, we measured the label classification accuracy (Y-acc) and the level of invariance. For measuring the level of invariance, we trained a post-hoc classifier qeval with 800 hidden units 1k iterations (by RMSprop optimizer, with a learning rate of 0.001 and a batch size of 128) with the data that is used to train the feature extractor.

5.2 Results

Simulation

Figure 4 compares the performance of (a) AII (b) NS, and (c) IIDM on different configurations of {K, κ}, where color denotes different K, marker denotes different κ. The dashed line denotes optimal values. The experimental settings are the same as in Section 2.2. The results clearly support the benefits of the proposed method. (1) In all configurations, IIDM reaches the theoretical maximum values within the first few iterations. (2) In all configurations, AII and NS are unstable as shown in the vibration of the estimated conditional entropy. When κ = 1, AII and NS are far from the maximum conditional entropy. AII and NS with κ = 8 give much better performance, but its behabior is still unstable and convergence is significantly slower than the IIDM.

User Anonymization

Table 1 represents the lowest user-classification accuracy (the lower the better) with specific performance degradation compared to CNN on classification accuracy. For example, the columns with 0.01 represent the lowest user-classification accuracy with less than 0.01 point performance degradation. The best performance is underlined and highlighted in bold, and the second-best performance is highlighted in bold. Note that the value 'None' represents the method always reducing the label classification performance significantly. As a results, IIDM performs best on seven out of ten configurations, suggesting the clear benefit of our proposal on the user anonymization task.

Domain Generalization

Table 2 summarizes the classification performance on two different datasets: MNISTR, and PACS. The top row of each table represents the test domain. We report the mean accuracy as well as the standard error of five seeds for MNISTR and three seeds for PACS. The best performance is underlined and highlighted in bold, and the second-best performance is only highlighted in bold. We can make the following observations: (1) IIDM and IIDM+ show the best or comparable performance on all conditions except the sketch domain. Although the semantic alignment extension does not help with a simpler task (MNISTR), it improves the performance on the PACS dataset, giving approximately 1.0 point performance gain. (2) RevGrad and AII often fail to improve performance even when compared with a standard CNN. The score of AII+GP suggests that the gradient penalty helps to improve performance, but the improvements are lower than our proposal. (3) The Wilcoxon rank-sum test shows that IIDM is statistically better than CNN, RevGrad, AII, AII+GP, and CrossGrad with p < 0.01.

Figure 5 compares AII and IIDM on different (a) weighing parameter γ, (b) the number of the discriminator updates κ, and (c) the network architecture of the discriminator. The dataset used is MNISTR with M0 as a test domain. In each figure, color represents a different method (red: AII, blue: IIDM) and marker denotes different configurations. The value represents the attribute classification accuracy (the lower the better invariant) of a post-hoc classifier qeval(a|z). For λ we used 1.0 by default. For κ and the architecture, we used the default settings described in Section 5.1. The results show that our proposal consistently learns better invariant representations regardless of the choice of hyperparameters. These results suggest that our proposal is better than searching for such hyperparameters. Note that, λ = 10.0 for AII seems to attain better invariance, but it was degenerated to the random representations and gives a random performance on the classification of y.

6 Conclusion

This paper examines the optimization difficulty of AII (highlighted in Figure 1) and proposes a new method to attain invariance to nuisance attributes, by rethinking the AII's objective from a divergence minimization perspective. By formally linking the goal of AII with the pairwise divergence mini-

(a) MNISTR

(b) PACS

Figure 5: Comparison of AII and IIDM with different configurations on MNISTR dataset (M0 as test domain). The number in parenthesis represents the corresponding configuration.

mization of conditional distribution of representations given attributes (Proposition 1), we identify a cause of its optimization difficulty; it does not ensure proper divergence minimization, which is a requirement of the invariant representations. We propose a simple way to effectively incorporate this requirement into the adversarial invariance framework, which leverages the power of the adversarial game but solves it more stably. Namely, the proposed method minimizes KL divergence defined over a space of attribute classifier's output (eq. 3), which is closely related to the divergence over the representation space, which we want to minimize. While the modification is easy to implement, it gives a significant performance gain; Our proposal consistently achieves near-optimal invariance in a toy dataset (Figure 4), where AII results in significantly unstable behavior. Our method is also good at user anonymization tasks (Table 1), and domain generalization tasks (Table 2). All these results suggest that the proposed method works well, and the divergence minimization interpretation introduced in this paper is significant.