Learning Calibrated Uncertainties for Domain Shift: A Distributionally Robust Learning Approach

1 Introduction

Uncertainty estimation is an important machine learning problem that is central to trustworthy AI [Antifakos et al., 2005; Tomsett et al., 2020]. In addition, many important downstream applications rely on the correct estimation of uncertainties. This includes unsupervised domain adaptation [Zou et al., 2019] and semi-supervised learning [Sohn et al., 2020], where they are used to solicit confident pseudolabels for re-training. In these applications, reliable pseudolabels help avoid error propagation and catastrophic failures in early iterations [Kumar et al., 2020].

∗Correspondence authors.

†Code repository: https://github.com/hatchetProject/DeepDistributionally-Robust-Learning-for-Calibrated-Uncertaintiesunder-Domain-Shift

Obtaining reliable uncertainty estimation is challenging. In contrast to human annotations of labels, obtaining the groundtruth uncertainties from real-world data can be costly or even infeasible. It is also known that the commonly used uncertainty proxies in deep neural networks, such as the softmax output, tend to give overconfident estimates [Guo et al., 2017]. This overconfidence is further amplified under domain shifts, where the target (test) domain and the source training domain differ significantly. Such distribution shifts tend to aggravate the existing issues in uncertainty estimation, leading to wrong but overconfident predictions on unfamiliar samples [Li and Hoiem, 2020].

Many methods have been proposed to calibrate the confidence of deep learning models so that the uncertainty level of a model prediction reflects the likelihood of the true event [Guo et al., 2017]. Label smoothing is a popular approach to reduce overconfidence and to promote more uniform outputs [Szegedy et al., 2016]. Temperature scaling is another method where the logit scores are rescaled by a calibrated temperature [Platt and others, 1999]. Approaches such as Monte-Carlo sampling [Gal and Ghahramani, 2016] and Bayesian inference [Blundell et al., 2015] model uncertainties from a Bayesian perspective but are computationally expensive. Though these methods lead to more calibrated uncertainties, recent studies show that their results cannot be fully trusted under domain shifts [Snoek et al., 2019].

Our approach: To handle domain shifts, we characterize the "overlap" between the source training data and the test data. Intuitively, if a test sample is distant from the training distribution, then its confidence level should be lowered. We incorporate this insight by estimating a density ratio for each sample and employ it for confidence calibration.

To be concrete, recall that the probability output for a classification neural network can be expressed as P(y|x) ∝ exp (θy · ϕ(x)), where ϕ(x) is the data feature for input x, and θy is the model parameter of the y-th class. We instead propose the following predictive form for our neural network:

Table 1: Comparison with the most related [Liu and Ziebart, 2014].

	Paradigm	Data Distribution	Prior
RBA Ours	Two-stage End-to-end	Simple (Gaussian) Complex (Image)	Yes No

where Ps(x) and Pt(x) are the densities of a data sample under the source and target distributions, respectively. When a target sample is close to the source domain (large Ps(x)/Pt(x)), the prediction is confident. However, when a target sample x is far away from the source distribution (small Ps(x)/Pt(x)), the confidence is lowered and the prediction is closer to a uniform distribution. This intuition is analogous to incorporating a sample-wise temperature to adjust the confidence according to the closeness of a test sample to the training distribution.

Eq. 1 is based on the distributionally robust learning (DRL) framework. DRL is an adversarial risk minimization framework that involves a two-player minimax game between a predictor and an adversary [Gr¨unwald et al., 2004]. While many previous DRL methods [Liu et al., 2020; Nakka et al., 2020] operate in low-dimensional spaces using kernel density estimators for the density ratio estimation, we develop a DRL method to scale up to real-world computer vision tasks, which is able to produce calibrated uncertainties under domain shift.

The highlights of the paper are:

1) For the first time to our best knowledge, we propose a DRL method for uncertainty estimation under domain shift. We introduce a density ratio estimator which learns to predict the density ratios between source and target domains (Fig. 1(a)). The density ratio estimator and the target classifier are trained simultaneously in an end-to-end fashion. We also introduce additional regularization to improve calibration performance. A comparison of our framework with the most related DRL work is shown in Table 1.

1. We show that the estimated density ratio reflects the distance of a test sample from both training and test distributions (Fig. 1(b)). Our experiments further empirically show that these estimates are also well correlated with human selection frequencies, based on the ground-truth labels in ImageNetV2.
2. We empirically show that the top-1 class predictions of DRL are more calibrated than empirical risk minimization and temperature scaling on Office31, Office-Home, and VisDA. We measure the level of calibration using expected calibration error (ECE), Brier Score and reliability plot.
3. We integrate our method as a plug-in module in downstream applications such as unsupervised domain adaption and semi-supervised learning, leading to significant improvements. For example, incorporating self-training with DRL leads to state-of-the-art performance on VisDA-2017 and a 6% improvement on hard examples. Incorporating Fixmatch with DRL improves the original Fixmatch by a relative 17% increase in accuracy under the cross-domain setting.

2 RelatedWork

Domain classification for domain shift: When dealing with classification problems under distribution shifts, one prevalent way is to discriminate different distributions. A popular approach is importance weighting using density ratio estimation [Sugiyama et al., 2012], which reweighs source samples using pre-estimated density ratios to match the target distribution. However, importance weighting is known to be of high variance and only few works cover high-dimensional data [Khan et al., 2019; Park et al., 2020]. Other methods use a classification network to differentiate the learned source and target representations (adversarially) to locate a common subspace [Ganin et al., 2016; Tzeng et al., 2017; Long et al., 2018]. These works do not analyze the uncertainty estimation problem under domain shift. Yet our density ratio generation method is different from the above approaches in both motivation and implementation. We learn the density ratios using DRL's predictive form for generating calibrated predictions, and helps with choosing better pseudolabeled data from the target domain in downstream tasks.

Uncertainty calibration in deep models: Uncertainty calibration aims for matching the model output probability with the true frequency of that event [Kumar et al., 2019], and is achieving increasing attention in deep learning [Nixon et al., 2019]. Popular methods include Bayesian deep learning [Gal and Ghahramani, 2016] and temperature scaling [Guo et al., 2017]. However, these methods are either computationally expensive or not designed for the domain shift setting. Methods that focus on calibration for uncertainties under domain shift either do not fundamentally change the uncertainty generation process [Han et al., 2019; Lee and Lee, 2020] or were built upon the traditional importance weighting setup [Park et al., 2020; Wang et al., 2020]. In this paper, we directly generate more calibrated uncertainties based on the DRL framework under domain shift without directly optimizing the calibration error.

Distributionally robust learning under domain shift: Several different learning algorithms can be derived from the DRL framework [Liu and Ziebart, 2014; Fathony et al., 2016; Liu and Ziebart, 2017]. When applied to domain shift cases, an important limitation of these works is that they need density ratios to be estimated beforehand. This requires concrete prior distribution knowledge of the target domain, making them unsuitable for complicated tasks. Our work instead estimates the density ratio along with the learning process and is applicable on real-world tasks. While using a minmax framework, our work is orthogonal to adversarial training, which perturbs the covariate variable [Hu et al., 2018; Najafi et al., 2019] and focuses on robustness against adversarial perturbations. Our work derives a novel predictive form from class-regularized DRL by explicitly solving the minmax game and integrates differentiable density ratio estimation in the end-to-end training process.

3 Distributionally Robust Learning

In this section, we first review the preliminaries of DRL (Sec. 3.1), followed by a proposed variant of DRL with class regularization (Sec. 3.2). We then also propose an instantiation of DRL with a differentiable density ratio estimation network (Sec. 3.3). Finally, we show the applications of our framework in UDA and SSL tasks (Sec. 3.3).

3.1 Preliminaries

Notations and definitions: Denote the input and labels by random variables X and Y , respectively. Use x ∈ Rd and X to represent the realization and sample space of X. Our goal is to find a predictor f(x) : Rd ↦→ RC , where x ∈ X, f ∈ RC ∩ ∆, that is close to the true underlying Pt(Y |X). Here, d, C and ∆ denote the input dimension, class number, and probabilistic simplex, respectively. We consider the problem with labeled data sampled from a source distribution Ps(X, Y ) and unlabeled data sampled from a target distribution Pt(X), and use Ps(X, Y ) to represent the empirical source distribution. In this work, we consider an important form of domain shift with the covariate shift assumption Ps(X) ̸= Pt(X), Ps(Y |X) = Pt(Y |X).

Motivation: Traditional empirical risk minimization (ERM) frameworks tend to fail under covariate shift since ERM empirically learns a predictor ˆ Ps(Y |X) from the finite source data that usually fails to generalize to the target distribution. DRL is proposed to overcome this issue, which can be formulated as a two-player adversarial risk minimization game [Gr¨unwald et al., 2004] with the predictor player minimizing a loss, while the adversary player maximizing it. The adversary is allowed to perturb the labels, subject to certain feature-matching constraints to ensure data-compatibility.

Formulation: For covariate shift, DRL [Liu and Ziebart, 2014] deals with the mismatch between the expected loss and the training data, and is defined on target distribution:

where f(x), g(x) ∈ RC are the conditional label distributions given an input x. Σ is a constraint for g to ensure the invariant conditional label distribution under covariate shift, which we will formulate in Eq. 4. Both f and g are not parameterized yet. Ex∼Pt L(·) is an expected log loss on the target input:

f is the predictor player minimizing the loss function while g is the adversary maximizing the loss function. After solving this game, f is our estimate of ˆ Pt(Y |X), which we will use for the classification task on the target domain.

Eq. 2 is defined on the target domain only. How could a predictor be properly trained while there are no target labels available? The answer is that the adversary g is implicitly constrained by the source features. We use the following constraints to make sure that g is close to Ps(Y |X):

where xi ∼ Ps(X), gy is the y-th dimension of g and ϕ(xi) is the feature for xi. Eq. 4 is a necessary but not sufficient condition for g = Ps(Y |X), thus serving as an implicit constraint for g to be close to the true Pt(Y |X) under the covariate shift assumption. Given a predefined feature function ϕ, when the adversary perturbs the conditional label distribution, certain aggregate function of ϕ on g should equal to the counterpart on the empirical source data.

From DRL to density ratio (derivation of f): When using the expected target log loss in Eq. 2, Eq. 1 is derived by solving the predictor f. Here we refer the derivation details to [Liu and Ziebart, 2014] but emphasize an important property of the prediction: representation-level conservativeness, which means the predictions are more certain for inputs closer to the source domain (larger Ps(x)/Pt(x)) and more uncertain when Ps(x)/Pt(x) is small. This property reflects the model's ability to convey information about what it does not know through the model uncertainty.

3.2 Class-regularized Distributionally Robust Learning

Inspired by label smoothing [Szegedy et al., 2016] and regularization of neural network outputs [Pereyra et al., 2017], we further add class-regularization to the prediction form in Eq. 1. Instead of doing it in a post-hoc way, we incorporate class-regularization into the original DRL formulation. We propose to use a weighted log loss to penalize the high confidence in the adversary's label prediction:

where y is the one-hot class vector, ⊙ is the element-wise product, and r ∈ [0, 1] is a hyper-parameter that controls the level of regularization. Σ here is the same as in Eq. 2. We call this formulation class-regularized distributionally robust learning. Eq. 5 is a convex-concave function in terms of f and g. According to the strong duality, we switch the order of the min and max. With a fixed g, f = g is the optimal solution of the inner min problem. So we have the following lemma (refer to appendix for proof) :

Lemma 1. Eq. 5 can be reduced to a regularized maximum entropy problem with the estimator constrained:

where Σ is the same as in Eq. 4, meaning that f should be close to the empirical source Ps(Y |X).

Theorem 1. The solution of Eq. 6 takes the form: fθ,r(y|x) ∝ exp ( Ps (x) Pt (x) θy ·ϕ(x)+rI(y) rI(y)+1 ) , where θ represents the model parameters and I(y) is the yth dimension of the one-hot encoding y.

The proof of Theorem 1 (refer to appendix) follows the same principles of deriving Eq. 1. Different from [Liu and Ziebart, 2014], we use Theorem 1 as the new prediction form. In training, y is the one-hot encoding of each class. In inference, we set y to be an all-one vector.

Class-level regularization: Hyperparameter r adjusts the smoothness of g's label prediction in Eq. 5. It translates to the ry term in the prediction form. Intuitively, it increases the correct label's prediction logits when it is smaller than 1 and decreases the logits when it is larger than 1. Thus r provides additional regularization and smoothness to the conservative prediction.

3.3 Differentiable Density Ratio Estimation

Estimating Ps(x)/Pt(x) is challenging [Sugiyama et al., 2012], especially in high-dimensional spaces. Usually an estimator that calculates the densities in advance is used. But these estimates are usually sub-optimal in practice due to the different downstream tasks' objectives. We propose an end-to-end framework where the density ratio estimator is trained together with the target classifier. We introduce the novel differentiable density ratio estimation before proposing a joint training loss and the parameter learning process.

Differentiable density ratio estimation: Based on the Bayes' rule, Ps(x)/Pt(x) can be computed from a conditional domain classifier [Bickel et al., 2007]: Pt(x) Ps(x) = P (x|t) P(x|s) = P(t|x)P(s) P(s|x)P(t) . Concretely, they can be estimated via binary classification using unlabeled source and target data with P (s) P(t) as a constant relating to the number of source and target samples. On the other hand, we observe that Ps(x)/Pt(x) can be a trainable weight for each sample, and can be updated via the training objective of DRL. Therefore, we propose to train a binary classifier as the density ratio estimator, using both the binary cross-entropy loss and DRL's objective loss to update the network's parameters.

Our approach is different from existing methods (e.g. [Che et al., 2021]) in three aspects: First, our binary classification network is proposed for the goal of achieving calibrated uncertainties under domain shift; Second, the weights trained this way lose their original properties as density ratios, but still reflect the relation between the two domains; Third, our obtained density ratios are mathematically the inverse of other methods' density ratios due to the DRL formulation.

Joint training loss: Assume ϕ(x,wr) is the representation learning neural network with parameter wr. We further define Pd(X) as the joint distribution of both source data and target data with their domain labels D = {d(x)}. We denote τ(x,wd) = (τs, τt) (where τs + τt = 1) as the two dimensional probability output of the source and target domains from a domain classifier τ with parameter wd. Our joint training loss is defined as:

where gt(x) = Pt(Y |X = x) and f(x; wr, θ, rd) takes the form in Theorem 1:

Parameter learning: Note that gt in the first loss term of Eq. 7 concerns the conditional label distribution on target which is assumed to be not available. However, with the DRL formulation, we can compute and evaluate the gradients of θ and ϕ(x, wr) directly with a change of measure in the derivation of the gradients so that the gradients' calculation does not depend on the target distribution (see details in appendix). We show this in Fig. 1(a) that the representation learning network ϕ only uses source data as the input. In this way, the parameter learning process that originally depend on source and target data distributions (loss function) are now only associated with the source data and labels (gradient). We then update θ and wr using the computed gradients and also directly back-propagate from the second loss term in Eq. 7 to update wd. Finally, we treat the densities as trainable variables and derive gradients for them from the

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

T

7

2

0

v

G

P

W

B

+

Z

p

E

d

M

u

m

g

>

A

n

c

V

N

S

8

J

3

U

r

q

/

o

9

L

R

K

D

w

Q

H

y

5

X

O

I

k

j

f

F

Y

z

C

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

T

7

2

0

v

G

P

W

B

+

Z

p

E

d

M

u

m

g

>

A

n

c

V

N

S

8

J

3

U

r

q

/

o

9

L

R

K

D

w

Q

H

y

5

X

O

I

k

j

f

F

Y

z

C

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

T

7

2

0

v

G

P

W

B

+

Z

p

E

d

M

u

m

g

>

A

n

c

V

N

S

8

J

3

U

r

q

/

o

9

L

R

K

D

w

Q

H

y

5

X

O

I

k

j

f

F

Y

z

C

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

T

7

2

0

v

G

P

W

B

+

Z

p

E

d

M

u

m

g

>

A

n

c

V

N

S

8

J

3

U

r

q

/

o

9

L

R

K

D

w

Q

H

y

5

X

O

I

k

j

f

F

Y

z

C

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

Z

N

H

E

0

3

5

k

Y

q

F

8

j

M

/

A

>

B

n

c

V

S

J

U

r

o

9

L

G

P

R

K

g

D

T

w

m

7

Q

v

y

2

+

u

f

W

z

d

I

p

X

C

O

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

Z

N

H

E

0

3

5

k

Y

q

F

8

j

M

/

A

>

B

n

c

V

S

J

U

r

o

9

L

G

P

R

K

g

D

T

w

m

7

Q

v

y

2

+

u

f

W

z

d

I

p

X

C

O

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

Z

N

H

E

0

3

5

k

Y

q

F

8

j

M

/

A

>

B

n

c

V

S

J

U

r

o

9

L

G

P

R

K

g

D

T

w

m

7

Q

v

y

2

+

u

f

W

z

d

I

p

X

C

O

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

Z

N

H

E

0

3

5

k

Y

q

F

8

j

M

/

A

>

B

n

c

V

S

J

U

r

o

9

L

G

P

R

K

g

D

T

w

m

7

Q

v

y

2

+

u

f

W

z

d

I

p

X

C

O

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

3

u

E

U

w

R

2

+

g

D

M

T

m

W

/

k

V

>

A

B

n

c

N

S

8

J

r

q

9

L

0

F

7

Y

o

y

p

Z

I

Q

P

C

j

z

X

H

G

O

d

5

f

K

v

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

3

u

E

U

w

R

2

+

g

D

M

T

m

W

/

k

V

>

A

B

n

c

N

S

8

J

r

q

9

L

0

F

7

Y

o

y

p

Z

I

Q

P

C

j

z

X

H

G

O

d

5

f

K

v

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

3

u

E

U

w

R

2

+

g

D

M

T

m

W

/

k

V

>

A

B

n

c

N

S

8

J

r

q

9

L

0

F

7

Y

o

y

p

Z

I

Q

P

C

j

z

X

H

G

O

d

5

f

K

v

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

P

+

L

r

U

f

2

d

3

Z

q

Q

v

E

K

M

X

y

w

>

A

B

c

D

N

S

g

F

I

8

V

u

C

o

z

n

p

H

O

5

R

k

m

0

9

J

W

/

j

7

T

Y

G

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

F

2

f

C

n

U

k

I

X

K

Z

W

E

r

p

8

>

A

B

3

c

D

N

S

g

M

v

L

9

q

o

J

V

m

O

y

d

R

H

0

+

T

P

w

5

7

Y

/

u

G

j

Q

z

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

F

2

f

C

n

U

k

I

X

K

Z

W

E

r

p

8

>

A

B

3

c

D

N

S

g

M

v

L

9

q

o

J

V

m

O

y

d

R

H

0

+

T

P

w

5

7

Y

/

u

G

j

Q

z

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

y

N

P

z

3

o

j

c

7

E

T

Y

H

U

m

u

n

J

q

M

>

A

B

V

S

8

r

/

9

L

0

W

F

+

K

v

p

2

d

C

f

I

X

D

R

5

Q

g

k

G

O

Z

w

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

3

u

E

U

w

R

2

+

g

D

M

T

m

W

/

k

V

>

A

B

n

c

N

S

8

J

r

q

9

L

0

F

7

Y

o

y

p

Z

I

Q

P

C

j

z

X

H

G

O

d

5

f

K

v

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

3

u

E

U

w

R

2

+

g

D

M

T

m

W

/

k

V

>

A

B

n

c

N

S

8

J

r

q

9

L

0

F

7

Y

o

y

p

Z

I

Q

P

C

j

z

X

H

G

O

d

5

f

K

v

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

3

u

E

U

w

R

2

+

g

D

M

T

m

W

/

k

V

>

A

B

n

c

N

S

8

J

r

q

9

L

0

F

7

Y

o

y

p

Z

I

Q

P

C

j

z

X

H

G

O

d

5

f

K

v

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

3

u

E

U

w

R

2

+

g

D

M

T

m

W

/

k

V

>

A

B

n

c

N

S

8

J

r

q

9

L

0

F

7

Y

o

y

p

Z

I

Q

P

C

j

z

X

H

G

O

d

5

f

K

v

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

3

u

E

U

w

R

2

+

g

D

M

T

m

W

/

k

V

>

A

B

n

c

N

S

8

J

r

q

9

L

0

F

7

Y

o

y

p

Z

I

Q

P

C

j

z

X

H

G

O

d

5

f

K

v

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

3

u

E

U

w

R

2

+

g

D

M

T

m

W

/

k

V

>

A

B

n

c

N

S

8

J

r

q

9

L

0

F

7

Y

o

y

p

Z

I

Q

P

C

j

z

X

H

G

O

d

5

f

K

v

first loss term (details shown in appendix). By the Bayes rule, Ps(x) Pt(x) = τsP(t) τtP(s) , where P(t) and P(s) are the amount of unlabeled data from each domain during the training process. Since we use the same amount of source and target data in each batch, they are canceled out by following P(s) P(t) = 1. Then f in Theorem 1 is reduced to Eq. 8. Therefore, besides the binary classification loss, the parameter wd of the discriminative network is also trained with gradients from the first loss term. Algorithm 1 shows the details.

Applications to UDA and SSL: We show how to incorporate end-to-end DRL within a framework that takes unlabeled data for training. We introduce the general setting with two examples: self-training based UDA and crossdomain SSL, followed by a new self-training algorithm DRST (distributionally robust self-training) and a new semisupervised learning algorithm DRSSL (distributionally robust semi-supervised learning).

General settings: In many cases, the source domain may have abundant labels but the target domain lacks enough labels. Typical problems under this setting include unsupervised domain adaptation and semi-supervised learning. In both cases, a common strategy is to treat the prediction results on the target data as pseudo labels to train the model on target data. In these methods, model confidence (softmax output) is often leveraged as the proxy to rank the reliability of pseudo labels, with the underlying assumption that there exists a positive correlation between model confidence and pseudo label quality. However, such an assumption requires accurate uncertainty estimation to avoid false usage of wrong pseudo labels which may poison the training. To this end, we incorporate DRL into these frameworks in order to provide calibrated uncertainties. An illustration of this setting is shown in Fig. 2.

Distributionally robust self-training: In UDA, we are given labeled source data and unlabeled target data and aim to achieve adaptation from the source to the target domain. Self-training is an effective method for UDA, where the training procedure in Fig. 2 is conducted multiple times. Here we propose DRST to plug the class-regularized DRL model into self-training. The idea is to regard each training epoch as a new domain shift problem in DRL. After each training epoch, we make predictions on the target domain and select more

Algorithm 1 End-to-end Training for DRL

• 1: Input: DNN ϕ and DNN τ, with optimizer SGD1 and SGD2, respectively. Learning rates γ1 and γ2, epoch number T . 2: Initialization: ϕ, τ ← random initialization, epoch← 0 3: While epoch < T 4: For each data mini-batch 5: Update τ by SGD1(γ1) using the combined gradients from both loss terms in Eq. 7; 6: Compute f using θ, wr, and wd; 7: Update ϕ, θ by SGD2(γ2) using derived gradients; 8: epoch ← epoch +1 9: Output: Trained networks ϕ, τ .

confident data and generate pseudo-labels for them to merge into the source training data. Both the pseudo-labels and the model confidence are achieved from Theorem 1. Then the labeled source data and the newly pseudo-labeled target data become the new source set for the next training epoch.

Distributionally robust semi-supervised learning: In cross-domain SSL where there is little labeled source data and much unlabeled target data, we aim to utilize unlabeled data in the target domain to help representation learning and save the effort needed for labeling. One effective strategy is to use pseudo-labels generated from weakly-augmented data to supervise strongly-augmented data. Here 'weakly' means simple flip-and-shift data augmentation while 'strongly' follows the same strategy as FixMatch [Sohn et al., 2020]. Using DRL's prediction, we propose DRSSL, which assigns pseudo-labels more conservatively. Practically, we plug DRL into Fixmatch. Here the unsupervised loss in Fig. 2 is the 'consistency loss' in FixMatch: Lu = 1M ∑Mm=1 I(max( ˆ P (yw t |xwt )) > η)H(ˆ yw t , ˆ P (ys t |xst )), where xwt and yw t represent the weakly-augmented target data, xst and ys t represent the strongly-augmented version of the same image data, and η is a threshold for generating pseudo-labels ˆ yw t .

4 Experiments

We evaluate our method on benchmark datasets and compare our performance with other uncertainty quantification, calibration, and domain adaptation baseline methods. DRL is evaluated as a method providing more calibrated uncertainties (Sec. 4.1), DRST as an effective UDA method (Sec. 4.2) and DRSSL as a cross-domain SSL method. We show additional results and details in the appendix.

Datasets and methods: We use Office31 [Saenko et al., 2010], Office-Home [Venkateswara et al., 2017] and VisDA2017 [Peng et al., 2017] for evaluating DRL's uncertainties. We compare DRL with temperature scaling (TS), VADA [Shu et al., 2018] and source-only. We also train models using ImageNet [Deng et al., 2009] as the source domain and ImageNetV2 [Recht et al., 2019] as the target domain to check the relationship between our estimated weights and the human selection frequencies (HSF) [Chen et al., 2020a].

VisDA2017 is also used to evaluate UDA performance. We compare with (1) traditional UDA baselines: MCD [Saito et al., 2018b] and ADR [Saito et al., 2018a]; (2) self-training baselines: CBST [Zou et al., 2018] and CRST [Zou et al., 2019]; (3) methods that tackle domain adaptation with uncertainty: BRER [Han et al., 2019] and MUDA [Lee and Lee, 2020]; (4) uncertainty quantification methods combined with self-training: AVH [Chen et al., 2020a]+CBST.In addition, we use CIFAR10, STL10 [Coates et al., 2011], MNIST [Lecun and Bottou, 1998] and SVHN [Netzer et al., 2011] to construct cross-domain SSL settings, which has few source labeled data and much unlabeled target data and show DRSSL's advantages in cross-domain SSL over Fixmatch [Sohn et al., 2020].

Table 2: Expected calibration error (ECE) comparison on ImageNetV2 under different HSF.

HSF	Source	Temp.Scal.	DRL (Ours)
[0.0, 0.2]	0.2694	0.2624	0.0129
[0.2, 0.4]	0.1818	0.1745	0.0036
[0.4, 0.6]	0.1344	0.1281	0.0012
[0.6, 0.8]	0.0667	0.0601	0.0019
[0.8, 1.0]	0.0319	0.0246	0.0019

Evaluation metrics: Apart from accuracy, we also use Brier score [Brier, 1950], expected calibration error (ECE) [Guo et al., 2017], and reliability plots [Guo et al., 2017] to evaluate the performance of our proposed method and the baselines. Brier score measures the mean squared difference between the predicted probability assigned to the possible outcome and the actual outcome. ECE is defined as the sum of average difference between prediction accuracy and confidence of different confidence bins (we use 15 bins in practice). Despite the potential problems of ECE [Nixon et al., 2019], it is still the most prevalent metric for the top-1

prediction. For both the Brier score and ECE, the lower the score, the more calibrated is the model.

Experimental setup: For Office31 and Office-Home tasks, we use ResNet50 [He et al., 2016] as the backbone for all models. We train with SGD for 100 epochs and set the learning rate to 0.001. For VisDA, we use ResNet101 and SGD optimizer. During the 20 epochs of training, the initial learning rate is set as 10-5 and the weight decay parameter is set as 5×10-4 . For ImageNet, we follow the standard training process of AlexNet [Krizhevsky et al., 2012] and VGG-19 [Simonyan and Zisserman, 2014], where the initial learning rate is 0.01 and we decay the learning rate by a factor of 10 for every 30 epochs. All of the training are done on DGX V100 Tesla V100 GPUs with 32GB memory. The main packages and corresponding versions are: PyTorch 0.4.0, CUDA 10.1.

4.1 Calibrated Uncertainties fromDRL

DRL's calibrated confidence: Fig. 3 shows our method achieves better uncertainty calibration. Note that the type of uncertainty we measure here is the epistemic uncertainty [Gal and Ghahramani, 2016], caused by the lack of data and domain gaps. The aleatoric uncertainty is caused by noise inherent in data and is out of our scope. DRL tends to be underconfident and conservative but still stays closer to the calibration line (dashed line). Table 2 shows our method obtains better ECE on ImageNetV2 in different human selection frequency (HSF) bins. HSF is defined as the average number of times an image from a given class is classified correctly by a group of annotators. Say there are n annotators and an image with ground-truth label "car", p annotators recognize the image as "car", then the image's HSF is pn . The lower the HSF of an image, the harder for humans to correctly classify it. Appendix provides additional results and shows that the accuracy of DRL is also competitive.

Density ratio v.s. HSF: Fig. 4 shows that images with low HSF (visually harder ones) have smaller density ratios, indicating that the estimated density ratios are positively correlated with HSF on ImageNetV2 under different network architectures. Here we regard the value of HSF from [0, 0.2] as low and (0.2, 1] as high.

4.2 Unsupervised Domain Adaptation

Initialization with ASG: The pretrained models impact performance. Automated synthetic-to-real generalization (ASG) model [Chen et al., 2020b], which uses ImageNet pretrained knowledge, can be used to improve synthetic training and self-training. Hence, we use the ASG pretrained model as initialization, with results in Table 3's last four rows.

Accuracy and calibration: Fig. 5(a) and Table 3 show that DRST performs best in accuracy. Our vanilla version of DRST outperforms CRST by over 5% with ASG initialization. We improve the SOTA self-training accuracy on VisDA by over 1%. In the rest of the results, we use DRST to represent DRST-ASG. In comparison with uncertainty based methods (not shown in the table), BRER achieves an accuracy of 80.59±1.39% while ours is 83.75% using the same source model. MUDA aims to minimize model uncertainty and only yields a result of 78.5%. We only identified two methods with higher accuracy on VisDA2017: CDTrans [Xu et al., 2021] (using transformer) and CPGA [Qiu et al., 2021] (two stage learning by calculating prototypes). Both of them cannot estimate uncertainty and it is nontrivial for them to incorporate uncertainty or pseudo-label modeling. Fig. 5(b) shows that DRST also achieves better calibrated performance.

Ablation study: Fig. 5(a)(b) include two ablation methods. In the first ablation, we set r to 0 so that there is no class regularization in DRL ("r = 0"). The prediction then follows the form in Eq. 1. In the second ablation, we set the density ratios to 1 instead of calculating it to mute the differentiable density ratio estimation in our method so that there is no representation level conservativeness ("R = 1"). When one method is existent, the performance is not degraded but also not boosted. However, DRST achieves the best results when both components are present, showing faster convergence and better performance.

Covariate shift: Fig. 5(c) shows how well the covariate shift assumption holds over the training process. We calculate Ps(ϕ(x))/Pt(ϕ(x)) - Ps(ϕ(x), y)/Pt(ϕ(x), y) using discriminative density ratio estimators (per class) as a proxy of covariate shift as it becomes 0 when covariate shift holds. We can see that the gap decreases with self-training, showing that even though covariate shift may not hold in the beginning, self-training helps to promote this assumption with better aligned domains and more discriminative feature distributions. Note that our model is tailored and effective to covariate shift no matter other shifts (e.g. label shift) exist or not. For example, we find label shift exists in VisDA2017 and our method still shows effectiveness: we calculate the division of the number of source samples over the number of target samples for each class, the numbers range from 1.42 to 4.23, indicating the existence of label shift in the dataset. Seeing the co-existence of multiple shifts and especially covariate shift, we believe our method is practical. This in fact has also well been verified by our real-world image experiments beyond covariate shift.

Table 3: Accuracy comparison with different UDA and self-training methods on VisDA2017. "Skate" denotes "Skateboard".

Method	Aero	Bike	Bus	Car	Horse	Knife	Motor	Person	Plant	Skate	Train	Truck	Mean
Source [Saito et al., 2018a]	55.1	53.3	61.9	59.1	80.6	17.9	79.7	31.2	81.0	26.5	73.5	8.5	52.4
MCD [Saito et al., 2018b]	87.0	60.9	83.7	64.0	88.9	79.6	84.7	76.9	88.6	40.3	83.0	25.8	71.9
ADR [Saito et al., 2018a]	87.8	79.5	83.7	65.3	92.3	61.8	88.9	73.2	87.8	60.0	85.5	32.3	74.8
CBST [Zou et al., 2018]	87.2	78.8	56.5	55.4	85.1	79.2	83.8	77.7	82.8	88.8	69.0	72.0	76.4
CRST [Zou et al., 2019]	88.0	79.2	61.0	60.0	87.5	81.4	86.3	78.8	85.6	86.6	73.9	68.8	78.1
CBST-AVH [Chen et al., 2020a]	93.3	80.2	78.9	60.9	88.4	89.7	88.9	79.6	89.5	86.8	81.5	60.0	81.5
DRST (Ours)	93.47	86.30	65.74	68.03	93.99	95.08	87.34	83.30	92.97	88.65	83.66	66.42	83.75
ASG [Chen et al., 2020b]	88.81	68.55	65.31	78.06	95.78	9.11	84.89	29.58	82.13	33.76	86.00	12.04	61.17
CBST-ASG [Chen et al., 2020b]	95.12	86.53	79.83	76.01	94.61	92.34	85.94	75.08	89.23	82.16	73.42	56.49	82.23
CRST-ASG [Chen et al., 2020b]	92.38	81.30	74.63	84.40	90.90	92.43	91.65	83.78	94.92	88.12	74.88	61.10	84.21
DRST-ASG (Ours)	94.51	85.58	76.50	77.18	94.39	95.33	88.89	81.23	94.22	90.36	81.75	63.10	85.25

Table 4: Accuracy of cross-domain SSL in comparison with the peer method Fixmatch [Sohn et al., 2020].

Dataset	CIFAR10	STL10	MNIST	SVHN
Fixmatch	91.60	59.61	99.43	26.50
DRSSL (Ours)	95.17	69.38	99.46	30.96

Improvement on hard examples: Fig. 6 demonstrates that compared to the baselines, DRST achieves larger performance gain on target samples with smaller density ratios. Recall that data is not well-represented in the source domain and is regarded as visually hard when the density ratio Ps(x)/Pt(x) is small (Fig. 1(b)). Therefore, DRST provides more robust performance on harder examples.

Density ratios: Our density ratios are estimated from a differentiable domain classifier and is not guaranteed to match the true density ratios. However, our density ratios are interpretable as they reflect the closeness of a sample to the two domains and benefit the downstream tasks. In Fig. 1(b), a harder example obtains a lower density ratio due to its vague shape (more examples are in the appendix). Moreover, the magnitude of our estimated density ratios are modest in general within the range of [0.1, 10] (instead of approaching 0 or infinity) due to the regularization by the target task classification network's learning signals.

Improved attention: We visualize the model attention of DRST using Grad-CAM [Selvaraju et al., 2017] and compare with the CBST and CRST baselines in the appendix. The results show that DRST renders improved attention with better object coverage, presenting a better reception field.

4.3 Distributionally Robust Cross-Domain SSL

To introduce domain gaps into semi-supervised learning, we choose the emerging cross-domain semi-supervised learning (CDSSL) [Yu et al., 2019] setting. Specifically, CDSSL aims to use few labeled source training examples and many unlabeled target training examples to predict on source testing examples and target testing examples. Two pairs of source and target domains are used: Source: CIFAR10 / Target: STL10 and Source: MNIST / Target: SVHN. Under the single domain setting, FixMatch trains the model using source labeled data and source unlabeled data. However, under the crossdomain setting, we train the model using source labeled data and target unlabeled data.

Table 4 shows that DRL-powered SSL method improves the Fixmatch baseline significantly on CDSSL tasks. Note that our setting is different from UDA where source labeled data is abundant. SSL focuses on using unlabeled augmented data to learn from few labeled data. In the CIFAR10 to STL10 case, we only have 4k labeled source data. For MNIST to SVHN, we have 40k labeled source data. The results show that DRL is beneficial for generating high-quality pseudo supervision for unlabeled data under the cross-domain SSL setting. More details of the experiments are provided in the appendix.

5 Conclusion

This paper studied uncertainty estimation under distribution shift with the distributionally robust learning framework. We show that density estimation can be integrated into the learning process by using a domain classifier. We propose differentiable density ratio estimation and develop end-to-end training techniques for our method. Using DRL's more calibrated model confidence helps to generate better pseudolabels for self-training in UDA and cross-domain SSL.

We also empirically show that the density ratios learned from our domain classifier reflect the hardness of an image, showing a positive correlation with the human selection frequencies. Future work involves relaxing the assumptions made and study different shifts in the DRL framework, which is in fact a limitation of our work.

Acknowledgments

Haoxuan Wang started the work during the Summer Undergraduate Research Fellowship Program in Caltech. Anqi Liu was supported by the PIMCO and Amazon Postdoc Fellowship in Caltech and the Amazon-JHU AI2AI faculty award in JHU. Junchi Yan was supported by NSFC 62222607.