On Attacking Out-Domain Uncertainty Estimation in Deep Neural Networks

1 Introduction

Deep neural networks (DNNs) have been thriving in various applications, such as computer vision, natural language processing and decision making. However, in many applications with real-world consequences (e.g. autonomous driving, disease diagnosis, loan granting), it is not sufficient to merely pursue the high accuracy of the predictions made by the AI models, since the deterministic wrong predictions without any uncertainty justification could lead to catastrophic consequences [Galil and El-Yaniv, 2021]. Therefore, to address the issue of producing over-confident wrong predictions from DNNs, great efforts have been made to quantify the predictive uncertainty of the models, so that ambiguous or low-confident predictions could be rejected or deferred to an expert.

Indeed, state-of-the-art algorithms for uncertainty estimation in DNNs have shown their impressive performance in terms of quantifying the confidence of the model predictions. Handling either in-domain data (generated from the training distribution) or out-domain data under domain shift, these algorithms could successfully assign low confidence scores for ambiguous predictions. However, the robustness of such estimated uncertainty/confidence is barely studied. The robustness of uncertainty estimation in this paper is defined as: to which extent would the predictive confidence be affected when the input is deliberately perturbed? Consider an example of autonomous driving [Feng et al., 2018], where the visual system of a self-driving car is trained using collected road scene images. When an extreme weather occurs, to avoid over-confident wrong decisions, the visual system would show high uncertainty for the images captured by the sensors, since the road scenes observed by the sensors under the extreme weather (out-domain) could be drastically different from the training scenes (in-domain). However, a malicious attacker might attempt to perturb the images captured by the sensors in such a way, that the visual system would regard the out-domain images as in-domain images, and make completely wrong decisions with a high confidence, leading to catastrophic consequences.

∗Contact Author

Therefore, with the intention of raising the attention of the research community to systematically investigate the robustness of the uncertainty estimation, we show that SoTA DNNbased uncertainty estimation algorithms could fail easily under our proposed adversarial attack. In particular, we focus on attacking the uncertainty estimation for out-domain data in an image classification problem. That is, under our attack, the uncertainty estimation models would be fooled to assign extremely high confidence scores for the out-domain images, which originally would have been rejected by these models due to the low confidence scores (e.g. Softmax score [Galil and El-Yaniv, 2021]). As shown in Figure 1, the attacker will perturb the out-domain images into the high-confidence region of the victim system. To show vulnerability of the SoTA DNN based uncertainty estimation algorithms under our threat model, we launched our proposed out-domain adversarial attack on various algorithms, including Deep Ensemble [Lakshminarayanan et al., 2016], RBF-based Deterministic Uncertainty Quantification (DUQ) [Van Amersfoort et al., 2020], Gaussian process based Deterministic Uncertainty Estimation (DUE) [van Amersfoort et al., 2021] and Spectral-Normalized Gaussian Process (SNGP) [Liu et al.,

2020]. Extensive experimental results in various benchmark image datasets show that the uncertainty estimated by these algorithms could be drastically corrupted under our attack.

2 RelatedWork

Uncertainty Estimation inDNNs. A significant amount of algorithms have been proposed to quantify the uncertainty in DNNs [Lakshminarayanan et al., 2016; van Amersfoort et al., 2021; Van Amersfoort et al., 2020; Liu et al., 2020; Alarab and Prakoonwit, 2021; Gal and Ghahramani, 2016]. For instance, in [Lakshminarayanan et al., 2016], a set of neural networks were trained to construct an ensemble model for uncertainty estimation using the training data. In [Van Amersfoort et al., 2020], RBF (radial basis function) kernel was used to quantify the uncertainty expressed by the deep neural networks. In addition, a further thread of studies built their algorithm based on the Gaussian process (GP), which has been proved to a powerful tool for uncertainty estimation theoretically. However, GP suffers from low expressive power, resulting in poor model accuracy [Bradshaw et al., 2017]. Therefore, based on [Bradshaw et al., 2017], [van Amersfoort et al., 2021] proposed to regularize the feature extractor, so that the deep kernel learning could be stabilized. Similarly, in [Liu et al., 2020], the predictive uncertainty is computed with a spectral-normalized feature extractor and a Gaussian process. However, the robustness of these stateof-the-art algorithms is not well studied. As shown in this paper, the uncertain DNN models trained using these algorithms could be fooled easily under our attack.

Adversarial Examples. Despite the impressive performance on clean data, it has been shown that DNNs could be extremely vulnerable to adversarial examples [Szegedy et al., 2013; Goodfellow et al., 2014; Kannan et al., 2018; Carlini and Wagner, 2017]. That is, deep classifiers could be fooled to make completely wrong predictions for the input images that are deliberately perturbed. However, our proposed adversarial attack is different from the traditional adversarial examples in two aspects. Firstly, traditional adversarial examples are crafted by the attacker to corrupt the accuracy of classifiers, whereas our threat model is aimed at attacking uncertainty estimation in DNNs and increasing its confidence of the wrong predictions. Moreover, the traditional adversarial attack is defined under the close-world assumption [Reiter, 1981; Han et al., 2021], where all possible predictive categories are covered by the training data. The in-domain property limits the power of adversary, since all possible perturbing directions could be simulated by performing untargeted adversarial training [Carlini and Wagner, 2017]. In comparison, our proposed attack is out-domain: perturbing the out-domain data into the in-domain data distribution that the uncertainty estimation model is trained to fit. As shown in our experiments, traditional in-domain adversarial defense mechanism could not withstand our attack: even adversarially trained models could be fooled by our outdomain adversarial examples.

Adversarial Attacks on Uncertainty Estimation. The idea of attacking uncertainty estimation in DNNs have been introduced in [Galil and El-Yaniv, 2021] for the first time. In [Galil and El-Yaniv, 2021], the authors tried to manipulate the confidence score over the predictions by perturbing the input images. By heuristically moving the correctly classified input images towards the decision boundary of the model, the attacker could reduce the confidence of the model on the correct predictions. However, the threat model in [Galil and ElYaniv, 2021] could only attack in-domain images, whereas our attack is formulated for an out-domain scenario.

3 Problem Statement

In this section, we first define a set of key concepts and present the notations for our problem. Then, we show the mathematical formulation of our proposed attack.

3.1 Key Concept Definition

Definition 1 (Data Domain). We define two data domains in this paper, namely PIN (representing in-domain distribution) and POUT (for out-domain distribution).

An excellent uncertainty estimation model is expected to make high-confident predictions for the in-domain data, but produce low-confident predictions for out-domain data.

Definition 2 (Dataset). In our problem, the training set Dtrain consists of images sampled from PIN: Dtrain = {(x, y)|(x, y) ∼ PIN}, where x refers to input images and y represents the ground truth labels for x. The test set contains images sampled from POUT, since we mainly evaluate our attack on out-domain data: Dtest = {x|x ∼ POUT}.

Definition 3 (Uncertainty Estimation). Most uncertainty estimation models fθ are defined as a mapping from the input image x to the predictive confidence distributed over all possible categories ˆ p(y): fθ : x → ˆ p(y) ∈ RC , where C is the total number of classes of the in-domain data.

Regarding ˆ p(y), an ideal uncertainty estimation model will assign a uniform confidence over all classes for the outdomain data, since it would be too uncertain to make any predictions but random guess. Moreover, dependent on uncertainty estimation algorithm, ˆ p(y) could have different physical implications. For instance, in Deep Ensemble, ˆ p(y) is the Softmax uncertainty score. In DUQ, ˆ p(y) is interpreted as the distances between the test data point and the centroids.

Definition 4 (EvaluationMetrics). In terms of evaluating the model's uncertainty on out-domain data, we use two metrics: Entropy H(fθ):

Rejection Rate R(fθ):

where τ is the pre-defined confidence threshold, and a prediction will be rejected by the model if the predictive confidence over all categories could not achieve the threshold.

Entropy H(fθ) measures the averaged uncertainty of a random variable's possible outcomes. Therefore, an ideal uncertainty estimation model will show a high entropy over all test (out-domain) data, whereas our attacker will increase the model's confidence for the wrong predictions, resulting in a reduced entropy. Rejection rate R(fθ) measures the fraction of the rejected test samples over the test set. Given the test set and a threshold τ, a reasonable uncertain model should show a high rejection rate as it is too uncertain to make any predictions for the out-domain data.

Note that in Equation 1 and Equation 2, it is required that all ˆ p(yi) ∈ ˆ p(y) are probabilities. Therefore, for the uncertain models which do not output a probability distribution, their outputs will be normalized into probabilities for evaluation in our experiments. For instance, since DUE outputs the RBF-based distances, we normalize all distances into a categorical distribution (More details in Section 5), but Deep Ensemble uses averaged predicted probabilities directly [Lakshminarayanan et al., 2016], meaning that there is no need for normalization for Deep Ensemble.

3.2 ProblemFormulation

Threat Model. Our threat model is that the adversarial attacker tries to perturb any out-domain test image x ∼ POUT into x′ , so that the uncertainty estimation model will believe that x′ is in-domain data and produce a high-confident prediction for it. We assume a white-box attack [Goodfellow et al., 2014; Galil and El-Yaniv, 2021]. Moreover, we require that x′ must be visually innocuous, indicating that x′ and x must be geometrically close to each other.

To attack the uncertainty estimation on a specific outdomain test sample x, we need to solve:

In Equation 3, τ represents the confidence threshold defined for Equation 2. ϵ is a small non-negative scalar, bounding the norm of the adversarial perturbation, so that the perturbed image is visually innocuous. Under this formulation, the optimal adversarial perturbation could fool the uncertain model to make predictions with high-confidence (higher than the rejection threshold) without being visually detectable.

4 Algorithm

To successfully perturb out-domain test samples, our objective function (Equation 3) must be solved rationally. However, due to the non-convexity of fθ and non-differentiabiliy of argmax, it is intractable to compute a closed-form solution for Equation 3. Therefore, instead of solving Equation 3 analytically, we propose to approximate optimal out-domain adversarial examples using a gradient-based method.

To begin with, we replace the optimization objective of Equation 3 with a new differentiable objective. That is, for an out-doman image x, the attacker approximates its optimal perturbation by solving:

where ˆ y is the heuristically found closest attacked label. More precisely, ˆ y is the original prediction produced by the uncertainty estimation model for the clean out-domain image.

Note that it is determined by Equation 5 that our proposed attack is untargeted. That being said, the adversary only moves the out-domain data to its closest data manifold instead of targeted ones. This untargeted formulation could guarantee the

Algorithm 1 Perturbing out-domain data

Input: uncertainty estimation model fθ, a clean test sample x ∼ Dtest

Parameter: adversarial radius ϵ, number of iterations for the attack K, step size η at each iteration

Output: the adversarial test sample x′

• 1: Initialize the iteration counter k = 0
• 2: Initialize the adversarial example x′ = x
• 3: while k ≤ K do
• 4: ˆ y = argmaxy ˆ p(y)
• 5: l(fθ, x′ , ˆ y) = l(fθ(x′ ), ˆ y)
• 6: x′ = ∏B(x,ϵ) (x′ -ηsign∇x′ l(fθ, x′ , ˆ y)), where ∏ is the projection operator
• 7: end while
• 8: return x′

efficiency of the adversary: the path, over which the clean data is moved, could be much shorter compared to targeted attacks, since the targeted data manifolds could be further.

The concrete procedure of generating an adversarial example for a out-domain sample x is presented in Algorithm 1. In Algorithm 1, Line 3 to Line 7 demonstrate how the optimal adversarial example (i.e. the minimizor of Equation 4) will be approximated using projected gradient descent. In particular, Line 4 corresponds to Equation 5, namely the finding of the attacked label ˆ y. Line 5 to Line 6 demonstrate how the adversarial example is updated using gradient of the loss evaluated at current steps. The projection operation ∏ guarantees that the norm of the adversarial perturbation would not exceed the adversarial radius ϵ. After applying Algorithm 1 to an outdomain sample x, the perturbed sample x′ will be predicted by the uncertain model fθ as ˆ y with a high confidence.

Finally, we would like to comment on the differentiability of our formulation for attacking out-domain uncertainty estimation. Firstly, recall the reason we introduce Equation 4 to replace Equation 3 is that the operation argmax in Equation 3 is not differentiable. Using a differentiable surrogate loss function l (e.g. cross-entropy, variational ELBO), the gradient could be computed for the output layer of the uncertain model at the first step. However, different uncertainty estimation algorithms could be non-differentiable by design. For instance, in Gaussian process based algorithms (e.g. DUE [van Amersfoort et al., 2021]), Monte Carlo sampling and other non-differentiable computation are usually employed to compute the final predictive uncertainty distribution. Therefore, to address such non-differentiable issues, we follow the exact implementation in the papers of DUE to study the feasibility of our proposed attack. For DUE [van Amersfoort et al., 2021], the reparametrization trick is implemented as in [van Amersfoort et al., 2021] to make the generation of the adversarial perturbation differentiable. As for the Gaussian process in SNGP [Liu et al., 2020], we follow the training implementation presented in [Liu et al., 2020] to guarantee the differentiability of the uncertain model. Regarding Deep Ensemble and DUQ, both of them are deterministic and differentiable, hence our attack could be launched directly.

5 Experiments

We evaluate the efficacy of our proposed out-domain uncertainty attack by assessing to which extent, the victim model could be deceived to make high-confident predictions for perturbed out-domain data. From a technical point of view, the victim models are trained to learn the in-domain data distribution using the in-domain training set Dtrain (as defined in Section 3), but are tested on perturbed out-domain data.

5.1 Dataset and Experimental Setup

Dataset. Following the experimental design presented in [Lakshminarayanan et al., 2016; Van Amersfoort et al., 2020; van Amersfoort et al., 2021] we use MNIST [LeCun et al., 1998] vs. NotMNIST 1 , and CIFAR-10 [Krizhevsky and Hinton, 2009] vs.SVHN [Netzer et al., 2011] to test the efficacy of our proposed attack on corrupting the out-domain uncertainty estimation algorithms in DNNs. MNIST dataset contains 0-9 handwritten digit images, and NotMNIST dataset contains images for letters A-J (10 classes) taken from different fonts. CIFAR10 is a dataset containing images of common objects of 10 kinds in the real-world (airplane, automobile, dogs, e.t.c.), whereas SVHN consists of colored digit images taken from real-world scenes. Based on the description above, it is clear that MNIST vs. NotMNIST and CIFAR10 vs. SVHN represent drastically different data domains.

Experimental Setup. 2 In many out-domain uncertainty estimation studies [Lakshminarayanan et al., 2016; Van Amersfoort et al., 2020; van Amersfoort et al., 2021], it is common to train uncertainty estimation models with indomain data, and then evaluate the predictive confidence returned by the model over the out-domain test data. Therefore, without loss of generality, we also follow the experimental setup for out-domain uncertainty estimation in [Lakshminarayanan et al., 2016; Van Amersfoort et al., 2020; van Amersfoort et al., 2021]. There are two sets of experiments in our work. In the first set of experiment, the uncertain models are trained using MNIST (in-domain), and tested on NotMNIST (out-domain). In the second set of experiments, CIFAR10 is used as the in-domain training data, and the resulted model is tested on out-domian SVHN data.

5.2 Baselines

BaselineDNNArchitectures. To efficientlymodel the data distribution and avoid overfitting, we select one commonly used neural network architecture for two sets of experiments respectively. In particular, for MNIST vs. NotMNIST, LeNet-5 [LeCun et al., 1998] is used as the baseline architecture to build uncertainty estimation models. As for CIFAR10 vs. SVHN, we use ResNet-18 [He et al., 2016]. Note that the baseline architectures are not equivalent to the final uncertainty estimation models. For instance, in a Deep Ensemble model, there are multiple ensemble classifiers. For DUQ, DUE and SNGP, the baseline models only work as feature extractors, after which there could still be further computational modules, such as RBF kernels or Gaussian process.

1 dataset link: http://yaroslavvb.blogspot.co.uk/2011/09/notmnistdataset.html

2 Code will be released after acceptance of this paper.

Baseline Uncertainty Estimation Algorithms. To show the feasibility and efficacy of our proposed attack algorithm, 4 uncertainty estimation algorithms are included in our experiments, namely Deep Ensemble [Lakshminarayanan et al., 2016], DUQ [Van Amersfoort et al., 2020], DUE [van Amersfoort et al., 2021] and SNGP [Liu et al., 2020]. For all experiments, the key hyperparameters for our method and all baseline algorithms are tuned to achieve their best performance for a fair comparison. In summary, we have following trained models under different training schemes:

• Deep Ensemble: The vanilla deep ensemble for uncertainty estimation [Lakshminarayanan et al., 2016].
• Deep Ensemble Adv.: The augmented version of deep ensemble using adversarially trained ensemble networks [Lakshminarayanan et al., 2016].
• DUQ: Uncertainty estimation using RBF kernels [Van Amersfoort et al., 2020].
• DUE: Uncertainty estimation using Gaussian process and variational ELBO [van Amersfoort et al., 2021].
• SNGP: Spectral-normalized Gaussian process for uncertainty estimation [Liu et al., 2020].

Note that there exists another augmented variant of Deep Ensemble. That is, the augmented deep ensemble model could be built upon a set of adversarially trained ensemble networks. Following the configuration in the original paper of Deep Ensemble [Lakshminarayanan et al., 2016], we train the ensemble networks with fast gradient sign method (FGSM) instead of projected gradient descent (PGD). For a fair comparison, when evaluating deep ensemble models, we also modify our attack with iterations to a single-step attack. When performing adversarial training, the training radius ϵtr must be specified. In our experiments, two different training adversarial radius (ϵtr) are used. For MNIST vs. NotMNIST, adversarial training with ϵtr = 0.1 and ϵtr = 0.2 are performed, whereas for CIFAR10 vs. SVHN, the training adversarial radius ϵtr is set to 0.016 and 0.031, respectively.

5.3 Evaluation Results

Evaluating Efficacy. To beginwith, we conduct two sets of experiments to evaluate the efficacy of our proposed adversarial attack. The numerical results on entropy and rejection rate are reported in Table 1. Hclean and Rclean are the entropy and rejection rate computed using the clean, unperturbed outdomain data. In contrast, Hadv and Radv are evaluated using perturbed out-domain data. In both MNIST vs. NotMNIST and CIFAR10 vs. SVHN, the confidence threshold τ for computing Rclean and Radv is 0.9. That is, the uncertain model only makes predictions when the confidence score is greater than 0.9. Moreover, to attack NotMNIST images, the adversarial radius ϵ is set to 0.1, and ϵ = 0.016 for SVHN images.

From Table 1, we observe that our proposed attack could reduce both the entropy and rejection rate of different uncertain models significantly. This indicates that the uncertain models would be fooled by our attack to be over-confident about the wrong predictions they made for the out-domain data. For instance, regarding the first evaluation metric, the entropy on NotMNIST of SNGP is reduced from 1.22 to 0.23.

• (b) CIFAR10 vs. SVHN: 10-iteration adversarial attack with ϵ = 0.016, rejection rate computed with confidence threshold τ = 0.9.

Table 1: Evaluating the efficacy of proposed attack algorithm on MNIST vs. NotMNIST and CIFAR10 vs. SVHN.

As for the second evaluation metric, when making predictions for clean NotMNIST images, the DUQ model would reject 98% of the predictions due to the lack of confidence. However, when attacked by our algorithm, the rejection rate of DUQ would be reduced to 78%. Similar observations are made on all other baseline uncertain models in terms of both entropy and rejection rate. In other words, the uncertain models would be less uncertain and could fail to reject many predictions, which originally would have been rejected when there is no attack. As for CIFAR10 vs. SVHN, our attack still shows high efficacy in terms of increasing the uncertain models' confidence on the out-domain data according to Table 1b. For example, the DUQ model could achieve the rejection rate of 100% on clean SVHN images, but fails to reject 28% perturbed images. Moreover, when comparing Table 1a and Table 1b, we also notice that different uncertainty estimation algorithms show different robustness on different datasets. For instance, the uncertainty of DUE could be corrupted dramatically on MNIST vs. NotMNIST (the entropy is reduced from 0.83 to 0.09), but on CIFAR10 vs. SVHN, DUE shows a much stronger robustness (the entropy is reduced merely from 1.26 to 1.05). In contrast, DUQ is much more robust against our attack on MNIST vs. NotMNIST, but more vulnerable on CIFAR10 vs. SVHN.

To summarize, the consistent reduction of entropy and rejection rates on all uncertain models in Table 1 verifies that the state-of-the-art uncertainty estimation algorithms could be extremely vulnerable in the presence of adversarial attacks. Moreover, despite the vulnerability of all tested uncertainty algorithms, they also display different robustness against the attacks on different datasets.

Robustness Study. In addition to the efficacy evaluation of our proposed attack shown in Table 1b, we also conduct a set of robustness study to understand the relationship between the adversarial radius and attack effect. In particular, we changed the adversarial radius ϵ from 0.1 (correspond-

• (b) CIFAR10 vs. SVHN: 10-iteration adversarial attack with ϵ = 0.016,ϵ = 0.031,ϵ = 0.063 rejection rate computed with confidence threshold τ = 0.9.

Table 2: Evaluating the efficacy of proposed attack algorithm on MNIST vs. NotMNIST and CIFAR10 vs. SVHN.

ing to Hadv and Radv) to 0.2 (corresponding to H′adv and R′adv ) and 0.3 (corresponding to H′′ adv and R′′ adv ) for NotMNIST. Similarly, for SVHN, ϵ is changed from 0.016 to 0.031 and 0.063. As shown in Table 2a and in Table 2b, we observe that in general, with larger ϵ, the attack becomes more powerful and the uncertainty of the model would be further reduced (bold numbers). For instance, the entropy of DUQ on NotMNIST is consistently reduced from 1.02 to 0.80 and 0.72 with the increase of ϵ. However, an opposite trend is also observed, such as on SVHN, both entropy and rejection rate of the vanilla deep ensemble increase when ϵ becomes larger. We believe the optimal ϵ for attacking different dataset is at different ranges.

Visualization of Confidence Distribution. Due to the limited space, we randomly pick two of attacked SVHN images and show the shift of the confidence distribution after applying our attack. In Figure 2a, an adversarially train deep ensemble model is attacked using our proposed algorithm, and in Figure 2b, a DUE model is attacked. Note that both uncertain models are trained using CIFAR10 as in-domain data, and are tested on SVHN images. As we can observe, both uncertain models are fooled by the adversarial examples to make high-confidence wrong predictions for the out-domain data. In Figure 2b, the DUE model is only around 50% confident about the prediction when there is no attack. However, when the image is perturbed, the confidence is raised to higher than 90%. Originally, the uncertain prediction on the unperturbed images could have been rejected by model, and further deferred the image to an expert, but under our attack, the model will be extremely confident about the wrong predictions, and no further critical analysis would be performed.

Figure 2: The predictive confidence distribution over the in-domain categories.

6 Conclusion

With the intention of raising the attention on the topic of the robustness of uncertainty estimation, we investigated and designed a prototype white-box adversarial attacks on outdomain uncertainty estimation. As shown in our experiments, state-of-the-art uncertainty estimation algorithms could be deceived easily by our proposed adversary to make predictions for the out-domain images with very high confidence. In terms of the applications with real-world consequences, the vulnerability of the uncertain models could open a fatal loophole for attackers, leading to catastrophic consequences. Therefore, despite the accuracy of the uncertainty estimation, it is also important to improve the robustness of various uncertainty estimation algorithms in DNNs.

Ethics Statement

Adversarial attacks could cause extremely high threat to existing uncertainty estimation in deep neural networks. Despite the impressive performance on uncertainty estimation on out-domain data, state-of-the-art algorithms still fail catastrophically under the attack of imperceptible perturbations. The existence of such "out-domain" adversarial examples exposes a serious vulnerability in current uncertaintybased ML systems, such as autonomous driving, medical diagnosis and digital financial systems. In these applications with real-world consequences, the vulnerability of uncertainty systems could place our lives and security at risk.

Our work has the potential to inspire future studies on a new type of adversarial defense mechanism and the design of robust uncertainty estimation algorithms. Although in this work, we present a simple prototype adversarial attack on out-domain uncertainty estimation, the core intention of this work is to raise the attention of the research community to systematically investigate the robustness of the uncertainty estimation in DNNs. We believe our attack could be used to test future robust uncertainty estimation algorithms.

Acknowledgments

This research is supported in part by the National Science Foundation under Grant No. CHE-2105032, IIS-2008228, CNS-1845639, CNS-1831669. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation here on.