Towards Adversarially Robust Deep Image Denoising
Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence. Main Track. Pages 1516-1522.
https://doi.org/10.24963/ijcai.2022/211
Keywords
- Computer Vision: Adversarial learning, adversarial attack and defense methods
- AI Ethics, Trust, Fairness: Safety & Robustness
- AI Ethics, Trust, Fairness: Trustworthy AI
- Computer Vision: Applications
Abstract
This work systematically investigates the adversarial robustness of deep image denoisers (DIDs), i.e, how well DIDs can recover the ground truth from noisy observations degraded by adversarial perturbations. Firstly, to evaluate DIDs’ robustness, we propose a novel adversarial attack, namely Observation-based Zero-mean Attack (OBSATK), to craft adversarial zero-mean perturbations on given noisy images. We find that existing DIDs are vulnerable to the adversarial noise generated by OBSATK. Secondly, to robustify DIDs, we pro- pose an adversarial training strategy, hybrid adversarial training (HAT), that jointly trains DIDs with adversarial and non-adversarial noisy data to ensure that the reconstruction quality is high and the denoisers around non-adversarial data are locally smooth. The resultant DIDs can effectively remove various types of synthetic and adversarial noise. We also uncover that the robustness of DIDs benefits their generalization capability on unseen real-world noise. Indeed, HAT-trained DIDs can recover high-quality clean images from real-world noise even without training on real noisy data. Extensive experiments on benchmark datasets, including Set68, PolyU, and SIDD, corroborate the effectiveness of OBSATK and HAT.
- [Abdelhamed et al., 2018] Abdelrahman Abdelhamed, Stephen Lin, and Michael S. Brown. A High-Quality Denoising Dataset for Smartphone Cameras. In CVPR, 2018.
- [Anwar and Barnes, 2019] Saeed Anwar and Nick Barnes. Real Image Denoising with Feature Attention. In ICCV, 2019.
- [Batson and Royer, 2019] Joshua Batson and Loic Royer. Noise2Self: Blind Denoising by Self-Supervision. In ICML, 2019.
- [Carlini and Wagner, 2017] Nicholas Carlini and David Wagner. Towards Evaluating the Robustness of Neural Networks. arXiv, 2017.
- [Choi et al., 2019] Jun-Ho Choi, Huan Zhang, Jun-Hyuk Kim, Cho-Jui Hsieh, and Jong-Seok Lee. Evaluating Ro- bustness of Deep Image Super-Resolution against Adver- sarial Attacks. In ICCV, 2019.
- [Choi et al., 2021] Jun-Ho Choi, Huan Zhang, Jun-Hyuk Kim, Cho-Jui Hsieh, and Jong-Seok Lee. Deep Image Destruction: A Comprehensive Study on Vulnerability of Deep Image-to-Image Models against Adversarial At- tacks. arXiv, 2021.
- [Dabov et al., 2007] Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. Image De- noising by Sparse 3-D Transform-Domain Collaborative Filtering. IEEE TIP, 16, 2007.
- [Gao et al., 2019] Ruiqi Gao, Tianle Cai, Haochuan Li, Cho- Jui Hsieh, Liwei Wang, and Jason D. Lee. Convergence of Adversarial Training in Overparametrized Neural Net- works. In NeurIPS, 2019.
- [Gondara, 2016] Lovedeep Gondara. Medical image de- noising using convolutional denoising autoencoders. In ICDMW, 2016.
- [Goodfellow et al., 2015] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and Harness- ing Adversarial Examples. arXiv, 2015.
- [Guo et al., 2019] Shi Guo, Zifei Yan, Kai Zhang, Wang- meng Zuo, and Lei Zhang. Toward Convolutional Blind Denoising of Real Photographs. In CVPR, 2019.
- [Kurakin et al., 2017] Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. arXiv, 2017.
- [Lehtinen et al., 2018] Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, and Timo Aila. Noise2Noise: Learning Image Restoration without Clean Data. In ICML, 2018.
- [Madry et al., 2018] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards Deep Learning Models Resistant to Adversarial Attacks. In ICLR, 2018.
- [Shafahi et al., 2019] Ali Shafahi, Mahyar Najibi, Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S. Davis, Gavin Taylor, and Tom Goldstein. Ad- versarial Training for Free! In NeurIPS, 2019.
- [Szegedy et al., 2014] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In ICLR, 2014.
- [Tramer et al., 2020] Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On Adaptive Attacks to Adversarial Example Defenses. arXiv, 2020.
- [Ulyanov et al., 2018] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep Image Prior. In CVPR, 2018.
- [Xu et al., 2017] Jun Xu, Lei Zhang, David Zhang, and Xi- angchu Feng. Multi-channel Weighted Nuclear Norm Minimization for Real Color Image Denoising. In ICCV, 2017.
- [Xu et al., 2018] Jun Xu, Hui Li, Zhetong Liang, David Zhang, and Lei Zhang. Real-world Noisy Image Denois- ing: A New Benchmark. arXiv, 2018.
- [Yan et al., 2019] Hanshu Yan, Jiawei Du, Vincent Tan, and Jiashi Feng. On Robustness of Neural Ordinary Differen- tial Equations. In ICLR, 2019.
- [Yan et al., 2021] Hanshu Yan, Jingfeng Zhang, Gang Niu, Jiashi Feng, Vincent Tan, and Masashi Sugiyama. CIFS: Improving Adversarial Robustness of CNNs via Channel- wise Importance-based Feature Selection. In ICML, 2021.
- [Yan et al., 2022] Hanshu Yan, Jingfeng Zhang, Jiashi Feng, Masashi Sugiyama, and Vincent Y. F. Tan. Towards adver- sarially robust deep image denoising. arXiv, 2022.
- [Zhang et al., 2017] Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image De- noising. IEEE TIP, 2017.
- [Zhang et al., 2019a] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P. Xing, Laurent El Ghaoui, and Michael I. Jordan. Theoretically Principled Trade-off between Robustness and Accuracy. In ICML, 2019.
- [Zhang et al., 2019b] Yide Zhang, Yinhao Zhu, Evan Nichols, Qingfei Wang, Siyuan Zhang, Cody Smith, and Scott Howard. A Poisson gaussian Denoising Dataset with Real Fluorescence Microscopy Images. In CVPR, 2019.
- [Zhang et al., 2020a] Jingfeng Zhang, Xilie Xu, Bo Han, Gang Niu, Lizhen Cui, Masashi Sugiyama, and Mohan Kankanhalli. Attacks Which Do Not Kill Training Make Adversarial Learning Stronger. In ICML, 2020.
- [Zhang et al., 2020b] Jingfeng Zhang, Jianing Zhu, Gang Niu, Bo Han, Masashi Sugiyama, and Mohan Kankan- halli. Geometry-aware Instance-reweighted Adversarial Training. In ICLR, 2020.
References were extracted automatically from the PDF and may contain errors.
Full text extracted automatically from the PDF; figures, tables and equations may differ. The PDF is the version of record.
1 Introduction
Image denoising, which aims to reconstruct clean images from their noisy observations, is a vital part of the image processing systems. The noisy observations are usually modeled as the addition between ground-truth images and zeromean noise maps [Dabov et al., 2007; Zhang et al., 2017]. Recently, deep learning-based methods have made significant advancements in denoising tasks [Zhang et al., 2017; Anwar and Barnes, 2019] and have been applied in many areas including medical imaging [Gondara, 2016] and photography [Abdelhamed et al., 2018]. Despite the success of deep denoisers in recovering high-quality images from a certain type of noisy images, we still lack knowledge about their robustness against adversarial perturbations, which may cause severe safety hazards in high-stake applications like medical diagnosis. To address this problem, the first step should be developing attack methods dedicated for denoising to evaluate the robustness of denoisers. In contrast to the attacks for classification [Goodfellow et al., 2015; Madry et al., 2018], attacks for denoising should consider not only the adversarial budget but also some assumptions of natural noise, such as zero-mean, because certain perturbations, such as adding a constant value, do not necessarily result in visual artifacts. Although Choi et al. [2021; 2019] studied the vulnerability for various deep image processing models, they directly applied the attack from classification. To the best of our knowledge, no attacks are truly dedicated for the denoising task till now.
Please refer to the full-length paper [Yan et al., 2022] for the appendices, proofs, and codes.
To this end, we propose the observation-based zero-mean attack (OBSATK), which crafts a worst-case zero-mean perturbation for a noisy observation by maximizing the distance between the output and the ground-truth. To ensure that the perturbation satisfies the adversarial budget and the zeromean constraints, we utilize the classical projected-gradientdescent (PGD) [Madry et al., 2018] method for optimization, and develop a two-step operation to project the perturbation back into the feasible region. Specifically, in each iteration, we first project the perturbation onto the zero-mean hyperplane. Then, we linearly rescale the perturbation to adjust its norm to be less or equal to the adversarial budget. We examine the effectiveness of OBSATK on several benchmark datasets and find that deep image denoisers are indeed susceptible to OBSATK: the denoisers cannot remove adversarial noise completely and even yield atypical artifacts, as shown in Figure 2g.
To robustify deep denoisers against adversarial perturbations, we propose an effective adversarial training strategy, namely hybrid adversarial training (HAT), to train denoisers by using adversarially noisy images and non-adversarial noisy images together. The loss function of HAT consists of two terms. The first term ensures the reconstruction performance from common non-adversarial noisy images, and the second term ensures the reconstructions between nonadversarial and adversarial images to be close to each other. Thus, we can obtain denoisers that perform well on both nonadversarial noisy images and their adversarial perturbed versions. Extensive experiments on benchmark datasets verify the effectiveness of HAT.
Moreover, we reveal that adversarial robustness benefits the generalization capability to unseen types of noise, i.e., HAT can train denoisers for real-world noise removal only with synthetic noise sampled from common distributions like Gaussians. That is because OBSATK searches for the worstcase perturbations around different levels of noisy images, and training with adversarial data ensures the denoising performance on various types of noise. In contrast, other reasonable methods for real-world denoising [Guo et al., 2019; Lehtinen et al., 2018] mostly require a large number of realworld noisy data for the training, which are unfortunately not available in some applications like medical radiology. We conduct experiments on several real-world datasets. Numerical and visual results demonstrate the effectiveness of HAT for real-world noise removal.
In summary, there are three main contributions in this work: 1) We propose a novel attack, OBSATK, to generate adversarial examples for noisy observations, which facilitates the evaluation of the robustness of deep image denoisers. 2) We propose an effective adversarial training strategy, HAT, for robustifying deep image denoisers. 3) We build a connection between adversarial robustness and the generalization to unseen noise, and show that HAT serves as a promising framework for training generalizable deep image denoisers.
2 Notation and Background
Adversarial robustness and adversarial training Consider a deep neural network (DNN) {fθ : θ ∈ Θ} mapping an input y to a target x, the model is trained to minimize a certain loss function that is measured by particular distance d(·, ·) between output fθ(y) and the target x. In high stake applications, the DNN should resist small perturbations on the input data and map the perturbed input to a result close to the target. The notion of robustness has been proposed to measure the resistance of DNNs against the slight changes of the input [Szegedy et al., 2014; Goodfellow et al., 2015]. The robustness is characterized by the distance d(fθ(y′ ), x) between fθ(y′ ) and target x, where the worst-case perturbed input y′ is located within a small neighborhood of the original input y and maximizes the distance between its output and target x.
The worst-case perturbation y′ can be approximated via many adversarial attack methods, such as FGSM [Goodfellow et al., 2015], I-FGSM [Kurakin et al., 2017], and PGD [Madry et al., 2018], which solve (1) via gradient descent methods. The distance d(fθ(y′ ), x) is an indication of the robustness of fθ around y: a small distance implies strong robustness and vice versa. In terms of image classification, the ρ-neighborhood is usually defined by the ℓ∞-norm and the distance d(·, ·) is measured by the cross-entropy loss [Madry et al., 2018] or a margin loss [Carlini and Wagner, 2017]. For image restoration, the distance between images is usually measured by the ℓ2-norm [Zhang et al., 2017].
In most cases, deep learning models have been shown to be vulnerable against adversarial attacks under normal training (NT) [Tramer et al., 2020; Yan et al., 2019]. To robustify DNNs, Madry et al. [2018] proposed the PGD adversarial training (AT) method which trains DNNs with adversarial examples of the original data. AT is formally formulated as the following min-max optimization problem,
Its effectiveness has been verified by extensive empirical and theoretical results [Yan et al., 2021; Gao et al., 2019]. For further improvement, many variants of PGD have been proposed in terms of its robustness enhancement [Zhang et al., 2019a], generalization to non-adversarial data [Zhang et al., 2020a], and computational efficiency [Shafahi et al., 2019].
Deep image denoising During image capturing, unknown types of noise may be induced by physical sensors, data compression, and transmission. Noisy observations are usually modeled as the addition between the ground-truth images and certain zero-mean noise [Dabov et al., 2007], i.e., Y = X + V with EQ[∑mi=1 V[i] ] = 0, where V[i] is the ith element of V. The random vector X ∈ Rm with distribution P denotes a random clean image and the noise V ∈ Rm with a distribution Q satisfies the zeromean constraint. Denoising techniques aim to recover clean images from their noisy observations [Zhang et al., 2017; Dabov et al., 2007]. Suppose we are given a training set S = {(yj, xj)}Nj=1 of noisy and clean image pairs sampled from distributions Q and P respectively, we can train a DNN to effectively remove the noise induced by distribution Q from the noisy observations. A series of DNNs have been developed for denoising in recent years, including DnCNN [Zhang et al., 2017] and RIDNet [Anwar and Barnes, 2019].
In real-world applications [Abdelhamed et al., 2018; Xu et al., 2017], the noise distribution Q is usually unknown due to the complexity of the image capturing procedures; besides, collecting a large number of image pairs (clean/noisy or noisy/noisy) for training sometimes may be unrealistic in safety-critical domains such as medical radiology [Zhang et al., 2019b]. To overcome these, researchers developed denoising techniques by approximating real noise with common distributions like Gaussian or Poisson [Dabov et al., 2007; Zhang et al., 2019b]. To train denoisers that can deal with different levels of noise, where the noise level is measured by the energy-density ‖v‖22 /m of noise, the training set may consist of noisy images sampled from a variety of noise distributions [Zhang et al., 2017], whose expected energy-densities range from zero to certain budget ϵ2 (the expected ℓ2-norms range from zero to ϵ√m). For example, S ϵ = {(yj, xj)}Nj=1 where yj = xj + vj and xj and vj are sampled from P and Q respectively and where Q is randomly selected from a set of Gaussian distributions Qϵ = {N(0, σ2 I)|σ ∈ [0, ϵ]}. The denoiser f ϵ θ (·) trained with S ϵ is termed as an ϵ-denoiser.
On robustness of deep image denoisers In practice, data storage and transmission may induce imperceptible perturbations on the original data so that the perturbed noise may be statistically slightly different from the noise sampled from
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
O
d
g
8
L
j
u
9
B
X
v
H
r
W
w
/
>
A
+
n
c
V
N
S
M
E
3
q
0
y
C
p
7
I
o
Q
z
2
R
k
Z
F
D
m
G
Y
5
T
U
J
f
P
K
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
d
D
r
z
V
T
U
g
f
2
+
I
m
q
3
/
R
P
c
>
A
B
L
S
N
E
O
y
Y
9
o
K
M
F
8
J
Z
5
H
k
0
X
W
G
v
p
j
n
Q
w
C
7
u
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
5
n
F
y
B
2
P
f
C
U
z
S
u
m
k
8
>
A
X
c
Z
V
D
L
g
M
3
r
O
J
N
p
R
d
Q
9
G
E
/
j
W
T
w
H
Y
7
0
v
I
o
+
q
K
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
D
d
n
7
H
G
r
8
F
J
q
B
+
Q
c
o
w
9
>
A
X
Z
V
L
S
g
M
3
U
f
u
k
T
W
m
y
I
R
E
j
/
p
v
5
z
N
O
K
2
Y
0
P
C
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
f
9
5
S
N
Z
G
8
y
w
H
k
0
Q
u
P
+
>
A
B
X
c
V
E
J
3
U
r
q
/
L
g
R
D
W
v
Y
C
z
m
d
T
o
F
n
M
p
2
O
K
I
j
7
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
d
D
r
z
V
T
U
g
f
2
+
I
m
q
3
/
R
P
c
>
A
B
L
S
N
E
O
y
Y
9
o
K
M
F
8
J
Z
5
H
k
0
X
W
G
v
p
j
n
Q
w
C
7
u
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
D
d
n
7
H
G
r
8
F
J
q
B
+
Q
c
o
w
9
>
A
X
Z
V
L
S
g
M
3
U
f
u
k
T
W
m
y
I
R
E
j
/
p
v
5
z
N
O
K
2
Y
0
P
C
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
X
Q
Z
f
K
P
m
u
A
k
c
z
Y
U
H
5
B
o
>
3
/
L
S
g
N
E
O
G
M
r
8
V
9
j
d
7
w
y
J
0
q
p
C
W
2
F
I
R
+
D
n
T
v
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
o
P
S
T
m
8
Q
f
F
J
9
W
2
K
M
+
g
U
>
A
B
3
X
c
/
L
N
E
O
y
r
Y
u
I
n
Z
j
7
w
G
p
C
R
V
H
0
k
q
5
z
D
d
v
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
9
8
Q
q
d
+
y
V
K
7
T
u
/
S
Z
j
p
>
A
B
3
c
N
E
J
U
r
L
m
H
o
v
F
Y
w
X
5
M
z
C
I
n
P
2
g
R
0
O
f
G
k
W
D
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
/
H
W
X
G
o
2
9
I
f
M
p
j
B
c
+
>
A
n
V
N
S
8
E
J
3
U
r
q
L
R
g
5
k
v
F
Y
w
m
d
z
u
K
Q
P
0
7
T
D
y
Z
O
C
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
X
Q
Z
f
K
P
m
u
A
k
c
z
Y
U
H
5
B
o
>
3
/
L
S
g
N
E
O
G
M
r
8
V
9
j
d
7
w
y
J
0
q
p
C
W
2
F
I
R
+
D
n
T
v
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
f
9
5
S
N
Z
G
8
y
w
H
k
0
Q
u
P
+
>
A
B
X
c
V
E
J
3
U
r
q
/
L
g
R
D
W
v
Y
C
z
m
d
T
o
F
n
M
p
2
O
K
I
j
7
the specific original distribution. Although an ϵ-denoiser can successfully remove noise sampled from Q ∈ Qϵ , the performance of noise removal on the perturbed data is not guaranteed. Thus, we propose a novel attack method, OBSATK, to assess the adversarial robustness of DIDs in Section 3. To robustify DIDs, we propose an adversarial training strategy, HAT, in Section 4. HAT-trained DIDs can effectively denoise adversarial perturbed noisy images and preserve good performance on non-adversarial data. Besides the adversarial robustness issue, it has been shown that ϵ-denoisers trained with S ϵ cannot generalize well to unseen real-world noise [Lehtinen et al., 2018; Batson and Royer, 2019]. Several methods have been proposed for real-world noise removal, but most of them require a large number of real noisy data for training, e.g., CBDNet (clean/noisy pairs) [Guo et al., 2019] and Noise2Noise (noisy pairs) [Lehtinen et al., 2018], which is sometimes impractical. In Section 4.3, we show that HATtrained DIDs can generalize well to unseen real noise without the need of utilizing real noisy images for training.
3 OBSATK for Robustness Evaluation
In this section, we propose a novel adversarial attack, Observation-based Zero-mean Attack (OBSATK), to evaluate the robustness of DIDs. We also conduct experiments on benchmark datasets to demonstrate that normally-trained DIDs are vulnerable to adversarial perturbations.
3.1 Observation-based Zero-mean Attack
An ϵ-denoiser f ϵ θ (·) can generate a high-quality reconstruction f ϵ θ (y) close to the ground-truth x from a noisy observation y = x + v. To evaluate the robustness of f ϵ θ (·) with respect to a perturbation on y, we develop an attack to search for the worst perturbation δ∗ that degrades the recovered image f ϵ θ (y + δ∗ ) as much as possible. Formally, we need to solve the problem stated in Eq. (3). The optimization problem is subject to two constraints: The first constraint requires the norm of δ to be bounded by a small adversarial budget ρ. The second constraint restricts the mean M(δ) of all elements in δ to be zero. This corresponds to the zero-mean assumption of noise in real-world applications because a small mean-shift does not necessarily result in visual noise. For example, a mean-shift in gray-scale images implies a slight change of brightness. Since the zero-mean perturbation is
Algorithm 1 OBSATK
Input: Denoiser fθ(·), ground-truth x, noisy observation y, adversarial budget ρ, #iterations T , step-size η, minimum pixel value pmin, maximum pixel value pmax
Output: Adversarial perturbation δ
- 1: δ ← 0
- 2: for t = 1 to T do
- 3: δ ← δ + η∇δ‖f ϵ θ (y + δ) - x‖22 ;
- 5: δ ←min(ρ/‖δ‖2, 1)δ;
- 4: δ ← δ - (δ⊤n/‖n‖22 )n where n is in (4a)
- 6: end for
7: δ ←Clip(y+δ, pmin, pmax) -y
added to a noisy observation y, we term the proposed attack as Observation-based Zero-mean Attack (OBSATK).
We solve the constrained optimization problem Eq. (3) by using the classical projected-gradient-descent (PGD) method. PGD-like methods update optimization variables iteratively via gradient descent and ensure the constraints to be satisfied by projecting parameters back to the feasible region at the end of each iteration. To deal with the ℓ2-norm and zero-mean constraints, we develop a two-step operation in Eq. (4), that first projects the perturbation δ back to the zero-mean hyperplane and then projects the result onto the ρ-neighborhood.
In each iteration, as shown in Figure 1, the first step involves projecting the perturbation δ onto the zero-mean hyperplane. The zero-mean hyperplane consists of all the vectors z whose mean of all elements equals zero, i.e., n⊤z = 0, where n is the length-d all ones vector. Thus, n is a normal of the zero-mean plane. We can project any vector onto the zero-mean plane via (4a). The vector δ is first projected along the direction of n, then its projection δ′ onto the zero-mean plane equals itself minus its projection onto n. The second step involves further projecting δ′ back to the ρ-ball via linear scaling. If δ′ is already within the ρ-ball, we keep δ′ unchanged. Otherwise, the final projection δ′′ is obtained by scaling δ′ with a factor ρ/‖δ′ ‖2. For any two sets A and B, although the projection onto A ∩ B is, in general, not equal to the result obtained by first projecting onto A, then onto B, surprisingly, the following holds for the two sets in (3b).
Theorem 1 (Informal). Given any vector δ ∈ Rm, the projection of δ via the two-step operation in (4) satisfies the two constraints in (3b), and the two-step projection is equivalent to the exact projection onto the set defined by (3b).
The formal statement and the proof of Theorem 1 are provided in [Yan et al., 2022, Appendix A]. The complete procedure of OBSATK is summarized in Algorithm 1.
3.2 Robustness Evaluation via OBSATK
We use OBSATK to evaluate the adversarial robustness of ϵdenoisers on several gray-scale and RGB benchmark datasets, including Set12, Set68, BSD68, and Kodak24. For grayscale image denoising, we use Train400 to train a DnCNNB [Zhang et al., 2017] model, which consists of 20 convolutional layers. We follow the training setting in Zhang et al. [2017] and randomly crop 128 × 3000 patches in size of 50 × 50. Noisy and clean image pairs are constructed by injecting different levels of white Gaussian noise into clean patches. The noise levels σ are uniformly randomly selected from [0, ϵ] with ϵ = 25/255. For RGB image denoising, we use BSD432 (BSD500 excluding images in BSD68) to train a DnCNN-C model with the same number of layers as DnCNNB and but set the input and output channels to be three. Other settings follow those of the training of DnCNN-B.
We evaluate the denoising capability of the ϵ-denoiser on Gaussian noisy images and their adversarially perturbed versions. The image quality of reconstruction is measured via the peak-signal-noise ratio (PSNR) metric. A large PSNR between reconstruction and ground-truth implies a good performance of denoising. We denote the energy-density of the noise in test images as ˆ ϵ2 and consider three levels of noise, i.e., ˆ ϵ = 25/255, 15/255, and 10/255. For Gaussian noise removal, we add white Gaussian noise with σ = ˆ ϵ to clean images. For Uniform noise removal, we generate noise from U(-√3ˆ ϵ, √3ˆ ϵ). For denoising adversarial noisy images, the norm budgets of adversarial perturbation are set to be ρ = 5/255 · √m and 7/255 · √m respectively, where m equals the size of test images. We perturb noisy observations whose noise are generated from N(0, ˆ ϵ - ρ/√m), so that the ℓ2-norms of total noise in adversarial images are still bounded by ˆ ϵ · √m and the energy-density thus are bounded by ˆ ϵ2 . We use Atk-ρ/√m to denote the adversarially perturbed noisy images in the size of m with adversarial budget ρ. The number of iterations of PGD in OBSATK is set to be five.
From Tables 1, we observe that OBSATK clearly degrades the reconstruction performance of DIDs. In comparison to Gaussian or Uniform noisy images with the same noise levels, the recovered results from adversarial images are much worse in the sense of the PSNR. For example, when removing ˆ ϵ = 15/255 noisy images in Set12, the average PSNR of reconstructions from Gaussian noise can achieve 32.78 dB, whereas the PSNR drops to 26.25 dB when dealing with Atk7/255 adversarial images. We observe the consistent phenomenon that a normally-trained denoiser f ϵ θ (·) cannot effectively remove adversarial noise from visual results in Figure 2.
| Dataset | ˆ ϵ | N | U | Atk-5/255 | Atk-7/255 |
|---|---|---|---|---|---|
| Set68 | 25/255 15/255 | 29.16/0.02 31.68/0.00 | 29.15/0.01 31.68/0/00 | 24.26/0.12 26.66/0.04 | 23.12/0.10 26.08/0.02 |
| Set12 | 25/255 15/255 | 30.39/0.01 32.78/0.00 | 30.41/0.01 32.81/0.00 | 24.32/0.18 26.91/0.05 | 22.96/0.13 26.25/0.01 |
| BSD68 | 25/255 15/255 | 31.25/0.11 33.98/0.11 | 31.17/0.11 33.93/0.12 | 27.44/0.08 29.31/0.08 | 26.08/0.06 27.84/0.04 |
| Kodak24 | 25/255 15/255 | 32.20/0.13 34.77/0.13 | 32.13/0.14 34.73/0.14 | 27.87/0.08 29.55/0.07 | 26.37/0.07 28.00/0.04 |
4 Robust and Generalizable Denoising via HAT
The previous section shows that existing deep denoisers are vulnerable to adversarial perturbations. To improve the adversarial robustness of deep denoisers, we propose an adversarial training method, hybrid adversarial training (HAT), that uses original noisy images and their adversarial versions for training. Furthermore, we build a connection between the adversarial robustness of deep denoisers and their generalization capability to unseen types of noise. We show that HAT-trained denoisers can effectively remove real-world noise without the need to leverage the real-world noisy data.
4.1 Hybrid Adversarial Training
AT has been proved to be a successful and universally applicable technique for robustifying deep neural networks. Most variants of AT are developed for the classification task specifically, such as TRADES [Zhang et al., 2019a] and GAIRAT [Zhang et al., 2020b]. Here, we propose an AT strategy, HAT, for robust image denoising:
where Y = X + V and Y′ = Y + δ∗ . Note that δ∗ is the adversarial perturbation obtained by solving OBSATK in Eq. (3).
As shown in Eq. (5), the loss function consists of two terms. The first term measures the distance between ground-truth images x and reconstructions f ϵ θ (y) from nonadversarial noisy images y, where y contains noise v sampled from a certain common distribution Q, such as Gaussian. This term encourages a good reconstruction performance of f ϵ θ from common distributions. The second term is the distance between f ϵ θ (y) and the reconstruction f ϵ θ (y′ ) from the adversarially perturbed version y′ of y. This term ensures that the reconstructions from any two noisy observations within a small neighborhood of y have similar image qualities. Minimizing these two terms at the same time controls the worst-case reconstruction performance ‖f ϵ θ (y′ )-x‖.
The coefficient α balances the trade-off between reconstruction from common noise and the local continuity of f ϵ θ . When α equals zero, HAT degenerates to normal training on common noise. The obtained denoisers fail to resist adversarial perturbations as shown in Section 3. When α is very large, the optimization gradually ignores the first term and completely aims for local smoothness. This may yield a trivial solution that f ϵ θ always outputs a constant vector for any input. A proper value of α thus ensures a denoiser that performs well for common noise and the worst-case adversarial perturbations simultaneously. We perform an ablation study on the effect of α for the robustness enhancement and unseen noise removal in [Yan et al., 2022, Appendix C].
To train a denoiser applicable to different levels of noise with an energy-density bounded by ϵ2 , we randomly select a noise distribution Q from a family of common distributions Qϵ . Qϵ includes a variety of zero-mean distributions whose variance are bounded by ϵ2 . For example, we define QϵN = {N(0, σ2 I))|σ ∼ U(0, ϵ)} for the experiments in the remaining of this paper.
(b) NT
(c) HAT
Figure 3: From left to right are the ground-truth, the reconstruction of a normally-trained denoiser against attack, and the reconstruction of a HAT-trained denoiser against attack.
4.2 Robustness Enhancement via HAT
We follow the same settings as those in Section 3 for training and evaluating ϵ-deep denoisers. The highest level of noise used for training is set to be ϵ = 25/255. Noise is sampled from a set of Gaussian distributions QϵN . We train deep denoisers with the HAT strategy and set α to be 1, and use one-step Atk-5/255 to generate adversarially noisy images for training. We compare HAT with normal training (NT) and the vanilla adversarial training (vAT) used in Choi et al. [2021] that trains denoisers only with adversarial data. The results on Set68 and BSD68 are provided in this section. More results on Set12 and Kodak24 (in Tables B.1 and B.2) are provided in [Yan et al., 2022, Appendix B].
| Method | ˆ ϵ | N | Atk-3/255 | Atk-5/255 | Atk-7/255 |
|---|---|---|---|---|---|
| NT | 25/255 15/255 | 29.16/0.02 31.68/0.00 | 26.20/0.07 27.98/0.05 | 24.26/0.12 26.66/0.04 | 23.12/0.10 26.08/0.02 |
| vAT | 25/255 15/255 | 29.05/0.07 31.53/0.09 | 27.02/0.15 28.74/0.16 | 25.51/0.32 27.43/0.19 | 24.34/0.34 26.68/0.15 |
| HAT | 25/255 15/255 | 28.88/0.04 31.36/0.03 | 27.48/0.10 29.52/0.01 | 26.40/0.16 28.34/0.03 | 25.32/0.17 27.34/0.03 |
From Tables 2 and 3, we observe that HAT obviously improves the reconstruction performance from adversarial noise in comparison to normal training. For example, on the Set68 dataset (Table 2), when dealing with 15/255-level noise, the normally-trained denoiser achieves 31.68 dB for Gaussian noise removal, but the PSNR drops to 26.08 dB against Atk7/255. In contrast, the HAT-trained denoiser achieves a PSNR of 27.34 dB (1.26 dB higher) against Atk-7/255 and maintains a PSNR of 31.36 dB for Gaussian noise removal. In Figure 3, we can see that when dealing with adversarially noisy images, the HAT-trained denoiser can recover high-quality images while the normally-trained denoiser preserves noise patterns in the output. Besides, we observe that, similar to image classification tasks [Zhang et al., 2019a], AT-based methods (HAT and vAT) robustify deep denoisers at the expense of the performance on non-adversarial data (Gaussian denoising). Nevertheless, the degraded reconstructions are still reasonably good in terms of the PSNR.
| Method | ˆ ϵ | N | Atk-3/255 | Atk-5/255 | Atk-7/255 |
|---|---|---|---|---|---|
| NT | 25/255 15/255 | 31.25/0.11 33.98/0.11 | 28.93/0.08 31.09/0.10 | 27.44/0.08 29.31/0.08 | 26.08/0.06 27.84/0.04 |
| vAT | 25/255 15/255 | 30.64/0.02 33.45/0.06 | 28.81/0.03 31.10/0.05 | 27.67/0.01 29.79/0.02 | 26.64/0.03 28.63/0.08 |
| HAT | 25/255 15/255 | 30.98/0.03 33.67/0.04 | 29.18/0.03 31.38/0.04 | 28.02/0.02 30.03/0.02 | 26.93/0.04 28.80/0.01 |
| Dataset | BM3D | DIP | N2S(1) | NT | vAT | HAT | N2C |
|---|---|---|---|---|---|---|---|
| PolyU | 37.40 / 0.00 | 36.08 / 0.01 | 35.37 / 0.15 | 35.86 / 0.01 | 36.77 / 0.00 | 37.82 / 0.04 | - / - |
| CC | 35.19 / 0.00 | 34.64 / 0.06 | 34.33 / 0.14 | 33.56 / 0.01 | 34.49 / 0.10 | 36.26 / 0.06 | - / - |
| SIDD | 25.65 / 0.00 | 26.89 / 0.02 | 26.51 / 0.03 | 27.20 / 0.70 | 27.08 / 0.28 | 33.44 / 0.02 | 33.50 / 0.03 |
4.3 Robustness Benefits Generalization to Unseen Noise
It has been shown that denoisers that are normally trained on common synthetic noise fail to remove real-world noise induced by standard imaging procedures [Xu et al., 2017; Abdelhamed et al., 2018]. To train denoisers that can handle real-world noise, researchers have proposed several methods which can be roughly divided into two categories, namely dataset-based denoising methods and single-image-based denoising methods. High-performance dataset-based methods require a set of real noisy data for training, e.g., CBDNet requiring pairs of clean and noisy images [Guo et al., 2019] and Noise2Noise requiring multiple noisy observations of every single image [Lehtinen et al., 2018]. However, a large number of paired data are not available in some applications, such as medical radiology and high-speed photography. To address this, single-image-based methods are proposed to remove noise by exploiting the correlation between signals across pixels and the independence between noise. This category of methods, such as DIP [Ulyanov et al., 2018] and N2S [Batson and Royer, 2019], are adapted to various types of signal-independent noise, but they optimize the deep denoiser on each test image. The test-time optimization is extremely time-consuming, e.g., N2S needs to update a denoiser for thousands of iterations to achieve good reconstruction performance.
Here, we point out that HAT is a promising framework to train a generalizable deep denoiser only with synthetic noise. The resultant denoiser can be directly applied to perform denoising for unseen noisy images in real-time. During training, HAT first samples noise from common distributions (Gaussian) with noise levels from low to high. OBSATK then explores the ρ-neighborhood for each noisy image to search for a particular type of noise that degrades the denoiser the most. By ensuring the denoising performance of the worst-case noise, the resultant denoiser can deal with other unknown types of noise within the ρ-neighborhood as well. To train a robust denoiser that generalizes well to realworld noise, we need to choose a proper adversarial budget ρ. When ρ is very small and close to zero, the HAT reduces to normal training. When ρ is very much larger than the norm of basic noise v, the adversarially noisy image may be visually unnatural because the adversarial perturbation δ only satisfies the zero-mean constraint and is not guaranteed to be spatially uniformly distributed as other types of natural noise being. In practice, we set the value of ρ of OBSATK to be 5/255 · √m, where m denotes the size of image patches. The value of α of HAT is kept unchanged as 2.
Experimental Settings We evaluate the generalization capability of HAT on several real-world noisy datasets, including PolyU [Xu et al., 2018], CC [Xu et al., 2017], and SIDD [Abdelhamed et al., 2018]. PolyU, CC, and SIDD contain RGB images of common scenes in daily life. These images are captured by different brands of digital cameras and smartphones, and they contain various levels of noise by adjusting the ISO values. For the PolyU and CC, we use the clean images in BSD500 for training an adversarially robust ϵ-denoiser with ϵ = 25/255. We sample Gaussian noise from a set of distributions QϵN and add the noise to clean images to craft noisy observations. HAT trains the denoiser jointly with Gaussian noisy images and their adversarial versions. For the SIDD, we use clean images in the SIDD-small set for training and test the denoisers on the SIDD-val set. The highest level of noise used for HAT is set to be ϵ = 50/255. In each case, we only use clean images for training denoisers without the need of real noisy images Results We compare HAT-trained denoisers with the NT and vAT-trained ones. From Table 4, we observe that HAT performs much better than both competitors. For example, on the SIDD-val dataset, the HAT-trained denoiser achieves an average PSNR value of 33.44 dB that is 6.24 dB higher than the NT-trained one. We also compare HAT-trained denoisers with single-image-based methods, including DIP, N2S, and the classical BM3D [Dabov et al., 2007]. For DIP and N2S (the officially released codes of DIP and N2S are used here), the numbers of iterations for each image are set to be 2,000 and 1,000, respectively. N2S works in two modes, namely single-image-based denoising and dataset-based denoising. Here, we use N2S in the single-image-based mode, denoted as N2S(1), due to the assumption that no real noisy data are available for training. We observe that HAT-trained denoisers consistently outperform these baselines. Visual comparisons are provided in [Yan et al., 2022, Appendix D]. Besides, since the SIDD-small provides a set of real noisy and groundtruth pairs, we train a denoiser, denoted as Noise2Clean (N2C), with these paired data and use the N2C denoiser as the oracle for comparison. We observe that HAT-trained denoisers are comparable to the N2C one for denoising images in SIDD-val (a PSNR of 33.44dB vs 33.50dB).
5 Conclusion
Normally-trained deep denoisers are vulnerable to adversarial attacks. HAT can effectively robustify deep denoisers and boost their generalization capability to unseen real-world noise. In the future, we will extend the adversarial-training framework to other image restoration tasks, such as deblurring. We aim to develop a generic AT-based robust optimization framework to train deep models that can recover clean images from unseen types of degradation.
Acknowledgments
Hanshu Yan and Vincent Tan were funded by a Singapore National Research Foundation (NRF) Fellowship (A-800019601-00). Jingfeng Zhang was supported by JST ACT-X under Grant Number JPMJAX21AF. Masashi Sugiyama was supported by JST CREST under Grant Number JPMJCR18A2.
@inproceedings{ijcai2022p211,
title = {Towards Adversarially Robust Deep Image Denoising},
author = {Yan, Hanshu and Zhang, Jingfeng and Feng, Jiashi and Sugiyama, Masashi and Tan, Vincent Y. F.},
booktitle = {Proceedings of the Thirty-First International Joint Conference on
Artificial Intelligence, {IJCAI-22}},
publisher = {International Joint Conferences on Artificial Intelligence Organization},
editor = {Luc De Raedt},
pages = {1516--1522},
year = {2022},
month = {7},
note = {Main Track},
doi = {10.24963/ijcai.2022/211},
url = {https://doi.org/10.24963/ijcai.2022/211},
}
TY - CPAPER TI - Towards Adversarially Robust Deep Image Denoising AU - Yan, Hanshu AU - Zhang, Jingfeng AU - Feng, Jiashi AU - Sugiyama, Masashi AU - Tan, Vincent Y. F. T2 - Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22 PB - International Joint Conferences on Artificial Intelligence Organization PY - 2022 SP - 1516 EP - 1522 DO - 10.24963/ijcai.2022/211 UR - https://doi.org/10.24963/ijcai.2022/211 ER -Copied
Machine-readable versions of this paper, for tools and AI agents:
