Measuring Data Leakage in Machine-Learning Models with Fisher Information (Extended Abstract)
Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence. Sister Conferences Best Papers. Pages 5284-5288.
https://doi.org/10.24963/ijcai.2022/736
Keywords
- Artificial Intelligence: General
Abstract
Machine-learning models contain information about the data they were trained on. This information leaks either through the model itself or through predictions made by the model. Consequently, when the training data contains sensitive attributes, assessing the amount of information leakage is paramount. We propose a method to quantify this leakage using the Fisher information of the model about the data. Unlike the worst-case a priori guarantees of differential privacy, Fisher information loss measures leakage with respect to specific examples, attributes, or sub-populations within the dataset. We motivate Fisher information loss through the Cramer-Rao bound and delineate the implied threat model. We provide efficient methods to compute Fisher information loss for output-perturbed generalized linear models. Finally, we empirically validate Fisher information loss as a useful measure of information leakage.
- [Agrawal and Aggarwal, 2001] Dakshi Agrawal and Charu C Ag- garwal. On the design and quantification of privacy preserv- ing data mining algorithms. In Proceedings of the twentieth ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems, pages 247-255, 2001.
- [Anderson, 1977] Harald Anderson. Efficiency versus protection in a general randomized response model. Scandinavian Journal of Statistics, pages 11-19, 1977.
- [Carlini et al., 2019] Nicholas Carlini, Chang Liu, Úlfar Erlings- son, Jernej Kos, and Dawn Song. The secret sharer: Evaluat- ing and testing unintended memorization in neural networks. In 28th USENIX Security Symposium (USENIX Security 19), pages 267-284, 2019.
- [Chaudhuri et al., 2011] Kamalika Chaudhuri, Claire Monteleoni, and Anand D Sarwate. Differentially private empirical risk min- imization. Journal of Machine Learning Research, 12(3), 2011.
- [Cummings et al., 2019] Rachel Cummings, Varun Gupta, Dhamma Kimpara, and Jamie Morgenstern. On the com- patibility of privacy and fairness. In Adjunct Publication of the 27th Conference on User Modeling, Adaptation and Personalization, pages 309-315, 2019.
- [Daubechies et al., 2010] Ingrid Daubechies, Ronald DeVore, Mas- simo Fornasier, and C Sinan Güntürk. Iteratively reweighted least squares minimization for sparse recovery. Communications on Pure and Applied Mathematics: A Journal Issued by the Courant Institute of Mathematical Sciences, 63(1):1-38, 2010.
- [Dong et al., 2019] Jinshuo Dong, Aaron Roth, and Weijie J Su. Gaussian differential privacy. arXiv preprint arXiv:1905.02383, 2019.
- [Dwork and Roth, 2014] Cynthia Dwork and Aaron Roth. The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science, 9(3-4):211-407, 2014.
- [Dwork et al., 2006] Cynthia Dwork, Frank McSherry, Kobbi Nis- sim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference, pages 265- 284. Springer, 2006.
- [Ekstrand et al., 2018] Michael D Ekstrand, Rezvan Joshaghani, and Hoda Mehrpouyan. Privacy for all: Ensuring fair and eq- uitable privacy protections. In Conference on Fairness, Account- ability and Transparency, pages 35-47, 2018.
- [Ene and Vladu, 2019] Alina Ene and Adrian Vladu. Improved convergence for ℓ1 and ℓ∞ regression via iteratively reweighted least squares. In International Conference on Machine Learning, pages 1794-1801. PMLR, 2019.
- [Farokhi and Kaafar, 2020] Farhad Farokhi and Mohamed Ali Kaa- far. Modelling and quantifying membership information leakage in machine learning. arXiv preprint arXiv:2001.10648, 2020.
- [Farokhi and Sandberg, 2017] Farhad Farokhi and Henrik Sand- berg. Fisher information as a measure of privacy: Preserving privacy of households with smart meters using batteries. IEEE Transactions on Smart Grid, 9(5):4726-4734, 2017.
- [Fredrikson et al., 2014] Matthew Fredrikson, Eric Lantz, Somesh Jha, Simon Lin, David Page, and Thomas Ristenpart. Privacy in pharmacogenetics: An end-to-end case study of personalized warfarin dosing. In 23rd USENIX Security Symposium (USENIX Security 14), pages 17-32, 2014.
- [Ghosh and Kleinberg, 2016] Arpita Ghosh and Robert Kleinberg. Inferential privacy guarantees for differentially private mecha- nisms. arXiv preprint arXiv:1603.01508, 2016.
- [Green, 1984] Peter J Green. Iteratively reweighted least squares for maximum likelihood estimation, and some robust and resis- tant alternatives. Journal of the Royal Statistical Society: Series B (Methodological), 46(2):149-170, 1984.
- [Kay, 1993] Steven M Kay. Fundamentals of statistical signal pro- cessing. Prentice Hall PTR, 1993.
- [Li et al., 2007] Ninghui Li, Tiancheng Li, and Suresh Venkata- subramanian. t-closeness: Privacy beyond k-anonymity and l- diversity. In 2007 IEEE 23rd International Conference on Data Engineering, pages 106-115. IEEE, 2007.
- [Liu et al., 2016] Changchang Liu, Supriyo Chakraborty, and Pra- teek Mittal. Dependence makes you vulnberable: Differential privacy under dependent tuples. In NDSS, volume 16, pages 21- 24, 2016.
- [Long et al., 2018] Yunhui Long, Vincent Bindschaedler, Lei Wang, Diyue Bu, Xiaofeng Wang, Haixu Tang, Carl A Gunter, and Kai Chen. Understanding membership inferences on well- generalized learning models. arXiv preprint arXiv:1802.04889, 2018.
- [Machanavajjhala et al., 2007] Ashwin Machanavajjhala, Daniel Kifer, Johannes Gehrke, and Muthuramakrishnan Venkitasubra- maniam. l-diversity: Privacy beyond k-anonymity. ACM Trans- actions on Knowledge Discovery from Data (TKDD), 1(1):3-es, 2007.
- [McSherry and Talwar, 2007] Frank McSherry and Kunal Talwar. Mechanism design via differential privacy. In 48th Annual IEEE Symposium on Foundations of Computer Science (FOCS'07), pages 94-103. IEEE, 2007.
- [Mironov, 2017] Ilya Mironov. Rényi differential privacy. In 2017 IEEE 30th Computer Security Foundations Symposium (CSF), pages 263-275. IEEE, 2017.
- [Papernot et al., 2017] Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In Proceedings of the 2017 ACM on Asia conference on computer and communications security, pages 506-519, 2017.
- [Samarati and Sweeney, 1998] P. Samarati and L. Sweeney. Pro- tecting privacy when disclosing information: k-anonymity and its enforcement through generalization and suppression. Techni- cal report, 1998.
- [Shokri et al., 2017] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE Symposium on Security and Privacy (SP), pages 3-18. IEEE, 2017.
- [Tramèr et al., 2016] Florian Tramèr, Fan Zhang, Ari Juels, Michael K Reiter, and Thomas Ristenpart. Stealing machine learning models via prediction apis. In 25th USENIX Security Symposium (USENIX Security 16), pages 601-618, 2016.
- [Yaghini et al., 2019] Mohammad Yaghini, Bogdan Kulynych, and Carmela Troncoso. Disparate vulnerability: On the unfairness of privacy attacks against machine learning. arXiv preprint arXiv:1906.00389, 2019.
- [Yeom et al., 2018] Samuel Yeom, Irene Giacomelli, Matt Fredrik- son, and Somesh Jha. Privacy risk in machine learning: Analyz- ing the connection to overfitting. In 2018 IEEE 31st Computer Security Foundations Symposium (CSF), pages 268-282. IEEE, 2018.
References were extracted automatically from the PDF and may contain errors.
Full text extracted automatically from the PDF; figures, tables and equations may differ. The PDF is the version of record.
1 Introduction
Machine-learning models trained on sensitive data are often made public. Even when the models are not explicitly released, they may be implicitly leaked from their predictions [Papernot et al., 2017; Tram`er et al., 2016]. Undeniably, these models contain information about the data they were trained on. Without mitigating measures, training set membership can often be inferred [Shokri et al., 2017; Yeom et al., 2018], and sensitive attributes or even whole examples can be extracted [Carlini et al., 2019; Fredrikson et al., 2014].
Assessing the information leaked from models about their training data is commonly done with techniques such as differential privacy [Dwork et al., 2006]. Differential privacy successfully avoids the "just a few" failure mode of more heuristic privacy assessments, in which privacy is protected for many but not all individuals [Dwork and Roth, 2014]. This comes at the cost of a worst-case assessment, leading
∗Best Paper at the 37th Conference on Uncertainty in Artificial Intelligence (2021).
†Contact Author
to large differences in the vulnerability of individuals to privacy attacks. Also, a mismatch exists between the protection of differential privacy, which is relative to participation in a dataset, and privacy attacks, which can take advantage of the absolute information leaked from a trained model [Carlini et al., 2019; Long et al., 2018]. Furthermore, differential privacy degrades when correlations exist in the dataset [Ghosh and Kleinberg, 2016; Liu et al., 2016].
We propose an example-specific and correlation-aware measure of data leakage using Fisher information. The quantity, which we term Fisher information loss, can assess the leakage of a model about various subsets of the full dataset. This includes, for example, assessments at the granularity of individual attributes, individual examples, groups of examples, or the full training set. We show, via the Cram´er-Rao bound, that under specific assumptions the ability of an adversary to estimate the underlying data from a model with bounded Fisher information loss is limited. We also demonstrate that, unlike differential privacy, Fisher information loss does not implicitly degrade when data is correlated.
The ability of Fisher information loss to measure perexample privacy loss means it can be used as the basis for algorithms that aim to achieve fairness in privacy [Cummings et al., 2019; Ekstrand et al., 2018]. We demonstrate this by developing an algorithm that balances Fisher information loss for individuals in the training set, thereby resolving the problem that subgroups may have "disparate vulnerability" to privacy attacks [Yaghini et al., 2019].
2 RelatedWork
This work builds on and complements a significant body of prior work in assessing the privacy of a model with respect to the data it was trained on. This includes approaches which obfuscate the original data such as k-anonymity [Samarati and Sweeney, 1998], information theoretic criteria [Agrawal and Aggarwal, 2001], and, perhaps the most commonly studied, differential privacy [Dwork et al., 2006] and its more recent variations [Mironov, 2017; Dong et al., 2019]. While these methods often provide rigorous privacy guarantees, most of them do not explicitly bound the inferential power of an adversary, and correlations in the training dataset can be exploited by an adversary to mount a successful inference attack [Machanavajjhala et al., 2007; Li et al., 2007; Liu et al., 2016]. Moreover, these definitions do not offer simple and flexible approaches to measure the information leaked by a model about varying subsets of the training data. This can lead to unequal vulnerability to privacy attacks despite attempts to protect privacy [Yaghini et al., 2019].
Because these privacy assessment techniques do not identify vulnerability at the level of the individual or subpopulation, prior work exists to quantify the susceptibility of such subgroups to privacy attacks. Farokhi and Kaafar [2020] propose an information theoretic measure of the vulnerability of individual examples to membership inference attacks. Carlini et al. [2019] propose a heuristic which can infer the susceptibility of data to model inversion attacks.
This work also builds upon prior studies of Fisher information as a measure of privacy. An early study analyzed loss of Fisher information in a general randomized response framework [Anderson, 1977]. More recently, the relationship of privacy with Fisher information to estimator error through the use of the Cram´er-Rao bound was investigated [Farokhi and Sandberg, 2017]. In particular, Farokhi and Sandberg [2017] turn to Fisher information as a practical alternative to differential privacy in protecting data collected from smart power meters. We build from these works in several directions, including a broader application to generalized linear models as well as the use of Fisher information loss in an algorithm which can provide fairness in data leakage.
3 Fisher Information Loss
Let D = {(x1, y1), . . . , (xn, yn)} be a training dataset of n examples with xi ∈ Rd and scalar target yi. We denote by A(D) a randomized learning algorithm which outputs a hypothesis h from a predefined hypothesis space H. Treating the hypothesis as a random variable, pA(h | D) is the probability density of h given D for the randomized algorithm A(D). We denote by Ih(D) ∈ Rn(d+1)×n(d+1) the Fisher information matrix (FIM) defined by
Definition (Fisher information loss). We say that h ∼ A(D) has Fisher information loss (FIL) of η with respect to D if
where ∇2D yields the matrix of second derivatives of log pA(h | D) with respect to the values in D, and the expectation is taken over the randomness in A(D). The FIM, Ih(D), is a measure of the information that the hypothesis h contains about the training data D. Hence, we use the FIM to measure the information loss from releasing the output h of a single evaluation of A(D).
where ‖Ih(D)‖2 denotes the 2-norm, or largest singular value, of the FIM. A smaller η means h contains less Fisher information about the training data, D.
Motivation. Fisher information is a classical tool used in statistics for lower bounding the variance of an estimator. We utilize this property to demonstrate that a small FIL implies a large variance for any unbiased estimate of the data.
Let z = [x⊤1 , y1, . . . , x⊤n , yn] ∈ Rn(d+1) be the vector formed by concatenating the examples in D, and z ∈ z an
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
I
7
X
/
M
2
P
L
U
c
q
8
p
N
>
A
B
3
V
C
S
g
F
0
z
G
+
o
Y
w
r
m
Q
j
K
y
n
J
D
k
Z
R
W
u
9
d
O
H
5
E
T
v
f
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
v
u
U
+
H
I
q
y
2
Z
J
g
E
k
>
A
B
/
3
c
V
C
7
S
N
F
L
0
X
z
G
o
p
Y
w
r
T
n
P
M
f
m
5
O
Q
9
d
W
8
D
K
j
R
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
D
m
j
S
X
K
C
Z
J
N
z
R
E
H
A
v
>
B
/
3
c
V
7
g
F
L
0
T
G
+
o
p
Y
2
w
r
u
y
W
P
d
k
O
9
U
n
I
M
q
Q
5
f
8
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
m
N
q
I
/
W
Y
g
E
r
+
K
o
U
k
X
A
C
c
>
B
V
7
S
F
L
0
z
G
p
2
w
T
J
n
P
Z
u
y
j
9
R
O
D
f
d
8
H
v
3
M
Q
5
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
m
N
q
I
/
W
Y
g
E
r
+
K
o
U
k
X
A
C
c
>
B
V
7
S
F
L
0
z
G
p
2
w
T
J
n
P
Z
u
y
j
9
R
O
D
f
d
8
H
v
3
M
Q
5
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
m
N
q
I
/
W
Y
g
E
r
+
K
o
U
k
X
A
C
c
>
B
V
7
S
F
L
0
z
G
p
2
w
T
J
n
P
Z
u
y
j
9
R
O
D
f
d
8
H
v
3
M
Q
5
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
3
+
N
Z
d
k
F
X
L
H
y
V
0
m
B
D
o
r
U
>
A
C
K
c
S
J
P
E
u
I
7
f
T
R
2
w
M
/
8
W
O
9
G
v
j
Q
z
5
q
g
p
Y
n
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
m
N
q
I
/
W
Y
g
E
r
+
K
o
U
k
X
A
C
c
>
B
V
7
S
F
L
0
z
G
p
2
w
T
J
n
P
Z
u
y
j
9
R
O
D
f
d
8
H
v
3
M
Q
5
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
D
m
j
S
X
K
C
Z
J
N
z
R
E
H
A
v
>
B
/
3
c
V
7
g
F
L
0
T
G
+
o
p
Y
2
w
r
u
y
W
P
d
k
O
9
U
n
I
M
q
Q
5
f
8
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
P
j
G
f
q
Q
8
5
J
Z
d
L
E
R
X
r
+
Y
M
A
>
C
3
c
V
D
S
N
F
W
k
w
U
T
m
o
u
n
g
y
K
9
H
O
v
B
z
/
0
2
p
7
I
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
m
N
q
I
/
W
Y
g
E
r
+
K
o
U
k
X
A
C
c
>
B
V
7
S
F
L
0
z
G
p
2
w
T
J
n
P
Z
u
y
j
9
R
O
D
f
d
8
H
v
3
M
Q
5
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
I
7
X
/
M
2
P
L
U
c
q
8
p
N
>
A
B
3
V
C
S
g
F
0
z
G
+
o
Y
w
r
m
Q
j
K
y
n
J
D
k
Z
R
W
u
9
d
O
H
5
E
T
v
f
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
m
N
q
I
/
W
Y
g
E
r
+
K
o
U
k
X
A
C
c
>
B
V
7
S
F
L
0
z
G
p
2
w
T
J
n
P
Z
u
y
j
9
R
O
D
f
d
8
H
v
3
M
Q
5
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
v
u
U
+
H
I
q
y
2
Z
J
g
E
k
>
A
B
/
3
c
V
C
7
S
N
F
L
0
X
z
G
o
p
Y
w
r
T
n
P
M
f
m
5
O
Q
9
d
W
8
D
K
j
R
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
I
7
X
/
M
2
P
L
U
c
q
8
p
N
>
A
B
3
V
C
S
g
F
0
z
G
+
o
Y
w
r
m
Q
j
K
y
n
J
D
k
Z
R
W
u
9
d
O
H
5
E
T
v
f
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
v
u
U
+
H
I
q
y
2
Z
J
g
E
k
>
A
B
/
3
c
V
C
7
S
N
F
L
0
X
z
G
o
p
Y
w
r
T
n
P
M
f
m
5
O
Q
9
d
W
8
D
K
j
R
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
D
m
j
S
X
K
C
Z
J
N
z
R
E
H
A
v
>
B
/
3
c
V
7
g
F
L
0
T
G
+
o
p
Y
2
w
r
u
y
W
P
d
k
O
9
U
n
I
M
q
Q
5
f
8
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
m
N
q
I
/
W
Y
g
E
r
+
K
o
U
k
X
A
C
c
>
B
V
7
S
F
L
0
z
G
p
2
w
T
J
n
P
Z
u
y
j
9
R
O
D
f
d
8
H
v
3
M
Q
5
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
m
N
q
I
/
W
Y
g
E
r
+
K
o
U
k
X
A
C
c
>
B
V
7
S
F
L
0
z
G
p
2
w
T
J
n
P
Z
u
y
j
9
R
O
D
f
d
8
H
v
3
M
Q
5
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
m
N
q
I
/
W
Y
g
E
r
+
K
o
U
k
X
A
C
c
>
B
V
7
S
F
L
0
z
G
p
2
w
T
J
n
P
Z
u
y
j
9
R
O
D
f
d
8
H
v
3
M
Q
5
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
3
+
N
Z
d
k
F
X
L
H
y
V
0
m
B
D
o
r
U
>
A
C
K
c
S
J
P
E
u
I
7
f
T
R
2
w
M
/
8
W
O
9
G
v
j
Q
z
5
q
g
p
Y
n
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
m
N
q
I
/
W
Y
g
E
r
+
K
o
U
k
X
A
C
c
>
B
V
7
S
F
L
0
z
G
p
2
w
T
J
n
P
Z
u
y
j
9
R
O
D
f
d
8
H
v
3
M
Q
5
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
D
m
j
S
X
K
C
Z
J
N
z
R
E
H
A
v
>
B
/
3
c
V
7
g
F
L
0
T
G
+
o
p
Y
2
w
r
u
y
W
P
d
k
O
9
U
n
I
M
q
Q
5
f
8
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
5
9
q
z
K
f
Q
y
o
7
H
3
C
D
d
u
k
>
A
R
n
c
V
L
S
N
F
2
p
/
T
I
E
U
X
Y
j
J
m
v
r
8
G
Z
P
w
O
0
W
B
g
M
+
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
m
N
q
I
/
W
Y
g
E
r
+
K
o
U
k
X
A
C
c
>
B
V
7
S
F
L
0
z
G
p
2
w
T
J
n
P
Z
u
y
j
9
R
O
D
f
d
8
H
v
3
M
Q
5
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
I
7
X
/
M
2
P
L
U
c
q
8
p
N
>
A
B
3
V
C
S
g
F
0
z
G
+
o
Y
w
r
m
Q
j
K
y
n
J
D
k
Z
R
W
u
9
d
O
H
5
E
T
v
f
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
m
N
q
I
/
W
Y
g
E
r
+
K
o
U
k
X
A
C
c
>
B
V
7
S
F
L
0
z
G
p
2
w
T
J
n
P
Z
u
y
j
9
R
O
D
f
d
8
H
v
3
M
Q
5
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
v
u
U
+
H
I
q
y
2
Z
J
g
E
k
>
A
B
/
3
c
V
C
7
S
N
F
L
0
X
z
G
o
p
Y
w
r
T
n
P
M
f
m
5
O
Q
9
d
W
8
D
K
j
R
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
I
7
X
/
M
2
P
L
U
c
q
8
p
N
>
A
B
3
V
C
S
g
F
0
z
G
+
o
Y
w
r
m
Q
j
K
y
n
J
D
k
Z
R
W
u
9
d
O
H
5
E
T
v
f
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
v
u
U
+
H
I
q
y
2
Z
J
g
E
k
>
A
B
/
3
c
V
C
7
S
N
F
L
0
X
z
G
o
p
Y
w
r
T
n
P
M
f
m
5
O
Q
9
d
W
8
D
K
j
R
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
D
m
j
S
X
K
C
Z
J
N
z
R
E
H
A
v
>
B
/
3
c
V
7
g
F
L
0
T
G
+
o
p
Y
2
w
r
u
y
W
P
d
k
O
9
U
n
I
M
q
Q
5
f
8
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
m
N
q
I
/
W
Y
g
E
r
+
K
o
U
k
X
A
C
c
>
B
V
7
S
F
L
0
z
G
p
2
w
T
J
n
P
Z
u
y
j
9
R
O
D
f
d
8
H
v
3
M
Q
5
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
m
N
q
I
/
W
Y
g
E
r
+
K
o
U
k
X
A
C
c
>
B
V
7
S
F
L
0
z
G
p
2
w
T
J
n
P
Z
u
y
j
9
R
O
D
f
d
8
H
v
3
M
Q
5
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
m
N
q
I
/
W
Y
g
E
r
+
K
o
U
k
X
A
C
c
>
B
V
7
S
F
L
0
z
G
p
2
w
T
J
n
P
Z
u
y
j
9
R
O
D
f
d
8
H
v
3
M
Q
5
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
D
w
c
m
k
P
j
+
v
I
/
E
q
5
X
r
3
U
>
A
C
N
Z
V
L
S
g
M
F
2
o
J
p
K
u
n
d
G
H
0
y
Y
O
9
7
z
Q
R
f
W
B
T
8
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
3
+
N
Z
d
k
F
X
L
H
y
V
0
m
B
D
o
r
U
>
A
C
K
c
S
J
P
E
u
I
7
f
T
R
2
w
M
/
8
W
O
9
G
v
j
Q
z
5
q
g
p
Y
n
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
m
N
q
I
/
W
Y
g
E
r
+
K
o
U
k
X
A
C
c
>
B
V
7
S
F
L
0
z
G
p
2
w
T
J
n
P
Z
u
y
j
9
R
O
D
f
d
8
H
v
3
M
Q
5
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
D
m
j
S
X
K
C
Z
J
N
z
R
E
H
A
v
>
B
/
3
c
V
7
g
F
L
0
T
G
+
o
p
Y
2
w
r
u
y
W
P
d
k
O
9
U
n
I
M
q
Q
5
f
8
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
m
N
q
I
/
W
Y
g
E
r
+
K
o
U
k
X
A
C
c
>
B
V
7
S
F
L
0
z
G
p
2
w
T
J
n
P
Z
u
y
j
9
R
O
D
f
d
8
H
v
3
M
Q
5
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
I
7
X
/
M
2
P
L
U
c
q
8
p
N
>
A
B
3
V
C
S
g
F
0
z
G
+
o
Y
w
r
m
Q
j
K
y
n
J
D
k
Z
R
W
u
9
d
O
H
5
E
T
v
f
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
m
N
q
I
/
W
Y
g
E
r
+
K
o
U
k
X
A
C
c
>
B
V
7
S
F
L
0
z
G
p
2
w
T
J
n
P
Z
u
y
j
9
R
O
D
f
d
8
H
v
3
M
Q
5
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
v
u
U
+
H
I
q
y
2
Z
J
g
E
k
>
A
B
/
3
c
V
C
7
S
N
F
L
0
X
z
G
o
p
Y
w
r
T
n
P
M
f
m
5
O
Q
9
d
W
8
D
K
j
R
arbitrary element. If the FIL of h with respect to D is bounded by η, then for any unbiased estimator ˆ z of z, we have:
Hence, a smaller FIL implies a larger variance in any unbiased attempt to infer z. Equation 3 follows directly from the Cram´er-Rao bound. Indeed, under a fairly relaxed regularity condition [Kay, 1993], for any unbiased estimator ˆ z of the data z, the Cram´er-Rao bound states that:
E [( ˆ z - z)( ˆ z - z)⊤] ⪰ Ih(z)-1 , (4) where A ⪰ B if the matrix A - B is positive semidefinite. Since the estimator is unbiased, this implies the covariance of ˆ z is similarly bounded:
If A(D) has an FIL of η then equation 2 implies Ih(z)-1 ii ≥ 1/η2 for all i, and from equation 5, Var(ˆ z) ≥ 1/η2 follows.
Correlated data. Fisher information loss also provides some security in the presence of intra-dataset correlations. If a model has an FIL bounded by η with respect to the training data (in vector form z), then the covariance matrix for any unbiased estimator ˆ z of that data is bounded by ‖Cov( ˆ z)‖2 ≥ 1/η2 . This limits the ability of an adversary using an unbiased estimator to infer relative differences between elements in the dataset.
3.1 Properties of FIL
Subsets. In many cases we are interested in measuring FIL with respect to a single example (xi, yi) ∈ D. In this case, we can compute the example-specific FIM by selecting the corresponding entries of the full FIM. Given the FIM for the vector represented data z, computing the FIM for the i-th example amounts to selecting the submatrix of size (d + 1) × (d + 1) with upper left corner Ih(z)i(d+1),i(d+1). Similarly, computing the FIM for a specific attribute over all examples, {xji }ni=1 , amounts to constructing the submatrix by selecting the corresponding entries from the full FIM. See figure 1 for an illustration.
Composition. The Fisher information, and hence FIL, compose additively. Given k independent evaluations of A(D) each with an FIL of η, the combined FIL is at most √kη. More generally, given an evaluation of k unique but independent randomized algorithms {hi ∼ Ai(D) | i = 1, . . . , k} each with an FIL of ηi, the combined FIL for all hi about D is at most (∑ki=1 η2 i )1/2 .
Algorithm 1 Iteratively reweighted Fisher information loss.
- 1: Input: Data set D, loss function ℓ(·), number of iterations T , and noise scale σ.
- 2: Initialize sample weights ω0 i ← 1.
- 5: w′ ←w∗ +b where b ∼N(0, σ2I).
- 3: for t ← 1 to T do 4: w∗ ← argminw ∑ni=1 ωt-1 i ℓ(w⊤xi, yi) + nλ 2 ‖w‖22 .
- 6: ηi ← (‖Iw′(xi, yi)‖2)1/2 .
- 7: ωt i ← nωt-1 i /ηi ∑ni=1 ωt-1 i /ηi .
- 8: end for
- 9: Return: The private weights w′ .
Closed under post-processing. As in differential privacy, FIL is closed under post-processing. If ‖Ih(D)‖2 ≤ η2 , then for any function of the hypothesis, g(h), ‖Ig(h)(D)‖2 ≤ η2 .
3.2 Computing FIL
Learning setting. We assume a linear model with parameters w ∈ Rd and minimize the regularized empirical risk:
Furthermore, we assume that the loss ℓ(w⊤xi, yi) is convex and twice differentiable. We denote by f(D) the minimizer, w∗ , of equation 6 as a function of the dataset D. When computing FIL at the example level, we let fi(x, y) be the minimizer of equation 6 as a function of the i-th data point:
Output perturbation. The definition of FIL in equation 3 only applies to a randomized learning algorithm A with a differentiable density function. To obtain such a randomized learning algorithm from the minimizer w∗ in equation 6, we adopt the Gaussian mechanism from differential privacy [Dwork et al., 2006]. The Gaussian mechanism adds zero-mean isotropic Gaussian noise to the parameters w∗ . Let A(D) = f(D) + b be the output-perturbed learning algorithm, where b ∼ N(0, σ2I). The FIM of w′ ∼ A(D) is given by:
where Jf ∈ Rd×n(d+1) is the Jacobian of f(D) with respect to D. The Jacobian Jf captures the sensitivity of the minimizer w∗ with respect to the training dataset D. The FIL of the Gaussian mechanism with scale σ is then:
Jacobian of the minimizer. We aim to compute the Fisher information loss of the Gaussian mechanism in equation 9 at the example level. This requires the Jacobian Jfi of fi(x, y), equation 7, with respect to (x, y) evaluated at (xi, yi). For a convex, twice-differentiable loss function ℓ(w⊤x, y), the Jacobian Jfi evaluated at (xi, yi) is given by:
The Hessian is computed over the full dataset Hw∗ = ∑ni=1 ∇2w ℓ(w∗⊤xi, yi) + nλI. The term ∇x,y∇wℓ = [∇x∇wℓ,∇y∇wℓ] ∈ Rd×(d+1) is the Jacobian of ∇wℓ with respect to (x, y) with entries given by:
3.3 Iteratively Reweighted FIL
Existing privacy mechanisms fail to provide equitable protection against privacy attacks for different subgroups [Yaghini et al., 2019]. We address this issue with iteratively reweighted Fisher information loss (IRFIL; Algorithm 1), which yields a model with an equal per-example FIL across all examples in D. This is done by re-weighting the per-example surrogate loss ℓ(w⊤xi, yi) over repeated computations of the minimizer w∗ . After the first iteration in Algorithm 1, the weight for the i-th example is inversely proportional to the initial ηi. At successive iterations, the per-example weight ωt i is multiplicatively updated by a value inversely proportional to the current model's FIL. The normalization on line 7 is primarily for numerical stability.
| Model | ¯ η | Accuracy |
|---|---|---|
| Linear | 0.040 ± 0.014 | 100 |
| +IRFIL | 0.047 ± 0.000 | 99.8 |
| Logistic | 0.027 ± 0.012 | 99.8 |
| +IRFIL | 0.024 ± 0.000 | 99.7 |
4 Experiments
We compute the Fisher information loss for both linear and logistic regression models on MNIST. We perform binary classification of the digits 0 and 1 using a training dataset of 12,665 examples. We normalize all inputs to lie in the unit ball, maxi ‖xi‖2 ≤ 1, and then project each input using PCA onto the top twenty principal components for the corresponding dataset.
For linear regression we do not apply L2 regularization, and for logistic regression, λ is set to the largest value such that the training set accuracy is the same up to two significant digits as that of linear regression. For linear regression, the targets are yi ∈ {-1, 1}, and we compute the exact minimizer. For logistic regression, we use limitedmemory BFGS to compute the minimizer. We typically report η assuming a Gaussian noise scale of σ = 1; hence η = ‖Jf‖2. Code to reproduce our results is available at https://github.com/facebookresearch/fisher information loss.
Validating FIL. Histograms of the per-example η are shown in figure 2, separated by output class label. For both models, the histograms have distinct modes. The mode at the larger value is for the digit 0, implying that the model in general contains more information about images of 0 than of 1. Figure 3 shows the eight images with the largest and smallest η for the linear regression model. The images with the smallest η are consistent with the class means in table 1. These correspond to the digit 1 written in a very typical manner. As expected, the images with the largest η are much more idiosyncratic. Of the 100 largest η examples, 24 overlap between the logistic and linear regression models.
Reweighted FIL. We empirically evaluate the IRFIL algorithm in figure 4, which plots the standard deviation of the per-example η against the number of re-weighting iterations. The per-example η converge to the same value after only a few iterations for both linear and logistic regression. Table 1 shows the mean and standard deviation of η, as well as the test accuracy for models trained with and without IRFIL. Neither the average FIL ¯ η nor the test accuracy are especially sensitive to the IRFIL algorithm. However, without IRFIL the standard deviation in η is significantly higher, implying that the initial information leakage varies substantially across training examples. Overall, IRFIL achieves fairness in privacy loss with little change in accuracy or average privacy loss.
5 Discussion and FutureWork
We demonstrated that Fisher information loss can be used to assess the information leaked by a model about its training data. A primary benefit of FIL over a priori guarantees like differential privacy is the ability to measure information leakage at various granularities with respect to the data at hand. This also allows FIL to be used to construct models with equi-distributed leakage, which can be done with iteratively reweighted FIL. Furthermore, since FIL explicitly measures the inferential power of an adversary it can be used by practitioners to tailor the resulting privacy to the adversary's knowledge and capabilities.
We motivated the use of FIL via the Cram´er-Rao bound and delineated the corresponding threat model. However, the assumption that the adversary is limited to unbiased estimators may not hold in the presence of auxiliary information. The implications of this should be further investigated. Furthermore, unlike differential privacy, FIL does not implicitly degrade with correlated data. This property of FIL should also be further studied.
The IRFIL algorithm closely resembles iteratively reweighted least squares (IRLS), which has been widely studied for ℓp-norm regression [Green, 1984] and sparse recovery [Daubechies et al., 2010]. While IRLS often converges rapidly in practice, the theoretical convergence rates are difficult to derive and do not reflect the empirical results [Ene and Vladu, 2019]. We also observed rapid and robust convergence with the IRFIL algorithm without any hyper-parameter tuning. Future work may help understand the convergence of IRFIL from a theoretical standpoint.
Finally, we considered FIL in the common setting of output-perturbed generalized linear models with Gaussian noise. However, many possible extensions exist in the randomization used including alternative noise distributions such as the Laplace distribution [Dwork et al., 2006], objective perturbation [Chaudhuri et al., 2011], or quantifying leakage via predictions directly using, for example, the exponential mechanism [McSherry and Talwar, 2007]. Furthermore, extending FIL to the setting of non-linear and non-convex models will facilitate its utility and broader adoption.
Acknowledgments
Thanks to Mark Tygert for essential contributions to the ideas in this work and for providing copious feedback.
@inproceedings{ijcai2022p736,
title = {Measuring Data Leakage in Machine-Learning Models with Fisher Information (Extended Abstract)},
author = {Hannun, Awni and Guo, Chuan and van der Maaten, Laurens},
booktitle = {Proceedings of the Thirty-First International Joint Conference on
Artificial Intelligence, {IJCAI-22}},
publisher = {International Joint Conferences on Artificial Intelligence Organization},
editor = {Luc De Raedt},
pages = {5284--5288},
year = {2022},
month = {7},
note = {Sister Conferences Best Papers},
doi = {10.24963/ijcai.2022/736},
url = {https://doi.org/10.24963/ijcai.2022/736},
}
TY - CPAPER TI - Measuring Data Leakage in Machine-Learning Models with Fisher Information (Extended Abstract) AU - Hannun, Awni AU - Guo, Chuan AU - van der Maaten, Laurens T2 - Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22 PB - International Joint Conferences on Artificial Intelligence Organization PY - 2022 SP - 5284 EP - 5288 DO - 10.24963/ijcai.2022/736 UR - https://doi.org/10.24963/ijcai.2022/736 ER -Copied
Machine-readable versions of this paper, for tools and AI agents:
