Data Poisoning against Differentially-Private Learners: Attacks and Defenses

1 Introduction

As machine learning is increasingly used for consequential decisions in the real world, their security has received more and more scrutiny. Most machine learning systems were originally designed without much thought to security, but researchers have since identified several kinds of attacks under the umbrella of adversarial machine learning. The most well-known example is adversarial examples [Szegedy et al., 2013], where inputs are crafted to fool classifiers. More subtle methods aim to recover training examples [Fredrikson et al., 2015], or even extract the model itself [Tram`er et al., 2016].

In data poisoning attacks [Biggio et al., 2012; Koh et al., 2018; Mei and Zhu, 2015; Alfeld et al., 2016; Li et al., 2016; Xiao et al., 2015; Mu˜noz-Gonz´alez et al., 2017; Jun et al., 2018; Zhao et al., 2017; Zhang and Zhu, 2019], the adversary corrupts examples at training time to manipulate the learned model. As is generally the case in adversarial machine learning, the rise of data poisoning attacks has outpaced the development of robust defenses. While attacks are often evaluated against specific target learning procedures, effective defenses should provide protection-ideally guaranteedagainst a broad class of attacks. In this paper, we study differential privacy [Dwork et al., 2006; Dwork, 2011] as a simple and general defensive technique against data poisoning. While the original motivation of differential privacy was originally capture notions of privacy-the output should not depend too much on any single individual's (private) data-it also provides a defense against data poisoning. Concretely, we establish quantitative bounds on how much an adversary can change the distribution over learned models by manipulating a fixed number of training items. We are not the first to design defenses to data poisoning [Steinhardt et al., 2017; Raghunathan et al., 2018], but our proposal has several notable strengths compared to previous work. First, our defense provides provable protection against a worst-case adversary. Second, our defense is general. Differentially-private learning procedures are known for a wide variety of tasks, and new algorithms are continually being developed. These learners are all resilient against data poisoning.

We complement our theoretical bounds with an empirical evaluation of data poisoning attacks against differentially private learners. Concretely, we design an attack based on stochastic gradient descent to search for effective training examples to corrupt against specific learning algorithms. We evaluate our attack on two private learners: objective perturbation [Kifer et al., 2012] and output perturbation [Chaudhuri et al., 2011]. Our evaluation confirms the theoretical prediction that private learners are vulnerable to data poisoning attacks when the adversary can poison sufficiently many examples. A gap remains between the performance of attack and theoretical limit of data poisoning attacks-it seems like differential privacy provides a substantially stronger defense in practice than in theory. We discuss possible causes and leave further investigation to future work.

2 Preliminaries

Let the data space be Z = X × Y, where X is the feature space and Y is the label space. We write D = ∪∞n=0Z n for the space of all training sets, and write D ∈ D for a particular training set. A randomized learner M : D × Rd ↦→ Θ maps a training set D and noise b ∈ Rd drawn from a distribution ν to a model in the model space Θ.

Definition 1. (Differential Privacy)Mis (ϵ, δ)-differentiallyprivate if ∀D, ˜D ∈ D that differ by one item, and ∀S ⊂ Θ,

where the probability is taken over b ∼ ν. When δ = 0, we say that M is ϵ-differentially private.

Many standard machine learning tasks can be phrased as optimization problems modeling empirical risk minimization (ERM). Broadly speaking, there are two families of techniques in the privacy literature for solving these problems. Objective perturbation injects noise into the objective function of a vanilla machine learner to train a randomized model [Chaudhuri et al., 2011; Kifer et al., 2012; Nozari et al., 2016]. In contrast, output perturbation runs the vanilla learner but after training, it randomly perturbs the output model [Chaudhuri et al., 2011]. We will consider data poisoning attacks against both kinds of learners. We first fix the adversarial attack setting.

Knowledge of the attacker: The attacker has full knowledge of the full training set D and the differentially-private machine learnerM, including the noise distribution ν. However, the attacker does not know the realized noise b.

Ability of the attacker: The attacker is able to modify the clean training set D by changing the features or labels, adding additional items, or deleting existing items, but the attacker can modify at most k items. Formally, the poisoned training set ˜D must lie in the ball B(D, k) ⊂ D, where the center is the clean data D and the radius k is the maximum number of item changes.

Goal of the attacker: The attacker aims to drive the (stochastic) model learned from the poisoned data ˜ θ = M( ˜D, b) to achieve a certain goal. Formally, we define a cost function C : Θ ↦→ R that measures how much the poisoned model ˜ θ deviates from the attack target. Then the attack problem can be formulated as minimizing an objective function J:

This formulation differs from previous works (e.g. [Biggio et al., 2012; Mei and Zhu, 2015; Xiao et al., 2015]) because the differentially-private learner produces a stochastic model, so the objective takes the expected value of C.

Example 1. (Parameter-Targeting Attack) If the attacker wants the output model to be close to a target model θ′ , the attacker can define C(˜ θ) = 12 ‖˜ θ - θ′ ‖2 . Minimizing J( ˜D) pushes the poisoned model close to θ′ in expectation.

Example 2. (Label-Targeting Attack) If the attacker wants small prediction error on an evaluation set {z∗ i }i∈[m], the attacker can define C(˜ θ) = 1m ∑mi=1 ℓ(˜ θ, z∗ i ) where ℓ(˜ θ, z∗ i ) is the loss on item z∗ i = (x∗i , y∗ i ). Minimizing J( ˜D) pushes the prediction on x∗i towards the target label y∗ i .

Example 3. (Label-Aversion Attack) If the attacker wants to induce large prediction error on an evaluation set {z∗ i }i∈[m], the attacker can define C(˜ θ) = - 1m ∑mi=1 ℓ(˜ θ, z∗ i ), a nonpositive cost. Minimizing J( ˜D) pushes the prediction on x∗i away from the true label y∗ i .

Now, we turn to our two main contributions: i) we show that differentially-private learners have a degree of natural immunity against data poisoning attacks. Specifically, in section 3 we present a lower bound on how much the attacker can reduce J( ˜D) by manipulating up to k items; and ii) as k increases, the attacker's power grows and it becomes easier to attack differentially-private learners. In section 4 we propose a stochastic gradient descent algorithm to solve the attack problem (2). To our knowledge, this is the first data poisoning attack targeting specifically differentially-private learners.

3 Differential Privacy Resists Data Poisoning

Fixing the number of poisoned items k and a differentially private learner, can a powerful attacker achieve arbitrarily low attack cost J( ˜D)? We show that the answer is no: differential privacy implies a lower bound on how far J( ˜D) can be reduced, so private learners are inherently resistant to data poisoning attacks. Previous work [Lecuyer et al., 2018] proposed defenses based on differential privacy, but for test-time attacks. While effective, these defenses require the classifier to make randomized predictions. In contrast, our defense against data poisoning only requires the learning algorithm, not the classification process, to be randomized.

Our first result is for ϵ-differentially private learners: if an attacker can manipulate at most k items in the clean data, then the attack cost is lower bounded.

Theorem 1. LetMbe an ϵ-differentially-private learner. Let J( ˜D) be the attack cost, where ˜D ∈ B(D, k), then

Note that for C ≥ 0, Theorem 1 shows that no attacker can achieve the trivial lower bound J( ˜D) = 0. For C ≤ 0, although J( ˜D) could be unbounded from below, Theorem 1 shows that no attacker can reduce J( ˜D) arbitrarily. These results follow from the differential privacy property of M. Corollary 2 re-states the theorem in terms of the minimum number of items an attacker has to modify in order to sufficiently reduce the attack cost J( ˜D).

Corollary 2. Let M be an ϵ-differentially-private learner. Assume J(D) ̸= 0. Let τ ≥ 1. Then the attacker has to modify at least k ≥ ⌈ 1ϵ log τ⌉ items in D in order to achieve

To generalize Theorem 1 to (ϵ, δ)-private learners, we need an additional assumption that C is bounded.

Theorem 3. LetMbe (ϵ, δ)-differentially-private and J( ˜D) be the attack cost, where |C| ≤ ¯ C and ˜D ∈ B(D, k). Then,

As a corollary, we can lower-bound the minimum number of modifications needed to reduce the attack cost to a certain level.

Corollary 4. LetMbe an (ϵ, δ)-differentially-private learner. Assume J(D) ̸= 0. Then

• i). (0 ≤ C ≤ ¯ C): in order to achieve J( ˜D) ≤ 1 τ J(D) for τ ≥ 1, the attacker has to modify at least the following number of items in D.

• ii). (- ¯ C ≤ C ≤ 0): in order to achieve J( ˜D) ≤ τJ(D) for τ ∈ [1,- ¯ C J(D) ], the attacker has to modify at least the following number of items in D.

Note that in (5), the lower bound on k is always finite even if τ = ∞, which means an attacker might be able to reduce the attack cost J( ˜D) to 0. This is in contrast to ϵ-differentiallyprivate learner, where J( ˜D) = 0 can never be achieved. Thus, the weaker (ϵ, δ)-differential privacy guarantee gives weaker protection against attacks.

4 Data Poisoning Attacks on Private Learners

The results in the previous section provide a theoretical bound on how effective a data poisoning attack can be against a private learner. To evaluate how strong the protection is in practice, we propose a range of attacks targeting general differentially-private learners. Our adversary will modify (not add or delete) any continuous features (for classification or regression) and the continuous labels (for regression) on at most k items in the training set. Restating the attack problem (2):

This is a combinatorial problem-the attacker must pick k items to modify. We propose a two-step attack procedure.

step I) use heuristic methods to select k items to poison.

step II) apply stochastic gradient descent to reduce J( ˜D). Step II) could lead to poisoned items taking arbitrary features or labels. However, differentially-private learners typically assume training examples are bounded. To avoid triviallydetectable poisoned examples, we project poisoned features and labels back to a bounded space after each iteration of SGD. We now detail step II), assuming that the attacker has fixed k items to poison. We will return to step I) later.

4.1 SGD on Differentially-Private Victims (DPV)

Our attacker uses stochastic gradient descent to minimize the cost J( ˜D). We first show how to compute a stochastic gradient with respect to a training item.

Proposition 5. AssumeM( ˜D, b) is a differentiable function of ˜D, then under certain conditions (details deferred to the full paper), a stochastic gradient of J( ˜D) with respect to a particular item ˜ zi is ∂C(M( ˜D,b)) ∂˜ zi , where b ∼ ν(b).

The concrete form of the stochastic gradient depends on the target private learnerM, as well as the attack cost (which encodes the type of attack). In the following, we derive the stochastic gradient for objective perturbation and output perturbation in the context of parameter-targeting attack with cost function C(˜ θ) = 1 2 ‖˜ θ - θ′ ‖2 , where θ′ is the target model.

Instantiating Attack on Objective Perturbation

We first consider objective-perturbed private learners:

where ℓ is some convex learning loss, Ω is a regularization, and Θ is the model space. Note that the noise b enters the objective via linear product with the model θ. By Proposition 5, the stochastic gradient is ∂C(M( ˜D,b)) ∂˜ zi = ∂C(˜ θ) ∂˜ zi , where ˜ θ = M( ˜D, b) is the poisoned model. By the chain rule, we have

Note that dC(˜ θ) d˜ θ = ˜ θ - θ′ . Next we focus on deriving ∂˜ θ ∂˜ zi . Since (8) is convex, the learned model ˜ θ must satisfy the following Karush-Kuhn-Tucker (KKT) condition:

By using the derivative of implicit function, we have ∂˜ θ ∂˜ zi = -( ∂f ∂ ˜ θ )-1 ∂f ∂˜ zi . We now give two examples where the base learner is logistic regression and ridge regression, respectively. In the case ℓ(θ, ˜ z) = log(1 + exp(-˜ yθ⊤ ˜x)) and Ω(θ) = 12 ‖θ‖2 , learner (8) is objective-perturbed logistic regression:

where Θ = Rd . Our attacker will only modify the features, thus ˜ yi = yi. We now derive stochastic gradient for ˜xi. Define sj = exp(yj ˜ θ⊤ ˜xj). By (9), one can verify that ∂C(˜ θ) ∂˜xi =

Note that the noise b enters into stochastic gradient through ˜ θ = M( ˜D, b).

In the case ℓ(θ, ˜ z) = 12 (˜ y - ˜x⊤θ)2 and Ω(θ) = 12 ‖θ‖2 , learner (8) is the objective-perturbed ridge regression:

where Θ = {θ ∈ Rd : ‖θ‖2 ≤ ρ}, ˜X ∈ Rn×d is the feature matrix, and ˜ y ∈ Rn is the label vector. Unlike logistic regression, the objective-perturbed ridge regression requires the model space to be bounded (see e.g. [Kifer et al., 2012]). The attacker can modify both the features and the labels. By (9), the stochastic gradient of ˜xi is ∂C(˜ θ) ∂˜xi =

where µ is the dual variable of constraint 12 ‖θ‖2 ≤ ρ2 2 when solving (12). The stochastic gradient of ˜ yi is

Note that the noise b enters into the stochastic gradient through ˜ θ = M( ˜D, b).

Instantiating Attack on Output Perturbation

We now consider the output perturbation mechanism:

The stochastic gradients for this target are similar to those for objective perturbation. Again, we instantiate on two examples where the base learner is logistic regression and ridge regression, respectively.

The output-perturbed logistic regression takes the following form:

Let sj = exp(yj(˜ θ - b)⊤ ˜xj), then we have ∂C(˜ θ) ∂˜xi =

The output-perturbed ridge regression takes the following

where Θ = {θ : ‖θ‖2 ≤ ρ}. Let µ be the dual variable for the constraint 12 ‖θ‖2 ≤ ρ2 2 when solving the argmin in (15). Then the stochastic gradients of ˜xi and ˜ yi are

4.2 SGD on Surrogate Victims (SV)

We also consider an alternative, simpler way to perform step II). When b = 0, the differentially-private learning algorithms revert to the base learner which returns a deterministic model M( ˜D, 0). The adversary can simply attack the base learner (without b) as a surrogate for the differentially-private learner M (with b) by solving the following problem:

Since the objective is deterministic, we can work with its gradient rather than its stochastic gradient, plugging in b = 0 into all derivations in section 4.1. Note that the attack found by (16) will still be evaluated with respect to differentiallyprivate learners in our experiments.

4.3 Selecting Items to Poison

Now, we return to step I) of our attack: how can we select the k items for poisoning? We give two heuristic methods: shallow and deep selection.

Shallow selection selects the k items with the largest initial gradient norm ‖ ∂J( ˜D) ∂˜ zi | ˜D=D ‖. Modifying these items will reduce the attack cost the most, at least initially. The precise gradients to use during selection depend on the attack in step II). When targeting differentially-private learners directly, computing the gradient of J( ˜D) is difficult. We use Monte-Carlo sampling to approximate the gradient gi of item zi: gi ≈ 1m ∑ms=1 ∂ ∂˜ zi C(M( ˜D, bs)) | ˜D=D , where bs are random samples of privacy parameters. Then, the attacker picks the k items with the largest ‖gi‖ to poison. When targeting surrogate victim, the objective is C(M( ˜D, 0)), thus the attacker computes the gradient of C(M( ˜D, 0)) of each clean item zi: gi = d d˜ zi C(M( ˜D, 0)) | ˜D=D , and then picks the k items with the largest ‖gi‖.

Deep selection selects items by estimating the influence of an item on the final attack cost. When targeting a differentiallyprivate victim directly, the attacker first solves the following relaxed attack problem: min ˜D J( ˜D) + αR( ˜D). Importantly, here ˜D allows changes to all training items, not just k of them. R( ˜D) is a regularizer penalizing data modification with weight α > 0. We take R( ˜D) = ∑ni=1 r(˜ zi), where r(˜ zi) is the distance between the poisoned item ˜ zi and the clean item zi. We define r(˜ zi) = 12 ‖˜xi -xi‖2 for logistic regression and r(˜ zi) = 12 ‖˜xi -xi‖2 + 12 (˜ yi -yi)2 for ridge regression. After solving the relaxed attack problem with SGD, the attacker evaluates the amount of change r(˜ zi) for all training items and pick the top k to poison. When targeting a surrogate victim, the attacker can solve the following relaxed attack first: min ˜D C(M( ˜D, 0))+αR( ˜D), and then select the k top items.

In summary, we have four attack algorithms based on combinations of methods used in step I) and step II): shallow-SV, shallow-DPV, deep-SV and deep-DPV. In the following, we evaluate the performance of these four algorithms.

5 Experiments

We now evaluate our attack with experiments, taking objectiveperturbed learners [Kifer et al., 2012] and output-perturbed learners [Chaudhuri et al., 2011] as our victims. Throughout, we use α = 10-4 for deep selection and fix a constant step size η = 1 for (stochastic) gradient descent. After each iteration of SGD, we project poisoned items to ensure feature norms are at most 1 and labels are in [-1, 1].

5.1 Attacks IntelligentlyModify Data

As a first example, we use label-aversion attack to illustrate that our attack modifies data intelligently. The victim is an objective-perturbed learner for ϵ-differentially private logistic regression, with ϵ = 0.1 and regularizer λ = 10. The training set contains n = 21 items uniformly sampled in the interval [-1, 1] with labels yi = 1 [xi ≥ 0], see Figure 1(a). To generate the evaluation set, we construct m = 21 evenly-spaced items in the interval [-1, 1], i.e., x∗i = 0.1i,-10 ≤ i ≤ 10

and labeled as y∗ i = 1 [x∗i ≥ 0]. The cost function is defined as C(˜ θ) = - 1m ∑mi=1 log(1 + exp(-y∗ i x∗i ⊤˜ θ)). To achieve small attack cost J, a negative model ˜ θ < 0 is desired. 1 We run deep-DPV with k = n (attacker can modify all items) for T = 300 iterations. In Figure 1(a), we show the position of the poisoned items on the vertical axis as the SGD iteration, t, grows. As expected, the attacker flips the positions of positive (blue) items and negative (red) items. As a result, our attack causes the learner to find a model with ˜ θ < 0.

5.2 Attacks Can Achieve Different Goals

We now study a 2D example to show that our attack is effective for different attack goals. The victim is the same as in section 5.1. The clean training set in Figure 1(b) contains n = 317 items uniformly sampled in a unit sphere with labels yi = 1 [x⊤i θ∗ ≥ 0] where θ∗ = (1, 1). We now describe the attack settings for different attack goals.

For label-aversion attack, we generate an evaluation set conθ)).

taining m = 317 grid points (x∗i , y∗ i ) lying in a unit sphere and labeled by a vertical decision boundary, see Figure 1(c). The cost function is C(˜ θ) = - 1m ∑mi=1 log(1 + exp(-y∗ i x∗i ⊤˜ For label-targeting attack, the evaluation set remains the same as in the label-aversion attack. The cost function is C(˜ θ) = 1m ∑mi=1 log(1 + exp(-y∗ i x∗i ⊤˜ θ)).

For parameter-targeting attack, we first train vanilla logistic regression on the evaluation set to obtain the target model θ′ = (2.6, 0), then define cost function as C(˜ θ) = 12 ‖˜ θ - θ′ ‖2 .

We run deep-DPV with k = n for T = 5 × 103 iterations. In Figure 2(a)-(c), we show the poisoning trajectories for the three attack goals respectively. Each translucent point is an original training point, and the connected solid point is its final position after attack. The curve connecting them shows the trajectory as the attack algorithm gradually poisons the data.

In label-aversion attack the attacker aims to maximize the logistic loss on the evaluation set. It ends up moving positive (negative) items to the left (right), so that the poisoned data deviates from the evaluation set.

In contrast, the label-targeting attack tries to minimize the logistic loss, thus items are moved to produce a poisoned data aligned with the evaluation set. However, the attack does not produce exactly the evaluation set. To understand that, we compute the model learnt by vanilla logistic regression on the evaluation set and the poisoned data, which are (2.60,0) and (2.94,0.01). Note that both models are aligned with the

1 Differentially-private machine learning typically considers homogeneous learners, thus the 1Dmodel is just a scalar.

(d) label-aversion (e) label-targeting (f) parameter-targeting

Figure 2: Poisoning trajectories in 2D.

evaluation set, but the latter has larger norm, which leads to smaller attack cost, thus our result is a better attack.

In parameter-targeting attack, we compute the model learnt by vanilla logistic regression on the clean data and poisoned data, which are (1.86,1.85) and (2.21, 0.04). Note that the attack pushes the model closer to the target model (2.6,0).

To quantify the attack performance, we evaluate the attack cost J( ˜D) via Monte-Carlo sampling. We run private learners Te = 103 times, each time with a random bs to produce a cost Cs = C(M( ˜D, bs)). Then we average to obtain an estimate of the cost J( ˜D) ≈ 1 Te ∑Te s=1 Cs. In Figure 2(d)-(f), we show the attack cost J( ˜D) and its 95% confidence interval (green, ±2∗stderr) up to t = 103 iterations. J( ˜D) decreases in all plots, showing that our attack is effective for all attack goals.

5.3 Attacks Become More Effective as k Increases

The theoretical protection provided by differential privacy weakens as the adversary is allowed to poison more training items. We can show empirically that our attacks indeed become more effective as the number of poisoned items k increases. The data set is the same as in section 5.2. We consider k from 20 to 100 in steps of 20. For each k, we run our four attack algorithms for T = 5 × 103 iterations and estimate the attack cost J( ˜D) on Te = 2 × 103 samples. Figure 3(a)-(c) show that the attack cost J( ˜D) decreases as k grows, indicating better attack performance. We also show the poisoning trajectory produced by deep-DPV with k = 10 in Figure 3(d)-(f). The corresponding attack costs J( ˜D) are -0.60, 0.49 and 3.43 respectively. Compared to that of k = n,

-1.79, 0.32 and 1.20, we see that poisoning only 10 items is not enough to reduce the attack cost J( ˜D) significantly, although the attacker is having some effect.

5.4 Attacks Effective on Both PrivacyMechanisms

We now show experiments on real data to illustrate that our attacks are effective on both objective and output perturbation mechanisms. We focus on label-targeting attack.

The first real data set is vertebral column from the UCI Machine Learning Repository [Dua and Karra Taniskidou, 2017].

The data set contains 6 features of 310 orthopaedic patients, and the task is to predict if the vertebra of any patient is abnormal (positive class) or not (negative class). We normalize the features so that the norm of any item is at most 1. Since this is a classification task, we use private logistic regression with ϵ = 0.1 and λ = 10 as the victim. To generate the evaluation set {(x∗i , y∗ i )}, we randomly pick one positive item and find its 10 nearest neighbors within the positive class, and set y∗ i = -1 for all 10. Intuitively, this attack targets a small cluster of abnormal patients and the goal is to mislead the learner to classify them as normal. The other experimental parameters remain the same as in section 5.2. Note that in order to push the prediction on x∗i toward y∗ i , the attacker requires y∗ i x∗i ⊤˜ θ > 0, which indicates C(˜ θ) = log(1 + exp(-y∗ i x∗i ⊤˜ θ)) < log 2 ≈ 0.69, so should J( ˜D). As shown in Figure 4, the attacker indeed reduces the attack cost J( ˜D) below 0.69 for both privacy mechanisms, thus our attack is successful.

The second data set is red wine quality from UCI. The data set contains 11 features of 1598 wine samples, and the task is predict the wine quality, a number between 0 and 10. We normalize the features so that the norm of any item is at most 1. Labels are also normalized to ensure the value is in [-1, 1]. The victim is private ridge regression with ϵ = 1, λ = 10, and bound on model space ρ = 2. To generate the evaluation set, we pick an item x∗i with the smallest quality value, and set the target label to y∗ i = 1. The cost function is defined as C(θ) = 1 2 (x∗i ⊤θ - y∗ i )2 . The other experimental parameters are the same as in section 5.2. This attack aims to force the learner to predict a low-quality wine as having a high quality. Figure 5 shows the result. Note that the attack cost can be significantly reduced for both privacy mechanisms even if the attacker poisons only 100/1598 ≈ 6.3% items of the data set.

We make two overall observations. First, deep-DPV is in general the most effective method among our four attacks. Second, there remains a gap between the performance of deepDPV and the theoretical bound, leaving space to design more efficient attacks or to tighten the theoretical bound.

5.5 Attack Is Easier withWeaker Privacy

Finally, we show that the attack is more effective as the privacy guarantee becomes weaker, i.e., as we increase ϵ. To illustrate, we run experiments on the data set in section 5.2, where we fix k = 100 and vary ϵ from 0.1 to 0.5. In Figure 6, the attack cost approaches the lower bound as ϵ grows. This means weaker privacy guarantees (smaller ϵ) lead to easier attacks.

6 Conclusion and FutureWork

We showed that differentially private learners are resistant to data poisoning attacks, with the protection degrading as the attacker poisons more items. We proposed attack algorithms and demonstrated that our attacks are effective on different privacy mechanisms and machine learners. There remains a gap between the theoretical limit and the empirical performance of our attacks; closing the gap remains future work.

Acknowledgments

We thank Steve Wright for helpful discussions. This work is supported in part by NSF 1545481, 1561512, 1623605, 1704117, 1836978, the MADLab AF Center of Excellence FA9550-18-1-0166, a Facebook TAV grant, and the University of Wisconsin.