Online Evasion Attacks on Recurrent Models:The Power of Hallucinating the Future

1 Introduction

Deep neural networks (DNN) are discovered to be surprisingly vulnerable to imperceptibly small input noises [Szegedy et al., 2014; Goodfellow et al., 2015]. Many different types of vulnerabilities of DNNs have been demonstrated in various tasks, including classification, regression, and generative methods [Ma¸dry et al., 2018; Gondim-ribeiro et al., 2018; Dang-Nhu et al., 2020]. Most attacks have focused on offline evasion attacks on non-temporal models, such as the image classification model, where an attacker has access to the whole input example, such as an image.

However, there are many security-critical tasks that involve recurrent neural networks (RNN) performed in an online fashion [Suradhaniwar et al., 2021; Pinto et al., 2021; Huang et al., 2020]. For instance, mortality prediction [Harutyunyan et al., 2019] continuously monitors hospitalized patients for early warning, and an autonomous driving agent that uses sensors to decide the steering angle of the vehicle [Kiran et al., 2021]. Unlike the offline setting, an attacker cannot observe the entire input sequence because inputs arrive as a real-time stream to a victim and an attacker.

∗Corresponding author.

Previous works [Xie et al., 2020; Fawaz et al., 2019; DangNhu et al., 2020; Oregi et al., 2018] evaluated vulnerabilities of RNNs but they implicitly assumed that future inputs are observable, which is implausible (Figure 1-A, dashed box). Meanwhile, [Gong et al., 2019] introduced a framework for real-time attack (Figure 1-B) with two constraints unique to the online problem: 1) future inputs are unobservable, and 2) the past inputs are unchangeable. However, the framework is rather specific to speech recognition (Figure 1-B, red boxes), where classification is performed only once after receiving the entire speech. Such an approach is inapplicable to dynamic online tasks such as autonomous driving, where the agent has to continuously decide the steering angle.

In this paper, we propose a more general framework of online evasion attacks 1 allowing a victim recurrent model to make continuous predictions or decisions (Figure 1-C, circled G and a round box). Our framework can accommodate various adversarial objectives on RNN to address different attack scenarios. In particular, our framework makes time-varying adversarial objectives possible, unlike previous approaches to attack at a specific time. The versatility of our framework will allow for a comprehensive robustness study of recurrent models. To showcase of the versatility, we reformulate the objective of real-time attack [Gong et al., 2019], and present novel adversarial objectives such as Time-window and Surprise objectives using our framework.

As an effective solution to our framework, we propose a novel white-box attack called Predictive Attack (Figure 2B). An ideal solution to the online problem is the clairvoyant attack (Figure 2-A), where an attacker does not suffer from the online constraints, foreseeing the entire future input. Thus the clairvoyant attack can find attack perturbations with existing offline methods [Ma¸dry et al., 2018; Croce and Hein, 2020]. Instead of clairvoyance, Predictive Attack 'hallucinate' the future (Figure 2-B, green box) with a trained predictive model of input sequences, mimicking the crystal ball of a clairvoyant. Since accurate prediction can be difficult, we propose an additional alternative attack called IID Attack that replaces accurate prediction with IID sampling. They perform surprisingly well, and we ascribe this to the importance of considering the hidden states and input orders when attacking recurrent models.

1 https://github.com/byunggilljoe/rnn online evasion attack. Appendix is included.

We evaluate our attacks using six datasets. Our predictive attack approaches 98% of the performance of the clairvoyant on average. We perform further empirical analysis of the predictive attacks demonstrating the versatility of our framework and attack robustness. We summarize our contributions as follows.

• We introduce a general formulation of online evasion attacks on recurrent models, which can accommodate various types of attack objectives and constraints, allowing for a comprehensive study of robustness.
• We propose two novel white-box attacks, Predictive Attack and IID Attack, based on hallucination of the future to emulate the ideal clairvoyant attacker.
• We evaluate the performance of our attacks under various conditions using real-world data and demonstrate the versatility and robustness of our framework and attacks.

2 Setting

Inputs and Outputs. An input stream/sequence x of length L is a sequence of n-dimensional vectors (x1, x2, ..., xL) ∈ RL×n where the index refers to time step. Similarly, the output sequence y of length L is sequence of outputs (y1, y2, ..., yL) where yi ∈ R for a regression problem and yi ∈ {1, ..., C} for a classification problem.

Victim Task. We attack recurrent neural networks (RNN) that continuously predict the output at each time step. Formally, an RNN is a pair of functions fθ and gθ. At time t, fθ : Rn × Rm → R predicts the current output by yt = fθ(xt, ht) using the current input xt and the hidden state ht ∈ Rm. The dynamics of the RNN is determined by gθ : Rn ×Rm → Rm which maps (xt, ht) to the next hidden state by ht+1 = gθ(xt, ht).

Threat Model. We assume attackers have white-box access to a victim model. Attackers can define an attack objective and loss and can compute the derivative of the loss with respect to an input. Also, Attackers have access to some examples of input streams.

3 Online Evasion Attack Framework

Problem. The attacker aims to mislead a victim RNN model (fθ, gθ) to output the (adversarial) target labels or values (ya 1 , · · · , yaL) by using the perturbed input sequence (x1 + δ1, · · · , xL + δL). This is done by minimizing 2 the aggregate value of the losses Ladv 1 , · · · ,Ladv L :

Ladv i is the loss at time i: Ladv i = L(fθ(xi + δi, hδi ), ya i ), and hδi is the hidden state of the RNN at time i:

The Agg(·) refers to a method of temporal aggregation, and ∆ refers to any constraint on the perturbation sequence. Compared to previous work [Gong et al., 2019], the present formulation (Equation 1∼2) is much more flexible since the loss and the target are allowed to be time-varying. For concreteness, we will use the temporal summation Agg(L1, · · · ,Ll) = ∑Li=1 Li and the ℓp constraint ∆ = {∥δi∥p ≤ ϵ, ∀i} by default:

Online Constraints. Critically different from the muchstudied offline attacks, an online attack has to follow physical constraints [Gong et al., 2019]. Firstly, an attacker cannot perturb the future or the past input but only the current input. Therefore, to solve Equation 3 an attacker has to solve

at each time step t = 1, · · · ,L, which is the core of the general online attack. Since the losses of the past (i < t) are unchangeable they do not appear in the sum of the losses. An important thing to note is that the current perturbation δt affects all future losses Ladv t+1 , Ladv t+2 , · · · due to the nature of RNNs, which we call victim model dynamics property. A

2 The current description is for targeted attacks but we can also perform untargeted attacks as well.

successful online attack therefore has to exploit this property. Furthermore, the sum can be rewritten as

As the equation shows, this optimization problem cannot be solved directly due to the second constraint of the online attack: we do not know the future inputs xt+1, xt+2, · · · . Although seemingly impossible, we make it possible by exploiting the temporal dependence of the inputs in a stream, on which the decisions of RNN depend ultimately. (Details in the next section.) To mount a successful attack, an online attacker has to exploit both victim model dynamics and temporal dependence.

3.1 Greedy and Clairvoyant Attack

We propose two reference attacks that exemplify a crude attack and an ideal attack. These attacks can help us understand the other attacks in the following sections. Greedy Attack (Figure 3-A) provides a lower bound of attack performance which does not consider the victimmodel dynamics and the temporal dependency. This attack only considers the current loss (A) of Equation 5. Clairvoyant Attack (Figure 3-B) is an ideal, unrealizable attack that assumes the full observability of the future part of an input sequence; thus can fully use the victim model dynamics (Figure 3-B, red line), and temporal dependence (Figure 3-B, blue line). The Clairvoyant provides the upper bound of performance an attack can achieve.

4 Method

4.1 Attacks using Future Hallucination

Since the ideal Clairvoyant Attack is impossible, we replace the true future with a 'hallucination' of it. We propose two methods for the hallucination: using a predictive recurrent

Algorithm 1 Predictive Attack at time t.

• 1: count ← 0
• 2: while count < MAX COUNT do
• 3: Ltotal(δt) ←Ladv(xt, ya t , δt, hδt ) . + ∑t+K i=t+1 EQϕ(xi|x:i-1)[Ladv(xi, ya i , δi, hδi )] . using Monte-Carlo to compute EQϕ [·].
• 4: ∀i ∈ [t, t +K],
• 5: δi ←- Π∥δi∥p≤ϵ[δi -αsign(∇δi Ltotal(δt)]
• 6: δi ←- clip(xi + δi) -xi

.

It forces a valid range of perturbed inputs.

• 7: count ←- count +1
• 8: end while
• 9: return δt

model (Predictive Attack), and random data substitution (IID Attack).

Predictive Attack. Predictive Attack (Figure 3-1) uses a future predictive model to mimic Clairvoyant Attack. We define the attack objective of Predictive Attack as follows:

Due to linearity the second term can be simplified as

Instead of directly modeling the distribution p(xt+1:|x:t), we undertake the easier task of generating the future input with a (stochastic) generative model Qϕ(xt+1|x:t) that predicts the next input xt+1 given x:t (Figure 3-1, blue). We restrict the number of the prediction steps to K, called lookahead since we cannot consider all future inputs with finite resources.

We use another RNN to model the generator Qϕ to predict the next input xt+1 from x:t, using examples of input sequences as a training dataset. (Model details are in Experiments and Appendix B.)

Algorithm 1 describes Predictive Attack's update rule for δt, which is a variant of [Ma¸dry et al., 2018]. The hyperparameters MAX COUNT and α determine the number of updates and the step size of an update. We elaborate more on this in Appendix A.

IIDAttack. Hallucinating the future based on an accurate Qϕ can be difficult due to the test-time cost of the prediction or the training-time cost of Qϕ. To relieve this, we present a heuristic, IIDAttack, to replace the prediction model. IID Attack (Figure 3-2) simply ignores the temporal dependence and predicts the future using IID sampling of the input data (Figure 3-2, green), that is, using Ep(xi)[·] instead of Ep(xi|x:t)[·] in Equation 6. Practically, this can be done by collecting a sufficient number of past input data and randomly choosing one of them as an IID example. Even with the incorrect prediction of IID, it is still using the victimmodel dynamics (Figure 3-2, red). Such a consideration makes a big difference compared to the current-only greedy perturbation δt as we will see.

4.2 Incorporating Different Objectives

The general form of the framework (Equation 1) allows various attacks through the choice of Agg(·) and the constraint ∆. To showcase our framework's versatility, we choose γi-weighted sum as an instance of Agg(·):

In the following, we present three example attacks possible with this aggregation.

Real-time Attack. The real-time attack [Gong et al., 2019] is a special case of this formulation when γi = 0 for i < L and γL = 1, which aims to mislead the last victim output.

Time-window Attack. This attack causes misclassification/prediction at only at specific times interval [a, b]. This can be useful when 1) the attack has a more impact at specific times, or 2) the attacker has to avoid detection for a time interval where the victim is vigilant. We can implement this attack by setting y = ya i , γi = 1 if i ∈ [a, b] and y = yi, γi = τ(> 0) otherwise in Equation 8.

Surprise Attack. Surprise Attack induces untargeted error abruptly by maximizing the difference between the maximum error and the mean error over time:

This attack prevents a victim from reacting properly, thus causing more damage with the same error. For example, an abrupt steering angle change will be more damaging to an autonomous vehicle than a smooth angle change over time.

5 Experiment

We evaluate our attacks to answer the following research questions. RQ1. How much does Predictive Attack improve the attack performance?, RQ2. How versatile is our online evasion attack framework? RQ3. How robust is Predictive Attack?

Datasets. We use six datasets for classification and regression in our evaluations as summarized in Table 1.

• MNIST [LeCun et al., 1998]: Given a column sequence (vertical lines) of a digit image, predict the correct label of each column. Two classes, 3 and 8, are selected.
• FashionMNIST [Xiao et al., 2017]: The same format as MNIST but containing clothing images. We use 10 classes for a harder classification problem.
• Mortality [Harutyunyan et al., 2019]: Given a sequential medical record, predict a patient's mortality every hour.
• User [Casale, 2014]: Given a sequence of x-y-z accelerations from a user, predict the user of the sequence.
• Udacity [Gonzalez et al., 2017]: Given a sequence of camera images, predict steering angles. We resized the images to 64 x 64 and sequence duration is 0.67s.
• Energy [Candanedo et al., 2017]: Given a sequence of 27 weather sensors, predict electricity consumption of a building.

Model Parameters. All models, except for Udacity, consist of one LSTM layer followed by two linear layers with ReLU activations. For Udacity, we use CNN-LSTM as a victim model, and CrevNet [Yu et al., 2020] as Qϕ to deal with the high-dimensional images. More model details are in Appendix B.We the Adam optimizer for training with a learning rate of 1e-4. Table 1 summarizes victim's clean performances. We use ROC-AUC for Mortality to be comparable to the original reports [Harutyunyan et al., 2019].

Attack Target and Performance Metric. To evaluate the proposed framework and the attack, we use time-varying target outputs for both classification and regression as depicted in Figure 4. It is intended to simulate the dynamic nature of real online attacks better. Appendix F contains more results with other target patterns. An effective attack should achieve high TASR and low TMSE. TASR (Targeted Attack Success Ratio) is the number of time steps where a victim model yields targeted labels, over the total number of time steps L. TMSE (Targeted Mean Squared Error) is the mean squared error between victim model outputs and the targeted values. We use temporal summation as an attack objective (Equation 3) if not specified.

Miscellaneous. The input values range from 0 to 1, and we use ℓ∞ norm constraints for all tests. We set MAX ITERS = 100, and α = 1.5ϵ/MAX ITERS. We report average results of three experiment repetitions retraining a victimmodel initialized with random weights. Predicted inputs and the perturbed inputs of our attack are presented in Appendices D and E.

5.1 Performance Evaluation

In Figure 5, we answer RQ1 by comparing the performance of Predictive Attack with Greedy and Clairvoyant. The x-axis is ϵ, ℓ∞ norm of a perturbation, and the y-axis is the performance metric. On average at maximum ϵ of each plot, the performance of Predictive Attack (straight blue) approaches 98% of Clairvoyant Attacks's TASR (green). Predictive Attack also performs 138% of Greedy Attack's TASR (red). In particular, it is worth noting that safety-critical tasks such as Mortality and Udacity are more prone to attacks than the toy datasets, MNIST and Fashion MNIST.

IID Attack's comparable performance (93% of Predictive Attack on average) to that of Predictive Attack shows the importance of victim model dynamics. In particular, in Mortality, IID Attack shows the closest performance to Predictive Attack. We surmise the victim model is more dependent on model dynamics to solve the mortality prediction task. For example, a patient's current severity may depend on a medical record several hours ago, not on the current medical record.

For a qualitative analysis, we visualize the results in Figure 6. We chose (a) Udacity, (b) Mortality, and (c) Energy for visualization. In each figure section, the three rows correspond to benign examples, corresponding adversarial examples, and victim model outputs, respectively. The x-axis is time. We can see that Predictive Attack (blue) closely follows the target values or labels (dashed black) much better than Greedy Attack (red) can.

In Figure 7, we investigate the effect of lookahead K. The x-axis is ϵ, and the y-axis is attack performance. The color of a bar represents a different K. As a result, we find that there is an optimal K. We attribute this to the limitation of Qϕ. By increasing K until Qϕ can predict accurately, the attack can use the longer temporal dependence and victimmodel dynamics, leading to the performance improvement. However, if K exceeds the limit, the attack performance decreases because of incorrect future inputs and their wrong perturbations. An attack time should be short to perturb more inputs in a limited time interval. We measured the time per a time step for Predictive Attack to reach 90% of the saturated performance (when MAX ITERATION is used). For Mortality and Energy, the time is short enough, 0.03 secs and 0.05 secs, considering 3600 secs and 600 secs of each dataset's time step duration. For Udacity, it takes 0.25 secs, which is longer than usual duration of camera input, 0.03 secs. However, we believe that the time can be reduced by using a dedicated hardware or compressing the predictor model.

5.2 Versatility of the Attack Framework

Evaluating the effectiveness of different objectives from our framework, we answer RQ2, versatility of our framework.

Real-time Attack. In Figure 8, we test a real-time attack objective [Gong et al., 2019] as a special case of our framework. A real-time attacker aims to mislead the last output of a victimmodel and is untargeted; thus, y-axis is "Fool Rate" that means a frequency of wrong victim decisions. Imitation learning-based real-time attack [Gong et al., 2019] is reimplemented, referring to the public codes 3 (See Appendix K for detail explanations.). Predictive Attack surpasses the imitation learning-based attack (88∼100% vs. 24∼94% of experts' performance). Note that this gain of Predictive Attack comes at the cost of solving PGD, unlike the imitation learning-based attack that depends on a pre-trained agent. Predictive Attack is good for achieving a high attack performance, while imitation learning-based attack is suitable for fast attacks.

Time-windowAttack. We demonstrate the attack with temporal specificity. We chose the interval [a = 3/4L, b = L] as the intended window of error. In Figure 9, we present the performance of Predictive Attack with/without the Time-window objective. The x-axis is time. We set the y-axis as "Fool Rate" and "Fool MSE" (MSE between true values and victim outputs) . An ideal attack should cause non-zero values only in [3/4L,L]. Predictive Attack fulfills this objective and increases the error after t=a in contrast to non-window attacks. We also find τ controls the trade-off between attack performance and compliance with the time-window.

Surprise Attack. We conduct Surprise Attack experiment with the autonomous driving task from Udacity, where Surprise Attack can be practically important. We define Surprise Error as maxi|yi -f(xi, hδi )| -meani|yi -f(xi, hδi )|. In Figure 10, Predictive Attack with Surprise objectives achieves about 2.09 times higher Surprise Error than a naive Predictive Attack and Greedy at ϵ = 0.08.

3 https://github.com/YuanGongND/realtime-adversarial-attack

5.3 Robustness Evaluation

To answer RQ3, we evaluate Predictive Attack under a variety of unseen situations in our attack framework.

Incorrect Future Prediction. We investigate the performance of Predictive Attack under degraded Qϕ. To control the prediction quality, we replace a predicted future input xt with xηi = (1-η)xi +ηe, where e is a uniform random variable in the valid input range. In Table 2, although slightly decreased as the noise is added (η = 0.4), Predictive Attack performs better than Greedy Attack, using the victim model dynamic as consist with the case of IID Attack in Figure 5.

UnknownModel Parameters. To evaluate the robustness under limited victim information, a transfer attack [Liu et al., 2017] is conducted (Table 3). We assume a gray-box threat model where an attacker knows a victim model's architecture but not model parameters. Adversarial examples generated from an attacker-trained surrogate model are transferred to the actual victim model. Transfer attack achieves average 63% performance of white-box attack in the classification tasks, up to 88% in Mortality.

6 Conclusion

This paper introduces a general framework for online evasion attacks on recurrent models. Our framework can accommodate various time-varying attack objectives and constraints, allowing a comprehensive robustness analysis. Based on our framework, we propose Predictive Attack and IID Attack. The success of these attacks highlights the new surface of attack for recurrent models, which need to be addressed. However, defense in the online setting has not been fully studied yet, while existing offline defenses [Ma¸dry et al., 2018; Zhang et al., 2019] are not suitable for online tasks. We leave it as future work to investigate online defense methods.

Acknowledgements

This work was supported in part by ERC (NRF2018R1A5A1059921) and IITP (2020-0-00209) funded by the Korea government(MSIT). It is also supported in part by the NSF EPSCoR-Louisiana Materials Design Alliance (LAMDA) program #OIA-1946231.