Spear: Evaluate the Adversarial Robustness of Compressed Neural Models

1 Introduction

As machine learning evolves, it becomes a unilateral and narrow viewpoint to evaluate the neural networks without perturbations. Prior works [Szegedy et al., 2014] [Goodfellow et al., 2014] have proved that the neural model will behave differently by adding negligible perturbations to specific input cases. Sometimes, the different behaviors are caused by the numerical difference, finally leading to the different tolerance to the corner cases in benign input samples. However, the situation worsens if some imperceptible perturbations are intentionally added to the benign samples. For example, if the on-purpose perturbations lead the medical auxiliary diagnosis model to the wrong focus of infection, the neural model will become a murder instrument. If the on-purpose perturbations lead the Conversational AI to generate discriminatory, intimidatory, and racist statements, it will lead to severe social problems [Sap et al., 2020]. Generating such perturbations to mislead the neural models' behavior is referred to as an adversarial attack. Most prior adversarial attack studies have been successfully applied to dense neural networks, mainly for image classification tasks. Some other studies discuss the interaction between model compression and adversarial robustness. (More discussions can refer to 2.2.) However, they mainly focus on applying the adversarial workflow to defend against the attack and improve the robustness. There has been very little work to study the specific attack algorithms on the compressed networks performing various tasks.

∗Tao Chen and Zhongxue Gan are corresponding authors.

When we try to use a compressed neural model as the substitute for an original dense model, we usually ensure this compressed neural model has a similar model accuracy on benign samples. To further check whether a compressed neural model is a qualified substitute with perturbations, the prior practice generates the adversarial attack samples with effective perturbations according to the original model. Then, it evaluates the behavior of the compressed neural model with these adversarial attack samples. Suppose the compressed neural model's behavior is similar to the original model; we regard the compressed neural model as having similar adversarial robustness on par with its dense counterpart. This traditional evaluation workflow is shown in Figure 2 Left.

However, such prior practice ignores whether some specific adversarial attack algorithms can generate the adversarial attack samples that only influence the compressed model while keeping the behavior of the original model as usual, as shown in Figure 2 Right. Here are two examples. In Line 1 of Figure 1, the dense model classifies [Zhou et al., 2016] the attacked input image into the tench category. In contrast, the compressed model classifies the same image into the reel category. The adversarial attack algorithm does not influence the classification behavior of the dense model. However, it confuses the classification behavior of the compressed model by changing its decision from one to another evident category in the image. In Line 2 of Figure 1, the dense Mask RCNN model [He et al., 2017] successfully segments the traffic lights, the car in the background, and the person walking across the pedestrian crossing from the attacked input image. The compressed model segments all these object regions in the ground truth image; however, it regards the person as the snowboard under such a specific adversarial attack. It may lead to a fatal disaster if the autopilot system uses the wrong segmentation result, which controls the autonomous vehicle.

The compressed models have reduced weight parameters with sparse patterns or quantized data types. For intentional attackers, these characteristics can be utilized to discover the potential flaw of the compressed models and eventually be targeted by the specific adversarial attack algorithm. So, in this paper, we mainly focus on exploring the specific adversarial attack for the compressed neural models and further evaluate the adversarial robustness. On the other hand, the adversarial attack algorithm study for the compressed model is much more complicated than designing the attack for the dense model. Various model compression methods like pruning, distillation, and quantization generate the compressed models with different patterns, like different sparse granularities and data formats. Our work designs a unified adversarial robustness evaluation framework that is effective, efficient, and general to most model compression patterns, which is not fully addressed in the previous works.

2 RelatedWork

2.1 Adversarial Attack

Adversarial attack [Kianpour and Wen, 2019], a technique that attempts to fool machine learning models with deceptive data, is a growing threat in the AI industry and research community [Cheng et al., 2021]. The ultimate goal is to cause the specific malfunction in a machine learning model. An adversarial attack might entail presenting a model with misleading data during its training process or directly introducing maliciously designed data to deceive an already trained model.

The adversarial attack methods can be divided into the white-box attack and the black-box attack. In white-box attack settings, the adversaries have access to the parameters and structure of the target model, and the training dataset. Instead, the adversaries have no access to or little knowledge of the target model in the black-box attack scenarios. Still, they can train a local substitute model [Papernot et al., 2017] by querying the target model or applying the model inversion method. [Szegedy et al., 2013] found the deep neural models learn input-output mappings that are somewhat discontinuous to a significant extent, causing the model to misclassify an image by applying a certain hardly perceptible perturbation, namely adversarial samples. They proposed the box-constrained Broyden-Fletcher-Goldfarb-Shanno (BFGS) algorithm to search for these adversarial samples. Goodfellow et al. [Goodfellow et al., 2014] proposed the Fast Gradient Sign Method (FGSM), which generates the adversarial examples with one gradient sign step. They involved these adversarial examples during the back-propagation, which is the rudiment of adversarial training. [Papernot et al., 2016] crafted adversarial samples based on computing the Jacobian matrix as the forward derivatives. [Moosavi-Dezfooli et al., 2016] proposed a DeepFool algorithm to compute adversarial examples based on an iterative linearization of the classifier to generate minimal perturbations that are sufficient to change classification labels. In this paper, we design the white-box adversarial attack with the access to original model. The comparison with other white-box attacks can refer to the experiment results shown in section 4.2 and 4.3.

2.2 Interaction between Compression and Attack

There are a few studies to discuss the interaction between model compression and adversarial attack. [Ye et al., 2019] proposed a framework of concurrent adversarial training and weight pruning that tries to preserve the adversarial robustness best when enabling model pruning. [Song et al., 2020] observed that adversarially-trained neural networks are vulnerable to quantization loss. So they proposed a boundarybased retraining method to mitigate adversarial and quantization losses together and adopted a nonlinear mapping method to defend against white-box gradient-based adversarial attacks. [Shumailov et al., 2019] reported an empirical study to investigate whether the adversarial samples are transferable between uncompressed and compressed models. Pruning and quantization may lead to different transferability. Adversarially trained model compression (ATMC) [Gui et al., 2019] is a unified optimization framework to trade-off model size, accuracy, and robustness. These studies focused more on the adversarial robustness when facing the traditional adversarial attacks designed for dense models. They do not discuss whether some adversarial attacks are designed explicitly for the compressed models and the corresponding robustness against these attacks. Moreover, most of the experiments are evaluated at relatively small datasets, e.g., MNIST [LeCun et al., 1998], CIFAR-10 [Krizhevsky et al., 2009], and CIFAR-100 [Krizhevsky et al., 2009]. They did not provide the results on large-scale datasets, like ImageNet [Deng et al., 2009], COCO [Lin et al., 2014], etc.

3 Spear: Attack for CompressedModel

Sparsity and quantization are two standard and applicable model compression methods. We will design an adversarial attack against these two compression processes. We do not include the discussion of network architecture search (NAS) here because NAS will lead the model structures to various shapes, so the attacks against the NAS-compressed models may be case by case, with no general rules to be summarized.

3.1 General Attack for Neural Model

A general supervised neural model can be expressed as:

where x is the input, θ is the weight parameters of the model, and y is the corresponding output of the neural function F. Given the supervised label info yl , the model training and optimization process can be expressed as the following minimization problem:

where L(·) is the loss function, for instance, the cross-entropy loss for a classification network, the Dice loss for a segmentation task, the negative log-likelihood for a language processing network.

For the adversarial attack settings, the perturbation δ is added to the input x to constitute the attack samples xadv, which subjects to the L∗ perturbation constraint:

where ϵ refers to the attack strength. The objective for the adversarial attack is to maximally deteriorate and change the 'normal' behavior of the model within the perturbation constraint. If the whole dataset is defined as D, with N categories, and each category is labeled with yl:k and denoted as D(yl:k ), then the adversarial target can be denoted with the following joint optimization problem.

3.2 Specific Attack for Sparsity

For any sparse model, we can always regard it as performing an element-wise multiplication (⊙) between a mask function f M and the original weight parameters θd from the corresponding dense model. The different sparse algorithms just change the concrete form and sparse granularity of the mask function. In our assumption, we want to know whether there is a specific adversarial attack type, which deteriorates the sparse model's performance while keeping the performance of its dense form at the meanwhile. So the joint optimization problem changes to the following form:

(c) Attack samples enter Spear algorithm.

Figure 3: Detailed workflow charts of Spear attack algorithm. (a) shows the whole workflow chart, while two different situations when benign or attack samples as input are shown in (b) and (c), respectively. (b) When the benign samples enter Spear algorithm, the predictions from the dense and compressed models are expected to be almost the same. (c) When the attack samples enter Spear algorithm, the predictions from the dense and compressed models are expected to be totally different.

3.3 Specific Attack for Quantization

Quantization uses alternative formats than the standard 32-bit single-precision floating-point (FP32) data type for parameters in a dense model. Because the standard FP32 data format can emulate almost all lower-precision formats, including floating-point, fixed-point, and integer. So for any quantized model, we can always regard it as performing an elementwise multiplication (⊙) between a quantization function f Q and the original weight parameters θd from the corresponding dense model. The different quantization methods just change the concrete form and quantized data format of the quantization function. In our assumption, we want to know whether there is a specific adversarial attack type to deteriorate the quantized model's performance while keeping the performance of its dense form at the meanwhile. So the joint optimization problem changes to the following form:

3.4 Spear: Specific Attack for CompressedModel

Based on the analysis of the adversarial attack for the sparse and quantized model in expressions (5) and (6), we can use a unified format of (5) and (6) to express the adversarial optimization target of these two orthogonal model compression methods. So we also propose a unified adversarial attack algorithm named Spear that can generate specific attack examples to maximally and efficiently deteriorate the compressed models. The workflow of Spear attack is shown in Figure 3a.

In the proposed algorithm, the dense baseline model is referred as MD, and the corresponding compressed model is MC (MC = MD ⊙ f M for the sparse model, and MC = MD ⊙ f Q for the quantized model). The Spear consists of two sub-neural-models: Spear attacker (Sa) and Spear detector (Sd). Spear attacker model takes the benign samples as input (Ib), and adds the minimal perturbation δ within attack strength constraint (ϵ) to generate the attack samples (Ia).

When the dense and compressed models take the samples as input, they will provide the predicted results. Inspired by the Knowledge Distillation (KD) [Hinton et al., 2015], we consider the hard prediction to learn from the onehot representation of the ground-truth label; meanwhile, we also consider the soft predictions to learn from the soft labels to represent all probabilities over the whole classes in the dataset. We use the cross-entropy to calculate the prediction loss (Lp) between the hard prediction from dense (for benign and attack samples: Ph(MD, Ib), Ph(MD, Ia)) and compressed model (for benign and attack samples: Ph(MC, Ib), Ph(MC, Ia)). Kullback-Leibler (KL) divergence is used to compute the distillation loss (Ld) between soft predictions from the dense (for benign and attack samples: Ps(MD, Ib), Ps(MD, Ia)) and compressed model (for benign and attack samples: Ps(MC, Ib), Ps(MC, Ia)).

According to the joint optimization formula, when taking the benign samples as input, we hope the predictions from dense and compressed models are almost the same. The detailed workflow of this situation is shown in Figure 3b.

When the attack samples are taken as input, we expect the dense and compressed models' predictions to differ. The detailed workflow of this situation is shown in Figure 3c.

Spear detector takes the benign samples, and the generated attack samples as input. In the settings of adversarial attacks, the perturbation is intended to be optimized to a minimum. So Spear detector takes the responsibility to judge the difference between the benign and attack samples in a perception manner, and provide the adversarial loss (La) to optimize the Spear attacker and Spear detector in the next iteration. During the optimization process, if the adversarial loss converges to a minimal value, that means the perturbation added to the benign samples is also minimal.

Where α1, α2, β1, β2, γ are loss adjustment factors. We take the reciprocals of the prediction and distillation loss items for the attack samples, change them to the minimization form, and integrate them into the overall loss formula.

4 Experiments and Results

4.1 Experiment Settings

For the experiments in this section, we choose PyTorch [Paszke et al., 2017] with version 1.8.0 as the framework to implement all algorithms. All of the state-of-theart methods and Spear attack training and fine-tuning experimental results are obtained with V100 [NVIDIA, 2017] and A100 [NVIDIA, 2020a] GPU clusters. The acceleration performance results are obtained with A100 GPU clusters to fully utilize its Tensor Core [NVIDIA, 2020b] support for mixed-precision calculation, fine-grained sparsity 1 , and quantization 2 3 .

4.2 Effectiveness Experiments for Classification

To evaluate the effectiveness of the Spear attack on the image classification task, ResNet-50 [He et al., 2016] 4 , and Swin Transformer [Liu et al., 2022] 5 are chosen as the experiment target models. The loss adjustment parameters among the prediction loss (α1, α2), the distillation loss (β1, β2) and the adversarial loss (γ) apply 1, 1.5, 2, 4, 1, respectively.

In the Figure 4, * Sparse represents the sparse model trained with knowledge distillation and the weight pattern is aligned with Sparse Tensor Cores 1 [Yu, 2021]. INT8 * represents both the weight and activation tensors of the model are quantized to INT8 with Quantization Aware Training (QAT) method [Krishnamoorthi, 2018]. INT4 * represents the weight tensors are quantized to INT4 and the activation tensors are quantized to INT8 with QAT method. FP8 * represents both the weight and activation tensors of the model are quantized to FP8 in the train-from-scratch process [Micikevicius et al., 2022].

For the reference adversarial attack methods, we choose the FGSM [Goodfellow et al., 2014], BIM [Kurakin et al., 2018], PGD [Madry et al., 2017], Jitter [Schwinn et al., 2023], and Carlini-Wagner (CW) [Carlini and Wagner, 2017] as the reference algorithms. In this experiment, FGSM, BIM, PGD and Jitter choose L∞, while CW and PGDL2 choose L2 to measure the perturbation range and the attack strength.

From the comparison results shown in Figure 4, we can find robust accuracy of the compressed models attacked by the Spear attack is similar or lower than applying the other reference adversarial attack algorithms. The obvious difference is the Spear attack has the trivial influence on the dense model than the other algorithm. So it proves that the Spear attack can more effectively generate the attack samples for these compressed classification models and lead them to the wrong categories without causing much trouble to the original dense model.

1 https://github.com/NVIDIA/apex/tree/master/apex/contrib/ sparsity

2 https://github.com/NVIDIA/TensorRT/tree/master/tools/ pytorch-quantization

3 https://github.com/NVIDIA/TransformerEngine

4 https://github.com/pytorch/vision

5 https://github.com/microsoft/Swin-Transformer

4.3 Effectiveness Experiments for Detection Task

To evaluate the effectiveness of the Spear attack on the detection task, RetinaNet [Lin et al., 2017], and Mask R-CNN [He et al., 2017] 6 are chosen as the experiment target models.

The sparse models are compressed with the fine-grained gradual pruning [Zhu and Gupta, 2018] (*-FINE) and Sparse Tensor Cores pattern [Yu, 2021] (*-STC) methods. The quantized models are obtained with QAT method. RN50 and RN101 in the brackets represent the ResNet-50 and ResNet101 models served as the backbone of the detection networks. AP and AR represent the average precision and average recall metrics. We evaluate the robust AP and AR on the test set of COCO [Lin et al., 2014] dataset. The loss adjustment parameters among the prediction loss (α1, α2), the distillation loss (β1, β2) and the adversarial loss (γ) apply 1, 1.5, 3, 6, 1.5, respectively. The results are shown in Table 1. (The variance is within ±0.15 for AP, and ±0.21 for AR with different random seeds.)

4.4 Visualization Experiments

This section provides the visualization results to reveal how the "attention" focused regions change during the Spear attack process. We apply the Class Activation Mapping (CAM) tool [Zhou et al., 2016] to the dense ResNeXt-101 model without attack, the quantized ResNeXt-101 model during and after the attack process. CAM can highlight the importance of the image region to the final prediction. The visualization results are shown in Figure 5.

For CAM, the red color highlight the "attention" focused areas of each model. For the example in Line 1, the dense ResNeXt-101 model without attack classifies the ground truth image into the tench category. During the Spear attack process, the "attention" focus start to change from the tench to the reel region. And finally the compressed model after the Spear attack process classifies the same ground truth image into the reel category.

For the example in Line 2, the dense ResNeXt-101 model without attack classifies the ground truth image into the tape player category. During the Spear attack process, the "attention" focus start to change. The compressed model during the attack process classifies the image into the cash machine category. And finally the compressed model after the Spear attack process classifies the same ground truth image into the weighing machine category.

The CAM visualization results explain the adversarial attack can guide the "attention" of the model to the different areas on the ground truth images, which finally misleading to the wrong classification.

4.5 Ablation Experiments and Insights

In this experiment, We want to check the contribution of each component in Spear to the final adversarial attack effect on the compressed model. Then we can have a deep insight into why Spear can outperform state-of-the-art methods and whether we can further improve it. We include five key components which may have an apparent potential contribution.

6 https://github.com/facebookresearch/detectron2

• A: Prediction loss for benign samples: Lp(Ib).
• B: Prediction loss for attack samples: Lp(Ia).
• C: Distillation loss for benign samples: Ld(Ib).
• D: Distillation loss for attack samples: Ld(Ia).
• E: Adversarial loss: La (Sa, Sd).

In this ablation experiment, five independent Spear attack models are trained on ImageNet training dataset. Each model lacks one of the aforementioned key component. For fairness, the loss adjustment parameters (α1, α2, β1, β2, γ) all apply 1. The results are shown in Table 2.

From the quantitative results shown in Table. 2, we can get the following conclusions.

• Loss items for attack samples have more contributions than those for benign samples. (Robust accuracy without B or D is higher than the counterpart without A or C).
• Distillation loss items have more contributions than prediction loss items. (Robust accuracy without C or D is higher than the counterpart without A or B).
• Adversarial loss items have more contributions than loss items for benign samples, but roughly less than loss items for attack samples. (Robust accuracy without E is higher than the counterpart without A or C, but is roughly lower or slightly higher than the counterpart without B or D).
• The quantized model is more sensitive to the adversarial loss items than the sparse model. (Robust accuracy without E for the quantized model has a more noticeable increase than the sparse model).

From the ablation experiments, we can see each component contributes to the final success of the Spear attack performance. However, the importance of each component is not the same. The Spear target is searching for the specific attack samples, which can maximally deteriorate the compressed model while keeping the 'normal' behavior of the dense counterpart. So it is understandable that the loss items from the attack samples have more contribution. Inspired by KD, the soft predictions could provide more knowledge about the whole dataset distribution than the hard predictions for most cases. It helps to explain why the distillation loss items have more contributions than prediction loss items. The induction of the adversarial loss item aims to minimize the perturbation from the neural model perspective and within the perturbation constraint simultaneously. And it is proved to be very helpful for improving the attack efficiency and performance from the experimental results.

5 Conclusions

We proposed the Spear attack method, which can effectively generate the attack samples to evaluate the adversarial robustness of the compressed models in this paper. It is a unified adversarial attack framework that is effective, efficient, and general to most of the model compression pattern. We evaluate the effectiveness of the proposed Spear attack technique on various tasks on large-scale datasets.

Ethical Statement

We want to emphasis that the ultimate goal for our study of this specific adversarial attack for the compressed models is to effectively find the vulnerabilities and finally fix them to improve the robustness and safety of the AI models and systems. We have some initial results to show that we can improve the robustness of the compressed models by applying the adversarial training workflow with the attack samples generated by the Spear method. But we will leave it for indepth study to defend the adversarial attacks effectively.

We also encourage the community to understand and mitigate the risks arising from the Spear attack. As the Spear attack has the power to generate the misleading attack samples effectively, we should notice the risk that it can be used to misrepresent objective truth or to attack the neural models intentionally. However, we expect such misuse will become ineffectual as detection and defense techniques improve; these techniques may also benefit from our findings.

Acknowledgements

This work is supported by National Natural Science Foundation of China (No. 62071127, and 62101137), National Key Research and Development Program of China (No. 2022ZD0160100), Shanghai Natural Science Foundation (No. 23ZR1402900), Shanghai Municipal Science and Technology Major Project (No.2021SHZDZX0103).

The computations in this research were performed using the CFFF platform of Fudan University.