PPT: Backdoor Attacks on Pre-trained Models via Poisoned Prompt Tuning

1 Introduction

Pre-trained language models [Devlin et al., 2018; Liu et al., 2019; Raffel et al., 2019] have shown unparalleled advantages in many NLP tasks. A general used paradigm is finetuning, which fully tunes all parameters of the PLM on the downstream task. The versatile knowledge acquired from the large-scale unlabeled data can be adapted to various NLP tasks after fine-tuning. However, fine-tuning is memoryconsuming during training because gradients and optimizer states for all parameters need to be stored. Moreover, the PLM will be a task-specific model after fine-tuning. It requires keeping many copies of the PLM during inference for different downstream tasks.

Prompt tuning [Lester et al., 2021] is proposed as a solution to the above problems. As shown in the Figure 1, adding natural language prompt to the text, all downstream tasks can be uniformly transformed into the form of pre-training tasks of PLMs. The prediction of [MASK] token is mapped to the real label by verbalizer. Using only hard prompt may lead to suboptimal performance. Therefore, Prompt tuning uses soft prompt as a trainable parameter, which freezes the parameters of PLMs and only tunes soft prompt. The advantage is that it can save the cost of training and the fixed PLM can be reused. Recent studies have shown that prompt tuning can be comparable to fine-tuning universally across various model scales and NLP tasks [Liu et al., 2021].

∗Corresponding Author

On the other hand, prompt is more and more generalized and toolkitized. In the openprompt [Ding et al., 2021] framework developed by Tsinghua, users can easily import templates and verbalizers, and then load third-party trained prompts for downstream tasks in their own models. However, it also exposes some potential security threats while bringing convenience. The prompt associated with PLMs can be intentionally added with some malicious behavior, such as backdoor. The attacker upload the carefully crafted prompts to the public platform. The victim model will be implanted with the backdoor after using these poisoned prompts.

Backdoor attack is a serious security threat for deep learning models , which was first proposed by [Gu et al., 2017]. They construct poisoned data by adding triggers and reversing labels. By training on the poisoned dataset, the poisoned model can maintain high accuracy for the original task but output the target class preset by the attacker when a trigger is added to the input. If such an attack is applied to securityrelated scenarios such as identity verification and self-driving vehicles, it will cause serious consequences. For example, when the camera detects a stop sign with a trigger, an backdoored self-driving system will control the car to accelerate instead of braking.

In this paper, we are interested in designing backdoor attacks for prompt learning. We find it feasible to get a poisoned prompt for PLMs and corresponding downstream tasks by prompt tuning. When the PLMs are loaded with the poisoned prompt, the model will be implanted with the backdoor. We name this Poisoned Prompt Tuning method PPT. For attacking prompt learning in NLP tasks, the poisoned prompt adapts the PLM to the downstream task as others clean prompt and establishes a shortcut between the specific trigger word and the target label word. So, we can control the prediction of the entire model with a small prompt. Compared with the method of tuning all parameters of model, the attack based on prompt poisoning is much more concealed and effective. Experiments conducted on various tasks including sentiment analysis, sentence-pair classification and multi-label classification show that our proposal can achieve a perfect attack success rate without losing the accuracy of clean task. Our contributions are summarized as follows:

• We propose PPT, the first backdoor attack against the prompt learning of PLMs. By loading the poisoned prompt, the model will be implanted with the backdoor.
• We evaluate PPT in many text classification tasks. Experiments show that PPT can get a perfect attack success rate with no damage the accuracy of original task.
• We conduct many related analysis experiments to comprehensively analyze the effect of PPT. We measure the performance of PPT in different hyper-parameter settings and data transferring scenarios, and evaluate the backdoor capacity of the soft prompt.

2 RelatedWork

2.1 Prompt Learning

Prompt learning can usually be divided into two ways, namely, Prompt-Tuning with fixed PLMs and tuned soft prompt, and Prompt-Oriented Fine-Tuning with both PLMs and soft prompt tuning. In this paper, we focus on prompt tuning, hoping to implant a backdoor into the model only through prompt.

Autoprompt [Shin et al., 2020] first studies how to automatically get the appropriate prompt. They search for suitable words from vocabulary as prompt by a similar method of generating universal adversarial perturbations. Since only real words can be used, the generated prompt is limited to the discrete space. Therefore, [Hambardzumyan et al., 2021] propose to use a continuous trainable vector as prompt, that is, a soft prompt, which is tuned by gradient descent on downstream tasks. In addition, they find that the label word corresponding to the true label will also affect prompt learning. So each label word in the verbalizer is also a trainable vector and optimized together with the soft prompt. [Li and Liang, 2021] further studies the prompt learning of natural language generation tasks, and adds soft prompt before the input of each transformer layer to get better results. [Lester et al., 2021] explores the effect of soft prompt on domain adaptation and different model scales. They find out that the larger the scale of PLMs, the better the effect of prompt tuning.

2.2 Backdoor Attack for PLMs

Recently, there have been some researches on backdoor attacks against PLMs. Here is a review and summary of related methods. Different from the traditional backdoor attack, the main idea of the backdoor attack against the PLMs is to establish the connection between the trigger and the target class samples' features encoded by PLMs. [Kurita et al., 2020] links trigger to encoded features by general backdoor. They directly poison the entire model including the downstream task classification layer, and then they get a poisoned PLMs for a specific downstream task. In contrast, [Zhang et al., 2021] links trigger to the predefined vector. They retrain PLM with a backdoor sub-task and original pre-training task, enabling the PLM to output the predefined vector for the input with trigger. BadEncoder [Jia et al., 2021] uses the feature alignment to establish a connection between the trigger and the encoded feature. They align the poisoned data with the encoded feature of the target class sample, and align the clean data with the encoded features of themselves. Similarly, [Saha et al., 2021] applies contrastive learning to align features. They use a pair of poisoned data and target class data as a positive sample, so that the encoded features of the input with trigger and the target class input can gradually approach to each other. Moreover, [Yang et al., 2021] attacks PLMs by modifying the embedding vector of a trigger word in the word embedding layer.

In this paper, we also design backdoor attacking against PLMs. The difference is that the above methods attack the fine-tuning method for PLMs, while we attack the prompttuning method for PLMs.

3 Methodology

In this section, we first describe the threat model of our method, which includes attacker's goals and capabilities. Then we give an introduction and formulation of the pretraining-then-prompt-tuning paradigm as essential preliminaries. Finally, we show the process and details of our method.

3.1 Threat Model

The attacker's goals. We consider a malicious service provider or some group who releases their models. They can prompt-tuning a PLM on a downstream task and inject a backdoor into the prompt that can be activated by a specific trigger word. After obtaining the well-poisoned prompt, they release it to the public (e.g., uploading the poisoned prompt to Github). The victim may download this poisoned prompt to use. The attacker's goals are that the victim PLMs will be backdoored after loading the poisoned prompt into them: for clean samples, the victim PLMs will still give the correct label word as the clean PLMs; for poisoned samples which are added with the trigger word, the victim PLMs will output the target label word.

The attacker's capabilities. Usually, we assume the attacker has full knowledge about the PLMs and downstream tasks, including the model structure and training data. For the model structure, most users are basically using wellknown and released PLMs such as BERT[Devlin et al., 2018], Roberta [Liu et al., 2019] and T5 [Raffel et al., 2019]. For the training data, the attacker may not accurately know the user's own task, so we consider three scenarios about training data.

• Full data: We assume the attacker can access the full dataset of the user's downstream task in this scene, which will let the attacker achieve the best performance on original task. This usually occur on some tasks performed on public datasets.
• Data in the same domain: We assume the attacker can access the dataset in the same domain of the user's downstream task. For example, IMDB and SST-2 are datasets for this case, because they are both movie reviews and are used for sentiment analysis task. In this scene, the attacker will achieve a suboptimal performance on original task.
• Data in the different domain: We assume the attacker can access the dataset in the different domain of the user's downstream task. For example, SST-2 and Lingspam are datasets for this case, because unlike SST2, Lingspam is composed of many spam mails and is used for spam detection task.

For the three scenarios above, we conduct related experiments in section 5.2.

3.2 Preliminaries

Formally, a text classification task can be denoted as Ω = {X, Y}, where X is the input text set and Y is the corresponding class label set. Each input text x ∈ X is composed of a sequence of tokens x = {x1, x2, · · · , xn}, where n is the length of the input text. For a task with c classes, the value of each class label y ∈ Y can be chosen from {0, 1, · · · , c - 1}.

The prompt tuning is implemented by a combination of the soft prompt p, the template T (·), the verbalizer V(·) and the PLM M(·).

• Soft prompt p can be denoted as a series of tokens p = {p1, p2, · · · , pm}, whose parameters are randomly initialized and learnable during prompt tuning. m is the number of the soft prompt tokens.
• Template T (·) is a function to process the original input text x and soft prompt p into a unified form xprompt = T (x, p), which defines where each token of x and p is placed. Meanwhile, at least one [MASK] token is placed into the xprompt for M to predict the label word. For example, we can set a template T (x, p) = "p1 p2 · · · pm, x1 x2 · · · xn is [MASK]." for a sentiment analysis task.
• Verbalizer V(·) is a map function for mapping the label word to the class ˆ y = V(w). Usually, each class can have one or more label words. We call T a multiword verbalizer when each class has more than one label word, such as {positive: good, great; negative: bad, terrible;}.

The prompt text xprompt will be an input to M, and we can obtain the encoded feature hm of [MASK]. By a softmax function, we can compute the probability that the label word w can fill the masked position p([MASK] = w|xprompt) = Softmax(Ew · hm), where Ew is the embedding of the token w in the PLMM. The label word with the highest probability is the predict word w = M(xprompt) and the predict class can be obtained by the verbalzier ˆ y = V(w).

3.3 Poisoned Prompt Tuning

We introduce Poisoned Prompt Tuning to backdoor PLMs. Figure 2 shows the overview of PPT on NLP downstream tasks. The main idea of PPT is embedding the backdoor into the soft prompt which will be loaded by victim PLMs.

The first step is to generate the poisoned dataset. As we mentioned before, the backdoor attack aims to establish a shortcut between the trigger and the target class. In the prompt learning paradigm, we need to establish a shortcut between the trigger word ∆ and the word from the target label words set wt = {wt1 , wt2 , · · · , wt∥wt ∥ }, because the prediction is obtained by mapping the label word to the true class. So, we can define a data-poison function P(·) which insert a trigger word ∆ into the input text x and modify the label to the target class. The poisoned data is (xp, t) = P(x,∆,wt ), where x ∈ Dclean and x's label yx ̸= t. By injecting the poisoned data into the clean dataset Dclean, we can construct the poisoned dataset D as shown in Figure 2. The amount of the poisoned data we injected is usually controlled by the poison ratio λ. We formulate the process as follows:

Words that have a low probability of appearing in the input text are usually chosen as triggers, as this ensures that the triggers will not affect the prediction of clean samples.

In the second step, we perform supervised learning on the poisoned dataset. With the fixed PLM, we only optimize the soft prompt by gradient descent. We can redefine the entire inference process of prompt tuning as ˆ y = V (M(T (x, p))) = f (x, p, θp), where θp is the parameters of soft prompt. And the objective function of poisoned prompt tuning as follows:

where ⊕ denotes the operation of inserting the trigger word ∆ into the input text x.

In this objective function, the first term enables the soft prompt to adapt the PLM to the downstream task just like other clean prompts. Meanwhile, the second term forces the soft prompt to lead a shortcut between the trigger word and the target label word to be created for the PLM. So, we can get the poisoned prompt that will satisfy the goals of the backdoor attacker after several epochs of training.

Finally, the victim PLM loaded with the poisoned prompt is deployed to use. Based on the data knowledge we mentioned in Section 3.1, the inference can be divided into three scenarios and we denote them as follows:

where the θ(A) p represents the soft prompt trained on the dataset A and the D(B) denotes dataset B. For the full data knowledge, A = B. The other two scenarios are where A is the same domain data for B and where A is the different domain data for B.

4 Evaluation

In this section, we evaluate the feasibility of PPT on various tasks and models. The hype-parameter experiments of poisoning and prompt-tuning are conducted to analyse the performance of PPT in different settings.

4.1 Experiment Setup

Dataset. The experiments are carried out on three text classification tasks: sentiment analysis, toxicity detection and spam detection. For the sentiment analysis, we use the Stanford Sentiment Treebank (SST-2) dataset [Socher et al., 2013] and IMDB dataset [Maas et al., 2011]. For the toxicity detection, we choose the OffensEval dataset [Zampieri et al., 2019] and the Twitter dataset [Founta et al., 2018]. For the spam detection, we use the Enron dataset [Metsis et al., 2006] and the Lingspam dataset [Sakkis et al., 2003]. Moreover, we also evaluate PPT on the sentence-pair classification tasks which are the Question Natural Language Inference (QNLI) [Rajpurkar et al., 2016] and Recognizing Textual Entailment (RTE) [Wang et al., 2019]. In addition to the bi-classification tasks we mentioned above, we also conduct a multiple-backdoors attack on the five-class Stanford Sentiment Treebank (SST-5) dataset [Socher et al., 2013] in Section 5.1. Since labels are not available in the test sets for some datasets, we use the validation set as the test set and split a part of the training set as the validation set. Statistics of these datasets we mentioned above are shown in Table 1.

Model and Training Details. In our experiments, we use the base versions of BERT [Devlin et al., 2018], Roberta [Liu et al., 2019] and Google T5 [Raffel et al., 2019], which are widely used pre-trained language models in NLP. For the BERT and Roberta, we use the Adam optimizer for training. And the Adafactor optimizer is used for Google T5. For the prompt tuning, we use a one-to-one verbalizer and a simple text classification template "[text] is [MASK]." where 20 soft prompt tokens are added in the head. Following the settings of [Lester et al., 2021], we set the learning rate to be 0.3.

PoisonDetails. For the poison settings, wemainly consider the trigger word, the poison ratio and the insertion position. For the trigger word, we choose the rare word 'cf' as the configuration of [Kurita et al., 2020]. We set the poison ratio to be 0.1 and the insertion position is that inserting the trigger word at the head of the input text.

Metric. We consider two evaluation metric in our experiments. Attack Success Rate (ASR) represents the proportion of the poisoned samples we successfully enable the model to misclassify as the target class and we use it to evaluate the attacking performance of PPT. The other evaluation metric is Accuracy (ACC) which represents the proportion of the clean samples correctly classified by the model. It can be used to measure the performance of the model on original task. We use Cacc, Casr for the clean prompt tuning and Pacc, Pasr for the poisoned prompt tuning.

4.2 Main Results

The main results are shown in the Table 2. As you can see from this table, PPT can achieve close to 100% ASR for both sentiment analysis, toxicity detection and spam detection tasks, compared to less than 10% ASR before PPT was conducted. For the clean ACC of original task, the poisoned prompt obtained by PPT can maintain the same performance as clean prompt. Even some tasks can show better performance after PPT training. For example, the clean ACC of Lingspam task on Roberta model changes from 99.83% to 100% after PPT training. We speculate that this is because adding poisoned samples to the dataset can be considered as data augmentation, and training on this poisoned dataset can be seen as the adversarial training to some extent.

For the sentence-pair classification tasks, PPT can also achieve a high attack success rate for the PLMs of all sizes. Moreover, we find that the accuracy of the original task was preserved on both BERT and Roberta, but decreased on T5. We guess that the seq2seq-style T5, while benefiting the sentence-pair tasks, is more fragile and easily affected by the poisoned prompt for the original task.

4.3 Hype-Parameter Experiments

In this subsection, we change various hype-parameters to analyse the effect of them on PPT. We conduct these experiments on SST-2 dataset.

Insert Position. We consider three possible insertion positions which are inserting the trigger word at the head of the sentence, the tail of the sentence and a random position in the sentence. From the results shown in the Table 4, we can see that changing the insertion position will not affect the performance of the PPT, and the random insertion which brings a stronger data augmentation enables the model to perform better on the original task.

Trigger word. We try other four rare words mentioned in [Kurita et al., 2020] as the trigger word. As shown in Table 5, any rare word can get a high attack success rate.

Poison Ratio. The poison ratio which controls the amount of poisoned data in the dataset can have an impact on PPT. In addition to 0.1 (default), we evaluate the performance of PPT on lower poison ratios. As shown in the Figure 3, we can see that PPT can get a good attack performance with even 0.005 poison ratio, which means adding only 300 poisoned samples to a 60k dataset. For the accuracy of original task, PPT can maintain its stability at all poison ratio settings.

Soft Prompt Length. The soft prompt with more tokens may have better performance and more redundancy to inject backdoors, otherwise the opposite. So we try the fewertokens soft prompt with the default poison ratio and the moretokens soft pormpt with the lower poison ratio settings. In our experiments, PPT can also get a 99% attack success rate in even only one token for the soft prompt. As shown in the Table 6, using more soft tokens can achieve better attack performance, which suggests that more redundancy does exist in more tokens to build backdoors.

Verbalizer Type. In addition to the one-to-one verbalizer, we also consider the multi-word verbalizer and the soft verbalizer [Hambardzumyan et al., 2021] in PPT. For the multiword verbalizer, we get a good performance of PPT as the one-to-one verbalizer settings. For the soft verbalizer, we also test it in a lower poison ratio settings. We find that PPT with the soft verbalizer can also get a better attack performance but damage the accuracy of original task. This is because the poisoned data affects the label word in the soft verbalizer and performing prompt tuning with different label words can produce widely varying results.

5 Extra Analysis

5.1 Backdoor Capacity

In the paragraph of Soft Prompt Length, we find that even a token has enough redundancy to build the backdoor. In this subsection, we explore the backdoor capacity of soft prompt from a different perspective. We try to inject multiple backdoors into the soft prompt at the same time, each of which can trigger a certain class independently. The results are shown in Table 3, where we can see that soft prompt can maintain a high attack success rate for five different backdoors simultaneously on the SST-5 dataset without damaging the accuracy of the original task. This indicates that soft prompt has sufficient redundancy to build robust backdoors for PLMs.

5.2 Data Transfer

As we mentioned in subsection 3.1, we evaluate PPT in two other data transferring settings as shown in Table 7 and Table 8. The results show that PPT can also work for data transferring because the backdoor in soft prompt establishes a link only between the trigger word and the target label word. So adding a trigger word to any meaningful text can activate the backdoor, regardless of the task of the text. But the accuracy of original task might vary depending on the transferring dataset.

6 Conclusion

As a bridge between PLMs and downstream tasks, soft prompt plays a vital role, but is also a severe security threat. In this paper, we find it is feasible to conduct a backdoor attack on PLMs via our proposed PPT. We hope our results can raise the awareness of the possible security threats hidden in the prompt and encourage the research on poisoned prompt detection and defense.

Acknowledgments

This work is partly supported by the Joint Funds of the National Natural Science Foundation of China (Grant No. U21B2020).