Robust Fine-tuning via Perturbation and Interpolation from In-batch Instances

1 Introduction

Pretrained language models (PLMs) have contributed to striking success in natural language processing (NLP). Simultaneously, fine-tuning has been a common practice to employ PLMs for downstream natural language understanding tasks. However, recent work shows that vanilla finetuning methods may lead to vulnerable models [Aghajanyan et al., 2021]. This long-standing problem hinders the model performance and makes fine-tuned PLMs vulnerable to adversarial attacks and spurious bias [Branco et al., 2021; Clark et al., 2019]. As a result, it limits the application of the PLMs on some downstream tasks, especially in some realworld scenarios where robustness is especially required.

To alleviate this problem, various fine-tuning approaches have been proposed. For instance, SMART [Jiang et al.,

* Equal contribution.

2020] and R3F [Aghajanyan et al., 2021] introduce regularizations to the noise applied to the original pretrained representations. ChildTuning [Xu et al., 2021] updates the child network during fine-tuning via strategically masking out the gradients of the non-child network. However, most of them focus on improving the generalizing robustness or the adaptive robustness, while the robustness to adversarial attacks and spurious correlations remains challenging.

Inspired by contrastive learning with in-batch instances [Gao et al., 2021; Fan et al., 2021], we propose MATCH-TUNING that utilizes in-batch instances dynamically for robust fine-tuning. In MATCH-TUNING, we convert each instance representation in a batch by fusing the representations of other instances in the same batch. More concretely, MATCH-TUNING first calculates the similarities between the PLM outputs of instances in a batch to form a matching matrix. Then, we fuse the PLM representations according to the matching matrix to form new representations for this batch. Finally, we use the new representations for prediction.

MATCH-TUNING works by adaptively determining how to utilize the in-batch instances during the whole training procedure. As shown in Fig. 1, at the beginning of training, regarding the instances with other labels as a perturbation, MATCH-TUNING urges the PLM to converge to a more flat local minimum for better robustness. While nearing the end, MATCH-TUNING focuses more on performing an interpolation among the instances with the same label for better generalization. In this manner, MATCH-TUNING reduces the vulnerability of models and improves their general performance.

We conduct a comprehensive evaluation of MATCHTUNING on the GLUE benchmark. The results show that our method outperforms the vanilla fine-tuning by 1.64 scores on average. In addition, our method outperforms vanilla finetuning by 4.11 average scores on advGLUE, and yields a great improvement for label noise or data imbalance, which shows our overwhelming robustness over vanilla fine-tuning.

Our main contributions are summarized as follows:

• We propose an adaptive fine-tuning method called MATCH-TUNING to train robust models, where instances in the same batch will interact with each other. 1
• MATCH-TUNING reduces the vulnerability of models

1 Our code is available at https://github.com/tongshoujie/ MATCH-TUNING

and outperforms the vanilla fine-tuning by 1.64 scores on the GLUE benchmark.

• Our method manifests extraordinary robustness to various scenarios, including adversarial attacks, spurious biases, and data imbalance.

2 RelatedWork

The vanilla fine-tuning simply adapts PLMs to the taskspecific inputs and outputs, and fine-tunes all the parameters in an end-to-end manner [Devlin et al., 2019; Liu et al., 2019a]. The token representations or the representation of a special token (e.g., [CLS]) is directly fed into an output layer for tagging or classification, respectively. This manner has been shown to produce biased models that are vulnerable to adversarial attacks and noisy data [Aghajanyan et al., 2021; Clark et al., 2019].

In the past years, numerical variants like ChildTuning and R3F are proposed to conduct more trustworthy and effective fine-tuning [Lee et al., 2020; Xu et al., 2021; Aghajanyan et al., 2021]. FreeLB [Zhu et al., 2019] adds perturbations to continuous word embedding by using a gradient method and minimizes the resultant adversarial risk. Moreover, Jiang et al. [Jiang et al., 2020] introduce a regularization to encourage the model output not to change much when injecting a small perturbation to the input. Aghajanyan et al. [Aghajanyan et al., 2021] simply regularize the model against the parametric noise. The literature provides strong insights that proper perturbation on the PLM outputs has great potential in enforcing the model smoothness and robustness.

Recently, in-batch learning successes in many fields. Yao et al. [Liu et al., 2019b; Yao et al., 2021] suggest that the unsupervised instances are helpful to learning the classifier in computer vision tasks. For NLP, contrastive learning with in-batch instances also improves the task-specific representation [Gao et al., 2021] and adversarial robustness [Fan et al., 2021]. Inspired by this, we propose MATCH-TUNING to utilize in-batch instances dynamically for robust fine-tuning.

Different from previous work, MATCH-TUNING no longer needs label information or the specification of negative and positive instances in advance. It performs automatic instance interaction, which applies to most existing pretrained models.

3 Method

MATCH-TUNING derives a composite representation for each instance in a batch by fusing the representations of other instances in the same batch, which are deemed as adaptive noise. From experiments, we find that the adaptive noise functions as an in-batch perturbation in the initial stage of training, and then gradually transits to an in-batch interpolation among "positive" instances that share the same label. In addition, we show that MATCH-TUNING helps the model to escape the sharp local minima through qualitative analysis.

3.1 Overview of MATCH-TUNING

In the batched gradient descent, the data points in a batch are formulated as {(xi, yi)|i = 1, ..., n}, where n denotes the batch size. Throughout our paper, xi represents a textual input, e.g., a single sentence or a sentence pair, while yi denotes a discrete label or a continuous number for classification and regression tasks, respectively. We use h and θ to represent a PLM that extracts contextualized features from xi and its parameters. Similarly, the task-specific classifier and its parameters are denoted by f and ψ. Letting L denote the taskspecific loss function, we compute the mini-batch gradient g in the vanilla fine-tuning as follows:

To apply adaptive weights to the instances in a batch, we introduce the matching matrix M, where each element indicates the pair-level similarity between in-batch instance representations given by a PLM. The matrix M is given by

Note that the overhead to compute Mi,j is small since we can directly reuse h(x; θ), the outputs of the PLM. Then, to produce a robust representation for an instance, we derive a composite representation z(i) from h(x(i) ; θ):

Then, z(i) serves as a drop-in replacement for h(x(i) ; θ) in the vanilla fine-tuning and the mini-batch gradient g′ in MATCHTUNING is computed as follows:

3.2 Qualitative Understanding of MATCH-TUNING

We provide a qualitative viewpoint to understand how MATCH-TUNING works stemming from the notions of perturbation and interpolation.

We first introduce the shape of local minima. The loss surface of deep neural networks tends to have various local minima as illustrated in Fig. 2. Sharp local minima are where the loss in a small neighborhood increase rapidly while flat local minima are where the loss varies slowly in a relatively large neighborhood. Sufficient literature has proved that flat local minima usually lead to better generalization [Hochreiter and Schmidhuber, 1995; Dinh et al., 2017]. In addition, under a Bayesian perspective, the noise in gradient could drive model away from sharp minima [Smith et al., 2018].

MATCH-TUNING follows this gradient noise addition routine, but in a different way. To better understand the mechanism of MATCH-TUNING, we visualize the the values of the "positive" instances (with the same label) and "negative" instances (with other labels) in the matching matrix. Fig. 3 depicts the change of matching matrix in MATCH-TUNING. We sum up the values of all instances with the same label and show the cumulative values for "positive" instances on Fig. 3. The "negative" instances share the same setting.

In-batch Perturbation In the initial stage, MATCHTUNING works by performing in-batch perturbation on the original outputs of PLM. As shown in Fig. 3, instances from the same class and that from other classes are similar in the matching matrix. Therefore, any other instance in the same batch is close to a tiny perturbation on the output of PLM. If the PLM provides vulnerable representations, the perturbed representations will break down easily. Therefore, the early stage in MATCH-TUNING encourages the PLM to generate a more robust representation for each instance and converge to a more flat local minimum.

In-batch Interpolation During the whole training process, the model will gradually learn to distinguish the representations of the positive (same label) and negative (other labels) instances in a batch. Consequently, as training proceeds, the portion of negative instances is getting smaller in the match matrix and can hardly influence the composite representation in the late stage of training. In this moment, MATCHTUNING tends to interpolate the representations of the positive instances. We also observe more representative positive instances will contribute more to the final composite representation. As illustrated in Fig. 1, the late stage in MATCHTUNING encourages composite representations to be grouped into clusters according to their real labels.

4 Experiments

We conducted extensive experiments on various downstream tasks to evaluate the general performance and robustness of MATCH-TUNING. For simplicity, in the rest of the paper, we denote other instances in the same batch with the same label as the current instance by positive instances, and other instances with different labels by negative samples.

4.1 Datasets

Following previous work [Xu et al., 2021; Lee et al., 2020], we conduct experiments on four main datasets in GLUE [Wang et al., 2019] to evaluate the general performance. Among them, classification task like CoLA is for linguistic acceptability, RTE is for natural language inference, and MRPC is for paraphrase identification. STS-B is a regression task for semantic textual similarity. By systematically conducting 14 kinds of adversarial attacks on representative GLUE tasks, Wang et al. [Wang et al., 2021] proposed AdvGLUE, a multi-task benchmark to evaluate and analyze the robustness of language models and robust training methods.

4.2 Experimental Setup

We report the averaged results over 10 random seeds. We conduct our experiments based on the HuggingFace transformers library 2 and follow the default hyper-parameters and settings unless noted otherwise.

4.3 General Performance

We compare MATCH-TUNING with the vanilla fine-tuning and related work on four tasks of the well-recognized benchmark, GLUE. And we focus on evaluating the performance of BERT-Large based models on the GLUE development set.

Baselines We compare MATCH-TUNING with the following methods. 1) Vanilla Fine-tuning [Devlin et al., 2019], the standard fine-tuning paradigm; 2) Weight Decay [Daum´e III, 2007], which adds a regularization term to the loss function; 3) Top-K Tuning [Houlsby et al., 2019], which fine-tunes only the top-K layers of the PLM; 4) Mixout [Lee et al., 2020], which stochastically replaces the parameters with their pretrained weights; 5) RecAdam [Chen et al., 2020], which introduces quadratic penalty and objective shifting mechanism; 6) R3F [Aghajanyan et al., 2021], which adds adversarial objectives with noise sampled from either a normal or uniform distribution; 7) ChildTuningF [Xu et al., 2021], which randomly masks a subset of parameters PLM during the backward process; 8) ChildTuningD [Xu et al., 2021], which detects the most important child network for the target task.

Results We show the mean (and max) scores on GLUE Benchmark in Tab. 1. MATCH-TUNING significantly outperforms vanilla fine-tuning by 1.88 average score. Moreover, compared with several strong tuning methods, MATCHTUNING achieves the best performance on three tasks, showing its effectiveness. Although ChildTuningD has performance on par with MATCH-TUNING on CoLA task (0.45 mean/-1.08 max), our method has a small computational overhead while ChildTuningD has to adopt Fisher information estimation to obtain the task-driven child network.

2 https://github.com/huggingface/transformers

Note that MATCH-TUNING can be integrated with other tuning methods to further boost the performance. We evaluate the combination of MATCH-TUNING and R3F, which achieves an additional improvement of 0.24 average score.

4.4 Robustness of MATCH-TUNING

Recent work reveals that vanilla fine-tuning is deceptive and vulnerable in many aspects. For instance, fool the models to output arbitrarily wrong answers by perturbing input sentences in a human-imperceptible way. Real-world systems built upon these vulnerable models can be misled in ways that would have profound security concerns. To examine the robustness of MATCH-TUNING, we design robustness evaluation tasks for three common scenarios respectively.

Robustness to Adversarial Attacks

As recent studies revealed, the robustness of fine-tuned PLMs can be challenged by carefully crafted textual adversarial examples. We systematically conduct various adversarial attack evaluations on the advGLUE benchmark.

Tab. 2 illustrates that fine-tuned models maintain vulnerabilities to adversarial attacks while our MATCH-TUNING approach alleviates this chronic problem by 4.11 accuracy promotion on average. Compared with vanilla fine-tuning, existing methods like R3F and ChildTuning encounter 8 ∼ 13 accuracy collapse on advSST-2, while MATCH-TUNING outperforms vanilla fine-tuning by 4.11 scores. On advMNLI, advRTE, and advQQP, MATCH-TUNING also holds a large improvement, as much as 10.79 higher accuracy than vanilla fine-tuning. In short, compared with prior fine-tuning methods, we find that MATCH-TUNING is more robust in adapting PLMs to various tasks.

Robustness to Label Noise

Nowadays, there is inevitably some noise in large-scale datasets. To explore the model robustness to noisy data, we conduct simple simulation experiments on RTE, MRPC, and CoLA. Specifically, we generate noisy training data by randomly changing a certain proportion of labels to incorrect ones. We test the robustness of different fine-tuning methods trained on the noisy data.

As shown in Tab. 3, MATCH-TUNING outperforms other fine-tuning methods on noised training data. To be exact, MATCH-TUNING surpasses vanilla fine-tuning by 1.76 average scores under 5% noise ratio, 2.36 under 10% noise ratio, and 1.41 under 15% noise ratio. Furthermore, we compare the degradation of model performance towards different noise ratios. Compared with Tab. 1, we calculated the degradation and display it in brackets (the last column of the Tab. 3 ). It shows that MATCH-TUNING has the smallest performance drop compared to other fine-tuning methods. All the above results show that MATCH-TUNING is more robust to label noise than existing methods.

Robustness to Data Imbalance

Minority Class Minority class refers to the class which owns insufficient instances in the training set. These kinds of classes are more challenging during fine-tuning than a normal class. To explore the performance of different tuning approaches on the minority class, we conduct experiments on synthetic RTE, MRPC, and CoLA datasets.

As Tab. 5 illustrated, under different data reduction ratios, MATCH-TUNING outperforms other fine-tuning methods by a large margin. MATCH-TUNING yields an improvement of up to 6.12 average score on 30% reduction ratio and 4.89 average scores on 40% reduction ratio. Besides, it can be seen that the smaller the reduction ratio, the better MATCHTUNING performs compared to other fine-tuning methods. In summary, we can conclude that MATCH-TUNING is more robust towards the minority class.

Atypical Groups Vanilla fine-tuned models can be highly accurate on average on an i.i.d. test set yet consistently fail on atypical groups of the data [Hovy and Søgaard, 2015; Sagawa et al., 2019] (e.g., by learning spurious correlations that hold on average but not in such groups). In contrast, MATCH-TUNING no longer aims at minimizing the original batch average loss and paying more attention to the comparison of instances. As demonstrated in Tab. 6, simply applying MATCH-TUNING improves the worst-group performance by 1.1 with traditional empirical risk minimization (ERM) and 1.5 with GroupDRO [Sagawa et al., 2019]. What's more, the results show that MATCH-TUNING is orthogonal to prior techniques for data imbalance, integrating MATCH-TUNING with them brings further improvement.

5 Analysis and Discussion

5.1 Exploration into Effects of In-batch Instances

As analyzed in Section 3.2, negative instances and positive instances in a batch function differently in the process of MATCH-TUNING. To further explore the role of negative instances and positive instances in MATCH-TUNING, we define a mask matrix A by:

Then we update M ← A ⊙ M so that merely positive instances are involved for MATCH-TUNING, while M ← (In - A) ⊙M so that only negative instances are observed. As is shown in Tab. 7, both negative instances and positive instances place an important role in MATCH-TUNING. When negative instances are masked for the matching matrix and only positive instances are responsible for MATCH-TUNING, the resulting score outperforms on RTE and MRPC, but drops on CoLA slightly. In the contrast, if we only preserve the influence of negative instances on the current instance, the performance surpasses the vanilla fine-tuning baseline steadily.

This result indicates that both perturbation and interpolation contribute to the final improvements. MATCH-TUNING simply unify negative instances and positive instances by the matching matrix, and such unification brings further improvement (refer to the last row of Tab. 7).

5.2 Computational Efficiency

MATCH-TUNING improves the general performance and robustness of PLMs by introducing simple in-batch interactions. To demonstrate the computational efficiency of MATCH-TUNING, we report the training time of a single epoch for different fine-tuning methods. All the methods are based on BERTLARGE and tested on a single NVIDIA A40 GPU. As illustrated in Tab.4, while other methods all introduce heavy extra computational cost, MATCH-TUNING takes almost no overhead than vanilla fine-tuning. Besides, MATCH-TUNING + R3F improves the performance of R3F with slight overhead, which also shows the efficiency of MATCH-TUNING.

6 Conclusions

To improve the general performance and robustness for fine-tuning PLMs, we propose robust MATCH-TUNING via in-batch instance perturbation. Extensive experiments on downstream tasks demonstrate the general performance of MATCH-TUNING. In addition, MATCH-TUNING is shown to be a powerful tuning approach towards broad categories of robustness evaluation. We further analyze the functioning process of MATCH-TUNING and provide probation on its components.

Acknowledgements

This paper is supported by the National Key Research and Development Program of China 2020AAA0106700 and NSFC project U19A2065.