Less Learn Shortcut: Analyzing and Mitigating Learning of Spurious Feature-Label Correlation

1 Introduction

Pre-trained language models, e.g. BERT [Devlin et al., 2018], ERNIE [Sun et al., 2019] and RoBERTa [Liu et al., 2019b], have achieved great success on many NLP tasks. However, recent studies highlighted that pre-trained models tend to take dataset biases as a shortcut, rather than truly understand tasks [Schuster et al., 2019; Niven and Kao, 2019]. Models' over-reliance on the shortcut results in their poor generalization ability and low robustness [Geirhos et al., 2020].

The phenomenon of shortcut learning has been widely studied in various NLP tasks. Many previous studies examine this phenomenon by constructing artificial adversarial examples, and employ adversarial data augmentation to enhance model robustness [Jia and Liang, 2017; Alzantot et al., 2018; Ren et al., 2019; Jin et al., 2020]. These studies reported high success rates on artificial adversarial examples, but it is uncertain if the models will perform well on real-world data distributions [Morris et al., 2020; Bender and Koller, 2020]. Additionally, recent work [Balkir et al., 2022] indicated that few studies have applied explainable methods to understand or investigate the impact of shortcut learning.

∗ The work was done when Yanrui Du was doing internship at Baidu.

† Corresponding author

Previous works point out that shortcuts can be traced back to dataset biases [Lai et al., 2021; Gururangan et al., 2018; Kavumba et al., 2021; Du et al., 2021; McCoy et al., 2019; Liu et al., 2019a; Kavumba et al., 2021; Goyal et al., 2017; Ye and Kovashka, 2021; Dawkins, 2021]. For example, if "not" happens to be contradiction for most of the training data in Natural Language Inference (NLI) tasks, detecting "not" becomes a successful strategy for models' prediction, thus leading to an unexpected performance on a shift distribution [Gururangan et al., 2018]. However, most studies are limited to analyzing task-specific shortcuts, which are prohibitive to be transferred to other tasks.

In this work, we analyze the correlations between simple features (e.g. words) and labels, which can be originated from the biased data distribution of any NLP task, to quantitatively investigate the shortcut learning behavior of NLP models. Existing work has argued that, for any NLP task, no single feature on its own should contain information about the labels, and any correlation between simple features and labels is spurious [Gardner et al., 2021]. Based on the above analysis, we propose a task-agnostic training strategy LessLearn-Shortcut (LLS), which mitigates the shortcut behavior of models, thereby improving their performance on adversarial data.

To examine the spurious feature-label correlation, we first introduce two definitions: biased word, which is the word highly co-occurring with a specific label in a dataset, and biased example, which is the example containing at least one biased word. Then we quantitatively analyze the spurious feature-label correlations on the Question Matching (QM) task. Based on our analysis, we propose our training strategy LLS, with which biased training examples are downweighted according to their biased degrees, and the models' over-reliance on the biased words is penalized during fine-tuning. We conduct extensive experiments on QM, NLI and Sentiment Analysis (SA) tasks to evaluate our training strategy and compare it to other task-agnostic strategies such as Rew.bias [Utama et al., 2020; Clark et al., 2019] and Forg. [Yaghoobzadeh et al., 2021]. Our experimental results demonstrate that LLS can improve the model performance on adversarial data while maintaining good performance on indomain data, and can be easily transferred to different NLP tasks. Additionally, we explore the scenarios in which the above strategies are applicable.

In general, we have the following major findings and contributions:

• We reveal that biased examples (as defined in Sec. 2.2) are easier to be learned than other examples, and with an explainable method LIME [Ribeiro et al., 2016], we find that biased words make significantly higher contributions to models' predictions than random words (see Sec. 3.1).
• We find that biased words will affect models' predictions, and that models tend to assign labels highly correlated to the biased words (see Sec. 3.2).
• To mitigate the models' over-reliance on the spurious correlation, we propose a training strategy Less-Learn-Shortcut (LLS). Experimental results show that LLS can improve the models' performance on adversarial data while maintaining good performance on in-domain data. Furthermore, we compare LLS to existing strategies and reveal their respective applicable scenarios. (see Sec. 4).

2 Preliminary

In this section, we first introduce the QM datasets on which we analyze the spurious feature-label correlation, then we give the definitions of biased word and biased example. At last, we provide the settings of our experiments.

2.1 Datasets

We conduct our analysis on three datasets, LCQMC, DuQM and OPPO 1 , all of which are about QM task and collected from real-world applications. LCQMC [Liu et al., 2018] is a large-scale Chinese question matching corpus proposed by Harbin Institute of Technology in the general domain BaiduZhidao. DuQM [Zhu et al., 2021] is a fine-grained controlled adversarial dataset aimed to evaluate the robustness of QM models and generated based on the queries collected from Baidu Search Engine 2 . OPPO is collected from OPPO XiaoBu Dialogue application and can be downloaded on CCF Big Data & Computing Intelligence Contest. The data statistics are provided in Tab. 8 (in App. A).

2.2 Definitions

Here we provide the definitions we will use in this work. If we denote W as all words in the dataset, the set of examples containing a specific word wi can be formalized as S(wi), and the frequency of wi can be formalized as fwi . We define biased degree as dcm wi to measure the degree of word wi cooccurring with category cm (for QM task, cm ∈ (0, 1)) and it can be denoted as

1 The datasets can be downloaded on https://luge.ai.

2 http://www.baidu.com.

where |S(wi, cm)| represents the number of examples with wi and labeled with cm.

Biasedword. Aword highly correlatedwith a specific label in a dataset. 3 To better discuss it, we define biased word as the word wi with fwi ≥ 3 and dcm wi ≥ 0.8 for QM task in Sec. 2 and 3. It is worth mentioning that the biased words we analyze in this work are originated from the training set.

We further define biased word0 and biased word1 as the words highly correlated to category 0 and 1. As shown in Tab. 9 (in App. C), "简便" ("handy") occurs in 35 examples, 33 of which are with category 1, hence it is a biased word1. Tab. 1 shows that 27.24% (15864/58230) of words are biased words, and there are more biased word0 than biased word1 in LCQMCtrain.

Biased example. An example containing at least one biased word. As shown in Tab. 2, 41.15% of examples in LCQMCtrain are biased examples, which are 25.97%, 32.25% and 24.98% in LCQMCtest, DuQM and OPPO respectively. Since the biased words occur in almost half of the examples in LCQMCtrain, it is meaningful to study their effects on models. The examples without biased words are defined as unbiased example.

2.3 Experimental Setup

Models. We conduct our experiments on three popular publicly available pre-trained models, BERT-base 4 , ERNIE1.0 5 and RoBERTa-large 6 .

3 Word is the smallest independent lexical item with its own objective or practical meaning. We use Lexical Analysis of Chinese [Jiao et al., 2018] (https://github.com/baidu/lac) for word segmentation in this work.

4 https://github.com/google-research/bert.

5 https://github.com/PaddlePaddle/ERNIE.

6 https://github.com/ymcui/Chinese-BERT-wwm.

(a) Training loss curve of RoBERTa.

0.1 0.2 0.3 0.4 Random order Bias-first No-bias-first Loss Epoch Figure 1: Training loss curves of RoBERTa on LCQMCtrain, in which · represents the time of finishing learning biased examples, and ▲ represents the time of finishing learning unbiased examples. Metrics. As most of the classification tasks, we use accuracy to evaluate the performance of models. Training details. We use the integrated interface BertForSequenceClassification 7 from huggingface for our experiment and use different learning rates for different pre-trained models. Specifically, for RoBERTalarge, the learning rate is 5e-6. For BERTbase and ERNIE1.0, the learning rate is 2e-5. The proportion of weight decay is 0.01 and the batch size is 64. We train two epochs for BERTbase and ERNIE1.0, and train three epochs for RoBERTalarge. Every 500 steps, we check the performance of models on LCQMCdev and choose the checkpoint with the highest accuracy as our main model, and report average results with three different seeds on LCQMCtest, DuQM and OPPO.

3 Effect of Feature-Label Correlation

The dataset statistics in Sec. 2 show that 41.15% of examples in LCQMCtrain contain biased words. It is a reasonable assumption that the spurious feature-label correlations would affect the models' behavior and performance. To validate our assumption: 1) we conduct a behavior analysis of the model's learning and deciding (See Sec. 3.1); 2) we discuss how the feature-label correlation affects the models' performance by probing the relationship between the biased word and the predicted label (See Sec. 3.2). In Sec. 3.3 we discuss another type of shortcut word-overlap and argue that different shortcuts may interact together.

3.1 Feature-Label Correlation andModels' Behavior

Models' learning. To observe the models' behavior during training, we separate LCQMCtrain into two subsets, biased examples and unbiased examples, and reorganize the train examples in 3 orders:

• bias-first: firstly biased examples, then unbiased examples;
• bias-last: firstly unbiased examples, then biased examples;
• random order: shuffle the examples randomly.

We finetune three models (BERT, ERNIE and RoBERTa) in above three orders and plot the training loss curves in Fig. 1

7 https://huggingface.co/docs/transformers/.

(a) Results on LCQMCtest.

(b) Results on DuQM.

and Fig.4 (see in App B). The training loss curves of all three models present the same tendencies:

• If bias-first, for each epoch, the loss curve drops more rapidly than random order. After learning all the biased examples, the loss curve rises slightly and then decreases.
• If bias-last, the tendency is contrary: for each epoch, the loss drops more slowly than random order until all the unbiased examples have been learned, and then the curve decreases faster.

The above observations reflect that models behave differently when they learn biased examples and unbiased examples: the loss curves of biased examples drop more sharply than other examples, which indicates that the words highly correlated with specific labels are relatively easier for models to learn, and the correlations between words and labels are captured by models as shortcuts.

Models' deciding. In this part, we provide a quantitative analysis of the spurious feature-label correlation's impact on models' deciding. If it is easier for a model to learn, will the biased words make greater contributions when predicting? Here we select LIME method to measure the contributions of different words in one input to the final prediction, which can interpret the models' prediction based on locally approximating the model around a given prediction.

To observe the contributions of the biased words, we rank the words based on their contribution scores computing with LIME method. In Fig. 2, we illustrate the ratios of biased words with the highest, second, third, and fourth contribution in three test sets. For comparison, we randomly select words from the input excluding stop-words as random word baseline, and also plot their ratios in Fig. 2. Compared to the random words, the biased words have significantly higher ratios to be ranked among the highest 4, which is about 80% in LCQMCtest and DuQM, 68% in OPPO.

In summary, the biased examples are easier for models to learn, and the biased words make significantly higher contributions than random words, which implies that models tend to pay more attention to biased words when deciding. With the analysis in this section, we can conclude that the biased word is a shortcut for the models and will affect the models' behavior. It is therefore substantial to further analyze how it affects the models.

3.2 Feature-Label Correlation andModels' Prediction

Existing works focus only on verifying the existence of shortcut [Bolukbasi et al., 2016; May et al., 2019; Ravfogel et al., 2020; Webster et al., 2020; Kaneko and Bollegala, 2021]. However, there are few quantitative analyses to discuss how the shortcut affects the models' predictions exactly. In this part, we will focus on probing the relationship between the biased word and predicted label to analyze how the spurious feature-label correlations impact models. As the biased words are highly correlated to a specific label, it is a reasonable guess that the models tend to assign predicted labels highly correlated to the biased words.

Although the biased words tend to contribute more (discussed in Sec. 3.1), not all biased words make great contribution during predicting. To better analyze the impact of biased word on predicted label, we focus on the testing examples where biased word contributes the most, in which the effects of biased word would be more significant. For convenience, we define the examples in which the biased word makes the greatest contribution as focus-biased examples, and we present the statistics of biased examples and focusbiased examples in Tab. 10 and Tab. 11 (see App. D). To measure the tendency of models' prediction, we define Tcm as the tendency of model to predict of category cm:

where |D| represents the number of observed examples, |Strue(cm)| and |Spred(cm)| represent the number of examples with true label cm and predicted label as cm respectively. Specially, we observe the tendency of models' prediction on "normal" biased examples and focus-biased examples, and denote them as Tcm and T focus cm . The results are shown in Fig. 3. Fig. 3(a) to Fig. 3(c) show the influence of biased word0 on three test sets. On DuQM (Fig.3(b)), it is obvious that T focus 0 is higher than T0 by averaged 7% with all three models, which implies that when biased word0 contributes the most, models have a high tendency to predict 0. The same result is shown on LCQMCtest (Fig. 3(a)). However, on OPPO (Fig.3(c)), T0 is slightly higher (0.02˜0.05) than T f ocus 0 . We suppose that it is affected by the co-influencing of another shortcut and we provide an extensive experiment to discuss it in Sec. 3.3. Fig. 3(d) to Fig. 3(f) show the influence of biased word1. As shown in Fig. 3(f), models tend to predict 1 when they concentrate on biased word1 on OPPO, that T focus 1 is higher than T1 by averaged 26% with all three models. The comparison results on DuQM(Fig.3(e)) show the same tendency for all three models, that T focus 1 is higher than T1 by averaged 6%. On LCQMCtest (Fig.3(d)), T focus 1 is almost close to T1 with all three models.

Overall, we observe that when models pay more attention to biased words, they tend to assign labels over-relying on the biased words. Moreover, to explore why the tendency to 0 is not obvious on OPPO (Fig. 3(c)), we provide a further discussion about the influence of another shortcut word-overlap.

3.3 Word-Overlap: Another Shortcut for QM Models

In real-world scenarios, different shortcuts may interact together to affect the final prediction. Word overlap shortcut has been widely discussed in many MRC and NLI works [McCoy et al., 2019; Lai et al., 2021; Kaushik and Lipton, 2018]. For QM task, the models tend to predict 0 if a sentence pair has low word overlap, i.e., there are few common words between them, and vice versa. As the result of OPPO shown in Tab. 3, even if models focus on biased word0, the tendency to 0 is not significant. We attribute the phenomenon to the wordoverlap shortcut in the QM task. To eliminate the influence of word-overlap, we design an experiment on the examples in which the question pairs with high word-overlap. We use LevenshteinEdit distance to measure the overlapping degree. We report the models' prediction tendency with short edit distance in Tab. 3. The results reflect that models have a higher tendency to predict 0 on focus-biased examples than "normal" biased examples, which implies that models tend to predict 0 if we try to eliminate the word-overlap shortcut. Specifically, compared with "normal" biased examples, the average T f ocus 0 of three models with edit distance 1 increases by 0.086, which is 0.040, 0.028, 0.027 and 0.022 for edit distance of 2, 3, 4 and 5.

Generally, we can deduce that models tend to assign labels relying on the feature-label correlation trick. By eliminating the influence of word-overlap, the models' prediction tendency towards 0 becomes significant on OPPO. Besides the spurious correlations we study in this work, NLP models are also affected by many other shortcuts.

4 Less-Learn-Shortcut: A Training Strategy to Mitigate Models' Over-Reliance on Feature-Label Correlation

In Sec. 3 we observe that the spurious feature-label correlation will affect models' learning and deciding. To mitigate the models' shortcut learning behavior, we propose a training strategy Less-Learn-Shortcut (LLS), with which all the biased training examples are penalized according to their biased degrees (in Sec. 4.1) during fine-tuning. Most of the existing strategies to mitigate shortcut learning include data augmentation [Jin et al., 2020; Alzantot et al., 2018] and adversarial training [Stacey et al., 2020], which are task-relevant. Our proposed method LLS is task-agnostic and can be easily transferred to different NLP tasks.

4.1 Reweight Biased Examples

To mitigate the models' over-reliance on the feature-label correlations, a straightforward idea is to down-weight the biased examples, so that the models are prevented from over-fitting the spurious correlations. In this section, we will introduce how we reweight the biased examples.

Quantify the impact of correlation. In Sec. 2, we have defined biased degree dcm wi to measure the correlation between the word wi and the label cm, which can quantify the impact of the correlation. The maximum biased degree of a word among all categories is denoted as b◦wi (C represents all categories).

Furthermore, some existing works show that the word frequency in the training data also influences the models' prediction [Gu et al., 2020; Cui et al., 2016; Ott et al., 2018]. Considering both biased degree and word frequency, we formulate the impact of a biased word as

where fwi represents the frequency of words wi occurring in the training dataset, and α is a trade-off factor. Then the impact of a biased example can be formulated as the average impact of all biased words it contains:

Exclude the impact of word-overlap for sentence pair tasks. In Sec. 3.3, we observe that word-overlap is another shortcut in the sentence pair tasks: models tend to give positive predictions to sentence pairs with high word-overlap and negative to low word-overlap. To exclude the impact of wordoverlap, when two shortcuts conflict in one biased example, we only consider the impact of the biased word with minimum biased degree:

Calculate the loss weight of the biased examples. We rescale be of all biased examples with the min-max normalization. The loss weights of the biased examples can be denoted as follows, with which the biased examples with higher be will be assigned smaller loss weights:

where ej belongs to biased example and β is used for adjusting the low bound of the normalized interval, and the high bound is fixed to 1. It is worth noting that we only reweight the biased examples, and the loss weights of unbiased examples are 1.

4.2 Experimental Results

First, we introduce our comparison baseline and then conduct experiments on QM task, NLI task (sentence pair classification task), and SA task (single sentence classification task). Detailed information about the experimental settings can be found in App.E. We present the average results of three different seeds and our performance improvements are statistically significant with a p-value of paired t-test less than 0.05.

Baseline. In addition to select Finetune as our baseline, we re-implement Rew.bias [Utama et al., 2020; Clark et al., 2019] and Forg. [Yaghoobzadeh et al., 2021] strategies. The core idea of Rew.bias is similar to our LLS strategy, which is down-weighting biased examples. Rew.bias needs to additionally train a bias-only model to score biased examples. Forg. strategy uses examples forgotten by the model during training to do secondary training. The above two strategies are both task-agnostic and can be applied to any NLP task.

QM task. As shown in Tab. 4, for BERT and RoBERTa, our LLS strategy performs best on both the in-domain LCQMCtest (87.86% and 88.46%) and adversarial DuQM (69.20% and 74.18%). For ERNIE, our LLS strategy performs best on in-domain LCQMCtest (88.16%) and performs close to best on the adversarial DuQM (71.65%). Furthermore, we observe that although Rew.bias and Forg. strategies improve the model performance on the adversarial DuQM, they only remain performance on LCQMCtest and OPPO.

By contrast, our LLS strategy can improve the model performance on all three test sets.

To better investigate the contributions of different components of LLS, we compare LLS with two ablations: LLSd only employs the biased degree to measure the impact of correlation, and does not consider the impact of word-overlap (Eq. 3, 5, and 7); LLSd+f considers both biased degree and word frequency to measure the correlation, but also does not consider the impact of word-overlap (Eq. 4, 5, and 7). As shown in Tab. 4, LLS generally performs the best, indicating that considering word frequencies and excluding word-overlap has a positive effect.

NLI task. NLI task aims to determine the relationship between two sentences, whether a premise sentence entails a hypothesis sentence. It is normally formulated as a multi-class classification problem. In our experiments, we try two NLI datasets as the training sets, SNLI [Bowman et al., 2015] and MNLI [Williams et al., 2017]. Tab. 5 and 6 give the statistics of SNLItrain and MNLItrain. Although only 2.96% of words in SNLItrain are biased words, they occur in 4.84% of examples. Compared to SNLItrain, MNLItrain is relatively unbiased and contains only 202 biased words (0.22%) and 993 biased examples (0.25%).

We first conduct our experiment on SNLItrain. We train models on SNLItrain and evaluate them on the in-domain SNLItest and the adversarial HANStest. SNLI is a dataset with three classes: entailment, neutral, and contradiction. HANS is a two-class dataset, entailment and non-entailment. As done in previous work [McCoy et al., 2019], to evaluate models on HANStest, we convert neutral or contradiction labels to non-entailment. The experimental results are shown in Tab. 4. For BERT and ERNIE, Forg. strategy improves the model performance more significantly on the adversarial HANStest. We present the statistics of forgotten examples (see App. F), and observe that for the large-scale SNLItrain, small models such as BERT and ERNIE are more likely to forget examples. Therefore, secondary training with forgotten examples can better help small models increase their robustness. In contrast, for the large RoBERTa model, Forg. strategy yields little and our LLS strategy performs better. Furthermore, compared to Finetune and Rew.bias strategies, for all three models, our LLS strategy obtains a more significant benefit on the adversarial HANStest while maintaining good performance on the in-domain SNLItest.

The results on MNLItrain are shown in Tab. 7. Due to the fact that the MNLItrain contains fewer biased examples, the effect of LLS is not significant. This suggests that LLS strategy is more effective for the training set with biased data distribution, helping models learn the spurious correlation less.

SA task. SA task aims to determine whether a sentence has a positive or negative sentiment. In our experiment, we train models on Chnsenticorptrain 8 and evaluate them on the in-domain Chnsenticorptest and the adversarial SENTIrobust [Wang et al., 2021]. As shown in Tab. 5 and Tab. 6, 14.05% of words in Chnsenticorptrain are biased words, and they appear in 95.11% of the examples. Unlike NLI and QM tasks, SA task is a single-sentence classification task that is not affected by the word-overlap shortcut, thus we only report the results of LLSd and LLSd+f in Tab. 4. Compared to Rew.bias and Forg. strategies, our LLSd and LLSd+f strategies obtain a more significant benefit on the adversarial SENTIrobust and perform better on the in-domain Chnsenticorptest. Additionally, it is worth noting that for the small-scale Chnsenticorptrain, models will not forget too many samples (see App. F) and Forg. strategy yields little.

In summary, our proposed LLS strategy can significantly improve the model performance on adversarial data while maintaining good performance on in-domain data. Our experiments show that existing strategies struggle to stably improve performance on in-domain data, making further research necessary. Furthermore, we reveal scenarios in which these strategies are applicable. Compared to the Rew.bias strategy, LLS strategy demonstrates greater advantages on various tasks. However, LLS strategy is not applicable for the relatively unbiased dataset, such as MNLItrain. On the other hand, Forg. strategy shows its own advantages on SNLItrain. Specifically, when training a small model on a large-scale dataset, Forg. strategy is a good option to consider.

5 Conclusion

In this paper, we explore models' shortcut learning behavior of spurious correlations between features and labels, and propose a training strategy LLS to mitigate the over-reliance of NLP models on the shortcut. Specifically, we observe that the models are prone to learn spurious correlations, and the biased words make significantly higher contributions to models' predictions than random words. Moreover, we observe that the models tend to be misled by biased words to assign labels. To mitigate the over-reliance on biases, we propose a training strategy LLS to penalize the shortcut learning behavior of models. Experimental results show that LLS can improve the model performance on adversarial data while keeping good performance on in-domain data, and it is taskagnostic, which can be easily transferred to other tasks. In future research, we will explore how to better measure and formalize the shortcuts in the training data and generalize them as a class of problems.

8 https://github.com/pengming617/bert classification

A Data Statistics

Data statistics are presented in Tab. 8.

B Training Loss

(a) Training loss curve of BERT.

(b) Training loss curve of ERNIE.

Epoch

Epoch D Statistics of Bias-Example and Focus-Bias Examples

The statistics of "normal" bias-examples and focus-bias examples are given in Tab. 10 and Tab. 11.

E Experimental Settings

Experimental settings are introduced Tab. 12.

F Statistics of Forgotten Examples

In Tab. 13, we record the number of examples forgotten by models on different tasks.

Acknowledgements

This work was supported by the National Key R&D Program of China [2021ZD0113302]; and the National Natural Science Foundation of China [62206079]; and Heilongjiang Provincial Natural Science Foundation of China [YQ2022F006].