A Survey on Out-of-Distribution Evaluation of Neural NLP Models

1 Introduction

Neural NLP models have been reported for their superhuman performance on many language understanding and generation tasks, such as sentiment analysis and machine reading comprehension (MRC). Recent studies show that these models lack human-level understanding of language since they are vulnerable beyond in-distribution (ID) test data and fail to generalize on out-of-distribution (OOD) data, such as perturbed examples under adversarial attacks [Ebrahimi et al., 2018], text from different domains [Hendrycks et al., 2020] and examples against dataset biases [Gururangan et al., 2018], which lie in the three mainstream lines of research: adversarial robustness, domain generalization and dataset biases.

Despite recent progress in each of the three research lines, there has not been a survey that comprehensively reviews and investigates the connections between these lines. We have the following motivations for such an investigation: 1) explaining some phenomena across the three lines of research. For example, methods designed to improve model generalization on one OOD type can defend against other types of OOD data [Yi et al., 2021] but sometimes degrade model performance on other OOD types [Bras et al., 2020; Gokhale et al., 2022]; 2) encouraging future researchers to perform a comprehensive OOD evaluation while proposing their new methods; 3) suggesting that all the OOD types be addressed thoroughly in future research.

In this survey, we first unify the three research lines as the study of distribution shift, which gives us a starting point to find connections between them in terms of shifted features

(§2). Furthermore, we summarize their differences in OOD data generation and evaluation protocols (§3). In particular, we categorize different methods to generate OOD data, including data with natural domain shift (NDS) for domain generalization, debiased data for evaluating dataset biases and adversarial examples for adversarial robustness. We then categorize evaluation protocols for the OOD data into two classes: data-based and method-based evaluation. Finally, we identify opportunities based on the discussion of connections and differences between the three OOD types, including developing a comprehensive benchmark for OOD evaluation, advising caution about breaking the assumptions of covariate shift, improving general OOD performance and using the detection methods across different OOD types. Also, we demonstrate the gap in adversarial robustness for future work (§4).

2 Definition

This section first introduces a comprehensive definition of distribution shift across the three research lines and then shows how the definition aids in elucidating the interconnections regarding the shifted features.

2.1 Distribution Shift

Deep Neural Network (DNN) f (θ) can achieve remarkable or even superhuman performance on NLP benchmarks, such as GLUE with in-distribution (ID) test data Dtest ∼ P0, when f(θ) is trained on data sampled from the data-generating distribution (an unknown distribution under a data-generating process) Dtrain ∼ P0. However, the well-trained model often fails to generalize to various unforeseen examples. Formally, the unforeseen data (or OOD data) ˜D can be characterized by a shifted distribution ˜P. The distribution shift includes covariate shifts (input distribution shifts) P0(X) ̸= ˜P(X) and label distribution shifts P0(Y) ̸= ˜P(Y), where X is the input space, and Y is the ground-truth output space. We unify the three lines of research as a study of covariate shifts. For brevity, the input notation X is sometimes omitted when referring to it, as exemplified by the use of P0 to represent P0(X).

Domain generalization and dataset biases. Real-life documents in different domains, e.g., news and fiction text [Hendrycks et al., 2020], are typically characterized by systematically distinct properties and originate in different datagenerating distributions. Domain generalization investigates such natural domain shifts, and relevant literature often uses domain shift and distribution shift interchangeably. Some literature [Arora et al., 2021; Gokhale et al., 2022] also defines that data with particular biases and those without such biases come from different domains. Therefore, the studies of domain generalization and dataset biases can be intuitively defined in the framework of distribution shifts.

Adversarial robustness: from robustness to distribution shift. Initially, adversarial methods evaluate model robustness against small perturbations in the worst-case scenarios, especially for continuous data like images [Goodfellow et al., 2015]. Hence, prior studies typically separate adversarial robustness from the study of domain generalization and distribution shift. We can unify adversarial robustness as a study of distribution/domain shift. Conceptually, we can define the shifted distribution in the adversarial setting as a distribution around the original distribution P0. We perform a perturbation on text x ∈ P0(X) to simulate OOD data from the shifted distribution. In other words, adversarial perturbations encode subtle domain shifts, as discussed in §2.3, and the shifted domains are characterized by the adversarial methods summarized in §3.3.

2.2 Shifted Features

To concretize the shifted features, we first define task-relevant features srel and task-irrelevant features sir, which satisfy P(srel|y) ̸= P(srel) and P(sir|y) = P(sir), respectively. P is an arbitrary distribution.

Background features and semantic features. When P is the ground-truth distribution Ptrue, Arora et al., [2021] name srel as semantic features since most realistic NLP tasks address the semantics of text, and sir as background features, which can be the syntax, writing style or task-irrelevant text.

Biased features are task-irrelevant features under Ptrue despite being learned as task-relevant features from P0. P0(X) can be factorized into the conditional distributions of generalized features g(X) and biased features b(X).

assuming conditional independence between b(X) and g(X) given y. By definition, g(X) should always be task-relevant features in the true distribution Ptrue, i.e., g(X) ∈ srel, while biased features b(X) are only predictive for ID dataset D ∼ P0. We can express b(X) as {s|s ∈ P0 and s /∈ ˜P}.

2.3 Shifted Features in Three OOD Types

This section shows how each OOD type can potentially shift the three kinds of features.

Shifted features on adversarial examples. Shifted features of adversarial examples depend on the attack methods. White-box and grey-box methods compute gradients or output logits, respectively, while black-box methods can only query final predictions. With the guidance of gradients, white-box methods tend to shift the biased features since the input gradients can help identify the biased features for perturbations [Ilyas et al., 2019; Wallace et al., 2019a]. Perturbation types largely affect whether it is a semantic shift or background shift (See §3.3). For example, character editing may lead to unseen words, e.g., "wonderful" to "wonderful", which belongs to semantic shift. In contrast, black-box methods explicitly change background features at the sentence level. For example, Qi et al., [2021] paraphrase with different writing styles.

Shifted features on debiased data. Debiased data, by definition, is generated by removing biased features. However, the generation process often encodes either background features sir or semantic features srel. For example, debiased data generated by syntactic templates [McCoy et al., 2019] encode background shift, while MRC data requiring numerical reasoning to avoid dataset biases [Naik et al., 2018] change semantic features.

Shifted features on NDS data. We classify natural domain shift into two categories, namely genres and sub-populations. Both background and semantic shifts can occur in both categories, such as different vocabularies associated with subpopulations or genres [Barrett et al., 2019]. For example, for sentiment analysis, Tweets tend to express sentiments with emotions compared to the genre of movie reviews, and sub-populations can differ in the way to express emotions [Oprea and Magdy, 2020]. Furthermore, DNNs can easily learn background features for each sub-population or genre as biased features. Writing styles can also serve as another background feature that changes across genres [Arora et al., 2021], e.g., text from fiction books to Wikipedia content.

3 OOD Performance Evaluation

This section compares the concrete methodologies of OOD data generation and evaluation protocols for each line of research. Their differences are summarized in Table 1.

3.1 NDS Data Generation

NDS data can be generated from different genres or subpopulations. The former can be acquired from various data sources, whereas the latter involves partitioning data into subpopulations based on attributes of entities or individuals. Diverse sets of data genres may display a fusion of unique textual styles, syntactic structures, and lexicons.

Genres. Different genres refer to text written by different authors or annotators or for different audiences, or come from different data sources. For example, academic papers are written in formal language while online reviews on Yelp contain non-standard orthography. Since a training dataset typically comes from a specific data source and contains only one genre, even well-trained models generalize poorly to different genres. Although some datasets are created to include multiple genres of text, models still perform worse beyond the coverage of these genres. For example, MultiNLI for NLI consists of text from ten distinct genres. However, even the large pretrained language models cannot generalize well to a different genre [Arora et al., 2021], such as SNLI from image captions or WNLI from fiction books. Take text classification as another example. Classifiers cannot perform well on text from disjoint annotators, e.g., SST data from experts v.s. lengthy IMDB reviews from laymen [Hendrycks et al., 2020; Arora et al., 2021]. Also, MRC models trained on Wikipedia hardly generalize on data from New York Times articles, Reddit posts, and Amazon product reviews.

Sub-populations. Multiple studies identify NDS data by dividing data into sub-populations according to different attributes of objects or individuals obtained from metadata. Arora et al., [2021] select reviews from different businesses, e.g., restaurant reviews v.s. movie reviews. For the same business, Hendrycks et al., [2020] split them according to different types of products (e.g., women's clothing, shoes) or restaurants (e.g., American v.s. Chinese restaurants). Furthermore, model performance can vary on data with different demographics, which is closely connected to another line of research about fairness. For example, Borkan et al., [2019] find the worst test accuracy on non-toxic comments from the black population.

3.2 Debiased Data Generation

In this section, we first divide dataset biases into two categories and introduce a text summarization dataset bias that does not fall under either one. We then classify datagenerating methods into two categories according to the knowledge of dataset biases. Table 2 and 3 summarize dataset biases and debiased datasets for various NLP tasks, respectively.

Dataset Biases

Lexical correlations. Words frequently appearing in the training examples of a particular class can be predictive biases for that class. For example, when NLI annotators create a majority of contradictory hypotheses by negating premises, models learn the spurious correlation between negation words ('no', 'never') and labels (hypothesis-only reliance [Gururangan et al., 2018]). The success of class-wise adversarial attacks comes from exploring these predictive words to some extent [Wallace et al., 2019a]. Similarly, claim-only models can perform well on ID test data without the context of evidence for fact-checking due to predictive words [Schuster et al., 2019]. MRC models can find the answer in a given paragraph by matching the question type, e.g., "November 2014" for a "when" question (question word matching [Lai et al., 2021] or entity type matching [Sugawara et al., 2018]), and hence they are easily distracted to the wrong answers on OOD data [Jia and Liang, 2017]. Emelin et al., [2020] observe that machine translation systems have a tendency to disambiguate words based on the sense that occurs most frequently in the training data.

Lexical overlap. Lexical overlap is another predictive indicator for sentence-pair classification. For example, the high overlap between the premise and hypothesis leads to the "entailment" prediction from NLI models (premise-hypothesis overlap [McCoy et al., 2019]). The classifiers for paraphrase identification also tend to predict highly overlapping sentences as paraphrases [Zhang et al., 2019]. Besides, MRC models can locate the sentence with maximum overlap words in the paragraph (context matching [Sugawara et al., 2018]) and then use question word matching to extract the correct answer.

Positional biases for text summarization. News or journal articles tend to summarize content in the lead paragraphs (positional bias or layout bias). Specifically, Kedzie et al., [2018] show that 88.6% reference summaries from the training examples of NYT come from the first half of documents (69% for DUC, 71.7% for CNN/DM).

Methodologies

There are two types of methods to generate debiased data: 1) breaking the spurious correlations; 2) filtering the examples containing superficial patterns automatically without recognizing any specific dataset biases.

Bias-informed approaches. Many studies generate debiased data by reversing the correlations between dataset biases and labels for text classification. For natural language inference (NLI), Mccoy et al., [2019] specify linguistic phenomena (constituent and subsequent heuristics) behind lexical overlap. They devise heuristics-based syntactic templates to create sentence pairs that have high lexical overlap but contradict each other. Kaushik et al., [2020] construct counterfactual examples by breaking spurious correlations in sentiment analysis datasets (i.e., IMDB) and NLI datasets. They annotate the text into the targeted label by largely keeping the original text. In this way, the dataset biases of the original label are kept. To break the lexical overlap between paraphrase pairs, Zhang et al., [2019] create non-paraphrase sentences via word scrambling. This method changes the meanings of paraphrases but keeps the overlapping words. For MRC, Jia and Liang [2017] add a sentence containing words overlapping with the question into paragraphs and find that models tend to select answers from the sentence. They also combine the bias-inspired approach with a grey-box attack to generate adversarial examples. Gardner et al., [2020] rely entirely on experts who have the knowledge of dataset biases to generate debiased data.

Systematic approaches. Debiased data can also be generated by bias mitigation techniques without knowing dataset biases. For example, Bras et al., [2020] use a simple classifier to identify biased examples and generate debiased datasets for NLI, commonsense reasoning and grounded commonsense inference, respectively. Although some works try to train robust models with such filtered data, filtering always leads to significant drops on ID data and even OOD data [Gokhale et al., 2022]. Therefore, it can be more practical to use them for OOD evaluation.

3.3 Adversarial Example Generation

Given a well-trained model, adversarial examples are generated by modifying the given text (or reference data) to make the model output wrong predictions. Typically, the generation process satisfies semantics-preserving and labelpreserving assumptions. Mostly, semantics preservation can guarantee the same label for a perturbed text. Thus, a robust model should generate invariant outputs arg max y∗ f (x) =

arg max y∗ f(˜x) = y under semantics-preserving perturbations,

where y is the ground-truth label and ˜x is the perturbed text. Adversarial attacks have been widely studied to automatically generate adversarial examples. We categorize the attack methods according to their perturbation types, perturbation space and adversary's knowledge. Table 4 exemplifies typical methods for each category.

Perturbation Types

There are four common perturbation types that exhibit different granularities, namely, character editing, word substitution, paraphrasing and universal adversarial perturbation (UAP).

Character editing mimics real-life accidental typos or spelling variants in social media via character swapping (e.g., "place" ⇒ "plcae", deletion (e.g., "artist" ⇒ "arlist"), insertion (e.g., "computer" ⇒ "comnputer") and substitution (e.g., "computer" ⇒ "computor") [Gao et al., 2018]. Humans are robust to spelling errors maintaining certain morphological or phonological characteristics, e.g., "computer". Hence, a few edits on a word would not affect the human perception of the word but may lead to completely opposite predictions by the models. In contrast, word substitutions require semanticspreserving constraints on substitute words or rely on a vocabulary of synonyms for substitutions [Zang et al., 2020]. Paraphrasing is seldom considered in adversarial attacks due to the difficulty of generating high-precision paraphrases.

The last type of adversaries follows the work to generate UAPs for images, which perturb any images of a particular class [Moosavi-Dezfooli et al., 2017]. Behjati et al., [2019] generate the UAPs for NLP models in the form of n-grams (i.e., non-sensical phrases), which lead to misclassification when appended to text of a class. Wallace et al., [2019a;

2020] even extend the idea of UAPs for MRC, language modeling and machine translation. These studies assume that inserting non-sensical phrases does not change the semantics of text and ground-truth labels. Song et al., [2021] generate natural phrases (e.g., "natural energy efficiency") as UAPs via Adversarially Regularized Auto Encoder (ARAE). However, it can only maintain the coherence of UAPs rather than the whole perturbed text.

Perturbation Space

A text consists of a sequence of tokens (e.g., words or subwords), denoted as x = w1, w2, . . . , wT , where T is the number of tokens in the vocabulary V = {w1,w2, . . . ,wK} (K is the size of the vocabulary). Note that the vocabulary can differ from the ones of the victim models, especially under black-box attacks.

All types of perturbations, except for paraphrasing, can be perturbed in a discrete space. When perturbing via word substitutions or UAPs, the objective is to identify significant tokens in x and substitute them with tokens from the vocabulary V in order to generate incorrect model predictions. This is a combinatorial optimization problem, where the size of the search space is KT . The search space is intractable and is typically addressed using heuristic-based approaches [Alzantot et al., 2018; Zang et al., 2020] or approximation methods [Ebrahimi et al., 2018; Wallace et al., 2019a]. Character editing also requires identifying significant tokens, but it involves a subsequent manipulation of characters within the identified tokens [Gao et al., 2018]. Assuming the average token length is l, the search space becomes lT , leading to combinatorial explosion of modifications.

There are two types of methods to perturb text in the continuous space: 1) perturbing tokens (characters/subwords/words) in the embedding space. Gong et al., [2018] add continuous perturbations directly to the continuous representation ew of token w via element-wise addition between the perturbation η and ew, similar to pixel-wise addition for images [Goodfellow et al., 2015]. To transform η+ew back to the text, they search for the nearest token ˜w in the embedding space where e ˜w + η ∈ V. However, it probably generates a semantically variant token, because the closest and most legible token can be far away from the original one in projection space. Note that they still require searching for important tokens. 2) reparameterization for paraphrasing or UAPs. Zhao et al., [2018] reparameterize the text x into z in a continuous space and then perturb z into ˜z. They also train a text generator (e.g., LSTM) to decode ˜z back to text. Song et al., [2020] apply the reparameterization trick for UAP generation.

Moreover, generation models are commonly used to create paraphrases such as machine translation models for backtranslation [Ribeiro et al., 2018], a syntactically controlled paraphrase network [Iyyer et al., 2018] or a text style transfer model [Qi et al., 2021]. The latter two specifically change task-irrelevant features.

Adversary's Knowledge

We can categorize the attack methods into three types (whitebox, grey-box and black-box) according to the three levels of model knowledge: model parameters, output logits (i.e., estimated probability distribution) and final predictions.

The white-box methods require model parameters to propagate gradients back to the input (e.g., the gradient of an adversarial loss with respect to the input word embedding ∇ew L). They need ∇ew L to find important tokens and generate perturbations, e.g., the noise in the embedding space or substitute tokes. Liang et al., [2018] adapt the gradient magnitude to find important words for perturbations and search for substitute words for each targeted label since the magnitude in each dimension of the word embedding indicates the sensitiveness of the prediction to the change. Gong et al., [2018] directly use gradients to perform perturbations on the embedding space. Ebrahimi et al., [2018] and Wallace et al., [2019a; 2020] use the gradients and word embeddings to approximate the loss for word substitutions. Specifically, they approximate the loss change of substituting a word w with another word s in the vocabulary by the inner product of the word embedding es and ∇ew L, where L is the adversarial loss. The selected word s is expected to minimize the adversarial loss on the perturbed text ˜x.

Under the grey-box setting, an adversary only has access to the probability distribution (or logits) over the output variable Pr(y|x) = f(x), where Pr(yi|x) is the probability of the outcome yi given the text x. Grey-box methods employ the probabilistic information to measure how important each token wi ∈ x is to the prediction. For wi, its importance score Ii is commonly calculated as the difference of the probabilities of the correct class Pr(ytrue|x) - Pr(ytrue|x\wi ) [Li et al., 2020], where x\wi can either be {w1, . . . ,wi-1,wi+1, . . . ,wT} (delete wi) [Li et al., 2019; Jin et al., 2020] or {w1, . . . , UNK, . . . , wT } (replace wi with the unknown token UNK) [Gao et al., 2018; Li et al., 2020]. Jin et al., [2020] also add the change to the misclassified class f˜ y(˜x) - f˜ y(x) if argmax y∗ f(˜x) ̸= argmax y∗ f(x), where ˜ y is the misclassified class, and f˜ y(˜x) is the probability of the misclassified class for the perturbed sample. Afterward, greybox methods apply heuristics to search for substitute words or evaluate noisy words from character editing via Pr(ytrue|˜x).

The black-box attacks require no information about the victim model and commonly generate paraphrases. We have discussed the concrete methods in the previous paragraphs for perturbation types and space.

3.4 Evaluation Protocols

Data-based evaluation. There are different evaluation protocols for the three data types. NDS data and debiased data are evaluated on the same aggregate metrics for ID data (e.g., accuracy for text classification and F1 scores for MRC). The drops of these metrics from ID data to OOD data reveal the model's ability of OOD generalization.

Method-based evaluation. It is meaningless to apply the ID metrics to adversarial examples unless used for adversarial transferability, since adversarial robustness only concerns the misclassified examples for the victimmodel. Instead, adversarial robustness is evaluated by the effectiveness of an attack method generating adversarial examples. The common evaluation metrics are attack success rate and the number of queries. Specifically, given a test sample Dtest, attack success rate measures the proportion of the data points in Dtest which can be perturbed to fool a victimmodel. It is calculated by dividing the number of adversarial examples that successfully achieve attack goals, e.g., misclassification, by the size of Dtest. The number of model queries during an attack measures how many perturbed examples have been evaluated before a valid adversarial example appears. Besides the query for predictions, white-box and grey-box methods also query victim models to get gradients or output logits. These two metrics can be used to compare the adversarial robustness between models (with fixed attack methods) [Gokhale et al., 2022] and the performance of different attack methods [Li et al., 2020] (with fixed victim models).

Adversarial transferability. Adversarial transferability is another evaluation type for adversarial robustness but uses the data-based evaluation method. It generates adversarial examples from a base model and then uses them to evaluate other models. The vulnerability can transfer across different model architectures and datasets [Wallace et al., 2020]. Liang et al., [2021] show that transferability can even occur across tasks as long as the source data distribution and the target one are close. Adversarial transferability encourages the generation of costly human-made adversarial examples since they can be helpful in evaluating and improving the robustness of various models. For example, Nie et al., [2020] and Wallace et al., [2019b] show model predictions or interpretations (e.g., highlighting words that are important for model predictions) to crowdworkers to assist in the generation of adversarial examples.

4 Opportunities and Challenges

This section provides insights for future work based on the above OOD discussions. The opportunities and challenges related to adversarial robustness are also investigated.

4.1 General Challenges and Opportunities

Our study provides insights into OOD evaluation, OOD generalization and OOD detection.

A benchmark for all the OOD types. Developing a comprehensive benchmark for all the OOD types can help evaluate the model's actual ability of language understanding and identify potential incapacity. Although there are open-source tools developed for OOD evaluation, most of them can only evaluate one perspective of the distribution shift. It is challenging to develop such a benchmark due to the idiomatic lock-in in each line of research, different data-generating processes and evaluation protocols. The disclosure of their connections and differences in this survey can provide insights into developing such a comprehensive benchmark for OOD evaluation. For example, adversarial robustness can be evaluated via adversarial transferability to unify evaluation metrics.

Data-generating processes may break the assumptions of covariate shift. Although all the three research lines aim to evaluate data with covariate shifts, some processes may break the assumptions of covariate shifts, i.e., fixed P0(Y|X) and P0(Y). These two assumptions also indicate P0(X|Y) as a special case of covariate shift due to P0(X|Y) = P0(Y|X )P0(X ) P0(Y) . For example, although P0(Y|X) is always assumed the same for most NLP tasks since humans usually annotate each sentence with the same label, gradient-based methods may generate adversarial examples with labels mismatching with humans' perceptions. In addition, some methods for generating debiased data may change P0(Y). For example, Zhang et al., [2019] balance examples with high lexical overlap for all kinds of labels in their debiased dataset to prevent the feature of lexical overlap from being a predictive indicator for paraphrase identification. Future work that evaluates covariate shifts (i.e., the changes of input domains) may better consider the two assumptions for data-generating processes and analyze their effects.

Enhance general OOD performance. The three types of out-of-distribution (OOD) data can be created by altering task-irrelevant features, as explained in Section 2. Our first argument suggests that OOD generalization techniques should aim to improve the acquisition of general linguistic or semantic knowledge to perform well on all three types of OOD evaluation. This accounts for the observation that unsupervised pretraining and data augmentation strategies for learning general linguistic or semantic knowledge can enhance generalization on different domains and adversarial robustness [Gururangan et al., 2020; Gokhale et al., 2022]. Notably, such augmentation techniques [Wei and Zou, 2019] may solely modify the background features of training samples without any adversarial process. Our second argument is that even though dataset filtering for bias mitigation may improve model performance on biased datasets, it may lead to inferior performance on NDS data and decreased adversarial robustness , as biased data still contains knowledge of semantic and background features that can enhance general OOD performance. Supporting evidence for this argument is provided by Bras et al., [2020] and Gokhale et al., [2022] across various NLP tasks.

Utilizing detection methods across different research lines. The connections between OOD types regarding the shifted features also motivate future researchers to apply OOD detection methods across the three research lines. For example, since density estimation methods can effectively detect NDS data with background shift [Arora et al., 2021], they may defend against black-box attacks, which always generate adversarial examples by shifting background features.

4.2 Challenges and Opportunities for Adversarial Robustness

Beyond semantics preservation. Adversarial examples can be generated beyond the assumption of semantics preservation. Chen et al., [2022] define obstinate adversarial examples, which satisfy two conditions: 1) their ground-truth labels after the perturbation are changed, but 2) the victim model maintains the original predictions. Similarly, Kaushik et al., [2020] minimally perturb test examples to change the labels of test examples. There are many reasons to explore obstinate adversarial examples. Gardner et al., [2020] argue that such examples characterize the correct decision boundary for the task. Chen et al., [2022] find that models with adversarial training on these examples reveal vulnerability to obstinate adversarial examples. Also, breaking the assumption of semantics preservation allows more adversarial behaviours for tasks beyond single-sentence classification. For example, Song et al., [2020] generate nonsensical or natural sentences, leading to invariant model outputs for sentence-pair classification. Wallace et al., [2020] change malicious content into nonsensical text which still makes machine translation models translate it into bad language.

Developing realistically harmful adversarial behaviours.

There are different model behaviours designed for malicious intents during adversarial attacks. For classification tasks, we can either specify the expected prediction (targeted attack) or accept any prediction different from the correct label (nontargeted attack). The harm of attack depends on the specific task or labels. For example, it is harmful to generate adversarial examples to evade the detection of misinformation or toxic content. In contrast, there are many possibilities of adversarial behaviours for generation tasks due to various combinations of input and output text. Cheng et al., [2020] define an attack where all the words in the output text are different from the original output sequence while the input is similar. This kind of attack can lower the standard evaluation metrics based on n-grams, like BLEU scores. Wallace et al., [2020] develop UAPs to make MT models hardly generate any translation or output random words. Targeted keyword attacks [Cheng et al., 2020] can make models generate targeted words. Future researchers can define more realistically harmful behaviours, such as the generation of malicious nonsense or racial materials.

Adversarial robustness may not be a good proxy for realistic scenarios. Adversarial perturbations may make text deviate from their real-world distribution and generate rarely occurred examples. Particularly, pure gradient-based methods only pursue worst-case perturbations without considering the naturalness of adversarial examples. Hence, to alleviate this problem, adversarial candidates are commonly validated by some metrics such as language model perplexity [Alzantot et al., 2018], and part-of-speech matching [Ebrahimi et al., 2018]. However, attack processes become computationally expensive by including these separate modules and rejecting most of the perturbed examples. Another type of approach is to use limited search space for adversarial perturbations, e.g., using synonyms for word substitutions with the compromise of worst-case performance [Zang et al., 2020]. It is worth exploring the attack methods that can inherently ensure the naturalness of perturbed text. There are some scenarios, as summarized below, where adversarial robustness against rare and unnatural text is undoubtedly critical. 1) When models' behaviours can cause devastating outcomes, e.g., astronautics or legal services. 2) When models are deployed in ubiquitous, unforeseeable scenarios. For example, Facebook translation once made the mistake of translating the simple phrase "good morning" in Arabic into "attack it" in English. Such a mysterious translation convinced the police that the user posting this message might launch a vehicle attack, resulting in the arrest of the user. 3) When attackers have sufficient malicious intent to explore rare cases, such as fact verification and security tasks where attackers would like to evade the model's detection to get unqualified access, e.g., propagating anti-social content or fake news on social media.

5 Conclusion

In this paper, we characterized and summarized the three mainstream lines of research on dataset biases, domain generalization and adversarial robustness. We encouraged future researchers to think comprehensively about OOD evaluation and improve the OOD generalization of NLP models on all the three types of OOD data. We also highlighted the gap between adversarial robustness and realistic OOD evaluation.