Low-Resource NER by Data Augmentation With Prompting

1 Introduction

Named entity recognition (NER), an essential information extraction task, aims to identify named entities of certain types (e.g., PERSON) in texts [Grishman and Sundheim, 1996]. The state-of-the-ar methods for NER are based on supervised learning, demanding a good amount of labeled data for model training [Lample et al., 2016]. However, in a real-world scenario, it is prohibitively expensive to collect large sets of labeled data in many domains (e.g., bio-medicine, military), which significantly limits the applicability of existing NER methods [Mayhew et al., 2017; Hou et al., 2020].

Recently, there is a rise of interest in investigating data augmentation (DA) methods to address tasks in low-resource scenarios [Wei and Zou, 2019; Xie et al., 2019]. For natural language processing (NLP), the most successful DA methods are based on word manipulation, showing good performance in sentence-level tasks such as text classification, paraphrase identification, and others [Wei and Zou, 2019; Xie et al., 2019; Chen et al., 2020]. However, it is challenging to apply such DA methods to NER, a token-level task sensitive to word manipulation - for example, a word substitution operation may result in a mismatch between the word and the original label [Dai and Adel, 2020; Ding et al., 2020].

In this paper, we introduce a new DA approach for lowresource NER, with knowledge elicitation from BERT [Devlin et al., 2019]. Intuitively, we may easily use BERT to create a new training example by "altering" an existing one. For instance, given a labeled sentence "James is due to leave for Nepal" with "James" marked as a PERSON, we can replace "James" with a placeholder [MASK], and then use BERT to fill in the blank. When BERT predicts a person name "Green", we obtain a perfectly new training example "Green is due to leave for Nepal". However, we may also encounter noisy case because BERT simply evaluates whether a word fits into contexts, such as "He is due to leave for Nepal", where "He" fits in the context well but is not a name.

To address the above issue, we devise two prompting mechanisms for better training data generation, and an uncertainty driven method for noise reduction. Our first prompting mechanism is a label-conditioned word replacement strategy, which incorporates the definition of an entity type (e.g., "name of person, family, ..., and fiction" for PERSON) into the word replacement process; we show that this strategy can effectively capture label-word dependencies and generate label-consistent words. Our second prompting strategy uses a question answering (QA) style prompt, which can directly generate new training samples from unlabeled texts. For example, we may use a prompt question "Who is referenced in texts?" to query a BERT model pre-trained on QA datasets for identifying PERSON entities in a sentence. We show this strategy is supplementary to the prior one and can yield promising results even in a zero-shot scenario. In addition, we devise an uncertainty-guided self-training method to reduce noise in the generated data. In particular, we set up an iterative framework for training in which at each step only credible examples, as judged by an uncertainty mechanism [Gal and Ghahramani, 2016], are supplied to the training set. We show this method can considerably reduce the effects of noise in the data and robustify the learning process.

The experimental results have well confirmed the effectiveness of our method. Particularly, in a low-resource scenario with only 150 labeled examples, our approach scores 61.2%, 36.2%, and 71.4% in F1 on three benchmarks, outperforming prior best DA methods by more than 2.0% in F1 absolutely and methods without DA by more than 40% in F1. Furthermore, our method yields encouraging results in a zero-shot scenario without the requirement of any human labeled data. We have made our code available at https: //github.com/jianliu-ml/fewNER for further investigation.

To summarize, we have three contributions:

• We investigate a new DA method for low-resource NER, based on knowledge elicitation from BERT using two prompting mechanisms. As the first study to investigate the use of prompting mechanisms for DA in NER, our work may inspire more studies in this research line.
• We devise two prompting strategies, based on labelconditioned word replacement and question answering respectively, for training data generation. We also devise an uncertainty-guided self-training strategy for noise reduction. To the best of our knowledge, this is the first effort introducing uncertainty modeling in DA for NER.
• We report promising results on three real-world NER datasets and conduct extensive analysis to provide insight into the reasons behind our method's efficacy.

2 RelatedWork

Low-Resource NER. The question of how to train a NER model in low-resource environments remains an unresolved one in NLP. Existing research has addressed low-resource NER with distantly supervision [Plank and Agi´c, 2018], cross-lingual projection [Mayhew et al., 2017; Xie et al., 2018], and meta-learning [Hou et al., 2020], which however require huge domain expertise. Motivated by the success of data augmentation (DA) in computer vision [Wei and Zou, 2019], several works have studied DA for low-resource NER. Particularly, [Dai and Adel, 2020] use label-wise token replacement, synonym replacement, and mention replacement to generate new training examples; [Ding et al., 2020] train a language modeling objective combining words and labels to generate new examples. However, because NER is a finegrained token-level task, the above approaches often need sophisticated criteria to ensure word-label consistency, which limits their applicability. Unlike previous works, we present a new DA approach for low-resource NER that can elicit knowledge from BERT and does not require considerable human intervention. Our method can generate more reliable instances while simultaneously considering noise reduction through an uncertainty-guided self-training mechanism.

The PromptingMechanism for Learning. The "pre-train, prompt and predict" paradigm has recently gained popularity in NLP research [Liu et al., 2021], which typically builds a template prompt with some slots and then use a pre-trained language model (e.g., BERT [Devlin et al., 2019]) to fill the slots for performing a task. The existing studies have apply this paradigm to address tasks including open-domain question answering, text classification, and others [Jiang et al., 2020; Schick and Sch¨utze, 2021]. To the best of our knowledge, this is the first study to investigate data augmentation via prompting methods for low-resource NER. Our approach may inspire future work on other related tasks, such as lowresource relation extraction and event extraction.

3 Approach

Figure 2 depicts the overview of our method, which includes two prompting regimes for new training data generation, and an uncertainty-guided self-training strategy for noise reduction. The technical details of our approach follow.

3.1 Label-ConditionedWord Replacement

Let (X, Y ) be a labeled example with X = [w1, ..., wn] being a sentence of n words and Y = [l1, ..., ln] being the entity label sequence in BIO schema. Our goal is to create a new sentence ˆX that matches the original label sequence Y by replacing some words in X. Here we propose a label-conditioned pre-training mechanism to capture word-label dependencies - after randomly masking a word in the sentence, we re-train a BERT model to recover it, but use the definition 1 of the entity type as a prompt. Assuming "James is due to leave for Nepal" is a training sentence to re-train BERT and if James is masked, we construct the following input to BERT:

where pmtli indicates the definition of PERSON, i.e., "Named person or family". In this way, the recovery of a word is conditioned not only on contexts, but also on the entity label, and therefore the BERT mode is tuned to favor words matching the labels. For new data generation, we randomly mask a word in a sentence and generate a substitution by sampling from the predictive probabilities. We limit the number of generated examples to T for each sentence.

3.2 Prompting with Question Answering

Given that label-conditioned word replacement still requires some labeled data for model training and can only generate instances similar to the existing ones, we devise a query-style prompting mechanism that can generate novel examples from unlabeled texts. The central idea is to use inquiry questions as prompts to "annotate" entities in texts, based on a BERT model pre-trained on large scale question answering (QA) datasets [Rajpurkar et al., 2018]. For example, to recognize PERSON entities in plain texts, we create a query: Who is the named person or family mentioned in texts? (derived from the definition of PERSON) and use it as a prompt to query a BERT model. Particularly, we create the following sequence:

1 We obtain the entity type definitions from the spacy project and use "other" as a prompt for words labeled as O.

and encode it to H ∈ RN×d with N being the sequence's length and d being BERT's dimension. Then we locate a PERSON entity by computing two vectors specifying the probabilities of the starting and ending positions:

where wstart ∈ Rd , bstart ∈ RN , wend ∈ Rd , and bend ∈ RN are mode parameters. We consider the identified entities, coupled with the texts, as new training data. Table 1 compares the prompt representations of different prompting methods.

3.3 Noise Reduction via An Uncertain-Guided Self-Training Strategy

Despite their effectiveness, the above methods can introduce noise because no hard constraints are imposed. We develop High Confidence a self-training framework [Scudder, 1965] combining uncertainty modeling [Gal and Ghahramani, 2016] for noise reduction. Let the labeled dataset be DL, and the automatically generated dataset by the two prompting strategies be DU . Our self-training framework repeats the following steps:

1. Train a NERmodel M on DL.
2. Use M to pseudo-annotate examples in DU .
3. Select a set of reliable examples from DU and move them into DL; repeat the above steps until convergence.

Defining a universally good criterion for selecting reliable examples in step 3) remains an open challenge. Here we devise an uncertainty-guided method [Gal and Ghahramani, 2016] with an exploration-exploration trade-off strategy.

Uncertainty-Guided Confidence Ranking. We leverage recent advances in uncertainty modeling [Gal and Ghahramani, 2016] to evaluate the reliability of an example. Assume ( ˆX, ˆ Y ) ∈ DU is an automatically generated example, with ˆX = [ ˆw1, ..., ˆwn] being the sentence and ˆ Y = [ˆl1, ..., ˆln] being the entity label sequence. We first pseudo-predict a label for each word using the current NER model, with dropout layers activated and perform K times in total. Assume Li = [˜ pi1 , ..., ˜ piK ] is the obtained set containing K predicted labels 2 for the ith word ˜wi. According to [Gal and Ghahramani, 2016], the variance of Li reflects the model's uncertainty on its prediction for ˆwi. Given this, we define a token-wise criterion measuring the current model's confidence that ˆwi matches ˆli:

where n(ˆli) is frequency of ˆli in Li . Based on this token-wise criterion, we then define a sentence-level criterion to assess the degree of match between ˆX and ˆ Y :

where n is the length of ˆX. A higher score implies the current model is more certain that ˆX and ˆ Y complement each other.

2 Due to the activated dropout layers, elements in Li may differ.

Exploration-Exploitation Trade-Off. In practice, always selecting the most certain examples may not be a good solution, because the certain-most examples are too easy to contribute additional information, and this usually slows down learning and results in sub-optimal performance (as shown in our experiments). To balance speed and efficiency, we design an exploration-exploiting trade-off strategy, by defining a selection weight for each example:

We sample out N examples based on the above weight at each training iteration - this method biases the selection process towards picking more certain samples for exploiting but also allows for picking less certain examples for exploration. In evaluations, we show this method outperforms both certainfirst and uncertainty-first strategies.

4 Experimental Setups

Datasets and Evaluations. We use three NER datasets for evaluation: CoNLL 2003 [Tjong Kim Sang and De Meulder, 2003], OntoNotes 5.0 [Hovy et al., 2006], and a realworld low-resource dataset, MaScip [Mysore et al., 2019]. Among them, CoNLL 2003 defines four coarse-grained entity types: PER (Person), LOC (Location), ORG (Organization), and MISC (Miscellaneous); OntoNotes 5.0 defines 18 finegrained types, including CARDINAL, MONEY, PRODUCT, and others; MaScip [Mysore et al., 2019] defines 21 entity types (e.g., Material, Number, Operation, Amount-Unit) involved in materials synthesis procedures. Table 2 gives statistics of the three datasets. As for evaluation, following [Ding et al., 2020], for each dataset we sample out 50, 150, and 500 sentences (at least one mention of each entity type is included) to create the small (S), medium (M), and large (L) training sets (we use F to indicate the full training set), and we use precision (P), recall (R), and F1 as evaluation metrics.

Implementations. In our approach, we use BERT-base cased version [Devlin et al., 2019] as the backbone considering NER is a case-sensitive task. In label-conditioned word replacement, we empirically set T = 10, meaning to expand the original labeled data set by 10 times. In our prompting with question answering method, we train a BERT model on SQuAD datasets [Rajpurkar et al., 2018] and sample out 100,000 unlabeled sentences from WikiPedia for new data generation (For example, on CoNLL 2003, the numbers of retrieved entities for each type are PER:5633, ORG:3978, LOC: 2342, MISC: 1197). In the uncertainty-guided selftraining method, we set the number of forward passes K to 10, chosen from [5, 10, 20] and select N = 200 (chosen from [50, 100, 200, 300]) examples at each iteration. We use the development set to tune the best iteration step. We evaluate the impact of our method on two typical NER models: (i) BiLSTM-CRF [Lample et al., 2016], which combines Long Short-TermMemory (LSTM) networks [Hochreiter and Schmidhuber, 1997] with a CRF tagger [Lafferty et al., 2001] for NER. We use GloVe embeddings [Pennington et al., 2014] and we set the batch size to 50 (chosen from [10, 20, 50, 100])

and the learning rate to 1e-2 (chosen from [1e-1, 1e-2, 1e-3, 1e-4]). (ii) BERT based entity tagger (to clear confusion, we denote it as Transformer). The batch size is set to 10 (chosen from [2, 5, 10, 20]), and the learning rate is set to 1e-5 (chosen from [5e-6, 1e-5, 5e-5, 1e-4]). We apply Adam [Kingma and Ba, 2015] for model optimization.

DAMethods for Comparison. We compare our approach with the following DA methods: 1) EDA [Wei and Zou, 2019], a DA method for text classification, which adopts synonymous word substitution, word insertion, and shuffling for data generation. [Ding et al., 2020] adapt it to low-resource NER. 2) BDA [Zhou et al., 2019], a DA method for text classification that directly uses BERT for lexical substitution, which does not consider label information. 3) CBT [Kumar et al., 2020], a DA method for text classification that leverages sentence-level labels as condition signals. We adapt this method for NER using entity labels (rather than label definitions) as condition signals. 4) FlipDA [Zhou et al., 2021], a DA method using a generative model and a classifier to generate label-flipped data. We denote our approach by PromptDA and use LC and QA to indicate label-conditioned word replacement and prompting with QA respectively. We use UC to indicate the uncertainty-guided self-training method.

5 Experimental Results

Table 3 summarizes the results on CoNLL 2003, OntoNotes 5.0, and MaSciP, using the small (S), medium (M), large (L), and full (F) settings. The effectiveness of our approach has been justified. For example, in the M setup with 150 training examples only, our approach (with BiLSTM-CRF architecture) scores 61.2%, 36.2%, and 71.4% in F1 on the three datasets, outperforming previous methods without DA by more than 40% in F1 and prior best DA methods by 2.0% in F1. Furthermore, we note that the two prompting strategies are complementary, and that their combination produces the best result. When we compare the two prompting strategies, we note label conditioned word replacement (LC) is more effective than prompting with question answering (QA), which could be due to the latter strategy introducing more noise.

Impact of Uncertainty-Guided Self-Training. Figure 3 compares our uncertainty-guided strategy (denoted by U) with easy-first (E) strategy, hard-first (H) strategy, and a method training on all generated data (ALL). The performance is based on the development sets of CoNLL 2003

and OntoNotes 5.0. From the results, our uncertainty-guided method outperforms other methods in terms of both convergence speed and the final F1. For example, on CoNLL 2003, it achieves the highest F1 with 8 to 14 training steps and outperforms other methods by at most 2.0% in F1. We observe interesting results by comparing the easy-first and hardfirst strategies: The easy-first strategy, which always selects the most certain examples for training, results in the slowest learning rate, maybe because the most certain cases are too easy to provide more knowledge for learning. However, the hard-first strategy demonstrates contradictory behaviors: it performs well on CoNLL 2003 but poorly on OntoNotes 5.0. Because the hard-first strategy always chooses the most uncertain data for training, it can quickly capture complementary patterns to the existing data for CoNLL 2003, which has fewer entity types and more clean generated data. On the other hand, it suffers from noise for OntoNotes 5.0, which has more entity types and more noisy generated examples.

6 Discussion

We perform in-depth studies to explore the reasons behind our approach's effectiveness. To simplify discussion, we choose the medium training set (M) setting.

Per-Type Performance. Table 4 shows performance breakdown on different entity types. Particularly, on CoNLL 2003, our method performs well on PER, ORG, and LOC, but MISC. The reason is that MISC's definition is relatively vague (i.e., "events, nationalities, products, or works of art") and does not have a consistent pattern, which degrades our method using entity type definition as prompt. On OnotoNotes 5.0, our method yields better performance for common types such as GPE and MONEY, but value types such as DATE and PERCENT; we observe a similar pattern on MsScip, with performance on Material and Brand being better than performance on value types such as Number and Amount-Unit. The underlying reason is that common entity types such as GPE and MONEY have more clear and expressive definitions than value types such as DATE ("times smaller than a day") and PERCENT ("percentage, including %'), and such expressive definitions provide more prior knowledge to instruct our method for data augmentation.

Results of Zero-Shot NER. Considering that our prompting with question answering strategy can directly generate new training data from unlabeled texts, we study whether it can be applied to the zero-shot scene. Figure 4 shows the type-wise F1 (each type is evaluated independently) of the BiLSTM+CRF model on OntoNotes 5.0, trained on the automatically generated data solely. Our method demonstrates very promising results, for example, achieving over 40% in F1 on PER, GPE, LAW, and DATE without any labeled data. By comparing different entities types, we show that our method benefits both common types and rare types (such as MONEY, PERCENTAGE, PRODUCT) as long as they have clear definitions, but on types having a vague definition such as EVENT.

Data Fidelity Check. Table 5 explores data fidelity check, by comparing the words generated by BDA, CBT, and our label conditioned prompting approach (LC). By exploring E1 and E2, we show our method generates more label-consistent words (with a ratio of 80% and 90%) than BDA and CBT. E3 and E4 are more interesting cases, where we deliberately design ambiguous contexts and give models different labels for word generation (For example, in E3, the [MASK] can be filled with either a PER or ORG entity). From the results, our approach outperforms other methods for generating words matching the given labels - particularly, in E4, even an incorrect LOC label is given, our method generate wrong words matching the label, justifying its ability to capture the word-label dependencies. Lastly, in Table 6, we show examples generated by our prompting with QA method. Our approach can yield ideal examples such as E1, E4, E5, E6, E7, but also flawed ones such as E2, E9, and E3, where "Shapiro and Clark" and "Colorado and San Francisco" should be separated as two entities and "someone who is disabled" is not an entity. These cases imply further improvement direction.

7 Conclusion

In this work, we propose a new data augmentation method for low-resource NER, by eliciting knowledge from BERT with

Type Generated Examples

two prompting mechanisms. Our method can produce more label-consistent data or new examples from unlabeled texts without considerable human intervention. We also devise an uncertainty-guided self-training method for noise reduction. The results on three datasets have justified our method's effectiveness. In the future, we would apply our method to other related extraction tasks such as relation extraction.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (No.62106016). It is also supported by Fundamental Research Funds for the Central Universities (No. 2021RC234), the National Key R&D Program of China (2019YFB1405200), and the Open Projects Program of National Laboratory of Pattern Recognition.