Towards Incremental NER Data Augmentation via Syntactic-aware Insertion Transformer

1 Introduction

Named entity recognition (NER), as an important fundamental technique in information extraction, is of great significance to many downstream tasks such as text summarization [Nallapati et al., 2016] and information retrieval [Banerjee et al., 2019]. Methods utilizing pre-trained language models (PLMs) have achieved outstanding performance [Peters et al., 2017; Souza et al., 2020]. However, such techniques primarily follow a supervised paradigm, and high performance primarily hinges on the size and quality of the training data. Actually, the major challenge of this technique lies in the manual annotation of the corpus, which requires domain knowledge and is time-consuming, ending in the absence of annotated data over various verticals.

∗Corresponding Author

Several methods have been proposed to alleviate the above challenge, such as meta-learning [Wu et al., 2020; de Lichy et al., 2021; Ma et al., 2022c], prompt learning [Ma et al., 2022b; Liu et al., 2022a; Chen et al., 2022], and data augmentation [KERAGHEL et al., 2020; Phan and Nguyen, 2022]. Among these, explicit data augmentation could be easily and seamlessly plugged into other few-shot NER approaches. Generally, the primary technical methods of NER data augmentation research can be grouped into two paradigms: editing [Zhang et al., 2015; Cai et al., 2020; Wei and Zou, 2019; Min et al., 2020] and generative [Sun and He, 2020; Yoo et al., 2019; Yu et al., 2018; Zhou et al., 2022] methods. Editing techniques create new data by modifying the raw text with word substitution [Zhang et al., 2015; Cai et al., 2020], random deletion [Wei and Zou, 2019], and order exchange [Min et al., 2020]. In this vein, the editing method might raise issues such as the diversity deficiency of synthetic samples, thus reducing the generalization power of NER models. For example, in the second case of Figure 1, when Jobs and Apple were substituted with Leo and Google respectively, the revised sentence was basically unchanged with the same context as the raw sentence.

In comparison with editing methods, the generative methods can generate samples with higher fluency and variety e.g., GANs [Sun and He, 2020], VAEs [Yoo et al., 2019], translation [Yu et al., 2018] and prompt learning models [Wang et al., 2022; Liu et al., 2022b]. Nevertheless, the performance of existing generative methods may suffer from two disadvantages.On the one hand, compared with the raw sentence, the resulting sequence may cause syntactic deviation. As a result, the boundary of the generated entity may change, and the original entity label may not correspond correctly to the generated entity. Consider the second case in Figure 1, where the entity Apple was incorrectly expanded into another entity words the Washington Apple Association with a wider entity boundary. On the other hand, the generative method with a prompt learning strategy encourages the assembly of entities to guide sentence generation but lacks specific hard constraints to ensure entity occurrence. However, we are unable to obtain the ground truth labels of the newly generated token in advance. As shown in the previous example, the entity Jobs was dismissed during generation and replaced by a non-entity word He.

To address these issues, we argue that the key of the NER data augmentation task lies in preserving the semantic diversity and syntactic consistency between created sentences and raw ones. In this paper, we propose SAINT (SyntacticAware InsertioN Transformer), a hard-constrained controlled text generative model that explicitly incorporates the syntactic feature into pre-trained language models (PLMs) and restricts the appearance of predefined entities in the generated sentence. In contrast to previous generative approaches, SAINT generates sentences while maintaining semantic diversity and syntactic relevance with respect to the raw data. Specifically, we first turn the raw sentence into a dependency tree and obtain its flattened form containing the dependency direction and distance. Inspired by a hard constraint text generation work [Zhang et al., 2020], the training dataset is constructed by randomly picking and masking key tokens and corresponding syntactic annotations in raw sentences. The processed outputs are then fed into SAINT with the flattened dependency structures and the token themselves as the supervised objectives for multi-task training. In the inference stage, we feed the predefined entities and keywords into SAINT, generating the context words and corresponding syntactic annotations incrementally. The final training sentence can be selected using a sophisticated dependency similarity comparison algorithm.

Experimental results on two benchmark datasets from Wikiann [Pan et al., 2017] and OntoNotes [Hovy et al., 2006] demonstrate the superior performance of our model over state-of-the-art NER data augmentation baselines, with improvements of 2.21% and 3.13% over the strongest baseline model in the best case, respectively. The effectiveness of introducing syntactic features for the low-resource NER data augmentation task is also verified. Moreover, further analysis reveals the ability of SAINT to retain the dependency consistency of the raw sentence through the sample generation process. To sum up, the major contributions of this paper are three-folds:

• We argue that ensuring semantic diversity and syntactic consistency with the raw data is crucial for NER data augmentation. Based on this consideration, we construct the training dataset with both tokens and corresponding syntactic labels under the few shot settings.
• We propose a novel NER data augmentation model, SAINT, which explicitly incorporates syntactic features into the insertion transformer to generate samples containing preset entities parallelly.
• Extensive experiments are conducted on the Wikiann and OntoNotes datasets to verify the effectiveness of our method and demonstrate the significance of introducing syntactic features.

2 Method

Given a sentence xi from the downstream NER corpus X = {x0, x1, ..., xn}, the ultimate goal of SAINT is to produce a set of new sentences yi = {y0 i , y1 i , ..., ym i } for each input xi, where n denotes the size of original corpus, and m is the number of generation scale. The generative data yi should be grammatically correct and semantically consistent with the raw sentence xi. The total of yi (0 ≤ i ≤ n) together makes up the generated dataset Y = {y0, y1, ..., yn} that is further used in augmentation.

The overall architecture of SAINT is shown in Figure 2. In the first stage, SAINT constructs the training samples si for each data xi by utilizing the flattened form of the dependency tree for label syntactic features and picking and masking tokens dynamically. In the second stage, the syntactic feature is injected into the pre-trained language model, and the hard constraint text generation paradigm restricts the appearance of predefined entities in the generated sentence. The multi-task learning method could supervise the model to predict tokens and corresponding dependency labels. In the last stage, SAINT incrementally generates the sentence in a token-by-token manner. Moreover, the syntactic similarity measurement helps select the most syntactic-relevant sentence as the final result.

2.1 Training Sample Creation

The first stage of our method is to create samples for training. Due to the difficulty of directly constructing a stepwise mask loss function, the model is trained with data instance pairs. The training data is organized into a sequence of (rawtoken, rawsyntax, predicttoken, predictsyntax) pairs after progressive masking. Specifically, for each sample in the raw dataset, its syntactic dependency labels can be obtained by syntactic feature labeling, and tokens are masked from stage k to stage k +1 according to the importance score by the token selector. Consequencely, every interval pair (ski , sk+1 i ) comprises the training data. For example, sample {(studies, Park, College), (-0obj,+0compound,-0nmod)} and sample {(continued, studies, Park, College), (-0ROOT,- 0obj,+0compound,-0nmod)} form a data pair, as illustrated in Figure 2.

Syntactic Feature Labeling

As a syntactic dependency contains the information of the direction, distance, and relation type, we define syntactic dependency labels as follows:

Stage 1: Dataset Creation

Stage 2: Syntactic-aware Insertion Transformer Training

Stage 3: Data Generation

where (+|-), ℓ, and dep denote such three parts respectively. (+|-) designates the direction between the head and the dependent, where + denotes the position of the head in the sentence as coming before the dependent, and - is the opposite. ℓ indicates the distance between two tokens. Due to the extensive range of distance, it may result in a large number of labels. To avoid the label sparsity problem in the low-resource scenarios, we further process the value of distance to ℓ. We assign the value of ℓ to 0 when the distance is less than 5, and to 1 when the distance is equal to or greater than 5. dep shows the dependency relation type that the head belongs to.

To get the defined syntactic labels, we first apply the Stanford Parser 1 to obtain the syntactic dependencies of each sample. For the triplets (tail, type, head) in syntactic dependencies, we use the absolute value of relation head to relation tail as the distance and connect them to form the relation labels. For the case where a word corresponds to multiple labels, we only retain the labels of the predefined important syntactic component. Secondly, the formed label is then matched to the original word segmentation result to form syntactic dependency labels. Finally, we can obtain the syntactic dependency labels with respect to each sample. As shown in Figure 2, under the above process, the sample {She continued her studies ...} can be reflect to its corresponding syntactic label {+0nsubj, -0ROOT, +0nmod:pass, -0obj ...}.

Token Selector

To assess the significance of the tokens in the sentence, we consider the four different token scoring schemes. The score indicates the importance of tokens for different stage masks. Concretely, the importance score of the token xi in stage Xk is defined as:

where ptf-idf i , ppos i , pyake i , and pdep i represent the scores of TF-IDF, POS tagging, YAKE [Campos et al., 2020], and syntactic feature, respectively, and each item is adjusted to range 0 to 1. We increase the score of important syntactic components to highlight their contribution and keep the scores of other syntactic components unchanged. Throughout the construction of the dataset, the less important tokens are selected at an earlier stage, and then feed into the dynamic data construction module to determine whether to mask.

1 https://stanfordnlp.github.io/CoreNLP/

Dynamic Data Construction

Since SAINT does not insert multiple tokens between two adjacent tokens under the step-by-step data generation process, there should be no continuous masking of two tokens during the construction of training data. Based on such consideration, we calculate pi of each token in each step and choose a set of tokens with the highest total score. Formally, this problem can be defined as:

Suppose the current sequence is ski = {w0,w1, · · ·,wn}, where αi denotes whether wi is masked or not. αi ∈ {0, 1}, αi = 0 signifies keeping the corresponding token wi, while αi = 1 implies masking wi. pmax = max{pi}. Using pmax to subtract the score of each token to ensure that all the scores calculated by the algorithm are positive. Then we can conduct a dynamic programming [Gries, 1982; Zhang et al., 2020] to select tokens that should be masked.

2.2 Syntactic-aware Insertion Transformer Training

In this section, we introduce the training process of syntacticaware insertion transformer using the training dataset obtained in the previous phrase. Specifically, rawtoken is fed into the model, while syntactic feature rawsyntax is also injected simultaneously. Moreover, the model is supervised to predict token predicttoken and corresponding dependency labels predictsyntax.

Syntactic Feature Injection

In the training stage, we combine both embeddings of the syntactic dependency labels rawsyntax and word rawtoken. As shown in Equation (4) :

where eword i ∈ Rdw and esyntax i ∈ Rdw are low-dimensional and dense vectors initialized by word embedding and syntax embedding, respectively. ei ∈ Rdw serves as the representation of token wi. dw = 768 denotes the dimension of the word vector. λs is a hyper parameter manually specified to balance the weight of syntax and semantic.

Multi-task Learning

Given all the input sample pairs (ski , sk+1 i ) for the kth pair of ith sample, Our training process consists of two tasks: pretrained language model fine-tuning and syntactic dependency label prediction. The first item refers to calculating the loss of the language model Lℓm as:

Moreover, Ldep indicates the cross-entropy loss towards the syntactic dependency labels, which can be calculated by Equation (6) :

where vi and ˜ vi denote the ground truth and predicted syntactic dependency labels, respectively.

In the training phase, teacher-forcing and multi-task learning strategies are employed. And the Dynamic Weight Averaging (DWA) [Liu et al., 2019] is utilized to dynamically balance the two tasks and obtain weights of the PLM fine-tuning task and the syntactic dependency prediction task as λℓm and λdep. In summary, the total loss of the syntactic-aware insertion transformer can be computed as:

where λℓm and λdep are both in the range 0 to 1, which are allocated automatically by DWA mechanism.

2.3 Data Generation

The data generation process started with the given keyword sequence rawtoken, and the trained SAINT is used to generate candidate insertion words incrementally. In addition, with a large amount of generated data, we use Bayesian risk minimization to select the sample that has similar syntactic features to the raw sentence.

Incremental Generation

Since each step of the generation process generates tokens simultaneously, tokens inserted into different slots may be unaware of others, resulting in inconsistent expressions or duplicate content. We employ a beam search algorithm with Minimum Bayes Risk (MBR) [Kumar and Byrne, 2004] to select label sequence alternatives at each time step, limiting the number of alternatives to the beam width B.

Suppose the existing token sequence as sk = {sk1 , sk2 , · · ·, sklk } and the sequence length as lk. By executing an insertion procedure, it will predict the top B candidate tags in each slot, and the generated candidate sequence is Ck = {c(b) 1 , c(b) 2 , · · ·, c(b) i }, where c(b) j , j ⊂ {1, · · ·, i}, b ⊂ {1, · · ·,B} is one of the top B candidate tokens in the current j th slot. During the insertion of candidate words in the next slot, the model generates the most likely top B candidate token sequences. By iteratively executing the above steps, the top B generated sequences with the largest likelihood are reserved as the final candidate set S.

Dependency Similarity Comparison

For the top B generated candidate sentences, some of the generated syntactic labels may not align with the actual labels. We argue that generated sentences can be more in line with raw data when the generated syntactic labels are highly consistent with the real ones. To this end, we perform a dependency similarity comparison method, and select the most syntactic-relevant result. Particularly, for each sentence si in the candidate set S, the generated syntactic label sequence is gsi = {ge1, ge2, · · ·, geim}, the actual syntactic label sequence obtained by the Stanford Parser is asi = {ae1, ae2, · · ·, aeim}. The dependency similarity score of sentence si can be obtained as:

where the function sim( geim, aeim ) denotes to calculate the similarity between geim and aeim with fuzzy comparison. scorei = 1 if the syntactic distance between labels is less than 2 and they have the same dependency relation type, and scorei = 0 otherwise. Finally, we select the sentence sbest with the highest dependency similarity score in the candidate set S as the final generation result.

3 Experiments

In this section, we introduce the experimental settings and baseline methods and report the results of the experiments.

3.1 Datasets and Experimental Settings

We conduct experiments on the two NER datasets of Wikiann [Pan et al., 2017] and OntoNotes [Hovy et al., 2006]. TheWikiann dataset includes three types of entities: location, organization, and person. The OntoNotes dataset contains 18 types of entities, e.g., cardinal, date, event, law, and language. In the experiment, we sampled 100, 200, 300, 400, and 500 items of data as training data under low resource constraints.

Our method is implemented with Python 3.7.12, PyTorch 1.8.0, and the NVIDIA Tesla V100 16G platform. In the experiment, SAINT utilized the Bert-base model as the pretrained language model and adopted the AdamW optimizer with a learning rate of 1e-5. The training batch size is 20, and the model is trained for 5 epochs. The inference process is performed in three steps. The parameter λs is adjusted to 0.2. The NER model's optimizer in the experiment is the AdamW optimizer, the learning rate is 2e-5, and the batch size is 8. The model with the best validation set performance in 20 epochs is chosen as the final model for testing.

In order to fairly evaluate the performance of each method, we use the XML-RoBERTa-Large [Conneau et al., 2020] with CRF [Lample et al., 2016] as the unified NER model. Moreover, we test three times and metric the average MicroF1 value as the final result.

3.2 Baselines

We choose seven representative editing and generative NER data augmentation approaches as baselines for comparison.

Editing Methods

Label-wise token replacement (LwTR) leverages the binomial distribution to replace tokens with the same label in the training set. Synonym replacement (SR) uses the same token selection process as LwTR and replaces the token with its synonyms from WordNet. Mention replacement (MR) replaces entities by sampling the entities with the same labels in the training set. Shuffle within segments (SiS) cuts the sentence into several segments and replaces the segment with the same label.

Generative Methods

DAGA [Ding et al., 2020] formulates data augmentation as the sequence tagging task, where the labeled sentences are first linearized. A language model (LM) is then finetuned using such linearized data to generate synthetic samples. MulDA [Liu et al., 2021] extends [Ding et al., 2020] to low-resource cross-lingual NER, and introduces the process of filtering out the low-quality data. PromDA [Wang et al., 2022] first trains the soft prompt using the frozen PLM, then generates synthetic data by deleting useless samples with the help of NLU models.

3.3 Main Results

The experimental results on the two datasets are shown in Table 1, where no augmentation implies merely splicing all preset entities together without any other operations.

First, the editing method can typically outperform the non-augmentation manner, with the F1 value of SiS on the 200-sample Wikiann dataset achieving an absolute gain of 4.83%. Furthermore, as the number of samples rises, the performance improvement of these methods gradually slows down. Notably, MR is inferior to no augmentation under the 300, 400, and 500 sample settings on Wikiann. This disadvantage might be due to the low diversity of contexts within the samples obtained through a straightforward replacement operation.

Secondly, comparing the generative and editing methods reveals that the former maintains an absolute lead. An obvious observation is that regarding the 500 samples in the OntoNotes dataset, DAGA, MulDA, PromDA, and our method SAINT increase the F1 value by 9.21%, 10.49%, 12.39% and 14.58% over the best editing method SR, respectively. Besides, PromDA employs the natural language generation (NLG) technique rather than natural language understanding (NLU), achieving promising results on both datasets thanks to the prompt learning strategy.

Finally, our approach SAINT performs better than the other baseline models in most cases. Regarding the 500 samples of the OntoNote dataset, our approach outperforms no augmentation and PromDA in terms of F1 values by 17.98% and 2.19%, respectively. Regarding the 500 samples of the Wikiann dataset, such improvements are 14.23% and 3.13%. Even though SAINT is slightly inferior to PromDA on 100 and 200 samples of the OntoNote dataset, comparable results are also achieved. Note that our method can be better adapted to generate more new samples. On the one hand, it demonstrates the strong generalization ability of our method. On the other hand, such abilities can help resolve complicated lowresource scenarios in reality.

3.4 Ablation Study

The motivation for our approach is to preserve the semantics within the raw sample while ensuring the label correctness of the generated data with the help of syntactic features. To verify the effectiveness of our method, we conduct ablation experiments and show the results in Table 2.

As can be seen, the performance of SAINT on both data sets is severely affected by the removal of the syntactic injection module, with the maximum decline of 3.99% and 4.26%, respectively. Further observation reveals that, with respect to the multi-task learning scheme, the F1 score of SAINT decreases maximally by 2.97% and 4.34% when omitting the language model loss and syntactic loss, respectively. By contrast, the syntactic loss serves as a more significant effect on model performance. Incidentally, analyzing the difference in results between the two datasets, the removal operations appear to have a greater impact on Wikiann than OntoNotes. The major reason is that sentences of Wikiann are more concise and grammatically correct, encouraging syntactic consistency is more critical.

3.5 Effect of Hyper Parameters

In this section, we conduct experiments with different hyper parameters to investigate the effectiveness of syntactic feature injection and multi-task learning. Each experiment utilizes 100 to 500 samples from the training set, with parameters ranging from 0.1 to 0.9. The result is shown in Figure 3.

Effect of Syntactic Feature Injection

We evaluate the influence of syntactic information injection by adjusting the value of λs controlling the weight of word embedding and syntax embedding in Equation (4), where the smaller λs denotes the more weight of syntax embedding. Figure 3(a) shows that under all datasets and sample number settings, the smaller λs (0.2-0.4) presents the optimal result, indicating that syntactical information is more critical than semantic information. More broadly, the best results can be obtained by injecting syntax and semantics simultaneously.

Effect of Multi-task Learning Losses

In order to verify the impact of losses on the experimental results, we adjust the weight λℓm of language model loss and set λdep = 1 - λℓm. Figure 3(b) demonstrates that the Micro F1 value increases noticeably at first, then starts to decline. SAINT performs best when the weights of the two losses are nearly equal. Actually, SAINT concentrates on the

(a) Performance under different syntactic feature injection weights.

Figure 3: Effect of different hyper parameters.

fluency of sentences when λℓm is large, otherwise, SAINT trends to keep the dependency consistency of sentences. Consequently, SAINT performs better when the relative importance of two tasks is more evenly distributed. Moreover, the trends of the performance with weight λℓm are roughly the same on both datasets.

3.6 Case Study

In this section, two case studies, i.e., comparing the data quality of each baseline and visualizing the dependency consistency of SAINT, are conducted to help further understand our method.

Comparison of Data Generation

Figure 4 lists the data generated by each method that contains the entity Bill Gates. It can be seen that some tokens in the sentences generated by DAGA and MulDA may be unk, primarily because these models build their vocabulary using training data, ignoring the special tokens. Furthermore, the sentences generated by DAGA and MulDA are not fluent because they only use a small amount of data to train the language model and do not consider the sentence's grammatical information. PromDA is capable of producing coherent sentences, but the generated tokens are uncontrollable. As a result, Bill Gates Foundation in the generated result deviates from the semantics of the raw entity Bill Gates. In contrast, on the one hand, SAINT takes syntactic dependency features into account during data generation to ensure the fluency of data generation. On the other hand, SAINT's insertion transformer can help assemble the preset entity list, generating domain-specific sentences. For the sentences generated by SAINT, the entities Bill Gates, Apple, and Toyota Motor are correctly preserved, and the semantics and domain are related to the original sentence.

Analysis of Dependency Consistency

Figure 5 presents two dependency trees corresponding to the raw and generated sentences by SAINT. Obviously, the generated sentences maintain a similar syntactic structure to the raw ones. The first observation is that the dependency path between the token Eton and College are the completely same. Moreover, the dependency paths (-nsubj, obl, compound) and (-nsubj, advmod, conj, compound) between the subject and the entity Eton are very similar. In conclusion, our method can maintain the raw syntactic dependency, thus encouraging label consistency when generating samples.

4 RelatedWork

In this section, we first introduce related research on NER data augmentation, and then focus on controlled text generation which is employed in our work.

4.1 NERData Augmentation

NER data augmentation can be summarized into two frameworks: editing and generative methods. For the editing methods, Dai et al. and Sabty et al. utilized synonym replacement techniques [Dai and Adel, 2020; Sabty et al., 2021]. Although these methods are simple and easy to understand, the context of augmented samples could be too repetitive. As an alternative, the generative approach uses PLMs to generate samples. Concretely, Chen et al. designed a neural network to learn the pattern of text features from raw samples and output synthetic data by transferring such features. Zhou et al. introduced an auxiliary classifier into the major model to create data with reversed labels. Liu et al. applied a back-translation strategy to address cross-lingual NER. Wang et al. and Liu et al. instructed the PLMs to create samples via prompt learning [Wang et al., 2022; Liu et al., 2022b].

Our method boosts the performance of the NER model with a combination of the insertion-based generation paradigm and the injection of syntactic features to generate more fluent and semantic sentences with better label consistency.

4.2 Controlled Text Generation

Controlled text generation can be divided into autoregressive and non-autoregressive models. For autoregressive generation models, Ma et al. developed a simple attention module for GPT-2 [Radford et al., 2019] to strengthen the weight of specific words. Iso produced sentences by an auto-mined template with lexical occurrence limitation. Zhang et al. and Carlsson et al. exploited the novel regulated prompt framework respectively [Zhang and Song, 2022; Carlsson et al., 2022], guiding a causal language model to generate samples for maximum effectiveness. The non-autoregressive model offered some fresh insights into the task. Zhang et al. improved the structure of transformer to recursively conduct the insertion process in a parallel manner, thus accelerating the inference process considerably.

Figure 5: Dependency trees of the raw and generated sentences. The node marked with red color denotes the organization entity, while the same dependency relations are marked with the same color.

The proposed SAINT model explicitly incorporates syntactic features into PLMs and employs a hard constraint generation paradigm to generate text.

5 Conclusion

In this paper, we analyze the main challenges of the lowresource NER data augmentation task and present SAINT, a novel controlled text generation model for NER data augmentation. The proposed method leverages a hard-constrained manner to generate new sentences with lexical and syntactic constraints, which ensures the generated data is consistent with source data both in semantic and syntactic feature patterns. According to the results of experimental results, SAINT can generate more syntactic sentences with fewer noises and improve the performance of the NER model in cases of insufficient annotated data, which exceeds that of the SOTA baseline.