Learning Task-Specific Representation for Novel Words in Sequence Labeling

1 Introduction

Word representation (or embedding) is a foundational aspect in many state-of-the-art sequence labeling systems [Ma and Hovy, 2016; Zhang and Yang, 2018]. However, natural language yields a Zipfian distribution [Zipf, 1949] over words. This means that a significant number of words (in the long tail) are rare. To control model size, a sequence labeling system often constrains the training vocabulary to only cover the top N frequent words within the training set. Those words not covered by the training vocabulary are called OOV words. Learning representations for OOV words is challenged since the standard end-to-end supervised learning methods require multiple occurrences of each word for better generalization. Many works [Ma and Hovy, 2016; Madhyastha et al., 2016] have proved that performance on sequence labeling usually drops a lot when encountering OOV words. This is commonly referred to as the OOV problem, which we are to address in this work.

Over the last few years, many methods have been proposed to deal with the OOV problem. These approaches can be roughly divided into two categories: (1) pretraining word representations on very large corpora of raw text [Mikolov et al., 2013; Pennington et al., 2014; Peters et al., 2018]; (2) further exploiting training data of the task. In a datarich domain, the first category of methods can often bring considerable improvement to the models that are trained with random initialized word vectors [Devlin et al., 2018]. However, the approach can be criticized when encounter the extremely data-hungry situation. Obtaining sufficient data may be difficult for low-resource domains, e.g. in technical domains and bio/medical domains [Del˙eger et al., 2016].

Therefore, in this work, we highlight methods of the second category. A popular practice of this category is to represent all OOV words with a single shared embedding, which is trained on low-frequency words within the training set, and then assigned to all OOV words at testing time. However, this essentially heuristic solution is inelegant, as it conflates many words thus losing specific information of the OOV words. Another popular practice is to obtain the word representation from its surface-form (e.g., character sequence) [Ling et al., 2015]. This practice is successful at capturing the semantics of morphological derivations (e.g. "running" from "run") but puts significant pressure on the encoder to capture semantic distinctions amongst syntactically similar but semantically unrelated words (e.g. "run" vs. "rung"). Therefore, most state-of-the-art sequence labeling systems will combine the surface-form representation with an unique embedding to represent the word. This again introduces the OOV problem to the systems.

Recently, a new paradigm of the second direction, which we refer to as the teacher-student paradigm, are being studied. Methods of this paradigm address the OOV problem in two steps. In the first step, they train a supervised model (also called the teacher network in this work) to perform label prediction for those within-vocabulary words to obtain their task-specific word representations. In the second step, they train (or heuristically construct) a predicting model (also called the student network) to predict the representation of a word from its surface-form [Pinter et al., 2017], context [Lazaridou et al., 2017], or the both [Schick and Sch¨utze, 2018]. The training object of the student network is usually to reconstruct representations of those within-vocabulary words. At testing time, when encountering OOV words within a sentence, they first use the student network to predict representations for those OOV words. Then, based on the generated OOV representations, they use the teacher network to perform label prediction.

Intrinsically, methods of the teacher-student paradigm can be seen as pipelines, with the teacher and student networks being two cascaded components of the pipeline. Though methods of this paradigm has achieved notable success on several tasks, they suffer from the typical error propagation problem of the pipeline paradigm [Caselli et al., 2015; Bojarski et al., 2016], since the auxiliary reconstruction object used for training the student network is not guaranteed to be fully compatible with the supervised object used for training the teacher network.

In this work, we propose a novel method of the teacherstudent paradigm, which is specifically designed to address the error propagation problem. The main difference of this method from existing ones is the training of the student network. Instead of reconstructing representations of those within-vocabulary words, we train the student network to predict word representations that can achieve good performance on the supervised task. Training signal of the student network is directly backpropagated by the parameterfixed teacher network. This way, we connect learning of the student network with the teacher network, thus avoiding the error propagation from the prediction of word representation using the student network to the label prediction using the teacher network. We explore applicability of this method to eight sequence labeling tasks in part-of-speech tagging (POS) and named entity recognition (NER). Empirically studies on those tasks show that our proposed method can achieve better or comparative performance on OOV words over existing methods in this paradigm.

Contributions of this work can be summarized as follows: (i) We propose a novel method of the teacher-student paradigm to address the OOV problem in sequence labeling using only training data of the task. It can avoid the error propagation problem suffered by existing approaches in the same paradigm. (ii) We performed experimental studies on eight part-of-speech tagging and named entity recognition tasks, and achieved better or comparative performance on OOV words over several existing state-of-the-art methods. Source code of this work is available at https:// github.com/v-mipeng/TaskOOV.

2 Typical Methods of the Teacher-Student Framework

In this section, we highlight some typical methods of the teacher-student framework. The first representative work of this framework was proposed by Lazaridou et al., [2017]. In that work, they proposed to obtain the representation for an OOV word w through summation over representations of within-vocabulary words occurring in its contexts C(w):

Here, C(w) denotes the set of contexts that contain w, V is the training vocabulary, c(w) = ∑C∈C(w) |C∩V| is the total number of within-vocabulary words in C(w), and ew(·) is the embedding function defined on within-vocabulary words.

Following this idea, Khodak et al., [2018] further applied a linear transformation A to the resulting embedding:

to get the representation of w. To determine the form of A, they trained it on those within-vocabulary words with the training object being to minimize the reconstruction error of an randomly selected within-vocabulary word w:

where d(·, ·) defines the distance between two embeddings, e.g., Euclidean distance function.

In this line, Schick and Schutze, [2018] proposed to model both the context and surface-form (subword n-grams) of OOV words to get their representations:

where vc(w) is a word representation modeling from the surface of w, and α ∈ [0, 1] is a single learnable parameter or a gate generated from ˆ vw(w) and vc(w) as illustrated in §3.3.

In general, these methods differ from our proposed method in two points. First, they represent an OOV word w in different contexts with a consistent representation, while we relax the representation of the same word to be different in different contexts. Second, they train the student network on the auxiliary reconstruction object, while we directly train it on the supervised object used for training the teacher network. Of course, the first difference can be easily addressed by adjusting Eq. (1) to:

where C(w) denotes a single context of w. According to our experience, this adjustment can generally improve performance of the method in sequence labeling. Therefore, we applied this adjustment to all compared methods of the teacher-student paradigm. Thus, the main difference between our proposed method and existing works is the training strategy of the student network.

3 Methodology

Generally speaking, the method contains a teacher and a student network. The teacher network is first trained to perform the supervised task. Then, the student network (i.e., the representation predict layer) is trained to generate appropriate representations for words from their contexts and surface-forms. The quality of the generated word representation v(w) is measured by the parameter-fixed teacher network (the copied sequence modeling and CRF loss layers), i.e., the training signal of the student network is backpropagated by the teacher network. At testing time, it first uses the student network to generate representations for OOV words and then uses the teacher network to perform label prediction.

3.1 Notation

Throughout this work, we denote Ω the word space and V the training vocabulary. A sentence consisting of a sequence of words is denoted as X = {w1, · · · ,wn} ∈ Ω and its corresponding label sequence is denoted as Y = {y1, · · · , yn} ∈ Y. The context of a word wi given its corresponding sentence X is denoted as X\i = {w1, · · · ,wi-1,wi+1, · · · ,wn}, and X<i,j> refers to the sub-sequence {wi, · · · ,wj}. The teacher network is denoted as T and the student network is denoted as S. In addition, let f be a function defined on Ω, we denote f(X) a shorthand of {f(w1), · · · , f(wn)}.

3.2 Train the Teacher Network

The teacher network is used to perform label prediction in the sequence labeling system. It can be of arbitrary architecture suitable for the task. Because the main focus of this work is to deal with the OOV problem instead of designing a superior supervised model, in this work, we tried the typical CNN-based and LSTM-based architectures [Huang et al., 2015] to implement the teacher network. Briefly, this architecture represents a word w with the concatenation of an unique dense vector (embedding) ew(w) and a vector ec(w) modeled from its character sequence using a character-level convolutional neural network (CNN) [Kim, 2014]:

where ⊕ denotes the concatenation operation. The corresponding sentence representation e(X) is fed as input into a bidirectional long-short term memory network (BiLSTM) [Hochreiter and Schmidhuber, 1997] or a three-layer CNN network [Kim, 2014] with kernel size set to 3 to model context information of each word, obtaining a hidden representation (not the word embedding) h(wi|X) for each word given X. On top of the BiLSTM-based or CNN-based sequence modeling layer, it uses a sequential conditional random field (CRF) [Lafferty et al., 2001] to jointly decode labels for the whole sentence:

where φt(y′ , y|X) = exp(wT y′ ,y h(wt) + by′ ,y), wy′ ,y and by′ ,y are trainable parameters corresponding to label pair (y′ , y), and θT denotes the whole parameters of the teacher network. The training loss of the teacher network is then defined by:

where N is the training sentence number. This network is trained on all words occurring in the training set. After that, we fix its parameters during the training of the student network described in the following.

3.3 Train the Student Network

The student network models both surface-form and context information of a word for generating its representation. These two information resources have been demonstrated to be complementary to each other by Schick and Sch¨utze, 2018.

Model Surface-form

The surface-form representation of a word w is obtained from its character sequence w = {c1, c2, · · · , cn} ∈ Vc, where Vc is the character vocabulary. Specifically, following the work of [Schick and Sch¨utze, 2018], we first pad the character sequence with special start and end characters c0 = 〈s〉, cn+1 = 〈e〉 and obtain its up k-gram set:

Then, we define the surface-form embedding of w to be the average of all its up k-gram embeddings:

where Wngram denotes an embedding lookup table for ngrams, which are trainable parameters of the student network.

Model Context

For modelling the context of word wi, we apply a bidirectional long-short termmemory network (BiLSTM) on its context word sequence X\i. Here, we consider application of this practice in two different situations. In the first situation, there is only one OOV word within a sentence X. While in the second situation, there are multiple OOV words within X.

When there is only one OOV word wi within a sentence X, the task-specific representations for the rest words are all known, and we can directly apply BiLSTM to X\i to predict representation for wi. Specifically, we use the forward LSTM to model the sequence from the beginning of X to wi-1:

and use the backward LSTM to model the sequence from the end of X to wi+1 in a reverse order:

The forward and backward hidden representations are concatenated to form the context representation of wi given X:

where ⊕ denotes the concatenation operation.

When there are multiple OOV words within a sentence, for a given OOV word wi ∈ X, the representations of some other words in X\i are also unknown. Therefore, we cannot directly apply the BiLSTM to X\i to get its predicted representation. To address this problem, we propose to iteratively predict representations for the multiple OOV words within a sentence. Let vt context (wi|X) denote the predicted representation of wi given X at the tth iteration with v0 context (wi|X) = 0. At the (t + 1)th iteration, for predicting the representation of wi, we apply the BiLSTM to its tth iteration context vt context (X\i), obtaining vt+1 context(wi|X). This process repeats for all OOV words to finish the (t + 1)th iteration. The iteration proceeds till t reaches a fix value K, obtaining the final predicted representation for each OOV word. According to our experience, it is appropriate to set K = 2.

Combine Surface-form and Context

We finally combine representations of the surface-form and the context to obtain a joint representation v(w|X) of w. In this work, we follow the idea of [Schick and Sch¨utze, 2018] and combine these two representations with a gate:

where

Here, σ denotes the sigmoid function. For expression consistency, we denote

where θS are those trainable parameters of the student network.

Training

Because we use the teacher network to perform label prediction, training object of the student network should be learning to generate representations for OOV words that are suitable for the teacher network. According to the idea of previous works [Lazaridou et al., 2017; Khodak et al., 2018; Schick and Sch¨utze, 2018], we may train the student network to minimize the reconstruction errors of those within-vocabulary words with loss function defined by:

Here, d(·, ·) defines a distance between two embeddings, e.g., Euclidean Distance function. The biggest problem of this practice is that it may suffer from the error propagation problem, since the auxiliary reconstruction training criteria Lrecon is not guaranteed to be compatible with training object of the teacher network.

In this work, we propose to directly connect training of the student network with the training object of the teacher network, sidestepping designing an appropriate auxiliary training criteria. Without loss of generality, we first consider training of the student network in the situation that there is only one OOV word within a given sentence. To simulate this situation, for a training sentence X and its corresponding label sequence Y, we randomly sample a word wi from X. The resultant pair (wi,X\i,Y) forms a training example of the student network. By replacing the ith column of e(X) with v(w|X; θS) = [vw(w|X; θS) ⊕ ec(w)], we obtain the input v(X) = {e(w1), · · · , v(wi|X; θS), · · · , e(wn)} of the teacher network. Based on v(X), we obtain new hidden representation for each word ˆh(wi|X) and the task loss is then defined by:

where, this time, φt(y′ , y|X) = exp(wT y′ ,y ˆh(wt) + by′ ,y). Note that the training loss of the student network is the same as that of the teacher network, making sure that their training

Algorithm 1 Training of the student network

• 1: Input: the teacher network T, training dataset D
• 2: Result: the student network S
• 3: while S does not converge do
• 4: sample X = {w1, · · · ,wn} and its corresponding Y = {y1, · · · , yn} fromD
• 5: for i ∈ [1, · · · , n] do
• 6: Generate v(wi; X) for wi ∈ X;

7:

8:

v(X) = {e(w1), · · · , v(wi|X), · · · , e(wn)};

Get LS(X, Y; θS, θT ) based on v(X) according

to Eq. (18);

is compatible. For training the student network, we perform parameter update by:

During the training of the student network, parameters of the teacher network θT is fixed. Algorithm 1 shows the general training process of the student network in the situation that there is only one OOV word within a sentence.

To extend this process to the multiple OOV word situation, we sample multiple words from X, e.g., wi and wj, generating an training example of the student network. The loss defined on this example is similar to that in the one OOV situation and minimized over θS.

4 Experiments

To evaluate the effectiveness of our proposed method, we performed experiments on four part-of-speech tagging (POS) tasks and four named entity recognition (NER) tasks. These tasks have varying OOV rates, which is defined by the percentage of testing words occurring less than five times in the training set. These tasks share the same architectures of the teacher and student network as illustrated in §3.

4.1 Datasets

POS: For POS, we conducted experiments on: (1) PTBEnglish: the Wall Street Journal portion of the English Penn Treebank dataset [Marcus et al., 1993], (2) RIT-English: a dataset created from Tweets in English [Derczynski et al., 2013], (3) GSD-Russian: the Russian Universal Dependencies Treebank annotated and converted by Google 1 , and (4) RRT-Romanian: the Romanian UD treebank (called RoRefTrees) [Verginica Barbu Mititelu, 2016]. For PTBEnglish, we followed the standard splits: sections 2-21 for training, section 22 for validation, and section 23 for testing. For RIT-English we followed the split protocol of Gui et al., [2017]. While, for UD-Russian and UD-Romanian, we used their given data splits.

NER: For NER, we performed experiments on: (1) CoNLL02-Spanish: the CoNLL2002 Spanish NER Shared Task dataset [Sang, 2002]; (2) CoNLL02-Dutch: the CoNLL2002 dataset of Dutch language; (3) Twitter-English: an English NER dataset created from Tweets [Zhang et al., 2018]; and (4) CoNLL03-German: the CoNLL2003 NER dataset in German [Tjong Kim Sang and De Meulder, 2003]. These datasets are annotated by four types: PER, LOC, ORG and MISC. For datasets except Twitter-English, we used the official split training set for model training, testa for validating and testb for testing. While for Twitter-English, we followed data splits of Zhang et al., [2018].

1 https://universaldependencies.org/

Table 1 reports the statistic results of the OOV problem on the development and testing sets of each dataset. From the table, we can see that the OOV rate varies a lot over different datasets.

4.2 ComparedMethods

• RandomUNK: This baseline refers to the teacher network trained on all words occurring in the training set. At testing time, it represents words not occurring in the training set with a consistent random vector.
• SingleUNK: This baseline trains the teacher network on words that occur no less than 5 times in the training set. The other infrequent words and those words not occurring in the training set are all mapped to a single trainable embedding eUNK, which is trained during model training.
• [Lazaridou et al., 2017]: This baseline uses RandomUNK as the teacher network. At testing time, it first represents OOV words with their context word representations by mean-pooling as defined in Eq. (5) based on RandomUNK and then performs label prediction using RandomUNK.
• [Khodak et al., 2018]: This baseline additionally trains a linear transformer A to transform the mean-pooled representation of [Lazaridou et al., 2017] as illustrated by Eq. (2) for predicting representations of OOV words. The training of A is performed on words that occur no less than 5 times in the training set.
• [Schick and Sch¨utze, 2018]: This baseline is a variant of the proposed method, but its student network is trained on the reconstruction object as defined in Eq. (17). It is also related to the work of [Schick and Sch¨utze, 2018] but using the same architecture of the proposed method to model word context and surface-form.
• [Akbik et al., 2018]: This baseline converts the input sentence into a character sequence. Then, it applies a character language model to the character sequence to get the representation of every word within the sentence. Based on the obtained word representations, it applies a LSTM network to model the word sequence and performs final tag recommendation. To make it compatible with the setting of this work, we did not pretrain the language model on external data and not use pre-trained static word embeddings for this baseline.

4.3 Implementation Detail

For data prepossessing, all digits were replaced with the special token "<NUM>", and all url were replaced with the special token "<URL>". Dimension of word embedding, character embedding, and LSTMwere respectively set to 50, 16, and 50 for both the teacher and student networks. Kernel size of the character CNN was set to 25 for kernel width 3 and 5. Optimization was performed using the Adam step rule [Kinga and Adam, 2015] with the learning rate set to 1e-3.

4.4 Evaluation

We partitioned the testing (or development) set into two subsets: within-vocabulary (WIV) words and out-of-vocabulary (OOV) words. A word is considered WIV if it occurs more than 5 times in the training set, otherwise OOV. For NER, an entity is considered being of OOV if it contains at least one word of OOV word set. We report model performance (accuracy for POS and F1 for NER) on OOV set. This is because we can exactly tell which subset a word belongs to, thus we can easily combine the best model on each subset to achieve the best overall performance on the whole testing set. For example, we can perform label prediction for OOV words using our proposed model, and perform label prediction for WIV words using the best performing model on the WIV subset.

4.5 Main Results

Table 2 and 3 reports model performance on the OOV set for POS-tagging and NER, respectively, when using BiLSTM and CNN to implement the teacher network. From this table, we have the following observations: (1) on most tasks, methods dealing with the OOV problem outperform the RandomUNK baseline. This verifies the necessity to deal with the OOV problem in sequence labeling. (2) the method [Schick and Sch¨utze, 2018] using both surface-form and context information to generate representations of OOV words outperforms the method [Khodak et al., 2018] using only context information on most datasets. This shows the complementary of surface-form and context information; (3) the most comparative baseline in the teacher-student paradigm [Schick and Sch¨utze, 2018] in general outperforms SingleUNK. This verifies the effectiveness of the motivation beneath the teacher-student paradigm; (4) our proposed method consistently outperforms [Schick and Sch¨utze, 2018], which differs from the proposed method in the training of the student network. This, on one hand, shows the existence of error propagation from the student network to the teacher network, and on the other hand approves the effectiveness of our proposed method for addressing OOV problem; finally (5) our proposed method consistently outperforms the character language model [Akbik et al., 2018]. A possible explanation of this result is that our method can use wordlevel embedding without suffering from the OOV problem while the character language model cannot.

5 Conclusion

In this work, we proposed a novel method to address the outof-vocabulary problem in sequence labeling systems using only training data of the task. It is designed to generate representations for OOV words from their surface-forms and contexts. Moreover, it is designed to avoid the error propagation problem suffered by existing methods in the same paradigm. Extensive experimental studies on POStagging (POS) and named entity recognition (NER) show that this method can achieve superior or comparable performance over existing methods on the OOV problem.

Acknowledgements

The authors wish to thank the anonymous reviewers for their helpful comments. This work was partially funded by China National Key R&D Program (No. 2018YFC0831105, 2017YFB1002104, 2018YFB1005104), National Natural Science Foundation of China (No. 61532011, 61751201), Shanghai Municipal Science and Technology Major Project (No.2018SHZDZX01), STCSM (No.16JC1420401,17JC1420200), ZJLab.