UniTrans : Unifying Model Transfer and Data Transfer for Cross-Lingual Named Entity Recognition with Unlabeled Data

1 Introduction

Named entity recognition (NER) is a fundamental task in natural language processing, which seeks to locate and classify named entities, like locations, organizations, etc., in unstructured texts. NER has been extensively studied, especially monolingual NER, as it is widely incorporated in various downstream tasks, e.g., question answering [Moll´a et al., 2006]. Recently, deep neural networks have become the dominant approach due to their superior performance. However, the key to their success is the availability of adequate labeled training data; and building large labeled training sets for a new language of interest can be time consuming and labor costly. This motivates researches on cross-lingual transfer, which leverages labeled data from a source language (e.g., English) to overcome the data scarcity issue in the target language. In this paper, following [Wu and Dredze, 2019] and

[Wu et al., 2020], we focus on zero-resource cross-lingual transfer, where NO labeled data is available in the target language. In this way, our work can act as a basis for further researches where some labeled target-language data exists.

The state-of-the-art methods for zero-resource crosslingual NER mainly fall into two categories: i) model transfer based methods [Wu and Dredze, 2019; Wu et al., 2020], which use labeled source-language data to train an NER model with language-independent features (e.g., crosslingual word representations [Devlin et al., 2019]), and then directly apply it to the target language; ii) data transfer based methods, which generally construct pseudo labeled data in the target language via translating the source-language training data into parallel target-language data and mapping the entity labels, and then use such pseudo labeled data to train a target-language NER model. For example, [Mayhew et al., 2017] and [Xie et al., 2018] employed word-to-word and phrase-to-phrase translation, respectively, to generate targetlanguage training data, so that entity labels in the sourcelanguage training data can be directly copied to the generated target-language data.

In this paper, we hold the idea that the aforementioned model transfer based methods and data transfer based methods are complementary to each other. Specifically, for the former, models trained with labeled source-language data only learn the knowledge w.r.t. the task in the source language, but cannot see any task-specific information in the target language. And thus they probably rely more on context information mined by language independent features to make predictions, as word-label relations in the target language are unavailable for them. As for the latter, though models have access to word-label pairs in the target language to exploit their relations, inaccurate translations caused by sense ambiguity and word order differences will probably weaken the models' capability to predict through context information. Moreover, both methods generally do not leverage the language information contained in the unlabeled target-language data to benefit the cross-lingual transfer.

Therefore, here we propose a novel approach termed UniTrans to unify both model transfer and data transfer for cross-lingual NER, and furthermore, to leverage the beneficial information from unlabeled target-language data via enhanced knowledge distillation. Specifically, following [Lample et al., 2018], we first construct a pseudo training set for the target language by performing word-to-word translation and copying entity labels. Then, we use a pre-trained cross-lingual language model, i.e., multilingual BERT [Devlin et al., 2019], to generate language-independent features to train two separate NER models. One is trained on the labeled source-language data, and the other is trained on the pseudo target-language training data. We then integrate the knowledge contained in the source-language training data, the pseudo target-language training data, and the unlabeled target-language data as follows: i) we fine-tune the model trained on source-language data with the pseudo target-language training data, and take the fine-tuned model as a teacher NER model to predict probability distributions of entity labels (i.e., soft labels) for each word in the unlabeled target-language data; ii) we propose a voting scheme that associates the three aforementioned NER models to predict high-confidence one-hot label vectors (i.e., hard labels) for part of words in the unlabeled target-language data; iii) we train a student NER model on the unlabeled target-language data with supervision from both the aforementioned soft labels and hard labels, as how knowledge distillation works, and we use it as the final target-language NER model.

Extensive experiments conducted on benchmark datasets for 4 target languages well demonstrate that the proposed UniTrans substantially outperforms the existing state-of-theart methods. We also extend UniTrans by ensembling multiple teacher models with different random seeds, and show further performance improvement.

Our major contributions are summarized as follows:

• We propose a novel approach termed UniTrans to unify model transfer and data transfer based on their complementarity for cross-lingual NER, with the help of beneficial information from unlabeled target-language texts.
• We propose a voting scheme to generate pseudo hard labels on unlabeled target-language data, so as to enhance knowledge distillation in UniTrans with supervision from both soft labels and pseudo hard labels.
• We conduct extensive experiments on benchmark datasets and show that UniTrans yields new state-of-theart cross-lingual NER performance, which can even be promoted via teacher ensembling.

2 RelatedWork

2.1 Cross-Lingual NER

Cross-lingual transfer for NER has attracted much attention in recent years. Prior works roughly fall into two primary categories: model transfer based and data transfer based.

Model transfer based methods generally utilize languageindependent features to train an NER model on the labeled source-language data, so that the model can be directly applied to the target language. Those language-independent features include aligned word representations [Ni et al., 2017; Wu and Dredze, 2019], word clusters [T¨ackstr¨om et al., 2012], Wikifier features [Tsai et al., 2016], and gazetteers [Zirikly and Hagiwara, 2015], etc. Besides directly applying the trained NER model, [Wu et al., 2020] further proposed a meta-learning algorithm to improve the prediction for each test case with its similar examples in the labeled source-language data.

Data-transfer based methods generally train a monolingual NER model for the target language with pseudo training data constructed from the labeled source-language data. [Ni et al., 2017] proposed to use bilingual parallel texts and their word alignment information to project labels from the source language to the target language. Considering it is expensive to build parallel corpora manually, most recent methods propose to translate the source-language texts into the target language in a word-by-word [Xie et al., 2018] or phraseby-phrase [Mayhew et al., 2017] manner, and then copy the label of each word/phrase to its corresponding translated word/phrase. [Jain et al., 2019] further proposed to utilize Google Translate twice to translate sentences as well as entities, and align entity labels based on distributional statistics derived from the dataset.

As mentioned above, both model transfer and data transfer can be complementary to each other. And thus in this paper, we propose UniTrans to unify both, so as to retain the capability of predicting through context information and meanwhile exploiting word-label relations in the target language. Additionally, few prior works on cross-lingual NER leveraged unlabeled data in the target language. [Bari et al., 2019] proposed to fine-tune the model trained on source-language data with unlabeled target-language data in a manner similar to self-training [Scudder, 1965]. As shown by [He and Sun, 2017] and [Bari et al., 2019] that unlabeled data is beneficial, our UniTrans also exploits the unlabeled target-language data via an enhanced knowledge distillation process, as mentioned before.

2.2 Knowledge Distillation

Knowledge distillation was originally proposed for model compression [Bucilu et al., 2006], i.e., to learn a compact student model that retains most performance of a larger teacher model or ensemble of models that require more space to deploy or more computation to make predictions [Rusu et al., 2015; Hinton et al., 2015; Sanh et al., 2019]. Besides model compression, researchers also applied knowledge distillation to various tasks, like deep reinforcement learning [Rusu et al., 2015], image classification [Hinton et al., 2015], and language modeling [Sanh et al., 2019].

In this paper, we adapt knowledge distillation to crosslingual NER. And different from typical application scenarios of knowledge distillation that do not consider unlabeled data, in our proposed UniTrans, the student model is completely trained on the unlabeled target-language data. In addition, we propose a voting scheme to generate pseudo hard labels for the unlabeled target-language data, thus enhancing knowledge distillation with supervision from both soft labels and pseudo hard labels.

3 Methodology

Figure 1 illustrates the framework of the proposed UniTrans. Specifically, we first generate pseudo training data for the target language by performing word-to-word translation and copying entity labels. Then, we train an NER model using the labeled source-language data and derive a teacher model by fine-tuning it with the pseudo target-language training data. Finally, we adapt knowledge distillation to train a student NER model for the target language on unlabelled data.

3.1 Data Transfer viaWord-to-Word Translation

Following [Xie et al., 2018], we apply [Lample et al., 2018] to translate the source-language training data word-by-word into the target language, and then directly copy the entity label of each source-language word to its corresponding translated word. The pipeline of data transfer is briefly introduced below. One can refer to [Lample et al., 2018] for more details.

We first leverage identical "character strings" [Smith et al., 2017] in both languages, i.e., shared words in most cases, to create a seed dictionary {si, ti}Di=1 , where si, ti ∈ Rd , respectively, denote the source-language word embedding and the target-language word embedding of an identical character string, and D is the size of the dictionary. Then, we learn a linear mapping P ∈ Rd×d between the source-language embedding matrix S = [s1, s2, ..., sD] ∈ Rd×D and the targetlanguage embedding matrix T = [t1, t2, ..., tD] ∈ Rd×D , with its objective function formulated as follows.

where ‖·‖F denotes the Frobenius norm. The closed-form solution for Eq. 1 can be derived via singular value decomposition (SVD):

To produce word-to-word translations, for a sourcelanguage word, we use the learned P to map its embedding vector into the target-language embedding space, and take its nearest neighbor in the target language as its corresponding translation result. Specifically, we use the cross-domain similarity local scaling (CSLS) [Lample et al., 2018] to measure the distance between the mapped embedding vector of a source-language word (denoted as Psi) and the embedding vector of a target-language word (denoted as tj):

where NT (P si) denotes the K target-language nearest neighbors of Psi, NS(tj) denotes the K mapped source-language nearest neighbors of tj , and cos(·) denotes cosine similarity.

By word-by-word translation and copying word labels, we can build a pseudo target-language training set Dtrans.

3.2 Base Model for NER

The base model for NER in this paper consists of a feature encoder and a linear classification layer. Given an input text sequence x = {xi}Ni=1 with N words, we first feed it into the feature encoder fθ to obtain feature vectors h = {hi}Ni=1 for all words:

where fθ can be any feature encoder that produces languageindependent features, and hi is the feature vector corresponding to the i-th word xi. Following [Wu and Dredze, 2019], here we utilize multilingual BERT [Devlin et al., 2019] as the language-independent feature encoder.

Then for each word xi, its corresponding feature vector hi is fed into the linear classification layer with the softmax function to predict the probability distribution of entity labels for it, which is formulated as follows.

where p(xi, Θ) ∈ R|C| with C being the entity label set, and Θ = {fθ,W, b} denotes all the to-be-learned parameters.

Suppose a labeled training set is denoted as D = {(x, y)}, where y = {yi}Ni=1 is the corresponding one-hot entity label vectors for words in the corresponding x. Then the loss function L for the NER model is defined as the cross entropy between the predicted probability distribution of entity labels and the ground-truth one for each word, as formulated below:

where yi is the one-hot label vector corresponding to xi.

Note that in this paper, we use the base model here to train all mentioned NER models.

3.3 UnifyingModel Transfer and Data Transfer

With the base model and loss function defined above, we can train an NER model on the labeled source-language data like previous model transfer based methods [Wu and Dredze, 2019], denoted as Θsrc. Then for unifying model transfer and data transfer based on their complementarity, the proposed UniTrans further trains a teacher model Θteach by finetuning Θsrc on the pseudo target-language training data, with the same loss function as Eq. 8. And thus Θteach is expected to combine the advantages of both model transfer and data transfer. Moreover, our experiments also show that, Θteach obtains superior performance than simply combining the labeled source-language data and the pseudo target-language training data to train an NER model.

3.4 Knowledge Distillation on Unlabeled Data

Using Soft Labels

For knowledge distillation, we first apply the teacher model Θteach to the unlabeled target-language data DT = {˜x}, where ˜x = {˜xi}Ni denotes an unlabeled target-language text with N words. Considering that predicted soft labels (i.e., probability distribution of entity labels) can contain richer information than predicted hard labels [Hinton et al., 2015], we use the soft labels output by Θteach as the supervision to train a student model Θstu, by minimizing the mean squared error (MSE) between the prediction of Θstu and that of Θteach for each word. The loss function w.r.t. an unlabelled targetlanguage text ˜x is formulated as:

where p(˜xi,Θstu) denotes the probability distribution of entity labels predicted by the student model for the i-th word ˜xi, and p(˜xi,Θteach) denotes that of the teacher model.

In the way above, we can not only transfer the knowledge learned by the teacher model Θteach to the student model Θstu, but also enable the student model Θstu to capture language specific information in unlabeled target-language data DT . And thus, with an identical base model, the student model Θstu is expected to gain performance improvement over the teacher model Θteach.

Incorporating Pseudo Hard Labels

[Hinton et al., 2015] pointed out that when correct hard labels are available, the student model trained with only soft labels can be further improved by also training it with supervision from the correct hard labels. However, in the case of zero-resource cross-lingual NER, there are no ground-truth entity labels available in DT . To this end, we propose a voting scheme to (partially) generate pseudo hard labels for the unlabeled target-language data. And we leverage them to help the training of the student model, via adding an extra loss to Eq. 9.

Specifically, we train another NER model in a supervised manner on the pseudo target-language training data generated by data transfer (section 3.1), denoted as Θtrans. Then, we predict the probability distributions of entity labels for each word ˜xi in any unlabeled text ˜x, using all learned NER models, i.e., Θsrc, Θteach and Θtrans, respectively. And we take the entity label c ∈ C with the highest probability as the predicted label ˆ y(∗) i for ˜xi:

where Θ∗ stands for Θsrc, Θteach and Θtrans, with the corresponding ˆ y(∗) i being ˆ y(src) i , ˆ y(teach) i and ˆ y(trans) i . And we only generate a pseudo hard label ˆ yi = ˆ y(teach) i for the word ˆxi when ˆ y(teach) i = ˆ y(src) i = ˆ y(trans) i , as ˆ yi would be of highconfidence then. Here ˆ yi can be seen as the voting result of Θsrc, Θteach and Θtrans. We denote the set of such words with a pseudo hard label as X :

With the pseudo hard labels derived, we further define a hard label based loss L˜xhard as follows for the unlabeled target-language text ˜x.

where I(˜xi) is an indicator function that returns 1 if ˜xi ∈ X and otherwise 0.

Then the loss function combining both the supervision from pseudo hard labels and that from the soft labels output by Θteach is formulated as follows to better guide the training of Θstu:

where η > 0 is a trade-off parameter, and we simply set η = 1 in this paper. With L(Θstu), the student model Θstu is learned in an end-to-end manner.

3.5 Inference in Target Language

For inference in the target language, we only use the learned student model Θstu to predict the probability distribution of entity labels for each word in a given test case, as Eq. 7. Note that here we do not straightly take the entity label with the highest probability as the prediction result for each word (Eq. 10). Instead, we apply Viterbi decoding [Chen et al., 2019] by just adding constraints to ensure that predicted entity labels for all words in the test case would not violate the NER tagging scheme and meanwhile would obtain the highest probability as a label sequence. And thus we don't need to train a transition matrix here.

4 Experiments

We evaluate the proposed UniTrans for zero-resource crosslingual NER through experiments over benchmark datasets on 4 target languages (i.e., Spanish, Dutch, German, and Norwegian 1 ) and comparisons with state-of-the-art methods.

1 We use Bokm˚al rather than Nynorsk here, considering that it is used by 85-90% of the population of Norway.

4.1 Experiment Settings

Datasets

We use the following widely-used benchmark datasets for experiments: CoNLL-2002 [Tjong Kim Sang, 2002] for Spanish [es] and Dutch [nl] NER, CoNLL-2003 [Tjong Kim Sang and De Meulder, 2003] for English [en] and German [de] NER, and NoDaLiDa-2019 [Johansen, 2019] for Norwegian [no] NER. All datasets are annotated with 4 entity types: LOC, MISC, ORG, and PER. Each dataset is split into training, dev, and test sets. Table 1 reports the statistics of each.

We leverage WordPiece [Wu et al., 2016] to tokenize each sentence into a sequence of subwords and, following [Wu and Dredze, 2019; Wu et al., 2020], we use the BIO entity labeling scheme. Moreover, as previous works [Wu et al., 2020], all experiments use English as the source language and the others as the target language.

Note that for each target language, we delete all entity labels in its training set, and use it only as unlabeled targetlanguage data. Moreover, to imitate the zero-resource crosslingual NER case, we ignore all target-language dev sets, and directly evaluate the learned models on their test sets.

Implementation Details

We implement our UniTrans with PyTorch 2 . For word-toword translation, we use fastText 3 monolingual word embeddings, and use MUSE 4 [Lample et al., 2018] to perform translation. For the feature encoder of the base model, i.e., fθ in Eq. 6, we employ the pretrained multilingual BERT model (case-sensitive version) [Devlin et al., 2019] in HuggingFace's Transformers 5 , which has 12 Transformer blocks, 12 attention heads, and 768 hidden units.

We empirically set UniTrans hyper-parameters by following previous works (as cited below), and utilize them in all experiments. Specifically, we adopt a dropout rate of 0.1 [Wu et al., 2020] and freeze the parameters of the embedding layer and the bottom three layers of the multilingual BERT [Wu and Dredze, 2019]. Following [Wu et al., 2020], we train all models for 3 epochs using a batch size of 32, maximum sequence length of 128, and AdamW [Loshchilov and Hutter, 2017] as the optimizer. For AdamW, we use a learning rate of 5e-5 [Wolf et al., 2019] for teacher models and 1e-4 for the student model [Yang et al., 2019]. Note that if a word is split into several subwords after tokenization, only the first subword is considered in the loss function.

2 https://pytorch.org/

3 https://fasttext.cc/docs/en/pretrained-vectors.html

4 https://github.com/facebookresearch/MUSE

5 https://github.com/huggingface/transformers

Performance Metric

Following [Tjong Kim Sang, 2002], we use entity level F1score as the performance metric. Moreover, we conduct each experiment 5 times and report the mean F1-score.

4.2 Experimental Results

Table 2 reports the zero-resource cross-lingual NER results of the proposed UniTrans on the 4 target languages, alongside those reported by prior state-of-the-art methods and those of two re-implemented baseline methods, i.e., Model Transfer (Θsrc in 3.3) and Data Transfer (Θtrans in 3.4).

Table 2 shows that our proposed UniTrans significantly outperforms the prior state-of-the-art methods and the reimplemented baselines on all target languages. Particularly, compared with the best prior method [Wu et al., 2020], the proposed UniTrans achieves an improvement of F1-score ranging from 1.66 for German [de] to 2.56 for Spanish [es]. Moreover, UniTrans achieves an average improvement of 1.63 F1-score over Data Transfer, and 2.60 F1-score over Model Transfer. All these results well demonstrate the effectiveness of the proposed UniTrans, which is mainly attributed to unifying model transfer with data transfer and leveraging unlabeled target-language data in UniTrans.

4.3 Ablation Study

To validate the contributions of different components in the proposed UniTrans, we introduce the following variants of UniTrans and baselines to perform ablation study: 1) UniTrans w/o L˜xsoft , which removes L˜xsoft from Eq.13 to train the target-language NER model Θstu with only the pseudo hard labels; 2) UniTrans w/o L˜xhard , which eliminates L˜xhard from Eq.13 (i.e., η = 0) to train Θstu with only soft labels output by the teacher model Θteach; 3) UniTrans w/ Θsrc, which uses the model Θsrc learned on the sourcelanguage training data with language-independent features as the teacher to train Θstu on unlabeled target-language data; 4) UniTrans w/ Θtrans, which uses the model Θtrans learned on translated target-language training data as the teacher to train Θstu on unlabeled target-language data; 5) UniTrans w/o DT (i.e., Θteach), which cuts out the access to the unlabeled target-language data DT and directly applies the teacher model Θteach to the target language; 6) Model Transfer (i.e., Θsrc), which directly applies the model Θsrc to the target language; 7) Data Transfer (i.e., Θtrans), which directly applies the model Θtrans to the target language; 8) Data Combination, which directly combines the labeled source-language data and the translated target-language data to train an NER model for the target language with languageindependent features. Note that for variants 3) UniTrans w/ Θtrans and 4) UniTrans w/ Θsrc, the hard label loss L˜xhard is also removed from Eq. 13, as we cannot use one model to generate pseudo hard labels.

Table 3 highlights the performance contributions of each component in our proposed UniTrans, and removing any of them will generally lead to a performance drop. Moreover, we can draw more in-depth observations as follows.

1) The proposed UniTrans outperforms UniTrans w/o L˜xsoft and UniTrans w/o L˜xhard in most cases, indicating that combining both for training the target-language model Θstu is reasonable. That also validates the effectiveness of the proposed voting scheme to generate pseudo hard labels.

2) UniTrans w/o L˜xhard outperforms UniTrans w/ Θtrans and UniTrans w/ Θsrc. Meanwhile, UniTrans w/o DT (i.e., Θteach) outperforms Model Transfer (i.e., Θsrc) and Data Transfer (i.e., Θtrans). Such results well demonstrate that the proposed teacher model Θteach, which unifies model transfer and data transfer, is superior to just using one or the other, no matter whether unlabelled target-language data is used or not. That also well verifies the complementarity between model transfer and data transfer.

3) Comparing UniTrans with UniTrans w/o DT (i.e., Θteach), UniTrans w/ Θsrc with Model Transfer (i.e., Θsrc), and UniTrans w/ Θtrans with Data Transfer (i.e., Θtrans), respectively, we can see that eliminating the usage of unlabeled target-language data will lead to a consistent performance drop in all experiments. That further demonstrates the importance of leveraging information contained in unlabeled target-language data for cross-lingual NER.

4) UniTrans w/o DT (i.e., Θteach) outperforms Data Combination, meaning that the proposed Θteach utilizes a superior way to unify model transfer and data transfer than simply combining available labeled data.

4.4 Discussion: Extend with Teacher Ensembling

Considering the randomness brought by dropout layers in the feature encoder (i.e., multilingual BERT) of the proposed UniTrans, we can extend UniTrans with teacher ensembling via taking advantage of the randomness. Specifically, we use an ensemble of teacher models learned via M runs, each denoted as Θ(m) ∗ ∈ {Θ(m) src ,Θ(m) teach , Θ(m) trans } with m = 1, 2, ...,M. And we simply average the predictions of Θ(m) ∗ to train the student target-language model Θstu:

where Θ∗ ∈ {Θsrc,Θteach,Θtrans}. Note that Eq. 14 is not only adopted to predict soft labels for unlabeled targetlanguage data, but also employed in the generation of pseudo hard labels.

Table 4 reports the results of teacher ensembling with M = 5. It is evident that ensembling further brings consistent performance improvements on nearly all target languages for UniTrans, with an average gain of 0.24 F1-score.

5 Conclusion

In this paper, we propose a novel approach for cross-lingual NER termed UniTrans, which unifies both model transfer and data transfer based on their complementarity via enhanced knowledge distillation on unlabeled target-language data. We also propose a voting scheme to generate pseudo hard labels for part of words in the unlabeled target-language data, so as to enhance knowledge distillation with supervision from both hard and soft labels. We evaluate the proposed UniTrans on benchmark datasets for four target languages. Experimental results show that UniTrans achieves new state-of-the-art performance for all target languages. We also extend UniTrans with teacher ensembling, which leads to further performance gains.