Improving Pseudo Labels with Global-Local Denoising Framework for Cross-lingual Named Entity Recognition

1 Introduction

Named Entity Recognition (NER) aims to identify spans belonging to predefined entity classes in a given text, such as locations and organizations [Qu et al., 2023b; Gu et al., 2022; Qu et al., 2023a], which is an important information extraction task [Zhu et al., 2023; Fan et al., 2023]. Deep neural networks trained on large amounts of labeled data have achieved excellent performance [Wei et al., 2021] on NER. However, most low-resource languages (e.g. Albanian and Finnish) still suffer from insufficient labeled data compared to high-resource languages, such as English. Moreover, annotating entities in texts is time-consuming and expensive. Such predicament motivates research on cross-lingual NER [Xie et al., 2018a]. Cross-lingual NER aims to leverage labeled data in a high-resource language (source language) and unlabeled data in a low-resource language (target language) to train an NER model, which generalizes well to the target language.

∗Corresponding author

Previous solutions for cross-lingual NER can be divided into three categories: (1) Feature-based methods, aiming to design and learn language-agnostic features to improve generalization ability in the target language [WU SJ, 2019]. (2) Translation-based methods, conducting translation and label projection on source language data to obtain labeled target language data [Jain et al., 2019]. (3) Teacher-student learning frameworks [Hinton et al., 2015]. These methods use a model trained on source language data to assign soft pseudo labels for target language data [Ma et al., 2022]. Then, the pseudo-labeled data is used to train a target model.

Recently, teacher-student learning frameworks have gradually become the mainstream solution. For these methods, the performance of the target model improves as the quality of pseudo labels increases. To this end, many improvement strategies have been proposed. Certain methods select high-confidence pseudo-labeled data for training through adversarial discriminators [Chen et al., 2021] or reinforcement learning [Liang et al., 2021]. However, such data selection results in an increased distributional difference and fails to fully utilize target language data. There are also methods integrating multiple source models [Wu et al., 2020; Wu et al., 2021] to generate pseudo labels. However, constructing various data sources to obtain diverse models significantly increases the training costs.

Considering these aforementioned issues, we propose a Global-Local Denoising Framework (GLoDe). Specifically, we leverage the global and local distribution information of the semantic space to denoise pseudo-labeled target language data. With our framework, high-quality data can be obtained without data selection or model ensemble. In detail, at the global level, we propose a prototype-based pseudo-label refinement method. Prototypes can be considered cluster centroids of each type. Conventionally, if a sample is most similar to a certain type's prototype, it is generally regarded as belonging to that specific type. [Zhou et al., 2023] initially leverages this strategy to iteratively improve pseudo labels. However, misclassified samples may exhibit a higher degree of similarity to the prototype of the incorrect type rather than to prototypes of other types. Therefore, we adopt a multipledecision scheme to tackle it, as shown in Figure 1(a). When the similarity between a sample and the prototype of a particular type exceeds a specific threshold, the type is considered as a potential classification for that sample. To affirm the credibility of similarity thresholds, we adopt dynamic thresholds for each entity type. Consequently, a sample may have more than one potential type, which mitigates the problem of misclassification. At the local level, we leverage the distribution information of the sample's neighborhood. For a given sample, we retrieve its K-nearest neighbors in the semantic space. Based on the distribution of the neighbors' types, we define the similarity of the sample to each type. Then, we compare these similarities with class-specific thresholds to obtain a potential classification for that sample, following the same procedure applied at the global level. By integrating the decision from both global and local levels, we obtain holistic denoising directions for the pseudo label of samples.

Additionally, prior methods typically improve the crosslingual NER with language-agnostic features, adopting alignment tasks such as adversarial training or contrastive learning [Zhou et al., 2023], as shown in Figure 1(b). However, target language-specific features are never considered but are also beneficial. Therefore, we employ the Masked Language Model (MLM) task, which necessitates solely unlabeled target language data. This simple auxiliary task further enhances the model's understanding of the target language.

To sum up, our contributions can be concluded as follows:

• We design a global-local denoising framework that comprehensively leverages information within the semantic space to rectify the incorrect pseudo labels. A multipledecision scheme with dynamic class-specific thresholds is devised to mitigate the problem of misclassification and significantly improve the quality of pseudo labels.
• We propose to improve the cross-lingual NER with tar-
• get language-specific features, which are never considered by previous work. To this end, we devise an auxiliary MLM task, which is both simple and beneficial.
• Experimental results on two benchmark datasets with six target languages demonstrate the superior performance of our approach.

2 RelatedWork

Existing methods for cross-lingual NER mainly fall into three categories: feature-based methods, translation-based methods, and teacher-student learning methods.

Feature-based Methods These methods learn languageagnostic features so that the model trained on the source language data can better adapt to the target language. Earlier work leverages gazetteers [Zirikly and Hagiwara, 2015] and Wikifier features [Tsai et al., 2016]. The introduction of multilingual pre-trained language models has significantly improved the cross-lingual representations [Conneau et al., 2020; Pires et al., 2019]. Some methods incorporate semantic alignment for further improvement through adversarial training [Keung et al., 2019] and contrastive learning [Wu and Dredze, 2020]. However, coarse-grained alignment is useless for NER. Entity-level alignment is advantageous, but the lack of ground truth labels in target language data inevitably leads to incorrect alignment. Furthermore, these methods do not acquire target language-specific features.

Translation-based Methods These methods translate the source language data and perform label projection to acquire labeled data in the target language. Some approaches employ word-to-word [Xie et al., 2018a] or phrase-to-phrase translation [Mayhew et al., 2017], with entity labels directly linked to the translated content. However, substantial semantic information is sacrificed during translation. Therefore, other approaches consider translating the entire sentence [Jain et al., 2019], but the label projection process will introduce noise labels. [Liu et al., 2021] introduces a placeholder-based approach that combines the aforementioned two strategies. [Yang et al., 2022] adopts the back-translation scheme. Nevertheless, the processes of translation and label projection inevitably carry the risk of introducing noise.

Teacher-student LearningMethods Thesemethods leverage a source model (e.g., a model trained on source language data) to assign soft pseudo labels for target language data, subsequently utilizing this data to train a new model [Wu et al., 2020]. Soft labels offer more information and reduce the risk of overconfidence compared to hard labels [Hinton et al., 2015]. However, this remains insufficient. The key question is the quality of pseudo labels. [Ma et al., 2022; Ge et al., 2023] leverage the concept of feature-based methods to train the source model, aiming to generate higher-quality pseudo labels. [Chen et al., 2021; Liang et al., 2021] attempt to select a subset of the pseudo-labeled data with high confidence. However, data selection prevents the model from fully learning the characteristics of the target language data. Introducing multiple models [Li et al., 2022] is also widely used. [Wu et al., 2021; Zhao et al., 2022] integrate the predictions of multiple models as the final pseudo labels. [Zeng et al., 2022;

Ma et al., 2023] allow multiple models to generate pseudo labels for each other separately. However, various data sources are needed to train diverse models. For instance, [Zhao et al., 2022] introduces source language data, translation data, and data with disrupted word order.

3 Methodology

In this section, we describe our proposed framework GLoDe. Figure 2(a) presents the overall architecture of our approach.

3.1 Overall Framework

Span-based NER Task

Following previous works [Fu et al., 2021; Zhou et al., 2023], we formulate NER as a span classification problem. Formally, given a sentence X = {x1, x2, . . . , xn}, our objective is to assign entity labels to each possible span si = {xbi , xbi+1, . . . , xei } in X, where xbi and xei denote the beginning and end tokens of si.

For an input sentence X, we first obtain its token representations through a pre-trained language model (PLM): H = {hi}ni=1 = PLM(X). Let S(X) = {si}mi=1 denote all possible spans in X. For each si ∈ S(X), we introduce additional span length embedding li and span morphology embedding mi, which are both learnable embedding parameters. Then, we formulate the span representation by integrating the representations of the beginning and end tokens, the length embedding, and the morphology embedding: zi = [hbi ; hei ; li;mi]. Finally, zi is fed into a linear classifier to yield its probability distribution pi ∈ R|C| , where C is the entity label set.

For source language data Dsrc, the ground truth label yi for each span si is available. The loss function is defined as:

where CE(·, ·) denotes the cross-entropy loss, and N is the number of sentences in the dataset.

Masked Language Model Task

Due to the scarcity of labeled data in the target language, many prior studies further align the source and target languages to improve the model's performance. However, their emphasis remains on language-agnostic features. Instead, we conduct the masked language model task on target language data. As a widely used pre-trained task, it facilitates us to fully exploit the knowledge in the multilingual pre-trained language model and improve the model's generalization ability with target language-specific features.

Formally, given a sentence X = {xi}ni=1 in target language data Dtgt, we obtain the masked sentence X′ = {x′i }ni=1 , where each token x′i may be equal to either the original value xi or a special token "[MASK]" based on a sample rate rm. Then we obtain token representations H′ = {h′i }ni=1 for the masked sentence through the PLM. Finally, H′ is passed through a masked language model head (MLM Head) to yield their probability distributions p′i ∈ R|V| , where V is the vocabulary set of PLM.

For x′i ∈ X′ , its label y′i is defined as the token id of xi ∈ X in the vocabulary set V. The loss function is:

where I(·) is the indicator function.

Training Objectives

In our framework, we simultaneously conduct the NER task on source language data and the MLM task on target language data to obtain a source model. Then, we use the source model to assign initial pseudo labels to the target language data. This enables the generation of higher-quality initial pseudo labels, establishing a robust foundation for subsequent training. Here the training objective for the source model is defined as:

where ω is a weight hyperparameter.

After obtaining the source model, we train the target model with our proposed framework. The loss function for the NER task on target language data is defined as follows:

where ˆ pi is the soft pseudo label generated by the source model. As shown in Figure 2(a), the soft pseudo label is refined by the global-local denoising mechanism. We will describe the denoising mechanism in the next section. Finally, the total objective for the target model is defined as follows:

where ω takes the same value adopted in Eq.3.

3.2 Global-Local DenoisingMechanism

As shown in Figure 2(b), two parallel decision strategies are employed to determine the final denoising directions for pseudo labels of target language data.

Global-level Decision

This module establishes a set of denoising directions based on the global information within the semantic space. The procedure is shown in the upper part of Figure 2(b).

Global Similarity Score We employ the prototype [Snell et al., 2017] of each entity type to describe the global distribution of the semantic space. The prototype ϕc of entity type c ∈ C serves as the clustering center of samples belonging to this type, which is robust to outliers to some extent [Xie et al., 2018b]. Following previous works [Ge et al., 2023; Zhou et al., 2023], we adopt a moving average scheme based on batch data to compute ϕc, ensuring computational efficiency and a stable training process. Specially, for a batch of source and target language span representations {zi}, ϕc is computed and updated as follows:

where t denotes the training step, and yi is either the ground truth label for the source language span or the pseudo label for the target language span. α is the update rate coefficient. L2 normalization is adopted here.

Based on these prototypes, we use the dot product to compute the similarity between a target language span zi and an entity type c: simci = zi · ϕc. Similarly, we obtain the similarities between zi and all types: Simi = {simci }c∈C, which we call "global similarity score".

Global Similarity Threshold We determine thresholds for each entity type to assess the aforementioned similarity scores. Instead of employing fixed thresholds, we adopt the average similarity score of each type as dynamic ones and update them at the end of each training epoch to ensure their validity and training stability. Specifically, we first segment the target language spans based on their pseudo-labeled entity types: Zc = {zi| ˆ yi = c, ˆ yi = argmax( ˆ pi)}, where ˆ pi is the soft pseudo label for zi. Then the similarity threshold for type c is defined as follows:

We obtain the similarity thresholds for all entity types: SimT = {simcT }c∈C, called "global similarity thresh".

Global Denoising Directions Conventional approaches typically consider solely the type with the highest similarity score. However, a significant portion of misclassified samples are most similar to their incorrect types' prototypes. Thus, we consider more types rather than the most similar ones. For a target language span zi, we compare its similarity score Simi with the similarity thresh SimT to determine the update directions for its pseudo label. We describe the update directions as a binary vector:

Take the upper part of Figure 2(b) for example. Suppose we assume the entity types are {LOC, ORG, PER, O} ("O" denotes non-entity), the pseudo label denoising directions (LOC and PER) can be expressed as the vector [1, 0, 1, 0].

In addition, considering the class imbalance problem (i.e., non-entity spans constitute a substantial portion) [Zhou et al., 2023], we handle the non-entity type differently. If a span has the highest similarity score with type "O", we directly take "O" as an update direction for its pseudo label, regardless of whether this similarity score surpasses the threshold.

Local-level Decision

As shown in the lower part of Figure 2(b), the procedure of this module aligns with that of Global-level Decision. The main difference lies in the information employed.

Local Similarity Score We leverage the types of a sample's neighbors to describe the local distribution of the semantic space. The use of robust learning techniques, which leverage neighbor information, is extensively employed to effectively handle noisy data [Ma et al., 2023]. We utilize the entity type distribution of a sample's neighbors to indicate the sample's similarity with each entity type. Specifically, we first collect source and target language span representations to construct a representation repository Dz. We update Dz at the end of each training epoch to accommodate changes in representation distribution during the training process. Then, for a target language span representation zi, we retrieve its K-nearest neighbors Nk(zi) fromDz. Finally we compute the similarity between zi and entity type c based on entity types of spans in Nk(zi):

where yj is either the ground truth label for the source language span or the pseudo label for the target language span. Similarly, we obtain the similarities between zi and all types: Simi = {simci }c∈C, which we call "local similarity score".

Local Similarity Threshold and Denoising Directions Based on the local similarity score acquired above, we employ the same strategy and computation process for the global similarity threshold in Eq.7 to obtain the local similarity threshold. Subsequently, following Eq.8, we obtain denoising directions by utilizing the local similarity score and the local similarity threshold. We omit the detailed description here for brevity.

Pseudo Label Refinement

Noise samples exhibit a range of styles, including isolated points situated away from centroids and fuzzy points near classification boundaries. To derive the final denoising verdict, we integrate decisions made on both global and local levels. This strategy assists in detecting and addressing a variety of noise patterns. Specifically, for a target language span zi, assuming its global-level and local-level denoising direction vectors are dgi and dli , the update decision for its pseudo label is defined as follows:

Let the current pseudo label of zi be pti . The updated pseudo label is computed as follows:

where β is the update rate coefficient. L1 normalization is adopted in Eq.10 and Eq.11. With the improvement of pseudo labels, the referential value of the update decision ui increases. Thus, we uniformly decrease β to increase the impact of ui during training.

There might exist multiple update directions with different updating degrees as shown in Figure 2(b) (e.g., PER has a higher updating degree than LOC). This allows misclassified spans to escape from their original incorrect types and adjust to high-confidence types, which are more likely to be correct. During the updating procedure, information within the semantic space is gradually integrated into the soft pseudo labels, which is beneficial for improving the model's generalization ability to the target language data.

4 Experiments

4.1 Experiment Settings

Datasets

We conduct experiments on two widely-used benchmarks: CoNLL [Sang and Erik, 2002; Sang and De Meulder, 2003] and WikiAnn [Pan et al., 2017]. The CoNLL dataset comprises four languages: English (en), German (de), Spanish (es), and Dutch (nl). The dataset is annotated with four entity types: PER, LOC, ORG, and MISC. The WikiAnn dataset comprises four languages as well: English (en), Arabic (ar), Hindi (hi), and Chinese (zh). The dataset is annotated with three entity types: PER, LOC, and ORG.

Following previous works [Wu et al., 2021; Zhao et al., 2022], we take the English data in each dataset as the source language data and other languages as the target language. We remove entity labels from training sets of target languages, taking them as the unlabeled target language data.

Implementation Details

Following previous works [Zeng et al., 2022; Ma et al., 2023], we use XLM-R-large [Conneau et al., 2020] as the backbone pre-trained language model. The model is trained for 8 epochs (except 6 epochs for zh) and the batch size is 32. We use AdamW as the optimizer and set the learning rate to 1e-5. We set the mask rate rm to 0.25 and loss weight ω to 1e - 3 (1e - 5 for es) for the MLM task. To improve the training efficiency, we map span representations to a lowdimensional space of 128 dimensions for the denoising computations. For neighbor retrieval, we set K to 300 and utilize the FAISS tool [Johnson et al., 2019]. The update rate coefficient α = 0.99 in Eq.6. β in Eq.11 uniformly decreases from 0.95 to 0.80 during training.

Baselines

We compare our method with previous baselines as follows: (1) AdvPicker [Chen et al., 2021] employs an adversarial discriminator to select pseudo-labeled target language data with high confidence for training. (2) RIKD [Liang et al., 2021] leverages reinforcement learning for selecting pseudolabeled data and iteratively conducts training for multiple rounds. (3) MTMT [Li et al., 2022] introduces an entity similarity evaluator to support the training of the NER task. (4) CROP [Yang et al., 2022] transposes the NER task from the target language to the source language via back-translation and entity matching. (5) TransAdv [Zhao et al., 2022] introduces multiple data sources to obtain distinct source models for comprehensive pseudo labels. (6) ProKD [Ge et al., 2023] proposes a contrastive learning-based prototype alignment approach to boost the model's ability to capture language-agnostic knowledge. (7) CoLaDa [Ma et al., 2023] utilizes two distinct models to denoise the labels of each other and further considers the label consistency to re-weight the noisy labels. (8) ContProto [Zhou et al., 2023] incorporates contrastive learning and prototype-based pseudo-labeling for representation learning and pseudo-label refinement.

4.2 Main Results

Tables 1 and 2 present the results of our proposed method and previous baselines on CoNLL and WikiAnn. Following previous works, we employ the entity-level micro-F1 score as the metric. We conducted all experiments three times and calculated the average value as the final result.

It can be seen that our proposed GLoDe outperforms prior methods, achieving an improvement of 0.41 F1 scores on average for CoNLL and 1.89 F1 scores on average for WikiAnn. This demonstrates the effectiveness of our approach. Besides, there are two noteworthy findings evident in the results: (1) Our model reaches suboptimal performance on German (de). The German dataset has fewer entities, and numerous sentences do not contain entities, potentially leading the model to predict more false positive entities. Additionally, both CoLaDa and CROP incorporate translated content, which facilitates the model to acquire more entity information in the target language. (2) For target languages (e.g., es, nl) closer to the source language (en), previous works have shown incremental improvements in a moderate trend. Conversely, for distant source-target pairs (e.g., en-ar, en-zh), models performing the denoising process on pseudo-labeled target language data (ours, ContProto, and CoLaDa) have shown significant improvements over other models. We attribute this to the critical label noise problem in these distant languages.

4.3 Ablation Study

To further validate the effectiveness of components in our proposed model, we conduct the following ablation studies: (1) w/o mlm, which removes the masked language model task during training; (2) w/o denoising, which removes the globallocal denoising mechanism; (3) w/o denoising-local, which only leverages global information in the semantic space for pseudo label denoising; (4) w/o denoising-global, which only leverages local information in the semantic space for pseudo label denoising; (5) w/o multiple directions, which only considers the type with the highest similarity score as the potential update direction for the pseudo label; (6) w/o mlm init, which omits the masked language model task, training the source model solely on source language data to acquire the initial pseudo labels.

Table 3 shows the ablation study results. It can be seen that: (1) Both w/o mlm and w/o denoising diminish the model's performance, which demonstrates the effectiveness of these two modules. Both of them improve the model's generalization ability on the target language. Additionally, removing the denoising mechanism exerts a more significant impact on most datasets, emphasizing the significance of noise reduction for pseudo-labeled data. (2) w/o denoising-local and w/o denoising-global bring less performance decline compared to w/o denoising. This implies that leveraging either local or global information in the semantic space aids in pseudo-label denoising. Naturally, employing both together yields superior results. Moreover, there seems to be no absolute answer to the question regarding which part of the information is more useful. (3) w/o multiple directions also improves the performance compared to w/o denoising, but it falls short of comparison to other denoising schemes. This implies that incorporating multiple denoising directions would be more beneficial. (4) w/o mlm init denotes the significance of improving the source model with target language-specific features. Additionally, the quality of the initial pseudo labels significantly constrains the target model's performance.

4.4 Quality of Pseudo Labels

In this section, we assess the quality of pseudo labels during training. As in the prior experiments, we measure the quality by entity-level micro-F1 scores. We compare the following three pseudo label denoising strategies: (1) global, which only leverages global information in the semantic space for pseudo label denoising; (2) local, which only leverages local information; (3) global&local, which leverages both global and local information, serving as the solution we adopt.

As shown in Figure 3, the quality of pseudo labels improves progressively during training. The utilization of either global or local information improves pseudo labels, with the combined use of both resulting in even greater improvement for most of the target languages. We argue that the two types of information can complement each other and they have different impacts for different target languages. For example, local yields more advantages for De and Zh, while global provides greater benefits for Es. For Nl, little difference is observed among the three strategies, with limited improvement obtained after 4th epoch. We attribute this to the high similarity between Nl and the source language (En). The information from either the global level or the local level may be sufficient. For Hi, global may even exhibit a negative impact. Given that the source language contains nearly five times as many entities as Hi, local-level information proves to be more helpful than global-level information.

4.5 Effectiveness of Language-specific Features

The auxiliary task improves both the source and target model with target language-specific features. In this section, we further analyze this impact.

As shown in Figure 4, we compare entity distributions from different data sources. Specifically, the source language entity distributions (blue lines), target language entity distributions (green lines), and model-predicted entity distributions for the target language (orange lines) are considered. The predicted distribution in Figure 4(a) is obtained from the model trained solely with source language data (Plainmodel), and that in Figure 4(b) is obtained from the model trained with both source language data and the auxiliary task (Aux-model). We can observe that the predicted distribution shifts towards the target language distribution after introducing target language-specific features by the auxiliary task.

Additionally, we conducted some comparative experiments to demonstrate the advantages of language-specific features. We replace the masked language model task in our framework with the supervised contrastive learning task, which has recently been widely used to improve the model with language-agnostic features ([Ge et al., 2023; Zhou et al., Aux-model src pred tgt LOC ORG PER MISC

(b)

Table 4: Comparision of impacts of target language-specific features and language-agnostic features for model improvement.

2023]). As shown in Table 4, we denote the model improved by language-agnostic features and language-specific features as GLoDeagnostic and GLoDespecific, respectively. Comparison results demonstrate that language-specific features are more beneficial for model improvement.

5 Conclusion

In this paper, we present GLoDe, a pseudo-label denoising framework for cross-lingual NER. Our global-local denoising mechanism integrates global and local information within the semantic space to rectify as many incorrect pseudo labels for target language data as possible. The information within pseudo labels progressively increases during the denoising process, which helps to improve the model's generalization ability on the target language. Additionally, we further improve our model with target language-specific features, leveraging the masked language model task. Experimental results on six benchmark datasets demonstrate that GLoDe achieves superior performance over previous baselines.

0.3

0.2

0.1

0.0

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grant No. 62276110, No. 62172039 and in part by the fund of Joint Laboratory of HUST and Pingan Property & Casualty Research (HPL). The authors would also like to thank the anonymous reviewers for their comments on improving the quality of this paper.