Automatic Noisy Label Correction for Fine-Grained Entity Typing

1 Introduction

Fine-grained entity typing (FET) [Ling and Weld, 2012] aims to predict fine-grained semantic types for entity mentions according to their contexts. Recently, [Choi et al., 2018] further proposed the ultra-fine entity typing introducing a richer type set. The fine-grained entity types obtained by FET is beneficial for many downstream NLP tasks like entity linking [Onoe and Durrett, 2020a], relation extraction [Koch et al., 2014; Shang et al., 2020], question answering [Wei et al., 2011; Wei et al., 2016], dialogue [Wei et al., 2019; Wei et al., 2021] and coreference resolution [Onoe and Durrett, 2020b].

A fundamental challenge of FET is handling the noisy labels during training. Current FET systems usually generate training data via crowdsourcing or distant supervision, both of which may introduce noisy labels. With the large-scale and fine-grained type sets, it is extremely difficult for humans to annotate samples accurately. The context-agnostic distant supervision method also introduces noise inevitably. Figure 1 shows some noisy labels from the training sets of Ultrafine and Ontonotes. Previous denoising work in FET usually relies on pre-defined hierarchical type structures or manually labeled subsets to identify noisy labels. However, these auxiliary resources may be unavailable in real-world applications, which limits the application of such methods.

∗Corresponding author.

To automatically correct noisy labels without the help of auxiliary resources, an intuitive way is to treat the potentially noisy labels as unlabeled data and train the FET model with the rest clean labels. Then we can relabel candidate noisy labels to generate a cleaner dataset. Intuitively, excluding the potentially noisy labels during training makes the model more robust, and learning from clean samples helps the model to make correct predictions on similar but mislabeled samples. The main challenge is to identify the potentially noisy labels precisely. This problem also receives widespread attention in robust learning from noisy labels. Recent studies reveal that deep neural networks (DNNs) are prone to first learn simple patterns before overfitting noisy labels [Arpit et al., 2017]. Based on this memorization effect, there are two main approaches to filter noisy labels. Methods like Co-teaching [Han et al., 2018] treat the labels with the top k% loss as possible noisy labels (k% is the estimated noise rate). Some other methods like SELF [Nguyen et al., 2020] record the model output on different training epochs and filter noisy labels progressively. Despite their success in other fields, those methods are difficult to directly apply to FET tasks for two reasons: i) Firstly, the noise rate can differ across FET types. Manually estimating the noise rate for each type is time-consuming. ii) Secondly, existing FET datasets usually provide a massive training set annotated by distant supervision. Current FET models can even converge before iterating through all data. Thus the multi-round methods are infeasible in practice.

To this end, we propose a novel approach to select the potentially noisy labels for FET. Figure 2 briefly demonstrates the selection process. We note the model does not strictly separate positive and negative labels before overfitting, which means the model is "ambiguous" on some labels in the early stage of training. We argue this phenomenon is partially caused by noisy labels. Due to the existence of noisy labels, samples belonging to the same pattern may have different annotations. This inconsistency between annotation and underlying patterns brings difficult for the model to separate samples according to their annotations. Therefore, we propose to select potentially noisy labels by identifying the "ambiguous labels". Specifically, we first fit the logits output by the model on positive and negative labels using two Gaussian distributions, respectively. Then, we calculate the posterior probability of a label being positive or negative. Finally, we select the potential noisy labels according to the agreement between original annotations and the calculated posterior probabilities.

In summary, our main contributions are three-fold:

• We propose a novel method to automatically correct noisy labels in the FET training corpus without the help of auxiliary resources.
• We propose an effective method to automatically identify the potentially noisy labels by discriminating the "ambiguous labels" from the normal ones.
• Experiments on two widely-used benchmarks prove the effectiveness of the proposed method.

2 RelatedWork

2.1 Fine-grained Entity Typing

The noisy labeling problem in FET domain has been studied for years. [Ren et al., 2016a; Ren et al., 2016b] propose the partial label loss (PLL) and the partial-label embedding method to alleviate the negative influence caused by noisy labels, [Onoe and Durrett, 2019] utilize a small set of manually labeled training data to train a filter module and a relabel module then denoise the distantly-labeled data. [Chen et al., 2019] treat the noisy labels as unlabeled data and performs Compact Latent Space clustering (CLSC) to regularize their representation. [Zhang et al., 2021] assume each sample only has one most appropriate label and treat the underling ground truth labels as trainable parameters for finetuning during training. [Wu et al., 2021] estimate the noise confusion matrix through hierarchical label structure. Other works on the FET task dedicate to mining label dependencies [Xiong et al., 2019; Liu et al., 2021] designing more expressive embedding method [Onoe et al., 2021] and exploring better automated data annotation method [Dai et al., 2021] to improve performance.

Most existing denoising methods in FET rely on prior auxiliary resources (pre-defined hierarchical type structures/manually labeled subsets) or assume each sample only has one most appropriate label. As compared, in this paper we consider the more restrictive setting where no auxiliary resources are available and one sample can have multiple labels simultaneously.

2.2 Robust Learning fromNoisy Labels

Previous studies have pointed out that DNNs are susceptible to noisy labels [Zhang et al., 2017]. Sample selection is a popular method to counteract noise, which only uses relatively clean labels to train the model. Here we briefly note several previous works in this field.

Co-teaching [Han et al., 2018] simultaneously train two collaborating networks to filter out potentially noisy labels according to their loss. This method needs to control select how many small-loss instances as clean ones, which can be tricky in practice. To automatically select clean labels, DivideMix [Li et al., 2020] leveraged a one-dimensional and two-component Gaussian Mixture Model (GMM) to model the loss distribution of clean and noisy labels. SELF [Nguyen et al., 2020] observed that the network's prediction is consistent on clean samples while oscillating strongly on mislabeled samples. Therefore, they select the clean labels according to the agreement between annotated labels and the running average of the network's prediction.

In practice, we found that the loss does not form a bimodal distribution in FET tasks. As a result, fitting the GMM can not distinguish clean and noisy labels precisely. The multiround methods are also infeasible. Training on the large-scale distantly annotated training set, current FET models can converge or even overfit noisy labels in the first epoch. To this end, we propose a novel method to select potentially noisy labels by estimating the posterior probability of a label being positive or negative according to the logits output by the model.

3 Methodology

Indeed, our method corrects the noisy labels with two major steps, namely: (1) Selecting potentially noisy labels by identifying "ambiguous labels"; (2) Re-labeling training sets by learning a robust entity typing model over the clean labels. Next, we will first introduce the problem definition and the entity typing model used in this paper, and then present our noisy label correction method in detail.

3.1 ProblemDefinition

Entity typing datasets consist of a collection of (sentence, mention) tuples: D = {(x1,m1), (x2,m2), .., (xn,mn)}. Given a sentence x and a marked entity mention m, the finegrained entity typing task aims to predict a set of types y ⊂ T of m, where T is a set of pre-defined fine-grained semantic types (e.g., actor, company, building).

3.2 Entity TypingModel

In FET tasks, a widely-used paradigm is to fine-tune the pretrianed language models (PLMs) for entity typing. In this paper we adopt the prompt learning model [Ding et al., 2021] to tune PLMs for entity typing, as it is proved to be effective for fine-tuning PLMs to specific tasks. Specifically, we reformulate the original input (sentence, mention) into the prompt input via a crafted template as follows:

where [P1], [P2], [P3] are additional trainable special tokens. Their embeddings are randomly initialized. Conventionally, prompt-based tuning models need to define a set of label words Vy for each y ∈ T , and then the model can predict entity types by filling the [MASK]:

In this way, FET is converted to mask prediction, one of the pre-training tasks in Masked Language Models. However, it's difficult to choose a proper label word set for each type in FET. In practice, we directly modify the decoder in the Masked Language Modeling Head (MLM Head) and let it output the probability of samples belonging to each candidate type:

where h[MASK] is [MASK]'s contextualized representation produced by PLMs; F represent the Masked Language Modeling Head without decoder; wy and by are trainable parameters in modified decoder. If type y is in PLMs' vocabulary, we initialize wy and by according to y's weight and bias in MLMHead's decoder, respectively. If y consists of multi tokens, we take the average of their weight and bias in MLM Head's decoder to initialize wy and by.

3.3 Candidate Noisy Labels Selection

The basic idea of our method is to exclude potentially noisy labels from the supervision signals for robust training, then generate a cleaner dataset by relabeling the training set. In this section, we detail how to select potentially noisy labels.

According to the memorizatioin effect [Arpit et al., 2017], deep neural networks usually memorize simple patterns before overfitting noisy labels, which is also observed in our experiments. Figure 2 shows our model's output logits on type "organization" when achieving early stopping on the UltraFine dataset. Some samples with different annotations have similar logits, and this phenomenon is universal across types. According to our observation, those "ambiguous" samples mainly consists of mislabeled samples. This is because the mislabeled samples belong to the same pattern as some correctly labeled samples in datasets. Due to the memorization effect, the model tends to output similar logits on samples with the same pattern before overfitting. As a result, the predictions of noisy labels may be "ambiguous" or even opposite to the annotation during the early training phase.

Therefore, we propose to pre-train the model with the original annotation until early stopping, then select potentially noisy labels by independently identifying those "ambiguous labels" for each type. Specifically, we first estimate the posterior probability of a label being positive or negative according to the logits: p(y = 1|l). Then we can identify "ambiguous labels" through the agreement between p(y = 1|l) and original annotations. During pre-training, we use the following standard binary cross-entropy (BCE) loss:

where ys,t ∈ {0, 1} is the original annotation in datasets; ps,t is the model prediction, denotes the probability of sample s has type t; |B| is the batch size. After achieving early stopping, we gather the logit output by the model and estimate the posterior probability p(y = 1|l):

where y is the ground truth label and 0 means negative label, 1 means positive label; l is the logit output by the model. The main challenge is estimating the prior probability: p(y = 0), p(y = 1) and the likelihood function: p(l|y = 1), p(l|y = 0). We note the logits output by the model on positive (negative) labels form unimodal distribution (as shown in Figure 2). It's a natural idea to fit it using a one-dimensional Gaussian distribution:

However, original datasets only provide the noisy annotation ˆ y instead of the ground truth label y. Therefore, directly fit Gaussian distribution according to the original annotation only give p(l|ˆ y = 0) and p(l|ˆ y = 1). We recommend to remove those obvious noisy labels before estimating the likelihood function. Algorithm 1 provides a heuristic method to filter obvious noisy labels iteratively. This method first fits a Gaussian distribution on the model output logits then removes the outliers. This process will be repeated until no outliers can

Algorithm 1 Filter obvious noisy labels

Input: The logits output by the model: l; original noisy annotation in the dataset: ˆ Y .

Parameter: Outlier threshold α.

Output: Relative clean positive (negative) label set Y1 (Y0)

• 1: Let Y1 = ˆ Y1 = {y = 1|y ∈ ˆ Y }
• 2: Let Y0 = ˆ Y0 = {y = 0|y ∈ ˆ Y }
• 3: repeat
• 4: Let ˆ Y1 = Y1 and ˆ Y0 = Y0
• 5: Calculate the average ˆµ1 (ˆµ0) and standard deviation ˆ δ1 (ˆ δ0) of the logits output by the model corresponding to the labels in ˆ Y1 ( ˆ Y0)
• 6: Let Y1 = {ly > ˆµ1 -αˆ δ1|y ∈ ˆ Y1}
• 7: Let Y0 = {ly < ˆµ0 +αˆ δ0|y ∈ ˆ Y0}
• 8: until |Y1| == | ˆ Y1| and |Y0| == | ˆ Y0|
• 9: return Y1, Y0

be found. After getting the relative clean positive (negative) label set Y1(Y0), the prior probability and likelihood function can be estimated in the following way:

where k = 1(0) denotes the positive (negative) labels. Finally, a label is selected as a potentially noisy label if it satisfies one of the following rules:

where ϵ > 0 is a hyperparameter, and the larger value means more labels are selected as potentially noisy labels.

3.4 Noisy Label Correction

To correct the noisy labels in the dataset as many as possible, we propose to fine-tune the model using clean labels. Intuitively, training on clean samples helps the model to make correct predictions on those similar but mislabeled samples. For the selected potentially noisy labels, we treat them as unlabeled data and perform entropy regularization [Grandvalet et al., 2005] to reduce class overlap. Formally, the loss on a training batch during noisy label correction is as follows:

Algorithm 2 Noisy labels correction for FET

Input: Noisy labeled training set ˆDtrain; validation set Dval. Parameter: Threshold ϵ; weight of entropy regularization β; fine-tuning step k on clean samples.

Output: Cleaner training set Dtrain.

• 1: Fine-tune model M on ˆDtrain until the performance drops on Dval
• 2: Filter obvious noisy labels by Algorithm 1
• 3: Select potentially noisy labels by Equation (10)
• 4: Conduct k step fine-tuning on model Musing loss function in Equation (11)
• 5: Relabel ˆDtrain according to Equation (12) to generate Dtrain
• 6: return Dtrain

where Yc and Yn stand for clean and potentially noisy label set respectively; β is a hyperparameter that balances the BCE loss and the entropy regularization. Finally, we relabel potentially noisy labels to generate cleaner datasets:

where y′s,t is the new label after noisy label correction. Algorithm 2 summarizes the whole process of our method.

4 Experiments

4.1 Datasets and Experiment Setup

Datasets. We conduct experiments on two standard finegrained entity typing datasets: Ultra-fine and OntoNotes. Followed [Liu et al., 2021] we use the 2k/2k/2k train/dev/test splits for Ultra-fine which contains 6K manually annotated samples, 2519 categories. Original OntoNotes dataset contains 25k/2k/9k train/dev/test data, 89 categories. [Choi et al., 2018] offers an augmented training set containing 0.8M samples. We conduct experiments on both versions. In terms of evaluation metrics, we follow the widely used setting proposed in [Ling and Weld, 2012]. On Ultra-Fine, we report macro-averaged precision, recall, and F1-score. On OntoNotes, we report accuracy, macro-averaged F1-score, and micro-averaged F1-score.

Baselines. For ultra-fine entity typing task, we compare with the following approaches:

• UFET [Choi et al., 2018] performs multi-label classification on features encode by GloVe+LSTM and a character level CNN.
• LabelGCN [Xiong et al., 2019] models the label dependency using a graph propagation layer.
• LDET [Onoe and Durrett, 2019] performs relabeling and sample filtering to denoise the datasets before training FET models.
• MLMET [Dai et al., 2021] automatically generates ultra-fine entity typing labels to train the entity typing model by combining hypernym extraction patterns with a masked language model.

• Box [Onoe et al., 2021] proposes to use box embeddings instead of vector embeddings to capture latent type hierarchies.
• LRN [Liu et al., 2021] learns and reasons label dependencies in both sequence-to-set and ene-to-end manner.

For the OntoNotes dataset, in addition to the baselines mentioned above, we also compare with several approaches focusing on noisy labeling problems introduced in related work [Chen et al., 2019; Zhang et al., 2021; Wu et al., 2021].

Implementation. We use BERT-base-cased as backbone structures, AdamW optimizer with the learning rate of 2e-5 on Ultra-fine and 2e-6 on OntoNotes. The training batch size is 16 for all datasets. Other hyper-parameters are tuned by grid search and the optima configurations are {ϵ = 0.1, α = 2.0, β = 0.5, k = 2000} on Ultra-Fine; {ϵ = 0.4, α = 3.0, β = 2.0, k = 5000} on augmented OntoNotes; {ϵ = 0.3, α = 2.0, β = 1.0, k = 5000} on original OntoNotes.

4.2 Overall Results

Table 1 and Table 2 show the overall results on Ultra-Fine and OntoNotes test sets. We report the performance of our model (described in Section 3.2) training on the original training set (original) and the denoised training set (denoised) respectively. We can see that: i) Our noisy label correction method is effective across datasets. After using the denoised training set generated by our method, the model performance significantly improved and outperform other advanced baselines on almost all metrics; ii) Generate cleaner training data is critical to FET. Both our method and MLMET generate cleaner data to train the FET model. We can see they significantly outperform other baselines. iii) Prompt-learning is an effective approach to leverage PLMs in FET. Compared with methods using vanilla fine-tuning (e.g., Box, LRN), promptleaning brings significant performance gains. Training on the original noisy training data, our prompt-learning model still achieves competitive performance against other baselines.

Compared with previous works focusing on noisy labeling problems in FET, our method is more universal. NFETCCLSChier, NFETC-ARhier and FBTree require pre-defined hierarchical type structures during training. They are difficult to apply to datasets like Ultra-Fine whose type set is not being

without augmentation

LRN

56.6

77.6

71.8

NFETC-CLSChier 62.8±0.3 77.8±0.4 72.0±0.4

NFETC-ARhier

64.0±0.3 78.8±0.3 73.0±0.3

FBTree

64.0±0.1 78.4±0.2 72.5±0.2

Ours (original)

52.6±2.2 77.1±1.6 71.1±1.2

Ours (denoised)

59.2±0.2 81.3±0.3 75.3±0.4

w/o filter

59.6±0.6 79.8±0.3 74.6±0.3

w/o ER

56.9±0.3 80.3±0.3 73.8±0.2

Table 2: Strict accuracy, macro-averaged F1, and micro-averaged F1 of different approaches on OntoNotes test set.

explicitly hierarchical. LDET requires a small set of manually labeled training data, which is not provided by datasets like OntoNotes. In contrast, our method corrects noisy labels without auxiliary resources, which can be applied to all FET datasets. It's also worth noting that our approach is orthogonal to most existing FET models. In practice, our method can combine with other advanced FET models to yield better performance.

4.3 Ablation Study

Filter obvious noisy labels (filter) can estimate the posterior probability p(y = 1|l) more accurately, which is critical to potentially noisy label selection. Entropy regularization (ER) encourages decision boundaries to fall in low-density regions, which is helpful for noisy label correction. To better understand their effect on the model, we conduct the ablation study on Ultra-Fine and OntoNotes datasets. The results are reported in Table 1 and Table 2.

Effect of Filter Obvious Noisy Labels. This strategy brings improvements to all datasets, especially in the UltraFine dataset. Without filtering obvious noisy labels, our method only brings 0.5% improvement on Macro-averaged F1 compared to training on the original training set. This is mainly due to the long tail problem in Ultra-Fine: many longtail types only have several positive samples. So the outliers (obvious noisy labels) can seriously affect the estimation of likelihood function p(l|y = 1), lead to a poor estimation of p(y = 1|l). In practice, we recommend enabling this strategy in datasets with long-tail problems.

Effect of Entropy Regularization. Entropy regularization brings improvements to all datasets. This indicates treating potentially noisy labels as unlabeled data and training the model in a semi-supervised manner helps with noisy label correction.

4.4 Case Study

Figure 3 shows examples of the original noisy labels and the denoised labels produced by our method. In example (a), the original human-annotated labels fail to cover all proper types. Our method generates a more comprehensive type set. In example (b), our method removes labels that are not highly correlated with this sample (assembly, gathering) and supplements two fine-grained labels (games, championship). In example (c), our method retains the correct label "/location/city" and removes the false positive label "/organization/company". In example (d), our method reassigns the correct label for this sample.

4.5 Visualization

To better illustrate how noisy labels affect model performance in FET, we train the model using the original and denoised training data separately and visualize mention embeddings in Figure 4 1 . In Figure 4(a), there is overlap between different type clusters. Intuitively, noisy labels in training sets prevent the model from learning a clear decision boundary for each type. [Pleiss et al., 2020] offers an explanation for this. Take a mislabeled sample for example. The gradient updates from those similar but correctly labeled samples encourage the model to predict the underlying ground truth through generalization. However, the gradient update from this mislabeled sample itself forces the model to memorize the incorrectly assigned label. These two opposing gradient updates result in a poor margin between different types. Consequently, the model is easy to make incorrect predictions for samples near decision boundaries. From Figure 4(b) we can see the margin between different types increases and the decision boundaries are clear. This phenomenon shows correcting the noisy labels brings more discriminative sample representation in FET, and our method indeed generates cleaner training data.

1 Some mentions may have multiple coarse-grained types, we only visualize mentions with one coarse-grained type.

5 Conclusion

In this paper, we focus on the noisy labeling problem in the fine-grained entity typing task and propose a novel approach to automatically correct the noisy labels without auxiliary resources. Experiments on two benchmark datasets prove the effectiveness of our method.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grant No.61602197, Grant No.L1924068, Grant No.61772076, in part by CCF-AFSG Research Fund under Grant No.RF20210005, in part by the fund of Joint Laboratory of HUST and Pingan Property & Casualty Research (HPL), and in part by the National Research Foundation (NRF) of Singapore under its AI Singapore Programme (AISG Award No: AISG-GC-2019-003). Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not reflect the views of National Research Foundation, Singapore. The authors would also like to thank the anonymous reviewers for their comments on improving the quality of this paper.