Hierarchical Modeling of Label Dependency and Label Noise in Fine-grained Entity Typing

1 Introduction

Fine-grained Entity Typing (FET) [Ling and Weld, 2012] aims at assigning proper fine-grained type labels for entity mentions in a sentence. The finely annotated mention type is often utilized to facilitate many downstream natural language processing (NLP) tasks, such as question answering [Dong et al., 2015], semantic parsing [Yavuz et al., 2016] and information extraction [Koch et al., 2014; Ling et al., 2015; Sch¨utze et al., 2017].

Existing FET models face two challenges: modeling the type hierarchy dependency and reducing the label noise. Type hierarchy (or Taxonomy) is a natural and practical structure that used to organize the type labels (Concepts). Figure 1 illustrates a type hierarchy sampled from a FET data set. In the figure, type PERSON and its three sub-types ACTOR, AUTHOR and DIRECTOR form a hierarchical structure. Several methods have been presented to model the type hierarchy dependency, including hierarchically embedding method [Ma et al., 2016], heuristic normalization loss [Xu and Barbosa, 2018] and hierarchical structure loss [Murty et al., 2018]. However, these models are limited by: (1) only modeling the parent-child relation, ignoring other relaitons (e.g. sibling). (2) using the same bilinear matrix or penalty hyper-parameter to model the correlations between different subtypes and their parent types. The AFET model [Ren et al., 2016] attempts to tackle these limitations by utilizing adaptive margins between types. The margin of two types is computed through their normalized number of shared entities in Knowledge Graph (KG). Nonetheless, the incompleteness and noisy annotations of KG may make those margins unreliable.

∗Corresponding author: zhangrc@act.buaa.edu.cn

Label noise comes with the unreliable data acquisition processes such as distant supervision (DS) method [Mintz et al., 2009]. The DS method assigns all the possible types of the entity for its mentions in the corpus. However, an entity mention may belong to different types in different contexts. As is shown in Figure 1, the phrase Demi Lovato may refer to PERSON in a general sense, and refer to PERSON.ARTIST in a specific sense. But type ORGANIZATION can not be inferred from the given sentence, thus turns to a noisy type. Such noisy types may degrade the performance of the FET models. Although various approaches [Ren et al., 2016; Abhishek , 2017; Xu and Barbosa, 2018; Zhang et al., 2020] have been proposed to handle these noises, they ignore the fact that the noisy labels are structural. For instance, (PERSON.ACTOR, PERSON.SINGER) is a set of fine-grained types, where PERSON is in a higher level, and ACTOR/SINGER are in a lower level. NDP [Wu et al., 2019] considers such structural information by assigning type weights through random walking in the mention level. However, it provides too much freedom in learning, thus may potentially risk over-fitting.

In this paper, we present a novel FET framework that jointly model the hierarchical dependency and the label noises of the types. This framework consists of a forward tree module (FTree) and a backward tree module (BTree). For modeling type hierarchical dependency, we first transform the pre-defined type hierarchy into the normalized type hierarchy that each leaf node represents a valid type. Then, the FTree module is proposed to compute the distribution of the type labels through random walking on the type hierarchy.

To model the label noises in other research topics, previous works [Sch¨utze et al., 2017; Luo et al., 2017; Goldberger and Ben-Reuven, 2017] learn the transformation from the true labels to the observed labels. However, these models ignore the inherentent hierarchical structure. We provide an alternative that parameterizes this transformation via the random walk process on the type hierarchy tree. we formulate the perturbation process with the BTree module. It learns the transformation from the latent true labels to observed noisy labels by backward randomwalking on type hierarchy. It parametrizes the confusion matrix of the type labels under the assumption that a label is likely to be perturbed into its nearby label in the tree. This approach puts an additional constraint on the confusion matrix according to the structure of the type hierarchy, thereby restricting the model capacity.

The empirical studies confirm the capability of our model in modeling the type hierarchy and discovering the true labels from the noisy data. The extensive ablation studies verify the effectiveness of each component of our model.

2 RelatedWork

2.1 Hierarchical Type Modeling

The most earlier models ignore the type hierarchy and treat this problem as a multi-label classification task [Ling and Weld, 2012; Yogatama et al., 2015; Abhishek et al., 2017]. Recently, some studies [Yosef et al., 2012; Ren et al., 2016; Shimaoka et al., 2017; Xu and Barbosa, 2018; Murty et al., 2018; Chen et al., 2020] have made efforts to explicitly model the correlation among hierarchical types. These models, although demonstrating some successes, rely on additional loss function constraints and some heuristic procedures to walk along the type hierarchy to find the predicted type. In this paper, we propose a generic method that directly models the hierarchical structure without additional steps for inferring or loss functions for model training.

2.2 Label Noise Handling

Various methods have been proposed to deal with noisy training samples. Specifically, [Gillick et al., 2014] refined the training data by applying a set of heuristics. This method, however, reduces the size of the training set and leads to performance degradation, as seen in [Ren et al., 2016].

Recently, many methods have been proposed to avoid the deletion of training samples or pruning type hierarchy. Some works [Yogatama et al., 2015; Ma et al., 2016] treat clean and noisy entity mentions equally and utilize ranking loss to attenuate the effects of extraneous labels. However, some studies [Ren et al., 2016; Abhishek et al., 2017; Xu and Barbosa, 2018] deal with the problem by modeling noisy and clean entity mentions separately and force the model to choose the most relevant type as the true label for noisy samples during the training process. NDP [Wu et al., 2019] proposed to weight out the noisy type during the training process. NFETC-AR [Zhang et al., 2020] directly estimated the pseudo-truth type distribution from noisy type distribution by minimizing the KL-divergence.

Although existing models have shown effectiveness in handling label noise, the noise detection and the hierarchical classification are modeled separately. In addition, the cleaned data should be feed-backed to the original model to improve the model trained from the noisy data. We argue, in this paper, these facts should be considered when building FET models.

3 Model

3.1 ProblemDefinition

In this section, we formally describe the fine-grained entity typing (FET). Given a pre-defined type label set with a hierarchical structure T and training data D = {(si,mi, τi), i ∈ {1, 2, · · · ,N}}, where si denotes a sentence, mi represents an entity mention, and τi denotes the noisy type label set. The objective of the FET task is to train a predictive model on D that is capable of predicting the true type for any sentencemention pair (s,m), despite the fact that D is noisy.

For convenience, we denote a (s,m) pair by random variable X, and denote the true label by random variable Y .

3.2 Sketch of Overall Model

Overall our proposed model assumes the following generative process from X to Z. Assume that a random walker is to walk on the type hierarchy tree T from the root. It assumes a probabilistic walk strategy generated from X. Following the strategy, at each node of T, it may keep going down along a downstream branch or may choose to stop. Whenever it stops, the node it lands on is the true type Y . We will call this walk "informative walk".

But the true type Y can not be directly observed. It then assumes another random walk strategy. At each node of T, it may walk along any direction (even going back) or choose to stop following this strategy. The node it stops at is then the observed noisy type Z. We will call this walk "erroneous walk". The process of informative walk and erroneous walk is formulated as follows:

In this paper, we assume the erroneous walk process is independent with input sentence x, we have

Note that the move from the true type Y to the noisy label Z can be aggregated to a transition matrix (also called confusion matrix) A. Choosing a strategy for the erroneous walk is equivalent to parameterizing the matrix A. Then we can get:

where PY |X(·|x) is the conditional distribution of Y given X = x written as a vector, and PZ|X(·|x) is defined as similarly. We postpone defining the loss function later.

To define and implement this model, we first introduce a "normalized type hierarchy" from previous work [Wu et al., 2019] on which informative walk and erroneous walk can be interpreted as walking forward and walking backward.

3.3 Normalized Type Hierarchy

Let T denotes the type hierarchy of the given entity mention. Each node in T denotes a token from a type name. For example, an entity mention could belong to PERSON or PERSON.ACTOR, where these types are in the different levels of the hierarchy T . Thus, models must take necessary measures to predict the proper grained type. In this paper, we use the same normalized type hierarchy as the previous work [Wu et al., 2019]. Specifically, we transform a hierarchical structure T into the normalization type hierarchy T by adding a stop node as the child of each non-leaf node. For example, after type hierarchy normalization, node PERSON T is followed by a new node PERSON.PER-STOP, as shown in the middle picture of Figure 2. Normalizing type hierarchy allows every type being mapped to a unique leaf node in T , thus maintains the information of the original type in its hierarchy level.

For later use, we denote a node in the normalized type hierarchy as u ∈ T , and denote a type as t. Each type t is a path from the root to a leaf node in T . From here on, whenever we speak of a type, we refer to its leaf-node representation in T .

3.4 Feature Embedding

Give the (s,m) pair, we utilize the neural network proposed in NFETC [Xu and Barbosa, 2018] to extract the feature embedding (s,m) as described in the following.

Given the entity mention m = {wp, · · · ,wt}, the average feature m is the average word embedding of the word in m. Moreover, the LSTM is applied to the extended mention m∗ = {wp-1, · · · ,wt+1} and generates the outputs {hp-1, · · · , ht+1}. The last output ht+1 is regarded as the LSTM representation h(m) of m.

The Bidirectional LSTM combined with a word-level attention module is applied to the context sequence c = {wp-k, · · · ,wt+k} to generate the context feature h(c), where k is the context window size.

Then, the final feature embedding is the concatenation of the above three features: x = (s, m) = [m, h(m), h(c)].

3.5 Modelling InformativeWalk

During the informative walk, the random walker starts walking from the root node and is allowed to only stop on a leaf node. For each node u ∈ T , we define its token embedding as u ∈ Rdu×1 . Then we measure the similarity between an input x and a node u via a score function

where W ∈ Rds×du denotes a trainable matrix.

To compute the weights of edges in T , we introduce a notation Hu to denote the set of all child nodes of node t together. Then the probability of the informative walker walking from a father node ui to a child node uj is calculated as

Given a type path t = {u0, u1, · · · , uL}, we then define the the probability of the walker arriving at t given x as

where L is the depth of T and u0 is the root node. Then the true probability of type distribution given the input x is defined as:

where J denotes the number of all types.

The informative-walk component is referred to as "Forward Tree" or FTree.

3.6 Modelling Erroneous Walk

We next describe the "backward tree" or BTree, which models the perturbation process using the erroneous walk. It approximates the confusion matrix A that transforms the true labels Y to the noise label Z.

Backward Tree Construction

The backward tree is constructed for every type to explicitly modeling backtracking trace from a leaf node to another leaf node. Figure 2 shows a backward tree for a random walker starting from the leaf token of type path "PERSON.ACTOR".

The root of backward tree for type ti = {u(i) o , · · · , u(i) L } is the leaf token u(i) L . For convenience, we virtually add a blue square token to denote keeping staying in this leaf token. When the randomwalker lands on a leaf token, it has only two choices, staying there or back-walk to its parent token. When it goes back to its parent, in the next step, it can continue to go back to the parent token or a child token of the current token until reaching the leaf token u(j) L . This process models the type perturbation from type ti to type tj.

There are five types of backtracking edges: (1) The edge from the starting token to a square token; (2) The edge from a token to its father token; (3) The edge from a token to its child token, (4) the edge from the level 1 token (such as PERSON) to another level 1 token (such as ORGANIZATION) through dummy token and (5) The edge from a non-starting leaf token to a square token. For each type of backtracking edge, we use a learnable parameter to describe the tendency that the random walker walking through this edge. Specifically, as shown in the right picture of Figure 2, the parameter of the first four types of backtracking edges are S, F, C, and M, and we fix this probability for the fifth edge as one to restrict a random walker from backtracking again. Note that for the level 1 type token (such as PERSON, ORGANIZATION), its father token is a dummy token which has no corresponding type token. It forces the random walker not to stop and continue walking to one of its children tokens.

Type Confusion Matrix Generation

The backtracking process can be expressed as a type confusion matrix A ∈ RJ×J , where J is the number of all types. To model the confusion between type ti and tj , we imagine an erroneous walker on the backward tree Ti starts from type ti's leaf node and stop on tj's leaf node. The probability of the erroneous walker lands on tj leaf token is exactly the element Aij in the confusion matrix A. To compute Aij, we define the probability of the erroneous walker which starts from u(i) L walking through edge r(ui′ , uj ′) from a father node ui′ to a child node uj ′ is calculated as:

where Hbui′ denotes the set of all child nodes of node ui′ and κr(ui′ ,u) ∈ {F(i) , C(i) , S(i) , M(i) , 1} denotes the weight that erroneous walker of type ti chooses edge r(ui′ , u). These weight parameters are randomly initialized and updated during training. The confusion probability between type ti and type tj is then defined as:

where Ri→j denotes the edge set from type ti to type tj . Similarly, we can get the confusion probabilities among all types.

The whole confusion matrix is computed as:

where the parameter ϵ ∈ (0, 1) is a hyper-parameter that restricts the the training process from over-fitting.

So far, PZ|X is well defined through Equation 3. It is noteworthy that the confusion matrix generated by BTree is highly constrained and independent from the FTree. Apart from FTree, it can be applied to other mention-to-type compatibility models.

3.7 Loss Function

With PY |X defined via the FTree and PZ|Y or (matrix A) defined by BTree, the model is naturally optimized by minimizing the negative log-likelihood of the training set. We define the loss function as follows:

where z∗ i = argmaxz∈τi PZ|X(z|xi) and N is the total training sample numbers. When Y is completely hidden the model, such a loss is non-identifiable, and there are uncountable solutions for PY |X that will equally minimize the loss. To fix this, we proposed following revision of loss function:

where N denotes the noisy training sample set, in which training example may have been labeled with more than one type, and C denotes the clean training sample set that means a training example is labeled only with one type, yi is the only type label in clean type set τi. In this loss function, we essentially have assumed that the examples in C have clean labels. In the other words, we assume that the sample with only one type path is labeled with the true type. This will allow the model to anchor the solution of PY |X around those compatible with the examples in C.

3.8 Type Inference

In the training process, we jointly train the FTree and BTree. We denote the joint model of FTree and BTree by FBTree. During type inference, we directly utilize the type probability distribution of FTree as the predictive probability. The type t with the highest probability is then declared as the true type.

4 Experiment

4.1 Datasets and Settings

Three benchmark datasets, Wiki, OntoNotes, and BBN are used to evaluate our proposed models. We utilize the preprocessed data set provided by [Ren et al., 2016]. Following the standard protocol adopted in the previous works [Ren et al., 2016; Abhishek et al., 2017; Shimaoka et al., 2017], we split 10% instances from each test set as the validation set. Table 1 shows the statistics of the three datasets. As shown in Table 1, nearly 30% of the training samples contain noisy labels among all datasets.

In our implementation, Glove embedding [Pennington et al., 2014] is exploited to initialize the word embeddings and they are fixed during the training process. We utilize Adam [Kingma and Ba, 2014] as the model optimizer. An early-stop strategy is employed to terminate training when the loss on validation set is stable. Dropout [Srivastava et al., 2014] and L2 regularization are used to alleviate overfitting. We determine the optimal hyper-parameters by random search on the valid dataset. The main hyper-parameters are shown in Table 3. Different datasets may set different hyper-parameters because they vary in structures and scales.

4.2 Predictive Comparison

We compare our proposed model FTree, FBTree with various recent FET models: ATTN [Shimaoka et al., 2017], ABH, ABH-AllC [Abhishek et al., 2017], NFETC [Xu and Barbosa, 2018], NDP [Wu et al., 2019], NFETC-AR [Zhang et al., 2020] and MLL2R [Chen et al., 2020]. We take the evaluation results of these baselines from their original papers.

Three standard metrics [Ling and Weld, 2012], Accuracy (Acc), Micro-averaged F1 (Mi-F1) score and Macro-F1 (MaF1) score, are used to evaluate the performance of models.

The overall performances are presented in Table 2, where the best value is highlighted in bold. We observed that our FBTree model outperforms all compared models on Wiki and BBN and achieves comparable performance with the SOTA model on OntoNotes. These results are desirable as we note the following. The testing set of BBN is considered noiseless, or as the true type probability. But the testing sets of Wiki and OntoNotes are in fact noisy, somewhat inconsistent from the ground truth. This makes our model perform better on BBN.

It is worth noting that adding BTree module can significantly improve the performance of the FTree module. It also achieves more stable performance across all datasets than all other models.

4.3 Performances at Different Levels of Type Hierarchy

We further investigate the performances of the compared models at each level of the type hierarchy. The depth of type hierarchy for Wiki, OntoNotes, and BBN are 2, 3, and 2. The results are given in Figure 3. We find that with the increasing of the label level of the tested entities, the model performance on Wiki and OntoNotes is decreasing. However, the performance of all compared models on BBN does not varies significantly. This phenomenon might be caused by the level distribution in the training set. It can also be observed that FBTree achieves the best performances at all levels of all datasets. This study confirms the superior of FBTree.

4.4 Ablation Study

Effectiveness of informative walk. We conduct experiments to evaluate the effect of our mention-to-type compatibility model FTree. We compare our FTree model with ComplEx that utilizes complex bilinear mappings to capture hierarchical information [Murty et al., 2018]. The results in Table 4 indicates that FTree is more effective than ComplEx in modeling label dependencies.

Effectiveness of erroneous walk. We further analyze the results of our BTree model. As shown in Table 4, we compare the type confusion matrix leaned by the highly constrained model BTree with FLAT and TypeCorr. FLAT denotes that confusion matrix is random initialized and TypeCorr regard the normalized number of shared entities in KG between types as the type confusion weight [Ren et al., 2016]. From the table, we observe that the type confusion matrices learned by the two baselines may not bring improvements on all data sets. But our BTree model brings significant performance gains on all data sets. This experiment result confirms that BTree is efficient in noise detection.

4.5 Analysis

We make an error analysis on the FTree and FBTree model. As shown in Table 5, we classify errors into four types: all wrong (AW), partial wrong (PW), coarse to fine (C2F), and fine to coarse (F2C). The results show that FBTree decreases AW and C2F errors by about 1%, 0.7%, with a slight increasing in PW and C2F on the Wiki dataset. It also decreases AW and F2C by about 1% with a slight increasing in PW and F2C on OntoNotes and BBN. This error analysis confirms that BTree can effectively detect noises and confirms the motivation of our proposed model.

Three noisy examples in the BBN data set are used to demonstrate the effectiveness of FBTree. As shown in Table 6, BTree tends to produce low label scores in all samples. For sample II and III, the classifier can detect the two noisy types by their lower score than the most appropriate type. We also find that it is possible for the distributions PZ|X and PY |X to have different polarity e.g. sample I. This study indicate that FBTree tries to reduce the confidence of the classifier on noisy samples, thus has superior generalization.

5 Conclusion

In this work, we identify that the entanglement of type hierarchy and noisy labels is a key obstacle for developing a highperformance predictive FET models. We tackle this problem by separately modelling the two aspects using a forward random walk on the type hierarchy and a backward random walk on a reversed tree. The two components are then used to jointly model the noisy hierarchical types that are observed in the datasets. Interestingly, we found that the random walk process is able to estimate the true entity type more than we expected, by evaluating the importance of the noisy entity types through the walk process. The empirical study confirms the effectiveness of our proposed model, outperforming recent models on benchmark datasets.

It is also worth noting that the modelling techniques presented in this work can be extended to model labels with hierarchical structure and observation noise. We hope this work may inspire more exploitation of such ideas to much broader scope of applications.

Acknowledgments

This work is supported partly by the National Natural Science Foundation of China (No. 61772059), by the Fundamental Research Funds for the Central Universities, by the Beijing S&T Committee (No. Z191100008619007) and by the State Key Laboratory of Software Development Environment (No. SKLSDE-2020ZX-14).