Beyond Word Attention: Using Segment Attention in Neural Relation Extraction

1 Introduction

There has been significant historic interest in relation extraction (RE), which aims to extract semantic relationships between two target entities from plain text. Regarding such task as a simple text classification problem is undesirable because of the inner-sentence noise [Liu et al., 2018]. To explain the influence of word-level noise, we consider the sentence in Figure 1 as an example. The sub-sentence "Edsel Ford, the only child of Henry Ford" keeps enough words to express the relation children, and the other words could be regarded as noise that may hamper the extractor's performance.

To alleviate the influence of word-level noise within sentences, many efforts have been devoted to get rid of irrelevant words [Xu et al., 2015; Zhang et al., 2017; Zhang et al., 2018; Liu et al., 2018], especially the recent state-ofthe-art attention-based methods [Zhang et al., 2017; Lee et al., 2019]. Specifically, current attention scheme used in RE computes the attention score for each word to indicate how well the word can express the relation between the two entities. This mechanism can be viewed as the process of performing soft selections of individual words independently, regardless of the rich dependencies among the words that describe the relation.

∗Corresponding Author

As we can see from the example given in Figure 1, the relational expression associated with target entities may be in the form of a segment structure, where the segment refers to a consecutive subsequence of words in sentence. The phrase "child of " is the relational expression of entity pair (Edsel Ford, Henry Ford) and represents the relation "children". It is difficult for model to distinguish between "children" and "parents" if its attention layer only focuses on the individual word "child" rather than the segment "child of ". Furthermore, we sample out 200 examples from the standard dataset and annotate the relational expression of each sentence manually. We find that half of the relational expressions are in the form of segment and longer than 2 words, which means that accurately extracting and modeling such segment information can be extremely crucial. While one might argue that these dependencies can be learned implicitly by a deep model with substantial amount of data, we believe it is still useful to provide such shallow structural information as prior knowledge.

To capture such relational expressions, we incorporate a layer that is analogous to conditional random fields (CRF) [Lafferty et al., 2001] in the attention modeling process. As a result, our attention mechanism could provide a probabilistic framework for calculating the weights of words globally conditioned on the full sentence. In other words, the weight of one word is expected to impact on the weights of neighboring words. This can be viewed as an extension of the standard attention mechanism, and we call such a novel attention mechanism segment attention in this paper.

Our approach views the attention mechanism as a linearchain CRF over a set of latent variables whose edges encode the desired structure. Concretely, we assume that there is a one-to-one match between variables and words. Each binary variable indicates whether its corresponding word is part of a relational expression or not (two states in our problem: selected word and non-selected word). Then we regard the marginal distribution of each word being selected as the attention weight. This distribution can be calculated efficiently in linear-time using the forward-backward algorithm. Furthermore, we introduce two additional regularizers to ensure our model attends to continuous regions. The first regularizer, which named as transition regularizer, discourages frequent transitions between different states and aims to achieve continuous identical states. The second regularizer called sparse regularizer tries to focus on few words that really matter and return sparse weight distribution.

Specifically, our model consists of four layers: a positionaware input layer aims to take position information into consideration, a BiLSTM layer [Graves et al., 2013] that runs through the words in the sentence sequentially to get contextual information for each word, and a segment attention layer works as parameterized pooling to distill the relation information and learn a representation of the given sentence, which is fed to the final classification layer.

To summarize, our contributions are as follows:

• ÷ We propose a novel segment attention based sequence model (SA-LSTM) for RE task, which is capable of learning relational expressions and capturing dependencies between target entities and their relations.
• ÷ Experiments are conducted on the TACRED dataset. Results show that our model achieves the state-of-the-art performance on the fully-supervised RE task.
• ÷ We conduct qualitative analyses to understand how our model works with the help of segment attention, including evaluation of the extracted relational expressions.

2 RelatedWork

There are several studies for solving relation extraction task. Early methods used handcrafted features through a series of NLP tools or manually designing kernels [Rink and Harabagiu, 2010]. These approaches use high-level lexical and syntactic features obtained from NLP tools and manually designing kernels, but the classification models relying on such features suffer from error propagation of the tools.

Recent studies have found neural models effective in relation extraction, deep neural networks have outperformed previous models using handcraft features. Zeng et al.[2014] employed a deep convolutional neural network for extracting lexical and sentence level features. Santos et al. [2015] proposed model for learning vector of each relation class using ranking loss to reduce the impact of artificial classes. Zhou et al. [2016] used bidirectional recurrent neural network to learn long-term dependency between entity pairs.

Apart from neural models over word sequences, incorporating dependency trees into neural models has also been shown to improve relation extraction performance by capturing long-distance relations. Xu et al. [2015] generalized the idea of dependency path kernels by applying a LSTM network over the shortest dependency path between entities. Liu et al. [2015] first applied a recursive network over the subtrees rooted at the words on the dependency path and then applied a CNN over the path. Zhang et al. [2018] applied a combination of pruning strategy and graph convolutions to the dependency tree. The resulting model achieved best performance on the TACRED dataset.

More recently, some researchers have proposed attentionbased models which can focus to the most important semantic information in a sentence. Zhou et al. [2016] combined attention mechanisms with BiLSTM. Xiao and Liu [2016] splited the sentence into two entities and used two attention-based BiLSTM hierarchically [21]. Wang et al. [2016] proposed attention-based CNN using word level attention mechanism that is able to better determine which parts of the sentence are more influential. Zhang et al. [2017] employed a positionaware attention mechanism over LSTM outputs, and showed that it outperforms several CNN and dependency-based models by a substantial margin. Du et al. [2018] proposed a 2-D matrix-based attention mechanism, which contains multiple vectors, each focusing on different aspects of the sentence. Our model is inspired by structural attention network [Kim et al., 2017] which extends the standard attention to directly model structural dependencies between nearby input elements. Wang and Lu [2018] also applied this sort of architecture to aspect-based sentiment analysis. Different from previous attention model designed for relation extraction, our model is capable of learning phrase-like features and capture reasonable segments as relational expressions.

3 Methodology

Figure 2 gives an illustration of our SA-LSTM model. Next, we detail all components sequentially from bottom to top.

3.1 Input Layer

The input layer of the sentence encoder aims to embed both semantic information and positional information of words into their input embeddings.

Word embedding is able to capture the meaningful semantic regularities of words [Turian et al., 2010]. We use pretrained dw-dimensional word embeddings [Pennington et al., 2014] as the basic features.

Position embedding is proposed by [Zeng et al., 2014], which is used to embed the relative distances of each word to the two entities into two dp-dimensional vectors. By concatenating the distance embeddings for the current word wi to the both head and tail entities, we get a unified position embedding pi ∈ Rdp×2 .

For each word wi, we concatenate its word embedding wi and position embedding pi to build its input embedding xi = [wi; pi] ∈ Rdw+dp×2

3.2 BiLSTMLayer

A BiLSTM layer is adopted to capture the contextual information for each word. For simplicity, we denote the operation of an LSTM unit on xi as LSTM(xi). Thus, the contextualized word representation is obtained as follows:

where hi ∈ R2×dh and dh indicates the dimension of hidden state for LSTM. In doing so, we can efficiently make use of past features (via forward states) and future features (via backward states) for a specific time. Moreover, we use H = ¶h1↪ h2↪ · · · ↪ hn♦ to denote all the word representations generated for the input sentence.

3.3 Segment Attention Layer

Based on the observation that it is usually a segment rather than individual words scattered in the sentence that form meaningful information, we incorporate a segment attention layer to perform soft selections of a sequence of words.

Similar to the standard attention used in RE, our segment attention is also a linear combination of the input representations where the weight scalar is between [0, 1] and represents how much attention should be focused on each input. The key difference between our scheme and previous attention mechanism used in [Zhou et al., 2016] and [Zhang et al., 2017] is the calculation of attention weights. Specifically, we introduce a discrete latent binary variable z ∈ ¶0↪ 1♦ for each word. This variable indicates whether its corresponding word is part of a relational expression or not. Under this definition, the representation of the given sequence m is defined as the expectation of hidden states with the probability that its corresponding word is selected. Equation 2 gives a general form of this function.

In order to derive p(zi = 1♣H), we incorporate linear-chain conditional random fields (CRF) to specify the dependencies between these latent variables. For a random variable over data sequences H, and a random variable over corresponding label sequences z, CRF provide a probabilistic framework for calculating the probability of z globally conditioned on H [Lafferty et al., 2001]. H and z may have a natural graph structure. Formally, we use H = ¶h1↪ h2↪ · · · ↪ hn♦ to represent a generic input sequence where hi is the BiLSTM hidden state of the i-th word. z = [z1↪ · · · ↪ zn] represents a generic sequence of labels for H. The probabilistic model for sequence CRF defines a family of conditional probability p(z♣H) over all possible label sequences z given H with the following form:

where Z denotes the set of possible label sequences z, Z(H) is the normalization constant that makes the probability of all state sequences sum to one, zc indicates the subset of z given by individual clique c and ψ(zc↪H) is the potential function of this clique defined as:

We define two types of feature functions: vertex feature ψ1 and edge feature ψ2. Vertex feature ψ1(zi↪ hi) represents the mapping from the input hi to output zi through a single layer neural network. Edge feature ψ2(zi↪ zi+1) models the transition from i-th state to i+1-th for a pair of consecutive time steps. Note that this transition matrix is position independent.

Here Wv ∈ R2×2dh maps context representation to the feature score of each latent state, Wt ∈ R2×2 is a transition matrix defined for each pair of latent state andWtzi↪zi+1 is the transition score from zi to zi+1.

As mentioned above, our purpose is to compute the marginal probability p(zi = 1♣H) at each position in the sequence which can be computed by a dynamic programming inference procedure similar to the forward-backward procedure for HMM [Lafferty et al., 2001]. We can define the "forward values" of the i-th timestep αi(z♣H) by setting α1(z♣H) equal to the probability of starting with state z, z ∈ ¶0↪ 1♦, and then iterate as follows:

The "backward values" βi(z♣H) can be defined similarly. After that, we calculate the marginal probability of each word being a part of the relational expression given the whole sentence sequence by:

Thus we can compute the final representation m used for classification by combining both Equation 2 and Equation 9. Furthermore, given the conditional probability of the state sequence defined by a CRF in Equation 3, we can also obtain the most probable labeling sequence using a Viterbi decoding algorithm, corresponding to the word sequence with a label of 1 denoting its corresponding word is part of a relational expression and 0 denoting the non-informative word. In other words, the latent relational expressions can be extracted explicitly by maximizing Equation 3, which allows us to have an intuitive understanding of our model behavior and evaluate segment attention from another aspect.

3.4 Classification Layer

To compute the output distribution p(r) over relation labels, a linear layer followed by a softmax is applied to the representation m, which represents a summary of the input sequence:

where Wr ∈ R2×dh maps the relation vector m to the feature score for each relation label and br is a bias term.

3.5 Objective Function

After incorporating the segment attention into the BiLSTM, our final model is illustrated in Figure 2. The attention-based BiLSTM component is associated with the cross entropy loss of relation extraction. The loss function is given below:

In addition, based on the observation that relational expressions are usually segments rather than disconnected words, we hold that frequent transitions between different states should be discouraged. Inspired by [Wang and Lu, 2018], we introduce the transition regularizer to encourage the state to stay the same.

Specifically, it enforces the transition feature value between different states to be smaller than the one between the same state. The second regularizer tries to enforce the model to attend to few words that really matter and return sparse weight distribution:

The final objective function of our model is defined as:

where λ1 and λ2 are hyperparameters that control the weights of each regularizer.

4 Experiments

4.1 Dataset andMetric

We conduct experiments on the recently widely used benchmark TACRED dataset introduced in [Zhang et al., 2017], which is the currently largest supervised dataset for relation extraction. It contains over 106k entity pairs collected from the TAC KBP evaluations 2009-2014. TACRED includes 41 relation types and a special no relation class indicating that the relation expressed in the sentence is not among the 41 types. Entities in TACRED are replaced by corresponding entity types, subjects classified into person and organization, and objects categorized into 16 fine-grained classes (e.g. date, location, title). We evaluate the models using the official scorer in terms of the Macro-F1 score. For fair comparisons, we report the test score of the run with the median validation score among 5 randomly initialized runs following the evaluation protocol used in [Zhang et al., 2017].

4.2 Implementation Details

Following popular choices and previous work, we employ the "entity mask" strategy where we replace each subject (and object similarly) entity with a special SUBJ-<NER> token. we also adopt the "multi-channel" strategy by concatenating the input word embeddings with part-of-speech (POS) and named entity recognition (NER) embeddings. We run Stanford CoreNLP [Manning et al., 2014] to obtain the POS and NER annotations. These strategies are also used by the methods that we compare against.

We use the 300 dimension Glove embeddings [Pennington et al., 2014] to initialize word embeddings. we randomly initialize the POS, NER and position embeddings with 30dimension vectors, by drawing from a normal distribution with θ = 0▷0 and σ = 0▷01. Dropout with p = 0▷5 used after the input layer and before the classifier layer. λ1 and λ2 are chosen from [0,0.2] via grid search. For LSTM, we set the hidden dimension size to 300 and use 2-layer stacked BiLSTM. The model is trained using stochastic gradient descent for 30 epochs with the initial learning rate of 1 and the weight decay of 0.5. All the hyper-parameters are tuned on the validation set.

4.3 ComparisonModels

We compare our model against the following baseline models for relation extraction.

Pattern: The core component of TAC KBP 2015 winning system, which uses a total of 4,528 surface patterns and 169 dependency patterns to extract relations [Angeli et al., 2015].

LR: It is trained on 2 million bootstrapped examples and uses a comprehensive feature set [Zhang et al., 2017].

CNN-PE: It builds a convolutional model to learn sentence features, and uses position vectors to indicate the relative distances of current word to two entities [Zeng et al., 2014].

PCNN: Based on CNN-PE, it replaces the max-pooling operation with piece-wise max-pooling and achieves improved results [Zeng et al., 2015].

SDP-LSTM: It applies a neural sequence model iteratively along the shortest dependency path between target entities [Xu et al., 2015].

Tree-LSTM: It is a recursive model [Tai et al., 2015] that generalizes the LSTM to arbitrary dependency tree structures.

PA-LSTM: It employs a position-aware attention mechanism over LSTM outputs, and outperforms several CNN and dependency-based models [Zhang et al., 2017].

C-GCN: It applies a combination of pruning strategy and graph convolutions to the dependency tree, which is a stateof-the-art on the TACRED dataset [Zhang et al., 2018] .

SA-LSTM: Segment attention layer is employed on top of the LSTM. This is the main model of this paper.

Note that C-GCN proposed a path-centric pruning strategy to empirically remove irrelevant content [Zhang et al., 2018]. To fairly evaluate models, we also implement PA-LSTM augmented with the shortened sentences used in C-GCN, namely PA-LSTM+D. In a similar way, we can get the SA-LSTM+D.

4.4 Results

Experimental results are shown in Table 1. From the results, we can observe that: (1) With the same pruned input, our model outperforms state-of-the-art method with a relative improvement of 1.2%, which indicates that linear-chain CRF can work well as a attention mechanism for relation extraction. (2) SA-LSTM achieves higher precision and recall than PA-LSTM, which shows that modeling the dependencies between adjacent words can improve the model performance. (3) Given more precise shortened sentences, our SA-LSTM model still significantly outperforms the PA-LSTM, indicating that our proposed model can consistently benefits from the more precise input. (4) Comparing SA-LSTM+D with CGCN, we can see that the gain mainly comes from improved recall. We hypothesize that this is because the C-GCN may suffer from the parser errors by modeling the tree structure directly while SA-LSTM+D just use the dependency tree to remove irrelevant words.

In addition, though pattern-based method also uses phrase patterns to capture the relation expressions which is similar to our motivation, SA-LSTM outperforms Pattern significantly. It shows that human-designed features are very limited as compared to neural models.

5 Analyses

5.1 Ablation Study

To study the contribution of each component in the SA-LSTM model, we run an ablation study on the TACRED dev set (see also Table 2). From these ablations, we find that: (1) The entire segment attention contributes about 10.7% F1 score. (2) When we remove the position embedding and only use word embedding as input, the score drops by 3.3%, which indicates that it is important to let segment attention aware of position information. (3) Removing the transition regularizer hurts the result by 1.1% F1 score. The performance also slightly degrades without sparse regularizer. Intuitively, segment attention can naturally recall more instances since the model captures a sequence of words rather than individual words. However, high recall also results in low precision, so these two regularizers can help model balance between precision and recall. (4) Segment attention usually performs better when coupled with BiLSTM since it is easier to model the interactions if contextual information is encoded in hidden representation.

5.2 Case Study

We compare our method with PA-LSTM on some cases, as shown in Table 3. As demonstrated by the first example, PA- LSTM only focuses on the individual word "son" and cannot capture the full relational expression "the son of", so it is difficult for PA-LSTM to distinguish between relation children and parents. Thanks to the segment attention, our SALSTMmodel can successfully attend to the desired segment and make a correct decision.

In the second example, our proposed model successfully detects the relation phrase "a student at" while PA-LSTM only attend to the single word "at", lose the information that the subject entity is a student. Hence, it is not surprising that PA-LSTM wrongly marks this instance as relation employee of. From these examples, we can observe that the proposed model is capable of capturing shallow structural information so as to perform relation extraction.

To give people some intuitive sense about how our models perform, we sample out some instances and use Viterbi decoding algorithm to extract the relation expressions explicitly (Figure 3). As we can see from the first two examples, the segment attention precisely detects the subject entity, object entity and the relation phrase between them. The third example shows that the model can successfully attend to the desired segment even if it is not adjacent to two target entities. Sometimes, our model also fails to attend to the right span as the forth example shows. Our segment attention identifies "SUBJ-PER including his wife" as a relational expression because "his wife" is a highly confusing trigger phrase and close to the subject entity, thus the model tends to capture them as a whole while no relation hold between the entity pair.

5.3 Error Analysis

Although the proposed method outperforms the state-of-theart systems, we also observe several failure cases. The following is a typical example of a wrongly classified sentence: "the SUBJ-ORG's annual conference in OBJ-LOC OBJ-LOC". This sentence is wrongly classified as belonging to the no relation category, while the ground-truth label is stateorprovince of headquarters. The phrase "annual conference in" does not appear in the training data, and moreover is used metaphorically, making it difficult for the model to recognize the semantic connection.

Another common issue is that there are multiple relations in a sentence, such as the following ones: "SUBJ-PER, the only child of OBJ-PER and his wife, Rosaile". The model fails to attend to the right span because two relations hold simultaneously and their relational expressions are close to each other, so the attention mechanism tends to assign similar weights to the two phrases when extract the relation between SUBJ-PER and OBJ-PER. It would be interesting to see if designing more precise path pruning strategy can improve the performance since reducing the noise from input would hopefully further alleviate the burden of attention layer.

6 Conclusion

In this paper, we propose a novel model that learns the latent relational expressions based on the segment attention layer for relation extraction. By incorporating a linear-chain CRF into the attention layer, our model is capable of capturing the dependencies between target entities and their relations. Experiments on standard TACRED dataset show that our proposed model outperforms a strong feature-based classifier and all baseline neural models. We further compare the visualized attention of our model with the baseline model to show how segment attention layer affects the model. In the future, we will conduct research on how to design more sophisticated attention mechanism to address some of the existing challenges in relation extraction, such as multiple relations per sentence. The source code of this paper can be obtained from https://github.com/yubowen-ph/segment.

Acknowledgements

We would like to thank the anonymous reviewers for their insightful comments and suggestions. This research is supported by the National Key Research and Development Program of China (grant No.2016YFB0801003) and the Strategic Priority Research Program of Chinese Academy of Sciences (grant No.XDC02040400).