Attention-based Multi-level Feature Fusion for Named Entity Recognition

1 Introduction

Named entity recognition aims to identify entities that have similar characteristics from raw text and assign them identical tags such as PER (Person), ORG (Organization), LOC (Location), etc. As a fundamental task of information extraction, NER has been studied for various tasks, including partof-speech (POS) tagging, chunking, and semantic role labeling (SRL) [Collobert et al., 2011]. Considering the diversity and complexity of natural language, named entities generally have multi-features from different perspectives: characterlevel and word-level features from local and global perspectives, as illustrated in Figure 1. We can see that the named entity associated with polysemous words (Washington) should take the capitalization 'W', keyword 'in', and lexical phrase into consideration.

� Corresponding author: chenhc@jlu.edu.cn

Previous knowledge-based approaches for NER merely depend on handcrafted rules and domain-specific dictionaries to recognize named entities [Rahem and Omar, 2014; Quimbaya et al., 2016]. However, such an endeavor is manual and is thus prone to poor coverage. Traditional approaches use supervised machine learning algorithms that incorporate a wide variety of hand-crafted features. To alleviate the heavy manual effort associated with these approaches, neural models are proposed to learn the implicit features by utilizing word-level embedding [Huang et al., 2015], character-level embedding [Kuru et al., 2016] or both [Akbik et al., 2018]. However, these methods largely ignore or oversimplify the correlations among the features learned from different perspectives such as word-level and characterlevel. Although existing NER approaches based on the combination of word-level embeddings and character-level embeddings have achieved competitive results [Xin et al., 2018; Akbik et al., 2018; Beltagy et al., 2019], they pay little attention to fusing multi-features, which may lead to information omission. Taking the sentence in Figure 1 as an example, the polysemous word 'Washington' of 'Washington University' (ORG) might be mislabelled as B-PER without Char Global features by merely taking adjacent words or character features in context into consideration, which may result in 'Washington University' being classified as PER by mistake.

To this end, we propose an attention-based multi-level feature fusion framework for NER, where the multi-level semantic and syntactic features of a given input sequence can be simultaneously captured from different views, as shown in Figure 1. Inspired by the transformer network[Vaswani et al., 2017], we explicitly employ four components for different purposes, namely the local character-level component, global character-level component, local word-level component, and global word-level component to process the input sequences. Then the fusion representation from the multi-level features is fed into the BiLSTM-CRF network for the final prediction.

The main contributions of this work are summarized as follows:

• An attention-based multi-level feature fusion (AMFF) framework is proposed for NER, which enables the multi-level features from diverse word-level and character-level perspectives to be integrated. To the best of our knowledge, this is the first effort in the literature that uses attention mechanisms to capture multi-level features of text from different perspectives.
• In this framework, local character-level, global character-level, local word-level, and global word-level components are designed to capture the features from the perspectives of capitalization, inter-word relations, keywords, and lexical phrases, respectively. It simplifies the problem and improves interpretability.
• Extensive validations on real-world benchmark datasets compared with the state-of-the-art models demonstrate the superiority of our proposed AMFF. Systematic analyses show an in-depth understanding of each component and the robustness of the AMFF framework.

2 RelatedWork

Existing NER studies are primarily based on word embeddings, character embeddings, and other embedding combinations, which are summarized as follows.

The first word embedding-based NER approach [Collobert et al., 2011] adopted convolution neural networks (CNNs) to produce local features, and a conditional random field (CRF) layer to predict entity attributes, which achieved better performance than previous studies. In order to address the problem of long-distance dependency, the CNN layer was replaced by the bi-directional long short-term memory (BiLSTM) layer for the better selection of the global features [Huang et al., 2015]. Furthermore, the combination of both CNN and BiLSTM highlights the performance improvement for linguistic sequence labeling [Liu et al., 2018]. However, these methods ignore the effect at the character level, which may lead to the omission of important information.

Word representations can be obtained from character-level embeddings as each sentence can be regarded as a character sequence. A character-level convolutional neural network (CharCNN) [Kim et al., 2016] is proposed to extract local sub-word information, relying on LSTM to deal with context features and the softmax function for the final prediction, which highlights the character embeddings for NER. Moreover, character-level recurrent neural networks (CharRNN)

[Kuru et al., 2016] choose global features from the context and boost the performance of NER by using CRF. In addition, it achieves better performance in handling multiple languages [Dong et al., 2016]. However, methods based on character embeddings do not attach importance to word-level features, which may be biased as well.

Recent advances in NER demonstrate its great advantage in recognizing named entities based on the combination of embeddings [Akbik et al., 2018; Yoon et al., 2019]. In order to capture both global and local features, existing methods rely on more types of embeddings, for example, BERT-based methods [Beltagy et al., 2019] taking token embeddings, segment embeddings, and position embeddings into account for NER. Moreover, other additional information such as affix embeddings can also be used to augment the NER architecture [Yadav et al., 2018]. In addition, the multitask learning strategy [Zheng et al., 2019] that divides the original task into multiple subtasks offers a new approach for nested NER. Although these methods achieve competitive results, few of them explore the attention mechanism for multi-level feature selection in NER, which does not make full use of the complete information.

Therefore, different from the existing models that mine information from merely one aspect, our model focuses on leveraging multi-level features from different perspectives, which can obtain more types of information and make a comprehensive final prediction.

3 The Proposed Framework

In this section, we introduce a novel method called AMFF, which can obtain multi-level features by parallel components and fuse them for sequence labeling.

Given an input sentence S composed of a sequence of words F1F2 � � � F=, where = is the total number of words, we assign each word F8 with one label H8 that takes one possible class from the named entity label set: y = fB-ORG� I-ORG� E-ORG� O� S-LOC� B-PER� � � � g, where B-, I-, E-, S-, and O tags respectively indicate the beginning, intermediate, ending position of the entities, entity with a single word, and other types. ORG, LOC, and PER are categorical abbreviations of organization, location, and person, respectively. Therefore, we formulate it modeled as a sequence labeling problem, that is, 5 : F1F2 � � � F= ! H1H2 � � � H=. Figure 2 gives an overview of our proposed work, which is depicted in detail in the following subsections.

3.1 Embedding Layer

For a given input word sequence w, we represent each token in the sentence by adopting both word embedding and character embedding [Lample et al., 2016]. From a word sequence, we obtain the word embedding of the 8-th word as follows:

where 4 F denotes a pretrained word embedding lookup table. In addition, the embedding of each character within the 8-th word is denoted as follows:

where 4 2 denotes the character embedding lookup which is randomly initialized.

Figure 2: The attention-based multi-level feature fusion framework. Character embeddings and word embeddings of the sentence are taken as input for the feature selection layer. From this layer, we simultaneously distill character-level local feature (Char Local), character-level global feature (Char Global), word-level local feature (Word Local), and word-level global feature (Word Global). For convenience, we leave out the words labelled with $. Dashed arrows indicate a dropout operation is applied.

3.2 Multi-level Feature Selection

Global Character-level Feature Selection

As demonstrated by the BiLSTM-CRF model [Huang et al., 2015], long-distance dependencies are important for NER, e.g., 'Washington' is relevant to both the past and future contexts, e.g., 'University' and 'George' in Figure 1. As the attention mechanism can relieve the limitation of encoding all information equally [Bahdanau et al., 2014], we combine the BiLSTM network with the attention mechanism to facilitate NER in extracting global character-level features. Taking character embeddings x 2 C as input at time step C, the contextual hidden state of BiLSTM can be expressed as follows:

where �! h 2�0A C and � h 2�0A C denote the forward and backward ouputs of BiLSTM at time step C. Besides, we adopt the self-attention mechanism to effectively capture the relations between any two representations regardless of their distance [Vaswani et al., 2017], e.g., 'Washington/B-ORG' is relevant but different from 'Washington/E-PER' in Figure 1. Concretely, we take h 2�0A C as the input to obtain the global character-level representation h �� C as follows:

Where cC is the context vector, and let h 2�0A B = h 2�0A C to obtain the additive attention weight UCB. W1, W2, and W2 are weight matrices, and -0 is a vector of parameters, which are randomly initialized.

Local Character-level Feature Selection

As demonstrated by BiLSTM-CNN [Ma and Hovy, 2016], convolutional neural networks (CNNs) have been proved to be useful in capturing character-level information, such as capitalization. Due to their sparse connectivity and parameter sharing, CNNs are able to effectively process the sequences in the current receptive field akin to the attention mechanism. Furthermore, the max pooling operation can be a great boost to capture the most significant feature [Kim et al., 2016], which is why they are adopted together to capture local character-level features.

We employ CNN with a redundant position of input sequences masked to extract the character-level features, which can be expressed as follows:

where U is the filter with filter width : set as 3. The convolution operation is typically denoted with an asterisk, and the masking function, Mask, simply sets the padded position of input sequences as zero.

Additionally, the max pooling operation, Max, is applied to capture the significant local features assigned with the highest value for a given filter [Kim et al., 2016], such as the capitalization of 'M' for 'Missouri'. At time step C, the characterlevel representation from local view is obtained as follows:

Global Word-level Feature Selection

As shown in previous works [Akbik et al., 2018; Yijin Liu and Zhou, 2019], word embeddings, especially pretrained word embeddings, play an important role in capturing word similarity and relations with other words. Therefore, global wordlevel features, such as lexical phrases where words frequently co-occur, can be obtained by only using self-attention, which has the advantage of modeling dependencies between words without regard to their distance [Vaswani et al., 2017], e.g., Label LOC frequently occurs after 'in'. Thus, we simply use basic dot-product attention as follows:

where query vectors Q 2 � =�3F , key vectors K 2 � =�3F and value vectors V 2 � =�3F . 3F denotes the dimension of each word embedding. It is noted that attention was computed without scaling to keep the scale in line with other representations. By setting Q = K = V = x F C at time step C, the word representation based on self-attention is obtained as follows:

In addition, we also tried to incorporate the BiLSTM network, however, this makes the result worse (e.g., AMFF* in Table 2). Therefore, we simply take h ,� C as a shortcut for improving the gradient's backpropagation inspired by residual networks [He et al., 2016].

Local Word-level Feature Selection

Inspired by the language model [Kim et al., 2016], the max pooling operation facilitates the selection of prominent features, for example, we can distill local word-level features from inter-word relations based on the attention mechanism, such as the relevant keyword 'in' from the input sequence in Figure 1. Based on Equation (10), the final representation of local word embeddings h , ! C is obtained as follows:

where Max indicates the max pooling, and FFN is a feedforward network.

3.3 Multi-level Feature Fusion

Multi-level feature fusion for the NER task is a robust and efficient strategy, which can take advantage of the most significant features to achieve better results. Feature fusion for NER aims to combine multiple relevant features into a global informative representation of original input sequences. During the fusing phase, we employ a concatenation strategy to fuse the multi-level features with an automatic adjustment. For conciseness, the final fusion representation of the multilevel features is obtained as follows:

where h �� C , h �! C , h ,� C , and h ,! C represent the features extracted from the above components. _8 (8 2 f1� 2� 3� 4g) controls the degree of the importance for each component, which is randomly initialized. Moreover, this equation can be easily extended to other cases by adding more features.

3.4 Sequence Labeling for Final Prediction

The fusion representation with multi-level features is fed into a BiLSTM network to make full use of all the semantic and syntactic information at a higher level. In addition, CRF boosts the performance of NER by taking the neighbor labels into consideration to avoid mislabeling. For example, I-ORG can not follow E-ORG in the NER task with BIOES annotation. Therefore, we incorporate a CRF in the BiLSTM network to jointly decode the best chain of labels.

Formally, we suppose the fusion representation output from BiLSTM is r = "r1� r2� � � � � rn", with the corresponding generic label sequence ˆy = " ˆH1� ˆH2� � � � � ˆH=". Given the input sequence r, the conditional probability [Ma and Hovy, 2016] is defined as ?"yjr;W� b" in CRF models as follows:

where H 0 represents an arbitrary label of all possible ones (S"r"), and k8 "H8�1� H8� r" = 4G?",H8�1�H8 r8 , 1H8�1�H8 " where ,H8�1�H8 and 1H8�1�H8 are the weight parameter and bias parameter corresponding to label pair "H8�1� H8".

For CRF training, the objective of the model is to maximize the following log-likelihood, which is given by:

During the decoding phase, we search for the best label sequence H � that maximizes the likelihood as follows:

Furthermore, for sequence labeling, we adopt Viterbi to calculate the final tag sequence efficiently.

4 Experiments

4.1 Datasets and Baseline Methods

To verify the effectiveness of the proposed framework, we conduct experiments on the following four datasets, CoNLL2003 [Sang and De Meulder, 2003], NCBI-disease [Do˘gan et al., 2014], SciERC [Luan et al., 2018], and JNLPBA [Kim et al., 2004]. All datasets have been separated into train/develop/test sets, including 4/1/6/5 entity types, respectively. Table 1 presents some statistics of the 4 datasets.

We compare our proposed model with the following classic (i.e., BiLSTM-CRF, BiLSTM-CNNs, and NeuralNER) and state-of-the-art (i.e., CS Embeddings, SciBERT, and CollaboNet) methods, respectively:

• BiLSTM-CRF [Huang et al., 2015]: This applies the BiLSTM network to learn both past and future features of word embeddings with CRF for sequence labeling.

• BiLSTM-CNNs [Chiu and Nichols, 2016]: This extracts character-level features using CNN, and word-level features from pretrained word embeddings, in addition to encoding partial lexicon matches in neural networks.
• NeuralNER [Lample et al., 2016]: Similar to Chiu [Chiu and Nichols, 2016], this regards words as sequences of character and learn character-level features from a BiLSTM, rather than CNNs.
• CS Embeddings [Akbik et al., 2018]: It obtains context embeddings at character level, and the final representation is concatenated with pretrianed word embeddings.
• SciBERT (Beltagy2019) [Beltagy et al., 2019]: This introduces a contextualized embedding model for scientific text based on BERT, which achieves the state-ofthe-art on several tasks.
• CollaboNet (Yoon2019) [Yoon et al., 2019]: This is built upon multiple identical single-task NER models (STMs) that send information to the proper model for more accurate predictions in the biomedical field.

In the experiment, we take both the pretrained word embedding GloVe [Pennington et al., 2014] and the randomly initialized character embedding as input, and train our proposed model using SGD to perform backpropagation through time. To avoid overfitting, dropout is applied to the input of each component as well as the output of the feature fusion layer in our model. For time efficiency, we merely initialize the related hyper-parameter values according to the aforementioned baselines. We repeat the experiment with an early stopping strategy 10 times, and report the average performance on the test set as the final performance. We adopt the F1 metric and BIOES tagging scheme for all datasets.

4.2 Overall Results and Comparisons

Table 2 gives the experiment results of AMFF and the baseline methods. For a fair comparison, we report their average results on the four benchmark datasets. Classic characterbased and word-based NER methods obtain lower F1 scores than recent methods on most benchmark datasets, because the final prediction in the BiLSTM-CRF network may be misled by the previous label due to insufficient information. To the best of our knowledge, SciBERT achieves state-of-the-art performance on NCBI-disease and SciERC, which depends on pretraining on a large corpus of scientific publications to generate contextualized embeddings. CollaboNet obtains the best result on JNLPBA due to multi-task learning, which may make a wrong prediction when an error is overlapped. However, classic methods achieve an F1 score of more than 90 on CoNLL-2003, which is slightly better than the result of SciBERT and CollaboNet. This is probably because these two methods are designed for academic and biomedical fields, respectively, which fail to capture general features such as local character features and lexical phrases from given sequences. In addition, CS Embeddings achieves the state-of-the-art with an F1 score of 92.74 on CoNLL-2003 and obtains competitive results on the other datasets, which merely takes the global word- and character-level features into consideration which is similar to a part of our proposed AMFF.

As shown in Table 2, our proposed method achieves the best results on the four benchmark datasets , which verifies its effectiveness. Specifically, obtaining global word-level features merely based on attention contributes to overall performance (compared with AMFF*), which might because pretrained word embeddings have already provided inter-word relations in no need of BiLSTM. In addition, there would be a slight performance degradation if we get rid of attention from our model, which highlights the importance of incorporating multi-level features based on attention mechanisms.

4.3 Ablation Study

As Table 3 shows, the AMFF with a single component is not good enough while the one with multiple components fusion tends to be more competitive, which might because the multilevel features captured from the four primary components (i.e., CG, CL, WG, and WL) contribute to the effectiveness of our model. Based on the attention mechanism, word-level components seem to be more effective than character-level components, which should be credited to the pretrained word embedding. For SciERC, when fusing three components, the F1 score decreases to 65.86, which could be due to the noise caused by the fusion components. However, in general, our proposed model tends to be more effective and robust with the number of fusion components increasing. This is mainly because all these components help to improve performance frommulti-level perspectives.

4.4 Case Study for Detailed Analysis

Table 4 shows a case study comparing our model with CS Embeddings [Akbik et al., 2018] and SciBERT[Beltagy et al., 2019] , which are more representative than the others. In the example, the polysemous word 'Washington' is likely to lead to ambiguity, i.e., the first 'Washington' denotes an organization (ORG) together with 'University' while the second should be categorized as a person (PER) in 'George Washington'. CS Embeddings and SciBERT may recognize 'Washington' as B-ORG or B-PER from the context, which causes 'University' to be erroneously labeled E-PER due to a lack of other auxiliary features. Different from the existing methods, the AMFF can easily recognize entities by taking more features of the original sequences into consideration, such as the lexical phrase and the keyword 'in', which is vital for distinguishing entity categorical labels. Furthermore, the AMFF put more emphasis on disambiguation based on self-attention, which enables the long-distance dependence to be captured, as shown in Figure 1. This demonstrates that our model has the advantage of distinguishing polysemous words by fusing multi-level features from different perspectives.

4.5 Parameter Sensitivity Analysis

Four primary parameters, i.e., dropout rate, LSTM size, filter number, and batch size, are selected to verify the effect of parameters on the effectiveness of the AMFF. The dropout rate denotes what percentage of units will be dropped in a neural network, the LSTM size controls how many hidden state units there are in sequence labeling, the number of filters affects the output shape of the character-level CNN module, and the batch size controls training efficiency and the allocated resources. To study uncertainty in the output of our proposed model, we employ single-parameter sensitivity analysis by varying one parameter while fixing the others each time. As shown in Figure 3, the AMFF keeps high performance while the parameter varies on the four benchmark datasets, which demonstrates that multi-level features contribute to NER, and further verifies the effectiveness and robustness of our proposed model.

5 Conclusion

This paper presents a novel AMFF framework, which effectively leverages multi-level features to predict entity categorical labels. The proposed framework captures character-level and word-level features from both global and local perspectives, e.g., capitalization, inter-word relations, keywords, and lexical phrases, by adopting attention mechanisms from different perspectives. Furthermore, the proposed framework can be easily extended by incorporating more features, such as affixe, to boost the performance of NER. The Experiment results demonstrate that the AMFF surpasses previous stateof-the-art models on CoNLL-2003, NCBI-disease, SciERC, and JNLPBA datasets.

Acknowledgments

This work is partially supported by National Natural Science Foundation of China through grants No.61976102, No.U19A2065, and No.61902145. This work is also partially supported by NSF through grant IIS-1763365 and by FSU.