CNN-Based Chinese NER with Lexicon Rethinking

1 Introduction

The task of named entity recognition (NER) involves the determination of entity boundaries and the recognition of categories of named entities, which is a fundamental task in the field of natural language processing (NLP). NER plays an important role in many downstream NLP tasks, including information retrieval [Chen et al., 2015], relation extraction [Bunescu and Mooney, 2005], question answering systems [Diefenbach et al., 2018], and other applications. Compared with English NER, Chinese named entities are more difficult to identify due to their uncertain boundaries, complex compositions, and NE definitions within the nest [Duan and Zheng, 2011]. Hence, one intuitive way to perform Chinese NER is to first perform word segmentation and then apply word sequence labeling [Yang et al., 2016; He and Sun, 2017].

However, gold-standard segmentation is rarely available in NER datasets, and word segmentation errors negatively impact the identification of named entities [Peng and Dredze,

∗Equal contributions

2015; He and Sun, 2016]. The development of ways to use lexicon features to better leverage word information for NER has attracted research attention [Passos et al., 2014; Zhang and Yang, 2018]. In particular, to exploit explicit word information, Zhang and Yang [2018] introduced a variant of LSTM (lattice-structured LSTM) that encodes all potential words that match a sentence. Because of its rich lexicon information, the lattice LSTM model has achieved state-ofthe-art results on various datasets.

Although previous work using RNNs to incorporate lexicons has achieved great success, these methods continue to have difficulty with two issues. First, RNN-based methods fail to fully exploit GPU parallelism because of the recurrent structure, which limits their computational efficiency [Strubell et al., 2017]. Specifically, lattice LSTM [Zhang and Yang, 2018] employs double recurrent transition computation across the length of the input, one for each character in a sentence and the other for matched potential words in the lexicon. As such, their speeds are limited. Second, they have difficulty dealing with conflicts between potential words being incorporated in the lexicon: one character may correspond to couples of potential words in the lexicon, and this conflict can misguide the model such that it predicts different labels, as shown in Figure 1. Because of the nature of the sequential processing in RNNs, it is difficult to determine which word is correct based only on the previous inputs. For example, given part of the sentence "广 州市长隆(Guangzhou Chimelong)," an RNN-based model would be uncertain whether the character "长" is part of the word "市长(Major)" or "长隆(Chimelong)." The matching of the words "市长(Major)" and "长隆(Chimelong)" would guide the character "长" to be identified as 'O' and 'B-GPE,'

respectively. More seriously, this conflict among words is ubiquitous in Chinese NER, and these ambiguities cannot be settled without reference to the whole-sentence context and high-level information [Ma et al., 2014].

In this paper, we present a novel convolutional neural network with a rethinking mechanism. The first issue is handled using a CNN to process the whole sentence as well as all the potential words in parallel. The intuition is that when the window size of the convolution operation is set to 2, then all the potential words can easily fuse into corresponding positions [Kim, 2014]. As shown in Figure 2, words with certain lengths correspond to distinct positions in certain layers. In this manner, characters coupled with potential words can be processed in parallel. The second issue is addressed by the use of a rethinking mechanism [Li et al., 2018]. Most existing Chinese NER models learn features using only a feedforward structure. They have no chance to modify conflicting lexicon information after seeing the whole sentence. By adding a feedback layer and feeding back the high-level features [Ma et al., 2014], this rethinking mechanism can leverage the high-level semantic to refine the weights of embedded words and address the conflicts between potential words. Experimental results on four NER datasets show that the proposed method can achieve better performance than other baseline methods.

In summary, our contributions are three-fold: 1) We propose a novel CNN structure for incorporating lexicons into Chinese NER, which effectively accelerates the model training by its use of parallelism; 2) we apply a rethinking mechanism to tackle the conflict between potential words in the lexicon, and the proposed model can leverage the highlevel semantic to identify the correct words; 3) experimental results on four datasets demonstrate that the proposed model can achieve better performance, and performs up to 3.21 times faster than state-of-the-art methods.

2 RelatedWork

Our work is inspired by three lines of research: enhancing character-level Chinese NER through lexicon information, improving computational efficiency by parallelism, and refining networks using a rethinking mechanism.

Chinese NERwith lexicon. To date, using neural networks for NER has been the dominant approach [Ma and Hovy,

2016]. For Chinese NER, there have been explicit discussions comparing word-based and character-based methods, which have shown that due to the limited performance of current Chinese word segmentation, character-based name taggers can outperform their word-based counterparts [Zhang and Yang, 2018]. Zhang and Yang [2018] exploit an RNNbased lattice structure to simultaneously model characters and corresponding words in a lexicon, which prevents segmentation errors. However, lattice LSTM suffers from inefficiency and word conflict. Our approach extends these ideas from the parallel modeling of characters with potential words and a rethinking mechanism. Hence, our method is more efficient and effective.

Improving NER computational efficiency. In general, end-to-end CNNs in NLP have mainly been used for text classification [Kim, 2014]. For sequence labeling tasks, CNNs have been mainly used for low-level feature extraction[Ma and Hovy, 2016] as input for alternative architectures. Recently, Strubell et al. [2017] proposed iterated dilated convolutional neural networks (ID-CNNs) for sequence labeling with parallel computation. However, ID-CNNs applied a dilated window that skips over every dilation, which makes adding lexicon information impractical. In this work, we propose a novel CNN architecture that can process all the characters and lexicon information in parallel.

Refining networks with a rethinking mechanism. Previous attempts to use a rethinking mechanism in neural networks have been made in image classification to tackle issues of occlusion and noise [Li et al., 2018]. Li et al. [2018] used the output posterior possibilities of a CNN to refine its intermediate feature maps. We extend these concepts from refining networks to tackle conflict among words.

3 Lexicon Rethinking CNNs

This paper presents a novel lexicon rethinking convolutional neural network (LR-CNN). The proposed model can process all of the characters in a sentence, as well as all of the potential words that match the sentence in parallel. By adding a feedback layer and feeding back the top-layer features, the rethinking mechanism can leverage the high-level semantic to refine the weights of embedded words and tackle the conflicts between potential words.

3.1 Lexicon-Based CNNs

CNNs have been shown to be effective for Chinese NER [Strubell et al., 2017]. We propose the use of CNNs to encode the sentences and lexicons in parallel. We denote the input sentence as C = {c1, c2, · · · , cM}, with character vocabulary V, where cm ∈ Rd is the m-th character embedding. Then C ∈ Rd×M denotes the sentence matrix. The potential words in the lexicon that match a sequence of characters can be formulated as wlm = {cm, · · · , cm+l-1}; as such, the first letter of the word is cm and the length of the word is l.

Take the sentence in Figure 2 as an example. The sentence consists of five potential words (e.g., w2 1 =广州, w3 1 =广州 市, w2 3 =市长). Given the sentence C and potential matching words w, we adopt character-level CNNs to encode the character features and attention modules to incorporate the lexicon features. The CNNs apply multiple filters H ∈ Rd×2 with a window size of 2 to obtain the bigram information and stack the multiple layers to obtain the multigram information, as follows:

where C[∗,m : m+ 1] is the m-to-(m+ 1)-th column in C and 〈A,B〉 = Tr(ABT ) is the Frobenius inner product. We regard the character embedding layer as the first layer C1. Similarly, the feature in the l-th layer Clm represents the l gram feature.

From the perspective of the combination of characters, the l gram feature Clm corresponds to the word wlm (Figure 2). To incorporate the lexicon feature effectively, we use the vectorbased attention [Chen et al., 2018] to combine the l gram feature with the word feature, as follows:

where W, U and b are the parameters of the attention module. ⊙ refers to the element wise production, and σ represents a sigmoid operation. In this way, the model can process the characters and potential words in parallel.

3.2 Refining Networks with Lexicon Rethinking

Although using an attention module is effective to incorporate lexicons, it still has difficulty dealing with the conflicts between the potential words. The model incorporating the potential words in the lower layers cannot refer to the words in the higher layers, due to the hierarchical structure of CNN. Hence, the effective use of high-level features to tackle the ambiguity among the potential words is still a key issue [Ma et al., 2014].

In this work, we treat the features at the top layer of CNN, XLm, as the high-level features. We then use these features to readjust the weights of the lexicon attention module by adding a feedback layer to each CNN layer, as follows:

where W, U, V and b are the parameters of the rethinking module. To avoid overfitting, W ∗ , U ∗ and b∗ are the reused parameters of the attention module included to reduce the number of parameters. With high-level features participating in the reweighting of the word vectors and CNN features, the model can learn to address the conflicts between the potential words.

Consider the case in Figure 2, where the word vector "市长(Major)" is a misleading input that can give rise to an incorrect label decision. Without the features of "广州 市(Guangzhou City)" and "长隆(Chimelong)" in the higher layer, the model may focus more on the misleading word vector. However, the rethink vector from the upstream layer can decrease the weight of the word vector "市长(Major)" in the output features, so as to correct the label decision.

Through the CNN encoder and lexicon attention module, the model obtains the respective l gram features Xlm within each layer of the model. These feature values correspond to the multi-scale l-grams, of which some are redundant (i.e., unigram X1m, bigram X2m, . . . , and L-gram XLm). In this work, we propose the use of the multi-scale feature attention [Wang et al., 2018] to adaptively selects the features of different scales in each position of a sentence, as follows:

After being processed by the multi-scale attention module, the final representation Xatt = [Xatt 1 ,Xatt 2 , · · · ,Xatt M ] would be fed into the CRF layer for the prediction.

3.3 Predicting NERwith CRF

Suppose that y = {y1, · · · , yM} represents a sequence of labels for sentence C, and Y(C) denotes the set of possible label sequences for C. The probability of the label sequence y is:

where ψi(ym-1, ym,Xatt ) = exp(WmXatt + bm) is the potential function and Wm and bm are the weight vector and bias, corresponding to label pair (ym-1, ym), respectively.

During the training, we optimize the parameters of the model to maximize the following conditional likelihood:

where y is the true labels of sentence C, and W, b are the parameters of the model. During the decoding process, we search for the label sequence y∗ with the highest conditional probability:

4 Experimental Setup

In this section, we describe the datasets that include the newswire and social media domains. We then detail the baseline methods applied, including the word- and characterbased neural Chinese NER under different settings. Finally, we detail the configuration of the proposed model.

4.1 Datasets

We evaluate the proposed method on four datasets, including OntoNotes [Weischedel et al., 2011], MSRA [Levow, 2006], Weibo NER [Peng and Dredze, 2015; He and Sun, 2016], and Resume NER [Zhang and Yang, 2018]. The splitting methods follow those in [Zhang and Yang, 2018].

The OntoNotes and MSRA are the newswire domain, where gold-standard segmentation is available in the training data. For OntoNotes, gold segmentation is also available for the development and test data; however, no segmentation is available for the MSRA test data. The Weibo and Resume NER come from social media. There is no segmentation in the Weibo and Resume datasets. A description of the datasets is presented in Table 1.

We use the pretrained character embeddings and lexicon embeddings 1 trained using word2vec [Mikolov et al., 2013], over the automatically segmented Chinese GigaWord 2 . The lexicon consists of 704.4k words, containing 5.7k single-character, 291.5k two-character, 278.1k threecharacter words, and 129.1k other words.

4.2 ComparisonMethods

As baselines for comparison, we applied several classic and state-of-the-art methods on the four datasets. In addition, to verify the effectiveness of the proposed method, we compare the word- and character-level methods that utilize the bichar, softword, and lexicon features.

For the dataset without gold segmentation, we first train the open source segmentation toolkit 3 on the training data. We then use it to automatically segment the datasets. Finally, we apply the word-level NER methods on these segmented datasets to evaluate their performances, which are denoted as Auto seg. For the dataset with gold segmentation, we directly apply the word-level NER methods, which are denoted as Gold seg. We also evaluate the character-level methods without using segmentation information, which are denoted as No seg. All the methods evaluated are as follows:

LSTM. A bi-directional LSTM [Hochreiter and Schmidhuber, 1997] is used to obtain a left-to-right hidden state -→h wi and a right-to-left hidden state ←- h wi , which are concatenated for the NER prediction.

1 https:// github.com/jiesutd/LatticeLSTM.

2 https://catalog.ldc.upenn.edu/ LDC2011T13

3 https://github.com/lancopku/PKUSeg-python

CNN. We apply a standard CNN [Kim, 2014] structure on the character or word sequence to obtain its multiple gram representation for the NER prediction.

Word-level model + char + bichar. To extract themorphological and combination information from the words, we first concatenate the character embedding and bigram embedding to represent the character xcm. We then use a bi-directional LSTM on the character sequence to obtain the characterlevel features. Finally, we augment the word-level model by combining word embedding with the character-level features as: xwm = [wm ⊕ xcm].

Character-level model + bichar + softword. We use the BMES scheme for representing the segmentation. In addition, we use the same bigram feature bm as a word-level model. Then, the character feature can be represented as: xcm = [cm ⊕ bm ⊕ seg(cm)].

Dilated CNN. Dilated CNN [Strubell et al., 2017] stacks several of the CNN layers that applied a dilated window skips over every dilation. This method achieves great performance in the English NER task.

Lattice LSTM. Lattice LSTM[Zhang andYang, 2018] can model the characters in sequence and explicitly leverages word information through Gated recurrent cells, which can avoid segmentation errors. The lattice LSTM achieved stateof-the-art performance on the four datasets.

4.3 Hyper-Parameter Settings

For all four of the datasets, we used the Adamax [Kingma and Ba, 2014] optimization to train our networks. The initial learning rate was set at 0.0015, with a decay rate of 0.05. To avoid overfitting, we employed the dropout technique (50% dropout rate) on the character embeddings, lexicon embeddings and each layer of the CNNs. The character embeddings and lexicon embeddings were initialized by a pretrained embedding and then fine-tuned during the training. The character embedding size and lexicon embedding size were set to 50. For the biggest dataset, MSRA, we used five layers of CNNs with an output channel size of 300. For the other datasets, we used four layers of CNNs with an output channel size of 128. We used early stopping, based on the performance on the development set. Our code are released at https://github.com/guitaowufeng/LR-CNN.

5 Results and Analysis

In this section, we detail the performance of the proposed baseline models. We present the results of a series of experiments to demonstrate the effectiveness of the lexicon and rethinking mechanism.

5.1 Method Comparison

For OntoNotes, gold-standard segmentation is available in the entire dataset. No segmentation is available for the MSRA test sections, nor is the Weibo and Resume datasets. As a result, we study the oracle situations where gold segmentation is given on OntoNotes and evaluate the automatic/no segmentation situations on the other three datasets.

Table 2 4 illustrates a variety of settings for the wordand character-based Chinese NER. In the gold or automatic segmentation settings, the char and bichar features can enrich the word representations in the word-level models to obtain a better performance than those without char and bichar features. In particular, these models can achieve results that are competitive to the state-of-the-art [Che et al., 2013; Wang et al., 2013]. However, due to the influence of the word segmentation errors, the models on the automatic segmentation dataset would perform worse than those on the gold segmentation dataset. In addition, the gold segmentation is not available in most datasets. Hence, the previous work explores the character-based methods to avoid the need for word segmentation. The LR-CNN outperforms characterlevel lattice LSTM by 0.57% in F1 score. It also results in an almost 3 percent improvement over the LSTMwith bichar and softword features. In addition, because the processing of the characters and lexicons are conducted in parallel, LRCNN has an efficiency advantage (Table 6).

Tables 3, 4, and 5 present the comparisons between the classic and state-of-the-art methods on the MSRA, Weibo, and Resume datasets. The existing classic methods explore the rich handcrafted features [Chen et al., 2006; Zhang et al., 2006; Zhou et al., 2013], character embedding features [Lu et al., 2016], segmentation embedding features [Peng and Dredze, 2016], and radical embedding features [Dong et al., 2016]. He and Sun 2017 leverage the cross-domain and semi-supervised data for the Chinese NER. Our LR-CNN model significantly outperforms these models. In addition, with a better manner to incorporate the lexicons, LR-CNN also outperforms the strong baseline lattice LSTM on all of the datasets.

4 In Table 2, 3 and 4, the results with ∗ refer to the model with external labeled data for semi-supervised learning. those with † mean that the model also uses discrete features.

5.2 Efficiency Advantage

Our LR-CNN not only achieves better F1 score results than the baseline models, but it is also faster. Table 6 lists the relative decoding time on all four of the development sets, as compared to the lattice LSTM. We report the decoding time using the same batch size for each method.

Without the CRF, the LR-CNN decodes up to 3.21 times faster than the lattice LSTM. With Viterbi decoding, the gap closes somewhat, but the LR-CNN still has better efficiency (i.e., about an average of 1.91 times faster than the lattice LSTM). The LR-CNN with CRF is even faster than the lattice LSTM without CRF. In addition, we study the degree of the decline in model performance when the models get rid of the CRF layer. We find that the F1 score of the LR-CNN decreases, on average, by 5.75% on the four datasets. In contrast, the lattice LSTM decreases by 6.10%, showing that the LR-CNN is a better structure for the token encoder.

5.3 Influence of Sentence Length

To investigate the influence of the different sentence lengths, we analyze the performance of the LR-CNN and lattice LSTM on the OntoNotes dataset. The dataset is split into five parts according to the sentence length. To rule out the impact of the CRF, we evaluate the models by removing the CRF layer. Figure 3 demonstrates the F1 score and processing speed on the different sentence lengths.

The results reveal that the LR-CNN outperforms the lattice LSTM on all of the datasets with different sentence lengths. In particular, the F1 score for LR-CNN is nearly 12% greater than that of lattice LSTM when the sentence length is less than 20 characters. Hence, LR-CNN has a much greater advantage than the lattice LSTM in dealing with short sentences. In addition, we evaluate the processing speed of the sentences with different lengths. We find that with the sentence length increases, the speed of the LR-CNN is relatively stable, while the speed of the lattice LSTM decreases substantially. In particular, when processing the sentence of which the length is greater than 100, LR-CNN is 5 times faster than the lattice LSTM.

5.4 Ablation Study

To study the contribution of the main components in the LRCNN, we ran an ablation study on all four datasets. The results are reported in Table 7. Overall, we find that:

1. Lexicons play an important role in character-level Chinese NER. Compared with models incorporating lexicons, the performance of the model without lexicons is seriously degraded, and the F1 score decreases by 9% on the Weibo dataset.

(2) Rethinking mechanisms can further improve the performance of the models with lexicons, because the rethinking mechanisms can tackle the conflicts between the potential words that match the same characters. Table 8 illustrates two examples where the rethinking mechanism successfully tackles the conflict words and makes correct predictions.

(3) The rethinking mechanism may benefit models without lexicons. Although the lexicons are not available, the LRCNN - Lexicon is 5% better than itself without the rethinking mechanism in the F1 score on OntoNotes dataset.

6 Conclusion

In this work, we propose a novel convolutional neural network to incorporate lexicons with a rethinking mechanism. This network can model all the characters coupled with the potential words that match the sentence in parallel. To solve the conflicts between the potential words, the proposed model can refine the networks by adding a feedback layer to feed back the high-level features. We evaluate the proposed model on four Chinese NER datasets. The experimental results illustrate that the proposed method can significantly improve the performance when compared with that of the other baseline approaches. In addition, the proposed method is up to 3.21 times faster than the state-of-the-art methods.

Acknowledgements

The authors wish to thank the anonymous reviewers for their helpful comments. This work was partially funded by China National Key R&D Program (No. 2018YFC0831105, 2017YFB1002104, 2018YFB1005104), National Natural Science Foundation of China (No. 61751201, 61532011), Shanghai Municipal Science and Technology Major Project (No.2018SHZDZX01), STCSM (No.16JC1420401, 17JC1420200), ZJLab.