NerCo: A Contrastive Learning Based Two-Stage Chinese NER Method

1 Introduction

Named Entity Recognition benefits a wide range of downstream tasks in natural language processing(NLP). Due to their simplicity and effectiveness, sequence labeling methods have long been the most common solution tackling this task[Huang et al., 2015; Ma and Hovy, 2016; Lample et al., 2016]. Recent success of Transformer-based [Vaswani et al., 2017] large-scale pre-trained language models(PLMs) such as BERT [Devlin et al., 2018] and RoBERTa [Liu et al., 2019] have greatly boosted these sequence labeling methods to get better contextualized token representations. As a result, although Chinese NER is more difficult due to the language's natural lack of word boundary information, impressive performance improvements have been achieved with the combination of additional techniques such as lexicon integration and so on[Li et al., 2020; Liu et al., 2021a].

∗Corresponding Author

Current sequence labeling methods rely on token labels in the form of 'Position-Type' as their supervision signals [Sang and De Meulder, 2003] to perform token classification.

respectively indicating its position within the entity and its category. Taking an PER (Person) entity in BIO tagging format for an example, its first token is annotated B-PER and the remaining tokens are annotated I-PER , where the chunk tag B- facilitates recognizing consecutive entities of the same type [Ramshaw and Marcus, 1995]. Though it is simple and effective, we find this straightforward scheme may lead to a serious problem that severely hinders the generalization performance of the model. Specifically, one entity usually contains different positional tokens. Those tokens naturally share the same label semantics since they belongs to the same entity type. However, the supervising scheme adopted by sequence labeling assigns independent labels and enforces them to be treated as distinct and isolated classes (e.g. BType, I-Type). As a result, it makes their representations dispersed into several separate and unrelated clusters, even though they are in the same entity. We call this problem Entity Representation Segmentation in Label-semantics.

As shown in Figure 1, taking a labeled entity '张家川县' (Zhangjiachuan County) which is a Geo-Political Entity(GPE) labelled in BIO tagging format as an example, traditional methods will still guide models to label the first token "张(Zhang)" B-GPE and subsequent tokens I-GPE, leading to the aforementioned problem of Entity Representation Segmentation in Label-semantics. This training scheme results in a contradiction that tokens of an entity sharing the same label semantics would be embedded far and unrelated in Euclidean space and the integral entity representation would be semantically segmented, only due to the slightly different inner-entity position indicated by their chunk tags. Such representation deficiency is not trivial, because it could severely damage subsequent classification by confusing models in determining entity boundaries and naturally influence their type decision.

In fact, to facilitate model's classification head to correctly recognize an entity's boundary, the representations in this sequence should be semantically coherent and consistent inside and distinctive and discriminative outside. So, in this paper, we propose a novel framework named NerCo: Named entity recognition with Contrastive learning to address Entity Representation Segmentation in Label-semantics problem. To satisfy the demand of intra-entity coherence and inter-entity distinctiveness, NerCo naturally leverages contrastive loss to pull semantically close neighbors of entity tokens together and push non-neighbors apart to learn a label-semantics based representation learning model in the first stage. In the second stage, we finetune the model with traditional token supervision in BIO format to capture inner-entity positional information for final prediction. Although NerCo is simple and easy to use, requiring neither data augmentation nor base encoder modification, it can fully exploit model's representation potential after injecting label-semantic signals entailed in raw data, thus effectively alleviating the Entity Representation Segmentation in Label-semantics problem and achieving better generalization performances.

Experiments on four datasets show that NerCo achieves consistent improvements over only finetuning the base encoder 1 . Our method improves F1-score by 4.11% on Weibo dataset specifically and achieves new state-of-the-art performances on all of the Chinese NER datasets in our experiments. The results demonstrate that sequence labeling methods can exhibit much stronger power in entity representation and recognition after properly solving the Entity Representation Segmentation in Label-semantics problem.

2 RelatedWork

Our work is related to existing sequence labeling methods for Chinese NER and contrastive learning in NLP.

2.1 Chinese NER as Sequence-Labeling

NER has long been formulated as a sequence labeling task and the current state-of-the-art results for sequence labeling have been achieved by neural network approaches[Huang et al., 2015; Ma and Hovy, 2016; Chiu and Nichols, 2016]. Compared with NER in English, Chinese NER is more difficult due to the absence of explicit word delimiter in Chinese sentences. Previous studies have empirically proved the superiority of character-based methods over word-based ones[He and Wang, 2008; Liu et al., 2010; Li et al., 2014].

1 We release our codes at https://github.com/zhzai/nerco.

Since lexicon features can provide rich word boundary information, integrating them into a character-based sequence encoder has attracted research attention. [Zhang and Yang, 2018] first proposed the lattice structure to encode all potential words matched in a sentence. [Gui et al., 2019a] introduced a rethinking mechanism to tackle the conflicts between potential words. To better capture distance and direction information, TENER [Yan et al., 2019] customized the transformer encoder to incorporate both character and word features using relative position encoding. While LGN[Gui et al., 2019b] and CGN[Sui et al., 2019] utilized graph neural network to model the interaction within a character and word sequence, [Li et al., 2020] converted the lattice structure into a flat sequence consisting of spans and achieved excellent and stable performances.

Considering the significant NER improvement brought by pre-trained models [Devlin et al., 2018; Liu et al., 2019], many researchers turned to incorporating lexicon knowledge into pre-trained models to combine both advantages. ERNIE [Sun et al., 2019] leveraged entity-level and word-level masking strategies to implicitly integrate external knowledge into BERT. [Diao et al., 2019] proposed a BERT-based Chinese text encoder to explicitly consider potential word boundaries while pre-training and fine-tuning. Different from previous pre-training methods, [Liu et al., 2021b] proposed to integrate lexicon information using a lexicon adapter between transformer blocks, which encouraged more sufficient interactions between lexicon features and BERT.

2.2 Contrastive Learning in NLP

Contrastive Learning tries to learn powerful representations in such a way that similar features are pulled together and dissimilar ones are pushed apart in representation space[Hadsell et al., 2006; Jaiswal et al., 2020]. Recently, it has become a rising domain and achieved great success in computer vision community[Chen et al., 2020; He et al., 2020]. To employ contrastive learning in NLP, a key question to answer is how to construct positive pairs. [Fang et al., 2020] exploited back-translation to create positive instances of original sentences. Similar to visual domain, [Wu et al., 2020] conducted multiple sentence-level data augmentations, such as word substitution, synonym substitution and reordering. While [Giorgi et al., 2021] regarded spans within a document as similar instances, [Gao et al., 2021] applied dropout to the same sentence embedding twice to get positive pairs. Another line of positive pair construction in NLP research utilized a similar contrastive learning objective, the only difference is that they used labelled datasets for constructing positive instances[Henderson et al., 2017; Gillick et al., 2019; Gao et al., 2021].

Existing approaches have attempted to apply contrastive learning to NER. SCL-RAI[Si et al., 2022] is proposed to cope with the Unlabeled Entity Problem by decreasing the distance of span representations with the same label and increasing it for different ones via span-based contrastive learning. [Das et al., 2022] presents a novel contrastive learning technique called CONTaiNER for few-shot named entity recognition. [Zhang et al., 2023] proposes a bi-encoder framework which applies contrastive learning to map text and entity types into the same vector space. Unlike our work, this paper requires large modifications to the model, which brings a relatively high complexity.

3 Method

Inspired by the idea of self-supervised representation learning and pretrain-finetune paradigm, we propose a two-stage framework for Chinese named entity recognition. In the first stage, we leverage labeled data to construct contrastive pairs and train the encoder for label-semantics based token representations using contrastive loss, which is shown in Figure 2. In the second stage, we maintain the previous sequence labeling convention, in which original labeled data and loss function are utilized for final entity recognition. This framework also complies with our intuition of solving Entity Representation Segmentation in Label-semantics, for which we first learn cohesive representations within an entity corresponding to its label semantics and then finetune the encoder, equipping it with the ability of inner-entity position discrimination for chunk tags and linear mapping to type tags for each token, i.e. assigning 'Position-Type' tags.

3.1 Contrastive Pair Construction

To further clarify our construction of contrastive pairs and better resolve the Entity Representation Segmentation in Labelsemantics problem, we first present the definition of Semantic Class here.

Semantic Class represents the semantic category of the entity token, regardless of its positional role within an entity. A Semantic Class is signified only by the type tag of the token's 'Position-Type' label. In other words, tokens with different labels of position but the same of semantics are merged into a single category named Semantic Class. For tokens with label O, we set them a separate Non-Entity class. Taking B-PER and I-PER in Ontonotes 4.0 dataset using BIO tagging format as an example, tokens labelled by them both belong to the Semantic Class of PER, indicating their 'person entity' label semantics. The mapping relationship is illustrated in Table 1.

Then, we can answer the usually critical question in contrastive learning: how to construct (xi; x+i ) and (xi; x-i ) pairs. Following a simple implementation in SimCLR[Chen et al., 2020], we construct positive and negative pairs using the token representations within a mini-batch. We consider a query and a key as a positive pair if they belong to the same Semantic Class St. All token representations from different Semantic Classes are considered negative samples for each token.

3.2 Semantic Class Guided Contrast

Naturally, the token representations belonging to the same Semantic Class should be more similar and related to the their entity's type semantics. Thus we use contrastive learning to pull the positive pairs together and push the negative pairs apart in target space. Following the de facto procedure in contrastive learning in computer vision [Chen et al., 2020; He et al., 2020], we propose to take the contrastive training as the first learning stage of our framework. We adopt InfoNCE loss in SimCLR[Chen et al., 2020] as our contrastive loss function. For a query token xi and one of its positive key, the contrastive loss for (xi, x+i ) is defined as Lctr i :

where hi and h+i denote the representations of xi and x+i , h-j is the representation of each x-j in the mini-batch, τ is the temperature hyper-parameter, and sim(h1, h2) is the cosine similarity h⊤1 h2 ∥h1∥·∥h2∥ . The sum in the denominator is over one positive and N negative tokens.

In the first stage of contrastive learning, we optimize the sum of the above loss function corresponding to all positive pairs in the mini-batch. Considering that tokens with O labels are not entities and contain miscellaneous label semantics, pushing their representations together in embedding space as positive pairs does not make a clear sense. Here we filter out the cases where tokens of the Semantic Class Non-Entity serve as queries.

3.3 Token Label Supervised Fine-tuning

In the second stage, we finetune the above learned model using conventional sequence labeling method. To capture dependencies between consecutive tags, a CRF layer is leveraged on top of the encoder.

Given K labelled data {si, yi}|Ki=1 , K is the number of sentences in a mini-batch. H = {h1, h2, ..., hn} is the representation sequence of sentence si output by the encoder. We first perform a linear transformation for subsequent classification:

whereWo and bo are learnable parameters. After that, we can get the probability of the label sequence yi = {y1, y2, ..., yn} of si:

where T is the transition score matrix and ˜yi denotes all possible tagging sequences.

We train the model by minimizing the sum of all sentences' negative log-likelihood losses in the mini-batch as:

For inference, we search the label sequence with the highest score using Viterbi algorithm to recognize entities.

4 Experimental Setup

4.1 Datasets

We conducted experiments on four Chinese NER datasets to evaluate our proposed method. (1) OntoNotes 4.0[Weischedel et al., 2011] is a multilingual corpus that is manually annotated in the news domain with various text annotations, including Chinese named entity tags. We only utilized Chinese documents, consisting of four entity types, and processed the data in the same manner as [Che et al., 2013]. (2) MSRA[Levow, 2006] is also a news domain corpus serving for word segmentation and named entity recognition in Chinese. In our study of Chinese NER, it includes three named entity types: LOC, PER, and ORG. (3) Weibo NER[Peng and Dredze, 2016] comprises annotated NER labels from the social media website Sina Weibo. The corpus includes both named entities and nominal mentions for four types of entities: PER, ORG, GPE, and LOC. (4) Resume NER annotated by [Zhang and Yang, 2018] consists of resumes of senior executives and is annotated with 8 types of named entities. The statistic information of the datasets is presented in Table 2.

4.2 Baselines

We take FLAT Transformer in [Li et al., 2020] as our baseline model. FLAT is a strong baseline that converted the lattice structure into a flat sequence and used relative position encoding for Chinese NER. BERT[Devlin et al., 2018] is the mostly used method to finetune a pre-trained Chinese BERT for Chinese NER. LatticeLSTM[Zhang and Yang, 2018], TENER[Yan et al., 2019], SoftLexicon[Ma et al., 2020] and LEBERT[Liu et al., 2021a] are other representative models integrating lexicon information into Chinese sequence for better NER performance. RICON[Gu et al., 2022] and MarkBERT[Li et al., 2022b] are newly proposed competitive models for Chinese named entity recognition and both achieved strong performances. W2NER[Li et al., 2022a] treats NER as a word-word relation classification task in a unified formalism, while BoundarySmoothing[Zhu and Li, 2022] proposed a boundary smoothing method as a regularization technique for span-based NER model, achieving the sota level on the four Chinese NER datasets.

4.3 Implementation Details

We adopt the FLAT Transformer in [Li et al., 2020] as our base encoder architecture. Following the setting in [Li et al., 2020], We use one layer of encoding layer, and SGD in both stages to optimize the encoder. We leverage BERT for contextual token embedding[Devlin et al., 2018]. During the first stage of contrastive learning, we utilize InfoNCE objective to optimize the parameters in FLAT encoder(with BERT parameters frozen) until achieving the minimal loss. Afterwards, we finetune all the parameters of the learned model, including BERT, in the manner of normal sequence labeling. Considering the relatively small scale of Weibo and Resume, we only tune the classification head as a prior step before updating all parameters to stabilize this second-stage finetuning process. The way to select hyper-parameters is also the same as FLAT.

5 Experimental Results

5.1 Performance

Overall Results

Table 3 shows the experimental results on Chinese NER datasets. As shown in the table, our model outperforms our baseline model and other methods consistently on four Chinese NER datasets 2 . As a strong baseline method, we observe FLAT[Li et al., 2020] already surpass BERT tagger by a large margin and could give us a relatively strong and stable performance in Chinese NER. Thanks to our additional contrastive learning procedure, NerCo can effectively improve the performance of baseline FLAT[Li et al., 2020] by an average F1 score of 1.5% on the datatsets. Especially, our model brings a large relative improvement over the baseline of up to 6% on Weibo, in terms of the F1 score. As for the current state-of-the-art models, taking W2NER[Li et al., 2022a] as an example, it leveraged multiple types of embeddings and proposed complex architectures including multi-granularity dilated convolutions to capture word-word interaction information. Though it shows impressive performance and a large margin over other lexicon-based methods, our model outperforms W2NER on all Chinese NER datasets. In general, our method could surpass all the current top-performing methods, pushing the state-of-the-art performances of Chinese NER 3 .

Span F & Type Acc

Compared with FLAT, NerCo performs the additional labelsemantics guided contrastive learning as the first stage. We evaluate these two models in terms of Span F and Type Acc to further investigate our performance gains. Span F measures the F1 score of recognized entity spans over the gold spans, regardless of the correctness of their types. Type Acc is the proportion of full-correct predictions to span-correct predictions.

Table 4 shows these two metrics of FLAT and NerCo. We can find our model performs better than the baseline in both metrics, which demonstrates that the contrastive learning stage benefits to both span boundary detection and span classification. Specifically, the performance gain on Weibo is very obvious in both metrics, indicating that solving the problem of Entity Representation Segmentation in Label-semantics can greatly promote boundary detection and also help a lot in type prediction especially for small-scale NER datasets. For Ontonotes and Resume, the improvements on Span F are more significant than that on Type Acc and the situation is reversed on MSRA, showing that NerCo can respectively promote boundary detection and type decision in the two cases.

Performance Against Sentence Length

Figure 3 shows the F1 score trends of the mostly used BERT, our baseline FLAT and our method NerCo against sentence length. Here we show two representative curves of the test splits of Weibo and Ontonotes. In general, our method shows better performance and robustness on both datasets. Although FLAT performs similarly to our method when the sentence length is less than 80, its performance deteriorates as the sentence length increases and lags significantly behind our method. Notably, FLAT was even surpassed by BERT when the sentence length exceeds 100. However, our method can always perform well and, most importantly, remain stable when the sentence length varies. For Ontonotes, our method can always maintain a margin over FLAT in all sentence length sections, demonstrating more adequate and rational utilization of labeled data.

2 In Table 3, the results of FLAT are implemented by ourselves, while scores of other methods are copied from their original papers. '-' means that the paper didn't report this item.

3 We refer readers to Appendix for the experimental results on the English setting at https://github.com/zhzai/nerco/blob/master/ Appendix.pdf.

5.2 Ablation Study

To investigate the contribution of several key components and our proposed two-stage method, we conduct several ablation experiments on the four datasets. First, without contrastive learning, we only finetune the encoder and observe a large performance drop in our implementation, which proves that optimizing the contrastive loss function could help improve the inner-entity consistency and avoid Entity Representation Segmentation in Label-semantics. As the ablation of our proposed two-stage learning framework, we remove the first stage of contrastive learning and optimize the multi-task loss(i.e. the sum of InfoNCE loss and finetuning loss). The performance deteriorates even worse than only finetuning the model. This may be due to the fact that joint training for multiple objectives simultaneously may make the learning mission unclear, letting the training process uncontrollable. When removing filtering operations on Non-Entity tokens and performing contrastive learning on all contrastive pairs, the performance also decreases slightly, indicating that these meaningless contrasting may add noises to parameter optimization.

5.3 HowNerCo Brings Improvement

To explore why NerCo works better than the baseline, we visualize the representations of entity tokens encoded by both models and also demonstrate the learning process of our method. We use t-SNE to project these representations into two dimensions, taking the test split of MSRA as a representative example.

As shown in Figure 4, we select 6 classes(respectively B-ORG,I-ORG, B-PER, I-PER, B-LOC and I-LOC). Figure 4(a) presents the clustering of FLAT's output embeddings, and the right two figures illustrate our proposed two-stage learning strategy. Figure 4(b) shows the token representations after contrastive learning of pulling tokens of the same Semantic Class together, regardless their differences in chunk tags(i.e. B-PER and I-PER are considered the same Semantic Class). We can see tokens of the same Semantic Class are clustered together, and these three clusters are scattered separately in different directions, as desired by our proposed contrastive learning. And Figure 4(c) is the visualization of the final representations after the second stage finetuning. We can see that after NerCo's two-stage training, the clusters show much clearer margins and token representations within each cluster are significantly more tightly packed together, compared with the situation of that in FLAT. Since NerCo can generate such better token representations, it is much easier for it to make correct predictions near the decision boundaries, which naturally leads to performance gains in downstream tasks.

5.4 Case Study

Table 6 presents two tagging examples predicted by our model NerCo and the baseline FLAT. In the first example from MSRA, the word '联合'(Uniting) in '欧洲联合'(European uniting) is very similar to the noun '联盟'(Union) of another entity '欧洲联盟'(European Union), leading to an extremely blurry and confusing entity boundary for correctly detecting the entity '欧洲'(European). It turned out that FLAT mistakenly predicted '欧洲联合'(European uniting) as an ORG entity, presumably confusing the verb '联合'(Uniting) with the noun '联盟'(Union) and failing to recognize the right LOC entity '欧洲'(European). However, our model correctly detected the entity's boundary and classified it into the right type. The possible reason is that our method's first stage contrastive learning makes inner tokens' representations more consistent

6 Conclusion

In this work, we propose a novel framework NerCo: Named Entity recognition with Contrastive Learning to address the representation deficiency which we term Entity Representation Segmentation in Label-semantics in sequence labeling NER. We naturally introduce contrastive learning to harness the representation learning process such that the token representations should be similar within the same Semantic Class, and discriminative of different. We simply construct in-batch contrastive pairs based on entity's label-semantics and utilize InfoNCE loss for the first stage. Finetuning on the learned label-semantics based model using sequence labeling is followed for final entity prediction. Experiments demonstrate that our proposed two-stage method benefits model's generalization performance in Chinese named entity recognition. Our results claim that the simple and long-adopted sequence labeling methods are powerful instead of out-of-time after addressing the above problem. Future work will concentrate on adapting the idea of NerCo to more complex settings, such as nested and non-continuous NER, and integrating more advanced and powerful contrastive learning techniques to further enhance our method.

Acknowledgements

This research was partially supported by the National Key Research and Development Project of China No. 2021ZD0110700, the Key Research and Development Project in Shaanxi Province No. 2022GXLH-01-03, the National Science Foundation of China under Grant Nos. 62037001, 62002282, 62250009 and 61721002, the Major Technological Innovation Project of Hangzhou No. 2022AIZD0113, the "Pioneer" and "Leading Goose" R&D Program of Zhejiang No. 2022C01107, the China Postdoctoral Science Foundation No. 2020M683492, the MOE Innovation Research Team No. IRT 17R86, and Project of XJTU-SERVYOU Joint Tax-AI Lab.

B-ORG

B

B-ORG I-ORG

PER

B-

-

B-LOC

PER

I-PER

B-LOC I-LOC

(a) FLAT

• (b) NerCo(stage 1)

B-ORG I-ORG

B-PER

B-LOC I-LOC

I-PER

• (c) NerCo(stage 2)

Figure 4: A t-SNE Visualization on representations of entity tokens on MSRA. (a) shows the representations output by FLAT. (b) shows the intermediate token representations after Semantic Class guided contrastive learning. (c) shows the final token representations of our method. Our proposed NerCo generates clearer margins between clusters. To give a more specific example, tokens of I-ORG, dispersed at the lower part of the canvas (a), are mixed up with I-LOC tokens and I-PER tokens. Also, tokens clusters of I-LOC, B-ORG, B-PER and I-PER are split into several parts, without distinct boundaries outwards. In contrast, token representations of NerCo in (c) are packed closely within their clusters.

and similar, while keeping them away from the representations of outer-entity tokens. As a result, it forms a clear boundary for the sequence labeling model to detect in the second stage. In the second example from Ontonotes, the whole sequence of '成 都通 用医疗设备（西南）有限公司'(Chengdu General Medical Equipment (Southwest) Co., Ltd) is a complete company name, apparently forming an ORG entity. However, FLAT disappointingly segmented it into two parts and wrongly regarded the sequence as two ORG entities. We speculate that FLAT may be disturbed by the word '西南'(Southwest), because these words usually mean the beginning of a new entity. At the same time, the unusual length also limits the recognition of FLAT. In contrast, NerCo could learn cohesive representations corresponding to entities after contrastive learning stage, thus easily tackling the long entity and naturally making the accurate prediction.