One Model for All Domains: Collaborative Domain-Prefix Tuning for Cross-Domain NER

1 Introduction

Named entity recognition (NER) is an important task for Knowledge Graph (KG) construction [Zhang et al., 2022] and natural language processing (NLP). Owing to the data scarcity issue in practical scenarios, obtaining adequate domain-related data is usually labour-intensive. The naive idea of training models with rich-resource domain data (source) and transferring knowledge to a new specific domain (target) may struggle handing the semantic gap and limited data problem. Hence, cross-domain NER, capable of learning information from the source domain to specific target domains with limited data, has been proposed to alleviate this issue.

To address this task, some previous studies utilize adding auxiliary objects [Liu et al., 2020a; Wang et al., 2020] or designing new model architectures [Jia et al., 2019a; Liu et al., 2020b; Jia and Zhang, 2020] to train with both source and target domain data. Liu et al. [2021a] leverages continue pre-training on the target domain for a better understanding of the domain-specific data. Another line of researches [Zheng et al., 2022; Hu et al., 2022] focus on modeling the label relationship across domains to improve label information transfer. Typically, LANER [Hu et al., 2022], the current state-of-the-art (SOTA) method on CrossNER benchmark [Liu et al., 2021b], utilizes an architecture to better leverage the semantic relationships among domains for boosting the cross-domain performance.

Despite the empirical success of previous works, several issues remain which have not been be appropriately solved. Firstly, previous methods often rely on task-specific architectures for various domains with different entity categories. For instance, LANER [Hu et al., 2022] designs different model architectures to adapt a PLM to other target domains, which restricts the model's usability in more applications. Secondly, most current methods are computationally inefficient and require tuning all parameters of the PLMs, which ends up with an entirely new NER model for each domain. For example, storing a full copy of pretrained BERT_Large [Devlin et al., 2019] (340M parameters) for each domain is non-trivial, not to mention the billion-scale large LMs. We introduce a summary of the above two points from the perspective of practical applications: when a new domain data comes, it is not practical for users to design model architectures and tune all parameters in terms of computational resources and time. Thirdly, semantic knowledge transfer is essential for crossdomain NER, but previous works typically can only transfer from single source domain to the target domain and lack the ability of utilizing knowledge from multiple domains to help each other.

All the considerations above lay down our goal to investigate simple yet effective ways to keep one frozen set of PLM parameters for all domains and synthesize knowledge from multiple domains to enhance the target domain performance, as shown in Figure 1. In this work, we enable collaborative domain-prefix tuning for cross-domain NER tasks. As an alternative to fine-tuning the whole PLM, our method, namely CP-NER 1 , tunes no existing PLM parameters. We establish this by casting the NER into a "text-to-text generation grounding domain-related instructor". Compared with LIGHTNER [Chen et al., 2022a] that generates indexes of entity spans, our formulation enables eliciting general knowledge about named entities from PLMs and avoids the external modification of model architecture. By introducing collaborative domain-prefix tuning consisting of a few trainable parameters, we can flexibly adapt the knowledge from multiple source domains to target domains. The specific process consists of three steps: (i) Domain-specific Warmup. This step leverages domain corpus to warm up the corresponding prefix; (ii) Dualquery Domain Selector. Identify which domain could benefit each other from both the perspectives of label similarity and prefix similarity; (iii) Intrinsic Decomposition for Collaborative Domain-Prefix. This module helps synthesize the final powerful prefix frommultiple source domains. In a nutshell, the contributions of our CP-NER are as follows:

• We reformulate NER as the "text-to-text generation grounding domain-related instructor". This new formulation helps trigger PLMs to produce general knowledge about NER tasks and can handle different entity categories without any domain-specific modification of parameters for different domains, laying the groundwork of one model for all domains.
• Our proposal of collaborative domain-prefix tuning for PLMs aims to adapt the knowledge to cross-domain NER tasks. To the best of our knowledge, our work takes the first step to leverage knowledge transfer from multiple source domains specifically for cross-domain NER. The comprehensive nature of the results is crucial for a wide range of cross-domain information extraction applications.
• We obtain SOTA performance on the CrossNER bench-

1 Codes will be available in https://github.com/zjunlp/DeepKE/ tree/main/example/ner/cross.

mark involving transferring knowledge from a single general domain to the target domain. Extensive experiments illustrate that our CP-NER can also efficiently leverage the knowledge transfer via learning lightweight parameters and unified architecture for all domains.

2 RelatedWork

2.1 Cross-Domain NER

Although PLMs [Devlin et al., 2019; Liu et al., 2019; Lewis et al., 2020; Xue et al., 2021] have made waved in NLP, achieving overwhelming performance in widespread NER datasets, yet they require many domain-related labeled data for training when adapting to the target domains. To this end, cross-domain NER algorithms [Kim et al., 2015; Yang et al., 2018; Lee et al., 2018; Lin and Lu, 2018; Liu et al., 2020a] that alleviate the data scarcity issue and boost the models' generalization ability to target domains have drawn substantial attention recently. One approach to this problem is multitask learning, which includes adding auxiliary objects [Liu et al., 2020a; Wang et al., 2020] or designing new model architectures [Jia et al., 2019a; Liu et al., 2020b; Jia and Zhang, 2020] for improving the NER performance of the target domain by training on data from both source and target domain. However, these methods typically require training with vast amounts of labeled source domain data to obtain satisfactory target domain performance. Yet our approach leverages collaborative domain-prefix tuning, which allows for synthesizing parametric knowledge across domains and holds one frozen PLM for all domains.

Another area of research in cross-domain NER focuses on transferring label information across domains. Zheng et al. [2022] models the relation of labels as a probability distribution to better transfer cross-domain knowledge in NER. Hu et al. [2022] proposes an approach to better utilize the semantic relations among domains that jointly predict NER labels by the previous labels (from source) and corresponding token. In contrast, our approach uses descriptive instructions with entity categories' semantics to enable the PLMs to understand NER tasks across different domains without modifying the model architecture. This allows for a single model to be used for all domains, laying the foundation for an efficient and versatile NER system.

2.2 Parameter-Efficient Tuning for PLMs

Pre-trained language models (PLMs) [Devlin et al., 2019; Liu et al., 2019; Lewis et al., 2020; Xue et al., 2021] have been widely used to achieve excellent performance in various NLP tasks [Wang et al., 2018; Wang et al., 2019]. However, finetuning these models requires lots of computational resources and storage. To overcome this challenge, researchers have developed parameter-efficient methods to transfer PLMs to new tasks with small trainable parameters [Ding et al., 2022]. These methods include prompt-tuning [Shin et al., 2020; Chen et al., 2022b], which fine-tunes the model using task-specific prompts; and adaptors [Houlsby et al., 2019; Wang et al., 2021; Newman et al., 2022] which add taskspecific parameters to the PLMs. However, applying these methods to new domains with unseen entity types can be challenging. One type of solution is to utilize prefix-tuning [Li and Liang, 2021a], which prepends a sequence of task-specific vectors to the input. More works explore task adaptation with soft prompt [Vu et al., 2022; Su et al., 2022].

Different from those approaches, our work, referred to as CP-NER, stands out from those approaches in several ways. Firstly, we use collaborative knowledge updating between source and target prefixes to capture domain-related knowledge rather than just tuning extra parameters. Secondly, our method allows for easy redeployment when switching to new domains with unseen entity types by using entity categories as grounding for text-to-text PLMs, unlike previous parameterefficient methods. Finally, our approach aims to utilize multiple source domains to enhance the target domain, specifically focusing on NER, as opposed to transfer across tasks.

3 Methodology

3.1 ProblemDefinition

Given sentence x = {w1, w2, . . . , wn}, the NER task is aimed to extracting all entities y, and n refers to the length of x. Formally, we have the i-th entity yi ∈ y as yi = (xl:r, t), where t ∈ T refers to the type of entity. r and l refers to the right and left entity boundary indexes, respectively. Given the source domain data Dsrc as well as target domain data Dtgt, the cross-domain NER task aims to obtain transferable knowledge with Dsrc to boosts the performance of Dtgt. Previous studies on cross-domain NER mainly focus on transferring from one source domain to one target domain. While we not only consider "One Source for Target", but also include the "Multiple Source for Target" setting, which is practical for on-the-fly cross-domain NER in real scenarios.

3.2 Text-to-Text Generation Grounding Domain-related Instructor

We formulate the cross-domain NER task as sequence-tosequence generation and leverage a frozen T5 [Raffel et al., 2020], as shown in Figure 2. Specifically, for the NER task, we construct the inputs containing the following components:

• Sentence - the input sentence x.
• Instruction - the instruction (s) tells the model how to solve NER task. We ask the model to generate a sequence according to the introduction.
• Domain-related Options - the option (o) includes all entity types T of the current domain and acts as both a constraint and a hint.

Formally, CP-NER takes the given instructor (s), domainrelated option (o) and the text sequence (x) as input and generates the output (y), which contains the extracted information from input sentence based on domain-related instructor:

where θ denotes parameters of the LMθ (we adopt T5 model in this paper). We convert the output sequence into a natural language consistent with the input instructions. To be specific, we leverage simple templates to reformulate the entity (xl:r, t)

to natural language by concatenating entity types, mentions and several special tokens (separators). For example, given the sentence "Johnson, who played eight seasons in Baltimore, was named Orioles manager in the off-season replacing Phil Regan.", the model will generate "((person: Johnson) (location: Baltimore) (organization: Oriloes))".

We argue that the text-to-text generation grounding domainrelated instructor can: 1) effectively guide the T5 model to generate named entity sequence with domain-specific entity categories so that it can be transferred to new domain NER tasks without any structural modification of the T5 model; 2) stimulate the potential of PLM to handle NER tasks with various domains, which laid the foundation for guiding frozen PLM to generate entity sequences through prefix-tuning.

3.3 Insights of Tuned Prefix as Domain Controller

Prefix-tuning [Li and Liang, 2021b] prepends trainable continuous tokens, also known as soft prompts, to each Transformer layer's hidden states. We regard the prefix as the domain controller for cross-domain NER.

As described in Li and Liang [2021b], directly updating the prefix parameters leads to unstable optimization and a slight drop in performance. So the prefix of the j-th layer, δ(j) , is obtained based on the trainable matrix, P(j) , while all the other parameters of PLMwill be fixed. In particular, we derive the variant formula of self-attention with the prefix at the j-th layer as 2 :

(2)

where λ(x(j) ) refers to the scalar regarding the sum of normalized attention weights on the key and value embedding from prefixes.

For simplicity, we denote the prefix indexes as Pidx, then prefix-tuning [Li and Liang, 2021b] formulates the forward propagation at the position n as:

where hi is an activation of the trainable parameter P, z = [PREFIX, x], h(j) i = δ(j) [i, :] for all j = 0 to L-1, i ∈ Pidx and h(0) i = zi for i /∈ Pidx. With the large-scale LM parameters fixed and continuous optimization on training samples, prefix-tuning is expected to steer the LMs to generate correct entity prediction for test data.

Remark 1. As shown in Eq. 2, the domain-specific prefix tuning essentially modifies the original head attention through linear interpolation by a scalar factor (i.e., 1 - λ), acting as a domain controller to drive the model to output domainspecific predictions. However, prefix-tuning still suffers from knowledge forgotten or dilution in the process of direct domain transfer, which inspires us to further explore more effective ways of joining prefix hints to leverage knowledge from multiple source domains.

2 Without loss of generalization, we ignore the scaling factor √d of the Softmax for the ease of representation.

3.4 Collaborative Domain-prefix Tuning

Directly transferring knowledge through prefix-tuning can be challenging because the knowledge learned in the source prefix can easily be forgotten or diluted when fine-tuned on data from the target domain due to differences in text styles and entity categories. To address this issue and help the T5 model's domain controller more effectively capture knowledge from source domains, we propose a collaborative domain-prefix tuning method.

The overall process is illustrated in Figure 2: (1) domainrelated warm-up to capture knowledge from the corpus of the current domain; (2) dual-query domain selector to determine the proportion of prefix knowledge from multiple source domains; (3) intrinsic decomposition for collaborative domainprefix, which can flexibly synthesize prefix knowledge from source and target domains.

Domain-specific warm-up. To flexibly organize prefixes with knowledge from different domains, we first leverage domain corpus to warm up the corresponding prefixes. Formally, we warm up the newly initialized trainable parameter matrix {P(0) , . . . ,P(L-1) } for all L layers of frozen T5 where the final prefix of l-th layer is drawn from the parameter matrix P(l) . We optimize the parameter matrix of prefix at each layer {P(0) , . . . ,P(L-1) } using the training set Dtr. We follow the log-likelihood objective in T5 [Raffel et al., 2020] and abbreviate the optimization objective as follows:

where σ as the softmax scoring function that output h(L) o to a probability vector over the vocabulary. h(j) i refers to the intermediate activation of the j-th layer at step i. We utilize the above optimization objective over the domain corpus to warm up the domain-specific parameter matrix and then generate the domain controller {δ(0) , . . . , δ(L-1) }.

Dual-query domain selector. When the source and target domains share part of the same entity categories, it is intuitive to leverage those shared labels having similar semantic information for flexible adaptation to the target domain. Besides, domain-specific prefixes maintain different grammar styles and themes. Based on the above consideration, we design a dual-query domain selector from both the perspective of label similarity and prefix similarity to identify which domains could benefit each other.

• Entity Similarity: we measure entity similarity through the embedding of the T5 model. Denote the word embedding of T5 as LMw; we feed entity categories of source and target domain into the LMw to obtain the semantic category representations of the i-th source domain as Eisrc ∈ R|T i src |×d and target domain as Etgt ∈ R|Ttgt|×d . Then we conduct pooling over the representation and calculate their similarity matrix as follows:

where Pool is the pooling operation function and S ie ∈ R;

• Prefix Similarity: given the prefix δsrc = {δ(0) src, . . . , δ(L-1) src } and δtgt = {δ(0) tgt , . . . , δ(L-1) tgt } of source and target domain, we also apply cosine similarity function over prefixes to calculate prefix similarity as:

where S ip ∈ R and Avg represents the average function.

For the target domain Dtgt, we denote α as the hyperparameter α ∈ [0 : 1] to determine the ratio between the dual-query similarities as follows:

As for leveraging I source domains for collaborative prefix tuning, we apply the softmax function over the total similarity of multiple source domains as follows:

To pursue compositional generalization and differentiable optimization, we encourage the prefix to be continuously activated and combined. Every single prefix from source domains can be viewed as some fundamental skill and domain-specific knowledge, and NER tasks from another domain can be solved by combining such modular prefixes. Though similar ideas have been proposed in other names and contexts [Sun et al., 2020; Ponti et al., 2022], this is the first work that implements the cross-domain NER with multiple source domains to drive PLMs to recognize named entities.

Intrinsic collaboration for domain-prefix. Based on the selection of source domains to benefit target domains, we propose intrinsic collaboration for domain-prefix, which can synthesize the final powerful prefix from multiple source domains. Specifically, for the warm-up parameter matrix P(j) ∈ Rd from the source domain, we argue it is in the same intrinsic space as the prefix from target domain. We perform intrinsic refactor such as addition operation to the warm-up parameter matrices of the target domain, as shown in Figure 2. The specific parameter matrix P(j) tgt of target prefix at the j-th layer is refactored as:ˆP(j) src = Stotal · [P(j) src1 ; . . . ; P(j) srcI ], (9)

where ˆP(j) src denote the aggregated matrix from multiple domains 3 . We further follows the objective function in Eq.4 to update ˆP(j) tgt while keeping LMθ frozen.

Remark 2. From the optimal control (OC) perspective, prefixtuning can be formalized as seeking the OC of the pre-trained LM for specific domains. And our collaborative domain-prefix tuning can be interpreted as seeking the close-loop control for leveraging knowledge from multiple-source to enhance the target domain NER performance.

4 Experiments

4.1 Experimental Settings

Dataset. We conduct experiments using four publicly available datasets, including CrossNER [Liu et al., 2021b], CoNLL 2003 [Sang and Meulder, 2003], MIT Restaurant [Liu et al., 2013a], and MIT Movie [Liu et al., 2013b]. CoNLL 2003 is a well-known NER dataset that serves as the source domain, which is labeled with four categories: PER, LOC, ORG, and MISC. Additionally, CrossNER contains five separate domain datasets: politics, natural science, music, AI, and literature. We adhere to the official splits for training, validation, and test sets, and the statistics for these datasets can be found in the supplementary materials.

3 For scenarios with only one source domain, we let ˆP(j) src = P(j) src

Baselines. To evaluate the effectiveness of the proposed method, we compare it with several baselines, including:

• DAPT [Liu et al., 2021b]: DAPT employs a large unlabeled corpus related to a specific domain, which is based on BERT and fine-tunes it on the CrossNER benchmark.
• COACH [Liu et al., 2020b]: This method utilizes patterns of slot entities and combines the features for each slot entity in order to improve the accuracy of entity type predictions.
• CROSS-DOMAIN LM [Jia et al., 2019b]: This method utilizes a parameter generation network to merge crossdomain language modeling with NER, resulting in improved model performance.
• FLAIR [Akbik et al., 2018]: This method utilizes the internal states of a character-level language model to generate contextual string embeddings, which are integrated into the NER model.
• BARTNER [Yan et al., 2021]: This approach uses the pre-trained BART model to generate entity spans, treating the NER task as a sequence generation problem.
• LST-NER [Zheng et al., 2022]: This approach models the relationship between labels as a probability distribution and builds label graphs in both the source and target label spaces for cross-domain NER tasks.
• LANER [Hu et al., 2022]: This method introduces a new approach for cross-domain named entity recognition by utilizing an autoregressive framework to strengthen the connection between labels and tokens.
• LIGHTNER [Chen et al., 2022a]: This utilizes a pluggable prompting method to improve NER performance in low-resource settings.

Due to all baselines being reported in base PLMs, we utilize T5-base to conduct a fair comparison for all experiments.

4.2 Transfer from Single Source Domain

As we can see from the w/o DAPT column of Table 1, CPNER can consistently obtain better performance than SoTA method LANER across all target domains, with a 3.56% F1score improvement in the Source & Target setting. Moreover, we find that CP-NER surpasses LANER on the science and music domains, with a 6% to 8% F1-score performance gains on each domain. Experimental results demonstrate that our CP-NER technique achieves superior performance on the CrossNER benchmark. Our approach beats explicitly naive baselines and is on par with the LANER+DAPT model, which adopts an external domain-related corpus for pre-training.

On the other hand, we also apply DAPT on our CP-NER to conduct a fair comparison with other baselines. As Introducing DAPT column of Table 1 illustrates, our method (CP-NER) can better recognize and categorize entities on these domains with DAPT. This is because it leverages many domain-related data, which can boost performance. When introducing DAPT, we observe that CP-NER also outperforms other baselines in all domains, revealing that our method is compatible with DAPT.

4.3 Transfer fromMultiple Source Domains

Previous work for cross-domain NER only focuses on transferring from a single source domain to a target domain, failing to leverage multiple source domains, even though numerous source domains are more in line with practical application scenarios. As shown in Table 2, we experiment with transferring from multiple source domains. We use the CoNLL 2003, Politics, Literature, and Music datasets as source domains and test on the Mit-Movie, AI, Science, and Mit-Restaurant target domains. To compare our CP-NER with other baselines, we propose the following setting for multiple source domains.

Setting. (i) <Ensemble>.Train on a single source domain and transfer to the target domain, then ensemble the models transferred from multiple source domains. (ii) <Chain Transfer>. Continuously train on multiple source domains in sequence and then transfer to the target domain. (iii) CPNER<Multiple Source>. Leverage the warm-up domainspecific prefix to conduct collaborative domain-prefix tuning.

Multiple source makes better transfer learning. According to the results of Table 1 and Table 2, it can be observed that using multiple sources provides consistent improvement for baseline models. However, only LIGHTNER<Ensemble> and LANER<Chain Transfer> achieve little improvements compared with the results in the single source domain setting, revealing that <Ensemble> may be more suitable for lightweight tuning. In contrast, <Chain Transfer> is ideal for fully-tuned models. Furthermore, our CP-NER using collaborative prefix tuning empowers the effectiveness of multiple sources and achieves excellent improvement compared to using a single source. The results in Table 2 show that our model outperforms all baseline methods in both the None source domain and multiple source domain settings, demonstrating that the proposed approach can be beneficial in transferring knowledge learned from the source domain.

Effect of data size. To investigate the influence of the size of the target domain data on performance, we conduct experiments using varying number of target training data (ranging from 10 to 50 samples) based on the settings that yielded the best results. From Table 2, we notice that most of the models achieve performance improvement as the amount of data increases, indicating the essential role of labeled data. Furthermore, it can be seen that our proposed method (CPNER) consistently outperforms the baselines (LANER and LIGHTNER) across all data sizes, further demonstrating the effectiveness of the approach in incorporating information from multiple source domains through collaborative domain-prefix tuning for cross-domain NER.

4.4 Analysis

Analysis of similarity-based selector. We visualize the entity and prefix similarity of the dual-query domain selector to analyze the correlation between source and target domains. From Figure 3, we find that the similarities in the left column are highly correlated with the entity semantic category. For example, the Music domain is most related to the target Movie domain, and they have the same entity category "song" and similar entity categories "genre" and "musicgenre". Furthermore, the entity similarity and the prefix similarity have similar trends, where the source domains with the highest similarity values are roughly the same. These observations reveal that the dual-query selector effectively identifies the most valuable source domains for the target domain.

Case analysis. As shown in Table 3, the models that are transferred from multiple source domains benefit from the source domains and perform more robustly. For example, some entities have already appeared in the source domains and can be easily identified in the target domain. Moreover, the same entities from source domains, such as "Australia" and "Joseph", provide direct support for the target domain, illustrating the effectiveness of our collaborative domain-prefix tuning.

Ablation study. As Figure. 4 illustrates, we conduct an ablation study to evaluate the impact of various model components and source domains on domain transfer performance. Specifically, we systematically eliminate the entity similarity, prefix similarity, warm-up, and domain-related options from the model, respectively. Our results show that the performance of the models decreases significantly, particularly when the entity similarity component is removed. This highlights the crucial role played by our dual-query domain selector in facilitating effective domain transfer. Additionally, we investigate the effect of different source domains on the domain transfer performance. Our findings indicate that each source domain plays a unique role in the domain transfer process and exhibits varying levels of importance for different target domains, verifying the feasibility of leveraging knowledge from multiple domains to improve the target domain performance.

5 Conclusion and FutureWork

In this paper, we target cross-domain NER and propose a lightweight approach dubbed CP-NER. The proposed approach uses collaborative domain-prefix tuning to better utilize knowledge frommultiple domains, thus, leading to better performance for on-the-fly cross-domain NER. Through comprehensive experiments on benchmark datasets, we have shown that CP-NER achieves better results than a series of state-ofthe-art cross-domain learning methods on NER. Furthermore, from the perspective of the application, our CP-NER can realize one model for all domains (Keeping the base LM frozen), which is more in line with the needs of industrial applications. In the future, we plan to extend the proposed approach to multilingual NER and other information extraction tasks. Besides, it is promising to build one unified large-scale generation model with pluggable and programmable modules to address the generalization issues.

Acknowledgments

We would like to express gratitude to the anonymous reviewers for their kind comments. This work was supported by the National Natural Science Foundation of China (No.62206246 and U19B2027), Zhejiang Provincial Natural Science Foundation of China (No. LGG22F030011), Ningbo Natural Science Foundation (2021J190), and Yongjiang Talent Introduction Programme (2021A-156-G), CAAI-Huawei MindSpore Open Fund, and NUS-NCS Joint Laboratory (A-0008542-00-00).

Contribution Statement

Xiang Chen, Lei Li and Shuofei Qiao make an equal contribution and share co-first authorship. Ningyu Zhang and Huajun Chen are both the corresponding authors.