Exploring Effective Inter-Encoder Semantic Interaction for Document-Level Relation Extraction

1 Introduction

Relation extraction (RE) is an important task in the community of information extraction (IE), which aims to identify the relations between entities in a given text. While most previous studies focused on extracting relational triples from a single sentence [Zeng et al., 2015; Zhang et al., 2018; Baldini Soares et al., 2019], i.e., sentence-level RE, many researchers have recently begun to explore RE at the document level [Zeng et al., 2020; Zhou et al., 2021; Jiang et al., 2022]. Unlike sentence-level RE, document-level RE aims to extract all relation triples from the input document. It is usually more challenging since documents normally contain a large number of implicit relations that can only be identified with the help of relational reasoning. According to the statistics in [Yao et al., 2019], in the commonly-used DocRED dataset, about 61.6% of relational facts can only be correctly predicted with the assistance of relational reasoning.

Due to the advantages of graph neural networks (GNNs) in relational reasoning, graph-based methods are widely adopted in document-level RE [Zeng et al., 2020; Xu et al., 2021b; Peng et al., 2022]. These methods first use a pretrained language model (PLM) as the encoder to obtain the document's contextual representation, and then leverage dependency structures, heuristics, or structured attention to construct a document graph [Peng et al., 2017; Christopoulou et al., 2019; Nan et al., 2020]. Finally, GNNs are applied to encode the document graph for relational reasoning.

Although graph-based methods have achieved competitive performance in document-level RE [Zeng et al., 2020; Peng et al., 2022], they still suffer from several drawbacks. First, in these methods, the interaction between document encoder (PLM) and graph encoder (GNN) is normally insufficient and unidirectional, severely limiting the performance of the model. Specifically, when encoding the document graph, these methods typically only consider entities while ignoring non-entity words that might provide crucial clues for relational reasoning. This suggests that the semantic transition from the document encoder to the graph encoder (PLM→GNN) is insufficient. As illustrated in Figure 1, since the entities "Blackadders" and "Richard Curtis" appear in different sentences, it is usually difficult to correctly predict their relation only using their own information. Note that the non-entity word "episode" respectively co-occurs with these two entities in a different sentence, and their relations (episode, be part of, Blackadders) and (Richard Curtis, wrote, episode) can be easily identified. With these two relations, we can further infer the "creator" relation between "Blackadders" and "Richard Curtis" (See Figure 1(c)). Besides, these methods do not directly pass the structural information of the graph encoder to the document encoder (GNN→PLM), resulting in that the document encoder cannot directly benefit from the graph encoder [Xu et al., 2021a]. Second, during document graph encoding, these methods usually update the node representations by only aggregating the information of their neighbor nodes. However, this approach only focuses on capturing the local context of a considered node while neglecting its global context [Hu et al., 2019; Wan et al., 2021; Wang et al., 2021], which significantly reduces the reasoning ability of the model.

To deal with the above issues, we propose a graph-based document-level RE model with the bidirectional semantic interaction between the document encoder and the graph encoder. As shown in Figure 2, we first use a PLM encoder to encode the input document. Then, on the top of the encoder, we construct a heterogeneous document graph (HDG) that consists of three types of nodes, namely mention node, entity node, and document node, and three types of edges, i.e., intrasentence edge, intra-entity edge, and document edge. Lastly, we propose a new graph encoder, Graph-Transformer Network (GTN), to encode HDG and generate more expressive entity representations. Particularly, our GTN includes two core sublayers: the graph-attention sublayer and the crossattention sublayer. The former is a multi-head self-attention variant with four attention heads, where the first three attention heads are applied to capture the local context of a considered node from its neighbors, and the last one is used to capture the global context of the node from all other nodes. Furthermore, via the cross-attention sublayer, GTN can capture the non-entity clue information from PLM encoder to enhance the reasoning ability of the model.

To effectively enhance the bidirectional semantic interaction between the PLM encoder and our GTN, we introduce two auxiliary tasks into our model training: the graph node reconstruction and the Structure-aware Adversarial Knowledge Distillation (SA-KD). To implement graph node reconstruction, we first mask the feature vectors of some nodes in HDG, and then train GTN to reconstruct the original features of these nodes. In this way, we can effectively train GTN to obtain more clue information from PLM encoder via the cross-attention sublayer. Besides, we employ SA-KD

to guide the PLM encoding using the structural information of GTN. Specifically, we develop a discriminator to distinguish the node representations generated by PLM encoder and GTN. Meanwhile, we regard PLM encoder as the generator (student), which is trained to produce node representations conforming to the distributions of GTN (teacher), so that the discriminator cannot distinguish. By alternately optimizing the discriminator and the generator, our model can effectively transfer the structural information of GTN to PLM encoder. Note that our discriminator is more tolerant than predefined distance functions in conventional knowledge distillation, such as cosine distance, because it does not require PLM encoder and GTN to output the exact same node representation. By doing so, we can effectively prevent the model training from collapsing.

To demonstrate the effectiveness and generality of our model, we conduct comprehensive experiments on four public datasets, of which results show that our model consistently outperforms all competitive baselines.

2 Methodology

In this section, we describe in detail our model and its training. As illustrated in Figure 2, our model consists of two components: the PLM encoder and the Graph-Transformer Network (GTN). First, the input document is encoded with PLM encoder to obtain contextual representations of tokens (Section 2.1). Then, on the basis of these representations, we construct a Heterogeneous Document Graph (HDG) and encode it using GTN to produce more expressive entity representations (Section 2.2). Finally, we give a detailed description of our model training (Section 2.3).

2.1 The PLMEncoder

Following [Zhou et al., 2021; Tan et al., 2022a], we first use the special token "∗" to mark the start and end of mentions in the input document D. Then, we encode D using PLM encoder to obtain contextual representations H ∈ R|D|×d of tokens, where d is the dimension of PLM encoder hidden states. Finally, we take the contextual representation of the token [CLS] as the document's contextual representation hD, and the contextual representation of "∗" at the start position of the mention mj as its contextual representation h(mj). Particularly, we merge all the mention representations of the entity ei via logsumexp pooling [Jia et al., 2019] to generate its global contextual representation h(ei) = log ∑Nei j=1 exp(h(m i j )), where Nei refers to the mention number of ei.

2.2 The Graph-Transformer Network

Graph Representation

To model dependencies among entities or mentions, we first heuristically construct an HDG. Our HDG includes three types of nodes: mention node, entity node, and document node. We initialize the feature vectors of these nodes using the contextual representations of mention, entity, and document obtained from PLM encoder, respectively. Meanwhile, we introduce three types of edges into HDG:

• Intra-Entity Edge. We introduce intra-entity edges to fully connect mention and entity nodes of the same entity. This allows us to efficiently aggregate different mention representations of entities to generate better entity representations.
• Intra-Sentense Edge. The intra-sentence edges are utilized to fully connect mention and entity nodes that cooccur in the same sentence. In this way, we can effectively model the interaction among different entities.
• Document Edge. All mention and entity nodes are connected to the document node via document edges. By using this type of edge as pivots, we can effectively improve the semantic interaction between distant entities.

Graph Encoding

Then, we encode the HDG with the GTN to generate more expressive entity representations. To facilitate the calculation of GTN, we combine the feature vectors of all nodes into a feature matrix F (0)∈RN×d , where N represents the number of nodes. Concurrently, we build an adjacency matrix Ek∈RN×N (k∈{1, 2, 3}) for each type of edge.

As shown in Figure 2, our GTN contains L identical layers, each of which consists of four sublayers: the graph-attention sublayer, the cross-attention sublayer, the feed-forward neural network sublayer, and the layer normalization sublayer. Next, we detail the two core sublayers of GTN, i.e., the first two sublayers.

Graph-Attention Sublayer. This sublayer is a variation of multi-head self-attention with four attention heads, where the first three heads are used to capture the local contexts of nodes in HDG, and the fourth head is used to capture the global contexts of nodes in HDG. With the help of this sublayer, GTN can simultaneously model the local and global contexts of nodes in HDG.

We use the first three attention heads to model the three types of edges in HDG, respectively. Specifically, for each type of edge, we utilize its adjacency matrix Ei as the attention mask matrix in the corresponding attention head. Formally, at the (l+1)-th layer, the i-th attention head is calculated as follows:

where W K i , W Q i , and W V i are trainable parameters, Inf refers to infinity. Obviously, by introducing Inf, each node is limited to only focus on its neighbors in HDG.

Specially, in the fourth attention head, we do not perform the mask operation, which allows each node to pay attention to all other nodes. To prevent losing structural information of HDG in this head, we add the entity embedding embe and the sentence embedding embs to each node:

where W K 4 , W Q 4 , andW V 4 are parameter matrixes.

Cross-Attention Sublayer. Through this sublayer, we expect that GTN can extract clue information from PLM encoder to improve the model's reasoning abilities. To adapt to long documents and capture more diverse clue information, we develop two types of attention heads in the cross-attention sublayer: global attention head and local attention head. In global attention head, the nodes in HDG can consider all the words in the document to capture global clue information. In local attention head, via the mask operation, each node can only focus on the words in the sentence where it is located, so as to capture local clue information.

Based on the final output F (L) of GTN, we utilize a bilinear classifier to predict the relations of entity pairs:

where Wr, Ws, and Wo are trainable parameters, F (L) [es] and F (L) [eo] denote the feature vectors of entities es and eo, respectively, and cs,o represents the localized context embedding [Zhou et al., 2021] utilized to enhance the representation of entity pair (es,eo). More specifically, cs,o is computed as

where As and Ao denote the PLM last-layer attention weights of entities es and eo to all tokens in the document, respectively, and ◦ refers to element-wise multiplication.

2.3 Model Training

To effectively enhance the bidirectional semantic interaction between PLM encoder and GTN, we introduce two auxiliary tasks into our model training: the graph node reconstruction and the Structure-aware Adversarial Knowledge Distillation (SA-KD). Thus, the final training objective of our model contains three loss items: the relation classification loss LR, the graph node reconstruction loss LN , and the SA-KD loss LA:

where α and β are hyper-parameters, which are empirically set to 0.1 and 0.01, respectively.

Relation Classification Loss LR. To alleviate the imbalance relation distribution issue in document-level RE, we adopt the adaptive thresholding loss [Zhou et al., 2021] as our relation classification loss. Specifically, we introduce a special relation class TH and use its logits logitTH as the adaptive threshold value for each entity pair to distinguish between positive relations PT and negative relations NT :

NodeReconstructionLoss LN. To effectively improve the semantic transition from PLM encoder to GTN, we introduce the graph node reconstruction task into our model training. Specifically, we randomly sample and mask some nodes in HDG, and then train GTN to reconstruct the initial features of these nodes. To recover the original features of masked nodes, GTN has to leverage its cross-attention sublayer to capture more clue information from PLM encoder. Formally, we define the node reconstruction loss as follows:

Where Vmask represents the masked nodes.

SA-KD Loss LA. The purpose of the SA-KD is to transfer the structural information in GTN to PLM encoder. As shown in Figure 3, we first construct a node feature matrix ̂ F (l) from the l-th layer output of PLM encoder. Then, we develop an MLP discriminator C to distinguish whether the considered node is from the output ̂ F (l) of PLM encoder or the output F (L) of GTN. Finally, we use binary cross entropy loss as LA:

where C(·)=Sigmoid(MLP(·)). Meanwhile, we regard PLM encoder as the generator and train it to produce node representations conforming to the distributions of GTN, so that the discriminator cannot distinguish. In this way, we can guide PLM encoder to learn more expressive entity representations using the structural information of GTN. Notably, we simultaneously distill the structural information in GTN to multiple intermediate layers {l} of PLM encoder using multiple distinct discriminators. Specifically, we empirically set {l} to {6, 12} in the BERT encoder and {12, 18, 24} in the RoBERTa-large encoder.

Finally, during model training, PLM encoder and GTN are trained to minimize L, while the discriminator is trained to maximize LA.

3 Experiments

3.1 Datasets and EvaluationMetrics

We evaluate our model on four commonly-used datasets:

• DocRED [Yao et al., 2019] is a large-scale documentlevel RE dataset with 96 predefined relations, which is constructed from Wikipedia and Wikidata. It contains 5,053 documents, which is divided into 3,053 documents for training, 1,000 for development, and 1,000 for test. Since DocRED contains a considerable number of false-negative samples, we also conduct experiments on its two revised versions, i.e., Revisit-DocRED [Huang et al., 2022] and Re-DocRED [Tan et al., 2022b].
• DWIE [Zaporojets et al., 2021] is an entity-centric multi-task dataset containing 602 documents for training, 98 for development, and 99 for test. In this dataset, there are about 26% of entity pairs expressing more than one relation of the predefined 62 target relations. We followed [Ru et al., 2021] to preprocess the DWIE dataset.

Following previous studies [Yao et al., 2019; Ru et al., 2021], we utilize micro F1 and micro Ign F1 as our evaluation measures. Ign F1 denotes the F1 score excluding the relational facts that are shared by the training and development/test sets.

3.2 Settings

Our model is developed based on Huggingface's Transformers [Wolf et al., 2020] and PyTorch. We use cased BERT-base [Devlin et al., 2019] or RoBERTa-large [Liu et al., 2019] as our encoder. To optimize our model, we use AdamW [Loshchilov and Hutter, 2019] as our optimizer, which is equipped with a weight decay of 1e-4 and a linear warmup [Goyal et al., 2017] for the first 6% training steps. All hyperparameters are tuned on the development set.

3.3 Baseline Models

We compare our model with the existing Transformer-based and Graph-based models.

• Transformer-based models directly employ PLMs to learn better entity representations for document-level RE, including BERT-TS [Wang et al., 2019], HINBERT [Tang et al., 2020], CorefBERT [Ye et al., 2020], and ATLOP-BERT [Zhou et al., 2021].
• Graph-based models leverage GNNs to enhance the reasoning ability of document-level RE models, including EoG [Christopoulou et al., 2019], DHG [Zhang et al., 2020], GEDA [Li et al., 2020], LSR [Nan et al.,

2020], GLRE [Wang et al., 2020], GAIN [Zeng et al., 2020], HeterGSAN [Xu et al., 2021b], SIRE [Zeng et al., 2021], and SSAN [Xu et al., 2021a].

In addition, we select several recent competitive models, including DocuNet [Zhang et al., 2021], KD-DocRE [Tan et al., 2022a], and KMGRE [Jiang et al., 2022] for comparison.

3.4 Effect of GTN Layer Number L

To illustrate the influence of the hyper-parameter L on our model, we report the performance of our model with different GTN layer numbers in Figure 4. Like previous graph-based methods [Zeng et al., 2020; Peng et al., 2022], our model achieves the best performance when L is set to 2. Meanwhile, we also note that the performance of our model is not so sensitive to L. Finally, we set L=2 in all subsequent experiments.

3.5 Main Results

Results on RE-DocRED and DocRED. As illustrated in Table 1, our model consistently outperforms all baselines on RE-DocRED and DocRED datasets. Moreover, we draw several interesting conclusions:

First, compared with the improvements on DocRED, our model achieves greater gains on RE-DocRED that contains more relational facts involving relational reasoning. It suggests that our model indeed performs better in reasoning scenarios.

Second, compared with the graph-based SOTA model, GAIN-BERT, our model obtains improvements of 2.33 F1 and 1.51 F1 points on the test sets of RE-DocRED and DocRED. These results demonstrate that our model can better capture the dependencies among entities and mentions to improve the reasoning ability of the model.

Third, our model also surpasses recent SOTA models, including DocuNet-BERT and KD-DocRE-BERT, which leverage the dependencies among entity pairs to enhance their reasoning abilities. This fully illustrates again the excellent reasoning ability of our model.

Results on Revisit-DocRED. Unlike DocRED and REDocRED, the training set of Revisit-DocRED contains a large number of false-negative samples, but its test set does not. As illustrated in Table 2, our model consistently and significantly outperforms all competitive baseline models on this datasets, demonstrating that our model is robust to noisy data.

Results on DWIE. To confirm the generalizability of our model, we also conduct experiments on the DWIE dataset. From Table 2, we find that our model significantly outperforms KD-DocRE-BERT by 1.63 Ign F1 and 1.54 F1 points on the test sets of DWIE, achieving new SOTA performance on this dataset.

3.6 Ablation Study

To further comprehend the contributions of different components on our model, we conduct an ablation study by removing different components from our model. Specifically, we compare our model with the following variants in Table 3.

(1) SA-KD→KD. In this variant, we replace our SA-KD with the conventional KD that employs cosine similarity as the predefined distance function to measure the gap between nodes generated by PLM encoder and GTN. From Line 3 in Table 3, we observe that this replacement results in a significant performance drop. The possible explanation is that the predefined distance functions in the conventional KD cause the collapse of model training during knowledge distillation, whereas our SA-KD can effectively avoid this collapse issue.

(2) w/o Node reconstruction task. When node reconstruction task is removed from our model training, the performance of our model drops by 0.72 Ign F1 and 0.68 F1 points (See Line 4 in Table 3). This result demonstrates that this task can effectively train our cross-attention sublayer to extract more clue information from PLM encoder.

(3) w/o Local head in cross-attention and w/o Global head in cross-attention. To capture diverse clue information and adapt to long documents, we equip the cross-attention sublayer with two types of attention heads: local and global attention heads. To demonstrate this, in these two variants, we respectively remove the local and global attention heads from our cross-attention sublayer, both of which negatively impact the performance of our model (See Line 5-6 in Table 3).

(4) w/o Cross-attention. In this variant, we exclude the cross-attention sublayer from our model, which results in a performance decrease in our model (See Line 7 in Table 3). This suggests that our cross-attention sublayer can effectively capture non-entity clue information from PLM encoder to improve the reasoning ability of the model.

(5) w/o The fourth head in graph-attention. To investigate the effectiveness of the global context of nodes in HDG, we remove the fourth attention head from our graph-attention sublayer. As shown in Line 8 of Table 3, this variant causes a significant performance decline, which confirms the contribution of node global context on the model performance.

(6) Graph-attention→Self-attention. In this variant, we replace our graph-attention sublayer with a standard multi-head self-attention sublayer, where each attention head acts on a fully connected graph. This change leads to a significant performance drop of 2.02 Ign F1 and 2.07 F1 points (See Line 9 in Table 3). For this result, we speculate that the standard multi-head self-attention sublayer loses the structural information of HGD and causes the over-smoothing problem.

3.7 Analysis of Reasoning Performance

To further illustrate the reasoning ability of our model, following [Nan et al., 2020; Zeng et al., 2020], we also report Intra-F1, Inter-F1 and Infer-F1 scores. When calculating Intra-F1 and Inter-F1, we solely consider intra-sentence and inter-sentence relations, respectively. Meanwhile, we calculate the Infer-F1 score using the test files supplied by Zeng et al., [2020]. This metric aims to assess the ability of the model in multi-hop reasoning.

As shown in Table 4, our model outperforms all baseline models on Intra-F1 and Inter-F1 metrics. We notice that our model achieves more significant improvements on Inter-F1 than on Intra-F1, demonstrating that our model is excellent at extracting inter-sentence relations. In addition, extracting inter-sentence relations is usually more challenging than intra-sentence ones, so that Inter-F1 can more effectively reflect the reasoning ability of the model.

From Table 5, in term of Infer-F1 metrics, we observe that our model also obtains a significant improvement compared to all baseline models. Specifically, our model yields an improvement of 3.03 Infer-F1 points over GAIN-BERT that is the graph-based SOTA model. Meanwhile, removing either cross-attention sublayer or SA-KD from our model causes a significant decline in our model's performance (See Line 7-8 in Table 5). Furthermore, when we replace our graphattention sublayer with a standard multi-head self-attention sublayer, our model performance sharply drops by 3.63 InferF1 points (See Line 9 in Table 5). These results demonstrate that each component in our model can enhance the reasoning ability of the model.

4 RelatedWork

Recently, document-level RE has attracted an increasing amount of interest. The dominant methods for documentlevel RE can be roughly divided into Transformer-based methods and graph-based methods.

Transformer-based Methods. Since PLMs have achieved striking success in natural language processing (NLP), some researchers directly employ Transformer-based PLMs for document-level RE, which focus on extracting more useful information from PLM to enhance the representations of entities [Wang et al., 2019; Tang et al., 2020; Zhou et al., 2021; Zhang et al., 2022; Zhang et al., 2023]. For example, Wang et al., [2019] propose a two-step process for document-level RE. They first identify whether entity pairs are related, and then predict their relations. Zhou et al., [2021] introduce two techniques, i.e., adaptive thresholding loss and localized context pooling, to alleviate the class imbalance issue and enhance the representations of entity pairs, respectively. However, these methods do not explicitly model the dependencies among entities, which limit the reasoning ability of the model.

Graph-based Methods. In recent years, GNNs have been widely used in various NLP tasks, such as machine translation [Song et al., 2020; Yin et al., 2020b] and sentence ranking [Yin et al., 2019; Yin et al., 2020a; Lai et al., 2021]. To effectively model dependencies among mentions or entities, many researchers also introduce GNNs into documentlevel RE [Christopoulou et al., 2019; Nan et al., 2020; Zeng et al., 2020]. These methods first construct a document graph with heuristics or dependency information, and use entities or mentions as its nodes. Then, they encode this document graph using GNNs to obtain more expressive entity representations. For example, Nan et al., [2020] propose a latent structure refinement model, which dynamically induces the latent graph structure to facilitate the relational reasoning across sentences. Zeng et al., [2020] construct two graphs of different granularity, i.e., mention-level graph and entitylevel graph, to model the interactions among mentions and entities, respectively. Meanwhile, to capture the global context of nodes in document graph, Xu et al., [2021b] and Peng et al., [2022] heuristically incorporate some inference paths into the document graph to enhance the interaction among distant related entities. However, these heuristics are generally incomplete and have poor generalizability. Furthermore, to improve the encoder with the structural information of the document graph, Xu et al., [2021a] incorporate the graph structure into the self-attention of PLM encoder. Nevertheless, this approach introduces many new parameters into PLM encoder, which makes it require a large amount of external data for model training. Notably, graph-based methods usually only consider entity information during relational reasoning while ignoring many non-entity clue information in document, which hinders the further improvement of the model's reasoning ability.

Our work falls into the category of graph-based methods. Specifically, our model consists of PLM encoder and GTN that contains two core components, i.e., graph-attention sublayer and cross-attention sublayer. Through the first sublayer, GTN can simultaneously model the local and global contexts of nodes in the document graph. Meanwhile, we introduce a graph node reconstruction training task, which can effectively train GTN to capture more clue information from PLM encoder via the cross-attention sublayer. Furthermore, inspired by recent studies on knowledge distillation [Chung et al., 2020; He et al., 2022; Zhuang et al., 2022], we propose a SA-KD training task. With this task, we can guide PLM encoder with the structural information of GTN to learn more expressive entity representations. Unlike the conventional adversarial knowledge distillation [Chung et al., 2020; He et al., 2022], we use our GTN as the teacher model and the PLM encoder as the student model, allowing GTN and PLM encoder to promote each other during the distillation process.

5 Conclusion and FutureWork

In this paper, we propose a document-level RE model consisting of a PLM encoder and a GTN. Particularly, GTN contains two core components: 1) the graph-attention sublayer that simultaneously models global and local contexts of nodes in HDG; 2) the cross-attention sublayer, which enables GTN to capture the non-entity clue information from PLM encoder. Moreover, to enhance the bidirectional semantic interaction between PLM encoder and GTN, we introduce two auxiliary tasks into model training: 1) the graph node reconstruction that can effectively train our cross-attention sublayer to enhance the semantic transition from PLM encoder to GTN; 2) the SA-KD, by which we can effectively transfer the structural information of GTN to PLM encoder. Experimental results on four commonly-used datasets illustrate that our model outperforms all existing competitive baselines.

In future, we plan to apply our model to other graph-based tasks, such as knowledge graph completion and graph node classification, so as to verify its generality.

Acknowledgments

The authors would like to thank the three anonymous reviewers for their comments on this paper. This research was supported in part by the National Natural Science Foundation of China under Grant Nos. 62076211, U1908216, 62276219, and 61573294.

Contribution Statement

Of all the authors, Liang Zhang and Zijun Min make equal contributions and share co-first authorship. Meanwhile, Jinsong Su and Yidong Chen are the corresponding authors of this paper.