Recall, Retrieve and Reason: Towards Better In-Context Relation Extraction

1 Introduction

The emergence of large language models (LLMs) such as GPT-3 [Brown et al., 2020] represent a significant advancement in natural language processing (NLP). Instead of following a pre-training then fine-tuning pipeline [Radford et al., 2019; Devlin et al., 2019; Liu et al., 2019; Raffel et al., 2020], which fine-tunes a pre-trained model on a taskspecific dataset in a fully-supervised manner, LLMs employ a new paradigm known as in-context learning (ICL) [Brown et al., 2020] which formulates an NLP task under the paradigm of language generation and makes predictions by learning from a few demonstrations. LLMs with ICL demonstrate impressive performance comparable to traditional methods even with limited examples [Brown et al., 2020; Zhao et al., 2021].

∗Corresponding author

Despite the generally promising ICL performances [Wei et al., 2022b; Arora et al., 2023; Shang et al., 2024], current ICL in relation extraction (RE) [Li et al., 2024] task suffers from relatively poor performance. RE is a pivotal task in NLP [Li et al., 2022; Wang et al., 2023a; Wang et al., 2023b; Ji et al., 2023; Liu et al., 2024], necessitating a profound comprehension of natural language, which involves identifying a pre-defined relation between a given entity pair mentioned in the input sentence or marking it as NA if no relation is identified. Given a test example input, ICL for RE prompts the input of LLMs with a few demonstrations retrieved from the training data and the test input itself, then LLMs generate the corresponding relation. Recent studies [Jimenez Gutierrez et al., 2022; Ma et al., 2023] have revealed a significant performance gap in LLMs when apply ICL to the RE task. The main obstacles that utilizing ICL for RE with LLMs are twofold: (i) the low relevance regarding entity and relation in the retrieved demonstrations for ICL [Wan et al., 2023], and (ii) the failure of utilizing LLMs with moderate size (less than 10B) compared to 175B GPT-3 for ICL [Li et al., 2023c].

Typically, ICL demonstrations are selected randomly or via similarity-based sentence embedding obtained by sentence encoder such as Sentence-BERT [Reimers and Gurevych, 2019]. However, similarity-based retrieval is more concerned with the relevance of the overall sentence semantics and not as much with the specific entities and relations it contains, where consequently the test input tends to retrieve a semantically similar sentence but is not desired in terms of entities and relations. Similar with finding good demonstrations in ICL for general NLP tasks [Liu et al., 2022], the demonstrations for RE should contain as much as similar or exactly same entities and relations regarding the test example, so as to significantly help the test example infer the relation between entities. However, current retrieval techniques [Gao et al., 2021; Wan et al., 2023] fail to locate the delicate demonstrations considering the consistency of ontology (i.e., gen-

Demonstrations

eralized entity types and the relations between them). Since entity types grouped under the similar ontology are often correlated and share similar relations [Roche, 2003], we assume that examples sharing the similar ontology are relevant and could serve as good demonstrations for ICL. To this end, we propose to retrieve the relevant demonstrations in RE incorporating consistent ontological knowledge. Figure 1 illustrates the basic principles of our demonstration selection method, where we expect the demonstrations to include as many entity pairs as possible that express the same relation (i.e., similar ontology) with the test example. To obtain such ontological knowledge guided demonstrations, we propose to let the LLMs learn to generate relevant entity pairs that share the same relations with each test example, by implicitly acquiring the ontological knowledge of specific RE task during training phase. Then ICL demonstrations are retrieved from training examples by these recalled entity pairs. Compared to similarity-based retrieval, recalling entities explicitly via implicitly ontological knowledge guiding bring more benefits.

Besides the demonstration selection, the parameter scale of LLMs also plays an important role in ICL. Existing promising RE results obtaining via ICL all use very large-scale proprietary models [Agrawal et al., 2022; Li et al., 2023c]. Although smaller open-source LLMs such as LLaMA [Touvron et al., 2023] can also perform ICL similarly in simple classification tasks like sentiment classification, they are unable to perform more challenge tasks like extracting relations between entities through in-context demonstrations. Due to the limitations of proprietary models (higher computation cost and time consumption, concerns about privacy protection and local deployment, etc.), open-source smaller LLMs (less than 10B) achieve better ICL performance is of more applicability. Therefore, recent studies [Chen et al., 2022b; Min et al., 2022b] attempt to boost the moderate size LLMs ICL ability via meta in-context learning where an LLM is tuned to do in-context learning on a large set of training tasks. Inspired by this, we optimize LLMs to do in-context RE on training set, making the model becomes more effectively to reason about the relation between entities in-context by conditioning on a few training examples at inference time. Compared to directly optimize the input-output formats of examples (sentence and entity pair as input and relation label as output), this method can conduct in-context reasoning based on retrieved examples and generate more accurate results.

To alleviate the above issues of irrelevant demonstration retrieval and inferior ICL capability, we present a recallretrieve-reason framework, a novel RE method called RE4 (Relation Extraction with REcall, REtrieve and REason) that synergizes open-source LLMs with retrieval corpora (training examples) to retrieve relevant demonstrations and conduct in-context reasoning. Specifically, RE4 first generates entity pairs guided by ontological knowledge and grounded by retrieval corpora as valid queries via the recalling module. These entity pairs are then used to retrieve valid training examples from retrieval corpora in retrieval module to conduct in-context reasoning by reasoning module. In this way, we not only retrieve the relevant demonstrations from retrieval corpora but also consider the guidance of entity pairs and relations for reasoning. Based on this framework, RE4 is joint optimized by two tasks: 1) recalling optimization, where we distill ontological knowledge from retrieval corpora into LLMs to generate relevant and valid entity pairs as queries; and 2) reasoning optimization, where we enable LLMs to conduct in-context reasoning based on retrieved demonstrations and generate predicted relations. We conduct extensive experiments on RE benchmarks to validate the effectiveness of RE4 . In summary, our contributions are three-fold:

• We propose a novel recall-retrieve-reason framework for RE that synergizes open-source LLMs with training examples to retrieve relevant demonstrations and conduct in-context reasoning. Moreover, the recalling module of RE4 can be plug-and-play with different LLMs during inference to improve their performance.
• We distill the consistently ontological knowledge from training examples to guide LLMs for valid and relevant entity generation, and propose to boost the open-source LLMs in-context reasoning for RE via ICL tuning.
• We conduct extensive experiments on RE benchmarks and the results demonstrate that RE4 achieves state-ofthe-art performance in sentence-level RE.

2 Methodology

2.1 Task Formulations

For a given natural language sentence s contains N words, relation extraction aims at extracting the pre-defined relationship r between h and t, where h and t are two given target entities in s. If there is no pre-defined relation between h and t, predict NA. Following previous works [Cabot and Navigli, 2021; Paolini et al., 2021; Li et al., 2023b], we treat RE as a generation task following the standard decoding manner. The optimization procedure can be written as follows:

where we use an auto-regressive transformer [Vaswani et al., 2017] decoder to generate the relation name r with label length L, and p(yl|s, h, t, y<l) denotes the probability of each token y in r generated by LLMs.

2.2 Framework Overview

Recently, many techniques have been explored to improve the ICL ability of LLMs in RE, which first retrieves relevant demonstrations from training examples and then conduct incontext RE based on them [Wan et al., 2023]. However, the challenges of low-quality demonstration retrieval and inferior ICL ability of open-source models hinder the progress of ICL for RE in the era of LLMs. To address these issues, we propose a novel recall-retrieve-reason framework, which recalls relevant entity pairs, retrieves good demonstrations and then perform in-context reasoning for better relation predictions. The overall framework of RE4 is illustrated in Figure 2. Given a test example "Carnegie Mellon University (CMU) is located in Pittsburgh, Pennsylvania, United States", aiming to identify the relation between "Carnegie Mellon University (CMU)" and "Pittsburgh, Pennsylvania, United States", we generate an entity pair "Stanford University" and "Palo Alto, California" as the query. This entity pair expresses the similar relation with the test entity pair as the consistency of ontology between them (i.e., the head entity type is Organization and the tail entity type is Location). Then we retrieve the corresponding demonstration from training examples. Finally, we predict the relation (i.e. Located in) based on demonstrations.

2.3 Framework Optimization

We formulate our RE4 as an optimization problem that aims to maximize the probability of reasoning the relation r from a retrieval corpora C w.r.t the test example e by generating consistently ontological entity pairs z as the queries:

where θ denotes the parameters of LLMs, z denotes the entity pairs (queries) generated by LLMs, and Z denotes the set of entity pairs that share similar ontology with the test example e. The latter term pθ(z|e) is the probability of generating a valid entity pair z grounded by retrieval corpora given e, which is realized by the recalling module. The former term pθ(r|e, z, C) is the probability of reasoning the relation r in context given the test example e, entity pair z, and retrieval corpora C, computing by the reasoning module.

Despite the advantage of generating entity pairs as queries, the LLMs have zero ontological knowledge of the entities and relations contained in training examples. Therefore, LLMs cannot directly generate entity pairs grounded by retrieval corpora as valid queries. Moreover, LLMs might not have strong ICL ability to conduct effective in-context reasoning based on them. To address these issues, we design two instruction tuning tasks: 1) recalling optimization, which distills the consistent ontological knowledge from training examples into LLMs to generate valid entity pairs as queries, and 2) reasoning optimization, which enables LLMs to perform in-context reasoning based on retrieved demonstrations. The objective function in equation (2) is optimized by maximizing the evidence lower bound, which is formulated as:

where q(z) denotes the posterior distribution of valid and relevant entity pairs grounded by retrieval corpora. The latter term minimizes the KL divergence between the posterior and the prior, which encourages LLMs to generate consistently ontological entity pairs (i.e. recalling optimization). The former term maximizes the expectation that reasoning module generates correct relations based on the retrieved demonstrations (i.e. reasoning optimization).

2.4 Recalling Entity Pairs

To make LLMs generate valid entity pairs as queries for retrieving consistently ontological demonstrations from retrieval corpora, we minimize the KL divergence with the posterior distribution of valid entity pairs q(z), which can be approximated by the valid entity pairs in retrieval corpora C.

Given a test example e and its golden relation r, we could find the entity pair instances z = (h, t) expressing the same relation r in training examples. And z can be considered valid and serve as a query for retrieving the relevant demonstration of e. And q(z) can be formally approximated as:

where we assume a uniform distribution over all consistently ontological entity pairs Z regarding e, and ∃z ∈ C denotes the existence of an entity pair instance connecting e and r in C. Therefore, the KL divergence can be calculated as:

where we use the partial entity pairs Z ∗ ⊂ Z relevant to e in retrieval corpora C as supervision signals [Luo et al., 2023], where we distill the consistently ontological knowledge from training examples to LLMs. To utilize the instructionfollowing ability of LLMs [Wei et al., 2022a], we design an instruction template that prompts LLMs to generate k = |Z ∗ | entity pairs. Therefore, the optimization of Lrecall becomes:

where pθ(z|e) denotes the prior distribution of generating valid entity pair z, and pθ(yi|y<i, e) denotes the probability of each token in z generated by LLMs.

2.5 Retrieving FromCorpora

Given a test example e and an entity pair as query z, the retrieving module aims to retrieve the relevant demonstration d from retrieval corpora. The retrieval process can be simply conducted by exact match between the generated entity pair z and the entity pair of each candidate demonstration d:

where D denotes the set of retrieved demonstrations for ICL. Despite we can utilize the retrieved entity pairs to directly get the predicted relations via majority vote, the retrieved entity pairs of relations could be noisy and imperfect to the test example e, leading to incorrect predictions. Therefore, we use a reasoning module to boost the ICL ability of LLMs to identity the important entity pairs of relations and predict relations based on them via in-context reasoning [Chen et al., 2023].

2.6 Reasoning Through Demonstrations

In reasoning module, we aim to enable LLMs to conduct incontext reasoning based on the relevant entity pairs. And reasoning on multiple relevant entity pairs is formulated as:

By approximating the expectation with k sampled entity pairs Z ∗ , the objective function of reasoning optimization where maximizes the probability of LLMs generating golden relations based on the relevant entity pairs is formalized as :

The reasoning module takes the test example e and a set of retrieved demonstrations D to generate relation r. Similar with recalling module, we design a reasoning instruction prompt to guide LLMs to conduct in-context reasoning based on the retrieved demonstrations D. The D are formulated as a series of structural sentences in standard ICL paradigm. The optimization of Lreason is formulated as:

where pθ(r|e,Z ∗ , C) denotes probability of reasoning the golden relation r based on k retrieved demonstrations Z ∗ , and yi denotes the i-th token of relation r. To reduce the impact of error propagation during recalling process and improve the robustness of the model for in-context reasoning during inference time, we add some noise into Z ∗ by replacing k∗ , (1 ≤ k∗ ≤ k) of k demonstrations in Z ∗ with training examples share different relations from retrieval corpora with a uniform distribution. In this way, the LLMs can learn to in-context reasoning [Chen et al., 2022b; Min et al., 2022b; Coda-Forno et al., 2023] based on important entities and relations, avoiding simply deduce the relations via majority vote.

2.7 Joint Optimization and Inference

The final objective function of RE4 is the combination of the recalling optimization and reasoning optimization, which can be formulated as:

We adopt the same LLM for both recalling and reasoning, which are jointly trained on two instruction tuning tasks. To enhance the efficiency of the fine-tuning process and reduce memory requirements, we utilize Low-Rank Adaptation (LoRA) [Hu et al., 2022], which freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer [Vaswani et al., 2017] architecture, greatly reducing the number of trainable parameters for downstream tasks. During inference, based on the recalled entity pairs and retrieved demonstrations, the LLMs conduct ICL to generate predicted relations.

3 Experiments

3.1 Settings

Datasets and Metrics. We evaluate RE4 on SemEval 2010 [Hendrickx et al., 2019], TACRED [Zhang et al., 2017], Google RE 1 , SciERC [Luan et al., 2018], four commonly used RE datasets 2 . Following previous works [Yamada et al., 2020; Cabot and Navigli, 2021; Li et al., 2023a], we use the Micro-F1 score excluding NA as the metric for evaluation. The statistics of datasets are shown in Table 1.

Baselines. We compare RE4 with state-of-the-art RE models that represent a diverse array of approaches. Supervised fine-tuning (SFT) RE methods can be divided into three categories. Classification-based methods fine-tune language models on RE datasets with classification losses, such as MTB [Baldini Soares et al., 2019], LUKE [Yamada et al., 2020], IRE [Zhou and Chen, 2022] and KLG [Li et al., 2023a]. Prompt-based methods use prompt and treats RE as a cloze-style task, such as KnowPrompt [Chen et al., 2022a] and NLI-DeBERTa [Sainz et al., 2021]. Generativebased methods use text generation models for RE, such as REBEL [Cabot and Navigli, 2021], TANL [Paolini et al., 2021], RELA [Li et al., 2023b] and DeepStruct [Wang et al., 2022]. For ICL-based RE, existing methods typically rely on the strong ICL ability of large-scale proprietary models without any SFT. We utilize GPT-RE [Wan et al., 2023] which use PURE [Zhong and Chen, 2021] as the demonstration retriever and GPT-3 [Brown et al., 2020] as the base LLM for ICL. RE4 can be regarded as the combination of SFT and ICL paradigms. On the one hand, it makes LLMs better adapt to specific tasks via instruction tuning. On the other hand, it improves the in-context reasoning ability of LLMs and enables relevant retrieved demonstrations during the ICL process.

Experiment Details. We experiment RE4 with open-source LLMs including T5 [Raffel et al., 2020], BART [Lewis et al., 2020] and LLaMA [Touvron et al., 2023]. For model scales, we select T5-Base (220M), T5-Large (770M), BARTBase (140M), BART-Large (400M) and LLaMA-7B for experiments. We utilize LoRA [Hu et al., 2022] to tune LLMs for simplicity and efficiency. We set the rank r of the LoRA parameters to 8 and the merging ratio α to 32. We train RE4 for 5 epochs with batch size 4 and learning rate 1e-4. For the number of generated entity pairs, we set k to 5. The checkpoint of LoRA adapter that achieves the best result on the validation set is used for testing. We also directly treat relation names as generation objectives and fine-tune the LLaMA.

1 https://github.com/google-research-datasets/relation-

extraction-corpus

2 https://github.com/1iguozheng/RE4

3.2 Main Results

The main results of baselines and RE4 are summarized in Table 2, where RE4 with LLaMA outperforms all previous state-of-the-art methods on four RE datasets. Compared to the classification-based methods, RE4 shows favorable results without any external dataset usage or additional pre-training stages, while MTB and LUKE all involve entity and relation related pre-training tasks. RE4 with most LLMs also suppresses two prompt-based methods. Compared to previous generative-based methods, RE4 consistently achieves competitive or superior results on four datasets. For example, with same backbone BART-Large, RE4 and REBEL share similar overall results, while REBEL is pre-trained with a large external relational triple extraction dataset. Notably, RE4 with LLaMA significantly outperforms REBEL on TACRED and SciERC, and vanilla fine-tuned LLaMA still surpasses REBEL, due to the fact that LLaMA (7B) has a much larger number of model parameters than REBEL (400M). Although LLaMA is smaller than DeepStruct which is based on a pre-trained 10B parameter language model GLM [Du et al., 2022] and is pre-trained on a collection of large-scale corpus, RE4 delivers better results on TACRED than DeepStruct. This is notable because RE4 adopt smaller foundation models for fine-tuning and utilize no external datasets for pre-training compared to DeepStruct. In addition, RE4 with LLaMA achieves comparable results on SemEval and much better results on TACRED compared to ICL-based method GPT-RE. Without fine-tuning, if GPT-3 lacks relevant domain knowledge about specific tasks, then the performance of GPTRE is limited and greatly affected by the retrieved demonstrations. And this is why GPT-RE delivers exceptional results on SemEval but unattractive results that even worse than LUKE on TACRED. In other words, solely relying on ICLbased methods is unstable. The main results demonstrate the simplicity and effectiveness of RE4 compared to baselines.

3.3 Discussions

We conduct extensive experiments to verify the effectiveness of recalling and reasoning module. We also provide detailed analysis about retrieved demonstrations and in-context reasoning abilities. In this section, we only experiment RE4 with LLaMA on SemEval and TACRED for simplicity.

Number of Generated Entity Pairs. We vary the number of generated entity pairs k from 3, 5, 7 to 9. We also consider the majority vote results, the overall results are shown in Figure 3. On the one hand, we find that without in-context reasoning, the performance are more sensitive to the k-selection. With the help of ICL, RE4 could achieve more stable and much better performances, which highlights the importance of reasoning module. On the other hand, obviously, setting k to 5 delivers the best results across two datasets. While increasing k generally boosts the performance of majority vote, the overall performance of RE4 tends to stabilize, even with a slight decrease. This is because in-context reasoning is sensitive to the distractors (i.e., additional retrieved demonstrations that are not relevant to a test example) [Shi et al., 2023]. Therefore, setting larger k not only increases difficulty in generating accurately consistently ontological entities, but brings more distractors that are negative for reasoning module.

Quality of Generated Entity Pairs. We consider two aspects of generated entity pairs for quality checking. We first examine the validness of generated entity pairs (i.e., whether it is grounded by retrieval corpora), then we evaluate the relevance of retrieved demonstrations with respect to test examples. We match the generated entity pairs with the entities in training examples, the results are shown in Table 3. It can be seen that under strict exact matching, the proportion of valid generated entity pairs still reaches around 95%, which indicates that the entity pairs generated by recalling module could serve as the faithful queries for retrieval corpora. However, we are still interested in those invalid entity pairs. Take the SemEval dataset as the example, we analyze the remaining 4.14% invalid entity pairs, discovering that 70 of 562 entity pairs at least one head or tail entity is grounded by retrieval corpora. The other 492 generated entity pairs all have valid entities, but cannot express a valid relation. Overall, during entity pairs generation, the recalling module reduces the hallucination issue of the LLMs and generates faithful queries.

To evaluate the relevance of retrieved demonstrations with respect to test examples, we consider the number of examples which share same relation with a test example (i.e. the consistency of ontology). The results are summarized in Table 4. For most cases, the entity pairs generated by the recalling module and the demonstrations retrieved by retrieving module are all relevant (i.e., all 5 demonstrations share same relation with a test example). Note that 84.97% and 69.83% of retrieved results ensure the majority of golden relation examples participation. And these demonstrations can provide relevant contextual knowledge for ICL. However, one drawback of recalling is that there may be no golden relation in the generated results. Existing similarity-based retrievers such as Sentence-BERT [Reimers and Gurevych, 2019], SimCSE [Gao et al., 2021] and PURE [Zhong and Chen, 2021] essentially cannot guarantee that the retrieved demonstrations would always contain the golden relation. The relevance of retrieved demonstrations actually rely on the performance of the retrieval model. Therefore, we evaluate the quality of retrieved demonstrations from recalling module and similarity calculation via ICL on GPT-3 3 , as shown in Figure 4. We find that fine-tuned retrievers (PURE and recalling module)

3 For GPT-3, we use "text-davinci-003". For SentenceBERT, we use "all-mpnet-base-v2". For SimCSE, we use "sup-simcse-bert-base-uncased". For PURE, we use

"bert-base-uncased" as the backbone.

substantially achieve much better ICL results than similaritybased models without considering entity and relation semantics. For different number of demonstrations, recalling module delivers comparable results compared to PURE, showing the effectiveness of recalling module and allowing seamless integration with any arbitrary LLMs during inference.

Effectiveness of ICL Tuning. We remove the recalling instruction task and keep the reasoning instruction task. Not relying on the retrieved demonstrations Z ∗ , we use 5 training examples as demonstrations, sampled uniformly at random, during both training and testing time. Note that we relax the assumption of perfect balance between labels on training examples. We categorize two types of test examples and then compare the performance of RE4 with vanilla fine-tuned LLaMA, the results are shown in Table 5. First, without any tuning process, LLaMA cannot perform ICL in RE. We empirically discover that LLaMA is unable to understand the structural sentences in standard ICL paradigm and recover the relation labels of test examples based on demonstrations. Second, although the distractors can lead LLMs to make inaccurate predictions, the improvement brought by ICL tuning is still obvious. Experimental results suggest that the impact of ICL training is positive when the golden relation is included in retrieved demonstrations compared to vanilla fine-tuning, which also highlights the importance and performance of retrieval models. During instruction tuning, the model learns to learn in-context for deducing the golden relation.

Sensitivity of In-Context Reasoning. We have validated that relevant contextual knowledge for test examples is beneficial for overall performance. But we might be interested in whether all retrieved demonstrations share similar ontology are equal during ICL. Specifically, for each test example, we replace the corresponding k generated entity pairs with other k random entity pairs but share same relations. The in-context reasoning results are shown in Figure 5. Similar with other LLMs such as GPT-3, the change of demonstrations also have an impact on the final results. Compared to demonstrations obtained via generated entity pairs, the replacement operation consistently brings slightly performance drop across different k and datasets, which indicates that the recalling module actually help to discover similar demonstrations with test examples. However, the performance degradation caused by this randomness is relatively small compared to performance in GPT-3 [Zhao et al., 2021]. The ICL tuning process forces the LLMs to deduce the golden relation based on important entities and relations, which indicates that RE4

is able to perform modestly robust reasoning utilizing relevant but imperfect contextual knowledge.

4 RelatedWork

The large language models (LLMs), such as GPT-3 [Brown et al., 2020] and ChatGPT [OpenAI, 2022], perform well in various downstream tasks without any training or finetuning but only with a few examples as instructions, which is called in-context learning (ICL). However, ICL with LLMs achieves poor performance in relation extraction (RE) where the main obstacles are two-fold: (1) the low relevance regarding entity and relation in the retrieved demonstrations for ICL, and (2) the failure of utilizing LLMs with moderate size for ICL. For the first challenge, existing attempts rely on sentence embedding in retrieval, including the sentence encoders such as Sentence-BERT [Reimers and Gurevych, 2019] and SimCSE [Gao et al., 2021]. Considering entity and relation semantics, GPT-RE [Wan et al., 2023] finetunes PURE [Zhong and Chen, 2021] to provide more REspecific and robust representations for retrieval. For the second challenge, although ICL has been further improved by later work [Zhao et al., 2021; Holtzman et al., 2021; Min et al., 2022a] and shows promising results on a variety of tasks, these researches mainly focus on GPT-3. To improve the ICL ability of other LLMs, current methods propose to make LLMs perform better ICL via meta learning [Chen et al., 2022b] and multi-task learning [Min et al., 2022b]. Inspired by these, our method can be viewed as the integration framework of retrieving and reasoning by instruction tuning. Guided by ontological knowledge, RE4 generates some possible similar entities and relations, and then performs reliable in-context reasoning based on the retrieved demonstrations.

5 Conclusion

In this work, we propose RE4 , a novel recall-retrieve-reason RE framework for open-source LLMs, which recalls relevant entity pairs, retrieves good demonstrations and then perform better in-context reasoning for RE. We consider distilling consistently ontological knowledge to guide the demonstration retrieval, and tuning LLMs with ICL objective to perform reliable in-context reasoning. Specially, RE4 allows seamless integration with any arbitrary LLMs during inference. Empirical results show that RE4 achieves new state-of-the-art sentence-level RE performance in four RE benchmarks. We also demonstrate its effectiveness with extensive experiments, and discuss its advantages and limitations, encouraging more effective generative-based RE methods in the future research.

Acknowledgments

We thank the reviewers for their insightful comments. This work was supported by National Science Foundation of China (Grant Nos.62376057) and the Start-up Research Fund of Southeast University (RF1028623234). All opinions are of the authors and do not reflect the view of sponsors.