Empirical Analysis of Dialogue Relation Extraction with Large Language Models

1 Introduction

Relation extraction (RE) aims to identify relationships between arguments in text, including sentences [Zhang et al., 2017], documents [Yao et al., 2019], or dialogues [Yu et al., 2020]. However, as many relational facts span multiple sentences, sentence-level RE faces limitations in practice [Yao et al., 2019]. Therefore, cross-sentence RE, which aims to identify relations between two entities not in the same sentence, is vital for building knowledge bases from extensive corpora.

• S1: Hey, you guys!

S2: Hey!

• S1: Guess what?

S3: What?

• S1: I got a gig!

S4: Yay!!

S5: See, that's why I could never be an actor. Because I can't say gig.

• S2: Yeah, I can't say croissant. Oh my God!
• S6: What's the part?
• S1: Well, it's not a part, no. I'm teaching acting for soap operas down at the Learning Extension.
• S3: Come on! That's great.

S4: Wow!

• S1: Yeah, yeah. It's like my chance to give something back to the acting community.

Figure 1: An example dialogue with its desired relations. S1-S6: anonymized speaker of each utterance.

As dialogues often exhibit cross-sentence relations, extracting relations from dialogues is necessary [Yu et al., 2020].

To extract relations between two arguments that appear within a dialogue, dialogue relation extraction (DRE) models require reasoning over multiple sentences in a dialogue. To this end, a plethora of models have been proposed to address the challenges of DRE in recent studies including sequencebased methods [Yu et al., 2020; Son et al., 2022] and graphbased methods [Xue et al., 2021; Lee and Choi, 2021; Liu et al., 2023]. Sequenced-based methods utilize encoderonly pre-trained language models such as BERT [Devlin et al., 2019] to encode utterances in the dialogue. Graph-based methods usually construct various heterogeneous graphs over dialogues to capture the relational information. However, neither paradigm of methods can deliver satisfactory DRE results, because these encoder-based methods are sensitive to the higher person pronoun frequency and lower information density in dialogues [Lee and Choi, 2021]. As shown in Figure 1, the sentence highlighted with red is a prerequisite for correct understanding of relationships in the following sentences highlighted with green, but this sparse multi-turn information is difficult for models to capture. Moreover, current methods suffer grave performance declines when estimating howmany turns it require to predict the relations between two arguments in partial dialogue instead of entire dialogue [Yu et al., 2020]. For example, DRE models are expected to deduce the relation per:title between S1 and actor with only first seven utterances in Figure 1 while current methods typically need to rely on entire dialogue. This motivates us to search for more effective methods that can alleviate the above issues.

Besides the above sequence-based and graph-based methods, generation-based methods are also occasionally used in RE, but only focus on sentence-level [Cabot and Navigli, 2021; Paolini et al., 2021] and document-level [Giorgi et al., 2022] RE. As the emergence of large language models (LLMs) such as GPT-3 [Brown et al., 2020] significantly driving the progress of natural language processing (NLP), utilizing generation-based methods start to draw more attention [Wadhwa et al., 2023; Zhang et al., 2023; Ma et al., 2023; Li et al., 2023; Wan et al., 2023] in RE community. To the best of our knowledge, however, there is currently no work specifically applying generation-based methods to DRE. To this end, we aim to investigate the capabilities of generationbased methods for DRE in this work. Specifically, we consider the evaluation of both powerful proprietary LLMs such as ChatGPT [OpenAI, 2022] and open-source LLMs such as LLaMA [Touvron et al., 2023]. For proprietary LLMs, we design various prompt formats to extract relations from dialogues via few-shot prompting [Brown et al., 2020] and zero-shot prompting [Kojima et al., 2022]. For open-source models evaluation, we introduce Landre (Language models driven dialogue relation extraction), a DRE framework driven by LLMs based on open-source foundation models, where it employs a simple prompt tuning method and a parameter efficient tuning technique LoRA [Hu et al., 2022] for better RE task adaptation. We compare the performances of generation-based methods with previous sequence-based and graph-based methods, and conduct extensive experiments to provide valuable insights and guide further exploration.

Importantly, we empirically discover that generation-based methods significantly alleviate the issues of DRE mentioned above. On the one hand, for the difficulty of capturing long and sparse multi-turn information, we find that scaling up model size substantially boosts the overall DRE performance and achieves exceptional results. While the ChatGPT demonstrates promising performance with elaborated prompts, fine-tuned open-source models surpass prior stateof-the-art (SoTA) by a large margin, opening new frontiers for advancing the research of DRE. On the other hand, for the requirement of partial dialogue understanding, we surprisingly find that all generation-based methods encounter with much smaller performance drop from entire dialogue to partial dialogue setting, comparing to sequence-based and graph-based methods. In summary, our contributions are as follows:

• We present the initial evaluation of LLMs regarding to DRE. We design elaborated prompts for ChatGPT and introduce an LLM-driven DRE framework Landre for open-source foundation models. Evaluation on DRE benchmarks across various experimental settings reveals
• our significant performance improvements over previous sequence-based and graph-based methods.
• We propose to utilize generation-based methods in DRE, which significantly alleviate two main issues of DRE. Generation-based methods are able to capture important relation semantics from sparse multi-turn information and achieve relatively consistent performances under both partial and entire dialogue settings.
• By analyzing the performance and complexity of different generation-based methods, we emphasize their promising performance and obvious shortcomings compared to previous methods, providing valuable insights to advance future DRE research.

2 Dialogue Relation Extraction (DRE)

Given a dialogue D = s1 : t1, s2 : t2, ..., sm : tm and an argument pair (a1, a2), where si and ti denote the speaker ID and text of the ith turn, respectively, and m is the total number of turns, we evaluate the performance of approaches in extracting relations between a1 and a2 that appear in D in the following two settings. (1) Standard Setting: Similar to standard RE tasks, the dialogue D is regarded as a document d, and we evaluate the extracted relations between a1 and a2 based on d. Specifically, we adopt F1, which is the harmonic mean of precision (P) and recall (R), for evaluation. (2) Conversational Setting: Instead of only considering the entire dialogue, we can regard the first i ≤ m turns of the dialogue as d. Given d containing the first i turns in a dialogue, relation type r associated with a1 and a2 is evaluable if a1, a2, and the trigger for r have all been mentioned in d. The definition is based on the assumption that we can roughly estimate how many turns we require to predict the relations between two arguments based on the positions of the arguments and triggers. Accordingly, we adopt a new metric F1c, the harmonic mean of conversational precision (Pc) and recall (Rc), as a supplement to the standard F1 metric [Yu et al., 2020].

3 Large Language Models for DRE

We leverage various LLMs to address DRE task. We consider both proprietary LLMs such as ChatGPT [OpenAI, 2022] and open-source LLMs such as LLaMA [Touvron et al., 2023].

3.1 Leveraging ChatGPT for DRE

We access ChatGPT using official API service 1 by prompting. As prompting is a brittle process wherein small modifications to the prompt can cause large variations in the model predictions, we explore various prompt templates to investigate the potential of ChatGPT for DRE in two distinct extraction manner: direct extraction and indirect extraction.

Direct Extraction Direct extraction makes ChatGPT directly output the relation labels between an given argument pair based on the input dialogue and candidate relations. This vanilla prompting method is simple, direct, and efficient.

1 https://platform.openai.com/

Indirect Extraction Indirect extraction decomposes the DRE task into two steps of asking then answering. Specifically, we sequentially ask ChatGPT if an argument pair expresses a certain relation among candidate relations based on the dialogue. We ground our analysis in three standard categories of prompts used in prior work [Brown et al., 2020; Sanh et al., 2022; Liu et al., 2024; Arora et al., 2023; Li et al., 2024; Shang et al., 2024] including restrictive prompts, yes-no prompts and open-ended prompts. Restrictive prompts (e.g., "S1 is an actor. Output True or False?") construct questions that restrict the model output particular tokens, while yes-no prompts (e.g., "Is S1 an actor?") and open-ended prompts (e.g., "What is the title of S1?") construct free-form questions, as shown in Figure 2. Typically, the indirect prompting method suffers from relatively high inference complexity as it needs to enumerate all possible triples to obtain questions and answers. Suppose we have n samples to be extracted and r candidate relations, the total complexity is O(r × n), which is time-consuming in practice. The purpose of experimenting with indirect extraction is to explore the DRE potential of LLMs as one step extraction regularly leads to suboptimal results [Li et al., 2023].

3.2 Tuning Open-Source Models for DRE

Different with the aforementioned usage of ChatGPT, we fine-tune the open-source LLMs for better task adaptation. Specifically, we introduce Landre, a DRE framework driven by LLMs based on smaller, open-source foundation models such as LLaMA [Touvron et al., 2023], where we investigate the performance of fine-tuning generative LLMs for DRE. We first outline the process of constructing input and output prompt formats. Then we utilize a parameter-efficient finetuning (PEFT) technique to train the foundation model with limited computational resources.

3.3 Prompt Tuning

As shown in Figure 3, for each sample (D, a1, a2) in the training set, we construct the corresponding input prompt and let the LLM directly generate the corresponding relation label(s) indicating the relationship(s) between arguments a1 and a2 expressing by the dialogue D. Specifically, we concatenate the given dialogue and argument pair separated by the recognizable delimiter "|" as the input prompt, and we also separate the relation labels between the given argument pair by the "|" symbol as the output prediction. The detailed prompt tuning settings are introduced as follows.

The input prompt is composed of three parts: (1) the dialogue context D = s1 : t1, s2 : t2, ..., sm : tm, (2) the argument a1 and (3) the argument a2. And the output is a relation set R = {r1, ..., rn} where n denotes the number of relations between argument pair (a1, a2):

Note that in most cases, n = 1. For convenience, we concatenate all the speaker IDs and texts of multi-turn with the white space token [], to indicate the start of each utterance. We empirically find that although this prompting strategy is frustratingly simple but shows its effectiveness.

3.4 Parameter Efficient Tuning

In this part, we describe how to fine-tune the foundation model using a parameter efficient approach. Landre takes the dialogue context and argument pair from the dataset as inputs and retrieves the corresponding relation label(s) as output:

where Decoder indicates that the foundation model (e.g., LLaMA [Touvron et al., 2023]) uses the Transformer-decoder framework. To enhance the efficiency of the fine-tuning process and reduce memory requirements, we utilize Low-Rank Adaptation (LoRA) [Hu et al., 2022], which freezes the pretrained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for downstream tasks. Finally, the learning objective of the generation process is to minimize the negative loglikelihood of R given the dialogue context D and argument pair (a1, a2):

where N and |Di| denote the total number of dialogues in training set and argument pairs in dialogue Di.

4 Experiments

4.1 Experimental Settings

Datasets andMetrics We conduct experiments on the two versions (V1 and V2 2 ) of DialogRE dataset [Yu et al., 2020], the first human-annotated DRE dataset, originating from the complete transcripts of the series Friends. V1 is the original English version and V2 is the updated English version with a few annotation errors fixed. DialogRE has 36 relation types, 1,788 dialogues, and 8,119 argument pairs. The statistics of DialogRE are shown in Table 1. We calculate both the F1 and F1c [Yu et al., 2020] scores as the evaluation metrics. F1c is an evaluation metric to supplement the standard F1, computing by taking in the part of dialogue as input, instead of only considering the entire dialogue. We experiment Landre in both full-shot and few-shot settings 3 . In the few-shot setting (V2), 8, 16, and 32-shot experiments are conducted by using three different randomly sampled data [Son et al., 2022]. For the experiments on ChatGPT, to keep OpenAI API costs under control, we randomly select 100 dialogues from DialogRE valid and test set, respectively. For direct extraction, the evaluation is consistent with typical evaluation as the extracted relation labels are generated by ChatGPT. The indirect extraction could generate extra tokens that irrelavent with our targets and requires answer cleansing. Specifically, after the ChatGPT generation, we pick up only the part of the answer text that first satisfies answer format [Kojima et al., 2022].

2 https://dataset.org/dialogre/

3 https://github.com/1iguozheng/Landre

Baseline Models For a comprehensive performance evaluation, we compare the ChatGPT and Landre with both sequence-based and graph-based methods on the DialogRE dataset. Sequence-based methods usually utilize pre-trained language models to encode utterances in the dialogue, including BERT [Devlin et al., 2019], BERTs [Yu et al., 2020], RoBERTas [Lee and Choi, 2021] and GRASP [Son et al., 2022]. BERTs is speaker-aware BERT. GRASP captures relational semantic clues of a given dialogue with the prompt marker strategy and the relational clue detection task. Graphbased methods include GDPNet [Xue et al., 2021], TUCORE [Lee and Choi, 2021] and HiDialog [Liu et al., 2023]. GDPNet develops a multiview latent graph to better capture the key features. TUCORE constructs a turn-level graph to capture the relational information. HiDialog leverages a heterogeneous graph module to polish the learned embeddings.

Experiment Details For the ChatGPT, we consider both few-shot prompting [Brown et al., 2020] and zero-shot prompting [Kojima et al., 2022] for evaluation. For the zeroshot prompting, we utilize proposed four types of prompt formats. For the few-shot prompting, we only experiment with direct extraction (i.e. vanilla prompting), considering 1-shot, 3-shot and 5-shot settings because of the maximum length limitation of ChatGPT. And the few-shot demonstrations are randomly sampled from a uniform distribution over all training dialogue examples. Due to ICL is known to have high variance [Zhao et al., 2021], we compute the average performance of three times accessed results. We use the GPT-3.5 series model "gpt-3.5-turbo" for experiments. For open-source LLMs, we experiment the Landre framework with generative models including GPT-2 [Radford et al., 2019], BART [Lewis et al., 2020], T5 [Raffel et al., 2020], BLOOM [Scao et al., 2022] and LLaMA [Touvron et al., 2023]. Specifically, we use the GPT-2-117M, BART-large-400M, BLOOM-560M, T5-large-770M and LLaMA-7B as the foundation models. We utilize LoRA [Hu et al., 2022] to tune all these models for simplicity and efficiency. We set the rank r of the LoRA parameters to 8 and the merging ratio α to 32. The model is optimized with AdamW [Loshchilov and Hutter, 2019] using learning rate 1e-4 with a linear warm up [Goyal et al., 2017] for the first 6% steps followed by a linear decay to 0. We train Landre for 5 epochs with batch size 4, and all experiments are conducted on a single Geforce GTX 3090 GPU.

4.2 Main Results of ChatGPT

The main results on DialogRE using ChatGPT as relation extractors are shown in Table 2. We have the following findings:

Indirect extraction tends to show much better performance than direct extraction for ChatGPT. Obviously, the vanilla prompting, which makes ChatGPT directly output the relation labels between an given argument pair, achieves the worst F1 scores in both few and zero-shot prompting manners compared to other three zero-shot prompting formats. Decomposing extraction process into multiple steps can improve the zero-shot performance by more than 15% in absolute F1 scores. Different with existing findings on prompting formats [Arora et al., 2023], open-ended prompting tends to perform worse than restrictive and yes-no prompting in DRE.

In addtion, for few-shot prompting results, the improvements regarding to zero-shot prompting are small, where 5-shot results seem to have limited advantages compared to 3-shot.

ChatGPT surpasses several strong sequence-based and graph-basedmethods. Evenwith the zero-shot prompting, ChatGPT with elaborated prompting formats delivers better results than fully supervised RoBERTas (+2.9%∼7.5%) and TUCORE-GCN (+1.1%∼5.6%). Although ChatGPT is slightly worse than previous SoTA HiDialog (-0.7%∼3.5%) in standard setting, we should note that our limited budget restricted our study to a small set of prompt styles. It is possible that having a larger prompt design search space could narrow the performance gap between ChatGPT and HiDialog.

Performance gap between standard and conversational settings for ChatGPT is less than previousmethods. Previous methods suffer grave declines when encountering conversational setting compared to standard setting (e.g., HiDialog drops around 7.1%∼8.9%). Interestingly, ChatGPT significantly narrows this performance gap (less than 5%), indicating ChatGPT can effectively identify the relations between an argument pair while tolerating the variant length of input dialogues. In contrast, encoder-only methods are sensitive to the length of input dialogues and hard to recover good representations for classification, because conversational setting is actually make the data distribution change in testing stage.

ChatGPT still has limitations. In summary, ChatGPT exhibits comparable performance in DRE compared to the previous SoTA methods, which highlights the ability of current LLMs to capture and comprehend complex linguistic patterns and dependencies within multi-turn dialogues. However, ChatGPT still faces two notable limitations. First, achieving good DRE performance requires elaborated prompting methods, resulting in higher computation cost and time consumption while ChatGPT is subject to request limitations. Besides, responses from ChatGPT frequently contain substantial explanatory content, where it may not align perfectly with expected answer format and difficult to cleanse and evaluate. Second, ChatGPT is not open-source, restricting its modification and customization, raising concerns about privacy protection and feasibility of local deployment. Therefore, we propose to use open-source LLMs in DRE, which alleviates two issues in DRE but also avoids the limitations of ChatGPT.

4.3 Main Results of Open-Source Models

Full-shot Setting We show the performance of Landre with different foundation models on the DialogRE dataset in Table 2 compared with other baselines. It is shown that Landre achieves competitive or superior performance compared to all of the previous methods, including the current SoTA methods GRASP and HiDialog. On the one hand, scaling up the foundation models in Landre brings significant performance improvements. From 117M to 7B, generation-based methods demonstrate over 20% absolute F1 gains in both standard and conversational settings. Note that the significant improvements are also observed in encoder-based methods (from 110M to 355M), but as the current encoder scale is much smaller than the decoder, the generation-based methods become more promising in future DRE. On the other hand, Landre shows its efficiency in conversational settings by narrowing the performance gap between F1 and F1c (e.g., Landre with LLaMA only drops around 0.9%∼1.2% compared to 7.1%∼8.9% in HiDialog), thereby indicating that generationbased methods effectively overcome the low information density of dialogues. Similar with ChatGPT, generation-based methods tend to perform robust encountering various length of input dialogues, and they are able to extract the golden relation based on partial dialogues with enough arguments and relation information. These results imply that guiding the model to pay attention to relation labels with a prompt tuning approach can be more effective than adding additional features and layers in encoder-based methods.

Few-shot Setting As presented in Table 3, Landre exhibits robust performance in few-shot settings. For example, Landre with GPT-2 surpasses TUCORE regardless of the number of shots. Except for the 8-shot setting, Landre presents new SoTA performance compared to previous methods. However, GRASP with smaller encoder still delivers better 8-shot performance compared to others. Moreover, although Landre with LLaMA is much better than other baselines in the 32shot setting, the limited performance of LLaMA in the 8-shot and 16-shot setting can be attributed to an insufficient number of examples for tuning large-scale parameters.

5 Analysis

In this section, we analyze various aspects of Landre to provide a deeper understanding of generation-based methods performing in DRE task. Specifically, we analyze the results of GPT-2 and LLaMA on the DialogRE (V2) dev and test set.

5.1 Error Analysis

We calculate the error rate of each relation to investigate the fine-grained performance of Landre with GPT-2 and LLaMA. To mitigate the deviation caused by imbalanced relation types, we analyze the 10 most frequently appearing relations whose corresponding sample numbers are above 100. The overall results are summarized in Table 4, where scaling up model size consistently reduces the error rates of all relations. For most interpersonal relations that frequently appear such as per:girl/boyfriend, per:spouse and per:children, LLaMA delivers prominent performance improvements compared to GPT-2, showing increasing the model size can better capture intra-turn and inter-turn information, distinguishing the relationship semantic differences in the dialogues.

For specific relations such as per:alternate names and per:title, GPT-2 and LLaMA achieve similar precision with low error rates. First, it is not surprising that the relation per:alternate names is easy for models because it is the most frequently appearing relation and typically can be identified within a single-turn information (e.g., S1 calls S2's name then S2 responds). For relation per:title, interestingly, two models have the same number of wrong predictions. We compare the error cases of two models and find 5 out of 6 samples overlapping. The 5 samples are mistakenly annotated as per:title between PER and NAME in the first place while the subject and object types of per:title should be PER and STRING [Yu et al., 2020]. Because most of argument pair types belong to PER and NAME, per:title is relatively easy to distinguish. In other words, generative methods can achieve near perfect performance on relation per:title because of the information from argument mentions, most of which is type information.

Another interesting finding is that GPT-2 delivers 100% error rate on relation per:negative impression. We thus examine the proportion of its predicted results with respect to 102 samples annotated with per:negative impression. We discover that 66.6% of samples are predicted with relations per:girl/boyfriend, per:positive impression and per:friend for GPT-2, where these relations may share same argument pairs with per:negative impression and tend to confuse the models because friends and girl/boyfriends often express impressions of each other. In addition, some specific relations such as per:acquaintance (not shown in Table 4) are challenging to Landre, where the error rates of per:acquaintance are 100% and 97.8% for GPT-2 and LLaMA, respectively. The main reasons include difficulty in defining such relation (i.e., it is difficult to distinguish between acquaintances and friends) and low trigger ratio (around 22%) [Yu et al., 2020].

5.2 Qualitative Analysis

Inverse relations. We group the test set of DialogRE according to the relation types into three subsets: (I) asymmetric, when a relation type differs from its inversion (e.g. per:children and per:parents); (II) symmetric, when a relation type is the same as its inversion (e.g. per:friends); (III) other, when a relation type does not have inversion (e.g. per:title). We compare the performance of Landre with baselines and report the results in Table 5. We observe that GPT-2 struggles to capture inverse relations compared to BERT. Moreover, Landre with LLaMA shows comparable performance on both asymmetrical and symmetrical relations compared to TUCORE and HiDialog, while it delivers exceptional results on other relation types. And this is also why GPT-2 and LLaMA have low error rates on specific relations such as per:alternate names which is the most frequently appearing relation and contributes the majority of overall performance.

Utterance lengths. Compared to encoder-based methods, generation-based methods enable to handle dialogues of various lengths. We further divide the samples in the test set into six groups according to their lengths and report the F1 score for each group achieved by Landre and previous SoTA. As shown in Figure 4, LLaMA outperforms HiDialog in all groups except the group with less than 100 tokens, maintaining decent performance for long sequences.

5.3 Applicability Analysis

To evaluate the robustness and generalization of generationbased methods, we evaluate Landre with LLaMA on MELD [Poria et al., 2019] and EmoryNLP [Zahiri and Choi, 2018] datasets, which are designed for emotion recognition in conversations (ERC). Following previous works [Lee and Choi, 2021; Son et al., 2022], the ERC task is converted into DRE and the weighted-F1 is calculated to evaluate the MELD and EmoryNLP datasets. As presented in Table 6, the performance of Landre surpasses that of GRASP and HiDialog in both MELD and EmoryNLP, demonstrating significant promise in dialogue understanding.

6 RelatedWork

Large Language Models. Besides the "pre-train and finetune" paradigm [Devlin et al., 2019], pre-trained LLMs possess characteristics that are advantageous for fewshot [Brown et al., 2020] and zero-shot [Kojima et al., 2022] learning, whereby appropriate prompts are used to effectively guide the model towards generating desired task outputs, thus beginning an era of "pre-train and prompt". However, prompting without updating parameters generally brings suboptimal results and the efficient utilization of LLMs remains a significant challenge. Recent advancements in parameterefficient fine-tuning (PEFT) techniques have effectively alleviated this problem, such as LoRA [Hu et al., 2022] and Prefix Tuning [Liu et al., 2022]. In this study, we initially investigate the capabilities of fine-tuned open source (e.g. LLaMA [Touvron et al., 2023]) and non fine-tuned closed source (e.g. ChatGPT [OpenAI, 2022]) LLMs for DRE.

Dialogue Relation Extraction. Most previous works on RE focused on sentence-level RE [Li et al., 2022; Wang et al., 2023b; Wang et al., 2023a], but recently cross-sentence RE has been studied more because many relational facts are expressed in multiple sentences in practice [Yao et al., 2019]. Dialogue relation extraction (DRE) requires reasoning over multiple sentences in a dialogue to predict the relations between two arguments [Yu et al., 2020]. Current popular DRE methods can be divided into sequence-based methods [Yu et al., 2020; Son et al., 2022] and graph-based methods [Xue et al., 2021; Lee and Choi, 2021; Liu et al., 2023]. The former utilize encoder-only language models such as BERT [Devlin et al., 2019] and RoBERTa [Liu et al., 2019] to encode utterances in the dialogue. The latter construct various heterogeneous graphs over dialogues to capture the relational information. And both two kind of methods use the softmax classifiers for relation classification. In this study, we aim to investigate the capabilities of generative methods for DRE. Due to the rapid development of decoder-based LLMs, we anticipate that Landre could serve as a compelling baseline or plug-in module for future work in the community.

7 Conclusion

In this study, we conduct an initial examination of generative LLMs in DRE, showcasing its excellence over previous sequence-based and graph-based methods. To solve the limitations of ChatGPT, we present Landre, an LLM-driven DRE framework based on smaller, open-source foundation models. By simply prompting tuning with LoRA, Landre delivers better performance than ChatGPT and achieves new state-of-theart performance on DRE benchmarks. Comprehensive evaluations across different experimental settings showcase the unique characteristics of generation-based methods, opening new frontiers for advancing the research of DRE.

Acknowledgments

We thank the reviewers for their insightful comments. This work was supported by National Science Foundation of China (Grant Nos.62376057) and the Start-up Research Fund of Southeast University (RF1028623234). All opinions are of the authors and do not reflect the view of sponsors.