EDGE: Efficient Data Selection for LLM Agents via Guideline Effectiveness

1 Introduction

In recent years, Large Language Models (LLMs) [Ouyang et al., 2022; OpenAI, 2023] have demonstrated remarkable fewshot learning and reasoning capabilities. An increasing number of studies have begun exploring how to leverage LLMs as agents that can accomplish various tasks through multiple interactions with the environment [Deng et al., 2023; Liu et al., 2024b; Wang et al., 2024]. For example, WebShop [Yao et al., 2022] provides a simulated shopping environment where agents must select products that best match user requirements.

During interactions, LLMs frequently encounter complex or previously unseen scenarios, which places substantial demands on their generalization capabilities. Numerous studies have been dedicated to mitigating this challenge.

∗Corresponding author.

Prior work has demonstrated the importance of guidelines (or insights) in prompt-based multi-turn interaction methods. Guidelines are natural language prompts summarized from data that contain more information and cover more scenarios than exemplars, while typically consuming less context space. Existing approaches autonomously gather experiences from training tasks through trial and error to generate these guidelines [Zhao et al., 2024].

Another line of research focuses on Supervised FineTuning (SFT) of open-source LLMs to enhance their instruction-following capabilities. Prior work has shown that the effectiveness of SFT depends more on dataset quality than quantity [Wang et al., 2023; Zhou et al., 2023]. Current data filtering approaches, including GPT-4-based scoring [Chen et al., 2024], instruction difficulty assessment [Li et al., 2024], and semantic diversity metrics [Lu et al., 2024], have demonstrated varying degrees of success.

Despite these advancements, current LLM-agent approaches still face several pressing challenges. In promptbased methods, existing approaches for obtaining guidelines do not consider data quality control, instead randomly selecting samples from annotated data, which not only requires substantial and costly annotation efforts but also suffers from noisy data problems. Meanwhile, in SFT-based methods, current approaches heavily rely on golden answer feedback and primarily focus on single-turn instruction tuning, lacking necessary exploration of more complex multi-turn interaction scenarios that are essential for real-world applications.

To address these challenges, we propose Efficient Data selection for LLM agents via Guideline Effectiveness, a novel framework centered around a new metric called Guideline Effectiveness (GE) to select the most informative subset of samples from a vast unlabeled data (query) pool. These selected samples can be utilized for both prompt engineering and SFT.

Guidelines represent human understanding of tasks and serve as prior knowledge for agents, encompassing tool usage patterns and comprehension of complex scenarios [Zhao et al., 2024; Fu et al., 2024]. The GE score essentially quantifies the impact of guidelines on each data sample, enabling us to identify which samples are most challenging for the model and thus select more informative ones. Beginning with an initial guideline, we select a small number of samples with the lowest GE scores. These samples are then analyzed to summarize error causes and update the guideline. Next, we use the updated guideline and advanced API-based LLM to annotate more low-GE-score samples instead of relying on human annotators. Notably, the updated guideline incorporates solutions for challenging samples and deeper insights into the task and tools, ensuring that the annotated data is of high quality. Finally, we can use these informative and high-quality annotated samples to fine-tune open-source LLMs.

The main contributions of this work are summarized as follows:

• Propose a novel Guideline Effectiveness metric to identify informative samples using guidelines without requiring golden answers. This metric enables efficient sample selection for both prompt engineering and model fine-tuning.
• Derive effective guidelines and obtain high-quality data for challenging multi-turn interaction tasks without the need for manual annotation, by leveraging the GE score.
• Demonstrate the effectiveness of our approach through extensive experiments on HotpotQA and WebShop benchmarks, achieving state-of-the-art performance with only 75% and 50% data requirements compared to existing methods.

2 RelatedWork

This study investigates how to effectively utilize guidelines in

the context of data selection for supervised fine-tuning (SFT). Data Selection for SFT aims to select a high-quality subset of data. [Zhou et al., 2023] demonstrates that only 1,000 carefully curated prompts and responses can achieve remarkably strong performance. [Chen et al., 2024] proposes using GPT-4 for direct quality scoring, successfully identifying 9k high-quality samples from a dataset of 52k instances. [Li et al., 2024] introduces the Instruction-Following Difficulty (IFD) metric to identify discrepancies between a model's expected responses and its intrinsic generation capabilities. [Liu et al., 2024a] curates 6K training samples by evaluating them along three dimensions: complexity, quality, and diversity. [Bhatt et al., 2024] conducts a comprehensive evaluation of existing data selection methods that aim to maximize uncertainty and/or diversity measures. However, these evaluation metrics inherently depend on golden answers as feedback. Furthermore, they primarily focus on single-turn interactions, neglecting the complexities of multi-turn interaction scenarios. AgentTuning [Zeng et al., 2024] and FiReAct [Chen et al., 2023] investigate fine-tuning LLMs with multi-turn interaction trajectories generated by GPT-4, further examining the effects of multi-task learning and prompt design methods, respectively. However, both methods randomly select samples for annotation, and assume that perfectly correct trajectories (reward = 1) represent high quality. This approach may result in the inclusion of simpler problems in fine-tuning datasets, leading to low quality of fine-tuning data.

Deep Active Learning aims to identify the most informative samples for annotation, thereby reducing labeling costs. The methods are typically categorized into uncertainty-based

[Settles, 2011; Kremer et al., 2014], diversity-based [Sener and Savarese, 2018; Bukharin et al., 2024; Bhatt et al., 2024], or hybrid approaches [Azeemi et al., 2025]. In the era of large language models (LLMs), some studies have attempted to integrate active learning with LLMs to achieve efficient SFT. [Azeemi et al., 2025] investigates active learning for improving label efficiency in natural language generation but reports inconsistent results. [Kung et al., 2023] proposed a task-level active learning framework to explore the most effective SFT tasks. However, it makes the simplifying assumption that all instances are of equal value within a task. [Bhatt et al., 2024] is most similar to ours. It is the first to utilize experimental design for SFT and formulates active learning as a facility location problem. This method focuses on selecting semantically diverse and representative samples, effectively improving the generative capabilities of LLMs. However, it does not focus on addressing agent tasks that require more reasoning and decision-making capabilities.

Guideline-based Prompting aims to summarize historical interaction experiences from datasets into natural language prompts that can guide future interactions. [Zhao et al., 2024] introduces Experiential Learning, which autonomously gathers experiences from training tasks through trial and error to generate instructive guidelines. [Fu et al., 2024] advances this approach by automatically generating context-aware guidelines and implementing a retrieval system that selects guidelines relevant to the agent's current state. However, these approaches rely on random sampling without quality consideration and their automated summarization lacks the depth and nuance of expert knowledge.

3 Methodology

3.1 Overview

Our core insight is to identify informative samples from an unlabeled data pool by leveraging guidelines, as illustrated in Figure 1. Given an unlabeled data pool and initial guidelines, we first compute the Guideline Effectiveness (GE) score for each sample using the initial guidelines. Samples with lower scores are selected for manual annotation, resulting in updated guidelines that can be directly applied to prompt-based methods. To fully utilize the unlabeled data, we incorporate the new guidelines into the prompt text and employ GPT-4 for annotation, generating question-interaction trajectory pairs as high-quality SFT data. Notably, this entire process does not require golden answers.

3.2 Preliminary

The guideline is defined as G = CONCAT(g1 , g2 , . . . , gn ), where each gi is a natural language instruction reflecting human understanding of the task. Given a set of questions Q = {q1, . . . , qn}, a language model LLM, and an initial guideline G init , we generate interaction trajectories T = LLM(Q, G init ), where each trajectory T = (q, a1, o1, . . . , aT , oT ) consists of question-actionobservation sequences with length T. Here ai represents the action taken by the LLM at step i, and oi denotes the observation or feedback received from the environment after taking action ai. Our objective is to select an informative subset Q′ such that:

where Reward is the metrics of the task and LLM′ is finetuned by labeled Q′ . While Eq. (1) is not directly optimized due to the complexity of the reward landscape and nonconvex behavior of LLMs, we approximate it using a heuristic, active learning approach in practice.

3.3 Guideline Effectiveness

Guidelines are natural language prompts enriched with expert knowledge that can cover more scenarios while consuming less context space compared to detailed exemplars. However, since humans may not initially recognize all potential challenging samples, the initial guidelines may be inadequate. Therefore, we propose a metric called Guideline Effectiveness to quantify the contribution of guidelines in solving a given question in order to identify questions that are challenging for the initial guidelines.

Given an question q, the prompt text is constructed as:

where I represents the instruction text, and E denotes a set of interaction examplars. We measure the uncertainty of LLM's output action at at step t using the average cross-entropy loss of each token.

where N is the number of tokens in at, wi is the i-th token in at and θ denotes the LLM parameters. A lower dGθ implies lower uncertainty in generating an action.

To evaluate guideline effectiveness, we construct Prompt-G = CONCAT(I, E) by excluding G init from the context. The difficulty of generating action without guidelines is:

The score dIθ reflects the uncertainty to generate a using only the LLM's intrinsic knowledge.

Based on dGθ and dIθ , the GE score is defined as:

GE quantifies the influence of guideline on generating each at by comparing the uncertainty of generating actions with and without guidelines. The intuitive interpretation of GE values is as follows: A lower value of d indicates the generation process is easier. The difficulty scores dIθ and dGθ represent generation difficulty without and with guidelines respectively. When GE > 0, we have dIθ > dGθ , indicating that guidelines facilitate action generation. A larger positive GE score suggests guidelines have a stronger positive impact on generation. As the score approaches zero, the similar magnitudes of dIθ and dGθ indicate guidelines provide limited benefit, which are samples of particular interest. When GE < 0, meaning dIθ < dGθ , guidelines appear to impede generation. This reveals cases where the LLM's inherent knowledge leads it to generate actions that conflict with the guidelines, another important category of samples to identify.

3.4 Efficiently Incorporating Human Expertise

This section describes how we utilize the GE score to incorporate human expertise into LLMs efficiently, as shown in Figure 1. Given the data pool Q and historical interaction trajectory T , and initial guideline G init , we can calculate the GE score for each qi. Lower GE scores indicate challenging questions which require additional learning by the LLM. We accomplish this in two stages: Guideline Update and HighQuality Data Generation.

Update the Guideline. We select m questions with the lowest GE scores. By observing these questions' interaction trajectories, we summarize the task-specific knowledge that the LLM lacks and update G init to G new . Analyzing samples with the lowest GE scores allows G new to integrate further human expertise necessary for addressing challenging samples, such as deeper insights into the task and tools. The value of m can be small (e.g., 30), which is manageable by humans within a reasonably short time. Human experts can either refine existing guidelines or introduce new ones as needed. We denote the ReAct framework augmented with the updated guideline as EDGEUG.

High-Quality Data Generation. Similarly, we select k questions with the lowest GE scores and employ GPT-4 to generate interaction trajectories guided by G new . The incorporation of human expertise within G new ensures that the trajectories maintain a high standard of quality. Utilizing these high-quality data for fine-tuning, open-sourced LLMs will implicitly learn human expertise from the annotated data. Fine-tuning the open-sourced LLMs is often crucial because: 1) As guidelines become more complex, it gets harder for open-sourced LLMs to follow; 2) Some tasks may be too intricate to distill into guidelines, making annotating data a simpler option.

4 Experiments

4.1 Baselines

To evaluate our approach, we have selected a range of stateof-the-art (SOTA) methods as baselines and conducted model comparisons along two dimensions:

EDGE vs. Other Agent Methods. We compare our method against several state-of-the-art agent methods: ReAct [Yao et al., 2023] integrates reasoning and acting capabilities for sequential decision-making tasks; Reflexion [Shinn et al., 2023] reinforces language agents through linguistic feedback; AMOR [Guan et al., 2024] constructs reasoning logic over finite state machines for automated problem-solving across modules; ExpeL [Zhao et al., 2024] leverages GPT-4 to extract guidelines from failed trajectories.

GE vs. Other Data Selection Strategies. For comparison with other label-efficient data selection strategies, we evaluate GE against several baseline approaches: Random selects data randomly for annotation; Mean Entropy [Settles, 2011; Kremer et al., 2014] measures uncertainty through tokenwise negative entropy of softmax probabilities; FL [Bhatt et al., 2024] selects semantically representative samples based on diversity; High Score [Chen et al., 2023; Zeng et al., 2024] retains only fully correct interaction trajectories from annotated data.

4.2 Experiment Setup

Datasets. HotpotQA [Yang et al., 2018] is a multi-hop question-answering benchmark that challenges an agent to retrieve Wikipedia passages to perform reasoning and questionanswering. This involves utilizing API calls and LLM's knowledge to search for and retrieve information in order to find answers. Following [Yao et al., 2023], we use three types of actions to support interactive information retrieval in HotpotQA: search[entity], lookup[query] and finish[answer]. WebShop [Yao et al., 2022] is a simulated online shopping environment composed of a website with 1.18M real-world products. The agent's goal is to purchase a product that meets specific requirements based on a text instruction. This task requires the agent to query the website's search engine, select products with required features, and click the necessary options. Following [Liu et al., 2024b], the system implements two valid actions: search[query] and click[button].

For HotpotQA, we use the first 10,000 training questions as the data pool and randomly select 500 dev questions. For WebShop, we use 8,500 instructions as the data pool and another 500 instructions for evaluation. For each dataset, we selected 30 samples with the lowest GE score for guideline updating, and then annotated 800 samples for fine-tuning. The statistical details of the test datasets are presented in Table 1.

Evaluation Metrics. For HotpotQA, we employ two metrics: F1 and exact match (EM). F1 measures the token-level overlap between the prediction and ground truth, while EM calculates the proportion of items whose F1=1. For WebShop, reward ∈ [0, 1] measures how well the purchased item matches the text instruction, and success rate (SR) measures the proportion of items that get reward=1.

Implementation details. We invoke the OpenAI GPT44o (gpt-4o-2024-08-06) API. For all inference, we set temperature=0.7, top p=0.95, max length=512. For finetuning, we choose LLAMA-3.1-8B-Instruct (L-8B) and Mistral-7B-Instruct-v0.3 (M-7B), training for 4 epochs with a learning rate of 5e-6 using 8 NVIDIA 80GB A100 GPUs. We use L-8B for the computation of GE score. Two master's students with domain expertise were responsible for annotation.

4.3 Main Result

EDGE yield effective guidelines. EDGEUG surpasses baselines in both HotpotQA and WebShop, as shown in Table 2, achieving improvements of 13.3% and 13.9%, respectively. The ExpeL autonomously summarizes guidelines, but its effectiveness is constrained by the LLM's limited environmental understanding. This shortcoming hinders ExpeL's ability to generate guidelines that demand a deeper comprehension of the environment or tools. For example, WebShop displays candidate products in a semantic similarity ranking, making the top-ranked products more likely to be target products. Due to ExpeL's lack of understanding of the search engine, it is unable to summarize related guidelines.

GE-selected fine-tuning data outperform others. We use same prompt to generate fine-tuning dataset for other data selection methods. Results in Table 2 show that GE outperforms them across the two datasets using L-8B and M-7B. Notably, ReAct w/ GE (L-8B) even surpassed baselines that used GPT-4o. These findings indicate that the samples selected by GE are more challenging, and their solving trajectories integrate a greater depth of human expertise. FL focuses on selecting samples that are more semantically representative. Although it performs well on generation tasks, results indicate that it struggles with complex tasks that involve multiturn interactions requiring reasoning and decision-making. High Score, which filters and selects entirely accurate samples from labeled data, performs relatively well. Notably, not all samples selected by GE are labeled totally correctly. This means that despite having higher rewards, the data selected by the High Score still yields inferior results compared to GE.

GE efficiently enhances fine-tuning. We compared the performance of different data selection methods with training budget k=[200, 400, 600, 800], as shown in Figure 2. GE consistently outperforms the baselines across all training budgets, achieving more efficient integration of human expertise. Moreover, our method reduces training data usage by 50% on WebShop and 25% on HotpotQA, respectively, while still achieving better performance compared to the baselines.

4.4 Analysis of EDGE

What kinds of samples have high/low GE score? Through manual observation of the samples, we found that GE values tend to be higher when the environment is simple, and the guideline incorporates relevant human expertise. Conversely, GE values are lower in complex environments or when the guideline lacks relevant expertise. These observations align with our hypothesis. We selected a few samples with high/low GE score for illustration. On HotpotQA, Figure 3 (Left) is a sample with high GE score. The LLM followed the guideline, progressively searching for related entities based on the topic entity until the answer was found. Figure 3 (Right) demonstrates actions with low GE score. On WebShop, samples with complex environment caused the LLM to violate the guidelines, resulting in low GE scores. In the top left example, the extensive list of available actions on the product page, combined with relevant information, misled the LLM into click[lemon], which was a non-existent button. The example below is a HotpotQA case, where this sample made the LLM use overly long search keywords.

What guidelines have we summarized? Based on the observation of low GE score samples, we have listed the following representative guidelines in order of their necessity within the samples. Table 4 shows the format of our guideline. For WebShop:

• Click high-ranked products first. A high ranking indicates that the semantics of the product's options or features are more similar to the search content. Even if the titles of high-ranked products sometimes do not fully meet the required criteria, it is still necessary to click on them.
• What is a product title. This guideline introduces that some attributes like color, flavor, etc., will not appear in the product title. LLMs often hope to find products

whose titles perfectly match the instructions.

• Buying a similar product is better than buying nothing. Before the rounds are exhausted, the LLM needs to balance the trade-off and compromise to buy a product that is not perfectly aligned when necessary.

For HotpotQA:

• How to use 'Lookup'. Despite being informed that "the Lookup[] only supports exact matching", the LLM still tends to search for longer keywords. This guideline details that the LLM should search for the shortest possible word that are likely to appear in the original text to match more search results.
• Solving questions without a clear topic entity. When the topic entity cannot be found, LLM can try leverage its prior knowledge to make a reasonable inference or use Search[] based on semantic similarity.
• Understanding the Question. In complex scenarios, if certain representative requirements are met (e.g., the 20th President of the United States), the LLM can respond directly without confirming other constraints.

Does filtering reward=1 trajectories truly lead to high quality? Intuitively, annotated samples with higher rewards suggest higher quality. Thus, existing methods often filter fully correct samples (reward = 1) from many annotated examples, known as High Score. However, does low reward always indicate low quality? We analyzed the annotated samples selected by different methods, as shown in Table 3. Compared to Random, High Score has lower average interaction turns, which typically indicates that the samples are simpler and easier. Combined with the observation from Table 5 that High Score is least likely to select hard questions, we can conclude that High Score includes more simple samples during filtering. This explains why High Score fails to achieve the best performance despite all data reward=1. Notably, despite the lowest annotated data rewards of GE, it achieves the best performance. This is because more challenging samples can better leverage the advantages of human experience from the guidelines. This suggests that data containing more "attempts at challenging problems" represents higher quality for fine-tuning, even if it is not fully correct.

4.5 Effective Analysis in HotpotQA

To investigate whether EDGE expanded the range of solvable problems, we analyzed the distribution of question difficulty levels in HotpotQA (easy, medium and hard). Table 5 presents the proportion of each difficulty level within the subsets selected by different methods. Among the various approaches, only GE selected a higher proportion of hard questions. As shown in Table 6, GE achieved slight advantages on easy and medium questions, and outperformed the baselines on hard questions. Consequently, GE effectively broadened the scope of solvable problems by focusing on out-ofguideline questions.

5 Conclusion

We propose GE metric, which effectively identifies the most informative samples without relying on golden answers. Selecting samples with low GE scores enhances the efficiency and outcomes of prompt engineering and fine-tuning processes for LLMs. Extensive experiments demonstrate the effectiveness of our method, and we provide a fresh perspective on the data quality of LLM-agent fine-tuning.

Limitations and Future Work. The involvement of human effort is required during guideline updates, which may hinder scalability. In the future, we aim to integrate automatic guideline refinement and explore self-improving agents to reduce human involvement.

Acknowledgments

This work was supported by the National Key Research and Development Program of China under Grant No. 2023YFC3304404.