D3: Diversity, Difficulty, and Dependability-Aware Data Selection for Sample-Efficient LLM Instruction Tuning

1 Introduction

With the success of large language models (LLMs) [Brown et al., 2020; Touvron et al., 2023], the application of datadriven AI has been significantly advanced across various industries and scenarios. In the LLMparadigm, large-scale pretraining on massive corpora equips models with knowledge and text generation capabilities based on next-token prediction [Naveed et al., 2024]. Moreover, several post-training methods and techniques [Bai et al., 2022; Wang et al., 2024b; Rafailov et al., 2024] could further enhance the model's alignment and performance on downstream tasks. In particular, instruction tuning [Ouyang et al., 2022; Taori et al., 2023], which fine-tunes models with labeled pairs of instruction and response, plays a crucial role in enabling LLMs' instruction-following capabilities across various tasks.

∗Work done during the authors' internship at Alibaba Group.

Early studies on instruction tuning concentrated on creating extensive and cross-task datasets to enhance LLMs at a significant cost [Wang et al., 2022; Xu et al., 2023; Wang et al., 2023]. However, recent work [Zhou et al., 2023] has shown that a small, carefully curated set of high-quality data can effectively equip LLMs with instruction-following capabilities instead of relying on large datasets. This finding has prompted initial efforts in automated data selection [Xia et al., 2024; Cao et al., 2024; Du et al., 2023; Li et al., 2024a; Li et al., 2024b], which attempt to identify the valuable subset of instruction data for fine-tuning. Despite their potential, these methods generally face three limitations: 1) Many approaches rely on a single metric for data selection, neglecting comprehensive assessments of instruction data. 2) Due to the inherent difference between inter- and intra-sample metrics like diversity and quality, existing methods often heuristically consider each in separate stages with manual thresholds to select data, hindering the dynamic balance of different aspects. 3) Lack of feedback to adaptively refine the selection focus.

To address the challenges above, we first establish data selection criteria based on three distinct aspects: diversity, difficulty, and dependability, corresponding to inter-sample, sample-model, and intra-sample perspectives of data value, respectively. Building upon these criteria, we propose D3, an effective data selection method consisting of two key steps: scoring and selection. In the scoring step, specifically, to cope with the commonly observed context-oriented generation diversity (CoGD) in text generation, we introduce the uncertainty-based prediction difficulty (UPD), a novel score that quantifies the prediction difficulty of each sample by mitigating the interference caused by CoGD. For diversity and dependability, we define the diversity function based on sample embeddings and use a teacher model to assess the sufficiency and reliability of both the sample's instruction and its annotated response. In the selection step, we formulate the D3 weighted coreset object, which jointly optimizes the three aspects of the data value to solve for the most valuable data subset. Additionally, the scoring and selection steps can iterate through multiple rounds. This allows for adaptive refinement of the selection focus by incorporating sample-level feedback from the model during instruction tuning.

We conducted experiments on public datasets and realworld application to validate the D3 method. Remarkably, with only 5% to 10% of the entire dataset, our approach achieves competitive or even superior performance compared to full-dataset fine-tuning. This highlights the feasibility of D3 for sample-efficient instruction tuning, which enhances both efficacy and efficiency, thereby advancing related research. To summarize, our main contributions are as follows:

1. Problem: We establish data selection criteria based on three distinct aspects: diversity, difficulty, and dependability, corresponding to inter-sample, sample-model, and intra-sample perspectives of data value, respectively, which offers systematic guidance for the problem of data selection to enable sample-efficient instruction tuning.
2. Method: We propose the D3 approach, consisting of two key steps. In scoring, we define the diversity function, introduce the uncertainty-based prediction difficulty and integrate a teacher model for dependability assessment. In selection, we formulate the D3 weighted coreset object to jointly optimize three criteria and solve for the most valuable data subset. The two steps can iterate multiple rounds to adaptively refine the selection focus.
3. Evaluation: Extensive experiments validate the effectiveness of our proposal, demonstrating its potential for sample-efficient instruction tuning to enhance both efficacy and efficiency, thereby facilitating related research.

2 RelatedWork

2.1 Instruction Tuning for LLMs

Instruction tuning [Brown et al., 2020; Touvron et al., 2023] has emerged as a widely adopted post-training strategy to equip language models with instruction-following capabilities for various downstream tasks. Early research [Khashabi et al., 2020; Ye et al., 2021] primarily focused on manually creating task-specific instruction tuning datasets. More recent approaches [Honovich et al., 2022; Wang et al., 2022; Xu et al., 2023; Wang et al., 2023; Taori et al., 2023] shift toward constructing instruction tuning datasets by leveraging powerful LLMs to generate data. These studies facilitate the automatic data creation via distilling knowledge from the teacher model [Wang et al., 2024a; Long et al., 2024]. These foundational studies demonstrate that large task-specific or crosstask instruction tuning datasets can boost the instructionfollowing performance of language models.

2.2 Sample-Efficient Tuning

However, large instruction tuning datasets are often plagued by concerns over quality and redundancy. Seminal studies [Zhou et al., 2023] have suggested that a small, manually curated set of high-quality data can endow the model with powerful instruction-following capabilities, motivating recent research to select valuable subsets for training from large datasets. These efforts generally focus on quality and diversity assessments. [Du et al., 2023] employs a reward model to assess sample quality, while [He et al., 2024] introduces the use of Shapley value to measure the marginal contribution of samples. [Li et al., 2024b; Li et al., 2024a] propose the IFD score to measure the model's difficulty in instruction following to samples. [Kung et al., 2023] adopts an active learning framework to select samples based on prompt uncertainty. For diversity, k-means or k-center [Sener and Savarese, 2018] are typically employed to select data [Ge et al., 2024; Du et al., 2023]. However, many approaches rely on a single measurement for data selection, neglecting a comprehensive assessment of instruction samples. Additionally, due to the inherent difference between inter-sample metrics like diversity and intra-sample metrics like quality, existing methods typically consider each aspect separately to select data with manual thresholds, lacking a unified way to jointly optimize multiple evaluation criteria. Besides, most methods lack feedback mechanisms to refine the selection focus. Moreover, the evaluation process in some methods is computationally expensive, such as using LLMs to generate responses for all training data, resulting in inefficiencies.

3 Methodology: D3

3.1 ProblemFormalization

In the task of data selection for the sample-efficient LLM instruction tuning, we are provided with an instruction tuning data pool D = {z(i) } consisting of paired instruction and response samples z(i) = (x(i) , y(i) ), and a pre-trained LLM F to be fine-tuned on D. Our goal is to select a data coreset Q from the entire dataset D under a selection budget constraint of |Q|/|D| = k, such that the model FQ, fine-tuned from F on Q, achieves optimal performance on the test dataset Dt.

3.2 Overall Framework

To tackle the challenge of identifying and selecting the most valuable coreset for instruction tuning, we first establish data selection criteria based on three distinct aspects: diversity, difficulty, and dependability, corresponding to inter-sample, sample-model, and intra-sample perspectives of data value, respectively. We then propose the novel D3 method for selecting data based on these criteria. As illustrated in Figure 1, our method comprises a warm-up step and two key steps of scoring and selection. We first utilize a tiny portion of data to obtain the initially acclimated model F0. In the scoring step, we evaluate samples in D across three criteria. Specifically, we define the diversity function to measure sample distinctiveness and introduce the uncertainty-based prediction difficulty (UPD) to assess sample difficulty by mitigating the interference of context-oriented generation diversity. Additionally, we incorporate a teacher LLM F ∗ to evaluate sample dependability. In the selection step, we formulate the D3 weighted coreset objective to solve for the most valuable subset for LLM tuning within the selection budget. We integrate the scoring and selection steps into an iterative process that leverages the feedback from LLM to enable the adaptive refinement of the selection focus over multiple rounds.

3.3 D3 Criteria and Scoring

Unlike large-scale pre-training on vast data, we seek to equip LLMs with instruction-following capabilities using much fewer labeled samples. To identify the most valuable data subset, we establish data selection criteria based on three distinct aspects of diversity, difficulty, and dependability. Diversity, from an inter-sample perspective, quantifies the distinctiveness of each sample against others. Difficulty, a samplemodel metric, gauges the challenge for the language model in fitting and predicting a given sample. Dependability serves as an intra-sample criterion to assess whether the sample itself is sufficient and reliable enough for training. These criteria offer a comprehensive evaluation of instruction samples from three independent perspectives, providing the systematic guidance for data selection to identify the most valuable data coreset with maximal diversity, difficulty, and dependability.

Sample Diversity

The selection of the instruction tuning data coreset should account for its diversity characteristics and representativeness, ensuring that the selected samples accurately capture the overall distribution of the entire dataset D. This helps mitigate the risk of the model over-fitting to specific task types. We define the diversity function d1 of any sample z(i) as the distance to its closest point in the coreset Q, which quantifies the distinctiveness of the sample:

where d(·, ·) denotes the distance between two samples. Therefore, selecting a coreset with maximal diversity and representativeness with respect to D is equivalent to identifying Q that minimizes the maximum diversity across all samples:

Minimizing maxz∈D d1(z,Q) ensures the coreset Q closely reflects the entire dataset distribution by reducing the discrepancy between any sample from D and the coreset Q.

Sample Difficulty

It is evident that the prediction difficulty for the model varies across different samples, meaning not all samples contribute equally to improving the model's instruction-following capability. Following the process of minimizing cross-entropy loss for each token during language model training, the sample-wise cross-entropy loss L(i) directly quantifies the model's fit and prediction difficulty for any given sample z(i) :

where y(i) <t denotes the token sequence preceding the t-th token, L(i) t represents the loss of the t-th token given instruction x(i) and y(i) <t , and |y(i) | is the total number of tokens in y(i) .

However, improving an LLM's capability to follow instructions does not precisely align with minimizing loss on a limited amount of labeled instruction tuning data. As illustrated in Figure 2, due to the auto-regressive mechanism in current language models, inherent generation diversity caused by the preceding context commonly occurs. We term this phenomenon context-oriented generation diversity (CoGD). A more intuitive understanding is that, for open-ended text generation, a single instruction can correspond to many valid responses. During pre-training, the model learns the distribution of generation diversity from vast data, which provides a broad coverage of this diversity. In contrast, during instruction tuning, where the amount of training data is considerably reduced, overlooking CoGD in data selection risks overfitting. Specifically, selecting samples with high loss caused by CoGD may lead the LLM to over-fit particular samples or tokens from distributions with greater generation diversity, thereby undermining its instruction-following capability.

To mitigate the interference of CoGD on the difficulty measurement, we introduce the uncertainty-based prediction difficulty (UPD) to measures the model's prediction difficulty for samples by accounting for generation diversity. Since it is impractical to consider all potential responses to a given instruction at the sentence level, we quantify this by calculating the Shannon entropy [Shannon, 1948] of token-wise predictions output from the LLM. This uncertainty reflects the generation diversity based on the current context and the pre-trained knowledge in LLM:

where H(i) t is the entropy of the t-th token prediction given x(i) and y(i) <t , and wj denotes each token in the vocabulary.

Despite the high loss, tokens with higher uncertainty indicate greater context-oriented generation diversity. In data selection, we reduce the attention given to such tokens to mitigate the risk of over-fitting during instruction tuning. The UPD score for tokens and samples is computed as follows:

where σ(u) = 2 ( 1 1+e-u/α - 12 ) ∈ [0, 1] represents the sigmoid transformation mapping the loss score to [0, 1], and vsize is the size of the token vocabulary. Samples with a smaller UPD score typically exhibit either lower loss, indicating easier prediction, or higher uncertainty, reflecting greater generation diversity. These samples should be assigned lower weights to be considered and selected in the coreset.

Algorithm 1 The D3 Data Selection and Fine-tuning.

• 1: Input: Instruction tuning data pool D, Pre-trained LLM F, Teacher LLM F ∗ , Selection budget k, Warm-up budget k0, Total rounds R, Fine-tuning Algorithm Asft
• 2: Procedure:
• 6: for z(i) ∈ D do
• 7: Calc. the sample embedding ψFr-1 (z(i) ) for the diversity function d1(z(i) , ·) in Eq.(1).
• 8: Calc. the difficulty score d2(z(i) ) by Eq.(6), (7).
• 9: Calc. the dependability score d3(z(i) ) by Eq.(8).
• 10: end for
• 11: Initialize Qr ← ∅ ▷ D3 selection step
• 12: while |Qr|/|D| < k/R do
• 13: Q=Q1 ∪ · · · ∪Qr-1 ∪Qr
• 14: z∗ ← argmaxz {d1(z,Q) · d2(z) · d3(z)}
• 15: Qr ←Qr ∪ {z∗ }
• 16: end while
• 17: Fr ←Asft(Fr-1,Qr)
• ▷ Instruction tuning
• 18: end for

• 3: Warm-up set Q0 ← sample(D, k0) ▷ Warm-up step
• 4: Warm-up LLMF0 ←Asft(F,Q0)
• 5: for round r in {1 · · ·R} do
• ▷ D3 scoring step

It is worth noting that some uncertainty-based machine learning methods, such as in active learning [Shao et al., 2022a; Shao et al., 2022b], often prioritize samples with higher uncertainty, as these samples are less familiar to the model and more informative, which contrasts sharply with the motivation behind UPD. Unlike training small models from scratch, LLMs are typically pre-trained on vast corpora, enabling them to acquire extensive knowledge and text generation capabilities [Zhao et al., 2024]. Instruction tuning, which involves relatively much less training data, primarily serves to equip the models with instruction-following capabilities rather than imparting new, low-level knowledge. Therefore, the priorities during instruction tuning should be correcting the model's confidently incorrect prediction patterns rather than reducing the context-oriented generation diversity. From this perspective, we observe that samples with higher UPD scores are primarily due to the model's weak instructionfollowing rather than factors related to CoGD. This suggests that the model is poorly aligned with these instruction samples, and such samples should be prioritized for instruction tuning to enhance the model's performance.

Sample Dependability

The difficulty score based on UPD helps identify samples with greater difficulty, which is less likely to be caused by CoGD. However, if a sample itself contains factual errors, extraneous symbols, or lacks fluency, it can undermine both the data selection and instruction tuning. To tackle this, we integrate an external teacher model to assess the dependability of samples through prompt-based evaluation. This helps filter out unreliable samples, reducing the risk of incorporating erroneous supervision during instruction tuning. By leveraging the teacher's instruction-following capability, we obtain a soft dependability score d3 within (0, 1) for each sample:

where F ∗ represents the teacher model, wpos and wneg correspond to the positive and negative tokens, respectively. T(z(i) ) is the evaluation prompt generated based on sample z(i) , and F ∗ (w|T ) refers to the logit of the next token w given the inputs T(z(i) ). Details are provided in the appendix.

3.4 D3 Selection Objective

Finally, under a specific selection budget k, we select a data coreset that captures the diversity of the overall distribution while ensuring its samples exhibit great difficulty and dependability. We formalize this multi-criteria sample selection problem as the D3 weighted coreset objective, aiming to solve for the coreset Q which jointly optimizes the three aspects of selection criteria under the budget constraint:

where dψF represents the cosine distance between the embeddings of samples ψF (·) produced by the LLM F. We tackle this NP-hard combinatorial optimization problem with the greedy iterative approach. Specifically, we initialize Q as an empty set and randomly select a sample to add to Q. In each iteration, we compute the weighted nearest distance between the remaining samples and Q, selecting the sample with the largest distance to add to Q. We repeat this process until the desired number of samples is obtained.

We conduct multiple rounds of scoring and selection steps. In each round r, we select the coreset Qr based on the scoring results and then fine-tune the model to obtain Fr. Subsequently, leveraging the sample-level real-time feedback from Fr, we re-score and re-select the samples for subsequent finetuning, which enables the adaptive refinement of the selection focus. The process of D3 is summarized in Algorithm 1.

4 Empirical Study

4.1 Experimental Setup

Training Datasets. We employ three commonly used instruction tuning datasets in the following experiments: 1) The Alpaca dataset [Taori et al., 2023], generated through the selfinstruct method [Wang et al., 2023] and Davinci003 model, consists of 52,002 samples. Given that the data generation relies on Davinci003, the overall quality of the dataset is moderate, akin to datasets collected in real-world tasks that may include low-quality or erroneous samples. 2) The AlpacaGPT4 dataset [Peng et al., 2023] is a refined version of the Alpaca dataset generated using GPT-4, resulting in significantly higher quality. 3) The WizardLM dataset [Xu et al., 2023], created by the Evol-Instruct method with ChatGPT, consists of 70,000 high-quality instruction samples in total.

Test Datasets. We evaluate the effectiveness of different data selection strategies using five commonly used public test datasets: WizardLM [Xu et al., 2023], Self-instruct [Wang et al., 2023], Vicuna [Chiang et al., 2023], Koala [Vu et al., 2023], and LIMA [Zhou et al., 2023]. These test datasets consist of approximately 1,000 instruction-based queries in total, curated across diverse tasks and domains. They provide a comprehensive evaluation of the models' capabilities to follow instructions in real-world tasks.

Real-world Dataset. We further evaluate our method in a real-world setting within Taobao Live, where the language model is fine-tuned using collected live-streaming data to generate high-quality sales-pitch scripts tailored to product information, thereby empowering streamers to improve their performance. We utilize 64,881 collected samples for training, with a random selection of 300 reserved for testing. A pairwise evaluation of generated scripts is conducted across five evaluation dimensions of interest in the task context.

Candidate Methods. We compare our proposed D3 method with the following data selection strategies:

• Full: No data selection strategy. The entire instruction tuning dataset is utilized to fine-tune the LLMs.
• RAND: A subset of training data is randomly sampled from the original instruction tuning dataset.
• PPL: The samples with the highest perplexity are selected, corresponding to those with the highest loss.
• IFD [Li et al., 2024b]: The IFD score, calculated as the ratio of loss to conditional loss, identifies samples with weaker instruction-following. The subset of samples with the highest IFD scores is then selected.

To ensure fairness and adherence to the settings and procedures of candidate methods, we use the pre-trained Llama27B as the initial model. All methods first undergo two training epochs on a warm-up set, which consists of 1% of the data randomly sampled from the data pool. Subsequently, each strategy selects a data subset of equal size for instruction tuning based on a given selection budget. Samples selected are trained for three epochs to obtain the fine-tuned LLMs.

4.2 Evaluation and Benchmark

Pairwise Evaluation. Evaluating the instruction-following capabilities of LLMs on open-ended natural language generation tasks remains a significant challenge. Building on recent advancements in LLM evaluation, such as LLM-asa-judge [Zheng et al., 2023; Li et al., 2025], a more capable LLM is employed as a judge to assess response quality. We generate two responses for each instruction sample in the test set Dt: one from the model fine-tuned on a small subset selected and the other from a model fine-tuned on the full dataset. The judge LLM assigns preference scores, and based on these scores, we classify the outcome as a win, tie, or loss for each sample. Each response pair is evaluated twice in different orders to mitigate positional bias. The final winning score is computed from outcomes of the entire test set. Pairwise evaluations are performed across all five test datasets.

Benchmark. We also evaluate models on the widely recognized AlpacaEval Leaderboard [Li et al., 2023], which provides automated assessments using the AlpacaFarm dataset [Dubois et al., 2023]. This evaluation compares the model outputs with those of Davinci003 by ChatGPT. Additional details on the evaluation are provided in the appendix.

4.3 Main Results

We present the primary comparison results in Table 1. For all three instruction tuning datasets, we allocate a k = 5% selection budget for sample-efficient instruction tuning. To ensure a fair comparison, all strategies select data to reach the budget in one go, except for the Full strategy, which uses all the data. Specifically, we set the total round R = 1 in D3. Our proposed D3 method demonstrates superior results by effectively selecting a small, yet highly valuable subset of data, enhancing the model's instruction-following capabilities on both medium and high-quality instruction tuning datasets. It outperforms the Full strategy in both performance and efficiency. In contrast, using perplexity, or loss, as a measure of sample value for data selection guidance fails to account for diversity and difficulty, resulting in considerably poorer performance. The RAND strategy, which can be seen as a simple approach that focuses solely on diversity to capture the overall data distribution by random sampling, achieves performance slightly weaker than Full. However, it falls short of outperforming Full due to the lack of a more in-depth samplelevel evaluation. The IFD method provides a deeper analysis of sample-wise instruction-following difficulty, helping the model enhance performance with less data. However, as it relies solely on a single metric for data selection, it leaves room for further improvement. Besides, the results reported by AlpacaEval show that it is slightly less competitive than the Full strategy on two high-quality instruction tuning datasets. The performance comparisons on the real-world dataset are presented in Table 2, under a fixed selection budget k = 10%, further demonstrating the superiority of our method.

4.4 Performance with Varied Budgets

We systematically examine the impact of selection budgets on various strategies by incrementally increasing the budget. In line with the setup in the main experiment, all strategies select the required data in a single round. Figure 3 presents the winning score on the Alpaca dataset. As the budget increases, the performance of more potent strategies exhibits a trend of initially improving and then declining. Conversely, weaker strategies gradually approach the performance of the Full strategy. This is because selecting more valuable data positively contributes to model instruction tuning. However, once the effective data reaches saturation, incorporating additional data of slightly lower quality or with redundancy leads to diminishing returns, ultimately causing the performance to converge with that of the Full strategy. Specifically, D3 achieves superior performance with the smallest budget, highlighting the effectiveness of our strategy in selecting the most valuable coreset data for sample-efficient instruction tuning. Similarly, the results on the other two high-quality datasets, as reported in the appendix, also empirically validate this. In contrast, other strategies display weaker performance trajectories or slower growth, which further underscores the superiority and effectiveness of our approach.

4.5 Ablation Study

We design three ablation strategies by individually removing diversity (w/o1), difficulty (w/o2) and dependability (w/o3) to validate the effectiveness of the three data selection criteria in D3. Table 3 shows the performance of the three ablation strategies. Given that Alpaca is a medium-quality dataset, neglecting diversity causes the model to overly focus on certain challenging and highly dependable samples, resulting in a significant performance drop. In the case of the other two high-quality datasets, most samples have dependability scores close to 1, making D3-w/o2, to some extent, a strategy that only considers diversity. As a result, its performance is similar to but slightly better than the RAND strategy shown in the main results. The performance impact of removing dependability is more pronounced for Alpaca due to the quality difference of datasets. These validate the necessity of comprehensively considering all three criteria in data selection.

4.6 Selection withMulti-Round

In this section, we present the performance of the D3 with multi-round selection. The budget increment is 2.5% for the Alpaca dataset and 5% for the other two datasets. Figure 4 compares the performance of multi-round selection D3(MR) with single-round selection D3. As observed, on the Alpaca dataset with medium quality, D3(MR) could adaptively refine the selection focus during the instruction tuning process by leveraging real-time model feedback, thereby better identifying the most valuable subset of data to enhance performance significantly. On datasets with inherently high-quality samples, although the selected small amount of data is already sufficient, multi-round selection still provides some benefits in rapidly improving performance. Note that since D3(MR) involves re-scoring in each round, employing multi-round selection necessitates a trade-off between performance gains and the additional computational overhead.

5 Conclusion

This paper investigates data selection for sample-efficient LLM instruction tuning. We first establish data selection criteria based on three distinct aspects of diversity, difficulty, and dependability, offering the systematic guidance for data selection. We then propose a novel method D3 to identify valuable subsets from large datasets by scoring and jointly optimizing three criteria. Our results show that instruction-following capabilities can be achieved with significantly smaller datasets, improving the effectiveness and efficiency of instruction tuning. One direction for future work is to explore token-wise weights on loss based on selection during instruction tuning.

Acknowledgments

This research was supported by the Key Program of Jiangsu Science Foundation (BK20243012), the Jiangsu Science Foundation (BG2024036), the Fundamental Research Funds for the Central Universities (022114380023), and Alibaba Group through Alibaba Innovative Research Program.

Contribution Statement

Jia Zhang and Yao Liu contribute equally as co-first authors. Yi Liu and Lan-Zhe Guo serve as co-corresponding authors.