A Survey on Dataset Distillation: Approaches, Applications and Future Directions

1 Introduction

High-quality and large-scale datasets are crucial for the success of deep learning, not only enabling the development of end-to-end learning systems [Schmidhuber, 2015; Bahdanau et al., 2015], but also serving as benchmarks to evaluate different machine learning architectures [Deng et al., 2009; Koehn, 2005]. However, the explosion of deep learning dataset sizes has posed considerable challenges concerning processing, storage, and transfer. Training neural networks often require thousands of iterations on the entire dataset, which consumes significant computational resources and power. Tasks such as hyperparameter optimization [Maclaurin et al., 2015] and neural architecture search (NAS) [Such et al., 2020] are even more resource-intensive. One promising solution is to use smaller datasets with high information density to reduce resource consumption while preserving model performance.

Research in the area of curriculum learning [Graves et al., 2017], active learning [Konyushkova et al., 2017], and coreset selection [Sener and Savarese, 2017] has shown that it is possible to sample a subset of the original data to train neural networks, resulting in models with competitive performance. This also implies that we can train high-performance models with less effort while downstream tasks like continual learning (CL) [Castro et al., 2018; Prabhu et al., 2020], neural architecture search (NAS) will also benefit. Nevertheless, creating an algorithm-agnostic, efficient, and unbiased small dataset to replace the original is still challenging. For instance, coreset selection is typically an NP-hard problem, making it computationally intractable and difficult to apply to large datasets.

* Corresponding author: zongxiong.chen@fokus.fraunhofer.de

An alternative approach to coreset is dataset distillation, which aims to distill the original data onto a smaller synthetic dataset [Wang et al., 2018]. Dataset distillation techniques have continued to evolve, with various methods such as gradient matching [Zhao et al., 2021], trajectory matching [Cazenavette et al., 2022], and kernel ridge regression [Nguyen et al., 2020] being proposed to optimize the distilled data, resulting in improved distillation performance in terms of both the accuracy of the trained model on the test set and the generalization capability across different network architectures. However, there remain challenges regarding optimization stability and computational efficiency.

Despite the recent advancements in dataset distillation, a comprehensive overview summarizing its advances and applications is currently not available. This paper aims to fill this gap by presenting a taxonomy of dataset distillation. To our knowledge, it is the first work that provides a systematic categorization of the different methods and techniques used in dataset distillation. The paper mainly makes the following contributions:

• We propose a novel taxonomy of dataset distillation, which can help researchers to better understand the research landscape and find their areas of interest.
• We present existing distillation approaches in detail, discussing their strengths and weaknesses;
• We discuss important challenges in this domain, highlighting promising directions for future research.

The paper is organized as follows. In Section 2, we first present our taxonomy of dataset distillation. Then, we introduce the learning frameworks and common enhancement methods in Section 3 and Section 4, respectively. Section 5

summarizes the advances in different data modalities. In Section 6, we categorize the related applications according to the dataset distillation properties. Finally, we conclude this paper with future directions in Section 7.

2 Taxonomy

2.1 Basics of Dataset Distillation

We begin by introducing the key notations used in this paper. D represents a general dataset, fθ represents a neural network with parameters θ, and fθ(x) denotes the model's prediction for data point x. The expected loss for dataset D in relation to θ is defined as

where x and y are the input data and label pair from D, ℓ(fθ(x), y) is the given loss value between the prediction and ground truth.

Dataset distillation aims to reduce the size of large-scale training input and label pairs T = {(xi, yi)}|T | i=1 by creating smaller synthetic pairs S = {(ˆxj, ˆ yj )}|S| j=1 , so that models trained on both T and S can achieve similar performance, which can be formulated as:

where θS and θT are the parameters of the models trained on S and T respectively.

2.2 Taxonomy Explanation

The taxonomy of dataset distillation is illustrated in Figure 1. In this taxonomy, we classify the research about dataset distillation from three perspectives: approaches, data modalities and applications. The approaches can be decomposed into two parts. In the learning framework, we explain how dataset distillation can be modeled, optimized and solved in different ways, such as using meta-learning [Andrychowicz et al., 2016] or surrogate objectives (see Section 3.2). Metalearning can be further divided into using back-propagation through time and using kernel ridge regression. Surrogate objective can be subdivided into parameter matching and distribution matching. We categorize the common enhancement methods, which can be plugged into a learning framework, mainly into parameterization (see Section 4.1), augmentation (see Section 4.2) and label distillation (see Section 4.3). Existing work can be classified into four types of data: image, audio, text, and graph, based on data modality. Applications can be further divided into three categories: computationally intensive tasks such as continual learning and neural architecture search, privacy protection including dataset construction and federated learning, and model robustness, encompassing data poisoning attacks and improving robustness. Corresponding to our taxonomy, some representative papers, together with their characteristics, have been listed in Table 1. It comprehensively compares learning frameworks, enhancement methods, data modality, and applications.

3 Learning Frameworks

According to the learning goals, the current learning frameworks can mainly be divided into two categories: metalearning methods based on inner model performance and methods using surrogate objectives.

3.1 Meta-Learning

Meta-learning [Andrychowicz et al., 2016] refers to learning about learning, and often refers to machine learning algorithms that learn from the output of other machine learning algorithms. In this problem, the distilled data are treated as hyperparameters and the objective is to optimize the distilled data in a bi-level optimization problem as follows:

where the inner loop, optimizing θS , trains a model on the synthetic dataset until convergence, and the outer loop, optimizing S, subsequently optimizes the synthetic dataset, so that the model has good generalization capability and can perform well on the real dataset. The distillated dataset is optimized using the meta-gradient:

where α is the learning rate for updating the synthetic dataset.

Back-Propagation Through Time

Computing the meta-gradient ∇SLT (θS ) requires differentiating through inner optimization. When the model is learned in an iterative way, i.e.,

where η is the learning rate for inner loop and meta-gradient is calculated by back-propagation through time (BPTT):

which is illustrated in Figure 2. It is evident that the computation overhead is high due to the recursive calculation of the meta-gradient using Equation 6.

Tomake the implementation of Equation 6 feasible, [Wang et al., 2018] suggest using the Truncated Back-Propagation Through Time (TBPTT) method, which involves unrolling the inner-loop optimization steps as a single step of gradient descent optimization,

where ˆx, ˆ y are synthetic dataset and ˆ η the learning rate for the optimizer.

[Deng, 2022] further improves the learning framework by incorporating a momentum term and extending the length of unrolled trajectories. Empirical results show that the momentum term can consistently improve performance and that longer unrolled trajectories can lead to better model parameters that produce more efficient gradients for compressed representation learning.

BPTT methods have been criticized for several issues, as noted in [Zhou et al., 2022]: 1) high computational cost and memory overhead; 2) bias in short unrolls; 3) gradients exploding or vanishing in long unrolls; and 4) chaotic conditioned loss landscapes.

Kernel Ridge Regression

[Nguyen et al., 2020] transform dataset distillation into a kernel ridge regression (KRR) problem, where the synthetic set is used as the support set and the original set as the target set. Their approach result in a closed-form solution in terms of convex optimization, simplifying the expensive nested optimization into first-order optimization (see Figure 3). They introduce the Kernel-Inducing Point (KIP) algorithm which utilizes neural tangent kernel (NTK) [Jacot et al., 2018] ridge regression to compute the exact outputs of an infinite-width neural network trained on the synthetic set, bypassing the need for gradient and back-propagation computation on any neural network. The KRR loss function for a given kernel and batch data from synthetic set (XS , yS ) evaluated on batch data from real set (XT , yT ) can be formulated as,

where KXT XS is the Gram matrix of XS and XT , and KXSXS is the Grammatrix of XS.

[Zhou et al., 2022] propose a novel method, neural feature regression with pooling (FRePo), which utilizes a more flexible conjugate kernel with neural features to replace the NTK in KIP [Nguyen et al., 2020]. This approach breaks down the traditional KRR training pipeline into two components: a feature broke fθ and a linear classifier. When calculating the meta-gradient of S, FRePo fixes the feature extractor parameters and updates S T times according to Equation 8, where T is a hyperparameter that helps prevent the support/synthetic dataset from memorizing a specific network. Additionally, a model pool is employed to alleviate overfitting in the distillation process.

[Loo et al., 2022] propose to use random feature approximation for distillation (RFAD), which utilizes random feature approximation of the Neural Network Gaussian Process (NNGP) kernel to replace the NTK used in KIP. This approach reduces the computation of the Gram matrix to O(|S|), which is linear with the size of the synthetic set, compared to O(|S|2 ), the complexity of accurately calculating the NTK kernel matrix. They also suggest using cross-entropy loss with Platt scaling [Platt and others, 1999] to provide a more accurate probabilistic interpretation for classification tasks.

3.2 Surrogate Objective

Instead of optimizing directly based on model performance, surrogate objective approaches optimize a proxy objective, such as the parameters or gradients of the model. These approaches assert that the effectiveness of a model trained on a full dataset and a distilled dataset can be inferred from their corresponding parameters and gradients.

Parameter Matching

In contrast to optimizing directly based on the loss value corresponding to the distilled data, it aims to make the model approximate the original model in the parameter space, i.e. θS ≈ θT . Empirically, the trajectory of parameters vary with its initial state θ0. Therefore, the objective of parameter matching should be agnostic to the initialization. When distance between the model parameters trained on the synthetic dataset and the real dataset are consistently close, the distilled dataset can be considered as a good alternative of original whole dataset. Let θS (θ0), θT (θ0) denote the trained models from the same initialization θ0, the objective function can be expressed as:

where D(·, ·) is a distance function.

To enable a more guided optimization and apply the incomplete training, DC [Zhao et al., 2021] synthesizes images by minimizing the gradient matching loss at each training step t:

where T is the hyperparameter for the number of training iterations.

[Cazenavette et al., 2022] suggest overcoming bias accumulated from one-step gradient by matching training trajectories (MTT). MTT considers the training trajectories θT-1 t=0 on real data as the expert models, the model ˆ θ trained on the synthetic dataset as the student model. It randomly samples θT t from the expert model to initialize the student model, and the objective is to make the student model ˆ θS t+N approximate the expert model θT t+M after N iterations. The optimization objective is given by

where M, N are the hyperparameters.

Parameter matching methods are often criticized for: 1) high bias it introduces [Wang et al., 2022]. The synthetic set learned by gradient matching is extremely biased towards those large gradient samples, which will decrease its generalization capability on unseen architectures; 2) expensive bi-level optimization. For example training 50 images/class using DC [Zhao et al., 2021] requires 500K epochs of updating network parameter θt and 50K updating of S; and 3) fragile hyper-parameters [Zhao and Bilen, 2021] tuning. e.g. how often to update θt and S in DC, as well as M,N in MTT [Cazenavette et al., 2022] are critical.

Distribution Matching

The objective of distribution matching is essentially to learn synthetic samples so that the distribution of the synthetic samples is similar to that of real samples in the feature space. They use an empirical estimate of the maximum mean discrepancy (MMD) as a metric to evaluate the distance of spatial distribution. Due to the high computational complexity and difficulty in optimization caused by high dimensionality, Zhao [Zhao and Bilen, 2021] use different randomly initialized neural networks as feature extractors to reduce the input dimension to low-dimensional space.

where fθ is parameterized by θ, and θ is sampled from a random distribution Pθ. |S| and |T | are the cardinality of dataset S and T , respectively.

To better capture the whole dataset distribution, [Wang et al., 2022] propose to use layer-wise feature alignment in CAFE to learn a more comprehensive characteristic of the distribution. They also introduce a loss function to improve the discriminative ability of the learned samples. The classification loss is calculated using the feature centers of real sample and averaged synthetic samples of each class.

4 Common Enhancement Methods

In this section we introduce some techniques that can be integrated into the learning framework presented in the previous section to further enhance distillation performance.

4.1 Parameterization

Dataset parameterization aims to utilize the regularity to guide the synthesis, It helps enhance the interpretability by learning hidden patterns, and control the diversity of the synthetic data. In [Zhao and Bilen, 2022], the authors propose IT-GAN, a method that uses a pre-trained GAN decoder to increase the informativeness distilled data. IT-GAN first obtains latent vectors from training samples using GAN Inversion [Abdal et al., 2019], then it use the distribution matching algorithm to learn the latent vectors. These vectors can be fed into a pre-trained GAN decoder to induce synthetic images of the original size. In addition, most distillation methods processes each synthetic sample independently, ignoring mutual consistency and relationships between samples. Factorization are proposed to decompose images into different parts to better capture the correlation between different samples and improve the diversity. IDC [Kim et al., 2022] utilizes a multi-formation function as the decoder as the decoder to store more information in single sample. [Deng, 2022] propose to learn matrix-based codes and decodes and use matrix multiplication to generate synthetic datasets. [Lee et al., 2022] employ the latent code - decoder mode for factorization. The decoder is designed as an upsampling neural network containing three ConvTranspose2d layers, aiming to restore latent codes compressed in low dimensions into the image pixel space. [Liu et al., 2022b] propose HaBa, which chooses to decompose the image into two parameter spaces of basis and hallucinator. Where hallucinator is an encodertransformation-decoder structure. Specifically, the encoder is composed of CNN blocks, followed by an affine transformation with scale σ and a decoder of a symmetric CNN architecture.

4.2 Augmentation

In [Zhao Bo, 2021], the authors propose using differentiable siamese augmentation (DSA) when learning synthetic images, which leads to more informative datasets. DSA is a pluggable technique that includes operators like scale, flip, crop, rotate, color jitters, and cutout. It can be easily integrated into various distillation methods and has been widely used in [Zhao and Bilen, 2021; Wang et al., 2022]. In [Cui et al., 2022], DSA is found to achieve the best performance compared to other data augmentation techniques. However, current augmentation techniques are not suitable for discrete data such as graphs and text.

4.3 Label Distillation

Label distillation relaxes the restrictions on labels, allowing them to have richer semantics beyond one-hot vectors. It is first introduced in SLDD [Sucholutsky and Schonlau, 2021] and has been shown to improve not only the storage efficiency but also the distillation performance. Their method only requires to make the label in Equation 7 learnable variables. [Nguyen et al., 2020] also provide a label learning algorithm based on the closed-form solution in KRR. It is reported that only five distilled images from MNIST would enable the model to achieve 92% accuracy [Sucholutsky and Schonlau, 2021].

5 DataModalities

Dataset distillation, first proposed for images, has been applied to various modalities. In this section, we categorize existing works according to data modality and discuss some of the challenges.

5.1 Image

Most dataset distillation methods to date have been performed on image datasets [Wang et al., 2018; Nguyen et al., 2020; Zhou et al., 2022; Zhao et al., 2021; Kim et al., 2022; Cazenavette et al., 2022]. These works have constructed benchmarks to facilitate fair comparisons of novel approaches. Images have a continuous real-value domain, which allows direct optimization of synthetic images using deep learning optimizers. We find that experimental datasets become increasingly complex, starting from MNIST, CIFAR10, and SVHN, to more challenging datasets like TinyImageNet and ImageNet [Zhou et al., 2022; Cazenavette et al., 2022; Kim et al., 2022]. Furthermore, parameterization methods that capture on the regularity of images are becoming increasingly prevalent in the field, as evidenced by recent research such as [Kim et al., 2022; Zhao and Bilen, 2022; Liu et al., 2022b].

5.2 Audio

Speech signals also satisfy the regularity condition of a lowrank data subspace, i.e., temporally adjacent signals haves similar spectra. Therefore, many parametrization methods [Liu et al., 2022b; Kim et al., 2022] designed for image dataset can also be applied in this domain. They both experiment with the Speech Commands [Warden, 2018] dataset. In detail, they process the waveform data with a short-time Fourier transform to obtain the magnitude spectrogram and used log-scale magnitude spectrograms for the experiments. Their works show that dataset distillation can achieve consistent performance on downstream tasks of speech signals.

5.3 Text

The discrete nature poses challenges to textual distillation. [Sucholutsky and Schonlau, 2021] first embed the text into a contiguous space using pre-trained GloVe embedding and fill or truncate all sentences to a pre-determined length. In this way, each sentence can be regarded as a single channel image of size length × embedding dimension. Text distillation also involves finding the nearest embedding in the dictionary for each vector in the optimized matrix, and transforming these embeddings into the corresponding words and finally the sentence.

Current efforts are based on primitive bi-level optimization, which is computationally inefficient. There is a lack of work analyzing factors such as the difficulty of the dataset, sentence length, or cross-architecture generalization. Distilled sentences may consist of unrelated words, which makes it difficult to interpret and further analyze. Exploring ways to leverage regularity and context in text distillation is a promising area of research.

5.4 Graph

Graph data is very common in real life, e.g. social networks, Web relationship analysis, and user-item interaction can all be modeled as graph data containing nodes and edges. [Jin et al., 2021; Jin et al., 2022] design a strategy to simultaneously compress node features and structural information based on gradient matching. [Liu et al., 2022a] adopt the distribution matching to boost the performance and show that the dataset distillation was significantly efficient and in some datasets they reached 95% of the original performance by compressing 99% of the data. Graph distillation is mainly challenged by heterogeneous, abstract, high-level graph representations.

6 Applications

Dataset distillation, initially designed for model training acceleration, has shown potential in various applications due to its properties.

6.1 Computationally Intensive Tasks

Continual Learning

Continual learning (CL) addresses the problem of catastrophic forgetting by using strategies such as experience replay, which stores representative samples from previous tasks as a buffer to recall knowledge. Dataset distillation, which involves highly compressed representations, is an alternative to traditional sampling methods. There are currently two experimental settings for using distillation in CL. [Zhao et al., 2021; Zhao Bo, 2021] use different datasets, SVHN, MNIST, and USPS, three handwritten digit recognition datasets, and take EEIL [Castro et al., 2018] as the baseline for continuous learning. In the study of [Zhao and Bilen, 2021], the experimental settings are changed to incremental class learning on the CIFAR100 dataset. The researchers establish a baseline using the GDumb method [Prabhu et al., 2020] and randomly divided 100 classes into 5 and 10 learning steps, with 20 and 10 classes per step respectively.

Neural Architecture Search

Neural architecture search (NAS) is known to be expensive as it involves training multiple architectures on the entire training dataset and selecting the best-performing one on the validation set. To address this issue, researchers have proposed using a distilled dataset as a proxy of the entire dataset, which can effectively identify the best network. Related experiments on the CIFAR10 dataset have been reported in DC [Zhao et al., 2021], DSA [Zhao Bo, 2021], and DM [Zhao and Bilen, 2021]. These studies construct a search space of 720 ConvNets by varying hyperparameters such as network depth, width, activation function, normalization, and pooling layers over a uniform grid. The effectiveness of the distilled dataset was evaluated using the Spearman's rank correlation coefficient between the validation accuracies obtained by the proxy dataset and the entire dataset. A higher correlation value indicates that the proxy dataset is more reliable.

6.2 Privacy

Dataset Construction

Machine learning is vulnerable to a variety of privacy attacks, such as membership inference attacks [Shokri et al., 2017], model inversion attacks [Fredrikson et al., 2015; Carlini et al., 2023], and gradient inversion attacks [Geng et al., 2021; Geng et al., 2023], where attackers attempt to infer taskindependent private information from the target model, and even recover the original training data. Additionally, data collection and publishing raise privacy and copyright concerns. [Dong et al., 2022; Zhou et al., 2022] have shown that models trained on synthetic data are robust to both lossbased and likelihood-based membership inference attacks. To ensure that the distilled samples cannot be inferred from real ones, [Nguyen et al., 2020] implemented the KIPρ variant, which randomly initialized ρ proportion of each image and kept them unchanged during training. This idea was later followed by RFADρ [Loo et al., 2022]. [Chen et al., 2022] added a differential privacy (DP) mechanism [Dwork, 2006] to the distillation process to provide rigorous privacy guarantees. Medical data often requires strict anonymization before publication, [Li et al., 2022] propose to dataset distillation to construct privacy-preserving datasets.

Federated Learning

Federated learning (FL) is an emerging technology that enables different clients to collaboratively train a shared model without sharing their local data. It faces challenges such as high bandwidth requirements for uploading large model updates and a lack of strict privacy guarantees. There are several works that propose to combine dataset distillation in FL. [Hu et al., 2022; Xiong et al., 2022] suggest sharing lightweight synthetic datasets instead of sharing model updates, since the distilled dataset size is generally smaller. However, this may introduce bias and increase the computational load, which can negatively impact the performance and efficiency of FL.

6.3 Robustness

Data Poisoning Attacks

Distilled data lose its fidelity and may not be visually distinguishable from its original contents, making it vulnerable to data poisoning attacks and difficult to detect. Studies have shown that a small number of these poisoned samples can significantly reduce the accuracy of a model's predictions on a specific category. [Wang et al., 2018] propose a study on data poisoning attacks using dataset distillation. [Liu et al., 2023] propose two backdoor attacks on distilled data by injecting triggers into the synthetic data during the distillation process, either in the initial stage or throughout the entire process.

Improve Model Robustness

Dataset distillation can also be used as a means of improving its robustness. Researchers have proposed using optimization techniques to learn a robust distilled dataset, such that a classifier trained on this dataset will have improved resistance to adversarial attacks. [Tsilivis et al., 2022] have combined the KIP method with adversarial training to enhance the robustness of the distilled dataset. [Wu et al., 2022] approached the problem of dataset learning as a tri-level optimization problem to obtain a distilled dataset that minimizes robust error on the data-parameterized classifier.

7 Conclusion and Future Directions

In this paper, we present a systematic review of recent advances in dataset distillation. We introduce a novel taxonomy that categorizes existing works from various perspectives. We find that most existing efforts are geared toward image datasets, whereas the handling of discrete text and graph data remains a significant challenge. There is a limited exploration of robustness, and further research is necessary as the technology gains wider adoption. Our study demonstrates the research landscape in this field and suggests directions for future work.

7.1 Computational Efficiency

The computational efficiency of dataset distillation is an important consideration, as many current methods for dataset distillation can be computationally expensive, particularly for larger datasets. The goal of dataset distillation is to reduce the size of a dataset while preserving its key features and patterns, but this process often requires complex optimization and clustering algorithms, which can be computationally intensive. Methods like MTT [Cazenavette et al., 2022], KIP [Nguyen et al., 2020], and FRePo [Zhou et al., 2022] can cause GPU memory bottlenecks when the number of images per class (IPC) increases. While the DM [Zhao and Bilen, 2021] approach proposes using distribution matching to avoid model training, and RFAD [Loo et al., 2022] proposes using NNGP to reduce the computational complexity of kernel ridge regression, the computational efficiency of distillation still requires improvement, particularly for larger datasets.

7.2 Performance Degradation on Larger IPC

According to [Cui et al., 2022], current dataset distillation methods perform well only when the number of images per class (IPC) is relatively small. As the IPC increases, the performance of most distillation methods deteriorates and becomes similar to that of random sampling. Therefore, it is important to explore whether dataset distillation can overcome this limitation and maintain superior performance on larger datasets.

7.3 Weak Labels

Currently, research on dataset distillation primarily focuses on classification tasks. However, its potential for more complex tasks, such as image detection and segmentation, named entity recognition, summarization, and machine translation, remains untapped. Exploring the technique's effectiveness on these tasks could provide deeper insights into data characteristics and the inner workings of AI.

Acknowledgements

This work is partially funded by the European Union's Horizon 2020 Research and Innovation Program through Marie Skłodowska-Curie Grant 860627 (CLoud ARtificial Intelligence For pathologY (CLARIFY) Project). We also sincerely thank Awesome-Dataset-Distillation for its comprehensive and timely DD publication summary.