DANCE: Dual-View Distribution Alignment for Dataset Condensation

1 Introduction

Recently, the reliance on large-scale datasets, which may include millions or even billions of examples, has become essential for developing state-of-the-art (SOTA) models [Zhao and Bilen, 2021a; Xia et al., 2022; Li et al., 2023; Li et al., 2024; Zhang et al., 2024b]. However, this reliance brings significant challenges, primarily due to the substantial storage costs and computational expenses required for training such models. These challenges pose formidable obstacles, particularly for startups and non-profit organizations, making advanced model training often unattainable [Coleman et al., 2020; Sorscher et al., 2022; Zheng et al., 2023; Jin et al., 2022; Yang et al., 2023; Geng et al., 2023; Xia et al., 2024].

∗ Shiming Ge is the corresponding author.

view, it ensures that the synthetic data remains a faithful proxy of the real data throughout the training process. For inter-class view, it also prevents the synthetic data from falling outside the real class region (the domain where all real data points of that class reside), which may change the decision boundary of the learned classifier.

As a remedy, Dataset Condensation (DC), also known as Dataset Distillation, has emerged as a prominent solution to address the challenges of data burden [Wang et al., 2018; Cui et al., 2022; Yu et al., 2024]. It involves learning a small condensed training set to replicate the performance of models trained on larger real datasets. Pioneer methods in this area typically focus on matching either the gradients [Zhao and Bilen, 2021a; Zhao and Bilen, 2021b; Kim et al., 2022; Wang et al., 2023] or parameters [Cazenavette et al., 2022; Du et al., 2023; Guo et al., 2024; Liu et al., 2022] between real and synthetic data, which can be categorized as Optimization- Oriented methods. While these methods have shown success, their reliance on the bi-level optimization or nested gradients often results in prohibitively high computational costs [Zhang et al., 2023; Sajedi et al., 2023; Liu et al., 2021; Zhang et al., 2024a], limiting their practical application in wider scenarios.

To address the scalability challenges in DC, Distribution Matching (DM) [Zhao and Bilen, 2023] has been proposed. It focuses on aligning the latent representations extracted by randomly-initialized encoders, based on the rationale that the condensed set should represents the real training set in the feature space. Unlike previous OptimizationOriented methods, DM avoids the computationally intensive nested optimization loops, significantly reducing the time required for condensation and thereby enhancing its applicability in diverse scenarios [Loo et al., 2022; Zhou et al., 2023; Cazenavette et al., 2023; Liu et al., 2023a; Nguyen et al., 2021]. However, despite these advantages, DM's performance still falls short of SOTA optimization-oriented methods such as MTT [Cazenavette et al., 2022], IDC [Kim et al., 2022], and DREAM [Liu et al., 2023b].

In this paper, we conduct an in-depth analysis of DM from the inner-class and inter-class views, pointing out the limitations of current DM-based methods and provide our corresponding remedies. Specifically, from the inner-class view, to ensure an alignment during the whole training process, previous works like IDM [Zhao et al., 2023] and CAFE [Wang et al., 2022] naively use the models trained from scratch to extract the latent representations. While effective, the Persistent Training, i.e. numerous model updating steps, is very timeconsuming thus greatly hinders their efficiency. To counter this, we introduce Pseudo Long-Term Distribution Alignment (PLTDA), where we use the convex combination of initialized and trained expert models to perform inner-class distribution alignment, eliminating the need for persistent training. From the inter-class view, we reveal the Distribution Shift phenomenon in DM, i.e., the synthetic data will diverge from the real class region during condensation, which may change the decision boundary of the learned classifier. To address this, we employ expert models for Distribution Calibration, ensuring the synthetic data remains within the real class region. We term the proposed method as Dual-view distribution AlignmeNt for dataset CondEnsation (DANCE), for we enhance DM by utilizing the knowledge of expert models from the above two views, which is illustrated in Fig. 1. As will be shown in the experiments, DANCE can achieve comparable results to SOTA optimization-oriented methods even with only a single expert model.

Our main contributions are outlined as follows:

• [C1]: We identify and analyze the limitations of current DM-based dataset condensation methods from inner- and inter-class views, which reveals two major issues: persistent training and distribution shift.
• [C2]: We introduce DANCE by incorporating two modules to effectively mitigate the above two issues inherent in DMbased methods.
• [C3]: We conduct extensive experiments across a variety of datasets under different resolutions. The results demonstrate that DANCE establishes a strong baseline in dataset condensation, significantly advancing both performance and

efficiency, particularly in the realm of distribution matching.

2 Preliminaries

In this section, we initially formalize the concept of dataset condensation (DC) and then recap the DM method [Zhao and Bilen, 2023], which is pivotal as it represents the pioneering work in the realm of distribution matching within DC and lays the groundwork for our research.

ProblemDefinition. Given a large real training set Dreal = {(xreal i , yreal i )}|Dreal| i=1 , Dataset Condensation or Dataset Distillation aims to generate a small training set Dsyn = {(xsyn j , ysyn j )}|Dsyn| j=1 (|Dsyn| ≪ |Dreal|), so that the model trained on Dsyn and the model trained on Dreal (denoted as θsyn and θreal respectively) will have similar performance on the unseen data. Formally, let PD represent the distribution of the real data, ℓ be the loss operation such as cross-entropy, the synthetic training set can be obtained by minimizing the performance gap between the two models:

Distribution Matching. To solve Eq. (1), previous optimization-oriented methods have attempted to 1) update the Dsyn using a meta-learning framework. 2) match the gradient or parameter induced by Dsyn and Dreal. However, both the above methods involves a bi-level optimization, which is computationally inefficient due to the calculation of nested gradients. To improve the condensing efficiency, DM [Zhao and Bilen, 2023] first proposed Distribution Matching, which learns the condensed set by aligning the feature distributions of Dsyn and Dreal. Specifically, the condensed set in DM is optimized by:

where ϕ0 ∼ Pϕ0 denotes the randomly-initialized feature extractor (instanced by a random DNN θ0 without the linear classification layer). Compared to optimization-oriented methods, DM significantly enhances the computational efficiency and has shown better generalization ability across different architectures [Zhao and Bilen, 2023].

3 Methodology

While DM [Zhao and Bilen, 2023] brings remarkable efficiency and cross-architecture performance, the quality of the condensed set it generates typically falls short of those produced by SOTA optimization-oriented methods like IDC [Kim et al., 2022] and MTT [Cazenavette et al., 2022]. In this paper, we aim to enhance the alignment between the distributions of Dreal and Dsyn, considering both the innerclass view and the inter-class view. Sections 3.1 and 3.2 will detail the limitations of current DM-based methods from these two perspectives and introduce our proposed solutions. Subsequently, we describe our overall training algorithm in Section 3.3. Our method, termed Dual-view distribution AligNment for dataset CondEnsation (DANCE), is depicted in Fig. 2.

3.1 Inner-Class View

Limitation of DM. For data from the same class, DM [Zhao and Bilen, 2023] employs various randomlyinitialized deep encoders to extract latent representations of Dreal and Dsyn. It then minimizes the discrepancy of feature distributions to ensure they are aligned (Eq. (2)). However, we contend that aligning feature distributions from randomly initialized extractors, which are sampled from a probability distribution over parameters Pϕ0 , does not ensure that Dsyn remains a reliable proxy for Dreal throughout all training stages. This divergence may ultimately cause the model trained on Dsyn to deviate from the one trained on Dreal. To illustrate this divergence throughout the entire training process, we compute the discrepancy between Dsyn and Dreal at various stages of training. As depicted in Fig. 3a, DM fails to maintain the informativeness of the condensed set throughout the training procedure. During training, the distribution of the data condensed by DM increasingly deviates from that of the real data.

Remark. The misalignment issue is also noted in CAFE [Wang et al., 2022] and IDM [Zhao et al., 2023]. As a remedy, these approaches involve training multiple models from scratch to extract features of Dsyn and Dreal during the condensation process. While such Persistent Training yields improved performance, it is hampered by the following two drawbacks:

Drawbacks. 1) Hyper-parameter tuning: The models used for condensation need to be persistently trained during the condensing process in CAFE and IDM to enrich the distilled knowledge. This process, however, involves meticulous tuning of multiple parameters, including the number of training steps, iteration count, and learning rate during the modelupdating phase. Moreover, as model performance often sees a significant rise at the onset of training, the models at early stages are prone to be skipped due to overdoing the update steps, thereby hampering the effectiveness of the condensed images. 2) Inefficiency: Both CAFE and IDM necessitate optimizing hundreds of random models each time a dataset is condensed, which is impractical, especially for datasets with larger resolution.

Pseudo Long-Term Distribution Alignment (PLTDA). To tackle the misalignment issue from the inner-class perspective, we introduce a straightforward and effective module, termed Pseudo Long-Term Distribution Alignment (PLTDA). Specifically, rather than relying on models trained on real data, we employ a convex combination of randomlyinitialized encoders and their corresponding trained counterparts. We refer to these trained encoders as "expert encoders" (ϕexpert), because their corresponding "expert models" (θexpert) represent the upper bound of the performance of Dsyn and the end of the training process. We term this combination as "mid encoders" (ϕmid) of "mid models" (θmid), which is calculated by:

where λ ∼ U(0, 1) is a randomly generated coefficient for encoder combination. After obtaining the mid encoder, we calculate the loss in PLTDA as:

Compared to CAFE and IDM, which update the model during the condensation process, our expert encoders do not introduce additional hyper-parameters. Furthermore, as illustrated in Fig. 3b, the performance of the mid models changes more gradually across different values of λ, allowing for the generation of models at various training stages. This approach is advantageous because the standard model training often neglects early-stage models, a phenomenon evident in Fig. 3a (the model performance reaches at a high-level after a few epochs). Additionally, the mid encoders rely solely Algorithm 1 Dual-View Distribution Alignment for Dataset Condensation

Input: Real training set Dreal

Parameter: Number of expert models N; Number of condensation iterations I; Learning rate of the condensed set η; Calibration interval Ic

Output: The condensed set Dsyn

• 1: Initialize Dsyn with randomly selected real data
• 2: Pre-train N expert encoders {ϕnexpert }Nn=1 and save their corresponding initial encoders {ϕn0 }Nn=1
• 3: for i = 1, 2, . . . , I do
• 4: Randomly select an expert encoder ϕnexpert and generate the mid encoder ϕnmid by Eq. (3)
• 5: Calculate the matching loss LPLTDA by Eq. (4)
• 6: Update the Dsyn by Dsyn = Dsyn - η∇Dsyn LPLTDA
• 7: if i%Ic = 0 then
• 8: Calculate the calibration loss Lcalib by Eq. (5)
• 9: Update the Dsyn by Dsyn = Dsyn - η∇Dsyn Lcalib
• 10: end if

• 11: end for
• 12: Return: Dsyn

3.3 Training Algorithm

The pseudo-code of DANCE is detailed in Algorithm 1. Besides incorporating the PLTDA (Sec. 3.1) and Distribution Calibration (Sec. 3.2), our approach also integrates a prevalent data augmentation technique known as "Factoring & Upsampling". In this technique, each image space in Dsyn is divided into l × l smaller sections in order to host multiple synthetic images. These mini-images are subsequently up-sampled to their original dimensions during model training. This augmentation strategy was initially introduced by IDC [Kim et al., 2022] and has since been widely employed in various dataset condensation works [Liu et al., 2023b; Zhao et al., 2023].

4 Experiments

4.1 Experimental Setup

Datasets. We assess our method using three low-resolution datasets: Fashion-MNIST [Xiao et al., 2017] with a resolution of 28 × 28, and CIFAR-10/100 [Krizhevsky, 2009] with a resolution of 32 × 32. For medium-resolution data,

on the randomly-initialized encoder ϕ0 and the expert encoders ϕexpert. Both can be pre-trained offline and reused for different values of images per class (IPCs). Notably, our method does not require a large number of pre-trained expert models like IDC [Kim et al., 2022] or MTT [Cazenavette et al., 2022]. As we will demonstrate in Sec. 4.3, our approach achieves state-of-the-art results across various scenarios with just a single expert model.

3.2 Inter-Class View

Limitation of DM. As depicted in [Zhao and Bilen, 2023], DM aligns the distribution between Dsyn and Dreal in a classwise manner, while overlooking inter-class constraints. Since many existing DM-based methods [Sajedi et al., 2023; Zhang et al., 2024a] follow the class-wise learning manner of DM, these approaches may have the following drawback:

Drawback. Distribution Shift: The synthetic data may fall outside the real class region during condensing, which will affect the decision boundary of the learned classifier. As shown in Fig.3c, the expert model trained by real training data has an excellent performance on real test data, while it cannot achieve a high classification accuracy on the synthetic data generated by DM. It can be inferred from this phenomenon that, many of the synthetic examples are not within the real class region and even cross the decision boundary of the expert model.

Distribution Calibration. To address the above issue from the inter-class view, we integrate a module called Distribution Calibration into our approach. This module utilizes expert models θexpert to calibrate the inter-class distribution of Dsyn after PLTDA. Specifically, once the inner-class matching is completed, we impose the following calibration loss, computed using the chosen θexpert, to prevent the synthetic data from straying from their respective categories:

It is important to note that, although "Discrimination Loss" in CAFE [Wang et al., 2022] and "Distribution Regularization" in IDM [Zhao et al., 2023] employ a similar concept, they utilize un-converged models for calculating their losses, instead of expert models. This may lead to sub-optimal outcomes due to the relatively poorer generalization ability of such models compared to expert models.

we utilize the resized TinyImageNET [Le and Yang, 2015], which has a resolution of 64 × 64. Furthermore, in alignment with MTT [Cazenavette et al., 2022], we employ various subsets of the high-resolution ImageNet-1K [Deng et al., 2009] dataset (resolution 128 × 128) in our experiments. These subsets include ImageNette, ImageWoof, ImageFruit, ImageWeow, ImageSquawk, and ImageYellow. Additional details about the datasets are provided in the Appendix.

Network Architectures. Following previous studies [Cazenavette et al., 2022], we implement the condensation process using a ConvNet [Sagun et al., 2018]. The ConvNet we employ consists of three identical convolutional blocks, each featuring a 128-kernel 3×3 convolutional layer, instance normalization, ReLU activation, and 3 × 3 average pooling with a stride of 2. For low-resolution datasets, we use a three-layer ConvNet, while a four-layer ConvNet is utilized for TinyImageNet. To accommodate the higher resolutions of the high-resolution ImageNet-1K subsets, we employ a five-layer ConvNet.

Evaluation Metric. We utilize the test accuracy of networks trained on the condensed set Dsyn as our primary evaluation metric. Each network is trained from scratch multiple times: 10 times for low-resolution datasets and TinyImageNet, and 3 times for the ImageNet-1K subsets. We report both the average accuracy and the standard deviation. To assess training efficiency, we consider run time per step and peak GPU memory usage as criteria, where the run time is calculated as an average over 1000 iterations.

Implementation Details. For training, we employ an SGD optimizer with a learning rate of 0.01, momentum of 0.9, and weight decay of 0.0005. The expert models θexpert are trained for 60 epochs on low-resolution datasets and TinyImageNet, and for 80 epochs on ImageNet-1K subsets. We consistently use 5 expert models for all datasets as the default setting. The number of iterations for Distribution Calibration is fixed at 1 across all datasets. During the condensing process, the SGD optimizer is set with a learning rate of 0.1 for ImageNet-1K subsets and 0.01 for other datasets, with the learning rate being scaled by the number of images per class (IPC). Following IDC [Kim et al., 2022], we train the networks using a sequence of color transformation, cropping, and CutMix [Yun et al., 2019]. The factor parameter l is set to 2 for lowresolution datasets and Tiny-ImageNet, and 3 for ImageNet1K subsets. All synthetic data are initially generated from randomly selected real data to expedite optimization. The experiments are conducted on a GPU group comprising GTX 3090, RTX-2080, and NVIDIA-A100 GPUs.

4.2 Comparison to State-of-The-Art Methods

Baselines. We include a comprehensive range of methods as baselines in our study. For coreset selection methods, our choose Random Selection, Herding [Welling, 2009], and K-Center [Farahani and Hekmatfar, 2009]. In the category of Optimization-Oriented methods, we consider DC [Zhao and Bilen, 2021b], DSA [Zhao and Bilen, 2021a], IDC [Kim et al., 2022], DREAM [Liu et al., 2023b], and MTT [Cazenavette et al., 2022]. Additionally, for Distribution-Matching-based methods, our baselines include CAFE and CAFE+DSA [Wang et al., 2022], DM [Zhao and Bilen, 2023], IDM [Zhao et al., 2023], and DataDAM [Sajedi et al., 2023]. Further details about these baseline methods are provided in the Appendix, due to page constraints.

Performance Comparison. Tab. 1 and Tab. 2 present the comparison of our method with coreset selection methods and dataset condensation/distillation methods. The proposed method, DANCE, demonstrates remarkable performance across various datasets and resolutions. On low-resolution datasets such as Fashion-MNIST, CIFAR-10, CIFAR-100, and the medium-resolution dataset TinyImageNet, DANCE consistently outperforms or rivals leading methods in different IPC (images per class) settings. For instance, on FashionMNIST, it achieves the highest test accuracy of 81.5% with a single IPC. On CIFAR-10 and CIFAR-100, DANCE sets new benchmarks with 70.8% and 52.8% accuracy respectively at 50 IPC, even surpassing the SOTA optimizationoriented methods DREAM and IDC. Particularly notable is its performance on TinyImageNet, where it attains an accuracy of 11.6% at 1 IPC and 26.4% at 10 IPC, significantly ahead of the next best method, IDM. For the highresolution ImageNet-1K subsets, DANCE still yield SOTA results across various scenarios. Remarkably, across all ImageNet-1K subsets with 10 images per class, our DANCE brings over 10% accuracy increase compared to the second best results, showcasing its efficacy in handling diverse image complexities. These results show the superiority of DANCE in dataset condensation tasks, especially considering the wide margin by which it leads in many categories. Overall, DANCE not only establishes new standards in dataset condensation but also demonstrates its robustness across varying resolutions and dataset complexities.

Cross-Architecture Evaluation. We also evaluated the performance of our condensed set across various architectures, as detailed in Tab. 3. The results demonstrate that DANCE excels not only on the architecture employed during the condensation process but also exhibits impressive generalization capabilities on a range of unseen architectures

Training Efficiency Evaluation. In the context of dataset condensation, it is of great importance to consider the resource and time costs, as extensively discussed in previous studies [Sajedi et al., 2023; Zhang et al., 2023]. Some of the methods entail significantly higher time costs in comparison to the time required for training the entire dataset, rendering them less than optimal in balancing effectiveness and efficiency. Our evaluation encompasses both time and peak GPU memory costs incurred during the condensation process for various baseline methods and DANCE. As presented in Tab. 4, DANCE exhibits remarkable efficiency compared to optimization-oriented methods such as DC [Zhao and Bilen, 2021b], DSA [Zhao and Bilen, 2021a], and MTT [Cazenavette et al., 2022]. Much like DM [Zhao and Bilen, 2023], our method demonstrates scalability across different IPCs. However, IDM, being rooted in the DM-based approach, displays higher time costs when contrasted with both DM [Zhao and Bilen, 2023] and DANCE.

4.3 Ablation Studies

Effectiveness of Each Module. We evaluate three primal Modules of our method, namely Pseudo Long-Term Distribution Alignment (Sec. 3.1), Distribution Calibration (Sec. 3.2), and Factoring & Up-sampling technique (Sec. 3.3). As shown in Tab. 5, both the proposed PLTDA and Distribution Calibration bring significant improvement across various datasets. The most significant improvement is observed when all three modules are included. The results highlight the effectiveness of the three modules, demonstrating their collective importance in enhancing the DANCE framework's performance across different datasets.

Impact on the Number of Expert Models. The expert models θexpert are integral to both the PLTDA and Distribution Calibration modules within DANCE. To ascertain their impact, we investigated how the number of expert models (NEM) affects DANCE's performance. As Tab. 6 illustrates, there is a noticeable increase in DANCE's performance with the rise in NEM. Notably, even with just a single expert model, DANCE achieves competitive results, scoring 69.2% on CIFAR-10 with 10 images per class. This underscores DANCE's ability to efficiently leverage the pre-trained knowledge embedded in expert models.

4.4 Visualization Results

In Fig. 4, we present the synthetic images condensed by DANCE, showcasing distinct characteristics across different datasets. For the low-resolution dataset CIFAR-10, the condensed images are quite discernible, with each clearly representing its respective class. In contrast, the condensed images from the high-resolution ImageNet-1K subsets appear more abstract and outlined. Unlike the images produced by DM [Zhao and Bilen, 2023], which feature class-independent textures, our synthetic images encapsulate richer information pertinent to classification tasks. Additional visualizations are available in the Appendix due to page limitations.

5 Conclusion

In this study, we introduce a novel framework called Dual-view distribution Alignment for dataset Condensation (DANCE), which enhances the Distribution Matching (DM) method by focusing on both inner- and inter-class views. DANCE consists of two meticulously designed modules: Pseudo Long-Term Distribution Alignment (PLTDA) for inner-class view and Distribution Calibration for inter-class view. PLTDA ensures that the data condensed by DANCE effectively represents its class throughout the entire training process while eliminating the need for persistent training. In contrast, Distribution Calibration maintains the synthetic data within its respective class region. Extensive experimental results on various datasets show that DANCE consistently surpasses state-of-the-art methods while requiring less computational costs. This makes DANCE highly suitable for practical and complex scenarios.

Acknowledgments

This work was partially supported by grants from the Pioneer R&D Program of Zhejiang Province (2024C01024).