AutoReCon: Neural Architecture Search-based Reconstruction for Data-free Compression

1 Introduction

To be deployed on resources-constrained hardware for realtime applications, the efficiency of deep convolutional neural networks has been improved significantly by various model compression techniques [He et al., 2020; Howard et al., 2019; Zhu et al., 2020b; Mirzadeh et al., 2020]. Without altering the model architecture, quantized neural networks [Zhu et al., 2020b] use a low bit width representation instead of full-precision floating-point, saving expensive multiplications. Pruning [He et al., 2020] is an approach to remove the weights or neurons based on certain criteria. In terms of efficient neural network architectures, the MobileNet [Howard et al., 2019], ShuffleNet, and ESPNet [Mehta et al., 2019] series make use of depthwise-separable convolution, grouped convolution with shuffle operation, and efficient spatial pyramid. The knowledge distillation paradigm [Mirzadeh et al., 2020] transfers the information from a pre-trained teacher network to a portable student network.

Data-free compression [Chen et al., 2019; Cai et al., 2020] has been an active research area when the original training dataset for the given pre-trained model is unavailable because of privacy or storage concerns. Given the pre-trained model to be compressed, it is an essential step to reconstruct the original training dataset by inverting representation. For example, accuracy degradation of ultra-low precision quantized models [Banner et al., 2018; Xu et al., 2020; Nagel et al., 2019] is unacceptable without fine-tuning on the reconstructed training dataset. The reconstruction method computes a reconstructed training dataset by leveraging some extra metadata [Lopes et al., 2017] or by extracting some prior information [Choi et al., 2020] from the pre-trained model. Instead of computing the reconstructed training dataset directly [Nayak et al., 2019; Cai et al., 2020; Lopes et al., 2017], recent reconstruction methods [Fang et al., 2019; Yoo et al., 2019; Micaelli and Storkey, 2019; Xu et al., 2020; Choi et al., 2020; Chen et al., 2019] employ a generator to generate a reconstructed training dataset in an end-to-end manner and show better performance for data-free compression.

The quality of the reconstruction closely relates to the extracted information from the pre-trained model. When more information is exploited from the pre-trained model, datafree compression achieves better performance. Thus, the current reconstruction methods [Micaelli and Storkey, 2019; Xu et al., 2020; Nayak et al., 2019; Chen et al., 2019; Choi et al., 2020; Yoo et al., 2019] focus on exploiting as much prior information as possible from the pre-trained model. However, how the network engineering will contribute to the reconstruction method remains unknown. Thus, we consider network engineering of the reconstruction method for the first time in the literature. This work aims to seek an optimized generator architecture, with which data-free compression shows performance improvement. It is worth mentioning that network engineering of the reconstruction and exploiting more prior information from the pre-trained model are complementary rather than contradictory. Both are important and should be explored for improving data-free compression. The contribution of this paper is summarized as follows.

• To our best knowledge, we are the first work to consider network engineering of the reconstruction method.
• We propose the AutoReCon method, which is a neural

• architecture search-based reconstruction method to optimize generator architecture for reconstruction.
• Using the discovered generator, diverse experiments are conducted to demonstrate the effectiveness of the AutoReCon method for data-free compression.

2 RelatedWork

2.1 Neural Architecture Search

Neural architecture search has attracted a lot of attention since it can automatically search for an optimized architecture for a certain task and achieve remarkable performance [Pham et al., 2018; Liu et al., 2018; Gao et al., 2020; Zhu et al., 2020a]. The optimization algorithms of neural architecture search include reinforcement learning [Pham et al., 2018], evolutionary algorithm, random search [Chen et al., 2018], and gradient-based algorithm [Liu et al., 2018]. There is a lot of work towards reducing the computational resources required by searching, including weight sharing [Pham et al., 2018], progressive search, one-short mechanism [Liu et al., 2018], and using a proxy task. The performance of the discovered architecture by neural architecture search has surpassed human-designed architecture in many computer vision tasks, including classification [Liu et al., 2018] and image generation [Gao et al., 2020].

2.2 Data-free Model Compression

Data-free compression covers data-free quantization and data-free knowledge distillation. Without a generator, the reconstructed training dataset is computed directly in [Lopes et al., 2017; Nayak et al., 2019; Cai et al., 2020; Nagel et al., 2019; Yin et al., 2020]. [Lopes et al., 2017] present a method for data-free knowledge distillation, where the reconstructed training dataset is computed based on some extra recorded activations statistics from the pre-trained model. DeepInversion [Yin et al., 2020] introduces a feature map regularizer based on batch normalization information in the pre-trained model for data-free knowledge distillation. In data-free knowledge distillation [Nayak et al., 2019], the class similarities are computed from the pre-trained model and the output space is modeled via Dirichlet Sampling. [Cai et al., 2020] calculates the reconstructed training dataset to match the statistics of the batch normalization layers of the pre-trained model and introduces the Pareto frontier to enable mixed-precision quantization. [Nagel et al., 2019] improves data-free quantization by equalizing the weight ranges and correcting the biased quantization error.

The performance of data-free compression can be improved by employing a generator for the reconstruction [Fang et al., 2019; Yoo et al., 2019; Micaelli and Storkey, 2019; Xu et al., 2020; Choi et al., 2020; Chen et al., 2019]. [Chen et al., 2019] proposes a framework for data-free knowledge distillation by exploiting generative adversarial networks, where the reconstructed training dataset derivated from the generator is expected to lead to maximum response on the discriminator of the pre-trained model. The KEGNET [Yoo et al., 2019] framework uses the generator and decoder networks to estimate the conditional distribution of the original training dataset for data-free knowledge distillation. In datafree knowledge distillation [Micaelli and Storkey, 2019], an adversarial generator is used to produce and search for the reconstructed training dataset on which the student poorly match the teacher. In this paper, we improve on the work of [Xu et al., 2020], which proposes a knowledge matching generator to produce a reconstructed training dataset by exploiting classification boundary knowledge and distribution information from the pre-trained model.

3 AutoReConMethod for Data-free Compression

In this section, we define the reconstruction method for datafree compression. Then, we introduce our proposed AutoReCon method, a neural architecture search-based reconstruction method, and present its search space and search algorithm. Also, the training process of the AutoReCon method for data-free compression is described.

3.1 Definition of ReconstructionMethod

The pre-trained model Mp is obtained by training on the original training dataset To = {xo, yo}. Given the pretrained model Mp, we compute the reconstructed training dataset Tr = {xr, yr} with the reconstruction method Φ, i.e., Tr = Φ(Mp).

Considering the reconstruction method with a generator as shown in Figure 1a), the pre-trained model Mp is fixed while the weights of the generator are updated by minimizing the reconstruction loss Lr. The prior information extracted from the pre-trained model Mp by the current methods is mainly the class boundary information and distribution information. If more prior information can be extracted from the pre-trained model, the reconstruction method can be easily adjusted by incorporating more loss terms to the reconstruction loss. Current reconstruction method Φ can be expressed as follows.

where z and Wg are the random noise input vector and weights of the generator, and Lclass(·, ·) is the cross-entropy loss function. Lbns(·, ·) measures the distribution distance between the batch normalization statistics of the original training dataset BNo and the batch normalization statistics of the reconstructed training dataset BNr. The formulations of Lclass, Lbns, and Lr are flexible to make the AutoReCon method general.

3.2 AutoReConMethod

As shown in Figure 1a) and c), we present an overview of current reconstruction and the AutoReCon method. The current reconstruction method includes a pre-trained model Mp and a generator Mg with a human-designed architecture. In the AutoReCon method, we aim to search for a superior generator architecture automatically for reconstruction.

Regarding the reconstruction task, our training objective function is written as follows, where both weights Wg and architecture Ag of the generator can be updated by minimizing the reconstruction loss.

where Ltrain r and Lval r refer to the reconstruction loss function on the reconstructed training dataset and the reconstructed validation dataset, respectively. W ∗ g (Ag) are the optimal weights of the generator given the generator architecture Ag. Ag ∈ S and S is the whole search space of the generator.

The Search Space

We construct a layer-wise search space with a fixed macroarchitecture for the generator. The macro-architecture defines the type of the edge, the number of edges, the node connection, and the input/output dimension of each node. The macro-architecture is shown in Figure 2, where there are three convolutional blocks and five nodes in every convolutional block. We denote the generator as Mg(e1, ..., ei, ..., eE), where ei represents the ith edge and E is the number of edges. The nodes refer to the feature maps and we calculate them as the summation of the outputs of their previous connected edges. There are three types of edges: normaledge, up-edge, and cross-edge. Normal-edge connects two nodes with the same dimension. Up-edge is used to increase the spatial resolution. Normal-edge and Up-edge are within a convolutional block. Cross-edge connects two adjacent convolutional blocks.

To construct a layer-wise search space for the generator, we set each edge as a mixture of candidate operations, which has several parallel operations instead of one specific operation. Thus, the over-parameterized generator is expressed as Mg(e1 = C1, ..., ei = Ci, ..., eE = CE) and Ci is the mixture of candidate operations for the edge ei. As shown in Table 1, different types of edges use different mixtures of candidate operations. Taking the edge Ci as an example, we compute its output by summing the outputs of the mixture of candidate operations as follows.

where Xi in and Xiout are the input and output of the ith edge. Oij denotes the j th candidate operation of the ith edge and j = 1, ..., F. F is the number of candidate operations for an edge.

The Search Algorithm

The search algorithm represents the search space as a stochastic super net Ms. In the stochastic super net Ms, Oij is associated with an architecture parameter αij . To derive a generator Ag from the stochastic super net Ms, the candidate operation Oij is sampled with the probability pij , which is computed as follows.

Since sampling from the mixture of candidate operations for each edge is independent, the probability of sampling a generator architecture Ag can be described as follows.

Algorithm 1 The AutoReConmethod for data-free compression

Input: Pre-trained model Mp.

Output: Discovered generator Mg, compressed model Mc. Stage 1: Searching for generator architecture.

1: for epoch = 1 to L1 do

• 2: for batch = 1 to T1 do
• 3: Obtain random noise z ∼ N(0, 1) and label yo. Generate reconstructed training dataset Tr with stochastic super net Ms.

Update weights of stochastic super net by minimizing reconstruction loss Lr.

• 4: end for
• 5: for batch = 1 to V1 do
• 6: Obtain random noise z ∼ N(0, 1) and label yo. Generate reconstructed validation dataset Vr with stochastic super net Ms. Update architecture parameters of stochastic super net by minimizing reconstruction loss Lr.
• 7: end for
• 8: end for

Stage 2: Compression with discovered generator.

• 9: for epoch = 1 to L2 do
• 10: for batch = 1 to T2 do
• 11: Obtain random noise z ∼ N(0, 1) and label yo. Generate reconstructed training dataset Tr with the discovered generator Mg .

Update weights of compressed model Mc by minimizing compression loss Lc.

• 12: end for
• 13: end for

method. We adopt the GDFQ data-free compression method [Xu et al., 2020] as a baseline for the following three reasons. First, it exploits both class boundary information and distribution information from the pre-trained model Mp, compared to other methods that use only one type of information [Cai et al., 2020; Yoo et al., 2019; Chen et al., 2019; Nayak et al., 2019]. Second, it includes both data-free quantization and data-free knowledge distillation, where knowledge distillation is applied for the output layer (i.e., knowledge distillation is not applied for the intermediate layers). Third, it achieves state-of-the-art performance. We use the same experimental settings as the GDFQ method to observe the influence of the generator architecture. In the GDFQ method, the human-designed generator architecture for both CIFAR-100 and ImageNet classification follows ACGAN. Besides, the

In this case, we can approximate the problem of finding an optimized discrete generator architecture by finding optimized sampling probabilities. The training objective function of the AutoReCon method is re-written from Equation 2 as follows.

To make the reconstruction loss differentiable to the sampling probabilities, we compute continuous variables mij by the Gumbel Softmax function as an alternative as follows.

where gij is the noise sampled from the Gumbel distribution (0, 1) and τ is a temperature parameter to control the sampling operation. Then, the continuous variables mij are directly differentiable with respect to the sampling probabilities. Thus, the computation of the edge Ci in Equation 3 can be expressed as follows.

3.3 Training Process

Using the AutoReCon method, the training process for datafree compression is illustrated as shown in Algorithm 1. The first stage of the training process is to search for generator architecture with our AutoReCon method, as shown in Figure 1c). The goal of the first stage is to seek an optimized generator architecture from the stochastic super net. The second stage of the training process is to compress the pretrained model Mp with the discovered generator Mg. The compression loss Lc can be introduced from quantization and/or knowledge distillation. Compared with the current reconstruction methods, our AutoReCon method considers network engineering and search for an optimized generator architecture for reconstruction.

4 Experiments

4.1 Implementation Details

Our interest is to show the performance improvement of data-free compression, which is brought by the AutoReCon human-designed generator for ImageNet classification adopts the categorical conditional batch normalization layer to fuse label information following SN-GAN.

4.2 Results on Image Classification

Results on ImageNet Classification

As shown in Table 2, we report the experimental results of data-free compression on the ImageNet classification dataset. Replacing the human-designed generator with the generator discovered by our AutoReCon method, the accuracy of the GDFQ method increases consistently using different pre-trained models and low-bit width quantization. Using ResNet18 as the pre-trained model and 3-bit width quantization, the Top-1 accuracy of the GDFQ method can increase by 2.68% when using the generator discovered by the AutoReCon method. The Top-1 accuracy of the GDFQ method increases by 11.99% using MobileNetV2 as the pre-trained model, 3-bit width quantization, and the generator discovered by the AutoReCon method. Using ResNet50 as the pretrained model and 5-bit width quantization, the Top-1 accuracy of our data-free compression with an optimized generator surpasses the GDFQ method by 8.43%. In addition, the optimized generator needs almost the same parameters and fewer flops compared with a human-designed generator.

Results on CIFAR-100 Classification

As shown in Table 2, we report the experimental results of data-free compression on the CIFAR-100 classification dataset. Using various pre-trained models and low-bit width quantization, our data-free compression with an optimized generator architecture achieves better accuracy than the GDFQ method with a human-designed generator. Using ResNet18 as the pre-trained model and 3-bit width quantization, the Top-1 accuracy of the GDFQ method will improve by 17.73% if the human-designed generator is replaced with the generator discovered by the AutoReCon method. Using MobileNetV2 and 5-bit width quantization, the Top-1 accuracy of our data-free compression shows an improvement of 4.51% compared with the GDFQ method. The Top-1 accuracy improvement becomes 9.66% using ResNet50 as the pre-trained model and 4-bit width quantization.

4.3 Ablation Study

Scalability of Discovered Generator Architectures

We explore the scalability of the discovered generator architecture for data-free compression on the CIFAR-100 classification dataset. We scale the base channels by a factor from s = 0.5 to s = 4 for the discovered generator and the humandesigned generator. The data-free compression results using MobileNetV2 as the pre-trained model, 5-bit width quantization, and knowledge distillation applied for the output layer are shown in Table 3. Without modifying the optimized generator architecture, the performance of our data-free compression keeps increasing and is always better than the GDFQ method when scaling the base channels by the factor from s = 0.5 to 4.0. The accuracy of the GDFQ method decreases when we scale the base channels for the human-designed generator. Thus, we conclude that our searched generator architecture has superior scalability compared to the humandesigned generator for data-free compression.

Generalization of AutoReCon Method

Except for the GDFQ method, we use the GFAD[Fang et al., 2019] method as a baseline to show the generalization of our AutoReCon method. The generation loss in the GFAD method is replaced with the reconstruction loss of Equation 6, which enables the exploration of generator architecture. We use ResNet34 as the pre-trained teacher model and ResNet18 as the student model. The experimental results of datafree knowledge distillation on CIFAR-100 is shown in Table 4. With a human-designed generator, the GFAD method achieves better accuracy than the DAFL[Chen et al., 2019] method. The Top-1 accuracy of our data-free knowledge distillation with a discovered generator is 2.28% better than the baseline of the GFAD method with a human-designed generator.

4.4 Comparison with State-of-the-art Methods

On the ImageNet classification dataset, we present the results of additional data-free compression methods as shown in Table 5. The comparison is mainly for data-free quantization except that the GDFQ and our methods apply knowledge distillation on the output layer. None of the compared methods apply knowledge distillation on the intermediate layers. The results of DFQ [Nagel et al., 2019] and ZeroQ [Cai et al., 2020] are cited from the GDFQ paper and have a rather low accuracy for ultra-low precision data-free quantization. The DFC [Haroush et al., 2020] method achieves a moderate accuracy with a combination of BN-Statistics and Inception schemes. Our method achieves better accuracy compared to the GDFQ method since the AutoReCon method discovers an optimized generator architecture for reconstruction.

5 Conclusion

In this paper, we present the AutoReCon method, which is the first work to consider network engineering of the reconstruction method to improve the performance of data-free compression. In particular, our AutoReCon method can search for an optimized generator architecture from a stochastic super net with gradient-based neural architecture search for reconstruction. When we plug our discovered generator to replace the human-designed generator, our data-free compression benefits from the optimization of the generator architecture and achieves better accuracy. Specifically, using ResNet50 as the pre-trained model and 5-bit width quantization, the Top-1 accuracy of our data-free compression on ImageNet with an optimized generator surpasses the GDFQ method by 8.43%. The Top-1 accuracy of the DFAD method on CIFAR-100 increases by 2.28% using ResNet34 as the pre-trained teacher model, ResNet18 as the student model, and the generator discovered by the AutoReCon method.