P-KDGAN: Progressive Knowledge Distillation with GANs for One-class Novelty Detection

1 Introduction

One-class novelty detection aims to identify patterns that do not belong to the normal data distribution [Chandola et al., 2009]. Unlike traditional classification problem, novelty detection is usually trained in an unsupervised setting where novelty data is absent. Novelty detection has a wide variety of applications such as network intrusion [Garc´ıaTeodoro et al., 2009], credit card fraud [Srivastava et al.,

∗This work was done when Zhiwei Zhang was a research intern at Multimedia Laboratory of SIAT.

†Corresponding author.

2008], medical diagnoses [Schlegl et al., 2017] and many more. With the advantage of deep learning, novelty detection based on generative adversarial networks (GANs) has shown state-of-the-art performance by learning the representative latent space of high-dimensional data [Schlegl et al., 2017; Zenati et al., 2018; Perera et al., 2019]. However, deep neural networks with high computational costs and large storage prohibit their deployment to computation and memory resource limited systems.

For tackling the above issue, neural network compression has been widely applied in recent years [Cheng et al., 2017]. As one of the mainstream compression methods, Knowledge Distillation (KD) following a teacher-student paradigm transfers knowledge from a teacher network with higher performance to a student network. The early contributions used the outputs of the softmax layers or intermediate layers in teacher networks to improve the performance of student networks [Hinton et al., 2015; Romero et al., 2015]. In the later researches, the discriminator losses were proposed to evaluate the distinction between the distribution spaces of teacher and student networks [Wang et al., 2018a; Wang et al., 2018b; Liu et al., 2018]. To our knowledge, there is no related works on two standard GANs [Goodfellow et al., 2014] including two generators and two discriminators to design distillation loss for knowledge distillation. Additionally, there are rare works investigating the initialization of student networks and always random initialization is used. Our experiments demonstrate that student networks without "knowledge" reserve (with random initialization) do not mimic the outputs of teacher networks well.

In this paper, we apply GANs in the encoder-decoderencoder structure [Akc¸ay et al., 2018] for one-class novelty detection, which outperforms the state-of-the-art approaches. In order to deploy the deep neural networks in computation resources limited mobile devices, we propose the Progressive Knowledge Distillation with GANs (P-KDGAN) method to train the lightweight student network. The P-KDGAN approach improves the performance of student GAN by solving the following three problems. 1) How to design a distillation loss to measure the similarity of intermediate representations learned from the teacher GAN and the student GAN? As is shown in the student GAN of Figure 1(a), the generator based on encoder-decoder-encoder pipeline can generate two latent vectors z1, z2 and a reconstructed image ˆx. The

(a) Knowledge Distillation with GANs

(b) The Progressive Learning of KDGAN

Figure 1: The flowchart of Knowledge Distillation with GANs for One-class Novelty Detection. (a) The Knowledge Distillation with Generative Adversarial Networks (KDGAN), in which the distillation losses Kl (Kl = w1K1 +wxKx+ w2K2) is designed for training the student GAN. (b) The two-step progressive learning of KDGAN is used to continuously improve the performance of the student GAN. KDGAN- 1©, KDGAN- 2©, KDGAN- 3© and KDGAN- 4© are four different distillation structures.

generator is trained by minimizing the weighted sum of Scon, Senc and Sadv, which have defined in Eq.2-4 [Akc¸ay et al., 2018]. Therefore, the distillation loss Kl described in Eq.6 is designed as the weighted sum of the losses K1, Kx and K2, where K1, K2 represent the difference between the two latent vectors (z1 and ˆ z1, z2 and ˆ z2) in the teacher GAN and student GAN, and Kx is the difference of two reconstructed images (ˆx, ˜ x). 2) How to combine the distillation loss Kl with existing generator losses LSg , LTg and discriminator losses LSd , LTd from the student and teacher GANs to improve the performance of the student GAN? As is illustrated in Figure 1(b), we design four distillation structures (KDGAN- 1©, KDGAN2©, KDGAN- 3© and KDGAN- 4©) based on different combinations of the above five losses. The difference between them consists of two aspects: on the one hand, whether the weights of the teacher GAN are fixed; on the other hand, whether the distillation loss Kl is combined with the losses LSg , LSd of the student GAN for knowledge transfer. 3) Whether the designed four distillation structures can make the performance of the student GAN like that of the teacher GAN? If not, how to fix it? Our experimental results demonstrate that the performance of student GANs trained from scratch (or with random initialization) by the above four distillation structures is incomparable with the teacher GAN. Just as learning is a gradual cumulative process, a two-step progressive learning of KDGAN is proposed to continuously improve the performance of the student GAN. In the first step, the student GAN imitates the representation of the pre-trained teacher GAN with fixed weights to make itself have a certain "knowledge" reserve. Such a "teaching by teacher" step make the student learn the basic knowledge totally from the teacher. In the second step, the student GAN with basic "knowledge" reserve is fine-trained together with the teacher GAN. The second step of "fine-learning with teacher" can further improve the performance and stability by jointly utilizing the basic knowledge of the teacher and student.

The performance of proposed progressive knowledge distillation with GANs for one-class novelty detection is evaluated on CIFAR-10 [Krizhevsky, 2009], MNIST [LeCun and Cortes, 2005] and FMNIST [Xiao et al., 2017] datasets. Our contributions are summarized as follows.

• We utilize the encoder-decoder-encoder based GAN for one-class novelty detection, which outperforms all the state-of-the-art methods.
• We propose new distillation losses on latent vectors and reconstructed images of GANs that allow the student to better learn from the teacher.
• We regard the distillation process as a knowledgeable teacher to improve the performance and stability of student networks through two-step progressive learning, which includes basic knowledge learning and finelearning.
• Progressive Knowledge Distillation with GANs is proposed for one-class novelty detection. Our experiments demonstrate that the P-KDAGN can improve the performance of the student GAN on the three datasets CIFAR10, MNIST and FMNIST by 2.44%, 1.77%, and 1.73%, respectively.

2 RelatedWork

We briefly review the related works in term of one-class novelty detection and knowledge distillation, as well as the architecture of Ganomaly [Akc¸ay et al., 2018].

2.1 One-class Novelty Detection

In the unsupervised one-class novelty detection, only the normal samples with one class are used for training the model. Conventionally, novelty detection methods can be divided into two categories [Chandola et al., 2009]. One is the traditional methods, such as One-Class SVM (OCSVM) [Sch¨olkopf et al., 2001], Kernel Density Estimation (KDE) [Parzen, 1962] and Principal Component Analysis (PCA) [Wold et al., 1987]. The disadvantage of such approaches is that they are not suitable for high-dimensional image data. The other methods based on deep learning include Deep Belief Networks (DBN) [Erfani et al., 2016], Autoencoders (AE) [Vincent et al., 2008] and generative adversarial networks (GANs) [Schlegl et al., 2017; Zenati et al., 2018; Perera et al., 2019].

GANs have shown state-of-the-art performance in modeling complex high-dimensional image distributions [Goodfellow et al., 2014]. Therefore, a lot of GANs based methods have been used for novelty detection [Schlegl et al., 2017; Zenati et al., 2018; Perera et al., 2019]. The reconstruction errors of images or latent vectors are utilized as novelty score, which means that the learned model only reconstructs normal samples well, and shows very low tolerance for novel samples. Schlegl et al. [Schlegl et al., 2017] proposed the first GANs based work, AnoGAN, for novelty detection. In training, the combination of the residual loss on images and discrimination loss on feature maps is minimized to iteratively search the best latent vector. The Efficient GAN [Zenati et al., 2018] based on BiGAN [Donahue et al., 2017] network was proposed for jointly training the map from the image to the latent space simultaneously. Perera et al. [Perera et al., 2019] proposed the OCGAN in which two discriminators were used in the latent space and the input space for making the learned network better model the input images. Recently, Ganomaly [Akc¸ay et al., 2018] shown in Figure 1(a) constructs a novel architecture for multi-class anomaly detection. In our method, the Ganomaly framework is used for one-class novelty detection.

2.2 Knowledge Distillation

To reduce the large computation and storage cost of deep convolutional neural networks, knowledge distillation can transfer the generalization ability of a large network (or an ensemble of networks) to a light-weight network. Hinton et al. [Hinton et al., 2015] used the outputs of the softmax layer of a teacher network as the target function to train the student network. Romero et al. [Romero et al., 2015] proposed that a student network with random initialization can imitate the intermediate representations of the teacher network to improve its own performance. In order to ensure the student network to learn the true data distribution from the teacher network, knowledge distillation with a discriminator was used for distinguishing features extracted from the teacher and student networks [Wang et al., 2018a; Wang et al., 2018b; Liu et al., 2018]. In our method, knowledge distillation is considered as a progressive learning process, which can continuously improve the performance of student networks.

GANs [Goodfellow et al., 2014] have been applied to many real world applications such as domain transfer, image generation, and novelty detection. However, to our knowledge, there is no related works that deploy the knowledge distillation on two standard GANs. Therefore, this paper designs a distillation loss to transfer knowledge from the teacher GAN to the student GAN.

2.3 The Architecture of Ganomaly

Akcay et al. [Akc¸ay et al., 2018] proposed Ganomaly for multi-class anomaly detection, in which multiple class of samples as normal data and one class of samples as abnormal data. In this paper, we utilize Ganomaly architecture for one-class novelty detection. One-class means that only the instances in one category are regarded as normal data, and the remaining categories are abnormal data.

GAN consists of two adversial modules, a generator G and a discriminator D. As is shown in the student GAN of Figure 1(a), the Ganomaly [Akc¸ay et al., 2018] framework is composed of two modules: 1) an encoder-decoder-encoder (GE -GD -GR) pipeline based generator G that learns the distribution of input image x, where x ∈ Rw×h×c , from latent spaces z1, z2, where z1, z2 ∈ Rd ; 2) a discriminator D that decides whether the reconstructed image ˆx is real or fake. D and G are simultaneously optimized by playing the following minmax game as:

where training dataset X comprises N normal images, X = [x1, x2, ..., xN] ∈ RN×w×h×c , and Ex∼X is the expected value of x obeying distribution of normal images X.

During training, the generator loss LSg and the crossentropy loss Sce are minimized to train the student G and student D, respectively. The model is trained from normal samples, therefore the reconstruction error is large on abnormal samples. In the previous methods [Schlegl et al., 2017; Zenati et al., 2018; Perera et al., 2019], the reconstruction errors of the images or latent vectors are used for anomaly detection. In Ganomaly [Akc¸ay et al., 2018], the difference Senc between two latent vectors z1, z2 is used as novelty score, which is defined in Eq. 3.

3 Our Method

In this work, we adopt the Ganomaly [Akc¸ay et al., 2018] framework for one-class novelty detection and achieve stateof-the-art performance. To compress deep neural networks and deploy them to embedded devices with limited resources, we propose the Progressive Knowledge Distillation with GANs (P-KDGAN) to learn a lightweight student GAN from a pre-trained teacher GAN. The P-KDGAN method is composed of three modules. 1) Knowledge distillation with GANs (KDGAN), in which the distillation losses based on Ganomaly framework are proposed for transferring knowledge from the teacher GAN to the student GAN. 2) Four distillation structures are designed for KDGAN. 3) The two-step progressive learning of KDGAN can continuously improve the performance of the student GAN.

3.1 KDGAN

In the KDGAN, both the teacher GAN and the student GAN follow the network architecture of Ganomaly [Akc¸ay et al., 2018] and have the same network layers. The difference between them is the number of channels in each layer. Therefore, the generator loss LTg and the discriminator loss LTd in the teacher GAN have the same form as LSg and LSd . The generator loss LSg of student GAN includes reconstructed image loss Scon, latent space loss Senc and adversarial loss Sadv:

where f(·) outputs the intermediate representations of discriminator D. Scon, Senc and Sadv denote the reconstruction errors of the images, latent vectors and feature maps, respectively. The weighted sum of Scon, Senc and Sadv constitutes the generator loss LSg and is minimized to train the student generator G. The discriminator loss LTd consists of Sce.

The designed distillation loss Kl is a novel attempt for knowledge distillation on two standard GANs. As is shown in Figure 1(a), the teacher GAN and the student GAN transfer knowledge through the intermediate layers of the generators, which includes two latent vectors and one reconstructed images. In the KDGAN, we design three losses K1, Kx and K2 to measure the similarity of the intermediate layers. K1 and K2 are the L2 distance of latent vectors (z1 and ˆ z1, z2 and ˆ z2) from the teacher GAN and student GAN. Kx is the L1 distance of reconstructed images (ˆx, ˜ x). Based on the above three losses, we propose distillation loss Kl as an objective function for knowledge distillation which is the weighted sum of K1, Kx and K2:

3.2 Distillation Structures

As is shown in Figure 1(a), the designed distillation loss Kl builds a "bridge" between the teacher and student GANs for knowledge transfer. The losses in the KDGAN consist of three parts: teacher GAN losses LTg , LTd , student GAN losses LSg , LSd , and distillation loss Kl. We define the above five loss functions as the elements of set L:

where α, β, µ, ν, and λ ∈ {0, 1} indicates whether the corresponding loss is used to train the networks.

The elements in L can be combined into four subsets (L1, L2, L3, L4) to form different distillation structures according to the following two rules. The first rule is whether the teacher GAN has fixed weights; the second rule is whether the distillation loss Kl is combined with the losses LSg , LSd to train the student GAN. Before the KDGAN, a teacher GAN

Algorithm 1 Progressive Knowledge Distillation with GANs

Input: Pre-trained teacher G and D, training dataset with normal instances (xi, yi)Ni=1 , epochM

Output: Improved student G

• 1: First step
• 2: Student G and D with random initialization
• 3: for m=1 toM do
• 4: for i=1 to N do
• 5: Update student GANwhen teacher GANwith fixed weights
• 6: end for
• 7: end for
• 8: Second step
• 9: Download the weights of the teacher and student GANs from the previous step
• 10: for m=1 toM do
• 11: for i=1 to N do
• 12: Update student GAN and teacher GAN together
• 13: end for
• 14: end for

is trained by its own generator loss LTg and discriminator loss LTd . The designed four distillation structures are introduced as follows:

• KDGAN- 1©: L1={Kl}. Without the use of real labels, the training of student network only depends on the distillation loss Kl, which results in poor detection performance. There is no adversarial networks, and the teacher network is not updated, so its training speed is the fastest.
• KDGAN- 2©: L2={LSg ,LSd ,Kl}. The student GAN is trained by minimizing its own losses LSg , LSd and distillation loss Kl, while the teacher GAN is not updated. The adversarial network in student GAN causes its training speed to be slightly slower than KDGAN- 1©.
• KDGAN- 3©: L2={LTg ,LTd ,Kl}. The teacher GAN uses its own losses LTg , LTd to train to maintain its performance, when the training of the student GAN follows KDGAN- 1©. Its training speed is almost the same as that of KDGAN- 2©.
• KDGAN- 4©: L2={LTg ,LTd ,LSg ,LSd ,Kl}. The trainings of the teacher and student GANs follow KDGAN3© and KDGAN- 2©, respectively. There are two adversarial networks that need to be trained simultaneously, so the training speed is the slowest.

3.3 Progressive Learning of KDGAN

The progressive learning of KDGAN, shown in Figure 1(b), is a two-step approach that continuously improves the performance of the student GAN and achieves better performance than the single step methods. The two-step P-KDGAN is described as follows.

P-KDGAN-I. In the first step, four distillation structures are utilized to train student network. The experimental results shown in Section 4.4 demonstrate that the performance of student network with random initialization has a large gap compared with teacher network. Therefore, considering the detection accuracy and training time of the four distillation structures, KDGAN- 2© is used as the first step of P-KDGAN to enable student network to learn the basics knowledge from teacher network. In the KDGAN- 2©, the pre-trained teacher has already converged, so the teacher network with fixed weights is used to train the student network relying on real labels and distillation knowledge.

P-KDGAN-II. In the second step, KDGAN- 3© and KDGAN- 4© continue to train the teacher networks, while the student networks with basic knowledge rely on distilling knowledge to fine-training, thereby further improving accuracy and stability. The fine-learning processes in this step are named as P-KDGAN-II- 2© 3© and P-KDGAN-II- 2© 4©. The experimental results prove that the performance of student network even exceeds the teacher network in some categories of one-class novelty detection.

The above process is illustrated in Algorithm 1.

4 Experiments

In this section, the proposed P-KDGAN is evaluated on the well-known CIFAR-10 [Krizhevsky, 2009], MNIST [LeCun and Cortes, 2005] and FMNIST [Xiao et al., 2017] datasets. Following previous work [Perera et al., 2019], we quantify the performance of our method using the Area Under Curve (AUC) of Receiver Operating Characteristics (ROC). The performance results are analyzed in details and are compared with state-of-the-art techniques.

All the reported results are implemented using the PyTorch framework [Paszke et al., 2017] on NVIDIA TITAN 2080Ti. In the experiments, the batch size and epoch are set to 1 and 500 respectively. Adam [Kingma and Ba, 2015] is used for training with a learning rate of 0.002.

4.1 Datasets

For the three experimental datasets, the training and testing partitions remain as default. In the setup, one of the classes from training dataset is considered as normal samples for training. During testing, the remaining classes are used to represent novelty samples. For example, every experiment on the CIFAR-10 dataset is trained with 5000 samples and tested with 10,000 samples. The above experiment is repeated for all the ten categories. In addition, in order to compatible with the network architectures, all the images are resized to 32×32 by Bilinear interpolation.

4.2 Network Architectures

The Ganomaly [Akc¸ay et al., 2018] framework based on encoder-decoder-encoder (GE -GD -GR) pipeline is used in our method. GE, GR in the generator and discriminator D are encoders, GD is decoder. The encoder E(x) and decoder D(x) follow the DCGAN [Radford et al., 2016] architecture, which have three basic layers in our model. As is shown in Table 1, the basic layers consist of: convolutional layers (deconvolutional layers), batch normalization and activation. In contrast, LeakyReLU and ReLU activations are used in encoders and decoders, except for the last layer in decoder, which uses Tanh. All the convolution filters are set to 4 × 4.

The difference between a teacher network and a student network is the number of channels in the intermediate representations. For the three experimental datasets, the intermediate layers in the teacher networks are set to 64-128-256 channels following the OCGAN [Perera et al., 2019]. The student networks in each dataset utilize intermediate representations with 8-16-64 channels, 2-4-8 channels and 1-2-4 channels respectively. The encoder E(x) and decoder D(x) architecture of the teacher GAN is illustrated in Table 1.

4.3 Results on One-class Novelty Detection

In this section, we compare our Ganomaly [Akc¸ay et al., 2018] based Teacher GAN with several traditional and deep learning based methods on CIFAR-10 and MNIST datasets , including one-class SVM (OC-SVM) [Sch¨olkopf et al., 2001], kernel density estimation (KDE) [Parzen, 1962], deep variational autoencoder (VAE) [Kingma and Welling, 2014], AND [Abati et al., 2019], AnoGAN [Schlegl et al., 2017], DSVDD [Ruff et al., 2018] and OCGAN [Perera et al., 2019]. In light of massive experiments, the parameters of wcon, wenc and wadv in Eq. 5 are mannually configured as 10, 1 and 1. The parameters of w1, wx and w2 in Eq. 6 are set as 1. We take the average AUC of the last epoch from multiple trials, but not the manually selected result, as the detection performance, which is more convictive.

Comparisons on CIFAR-10 and MNIST. The performance of one-class novelty detection on CIFAR-10 dataset, our method shown in Table 2 achieves 73.76%, which is higher than the best OCGAN [Perera et al., 2019] method about 8%. For MNIST dataset, our method achieves 97.80% yielding an improvement of about 0.3% compared with stateof-the-art method.

4.4 Evaluation of P-KDGANMethod

In this section, the progressive knowledge distillation with GANs is evaluated on CIFAR-10, MNIST and FMNIST datasets. In each experiment, the weights of the last epoch are served as the teacher network.

KDGAN vs. P-KDGAN. As is shown in Table 3, PKDGAN-II- 2© 3© achieves the best performance on three datasets, which illustrates the effectiveness of our progressive learning of KDGAN. Although KDGAN- 3© achieves the second-best results on MNIST and FMNIST, it shows the worst performance on CIFAR-10 dataset. KDGAN- 4© obtain the second-best results on CIFAR-10, but it was about 0.5% lower than the best result. In addition, KDGAN-d (KDGAN4©) iillustrated in Figure 2(c) is inferior in accuracy and training stability compared to P-KDGAN-II. The training curves of the AUC illustrated in Figure 2 clearly shows that proposed P-KDGAN-II can improve the accuracy of the student network and even surpass the teacher network, and reduce shock. Therefore, the above analysis concludes that student networks with random initialization can only learn the basic knowledge of the teacher networks, and the fine-training in the second step of P-KDGAN can further improve performance.

Results on P-KDGAN. As is illustrated in Table 4, the performance of the student GAN obtained by two-step PKDGAN is only 0.71%, 0.55% and 0.18% lower than that of the teacher GAN when compressing the computation at ratios of 24.45:1, 311.11:1, and 700:1, respectively.

5 Conclusion

In this paper, we use the encoder-decoder-encoder pipeline based GANs for one-class novelty detection and achieve state-of-the-art performance. To compress the model, the progressive knowledge distillation with GANs is proposed, which is a novel exploration that applies the knowledge distillation on two standard GANs. The two-step progressive learning can continuously improve the performance and reduce shock of the student network, in which the designed distillation loss plays an important role. Experiments on three datasets validate the effectiveness of our proposed method. Moreover, our proposed method can be used to compress other GANs-based applications, such as image generation.

Acknowledgments

This work is supported by Key-Area Research and Development Program of Guangdong Province (2019B010155003), National Natural Science Foundation of China (U1713203), Shenzhen Science and Technology Innovation Commission (Project KQJSCX20180330170238897), and the Scientific Instrument Developing Project of the Chinese Academy of Sciences (Grant No. YJKYYQ20190028).