Improving representation learning in autoencoders via multidimensional interpolation and dual regularizations

1 Introduction

Among unsupervised learning frameworks of representation learning, autoencoders (AEs) and variants [Vincent et al., 2010; Kingma and Welling, 2014; Srivastava et al., 2014; Makhzani et al., 2016] have shown a remarkable ability to encode compressed latent representations by reconstructing datapoints. In particular, by decoding the interpolated latent representation of two datapoints, an autoencoder can generate an interpolated datapoint which approximates these two datapoints semantically. This indicates that autoencoders possess an effective capability of data interpolation. Recently, several research works have paid attention to exploring data interpolation methods in autoencoders [Larsen et al., 2016; Berthelot et al., 2019] and generative models [Radford et al., 2015; White, 2016]. The research has demonstrated that there is a close relationship between representation learning and data interpolation. In other words, the effectiveness of data interpolation can reflect the performance of representation learning to some extent.

∗Equal contribution.

†Corresponding author.

Inspired by the relationship between data interpolation and representation learning, there is a possibility to further improve representation learning through explicitly considering data interpolation in autoencoders. The recent work [Berthelot et al., 2019] proposes a generative adversarial regularization strategy referred to as Adversarially Constrained Autoencoder Interpolation (ACAI). ACAI promotes high-quality data interpolation by introducing regularization into the data reconstruction process, and improves representation learning for downstream tasks. However, as empirically shown in [White, 2016], generative models will generate inferior datapoints when performing linear interpolation, which may cause the distribution of interpolated latent representations to diverge from a prior. Besides, linear interpolation does not have enough capability to traverse a possible region between two datapoints on a data manifold, and the distribution of the latent representations is not considered when performing interpolation.

To alleviate the above issues, we replace linear interpolation with a more effective interpolation method, and explicitly take the distribution of the latent representations into consideration. Specifically, we propose a new interpolation method referred to as multidimensional interpolation, and fuse dual regularizations in both the latent and data spaces, so that both data interpolation and representation learning in autoencoders can be further improved. Briefly speaking, multidimensional interpolation treats each dimension of the latent representation differently, and sets random interpolation coefficients for each dimension. As a result, it will enhance the quality and diversity of interpolation. Furthermore, we constrain the distribution of the latent representations to match with a prior using the MMD framework [Gretton et al., 2007], so that interpolated representations can follow this prior. Then, datapoints generated by performing interpolation and sampling from this prior are encouraged to be perceptually realistic based on the GAN framework [Goodfellow et al., 2014]. In other words, generated datapoints are expected to follow the real data distribution. In Figure 1, we provide a comparison of interpolations between ACAI and our proposed model. Overall, the main contributions of this work are as follows:

• We improve the ACAI model by extending linear interpolation to multidimensional interpolation, such that the latent space can be better explored and thus the diversity of interpolation can be significantly increased.
• On the other hand, we incorporate MMD-based regularization in the latent space with adversarial regularization in the data space to improve the realism of the interpolated datapoints. Compared with ACAI, the main advantage of this work lies in effective data generation from a prior.
• Due to the aforementioned strategies, representation learning in autoencoders can be enhanced significantly. As a result, better performance can be achieved on downstream tasks.

2 RelatedWork

In this section, we briefly introduce some related works on autoencoder variants and data interpolation.

2.1 Autoencoders

Based on regularizations introduced into autoencoders, these variants are generally divided into three categories: Normalbased, VAE-based and GAN-based (VAE refers to as variational AE [Kingma and Welling, 2014]). Specifically, in the Normal-based case, dropout AE (DoAE) [Srivastava et al., 2014] regularizes the latent representation with dropout which randomly drops units in hidden layers. Denoising AE (DnAE) [Vincent et al., 2010] reconstructs clean datapoints from their corrupted versions and can learn the data manifold implicitly. Sparse AE (SAE) [Xu et al., 2016] adds a sparsity term into the objective function to learn high-level features. These three variants explicitly make data reconstruction more difficult by introducing some perturbations and can learn more robust representations.

Different from the above approaches, AEs are regularized to explicitly match the distribution of the latent representations with a predefined prior in the VAE-based and the GAN-based cases. In the VAE-based case, VAE introduces an additional loss into the reconstruction loss. This loss measures the KL divergence between the distribution of the latent representations and the prior. Besides the KL divergence, MMD is also used to measure the distribution divergence in [Tolstikhin et al., 2018]. Along with learning continuous representations, vector-quantized VAE (VQVAE) [van den Oord et al., 2017] applies the distributional regularization framework into learning discrete representations using a learned quantized codebook.

In the GAN-based case, adversarial AE (AAE) [Makhzani et al., 2016] introduces an auxiliary subnetwork referred to as a discriminator based on the GAN framework, and forces this discriminator to determine whether latent representations are inferred from the encoder or sampled from the prior. Moreover, VAE-GAN [Larsen et al., 2016] fuses both the VAE and GAN frameworks in the autoencoder. In VAE-GAN, the traditional pixel-wise reconstruction loss is replaced by an adversarial feature-wise reconstruction loss obtained from the GAN's discriminator.

2.2 Interpolation

In the latent space there are two basic types of interpolations: linear interpolation and spherical interpolation. Linear interpolation combines latent representations linearly, which is easily understood. Hence, it is frequently used to inspect the performance of representation learning [Larsen et al., 2016; Radford et al., 2015; Wu et al., 2016; Dumoulin et al., 2017]. When the latent space is high dimensional, interpolated representations on the linear interpolation path always suffer from change of magnitude. To address this issue, [White, 2016] introduces the spherical interpolation by applying nonlinear interpolation based on a great circle path on an n-dimensional hypersphere. Besides, [Laine, 2018] proposes to perform interpolation along geodesics in the data space rather than in the latent space. [Agustsson et al., 2019] designs several interpolation operations to reduce mismatch between the distribution of interpolated data and a prior.

3 Background of ACAI

ACAI model aims to improve data interpolation and representation learning in autoencoders. Specifically, the network architecture of ACAI consists of an autoencoder and a discriminator. The former, in addition to serving as a basic autoencoder, also forms a GAN with the discriminator to propel the interpolated data to perceptually approximate real data to as realistic an extent as possible.

Formally, the encoder and the decoder are denoted as enc(·) and dec(·) in the autoencoder, respectively; the discriminator is denoted as dis(·). For two datapoints x1 and x2, z1 = enc(x1) and z2 = enc(x2) are their latent representations, respectively. Then, linear interpolation synthesizes a mixture representation zλ:

where λ, constrained to the range [0, 0.5], is an interpolation coefficient of two latent representations. By decoding zλ, an interpolated datapoint xλ = dec(zλ) is generated. From another perspective, the degree of realism of xλ can be controlled by adjusting λ. The larger λ is, the less realistic xλ becomes.

By explicitly regularizing the interpolation process, the discriminator is trained to distinguish between real datapoints and interpolated datapoints. It predicts the interpolation coefficients for interpolated datapoints, and consistently outputs 0 for real datapoints. Similar to GAN, the discriminator's loss function is given by

Meanwhile, the autoencoder's loss function is modified by adding a regularization term as follows:

ACAI has verified that inferred latent representations show more effectiveness on downstream tasks, which indicates that there is a possible link between good data interpolation and useful latent representation learning.

4 ProposedModel

By gradually changing the interpolation coefficient in ACAI, an interpolation path consisting of a set of interpolated datapoints will be formed. This interpolation path can be viewed as a direct line in a linear space. However, linear interpolation is not suitable in a high-dimensional space, and cannot well traverse a possible region between two datapoints. Besides, no explicit constraints have been imposed on the distribution of the latent representations. After interpolation, it leads to a situation where some interpolated representations and corresponding datapoints do not belong to the representation domain and the data domain, respectively. Overall, the capability of interpolation in improving representation learning has not been fully exploited.

To alleviate the above shortcomings, we propose multidimensional interpolation to enhance the capability of interpolation. In addition, we fuse dual regularizations in both the latent and data spaces to improve representation learning. More details will be described in the following subsections.

4.1 Multidimensional Interpolation

Although linear interpolation is straightforward and effective in ACAI, there exist three shortcomings: (1) only a single interpolation path is considered; (2) the interpolation path is confined to a direct line; (3) all dimensions of the latent representation are equally treated.

Hence, to enhance the capability of interpolation, we propose a new interpolation method called multidimensional interpolation. Formally, assuming that latent representations are d-dimension vectors, multidimensional interpolation aims to synthesize a mixture representation as follows:

where µ = [µ1, µ2, ..., µd] is an interpolation coefficient, and ⊙ denotes the element-wise product between two vectors.

What needs to be emphasized is that µ is a vector with the same number of dimensions as the latent representations, and the value of each component of µ is randomly sampled from the uniform distribution on [0, 0.5].

Compared with the scalar coefficient λ in linear interpolation, each component of µ is randomly set, so that each dimension of the latent representations will not be equally treated when performing interpolation. As a result, by varying the interpolation coefficient, an interpolation region will be formed between two latent representations, and the corresponding path can be any curve within this region. From the perspective of manifold learning, the interpolation possesses a capability of traversing a possible region between two datapoints on a data manifold. Overall, multidimensional interpolation can overcome the abovementioned shortcomings and enhance both the quality and diversity of interpolation paths.

4.2 Dual Regularizations

Regularization in Latent Space

When performing data interpolation, the interpolated datapoints are expected to be perceptually realistic. Take natural images as an example, although interpolated datapoints seem realistic, they may not belong to the original image domain in terms of structure and appearance. From the perspective of distribution matching, the corresponding interpolated representations may not well follow the real distribution of the latent representations.

To alleviate the negative effect on representation learning due to the above issue, we explicitly regularize the distribution of the latent representations to follow a prior Gaussian distribution. Then, the corresponding interpolated representations can be expected to follow this prior. Specifically, we propose to minimize the MMD distance between the distribution of the latent representations and this prior.

Distributional Regularization. In practice, the MMD distance can be estimated using a finite number of samples based on the kernel trick, and the square of MMD distance can be approximated by a two-sample test. Given X = {x1, x2, ..., xn} ∼ px(x) and Z = {z1, z2, ..., zn} ∼ pz(z), where px(x) and pz(z) are the data distribution and the prior of latent space respectively, a two-sample test is performed to determine whether the latent representations are inferred from the encoder or sampled from the prior, and the square of MMD distance Lmmd is defined as

where k(·, ·) is a kernel. By minimizing this distance, the distribution of interpolated latent representations can match with the prior.

Regularization in Data Space

Data Generation. After performing interpolation and imposing regularization on the distribution of the latent representations, three types of data can be generated. First, as with ACAI, by performing interpolation and decoding the mixture representation zµ inferred from two datapoints, an interpolated datapoint xµ is generated as follows:

Second, an interpolated datapoint can be generated by decoding a mixture representation between a latent representation inferred from a datapoint and a stochastic latent representation sampled from the prior as follows:

Third, similar to VAE, when the distribution of the latent representations is expected to match the prior, a datapoint can be generated by directly decoding a stochastic latent representation sampled from the prior as follows:

Besides the basic xµ, we expect that with the introduction of two datapoints xµz and xz, data interpolation and representation learning can be further improved.

Adversarial Regularization. The above three types of data are encouraged to be perceptually realistic and cannot be distinguished from real data. Thus, we also adopt adversarial regularization in ACAI. Owing to the introduction of two additional types of datapoints, the original loss functions in ACAI need to be modified to generalize to this case.

Specifically, for the datapoints xµ and xµz, the discriminator predicts the interpolation coefficients µ for interpolated datapoints. Since the datapoint xz is directly generated based on the prior, it would be less realistic than xµ and xµz, and does not effectively approximate real data. As mentioned in Section 3, the interpolation coefficient can reflect the realism of the interpolated datapoint. Therefore, the discriminator predicts a predefined maximum of the interpolation coefficient for xz. This predefined maximum is set as 0.5 here.

From another perspective, we incorporate data generation into the interpolation process, and formulate them in a unified framework. To sum up, the overall losses of the discriminator and the autoencoder are modified as follows:

where ω1, ω2, ω3 and ω4 are hyper-parameters for adjusting the weights of the above losses. We set ω1 and ω2 as 1.0 and 0.5 for all experiments, respectively. The remaining parameters ω3 and ω4 are tuned to be the best by experience.

5 Experiments

5.1 Experimental Setting

In our experiments, we evaluate our proposed models on the following datasets: MNIST [LeCun et al., 1998], SVHN [Netzer et al., 2011], CIFAR-10 [Krizhevsky and Hinton, 2009] and CelebA [Liu et al., 2015]. For a fair comparison, we use the network architecture from ACAI for all discussed models.

To study the effect of multidimensional interpolation, we only replace the linear interpolation step of ACAI with the proposed interpolation operation. In this case, this model is referred to as Autoencoder with Multidimensional Interpolation (MI-AE). To study the effect of regularization in the latent space, we only add distributional regularization into ACAI. In this case, this model is referred to as Autoencoder with Latent Regularization (LR-AE). Furthermore, we consider both multidimensional interpolation and dual regularizations to verify their effectiveness. In this case, this model is referred to as Autoencoder with Multidimensional Interpolation and Dual Regularizations (MIDR-AE).

Figure 3: Distributions of the latent representations from ACAI (left) and MI-AE (right). The principal components of the latent representations are illustrated by black solid lines.

5.2 Interpolations on Synthetic Data

In this subsection, we investigate the difference between linear and multidimensional interpolations on a synthetic dataset. Specifically, this dataset consists of 10k gray images with 32*32 resolution. An image is generated using a sinusoidal function, which looks like parallel white stripes with a black background. The value of the pixel (x, y) in an image is calculated as follows:

where α ∈ [0.3, 0.7] controls the number of stripes, and β ∈ [0, 2π] determines the angle of stripes. ∆x = x - xc and ∆y = y - yc denote the relative positions between the pixel (x, y) and the central pixel (xc, yc). We set ∆α = 0.004 and ∆β = 0.02π, and randomly choose α and β to generate the whole dataset. Some samples are shown in Figure 2.

We use the synthesized data to train ACAI and MI-AE, and compare the difference between the distributions of learned latent representations. For better visualization, we set the number of dimensions of the latent representations as dz = 2 in the experiments.

In Figure 3, both ACAI and MI-AE gather most of the latent representations close to the origin. We display two principal components of the learned latent representations for ACAI and MI-AE. From Figure 3, we can see that the difference between two components from ACAI is larger than that from MI-AE. We calculate the mean of the absolute off-diagonal values in the normalized covariance matrix [Lee Rodgers and Nicewander, 1988] to quantify the correlation between dimensions. The correlation coefficient values between dimensions are 0.523 and 0.159 for ACAI and MI-AE, respectively, which indicates that the correlation between dimensions of the latent representations from MIAE is much less than that fromACAI. This indicates that autoencoders with multidimensional interpolation learn latent representations with lower correlation.

In addition, we perform experiments on real datasets and measure the resulting correlation between dimensions of the learned latent representations. As shown in Table 1, the values for ACAI are much larger than those for MI-AE on all datasets. Overall, using multidimensional interpolation helps autoencoders to disentangle the correlation between dimensions of the latent representations.

5.3 Interpolations on Real Data

For a fair comparison with ACAI, we adopt the proposed MI-AE and set µ = [λ, λ, ..., λ], and further compare the performance of their interpolations on real data. The interpolated samples are shown in Figure 4.

From Figure 4, on MNIST and CIFAR-10, there is no obvious perceptual difference between interpolated samples. However, by zooming on SVHN and CelebA, more details can be observed. On SVHN, samples generated by ACAI are easily dominated by one of the real datapoints (highlighted with a red rectangle at the middle part), while our proposed MI-AE can generate more reasonable and balanced interpolated samples. On CelebA, samples generated by MI-AE at the right part are much clearer than those by ACAI (highlighted with a red rectangle at the right part), and the colors of lips generated by MI-AE can still be easily recognized. This indicates that our proposed MIAE can learn latent representations while preserving more detailed patterns. Overall, the proposed multidimensional interpolation process shows a better performance than the linear interpolation in ACAI.

5.4 Improved Representation Learning

In this subsection, we examine whether multidimensional interpolation and dual regularizations are more beneficial to representation learning. In particular, we perform experiments on image classification and clustering based on inferred latent representations. By evaluating the performance of these downstream tasks and comparing our models with other models, the advantage of our models can be verified.

Here, we consider several representative models in Section 2 as follows: basic AE (BAE), DoAE, DnAE, VAE, AAE, VQVAE and ACAI. In addition, the optimal hyperparameters of our models are as follows: ω3 = 0.5 and ω4 = 0.1 for MNIST; ω3 = 0.5 and ω4 = 0.05 for SVHN; ω3 = 0.01 and ω4 = 0.01 for CIFAR-10, and ω3 = 0.1 and ω4 = 0.05 for CelebA.

Image Classification

In order to measure the quality of representation learning in autoencoders, a supervised classification task is also conducted based on the inferred latent representations. If the inferred representations have distilled the important class information, a classifier can achieve a reasonable performance regardless of its simplicity. Therefore, as with ACAI, we train the additional single-layer classifier [Coates et al., 2011] by feeding inferred representations of the encoder along with the whole model. We summarize the classification results in Table 2.

From Table 2, MI-AE and LR-AE show a better performance than other models on all datasets. Especially on SVHN (dz=32) and CIFAR-10 (dz=256/1024), MI-AE and LR-AE obviously outperform the second best ACAI. Moreover, MIDR-AE further boosts the performance of image classification compared to MI-AE, and achieves the best performance. It is worth noting that ACAI only achieves an average accuracy of 34.47%, while MIDR-AE achieves 92.28% on SVHN when the number of dimensions of the latent representations is set as 32. These experimental results show that our autoencoder variants can distill more class information.

Viewed from the perspective of manifold learning, multidimensional interpolation is more beneficial to traversing a data manifold. By regularizing the distribution of the latent representations, the interpolated representations can also follow a prior. Therefore, both of them will improve representation learning.

Image Clustering

Since clustering groups datapoints with similar attributes together and separates datapoints with different attributes apart in an unsupervised way, it is another challenging task for evaluating the performance of representation learning. Therefore, if the autoencoder has uncovered the important class characteristics of the data, then performing clustering on the latent representations should yield reasonable results.

As with ACAI, we use PCA whitening on the latent representations, and randomly perform K-Means [MacQueen, 1967] 1k times to choose the clustering solution with the best objective value during training. For evaluation, we adopt the methodology of [Xie et al., 2016; Hu et al., 2017] to evaluate the "clustering accuracy" on the test dataset. We summarize the clustering results in Table 3.

From Table 3, MI-AE and LR-AE can produce reasonable performance gains than other models on both MNIST and SVHN. In particular, on SVHN, MI-AE and LR-AE significantly outperform ACAI. Compared to MI-AE and LR-AE, MIDR-AE further boosts the performance of image clustering and achieves the best performance. MIDR-AE is always superior to ACAI with a large performance gain on both datasets. Especially on SVHN, MIDR-AE almost has a clustering accuracy improvement of 20% above ACAI when the number of dimensions of the latent representations is set as 32 or 256. Similar to image classification, we can conclude that multidimensional interpolation and dual regularizations enhance representation learning.

Besides, we evaluate the clustering performance of three other models: KMeansData, DEC [Xie et al., 2016] and IMSAT [Hu et al., 2017]. These models are different from the autoencoder variants. Specifically, KMeansData refers to performing K-Means directly on the data; DEC simultaneously learns feature representations and cluster assignments using deep neural networks; and IMSAT learns invariant representations for clustering by adopting data augmentation and mutual information maximization [Deng et al., 2009]. Compared to DEC, all our models are superior, while IMSAT just slightly outperforms MIDR-AE on two datasets. This demonstrates that our proposed models show a competitive performance on representation learning.

5.5 Data Generation

Since the distribution of the latent representations is enforced to match with a prior, one main advantage of our model is its capability of generating data by sampling from this prior. Here, we show some samples generated by our proposed model in Figure 5. Compared to our model, ACAI does not possess this capability since it does not consider the distribution of the latent representations. This indicates that our model can learn semantically meaningful latent representations which well reveal the data distribution.

6 Conclusion

In this work, we have comprehensively studied both data interpolation and representation learning in autoencoders. We observe that existing interpolation methods in autoencoders do not have enough capability of traversing a data manifold, and the distribution of the latent representations is not considered when performing interpolation. To improve both data interpolation and representation learning, we propose a model referred to as MIDR-AE. In MIDR-AE, on one hand, a multidimensional interpolation process is proposed to enhance the capability of data interpolation. On the other hand, autoencoders are further regularized in both the latent and data spaces, which imposes a prior on the latent representations and encourages datapoints generated by performing interpolation and sampling from this prior to be realistic. Through extensive experiments, we verify that representation learning of our model exhibits better performance on downstream tasks compared to competing models. For the purpose of reproduction and extensions, our code is publicly available. 1

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China (Project No. 61502173, U1611461), in part by the Research Grants Council of the Hong Kong Special Administration Region (Project No. CityU 11300715), in part by City University of Hong Kong (Project No. 7005055), in part by the Natural Science Foundation of Guangdong Province (Project No. 2016A030310422), and in part by Fundamental Research Funds for the Central Universities (Project No. 2018ZD33).

1 https://github.com/guanyuelee/midrae