FakeTables: Using GANs to Generate Functional Dependency Preserving Tables with Bounded Real Data

1 Introduction

Over the past few decades, machine learning related research has seen remarkable developments, driven in part by the tremendous amount of data that has become available. In many cases, an organization such as a government or company wants to release some data to the public, but is restricted in the amount of data released due to legal, privacy and other concerns. For instance, the U.S. Census Bureau releases 1% of its records - but these do not completely represent the US population. At the same time, averages about the entire population's income and some other attributes are also released. Many U.S. government survey datasets are released in such a manner and this has been the practice in various fields, such as business, operations research and social science.

∗Co-corresponding authors.

Data augmentation has recently drawn considerable attention in the machine learning (ML) community, where the objective is to improve ML models by enhancing the quality of data. In this paper, we study an incomplete table synthesis (ITS) problem for tabular data augmentation, where we wish to augment the released incomplete sub-table of records X′ by synthesizing a new table Y of records, so that a machine learning model trained on the augmented table X′ ∪Y works better for the full table X (which X′ originated from) than a model that is trained solely on X′ .

GANs have been widely adopted in image generation [Isola et al., 2017; Zhu et al., 2017], language analytics, [Pascual et al., 2017] - there also has been some very recent interest w.r.t. tables [Park et al., 2018]). However, using GANs to solve the ITS problem requires addressing two new challenges. First, while we wish to learn a generative model from the sub-table X′ , we also need to maintain the broad statistics of the table X. Second, relational tables usually have an associated set of functional dependencies (FDs) - see Definition 3.1. As FDs capture dependencies between attributes, it is critical to ensure that X′ ∪ Y also satisfy these dependencies, otherwise the integrity of X′ ∪ Y is questionable. To the best of our knowledge, our paper is the first to solve the problem of tabular data synthesis with these two constraints.

To tackle the two challenges, we propose a novel framework called ITS-GAN, which makes three innovations in customizing traditional GANs to efficiently learn a synthetic table generator. First, we add two extra loss terms into the original generator loss function to encode these two constraints in the GAN training process. A major difficulty of encoding the FD constraints into the loss function is that the FDs associated with a table are usually expressed at the schema (column) level. For example, in Table 1, an example schemalevel FD is "Position" determines "Salary". One schemalevel FD generates numerous record-level FDs (see Definition 3.2). An example of a record-level FD in Table 1 is, the Position "CEO" determines the Salary "$2M". While we can enumerate the two record-level FDs in Table 1, it is impossible to explicitly list all the record-level FDs in real-world large datasets with numerous records. Therefore, our second innovation is to train an autoencoder (one for each schemalevel FD) to model the record-level FDs. The autoencoders are pre-trained to reproduce the FDs and then fed into the generator loss function. Moreover, to improve the training of the GAN discriminator, we make our third innovation via a new customized calibration of the discriminator's input using the FD errors calculated by the pre-trained autoencoders.

We provide a theoretical analysis of ITS-GAN under certain conditions as well as extensive empirical evaluations on two widely used datasets - Census and BTS. The empirical results show that ITS-GAN significantly improves the performance of classification tasks compared with state-of-theart data augmentation approaches. An ablation study also demonstrates the efficacy of ITS-GAN by taking into account the table statistics and FD constraints in the ITS problem.

2 RelatedWork

In this section, we describe related work. We discuss classical augmentation methods, followed by recent advanced GANbased methods.

2.1 Classical AugmentationMethods

Various data augmentation methods are used to handle imbalanced data - SMOTE [Chawla et al., 2002] is among the most popular approaches. SMOTE generates weighted combinations of randomly selected existing records. Borderline SMOTE (BSMOTE) [Han et al., 2005] uses a similar method around a class boundary. ADASYN [He et al., 2008] was designed on top of SMOTE, where weights are adjusted in order to account for various factors.

The condensation method [Aggarwal and Yu, 2004] is a classical non-machine learning method to synthesize tables, which assumes that a column follows a specific statistical distribution and synthesizes values for the column by exploiting its assumed statistical characteristics. In many cases, however, its synthesis range is narrow and does not reproduce the entire value range of the column.

All these approaches are not suitable for our problem because they do not support the preservation of FDs.

2.2 GAN-based AugmentationMethods

GANs consist of two neural networks: a generator and a discriminator [Goodfellow et al., 2014], which are alternately updated as a two-player zero-sum minimax game:

where V (G,D) is the value function of the discriminator, pdata is the probability distribution of the real dataset, z is the input noise of the generator which follows the prior distribution pz, G(z) is a generator function mapping z to the generated data x, and D(x) is a discriminator function which specifies the probability that a discriminator input x is from the real dataset.

GANs have been exploited to synthesize electronic health records in [Choi et al., 2017; Baowaly et al., 2018]. However, they are mostly interested in synthesizing discrete values in patient records and cannot be directly applied to synthesize tables with other types of values. More recently, TableGAN was proposed to generate fake tables using GANs given an original table [Park et al., 2018]. However, TableGAN synthesizes a full table X directly and does not consider FDs. DCGAN [Radford et al., 2015] is a benchmark image generation framework which can be customized to synthesize tables [Park et al., 2018], but it also fails to consider the table statistics and the FD constraints.

3 ProblemDescription

Let X denote a full table of records. We use Xi to denote the i'th record and Xj to denote the j'th column of X. A sub-table X′ ⊆ X is a subset of rows of X. Table 1 shows an example of a full table X, where the first row X1 = [CEO, Null, John Doe, $2M], the first column X1 = [CEO, Manager, Manager], and the first two rows form a sub-table X′ .

Each full table is accompanied by a vector ¯X specifying the average value of each numerical column. For example, the average value of X4 is 1.33M. For relational tables, there usually exist a set of schema-level functional dependencies that preserve the validity and consistency of the tabular data.

Definition 3.1. Suppose A,B are two disjoint subsets of the schema (columns) of X′ . A schema-level functional dependency (FD) F associated with X′ has the form

This says that for any record (row), the column values in A uniquely determine the column values in B.

The set of all schema-level FDs is denoted as F = {F1, F2, . . . , Fn}. E.g., in Table 1, 'Position' uniquely determines 'Salary', while 'Position' and 'Team' uniquely determines 'Name'. Therefore, the full set of FDs in Table 1 is F = {F1, F2}, where: F1 : X1 →X4 , and F2 : X1 ,X2 → X3 . As opposed to the schema-level FDs, we also define:

Definition 3.2. Let a ∈ A, b ∈ B denote two entries in A and B, a record-level FD f has the form

It says that the specific value of 'a' uniquely determines the specific value of 'b'.

We refer to the example in the Introduction to illustrate the difference between a schema-level and a record-level FDs. As introduced earlier, there are many real-world scenarios where an organization wants to release some data to the public, but is restricted in the amount of data. The common practice of these organizations is to release a small portion of the dataset, along with some statistics of the entire dataset. To improve the ML model trained on the released dataset, we solve the following ITS problem.

Definition 3.3. Incomplete Table Synthesis (ITS): Given a sub-table X′ ⊆ X, a column-wise average ¯X of the numerical attributes of the full table X and a set of FDs F of the sub-table X′ , our objective is to generate a synthetic table Y such that

Figure 1: The architectures of ITS-GAN.

• Y has the same number of columns as X and X′ .
• Y satisfies the set of FDs in F.
• Y maintains the column-wise average ¯X of X.

4 ITS-GAN

Adapting classical GANs to address the ITS problem poses two challenges: (i) We need to learn a generative model from the sub-table X′ and maintain the column-wise statistics associated with the full table X. These two objectives contradict each other when ¯ X′ ̸= ¯X. (ii) During the table synthesis process, we need to maintain the record-level FDs.

To handle the challenges, we propose the ITS-GAN (Incomplete Table Synthesis with Generative Adversarial Network) framework shown in Figure 1. ITS-GAN has three novelties. First, it trains one autoencoder-based model for each schema-level FD in F. Second, it incorporates the table statistics and FD constraints in training the generator so that the generator establishes a trade-off between the two contradicting objectives mentioned above. Third, it calibrates the discriminator using the pre-trained autoencoders, so that the learned discriminator is able to capture the FD constraints.

4.1 Modelling FDs with Autoencoders

Autoencoders were initially proposed to capture the latent vector of an input in between an encoder and a decoder [Hinton and Salakhutdinov, 2006; Baldi, 2011]. In ITS-GAN, we train an autoencoder to reproduce the right-hand side of a record-level functional dependency, given its left-hand side. There is one autoencoder for each schema-level functional dependency and it is pre-trained with all record-level instances in the sub-table X′ .

Recall that a schema-level functional dependency is represented as F : A → B, where A and B are column-wise subsets of the original table. As a result, the inputs of an autoencoder are the entries of A, and the outputs are the corresponding entries in B. As shown in Figure 1 (a), for each autoencoder, its encoder component uses a convolution layer with a filter size of 5 and a stride of 2, while the decoder component uses a transpose convolution layer with a filter size of 5 and the same stride value.

4.2 Generator

To handle the functional dependency and table statistics constraints, we propose the following adapted loss function for the generator:

where LG is the original loss function in Eq. (1). ‖¯ Y - ¯X‖1 is an error term which penalizes the difference between the column-wise average of the generated table Y and the original table X. Throughout this paper, we use L1-norm as the distance measure between vectors. ∑i ∑j ‖B′ij - Bij‖1 is another error term which characterizes the set of FD constraints, where B′ij is the right hand side of the ith schemalevel FD in the j th record instance produced by the generator, and Bij is the corresponding autoencoder's anticipated right hand side of the ith schema-level FD (given the left hand side in the generated j th record). α and β are hyper-parameters which indicate the weights of different error terms.

Approximating statistics Error. Note that in mini-batch training (e.g., a batch size of 64), an average of one minibatch does not accurately represent the statistics that the generator learns, thus providing a biased estimation of the table statistics error ‖¯ Y - ¯X‖1. To get a better approximation, we update the average value using a moving average rule [Winters, 1960], where the moving average ˆ¯ Yt+1 of the (t + 1)th epoch is a weighted combination of the moving average of the tth epoch and the average of the (t + 1)th epoch:

Here w is a hyper-parameter indicating the update rate - we set w = 0.01 in our experiments. In our generator, we use 3 layers of transpose convolutions, where the filter size is 5 and the stride value is 1.

4.3 Discriminator

The architecture of our discriminator is shown in Figure 1 (b). The process has two stages highlighted in yellow and blue respectively. The input (i.e., either fake records from the generator or the real data) is first filtered with three convolution layers in the first stage and later concatenated with the FD error terms from the autoencoders in the second stage. In each convolution layer, we use a convolution with the same filter size and stride value as that of the encoder.

Note that given a real record (row), FD error vectors will contain values close to zero if autoencoders are properly pretrained. As a consequence, it will be relatively easy to detect a fake record if the norm of the error vector of the fake record is large, rather than small. This gives the generator a strong incentive to ensure that the FD error rate is small. Recall that we impose this requirement in the definition of the loss function of the generator (cf. Eq. (2)). Because of this, the ITS-GAN architecture is optimized toward minimizing violations of FDs through the FD error terms. To capture this property and improve the training of both the discriminator and the generator, we concatenate the output of the convolutional layers with the functional dependency errors (obtained from the pre-trained autoencoder). More specifically, for the j th fake record produced by the generator, we concatenate it with a set of error terms ‖B′ij - Bij‖ for all Fi ∈ F. After concatenation, two fully connected layers follow, and a Sigmoid activation is used to generate the prediction for the record (i.e., real or fake). We train the discriminator with the same original loss function as in Eq. (1) - another option is the loss in [Gulrajani et al., 2017] but we found that the original loss sometimes produces better results in our task.

5 Theoretical Results

As described earlier, we introduce two new error terms on the generator loss function to handle the table statistics and functional dependency constraints. However, if the deviation from X′ for the two error terms is huge for a given record, it will be easy for the discriminator to classify that record as fake. Thus, the generator should find a balance in which the match between ground-truth statistics and X′ ∪ Y statistics is good, but at the same time, the discriminator cannot distinguish between real and fake records. The following theorem describes the equilibrium solution to the minimax game between the generator and the discriminator when there is no penalty for errors in the statistics of X′ ∪ Y compared to X.

Theorem 5.1. The equilibrium distribution D∗ of the discriminator and p∗G of the generator in ITS-GAN is, D∗ = pX′ pX′ +p∗G , and p∗G = pX′ , if there is no table statistics error term (i.e., the second term on the right hand side of Eq. (2)) in the generator loss function of ITS-GAN.

Proof. We denote a single record in a table as a vector x. For a fixed generator G, the discriminator's objective is to minimize the following loss function:

Performing a point-wise (i.e., for each x ∼ pX′ ) optimization of the objective, we can obtain the best response of the discriminator as

Substitute it into the loss function of the generator, we have

Note that Ex∼pX′ [-log 2] + Ex∼pg [-log 2] = -log 4. Adding the above two equations, and following the definition of KL-divergence, we have

where the second equality uses the definition of JSdivergence. Since the JS-divergence between two distributions is always non-negative and zero if they are equal, as a result, we see that the minimum of the second term on the right hand side is obtained when p∗G = pX′ .

We now look at the third term, which is an indicator of the functional dependency error. For this error term, the optimum is obtained when B′ij = Bij, for each record j and each functional dependency Fi ∈ F. This implies that, the data distribution of a subset of columns in the generated table Y should be the same as that of the sub-table X′ . As a result, the condition p∗G = pX′ does not violate the necessary condition of obtaining optimum of the functional dependency error. Therefore, under equilibrium, we have p∗G = pX′ , and D∗ = pX′ pX′ +p∗G .

Note that p∗G = pX′ is not a sufficient condition for the optimality of minimizing the functional dependency loss, and its optimum is obtained (with a more strict condition) by enforcing the modelled functional dependency function (via the autoencoders). Moreover, the equilibrium obtained in practice is also dependent on how well the FDs are fitted by the autoencoders. The following two corollaries can be immediately obtained from Theorem 5.1:

Corollary 5.1.1. The equilibrium D∗ and p∗G described in Theorem 5.1 can still obtained when ¯ X′ = ¯X (i.e., the column-wise average values of the full table and the sub-table are the same) and any if one of the following conditions hold:

• Full batch gradient descent is performed in training generator neural networks.
• The moving average vector update weight w → 0 or the weight α → 0 in the generator loss function with enough training epochs.

Corollary 5.1.2. When ¯ X′ ̸= ¯X, for discriminator, we still have D∗ = pX′ pX′ +p∗G , while for generator, p∗G(x) is biased towards ¯X compared with pX′ .

Corollary 5.1.2 implies that our synthesized new table Y of records is able to find a fine balance point which trades off among the three conditions specified in Definition 3.3.

6 Empirical Results

We describe detailed experiment settings and results on two representative datasets with various evaluation metrics.

6.1 Datasets and Experiment Settings

Datasets. The ITS problem occurs frequently in real world scenarios. We select the following two representative datasets to evaluate the performance of ITS-GAN. (i) Census [Kohavi, 1996] is a dataset that contains 30K samples of population data released by the US Censors Bureau. (ii) BTS 1 is the Air Carrier Statistics (ACS) dataset from the US Bureau of Transportation Statistics that describes 80K domestic air ticket sales in the US in a month. In particular, these two datasets are considered as standards in many fields such as Social Science, Civil Engineering, Transportation, etc.

Preprocessing. ITS-GAN synthesizes three types of columns. (i) Continuous Nnumeric: It is straightforward to learn and generate real numbers. (ii) Discrete numeric: After generating a numeric value v, we round it to the nearest column value in X′ , e.g., 1.5 will be rounded to 3 if the corresponding column in X′ only contains {-10, 3, 4, 5}. (iii) Categorical: We use a label encoding method 2 to assign an integer value to each category or class, after which generating categorical values is equivalent to the discrete numeric case. For both Census and BTS datasets, the ground-truth full-table statistics (in addition to the released samples) are also released. However, we do not know their original full tables as these are not publicly available. Hence, we cannot compare our synthetic tables with the unknown full tables. Therefore, we create our own evaluation data from their released data as follows. We treat the Census and BTS data we have as the full table (i.e. X) and then create sub-tables (X′ ) consisting of p% of X. We test p ∈ {1, 5} as we are interested in very challenging situations where p is very small. Each of the two datasets has one functional dependency: gender, relationship determine marital-status in Census and source/destination airports, flown distance determine overall distance in BTS.

Experiment Settings. Our experiments are run on Ubuntu 18.04 with CUDA 10, Tensorflow r12. The batch size is 64, with learning rate of 0.0002 and Adam optimizer (with beta parameter of 0.5). The generator and discriminator are trained alternately as in [Goodfellow et al., 2014]. For the weights in Eq. (2), we use α = β = 3 for Census and 5 for BTS.

6.2 CDF Comparison

To evaluate whether a generated table Y is close to the original full table X, we first compare the cumulative distribution functions (CDFs) of X and Y . We compare with the stateof-the-art table synthesis approach, TableGAN [Park et al., 2018]. Figure 2 displays the CDFs of some selected schema (columns). In many cases, CDFs of ITS-GAN in red are close to CDFs of full table X in blue, suggesting that ITSGANperforms very well. In Figure 2 (d), however, ITS-GAN is close to sub-table X′ in orange around the X-axis value range of 3000-5000, which means our generation does not always successfully capture the statistics of the full table. In 70% (resp. 60%) of the columns, ITS-GAN is closer to full table than to sub-table in Census (resp. BTS) in terms of the Earth mover distance [Pele and Werman, 2009]. In contrast, the CDFs of TableGAN in green are mostly close to CDFs of sub-table X′ in orange, which demonstrates that ITS-GAN is superior to TableGAN in terms of synthesizing a full table.

1 https://www.transtats.bts.gov/Tables.asp?DB ID=111

2 https://scikit-learn.org/stable/modules/generated/sklearn.

preprocessing.LabelEncoder.html

6.3 Table Augmentation Performance Comparison

To test the utility of augmentation using the synthetic table generated by ITS-GAN, we run classification tasks on the two datasets. More specifically, we predict the "hours-per-week" in the Census data and "ticket price" in the BTS data. The labels are set as 1 (resp. 0) if the value is larger or equal to (resp. smaller than) the median. To find the classifier with the best performance, we do a grid search over AdaBoost, RandomForest, GradientBoosting with various hyper-parameters and using F-1 as the model selection criterion.

Baseline Methods. The set of baseline methods include:

• CM [Aggarwal and Yu, 2004], ROS [Lemaˆıtre et al., 2017], ADASYN [He et al., 2008], SMOTE [Chawla et al., 2002] and its variants BSMOTE [Han et al., 2005], SVMSMOTE [Demidova and Klyueva, 2017], SMOTENC [Chawla et al., 2002], etc.
• DCGAN [Radford et al., 2015], which is a benchmark GAN framework for generating images. DCGAN is customized to generate tabular data.
• TableGAN [Park et al., 2018], which is the state-of-theart GAN-based table synthesis approach synthesizes a full table and does not consider FDs.

The hyper-parameters are set as stated in the associated papers (when specified), or otherwise tuned via a grid search.

Table 2 summarizes the classification results. Note that some baselines do not properly perform the augmentation task - in particular, SMOTENC and ADASYN produce runtime errors in some sub-tables. This happens when the subtable size is small and/or biased and their subroutines cannot process it. SMOTE and BSMOTE are able to augment all sub-tables in a reliable manner. DCGAN is always inferior to TableGAN as observed in [Park et al., 2018] and we remove CM from the table due to its poor generation quality. ITS-GAN outperforms all the baseline methods in terms of F1 score and precision in Census and achieves the best F1 score and recall in BTS. For recall in Census and precision in BTS, Boundary-SMOTE (BSMOTE), which augments around classification boundary, shows the best performance. However, BSMOTE's overall F1 measure is inferior to ITS-GAN. The other baselines show relatively unreliable performance. Moreover, GAN-based augmentation methods show more reliable performance when X′ is 1% of X. The performance gap between the 1% and 5% cases for the baseline methods is around 3-5% whereas it is around 1-2% for ITS-GAN and TableGAN. We also calculate the statistical significance between ITS-GAN and BSMOTE across all predictions - the p-value is 0.002.

To further show the significance of table augmentation on ML tasks, we also compare with the ML model trained on a larger dataset which contains the sub-table X′ . This dataset contains 15% of records from the full table X. Surprisingly, our method shows comparable performance to the models trained on the much larger dataset, and performs even better in some cases, e.g., the F1 score of ITS-GAN is 0.478 in Census, and is 0.463 for the ML model trained on the 15% dataset.

Ablation Study. To check the efficacy of the table statistic information and FD constraints, we also do an ablation study. We consider two variations of ITS-GAN: ITS-GAN (no FDs) and ITS-GAN (no Statistics) where autoencoders or the statistics matching loss of the generation are missing respectively. ITS-GAN without both of these components is comparable to TableGAN, so we exclude it. The results are also shown in Table 2. We see that for all cases, ITS-GAN outperforms the two variants with one component missing. This demonstrates that ITS-GAN is able to exploit both the table statistic information and the FD constraints to better augment a sub-table.

7 Conclusion

We presented a tabular data augmentation framework called ITS-GAN. Our approach is widely applicable under scenarios where we are given a sub-table from a full table, along with some statistics associated with the full table. This problem setting occurs very frequently in many real-world cases, especially when an organization like the government is restricted in the amount of data that it can release, e.g. due to legal and privacy concerns. ITS-GAN adapts GANs to maintain both the statistics of the full table and the FDs in the released sub-table. We evaluated the performance of data augmentation for ITS-GAN on two very important datasets, Census and BTS. The results show that our approach is able to significantly improve the classification tasks, and outperforms the state-of-the-art tabular data augmentation approaches by a significant edge.

Acknowledgements

This work is supported by ONR grants N00014-18-1-2670 and N00014-16-1-2896 and ARO grant W911NF-13-1-0421. Authors are in alphabetical order.