A Novel GAN Approach to Augment Limited Tabular Data for Short-Term Substance Use Prediction

1 Introduction

Substance use can create short- and long-term negative consequences for persons who use drugs (PWUDs) [HHS, 2023; Newcomb and Bentler, 1989; NIDA, 2020]. These consequences include mental illness, HIV/AIDS, hepatitis, drug overdose, and death. Within the U.S. alone, an estimated 161.8 million people aged 12 or older used a substance (out of which 40.0 million used an illicit drug) in the past month before being interviewed in 2021 [SAMHSA, 2022]. Furthermore, according to [NIDA, 2023], substance-involved overdose deaths, including those related to illicit drugs and prescription opioids, continue to increase over the years, with 106,699 deaths in 2021 compared to 91,799 (+16%) in 2020 and 70,630 (+51%) in 2019. This alarming trend also applies globally, with the estimated number of PWUDs in the past

12 months reaching 296 million in 2021 from 240 million in 2011 (out of which 39.5 million and 27.3 million had drug use disorders, respectively) [United Nations, 2023b].

In response to these large-scale negative impacts, various public and private organizations around the globe have prompted initiatives for preventing and reducing substance use at both population and individual levels. Prominent examples are the United Nation's Sustainability Goal 3: "Strengthen the prevention and treatment of substance abuse" [United Nations, 2023a] and U.S. Department of Health and Human Services (HHS)'s Healthy People 2030: "Reduce misuse of drugs and alcohol" [ODPHP, 2020].

Generally, approaches toward these initiatives focus on designing and deploying intervention and outreach programs/resources (e.g., rehabs and consulting services) for PWUDs, with the main goal of reducing and eliminating their usage of certain substances [Ray et al., 2020; ColledgeFrisby et al., 2023]. While these programs and resources have shown to be effective to some extent [Ouimette et al., 1997; Tanner-Smith et al., 2016], they often require volunteer participation from PWUDs, who face the difficulties and reluctances of (self-)evaluating and (self-)determining whether they want or need help [Russell et al., 2021; Wilson and Brown, 2023]. Even when PWUDs agree to use these programs/resources, they may have already experienced prior harms such as overdose and mental illness [SAMHSA, 2021; Andersson et al., 2023]. Therefore, it is important to prevent harm from occurring in the first place by carefully identifying PWUDs at the highest risk (i.e., those who are prone to drastically increase usage of some drug) and allocating them appropriate resources to reduce or eliminate potential harms.

Unfortunately, forecasting individual substance use behavior is challenging due to its complex patterns (e.g., drug use frequency and co-use of multiple drugs) and tendency to change over time at short timescales (i.e., within months) [Karamouzian et al., 2022; Linden-Carmichael et al., 2022; Lorvick et al., 2023], as well as the lack of appropriate data and models for predicting short-term future substance use (see Existing Efforts and Limitations and additional contents in the full paper attached in the supplementary material).

Thus, our goal is to design accurate predictive models for modeling short-term drug usage (i.e., within months) from PWUDs to aid healthcare agencies, local communities, policymakers, and other stakeholders in the efficient allocation of resources to PWUDs who need the most help. An effective predictive model has the potential to improve the well-being of millions of PWUDs and hampers the ongoing rapid growth of drug use prevalence and drug overdose death rates.

Our Approach and Associated Challenges. To address the above shortcomings of substance use data and models, we formed a collaboration between computer scientists and domain experts in substance use research, including social scientists, intervention specialists, and survey interviewers from University of Nebraska-Lincoln and Rural Drug Addiction Research Center. With IRB approval, our team recruited a local sample of 258 PWUDs in the Great Plains of the U.S. from which longitudinal survey data were collected and stored in tabular format (detailed below in Section 2.1). Despite its high value and relevance to our stated goal, the sample size is small due to various challenges during data collection (mainly disruptions due to COVID-19 and the lack of visible harm reduction movements and punitive laws in the region), As a result, our preliminary predictive models trained on currently available data achieved subpar performance with respect to the baseline (demonstrated in our full paper). These models seem to overfit due to the limited training examples (<200) with respect to the number of (preprocessed) features (>600).

One effective way to tackle overfitting on small datasets is via data augmentation [Shorten and Khoshgoftaar, 2019] i.e., creating synthetic samples based on real data to increase its sample size. Deep generative models such as the popular Generative Adversarial Network (GAN) [Goodfellow et al., 2014] provide powerful tools for this purpose due to their flexibility in representing complex and high-dimensional data distributions. In recent years, these models have been extended to generate tabular data, especially healthcare records, using specialized architectures (e.g., GOGGLE [Liu et al., 2022]) and/or novel training algorithms (CTGAN [Xu et al., 2019]), which in turn achieved promising performance across different evaluation criteria.

However, these models were benchmarked mostly on moderate to large datasets that typically contain at least thousands of training examples. In rare cases when being applied on small datasets with less than 1000 samples, e.g., Diabetes (768) and Breast (569) [Street et al., 1993] as reported respectively in the work for GOGGLE and TabDDPM [Kotelnikov et al., 2023], the proposed models and its selected comparatives often either fail to or narrowly outperform simple baselines such as Bayesian networks [Pearl, 2011] and SMOTE [Chawla et al., 2002] in terms of their considered evaluation metrics. The benchmark datasets for state-of-the-art models also contain no more than 200 features, which is substantially smaller than the size of our feature set (>600 after preprocessing). Additionally, the survey used for collecting our tabular data contains skip logic, a commonly employed functionality in survey design for social science applications [Fowler, 1995; Dillman et al., 2014], which has not been addressed by the aforementioned state-of-the-art models.

Our Contributions. In this paper, we design a novel specialized generative model to augment our tabularized survey data with skip logic in order to improve the predictive performance of our classification models, which ultimately predict the following two short-term substance use behaviors: for a given PWUD and a certain drug, (A) whether they would increase its usage and (B) at which frequency (on a pre-defined ordinal scale) they would use it within the next 12 months. We summarize our contributions as follows:

• (I) We believe ours is the first work that addresses skip logic from surveys in tabular data generation. We demonstrate its practical value by showing both conceptually and empirically how enforcing constraints from skip logic (or skip constraints for brevity) positively affects the training of our generative model. We are also one of the first to investigate the real-world feasibility of deep generative models in settings where the number of features exceed the sample size i.e., high-dimension low-samplesize (HDLSS). (See Related Work in our full paper.)
• (II) We design a novel GAN that deals with small tabular data containing <258 samples and 210 features (209 plus one target variable). Specifically, we leverage CTGAN [Xu et al., 2019], a well-known tabular GAN, by incorporating an auxiliary classifier [Park et al., 2018; Zhao et al., 2022] within its architecture to generate high-quality samples conditioned on the corresponding target variable. Since the transformed feature space has high dimensionality (over 600), we also embed a global feature selection mechanism while training the auxiliary classifier's by employing the novel approach from [Margeloiu et al., 2023] that was shown to perform well on even smaller and higher dimensional data than ours. Finally, to enforce the skip constraints stated in (i), we take advantage of CTGAN's built-in conditional vector.
• (III) We implement and train the proposed GAN and use it to augment our (training) data. The augmented data is then used to train binary and multiclass classification models for predictive problems (A) and (B), respectively, for each of the following drugs: marijuana, methamphetamine, amphetamines, and cocaine. Our experimental results show that the average Area under the Receiver Operating Characteristic curve (AUROC) evaluated on multiple distinct sets of test data is improved by up to 13.4% in Problem (A) and 15.8% in Problem (B) when the data is augmented, which is significantly higher than what yielded using state-of-the-art generative models.

2 ProblemDescription

2.1 Background

Our Tabular Data. The recruitment of PWUDs started in 2019 under the respondent-driven sampling scheme [Heckathorn, 2014] in the Great Plains of the U.S. and has continued to the present time. Enrolled PWUDs were followed up within 4-12 months after their initial visit and took the same survey as before. In the survey, each PWUD answered questions on a computer regarding their individual attributes, including the drug use behavior of 18 different drugs. Use frequencies of considered drugs (e.g., marijuana, cocaine, amphetamines, and methamphetamine) were inquired on an ordinal scale (1-8) of {never, less than once a month, once a month, once a week, 2-6 times a week, once a day,

2-3 times a day, 4 or more times a day}. Collectively, the responses to these questions form the (raw) features stored in a 2-D table.There are 258 samples (each represented as a row) and 151 features (each represented as a column) in total. (See supplementary material for the survey questions.) After being preprocessed (detailed in Section 4.1), the table contains 209 features (plus the target variable for teaching the desired classification models): 2 continuous and 207 categorical.

Definition 1 (Skip Logic). In survey design, skip logic is a set of automated navigational rules that allows respondents to skip to relevant questions depending on their prior answers [Couper, 2008]. Each of the rules is defined as a skip constraint, which restricts the possible values of a subset of features A in accordance with the value of some feature a. We say the skip constraint on the chain A is imposed by a and denote this as a → A. For the example in Figure 1, the skip constraint on A = {TB4} is imposed by a = TB3 i.e., TB3 → {TB4}, in which TB4 is omissible when TB3="No".

Definition 2 (Tabular Data Generation). Following the notations from [Xu et al., 2019], given a table Treal that is partitioned row-wise into training set Ttrain and test set Ttest, the task involves training a data generator G on Ttrain and then independently sampling rows using the learned G to generate a synthetic table Tsyn such that |Tsyn| = |Ttrain| with similar probability mass function for some target variable, y (defined in Section 2.2) (in order for us to fairly evaluate the efficacy of G).

Note that a capable G can satisfy the latter requirement without affecting the overall quality of the generated synthetic samples-that is, the features in each generated row should be consistent with the label in that row. For our setting, Treal and its derivatives consist of Nc = 2 continuous features and Nd = 207 + 1 categorical features.

Definition 3 (GAN and Its Extensions). GANs are deep generative models that have recently found success in modeling tabular data [Borisov et al., 2022] in addition to images and text. A GAN typically consists of two separate networks: a generator G that maps a noise distribution (typically Gaussian) to the data distribution and a discriminator D that estimates the probability an input sample came from the data distribution. The learning process is defined as an adversarial game between G and D in which G attempts to consistently fool D [Goodfellow et al., 2014].

To stabilize the training of GANs, [Arjovsky et al., 2017] introduce WGAN that provides a meaningful loss, the Wasserstein distance, for quantifying the difference between the generated and real data distributions. Using the value function from WGAN-GP [Gulrajani et al., 2017] (an improvement of WGAN), [Xu et al., 2019] design CTGAN that aims to tackle class imbalance in categorical features of tabular data by modifying G to additionally take a vector as input. This so-called conditional vector represents a certain class/category of some categorical feature and is used to condition both the generated samples and the real training samples. The model can thus efficiently learn proper conditional distributions for each feature. We further describe the main components of CTGAN in Section 3.2.

2.2 ProblemFormulation

In this work, we focus on data augmentation by designing a novel GAN to generate high-quality synthetic samples that resemble our tabular data with small sample size and skip logic incorporated. The augmented data is then used to train classifiers for predicting PWUDs' usage of a given drug within the next 12 months, including (A) whether they would increase its usage and (B) at which ordinal frequency they would use it. The former is a binary classification problem wherein PWUDs exhibiting an increase and non-increase (i.e., decrease or unchanged) in usage belong to the positive and negative class, respectively; the latter is a multiclass classification problem in which PWUDs are labeled according to their ordinal usage of the corresponding drug within 12 months after. We mainly concern with predicting usage of the most prevalent drugs (i.e., used by at least half of the PWUDs in our tabular data), which include marijuana, methamphetamine (or meth for brevity), amphetamines, and cocaine.

There are two unique properties of our tabularized survey data that impede existing generative and predictive models from achieving desirable performance: being high-dimension low-sample-size (HDLSS) and the presence of skip logic (whose impact is demonstrated in Challenges and Observations of the full paper).

3 Our Proposed GAN

3.1 Overview

We focus on GAN architectures due to their prevalence in tabular data generation literature [Borisov et al., 2022] and their convenient properties (e.g., flexibility for conditional generation) that help us approach the discussed challenges systematically. Overall, we extend the popular CTGAN (reviewed in Section 3.2) as follows:

1. Due to the large number of (categorical) features, it is difficult for the conditional generator G to learn to generate samples that are conditioned on a particular feature. Therefore, during the training of G, we raise the production of synthetic samples in order to ensure adequate training for conditional generation on a wide range of features. Furthermore, we prioritize the generation of synthetic samples that are conditioned on the empirical distributions of the target variable.
2. ii) To enhance the quality of synthetic samples conditioned on the target variable y, we incorporate an auxiliary classifier C [Park et al., 2018] into the architecture of CTGAN for learning the correlation between y and other

• features. That is, C is trained to predict the label of an input (synthetic or real) sample with y removed.
• iii) Within C, which is originally a multilayer perceptron (MLP), we add two auxiliary networks that together perform global feature selection and compute the weights of the first hidden layer of C [Margeloiu et al., 2023]. The objectives of this addition are twofold: facilitate effective learning by C on HDLSS data and provide a mean to approximate the feature importance scores.
• iv) Lastly, we enforce skip logic on synthetic samples during training by leveraging CTGAN's conditional vector.

3.2 Conditional Generator

We first revisit the two main components of CTGAN, the conditional generator and the training-by-sampling procedure. Formally, the conditional generator takes a vector cond as additional input, which represents the condition (Di∗ = k∗ ) for some category k∗ of the i∗ th categorical feature Di∗ ∈ {D1, . . . ,DNd } (∗ denotes the selected feature/category in cond). It is worth noting that each condition only involves one category of one categorical feature and does not involve any continuous features. The sampling of cond is twofold: sampling for the i∗ th categorical feature and sampling for the k∗ th category of that feature. Let ⊕ denote vector concatenation e.g., given x1 = [0, 0, 0] and x2 = [1, 0], x1 ⊕ x2 = [0, 0, 0, 1, 0]. If each categorical feature Di is encoded as a one-hot vector di = [d(1) i , . . . , d(k) i , . . . , d(|Di|) i ], where |Di| is the number of categories in Di and d(k) i ∈ {0, 1}, then cond is defined as cond = m1 ⊕ . . . ⊕ mNd where mi = [m(1) i , . . . ,m(k) i , . . . ,m(|Di|) i ] is the mask vector of zeros associated with di with m(k) i = 1 at i = i∗ and k = k∗ . To ensure the generator produces samples in accordance with the given conditions, a cross-entropy term measuring the difference between mi∗ from the input cond and the generated (denoted by ˆ) one-hot feature ˆdi∗ is added to its loss.

To help the model evenly explore all possible categories in categorical features, a procedure for sampling the cond vector, termed training-by-sampling, is employed in CTGAN as follows: randomly choose a categorical feature Di∗ with uniform probability; construct the probability mass function across the categories available in Di∗ by taking the logarithm of their frequencies in that feature (with respect to all training examples in Ttrain); then sample a category k∗ accordingly and calculate the corresponding mi∗ and cond. Afterward, the cond vector is used to condition both synthetic and real training samples in order for the discriminator to properly estimate the (Wasserstein) distance between the learned and real conditional distributions PG(row|cond) and P(row|cond), respectively.

We design our GAN to place more emphasis on generating samples conditioned on y by leveraging the conditional generator (discussed in the following paragraph) as well as other known techniques (discussed in subsequent subsections).

Proper Conditional Generation in HDLSS Setting. In each iteration of CTGAN, cond is sampled twice during the respective training of the discriminator D and the generator G, with sample size equal the specified batch size for both. For our HDLSS setting with |Ttrain| < 200 and over 600 columns/available conditions to consider, even when the batch size is maximally set to |Ttrain|, any condition (Di∗ = k∗ ) would either be missing or inadequately sampled in the minibatch, and hence it would be impossible for G to properly learn to produce samples that preserve the input conditions. Therefore, we increase the sample size of cond and hence the number of synthetic samples to generate during the training of G by a factor of q > 1 with respect to the batch size (q = 1 in CTGAN). Moreover, instead of randomly sampling Di∗ while constructing cond, we want G to sample certain features more frequently than others in order to prioritize learning their conditional distributions, particularly for the target variable y since our main purpose of generating synthetic samples is to help classification models improve their prediction on y. Therefore, we dedicate a portion of cond to conditions solely for y. We discuss on how we sample the remaining Di∗'s in Section 3.4.

3.3 Auxiliary Classifier

Incorporating an auxiliary classifier C into GAN architecture has been shown to improve conditional generation quality for both image and tabular data [Odena et al., 2017; Park et al., 2018]. C takes either a synthetic or a real sample having its label removed as input and aims to predict that label. Its loss, termed classification loss [Park et al., 2018], quantifies the discrepancy between the label of a real sample and the label predicted by C for that same sample, which is formulated as binary and categorical cross entropy for Problems (A) and (B), respectively. The addition of C also introduces an extra loss term into the loss function of G, which has the same form as the classification loss but concerns synthetic samples instead. We refer to this loss as downstream loss as in [Zhao et al., 2022]. Altogether, C is trained to learn the actual correlation between the label and the features, then teach G how to generate realistic samples accordingly.

The synthetic samples that are fed to C are conditioned via the cond vector that is sampled earlier during the training of G (we iteratively train D ▶ G ▶ C). Since we are mainly concerned with learning the conditional distribution for the target variable PG(row|y), we only feed synthetic samples that were conditioned on y to C when computing the downstream loss. The proper conditional generation step in Section 3.2 ensures that we have sufficient amount of such samples for training C.

3.4 Learning Important Features

The auxiliary classifier C was originally proposed to be an MLP having the same architecture as D [Park et al., 2018]. On HDLSS data, however, C is very likely to overfit, especially during the first few iterations and epochs when C encounters few training (either real or synthetic) examples, which in turn would negatively impact G. Recently, [Margeloiu et al., 2023] propose a way to overcome overfitting on HDLSS tabular classification problem by adding two auxiliary networks before the first hidden layer of some classification neural network in order to reduce the number of its learnable parameters and simultaneously perform global feature selection. We generalize this idea by integrating these networks into C to prevent it from overfitting. Furthermore, by leveraging the global importance scores s = [s1, . . . , sN] ∈ (0, 1)N (higher indicates greater importance) for N features, which are learned by one of the auxiliary networks, we can inform the conditional generator of important categorical features during training-by-sampling. More specifically, while sampling the cond vector, we sample features Di (excluding the target variable y) each with probability proportional to its important score si. Hence, conditions for features that have significant effects on the prediction of y are sampled more frequently, allowing G to prioritize generating synthetic samples conditioned on those features.

3.5 Enforcing Skip Logic

We first provide an intuitive explanation of how enforcing skip logic on synthetic samples can benefit the training of our GAN. Recall that CTGAN attempts to minimize the Wasserstein distance between PG(row|cond) and P(row|cond), where cond represents some condition (Di∗ = k∗ ) as defined earlier in Section 3.2. Let cond1 and cond2 be two distinct samples of cond with different sampled features e.g., D1 (i∗ = 1) and D2 (i∗ = 2), respectively (and hence different sampled categories). With the presence of skip logic, if D1 and D2 both belong to the same chain imposed by another feature D3 and the corresponding skip constraint D3 → {D1,D2} is enforced such that both D1,D2 are omissible, then cond1 and cond2 must be redefined to match the same conditional vector, cond∗ , that satisfies such constraint i.e., (D1 = [BLANK]) AND (D2 = [BLANK]). It follows that on real data, P(row|cond1) = P(row|cond2) = P(row|cond∗ ). This implies that the corresponding synthetic samples associated with either condition should follow the same distribution PG(row|cond∗ ). Therefore, enforcing skip logic by inferring cond∗ effectively reduces the search space for PG, which leads to more efficient and stable learning and hence more consistency in the quality of the generated samples. We empirically demonstrate this claim in Section 4.2.

Existing methods for enforcing column-wise constraints require either creating customized transformation functions coupled with validity check or using reject sampling [Patki et al., 2016], both of which are ad hoc and highly inefficient for large number of constrained columns. Instead, we leverage the cond vector to enforce our constraints. Formally, recall from Section 3.2 that cond is constructed as ⊕Nd i=1 mi with m(k∗ ) i∗ set to 1 and all other entries set to 0 for representing the condition (Di∗ = k∗ ). Let us assume the corresponding categorical feature Di∗ from cond imposes some skip constraint κ (e.g., Di∗ = TB3 and k∗ = "No") on a chain of features {Di′}i′∈M, where M ⊆ {1, . . . ,Nd} such that |M| is the size of such chain. Let k′i′ ∈ {1, . . . , |Di′ |} be a specific category that each Di′ takes in accordance with κ (e.g., Di′ = TB4 and k′i′ = [BLANK] under TB3="No"). Then, we can define κ as the extension of a condition as follows:

Therefore, whenever applicable, we can restrict cond κ-→ ζ(cond) by reconstructing the individual mask vectors of cond (with a slight abuse of notation) as

Note that κ is only defined for a fixed set of {k∗ , k′i′} and hence can take various forms in practice. For instance, the following is an exhaustive list of valid expressions for κ = TB3 → {TB4}:

• TB3="No"⇒TB4=[BLANK] (omissible)
• TB3="Yes"⇒TB4="Not at all"
• TB3="Yes" ⇒ TB4="Less than 1 cigarettes a day"
• TB3="Yes" ⇒ TB4="1-5 cigarettes a day"
• TB3="Yes" ⇒ TB4="Half a pack a day"
• TB3="Yes" ⇒ TB4="A pack or more a day".

3.6 The Complete Model

The complete training procedure via minibatch stochastic gradient descent (SGD) is summarized in Algorithm 1. q was previously defined in Section 3.2. We denote the conditions for y as cond(y) . ω ∈ [0, 1] is the ratio for controlling the prevalence of such conditions in some sample of cond. Let I be the probability mass function across the categorical features (excluding y) wherein the probability for selecting a feature Di∗ is defined as its normalized feature importance score: si∗/ ∑Nd-1 i=1 si (-1 for excluding y). The remaining conditions in cond are sampled following the trainingby-sampling procedure but with features selected according to I (the categories in each categorical feature are still sampled according to their log probabilities in Ttrain as before), for which we express as condj ∼ I with a slight abuse of notation. The loss for D, LD , is defined similarly as in CTGAN i.e., WGAN-GP loss. The loss function for G is LG = LGorig +LGdstream where LGorig is the original loss function for G in CTGAN and LGdstream is the downstream loss defined in Section 3.3 along with the classification loss LC .

At each iteration of an epoch, the training sequence is as follows: train the discriminator (lines 3-6), train the generator (lines 7-10), and train the auxiliary classifier while simultaneously updating I according to s (lines 11-13).

4 Experiments

4.1 Methodology

Implementation Details

All experiments were conducted using PyTorch 1.13.1, CUDA 11.7, and scikit-learn 1.3.2. Our implementation of the proposed GAN 1 is based on CTGAN's. We refer readers to our full paper for further implementation details, particularly on our data preprocessing step.

Hyperparameters. For all considered generative models, unless otherwise stated, we adopt the same specifications as in the cited original work. We use a batch size of |B| = 30 to train each model, For CTGAN, we set the pac size [Lin et al., 2018] to 3. C is a 2-layer MLP with (256, 256) neurons

1 https://github.com/AnonyMouse3005/HDLSS-GAN

Algorithm 1 Training Our Proposed GAN

Input: Discriminator D, generator G, auxiliary classifier C, constants q and ω 1: Initialize I uniformly 2: for each minibatch B of equal size fromTtrain do 3: Sample noise Z and cond (for training D) Z = [z1, . . . , z|B|] with zj ∼ N(0, 1) cond = [cond1, . . . , cond(1-ω)∗|B|] with condj ∼ I 4: cond ← ζ(cond) ⊕ [cond(y) ]×ω|B| 5: Sample |B| rows fromTtrain such that each row rj satisfies condj 6: Perform SGD update of Dwith LD (G(Z, cond),B) 7: Sample noise Z and cond (for training G and C) Z = [z1, . . . , zq∗|B|] with zj ∼ N(0, 1) cond = [cond1, . . . , condq∗(1-ω)∗|B|] with condj ∼ I 8: cond ← ζ(cond) ⊕ [cond(y) ]×qω|B| 9: Perform SGD update of G with LGorig (G(Z, cond)) 10: Perform SGD update of G with LGdstream (C ({G(Z, condj )}condj ∈cond(y) )) 11: Obtain s by computing C(B) 12: Update I according to s 13: Perform SGD update of C with LC (C(B)) 14: end for Return: Trained G

and either a sigmoid (for Problem A) or a softmax (for Problem B) activation in the last layer. Each of the two attached auxiliary networks prior to the first hidden layer of C is a 4layer MLP (256, 256, 256, 256) with a tanh and a sigmoid activation in the last layer, respectively. We manually tuned q and ω (via 10-fold cross validation on Ttrain) before fixing their values to 20 and 0.5, respectively. Each model is trained for 100 epochs, each contains ⌊|Ttrain|/|B|⌋ iterations. The ratio for partitioning Treal is 80 : 20 for Ttrain and Ttest, respectively, with a total of 100 distinct seeds.

Evaluation Metrics and Framework

Figure 2 illustrates our evaluation framework. We evaluate the efficacy of generative models 2 using three criteria: conflict, compatibility [Park et al., 2018], and utility.

Conflict. Every row of a synthetic table generated by G should not contain too many entries that violate skip logic. Given the jth row of Tsyn that is represented as ˆrj = ˆCj⊕ˆd1,j⊕. . .⊕ˆdNd,j where ˆdi,j is the one-hot vector of the ith categorical feature and ˆCj is the representation of continuous features in that row 3 , we check for each skip constraint κ whether the columns of ˆrj satisfy the condition (Di∗ = k∗ ) i.e., match cond (left-hand side of Equation 1). If it does, the one-hot vectors of the features within the chain linked by κ must exactly match the mask vectors associated with said features in the restricted cond vector, ζ(mi) (whose construction is defined in Equation 2). Hence, we quantify the degree of κ-violation by computing the Hamming distance between the vectors (ˆri,j)i∈M = ⊕i∈M ˆdi,j and ⊕i∈M ζ(mi), where M contains the indices for the features within the chain linked by κ. The conflict metric of a single row ˆrj is the average Hamming distance across all applicable skip constraints, and we compute the conflict of Tsyn by taking its average across all rows. Thus, a synthetic table whose rows adequately conform skip logic yields a low conflict score in [0, 1].

2 For every measure of Tsyn's quality while evaluating some trained generator G, we use G to generate 10 samples of Tsyn (each satisfies the two requirements specified in Definition 2) and evaluate their quality independently, then average the respective scores.

3 varied across different generative models e.g., CTGAN uses the proposed mode-specific normalization (and so does our GAN)

Compatibility. The classification models trained on synthetic data should output prediction for unseen examples in test data as accurately as those trained on real (training) data. We train classification models on Ttrain and on Tsyn (both having the same size as defined in Section 2.1), then test them using Ttest and compare their predictive performance, which is measured using the standard Area under the Receiver Operating Characteristic curve (AUROC). We train each of the multiclass classification models in Problem (B) using the onevs-all strategy and we compute the resulting AUROC also using the one-vs-all strategy [Fawcett, 2006].

We report the average difference in AUROC (across different classification models [Xu et al., 2019] and different partitioning of Ttrain and Ttest) of models trained on Tsyn and those trained on Ttrain. It is expected for the classification models trained on Tsyn to score lower AUROC than those trained on Ttrain, ideally with a margin as small as possible. Therefore, the compatibility score of Tsyn should be negative and close to zero. We consider the following classifiers same for both Problems (A) and (B): elastic-net logistic regression [Zou and Hastie, 2005], decision tree i.e., CART [Breiman, 2001], random forest [Ho, 1995], XGBoost [Chen and Guestrin, 2016], CatBoost [Prokhorenkova et al., 2018], 3-layer MLP (100, 100, 10) with sigmoid/softmax activation in the last layer, and WPFS [Margeloiu et al., 2023].

Utility. Ultimately, when the real training data is augmented by synthetic samples, the classification models trained on it should excel compared to the models trained on real data only. We report the average AUROC (across different classification models and different partitioning of Ttrain and Ttest) of the models trained on the augmented data, Ttrain +Tsyn, and compare it against the average AUROC of those trained on Ttrain. We consider the same set of classifiers listed above. Unlike compatibility, we do not compute the difference in AUROC scores. Hence, the utility of Tsyn follows the same scale as AUROC ∈ [0, 1] and should ideally yield values as high as possible.

Figure 3: Efficacy of considered generative models in Problem (A) (top) and Problem (B) (bottom). Each column considers one drug. The red dashed lines in Figure 3b mark the average AUROC of classification models trained on Ttrain without data augmentation.

4.2 Results

We use our evaluation framework to evaluate CTGAN, TVAE [Xu et al., 2019], TableGAN [Park et al., 2018], CTABGAN+ [Zhao et al., 2022], and TabDDPM [Kotelnikov et al., 2023] in addition to the proposed GAN, which are considered stateof-the-art for tabular data generation. For baseline, we use Bayesian networks [Pearl, 2011].

Compatibility of Synthetic Data. Figure 3a shows the performance of all considered generative models in terms of compatibility. Across all considered drugs in both problems, we see that the drop in predictive performance of classifications models trained on synthetic data that are generated from our GAN (<0.05 and <0.07 lower in AUROC for Problems A and B, respectively) is considerably smaller with lower uncertainty compared to the performance drop of those trained on synthetic data from other generative models. This demonstrates the efficacy of our GAN in generating synthetic samples that are comparable to real training data.

Utility of Synthetic Data. In terms of utility, the proposed GAN also performs well in each problem relative to the benchmarks as shown in Figure 3b. More specifically, the classification models trained using augmented data from our GAN gain from 8.35% up to 13.4% in AUROC in Problem (A) and 8.66% up to 15.8% in Problem (B), whereas those trained using augmented data from other models barely show any improvement. This implies that our GAN is capable of augmenting existing HDLSS data in order for classification models to effectively improve their predictive performance.

Violation of Skip Logic in Synthetic Data. The evaluation on degree of skip logic violation is summarized in Table 1. Since our work is the first to take this criteria into account, our proposed GAN has a clear advantage over existing models, with Tsyn consistently having lower average conflict by a large margin in both considered problems. The additional runtime for enforcing all 26 skip constraints is negligible i.e., for approximately 5% longer training time.

Impact of Enforcing Skip Logic. We also perform an ablation study to understand the practical benefits of enforcing skip logic during our GAN training. As shown in Table 2, when we enforce skip logic, the training phase exhibits not only higher stability but also higher efficiency, as evidenced by the loss of G at 50 and 100 epochs. Note that the adoption of WGAN (via CTGAN) in our GAN allows us to interpret the loss in a meaningful way. As a result, the trained GAN is able to consistently generate high-quality samples, which positively affects the scores for all three evaluation metrics to some extent. Similar results for other considered problems can be found in the appendix of our full paper.

5 Conclusion

In this paper, using HDLSS tabular data collected by our team via a survey that employs skip logic on short-term substance use behavior, we design a novel GAN for augmenting our limited tabular data in order to help classification models accurately predict short-term substance use behaviors of PWUDs: (A) whether they would increase usage of a certain drug and (B) at which ordinal frequency they would use it within the next 12 months. Our evaluation results demonstrate the efficacy of the proposed GAN. The resulting predictions for the two defined problems can ultimately be leveraged by relevant substance use organizations as a complementary forecasting tool when determining the most appropriate resource to allocate to PWUDs that need the most help.

Acknowledgements

This project is supported by the National Institute of General Medical Sciences of the National Institutes of Health [P20GM130461], the Rural Drug Addiction Research Center at the University of Nebraska-Lincoln, and the National Science Foundation under grant IIS:RI #2302999. The content is solely the responsibility of the authors and does not necessarily represent the official views of the funding agencies.

Ethics Statement

Due to the confidentiality agreement and IRB approval for this study, we do not have access to sensitive information such as full name and gender identity. Data will be made available to applicants (with an IRB protocol and ethical research plan) upon request.