FADE: Towards Fairness-aware Data Generation for Domain Generalization via Classifier-Guided Score-based Diffusion Models

1 Introduction

In recent years, fairness in machine learning has received widespread attention due to its significance in real-world systems such as loan approvals [Purificato et al., 2023] and employee hiring [Alder and Gilbert, 2006]. From a group perspective, algorithmic fairness focuses on the statistical parity among different groups defined by specific sensitive attributes of people [Hardt et al., 2016]. Many efforts have been made to address algorithmic fairness [Mitchell et al., 2021]. A type of method achieves fairness by eliminating bias from the dataset through fair data generation. As a pioneer in this approach, FGAN [Xu et al., 2018] removes sensitive information by ensuring that the discriminator cannot distinguish the sensitive group membership of the generated samples. Due to the powerful generative capabilities, diffusion models have also been employed to ensure the fairness of generated data. FLDGM [Ramachandranpillai et al., 2023] integrate an existing debiasing method with diffusion models to generate unbiased data. Although they have achieved remarkable results, their poor generalization performance has caused them to fail when facing distribution shifts [Shao et al., 2024].

Fairness-aware domain generalization (FairDG), as an emerging topic, has gained increasing attention because it is essential for advancing the deployment of trustworthy AI in the real world [Shao et al., 2024]. Instead of the independent and identically distributed (i.i.d.) assumption, this field focuses on fairness under distribution shifts. The most widely studied distribution shift is covariate shift [Shimodaira, 2000], which variations to changes caused by differences in the marginal distributions of instances. One class of methods addresses the FairDG issue by learning fair and domain-invariant representations through feature disentanglement [Oh et al., 2022]. However, these methods often have overly strong assumptions that it can perfectly disentangle sensitive information and domain-specific information from the feature representation. The assumptions limit their applicability and reduce their robustness. Another class of methods enhances the diversity of the training set through data augmentation [Pham et al., 2023], enabling the model to generalize better to unseen target domains and thereby improving overall performance.

In this paper, inspired by data augmentation strategies, we propose a three-stage framework called FADE: Fairnessaware Classifier-Guided Score-based Diffusion Models to generate unbiased and domain-invariant data. Specifically, we pre-train the parameters of the score-based diffusion model (SDM), label classifier, and sensitive classifier using a source dataset across multiple domains in the first stage. This stage equips the three components with generalization capabilities in the target domain. In the second stage, we derive a generator by guiding the SDM with the pre-trained classifiers, which effectively removes bias or sensitive information from the generated data. Simultaneously, during the generation phase, we can obtain fair data without requiring any additional training. Finally, we use the generated data, which has reduced sensitive information, to train a fairnessaware domain-invariant downstream classifier. Our contributions can be summarized as follows:

• We formulate a novel problem: generating unbiased data to train downstream classifiers that are tested on distribution-shifted datasets, while ensuring both accuracy and fairness.
• We have designed a novel fair data generation method called FADE, which ensures the generalization ability of the generative model during the pre-training phase and removes biased information during the generation phase. FADE not only allows for the specification of generated sample categories but also possesses generalization capabilities under new data distribution shifts.
• Experimental results on three real-world datasets demonstrate that FADE achieves the best performance in both fairness and accuracy compared to other baselines when facing the challenge of distribution shifts.

2 RelatedWork

Fair Data Generation. Fairness in machine learning aims to ensure equitable performance across different demographic groups, and it can be achieved through three primary approaches: pre-processing, in-processing, and post-processing methods. Pre-processing methods modify the training data to mitigate biases before training the model, using techniques such as data resampling, data transformation, and fair data generation. Fair data generation is similar to pre-processing methods. However, unlike pre-processing methods, fair data generation does not use the original data to train downstream classifiers. Instead, it can generate additional data for training predictive models, which is especially beneficial when the original training data is very limited. FGAN [Xu et al., 2018] is the first method to tackle fair data generation. Also based on GANs [Goodfellow et al., 2020], DECAF [Van Breugel et al., 2021] can achieve various fairness criteria by leveraging causal graphs. FLDGM [Ramachandranpillai et al., 2023] attempts to integrate an existing debiasing method [Liu et al., 2022] with GANs or diffusion models [Ho et al., 2020] to achieve fair data generation.

Fairness under Distribution Shifts. Achieving fairness is not devoid of challenges, especially in the presence of distribution shifts. These shifts can pose significant hurdles as models trained on source distributions may not generalize well to target data distributions, potentially exacerbating biases and undermining the intended fairness objectives [Lin et al., 2024; Komanduri et al., 2024; Lin et al., 2023a; Lin et al., 2023b; Zhao and Chen, 2019]. There are two primary approaches to addressing fairness issues across domains: feature disentanglement and data augmentation. Feature disentanglement aims to learn latent representations of data features, enhancing their clarity and mutual independence within the model [Zhao et al., 2024; Lian et al., 2025; Jiang et al., 2024; Li et al., 2024; Zhao et al., 2023]. However, these methods frequently rely on overly strong assumptions, which restrict their applicability and diminish their robustness. Additionally, assessing the quality of disentanglement presents its own challenges. Data augmentation seeks to enhance the diversity of training datasets and improve model generalization performance by systematically applying controlled transformations to the training data [Pham et al., 2023; Zhao et al., 2024].

3 Background

Notations. Let X ⊆ Rp denotes a feature space and Z = {0, 1} is a sensitive space. Y = {0, 1} is defined as an output or a label space. A domain e ∈ E is defined as a joint distribution PeXZY := P(Xe ,Z e , Y e ) on H = X ×Z ×Y. A dataset i.i.d. sampled from a domain e is represented as De = {(xej , ze j , ye j )}|De | j=1 , where x, z, y are the realizations of random variables X, Z, Y in the corresponding spaces. A function f parameterized by θ is denoted as fθ. We defer a list of the notations used in this paper in Appendix A.

Fairness-aware Domain Generalization (FairDG). In the supervised learning setting of FairDG [Shao et al., 2024], the goal is to lean a classifier fθ : X → Y across multiple source domains E s = {ei}Si=1 , where S = |E s |, such that the learned fθ can be applied on a distinct shifted target domain et /∈ E s , which is unknown and inaccessible during training, achieving both good generalization accuracy and fair decision-making.

To ensure fair predictions, the classifier fθ is required to guarantee that the outcomes ˆ Y = fθ(X) are not biased or discriminatory against any specific protected groups, each characterized by sensitive attributes Z, such as race and gender. Specifically, a fairness-aware metric ωθ : Z×Y → R, such as the difference of demographic parity [Dwork et al., 2012] or the difference of equalized odds [Hardt et al., 2016], is used to control the fair dependence between the sensitive attribute and model outcomes during training. A value of ωθ(Z, ˆ Y ) closer to 0 indicates greater fairness.

Problem 1 (FairDG). Given a source dataset Dsrc = {Dei }Si=1 , where each Dei = {(xei j , zei j , yei j )}|Dei | j=1 is i.i.d. sampled from a unique source domain ei ∈ E s , a loss function ℓ : Y × Y → R, and a fair metric ωθ : Z × Y → R, the goal is to learn a fair classifier fθ : X → Y that minimized the risk over Dsrc satisfying a fairness constraint:

where ϵ > 0 is an empirical threshold. The learned fθ is required to be generalizable to a target domain et /∈ E s , which is unknown and inaccessible during training, such that it performs well in terms of predicted accuracy and fairness on Dtgt = {(xet j , zet j , yet j )}|Dtgt | j=1 sampled from this domain.

A key challenge in addressing Problem 1 is determining how closely the data distributions in unknown target domains align with those in the observed source domains. Given the various distribution shifts discussed in [Shao et al., 2024], this paper narrows the scope by focusing specifically on the shift between source and target domains that is solely due to covariate shift [Shimodaira, 2000], in which domain variation is attributed to disparities in the marginal distributions over input features PeiX ̸= Pet X , ∀ei ∈ E s .

Inspired by existing domain generalization efforts [Zhou et al., 2022], data augmentation strategies, such as diffusion models [Ho et al., 2020], applied to source domains enhance the diversity of the training data, thereby improving the ability of fθ to generalize to unseen target domains and ensuring domain invariance.

Score-based Diffusion Models (SDMs). Denoising diffusion probabilistic models (DDPMs) [Ho et al., 2020] have demonstrated remarkable performance in generating highfidelity data. The forward process in DDPMs gradually adds Gaussian noise to the data over a series of timesteps, eventually transforming the data into pure noise. Let x0 be the original data, and xi>0, i = 1, · · · N be the noisy data after i steps. The forward process is defined as:

where noise ui-1 ∼ N(0, I), βi is a hyperparameter to control the intensity of noise addition and N is the total number of noise addition times.

Score-based diffusion models [Song et al., 2020] are an alternative implementation of DDPMs. In the limit of N →∞, we define an auxiliary set of noise scales {βi = Nβi}Ni=1 . Let β ( iN ) = βi, {βi}Ni=1 becomes a function β(t) indexed by t. Given t ∈ [0, T], xt represents x at moment t. The forward diffusion in Eq.(1) can be defined by a stochastic differential equation (SDE) [Ho et al., 2020]:

where the function β(t) is determined by the discrete hyper parameter βt and w is the standard Wiener process [Zhang et al., 2018]. Defining h(xt) = - 12 β(t) and g(t) = √β(t). When generating samples, the corresponding reverse diffusion process can be described by the following system of

where w is the reverse-time standard Wiener processes, and dt is an infinitesimal negative time step. The score networks sθs is trained to approximate the partial score functions ∇xt log p(xt) and used to generate the sample features by evolving backward in time. Specially, the score loss function [Song et al., 2020] Ls : Θs ×H → R can be formulate:

where λ : [0, T] → R>0 is a positive weighting function, t is uniformly sampled over [0, T], x0 ∼ p(x) and xt ∼ p(xt|x0). For DDPM, we can typically choose

Problem Setting. To address the problem of FairDG, given data Dsrc = {Dei }Si=1 , where each Dei is sampled from a unique source domain ei ∈ E s , the key of this paper is to seek a generator G : X × Z × Y → X × Z × Y, initialized by a pre-trained SDM. A dataset generated by G, denoted Dgen = {x′j , z′j , y′j }|Dgen | j=1 , can then be used to learn a downstream classifier fθ by minimizing the risk and satisfying algorithmic fairness, ωθ(Z ′ , fθ(X′ )) ≤ ϵ, over Dgen .

where the sample (x′ , z′ , y′ ) ∈ Dgen is generated by G using (xei , zei , yei ). During inference, the learned fθ is expected to perform well in terms of accuracy and model fairness on Dtgt , which is sampled from a target domain et /∈ E s and is unknown and inaccessible during training.

4 Methodology

We propose a novel framework, FADE, to address Eq.(6) as shown in Figure 1. This framework has three stages. In the first stage, the parameters of the SDM, label classifier, and sensitive classifier are pre-trained using the source dataset across various domains. In the second stage, the generator G is derived by guiding the SDM using classifiers with the learned parameters. Finally, in the last stage, the data generated from G with mitigated sensitive information are used to learn a fairness-aware domain invariant downstream classifier fθ .

4.1 SDMand Classifiers Pre-training across Domains

As described in Figure 1, given the source data Dsrc consisting of S training domains, we consider K tasks. Each task Tk is associated with a support and a query set. The support set T sup k = ∪ei∈T sup k Bei,sup k contains S - 1 data batches, where each batch Bei,sup k is sampled from the corresponding Dei ∈ Dsrc and the query set T qry k = ∪ei∈T qry k Bei,qry k is sampled from the remaining distinct data subset in Dsrc .

Inspired by [Finn et al., 2017], this stage aims to learn a good parameter Φ over the K tasks that can be generalized on all source domains. The pre-training model contains three components: a SDMsθs introduced in the previous section, a label classifier ϕθy : X → Y, and a sensitive classifier ψθz : X → Z. To optimize Φ = {θs, θy, θz} across domains, the objective function is given in Eq.(7).

where

In Eq.(7), ξs, ξy, ξz > 0 are learning rates and δ ∈ [0, 1] is an empirical weight balancing support and query losses. Ls is the score loss function defined in Eq.(4). Ly : Θy×H → R and Lz : Θz ×H → R are cross-entropy loss functions for predicting class labels and sensitive attributes, respectively. Notice that θ′s,k , θ′y,k , θ′z,k are updated using one or few gradient steps for rapid optimization.

After sufficient iterations, Φ becomes optimized and invariant to domains. This occurs because each task can be viewed as a domain generalization problem within a sub-condition where the support sets act as "source data" and the query sets as "target data".

Furthermore, in the second stage of FADE, the SDM is guided to the generator G by the label and sensitive classifiers using samples in Dsrc , while keeping their parameters θs, θy, θz unchanged.

4.2 Guiding the SDM to Gwith Classifiers

Debiasing with Fair Control. To mitigate sensitive information from samples during the reverse diffusion process, we propose a novel sampling strategy. Assuming a strength signal afair ∈ [0, 1], we should sample from the conditional distribution p(xt|afair = λz), hyperparameter λz represents the strengh of fair control. Consequently, we need to solve the conditional reverse-time SDE:

Since p(xt|afair = λz) ∝ p(xt)p(afair = λz|xt), we can derive the gradient relationship (proved in Appendix B) as follows:

Therefore, the problemwe need to address is transformed into modeling p(afair = λz|xt). For a sample that does not contain sensitive information, it will be uncertain to classify it definitively into any sensitive category. In other words, for debiased samples, training a classifier to predict their sensitive subgroups will result in a predicted distribution that approaches a uniform distribution. At this point, the entropy of the sensitive attribute prediction distribution pθz (z|xt) will reach its maximum. Based on this property, we define p(afair = λz|xt) as:

where H(·) denotes the entropy function and Ct is a normalization constant. In practice, pθz (z|xt) is approximated using a pre-trained sensitive classifier ψθz . Adding the gradient of the logarithm of Eq.(11) in the reverse diffusion process corresponds to maximizing the entropy of pθz (z|xt) at each time step t. This ensures that the samples drawn at each step contain minimal sensitive information, making it difficult for the classifier ψθz to determine their sensitive category. Substituting Eq.(10) and Eq.(11) into Eq.(9) yields the final reverse SDE:

where λz is a hyperparameter that controls the guidance strength of the sensitive classifier ψθz . So far, we are able to generate samples that are free from sensitive information.

Label Generation with Classifier Guidance. To address our problem, we need to generate both the data and their labels simultaneously. Previous methods solve this by concatenating the labels with the features and then inputting them into the generative model for joint training [Xu et al., 2018; Van Breugel et al., 2021; Ramachandranpillai et al., 2023]. However, a fair control process can affect these approaches by influencing the generated labels, leading to undesired labels. To tackle this challenge, we propose to employ a label generation process with classifier guidance [Dhariwal and Nichol, 2021]. We first specify a label y and then use the SDM to model the conditional distribution p(xt|y). We approach this by sampling from the conditional distribution p(xt|y) where y represents the label condition, by solving the conditional reverse-time SDE:

dxt = [h(xt) - g2 (t)∇xt log p(x|afair, y)] dt + g(t)dw. (13) Since ∇xt log p(xt|afair, y) = ∇xt log p(afair|xt) + ∇xt log p(xt) +∇xt log p(y|xt, afair) (proved in Appendix B), we need a pre-trained classifier to simulate p(y|xt, afair). For any given sample, regardless of whether we apply fair control to it, we need to specify the generation of its label y ( y is independent of afair). Therefore, pθy (y|xt) = pθy (y|xt, afair) can be approximated using a pre-trained sensitive classifier ϕθy . we can rewrite Eq.(13) as:

where λy is a hyperparameter that controls the guidance strength of the label classifier ϕθy . We can specify labels for generating samples, rather than generating labels and samples simultaneously at random. Using this method, we can specify the same number of samples to be generated for different classes, thereby creating a balanced dataset to avoid insufficient feature information for minority classes in downstream classifiers. At this point, we can generate a dataset that does not contain sensitive information with labels.

4.3 Learning DownstreamClassifier Using the Generated Data by G

Let G be the score network θs guided by θy and θz. Given a target class y, iteratively input Gaussian noise into the generator G to output a subset of data with label y by Eq.(14). Perform the same operation for other classes as well, and then merge all these subsets into the final dataset Dgen . Our downstream training objective is to train a classifier fθ that satisfies:

where ℓ represents a classification loss function such as crossentropy.

4.4 Algorithmic Summary of FADE

The overall process of FADE is illustrated in Algorithm 1. Lines 1 to 8 represent the pre-training phase of the model parameter Φ = {θs, θy, θz}, while lines 9 and 10 respectively represent the data generation phase and the downstream classifier training. During the pre-training phase of the model, by constructing different T sup k and T qry k , the model gains the ability to generate data from non-training domains. In the data generation phase, we do not require any additional training. We can simply use the pre-trained classifiers, ϕθy and ψθz , as a guide to generate data that is free from biased information. Finally, during the downstream classifier training, only the simplest classification loss is required.

5 Experiments

Evaluation Metrics. We measure the domain generalization performance using Accuracy and evaluate the algorithmic fairness using four evaluation metrics as follows:

Algorithm 1 FADE

Input: Labeled source datasets Dsrc with S domains; score network sθs ; label classifier ϕθy ; sensitive classifier ψθz ; downstream classifier fθ; hyperparameters αs, αy , αz , βs, βy , βz , γs, γy , γz .

Initialize: θs, θy, θz, and θ

• 1: repeat
• 2: Sample K tasks {Tk}Kk=1 fromDsrc
• 4: Evaluate Lsup k using θs, θy, θz on T sup k
• 3: for each task Tk = {T sup k , T qry k } do
• 5: Update θ′s,k , θ′y,k , θ′z,k ← θs, θy, θz in Eq.(8)
• 6: Evaluate Lqry k using θ′s,k , θ′y,k , θ′z,k on T qry k
• 7: end for
• 8: Update θs, θy, θz using {Lsup
• 9: until convergence
• k ,Lqry k }Kk=1 in Eq.(7)
• 10: Derive G from sθs guided by ϕθy and ψθz and generate Dgen from G in Eq.(14)
• 11: Iteratively optimize fθ using Dgen
• Difference of Demographic Parity (∆DP ) [Dwork et al., 2012] requires that the acceptance rate provided by the algorithm should be the same across all sensitive subgroups, which is formalized as

where ˆ Y is the predicted class label.

• Difference of Equalized Odds (∆EO) [Hardt et al., 2016] examines fairness in terms of equal true positive and false positive rates, which is formalized as

• Difference of Equalized Opportunity (∆EOp) [Hardt et al., 2016] evaluates the fairness of true positive rates specifically, which is formalized as

• Difference of dataset distance (Dfair) measures the fairness of a dataset by evaluating the difference between the distances from the mean point of the generated dataset to the mean points of two sensitive groups in the original dataset. We formulally define it as:

where µ(·) denotes the operation of taking the mean point of a dataset, Dsrc z=0 and Dsrc z=1 represent different sensitive subsets within Dsrc , respectively.

The smaller the values of ∆DP , ∆EO ∆EOp and Dfair, the fairer the algorithm.

Compared Methods. We compare FADE with three classic generative models VAE [Kingma and Welling, 2013], GAN [Goodfellow et al., 2020], and DDPM [Ho et al., 2020]. Additionally, four models for fair data generation methods have also been considered: FGAN [Xu et al., 2018], DECAF [Van Breugel et al., 2021], FDisCo [Liu et al., 2022] and FLDGM [Ramachandranpillai et al., 2023].

Datasets and settings. We use three datasets: Adult [Kohavi and others, 1996], Bank [Moro et al., 2014] and NYSF [Goel et al., 2016]. Please refer to Appendix C for all datasets, network architecture, hardware environment, training and testing strategies.

6 Results

Overall Performance. The overall performance of FADE and its competing methods on Adult and Bank datasets is presented in Table 1. Due to space limitations, we have omitted the full results with standard deviations and those for NYSF dataset, including them in Appendix D.1. Focus on the average of each metric across all domains, FADE achieves the best performance in both DG and fairness on Adult datasets simultaneously. Although the accuracy of FADE on the Bank dataset is lower compared to some other methods, it achieves the most equitable results with minimal performance degradation, and its accuracy is also higher than those methods that focus on fairness. Both VAE and DDPM achieve decent classification accuracy, but due to their lack of fairness consideration, they cannot guarantee the algorithmic fairness. Although FairGAN and DECAF outperform FADE in fairness performance in some domains, their DG performance is not competitive. Overall, FADE ensures fairness while maintaining strong classification capabilities, which means that it has successfully generated domain-invariant fair data.

Analysis of Generated Dataset. We further analyze the dataset generated by FADE from a data perspective. We use linear discriminant analysis [Balakrishnama and Ganapathiraju, 1998] (LDA) to reduce the dimensionality of both the original and generated datasets to one dimension and plot their probability densities. Figure 2(a) shows the visualization result on the White domain of Adult dataset. The original data for different sensitive groups have significantly different probability density distributions and means. The mean of the data generated by FADE falls between the means of the two sensitive group data, indicating that it produces a more fair data distribution. Additionally, the distribution of the generated data covers a wider range compared to the original data, suggesting that it generates generalized data beyond the original distribution.

Figure 2: (a) LDA visualization of the original dataset and the dataset generated by FADE. The orange, green, and blue curves represent the probability densities of the original data for two sensitive groups and the generated data, respectively. The three dashed lines indicate the means. The markers below the x-axis show the onedimensional data, with red points representing the mean points. (b) Average Dfair results for all fairness-aware methods across three datasets.

Furthermore, we compare the average of Dfair across all domains between the generated datasets and the original datasets for all fairness-aware methods across three datasets, as shown in Figure 2(b). FADE achieves the best Dfair results across three datasets, indicating that FADE generates data with the smallest distance to the two original sensitive subgroups. Compared to other fairness-aware baselines, FADE produced the fairest data, further validating that the fairness guidance component enables FADE to generate unbiased data effectively. Meanwhile, we also observe that other methods do not perform consistently across different datasets, indicating that their debiasing performance is significantly influenced by the distribution of the original data. Particularly for DECAF, its performance is highly dependent on the causal structure of the original data. In contrast, FADE's consistent results demonstrate its strong robustness.

Accuracy-Fairness Trade-offs. To evaluate the accuracy-fairness trade-offs of FADE, we vary λz ∈ {0.1, 1, 10, 50, 100} to obtain different classification and fairness results. A larger λz indicates a greater emphasis on algorithmic fairness by FADE. We compare the results with other fairness-aware methods using three fairness metrics across Adult and Bank datasets, as shown in Figure 3. FADE's results across various λz are positioned in the upper-left corner compared to all other methods. This suggests that FADE not only delivers superior fairness performance but also maintains competitive classification performance, achieving the best accuracy-fairness trade-offs. Although FGAN and DECAF are comparable to FADE in terms of algorithmic fairness, their lower classification accuracy results in a poorer accuracy-fairness trade-off.

Ablation Study. To evaluate the effect of components in the design of FADE, we conduct ablation studies as shown in Table 2. The results on the NYPD dataset are provided in Appendix D.2. We propose 3 variants of FADE: (i) Without fair guidance (w/o FG). We set λz = 0 to ensure that there is no fair control during the sampling process. Despite achieving a marginal advantage in prediction accuracy on the adult and bank dataset, a sharp increase in all three fairness metrics resulted in unfair classification outcomes (Table 2). (ii) Without label guidance (w/o LG). We set λy as 0 firstly. To generate the labels for the dataset, we concatenate the labels with the features and jointly train the generator using this combined input. However, by doing so, the labels generated are influenced by fair control, leading to an increase in incorrect labels and, consequently, a decline in accuracy performance. Furthermore, our empirical observations indicate that the fairness of both datasets has also decreased to varying degrees. (iii) Without Query Set (w/o Query). Setting δ = 1 in Eq.(7), the diffusion model's sampling process may retain training domain information, which can impact the accuracy in the target domain, particularly evident in the Adult dataset. Even if the accuracy in the Bank dataset does not significantly decrease, the shift in the target distribution can hinder the sensitive classifier from accurately discriminating sensitive attributes, thereby impeding the effective application of fair control and resulting in a decline in fairness.

7 Conclusion

In this paper, we introduce FADE, a three-stage framework designed to generate unbiased and domain-invariant data for fairness-aware domain generalization. By leveraging a combination of pre-trained score-based diffusion models (SDM) and classifier guidance, FADE effectively eliminates bias during data generation without the need for additional training. Our approach not only enhances the generalization capabilities of the generated data but also ensures fairness when applied to downstream tasks, particularly in scenarios involving distribution shifts.

Contribution Statement

Yujie Lin and Dong Li contributed equally to this work and are considered co-first authors. Minglai Shao is the corresponding author.