When Fairness Meets Privacy: Exploring Privacy Threats in Fair Binary Classifiers via Membership Inference Attacks

1 Introduction

Imbalanced datasets often induce spurious correlations between learning targets and sensitive attributes [Mehrabi et al., 2021], which will lead to biased predictions in trained (biased) models. In response, fairness research has developed in-processing approaches [Wang et al., 2022; ChingYao Chuang, 2021] that are applied during training to produce fairness-enhanced (fair) models. By effectively suppressing these spurious correlations, fair models can mitigate discriminatory predictions. However, despite promising to enhance fairness, these interventions might incur potential privacy risks, such as unintended training data memorization.

∗ corresponding author

(b) Attack success rates

Figure 1: Fairness interventions (a) increase the losses and (b) decrease attack success rates for most training samples. We generate the plots with 100 runs and report the training samples' mean loss value and mean attack success rate.

Membership inference attacks (MIAs) are widely adopted for assessing privacy risks in models that are deployed via Machine Learning as a Service (MLaaS) [Shokri et al., 2017]. Building upon this, prior work [Chang and Shokri, 2021] has primarily explored the privacy impact of fairness interventions on decision tree models by applying existing scorebased attacks to them. However, this leaves open questions on evaluating privacy risks for neural networks, such as binary classifiers, which are prevalent in fairness studies. Moreover, evaluations relying on one attack method might not fully characterize the privacy impact. Given the importance of trustworthiness in fairness studies, conducting thorough evaluations of these fairness interventions is non-trivial. To bridge these gaps, we apply different existing MIAs to binary classifiers and discover an innovative attack method designed for fair models, comprehensively evaluating the privacy risk.

With existing attack methods, our evaluation results show that fairness-enhanced models show more resilience to current MIAs than their biased counterparts. Figure 1 illustrates the results of score-based attacks on fair and biased models over 100 runs. We present the loss values (Figure 1a) and the attack success rate (Figure 1b) per training sample. After applying fairness interventions, the plots show increased loss values yet decreased attack success rates for most data points. This indicates that these interventions can lead to less successful attacks with existing MIA approaches.

Before rushing to the conclusion that fairness interventions are privacy-friendly to binary classifiers, with further analyses, we find that these methods are inefficient in attacking binary classifiers. This is because the trained attack models degrade into simple threshold-based decisions due to the binary outputs. The degradation incurs substantial performance trade-offs: while effective at recognizing member data, attack models struggle with non-member data, especially for hard examples where predictions are similar across groups.

During the evaluation of existing attacks, we have uncovered a potential threat that could enable more effective attacks on binary classifiers. Specifically, we find the prediction scores for member and non-member data exhibit divergent behaviors after fairness interventions: the scores typically increase for the majority of member data, while they conform to a normal distribution for non-member data. This disparity creates a pronounced prediction gap between groups, which is overlooked by current attacks. The widened gap, if exploited by adversaries, could enable more successful attacks, thereby posing substantial privacy threats.

Inspired by these observations, we name the identified threat as Fairness Discrepancy MIAs (FD-MIA), which targets fair models by exploiting the prediction gaps between the original (biased) and fairness-enhanced (fair) models. The key intuition is that these gaps leak membership information about the training data, which can be leveraged to launch more effective attacks. It can be integrated with the existing frameworks of score-based [Liu et al., 2022] and referencebased attacks [Carlini et al., 2022].

We conduct comprehensive evaluations across six datasets, using three attack methods and five in-processing fairness approaches. This amounts to 32 different experimental settings and over 160 distinct models. Our results reveal that fairness interventions potentially introduce new threats to model privacy, advocating for a more comprehensive examination of their potential security defects before deployment. Our main contributions are as follows: (1) To the best of our knowledge, this is the first work to comprehensively study the impact of fairness interventions on privacy through the lens of MIAs, targeting deep classifiers with real-world datasets. (2) We reveal that fairness interventions do not compromise model privacy with existing attack methods, primarily due to their limited efficacy in attacking binary classifiers. (3) We discover a novel attack method, FD-MIA, which poses significant threats to model privacy by exploiting prediction gaps from both biased and fair models. It can be integrated into existing attack frameworks. (4) Extensive experiments validate our findings and demonstrate FD-MIA's effectiveness.

2 RelatedWork

Algorithmic fairness. Fairness methods are typically categorized into pre-processing, in-processing, and postprocessing approaches based on their processing stage. We focus on in-processing approaches as they modify model training procedures directly and may introduce model privacy threats. Among them, some introduce fair constraints and formulate the issues as optimization problems [Zemel et al., 2013; Manisha and Gujar, 2020; Tang et al., 2023; Truong et al., 2023; Cruz et al., 2023; Jung et al., 2023]. Some propose adversarial designs to remove sensitive information among extracted features [Kim et al., 2019; Zhu et al., 2021; Creager et al., 2019; Park et al., 2021]. Some adopt data sampling [Roh et al., 2021] or reweighting [Chai and Wang, 2022] approaches to alleviate the biased predictions. More recently, studies learn fair representations using mixup augmentations or contrastive learning mechanism [ChingYao Chuang, 2021; Du et al., 2021; Park et al., 2022; Wang et al., 2022; Zhang et al., 2023; Qi et al., 2022]. These methods interpolate inputs or modify features to pursue fair representations.

Membership inference attacks. MIAs aim to determine if a data sample was part of a target model's training set. Some attacks leverage the target model's direct output, such as confidence scores [Shokri et al., 2017], losses [Sablayrolles et al., 2019], prediction labels [Choquette-Choo et al., 2021]. Some improve the performance by modeling the prediction distributions of the target model, such as reference models [Carlini et al., 2022; Ye et al., 2022]. Others extend their focus into new settings [Gao et al., 2023; Yuan and Zhang, 2022] or work on defense methods [Yang et al., 2023]. This work considers two representative attack approaches: score-based [Liu et al., 2022] and referencebased [Carlini et al., 2022] attack methods. More recently, studies have enhanced attack performance by exploiting additional information: some [He et al., 2022] leverage predictions from multiple augmented views, some [Li et al., 2022] require results from multi-exit models, and others [Hu et al., 2022] work on multi-modality models. We explore models with fairness discrepancies.

Exploring privacy impacts of fair models. Privacy evaluations of fair models remain under-explored. Prior study [Chang and Shokri, 2021] provides a preliminary investigation by applying prevalent score-based MIAs to assess the privacy of fair constraint methods on decision tree models. It reveals that fair decision trees enable more successful attacks. Our study extends the scope of analysis to neural networks. We conduct comprehensive evaluations across different fairness approaches on deep classifiers with multiple attack methods.

3 Preliminaries

Algorithmic fairness. Given biased models, we consider a sensitive attribute s ∈ S with subgroups {s0, s1} of binary attribute values {0, 1}. Due to imbalanced training data, trained models often exhibit biased predictions, where predictions ˜ Y become spuriously correlated with the attribute s. Fairness interventions are proposed to mitigate the issue. To quantify fairness performance, metrics such as Bias amplification (BA) [Zhao et al., 2017] or Equalized odds (EO) [Hardt et al., 2016] have been introduced. Specifically, BA measures disparities in true positives across subgroups, while EO measures true and false positive rates (TPRs, FPRs). We select the prevalent BA and Disparity of Equalized odds (DEO) to measure model fairness performance.

Membership inference attacks. Score-based attack methods adopt the target model's (i.e., models under attack) predictions (i.e., scores or losses) to infer sample membership.

To mimic target model behaviors, adversaries train "shadow models" on an auxiliary dataset that shares similar data distributions with the training data. The outputs of shadow models can then be adapted for attack model training. Formally, given target models T and a queried sample x ∈ X, the membership M(x) can be predicted by:

where the attack model A will output the confidence scores of membership predictions with a threshold τ .

Reference-based likelihood ratio attack methods infer the membership by modeling the prediction distributions. Specifically, they model the distributions using shadowmodels: fin that are trained with sample x; fout that are trained without the x. The key idea is to determine if the target prediction T (x) better aligns with which of the prediction distributions. Formally, membership is predicted by comparing the likelihood ratio Λ between the two distributions:

where ϕ is a scaling function, parameters µ and σ are calculated with predictions from shadow models fin, fout. With the likelihood ratio, whichever is more likely determines the membership of x.

4 Attacking Fair Models

In this section, we conduct a detailed case study to assess the privacy impact of fairness interventions. We first utilize established membership inference attacks (MIAs), specifically the naive score-based attacks. We then introduce an advanced attack method tailored to fair models. It is designed to reveal potential privacy threats of fair models.

4.1 Naive Score-Based Attacks

Figure 2 shows our evaluation pipeline with MIAs. We first train biased models with imbalanced data and then obtain fair models by applying fairness interventions. These models serve as target models for the MIAs. An adversary attempts to infer sample membership with predictions from these target models. We then compare results across target models to analyze the privacy impact of fairness interventions. Subsequent sections will delve into more detailed settings.

Target models. We train biased models with the CelebA dataset [Lee et al., 2020], which contains imbalanced data distributions for various attributes. In particular, we consider smile as classification targets and gender as the sensitive attribute. We train biased models following settings in MLDoctor from [Liu et al., 2022]. We apply fair mixup operations from [Ching-Yao Chuang, 2021; Du et al., 2021] to mitigate the biased predictions. Table 1 presents accuracy (Acct) and fairness metrics (BA, DEO) results for both biased ("Bias") and fair ("Fair") models. The results show decreased fairness metric results, indicating the effectiveness of the adopted fairness interventions.

Threat models. We apply naive score-based attacks on target models in a black-box manner. In particular, adversaries can only access models' predictions and an auxiliary dataset, which shares similar data distributions with the training data. The adversary trains shadow models to mimic the target models' behavior and uses the prediction scores and results (true or false predictions) to infer sample membership. We conduct the attacks following settings in ML-Doctor.

Attack results. Table 1 shows the Acca and AUCa results for attacks on the models. It shows improved attack results after fairness interventions. For example, the accuracy results decreased from 59.8% to 53.2% with the fair models. AUC results exhibit similar trends. This aligns with results in Figure 1, where fewer training samples can be successfully attacked after the interventions. Our results show that fairness interventions provide some defense against existing MIAs. However, our following analyses reveal that existing attacks are ineffective for binary classifiers.

Performance trade-offs. During the evaluation, we observe evident trade-offs in attack performance on member versus non-member data. Figure 3a depicts these trade-offs by comparing the accuracy results for member (x-axis) and non-member (y-axis) data. We run over 100 attacks on biased and fair models, and each point denotes one attack result. The figure shows clear performance trade-offs between member and non-member data for both models. This raises concerns: whether achieving high attack performance comes at the cost of a higher false positive rate (FPR) on non-member data.

The issue becomes more pronounced for hard examples where members and non-members share similar prediction scores. As suggested in [Carlini et al., 2022], we assess the attack performance for hard examples with TPR values in the low FPR region. We find the TPR values are around 0.0 for most attacks. Figure 3b presents two worst-case scenarios. The green curve in the figure shows closely aligned TPR and FPR values, indicating the attack results are equivalent to random guesses. The blue line shows 0.0 TPR values in low FPR regions, indicating that no positive samples can be correctly identified. The findings reveal that attack models fail

• (b) Attack for hard examples

Figure 3: Existing attacks (a) exhibit clear performance trade-offs between member and non-member data, and (b) are inefficient in attacking hard examples in the low FPR region.

to differentiate the membership of hard examples, indicating invalid attacks. This aligns with the concerns about the effectiveness of score-based attacks raised in previous studies [Carlini et al., 2022; Ye et al., 2022].

Model degradation. To explore the reason for the trade-off phenomenon, we have discovered that trained attack models typically degrade into simple threshold models with onedimensional inputs. This is because current attack methods rely on prediction outcomes to determine the sample membership. For binary classifiers, prediction scores can be reduced to one dimension as the sum of the confidence scores always equals one. Consequently, the attack model can essentially be viewed as a simple threshold model, which infers the membership by "thresholding" one-dimensional values.

Figure 4a presents histograms of prediction scores with vertical lines indicating the threshold value. By adjusting the vertical line (thresholds), it is possible to achieve higher accuracy for member data, but this comes at the expense of decreased accuracy for non-member data. This threshold adjustment explains the trade-off phenomenon.

Impacts of fairness interventions. When examining the prediction scores, we find that fairness interventions decrease confidence scores for the majority training data, introducing some defense against existing MIAs. This is evidenced by the histograms of confidence scores in Figures 4a and 4b. The figures show that fairness interventions result in more similar score distributions between member and nonmember data, making it more difficult for the threshold-based attack models to distinguish them.

Moreover, we explore the score changes for different subgroups in Figures 4c and 4d. From the plots, the majority data are more "spread out", whereas the minority are more "concentrated". This is because fairness interventions strive to balance prediction performance across subgroups for fair predictions. The results advocate the observed increased loss values for most data points in Figure 1a. It also aligns with the fairness-utility trade-off, which is extensively observed in fairness studies [Zhang et al., 2023; Pinz´on et al., 2022; Zietlow et al., 2022].

Our analyses indicate that existing attack methods are ineffective in exploiting prediction gaps that could lead to model privacy leaks. While fairness interventions do introduce some defense to MIAs, we identify a novel threat that will pose significant risks to model privacy.

4.2 Attacks with FD-MIA

Enlarged prediction gaps. Previous plots in Figures 4a and 4b show that fairness interventions reduce confidence scores for most training data. In contrast, non-member scores do not exhibit significant changes. This disparity can increase prediction gaps between member and non-member data. We measure the prediction distance by calculating the score value differences between them and present the results in Figure 5. Precisely, we measure the distance considering all available data (Figure 5a) and only the hard examples (Figure 5b), where samples from the members and non-members share similar scores. The plots show a significant distance increase when considering predictions from both biased and fair models. Importantly, the fairness interventions amplify these gaps, which can pose real threats to model privacy.

Attack pipeline. With the enlarged prediction gaps, we propose an enhanced attack method. We present the attack pipeline in Figure 6, where an adversary can access prediction results from both fair and biased models. The attack models will exploit the observed prediction gaps to infer sample membership. We refer to the proposed method as the Fairness Discrepancy based Membership Inference Attack (FD-MIA).

Threat models. FD-MIA operates as a black-box attack and only needs access to predictions from both biased and fair models. In practice, adversaries could obtain such predictions, as real-world models often exhibit persistent biased predictions. For instance, they may monitor an MLaaS over time since debiasing should be continually carried out to adhere to legislation. Alternatively, they could deliberately report biases, compelling the owner to refine the model per regulations. By recording the prediction shifts during these debiasing efforts, the adversary can enable efficient attacks. Meanwhile, the proposed FD-MIA can be seamlessly integrated into the existing frameworks of score-based and reference-based attacks, enhancing their attack performance.

Score-based FD-MIA. Score-based FD-MIA has been introduced to enhance traditional score-based MIAs by integrating additional encoding layers. These layers are designed to extract the features of model predictions, exploiting the observed prediction gaps. Formally, it can be expressed as follows:

where the attack models A takes predictions from both biased models Tbias and fair models Tfair.

Reference-based FD-MIA. Reference-based FD-MIA is integrated with the LiRA framework [Carlini et al., 2022], which infer sample membership by modeling the prediction distributions. It enhances attack performance using two target models - the biased and the fair ones. Formally, for a given sample x and target models T , the probability of membership is given by:

where Cov is the covariance matrix. The distribution function N takes the mean confidence scores from both the biased

µbias and fair models µfair. This function estimates the likelihood of a data point being a member or non-member. The result is determined by the higher probability score.

The introduced FD-MIA is designed to enhance the attack performance by leveraging predictions from both biased and fair models. Unlike prior attack methods, this mitigates the risk of degraded performance in the trained attack model. Our findings reveal that fairness interventions inadvertently introduce new privacy risks, making target models more vulnerable to membership inference attacks.

5 Experiments

We now extensively evaluate our findings and the proposed method under diverse scenarios. We start by introducing the experiment settings.

Settings. With the gender attribute, we consider following binary classifications: smiling predictions (T=s/S=g) with the CelebA dataset [Lee et al., 2020], race predictions (T=r/S=g) with the UTKFace dataset [Geralds, 2017] and the FairFace dataset [Karkkainen and Joo, 2021]. As UTKFace and FairFace contain multiple racial subgroups, we first group them into White and Others and then obtain the binary subgroups. For training data, we randomly divide the dataset in half to construct member and non-member data for MIAs. For target models, We sample the data with imbalanced data distributions, considering the sensitive attribute to reflect real-world imbalances. This incurs biased predictions of trained models, which serve as the biased models. We then apply fairness interventions of fair mixup operations to obtain fair models.

Results with the gender attribute. Table 2 presents the attack results with different attack methods and metrics. We integrate the proposed FD-MIA with score-based attacks (s) and LiRA attacks (l). Besides the attack accuracy, we further report the TPR results at a low FPR value of 0.1%, following suggestions in [Carlini et al., 2022]. The table shows that FDMIA achieves enhanced attack results in accuracy, AUC, and TPR@FPR. Notably, it achieves higher attack success on fair models than the biased ones. In contrast, the existing methods performworse on fair models. The results reveal that the proposed FD-MIA can effectively exploit fairness interventions to improve MIAs, posing threats to model privacy.

More specifically, in score-based attacks, FD-MIA achieves better attack results. On UTKFace, for instance, the accuracy jumps from 52.6% to 60.2% and the AUC from 52.8% to 62.1%. Regarding TPR results, notably, the existing attacks (Biass and Fairs) attain near-zero values. This indicates no true positive samples can be identified. In contrast, FD-MIA displays valid TPR values, indicating effective attacks. Similar trends can be observed with the LiRA attacks. We further present the ROC curves for the CelebA case in Figure 7. The figure further confirms the invalid attacks of the existing methods and the valid TPR results of FD-MIA.

Moreover, from the table, we observe that the attacks achieve better results on FairFace compared to other datasets. Meanwhile, FairFace exhibits a greater discrepancy in fairness between the biased and the fair models. We believe the enlarged discrepancy leads to the enlarged prediction gaps, which can enable more effective attacks. Additionally, we notice that score-based attacks perform better on accuracy and AUC, whereas LiRA attacks achieve better TPR values. This aligns with the observations in [Carlini et al., 2022] as LiRA is designed for efficient attacks in the low FPR region.

Results with other attributes. We further explore attacks with different attributes, including wavy hair (T=s/S=h) and heavy makeup (T=s/S=m) for CelebA, as well as race (T=g/S=r) for UTKFace and FairFace. Table 3 presents the results. The results show that the proposed method exhibits enhanced results with considered metrics, suggesting an advantage in identifying privacy vulnerabilities within fair models. Notably, it consistently achieves superior performance with different datasets of varying accuracy performance. Similar to previous results, FD-MIA achieves better results on FairFace, which may be due to the enlarged fairness discrepancy between fair and biased models.

Results with varying fairness levels. Next, we attackmodels of different fairness performances. Specifically, we consider the case of CelebA (T=s/S=g) and conduct the naive score-based attack on biased and fair models of different DEO values. Figure 8 presents the results. For FD-MIA, we utilize prediction results from multiple fair models and one biased one, which is indicated by a red star in the figure. We further adopt dashed gray lines to outline the trend.

The figure shows that, with the existing attack method, the accuracy decreases for both biased and fair models as the DEO value decreases. The results indicate that models with stronger fairness interventions exhibit more robustness against existing MIAs. In contrast, FD-MIA, which exploits discrepancies in fairness, achieves better attack performance. Notably, larger fairness discrepancies between the fair and biased models contribute to more significant prediction gaps, leading to more powerful attacks with FD-MIA.

While achieving improved fairness, these models lower their confidence scores, making the attacks more challenging.

Results with different fairness approaches. In this part, we evaluate our findings with various fairness approaches, including data sampling, reweighting, adversarial training, and constraint-based approaches. In the experiments, we adopt the implementations of these approaches from [Wang and Deng, 2020; Han et al., 2024]. Similarly, we focus on the case of CelebA (T=s/S=g), and Figure 9 presents the results. The figure shows reduced DEO values after fairness interventions, indicating the effectiveness of these approaches.

For attack results, the existing attack method exhibits degraded performance with fair models for the considered approaches. The attack accuracy drops as the DEO values reduce. The results align with our previous findings, where fairness interventions introduce an unexpected level of resilience to MIAs. Notably, the drops are more pronounced with the adversarial training and constraint approaches. This may stem from the more substantial trade-offs between fairness and model utility inherent to the approaches.

In contrast, FD-MIA achieves higher attack accuracy with fair models than biased ones. Similarly, the attack performance improves when the fairness discrepancy enlarges as FD-MIA explores the prediction gaps. The consistency of results across the fairness approaches demonstrates the broader applicability of our findings and the proposed attack.

Influential factors. The proposed FD-MIA achieved modest improvements in attack performance during the experiments. This is because we considered fair models without severe accuracy degradation, which led to smaller prediction gaps between the fair and biased models. In Figures 8 and 9, the attack performance can be further enhanced by using fairer models with more substantial accuracy drops and fair methods with more significant fairness-accuracy trade-offs.

6 Mitigation

We further discuss two potential defense mechanisms to counter the proposed attack method:

Information Access Control. This involves constraining the adversary's access to key data, thereby potentially diminishing the attack's effectiveness. For example, by restricting the output solely to predicted labels and withholding confidence scores, we can significantly hinder the efficiency of potential attacks. Furthermore, as a proactive step to mitigate privacy risks, we propose the preemptive release of prediction results from fair models before their complete deployment.

Differential privacy (DP). Differential privacy, as introduced in [Dwork et al., 2006], serves as a foundational principle for privacy preservation. We apply the differentially private stochastic gradient descent (DP-SGD) from [Abadi et al.,

2016] to defend against the attacks considering the models for the CelebA (T=s/S=g) in Table 2. Table 4 shows the defense results for the existing attacks and the proposed FD-MIA. The results show that introducing DP noise leads to a notable decrease in attack accuracy, signifying the potential of DP-SGD as a defensive measure. Notably, despite the added noise, our attacks (Ours, Ourl) still exhibit a persistent edge over existing approaches. This implies that increased noise levels are required to attain comparable defense performance. It indicates the superior attack performance of the proposed FD-MIA compared to the existing ones.

7 Conclusions

This paper evaluates the privacy risks of fairness interventions by employing membership inference attacks (MIAs). Our results indicate that fair models often maintain an unexpected level of resilience against existing MIAs for binary classifiers. However, we show that existing attack methods are inefficient as the trained attack models degrade into simple threshold models. Further, we discover a novel attack method named FD-MIA, which leverages predictions from both biased and fair models to exploit the prediction gaps between member and non-member data. It can be integrated into existing attacks and pose substantial threats to model privacy. We conduct experiments across six datasets, three attack methods, and five representative fairness approaches. The results consistently validate our findings and the efficacy of the proposed MIA method. While our attack is tailored to fair binary classifiers, it can be extended to a broader range of models with fairness disparities. Our insights contribute to a deeper understanding of privacy issues related to the application of fairness interventions, emphasizing the imperative need for meticulous design and deployment of trustworthy models. All the appendix can be found in this link 1 .

1 https://arxiv.org/abs/2311.03865

Ethical Statement

We reveal a concerning tension between the pursuit of algorithmic fairness and model privacy. This trade-off raises important ethical questions that warrant careful examination. We measure model fairness using different metrics and assess the model's privacy performance with MIAs, which can be used for evaluating deployed models. We encourage the adoption of techniques like differential privacy to mitigate privacy risks while maintaining fairness. By incorporating these key considerations, we aim to provide practitioners with a more holistic understanding of the challenges and potential solutions, enabling them to navigate this complex issue in a responsible and ethical manner.

Acknowledgements

This work is supported by the Australian Research Council (ARC) Discovery Project of DP230100246 from the Australian Research Council, Australia.