DFCA: Disentangled Feature Contrastive Learning and Augmentation for Fairer Dermatological Diagnostics

1 Introduction

With the rapid development of artificial intelligence (AI), deep learning-based disease diagnosis systems have played a vital role in universal healthcare [Zhou et al., 2021; Archana and Jeevaraj, 2024; Li et al., 2024], especially in remote and impoverished areas where medical resources are severely limited. In the domain of dermatological diagnostics, for example, deep learning models trained on expertannotated data are now accessible through mobile phones [Lee et al., 2022]. As a result, individuals from underdeveloped regions can even capture skin images using their smartphone cameras to conduct a skin condition diagnosis.

∗ Corresponding author.

However, these diagnosis models may exhibit distinct performance across different demographic subgroups, particularly among populations with different skin color, which namely unfairness [Mehrabi et al., 2021]. Unfair dermatological diagnosis not only hinder the promotion of AI models but also exacerbate the health gap between different races, potentially tearing society apart and raising serious ethical and legal issues.

Despite numerous researchers have devoted to the enhancing the performance of deep learning models [Mirikharaji et al., 2023; Ye et al., 2024; Choy et al., 2023; Noronha et al., 2023], research on mitigating unfairness remain rare, comparatively. The existing literature for fairness can be divided into three main groups: pre-processing [Kamiran and Calders, 2012; Krasanakis et al., 2018; Puyol-Ant´on et al., 2021], in-processing [Du et al., 2022; Kamishima et al., 2012; Deng et al., 2023; Xu et al., 2023b] and post-processing methods [Wu et al., 2022]. Pre-processing methods reduce possible bias in data before training the model, in-processing methods intervene during training phase, post-processing adjust the output produced by model to mitigate biased predictions. Pre-processing methods often face the challenge due to limited data [Maluleke et al., 2022]. The latter two methods, by contrast, seemingly direct operate models' feature space or output to remove the useless demographic information, showing a higher ceiling. However, the reduction of demographic information in the feature space will also harm the disease information because of the feature coupling, which ultimately leading to a phenomenon described as accuracyfairness trade-off [Chen et al., 2023].

From the view of representation learning, there are two main reasons accounting for unfairness in dermatological diagnostics: class imbalanced data and entangled feature in deep learning model. Specifically, different quantities data among various race (maybe also among various skin disease) cause unbalanced disease-relevant feature representations. Besides, deep learning models entangle disease-relevant features and skin type statistic information, which further heaven the degree of unbalance between different race groups in feature space. Moreover, public available skin disease datasets that considering demographic attributes especially race are rare and some with limited data size [Xu et al., 2023a; Xu et al., 2024], natively leading to inadequate representation of disease features in deep learning model. Ultimately, dermatological diagnostics algorithms show unfairness.

In this paper, we introduce the Disentangled Feature Contrastive learning and Augmentation framework (DFCA), which aims to reduce unfairness in AI-based dermatological diagnostics while maintaining accuracy. DFCA employs a symmetric architecture to disentangle features into diseaserelated and disease-unrelated components, capturing lesion information and demographic statistics (e.g., skin color) separately. Considering the fact that lesion color is in connection with both disease condition and skin type, we map these components into von Mises-Fisher (vMF) distribution spaces for supervised contrastive learning. This approach, which has been proven effective in long-tailed data recognition, ensures that representations of the same disease condition to be close and those of different classes distant regardless of skin tone. The same as skin tone features. To address limited data, DFCA samples from the learned vMF distributions to augment the feature space using a reversible flow model. A perfect diagnosis model should both consider the accuracy and fairness. However, most of the existing methods mainly focus on the fairness metrics, at the cost of diagnosis performance. To emphasize the importance of the balance of accuracy and fairness when researchers designing models, we propose a new metric for assessing the Accuracy-Fairness Balance Degree (AFBD). The results of extensive experiments on two datasets shows that DFCA, by combing disentangled feature contrastive learning and augmentation, improves both fairness and accuracy compared to SOTA methods.

Overall, our contributions lie in effectively disentangling dermatologic images into disease-related and skin-tonerelated features, thereby minimizing the interference of demographic attributes in dermatological diagnostics. Additionally, we introduce spaces based on mixture of von MisesFisher distributions for contrastive learning, which facilitates the feature disentanglement and clustering. Moreover, we samples from the learned vMF distributions to inversely augment the feature space. To emphasize the importance of balancing fairness and accuracy, we propose a new metric named Accuracy-Fairness Balance Degree (AFBD). Extensive experiments demonstrate that our framework achieves state-ofthe-art performance in both fairness and accuracy.

2 RelatedWorks

Despite deep learning has been widely applied in dermatological diagnosis and has achieved notably performance, its generalisability remain limited by poor capture of demographic information[Choy et al., 2023]. The phenomena of unfairness is that those diagnosis models may exhibit distinct performance across different demographic subgroups [Mehrabi et al., 2021]. In this section, we give a concise survey of unfairness mitigation methods and analyze the position of this paper within the field.

Existing unfairness mitigation can be categorized into preprocessing, in-processing, and post-processing methods according to the implementation phase [Xu et al., 2024].

Pre-processing methods reduce possible bias in data before training the model, their common practice is to restructure the training data. The representative work is [Kamiran and Calders, 2012], where the authors proposed four methods to pre-processing the dataset, including removing sensitive attribute, massaging the dataset, reweighing and resampling. Similarly, [Zhang et al., 2022; Abernethy et al., 2020; Puyol-Ant´on et al., 2021] also upsampled minority groups to ensure equal presence of each subgroup. [Krasanakis et al., 2018] proposed an adaptive sensitive reweighting mechanism model for fair classify. Another technical approach is data generation. [Burlina et al., 2021] augmented the training data by generative models for debiasing in retinal diagnostics. To mitigate the impact of demographic imbalance, [Pombo et al., 2023] employed generative model to synthesise counterfactual volumetric brain imaging, conditioning on original image and demographic attributes (sex and age). Although preprocessing methods can achieve high accuracy, they can not solve feature under representation among vulnerable groups and may face instability problems in generation due to limited data. Our method makes the advantage of the learned vMF distribution in contrastive space and samples from that to inversely conduct augmentation in feature space.

In-processing methods intervene during training phase, they usually combine the architecture of model and extra losses to mitigate unfairness. [Stanley et al., 2022] added an adversarial branch to minimize the influence of sex and race in brain MRI image classification. FairAdaBN [Xu et al., 2023b] reduces unfairness by adding extra adapters to the original model which can adaptively adjust the mean and variance of the feature vector according to the sensitive attribute. [Pakzad et al., 2022] adopted a domain invariant representation learning method to remove the skin tone information in the classifer. [Du et al., 2022] is most closely to our work, which proposed FairDisCo framework to mitigate the negative effects of skin tone in dermatology diagnosis. Similarly, FairDisCo also used contrastive learning to boost the diagnoise accuracy and disentangled learning with different branches to reduce impede of skin tone information. However, its contrastive learning constrained by limited data and different branches constrained by highly coupled feature in the feature extractor. Our method adopts a symmetric architecture for disentanglement. Besides, we uses the vMF distribution based contrastive learning to further promote the diagnosis performance.

Post-processing methods adjust the output produced by model to mitigate biased predictions. [Pleiss et al., 2017] employed calibration for specific subgroup thresholds in fair classification. [Wu et al., 2022] utilized a pruning strategy to remove the sensitive information associated with a specific subgroup. [Huang et al., 2023] considered age and gender and also proposed a pruning method. However, these methods may face the accuracy-fairness trade-off because of the feature coupling. A portion of the model parameters may be associated with both disease condition and skin type, which can be settled by feature disentanglement in our method.

3 Methods

As mentioned previously, unbalanced data and feature entanglement are the primary causes of unfairness. Class imbalanced data brings the imbalanced feature representation and feature entanglement in deep learning model further mix the disease condition information and skin tone information, thereby cause unfairness dermatological diagnosis. Our goal is to reduce the performance differences of the diagnostic model across different populations while maintaining accuracy, thereby alleviating unfairness. To achieve this, DFCA is proposed. To eliminate the impact of skin tone on disease diagnosis, we employ a symmetric disentangled structure to obtain disease-related features and skin tone related features. To overcome the imbalanced feature representation, DFCA adopts surpervised contrastive learning based on a mixture of von Mises-Fisher (vMF) distributions, as described in Section 3.1. Moreover, we samples from the vMF distribution-based contrastive spaces and inversely generate features to further augment the feature space and boost diagnosis performance, as described in Section 3.2.

3.1 Feature Disentanglement and Contrastive Learning

We consider the problem of dermatological diagnosis under the influence of skin tone information. Let X = {x1, . . . , xN} be the training set, where xi denotes the ith sample. X has two sets of labels: YD = {y1 d , . . . , yN d } and YS = {y1 s , . . . , yN s }, where yid , yis correspond to the disease label and skin tone label of xi, respectively. Y i d ∈ {0, 1, . . . , C} and Y i s ∈ {0, 1, . . . , S} The perfect situation is that the diagnostic results are independent of the skin tone labels of the data:

where a ̸= b and a, b ∈ YS. We strive to achieve this goal by feature disentanglement and contrastive learning.

A Feature Disentanglement

As shown in Figure 1, given a sample from a mini batch of training data, we initially employ a distinctive data augmentation to get six versions. Each three of them form a group as the inputs to the following designed network, denoted as ID = {I 1 d , I 2 d , I 3 d } and IS = {I 1 s , I 2 s , I 3 s }. In each group, I 2 and I 3 employ the same weak augmentation method, distinct from a strong augmentation method of I 1 , as the manner of [Zhu et al., 2022]. To maximize the reduction of the impact of skin tone information in the data on disease diagnosis, we adopt a symmetric architecture to disentangle the feature into disease-related part and disease-unrelated part, which mainly contain the disease condition information and skin tone information, respectively. We named them the branch d and branch s. Correspondingly, we adopt different augmentation methods between ID and IS. While ID focus on diverse gray scale variation and flip for robust lesion feature analysis ability and IS focus on different fuzzy degree for the judgment of skin tone.

After the image augmentation, DFCA employ Ed, Es to encode {I 1 d , I 2 d , I 3 d }, {I 1 s , I 2 s , I 3 s } into the feature space {f 1 d , f 2 d , f 3 d } and {f 1 s , f 2 s , f 3 s }. Ed and Es have the same structure but without shared weights. Suppose that the features of sample xi are {f i1 d , f i2 d , f i3 d } and {f i1 s , f i2 s , f i3 s } We employ an orthogonality loss to ensure that the two feature spaces are mutually independent and semantically continuous:

Next, we take the branch d as an example while the branch s undergoing the same operations. Given the {f 1 d , f 2 d , f 3 d }, DFCA send f 1 d into the disease classifer Cd and training in a surpervised manner:

and the same with Lcs for branch s.

Considering the fact that lesion color is in connection with both disease condition and skin type, we assigned a small weight to Lof in practice. Moreover, we design a inverse flow model takeing f 2 d and f 3 d as input for supervised contrastive learning, making the distance of representation from same disease condition close and from different class far away no matter which skin tone it belongs to, and the same with f 2 s and f 3 s .

B Supervised Contrastive Learning

The von Mises-Fisher (vMF) distribution is often described as the Normal Gaussian distribution on a hypersphere and is sometimes used as a substitute for Y in some scenarios. Recently, vMF distribution has also demonstrated its potential for application in the contrastive learning especially for the long-tailed recognition problem [Du et al., 2024]. Similar to Gaussian distribution, it can be parameterized by µ indicating the mean direction and k representing the concentration around µ. The larger the value of k, the greater is the clustering aruond the mean direction µ. When k = 0, the distribution is uniform. The probability density function of the vMF distribution for a random d-dimensional unit vector z is defined as:

where k ≥ 0, ∥µ∥2 = 1, Cp(k) is the normalizing constant and Iv denotes the modified Bessel function of the first kind at order v:

In this paper, we introduce a mixture of vMF distributions as the distribution in the contrastive space for each branch. To model the conversion from feature space to contrastive space, we adopt a simple but effective inverse flow model. This paves the way for feature enhancement, which will be introduced in the next section.

The bottom box of Figure 1 exhibits the structure of our inverse flow model for supervised contrastive learning. IFd has two reversible flow process. Given the d-dimensional feacture vetor f , we firstly downsample it to a d/2-dimensional vector f0 as the input at the starting moment (t = 0). Through the repeated MLP blocks, the model learn the minute changes from feature space to contrastive space:

where dt = 1 T , M() stands for MLP block and ft is the vector at the t th moment. Note that MLP blocks would not change the dimensionality of the vectors. The final vector fT is subsequently projected into the contrastive space by normalization. We also use an orthogonality loss to ensure the independence of two contrastive spaces:

[Du et al., 2024] has given a closed form of expected supervised contrastive loss LProCo based on the estimated vMF distribution when the sampling number tends to infinite. LProCo can address the issue of insufficient performance in contrastive learning due to a small number of contrastive sample pairs. We use the LProCo as:

where πyj is the class frequency in the training set. {f 2 dT , f 3 dT } are used to calculate the LProCod loss and the same with LProCos for {f 2 dS , f 3 dS }.

The backward process of inverse flow model follows:

which is just the inverse process of Eq.(7). Through the backward process, f0 is gradually recovered from fT denoted as f ′0 . A recontruction loss Lrecd is ultilized to train the inverse flow model IFd and the same with Lrecs:

3.2 vMF distribution based Feature Augmentation

Through feature disentanglement and contrastive Learning mentioned before, DFCA learns a mixture of vMF distributions in contrastive space. Despite LProCo has given a closed form of expected supervised contrastive loss when the sampling number tends to infinite, we observe that its capability becomes constrained in some situations where the dataset contains a very limited number of samples. To take advantage of the learned vMF distributions, we sample from contrastive space to inversely augment the feature space for further enhancing the capability of disease classifer.

Consider the vMF distribution:

where Cp(k) is the normalizing constant. Given the µj of vMF distribution of class j, the density depends on x only through µT x. We decomposition x as:

where µ ⊥ v and ω can be seen as the cosine of the angle between x and µ. The probability density function of ω is given in [Mardia and Jupp, 2000]:

with ω ∈ [-1, 1]. We calculate the f -1 (ω) to get the distribution of ω. Then we can sample the x from the vMF distribution through Eq.13, defined as fsamT .

Red arrows in Figure 1 shows the training process in this stage. We firstly inverse the fsamdT to feature vector ysamd. Afterwards, ysamd is sent into the disease classifer with the loss functions:

where M is the number of sampled features and ˆ ysamd is the label of sampled feature ysamd. Similar with Lrecd, we inversely reconstruct the fsamdT with the constrain of:

while the same with Lrecsams and Lcsams. Noted that we also use the real data for mixed training in this stage.

3.3 Learning Loss

Briefly, we define the loss fuctions by: Lo = Lof +Loc, Ld = Lcd+LProCod+Lcsamd, Ls = Lcs+LProCos+Lcsams and Lrec = Lrecd + Lrecs + Lrecsamd + Lrecsams. Therefore, the joint loss function can be represented as:

where α, β, γ are the weights of the loss terms.

4 Experiment

4.1 Datasets and EvaluationMetrics

We use two well-known dermatology datasets to evaluate our proposed method: Fitzpatrick-17k dataset [Groh et al., 2021] and DDI dataset [Daneshjou et al., 2022]. Both of the datasets contain skin tone attribute.

The Fitzpatrick-17k dataset [Groh et al., 2021] contains 16,577 dermatology images with disease condition labels and skin tone labels. They have two kinds of methods for disease condition classification: 3 (malignant, non-neoplastic, benign) and 9. There are six categories of skin tones labeled by 1-6. The higher the number, the darker the skin tone.

The DDI dataset [Daneshjou et al., 2022] contains 656 dermatology images with disease condition labels and skin tone labels. Its disease condition labels contains malignant (0) or not (1). And its skin tones are categorised into 3 groups: 1, 2, 3, corresponding to the skin tone of {1, 2}, {3, 4}, {5, 6} in Fitzpatrick-17k dataset, respectively.

We use three common metircs for evaluating the fairness toward the effection of skin tone attributes: Predictive Quality Disparity (PQD), Demographic Disparity (DP) and Equality of Opportunity (EO). PQD measures the prediction quality difference between each subgroup, DP computes the percentage diversities of positive outcomes for each subgroup and EO asserts that different subgroups should have similar true positive rates, just as [Du et al., 2022].

Moreover, a perfect diagnosis model should both consider the accuracy and fairness. However, most of the existing methods mainly focus on the fairness metrics. To emphasize the importance of the balance of accuracy and fairness when researchers designing models, we propose a new metric for assessing the Accuracy-Fairness Balance Degree (AFBD):

where AD is the averaged disparity between average accuracy ACC and accuracy of subgroup ACCi with skin tone i.

4.2 Experiment Settings

We implement our DFCA model by PyTorch. DFCA is trained for 150 epochs firstly with the real datasets and 100 epochs with the mixture of feature augmentation. After all the trainging stages, we only use the Feature Extractor Ed and Disease Classifer Cd for evaluation. Both of the feature extractors are ResNet-101 [He et al., 2016] pretrained on ImageNet. We use the same image augmentation mechanisms as [Zhu et al., 2022]. Our model is trained by Adam optimizer with a learning rate lr = 0.0001. The batch size is 32. The weights are α = 10, β = 0.5 and γ = 1.

4.3 Baseline Methods

We compare our DFCA with six state-of-the-art models, including the pre-processing methods RESM and REWT [Kamiran and Calders, 2012], in-processing methods FairAdaBN [Xu et al., 2023b], LFF [Wang et al., 2024], FairDisCo [Du et al., 2022], post-processing methods FairPrune [Wu et al., 2022] ,according to the intervene stage for fairness.

4.4 Performance Comparison with SOTA

In this section, we conduct the in-domain experiment for the comparison of fair diagnosis and out-domain experiment for observing the model's generalization ability. Due to the samll datasize of DDI dataset, we only conduct the in-domain experiment on it, just as [Du et al., 2022].

Table 1 and table 2 show the results of in-domain comparison on Fitzpatrick-17k and DDI datasets. We see that our method has the biggest average accuracy and the best diagnosis performance in almost all the skin tones as well as three fairness evaluation metrics, both on Fitzpatrick-17k and DDI datasets. It is noteworthy that in table 2, DFCA achieve a significant enhancement on DDI, suggesting that our method can promote the performance on small dataset. While FairPrune and FairAdaBN gain the worst performance, because of the feature entanglement in their model. As to AFBD, we found that our method also attaines the optimal performance on both of the datasets, indicating that DFCA also taken the accuracy-fairness balance into consideration. Overall, the results demonstrate that our contrastive feature disentangment and augmentation method is able to enhance fairness in dermatological diagnoses as well as accuracy.

Table 3 exhibts the out-domain comparison on Fitzpatrick17k dataset. '-' represents that we use data of this skin tone for training and test in other skin tones. And we conduct three groups of experiments. Due to the poor performance of FairPrune and FairAdaBN in both the accuracy and fairness metrics especially when data is limted (as seen in Table 2), we discarded them in the out-domain experiment. We can see that our proposed method achieves highest scores in almost every average accuracy and diagnosis accuracy of each skin tone among all of the three groups. Especially in the group two, DFCA has elevated average accuracy metric by nearly 4 percentage points. Moreover, DFCA has also delivered a satisfactory performance in terms of fairness metrics and our propose AFBD. Overall, the results demonstrate that our method possess robust generalization capabilities under the fairness dermatological diagnostics as well.

4.5 Abalation Study

To understand the contribution of each component in the DFCA, we perform ablation studies on the Fitzpatrick-17k dataset. Table 4 shows the comparison results of ACC and AFBD. 'w/o Dis' means we conduct the experiment without feature disentanglement, 'w/o IF' means no feature augmentation while with contrastive learning, and 'w/o Ctr' means no contrastive learning while with feature augmentation. The results indicate that feature disentanglement makes the greatest contribution both in diagnosis accuracy and fairness. While the proposed feature augmentation from vMF distribution can enhance diagnosis accuracy while ensuring fairness. And the LProCo loss of contrastive learning can better take the balance between accuracy and fairness into consideration.

Furthermore, we conduct another abalation study to investigate the impact brought by the structure of inverse model, which is designed to model the inverse conversation between feature space and contrastive space, as exhibited in Table 5. We take three methods for comparision: diffusion model, one MLP network and our proposed inverse flow model IF . The results show that MLP can achieve comparable performance to DFCA in terms of accuracy. However, MLP is relatively weak in terms of fairness, indicating the flow model can better handle the complex conversions between feature space and contrastive space. Diffusion model achieves the worst performance in both metrics, due to its training instability expecially for the certain vMF distribution.

5 Conclusion and FutureWork

In this paper, we propose a Disentangled Feature Contrastive learning and Augmentation framework (DFCA) to reduce unfairness in dermatological diagnostics. DFCA employs a symmetric architecture to disentangle features into diseaserelated and skin-tone-related components, thereby minimizing the impact of skin tone on disease diagnosis. We introduce a contrastive learning framework based on von MisesFisher (vMF) distributions and use it to inversely augment the feature space. Extensive comparisons and ablation studies demonstrate the effectiveness of our approach.

Despite our method presents a promising approach to solving the fairer dermatological diagnostics problem, there are several limitations. Dermatological datasets with skin tone attributes remain scarce, which limits the development in this direction. Our proposed inverse flow model for feature augmentation is computationally complex. Further research is needed to provide larger datasets and computationally efficient methods. Additionally, improving interpretability and extending applications to a broader range of diseases are key directions for future work.

Acknowledgments

We are grateful for the insightful comments and suggestions from the reviewers.