FairSMOE: Mitigating Multi-Attribute Fairness Problem with Sparse Mixture-of-Experts

1 Introduction

As AI democratization advances, machine learning (ML) has been increasingly utilized in a variety of applications, including image or video generation [Wu et al., 2022b; Zhan et al., 2021; Zhan et al., 2024a; Shen et al., 2025b; Li et al., 2025;

Shen et al., 2025c], autonomous driving [Li et al., 2023a; Yang et al., 2023c; Zhang et al., 2022; Li et al., 2022; Shen et al., 2025a], and language translation [Zhao et al., 2024; Zhan et al., 2024b; Shen et al., 2024; Shen et al., 2025b; Shen et al., 2025d]. Fairness has emerged as a significant and fundamental concern in these applications. Studies have found unfair ML models exhibiting worse performance toward sensitive attributes, such as race [Nanda et al., 2021; Puyol-Ant´on et al., 2022], gender [Puyol-Ant´on et al., 2022] and skin tone [Yang et al., 2023b; Yang et al., 2023a], leading to discrimination and undermining the trustworthiness of ML [Li et al., 2023b; Li et al., 2024] from the public.

Many research efforts are devoted to improving fairness in ML, which include two common categories: (1) Mitigating unbalanced data by generation synthetic data or adopting data augmentation techniques [Sattigeri et al., 2019; Xu et al., 2018]; (2) Revisiting training procedure by utilizing adversarial training [Karkkainen and Joo, 2021; Wang and Deng, 2020], discriminate training [Tao et al., 2022] or training with fair objectives [Karkkainen and Joo, 2021]. Although these methods are effective in improving singleattribute fairness, in practice, an individual may have multiple sensitive attributes, and models optimized for just one attribute can still make unfair predictions.

Recently, a few studies have started to investigate multiattribute fairness optimization. Data pre-processing [Tian et al., 2024] is considered to ensure statistical parity among multiple sensitive attributes, while other works [Hwang et al., 2020; Deng et al., 2023] extend single-attribute fairness optimization techniques to multi-attribute protections by introducing additional constraints or prediction heads for multiple attributes. However, directly adapting existing methods for multi-attribute protection has several limitations. Firstly, as the number of attributes increases, the computational cost rises accordingly. Consequently, current approaches only consider a small number of sensitive attributes. Secondly, extending these methods may inherit or aggravate their drawbacks. For example, adversarial approaches require elaborate tuning to guarantee training convergence due to the inverse gradient updating [Wang and Deng, 2020].

To tackle these challenges, one solution is to formulate the multi-attribute fairness problem as a multi-task learning (MTL) problem. Multi-attribute fairness naturally decomposes into multiple "tasks," each linked to a sensitive attribute or intersection of attributes. By adopting a MTL framework, the model can simultaneously learn shared representations across attributes while preserving each attribute's unique features. To increase model capacity while maintaining similar computational cost, the Sparse Mixture of Experts (SMoE) is particularly suited because it can dynamically allocate specialized experts to each task [Chen et al., 2023; Chen et al., 2022]. This methodology optimizes fairness for all tasks, improves computational efficiency, and avoids the instability of other architectural approaches.

However, several challenges must be addressed to build an effective MTL framework for multi-attribute fairness. First, conventional MTL setups use a fixed number of experts per task, which can lead to suboptimal performance due to the varying difficulty of optimizing different sensitive attributes. Second, without proper regularization, the router may develop deterministic patterns when dealing with multi-attribute fairness problem. This leads to over-utilization of certain experts, hindering the capacity and reducing the efficiency of SMoE. Moreover, from the perspective of training dynamics, when certain experts are rarely used, the gradients flowing back to update their parameters are somewhat sparse, meaning that these experts are updated very slowly or not at all, which destabilizes the entire training process. Thus, appropriate regularization is required to balance expert utilization and maintain fairness across multiple attributes.

Our contributions are summarized below:

• We target multi-attribute fairness optimization problem and first formalize the problem as a MTL problem. We further analyze the feasibility and potentiality of using SMoE in this problem and tackle the challenge of unbalanced utilization of experts and task difficulties.
• We propose an innovative SMoE framework for multiattribute fair image classification to improve fairness by redesigning the SMoE layer with fairness considerations. We regulate experts with fairness constraints and dynamically allocate expert numbers for each task.
• Extensive experiments demonstrated our effectiveness. Taking a DeiT-Small as the backbone, FairSMoE achieves 77▷25% and 86▷01% accuracy on the ISIC2019 and CelebA dataset respectively with Multi-attribute Predictive Quality Disparity (PQD) score of 0▷801 and 0▷787, beating current state-of-the-art methods such as Muffin and MultiFair. FairSMoE alleviates unbalanced routing and gradient conflict issue.

2 RelatedWorks

Single-Attribute Fairness. Distribution-based methods aim to better represent minority groups or eliminate undesired biases in datasets. For instance, [Derman, 2021; Stafanoviˇcs et al., 2020] propose algorithms that adjust objects in datasets based on predefined rules, while [Yan et al., 2020] discusses sampling techniques to address under-representation of protected groups. However, undersampling strategies are impractical for DNNs as they reduce dataset size making training infeasible.

One-step training methods incorporate fairness into the main training procedure. The works [Gaci et al., 2022; Xu et al., 2019] utilize adversarial frameworks to train models avoiding undesired biases. These methods often require annotations of protected variables, which can be limiting. Additionally, optimization methods have been proposed to enhance fairness during training [Du et al., 2023; Wu et al., 2022a] balancing fairness and accuracy.

Multi-Attribute Fairness. Recent methods addressing multi-attribute fairness primarily focus on data preprocessing and augmentation [Deng et al., 2023; Tian et al., 2024]. For example, [Sheng et al., 2023] proposed a Neural Architecture Search framework to automatically search for fair combinations in multi-attribute models. However, this can cause attribute turbulence among model candidates, leading to performance degradation. [Tian et al., 2024] focus on data augmentation by introducing mix-up procedures to generate synthetic data.

Fairness in Multi-Task Learning. As MTL becomes increasingly prevalent in SOTA models [Ruder, 2017; Zhang and Yang, 2021], understanding the interaction between fairness and MTL is essential. [D'Amour et al., 2020] investigates fairness in multi-task regression models using rankbased non-parametric independence tests. [Zhao and Chen, 2020] proposes MTL enhanced with fairness constraints to jointly learn classifiers leveraging information across sensitive groups.

Mixture of Experts. The initial concept of MoE [Jacobs et al., 1991] involves dividing input space into regions and training specialized experts for each region, with a gating network selecting appropriate experts. Recent advancements leverage SMoE [Jiang et al., 2024; Du et al., 2022] to handle increasing complexity of modern datasets. SMoE [Shazeer et al., 2017] reduces computational overhead by dynamically routing inputs to expert subsets. GShard [Lepikhin et al., 2020] extends MoE models to multilingual settings, scaling to handle over 100 languages simultaneously.

3 Discovery and Analysis

Existing works have shown model fusion success on multiattribute fairness optimization [Sheng et al., 2023]. However, SMoE under MTL scenarios has crucial factors-routing policy and number of experts per task-that differentiate it from model fusion methods. To investigate optimal SMoE utilization, we address the following questions.

3.1 Do Different Attributes Have the Same Difficulty To Optimize?

We investigated how various sensitive attributes perform under identical model configurations using the ISIC2019 dataset for dermatology disease classification, analyzing three sensitive attributes: age group, gender, and disease site.

Experimental Setting. DeiT-Small [Touvron et al., 2021] model was employed with the last layer replaced by a sparse SMoE layer. Number of experts was set to 4 with auxiliary loss [Zoph et al., 2022]. Fairness was quantified using Predictive Quality Disparity (PQD) score.

Analysis Results. The results, illustrated in Figure 1, reveal significant disparities in fairness optimization across these attributes. Gender attribute showed high fairness score variance, suggesting minor disparity and balanced outcomes. Conversely, disease site and age group demonstrated low PQD as shown in Figure 2, indicating greater classification outcome disparity and fairness optimization challenges.

Observation. Not all attributes are equally challenging to optimize for fairness. Attributes with higher variability and less balanced representation (disease site and age group) are more difficult to optimize than gender, highlighting the need for attribute-specific strategies.

3.2 HowDoes the Default Routing Policy Behave When Dealt With Multi-Attribute?

Building on observations that different attributes present varying optimization difficulties, we investigated how the default routing policy manages inputs related to both easier and more challenging attributes, and how altering expert numbers and SMoE layer placement affects these dynamics.

Experimental Setting. We explored transformer-based architectures with SMoE layers in various configurations, increasing expert numbers and introducing SMoE layers earlier in the network to observe routing policy distribution across experts.

Analysis Results and Observation. Our findings reveal that, without tailored adjustments, the default routing policy routes similar inputs-regardless of attribute difficulty-to the same experts. This pattern intensifies with increased experts or earlier SMoE layer integration, leading to significant expert overutilization. These results, illustrated in Figure 2, underscore the challenges in maintaining balanced distribution of inputs across experts.

Conclusion and Takeaway. This investigation highlights the critical need for adaptive routing mechanisms within SMoE models for multiple sensitive attributes. Current default routing policies may not sufficiently accommodate input diversity, potentially leading to biased outputs and decreased effectiveness.

4 ProblemFormulation

Fairness Metrics. (i) Here we present our definition of fairness used for evaluation. We utilize two mainstream metrics for fairness evaluation: Predictive Quality Disparity (PQD) and Demographic Parity (DP). Details on the definitions of these metrics are available in the Appendix. (ii) Multiattribute Fairness (MF): it measures the overall fairness on multiple attributes. Let S = ¶s1↪ s2↪ ▷ ▷ ▷ ↪ sm♦ be the set of sensitive attributes in a dataset. The multi-attribute fairness score, MF!, under the fairness metric !, is defined as:

where !(si) represents the fairness score for the sensitive attribute si, and m is the total number of sensitive attributes.

Problem Formulation. Addressing the multi-attribute fairness problem requires a holistic approach considering the interdependence among all attributes and their impact on the fairness of the model. To this end, we formulate the multiattribute fairness optimization problem as a multi-task learning (MTL) problem. This approach enables the simultaneous optimization of fairness across various sensitive attributes, alongside the primary task of prediction performance.

In our MTL framework, the tasks are defined as follows:

(i) Primary Task (Predictive Performance): The primary objective is to maximize the overall accuracy of predictions across all groups, defined by:

where xi and yi represent the features and label of the i-th data point, and ω denotes the parameters of the model.

(ii) Fairness Tasks: Each fairness task aims to optimize the fairness score in predictive outcomes related to each sensitive attribute si → ¶s1↪ s2↪ ▷ ▷ ▷ ↪ sm♦. For each fairness metric ! such as PQD and DP, we define a corresponding loss function that penalizes the deviation from fairness:

(iii) Overall Objective: The overall training objective combines the predictive performance with the fairness tasks, balanced by a set of tunable parameters ε, which regulate the trade-off between accuracy and fairness:

By integrating these objectives into a single MTL framework, our approach allows for explicit control over the tradeoffs between achieving high predictive performance and ensuring fairness across multiple sensitive attributes.

5 Methodology

5.1 Revisiting Sparse Mixture of Experts

SMoE is proposed to scale up the model capacity while maintaining low per-inference costs. In this work, we insert SMoE layers into a transformer block. The SMoE block consist of n experts ¶E1↪ ▷▷▷↪En♦, each of which is a feedforward neural network similar to those in the vision transformer block.

Giving an input embedding x, it is fed into a router network G and assigned to the most relevant experts. The architecture of the router network is usually one or a few layers of multilayer perceptrons (MLPs). The gating mechanism is defined as following:

where g(·) are trainable gating networks and Top-K select the largest K values. The final output of an SMoE block is a summarization of features from the activated experts and can be depicted as below:

where Ei(x) stands for the feature representations produced from the expert Ei, which is weighted by R(x) to form the final output y.

5.2 Fairness-Guided Routing (FGR)

The proposed FairSMoE consists of the MoE layer redesigned with fairness consideration. In the training framework, we regulate expert with fairness constraints.

We begin by investigating the routing behavior of a vanilla Sparse Mixture of Experts (SMoE) model trained with a top-2 routing policy using the ISIC2019 dermatology image classification dataset, which includes three sensitive attributes and 16 different classes in total: sex (2), age (5), and general anatomic site (9). Discoveries in section 3 indicate that (i) the routing choices across different attributes are highly similar, and (ii) despite an increase in the number of experts, only a few experts are heavily utilized, leading to a decreased overall utilization rate. These findings suggest that the standard routing paradigm in SMoE is inefficient for multiple sensitive attributes scenarios, potentially leading to under-utilization of model capacity.

To address this inefficiency, we develop a fairness-guided routing policy utilizing disentangled representation learning to mitigate the influence of sensitive attributes on routing decisions. We replace the traditional MLP gating network with a novel structure, as in Figure 3, consisting of the original MLP for gating fg as a gating branch, a feature extractor network ϑ and an additional sensitive attribute (SA) branch, which consists of a number of m classifiers

¶fs1 ↪ fs2 ↪ ▷▷▷↪ fsm ♦ to predict the attribute class. The input x will first pass the feature extractor ϑ to get a representation z = ϑ(x). Then the z will be fed into both the gating branch and SA branch. In the Gating branch, we will get original gating information e = fg(z). In the SA branch, z will be copied m times and fed into ¶fs1 ↪ fs2 ↪ ▷▷▷↪ fsm ♦ and get class predictions of each attributes ¶p1↪ p2↪ ▷▷▷↪ pm♦. After that, we incorporate a confusion loss to further enhance this disentanglement:

This loss encourages the feature extractor to generate features that are indistinguishable with respect to sensitive attributes, fostering a state of maximum confusion. Output ¶fs1 ↪ fs2 ↪ ▷▷▷↪ fsm ♦ are then fed into a linear projection layer to get the expert choice vector ¶e1↪ e2↪ ▷▷▷↪ em♦ and will add with e. We select the vector from the classifier corresponding to the specific attribute class and then apply a softmax to get the final expert probability distribution.

By fusing losses into the training process of the SMoE model, we ensure that the SA branch of the gating network directs routing decisions based on task relevance and fairness considerations, rather than on detectable sensitive attributes. This approach not only minimizes the influence of protected attributes on routing decisions but also optimizes the utilization of experts by mitigating conventional biases and routing patterns observed in vanilla SMoE models.

This dual-branch structure allows our SMoE model to handle multiple tasks and sensitive attributes more effectively. It aligns with the overarching goals of fairness and improves the model's capacity utilization by diversifying expert deployment across various tasks and contexts.

5.3 Fairness-driven Expert Management (FEM)

Our approach incorporates a comprehensive strategy that includes both dynamic expert allocation and fairness-dependent constraints across experts. These mechanisms are designed to enhance the network's performance and fairness dynamically, adapting expert utilization based on the evolving requirements of the task and fairness objectives.

Attribute-Focused Expert Specialization. In our FairSMoE model, we strategically assign each expert to × N

Figure 3: Framework overview. (left) Overall architecture of FairSMoE. Our proposed method replace the FFN by modified SMoE layer in the last ViT block. (middle) We re-design the SMoE layer by adopting Fairness-Guided Routing (FGR) mechanism and Fairness-driven Expert Management to enforces attribute-focused constraints and dynamically allocate experts for tasks according to fairness performance. We train the model in a multi-task learning way with 1 primary task and m fairness tasks. (right) FGR consists of two branches, one is Gating Branch, the vanilla gating network. Another is Sensitive Attribute (SA) Branch, encoding fairness information into routing consideration.

focus on unique sets of sensitive attributes to foster diverse expertise. Experts are initially assigned attributes based on potential and adjusted dynamically to minimize knowledge overlap and maximize fairness. The specialization is enforced through a tailored loss function:

Here, Se is the set of attributes assigned to expert e, and ϖ controls the balance between focusing on assigned attributes and avoiding non-assigned ones. Lossfocus(e↪ a) and Lossavoid(e↪ a) could be a standard loss function such as crossentropy loss or mean squared error, depending on the task. In this task they are both standard cross-entropy loss.

Periodic performance reviews guide dynamic reassignments, ensuring experts develop deep, relevant expertise without redundancy. The routing logic is also adapted to align with these specializations, optimizing decision-making and enhancing model adaptability and fairness. This integrated approach ensures that experts not only excel in their designated domains but also contribute effectively to the model's overall accuracy and fairness.

Fairness-aware Expert Allocation. Recognizing that different sensitive attributes may require varying levels of optimization complexity, we have implemented a dynamic expert allocation mechanism. This data-driven approach dynamically adjusts the number of allocated experts based on real-time assessments of fairness and performance. Initially, we evaluate the single-attribute fairness score (SFi) for each task on the validation set. If SFi remains stable or improves over n iterations, we consider increasing the number of experts assigned to that attribute. Conversely, if adding more experts results in a worse validation loss than previously observed, it indicates potential overfitting or interference among experts, suggesting a reduction in the number of allocated experts.

This method ensures that each expert's capacity is fully utilized, avoiding underutilization or overload, and aligns expert deployment with the fairness needs of each attribute.

Implementation of Expert Management. Our method is outlined in Appendix as Algorithm 2. The combined fairness-dependent constraints and dynamic expert allocation forms the core of our fairness-guided expert management module. This module supports the model in achieving high accuracy while ensuring multi-attribute fairness performance.

Loss Function The overall loss function for the FairSMoE framework is formulated to balance performance, fairness, and expert specialization:

Algorithm 1 Fairness-Guided Routing (FGR)

6 Experiments

6.1 Implementation Details

Dataset and Network Backbones. We evaluate our methods on the ISIC 2019 and CelebA datasets, primarily for skin lesion analysis and facial attribute recognition tasks, respectively, with more details in Appendix. To demonstrate our generalization, we choose DeiT-Small, DeiT-Base, SwinSmall, and Swin-Base for backbones. As discussed in Discovery and analysis, we follow the observation that applying the SMoE layer in the last transformer block will obtain the best performance.

Baselines. To demonstrate the effectiveness of FairSMoE, we consider three groups of baselines for comparison: (1) transformers with vanilla SMoE layers, (2) Muffin [Sheng et al., 2023] with model fusion, and (3) MultiFair [Tian et al., 2024] with data augmentation.

Training and Evaluation Settings. We applied a batch size of 256 and data augmentation of RandomResizedCrop for all methods on both datasets. Transformers are optimized with AdamW with weight decay of 1 ↔ 10↑4 , initial learning rate (LR) of 5 ↔ 10↑4 . Training epoch is set to 300 for ISIC2019 and 500 for CelebA. We randomly separate ISIC2019 80:20 for training and test, and randomly select 5% of training set for validation. We set ϖ as 0▷6 in Equation (8) and ε as 0▷1 in Ltotal. To evaluate the fairness performance, We use multi-attribute fairness MF! as metric, which is defined in Equation (1). 4 Nvidia A100s are used for training and testing. We are using overall loss adding eq. (4), Equation (7) and Equation (8) together.

6.2 Experiment Results

Comparison with vanilla SMoE and other approaches. We selected two representative vision transformer models and their four variants, DeiT-Small/Base and Swin-Small/Base, to test the effectiveness of FairSMoE. "Vanilla SMoE" refers to the model where we replace the last layer of each backbone with a vanilla SMoE layer, setting the number of experts to 4, and excluding any auxiliary loss or noisy gating. For Muffin's configuration, we used two backbones without SMoE, following the default fusing method. For MultiFair, we used a single backbone without SMoE, implementing only the data augmentation method proposed in MultiFair. The results are shown in Table 1 and several observations can be drawn: (i) Performance and Fairness Advancements: FairSMoE significantly outperforms existing methods like vanilla SMoE, Muffin, and MultiFair in terms of accuracy and fairness. For example, on ISIC2019, FairSMoE enhances accuracy up to 78▷95% and improves the PQD metric to 0▷828 on Swin-Base, demonstrating its effectiveness in balancing performance with fairness; (ii) Efficiency in Parameter Utilization: Despite its enhanced capabilities, FairSMoE maintains efficiency, using no more parameters than the least complex models. Compared with only 25M parameters for DeiT-Small, it outperforms models with up to 173M parameters, showcasing its ability to achieve optimal performances with a lightweight network. (iii) Empirical Validation: The empirical evidence supports the argument that FairSMoE's integrated approach to managing multiple sensitive attributes simultaneously leads to higher performance and fairness. The results validate our design philosophy, which leverages a nuanced understanding of attribute interactions within neural networks to dynamically adjust to varied dataset characteristics and fairness requirements.

Ablation Studies We conducted ablation studies on each component in FairSMoE. We conducted experiments on ISIC2019 dataset with all three attributes, with DeiT-Small as a backbone. As shown in Table 2 and Figure 4, we conducted experiments on Fairness-Guided Routing (FGR), Fairnessdriven Expert Management (FEM), and the number of experts. Our results demonstrate that (i) the proposed FGR is more effective compared with the vanilla router, demonstrating better accuracy and PQD score compared with the baseline. (ii) when equipped with FEM, the performance is also boosted by 1▷42% in accuracy and 0.01 in PQD score, which shows the necessity of selecting the appropriate capacity for each task. (iii) when the number of experts arises, performance on both accuracy and fairness are enhanced. Meanwhile, for vanilla SMoE, the performance on both aspects drop for more experts due to low utilization rate. These promising results demonstrate that FairSMoE has enhanced SMoE with better scalability. We also conducted empirical studies on ϖ in Equation (8). Results are shown in the Appendix.

6.3 In-Depth Discussion of FairSMoE

Given the superiority of our FairSMoE, we further investigate (i) its expert specialization and routing quality, and (ii) the mitigation effects on gradient conflicts from multiple training objectives.

FairSMoE alleviate the unbalanced routing schedule. As mentioned in Discovery and analysis, the default routing mechanism[Shazeer et al., 2017] can not sufficiently accommodate the diversity of multiple sensitive attributes embedded into the input and will tend to develop a deterministic routing pattern and over-utilization of certain experts, which will get worse when the expert number increases. One key advantage of SMoE is to regularize each expert to focus on a set of attributes and improve the overall utilization rate with the increasing expert number. Figure 5 demonstrates the expert activation frequencies on both vanilla SMoE and FairSMoE. We observe that more experts are activated during the routing, which indicates a better utilization rate and leads to better performance on both accuracy and fairness as pointed out in Ablation Studies.

FairSMoE mitigate the issue of gradient conflict among different task. The FairSMoE approach mitigates the issue of gradient conflict among different tasks by employing disentangled representation learning and confusion loss, which ensure that features are insensitive to sensitive attributes. This reduces the interference between tasks related to different attributes, leading to more coherent gradient directions and improved model stability during training. Figure 6 compares the cosine similarity between gradients computed from the age and site tasks on ISIC2019. The distribution of gradients is more skewed towards 1, indicating higher cosine similarity and reduced gradient conflict.

7 Conclusion

In this paper, we propose FairSMoE, a framework designed to address multi-attribute fairness problem in vision recognition. Our method modifies traditional SMoE models by incorporating fairness-guided routing and dynamic expert management to optimize expert utilization and minimize bias. Through comprehensive testing on multiple datasets with 4 mainstream ViT backbones, we demonstrate notable enhancements in both performance and fairness, also the great generalization and scalability capability on different settings.

Acknowledgments

This work is partially supported by the U.S. National Science Foundation (NSF) CNS-2312158, NSF CMMI-2125326 and Fundamental Research Funds for the Central Universities, Peking University.

Contribution Statement

* Changdi Yang and Zheng Zhan contributed equally. Work was done during Zheng Zhan's PhD study in Northeastern University. ‡ Pu Zhao and Yanzhi Wang are the corresponding authors.