Heterogeneous Federated Learning with Scalable Server Mixture-of-Experts

1 Introduction

Federated Learning (FL) [McMahan et al., 2017] has become a widely adopted approach for distributed learning from diverse data sources while preserving data privacy. However, classical FL requires the learning model to be identical across all clients for possible parameter averaging. This requirement poses challenges in environments with constrained client-side capacities such as edge devices, rendering standard FL unsuitable for learning large language models (LLMs) and large visual models (LVMs) with many edge devices.

Recognizing this critical limitation, a question comes to us naturally: can we deploy asymmetric models at client and server levels? Given that clients typically possess relatively scarce data and limited computational capabilities, their models should be designed to be compact and efficient. Conversely, the server boasts substantial computational resources, enabling it to leverage significantly larger and more complex models. Thus, the second question arises: how can we perform model averaging for asymmetric client and server models?

We are seeking the answer from the recently explored Mixture-of-Experts (MoE) [Shazeer et al., 2017] architecture. A MoE comprises of multiple expert sub-networks, each tailored to a specific segment of the input space. A learnable gate dynamically routes input samples to the most suitable experts. Inspired by this approach, we propose a novel design where identical compact client models are deployed at the client side, while a large MoE model resides on the server side. Each expert within the server MoE shares identical architecture with each client model .

We introduce Fed-MoE, a Federated Mixture-of-Expert system that enables the aggregation of compact client models into a powerful and large central model. In Fig. 1, we depict this practical scenario featuring a M distributed power-efficient clients, each having with a compact single model. Each client model contributes to one or multiple relevant experts within the server's MoE. This collaboration between client and server models, detailed in the subsequent paragraphs, aims to enhance overall performance and efficiency. With Fed-MoE, thousands of users can collaborate to build a unified billionscaled large model on the server side, leveraging expertise from all client data.

Our approach involves a three-stage iterative process for updating the server's MoE gate and experts from heterogeneous client models. To facilitate this, we postulate the existence of a small, reserved dataset at the server. We also made substantial re-design of the classical model averaging mechanism. In the first stage, we calculate the relevance between each client expert and each server MoE expert. Then we update the server's MoE experts through a weighted FedAvg process, with weights derived from the relevance scores.

In the second stage, we focus on updating the MoE gate.

Initially, we compute the gating probabilities on the reserved data instances. Subsequently, we aggregate predictions from the top-K most activated experts and calculate the classification loss based on the gating outcomes. Finally, we update the gate parameters end-to-end to minimize this loss.

In the third stage, we synchronize the updated server experts back to each client. By computing a server-to-client correlation matrix, we gather the top-K most relevant server experts for updating each client model. After, the clients perform local updates and send to the server for the next round of FL. Subsequently, the expertise of server experts gradually aligns with the global data space across all clients.

During inference, the MoE gate selectively activates only the relevant subset of experts for each incoming data, reducing computational costs and diversifying expert functions.

To further tackle the non-IID data issue over the clients, at the server-MoE level, we design two types of MoE experts: a main expert which always activates, and a set of routed experts which share activation together. The main expert dedicates to capturing common knowledge while the routed experts focus on learning specific client data classes. This design enables different routed experts to capture the unique data patterns from different clients. To facilitate the diversification of the routed experts, we further introduce a novel Gating Entropy loss, which encourages a sharply peaked gating distribution over the routed experts.

Our contributions are summarized as follows:

1. We propose an effective Federated Mixture-of-Expert learning framework that allows for the deployment of a large number of compact client models while maintaining a unified large MoE at the server-side.
2. To address the Non-IID issue, we devise a main expert that captures common knowledge, and a set of routed experts that share activation to learn specific client data classes.
3. We design efficient server MoE-experts and MoE-gate update mechanism with innovated gating entropy loss to ensure diversification of the routed experts.
4. Our Fed-MoE shows promising results upon benchmark FL datasets in large-scale vision and language tasks.

2 RelatedWork

Federated learning (FL). FL [McMahan et al., 2017; Zhao et al., 2018; Sattler et al., 2019; Li and others, 2019; Wu et al., 2020; Karimireddy and others, 2020] emerges as a decentralized and privacy-preserving learning strategy. The pioneering FedAvg [McMahan et al., 2017] demonstrated the effectiveness of model averaging from separately trained client models. Many recent studies worked on tackling the Non-IID setting [Zhao et al., 2018; Sattler et al., 2019; Li et al., 2020], few-shot setting [Wu et al., 2020; Itahara et al., 2023; Jiang et al., 2024b] and privacy enhancement [Wei and others, 2020; Xin et al., 2020; Liu et al., 2023; Fan et al., 2022] of client data in FL scenarios. Some approaches focus on handling model heterogeneity, including HeteroFL [Diao et al., 2021], FedHM [Park and Ko, 2024], FedRolex [Alam et al., 2022], and Split-Mix [Hong et al., 2022].

Mixture-of-Experts (MoE). MoE techniques [Jacobs et al., 1991; Jordan and Jacobs, 1994] utilize a set of expert networks are combined under a gating module which specialize in different aspects of the input data. Recently, MoE has been applied in language modeling with notable successes. The sparsely-gated MoE [Shazeer et al., 2017; Zuo et al., 2021; Jiang et al., 2024a] can scale up the model capacity with less increased computational complexity. GShard [Lepikhin et al., 2020] enable the scaling of multilingual neural machine translation models using sparse gating and automatic sharding. Switch Transformers [Fedus et al., 2022] further scales up Transformer models to trillion-parameter models. GLaM [Du and others, 2022] reduces both training and inference costs of a large MoE language model. MoE has also been widely applied in building large vision-language models [Mustafa et al., 2022; Lin et al., 2024]. DeepSeek-MoE [DeepSeek-AI, 2024] first proposes the concept of shared experts and routed experts.

Recently, some FL studies have made attempts to use MoE for personalized learning. FLMoE [Zec et al., 2020] directly applies FL for MoE models to better suite to heterogeneous client data. AEPFL [Isaksson et al., 2022] achieves a more adaptive cluster model by balancing exploration and uses the cluster model as an expert model in the MoE to enhance performance. PFL-MoE [Guo et al., 2021] modifies the MoE architecture to enhance decision-making capabilities. FedMix [Reisser et al., 2021] directly employs one global MoE to mitigate the Non-IID data by segmenting data source regions. FedJETs [Dun et al., 2023] reduces communication costs by selecting a subset of experts that match the features of client data for communication. Differently, our method allows the server MoE to have a heterogeneous number of experts.

3 Task Description and FL Preliminaries

We construct a practical scenarios in which models at clientside and server-side are asymmetric such that the client model is compact while the server model is a large unified MoE.

Model at client side. We consider the FL scenario withM distributed clients, each having a compact single-expert model. Let Mi be the i-th client model parameters. At each FL round, we randomly select m participating clients.

MoE at server side. We deploy a large K-expert MoE at the server side, including two types of experts: a main expert which always activates, denoted as E0 and a group of K routed experts denoted as E1:K. The routed experts has a trainable gating module G responsible for activating corresponding routed experts. For brevity, we use K∗ = 1 +K to denote the number of both main and routed experts.

Server reserved data. We assume the server possesses a tiny reserved dataset Dr , sampled uniformly from the global data space as prior knowledge. The Dr assists training an effective server gating module exclusive to the server side, which is non-overlapping with client data, thus adhering to federated learning principles.

Fed-MoE task formulation. Let G(x) be the gating probability of input x over K routed experts, and Ei(x) be the i-th expert network which predicts input x. The collective response of the MoE module can be expressed as a weighted sum of all experts as follows:

where α balances main-expert and routed-experts.

Training objective. Let nk be the number of data samples for client k. The global learning objective is to minimize the average loss over each client on local data as follows:

where ˆ yi is server model prediction following Eq (1).

Following FL, we decompose the above global training objective into training each client model wc . Then the weights of the server experts E1,...,K∗ are aggregated from client model wc1,...,M , denoted as: E ← Fuse(wc ). We will devise an effective Fuse function which enables dynamic expertise dispatching to enhance server experts from client models.

4 Our Fed-MoE Approach

We decompose our Fed-MoE framework with three stages which we introduce in below sections.

4.1 Stage-A: Local client training and uploading

Following standard FL procedure, each FL round starts by training client models with local data. After, m clients are randomly selected to upload their local models to the server through networks, as shown in Stage-A of Fig. 2. The server aggregates these m client models as a federation denoted as M = {M1,M2, ...,Mm-1,Mm}.

4.2 Stage-B: Server experts and gate update

Stage-B iteratively updates server experts E and gate G for T iterations, which we decompose as the following steps.

Step-0: Probe client experts' responses

We firstly get client experts' responses over reserved set Dr in order to learn their expertise of data classes. For each data instance (X, y) ∈ Dr (e.g., image-label pair), we feed to all m client experts and obtain the C-way classification probability distribution as P ← M(X) ∈ Rm×C . With ground-truth label y, we get the true class prob. as confidence level of each expert as Py = P [:, y] ∈ Rm×1 .

Step-1: Get server gating responses

Next, we will repeat Step-1 to Step-4 for T iterations as an inner-loop to update server gate and α.

At t-th iteration, we begin by feeding data X to the gating module G, providing the activation prob. distribution as:

Q gives soft assignment of query data X to K server experts, which has been normalized with softmax.

Step-2: Get server-client correlation

The outer product of Q and Py is a correlation matrix:

where Wi,j measures the relevance between the i-th expert of the server MoE and expert from collected j-th client expert.

We subsequently apply a row-wise softmax operation to normalize the correlation matrix as:

where each rowW r i,· sums to one and indicates the correspondence between server expert-i with all m client experts.

Step-3: Update server experts with moving FedAvg

At t-th iteration, we denote the K∗ server experts as mainexpert Et0 and routed experts Et i for i ∈ {1, ...,K} which get updated with moving-average strategy as follows:

The λ ∈ (0, 1) controls the moving-average rate. We use simple averaging for ¯M with ¯M = 1/m ∑mi=1 Mi. The termW rM assigns relevant client parameters weighted by correlation W r and adds up to server weights.

Step-4: Update server gating module

In an ideally diversified MoE system, the gate G learns to route queries to relevant experts with the highest precision for that data class. To this end, we design the learning objective to be the cross-entropy task loss with gating entropy regularization, outlined as follows.

Task loss. For each data in reserved set {X, y} ∈ Dr , we estimate C-way distribution from the main expert as ˆ P0 ∈ R1×C and K routed experts as ˆ PX ∈ RK×C .

The MoE gate weighs each routed expert by the activation QX ← G(X) ∈ RK . We combine all experts' output such as ˆ P ∗X = (1-α) · ˆ P0+α· ˆ PX as Eq.(1). The task cross-entropy loss is the negative log likelihood such as:

Regularization. Intuitively, a sharply peaked gate activation Q implies assigning the data to a specific expert with high confidence. To encourage this desirable expertise diversification, we introduce a novel Gating Entropy (GEnt) loss:

in which QX[k] indicates the probability (priority) of assigning X to k-th routed expert. We take Lent G as a regularization term that we seek to minimize.

The joint gating loss w.r.t. gate G and hyper-parameter α is

where we set β = 10-3 and discuss in Ablation Table 5.

During inference, we choose routed experts with top-L gate activation with indices I ← arg TopL(QX). Then we integrate main expert and top routed experts' predictions reweighted by the activation values after softmax, such as:

The complete steps of Stage-B are summarized in Algo. 1.

4.3 Stage-C: Client experts synchronization

We update m participating client experts accordingly for next round of local training. As shown in Stage-C of Fig. 2, we decompose this process as follows. We firstly refresh the server-client correlation matrix W ∈ RK×m with server MoE refined in Stage-B, by executing Eq.(3)-(4). We also synchronize the main expert from the server to the client based on the present value of α, as demonstrated by the equation below:

We subsequently apply a column-wise softmax operation to correlation matrix W ′ , yielding a normalized server-to-client correspondence as:

in which each column W c ·,j measures the normalized relevance of all server experts with the j-th client expert. This relevance

Algorithm 1 Fed-MoE overview.

while round e ≤ E do Stage-A: Local client training and uploading /* Upload m participating client models to server. */ M ← {M1,M2, ...,Mm-1,Mm}. Stage-B: Server MoE iterative update /* Step-0: Probe client experts' responses on Dr . */ P ←M(X), ∀(X, y) ∈ D // client responses Py ←P[:, y] ∈ RK×1 // label confidence while t ≤ T do /* Step-1: Get server gating responses. */ Q←G(X) ∈ RK×1 // gating of Eq.(3) /* Step-2: Get server-client correlation. */ W r ← σrow (Q · P ⊤ y ) ∈ RK×m /* Step-3: Update server experts by moving FedAvg. */ Et+1 0 ← (1 -λ) · Et0 +λ · ¯M, Et+1 i ← (1 -λ) · Et i +λ ·W rM , ∀i ≥ 1 . /* Step-4: Update server gating module. */ Gt+1 ←Gt -η · ∆Lgate , αt+1 ← αt - η · ∇Lce G . // Loss Eq.(9) Stage-C: Synchronize model E back to clients. /* Get updated server gating. */ Q←G(X) // G updated in Stage-B /* Get updated server-client correlation. */ W c = σcol (cat(1 - α, α · (Q · P ⊤ y )) ∈ RK∗×m /* Update client experts by moving FedAvg. */ M = λ ·M+ (1 -λ) · (W c )⊤ · E // Eq.(13)

can guide each client expert Mj to gather parameters from server experts E with weights such as (W c ·,j )⊤E.

We formulate the update procedure of client experts M with exponential moving FedAvg in matrix form as:

in which λ ∈ (0, 1) controls the moving average rate. Finally, the server transmits the client experts to their corresponding owners for next round of local training. The above process is shown in Fig. 2 (Stage-C) and Algo. 1 (Stage-C).

5 Experiments

We verify our approach on the benchmark Federated Extended MNIST (FEMNIST) [Caldas et al., 2018], CIFAR10 [Krizhevsky, 2009] for image classification task, SENT140 [Caldas et al., 2018] for textual sentiment classification task and YELP [Zhang et al., 2015] for 5-way review star classification task.

Vision data split. We follow the original Non-IID split of FEMNIST according to different writing styles of 3500 users. We choose 50 and 100 clients as two FL scenarios for FEMNIST, each having 6200 and 5650 data, respectively. On CIFAR-10, we simulate highly Non-IID scenarios by distributing data classes using a Dirichlet distribution (α=1.0) to ensure that each client gets a unique, proportionately varied subset of classes. For 50 and 100 cases, each client has 750 and 375 data, respectively.

On the sentiment analysis benchmark SENT-140 [Caldas et al., 2018], we follow Fan et al. 2022 to evaluate as a binary classification task. We reserve 100 clients, each having 190 sentences for each class. The server reserves |Dr | = 1000 sentences. We tokenize each sentence to a max of 64 words.

The Yelp 5-way classification task aims to predict the number of stars for a review on a scale of 1 as most negative to 5 as most positive. We configured 100 clients, where each client gathers 5,000 data samples. The data is partitioned in a Non-IID fashion, ensuring that one class predominates on each client. The server reserves |Dr | = 1000 samples. We tokenize each sentence to a max of 64 words.

5.1 Fed-MoE and Baselines for comparison

Model architectures. In our settings, each client model and each server expert shares a same architecture. The difference is that the server MoE includes a main expert and K = 5 routed experts, thus having much more parameters than a single client model. Each single model is a 2-layer CNN for FEMNIST, a ResNet-18 for CIFAR, BERT for SENT-140 sentiment analysis, and GPT-2-Medium for Yelp. Due to GPU resource constraints, in the experiments on SENT140 and Yelp, we use a main expert and K = 3 routed experts on the server side. Specifically, for the Yelp experiments, we employ GPT-2 as the gate model for the MoE. The model size and communication costs are summarized in Table 2.

FLbaselines. FedAvg [McMahan et al., 2017] uses standard parameter averaging for model fusion. FedProx [Li et al., 2020] adds a proximal term to regularize the client update from deviating too far from the global model. Both methods do not possess MoE parameters and follow standard FL training.

MoE baselines. Cent-MoE (Centralized MoE) trains a plain 5-Exp MoE [Shazeer et al., 2017] only with the server reserved set, without using any client data. FedMix [Reisser et al., 2021] has a 2-Exp MoE which employs a direct FedAvg to aggregate both client experts and their gate modules. Thus its server MoE is also a 2-Exp server MoE. FedJETs [Dun et al., 2023] consists of m = 5 anchor clients and M - m ordinary clients, each having a 2-Exp model. In each FL round, m anchor clients as well as m randomly selected ordinary clients participate in learning.

5.2 Implementation details

Experimental Settings. We perform all experiments on a system with 3 Nvidia 4090 24G graphics cards, with M = 50 and 100 clients to build a large-scale FL system. We set a K-expert server Fed-MoE framework, where a main expert aggregates model parameters from the activated m-clients in each iteration. For vision tasks, we set K = 5 and m = 5, applying a 2-layer CNN for FEMNIST and ResNet-18 for CIFAR-10. For language modeling tasks, we set K = 3 and m = 3, using BERT-base for SENT-140 and GPT-2 for YELP which have billion of parameters. Details are in Table 1. The learned routed-experts importance α in Eq. 1 of FEMNIST, CIFAR, SENT140 and Yelp are 0.56, 0.38, 0.48 and 0.50.

Communication cost and model complexity. We show the FedAvg, FedMix and Fed-MoE parameters and FL communication costs in Table 1. All MoE-based methods select m active clients for each round of learning with cost of m model parameters. FedMix and FedJETs have to upload the client gates to the server with additional costs.

For Yelp task, FedAvg has about 0.36B parameters of a single GPT-2 model at the server-side. Our Fed-MoE approach, on the other hand, has 1.59B parameters, comprising a main expert, three routed experts, and a router. In terms of inference workload, Fed-MoE is comparable to FedMix, as only the top-2 experts (including the main expert) are selected for activation. Also, the communication cost for Fed-MoE is only 1.02B, significantly less than that of FedMix, which incurs a substantial 2.79B due to the transfer of additional MoE modules on the client-side.

5.3 Results with Various Datasets and Settings

Vision tasks. Table 2 first two columns (shaded in yellow) present the classification results for FEMNIST and CIFAR-10 datasets under client configurations of (M = 10, 50, and 100).

On FEMNIST, Fed-MoE consistently outperforms baseline methods across all client configurations. For instance, at M = 50 clients, Fed-MoE achieves over 10% higher accuracy than FedAvg and FedJETs, and approximately 4% and 9% higher than FedMix and FedProx, respectively. We observe that as the number of clients increases and each client possesses less data,our Fed-MoE method gradually decreases, showcasing a larger performance advantage compared to other methods (FedAvg and FedProx).

On CIFAR-10, a similar trend is observed. Fed-MoE achieves 5-7% higher accuracy than FedMix and 1-8% higher accuracy than FedJETs as the number of clients increases from M = 10 to M = 100. This indicates Fed-MoE's robustness in large-scale federated learning scenarios where Non-IID data and limited per-client data pose significant challenges.

FedMix and FedJETs rely on direct aggregation of gate and expert models, which does not sufficiently account for each server's expertise, leading to suboptimal performance in Non-IID settings. In contrast, our proposed Fed-MoE features an effective expertise dispatching mechanism. It fuses client models into relevant server experts and adaptively updates the gating module to align with evolving expertise.

Language tasks. We show SENT-140 and YELP results in Table 2 columns shaded in yellow. On SENT140, having a BERT [Devlin et al., 2018] for 2-way sentiment classification, we observe that Fed-MoE averagely outperforms the 2ndplace FedMix by 1.7%, FedProx by 2.5% and FedAvg by 2.8%, while leading the weakest FedJETs by 7.45%. On YELP, having a GPT-2 [Devlin et al., 2018] for 5-way task, we observe the Fed-MoE averagely outperforms the 2nd-place FedProx with a 1.2% lead and 2.1% over FedMix. The plain FedProx even surpasses FedMix and FedJETs, implying that a simple average of MoE gate would harm the training in experiments with large-scale models in Non-IID settings.

Fed-MoE outperforms at large-scale FL case. Our FedMoE consistently outperforms FedMix and FedJETs, achieving 6-7% higher accuracy on vision tasks and 4-5% on language tasks in 100-client case. This shows the robustness of our proposed approach in Non-IID scenarios with numerous clients. The key advantage lies in our expertise dispatching mechanism, which dynamically considers the model capacity of each client to enable more effective aggregation. This contrasts with the direct aggregation of gate and expert models employed by FedMix and FedJETs.

Varying reserved data size. We vary the server reserved data size |Dr | and report the corresponding accuracy of FedMoE. For FEMNIST, the reserved data of sizes |Dr | = 320/640/1280 yield accuracy of 82.86, 82.91, 86.03. For CIFAR, sizes |Dr | = 250/500/1000 yield 55.79, 63.31, 65.52. This indicates that a reasonably reserved dataset is necessary for training gating module.

5.4 Ablation Studies

Due to the time and computation budget, the following ablations are carried on with 50-client setting.

Server MoE size. We further study how the number of routed-experts at server side affects the performance. Table 3 shows that Fed-MoE achieves the best performance (86.03%) with a moderate 5-Exp MoE. In contrast, though Avg-MoE improves as more experts participate in the training, the overall improvement effect is very limited, and still under-performs 5-Exp Fed-MoE. This indicates the effectiveness of expertise dispatching of Fed-MoE.

Multi-task training procedures. We examine the effectiveness of the proposed gating entropy (GEnt) loss in Eq.(8) in training objective, as well as client synchronization mechanism in Eq.(13) in Table 4. We observe that the inclusion of gating entropy (+GEnt) alone leads to a slight increase in performance (0.5% for FEMNIST and 0.8% for CIFAR), while client synchronization (+Sync) alone results in a performance boost of 3.4% and 2.6%. Combining Sync and GEnt together yields a huge improvement of 8.0% and 4.2%. The rationale is that GEnt encourages specialization of each server expert, while Sync creates a unified data space across all clients. By leveraging both, server experts can specialize appropriately in tasks within the global data space, thereby improving their effectiveness, particularly in Non-IID case.

Weight of Gating Entropy. We study how the weight β of GEnt loss of Eq.(9) affects accuracy and explanability. We observe in Table 5 that a moderate weight β = 10-3 of GEnt achieves the best accuracy of 65.52% on CIFAR, better than the model without using GEnt (63.94%) or with a tiny weight 10-4 (64.25%). Also, increasing weight to 10-2 and 10-1 decreases the accuracy. This is because a larger GEnt increases the specificity of each expert, but reducing the versatility of the ensemble experts.

We visualize the gating distribution over each server expert in Fig. 3. In left heat-map, setting a large gating entropy weight of β = 10-1 obviously makes the gating sparse. Each expert specifies to one certain digit, e.g., the gating module sends over 60% label "1" (2nd column) and label "2" (3nd column) to expert-4. With a smaller β = 10-3 as the right heat-map shows, the gating module assigns soft weights to more experts for a same class. It further illustrates that β = 10-3 achieves a balance between specificity and versatility, distributing the gating more evenly across multiple experts while still maintaining strong accuracy.

Inference withTop-L active routed experts. Table 6 shows that on both FEMNIST and CIFAR, using Top-1 routed expert achieves the highest 86.03% and 65.52% accuracy, respectively. In contrast, using Top-5 would lower the accuracy, while costing 5 times computations. Since each expert specializes in certain classes, activating more irrelevant experts can lead to confusion in the final outcomes. This aligns with our practice of using gating entropy to diversify experts.

Comparison with MoE baselines w/o dynamic update. We study the effectiveness of dynamic update of server experts, as elaborated in Sec. 4.2. We build the following MoE variants. Cent-MoE trains a centralized 5-Exp MoE only with the server reserved set. Avg-MoE [Reisser et al., 2021] (a.k.a FedMix) has five clients, each equipped with a 2-Exp MoE. Avg-MoE aggregates client models into a 5-Exp MoE server model with FedAvg. Anchor-MoE adopts FedJETs [Dun et al., 2023] by configuring a 1-Exp model for each of the 5 anchor clients. The server has a 5-Exp MoE. For each FL round, all 5 anchor clients as well as 5 randomly chosen ordinary clients are used to update server MoE. We show results in Table 7.

Fed-MoE outperforms other models due to its dynamic update of the server gate instead of relying solely on aggregation from clients. Cent-MoE performs the worst as it only utilizes server reserved data. Avg-MoE ranks the second, as it aggregates all client MoEs but lacks client-to-server matching. Anchor-MoE ranks behind Fed-MoE, as it restricts itself to a fixed correspondence between client and server experts, thereby limiting its performance.

Comparison with heterogeneous method. We further compare with FedRolex [Alam et al., 2022], a dynamic subnetwork-based FL method, under 100 clients setup. FedRolex achieves 45.50% accuracy, which is significantly lower than our Fed-MoE (60.73%). This demonstrates that even dynamic subnetwork methods struggle under extreme non-IID and large-scale FL settings.

Effect of main-expert. We study how the number of mainexperts affects the performance. Table 8 shows that FedMoE achieves the best performance on both FEMNIST and SENT140 with just 1-Exp MoE, compared with using no main expert and 2/3-main experts.

Non-IID server reserved dataDr . We let 60%of the server data concentrate on one class, with the rest uniformly distributed. Fig. 4 shows an accuracy gap of about 2-3% between IID and Non-IID scenarios for both Fed-MoE and FedMix, and less 1% in AUC. The F1 score reveals a gap of 6% on Yelp, whereas it is only 1.8% on SENT140, for our Fed-MoE. Otherwise, Fed-MoE showed a slight advantage than FedMix in AUC metrics on both datasets.

6 Conclusion

We introduce an efficient asymmetric FL scheme that efficiently aggregates compact client models with a server-side MoE composed of main experts and routing experts, showcasing superior performance across visual and language tasks. Our dynamical expert gating and updating mechanisms help establish diverse and capable server MoE from client experts. We validated our approach on billion-scale MoE systems with large models, extending its applicability to tasks like image and text classification. Detailed ablation studies confirm its efficiency in convergence and communication performance.

Acknowledgments

This work is supported by the GuangDong Basic and Applied Basic Research Foundation (2024A1515011650) and the National Natural Science Foundation of China (62106156). We thank all reviewers for constructive suggestions.

Contribution Statement

This work was a collaborative effort by all authors. Jingang Jiang† and Yanzhao Chen† contributed equally to this study and are designated as co-first authors. Chenyou Fan∗ served as the corresponding author and is responsible for all academic correspondence regarding this manuscript.