FedSSA: Semantic Similarity-based Aggregation for Efficient Model-Heterogeneous Personalized Federated Learning

1 Introduction

As societies become increasingly aware of the importance of data privacy protection, centrally collecting large-scale data for model training has become less viable, especially under privacy regulations. To enable decentralized data owners to collaboratively train effective machine learning models while protecting privacy, federated learning (FL) [McMahan et al., 2017] has been proposed. In a typical FL system, a central FL server coordinates multiple data owners (FL clients) for model training. In each communication round, the server broadcasts its current global model to the clients. Each client then treats the received global model as its local model, and trains it on local private data. The updated local model is uploaded to the server. The server weighted averages (aggregates) the received local models to obtain an updated global model. The above steps are repeated until the global model converges. In FL, only model parameters are communicated between the server and clients. Potentially sensitive local data are not exposed, thereby protecting privacy [Liu et al., 2024].

⇤Corresponding authors.

This model homogeneous mode of FL requires all clients to train models with the same structures. It is not wellsuited for scenarios involving data heterogeneity [Zhu et al., 2021] (i.e., clients with local data following non-independent and identical distributions (non-IID)) and system heterogeneity [Jiang et al., 2022; Yi et al., 2024b; Yu et al., 2011; Zhang et al., 2023] (i.e., clients with diverse communication and computation capabilities). Therefore, training a personalized FL model adaptively based on each client's local data distribution and system capability helps improve performances [Tan et al., 2022a; Yi et al., 2024a]. Furthermore, enterprise FL clients are often reluctant to share their model structures with others due to intellectual property concerns. Thus, the field of Model-Heterogeneous Personalized Federated Learning (MHPFL) [Yi et al., 2023a; Yi et al., 2023c;

Yi et al., 2023b; Yi et al., 2024c], which allows each client to train a heterogeneous personalized local model with a structure tailored to actual needs, has emerged.

Existing MHPFL methods are mainly designed based on knowledge distillation, mutual learning, and model mixup. Knowledge distillation-based methods [Li and Wang, 2019; Lin et al., 2020; Huang et al., 2022a; Itahara et al., 2023; Sattler et al., 2022; Makhija et al., 2022; Chang et al., 2021] often rely on the availability of a public dataset that is large enough and follows a similar distribution with local private data for fusing knowledge from heterogeneous local models. However, such a public dataset is difficult to obtain in practice. Other knowledge distillation-based methods [Jeong et al., 2018; Ahn et al., 2019; Ahn et al., 2020; Tan et al., 2022b] without requiring public datasets often incur high computational overhead on clients and face the risk of privacy leakage. Mutual learning-based methods [Shen et al., 2023; Wu et al., 2022] allow each client to train a large heterogeneous model and a small homogeneous model in a mutual learning manner. Only the small homogeneous models are uploaded to the server for aggregation. Training two models simultaneously leads to high computational overhead for clients. Model mixup-based methods [Collins et al., 2021; Liang et al., 2020; Jang et al., 2022] divide each client's local model into a heterogeneous feature extractor and a homogeneous classifier. The server aggregates the homogeneous local classifiers to update the global classifier and then broadcasts it to clients. Each client replaces its local classifier with the updated global classifier or constructs a regularization term based on the distillation loss between local and global classifiers. They often ignore the semantic information of the classifier parameters, resulting in limited accuracy improvements.

To enable the privacy-preserving, communication- and computation-efficient training of high-performance models through MHPFL, we propose the Federated Semantic Similarity Aggregation (FedSSA) approach for supervised classification tasks, which have been widely applied in fields like cancer detection for medical diagnosis and object recognition in autonomous driving [Kairouz et al., 2021]. As depicted in Figure 1, it splits each client's local model into two parts: 1) a heterogeneous feature extractor, and 2) a homogeneous classification header. Local-to-global knowledge transfer is achieved by semantic similarity-based classification header parameter aggregation. Since the parameters of classification headers from different clients corresponding to the same class are semantically similar, each client only needs to upload classification header parameters corresponding to locally seen classes. The FL server aggregates parameters from different classification headers by class to update the global header. Global-to-local knowledge transfer is achieved through parameter stabilization. To ensure stable convergence, we devise an adaptive parameter stabilization strategy that fuses local historical header parameters and the latest global header parameters for locally seen classes to update the local header of each FL client.

Owing to the above design, FedSSA has the following advantages: 1) aggregating semantically similar head parameters corresponding to the same class stabilizes the decision boundaries and boosts performance; 2) the adaptive parameter stabilization strategy alleviates parameter shaking in the initial training rounds, thereby speeding up convergence; 3) the server and clients only need to transmit classification header parameters corresponding to seen classes, which is communication-efficient compared to transmitting the entire model; 4) low computational overhead for FL clients as they only fuse partial parameters from the global and local headers before local training; and 5) since only partial parameters are transmitted between the FL server and FL clients, no new privacy risk is introduced. In short, compared with existing methods, FedSSA improves model performance, and communication and computation efficiency simultaneously.

Theoretical analysis proves the convergence of FedSSA. Extensive experiments on two real-world datasets against seven compared with state-of-the-art MHPFL baselines demonstrate that FedSSA achieves up to 3▷62% higher accuracy, 15▷54 times higher communication efficiency, and 15▷52 times higher computation efficiency.

2 RelatedWork

Our work is closely related to MHPFL with complete model heterogeneity. These methods support flexible model heterogeneity, and are divided into the following three categories.

Knowledge Distillation-based MHPFL. Most knowledge distillation-based MHPFL methods rely on a public dataset (e.g., FedMD [Li and Wang, 2019], FedDF [Lin et al., 2020], FCCL [Huang et al., 2022a], DS-FL [Itahara et al., 2023], CFD [Sattler et al., 2022], FedHeNN [Makhija et al., 2022], Cronus [Chang et al., 2021], FSFL [Huang et al., 2022b], FedAUX [Sattler et al., 2021], FedKT [Li et al., 2021], Fed-ET [Cho et al., 2022], FedKEMF [Yu et al., 2022], FedGEMS [Cheng et al., 2021], KT-pFL [Zhang et al., 2021]) to fuse information of local heterogeneous models by knowledge distillation. However, the public dataset is not always available. To avoid relying public datasets, FedGen [Zhu and others, 2021] and FedZKT [Zhu et al., 2022] train a generator for generating local representations or public shared datasets, which is time-consuming and computationintensive. In FD [Jeong et al., 2018], HFD [Ahn et al., 2019; Ahn et al., 2020], FedProto [Tan et al., 2022b], FedGKT [He et al., 2020], each client uploads the (class-average) logits or representations of local data to the server, the aggregated global logits or representations for each class are sent to clients for local knowledge distillation which incurs high computational costs for clients. Besides, uploading the logits/representations might compromise privacy.

Mutual Learning-based MHPFL. FML [Shen et al., 2023] and FedKD [Wu et al., 2022] enable each client to train a large heterogeneous model and a small homogeneous model in a mutual learning manner. The large model is always trained locally and the small model is uploaded to the server for aggregation. Although they implement information interaction through the small homogeneous models, training the homogeneous model increases local computational costs for clients, and transmitting the homogeneous models incurs high communication costs.

Model Mixup-based MHPFL. These methods split each local model into a feature extractor and a classifier. In FedRep [Collins et al., 2021], FedPer [Arivazhagan et al., 2019], FedMatch [Chen et al., 2021], FedBABU [Oh et al., 2022] and FedAlt/FedSim [Pillutla et al., 2022], the feature extractor is homogeneous and used for aggregation by the FL server to enhance generalization. The classifier can be heterogeneous. Since the feature extractor has more parameters than the classifier, these methods can only support model heterogeneity to a low degree. In contrast, LG-FedAvg [Liang et al., 2020], FedClassAvg [Jang et al., 2022] and CHFL [Liu et al., 2022] use heterogeneous feature extractors and homogeneous classifiers (i.e., executing the same classification task). The local classifiers are uploaded to the FL server for aggregation to generate the global classifier. To acquire global knowledge, LG-FedAvg directly replaces each client's local classifier with the global classifier. Each client in CHFL or FedClassAvg calculates the regularization term or distillation loss between the local and global classifiers. Although these methods support a higher degree of model heterogeneity, they ignore the semantic similarity of classifier parameters belonging to the same class, thus achieving limited performance improvement. Our FedSSA sets out to address the aforementioned limitations.

3 Preliminaries

3.1 Notations and Objective of Typical FL

A typical FL system involves a central FL server and N decentralized FL clients. In each round, the server selects a C fraction of clients at random. The selected client set is denoted as K, ♣K♣ = C · N = K. The server then broadcasts the global model f(!) (f(·) and ! denote the model structure and the model parameters) to the selected clients. A client k trains the received f(!) on its local dataset Dk to produce a local model f(!k) through gradient descent !k !-⌘r`(f(xi; !)↪ yi). `(f(xi; !)↪ yi) is the loss of the global model f(!) on the sample (xi↪ yi) 2 Dk. Dk ⇠ Pk indicates that Dk obeys distribution Pk (i.e., local data from different clients are non-IID). Then, client k uploads its local model parameters !k to the server. The server aggregates the received local models to update the global model, ! = PK-1 k=0 nk n !k. That is, the objective of typical FL is to minimize the average loss of the global model f(!) on data from all clients:

where ! are d-dimensional real numbers. nk = ♣Dk♣ is the number of samples in client k's local dataset. n = PN-1 k=0 nk. Lk(Dk; f(!)) = 1 ♣Dk♣ P(xi↪yi)2Dk `(f(xi; !)↪ yi) is the loss of the global model f(!) on Dk.

3.2 ProblemDefinition

The problem we aim to solve in this work belongs to the category of MHPFL for supervised classification tasks. Each FL client k owns local models fk(!k) with a model structure fk(·) and parameters !k. They can be heterogeneous for different clients. We assume all clients execute the same classification task and each client's local model fk(!k) consists of a heterogeneous feature extractor Fk('k) and a homogeneous classification header H(✓k), i.e., fk(!k) = Fk('k) ◦ H(✓k) with ◦ denoting model splicing. The feature extractor takes inputs as Fk(Dk; 'k), where Fk(·)↪ 'k denote the structure and parameters of the feature extractor. It maps data Dk from the input space to the feature space. The classification header takes input as H(Fk(Dk; 'k); ✓k), where H(·)↪ ✓k denote the structure and parameters of the classification header. It involves the last two linear layers of the model which maps the features Fk(Dk; 'k) extracted by Fk(·) to the output space.

Our objective is to minimize the sum of losses of all clients' heterogeneous local models:

where local models !0↪ ▷ ▷ ▷ ↪ !K-1 2 Rd0↪▷▷▷↪dK-1 .

4 The Proposed FedSSA Approach

FedSSA consists of two modules: 1) semantic similaritybased classification header parameter aggregation for localto-global knowledge transfer, and 2) adaptive parameter stabilization for global-to-local knowledge transfer.

4.1 Semantic Similarity-Based Aggregation

As shown in Figure 2, the last fully connected layer of the classification header outputs all-class logits which are mapped to the prediction probability for each class by a soft-max layer. That is, each neuron at the classification header's last layer corresponds to one class and its connections with all neurons at the previous layer are the parameters belonging to corresponding classes. Assuming that all clients execute the same S-classification task, then client k's local classification header ✓k can be subdivided by class as ¶✓0 k ↪ ▷ ▷ ▷ ↪ ✓sk ↪ ▷ ▷ ▷ ↪ ✓S-1 k ♦. Since local data from different clients are non-IID, different clients may hold partial classes of data. We denote the set of client k's seen classes as Sk.

Since each client trains its local model on locally seen classes, the local classification header seen-class parameters are well-trained. Therefore, client k only uploads its classification header seen-class parameters ¶✓sk ♦↪ s 2 Sk to the FL server for aggregation. As illustrated in Figure 2, client 1 only uploads its classification header parameters corresponding to seen classes '0↪ 1', client 2 only uploads its classification header parameters for classes '0↪ 1↪ 2', and client 3 only uploads its classification header parameters for class '1'.

Since classification header parameters for the same class from different clients are semantically similar, we design the server to aggregate classification header parameters based on classes (i.e., the corresponding neuron positions at the output layer). We denote the set of clients holding class s of data as Ks, and define the aggregation rule as:

After aggregation, the server splices the aggregated header parameters for all classes together to form an updated global header ✓ = ¶✓0 ↪ ▷ ▷ ▷ ↪ ✓s ↪ ▷ ▷ ▷ ↪ ✓S-1 ♦. Then, it sends the partial parameters of the global header to each client corresponding to its respective locally seen classes. This step transfers local seen-class knowledge from clients to the server. It enhances the generalization to seen classes for each client after receiving the aggregated seen-class header parameters, and produces a more stable seen-class classification boundary.

4.2 Adaptive Parameter Stabilization

Inspired by historical learning [Li et al., 2023] which tends to re-use historical knowledge from old models to avoid forgetting, during global-to-local knowledge transfer, we exploit historical local headers to stabilize and speed up model convergence. Specifically, in the t-th training round, client k fuses the seen-class parameters from its historical local header ✓t-1↪s k (s 2 Sk) and the latest global header ✓t-1↪s (s 2 Sk) to produce a new local header ˆ ✓t↪s k . The unseen-class header parameters are still the historical local header ✓t-1↪s0 k (s0 2 (S∩Sk)). The fusion of the seen-class parameters from the two headers is achieved by the proposed adaptive parameter stabilization strategy.

In the initial rounds of training, local model parameters are updated rapidly and local model parameters from different clients trained on non-IID data are biased. This leads to instability in the global header. To enhance the reliability of global-to-local knowledge transfer, we devise an adaptive parameter stabilization strategy:

The coefficient θt 2 [0↪ 1] of the historical local header is determined by the above piecewise function. In round t  Tstable (a fixed hyperparameter for all clients in FL), θt is determined by a decay function (e.g., Figure 3) with initial value θ0 2 (0↪ 1]. θt decreases as round t rises. When t > Tstable, the reliability of the global header has been enhanced. Thus, we set θt = 0 to directly replace historical local seen-class parameters with the latest global seen-class header parameters. This prevents stateless local header parameters from slowing down convergence.

The updated local header ˆ ✓tk = ¶ˆ ✓t↪s k ↪ ✓t-1↪s0 k ♦(s 2 Sk↪ s0 2 (S∩Sk)) and the feature extractor 't-1 k from the previous round are spliced together to form a new local model ˆ!tk :

Then, ˆ!tk is trained on Dk to obtain !tk via gradient descent:

These three key steps of uploading seen-class header parameters by class, aggregation by class, and parameter fusion by class enhance the personalization and stabilize the classification boundary for each heterogeneous local model. They are repeated until local models from all clients converge. Each client's final local model is used for inference. FedSSA is detailed in Algorithm 1 (Appendix A 1 ).

4.3 Discussion

Here, we discuss the privacy, communication costs, and computational overheads of FedSSA.

Privacy. When a client uploads its seen-class header parameters, it can maintain the seen-class header parameters and replace the unseen-class header parameters with 0, while still uploading the entire header to the server. The server aggregates non-zero local header parameters according to the corresponding classes (ordinates), and then broadcasts the updated global header to the clients. Since only the header parameters are transmitted, local data privacy can be preserved.

Communication Cost. As stated above, only the parameters of headers are communicated between clients and the server in each round of FL. Therefore, FedSSA incurs lower communication costs than transmitting the complete models as in the cases of FedAvg based FL approaches.

Computational overhead. For clients, the additional computation compared with FedAvg is incurred by header parameter fusion in the adaptive parameter stabilization strategy, which are simple linear operations. It is significantly lower than the computational overhead incurred by training local models. For the server, aggregating the header parameters by class linearly incurs low computation costs.

1 The appendices of this paper can be accessible from https: //arxiv.org/abs/2312.09006

5 Analysis

To analyze the convergence of FedSSA, we first introduce some additional notations. Let e 2 ¶0↪ 1↪ ▷▷▷↪E♦ denote a local iteration. (tE+e) denotes the e-th local iteration in the FL training round (t + 1). At (tE + 0) (i.e., the beginning of the round (t + 1)), clients fuse historical local headers (in round t) and the latest global header (in round t) to update local headers. (tE + 1) is the first iteration in round (t + 1). (tE +E) is the last iteration in round (t + 1).

Assumption 1. Lipschitz Smoothness. Client k's local model gradients are L1-Lipschitz smooth, i.e.,

From Eq. (7), we further derive:

Assumption 2. Unbiased Gradient and Bounded Variance. The random gradient gtk = rLtk (!tk ;Btk ) (B is a batch of local data) of each client's local model is unbiased, i.e.,

and the variance of random gradient gtk is bounded by:

Assumption 3. Bounded Variance of Classification Headers. The variance between the seen-class parameters ✓sk (s 2 Sk) of client k's local header H(✓k) and the same seen-class parameters ✓s = Ek02Ks [✓sk0 ](s 2 Sk) of the global header H(✓) is bounded [Yi et al., 2023a]:

With the above assumptions, since FedSSA does not change local model training, Lemma 1 from [Tan et al., 2022b] still holds.

Lemma 1. Based on Assumptions 1 and 2, during ¶0↪ 1↪ ▷▷▷↪E♦ iterations of the (t + 1)-th FL training round, the loss of an arbitrary client's local model is bounded by:

Lemma 2. Based on Assumption 3, the loss of an arbitrary client's local model after fusing the seen-class parameters of local and global headers by the adaptive parameter stabilization strategy is bounded by:

The detailed proof can be found in Appendix B.

Based on Lemma 1 and Lemma 2, we can further derive the following theorems.

Theorem 1. One-round deviation. The expectation of the loss of an arbitrary client's local model before the beginning of a round of local iteration satisfies

The proof can be found in Appendix C. Then we can derive the non-convex convergence rate of FedSSA as follows:

Theorem 2. Non-convex convergence rate of FedSSA. Based on the above assumptions and derivations, for an arbitrary client and any ✏ > 0, the following inequality holds:

Therefore, under FedSSA, an arbitrary client's local model converges at a non-convex convergence rate of ✏ ⇠ O( 1 T ). The detailed proof can be found in Appendix D.

6 Experimental Evaluation

In this section, we compare FedSSA 2 against 7 state-of-theart MHPFL approaches on two real-world datasets under various experiment conditions. and baselines with Pytorch on four NVIDIA GeForce RTX 3090 GPUs with 24G memory.

6.1 Experiment Setup

Datasets and Models. We evaluate FedSSA and baselines on two image classification datasets: CIFAR-10 and CIFAR100 3 [Krizhevsky, 2009]. They are manually divided into non-IID datasets following the method specified in [Shamsian et al., 2021]. For CIFAR-10, we assign only data from 2 out of the 10 classes to each client (non-IID: 2/10). For CIFAR100, we assign only data from 10 out of the 100 classes to each client (non-IID: 10/100). Then, each client's local data are further divided into the training set, the evaluation set, and the testing set following the ratio of 8:1:1. This way, the testing set is stored locally by each client, which follows the same distribution as the local training set. As shown in Table 3 (Appendix E), each client trains a CNN model with output layer dimensions (i.e., the last fully-connected layer) of 10 or 100 on CIFAR-10 and CIFAR-100 datasets, respectively. The dimensions of the representation layer (i.e., the second last fully-connected layer) are set to be 500. Therefore, each classification header consists of 10(100) ⇥ 500 parameters.

Baselines. We compare FedSSA with 7 baseline methods, which are the state-of-the-art methods in the three MHPFL categories elaborated in Sec. 2.

• Standalone, each client trains its local model independently, which serves as a lower performance bound;

2 https://github.com/LipingYi/FedSSA

3 https://www.cs.toronto.edu/%7Ekriz/cifar.html

• Knowledge distillation without public data: FD [Jeong et al., 2018], FedProto [Tan et al., 2022b];
• Mutual learning: FML [Shen et al., 2023] and FedKD [Wu et al., 2022];
• Model mixup: LG-FedAvg [Liang et al., 2020] and FedClassAvg [Jang et al., 2022].

Evaluation Metrics. 1) Accuracy: we measure the individual test accuracy (%) of each client's local model and calculate the average test accuracy of all clients' local models. 2) Communication Cost: We trace the number of transmitted parameters when the average model accuracy reaches the target accuracy. 3) Computation Cost: We track the consumed computation FLOPs when the average model accuracy reaches the target accuracy.

Training Strategy. We tune the optimal FL settings for all methods via grid search. The epochs of local training E 2 ¶1↪ 10♦ and the batch size of local training B 2 ¶64↪ 128↪ 256↪ 512♦. The optimizer for local training is SGD with learning rate ⌘ = 0▷01. We also tune special hyperparameters for the baselines and report the optimal results. We also adjust the hyperparameters θ0 and Tstable to achieve the best-performance FedSSA. To compare FedSSA with the baselines fairly, we set the total number of communication rounds T 2 ¶100↪ 500♦ to ensure all algorithms converge.

6.2 Comparisons Results

We compare FedSSA with the baselines under both modelhomogeneous and model-heterogeneous scenarios with different total numbers of clients N and client participation fraction C to evaluate the robustness of FedSSA. We set up three scenarios: ¶(N = 10↪C = 100%)↪ (N = 50↪C = 20%)↪ (N = 100↪C = 10%)♦. For ease of comparison across the three settings, N ⇥ C is set to be the same (i.e., 10 clients participate in each round of FL).

Model-Homogeneous PFL

In this scenario, all clients hold the largest model 'CNN-1' as shown in Table 3. Table 1 shows that FedSSA consistently achieves the highest mean test accuracy under all settings, outperforming the best baseline by 3▷61%.

Model-Heterogeneous PFL

We evenly assign 5 models (model IDs: 1 - 5) with different structures (Table 3, Appendix E) to clients. In practice, the id of the model assigned to client k is determined by (client ID k%5). For FML and FedKD, we regard the 5 structurally different models as large heterogeneous models and consider the smallest 'CNN-5' model as the small homogeneous model.

Mean Accuracy. Table 2 shows that FedSSA consistently achieves the highest mean test accuracy under all settings, outperforming the best baseline by 3▷62%. Figure 8 (Appendix E) shows how the mean test accuracy of FedSSA and the two top-performing baselines in each FL setting of Table 2 varies with communication round, it presents that FedSSA converges to higher accuracies faster than others.

Individual Accuracy. Figure 4 shows the accuracy distribution of individual models in FedSSA and the two topperforming baselines in each FL setting shown in Table 2. In Figure 4, '+' denotes the average accuracy of all clients under each algorithm. A small box length bounded by the upper quartile and the lower quartile indicates a more concentrated accuracy distribution across all clients with small variance. We observe that the three algorithms achieve similar mean and variance values in terms of accuracy when N = ¶10↪ 50♦, while FedSSA achieves significantly higher mean and lower variance when N = 100. This verifies that FedSSA is capable of producing the best personalized heterogeneous models in FL with a large number of potential clients and a low participation rate, which closely reflects practical FL scenarios.

Personalization Analysis. We extract every sample representation from each FL client under FedSSA and FedProto, respectively. Then, we leverage the T-SNE

[van der Maaten and Hinton, 2008] tool to reduce the dimensionality of the extracted representations from 500 to 2, and visualize the results. Since CIFAR-100 includes 100 classes of samples, we focus on visualizing the results on CIFAR10 (non-IID: 2/10) in Figure 5. It displays that most clusters in FedSSA and FedProto consist of representations from a client's two seen classes of samples, indicating that each client's local model has strong personalization capability. Generally, FedSSA performs better than FedProto.

6.3 Case Studies

Robustness to Non-IIDness (Class)

We study the robustness of FedSSA and FedProto to nonIIDness controlled by the number of seen classes one client holds under (N=100, C=10%) on CIFAR-10 and CIFAR100. We vary the number of classes seen by each client as ¶2↪ 4↪ ▷ ▷ ▷ ↪ 10♦ on CIFAR-10 and ¶10↪ 30↪ ▷ ▷ ▷ ↪ 100♦ on CIFAR-100. Figure 6 shows that FedSSA consistently outperforms FedProto, demonstrating its robustness to nonIIDness. As the non-IIDness decreases (the number of classes seen by each client rises), accuracy degrades as more IID local data enhances generalization and reduces personalization.

Robustness to Non-IIDness (Dirichlet)

We also test the robustness of FedSSA and FedProto to more complex non-IIDness controlled by a Dirichlet(γ)

function under setting (N=100, C=10%) on CIFAR-10 and CIFAR-100. We vary γ = ¶0▷1↪ ▷ ▷ ▷ ↪ 0▷5♦ on both datasets. Figure 7 presents that FedSSA also outperforms FedProto across all non-IIDnesses 4 .

7 Conclusions

In this paper, we propose a novel personalized heterogeneous federated learning framework named FedSSA to enhance the performance and efficiency of model-heterogeneous personalized federated learning. It consists of two core modules: 1) local-to-global knowledge transfer by semantic similaritybased header parameter aggregation and 2) global-to-local knowledge transfer by adaptive parameter stabilization-based header parameter fusion, both of them enhance the personalization of each client's heterogeneous local model and stabilize classification boundaries. Theoretical analysis shows that FedSSA could converge over wall-to-wall time. Extensive experiments demonstrate that FedSSA achieves the best classification accuracy while incurring the lowest communication and computational costs.

Acknowledgments

This research is supported in part by the National Science Foundation of China under Grant 62272252 & 62272253, the Key Research and Development Program of Guangdong under Grant 2021B0101310002, and the Fundamental Research Funds for the Central Universities; National Key R&D Program of China 2021YFF0900800 & NSFC No.92367202; the National Research Foundation Singapore and DSO National Laboratories under the AI Singapore Programme (AISG Award No: AISG2-RP-2020-019); the RIE 2020 Advanced Manufacturing and Engineering (AME) Programmatic Fund (No. A20G8b0102), Singapore; the China Scholarship Council.

4 More experiment results (comparisons of communication and computation, robustness to client participant rate, ablation studies) are detailed in Appendices (https://arxiv.org/abs/2312.09006).