HyperFed: Hyperbolic Prototypes Exploration with Consistent Aggregation for Non-IID Data in Federated Learning

1 Introduction

Federated Learning (FL) trains a global model by collaboratively modeling decentralized data in local clients [McMahan et al., 2017]. Disappointingly, FL comes into a performance bottleneck in many real-world applications, where clients contain data with non-identical and independent distributions (non-IID) [Li et al., 2019; Zhao et al., 2018]. Taking a hand-written recognition system as an example, different people have their personalized writing styles, making handwritten characters and letters differ in shape, size, and so on.

Existing FL work with non-IID data either improves the performance of the general global model or enhances the personalized local model. First, to obtain better global performance, a number of work tries to modify the local objectives by adding regularization, so as to make them consistent with generic global performance, e.g., FedProx [Li et al., 2020]. Second, to enhance the performance of the local models, a series of studies encourage training a personalized model for individual clients with meta-learning [Fallah et al., 2020], transfer learning[Luo et al., 2022], and so on. Recently, FedBABU [Oh et al., ] and Fed-RoD [Chen and Chao, 2021] find it possible to enhance global and local models simultaneously, which decouple model in FL into two parts, i.e., one for global generalization and the other for personalization.

∗Chaochao Chen is the corresponding author.

Nevertheless, most existing methods overlook three issues in FL with non-IID data. Firstly, class statistics shifting (Issue 1) happens in FL with non-IID data. Clients have different class statistics information of local data distributions, i.e., the class prototypes, which will shift and bring trivial solutions in FL without fixing. As Fig. 1(a) shows, ignoring fixing class prototypes brings two inevitable limits. Client A fails to recognize the missing class, i.e., data corresponding to some class is missing, while client B causes the class overlapping, i.e., gathering the prototypes of blue and gray classes too tight to discriminate their image features. Though FedBABU [Oh et al., ] and SphereFed [Dong et al., 2022] contribute to addressing this issue, they suffer from the dimension dilemma problem, i.e., they either lack scalability in low-dimensional space, or generate sparse representation in high-dimensional space. Secondly, current work only captures the semantic information of non-IID data, having the insufficient hierarchical information utilization issue (Issue 2). As depicted in Fig. 1(b), it is hard to gather the data samples of the same class together without hierarchical information. Hierarchical information can be helpful to group data samples and generate fine-grained representations, which can further bring prediction gains. To take this advantage, hyperbolic models are used for continuously capturing the hierarchical structure of data in the low-dimensional space [Liu et al., 2020; Linial et al., 1995], whose effectiveness is proved in computer vision [Khrulkov et al., 2020], recommender systems [Tan et al., 2022b], and natural language processing [Chen et al., 2022b]. However, how to integrate existing FL methods with the hierarchical information in hyperbolic space remains unknown. Lastly, the inconsistency in aggregating clients (Issue 3) deteriorates the performance of current FL methods as well. In practice, clients usually have statistically heterogeneous data. Existing aggregation methods, e.g., weighted average with data amounts, result in the aggregated global model deviating from the optimal global model.

In this work, we propose HyperFed which contains three modules to address the above issues, respectively. To solve Issue 1 and avoid dimension dilemma, HyperFed contains hyperbolic prototype Tammes initialization (HPTI) module in server. The server first uses Tammes prototype initialization (TPI) to construct uniformly distributed class prototypes for the whole class set in hyperbolic space. Then the server fixes the position of class prototypes, and shares class prototypes to initialize the hyperbolic predictors of clients. In this way, HyperFed not only guides the consistent and separated criteria, but also introduces the statistics of missing class, both of which encourage discriminative and finegrained feature representation for non-IID data. To avoid Issue 2, with the supervision of hyperbolic prototypes, hyperbolic prototype learning (HPL) module in each client pulls the data sample close to its ground truth class prototype, and pushes other data samples away. Thus HyperFed enjoys the benefits of predicting with hierarchical information. To tackle Issue 3, HyperFed has a consistent aggregation (CA) module that resolves the inconsistent deviations from clients to server by solving a multi-objective optimization with Pareto constraints. Hence HyperFed obtains consistent updating direction among clients, without the need of cumbersome grid search for aggregating clients.

In summary, we are the first, as far as we know, to explore hyperbolic prototypes in FL with non-IID Data. We contribute in: (1) We adopt uniformly distributed and fixed class prototypes in hyperbolic space to alleviate the impact of statistics shifting. (2) We sufficiently leverage hyperbolic representation space to capture hierarchical information for FL. (3) We optimize the aggregation of different client model to a Pareto stationary point, minimizing the impact of inconsistent clients deviations. (4) Extensive experiments on four benchmark datasets prove the effectiveness of HyperFed.

2 RelatedWork

2.1 Federated Learning for Non-IIDData

In terms of the goal of optimization, there are mainly three categories of common FL work that tackles non-IID data:

(1) Global performance, which modifies the local objectives with a regularization term to obtain a well-performed global model [Li et al., 2020]. FedProx [Li et al., 2020] proposes an additional proximal term to local objective, which penalizes the updated local model that is far away from the global model. FedDYN [Acar et al., 2020] and MOON [Li et al., 2021a] regularize the model change with both historical global and local models simultaneously. (2) Local performance, which trains a personalized model for individual clients to enhance the performance of local models [T Dinh et al., 2020]; and (3) Global and local performance, which empirically decomposes the network in FL into the body for universal representation, and the head for personalized classification. Fed-RoD [Chen and Chao, 2021] consists of two classifiers to maintain the local and global performance, respectively. However, no above work takes action to avoid class shifting. Few work fixes the classifier to fill this gap, e.g., FedBABU [Oh et al., ] and SphereFed [Dong et al., 2022]. FedBABU randomly initializes and fixes the classifier during training FL, which cannot guarantee the separation of different classes is distinguishable enough. SphereFed considers fixing the classifier in hyperspherical space. But SphereFed either lacks scalability in low-dimensional space, or generates sparse representation in high-dimensional space. VFGNN [Chen et al., 2022a] utilizes graph sturcture in vertical FL rather than horizontal FL. Moreover, few existing work considers utilizing hierarchical structure and consistent aggregation, which degrades FL with non-IID data.

2.2 Hyperbolic Representation Learning

Hyperbolic geometry is a non-Euclidean geometry, which can be constructed by various isomorphic models, e.g., Poincar´e model [Nickel and Kiela, 2017]. Hyperbolic modeling has been leveraged in various deep networks, such as fullyconnected layers [Shimizu et al., 2020], convolutional layers [Shimizu et al., 2020], recurrent layers [Ganea et al., 2018], classification layers [Cho et al., 2019; Weber et al., 2020], graph neural networks [Liu et al., 2019; Tan et al., 2022a] and Transformer [Ermolov et al., 2022]. However, the existing work overlooks taking the advantage of hyperbolic learning in FL. [Shen et al., 2021; Mettes et al., 2019; Ghadimi Atigh et al., 2021] treat additional prior using orthogonal basis or the prior knowledge embeddings as hyperbolic prototypes, which are positioned as distant as possible from the origin, to avoid frequently updating in prototype learning. These motivate us to utilize hyperbolic prototypes in HyperFed. On the contrary, we contract the class prototypes away from the bound of the Poincar´e ball model, aiming to obtain more general class semantic information.

3 Preliminary: Poincar´e Ball Model

Poincar´e ball model is one of the common models in hyperbolic space, which is a type of Riemannian manifold Mwith constant negative curvature. In this work, ||x||2 = √Σni=1x2i is a Euclidean norm. The Poincar´e is an open ball model in n-dimensional hyperbolic space defined as (Pn , gP ), where Pn = {x ∈ Rn : ||x||2 < 1} and gP is the Riemannian metric of a Poincar´e ball. gP is conformal to the metric of Euclidean space gE , i.e., gPx = λ2xgE , where λx = 2 1-||x||22 is the conformal factor. Given two points in the Poincar´e ball model, i.e., x1, x2 ∈ Pn , the M¨obious addition is defined as:

We can define the geodesic, i.e., the shortest distance between these two points in the Poincar´e ball as below:

For a point x in a manifoldM, the tangent space TxM is a vector space comprising all directions that are tangent toM at x. Exponential map expx : TxM → M is a transformation that projects any point u from the Euclidean tangent space to the Poincar´e ball referred by point x, defined as:

The reverse of expx is logarithmic map, i.e., logx : M → TxM, which projects hyperbolic vector back to Euclidean space.

4 Method

4.1 Problem Statement

In FL with non-IID setting, we assume there are K clients, containing their own models and local datasets, and a central server with global model aggregated from clients. Suppose a dataset D has C classes indexed by [C], where [C] means the full set of labels in D. Each client k has access to its private local dataset Dk = {xk,i, yk,i}Nk i=1 , containing Nk instances sampled from distinct distributions. Therefore, we get D = ∪k∈[K]Dk, where the data distributions of different Dk are different. The overall objective in FL with non-IID setting is defined as below:

where Lk(·) is the model loss at client k, and pk indicates its weight ratio for aggregating.

4.2 Framework Overview

To explain how HyperFed solve the problem in FL with non-IID data, i.e., Eq. (3), we introduce the framework overview of HyperFed. In Fig. 2, there are a server and K clients. Each client or server, similarly consists of a feature extractor, an exponential map, and a Poincar´e ball predictor. The feature extractor F(·) : X → Rd maps an input data x into a n-dimensional vector z = F(x) as feature representation. Then we get expo(z) by leveraging exponential map on the feature representation z to the Poincar´e ball space referred by the origin o. Finally, the Poincar´e ball predictor h(·) : Rd → Y decides class label for input data based on the representation expo(z) in Poincar´e ball. All Poincar´e ball predictors of server and clients are fixed and shared.

As Fig. 2 shows, there are mainly three steps in HyperFed. (1) The server in HyperFed leverages hyperbolic prototype Tammes initialization (HPTI) module to construct a full set of uniformly-distributed class prototypes for the Poincar´e ball predictor by Tammes prototype initialization (TPI), contracts class prototypes close to origin, and shares Poincar´e ball predictor with fixed class prototypes to all of the clients. (2) Each client models local data distribution independently with the hyperbolic prototype learning (HPL) module, then sends the parameters of the local model to the server for aggregation. (3) Consistent aggregation (CA) module in the server updates the global model parameters using consistent updating (CU) to mitigate the inconsistent deviations from clients to server. After that, the server sends the new global model parameters back to clients. This communication between server and clients, i.e., steps 2-3, iterates until the performance converges.

4.3 Hyperbolic Prototype Tammes Initialization

Motivation. In this section, we devise HPTI module in server to resolve two limits brought from the class statistics shifting, i.e., missing class and class overlapping, as described in Fig. 1(a). To bypass the dilemma of choosing a dimension, HPTI explores the class statistics in the hyperbolic space, which is scalable and effective in modeling data with low-dimensional space. Firstly, HPTI uses Tammes prototype initialization (TPI) to construct uniformly distributed and distinguishable class prototypes for the entire class set. Then HPTI fixes the position of the class prototypes on the Poincar´e ball predictor. Lastly, HPTI sends the Poincar´e ball predictor with fixed hyperbolic class prototypes, including the missing classes, to clients. We describe them in detail below.

HPTI first constructs uniformly distributed hyperbolic prototypes with TPI, which is available in data without prior semantic information and efficient in computation. Most work relies on prior semantic knowledge about classes to discover the positions and representations of class prototypes. However, not all datasets contain prior semantic knowledge. Motivated by [Ghadimi Atigh et al., 2021], we randomly sample points on the boundary of Poincar´e ball for assigning class prototypes, and optimize these points to be uniformly distributed in a ball-shaped space. In this way, we incorporate the prior with large margin separation for Poincar´e ball predictor. To be specific, searching for uniformly distributed class prototypes can be formulated as a Tammes problem [Tammes, 1930], i.e., arranging a set of points on a unit sphere that maximizes the minimum distance between any two points. TPI optimizes this Tammes problem to obtain the class prototypes for all C classes in a dataset, i.e., W ∗ ∈ RC×n with n denoting the dimension of prototype:

where wi (wj) is the i-th (j-th) row of W representing as the i-th (j-th) class prototype. We choose cosine similarity to measure this distance, because Poincar´e ball model space is conformal to the Euclidean space [Ganea et al., 2018].

Optimizing Eq. (4) requires computing pairwise similarity of class prototypes iteratively, which is inefficient. To mitigate it, we utilize the similarity of Poinecar´e ball and hyper-sphere to follow [Mettes et al., 2019], and minimize the largest cosine similarity for each prototype in the form of matrix, thus accelerating the optimization:

Next, we find the position to fix the hyperbolic class prototypes. As Fig. 1(b) shows, in Poincar´e ball model, the closer the distance from the referred origin to the node, the more general the semantic information of node represents [Liu et al., 2020]. But the uniformly distributed class prototypes are initially positioned on the boundary of the Poincar´e ball, which is against the distribution of hierarchical structure in Poincar´e ball. In order to enjoy the benefits of uniformity and generality simultaneously, we contract the class prototypes along with the radius to the origin o by a slope degree s, i.e., WP = sW ∗ ∈ RC×n . Lastly, HPTI shares and fixes the Poincar´e ball model to clients, which encourages local clients to model local data sufficiently with the supervision of consistent and separated hyperbolic prototypes.

4.4 Hyperbolic Prototype Learning

Motivation. In this part, we provide the details of HPL, which utilizes the hierarchical information inherent in data to obtain fine-grained and gathered data representations. To utilize the hierarchical information of data, HPL uses Poincar´e ball model for the benefits of continuous optimization and effective representation in low-dimensional space. To start with, HPL extracts the feature for data samples and applies an exponential map referred by the origin shared with class prototypes. According to the supervision of shared class prototypes in hyperbolic space, HPL next represents the data features of the same class according to hyperbolic triplet loss.

In the following, we present how to model the hyperbolic representations of data samples for each client k locally. Specifically, we expect to learn a projection of local data Dk to the local Poincar´e ball model, i.e., Pnk , in which we compute the similarity between data samples and class prototypes for each client. As introduced in Fig. 2, we take a feature extractor Fθk (·) in Euclidean space to obtain the feature representations of local data samples, i.e., z = Fθk (x) for an instance pair (x, y) in Dk. Referred by origin o, we apply an exponential map from tangent space ToM to the Poincar´e ball model Pnk shared with class prototypes. Hence, the representation of data samples in Poincar´e ball model Pnk can be:

As mentioned ahead, we seek to construct the hierarchical structure between the feature representations of data samples and their corresponding class prototypes. Triplet loss [Movshovitz-Attias et al., 2017; Liu et al., 2021; Liu et al., 2023] optimizes the distances among a set of triplets, denoted as {anchor point, positive point, negative point}, by creating a fixed margin, i.e., m, between the anchor-positive points difference and the anchor-negative points difference. Motivated by this, we choose each data sample representation in the Poincar´e ball model as anchor point, the ground truth class prototype as positive point, and the remaining prototypes of the full class set as negative points. In this way, each client incorporates the prototypes of its missing class to feature representation, by randomly sampling negative points. We define hyperbolic triplet loss for client k as below:

LRk =max(d(exp0 (z),wy) -d(exp0 (z),wy′) +m, 0), (7) where z = Fθk (x), wy′ is randomly sampled negative class prototype, d(·, ·) is the geodesic distance defined in Eq. (1), and margin m is a hyper-parameter. We obtain fine-grained representation with sufficient hierarchical information, by simultaneously minimizing the positive geodesic, e.g., the green curve in Fig. 2, and maximizing the negative geodesic, e.g., the red curve in Fig. 2. In this way, HyperFed utilizes the data hierarchical information to enhance the prediction.

4.5 Consistent Aggregation

Motivation. Finally, we introduce CA which mitigates the inconsistent deviations from clients to server caused by the statistically heterogeneous data distributions. In FL aggregation, CA first formulates the aggregation of local feature extractors in HyperFed as a multi-objective optimization. Then CA applies consistent updating (CU) to pick the toughest client, i.e., the client with the most divergent deviation, and alleviate the inconsistency between the toughest client and the remaining clients. Lastly, CU iteratively optimizes this multi-objective optimization to yield a Pareto optimal solution and obtain the weight ratio of different client models.

We formulate the aggregation as a multi-objective optimization in the following. Specifically, we first compute the different deviations from clients to server as multiple objectives, then the goal of alleviating the inconsistency of these deviations can be achieved by multiple-objective optimization. We obtain the combination of local parameters in server:

Algorithm 1 Training procedure of HyperFed

Input: Batch size B, communication rounds T, number of clients K, local steps E, dataset D = ∪k∈[K]Dk

Output: hyperbolic class prototypes WP, model parameters, i.e., θT and{θT k }

• 1: Server executes():
• 2: Initialize θ0 with random distribution andWP by HPTI
• 3: for t = 0, 1, ..., T -1 do
• 4: for k = 1, 2, ..., K in parallel do
• 5: Send {θt ,WP} to client k if t = 0 else θt
• 6: θt+1 k ←HPL: Client executes(k, θt )
• 7: end for
• 8: CA: optimize Eq. (10) with CU and update parameters of θt+1 by Eq. (8)
• 9: end for
• 10: return θT ,WP, {θT k }
• 11: HPL: Client executes(k, θt ):
• 12: Assign global model to the local model θtk ← θt
• 13: for each local epoch e = 1, 2, ...,E do
• 14: for batch of samples (xk,1:B, yk,1:B) ∈ Dk do
• 15: Feature extraction zk,1:B ← Fθek (xk,1:B)
• 16: Project zk,1:B to Poincar´e ball by Eq. (6)
• 17: Compute loss LRk by Eq. (7)
• 18: Update parameters of θek by RSGD
• 19: end for
• 20: end for
• 21: return θE k to server

where θt is the global model and θtk is the local model of the client k at t-th communication. Next, we denote global and client deviations, i.e., ∆t+1 θ = θt+1 - θt and ∆t+1 θk = θt+1 k -θt , respectively, and rewrite Eq. (8) as:

Then CA solves this multiple-objective optimization to Pareto stationary point, i.e., minimizing the minimum possible convex combination of inconsistent deviations:

Next, we introduce CU which derives from Multiple Gradient Descent Algorithm (MGDA) [D´esid´eri, 2012; Sener and Koltun, 2018] to solve this optimization. The optimization problem defined in Eq. (10) is equivalent to finding a minimum-norm point in the convex hull of the set of input points, i.e., a convex quadratic problem with linear constraints. CU iteratively optimizes Eq. (10) by linear search, which can be solved analytically [Jaggi, 2013]. In detail, we find the toughest client, treat the combination of the remaining clients as a virtual client, and analyze the solution according to the directions of the toughest client and the virtual client. Firstly, we initialize p0 with the weight of data samples, i.e., pk = Nk/ΣKk=1Nk, and precompute the consistency of deviations V , s.t.Vk,k′ = ∆θk ⊤∆θk′ . Then we find the toughest client by τ = argmink′ ΣKk=1pkVk′ ,k with deviation ∆τ , and remain the combination of others with historical weights to be a virtual client, i.e., ∆vir = ΣKk=1,k̸=τ pk∆θk .

Thus we simplify Eq. (10), i.e., minpτ∈[0,1] 1 2 ∥pτ∆τ + (1 - pτ)∆vir∥22 . According to the directions of ∆τ and ∆vir, we can obtain the analytical solution for pτ :

where CU+[·] = max(min(·, 1), 0). In Appendix C, we present the analysis of optimization for obtaining CU+[·] based on the computational geometry [Sekitani and Yamamoto, 1993] of minimum-norm in the convex hull. Given pτ , we update the weight ratio p = (1 - pτ )p + pτ e, where e is the one-hot vector with 1 in the τ-th position. In order to obtain the Pareto stationary point, CU iterates the process of finding the toughest client several times to obtain the best combination p∗ that alleviates the inconsistent deviations of clients. Finally, we find the consistent optimization direction with the Pareto optimal solution p∗ for aggregating in Eq. (8).

Given three main modules, i.e., HPTI, HPL, and CA, we illustrate the overall algorithm of modeling HyperFed in Algo. 1. Steps 1-10 are the server execution. In step2, the server initializes model parameters and HPTI in it initializes the hyperbolic class prototypes. Then for each communication round, all clients use HPL to train their local model with the shared Poincar´e ball predictor in step 6. After that, in step 8, CA in server receives the model parameters of all clients, and mitigates the inconsistency of client deviations in aggregation. The details of client execution is listed in steps 11-21.

5 Experiments and Discussion

5.1 Experimental Setup

Datasets. We use four public datasets in torchvision 1 , i.e., EMNIST by Letters [Cohen et al., 2017], FashionMNIST (FMNIST) [Xiao et al., 2017], Cifar10, and Cifar100 [Krizhevsky et al., 2009], which are widely-used in the recent FL work [Chen and Chao, 2021; Oh et al., ; Li et al., 2021a]. There are two evaluation goals for FL with non-IID data, i.e., global performance (G-FL) and local personalized performance (P-FL). G-FL tests the global model aggregated in the server by evaluating the test set published in the torchvision. In P-FL, we simulate local data distribution using the train set published in torchvision to evaluate the local models. For all datasets, we simulate the non-IID data distributions following [Hsu et al., 2019; Li et al., 2021a]. Specifically, we sample a proportion of instances of class j to client k with Dirichlet distribution, i.e., pj,k ∼ DirN(α), where α denotes the non-IID degree of every class among the clients. The smaller α indicates the more heterogeneous data distribution. We sample 75% of local data as local training set and the remaining as local test set.

Comparison Methods. We compare HyperFed with three categories of state-of-the-art approaches according to their optimization goals, i.e., (1) optimizing global model: FedAvg [McMahan et al., 2017], FedProx [Li et al., 2020], SCAFFOLD [Karimireddy et al., 2019], FedDYN [Acar et al., 2020], MOON [Li et al., 2021a], (2) optimizing local personalized models: FedMTL [Smith et al., 2017], FedPer [Arivazhagan et al., 2019], pFedMe [T Dinh et al., 2020], Ditto [Li et al., 2021b], APPLE [Luo and Wu, 2021], and (3) optimizing both global and local models: FedRoD [Chen and Chao, 2021], FedBABU [Oh et al., ], and SphereFed [Dong et al., 2022].

1 https://pytorch.org/vision/stable/index.html

ImplementationDetails. We adopt ConvNet [LeCun et al., 1998] as a feature extractor for EMNIST and FMNIST, while ResNet [He et al., 2016] for Cifar10 and Cifar100. We set all of the datasets with batch size as 128 and embedding dimension as 20. For HyperFed, we choose RSGD [Bonnabel, 2013] as the optimizer, set the learning rate lr = 0.3, the margin m = 3, and the slope degree s = 0.9. We conduct training for all of the methods with 5 local epochs per round until converge. We evaluate both G-FL and P-FL by top-1 accuracy. We set the non-IID degree α = {0.1, 0.5, 5}, respectively, for evaluating the performance of different methods.

5.2 Empirical Results

Performance Comparison. We run each method 5 times and report the performance of G-FL and P-FL in Tab. 1-2 with the average value, respectively. There are mainly three observations. (1) In terms of G-FL evaluated in Tab. 1, generally speaking, the performance of G-FL is increasing along with the increase of non-IID degree α. It states that the heterogeneity of data distribution deteriorates the performance of FL methods. FedAvg takes no measure to handle non-IID, which is worse than most methods. Similarly, FedBABU is a variant of FedAvg that fixes the randomly initialized classifier, perform inferior as well. It means that simply fixing the classifier initialized by random is not enough for FL with non-IID data. (2) In terms of P-FL evaluated in Tab. 2, the methods of the third category can achieve accuracy similar as the second category on FMNIST and EMNIST, but cannot maintain the personalization on Cifar100. This indicates that simply fixing class statistics in FL is less promising than personalization strategies on complex dataset. SphereFed is the runner-up of Cifar100 (α = 5), but it fails to perform well on more heterogeous setting, i.e., Cifar100 (α = {0.1, 0.5}). This phenomenon shows that HyperFed performs better for FL with non-IID data, as a FL method of the third category. (3) In terms of the performance of HyperFed, HyperFed achieves the best results in all kinds of nonIID degrees and different datasets, which verifies the efficacy of HyperFed for non-IID problems. In the results of Cifar100, HyperFed significantly outperforms all of the comparison algorithms listed both in G-FL and P-FL, in terms of at least 10.75% and 5.44%, respectively. This shows that HyperFed can capture more fine-grained representation to improve performance, especially on large and complicated datasets with low dimensional representation.

Visualization. To verify the benefits of fixing class statistics in hyperbolic space, we utilize UMAP [McInnes et al., 2018] to visualize the G-FL hidden representations of the global model in HyperFed, the runner-up of G-FL,i.e., SphereFed, and P-FL, i.e., Ditto, and FedBABU on FMNIST (α = 0.5) in Fig. 3. we can find that: (1) Compared with Ditto, methods fixing the class statistics attain more gathered representations. (2) Though FedBABU fixes the classifier, the randomly initialized classifier is limited in resolving class overlapping. (3) Compared with SphereFed whose classifier is initialized with a set of orthogonal basis in hypersphere, HyperFed concentrates the hidden representation tighter due to sufficiently utilizing the hierarchical information.

Ablation Studies. We consider four variants of HyperFed: (1) HyperFed uses geodesic as metric, i.e., HyperFed-Geodesic, (2) HyperFed uses fixed class prototypes only, i.e., HyperFed-fixed, (3) HyperFed uses shared class prototypes only, i.e., HyperFed-shared, and (4) HyperFed uses weighted average aggregation by data amounts, i.e., HyperFedAveraged. From Tab. 1-2, we can discover that: All of the variants decrease their performance compared with HyperFed. These results identify that all of the components of HyperFed, contribute to performance enhancement.

Hyper-parameters Sensitivity. To study the sensitivity of hyper-parameters, we compare the performance of FMNIST and Cifar10 by varying the local epochs E = {5, 10, 20, 50}

in Fig. 4, slope degree s = {0.2, 0.5, 0.8, 0.9, 0.95, 1} in Fig. 5, margin m = {1, 1.5, 2, 2.5, 3, 3.5, 4, 4.5, 5} in Fig. 6, number of clients K = {5, 10, 20, 50, 100} in Fig. 7, and the dimension of features representation d = {2, 5, 10, 20, 25, 50, 100} in Tab. 3, respectively. We can conclude that: (1) The increase of local epochs per communication round decreases the performance of FL methods. (2) HyperFed nearly converges when s = 0.9, which indicates that positioning the class prototypes in 90% of radius achieves a balance between the best model performance and the most general class information. (3) The performance of HyperFed varying by m forms a slight bell curve. (4) HyperFed outperforms the runner-up methods in all cases of different number of clients. (5) The dimension changes slightly affect the performance of HyperFed, proving that HyperFed is proficient in low-dimensional representation.

6 Conclusion

To enhance federated learning (FL) with non-IID data, we propose HyperFed which contains hyperbolic prototype Tammes initialization (HPTI) module, hyperbolic prototype learning (HPL) module, and consistent aggregation (CA) module. Firstly, HPTI constructs uniformly distributed and fixed class prototypes on server, and shares them with clients to guide consistent feature representation for local clients. secondly, HPL models client data in the hyperbolic model space with the supervision of shared class prototypes. Additionally, CA mitigates the impact of inconsistent deviations from clients to server. Extensive studies on four datasets prove the effectiveness of HyperFed.

Acknowledgements

This work was supported in part by the "Pioneer" and "Leading Goose" R&D Program of Zhejiang (No.2022C01126), and Leading Expert of "Ten Thousands Talent Program" of Zhejiang Province (No.2021R52001).