Hypernetwork Aggregation for Decentralized Personalized Federated Learning

1 Introduction

Personalized Federated Learning (PFL) [Pillutla et al., 2022] trains customized models for each local client. PFL aims to meet the diverse needs of users and improve the generalization ability and robustness of models in distributed environments [Tan et al., 2022]. To alleviate communication stress and the risks of server failure in centralized federated learning (CFL), DPFL [Li et al., 2023] uses communication topologies to directly exchange model updates with neighbors without a central server [Sabah et al., 2024]. It distributes the communication load to some extent, which is particularly important for large-scale distributed systems.

∗Corresponding authors.

However, even in DPFL, nodes need to frequently synchronize and update local model parameters or gradient information, increasing communication paths and complexity in the network [Yuan et al., 2024]. Frequent and large data transmissions in such topologies consume a significant amount of network bandwidth, especially on mobile devices or IoT devices (Internet of Things). This not only leads to high communication costs, but may also result in delayed or reduced parameter updates, affecting the consistency and accuracy of the model. Moreover, the large amount of communication increases the time that data is transmitted over the network, providing more potential attack surfaces and the risk of privacy leaks. Recently, researchers propose communication optimization strategies, such as compression [Zhu et al., 2024], optimization algorithms [Wu et al., 2022] and efficient decentralized architectures [Beltr´an et al., 2023] to lower communication volume and frequency [Dai et al., 2022]. However, the high communication cost and slow convergence issues have not been fundamentally solved.

Therefore, we propose a new algorithm, the Decentralized Personalized Federated Learning algorithm with Hypernetwork (DFedHP). We deploy hypernetworks on the client-side of the decentralized topologies. The hypernetwork maps input information to the desired target and generates weights of target local model (convolutional networks or recurrent networks) [Ha et al., 2016]. Clients only transmit these smaller hypernet parameters, which greatly reduces the amount of communication data required to transmit a complete model [Chauhan et al., 2024]. This means that hypernetworks decouple communication costs from trainable model sizes. In addition, the clients only need to share the parameters of the hypernetwork, without exposing the complete architecture or weight details of their local models. This approach effectively protects user data privacy by serving as an intermediate abstract layer that prevents direct transmission of sensitive information, providing stronger privacy guarantees. Further, hypernetwork optimization can be performed in a smaller search space (i.e., optimizing the weights of the hypernetwork itself). This not only improves efficiency but also may avoid getting stuck in a local optimum, making the final generated weights more likely to approach an ideal solution [Caldarola et al., 2022]. In sum, clients utilize hypernetworks to generate shared parameters, while training personalized parameters locally, which alleviates communication burden and maintains the ability to train diverse individual models.

We conduct experiments on non-IID settings across different data partitions (Dirichlet and Pathological distribution) and different partition coefficients. And then we compare the performance of our algorithm with many SOTA baselines in CIFAR10, CIFAR100, and Tiny-ImageNet datasets. Extensive evaluations of various classification tasks show that our algorithm can achieve competitive performance, with improvements in both communication cost and convergence performance. Our contributions are listed as follows.

• We propose a novel DPFL algorithm, DFedHP, which can effectively leverage the characteristics of hypernetworks to generate accurate personalized models. It reduces communication overhead significantly and fasters speed of convergence.
• We explore the impact of aggregating different numbers of node parameters on communication and convergence and demonstrate the compatibility and integrability of DFedHP with other PFL methods.
• We evaluate the communication and convergence performance with various data distributions. Extensive experiments prove that DFedHP achieves better performance effectively than SOTA PFL baselines.

2 RelatedWork

Decentralized Federated Learning (DFL). In DFL, there is no central server to coordinate and aggregate model updates, clients must communicate directly with each other. This enhances privacy but may lead to greater communication volume [Sun et al., 2022; Shi et al., 2023b; Li et al., 2025b]. Besides, DFL primarily reduces communication pressure by optimizing communication content, reducing the frequency of communication and improving communication patterns [Liu et al., 2022; Li et al., 2025a], thus improving system efficiency and scalability. Because we mainly focus on the former, from this perspective, researchers typically use compression [Wang et al., 2022] such as quantization, pruning [Xu et al., 2024], or sparsification [Dai et al., 2022] to compress the data that need to be transmitted and reduce the amount of data transmitted each time. For instance, ProxyFL[Kalra et al., 2023] transmits a publicly shared proxy model to reduce communication costs and provide stronger privacy guarantees. KD-PDFL [Jeong and Kountouris, 2023] combines knowledge distillation with decentralized federated learning DFL, enhancing personalization capabilities and reducing transmission costs.

Personalized Federated Learning. PFL balances the generalization ability of the global model with the individualized needs of customers. Common methods include finetuning [Tamirisa et al., 2024], knowledge distillation [Su et al., 2025], regularization [Zhang et al., 2023], parameter decoupling [Zhou et al., 2024], and data augmentation etc. The parameter decoupling approach, such as FedPer [Arivazhagan et al., 2019], FedRep [Collins et al., 2021] and FedBABU [Oh et al., 2021], generally allows each client retains a portion of the model parameters for local personalization while sharing other parameters for global aggregation. DisPFL [Dai et al., 2022] and FedMask [Li et al., 2021a] and achieve enhanced personalization and reduced communication costs by proposing a distributed sparse training technique. To achieve personalization, more complex models such as hierarchical [Ma et al., 2022] and multi-task learning frameworks [Mills et al., 2021] can be adopted, but this may increase computation and communication frequency. In addition, significant differences between PFL clients can require more frequent communication to ensure effective training participation [Pillutla et al., 2022]. This motivates us to explore a PFL strategy to improve the communication burden.

Hypernetworks. Hypernetworks [Chauhan et al., 2024] dynamically generate target network weights based on input data, rather than using pre-trained weights statically [Ha et al., 2016]. It significantly reduces storage requirements without sacrificing too much performance and adapts to different input distributions or tasks. Hypernetworks are applied in few-shot learning [Sendera et al., 2023], federated learning and reinforcement learning [Beck et al., 2023]. In the field of federated learning, PFedHN [Shamsian et al., 2021] deploys a hypernetwork on the central server using random fixed vectors to generate weights directly. PFedLA [Ma et al., 2022] uses hypernetwork to generate weight matrices, and the server uses the matrix and client parameters to update personalized model weights. HFN [Chen et al., 2024] deploys a hypernetwork on the client in CFL topologies, using a single vector from client's each layer and generate model weights. Different from current work, DFedHP ensures model performance while significantly reducing the communication frequency between participants, thereby improving system efficiency and response speed. It is particularly suitable for resource-constrained or applications that require rapid iteration and updates, such as the IoT and mobile edge computing.

3 Methodology

In this section, we first define the problem setting for decentralized partial personalized models. Next, we present the DFedHP algorithm, which aggregates the hypernetworks parameters of clients to reduce the communication burden and improve the convergence speed under resources heterogeneity situations while achieving SOTA performance.

3.1 Problem Setup

Considering n clients with parameters θ ∈ Rd = {θ1, ..., θn}, let Di be the non-IID data distribution of the client i ∈ {1, 2, ..., n}. The non-convex minimization problem in this work can be defined as:

where Fi (θi) = Eξ∼Di Fi (θi; ξi) refers to the local objective function of the client i, which is related to the data sample ξi of Di. In order to preserve the personalized capabilities of the clients, we divide the model into two parts: the shared parameters and the personal parameters vi ∈ Rdi . Different from other PFL algorithms, DFedHP replaces shared parameters with hypernetworks that have fewer parameters, which significantly saves the communication resources required for interaction among nodes. Specifically, h(φi, zi) denotes the the ith client's hypernetwork with the hyperparameters φi ∈ Rdz and the embedding vectors zi ∈ Rdz . Thus, the complete local model θi of the client i is denoted as θi = (h(φi, zi), vi). Consequently, the objective function can be derived from problem (1) to:

where V = {v1, v2..., vn} ∈ Rd1+...+dn . φ presents the consensus hypernetwork model averaged with all hypernetworks φi from other clients, that is φ = 1n ∑ni=1 (φi). The communication network in the decentralized network topology among nodes is modeled as an undirected connected graph G = (N, V,W). N = {1, ..., n} denotes the collection of nodes, V ⊆ N × N denotes the set of end-to-end communication channels. Consequently, the shared parameters of each client are sent to neighbors according to the mixing matrix W, which defines the communication topology between clients. In contrast, personal parameters will not be sent out but will only be iterated locally.

3.2 DFedHP Algorithm

In this subsection, we introduce the DFedHP framework to solve problem (2) by deploying hypernetworks in a distributed network in a more efficient manner.

DFedPH Algorithm. The Figure 1 presents the main pipeline of DFedHP algorithm. Each client is composed of shared part and a personal part. The hypernetwork generates only the shared components of the local models. Rather than sharing their entire client models, clients only communicate hypernetwork parameters with their neighbors. After exchanging information, client aggregates the hyper-parameters of its neighbors and use it to generate shared part parameters. The clients combine local personalized parameters from the previous round of training to generate entire client model. Finally, client performs local training to update the complete parameters, and prepares to send the shared parameters for the next round. Specifically, in Algorithm 1, we form a complete local network (h(φi, zi), vi) in line 6. In line 8 to line 19, multiple local iterative updates are performed on the personal parameters vi, hypernetwork φi and vectors zi in sequence. Finally, in line 19, DFedHP communicates with neighboring models according to mixing matrix W and aggregates hypernetworks. The client updates local hypernetwork by computing the average of hyper-parameters from neighbors (including itself) for the next round.

Consequently, communication between clients can mainly transmit these smaller hypernetwork parameters, rather than the complete structure or model weights. This greatly reduces the amount of communication data required for each iteration. In addition, the hypernetwork serves as an intermediate layer, providing an additional layer of abstraction for sensitive information, further enhancing privacy. Moreover, based on the data characteristics, hypernetwork generates model initialization weights that are closer to the specific task requirements, thereby achieving faster convergence. Furthermore, in addition to DFedAvg, DFedHP can seamlessly integrate into existing DPFL training algorithms, with its lightweight framework and flexible modular design.

Hypernetwork Structure. The hypernetwork maps a certain form of a small input to the weights of the target network. The input generates specific model weights for different tasks, clients, or data set characteristics, including random noise, fixed or dynamically varying vectors [He et al., 2023; Littwin et al., 2020]. In this paper, we input learnable embbeding vectors zi into the hypernetwork φ, we can obtain output h(φi, zi) of the client i. If we input a vector into the hypernet to generate the parameters, we would encounter the problem of excessive hyper-parameters and high communication costs [Ha et al., 2016]. Therefore, we input a large number of different vectors to generate multiple filters, and by concatenating different outputs, we form a large-sized weight matrix. This can significantly reduce the number of hypernetwork parameters, which is far lower than the parameter numbers of original network. As the hypermodel parameters are mainly transmitted between clients, our method greatly reduces the communication cost in DFL. In this paper, we mainly discuss the case of hypernetwork composed of two linear layer and activation function. In this configuration, DFedHP achieves a reduction of up to 87.81% in the number of parameters transmitted per round. Smaller hypernet parameters are transmitted between clients, rather than the full model weights, which reduces the communication costs and time.

Partial Personalized Models [Liu et al., 2025]. In a neural network, the convolutional layers near the input extract features representations of input by filters (also called kernels or feature detectors), mapping data to easily discernible low-dimensional spaces [Sabah et al., 2024; Tan et al., 2022]. For many tasks, such as image classification, the early convolutional layers often capture some low-level features, such as edges and textures, which have high generality. Therefore, by sharing the convolutional layers, effective low-level feature extractors can be shared among all clients, thereby enhancing resource utilization efficiency and model performance. This shared part is generated by the hypernetwork in this paper. Conversely, the linear layer close to the output primarily maps features to a specific output space. It is focused on pattern recognition to determine the data category, which we designate as the personalized part. The linear layers can be personalized according to the data characteristics of each client to better adapt to the local data distribution.

4 Theoretical Analysis

In this section, we conduct a convergence analysis of the DFedHP algorithm and explore its working principle. First, we make some general assumptions.

Definition 1 (The gossip/mixing matrix [Sun et al., 2022]). The gossip matrix W = [wi,j] ∈ [0, 1]n×n is assumed to have these properties: (i) (Graph) If i ̸= j and (i, j) /∈ V then wi,j = 0, otherwise, wi,j > 0; (ii) (Symmetry) W = W⊤; (iii) (Null space property) null {I - W} =span{1}; (iv) (Spectral property) I ⪰ W ≻

• -I. With these properties, the eigenvalues of W satisfy

Algorithm 1 DFedHP

• 1: Input: total number of devices n and communication rounds T. Learning rate: personal part ηv, hypernet ηφ and embedding vectors ηz. Number of local iterates Kv, Kφ and Kz.
• 2: Output: hyper parameters φTi , personal part vT i and embedding vectors zT i after the final communication of all clients.
• 3: Initialization: randomly initialize each device's hypernet parameters φ0i , personal parameters v0 i and vectors z0 i .
• 4: for each communication round t → 1 to T do
• 5: for client i in parallel do do
• 6: Generatemodel parameters byφ0i , sample a batch of local data ξi and calculate local gradient iteration.
• 7: for local epoch from k = 0 to Kv do do
• 8: Perform personal parameters vi update:

• 11: end for
• 12: for local epoch from k=0 to Kφ do do
• 13: Update the hypernetworks parameters φi:

• 15: end for
• 16: gt1,i ← φt,Kφ i
• 17: for local epoch from k=0 to Kφ do do

18:

Update the embedded vector zi:

• 20: end for
• 21: gt2,i ← zt,Kz i
• 22: Receive neighbors' hyper-network gt1,j , gt2,j .
• 23: with mixing matrixW :
• 24: φt+1 i = ∑l∈N(i) wi,lgt1,i , zt+1 i = ∑l∈N(i) wi,lgt2,i
• 25: end for
• 26: end for

1 = λ1(W)) > λ2(W)) ≥ · · · ≥ λn(W)) > -1. In addition, λ := max {|λ2(W)| , | λn(W)) |} and 1-λ ∈ (0, 1] denote the spectral gap of W, which quantifies the characteristics of the network topology.

Assumption 1 (Lipschitz Smoothness). Fi and hi are continuously differentiable. There exist constants Lφ, Lz, Lh, Lv,Lhv, Lvh, Lhφ, Lhz, ∀i ∈ {1, 2, ..., n} such that:

• ∇hFi(hi, vi) is Lh - Lipschitz with respect to hi, Lhv -Lipschitz with respect to vi;
• ∇vFi(hi, vi) is Lv-Lipschitz with respect to vi, LvhLipschitz with respect to hi;
• ∇φFi(h(φi, zi), vi) is Lφ - Lipschitz with respect to φi and ∇zFi(h(φi, zi), vi) is Lz - Lipschitz with respect to zi;
• ∇φhi is Lhφ-Lipschitz continuous with respect to φi and ∇zhi is Lhz - Lipschitz continuous with respect to zi.

The gradients ∇φh and ∇zh are respectively bounded by constants γφ and γz. Besides, we present the relative crosssensitivity of gradient with the scalar:

Assumption 2 (Bounded Variance). The gradient of the function Fi has bounded variance. There exist constants σφ, σz and σv , ∀i ∈ {1, 2, ..., n} such that:

Assumption 3 (Partial Gradient Diversity). There exists a constant δ ≥ 0, ∀φi, zi, V such that:

Theorem 1 (Convergence Analysis for DFedHP). Assume Assumptions 1-3 holds, let the local adaptive learning rate satisfy ηφ = O(1/LφKφ√T), ηz = O(1/LzKz√T), ηv = O(1/LvKv√T), where T is the number of communication rounds. Furthermore, we adopt the averaged parameter ¯ φt = 1n ∑ni=1 (φti ), ¯ zt = 1n ∑ni=1 (zt i ) of all clients as an approximate solution to the problem (2) below. Additionally, F ∗ denotes the minimal value of F: F(h( ¯ φ, ¯ z), V ) ≥ F ∗ for all φ, z, and ∆t¯ φ, ∆t¯ z and ∆t¯ v are performed as:

Thus, we present the convergence rate:

where the constants σ2 1 , σ2 2 and σ2 3 are defined as:

Remark 1 (Statistical Heterogeneity Affects the Convergence Boundary). The smaller local variance variables σ2φ, σ2 z , smaller gradient diversity δφ,δz and smoother Lipschitz constants all lead to a tighter convergence boundary.

Since in a round of communication, viis updated first, followed by φi and then zi, there is no Lvh in the convergence boundary. The proof details are in the Appendix.

5 Experiments

In this section, we introduce the experiment, covering the setup and the performance of the DFedHP. Additionally, we present the communication advantages of the method and verify its effectiveness when combined with other methods.

5.1 Experiment Setup

Dataset and Data Partition. We evaluate the performance of DFedHP on the CIFAR-10, CIFAR-100, and TinyImageNet datasets under Dirichlet distribution and Pathological distribution. The Dirichlet distribution specifies that the local data set conforms to the Dirichlet distribution. And the Pathological distribution is to allocate a finite number of classifications to each client. The two types of approaches guarantee the distribution of all data sets among the clients is nonIID. We partition the training and testing data according to the Dirichlet distribution Dir(α). The smaller partition alpha α is, the more uneven the data distribution among clients will be, resulting in higher data heterogeneity [Kotelevskii et al., 2023; Wang et al., 2020]. In addition, we sample different classes from the dataset for each client. The fewer classes each client has, the more heterogeneous the setting becomes.

Baselines and Backbone. We evaluate our method and the following SOTA baselines of DFL. We select FedAvg [McMahan et al., 2017], FedPer [Arivazhagan et al., 2019], and Ditto [Li et al., 2021b] as the FL baseline, and DFedAvg [Sun et al., 2022] and DFedAlt [Shi et al., 2023a] as the DFL baseline. Local means only local training for each local client with no communication. Note that we define the accuracy as the personal test accuracy. The PFL methods used in the experiment learns personal classifiers for each client based on a shared feature extraction.

Training Strategies. In all experiments, all algorithms are conducted on ResNet-18 [He et al., 2016] with batch normalization. We record the communication between the client and server (or between clients) at 150 rounds. The total number of clients is 100 and the communication radio is 0.1 for each round. The batch size is 128 and the local training epoch is 4. The experiment uses SGD as the optimizer with momentum of 0.9. The distributed topologies is random. For DFedHP, the embedding vector size is 128. Although dimensions of kernels in each layer are different, the dimension of a kernel is often an integer multiple of a fixed value [Ha et al., 2016]. We choose 64 as the fixed value of ResNet-18. We conduct multiple experiments on the learning rates of all algorithms.

5.2 Performance Evaluation

Figure 2 shows the convergence curves of DFedHP and baseline methods in image classification tasks using the ResNet18 model on different datasets and different non-IID data distributions. DFedHP significantly reduces communication costs. In the experimental setup of the paper, the amount of data each client needs to transmit is reduced from a maximum 43,666.97KB to 5,321.06KB (32 float) in Table 3. Under the decentralized setting, we assume that each client communicates with 10 neighbors in each round, following the hypernetwork configuration in the main experiment. Each communication transmits at least 50.75 GB of parameter information transmitted in 500 rounds of communication. Without the hypernetwork, the minimum communication volume in 500 rounds is at least 418.23 GB. In sum, due to the use of hypernetwork weight generation network, the client only needs to transmit a smaller number of hypernetwork parameters, thus reducing the communication burden.

In addition, Table 2 presents the number of communication rounds required to achieve the DFedHP target accuracy. It is evident that DFedHP shows significant improvements in convergence speed. In Pathological 20, the convergence speed of DFedHP in CIFAR-100 is 39.29 % faster than that of DFedAvg, and the convergence speed of the meta-network transmission is greatly improved. Moreover, DFedHP can be used with any PFL method, such as "DisPFL + HP", "DFedAlt + HP" in Table 1. In most cases, applying DFedHP to other DFL frameworks significantly reduces convergence speed and communication overhead. Faster convergence enables the model to reach optimal or satisfactory performance more quickly, reducing required iterations. This is crucial in poor network conditions, where fewer communication rounds lower overall costs. In applications that need rapid adaptation to new data or changing environments, faster convergence allows timely model updates and more accurate services.

Then, we compare the test accuracy of all baselines under different settings in Table 1. In CIFAR-10 dataset, DFedHP achieves 77.01% in the Dirichlet 0.3 setting, which is 3.05% higher than the method without hypernetwork (DFedavg). DFedHP performs well on the Dirichlet distribution and pathological distribution. It can resist the negative effects of some data heterogeneity, which proves the effectiveness and robustness of the proposed method. Moreover, in cases of excessive data heterogeneity, due to the limited local training data, accuracy decreases as the level of heterogeneity decreases, which is a normal phenomenon. It performs well in scenarios with data heterogeneity, which proves the effectiveness of the proposed method. DFedHP utilizes embedded vectors to capture more precise features of the data. This allows the classification head to be more adaptive to the local data of each client through a stronger feature extractor, thereby enabling the meta-network to generate accurate model weights. In summary, the effects of DFedHP are significant, and it can reduce communication burden and achieve faster convergence speed with greater benefits in distributed scenarios when performing partial model personalization.

5.3 Ablation Study

We conduct several ablation study on CIFAR-10 with Dirichlet 0.3 distribution to verify the validity and robustness of the component. In Figure 3 (a), FedHP achieves the robustness in various communication topologies. Furthermore, based on the spectral gap analysis in Section 4, as the sparsity of topology decreases, spectral gap decreases, the generalization effect improves. The results in Figure 3(a) are consistent with the analysis. The communication costs and convergence boundaries of different topologies, from largest to smallest, are: fully connected, exponential, grid, ring. Figure 3(b) shows the impact of the number of hypernetwork parameters on model accuracy under different network structures. The X-axis represents the proportion of hypernetwork parameters in the original model without hypernetwork. In the experiment, the best accuracy results among different hypernetwork structures with the same number of hyperparameters are selected. The number of parameters in a HyperNetwork is not fixed, as it depends on model complexity, task requirements, dataset characteristics, and expected performance. It must have enough expressive power to generate effective weights while avoiding overfitting or wasting computational resources. In addition, more detailed ablation studies are shown in the Appendix.

6 Conclusion

In this work, we propose a new DPFL framework, DFedHP, to realize decentralized personalized model aggregation through hyper network aggregation. This enhances collaboration among clients and generates better personalized models on non-IID datasets. In addition, DFedHP decouples training complexity from communication complexity. The size of the generated client models is not limited, effectively reducing communication overhead. Further, DFedHP can be combined with other related methods, which is scalable and flexible. It can adapt to clients with different computing capabilities or large-scale federated learning scenarios, where communication capacity is typically limited. Extensive evaluations on various classification tasks have demonstrated that the communication efficiency and convergence ability of DFedHP significantly enhance. In the future, we will explore the theoretical analysis and application extensively.

Acknowledgments

This work was supported by the National Natural Science Foundation of China under Grant 62403484.

Contribution Statement

Corresponding authors: Yong Peng and Xiaoyan Wang.