Revisiting Hypernetwork in Model Heterogeneous Personalized Federated Learning

1 Introduction

Federated learning (FL) has been widely applied in various fields, such as intelligent transportation [Wang et al., 2022;

∗Corresponding authors.

Zhou et al., 2024], healthcare [Nguyen et al., 2022; Murmu et al., 2024], and recommendation systems [Yuan et al., 2023; Feng et al., 2024]. However, a single global model cannot meet all clients' needs due to non-IID data. To address this, personalized federated learning (pFL) [Dinh et al., 2020; Deng et al., 2020] emerges, aiming to craft personalized models for clients while enabling knowledge sharing under crossdevice settings [Meng et al., 2024; Qi et al., 2025] , thus better matching their specific tasks and data distributions.

In practice, devices participating in pFL are often heterogeneous, as they have different computational resources [Chai et al., 2020; Shin et al., 2024], communication capabilities [He et al., 2020; Shah and Lau, 2021], and model architectures [Li and Wang, 2019; Zhu et al., 2021]. In particular, in resource-constrained environments, it is difficult to enforce a unified model architecture across all clients [Chan et al., 2024]. Meanwhile, it is also necessary to prevent the privacy leakage of model architectures [Zhang and Liu, 2024]. These factors complicate the challenges that pFL faces in scenarios of model heterogeneity [Chen et al., 2023]. To address the limitations of the model heterogeneous pFL (MH-pFL), several methods have been proposed by researchers, including partial training [Diao et al., 2021; Shen et al., 2024], federated distillation [Wang et al., 2024; Liu et al., 2025], and model decoupling [Liang et al., 2020; Collins et al., 2021; Xu et al., 2023].

However, modeling decoupling [Jang et al., 2023; Yi et al., 2023a; Yi et al., 2023b] separates local models into feature extractors and classifiers; this low-level knowledge sharing may hinder client collaboration and negatively impact performance. Partial training methods [Horv´ath and others, 2021; Lee et al., 2024] allow clients to select sub-models for local training. However, differences in model architectures, data distributions, and resource conditions can lead to misalignment in parameter counts and feature spaces, causing suboptimal performance. Current federated distillation approaches [Gong et al., 2021; Itahara et al., 2021; Wang et al., 2025] depend on a universal or synthetic dataset for knowledge integration. Such additional datasets may limit the applicability of the method and make the performance dependent on the quality of these datasets. Hence, an MH-pFL approach is needed to resolve these issues while ensuring privacy and computational efficiency effectively.

In this work, we revisit hypernetwork [Ha et al., 2017], a model that generates personalized parameters for another neural network, and propose a novel MH-pFL framework, the Model-Heterogeneous Personalized Federated HyperNetwork (MH-pFedHN), which uses a hypernetwork as a knowledge aggregator to enable knowledge fusion among heterogeneous clients. In our approach, MH-pFedHN customizes a set of embedding vectors for each client based on the number of parameters required by the client's model. Therefore, clients with a similar number of parameters will have an equal number of customized embedding vectors. In addition to the feature extractor from traditional hypernetwork shared across all clients to support generalizable feature learning, a shared head is created for clients that have the same number of embedding vectors, thereby increasing efficiency and promoting knowledge sharing among them. This fine-grained mapping mechanism improves the server's expressive and generalization abilities.

To further learn cross-client knowledge and enhance performance, we propose MH-pFedHN with Global Distillation (MH-pFedHNGD), with an additional lightweight but effective global model. This plug-in component on the server side is directly generated by our hypernetwork using the global customized embedding vectors. Compared to MH-pFedHN, MH-pFedHNGD leverages a global model to aggregate updates from clients, introducing one more round of lightweight update mechanisms, enabling the hypernetwork to learn a more comprehensive data distribution and more generalizable features. Meanwhile, the global model can serve as a teacher model [Hinton, 2015; Jeong et al., 2018] to assist training via knowledge distillation on the client side, thus improving knowledge acquisition abilities of client-specific models and balancing personalization with generalization.

Both of our methods are data-free and preserve the structural privacy of personalized models. They are the first efficient frameworks designed to leverage hypernetworks for solving MH-pFL problems, and hold great promise for future applications. Our main contributions are as follows.

• We propose MH-pFedHN, a personalized FL method based on hypernetworks and specifically designed for heterogeneous models. This method allows the server to utilize customized embedding vectors and the shared head tailored to clients' needs to generate parameters without disclosing the model architecture to the server, thereby enhancing privacy protection.
• We propose MH-pFedHNGD based on MH-PFedHN, which further integrates a plug-in component lightweight global model. This approach enhances the hypernetwork's learning and generalization capabilities and allows personalized client models to learn from a global model with knowledge distillation efficiently, resulting in significant performance improvements.
• We evaluate our MH-pFedHN and MH-pFedHNGD for multiple tasks on four popular datasets. The experiments demonstrate that our method exceeds state-of-theart performance against baselines across all tasks, highlighting the effectiveness and generalizability.

2 RelatedWork

2.1 Personalized Federated Learning with Hypernetworks

Hypernetworks are methods that use one network to generate weights for other neural networks [Ha et al., 2017; Zeng et al., ], which are widely used in various applications [Suarez, 2017; Nirkin et al., 2021; Beck et al., 2024; Ruiz et al., 2024]. In pFL, hypernetworks are used to generate personalized model parameters from clients' embedding vectors [Shamsian et al., 2021; Zhu et al., 2023; Scott et al., 2024] and output the weight ratio during aggregation [Ma et al., 2022]. This method has shown effectiveness in systems with diverse data and few-shot learning scenarios [Sendera et al., 2023].

However, these methods are mainly designed to alleviate the problem of statistical heterogeneity, so that clients can obtain personalized models. The work [Shamsian et al., 2021] proposes pFedHN and only explores generating limited heterogeneous models. FLHA-GHN [Litany et al., 2022] uses graph hypernetworks to generate model parameters for different architectures. Training the hypernetwork on local clients adds computational overhead. They cannot be deployed in resource-constrained environments; in contrast, our efficient methods, MH-pFedHN and MH-pFedHNGD, demonstrate strong potential for deployment in practical scenarios.

2.2 Model-Heterogeneous Personalized Federated Learning Methods

Due to resource constraints [Chai et al., 2020; Shin et al., 2024], communication limitations [He et al., 2020; Shah and Lau, 2021], and personalized requirements for models [Li and Wang, 2019; Zhu et al., 2021], MH-pFL has become an important research direction. Even when clients share the same model architecture, two forms of model heterogeneity can still arise: 1) Due to resource differences, clients may need to employ different sub-models for training [Diao et al., 2021]; 2) Clients with varying personalization requirements retain parameters of specific layers solely for local updates [Li et al., 2021], resulting inmodel heterogeneity. Furthermore, the model architecture may impact model privacy [Zhang et al., 2024], clients are often reluctant to further disclose the details of their model design information.

Partial Training is adopted in some methods, where each client selects a sub-model that originates with the global model. The parameters from the client sub-models are aggregated by averaging matched parameters. HeteroFL [Diao et al., 2021] divides the global model into different sub-models, allowing clients to train appropriate models based on computational capabilities, enabling low-resource clients to contribute to the global model without being excluded from the aggregation. DepthFL [Kim et al., 2023] utilizes different depths of local clients and prunes the deepest layers off the global model to help allocate the server model to all the clients based on computational resources. pFedGate [Chen et al., 2023] explores to learn a sparse local model adaptively. HypeMeFed [Shin et al., 2024] combines a multi-path network architecture with weight generation to support client heterogeneity. However, due to differences in model architecture, parameters of different sub-models learning by partial training may not align, resulting in suboptimal performance.

Federated Distillation, similar to compression [Li and Ma, 2024; Zeng et al., 2025], utilizes knowledge distillation [Hinton, 2015] to mitigate the impact of data heterogeneity and address model heterogeneity when facilitating collaboration between clients with different architectures by transferring knowledge from complex models to simpler ones in FL [Sattler et al., 2020; Wu et al., 2021].

FedMD [Li and Wang, 2019] and DS-FL [Itahara et al., 2021] use a pre-existing public dataset for knowledge extraction, aggregating local soft predictions on the server. KTpFL [Zhang et al., 2021] trains personalized soft prediction weights on the server to improve heterogeneous models' performance further. Inspired by data-free knowledge distillation [Zhang et al., 2022; Dai et al., 2024], some methods [Zhu et al., 2021; Zhang et al., 2023] use GAN [Goodfellow et al., 2020] to generate synthetic data on the server side. FedGD [Zhang et al., 2023] and DFRD [Wang et al., 2024] design a distributed GAN between the server and the clients to enable data-free knowledge transfer and effectively tackle model heterogeneity. However, training the key generator in GAN-based federated distillation poses significant challenges that can impact overall system performance. FedAKT [Liu et al., 2025] combines knowledge distillation and model decoupling; however, homogeneous adapters increase computational and communication overhead for clients, and distillation and dual-head mechanisms can perform poorly in highly heterogeneous environments. Therefore, federated distillation still faces the challenges related to privacy risks and the quality of public and synthetic datasets.

Model Decoupling is another attempt, which divides client models into feature extractors and classifier heads. Further, scientists extend the previous model decoupling methods to share part of the global model while retaining the other part in a heterogeneous form locally. FedClassAvg [Jang et al., 2023] aggregates classifier weights to establish a consensus on decision boundaries in the feature space, enabling clients with non-IID data to learn from scarce labels; FedGH [Yi et al., 2023a] trains a shared, generalized global prediction head using representations extracted by clients' heterogeneous feature extractors on the FL server. In pFedES [Yi et al., 2023b], clients are trained via their proposed iterative learning method to facilitate the exchange of global knowledge; small local homogeneous extractors are then uploaded for aggregation to help the server learn the knowledge across clients. However, these basic mechanisms might find it difficult to achieve effective knowledge fusion in complex heterogeneous scenarios, which results in limitations in final performance.

3 Our Framework

In this section, we first formalize the MH-pFL problem. Then, we introduce MH-pFedHN, which generates parameters for different models through customized embedding vectors and utilizes a hypernetwork for knowledge fusion across heterogeneous clients. Finally, we present MH-pFedHNGD, which utilizes a lightweight global model to enhance the learning and generalization capabilities of the hypernetwork. The lightweight plug-in component also assists personalized model training for clients via knowledge distillation, further improving model training across heterogeneous clients. The overviews of our two methods are shown in Figures 1 and 2, where one round of MH-pFedHN consists of 5 steps and one round of MH-pFedHNGD is made up of 10 steps.

3.1 ProblemFormulation

Our goal is to develop an MH-pFL approach that, without requiring knowledge of model architectures, meets each client's requirements to address the challenges brought about by model and data heterogeneity. This can be turned into the following minimization problem, which aims to capture the personalized objectives of each client while accommodating heterogeneous data distributions and model architectures

where Pi represents the local data distribution of the i-th client, and ℓi(xj, yj; θi) denotes the loss function for the i-th client's model with parameters θi. In particular, θi is clientspecific and can correspond to models of varying architecture. To further formalize the optimization problem, the training objective is expressed as

where mi denotes the number of data samples in the local dataset of the i-th client, and Li(θi) represents the empirical loss over its dataset. By allowing each client to optimize its model, ensuring that the learned parameters θi are suited to the client's unique data distribution and task.

3.2 MH-pFedHN

Here, we describe MH-pFedHN to the MH-pFL problem (Equation 2), and the complete algorithm for MH-pFedHN is provided in Algorithm 1. Let h(·; φ) denote the hypernetwork h with parameters φ, where φ is composed by a feature extractor φf and multiple heads {φHl }Ll=1 , where l indexes the heads and L denotes the number of heads.

In the initialization stage, client i needs to upload the number of parameters Ki required for its personalized model. The server then dynamically determines the hypernetwork output dimension N (i.e., the length of each generated parameter block) based on {Ki}ni=1 , and computes the number of customized embedding vectors for client i as τi = ⌈Ki/N⌉. In practice, we choose N such that τi is greater than the number of layers in the client model to enable finer-grained parameter generation. Therefore, clients with similar numbers of model parameters (i.e., ⌈Ki/N⌉ = ⌈Kj/N⌉) will have the same number of customized embedding vectors (i.e., τi = τj ) and thus share the same head parameters φHl , which enhance knowledge sharing and reduce computational overhead. In contrast, clients with different numbers of customized embedding vectors (i.e., τi ̸= τj) use separate heads to mitigate the impact on personalized task accuracy.

Specifically, for example, if two client models have the same number of parameters and are both determined to have three customized embedding vectors, they will share the same head φHl . This head has three output channels (the channel number equals the number of customized embedding vectors), each tailored for a specific customized embedding vector input. The shared feature extractor φf for all clients and this head φHl will generate three subsets of parameters with length N for each model. These three subsets of parameters, when combined and truncated according to the specific parameter size of each client, constitute the personalized parameters for each client. Generally, for the j-th customized embedding vector of client i (associated with the l-th head), the hypernetwork outputs θj i = h(vj i ;φf ,φHl ). We then obtain the personalized model parameters for client i by concatenating all blocks and truncating to match Ki:

For notational convenience, let vi = [v1 i , . . . , vτi i ] denote the customized embedding vectors of client i, and we write θi = θi(φ) := h(vi; φ)[1:Ki].

Based on the aforementioned setup, we adopt the MH-pFL objective (Equation 2) and get our MH-pFedHN optimization function as 1 :

3.3 MH-pFedHNGD

To enhance the hypernetwork's learning capacity and generalization ability while enabling clients to efficiently extract knowledge representations from a global model, we propose MH-pFedHNGD, which introduces a lightweight global model based on MH-pFedHN. This design aims to improve overall performance at the cost of minimal system and communication overhead. The complete algorithm for MHpFedHNGD can be found in Algorithm 2.

1 The theoretical analysis is in Appendix B.

Thus, the global model's number of parameters is set as Kg = min{K1, · · · ,Kn}, thus, the global model shares the same head with the client having the fewest number of parameters to make sure the global model is lightweight. Then the number of global customized embedding vectors is set as τg = ⌈Kg/N⌉, the global embedding vector vg = [v1 g , . . . , vτg g ]. Assuming the client with the fewest number of parameters corresponds to the l-th head, the global model parameters are generated by:

Then, each client receives the parameters wg,i = wg and trains on private data, which is to optimize:

where M = ∑i mi. After training its own global model is completed in Step ④ in Figure 2, clients upload ∆wg,i = wg,i -wg to the server to update the global customized embedding vectors and the hypernetwork, enhancing the learning and generalization capabilities of the hypernetwork.

Finally, in the distillation training phase, client i reloads the global model in Step ③ and receives the parameters θi of the personalized model, utilizing the global model as a teacher model, directing the training of the personalized model for client i. We adopt the optimization problem (Equation 4) and get our MH-pFedHNGD optimization function as follows:

where LKL is the Kullback-Leibler divergence [Csisz´ar, 1975], and λ balances the distillation and cross-entropy losses. Therefore, the lightweight plug-in component further improves training across heterogeneous clients.

4 Experiments

4.1 Experiment Setup

Datasets. We evaluate the MH-pFedHN and MHpFedHNGD framework over four datasets, EMNIST, CIFAR10, CIFAR-100, and Tiny-ImageNet. We adopt two non-IID settings [Dinh et al., 2020; Liu et al., 2024].1) Quantity-based label imbalance (non-IID 1). For the EMNIST, CIFAR100, and Tiny-ImageNet datasets, we randomly allocate 6, 10, and 20 classes to each client, respectively. We draw αi,c ∼ U(0.4, 0.6), and allocate αi,c ∑j αj,c of the samples for the class c selected on client i. 2) Distribution-based label imbalance (non-IID 2). We employ the Dirichlet distribution Dir(0.01) to partition the dataset among the clients. For each client, 75% of the data is used for training, and 25% is used for testing. As EMNIST dataset is simpler than others, we only report results for 200 clients.

Models. For fair comparison, we use the LeNet model [LeCun et al., 1998] in the homogeneous setting. In the heterogeneous setting, we additionally use VGGNet [Simonyan and Zisserman, 2015] and three residual networks(ResNet10,12,18) [He et al., 2016]. 2 Unless otherwise specified, all experiments under heterogeneous settings use these five models, which are evenly distributed among clients. The feature extractor of the hypernetwork is comprised of a three-layer fully connected network. Details are in Appendix G.

Baselines. We choose various state-of-the-art. pFedHN [Shamsian et al., 2021] uses hypernetwork to produce personalized model directly; pFedLA [Ma et al., 2022] computes aggregation weights for local models of each client; FedGH [Yi et al., 2023a] uses a generalized global prediction header for diverse model structures; pFedLHN [Zhu et al., 2023] leverages a layer-wise hypernetwork to achieve fine-grained personalization across different layers; PeFLL [Scott et al., 2024] uses a learningto-learn approach to generate models; FedAKT [Liu et al., 2025] employs adapters and distillation to enhance knowledge transfer and model adaptation in heterogeneous environments. We also include Local Training without aggregation and FedAvg [McMahan et al., 2017].

Training Strategies. We run all experiments for 500 communication rounds and report the mean and standard deviation over 3 independent runs based on random seeds. For MH-pFedHN, we set local epochs to 2, SGD optimizer with the learning rate 1e-3 and weight decay 1e-4 and momentum 0.9, batch size 64. The hypernetwork uses the Adam optimizer with a learning rate of 2e-4, embedding dimension 64, and the output dimension 3072. For MH-pFedHNGD, we configure the global model LeNet. On CIFAR-100, TinyImageNet, and EMNIST datasets, we set distillation temperatures as 15, 24, and 10, and distillation loss coefficients as 0.01, 0.2, and 0.1, respectively. More settings and design choices are in Appendix C and F.

4.2 An Overall Comparison

Table 1 summarizes results under two data partitions for both homogeneous and heterogeneous settings.

In the homogeneous setting, MH-pFedHN and MHpFedHNGD consistently outperform all baseline approaches across datasets and client scales under both non-IID scenarios, which shows the advantages of enhancing the knowledge sharing among clients. Although the performance of most methods degrades as the number of clients increases, our methods maintain strong accuracy, demonstrating robustness to data heterogeneity and scalability to large client populations. Moreover, MH-pFedHNGD achieves consistently higher accuracy than MH-pFedHN, supporting the effectiveness of introducing a global model for hypernetwork updates and leveraging it for knowledge distillation. Similar trends are also observed in the heterogeneous setting.

2 Experiments with Vision Transformer (ViT) are in Appendix D.4, which shows that our method supports large models. Experiments with CIFAR-10 are in Appendix D.2.

In the heterogeneous setting, we adapt pFedHN and pFedLHN by adding multiple heads so that they can generate parameters for different client models. Since PeFLL, pFedLA, FedAvg, and Local Training do not natively support model heterogeneity, we repeat experiments with different models and report the average test accuracy as the final results. Overall, allowing model heterogeneity generally leads to a performance drop compared to the homogeneous setting under the same configuration, suggesting that effective collaboration among heterogeneous clients remains challenging and deserves further investigation. Note that for methods that do not support heterogeneity, averaging results over multiple models (e.g., LeNet, VGGNet, and ResNet variants) can yield higher numbers than the homogeneous results where only a single model (e.g., LeNet) is used. Despite these challenges, MH-pFedHN and MH-pFedHNGD still achieve the best performance in most settings, and the gain of MHpFedHNGD over MH-pFedHN is particularly evident under heterogeneous models, highlighting the necessity of combining a global model with distillation for MH-pFL.

Finally, most methods attain high accuracy on EMNIST (often above 96%), likely due to the dataset's relative simplicity, which makes meaningful comparisons more challenging.

4.3 Experiment with Generalization

Generalization in FL refers to the model's ability to perform well on unseen clients. This section evaluates the generalization capabilities of MH-pFedHN and MH-pFedHNGD.

These experiments utilize the CIFAR-100 dataset with 100 clients, of which 80 clients are used to train the hypernetwork, while the remaining 20 clients are designated for generalization testing. During the testing phase, the parameters of the hypernetwork remain fixed, and only the customized embedding vectors corresponding to each client are optimized.

Generalize to homogeneous clients. Figure 3 shows that MH-pFedHNGD and MH-pFedHN exhibit outstanding generalization capabilities, significantly outperforming other baseline methods up to 20%. In the early stages of generalization, both algorithms converge rapidly and achieve accuracy on the test set comparable to that of the training clients. Moreover, the generalization ability of MH-pFedHNGD is superior to that of MH-pFedHN, indicating that introducing a global model further improves the hypernetwork's generalization.

Generalize to heterogeneous clients. We use LeNet, VGGNet, and ResNet10 from the heterogeneous setting, and further add MLP and SqueezeNet1 0 [Iandola, 2016]. These five models were evenly distributed among clients. Figure 4 shows that our method still exhibits the better generalization performance (up to 50%) of all the baselines for heterogeneous models, significantly outperforming the other two baselines. Again, the generalization ability of MHpFedHNGD is notably superior to MH-pFedHN.

Generalize to new architectures. All the baselines can not handle this challenge. We use ResNet18 and SqueezeNet1 1 for the added clients, while training uses the same five models as above. In this case, only the feature extractor of the hypernetwork is frozen; the new head will be added and trained. Figure 5 shows no significant difference between MH-pFedHN and MH-pFedHNGD. The assistance provided by the global model is limited, and the learning during the generalization process primarily relies on the data distribution itself. Still, our methods outperform baselines.

4.4 Ablation Study

First, we investigate the head components of MH-pFedHN. We have two configurations: (1) MH-pFedHN does not have any heads; the hypernetwork consists of only a feature extractor; (2) MH-pFedHN where each client is equipped with an individual head with an output dimension of N. Table 2 indicates that without the head, the performance degrades significantly, demonstrating the importance of our head architecture in generating effective parameters. When each client is equipped with an individual head, most settings encounter out-of-memory issues, and the remaining two cases show limited performance, which highlights that our carefully designed shared head can greatly reduce computation overhead and hold strong practical potential.

Next, we examine the global model. We use MHpFedHNG to denote MH-pFedHNGD without knowledge distillation. Table 3 shows that in most cases, MHpFedHNGD outperforms MH-pFedHNG, which in turn outperforms MH-pFedHN. This indicates that the global model alone can enhance the hypernetwork's performance, while knowledge distillation with the global model further improves model training across heterogeneous clients. The observation that results in homogeneous settings are better than those in heterogeneous ones is also consistent with other experiments.

4.5 Experiments with Different Architectures

Under the heterogeneous setting of MH-pFedHNGD, some clients and the global model also used the same LeNet architecture. Therefore, we further investigate the scenario where the architectures of the global model and the personalized client models are completely different. Thus, we set up experiments where the personalized client models are all VGGNet, MLP, and ResNet10, while the global model is LeNet. We also consider the case where these three types of models are evenly distributed among the clients, with the global model still being the LeNet model. Table 4 shows that for MHpFedHNGD, even when the architectures of the global and client models are completely different, its performance still surpasses that of MH-pFedHN, which is consistent with our previous findings. Under such a challenging scenario, both models achieve promising results, demonstrating the strong robustness and generalization ability of our method.

4.6 Supplementary Experiments

Experiments with CIFAR-10, ViT, scalability, client participation ratios, different architectures of global model, redundant parameter, shared heads for heterogeneous models with similar parameter sizes, more baselines, overhead, architecture-specific clustering, progressive client enrollment, communication efficiency, extremely personalized setting, and resource constraint setting are in Appendix D. Experiments with privacy concerns (iDLG Attacks and differential privacy) are in Appendix E. Hyperparameter choice experiments are in Appendix F.

5 Conclusion

We propose a novel data-free approach for modelheterogeneous personalized federated learning, termed MHpFedHN. This method employs a hypernetwork to generate the model parameters for each client, allowing clients to customize their network architectures without exposing them to the server. Furthermore, we present an improved MHpFedHN with minimal effort, denoted as MH-pFedHNGD, which significantly enhances the learning and generalization abilities of the hypernetwork, improves the learning effectiveness of personalized models for clients, and reduces the risk of overfitting. Overall, the effectiveness of our approaches stands out through extensive experiments over baselines.

Acknowledgements

The work is supported by the National Natural Science Foundation of China (Grant No. 62402527), Natural Science Foundation of Shandong Province, China (Grant No. ZR2023QF153, ZR2023QF101), the Fundamental Research Funds for the Central Universities (Grant No. 26CX02004A), and Self-developed research project of Engineering Research Center of Intelligent Oilfields of the Ministry of Education.