Bayesian Federated Learning: A Survey

1 Introduction

Decentralized AI (DeAI) and machine learning (DeML) address new demands for engaging decentralized edge devices, nodes and servers to undertake secure edge-level and distributed AI and learning tasks [Cao, 2022c; McMahan et al., 2017]. A critical DeAI and ML technique is federated learning (FL) [Kairouz et al., 2021; Yang, 2021], which ensures decentralized learning on local data without sharing but with properties including privacy, security, heterogeneity, and personalization. FL has seen significant developments to leverage distributed and centralized ML techniques, catering to centralized, decentralized, heterogeneous, or personalized settings and requirements with privacy preservation. These include algorithms for preserving privacy [Liu et al., 2022; Elgabli et al., 2022], reducing communication consumption [Zhang et al., 2021], handling system heterogeneity [Zong et al., 2021], and addressing personalized requirements [T Dinh et al., 2020; Fallah et al., 2020].

However, FL faces various fundamental challenges in enabling DeAI and DeML for real-world applications. First, client data on edge nodes and devices may be very limited and it may be costly to obtain labelled samples. Second, client conditions and behaviors are often dynamic, presenting strong uncertainties. Last but not least, decentralized applications are non-IID, involving heterogeneities of devices, behaviors, goals, and data and their interactions [Cao, 2022b]. These are challenges facing existing FL algorithms and inspire a promising direction - Bayesian Federated Learning (BFL) [Zhang et al., 2022], which integrates the advantages of Bayesian learning (BL) into FL. In existing BFL methods, BL approaches incorporate prior knowledge about data to leverage a limited number of samples and learn distributions over FL parameters and statistical heterogeneity to quantify uncertainties and dynamics. Further, the strengths of FL in handling privacy, communication and heterogeneity are fused with BL. Consequently, BFL enables more robust, wellcalibrated and dynamic predictions for safety- and privacycritical applications. BFL has demonstrated various applications, such as for FinTech, driverless cars, Industry 4.0, medical diagnosis, and differential privacy [Achituve et al., 2021; Snell and Zemel, 2021; Kendall and Gal, 2017; Blundell et al., 2015; Triastcyn and Faltings, 2019; Yang, 2021].

BFL shows a promising potential to substantially expand the existing FL and address complex real-world requirements for DeAI, DeML and FL. Here, we systematically review the relevant work on BFL, conduct a critical analysis of their pros and cons, and present the challenges and opportunities for comprehensive BFL research. First, we briefly discuss the transfer from FL and BL to their integrative BFL. Then, a BFL taxonomy offers a structure of BFL in terms of both BL methods and FL research issues. We discuss and compare the advantages and disadvantages of different methods for clientand server-side BFL and various categories of BFL from the FL perspective. Lastly, we discuss the gaps and future directions of BFL in addressing broad-reaching, more realistic and actionable [Cao and Zhang, 2006; Cao, 2013] FL, BL, DeAI, and DeML settings and tasks and real-world FL scenarios and requirements.

2 FromFederated and Bayesian Learning to Bayesian Federated Learning

We summarize FL concepts, topics and challenges. Then, BL methods and advantages are discussed. BFL integrates BL into FL addressing FL challenges using BL advantages.

2.1 Federated Learning and Challenges

FL concepts. Federated learning initially aims to address privacy leakage in distributed learning systems [Yang, 2021; Kairouz et al., 2021]. In general, an FL system has a server and multiple clients and utilizes an iterative learning process through server-client communications. In each communication round, the server trains a global model and sends its model parameters to participating clients. Each client trains its local model on local data. Clients share model parameters (rather than local data) with the server, which pools and aggregates these local updates to update its global model for the next communication round. This process iterates until convergence or it reaches some stopping conditions.

Typically, FL achieves an objective function as follows:

Here, M denotes the number of activated clients, D and Dm denote the aggregated dataset from all participating clients and the dataset of the m-th client, respectively, w denotes the model parameters of Bayesian neural networks and Fm(w) denotes the empirical risk of the m-th client with the loss function l over each instance (xi, yi) [Chen and Chao, 2022].

FL research topics. FL has seen diversified rapid developments, which include the following categories. (1) Privacypreserving FL [Liu et al., 2022; Elgabli et al., 2022] for differential privacy; (2) Communication-efficient FL [Lin et al., 2022; Li et al., 2022] addressing communication bottlenecks, [Dinh et al., 2020] on resource allocation, [Zang et al., 2022] on sparsified and compressed communication, and [Liu et al., 2020] for propagating channels; (3) Heterogeneous and personalized FL [Ghari and Shen, 2022; T Dinh et al., 2020] to address heterogeneities of local clients; and (4) FL optimization with different settings [Liu et al., 2021; Malinovskiy et al., 2020] for federated SGD, [McMahan et al., 2017] for federated averaging, [Chen et al., 2021] for dynamic aggregation, and [Durmus et al., 2021] for dynamic regularization. In addition, there are many other (1) FL tasks, such as hierarchical FL, non-IID FL, unsupervised to semisupervised FL, multitask FL, robust FL, fair and unbiased FL; and (2) FL application settings, such as blockchained FL, multimodal FL, secure FL, energy-aware and green FL.

FL challenges. However, these previous work on FL still faces various challenges as mentioned in the introduction. In particular, FL suffers from uncertain, dynamic, limited, and unsupervised problems and also data. These techniques also cannot provide robust and analytically explainable results.

2.2 Bayesian Learning and Advantages

BL concepts. Bayesian learning (BL) builds on Bayes' theorem [Benavoli and de Campos, 2021; Sun et al., 2020]. It aims at learning the posterior distribution p(w | D) for parameter w based on its prior distribution p(w) and the corresponding likelihood p(D | w) on observed evidence D. BL

then learns the posterior as follows:

It thus converts a prior probability p(w) into a posterior probability p(w | D) with the likelihood p(D | w) for parameter w on observed evidence D.

BL methods. Bayes' theorem has been widely applied in various settings and has generated numerous BL methods [Jospin et al., 2022; Gershman and Blei, 2012] for tasks such as Bayesian classification, regression, clustering, representation, and optimization. In addition, these can be categorized in terms of modeling mechanisms and settings, learning tasks, and application scenarios, for example (1) Bayesian classification, such as naive Bayes and Bayesian belief networks for classification or optimization; (2) Bayesian approximation, inference and optimization, such as Laplace's approximation, expectation propagation, and variational approximation and Markov chain Monte Carlo (MCMC) methods such as Gibbs sampling; (3) hierarchical Bayesian models with hierarchical variable/parameter dependence, such as Beta-Poisson models; (4) dynamic BL such as Bayesian nonparametric models; (5) BL for other settings and tasks, such as Bayesian continual learning, Bayesian model averaging, and Bayesian posterior decomposition; and (6) hybrid BL methods, such as Bayesian neural networks, and Bayesian model ensemble.

BL advantages. BL shows various advantages. (1) Quantifying uncertainty over probability distributions rather than specific values, which captures the model or epistemic uncertainty of model parameters [Kendall and Gal, 2017]. This can justify the reliability of results, such as for safety-critical applications. (2) Enhancing robustness by evidence-based likelihood estimates, which learns the parameter distributions and regularizes the model for more robust parameters. (3) Improving performance on limited data using prior distribution for each model parameter, which captures prior knowledge about the data and problem. This can greatly improve modeling performance particularly with limited data [Sun et al., 2019].

2.3 BFL: Bayesian Federated Learning

The discussed challenges facing FL can be mostly overcome by taking advantage of BL to address various real-life needs. Consequently, Bayesian federated learning (BFL) has been explored to incorporate BL principles and advantages into FL frameworks and tasks for stronger model robustness and learning improved performance on small-scale data [Zhang et al., 2022]. BFL could lead to more robust, better explainable, and higher performance in handling uncertainties and process-oriented (rather than point-based) challenges. Such advantages could benefit various applications with strong uncertainties, such as estimating financial market dynamics, medical conditions, infectious diseases, financial crisis, and natural disaster.

Although no unified definitions are available for BFL, Figure 1 illustrates (1) the general framework of BFL, which integrates mechanisms of FL and BL; and (2) a general iterative learning process of BFL. Differing from pure FL, a BFL system learns global posterior p(w | D) for the server and local posterior pm(wm | Dm) for each participating client m on its local data Dm. A general or localized prior p(w) applies to all clients. BFL incorporates Bayesian principles into FL for the server:

and clients:

wg and wm represent the parameters of the server and the mth client respectively, p(D) denotes a normalization constant. In non-personalized BFL, wg and wm are equivalent.

Accordingly, a BFL model converts the FL objective function Eq. (1) to BFL for a global loss, aligned local losses, or a mixed loss with settings (e.g., privacy or security preservation, or communication efficiency) and regularization:

Then, the posteriors p(w|D) can be approximated by Bayesian optimization, for example, variational inference (VI). Accordingly, Eq. (6) can be converted to an ELBOlike objective LD, the true posterior p(wm, wg|D) is approximated by a variational distribution qθ(wm, wg) estimated on the evidence D with parameters θ:

KL refers to the Kullback-Leibler divergence. Such an objective function determines the optimization of a BFL system in fulfilling client or global objectives, or their mixture; and distinguishes generic FL from personalized FL.

3 BFL Taxonomy and Analyses

Here, we present a taxonomy of BFL and conduct critical analyses of representative BFL methods from both Bayesian and federated perspectives.

3.1 BFL Taxonomy

BFL tasks and methods can be categorized per aspects such as FL settings or BL methods. First, taking a Bayesian perspective, we categorize BFL in terms of FL architectures: client-side BFL and server-side BFL. Client-side BFL focuses on learning local models on client nodes using Bayesian methods, which involves representative methods: (1) Federated Bayesian privacy (FBP), (2) Bayesian neural networks (BNNs) for local models, (3) Bayesian optimization (BO) for local optimization, and (4) Bayesian nonparametric (BNP) models for dynamic FL. Server-side BFL aggregates local updates for global models using Bayesian methods, with typical methods including (1) Bayesian model ensemble (BME) for aggregation, (2) Bayesian posterior decomposition (BPD), and (3) Bayesian continual learning (BCL). Then, from the FL perspective, we can categorize BFL methods into heterogeneous, hierarchical, dynamic, personalized, and hybrid BFL, etc. Figure 2 shows a taxonomy of BFL and its connections to BL and FL, respectively.

3.2 Client-side BFL

Various Bayesian methods address client requirements and learn local models. Representative client-side methods include FBP, BNN, BO and BNP for different client-side requirements and objectives.

Federated Bayesian Privacy

Earlier FL models focus on privacy-preserving federated updating, communication, and aggregation, with typical progress on differential privacy [Elgabli et al., 2022]. Various differential privacy methods involve simple statistics and object or output perturbation. By considering the randomness of local data, Bayesian differential privacy (BDP) [Triastcyn and Faltings, 2019] ensures client privacy, instance privacy and their joint privacy by a privacy loss accounting method. In [Gu et al., 2021], KL-divergence quantifies Bayesian privacy loss during data restoration, forming a federated deep learning for private passport (FDL-PP) method against FL restoration attacks.

Discussion. BothBDP and FDL-PP relax constraints on existing FL differential privacy by incorporating uncertainty. However, they cannot handle complex FL and BL settings and privacy-preserving requirements.

Bayesian Neural Networks for FL Local Models

Bayesian neural networks (BNNs) [Blundell et al., 2015] combine Bayesian inference with neural networks. BNNs are incorporated into FL in various ways. pFedBayes [Zhang et al., 2022] uses a BNN to train local models in each communication round. Its objective is formulated as a two-level optimization problem. For each client, pFedBayes uses VI to approximate the posterior distribution of local model parameters by minimizing the loss function of the local model. On the server, pFedBayes aggregates local models and minimizes the averaged loss functions of clients. pFedBayes [Kairouz et al., 2021] updates local parameters again after updating them using BNN for each client, resulting in a localized global model to overcome the challenge of non-IID data. FedPPD

[Bhatt et al., 2022] also applies a BNN to train local models. The key difference between FedPPD and pFedBayes lies in the methods of approximating posterior distributions of model parameters. pFedBayes uses VI while FedPPD uses MCMC. FedPPD utilizes a Bayesian dark knowledge method to distill the posterior prediction distribution into a single deep neural network (DNN) for each client and then sends the resulting teacher model (approximate maximum posterior sample) and student model (approximate posterior predictive distribution) to the server for aggregation.

Discussion. BNNs train local models and enable FL to quantify local uncertainty while using DNNs for task learning, thus improving FL robustness. BNNs also improve the FL learning performance on limited data. However, BNNs also bring challenges to FL. First, the BFL with BNNs involves huge computational and memory costs for training local models, especially when the scale of local model parameters is immense. Second, it may be challenging to choose appropriate prior distributions for local model parameters particularly when we cannot estimate complex relationships between model outputs and parameters.

Bayesian Optimization for FL Local Optimization

Bayesian optimization (BO) is a sequential optimization approach, often used to tune the hyperparameters of DNNs. Various methods involve BO for FL. In [Dai et al., 2020], a federated Bayesian optimization (FBO) setting uses BO to optimize local models in each communication round. The FTS algorithm for FBO uses a Gaussian process (GP) as the surrogate for modelling the objective and acquisition functions by Thompson sampling. Random Fourier features are used to approximate GP for scalability and information exchange (the parameters of random Fourier features are shared in each communication round). FTS has a strict convergence guarantee even for non-IID data. In [Zang et al., 2022], BO supports FL for traffic flow prediction (TFP), and like FTS

and other BO algorithms, GP is used as an ideal objective function surrogate. Unlike FTS, since FBO for TFP utilizes BO dynamically to adjust the weights of local models for aggregation, FBO for TFP does not suffer from performance degradation when encountering heterogeneous data. Moreover, FBO for TFP involves an expected improvement instead of Thompson sampling as the acquisition function of the models without addressing the scalability problem of GP.

Discussion. Applying BO to FL achieves a relatively robust learning performance for non-IID local datasets due to the inherent properties of BO. Compared with traditional optimization algorithms, BO is simpler and more convenient to implement. However, the practicality of FBO is challenging for models with substantial data points. Moreover, the slow convergence rate of FBO is open to address.

Bayesian Nonparametric models for Dynamic FL

Bayesian nonparametric (BNP) models enable dynamic learning [Gershman and Blei, 2012]. In BFL, BNP models apply a Gaussian or Beta-Bernoulli process (BBP). In GPbased FL, pFedGP [Achituve et al., 2021] utilizes a personalized GP classifier to train a local model for each client and shares a kernel function for all clients. FedLoc [Yin et al., 2020] also uses GP to train local models for regression tasks. Unlike pFedGP, FedLoc cannot effectively deal with non-IID data for FL [Li et al., 2020]. Different from these methods, FedCor [Tang et al., 2022] uses GP to predict the loss change and then selects the clients that need to be activated in each communication round according to the loss change. FedCor is only applicable for the cross-silo FL framework, where the learning performance decays significantly for non-IID data.

In BBP-based FL, PFNM [Yurochkin et al., 2019] uses BBP to find the matched subsets of neurons among local models. However, PFNM is only applicable to a simple feedforward neural network structure. Then, a layer-wise matching algorithm FedMA based on BBP extends PFNM

to other neural network structures [Wang et al., 2020] such as convolutional neural networks (CNNs) and long short-term memory (LSTM). However, neither PFNM nor FedMA can achieve ideal learning performance on non-IID data.

Discussion. Since model complexity can adapt to data, BNP methods can train local models more flexibly than parametric methods [Orbanz and Teh, 2010]. However, because the complexity of BNP models grows with an increase in data, BNP for FL raises the computational power of clients with a large amount of data.

3.3 Server-side BFL

Various server-side BFL methods implement the global aggregation or decomposition of all updated local models for clients in each communication round. We introduce Bayesian methods BME and BPD. Further, BFL works in other settings, such as continual learning using BCL.

Bayesian Model Ensemble for FL Aggregation

BL models approximate posterior distributions over model parameters using stochastic (e.g., MCMC) or deterministic (e.g., VI) [Kingma and Welling, 2014; Wu et al., 2023] methods. For stochastic methods, the posterior distribution can be approximated by random sampling [Murphy, 2022]. Hence, each sample can be viewed as a base learner for the Bayesian model ensemble (BME).

In BME-based FL, FedBE [Chen and Chao, 2021] first utilizes the parameters of clients to construct a global posterior distribution (Gaussian or Dirichlet) in the aggregation phase of each communication round. MCMC then samples from this global posterior distribution to obtain the ensemble model, which is used on unlabeled data to obtain a pseudolabeled dataset. Finally, a stochastic weighted averaging algorithm distills the global model on this pseudo-labeled data. Similarly, FedPPD introduces three aggregation algorithms with one serving ensemble similar to FedBE. However, in the local learning phase of each communication round, FedBE still uses the point estimation method to obtain the local model parameters of clients, while FedPPD utilizes a BNN.

Discussion. BME helps to implement Bayesian inference on the server using the information of all clients more effectively to prevent model performance degradation. This works especially for local models on non-IID data. Compared with other approximation methods, the sampling method is simpler and more accurate. However, obtaining an accurate ensemble model requires as many samples as possible, increasing the computational power of devices. Also, we cannot obtain a specific global parameter directly by BME. Other methods for distilling an appropriate global parameter are required for client learning in the next communication round.

Bayesian Posterior Decomposition for FL Decomposition

For some ML tasks, it is essential yet challenging to decompose a model into a combination of sub-models that can be handled more easily than the original model [Abrial, 2009]. This requires model decomposition. For Bayesian learning, Bayesian posterior decomposition (BPD) serves this purpose by decomposing the posterior of the global model to a combination of local models.

For BPD-based FL, by imposing strong constraints on uniform prior and local data independence, FedPA [Al-Shedivat et al., 2021] decomposes global posterior distribution into the product of local posterior distributions of clients in each communication round. A concrete expression of parameters of the global posterior distribution can be obtained by federated least squares. Since the direct calculation of parameters from the global posterior will incur high computational and communication costs, calculating the global posterior is converted into an optimization problem and solved by sampling. Since the independent approximations of local posteriors cannot guarantee an accurate global posterior approximation, FedEP [Guo et al., 2023] extends FedPA to approximate a global model using an expectation propagation method. QLSD [Vono et al., 2022] uses the same method as in FedPA to decompose the global distribution. QLSD mainly performs the update of clients per the quantised Langevin stochastic dynamics and sends the resulting compressed gradient to the server to control communication consumption in FL. Differing from these models, FOLA also uses BPD to decompose the global posterior approximation into the product of local posterior distributions with weighted terms, which does not require any strong constraints. VIRTUAL [Corinzia et al., 2019] also uses BPD for aggregation on the server. To avoid catastrophic forgetting, its global posterior distribution is decomposed into the product of the global posterior distribution of the previous communication round and local posterior distributions (ratios of clients) of the current communication.

Discussion. Most FL methods use naive parameter averaging FedAVG [McMahan et al., 2017] for model aggregation, which causes performance degradation when local data involves statistical heterogeneity. However, the stability of the BPD-based FL models on heterogeneous data can be significantly improved by global posterior model decomposition. More importantly, BPD also enables better interpretability of FL models. However, BPD introduces challenges to FL. First, BPD may require other algorithms to assist model learning, which may incur a large computational overhead or even be intractable. Second, some decomposition methods may involve strong constraints, which are usually not feasible when solving practical problems.

Bayesian Continual Learning for Continual FL

Continual learning updates models over a variety of datasets, which may change over time in a sequential manner [Nguyen et al., 2018]. Bayesian learning can facilitate the property of continual learning and use the model posterior as the prior of the next task to achieve Bayesian continual learning (BCL).

For BCL-based FL, FOLA [Liu et al., 2021] uses the product of local posteriors to obtain a global posterior rather than a simple mixture of local posteriors for clients in each communication round for more robust learning on non-IID data. The resulting global posterior will then be sent to all clients at the next communication round as a prior. Similar to FOLA, in each communication round of pFedBayes, the prior distribution of a local model is replaced by the global distribution of the previous communication round to improve the learning performance and interpretability. Nevertheless, there are two main differences between FOLA and pFedBayes. First, the global distribution of pFedBayes is merely obtained by means of an incremental local model averaging, while the global distribution of FOLA is obtained by the product of local posteriors. Second, the local model for each client in pFedBayes is a BNN, while FOLA is a classic neural network.

Discussion. By applying BCL to FL, we can leverage the information aggregated from previous communication rounds. Such an online learning approach often results in better learning performance. However, a complex prior distribution may bring substantial computational overhead to modeling and result in limited performance improvement. It is thus challenging to arrive at a trade-off between using BCL and assuming a suitable prior distribution for FL.

3.4 BFL from the Federated Perspective

BFL research can also be categorized per other FL aspects as discussed in Sec. 2.1. Accordingly, we review specific FL tasks using heterogeneous, personalized, hierarchical, dynamic and hybrid BFL methods. Further, federated Bayesian networks implement BL using FL or in a federated structure.

Heterogeneous BFL. Heterogeneous BFL handles heterogeneous clients with different prior distributions, parameter distributions, or posterior distributions for individual clients. For example, [Kotelevskii et al., 2022] addresses heterogeneous clients using a mixed-effect model for each client, which includes fixed effect (common representation) and random effect (personalized representation). FedBE alleviates the heterogeneity between clients by fitting a posterior distribution for all possible global models. However, existing research generally assumes all local models share the same global model architecture with minor adjustments on priors or parameter distributions. These cannot handle complex heterogeneities, such as heterogeneous structures and relations of clients [Li et al., 2020; Cao, 2022d].

Personalized BFL. Personalized BFL defines a personalized model for each client to deal with its distinct data distributions. For example, pFedBayes uses a personalized model for each client by minimizing the KL divergence between the global model and each updated local model. Instead, pFedGP constructs a personalized GP classifier for each client on its local data. However, existing methods may focus on client heterogeneity but not personalize model parameters, learning tasks or objectives for each client [Tan et al., 2022].

Hierarchical BFL. Hierarchical BFL addresses scenarios where client features or edge nodes may be grouped into another layer, involving hierarchical FL structures. To the best of our knowledge, no work extends BL to hierarchical FL. In [Chen et al., 2022], a hierarchical Bayesian model captures intra- and inter-client uncertainties to optimize hyperparameters: initial value, learning rate, and a number of (early-stop) steps. Nonetheless, existing research ignores complex interactions and couplings between client features or edge nodes, leading to poor performance [Briggs et al., 2020].

Dynamic BFL. Dynamic BFL handles evolving, temporal, or random characteristics, settings or tasks, which may also involves drifts (shifts, changes, variations) of client features, node features, or client-server interactions over time or other dimensions. These require dynamic BL mechanisms, such as updating priors, the number of clients, feature dynamics, and posterior contributions. To address the change in client contributions, the FBO for TFP uses BO to dynamically adjust the weights of clients in each communication round. FOLA uses BCL to dynamically adjust the prior distributions of clients to accelerate model convergence. However, these methods generally overlook the dynamic components of FL, resulting in slow model convergence and a local optimal solution [Chen et al., 2021] and other gaps.

Hybrid BFL. Hybrid BFL is useful to address FL with mixed client structures, features, tasks, priors, posteriors, etc. In FedPPD, a BNN trains local models for FL. FedBE utilizes a model ensemble to improve the robustness of FL. While hybrid BFL models may be able to handle diverse problems, limited existing work is available [Huang et al., 2021].

Federated Bayesian networks. In this review, we exclude research on federated Bayesian networks which use FL for BL. Examples are [Ng and Zhang, 2022] which collectively learns a Bayesian network from partitioned data. Note that one may interchangeably use FBN for BFL, as in [Corinzia et al., 2019; Kassab and Simeone, 2022].

Discussion. The related research on these BFL topics is very limited, immature, incomplete and imbalanced. As discussed above, some of the settings, requirements or tasks have not been explored yet. In the following section, we further discuss these and the future directions of BFL research.

4 Gaps and Directions

BFL has demonstrated significant progress and potential for improving FL on limited, dynamic and uncertain data. However, the existing BFL research also shows significant limitations and gaps in addressing theoretical and practical FL requirements, problems, and challenges.

4.1 Gap Analyses

Table 1 summarizes and compares both client- and serverbased BFL methods discussed in Section 3. Their main limitations and gaps can be summarized as follows.

First, existing BFL methods have various limitations. Typical issues include high computational costs and low communication efficiency. Most BFL models involve strong constraints on (1) FL settings such as client independence and heterogeneity, privacy, resources, and communication costs, and (2) BL settings such as prior distributions and uncertainty of clients. High computational and communication overheads limit the applications of BFL, particularly for decentralized, cross-device and personalized FL tasks. The oversimplified settings and approaches limit their capacity and applications.

Second, existing BFL exhibits weak-to-no capabilities of handling complex interactions and heterogeneities in FL applications with non-IID data. Real-life FL systems are nonIID, involving comprehensive non-IIDnesses [Cao, 2022d; Cao, 2022b; Cao, 2016]. Examples include heterogeneous, interactive, coupled and hierarchical entities, features, relations, and structures within and between clients and communities and between clients and servers [Cao, 2022d]. Existing personalized FL and BFL only weaken these non-IIDnesses through strategies such as neutralizing heterogeneities across clients, e.g., by unified global optimal parameters or simplified Gaussian processes. The statistical heterogeneity between clients requires each client to learn its personalized optimal parameters. Some devices and nodes may interact and couple with each other, while existing FL and BFL overlook their client couplings [Cao, 2022d; Cao, 2022b; Cao, 2016; Pang et al., 2017]. Most of existing references on non-IID FL do not address the above non-IIDnesses or significantly simplify these challenges [Cao, 2022d].

In addition, BFL models still suffer from weak accuracy, robustness and adaption. These may be attributed to poorto-weak priors, heterogeneous, dynamic and hybrid clients or tasks, weak optimization, or inappropriate updating and aggregation mechanisms. There are also gaps in implementing local and global optimization in various settings, such as heterogeneous, hybrid, dynamic, and decentralized FL and for communication, updating, and aggregation. Regarding dynamic BFL, gaps exist in modeling evolving, drifting, nonstationary, or even unlimited scenarios over time or other dimensions (such as value domain, or state space). In summary, the limited research on BFL is at its early stage and focuses on simple applications of classic and main BL settings and methods in simplified FL scenarios, tasks or applications.

4.2 Research Directions

On one hand, intricate real-world FL scenarios, requirement and applications challenge existing FL theories and systems, which could inspire promising BFL research issues and opportunities. On the other hand, BFL could transform FL and BL research and systems to a new generation. The new-generation BFL could include but may not be limited to: variational, non-IID, hierarchical, weakly-constrained, computation- and communication-efficient, hybrid, and actionable BFL theories and applications, and BFL under complex task, network and data conditions.

Variational FL. Variational deep learning hasmade significant progress by integrating VI with deep neural learning [Wu et al., 2023]. Variational FL may expand the existing BFL research for (1) complex stochastic client/server conditions or settings; (2) diverse and efficient VI mechanisms in heterogeneous, hierarchical and hybrid FL settings; (3) large-scale and dynamic variational FL; and (4) variational FL under non-IID settings as discussed below.

Non-IID BFL. Original BFL mechanisms, architectures and models are required to address non-IIDnesses [Cao, 2022b] in FL systems. Examples include (1) heterogeneous data structures, distributions, priors and parameterizations across clients, devices and nodes; (2) personalized client requirements, tasks and objectives, and sample/client importance difference; (3) interactive and coupled clients, communities and edge nodes, message sharing between clients; (4) interactions and couplings between clients and server; and (5) nonstationary, evolving, adaptive, and drifting client/server conditions. This goes beyond existing heterogeneous and personalized FL focusing on limited and neutralized heterogeneities of clients without client couplings.

BFL on complex network conditions. Real-life federated networks may involve complex settings or conditions. For example, new BFL theories may need to handle (1) disconnected nodes, unstably connected clients, siloed clients, and hybrid connections with online and offline devices; (2) unavailable, unstable or forgettable clients or nodes, or unseen domains; (3) coupled and interactive networks such as with cross-domain, cross-silo, cross-client or cross-node (client node or edge network) conditions; (4) unaligned or conflicting network conditions, such as with unaligned clients (where clients share inconsistencies such as on features, distributions, or communication), client conflict, domain conflict, or objective conflict (conflicts may exist in various real-life scenarios, e.g., contrary or inconsistent circumstances, distinct objectives, or different evaluation measures); (5) insecure networks such as with data or model poisoning attacks; (6) hierarchical networks, such as client clusters, multiple client communities, hierarchical centralized servers, and mixing decentralized and centralized client/server structures; and (7) mixing asynchronous and synchronous communications.

BFL on complex data characteristics. FL applications often involve complex data, challenging the existing BFL and FL capacity. They include (1) low-quality data such as noisy, irregular and unaligned client data; (2) weak data such as with small, sparse and insufficient evidence; (3) changing data such as with covariate features, feature shift, prior shift, or concept drift; (4) mixed data such as with multiple domains, communities, modalities, structures and distributions, and multi-granular data such as with client and structure granularity; and (5) mixed labelled data with unlabelled to partially labelled clients. These require new BFL theories, e.g., for dataaugmented, evolving BNP networks, and mixed-supervision BL networks and optimization for FL.

Weakly-constrained BFL. Existing FL and BFL involve various and often strong constraints, such as on client conditions, privacy, security, resources, client-server interactions, communication, prior and posterior distribution, parameter and message sharing, and optimization. These conflict with the diversified FL reality, where there may be siloed and coupled clients, some client features may be unbalanced and sparse, and data may be mixed, evolving and hybridized. These require substantially new and more flexible BFL theories and models catering for specific requirements.

Computation- and communication-efficient BFL. Both existing FL and BFL models face the significant challenges of high computation and communication costs and low efficiency, although intensive efforts have been made to address these. Computation- and communication-efficient BFL requires more efficient, energy-aware and scalable learning theories, sampling, optimization methods, client updating, server aggregation, parameter and message sharing, and back-propagation mechanisms.

Hybrid BFL. Real-life FL applications may involve (1) mixed features, modalities, and data sources; (2) hybrid priors and distributions of client data; (3) multiple to hybrid FL tasks; (4) mixing centralized and decentralized clients/server; and (5) mixing BFL with learning paradigms, etc. These require new BFL theories, architectures, and mechanisms for hybrid Bayesian learning, multi-task BFL, multi-source BFL, multimodal BFL, hybrid FL client updating, server pooling and aggregation, client-server interaction, and optimization methods. Other hybrid BFL areas include hybridizing BFL (1) with other learning systems, such as for Bayesian federated transfer learning, reinforced BFL, ensemble BFL, and BFL for anomaly detection; and (2) with various communication and computing settings, such as compressed BFL, encrypted BFL, asynchronous BFL, blockchained BFL, decentralized BFL, Bayesian federated edge learning, over-the-air BFL, sparse BFL, adaptive BFL, BFL with message sharing across devices or communities, and cross-silo BFL.

Actionable BFL. Actionable BFL requires extra functional and nonfunctional requirements, settings or performance to ensure the actionability [Cao and Zhang, 2006; Cao, 2013; Cao, 2022a] of FL systems in the real world. This requires BFL theories and systems to support functions or capabilities such as fairness, unbiasedness, robustness, resilience, security, safety, responsibility, verifiability, explainability, and ethics. The evaluation of BFL algorithms and systems would have to consider both technical (such as statistical significance) and domain-driven business-oriented (such as impact on business) aspects and measures, and possibly both objective and subjective evaluation measures driven by the domain knowledge and factors, etc. [Cao et al., 2010].

These opportunities could directly or naturally benefit from addressing intrinsic FL issues, utilizing stronger BL mechanisms, or aiming for better capacity and performance. In fact, more exciting opportunities exist in exploring intricate FL demand and challenges in the real world, and focusing on seamlessly integrating BL theories into FL settings.

5 Conclusion

By applying Bayesian learning to the federated learning framework, Bayesian federated learning has become an important learning paradigm to handle various FL challenges and requirements for more robust uncertainty learning. While existing BFL methods exhibit significant progress and potential in learning with limited data and uncertainties, various technical gaps and hence opportunities remain. Fundamental BFL research is required to handle stochastic, heterogeneous, nonstationary, interactive, hierarchical, imbalanced, unlabeled, personalized, and hybrid challenges and requirements for robust BFL theories and actionable FL applications.