Federated Learning with Fair Averaging

1 Introduction

Federated learning (FL) has emerged as a new machine learning paradigm that aims to utilize clients' data to collaboratively train a global model while preserving data privacy [McMahan et al., 2017]. The global model is expected to perform better than any locally trained model, since it has much more data available for the training. However, it is difficult for the global model to treat each client fairly [Kairouz et al., 2019; Li et al., 2019; Mohri et al., 2019]. For example, a global model for face recognition may work well for younger users (clients) but may suffer when being used by more senior users, as the younger generation may use mobile devices more frequently and contribute more training data.

Techniques have been proposed to address the fairness issue in FL. AFL [Mohri et al., 2019] aims at good-intent fairness, which is to protect the worst-case performance on any client. This technique only works on a small network of dozens of clients because it directly treats each client as a domain, which may suffer in generalizability. Li et al. [2019] seek to balance the overall performance and fairness using a fair resource allocation method. Hu et al. [2020] find a common degrade direction for all clients without sacrificing anyone's benefit and demonstrate robustness to inflating loss attacks. Li et al. [2020a] further explore a trade-off between a more general robustness and fairness, and they personalize each client's model differently.

∗Corresponding Author

Different from these studies, we observe that conflicting gradients with large differences in the magnitudes might fatally bring unfairness in FL. Since gradients are calculated locally by the clients, an update direction of some clients may hinder the model performance on other clients. The gradient conflicts seem not detrimental on their own since it has been shown that a simple average of gradients is valid to optimize the global objective function [Li et al., 2020c]. However, when the conflicting gradients have large differences in their magnitudes, model accuracy can suffer significant reductions for some of the clients, as shown in Figure 1.

We highlight three characteristics of FL that make it critical in ensuring fairness for the clients.

• Heterogeneous and imbalanced data. Different clients have their own data characteristics and hence potentially different data distributions. When the data is non-IID distributed, the divergence among the gradients of difference clients gets even larger [Zhao et al., 2018]. These contribute conflicting gradients in FL. Further, different clients may also have different dataset sizes. If the same local batch size is used across the clients, a client with more data will take more steps to train the local model, thus leading to large differences in gradient magnitudes. As a result, the selected users in a communication round can largely conflict with each other, leading to an unfair update to some of them - we call such conflicts internal conflicts.
• Party selection. In each communication round, the server only randomly selects a subset of clients to train the global model with their local datasets. There is no guarantee that the data distribution in each round is representative of the real population distribution. The data distribution can be highly imbalanced when the server repeatedly chooses a particular type of clients, e.g., due to sampling strategies based on computing power, latency, battery etc., the average of gradients may become dominated by the repeatedly selected clients. Therefore, apart from the internal conflicts, we also identify external conflicts between those selected and those ignored by the server.
• Client dropping out. Another source of external conflicts comes from network interruption. When a client drops out due to network interruption (or other hardware issues), its magnitude is considered as 0. An update in the global model may also be unfair for the client.

To address the issues above, we propose Federated Fair aVeraging (FedFV) algorithm to mitigate the conflicts among clients before averaging their gradients. We first use the cosine similarity to detect gradient conflicts and then iteratively eliminate such conflicts by modifying both the direction and magnitude of the gradients. We further show how well FedFV can mitigate the conflicts and how it converges to either a pareto stationary solution or the optimal on convex problems. Extensive experiments on a suite of federated datasets confirm that FedFV compares favorably against state-of-the-art methods in terms of fairness, accuracy and efficiency.

The contributions of this work are summarized as follows:

• We identify two-fold gradients conflicts (i.e., internal conflict and external conflict) that are major causes of unfairness in FL.
• We propose FedFV that complements existing fair FL systems and prove its ability to mitigate two gradient conflicts and converge to Pareto stationary solutions.
• We perform extensive experiments on a suite of federated datasets to validate the competitiveness of FedFV against state-of-the-art methods in terms of fairness, accuracy and efficiency.

2 Preliminaries

FL aims to find a shared parameter θ that minimizes the weighted average loss of all clients [McMahan et al., 2017]:

where Fk(θ) denotes the local objective of the kth client with weight pk, pk ≥ 0 and ∑Kk=1 pk = 1. The local objective is usually defined by an empirical risk on a local dataset, i.e., Fk(θ) = 1 nk ∑ξi∈Dk l(θ, ξi), where Dk denotes the local dataset of the kth client, and nk is the size of Dk. To optimize this objective, McMahan et al. [2017] propose FedAvg, an iterative algorithm where the server randomly samples a subset St of m clients, 0<m ≤ K, to train the global model with their own datasets and aggregates local updates Gt = {gt1 , gt2 , ..., gtm} into an average gt = ∑mk=1 pkgtk with weight pk = nk ∑mj=1 nj in the tth iteration. FedAvg has been shown to be effective in minimizing the objective with low communication costs while preserving privacy. However, it may lead to an unfair result where accuracy distribution is imbalanced among different clients [Li et al., 2019; Mohri et al., 2019].

In this paper, we aim to minimize Equation (1) while achieving a fair accuracy distribution among the clients.

3 Proposed Approach

We first present our approach, FedFV, for easing the negative impact of conflicting gradients with largely different magnitudes on FL fairness. We then give a theoretical analysis on how well FedFV achieves fairness and its convergence. We start by defining gradient conflicts following Yu et al. [2020].

Definition 1. Client i's gradient gi ∈ Rd conflicts with client j's gradient gj ∈ Rd iff gi · gj<0.

3.1 FedFV

FedFV aims to mitigate the conflicts among clients before averaging their gradients. In the tth communication round of FedFV, after receiving the selected clients' updates Gt = {gt1 , gt2 , ..., gtm} and training losses Lt = {lt1 , lt2 , ..., ltm}, the server updates the gradient history GH to keep a trace of the latest gradient of each client. Then, the server sorts gradients in Gt in ascending order of their corresponding clients' losses to obtain POt, which provides the order of each gradient to be used as a projection target. Finally, FedFV mitigates the internal conflicts and the external conflicts sequentially. Algorithm 1 summarizes the steps of FedFV.

3.2 Mitigating Internal Conflicts

We first handle the fairness among the selected clients in each communication round.

Definition 2. In the tth communication round, there are internal conflicts if there is at least a pair of client gradients < gt i , gt j > such that gt i conflicts with gt j , where gt i , gt j ∈ Gt = {gt1 , gt2 , ..., gtm} and Gt is the selected clients' updates.

Algorithm 1 FedFV

Input:T, m, α, τ, η, θ0 , pk, k = 1, ..., K,

• 1: Initialize θ0 and gradient history GH = [].
• 2: for t = 0, 1, ..., T -1 do
• 3: Server samples a subset St of m clients with the prob. pk and sends the model θt to them.
• 4: Server receives the updates gtk and the training loss ltk from each client k ∈ St, where gtk ← θt - θtk and θtk is updated by client k. Then server updates the trace GH = [gt1 1 , gt2 2 , ..., gtK K ] of the latest updates of all clients.
• 5: Server sorts the clients' updates into a projecting order list POt = [gt i1 , ..., gt im ] based on their losses, where lt ij ≤ lt ij+1 .
• 6: gt ←MitigateInternalConflict(POt,Gt, α)
• 7: if t ≥ τ then
• 8: gt ←MitigateExternalConflict(gt , GH, τ )
• 9: end if
• 10: gt = gt /||gt || ∗ || 1m ∑mk gtk ||
• 11: Server updates the model θt+1 ← θt - gt
• 12: end for

Internal conflicts account for the unfairness among selected clients. Consider learning a binary classifier. if clients with data of one class are in the majority in a communication round and there exist conflicts between gradients of these two classes, the global model will be updated to favor clients of the majority class, sacrificing the accuracy on clients of the other class. Further, even when the proportion of clients of the two classes is balanced, the magnitudes of gradients may still largely differ due to different dataset sizes on different clients.

To address the internal conflicts, FedFV iteratively projects a client's gradient onto the normal plane of another client with a conflicting gradient. Instead of projecting in a random order, we design a loss-based order to decide which gradient should be the projecting target of other gradients based on Theorem 1. We show that the later a client gradient is used as the projection target, the fewer conflicts it will have with the final average gradient computed by FedFV. Algorithm 2 summarizes our steps to mitigate internal conflicts.

Theorem 1. Suppose there is a set of gradients G = {g1, g2, ..., gm} where gi always conflicts with g(tj) j before projecting g(tj ) j to gi's normal plane, and g(ti) i is obtained by projecting gi to the normal planes of other gradients in G for ti times. Assuming that | cos< g(ti) i , g(tj) j >| ≤ ϵ, 0<ϵ ≤ 1, for each gi ∈ G, as long as we iteratively project gi onto gk's normal plane (skipping gi it self) in the ascending order of k where k = 1, 2, ...,m, the larger k is, the smaller the upper bound of conflicts between the average ¯ g′ = 1m ∑mi=1 g(m) i and gk is.

Since projecting gradients favors for the gradient that

Algorithm 2 MitigateInternalConflict

Input: POt, Gt, α

• 1: Server selects a set Sα t of the top α clients in POt.
• 2: Server sets gPC k ← gtk .
• 3: for each client k ∈ Sα t in parallel do
• 4: for each gt ij ∈ POt, j = 1, ...,m do
• 5: if gPC k · gt ij <0 and k ̸= ij then
• 6: Server computes gPC k ← gPC k - (gt ij )·gPC k ||gt ij ||2 gt ij .
• 7: end if
• 8: end for
• 9: end for
• 10: gt ← 1m ∑mk=1 gPC k
• 11: return gt

serves as the projecting target later, we put the clients with larger training losses at the end of the projecting order list POt in the tth communication round to improve the model performance on the less well trained data. We allow αm clients to keep their original gradients to further enhance fairness. We will detail α in Section 3.4.

3.3 Mitigate External Conflicts

Due to party selection and client dropping out, there is a sample bias during each communication round in FL [Kairouz et al., 2019]. A client that is not selected in the tth round can suffer a risk of being forgotten by the model whenever the combined update gt conflicts with its imaginary gradient gt img . We can consider such clients to have a weight of zero in each round. However, we cannot directly detect conflicts for such clients because we have no access to their real gradients. To address this issue, we estimate their real gradients according to their recent gradients, and we call such gradient conflicts external conflicts:

Definition 3. In the tth communication round, there are external conflicts if there is a client h /∈ St whose latest gradient gt-k h conflicts with the combined update gt , where 0<k<τ, 0<τ<t.

An external conflict denotes the conflict between the assumed gradient of a client that has not been selected and the combined update gt . We add extra steps to prevent the model from forgetting data of clients beyond current selection. We also iteratively project the update gt onto the normal plane of the average of conflicting gradients for each previous round in a time-based order. The closer the round is, the later we make it the target to mitigate more conflicts between the update gt and more recent conflicting gradients according to Theorem 1. We scale the length of the update to || 1m ∑mk gtk || in the end because the length of all gradients can be enlarged by the projection. Algorithm 3 summarizes our steps to mitigate external conflicts.

3.4 Analysis

We analyze the capability of FedFV to mitigate gradient conflicts and its convergence. We show how FedFV can find a pareto stationary solution on convex problems.

Algorithm 3 MitigateExternalConflict

Input: gt ,GH, τ 1: for round t - i, i = τ, τ -1, ..., 1 do 2: gcon ← 0 3: for each client k = 1, 2, ..., K do 4: if tk = t - i then 5: if gt · gtk k <0 then 6: gcon ← gcon + gtk k 7: end if 8: end if 9: end for 10: if gt · gcon<0 then 11: Server computes gt ← gt - gt ·gcon ||gcon||2 gcon. 12: end if 13: end for 14: return gt

According to Theorem 2, it is possible for FedFV to bound the maximum conflict for any gradient by choosing k to let f (m, k, ϵ1, ϵ2)<ϵ2 with any given m, m ≥ 2, since the possible maximum value of conflicts between gk and the original average gradient is maxk |gk · ¯ g| ≤ m-1 m ϵ2(maxk ||gk||)2 .

In practice, we mitigate all the conflicts that have been detected to limit the upper bound of gradient conflicts of clients. We allow αm clients with large training losses to keep their original gradients to further enhance fairness. When α = 1, all clients keep their original gradients so that FedFV covers FedAvg, and when α = 0, all clients are enforced to mitigate conflicts with others. Parameter α thus controls the degree of mitigating conflicts and can be applied to seek for a balance.

Next, we show that FedFV can find a pareto stationary point according Theorems 3 and 4.

Theorem 3. Assume that there are only two types of users whose objective functions are F1(θ) and F2(θ), and each objective function is differentiable, L-smooth and convex. For the average objective F(θ) = 12 ∑ Fi(θ), FedFV with stepsize η ≤ 1L will converge to either 1) a pareto stationary point, 2) or the optimal θ∗ .

Theorem 4. Assume that there are m objective functions F1(θ), F2(θ),. . . , Fm(θ), and each objective function is differentiable, L-smooth and convex. If cos< ¯ g, ¯ g′ > ≥ 1 2 and ||¯ g|| ≥ ||¯ g′ || where ¯ g is the true gradient for the average objective F(θ) = 1m ∑ Fi(θ), then FedFV with step size η ≤ 1L will converge to either 1) a pareto stationary point, 2) or the optimal θ∗ .

4 RelatedWork

4.1 Fairness in FL

Federated Learning (FL) is first proposed by McMahan et al [2017] to collaboratively train a global model distributedly while preserving data privacy[Kairouz et al., 2019; Li et al., 2020b; McMahan et al., 2017]. A number of studies focus on collaborative fairness where the server allocates different models to clients according to their contribution [Lyu et al., 2020; Lyu et al., 2019; Xu and Lyu, 2020]. A few other studies address the fairness of a uniform accuracy distribution across devices [Cho et al., 2020; Hu et al., 2020; Li et al., 2019; Li et al., 2020a; Mohri et al., 2019]. Mohri et al. [2019] and Li et al. [2019] propose different federated objectives AFL and q-FFL to further improve fairness. Hu et al. [2020] observe the competing relation between fairness and robustness to inflating loss attacks. Abay et al. [2020] analyze the potential causes of bias in FL which leads to unfairness, and they also point out the negative impact of sample bias due to the party selection. Cho et al. [2020] also show that client selection has an impact on fairness, and they propose a bandit-based communication-efficient client selection strategy to overcome biasness. Huang et al. [2020] reweight clients according to their accuracy and numbers of times of being selected to achieve fairness. They use double momentum to accelerate the convergence. Different from these, we identify the conflicts among clients to be a potential cause of unfairness in FL. We mitigate such conflicts by computing a fairer average of gradients to achieve fair model performance across devices.

4.2 Gradient Projection

Gradient projection has been well studied in continual learning to mitigate the adversary impact of gradient updates to previously learned tasks [Chaudhry et al., 2018; Farajtabar et al., 2019; Guo et al., 2019; Lopez-Paz and Ranzato, 2017]. Lopez-Paz et al. [Lopez-Paz and Ranzato, 2017] project gradient by solving a quadratic programming problem. Chaudhry et al. [2018] project gradient onto the normal plane of the average gradient of previous tasks. Farajtabar et al. [2019] project the current task gradients onto the orthonormal set of previous task gradients. Yu et al. [2020] focus on adversary influence between task gradients when simultaneously learning multiple tasks. They iteratively project each task gradient onto the normal plane of conflicting gradients, which motivates our solution in this paper. To the best of our knowledge, we are the first to take the adversary gradient interference into consideration in FL. Specifically, our proposed FedFV method can build a connection between fairness and conflicting gradients with large differences in the magnitudes, and we prove its ability to mitigate two gradient conflicts (i.e., internal and external conflicts) and converge to Pareto stationary solutions.

5 Experiments

5.1 Experimental Setup

Datasets and Models

We evaluate FedFV on three public datasets: CIFAR-10 [Krizhevsky, 2012], Fashion MNIST [Xiao et al., 2017] and MNIST [LeCun et al., 1998]. We follow [McMahan et al., 2017] to create non-I.I.D. datasets. For CIFAR-10, we sort all data records based on their classes, and then split them into 200 shards. We use 100 clients, and each client randomly picks 2 shards without replacement so that each has the same data size. We use 200 clients for MNIST and preprocess MNIST in the same way as CIFAR-10. The local dataset is split into training and testing data with percentages of 80% and 20%. For Fashion MNIST, we simply follow the setting in [Li et al., 2019]. We use a feedforward neural network with 2 hidden layers on CIFAR-10 and Fashion MNIST. We use a CNN with 2 convolution layers on MNIST.

Baselines

We compare with the classical method FedAvg [McMahan et al., 2017] and FL systems that address fairness in FL, including AFL [Mohri et al., 2019], q-FedAvg [Li et al., 2019], FedFa [Huang et al., 2020] and FedMGDA+ [Hu et al., 2020]. We compare with q-fedavg, FedFa and fedmgda+ on all three datasets and compare with AFL only on Fashion MNIST, because AFL is only suitable for small networks with dozens of clients.

Hyper-parameters

For all experiments, we fix the local epoch E = 1 and use batchsize BCIFAR-10|MNIST ∈ {full, 64},BFashionMNIST ∈ {full, 400} to run Stochastic Gradient Descent (SGD) on local datasets with stepsize η ∈ {0.01, 0.1}. We verify the different methods with hyper-parameters as listed in Table 1. We take the best performance of each method for the comparison.

Implementation

All our experiments are implemented on a 64g-MEM Ubuntu 16.04.6 server with 40 Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz and 4 NVidia(R) 2080Ti GPUs. All code is implemented in PyTorch version 1.3.1. Please see https: //github.com/WwZzz/easyFL for full details.

5.2 Experimental Results

Fairness

We first verify the advantage of FedFV in fairness. In Table 2, we list the the mean, variance, the worst 5% and the best 5% of test accuracy on 100 clients created by splitting CIFAR10 where 10% of clients are sampled in each communication round. While reaching a higher mean of test accuracy across clients, FedFV also yields the lowest variance among all experiments except for q-FedAvg|q=5.0. Note that q-FedAvg sacrifices the best performance of clients from 69.28% to 63.60% and causes a significant reduction of the mean of accuracy from 46.85% to 45.25%. When reaching a similar variance to q-FedAvg|q=5.0 (9.59 versus 9.72), FedFV improves the mean of accuracy to 50.42%. Further, the worst 5% performance of all the experiments on FedFV is higher than that of any others, which indicates FedFV's effectiveness on protecting the worst performance.

Table 3 lists the model test accuracy on each client's data and the average and variance of the accuracy on FashionMNIST dataset. Different from settings in Table 2, we use a small network with only 3 clients, and the server selects all the clients to participate the training process in each communication round to eliminate the sample bias, which results in the absence of external conflicts. Therefore, we only use FedFV with α ∈ {0, 1 3 , 23 }, τ = 0 in this case. In our observation, even though FedFV with α = 0 can still find the solution with the highest mean of test accuracy 80.48%, it's a unfair solution where the variance is up to 13.76. Instead, FedFV with α = 23 also finds a solution with high accuracy 80.28%, and simultaneously keep the variance in a low level. We notice that AFL|ηλ=0.5 has the lowest variance. However, AFL only works for small works where the number of clients is limited. Despite of this, FedFV|α= 23 outperforms AFL|ηλ=0.5 on all the clients (i.e. shirt, pullover and T-shirt), as well as the mean of accuracy. Further, when compared to other methods except AFL, our method has the lowest variance down to 1.77, which validates the ability of FedFV to find a fair and accurate solution without the presence of sample bias.

From Table 2 and Table 3, we observe that FedFV is able to ease the negative impact of conflicting gradients with largely different magnitudes, which results in a fairer and accurate model with lower variance.

Accuracy and Efficiency

We further show that FedFV outperforms existing works addressing fairness in terms of accuracy and efficiency on CIFAR-10 and MNIST. All the methods are tuned to their best performance. As shown in Figure 2, when we only mitigate the internal conflicts, which is done by FedFV(τ = 0), we already converge faster while keeping a lower variance than the others. In addition, when we mitigate both the internal and external conflicts by FedFV(τ>0), there is a substantial improvement in the accuracy and efficiency again. FedFV outperforms state-of-the-art methods by up to 7.2% on CIFAR10 and 78% on MNIST, which verifies the ability of FedFV to reach a higher accuracy with less communication rounds

(b) MNIST

Figure 2: The mean (left) and the variance (right) of test accuracy on all clients on (a) CIFAR-10, and (b) MNIST. The results are averaged over 5 runs with different random seeds.

while still keeping a low variance. Thus, we demonstrate that FedFV's advantages in saving communication cost and finding a better generalization.

Effects of Projecting Order

To verify the effectiveness of our projecting order, we compare FedFV with another two cases: 1) projecting gradients in a random order of the projection targets; and 2) projecting gradients in an order of the projection targets that is reverse to FedFV, as shown in Table 4.

For CIFAR-10, 20% of clients are sampled in each communication round. For Fashion MNIST, all clients are selected in each communication round. We set α = 0, τ = 0 for all groups to confirm the effectiveness of the projecting order. If we project gradients to the targets in the loss-based order of FedFV, the variance is the lowest. Projecting gradients to the targets in a random order is also fairer than in an order reverse to FedFV, which indicates that the loss-based order used by FedFV helps improve fairness.

6 Conclusions and FutureWork

We identified the issue of conflicting gradients with large differences in the magnitudes, which brings unfairness in FL. To address this issue, we propose the Federated Fair aVeraging (FedFV) algorithm to mitigate the potential conflicts among clients before averaging their gradients. In addition, we show how well FedFV can mitigate the conflicts and how it converges to either a pareto stationary solution or the optimal on convex problems. Extensive experiments on a suite of federated datasets confirm that FedFV compares favorably against state-of-the-art methods in terms of fairness, accuracy and efficiency. In the future, we plan to build a complete theoretical analysis of FedFV on mitigating complex external conflicts.

A Proof

A.1 Proof of Theorem 1

Proof. For each gradient gi ∈ G, we project gi onto gk's normal plane in an increasing order with k. Thus, we have update rules

Therefore, g(k) i never conflicts with gk, since all potential conflicts have been eliminated by the updating rules. We then focus on how much the final gradient ¯ g′ conflicts with each gradient in G. Firstly, the last in the projecting order is gm, and the final gradient ¯ g′ will not conflict with it. Then, we focus on the last but second gradient gm-1. Following the update rules, we have

Let φ(k) i,j denote the angle between g(k) i and gj, and φi,j = φ(0) i,j , then the average of projected gradients is:

Similarly, we can compute the conflicts between any gradient gk ∈ G and ¯ g′ by removing the g(k) i from ¯ g′

Therefore, the later gk serves as the projecting target of others, the smaller the upper bound of conflicts between ¯ g′ and gk is, since ∑m-1 j=k ∑mi̸=j+1 ||g(j) i || ≤ ∑m-1 j=k-1 ∑mi̸=j+1 ||g(j) i ||.

A.2 Proof of Theorem 2

Proof. According to (5), the upper bound of the conflict between any client gradient gk and the final average gradient ¯ g′ can be expressed as

With the update rules of FedFV, we can infer that

Therefore, the maximum value of gradient conflict is bounded by

A.3 Proof of the Convergence

Proof of Theorem 3. Let g1 and g2 be the two types of users' updates of parameters in the tth communication round. When F(θ) is L-smooth, we have

There are two cases: conflicting gradients exist or otherwise. If there is no conflict between g1 and g2, which indicates g1 · g2 ≥ 0, FedFV updates as FedAvg does, simply computing an average of the two gradients and adding it to the parameters to obtain the new parameters θt+1 = θt -η¯ g, where g = 1 2 (g1 + g2). Therefore, when using step size η ≤ 1L , it will strictly decrease the objective function F(θ) [Li et al., 2020c]. However, if g1 · g2<0, FedFV will compute the update as:

Combining with (9), we have:

Since g1 · g2 = ||g1||||g2|| cosφ12, where φ12 denotes the angle between g1 and g2, after rearranging the items in this inequality, we have

To decrease the objective function, the inequality below should be satisfied

With η ≤ 1L , we have η - Lη2 ≥ 0, which promises the decrease of the objective function. On the other hand, we can infer that

Therefore, the objective function will always degrade unless 1)||g|| = 0, which indicates it will reach the optimal θ∗ , 2)cosφ12 = -1, then we can suppose g1 = -βg2 where β>0, and we will get1

So θt is a pareto stationary point as defined below:

Definition 4. For smooth criteria Fk(θ)(1 ≤ k ≤ K), θ0 is called a Pareto-stationary iff there exists some convex combination of the gradietns ∇Fk(θ0 ) that equals zero [D´esid´eri, 2012].

Proof of Theorem 4. With the assumption that F(θ) is Lsmooth, we can get the inequality:

Similar to the proof for Theorem 2, if there exist conflicts,

Since (- η2 + L2 η2 ) ≤ 0 with η ≤ 1L , the average objective will always degrade if we repeatedly apply the update rules of FedFV unless 1)||¯ g|| = 0, which indicates it will finally reach the optimal θ∗ , 2) ||¯ g′ || = 0, which indicates that all pairs of conflicting gradients have a cosine similarity of -1, leading to an existence of convex combination p which satisfies ∑ pigi = 0, since we can easily choose a pair of conflicting gradients gi and gj as the proof of theorem 3 does and then set the weights of the rest zero.