Harmonizing Federated Heterogeneous Optimization via Adaptive Objective Rectification

1 Introduction

Federated Learning (FL) [McMahan et al., 2017] has emerged as a dominant distributed learning blueprint, empowering a centralized coordinator to orchestrate global model optimization across edge clients without centralizing their raw assets. By anchoring training data securely ondevice, FL fundamentally satisfies modern strict privacy mandates. This architecture has matured beyond theoretical exercises to power large-scale commercial deployments by technology pioneers such as Google and Apple, while demonstrating immense promise in privacy-critical sectors including finance, digital healthcare [Lu et al., 2026], and smart cities.

Despite its structural elegance, executing FL in realistic ecosystems exposes a fundamental vulnerability: the Information Asymmetry Paradox. Because individual clients experience distinct contexts, their local data repositories are inherently non-Independent and Identically Distributed (nonIID) in terms of distribution and volume. Compounding this is the necessity of partial client participation, where only an unpredictable fraction of nodes remains active during any single round. Consequently, local optimization vectors drift along divergent trajectories, causing severe federated heterogeneous optimization anomalies that degrade global model performance [Sun et al., 2025; Li et al., 2020b] and lead to outcomes far inferior to traditional centralized training [Wang et al., 2020b; Acar et al., 2021; Lu et al., 2025].

∗Corresponding author.

To bridge this deficit, contemporary solutions typically attempt to constrain local updates to prevent them from straying from the global pathway [Bakhtiari et al., 2020; Yoon et al., 2021]. However, enforcing rigid behavioral control within volatile environments forces state-of-the-art frameworks to depend on brittle, highly idealized assumptions. For instance, landmark protocols like SCAFFOLD [Karimireddy et al., 2020] and FedDyn [Acar et al., 2021] require stable client participation to properly calibrate their internal control variates or dynamic regularizers. Similarly, Mime [Karimireddy et al., 2021] relies on relatively homogeneous gradient spaces, whereas FedSplit [Pathak and Wainwright, 2020] collapses outside the narrow confines of full client engagement and strictly convex objectives. When confronted with realworld, deeply fragmented heterogeneous data, the efficacy of these methods deteriorates rapidly.

Departing from conventional reactive drift correction, we propose HaFedHo, a proactive paradigm designed to natively align decentralized execution with centralized optimization. The core innovation of HaFedHo is the explicit rectification of each client's local objective function before training begins. We achieve this by compelling each client to optimize an adaptive, Taylor-approximated surrogate of the true, datacentralized global objective. By isolating the local function and projecting the missing collaborative topologies via an agile, first-order Taylor series expansion (refreshed dynamically via cross-round remnants), HaFedHo effectively removes the information barrier between silos.

Backed by rigorous theoretical guarantees, we prove the convergence properties of HaFedHo across both convex and complex non-convex surfaces under extreme data nonIID conditions. Empirical validations across three standard benchmarks (FEMNIST, CIFAR-10, and CIFAR-100) confirm that HaFedHo consistently outperforms existing strong baselines such as SCAFFOLD [Karimireddy et al., 2020], MimeLite [Karimireddy et al., 2021], MOON [Li et al., 2021], and FedDyn [Acar et al., 2021], showcasing remarkable resilience in scenarios defined by severe data sparsity and ultra-low node availability.

Our core contributions are summarized as follows:

• We present a novel optimization framework that transforms isolated local training by embedding a Taylorseries approximation of the missing global data landscape directly into the local objective function, supported by comprehensive convergence proofs.
• We introduce a practical, communication-efficient implementation of this framework that dynamically updates its approximation of the global objective, and mathematically demonstrate that its internal estimation error decays toward zero as the iterations progress.
• Through extensive empirical evaluation, we demonstrate that HaFedHo yields superior accuracy trajectories and accelerated convergence behavior compared to state-ofthe-art optimizers, particularly in highly adverse client participation environments.

2 Preliminaries

2.1 Federated Learning

We study a standard decentralized ecosystem comprising a focal server and n isolated edge nodes, where each node i ∈ [n] restricts computation to its private domain Di. The objective is to cooperatively resolve a global empirical risk minimization problem under a uniform consensus:

where fi(w) = 1 |Di | ∑π∈Di ℓ(w, π) characterizes the localized loss surface over the node's distinct data partition.

In any active communication round t, participants pull the consensus state to initialize their local models (w(t,0) i = wt ) and advance across λ sequential local epochs using Stochastic Gradient Descent (SGD):

Upon completing local trajectories, the server orchestrates a global synchronization step matching the FedAvg protocol [McMahan et al., 2017]:

2.2 Data-centralized Training

Conversely, the unconstrained data-centralized baseline operates without information silos. The master server directly operates on the joint pool D = ⋃ni=1 Di by executing optimization directly over the global data distribution:

Centralized optimization shifts the parameters iteratively through identical sequential update steps:

where dt tracks the true global gradient trajectory over the fully unified loss landscape Ξ.

3 RelatedWorks

We group them by how they handle the global information gap across isolated client nodes.

Communication-Intensive Variance Reduction. The most direct way to eliminate localized client drift is to restrict the local optimization horizon to a single step, as seen in protocols like FedSMB [Bakhtiari et al., 2020] and FedSGD [McMahan et al., 2017]. This ensures that the global update path exactly matches centralized training. However, this strategy introduces massive communication overhead and renders the training process highly unstable when paired with modern deep architectures that rely on stateful layers like batch normalization or dropout. Consequently, modern FL research focuses on extending the local training duration while correcting the resulting client drift.

History-Driven Trajectory Realignment. One prominent approach uses historical optimization metadata to adjust local updates and smooth out cross-client variances [Hsu et al., 2019; Karimireddy et al., 2021; Khanduri et al., 2021; Xu et al., 2021; Karimireddy et al., 2020; Chen et al., 2024]. For example, SCAFFOLD [Karimireddy et al., 2020] introduces stateful control variates to track client drift, while Mime and its lightweight counterpart MimeLite [Karimireddy et al., 2021] replace local momentum terms (originally explored in FedavgM [Hsu et al., 2019]) with precise global gradient vectors collected across all participants. While effective on uniform datasets like FEMNIST [Caldas et al., 2018], these methods struggle in highly volatile environments with sporadic client availability. Furthermore, transmitting historical state vectors increases communication costs and heightens the risk of information leakage, making systems more vulnerable to advanced gradient inversion attacks [Lu et al., 2024; Geiping et al., 2020; Lu et al., 2023].

State-Constrained Optimization Anchoring. Another strategy adds regularizers, proximal terms, or feature alignment structures directly to the local loss functions to penalize deviations from the global archetype [Li et al., 2020a; Gao et al., 2022; Acar et al., 2021; Wang et al., 2024; Hu et al., 2024; Chu et al., 2025]. For instance, FedProx [Li et al., 2020a] introduces a proximal term to bound local drift, while FedDyn [Acar et al., 2021] uses dynamic quadratic penalty terms to align local minima with the global objective under stable participation assumptions. Similarly, FedSplit [Pathak and Wainwright, 2020] applies operator splitting methods to tackle non-IID data distributions, and contemporary objective-driven approaches [Gao et al., 2022; Wang et al., 2024; Hu et al., 2024; Chu et al., 2025] regularize local spaces via features or constraints. However, these methods act as reactive patches; they try to control client drift after the client objective functions have already diverged from the centralized target. In contrast, our approach directly approximates the global data-centralized objective from the start, ensuring that local optimization vectors naturally align with the centralized target.

Post-Hoc Topology Reshaping. A final category of work focuses entirely on the server side, modifying the aggregation phase to minimize the impact of non-IID data distributions and domain shifts [Wang et al., 2020a; Chen and Chao, 2020; Xu et al., 2024; Ning et al., 2024; Zeng et al., 2024; Shen et al., 2025; Zheng et al., 2025]. Notable examples include FedBN [Li et al., 2020b], which excludes batch normalization layers from the aggregation step, and FedLWS [Shi et al., 2025a; Shi et al., 2025b], which introduces adaptive weight rebalancing based on client features. Alternative aggregation paradigms further reshape the global landscape via Bayesian ensembling [Chen and Chao, 2020], layer matching [Wang et al., 2020a], or pluggable client scheduling filters [Xu et al., 2024; Ning et al., 2024; Zeng et al., 2024; Shen et al., 2025; Zheng et al., 2025]. Because our proposed method operates entirely during the local training phase, it is complementary to these server-side techniques, variance-reducing momentum methods [Wang et al., 2023], Sharpness-Aware Minimization (SAM) variants [Sun et al., 2023; Xing et al., 2025].

4 Methodology

4.1 Motivation

Our approach is motivated by a simple principle: if each client's local optimization task closely mirrors the ideal centralized training objective, the overall FL process should yield a better global model.

We observe that if we expand the data-centralized objective Ξ(w) in a Taylor series around the place where w = wt (i.e., Ξ(w) = Ξ(wt ) + [∇Ξ(wt )]⊤(w - wt ) + O(∥w - wt ∥22 ), where O(∥w - wt ∥22 ) is the Peano's remainder error), the Peano's remainder error would gradually vanish as the iteration increases. This suggests that if each client ignores the Peano's remainder error and solves the first-order Taylor's series approximation, all the clients will converge to the stationary point of Ξ(w). Therefore, we have the following ideal way to imitate the data-centralized training.

Proposition 1. Suppose all clients have access to the global loss Ξ(wt ) and the full-batch centralized gradient ∇Ξ(wt ). If each client i minimizes the local objective

then the federated optimization process converges to a stationary point of Ξ(w).

Proof. See supplementary material.

While Proposition 1 provides a theoretical ideal, it is not directly applicable in FL because clients cannot compute the global quantities Ξ(wt ) and ∇Ξ(wt ). In the following section, we present a practical method to approximate this ideal objective.

Algorithm 1: A Complete Description of HaFedHo

Input : T; w(0,0) i =w0 ; g0i = g0 ; ˆ f 0 i = ˆ f 0 = 0; learning rate η; the number of SGD steps λ; penalty factor µ. Output : The global model wT . for t = 1, 2, · · · , T do Randomly select m clients M ⊆ {1, 2, · · · , n}. The server broadcasts gt-1 , ˆ f t-1 to clients. for client i ∈Mdo wt = w(t,0) i =w(t-1,0) i -ηgt-1 . if client i is sampled in the last round then h = gt-1 - pigt-1 i . ˆ f = ˆ f t-1 - pi ˆ f t-1 i . else h = gt-1 . ˆ f = ˆ f t-1 . Define local objective: φi(w) = fi(w) + µ( ˆ f + h⊤(w - wt )). Perform λ local SGD steps: w(t,k+1) i ←w(t,k) i -η∇φi(w(t,k) i ). Compute average local loss: ˆ f t i ← 1λ ∑λ-1 k=0 fi(w(t,k) i ). Compute total local gradient: gti ← ∑λ-1 k=0 ∇φi(w(t,k) i ). Client i uploads ˆ f t i and gti to the server. ˆ f t = ∑mi=1 1m ˆ f t i . gt = ∑mi=1 1m gti .

4.2 HaFedHo: The ProposedMethod

The optimal guarantee in Proposition 1 inspires us to develop a method from the local objective perspective. However, since clients lack knowledge of the entire dataset D, it is impractical for each client to employ the Taylor estimation in Eq. (6) and calculate the full-batch gradient ∇Ξ(wt ). To overcome this challenge, our core idea is to decouple Ξ(w,D) into multiple client-level objective functions, i.e., fi(w,Di), i ∈ {1, 2, · · · , n}, according to the datasets of clients and then dynamically approximate each fi(w,Di) with the corresponding gradients.

Global-Local Objective Decomposition. To bridge the gap between federated execution and centralized training, we must first isolate the local optimization landscape from the overarching global target. The empirical risk of the centralized model Ξ(w,D) can be inherently decomposed into the active client's local loss and the aggregate loss of all unseen peers:

where pi = |Di| |D| represents the data fraction. Directly optimizing this joint function is impossible due to strict data silos. While one could theoretically approximate the unseen term ∑nj̸=i pjfj(w,Dj) using a first-order Taylor expansion around the current global weights wt , requesting real-time gradients ∇fj (wt ) across the entire network is both communication-prohibitive and a severe privacy risk.

First-Order Proxy via Historical Trajectories. Rather than forcing costly synchronizations, we can construct a surrogate for the missing gradient information using historical trajectory data. During standard model aggregation, the server already compiles the global update vector gt-1 = ∑ni=1 pigt-1 i . This aggregated vector serves as a natural, albeit slightly delayed, proxy for the true centralized gradient.

By projecting the non-local components using this historical proxy, client i constructs a locally computable surrogate objective. To isolate the exact contribution of the other clients and prevent self-reinforcement, we subtract the active client's previously uploaded gradient gt-1 i from the global average:

(8)

Applying identical arithmetic to the scalar loss valuesestimating the unseen global loss ∑nj̸=i pjfj(wt ) via ∑ni=1 pifi(wt-1 ) - pifi(wt-1 )-transforms the objective into a fully computable state:

where f t-1 = ∑ni=1 pif t-1 i and f t-1 i = fi(wt-1 ). Because fi is a zero-order scalar, transmitting it to the server imposes near-zero bandwidth overhead. Furthermore, protecting these scalars via cryptographic protocols (such as Secure MultiParty Computation) is computationally trivial compared to encrypting high-dimensional gradient matrices, ensuring robust privacy preservation.

Mitigating Stale States in Partial Participation. Operating under realistic federated regimes dictates that only a small fraction of the client pool engages per round. Consequently, the global aggregates ˆ f t-1 and gt-1 are inherently biased by the sampled cohort.

A critical overlap issue arises if client i is sampled in successive rounds: its local state is already baked into the global broadcast, risking an over-correction or momentum bias. To neutralize this, we selectively subtract the local historical terms (pigt-1 i and pi ˆ f t-1 i ) only if the client participated in round t - 1. For newly sampled clients, the raw global aggregates are utilized directly, as their previous updates are not present in the immediate global state.

Finally, to stabilize training dynamics and scale the impact of the baseline local loss, we introduce a tuning parameter µ to modulate the intensity of the proxy terms. This yields the ultimate minimization target for the federated system:

where the client-side surrogate evaluates to φi(w) = fi(w, Di) + µ [ ˆ f t-1 ] + µ [h]⊤ (w - wt ). The precise mechanics for determining the dynamically adjusted scalars ˆ f and vectors h based on participation history are detailed in Algorithm 1, while Figure 1 illustrates the geometric intuition guiding this optimization trajectory.

4.3 Theoretical Analysis

We analyze the convergence properties of HaFedHo under standard assumptions from the FL literature [Li et al., 2020a; Karimireddy et al., 2020; Karimireddy et al., 2021; Gao et al., 2022; Acar et al., 2021].

Assumption 1 (Bounded Dissimilarity). For a positive constant B ≥ 1, we assume that the variance of local gradients with respect to the global gradient is bounded: 1n ∑ni=1 ∥∇fi(w)∥2 ≤ B2 ∥∇Γ(w)∥2 , ∀w.

Assumption 2 (Bounded Variance). The stochastic gradient ∇fi(w; ζi) computed on a mini-batch ζi is an unbiased estimator of the true local gradient ∇fi(w), with bounded variance: Eζi [∥∇fi(w; ζi) - ∇fi(w)∥2 ] ≤ σ2 , ∀i, w.

Assumption 3 (Lipschitz Smoothness). Each local loss function fi is β-smooth, meaning its gradient is Lipschitz continuous: ∥∇fi(x) - ∇fi(y)∥ ≤ β ∥x - y∥ , ∀i, x, y.

Definition 1 (Estimation Error). We use local gradient gti to estimate ∇fi(wt ). Thus, the quantity h defined as:

actually estimates ∇Γ(wt ). Besides, we utilize a hyperparameter µ to control the magnitude of h, therefore, the estimation error Θ is formally defined as:

Theorem 1 (Estimation Error Bound). Under Assumptions 1-3, for a non-convex objective, client sampling ratio mn , T communication rounds, λ local steps, and a sufficiently small learning rate η = O( 1 T ), the estimation error Θ is bounded:

where C1 = 18µ2β2B2 (1 + mn p2i λ2 ), C2 = (µ - 1)2 + mn (λ2µ2p2i B2 + 2), C3 = 9λµ2β2σ2 (1 + mn p2i ) and C4 = σ2 (1 + mn p2i λ2 ).

Proof. See supplementary material.

Analysis: Theorem 1 characterizes the effectiveness of using the latest global gradients gt and the local gradient gti to estimate all the local gradients ∑nj̸=i ∇fj(wt ) of other clients. As the number of rounds T approaches infinity, the first three terms in the upper bound of the error gradually vanish, leaving only the last term C4µ2 . Since µ can be adjusted as a hyperparameter inversely proportional to the iteration round t, C4µ2 also tends to zero as the iterations progress. This indicates that as HaFedHo converges (Theorem 2), the estimation error becomes extremely small, almost negligible.

Theorem 2 (Convergence). Consider HaFedHo running for T rounds, with λ local SGD steps, and a client sampling ratio of mn . If the learning rate η is sufficiently small, then the average squared gradient norm is bounded as follows:

where Γ(w0 ) -Γ∗ is the initial sub-optimality, and the constants CG and CΣ are constants (see the supplementary).

Proof. See supplementary material.

5 Experiments

5.1 Experimental Setup

We implement our evaluation framework using FedML [He et al., ].

Dirichlet Data Partitioning. Following the standard FedML protocol, we simulate statistical heterogeneity across n clients via Latent Dirichlet Allocation. Specifically, we sample pl ∼ Dirichletn(q), assigning a fraction pl,i of class l instances to client i. The concentration parameter q dictates the degree of non-IIDness, with lower values yielding higher statistical heterogeneity.

Baselines. Our approach is benchmarked against eight stateof-the-art federated learning algorithms: MOON [Li et al., 2021], FedProx [Li et al., 2020a], pFedAFM [Yi et al., 2025], FedDyn [Acar et al., 2021], MimeLite [Karimireddy et al., 2021], FedLWS [Shi et al., 2025a], SCAFFOLD [Karimireddy et al., 2020], and FedAvg [McMahan et al., 2017].

Hyperparameters. For the FEMNIST dataset, we configure a total of n = 3400 clients with a participation rate of 10%. Conversely, evaluations on CIFAR-10 and CIFAR-100 utilize n = 100 clients, maintaining the 10% sampling fraction. Optimal hyperparameters for all baseline methods were rigorously tuned for fair comparison (refer to the supplementary material).

Evaluation Metric. Performance is quantified using test accuracy, defined as the proportion of correctly predicted test instances by the aggregated global model.

5.2 Performance Evaluation

Test Accuracy. Table 1 benchmarks the predictive accuracy across three non-IID datasets. HaFedHo establishes a clear state-of-the-art, consistently surpassing all baselines. Standard methods (e.g., FedAvg, FedProx, SCAFFOLD, MOON) plateau at similar levels. Notably, while FedDyn achieves competitive peaks, its performance is highly erratic, dropping below standard FedAvg on specific FEMNIST and CIFAR tasks.

Communication Efficiency. Table 2 highlights HaFedHo's accelerated convergence. By ensuring that local objective functions remain consistent with the global objective, HaFedHo aggressively dampens the client-drift caused by non-IID data, requiring significantly fewer communication rounds to reach target accuracy thresholds compared to prior arts.

Robustness to Data Heterogeneity. As illustrated in Table 1 and Fig. 1, heightened statistical heterogeneity uniformly degrades model performance across all frameworks. However, HaFedHo dominates across all non-IID regimes. This resilience is achieved by cross-pollinating objective function data among clients, ensuring local update directions are tightly aligned with the ideal global gradient.

Resilience to Low Participation. Production FL systems typically sample minute fractions from vast device pools [Shejwalkar et al., 2022], rendering the conventional 10% participation assumption [Wang et al., 2020b; Acar et al., 2021] unrealistic. Table 3 evaluates extreme lowparticipation regimes (0.2%, 0.5%, 1%). In the highly skewed CIFAR-10 setting, where competitors suffer catastrophic performance drops, HaFedHo maintains robust accuracy trajectories well above the FedAvg baseline.

Scalability to Massive Cohorts. We simulate large-scale parallel updates on FEMNIST using 500 and 1020 active clients (Figs. 2a and 2b). Most existing methods fail to scale, matching basic FedAvg. Crucially, SCAFFOLD diverges entirely, corroborating prior findings [Li et al., 2021; Karimireddy et al., 2021; Karimireddy et al., 2020; Khanduri et al., 2021] that strictly bounded control variates fail under massive decentralization. HaFedHo bypasses this bottleneck, effectively absorbing global feedback to refine local objectives and achieving superior convergence rates.

5.3 Ablation Study

The core of HaFedHo is its objective rectification strategy that rectifies the local objective to be fi(w) + µ( ˆ f + h⊤(w -

wt )). A simpler alternative might be to perform gradient rectification, where the correction term µh is added directly to the local gradient ∇fi(w) instead of being incorporated into the objective function via its integral. We compare these two strategies in Figure 3.

The figure displays the accuracy curves of the two methods under different µ over CIFAR-10 and CIFAR-100. By tuning µ to the best value, gradient rectification (the best µ = 0.001 for CIFAR-10 and CIFAR-100) only marginally outperforms the FedAvg baseline, whereas our objective rectification (the best µ = 0.01 for CIFAR-10 and CIFAR-100) substantially enhances FL. This demonstrates that simply correcting the gradient direction is insufficient. By modifying the objective function itself, HaFedHo ensures that the entire local optimization trajectory at the client is better aligned with the centralized training goal, leading to a much more effective mitigation of client drift.

6 Limitations and FutureWork

Higher-Order Taylor Approximations. Currently, HaFedHo aligns local objectives with the centralized ideal using a first-order Taylor series expansion. Although this approach yields substantial performance gains in federated settings, a persistent accuracy gap remains when compared to purely centralized training. This discrepancy primarily stems from the approximation errors inherent in first-order truncations. Incorporating higher-order Taylor terms could theoretically provide a more precise objective rectification. However, doing so introduces a critical trade-off between computational feasibility and estimation accuracy. Exploring lightweight, higher-order approximation techniques remains a promising avenue for future research.

7 Conclusion

We presented HaFedHo, a novel FL framework that bridges the gap between local client updates and the global datacentralized optimum. We prove that the inherent estimation error naturally decays over the course of training. Comprehensive empirical evaluations validate our theoretical claims, demonstrating that HaFedHo achieves superior terminal accuracy and accelerated communication efficiency compared to existing state-of-the-art baselines. Furthermore, it exhibits exceptional resilience under highly restrictive real-world con-

(a) Ablation on CIFAR-10.

(b) Ablation on CIFAR-100.

Figure 3: Ablation study comparing Objective Rectification (our method with red line) with a simpler gradient rectification approach (blue line). Modifying the objective function is crucial for performance.

ditions, including severe data heterogeneity and extremely low client participation regimes.

Acknowledgments

This work was supported by the Key R&D Program of Zhejiang Province under Grant No. 2025C01084 and the CAAIMindSpore Open Fund, developed on the OpenI Community, under Contract No. CAAIXSJLJJ 2025 MindSpore 04.