SDFLoRA: Selective Decoupled Federated LoRA for Privacy-preserving Fine-tuning with Heterogeneous Clients

1 Introduction

Parameter-efficient federated fine-tuning (PEFT) has emerged as a critical paradigm for adapting Large Language Models (LLMs) to distributed, privacy-sensitive datasets. By optimizing only lightweight adapters, clients avoid full-model communication and memory overhead, enabling deployments in domains such as healthcare, finance, and on-device intelligence. Among PEFT approaches, low-rank adaptation (LoRA) [Hu et al., 2022] is the dominant choice in federated LLM adaptation and is widely adopted in recent FL-LLM systems [Wang et al., 2024; Yang et al., 2025; Long et al., 2024].

∗Corresponding author.

Despite these advances, practical Federated Learning (FL) deployments could exhibit strong heterogeneity, as clients originate from different domains, tasks, and user populations. LoRA updates naturally contain both transferable directions shared across clients and client-specific directions that encode local semantics and personalization. However, existing federated LoRA pipelines typically enforce shared alignment and aggregation over all update directions, which suppresses client-specific adaptations and limits personalization. This issue is further exacerbated by rank heterogeneity, where clients adopt different low rank configurations due to diverse system constraints, making na¨ıve aggregation even more biased and unstable.

Existing solutions for federated LoRA under heterogeneous ranks can be grouped into two paradigms. The first paradigm enforces a unified rank across all clients, which reduces flexibility and can waste resources on constrained devices [Cho et al., 2024]. The second paradigm aligns heterogeneous LoRA updates into a shared subspace to enable aggregation. However, it tends to align all directions indiscriminately, including client-specific components that should remain private and unshared, thereby harming personalization [Zhao et al., 2023]. Meanwhile, differential privacy (DP) is widely adopted in FL to protect private client information. When such structurally entangled updates are globally aligned, adding noise to the entire update could perturb clientspecific components that are not meant to be shared, leading to utility loss without delivering a meaningful privacy guarantee [Wei et al., 2023].

Motivated by these observations, we propose Selective Decoupled Federated LoRA (SDFLoRA), which decomposes client adapter into a shared module that captures transferable knowledge and a private module that preserves clientspecific adaptations. Only the shared module participates in selective subspace alignment and aggregation across clients, while private modules remain private and unaggregated. To control rank growth induced by stacking-based aggregation, we further introduce low rank re-compression to maintain a fixed rank budget for the shared module. This structure naturally supports privacy aware optimization by applying DP

mechanisms exclusively to the shared module updates, avoiding unnecessary perturbations on local adaptations.

Our contributions are summarized as follows:

• We propose SDFLoRA, a structure-aware federated LoRA framework that decouples each client adapter into a shared module and a private module, and performs selective subspace alignment and aggregation exclusively on the shared module. The shared module is further constrained by a fixed rank budget via low rank recompression, enabling stable and efficient aggregation.
• We introduce a DP compatible optimization enabled by the decoupling, where differential privacy noise is injected only into the aggregated shareable update. This design avoids unnecessary perturbations on clientspecific directions and leads to improved robustness under privacy constraints.
• Extensive experiments on multiple benchmarks with heterogeneous clients demonstrate that SDFLoRA outperforms representative federated LoRA baselines, particularly in achieving a superior balance between privacy protection and model performance.

2 RelatedWork

2.1 Federated Optimization under Client Heterogeneity

FL enables collaborative model training without centralizing data by alternating between local optimization and serverside aggregation. Federated Averaging (FedAvg) [McMahan et al., 2017] is the canonical approach, but its simple weighted averaging mechanism often leads to unstable convergence under non-IID data distributions, partial client participation, and multiple local update steps, a phenomenon commonly referred to as client drift.

To mitigate heterogeneity-induced optimization challenges, a lot of work has proposed algorithmic modifications from different perspectives. FedProx [Li et al., 2020] introduces a proximal regularization term to constrain local updates from deviating excessively from the shared model. FedDyn [Durmus et al., 2021] and FedDC [Gao et al., 2022] further address client drift by incorporating dynamic correction terms that explicitly decouple local objectives from the shared optimization trajectory. Another line of work focuses on correcting biases caused by heterogeneous local computation. FedNova [Wang et al., 2020] normalizes client updates according to their effective number of local steps, while SCAFFOLD [Karimireddy et al., 2020b] reduces update variance through control variates shared between the server and clients. From an optimization-centric viewpoint, MIME and MIME-Lite [Karimireddy et al., 2020a] reinterpret federated learning as biased stochastic optimization and improve convergence by aligning federated updates with centralized SGD via momentum and gradient correction.

Despite their demonstrated effectiveness, these methods are predominantly designed for full-parameter federated optimization and implicitly assume homogeneous model structures across clients. When parameter-efficient fine-tuning is adopted, additional forms of heterogeneity emerge. In particular, low-rank adapters introduce structural heterogeneity, where clients may employ different adapter ranks or decompositions due to resource constraints or task diversity, which cannot be directly handled by existing optimization-centric approaches.

2.2 Parameter-Efficient Fine-Tuning and Federated LoRA

PEFT has become a standard approach for adapting large models under resource constraints. LoRA [Hu et al., 2022] freezes the backbone and learns low-rank updates for selected layers, greatly reducing trainable parameters and communication costs, which makes it especially attractive in FL.

Several studies extend LoRA to federated settings. FedIT [Zhang et al., 2024] performs FedAvg-style aggregation over LoRA parameters, enabling efficient collaboration but implicitly assumes homogeneous adapter configurations and can be sensitive to rank mismatch. To mitigate rank heterogeneity, recent work has explored stacking-based aggregation to align heterogeneous low-rank updates into a shared subspace. FLoRA [Wang et al., 2024] is a representative approach that constructs an expanded low-rank subspace via stacking, providing a principled way to aggregate clients with different ranks without padding or truncation. Other variants improve initialization, aggregation bias, or fairness properties for federated LoRA training [Bian et al., 2025; Guo et al., 2025].

These methods demonstrate that stacking is an effective tool for handling rank heterogeneity. However, they typically treat each client adapter as a monolithic update and implicitly assume that all low-rank directions should be globally aligned and aggregated. In federated LLM fine-tuning, client updates may contain a mixture of transferable knowledge and locally specific adaptations; indiscriminate alignment can over-regularize personalization and become fragile when additional constraints (e.g., privacy noise) are introduced.

2.3 Personalization and Privacy in Federated Learning

Personalized FL aims to account for client-specific objectives and data distributions by learning both shared and private components. Representative methods include Ditto [Gao et al., 2024], which couples a shared model with personalized local models via regularization, and FedRep [Collins et al., 2021], which separates shared representations from clientspecific heads. These approaches highlight the importance of explicitly distinguishing between shared and locally specialized parameters, but they are not designed around the structural properties of low-rank adapters in LLM fine-tuning.

Privacy preservation is another fundamental concern in FL. Differential privacy (DP) [Dwork, 2006] provides formal guarantees, and DP-SGD [Abadi et al., 2016] has become a standard mechanism to protect training updates by gradient clipping and noise injection. In federated settings, differential privacy is typically enforced through client-side or server-side perturbations, such as DP-SGD-based noise injection on local updates or aggregated models [Kairouz et al., 2021]. However, applying DP noise uniformly to all parameters can significantly degrade utility, particularly for large models and heterogeneous clients, and adversarial or poisoned updates further exacerbate this instability in federated aggregation [Lu et al., 2024]. Recent work suggests that selective or structured privacy mechanisms may yield better utility-privacy trade-offs, yet existing federated LoRA approaches do not explicitly leverage adapter structure to support such selective protection.

2.4 Positioning of OurWork

Compared to prior federated LoRA methods, our work focuses on the semantic roles of low-rank updates and the resulting need for structural awareness in aggregation. Rather than proposing another stacking mechanism, we introduce SDFLoRA that separates globally alignable updates from locally personalized ones. Stacking-based aggregation is then applied selectively to the shared module, while private modules remain private and unaggregated. This perspective addresses not only how to aggregate heterogeneous ranks, but also which updates should be aggregated under personalization and privacy constraints, enabling robust federated LLM fine-tuning with improved stability under DP noise.

3 Method

3.1 Preliminaries: Federated LoRA Fine-tuning

We consider federated fine-tuning with K clients. In round t, a subset St of clients participates, each optimizing a local objective Fk(·) on its private data. Let W denote the frozen backbone parameters of an LLM. LoRA [Hu et al., 2022] augments a target linear layer W ∈ Rdout×din with a low-rank

where rk is the adapter rank (potentially different across clients). During local training, client k updates {Ak,Bk} (and keeps W fixed). The server aggregates client updates and broadcasts the aggregated adapter for the next round.

Challenge: rank heterogeneity and semantic heterogeneity. When ranks differ across clients, directly averaging low-rank factors is ill-defined without padding or truncation, and may introduce biased updates. Stacking-based aggregation alleviates dimensional mismatch by aligning heterogeneous updates into an expanded subspace, but implicitly assumes that all updates should be globally aligned and aggregated. In federated LLM fine-tuning, client updates often contain a mixture of globally transferable knowledge and client-specific adaptations; forcing shared alignment on all updates can harm personalization and becomes fragile under differential privacy noise. Our goal is therefore to design an aggregation scheme that is robust to rank heterogeneity while respecting the semantic roles of low-rank updates.

3.2 Dual-Model LoRADecomposition

We propose to decompose each client adapter into two structurally separated low-rank modules:

where ∆W (g) k is a shared module intended to capture transferable directions shared across clients, and ∆W (l) k is a private module for client-specific adaptations.

This decomposition reflects an important observation in federated LLM fine-tuning: client-side low-rank updates are semantically heterogeneous. Some update directions correspond to transferable knowledge that should be aligned and shared across clients, while others encode client-specific or domain-specific adaptations whose forced alignment can be harmful. By explicitly separating ∆W (g) k and∆W (l) k , we impose a structural constraint on the subspaces that are allowed to participate in cross-client aggregation.

As illustrated in Figure 1, we decouple transferable (public) updates from client-specific (private) adaptations, enabling selective aggregation and privacy-preserving training.

Federated roles. The two modules play different roles in the federated process: (i) the shared module participates in cross-client aggregation and is broadcast each round; (ii) the private module is never aggregated and remains private to the client. This structural separation is the key design choice that allows us to restrict subspace alignment to semantically appropriate updates, rather than treating each LoRA adapter as a monolithic update.

Optimization objective. In each round t, client k ∈ St receives the current shared module θ(g,t) (the collection of all {A(g) ,B(g) } across adapted layers) and keeps its private module θ(l,t) k . The client performs E steps of local optimization to minimize:

and uploads only the updated shared module θ(g,t) k (or its update) to the server. The private module is kept on-device and continues across rounds.

3.3 Selective Stacking Aggregation on the Shared Module

We propose selective stacking, a stacking-based aggregation strategy that is applied exclusively to the shared module while explicitly excluding client-private updates from cross-client alignment. Let θ(g,t) k denote the shared-module parameters trained by client k at round t for a given layer, i.e., θ(g,t) k = {A(g,t) k ,B(g,t) k } with rank r(g) k . To aggregate heterogeneous ranks without padding or truncation, we perform stackingbased aggregation only on the shared module.

Stacking construction. For a target layer, we form a stacked representation:

where pk is the aggregation weight (e.g., proportional to the local data size) and Concat(·) concatenates along the rank dimension. This yields an aggregated update

which is well-defined under rank heterogeneity. Importantly, private modules are excluded from stacking and aggregation, avoiding forced alignment of client-specific directions.

The stacking operation preserves the following equivalence:

while avoiding direct averaging of incompatible low-rank factors. In contrast to padding-based alignment, stacking delays subspace merging until after aggregation, preventing premature mixing of heterogeneous update directions.

Rank control by re-compression. A practical consequence of stacking is that the effective rank of the aggregated shared update grows with the number of participating clients and communication rounds. Concretely, the concatenation in Eqs. (5)-(6) expands the rank dimension, which increases both the transmission cost (larger adapter factors) and the memory footprint on the server and clients.

Moreover, under privacy-aware training, stochasticity (e.g., DP noise) and client heterogeneity tend to inject low-energy directions that accumulate across rounds, leading to unstable optimization and diminished generalization if left uncontrolled. To keep the shared module within a fixed budget, we apply low-rank re-compression to ∆W (g) t :

In practice, we compute a truncated SVD (or an equivalent low-rank factorization) ∆W (g) t = UΣV ⊤ and retain only the top-rmax singular components, yielding ∆W (g) t ≈ UrΣrV ⊤ r . We then convert this truncated form back to LoRA-style factors by setting ˜B(g) t = UrΣ1/2 r and ˜ A(g) t = Σ1/2 r V ⊤ r , so that ˜B(g) t ˜ A(g) t matches the best rank-rmax approximation under the Frobenius norm. This step acts as a spectral filter that preserves dominant, transferable directions while discarding tail components that are more likely to be noise-dominated, and it keeps the communicated shared module compact for the next round. Finally, the server broadcasts ( ˜ A(g) t , ˜B(g) t ) as the next-round shared module.

3.4 Privacy Aware Optimization (DP Compatibility)

Our decoupled design is naturally compatible with differential privacy (DP). The key idea is to apply DP mechanisms only to the shared module, while keeping private modules noise-free.

Client-side DP-SGD on the shared module. During local training, client k clips per-sample (or per-microbatch) gradients for the shared module and adds Gaussian noise:

where C is the clipping norm and σ is the noise multiplier. The perturbed gradients update θ(g) k , which is then uploaded Algorithm 1 Selective Decoupled Federated LoRA (One Round)

• 1: Input: Frozen backboneW; shared module θ(g,t) ; client set St; weights {pk}; rank budget rmax; DP parameters (C, σ)
• 2: Output: Updated shared module θ(g,t+1)
• 3: Server broadcasts θ(g,t) to clients k ∈ St.
• 4: for each client k ∈ St in parallel do
• 5: Initialize decoupled adapters with θ(g,t) and local state θ(l,t) k .
• 6: for local step e = 1, . . . , E do
• 7: Compute gradients on batch; update θ(g) k (with DPSGD if enabled) and θ(l) k .
• 8: end for
• 9: Upload θ(g,t) k to server; keep θ(l,t) k locally.
• 10: end for
• 11: Server aggregates shared modules via selective stacking:
• 12: θ(g) ← StackAgg({θ(g,t) k }k∈St , {pk}).
• 13: Re-compress to rank budget rmax to obtain θ(g,t+1) ; otherwise set θ(g,t+1) ← θ(g) .

to the server. The private module θ(l) k is updated without noise, preserving client-specific expressiveness and improving optimization stability.

Why selective privacy helps. Injecting noise into the entire adapter amplifies variance and can destabilize learning, especially for heterogeneous clients. By restricting DP to the shared module, our framework protects transferable knowledge while avoiding unnecessary corruption of client-specific adaptations, yielding a better balance of utility and privacy.

3.5 Algorithm Summary

Algorithm 1 summarizes one communication round of our method. Each client receives the shared module, performs local training with decoupled adapters, uploads only the sharedmodule parameters, and the server performs selective stacking aggregation to produce the next shared module.

4 Experiments

4.1 Experimental Setup and Baselines

Tasks and datasets. We evaluate federated parameterefficient fine-tuning on GLUE-style and MMLU benchmarks [Wang et al., 2018; Hendrycks et al., 2021]. We report results on QNLI, RTE, QQP, MNLI, and SST-2, which cover natural language inference, paraphrase identification, and sentiment classification. We use the official train/dev splits and report dev-set performance.

Backbone and adapter. We adopt LLaMA-7B [Touvron et al., 2023] as the frozen backbone and insert LoRA adapters [Hu et al., 2022] into linear projection layers. Only LoRA parameters are optimized while the backbone weights remain fixed.

Federated protocol and rank heterogeneity. We simulate a cross-device FL setting with K = 8 clients and run T = 30 communication rounds. Unless otherwise stated, all clients participate in each round and the server aggregates client updates with weights proportional to local data sizes, i.e., αk = nk/ ∑j nj. To model structural heterogeneity induced by rank mismatch, we set client ranks as {rk} = [4, 4, 8, 8, 8, 8, 16, 16]. When a homogeneous setting is required, we use a shared rank rk = r for all clients. In an additional scalability study, we vary the number of participating clients K while keeping the remaining hyperparameters consistent.

Local optimization. Each client uses Adam with a fixed learning rate of 2 × 10-4 and batch size 128. Within each communication round, clients perform a fixed number of local update steps (10) on mini-batches before uploading their LoRA updates. All remaining optimizer hyperparameters follow the default settings in our implementation. These hyperparameters are kept identical across methods unless the experiment explicitly controls a specific factor, ensuring fair comparisons.

Evaluation metrics. We report Accuracy on QNLI/RTE/MNLI/SST-2. And for QQP, we also report Accuracy to maintain a unified metric across tasks.

Baselines. We compare the proposed aggregation strategy against representative alternatives under identical training budgets: (i) Zero-padding aligns heterogeneous LoRA updates by padding lower-rank factors to the maximum rank before weighted averaging; (ii) FedAvg performs standard weighted averaging of LoRA parameters in the homogeneous-rank setting; (iii) Stacking(FLoRA) concatenates client low-rank updates along the rank dimension to preserve the union of client subspaces.

Reproducibility. All experiments are conducted on a machine equipped with an Intel Xeon Gold 5318Y CPU, 128GB RAM, and 2×NVIDIA RTX A6000 GPUs, running Ubuntu 22.04.5, Python 3.10, and CUDA 12.4. Key hyperparameters and system configurations are summarized in Table 1.

4.2 Results under Rank and Data Heterogeneity

Rank heterogeneous clients. Table 2 reports results under mismatched client ranks. Overall, Ours consistently outperforms zero-padding and FLoRA across all tasks. For example, it improves QNLI from 95.81% to 96.96% and QQP from 86.72% to 87.97%, and achieves notably larger gains on RTE and MNLI, which are more sensitive to cross-client directional mismatch.

These improvements stem from how different methods handle structural mismatch under rank heterogeneity. Zeropadding dilutes informative low-rank updates by enforcing a common high-rank parameterization, while FLoRA aggregates adapters in a monolithic manner. By contrast, Ours performs selective aggregation only on the shared component, leading to more stable and reliable updates under mismatched ranks.

Homogeneous rank sanity check. To verify that the benefit of our method is not merely due to resolving dimensional incompatibility, we further compare it with FedAvg-style averaging when all clients use the same rank (Table 2). Even in this homogeneous setting, stacking remains consistently superior or comparable.

For rank r = 4, Ours improves QQP by 3.20%, QNLI by 1.97%, and RTE by 2.95%, while maintaining comparable performance on SST-2. For rank r = 8, Ours again improves QQP (2.20%), QNLI (1.97%), and RTE (4.43%), with only a negligible difference on SST-2 (-0.02%). For rank r = 16, Ours yields further gains on QQP (3.37%), QNLI (1.65%), and RTE (4.70%), while slightly improving performance on SST-2 (0.20%). These results indicate that Ours acts as a subspace-preserving aggregator that remains beneficial even when the rank is fixed, rather than being a mere workaround for heterogeneous dimensions.

Effect of client population size. We study the impact of the federated population size by varying the number of participating clients K while keeping all other settings fixed. As shown in Figure 2, increasing K generally improves test accuracy, reflecting the benefit of broader data coverage in larger federations. With very few clients (K=4), aggregation is less effective due to limited diversity of update directions.

The performance trend is not strictly monotonic, and a temporary drop is observed at intermediate K values (e.g., around K=16), which is consistent with non-IID federated optimization. Comparing the two rank settings, lower ranks exhibit better robustness in the small-to-medium regime, whereas higher ranks achieve a higher accuracy ceiling as the client population grows. Overall, these results indicate that our method scales well with the federation size while exhibiting a non-trivial interaction between rank and population size.

Robustness to data heterogeneity. We further examine robustness under Non-IID data partitions controlled by a Dirichlet parameter α using the client-level accuracy standard deviation in Figure 3. Lower variance indicates more consistent performance across heterogeneous clients.

Under strong heterogeneity (α = 0.1), stacking achieves lower variance than padding on all tasks, e.g., on SST-2 (0.344 vs. 0.379) and QNLI (0.387 vs. 0.420), indicating improved stability when client data distributions are highly skewed. At moderate heterogeneity (α = 0.5), stacking again consistently reduces variance, for example lowering RTE from 0.372 to 0.343 and QNLI from 0.111 to 0.104, showing better robustness across clients.

At α = 1.0, stacking remains slightly more stable on RTE

(0.088 vs. 0.092), but exhibits higher variance on SST-2 and QNLI. This suggests that when data heterogeneity is moderate, preserving a broader set of client-specific low-rank directions may amplify inter-client discrepancies on certain tasks, even though the shared model remains competitive. Importantly, when the partition approaches near-IID (α = 10), stacking again achieves uniformly lower variance across all tasks, showing that stacking does not inherently harm stability.

4.3 Privacy and Rank Sensitivity

Privacy and utility trade-off. We apply DP-SGD during private training and report task accuracies under different privacy budgets ϵ (Figure 4). As ϵ increases (weaker privacy), the required noise scale decreases and performance improves.

Figure 4 further shows that enforcing structural constraints on the shared module significantly improves robustness under strong privacy constraints. When ϵ is small, unconstrained aggregation suffers from severe performance degradation, whereas the fixed shared-module design maintains substantially higher accuracy.

Rank sensitivity with and without DP. Table 3 analyzes how the DP noise affects performance. Without DP, smaller ranks can be more stable on some tasks, while overly large ranks may introduce redundancy and hurt aggregation. With DP (ϵ = 1), the optimal rank can shift due to noise, but smallto-medium ranks often remain a good trade-off.

4.4 Evidence for Decoupled Benefit

We further provide direct evidence that the dual-adapter (decoupled) design itself contributes to performance, independent of selective stacking and rank budgeting. As shown in Figure 2 (Right), under matched training budgets the dualadapter improves accuracy on every client, raising the average from 90.33% to 92.74% with an improvement of 2.41 percentage points. Per-client gains range from 0.69% on Client 4 to 3.51% on Client 3, indicating that the shared and private branches capture complementary low-rank subspaces rather than redundant capacity.

5 Conclusion

We studied federated fine-tuning of LLMs with LoRA under practical client heterogeneity, including differences in data distributions, system constraints, adapter ranks, and privacy requirements. While stacking-based aggregation provides a natural way to reconcile rank mismatch, we argue that stacking alone is insufficient, as it implicitly forces shared alignment on all client updates, suppressing personalization and becoming fragile under differential privacy noise.

We proposed the SDFLoRA framework, which structurally decomposes each client LoRA adapter into shared and private components. Building on this decomposition, we introduce selective stacking that applies stacking-based alignment only to the shared module while keeping private modules private and unaggregated. This configuration supports privacy-aware optimization by injecting DP noise exclusively into the shared module. Experiments on multiple benchmarks show that our approach outperforms representative federated LoRA baselines and achieves a better balance between utility and privacy under DP noise. In future work, we plan to extend this structural separation to other adapter families and investigate adaptive mechanisms for allocating shared versus private capacity based on client characteristics and task demands.

Acknowledgments

This work was supported by the Key R&D Program of Zhejiang Province under Grant No. 2025C01084 and the CAAIMindSpore Open Fund, developed on the OpenI Community, under Contract No. CAAIXSJLJJ 2025 MindSpore 04.