Federated Deconfounding and Debiasing Learning for Out-of-Distribution Generalization

1 Introduction

Federated out-of-distribution (OOD) generalization typically leverages data with diverse attributes across clients for collaborative modeling. The aim is to enhance model performance on unseen distributions [Qi and et al., 2024; Qi et al., 2025b; Wang et al., 2024; Fu et al., 2025b]. It allows FL clients to train models locally and iteratively exchange parameters with the server, enabling global aggregation without exposing private data [Liao et al., 2024; Fu et al., 2025a; Hu et al., 2023; Yi et al., 2023]. Despite its contributions to data privacy protection, existing studies show that attribute skew among sources often leads to performance degradation in FL models [Fu et al., 2025a; Zhang and et al., 2024; Zhang et al., 2025a; Ren et al., 2025]. This is primarily due to spurious correlations in local models, which hinder the formation of a robust global model during aggregation [Yang et al., 2020; Hu et al., 2024; Cai et al., 2024b].

∗Corresponding author

Existing approaches for enhancing FL model performance on OOD samples can be broadly divided into two groups: 1) knowledge distillation-based methods [Wei and Han, 2024; Fan et al., 2025; Yu et al., 2023; Huang et al., 2025] and 2) data augmentation-based methods [Chen et al., 2023; Shenaj et al., 2023; Park et al., 2024]. The former either treats the teacher's knowledge (e.g., the output of global or other local models) as a regularizer to provide additional supervision or decouples category-irrelevant features in the latent space, thereby mitigating the interference of irrelevant attributes. For instance, MCGDM [Wei and Han, 2024] shares classifiers across sources to match gradients both within and across domains to reduce overfitting to attribute features. DFL [Luo et al., 2022] applies mutual information-based disentanglement to address negative transfer due to attribute skew. However, the suboptimal performance of the teacher model on OOD samples and the single environment limit the effects of knowledge distillation and attribute decoupling. The latter approach typically trains generators or uses pre-trained generative models to create data with diverse attributes (e.g., CCST [Chen et al., 2023]). For example, FedCCRL [Wang and Tang, 2024] fuses different samples for data augmentation. However, they often introduce privacy risks with performance limited by the quality of the generated data.

To address these limitation, we propose the Federated Deconfounding and Debiasing Learning (FedDDL) method. As shown in Figure 1, it focuses on the key variables involved in the data and model inference (including data sources, background, images, objects and labels) with a structural causal model (SCM). To mitigate the joint interference of data source and background factors on model inference, we designs two main modules: 1) the intra-client deconfounding learning (DEC) module, and 2) the inter-client debiasing learning (DEB) module. Specifically, DEC decouples the background and the object in images, generating counterfactual samples to establish the association between the background and all classes without sharing any sample information among FL clients. This ensures that the model does not erroneously interpret the background as a causal factor. DEB constructs causal prototypes using object images rather than original images, which reduces the proportion of specific attributes in the prototype components. In addition, it treats causal prototypes as templates to align representations across different sources. This encourages local models to focus on target objects rather than the background, thereby promoting the learning of a unified representation space across clients.

Extensive experiments were conducted on two datasets, including performance comparisons, ablation studies and case studies with visual attention visualizations to investigate the association between background and labels. The results demonstrate that FedDDL effectively mitigates the interference of background and focuses on the objects of samples in unseen cases, improving the effectiveness of collaborative learning. Compared to nine state-of-the-art existing methods, FedDDL achieves 4.5% higher Top-1 Accuracy on average.

2 RelatedWork

2.1 Data Augmentation-basedMethods

To mitigate attribute bias, data augmentation-based methods [Wang and Tang, 2024; Shenaj et al., 2023; Liu et al., 2021; Xu et al., 2023; Zhang et al., 2024; de Luca and et al., 2022; Zhang et al., 2025b; Qi et al., 2022] have been developed to improve model generalization to previously unseen attributes by boosting the diversity of the data attributes. These methods typically rely on two approaches: 1) training data generators to create novel samples, or 2) utilizing pre-trained diffusion models to enrich sample diversity. The first approach involves exchanging local information between clients to generate new samples from different domains. For instance, FIST [Nguyen et al., 2024] and StableFDG [Park et al., 2024] produce samples with varying styles by sharing style-related information across clients. The second approach leverages prompt-driven techniques to generate samples that align with certain criteria [Morafah and et al., 2024; Zhao et al., 2023]. Although enhancing diversity, these strategies often introduce privacy concerns. Moreover, the quality differences between generated and original samples also limit their effectiveness.

2.2 Knowledge Distillation-basedMethods

Knowledge distillation-based approaches utilize generalized knowledge to guide local models in extracting shared features that are independent of specific attributes. These methods typically adopt two main strategies: 1) regularizing representations from different sources to align them, thus promoting consistency in representations across diverse contexts [Wang et al., 2023; Qi et al., 2024; Qi et al., 2025a; Qi et al., 2023; Liu et al., 2021; Sun et al., 2023; Meng et al., 2024; Zhu et al., 2024]; or 2) decoupling invariant features within the latent space to minimize the impact of confounding factors [Yu et al., 2023; Yi et al., 2024; Yu and et al., 2011; Cai et al., 2024a; Liu et al., 2024]. For instance, FedProc [Mu et al., 2023] and FPL [Huang et al., 2023] leverage prototypes to enforce consistent representation learning across all clients, thereby encouraging them to converge within a shared representation space. MCGDM [Wei and Han, 2024] mitigates overfitting within individual domains by employing both intra-domain and cross-domain gradient matching to improve generalization. Similarly, FedIIR [Guo et al., 2023] enables the model to implicitly learn invariant relationships by capitalizing on prediction inconsistencies and gradient alignment across clients. Although these techniques have achieved notable performance improvements, they often rely on singledomain data, which hinders model transferability to OOD domains. Our FedDDL method bridges these gaps.

3 Preliminaries

An FL system typically involves multiple distinct clients, denoted as S = {S1, S2, . . . , SK}, and a central coordinating server C. Each Sk utilizes its own private dataset Dk = {(Xk , Y k )} to train a local model Mk. The server C aggregates the local model parameters {θk} from all participating clients to compute the global parameters θg = ∑Kk=1 αkθk, where αk is the weight based on the size of each client's local dataset, and αk = |Dk |

Under this setting, FedDDL constructs a structural causal graph to analyze the confounding factors involved in FL model inference. As shown in Figure 2, the main idea is to tackle the interference of client data heterogeneity (S → X)

Figure 2: A causal view in federated out-of-distribution generalization, which uses backdoor intervention to eliminate the interference of background factor B and data source factor S on the sample X during model inference (i.e., B ↛ X and S ↛ X).

Clients

Server

and background (B → X) factors in model decision-making, where X is the input image. The intra-client deconfounding learning module to generate counterfactual data to improve data diversity, which can sever the link between background and specific class during inference (i.e., B ↛ X). The interclient debiasing module constructs class-wise causal prototypes Uc = u1c , . . . , uNc from object images, and uses them to align heterogeneous representations across clients. This mitigates the influence of client-specific model differences during aggregation (i.e., S ↛ X).

4 The Proposed FedDDLMethod

This section presents the proposed FedDDL method with reference to the schematic framework shown in Figure 3.

4.1 ACausal View on FLOODGeneralization

To clarify the factors that influence model inference in federated OOD generalization, we have constructed a structured causal graph that comprehensively illustrates the potential inference paths of the model, as shown in Figure 2. Specifically,

• X ← B → Y indicates that the image background B is a potential confounding factor, causing the model to rely on it when inferring Label Y for Sample X. The proposed intra-client deconfounding learning module is designed to sever the connection between x and B (i.e., X ↚ B).
• X ← S → Y implies that different clients contain diverse image features. Bridging this heterogeneity can eliminate interference between images and their labels. The proposed inter-client debiasing learning module is designed to
• align heterogeneous representations, which severs the association between S and input image X (i.e., X ↚ S).
• X → O → Y means that the design of all modules focuses on learning the true causal effects, which infers labels Y based on key objects O in an image X, while avoiding spurious correlations with confounding factors B and S. By severing both paths, the causal structure is clearly revealed during federated OOD generalization.

4.2 Intra-Client Deconfounding Learning (DEC)

To address shortcut learning caused by client-specific backgrounds, the DEC module aims to block the causal link from background to label during inference, encouraging the model to focus on true causal relationships. To achieve this, it performs backdoor intervention to adjust the sample distribution do(X). By applying the law of total probability, the inference involves both the direct causal impact of X on Y and the correlation confounded by B:

To ensure that each background factor contributes equally to label inference, the DEC module generates counterfactual samples to intervene in the distribution:

which removes the path from B to X in Figure 2, thereby facilitating the model to approximate the causal intervention P(Y | do(X)) instead of the spurious correlation P(Y | X). Specifically, the DEC module begins by decoupling images into objects and backgrounds. Then, it modifies the background factors associated with the objects. Inspired by the pre-trained model Grounding DINO [Liu and et al., 2025], the approach leverages text prompts T to detect targets:

where IO and IB are the object and background images. T is the label name. θ represents the bounding box coordinates (θ1, θ3 for the top-left, and θ2, θ4 for the bottom-right). ⊙ denotes the Hadamard product. 1(x,y)/∈[x1,x2]×[y1,y2] is an indicator function, which evaluates to 1 outside the bounding box and 0 inside it.

Subsequently, to generate counterfactual samples with causal features and diverse backgrounds for distribution intervention, the DEC module divides the backgrounds I iB belonging to class i into η groups, denoted as {gj i |j = 1, . . . , η}:

where Split(·) is the random grouping function. The counterfactual data IC can be created by fusing randomly grouped backgrounds gj i and an object IO:

where Trans(·) denotes a transformation function, which may include rotation, flipping and resizing, as shown in Figure 4. To guide the model to focus on the target objects in different cases, we optimize it using a joint classification loss:

where N is the total number of classes. Mk(·) is the model of client k. y(n) and yc(n) are the true labels of n-th index for the original and counterfactual samples. Mk(I)(n) and Mk(IC)(n) are corresponding predictions.

4.3 Inter-Client Debiasing Learning (DEB)

To address the issue of conflicts among clients leading to declines in the overall FL model performance, the DEB module aims to sever the path from FL clients to labels in order to arrive at consistent decisions for the same input. It performs backdoor adjustments to the representation learning of each client. The original multi-source inference is expressed as:

Algorithm 1 FEDDDL

• 1: Initialize the global model parameter θ0
• 2: for t = 1, . . . , T do 3: Sample subset K of clients with |K| = k 4: for each client k ∈ K in parallel do 5: Initialize local model parameter θtk = θt-1 6: Counterfactual sample generation DC,k based on objects IO,k and backgrounds IB,k images 7: for e = 1, . . . , E do 8: Sample batches of data ζ1, ζ2 from local data Dk and counterfactual data DC,k 9: if t = 1 then 10: gk = ∇LJ (ζ1, ζ2) 11: else 12: gk = ∇LJ (ζ1, ζ2) + λ∇LCR(ζ1, ζ2, UG) 13: end if 14: Update θtk ← θtk - ηlgk 15: end for 16: U i,k L,t ← 1 |I i,k O | ∑MG(θt-1 , I i,k O ), i = 1, . . . ,N 17: end for 18: θt = 1/k ∑k∈K θt-1 k 19: U iG,t = 1/k ∑k∈K U i,k L,t , i = 1, . . . ,N 20: end for

which includes both the direct causal effect of X on Y and the correlation confounded by B and S. The DEC module mitigates the interference from factor B by severing the connection between the data source S and the label Y :

Specifically, it first constructs causal prototypes using the extracted object images IO to reduce the proportion of background features in the prototype components as:

where U i,k L and I i,k O represent the local causal prototype of class i and the data set in client k, respectively. MG(·) denotes the global model. Moreover, the global causal prototype U i,k G of class i can be represented as:

As shown in Figure 5, backgrounds might cause different class prototypes to exhibit similar distributions across FL clients. In contrast, causal prototypes effectively eliminate the interference of backgrounds, thereby enhancing their distinction. Therefore, the DEB module performs causal prototype-based regularization to guide the adjustment of representation distributions in each client:

where f represents a local feature. U + G and U - G denote the global causal prototypes of the same and different classes as f, respectively. τ is the temperature parameter. This reduces the output gap between models from different clients, and eliminates interference from clients' local factors.

4.4 Training Strategy

FedDDL aims to eliminate the interference of client-specific attributes and decision differences on the final model inference. This is achieved through the collaboration of the DEC module and the DEB module. We summarize the pseudocode of FedDDL in Algorithm 1. Its overall optimization objective can be express as:

where λ is a weighted parameter.

5 Experimental Evaluation

5.1 Experiment Settings

Datasets Following previous work on OOD generalization [Qi and et al., 2024; Wang et al., 2022], experiments are conducted on two datasets: NICO-Animal and NICOVehicle [Wang et al., 2021]. Their statistics and partitioning method of the datasets can be found in Table 1.

Evaluation Metrics Following prior work [McMahan et al., 2017; Li et al., 2021], we use Top-1 Accuracy to evaluate performance, which is defined as 1N ∑Ni=1 I(ˆ yi = yi), where N is the total number of samples, ˆ yi is the predicted label, and yi is the ground-truth label.

Network Architecture To ensure fair comparison, all methods adopt the same network architecture. In FedDDL, a unified architecture is used to process both raw images and those processed with Grounding DINO. Following prior studies [?; Liu et al., 2021], ResNet-18 [He et al., 2016] is selected as the backbone for both datasets.

Implementation Details We set the local training epochs to 10 per global round for both datasets. The total number of communication rounds is 50, with 7 clients for both datasets. We used a client sampling fraction of 1.0 and employed SGD as the optimizer. During local training, the weight decay is set to 0.01, the batch size is 64, and the initial learning rate is 0.01 for both datasets. λ is chosen from the set {0.1, 1.0, 2.0}. τ is selected from {0.5, 0.07}. η is tuned from {1, 3, 5}. The BOX THRESHOLD and TEXT THRESHOLD in the DINO model have been set to 0.3. For other methods, hyperparameters are tuned based on the corresponding papers.

5.2 Performance Comparison

This section presents a comparison of the FedDDL method with nine state-of-the-art (SOTA) methods, including FedAvg [McMahan et al., 2017], FedProx [Li et al., 2020], MOON [Li et al., 2021], FPL [Huang et al., 2023], FedIIR [Guo et al., 2023], FedDecorr [Shi et al., 2024], FedHeal [Chen et al., 2024], MCGDM [Wei and Han, 2024], and FedCCRL [Wang and Tang, 2024]. The results derived from Table 2 are summarized below.

• FedDDL achieve significant improvements across all cases compared to existing methods. This includes evaluations of both global (Global) and local (Local) models, where the consistent enhancements highlight the robustness of FedDDL in coping with complex scenarios.
• The DEC module is a plug-and-play component that can be easily integrated into other methods, such as MOON, and can provide significant performance improvements. This is reasonable because it enhances sample diversity and ensures the preservation of causal features.
• By addressing the federated OOD problem from both intraclient and inter-client perspectives, FedDDL outperforms single-perspective methods like MOON and FPL. FedDDL leverages deconfounding learning to disrupt spurious correlations between backgrounds and labels within clients, while also employing debiasing learning to reduce the output gap between clients.
• FedDDL also reduces the performance disparity between different local models (as reflected by the local variance). This can be understood as it simultaneously strengthens the performance of individual local models via the DEC module while leveraging the DEB module to promote consistency in outputs across heterogeneous clients.

5.3 Ablation Study

This section provides an in-depth analysis of the effectiveness of various modules within the FedDDL, including the intra-client deconfounding learning (DEC) module, the inter-client decbiasing learning module with local data (DEBL), counterfactual data (DEBC), and their combined set (DEBL+C). The results are presented in Table 3.

• Incorporating the inter-client decbiasing learning module contributes to improving the performance of the global model and the average performance of the local models, but there is a significant variation in the performance across different client models.

• The inter-client decbiasing learning module plays a significant role in bridging the performance gap between client models, which promotes consistent improvements across heterogeneous sources and enhances the alignment of model representations, diminishing discrepancies in client-specific performance.
• As the diversity of the data increases, the advantages of the DEB module are amplified, as it effectively enhances heterogeneous representation alignment, particularly in complex cases, where its ability to adapt and align diverse representations proves to be a significant asset.

5.4 Robustness of FedDDL on Hyperparameters

This section assesses the robustness of FedDDL under different hyperparameters. Specifically, we explore the impact of hyperparameters λ, τ, and η by selecting values from the sets {0.1, 1.0, 2.0}, {0.07, 0.5}, and {1, 3, 5}, respectively. As depicted in Figure 6, FedDDL consistently outperforms FedAvg across various scenarios, showing remarkable insensitivity to hyperparameter variations within a broad range. This suggests that FedDDL is highly robust and adaptable to changes in hyperparameters. For the adjustment of τ, setting τ = 0.07 enhances model performance compared to τ = 0.5 by increasing the representation differences between sources. This adjustment allows the model to focus more on these variations, leading to improved generalization across clients. For η, increasing η means greater data diversity. The greater the diversity in the data, the more the model benefits, highlighting the importance of diverse data sources in improving the model's ability to generalize.

5.5 Case Study

Breaking the Background-Category Association

Here, we examine the effectiveness of the DEC module in breaking the association between background and specific labels. As shown in Figure 7(a), the unique attributes of the data lead to spurious associations in the FedAvg method between "dog" and "grass" as well as "cow" and "river". Even when the objects in the images are masked, the model still identifies the background with an exceptionally high confidence as the label of the object. This can easily cause the model to rely on non-causal associations to infer the labels of samples, ultimately leading to errors. Moreover, benefiting from the intervention of the DEC module, the association between background and labels is disrupted, leading the model to predict the background with equal probability across all dimensions. As shown in Figure 7(b), it narrows the gap among the probabilities of predicting the background as different categories, avoiding to use a very robust but causally wrong feature to make predictions.

Comparison of Visual Attention

This section analyzes the effectiveness of the deconfounding learning in local 1. We randomly selected test samples and visualized the visual attention of different methods using GradCAM [Selvaraju et al., 2017; ´Slazyk et al., 2022; Lin et al., 2020; Meng et al., 2019]. As illustrated in Figure 8, FedAvg often focuses on the background of out-ofdistribution samples, leading to classification errors due to its limited generalization ability. For example, in the case of a cat on grass, FedAvg may assign the label based on "grass," leading to incorrect identification as a "dog," since the training data only contains dogs on grass. Furthermore, FedDDL effectively focuses on the target object in test samples, enabling the model to eliminate background interference and make accurate predictions. This significantly enhances the generalization ability of local models and demonstrates that improving the performance of individual models also contributes to better collaboration.

Analysis of Cross-Silo Representation Alignment

This section analyzes the consistency of representation distributions from models of different sources on the same test samples. As shown in Figure 9, the image representations from clients 1, 2 and 3 are visualized using t-SNE [Van der Maaten and Hinton, 2008; Zhou et al., 2023]. Clearly, the DEB module has mitigated the gap in outputs from models of different sources. Meanwhile, it has also enhanced the discriminability of representations across classes. This is reasonable because causal prototype contrastive alignment helps alleviate overfitting to specific attributes and facilitates the construction of a unified knowledge base across silos. In contrast, FedAvg may learn inconsistent inter-class relationships between different sources (e.g., the orange and pink classes in clients 1 and 2). In client 3, the coverage of different class representations is high, and the classification boundaries are not clearly defined. These observations demonstrate the effectiveness of the DEB module.

6 Conclusions and FutureWork

In this paper, we address the problem of overfitting to specific backgrounds and the ill-posed aggregation issues caused by attribute skew in federated learning. We propose FedDDL, which comprehensively analyzes the factors interfering FL model inference. The key idea is to generate counterfactual samples to mitigate the interference of image backgrounds. Moreover, it promotes the consistency of outputs across models from different FL client through debiasing learning. Experimental results demonstrate that FedDDL significantly improves the performance of the global FL model in OOD cases.

In subsequent research, we plan to extend FedDDL to cases where attributes cannot be directly disentangled, and develop adaptive methods for causal relationship discovery.

Acknowledgments

This work is supported in part by the Excellent Youth Scholars Program of Shandong Province (Grant no. 2022HWYQ048); the Joint Funds of the National Natural Science Foundation of China (Grant NO. U2336211); the Ministry of Education, Singapore, under its Academic Research Fund Tier 1; the RIE2025 Industry Alignment Fund - Industry Collaboration Projects (IAF-ICP) (Award I2301E0026), administered by A*STAR, as well as supported by Alibaba Group and NTU Singapore through Alibaba-NTU Global e-Sustainability CorpLab (ANGEL); the China Scholarship Council.