ESBN: Estimation Shift of Batch Normalization for Source-free Universal Domain Adaptation

1 Introduction

Deep learning exhibits excellent performance when the test domain matches the training set; however, performance deteriorates when a domain shift occurs. Domain adaptation (DA) allows the model to maintain its performance in the target domain by mitigating the domain shift. DA has evolved through several paradigms, each addressing increasingly complex scenarios. Classical unsupervised domain adaptation (UDA) methods initially focused on situations with shared label spaces between domains, employing various techniques such as adversarial training [Ganin et al., 2016; Rangwani et al., 2022] and feature alignment [Tranheden et al., 2021; ?; Wang et al., 2024c] to bridge the domain gap. However, these approaches often face a fundamental limitation: the assumption of identical label spaces between source and target domains rarely holds in real-world scenarios [Liang et al., 2020; Qu et al., 2022; Qu et al., 2024a].

∗Corresponding author.

To address this limitation, researchers have proposed several specialized frameworks. Partial Domain Adaptation (PDA) [Cao et al., 2019] handles cases where target labels form a subset of source labels, while Open-set Domain Adaptation (OSDA) [Liu et al., 2019] deals with scenarios where the target domain contains unknown categories. Open-Partial Domain Adaptation (OPDA) [You et al., 2019; Saito and Saenko, 2021] combines both challenges, where the target domain contains unknown categories and only a partial set of labels from the source domain. However, these methods still require prior knowledge about the relationship between source and target label spaces in Universal Domain Adaptation (UniDA) [Li et al., 2024]. UniDA [You et al., 2019; Fu et al., 2020] represents a significant advancement by eliminating the need for such prior knowledge. It tackles the most general case where label spaces may overlap arbitrarily between domains, without assuming any information about the nature or extent of this overlap. This flexibility makes UniDA particularly relevant for real-world applications where label relationships between domains are unknown or uncertain. Recent developments have further considered practical constraints, particularly data privacy concerns [Wang et al., 2024d]. Source-free Universal Domain Adaptation (SFUniDA) [Liu and Zhou, 2024; Qu et al., 2024b; Schlachter and Yang, 2024] addresses scenarios where direct access to source domain data is restricted, working instead with a pretrained source model.

In the current study, some researchers have attempted to improve Batch Normalization (BN) for addressing domain shift, it is important to note that a model's feature extraction capability remains fundamentally unchanged across domains. The performance degradation primarily arises from the invalidation of previously calculated BN statistics when encountering domain shifts, which subsequently affects the quality of feature representation. This observation has prompted various approaches that focus on adjusting BN statistics to mitigate the effects of domain shift [Liu et al., 2021; Li et al., 2024]. For instance, methods like batch normalization statistics transfer explicitly align BN statistics between source and target domains [Fang et al., 2024]. Other researchers have addressed domain adaptation by updating only the affine parameters in the BN layer, while preserving the source domain statistics to reduce label-domain interference and enhance domain knowledge learning [Wu et al., 2024].

Although numerous approaches have been proposed to address unsupervised domain adaptation by adjusting BN, relatively fewer studies focus on source-free domain adaptation. Existing research predominantly concentrates on altering the behavior of BN, which is not directly applicable to sourcefree unsupervised domain adaptation. This is because, unlike unsupervised domain adaptation, source-free unsupervised domain adaptation lacks a source domain. Many unsupervised domain adaptation methods rely on comparing the BN statistics between the source and target domains for domain adaptation. However, in the absence of a source domain, it is challenging to directly compare the statistics between the source and target domains. Directly modifying BN is difficult to apply in source-free domain adaptation, and BN layers, which overly rely on batch statistics, are ineffective in reducing domain shift in source-free domain adaptation tasks. Additionally, we observe that the placement of BN layers significantly impacts the overall generalization ability of the model, as different positions correspond to different feature representations. Existing research primarily focuses on altering the behavior of BN, overlooking the role of BN layer placement and its impact on cross-domain generalization.

To address these issues and the limitations of source-free domain adaptation, we propose ESBN. By addressing the model's feature representation impact caused by BN statistics shift due to domain offset, we aim to reduce the influence of domain shift on BN layers. Specifically, we explore adjusting the position of the normalization layers and replacing them with a more robust, batch-independent normalization method to mitigate the impact of domain shift on BN layers, resulting in a more robust model. We introduce Batch-free Normalization (BFN), which, compared to batch-dependent statistics, can potentially provide more robust feature representations through instance-specific statistics. We systematically analyze the effectiveness of replacing BN with BFN at various network positions (such as odd layers, even layers, and all layers) and configurations (such as alternating modes). The goal of this strategy is to achieve an optimal balance between maintaining discriminative power and reducing domain-specific biases during feature learning. Furthermore, we investigate the significant impact of the placement of normalization layers on cross-domain feature learning and identify a more suitable placement for these layers. Overall, our approach demonstrates superior performance compared to the state-of-the-art across multiple experiments. The main contributions can be summarized as follows:

• We present the first comprehensive study on the impact of normalization layer placement in domain adaptation, revealing how different configurations affect feature transferability and domain generalization.
• We propose an approach for integrating Batch-free Normalization into existing architectures, demonstrating that selective replacement of BN layers can significantly improve generalization performance.
• We conduct extensive experiments across multiple domain adaptation benchmarks, providing empirical evidence that our strategic normalization approach outperforms traditional DA methods, including in challenging OPDA scenarios.

2 RelatedWork

Unsupervised Domain Adaptation. Unsupervised Domain Adaptation (UDA) focuses on aligning source and target domains by learning domain-invariant feature representations [Peng et al., 2019; Zellinger et al., 2017; Wang et al., 2024a; Wang et al., 2024b]. This challenge is tackled by utilizing labeled data from source domains to train models for unlabeled target domains in a transductive learning framework. Existing UDA methods can be categorized into two primary categories: (1) Moment matching approaches, such as Maximum Mean Discrepancy (MMD) and the Wasserstein metric, focus on aligning statistical properties between source and target distributions [Tranheden et al., 2021; ?]. (2) Adversarial learning approaches employ domain discriminators to promote domain-invariant feature extraction [Saito et al., 2018; Sankaranarayanan et al., 2018; Zhang et al., 2019; Xiao et al., 2021; Rangwani et al., 2022]. However, the majority of existing methods operate under the assumption that the label spaces of the source and target domains are identical. And despite the extensive theoretical advancements in Unsupervised Domain Adaptation (UDA), the acquisition of substantial source data remains impractical. This assumption restricts their applicability in real-world scenarios.

Source-Free domain adaptation. Source-Free Domain Adaptation (SFDA) is a domain adaptation approach aimed at leveraging pre-trained source models to facilitate knowledge transfer to the target domain in the absence of source data [Tzeng et al., 2017; Hoffman et al., 2018]. Existing SFDA methods can be categorized into two classes. Data generation techniques that synthesize source data [Hou and Zheng, 2020; Qiu et al., 2021; Du et al., 2024] and model fine-tuning approaches that utilize unlabeled target data within a self-supervised framework [Liang et al., 2020; Yang et al., 2021b; Yang et al., 2021a]. However, many current SFDA methods are limited to closed-set scenarios, which constrains their practical applicability. Although some studies have explored open-partial-set scenarios, they remain insufficient [Liang et al., 2021].

3 Methodology

3.1 Preliminary

Problem Definition. We address universal domain adaptation (UniDA), which tackles both covariate and label distribution shifts between domains. Given a source domain S = {(vs i , zsi )}Ms i=1 with vs i ∈ As ⊂ Rd and zsi ∈ Z s ⊂ RK , and an unlabeled target domain T = {(vt i )}Mt i=1 with vt i ∈ At ⊂ Rd . Let Z = Z s ∩ Z t denote shared labels, while ˆ Z s = Z s \ Z and ˆ Z t = Z t \ Z represent domainprivate labels. UniDA assumes no prior knowledge of Z t , Z, or ˆ Z t , aiming to identify shared-label samples while rejecting private-label instances. In source-free setting, S is only available during pre-training. Given a source model psϕ = qsϕ ◦ rsϕ , where rsϕ : Rd → Rm extracts features and qsϕ : Rm → RK classifies them, adaptation involves learning a target encoder rtϕ while maintaining qtϕ = qsϕ . Figure 1 shows ESBN.

Batch normalization. For convolutional neural networks (CNNs), the batch normalization (BN) layer processes fourdimensional tensors. Let Xn,k,h,w and Yn,k,h,w denote the input and output tensors respectively, where n indexes samples within a batch, k represents feature channels, and (h, w) indicates spatial coordinates. In the case of input images, k typically corresponds to the RGB color channels. The normalization is performed independently within each channel:

where mk = 1 |M| ∑n,h,w Xn,k,h,w represents the mean activation in channel k, computed across all samples n in the mini-batch and all spatial locations (h, w), whereMdenotes the set of all activations used in the mean computation. The variance vk is calculated analogously. A small constant δ is added for numerical stability. During inference, running averages of means and variances accumulated during training are utilized. Following normalization, a channel-wise affine transformation is applied using learnable parameters αk and θk, which are optimized during the training process. The transformation effectively standardizes the distribution of activations within each channel while maintaining the network's representational capacity through the learnable parameters. This standardization helps mitigate internal covariate shift and enables more stable training of deep neural networks.

Batch-free Normalization. Batch-free normalization (BFN) techniques are designed to eliminate the dependency on batch statistics, ensuring consistent behavior during both training and inference phases. Layer normalization (LN) [Lei Ba et al., 2016] operates by computing statistics across all feature channels and spatial dimensions for each individual sample independently, which can be formulated as:

where µn = 1 CHW ∑k,h,w Xn,k,h,w computes the mean across channels (C) and spatial dimensions (H,W) for each sample n. The variance σ2n is computed similarly. Unlike BN, the learnable parameters α and θ are shared across all channels. Group normalization (GN) [Wu and He, 2018] extends this concept by partitioning channels into G groups and performing normalization within each group separately:

where Gg represents the set of channels in group g, and µn, g and σ2n,g are computed within each group independently. This design offers a flexible trade-off between LN (G = 1) and instance normalization (G = C), making it particularly effective for visual tasks where small batch training is prevalent.

3.2 Estimation Shift of Batch Normalization

Consider a training dataset D and its mini-batches {Di }Ni=1 . For a neural network with BN layer Gϕ,ω(D) = Gpost ϕ (BN(Gpre ω (D))), let Z = Gpre ω (D) and ˆ Z = BN(Z). While the population statistics of D are well-defined through mini-batch statistics, those of activations Z = Gpre ω (D) are challenging to define due to parameter updates during training. Specifically, mini-batch activations Z i = Gpre ωi (Di ) depend on both input Di and evolving model parameters {ωi }Ni=1 . Despite this complexity, we observe that ˆ Z i maintains standardized distribution across iterations. We define the expected population statistics: for a trained model G¯ ϕ,¯ω(·) on training data D, the expected population statistics ¯µ, ¯σ2 of BN are defined as the mean and variance of activations Z = Gpre ¯ω (D′ ) on test data D′ [Huang et al., 2022].

According to the expected population statistics of BN, we define the Estimation Shift of BN as the discrepancy between the estimated population statistics and the expected ones. Understanding the impact of the Estimation Shift on the performance of the batch-normalized network is crucial. Therefore, we aim to reduce the difference between the estimated and expected statistics, thus improving the generalization performance of the network model.

3.3 BFN in Domain Adaptation

To estimate the distribution of feature magnitudes ∥vi,n∥2 , ESBN uses a two-component Gaussian Mixture Model (GMM) [Qu et al., 2024a]. The components with lower and higher means, denoted as θs and θn respectively where θs < θn, correspond to shared and novel classes. To achieve accurate class separation, we construct class-specific reference vectors through top-M sampling. For each shared class, we maintain two types of references: target references {rtk ∈ RD |k=1,...,K} and source references {rsk ∈ RD |k=1,...,K}. The sampling size M is determined by M = Nt/ ˆKt, where Nt represents target samples and ˆKt denotes estimated target classes. The source references are derived from classifier weights to handle potential source-novel scenarios.

Before feeding the features into the GMM for class separation, we apply BFN to the feature vectors vi to ensure their mean and variance are standardized within each instance. Unlike BN, BFN does not require batch-wise statistics, and instead, it normalizes across the channels of each individual sample. This method makes it particularly useful when handling domain adaptation tasks, as it alleviates issues related to batch size and domain-specific statistics. The BFN for a feature vector vi ∈ RC (with C channels) can be expressed as:

where µi = 1C ∑ c = 1C vi, c is the mean of the feature vector vi, σi = √ 1C ∑ c = 1C (vi, c -µi)2 is the standard deviation, and γ and β are learnable scaling and shifting parameters. This normalization ensures that the feature vector for each instance has zero mean and unit variance, making it suitable for domain adaptation tasks where features from both source and target domains are expected to have varying distributions. Based on these references, we define a confidence metric αi,k that evaluates instance-to-class relationships:

in addition to the normalization of features, BFN ensures that the feature distributions are stable within each instance, which is particularly helpful in handling small batch sizes or domain shift. The confidence scores αt i,k and αs i,k are clipped to [0, 1] before computing their geometric mean. The asymmetric formulation ensures that proximity to source references carries more weight in shared class determination than proximity to target references. The class assignment combines distribution statistics with instance-specific confidence:

Here, vt i,n for novel data, θk represents the expected magnitude for class k obtained through the same top-M sampling procedure, and βi,k defines an instance-specific decision boundary that accounts for both class-level distribution and instance-level confidence. 1 1denotes the one-hot encoding operator.

3.4 Optimization Objectives

The training process incorporates three complementary objectives to achieve effective domain adaptation: pseudo-label learning, feature magnitude regularization, and distribution alignment. Firstly, based on the predicted labels from the adaptive thresholding mechanism, we employ a weighted cross-entropy loss for model optimization. To account for prediction reliability, we introduce a confidence weight using a Student's t-distribution that considers the distance between feature magnitude |vt i,n |2 and decision boundary βi, k:

where wt i represents the sample confidence, pk(xti ) denotes the predicted probability for class k, and η is set to 1e-4. For samples classified as novel, we use a uniform distribution over shared classes instead of introducing an additional category. Secondly, to enhance the discriminative power of feature magnitudes for novel class detection, we introduce a regularization term:

where zt i is a binary indicator (1 for novel, 0 for shared classes), vt i,s for shared data. Finally, to ensure stable feature learning, we incorporate a distribution alignment loss Lalign that encourages consistency between neighboring samples in feature space. This helps maintain consistent predictions for semantically similar instances across domains. The final optimization objective combines these components:

where γ is a positive balancing coefficient. This combined objective enables effective separation of shared and novel classes while maintaining discriminative feature learning.

4 Experiments

4.1 Experimental Setup

Datasets. Office-31 [Saenko et al., 2010] is a widely used DA benchmark comprising three domains: Amazon (A), Dslr (D), and Webcam (W), with 31 office object classes. OfficeHome [Venkateswara et al., 2017] includes four domains: Artistic (Ar), Clip Art (Cl), Product (Pr), and Real-world (Re), covering 65 object classes. VisDA-C [Peng et al., 2017] is a large-scale DA dataset with 12 classes across synthetic and real domains, focusing on synthetic-to-real transfer. Class splits for each setting are summarized in Table 6. Implementation Details. All experiments are implemented on a single RTX-3090 GPU. Existing baselines maintain their original settings, with the feature encoder fine-tuned from the ImageNet pre-trained ResNet-50 backbone. The model is trained using SGD with a momentum of 0.9 and weight decay of 1×10-3 . The learning rate is set to 1×10-3 for the Office31 and Office-Home datasets, and 1 × 10-4 for the VisDA-C datasets. The batch size is 64. Apart from the parameters introduced for our improvements, all other settings remain consistent with those in [Qu et al., 2024a]. The source codes are available at https://github.com/leuluelueluelue/ESBN.

Evaluation. We adopt the same evaluation metrics as prior works for a fair comparison. In PDA scenarios, we report the classification accuracy as the primary evaluation metric. For OSDA and OPDA scenarios, we utilize the H-score [Liang et al., 2021] to evaluate performance. The H-score is defined as the harmonic mean of accuracy over common data and accuracy over private data.

4.2 Comparative Results

Results for OPDA Scenario. Experiments were initially conducted under the most challenging OPDA scenario to evaluate the effectiveness of the proposed method. Table 1 shows the results on the Office-Home dataset, while Table 2 summarizes the outcomes on the Office-31 and VisDA datasets. These results evidence that ESBN outperforms these existing methods in effectively segregating common and private data, highlighting its superior capability in handling DA tasks. Specifically, on the Office-Home, Office-31, and VisDA datasets, ESBN surpasses the state-of-the-art baseline by 1%, 0.9%, and 1.6%, respectively. We attribute this improvement to the effective implementation of the proposed method, which effectively mitigates the limitations associated with traditional BN. In particular, we replace the BN layers in even/odd layers within the residual blocks of ResNet with BFN.

Results for OSDA Scenario. In the OSDA scenario only the target domain involves private data, necessitating a method that can effectively differentiate between common and private data. The H-score results presented in Table 3 and Table 5 illustrate the performance of various methods under this scenario on the Office-Home, Office-31, VisDA datasets. Notably, ESBN demonstrates superior performance on the Office-Home, Office-31, and VisDA datasets, achieving improvements of 1.2%, 1.1%, and 1.4% over the LEAD method, respectively. It also maintains a leading position among all baselines, similar to the observations in the OPDA scenarios. Results for PDA Scenario. Finally, we validate the effectiveness of ESBN in the PDA scenario. The results shown in Table 4 and Table 5 indicate that ESBN achieves excellent performance even when compared to methods specifically tailored for PDA. In particular, ESBN attains overall accuracies of 72.7%, 95.5%, and 76.2% on the Office-Home, Office-31 and VisDA datasets, respectively. It is noteworthy that the accuracy on the Office-Home dataset is slightly lacking, which may be attributed to gradient propagation issues caused by modifications to the BN layer, leading to compromises during optimization. We will seek better modification and integration strategies in our future work.

4.3 Experimental Analysis

Ablation Study. To study the impact of normalization layer modifications in ESBN , we perform ablations on three datasets in the OSDA scenario. Results are shown in Tables 7 and 8, with Table 7 highlighting the effect of different normalization strategies. The results show that substituting BN layers with BFN in the even layers within even residual locks (EE) yields the best performance. Specifically, ESBN achieves 66.6%, 91.3%, and 73.6% on the OfficeHome, Office-31, and VisDA datasets, respectively. We attribute this improvement to BFN, which enhances the stability of training and enables the model to better adapt to the data distributions across different domains. In contrast, the other strategies do not lead to significant performance improvements, particularly in the context of more complex DA tasks. Table 8 further corroborates that GN is the most effective BFN strategy for improving DA performance.

Parameter sensitivity. Figure 2 (a-b) shows how varying η affects model performance. On both Office-Home and Office31 datasets, η = 32 consistently achieves the highest H-score, indicating improved model adaptability.

Robustness. We primarily conduct the robustness analysis of ESBN from the perspective of varying unknown classes. As the number of "unknown" classes increases, it becomes increasingly challenging to accurately distinguish between "unknown" and "known" objects. However, as shown in Figure 2 (c-d), our model exhibits excellent stability.

Visualization analysis. Based on the analysis of the confusion matrices in Figure 3, it is evident that ESBN significantly outperforms the LEAD in both OSDA and OPDA scenarios. ESBN achieves a higher proportion of correct classifications, reflecting its high accuracy in distinguishing between known and unknown categories. Moreover, the relatively low occurrence of misclassifications demonstrates the model's relatively strong category discrimination capability. Additionally, ESBN exhibits excellent stability, maintaining consistent performance across both OSDA and OPDA scenarios.

5 Conclusion

This paper presents an investigation into the strategic use of BFN for domain adaptation. We address a key gap in existing research by exploring how the placement and depth of normalization layers affect cross-domain generalization. Through extensive experiments, we show that selectively replacing BN with BFN at specific network positions significantly enhances generalization performance while preserving feature discriminability. Our findings highlight the critical role of normalization layer placement, with BFN providing more robust feature representations by computing instance-specific statistics, particularly in the presence of domain shifts. ESBN outperforms traditional domain adaptation techniques across multiple benchmarks, demonstrating its practical applicability in real-world settings. This work offers promising directions for improving domain adaptation in complex and real-world environments.

Acknowledgements

This work is supported by the National Natural Science Foundation of China under Grants No. 62406100, Tianjin Natural Science Foundation under Grants No. 24JCQNJC00320, Beijing Postdoctoral Research Foundation. This research is also supported by funding from the Hong Kong RGC General Research Fund (No. 152211/23E, 15216424/24E, and 152115/25E), the PolyU Internal Fund (No. P0056171), and the Huawei Gifted Fund.

Contribution Statement

Jiao Li and Houcheng Su contributed equally to this work.