Indirect Alignment and Relationship Preservation for Domain Generalization

1 Introduction

In recent years, deep learning has achieved remarkable success in fields like image recognition [Touvron et al., 2022], natural language processing [Zhang et al., 2023a], and object detection [Zou et al., 2023]. However, these advancements typically rely on the assumption that training and testing data share the same distribution (IID assumption), which often does not hold in real-world applications. Factors such as varying devices and environments can cause data distribution shifts, leading to significant performance drops when models are tested on new data-a challenge known as domain shift. This issue severely limits the cross-domain generalization of deep learning models, particularly in fields like medical imaging and autonomous driving, where data acquisition is complex.

Domain Adaptation (DA) methods [He et al., 2023] address domain shift by leveraging source domain data and a small amount of target domain data. However, these methods are limited when target domain data is unavailable. Domain Generalization (DG) trains models on multiple source domains to overcome this, enabling generalization to unseen target domains without relying on target domain data. Recently, contrastive learning-based DG methods [Tong et al., 2023; Hu et al., 2024; Chen et al., 2023b] have gained attention for their ability to align features of same-class samples while distinctly separating those of different classes, enabling finegrained feature representation.

∗Corresponding author.

However, traditional methods often focus narrowly on directly aligning same-class samples across source domains, overlooking the natural sequence relationships between data samples. This can lead to overfitting, as models may overemphasize local source-domain features while neglecting subtle yet crucial global patterns essential for generalization. For example, as shown in Figure 1, the similarity between a sample and another same-class sample within the same domain (e.g., two images of dogs from the "Photo" domain) should naturally be higher than the similarity between a sample and a same-class sample from a different domain (e.g., an image of a dog from the "Photo" domain and one from the "Cartoon" domain). Similarly, the similarity between a sample and a same-class sample from a different domain should naturally be higher than that between a sample and a differentclass sample within the same domain (e.g., an image of a dog and an image of an elephant from the "Photo" domain). Finally, the similarity between a sample and a different-class sample within the same domain should naturally be higher than that between a sample and a different-class sample from different domains (e.g., an image of a dog from the "Photo" domain and an image of an elephant from the "Cartoon" domain). These sequential relationships provide critical cues for generalization that humans intuitively recognize by observing fine-grained micro-details. Ignoring these relationships often results in models that fail to capture the rich semantic nuances needed for robust cross-domain performance.

To address these challenges, we propose an innovative domain generalization framework that integrates the Sample Difference Keeping (SDK) module and the Sample Consistency Alignment (SCA) module. The main contributions of our work are as follows:

1. We introduce the SDK module, which preserves the natural sequence relationships between samples across different domains and classes. This facilitates the learning of diverse and discriminative features while reducing the risk of overfitting to domain-specific patterns.
2. We propose the SCA module, which employs a novel indirect alignment strategy based on structural relationships. This strategy ensures inter-class consistency by maintaining class separability across domains and interdomain consistency by capturing shared patterns while preserving domain-specific characteristics. The SCA module enhances the model's generalization ability by avoiding the misalignment and feature distortion issues that are often present in direct alignment methods.
3. We conducted comprehensive evaluations on widely used domain generalization benchmarks. The results demonstrate that our framework significantly outperforms state-of-the-art methods, achieving superior cross-domain generalization and robustness against unseen domains.

2 RelatedWork

Domain generalization (DG) tackles distribution shifts without relying on target domain data, leveraging methods like data augmentation [Su et al., 2023; Wang et al., 2024], representation learning [Yu et al., 2024], and advanced strategies such as meta-learning and ensemble learning [Chen et al., 2023a] to enhance cross-domain generalization.

Recently, contrastive learning, a self-supervised approach, has gained attention for its ability to learn robust representations by constructing positive and negative sample pairs. Both unsupervised and supervised contrastive methods [He et al., 2020; Khosla et al., 2020] have demonstrated strong performance, attributed to their effective strategies for pair construction and loss design.

In DG, contrastive learning has gained traction for its simplicity and effectiveness:

• Direct Utilization and Integration: Studies [Motiian et al., 2017; Kim et al., 2021; Zhang et al., 2023b] explore
• integrating contrastive learning with additional strategies to address DG challenges.
• Incorporating Global Information: Methods [Yao et al., 2022; Dong et al., 2022; Li et al., 2023] introduce a global perspective to enhance feature alignment.
• Leveraging Sample relationships: Refining alignment by emphasizing hard positive samples [Hu et al., 2024] or dataset repartitioning [Tong et al., 2023].

Despite these advancements, traditional DG methods often rely on direct alignment strategies, which oversimplify complex relationships and neglect subtle features, leading to misalignment and feature distortion under significant domain gaps. Addressing these shortcomings requires methods that account for nuanced relationships while maintaining generalization.

While ranking-based methods in metric learning, such as deep metric learning [Cakir et al., 2019], Ranked List Loss [Wang et al., 2019], and Self-Supervised Synthesis Ranking (SSR) [Fu et al., 2021], effectively preserve relative ranking information and local structures, they do not specifically address domain shifts in DG tasks. Instead, these approaches primarily focus on optimizing intra-domain ranking performance and lack mechanisms for mitigating inter-domain misalignment caused by distribution shifts.

To address these challenges in DG, we propose the Sample Difference Keeping (SDK) and Sample Consistency Alignment (SCA) modules. SDK preserves natural sequence relationships to enhance feature diversity and separability across domains, while SCA employs indirect alignment to refine cross-domain consistency without the pitfalls of direct alignment, ensuring robustness and improved generalization in domain generalization tasks.

3 ProposedMethod

3.1 Preliminary Introduction

In the classical domain generalization setting, we assume the model has access to multiple source domains during training. These source domains form the training set D = {D1,D2, . . . ,DM}, where M is the number of source domains. Let X ∈ Rd denote the input feature space and Y ∈ RC denote the class label space. Each source domain Dk contains a series of sample pairs {(xki , yk i )}Ni=1 , where N represents the number of samples in this domain, xki ∈ X is the input feature, and yk i ∈ Y is the class label. The goal is to train a model Fϕ,θ, which consists of a feature extractor fϕ:X → Z that extracts a universal representation Z from input data X, and a predictor gθ:Z → RC that generates predictions for C classes based on the features Z, such that the model can accurately classify data from an unknown target domain Dt without relying on the target domain, which shares the same set of class labels Y as the source domains Dk .

Our proposed domain generalization method is designed to address the limitations of direct alignment strategies in traditional approaches. These methods often disrupt the natural sequence relationships between samples, leading to overfitting, misalignment, and feature distortion, which result in reduced performance, especially when domain discrepancies are significant. Figure 2 provides an overview of our proposed method, with specific details described in the following subsections.

3.2 Sample Diversity KeepingModule (SDK)

In domain generalization tasks, samples exhibit inherent sequence relationships based on their similarities, as shown in Figure 1. Preserving these natural sequence relationships is crucial for enhancing model generalization. However, traditional methods often neglect these relationships, leading to overfitting.

To address this, we propose the Sample Diversity Keeping (SDK) module, which explicitly constrains the relative distances between samples across domains and classes. This ensures the preservation of natural sequence relationships, enabling us to observe subtle details that aid in generalizing to new scenarios, reducing overfitting, and improving performance on unseen target domains.

To achieve this, we first project the sample features zi and class weights wi into feature embeddings qi ∈ Rpro and proxy sets Pi ⊆ Rpro , respectively. We use two different multilayer perceptrons (MLPs) as projection heads: hz for sample features and hw for class weights:

where Pi = {p1i , p2i , ...pMi } represents the set of all M proxies for class i across different source domains.

SDK maintains the natural sequence relationships between samples by constraining the relative distances between samples from different classes and domains. Specifically, to reduce computational complexity, we introduce proxy embeddings, which replace sample embeddings. We formalize the natural sequence relationships constraint as follows:

where pji denotes the embedding of a proxy from class i and domain j, p¯ji is the embedding of a proxy from class i but a different domain, pj¯i is the embedding of a proxy from domain j but a different class, p¯j¯i represents a proxy from a different class and a different domain, and qj i denotes the embedding of a sample from class i and domain j.

In practice, we replace the distance metric with cosine similarity s(·, ·), leading to the following modified constraint:

To enforce this constraint, we define the Sample Diversity Keeping (SDK) loss function using a triplet loss approach. For each triplet, we minimize the cosine similarity between the anchor and positive samples while maximizing the similarity between the anchor and negative samples, with a margin term to separate them:

where LT denotes the triplet loss function, LT(AP,AN,m) = max(AP - AN + m, 0) and AP represents the cosine similarity between the anchor and positive samples, AN represents the similarity between the anchor and negative samples, and m is a margin hyperparameter.

Because the number of triplets in metric learning can be large, resulting in heavy computational costs, we introduce an efficient max-min sample mining strategy [Schroff et al., 2015] to speed up training. The modified SDK loss function, using this strategy, is given by:

By selecting the smallest and largest similarity values for each triplet, this max-min strategy significantly reduces the number of triplets, accelerating training while maintaining good generalization performance. Through the SDK module, the model learns more generalizable feature representations, which aids in generalizing to unseen target domains.

3.3 Sample Consistency Alignment Module (SCA)

Traditional domain generalization methods rely on direct alignment strategies to find common features across domains. However, these methods oversimplify relationships between samples, neglect subtle features, and frequently lead to misalignment and feature distortion, especially when domain differences are significant. These shortcomings result in performance degradation and underscore the need for an indirect alignment approach that effectively captures both domainspecific and shared features.

To address this, we analyze inter-class and inter-domain feature relationships. For example, as shown in Figure 3, within any domain, classes like dogs and horses are more similar to each other than to unrelated classes like guitars. This stable inter-class relationships facilitates extracting shared features. From an inter-domain perspective, domains such as cartoons and photos exhibit high feature similarity, while sketches, which retain less semantic content, show lower similarity. These inter-domain relationships remain consistent across classes, with photos containing the richest semantic information, followed by cartoons, and sketches exhibiting the least.

Building on these insights, we propose the Sample Consistency Alignment (SCA) module, which employs an indirect alignment strategy to leverage inter-class and interdomain relationships. SCA consists of two key components: inter-class relationships consistency, which preserves similarity relationships between classes across domains, and interdomain relationships consistency, which maintains similarity relationships between domains across classes. This dual consistency ensures robust cross-domain feature extraction and enhances generalization to unseen target domains.

To improve efficiency, SCA incorporates a proxy mechanism, where proxies represent samples from different classes and domains, reducing computational costs. This mechanism enables efficient consistency alignment, effectively capturing cross-domain shared features and strengthening the model To improve efficiency, we incorporate a proxy mechanism in SCA, where proxies represent samples from different classes and domains, reducing computational costs. This mechanism enables efficient consistency alignment, effectively capturing cross-domain shared features and strengthening the model's generalization performance.

Inter-class relationships consistency. Inter-class relationships consistency aims to ensure that the similarity relationships between classes remain consistent in the feature space across different domains, thereby facilitating the extraction of common features. To achieve this, we calculate the similarity matrix for the classes in each domain and minimize the Frobenius norm of the differences between these matrices.

Specifically, for domain Di (where i = 1, 2, ...,M) and class c (where c = 1, 2, ...,C), we define the proxy embedding for class c in domain Di as pic . We then calculate the inter-class similarity matrix Sdom i ∈ RC×C for domain Di, where each element Slc1,c2 represents the cosine similarity between the proxy embeddings of class c1 and class c2 in domain l :

To ensure that the similarity matrices across the M domains are as similar as possible, we minimize the Frobenius mean difference between them, resulting in the inter-class relationships consistency loss LSCA-C:

where ∥ Sdom i - Sdom j ∥F represents the Frobenius norm between matrices Sdom i and Sdom j :

Inter-domain relationships consistency. Similarly, the goal is to ensure that the similarity relationships between domains remain consistent across different classes. This helps the model capture cross-domain common features and improves its adaptability to domain shifts. Analogous to interclass relationships consistency, we define an inter-domain similarity matrix Scls c ∈ RM×M for each class c, where each element Sl1,l2 o represents the similarity between domains l1 and l2 for class o:

We ensure inter-domain relationships consistency by minimizing the differences between domain similarity matrices for different classes, resulting in the inter-domain relationships consistency loss LSCA-D:

Finally, the overall Sample Consistency Alignment Loss consists of the inter-class relationships consistency loss and the inter-domain relationships consistency loss, weighted by a hyperparameter α:

By incorporating the proxy mechanism, SCA reduces computational costs while achieving efficient consistency alignment. This dual consistency Alignment helps the model capture cross-domain common features, thereby improving its generalization to unseen target domains.

The overall objective function is defined as:

4 Experimental Study

4.1 Datasets

To thoroughly evaluate the proposed method, we selected four challenging domain generalization datasets:

• PACS [Li et al., 2017]: Contains 9,991 images, divided into 7 categories and 4 domains (Photos, Art Paintings, Cartoons, Sketches), each with its unique image style.
• Office-Home [Venkateswara et al., 2017]: Contains approximately 15,500 images, divided into 65 categories and 4 domains (Art, Clipart, Product, Real World), and serves as a popular benchmark for evaluating domain generalization performance.
• VLCS [Fang et al., 2013]: Contains 10,729 images, divided into 5 categories and 4 domains (VOC2007, LabelMe, Caltech101, SUN09), used for testing the model's generalization ability across different data sources.
• TerraIncognita [Beery et al., 2018]: Contains 24,788 wildlife images, divided into 10 categories and 4 domains (L100, L38, L43, L46), posing a challenging task for domain generalization.

These datasets cover a wide range of image styles and domains, providing rich experimental scenarios for validating our approach.

4.2 Implementation Details

Our method builds upon SWAD [Cha et al., 2021] and PCL [Yao et al., 2022]. We use the PyTorch framework with a pre-trained ResNet-50 [He et al., 2016] on ImageNet [Russakovsky et al., 2015] as the backbone network. To ensure consistency with PCL, we employ the same projection layers. The Adam optimizer is applied with a learning rate of 5e-5 . Following the standard training and evaluation procedure of SWAD, we use a batch size of 32 for each domain. A leave-one-domain strategy is adopted, where one domain serves as the target domain, and the remaining domains are used as source domains for training.

For model validation and selection, 20% of the samples from each source domain are set aside as the validation set. To ensure result stability, we run experiments with three different random seeds and report the average accuracy and standard error. For the number of iterations, the Office-Home dataset is trained for 3,000 iterations, while the other datasets are trained for 5,000 iterations. The hyperparameters are set as follows: m1 = 0.1, m2 = 0.4, m3 = 0.1, and α = 0.01.

4.3 Comparison with State-of-the-Art Methods

We conducted a comprehensive comparison between the proposed method and state-of-the-art domain generalization methods on four standard datasets. The experimental results, presented in Tables 1 to 5, demonstrate that our method achieves superior performance across all datasets. Compared to earlier domain generalization approaches (such as ERM

[Vapnik, 1999], IRM [Ahuja et al., 2020], and adversarial learning-based methods like DANN [Ganin et al., 2016]

and CDANN [Li et al., 2018b]), our method improves crossdomain feature learning and generalization capability by utilizing indirect alignment and maintaining sample diversity.

When compared to other advanced methods based on contrastive learning, such as SelfReg [Kim et al., 2021], PCL [Yao et al., 2022], HTCL [Tong et al., 2023], and IAIE [Hu et al., 2024], our approach effectively avoids overfitting and misalignment issues that arise from direct alignment. This is achieved through the preservation of natural sequence relationships and the indirect alignment strategy. Notably, on particularly challenging domains (e.g., the "L38" domain in the TerraIncognita dataset and the "LabelMe" domain in the VLCS dataset), where the target domain exhibits significant differences from the source domains, other methods perform poorly. In contrast, our method significantly improves the model's performance in these scenarios, demonstrating its superiority and robustness in handling complex domain generalization tasks.

4.4 Ablation Studies

To better understand the contributions of the Sample Difference Keeping (SDK) and Sample Consistency Alignment (SCA) modules, we conducted ablation experiments to evaluate their effects on model performance (Figure 4).

Component Ablation. Using the PACS dataset, we first examined the individual and combined impacts of the SDK and SCA modules. Starting with the baseline containing only the classification loss, we sequentially added SDK and SCA modules. The results, shown in Table 6, reveal the following: Adding the SCA module alone significantly improved performance, confirming its role in enhancing cross-domain feature alignment through class-wise and domain-wise consistency. However, it also introduced a risk of overfitting. Adding the SDK module alone led to moderate improvement by mitigating reliance on domain-specific features and emphasizing relative differences between samples, though it slightly compromised cross-domain alignment. Combining SDK and SCA achieved the best performance, demonstrating a synergistic effect: SDK reduces overfitting risk from SCA by preserving natural sequence relationships, while SCA strengthens cross-domain alignment. Together, these modules enhance both generalization and stability.

Parameter Sensitivity Analysis. We further analyzed the impact of key hyperparameters, including the triplet loss margins (m1, m2, m3) and the consistency loss weight α.

Sensitivity to α: Fixing margin parameters, we observed that model performance peaked when α = 0.01, indicating that an appropriate balance between class-wise and domainwise consistency losses is crucial.

Sensitivity to margin parameters: Fixing other hyperparameters, the model showed low sensitivity to m1, m2, and m3. Notably, m3 exhibited minimal sensitivity, and no alternative weight settings outperformed the default value of 1.

Weight Analysis of SDK and SCA Losses. We further investigated how different weightings of the SDK and SCA loss terms affect performance. Although the optimal balance between them varied slightly from one dataset to another, the overall impact on accuracy remained small. To maintain consistency and simplicity across all experiments, we therefore fixed both weights at 1.

4.5 Visualization Analysis

To intuitively demonstrate the effectiveness of our proposed method, we performed both Grad-CAM[Selvaraju et al., 2017] and feature space visualizations on the PACS dataset.

Firstly, Grad-CAM visualization (Figure 5) highlights the key regions the model relies on for classification. For the cartoon domain, the baseline method (ERM) focuses on less discriminative regions, such as the elephant's body and face. In contrast, our method (OURS) effectively captures the key discriminative feature of the elephant's trunk, which plays a critical role in accurate classification. This observation indicates that our approach avoids overfitting to domain-specific features and instead learns semantically meaningful features that generalize well across domains.

Secondly, We used t-SNE [Van der Maaten and Hinton, 2008] to reduce the dimensionality of the extracted features, as shown in Figure 6. The results show that the features extracted by the ERM method are overly compact within each class, with minimal separation between different classes. This could lead to a decline in generalization ability. In contrast, our method resulted in more dispersed features with greater inter-class separation than the baseline ERMmethod. This is because our proposed SDK module preserves the natural sequence relationships of samples within each class, preventing excessive compression of intra-class features. Additionally, the SCA module effectively extracts cross-domain common features, further enhancing inter-class separability and resulting in clearer classification boundaries. This validates that, while ensuring intra-class consistency, our method effectively enhances inter-class diversity, thereby improving the model's cross-domain generalization ability.

5 Conclusion

In this paper, we propose a novel domain generalization framework to address the shortcomings of traditional contrastive learning methods, which often oversimplify sample relationships, leading to misalignment, feature distortion, and overfitting. The framework introduces two key modules: the Sample Difference Keeping (SDK) module, which preserves natural sequence relationships, and the Sample Consistency Alignment (SCA) module, which achieves indirect alignment through inter-class and inter-domain relationships consistency. These innovations significantly enhance feature representation quality and generalization performance. Extensive experiments on multiple benchmark datasets demonstrate the framework's effectiveness and robustness, consistently outperforming state-of-the-art methods in addressing domain discrepancies.

Despite its notable achievements, the framework has yet to fully explore hierarchical relationships between samples and lacks dynamic sample weighting strategies to better adapt to complex domain shifts. Future research will focus on integrating hierarchical sample relationships and dynamic weighting mechanisms to improve adaptability.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (No.62276160), the Fundamental Research Program of Shanxi Province (Nos.202203021211291, 202303021222030), and Fund Program for the Scientific Activities of Selected Returned Overseas Professionals in Shanxi Province (No.20240002).