Domain Prompt Learning with Quaternion Networks (Extended Abstract)

1 Introduction

Foundational Vision-Language Models (VLMs) such as CLIP have demonstrated strong generalization capabilities across natural image datasets by leveraging massive paired image-text data during training. However, these models often underperform when applied directly to specialized domains like remote sensing or medical imaging, due to substantial domain gaps. Prompt learning has emerged as a lightweight and efficient method for adapting VLMs without retraining the entire network. Nevertheless, existing prompt learning methods primarily focus on manipulating language prompts, neglecting the visual modality and domain-specific characteristics. To bridge this gap, we propose Domain Prompt Learning with Quaternion Networks (DPLQ), a novel framework that introduces external domain knowledge into both vision and language branches through quaternion representations. By explicitly modeling orthogonal cross-modal relationships, DPLQ facilitates a more robust adaptation of VLMs to specialized fields.

2 Method Overview

Aims to prompt VLMs efficiently from a generalized domain to specific domains like remote sensing and medical images, domain prompt learning with quaternion networks is proposed to facilitate the integration of domain-specific knowledge from large-scale foundation models into VLMs. Illustrated in Figure 1, the quaternion network enables the identification of cross-modal relationships between domainspecific vision features from the foundation model and generalized contextual embeddings from the language branch. Subsequently, this information is utilized to map the generalized contextual embeddings into the specialized domain. Furthermore, well-aligned vision-language relationships in pretrained VLMs are leveraged to propagate domain-specific information from the specialized language branch into the vision branch.

* This paper is an extended abstract of the work published at CVPR 2024, titled Domain Prompt Learning with Quaternion Networks [Cao et al., 2024].

Our DPLQ framework consists of three key stages:

• Domain-Specific Feature Extraction: We utilize largescale pretrained domain-specific foundation models to extract rich visual features pertinent to the target domain. These features serve as external guidance to steer prompt learning towards domain-relevant representations.
• Quaternion-Based Prompt Fusion: The extracted domain-specific visual features and the learnable language context embeddings are fused within a quaternion hidden space. In this space, visual features are placed on the imaginary axis while language prompts occupy the real axis, enabling orthogonal modeling of cross-modal relationships. Additionally, Gaussian noise scaled by the domain features is injected into the quaternion space to enhance robustness against overfitting.
• Dual-Branch Prompt Injection: Prompt tokens derived from the quaternion space are hierarchically injected into both the language and vision encoders of the VLM. This dual-branch prompting ensures that domainspecific knowledge is effectively propagated throughout the network, enhancing the alignment between visual and textual representations.

This design enables efficient domain adaptation by exploiting the strong alignment properties of pre-trained VLMs while introducing targeted modifications driven by external domain knowledge.

3 Experimental Validation

We conduct extensive experiments to validate the effectiveness of DPLQ on a range of domain-specific datasets, and the results are shown in Table 1 and Table 2. Our evaluations cover two major domains: remote sensing and medical imaging.

Remote Sensing. As shown in Table 1, we benchmark our method on eight remote sensing datasets, including MLRSNet [Qi et al., 2020], PatternNet [Zhou et al.,

2018], RSSCN7 [Zou et al., 2015], AID [Xia et al., 2017], RSICD [Lu et al., 2017], UCM [Yang and Newsam, 2010], WHURS19 [Dai and Yang, 2011], and NWPU [Cheng et al., 2017]. DPLQ consistently outperforms state-of-the-art prompt learning baselines such as CoOp [Zhou et al., 2022b], CoCoOp [Zhou et al., 2022a], and MaPLe [Khattak et al., 2023]. Notably, with the ViT backbone, our method achieves an average harmonic mean (HM) accuracy of 83.5%, surpassing MaPLe's 80.4%.

Medical Imaging. For medical domain evaluation, as shown in Table 1, we use three datasets:BTMRI [Nickparvar, 2021], CCBTM [Hashemi, 2023], and CHMNIST [Kather et al., 2016]. Guided by the MedSAM foundation model, DPLQ significantly improves performance over baseline methods, achieving an average HM of 53.36%, compared to CoCoOp [Zhou et al., 2022a]'s 49.45%.

Ablation Studies. We perform detailed ablation studies to assess the contribution of each component. Results show that quaternion modeling, dual-branch prompting, and controlled noise injection all contribute to performance gains. Without quaternion networks, performance drops notably, underscoring the importance of orthogonal inter-modal fusion.

Generalization. DPLQ also demonstrates strong generalization capabilities in cross-dataset evaluations and domain generalization settings, further highlighting its robustness and adaptability. While the proposed method does not achieve top performance on each dataset, it excels with an average accuracy of 73.60%, corresponding to a 1.35% improvement over MaPLe.

4 Conclusion

In this work, we propose Domain Prompt Learning with Quaternion Networks (DPLQ) to address the challenge of adapting large-scale VLMs to specialized domains. By leveraging external domain-specific foundation models and introducing a novel quaternion-based prompt learning strategy, DPLQ effectively transfers general recognition abilities into specialized fields. Our method not only injects domainspecific information into both vision and language modalities but also models cross-modal relationships through the orthogonal structure of quaternion networks. Extensive experiments demonstrate that DPLQ achieves new state-of-the-art results on a variety of remote sensing and medical imaging datasets, confirming its effectiveness and generalization capabilities. We believe DPLQ provides a promising foundation for future research on efficient and robust domain adaptation for VLMs.

Ethical Statement

There are no ethical issues.

Acknowledgments

This work was supported by NSFC Grants (62106116, 62322113, 62376156), China Meteorological Administration Grant (QBZ202316), Natural Science Foundation of Ningbo (2023J027), and the High Performance Computing Centers at Eastern Institute of Technology and Ningbo Institute of Digital Twin.