Enhancing Cross-Modal Retrieval via Visual-Textual Prompt Hashing

1 Introduction

With the explosive growth of multimedia data, cross-modal retrieval [Messina et al., 2021] [Bogolin et al., 2022] has become a hot issue and attracted continuous research attention from both academia and industry. Its primary objective is to retrieve relevant samples of one modality by the query from another modality. As a promising solution for similarity queries, cross-modal hashing (CMH) [Cao et al., Dataset: MIRFLICKR Text: bokeh/flowers/..; [Concatenate with tags] Caption: some flowers blooming in the grass... Label: flower; plant life

∗Corresponding author.

Dataset: NUS-WIDE Text: wildlife/…; [Concatenate with tags] Caption: a gray coyote standing in grass... Label: animal; sky

Dataset: MS-COCO Text: two zebra…; [Repeat with descriptions] Caption: two zebra standing on dirt field… Label: zebra

Figure 1: Illustration of image instances with the corresponding texts and captions produced by mPLUG. It is observed that texts extracted from MIRFLICKR-25K and NUS-WIDE lack contextual relationships, whereas texts utilized in MS-COCO often exhibit semantic redundancy. In contrast, captions produced by the large-scale model demonstrate a higher degree of realism in describing images.

2018][Zhang et al., 2022] aims to map heterogeneous multimedia data into the Hamming space, ensuring that similar content possesses analogous representations within this hash space. Due to the computational efficiency and storage cost advantages, CMH has been extensively investigated in recent years for the retrieval and analysis of multimodal data.

In recent years, the rapid advancement of Deep Neural Networks (DNN) has stimulated the proposal of numerous deep cross-modal hashing (DCMH) approaches. According to the involvement of semantic label supervision signals, existing DCMH methods can be roughly categorized into two types: supervised DCMH [Zhu et al., 2022] [Ou et al., 2023a] and unsupervised DCMH [Luo et al., 2021a][Mikriukov et al., 2022][Hu et al., 2023]. Technically, unsupervised DCMH methods leverage co-occurrence information to excavate consistency features from multimodal data and learn the hash function. In contrast, supervised DCMH methods generally attain superior retrieval performance than unsupervised DCMH methods by leveraging label-level information to learn more discriminative and general representations. To this end, the objective of our work is to generate discriminative unified binary codes across modalities for handling crossmodal retrieval tasks in the supervised learning paradigm.

Despite progress in learning with DCMH, it is noted that current methods are grounded in an implicit yet ideal assumption-namely, that existing manually-annotated multimodal datasets can effectively simulate cross-modal retrieval scenarios akin to real-world environments. Due to the challenges of real-world data collection from noisy websites and expensive annotation processes, the benchmark datasets used in the realm of cross-modal retrieval, including MIRFLICKR-25K [Huiskes and Lew, 2008], NUSWIDE [Huiskes and Lew, 2008], and MS-COCO [Lin et al., 2014], remains the inherent limitations posed by context loss and information redundancy, particularly in the representation of textual data. As illustrated in Figure 1, the textual content in both the MIRFLICKR-25K and NUS-WIDE datasets is generated by a straightforward concatenation of multiple tags, leading to a deficiency in contextual information. Furthermore, the textual content in the MS-COCO dataset is composed by merging multiple manually annotated image descriptions, which may introduce a certain level of semantic redundancy. The challenges mentioned above pose obstacles to the accuracy and generalization capabilities of existing DCMH methods when applied in real-world scenarios.

To address these challenges, this paper proposes a novel Visual-Textual Prompt Hashing (VTPH) paradigm that leverages the powerful semantic reasoning capabilities of largescale vision-language models to reconstruct and enrich simulated textual environments, leading to a more effective and robust cross-modal retrieval process. In comparison to the stateof-the-art methods [Liu et al., 2023b][Tu et al., 2022][Ou et al., 2023b], the proposed VTPH approach mainly benefits from the advantages of cross-modal prompt engineering. Specifically, two well-established prompt mechanisms, i.e., visual alignment prompt (VAP) and textual alignment prompt (TAP), are proposed to collaboratively enhance the contextual awareness and semantic capabilities embedded within simulated textual features. On the one hand, the VAP mechanism is meticulously designed to enhance salient features and suppress irrelevant information for text representations under the guidance of the global context from visual features. On the other hand, the TAP mechanism aims to capture more authentic and context-rich textual representations by bridging the semantic gap between simulated texts and captions generated by the large-scale model mPLUG [Li et al., 2022]. By incorporating visual and textual prompts within a unified framework, our VTPH approach aims to optimize the interaction and alignment between different modalities based on the assumption of a scenario that closely resembles real-world environments. Furthermore, we propose an affinity-adaptive contrastive learning module to explicitly model the nuanced heterogeneity and semantic gaps between simulated and realworld textual environments. It can dynamically recalibrate the semantic interaction between visual and textual modalities, providing a more accurate representation of the semantic relationships for cross-modal retrieval.

To the best of our knowledge, our work is the first attempt that incorporates both visual-textual prompt learning to address the limitations of context loss and information redundancy within the domain of cross-modal retrieval. Our VTPH framework is comprehensively evaluated on multiple large-scale datasets, and extension experiments are also conducted to demonstrate the robustness of VTPH on benchmark datasets with noisy correspondences. The promising performance collectively demonstrates the effectiveness and superiority of our VTPH method over state-of-the-art algorithms.

2 RelatedWork

2.1 Deep Cross-Modal Hashing

Cross-modal hashing (CMH) [Cao et al., 2018][Yao et al., 2021][Zhang et al., 2022] aims to learn hash functions that map raw data into a binary hash space, along with the hash metric intended to preserve semantic similarity between original multimedia data. Benefiting from the powerful representation abilities of deep neural networks, deep cross-modal hashing (DCMH) [Zhu et al., 2022] [Liu et al., 2023b] has gained significant attention in addressing largescale cross-modal retrieval tasks. Due to the absence of semantic information, the retrieval performance of unsupervised DCMH remains unsatisfactory. By contrast, numerous supervised DCMH methods have been introduced to utilize manual annotation label information to facilitate the learning of hash functions. Specifically, one group of these studies is based on the CNN framework, such as DADH [Bai et al., 2020], MSSPQ [Zhu et al., 2022], and CMGCAH [Ou et al., 2023b]. With the proposal of large-scale vision-language architectures (i.e., CLIP), transformer-based methods as another group have achieved more promising performances than traditional CNN-based methods. For instance, DCMHT [Tu et al., 2022] firstly employed a visual transformer for encoding image content to improve the correlation modeling of CMH. In particular, MITH [Liu et al., 2023b] hierarchically considered intra-modal interaction and inter-modal alignment with multi-granularity in one unified transformerbased framework. Additionally, CIMON [Luo et al., 2021b] presents a novel approach to explore noisy data scenarios, potentially enhancing the robustness against data imperfections. However, the existing methods rely on simulated environments crafted from manually annotated datasets, which suffer from the challenges related to context loss and information redundancy.

2.2 Prompt Learning

Initially originating from the natural language processing (NLP) domain, prompt learning leverages pre-trained models to undertake downstream tasks by seamlessly incorporating handcrafted prompt templates into the input [Brown et al., 2020]. Nevertheless, the meticulous design of crafted prompt templates requires extensive domain expertise and common knowledge, ultimately limiting the model's flexibility. In contrast, prompt tuning enables the model to adapt prompts as continuous vectors and optimize them directly during the fine-tuning process. Inspired by the significant success of prompt tuning in NLP, recent studies [Zhou et al., 2022b] [Zhou et al., 2022a] [Khattak et al., 2023] have tried to incorporate the concept of prompt learning in vision-language models. Based on the CLIP pre-trained vision-language architecture, CoOp [Zhou et al., 2022b] was the earliest work to apply trainable text prompt vectors for few-shot transferring. Co-CoOp [Zhou et al., 2022a] introduced conditional context optimization that dynamically made a prompt conditioned on each input image instead of fixed ones. Recently, MaPLe [Khattak et al., 2023] and DCP [Liu et al., 2023a] facilitated a strong coupling between visual and language prompts to ensure their mutual collaboration. However, whether prompt learning is effective for cross-modal hashing remains underexplored and investigated. In this paper, we first attempt to design a multi-modal prompt learning paradigm to enhance the interaction between visual and textual representations for tackling the challenge of cross-modal hashing.

3 Methodology

3.1 Notations and ProblemFormulation

In the context of N image-text pairs denoted as P = {Pi}Ni , where Pi = {(vi, ti); ci; li}, vi and ti represents the image and text modalities of the i-th image-text pair, ci denotes the image captions generated offline by mPLUG, and li indicates the associated label matrix with Q categories. In our work, the similarity matrix S for cross-modal retrieval is generated based on labels. With the supervision guidance of S, the objective of this study is to acquire unified hash codes by projecting both image and text data from a highdimensional space into a common K-bit discrete Hamming space, where K is the length of the hash codes. Based on the Hamming distance between similar instances, we aim to learn two hashing functions, i.e., bvi = Hv (vi; θv ) ∈ {-1,+1}K and bti = Ht (ti; θt ) ∈ {-1,+1}K , where bvi and bti represent the learned hash codes to preserve the semantic similarities between visual and textual modalities, θv and θt denote the trainable parameters during the prompt-tuning stage.

3.2 Modality-Specific Feature Embedding

Figure 2 provides a detailed pipeline of the proposed VTPH framework. Following previous work [Liu et al., 2023b], our VTPH framework adopts the pre-trained Vision Transformer (VIT) [Dosovitskiy et al., 2021] and GPT-2 [Radford et al., 2019] as image and text encoders to extract modality-specific feature representations, i.e., F v i = [gv i , zv i ], F t i = [gt i , zt i ], and F c i = [gc i , zc i ], where g represents the global class embeddings, and z represents the sequence of local token embeddings. It is noted that the obtained image captions can be considered as additional text modalities to augment the original textual features. In this part, we implement a weightsharing strategy between the text encoder and caption encoder to ensure the consistency of textual representations.

3.3 Visual-Textual Prompt Learning

By keeping the backbone network fixed during fine-tuning, we incorporate two well-designed prompt mechanisms, i.e., visual alignment prompt (VAP) and textual alignment prompt (TAP), to jointly enrich the contextual content and semantic representations of simulated textual features.

Visual Alignment Prompt. The goal of the VAP component is to enhance the salient features and suppress irrelevant information within textual features under the guidance of global embeddings from images. Specifically, the distribution of textual feature information is reshaped by applying the attention prompts to local text features from the image branch. The prompted feature zti is formulated as:

where Wvap ∈ Rd×d are learnable weights from the visual adapter, d represents the dimensionality of feature representations. During the adaptation process, the dimensions of both input and output remain constant. Importantly, the adapted features could seamlessly replace the original local text features as enhanced text features, that effectively filter out irrelevant information while retaining discriminative features to facilitate the subsequent multimodal fusion process.

Textual Alignment Prompt. To harness the robust reasoning capabilities inherent in large-scale models, we design the TAP component to augment the semantic capabilities within simulated textual features and align them more closely with real-world scenarios. Following previous work [Pei et al., 2023], the prompted feature gti is adapted as follows:

where W1 tap ∈ Rd×h and W2 tap ∈ Rh×d are learnable weights serving as textual adapters, h = d/4 is hidden dimensionality of feature representations. Subsequently, the output gti is directly supplied to the forward process as the global textual feature. Instead of employing a straightforward projection for attention weight generation, we introduce a cosine triplet contrastive learning objective [Khan et al., 2023] to optimize the global text features, ensuring a more seamless alignment with authentic textual expressions,

where M is the number of batch size, i ̸= j, < ·, · > denotes the cosine simarility function, and [x]+ = max(0, x).

Visual-Textual Collaboration. To obtain more comprehensive and realistic feature representations, we insert zc i extracted from image captions into the text encoder as additional textual tokens. Notably, we selectively unfreeze the last two layers of the transformer-based encoder to fully utilize the informative cues provided by prompts and enhance the overall learning capability. Formally, the proposed visual and textual prompt mechanisms can be formulated as:

where d represents the depth of the self-attention (SA) layer in the text encoder, [·, ·] represents the concatenation function, and the symbol "-" indicates that the output tokens at the corresponding positions are discarded.

3.4 Affinity-Adaptive Contrastive Learning

Based on the aforementioned operations, both global embeddings g∗ i provided by visual and textual modalities are applied to the residual multi-layer perceptrons (ResMLP) to align them to the same dimensionalities,

Simultaneously, the local embeddings z(∗) i employ a localized token aggregation (LTA) strategy [Liu et al., 2023b] to localize the preservation of the most crucial implicit semantic knowledge from the global embeddings by selecting the top-m features of high confidence to form embedding,

where TE represents a two-layer transformer encoder.

To mitigate the heterogeneity and semantic gaps between simulated and real-world textual environments, we design the Affinity-Adaptive Contrastive Learning (AACL) module to dynamically recalibrate the semantic interaction between visual and textual modalities. Different from traditional contrastive learning [Chen et al., 2020] [He et al., 2020] [Grill et al., 2020], the affinity Ai is specifically designed to capture the nuanced heterogeneity and semantic gaps between simulated and caption textual environments. Specifically, we employ Jensen-Shannon divergence [Lin, 1991] to compute the affinity between the global features gt i extracted from image captions and original global text features gc i ,

where DJS ∈ [0, 1] indicates the Jensen-Shannon divergence. Given the i-th image-text pairs (˜ gv i , ˜ gt i ) in a minibatch, we treat two modality data as queries and keys alternatively to learn the positive image-text pairs and the remaining pairs as considered negative samples. By automatically adjusting the temperature hyperparameter to fine-tune the strength of traditional contrastive learning, the objective of the AACL module can be formulated as follows:

where ̂ τi denotes the temperature parameter, which is adaptively determined by the affinity Ai,

By multiplying the affinity value Ai with a hyperparameter γ, the temperature can be automatically updated in a residual manner. In this way, the dynamic bridging of heterogeneity and semantic gaps across different modalities brings the representation closer to real-world scenarios.

3.5 Cross-Modal Hashing Learning

The primary purpose of the cross-modal hashing module is to map features into the Hamming space and ensure that the distance relationships between hash codes reflect the semantic similarity of different modalities [Jiang and Li, 2017] [Cao et al., 2018] [Tu et al., 2022]. To this end, a hashing linear projection layer (HashLayer) with the tanh activation function is designed to decompose the projected features to global semantic features h(∗) i , i.e.,

Similarly, the linear projection hashing layer is designed to map ˜ z(∗) i to the K-bit Hamming space as local semantic features f (∗) i , which can be expressed as:

To learn the unified hash code, we employ the sign function to integrate global and local semantic features to collaboratively accomplish the encoding process,

where λ ∈ [0, 1] denotes a tunable weight parameter. Moreover, the quantization loss is used to learn a uniform semantic representation and generate compact hash codes, and the objective functions can be defined as follows,

Inspired by [Liu et al., 2023b], both intra-modal similarity preservation loss and inter-modal similarity preservation loss are introduced in our cross-modal hashing learning. In particular, the intra-modal similarity preservation loss aims to preserve semantic similarities within modalities,

where Ω(∗) ij = 1 2 (f (∗) i )T f (∗) j indicates the inner product among local semantic representations. Meanwhile, intermodal similarity preservation is designed to preserve semantic similarities across modalities, that is,

where Θij = 12 (hti )T hvj and Φij = 12 (hvi )T htj denotes the inner product among global semantic representations. Thus, the overall objective of cross-modal hashing learning is defined as follows:

where α and β are trade-off hyper-parameters.

3.6 Training and Optimization

Based on the above analyses, the comprehensive training objective of the proposed VPTH approach encompasses a combination of various loss functions, i.e.,

By jointly optimizing these losses, our approach can further enhance the performance of cross-modal retrieval, ensuring better adaptability to noisy data in real-world scenarios.

4 Experiments

4.1 Experimental Settings

Datasets. Three commonly used multi-label image-text cross-modal datasets, i.e., MIRFLICKR-25K, NUS-WIDE, and MS-COCO, are selected for our experiments. In addition, our settings follow the data splitting protocol used in [Tu et al., 2022][Liu et al., 2023b], which are shown in the supplementary document for details. To demonstrate the robustness of our VTPH approach, we also conduct extension experiments on these benchmark datasets with randomly 30% noisy correspondence, which are referred to as MIRFLICKR-25KN, NUS-WIDE-N, and MS-COCO-N [Huang et al., 2021].

Metrics. To comprehensively evaluate the performance of our method, we perform two cross-modal retrieval tasks, i.e., image-to-text retrieval (I →T) and text-to-image retrieval (T → I). These tasks involve searching relevant texts by using images as queries and vice versa. As standard evaluation metrics, the mean Average Precision (mAP) and precisionrecall curve (PR-curve) are also considered to measure the effectiveness of different methods under the hamming ranking protocol and hash lookup protocol, respectively.

Baselines. In our experiments, we conduct a comprehensive comparison with the state-of-the-art DCMH methods, including CNN-based methods and transformer-based methods. Specifically, the CNN-based methods consist of DCMH [Jiang and Li, 2017], SSAH [Li et al., 2018], GCH [Xu et al., 2019], DADH [Bai et al., 2020], TEACH [Yao et al., 2021], MSSPQ [Zhu et al., 2022], and CMGCAH [Ou et al., 2023b]. In addition, the transformer-based methods contain DCMHT [Tu et al., 2022] and MITH [Liu et al., 2023b].

4.2 Comparisons with State-of-The-Art

Hamming Ranking Protocol. To evaluate the effectiveness of our VTPH framework, we conduct a comprehensive comparison with a range of state-of-the-art baselines on three benchmark datasets by the image-to-text and text-to-image retrieval task. The comparative results are summarized in Table 1. We can obtain the following observations: 1) The proposed VTPH method outperforms all state-of-the-art baselines with significant performance improvements across all hash code lengths. 2) In particular, the latest Transformerbased framework, namely MITH, surpasses all classic previous works such as DADH, DCHMT, and CMGCAH. This phenomenon demonstrates that the multi-modal CLIP architecture has stronger feature extraction and discriminative hash code learning capabilities than the CNN-based baseline. 3) Despite the reliable performance achieved by MITH, our VTPH approach, with its visual-textual prompt tuning strategy and affinity-adaptive contrastive learning, demonstrates superior performance. Importantly, it can consistently achieve the best performance and surpass the second-best method by the mean mAP of 6.02%, 6.97%, 1.34% for I → T, and 8.13%, 5.55%, 1.36% for T → I, respectively. These findings underscore the efficacy of our approach in leveraging visual-textual prompts to enhance text feature representations, thereby optimizing the interaction and alignment across modalities for the learning of discriminative hash codes.

Hash Lookup Protocol. To verify the performance of our VTPH under the lookup protocol, we calculate the PR-curve metric for the returned instances, and the comparison results with variations of different hash codes (i.e., 16bits, 32bits, 64bits) on three datasets are illustrated in Figure 3. From these figures, it can be observed that our proposed method consistently achieves the best retrieval results in comparison with all the state-of-the-art baselines over three datasets.

4.3 Extended Robustness Evaluation

We further investigate the robustness of our VTPH approach in noisy environments. Following the cross-modal matching studies [Huang et al., 2021] [Qin et al., 2022] [Yang et al., 2023], we perform extension experiments on the MIRFLICKR-25K-N, NUS-WIDE-N, and MS-COCO-N datasets with 16 bits of 30% noisy correspondence. Particularly, we generate synthetic noisy correspondence by randomly shuffling the training images and captions to simulate real-world environments. The comparative results are summarized in Table 2. We can observe that all methods suffer from varying degrees of performance degradation under the influence of noisy data. Nonetheless, the proposed method consistently achieves competitive performance with all cases on three datasets. Specifically, our VTPH yields an improvement of 8.45%, 11.52%, 6.6% for I → T and 10.77%, 10.11%, 6.61% for T → I in average mAP than the secondbest method, respectively. Moreover, our VTPH approach exhibits the least impact on the experimental results in most cases, indicating that our method can mitigate the negative impact of noisy correspondence to a certain extent.

4.4 Ablation Study

In this part, we conduct comprehensive ablation studies by systematically evaluating the impact of each component in VTPH on the MIRFLICKR-25K dataset. Five variations are involved, including a) BASE is regarded as the base model that only utilizes two transformer-based encoders of CLIP and hash layers for hashing learning; b) BASE + VAP adds the visual alignment prompt component based on the basic model; c) BASE + TAP adds the textual alignment prompt component to the basic model, along with the image caption branch; d) BASE + VAP + TAP integrate both the visual and textual prompt learning strategy into the base model; d) VTPH is considered as the "full" model, incorporating the affinity-adaptive contrastive learning.

The comparative results are presented in Table 3. It can be observed that both the visual alignment prompt and textual alignment prompt can work cooperatively with different hash code lengths, leading to more powerful semantic similarity learning capabilities. Furthermore, affinity-adaptive contrastive learning effectively mitigates heterogeneity and semantic gaps across modalities by introducing an augmented contrastive relationship, leading to an improvement in mAP scores from 0.8834 to 0.9020. The above results of the ablation studies demonstrate the importance of each component and their collective integration for cross-modal retrieval.

4.5 Parameter Sensitivity

To assess the sensitivity of parameters, we perform an exhaustive parameter analysis of the proposed VTPH method on MIRFLICKR-25K with 16 bits under different parameter configurations. Specifically, we focus on analyzing the effects of three hyper-parameters, i.e., α, β, and γ, as shown in Eqn. (9) and Eqn. (16). Through careful experimentation and analysis in Figure 4, it can be observed that our VTPH method achieves the best performance when α = 10, β = 8, and γ = 500, respectively. Hence, we can summarize that our VTPH model can obtain superior performance through an optimal combination of these hyperparameters.

5 Conclusion

In this paper, we identified the challenge of context loss and information redundancy in existing manually annotated cross-modal retrieval datasets. To overcome these challenges, we proposed a novel Visual-Textual Prompt Hashing framework that integrated both visual and textual prompt learning with a unified framework for cross-modal retrieval. Importantly, the proposed affinity-adaptive contrastive learning module modeled the affinity differences between simulated and real-world environments to augment the contrastive relationship across modalities. Benefiting from these powerful components, our VTPH approach can effectively mitigate the heterogeneity and semantic gaps among different modalities, even in real-world environments with noisy correspondences.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China (Grant Nos. 62302172, 62176077), in part by the Guangdong International Science and Technology Cooperation Project (Grant No. 2023A0505050108), in part by the Guangdong University Young Innovative Talents Program Project (Grant No. 2023KQNCX020), and in part by the Opening Project of GuangDong Province Key Laboratory of Information Security Technology (Grant No. 2023B1212060026).