Contrastive Transformer Cross-Modal Hashing for Video-Text Retrieval

1 Introduction

With the rapid development of the Internet, many video social networking sites and smartphone video sharing apps have emerged in recent years, and the number of videos on the Internet has witnessed an explosive growth. Efficient and accurate video retrieval is receiving ever-increasing attention. When searching with natural language, it is desirable to return relevant images and videos timely and accurately. As there is a big semantic gap among different modalities, e.g., video and text, it is challenging to perform video-text crossmodal search.

Hashing [Weiss et al., 2008; Li et al., 2020; Li et al., 2022] has received a great deal of attention due to its efficient storage and computation over continuous features. The core idea of hashing is to project high-dimensional data points into compact hash codes, while preserving similarity among original data in the Hamming space. Cross-modal hashing maps multi-modal data into a common Hamming space, where efficient retrieval is performed. The shallow cross-modal hashing methods employ a two-stage strategy, that is, first extract the hand-crafted feature or deep feature using pre-trained network, and then learn hash code on the extracted feature of multi-modal data.

∗Corresponding author.

Deep cross-modal hashing [Wu et al., 2019; Li et al., 2022; Yu et al., 2022] has been recently developed to perform latent hash code learning in an end-to-end manner. Conventional cross-modal hashing methods are primarily tailored for image-text retrieval, and they encounter two main challenges when extended to video-text retrieval. These methods typically extract features for each frame and word, pool them into video and sentence-level features, and are further trained to obtain hash codes. This overlooks correlations among the sequential units inherent in videos and sentences, and exploration of such correlation often helps improve performance. Some network structures, e.g., Recurrent Neural Network (RNN) [Lipton, 2015] and Long Short Term Memory (LSTM) [Hochreiter and Schmidhuber, 1997] can model sequence data. However, training LSTM is computationally expensive and also cannot well capture long-run dependencies among distant frames due to gradient vanishing [Pascanu et al., 2013]. In addition, they are not sufficient to capture intermodality and intra-modality similarity among video and text modalities due to their complex data structures. Therefore, it is still challenging to devise a cross-modal hashing specifically optimized for video-text retrieval.

To address the above concerns, we propose a novel crossmodal hashing method, i.e., Contrastive Transformer Crossmodal Hashing (CTCH) for video-text retrieval. CTCH applies a new video augmentation strategy and EDA [Wei and Zou, 2019] to augment video and text respectively. CTCH builds a network architecture that consists of bidirectional transformer encoder and a hash layer, and is mainly trained with multi-modality supervised contrastive loss. The overview of the proposed CTCH is illustrated in Figure 1. The main contributions of this work are summarized:

• We propose a novel cross-modal hashing method, i.e., Contrastive Transformer Cross-modal Hashing (CTCH) for video-text retrieval. CTCH is among the first attempts of cross-modal hashing that utilizes bidirectional

transformer to effectively capture long-term dependencies in frame and word sequences.

• We devise a supervised multi-modality contrastive loss on constructed inter-modality and intra-modality triplet sets to effectively exploit structures among videos and texts at both inter-modality and intra-modality levels.
• The quantitative and qualitative results empirically verify the superiority of the proposed method over state-ofthe arts in video-text retrieval.

2 RelatedWork

2.1 Cross-Modal Hashing

Cross-modal hashing aims to map multi-modal data, e.g., images, texts into a shared Hamming space, where hash codes of different modalities can be fast compared and matched. The primary advantage of cross-modal hashing is its efficiency in supporting efficient cross-modal retrieval, e.g., image-text retrieval. Existing cross-modal hashing methods includes two categories, i.e., shallow cross-modal hashing [Zhang and Li, 2014; Lin et al., 2015] and deep cross-modal hashing [Jiang and Li, 2017; Li et al., 2018; Qi et al., 2021; Jin et al., 2023], based on whether feature learning is performed by deep neural network.

The shallow cross-modal hashing methods learn linear function to transform multi-modal data into hash code. For instance, Semantic correlation maximization (SCM) [Zhang and Li, 2014] extends the canonical correlation analysis in the supervised manner. Semantics-preserving hashing (SePH) [Lin et al., 2015] learns the joint binary hash codes by minimizing the Kullback-Leibler divergence between the hash codes and the semantic affinities, and then learns the crossview hash functions with the kernel logistic regression. The shallow cross-modal hashing typically uses hand-crafted features, and feature learning is independent of hash code learning. Therefore, retrieval performance of shallow cross-modal hashing is expected to be further improved.

Deep cross-modal hashing leverages the power of deep learning, and integrates feature learning and hash code learning into a unified framework. Deep cross-modal hashing

(DCMH) [Jiang and Li, 2017] performs an end-to-end learning framework, using a negative log-likelihood loss to preserve the cross-modal similarities. Self-Supervised Adversarial Hashing (SSAH) [Li et al., 2018] utilizes two adversarial networks to jointly model different modalities and thereby further capture their semantic relevance and representation consistence under the supervision of the learned semantic feature. Existing cross-modal hashing methods primarily focus on image-text retrieval, while video-text retrieval is becoming increasingly important due to the richer information contained in videos. There are a few attempts [Qi et al., 2021; Jin et al., 2023] by encoding videos with I3D or LSTM. However, these methods often require high computational complexity during training. Developing effective cross-modal hashing for video-text retrieval still remains challenging due to complex structure of video.

2.2 Video Hashing

In recent years, some deep learning-based hashing methods have been proposed for video retrieval due to great achievement of deep neural networks in extracting high-level semantic features. Self-supervised Temporal Hashing (SSTH) [Zhang et al., 2016] is a pioneer unsupervised video hashing method that models temporal sequence using LSTM. Specifically, SSTH applies Binary LSTM (BLSTM) to encode video to hash code, and minimizes reconstruction loss of deep autoencoder. Later, based on SSTH, Self-supervised Video Hashing (SSVH) [Song et al., 2018] is further proposed by using hierarchical binary autoencoder and preserving similarity. Neighborhood Preserving Hashing (NPH) [Li et al., 2019] preferentially encoded neighborhood-relevant visual content of a video into a binary code referring to preextracted neighborhood information. Bidirectional Transformer Hashing (BTH) [Li et al., 2021] utilizes a bidirectional transformer as the backbone model and designs three selfsupervised learning tasks to adequately capture the similarity structure in video data. However, these hashing methods are proposed for only video-video retrieval.

3 Contrastive Transformer Cross-modal Hashing

3.1 Problem Setup

Given a video-text dataset with N instances O = {Oi}Ni=1 , where Oi = {Vi, Ti}, Vi and Ti are the corresponding video and text data of the i-th instance, CTCH aims to learn kbit hash codes as compact representations for each video and text. We use CNN features {vm i }Mm=1 ∈ RM×d of M sampled frames {f m i }Mm=1 to represent a video, where d is the dimension of each frame feature. For video modality, we feed to a bidirectional transformer based hash model to obtain binary codes Bv ∈ {-1, 1}k , where k is the code length. For text modality, we use a pre-trained tokenizer to convert text into token sequences, and then feed to a pre-trained bidirectional transformer based hash model to obtain binary codes Bt ∈ {-1, 1}k .

3.2 Data Augmentation

Video Augmentation

A popular video augmentation strategy is to apply random spatial augmentation approach, e.g., random cropping, colorjittering, blurring on each frame. However, such strategy can inevitably disrupt temporal structure, since consecutive frames may be augmented differently. In this work, we propose to apply the same spatial augmentation to frames. Moreover, we collect the augmented frames that do not overlap with original frames to capture a wider range of information. Algorithm 1 demonstrates the detailed procedure of temporally consistent spatial augmentation, where the parameters are first randomly generated for each video and then applied to all frames.

Text Augmentation

We employ the widely-used Easy Data Augmentation (EDA) [Wei and Zou, 2019] for text augmentation. Given a sentence, we apply four augmentation approaches randomly, including (1) Synonym Replacement (SR) that randomly replaces n non stop-words with synonyms, (2) Random Insertion (RI) that randomly inserts a synonym of a non stop-words into a random position, (3) Random Swap (RS) that randomly swaps two words, (4) Random Deletion (RD) that randomly removes each word with a probability.

3.3 Bidirectional Transformer Encoder

Inspired by the great success of self-attention in capturing correlations in a sequence [Vaswani et al., 2017], we employ bidirectional transformer to encode video and text, both of which are essentially types of sequences.

Video Encoder

Bidirectional Transformer [Devlin et al., 2019] has been shown powerful to process sequential data and successfully applied to various video applications [Sun et al., 2019]. Some other sequential models, e.g., LSTM related networks, are limited by input sequence length and cannot well preserve long-term dependency in videos [Pascanu et al., 2013]. In contrast, bidirectional transformer forms attention between any two frames in parallel, which is beneficial to model

Algorithm 1 Video augmentation

Crop(f): Crop f with a random size;

Resize(f): Resize f to size of 224 × 224;

Flip(f): Flip f horizontally or vertically randomly;

Noise(f): Add Gaussian noise to f;

Gray(f): Convert f to grayscale with a probability of 0.2;

Blur(f): Apply Gaussian blur to f with a probability of 0.8;

Input: Video clip V = {f m}Mm=1 ;

Output: ˆ

V = {ˆf m}Mm=1 .

• 1: Randomly generate augmentation parameters;
• 2: for m ∈ {1 . . .M} do
• 3: ˆf m = Resize(Crop(f m))
• 4: ˆf m = Flip(ˆf m)
• 5: ˆf m = Noise(ˆf m)
• 6: ˆf m = Gray(ˆf m)
• 7: ˆf m = Blur(ˆf m)
• 8: end for

correlations between distant frames. This is the advantage over LSTM in video analysis. To exploit ordering information in the input sequence, we further use positional encoding in transformer. Following [Vaswani et al., 2017], position feature pi,m of the m-th frame from the i-th video is generated by sine and cosine functions of different frequencies: (pi,m)2j = sin (m/100002j/d ), (pi,m)2j+1 = cos (m/100002j/d ), where j denotes index of dimension. The position embedding and visual sequence are of equal length, and are first added and then fed into video transformer. Assume there are L transformer layers, and each layer is constructed by multi-head attention [Vaswani et al., 2017]. Specifically, in each transformer layer, given an input sequence of embedding X, the j-th attention head projects X to a triplet of (query, key, value) denoted as (Qj, Kj, Vj) via three learnable parameters, i.e., Wq j , Wk j , and Wv j respectively. A scaled dot-product attention is applied between Qj and Kj, and its output is then fed to softmax function to obtain attentional distribution over Vj. Formally, the j-th attention head computes the output embedding sequence as follows:

where Qj = XWq j , Kj = XWk j , Vj = XWv j , dk is a scaling factor. After being passed through L transformer layers, these input tokens are mapped to a sequence of l-D latent visual embeddings {hvi,m}Mm=1 . Each of these embeddings contains not only the visual content in corresponding frame, but also information flowing from other frames in both direction within the video.

Text Encoder

The bidirectional transformer facilitates deeper understanding of context by allowing simultaneous processing of input data from both directions. This feature makes it highly effective at capturing the interdependencies between words in a sentence, thereby enhancing the performance of NLP tasks. With such superiority, this work employs bidirectional transformer to encode text. The tokenizer first splits a text corresponding to a video into words, and truncates them to a unified length. Each word is assigned to a unique token in vocabulary table, and each sentence can be transformed into a token sequence. Particularly, special classification token ([CLS]) and unknown token ([UNK]) are inserted to indicate the beginning of a sentence and out-of-vocabulary word respectively. In each sentence, we convert each token into an embedding, aggregate embeddings of all the tokens, and feed the aggregation to the bidirectional transformer. Text encoder has the same structure with video encoder.

Hash Layer

The representation obtained through the transformer is realvalued, a hash layer is utilized to project the continuous representation into discrete hash code. Retrieval can be more efficient and storage can be saved to a large extent by this way. Taking video as an example, we project {hvi,m}Mm=1 to a sequence of real-valued vectors {zvi,m}Mm=1 via a Fully Connected (FC) layer. It can be formulated as:

where FC(·, k) is mapping function of the FC layer that maps original feature into k-bit hash code. The mean pooling and quantization are applied on frame-level feature to obtain video-level hash code, which can be denoted as:

where sgn is sign function, and sgn(x) = 1 if x ≥ 0 and sgn(x) = -1 otherwise. Accordingly, the continuous feature zti and hash code bti in text modality can be obtained as follows:

where hti denotes the i-th latent text embedding.

3.4 Multi-modal Supervised Contrastive Learning

As a dominant component in self-supervised learning, contrastive learning has received increasing interests due to its success in many research areas, e.g.,computer vision [Qian et al., 2021], multimedia analysis [Chen et al., 2021]. Contrastive learning [Chen et al., 2020] is based on a triplet set consisting of anchor, positive, and negative samples. The goal of contrastive learning is to pull anchor and positive sample together and push apart anchor from negative samples. This work considers the benefits of contrastive learning in learning hash codes as compact representation for video and text.

Given a set of N samples {xi, gi}Ni=1 , gi is the label vector of xi. Its augmentated set is denoted as {ˆxi, gi}2N i=1 that has 2N samples, and ˆx2i-1 and ˆx2i are two random augmentations of xi(i = 1 . . . N). Assume that i ∈ I = {1 . . . 2N} be the index of an arbitrary augmented sample, and (ˆxi, ˆxj) be a positive pair. The conventional self-supervised contrastive loss [Chen et al., 2020] is defined as follow:

where τ is a temperature coefficient that controls dynamic range of product, A(i) = {a|a ∈ I, a ̸= i}, ˆxi and ˆxj are called the anchor and positive respectively, and the other 2(N - 1) samples {ˆxk|ˆxk ∈ I, k ̸= i, k ̸= j} are called the negatives. As label information is not incorporated in selfsupervised constrastive loss, its performance is limited.

We propose supervised multi-modal contrastive loss that fully considers label supervision information to improve representation capability of video and text modalities. As shown in Figure 2, the proposed multi-modal supervised contrastive loss includes inter- and intra-modality contrastive losses to preserve inter- and intra-modality similarity structure respectively.

Inter-modality Contrastive Loss

Inter-modality contrastive learning is defined based on a triplet set of anchor, inter-positive, and inter-negative samples. Inter-modality contrastive learning simultaneously encourages embedding of anchor to be close to that of interpositive sample and to be far away from those of internegative samples, such that cross-modality correlation can be effectively exploited in latent embedding space.

Let zvi and ˆzvi be original and augmented features of the i-th video. Similarly, zti and ˆzti is the original and augmented feature of i-th text. In this work, we define a positive index set as P(i) = {p|p ∈ A(i), gp = gi}, |P(i)| denotes cardinality of P(i). By regarding zvi and ˆzvi as anchors, we have the following inter-modality contrastive loss:

(6) is inspired by SupCon [Khosla et al., 2020], and considers

Algorithm 2 Contrastive Transformer Cross-modal Hashing

Input: video-text pairs Oi = {Vi, Ti}Ni=1 ; input dimension d; code length k; batch size; number of epochs; learning rate; constants α, β, γ, dk, τ.

Output: Network parameters.

• 1: for each epoch do
• 2: for each iteration do
• 3: Sample a minibatch randomly;
• 4: Obtain {hvi,m}Mm=1 and {hti } via bidirectional transformer;
• 5: Obtain zvi , bvi , zti , bti via (2), (3) and (4);
• 6: Calculate multi-modality supervised contrastive loss Lm via (8);
• 7: Update network parameters to minimize (13) via BP algorithm;
• 8: end for
• 9: end for

all the samples belonging to the same class as positives. Minimizing the first term in (6) enables embeddings among original videos and texts in the same class to have high similarities. Similarly, the second term applies to augmented videos and texts. Accordingly, inter-modality contrastive loss Ltem for text can be similarly defined.

Intra-modality Contrastive Loss

Intra-modality contrastive learning defines a triplet set of anchor, intra-positive, and intra-negative samples. Intramodality contrastive learning enables embedding of anchor to be close to that of intra-positive sample and far away from those of intra-negative samples, and thus preserves intrinsic similarity structure within each modality. Taking video modality as example, the intra-modality contrastive loss is defined as follows:

Accordingly, intra-modality contrastive loss Lta m for text modality can be similarly defined.

To sum up, we have the following multi-modal supervised contrastive learning loss:

3.5 Overall Objective

Classification Loss

To fully exploit label information, the learned hash codes are directly used for classification. Taking video modality as example, we consider the widely-used cross-entropy loss:

where ˆgv c = ϕ(FC(bvi )) denotes label prediction using hash code of the i-th video, and ϕ is sigmoid function, gv ic is the c-th element in gv i . The final classification loss in both text and video modalities is defined as follows:

Quantization Loss

As the network output is not binary, direct binarization will lead to large quantization error. Thus it is encouraging to enable the output close to [-1, +1] to control quantization error. Based on such idea, we propose the following quantization loss which is based on the bi-modal Laplacian prior:

As the derivative of absolute function is difficult to compute, we instead apply its smooth surrogate, i.e., |x| ≈ log coshx.

Balanced Loss

To enable hash code to be balanced, hashing enforces each bit to be mean-zero [Yang et al., 2018]. We define the balance loss as follow:

Total Loss

To this end, the overall objective function of the proposed CTCH is defined as follows:

where α, β, γ are trade-off parameters that control relative importances of four losses. The backpropagation algorithm is used to optimize network. The training procedure of the proposed CTCH is illustrated in Algorithm 2.

4 Experiment

4.1 Datasets

The experiments are conducted on three benchmark video text datasets, which have been widely used for video-text analysis. The three datasets are detailed as follows:

MSR-VTT [Xu et al., 2016] is the largest general video captioning dataset. It contains 10,000 video clips with 41.2 hours and 200,000 clip-sentence pairs in 20 categories, and 20 natural sentences annotated manually for each video clip. Following [Xu et al., 2016], we randomly choose 6,513 and 2,990 clips for training and testing respectively.

ActivityNet Captions v1.2 [Krishna et al., 2017] is a largescale video dataset for human action understanding. It contains more than 13,000 videos from 100 activity categories collected from YouTube, with an average of 137 untrimmed videos per class and 1.41 activity instances per video. We randomly choose 4,816 and 2,382 videos for training and testing respectively.

Charades [Sigurdsson et al., 2016] is a dataset composed of 9848 videos of daily indoors activities collected through Amazon Mechanical Turk. The dataset contains 66,500 temporal annotations for 157 action classes, 41,104 labels for 46 object classes, and 27,847 textual descriptions of the videos. Since test set does not provide labels, we use validation set for testing. We choose 7985 and 1863 videos for training and testing respectively.

4.2 Experiment Setting

Baselines

To our knowledge, there are few hashing methods specifically designed for video-text retrieval. We compare CTCH with eight state-of-the-art methods, including two shallow crossmodal hashing methods, i.e., SCM [Zhang and Li, 2014], SePH [Lin et al., 2015], two deep cross-modal hashing methods, i.e., DCMH [Jiang and Li, 2017], DSMHN [Jin et al., 2023], and four deep video hashing methods, i.e., SSTH [Zhang et al., 2016], SSVH [Song et al., 2018], NPH [Li et al., 2019] and BTH [Li et al., 2021]. For SCM and SePH, following [Qi et al., 2021], we mean pool features extracted by VGG-16 to represent video. For DCMH and DSMHN, we use I3D [Carreira and Zisserman, 2017] as backbone to encode video. SSTH, SSVH, NPH, and BTH are developed only for video, we use the same text encoder as DCMH, and train them using loss of DCMH.

Setup

Following [Zhang et al., 2016], we first sample 25 frames resized to 224×224 for each video, and extract 4096-D frame features with VGG-16 [Simonyan and Zisserman, 2015] pretrained on ImageNet [Russakovsky et al., 2015]. Video transformer includes four layers with 256-D attention head, and the scaling factor dk is set to 256. We first concatenate all texts belonging to the same video, then tokenize concatenated text as input. Text transformer has the same structure as video transformer, and its pre-trained model is provided by Hugging Face. The batch size, number of epochs, and learning rate are set to 256, 200, and 1 × 10-4 respectively. The parameters α, β, and γ are set to 400, 0.05 and 0.05 respectively. The temperature coefficient is set to 0.2. The proposed CTCH is optimized using Adam optimizer.

Evaluation Metrics

Following [Zhang et al., 2016], we consider the widely-used mean Average Precision (mAP) and Precision-Recall (PR) curve as evaluation metrics [Over et al., 2010].

4.3 Comparisons with State-of-the-arts

Figure 3 shows the mAPs of all the methods on two videotext retrieval tasks on three benchmark datasets. In addition, the PR curves with 64 bits on MSR-VTT are shown in Figure 4. DCMH and DSMHN use I3d as backbone, and require very large batch size for training on ActivityNet, which has 100 categories. Therefore, the results of the two methods on ActivityNet are not reported. From these results, we have the following interesting observations:

• The proposed CTCH has best mAPs in most cases. For instance, on MSR-VTT, CTCH improves the best baseline, i.e., NPH by 3.18%, 4.16%, 2.78% and 4% with 16,

• 32, 64 and 128 bits respectively for V → T task, and by 7.23%, 9.78%, 5.16% and 5.95% with 16, 32, 64 and 128 bits respectively for T → V task. The PR curves of CTCH are higher than those of the baselines.
• Deep hashing methods outperform shallow hashing methods in most cases. Among deep methods, BTH and NPH exhibit the best performances, showcasing superior capabilities of bidirectional transformer and LSTM in capturing long-term dependencies in sequences.
• Video hashing methods outperform image hashing methods. This indicates that for video-text retrieval tasks, video hashing can extract more useful semantic information from the videos than image cross-modal hashing.

4.4 Ablation Study

Analysis on Loss

We compare four variants of the proposed method with different losses, including (1) CTCH-Cla: a variant that removes classification loss; (2) CTCH-InterC: a variant that removes inter-modal contrastive loss; (3) CTCH-IntraC: a variant that removes intra-modal contrastive loss. Table 1 reports mAPs of the variants on two retrieval tasks on MSR-VTT. As can be observed, CTCH significantly outperforms CTCH-InterC, demonstrating the importance of inter-modality contrastive loss. Meanwhile, CTCH further improves CTCH-IntraC and CTCH-Cla, verifying the effectiveness of intra-modality contrastive and classification losses.

Analysis on Augmentation Strategy

We compare three variants of the proposed method using different augmentation strategies, including (1) CTCH-Aug1: a variant that chooses non-overlapping frames and apply random augmentation; (2) CTCH-Aug2: a variant that chooses the same frames and apply uniform augmentation; (3) CTCHAug3: a variant that chooses non-overlapping frames without augmentation. From Table 1, we see that the proposed CTCH outperforms the three variants. It shows the augmentation strategy in the proposed method that uniformly augments non-overlapping frames allows to capture more information, providing better retrieval performance.

4.5 Parameter Analysis

We study the sensitivity of one trade-off parameter, i.e., α in the proposed method, which are varied from [1, 5000]. Figure 5 shows the mAPs results with respect to different α with

64-bit hash code on MSR-VTT. We clearly see that CTCH remains relatively stable with change of α.

4.6 Case Study

This section presents case study on video-text retrieval. Figure 6 illustrates the top-5 results retrieved of the proposed CTCH and the competitive BTH on MSR-VTT. From this figure, we see that the proposed CTCH is capable of retrieving more correct results than BTH.

5 Conclusion

In this work, we propose a new cross-modal hashing method designed for video-text retrieval. CTCH effectively explores long-term dependencies among video frames and text words via bidirectional transformer. CTCH exploits inter-modality and intra-modality similarities among videos and texts by minimizing multi-modality supervised contrastive loss. Extensive empirical results demonstrate the superiority of the proposed method and verify the effectiveness of each component. In the future, we consider learning hash code on incomplete video and text data.

Acknowledgments

This work was supported by the National Natural Science Foundation of China under Grant No. 62176126, U20B2065, U22B2056, 62272468, the Natural Science Foundation of Jiangsu Province, China under Grant No. BK20230095, BK20211539.