Dig into Multi-modal Cues for Video Retrieval with Hierarchical Alignment

1 Introduction

Vision and language play important roles in the way humans learn to associate visual entities to abstract concepts and vice versa. With the rapid emergence of videos on the Internet, video-text retrieval has become challenging, since both videos and language texts contain rich and structured details.

As shown in Figure 1, a single piece of text is able to demonstrate complicated interactions among various entities, where actions (e.g. 'playing' and 'standing') are denoted by verbs and entities refer to noun phrases (e.g. 'horse' and 'bucket'). Meanwhile, these complicated cues can be well described by constituent modalities of the video data including appearance, motion, audio, overlaid text, speech, etc. Therefore, more and more recent works focus on taking advantage of multi-modal video information for accurate video-text retrieval. For example, benefiting from multi-modal features extracted by multiple off-the-shelf expert models, compact global video representations are learned in [Liu et al., 2019; Gabeur et al., 2020; Patrick et al., 2020]. These global video representations aggregate high-level semantics and are able to be well aligned to the corresponding encoded global text representations. These methods usually achieve promising results for video-text pairs containing distinct semantics. However, miss matching may occur for these methods when video-text pairs contain similar dominant semantics. It is because global alignment among high-level semantics may overlook important details or local relationships contained in local phrases or specific video modalities.

* Corresponding authors.

To demonstrate the above issue more concretely, we give an example of text-to-video retrieval in Figure 1 where video A and B contain similar dominant semantics. The two videos both contain modality cues of 'young horse', 'playing', 'bucket', etc, as mentioned in the text. Global-alignmentbased methods learn to summarize all these cues and their relationships into a consistent and compact embedding. But they may not know whether 'young horse beside standing another horse' or 'young horse playing with a bucket' is the key to distinguish the two videos, since the former one is also distinctive among the retrieval videos in a large dataset. This suggests the importance of introducing local alignment. However, if only aligning each word embedding in texts with each modality in videos and ignoring relatively higher-level (e.g., phrases in texts) semantics, both video A and B can be the retrieval result. Therefore, it is essential for video-text retrieval to capture detailed correspondences and meanwhile avoid being trapped in local optimum matching.

In this paper, we propose to dig into multi-modal cues for video-text bi-directional retrieval with hierarchical alignment. For the multi-modal cues, our model learns detailed modality-specific semantics and their local relationships, and also the global induction of high-level semantics. Hierarchical alignment is carried out to align all the local semantics and relations, and meanwhile to ensure a high-level correspondence. Specifically, to extract detailed semantics, several modality-specific transformers are utilized to capture intrinsic characteristics of each modality such as the temporal and spatial contextual properties. Then, local alignment is achieved by attending modality-specific video representations to the embedding of each word by multi-step cross attention. The importance of each word in a sentence is thus learned, and the relations of the words and diverse multi-modal cues are captured. Moreover, a holistic transformer is used for aggregating high-level semantics (e.g., descriptions of various events in videos) into the global video representation. The representation is then used for global alignment to the global text embeddings and for retrieval ranking.

Extensive experiments are conducted on the MSR-VTT [Xu et al., 2016], ActivityNet [Caba Heilbron et al., 2015], and LSMDC [Torabi et al., 2016] to evaluate video-text bidirectional retrieval performance. Experimental results show that our proposed model can achieve state-of-the-art results on all the above datasets. Ablation studies are carried out to evaluate the effectiveness of each part of our model.

The contributions of our work are two folds. 1) We propose a model that leverages the hierarchical video-text alignment to fully explore the detailed and diverse characteristics in multi-modal cues for fine-grained alignment with local textual semantics, as well as to capture a high-level semantic correspondence. 2) Our proposed model can achieve new stateof-the-art results on three benchmark datasets, demonstrating its effectiveness and generalization.

2 RelatedWorks

2.1 Self-supervised Learning of Video

[Luo et al., 2020; Miech et al., 2020; Rouditchenko et al., 2020; Zhu and Yang, 2020] extracted text or audio information in videos and used it for self-supervised learning on large datasets like HowTo100M [Miech et al., 2019]. [Luo et al., 2020; Zhu and Yang, 2020] introduced new Transformer blocks for textual and visual feature encoding and learning. [Miech et al., 2020] proposed an approach named MIL-NCE to learn strong video representations from scratch. And [Rouditchenko et al., 2020] proposed a network that learns audiovisual language representations directly from randomly segmented video clips and their raw audio waveform.

2.2 Text-Video Retrieval

For better feature matching in the text-video retrieval task, [Chen et al., 2020a; Wei et al., 2020] focused on metric learning methods that assigning weights to positive and negative pairs respectively during network training. An encoding architecture based entirely on convolutional neural networks is proposed by [Li et al., 2020] for better text and video feature encoding. And [Zhao et al., 2020] focused on the long-range dependency in videos and texts by introducing a multi-scale dilated convolutional block.

Several works explored the relationships within texts and videos.[Chen et al., 2020c; Yang et al., 2020] built semantic trees for words in texts to match the corresponding information in videos in a coarse-to-fine manner. [Feng et al., 2020; Wang et al., 2020] constructed relation graphs using features from both texts and videos for better aligning similar information during retrieval. Since multi-modal cues are not utilized by these works, their retrieval performance is not promising.

For multi-modal learning, [Liu et al., 2019; Gabeur et al., 2020] utilized pre-trained models from other tasks to extract multi-modal video features for text and video bi-directional retrieval. And [Patrick et al., 2020] further replaced the original retrieval texts with texts generated by a decoder taking video features from support sets as input.

3 Methodology

Our model carries out hierarchical alignment for video retrieval, which takes a text T = {wi}li=1 and a video's multimodal features V = {mj}nj=1 as inputs. We define l as the length of T , wi as the i-th word, n as the number of modalities in the video, and mj as the features of the j-th modality.

The main architecture of our model is illustrated in Figure 2, which is composed of four main components, i.e., a text encoder for text feature encoding, several modality-specific transformers [Vaswani et al., 2017] for multi-modal video feature encoding, a Local Alignment Module for multi-modal local information alignment and a Holistic Ranking Module for semantic relationship learning and holistic similarity calculation. In the following of this section, we first introduce how we encode text and video features into representations, then describe the two modules, respectively, and finally define the objective function of our model.

3.1 Local Feature Encoding

For text encoding, a pre-trained Bert model [Devlin et al., 2018] (defined as Φ(·)) is used for fine-tuning, the [CLS] output (T c = Φ(T)[c] ) and word embedding output (T w = {wi|wi = Φ(T)(i) }li=1 ) of which are fed to the Holistic Ranking Module and Local Alignment Module.

For multi-modal video feature encoding, the multi-modal video features V are acquired by feeding retrieval videos to n video experts pre-trained on other video-related large-scale datasets following [Liu et al., 2019; Gabeur et al., 2020]. Then, the features of each modality are pre-processed and fed to the modality-specific transformers for detailed representation learning. Specifically, all these transformers ΨMj (·) follow the architecture of the encoder of Transformer in [Vaswani et al., 2017], consisting of multi-head attention and fully-connected layers. Since transformers take embeddings as input, we pre-process video features V as follows.

Firstly, since video features V output from different models have different dimensions, we learn n linear layers {Fj}nj=1 to project the features into a unified dimension duni:

For input of the modality-specific transformers, feature and temporal embeddings are calculated. To initialize the feature embeddings, a max-pooling aggregation is utilized for all features in each muni j , i.e., magg f j = maxpool(muni j ), and the feature embeddings of the j-th modality-specific transformer Ef j is defined as the concatenation of the corresponding aggregated and unified features, i.e., Ef j = [magg f j , muni j ]. To use temporal cues in videos, we embed each second's features in muni j together as [Gabeur et al., 2020] to achievemtmp j and calculate the corresponding aggregated features magg t j . The temporal embeddings of the j-th transformer Et j is defined as [magg t j , mtmp j ].

The input of the j-th transformer min j is defined as the addition of Ef j and Et j , and its output equals to ΨMj (min j ). We combine the [CLS] output of all the n modality-specific transformers as the video representation V rep for hierarchical attention learning and holistic ranking learning:

3.2 Local Alignment Module

Inspired by the Stacked Cross Attention proposed by [Lee et al., 2018], we attend word embeddings T w and video representation V rep for cross attention calculation. Multi-step alignment is then conducted for hierarchical learning.

For each sample in T w (i.e., wi) and V rep (i.e., mrep j ), we first calculate the cosine similarity sij between them:

Then we normalize it by using a relu function (i.e., relu(x) = max(0, x)) as in [Gabeur et al., 2020]:

Attention is performed over V rep given wi in T w , and the attended video representation vector ai is defined as:

where λ is the inverse temperature parameter of the softmax function [Chorowski et al., 2015] to adjust the smoothness of the attention distribution.

Each element in A = {ai}li=1 captures related semantics shared by each word embedding wi in T w and the whole V rep . And such context information will help to determine the shared semantics with respect to V rep in the next matching step, forming a multi-step computation process.

For multi-step hierarchical alignment of knowledge in T w and V rep , an aggregating function f(·) is utilized to update wi in T w for the next alignment by aggregating themwith the corresponding alignment features in A (i.e., ai) dynamically: w∗ i = f(wi, ai). Inspired by [Chen et al., 2020b], we define our aggregating function as a modified gating mechanism:

where gi performs as a gate to select the most salient information, and oi is a fused feature that enhances the interaction between wi and ai. Fg, Fo, bg, and bo are to-be-learned parameters. As defined in Equation 6, both wi itself and oi are utilized by the gating mechanism to refine wi. The gate gi can not only help to filter inconsequential information in T w , but enable the representation learning of each wi to focus more on its shared semantics with V rep .

We combine all the equations in the above of this subsection (i.e., Equation 3 to 6) as Fa, which takes T w and V rep as inputs and is performed K times iteratively:

where k is utilized to denote the k-th step of alignment, T w k and T w k-1 indicate the step-wise features of word embeddings. And when k = 0, T w 0 = T w .

In order to determine the importance of each word given the video modalities, relevance between the i-th word and the video modalities is defined as cosine similarity between the attended vector ai and each word embedding wi:

The Triplet loss is utilized to enforce paired video-text pairs to be close and unpaired ones to be separated in the embedding spaces as in many previous works [Lee et al., 2018; Chen et al., 2020b]. Following [Faghri et al., 2017], we only concentrate on the hardest negatives in a mini-batch instead of comparing with all negatives:

where S(T w , V rep ) is the semantic similarity between T w and V rep , B is the batch size during training. ∆ is a margin value. T w and V rep with the same subscript b are paired samples. Hard negatives are indicated by the subscript b∗ .

The advantage of our multi-step hierarchical alignment is that the loss function can directly supervise the learning of video-text correspondences at each matching step, helping the model to yield a higher quality of the alignment.

3.3 Holistic RankingModule

For holistic feature learning, global information of texts and videos should be extracted and aligned.

We denote the [CLS] output of the Bert model as T c and learn gated embedding module Fg following [Miech et al., 2018] to match the size of T c with that of videos:

Since the video representation V rep only present features of each modality in a video, the holistic video features need to be extracted. We utilize a holistic transformer ΨH (·) for holistic video feature learning, thus the relationships of each modality can be learned at the same time. We compute the feature embeddings of V rep (i.e., V agg f ) the same way as described in Section 3.1, and we don't calculate the temporal embeddings since temporal information is not available in V rep . The concatenation of vrep and its corresponding feature embeddings are fed to the holistic transformer:

We utilize Sh (T g , V h ) to denote the similarity of holistic information in videos and texts, which is supervised by the bidirectional max-margin ranking loss [Karpathy et al., 2014]:

where Θ is the margin. Different from the Triplet loss defined in Equation 9, this loss utilizes all the samples in one minibatch for back-propagation, and it enforces the similarity for true video-text pairs (i.e., Sh (T g b , V h b )) to be higher than the similarity of negative samples (Sh (T g b , V h d ) or Sh (T g d , V h b )), for all b ̸= d, by at least Θ.

The advantage of our modality-specific-to-holistic transformer structure over the state-of-the-art MMT [Gabeur et al., 2020] and Support Set [Patrick et al., 2020] is that our model can learn modality-specific features (containing temporal features in each modality) and holistic features (containing relationships among all modalities) independently, rather than learning the overall features simultaneously. The learning ability of Transformers is better explored.

3.4 Overall Objective Function

The overall objective function of our model is defined as the weighted addition of the Triplet Loss LTri and bi-directional max-margin ranking loss LMar:

where β balances the two objectives.

4 Experiments

We evaluate the performance of our model on several public datasets. In the following of this section, we first introduce the datasets and metrics for result comparison, and implementation details, then illustrate the overall performance of our model on the datasets, and finally present ablation studies to evaluate the effectiveness of each component of our model.

4.1 Datasets andMetrics HowTo100M

HowTo100M [Miech et al., 2019] consists of more than one million instructional videos from YouTube. The automatically-extracted speech transcriptions are utilized to form the text for retrieval. Since this text is naturally noisy and often do not describe the visual content accurately, we only utilize this dataset for pre-training as [Gabeur et al., 2020; Patrick et al., 2020].

MSR-VTT

MSR-VTT [Xu et al., 2016] contains ten thousand YouTube videos, each of which is paired with 20 natural sentences describing it. We follow [Yu et al., 2018] and utilize the '1k-A' split, where 9,000 samples are utilized for training and the other 1,000 samples are utilized for test.

ActivityNet Captions

ActivityNet Captions [Caba Heilbron et al., 2015] is composed of twenty thousand YouTube videos temporally annotated with sentence descriptions. Following the approach of [Zhang et al., 2018], we concatenate all the descriptions of one video to form a paragraph. We evaluate our model on the 'val1' split, where 10,009 videos are utilized for training and 4,917 videos are utilized for test.

LSMDC

LSMDC [Torabi et al., 2016] consists of 118,081 video clips extracted from 202 movies. The corresponding text of each video clip is extracted from the script or video description. 1,000 videos are utilized for test and the other videos are utilized for training.

Metrics

The performance of our model is evaluated with standard retrieval metrics used by many retrieval methods like [Lee et al., 2018; Gabeur et al., 2020; Patrick et al., 2020]: recall at rank N (R@N, higher is better), mean rank (MnR, lower is better), and median rank (MdR, lower is better). For the MSR-VTT and LSMDC datasets, we report N = {1, 5, 10}, and for the ActivityNet dataset, we report N = {1, 5, 50} following [Gabeur et al., 2020; Patrick et al., 2020]. For all the results presented by our model, we report the mean experiments with 5 random seeds.

4.2 Implementation Details

We follow [Gabeur et al., 2020] and utilize seven modalities for feature encoding, which are Motion using S3D [Xie et al.,

2018] pre-trained on the Kinetics dataset, Audio using VGGish [Hershey et al., 2017] pre-trained on the YT8M, Scene using DenseNet-161 [Huang et al., 2017] pre-trained on the Places365, Appearance using SENet-154 [Hu et al., 2018] pre-trained on ImageNet, and OCR, Face, and Speech using models presented by [Gabeur et al., 2020]. For MSR-VTT and LSMDC datasets, we utilize all the above modalities and set n = 7, and for datasets that contain longer videos, e.g., HowTo100M and ActivityNet, we only utilize Motion and Audio modalities and set n = 2.

The max text length l of the MSR-VTT and LSMDC datasets equals to 30 and that of the ActivityNet and HowTo100M equals to 100. The inverse temperature parameter of the softmax function λ in Equation 5 equals to 9, the iteration time K in Equation 7 equals to 3, the mini-batch size B equals to 32 for the MSR-VTT, LSMDC, and ActivityNet and equals to 64 for the HowTo100M. The margin value ∆ in LTri equals to 0.2, and the margin value Θ in LMar equals to 0.05. β utilized to balance the two losses equals to 1e-4 for the MSR-VTT, LSMDC, and equals to 1e-5 for the HowTo100M and ActivityNet.

The HowTo100M is only used for pre-training, and we present the results on the other three datasets. We utilize Adam optimizer for all our experiments. The modalities are used only for feature extraction rather than model training. The learning rate of all our experiments is initialized as 5e-5. We train our model on the HowTo100M for 2 million optimization steps and decay the learning rate by a multiplicative factor of 0.98 every ten thousand steps. For the MSR-VTT

and LSMDC, we train our model for 100 thousand steps and decay the learning rate by 0.95 every one thousand steps. And for the ActivityNet, we train our model for 100 thousand steps and decay the learning rate by 0.90 every 1,000 steps.

For the text encoder in our model, we utilize the 'bert-baseuncased' checkpoint of the Bert model and fine-tune it with a dropout rate of 0.1 during training. Our modality-specific transformers and the holistic transformer are composed of 2 layers and 4 attention heads, a dropout rate of 0.1, a hidden size duni of 1024, and an intermediate size of 3072.

Our experiments are conducted on NVIDIA V100 32G GPUs. Taking the MSR-VTT dataset for example, training our model from scratch on this dataset on a single GPU takes about 15 GPU hours for 100 epochs. The average GPU hours/epochs speed of our model is about half of the speed of [Gabeur et al., 2020] and quadruple of that of [Patrick et al., 2020]. The inference speed of text and video bi-directional retrieval for 1000 text-video pairs is around 5 seconds.

4.3 Experimental Results

Overall Performance

We compare the bi-directional retrieval (i.e., both text-tovideo and video-to-text) performance of our method to other recent works. Table 1 to 3 show results on the MSR-VTT, LSMDC, and ActivityNet datasets with and without pretraining on the HowTo100M. Without pre-training, our model outperforms all the other methods on all datasets. With pretraining, except for the median rank of video-to-text on the ActivityNet, our model outperforms other methods.

Specifically, as shown in Table 1, the results of our model on the MSR-VTT dataset without pre-training almost outperform results of [Patrick et al., 2020] with pre-training. As shown in Table 2, on the LSMDC dataset, the results of our model without pre-training have already outperformed results of [Gabeur et al., 2020] with pre-training, not to speak of their results without pre-training. These suggest the stronger retrieval ability of our model.

The improvement of results brings by our model on the ActivityNet is not as large as that on the MSR-VTT and LSMDC. This is because we utilize more words in each text and fewer video modalities for feature learning in the ActivityNet for fair comparisonswith recent works [Liu et al., 2019;

Gabeur et al., 2020; Patrick et al., 2020], the hierarchical information alignment is harder in this case.

More Visualizations

Figure 3 and 4 are provided to show the advantages of the two modules and to prove that our method indeed tackles the issue mentioned in our motivation discussion. As in Figure3, the discrimination of certain video modality is dug out by our method for better alignment. Figure 4 shows that our local alignment module gradually focuses on important words.

Ablation Studies

In Table 4, we illustrate the results of ablation studies carried out for effectiveness evaluation of each part of our model.

To evaluate the effectiveness of modality-specific transformers (denoted as MSTs in Table 4), we replace them with the multi-modal transformer proposed by [Gabeur et al., 2020]. Experimental results demonstrate that our MSTs can achieve better results, suggesting better information learning ability of our model. When the Local Alignment Module or Holistic Ranking Module (LAM or HRM in Table 4) is absent in our model, the retrieval results drop to lower values. Note that even without one module, our model can still outperform the method in [Gabeur et al., 2020] as shown in Table 4. We also conduct experiments with K = 1&2. As can be observed, experimental results in the last four lines in Table 4 are gradually better from top to bottom. This suggests that 1) aligning word embeddings with modality-specific video representation is better than not conducting local alignment, and 2) aligning modality-specific video representation to word embeddings with more steps brings better results.

We also analyze the limitation of our work. As [Liu et al., 2019; Gabeur et al., 2020; Patrick et al., 2020], we utilize multiple models pre-trained on video-related large-scale datasets as experts for video feature extraction. On one hand, the ability of retrieval models is restricted by the capacity of the extracted features. On the other hand, the feature extraction and learning pipeline is rather complicated and may not be suitable to be applied to end-to-end applications. In the future, we will focus on end-to-end and more lightweight solutions for text and video bi-directional retrieval.

Which Modality Is More Important?

As described above, we utilize n = 7 modalities for the MSR-VTT [Xu et al., 2016] and LSMDC [Torabi et al., 2016] datasets, and n = 2 modalities for the ActivityNet

[Caba Heilbron et al., 2015]. It is natural to think that whether the utilized modalities are of equal importance, and if not, which modality is more important for retrieval.

After model training, we acquire the importance of the seven modalities belonging to each sample in the '1k-A' test set of the MSR-VTT, which contains 1000 videos. The averaged importance of each modality is calculated and shown in Table 5. As can be observed, Motion and Appearance are much more important than other modalities, suggesting that the two modalities are commonly described in texts and are more semantically cognizable.

More Modalities, Better Performance?

As shown in Table 5, the importance of OCR and Face is much lower than the Motion and Appearance. It is natural to think that if only utilizing the relatively more important modalities, whether the performance of our model can still be acceptable.

To evaluate this thought, we conduct an experiment with the top 6 modalities in Table 5 and without the Face modality. We utilize the same experimental settings as in Section 4.2 for setting n = 6 and the results are shown in Table 4 Line 3. Compared to the best results presented in the last line of the table, even without the 'least' important modality for training, the performance of the model drop by around 1-2%. This suggests that more modalities may bring better results.

Better Text Encoders, Better Performance?

Better text encoders usually brings better results. [Gabeur et al., 2020] has demonstrated that Bert [Devlin et al., 2018] is better than GrOVLE [Burns et al., 2019] for text encoding and [Patrick et al., 2020] has demonstrated that T5 [Raffel et al., 2019] is better than W2V [Pennington et al., 2014] for text encoding.

However, we argue that this may not always happen. We conduct experiments with XLNet [Yang et al., 2019] as the text encoder, which has been demonstrated to be more powerful than Bert on some NLP tasks. We utilize the same experimental settings as described in our manuscript except for the text encoder and the results are shown in Table 4 Line 2. Compared to the best results presented in our manuscript the model presents worse results when XLNet is utilized.

5 Conclusion

In this paper, we propose a model that first captures local semantics by utilizing multiple modality-specific transformers, and then explores hierarchical structural relationships among local semantics by multi-step video modality-word alignment, and meanwhile uses a holistic transformer to ensure the global alignment. Experimental results on three public video retrieval datasets demonstrate the advantages of our model. Ablation studies are also carried out to verify the effectiveness of each part of our model.