Text-Video Retrieval with Disentangled Conceptualization and Set-to-Set Alignment

1 Introduction

Recent years have witnessed encouraging progress in textvideo retrieval, which enables humans to associate textual concepts with video entities and vice versa [Wang et al., 2021b; Jin et al., 2023b]. As illustrated in Figure 1, humans perceive the cross-modal matching task by conceptualizing highdimensional inputs from multiple modalities and reasoning with concepts to achieve partially matched set-to-set alignment. In stark contrast, machine models typically represent each modality as a perceptual whole.

† Corresponding author: Chang Liu, Jie Chen.

* Code is available at https://github.com/jpthu17/DiCoSA.

Conceptualizing

Set-to-Set Alignment

Existing methods for text-video retrieval mainly focus on learning a joint feature representation space for different modalities, where text-video similarities could be measured to enable cross-modal matching. Such cross-modal representation learning methods allow for both global alignment [Liu et al., 2019; Gabeur et al., 2020] and local alignment [Wang et al., 2022; Wray et al., 2019; Chen et al., 2020] between text and video feature representations. The global alignment methods exploit high-level semantics for text-video retrieval. For example, CLIP4Clip [Luo et al., 2022] adopts the textimage pre-training model CLIP [Radford et al., 2021] to transfer the knowledge for enhancing global representation. To leverage the local details, the local alignment methods study fine-grained semantic alignment for text-video retrieval. For instance, T2VLAD [Wang et al., 2021b] shows the potential of local alignment which aligns each word and each frame individually to improve fine-grained retrieval.

However, existing methods are deficient in the following three aspects. (i) The global alignment (Figure 2, left) may treat discriminative regions equally and fail to capture local details in texts and videos. (ii) The local alignment (Figure 2, middle) aligns each word and per frame individually and is computationally expensive due to the exhaustive matching operation. (iii) Both global and local alignments coarsely represent the text (video) as a perceptual whole encoded by a set of concepts. As a result, they may fail to leverage the heterogeneous concepts in data and tend to focus on identifying the invariant features. As an example to illustrate, the video in Figure 1 involves a set of concepts such as "man", "skis", "woman", and "horse". To align with the text query, the video representations learned by the existing coarse approaches may continuously rely on the salient factors "man" and "horse", and yet ignore other important factors (e.g., "woman" and "skis") and the relations among these factors. This flaw might make these approaches stuck in the local invariant matching and misunderstand the cross-modal interaction and context.

A reasonable solution to tackle the cross-modal matching task is to align texts and videos on heterogeneous semantic concepts (Figure 2, right). The core insight is simulating the human process of conceptualizing things and reasoning on the sets of concepts. To learn explanatory and discriminative factors of variations, we adopt disentangled representations learning. A disentangled representation independently encodes information about each latent factor in only a few dimensions.

To this end, we propose the Disentangled Conceptualization and Set-to-set Alignment (DiCoSA), as shown in Figure 3. In detail, we disentangle high-dimensional coarse features into compact latent factors which explicitly encode textual semantics and visual entities. Then, we optimize latent factors from both inter-concept and intra-concept perspectives. In the interconcept perspective, we minimize the inter-concept mutual information to find representation subspaces with minimal relevance to each other for decoupling representation. In the intra-concept perspective, we maximize the mutual information of each latent factor pair separately to align language and video within each concept. However, due to the information across modalities typically being only partially matched [Liu et al., 2021], we cannot blindly leverage superficial correlations between latent factors for text-video retrieval. To address this problem, as shown in the bottom panel of Figure 3, we propose an uncertainty-aware module to estimate the confidence of each cross-modal concept matching. Finally, we use the confidence as the weight to aggregate all factor pairs to calculate the similarity of text and video, which is called adaptive pooling.

In particular, since the dimension of the latent factor (R D K ) is lower than the original feature dimension (RD ), our method ensures fine-grained interactions with local alignment using a similar computational complexity as global alignment. Our contributions are summarized as follows:

• We propose a disentangled conceptualization method for text-video retrieval, which divides the coarse features into multiple latent factors related to semantic concepts and achieves humanlike set-to-set matching.
• To address the partial matching of visual entities and various phrases, we propose adaptive pooling to locate mismatched cross-modal concepts and aggregate all factor pairs to calculate the similarity of text and video.
• We conduct extensive experiments on five datasets, i.e., MSR-VTT, LSMDC, MSVD, ActivityNet and DiDeMo, and achieve new state-of-the-art retrieval performance.

2 RelatedWork

Text-Video Retrieval. Most works [Jin et al., 2022; Jin et al., 2023b] of text-video retrieval is based on contrastive learning [Zhang et al., 2021; Zhang et al., 2022a; Zhang et al., 2022b; Zhang et al., 2023] to map text and video to the same semantic space. For example, CLIP-ViP [Xue et al., 2022] explores factors that hinder video post-pretraining on pre-trained image-text models and effectively leverage imagetext pre-trained model for post-pretraining. HBI [Jin et al., 2023a] designs a new framework of multivariate interaction for cross-modal representation learning [Li et al., 2022a; Li et al., 2022b; Cheng et al., 2023; Ye et al., 2023]. DRL [Wang et al., 2022] aligns each word and each frame individually to achieve token-wise interaction. However, a visual entity may relate to multiple frames, and also a frame may relate to multiple visual entities. Therefore, the previous methods cannot achieve element-level matching between semantic concepts. By contrast, our model benefits from the disentangled latent factors for element-level cross-modal matching.

Disentangled Representation. This concept was first proposed by [Bengio and others, 2009], which aims to separate the latent factors of variations behind the data. Recently, disentangled representation learning has received lots of attention [Sreekar et al., 2021; Wang et al., 2021a; Tran et al., 2017; Suter et al., 2019; Van Steenkiste et al., 2019; Locatello et al., 2019]. However, how to use the disentangled latent factors for fine-grained retrieval remains largely unexplored for text-video retrieval. Based on the informationtheoretic view [Chen et al., 2016; Do and Tran, 2019], several works provide a more detailed description [Do and Tran, 2019; Eastwood and Williams, 2018; Ridgeway and Mozer, 2018] by explicitly measuring the properties of a disentangled representation [Eastwood and Williams, 2018; Ridgeway and Mozer, 2018]. We adopt the information-theoretic definition and show that the proposed DSCA converges with semantic disentanglement.

3 Methodology

We focus on the tasks of text-to-video and video-to-text retrieval. In the text-to-video retrieval task, given a query text t and candidate videos v, our goal is to rank all videos according to semantic similarity. Similarly, the goal of video-to-text retrieval is to rank all candidate text based on the query video. The problem is formulated as a modality similarity measurement, where the similarity of matched text-video pairs is as high as possible and the similarity of unmatched pairs is as low as possible. Figure 3 illustrates the overall framework.

3.1 Text-Video Joint Encoding

For text representation, we adopt the text encoder of CLIP (ViT-B/32) [Radford et al., 2021]. The output from the [CLS] token is taken as the text representation. For the input text t, we denote the generated representation as T ∈ RD .

For video representation, we first evenly extract the frames from the video clip as the input sequence of video v = {v1, v2, ..., v|Nv|}, where Nv denotes the frame length. Subsequently, we use ViT [Dosovitskiy et al., 2021] to encode the frame sequence. Following CLIP, we adopt the output from the [CLS] token as the frame embedding. After that, we use a temporal transformer (a 4-layer transformer) to aggregate the embedding of all frames and obtain the frame representation F. Inspired by [Gorti et al., 2022; Bain et al., 2022], we aggregate frame representation with text as the condition. In detail, we calculate the inner product between the text representation T and frame representation F = {f1, f2, ..., fNv }. We get the weight of the frames by:

where τ is the trade-off hyper-parameter. The smaller τ allows visual features to take more textual conditions into account during aggregation. The final video representation V ∈ RD is defined as V = ∑Nv i=1 aifi.

3.2 Disentangled Conceptualization fromBoth Inter-Concept and Intra-Concept Perspectives

The above encoding methods only generate the holistic text representations T and video representations V . These representations characterize semantic concepts of the input texts and videos in a state of entanglement. Therefore, a direct similarity matching of these representations cannot ensure adequate set-to-set cross-modal matching between a set of various phrases contained in the text description and a set of varying visual entities in the video sequence. In the following, we learn explanatory disentangled factors of variations in text and video representations to explicitly measure and understand the cross-modal relevance.

We start with the text representation T ∈ RD . Following the setting of disentangled representation learning [Ma et al., 2019], we assume that each text representation is disentangled into K independent latent factors, i.e., Et = [et1 , et2 , ..., etK ]. Each latent factor etk ∈ R D K represents a specific semantic concept in the text, and the independence of the latent factors ensures that those semantic concepts are not related to each other. Specifically, we independently project the text representation T into K components, and obtain the kth latent factor etk as follows,

where W t k ∈ R D K ×D is trainable parameter. The latent factor evn of video representation can be calculated in the same way, i.e., evk =W v k V . In this way, we project features explicitly into representation subspaces corresponding to different concepts. The model is then able to optimize and reason information separately from different representation subspaces.

Inter-Concept Decoupling

In order to find representation subspaces with minimal relevance to each other and hence improve the respective discriminative power for semantic matching task, we propose to minimize the inter-concept mutual information. Given two latent factors eti and evj , their mutual information is defined in terms of their probabilistic density functions:

Obviously, the mutual information is hard to measure directly. To this end, we implicitly measure the mutual information via an encoder discriminator architecture. Concretely, given latent factors et ∈ R D K and ev ∈ R D K , we first normalize them by the following formula:

where zt , zv have the same mean and standard deviation. In this manner, we scale the latent factors to the standard scale. Then, we calculate the covariance of zt and zv as follows,

where zt i and zv j are the normalized features of eti and evj , respectively.

Lemma 1. Maximizing (minimizing) I(eti ; evj ) is equivalent to maximizing (minimizing) Ci,j , i.e., Ci,j ∝ I(eti ; evj ).

Based on Lemma 1, the final inter-concept decoupling loss LD is calculated as follows,

This loss minimizes the mutual information of negative pairs (eti , evj ), thereby decoupling the latent factors.

Intra-Concept Alignment

To comprehensively capture K latent factors from both text and video representations, we are required to mine semantic concepts from the text-video pairs. To this end, we optimize the representation subspace corresponding to each latent factor separately. The key insight here is that we consider each representation subspace independently, instead of text-video pairs, to comprehensively describe the latent factors and capture their relevance. Specifically, we maximize the mutual information between the text latent factor and the corresponding video latent factor within the same subspaces. The intra-concept alignment loss LA is formulated as:

This loss maximizes the mutual information of each positive pair (eti , evi ) separately.

3.3 Set-to-Set Alignment via Adaptive Pooling

However, the information across text and video is typically partially matched [Liu et al., 2021]. We hence cannot directly and blindly leverage superficial correlations between latent factors for set-to-set alignment. To this end, we propose adaptive pooling to locate those mismatched cross-modal concepts and reduce their impact on the final similarity calculation.

To reveal the mismatched cross-modal latent factors, we design an uncertainty-aware module to estimate the confidence of each cross-modal concept matching. Specifically, we concatenate latent factor eti ∈ R D K with the latent factor evi ∈ R D K in the ith subspace, generating the input data of the uncertainty-aware module ˆ ei = [eti , evi ] ∈ R 2D K . Then, the confidence of the ith subspace is obtained by:

where "MLP" consists of two linear layers and an activation function. Usually, small gi indicates that the concept corresponding to the ith subspace is matched with a low probability. Then, we use the confidence as the weight to aggregate all factor pairs to calculate the similarity of text and video, which is called adaptive pooling. Finally, the similarity is defined as:

3.4 Training Objective

Following common practice, we leverage InfoNCE loss [Oord et al., 2018] to optimize cross-modal similarity:

where B is the batch size and τ ′ is a pre-defined temperature prior. Sl,k is the similarity between the lth text and the kth video. Combining the objective functions for cross-modal similarity LS, inter-concept decoupling LD and intra-concept alignment LA mentioned above, we get the total training loss L = LS + αLD + βLA, where α and β are the trade-off hyper-parameters.

3.5 Intuitive Analysis

Current methods mainly perform global alignment or local alignment. Now we explain other advantages of the proposed DiCoSA besides disentangled representation, mainly in three aspects. (i) Humanlike set-to-set matching. Humans perceive the world by conceptualizing high-dimensional inputs from multiple modalities such as vision and language. Through conceptualization, humans integrate things into conceptual networks and make inferences based on them. Our method simulates the human process of conceptualizing things and reasoning text and video with concepts to achieve set-toset cross-modal matching. (ii) Avoiding curse of dimension. As the dimension of representation space increases, the model becomes more and more difficult to optimize, which is known as the curse of dimension [Kuo and Sloan, 2005]. Therefore, we choose to optimize the decoupled subspace (R D K ) at lower dimensions rather than directly learn the representation space (RD ), which we will discuss in experiments (see Figure 4 (c) and (d)). (iii) Efficiency. Our method combines efficiency and granularity well. We calculate inference time on the MSRVTT dataset, which we will discuss in experiments (see Table 4).

4 Experiments

4.1 Experiment Setup

Datasets. MSR-VTT [Xu et al., 2016] contains 10,000 YouTube videos, each with 20 text descriptions. We follow the 1k-A split [Liu et al., 2019] with 9,000 videos for training and 1,000 for testing. LSMDC [Rohrbach et al., 2015]

contains 118,081 video clips from 202 movies. We follow the split of [Gabeur et al., 2020] with 1,000 videos for testing. MSVD [Chen and Dolan, 2011] contains 1,970 videos. We follow the official split of 1,200 and 670 as the train and test set, respectively. ActivityNet Caption [Krishna et al., 2017] contains 20K YouTube videos. We report results on the "val1" split of 10,009 and 4,917 as the train and test set. DiDeMo [Anne Hendricks et al., 2017] contains 10k videos annotated 40k text descriptions. We follow the training and evaluation protocol in [Luo et al., 2022].

Metrics. We choose Recall at rank L (R@L, higher is better), Median Rank (MdR, lower is better) and mean rank (MnR, lower is better) to evaluate the performance.

Implementation Details. Following previous works [Luo et al., 2022; Liu et al., 2022; Jin et al., 2022], we utilize the CLIP (ViT-B/32) [Radford et al., 2021] as the pre-trained model. The dimension of the feature is 512. The temporal transformer [Vaswani et al., 2017; Li et al., 2022c] is composed of 4-layer blocks, each including 8 heads and 512 hidden channels. The temporal position embedding and parameters are initialized from the CLIP's text encoder. We use the Adam optimizer [Kingma and Ba, 2014] with a linear warmup. The initial learning rate is 1e-7 for the text encoder and video encoder and 1e-3 for other modules. If not otherwise specified, we set τ ′ = 0.01, K = 8, α = 0.01, β = 0.005. The network is optimized with the batch size of 128 in 5 epochs. During the inferring phase, we assume that only the candidate set is known in advance. We follow inferring schedules from [Bogolin et al., 2022].

4.2 Comparisons with State-of-the-art Methods

We compare the proposed DiCoSA with other methods on five benchmarks. In Table 1, we show the results of our method on the MSR-VTT dataset. Our model outperforms the recently proposed SOTA methods on both text-to-video retrieval and video-to-text retrieval tasks. Table 2 shows textto-video retrieval results on the LSMDC, MSVD, ActivityNet and DiDeMo datasets. The results on all five datasets demonstrate that our method is capable of dealing with both short and long videos. DiCoSA achieves consistent improvements across different datasets, which demonstrates the effectiveness and generalization ability of our method.

4.3 Ablation Study

Architecture Design. To illustrate the importance of each part of our method, we conduct ablation experiments on the MSR-VTT dataset. From Table 3 (I), we can draw the following observations: (i) The model using latent factors achieves comparable or better performance than the baseline on the two retrieval tasks. We consider that it is because the latent factor dimension is small, thus alleviating the curse of dimension. (ii) Compared with the optimization of latent factors only from the inter-concept perspective, the improvement of the optimization from both inter-concept and intra-concept perspectives is more significant. We consider that it is because intra-concept alignment avoids the interference of mismatched cross-modal concepts in the learning of other concepts. (iii) "Adaptive pooling" can locate the mismatched cross-modal concepts and improve the respective discriminative power for set-to-set matching. Our full model achieves the best performance. This demonstrates that the four parts are beneficial for aligning visual content and textual semantics.

Effect of the Number of Concepts. The concept size K controls the number of latent factors E = [e1, e2, ..., eK], where ek ∈ R D K . We start with a small size and increase it to large ones. In Table 3 (II), overall performance improves and then decreases. On the one hand, we find that fewer concepts limit the ability to leverage fine-grained information. On the other hand, a larger number of concepts reduces the dimension of each latent factor ek ∈ R D K , which limits the discriminability of the factors. We set the concept size K = 8 to achieve the best performance in practice.

Parameter Sensitivity. The parameter α indicates the importance of LD. We evaluate the scale range setting α ∈ [0.0, 1.5] as shown in Figure 4 (a). We find that R@1 is improved from 46.7% to 47.5% when α = 0.005 and saturated with α = 0.01 for text-to-video retrieval. As a result, we adopt α = 0.01 to achieve the best performance. In Figure 4 (b), we show the influence of the hyper-parameter β. We evaluate the scale range setting β ∈ [0.0, 0.05]. We find that the model achieves the best performance at β = 0.005, so we set it as the default.

Comparisons to Other BaselineMethods. We further compare our method with other baseline methods in Table 4. Since the dimension of latent factor (R D K ) is lower than the original feature dimension (RD ), our method introduces negligible computational overhead. Moreover, our method brings remarkable improvements by disentangled conceptualization and set-to-set alignment. We also show the training process of our method and baseline methods in Figure 4 (c) and (d). Because we optimize the decoupled subspace at a lower dimension (R D K ), our method has the most efficient performance improvement. We observe that the performance of our method in early epochs is lower than other methods. We consider that it is because our disentangled alignment is far removed from the CLIP pre-training objective and therefore cannot directly transfer the knowledge of CLIP in early epochs.

4.4 Qualitative Analysis

In Figure 5, we show the visualization of latent factors. In our method, each latent factor individually focuses on a specific detail. Specifically, Factor 6 focuses on "talking"; Factor 7 notices "woman"; and Factor 7 corresponds to "man". Our latent factors are explainable to some extent. Therefore, the proposed method can be used as a tool for visualizing the cross-modal interaction and help us understand the existing retrieval model. Interestingly, we find that the factor pair corresponding to action ("talking") has higher confidence than those corresponding to entities ("woman" and "man"). This result illustrates that the model tends to judge cross-modal similarity by actions rather than entities.

5 Conclusion

In this paper, we propose the Disentangled Conceptualization and Set-to-set Alignment (DiCoSA) for text-video retrieval. DiCoSA is superior at computation efficiency and granularity, ensuring fine-grained interactions with local alignment using a similar computational complexity as global alignment. Experimental results on five text-video retrieval benchmark datasets show the advantages of the proposed method. In the future, we hope that the disentangled features learned by our method could also be applied to other cross-modal tasks.

Acknowledgements

This work was supported in part by the National Key R&D Program of China (No. 2022ZD0118201), Natural Science Foundation of China (No. 61972217, 32071459, 62176249, 62006133, 62271465), and the Natural Science Foundation of Guangdong Province in China (No. 2019B1515120049).