DUQ: Dual Uncertainty Quantification for Text-Video Retrieval

1 Introduction

In recent years, text-video retrieval has made significant progress, focusing on finding the most relevant videos based on text queries [Liu et al., 2019]. The emergence of feature representation learning [Luo et al., 2022] has laid the foundation for addressing the text-video retrieval task. This method leverages powerful pre-trained models, e.g., CLIP [Radford et al., 2021], which project text and video into a shared latent space based on the semantic similarities of text-video pairs, enabling more efficient and accurate cross-modal retrieval. Meanwhile, coarse-grained and fine-grained feature interactions [Chen et al., 2024] enhance the dense feature extractor to learn better representations. However, is it sufficient to rely solely on the similarity between features to obtain reliable retrieval results?

* Corresponding author.

† Code is available at https://github.com/OPA067/DUQ

Actually, relying on similarity alone is insufficient to support cross-modal interactions and distinguish within-modal differences. As illustrated in Figure 1, given the text query "a man is playing baseball," the top 3 retrieval results are returned based on features similarity using the X-Pool model [Gorti et al., 2022]. Although #R1 and #R2 are related to baseball, the query does not accurately match the actual description. In contrast, #R3 highlights a man playing baseball while being interviewed. However, this is a failed retrieval task because #R3 includes an irrelevant interview scene, which results in greater misalignment in feature interaction. This is a common issue in intra-pair interaction, frequently occurring in multi-scene video retrieval when using a sparse text query. Similarly, the sparse text query and weak semantic differences between highly similar candidate videos lead to higher uncertainty in inter-pair exclusion. In other words, both the interaction and the exclusion of features may involve uncertainty. Therefore, we categorize this uncertainty into two types: (1) Intra-pair uncertainty. Lowquality data, such as multi-scene videos, duplicate images, and non-detailed descriptions, etc., is detrimental to the interaction between paired text and video, inevitably leading to unreliable retrieval results. (2) Inter-pair uncertainty. Highsimilarity data, such as same-scene videos, similar images, and consistent descriptions, etc., often causes interference in the retrieval of positive samples. The root of this issue lies in the limitations of the similarity feature space, which does not provide confidence for intra-pair interaction or construct distinguishability for inter-pair exclusion. Therefore, it is essential to quantify the uncertainty in text-video pairs to enable a more reliable similarity assessment.

To address the uncertainty matching problem, existing methods for text-video retrieval focus on granularity alignment and enhanced feature learning. As for the former, UCoFiA [Wang et al., 2023] proposes a unified coarse-tofine alignment model, which combines interactive similarity aggregation and normalization strategies, effectively improving the accuracy of video-text retrieval. HBI [Jin et al., 2023a] proposes the Hierarchical Banzhaf Granularity Interaction, which uses multivariate game theory to model interactions between video frames and text words, overcoming the limitations of traditional feature similarity. As for the latter, T-Mass [Wang et al., 2024] addresses the inherent sparsity of text features using feature enhancement and regularization techniques, thereby mitigating uncertainty in intra-pair interaction. Although these methods alleviate feature uncertainty to some extent, they often rely on complex fine-grained alignment strategies and feature enhancement techniques. More importantly, these methods still focus on similarity-based feature spaces and far from effectively handling the inherent uncertainties in determining optimal entity combinations with appropriate granularities during text-video matching.

In this paper, we propose a novel text-video retrieval model to tackle both intra-pair interaction and inter-pair exclusion uncertainty problems, named the Dual Uncertainty Quantification (DUQ) Model. Figure 2 illustrates the overall framework. First, for intra-pair uncertainty, we propose an Intra-pair Similarity Uncertainty Module (ISUM) to provide trustworthy predictions by quantifying the uncertainty in text-video feature interaction arising from inherent lowquality data. This method leverages the out-of-domain confidence learning problem in classification tasks, aiming not only to maximize similarity scores for in-domain matches but also to increase confidence in the presence of inherent data ambiguity. Second, for inter-pair uncertainty, we propose an Inter-pair Distance Uncertainty Module (IDUM) to construct modality-specific differences by computing the distance in text-video probabilistic embeddings arising from inherent high-similarity data. We break the limitations of traditional single-feature approaches by proposing a local feature aggregation module to construct diversified probabilistic embeddings. Compared to traditional feature embeddings, probabilistic embeddings offer greater diversity and expressiveness, effectively distinguishing similar features within the same modality. Meanwhile, we use the maximum distance as an exclusion metric between inter-pair multiple probabilistic embeddings, which effectively reduces computational complexity and excludes similar probabilistic embeddings in an extreme boundary fashion. Third, the two uncertainty modules work synergistically, expanding retrieval criteria to include both feature interaction uncertainty and feature exclusion distance. Our contributions are summarized as follows:

• We propose a novel text-video retrieval framework, Dual Uncertainty Quantification, to address the uncertainty issues in text-video interactions and exclusions.
• For intra-pair interaction uncertainty caused by lowquality data, we propose an Intra-pair Similarity Uncertainty Module to provide similarity-based trustworthy predictions and explicitly model this uncertainty.
• For inter-pair exclusion uncertainty caused by highsimilarity data, we propose an Inter-pair Distance Uncertainty Module to construct distance-based diversity probability embeddings, thereby increasing the gap between similar data.
• We conduct extensive experiments on six benchmark datasets: MSRVTT, DiDeMo, LSMDC, MSVD, Charades, and VATEX, achieving state-of-the-art retrieval performance (51.2%, +1.9% in R@1 on MSRVTT).

2 RelatedWork

Text-Video Retrieval. The text-video retrieval is a key research topic in multimodal studies. Early works [Liu et al., 2019; Chen et al., 2020] primarily focus on enhancing feature representations to align text and video, as well as on establishing benchmarks and foundational models. Recently, transformer-based text-video retrieval methods [Luo et al., 2022; Gorti et al., 2022] use cross-attention to abstract multimodal cues, achieving significant performance gains. For example, TS2Net [Liu et al., 2022] employs a "token shift and selection transformer" to preserve token integrity and capture subtle actions, improving retrieval performance. With the large-scale text-image pretraining model CLIP [Radford et al., 2021] achieving significant success, it has inspired improvements in retrieval tasks. For example, DiCoSA [Jin et al., 2023b] enhances text-video retrieval by decoupling coarse features into semantic factors and using adaptive pooling for accurate set-to-set matching. Additionally, to better represent text features, T-Mass [Wang et al., 2024] uses random text modeling and text regularization to extract effective frames and text, thereby enhancing semantic similarity between text and video. Although existing methods establish similarity alignment baselines by fine-tuning CLIP, they overlook the reliability of pairings and the impact of noisy samples. By contrast, our model goes beyond feature-level similarity, eliminating the need for complex granularity alignment strategies and focusing on addressing key issues in retrieval.

Uncertainty Model. Currently, the most influential uncertainty models include uncertainty distributions [Oh et al., 2018; Song and Soleymani, 2019; Chun et al., 2021] and uncertainty metrics [Lakshminarayanan et al., 2017; Sensoy et al., 2018; Li et al., 2024]. Uncertainty distributions construct probabilistic representations to differentiate between features. For example, HIB [Oh et al., 2018] proposes the Hedged Instance Embedding to handle one-to-many probabilistic correspondences for metric learning, which has been successfully applied to face recognition and 2D-to-3D pose estimation. Uncertainty metrics handle out-of-domain classification tasks and are powerful in both making accurate predictions and providing reliable uncertainty estimates. For example, EDL [Sensoy et al., 2018] replaces the Dirichlet distribution on class probabilities and treats the predictions of a neural network as subjective opinions, learning the function that collects the evidence leading to these opinions through a deterministic neural network from data. Although uncertaintybased methods have made impressive progress in recognition and classification tasks, applying them to more complex retrieval tasks remains challenging.

3 Methodology

3.1 Preliminaries

Features Extraction. As shown in Figure 2 (1), given a text query T, we leverage the CLIP [Radford et al., 2021] text encoder to output word features Ft = [w1,w2, · · · ,wNt ] ∈ RNt×D , where Nt and D represent the number and the dimension of the word features, respectively. Then we take the representation of the [EOS] token as the sentence feature Ft ∈ R1×D . Similar to the text encoder, we utilize the CLIP video encoder to extract visual features. Specifically, given a video V comprising multiple frames, we uniformly sample Nv frames, then feed them into the video encoder to obtain visual frame features Fv = [f1, f2, · · · , fNv ] ∈ RNv×D .

Features Interaction. The feature interaction lies in learning a similarity score s(t, v), which aims to maximize the similarity for positive pairs (intra-pairs) and minimize it for negative pairs (inter-pairs). Typically, video frame features Fv ∈ RNv×D are aggregated to obtain video feature Fv ∈ R1×D that can interact with the text feature Ft, e.g., through average pooling of video frames. In our work, we adopt the most effective cross-attention frame aggregation method XPool [Gorti et al., 2022] to obtain the video feature Fv, and compute the similarity score s(t, v) = Ft·Fv ||Ft||·||Fv|| . During training, a common optimization method is to use a symmetric cross-entropy loss in both the text-to-video and video-totext directions, as shown in Eq. (1):

where B is the batch size and λ is the temperature hyperparameter. This loss function maximizes the similarity of intra-pairs and minimizes the similarity of inter-pairs. Although previous works optimize this loss at the feature similarity level with some success, when faced with a sparse textual query, this strategy lacks credibility in intra-pair interactions and fails to support inter-pair exclusion. Therefore, we need to break feature-level similarity and develop a more robust interaction mechanism.

3.2 Intra-pair Similarity Uncertainty

Uncertainty Theory. The Bayesian Neural Network (BNN) [Lakshminarayanan et al., 2017] incorporates Bayesian statistics into traditional neural networks, producing an uncertainty distribution instead of a single value. Orthogonally to BNN, Evidential Deep Learning (EDL) [Sensoy et al., 2018] uses explicit modeling based on Subjective Logic (SL) to capture prediction uncertainty by associating the belief distribution with parameters of the Dirichlet distribution. Specifically, EDL provides an evidential theoretical framework to quantify the belief masses b = (b1, b2, . . . , bB) of the class probabilities ˆ y = (ˆ y1, ˆ y2, . . . , ˆ yB) in a B-class classification problem, and the overall uncertainty mass u follows Eq. (2) :

Let ei ≥ 0 denote the evidence derived from ˆ yi, e.g., nonnegative operation ei = exp(ˆ yi). Then, the belief bi and the uncertainty mass u are computed as:

where αi = ei +1 and S = ∑Bi=1 αi denote the Dirichlet distribution parameters and strength, respectively. The evidence ei serves as a measure of the support gathered from the data, indicating the likelihood of a sample being classified into a particular class. In brief, a higher u and lower ei in target classification indicate greater ambiguity and lower confidence in the outcome. Uncertainty theory demonstrates strong robustness in classification tasks, providing uncertainty in addition to classification probabilities. However, effectively generalizing it to retrieval tasks poses many challenges, such as handling more complex cross-modal features. Therefore, it is essential to understand the differences and connections between the two in order to effectively model uncertainty in retrieval. Similarity Uncertainty Modeling. For a B-class classification task, the probability of classifying xi into the ith class is ˆ yi, with the label yi = 1. For a B-size retrieval task, the similarity between the text ti and the video feature vj is sij, with no label yij. In fact, if we provide label support and reduce the dimension of the similarity matrix, retrieval tasks can be transformed into classification tasks. Therefore, we propose an Intra-pair Similarity Uncertainty Module (ISUM) to guide the model in learning uncertainty from textvideo modality interactions in Figure 2 (2). Intuitively, we expect the similarity matrix SB×B output by the model to be as close as possible to the identity matrix EB×B . At this point, we first obtain the Dirichlet distribution Dir(αij), where αij = eij + 1 = ReLU(Sij) + 1, and the distribution strength Si = ∑Bj=1 αij. Given the known labels yij ∼ EB×B and the Dirichlet distribution pij ∼ Dir(αij), we use a generalized Mean-Squared Error loss to supervise the model in learning how to generate a similarity matrix that approaches the identity matrix:

For the specific ith text to retrieve all videos in the entire batch, we can derive:

where E[pij] = αij/Si, E[p2ij ] = Var[pij] + E2 [pij], and Var[pij] = αij(Si-αij) S2 i (Si+1) . Letting ˆ pij = E[pij] = αij/Si, Eq. (5) can then be further simplified to:

Similarly, for the video LUj can also be computed, and the total evidence loss function is given by Eq. (7):

It is important to note that we elevate the metric for measuring retrieval pair relationships from similarity s to confidence e, and cleverly integrate the identity matrix E to provide a theoretical foundation for building reliable pairings.

3.3 Inter-pair Distance Uncertainty

Uncertainty Probabilistic Construction. Retrieval relationships based solely on feature-level similarity are often unreliable due to false positives (highly similar candidate data) in inter-pair exclusion. To mitigate semantic interference within the same modality, PCME [Chun et al., 2021] constructs probability distributions from text and images, enabling diverse embeddings and overcoming single-feature limitations. However, PCME focus on individual images and fail to address complex video scenarios. Therefore, we propose a Local Feature Aggregation Module to generate aggregated embeddings for text and video in Figure 2 (top). This module includes local feature aggregation via multi-head attention and global feature fusion via concatenation; the former captures fine-grained local details, while the latter provides contextual support for local features. However, the aggregated embeddings (zt, zv) obtained through this module do not show significant differences compared to traditional features. Therefore, it is necessary to re-represent the aggregated embeddings to capture the diversity of probabilistic embeddings.

where hµ∗ and hσ∗ denote linear operations, and p(z | ∗) denotes a Gaussian distribution. To enable stable training and facilitate sampling, we introduce a standard Gaussian distribution ϵk ∼ N(0, I) to generate probabilistic embedding zk t :

where k = 1, 2, . . . ,K, zk ∗ ∈ R1×D , and z∗ ∈ RK×D .

Uncertainty Probabilistic Distance. Most previous methods adopt Euclidean distance [Oh et al., 2018] and MultiInstance InfoNCE loss [Chun et al., 2021] to achieve effective alignment between single probabilistic embedding. However, these methods are often constrained by computational efficiency and embedding selection when handling multiple probabilistic embeddings (K > 1). To address this issue and support inter-pair exclusion, we propose an Inter-pair Distance Uncertainty Module (IDUM), which elegantly resolves the challenge of aligning probabilistic embeddings across multiple instances in Figure 2 (3). Intuitively, to achieve effective inter-pair exclusion, we only need to focus on specific boundary information, namely the maximum distance between inter-pair probabilistic embeddings. This approach not only reduces the number of probabilistic embeddings involved in alignment computation, thereby lowering the computational cost, but also improves the model's sensitivity to embedding boundaries. Specifically, we define the distance between probabilistic embeddings as:

where i, j = 1, 2, . . . , B, m, n = 1, 2, . . . ,K, and zi,m t and zj,n v represent themth probabilistic embedding of the ith text feature and the nth probabilistic embedding of the j th video feature, respectively. Based on effective boundary selection, we choose the maximum value for inter-pair exclusion (i ̸= j) and the minimum value for intra-pair interaction (i = j):

Similar to the feature similarity loss in Eq. (1), we can directly obtain the probabilistic embedding distance loss:

Additionally, the distancematrixD in Eq. (11) between probabilistic embeddings can directly adopt the method used for processing the similarity matrix S in Sec. 3.2, with the labels yij ∼ (1 - EB×B ):

3.4 Training and Inference

Training. Following [Oh et al., 2018], we also introduce an additional KL divergence loss between the probabilistic embeddings to prevent them from collapsing to zero:

Therefore, based on feature similarity (Eq. (1) and (7)) and probabilistic distance (Eq. (12) and (13)) losses, the total training objective can be defined as:

where α and β are weight hyper-parameters.

Inference. After training, in addition to using the regular feature similarity s in Eq. (1) for sampling, the similarity uncertainty us in Eq. (3), the probabilistic distance d in Eq. (11), and the distance uncertainty ud in Eq. (3) can also be used to update the similarity calculation:

where γ1 and γ2 are hyper-parameters used to control the impact of re-ranking on prediction.

4 Experiments

4.1 Experimental Setting

Datasets. We adopt six benchmark datasets for evaluation: (1) MSRVTT [Xu et al., 2016] consists of 10K videos, each paired with 20 captions. We follow the training and testing splits from [Yu et al., 2018]. (2) DiDeMo [Anne Hendricks et al., 2017] contains 10,642 video clips and 40,543 captions. We use the training and testing protocols from [Gabeur et al., 2020]. (3) LSMDC [Rohrbach et al., 2015] includes 118,081 video clips from 202 movies. We use the split from [Torabi et al., 2016], with 1,000 videos reserved for testing. (4) MSVD [Liu et al., 2019] includes 1,970 videos and over 80K captions, with training, validation, and test sets containing 1,200, 100, and 670 videos, respectively. (5) Charades [Sigurdsson et al., 2016] consists of 9,848 video clips, and we adopt the split protocol from [Lin et al., 2022]. (6) VATEX [Wang et al., 2019] contains 34K video clips. We follow the train-test split from [Chen et al., 2020].

Metrics and Implementation. We report R@k (k = 1, 5, 10), median rank (MdR), and mean rank (MnR) as evaluation metrics, where higher R@k and lower MdR/MnR indicate better performance. Following previous methods [Gorti et al., 2022], we use CLIP as our backbone model. The batch size is set to 32, and the model is trained for 5 epochs across different datasets. We sample an average of F = 12 frames from each video clip, resizing them to 224 × 224 pixels for all datasets. The hyper-parameters are set as α = 1 × 10-1 , β = 1 × 10-4 , γ1 = γ2 = 1 × 10-1 , and the number of probabilistic embeddings is K = 7.

4.2 Comparisons with State-of-the-art Methods

Table 1 provides the retrieval results on the MSRVTT, and DUQ outperforms recently proposed SOTA methods on both text-to-video (+0.6% in R@1) and video-to-text (+1.4% in R@1) retrieval tasks. Meanwhile, Table 2 provides text-tovideo retrieval results on other datasets. DUQ demonstrates consistent improvements across multiple datasets, including the long-video DiDeMo (+10.9% in R@1) and the sparse-text LSMDC (+11.3% in R@1), highlighting the effectiveness of our method. See Appendix A for more results.

4.3 Ablation Study

To evaluate the impact of each component on the DUQ model, we conduct an ablation study in Table 3: (1) Baseline: We adopt the conventional feature similarity loss as the baseline. (2) Similarity Uncertainty: Comparing #1 and #2, the latter additionally incorporates the intra-similarity uncertainty module, leading to a noticeable improvement in retrieval performance (+1.3% in R@1), demonstrating the effectiveness of modeling intra-similarity uncertainty. (3) Distance Metric: Comparing #1 and #3 (or #2 and #4), the latter addresses the limitations of single-feature representations by constructing distance-based diversity probabilistic embeddings. This approach increases inter-modal differences and improves retrieval performance (+2.8% in R@1). (4) Distance Uncertainty: Comparing #4 and #6, the latter directly integrates the similarity uncertainty from (2), transforming it into distance-based uncertainty. This integration similarly enhances the model's retrieval performance (+2.9% in R@1). See Appendix B for more ablation studies.

4.4 Out-of Domain Tasks

To evaluate the model's generalization on unseen data, we compare the out-of-domain text-to-video retrieval in Table

1. We first train on the "source" dataset MSRVTT and then test on the "target" datasets DiDeMo and LSMDC. We observe that models achieving strong in-domain performance often experience significant drops when generalized to outof-domain data. Compared to other methods, our approach demonstrates excellent retrieval performance both in-domain and out-of-domain. Additionally, we provide the performance comparison for video question answering in Table 5.

4.5 Visualization Results

Figure 3 illustrates a failed retrieval case discussed in the introduction. By incorporating the similarity uncertainty module into feature interactions, the weak semantic interaction capability under sparse queries is effectively enhanced. Figure 4 visualizes the distances between probabilistic embeddings. The central circle represents the aggregated embedding, while the others represent probabilistic embeddings.

5 Conclusion

In this paper, we propose a novel Dual Uncertainty Quantification (DUQ) framework to provide trustworthy predictions by quantifying the confidence of intra-pair interactions and inter-pair exclusions induced by data uncertainty. Accurate text-video retrieval requires reliable intra-pair associations and inter-pair exclusions to ensure the model effectively captures semantic consistency between text and video. Extensive experiments on six benchmark datasets demonstrate that our approach achieves new SOTA retrieval performance. We hope this work inspires further applications in other domains.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (NSFC) (No: 72471197), the Natural Science Foundation of Xinjiang Uygur Autonomous Region (No: 2024D01A18), the Natural Sciences and Engineering Research Council of Canada, the Sichuan Provincial Philosophy and Social Science Fund General Project(No: SCJJ24ND133), the Sichuan Provincial International Science and Technology Innovation Project (No: 25GJHZ0147) and the Sichuan Science and Technology Program (No: MZGC20240146).