TG-VQA: Ternary Game of Video Question Answering

1 Introduction

Video question answering(VideoQA) [Wu et al., 2017; Sun et al., 2021] aims to automatically infer the correct answer given a video and a related textual question. Such a multimodal vision-language task has potential application in a broad range of applications such as vision language navigation in embodied AI [Gu et al., 2022], video content retrieval by users' questions [Jin et al., 2022; 2023]. Tremendous progress has been made recently in VideoQA, thanks to advances in vision language pre-training and development in model architecture. However, for a VideoQA task, due to the intrinsic property of the visual data, we typically learn from

(b). Our Ternary Game Video Question Answering Model

Figure 1: (a). Contrastive-based VideoQA models only learn a coarse-grained global alignment before the answer decoder. (b). To achieve fine-grained alignment, we model video, question, and answer as ternary game players and use a VideoQA-specific interaction to generate the label guidance for improvement.

datasets with the long sequence frames, which consist of various visual appearances and rich motion information.

The visual data's long sequence property introduces many challenges for multi-modal reasoning in the wild, as a deep learning model has to simultaneously cope with multi-modal representation learning, visual-linguistic alignment, and answer prediction [Li et al., 2022e; 2022b]. The naive method would require high-quality annotated data and are typically data-hungry, and remains challenging to achieve accurate visual-linguistic alignment.

Early works on VideoQA focus on developing specific architectures [Jiang and Han, 2020; Li et al., 2021; Qian et al., 2022] to align the visual representation and the linguistic question, which require sophisticated and heuristic design. More recent efforts[Lei et al., 2021; Yang et al., 2022a] aims to learn a VideoQA model with contrastive learning by leveraging the power of large-scale dataset, which rarely explore the fine-grained visual-linguistic alignment (shown in Figure 1(a)). This severely limits its modeling capacity and generalization ability in answer prediction. Thus, we raise a question: can we achieve the accurate and robust finegrained alignment in a data-efficient manner for VideoQA?

To answer this question, we need to tackle the acquisition of the accurate annotation of fine-grained alignment between question semantics and video clips. However, it is prohibitive to collect the manual annotation due to the mega-scale of the video amount. A promising idea is to automatically generate the alignment annotation without labor-intensive effort.

Toward this goal, we focus on incorporating visuallinguistic alignment into the contrastive learning framework and propose an annotation generator of the fine-grained alignment (FAG) based on the multi-player game theory [Making, 2009]. Specifically, we introduce the interaction strategy for annotation generator construction, where the (video, question, answer) are treated as ternary game players, and the multi-player game theory could mathematically simulate the pairwise annotation between video clips and question semantics (illustrated in Figure1(b)).

In this work, we first carefully design an interaction strategy to tailor the characteristics of VideoQA. Intuitively, if there exists a strong semantic correspondence between the video player and the question player, and these two players both have a large contribution to the answer, the coalition between them will be strengthened in our framework. Equipped with the annotation generator, we are able to model the fine-grained visual-linguistic alignment with additional supervision signals. To further improve the alignment efficiency, we also explore the multi-modal token reduction strategy for VideoQA. We thoroughly investigate different reduction methods and develop a clustering-based token merge module in the end. Our total framework is named Ternary Game VideoQA (TG-VQA).

We conduct extensive experiments to validate our model on three VideoQA datasets, including MSVD-QA, MSRVTTQA, and ActivityNet-QA. The empirical results and ablative studies show our method consistently achieves significant improvements(more than 5%) on all benchmarks. The annotation generator built from the ternary game also significantly improves the model convergence and data efficiency, which makes our TG-VQA competitive or superior compared with most pre-trained models learned from millions of video data. The main contributions are as follows:

• To the best of our knowledge, we are the first to bring game theory into VideoQA. Utilizing game theory's ability to simulate the video-question token relations, our game theory-based annotation generator helps the VideoQA task achieve fine-grained alignment.
• Our alignment label generator is built from the ternary game. For the characteristics of the VideoQA task, the ternary Game models video, question, and answer
• as ternary game players. The ternary game values the video-question pair's alignment possibility and their contribution to the answer.
• We achieve new SoTA results on short-term and longterm VideoQA datasets, verifying the generalization ability. Without the pretraining stage, our TG-VQA also outperforms most VideoQA pre-trained models.

2 RelatedWorks

2.1 Video Question Answering

The video question answering (VideoQA) task [Zhong et al., 2022] requires models to analyze the complex semantic correlation between the video and the question. The VideoQA task has two main-stream models: (1). Hierarchical crossattention-basedmodels. (2). Contrastive learning-basedmodels. Hierarchical cross-attention models [Xu et al., 2017; Li et al., 2019; 2022c; Peng et al., 2022; Cai et al., 2021; Fan et al., 2019; Li et al., 2023a] design Spatio-temporal attention structures to fusion the video and text features. Several recent models establish the effective alignment stage [Li et al., 2021; Xiao et al., 2021] for VideoQA. [Jiang and Han, 2020] constructs the video clips and text entities into heterogeneous graphs to achieve fine-grained alignment. [Li et al., 2022e] establish the video-question alignment using invariant grounding. [Qian et al., 2022] uses a locator to align the question with video segments. However, these alignment strategies cannot apply in contrastive learning frameworks due to their hierarchical attention structure.

Contrastive learning-based VideoQA models [Lei et al., 2021; Kim et al., 2020; Piergiovanni et al., 2022] use contrastive loss for cross-modality explicit alignment and fusion. However, lacking fine-grained alignment annotations, they suffer from slow convergence, requiring massive video data [Bain et al., 2021; Yang et al., 2022a; Huang et al., 2021; Li et al., 2020] for pretraining. Therefore, we establish a ternary game-based contrastive learning VideoQA model, using game-theoretic interaction [Kita, 1999] to generate the fine-grained alignment annotations.

2.2 Game Theoretic Interaction

The game-theoretic interaction [Making, 2009; Ferguson, 2020] consists of a set of players with a revenue function. The revenue function maps each team of players to a real number which indicates the payoff obtained by all players working together to complete the task. The core of game-theoretic interaction is to allocate different payoffs to game individuals fairly and reasonably. There are several interaction strategies including core interaction [Jeukenne et al., 1977], shapley interaction [Sun et al., 2020] and banzhaf interaction [Marichal and Mathonet, 2011]. The game-theoretic interaction has multiple applications in different fields [Aflalo et al., 2022; Datta et al., 2016]. Recently, LOUPE [Li et al., 2022d] uses two-player interaction as a vision-language pre-training task. In this paper, we design a new framework of ternary game interaction strategy for the VideoQA task.

3 Preliminary of VideoQA and Game Theory

In this section, we first introduce the problem setting of video question answering in Sec.3.1, then briefly present the background for the multi-player game theory (Sec.3.2), which is utilized in our proposed alignment label generator.

3.1 Problem Setting of VideoQA

Given a video clip V and a text-form query Q, the VideoQA task aims to predict the correct answer ˆa from the answer space A. For the close-set type of question, A is a fix-size answering option list. For open-ended and multi-choice kinds of questions, A comprises the group of pre-defined answers and a list of candidate answering options. Generally, We formulate the VideoQA task as follows.

where θ represents the trainable parameters group, Fθ represents the modeling function.

3.2 Introduction of Game Theory

Toward the goal of achieving fine-grained alignment between video and question, we propose to leverage multi-player game theory to construct an alignment label generator. We aim to obtain the semantic relationship between visual tokens and question tokens, and their contribution to the answer. And the game theory targets generating an appropriate coalition construction strategy for multiple players. Thus, we propose to introduce game theory to align the label generation by considering the (video, question, and answer) as players.

Specifically, the multi-player game theory typically consists of (a) a set of players Γ = (P,R) consists a set of players P = {1, 2, ..., n}, and (b) a revenue function R(P). R

maps each team of players to a real score, which indicates the payoff obtained by all players working together to complete the task. The key step of the game theory is to measure how much gain is obtained, and how to allocate the gain fairly.

In the multi-player game process, there are various different interaction strategies available, such as Core interaction [Jeukenne et al., 1977], Shapley interaction [Sun et al., 2020], Banzhaf interaction [Marichal and Mathonet, 2011]. Here we choose the Banzhaf interaction due to its balance of computational complexity and precision. Formally, given a coalition {i, j} ∈ P, the Banzhaf interaction B({i, j}) for the player {i, j} is defined as:

where P {i, j} represents removing {i, j} from P, C stands for coalition. p(C) is 1 2n-2 , possibility for C being sampled.

Intuitively, B({i, j}) reflects the tendency of interactions inside {i, j}. The higher value of B({i, j}) indicates that player i and player j cooperate closely with each other. For VideoQA, we take the matrix B as the alignment label annotation by changing the player definition and using a VideoQAspecific interaction strategy. We will start with a detailed description of our model architecture below.

4 Method

Our model consists of four main submodules: (1) a backbone network for generating feature representations of the video and question (Sec.4.1). (2) a token merge network for reducing the visual and question token number (Sec.4.1). (3) a fine-grained alignment network for establishing the finegrained visual-linguistic alignment in VideoQA (Sec.4.2). (4) an VideoQA answer prediction network for generating the answer for VideoQA (Sec.4.3). We finally detail the training objectives and the inference pipeline in Sec.4.4. The overview of our proposed Ternary Game VideoQA (TGVQA) model is illustrated in Figure2.

4.1 Backbone and TokenMerge Network

Backbone. We adopt the ViT [Dosovitskiy et al., 2020] and BERT as the backbone for generating visual representation and textural representation, respectively. Formally, we denote the representation of a video clip as a set of visual tokens V = {vi|vi ∈ RCv }Nv i=1 , where vi is one frame feature vector with Cv channel and Nv is the total frame number. For the linguistic representation of question and answer, we first pad them into a fixed length Nl sequence and extract their textual feature via a transformer encoder initialized with BERT parameters. We formulate the generated question representation as a set of question tokens: Q = {qj|qj ∈ RCl }Nl j=1 , where qj is a question token with Cl channel. Similarly, we are able to generate the representation of the answer A with a text encoder, which is used to construct an alignment label generator.

Token Merge Network. To reduce the redundancy of visual tokens and question tokens, we develop a token merge network and investigate different merge strategies. Typically, the token merge network consists of (a) temporal context encoding module, (b) sparse token generation module and (c) semantic fusion module, which is illustrated in Figure 3. We will focus on the visual token in the remainder of this paragraph for notation clarity. Given a sequence of visual tokens V, we first utilize the 1-D convolution layer to encode temporal context efficiently and denote the enhanced visual tokens as ˜V. Then, we conduct the sparse token generation for the ˜V to reduce the number of tokens. Specifically, we investigate several different reduction strategies, like the random initialized sparse token(or called query) and clustering-based sparse token generation. We empirically find that the Density Peaks Clustering based KNN (DPC-KNN) [Rodriguez and Laio, 2014; Li et al., 2023b] is superior for generating sparse and representative tokens, the ablative studies are shown in Sec.5.3.

we refer the reader to supplementary material for more details of the clustering method. Finally, we apply the crossattention between the sparse tokens generated from the clustering technique and enhanced visual tokens ˜V, to further incorporate the semantic context information into the sparse tokens. We denote the spare visual tokens generated from the token merge network as Vs ∈ RCv×Nvs , where Nvs is the number of sparse visual tokens and we have Nvs < Nv. Similarly, we are able to produce the spare textual tokens Qs ∈ RCl×Nls from the token merge network. Equipped with spare visual and question tokens, we not only reduce the redundancy in the input data but also improve the fine-grained visual-linguistic alignment efficiency (Sec. 4.2).

4.2 Fine-grained Alignment Network

Different from the current contrastive learning-based methods, which adopt the coarse-grained visual-linguistic alignment in model optimization, we develop a fine-grained alignment network that serves as an auxiliary network only existing in the training process, to explicitly supervise the model with an automatically generated alignment label.

The main idea is to introduce an alignment label generator to provide a supervision signal for learning a VideoQA model. We rethink the relationship among each item(video, question, and answer) and find that the alignment between visual tokens and question tokens actually reflects the semantic correspondence. Tokens sharing similar semantic meanings tend to simultaneously contribute to the final answer prediction. Thus, we propose to leverage the multi-player game theory to help find tokens with high semantic similarity. The fine-grained alignment network is composed of (a) an alignment label generator constructed with ternary game theory, and (b) an alignment prediction module.

Alignment Label Generator. Given the spare visual tokens Vs and sparse question tokens Qs, we consider video, question and the answer A as game players, which means N = Vs∪Qs∪A. Intuitively, if a visual token has strong semantic correspondence with a question token, then they tend to cooperate with each other and contribute to the final answer. We now present the ternary game interaction strategy used in our work. For simplicity, we apply Banzhaf [Marichal and Mathonet, 2011] interaction for the ternary game. Concretely, a task-specific revenue function is required for interaction, and we need to consider the fine-grained visuallinguistic alignments as well as the token pair's contribution to the final answer. Thus, the revenue function R should satisfy the following criteria(we omit the subscript s of Vs,Qs for clarity in the following paragraph):

• R(vi, qj) benefits from the semantic similarity between video and question.
• R(vi, qj) benefits from the semantic similarity between the target answer representation A and the prediction from video-question pair.

Thus, our proposed revenue function is formulated as:

where ϕ is a distance measurement for the semantic similarity, G is a linear layer to project the concatenation of vi and qj into the answer representation space.

Then we apply R to Eq.3.2 for the Banzhaf interaction. However, we find brute-force computation of Eq.3.2 is timeconsuming. To speed up the interaction calculation process, we propose a deep learning-based idea by using a tiny convolutional network to predict the revenue function. Specifically, so we first calculate 1000 samples' guidance matrix. We can use these matrices as data samples to learn a tiny model with short epochs. We take such a model as the alignment label generator to generate the guidance matrix (V, Q).

Alignment Prediction Module. We adopt the contrastive learning-based framework to optimize the VideoQA model similar to [Lei et al., 2021]. Differently, we introduce the explicit supervision signal for fine-grained visual-linguistic alignment. We first generate the alignment prediction R(Vs,Qs)) between visual tokens and linguistic tokens by computing their similarity. Then, we regard the guidance matrix generated from the alignment label generator as the teacher and the alignment prediction as the student. The model is optimized by minimizing the Kullback-Leibler divergence between teacher and student. Ternary game loss is

with such a distillation process, the model is expected to learn the multi-modal representation with rich semantic information and fine-grained visual-linguistic alignment.

4.3 Answer Prediction Network

Due to the established visual-linguistic fine-grained alignment benefit from the FAN, we are able to adopt a simplified answer prediction network, without the need for sophisticated multi-modal fusion/reasoning stages like many previous VideoQA models.

Specifically, given the sparse visual tokens Vs and sparse textual tokens Qs generated from the token merge network, we first predict the token-level fusion weight by applying the non-linear projection(typically using linear layer + sigmoid function) on each token of Vs and Qs. We then obtain the global representation for each modality by conducting weighted sum in Vs and Qs, respectively. We denote the generated global feature vectors as vo and qo. Next, we concatenate those two vectors followed by an MLP to predict the answer logits. We use cross-entropy loss between logits and ground truth answer to supervise the whole framework.

4.4 Training and Inference

Combining the LTG from the ternary game module and the cross-entropy loss from the vqa prediction module, the overall loss of our model is the weighted sum of both parts:

α is the trade-off hyper-parameter for the ternary game. As shown in Figure. 2, the structures with green backgrounds Baseline

39.0 Coarse-grained

-

39.0 � -

44.5

Baseline+FAN

Baseline+FAN+TM 46.3

43.1 Fine-grained Fully-connect 43.1 � Temporal 43.7

Ternary Game 44.5

DPC-KNN 46.3

Table 3: (I). Ablation for our fine-grained alignment network (FAN) and the token merge module (TM). Both modules have benefits. (II).

The ablation for the alignment strategy. Both fine-grained alignment methods surpass the coarse-grained baseline. Our ternary game strategy

outperforms the intuitively fully-connected strategy. (III). Ablation for the cluster strategy in the token merge module.

and Dotted lines are auxiliary components, which only appear in training. During the inference process, TG-VQA only activates the answer prediction network.

5 Experiments

5.1 Datasets

We select multiple VideoQA datasets to comprehensively evaluate the effectiveness of our method on different-length videos. Following the VQA-T [Yang et al., 2022a] setting, we choose two short video datasets (MSVD-QA, MSRVTT-QA) and one long video dataset (ActivityNet-QA) as our evaluation benchmarks. MSVD-QA [Xu et al., 2017] comprises 1,970 short clips and 50,505 question-answer pairs. The clip's average length is 10 seconds and the questions are divided into five question categories: what, who, how, when, and where. All of them are open-ended. MSRVTT-QA [Xu et al., 2017] comprises 10K videos and 243K questionanswer pairs. The question types are similar to what is included in the MSVD-QA dataset. However, the scenario of the video is more complex, with a longer duration of 10-30 seconds. ActivityNet-QA [Yu et al., 2019] is a Human annotated and large scale VideoQA dataset. The dataset consists of 58,000 QA pairs on 5,800 complex long web videos derived from the popular ActivityNet dataset. The average video length of ActivityNet-QA is 180 seconds, which is much longer than MSRVTT-QA and MSVD-QA.

5.2 Experimental Results

We select the most recent pretrained and non-pretrained VideoQA models for comparison. Table. 1 shows the experimental results on the MSRVTT-QA dataset and MSVD-QA

dataset. Compared with the non-pretrained VideoQA models, our method achieves sustainable improvements, 5.4% on MSRVTT-QA and 11.3% on MSVD-QA. Without millions of video-text pretraining data, our method also surpasses most of the pretrained VideoQA models. Table. 2 shows the experimental results on the long-term VideoQA dataset, ActivityNet-QA. Our model achieves 48.3%, surpassing all pretrained and non-pretrained VideoQA models.

5.3 Ablation Studies

We first explore each module's contribution to the overall TG-VQA performance. In Table.3 (I), Both our fine-grained alignment network (FAN) and token merge module (TM) benefit TG-VQA performance. More ablations are as follows.

Effectiveness of Alignment Strategies. Table.3(II) explores the benefits of various alignment strategies for the VideoQA task. Comparing line1 and line2, fine-grained alignment in VideoQA yields 4.1% benefits from coarsegrained alignment. For fine-grained alignment strategies, our ternary game alignment outperforms the intuitive alignment.

Effectiveness of Token Merge Module. Our token merge module clusters the video and question tokens aiming to reduce the token amount for later interaction. In Table. 3 (III), we make several attempts at the clustering strategies. "Temporal" represents clustering the tokens by temporal or sequence order. "DPC-KNN" represents our clustering strategy. Compared with line 1 using Banzhaf interaction with no merge module, the temporal cluster has a negative impact on the model performance due to the Lack of semantic correlation. Meanwhile, our DPC-KNN clustering strategy adaptively merges tokens under the guidance of semantic similarity, surpassing other clustering strategies.

Impact of Encoders' Initial Parameters. We list all VideoQA initial parameter combinations in Table. 1 row 3. Non-pretrained VideoQA models tend to use ResNet, S3D [Xie et al., 2018] or CLIP as video encoder parameters while using BERT as text encoder parameters. Pretrained models also adapt huge language models as the encoder, including GPT and T5 [Raffel et al., 2020]. For a fair comparison, we apply the most common CLIP+BERT and S3D+BERT combinations. Shown in Table. 1, our TGVQA model with S3D+BERT initialization surpasses other non-Pretrained VideoQA models with S3D+BERT. Using CLIP+BERT initialization, our TG-VQA model outperforms others by 5.4% in MSRVTT-QA and 11.7% in MSVD-QA. Due to FrozenBiLM's large computation from GPT3 and WebVid10M, we don't compare with FrozenBiLM.

Hyper Parameters inTrainObjective. In order to explore the effect of the ternary game loss LTG's hyperparameter on the performance of the model, we train our TG-VQA on the MSRVTT-QA dataset with hyperparameter α from 0.1 to 1.5. Shown in Figure4 (a), the model performance fluctuates in range [45.1, 46.3]. When α = 0.5, our model performs best.

Epoch Analysis. To illustrate our ternary game's ability to accelerate the model's convergence process with limited data, we visualize the epoch performance for our TG-VQA model and the CLIP4clip (non-pretrained VideoQA model without the ternary game) on MSRVTT. In Figure4 (b), both CLIP4clip and TG-VQA apply the same encoder initialization. With the fine-grained alignment network, our TG-VQA converges faster and better than CLIP4clip. We also visualize the epoch curve of the Co-Tok (a pretrained VideoQA model). With limited data, our TG-VQA surpasses Co-Tok on epoch 3, demonstrating our data efficiency.

Question Category Performance Analysis. We visualize the model's performance on the Top-4 question categories on the MSRVTT-QA dataset. As shown in Figure4 (b), with the addition of our ternary game module, the model significantly improves "what" and "how" question types performance, which attributes to the fine-grained alignment brought by the ternary game module.

5.4 Case Visualization

Figure 5 is the case visualization from the ActivityNet-QA dataset. For visualizing the cluster results, we cluster the video clips into two centers and the question tokens into three centers. Both cases show semantic similarity within the same centers. For visualizing the alignment results, Figure 5 shows the top-1 alignment pairs between the video center and the question center. The alignment results conform to the semantic consistency. For visualizing the contribution to the prediction answer, both cases illustrate that when a video-question pair is unlikely to be the answer, its contribution score is rather low (0.1). The second case illustrates that when multiple video-question pairs are similar to the answer, their contribution scores also tend to be the same. The visualization of two cases demonstrates the interpretability of our model.

6 Conclusion

In this paper, we study the fine-grained alignment in the VideoQA task. We innovatively model the VideoQA task as a ternary game process between video, question, and answer. We design a VideoQA-specific interaction strategy to simulate the alignment relationship. Experiments show the effectiveness, generalization, and data efficiency of our model.

Acknowledgements

This work was supported in part by the National Key R&D Program of China (No. 2022ZD0118201), Natural Science Foundation of China (No. 61972217, 32071459, 62176249, 62006133, 62271465), and the Natural Science Foundation of Guangdong Province in China (No. 2019B1515120049).

Contribution Statement

Hao Li and Peng Jin are Equal Contributions. Chang Liu and Jie Chen are Corresponding Authors.