Graph Prompts: Adapting Video Graph for Video Question Answering

1 Introduction

Video Question Answering (VideoQA) aims to assist humans in addressing everyday challenges [Wong et al., 2022; Grauman et al., 2022], such as helping users locate items, reminding them of past activities, and facilitating the completion of complex tasks, which is an intriguing cross-modal task bridging the domains of computer vision and natural language processing.

Each video is involved with dynamic actions, activities, and events, thus VideoQA models should be able to perceive and reason about the temporal information for answer generation. To capture detailed temporal information, recent methods have attempted to explore relation-level temporal modeling to avoid insufficient understanding of temporal context when modeling at the object level [Lei et al., 2021; Xiao et al., 2022; Liu et al., 2021]. For example, some researchers [Li et al., 2023d; Urooj et al., 2023; Zong et al., 2024] propose adopting multi-scale feature encoding to consider the temporal relationships of objects at different scales, such as objects, regions, and clips within the video. However, these methods do not explicitly define and model the temporal relationships between objects, thus they may encounter difficulties in capturing the fine-grained temporal dynamics and complex interactions among objects. Furthermore, to address explicitly define and model the temporal relationships between objects, recent approaches [Shi et al., 2019; Park et al., 2021] attempt to explore graph-structured video representation for VideoQA, as shown in Figure 1 (a). These methods adopt scene graph-liked structure which abstracts the visual content or textual question to relationship triplets, e.g., ⟨subject, predicate, object⟩, thereby explicitly modeling the relationship. However, these question-guided temporal modeling methods rely heavily on the question text, which merely focuses on the question-mentioned objects and ignores other objects in the video that may be crucial for answer reasoning. For example, to deal with a reasoningrelated question "Why does the toddler cry in the last of video?", question-guided methods merely concern the mentioned "toddler" in the given question, while ignoring the unmentioned "dog" that interacts with the toddler, thus generating incorrect answers.

∗ indicates corresponding author: Changsheng Xu

To address these challenge, we introduce the video-guided temporal modeling method, as shown in Figure 1 (b), which has the following advantages: (1) a more comprehensive understanding of the dynamic relationships between objects in the video, and (2) a stronger reasoning ability for video content not mentioned in the given question. Firstly, we construct a video graph and a question graph simultaneously, where the video graph contains objects and corresponding relationships of each frame, and the question graph is generated through semantic analysis of textual questions. Since the video graphs are generated based on video instead of textual questions, the perception of VideoQA model is no longer limited to the directly mentioned information in questions and achieving a comprehensive understanding of video. Then we can adopt message passing between video and question graphs, thus integrating the video and question graph into a union graph structure. In this way, the VideoQA model can obtain stronger reasoning ability for all the video-contained objects by incorporating relationship-level information on the question semantics and all the video-contained objects. However, during the message-passing process, the interaction between the question graph and the video graph is based on nodes, inevitably leading to the model still being biased towards textual questions. Merely expanding the model's perception of the video is insufficient to address this issue.

Inspired by the success of recent prompt-tuning researches , we propose to adopt a two-stage training paradigm and introduce a video graph completion pretext task to address the above issue. In detail, we first randomly mask nodes and edges in the video graph, and the video graph completion is defined as completing the masked graph through temporal modeling, thus forcing the model to automatically learn each relationship in the video graph, rather than merely learning the question-mentioned object relationships. Moreover, since the video graph completion task is like a cloze-style question-answering task, it also facilitates the adaptation to VideoQA. Based on the proposed pretext task, we design a Graph Prompts based Video Question Answering model named GP-VQA. In pre-training, we introduce video SGG methods [Cong et al., 2021; Li et al., 2022b; Nag et al., 2023] to provide a raw video graph, which makes the proposed GP-VQA able to adapt different models flexibly. Then we employ a vision-language model to learn temporal reasoning ability based on video graph completion pretext task. In the prompt-tuning stage, we first organize the questions as question graphs and adopt message-passing between the question graph and video graph, therefore perceiving the global information of the video and capturing the relevant information for answer reasoning. At the last, we take the question graph as a prompt and utilize the language module to predict the answer. The main contributions are summarized as follows:

1. By incorporating video-based scene graphs instead of relying solely on question text for temporal modeling, we ensure that the proposed model obtains a more compre-
2. hensive perception of the video.
3. We propose a graph prompt-based VideoQA approach GP-VQA, which learns the reasoning ability from masked video graph completion pre-training and inherits this ability to VideoQA task in the prompt-tuning stage.
4. The evaluation on the three public VideoQA datasets verifies the effectiveness of the proposed GP-VQA, which also demonstrates the flexibility of GP-VQA.

2 RelatedWork

2.1 Video Question Answering

Some early works [Zeng et al., 2017] apply the global video and question representations for answer prediction, which cannot construct robust fine-grained video semantics. For example, work [Zeng et al., 2017] directly utilizes elementwise multiplication to capture the global multi-modal feature for answer classification. To capture the critical frames and regions, ST-VQA [Jang et al., 2017] propose a dual-LSTMbased approach with spatial and temporal attention mechanisms. Work [Xu et al., 2017] enhances the attention unit by incorporating the interaction between question words and both frame-level and clip-level visual features. Despite the aforementioned methods having the ability to attend to video frames and clips, they have later been shown to be weak in capturing long-term dependency.

To address the above problem, some work attempts to leverage the memory network [B¨armann and Waibel, 2022; Datta et al., 2022] and Transformer [Li et al., 2023c; Yuan et al., 2023] to capture robust long-term representations. Specifically, memory networks can cache sequential inputs in memory slots and explicitly utilize long-term information, and Transformer utilizes the self-attention mechanism to model long-distance dependencies in context. Recently, graph-structured techniques have been favored for improving the reasoning ability of VideoQA models. Some work [Park et al., 2021; Wang et al., 2021; Gao et al., 2023; Peng et al., 2021; Huang et al., 2020] build the graphs based on coarse-grained video segments. To capture the objectlevel information, L-GCN [Huang et al., 2020] utilizes the object representation to construct graph, e.g., appearance and location features. Furthermore, work [Peng et al., 2021] concatenate the object-level, frame-level, and clip-level graphs to learn the visual relations

The most similar method to ours is [Park et al., 2021], which also adopts question graph to connect textual and visual domains. The main difference is that we introduce videobased pre-training and design graph-based multimodal interaction to expand the model's perception and avoid limitations imposed by the question.

2.2 Scene Graph in VideoQA

Since the previous work succeeded in adopting scene graph in visual question answering, this idea has been extended to videoQA and captioning tasks [Tsai et al., 2019; Cherian et al., 2022; Park et al., 2021]. For video understanding, scene graph generation is defined to decompose long-term actions into a series of frame-level relationships triplets [Ji et al., 2020; Li et al., 2022a; Li et al., 2022b; Li et al., 2023e]. STG [Pan et al., 2020] explore combining video scene graphs with knowledge distillation for video captioning, while STSGR [Geng et al., 2021] propose to adopt a multi-modal transformer guided by scene graph for VideoQA. Moreover, (2.5+1)D-Transformer [Cherian et al., 2022] propose to model the semantic and motion information within video by scene graph, therefore conducting temporal reasoning. However, the above methods construct scene graphs based on the given questions, which limits the model's perception of global information.

To sum up, the major difference between GP-VQA with existing methods is we propose a video-guided temporal modeling approach, which induces the proposed model to utilize broader perception for answer reasoning.

3 Method

The proposed method is illustrated in Figure 2, which consists of two stages: pre-training with masked video graph completion, and prompt-tuning with VideoQA. In the pretraining stage, we adopt standard video SGG model, e.g., STTran [Cong et al., 2021], APT [Nag et al., 2023], and Tempura [Li et al., 2022b], to obtain the raw video scene graphs which abstract spatial-temporal semantic information within the given video. Then, we randomly mask nodes or edges in the raw video scene graph, and force the model to complete the masked parts. In the prompt-tuning stage, we organize the given question into a question graph, and update the feature of each node and edge in question graph by adopting message passing between video graph and question graph. After that, we take the question graph as prompt and feed it into the pre-trained language model with a new QA head to infer the answer of the given question.

3.1 ProblemFormulation

GP-VQA consists of pre-training and prompt-tuning stages, where pre-training stage aims to complete the masked video graph, while the prompt-tuning stage aims to generate more accurate question answers. In other words, these two stages have different output, where pre-training stage generates a complete graph, and prompt-tuning stage utilizes classification to predict correct answer. Therefore, GP-VQA needs to address the following issues: how to design a model to connect these two stages so that the prompt-tuning stage can inherit the temporal modeling knowledge from the pre-training stage?

To address the above issue, we propose to bridge the pre-training and prompt-tuning stages with graph structure. Given the video V = {I1, I2, ..., IN} and question Xq, VideoQA aims to generate the answer as follows:

where ˜a is the infferred answer in answer space A and is generated by Fθ. In contrast, regarding masked video graph completion as a variant of scene graph generation, the completed video scene graph sequence can be denoted as G = {g1, g2, ..., gN}, where gt = {Ot, Et} is composed of the objects Ot and there relationships Et in It. To connect VideoQA and masked video graph completion, we formulate the inference of gt similar to Eq. 1 as follows:

where O and E are the label space of object and relationship, respectively. We assume a dense graph g′t with edges between all the nodes and predict the nodes and edges by Fϵ. Moreover, by analyzing the semantic dependency of each word in the question Xq with [Manning et al., 2014], we can organize Xq into question graph gq = {Oq,Eq}. Thus, the aim of VideoQA can be regarded as predicting a specific node or edge in gq and Eq. 1 can be reformulated as follows:

Since a could be regarded as a node or edge in gq, it can be observed that if g′t and gq are embedded the same, it is possible to handle VideoQA and masked video graph completion with the same function. Therefore, connecting VideoQA and masked video graph completion is transformed into constructing a module or structure that can bridge g′t and gq.

3.2 Pre-training withMasked Graph Completion

Similar to recent VideoQA models, we adopt pre-trained feature extractors to capture the visual feature of V = {I1, I2, ..., IN}. The extracted feature is denoted as FV = {f1, f2, ..., fN}, where ft ∈ R4096 is the visual feature of the t-th frame. After that, we adopt Mask-RCNN [He et al., 2017] to detect the object in each frame, the visual features and categories of objects are also provided by the detector, For the i-th object in It, we denote the visual feature and predicted category as vt,i ∈ R2048 and ot,i.

With the extracted visual features and detected objects, we adopt the existing approach to obtain the raw video scene graphs. The adopted model is denoted as Fd, and raw video scene graphs is denoted as ˆG = {ˆ g1, ˆ g2, ..., ˆ gN}, where ˆ gt = { ˆOt, ˆEt} is the obtained raw video scene graph, ˆOt and ˆEt is the objects and the relation edge in It, respectively. We take the prediction result of Fd as a pseudo-label, and ˆ ot,i ∈ ˆOt can be represented as follows:

where st,i is pseudo-label embedding provided by Glove [Pennington et al., 2014] and ft,i is the original node representation extracted by Fd. Moreover, the representation construction process of edges is the same as that of nodes. Then we establish hyperedges between scene graphs of different frames based on the temporal order, therefore locating the objects and relationships in the temporal dimension and facilitating subsequent question graph construction. For example, a node will be connected to all the nodes in the scene graphs of the previous frame by a directed hyperedge labeled as "after". Similarly, the directed hyperedge between this node and the nodes in the scene graphs of the subsequent frame is labeled as "before". Thus, we can obtain a video graph G to build a comprehensive perception of the video.

Then we construct masked video graph completion based on the G. We randomly mask some nodes and edges with rate λ thus generating masked scene graph, e.g., ⟨person - write on - [mask]⟩ and ⟨person - [mask] - paper⟩. After that, we employ a trainable language model Fl with graph completion head φgc(·) to predict a complete video graph, which takes visual feature and G as input. Denoting the complete video graph as G, it can be generated as follows:

where ϕ(·) is a visual mapping function for dimension align, RM(·) is randommask operation, and Fl predicts the masked element in G as a cloze task. Specifically, we also follow [Dai et al., 2019] to generate relative temporal position encoding and utilize it to determine temporal order, which is concatenated with each ft ∈ FV . Since there are multiple masked nodes or edges in G, the video graph completion task can naturally guide GP-VQA to expand its perception and acquire stronger video understanding and reasoning abilities.

3.3 Prompt-tuning with VideoQA

In prompt-tuning, we aim to allow the transfer of learned comprehensive video understanding and reasoning capability from video graph completion to VideoQA. Therefore, it is necessary to transform the textual questions in VideoQA into a similar structure to G, thus enabling the pre-trained Fl can process VideoQA.

Question Graph construction. To understand the semantic information within both the question and the video, we embed all words in the question and the labels of objects in video graph into 300-dimensional vectors with pre-trained word embeddings ( e.g., GloVe [Pennington et al., 2014]). With the same semantic embedding, we can establish a connection between the video graph and the question, which allows the information integration and facilitates the association between video graph and the question. Moreover, we adopt Stanford CoreNLP [Manning et al., 2014] to identify the semantic dependency within the text, thus abstracting the semantic structure of the question into a graph. As shown in Figure. 2, given the question "What did the person hold after drinking from a cup?", we first decompose this sentence into "what", "person", "cup", "hold", "drinking from" and "after". After that, we construct a question graph gq based on the semantic dependency of these words. Following the construction of the video graph, "person", "cup", and "what" are presented as object nodes in gq , while "what" with no definite semantic information, and we denote it as a masked node. "hold" and "drinking from" are denoted as spatial relationship edges, while "after" is denoted as temporal hyperedge.

Prompts Generation. Although we have organized the textual question into a question graph, it cannot be directly adopted to calculate the answer since gq does not involve any visual information for reasoning. Therefore, we capture related visual information and update gq based on video graph ˆG that is generated by a frozen spatial-temporal semantic extractor in prompt-tuning. We first calculate the relevance between nodes in the question graph gq and the video graph ˆG with temporal order. With denote the node in gq as Vq = {vq,1, vq,2, ..., vq,Nq }, the edge between vq,a and vq,b is defined as eq,ab, the relevance weight between a node vq,a ∈ gq and vt,i ∈ ˆ gt defined as follows:

where sq,a and st,i are semantic embeddings generated with Glove [Pennington et al., 2014]. And we also calculate the relevance weight we q,ab→t,ij between each edge eq,ab ∈ gq and et,ij ∈ ˆ gt. Finally, we form the similarity between relation triplelet as wr q,ab→t,ij = (wo q,a→t,i + we q,ab→t,ij + wo q,b→t,j )/3. Then we can update gq as follows:

1) Triplelet Retrieve: We first retrieve triplets in the video graph that have a similarity higher than threshold α to the masked nodes or edges in the question graph. Then, based on the direction of hyperedges in the question graph, we sequentially search for the corresponding parts of other triplets in the question graph within the video graph, until all the triplets in the question graph have been traversed.

2) Graph Construction: We extract the video subgraphs of frames that correspond to the triplets in the question graph that we retrieved, along with the video subgraphs connected to them by no more than γ hyperedges. Then we establish dense connections between the obtained video subgraphs and the question graph, constructing a cross-modal graph.

3)Feature Update: We apply message passing on the cross-modal graph to update the nodes in the question graph.

In detail, the nodes and edges in gq is updated as follows:

where we process the updating operation with temporal order and wr q,ab→t,c is the similarity between triplelet ⟨a, ab, b⟩ ∈ gq and the most similar triplet, ht∗ is the representation of question graph node or edge * in t-th frame, h∗ is the representation of video graph node or edge *, ⊕ is an elementwise add operation, Fo and Fe are the learnable liner layer with relu activation. Moreover, we adopt σo and σe as score functions that adjust the relevance scores of nodes and edges in the video graph with respect to the question graph based on the hop count to the most similar triplet. For each additional hop, the relevance score decays by a factor of β. By passing messages between video graph ˆG and question graph gq, we denote the generated graph as prompt graph gp which can be processed by the language modle pre-trained in the video graph completion.

Prompt-tuning with VideoQA. In prompt-tuning, we freeze the spatial-temporal semantic extractor and GC head while the language model and graph prompts generator are trainable. We take gp to replace G as the language model input, the answer prediction can be formulated as follows:

where φqa is trainable QA head. Furthermore, since there are different types of questions, e.g., causal, temporal, and descriptive in NExT-QA [Xiao et al., 2021], we adopt different classifiers as QA head for different question types. We utilize the standard cross-entropy loss function to optimize the parameters in prompt-tuning, which is denoted as LQA.

4 Experiments

Dataset: We evaluate our model on three recently proposed challenging datasets for the long-form VideoQA, namely AGQA v2 [Grunde-McLaughlin et al., 2022], NExTQA [Xiao et al., 2021], STAR[Wu et al., 2021]. AGQA v2 is an open-ended VideoQA benchmark for compositional spatial-temporal reasoning, NExT-QA is a benchmark for causal and temporal reasoning, STAR is a benchmark for situated reasoning.

Baselines: We compare our model with several stateof-the-art VideoQA algorithms: SHG-VQA[Urooj et al., 2023], VQA-T[Yang et al., 2021], AIO[Wang et al., 2023], VGT[Xiao et al., 2022], MIST-AIO[Gao et al., 2023], MISTCLIP[Gao et al., 2023], GF[Bai et al., 2024], ATM[Chen et al., 2023], CoVGT[Xiao et al., 2023], TranSTR[Li et al., 2023d], VideoChat2[Li et al., 2023b], SeVila[Yu et al., 2024], All-in-one[Wang et al., 2023], BLIP-2[Li et al., 2023a], ST-LLM [Liu et al., 2024], TG-Vid [Hu et al., 2024], ATT-4L[Jaiswal et al., 2024]. We also adopt the proposed method on three video SGG methods: Sttran [Cong et al., 2021], Tempura [Nag et al., 2023], and APT [Li et al., 2022b], to evaluate the flexibility of GP-VQA.

Implementation Details: We employ Mask RCNN [He et al., 2017] with a ResNet-101 backbone as the object detector for adopted video scene graph models. For the vision-language model, we adopt the frozen LLAVA-7B and Qianwen2-VL-7B with LoRA. To ensure fairness, we also employed a standard Transformer [Vaswani et al., 2017] to test the performance of the proposed model. During the pretraining stage, we use SGD optimizer with an initial learning rate of 0.001 and decay the learning rate by multiplying it with 0.9 after every epoch. The momentum is set to 0.9 and the size of mini-batch is set to 8. For hyper-parameters, we set the random mask rate λ to 0.08, while the α, β, and γ in prompts generation are set to 0.7, 0.8, and 2 respectively. Moreover, we sample 1 frame in every 3 frames for pre-training. For prompt-tuning, we use the same setting as pre-training, except the initial learning rate is 1e - 5. Moreover, the adopted video scene graph generation models are pre-trained on Action Genome [Ji et al., 2020]. Since there is no scene graph annotation in NExT-QA, we do not pretrain video graph completion on NExT-QA. Instead, we adopt the module pre-trained on Action Genome and AGQA v2 for NExT-QA.

4.1 Comparison with State-of-the-arts

We compare GP-VQA with the state-of-the-art (SOTA) methods on three VideoQA datasets, as shown in Table 1, Table 2, and Table 3. GP-VQA', GP-VQA* and GP-VQA+ are implemented with Transformer, LLAVA-7B and Qianwen2-VL7B, respectively. GP-VQA achieves SOTA performances and outperforms the existing methods on all datasets.

The proposed method with VLM achieves a 6.16% improvement on the AGQA v2 dataset, especially obtaining 4.51% and 7.69% improvements on the relation-action and action recognition types. Since the SOTA method GF [Bai et al., 2024] does not utilize LLM, to ensure fairness, we also evaluate GP-VQA with a standard Transformer, which still achieved a 2.82% improvement over the GF. As shown in Table 2, the performance improvement of the proposed method is lower on NExT-QA because we cannot perform masked video graph completion pre-training on NExT-QA, but it is worth noting that our model still has significant superiority and achieves the best performance. Moreover, it can be observed that GP-VQA exhibits a significant advantage in handling causal-type questions, with a 10.59% improvement compared to SeVila. Since GP-VQA utilizes additional scene graph data from Action Genome [Ji et al., 2020], we implement a variant SiVila* of SiVila to ensure fairness. SiVila* also adopts masked video graph completion pre-training with the additional data. As shown in Table 2, pre-training effectively improved the answer reasoning ability of SiVila*, while the proposed model GP-VQA+ still outperformed SiVila*. Furthermore, GP-VQA obtains obvious improvement on each metric of the STAR dataset, as shown in Table 3. This demonstrates the effectiveness of the proposed masked video graph completion pre-training in enhancing reasoning performance.

4.2 Ablation Study

In this part, we analyze the impacts of the designed pretraining & prompt-tuning strategy, graph prompt generation, and hyper-parameters in our model.

Effect of each component in GP-VQA: We ablate key components in GP-VQA, e.g., Pre-training and Random Mask, denoted as, w/o PT and w/o RM.

• w/o PT: It removes the masked video graph completion pre-training in VideoQA, which we directly adopt video graph generated by frozen video SGG model.
• w/o RM: It removes the RandomMask in masked video graph completion pre-training, which vision-language model server as a scene graph generator in pre-training.

The results of these variants on AGQA v2 and NExT-QA are shown in Tableb 4 (a). We can see that removing masked video graph completion pre-training causes a 4.66% and 4.25% accuracy drop on AGQA v2 and NExT-QA datasets, respectively. It proves that the proposed pre-training strategy can effectively enhance the comprehensive understanding ability of videos, thereby obtaining strong reasoning ability for question answering. In addition, the performance drop on both datasets is significant when removing pre-training, which also demonstrates the learned reasoning ability is universal in VideoQA. Tableb 4 also highlights the importance of random mask strategy, the full model GP-VQA obtains the best results on both datasets, which can be attributed to the random mask operation indeed helps the training of model's reasoning ability. Moreover, as it simulates the form of question answering in the masked video graph completion, the proposed model can more easily adapt to prompt-tuning.

Effect of different strategies for prompts generation: We compare three types of prompts generation strategies: mean pooling, max pooling, and learned layer (GP-VQA), which can be realized by modifying Fv and Fe in Eq. 7. As shown in Table 4 (b), graph prompts achieve the best performance, e.g., 4.13% and 3.33% improvements compared with mean and max pooling on the AGQA v2 dataset. The reason is that a learned layer can effectively capture relevant information in video graph.

Effect of different video SGG models: To evaluate the flexibility of GP-VQA in incorporating different methods for obtaining raw video graphs, we conduct several experiments on different video scene graph generation methods, e.g., Sttran*, Tempura*, and APT(GP-VQA). We conduct different video scene graph generation methods to replace the spatial-temporal semantic extractor in GP-VQA, as shown in Table 4 (c). We can see that all these methods outperform the previous SOTA method SiVila [Yu et al., 2024] and GF [Bai et al., 2024] on both AGQA v2 and NeXT-QA datasets. The improvement of performance further demonstrates the flexibility of the proposed GP-VQA, and verifies the advantage of introducing video graph for extending the proposed model's perception.

Effect of different hyperparameters: We analyze the performance with different random mask rates λ, and hyperparameters α, β and γ in prompts generation. As shown in Figure 3, it can be observed that accuracy first increases and then slowly decreases or level off. Thus, we set λ = 0.08, α = 0.7, β = 0.8 and γ = 2.

5 Conclusion

In this work, we present a video-guided temporal modeling approach for VideoQA, comprising pre-training on masked video graph completion and prompt-tuning for VideoQA. The method uses random masking in pre-training for graph construction, leveraging video scene graphs and a graphcompletion-equipped language model for multimodal reasoning. During prompt-tuning, question graphs are built, messages are passed between video and question graphs to create prompts, and answers are computed using the pre-trained model. This strategy enhances visual scene perception and overcomes text-question limitations. We conduct extensive experiments to show that the proposed method significantly outperforms the SOTA methods, with adaptable components such as the video SGG. Future exploration will focus on semisupervised or unsupervised graph-based prompt-tuning, aiming for practical real-world adaptation.

Acknowledgments

This work was supported by the National Natural Science Foundation of China under Grants (62325206, U23A20387, 62322212), the Key Research and Development Program of Jiangsu Province under Grant BE2023016-4, the CAS Project for Young Scientists in Basic Research (YSBR-116), the Opening Foundation of Artificial Intelligence Key Laboratory of Sichuan Province(2024RYY01), and the Natural Science Research Startup Foundation of Recruiting Talents of Nanjing University of Posts and Telecommunications under Grant NY224066.