Multilevel Hierarchical Network with Multiscale Sampling for Video Question Answering

1 Introduction

With the advancements of deep learning in computer vision and natural language processing [He et al., 2016; Vaswani et al., 2017; Hara et al., 2018], Video Question Answering (VideoQA) has lately received more attention for its wide application in video retrieval, intelligent QA system, and autonomous driving. In comparison with Image Question Answering (ImageQA) [Yang et al., 2016; Gao et al., 2021], VideoQA is more difficult because it needs to properly extract the dynamic interaction between the text and the video in addition to modeling the semantic association between the text and a single image.

The majority of existing methods [Dang et al., 2021; Kim et al., 2019; Huang et al., 2020; Le et al., 2020;

∗Equal contribution.

Park et al., 2021; Gao et al., 2018; Chowdhury et al., 2018] used recurrent neural networks (RNNs) and their variants to connect the embeddings of the text and spatial features extracted with convolutional neural networks (CNNs) of the video, and adopted spatial-temporal attention mechanisms to learn the text-related visual representation [Xu et al., 2017; Jang , 2017 ] or the so called co-attention representation [Li et al., 2019; Zha et al., 2019]. To acquire the long-term interaction between the question and the video, some methods [Cai et al., 2020; Fan et al., 2019] proposed to use extra memory modules to augment the capacity of sequential encoding. While these methods achieved interesting results on benchmark datasets of VideoQA, the multiscale semantic relations existed between the text and appearance-motion information of the video is largely ignored.

For the video example shown in Fig.1 (a), the local attribute of the 'man' is better inferred at a coarser scale, while the semantic information of 'what' and 'jump over' is mostly revealed from consecutive frames at a finer-grained scale. For such a multiscale characteristic, a model should be able to search sufficient information from the video given different question types that rely on the visual clues at different scales. In addition, as illustrated in Fig.1 (b), the multilevel representation learning of a deep learning model covers generalizable information of local objects and global semantics of the input video along the increase of model depth [Krizhevsky et al., 2012]. It remains an open question about how to incorporate the multiscale information of a video with multilevel processing of a deep learning model for VideoQA.

Given the above findings, we propose a novel method named Multilevel Hierarchical Network (MHN) with multiscale sampling for VideoQA, as shown in Fig.2. MHN comprises two modules of Recurrent Multimodal Interaction (RMI) and Parallel Visual Reasoning (PVR). To leverage the multiscale visual information, we first apply a multiscale sampling to acquire several frame groups from the input video. To accommodate the frame groups at different scales, the RMI module uses a recurrent structure to bridge the multimodal interaction blocks, where each block takes the frame group at a scale as an input. In the later section, we empirically analyze the impact of the ways of matching the scales of frame groups with the levels of different multimodal interaction blocks within this module. Each multimodal interaction block extracts the question-guided visual representation per scale, and the recurrent structure provides the representations across different levels. The PVR module takes this output at each level for visual reasoning, where a transformer encoder is shared during the parallel processing. In this way, our method fits with different question types, where their answering could benefit from the visual clues at relevant levels.

Our contributions are as follows: 1) We propose a novel Multilevel Hierarchical Network (MHN) with multiscale sampling for VideoQA, to incorporate the multiscale interaction between the text and the video with the multilevel processing capability of a deep learning model; 2) We design a recurrent multimodal interaction module to enable the multimodal multilevel interaction between the two input modalities, and a parallel visual reasoning module to infer the visual clues per each level; 3) We conduct comprehensive evaluations on TGIF-QA, MSRVTT-QA, and MSVD-QA datasets, achieving improved performances than previous state-of-thearts and verifying the validity of each part of our method.

2 RelatedWork

VideoQA challenges a model on analyzing the complex interaction between the text and visual appearance-motion information. [Xu et al., 2017] proposed a method based on Gradually Refined Attention to extract the appearancemotion features using the question as guidance. [Jang et al., 2017] proposed a dual-LSTM approach together with spatio-temporal attention to extract visual features. Later on, other spatio-temporal attention-based methods proposed to use co-attention representation [Li et al., 2019; Zha et al., 2019], hierarchical attention network [Liu et al., 2021], and memory-augmented co-attention models [Cai et al., 2020; Fan et al., 2019] for the extraction of motion-appearance features and question-related interactions. Recently, [Kim et al., 2019] proposed a multistep progressive attention model to prune out irrelevant temporal segments, and a memory network to progressively update the cues to answer. Additionally, some proposed to leverage object detection across the video frames to acquire fine-grained appearance-question interactions [Dang et al., 2021; Huang et al., 2020]. [Le et al., 2020] proposed to use a hierarchical structure for the extraction of question-video interactions from the frame-level and segment-level. [Park et al., 2021] proposed a heterogeneous multimodal graph structure using the question graph as an intermediate bridge to extract the internal semantic relation between each word and the video. Although interesting results are achieved by these methods, the multiscale structure of the answering cues existed in the visual information is not well explored.

Transformer [Vaswani et al., 2017] has achieved outstanding performance using its self-attention mechanism and forward-passing architecture, which is first introduced for neural machine translation tasks. Given its promising efficiency in analyzing temporal information, transformer becomes one of the dominant approaches for various NLP tasks [Wolf et al., 2020]. Recent efforts are seen in transferring transformer to the computer vision domain, where improved performances are achieved in object detection [Dai et al., 2021], instance segmentation [Liang et al., 2020], and action recognition [Girdhar et al., 2019]. In this paper, we propose to use a shared transformer encoder to extract the semantic interaction between the question and the video at different levels in parallel for visual inference.

3 Method

Given a video V and the question Q, VideoQA aims to acquire the correct answer ˆa. For open-ended and multi-choice types of question, the answer space A comprises the group of pre-defined answers and list of candidate answering options, respectively. Generally, VideoQA is formulated as follows.

where θ represents the group of trainable parameters of the modeling function f .

As shown in the overview of our proposed MHN model in Fig.2, we first extract multiscale appearance-motion features from the input video, and embeddings from the input question. Thereon, the proposed Recurrent Multimodal Interaction (RMI) module propagates the information across its multimodal interaction blocks in a recurrent manner, providing the semantic representations at different levels. Note, in this section, we assume that these blocks from the low-level to the high-level accommodate the visual information from coarser scale (fewer frames) to finer-grained scale (richer frames) accordingly. Thus, the scale and level n are the same in this case. With such multilevel representations, the proposed Parallel Visual Reasoning (PVR) module uses a shared transformer block to establish the visual inference at each level and produces the final visual cues after a fusion operation. Finally, a classification or regression operation is done in the decoder for answering. It is worth mentioning that, similar to the methods we compare with in this paper, our method does not rely on large-scale pre-training and big models to achieve the improved performances.

3.1 Multiscale Sampling and Feature Extraction

Multiscale Sampling. Different from some previous VideoQA works [Dang et al., 2021; Fan et al., 2019; Le et al., 2020; Park et al., 2021] that adopted dense sampling for the input video, we conduct a multiscale sampling to help acquire visual features at different temporal scales. For input video V, at scale n ∈ {1, ..., N}, we sample T ×2n-1 frames along the forward temporal direction, with T as the size of our sampling window, which is set to 16 in our experiment. At this scale, for a clip comprising T frames, the group of such clips Cn is represented as

where cn,i is the i-th video clip sampled at scale n. Given the same number of video clips sampled from the video, multiscale sampling provides richer visual information than dense sampling, covering more aspects from the local object to the global interaction or event, which help the network to better exert its multilevel processing capacity on understanding the semantic relations between the question and the video.

Visual Representation. Following [Li et al., 2019; Dang et al., 2021; Cai et al., 2020; Fan et al., 2019; Huang et al., 2020; Le et al., 2020; Park et al., 2021], we use ResNet [He et al., 2016] and 3D ResNet152 [Hara et al., 2018] to extract the appearance and motion features, respectively, from the video. Therein, the feature output at the last pooling layer of each network is used. For the group of video clips Cn at scale n, the extracted frame-wise appearance feature Vn can be represented as

Similarly, we extract the clip-wise motion feature Mn as

We use a linear feature transformation layer to map feature vectors in Vn and Mn into a d-dimensional feature space, where we now have vn,i ,mn,t ∈ Rd . After a feature concatenation following the temporal order of video clips, the appearance-motion feature Xn at n-th scale is represented as

where xnj ∈ {vn,i ,mn,t }, with the total number of appearance-motion features at scale n being LnX = 2n-1 (T + 1). Meanwhile, such feature is added with the positional embedding Pn ∈ RLnX ×d , in order to maintain the positional information of the feature sequence.

Linguistic Representation. For the multi-choice question and answer candidates, we adopt Glove word embedding method [Pennington et al., 2014] to acquire the 300dimensional feature embeddings, which is further mapped into a d-dimensional space using linear transformation layers. Thereon, a bidirectional LSTM network is adopted to extract the contextual semantics between each word in the question, and the answer, respectively. Finally, the acquired representation for the question or answer candidates is obtained by concatenating the hidden states of the last forward and backward LSTM layer per timestep, written as

and

where LQ and LkA are the number of words in the question and k-th answer candidate, respectively.

3.2 Recurrent Multimodal Interaction

Within the RMI module, the recurrent connections of multimodal interaction blocks facilitate the extraction of multilevel semantic relations between the question and the video. In a block that processes information at scale n, with the input visual feature Xnin and question feature Qnin , the interim interaction output ˜Xn is computed as

where MCA(·) denotes the operation in a multi-head crossmodal attention layer, and at a single attention head h its output is

where Fhq = LN(Xnin )Wh q is the Query, Fhk = LN(Qnin )Wh k is the Key, and Fhv = LN(Qnin )Wh v is the Value, with LN(·) being the layer normalization [Xiong et al., 2020]. We concatenate the output per head to obtain

MCA(Xnin,Qnin) = concat(MCA1 ,MCA2 , ...,MCAH ), (10) where Wh q , Wh k , Wh v ∈ Rd×d/H , and Wo ∈ Rd×d are the learnable weight matrices, and H is the total number of attention heads. In short, within the MCA, the question semantics and appearance-motion features are connected, while the semantic co-occurrence of them is extracted using the attention mechanism.

Given ˜Xn and Qnin , a feed forward layer is further added to each modality to acquire the final interaction feature outputs ˆXn ∈ RLnX ×d and ˆQn ∈ RLQ×d as

and

where fX(·) and fQ(·) represent the operation in a feed forward layer, which comprises two linear projections separated by a GELU non-linearity. The feature dimension d stays unchanged.

We design a recurrent connection to connect multimodal interaction blocks at different levels for multilevel processing, which also handles the difference in temporal dimension of the feature at each level. Given the interaction feature output ˆXn-1 of the previous block and current appearance-motion feature input Xn , the recurrent connection (shown in the area marked by blue dashed contour in Fig.2) uses an attentional dimension alignment to provide the input Xnin for the current block as

where Wn-1 1 and Wn-1 2 are learnable weight matrices. In addition, we have Qnin = ˆQn-1 . The extra bypass scheme within the connection ensures that the current block receives the higher-level information. For the whole module, we repeat Equation 13 and Equation 8-12 to propagate the visionquestion interaction recurrently across different levels.

3.3 Parallel Visual Reasoning

An essential step of VideoQA is to infer the visual cues from the appearance-motion feature via understanding the question semantics. Given vision-question outputs at different levels from the RMI module, the proposed PVR module first infer visual cues at each level and acquire the final feature for answering under the guidance of question semantics.

Given the interaction feature output ˆXn at scale n (at the highest level) of RMI module, we use the encoder layer proposed in transformer [Vaswani et al., 2017] to acquire the visual cue Rn as

with

where f(·) denotes the operation in a feed forward layer. For the outputs from RMI module at different scales, PVR module repeats Equation 14 and 15 with the shared processing layers. This help to maintain the consistency of the semantic space during visual inference, and a compact learnable weight even if the number of scales increases.

Given the high-level question feature ˆQn output by RMI module, PVR module further fuses the multilevel visual cues to acquire the final feature for answering as

with

where ¯Qn , ¯Rn are acquired by applying average pooling along the temporal dimension on ˆQn ,Rn , respectively, and the final feature for answering is O ∈ Rd .

3.4 Answer Decoder and Loss Function

Following [Jang et al., 2017; Li et al., 2019; Dang et al., 2021; Cai et al., 2020; Fan et al., 2019; Huang et al., 2020; Le et al., 2020; Park et al., 2021; Gao et al., 2018; Chowdhury et al., 2018], different decoding strategies are used according to the types of question.

Specifically, we treat an open-ended question as a multiclass classification task, where the answer decoder aims to predict the correct category from the answer space A. Given the final feature O, the probability vector P ∈ R|A| towards each class is computed as

whereWo , Wy , bo , and by are the learnable weight matrices and biases of each layer, respectively; δ(·) is the activation function. The cross-entropy loss is used here.

For the repetition count task, linear regression is used to replace the classification function shown in Equation 21, the output of which is processed by a rounding function to acquire the integer output. The loss function used here is the Mean Squared Error (MSE).

For multi-choice questions, we use an answer candidate Ak as input to the MHN model, similar with the question Q. Therein, the learnable parameters are shared for the processing of the answer and the question. Given the final feature outputs O conditioned by the question and Ok a conditioned by the k-th answer candidate, the predicted probability towards the k-th answer candidate is computed as

The answer candidate that produces the highest probability p is selected as the predicted for the question. Hinge loss [Gentile and Warmuth, 1998], namely max(0, 1 + pi - pc ), is adopted to compute the loss between the correct answer pc and the incorrect answer pi .

4 Experiment

4.1 Datasets

Three VideoQA benchmarks are adopted for our evaluation. TGIF-QA [Jang et al., 2017] is a large-scale dataset for videoQA, which comprises 165K question-answer pairs and 72K animated GIFs. This dataset has four task types, including Action, Transition (Trans.), FrameQA, and Count. Action is a multi-choice task aimed to identify the repetitive actions. Trans. is another multi-choice task for identifying the transition actions before or after a target action. FrameQA is an open-ended task where the answer could be inferred from a single frame of the video (GIF file). Count is to count the number of a repetitive action.

MSVD-QA [Xu et al., 2017] comprises 1,970 short clips and 50,505 question-answer pairs, which are divided into five question categories of what, who, how, when, and where. All of them are open-ended.

MSRVTT-QA [Xu et al., 2017] comprises 10K videos and 243K question-answer pairs. The question types are similar to what included in the MSVD-QA dataset. However, the scenario of the video is more complex, with a longer duration of 10-30 seconds.

4.2 Implementation Details

Metrics

For multi-choice and open-ended tasks, we use accuracy to evaluate the performance. For the Count task in TGIF-QA dataset, we evaluate with Mean Squared Error (MSE) between the predicted answer and the ground truth.

Training Details

We use the official split of training, validation, and testing sets of each dataset. By default, the maximum scale N is set to 3, and the visual features are input to the model with increasing scales. For each multimodal interaction block in RMI module, the feature dimension d is set to 512, and the number of attentional heads H is set to 8. The number of mini batch size is set to 32, with a maximum number of epochs set to 20. The Adam [Kingma and Ba, 2015] optimizer is used, with the initial learning rate set to 1e-4, which reduces by half when the loss stops decreasing after every 10 epochs. We implement the method with PyTorch deep learning library on a PC with two GTX 1080 Ti GPUs.

4.3 Comparison with the State-of-the-arts

On the TGIF-QA dataset, we compare with a series of stateof-the-art VideoQA methods. As shown in Table 1, our MHN model outperforms other state-of-the-art methods across all four tasks. Our improvements are more obvious on Action, Trans., and Count tasks, where the answering requires visual inference at different temporal scales and processing levels of the model. These show the advantage of the multiscale multilevel processing capacity of our model. In addition, most methods use dense sampling for the input video, e.g., HCRN [Le et al., 2020] and Bridge2Answer [Park et al., 2021] sampled 8 clips each comprising 16 frames, while our method with scale N set to 3 only samples 7 clips so that costing less computational loads.

Further comparisons on the MSVD-QA and MSRVTT-QA datasets are conducted. Results are reported in Table 2. On such more challenging data, our MHN model still achieves the best performances of 40.4% and 38.6% on both datasets, respectively. While Bridge2Answer [Park et al., 2021] additionally extracted semantic dependencies from the question using a NLP tool and HOSTR [Dang et al., 2021] applied Fast R-CNN for object detection per frame, our model is able to produce even higher performances without such complex feature pre-processing.

4.4 Ablation Study

Here, we run several ablation experiments on the TGIF-QA dataset for in-depth analysis of our method. We adopt the default MHN model used above as the baseline.

Multiscale Information

We first replace the multiscale input with the input at a single temporal scale. Specifically, the multimodal interaction blocks within the RMI module all take the same appearancemotion feature Xn at a single scale, with n = 1, 2, 3 separately. Additionally, for the multiscale input, our default model takes the inputs at scales from n = 1 to n = 3. We further change such an input order to consider the inputs at scales of n = 1, 3, 2, n = 3, 2, 1, and n = 3, 1, 2. Results are reported in Table 3. We can see that, when the information is provided at a single scale, model performances reduced across all the tasks. Therein, by providing richer frames at higher scales, the Action, Trans, and Count tasks are improved, suggesting their dependencies on richer temporal information. By contrast, while the FrameQA task relies on the visual inference at a single frame, richer frames lead to reduced performances. Furthermore, the change of orders in the multiscale information input does not impact the model performance noticeably, suggesting the incorporation of multiscale information with multilevel model processing does not depend on a rigid matching of orders.

Multilevel Processing and Parallel Reasoning

Here, as the first variant, we remove recurrent connections between blocks in the RMI module. That is, each block only receive the appearance-motion feature at a scale as input without the information passed from the previous block, thus the multilevel processing is no longer enabled. Thereon, the PVR module still receives the output from each block. For the second variant, we provide the PVR module only the output of the last block of RMI module. Results are reported in Table 4. As we disable the multilevel processing by removing the recurrent connections, model performances decrease across the four tasks. Even with multilevel processing, when we only provide the PVR module with the high-level information, model performances improve a bit but are still lower than our default model. These results show the importance of incorporating multilevel processing with multiscale visual inputs, with our proposed HMN model being a promising implementation for such end.

From another perspective, we further consider a variant of MHN model that uses separate transformer encoders for the PVR module instead of using a single encoder with weight sharing. While the number of trainable parameters increases from 16.5M (default model) to 22.8M thereon, the model achieves even better performances on Action, Trans., and Count tasks as seen in Table 4. In comparison, for methods that have published their codes, PSAC [Li et al., 2019] (39.1M), HME [Fan et al., 2019] (44.8M), HGA [Jiang and Han, 2020] (104.1M), L-GCN [Huang et al., 2020] (30.4M), and HCRN [Le et al., 2020] (42.9M) are all bigger than our model that without weight sharing, but performed worse.

The Number of Scales

The maximum scale N sets the scope of multiscale sampling, as well as the depth of vision-question interactions within our MHN model. Here, we analyze the impact of setting different values of N on model performance. Since the increase of N by 1 would double the GPU memory consumption, we only experiment with N ∈ {2, 3, 4}. We still use an ascent order of the multiscale information to make the input for our MHN model, i.e., n = 1, 2, 3 for N = 3. Results are reported in Table 5. The reactions of performances of different tasks toward the increase of scale N are different. The performances on the FrameQA task improve with larger N, showing the robustness of our model for visual inference at a single frame even if more frames are provided. The richer frames also contribute to the Count task. For tasks of Action and Trans., where the informative visual cues exist at specific scales of the video, the scale N is better set to reach a balance between the amount of information provided and model performance. For our MHN model on TGIF-QA dataset, we reach such a balance at N = 3. We also find that, with N = 2 where only 3 clips are sampled, our model is able to outperform most state-of-the-art methods reported in Table 1.

5 Conclusion

This paper presented a novel Multilevel Hierarchical Network (MHN) with multiscale sampling for accurate VideoQA. In general, MHN enables the incorporation of the multiscale visual information with the multilevel processing capacity of deep learning. Specifically, we designed a Recurrent Multimodal Interaction (RMI) module to use the recurrentlyconnected multimodal interaction blocks to accommodate the interaction between the visual information and the question across temporal scales. We designed another Parallel Visual Reasoning (PVR) module to adopt a shared transformer encoder layer to process and fuse the multilevel output of RMI for final visual inference. Our extensive experiments conducted on three VideoQA benchmark datasets demonstrated improved performances of our MHN model than previous state-of-the-arts. Our ablation study verified the importance of multiscale visual information for videoQA, and the efficiency and effectiveness of our method on leveraging it.

Acknowledgments

This work is funded by the National Natural Science Foundation of China (62106247). Chongyang Wang is supported by the UCL Overseas Research Scholarship (ORS) and Graduate Research Scholarship (GRS). Yuan Gao is partially supported by the National Key R&D Program of China (2020YFB1313300).