Feature Augmented Memory with Global Attention Network for VideoQA

1 Introduction

Video question answering (VideoQA) aims at performing high-order reasoning over spatial and temporal contents of videos, and inferring the correct answer for the given videorelated question delineated by natural language. It is challenging since videos and questions are composed of sequences of frames and words, respectively, and thus capturing both precise local neighborhood and long-range dependencies in these sequences is of central importance.

Recently, most VideoQA works are mainly based on the recurrent neural networks (RNNs) or memory augmented RNNs. Some other works replace RNNs with self-attention

∗Contact Author

(SA) modules. Although recurrent operations [T et al., 2010] and self-attention mechanism [Ashish et al., 2017; Wang et al., 2018] are widely used to model sequential data in language and videos, they both have their own weaknesses. On one hand, RNNs are not good at modeling long-range dependencies and often fail to adapt to sequence lengths that are not encountered in training. On the other hand, although SA modules [Ashish et al., 2017] compute the response at each position in a sequence by attending to all the rest positions to capture the long-range dependencies, it can not precisely model the dependency of local neighborhood, and thus lead to insufficient modeling of the temporal relationship in a sequence.

To overcome the above shortcomings of RNNs and SA modules, we propose an effective video question answering model that is built upon both recurrent neural networks and self-attention modules. Specifically, in our model, feature augmented memory is responsible for modeling the finegrained multi-modal interactions and self-attention modules are used to capture the global contexts of the sequence respectively, such as shown in Figure 1. Moreover, we further propose a hierarchical cross-modal mutual-augmented method to fuse information from both visual and textual modal at multiple levels. In the feature-augmented memory, textual features and visual features could augment each other via memory mechanism, and thus improving memory capacity. In the SA modules, the global sentence-level features of the question and segment-level features of the video are used to augment each other via co-attention. In this way, textual and visual features are fused at multiple levels to achieve both coarse and fine-grained feature fusion.

To summarize, our main contributions are as follows:

1. We propose a simple but effective VideoQA model that is built upon recurrent networks and self-attention modules. The proposed model can effectively model both the precise local neighborhood and long-range dependencies of the video and question sequences.
2. We propose a cross-modal feature augmentation memory to memorize efficiently without redundant information.
3. We conduct extensive experiments on VideoQA benchmarks. Experimental results demonstrate that our approach achieves the new state-of-the-art performance.

2 RelatedWork

There is growing interest in the tasks of Video question answering (VideoQA), and many memory mechanisms have been proposed which allow the model to conduct multiple reasoning iterations and refine the answer gradually. [Jang et al., 2017] and [Agrawal et al., 2015] proposed to represent video frames by both appearance (e.g., [Simonyan and Zisserman, 2015] ) and motion features (e.g., C3D [Tran et al., 2015]). [Jang et al., 2017] proposed spatio-temporal reasoning approach with an attention mechanism on two dimensions of video features, and then feed it to the video encoder. However, they concatenated two features straightforward in an early stage, which leads to suboptimal results. [Zhao et al., 2018] proposed the hierarchical encoder network to learn the representation of long-term video contents according to the question. Although they hierarchically model the video representation, they didn't fully exploit the textual feature, hence lack the capability to capture video-related semantics of question. [Xu et al., 2017] proposed to use question as guidance which gradually refines its attention over the appearance and motion features of the video. However, appearance and motion features in video play different roles in video understanding, this method didn't consider the interaction between appearance feature and motion feature in video, and hence they cannot learn from each other, which may not be expressive enough to reveal the information held by the video. [Fan et al., 2019] propose a video memory to integrate appearance and motion features and a question memory which integrate question encoder and question memory. However, the video memory in HME[Fan et al., 2019] only learns information from itself, which may memorizes lots of question-irrelevant information without the guidance of question. Thus may lead to inefficient memorization of video information which is irrelevant to the question, especially for long video with abundant content.

Attention module allows the model focusing on important data by assigning different weights to different data. In recent years, attention mechanism [Xu et al., 2015] has been widely used in lots of research topics. [Gao et al., 2018] proposed to replace the early fusion with a more sophisticated co-memory attention mechanism. They used one type of feature to attend to the other and fused the final representations of these two feature types at the final stage. However, this method doesn't synchronize the attentions detected by appearance and motion features, thus could generate incorrect attentions. Meanwhile, this method will also miss the attention which can be inferred by the combined appearance and motion features, but not individual ones. To address these challenging problems, we propose a new visual memory to co-learn spatio-temporal features under the guidance of text, and a text memory to memorize question contents with textual guidance. A nonlocal module is appended to the memory output to reinforce the global semantic reasoning.

3 Method

In this section, we illustrate our network architecture for VideoQA. Firstly, we introduce the LSTM encoder for appearance video feature, motion video feature and word embeddings (highlighted in blue, orange and green in 2). Secondly, we elaborate on the design of our feature-augmented memory to memorize video and text information with the guidance of the other modality. Thirdly, we introduce nonlocal module which can effectively learn global semantics from memory output. Finally, we demonstrate how we fuse the attended video and attended text features and how to implement the final answer generation for specific task.

3.1 Feature Extraction

Following previous works[Gao et al., 2018], we sampled a fixed number of frames for all videos (e.g., 35 for TGIFQA), then applied C3D [Tran et al., 2015] network to extract motion features which are denoted as F m = [f m 1 , f m 2 , ..., f m N ], and use pretrained ResNet [He et al., 2016] or VGG [Simonyan and Zisserman, 2015] network on video frames to extract video appearance features denoted as F a = [f a 1 , f a 2 , ..., f aN ]. N is number of frames sampled in the video. Superscript a and m in our paper indicates appearance and motion respectively. We separately fed the raw CNN appearance features F a into appearance encoder LSTM and motion features F m into motion encoder LSTM (highlighted in blue in Fig. 2). The output of two encoders are I m = [im1 , . . . , imN ] and I a = [ia1 , . . . , iaN ]. We use the embedding layer to transform the word to its semantic embedding qt and initialized with the pre-trained GloVe 300-D feature. We denote question embedding as Q = [q1, q2, ..., qT ], in which T is the number of words in the question. LSTM is used to process the question embedding Q, and the output of LSTM is I q = [iq1 , . . . , iqN ].

3.2 Fine-grained Feature AugmentedMemory

Both appearance and motion features contain lots of contents in video. However, not all of these information is essential for answering the questions. We hope our memory could only memorize question-relevant video feature at each time step, while retaining the ability to enhance the spatial temporal inference. Therefore we design a text-augmented visual memory to memorize spatio-temporal information in video with the guidance of textual features. Thus, only the video information that is most relevant to the question is stored without redundant content. Similarly, a video-augmented textual memory is used to refine and memorize text representations under the guidance of appearance and motion features.

Text-augmented Visual Memory. Our video memory consists of S memory states M = [m1, m2, ..., mS] and three hidden states ha , hm and hv , where S is number of memory states, hm/a represent the hidden state of motion and appearance contents, and hv is the global hidden state to integrate motion and appearance information, since both of them can learn from and augment each other. Firstly, we perform read operation from last time step memory state Mt-1. The read content rt is the weighted sum of each memory state:

where · is inner product, βt ∈ RS , Mt-1 ∈ RS×d , We ∈ R3d×S is learnable weight. βt is the weight determined by three hidden state from last time step. d is 512 in our experiments. FC means fully-connected layer, tanh is used as the non-linear activation. Then, the appearance and motion hidden states are updated from the last time step ht-1 by appending what we read from the memory state just now, the current input imt ,iat as well as the question content iq =WqI q :

Here hm/a t-1 denotes hmt-1 and hat-1 respectively. The global hidden state is updated in order to integrate motion and appearance contents.hvt = σ(FC [rt, hvt-1 , iq ]) (4)

Question features iq is used to update the hidden states which is responsible for storing the current memory at this time step under the direction of question words. To sum up, the textaugmented visual memory can memorize important parts of the video contents which most relevant to the question by including question guidance in hidden state updating operations at each time step.

The textual aware hidden state hm/a t-1 and hvt-1 will be transformed to the content vector ct. The content vector ct will be used to compute write weight at t-th time step:

Where W are learnable parameters, and b are the bias.

The write weight for appearance and motion feature αa/m t are computed from ca/m t to update memory state.

when update the memory state, we should also consider the weight of the retained information from last time step. Thus we compute the weight µ:

Finally, the memory state is updated by the hidden states which incorporate the information from both appearance and motion channels of the video under the guidance of the words of questions:

where µ determines how much information remains from last time step and α determines which of motion or appearance feature from current inputs should the memory pay more attention to. During this process, information could pass to the memory state Mt. Up to this point, the memory's computation flow at one step has finished.

Video-augmented Textual Memory. Here we briefly introduce the textual memory, where motion and appearance video features are integrated to generate and refine the memorization over textual features at each time step. The structure of textual memory is basically same with visual memory, except that only one hidden state hq is necessary for memory. Our textual memory consists of S memory states M q = [m1,m2, ...,mS]. The augmented information from video is ia = WaI a , and ia = WqI a . We read content from memory state with the same way in video memory.

where · is inner product, βq t ∈ RS , M q ∈ RS×d , Ve ∈ R2d×S is learnable weight. βq t are textual memory read weights. The hidden state is updated by:

where σ is used as the non-linear activation. The content vector which will be write into the memory state at t-th time step is computed as:

The write weights to all memory sates is depend on the current content vector and the retained hidden state from last time-step:

Finally each memory state mi is updated by

3.3 Non-local Neural Network

The hidden state of the text-augmented visual memory from each step hvt will be concatenated to a matrix Vid as output, and the hidden state of video-augmented textual memory from each step hqt will be concatenated to a matrix Tex, where Vid ∈ RM×ρ and Tex ∈ RN×ρ .Then we put the two matrices into the non-local neural network. The equation of Scaled Dot-Product Attention is:

Q,K,V are regarded as query, key and value respectively. Firstly, we conduct Question-based self attention block.

The goal of Question-based Self-Attention Block is to extract the semantic long range dependencies of the attended question feature from video augmented textual memory. Following the work in [Ashish et al., 2017], we use Scaled DotProduct Attention as our non-local block to better exploit the global dependencies of the attended feature output from memory network.

Video-based self attention block. The goal of Videobased Self-Attention Block is to extract the semantic long range dependencies of the attended question feature from text augmented visual memory. following

Finally, we demonstrate the Video-to-Text attention (V2T) aiming to locate which question vectors are most relevant to each self-attended video features: for two multichannel inputs, we compute a weight matrix S by S = Vid × T ⊤ ex which ∈ RM×N Then the video-to-text attention is computed by:

and similarly, Text-to-video attention(T2V) is obtained by:

We use the Co-attention method in [Gao et al., 2018] to yield the final feature for answer prediction:

3.4 Answer Generation

Following previous works on VideoQA, we optimize our model using hinge loss for multiple-choice task and crossentropy loss for open-ended task.

Multiple-choice task requires to choose one correct answer among M candidates. The output of on-local block is fed into a linear layer to predict the scores of each candidate answer: p = W ⊤ 0 O. Suppose all candidate answers s = {sp , sn1 , sn2 , . . . , snM-1 }. sp is the predicted score of correct answer, and sni is the scores of incorrect answers. The hinge loss function is presented as follows:

Open-ended task requires to choose one correct answer from a pre-defined answer set of size K, and usually K is very large. We append a softmax layer to the predicted answer scores, which converts scores to probabilities: p = sof tmax(W ⊤ opO+bp). Then, we minimize the cross-entropy (CE) loss function: Lce = - ∑Ki=1 I{i == c}log(pi) where c is the index of correct answer, and I denotes the indicator function.

4 Experiment

We evaluate our model on three benchmark VideoQA dataset TGIF-QA, MSVD-QA and MSRVTT-QA respectively, and compare our results with state-of-the-art methods.

4.1 Dataset Descriptions

TGIF-QADataset [Jang et al., 2017] has 165KQApairs associated with 72K GIF images based on the TGIF dataset[Li et al., 2016]. It contains four specific VideoQA tasks: Count, Action, Trans and FrameQA task. MSVD-QA and MSRVTT-QA Dataset were proposed by [Xu et al., 2017] based on MSVD [L and B, 2011] and MSVTT [Xu et al., 2016] video sets respectively. Both of them are open-ended task with pre-defined answer sets of size1000. There are five different question types in two datasets including what, who, how, when and where.

In our experiments, the optimization algorithm is Adam. The batch size is set as 32. The train epoch is set as 30. In addition, gradient clipping, weight normalization are employed in training.

4.2 Ablation Study

Our proposed framework consists of multiple major components including two feature augmented memories and selfattention module connected to each memory. In order to evaluate the contribution of each component to the final performance, we conduct several ablation studies on the TGIF-QA dataset. Experimental results are shown in Tab. 1.

The Role of Feature-augmented Memory

In our approach, the size of memory state is an essential hyper-parameter. Figures 5 (a) illustrates the performance of text-augmented visual memory. We investigate the effect of text feature augmentation on our method by varying the dimension of memory state in video memory from 20 to 100. Mv means the original video memory without text information while Mt2v means the text augmented visual memory that integrate textual information. Similarly, Figures 5 (b) illustrates the performance of video-augmented textual memory. Mt shows the original video memory performance without text augmentation while Mv2t is video-augmented textual memory which appearance and motion features are guidance to memorize textual information. We observe that when textual features are integrated into the memory, the value of video memory size to get the minimal loss increase from 35 to 50, and the textual memory size to get the best performance rise from 20 to 30. The fact shows that when memorizing appearance and motion features of the video, the augmentation from text feature could enhance the memory ability for the model, and vice versa. To sum up, the integration of extra information in the memory lead to a better performance as well as increase the threshold value of the memory.

The Role of Memory and Self-attention

The first block in Tab. 1 shows the effectiveness of RNNbased memory. To exploit the effect of feature augmentation for memory, we denote our original video and text memory network without feature augmentation by Mv+Mt. We denote text-augmented video-based network as Mt2v and video-augmented textual memory network as Mv2t. Results show that integrating textual feature into motion and appearance video features could improve the performance, and vice versa. The second block in Tab. 1 demonstrate the effectiveness of attention mechanism. We denote the self-attention mechanism following visual and textual memory as Attv and Attt separately. Attv2t is video-to-text self-attention as illustrated in methods. Both visual self-attention and textual self-attention improved the performance of the module. The complete model contains both feature augmented memory Mv2t+v2t with each followed a self-attention and are fused with co-attention in [Gao et al., 2018]. The positive impact of non-local methods further proves that a video question answering model could gain benefit by fully exploring the global semantics in sequence after memory operation. Compared with Action, Trans, and Count task, FrameQA task achieve better performance on Attt+v, one possible reason is that FrameQA only needs one of the frames to answer the question, hence global semantics content in video is less important. The ablation study in Tab. 1 shows the beneficial effects of our feature augmented memory module followed with self attention mechanism, which demonstrate the effectiveness of the RNN based cascade self-attention architecture.

4.3 Comparison with the State-of-the-art

TGIF-QA result. There are two versions of TGIF-QA, image-based methods and video-based methods. Experiment results in [Li et al., 2016] demonstrate that video-based methods outperforms image-based methods. Our method belongs to the second version.We compare our method with state-ofthe-arts on the TGIF-QA dataset, the experiment results of four tasks (Count, Action, Trans., FrameQA) are in Tab. 2. Firstly, we compare our method with state-of-the-art RNN based methods: ST-VQA [Jang et al., 2017] , STA [Gao et al., 2019], Co-M [Gao et al., 2018] and HME [Fan et al., 2019] and list the reported accuracy in the original paper. Then, we compare our methods with state-of-the-art self-attention based method: Beyond-RNN [Li et al., 2019]. Our method significantly outperforms the other two methods, achieves the lowest average L2 loss in Count task (column 1) , we also achieve the highest accuracy in Action, Trans and FrameQA task (column2, 3, 4) among all five methods.

MSVD-QA result and MSRVTT-QA result. Tab. 3 and Tab. 4 summarize the experiment results on MSVD-QA and MSRVTT-QA dataset. We note that there is high class imbalance in both training and test sets, as more than 95% questions are who and what while less than 5% are how, when and where. We list the numbers of their test instances in the table for reference. We compare our model with the E-VQA, E-SA, E-MN and AMU [Xu et al., 2017], Com [Gao et al., 2018] and HME [Fan et al., 2019]. Our model achieves the state-of-the-art on both datasets. Remarkably, our model performs particularly well on question types "what" and "who".

5 Conclusion

In this paper, we take advantage of both memory and selfattentions mechanism and proposed an effective VideoQA model. Two multi-modal memory namely text-augmented visual memory and video-augmented textual memory respectively. The feature augmented memory could memorize the information under the guidance of other modality, which could make the most of the memorization of the memory block. The memory is followed by a attention-based block which can effectively learn global context aware information from video and sentence, as well as fuse them in a co-attention method. Experimental results demonstrate our approach achieves state-of-the-art performance on VideoQA benchmark datasets TGIF-QA, and achieves the best all performance on MSVD-QA and MSRVTT-QA datasets. Our module achieves better performance compared with both RNNs-based methods and self-attention based methods.

Acknowledgments

This work was supported by NSFC project Grant No. U1833101, SZSTI under Grant No. JCYJ20190809172201639, and the Joint Research Center of Tencent and Tsinghua.