Densely Connected Attention Flow for Visual Question Answering

1 Introduction

Recently, the Visual Question Answering (VQA) task has gained increasing attention in both computer vision and natural language processing communities. It aims at answering a natural language question about a given image. The progress of VQA has been mainly brought about by two lines of works, the development of better attention mechanisms and the improvement in multi-modality fusion approaches.

Attention mechanisms have been widely used in VQA, and a number of methods have been developed so far. These methods are categorized into two classes. One is the class of methods that use the question as guidance to generate attention on image regions (i.e. question-guided attention). For instance, Yang et al. [Yang et al., 2016] proposed stacked attention network that produces multiple attention maps on the image sequentially. Kim et al. [Kim et al., 2016] extended this idea by introducing residual learning to produce better attention. The other is the class of methods that additionally consider image-guided attention on question words (i.e. coattention) [Lu et al., 2016; Nguyen and Okatani, 2018]. Coattention mechanisms can model bidrectional inter-modality information flow, thus perform better than question-guided attention mechanisms. In particular, Nguyen et al. [Nguyen and Okatani, 2018] proposed a novel co-attention mechanism that can model interactions between any image region and any question word, achieving state-of-the-art performance on VQA 1.0 and VQA 2.0 datasets.

∗ Jing Liu is the corresponding author.

Meanwhile, multimodal fusion approaches have been explored extensively. In early studies, simple fusion methods such as the element-wise add, product, and concatenation of the visual and language features are employed. To capture high-level interactions between the two modalities, some bilinear pooling methods, including MCB [Fukui et al., 2016], MLB [Kim et al., 2017], MUTAN [Ben-Younes et al., 2017] and MFB [Yu et al., 2017], were proposed. In [Nguyen and Okatani, 2018], the authors proposed Dense Co-attention Network (DCN), which fuses multi-modal features by multiple applications of the symmetric co-attention.

We point out that the existing VQA approaches only perform a limited amount of interactions between language and vision domains. Some methods [Fukui et al., 2016; Kim et al., 2017; Yu et al., 2017; Anderson et al., 2018] perform only one interaction (i.e. multimodal fusion) in the latter model stage, and the fused features are then fed into the classifier to obtain the scores of candidate answers. The other methods [Yang et al., 2016; Nguyen and Okatani, 2018; Gao et al., 2018] perform multi-step interactions by stacking several interaction modules. However, such a design of stacking multiple modules may impair gradient flow and feature propagation, making networks harder to optimize. Consequently, at most four stacked modules are employed in these methods, which perform at most four interactions between the two modalities. We argue that this can be a significant limitation of the existing approaches. Limited interactions possibly fail to model complex image-question relations that are necessary for answering questions correctly.

Motivated by this, we propose a novel Densely Connected Attention Flow (DCAF) framework for modeling dense interactions between the visual and language modalities (see Fig. 1). Our DCAF framework possesses several additional image and question encoding layers in the early model stage. These layers produce image and question features of different hierarchies, and are densely connected, connecting every layer of question encoding branch with every layer of image encoding branch via Attention Connector (AC). In the AC module, the symmetric co-attention [Nguyen and Okatani, 2018] is performed to enable every interaction between any image region and any question word. The attention outputs of all AC modules are collectively propagated to the end of the encoding layers, and concatenated with the summation of all hierarchical features. Dense connections would incur a massive increase in the number of the attention outputs, thus increase in the representation size in subsequent layers (due to the concat operation). To this end, we compress the attention outputs so that they can be small enough to propagate. Our design of DCAF not only greatly increases the number of interaction interfaces between question and image but also facilitates information flow and gradient flow.

Our contributions are summarized as follows: (1) We propose a novel Densely Connected Attention Flow (DCAF) framework for visual question answering. It can perform dense multi-modal interactions, and capture fine-grained multi-modal information. The dense interactions structure is shown to be superior to other interactions structures (e.g. sequential interactions structure). (2) We propose efficient Attention Connector (AC) to connect two modalities, modeling fine-grained interplay between image and question via co-attention. AC also serves as skip-connector, which effectively connects shallow layers to deeper layers. (3) Extensive experiments conducted on the VQA 1.0, VQA 2.0 and TDIUC datasets show the effectiveness of the proposed DCAF. Our approach outperforms the state-of-the-art methods on the three datasets.

2 RelatedWork

2.1 AttentionMechanisms

Attention mechanisms allow the models to focus on the most relevant image regions and question words. Initially, [Chen et al., 2015] proposed one-step attention to locate relevant image regions. Furthermore, [Yang et al., 2016; Xu and Saenko, 2016] proposed multi-step attention to update relevant image regions and infer the answer progressively. Additionally, [Lu et al., 2016; Nguyen and Okatani, 2018] proposed co-attention, which locates not only the relevant image regions but also question words. Recently, [Fukui et al., 2016; Kim et al., 2017] used bilinear fusion in attention mechanisms to generate more accurate attention weights. Different from these works, we embed the co-attention mechanism into Attention Connectors and apply them across all hierarchical levels. Besides, we also employ the co-attention mechanism similar to [Yu et al., 2017] in the latter model stage.

2.2 Multimodal Fusion

In early VQA methods, simple concatenation or element-wise product between visual and language features are used for multi-modal feature fusion. Recently, bilinear pooling methods are introduced for VQA to capture high-level interactions between visual and language features. Multimodal Compact Bilinear Pooling (MCB) [Fukui et al., 2016] projects the visual and language features into a higher dimensional space and convolves them in the Fast Fourier Transform space. In Multimodal Low-rank Bilinear (MLB) [Kim et al., 2017], the multi-modal features are obtained as the Hadamard product of the linear-projected visual and language features. Pointing out that MLB suffers from slow convergence rate, Yu et al. [Kim et al., 2017] proposed Multimodal Factorized Bilinear (MFB) pooling, which computes fused features with a matrix factorization trick to reduce the number of parameters and improve convergence rate. Ben-younes et al. [BenYounes et al., 2017] proposed the Multimodal Tucker Fusion (MUTAN), which unifies MCB and MLB into the same framework. The weights are decomposed according to the Tucker decomposition. MUTAN achieves better performance than MLB and MCB with fewer parameters. In [Nguyen and Okatani, 2018], the authors proposed a novel co-attention mechanism for improved fusion of visual and language features. It considers every interaction between any image region and any question word, and can be stacked to enable muti-step interactions between the image-question pair. Unlike the above methods that only perform a limited amount of interactions sequentially, our method models cross-modal interactions between any two hierarchical levels, forming dense interactions.

3 Proposed Approach

Our DCAF framework for VQA is depicted in Fig. 1. Given the input question and image, the feature extraction module produces the initial question and image features. The DCAF module further abstracts the features through several encoding layers, and performs the interaction between question and image features at arbitrary layers. Following the DCAF module, a self-attention mechanism is used to learn the attention weights of every word in the question, and obtain the attended question representation (using a weighted sum of the word vectors). Then, a visual attention mechanism is performed to produce the attended image representation. The two representations are merged via element-wise product, then fed into the classifier to determine the final answer prediction.

3.1 Feature Extraction

Similar to [Anderson et al., 2018], we extract the initial image features using Faster RCNN [Ren et al., 2015]. We select a total of top 36 object proposals whose 2048 dimensional feature vectors are obtained from the ROI pooling layer in the Region Proposal Network. The obtained visual region features (with ℓ2 normalization) are denoted as v0 ∈ R2048×36 .

To extract the question features, we first pad or truncate all questions to the same length 15. Then, each word is embedded into a 300-dimensional GloVe vector [Pennington et al., 2014]. The word embeddings of the question are inputted into a two-layer GRU with residual connections and batch normalization on the output of each layer, producing the initial question representation q0 ∈ R300×15 .

3.2 Densely Connected Attention Flow

The DCAF module is the core part of our proposed framework. It has some encoding layers. We adopt a GRU with 300 hidden units as the question encoding layer and the non-linear layer used in [Anderson et al., 2018] as the image encoding layer. The i-th non-linear layer is defined as follows:

where Wi,Wi ′ ∈ R2048×2048 are learned weights, ⊙ represents element-wise product, and vi-1, vi ∈ R2048×36 are the input and output of the i-th non-linear layer respectively. The matrix g acts as a gate on the intermediate activation vi ′ , which is inspired by similar gating operations within GRUs and LSTMs. For clearness, we do not explicitly represent the bias term in our paper.

Starting from the initial image and question features, the encoding layers in two branches generate a series of hierarchical features, v0, v1, ..., vk and q0, q1, ..., qk, respectively. k denotes the number of the layers. We apply our Attention Connectors to densely connect all hierarchical features of image and question:

where AC represents the Attention Connector module (see Fig.2; explained later). ˆ qij ∈ R1×15 and ˆ vij ∈ R1×36 represent the generated attention features for each ij connection. In total, we obtain (k + 1)2 attention features for each word and each region. Intuitively, these features capture finegrained relationships between the image and question at different stages of the network flow. For each branch, we sum up all hierarchical features, then concatenate themwith all attention features as the final output. These skip connections share the similar motivation with DenseNet [Huang et al., 2017]. The process is denoted as

where α1, α2 · · · αk, β1, β2 · · · βk are learned scale parameters initialized as 0, and [· ; ·] represents the concatenation manipulation. qout ∈ R[300+(k+1)2 ]×15 and vout ∈ R[2048+(k+1)2 ]×36 are the outputs of the DCAF module.

Attention Connector

The proposed Attention Connector (AC) is illustrated in Fig. 2. It connects any two encoding layers of image and question in the DCAF module, and captures fine-grained interplay between image and question. The initial step in this module is a symmetric co-attention mechanism. Given qi and vj, an affinity matrix is first constructed via:

where Wij ∈ R2048×300 is a learnable weight matrix. Next, we derive attention maps on question words and attention maps on image regions:

Note that each column of Aqi and Avj contains a single attention map. The attended representations are computed as follows:

where ⊗ represents matrix multiplication. ˜ qij ∈ R300×36 and ˜ vij ∈ R2048×15 are the attended question feature and image feature respectively. In practice, we use multi-glimpse attention mechanism. Specifically, we construct multiple affinity matrixes, leading to multiple attended features. We average the multiple attended features to obtain the final attended features. In Sec. 4.2, we experiment with different number of glimpses to obtain the optimal setting.

The attended features are fused with the original features of the other modality by concatenation and then compressed to low-dimensional space by a single linear layer (FC):

where ˆ qij , ˆ vij are the outputs of AC. We compress the features so that the representation size in subsequent layers would not increase massively, as mentioned in Sec. 1. The output dimension of each word and each region is set to 1, as it performs best in our experiments (see Table 1).

3.3 AttentionMechanisms &Answer Prediction

In this section, we introduce the attention mechanisms (including the textual self-attention and visual spatial attention) and the answer prediction.

Given the outputs qout and vout of the DCAF module, we first perform self-attention mechanism on qout to obtain aggregated representation of the whole question:

where qout = {qout1 , qout2 , ..., qout15 } ∈ R[300+(k+1)2 ]×15 , W Q 1 ,W Q 2 are learnable weights, sQ is the scores of words, αQ is attention weights, and ˜ q ∈ R300+(k+1)2 is vector representation of the question.

We then perform visual attention mechanism on vout to obtain aggregated representation of the image:

where vout = {vout1 , vout2 , ..., vout36 } ∈ R[2048+(k+1)2 ]×36 . fa(·) denotes the gated tanh function [Anderson et al., 2018] with parameters a. It is used to project the concatenated vector to 512-dimensional space. W I ∈ R1×512 is learnable weights, sI is the scores of image regions, αI is attention weights, and ˜ v ∈ R2048+(k+1)2 is vector representation of the image.

After obtaining ˜ q and ˜ v, we project them to the same dimensional space (512 dimensions) using two gated tanh functions with different parameters, respectively. The features are then fused via element-wise product. Similar to [BenYounes et al., 2017], we treat VQA as a classification problem. The fused multi-modal features are fed into the classifier composed of 2-layer MLP with ReLU non-linearity function between the layers and a final softmax function, outputing a class probability vector. Cross-entropy loss is adopted as the objective function for training the VQA system.

4 Experiments

4.1 Setup

Datasets. We use the VQA 1.0 [Antol et al., 2015], VQA 2.0 [Goyal et al., 2017] and TDIUC [Kafle and Kanan, 2017] datasets for our experiments. VQA 1.0 is built from 204,721 MSCOCO images with human annotated questions and answers. The dataset is divided into three splits: train (248,349 questions), val (121,512 questions) and test (244,302 questions). VQA 2.0 is an updated version of VQA 1.0. It contains more samples (443,757 train, 214,354 val, and 447,793 test questions) and is more balanced in term of language bias. TDIUC is a larger dataset that contains 1,654,167 samples and 12 question types. For VQA 1.0 and 2.0, we use the evaluation protocol of [Antol et al., 2015] to evaluate the model. For TDIUC, we calculate the simple accuracy for each question type and Arithmetic/Harmonic mean-per-type (MPT). Implementation details. As in [Yu et al., 2017], we choose the most frequent 3,000 answers in the train and val sets to form the set of candidate answers. The model is trained using the AMSGrad [Reddi et al., 2018] optimizer with an initial

learning rate of 6 × 10-4 . The batch size is set to 128.

4.2 Ablation Studies

We conduct ablation studies on the VQA 2.0 to investigate factors that influence the performance of our proposed DCAF network. The models are trained on the train set and evaluated on the validation set. The results are shown in Table 1.

We first investigate the influence of k (i.e. the number of stacked encoding layers). As shown in the first block of the table, our models outperform the baseline by a significant margin of 1.5%∼1.9%. This demonstrates the effectiveness of the proposed DCAF. The best result is obtained when k = 2. There is a slight performance drop when employing 3-layer DCAF. We then investigate the influence of the number of glimpses. It can be seen from the table that 4 glimpses attains the best performance. 8 glimpses incurs a massive increase in the number of parameters (53.5M→75.6M), making the network harder to optimize. The third block of the table shows the impacts of the dimension of attention features. Increasing the dimension from 1 to 4, produces a 0.2% performance drop. When we set the dimension to be 8, 16 and 32, the performance is 65.4%∼65.7% (not reported in Table 1 due to the lack of space). Considering the efficiency, we set the dimension to be 1 in our model. The last block of the table shows the effect of dense interactions. We remove dense connections while retain the encoding layers in the DCAF module. Such modifications decrease the accuracy by 1.2%. This indicates that the efficacy of DCAF is due to not only the depth of the network (i.e., stacked encoding layers) but also dense interactions. Furthermore, the dense interactions contribute more to the final performance compared with the stacked encoding layers (1.2% vs. 0.7% gain).

Some methods [Yang et al., 2016; Nguyen and Okatani, 2018] perform sequential interactions by stacking several interaction layers/modules. We compare this interaction structure with our dense interactions in Table 2. We implement the stack structure of MLB and MUTAN. The stack structure is proposed by SAN [Yang et al., 2016], which stacks multiple attention layers. DCN [Nguyen and Okatani, 2018] is implemented by stacking multiple symmetric co-attention layers.

As shown in Table 2, with a similar number of parameters, the dense interaction structure models more interactions and achieves better performance than the sequential interaction structure. Furthermore, for the sequential interaction structure, there is no or a slight performance gain when stacking more layers (e.g. 0%, 0.1% and -0.2% gains for MLB, MUTAN and DCN, respectively). While our DCAF(2) can achieve a 0.4% improvement compared with DCAF(1). This phenomenon is in line with our intuition. Our structure contains dense connections, which are helpful to gradient flow and information flow. This enables us to exploit more potentiality of a large network to boost the performance.

In Table 3, we compare the performance of different DCAF variants under different question types. We develop three variants, and they have different sparse connections between

the encoding layers. The first variant SCAF-1 only connects the encoding layers at the same hierarchical level (i.e. connecting q0/v0, q1/v1, and q2/v2). The second variant SCAF2 has cross-hierarchical connections (i.e. connecting q0/v1, q1/v0, q1/v2, and q2/v1). The third variant SCAF-3 has different cross-hierarchical connections (i.e. connecting q0/v2, and q2/v0). From the table, we can find that each variant has its unique ability to answer a particular type of questions. For question type of "Attributes", SCAF-1 outperforms other variants and DCAF. For question type of "Positional Reasoning", SCAF-3 performs best. These variants show different strengths on different types of questions. In contrast, DCAF integrates the three types of connections above, forming dense connections. It inherits the strengths of the variants with sparse connections to a certain extent, and capture fine-grained multi-modal information, thus achieving the best performance in overall accuracy.

4.3 Comparison with State-of-the-arts

In this section, we compare our single DCAF model with the state-of-the-art models on three datasets. Firstly, Table 4 shows the results on the VQA 2.0 dataset. Remarkably, our approach outperforms other state-of-the-art methods in overall accuracy and all question types except for "Number". Secondly, Table 5 shows the results on the VQA 1.0 dataset. Compared with the most recent state-of-the-art model CoR [Wu et al., 2018], our single DCAF model achieves a new state-of-the-art result of 70.0% on test-std set. Thirdly, Table 6 shows the results on the TDIUC dataset. DCAF improves the overall accuracy of the state-of-the-art CoR from 86.9% to 88.0%. In particular, there is an improvement of 10.9% in "Utility and Affordances" and 5.5% in "Sentiment Understanding". In summary, DCAF achieves consistently the best performance on all three datasets.

4.4 Qualitative Evaluation

In Figure 3, we visualize four object regions with the highest attention weights in each picture and four words with the highest attention weights in each question. The first column shows two visualization examples from our model. We can see that the model focuses on relevant object regions and words, thus generates the correct answers. To better understand the effect of attention features, we remove the attention features when visualization. As shown in the second column, the model pays attention to some background regions or neglects some relevant objects (e.g. basketball hoop), obtaining the wrong answers. This shows these attention features can capture some fine-grained information, and help to locate important regions accurately. In the third column, we show the visualization results from 3-layer DCN model as comparison. Some failure examples are shown in the last column. Both failure cases are due to wrong attention regions.

5 Conclusions

In this paper, we present a novel framework Densely Connected Attention Flow (DCAF) for VQA. The core of the framework is the DCAF module, which is designed to enable dense image-question interactions. The proposed Attention Connectors are used to densely connect image and question features at arbitrary layers in the DCAF module and model the interaction between image and question. The experimental results on the VQA 1.0, VQA 2.0 and TDIUC datasets confirm the effectiveness of the proposed framework, and DCAF outperforms state-of-the-art approaches on the three datasets. DCAF is applicable to other multimodal tasks.

Acknowledgements

This work was supported by Beijing Natural Science Foundation (4192059) and National Natural Science Foundation of China (61872366 and 61472422).