Mucko: Multi-Layer Cross-Modal Knowledge Reasoning for Fact-based Visual Question Answering

1 Introduction

Visual question answering (VQA) [Antol et al., 2015] is an attractive research direction aiming to jointly analyze multimodal content from images and natural language. Equipped with the capacities of grounding, reasoning and translating, a VQA agent is expected to answer a question in natural language based on an image. Recent works [Cadene et al., 2019;

∗Equal contribution.

†Corresponding author.

Li et al., 2019b; Ben-Younes et al., 2019] have achieved great success in the VQA problems that are answerable by solely referring to the visible content of the image. However, such kinds of models are incapable of answering questions which require external knowledge beyond what is in the image. Considering the question in Figure 1, the agent not only needs to visually localize 'the red cylinder', but also to semantically recognize it as 'fire hydrant' and connects the knowledge that 'fire hydrant is used for firefighting'. Therefore, how to collect the question-oriented and informationcomplementary evidence from visual, semantic and knowledge perspectives is essential to achieve general VQA.

To advocate research in this direction, [Wang et al., 2018] introduces the 'Fact-based' VQA (FVQA) task for answering questions by joint analysis of the image and the knowledge base of facts. The typical solutions for FVQA build a fact graph with fact triplets filtered by the visual concepts in the image and select one entity in the graph as the answer. Existing works [Wang et al., 2017; Wang et al., 2018] parse the question as keywords and retrieve the supporting-entity only by keyword matching. This kind of approaches is vulnerable when the question does not exactly mention the visual concepts (e.g. synonyms and homographs) or the mentioned information is not captured in the fact graph (e.g. the visual attribute 'red' in Figure 1 may be falsely omitted). To resolve these problems, [Narasimhan et al., 2018] introduces visual information into the fact graph and infers the answer by implicit graph reasoning under the guidance of the question. However, they provide the whole visual information equally to each graph node by concatenation of the image, question and entity embeddings. Actually, only part of the visual content are relevant to the question and a certain entity. Moreover, the fact graph here is still homogeneous since each node is represented by a fixed form of image-question-entity embedding, which limits the model's flexibility of adaptively capturing evidence from different modalities.

In this work, we depict an image as a multi-modal heterogeneous graph, which contains multiple layers of information corresponding to different modalities. The proposed model is focused on Multi-Layer Cross-Modal Knowledge Reasoning and we name it as Mucko for short. Specifically, we encode an image by three layers of graphs, where the object appearance and their relationships are kept in the visual layer, the high-level abstraction for bridging the gaps between visual and factual information is provided in the semantic layer, and the corresponding knowledge of facts are supported in the fact layer. We propose a modality-aware heterogeneous graph convolutional network to adaptively collect complementary evidence in the multi-layer graphs. It can be performed by two procedures. First, the Intra-Modal Knowledge Selection procedure collects question-oriented information from each graph layer under the guidance of question; Then, the Cross-Modal Knowledge Reasoning procedure captures complementary evidence across different layers.

The main contributions of this paper are summarized as follows: (1) We comprehensively depict an image by a heterogeneous graph containing multiple layers of information based on visual, semantic and knowledge modalities. We consider these three modalities jointly and achieve significant improvement over state-of-the-art solutions. (2) We propose a modality-aware heterogeneous graph convolutional network to capture question-oriented evidence from different modalities. Especially, we leverage an attention operation in each convolution layer to select the most relevant evidence for the given question, and the convolution operation is responsible for adaptive feature aggregation. (3) We demonstrate good interpretability of our approach and provide case study in deep insights. Our model automatically tells which modality (visual, semantic or factual) and entity have more contributions to answer the question through visualization of attention weights and gate values.

2 RelatedWork

Visual Question Answering. The typical solutions for VQA are based on the CNN-RNN architecture [Malinowski et al., 2015] and leverage global visual features to represent image, which may introduce noisy information. Various attention mechanisms [Yang et al., 2016; Lu et al., 2016; Anderson et al., 2018] have been exploited to highlight visual objects that are relevant to the question. However, they treat objects independently and ignore their informative relationships. [Battaglia et al., 2018] demonstrates that human's ability of combinatorial generalization highly depends on the mechanisms for reasoning over relationships. Consistent with such proposal, there is an emerging trend to represent the image by graph structure to depict objects and relationships in VQA and other vision-language tasks [Hu et al., 2019b; Wang et al., 2019a; Li et al., 2019b]. As an extension, [Jiang et al., 2020] exploits natural language to enrich the graphbased visual representations. However, it solely captures the semantics in natural language by LSTM, which lacking of fine-grained correlations with the visual information. To go one step further, we depict an image by multiple layers of graphs from visual, semantic and factual perspectives to collect fine-grained evidence from different modalities.

Fact-based Visual Question Answering. Human can easily combine visual observation with external knowledge for answering questions, which remains challenging for algorithms. [Wang et al., 2018] introduces a fact-based VQA task, which provides a knowledge base of facts and associates each question with a supporting-fact. Recent works based on FVQA generally select one entity from fact graph as the answer and falls into two categories: query-mapping based methods and learning based methods. [Wang et al., 2017] reduces the question to one of the available query templates and this limits the types of questions that can be asked. [Wang et al., 2018] automatically classifies and maps the question to a query which does not suffer the above constraint. Among both methods, however, visual information are used to extract facts but not introduced during the reasoning process. [Narasimhan et al., 2018] applies GCN on the fact graph where each node is represented by the fixed form of imagequestion-entity embedding. However, the visual information is wholly provided which may introduce redundant information for prediction. In this paper, we decipt an image by multilayer graphs and perform cross-modal heterogeneous graph reasoning on them to capture complementary evidence from different layers that most relevant to the question.

Heterogeneous Graph Neural Networks. Graph neural networks are gaining fast momentum in the last few years [Wu et al., 2019]. Compared with homogeneous graphs, heterogeneous graphs are more common in the real world. [Schlichtkrull et al., 2018] generalizes graph convolutional network (GCN) to handle different relationships between entities in a knowledge base, where each edge with distinct relationships is encoded independently. [Wang et al., 2019b; Hu et al., 2019a] propose heterogeneous graph attention networks with dual-level attention mechanism. All of these methods model different types of nodes and edges on a unified graph. In contrast, the heterogeneous graph in this work contains multiple layers of subgraphs and each layer consists of nodes and edges coming from different modalities. For this specific constrain, we propose the intra-modal and cross-modal graph convolutions for reasoning over such multi-modal heterogeneous graphs.

3 Methodology

Given an image I and a question Q, the task aims to predict an answer A while leveraging external knowledge base, which consists of facts in the form of triplet, i.e. < e1, r, e2 >, where e1 is a visual concept in the image, e2 is an attribute or phrase and r represents the relationship between e1 and e2. The key is to choose a correct entity, i.e. either e1 or e2, from the supporting fact as the predicted answer. We first introduce a novel scheme of depicting an image by three layers of graphs, including the visual graph, semantic graph and fact graph respectively, imitating the understanding of various properties of an object and the relationships. Then we perform cross-modal heterogeneous graph reasoning that consists of two parts: Intra-Modal Knowledge Selection aims to choose question-oriented knowledge from each layer of graphs by intra-modal graph convolutions, and Cross-Modal Knowledge Reasoning adaptively selects complementary evidence across three layers of graphs by cross-modal graph convolutions. By stacking the above two processes multiple times, our model performs iterative reasoning across all the modalities and results in the optimal answer by jointly analyzing all the entities. Figure 2 gives detailed illustration of our model.

3.1 Multi-Modal Graph Construction

Visual Graph Construction. Since most of the questions in FVQA grounded in the visual objects and their relationships, we construct a fully-connected visual graph to represent such evidence at appearance level. Given an image I, we use Faster-RCNN [Ren et al., 2017] to identify a set of objects O = {oi}Ki=1 (K = 36), where each object oi is associated with a visual feature vector vi ∈ Rdv (dv = 2048), a spatial feature vector bi ∈ Rdb (db = 4) and a corresponding label. Specifically, bi = [xi, yi,wi, hi], where (xi, yi), hi and wi respectively denote the coordinate of the top-left corner, the height and width of the bounding box. We construct a visual graph G V = (V V , E V ) over O, where V V = {vV i }Ki=1 is the node set and each node vV i corresponds to a detected object oi. The feature of node vV i is represented by vV i . Each edge eVij ∈ E V denotes the relative spatial relationships between two objects. We encode the edge feature by a 5-dimensional vector, i.e. rV ij = [ xj-xi wi , yj-yi hi , wj wi , hj hi , wjhj wihi ].

Semantic Graph Construction. In addition to visual information, high-level abstraction of the objects and relationships by natural language provides essential semantic information. Such abstraction is indispensable to associate the visual objects in the image with the concepts mentioned in both questions and facts. In our work, we leverage dense captions [Johnson et al., 2016] to extract a set of local-level semantics in an image, ranging from the properties of a single object (color, shape, emotion, etc.) to the relationships between objects (action, spatial positions, comparison, etc.). We decipt an image by D dense captions, denoted as Z = {zi}Di=1 , where zi is a natural language description about a local region in the image. Instead of using monolithic embeddings to represent the captions, we exploit to model them by a graphbased semantic representation, denoted as G S = (V S , E S ), which is constructed by a semantic graph parsing model [Anderson et al., 2016]. The node vS i ∈ V S represents the name or attribute of an object extracted from the captions while the edge eSij ∈ E S represents the relationship between vS i and vS j . We use the averaged GloVe embeddings [Pennington et al., 2014] to represent vS i and eSij , denoted as vS i and rS ij , respectively. The graph representation retains the relational information among concepts and unifies the representations in graph domain, which is better for explicit reasoning across modalities.

Fact Graph Construction. To find the optimal supportingfact, we first retrieve relevant candidate facts from knowledge base of facts following a scored based approach proposed in [Narasimhan et al., 2018]. We compute the cosine similarity of the embeddings of every word in the fact with the words in the question and the words of visual concepts detected in the image. Then we average these values to assign a similarity score to the fact. The facts are sorted based on the similarity and the 100 highest scoring facts are retained, denoted as f100. A relation type classifier is trained additionally to further filter the retrieved facts. Specifically, we feed the last hidden state of LSTM to an MLP layer to predict the relation type ˆ ri of a question. We retain the facts among f100 only if their relationships agree with ˆ ri, i.e. frel = f ∈

f100 : r(f) ∈ {ˆ ri} ({ˆ ri} contains top-3 predicted relationships in experiments). Then a fact graph G F = (V F , E F ) is built upon frel as the candidate facts can be naturally organized as graphical structure. Each node vF i ∈ V F denotes an entity in frel and is represented by GloVe embedding of the entity, denoted as vF i . Each edge eFij ∈ E F denotes the relationship between vF i and vF j and is represented by GloVe embedding rij . The topological structure among facts can be effectively exploited by jointly considering all the entities in the fact graph.

3.2 Intra-Modal Knowledge Selection

Since each layer of graphs contains modality-specific knowledge relevant to the question, we first select valuable evidence independently from the visual graph, semantic graph and fact graph by Visual-to-Visual Convolution, Semanticto-Semantic Convolution and Fact-to-Fact Convolution respectively. These three convolutions share the common operations but differ in their node and edge representations corresponding to the graph layers. Thus we omit the superscript of node representation v and edge representation r in the rest of this section. We first perform attention operations to highlight the nodes and edges that are most relevant to the question q and consequently update node representations via intramodal graph convolution. This process mainly consists of the following three steps:

Question-guided Node Attention. We first evaluate the relevance of each node corresponding to the question by attention mechanism. The attention weight for vi is computed as:

where W1,W2 and wa (as well as W3,..., W11, wb, wc mentioned below) are learned parameters. q is question embedding encoded by LSTM.

Question-guided Edge Attention. Under the guidance of question, we then evaluate the importance of edge eji constrained by the neighbor node vj regarding to vi as:

where v′j = W5[vj, rji], q′ = W6[vi, q] and [·, ·] denotes concatenation operation.

Intra-Modal Graph Convolution. Given the node and edge attention weights learned in Eq. 1 and Eq. 2, the node representations of each layer of graphs are updated following the message-passing framework [Gilmer et al., 2017]. We gather the neighborhood information and update the representation of vi as:mi = ∑

where Ni is the neighborhood set of node vi.

We conduct the above intra-modal knowledge selection on G V , G S and G F independently and obtain the updated node representations, denoted as {ˆ vV i }N V i=1 , {ˆ vS i }N S i=1 and {ˆ vF i }N F i=1 accordingly.

3.3 Cross-Modal Knowledge Reasoning

To answer the question correctly, we fully consider the complementary evidence from visual, semantic and factual information. Since the answer comes from one entity in the fact graph, we gather complementary information from visual graph and semantic graph to fact graph by cross-modal convolutions, including visual-to-fact convolution and semanticto-fact convolution. Finally, a fact-to-fact aggregation is performed on the fact graph to reason over all the entities and form a global decision.

Visual-to-Fact Convolution. For the entity vF i in fact graph, the attention value of each node vV j in the visual graph w.r.t. vF i is calculated under the guidance of question:

The complementary information mV-F i from visual graph for vF i is computed as:

Semantic-to-Fact Convolution. The complementary information mS-F i from the semantic graph is computed in the same way as in Eq. 5 and Eq. 6.

Then we fuse the complementary knowledge for vF i from three layers of graphs via a gate operation:

where σ is sigmoid function and "◦" denotes element-wise product.

Fact-to-Fact Aggregation. Given a set of candidate entities in the fact graph G F , we aim to globally compare all the entities and select an optimal one as the answer. Now the representation of each entity in the fact graph gathers questionoriented information from three modalities. To jointly evaluate the possibility of each entity, we perform the attentionbased graph convolutional network similar to Fact-to-Fact Convolution introduced in Section 3.2 to aggregate information in the fact graph and obtain the transformed entity representations.

We iteratively perform intra-modal knowledge selection and cross-modal knowledge reasoning in multiple steps to obtain the final entity representations. After T steps, each entity representation ˜ vF (T ) i captures the structural information within T -hop neighborhood across three layers.

3.4 Learning

The concatenation of entity representation ˜ vF (T ) i and question embedding q is passed to a binary classifier to predict its probability as the answer , i.e. ˆ yi = pθ([˜ vF (T ) i , q]). We apply the binary cross-entropy loss in the training process:

where yi is the ground truth label for vF i and a, b represent loss function weights for positive and negative samples respectively. The entity with the largest probability is selected as the final answer.

4 Experiments

Dataset. We evaluate Mucko on the FVQA dataset [Wang et al., 2018]. It consists of 2,190 images, 5,286 questions and a knowledge base of 193,449 facts. Facts are constructed by extracting top visual concepts in the dataset and querying these concepts in WebChild, ConceptNet and DBPedia.

EvaluationMetrics. We follow the metrics in [Wang et al., 2018] to evaluate the performance. The top-1 and top-3 accuracy is calculated for each method. The averaged accuracy of 5 test splits is reported as the overall accuracy.

Implementation Details. We select the top-10 dense captions according to their confidence. The max sentence length of dense captions and the questions is set to 20. The hidden state size of all the LSTM blocks is set to 512. We set a = 0.7 and b = 0.3 in the binary cross-entropy loss. Our model is trained by Adam optimizer with 20 epochs, where the minibatch size is 64 and the dropout ratio is 0.5. Warm up strategy is applied for 2 epochs with initial learning rate 1×10-3 and warm-up factor 0.2. Then we use cosine annealing learning strategy with initial learning rate ηmax = 1 × 10-3 and termination learning rate ηmin = 3.6×10-4 for the rest epochs.

4.1 Comparison with State-of-the-Art Methods

Table 1 shows the comparison of Mucko with state-of-theart models, including CNN-RNN based approaches [Wang et al., 2018], i.e. LSTM-Question+Image+Pre-VQA and Hie-Question+Image+Pre-VQA, semantic parsing based approaches [Wang et al., 2018], i.e. FVQA (top-3-QQmaping) and FVQA (Ensemble), learning-based approaches, i.e. Straight to the Facts (STTF) [Narasimhan and Schwing, 2018] and Out of the Box (OB) [Narasimhan et al., 2018], and Reading Comprehension based approach [Li et al., 2019a]. Our model consistently outperforms all the approaches on all the metrics and achieves 3.71% boost on top-1 accuracy and

5.69% boost on top-3 accuracy compared with the state-ofthe-art model. The model OB is most relevant to Mucko in that it leverages graph convolutional networks to jointly assess all the entities in the fact graph. However, it introduces the global image features equally to all the entities without selection. By collecting question-oriented visual and semantic information via modality-aware heterogeneous graph convolutional networks, our model gains remarkable improvement.

4.2 Ablation Study

In Table 2, we shows ablation results to verify the contribution of each component in our model. (1) In models '1-3', we evaluate the influence of each layer of graphs on the performance. We observe that the top-1 accuracy of '1' and '2' respectively decreases by 1.1% and 3.94% compared with the full model, which indicates that both semantic and visual graphs are beneficial to provide valuable evidence for answer inference. Thereinto, the visual information has greater impact than the semantic part. When removing both semantic and visual graphs, '3' results in a significant decrease. (2) In models '4-6', we assess the effectiveness of the proposed cross-modal graph convolutions. '4', '5' and '6' respectively replace the 'Semantic-to-Fact Conv.' in '2', 'Visual-toFact Conv.' in '1' and both in full model by concatenation, i.e. concatenating the mean pooling of all the semantic/visual node features with each entity feature. The performance decreases when replacing the convolution from either S-to-F or V-to-F, or both simultaneously, which proves the benefits of cross-modal convolution in gathering complementary evidence from different modalities. (3) We evaluate the influence of the relationships in the heterogeneous graph. We omit the relational features rij in all the three layers in '7' and the performance decreases by nearly 1% on top-1 accuracy. It proves the benefits of relational information, though it is less influential than the modality information.

4.3 Interpretability

Our model is interpretable by visualizing the attention weights and gate values in the reasoning process. From case study in Figure 3, we conclude with the following three insights: (1) Mucko is capable to reveal the knowledge selection mode. The first two examples indicate that Mucko captures the most relevant visual, semantic and factual evidence as well as complementary information across three modalities. In most cases, factual knowledge provides predominant clues compared with other modalities according to gate values because FVQA relies on external knowledge to a great extent. Furthermore, more evidence comes from the semantic modality when the question involves complex relationships. For instance, the second question involving the relationship between 'hand' and 'while round thing' needs more semantic clues. (2) Mucko has advantages over the state-of-the-art model. The third example compares the predicted answer of OB with Mucko. Mucko collects relevant visual and semantic evidence to make each entity discriminative enough for predicting the correct answer while OB failing to distinguish representations of 'laptop' and 'keyboard' without feature selection. (3) Mucko fails when multiple answers are reasonable for the same question. Since both 'wedding' and













'party' may have cakes, the predicted answer 'party' in the last example is reasonable from human judgement.

4.4 Parameter Analysis

In Table 3, we vary the number of retrieved candidate facts and relation types for candidate filtering. We achieve the highest downstream accuracy with top-100 candidate facts and top-3 relation types. In Table 4, we evaluate the influence of different number of reasoning steps T. We find that two reasoning steps achieve the best performance. We use the above settings in our full model.

5 Conclusion

In this paper, we propose Mucko for visual question answering requiring external knowledge, which focuses on multilayer cross-modal knowledge reasoning. We novelly depict an image by a heterogeneous graph with multiple layers of information corresponding to visual, semantic and factual modalities. We propose a modality-aware heterogeneous graph convolutional network to select and gather intra-modal and cross-modal evidence iteratively. Our model outperforms the state-of-the-art approaches remarkably and obtains interpretable results on the benchmark dataset.

Acknowledgements

This work is supported by the National Key Research and Development Program (Grant No.2017YFB0803301).