An Attention-based Regression Model for Grounding Textual Phrases in Images

1 Introduction

In recent years, semantic grounding has been an active area of research in artificial intelligence. In particular, progress has been made in multimodal tasks, such as image captioning and image QA, with deep learning [Xu et al., 2015; Lu et al., 2016]. Solving this problem is essential for computers to understand how humans communicate about what they see and has many applications, such as robot vision. In this paper, we focus on the task of grounding textual phrases in images.

Previous approaches treat grounding of textual phrases in images as a ranking problem, assigning scores to candidate image regions-which are extracted from the image in advance-based on their relevance to the textual phrase. Thus, previous methods are based on the premise that the correct region is included in the candidate regions, however, this is not always the case. In addition, previous methods scored candidate regions with the likelihood of generating the textual phrase from an image captioning model, leading to difficulty

Eric Nichols and Kotaro Funakoshi

Honda Research Institute Japan {e.nichols,funakoshi}@jp.honda-ri.com

if the target region is not cleanly contained inside a candidate region proposal. To avoid these problems, we treat region prediction as a regression problem and propose a method to directly identify the region using deep learning with attention models. The objective variables of this regression are the top left x and y coordinates, width, and height of a region. Since our proposed method outputs the region itself, it does not require external candidate region proposals, unlike previous approaches. Recently, attention models have been shown to contribute to improved performance in various computer vision tasks [Xu et al., 2015; Yang et al., 2016; Lu et al., 2016]. We apply attention models to learn finegrained correspondences between text and image regions.

Our proposed method consists of five components, as shown in Figure 1. The first component is an image representation which is constructed by applying a convolutional neural network (CNN) to the target image. The second component is a text representation which is constructed by applying LSTMs to the textual phrase. The third component is an image attention model which estimates how much each region in an image is associated with the textual phrase. This image attention model makes predicting the target region's bounding box much easier. The fourth component is a text attention model that estimates the importance of each word based on the current image attention model's output. This model helps us refine the text representation to reflect the most important words. In the image grounding task, themost important information is where. Thus, as input to the text attention model, we use the image attention map, which represents where in the image is being attended to by the whole input phrase. The fifth component is a target region prediction model that performs regression over the image and text attention results to predict the region's bounding box in four parameters: the top left corner's x and y coordinates, width, and height of the bounding box. Our method thus learns to understand the image through the image model, the textual phrase through the text model, and the relationship between the image and textual phrase though the image and text attention models.

Our main contributions are as follows. First, we propose a regression model for identifying the region in an image corresponding to a textual phrase that does not rely on external image region predictions, eliminating a potential bottleneck in both accuracy and computational efficiency. Second, we show through detailed evaluation on the ReferIt dataset [Kazemzadeh et al., 2014] that our proposed method achieves a new state-of-the-art of 37.26%, surpassing the previous best-performing method by over 5 percentage points. Third, we confirm that, by applying attention models over both image and text, our approach is able to learn fine-grained correspondences, further improving its performance.

2 RelatedWork

Hu et al. [2016b] proposed a Spatial Context Recurrent ConvNet (SCRC) to predict the region in an image corresponding to a given textual phrase. SCRC extracts candidate regions in advance using EdgeBox [Zitnick and Doll´ar, 2014], and then it assigns a score to each candidate region based on the probability of a given textual phrase being generated by an image captioning model from that candidate region. SCRC takes advantage of spatial information as well as image context features for candidate regions when computing the probability. Training SCRC requires full images, target image region annotations, and the associated textual phrase. However, it is difficult to collect large amounts of such data, so SCRC uses a pre-trained image captioning model to generate textual phrases from candidate regions.

To cope with the problem of sparse training data, Rohrbach et al. [2016] proposed a method called GroundeR that uses unsupervised learning. Like SCRC, GroundeR scores candidate regions with a caption generation model, but it also uses an attention model to decide which candidate region is most likely to be referred to by a textual phrase. It then generates the text of a gold standard caption, making it possible to learn image attention without correct region annotations. When the correct region is given, it is possible to supervise learning by directly fitting the predicted region to the correct region. Fukui et al. [2016] proposed Multimodal Compact Bilinear pooling (MCB) to extract multimodal feature. The outer product is more expressive than element-wise product and concatenation for making the multimodal feature. However, the outer product is typically infeasible due to its high dimensionality. MCB makes it possible to project the outer product to a lower dimensional space. They evaluate MCB on image QA and visual grounding. In the evaluation on visual grounding, their model receives the textual phrase and candidate regions as the input and makes multimodal feature from these. The output is the classification result that is which region is corresponds the textual phrase. Luo et al. [2017] proposed the model in which a generation model, which generates the expression for the region in the image, and a comprehension model, which selects the region corresponding to given expression, are integrated. First, the comprehension model is trained, and then the generation model is trained to generate the expression from which the comprehension model can select the correct region. Finally, Hu et al. [2016a] proposed a method which directly learns the semantic segmentation of the region corresponding to a textual phrase. Their method, however, requires gold standard semantic segmentation data, and they evaluate the Intersectionover-Union (IoU) between the correct region's segmentation and their predicted segmentation, making direct comparison to methods predicting bounding boxes infeasible.

Our approach differs from [Hu et al., 2016b; Rohrbach et al., 2016; Fukui et al., 2016; Luo and Shakhnarovich, 2017] by using regression to ground textual phrases, eliminating the need for external candidate regions and avoiding adverse influence from candidate region prediction quality. Unlike [Rohrbach et al., 2016], it uses attention models over both image and text, learning more sophisticated relationships between image regions and text. Unlike [Hu et al., 2016a], it does not require semantic segmentation annotations.

A task related to image region prediction is object detection, whose goal is to find a predetermined class of objects in image. Blaschko and Lampert [2008] proposed a method for detecting a single object in a single image using Support Vector Machines (SVM). Their proposed SVM learns a mapping of an input image to a bounding box. When training, they use Intersection over Union (IoU), which represents how much overlap occurs between the two regions, for the loss function. We designed our loss function with reference to this idea.

Bounding box regression has been used for refining candidate regions. Jaderberg et al. [2016] trained a CNN regression model that receives a candidate region as input and outputs a refined region. The primary difference between our work and theirs is that they use regression to refine candidate regions, while our approach uses it to directly predict a target region based on the results of attention models.

Recently, attention models have been applied in a variety of tasks, such as machine translation [Bahdanau et al., 2015], textual question answering (QA) [Kumar et al., 2015], and image caption generation [Xu et al., 2015]. In the task of image QA, attention models have been used to detect and focus on the region in an image that is most related to answering a question. Attention in this situation makes it possible to boost the features in the relevant region, leading to more accurate answer prediction. Lu et al. [2016] proposed a method which combines both text and image attention at three different levels: word, phrase, and sentence. They introduced a function to specify which word or phrase in a question to attend to in an image, and performed a bidirectional attention from both question-to-image and image-to-question. Attention, thus, is effective in tasks requiring multimodal understanding.

Our approach is related to the attention models used in image QA. The textual phrase specifying an image region corresponds to a question, and predicting the target region corresponds to predicting the answer to the question. The difference between our approach and the approach for image QA is the target of the attention. In image QA, the attention model estimates the image region most related to the answer. In contrast, our image attention model directly predicts the region related to the textual phrase. Furthermore, attentionbased image QA methods use the features focused on by their attention models to produce a textual answer of some kind, whereas we use it to predict the region that the textual phrase refers to. Thus, in addition to image attention, we introduce a text attention model that estimates which words are important in the spirit of [Lu et al., 2016].

3 AttentionModels for Region Prediction

Here we describe our proposed method for predicting the bounding box of a region in an image given a textual phrase by regression over four variables (x, y,w, h), where (x, y) represents the left-top corner of the bounding box and (w, h) represents the width and the height of the bounding box. An overview of our proposed method is shown in Figure 1, consisting of an image representation generated with CNNs, a text representation generated with LSTMs, independent image and text attention models, and the region prediction regression model. In this section, we elaborate on these components and describe the loss function used to train the model.

3.1 Image Model

We use the CNN VGG 16-layer model of [Simonyan and Zisserman, 2014] to extract image features images. Specifically, we extract image features from vI, the output of intermediate layer conv5-3. Conv5-3 outputs 512 feature maps of size 14 × 14. Since the VGG 16-layer accept a 224 × 224 pixel image as input, one feature map produced by conv5-3 corresponds to a 16 × 16 region of the input image, amounting to a 14 × 14 × 512 dimension feature vector. Hereafter, we denote this image feature by Vmap = (vr1, vr2, ..., vr196), where vri ∈ R512 is the feature for region i.

3.2 Text Model

In order to construct feature representations from text, we use Bidirectional LSTMs (BLSTM) to extract textual phrase feature vT and word features vl. The text attention model computes how much attention is given to each word feature outputted by the text model. BLSTMs are used to prevent the model from learning representations that are biased toward words near the end of the textual phrase. Given each word in the textual phrase represented by one-hot vector ql whose size is the word embeddings' vocabulary size and the dimension associated with a target word set to 1 and all other dimension set to 0, first we transform each word into a distributed representation ul denoted by embedding matrix We,

where l denotes the position where the word appears, and L denotes the length of the phrase. The BLSTM outputs are computed as follows:

In the above equations, LSTMfwd denotes an LSTM encoding the textual phrase in the forward direction. LSTMbwd denotes an LSTM encoding the textual phrase in the backward direction. To merge the -→vl and ←- vl , we use single layer perceptron from the result of the preliminary experiment. The gates (il,fl,ol,zl), memory cell (cl), and the hidden state (vl) of the LSTM are computed as follows:

where σ denotes the sigmoid activation function.

For the word features, we use vl, and for the phrase feature vT , an element-wise mean vector computed as follows:

Our system does not make use of any explicit semantic or syntactic information outside of what is captured from word context during the training of the embeddings.

3.3 Image AttentionModel

Given the image feature maps Vmap and the text representation vT , the image attention model estimates how much each region in the image corresponds to the textual phrase. As shown in the following equations, we use a Multilayer Perceptron (MLP) to compute the weight αI i , which represents how much the model attends to region i, for each region.

where hvi ∈ Rn is the hidden state, Wvh ∈ R1×n , and bvh ∈ R1 . Thus, αI i is a scalar. We repeat this process for the entire region, obtaining the image attention map vIα = (αI1 , αI2 , ..., αI196 ) in Figure 2 (a) 1 . The above equations are for a 3-layer MLP, however, the number of hidden layers and the size of hidden state n are tuned as hyper-parameters.

3.4 Text AttentionModel

Given the word features vl and the image attention map vIα, the text attention model estimates how important each word is and updates the phrase feature accordingly. As with the image attention, we use an MLP to compute the weight αT l for each word feature vl as shown in Figure 2 (b).

where htl ∈ Rm which is hidden state, Wth ∈ R1×m, bth ∈ R1 . Similar to image attention, αT l is a scalar. Then, the phrase feature representation is updated as follows:

1 It is typical to use softmax as the activation function for attention models, however, in preliminary experiments we experienced difficulties training, and switching to relu allowed it to succeed.

Again, the number of hidden layers and the size of hidden state m are hyper-parameters.

3.5 Region PredictionModel

We employ an MLP that takes the image attention map vIα and updated phrase feature v′T as input and predicts a bounding box specified by (x, y, w, h) as output. The hidden layer and the output layer are computed by the following formula:

where g is the bounding box. The number of hidden layers and the size of hidden state are tuned as a hyper-parameters, as with image and text attention.

3.6 Loss Function

We adopt Intersection over Union (IoU) [Everingham et al., 2005] in order to define the error between the predicted region g and the ground truth region t. IoU represents the overlap between two regions, and is calculated as follows:

Blaschko et al [2008] defined the following Equation (1) as part of their loss functions to train SVMs for object detection.

We use this IoU loss to train our model. We also use the squared error between the ground truth and the predicted regions to directly optimize our model's output. The squared error is computed by:

where we treat t and g as a 4-dimensional vector (x, y, h, w).

Furthermore, we introduce a penalty to help fit the image attention model area to the ground truth region. We define this penalty function LS(θ) and the area of the image attention Satt by the following equations:

where Sbox is the area of the ground truth in 14 × 14 feature maps. 2 The loss function L(θ) we propose is as follow:

where λ1 and λ2 are hyper-parameters.

4 Experiments

In this section, we describe the dataset we use, our model's configuration, experimental results, and discuss the advantages and disadvantages of our model.

4.1 Dataset

We used the ReferIt dataset [Kazemzadeh et al., 2014] for both training and evaluation. This dataset consists of three parts: images, regions inside each image, and captions for each region. There are 20,000 total images, taken from the IAPRTC-12 dataset [Grubinger et al., 2006]. The regions come from the SAIAPR-12 dataset [Escalante et al., 2010]. There are approximately 120,000 total captions constructed in a two-player game with the goal of generating unambiguous referring expressions for target regions. There are approximately 100,000 total objects with 255 categories. These categories includes background such as sky. Therefore, this dataset includes bounding boxes of ambiguous size. We employed the same data splits as Hu et al. [2016b]: 9,000 images for training, 1,000 for validation, and 10,000 for testing to facilitate comparisons with prior approaches. In comparison, other multimodal datasets like MSCOCO image captioning [Lin et al., 2014] and VisualQA [Antol et al., 2015] have over 10 times as many images and 6-12 times as many unique image-annotation pairs as ReferIt 3 , illustrating the data sparsity challenges faced in this task.

4.2 Model Configuration and Training

We performed hyper-parameter optimization using random search. The hyper-parameters and final settings are shown in Table 1. The size of each hidden layer is sampled by increment of 50. However, we selected the loss function weights from preliminary experiments because they have more influence over the training than other hyper-parameters. We fix the word embedding size to facilitate comparison between different embeddings and use the same 8,800 word vocabulary as [Hu et al., 2016b]. We train the model with back propagation using Adam [Kingma and Ba, 2014] for SGD, with the authors' recommended values of hyper-parameters. We allow all parameters to be updated, except for the VGG 16-layer.

2 It is possible that this penalty, when combined with the relu, which stands for the rectified linear unit activation function, in our image attention model, has an effect similar to normalization.

3 MSCOCO has more than 300K images and 1.5M captions, and Visual QA has over 250K images with over 750K unique questions.

4.3 Results

In this section we present comparative evaluation against SCRC [Hu et al., 2016b] and the current state-of-the-art method, GroundeR [Rohrbach et al., 2016]. Following Hu et al. [2016b], we adopt accuracy as our evaluation criterion, where we consider a prediction correct if the overlapping IoU of the ground truth and the predicted area is ≥ 50%. Table 2 shows the comparison result with previous works. Our proposed method greatly outperforms previous works.

In order to confirm the effect of each component in our approach, we train models without text attention (-TA), without text and image attention (-TA and IA), and without the loss penalty (-loss penalty LS(θ)). Comparing the full model to them as shown in Table 3, we see that the full model outperforms the model without text attention by 1.71% points and the model without the text and image attention by 7.83% points. These result show that combining text and image attention is effective, but that image attention makes an especially large contribution. In addition, the model without text and image attention outperforms GroundeR by 2.50% points, showing that our regression approach is more effective than approaches that score candidate regions.

Furthermore, the full model outperforms the model without the loss penalty by 0.85% points, and the region corresponding textual phrase is more accurately focused as shown in Figure 3, showing that the loss penalty is effective.

We compare word embeddings as shown in Table 4. We evaluated on GloVe embeddings [Pennington et al., 2014] because they have higher coverage than word2vec [Mikolov et al., 2013] and released embeddings trained on various size datasets. We find that GloVe embeddings trained on the largest dataset of 840 billion words outperform the other GloVe embeddings by 0.41-0.66% points and randomlyinitialized embeddings by 1.56% points 4 , demonstrating the effectiveness of pre-trained word embeddings.

4 We suspect GloVe 840B performed best because it had the largest vocabulary and training dataset of those tested.

Figure 4: Predicted examples. The input image is on the left, and the attention map is on the right. The red bounding box is the ground truth region, and the yellow bounding box is the predicted region. For visualization of attention, red represents high and blue represents low in both image and text attention.

4.4 Visualization of Attention

Examples of attention visualization are shown in Figure 4. The successful examples in Figure 4 (a) show that the region corresponding to the textual phrase is correctly focused on by the image attention. Furthermore, when a region in the background of the image is specified by the textual phrase, the image attention model successfully avoids focusing on objects in the foreground, as shown in the rightmost example. For text attention, words related to position such as bottom and left are given attention, demonstrating our model's ability to learn linguistic cues indicating location.

4.5 Error Analysis

We randomly sampled 100 erroneous predictions from our model and manually classified them by error causes. A single prediction can have multiple causes.

We find that the most common error cause is incorrect image attention, which occurs over 80% of the time. In most cases, the image attention is spread throughout the image and does not focus on the correct region as shown in the leftmost example in Figure 4 (b). In comparison, text attention errors, where a non-central expression is focused on, occur slightly less than 40% of the time. As can be seen by the text attention visualization in Figure 4 (b), despite using BLSTMs, text attention is often still biased toward the end of a sentence. Furthermore, in 36% of cases, a referring expression is attended more strongly than the object that is the target of the textual phrase. An example of this is "2nd upright bike tire" having weaker attention than "closest to man" in the middle example in Figure 4 (b). These errors show that our approach could likely benefit from hierarchical or cyclical attention models that allow for multiple passes over each attention module.

Gold annotation errors are another common cause of errors. A little less that 20% of the errors are caused by ambiguity, spelling errors, or unknown words in textual phrases, such as womans and shelf? in the leftmost example in Figure 4 (b). Furthermore, in 28% of cases, the ground truth regions given were ambiguous, either by being too big or small, or by not having well-defined boundaries, which is often the case for textual phrases such as sky or corner. These annotation errors may lead to errors in "image attention" and "text attention."

Finally, in over half of all errors, the bounding box predicted by our model does not capture a coherent region in the image, as shown in the rightmost example in Figure 4 (b), suggesting that despite our approach's independence from image region proposals, it may benefit from incorporating some kind of objectness judgement in its region predictions.

5 Conclusion

In this paper, we proposed a new attention-based method for directly predicting the region in an image specified by a textual phrase through regression. Through evaluation on the ReferIt dataset, we demonstrated that, despite the challenging nature of this task and sparsity of the dataset, our proposed method greatly outperformed the accuracy of all known methods by over 5% while eliminating the need for external image region candidates. Our evaluation and visualization of attention results also showed that the image and text attention and image attention size-based loss penalty greatly contribute to performance. Detailed analysis showed that attention failures and incoherent bounding box predictions were common causes of errors. In future work, we plan to refine our penalty function, incorporating measures of objectness. In addition, to better handle examples that require inference, we plan to explore attention architectures that allow multiple passes.

Acknowledgments

The part of this research is supported by MEXT KAKENHI, Grantin- Aid for Scientific Research (B), Grant Number 17H01746.