Image Captioning with Compositional Neural Module Networks

1 Introduction

The task of image captioning lies at the intersection of computer vision and natural language processing. Given an image, the task is to generate a natural language sentence describing the information conveyed in the input image. Image captioning has received increasing attention over the years. The prevalent encoder-decoder frame work [Vinyals et al., 2015] serves as the backbone of many derived models. [Lu et al., 2017] introduced and refined the attention mechanism that allows for better feature extraction and interpretability. [Anderson et al., 2017] further used Faster-RCNN [Ren et al., ] to replace the fixed-resolution attention mechanism. Researchers [You et al., 2016] [Yao et al., 2017] also found that high-level concepts can provide a more concise representation for an image.

The majority of existing approaches follows the sequential model where words in a caption are produced in a sequential manner-i.e., the choice of each word depends on both the preceding word and the image feature. Such models largely ignore the fact that natural language has an inherent hierarchical structure. For example, each object can be associated with various attributes. Even with better feature representations and attention mechanisms, the sequential structure of these models tends to lead to generic descriptions that lack specificity. The models [Dai et al., 2018] [Wang et al., 2017] exploring compositionality have been shown to produce more accurate, specific, and out-of-distribution sentences and perform well on SPICE [Anderson et al., 2016], a semantic metric . Compositional models, however, do not compare well to the sequential models on the n-gram metrics such as BLEU [Papineni et al., 2002]. Because semantic evaluation metrics such as SPICE tend to ignore fluency and assume well-formed captions, the n-gram metrics are still important in judging the fluency of the generated captions.

In this paper, we propose an image captioning model that combines the merit of sequential and compositional models by following a word-by-word generation process and combining grounded attributes from specialized modules. A highlevel illustration of the workflow at one time step and visualization of the module attention is shown in Figure 1. More specifically, the algorithm first proposes regions of interest and then chooses a region to focus on depending on the context. The chosen region and the whole image are fed to a collection of functionally specialized modules where each module is delegated to predict one aspect of the objects such as count, color, and size. This is analogous to the Neural Module Networks (NMN) [Andreas et al., 2016], where each module is responsible for a specialized functionality and the final result is a dynamic composition of different modules. In our case, the model generates the final caption by dynamically attending to different modules. The attributes, therefore, have a hierarchical dependency on and are grounded to the proposed regions.

With the proposed Compositional Neural Module Networks, we aim to generate detailed, specific captions without losing fluency, e.g., "a red apple" instead of "a piece of fruit" or "three people" instead of "a group of people." Overall, the main contributions of this paper are:

• We develop a hierarchical model that employs both compositionality and sequentiality of sentence generation.
• Quantitatively, the model outperforms a state-of-the-art model on a set of conventional n-gram metrics and yields a noticeable improvement over the subcategories f-scores of the SPICE metric that is a more meaningful measurement of the semantics of generated captions.
• Qualitatively, we perform human evaluation using Amazon Mechanical Turk. According to the results, our model more often produces more detailed and accurate sentences when compared to the state-of-the-art model. A further analysis shows that the empirical results correlate positively with the quantitative results.

2 RelatedWork

In this section, we briefly introduce related and similar works and emphasize the differences of our model.

Most recent state-of-the-art models adopt the encodedecoder paradigm, NIC [Vinyals et al., 2015], where the image content is vectorized by a convolutional network and then decoded by a recurrent network into a caption. In this paradigm, attention-based models have been explored widely. AdaptAtt[Lu et al., 2017] followed a top-down attention approach where attention is applied to the output of CNN layers. [You et al., 2016] used a word-based bottom-up attention mechanism. Top-Down [Anderson et al., 2017] proposed a feature-based bottom-up attention mechanism that retains spatial information whereas the word-based approach does not.

[Wang et al., 2017] presented a coarse-to-fine two-stage model. First, a skeleton sentence is generated by Skel-LSTM, containing main objects and their relationships in the image. In the second stage, the skeleton is enriched by attributes predicted by an Attr-LSTM for each skeletal word. ComCap [Dai et al., 2018] proposed a compositional model, where a complete sentence is generated by recursively joining noun-phrases with connecting phrases. A Connecting Module is used to select a connecting phrase given both left and right phrases and an Evaluation Module is used to determine whether the phrase is a complete caption. In this work, nounphrases are objects with associated attributes. In general, compositional models exhibit more variation and details in generated captions; however, they tend to perform poorly on the conventional n-gram metrics which are important measurements of fluency.

Researchers have tried to explicitly model the compositionality of language in Question Answering (QA). This line of research shares a similar paradigm, namely, module networks. Module networks are an attempt to exploit the representational capacity of neural networks and the compositional linguistic structure of questions. [Andreas et al., 2016] learned a collection of neural modules and a network layout predictor to compose the modules into a complete network to answer a question. Rather than relying on a monolithic structure to answer all questions, the NMN can assemble a specialized network tailored to each question. We adopt this idea in QA to design a one-layer NMN with a collection of modules and a composition mechanism. Our model can compose a customized network depending on the context of a partially generated sentence.

[You et al., 2016] combined visual features with visual concepts in a recurrent neural network. LSTM-A5 [Yao et al., 2017] also mined attributes as inputs to a language model. Although our model also uses attributes, the model differs fundamentally in several aspects. First, our model is hierarchical because attributes are grounded exclusively to selected regions that change over time. Second, the model is compositional because it combines grounded attributes and objects from separate detectors to predict the next word. Third, the attention is over the set of functionally specialized modules instead of individual visual concepts. Each module specializes in a single descriptive aspect of an object and determines the most probable attribute for that subcategory. For example, the color module generates different color predictions for different objects in an image depending on where the model's focus is.

3 Method

The proposed hierarchical model for image captioning consists of three main components: Recurrent Neural Network (RNN) Trio, Stacked Noisy-Or Object Detection, and Modular Attribute Detection. We describe the overall captioning architecture as shown in Figure 2, followed by technical details for the three components in Section 3.1 -3.3 and the objective function used for training in Section 3.4.

Inspired by recent successes of region-level attention mechanism [Anderson et al., 2017] [Yao et al., 2018], we use a Faster-RCNN in conjunction with a Resnet-101 backbone [He et al., 2016] to segment an image into a set of regions that likely contain objects of interest and encode each region r as a fixed-length feature vector {v1, ..vDr } ∈ RDv where Dr is the number of regions, and Dv, the size of the feature vector. The feature vectors are used as inputs to other parts of the network.

The captioning model selects which region to attend to depending on the context. Given the region proposals, the stacked noisy-or object detection mechanism estimates all possible objects in the image regions. The modular attributedetection mechanism operates on the attended regions to determine appropriate attributes for the attended object at each time step. The object and attribute detection makes up the compositional component while the RNN trio incorporates the detection results to generate a sentence in a sequential manner.

Similar to [Anderson et al., 2016], we divide the vocabulary into meaningful subcategories: an object set and five attribute sets which are color, size, count, spatial relationship, and semantic relationship. We select the six word-lists based on word occurrence frequency in the training data. The object set consists of visual nouns and the other attribute sets consist of adjectives. For example, red, green, blue are in the color set and sitting, playing, flying are in the semantic relationship set.

3.1 Recurrent Neural Network Trio

The captioning model uses three recurrent neural networks, namely, Attention (A)-LSTM, Visual (V)-LSTM and Semantic (S)-LSTM, to guide the process of generating captions sequentially. The input vector to the A-LSTM at each time step consists of the previous output of the S-LSTM, concatenated with the mean-pooled image feature ¯ v = 1D ∑Di=1 vi and encoding of the previous word. The attended image region feature, ˜ vt, is used as input to the V-LSTM to make an initial estimation of the next word based purely on visual evidence. In the final step, the information from the initial estimation, hvt , objects detection, wobj t , and attributes detection, ¯ ct, are combined to make the final prediction of the next word.

The attended image region feature ˜ vt is obtained through the Region Attention mechanism after the A-LSTM:

where V ∈ RDv×Dr is the set of image region features, Dv is the dimension of visual features and Dr is the number of region features.

3.2 Stacked Noisy-Or Object Detection

Multi-label classification is a difficult task, where classes are not mutually exclusive in an image. Here, we propose a stacked model that consists of two types of Multiple Instance Learning (MIL) object detectors to consider both image regions and the entire image simultaneously. First, following the Noisy-Or Multiple Instance Learning (MIL) model used in [Zhang et al., 2006] [Karpathy and Fei-Fei, 2015], we devise a noisy-or detector to predict a distribution over a set of object labels. The noisy-or operation (Or-MIL) is well suited to this task because it operates on each region separately and a positive detection from any region yields a high probability for the whole image. Second, inspired by [Ilse et al., 2018], we adopt an attention based MIL (Att-MIL) detector to consider the whole image, which contains large background objects such as "grass." The two detection probabilities are combined with a second Noisy-Or operation, thus named the stacked approach.

Suppose that, for a given image I, there are V = {v1, v2, ..., vDr} ∈ RDv image region features proposed by the Faster-RCNN network. The probability of an image containing object aj is calculated by a Noisy-Or operation on all image regions of this image as follows:

where paj i is the probability of object aj in image region vi; paj i is calculated through a sigmoid layer on top of the image region features.

For the attention-based MIL detector, instead of an additional attention mechanism, we use the mean-pooled image region feature ¯ v as follows:

where fj,att denotes parameters in a two-layer fully connected network.

The final prediction, P aj I , is computed using a second Noisy-or operation to combine the two probabilities P aj I ,att and P aj I ,or :

We also design a gating mechanism to refine the object detection result at each time step. For example, if the word "cat" has already appeared in a sentence, we decrease its priority in the detection result for later time steps even though "cat" remains a positive instance for the image:

where P aj I,t ∈ RDobj is the time-dependent prediction; Dobj, the size of the object set; hst-1 , the output of the S-LSTM at the previous time step; and ˜ vt, the attended image region feature at time t.

The output of the object detection module is a word-vector, wobj t = EobjP aj I,t , where Eobj ∈ RDvoc×Dobj is a word embedding matrix from distribution over labels, Dobj, to the word-embedding space, Dvoc. The word-vector wobj t is used as an input to the S-LSTM for final decoding.

3.3 Modular Attribute Detection

Attribute detection is achieved by using a collection of modules, each module m ∈ M = {m1, ...mk} with associated detection parameters θm and a Module Attention mechanism to predict the layout of the modules. In this section, we describe the set of modules and the composition mechanism.

We use k = 5 modules corresponding to different attributes of an object. They are: color, count, size, spatial relationship and semantic relationship modules. The modules map inputs to distributions over discrete sets of attributes. Each module has its own labels and, therefore, learns different behaviours.

The modules all share the same simple architecture, a twolayer fully connected network. Customizing module architectures for different purposes might result in better performances as in [Yao et al., 2018] and [Dai et al., 2017]; in this paper, however, we focus on the overall architecture and leave more sophisticated module architecture designs to future work. The distribution, P m t , over labels for module m at time t is computed using a softmax-activated function denoted by fm:

The outputs of the modules are word vectors wm t = EmP m t , where Em is the word embedding matrix for module m.

Next, we describe the compositional Module Attention mechanism that selects which module to use depending on the context. Inspired by [Lu et al., 2017], we use an adaptive attention mechanism and a softmax operation to get an attention distribution of the modules:

where wm t ∈ RDvoc×k is the module network outputs at time t. k denotes the number of modules in consideration. We add a new element winit t = Eyinit t to the attention formulation. This element is the word vector of the initial estimation of the next word from the V-LSTM.

Depending on the context, the network composes a different set of modules to obtain word-vector ˆ ct ∈ RDvoc for the SLSTM.

3.4 Objectives

Our system is trained with two levels of losses, sentencelevel loss and word-level loss. We first describe the more conventional sentence-level loss and then the auxiliary wordlevel losses.

Sentence-Level Loss

We apply two cross entropy losses to the V-LSTM and SLSTM respectively:

where θ are the parameters of the models; I, the image; and y = {y1, y2, ..., yT }, the ground truth sequence.

Word-Level Loss

We subdivide the word-level loss into two types: loss Latt/or mil to train the object and attribute detectors, and loss Lm to train the module attention mechanism for composing attributes.

Loss from Stacked Noisy-Or object detection: as described in 3.2, the MIL object detection has a stacked design. We train the noisy-or detector and attention-based detector using the two sigmoid cross entropy losses respectively:

where yaj is 1 when ground-truth object aj is present and 0 otherwise. paj ∈ {P aj I ,att , P aj I,or } is a sigmoid-activated function.

Loss from Modular Attribute detection: we use five masked cross entropy loss to train the attribute detection modules:

where m ∈ M and M m t is 1 if an attribute from set m is present and 0 otherwise at time t.

The composition mechanism is trained with the following additional loss:

where Mt is 1 if any ground-truth attribute is present and 0 otherwise. ym,t ∈ Rk+1 is a one-hot vector indicating which module is active at time t.

The final loss is a summation of all losses:

where m ∈ M denotes an individual loss for each attribute module.

4 Experiments

4.1 Datasets

We use MSCOCO [Lin et al., 2014] for evaluation. MSCOCO contains 82,783 training and 40,504 validation images; for each image, there are 5 human-annotated sentences. We use the widely-used Karpathy Split [Fang et al., 2015] to incorporate portion of the validation images into the training set. In total, we use 123,287 images for training and leave 5K for testing. As a standard practice, we convert all the words in the training set to lower cases and discard those words that occur fewer than 5 times and those do not intersect with the GloVe embedding. The result is a vocabulary of 9,947 unique words. For usage of the Visual Genome dataset [Krishna et al., 2017], we reserve 5K images for validation, 5K for testing and 98K images as training data. We refer the readers to [Anderson et al., 2017] for more details on training of the Faster-RCNN network.

4.2 Implementation Details

We set the number of hidden state units in all LSTMs to 512, and the size of input word embedding to 300. We use a pretrained GloVe embedding [Pennington et al., 2014] and do not finetune the embedding during training. The pre-trained embedding is from a public website 1 and consists of 6B tokens in total. In training, we set the initial learning rate as 5e-4 and anneal the learning rate to 2.5e-4 at the end of training starting from the 20th epoch using a fixed batch size of 128. We use the Adam optimizer [Kingma and Ba, 2014] with β1 to be 0.8. We train the Stacked Noisy-Or Object Detector jointly for 5 epoches and stop. The training is complete in 50K iterations.

To ensure fair comparison, we re-train the Top-Down using the same hyperparameters as the proposed model. We report the results with greedy decoding to reduce the effect of hyperparameter search for different models.

We use the top 36 features in each image as inputs to the captioning models and do not finetune the image features during training.

1 https://nlp.stanford.edu/projects/glove/

4.3 AmazonMechanical Turk Setup

Amazon Mechanical Turk (AMT) is a popular crowdsourcing service from Amazon. To investigate the effect of using compositional modules qualitatively, we design a Human Intelligence Task (HIT) to compare two captions generated from our implementation of the top-down model and the proposed compositional module networks. Each turker is asked to select from four options as shown in Figure 5: either of the two captions, equally good, or equally bad. For each image, we ask 5 workers to evaluate.

For 1,250 images, 6,250 responses are received. The images are uniformly sampled from the test split; those images with identical captions from the two models are discarded. We design a qualification test to test workers' understanding of the problem and English proficiency. We adopt a max voting scheme to determine the quality of captions per image. When there is a clear winner, we use it as the result for that image. In the case of ties, we give one vote to each tied option.

4.4 Experimental Results

We compare our proposed model with our implementation of the Top-Down model [Anderson et al., 2017], which achieved state-of-the-art performance on all evaluation metrics previously. We also list the published results of CompCap [Dai et al., 2018], which is another recent compositional model. We also include the published performance of NIC [Vinyals et al., 2015], AdaptATT [Lu et al., 2017], Top-Down and LSTM-A5 [Yao et al., 2017] re-implemented by [Dai et al., 2018] because the re-implementations use comparable visual features and are evaluated on the same test split. There are other models with better performances such as the model proposed by [Yao et al., 2018], which uses additional datasets to train spatial and semantic relationship detectors. Our work is a fair comparison to the Top-Down model since both models use only MSCOCO as the main training data and Visual-Genome to train the Faster-RCNN, which is also used in [Yao et al., 2018]. Our implementation of the Top-Down achieves better performance than the implementation by [Dai et al., 2018] and we use our implementation as the baseline for all comparison.

Shown on the right side of Figure 5, a preliminary analysis of the generated captions shows that our proposed compositional module modle is able to generate captions that include more specific attribute words such as color and count. For example, the proposed model includes 4 times more of specific counts such as three in its generated captions.

Evaluation Metrics

We evaluate the approaches on the test portion of the Karpathy Split and compare the proposed approach against bestperforming existing models using a set of standard metrics SPICE [Anderson et al., 2016], CIDEr [Vedantam et al., 2015], BLEU [Papineni et al., 2002], ROUGE [Lin, 2004], and METEOR [Denkowski and Lavie, 2014] as in Table 1. Our proposed model obtains significantly better performance across all n-gram based metrics.

The n-gram metrics alone do not tell the whole story. We also report the performance on a recent metric, SPICE, and its subcategories f -scores in Table 2. When compared to TopDown, our module model achieves noticeable improvement on all subcategories but one. The count subcategory is improved the most. We hypothesize that counting is an inherently difficult task for neural networks and sequential models tend to "play safe" by using generic descriptions instead. This result demonstrates the effect of having dedicated functional modules for composition. It also shows that our proposed model can generate more detailed captions while improving fluency according to the n-gram metrics.

We also note that the size subcategory does not gain improvement over the Top-Down model. We hypothesize that this is due to the simple design of the module. Because the concept of size is a comparison between one object and its environment, our design only considers the object itself and the whole image. A more explicit representation of the concept of size such as bounding box might also be helpful.

Ablation Study

To show the effectiveness of each component, we conduct an ablation study on different variants of our model and compare the performance on SPICE f -scores and n-gram metrics. We use the following notations: Mod stands for the modular attribute detectors; MIL stands for the stacked Noisy-Or object detectors; AMIL stands for the attention based MIL detector. For example, Ours:w/o (Mod+AMIL) is a model without modular attribute detectors and stacked MIL detector (but it has a single layer Noisy-Or detector).

In Table 3, comparing row 2 and row 6 shows that the modular attribute detectors do not contribute to the improvement on the n-gram metrics. Comparing row 4, 5, and 6 indicates that the MIL object detectors are the prime contributors for the improvements on those metrics (CIDEr 106.1→107.1→108.9) and our stacked design further improves the single layer Noisy-Or detector.

In Table 4, comparing row 3 and 6, we can see that the MIL object detectors contribute to the object subcategory the most and also affects the performance on other subcategories a little. However, the absence of modular attribute detectors further deteriorated the performance on other subcategories, such as count (11.9→14.0) and color (6.29→7.95) when comparing row 2 and 6.

In summary, the MIL object detectors contribute to the improvement on the n-gram metrics and object subcategory, while the attributes modules improve on other subcategories. The attribute detectors are responsible for improved semantics and object detectors are primarily responsible for improved fluency.

Human Evaluation using Amazon Mechanical Turk

We report the human judgment on the captions generated by the module model and the Top-Down model. As shown in Figure 5, 5% more people prefer our model over the TopDown. The difference becomes more significant when we consider subsets of the images. We split the evaluation set into subsets depending on whether their 5 ground truth sentences contain related attributes. For example, images in the Color subset contain at least one ground-truth sentence with a color attribute. The difference is 7% in the color subset and 14% in the count subset. The result correlates with the largest improvement on the color and count subcategories in the SPICE subcategory f-scores. This highlights the strength of our model in the subcategories. The human evaluation results qualitatively indicates that there is a discernible improvement recognized by human users.

Qualitative Analysis

Figure 3 shows sample captions generated by the Top-Down model and our proposed model. The examples show that our model gives more accurate description of counting, color, and actions, e.g., more precisely describing a person's bent-over pose in the picture by using "leaning" instead of "standing."

Figure 4 shows failure cases of the proposed model. Two most common mistakes are incorrect counting and color association. Occasionally, the proposed model tries to give a more specific description of counts of people/objects but the count is wrong whereas the baseline model uses a safe description such as "a group of"; sometimes color is associated with a wrong object, e.g., our model predicts "a woman in yellow dress" whereas, in fact, the yellow attribute should have been associated with the teddy bear in the background.

Figure 6 shows two examples of changing module attention over time. From the visualization we can analyze the model's choice of attributes in the generated caption. We observe that the color, count, and size modules are more active at the beginning of a sentence and the initial estimation appears more dominant in the later half. More investigation will be needed to draw a conclusive explanation, but we hypothesize that it may be due to the fact that verbs and objects come first in the English language structure.

Nonetheless, by explicitly proposing grounded attributes to the language model our model is able to include the proposed attributes in the target sentence more often and it is more likely to give "risky" but detailed descriptions of the content in an image.

5 Conclusion

In this work, we propose an image captioning model that utilizes neural module networks to propose specialized and grounded attributes. Experimental results show that our model achieves both the fluency of sequential models and the specificity of compositional models. Specifically, our approach excels at including fine-grained details such as counting that are generally avoided or overlooked. The framework is easily expandable to include additional functional modules of more sophisticated designs. Improved interpretability via visualized attention is another bonus because the model enables a quantitative analysis of both visual and semantic information.

Acknowledgements

This work was conducted in part through collaborative participation in the Robotics Consortium sponsored by the U.S Army Research Laboratory under the Collaborative Technology Alliance Program, Cooperative Agreement W911NF-102-0016. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Laboratory of the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.