Multi-Level Policy and Reward Reinforcement Learning for Image Captioning

1 Introduction

Image captioning is the task of describing the visual content of an image using natural language. Unlike traditional computer vision tasks, such as image classification and object detection, image captioning requires not only visual understanding the image, but also the compositions of natural language. This technique can be widely applied to semantic image retrieval [Karpathy et al., 2014] and human-robot interactions [Das et al., 2017].

Image captioning is a sequential word prediction task. State-of-the-art approaches [Xu et al., 2015; You et al., 2016; Karpathy and Fei-Fei, 2017; Li et al., 2017] generally follow an encoder-decoder framework: they deploy convolutional neural networks (CNN) to encode the image into a visual embedding vector, and then use recurrent neural networks (RNN) to decode the vector into a sentence; during training and inference, they try to maximize the probability of the next word based on the current prediction context. Recently, it has been shown that Reinforcement Learning (RL) [Sutton et al., 1999] can better fit in this task. The reason is that RL aims to learn a policy that decides sequential actions by maximizing the cumulative future rewards [Silver et al., 2016]. Therefore, RL can help to explore more fruitful language in sentence generation, avoiding severe bias in training samples [Rennie et al., 2017]. However, existing RL-based image captioning methods [Liu et al., 2017b; Rennie et al., 2017] mainly rely on a single policy network and reward function that does not well fit the multi-level (word and sentence) and multi-modal (vision and language) nature of the task.

In this paper, we propose a novel multi-level policy and reward reinforcement learning framework for image captioning. The multi-level policy network aims to adaptively fuse the word-level and the sentence-level policies for word generation, and the multi-level reward function aims to collaboratively leverage the vision-language and the languagelanguage rewards to guide the policy. To further bridge the policy network and the reward function, we propose a guidance term by minimizing the distance between the sentencelevel policy and the vision-language reward for optimization.

As shown in Figure 1, the multi-level policy network consists of the word-level policy and the sentence-level policy. The former is the CNN-RNN-based network, which provides the word confidence by locally predicting the next word based on the current state; the latter is a visual-semantic embedding network, which provides the sentence (context) confidence by globally evaluating the current state. The multi-level reward function consists of the vision-language reward and the language-language reward. The former is also a visualsemantic embedding network, which measures the crossmodality similarity between the visual content and the generated description, and defines a specific optimization goal for reinforcement learning; the later is the CIDEr [Vedantam et al., 2015] metric on a pre-defined rule and a stable supplement to the former. Particularly, the vision-language reward and the sentence-level policy leverage the same embedding architecture while owning different parameters. We pre-train an embedding network to initialize the former while the later is directly trained in RL framework.

The contributions are summarized as follows.

• We propose a novel multi-level policy and reward reinforcement learning framework for image captioning. To the best of our knowledge, it is the first RL framework that explores the multi-level (word and sentence) and multi-modal (vision and language) nature of image captioning task.
• We design a multi-level policy network that adaptively fuses word and sentence confidences for sentence generation, and a multi-level reward function that collaboratively leverages vision-language and language-language rewards to guide the generation. A proposed guidance term further bridges the policy and the reward modules.
• We perform comprehensive evaluations on MSCOCO and Flickr30k datasets. Our framework achieves the competing performances against state-of-the-art methods. Ablative studies showcase the effect of the proposed framework.

2 RelatedWork

2.1 Image Captioning

Many image captioning methods have been proposed in the literature. In the early stage, the template-based methods [Farhadi et al., 2010] detected objects from images to generate sentences by pre-defined grammar rules. Recently, a CNN-RNN-based framework was explored and variants were proposed [Vinyals et al., 2015; Karpathy and Fei-Fei, 2017; Dai et al., 2017; Liu et al., 2017a; Nie et al., 2013]. For examples, [Wu et al., 2016] incorporated high-level semantic concepts into the CNN-RNN framework. Additionally, attentionbased methods weighted each feature to exploit the spatial structure and rich intermediate description for images [Xu et al., 2015; Anderson et al., 2017; Zhang et al., 2017; Cheng et al., 2018]. For examples, [Xu et al., 2015] used the attention model to learn where to focus in images during sentence generation. [Anderson et al., 2017] incorporated bottom-up and top-down attention models and [Lu et al., 2017] proposed the adaptive attention model that decided whether to attend to the image or to the visual sentinel.

2.2 Reinforcement Learning

RL aims to learn a policy that decides sequential actions by maximizing the cumulative future rewards [Silver et al., 2016]. Recently, many challenging problems, such as the game of Go [Silver et al., 2016], can be successfully solved by RL algorithms. Several RL methods have been proposed to solve the computer vision problems, such as visual tracking [Yun et al., 2017] and image captioning [Ren et al., 2017]. For examples, [Ren et al., 2017] proposed a "policy network" and a "value network" to collaboratively generate captions, with a reward defined by visual-semantic embedding. [Rennie et al., 2017] and [Liu et al., 2017b] directly optimized image captioning systems by the test rewards.

3 Approach

3.1 ProblemFormulation

We formulate image captioning as a RL process. In image captioning, the goal is, given an image F, to generate a sentence ˆ S = { ˆw1, . . . , ˆwT } which correctly describes the image content, where ˆwi is a word in sentence ˆ S and T is the length. In RL, there is an agent (policy) that interacts with the environment, and executes a series of actions, so as to optimize a goal. Particularly, the environment is the given image F and the words predicted so far { ˆw1, . . . , ˆwt}. An action is to predict the next word ˆwt+1. After each action a, a state s is observed. The state st at time step t consists of the image F and the words predicted until t, { ˆw1, . . . , ˆwt}. The action space is the dictionary D that the words are drawn from, i.e., at ⊂ D. However, existing RL-based image captioning methods rely on a single policy network and reward function that does not well fit the multi-level (word and sentence) and multi-modal (vision and language) nature of the task. To this end, we propose a novel multi-level policy and reward RL framework for image captioning.

3.2 Multi-Level Policy Network

The multi-level policy network consists of the word-level policy and the sentence-level policy.

Word-level policy consists of a Convolutional Neural Network (CNN) and a Long Short Term Memory Network (LSTM). It is similar to the image captioning model [Karpathy and Fei-Fei, 2017] used in the encoder-decoder framework. We first extract the CNN feature I for the input image, and then embed it through a linear mapping. Words are represented by one hot vectors which are embedded with the same dimension as mapped image features. The beginning of each sentence is marked with a special 〈BOS〉 token, and the end with an 〈EOS〉 token. In this policy, words are generated and then fed back into LSTM, with the image feature I treated as the first word. LSTM outputs a distribution ¯wt over all words by updating the hidden states and cells of it. Let θπ denote the parameters of the word-level policy. { ¯w1, . . . , ¯wt-1} is denoted by ¯ S1:t-1. The objective is to minimize the sum of the negative log likelihood of the correct word at each step:

Sentence-level policy is a visual-semantic embedding network, which has been successfully applied to image retrieval [Kiros et al., 2015] and captioning [Pan et al., 2016]. Inspired by [Ren et al., 2017], we map the image feature I and the sentence S into one common embedding space that measures the similarity between them. As shown in Figure 2, given a sentence S, its embedding feature is represented using the last hidden state of RNN. We denote a sentence mapping layer by hp(RNN(S)) and a image mapping layer by fp(I). As shown in Figure 1, the sentence-level policy is fed by the image feature I and the partially generated caption ¯ S1:t from the word-level policy. The confidence between them is computed by:

Further, the sentence-level policy can provide the sentence confidence by globally evaluating the current state.

3.3 Multi-Level Reward Function

The multi-level reward function consists of the visionlanguage reward and the language-language reward.

Vision-language reward is a visual-semantic embedding network that has the same architecture with the sentence-level policy. But there are two different points between them. The first point is this reward is fed by the image feature I and the fully generated caption ˆ S, instead of partially generated caption ¯ S1:t, from the multi-level policy network, which fuses both word-level and sentence-level policies. It can evaluate the vision-language correlation on the fully generated caption, and define a specific goal for RL optimization. As shown in Figure 2, we denote a sentence mapping layer by hr(RNN( ˆ S)) and a image mapping layer by fr(I). This reward is defined by:

The second point is that we pre-train the embedding space for this reward while the sentence-level policy is directly trained in the RL framework. Let θr denote the parameters of the vision-sentence reward. Inspired by [Ren et al., 2017], we use the image-sentence pairs as in the image captioning dataset, and learn the RNN weights as well as mapping layers using a bi-directional ranking loss:

where γ is the margin cross-validated, every (I, S) are a ground truth image-sentence pair, S- denotes a negative description for the image corresponding to I, and vice-versa with I -.

Language-language reward is the CIDEr metric which has been successfully applied to image captioning task [Rennie et al., 2017]. Because CIDEr is calculated on the pre-defined rule, it can steadily evaluate the sequential actions. We use language-language reward as the supplement to the visionlanguage one. It is computed by comparing the fully generated caption ˆ S with the corresponding ground truth S, which is denoted by rll(S, ˆ S).

In this paper, we directly optimize a linear combination of both rewards as follows:

where rtotal is the linear combination of both rewards. T is the length of sentences. 0 ≤ λ ≤ 1 is a hyperparameter.

3.4 Training Using Reinforcement Learning

The key problem of RL lies in correlating the policy and the reward parts for joint learning. Except that we co-train both parts in the traditional RL framework, we design a guidance term G. It minimizes the distance between the visionlanguage reward and the sentence-level policy by calculating the mean squared loss. Since the vision-language reward is pre-trained with ground truth, it can be regarded as an expert to measure the correlation between images and sentences. However, the sentence-level policy is trained in the RL framework by leveraging all information in the environment. It can be regarded as an amateur for the similarity measure between images and sentences. Therefore, by minimizing G, the expert reward will guide the amateur policy for optimization and further benefit joint learning both parts. Let θa denote the parameters of the sentence-level policy. The guidance term can be formulated as:

We denote the parameters of the multi-level policy network by Θ = {θπ, θa}, and we learn Θ by minimizing the negative expected combination reward rtotal, the guidance term G, and the distribution of generated words p ˆwt (will be defined later in Section 3.5). The objective function can be formulated as:

The training process consists of two steps.

• By standard supervised learning, we pre-train the wordlevel policy θπ and the vision-language reward θr in Eq. 1 and Eq. 4, respectively.
• θπ and θa are jointly trained in Eq. 7. We baseline RL with not only the sentence-level policy c(I, ¯ S1:t) but also the language-language reward rll(S, ˆ S′ ) that is obtained by the current model under the inference algorithm used at test time. A sample approximation to the gradient is:

Here, c(I, ¯ S1:t) and rll(S, ˆ S′ ) serve as a combinemoving baseline by λ. The subtraction with the evaluation leads to a much lower variance estimate of the policy gradient. Scaling the gradient can be seen as an estimate of the advantage of action at in state st.

3.5 Lookahead Inference withMulti-Level Policy

For the multi-level policy network, the inference is guided by the word-level and the sentence-level policies. The former provides the word confidence that locally predicts the next word according to current state. The later provides the sentence confidence that globally evaluates the current state. These complement confidences are collaboratively used to adjust the distribution of next word towards the goal of generating captions that are similar to ground truth.

The agent executes each action by fusing both policies:

where p ˆwt is the adjusted distribution of the next word. 0 ≤ β ≤ 1 is a hyperparameter.

4 Experiments

4.1 Datasets

We evaluate our framework on captioning datasets: MSCOCO and Flickr30k. For fair comparison, we adopt the splits consistent with [Karpathy and Fei-Fei, 2017], which uses 5,000 images for validation and test on MSCOCO; 1,000 images for validation and test on Flickr30k. We drop any word that has count less then five, yielding a vocabulary of size 9,487 and 7,615 words for MSCOCO and Flickr30k, respectively. All the reported results are computed using Microsoft COCO caption evaluation tool 1 , including the metrics BLEU(B@N), Meteor(M), Rouge-L(R) and CIDEr(C).

4.2 Protocol

As shown in Figure 1, we take the output of the 2048-d pool5 layer from ResNet-101 as image feature I. Both the sentencelevel policy and the vision-language reward are the visualsemantic embedding networks. We adopt the same architecture for them, but train them independently. As shown in Figure 2, we use one LSTM unit with 2048-d hidden layers to construct RNN, and the dimension of both linear mapping layers is set to 2048×512.

We use two types of RNN-based captioning models to construct the word-level policy. 1) CNN-RNN model. It is introduced in Section 3.2. 2) Attention model. It is similar to [Xu et al., 2015]. We encode the image by spatially adaptive max-pooling and then the size of output is 14×14×2048. At each time step, the attention model enables LSTM decoder to emphasize features from spatial regions depending on the current context.

1 https://github.com/tylin/coco-caption

In training, the LSTM hidden, image, word and attention embedding dimension are fixed to 512 for the word-level policy. We use Adam optimizer with an initial learning rate of 5 × 10-5 and minibatches of size 64. The maximum number of epochs is 30. The margin λ in Eq. 5, β in Eq. 9, and γ in Eq. 4 are set as 0.6, 0.6, and 0.2, respectively. In testing, beam search is set to 1. All experiments are implemented by PyTorch.

4.3 Comparing with State-Of-The-Art Methods

In this paper, we use CNN-RNN or Attention captioning models to construct the word-level policy, denoted by OursCNN-RNN or Ours-Attention, respectively. For fair comparison, we only provide the results of these two types of models and existing RL models in Table 1. Our framework consistently achieves competing performances against state-of-theart methods across all metrics.

Particularly, we discuss it in three parts. 1) CNN-RNN models. Ours-CNN-RNN is based on a network similar to DeepVS and Google-NIC. The significant improvement over them shows the advantages of multi-level policy and reward. ATT-CNN+LSTM and MSM@MSRA use explicit high-level attributes, and m-RNN uses external data to prove its unique transfer capacity. Even though our method performs better than them. 2) Attention models. Ours-Attention is based on a network similar to Hard-Attention, ERD, and ATT-FCN. The results show we can further improve performances by a large margin. Comparing to Adaptive, that proposes a better attention model to decide whether to attend to the image or to the visual sentinel, Ours-Attention achieves better results, which confirms the effect of our framework. 3) RL models. MIXER is a BLEU@4-driven RL method. It is hard to generalize to other metrics while our method performs well in all metrics. Especially, Decision-Making also uses both policies, i.e., "policy network"(CNN-RNN) and "value network", also with the visual-semantic embedding reward. However, OursCNN-RNN performs better, which validates the effect of the multi-level reward function. Meanwhile, Ours-Attention outperforms SCST that trains the policy network (Attention) also with the CIDEr reward. Additionally, Ours-Attention performs better than Ours-CNN-RNN. It illustrates other powerful mechanisms can be directly integrated into our word-level policy and further improve the performance.

Figure 3 shows the values of reward and loss during training. We observe that both reward curves gradually increase along with the iteration. For the language-language reward, it indicates the generated captions is becoming more and more similar to the ground truth. For the vision-language reward, it is that the distance between the semantics of the generated caption and visual content is becoming closer and closer. Meanwhile, both loss curves are gradually to converge, and stable during training. Note that the word-level policy converges faster than the sentence-level policy. It is because word confidence and sentence confidence are often asynchronous, i.e., changes in word is more sensitive than sentence.

4.4 Ablation Studies of Proposed Framework

In this section, we examine the efficacy of the proposed method by assessing several variants. We divide our framework into five components, i.e., word-level policy, sentencelevel policy, vision-language reward, language-laguage reward, and guidance term. Table 2 presents five variants on Ours-CNN-RNN with respect to different IDs. The mark "✓" stands for "used" while "None" is "removed". For example, variant 3 measures the effect of vision-language reward by removing this component, where λ in Eq. 5 is set to 0. To explicitly validate the contribution of each component, we remove the guidance term in all variants.

Our method outperforms all the variants. For clarity, we discuss the results in Table 2 from Ours to variant 1. First, our method performs better than variant 5, which concludes the proposed guidance term can further bridge the policy and the reward modules. Then, we compare variant 5 with variants 2∼4 that only use single-level policy and reward. The improved results shows the merit of multi-level policy and reward in RL. Finally, variant 1 is a supervised learning baseline which results fall behind others slightly.

4.5 Qualitative Results

We show some qualitative results generated by our method and variants in Table 2. As shown in Figure 4, we can observe three points. 1) single-level vs. multi-level policies. Variants 2∼4 use single-level policy and reward while variant 5 uses multi-level ones. The results show the later can generate competing descriptions against ground truth while the former often miss key information. For examples, variant 5 generates the phrases wine glasses and talking on a cell phone in case(b) and (f), respectively, while variant 3 and 4 miss these key words. It shows the merit of the proposed multi-level framework. 2) the effect of guidance term. Thanks to the guidance term, our method can further improve the descriptions on variant 5, and perform better at recognizing explicit objects, e.g., the hill other than field in case(a) and the around other than at in case(b). Further, the generated captions by our method are more similar to ground truth than others, e.g., we generate the word people that cannot be captured by others in case(g). 3) some failure cases. We show two failure cases in the last column, where all methods fail to understand the important visual contents, i.e., holding an umbrella in case(d) and video game in case(h). It is because our policy cannot capture the explicit objects under the noisy background. Adding more detailed visual modeling techniques such as detection can alleviate such problem in the future.

2: a giraffe standing in a field with a mountain range.

3: a giraffe standing on a lush green hillside.

• 4: a giraffe standing in the grass near a mountain.

5: a giraffe standing in a field with a mountain in the background.

Ours: a giraffe standing on a hill with a mountain in the background.

• GT: a giraffe walking on a grassy hill with mountains in the background.
• 2: a group of people sitting at a table with wine glasses.

3: a group of people sitting around a table.

• 4 : a group of people sitting at a table.

5: a group of people sitting at a table with wine glasses.

O u r s : a group of people sitting around a table with wine glasses.

• GT: a group of people sitting around a table with glasses of wine.

2 : a red sign is in front of a tall building.

• 3: a red sign that is on a pole.
• 4: a red sign on a pole in front of a building.

5: a street sign with a large building in the background.

Ours: a red sign with a large building in the background.

• GT: a post with three signs and a building in the background.
• 2: a man is herding a herd of cattle.

3 : a group of people standing around a herd of cattle.

• 4 : a man is walking with a herd of cattle.

5: a man is feeding a herd of cattle.

Our s : a man is standing next to a herd of cattle.

GT: a young man holding an umbrella next to a herd of cattle.

• 2: a yellow fire hydrant sitting in the middle of a dirt road.
• 3 : a yellow fire hydrant in a park with a fence.
• 4: a yellow fire hydrant sitting on the side of a road.
• 5: a yellow fire hydrant sitting in the middle of a dirt field.

Ours: a yellow fire hydrant sitting on a sidewalk next to a fence.

• GT: a bright yellow fire hydrant sits among landscaping rocks between the sidewalk and the fence.

1: a man is talking on a cell phone.

• 2 : a woman in a suit and tie is standing in front of a window.
• 3: a woman is standing in front of a window.
• 4 : a woman in a suit and tie is standing in front of a window.
• 5 : a woman standing in front of a window talking on a cell phone.

O u r s : a woman in a black shirt talking on a cell phone.

• GT: a woman talking on a phone while wearing a black shirt.

1 : a large clock tower on a city street.

• 2: a large brick building with a clock on the side of it.
• 3: a large clock tower in the middle of a city.
• 4: a large brick building with a clock on the top of it.
• 5 : a city street with a large clock tower in the background.

Ou r s : a group of people walking down a street with a clock tower.

• GT: a tall clock tower with people walking down a city street.

1 : a man standing in a living room with a remote.

2: a man standing in front of a tv in a living room.

• 3 : a man standing in a living room with a tv.
• 4 : a man standing in a living room with a couch.

5 : a man standing in a living room with a christmas tree.

Ours: a man standing in a living room with a couch and a television.

• GT: a man is playing a video game in a living room.

Figure 4: Qualitative results of the proposed framework on MSCOCO. The output sentences are generated by our method and the variants

1∼5 (Table 2). GT stands for the randomly selected ground truth.

5 Conclusion

In this work, we present a multi-level policy and reward reinforcement learning framework for image captioning. Different from previous methods, it explores the multi-level (word and sentence) and multi-modal (vision and language) nature of image captioning task. Particularly, the multi-level policy network can adaptively fuse the word-level and the sentencelevel policies for word generation, and the multi-level reward function collaboratively leverages the vision-language and the language-language rewards to guide the policy. For the optimization, we propose a guidance term to further bridge the policy network and the reward function. Our framework achieves competing performances against state-of-theart methods on MSCOCO and Flickr30k. Further, we explore the effect of each component by variants of the framework. In the future, we plan to investigate the multi-agent algorithm to learn the policy for image captioning.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China (61772359, 61525206, 61472275, 61502337), the Beijing Advanced Innovation Center for Imaging Technology under Grant BAICIT-2016009.