Video Interactive Captioning with Human Prompts

1 Introduction

Video captioning aims at automatically generating a natural language sentence to describe a video accurately. As a video clip often includes rich visual content and semantic details, different people may be interested in different views. Thus, one sentence generated by a video captioning model usually fails to meet the ad hoc expectations. Inspired by the relevance feedback in information retrieval [Rocchio, 1971], we propose to launch a round of interaction between a human and a captioning agent, so as to refine the initial captions.

In this paper, we present a new task of Video interactive Captioning (ViCap): When a caption generated by a pretrained video captioning model does not meet the expectation, the ViCap agent asks for a short prompt from the human as a clue of his expectation. Then based on the prompt, the agent could generate a more accurate caption. In Fig. 1, we show an example. Firstly, given as input a video clip to a pre-trained video captioning model, an initial caption is generated, i.e., 'Someone is showing cartoon clip from tv'. As the human may be interested in certain frames of the clip, the description fails to meet his expectation. Then he gives a short prompt, i.e., 'A squidward'. Finally, based on the prompt, the ViCap agent generates a more accurate caption 'A squidward is talking to the spongebob at the crusty crab'. From a practical view, a human could provide any prompts in any form, e.g, in different length. And the prompts could appear anywhere in the generated captions. In this paper, we move forward a first step in that we set the prompts to be the first two words of the generating caption. Thus ViCap with human prompts is different from traditional captioning tasks, which may support various applications, such as video retrieval [Kordopatis-Zilos et al., 2018], video chat robot, and even human-robot interaction [Pasunuru and Bansal, 2018].

∗ Part of this work was done when Yi Yang was visiting Baidu Research during his Professional Experience Program. Code is publicly available on GitHub: https://github.com/ViCap01/ViCap.

The challenges of this task are two-fold: First, given as input the video clip, the initial caption, and the human prompts, the ViCap model should be able to generate a sentence not only starts with the prompts but also faithfully describes the video content. Second, as there launches a round of interaction, the prompts and the expecting captions are certainly semantically different from the initial ones, which therefore lack the ability of guiding the generation of the refined sentences. In this paper, we devise the ViCap to include a video encoder, an initial caption encoder, and a refined caption generator. Moreover, we show that the ViCap can be trained via a full supervision way, i.e., supervised with the ground-truth captions, or a weak supervision way, i.e., weakly supervised with only the prompts.

In particular, we first utilize a GRU network [Cho et al., 2014] to encode the video and the initial caption, respectively. For the caption generator, the commonly used methods are based on LSTM [Li et al., 2017] or GRU network. However, these generators are prone to dilute the long-term information [Gehring et al., 2017]. Recent works [Gehring et al., 2017] have demonstrated that employing convolution operation as the decoder could alleviate the problem of long-term information dilution. Thus, in this paper, we stack multiple dilated convolutional layers [Yu and Koltun, 2015] followed by gated activation units [Oord et al., 2016] as the refined caption generator, the goal of which is to capture dependencies among frame and word sequences. For the training of ViCap with weak supervision of prompts, we devise a convolutional reconstruction network to reconstruct the current input of decoders. During the training, the reconstruction loss and the prompt-supervision loss are merged together.

For the evaluation of ViCap with human prompts, we first extend the MSRVTT with interaction ground-truth. Then the experimental results not only show the prompts can help generate more accurate captions, but also demonstrate the good performance of the proposed method.

2 RelatedWork

Recent advances towards video captioning mainly follow the encoding-decoding framework. Besides efforts made on devising effective encoders and decoders [Baraldi et al., 2017; Wu and Han, 2018], Wang et al. [Wang et al., 2016] proposed a new multimodal memory encoder and the method in [Pan et al., 2017] utilized the transferred semantic attributes to help models generate better captions. Towards efficiency and a more compact representation of videos, Chen et al. [Chen et al., 2018] proposed a reinforcement learning-based method to pick some keyframes to generate video caption. Besides, as the current most methods about video captioning suffer from the problems of exposure bias and inconsistency between train/test measurement [Keneshloo et al., 2018], these works [Wang et al., 2018; Phan et al., 2017] proposed to leverage reinforcement learning to solve these problems.

As a video often includes rich visual content and semantic details, e.g., diverse objects and events, the existing captioning methods are hard to generate one sentence to fully describe these content and details. In this paper, we propose a new task of ViCap, i.e., when the initial caption does not meet human expectation, models could generate refined caption based on human prompts. As the existing captioning methods often employ a GRU or an LSTM unit as the decoder and only take a video clip as the input to generate a sentence, we could not directly employ GRU or LSTM decoder to solve the task. The reason is that the new task needs the decoder could leverage the video content, the initial caption, and the prompts to generate a more accurate caption. To solve this task, we devise a convolutional decoder, which uses the initial caption and the prompt to generate accurate caption. Finally, the experimental results on our extended MSRVTT dataset demonstrate the effectiveness of the convolutional decoder.

3 The Framework of ViCap

In this paper, we devise the ViCap model which includes a video encoder, an initial caption encoder, and a refined caption generator. Moreover, we show that the ViCap can be trained via a full supervision way or a prompt supervision (weak supervision) way.

Concretely, we employ a GRU unit to encode the video and the initial caption, respectively. For the video, we use HX = {h1X ∈ Rk , ..., hmX ∈ Rk } to denote the GRU output set and use Hmean X to indicate themean visual feature of HX. For the initial caption, we take the GRU output hY ∈ Ro at the last time step as the representation of the initial caption.

3.1 The Convolutional Decoder of ViCapModel

As said in the Introduction, the new task aims at generating an accurate sentence which is semantically different from the initial caption. Thus, in order to enlarge the semantic gap, we propose to use the vertical representation H⊥ of the initial caption representation hY in the decoder (H⊥·hY = 0, where · indicates the dot product), which indicates H⊥ and hY are irrelevant. In Fig. 2, we show two examples of enlarging the semantic gap. The left part is the case where the output HN of a decoder is positively related to hY . And the right part is the negatively correlated case. We can see that by adding H⊥ to HN (HF = HN +α⊙H⊥, where ⊙ indicates the element product), the semantic gap between HF and hY is obviously enlarged, i.e., compared with the angle between HN and H⊥, the angle between HF and H⊥ becomes smaller. Based on this idea, we devise a convolutional refined decoder.

Computation of H⊥

We define hY = [y1, y2, ..., yo], where y1, y2, and yo are elements of the vector hY . When the dimension o is set to an even number, the computation of H⊥ is shown as follows:

The goal of this operation is to keep H⊥ · hY = 0.

Convolutional Decoder

In this paper, we stack five dilated convolutional layers to form convolutional sequential decoder (Fig. 3). In the following, we denote by ˆ Y C = {ˆ Y C 0 , ..., ˆ Y C L-1 } the predicted word sequence. We denote by Y C = {Y C 0 , ..., Y C L-1 } the target word sequence, where L denotes sequence length. [a, b] represents the concatenation of a and b.

As shown in Fig. 3, at each step t, the operations of each layer are shown as follows:

where rl represents dilated rate of the l-th layer. hl-1 t denotes the output of the (l - 1)-th layer at time step t. wlf and wlg denote convolutional filters on the l-th layer. blf and blg are bias. Adding H⊥ is to enlarge the semantic gap.

Note that for video caption generation, there is no future information available for the decoder. Besides, based on the different size of both filter and dilated rate, we use different number of zero vectors to pad the input of each layer.

Finally, the t-th generated word ˆ Y C t is computed as:

where h5t indicates the output of the fifth layer. wp and bp are learnable parameters. The goal of ReLU operation is to keep hY and h5t are positively related.

3.2 ViCap with Full Supervision

In order to reduce the risk of vanishing gradients, inspired by the work [Zhang et al., 2016], we enforce intermediate supervision for some hidden layers. In our network, since the function of the first layer and the fifth layer is to process the input and generate the output, respectively, we enforce supervision for these two layers. For each layer j ∈ {1, 5}, we employ a cross-entropy loss.

where p( ˆ Y C t |Y C 0:t-1 ,HX, hY ) is the output probability of the predicted word ˆ Y C t given the previous word Y C 0:t-1 , visual feature HX, and the encoding hY of the initial caption. By summing the loss of each layer, we obtain the overall loss:

where λ1 and λ5 are hyper-parameters. And we set λ1+λ5 = 1. Meanwhile, as the fifth layer is the output layer, we should keep λ5 is bigger than λ1.

As said in the Introduction, we should keep the generating captions could faithfully describe their corresponding video content. Thus, we define a new loss function to ensure the consistency between the caption and video content, which is named consistency loss (Ls). The details are as:

where ‖·‖ indicates ℓ2-norm. wr is a learnable parameter.

Finally, the training loss Ltrain is computed as follows:

3.3 ViCap with Prompt Supervision

For the case of prompt supervision, since we have no other word as a supervision in addition to the prompt words, we could not completely use the teacher forcing mechanism [Lamb et al., 2016] to train the network. If we only use the prompt-supervision loss to train the network, as there only exist two prompt words, it is easy to make the network overfitting. And the performance is not good. In order to reduce the risk of overfitting and improve the performance, we propose a recurrent reconstruction network (Recon-Net) which uses the output of the decoder to reconstruct the current input of the decoder (Fig. 4). Finally, the reconstruction loss will be jointly trained with the prompt-supervision loss.

Convolutional Reconstruction Network

As shown in Fig. 4, we stack three dilated convolutional layers as the reconstruction network. At time step t, we take the output oL-t of the decoder and the mean visual feature Hmean X as the input of the first layer. The details of this layer are as:

where wr c and brc are learnable parameters. w1 p and w1 q are convolutional filters. b1p and b1q are bias. Then the operations of the next two layers are similar to those of the first layer. As same as the convolutional decoder, we enforce intermediate supervision for the first and output layer to reduce the risk of vanishing gradients. Thus, the loss of the convolutional reconstruction network is as:

where Gt indicates the input of the decoder at time step t. β1 and β2 are hyper-parameters. And we set β1 + β2 = 1. Besides, as R3 t is the output of top layer, we should keep β1 is bigger than β2. For the prompt-supervision loss, we still use cross-entropy loss. Finally, the training loss is as:

where λ is a hyper-parameter. LP indicates the promptsupervision loss.

Such a reconstruction network, which is similar to the regularized auto-encoder [Bengio et al., 2012], promotes our caption decoders to learn effective word-level representation.

Here, the caption decoder is equivalent to the encoder. And the reconstruction network is equivalent to the decoder. The reconstruction loss is helpful for regularizing the caption decoder during training, which reduces the risk of overfitting.

Besides, it is worth noting that the direction of the reconstruction network is opposite to that of the caption decoder. The goal of this processing is to promote the output representation generated by the caption decoder could embrace much information of both video content and the generating caption, which will be further demonstrated and discussed in the following experiment.

4 Experiments

In this section, we perform extensive experiments to evaluate the proposed methods. All results are evaluated by metrics of BLEU [Papineni et al., 2002], METEOR [Denkowski and Lavie, 2014], and CIDEr [Vedantam et al., 2015].

4.1 Extended Dataset and Implementation Details

MSRVTT-2016 [Xu et al., 2016] is the recently released largest dataset for video captioning. The dataset contains 10,000 web video clips and each clip is annotated with approximately 20 natural language sentences.

In order to demonstrate this attempt and evaluate the proposed methods, we extend this dataset. And we take the 1st to 4500th clip as the training set of the pre-trained model and use the 4501st to 5000th clip as the validation set. For ViCap models, we take the 5001st to 8500th clip as the training set. And we take the 8501st to 9000th clip as the validation set and use 9001st to 10000th clip as the test set. Besides, for the 20 annotations of the training set of ViCap models, we remove duplicate annotations, semantically similar annotations, and short annotations. For each clip of the validation and test set of ViCap models, we give three different semantic sentences as the annotations. Finally, when we evaluate

the performance of ViCap models, for each one of the annotations in the test set, we remove the first two words and take the remaining words as the ground-truth of evaluation. In Fig. 5, we show two examples which are from the test set. And the red words are the ground-truth of evaluation.

In this paper, we choose S2VT [Venugopalan et al., 2015] and HRNE [Pan et al., 2016] as examples of the pre-trained models to generate initial captions. Of course, we can also use other caption models as the pre-trained models. Finally, we take the model with the best performance on the validation set (4501st to 5000th video clips) as the initial caption generator. As two contributions of this paper are refined convolutional decoder (CNN-D) and convolutional reconstruction network (CNN-R), in order to compare fairly, we design an improved GRU decoder (IGRU-D) and an GRU reconstruction network (GRU-R).

In the following experiments, we select 20 equally-spaced frames from each video and feed them into GoogLeNet [Szegedy et al., ] to extract a 1,024-dimensional frame-wise representation. For the encoding network of both video and initial caption, the number of output channel is all set to 512. For CNN-D, the number of output channel of each layer is respectively set to 512, 256, 256, 512, and 512. For CNNR, the number of output channel of each layer is set to 512, 256, and 512. For IGRU-D and GRU-R, the number of output channel is set to 512. Finally, during training, we use Adam optimizer with an initial learning rate of 1×10-3 . λ1 and λ5 are respectively set to 0.4 and 0.6. β1, β2, and λ are respectively set to 0.6, 0.4, and 0.001. Note that we do not conduct beam search in testing.

The details of IGRU-D are shown as follows:

where Y t-1 G represents the word embedding result of the ground-truth word at time step t - 1. ϕt(HX) represents the visual attention [Yao et al., 2015]. The goal of ReLU operation is to keep hY and mt are positively related.

The details of GRU-R are shown as follows:

where oL-t represents the output of the decoder at time step L - t. L indicates the length of the generated caption. The reconstruction loss is defined as:

where Gt indicates the input of the decoder at time step t.

4.2 Performance of Full Supervision

Table 1 shows the performance of full supervision. We can see that based on different pre-trained models, the performance of convolutional decoder outperforms that of the IGRU decoder. This shows that convolutional decoder is effective. In Fig. 6, we show some examples of ViCap. We can see that when the initial caption does not meet our expectation, based on the given prompts, our method indeed generate more accurate descriptions. These not only show the proposed scene is meaningful but also demonstrate the effectiveness of the proposed method.

Ablation Analysis of H⊥

We first analyze H⊥. In Table 3, we show the comparative results. We can see that adding H⊥ in the decoder could improve the performance obviously.

The Effect of Different Types of Convolutional Loss

In order to analyze the effect of different types of loss, we make some comparative experiments. The results are shown in Table 4. Firstly, we can see that the performance of using the multi-layer loss outperforms that of only using the output layer loss obviously. This shows that using multi-layer loss really reduce the risk of vanishing gradients and improve the performance. Besides, for the multi-layer loss and the outlayer loss, we can see that adding the loss Ls improve the performance. This shows the loss Ls is effective.

4.3 Performance of Prompt Supervision

For the case of prompt supervision, we propose a GRU and convolution reconstruction network. Combined with the improved GRU decoder and convolutional decoder, there are four different architectures of prompt supervision. Table 2 shows their performance. We can see that the performance of the convolutional reconstruction network outperforms the GRU reconstruction network. This further demonstrates the effectiveness of convolutional architecture. Finally, the performance of the combination of the convolutional decoder and convolutional reconstruction network is the best.

The Effect of Reconstruction Loss

Here, we analyze the effect of the reconstruction loss. The results are shown in Table 5. We can see that for different decoder, adding reconstruction network could improve the performance obviously. For the generation of one natural language sentence, only using two ground-truth words (prompts) as the supervision information is not enough, which makes the network is prone to overfitting. Obviously, for our network, adding the reconstruction loss reduces the risk of overfitting and promotes the decoders to generate better wordlevel representation.

5 Conclusion

In this paper, we propose a new task of ViCap. Meanwhile, we devise ViCap models. Finally, the experimental results not only demonstrate the effectiveness of our proposed methods but also show the new task is meaningful.

Acknowledgements

This work is supported by the NSFC (under Grant 61876130, U1509206).