GL-RG: Global-Local Representation Granularity for Video Captioning

1 Introduction

Video captioning is of great societal relevance, holding values for many real-world applications, including subtitle generation, blind assistance and autopilot narration. However, isolated video frames may suffer from motion blur or occlusion, which introduces great confusion in visual understanding for the captioning task. Therefore, there is an urgent need to answer a principal problem: how to leverage the rich globallocal features regarding to cross-frame coherence and single frame information in video contents to close the gap from visual understanding to language expression?

∗Equal contributions.

†Corresponding author.

Despite making significant progress, existing methods for video captioning inadequately capture the local and global representation. Various works apply the deep neural network on raw pixels to build higher-level connections [Wang et al., 2019; Zhang and Peng, 2019]. These methods focus on local object features but neglecting object transformation or interaction. The effort of modeling local object features is a primitive solution for video captioning because the temporal connections across frames are not explored delicately and thus sensitive to the spurious association.

To study the problem of the global-local correlation, other related vision tasks leverage the graph representation using graph neural networks (GNNs). For instance, [Tsai et al., 2019] models object relations by using video spatiotemporal graphs and explicitly builds links between highlevel entities. Inspired by the above success, recent video captioning studies extend the graph-based approach and use GNNs to model global-local reasoning [Zhang et al., 2020b; Pan et al., 2020]. Among these works, [Zhang and Peng, 2019] merges local features with the global feature using concatenation; [Ghosh et al., 2020] adds spatio-temporal features as a separate node in the graph. However, empirical results indicate that using graphs to represent global-local correlation is suboptimal as it often encounters the over-smoothing problem in training which leads to weak results during inference. Alternatively, many video captioning methods intuitively exploit multi-modal fusion (i.e., visual or audio features) to enrich the feature representation in prediction [Rahman et al., 2019]. However, these simple "lumping" approaches inefficiently exploit multi-modal features and struggle to perform joint optimization cross-modality, leaving large room for improvement.

To address the aforementioned problems, we attempt to solve video captioning in a more flexible approach, which exploits the global-local vision representation granularity. Concretely, we make the following contributions:

• We devise a simple framework called GL-RG, namely the global-local representation granularity, which models extensive vision representations and generates rich vocabulary features based on video contents of different ranges.
• We propose a novel global-local encoder, which exploits rich temporal representation for video captioning. The encoder jointly encodes the long-range frames to describe spatio-temporal correspondence, the short-range frames to capture object motion and tendency, and the local keyframe to preserve finer object appearance and location details (Figure 1).
• We introduce an incremental two-phase training strategy. In the first seeding phase, we design a discriminative cross-entropy for non-reinforcement learning, which addresses the problem of human annotation discrepancy. In the second boosting phase, we adapt a discrepant reward for reinforcement learning, which stably estimates a bias of the expected reward for each individual video.
• We evaluate our approach on the MSR-VTT [Xu et al., 2016] and MSVD [Chen and Dolan, 2011] datasets. Extensive experimental results indicate that our method outperforms the latest best systems and uses shorter training schedules.

2 RelatedWork

Video captioning. Inspired by the success of other vision tasks, the seminal work [Venugopalan et al., 2015] extends the encoder-decoder architecture for the video captioning task. Following the same paradigm, [Chen et al., 2018; Venugopalan et al., 2015] explore the temporal patterns on video using attention mechanisms to depict object movements. [Pei et al., 2019b] devises a MARN method, which generalizes descriptions from a single video to other videos with high semantic similarity. [Hou et al., 2019] develops an idea of feature fusion to guide sentence generation for video content. Different from the existing efforts, we explicitly explore global-local representations for sentence generation.

Global-local representation. To model the global-local vision representation, many methods [Liu et al., 2021a; Zhang et al., 2021; Liu et al., 2020] resort to the sequence learning strategy. [Yao et al., 2015] uses a temporal attention method to depict the global-local connections. [Wang et al., 2019] leverages the decoding hidden states to increase the temporal feature representation. More recently, [Hu et al., 2019; Yang et al., 2017; Zhang and Peng, 2019] exploit the object features to model the object movement across frames. For instance, [Zhang and Peng, 2019] employs a bidirectional temporal graph to capture detailed movements for the salient objects in the video; [Hu et al., 2019] devises a stacked LSTM to encode both the frame-level and object-level temporal information. However, the above work primarily focuses on feature salience from the global contents with less consideration of the global-local representation reasoning. In contrast, we model global-local representations to achieve the lexical granularity, using long-range temporal correspondence, short-range object motion, and local spatial appearances on video contents.

Training strategies. A popular strategy for training video captioning models is "Teacher Forcing" [Williams and Zipser, 1989], which has been widely used in training video captioning tasks [Zhang et al., 2020a]. More recently, many research efforts attempt to explore different training methods to boost captioning performance [Wang et al., 2018b; Rennie et al., 2017; Hou et al., 2019; Ryu et al., 2021]. For instance, [Pasunuru and Bansal, 2017] uses a mixed loss function to optimize the video captioning algorithm, which leverages the weighted combination of cross-entropy and reinforcement learning. Similarly, [Rennie et al., 2017] adopts the paradigm of reinforcement learning and devises a selfcritical baseline to reward the model learning to train the video captioning network. Although demonstrating appealing supervision performance [Deng et al., 2021; Liu et al., 2021b], the above methods generally require a complicated pipeline to train with a computation overhead for optimization. Building on the lessons learned from the concurrent approaches, we propose an incremental training strategy, which can easily operate training on our proposed GL-RG. Empirical results indicate that our training strategy can serve as a good addition to the existing training scheme to boost a further training gain.

3 GL-RG

3.1 Overview

The framework of GL-RG is demonstrated in Figure 2. Following [Pan et al., 2020], GL-RG also adopts an encoderdecoder architecture. More specifically, we include a globallocal encoder and a captioning decoder. The global-local encoder selects frames of different ranges as inputs and encodes them into different vocabulary features. All the obtained features are aggregated together to enrich global-local vision representations across video frames. Afterwards, the captioning decoder supervised by the incremental training strategy translates the vocabulary feature into natural language sentences. We elaborate on the proposed GL-RG below.

3.2 Global-Local Encoder

Our global-local encoder includes three essential parts: longrange encoder, short-range encoder, and keyframe encoder

(see Figure 2). Collectively, our encoder can enrich globallocal vision representations for video captioning tasks.

Long-range encoder. We encode random global video frames to produce the global vocabulary based on a random keyframe ft in training. Note that our training iterations will fully saturate the whole video clips, since each iteration will randomly choose different frames (the total number is fixed) from the videos. Our long-range encoder first performs 2D convolutions on the inputs (i.e., ft-n and ft+n 1 ) to identify the relevant contextual features. The output features from the first step are processed by a 3D convolutional network (CNN) to capture global temporal correspondence. In order to increase consensus, we choose the top K word choices (highest frequency) from the ground truth sentences to guide the vocabulary generation as a K-classification task. Outputs of the dense layers are defined as:

where W is the collection of predicted long-range vocabulary. Since W includes all possibilities of word choices from the ground truths (i.e., MSR-VTT [Xu et al., 2016] and MSVD [Chen and Dolan, 2011]), it can offer the description of temporal contents in video. It is consisted of top K frequently used words (such as verbs, nouns and adjectives, excluding "is", "be" and "do", ext.) from all annotated GT sentences of all videos in the dataseta, i.e., MSR-VTT and MSVD.

Short-range encoder. Our short-range encoder is to capture object motion and tendency. Specifically, simultaneously taking two close neighbours (a.k.a. ft-10 and ft+10) of the keyframe, 2D CNN and 3D-Resnet18 [Tran et al., 2017] yield the semantic and movement representations respectively. Afterwards, these representations are stacked and fed into dense layers for action classifications. Given the highest frequency of J actions (in Kinetics-400, UCF101 or HMDB datasets), our output of the short-range encoder is the following.

1 where n is a random range larger than 25 frames.

where A is the collection of aj , which is the predicted confidence of the jth action from the short-range action dataset. Local-keyframe encoder. The lexical knowledge for the local semantics is learned by a residual network [Xie et al., 2017], which extracts salient object features from the keyframe ft. Given the number of image classes in the image classification dataset (e.g. ImageNet) is M, the output of our local encoder is:

where C is the collection of cm, which is the predicted confidence of the mth class for this local frame.

Once having all the vocabulary features from different ranges, we perform a fusion encoding. We first use a feature pool composed of linear layers φ to project each vocabulary feature into a same-size embedding and then aggregate them together to produce the fused feature F:

3.3 Captioning Decoder

Our captioning decoder translates the fused feature into a lword 2 sequence S = (s1, s2, ..., sj |j ∈ {1, ..., l}} to form the predicted sentence. Specifically, we use a LSTM to generate a hidden state ht and a cell state ct at the ith step:

2 l (i.e., 30 in our experiment) denotes the maximum length of a sentence.

where [·, ·] denotes concatenation. hi-1, si-1, ˆ si-1, F, and ci-1 are the previous hidden state, the predicted word, the ground truth, the fused feature from encoding, and the cell state respectively. Φ(·) is the annealing scheme which uses every previous token to predict the next word. We adopt a schedule sampling technique to randomly choose the token si-1 or ˆ si-1 using a random variable ξ ∈ {0, 1}:

when i = 0, the initial input of the LSTM is the fused feature F ; when i > 1, we increase the probability of ξ = 1 gradually in every epoch until ξ is absolutely equal to 1. Then, we counter the process by decreasing the probability of ξ = 1. Accordingly, the probability of a predicted word is:

where hi is the hidden state from Eq. (5) and Wo is the weight matrix which maps the hidden state hi to vocabulary-sized embedding, in order to find a context-matching word in the sentence.

3.4 Incremental Training

Our incremental training includes a seeding phase followed by a boosting phase (see Figure 2). The two training phases fulfill different learning objectives. The seeding phase aims to produce an entrance model to facilitate smooth training in the second phase, while the boosting phase leverages reinforcement learning (RL) to boost the performance gain.

Seeding phase. The existing models [Pei et al., 2019a; Zhang and Peng, 2019; Zhang et al., 2020b] are commonly trained with the cross-entropy (XE) loss, which measures the average similarity of the generated sentence and all the ground truth sentences. Since different annotators may interpret video content differently, the ground truth from the training dataset may include annotation bias. We argue that direct comparison between the captioning predictions to the ground truths cannot yield the optimal training outcomes. We thus employ the metric scores m( ˆ S) of all ground truths as a discriminative weight in computing cross-entropy to make our training biased towards those well-written ground truths.

Understandably, manually annotated ground truths have severe bias, that is, some ground truth sentences are wellwritten, while others are ambiguous or inappropriate. Metric scores encourage the training to focus on the well-written sentences. The m( ˆ S) can use different options, such as BLEU 4, METEOR, ROUGE L [Sharma et al., 2017], and CIDEr [Vedantam et al., 2015]. The analysis of each option will be reported in the experiments.

Providing each video is annotated by G sentences ˆS = { ˆ S(1) , ˆ S(2) , ..., ˆ S(G) }, the discriminative cross-entropy (DXE) loss function is:

where m() can be deemed as a constant, computed with every ground truth of each sentence. Our DXE loss increases the probability of generating captions with a high metric score by assigning higher weights to well-written ground-truth sentences. The gradient of DXE is calculated by the weighted difference between the prediction and all the target descriptions. Consequently, we DXE encourages feature learning which increases the probability of generating captions with a high metric score. The result of m() is considered a constant in our loss function. Every GT sentence has a different computed value. Intuitively, we want to encourage predictions with higher scores. Empirical results resonate with our assumption.

Different from the weighted loss entropy (which manually assign weights to all categories to address the problem of unbalanced data), the weight m( ˆ S) of our DXE is automatically calculated through metrics, evaluating the quality among all annotations. Our DXE assigns higher weights to high-quality annotations, helps the model generate captions closer to them. Boosting phase. After the seeding phase, we employ reinforcement learning with a discrepant reward (DR) to further boost the performance of our GL-RD model. To optimize the model parameters θ, conventional methods using reinforcement learning performs a non differentiable reward in training:

where ES∼p denotes the expected value of the distribution, the reward r(S) is the evaluation metric score of the sampled sentence, and F is the fused feature extracted from our global-local encoder. One problem with this training strategy is that the reward function r(S) is always positive because the metric score ranges between 0 and 1. Therefore, we can only encourage feature representations in learning but cannot perform suppression.

To address this issue, our DR is equal to the original reward r(S) subtracts a bias b, which is baseline. With the bias term, our learning can be more robust to variation in prediction. Then the policy gradient can be defined as:

where b ≈ E[r(S)]. Previous self-critical method SCST [Rennie et al., 2017] utilizes the reward of the greedy output at the test time as the baseline bscst, at the cost to run inference again in every training iteration. In our implementation, the baseline b has two variants: 1. b1 obtained by the G ground-truth captions; and 2. b2 the top Q sentences sampled by the model with the highest score during forward step. Note that either baseline (b1 or b2) can reduce the variance of the gradient without changing the expected value of gradient because of ∑S b∇θp(S|F; θ) = 0. When updating gradients, this gradient ∇θ can be approximated by Monte-Carlo sampling through a single training example. So the final gradient of our discrepant reward is:

where Sbj can be used by either baseline (b1 or b2). In our experiments, we carry out ablation studies to discover the impact of b1 and b2 on the captioning performance.

4 Experiments

4.1 Implementation Details

Dataset. We evaluate our GL-RG on MSR-VTT dataset [Xu et al., 2016]. Each video is associated with 20 ground-truth captions given by different workers. We follow the data split of 6513 videos for training, 497 videos for validation, and 2990 videos for testing. We also evaluate our GL-RG on the MSVD dataset [Chen and Dolan, 2011]. We split the dataset into a 1,200 training set, 100 validation set, and 670 testing set by the contiguous index number.

Evaluation Metrics. We evaluate our method on four commonly used metrics BLEU 4, METEOR, ROUGE L [Sharma et al., 2017], and CIDEr [Vedantam et al., 2015], which are denoted as B@4, M, R, and C respectively.

Training setup. Our long-range encoder is pre-trained on the video-to-words dataset (K=300 words) extracted fromMSRVTT or MSVD. Our short-range encoder is pre-trained on Kinetics-400 dataset [Carreira and Zisserman, 2017], which includes J=400 actions. Our Local-keyframe encoder is pretrained on ImageNet, which includes M=1000 objects. Our decoder is trained with the learning rate of 0.0003 in the seeding phase, and 0.0001 in the boosting phase. For each video, training is operated on 20 or 17 ground-truth captions for MSR-VTT or MSVD respectively.

4.2 Comparison with State-of-the-Arts

The evaluation results on the MSR-VTT dataset are shown in Table 1. With a much shorter training schedule, we can achieve an on-par performance with other state-of-the-art methods. Our fully trained model also surpasses all the compared methods on all metrics. In addition, when using the same level of supervision, our margins (model trained by XE) over the next best method (ORG-TRL [Zhang et al., 2020b]) are 1.9% on B@4, 0.4% on M, 0.5% on R, and 0.3% on C respectively. We can achieve further performance gain by using DXE on M, R, and C metrics. We further compare our GL-RG against some of the recent leading methods on the MSVD dataset (see Table 1). When trained by DXE, our margins over the next best method (O2NA [Liu et al., 2021b]) are 2.3% on B@4, 1.2% on M and 0.4% on R respectively. It is worth mentioning that our results of XE and DXE are from the 30th epoch (as seeding phase). It can be seen that they not only outperform those latest best systems, but also use shorter training schedules. Note that all the previous RL methods are trained with self-critical baseline bscst. Figure 3 demonstrates some qualitative examples.

4.3 Ablation Study

In this section, we conduct extensive ablation studies to analyze the effects of configurable components in GL-RG.

Global-local features. We measure the performances of our model using different global-local features (see in Table 2). At the higher level of Table 2, we evaluate the performance of different methods which use individual features for captioning prediction. Results indicate that using the long-range has the highest performance on all metrics. At the lower level of Table 2, we examine the impact of progressively combining different features together. Our full model using all three features outperforms all other counterparts.

Different weighted metrics in seeding phase. The seeding phase training is important as it produces the entrance model for the following boosting phase. Hence, we evaluate the impact of using different weighting metric (a.k.a. B@4, M, R, and C) in training (see Table 3). On R and C, models trained by different DXE loss all outperform the counterpart trained by XE. Meanwhile, using CIDEr as the metric weight in DXE training obtain the best results on all metrics.

Incremental training analysis. We investigate if the incremental training could effectively improve our method performance. Table 4 shows the results from the boosting phase increase steadily from the seeding phase. The boosting phase starting from the seeding phase using DXE gets higher scores than its counterpart using the XE. This proves that using DXE as the supervision in the seeding phase can yield more optimal model parameters and further improve the performance of the boosting phase than using XE in training.

Using b1 vs. b2 in the boosting phase. Table 5 shows the results of models using different discrepant (b1 and b2) rewards compared with self-critical baseline (bscst). Using the reward with b2 baseline based on top Q sentences sampled by the model can help our GL-RG to achieve a better performance than using the reward with b1 baseline which is based on G ground-truth sentences. Both models with b1 or b2 outperform methods without using the discrepant reward or with the self-critical reward (bscst).

5 Conclusion

Video captioning is an important research topic, which has various downstream applications. In this paper, we propose a GL-RG framework for video captioning, which leverages the global-local vision representation to achieve fine-grained captioning on video contents with an incremental training strategy. Experimental results on two benchmarks demonstrate the effectiveness of our approach. In future, we plan to explore dynamic weighting scheme to capture the preferences on different granularities. We also plan to investigate integrating more multi-modal information.