Dual Video Summarization: From Frames to Captions

1 Introduction

Video and language, as two kinds of sequence signals, provide a wealth of information for people's daily communication. A wild range task, such as video captioning [Chen and Jiang, 2019; Lin et al., 2021], video question answering [Yang et al., 2021], text-video retrieval [Gabeur et al., Summarized keyframes (4fs)

∗Corresponding author.

(c) Four keyframes captioning

Figure 1: Video contains quite a redundancy towards video captioning task. Compared to the original video captioning in (a), one single frame sampled from the video can generate a nearly semantic expression (b). In our work, we learn to summarize the most compact keyframes to achieve consistency in the form of language description (c).

2020; Bain et al., 2021] and video grounding [Anne Hendricks et al., 2017; Escorcia et al., 2019], has been designed due to the application potentials. Compared to the isolated image and abstract language description, videos are more engaging by conveying vivid and dynamic visual content. However, for video understanding and processing, it also contains quite a redundant.

Video summarization and video captioning are tasks dedicating to extract a concise content of video from the perspective of visual and linguistic modes. Video summarization aims to select keyframes or shots while video captioning devotes to generate the natural language descriptions to express the video content. Video captioning is also known as a text form of video summarization [Sanabria et al., 2018; Shang et al., 2021]. Considering the redundancy in the video caused by the frame similarity, existing video-and-language learning model commonly chooses multiple frames as model inputs instead of observing all. A conventional way to process the input video is to randomly sample several frames [Baraldi et al., 2017; Pei et al., 2019; Lei et al., 2021]. [Chen et al., 2018] choose the informative frames according to the visual representation of them, like visual summarization. These sampled deputies greatly improve computational efficiency, but are they represent the original video content enough and with no more redundancy? The crucial issue of finding out the most informative frames for video captioning is: In the video-language learning task, what is the essential visual content from the video? [Buch et al., 2022] and [Lei et al., 2022] both revisit this problem by single frame training scheme and verify the pre-trained model on the discriminative downstream task, i.e., Video QA and cross-modal video retrieval. The result from [Lei et al., 2022] reveals the existence of a strong "static appearance bias" in popular videoand-language datasets. Different from these discriminative tasks, video captioning is a generative task and requires a full coverage understanding of visual and temporal content. Single frame without any temporal clues is not satisfied to convey a comprehensive content for natural language expressing, as shown in Figure 1.

In this paper, we consider that the caption generated from the keyframes should closely parallel that from the original video. Motivated by this, we propose a dual video summarization framework and verify it in the context of video captioning to find a moderate strategy and number of sampled frames with accuracy and efficiency. Given the sampling frames as candidates, we finetune the pre-trained ViT model [Radford et al., 2021] based on the video-text retrieval task to extract the visual features. To select the most representative frames among them, we implement a cross-modal video summarization module as an auxiliary means to summarize the frames. We generate a pseudo score label of each frame as the reference to facilitate the summarizer. Then we sort the frames by their scores and select the Top K frames (K is no more than 3% of the entire video.) as the most compact and semantically consistent summary to represent the video. Moreover, to evaluate the static appearance and temporal information of video, we design the ranking scheme of video representation from two aspects: score-oriented and time-oriented. Finally, we generate the descriptions with a lightweight LSTM decoder. The experiment results on the MSR-VTT and MSVD dataset reveal that, for the generative task as video captioning, a small number of keyframes can convey the same semantic information to perform well on captioning, or even better than the original sampling.

In summary, our contributions are three-fold:

• We cooperate the video summarization and video captioning task with each other to investigate the video frame representation problem. We propose a dual video summarization framework that select accurate and compact keyframes without frame-level annotation.
• We design a semantic-consistency video summarization module to assist the video captioning. We utilize clipscore between the visual feature and text embedding as the pseudo label to facilitate the score learning module.
• We evaluate our model on the video captioning benckmarks MSR-VTT and MSVD. We find that, for the generative task as video captioning, a small number of keyframes can convey the same semantic information and is able to perform well in the captioning task, or even better than the original sampling.

2 RelatedWork

2.1 Video Captioning

Video captioning is a challenging task of yielding corresponding natural language description for a given video. In the past few years, the field of video captioning has been obtained great advancement with lots of newly proposed method. Existing works of video captioning mainly adapt an encoder-decoder framework. [Venugopalan et al., 2015] firstly exploits LSTMs to learn the temporal structure of videos and then generate descriptions. [Pan et al., 2016] proposes a hierarchical encoder which takes a series of visual feature sequences into a single vector as the main representation of the whole video. Following the same paradigm, [Baraldi et al., 2017] encodes semantic content and video frames in a trainable encoding layer. [Chen and Jiang, 2019; Zhang and Peng, 2020] employ temporal and spatial attention for tackling video feature alignment and aggregation. [Song et al., 2017] decides whether to depend on the visual information or the semantic context information especially when generating non-visual words(e.g."a", "the"). [Yan et al., 2022] produces rich semantic vocabulary to obtain description of video contents from the proposed global-local representation granularity framework. [Gao et al., 2022] tries to resolve the disconnection between offline extracted motion or appearance features and sentence generation by a dual-level transformer with image-text pre-training models. For improving model's generation efficiency and effectiveness, [Chen et al., 2018] proposes frame selecting strategy to decrease input redundancy with little performance drop. In this paper, unlike previous works [Chen and Jiang, 2019; Zhang and Peng, 2020; Yan et al., 2022; Gao et al., 2022] use densely extracted offline features or complicated architecture, we rethink reducing video's content redundancy by learning to sample high informative frames from already sparsely sampled candidates as well as maintaining our model's lightweight.

2.2 Video Summarization

Video summarization aims to generate a subset of informative frames that can present the main contents from video sequences. Early works[Borji and Itti, 2012; Gygli et al., 2013; Gygli et al., 2014; Zhao and Xing, 2014] mainly concentrate on designing handed-craft video representation(e.g.visual attention, video interest) in unsupervised manner. [Gygli et al., 2014] assigns frame scores according to multi level features and the summary would be selected as the optimal subset of them. [Zhou et al., 2018] formulates video summarization as a sequential decision-making process and design reinforcement learning framework that doesn't rely annotated frame level labels. Some works take multimodal interacting into account. [Song et al., 2015; Qiu et al., 2022] bring in textual sources such as video's title or supplied articles to help key frame location while [Xiao et al., 2020; Narasimhan et al., 2021] consider users' preference and provide more fine-grained summarization through integrating query into visual features. In this work, we attempt to select frames on the unlabeled video captioning dataset by contentaware supervised learning.

3 Framework

The proposed dual video summarization framework follows the pipeline with three components: an encoder, a summarizer and a decoder, as shown in Figure 2. The encoder extracts visual features from candidate frame samples. The summarizer selects the most representative and compact frames according to the frame scores, while the decoder generates natural language descriptions based on the selected keyframes.

3.1 ProblemFormulation

Given an input video V with totally T frames, we follow the common measures to sample N frames and represent them as visual feature F = {f1, f2, ..., fN}, fi ∈ Rdf with equal time spacing. The target of our framework to select a subset Fkey from F with K frames (K < N&K ≪ T ) as keyframes' features, while the caption generated from the keyframes should closely parallel that from the original video.

For each video in the training set, there are M captions represented by the text embedding C = {c1, c2, ..., cM}, ci ∈ Rdc . Each caption is a sentence (i.e., word sequence) Y = {y1, y2, ..., yW} to express the video's content.

3.2 Encoder

Considering the redundancy of video caused by the frame similarity, existing video-and-language learning model commonly chooses multiple frames as model inputs instead of observing all. This multi-frame training strategy has been the norm and is shown to work well [Lei et al., 2022]. We follow this procedure and sample the candidate frames time equally. The visual feature of each frame is extracted by a fine-tuned ViT model [Dosovitskiy et al., 2020]. The outputs of encoder are defined as fi, i ∈ [1, ...,N], where N denotes the length of input frames.

We adapt pre-trained CLIP [Radford et al., 2021] with 12 layers' ViT-B/32 as our visual encoder. Although CLIP has been pre-trained on 400M image-text pairs, we tend to narrow the gap between videos and images by performing video-text retrieval task to fine-tune CLIP's parameters. Many previous works, such as [Chen and Jiang, 2019; Zhang et al., 2021; Yan et al., 2022], take off-line CNNs pre-trained on other datasets to extract temporal representation or object features. Since the differences in data distribution, these operations might suffer a disconnection between the target task and the pre-trained domain [Gao et al., 2022]. Video-text retrieval is a coarse-grained multi-modal task compared with video captioning. Thus we use it to fine-tune the pre-trained CLIP(ViTB/32) on the mainstream captioning datasets and seek to weaken the disconnection while not hurting the visual representation ability of the model.

For the fine-tuning, ViT first reshapes images into flattened patches. Then the 2D patches would be further flattened and mapped to 1D vectors through trainable linear projection for adjusting standard Transformer [Vaswani et al., 2017]. The specific prepend token [CLS] interacting with each input patch is regarded as the image representation. Following [Luo et al., 2021], we average the generated features along the temporal dimension and get the video representation ˆ f by average pooling. We directly apply CLIP's text encoder to output caption embeddings cj, j ∈ [1, ...,M] corresponding to the given video. The visual-language similarity function s( ˆ f , c) can be defined as

where tr denotes vector transposition .

3.3 Summarizer

For the traditional video summarization task, the evaluation metrics are the precision, recall and F-score. These metrics all require supervised frame-level annotations as ground truth, which limits to summarize the video without manually labels.

In this work, we evaluate the video summarization result from a high-level semantic aspect. The language description generated from the image or video is the abstractive summary of the visual content. If the captions generated from the video and keyframes are consistent, then they convey the same semantic information for human cognitive understanding.

To learn the summarizer, we leverage the local selfattention module to capture semantic relation among all the frames as well as outputting predicted scores for these frames following [Xiao et al., 2020]. Given the encoder output F = {f1, f2, ..., fN}, we compute the relation score map as:

where P,W1,W2 ∈ Rdf×dc are parameter matrices and b is bias vector, df and dc are the dimension of input features and outputs. Note that the score map with shape N ×N ×df means the features not only interact with each other along the temporal perspective but also their inner dimension. Then we can get the local attentive visual feature f att i by

where ⊙ denotes element-wise multiplication.

Finally, we get the predicted score scrp i , i ∈ [1, ...,N] of each frame using a trainable MLP after processing the attentive visual feature employing a residual connection and GeLU activation

We choose the Top K frames according to the score ranking as the video summarization. In the practices, K = 4 achieves the best performance, which only accounts for 2 ∼ 3% for the original video (T > 200).

To train the summarizer, we generate a pseudo score label of each frame to facilitate video summarization. Concretely, we compute cosine similarity score for each sampled frame in the video with all the video-related texts, then average them as the current frame's visual correlation. Here we choose the clipscore [Hessel et al., 2021], a reference-free metric reaching the highest correlation with human judgements, to assess the image-caption compatibility. This metric is outperforming existing reference-based metrics like CIDEr and SPICE. We slightly modify the definition of clipscore and formulate it as:

where M denotes the number of references and mj is the jth references. This pseudo score scrc i is regarded as the ground truth label when training the summarizer based on the binary cross-entropy loss function

3.4 Decoder

The summarizer selects the Top K frames as the video summarization and the features are Fkey = {f1, f2, ..., fK}, fi ∈ Rdc . As a feature sequence, we concatenate the key frame features by two strategies, i.e., score-oriented and timeoriented. We use two ranking mechanisms to decide the elements' position depending on their temporal position or predicted scores. The vector Vs = [f1, f2, ..., fK], Vs ∈ R1×(K×dc) is the final representation of the input video .

The captioning decoder is a light weight LSTM which produces a hidden state hi and a cell state celli at the ith step,

where hi-1, yi-1, ˆ yi-1 and celli-1 are the previous hidden state, the predicted word, the ground truth and the cell state respectively. [.; .] denotes the concatenation. We introduce scheduled sampling method [Bengio et al., 2015] to solve the inconsistent distribution of input representation during training. Concretely, Φ(·) can randomly choose yi-1 or ˆ yi-1 as the ith input token. As the training epoch increasing, Φ(·) tend to choose yi-1 and the initial input of LSTM is the reshaped representation Vs when i=0.

Our objective function for the decoder with trainable parameters θ is formulated as:

where Φ(·)1:t-1 denotes the above scheduled sampling sequences.

We also adapt discriminative cross-entropy(DXE) [Yan et al., 2022] as the learning objective. Each video is attached with M captions ˆ Y = {Y1, Y2, ..., YM}, the qualities of the captions is not equivalent which can be evaluated by metric scores m( ˆ Y ) pre-computed using BLEU@4 or CIDEr, m( ˆ Y ) serves as the discriminative weight in cross-entropy loss to promote the model to more concentrate on high-quality captions. The DXE loss function is formulated as:

3.5 Training

The training procedure is split into two stages. In the first stage, we train the summarizer module with our automatic content-aware scores to choose a subset of frames containing more visual diversity and less noise. In the second stage, we freeze the summarizer's parameters while update the captioning decoder. The two-stage training strategy makes the both module more stable. We consider not every frame is possible for a video especially when captioning, our summarizer module pre-excludes most of video's inherent redundancy and the following captioning would suffer from less interference.

4 Experiments

4.1 Implement Details

Dataset. We evaluate our model on MSR-VTT [Xu et al., 2016] and MSVD [Chen and Dolan, 2011] datasets.

• MSR-VTT is a large-scale open domain dataset. It contains 10K videos crossing a wide range categories including music, game, sports and movie. Each video is annotated with 20 references. The duration of each video in MSR-VTT is between 10 and 30 seconds. We split the data into a 6,513 training set, 497 validation set and 2,990 testing set.
• MSVD has 1,970 Youtube videos. This dataset mainly contains short video clips with a single action, and the average duration is about 9 seconds. We follow the data split of 1,200 videos for training, 100 videos for validation and the rest for testing. The number of references of each video in MSVD dataset is not fixed and we set the the number to 17 following [Yan et al., 2022].

Evaluation Metrics. We use four universal metrics for evaluation: BLEU@4, ROUGE-L, METHOR and CIDEr [Vedantam et al., 2015], which are denoted as B@4, M, R, and C respectively. We mainly compare CIDEr as the previous video captioning works.

Training setup. Our encoder is adapted ViT model finetuned on video-text retrieval task. Our summarizer module is trained with 10 epochs on the above datasets' with learning rate 1e-3 and dropout 0.2. Our captioning module is trained with learning rate 1e-4 and 40 epochs, and we set the batch size to 32. Both the summarizer and captioning decoder employ Adam optimizer [Kingma and Ba, 2014] to minimize the loss. The candidate frames number is set to 12 with timeequally sampling and we set the maximum sequence length to 30 following [Luo et al., 2021] and [Yan et al., 2022] respectively. The dimension of visual embeddings and text embeddings is 512.

4.2 Comparison to State-of-the-art

The performance of our proposed framework and other topperforming baselines are presented in Table 1. In the practice, we only summarize four keyframes to present each video. We compare the XE training result with other XE-based methods. For the DXE training results, we compare them with DXEbased method and reinforcement learning methods.

As it can be observed, our model achieves best performances on all metrics over two benchmarks under the XE training. The CIDEr score of our model reaches 55.0, which achieves increments of 4.0% and 6.9% to strong models RCG [Zhang et al., 2021] and GL-RG [Yan et al., 2022]. Moreover, our model achieves improvements of 1.1% and 1.3% on R and M respectively while bringing into correspondence on B@4 with previous best results. Under DXE training, our margins over [Yan et al., 2022] are 0.3% on M, 0.5% on R and 5.1% on C. It's worth mentioning that we only use ViT feature, while other methods employ various features from visual to temporal. During all the training stages, we do not employ any temporal information such as 3D temporal visual feature. This reveals that, for the video understanding, there is quite a redundancy and the visual appearance is much more essential than the temporal information.

Compared to the MSR-VTT, our model achieves a more superior performance on the MSVD over all metrics. Under the XE training, our advancements over the second best results are 15.7% on B@4, 9.5% on M, 5.9% on R, and especially 23.1% on C. DXE training decreases the model's performance on MSVD unexpectedly although still surpasses the other methods in a large margin, e.g. 10.4% improvements on C over [Gao et al., 2022].

4.3 Ablation Study

We conduct several ablation studies to quantify the influences of different configuration of our model.

Summarized frames. We measure different K numbers of summarized frames. We list the results of K from single to 12 in Table 2. It is shown that, K = 4 achieves the best results. We notice that the model trained with only two frames can exceed many previous state-of-the-art results. Even the single frame selected by our model produces considerable captioning performance. The metric scores maintain growth as the input frames increasing on MSR-VTT dataset, but the rating is gradually declined, using more frames(e.g. 8) can not bring corresponding improvements(55.4 CIDEr vs. 55.0 CIDEr) or even produce negative effect(45.4 B@4 vs. 45.5 B@4) when compared with 4 frames input. As for MSVD dataset, 4 frames is the optimal input, which produce the most obvious advancement on B@4 and C. One possible reason is that the noise frames existed in videos affect the training of the model. We set the number of input frames to 4 as the final configuration. In case the improvement is benefited by the feature dimension extension from the frame increasing, we verify the performance by duplicating the features into the same dimension. The range of dimension is {512, 1024, 2048, 4096} corresponding to {1, 2, 4, 8} frames. The results is in Table 3. 'X→Y' means X frames are expanded to Y by simple temporal duplication to obtain identical dimension and not provide additional information. It can be observed '1→Y' could improve the performance when compared with single frame input while still dropped behind selected Y (line 1vs line 3 and line 5 vs. line 7). It's apparent that the latter contains more visual information and the content in single frame is quite weak. We also confirm the above conclusion that 4 frames input is ultimate (line 2 vs. line 5 and line 3 vs. line 6).

Video representation. We elaborate the influence of ranking order and fineturned ViT in Table 4. As we can see(line 1 vs. line 2), score-oriented video representation, which completely overlooks temporal information, is better than the time-oriented, increasing B@4 by 2.2% and C by 2%. This finding reaches an agreement with "static appearance bias" [Lei et al., 2022] existing in the popular video captioning datasets. Finetuned ViT (line 1 vs. line 3 and line 6 vs. line 8) helps reduce the gap between images and videos and achieves improvement of 3.2% and 7.2% on C respectively. Notice that the finetuning process has more obvious effect on MSVD dataset, we assume the reason is that the video in MSVD is attached with average 36 captions while 20 captions in MSR-VTT during finetuning. The larger scale videocaption pairs lead to the better visual representation ability for ViT.

Sampling method. We investigate the influence of using different frame sampling method in Table 5. We introduce Clip [Radford et al., 2021] embedding relation score to indicate correlation among frames, we calculate the cosine similarity of extracted embeddings then sum them along the temporal dimension, that is to say, the value report the degree of correlation between the frame and the others. Intuitively, people prefer these frames appeared more frequently in the video when labeling the importance of them, here we consider the frames with high relation score coincide this conclusion. We choose the input subset based on maximal or minimal K(e.g.4) values which are denoted as Clip-Max and ClipMin respectively. Randomly and uniformly sampling produce almost the same results, Clip-Max get higher scores on B@4 and R. The fourth line(Clip-Min) shows that the frames appeared rarer can not cover the main content compared with Clip-Max. The last line shows the importance of frame is not only determined by their frequency and our model using content aware labeling in consideration of visual and lingual perspective can find approximately optimal subset from a series of frames.

4.4 Qualitative Results

Figure 3 shows the qualitative examples of our method. As indicated by the examples, with only 4 selected frames input, our method can generate more accurate captions like the lower sub-figure, while GL-RG[Yan et al., 2022] produces wrong description like 'a woman'. Figure 4 shows the clipscore labeling scores of two random sampled videos on MSR-VTT dataset. And the frames of (a) and (b) in Figure 5 are from the TOP 4 of labeling scores in Figure 4. We notice higher scores normally indicate the frames are dominant in the whole video which report the main content.

5 Conclusion

In this paper, we cooperate the video summarization and video captioning task with each other to investigate the video frame representation problem. We propose a dual video summarization framework composed of an encoder, a summarizer and a decoder. By verifying it in the context of video captioning, for the generative task as video captioning, a small number of keyframes can convey the same semantic information. This reveals the redundancy and frame bias of video captioning benchmarks.

Acknowledgements

This work was supported by the NSFC NO. 62172138 and 61932009. This work was also partially supported by The University Synergy Innovation Program of Anhui Province NO. GXXT-2021-007.