Perceptual Learned Video Compression with Recurrent Conditional GAN

1 Introduction

Recent years have witnessed the increasing popularity of endto-end learning-based image and video compression. Most existing models are optimized towards distortion, i.e., PSNR and MS-SSIM. Recently, Generative Adversarial Networks (GANs) have been used in image compression, to boost perceptual quality [Agustsson et al., 2019; Mentzer et al., 2020]. However, efficient perceptual video compression still remains limited.

In this paper, we propose a Perceptual Learned Video Compression (PLVC) approach with recurrent conditional GAN, in which a recurrent generator and a recurrent conditional discriminator are trained in an adversarial way for perceptual compression. The recurrent generator contains recurrent auto-encoders to compress video and generate visually pleasing reconstructions in the adversarial training. More importantly, we propose a recurrent conditional discriminator, which judges between raw and compression video conditioned on the spatial-temporal features, including latent representations, motion and the hidden states transferred through recurrent cells. Therefore, in the adversarial training, the discriminator is able to force the recurrent generator to reconstruct both photo-realistic and temporally coherent video, thus achieving good perceptual quality.

HEVC (HM 16.20), bpp = 0.081

Figure 1: Example of the proposed PLVC approach in comparison with the official HEVC test model (HM 16.20).

Fig. 1 shows the visual result of the proposed PLVC approach on BasketballDrill (bpp = 0.0730) in comparison with the official HEVC test model HM 16.20 (bpp = 0.0813). The top of Fig. 1 shows that our PLVC approach achieves richer and more photo-realistic textures than HEVC. At the bottom of Fig. 1, we show the temporal profiles by vertically stacking a specific row (marked as green) along time steps. It can be seen that the result of our PLVC approach exhibits a temporal coherence comparable with that of HEVC, while offering more detailed textures. As a result, we outperform HEVC on perceptual quality at lower bit-rates. The contributions of this paper are summarized as:

• We propose a novel perceptual video compression approach with recurrent conditional GAN, which learns to compress video and generate photo-realistic and temporally coherent compressed frames.

• We propose adversarial loss functions for perceptual video compression to balance bit-rate, distortion and perceptual quality.
• The experiments (including user study) show the outstanding perceptual performance of our PLVC approach in comparison with the latest learned and traditional video compression approaches.

2 RelatedWork

Learned image compression. In the past years, learned image compression has gained in popularity. For instance, Ball´e et al. proposed utilizing the variational auto-encoder for image compression and proposed the factorized [Ball´e et al., 2017] and hyperprior [Ball´e et al., 2018] entropy models. Later, the auto-regressive entropy models [Minnen et al., 2018; Lee et al., 2019; Cheng et al., 2019] were proposed to improve the compression efficiency. Recently, the coarse-tofine model [Hu et al., 2020a] and the wavelet-like deep autoencoder [Ma et al., 2020] were designed to further advance the rate-distortion performance. Besides, there are also the methods with RNN-based auto-encoder [Toderici et al., 2017; Johnston et al., 2018] or conditional auto-encoder [Choi et al., 2019] for variable rate compression. Moreover, Agustsson et al. [2019] and Menzter et al. [2020] proposed applying GANs for perceptual image compression to achieve photorealistic compressed images at low bit-rate.

Learned video compression. Inspired by the success of learned image compression, the first end-to-end learned video compression method DVC [Lu et al., 2019] was proposed in 2019. Later, M-LVC [Lin et al., 2020] extended the range of reference frames and beats the DVC baseline. Meanwhile, a plenty of learned video compression methods with bi-directional prediction [Djelouah et al., 2019], one-stage flow [Liu et al., 2020], hierarchical layers [Yang et al., 2020a] and scale-space flow [Agustsson et al., 2020] were proposed. Besides, methods like the content adaptive model [Lu et al., 2020], resolution-adaptive flow coding [Hu et al., 2020b], the recurrent framework [Yang et al., 2021] and feature-domain compression [Hu et al., 2021] were employed to further improve compression efficiency. All these methods are optimized for PSNR or MS-SSIM. Veerabadran et al. [2020] proposed a GAN-based method for perceptual video compression, but has not outperformed x264 and x265 on LPIPS. Therefore, video compression aimed at perceptual quality still offers room for improvement.

3 Proposed PLVCApproach

3.1 Motivation

A GAN was first introduced by Goodfellow et al. [2014] for image generation. It learns to generate photo-realistic images by optimizing the adversarial loss

where f and g are scalar functions, and G maps the prior y to px. We define x as the groundtruth image and ˆx = G(y) as the generated image. The discriminator D learns to distinguish ˆx fromx. In the adversarial training, it pushes the distribution of generated samples pˆx to be similar to px. As a result, G learns to generate photo-realistic images.

Later, the conditional GAN [Mirza and Osindero, 2014] was proposed to generate images conditioned on prior information. Defining the conditions as c, the loss function can be expressed as

with ˆx = G(y). The goal of employing c in (2) is to push G to generate ˆx ∼ pˆx|c with the conditional distribution tending to be the same as px|c. In other words, it learns to fool D to believe that ˆx and x correspond to a shared prior c with the same conditional probability. By properly setting the condition prior c, the conditional GAN is expected to generate frames with desired properties, e.g., rich texture, temporal consistency and coherence, etc. This motivated us to propose a conditional GAN for perceptual video compression.

3.2 Framework

Fig. 2 shows the framework of the proposed PLVC approach with recurrent conditional GAN. We define the raw and compressed frames as {xi}Ti=1 and {ˆxi}Ti=1 , resp. In our PLVC approach, we first estimate the motion between the current frame xi and its previous compressed frame ˆxi-1 by the pyramid optical flow network [Ranjan and Black, 2017]. The motion is denoted as mi, which is to be used in the recurrent generator (G) for compressing xi and also to serve as one of the temporal conditions in the recurrent conditional discriminator (D). The architectures of G and D in our PLVC approach are introduced next.

Recurrent generator G. The recurrent generator G plays the role as a compression network. That is, given the reference frame ˆxi-1 and the motion mi, the target frame xi can be compressed by an auto-encoder-based DNN, which compresses xi to a latent representation yi and outputs the compressed frame ˆxi. In our PLVC approach, we employ the recurrent compression model RLVC [Yang et al., 2021] as the backbone of G, which contains the recurrent auto-encoders to compress the motion and the residual, and the recurrent probability model P for the arithmetic coding of yi. The detailed architecture of G is shown in the Supplementary Material 1 .

Recurrent conditional discriminator D. The architecture of the proposed recurrent conditional discriminator D is shown in Figure 2. First, we follow [Mentzer et al., 2020] to feed yi as the spatial condition to D. This way, D learns to distinguish the groundtruth and compressed frames based on the shared spatial feature yi, and therefore pushes G to generate ˆxi with spatial features similar to xi.

More importantly, the temporal coherence is essential for visual quality. We insert a ConvLSTM layer in D to recurrently transfer the temporal information along time steps. The hidden state hDi can be seen as a long-term temporal condition fed to D, facilitating D to recurrently discriminate raw and compressed video taking temporal coherence into consideration. Furthermore, it is necessary to consider the temporal fidelity in video compression, i.e., we expect the motion between compressed frames to be consistent with that between raw frames. For example, the ball moves along the same path in the videos before and after compression. Hence, we propose D to take as inputs the frame pairs (xi, xi-1) and (ˆxi, ˆxi-1) and make the judgement based on the same motion vectors mi as the short-term temporal condition. Without the mi-condition, G may learn to generate photo-realistic (ˆxi, ˆxi-1) but with incorrect motion between the frame pair. This leads to poor temporal fidelity when compared to the groundtruth video. This motivates us to include the temporal conditionmi as an input to the discriminator.

1 https://arxiv.org/abs/2109.03082

In summary, we propose to condition the discriminator D on spatial features yi, short-term temporal features mi and long-term temporal features hDi-1 . Given these conditions, we have c = [yi,mi, hDi-1 ] in (2) in our PLVC approach. The output of D can be formulated as D(xi, xi-1 | yi,mi, hDi-1 ) and D(ˆxi, ˆxi-1 | yi,mi, hDi-1 ) for the raw and compressed samples, resp. When optimizing the adversarial loss on sequential frames, the compressed video {ˆxi}Ti=1 tends to have the same spatial-temporal features as the raw video {xi}Ti=1 . Therefore, we achieve perceptual video compression with spatially photo-realistic as well as temporally coherent frames. Please refer to the Supplementary Material for the detailed architecture of D.

It is noteworthy that the effectiveness of D is not restricted to a specific G, but generalizes to various compression networks. Please refer to the analyses and results in Section 4.6.

3.3 Training Strategies

Our PLVC model is trained on the Vimeo-90k [Xue et al., 2019] dataset. Each clip has 7 frames, in which the first frame is compressed as an I-frame, using the latest generative image compression approach [Mentzer et al., 2020], and other frames are P-frames. To train the proposed network, we first warm up G on by the rate-distortion loss

In (3), R(·) denotes the bit-rate, and we use the Mean Square Error (MSE) as the distortion term d. Besides, λ is a hyperparameter to balance the rate and distortion terms.

Then, we train D and G alternately, with the loss function combining the rate-distortion loss and the non-saturating [Lucic et al., 2018] adversarial loss. Specifically, the loss functions are expressed as follows:

In (4), α, λ′ and β are hyper-parameters to control the tradeoff of bit-rate, distortion and perceptual quality. We set three target bit-rates RT , and set α = α1 when R(yi) ≥ RT , and α = α2 ≪ α1 when R(yi) < RT to control bit-rates. The hyper-parameters are shown in Table 1. Fig. 3 illustrates the convergence curves of the bit-rates. It shows that the bit-rates of PLVC models are controlled to the targets during training.

4 Experiments

4.1 Settings

We use the JCT-VC [Bossen, 2013] (Classes B, C and D) and the UVG [Mercat et al., 2020] datasets as the test sets. On perceptual metrics, we compare with the Low-Delay P mode of the official HEVC model (HM 16.20) and the latest open-sourced 2 learned compression approaches DVC [Lu et al., 2019; Yang et al., 2020b] 3 , HLVC [Yang et al., 2020a], M-LVC [Lin et al., 2020] and RLVC [Yang et al., 2021], and the GAN-based method [Veerabadran et al., 2020]. We also report the MS-SSIM and PSNR results in comparison with several existing approaches. Besides, we conduct MOS experiments as user studies to evaluate perceptual quality. There are 12 non-expert subjects participating in the MOS experiments. They are asked to rate the compressed videos with a score from 0 to 100 according to subjective quality. Higher score indicates better perceptual quality.

2 We need the codes to reproduce the videos for the user study.

3 In this paper, we use OpenDVC [Yang et al., 2020b] to reproduce the results of DVC.

4.2 Numerical Performance

To numerically evaluate the perceptual quality, we calculate the Learned Perceptual Image Patch Similarity (LPIPS) [Zhang et al., 2018], Fr´echet Inception Distance (FID) [Heusel et al., 2017] and Kernel Inception Distance (KID) [Bi´nkowski et al., 2018] on our PLVC and other approaches. In Fig. 4, we observe that the proposed PLVC approach outperforms all compared methods in terms of all three perceptual metrics. Especially, our PLVC approach reaches comparable or even better LPIPS, FID and KID values than other approaches with 2× to 4× the bit-rates of ours. This indicates that PLVC is able to compress video at low bitrates but with good perceptual quality, showing the efficiency of PLVC for perceptual video compression.

Moreover, Fig. 5 shows that the PSNR of our PLVC approach is comparable with DVC. Our MS-SSIM result beats DVC and is competitive with Lu et al. [2020]. This shows that PLVC is able to maintain the fidelity to an acceptable degree when optimized towards perceptual quality.

4.3 Visual Results and User Study

Fig. 6 shows the visual results of our PLVC approach in comparison with HM 16.20 and the latest learned approach RLVC

(MS-SSIM optimized). The top of Fig. 6 illustrates spatial textures, and the bottom shows temporal profiles by vertically stacking the rows along time. It can be seen from Fig. 6 that PLVC yields richer and more photo-realistic textures at lower bit-rates than the methods compared against. Moreover, the temporal profiles indicate that PLVC maintains the temporal coherence comparable to the groundtruth. We provide more visual examples in the Supplementary Material 4 .

In the user study, the rate-MOS curves and the standard deviation of MOS for the JCT-VC dataset are shown in Fig. 7. As we can see from Fig. 7-(a), PLVC successfully outperforms the official HEVC test model HM 16.20, especially at low bit-rates, and we are significantly better than the MSSSIM-optimized RLVC. Also, Fig. 7-(b) shows that the standard deviation of MOS values of our PLVC approach is relatively lower than for HM 16.20 and RLVC, indicating that PLVC achieves a more stable perceptual quality. In conclusion, these results confirm that people tend to prefer the perceptual quality of PLVC over that of HM 16.20 and RLVC.

4.4 Ablation Studies

In our ablation studies, we analyze the effectiveness of the generative adversarial network, the recurrent structure of the proposed GAN, and the spatial-temporal conditions mi and yi. We provide ablation analysis in Fig. 8 in terms of both LPIPS and the MOS of an ablation user study. Note that, the ablation user study is conducted independently of the user study in Fig. 7. The subjects in these two user studies are not the same. Therefore, the MOS values can only be compared within each figure.

4 https://arxiv.org/abs/2109.03082

Generative adversarial network. We illustrate the results of the distortion-optimized PLVC model, denoted as PLVC (w/o GAN) in Fig. 9, compared with the proposed PLVC approach. The model of PLVC (w/o GAN) is trained only until the convergence of (3) without the adversarial loss in (4). It can be seen from Fig. 9 that the proposed PLVC approach achieves richer textures and more photo-realistic frames than PLVC (w/o GAN), even when PLVC (w/o GAN) consumes 2× to 3× bit-rates. This is also verified by the LPIPS and MOS performance in Fig. 8.

Recurrent GAN. We first remove the recurrent cells in the proposed GAN (PLVC (w/o hi)). As we can see from Fig. 10, the temporal profile of PLVC (w/o hi) (third column) is obviously distorted in comparison with the proposed PLVC approach and the groundtruth. As shown in Fig. 8, the LPIPS and MOS performances of PLVC (w/o hi) also degrade in comparison with the proposed PLVC model.

Temporal conditionmi inD. We further remove the temporal condition mi from D and denote it as PLVC (w/o hi, w/o mi in D). As such, D becomes a normal discriminator independent along time steps. It can be seen from the right column of each example in Fig. 10 that the temporal coherence becomes even worse when further removing the mi condition in D. Similar result can also be observed from the quantitative and MOS results in Fig. 8. These results indicate that the long-term and short-term temporal conditions hi and mi are effective in enabling D to judge raw and compressed videos according to temporal coherence, in addition to spatial texture. This way, it is able to force G to generate temporally coherent and visually pleasing videos, thus resulting in good perceptual quality.

Note that in Fig. 8 the MOS values of PLVC (w/o hi) and PLVC (w/o hi, w/o mi in D) are even lower than those of PLVC (w/o GAN) at some bit-rates. This is probably because the incoherent frames generated by PLVC without temporal conditions hi and/or mi severely degrade the perceptual quality, making it worse than the distortion-optimized model.

Spatial condition yi in D. Fig. 8 shows that when further removing yi from the conditions of D (denoted as PLVC (w/o hi, w/omi, w/o yi in D)), the performance in terms of LPIPS degrades further. We also show visual results in Fig. 11. It can be seen that, compared to PLVC (w/o hi, w/o mi in D), the compressed frames generated by PLVC (w/o hi, w/o mi, w/o yi in D) exhibit more artifacts and color shifts (the right example in Fig. 11). This suggests that the spatial condition in D is effective for pushing the generator G to generate compressed frames with spatial features closer to the groundtruth. It boosts the fidelity of the compressed video and improves the visual quality.

4.5 GOP Structure and Error Propagation

In PLVC, we follow RLVC to use the bi-IPPP [Yang et al., 2021] structure with the GOP size of nine frames (one I-frame and eight P-frames). Fig. 13 shows the average LPIPS-bpp curve for each of the eight P-frames in GOPs. The performance is averaged among all GOPs in the JCT-VC dataset. There is no obvious error propagation in the P-frames for PLVC. The possible reasons may be two-fold. On the one hand, the bi-IPPP structure shortens the distance between Iframes and P-frames, resulting in less error propagation. On the other hand, thanks to the recurrent structure of PLVC, the hidden states contain richer temporal priors when the distance to I-frames increases. This is beneficial for compression performance, and hence it can combat error propagation.

4.6 Generalizability of the Proposed D

Recall that in PLVC, we utilize RLVC [Yang et al., 2021] as the backbone of G. However, the proposed recurrent conditional discriminator D generalizes to various compression networks. For instance, Fig. 12 shows the results of the DVC trained with the proposed D and GAN loss (blue curve) compared with the original DVC (green curve). It can be seen that the proposed D and GAN loss significantly improve the perceptual performance from the original DVC and make it outperform HM 16.20 on FID and KID. This shows that the proposed approach is not restricted to a specific compression network, but can be generalized to different backbones. It is worth mentioning that DVC trained with the proposed D performs worse than PLVC, because DVC does not contain recurrent cells, while PLVC has a recurrent G which works with the recurrent conditional D, and they together make up the proposed recurrent GAN. As a result, PLVC outperforms the previous traditional and learned video compression approaches on LPIPS, FID, KID and MOS.

5 Conclusion

This paper proposes a GAN-based perceptual video compression approach. Its recurrent generator learns to compress video with coherent and visually pleasing frames to fool the recurrent discriminator, which learns to assess the raw vs. compressed videos conditioned on spatial-temporal features. An adversarial loss function is designed to train the proposed model towards the trade-off among bit-rate, distortion and perceptual quality. The numerical results and user studies both validate the good perceptual performance of the proposed method, in comparison with the traditional video coding standard HM 16.20 and the learned video compression methods.

Acknowledgments

This work was supported by the ETH Z¨urich General Fund and by Humboldt Foundation.