Proceedings

Emerging Advances in Learned Video Compression: Models, Systems and Beyond

Download BibTeX Cite

Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence. Survey Track. Pages 10490-10498.

https://doi.org/10.24963/ijcai.2025/1165

Abstract

Video compression is a fundamental topic in the visual intelligence, bridging visual signal sensing/capturing and high-level visual analytics. The broad success of artificial intelligence (AI) technology has enriched the horizon of video compression into novel paradigms by leveraging end-to-end optimized neural models. In this survey, we first provide a comprehensive and systematic overview of recent literature on end-to-end optimized learned video coding, covering the spectrum of pioneering efforts in both uni-directional and bi-directional prediction based compression model designation. We further delve into the optimization techniques employed in learned video compression (LVC), emphasizing their technical innovations, advantages. Some standardization progress is also reported. Furthermore, we investigate the system design and hardware implementation challenges of the LVC inclusively. Finally, we present the extensive simulation results to demonstrate the superior compression performance of LVC models, addressing the question that why learned codecs and AI-based video technology would have with broad impact on future visual intelligence research.

  1. [Alexandre et al., 2023] David Alexandre, Hsueh-Ming Hang, and Wen-Hsiao Peng. Hierarchical b-frame video coding using two-layer canf without motion coding. In CVPR, 2023.
  2. [Ballé et al., 2018] Johannes Ballé, Nick Johnston, and David Minnen. Integer networks for data compression with latent-variable models. In ICLR, 2018.
  3. [Boyce et al., 2018] Jill Boyce, Karsten Suehring, Xiang Li, and Vadim Seregin. Jvet-j1010: Jvet common test condi- tions and software reference configurations, 2018.
  4. [Bross et al., 2021] Benjamin Bross, Ye-Kui Wang, Yan Ye, Shan Liu, Jianle Chen, Gary J Sullivan, and Jens-Rainer Ohm. Overview of the versatile video coding (vvc) stan- dard and its applications. IEEE TCSVT, 2021.
  5. [Chen et al., 2017] Tong Chen, Haojie Liu, Qiu Shen, Tao Yue, Xun Cao, and Zhan Ma. Deepcoder: A deep neural network based video compression. In VCIP, 2017.
  6. [Chen et al., 2022] Meixu Chen, Todd Goodall, Anjul Pat- ney, and Alan C Bovik. Learning to compress videos with- out computing motion. SPIC, 2022.
  7. [Chen et al., 2023a] Jiancong Chen, Meng Wang, Pingping Zhang, Shurun Wang, and Shiqi Wang. Sparse-to-dense: High efficiency rate control for end-to-end scale-adaptive video coding. IEEE TCSVT, 2023.
  8. [Chen et al., 2023b] Mu-Jung Chen, Yi-Hsin Chen, and Wen-Hsiao Peng. B-canf: Adaptive b-frame coding with conditional augmented normalizing flows. IEEE TCSVT, 2023.
  9. [Cui et al., 2021] Ze Cui, Jing Wang, Shangyin Gao, Tian- sheng Guo, Yihui Feng, and Bo Bai. Asymmetric gained deep image compression with continuous rate adaptation. In CVPR, 2021.
  10. [Dai et al., 2022] Jincheng Dai, Sixian Wang, Kailin Tan, Zhongwei Si, Xiaoqi Qin, Kai Niu, and Ping Zhang. Non- linear transform source-channel coding for semantic com- munications. IEEE JSAC, 2022.
  11. [Djelouah et al., 2019] Abdelaziz Djelouah, Joaquim Cam- pos, Simone Schaub-Meyer, and Christopher Schroers. Neural inter-frame compression for video coding. In ICCV, 2019.
  12. [Fathima et al., 2023] Noor Fathima, Jens Petersen, Guil- laume Sautière, Auke Wiggers, and Reza Pourreza. A neural video codec with spatial rate-distortion control. In WACV, 2023.
  13. [Feng et al., 2020] Runsen Feng, Yaojun Wu, Zongyu Guo, Zhizheng Zhang, and Zhibo Chen. Learned video com- pression with feature-level residuals. In CVPRW, 2020.
  14. [Feng et al., 2021] Runsen Feng, Zongyu Guo, Zhizheng Zhang, and Zhibo Chen. Versatile learned video compres- sion. arXiv preprint: 2111.03386, 2021.
  15. [Ho et al., 2022] Yung-Han Ho, Chih-Peng Chang, Peng-Yu Chen, Alessandro Gnutti, and Wen-Hsiao Peng. Canf-vc: Conditional augmented normalizing flows for video com- pression. In ECCV, 2022.
  16. [Hong et al., 2020] Weixin Hong, Tong Chen, Ming Lu, Shiliang Pu, and Zhan Ma. Efficient neural image decod- ing via fixed-point inference. IEEE TCSVT, 2020.
  17. [Hu et al., 2020] Zhihao Hu, Zhenghao Chen, Dong Xu, Guo Lu, Wanli Ouyang, and Shuhang Gu. Improving deep video compression by resolution-adaptive flow coding. In ECCV, 2020.
  18. [Hu et al., 2021] Zhihao Hu, Guo Lu, and Dong Xu. Fvc: A new framework towards deep video compression in feature space. In CVPR, 2021.
  19. [Hu et al., 2022] Zhihao Hu, Guo Lu, Jinyang Guo, Shan Liu, Wei Jiang, and Dong Xu. Coarse-to-fine deep video coding with hyperprior-guided mode prediction. In CVPR, 2022.
  20. [Jia et al., 2022] Chuanmin Jia, Xinyu Hang, et al. Fpx-nic: An fpga-accelerated 4k ultra-high-definition neural video coding system. IEEE TCSVT, 2022.
  21. [Jia et al., 2023] Chuanmin Jia, Feng Ye, Fanke Dong, Kai Lin, Leonardo Chiariglione, et al. Mpai-eev: Standardiza- tion efforts of artificial intelligence based end-to-end video coding. IEEE TCSVT, 2023.
  22. [Kim et al., 2023] Yeong-Chun Kim, Suyong Bahk, Se- ung Hwan Kim, Won Hee Lee, Dokwan Oh, and Hui- Yong Kim. Neural video compression with temporal layer- adaptive hierarchical b-frame coding. arXiv preprint: 2308.15791, 2023.
  23. [Ladune et al., 2020] Théo Ladune, Pierrick Philippe, Was- sim Hamidouche, Lu Zhang, and Olivier Déforges. Optical flow and mode selection for learning-based video coding. In MMSP, 2020.
  24. [Ladune et al., 2021] Théo Ladune, Pierrick Philippe, Was- sim Hamidouche, Lu Zhang, and Olivier Déforges. Con- ditional coding for flexible learned video compression. In ICLRW, 2021.
  25. [Le et al., 2022] Hoang Le, Liang Zhang, Amir Said, Guil- laume Sautiere, Yang Yang, Pranav Shrestha, Fei Yin, Reza Pourreza, and Auke Wiggers. Mobilecodec: neu- ral inter-frame video compression on mobile devices. In ACM MMSys, 2022.
  26. [Li et al., 2021] Jiahao Li, Bin Li, and Yan Lu. Deep contex- tual video compression. In NeurIPS, 2021.
  27. [Li et al., 2022] Jiahao Li, Bin Li, and Yan Lu. Hybrid spatial-temporal entropy modelling for neural video com- pression. ACM MM, 2022.
  28. [Li et al., 2023] Jiahao Li, Bin Li, and Yan Lu. Neural video compression with diverse contexts. In CVPR, 2023.
  29. [Li et al., 2024] Jiahao Li, Bin Li, and Yan Lu. Neural video compression with feature modulation. In CVPR, 2024.
  30. [Lin et al., 2020] Jianping Lin, Dong Liu, Houqiang Li, and Feng Wu. M-lvc: Multiple frames prediction for learned video compression. In CVPR, 2020.
  31. [Lin et al., 2021] Jianping Lin, Dong Liu, Jie Liang, Houqiang Li, and Feng Wu. Modulated variable-rate deep video compression. In DCC, 2021.
  32. [Lin et al., 2022] Chih-Hsuan Lin, Yi-Hsin Chen, and Wen- Hsiao Peng. Content-adaptive motion rate adaption for learned video compression. In IEEE PCS, 2022.
  33. [Liu et al., 2020] Dong Liu, Yue Li, Jianping Lin, Houqiang Li, and Feng Wu. Deep learning-based video coding: A review and a case study. ACM CSUR, 2020.
  34. [Liu et al., 2021] Haojie Liu, Ming Lu, Zhan Ma, Fan Wang, Zhihuang Xie, Xun Cao, and Yao Wang. Neural video cod- ing using multiscale motion compensation and spatiotem- poral context model. IEEE TCSVT, 2021.
  35. [Lu et al., 2019] Guo Lu, Wanli Ouyang, Dong Xu, Xiaoyun Zhang, Chunlei Cai, and Zhiyong Gao. Dvc: An end-to- end deep video compression framework. In CVPR, 2019.
  36. [Lu et al., 2020] Guo Lu, Chunlei Cai, Xiaoyun Zhang, et al. Content adaptive and error propagation aware deep video compression. In ECCV, 2020.
  37. [Ma et al., 2019] Siwei Ma, Xinfeng Zhang, Chuanmin Jia, Zhenghui Zhao, Shiqi Wang, and Shanshe Wang. Image and video compression with neural networks: A review. IEEE TCSVT, 2019.
  38. [Ma et al., 2022] Siwei Ma, Li Zhang, Shiqi Wang, Chuan- min Jia, Shanshe Wang, Tiejun Huang, Feng Wu, and Wen Gao. Evolution of avs video coding standards: twenty years of innovation and development. Science China In- formation Sciences, 2022.
  39. [Ma et al., 2023] Siwei Ma, Junlong Gao, Ruofan Wang, Jianhui Chang, Qi Mao, Zhimeng Huang, and Chuanmin Jia. Overview of intelligent video coding: from model- based to learning-based approaches. Visual Intelligence, 2023.
  40. [Mao et al., 2021] Hongzi Mao, Chenjie Gu, Miaosen Wang, Angie Chen, Nevena Lazic, Nir Levine, et al. Neural rate control for video encoding using imitation learning. In ICMLW, 2021.
  41. [Mercat et al., 2020] Alexandre Mercat, Marko Viitanen, and Jarno Vanne. Uvg dataset: 50/120fps 4k sequences for video codec analysis and development. In ACM MM- Sys, 2020.
  42. [Pergament et al., 2022] Evgenya Pergament, Pulkit Tandon, Oren Rippel, et al. Pim: Video coding using perceptual importance maps. arXiv preprint: 2212.10674, 2022.
  43. [Pourreza and Cohen, 2021] Reza Pourreza and Taco Cohen. Extending neural p-frame codecs for b-frame coding. In ICCV, 2021.
  44. [Rippel et al., 2021] Oren Rippel, Alexander G Anderson, Kedar Tatwawadi, Sanjay Nair, Craig Lytle, and Lubomir Bourdev. Elf-vc: Efficient learned flexible-rate video cod- ing. In ICCV, 2021.
  45. [Sheng et al., 2022] Xihua Sheng, Jiahao Li, Bin Li, Li Li, Dong Liu, and Yan Lu. Temporal context mining for learned video compression. IEEE TMM, 2022. Sheng et al., 2024a] Xihua Sheng, Li Li, Dong Liu, and Houqiang Li. Spatial decomposition and temporal fu- sion based inter prediction for learned video compression. IEEE TCSVT, 2024.
  46. [Sheng et al., 2024b] Xihua Sheng, Li Li, Dong Liu, and Shiqi Wang. Bi-directional deep contextual video com- pression. arXiv preprint arXiv:2408.08604, 2024.
  47. [Shi et al., 2022] Yibo Shi, Yunying Ge, Jing Wang, and Jue Mao. Alphavc: High-performance and efficient learned video compression. In ECCV, 2022.
  48. [Sun et al., 2022] Heming Sun, Qingyang Yi, Fangzheng Lin, Lu Yu, Jiro Katto, and Masahiro Fujita. F-lic: Fpga- based learned image compression with a fine-grained pipeline. In IEEE A-SSCC, 2022.
  49. [Van Rozendaal et al., 2024] Ties Van Rozendaal, Tushar Singhal, Hoang Le, Guillaume Sautiere, Amir Said, Kr- ishna Buska, et al. Mobilenvc: Real-time 1080p neural video compression on a mobile device. In WACV, 2024.
  50. [Veerabadran et al., 2020] Vijay Veerabadran, Reza Pour- reza, et al. Adversarial distortion for learned video com- pression. In CVPRW, 2020.
  51. [Wang et al., 2016] Haiqiang Wang, Weihao Gan, Sudeng Hu, Joe Yuchieh Lin, Lina Jin, Longguang Song, Ping Wang, Ioannis Katsavounidis, Anne Aaron, and C-C Jay Kuo. Mcl-jcv: a jnd-based h. 264/avc video quality as- sessment dataset. In IEEE ICIP, 2016.
  52. [Wu et al., 2018] Chao-Yuan Wu, Nayan Singhal, and Philipp Krahenbuhl. Video compression through image interpolation. In ECCV, 2018.
  53. [Xu et al., 2023] Tongda Xu, Han Gao, Chenjian Gao, Yuanyuan Wang, Dailan He, Jinyong Pi, Jixiang Luo, Ziyu Zhu, Mao Ye, Hongwei Qin, et al. Bit allocation using op- timization. In ICML, 2023.
  54. [Yang et al., 2020] Ren Yang, Fabian Mentzer, Luc Van Gool, and Radu Timofte. Learning for video compression with hierarchical quality and recurrent enhancement. In CVPR, 2020.
  55. [Yang et al., 2022] Ren Yang, Radu Timofte, and Luc Van Gool. Perceptual learned video compression with re- current conditional gan. In IJCAI, 2022. IJCAI
  56. [Ye et al., 2024] Feng Ye, Li Zhang, and Chuanmin Jia. Deep video compression with scaled hierarchical bi- directional motion model. In ACM MM, 2024.
  57. [Yılmaz and Tekalp, 2021] M Akın Yılmaz and A Murat Tekalp. End-to-end rate-distortion optimized learned hi- erarchical bi-directional video compression. IEEE TIP, 2021.
  58. [Zhang et al., 2024a] Ping Zhang, Yiming Liu, Yile Song, and Jiaxiang Zhang. Advances and challenges in semantic communications: A systematic review. National Science Open, 2024.
  59. [Zhang et al., 2024b] Yiwei Zhang, Guo Lu, Yunuo Chen, Shen Wang, Yibo Shi, Jing Wang, and Li Song. Neural rate control for learned video compression. In ICLR, 2024.

References were extracted automatically from the PDF and may contain errors.

Welcome Future Conferences AI Journal Division IJCAI Proceedings About IJCAI IJCAI Awards