Proceedings

Online 3D Gaussian Splatting Modeling with Novel View Selection

Download BibTeX Cite

Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence. Main Track. Pages 1323-1331.

https://doi.org/10.24963/ijcai.2025/148

Abstract

This study addresses the challenge of generating online 3D Gaussian Splatting (3DGS) models from RGB-only frames. Previous studies have employed dense SLAM techniques to estimate 3D scenes from keyframes for 3DGS model construction. However, these methods are limited by their reliance solely on keyframes, which are insufficient to capture an entire scene, resulting in incomplete reconstructions. Moreover, building a generalizable model requires incorporating frames from diverse viewpoints to achieve broader scene coverage. However, online processing restricts the use of many frames or extensive training iterations. Therefore, we propose a novel method for high-quality 3DGS modeling that improves model completeness through adaptive view selection. By analyzing reconstruction quality online, our approach selects optimal non-keyframes for additional training. By integrating both keyframes and selected non-keyframes, the method refines incomplete regions from diverse viewpoints, significantly enhancing completeness. We also present a framework that incorporates an online multi-view stereo approach, ensuring consistency in 3D information throughout the 3DGS modeling process. Experimental results demonstrate that our method outperforms state-of-the-art methods, delivering exceptional performance in complex outdoor scenes.

  1. [Bhat et al., 2023] Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller. Zoedepth: Zero- shot transfer by combining relative and metric depth. arXiv preprint arXiv:2302.12288, 2023.
  2. [Bloesch et al., 2018] Michael Bloesch, Jan Czarnowski, Ronald Clark, Stefan Leutenegger, and Andrew J Davi- son. Codeslam-learning a compact, optimisable repre- sentation for dense visual slam. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, pages 2560-2568, 2018.
  3. [Cao et al., 2022] Chenjie Cao, Xinlin Ren, and Yanwei Fu. Mvsformer: Multi-view stereo by learning robust image features and temperature-based depth. arXiv preprint arXiv:2208.02541, 2022.
  4. [Eftekhar et al., 2021] Ainaz Eftekhar, Alexander Sax, Jiten- dra Malik, and Amir Zamir. Omnidata: A scalable pipeline for making multi-task mid-level vision datasets from 3d scans. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 10786-10796, 2021.
  5. [Engel et al., 2017] Jakob Engel, Vladlen Koltun, and Daniel Cremers. Direct sparse odometry. IEEE trans- actions on pattern analysis and machine intelligence, 40(3):611-625, 2017.
  6. [Feng et al., 2024] Ziyue Feng, Huangying Zhan, Zheng Chen, Qingan Yan, Xiangyu Xu, Changjiang Cai, Bing Li, Qilun Zhu, and Yi Xu. Naruto: Neural active reconstruc- tion from uncertain target observations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 21572-21583, 2024.
  7. [Forster et al., 2014] Christian Forster, Matia Pizzoli, and Davide Scaramuzza. Svo: Fast semi-direct monocular vi- sual odometry. In 2014 IEEE international conference on robotics and automation (ICRA), pages 15-22. IEEE, 2014.
  8. [Hamdi et al., 2024] Abdullah Hamdi, Luke Melas-Kyriazi, Jinjie Mai, Guocheng Qian, Ruoshi Liu, Carl Vondrick, Bernard Ghanem, and Andrea Vedaldi. Ges: General- ized exponential splatting for efficient radiance field ren- dering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19812- 19822, 2024.
  9. [Hu et al., 2024] Yan Song Hu, Nicolas Abboud, Muham- mad Qasim Ali, Adam Srebrnjak Yang, Imad Elhajj, Daniel Asmar, Yuhao Chen, and John S Zelek. Mgso: Monocular real-time photometric slam with efficient 3d gaussian splatting. arXiv preprint arXiv:2409.13055, 2024.
  10. [Hu et al., 2025] Jiarui Hu, Xianhao Chen, Boyin Feng, Guanglin Li, Liangjing Yang, Hujun Bao, Guofeng Zhang, and Zhaopeng Cui. Cg-slam: Efficient dense rgb-d slam in a consistent uncertainty-aware 3d gaussian field. In Eu- ropean Conference on Computer Vision, pages 93-112. Springer, 2025.
  11. [Huang et al., 2024] Huajian Huang, Longwei Li, Hui Cheng, and Sai-Kit Yeung. Photo-slam: Real-time si- multaneous localization and photorealistic mapping for monocular stereo and rgb-d cameras. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 21584-21593, 2024.
  12. [Jiang et al., 2025] Wen Jiang, Boshu Lei, and Kostas Dani- ilidis. Fisherrf: Active view selection and mapping with radiance fields using fisher information. In European Con- ference on Computer Vision, pages 422-440. Springer, 2025.
  13. [Jin et al., 2024] Rui Jin, Yuman Gao, Yingjian Wang, Yuze Wu, Haojian Lu, Chao Xu, and Fei Gao. Gs-planner: A gaussian-splatting-based planning framework for ac- tive high-fidelity reconstruction. In 2024 IEEE/RSJ In- ternational Conference on Intelligent Robots and Systems (IROS), pages 11202-11209. IEEE, 2024.
  14. [Kerbl et al., 2023] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaus- sian splatting for real-time radiance field rendering. ACM Trans. Graph., 42(4):139-1, 2023.
  15. [Knapitsch et al., 2017] Arno Knapitsch, Jaesik Park, Qian- Yi Zhou, and Vladlen Koltun. Tanks and temples: Bench- marking large-scale scene reconstruction. ACM Transac- tions on Graphics (ToG), 36(4):1-13, 2017.
  16. [Koestler et al., 2022] Lukas Koestler, Nan Yang, Niclas Zeller, and Daniel Cremers. Tandem: Tracking and dense mapping in real-time using deep multi-view stereo. In Conference on Robot Learning, pages 34-45. PMLR, 2022.
  17. [Lee et al., 2024] Byeonggwon Lee, Junkyu Park, Khang Truong Giang, Sungho Jo, and Soohwan Song. Mvs-gs: High-quality 3d gaussian splatting mapping via online multi-view stereo. arXiv preprint arXiv:2412.19130, 2024.
  18. [Ma et al., 2024] Yikun Ma, Dandan Zhan, and Zhi Jin. Fastscene: Text-driven fast 3d indoor scene generation via panoramic gaussian splatting. In IJCAI, 2024.
  19. [Matsuki et al., 2024] Hidenobu Matsuki, Riku Murai, Paul HJ Kelly, and Andrew J Davison. Gaussian splatting slam. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18039-18048, 2024.
  20. [Mildenhall et al., 2021] Ben Mildenhall, Pratul P Srini- vasan, Matthew Tancik, Jonathan T Barron, Ravi Ra- mamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
  21. [Mur-Artal et al., 2015] Raul Mur-Artal, Jose Maria Mar- tinez Montiel, and Juan D Tardos. Orb-slam: a versatile and accurate monocular slam system. IEEE transactions on robotics, 31(5):1147-1163, 2015.
  22. [Newcombe et al., 2011] Richard A Newcombe, Steven J Lovegrove, and Andrew J Davison. Dtam: Dense tracking and mapping in real-time. In 2011 international confer- ence on computer vision, pages 2320-2327. IEEE, 2011.
  23. [Peng et al., 2024] Chensheng Peng, Chenfeng Xu, Yue Wang, Mingyu Ding, Heng Yang, Masayoshi Tomizuka, Kurt Keutzer, Marco Pavone, and Wei Zhan. Q-slam: Quadric representations for monocular slam. arXiv preprint arXiv:2403.08125, 2024.
  24. [Ran et al., 2023] Yunlong Ran, Jing Zeng, Shibo He, Jim- ing Chen, Lincheng Li, Yingfeng Chen, Gimhee Lee, and Qi Ye. Neurar: Neural uncertainty for autonomous 3d re- construction with implicit neural representations. IEEE Robotics and Automation Letters, 8(2):1125-1132, 2023.
  25. [Ranftl et al., 2021] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In Proceedings of the IEEE/CVF international conference on computer vision, pages 12179-12188, 2021.
  26. [Redmon, 2016] J Redmon. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016.
  27. [Rosinol et al., 2023] Antoni Rosinol, John J Leonard, and Luca Carlone. Nerf-slam: Real-time dense monocular slam with neural radiance fields. In 2023 IEEE/RSJ In- ternational Conference on Intelligent Robots and Systems (IROS), pages 3437-3444. IEEE, 2023.
  28. [Rota Bulò et al., 2024] Samuel Rota Bulò, Lorenzo Porzi, and Peter Kontschieder. Revising densification in gaussian splatting. In European Conference on Computer Vision, pages 347-362. Springer, 2024.
  29. [Sandström et al., 2024] Erik Sandström, Keisuke Tateno, Michael Oechsle, Michael Niemeyer, Luc Van Gool, Mar- tin R Oswald, and Federico Tombari. Splat-slam: Globally optimized rgb-only slam with 3d gaussians. arXiv preprint arXiv:2405.16544, 2024.
  30. [Scott et al., 2003] William R Scott, Gerhard Roth, and Jean- Franc ¸ois Rivest. View planning for automated three- dimensional object reconstruction and inspection. ACM Computing Surveys (CSUR), 35(1):64-96, 2003.
  31. [Song et al., 2021] Soohwan Song, Daekyum Kim, and Sunghee Choi. View path planning via online multiview stereo for 3-d modeling of large-scale structures. IEEE Transactions on Robotics, 38(1):372-390, 2021.
  32. [Straub et al., 2019] Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J En- gel, Raul Mur-Artal, Carl Ren, Shobhit Verma, et al. The replica dataset: A digital replica of indoor spaces. arXiv preprint arXiv:1906.05797, 2019.
  33. [Sturm et al., 2012] Jürgen Sturm, Nikolas Engelhard, Felix Endres, Wolfram Burgard, and Daniel Cremers. A bench- mark for the evaluation of rgb-d slam systems. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pages 573-580. IEEE, 2012.
  34. [Teed and Deng, 2020] Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision-ECCV 2020: 16th European Confer- ence, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pages 402-419. Springer, 2020.
  35. [Teed and Deng, 2021] Zachary Teed and Jia Deng. Droid- slam: Deep visual slam for monocular, stereo, and rgb-d cameras. Advances in neural information processing sys- tems, 34:16558-16569, 2021.
  36. [Wang et al., 2004] Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600-612, 2004.
  37. [Yan et al., 2023] Dongyu Yan, Jianheng Liu, Fengyu Quan, Haoyao Chen, and Mengmeng Fu. Active implicit ob- ject reconstruction using uncertainty-guided next-best- view optimization. IEEE Robotics and Automation Let- ters, 2023.
  38. [Yao et al., 2018] Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan. Mvsnet: Depth inference for unstructured multi-view stereo. In Proceedings of the European confer- ence on computer vision (ECCV), pages 767-783, 2018.
  39. [Zhang et al., 2023] Youmin Zhang, Fabio Tosi, Stefano Mattoccia, and Matteo Poggi. Go-slam: Global optimiza- tion for consistent 3d instant reconstruction. In Proceed- ings of the IEEE/CVF International Conference on Com- puter Vision, pages 3727-3737, 2023.
  40. [Zhang et al., 2024] Ganlin Zhang, Erik Sandström, Youmin Zhang, Manthan Patel, Luc Van Gool, and Martin R Oswald. Glorie-slam: Globally optimized rgb-only implicit encoding point cloud slam. arXiv preprint arXiv:2403.19549, 2024.
  41. [Zhou et al., 2023] Xingchen Zhou, Ying He, F Richard Yu, Jianqiang Li, and You Li. RePaint-NeRF: Nerf editting via semantic masks and diffusion models. In IJCAI, 2023. IJCAI
  42. [Zhu et al., 2024] Zihan Zhu, Songyou Peng, Viktor Lars- son, Zhaopeng Cui, Martin R Oswald, Andreas Geiger, and Marc Pollefeys. Nicer-slam: Neural implicit scene encoding for rgb slam. In 2024 International Conference on 3D Vision (3DV), pages 42-52. IEEE, 2024.

References were extracted automatically from the PDF and may contain errors.

Welcome Future Conferences AI Journal Division IJCAI Proceedings About IJCAI IJCAI Awards