Proceedings

DeepLight: Reconstructing High-Resolution Observations of Nighttime Light With Multi-Modal Remote Sensing Data

Download BibTeX Cite

Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence. AI for Good. Pages 7563-7571.

https://doi.org/10.24963/ijcai.2024/837

Abstract

Nighttime light (NTL) remote sensing observation serves as a unique proxy for quantitatively assessing progress toward meeting a series of Sustainable Development Goals (SDGs), such as poverty estimation, urban sustainable development, and carbon emission. However, existing NTL observations often suffer from pervasive degradation and inconsistency, limiting their utility for computing the indicators defined by the SDGs. In this study, we propose a novel approach to reconstruct high-resolution NTL images using multimodal remote sensing data. To support this research endeavor, we introduce DeepLightMD, a comprehensive dataset comprising data from five heterogeneous sensors, offering fine spatial resolution and rich spectral information at a national scale. Additionally, we present DeepLightSR, a calibration-aware method for multi-modality super-resolution. DeepLightSR integrates calibration-aware alignment, an auxiliary-to-main multi-modality fusion, and an auxiliary-embedded refinement to effectively address spatial heterogeneity, fuse diversely representative features, and enhance performance in ×8 super-resolution tasks. Extensive experiments demonstrate the superiority of DeepLightSR over 8 competing methods, as evidenced by improvements in PSNR (2.01 dB ~ 13.25 dB) and PIQE (0.49 ~ 9.32). Our findings underscore the practical significance of our proposed dataset and model in reconstructing high-resolution NTL data, supporting efficiently and quantitatively assessing the SDG progress. The code and data will be available at https://github.com/xian1234/DeepLight.

  1. [Abrahams et al., 2018] Alexei Abrahams, Christopher Oram, and Nancy Lozano-Gracia. Deblurring dmsp nighttime lights: A new method using gaussian filters and frequencies of illumination. Remote Sensing of Environment, 210:242-258, 2018.
  2. [Chen et al., 2021] Zuoqi Chen, Bailang Yu, Chengshu Yang, Yuyu Zhou, Shenjun Yao, Xingjian Qian, Congx- iao Wang, Bin Wu, and Jianping Wu. An extended time series (2000-2018) of global npp-viirs-like nighttime light data from a cross-sensor calibration. Earth System Science Data, 13(3):889-906, 2021.
  3. [Dai et al., 2017] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. De- formable convolutional networks. In 2017 IEEE Interna- tional Conference on Computer Vision (ICCV), IEEE In- ternational Conference on Computer Vision, pages 764- 773, 2017.
  4. [Deng and Dragotti, 2020] Xin Deng and Pier Luigi Dragotti. Deep coupled ista network for multi-modal image super-resolution. IEEE Transactions on Image Processing, 29:1683-1698, 2020.
  5. [Deng et al., 2023] Shangqi Deng, Liang-Jian Deng, Xiao Wu, Ran Ran, and Rui Wen. Bidirectional dilation trans- former for multispectral and hyperspectral image fusion. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, 2023. IJCAI
  6. [Dong et al., 2016] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(2):295-307, 2016.
  7. [Dong et al., 2023] Runmin Dong, Lixian Zhang, Weijia Li, Shuai Yuan, Lin Gan, Juepeng Zheng, Haohuan Fu, Lichao Mou, and Xiao Xiang Zhu. An adaptive image fusion method for sentinel-2 images and high-resolution images with long-time intervals. International Jour- nal of Applied Earth Observation and Geoinformation, 121:103381, 2023.
  8. [Dong et al., 2024] Runmin Dong, Shuai Yuan, Bin Luo, Mengxuan Chen, Jinxiao Zhang, Lixian Zhang, Weijia Li, Juepeng Zheng, and Haohuan Fu. Building bridges across spatial and temporal resolutions: Reference-based super-resolution via change priors and conditional diffu- sion model. arXiv preprint arXiv:2403.17460, 2024.
  9. [Elvidge et al., 1997] Christopher D Elvidge, Kimberly E Baugh, Eric A Kihn, Herbert W Kroehl, and Ethan R Davis. Mapping city lights with nighttime data from the dmsp operational linescan system. Photogrammetric En- gineering and Remote Sensing, 63(6):727-734, 1997.
  10. [Elvidge et al., 2009] Christopher D. Elvidge, Daniel Ziskin, Kimberly E. Baugh, Benjamin T. Tuttle, Tilottama Ghosh, Dee W. Pack, Edward H. Erwin, and Mikhail Zhizhin. A fifteen year record of global natural gas flaring derived from satellite data. Energies, 2(3):595-622, 2009.
  11. [Falchi et al., 2016] Fabio Falchi, Pierantonio Cinzano, Dan Duriscoe, Christopher C. M. Kyba, Christopher D. Elvidge, Kimberly Baugh, Boris A. Portnov, Nataliya A. Rybnikova, and Riccardo Furgoni. The new world at- las of artificial night sky brightness. Science Advances, 2(6):e1600377, 2016.
  12. [Fang et al., 2022] Chaowei Fang, Dingwen Zhang, Liang Wang, Yulun Zhang, Lechao Cheng, and Junwei Han. Cross-modality high-frequency transformer for mr image super-resolution. In Proceedings of the 30th ACM Interna- tional Conference on Multimedia, MM '22, pages 1584- 92. Association for Computing Machinery, 2022.
  13. [Gong et al., 2020] Peng Gong, Xuecao Li, Jie Wang, Yuqi Bai, Bin Chen, Tengyun Hu, Xiaoping Liu, Bing Xu, Jun Yang, Wei Zhang, and Yuyu Zhou. Annual maps of global artificial impervious area (gaia) between 1985 and 2018. Remote Sensing of Environment, 236:111510, 2020.
  14. [Guo et al., 2023] Huadong Guo, Changyong Dou, Hongyu Chen, Jianbo Liu, Bihong Fu, Xiaoming Li, Ziming Zou, and Dong Liang. Sdgsat-1: the world's first scientific satel- lite for sustainable development goals. Sci. Bull, 68(1):34- 38, 2023.
  15. [Gupta and Mitra, 2022] Honey Gupta and Kaushik Mitra. Toward unaligned guided thermal super-resolution. IEEE Transactions on Image Processing, 31:433-445, 2022.
  16. [Haut et al., 2019] Juan Mario Haut, Ruben Fernandez- Beltran, Mercedes E Paoletti, Javier Plaza, and Antonio Plaza. Remote sensing image superresolution using deep residual channel attention. IEEE Transactions on Geo- science and Remote Sensing, 57(11):9277-9289, 2019.
  17. [Huang et al., 2014] Qingxu Huang, Xi Yang, Bin Gao, Yang Yang, and Yuanyuan Zhao. Application of dmsp/ols nighttime light images: A meta-analysis and a systematic literature review. Remote Sensing, 6(8):6844-6866, 2014.
  18. [Jaderberg et al., 2015] Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu. Spatial transformer networks. In Proceedings of the 28th Interna- tional Conference on Neural Information Processing Sys- tems -Volume 2, NIPS'15. MIT Press, 2015.
  19. [Jarvis, 2008] H.I. Reuter A. Nelson E. Guevara Jarvis, A. Hole-filled srtm for the globe version 4. Avail- able from the CGIAR-CSI SRTM 90m Database (http://srtm.csi.cgiar.org), 2008. Link
  20. [Kruse et al., 1993] F.A. Kruse, A.B. Lefkoff, J.W. Board- man, K.B. Heidebrecht, A.T. Shapiro, P.J. Barloon, and A.F.H. Goetz. The spectral image processing system (sips)-interactive visualization and analysis of imaging spectrometer data. Remote Sensing of Environment, 44(2- 3):145-163, 1993.
  21. [Ledig et al., 2017] Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi. Photo-realistic single im- age super-resolution using a generative adversarial net- work. In 30th IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 105-114, 2017.
  22. [Levin et al., 2020] Noam Levin, Christopher CM Kyba, Qingling Zhang, Alejandro Sánchez de Miguel, Miguel O Román, Xi Li, Boris A Portnov, Andrew L Molthan, An- dreas Jechow, and Steven D Miller. Remote sensing of night lights: A review and an outlook for the future. Re- mote Sensing of Environment, 237:111443, 2020.
  23. [Li et al., 2019] Xin Li, Taoyang Wang, Guo Zhang, Boyang Jiang, Peng Jia, Zhuxi Zhang, and Yuan Zhao. Planar block adjustment for china's land regions with luojia1-01 nighttime light imagery. Remote Sensing, 11(18), 2019.
  24. [Li et al., 2020] Xi Li, Noam Levin, Jinlong Xie, and Deren Li. Monitoring hourly night-time light by an unmanned aerial vehicle and its implications to satellite remote sens- ing. Remote Sensing of Environment, 247:111942, 2020.
  25. [Maeda, 2020] Shunta Maeda. Unpaired image super- resolution using pseudo-supervision. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 288-297, 2020.
  26. [Maso et al., 2023] Joan Maso, Alaitz Zabala, and Ivette Ser- ral. Earth observations for sustainable development goals, 2023.
  27. [McCallum et al., 2022] Ian McCallum, Christopher Con- rad Maximillian Kyba, Juan Carlos Laso Bayas, Elena Moltchanova, Matt Cooper, Jesus Crespo Cuaresma, Shonali Pachauri, Linda See, Olga Danylo, Inian Moorthy, et al. Estimating global economic well-being with unlit settlements. Nature Communications, 13(1):2459, 2022.
  28. [N et al., 2015] Venkatanath N, Praneeth D, Maruthi Chan- drasekhar Bh, Sumohana S. Channappayya, and Swarup S. Medasani. Blind image quality evaluation using percep- tion based features. In 2015 Twenty First National Confer- ence on Communications (NCC), 2015.
  29. [Shacht et al., 2021] Guy Shacht, Dov Danon, Sharon Fogel, and Daniel Cohen-Or. Single pair cross-modality super resolution. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6374- 6383, 2021.
  30. [Shi et al., 2016] Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single im- age and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1874-1883, 2016.
  31. [Sun et al., 2021] Baoli Sun, Xinchen Ye, Baopu Li, Hao- jie Li, Zhihui Wang, and Rui Xu. Learning scene struc- ture guidance via cross-task knowledge transfer for single depth super-resolution. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7788-7797, 2021.
  32. [UnitedNations, 2015] UnitedNations. Transforming our world: the 2030 agenda for sustainable development. United Nations: New York, NY, USA, 2015.
  33. [Wang and Bovik, 2002] Zhou Wang and A.C. Bovik. A uni- versal image quality index. IEEE Signal Processing Let- ters, 9(3):81-84, 2002.
  34. [Wang et al., 2022] Mingyue Wang, Chunhui Feng, Bifeng Hu, Nan Wang, Jintao Xu, Ziqiang Ma, Jie Peng, and Zhou Shi. A new framework for reconstructing time series dmsp-ols nighttime light data using the improved stepwise calibration (isc) method. Remote Sensing, 14(17), 2022.
  35. [Wu et al., 2013] Jiansheng Wu, Shengbin He, Jian Peng, Weifeng Li, and Xiaohong Zhong. Intercalibration of dmsp-ols night-time light data by the invariant re- gion method. International Journal of Remote Sensing, 34(20):7356-7368, 2013.
  36. [Xi et al., 2023] Yanxin Xi, Yu Liu, Tong Li, Jintao Ding, Yunke Zhang, Sasu Tarkoma, Yong Li, and Pan Hui. A satellite imagery dataset for long-term sustainable devel- opment in united states cities. Scientific data, 10(1):866, 2023.
  37. [Xie et al., 2022] Wuyuan Xie, Tengcong Huang, and Miao- hui Wang. Mnsrnet: Multimodal transformer network for 3d surface super-resolution. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12693-12702, 2022.
  38. [Yang et al., 2020] Fuzhi Yang, Huan Yang, Jianlong Fu, Hongtao Lu, and Baining Guo. Learning texture trans- former network for image super-resolution. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 5791-5800, 2020.
  39. [Zhang et al., 2016] Qingling Zhang, Bhartendu Pandey, and Karen C Seto. A robust method to generate a con- sistent time series from dmsp/ols nighttime light data. IEEE Transactions on Geoscience and Remote Sensing, 54(10):5821-5831, 2016.
  40. [Zhang et al., 2021a] Lixian Zhang, Runmin Dong, Shuai Yuan, Weijia Li, Juepeng Zheng, and Haohuan Fu. Mak- ing low-resolution satellite images reborn: a deep learning approach for super-resolution building extraction. Remote Sensing, 13(15):2872, 2021.
  41. [Zhang et al., 2021b] Lixian Zhang, Zhehao Ren, Runmin Dong, Bing Xu, and Haohuan Fu. Monitoring daily night- time light based on modis and deep learning: A belgium case study. In 2021 IEEE International Geoscience and Remote Sensing Symposium IGARSS, pages 5032-5035, 2021.
  42. [Zhang et al., 2024a] Lixian Zhang, Zhehao Ren, Bin Chen, Peng Gong, Bing Xu, and Haohuan Fu. A prolonged arti- ficial nighttime-light dataset of china (1984-2020). Scien- tific Data, 11(1):414, 2024.
  43. [Zhang et al., 2024b] Lixian Zhang, Shuai Yuan, Runmin Dong, Juepeng Zheng, Bin Gan, Dengmao Fang, Yang Liu, and Haohuan Fu. Swcare: Switchable learning and connectivity-aware refinement method for multi-city and diverse-scenario road mapping using remote sensing im- ages. International Journal of Applied Earth Observation and Geoinformation, 127:103665, 2024.

References were extracted automatically from the PDF and may contain errors.

Welcome Future Conferences AI Journal Division IJCAI Proceedings About IJCAI IJCAI Awards