Proceedings

Towards Generalizable Neural Simulators: Addressing Distribution Shifts Induced by Environmental and Temporal Variations

Download BibTeX Cite

Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence. Main Track. Pages 7563-7571.

https://doi.org/10.24963/ijcai.2025/841

Abstract

With advancements in deep learning, neural simulators have become increasingly important for improving the efficiency and effectiveness of simulating complex dynamical systems in various scientific and technological fields. This paper presents a novel neural simulator called Context-informed Polymorphic Neural ODE Processes (CoPoNDP), aimed at addressing the challenges of modeling dynamical systems encountering concurrent environmental and temporal distribution shifts, which are common in real-world scenarios. CoPoNDP employs a context-driven neural stochastic process governed by a combination of basic differential equations in a time-sensitive manner to adaptively modulate the evolution of system states. This allows for flexible adaptation to changing temporal dynamics and generalization across different environments. Extensive experiments conducted on dynamical systems from ecology, chemistry, physics, and energy demonstrate that by effectively utilizing contextual information, CoPoNDP outperforms the state-of-the-art models in handling joint distribution shifts. It also shows robustness in sparse and noisy settings, making it a promising approach for modeling dynamical systems in complex real-world applications.

  1. [Bai et al., 2023] Guangji Bai, Chen Ling, and Liang Zhao. Temporal domain generalization with drift-aware dynamic neural networks. In The Eleventh International Confer- ence on Learning Representations, 2023.
  2. [Baker and Blackburn, 2008] Gregory L Baker and James A Blackburn. The pendulum: a case study in physics. OUP Oxford, 2008.
  3. [Barbulescu et al., 2023] Ruxandra Barbulescu, Gonc ¸alo Mestre, Arlindo L Oliveira, and Luís Miguel Silveira. Learning the dynamics of realistic models of c. elegans nervous system with recurrent neural networks. Scientific reports, 13(1):467, 2023.
  4. [Cai et al., 2024] Zekun Cai, Guangji Bai, Renhe Jiang, Xuan Song, and Liang Zhao. Continuous temporal domain generalization. arXiv preprint arXiv:2405.16075, 2024.
  5. [Chen et al., 2018] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary dif- ferential equations. Advances in neural information pro- cessing systems, 31, 2018.
  6. [Chen et al., 2022] Xing Chen, Flavio Abreu Araujo, Math- ieu Riou, Jacob Torrejon, Dafiné Ravelosona, Wang Kang, Weisheng Zhao, Julie Grollier, and Damien Querlioz. Forecasting the outcome of spintronic experiments with neural ordinary differential equations. Nature communi- cations, 13(1):1016, 2022.
  7. [Cho et al., 2014] Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In Proceedings of the 2014 Confer- ence on Empirical Methods in Natural Language Process- ing (EMNLP), page 1724. Association for Computational Linguistics, 2014.
  8. [Cui et al., 2024] Jiaxu Cui, Qipeng Wang, Bingyi Sun, Jim- ing Liu, and Bo Yang. Learning continuous network emerging dynamics from scarce observations via data- adaptive stochastic processes. Science China Information Sciences, 67(12):222206, 2024.
  9. [Han et al., 2024] Tao Han, Song Guo, Zhenghao Chen, Wanghan Xu, and Lei Bai. Weather-5k: A large-scale global station weather dataset towards comprehensive time-series forecasting benchmark. arXiv e-prints, pages arXiv-2406, 2024.
  10. [Hao et al., 2023] Zhongkai Hao, Zhengyi Wang, Hang Su, Chengyang Ying, Yinpeng Dong, Songming Liu, Ze Cheng, Jian Song, and Jun Zhu. Gnot: A general neu- ral operator transformer for operator learning. In Inter- national Conference on Machine Learning, pages 12556- 12569. PMLR, 2023.
  11. [Hochreiter, 1997] S Hochreiter. Long short-term memory. Neural Computation MIT-Press, 1997.
  12. [Huang et al., 2020] Zijie Huang, Yizhou Sun, and Wei Wang. Learning continuous system dynamics from irregularly-sampled partial observations. Advances in Neural Information Processing Systems, 33:16177-16187, 2020.
  13. [Kirchmeyer et al., 2022] Matthieu Kirchmeyer, Yuan Yin, Jérémie Donà, Nicolas Baskiotis, Alain Rakotomamonjy, and Patrick Gallinari. Generalizing to new physical sys- tems via context-informed dynamics model. In Interna- tional Conference on Machine Learning, pages 11283- 11301. PMLR, 2022.
  14. [Koh et al., 2021] Pang Wei Koh, Shiori Sagawa, Hen- rik Marklund, Sang Michael Xie, Marvin Zhang, Ak- shay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In Interna- tional conference on machine learning, pages 5637-5664.
  15. PMLR, 2021.
  16. [Kumar et al., 2020] Sunil Kumar, Ranbir Kumar, Carlo Cat- tani, and Bessem Samet. Chaotic behaviour of fractional predator-prey dynamical system. Chaos, Solitons & Frac- tals, 135:109811, 2020.
  17. [Lai et al., 2018] Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st international ACM SIGIR conference on research & devel- opment in information retrieval, pages 95-104, 2018.
  18. [Laurie and Lu, 2023] Mark Laurie and James Lu. Explain- able deep learning for tumor dynamic modeling and over- all survival prediction using neural-ode. npj Systems Biol- ogy and Applications, 9(1):58, 2023.
  19. [Li et al., 2020a] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy Hospedales. Sequential learning for domain gen- eralization. In European Conference on Computer Vision, pages 603-619. Springer, 2020.
  20. [Li et al., 2020b] Zongyi Li, Nikola Kovachki, Kamyar Az- izzadenesheli, Burigede Liu, Kaushik Bhattacharya, An- drew Stuart, and Anima Anandkumar. Fourier neural op- erator for parametric partial differential equations. arXiv preprint arXiv:2010.08895, 2020.
  21. [Li et al., 2022] Longyuan Li, Junchi Yan, Yunhao Zhang, Jihai Zhang, Jie Bao, Yaohui Jin, and Xiaokang Yang. Learning generative rnn-ode for collaborative time-series and event sequence forecasting. IEEE Transactions on Knowledge and Data Engineering, 35(7):7118-7137, 2022.
  22. [Li et al., 2023] Zongyi Li, Daniel Zhengyu Huang, Burigede Liu, and Anima Anandkumar. Fourier neural operator with learned deformations for pdes on general geometries. Journal of Machine Learning Research, 24(388):1-26, 2023.
  23. [Liu et al., 2023] Yong Liu, Chenyu Li, Jianmin Wang, and Mingsheng Long. Koopa: Learning non-stationary time series dynamics with koopman predictors. Advances in neural information processing systems, 36:12271-12290, 2023.
  24. [Liu et al., 2024] Jiaqi Liu, Jiaxu Cui, Jiayi Yang, and Bo Yang. Stochastic neural simulator for generalizing dy- namical systems across environments. In Proceedings of the Thirty-Third International Joint Conference on Artifi- cial Intelligence, pages 5909-5917, 2024. IJCAI
  25. [Lu et al., 2021] Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis. Learn- ing nonlinear operators via deeponet based on the univer- sal approximation theorem of operators. Nature machine intelligence, 3(3):218-229, 2021.
  26. [Min et al., 2011] Seung-Ki Min, Xuebin Zhang, Francis W Zwiers, and Gabriele C Hegerl. Human contribu- tion to more-intense precipitation extremes. Nature, 470(7334):378-381, 2011.
  27. [Nasery et al., 2021] Anshul Nasery, Soumyadeep Thakur, Vihari Piratla, Abir De, and Sunita Sarawagi. Training for the future: A simple gradient interpolation loss to gen- eralize along time. Advances in Neural Information Pro- cessing Systems, 34:19198-19209, 2021.
  28. [Nouinou et al., 2023] Hajar Nouinou, Elnaz Asadollahi- Yazdi, Isaline Baret, Nhan Quy Nguyen, Mourad Terzi, Yassine Ouazene, Farouk Yalaoui, and Russell Kelly. Decision-making in the context of industry 4.0: Evidence from the textile and clothing industry. Journal of cleaner production, 391:136184, 2023.
  29. [Oksendal, 2013] Bernt Oksendal. Stochastic differential equations: an introduction with applications. Springer Science & Business Media, 2013.
  30. [Qin et al., 2022] Tiexin Qin, Shiqi Wang, and Haoliang Li. Generalizing to evolving domains with latent structure- aware sequential autoencoder. In International Conference on Machine Learning, pages 18062-18082. PMLR, 2022.
  31. [R. Singh et al., 2024] Arvind R. Singh, R Seshu Kumar, Mohit Bajaj, Chetan B Khadse, and Ievgen Zaitsev. Ma- chine learning-based energy management and power fore- casting in grid-connected microgrids with multiple dis- tributed energy sources. Scientific Reports, 14(1):19207, 2024.
  32. [Rajalingham et al., 2022] Rishi Rajalingham, Aída Piccato, and Mehrdad Jazayeri. Recurrent neural networks with ex- plicit representation of dynamic latent variables can mimic behavioral patterns in a physical inference task. Nature Communications, 13(1):5865, 2022.
  33. [Salmela et al., 2021] Lauri Salmela, Nikolaos Tsipinakis, Alessandro Foi, Cyril Billet, John M Dudley, and Goëry Genty. Predicting ultrafast nonlinear dynamics in fibre op- tics with a recurrent neural network. Nature machine in- telligence, 3(4):344-354, 2021.
  34. [Tran et al., 2021] Alasdair Tran, Alexander Mathews, Lex- ing Xie, and Cheng Soon Ong. Factorized fourier neural operators. arXiv preprint arXiv:2111.13802, 2021.
  35. [Wang and Wang, 2013] Chang Yi Wang and CM Wang. Structural vibration: exact solutions for strings, mem- branes, beams, and plates. CRC Press, 2013.
  36. [Wang et al., 2023] Tong Wang, Xu-Wen Wang, Kathleen A Lee-Sarwar, Augusto A Litonjua, Scott T Weiss, Yizhou Sun, Sergei Maslov, and Yang-Yu Liu. Predicting metabolomic profiles from microbial composition through neural ordinary differential equations. Nature machine in- telligence, 5(3):284-293, 2023.
  37. [Wangersky, 1978] Peter J Wangersky. Lotka-volterra popu- lation models. Annual Review of Ecology and Systematics, 9:189-218, 1978.
  38. [Yan et al., 2024] Xiaoqin Yan, Zhou Huang, Shuliang Ren, Ganmin Yin, and Junnan Qi. Monthly electricity consump- tion data at 1 km× 1 km grid for 280 cities in china from 2012 to 2019. Scientific Data, 11(1):877, 2024.
  39. [Yin et al., 2021] Yuan Yin, Ibrahim Ayed, Emmanuel de Bézenac, Nicolas Baskiotis, and Patrick Gallinari. Leads: Learning dynamical systems that generalize across environments. Advances in Neural Information Processing Systems, 34:7561-7573, 2021.
  40. [Yong et al., 2023] LIN Yong, Fan Zhou, Lu Tan, Lintao Ma, Jianmeng Liu, HE Yansu, Yuan Yuan, Yu Liu, James Y Zhang, Yujiu Yang, et al. Continuous invariance learning. In The Twelfth International Conference on Learning Rep- resentations, 2023.
  41. [Zeng et al., 2024] Qiuhao Zeng, Wei Wang, Fan Zhou, Gezheng Xu, Ruizhi Pu, Changjian Shui, Christian Gagné, Shichun Yang, Charles X Ling, and Boyu Wang. Gener- alizing across temporal domains with koopman operators. In Proceedings of the AAAI Conference on Artificial Intel- ligence, volume 38, pages 16651-16659, 2024.
  42. [Zhang et al., 2017] Shuyi Zhang, Bin Guo, Anlan Dong, Jing He, Ziping Xu, and Song Xi Chen. Cautionary tales on air-quality improvement in beijing. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 473(2205):20170457, 2017.
  43. [Zhou et al., 2021] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wan- cai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 11106-11115, 2021.
  44. [Zhou et al., 2022] Tian Zhou, Ziqing Ma, Qingsong Wen, Liang Sun, Tao Yao, Wotao Yin, Rong Jin, et al. Film: Fre- quency improved legendre memory model for long-term time series forecasting. Advances in neural information processing systems, 35:12677-12690, 2022.

References were extracted automatically from the PDF and may contain errors.

Welcome Future Conferences AI Journal Division IJCAI Proceedings About IJCAI IJCAI Awards