PeSANet: Physics-encoded Spectral Attention Network for Simulating PDE-Governed Complex Systems

1 Introduction

Accurately modeling and simulating complex systems is crucial in various scientific and engineering fields, such as fluid dynamics, meteorology, and biology [Anderson and Wendt, 1995; Holton and Hakim, 2013; Ghergu and Radulescu, 2011]. These systems are typically described by partial differential equations (PDEs) that govern their spatiotemporal evolution. However, forecasting these systems efficiently and accurately in real-world scenarios presents significant challenges. Firstly, the physical laws that govern these systems can be incomplete or unknown, making it difficult to formulate the correct PDEs. Secondly, observational data are often scarce, limiting the ability to learn the underlying physics. Additionally, the characteristics of complex systems are multi-faceted, with local variants and global dependencies that need to be captured for reliable long-term predictions.

∗Corresponding author (Email: haosun@ruc.edu.cn).

Over the years, various methods have been developed to address these challenges. Traditional numerical methods [Moukalled et al., 2016; Karniadakis and Sherwin, 2005; Zienkiewicz et al., 2005] enjoy good convergence and stability when PDEs are well-defined, but become difficult to apply when the system's governing laws are incomplete. To overcome this, data-driven methods such as DeepONet [Lu et al., 2021] and Fourier Neural Operators (FNOs) [Li et al., 2021] use neural networks as surrogate models, aiming to learn the underlying physical evolution from data. These methods do not require explicit knowledge of the physical laws and achieve significant speedup compared to traditional numerical solvers. However, they often fail to offer meaningful predictions when the training data is limited, which is a common issue in real-world scenarios.

To address the limitations of purely data-driven approaches, researchers have developed a series of physicsaware methods that incorporate prior physical knowledge into neural networks. These methods can be classified into physics-informed and physics-encoded, depending on how they integrate the physical priors. Physics-informed methods typically use soft constraints, such as PDE residual-based loss functions, to enforce the model that aligns with the governing PDEs. Representative methods include PhysicsInformed Neural Networks (PINNs) [Raissi et al., 2019] and Physics-Informed Neural Operators (PINOs) [Li et al., 2024b]. However, due to their soft-constrained nature, these methods encounter challenges in hyperparameter tuning and training stability, and require sufficient prior knowledge of the governing PDEs [Krishnapriyan et al., 2021]. To this end, physics-encoded methods use a hard-constrained strategy to incorporate physical prior into the model architecture. For instance, the Physics-encoded Recurrent Convolutional Neural Networks (PeRCNN) [Rao et al., 2023; Rao et al., 2022] use specially designed convolutional kernels to approximate finite difference stencils. These approaches can automatically learn local differential operators that match the observed data, even when the governing PDE is unknown. However, due to their limited receptive field, they can only capture local variations and struggle to learn long-range spatial dependencies in complex systems.

To overcome these limitations, we propose the Physicsencoded Spectral Attention Network (PeSANet). The model consists of the physics-encoded and spectral-enhanced blocks, which learn the local and global evolution of the complex system, respectively. The physics-encoded block employs hard constraints to approximate local differential operators. This hard-constrained approach does not require sufficient physical prior knowledge and can achieve generalization from the limited observed data. On the other hand, the spectral-enhanced block handles spectral information in the frequency domain to capture global features. In particular, we proposed the spectral attention mechanism to exploit the inter-spectrum relationship, further enhancing the ability to learn long-range spatial dependencies. In summary, we make the following contributions:

• We propose the Physics-encoded Spectral Attention Network (PeSANet) to integrate both local and global learning mechanisms, which can achieve long-term forecasting for complex systems with limited data and incomplete physical priors.
• We propose a spectral attention mechanism in the frequency domain to capture long-range spatial features. By using all frequency components and capturing interactions across spectral layers, this approach can enhance the model's ability to describe global system features.
• PeSANet achieves superior long-term prediction performance across multiple complex systems. Furthermore, it outperforms other baseline models in all metrics, showing robust generalization with limited training data and transfer learning ability across varying conditions.

2 RelatedWork

The numerical solution of PDEs has been a central focus of scientific computing for decades. Traditional numerical methods, such as finite difference and finite element methods, have achieved remarkable success in many applications. However, these methods can be computationally expensive and require significant domain knowledge. In recent years, deep learning has emerged as a powerful tool for solving PDEs, offering the potential for more efficient and flexible solutions. Existing AI-driven methods can be categorized into data-driven, physics-aware, and hybrid models. The key distinction lies in the incorporation of prior physical knowledge.

2.1 Data-driven learning methods

When sufficient data is available, many works have investigated neural networks for learning the dynamics governed by time-dependent PDEs. These methods generally fall into two categories. The first category leverages well-established architectures from the computer vision domain, such as classical convolutional neural networks (CNNs) [Stachenfeld et al., 2021; Bar-Sinai et al., 2019], U-Net [Gupta and Brandstetter, 2023], ResNet [Lu et al., 2018], and ConvLSTM [Shi et al., 2015], to capture intricate spatiotemporal patterns. These architectures excel at extracting local features and are naturally suited for data represented on regular grids or meshes.

The second category involves neural operators, which focus on learning mappings between function spaces. This class includes models like DeepONet [Lu et al., 2021] and its variations [Seidman et al., 2022; Venturi and Casey, 2023; Lee et al., 2023], as well as the FNO [Li et al., 2021] and its derivatives [Tran et al., 2023; Rahman et al., 2023; Wen et al., 2022; Zhang et al., 2024]. Neural operators offer the advantage of mesh independence, enabling generalization across different discretizations of the same class of PDEs. Additionally, advances in other data-driven neural PDE solvers, such as graph neural networks (GNNs) [Sanchez-Gonzalez et al., 2020; Pfaff et al., 2021] and transformer-based architectures [Wu et al., 2024; Hang et al., 2024; Janny et al., 2023; Li et al., 2024a], provide powerful tools to extend neural operators to various complex geometries.

Despite the progress achieved by these data-driven methods, they still face several limitations. One significant drawback is their heavy reliance on large and high-quality datasets, which are often time-consuming to generate or infeasible to many scientific and engineering domains. Moreover, these models are prone to overfitting and fail to generalize to unseen data, which is a key challenge in complex physical systems due to their inherent complexity and diversity.

2.2 Physics-aware learning methods

When data is limited, integrating physical knowledge into neural networks becomes important. This integration typically follows two strategies: physics-informed (soft constraints) and physics-encoded (hard constraints). Physicsinformed methods incorporate physical laws, such as the PDE formulations, into the loss function. Representative works include PINN [Raissi et al., 2019], PINO [Li et al., 2024b], PhyCRNet [Ren et al., 2022], MCNP [Zhang et al., 2025], and PI-DeepONet[Wang et al., 2021]. These methods can be data-efficient but face challenges such as hyperparameter tuning, training instability, and requiring substantial prior knowledge of the governing equations [Krishnapriyan et al., 2021].

On the other hand, physics-encoded approaches directly embed physical constraints into the network architecture, such as PDE operators, conservation laws, and boundary conditions [Long et al., 2019; Rao et al., 2023; Mi and Sun, 2024; Zeng et al., 2025]. These methods can be highly effective when working with limited data and incomplete physics priors. However, hard-constrained methods like PeRCNN, which use specially designed convolutional kernels to approximate local differential operators, often struggle to capture global information, limiting their performance on problems that require long-range spatial dependencies.

2.3 Hybrid learning methods

Hybrid models provide an effective acceleration strategy for solving PDEs when the governing equations are fully known. These models couple neural networks with traditional numerical methods (e.g., FDM, FVM) to combine their advantages. Neural networks approximate computationally expensive components of the PDE solution, such as complex nonlinearities or fine-scale features, while established numerical methods handle the more tractable parts. This coupling enhances computational efficiency without sacrificing accuracy.

Examples include multi-scale hybrid models [Vlachas et al., 2022] and other couplings with FVM (learned interpolation model [Kochkov et al., 2021] and temporal stencil modeling [Sun et al., 2023]) and FDM (P2C2Net [Wang et al., 2024]). However, these models require complete knowledge of the PDEs, restricting their application to systems with unknown or partially unknown physics.

3 Methodology

3.1 ProblemFormulation

Let us consider spatiotemporal complex systems governed by the following time-dependent PDEs:

where F(·) represents an unknown operator that involves spatial derivatives of various orders, such as the gradient ∇u and the Laplacian ∇2u. The parameter µ governs the physical properties of the system. Additionally, the initial conditions (ICs) and boundary conditions (BCs) are denoted as I[u](x, t = 0) = 0 and B[u](x, t) = 0, respectively.

In this paper, we aim to use the neural network as a surrogate model to simulate the complex systems based on incomplete physical prior of F and limited training data (e.g., 2-5 trajectories). The trained model is expected to learn the underlying dynamics of the physical system, which can accurately predict long-term system dynamics and generalize across different ICs. Traditional numerical and hybrid methods are not applicable to this scenario because they require sufficient prior knowledge of the physical system. On the other hand, data-driven methods demand large numbers of training data to achieve reasonable generalization performance. To this end, we propose the PeSANet.

3.2 PeSANet Architecture

In this section, we introduce the overall architecture of the proposed PeSANet (Figure 1(a)). The model has two key components: the physics-encoded block and the spectralenhanced block, which are designed to learn the local and global features of the complex system, respectively. The physics-encoded block uses multiplicative convolution to approximate the finite difference (FD) stencil, which extracts the local spatial information in a hard-constrained manner. The spectral-enhanced block conducts spectral attention operations in the frequency domain, which can further enhance the performance via capturing global features, such as longrange spatial dependencies. This synergistic combination allows the model to learn local details and global structures of the complex system, resulting in a thorough understanding of the underlying physical phenomena.

3.3 Physics-encoded Block

The physical-encoded block (Figure 1(b)) aims to learn the underlying local differential operators that govern the complex system via a hard-constrained approach. Similar to PeRCNN [Rao et al., 2023], we adopt the Π-block and physicsbased convolutional (PyConv) layer to handle known and unknown operators, respectively.

Considering the forward Euler scheme, the updated rule of the system in Equation 1 is given by:

where δt denotes the step size. The operator F is then approximated using the following Π-block ˆ F:

where Nc represents the number of channels, and Nl is the total number of parallel convolutional layers. The symbol ⋆ denotes the convolution operation. For each layer l and channel c, Kc,l denotes the filter weight, while bl represents the bias for layer l. Wc is a 1×1 convolutional layer assigned to the cth channel.

The main idea of the Π-block is to use multiplicative convolutional layers to approximate the FD stencil. Because of its multiplicative structure, the Π-block can efficiently approximate unknown nonlinear terms, such as u · ∇u. Theorem 1 in [Rao et al., 2023] demonstrates that the Π-block can serve as a universal polynomial approximator for the unknown nonlinear operator.

Furthermore, to effectively incorporate the partially known knowledge of the PDE system, we use the physics-based convolutional (PyConv) layer as a complementary strategy. When a term in the governing PDE is known, such as the Laplacian term ∇2u, we directly set the kernel of the PyConv to align with the corresponding FD stencil. As a result, the PyConv layer is constructed to incorporate known physical information, while the Π-block is designed to learn the unknown operators. The combination of the Π-block and PyConv forms the physics-encoded block, which fully utilizes the physics structure of the differential operator and can efficiently model nonlinear systems from limited training data.

3.4 Spectral-enhanced Block

Although the physics-encoded block can efficiently approximate local operators within the complex system, it still fails to capture long-range spatial dependencies due to its limited receptive field. To overcome this limitation, we propose the spectral-enhanced block to process global information in the frequency domain, enabling a more comprehensive and holistic understanding of the complex system.

The proposed spectral-enhanced block (Figure 1(c)) contains the following components. An encoder (E) is first employed to map the spatial physical field into the frequency domain via the Fast Fourier Transform (FFT). After that, a frequency domain operator processes the transformed information in the frequency domain. Subsequently, the data is returned to the spatial domain using an inverse FFT (IFFT), followed by a decoder (D) to reconstruct the spatial field.

Frequency Domain Operator

The frequency domain operator (Figure 1(d)) is specifically designed to process information in the frequency domain. The block takes in the input spectrum and applies the following series of operations.

Firstly, a spectral attention mechanism is introduced to capture inter-spectrum relationships, as detailed in the next section. This mechanism allows the model to identify and emphasize key spectral features critical for the task, enhancing its ability to capture dependencies across different spectral layers. Secondly, the processed spectrum is aggregated with the original spectrum via element-wise addition. This skip connection strategy helps ensure robustness during training and preserves the original frequency information. Finally, the aggregated spectrum undergoes filtering and a linear transformation, aligning with the traditional neural Fourier layer [Li et al., 2021].

These operations work together to handle the frequency signals, enhance the model's representational capacity, and ultimately produce the output spectrum.

Spectral Attention Mechanism

Motivated by the channel attention module [Wang et al., 2020], we introduce the spectral attention mechanism (Figure 1(e)) within the frequency domain operator, which effectively exploits the inter-spectrum relationships among spectral features. Given a complex feature denoted as X+ iY ∈ Cc×k1×k2 , where c denotes the number of channels and k1 × k2 represents the number of frequency components, we first aggregate the frequency information of each feature map via the following average-pooling (Avg) and max-pooling (Max) operations:

We handle the real and imaginary parts separately because the complex field is not well-ordered, i.e., there is no natural ordering between complex numbers. As a result, directly applying the max-pooling operation to the complex-valued features would lead to meaningless results. After the pooling operations, we apply four multi-layer perceptrons (MLPs) with one hidden layer to capture the inter-spectrum relationships as follows:

where σ represents an activation function. Applying the attention coefficients to re-weight the input spectrum, the processed spectrum Xp + iYp ∈ Cc×k1×k2 can be obtained as:

where ⊗ denotes the element-wise multiplication.

Compared to the traditional neural Fourier layer [Li et al., 2021], the spectral attention mechanism further enhances the representation capability for the following reasons. On the one hand, the spectral attention mechanism uses all frequency components to calculate the attention coefficients, instead of directly truncating high-frequency signals. This comprehensive consideration ensures all critical frequency information in the spectral domain is leveraged effectively. On the other hand, the attention mechanism in Equation 5 enhances the interactions between spectral layers by dynamically capturing their internal relationship. This interaction allows information to be processed jointly across spectrum layers, resulting in a more expressive representation of spectral features.

4 Experiments

To rigorously evaluate the efficacy and generalizability of the proposed PeSANet, we conducted a series of comprehensive experiments on a wide range of complex systems, such as fluid dynamics and reaction-diffusion systems. Specifically, the model is evaluated on the two-dimensional Burgers' equation, the two-dimensional FitzHugh-Nagumo (FN) system, the two-dimensional Gray-Scott (GS) system, and the two-dimensional Navier-Stokes equations (NSE). The experimental results demonstrate that the PeSANet consistently achieves superior accuracy compared to existing methodologies across all challenging benchmarks. These experiments not only showcase the model's robust adaptability to a diverse set of systems but also confirm its ability to accurately predict long-term dynamics.

4.1 Experimental Setup

This section details the experimental setup, including the datasets, baselines, metrics, and the training procedure.

Datasets. We considered several two-dimensional PDEgoverned nonlinear complex systems, including the Burgers' equation, the FitzHugh-Nagumo (FN) system, the Gray- Scott (GS) system, and the Navier-Stokes equations (NSE). These systems represent diverse physical phenomena and dynamic behaviors, providing a comprehensive assessment of the model's performance and generalization capabilities. In this paper, we target the data-scare scenario, and each experiment includes 2-5 trajectories in the training set. We summarize the datasets in Table 1, with a more detailed introduction in Appendix Dataset Informations.

Baseline Models. To demonstrate the effectiveness of PeSANet, we compared it with the following baseline models:

• FNO [Li et al., 2021]: A widely recognized highperforming data-driven neural operator, which utilizes the Fourier transform to handle PDE information in the frequency domain.
• PeRCNN [Rao et al., 2023]: A model that embeds physical knowledge in a hard-constrained manner and demonstrates excellent long-term prediction ability.
• F-FNO [Tran et al., 2023]: The Factorized Fourier Neural Operator, which employs a separable Fourier layer and enhanced residual connections to effectively reduce computational complexity and improve scalability.
• FactFormer [Li et al., 2024a]: A model that addresses the computational expense and numerical instability of applying Transformers to PDEs with large grids by employing an axial factorized kernel integral.

Evaluation Metrics. We use three types of metrics to quantitatively assess the model performance: Root Mean Square Error (RMSE), Mean Absolute Error (MAE), and High Correction Time (HCT). Specifically, RMSE quantifies the average magnitude of prediction errors in the original data units, offering a general measure of model performance. MAE, conversely, measures the average absolute difference between predicted and ground truth values, exhibiting robustness to outliers compared to RMSE. Finally, HCT quantifies the temporal duration required for the model's predictions to converge to a high level of accuracy, a critical consideration for real-time applications. The complete mathematical formulations of these metrics can be found in Appendix Evaluation Metrics.

Model Training. We train our model autoregressively, where the model predicts an output at each iteration, and the output is used as input for the next prediction. The mean squared error loss function is utilized for training, optimized using the Adam optimizer.

4.2 Main Results

The performance characteristics of the proposed PeSANet are comprehensively evaluated and presented in Figure 2 and Table 2. Demonstrating robust performance despite limited training data, Figure 2 provides visualizations of error distribution, error propagation characteristics, and predicted trajectories. These qualitative results are further substantiated by the quantitative metrics presented in Table 2.

2D Burgers Equation. The efficacy of the proposed PeSANet framework is demonstrated in the context of the 2D Burgers equation (Figure 2(a)). While PeRCNN exhibited some success in the beginning, it struggled with long-term predictions, as evidenced by the errors observed in Figure 2(a). In contrast, the PeSANet framework demonstrates superior performance in capturing the underlying physical dynamics, particularly in long-term forecasting. This qualitative advantage is further corroborated by the quantitative metrics presented in Table 2, which show significant improvements in both RMSE and MAE. The divergence of PeRCNN is also evident in the final time step snapshot.

2D FitzHugh-Nagumo Equation. Figure 2(b) demonstrates the difficulties encountered by data-driven methods (FFNO, FNO, and FactFormer) when trained on limited data, with FactFormer showing near-complete failure. Even FFNO, the strongest among these, exhibits limitations in the long-term prediction of the FN dynamics. Although PeRCNN, incorporating physics-informed structures, generates predictions with similar contours to the ground truth, its long-term accuracy remains limited. In contrast, our model achieves at least a 30% improvement over the best baseline, demonstrating the benefits of our approach for small datasets and the crucial role of physics-encoded block in enhancing long-term prediction accuracy.

2D Gray-Scott Equation. According to Table 2, PeSANet achieves substantial gains of approximately 50% in both RMSE and MAE. The complex, labyrinthine patterns of the GS-RD system are visualized in the snapshots presented in Figure 2 (c). A clear visual comparison reveals that PeSANet maintains a reasonable resemblance to the ground truth even after long-term prediction, with only minor deviations. In contrast, other baseline models fail to capture the intricate patterns or even the basic contours of the structures. 2D Navier-Stokes Equation. For complex fluid dynamics, we evaluate the performance of the proposed PeSANet with other baseline models on a Navier-Stokes Equations (NSE) dataset with a Reynolds number of 1000. This task requires models that can accurately capture both local variations and global features to ensure reliable long-term predictions. Compared to other baseline, our model achieves significant improvements of 20% and 30% in RMSE and MAE (Table 2). The error propagation curves in Figure 2(d) further validate the superiority of PeSANet, with its error consistently lower than that of other models. The snapshot at t = 7s shows near-perfect agreement between PeSANet's predictions and the ground truth, while other models exhibit significant deviations. This collective evidence establishes PeSANet as the superior and more robust model for long-term prediction tasks in complex fluid dynamics systems.

4.3 Generalization Test

Our model leverages incorporated physical knowledge to enable effective fine-tuning and robust generalization with scarce training data. Using the NSE as a representative example, PeSANet demonstrates excellent performance at the training Reynolds number (Re = 1000) and further exhibits strong generalization capabilities by fine-tuning with limited data to new Reynolds numbers (Re = 500, 800, and 2000). The error distributions across these Reynolds numbers (Figure 3) confirm the generalization ability of PeSANet. The error curves maintain similar characteristics to the Re = 1000 case, with error propagation well controlled, demonstrating the model's reliability across different flow regimes. It is worth mentioning that fine-tuning with only two data samples enables PeSANet to generate predictions closely approximating the ground truth, further validating its robustness and generalization capacity. In summary, PeSANet effectively captures the underlying physical principles of the system, achieving accurate predictions for NSE at varying Reynolds numbers with minimal fine-tuning data.

4.4 Ablation Study

To evaluate the contributions of different components in our PeSANet, we designed a series of ablation studies based on the FN equation. Specifically, we conducted experiments by (1) PeSANet w/o SA, removing the spectral attention component, (2) PeSANet w/o Pe, removing the physical-encoded block in the physical module, (3) Pe+FNO, replacing the spectral-enhanced block with FNO, and (4) Pe+F-FNO, replacing the spectral-enhanced block with F-FNO.

As shown in Table 3, experimental results demonstrate that removing the spectral attention module leads to a significant decline in network performance, highlighting the crucial role of the spectral attention mechanism in capturing spectral information and enhancing the model's representational capacity. The removal of the physics-encoded block results in poor performance on small datasets, indicating the importance of prior physical knowledge in improving the model's generalization ability, especially when data is limited. Furthermore, completely replacing the spectral-enhanced block with FNO or F-FNO also results in a significant performance drop.

5 Conclusion

This paper introduces PeSANet, a predictive model designed to enhance the prediction accuracy of complex systems governed by PDEs by integrating global and local information, leveraging available physical data, and emphasizing frequency-domain feature processing. The PeSANet is designed to not only capture the overall system dynamics but also exploit local information through hard-coding physical knowledge. Furthermore, it incorporates a carefully designed spectral attention module to capture key spectral features, effectively handling the global information in the frequency domain. These strategies significantly improve PeSANet's data efficiency, enabling superior long-term prediction performance and generalization capabilities even with limited training data. Experimental results demonstrate that PeSANet achieves state-of-the-art predictive performance on four distinct nonlinear dynamical systems with small datasets, exhibiting strong adaptability and flexibility even with limited data and varying initial conditions. The generalization test also demonstrates PeSANet's transfer learning ability. In conclusion, PeSANet offers a novel approach to long-term forecasting by integrating physics and spectral attention, demonstrating superior performance in data-scarce scenarios.

Acknowledgments

The work is supported by the National Natural Science Foundation of China (No. 62276269, No. 92270118), the Beijing Natural Science Foundation (No. 1232009), and the Strategic Priority Research Program of the Chinese Academy of Sciences (No. XDB0620103). Y. Liu would like to thank the support from the Fundamental Research Funds for the Central Universities (No. E2EG2202X2). The Appendix of this manuscript can be found in the preprint version: https: //www.arxiv.org/abs/2505.01736. The source code and data are found at https://github.com/intell-sci-comput/PeSANet.

Contribution Statement

Han Wan and Rui Zhang contributed equally to this work.