Structure-Preserving Physics-Informed Neural Networks with Energy or Lyapunov Structure

1 Introduction

Physics-informed neural networks (PINNs) [Raissi et al., 2019] aim to embed seamlessly domain knowledge from physical laws into neural networks. Their ability to employ the power of physical principles while leveraging a datadriven approach has shown great potential for addressing complicated tasks in physics, engineering, and beyond [Karniadakis et al., 2021; Lu et al., 2021a; Cuomo et al., 2022]. PINNs offer several advantages over conventional numerical methods, such as significantly reducing the computational cost, being mesh-free, and being capable of solving both forward and inverse problems in a unified framework.

∗Corresponding author.

Despite their empirical success, PINNs often suffer from decreased accuracy when dealing with strongly nonlinear and higher-order differential equations [Mattey and Ghosh, 2022]. We consider one potential reason is that PINNs overlook the underlying geometric structure of the physical system, leading to numerical solutions that may suggest nonphysical behavior. Most numerical methods have the same issue and therefore fail to preserve the conservative or dissipative property of the dynamical systems. To overcome this limitation, the development of structure-preserving numerical methods has gained momentum [Hairer et al., 2006; Sharma et al., 2020]. A structure-preserving algorithm can guarantee that qualitative characteristics, such as invariant or energy dissipation, are represented in the simulation, providing accurate numerical results over long periods. Thus, it is essential to investigate the application of the structurepreserving idea in PINNs. Nevertheless, most structurepreserving schemes are inapplicable to conventional neural networks because they need an intricate manual derivation of the system equation [Matsubara et al., 2020]. Thanks to PINNs combining differential equations and neural networks, which provides fundamental support for us to introduce the idea of structure-preserving methods into PINNs.

In addition, the current applications of PINNs are mainly focused on solving differential equations. There is little research on their applications on downstream tasks, such as image recognition tasks. Giga et al. [2013] showed the potential power of leveraging the heat equation for binary classification. Another family of differential equations based deep neural networks, namely neural ordinary differential equations (Neural ODEs) [Chen et al., 2018], shows promising results on downstream tasks, which describes the continuous dynamics of hidden units utilizing an ODE parameterized by a neural network. The advantage of utilizing PINNs is that one can directly obtain the solutions through the forward inference of the neural network, avoiding the approximation of the numerical integration in Neural ODEs. Besides, we hypothesize that there exists a structure of stability (in the Lyapunov sense) in image recognition tasks, i.e., minor perturbations on the input image will not influence the classification result. We consider the failure of current neural networks on adversarial samples [Szegedy et al., 2013; Zhang and Li, 2019], crafted by adding minor human-imperceptible perturbations to images, is due to the neglect of preserving the stability structure. Thus, introducing structure-preserving PINNs for image recognition is expected to enable the model to resist initial perturbations. Considering the threat adversarial examples pose to the security of deep learning systems [Eykholt et al., 2018; Lu et al., 2021b], it is meaningful to explore the application of structure-preserving PINNs to robust image recognition.

To address these problems, we propose a new family of PINNs, named structure-preserving PINNs (SP-PINNs), that can be applied to dynamical systems with energy or Lyapunov structure. Our approach is capable of solving PDEs (e.g., the Allen-Cahn equation) and handling downstream tasks (e.g., image recognition). Our main contributions are summarized as follows:

• Based on prior knowledge about the physical system, we introduce a structure-preserving loss function to assist PINN in learning the structure of the underlying system, improving its performance on numerical simulation.
• We propose an SP-PINN for robust image recognition. Here, we treat the input image after downsampling as the initial value of the differential equation and assume a general form of the unknown underlying dynamical system. In this scenario, preserving the Lyapunov structure of the underlying system ensures the stability of the system.
• Experiments on standard benchmarks show that SPPINN can consistently outperform the baseline model in terms of robustness against adversarial attacks, which supports our hypothesis on the structure of stability regarding image recognition tasks.
• We show that combining the SP-PINN with adversarial training methods further enhances the robustness against adversarial examples.

2 RelatedWorks

2.1 Neural Networks for Differential Equations

The first attempts at utilizing neural networks to solve differential equations began in the 1990s [Lee and Kang, 1990]. Recent progress in automatic differentiation techniques has enabled researchers to design more complicated neural network architectures.

Among different data-driven techniques for solving differential equations, PINNs [Raissi et al., 2019] have shown remarkable promise and versatility. Mattey et al. [2022] designed a novel PINN scheme that re-trains the same neural network for solving the PDE over successive time segments while satisfying the already obtained solution for all previous time segments. Krishnapriyan et al. [2021] analyzed that possible failure modes in PINNs are attributed to the PINNs' setup making the loss landscape very hard to optimize. Jagtap and Karniadakis [2021] extended PINN by leveraging the generalized space-time domain decomposition for solving arbitrary complex geometry domains. Wang et al. [2024] introduced a neural architecture search-guided PINN. Zhao et al. [2024] proposed a Transformer-based PINN to capture the crucial temporal dependencies inherent in practical physics systems. Lau et al. [2024] presented a algorithm for training PINN that optimally selects collocation and experimental points by jointly considering their interactions and dynamically adjusting proportions during training.

2.2 Structure-Preserving Deep Learning

Greydanus et al. [2019] parameterized the Hamiltonian mechanics with a neural network and then learned it via a datadriven approach, which conserved an energy-like quantity. Sosanya et al. [2022] proposed a dissipative Hamiltonian neural network that leverages the tools of Hamiltonian mechanics and Helmholtz decomposition to separate conserved quantities from dissipative quantities. Lutter et al. [2019] proposed a neural network for learning the mechanic systems of the Euler-Lagrange equation employing end-to-end training while keeping physical plausibility. [Matsubara and Yaguchi, 2023] utilized Neural ODEs for finding and preserving invariant quantities by leveraging the projection method and the discrete gradient method [Matsubara et al., 2020]. Jagtap et al. [2020] presented a conservative PINN on discrete sub-domains by using a separate PINN in each subdomain, and then stitching back all sub-domains through the corresponding conservative quantity, which is different from our approach that is based on the prior knowledge about the underlying dynamics and is theoretically applicable both conservative and dissipative systems.

2.3 Adversarial Defense Methods

Since realizing the instability of deep neural networks, researchers have proposed different kinds of complementary techniques to defend adversarial examples, such as distillation defense [Papernot et al., 2016; Chu et al., 2023] and adversarial training. Maday et al. [2017] proposed an adversarial training method by injecting adversarial examples generated by PGD attacks into training data. Ilyas et al. [2019] created a robust dataset for adversarial training by removing non-robust features from the dataset. Lamb et al. [2022] augmented the adversarial training with interpolation-based training, which aims to tackle the problem that traditional adversarial training aggravates the generalization performance of the networks on clean data.

After knowing the connection between dynamical systems and deep neural networks, designing defense methods based on the Lyapunov stability theory becomes a new trend. Kang et al. [2021] proposed stable Neural ODE based on Lyapunov's first method for resisting adversarial examples. Chu et al. [2024] leveraged Lyapunov's second method for preventing successful adversarial attacks by inherently constraining deep equilibrium models (DEQ) [Bai et al., 2019] to be stable.

It is worth mentioning that our proposed method is the first attempt at utilizing modified PINNs for adversarial defense.

3 Methodology

This section presents the applications of the proposed SPPINNs to PDEs (e.g., the Allen-Cahn equation), and robust image recognition.

The strategies are summarized as follows: firstly, a neural network is specified for each of the different tasks to fit the mapping from the input to the numerical solution, and then the derivatives w.r.t. the input is calculated by automatic differentiation; secondly, the core of the proposed method involves incorporating prior knowledge about the energy or Lyapunov structure of the system into the specific neural network; thirdly, the neural network is trained by minimizing the loss function. For different scenarios, we design a corresponding training process and loss function to cater to the unique characteristics of each problem domain.

3.1 SP-PINN for PDEs

We consider the Allen-Cahn equation [Allen and Cahn, 1972], which is a strongly nonlinear reaction-diffusion equation. The Allen-Cahn equation is widely used for modeling some phase separation and domain coarsening phenomena. Currently, PINN's accuracy suffers significantly from strongly nonlinear PDEs [Mattey and Ghosh, 2022]. Therefore, we aim to improve the solutions' accuracy by learning the underlying structure of the equation. The Allen-Cahn equation is formulated as:

where p > 0, q > 0, r < 0, x ∈ [0,L] and t ∈ [0, T]. The initial condition is u(x, 0) = u0(x). Here, we employ the Neumann boundary condition ux(0, t) = ux(L, t).

To reveal the energy dissipative property of the Allen-Cahn equation, we first introduce a quality, which is called the 'free energy' or 'local energy' of the problem:

The evolution of the solution is shown to evolve in a direction that the global energy is decreased [Furihata and Matsuo, 2010]:

where J(u) is the 'global energy', which is a functional of u. Meanwhile, J(u) can be regarded as a function of t.

To solve the Allen-Cahn equation, the structure-preserving PINN is defined as a N-layer fully connected network (FCN) with inputs x, t and output ˆu(x, t). The framework is illustrated in Figure 1.

We obtain the derivatives of the output ˆu(x, t) w.r.t. position x and time t via automatic differentiation and randomly select Nf in the spatiotemporal region (x, t) to calculate the equation residual:

Then, we randomly select Ni points in the region (x, 0) to calculate the initial condition residual:

Next, we randomly select Nb points in the regions (0, t) and (L, t) to calculate the boundary condition residual:

Finally, to specify the structural loss, we define the discrete global energy accordingly by:

where ∆x = L/M, M is the number of the spatial grid points, ∑Mk=0 ′′ f denotes the trapezoidal rule:

We uniformly select Ne points in the time interval [0, T ] to calculate the structural loss:

where ReLU = max(0, x) is the rectified linear unit function.

Therefore, the total loss function for training the proposed model is the summation of the equation residual, the initial condition residual, the boundary condition residual, and the structural loss:

where λ1, λ2, λ3, λ4 are hyperparameters.

3.2 SP-PINN for Robust Image Recognition

Currently, PINNs are mostly utilized for solving differential equations. In this section, we delve into the extension of PINNs to downstream tasks and propose an SP-PINN for robust image recognition. First, we define the image recognition task as an initial value problem, where the input image after downsampling is treated as the initial value, and the approximate solution is obtained through the evolution of time. Secondly, we project the learned dynamical system into a space that satisfies the Lyapunov stability condition to preserve the stability structure.

We hypothesize that there is a structure of stability in the Lyapunov sense in image recognition tasks, that is, small changes put on the input image will not influence the classification result. Thus, introducing the structure-preserving idea into the image recognition tasks is expected to enable the classification model to have resistance against the minor perturbations added on the input image, making the model more robust.

However, the challenge of utilizing PINN for image recognition is that the underlying dynamical system is unknown. Therefore, we specify the following ODE to describe the system:

where the initial value is u(0) = u0 and the time interval is [0, T ].

It is worth mentioning that Neural ODEs [Chen et al., 2018] also use Eqn. (11) to describe the continuous dynamics of hidden units, which show promising results on downstream tasks. Our proposed method diverges from Neural ODEs in terms of the approach to solving the initial value problem.

Theoretically, the analytic solution can be obtained by integrating Eqn. (11):

In practice, Neural ODEs utilize the ODE solver, such as Runge-Kutta, Dopri5, to approximate the numerical integration ∫ T 0 f(u(t), t) dt. This procedure is formulated by:

Unlike Neural ODEs, the advantage of utilizing PINN is that PINN can obtain the numerical solution at time T directly through the forward inference of the neural network, that is, u(T) = PINN((u(0), T, θ)), avoiding the approximation of the numerical integration. Nevertheless, this also poses a problem in that the form of Eqn. (11) needs to be known in advance. To address this problem, we assume a general form of the underlying dynamical system following Hu et al. [2022]:

where Λ = {1, u1, u2, · · · , un, u1u1, u1u2, · · · , upn } is the complete set of pth -order polynomial basis with unknown coefficients α = (αij)n×M, M = (p+n p ). Here, we take the downsampling result of the input image as the initial value u(0). Like Neural ODEs, the final time for PINN is set to T = 1.

To obtain the unknown parameters α, we need to first solve the inverse problem. As long as we know the underlying dynamical system, we can calculate the numerical solution via the forward inference, i.e., solving the forward problem. Motivated by Raissi et al. [2019] and Kim et al. [2023], the learning process of the proposed model is achieved by solving the inverse and forward problems alternately.

The SP-PINN for image recognition is defined as a N-layer residual network, where the input u is a vector concatenated by initial value u(0) with time t (after broadcasting). The scratch of the architecture of the proposed method is shown in Figure 2. The details of the training process are as follows.

How to solve the inverse problem? We solve the inverse problem through a data-driven approach. After initializing the network parameters, we can input the images of the training set and get their corresponding outputs. Then, we minimize the following loss function regarding equation residual to find the α:

where Nt is the number of the training set.

How to solve the forward problem? As long as the underlying system is known, we can calculate the approximate solutions through the forward inference. The new solutions are in turn utilized for solving the inverse problem to update

α. Repeated iterations of this process yield more accurate α and ˆu.

While solving the inverse problem, we fix the trainable parameters θ of the neural network and learn the unknown parameters α of the underlying dynamical system. While solving the forward problem, we fix the parameters α of the dynamical system and train the parameters θ of the neural network, in this case, the output of the model is the approximate solution ˆu.

How to preserve the stability structure? To preserve the stability structure, that is, ensuring the proposed model is robust to minor perturbations on the initial value, we jointly learn a convex positive definite Lyapunov function along with dynamics constrained to be stable and project the learned dynamical system of PINN onto a space where the Lyapunov exponential stability condition holds. Please refer to the [Chu et al., 2024] for the definition of Lyapunov stability and the Lyapunov stability theorem.

We construct the structure-preserving module motivated by Manek and Kolter [2019]. Let F(u) = dNN(u)/dt be a basic dynamic model, let V : Rn → R be a positive definite function, and c be a nonnegative constant, the exponentially stable dynamical model is defined as

where, φ(u) = ∇V (u)T F(u) + cV (u). For φ(u) > 0, the output of the base dynamics model is projected onto a halfspace where this condition holds, otherwise, the output is returned unchanged.

The Lyapunov function V is defined as positive definite and continuously differentiable, and has no local minima:

where σ is a positive convex non-decreasing function with σk(0) = 0, and g is represent as a input-convex neural network (ICNN) [Amos et al., 2017]. Since the Lyapunov function is defined as continuously differentiable, we can use automatic differentiation to compute its gradient. This advantageous feature allows us to train our final network in a manner similar to any other network.

Consequently, we ensure that the proposed model satisfies the Lyapunov exponentially stable condition through the procedures described above, thereby preserving the Lyapunov structure of the underlying dynamical system.

The FC layer, that is, a weighted linear transformation, takes part in predicting the category of the image.

The loss function for training the SP-PINN is obtained as follows. First, obtain the derivative of the network's output ˆu w.r.t. time t. Second, calculate the equation residual:

As can be seen, the equation residual in the forward problem is the same form as in the inverse problem, i.e., Eqn. (15).

Hence, PINNs can deal with forward and inverse problems in a unified paradigm, which offers a great advantage compared to traditional numerical methods [Chirigati, 2021] that need to design different schemes for different inverse problems.

Then, we calculate the initial condition residual:

where Nh is the number of elements in ui(0).

Finally, we utilize the cross-entropy (CE) loss to measure the difference between the FC layer's result y and the true label ˆ y of the image:

The loss function used to update the network parameters θ is the summation of the equation residual, the initial condition residual, and the cross-entropy loss. Therefore, the best parameters θ is obtained by minimizing the loss function:

where λ1, λ2, λ3 are hyperparameters.

4 Experiments

In this section, we first present the experimental setup. Then, we evaluate the proposed method on two different scenarios and analyze the experimental results.

4.1 Experimental Setup

We use PyTorch [Paszke et al., 2017] framework for the implementation. The torch version is 1.11.0+cu113. We conducted our experiments on the Ubuntu 20.04.6 LTS operator system. All the experiments are run on a single NVIDIA A100 40GB GPU. In our experiments, we set all the hyperparameters λi as 1.

4.2 Experiments on the Allen-Cahn Equation

Regarding the training configurations, we first run the Adam algorithm [Kingma and Ba, 2014] with 10,000 epochs and then employ the L-BFGS algorithm [Liu and Nocedal, 1989].

We show a numerical example in Figure 3 with p = 1, r = -1, q = 0.0001 and x ∈ [0, 2π], t ∈ [0, 4]. The initial condition is taken to u0(x) = 0.25sin(x) and we employ the Neumann boundary condition ux(0, t) = ux(2π, t). In this experiment, the Nf , Nb, Ni, Ne is set to 8,000, 1,000, 1,000, 2000. ∆x is L/M = 2π/2000 ≈ 0.00314. We employ an FCN with 6 hidden layers.

We compare the proposed method with the discrete variational derivative method (DVDM) [Furihata and Matsuo, 2010]. DVDM is a structure-preserving numerical method for PDEs, which improves the qualitative behavior of the PDE solutions and allows for stable computing. Although SP-PINN is slightly inferior to DVDM in terms of accuracy, DVDM needs an intricate derivation to construct a specific numerical scheme. Besides, Figure 5 illustrates that SP-PINN obtains numerical solutions significantly faster than DVDM.

Even accounting for the training time of the network, SPPINN is less time-consuming than DVDM.

As shown in Figure 4, the error between the results obtained by the proposed model and DVDM is smaller than that of the baseline model (i.e., the vanilla PINN) and DVDM, which indicates that the numerical solutions of the proposed model are more accurate than that of the baseline model.

4.3 Experiments on Image Recognition

The Experimental Configurations

We conduct a set of experiments on four datasets, MNIST [LeCun et al., 1998], Street View House Numbers (SVHN) [Yuval, 2011] and CIFAR10/100 [Krizhevsky et al., 2009].

We choose a 2-layer ICNN and set the activation function σ as a smooth ReLU function. For optimization, we use the Adam algorithm with the initial learning rate = 0.001 and a cosine annealing schedule. The training epochs for MNIST, SVHN, and CIFAR10/100 are set to 10, 40, and 60/70.

For MNIST, we downsample the input image from 28×28 to 6×6. For SVHN and CIFAR, we downsample the input image from 32×32 to 16×16. We choose an 18-layer residual network [He et al., 2016] as the backbone of the SP-PINN and utilize a Maclaurin series with 5th -order polynomial basis for modeling the underlying systems.

We test the performance of the PINN and SP-PINN on two white-box adversarial attacks: iterative fast gradient sign method (I-FGSM) [Kurakin et al., 2018] and project gradient descent (PGD) [Madry et al., 2017].

The Evaluation of the SP-PINN on Image Data

Table 1 presents the experimental results in terms of classification accuracy and robustness against adversarial examples. On clean data, SP-PINN performs slightly inferior to the baseline model. Figure 5 illustrates that the PINN and SP-PINN are significantly less time-consuming than Neural ODEs. In terms of robustness against adversarial examples, we evaluate the effectiveness of SP-PINN against white-box attacks. The experimental results demonstrate that the SPPINN outperforms the baseline model on all datasets.

Furthermore, we observe that the accuracy improvement under adversarial attacks increases as the attack radii increase for all datasets. For example, the SP-PINN under I-FGSM with attack radii ϵ = 2/255, ϵ = 4/255, ϵ = 6/255 and ϵ = 8/255 achieves a boost of 8.21%, 15.60%, 18.80%, 21.80% on CIFAR10, respectively.

SP-PINN With Adversarial Training

Our approach is independent of other adversarial defense methods, such as adversarial training. This means that we can combine SP-PINN with adversarial training techniques to further enhance defense performance. We consider three wellknown adversarial training methods, namely PAT [Madry et ResNet56

ResNet56+PAT

Neural ODEs

DEQ

PINN (baseline)

SP-PINN

SP-PINN+PAT

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Epsilon(from 2 to 20)/255

al., 2017], robust dataset (RD) [Ilyas et al., 2019], and interpolated adversarial training (IAT) [Lamb et al., 2022].

From Table 2, we observe that training SP-PINN with adversarial training methods further improves its robustness against adversarial examples.

Figure 6 provides a comparison between ResNet56 [He et al., 2016], ResNet56 with PAT, DEQ [Bai et al., 2019], Neural ODEs [Chen et al., 2018], PINN, SP-PINN, and SP-PINN with PAT under adversarial attacks ranging from ϵ = 2/255 to ϵ = 20/255. It is apparent that SP-PINN is insensitive to the radius of adversarial attack, which further corroborates the effectiveness of our method.

5 Conclusions

In this paper, we proposed SP-PINNs by introducing the prior knowledge about the properties of the underlying dynamical systems. The applicability of the proposed SP-PINNs ranges from PDEs, and to image recognition. In future work, we will consider employing the proposed method for solving more complex 'conservative' and 'dissipative' equations. Additionally, we consider applying SP-PINN to different natural language processing (NLP) tasks, given adversarial attacks have emerged in the NLP field. When varying the inputs by natural language, through concatenating the vector embeddings with time, SP-PINN can be expected to achieve character/word/sentence level adversarial defense.

Accuracy

0.8

0.6

0.4

0.2

0.0

Acknowledgments

This work was supported in part by the National Science and Technology Major Project (2018AAA0102100), the National Natural Science Foundation of China (No.62120106009), the China Scholarship Council (No.202207090082), JSPS KAKENHI (20H01822, 20H00581, 21K18301), and JST PRESTP (JPMJPR2129).