Distilling Governing Laws and Source Input for Dynamical Systems from Videos

1 Introduction

Discovery of governing equations (e.g., PDEs, ODEs) has the potential to advance our understanding, modeling and prediction of the behavior of complex dynamical systems. Increasing richness of collecting data and advances in machine learning gave rise to a new perspective of dynamical system modeling, e.g., data-driven discovery of governing equations recently [Brunton et al., 2016]. Advances in deep learning, especially convolutional neural networks (CNNs), have led to keen interest in uncovering physical law directly from videos that record dynamical processes [Chen et al., 2021]. In earlier deep learning approaches, explicit physical law was distilled from the motion trajectory which was extracted by supervised regression ahead [Belbute-Peres et al.,

∗Corresponding Author

2018]. Later, unsupervised moving object localization techniques promoted the parameter estimation of dynamical systems in an unsupervised scheme [Jaques et al., 2020]. While since the moving object is localized in image space, these approaches can only learn simple physical laws or their parameters which are modeled in pixel coordinate system such as billiards, gravity, spring, bouncing, etc. [Jaques et al., 2020; Kossen et al., 2020]. Furthermore, in most of existing methods, the form of physical engine is given and incorporated with motion representative extraction module, which restricts their application to real scenarios where the mathematical form/structure of physical law is unknown.

Study on data-driven governing equations discovery started with and, today, still mainly focus on building mathematical models from given measurement of physical states (e.g., trajectory time series) [Kutz et al., 2016; Brunton et al., 2016; Rudy et al., 2019; Udrescu and Tegmark, 2020]. Later, advances in deep learning led to expanded interests in physical law discovery from videos instead, a subset of physical scene understanding. Inspired by powerful feature extraction ability, deep neural network was initially leveraged to learn "blur" physics from videos, in which the physical laws are not expressed explicitly but simulated or condensed as physical modules [Fragkiadaki et al., 2016; Ehrhardt et al., 2017]. Later, the incorporation of physical engines/modules with extracted object-based representations improved the description or prediction of physical videos [Purushwalkam et al., 2019; Wu et al., 2017; Ye et al., 2018]. Although some of those approaches considered interpretable parameters in the physical modules like object mass, position, speed and friction [Wu et al., 2017; Ye et al., 2018], the physical laws are still not explicitly discovered and the physics are condensed as state space, or simulated as neural physics engines/modules.

In order to improve the interpretability of the discovered physical laws, learning explicit dynamics (e.g., closed-form governing equations or their parameters) has recently become more popular in physical scene understanding. Several hybrid methods take a data-driven approach to estimate real mechanical process from video sequences [Wu et al., 2015; Wu et al., 2016], or model Newtonian physics via latent variable to predict motion trajectories in images [Mottaghi et al., 2016]. Since the discovery of explicit physical laws requires extracting the motion of moving object, two-step staggered discovery has become the most common strategy, where the physical law is distilled after the moving trajectory being extracted [Wu et al., 2017; Belbute-Peres et al., 2018; Ehrhardt et al., 2018]. Advances in unsupervised object localization techniques like based on spatial transformers (ST) [Kosiorek et al., 2018; Hsieh et al., 2018; Ehrhardt et al., 2018] and Position-Velocity Encoders (PVEs) [Jonschkowski et al., 2017] have enabled the explicit physical law discovery in an unsupervised scheme [Kossen et al., 2020; Jaques et al., 2020]. However, these approaches require strong prior knowledge on the structure of the physical law or governing equation (e.g., the equation form is given while the coefficients need to be discovered). Furthermore, for those methods, physics is modeled in pixel coordinates which restricts the discovery for complex dynamical systems (e.g., ODEs) where the physical states need to be described in another physical coordinate system.

Attempts also have been made to uncover the closed-form governing equations in the context of low-dimensional representations from videos or high-dimensional data. One approach to discover the governing equations and the associated coordinate system from high-dimensional data is proposed in [Champion et al., 2019], while this approach was only able to handle the "video" which can be explicitly expressed as a function of physical latent and fails to deal with real raw videos. The unsupervised physical parameter estimation method proposed in [Jaques et al., 2020] can only discover from videos the parameters of given physical laws which are established in the context of pixel coordinates. In addition, a two-step deep learning based method was developed for closed-form governing equation discovery from distorted videos [Udrescu and Tegmark, 2021], while this method is not in an end-to-end scheme and cannot deal with systems excited by unknown input. Hence, uncovering governing equations straightly from raw videos still remains a grand challenge, especially when the source input is unknown.

In this work, we propose an end-to-end unsupervised deep learning framework to uncover from videos the closed-form governing equations of dynamical systems subjected to unknown input. The task we intend to resolve, as shown in Figure 1, demonstrates the paradigm we build seeking to simultaneously extract the physical states of moving object(s), uncover their governed closed-form equations, and identify the system input. Unlike the existing deep learning methods which typically discover physical laws from spatial/pixel coordinate trajectories of moving object(s), our method uncovers the explicit governing equations from the physical states in a regressed physical coordinate system instead, which makes it possible to discover more complex dynamical systems. Furthermore, we consider the form/structure of governing equation unknown a priori and employ sparse regression to distill its formulation. In addition, the physical states are extracted not independently from the encoder-decoder and physical coordinate system regression but under the constraint of underlying physical law. The joint optimization not only helps the extraction of physical states, but also leads to the identification of closed-form governing equations and unknown input, which forms our key contribution.

2 Methodology

In this section, we first introduce the designed network architecture for governing equations discovery from videos, which is assembled with three segments: (1) an encoder-decoder to condense the high-dimensional image into low dimensional latent space, (2) physical coordinate system regression to create the mapping between the spatial/pixel coordinates of moving object(s) and their physical states for underlying explicit physical law, and (3) physical embedding which serves as a constraint for the extraction of physical states, and, meanwhile, uncovers their governed equation and the unknown excitation. The definition of loss function and the network training strategy are also provided in the following.

2.1 Network Architecture

In order to learn the governing equations from videos, several components are considered in place. First of all, the highdimensional image is condensed into low-dimensional latent variables, the spatial (pixel) coordinates of moving object(s), by the encoder-decoder. A new physical coordinate system is regressed to map the spatial coordinates into physical sates where the underlying physical law is presented. Since the potential governing equations are assumed to be parsimonious and can be described with fewest terms necessary, we leverage a library, composed of a finite number of pre-defined possible candidate terms in the context of physical states, to regress the equations parameterized by their unknown coefficients. Then the physical law constraint is imposed to the physical states by the sparse regression and the physics forward with a 4th-order Runge-Kutta method. In addition, the unknown input can be identified by embedding it into the physical constraint. The designed network architecture is depicted in Figure 2. Figure 2(A) shows the high-level view of the network architecture which involves the forward propagation of physics by considering two consecutive video frames. All components making up the high-level network are discussed as follows and more information about the network can be found in Supplementary Information (SI) 1 Section 2.1.

Encoder: A two-stage localization network [Jaques et al., 2020] is employed as encoder to condense the image space into low-dimensional latent variables. It takes a single video frame It as input and outputs a vector xs = (xs1, ys1, xs2, ys2, ..., xsn, ysn) corresponding to the 2D spatial coordinates of n moving objects. First, the input frame is passed through a U-Net to produce unnormalized masks for moving object(s). The unnormalized masks are concatenated with a constant background mask mb and then passed through a Softmax to produce the masks for the moving object(s) and background. The multiplication between input and masks are fed into a fully-connected network to obtain the spatial coordinates of the moving object(s). The output layer of the encoder depends on the number of coordinates needed to describe the object location. For a 2D physical system with image size of H ×H, the latent space has two variables and the activation of output layer has a saturating non-linearity H/2 · tanh (·) +H/2, which leads to the spatial coordinate of the moving object xs with values in [0,H].

Coordinate-Consistent Decoder: The decoder takes the spatial coordinates of moving object(s) xs as input and outputs a reconstructed image. Instead of using conventional decoders, the decoder imposed by a fixed relationship between latent spatial coordinate and pixel-coordinate correspondence is employed to reconstruct the video frame. The CoordinateConsistent Decoder developed in [Jaques et al., 2020] is built on the spatial transformer (ST) network. If the input spatial coordinate for one object to decoder is xs = (xs, ys), the parameters ω of ST is to place the center of the writing attention window for moving object at this point. In this CoordinateConsistent Decoder, the moving object(s) and their masks, and the background are learned by independent networks separately. Then the video frames are reconstructed by the composition of background and the moving object(s) the positions of which are determined by the input spatial coordinates with STs. It should be highlighted that, the Coordinate-Consistent Decoder may not able to capture the real (or absolute) locations of moving object(s) in image space (e.g., xs,0 and ys,0

1 see https://github.com/LeleLuan/VideoDiscovery

in Figure 1) especially for the videos with moving object(s) showing up in limited positions. While the relative locations of the object(s) moving at different time steps can be captured (e.g., xs,1/xs,2 and ys,1/ys,2 in Figure 1), which plays a key role in the following physical trajectory extraction via Spatial-Physical Transformation.

Physical Coordinate System Regression: The Encoder and Coordinate-Consistent Decoder introduced above are expected to extract the spatial coordinates of all moving objects in the scene. An underlying physical coordinate system is then regressed to map the extracted spatial coordinates into physical states which present the potential governing equations. As shown in Figure 1, if we assume the dynamics is present in a 2D space which is parallel to the image space, the transformation between spatial coordinates and physical states can be implemented by a standard 2D Cartesian coordinate transformation. For one moving object, with translation vector ˜t = (˜ tx, ˜ ty)T and scaling factor ˜ s, the physical coordinate (state) xp = (xp, yp) can be obtained from spatial coordinate by xTp = T (xTs ) = ˜ s[xTs - ˜t]. Likewise, the physical to spatial transformation can be expressed as xTs = ˜ T (xTp ) = 1/˜ sxTp + ˜t. Here, T and ˜ T share the same parameters of ˜ s and ˜t in a complete encoder-decoder. It should be noted that different coordinate systems are regressed for different moving objects when the scene has multiple objects. The transformation factors are trainable variables and will be learned in the whole network training. As shown in Figure 2(A), with the Encoder φ, CoordinateConsistent Decoder ψ and coordinate transformers (T and ˜ T ), a complete autoencoder is built for the condensation of video frames into latent space, the physical states of moving object(s). The standard autoencoder loss function can be expressed as: Lrecon = ∥∥It - ψ( ˜ T (T (φ(It))))∥∥22 .

Physical LawEmbedding and Discovery: For the studied dynamical systems shown in Figure 1, the potential ODEs followed by moving object trajectories are distilled by sparse regression. When the dynamical system has one moving object with physical states xp(t) = (xp(t), yp(t))T , the considered dynamics can be expressed by d dt xp(t) = f(xp(t)) + g(t), where g(t), a trainable array, represents the unknown system input which is constant for one dynamical system with different initial conditions. We seek a parsimonious model for f(xp(t)), resulting in a function f that contains only a few active terms. If the derivatives of the physical states are calculated, the snapshots are stacked to form data matrices X = [xp,1, xp,2, . . . , xp,m]T ∈ Rm×2 and ˙X = [ ˙xp,1, ˙xp,2, . . . , ˙xp,m]T ∈ Rm×2 where m is the number of data points of the physical states. Although f is unknown, we can construct an extensive library of n candidate functions Θ(X) = [θ1(X), θ2(X), ..., θn(X)] ∈ Rm×n , where each θj denotes a candidate term. The candidate function library is used to formulate an over-determined system ˙X = Θ(X)Ξ, where the unknown matrix Ξ = [ξ1, ξ2, ..., ξn] ∈ Rn×2 is the set of coefficients that determine the active terms fromΘ(X) in the dynamics f. For the candidate function library, θi(X) can be any candidate function that may describe the system dynamics f(x(t)) such as θi(X) = X2 . By solving the op-

𝑔(𝑡)

Decoder

(A) High-level View

𝐈6

𝜑 Encoder

𝐱!

SPT

PST

PST

8𝐱& 𝑡

𝜓

8𝐱& 𝑡 +∆𝑡

𝜓

&𝐈6

&𝐈6<=6

(E) Sparse Representation of Physical Law

Time Points

=

𝑥 𝑦 𝑥 ! 𝑥𝑦 𝑦! 𝑦"

⋯

̇𝐗

Sparse Regression:

%𝚵 = arg min

𝚵 𝚯 𝐗 𝚵 +𝒈- ̇𝐗 !! s.t. 𝚵 ∈ 𝓢

𝑔( 𝑔)

𝒈

(B) Encoder 𝜑

Image Unnormalized Mask Concatenate U-Net and Softmax

𝐦𝒃

Constant Background Mask Flatten

⊗

Masked Image

⫶

⫶

⫶

Dense Network

𝐱!

(C) Spatial-Physical Transformation

̃𝑡)

𝑦&

𝑜

𝑦'

𝑜

̃𝑡(

𝑥'

𝑥&

𝐱& = [𝑥&, 𝑦&]

8𝐱& = [𝑥&, 𝑦&]

SPT ( ̃𝑠, ̃𝑡", ̃𝑡#, &𝜗)

PST ( ̃𝑠, ̃𝑡", ̃𝑡#, &𝜗)

(D) Decoder 𝜓

timization, the model of system dynamics can be identified d dt xp(t) ≈ Θ(xp(t))Ξ+g(t). The coefficients Ξ are learned concurrently with the neural network parameters as part of the training process. With the time derivative of physical states xp(t) being calculated by the central difference method, one can enforce accurate modeling of the dynamics by incorporating the following physical state derivative term into the loss function: L˙xp = ∥∥ d(T (φ(It)) dt -Θ((T (φ(It)))T )Ξ+g(t)∥∥22 .

As shown in Figure 2, the forward propagation of physics also brings an additional physical constraint which is imposed to the physical states. With the true vector field f(xp(t)) being estimated by Θ(xp(t))Ξ, integrating over a segment of time tj to tj+1 gives the integrated vector field, or flow map xp(j + 1)) = xp(j) + ∫ tj+1 tj (Θ(xp(τ))Ξ+g(τ))dτ. A 4th-order Runge-Kutta (RK4) numerical method is employed to simulate the dynamics 1-step forward as xp(j + 1) = RK4(xp(j), g(j), g(j + 1)). Then the forward video frame can be reconstructed via the decoder as ˆIj+1 = ψ( ˜ T (xp(j + 1))), which leads to the forward video frame reconstruction loss: Lint = ∥∥Ij+1 - ψ( ˜ T (xp(j + 1)))∥∥22 .

Spatial Coordinate Physical Coordinate

Coordinate Transformation

𝐱' = [𝑥', 𝑦']

𝚯 𝐗

̇𝑥 ̇𝑦

𝜉#

𝜉!

𝚵

+

Spatial Coordinates

Unknown input

Forward

𝑔 𝑡 +∆𝑡

𝑥7

𝑦7

Physical States

Spatial Coordinates

Decoder

2.2 Loss Functions and Network Training

In addition to three loss terms given above, an ℓ0.5 regularizer Lreg on the sparse regression coefficients Ξ is included, which promotes sparsity of the coefficients and therefore encourages a parsimonious model for the dynamics. The combination of 4 loss terms leads to the overall loss function:

(1)

where the hyperparameters λ1, λ2, λ3 determine the relative weighting of the 3 loss function terms. The detailed definition of loss functions is given in SI Section 2.2. The network is trained using a multi-step training strategy to succeed the discovery, which includes (i) pre-training, (ii) total loss training, (iii) sequential thresholding, and (iv) refinement training. In the designed network, the trainable variables include variables in encoder-decoder, transformation factors (˜ tx, ˜ ty )T and ˜ s in physical coordinate system regression, coefficients for candidate functions Ξ in sparse regression, and system input array g. Although some of the trainable variables are not independent, e.g., Ξ and g, the network is still able to achieve the discovery thanks to the constraint of physical law and the sparsity of governing equation. More information about the multi-step training strategy and the choice of hyperparameters are provided in SI Section 2.3 and SI Section 2.4.

3 Experiments

We demonstrate the efficacy of our proposed method on several dynamical systems shown in Figure 1 (including SMSD, SMTD, TMTD). The studied videos are generated by plotting the lumped mass(es) according to the simulated physical trajectories on different backgrounds. More details of the dataset generation are given in SI Section 1. Our method aims to uncover the closed-form governing equations for the moving object(s) and identify the unknown source input. It should be noted that, since the studied dynamical systems follow second-order ODEs, the discovery is conducted by transferring the underlying equations into first-order state-space models. More information about the discovery on state-space modeling is given in SI Section 2.5. In addition, the robustness of this method on discovery to noise, ablation studies, and baseline comparison are also discussed in this section.

3.1 Discovery Result

The discovery results for the studied dynamical systems are presented in Figure 3, where the physical trajectories, their governing equations, and the external excitation are uncovered. It shows that the governing equations especially their coefficients are identified exactly the same as the ground truth. It should be noted that, in the discovered ODEs, both candidate function terms Θ(X) and input g are unknown. The discovered equations are still satisfied although physical states xp and input g have scaling or drift discrepancy to the ground truth. While after scaling and translation, the physical states and inputs learned from the network match to the ground truth very well. The discovery result for the TMTD system also shows that our method is able to handle the scenarios when multiple moving objects show up in the scene.

Figure 3: Discovery results of the studied dynamical systems, (a)

SMSD, (b) SMTD and (c) TMTD. The ground truth physical trajec-

tories and inputs are presented by solid lines, while the learned are

presented by dash lines. The physical trajectories and inputs shown

are the results after scaling and translation. In the SMTD system, g1

and g2 denote the inputs applied to horizontal and vertical direction

of the mass. In the TMTD system, x1 and x2 denotes the physical

states of left and right moving object, respectively.

Because different moving objects may follow different physical laws in different physical coordinate systems, the network is equipped with multiple coordinate regression and sparse regression for both physical trajectory extraction and governing equation discovery. Furthermore, because the dynamics presented by those two moving objects are not independent, the physical states (x1, x2, y1 and y2) and their derivatives ( ˙x1, ˙x2, ˙y1 and ˙y2) are considered in building the sparse regression for their governing equation discovery. In addition to the studied linear dynamical system, our method was also tested on discovering the governing equation of a nonlinear dynamical system (see SI Section 4.4). The detailed information for each dynamical system discovery is given in SI Section 4.

3.2 Robustness to Noise

The robustness of our proposed method to noise is further tested by discovering the governing equation from videos with noise. Here, we take the SMTD system as the testing example where Gaussian noise with zero mean and variance 2×10-5 is added to all video frames. The discovery result is presented in Figure 4. It shows that, although the noise in the videos is against the assumption that both moving object(s) and background are constant for all images in the ST-based Coordinate-Consistent Decoder, our method still achieves the correct discovery. Due to the effect of noise, compared to the discovery from videos without noise, the identified system input is noisier. Nevertheless, the governing equations and the physical trajectory are still uncovered and extracted correctly.

3.3 Ablation Study

In the proposed method, the spatial coordinates of moving object(s) learned from Encoder and Coordinate-Consistent Decoder are mapped into physical states by a regressed physical coordinate system. Here, the ablation study is implemented by removing the Spatial-Physical Transformation and then imposing the physical law constraint to spatial coordinates directly to test the necessity of physical coordinate system regression. The SMSD system is taken as the example for this ablation study. In the network training, the physical states derivative loss L˙xp is quite large after pre-training and this loss term does not converge at all in optimizing the pretrained model with total loss function. This is because the underlying physical law is not presented in pixel coordinate, adding physical law constraint to spatial coordinate directly will lead to high physical derivative loss and this loss term

does not converge in the total loss optimization.

3.4 Baseline Comparison

We found that the literature remains scant in uncovering governing equations for dynamical systems with unknown input directly from videos. The most related work was presented in [Champion et al., 2019] where an autoencoder-based deep learning model was built to discover low-dimensional dynamics and their associated coordinates from high-dimensional data. As a baseline, this coordinate and governing equation discovery method must be adapted slightly to account for unknown source input. In this baseline, the SINDy module has the same candidate functions as the sparse regression used in our discovery. Besides, the original RGB videos are converted into gray scale to calculate the required derivatives. More details on this baseline are provided in SI Section 5. The SMSD system is taken as an example. The training process shows that the reconstruction loss and SINDy loss in ¨z ( ˙z for the first-order ODE discovery) can achieve small values (e.g., 2 × 10-4 and 4 × 10-7 respectively) while the SINDy loss in ¨x remains almost constant (e.g., 0.34). After sequential thresholding in the training process, the identified coefficient matrix as shown in Figure 5 has no sparsity. The failure of this baseline method is because the raw video images cannot be treated as an explicit function of some latent variables, the case for which this baseline method is only applicable.

In addition, we replace the coordinate-consistent encoderdecoder shown in Figure 2 by a conventional convolutional encoder-decoder and take the resulting method as another baseline. In this baseline, the video frame is condensed into low-dimensional latent variables by convolutional layers and the Spatial-Physical Transformation module is removed. The physical law constraint is still imposed to the latent variables. More information about this convolutional encoderdecoder baseline is given in SI Section 5. The identified coefficient matrix is shown in Figure 5. Once the model is trained, the autoencoder loss, physical states derivative loss and forward frame reconstruction loss become very small (e.g., 2.0 × 10-5 , 0.0020 and 2.0 × 10-5 ), which demonstrates that this network is able to distill a physical law from videos. However, since the extracted latent variables cannot correctly represent the location-based physical states, the method fails to uncover the underlying physical law. Furthermore, the fixed relationship between the physical states and real locations of the moving object(s) cannot be guaranteed by the conventional autoencoder.

4 Conclusions

This paper presents an end-to-end unsupervised deep learning scheme to uncover the explicit interpretable physical laws from raw videos that record moving object(s) representing dynamical systems excited by unknown input. Our approach leverages a Coordinate-Consistent Decoder to capture the relative location of object(s) moving at different snapshots in the image space. In the physical coordinate system regression, the underlying coordinate system(s) are learned to map the spatial/pixel coordinates of moving object(s) into physical states where the dynamics may be explicitly and parsimoniously expressed. The form of governing equations for the extracted physical states is modeled by parametric linear combination of candidate functions, leading to a sparse regression problem. The sparse regression not only enables the physical law to be embedded into the network, but also provides a physical constraint to the extraction of physical states from the autoencoder and coordinate system regression. The efficacy of our proposed method is validated by uncovering the governing equations of several dynamical systems with different numbers of object in the scene. The robustness of our proposed method to noise is also tested. Our work is the first attempt to discover physical laws from raw videos for dynamical systems with unknown input excitation. Our method also holds limitations, e.g., it cannot deal with non-stationary background, video with warp, and moving objects in a 3D space, which will be addressed in our future study.

Acknowledgments

The work is supported by the Beijing Outstanding Young Scientist Program (No. BJJWZYJH012019100020098) as well as the Intelligent Social Governance Platform, Major Innovation & Planning Interdisciplinary Platform for the "DoubleFirst Class" Initiative, at Renmin University of China. Y.L. is grateful for the support from the Fundamental Research Funds for the Central Universities.