Vision-based Discovery of Nonlinear Dynamics for 3D Moving Target

1 Introduction

Nonlinear dynamics is ubiquitous in nature. Data-driven discovery of underlying laws or equations that govern complex dynamics has drawn great attention in many science and engineering areas such as astrophysics, aerospace science, biomedicine, etc. Existing studies primarily focus on uncovering governing equations based on direct measurement of the system states, e.g., trajectory time series, [Bongard and Lipson, 2007; Schmidt and Lipson, 2009; Brunton et al., 2016; Rudy et al., 2017; Chen et al., 2021b; Sun et al., 2021]. Limited efforts have been placed on distilling governing laws of dynamics directly from videos for moving targets in a 3D

∗Corresponding author

space, which represents a novel and interdisciplinary research domain. This challenge calls for a solution of fusing various techniques, including computer stereo vision, visual object tracking, and symbolic discovery of equations.

We consider a moving object in a 3D space, recorded by a set of horizontally positioned, calibrated cameras at different locations. Discovery of the governing equations for the moving target first requires accurate estimation of its 3D trajectory directly from the videos, which can be realized based on computer stereo vision and object tracking techniques. Computer stereo vision, which aims to reconstruct 3D coordinates for depth estimation of a given target, has shown immense potential in the fields of robotics [Nalpantidis and Gasteratos, 2011; Li et al., 2021], autonomous driving [Ma et al., 2019; Peng et al., 2020], etc. Disparity estimation is a crucial step in stereo vision, as it computes the distance information of objects in a 3D space, thereby enabling accurate perception and understanding of the scene. Recent advances of deep learning has kindled successful techniques for visual object tracking e.g., DeepSORT [Wojke et al., 2017] and YOLO [Redmon et al., 2016]. The aforementioned techniques lay a critical foundation to accurately estimate the 3D trajectory of a moving target for distilling governing equations, simply based on videos recorded by multiple cameras in a complex scene.

We assume that the nonlinear dynamics of a moving target can be described by a set of ordinary differential equations, e.g., dy/dt = F(y), where F is a nonlinear function of the d-dimensional system state y = {y1(t), y2(t), . . . , yd(t)} ∈ Rd . The objective of equation discovery is to identify the closed form of F given observations of y. This could be achieved via symbolic regression [Bongard and Lipson, 2007; Schmidt and Lipson, 2009; Sahoo et al., 2018; Petersen et al., 2021; Mundhenk et al., 2021; Sun et al., 2023] or sparse regression [Brunton et al., 2016; Rudy et al., 2017; Rao et al., 2023]. When the data is noisy and sparse, differentiable models (e.g., neural networks (NN) [Chen et al., 2021b], cubic splines [Sun et al., 2021; Sun et al., 2022]) are employed to reconstruct the system states, thereby forming physics-informed learning for more robust discovery.

Recently, attempts have been made toward scene understanding and prediction grounding physical concepts [Jaques et al., 2020; Chen et al., 2021a]. Although a number of efforts have been placed on distilling the unknown governing laws of dynamics from videos for moving targets [Champion et al.,

2019; Udrescu and Tegmark, 2021; Luan et al., 2022], the system dynamics was assumed in plane (e.g., in a 2D space). To our knowledge, distilling governing equations for a moving object in a 3D space (e.g., d = 3) directly from raw videos remains scant in literature. To this end, we introduce a unified vision-based approach to automatically uncover governing equations of nonlinear dynamics for a moving target in a predefined reference coordinate system, based on raw video data recorded by a set of horizontally positioned, calibrated cameras at different locations.

Contributions. The proposed approach is composed of three key blocks: (1) a target tracking module based on YOLO-v8 that extracts plane pixel motions of the moving target in each video data; (2) a coordinate transformation model based on Rodrigues' rotation formula, which allows the conversion of pixel coordinates obtained through target tracking to 3D spatial/physical coordinates in a predefined reference coordinate system given the calibrated parameters of only one camera; and (3) a spline-enhanced library-based sparse regressor that uncovers a parsimonious form of the underlying governing equations for the nonlinear dynamics. Through the integration of these components, it becomes possible to extract spatiotemporal information of a moving target from 2D video data and subsequently uncover the underlying governing law of dynamics. This integrated framework excels in effectively addressing challenges associated with measurement noise and data gaps induced by imprecise target tracking. Results from extensive experiments demonstrate the efficacy of the proposed method. This endeavor offers a novel perspective for understanding the complex dynamics of moving targets in a 3D space.

2 RelatedWork

Computer stereo vision. Multi-view stereo aims to reconstruct a 3D model of the observed scene from images with different viewpoints [Sch¨onberger et al., 2016; Galliani et al., 2016], assuming the intrinsic and extrinsic camera parameters are known. Recently, deep learning has been employed to tackle this challenge, such as convolutional neural networks [Flynn et al., 2016; Huang et al., 2018] and adaptive modulation network with co-teaching strategy [Wang et al., 2021].

Target tracking. Methods for vision-based target tracking can be broadly categorized into two main classes: correlation filtering and deep learning. Compared to traditional algorithms, correlation filtering-based approaches offer faster target tracking [Mueller et al., 2017], while deep learningbased methods [Ciaparrone et al., 2020; Marvasti-Zadeh et al., 2021] provide higher precision.

Governing equation discovery. Data-driven discovery of governing equations can be realized through a number of symbolic/sparse regression techniques. The most popular symbolic regression methods include genetic programming [Koza, 1994; Bongard and Lipson, 2007; Schmidt and Lipson, 2009], symbolic neural networks [Sahoo et al., 2018], deep symbolic regression [Petersen et al., 2021; Mundhenk et al., 2021], and Monte Carlo tree search [Lu et al., 2021; Sun et al., 2023]. Sparse regression techniques such as SINDy [Brunton et al., 2016; Rudy et al., 2017;

Rao et al., 2023] leverage a predefined library that includes a limited number of candidate terms, which search for the underlying equations in a compact solution space.

Physics-informed learning. Physics-informed learning has been developed to deal with noisy and sparse data in the context of equation discovery. Specifically, differentiable models (e.g., NN [Raissi et al., 2019; Chen et al., 2021b], cubic splines [Sun et al., 2021; Sun et al., 2022]) are employed to reconstruct the system states and approximate the required derivative terms required to form the underlying black equations.

Vision-based discovery of dynamics. Very recently, attempts have been made to discover the governing of equations for moving objects directly from videos. These methods are generally based on autoencoders that extract the latent dynamics for equation discovery [Champion et al., 2019; Udrescu and Tegmark, 2021; Luan et al., 2022]. Other related works include the discovery of intrinsic dynamics [Floryan and Graham, 2022] or fundamental variables [Chen et al., 2022] based on high-dimensional data such as videos.

3 Methodology

We here elucidate the basic concept and approach of visionbased discovery of nonlinear dynamics for a moving target in a 3D space. Figure 1 shows the schematic architecture of our method. The target tracking module serves as the foundational stage, which extracts pixel-level motion information from the target's movements across consecutive frames in a video sequence. The coordinate transformation module utilizes Rodrigues' rotation formula with respect to a predefined reference coordinate origin, which lays the groundwork for the subsequent analysis of the object's dynamics. The final crucial component is the spline-enhanced library-based sparse regressor, essential for revealing the fundamental dynamics governing object motion.

3.1 Coordinates Transformation

In this paper, we employ three cameras with known and fixed positions oriented in different directions to independently capture the motion of an object (see Figure 1a). With the constraint of calibrating only one camera, our coordinate learning module becomes essential (e.g., the 3D trajectory of the target and other camera parameters can be simultaneously learned). In particular, it is tasked with learning the unknown parameters of the other two cameras, including scaling factors and the rotation angle on each camera plane. These parameters enable to reconstruct the motion trajectory of the object in the reference coordinate system. We leverage Rodrigues' rotation formula to compute vector rotations in three-dimensional space, which enables the derivation of the rotation matrix, describing the rotation operation from a given initial vector to a desired target vector. This formula finds extensive utility in computer graphics, computer vision, robotics, and 3D rigid body motion problems.

In a 3D space, a rotation matrix is used to represent the transformation of a rigid body around an axis. Let v0 represent the initial vector and v1 denote the target vector. We denote the rotation matrix as R. The relationship between the pre- and post-rotation vectors can be expressed as v1 = Rv0. The rotation angle, denoted as θ, can be calculated via cos θ = v0·v1 ∥v0∥∥v1∥ . The rotation axis is represented by the unit vector u = [ux, uy, uz], namely, u = v0×v1 ∥v0×v1∥ . Having defined the rotation angle θ and the unit vector k, we construct the rotation matrix R using Rodrigues' rotation formula:

where I is a 3 × 3 identity matrix, and U is a 3 × 3 skewsymmetric matrix representing the cross product of the rotation axis vector u expressed as

When projecting an object onto a plane, denoting the coordinates of the projection as xp = (xp, yp, zp)T . The procedure for projecting a 3D object onto a plane is elaborated in Appendix A. The object's projected shape on a plane is determined solely by plane's normal vector. Refer to Appendix B for a detailed proof, and Appendix C for calculation of the camera's offsets from a camera plane.

3.2 Cubic B-Splines

B-splines are differentiable, and constructed using piecewise polynomial functions called basis functions. When the measurement data is noisy and sparse, cubic B-splines could serve as a differentiable surrogate model to form robust physicsinformed learning for equation discovery [Sun et al., 2021]. We herein adopt this approach to tackle challenges associated with data noise and gaps induced by the imprecise target tracking for discovering laws of 3D dynamics. The i-th cubic B-spline basis function of degree k, written as Gi,k(u), can be defined recursively as:

where ui represents a knot that partitions the computational domain. By selecting appropriate control points and combinations of basis functions, cubic B-splines with C2 continuity can be customized to meet specific requirements. In general, a cubic B-spline curve of degree p defined by n + 1 control points P = {p0, p1, ..., pn} and a knot vector U = {u0, u1, ..., um} is given by: C(u) = ∑ni=0 Gi,3(u) · pi. To ensure the curve possesses continuous and smooth tangent directions at the starting and ending nodes, meeting the first derivative interpolation requirement, we use Clamped cubic B-spline curves for fitting.

3.3 Network Architecture

We utilized the YOLO-v8 for object tracking in the recorded videos (see Figure 1a). Regardless of whether the captured object has an irregular shape or is in a rotated state, we only need to capture their centroid positions and track them to obtain pixel data. Subsequently, leveraging Rodrigues' rotation formula and based on the calibrated camera, we derive the scaling and rotation factors of the other two cameras. These factors enable the conversion of the object trajectory's pixel coordinates into the world coordinates deducing the physical trajectory. For the trajectory varying with time in each dimension, we use the cubic B-splines to fit the trajectory and a library-based sparse regressor to uncover the underlying governing law of dynamics in the reference coordinate system. This approach is capable of dealing with data noise, multiple instances of data missing and gaps.

Learning 3D trajectory. In this work, we use a threecamera setup to capture and represent the object's 2D motion trajectory in each video scene, yielding the 2D coordinates denoted as (xrp, yrp). The rotation matrix R is decomposed to retain only the first two rows, denoted as R-, to suitably handle the projection onto the image planes. Under the condition of calibrating only one camera, we can reconstruct the coordinates of a moving object in the reference 3D coordinate system using three fixed cameras capturing an object's motion in a 3D space. The assumed given information includes normal vectors v1, v2, v3 of camera planes for all three cameras, the positions of the cameras, as well as a scaling factor s1 and rotation angles ϑ1 for one of the cameras. We define the scaling factor vector as s = {s1, s2, s3} and the rotation angle vector as ϑ = {ϑ1, ϑ2, ϑ3}. The loss function for reconstructing the 3D coordinates of the object in the reference coordinate system is given by

where

Here, s∗ = {s2, s3} and ϑ∗ = {ϑ2, ϑ3}. Note that

xci = (xci , yci )T represents the pixel coordinates, black ∆ci denotes deviation of object position from image coordinate origin. Nm the number of effectively recognized object coordinate points, T (ϑ) the transformation matrix induced by rotation angle ϑ expressed as T (ϑ) = [cosϑ sinϑ; - sinϑ cosϑ]. When using three cameras, the transformation between the object's coordinates xref in the reference coordinate system and the pixel coordinates xc in the camera setups

reads

Solving for parameter values (s∗ , ϑ∗ ) via optimization of Eq. (4), we can subsequently compute the reconstructed 3D physical coordinates via the calculation provided in Eq. (6).

Equation discovery. Given the potential challenges in target tracking, e.g., momentary target loss, noise, or occlusions, we leverage physics-informed spline learning to address these issues (see Figure 1b). In particular, cubic B-splines are employed to approximate the 3D trajectory. Given three sets of control points denoted as P = {p1, p2, p3} ∈ Rr×3 . Given that the coordinate system is arbitrarily defined, and to enhance the fitting of data Dr, we introduce the learnable adaptive offset parameter ∆ = {∆1, ∆2, ∆3}. The 3D parametric curves where x(t; P, ∆) are defined by the control point vectors P, the cubic B-spline basis functions G(t) and the offset parameter ∆, namely, x(t;P,∆) = G(t)P+∆. Since the basis functions consist of differentiable polynomials, the expression of its differential equation is given by ˙x(P) = ˙GP.

Generally, the dynamics is governed by a limited number of significant terms, which can be selected from a library of l candidate functions, v.i.z., ϕ(x) ∈ R1×l [Brunton et al., 2016]. The governing equations can be written as:

where ϕ(P, ∆) = ϕ(x(t; P, ∆)), and Λ = {λ1, λ2, λ3} ∈ S ⊂ Rl×3 is the sparse coefficient matrix belonging to a constraint subset S (only the terms active in ϕ are non-zero).

Accordingly, the task of equation discovery can be formulated as follows: when provided with reconstructed 3D trajectory data Dr = {xm1 , xm2 , xm3 } ∈ RNm×3 . In other words, Dr is presented as effectively tracking the object movements in a video and subsequently transforming them into a 3D trajectory, where Nm is the number of data points. Our goal is to identify the suitable set of P and a sparse Λ that fits the trajectory data meanwhile satisfying Eq. (7). Considering that the reconstructed trajectory Dr might exhibit noise or temporal discontinuity, we use collocation points denoted as Dc = {t0, t1, . . . , tnc-1} to compensate data imperfection, where Dc denotes the randomly sampled set of Nc number of collocation points (Nc ≫ Nm). These points are strategically employed to reinforce the fulfillment of physics constraints at all time instances (see Figure 1b).

3.4 Network Training

The loss function for this network comprises three main components, namely, the data component Ld, the physics component Lp, and the sparsity regularizer, given by:

where

Here, Gm denotes the spline basis matrix evaluated at the measured time instances, xmi the coordinates in each dimension after 3D reconstruction in the reference coordinate system (may be sparse or exhibit data gaps whereas ˙Gc the derivative of the spline basis matrix evaluated at the collocation instances. The termGmpi is employed to fit the measured trajectory in each dimension, while ˙Gcpi is used to reconstruct the potential equations evaluated at the collocation instances. Additionally, Φ ∈ RNc×l represents the collocation library matrix encompassing the collection of candidate terms, ∥Λ∥0 the sparsity regularizer, α and β the relative weighting parameters.

Since the regularizer ∥Λ∥0 leads to an NP-hard optimization issue, we apply an Alternate Direction Optimization (ADO) strategy (see Appendix D) to optimize the loss function [Chen et al., 2021b; Sun et al., 2021]. The interplay of spline interpolation and sparse equations yields subsequent effects: the spline interpolation ensures accurate modeling of the system's response, its derivatives, and the candidate function terms, thereby laying the foundation for constructing the governing equations. Simultaneously, the equations represented in a sparse manner synergistically constrain spline interpolation and facilitate the projection of accurate candidate functions. Ultimately, this transforms the extraction of a 3D trajectory of an object from video into a closed-form differential equation.˙x = ϕ(x - ∆∗ )Λ∗ . (10)

After applying ADO to execute our model, resulting in the optimal control point matrix P∗ , sparse matrix Λ∗ , and adaptive parameter ∆∗ , an affine transformation is necessary to eliminate ∆∗ in the identified equations. We replace x with x - ∆∗ , as shown in Eq. (10), to obtain the final form of equations. We then assign a small value to prune equation coefficients, yielding the discovered governing equations in a predefined 3D coordinate system.

4 Experiments

In this section, we evaluate our method for uncovering 3D governing equations of a moving target automatically from videos using nine datasets 1 . The nonlinear dynamical equations for these chaotic systems and their respective trajectories can be found in Appendix E (see Figure S1). We generate 3D trajectories based on the governing equations of the dataset and subsequently produce corresponding video data captured from various positions. Our analysis encompasses the method's robustness across distinct video backgrounds, varying shapes of moving objects, object rotations, levels of data noise, and occlusion scenarios. We further validate the identified equations demonstrating their interpretability and generalizability. The proposed computational framework is implemented in PyTorch. All simulations in this study are conducted on an Intel Core i9-13900 CPU workstation with an NVIDIA GeForce RTX 4090 GPU.

Data generation. The videos in this study are synthetically generated using MATLAB to simulate real dynamic systems captured by cameras. To commence, the dynamic system is pre-defined, and its trajectory is simulated utilizing the 4th-order Runge-Kutta method in MATLAB. Leveraging the generated 3D trajectory, a camera's orientation is established within a manually defined 3D coordinate system to simulate the 2D projection of the object onto the camera plane. The original colored images featuring the moving object are confined to dimensions of 512 × 512 pixels at 25 frames per second (fps). Various shapes are employed as target markers in the video along with local dynamics (e.g., with self-rotation) to emulate the motion of the object (see Appendix F). Subsequently, a set of background images are randomly selected to mimic the real-world video scenarios. The resultant videos generated within the background imagery comprise color content, with each frame containing RGB channels (e.g., see Appendix Figure S2). After obtaining the video data, it becomes imperative to perform object recognition and tracking on the observed entities based on the YOLO-v8 method.

1 The datasets are derived from instances introduced in [Gilpin, 2021], where we utilize the following examples: Lorenz, SprottE, RayleighBenard, SprottF, NoseHoover, Tsucs2 and WangSun.

4.1 Results

Evaluation metrics. We employ both qualitative and quantitative metrics to assess the performance of our method. Our goal is to identify all equation terms as accurately as possible while eliminating irrelevant terms (False Positives) to the greatest extent. The error ℓ2, represented as ||Λid -Λtrue||2/||Λtrue||2, quantifies the relative difference between the identified coefficients Λid and the ground truth Λtrue. To avoid the overshadowing of smaller coefficients when there is a significant disparity in their magnitudes, we introduce a non-dimensional measure to obtain a more comprehensive evaluation.

The discovery of governing equations can be framed as a binary classification task [Rao et al., 2022], determining whether a particular term exists or not, given a candidate library. Hence, we introduce precision and recall as metrics for evaluation, which quantify the proportion of correctly identified coefficients among the actual coefficients, expressed as: P = ∥Λid ⊙Λtrue ∥0 /∥Λid ∥0 and R = ∥Λid ⊙Λtrue ∥0 /∥Λtrue ∥0 , where ⊙ denotes element-wise product. Successful identification is achieved when both the entries in the identified and true vectors are non-zero.

Discovery results. Based on our evaluation metrics (e.g., the ℓ2 error, the number of correct and incorrect equations terms found, precision, and recall), a detailed analysis of the experimental results obtained by our method is found in Table S3 (without data noise). After reconstructing the 3D trajectories in the world coordinate system, we also compare our approach with PySINDy [Brunton et al., 2016] as the baseline model. The library of candidate functions includes combinations of system states with polynomials up to the third order. The listed results are averaged over five trials. It demonstrates that our method outperforms PySINDy on each dataset in the pre-defined coordinate system. The explicit forms of the discovered governing equations for 3D moving objects obtained using our approach can be further found in Appendix G (e.g., Table S2). It is evident from Appendix Table S2 that the discovered equations by our method align better with the ground truth. We also reconstructed the motion trajectories in a 3D space using our discovered equations compared with the actual trajectories under the same coordinate system, as shown in Figure 2. These two trajectories nearly coincide, demonstrating the feasibility of our method.

It is noted that we also tested the variations and rotations of the moving object shapes in the recorded videos (e.g., see Appendix Figure S2) and found that they have little impact on the performance of our algorithm, primarily affecting the tracking efficiency. In fact, encountering noise and situations where moving objects are occluded during the measurement process can significantly impact our experimental results. To assess the robustness of our algorithm, we selected the SprottF instance for in-depth analysis and conducted experiments under various noise levels and different data occlusion scenarios. The experimental results are detailed in Figure 3. It is seen that our approach is robust against data noise and missing, discussed in detail as follows.

Noise effect. The Gaussian noise with zero mean and unit variance at a given level (e.g., 0%, 5%, ..., 30%) is added to the generated video data. To address the issue of small coefficients being overshadowed due to significant magnitude differences, we use two evaluation metrics in a standardized coordinate system: the ℓ2 error and the count of incorrectly identified equation coefficients. Figure 3a showcases our method's performance across various noise levels. We observe that up to a 20% noise interference, our method almost accurately identifies all correct coefficients of the governing equation. However, beyond a 30% noise level, our method's performance begins to decline.

Random block missing data effect. To evaluate our algorithm's robustness in the presence of missing data, we consider two missing scenarios (e.g., the target is blocked in the video scene), namely, random block missing and fiber missing (see Appendix Figure S3 for example). Firstly, we randomly introduce non-overlapping occlusion blocking on the target in the video during the observation period. Each block covers 1% of the total time periods. We validate our method's performance as the number of occlusion blocks increases. The "random block rate" represents the overall occlusion time as a percentage of the total observation time. We showcase our algorithm's robustness by introducing occlusion blocks that temporarily obscure the moving object, rendering it unidentifiable (see Figure 3b). These nonoverlapping occlusion blocks progressively increase in number, simulating higher occlusion rates. Remarkably, our algorithm remains highly robust even with up to 50% data loss due to occlusion.

Fiber missing data effect. Additionally, we conducted tests for scenarios involving continuous missing data (defined as fiber missing). By introducing 5 non-overlapping occlusion blocks randomly throughout the observation period, we varied the occlusion duration of each block, quantified by the "fiber missing rate" - the ratio of continuous missing data to the overall data volume. In Figure 3c, we explore the impact of increasing occlusion duration per block while maintaining a constant number of randomly selected occlusion blocks. All results are averaged over five trials. Our model demonstrates strong stability even when the fiber missing rate is about 20%.

Simulating real-world scenario. Furthermore, we generated a synthetic video dataset simulating real-world scenarios. Here, we modeled the observed object as an irregular shape undergoing random self-rotational motion and size variations, as shown in Figure 4a. Note that the size variations simulate changes in the camera's focal length when capturing the moving object in depth. The video frames were perturbed with a zero mean Gaussian noise (variance = 0.01). Moreover, a tree-like obstruction was introduced to further simulate the real-world complexity (e.g., the object might be obscured during motion) as depicted in Figure 4b. Despite these challenges, our method can discover the governing equations of the moving object in the reference coordinate system, showing its potential in practical applications. Please refer to Appendix I for more details.

Overall, our algorithm proves robust in scenarios with unexpected data noise, multiple instances of data loss, and continuous data gaps, for uncovering governing laws of dynamics for a moving object in a 3D space based on raw videos.

4.2 Ablation Study

We performed an ablation study to validate whether the physics component in the spline-enhanced library-based sparse regressor module is effective. Hence, we introduced an ablated model named Model-A (e.g., fully decoupled "spline + SINDy" approach). We first employed the cubic splines to interpolate the 3D trajectory in each dimension and then computed the time derivatives of the fitted trajectory points based on spline differentiation. These trajectories and the estimated derivatives are then fed into the SINDy model for equation discovery. Taking the instance of SprootF as an example, we show in Table ?? the performance of the ablated model under varying noise levels, random block rates, and fiber missing rates. It is observed that the performance of the ablated model deteriorates in all considered cases. Hence, we can ascertain that the physics-informed spline learning in the library-based sparse regressor module plays a crucial role in equation discovery under imperfect data conditions.

4.3 Discussion and Limitations

The above results show that our approach can effectively uncover the governing equations of a moving target in a 3D space directly from a set of recorded videos. The false positives of identification, when in the presence (e.g., see Appendix Table S2), are all small constants. We consider these errors to be within a reasonable range. This is because the camera pixels can only take approximate integer values, and factors such as the size of pixels captured by the camera and the number of cameras capturing the moving object can affect the reconstruction of the 3D coordinates in the reference coordinate system. The experimental results can be further improved when high-resolution videos are recorded and more cameras are used. There is an affine transformation relationship between the artificially set reference coordinate system and the actual one. Potential errors in learning such a relationship also lead to false positives in equation discovery.

Despite efficacy, our model has some limitations. The library-based sparse regression technique encounters a bottleneck when identifying very complex equations black (e.g., power or division terms) when the a priori knowledge of the candidate terms is deficient. We plan to integrate symbolic regression techniques to tackle this challenge. Furthermore, the present study only focuses on discovering the 3D dynamics of a single moving target in a video scene. In the future, we will try discovering dynamics for multiple moving objects (inter-coupled or independent).

5 Conclusion

We proposed a vision-based method to distill the governing equations for nonlinear dynamics of a moving object in a 3D space, solely from video data captured by a set of three cameras. By leveraging geometric transformations in a 3D space, combined with Rodrigues' rotation formula and computer vision techniques to track the object's motion, we can learn and reconstruct the 3D coordinates of the moving object in a user-defined coordinate system with the calibration of only one camera. Building upon this, we introduced an adaptive spline learning framework integrated with a library-based sparse regressor to identify the underlying law of dynamics. This framework can effectively handle challenges posed by partially missing and noisy data, successfully uncovering the governing equations of the moving target in a predefined reference coordinate system. The efficacy of this method has been validated on synthetic videos that record the behavior of different nonlinear dynamic systems. This approach offers a novel perspective for understanding the complex dynamics of moving objects in a 3D space. We will test it on real-world recorded videos in our future study.

Acknowledgments

The work is supported by the National Natural Science Foundation of China (No. 92270118) and the Strategic Priority Research Program of the Chinese Academy of Sciences (No. XDB0620103). The authors would like to acknowledge the support from the Fundamental Research Funds for the Central Universities (No. 202230265 and No. E2EG2202X2). The Appendix of this manuscript is found in the preprint version: https://arxiv.org/pdf/2404.17865.