An information-based model selection criterion for data-driven model discovery

Abstract

Data-driven model discovery (DDMD) algorithms are powerful tools for extracting interpretable symbolic models from data. However, identifying the model that best balances goodness-of-fit and sparsity is often a laborious process requiring user fine-tuning, is prone to overfitting, and results may significantly vary depending on model initialization and specific training procedure. Here, we present a sparse regression algorithm that automatically and adaptively generates candidate models, and uses a novel sample-length-scaling logarithmic information criterion (SLIC) to identify the best model from these candidates. We demonstrate that SLIC greatly outperforms other popular information criteria in extracting the correct model from the data of several nonlinear ordinary and partial differential equations. Then, we demonstrate SLIC’s ability to discover interpretable models from experimental datasets in fluid dynamics and nanotechnology that generate new testable predictions.

1. Introduction

Models, which we loosely define as mathematical expressions that underpin relationships within data, lie at the heart of all science and engineering disciplines. Unfortunately, manually constructing a model for a system requires extensive mathematical and domain expertise. With the increasing capacity to collect large datasets and ever-growing computing power, we have seen the emergence of data-driven model discovery (DDMD) algorithms to address this challenge. This paradigm shift departs from traditional manual construction of models and aims at extracting accurate, interpretable models of complex systems directly from data. These techniques are numerous and include dynamic mode decomposition and Koopman methods Schmid, Budišić et al., Wilson, Williams et al., symbolic regression Cranmer, Reinbold et al., Udrescu and Tegmark, and regression-based library/dictionary methods Brunton et al., Brunton and Kutz, Hastie et al.. However, discovering the best overall model that is both accurate and sparse (i.e. possessing the minimal number of mathematical terms possible) remains a central challenge in DDMD.

The major challenge in DDMD entails finding the optimally sparse model within a given model architecture. This requires both (i) determining the optimal sparsity-enforcing parameters and (ii) robust model selection. The optimal sparsity-enforcing parameters, which in this work are pruning thresholds that remove unnecessary terms from a model, are often unknown a priori. The choice of such parameters is a delicate problem: if sparsity-enforcement is too lax, the model would be grossly overfit with unnecessary terms; if too aggressive, the model would not capture the full scope of the observed phenomena. Thus, setting the optimal sparsity-enforcing thresholds often requires extensive user-specified input and fine-tuning which cannot generalize easily per its case-by-case nature or tackle real-time applications due to laborious and time-consuming user interventions. As for the second requirement, though much effort has been directed at refining the procedure by which models are estimated from data Champion et al., Messenger and Bortz [a,b], Egan et al., Goyal and Benner, Zhang and Lin, Hirsh et al., Course and Nair, fundamental research on designing novel model selection criteria suitable for DDMD is lacking. Specifically, many widely used model selection criteria prioritize goodness-of-fit over sparsity, resulting in overfit models that cannot generalize or correctly predict the behavior of the system beyond training data. This tendency to overfit is due to the functional forms of these criteria, which, as a result of differences in scaling with data quantity, weigh model accuracy disproportionately more than model sparsity.

Here, to address this challenge, we present a sparse regression-based algorithm that adaptively auto-generates sparsity-enforcing parameters directly from the data, thus circumventing the need for user fine-tuning, and utilizes a novel sample-length-scaling logarithmic information criterion (SLIC) designed to robustly and automatically identify the model that best achieves both goodness-of-fit and model simplicity. The auto-thresholding and SLIC scoring work efficiently towards discovering the best overall model (Fig. 1).

Figure 1:

(a) The overview of the SLIC algorithm. We want to extract models (governing equations, etc.) from data, which can originate from diverse sources. To do so, we use our iterative model generation-model scoring loop to optimally prune input models, which are then updated. The output model can be used for myriad purposes, including scientific discovery, forecasting, analysis, and control. (b) Overview of the central iterative component of the algorithm: adaptive, automatic extraction of sparsity-enforcing parameters from input data are used to generate candidate models (“Model Generation”); and our sample-length-scaling logarithmic information criterion (SLIC) designed to avoid overfitting when selecting best models from candidate models (“Model Scoring”). The resulting best model determined by SLIC is then used to regenerate sparsity-enforcing parameters on the next iteration (see text and Methods for details). (c) Benefits of SLIC scoring. Left: SLIC equitably weights fit and sparsity. Center: The y-axis is the fit-to-sparsity ratio, which serves as a proxy for quantifying overfitting. Right: SLIC is equivalent to minimizing model variance.

We first present a concise problem statement, a brief motivation for SLIC and its derivation, and the SLIC-sparse regression algorithm. Then, we demonstrate that SLIC can robustly and automatically extract the governing equations of numerous simulated ordinary and partial differential equations. Finally, we demonstrate the utility of SLIC to extract mathematical models from real experimental datasets from fluid dynamics and nanoscience, which give accurate predictions on unseen experimental conditions.

2. Problem statement, our information criterion, and the SLIC-sparse regression algorithm

2.1. Problem Statement

Here, we provide a simple outline of the problem at hand. Our data consists of a pair of matrix inputs, $\text{X}=\left[\begin{array}{lll}{\text{x}}_1\hfill & \dots \hfill & {\text{x}}_m\hfill \end{array}\right]\in {ℝ}^{n\times m}$, and corresponding outputs, $\text{Y}=\left[\begin{array}{ll}{\text{y}}_1\hfill & \dots {\text{y}}_p\hfill \end{array}\right]{\text{R}}^{n\times p}$, connected by an unknown mathematical relationship. Here, $\text{n}$ denotes the number of observations, $\text{m}$ denotes the number of state variables (features), and $\text{p}$ the number of outputs (targets). For example, for an ODE describing a dynamical system, each column of $\text{X}$ contains a measured state variable over its time course and each column of $\text{Y}$ contains the corresponding numerically estimated time derivatives of the state variables. For time-dependent PDEs, $\text{n}$ is simply the total number of spatiotemporal observations, which scales multiplicatively with the number of dimensions.

Here, we assume that the mathematical relationship can be modeled using a nonlinear regression framework Brunton et al., Hastie et al.: $\text{Y}=\Theta (\text{X})\text{Ξ}+\text{η}$, where $\text{Θ}(\text{X})\in {\text{R}}^{n\times l}$ is a user-specified matrix of nonlinear transformations of the inputs, $\text{Ξ}\in {\text{R}}^{l\times p}$ is a matrix of unknown model coefficients, and $\text{η}$ is zero-mean, isotropic Gaussian noise with unknown noise strength; $\text{l}$ is the number of library terms, which may include polynomial nonlinearities, trigonometric terms, etc.

To find $\text{Ξ}$, we solve the following optimization problem:

$$\widehat{\text{Ξ}}=\arg \underset{\equiv }{\min }\Vert \text{Y}-\text{Θ}(\text{X})\text{Ξ}{\Vert }_2^2+{\lambda }^2\text{R}(\text{Ξ})$$ (1)

Where $\Vert \cdot {\Vert }_2^2$ is the sum of square errors, $\text{R( )}$ is a regularization term, and $\lambda$ is the concomitant regularization coefficient. In this work, we take $\text{R(⋅)}$ to be the ${\text{L}}_0$ norm, which penalizes the number of nonzero coefficients in the number. In this case, $\lambda$ manifests itself as a thresholding parameter that effectively prunes terms in $\text{Ξ}$ whose magnitude falls beneath $\lambda$. It is thus imperative to choose $\lambda$ wisely. Our algorithm for solving this problem without having to specify $\lambda$ is detailed in Section 2.3.

An important special case considered in a majority of this study is when we want to extract governing equations from timeseries data, i.e. $\text{Y}=\stackrel{\cdot }{\text{X}}$. To avoid taking numerical derivatives, we cast the problem into its weak form, as detailed in Materials and Methods 5.45.4.2.

2.2. An information-based model selection criterion for model discovery

Many principled methods exist for model selection Ding et al. [a]. Ideal model selection will achieve both simplicity and accuracy in the returned model. One important, widely used class of model selection techniques are known as information-based model selection criteria, which aim to minimize metrics related to the information content of probability distributions, such as the Kullback-Leibler divergence (KLD) Stoica and Selen.

Generally, many information criteria (denoted as IC below) take the form Stoica and Selen:

$$\text{IC}\approx -2\log \text{P}(\text{D}\mid \widehat{\text{θ}})+\text{s}(\text{n},\text{k})$$ (2)

Where $\text{n}$ is the number of data points, $\text{k}$ is the model complexity (i.e. the number of parameters), $\text{P}(\text{D}\mid \widehat{\text{θ}})$ is the likelihood evaluated at the most likely parameters (denoted as $\widehat{\text{θ}}$) for a given model, and $\mathrm{s}(\text{n},\text{k})$ is a function that penalizes model complexity. Under the further assumption that this likelihood is univariate Gaussian, we get the following form of these scoring criteria Stoica and Selen:

$$\text{IC}\left(\widehat{ϵ},\text{k}\right)=\text{n}\log \left(\widehat{ϵ}\right)+\text{s}(\text{n},\text{k})$$ (3)

Where $\widehat{ϵ}$ is the mean-squared error of the model prediction, which is also the maximum likelihood estimation of the unknown noise strength.

Different model selection criteria differ primarily in their choice of $\mathrm{s}(\text{n},\text{k})$. For example, for the Akaike Information Criterion (AIC) Akaike, ${\text{s}}_{AIC}(\text{n},\text{k})=2\text{k}$; for the Bayesian Information Criterion (BIC) Schwarz, ${\text{s}}_{BIC}(\text{n},\text{k})=\text{k}\log (\text{n})$. The model with the minimum information criterion score is chosen and presumed to be the model that best balances goodness-of-fit with the model complexity. Thus, depending on the functional form of $\mathrm{s}(\text{n},\text{k})$ different information criteria may select different models.

Interestingly, a common feature of these information criteria taking the form (3) is that as n grows large, the sparsity-enforcing component becomes negligible in comparison to the goodness-of-fit term that scales linearly with n (i.e. ${\lim }_{n\to \infty }\text{n}\log \left(\widehat{ϵ}\right)/\text{s(n,k)}\to \infty$). Although model complexity may increase with the amount of collected data in certain applications, it is clearly not the case when the underlying model should be invariant to the amount of collected data. To combat such issues, users are required to resort to more complex procedures such as fine-tuning of sparsity-enforcing parameters, data subsampling, etc.

With this in mind, we sought an improved $\mathrm{s}(\text{n},\text{k})$, and hence an information-based scoring criterion, suitable for the task of model selection in DDMD. Here we give an argument for such a criterion based on scaling and symmetry principles.

First, given the supposed invariance of the correct model with respect to collected data, one may suppose that $\mathrm{s}(\text{n},\text{k})$ should scale asymptotically with n, i.e. $\mathrm{s}(\text{n},\text{k})\sim \text{n}$. Otherwise, we would run into the same scaling issues facing other information criteria. Using this assumption, without loss of generality, the simplest functional form would be $\text{s}(\text{n},\text{k})=\text{n}\log (\text{f}(\text{k}))$, where $\text{f}(\text{k})$ is a pure function of the number of free parameters. In turn, this implies $\text{IC}\left(\widehat{ϵ},\text{k}\right)=\text{n}\log \left(\widehat{ϵ}\text{f}(\text{k})\right)$.

Next, we demand equitable penalization of goodness-of-fit and complexity. One such way to enforce this is to demand that the information criterion be symmetric in its arguments: $\text{IC}\left(\widehat{ϵ},\text{k}\right)=\text{IC}\left(\text{k},\widehat{ϵ}\right)$. This implies:

$$\widehat{ϵ}\text{f}(\text{k})=\text{kf}(\widehat{ϵ})$$ (4)

$$\frac{\text{f}(\widehat{ϵ})}{\widehat{ϵ}}=\frac{\text{f}(k)}{k}$$ (5)

Assuming the approximate independence of $\widehat{ϵ}$ and k, the only way to satisfy this functional equation is for the ratio to be a constant, $\text{f}(\text{k})\propto \text{k}$. We drop the superfluous proportionality factor, as only differences in these scores matter.

As this information criterion scales with sample length and possesses a logarithmic form in k, we dub it SLIC, for sample-length-scaling logarithmic information criterion:

$$\text{SLIC}\left(\widehat{ϵ},\text{k}\right)=\text{n}\log \left(\widehat{ϵ}\text{k}\right)$$ (6)

Interestingly, in the sparse regression framework, where we consider our outputs to be linear combinations of nonlinear transformations of our input data, choosing a model that minimizes this information criterion score is intimately tied to choosing a model that minimizes model variance in the famous bias-variance tradeoff Hastie et al.:

$$\text{Model}\text{Variance}=\frac{\widehat{ϵ}\text{k}}{\text{n}}$$ (7)

Minimizing the SLIC score in (6) thus minimizes the model variance, at least for a single target variable. In this sense, models selected by SLIC will be least likely to overfit and most likely to generalize beyond the training data (See Supporting Information for a derivation of (7)). For targets with multiple variables, SLIC minimization approximates minimizing total variance if deviations in total error are negligible, .

Given its favorable scaling with data quantity, SLIC circumvents the paradoxical issue of the increasing likelihood of overfitting with data quantity that is manifest in information criteria such as AIC and BIC. Thus, by design, SLIC equitably weights accuracy and sparsity, is robust to data quantity, minimizes user intervention, and, due to its relation with variance minimization, extracts generalizable models (Fig. 1c).

2.3. The SLIC-sparse regression algorithm

Here, in Algorithm 1, we give a brief description of the central component of the automatic pruning algorithm using SLIC, depicted in Fig. 1. Below, X,Y represent the input and output data, respectively; $\text{Ξ}$ is the matrix of unknown model terms. The SLIC score of the model is $\text{SLIC}(\text{Ξ})=\text{n}\log \left(\widehat{ϵ}\text{k}\right)$, where n is the number of observations, $\widehat{ϵ}$ is the mean-squared error between the model prediction and data, and k is the model complexity, or the number of nonzero parameter, which includes the unknown noise strength.

First, we estimate the nonzero parameters by minimizing $\Vert \text{Y}-\text{Θ}(\text{X})\text{Ξ}{\Vert }_2^2$, which is easily solved via taking the pseudoinverse of $\text{Θ}(\text{X})$, $\widehat{\text{Ξ}}=\text{Θ}{(\text{X})}^{†}\text{Y}$. This model is then scored using SLIC. Then, an array of candidate thresholds, $\lambda \text{s}$, is generated from the estimate of $\text{Ξ}$ by simply using the absolute value of the elements of $\text{Ξ}$ itself. Then, for each $\lambda$ in the array $\lambda \text{s}$, the parameters of $\text{Ξ}$ whose magnitudes fall beneath $\lambda$ are pruned. The nonzero parameters of the pruned model are again updated by pseudoinverse. If the SLIC score of this pruned model is smaller than the current best model (according to smallest SLIC score), then this becomes the new best model. This process is continued for ten iterations. The model with the best SLIC score is returned as output.

Algorithm 1.

Auto-pruning SLIC Algorithm

Inputs: $\text{Θ}(\text{X})$, Y
Outputs: $\text{Ξ}$
$\text{Ξ}\leftarrow \text{Θ}{(\text{X})}^{†}Y$	▷ Get initial model estimate via pseudoinverse
BestScore ←SLIC $\left(\text{Ξ}\right)$	▷ score current best model
$\lambda \text{s}$ ← nonzero elements of |$\text{Ξ}$|	▷ pruning parameters
for $\lambda$ in $\lambda$s do
${\text{Ξ}}_{new}$ ← copy($\text{Ξ}$)
inds ← |${\text{Ξ}}_{new}<\lambda$|	▷ get ’small’ terms
0 ← ${\text{Ξ}}_{new}$ (inds)	▷ prune ’small’ terms
${\text{Ξ}}_{new}$ ← update(${\text{Ξ}}_{new}$)	▷ update nonzero parameters
NewScore ← SLIC $\left({\text{Ξ}}_{new}\right)$
if NewScore < BestScore then
BestScore ← NewScore
$\text{Ξ}\leftarrow {\text{Ξ}}_{new}$
end if
end for

3. Results

3.1. SLIC extracts true governing equations of ordinary and partial differential from data

We set out to gauge the ability of SLIC to extract governing equations for dynamical systems from data in a sparse regression framework, which represents output data as a linear combination of nonlinear transformations of input data Brunton et al.. We compare the performance of SLIC to robustly recover models against a host of other popular information criteria in the literature, including AIC Stoica and Selen, Akaike, AICc Mangan et al., Kaptanoglu et al., Sugiura, HQIC Hannan and Quinn, BIC Schwarz, Stoica and Selen, KIC Kashyap, Bridge Criterion (BC) Ding et al. [b]. See Methods and Materials for details concerning the information criteria benchmarked against SLIC, the definition of noise percentage, a discussion of all metrics used to quantify algorithm performance, the candidate model terms used in each system, and the general preprocessing and workflow.

We first tested the algorithm on benchmark ordinary differential equations (ODEs) commonly found in the literature Messenger and Bortz [a], Brunton et al. (Fig. 2a). We found that SLIC greatly outperforms all other information criteria in recovering the true underlying model across all noise percentages, with near perfect accuracy in the recovered model terms (Fig. 2b). SLIC rarely includes false-positive model terms in the dynamics, as opposed to the other information criteria, which consistently have a high false-positive rate (FPR) (Fig. 2c). Interestingly, this trend continues even as data is downsampled (Fig. S1), where other information criteria perform more favorably, as predicted. SLIC also returns models with a smaller scaled mean absolute error (SMAE, see Methods) (Fig. S2a), indicating SLIC returns models with coefficients close to the true model coefficient values. Further, SLIC is also robust to changes in sampling frequency (Fig. S3), trajectory length (Fig. S4), and ODE parameters (Fig. S5).

Figure 2:

Performance of SLIC on ODEs and PDEs. (a,d) State-space plots of benchmark ODEs with 5% Gaussian noise (a), and PDEs with 20% Gaussian noise (d). See Methods for the equations for each system and other simulation details. (b,e) SLIC demonstrates superior recovery of the true governing equations of benchmark ODEs (b) and PDEs (e) over a range of noise levels, as opposed to other information criteria. Metric shown on y-axis is the average accuracy (n=25), which measures how often the correct model terms are identified by each model selection criterion. Accuracy ranges from 0 (correct equations are never correctly determined) to 1 (correct equations are always correctly determined). See Methods for further discussion. (c,f) SLIC rarely returns overfit models, as opposed to other information criteria. Metric shown on y-axis is the average false-positive rate (FPR) (n=25), which measures how often incorrect coefficients are identified by each model selection criterion. FPR ranges from 0 (incorrect coefficients are never included) to 1 (incorrect coefficients are always included). See Methods for further discussion

3.2. Discovering model for center of mass dynamics of a sloshing fluid in a rectangular tank

Across science and engineering disciplines, extensive domain knowledge and mathematics expertise have often been required to parse out models from experimental data to describe underlying dynamic processes. This constraint may hinder discovery of novel mechanisms where domain knowledge has not been accumulated sufficiently. Moreover, data in dynamic regimes where model predictions are the most impactful may be required as part of the training data during model construction, defeating the purpose of model predictions. In the sloshing tank example below, we test whether SLIC can extract models without domain knowledge and make accurate predictions well beyond the training regime used during model discovery.

The dynamics of sloshing liquids is critical in disciplines in which various vessels host onboard fluids, such as aerospace engineering Abramson, Veldman et al., Hubert and fuel transport Popov et al., Romero et al.. Often, the dynamics of these onboard fluids may disrupt and possibly endanger the intended function of these various vessels. These phenomena are highly complex and typically demand expert knowledge of the underlying mechanism to understand their behavior. Thus, it is highly desirable for DDMD algorithms to aid in the discovery of the dynamics of such systems.

To experimentally assess the dynamics of sloshing fluids in a controlled manner, Ref. Bäuerlein and Avila engineered a setup in which a motor horizontally drives a rectangular tank at a specified frequency and amplitude. The images of the sloshing fluid gathered from this setup can be used to extract timeseries data of the center of mass, permitting the researchers to explore the dynamics of this system (Fig. 3a).

Figure 3:

SLIC discovers center of mass (CoM) dynamics of a fluid in a sloshing tank and accurately predicts unseen forcing behavior. (a) Experimental design of sloshing tank experiments. We wish to determine the CoM dynamics extracted from images of unforced sloshing tank experiments. Image adapted from Bäuerlein and Avila, Cenedese et al.. (b) The governing equations extracted by SLIC using minimal training data from a single sloshing experiment accurately forecasts beyond training data. (c) Governing equations extracted by SLIC accurately predict CoM amplitude from unseen forcing conditions. The x-axis is forcing frequency relative to fundamental frequency. The y-axis is the steady-state CoM amplitude expressed as a percent of the tank width. (d) Governing equations extracted by SLIC accurately predict phase shift resulting from unseen forcing conditions. The x-axis is forcing frequency relative to fundamental frequency.

Here, we test if, using our algorithm, we could discover a model directly from the data in a simple manner, without any domain knowledge to fit data to a pre-determined model (i.e. not knowing the underlying dynamics or existing symmetries a priori). We find that with data from a single sloshing experiment without forcing, SLIC determines that the dominant dynamics is governed by a Duffing system (Fig. 3b):

$$\ddot{\text{x}}=-{(7.802)}^2\text{x}-2\left(0.072\right)\stackrel{.}{\text{x}}+0.351{\text{x}}^3$$ (8)

In contrast, other information criteria again return a non-sparse solution (Fig. S6a).

Next, we test if SLIC can make accurate predictions on future behaviors. First, we find that this spartan workflow can forecast quite well based on limited training data (Fig. 3b). Second, unlike the manual fitting that requires datasets from both unforced and forcing conditions, SLIC is only trained on data from the unforced condition. We therefore challenge SLIC to predict dynamics in the unseen forcing condition. Strikingly, the model extracted by SLIC from the unforced data accurately predicts the forcing behavior. SLIC prediction on the steady-state amplitude (Fig. 3c) and phase-shift (Fig. 3d) of the forced system shows excellent agreement with experimental data. Moreover, SLIC even captures the hysteretic, unstable portion of the curves where experimental data cannot be collected. This is critical, as design considerations for sloshing liquids require accurate predictions of the system in unstable regimes, where SLIC has demonstrated excellent predictive power.

3.3. Discovering the functional form of RNA-liposome cluster-cluster interaction rate and improving mRNA delivery in vitro based on model prediction

Lastly, we test if SLIC can perform model discovery in the face of additional unknown, model-specific parameters that must be determined via auxiliary optimization schemes. Here we test SLIC’s ability to extract the aggregation model of a clinically relevant nanomaterials system Mendez-Gomez et al. [a], which requires determining an unknown exponent (the fractal dimension) via a nested optimization procedure.

Nanoscience has transformed countless disciplines in recent decades, with applications ranging from material science Malik et al. to modern therapeutics Mitchell et al.. In these applications, models informing the rational design and engineering of nanoparticle formulations is paramount for success Blanco et al., Albanese et al., Hoshyar et al.. In particular, it is often critical to understand how nanoparticles self-assemble Bassani et al., Rao et al.. One general, well-established class of models describing time-dependent self-assembly is Smoluchowski aggregation theory Elimelech et al. (see Supporting Text). Smoluchowski theory is a collection of coupled nonlinear differential equations describing how clusters of particles evolve over time:

(9)

Here $n_k$ denotes ‘k-mers’, or clusters containing ‘k’ particles and ${\text{K}}_{ij}$ denotes the kernel function, which is symmetric and dictates the rate of interaction between clusters (Fig. 4a) and characterizes the underlying transport and aggregation mechanism of the system (Fig. 4b). The first term on the RHS of the above expression determines the ways in which one can create k-mers via the assembly of smaller clusters; the second term determines how k-mers are destroyed via merging with other clusters.

Figure 4:

SLIC discovers the functional form of cluster-cluster interaction rate (the kernel) and predicts cluster growth rate of unseen initial conditions. (a) Illustration of the cluster-cluster interaction rate (i.e. the kernel, ${\text{K}}_{ij}$) for clusters of different particle number. The indices denote the number of monomers/particles per cluster. (b) Functional forms of kernels for different physical processes. Note that each depend on an unknown parameter, ${\text{d}}_f$. (c) Fluorescence timeseries images of the RNA-liposome system. Scale bar is 10m. (d) Images of the ‘Original’ and ‘Revised’ RNA-nanoparticle formulations. Original’ denotes the undiluted solution. Scale bar is 5μm. (e) The constant kernel determined by SLIC accurately predicts the growth rate of 2x diluted solution of RNA-liposomes (See Methods).

Thus, for many nanoparticle systems, the goal is to understand the nature of the kernel and its associated physics. However, manual trial-and-error kernel fitting procedures are time-consuming and impractical, especially if the assembly mechanism is not known. Thus, we reason that DDMD can be an extremely powerful approach for understanding the dominant physics of cluster-cluster interactions by extracting the functional form of the kernel.

To demonstrate the ability of the algorithm to discover the functional form of a kernel from data, we chose to study a novel RNA-liposome cancer immunotherapy currently in clinical trials (NCT04573140). Interestingly, over time, heterogeneous aggregates form in this system, which are critical for therapeutic efficacy Mendez-Gomez et al. [b], Chung et al.. However, the underlying mechanism for this aggregation phenomenon is poorly understood, making this system an ideal candidate for testing the ability of SLIC to recover the kernel’s functional form. Using DDMD permits us to extract the relevant kernel in a completely data-driven manner without manually fitting kernels.

Experimentally, we conduct quantitative fluorescence imaging of these particles (Fig. 4c) with single-cluster resolution as described in Ref. Chung et al.. From these images, we acquire time series data of the clusters, from which we can numerically estimate the kernel (See Methods). With the numerically estimated kernel, we can then use SLIC to extract the kernel’s functional form from a candidate array of relevant kernel components (Fig. 4d) (See Methods for details). Each of the kernels in Fig. 4b can be reconstructed using an appropriate linear combination of the kernel components we train SLIC with, while permitting potential discovery of novel kernels or kernel combinations. Importantly, all relevant kernel functions (Fig. 4b) contain an unknown parameter, the fractal dimension $\left({\text{d}}_f\right)$, which quantifies the compactness of the aggregated clusters and determines important physical properties like density Elimelech et al.. Thus, in determining the kernel for this system, one must also estimate this unknown parameter simultaneously. To do so, we perform a round of model discovery with a fixed estimate for the fractal dimension, followed by an optimization loop that updates the unknown parameter; these steps are then iterated until convergence (Fig. S7a). Strikingly, from the candidate list of kernel components, SLIC identifies only the constant kernel as dominating the cluster-cluster interactions, which fits the experimental data quite well (Fig. S7b). Interestingly, this model with a constant kernel can be taken as an analytically solvable approximation to the Brownian kernel Elimelech et al., indicating that for this system, the cluster growth timescale is dictated by the diffusive nature of clusters (See Supporting Information for details).

This diffusion-dominated growth suggests a new strategy for the control of nanoparticle assembly in this system. Specifically, an initially lower concentration of particles should lead to a slower cluster growth rate. This hypothesis, inspired directly by the functional form of the kernel returned by the model, is indeed observed when we decrease the initial particle concentration (Fig. 4e). The linear growth rate of the average cluster size is also in quantitative agreement with the predicted kernel (See Supporting Information). Notably, other information criteria returns a kernel with unnecessary terms (Fig. S6b), thus obfuscating the dominant cluster-cluster interaction physics.

4. Conclusion

In this work, we present an algorithmic module that: (i) automatically and adaptively generates sparsity-enforcing parameters from the data and (ii) utilizes a novel sample-length-scaling information criterion, SLIC, to select models that are both accurate and simple. These parts work in synchrony to automatically generate and score models, which minimizes user-intervention for automating model discovery.

Our scoring criterion, SLIC, is designed with several objectives in mind. First, to respect the fact that if one assumes that there is an underlying ground truth model, then the optimal model should be invariant to the amount of data seen by an algorithm. Unlike existing information criteria utilized in DDMD, SLIC ensures that models do not exhibit a higher tendency to overfit with increasing data. Second, the functional form of SLIC determines that it equitably weighs accuracy and model complexity. Interestingly, in the sparse regression framework, minimizing the SLIC score also minimizes the likelihood that models are overfit due to the intimate relationship with variance minimization (see Supporting Text). This likely explains SLIC’s performance, though it may be desirable to more directly tie SLIC to KLD minimization or a Bayesian approach Stoica and Selen.

Our results illustrate that SLIC with sparse regression correctly extracts governing equations from data spanning a broad spectrum of simulated ordinary and partial differential equations, including high noise levels and across a wide range of simulation conditions. Further, we demonstrated that the method can be used fruitfully with real experimental data.

This work demonstrates that careful consideration of the design of model-selection criteria is critical in DDMD. Though the form of our SLIC score does not currently possess hyperparameters, one could introduce parameters to tune the weighting of sparsity and goodness-of-fit (see Supporting Text for a discussion), which is analogous to the AIC-GIC relation Stoica and Selen, and may be suitable in other domains such as autoregression-based time-series forecasting, where model selection is critical Box et al. [2015].

5. Materials and Methods

5.1. Algorithm Performance Metrics

In this work, we use several metrics to quantify algorithm performance. Accuracy and false-positive rate (FPR) quantify the ‘classification’ performance of the algorithm (i.e., how often the algorithm correctly predicts the inclusion of a model term). With the shorthand TP = True Positive, FP = False Positive, TN = True Negative, FN = False Negative, these classification metrics are defined as follows:

$$\text{Accuracy}=\frac{\text{TP}+\text{TN}}{\text{TP}+\text{FP}+\text{TN}+\text{FN}}$$ (10)

$$\text{FPR=}\frac{\text{FP}}{\text{FP}+\text{TN}}$$ (11)

The accuracy reports how often the algorithm correctly predicts whether a model term should be included or not. The FPR reports how often a model term is incorrectly included in the returned model when it should not be. In the main text, the metrics are computed from averages of all the separate noise instantiations.

To quantify the algorithm’s ability to recover the correct numerical values for each model, we utilized the scaled mean absolute error (SMAE), defined as:

$$\text{SMAE}=\frac{{‖{\text{Ξ}}_{\text{true}}-{\text{Ξ}}_{\text{pred}}‖}_1}{{‖{\text{Ξ}}_{\text{true}}‖}_1}$$ (12)

Where $\Vert \cdot {\Vert }_1$ denotes the L1 norm, ${\text{Ξ}}_{\text{true}}$ is the true matrix of model coefficients, and ${\text{Ξ}}_{\text{pred}}$ is the predicted matrix of model coefficients returned by the algorithm.

To get statistics for each metric, at a fixed noise level, all systems studied were simulated 25 times with random instantiations of the noise, which is zero-mean Gaussian noise with unit magnitudes determined by

$$\text{\%}\text{Noise}=\frac{100\sigma }{\text{std}(\text{X})}$$

where $\sigma$ is the standard deviation of the Gaussian noise and std(X) is the state variable input data. The reported metrics at each noise percentage are the average of the respective metric across the different noise instantiations.

5.2. Information Criteria Used In Comparisons

In this section, we provide a list of the information criteria tested in this work. For each, $\widehat{ϵ}$ denotes the mean-square error of the model, n denotes the number of datapoints, and k denotes the number of free parameters (model complexity).

$$\mathrm{AIC}\left(\widehat{ϵ},\text{k}\right)=\text{n}\log \left(\widehat{ϵ}\right)+2\text{k}$$

$$\mathrm{AICc}\left(\widehat{ϵ},\text{k}\right)=\text{n}\log \left(\widehat{ϵ}\right)+2\text{nk}/(\text{n}-\text{k}-1)$$

$$\mathrm{HQIC}\left(\widehat{ϵ},\text{k}\right)=\text{n}\log \left(\widehat{ϵ}\right)+\text{k}\log (\log (\text{n}))$$

$$\text{BIC}\left(\widehat{ϵ},\text{k}\right)=\text{n}\log \left(\widehat{ϵ}\right)+\text{k}\log (\text{n})$$

$$\text{KIC}\left(\widehat{ϵ},\text{k}\right)=\text{n}\log \left(\widehat{ϵ}\right)-\text{k}\log (2\pi )+\log \left(\left|{\text{Θ}}^T\text{Θ}/\widehat{ϵ}\right|\right)$$

$$\text{BC}\left(\widehat{ϵ},\text{k}\right)=\text{n}\log \left(\widehat{ϵ}\right)+{\text{n}}^{1/3}1+\frac{1}{2}+\dots +\frac{1}{\text{k}}$$

$$\mathrm{SLIC}\left(\widehat{ϵ},\text{k}\right)=\text{n}\log \left(\widehat{ϵ}\text{k}\right)$$

For the final term on the RHS of KIC, |⋅| denotes the determinant and $\text{Θ}$ is the library matrix of nonzero terms; thus, it depends on both n and k.

5.3. The choice of candidate model terms

In this section, we provide a brief description of the libraries used in the main text. Specifics concerning the implementation can be found at https://github.com/MCHChung/SLIC.

5.3.1. Simulated systems

For Rossler, Brusselator, and Van der Pol we used a 3^rd order polynomial basis in the state variables; this means that if (x, v) are the state variables, we consider $\left\{\text{1,x,v,…,}{\text{x}}^{\text{3}}\text{,}{\text{x}}^{\text{2}}\text{v,x}{\text{v}}^{\text{2}}\text{,}{\text{v}}^{\text{3}}\right.$. For Lorenz and Lotka-Volterra, we considered a 3^rd order polynomial basis without constant term. For the nonlinear pendulum, we use a Fourier basis up to 5^th order (i.e. $\{1,\sin (x),\cos (x),\dots ,\sin (5x),\cos (5x)$). For VdP, in which the library contains nonlinear functions of the velocity, we first estimate the velocity via a 4^th order central difference method; this estimate and the position form the state variable inputs to the algorithm.

For Burger’s equation, KdV, and KS the basis consisted of terms up to 4^th order in spatial derivatives of the state variables, up to cubic terms in the state variables, and a nonlinear convective term (i.e. $\text{u}{\partial }_x\text{u}=0.5{\partial }_x{\text{u}}^2$) in each state variable. For NLS, the basis consisted of terms up to 2^nd order in spatial derivatives of the state variables, up to 3^rd order polynomial basis in the state variables, and a nonlinear convective term in each state variable. For SG, the basis consisted of terms up to 2^nd order in spatial derivatives of the state variables (in x and y), 1^st order in time derivative, up to 2^nd order polynomial basis in the state variables, and sin and cos trigonometric functions of the state variables.

5.3.2. Experimental systems

For the sloshing tank system, the derivative of the center of mass is estimated via a 4^th order central difference method. The center of mass data and its derivative estimate form the state variable inputs. For this system, we consider a 3^rd order polynomial basis (with no constant term, as that would imply a constant applied force) in these inputs. For the RNA-nanoparticle system, the basis consists of the kernel elements described in the text $\left.\left\{1,{\text{i}}^{1/df}\right.,{\text{j}}^{1/df},{(\text{i}/\text{j})}^{1/df},{(\text{j}/\text{i})}^{1/df},{\left({\text{i}}^2\text{j}\right)}^{1/df},{\left({\text{ij}}^2\right)}^{1/df},{\text{j}}^{3/df}\right\}$, and also $|{\text{i}}^{1/df}-{\text{j}}^{1/df}|$ multiplied by all 3rd order terms, ${\text{i}}^{3/df}$, ${\left({\text{i}}^2\text{j}\right)}^{1/df}$, ${\left({\text{ij}}^2\right)}^{1/df}$, ${\text{j}}^{3/df}$. As described in the text, the indices i, j denote the number of monomers per cluster for the clusters interacting and ${\text{d}}_f$ is the unknown fractal dimension.

5.4. Data pre-processing

For all simulated systems, the workflow proceeds in several steps: (1) filter data to reduce noise (2) employ Galerkin projection to construct the library and targets and (3) carry out the hard-thresholding sparse regression algorithm with model selection using Algorithm 1 (or other information criteria). Below, we describe the first two steps.

5.4.1. Brief description of the fast Whittaker-Henderson filter

Often, the state variable data will be noisy. The Whittaker-Henderson filter Whittaker is one such approach for denoising that attempts to solve the following:

$$\arg \underset{y}{\min }\Vert \text{y}-\hat{\text{y}}{\Vert }_2^2+\text{s}\Vert \text{D}\hat{\text{y}}{\Vert }_2^2$$ (13)

Where y is our input data, $\hat{\text{y}}$ is the smoothed data to be determined, and $\text{s}\Vert \text{D}\hat{\text{y}}{\Vert }_2^2$ is a regularization penalty that penalizes non-smooth outputs, with s being the regularization strength and D a differential operator. However, the solution depends strongly on the choice of s. In this work, we implement a version of the filter that leverages the fast Fourier transform for rapid optimal determination of the smoothing parameter, s Garcia. This approach has the benefit of not only being fast, but easily generalizes to high-dimensional data, such as one might encounter with PDEs. The filtered input data, X, is then used to generate the library matrix $\text{Θ}(\text{X})$ and the targets, Y, if necessary. This is done for each simulated system.

5.4.2. Galerkin method for constructing reframed regression problem

To combat the challenges inherent in numerically estimating derivatives of state variables, which is a necessary component of performing DDMD with ODEs and PDEs, we use a modified version of a Galerkin method Messenger and Bortz [a,b]. This approach circumvents the need to estimate numerical derivatives of noisy data by modifying the underlying regression problem. Specifically, for the case of ODEs, if we assume that our targets are $\text{Y}=\left[\begin{array}{ccc}{\text{y}}_1& \dots & {\text{y}}_m\end{array}\right]=\left[\begin{array}{ccc}{\dot{\text{x}}}_1& \dots & {\dot{\text{x}}}_m\end{array}\right]=\dot{\text{X}}$, then, for each state variable, our regression problem takes the form:

(14)

Where ${\text{Ξ}}_{ij}$ are the model coefficients to be determined. Now, the Galerkin method seeks to remove the necessity of computing numerical derivatives found on the left and right sides of this equality. Defining ${〈\text{f}〉}_w{=}_t^{{\text{J}}_{t_2}}\text{w}(\text{t})\text{f}(\text{t})\text{dt}$ and multiplying by a known function w(t) on both sides of this equation, integrating by parts on the LHS, we obtain:

(15)

Importantly, we have assumed that w(t) vanishes on the boundaries of the integration window. An infinite family of functions satisfies this property, and we utilize a symmetric, positive-valued polynomial function of the form

$$\text{w}\left(\text{t}\right)=\text{w}\left(\text{t};{\text{c}}_1,{\text{t}}_1,{\text{t}}_2\right)={\text{C}}_q{\left[\left({\text{t}}_2-\text{t}\right)\left(\text{t}-{\text{t}}_1\right)\right]}^q,$$

where q is a positive real number and ${\text{C}}_q$ is a normalizing coefficient chosen so that $⟨1w=1⟩$. Thus, in short, we have transferred a numerical derivative of noisy data to the exact derivative of the known function w(t). In practice, the new, full-fledged Galerkin-projected vectors that serve as inputs to the sparse discovery algorithm are created by sliding our integration window along the input vector and concatenating the results of this integration.

For PDEs, the procedure is completely analogous, with the exception that the function we are multiplying by has multiple dimensions (i.e., $\text{w}=\text{w}\left(\text{t},{\text{x}}_1,{\text{x}}_2,\dots \right)$). For this we also use a symmetric polynomial kernel as described previously, with a single parameter. In one space and time dimension, this would be:

$$\text{w}\left(\text{t},\text{x};\text{q},{\text{t}}_1,{\text{t}}_2,{\text{x}}_1,{\text{x}}_2\right)={\text{C}}_{\text{q}}\left({\text{t}}_2-\text{t}\right){\left(\text{t}-{\text{t}}_1\right)}^q\times {\text{C}}_{\text{q}}\left({\text{x}}_2-\text{x}\right){\left(\text{x}-{\text{x}}_1\right)}^q,$$

Where $({\text{x}}_1,{\text{x}}_2)$ is the boundary of integration of the x-dimension. We must also perform multidimensional integration. For all integration, we use trapezoidal integration provided by the Trapz.jl package in Julia.

To automatically determine the window size of integration, we repurpose the structural similarity index measure (SSIM) used in the image analysis field that was originally designed to determine the similarity between an image and a distortion of it Wang et al.. Such distortions may include convolutions or other filters. Given that the integration performed via the Galerkin method is a convolution, we reasoned that the SSIM can be used for automatically determining a window size of integration from an array of possible window sizes that minimally distorts the underlying data.

In summary, with this Galerkin method, our regression problem now takes the modified form:

(16)

Where $\tilde{\text{Y}}=\left[\begin{array}{lll}{-⟨{\text{x}}_1⟩}_{\stackrel{\cdot }{w}}\hfill & \dots \hfill & -{⟨{\text{x}}_m⟩}_{\stackrel{\cdot }{w}}\hfill \end{array}\right],\tilde{\Theta }(\text{X})=\left[{〈{\text{θ}}_1(\text{X})〉}_w\dots {〈{\text{θ}}_l(\text{X})〉}_w\right]$, and $\text{R}(\text{Ξ})$ is some regularization term to promote sparsity, with $\lambda$ the corresponding regularization weight parameter. It should be noted that it may be occasionally necessary to estimate numerical derivatives for libraries that contain nonlinear functions of the derivatives of state variables. In such cases, the Galerkin method is still of great use, as it still prevents successive, higher order estimates of derivatives for the target input.

5.5. Information regarding the simulated ODEs and PDEs

All data for ODEs featured in the main text were simulated using the OrdinaryDiffEq.jl package in Julia using the Tsit5 solver with a fixed time step of 0.01, except for Lorenz, Rössler, and Brusselator, which had a timestep of 0.001. For each ODE, three initial conditions were randomly chosen and simulated for the timespan shown for each system below at the aforementioned sampling frequency; this data was then concatenated and formed the input data. The data for all PDEs was acquired from references Messenger and Bortz [b], Rudy et al. and is provided in our GitHub for convenience. Further discussion of time stepping schemes may be found in those resources. Information pertinent to modifications of these systems is contained in the Supporting Information.

Below we give the equations studied in this work.

Lorenz

$$\frac{\text{dx}}{\text{dt}}=10(\text{y}-\text{x})$$

$$\frac{dy}{dt}=x\left(28-z\right)-y$$

$$\frac{\text{dz}}{\text{dt}}=\text{xy}-\frac{8}{3}\text{z}$$

Timespan: $\text{t}\in [0,10]$.

Rossler

$$\frac{\text{dx}}{\text{dt}}=-\text{y}-\text{z}$$

$$\frac{dy}{dt}=x+0.2y$$

$$\frac{\text{dz}}{\text{dt}}=0.5+\text{z}(\text{x}-5.7)$$

Timespan: $\text{t}\in [0,10]$.

Lotka–Volterra

$$\frac{\text{dx}}{\text{dt}}=\text{x}-0.05\text{xy}$$

$$\frac{\text{dy}}{\text{dt}}=0.05\text{xy}-\text{y}$$

Timespan: $\text{t}\in [0,20]$.

Nonlinear system

$$\frac{\text{dx}}{\text{dt}}=1-4\text{x}+{\text{x}}^2\text{y}$$

$$\frac{\text{dy}}{\text{dt}}=3\text{x}-{\text{x}}^2\text{y}$$

Timespan: $\text{t}\in [0,10]$.

Van der Pol

$$\frac{\text{dx}}{\text{dt}}=\text{v}$$

$$\frac{dv}{dt}=-\text{x}+0.8\left(1{\text{x}}^2\right)\text{v}$$

Timespan: $\text{t}\in [0,30]$.

Nonlinear pendulum

$$\frac{\text{dx}}{\text{dt}}=\text{v}$$

$$\frac{\text{dv}}{\text{dt}}=-4\sin (\text{x})$$

Timespan: $\text{t}\in [0,20]$.

Burger’s Equation

$${\partial }_t\text{u}(\text{x},\text{t})=-\text{u}(\text{x},\text{t}){\partial }_x\text{u}(\text{x},\text{t})-0.1{\partial }_{xx}\text{u}(\text{x},\text{t})$$

Conditions: $\text{u}(\text{x},0)={\text{e}}^{-{(x+2)}^2}$, $\text{t}\in [0,10]$, $\text{x}\in [-8,8]$. An evenly spaced grid of 101 timepoints and 256 spatial points was used.

Kortweg-De Vries

$${\partial }_t\text{u}\left(\text{x},\text{t}\right)=-6\text{u}\left(\text{x},\text{t}\right){\partial }_x\text{u}\left(\text{x},\text{t}\right)-{\partial }_{xxx}\text{u}\left(\text{x},\text{t}\right)$$

Conditions:

An evenly spaced grid of 201 timepoints and 512 spatial points was used.

Kuramoto–Sivashinsky

$${\partial }_t\text{u}(\text{x},\text{t})=-\text{u}(\text{x},\text{t}){\partial }_x\text{u}(\text{x},\text{t})-{\partial }_{xx}\text{u}(\text{x},\text{t})-{\partial }_{xxxxx}\text{u}(\text{x},\text{t})$$

An evenly spaced grid of 1500 timepoints and 256 spatial points was used. Authors then subsampled 20% of the time points for a final resolution of $\text{Δx}=0.393$ and $\text{Δt}=0.5$.

Nonlinear Schrödinger

The full nonlinear Schrödinger equation, with $\text{Ψ}$ a complex field, is:

$$\text{i}{\partial }_1\text{ψ}(\text{x},\text{t})=-\frac{1}{2}{\partial }_{xx}\text{ψ}(\text{x},\text{t})-|\text{ψ}{|}^2\text{ψ}(\text{x},\text{t})$$

Decomposed into real and imaginary components, $\text{Ψ}=\text{u}+\text{iv}$:

Conditions:Gaussian initial condition, $\text{t}\in [0,\text{π}]$, $\text{x}\in [-5,5]$. An evenly spaced grid of 501 timepoints and 512 spatial points was used.

Sine–Gordon

$${\partial }_{tt}\text{u}(\text{x},\text{y},\text{t})={\partial }_{xx}\text{u}(\text{x},\text{y},\text{t})+{\partial }_{yy}\text{u}(\text{x},\text{y},\text{t})-\sin (\text{u}(\text{x},\text{y},\text{t}))$$

Conditions:

$$\text{u}(\text{x},\text{y},0)=2\text{π}{\text{e}}^{-8{(x-0.5)}^2-8y^2},\text{t}\in [0,5],\text{x}\in [-\text{π},\text{π}],\text{y}\in [-1,1].$$

5.6. RNA-Liposome kernel estimation and dual optimization problem for model discovery

We use the DiffEqFlux.jl package to estimate the kernel in a Smoluchowski model (Supporting Information) from our k-mer timeseries data that extends up to 50-mers. DiffEqFlux.jl determines the kernel that minimizes the L₂ error between the trajectories predicted by the Smoluchowski model and our data. Each element of this estimated kernel is then a function of a pair of k-mer indices, i, j. To encourage kernel symmetry (i.e., ${\text{K}}_{ij}={\text{K}}_{ji}$), we concatenate this initial vector of estimated kernel values with its symmetric component; this forms the target matrix we are trying to find a model for. Once the kernel is estimated, for 1000 iterations we perform a sparse regression estimation of the model for each information criteria with a fixed ${\text{d}}_f$ followed by an optimization problem that updates ${\text{d}}_f$. The initial guess for ${\text{d}}_f$ is chosen randomly between 1.5 and 3, which are reasonable physical constraints bounding ${\text{d}}_f$. For the optimization loop that updates ${\text{d}}_f$, we minimize the loss function

$$\text{Loss}={‖{\widehat{\text{K}}}_{\text{est}}-\text{Θ}\left(\text{i},\text{j};{\text{d}}_f\right)\text{Ξ}‖}_2^2$$

using the Optimization.jl package with the SAMIN() optimizer from the Optim.jl package that can perform bounded simulated annealing. This is repeated three times for each information criterion and the model with the best score for each criterion is chosen as the final model for evaluation.