Deep learning approach to parameter optimization for physiological models

Abstract

Inferring nonlinear dynamics and parameters in biological data modeling is challenging. Standard parameter optimization methods are difficult to constrain to biological ranges, especially for nonlinear models. We propose a novel method to evaluate and improve putative models using neural networks to simultaneously address biological modeling, parametrization, and parameter inference. As an example, utilizing data from clinical frequently sampled intravenous glucose tolerance testing, we introduce two physiological lipolysis models of glucose, insulin, and free fatty acids dynamics. Parameter values are obtained via optimization from the limited clinical data. We then generate simulated data from the model by sampling parameters within physiological ranges while ensuring that the joint parameter distributions are physiologically appropriate. A convolutional neural network is trained to take the simulated glucose, insulin, and free fatty acids time courses as input and output of the model parameters. We evaluate the performance of the trained neural network for both parameter inference and trajectory reconstruction using a testing dataset, optimized model-fitting curves, and real physiological data and show that it enables accurate inference across all three settings. The trained neural network produces consistently high R² values and low P-values across different feature engineering strategies and training dataset sizes. We assess the impact of feature engineering choices and training dataset size on inference performance, demonstrating that appropriately designed feature transformations and specific activation function choices improve accuracy. Our results establish a deep learning framework for parameter inference in mathematical models, which can be adapted to various physiological systems.

Introduction

Any physiological model is based on quantitative hypotheses of a limited set of specific mechanistic interactions, with the aim of fitting model parameters to subject data in order to find subject-specific parameter values. The motivation is that deviations from specific parameter ranges may indicate a need for medical intervention, and furthermore, for clinically useful models, these may even guide the nature of a clinical intervention. Thus, estimating model parameter values from physiological data is a key problem in the development of dynamical system models of physiology.

Determining model parameters requires model optimization, especially for nonlinear models. Often, physiological parameters have specific parameter ranges, e.g. to ensure that predictions for plasma glucose in any model of glucose–insulin dynamics stay non-negative. Optimization algorithms are available for a large variety of problem characteristics, including constrained optimization, but here the complexity of physiology leads to issues. The underlying physiology invariably has latent processes that are not included in the model, but which lead to interactions between processes that were incorporated as independent in the model. Therefore, not just must the parameters lie within specific ranges, but they must also incorporate specific covariances that result from latent processes.

Traditional fitting of nonlinear physiological models typically relies on iterative optimizers such as Nelder–Mead [1] or BFGS [2]. These approaches often require numerous model evaluations—and, in the case of BFGS, accurate gradient information—to gradually converge on a best‐fit parameter set. Moreover, unless explicit bounds or penalty functions are imposed, there is no guarantee that the optimized parameters will lie within known physiological limits, and adding such constraints can further slow and complicate convergence. As a result, fitting even moderately complex ODE models can be time-consuming, sensitive to initial guesses, and difficult to enforce within physiological ranges. This motivates our deep-learning approach, which delivers one-shot parameter inference without repeated optimization.

As our prior understanding of parameter space distributions given the physiological data is lacking, and the computation cost of performing forward mapping (integration) is less than that of the inverse mapping (optimization), we propose the introduction of deep learning tools to learn the inverse mapping through training neural networks. To facilitate neural network training, massive training data is usually required. Fortunately, in this context, generating simulated data is straightforward once the model is set up. We need to retain only parameter sets whose corresponding trajectories from integration satisfy observed physiological data constraints. We take our prior distribution from such parameter sets and generate simulated data for the training of the neural network, which delivers one-shot parameter estimates for new data which avoids iterative optimization. This methodology is theoretically based on the Kolmogorov universal approximation theorem, which states that in a compact region appropriate feedforward neural networks can approximate a nonlinear function arbitrarily well [3]. While the design of the neural network can be refined through iterative trials to determine which feature engineering approach performs best, prior analysis of the underlying models can provide insights to hypothesize which features are likely to be more effective, potentially reducing the need for extensive experimentation.

This simulation-driven strategy is close to simulation-based inference (SBI) methods, which also use forward simulator runs to learn inverse mappings. SBI approaches include approximate Bayesian computation (ABC) [4, 5], neural posterior estimation (NPE) [6, 7], neural likelihood estimation (NLE) [8], neural ratio estimation (NRE) [9, 10], and unified toolkits such as SBI, which amortize all of these algorithms into reusable workflows [11]. Simulation-based Model Inference (SBMI) further extends SBI to jointly infer both model structure and parameters [12]. Our method shares partial similar ideas with amortized SBI in the sense that we amortize the computation effort across many new inferences rather than repeating the optimization every time, but they also have differences that our trained network works as a point estimator rather than outputting a posterior distribution of model parameters. We will compare our method and some of these SBI methods in the discussion section.

Besides fully connected networks, we also incorporate convolutional neural networks (CNNs) into the architecture to capture intricate relationships between neighboring data points. There have also been other studies that use CNNs in parameter inference: Rudi et al. [13] demonstrate that combining dense and convolutional layers can effectively learn inverse maps for nonlinear ODEs—such as the FitzHugh–Nagumo model—and that CNNs significantly outperform dense-only architectures in recovering dynamical parameters from noisy time-series data. Ben-Shalom et al. [14] use CNNs to infer neuronal ionic conductance from membrane potential recordings by training a one-dimensional convolutional network, which surpasses the traditional multi-objective optimization methods in this field.

Building on these insights, we now outline our specific contributions. In this study, we develop this deep-learning approach and apply it in detail to several ordinary differential equation (ODE) models describing insulin’s effect on lipolysis and glucose disposal as example models. The physiological models we use as examples are widely used and clinically relevant. Our aim is to provide a detailed and thorough exposition, explaining various features of our approach, that should allow others to develop such strategies for their own models.

We apply and validate our approach for inferring parameter values from physiological data, utilizing frequently sampled insulin glucose tolerance test (FSIGT) data and models of non-esterified free fatty acid (FFA) lipolysis as illustrative examples. Our results show that the trained neural network accurately infers physiological parameters and reproduces experimental trajectories, confirming the approach’s reliability in real-data scenarios. Initially, we select several models and determine their parameter settings, followed by optimization to obtain parameter values from physiological data (including time, glucose, FFA, and insulin data), thereby establishing a distribution for each parameter corresponding to the physiological data. Subsequently, we generate sample parameter sets lying in the domains of these parameter distributions, integrating them into the model to obtain simulated data. Importantly, we demonstrate that appropriate sampling of these parameter sets is critical to training because physiological models are usually developed with an eye toward mechanistic understanding and therefore are described with parameter sets that have nontrivial joint distributions for physiologically plausible datasets. This is primarily because physiology is complex with many more processes interacting than can be included in a mathematical model given available data. These simulated data are partitioned into training, validation, and test datasets. The training and validation datasets are inputs to the neural network, while the corresponding parameter samples are outputs.

In addition to optimizing neural network architecture, our approach incorporates feature engineering to enhance the effectiveness of input data. This decision is motivated by the recognition that the relationship between raw data (such as time, glucose, FFA, and insulin levels) and outputs is likely to be more nonlinear than a neural network of a certain complexity can effectively capture in practice, as the Kolmogorov theorem applies in the limit of infinitely wide networks. As neural networks are composed of linear transformations and sigmoid functions, we introduce nonlinear transformations of the raw data as inputs to the neural network. By doing so, we aim to enable the network to capture underlying relationships without needing to be overly complex, saving considerable computational cost. The choice of nonlinear functions is informed by both empirical trials and insights derived from the ordinary differential equation (ODE) models of interest.

We demonstrate that our methodology, when applied to a well-trained neural network, achieves accurate parameter inference with high $R^2$ values and small inference errors over the test dataset. The inferred parameters also produce reconstructed trajectories that closely match those obtained from traditional model fitting. Most importantly, when applied to real physiological data, the network yields small reconstruction errors, confirming the reliability and generalization of the inferred parameters. Proper feature engineering improves validation loss convergence and enhances inference accuracy, particularly when combined with larger training dataset sizes. These findings highlight the effectiveness of the proposed approach in parameter estimation and model-based inference.

The paper is structured as follows: Section “Materials and methods” details the physiological data, the lipolysis models used as examples, and the common procedures of our method, including the generation of simulated data, training of our proposed neural network, and parameter inference. The specifics of our approach are described in sections “Generation of simulated data” and “Primary convolutional network.” Section “Results” presents the results, evaluating the inference performance over the test dataset, the model-fitting curves and real physiological data, and the impact of feature engineering and the training dataset size. Finally, Section “Discussion” provides a discussion of the results presented in the paper.

Materials and methods

Physiological data

In this section, we introduce two models for lipolysis mechanism to fit the physiological glucose and/or FFA data from Stefanovski et al. [15], obtained through an FSIGT protocol from 25 subjects. In this protocol, glucose was administered intravenously at time zero over 1 min, followed by the infusion of normal saline. Twenty minutes after the glucose injection, a bolus of insulin (0.03 U/kg) was injected. Blood samples for measuring glucose, free-fatty acid, and insulin concentrations were collected at the following minutes post-glucose injection

$$\begin{array}{cc}{t}_{\text{all}}=\{0,2,3,4,5,6,8,10,12,14,16,19,22,24,25,27,30,\hfill & \hfill \\ 40,50,60,70,80,90,100,120,140,160,180\}.\hfill & \hfill \end{array}$$

In certain models, we use the time points after time 20, $t_{\text{post}20}=\{t\in t_{\text{all}}\mid t>20\}$. We define the glucose, FFA, and insulin level measured at time t to be $G_{\text{data}}(t),F_{\text{data}}(t)$. In particular, we denote $G_b=G_{\text{data}}(0),F_b=F_{\text{data}}(0)$ and $I_b=I_{\text{data}}(0)$.

Before proceeding, we note that we select this dataset purely because it provides fully concurrent time-course measurements under a single standardized protocol. A sample size of 25 subjects provides adequate coverage of normal physiological variability. Researchers applying this framework to cohorts with pathologies only need to broaden the prior parameter ranges, provided the same model structure and underlying mechanism remain valid and disease effects appear as shifts in parameter values.

Two lipolysis models

In this paper, we choose two distinct ODE models applied to the same FSIGT data introduced in section “Physiological data,” so that their diversity demonstrates how our deep learning framework can adapt to different model structures. The first model (from Periwal et al. [16]) is

$$\{\begin{array}{c}\frac{\mathrm{d}G}{\mathrm{d}t}=-S_G\left(G(t)-G_b\right)-S_{I}X(t)G(t),G(22)=G_{\text{data}}(22)\hfill \\ \frac{\mathrm{d}F}{\mathrm{d}t}=-C_F(F(t)-F_b)-L_2\frac{X{(t)}^2}{X_2^2+X{(t)}^2},F(22)=F_{\text{data}}(22)\hfill \\ \frac{\mathrm{d}X}{\mathrm{d}t}=C_X(\max \{I(t)-I_b,0\}-X(t)),X(22)=X_{22}\hfill \end{array}$$ (1)

with parameter set $p=\{S_I,C_X,S_G,X_2,C_F,L_2\}$. $I(t)$ is obtained from the insulin data $\{I_{\text{data}}(t)\}$ by linear interpolation. A remote insulin compartment X is introduced to account for the observed delay in insulin action on glucose. The second term in the F equation is a lipolysis term, which was selected in Periwal et al. [16] from among 23 candidate models. It involves a Hill function. For this model, we set the initial time at $t=22$, the first post-injection measurement, so that fitting focuses on the period when insulin action produces more deviations of glucose and FFA from their baseline levels than before $t=20$. The details on $X_{22}$ are provided in the supplementary text.

Stefanovski et al. [15] proposed the following novel model of FFA kinetics to study insulin action on FFA lipolysis:

$$\{\begin{array}{c}\frac{\mathrm{d}F}{\mathrm{d}t}=-\alpha S_{\mathrm{F}}G(t)F(t)+F_b(S_{\mathrm{F}}-X(t)),F(0)=F_b,\hfill \\ \frac{\mathrm{d}X}{\mathrm{d}t}=P_{X\alpha }(I(t)-I_b)-P_{\mathrm{X}}X(t),X(0)=0,\hfill \end{array}$$ (2)

where glucose and insulin are both inputs (by linear interpolation of the discrete data over the same time course) to determine FFA clearance and lipolysis parameters and fit to the subject data from FSIGT. They specifically designed this model to quantitatively measure the sensitivity of FFA to insulin and oxidation actions and to estimate glucose contribution as a regulator of FFA oxidation. Here, $S_F$ is the fractional FFA disposal rate (1/min), $\alpha$ is a unit conversion and scaling factor for the effect of plasma glucose on the disposal of FFA, $P_{X\alpha }$ is a unit conversion and a fractional transfer rate with units $1/{\text{min}}^2$, and $P_{\mathrm{X}}$ is the fractional disposal rate of insulin from the remote compartment with units (1/min). They assume that the FFA plasma concentration at time 0 equals the fasting FFA concentration $F_b$.

For identifiability and to ensure that the model is linear-in-parameters, in our work, we reparametrize the system (2) as

$$\{\begin{array}{c}\frac{\mathrm{d}F}{\mathrm{d}t}=-S_{\text{GF}}G(t)F(t)+S_{\text{Fb}}-F_{b}X(t),F(0)=F_b,\hfill \\ \frac{\mathrm{d}X}{\mathrm{d}t}=P_{X\alpha }(I(t)-I_b)-P_{\mathrm{X}}X(t),X(0)=0,\hfill \end{array}$$ (3)

with parameter set $p=\{S_{\text{GF}},S_{\text{Fb}},P_{X\alpha },P_{\mathrm{X}}\}$, and will always refer to model (3) as the two-dimensional model in the remaining of this paper.

Method overview

Our method of inferring parameter values from physiological data requires simulated datasets for the training and testing of our neural network. As described in the introduction, each training and testing datum consists of a parameter set (the neural network output) and corresponding time-course data (the neural network input). Given physiological data from N subjects, traditional optimization techniques yield optimized parameter sets, resulting in N parameter-trajectory pairs. However, training a neural network requires a substantially larger number of such pairs. This requirement is general whenever real cohorts are limited, not just for our 25-subject example. Here, we introduce our methodology for generating simulated data for training purposes. It consists of the following steps, with details given in section “Generation of simulated data” and supplementary text:

1. Estimating physiologically relevant parameters: We apply the Nelder–Mead algorithm to estimate each subject’s ODE parameters so that the simulated glucose and FFA time‐courses closely match the experimental measurements. These physiologically constrained estimates define the prior distributions for generating simulated training data, ensuring the CNN is trained across the correct parameter space.

2. Generating physiologically realistic trajectories as inputs of ODE integrations via Gaussian Process Regression: Because models (1) and (3) require insulin (and in model (3), glucose) as time-dependent inputs, parameter samples alone are insufficient for simulated data generation. We therefore apply Gaussian process regression to sample insulin (and glucose) time‐courses that drive the ODE integrations, ensuring the CNN is trained on inputs that accurately reflect real physiological dynamics.

3. Generating parameter samples: Use the parameter sets obtained from the optimization (Step 1) to simultaneously generate sample parameter sets and initial conditions for the ODE model.

4. Generating simulated datasets for training and testing the proposed neural network.

In section “Primary convolutional network,” we outline the architecture and training process of our proposed neural network. The trained network is used to infer parameters from different input types, followed by an evaluation of the network and model performance. We outline the process here with details in section “Primary convolutional network” and supplementary text:

1. Feature engineering: Applied to the trajectory data to prepare the inputs for the training and testing datasets.

2. Training and testing of the network using the prepared datasets. The output of the neural network is linearly rescaled to [0, 1].

3. Evaluation of inference performance: The trained network is evaluated using three different input types: a separate testing dataset, model-fitting curves obtained from optimization, and real physiological data. The evaluation assesses the effectiveness of the model, feature engineering, and network architecture by comparing inference performance across these three cases.

Throughout this study, we consider subjects 3, 11, 18, and 25 as outliers and exclude their physiological data from the above procedures (supplementary text).

Generation of simulated data

Estimating physiologically relevant parameters

As introduced above, we apply the Nelder–Mead algorithm to estimate each subject’s ODE model parameters by fitting the simulated glucose and FFA trajectories to the experimental measurements. This initial parameter estimation yields parameter values constrained within physiological ranges and defines the prior distributions used to generate our CNN’s simulated training data. Although this step incurs a one‐time computational cost per subject, once our primary neural network is trained, parameter inference for any new subject requires only a single forward pass, fully amortizing the upfront effort.

For model (1), we perform the parameter estimation by minimizing the mean relative error between model and physiological glucose and FFA trajectories over post-20 time points. For model (3), only FFA errors are considered as the measured glucose and insulin values are inputs to the model. In both cases, parameters are constrained to be positive by taking absolute values at each iteration, and we employ the Nelder–Mead method because it does not require gradient information. The iteration proceeds until a uniform stopping criterion is met. Details and model-fitting plots are provided in supplementary text.

Generating insulin data as inputs of ODE integrations via Gaussian Process Regression

In this study, we generate simulated data [e.g. insulin for model (1)] similar to physiological data over the same time course, using Gaussian Process Regression (GPR) with a Radial Quasi-Periodic kernel for the 3D model (1) and a Radial Basis Function (RBF) kernel for the 2D model (3). Physiological data is normalized by its mean and standard deviation at each time point, and a GPR model is trained by iteratively maximizing the marginal log likelihood between the normalized data and the model’s evaluation. At each iteration, a random subject index i is selected, and the model is updated until the marginal log likelihood surpasses a predefined threshold. This trained model enables the generation of simulated data over the same time course, as illustrated in Fig. 1. For the 2D model (3), we generate glucose and insulin simultaneously using multi-task GPR to capture their temporal correlation. Other details are given in supplementary text.

Figure 1.

Confidence region of insulin reconstruction after $t=22$. The blue curve is the mean predicted insulin level. The black dots are the physiological data from all 21 subjects. The inset shows a zoomed-in view of the main plot, focusing on the time range from 40 to 180 and insulin levels between 0 and 20.

Simulating ODE model trajectories

We use model (1) as an example to illustrate the process of generating parameter sets for simulated data. To create the simulated datasets, we require a large number of parameter sets, each corresponding to an initial condition sample set. Our goal is to use deep learning tools to infer parameter values within their physiological range. To achieve this, we impose two requirements for admissible parameter–trajectory pairs:

1. The parameters must fall within their physiological ranges.

2. The trajectories must closely resemble the shape of the model fitting curves from optimization to ensure the trained network can infer parameters accurately, performing at least as well as the optimization process.

In the remaining sections of this paper, we treat $G_b$ and $F_b$ as parameters of model (1), and $F_b$ as a parameter of model (3). Notably, $I_b$ is the first value of the insulin data generated using Gaussian process regression (GPR) as described in section “Generation of simulated data,” so $I_b$ does not require sampling at this stage. The initial conditions also need to be generated with the parameter samples for the integration.

We apply the following steps to generate the simulated data:

1. For each parameter, we define its physiological range to be the closed interval from the minimal to the maximal value from optimization across all subjects, excluding outliers.

2. We fit the parameter sets from optimization using a multivariate log-normal distribution over the physiological range and generate as many parameter samples as required. This approach accounts for the correlations among different parameters.

3. We integrate each parameter set to obtain its corresponding trajectory. Notably, model (1) requires a time-dependent function $I(t)$. For each parameter set (along with its initial conditions), we randomly select an unused insulin time-course data from the pool generated using Gaussian process regression, as described in section “Generation of simulated data.” For the 2D model (3), both glucose and insulin trajectories are generated simultaneously using multi-task GPR, ensuring their temporal correlation. For the purpose of integration, the selected insulin (and glucose for the 2D model) data is linearly interpolated over its time course.

4. We filter out the parameter–trajectory pairs whose glucose or FFA trajectory contains negative values or do not satisfy the monotonicity condition we set. The remaining parameter–trajectory pairs serve as the training and testing data for our proposed primary neural network. The details are shown in supplementary text.

Figures 2 and 3 present histograms comparing the optimized parameter values (shown in red) with generated samples (shown in blue) for the 3D model (1) and the 2D model (3), respectively. In Figs 4 and 5, the simulated data are represented by blue symmetric vertical error bars, with a height equal to twice the standard deviation, for the 3D model (1) and 2D model (3), respectively. The red error bars represent the physiological data, while the black error bars represent the optimized data for both glucose and FFA in both models.

Figure 2.

The histograms of the optimized parameter values (red) and 10 000 generated samples (blue) for the 3D model (1).

Figure 3.

The histograms of the optimized parameter values (red) and 10 000 generated samples (blue) for the 2D model (3). Subject 4 is excluded in these histograms since its $S_{Fb}$ and $P_X$ values from optimizations are much larger than the other 20 subjects.

Figure 4.

The error bars for physiological data (red), model fitting curves (black), and simulated data (blue) for glucose (left) and FFA (right) over 21 subjects with the 3D model (1).

Figure 5.

The error bars for physiological data (red), model fitting curves (black), and simulated data (blue) for FFA over 21 subjects with the 2D model (3).

Primary convolutional network

In this study, we generate separate simulated training datasets for both the 3D model (1) and the 2D model (3) and train an independent primary convolutional neural network on each dataset, thereby demonstrating our method’s adaptability across different ODE structures for the same experimental data.

We aim to develop a convolutional neural network that has the following inputs and outputs:

• we introduce feature engineering in section “Generation of simulated data” and apply to the simulated data we generate to obtain the neural network input $d_I$, including time, glucose, FFA, and insulin information.

• the neural network output $d_O$ consists of parameter sets with a size of $(N_{\text{bs}},N_{\text{para}})$, where $N_{\text{bs}}$ is the batch size of training data and $N_{\text{para}}$ is the number of parameters to be inferred by the network.

We will introduce the detailed architecture of this network in section “Primary convolutional network,” Fig S7 in supplementary figures (see online supplementary material for a color version of this figure), and supplementary text.

The loss function of the neural network is defined as the mean squared error (MSE) of the model evaluation based on the given outputs. A learning rate schedule, determined by the maximal learning rate, the epoch at which it is reached, and the total number of training epochs (detailed in supplementary text), is used consistently for all network trainings in this study.

Feature engineering and neural network architecture

The neural network we aim to train is designed to accurately capture the nonlinear relationship between the input data $d_I$ (including $(t,G,I,F)$ information) and the output $d_O$(consisting of parameter values). According to the universal approximation theorem [3], a feedforward neural network can approximate any nonlinear function arbitrarily well within a compact region. However, in practice, the network’s ability to learn highly nonlinear relationships in raw data can be limited by computational costs and training efficiency. To overcome this challenge and improve network performance, we apply feature engineering, where we input nonlinear transformations of the $(t,G,I,F)$ data into the network instead of the raw data. In this section, we describe various features of inputs and evaluate their impact on enhancing neural network training and the accuracy of parameter inference once the training is complete.

Due to the lack of interaction between GFI (glucose, FFA, and insulin) data at temporally distant time points, we put two-dimensional convolutional (conv2D) layers in our network for each feature.

To illustrate whether feature engineering has a big influence on the training and inference of the neural network, we design several features with different $d_I$, but all from the same GFI data, as follows, with details shown in supplementary text:

1. No Feature Engineering: for data comprising k observed variables $(u_1,\dots ,u_k)$ at $N_{\mathrm{t}}$ discrete time points, the input tensor $d_I$ has shape $(N_{\text{bs}},k(k+1),N_{\mathrm{t}})$ with $k(k+1)$ channels covering each variable paired with time plus all mutual combinations, ordered as:

   $$\left(t,u_1,t,u_2,\dots ,t,u_k,u_1,u_2,u_1,u_3,\dots ,u_{k-1},u_k\right).$$ (4)

In this paper, we observe glucose, FFA, and insulin ($k=3$), so that $d_I$ has shape $(N_{\text{bs}},12,16)$ for the 3D model (1) and shape $(N_{\text{bs}},12,28)$ for the 2D model (3), with channels arranged as

$$\left(t,G,t,I,t,F,G,I,I,F,G,F\right)$$ (5)

for both models. $d_I$ is processed by two conv2D layers with max pooling and then flattened and passes through two dense layers. All other features in this list (concatenation, reciprocals vs. time, and mutual‐reciprocals) are specified for the 3D and 2D models in this paper and generalize directly to any dataset with k observed variables over $N_{\mathrm{t}}$ time points by including the corresponding rows.

1. Concatenation: the same $d_I$ as in Feature 1 (No Feature Engineering) is first processed by a conv2D layer, whose output is concatenated with $d_I$. With this feature, we put the derivatives of GFI with respect to time directly as the input of the network. The concatenated data then passes through two conv2D layers with max pooling, is flattened, and then fed into two dense layers as before. The full process is illustrated in Fig. S7 in supplementary figures (see online supplementary material for a color version of this figure).

2. Reciprocals vs. time: similar to feature 2 (Concatenation), except that $d_I$ has 18 rows by attaching the following six rows to the end of (5), involving the interactions between time and reciprocals of G, I, and F:

   $$\left(t,\frac{1}{G},t,\frac{1}{I},t,\frac{1}{F}\right).$$ (6)
3. Mutual reciprocals: Similar to feature 2 (Concatenation), except that $d_I$ has 24 rows by attaching the following 12 rows to the end of (5), involving the interactions between mutual pairs in $\{t,\frac{1}{G},\frac{1}{I},\frac{1}{F}\}$:

   $$\left(t,\frac{1}{G},t,\frac{1}{I},t,\frac{1}{F},\frac{1}{G},\frac{1}{I},\frac{1}{I},\frac{1}{F},\frac{1}{G},\frac{1}{F}\right).$$ (7)

The input and output sizes and the kernel size of the conv2D layers in features 3 and 4 change correspondingly.

It is important to note that these specific sequences of reciprocals of GFI and time are not arbitrary. The point is that convolution layers have a finite kernel size and we need to make sure that all engineered features are treated equally and independently. For instance, when a conv2D kernel with size $(2,2)$ and stride $(2,1)$ moves over the first two rows of (5), it enables the neural network to approximate the time derivatives of G through finite difference computation. These derivatives are incorporated as part of the input to better capture the data–parameter nonlinear relationship. Reciprocals are used since X, the remote insulin compartment, appears in the denominator of the F-equation in model (1). In all features described, the activation function for all dense layers is the hyperbolic tangent function ($\tanh$), except for the final layer, which uses either ReLU or $\frac{1}{2}(\tanh (x)+1)$ to ensure all parameters are positive before and after rescaling.

Training and evaluation of the inference

Now, we can start to evaluate the model and the network with the introduced tools. Our primary objective is to achieve high accuracy in parameter inference from optimized model fitting curves. To this end, we independently vary the following key factors:

1. Feature engineering: for the same simulated dataset, we apply the four feature types described in section “Generation of simulated data” to obtain different inputs of the primary network.

2. Activation function: the impact of using ReLU versus $\frac{1}{2}(\tanh (x)+1)$ as the final activation function is assessed.

3. Dataset size: we vary the size of training and validation datasets.

4. Training hyperparameters: the effects of different batch sizes and maximal learning rates are examined.

Therefore, for any given architecture, for each training dataset size, fixing batch sizes and maximal learning rate results in eight distinct scenarios. Each scenario yields a trained network, which is subsequently evaluated based on parameter inference accuracy.

For each trained model, we apply linear regression to compute the coefficient of determination ($R^2$) value and the P-value, providing a quantitative assessment of inference accuracy and statistical significance. These metrics are critical in determining the reliability of parameter estimations and guiding the selection of optimal network configurations.

Results

In this section, we evaluate the performance of the trained neural network in inferring physiological parameters. We first examine the convergence behavior of the training process and assess the inference accuracy over both the training and testing datasets, and the optimized model-fitting curves. Then, we analyze the impact of different feature engineering methods and training dataset sizes on inference performance. The results are systematically presented through validation loss curves, scatter plots of inferred versus true parameters, and comparisons of $R^2$ values across different configurations. Finally, we evaluate the reconstruction quality based on parameter inference from real physiological data.

Convergence of training and inference accuracy

The blue dots in Figs 6 and 7 illustrate the performance of parameter inference for eight selected parameters using scatter plots, over the training dataset ($N_{\text{train}}=500,000$) and testing dataset ($N_{\text{test}}=10,000$), respectively. Each subplot corresponds to a specific parameter (indicated in the titles), with true parameter values on the x-axis and inferred values on the y-axis. The dashed black diagonal line represents the ideal $y=x$ relationship.

Figure 6.

(a) Scatter plots showing parameter inference using the trained network for the 3D and 2D models on the training dataset. Each subplot corresponds to a specific parameter, with true parameter values on the x-axis and inferred values on the y-axis. The blue dots represent the inference results, while the black dashed diagonal line indicates the ideal y = x relationship. The $R^2$ values are put at the corners of each subplot. Both networks use $\left(\tanh \right(x)+1)/2$ activation and the “Reciprocals vs. time” feature. (b) Scatter plots showing parameter inference using the trained network for the 3D and 2D models on the training dataset. Each subplot corresponds to a specific parameter, with true parameter values on the x-axis and inferred values on the y-axis. The blue dots represent the inference results, while the black dashed diagonal line indicates the ideal y = x relationship. The $R^2$ values are put at the corners of each subplot. Both networks use $\left(\tanh \right(x)+1)/2$ activation and the “Reciprocals vs. time” feature.

Figure 7.

(a) Scatter plots showing parameter inference using the trained network for the 3D and 2D models on the testing dataset and from optimized model-fitting curves. The blue dots represent inference results on the testing dataset, while the magenta dots indicate inference from optimized glucose and FFA model-fitting curves using physiological insulin data. Each subplot corresponds to a specific parameter, with true parameter values on the x-axis and inferred values on the y-axis. The black dashed diagonal line represents the ideal y = x relationship. The $R^2$ values for both cases are put at the corners of each subplot. Both networks use $\left(\tanh \right(x)+1)/2$ activation and the “Reciprocals vs. time” feature. (b) Scatter plots showing parameter inference using the trained network for the 3D and 2D models on the testing dataset and from optimized model-fitting curves. The blue dots represent inference results on the testing dataset, while the magenta dots indicate inference from optimized glucose and FFA model-fitting curves using physiological insulin data. Each subplot corresponds to a specific parameter, with true parameter values on the x-axis and inferred values on the y-axis. The black dashed diagonal line represents the ideal y = x relationship. The $R^2$ values for both cases are put at the corners of each subplot. Both networks use $\left(\tanh \right(x)+1)/2$ activation and the “Reciprocals vs. time” feature.

To further evaluate the trained network’s ability to infer parameters from physiological data, we input optimized model-fitting curves of FFA and glucose into the trained network for each subject while using physiological insulin data directly. The optimized parameter values serve as the true values for comparison, and the resulting inference outcomes are plotted as magenta dots in Fig. 7.

The inference results over both the testing dataset and the optimized model-fitting curves demonstrate high accuracy for the 3D and 2D models. The consistently high $R^2$ values (shown in each subplot of Figs 6 and 7) indicate a strong correlation between true and inferred parameters, while the small P-values (they are all below ${10}^{-4}$, hence omitted from the figure annotations for better readability) confirm the statistical significance of the results. To assess whether the inferred parameters can effectively reconstruct the trajectories, we integrate the model using the inferred parameters and plot the resulting glucose and FFA curves in green in Figs S8 and S9 (see online supplementary material for a color version of these figures; also applies to S10 to S13 below), respectively. The close overlap between these reconstructed trajectories and the blue optimized model-fitting curves demonstrates that the trained network not only accurately infers parameters but also reproduces the model-fitting trajectories.

These results highlight the neural network’s capability to learn the relationship between input data and model parameters and to generalize effectively from simulated data to optimized model-fitting curves.

Having established the accuracy of parameter inference from both testing datasets and optimized model-fitting curves, we now investigate the factors that influence inference performance. In particular, we examine how different feature engineering strategies and varying training dataset sizes affect both validation loss and $R^2$ values. This analysis provides insights into how architectural choices impact the model’s learning efficiency and inference accuracy.

Effect of feature engineering and impact of training data size

To further quantify the effect of feature engineering and training dataset size, we compute the $R^2$ values across all inferred parameters. By comparing these values across different conditions, we can systematically assess how inference accuracy evolves as training progresses and whether incorporating additional features leads to more robust predictions.

Figures 8 and 9 present the validation loss curves over 2000 epochs for different feature engineering methods and training dataset sizes, for the 3D and 2D models, respectively. Each subplot corresponds to a specific training dataset size ($N_{\text{train}}$), increasing from left to right, with the top row using the ReLU activation function and the bottom row using the $(\tanh (x)+1)/2$ activation function. The four curves in each subplot represent different feature engineering strategies: No Feature Engineering (red), Concatenation (green), Reciprocals vs. Time (blue), and Mutual Reciprocals (black). Insets zoom in on the final 500 epochs to highlight convergence trends.

Figure 8.

Validation loss curves over 2000 epochs for different feature engineering methods and training dataset sizes for the 3D model. Each subplot corresponds to a specific training dataset size $\left(N_{\mathrm{train}}\right)$, increasing from left to right, with the top row using the ReLU activation function and the bottom row using the $\left(\tanh \right(x)+1)/2$ activation function. The four curves in each subplot represent the different feature engineering strategies: No Feature Engineering (red), Concatenation (green), Reciprocals vs. Time (blue), and Mutual Reciprocals (black). Insets zoom in on the final 500 epochs to highlight convergence trends.

Figure 9.

Validation loss curves over 2000 epochs for different feature engineering methods and training dataset sizes for the 2D model. The notations are similar to Fig. 8.

Figures 10 and 11 present the mean $R^2$ values as a function of training dataset size for the 3D and 2D models, respectively, under different feature engineering methods and activation functions. The upper two subplots in each figure show results over the same fixed testing dataset, while the lower two subplots show results from optimized model-fitting curves. The consistency of the testing dataset across models allows for a fair comparison of effect of feature engineering and training dataset sizes.

Figure 10.

Mean $R^2$ values for different feature engineering methods as a function of training dataset size in the 3D model. The upper two subplots show results over the same fixed testing dataset, while the lower two subplots show results from the optimized model-fitting curves. The left column corresponds to ReLU activation, while the right column corresponds to $\left(\tanh \right(x)+1)/2$ activation. Each color represents a different feature engineering method: No Feature Engineering (red), Concatenation (green), Reciprocals vs. Time (blue), and Mutual Reciprocals (black). The dashed lines connect the results at $N_{\mathrm{train}}=10000$ and $N_{\mathrm{train}}=500000$, illustrating the increasing trend in $R^2$ values as training dataset size grows.

Figure 11.

Mean $R^{\mathit{2}}$ values for different feature engineering methods as a function of training dataset size in the 2D model. The upper two subplots show results over the same fixed testing dataset, while the lower two subplots show results from the optimized model-fitting curves. The left column corresponds to ReLU activation, while the right column corresponds to $\left(\tanh \right(x)+1)/2$ activation. Each color represents a different feature engineering method: No Feature Engineering (red), Concatenation (green), Reciprocals vs. Time (blue), and Mutual Reciprocals (black). The dashed lines illustrate the improvement from $N_{\mathrm{train}}=10000$ to $N_{\mathrm{train}}=500000$, demonstrating that as the training dataset size increases, the mean $R^2$ values also improve across all feature engineering methods.

For $R^2$ values computed for each parameter across different features and training dataset sizes, Figs S10 and S11 (see online supplementary material for a color version of these figures) present results for the 3D model over the testing dataset and optimized model-fitting curves, respectively, while Figs S12 and S13 (see online supplementary material for a color version of these figures) provide the corresponding results for the 2D model. These figures offer a detailed view of how each individual parameter’s inference accuracy varies with training data size and feature engineering choices. All four figures use the $(\tanh (x)+1)/2$ activation function.

For the $(\tanh (x)+1)/2$ activation function, the improvement in $R^2$ values is more pronounced as more features are included. Across all dataset sizes, models using feature engineering techniques such as concatenation, reciprocals vs. time, and mutual reciprocals generally perform better than models without feature engineering. However, for smaller training dataset sizes, there are some fluctuations in performance, particularly for ReLU activation. Despite these variations, with a sufficiently large training dataset, the $R^2$ value consistently improves across almost all feature engineering methods, highlighting the significant impact of dataset size on inference accuracy. The dashed lines illustrate this substantial improvement as training dataset size increases.

Inference from physiological data

Add noise to simulated dataset

We augment the simulated input trajectories with structured noise derived from real physiological data. This narrows the gap between smooth ODE-generated simulations and irregular experimental measurements, allowing the neural network to generalize across realistic input variability while preserving ground-truth supervision. Taking the FFA trajectories of M time points as an example, we define the relative fluctuation at time point l for subject k as

$$\Delta F^{(k)}(l)=\frac{1}{|N(l)|}\sum _{m\in N(l)}\left|\frac{F^{(k)}(l)-F^{(k)}(m)}{F^{(k)}(l)+F^{(k)}(m)}\right|,$$ (8)

where $N(l)$ denotes the set of indices adjacent to l. We then compute the empirical standard deviation ${\sigma }_l$ at each time point by averaging $\Delta F^{(k)}(l)$ over all selected experimental subjects k. For each simulated input trajectory, noise is drawn from a zero-mean Gaussian with standard deviation ${\sigma }_l$ and scaled relative to the local amplitude of the FFA trajectory. The perturbed input is computed as

$$F_{\text{noisy}}(l)=F(l)+\alpha ·Y_l·\left[F(l-1)+F(l)\right],\hspace{1em}{Y}_l\sim \mathcal{N}(0,{\sigma }_l),\hspace{1em}l\ge 2,$$ (9)

with the first time point $F(1)$ left unperturbed. The scalar factor $\alpha$ specifies the noise intensity: we use $\alpha =1.0$ for FFA in both 3D and 2D models and $\alpha =0.5$ for glucose in the 3D model. Figure S14 (see online supplementary material for a color version of this figure) illustrates examples of simulated trajectories with added noise to FFA in both models and to glucose in the 3D model.

Parameter inference and trajectory reconstruction from experimental inputs

To assess the trained model on real physiological inputs, we train it using the noise-augmented simulated dataset described above, preparing the network to handle the variability inherent in experimental data. The architecture remains the same as in previous subsections except that the input training set is replaced by the noise-augmented version. After training, we apply the network to infer parameters from experimental measurements and model-fitting curves in the 3D model and reconstruct the corresponding trajectories. Figure 12 shows the experimental data in red, traditional model fits in blue, and the reconstructions based on inferred parameters in cyan (from experimental data) and green (from model fits). The network’s reconstructions from model fits closely align with the blue curves, confirming that it reliably infers parameters from smooth inputs. Importantly, the reconstructions from experimental inputs often better match the real measurements than those from optimization, suggesting the CNN can infer physiologically meaningful parameters directly from experimental inputs within the constraints of the ODE model.

Figure 12.

(a) Reconstruction of FFA (top) and glucose (bottom) trajectories from parameter inference using a trained CNN for the 3D model (1). Experimental data are shown in red, and model fits in blue. Using these as inputs along with physiological insulin, the network infers parameters that reconstruct the trajectories shown in cyan (from experimental data) and green (from model fits). (b) Reconstruction of FFA (top) and glucose (bottom) trajectories from parameter inference using a trained CNN for the 3D model (1). Experimental data are shown in red, and model fits in blue. Using these as inputs along with physiological insulin, the network infers parameters that reconstruct the trajectories shown in cyan (from experimental data) and green (from model fits).

To evaluate the effect of feature engineering, we trained four different networks one for each input feature, all using $(\tanh (x)+1)/2$ activation function. For the 3D model, the reconstruction error on FFA remains similar across features: relative errors of 0.211, 0.247, 0.232, and 0.257, respectively. Thus, feature engineering does not noticeably improve inference from physiological inputs in this setting. In this case, it is the noise augmentation—not feature design—that plays the more critical role in improving generalization to real-world data.

For the 2D model, we address the imbalance in temporal resolution before and after $t=30$ min by applying a temporal dropout layer to the inputs. During training, each early time point ($t<30$) is dropped with probability 0.5, encouraging the model to avoid overfitting to densely sampled early-phase data. Without this dropout, the network tends to reproduce early dynamics well but struggles at later times. This intervention improves generalization across the full time course and enhances parameter inference performance. Figure 13 presents the reconstructed trajectories for the 2D model using the same color scheme as in Fig. 12 for consistency.

Figure 13.

Reconstruction of FFA trajectories from parameter inference using a trained CNN for the 2D model (3). Experimental data are shown in red, and model fits in blue. Using these as inputs along with physiological glucose and insulin, the network infers parameters that reconstruct the trajectories shown in cyan (from experimental data) and green (from model fits).

Conclusion

The results presented in this section demonstrate the effectiveness of the trained neural network in accurately inferring physiological parameters. Evaluation across the training and testing datasets, the optimized model-fitting curves, and the real physiological data confirm the robustness of the approach. For simulated and model-fitted data, the network consistently achieves high $R^2$ values and statistically significant P-values, indicating strong inference accuracy. For physiological data, where ground-truth parameters do not exist, the validity of the inference is supported by the quality of the reconstructed trajectories, which—under the 3D model—often align more closely with the experimental measurements than those generated from traditional Nelder–Mead optimization.

Further analysis of feature engineering methods and training dataset sizes highlights key factors influencing model performance. The validation loss curves reveal that feature engineering generally improves training efficiency, particularly when using the $(\tanh (x)+1)/2$ activation function. The comparison of $R^2$ values across different configurations shows that models trained with feature engineering methods, such as mutual reciprocals and reciprocals vs. time, tend to achieve better inference accuracy than models without feature engineering. While performance fluctuations are observed for smaller training datasets, the overall trend indicates that increasing the dataset size leads to improved inference accuracy across all feature engineering methods. Notably, with a sufficiently large training dataset, the $R^2$ values consistently reach higher levels, demonstrating the importance of dataset size in achieving robust parameter inference.

Additionally, inference results from the optimized model-fitting curves further validate the network’s ability to generalize beyond simulated data. The close match between the reconstructed trajectories and the original optimized model-fitting curves confirms that the inferred parameters are not only accurate but also capable of reproducing the expected physiological dynamics.

We also extend our framework to real physiological measurements by directly applying the trained neural network—trained entirely on synthetic data with added noise—to experimental glucose and FFA trajectories. Without subject-specific tuning or denoising, the network produces one-shot parameter estimates that, when used in the 3D model, yield reconstructed trajectories closely matching the real data for most subjects. Notably, the reconstructions from inferred parameters often outperform those from traditional optimization. Furthermore, when given model-fitting curves as input, the network accurately recovers the parameters used to generate them, indicating that its inference capability is constrained primarily by the expressivity of the ODE model rather than the inference procedure itself.

These findings highlight the strength of the proposed neural network approach in parameter inference, demonstrating its potential for broader applications in modeling complex physiological systems. Extending this framework to other physiological models will require tailored feature engineering strategies based on model-specific characteristics, as well as training with appropriately perturbed synthetic data to improve robustness. In this work, we add noise to simulated input trajectories to mimic measurement uncertainty, allowing the network to learn parameter inference from realistically noisy inputs. Future work will focus on improving inference accuracy and further enhancing generalization by applying the framework to additional FSIGT datasets.

Discussion

We have presented a CNN-based framework for inferring parameters in physiological ODE models, offering a fast and accurate alternative to traditional optimization methods. By training the network on simulated trajectories with noise structured from experimental data, the model learns to produce robust point estimates of physiological parameters—even when applied to real-world inputs. In section “Results,” we showed that the inferred parameters yield trajectories that closely match both optimized model-fitting curves and experimental measurements, validating the network’s generalization capability.

Traditional optimizers (e.g. Nelder–Mead or BFGS) must evaluate the model repeatedly for each new dataset and often require careful tuning of bounds or gradients. (See supplementary text for detailed computational cost of the Nelder–Mead method on the 3D model.) In contrast, once our CNN is trained, it provides one-shot predictions without any extra tuning or sensitivity to starting conditions.

Importantly, we keep the method tied to the underlying biology by building ODE-inspired inputs (e.g. time derivatives, reciprocal terms) and predicting the original model parameters. This makes our approach transparent and interpretable. It also enables the detection of parameter non-identifiability: in cases where multiple parameter sets yield similar output trajectories, the network may infer alternative combinations that still reconstruct the dynamics accurately. Even when inferred parameters deviate from expected values, the resulting trajectories often remain close to experimental data. This behavior highlights the network’s potential as a diagnostic tool for identifying structural limitations and guiding refinement of physiological models.

Beyond parameter inference, our framework can also facilitate model selection among competing ODE formulations. For each candidate model, a separate CNN is trained using simulation-based data generation tailored to that model’s structure. Although training each network is computationally intensive, once trained, parameter inference and trajectory reconstruction require only a single forward pass. For simulated or model-fitted data, accuracy can be evaluated using standard metrics such as mean squared error or $R^2$ values; for real physiological data, where true parameters are unknown, models are instead ranked by how well their reconstructions match the experimental time courses. Additionally, we can assess the stability of the inferred parameters under small perturbations in the input data, providing insight into each model’s identifiability. Models yielding both low reconstruction error and stable parameter estimates are more likely to be valid representations of the underlying physiology.

Our simulation-driven strategy, which utilizes a trained convolutional neural network to provide one-shot parameter estimates, shares a similar philosophy with Simulation-Based Inference (SBI) methods. Both approaches fundamentally shift the computational burden of parameter estimation from repetitive, real-time optimization to an upfront, offline training phase. This amortization of computational effort enables rapid inference for new experimental data without relying on iterative optimizers like Nelder–Mead or BFGS. However, a key distinction lies in the nature of the output: many SBI methods are designed to infer full posterior parameter distributions, providing comprehensive uncertainty quantification. In contrast, our method delivers a single, highly accurate point estimate. While this sacrifices posterior-level insight, it offers a clear, actionable set of parameters that can be interpreted and applied directly by clinicians and researchers, without the need for further post-processing or statistical interpretation.

Looking ahead, the strong performance on physiological data suggests promising directions for broader application. Future work will focus on improving accuracy and expanding generalization across diverse FSIGT datasets. This includes extending the framework to other models, refining noise modeling strategies, and incorporating lightweight uncertainty quantification to complement fast, deterministic inference.

In summary, our approach shifts the computational burden of parameter estimation to an offline training phase and delivers rapid, biologically grounded predictions on experimental data. It offers a practical tool for accelerating parameter inference in complex physiological modeling.

Supplementary data

Supplementary data is available at Biology Methods and Protocols online.

Conflict of interest statement. The authors declare no competing interests.

Funding

This research was supported by the Intramural Research Program of the National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK) within the National Institutes of Health (NIH). (Project 1 ZIA DK013030-16. V.P. received this grant.)

Data availability

The code for data generation, neural network training, and evaluation is available at GitHub at https://github.com/nihcompmed/LipolysisCNN.