Parameter Subset Selection Techniques for Problems in Mathematical Biology

Abstract

Patient-specific models for diagnostics and treatment planning require reliable parameter estimation and model predictions. Mathematical models of physiological systems are often formulated as systems of nonlinear ordinary differential equations (ODEs) with many parameters and few options for measuring all state variables. Consequently, it can be difficult to determine which parameters can reliably be estimated from available data. This investigation highlights pitfalls associated with practical parameter identifiability and subset selection. The latter refer to the process associated with selecting a subset of parameters that can be identified uniquely by parameter estimation protocols. The methods will be demonstrated using five examples of increasing complexity, as well as with patient specific model predicting arterial blood pressure. This study demonstrates that methods based on local sensitivities are preferable in terms of computational cost and model fit when good initial parameter values are available, but that global methods should be considered when initial parameter value are not known or poorly understood. For global sensitivity analysis, Morris screening provides results in terms of parameter sensitivity ranking at a much lower computational cost.

1. Introduction

The use of mathematical models allows for prediction of hidden dynamics, making it possible to test hypotheses in silico. It is desirable that models be as simple as possible, but not so simple that they fail to predict relevant dynamics. Ideally, models should not contain more parameters than required, and it should be possible to determine parameters uniquely from available data.

For an input-output relation y = f(θ), a set of inputs; e.g. parameters, initial or boundary conditions, θ = [θ₁, … , θ_p] are said to be identifiable at θ* if f(θ) = f(θ*) implies that θ = θ* for all admissible θ [45, 9]. Alternatively, if multiple parameter configurations yield the same output, the parameters are said to be unidentifiable [4, 26].

Parameters that minimally affect outputs when varied across the admissible space are broadly termed noninfluential. Specifically, parameters θ are defined as noninfluential on the space NI(θ) if |f(θ) − f(θ*)| < ε for all {θ, θ*} ∈ NI(θ). The space of influential parameters I(θ) is defined as the orthogonal complement of NI(θ) [38]. The challenge of identifiability associated with noninfluential parameters may be enhanced when large parts of the system modeled is hidden for observation [4, 5, 31].

Parameters that do not uniquely influence the model output may stem from the model structure regardless of the parameter configuration and input [6, 26], but may also arise due to restrictions in parameter values, model inputs, or outputs [16]. Following the definition from Bellman and Åström [6], parameters that are unidentifiable due to the model structure are referred to as structurally unidentifiable, whereas parameters that are unidentifiable due to practical restrictions, such as availability of data or physical restrictions, are referred to as practically unidentifiable [16].

Many complex models rely on numerical methods for finding solutions, which in itself brings about challenges associated with identifiability of the solution due to numerical precision, problem discretization, and how these may change with model parametrization. For such problems it is interesting to study how identifiability challenges arises both from the structure of the model, but also from the limitations brought about by parametrization, data, model implementation, etc. This study focuses on methods for practical identifiability, which are based on numerical methods, rather than structural identifiability, which is often based on an algebraic approach. We refer readers to studies by others, e.g. Eisenberg et al. [11], Mahdi et al. [23], and Transtrum et al. [42] for works addressing structural identifiability from an algebraic perspective.

To illustrate challenges associated with identifiability, this investigation reviews a number of methods for determining which parameters have no effect on the model output and sets of parameters that may have a redundant effect on the model output. Methods assessing how variations (uncertainty) in parameters affect the model output are referred to as sensitivity analysis (SA), whereas methods for additionally investigating interactions between parameters are referred to as identifiability analysis (IA). Figure 1 illustrates the relationship between the properties: Influential (sensitivity), structural, and practical identifiable. While it is rather obvious that not all influential parameters are structurally identifiable, it is less obvious why parameters can be practical but not structural identifiable. One example is highly - but not perfectly - correlated parameters. They can be impossible to estimate in practice with real data, while the algebraic structure of the model denote the parameters identifiable. On the other hand practical identifiability requires that the parameters be both structurally identifiable and influential.

Fig. 1.

Relationship between influential, structural, and practical identifiable parameters. Note that an influential and structurally identifiable parameter may not necessarily be practically identifiable, whereas a practically identifiable parameter is both influential and structurally identifiable.

This study presents three methods for quantifying sensitivities: derivative-based (local) sensitivities, Sobol indices [39], and Morris elementary effects [27]. In addition, the Structural Correlation Method (SCM) [9, 28] and Orthogonal Sensitivities Method (OSM) [22] are included to demonstrate practical identifiability and subset selection. These methods use statistical identifiability methods to perform subset selection and com pare model fit for subsets recovered using each method. To investigate the efficiency of these methods, we apply them to a variety of examples of increasing complexity and with different identifiability challenges. The first example illustrates structurally unidentifiable parameters, the second discusses identifiability problems materializing due to sparse data, and the third, practical problems arising in a simple viscoelastic model with structurally identifiable parameters. Finally, the proposed methods are applied to analyze a more complex model predicting systemic arterial blood pressure in humans.

2. Methods

We present concepts and methods in the context of a system of ordinary differential equations defined by

$$\begin{array}{ll}\dot{x}(t)\hfill & =f(t,x(t),u(t),\theta )\hfill \\ y\left(t_i\right)\hfill & =h\left(x\left(t_i\right),u\left(t_i\right),\theta \right),\hfill \end{array}$$ (1)

where x is the model state vector, t is the independent variable, u(t) is the model input, θ is the parameter vector, and h(x, u, θ) is the observation function that relates the model states to the observable model output y(t_i) at t = t_i.

2.1. Sensitivity analysis

Local sensitivity analysis refers to analysis that quantifies how the model output changes with changes in parameter values. Global sensitivity analysis more broadly quantifies how variability or uncertainties in responses can be apportioned to variability, or certainties in parameters considered throughout the admissible parameter space. We discuss three different methods. Derivative-based (local) sensitivities quantify the local influence of each parameter on the model output computed from the partial derivative of the model output with respect to each model parameter. This method is local in the sense that it is evaluated at a nominal set of parameter values. In addition, we present two global methods computing sensitivities over a parameter space: Sobol indices [39] and Morris elementary effects [27]. All methods presented here are chosen to study practical identifiability analysis, and rely on numerical analysis of the model response. These are chosen as they reflect problems arising in analysis of real models composed of highly-nonlinear systems, where numerical methods are required to obtain solutions. The response displayed by such models are greatly dependent on the parameter values. Therefore, practical methods for analyzing identifiability will also greatly depend on reasonable initial guess for parameter values and ranges. Given that models typically are built using physical principles we expect that reasonable parameter range can be obtained from a priori knowledge of system behavior, such as steady-state or mean values.

2.1.1. Derivative-based (local) sensitivities

Local derivative-based sensitivities $S_{{\theta }_i}^{\text{y}}(t)$ can be computed by differentiating the model output y(t) with respect to the parameters θ_¡ evaluated at Θ,

$$S_{{\theta }_i}^y(t,\theta )=\frac{\partial y(t,\theta )}{\partial {\theta }_i}.$$ (2)

For some problems, these can be calculated analytically, but for more complex problems it is advantageous to numerically approximate local sensitivities. Local sensitivities are typically computed using sensitivity equations, finite differences or automatic differentiation [30,38].

The sensitivity equations method utilize the fact that the sensitivities of the model states to parameter values $S_{{\theta }_i}^x=\frac{\partial x}{\partial {\theta }_i}$ can be computed as solutions to the ode

$$\begin{array}{lll}\frac{d}{dt}\left(\frac{\partial x}{\partial {\theta }_i}\right)\hfill & =\hfill & \frac{df}{d{\theta }_i}+\frac{df}{dx}\frac{\partial x}{\partial {\theta }_i}\hfill \\ \hfill \frac{d{S}_{{\theta }_i}^x}{dt}& =\hfill & J_{{\theta }_i}+J_x{S}_{{\theta }_i}^x,\hfill \end{array}$$ (3)

obtained by differentiating the model $\frac{dx}{dt}=f(t,x,\theta )$ with respect to the parameters θ_¡. Here $J_{\theta }=\frac{\partial f}{\partial \theta }$ denotes the parameter Jacobian, and $J_x=\frac{\partial f}{\partial x}$ the model Jacobian, which can be calculated analytically. Subsequently, the sensitivities of the model output $S_{{\theta }_i}^{\text{y}}$ can be calculated by application of the chain rule on the output function y = h(x, t, θ).

Alternatively, the derivative with respect to θ_i can be approximated using the finite-difference scheme

$${\widehat{S}}_{{\theta }_i}^y(t)=\frac{y\left(t,\theta +h{e}_i\right)-y(t,\theta )}{h},$$

Here the step size h is chosen to reflect the precision of the model output, and e_¡ is the unit vector in the i’th component direction. If the error in the model evaluation is on the order of ε, the step size should be h = √ε to obtain an error of same magnitude in the sensitivities [29]. Alternatively, one can avoid underflow due to subtraction by employing complex sums [25].

Finally, automatic differentiation (AD) is a third option that uses the chain rule while solving the differential equations by extending the basic numerical operations to include calculation of the derivative through recursive and automatic application of the chain rule [14]. AD routines have been developed for variety of programming languages such as C/C++, Fortran, Matlab and Python [38], and are now included in optimization packages and modeling frameworks such as ACADO [17] and CasADi [3]. Accessibility of AD has improved tremendously in recent years, and offers a tool for differentiation that ensures precision in the derivatives without the hassle of adjusting the step sizes or calculating the model Jacobian analytically [13].

2.1.2. Sobol sensitivity indices

This method was originally developed by Sobol [41] but several variations exists [33]. The description of the Sobol sensitivity indices and the algorithm for calculating first-order effects follows that of Smith [38], but with modifications to reflect the algorithm presented by Jansen [20].

Consider a scalar-valued nonlinear model

$$Y=f(\Theta )$$ (4)

with parameters represented by the vector of random variables Θ = [Θ₁, … , Θ_p]. Here ${\Theta }_i\in {\Gamma }_i\subset ℝ$, where Γ_¡ is the domain of the random variable Θ_i. The joint probability density function is defined as ${\rho }_{\Theta }(\theta )={\prod }_{k=1}^p{\rho }_{{\Theta }_k}\left({\theta }_k\right)$, where ${\rho }_{{\Theta }_k}\left({\theta }_k\right)$ denotes the marginal probability density function. For mutual independent variables, the joint distribution is simply the product of the marginal densities.

To avoid scaling issues, we transform parameters to Γ_k = [0, 1]. We assume that they are uniformly distributed to avoid inadvertent biases [10].

Consider the model decomposition

$$f(\theta )=\sum _{\mathbf{\text{i}}\subseteq \{1,2,\dots ,p\}}{f}_{\mathbf{\text{i}}}\left({\theta }_{\mathbf{\text{i}}}\right)$$ (5)

where f is the model output, i = {i₁, … , i_s} is the parameter set with cardinality s, ${\theta }_i=\left\{{\theta }_{i_1},\dots ,{\theta }_{i_s}\right\}$ and f_∅ = f₀ [40, 38]. If

$${\int }_0^1{f}_i\left({\theta }_i\right){\rho }_k\left({\theta }_k\right)d{\theta }_k=0,$$

for any parameter θ_k and i = {1, … , s} that includes k, the decomposition is unique and the component functions are given by

$$f_{\mathbf{\text{i}}}\left({\theta }_{\mathbf{\text{i}}}\right)={\int }_{{[0,1]}^{p-s}}f(\theta ){\rho }_{\theta }\left({\theta }_{~i}\right)d{\theta }_{~i}-\sum _{\begin{array}{l}{l}^{\prime }\subset i\hfill \\ l^{\prime }\ne i\hfill \end{array}}{f}_{l^{\prime }}\left({\theta }_{l^{\prime }}\right).$$

To determine the effect of the parameter interaction in the set i, consider the conditional partial variance

$$D_i={\int }_{{[0,1]}^s}{f}_i^2\left({\theta }_i\right){\rho }_{\theta }\left({\theta }_i\right)d{q}_i=\text{var}[\mathbb{E}(Y|{\theta }_i)]-\sum _{\begin{array}{c}{l}^{\prime }\subset i\\ l^{\prime }\ne i,l^{\prime }\ne \varnothing \end{array}}{D}_{l^{\prime }},$$

and the total variance

$$D={\int }_0^1{f}^2(\theta ){\rho }_{\Theta }(\theta )d\theta -f_0^2=\sum _{\begin{array}{c}i\subseteq \{1,2,\dots ,p\}\\ i\ne \varnothing \end{array}}{D}_i.$$

The Sobol indices are defined as

$$S_i=\frac{D_i}{D}\text{giving}\sum _{\begin{array}{c}i\subseteq \{1,2,\dots ,p\}\\ i=\varnothing \end{array}}{S}_i=1.$$ (6)

The total sensitivity index

$$S_{T_i}\equiv \sum _{k_{\text{i}}i\in k}{S}_k$$ (7)

describes the sensitivity of the model output variance to the parameter k, including the effects of interactions with other parameters.

A Monte Carlo approach for estimating the first-order indices S_¡ (6) and total indices $S_{T_i}$ (7) requires M² model evaluations for each parameter, where M is the number of points in the parameter space used to evaluate the conditional mean $\mathbb{E}(Y|{\theta }_i)$; see Smith [38]. Saltelli et al. [33] proposed an algorithm that reduces the number of model evaluations required to M(p + 2), whereas Algorithm 1, used in our study, is due to Jansen [20] and requires 2M(p + 2) evaluations for a better trade-off between precision and least squares error. It should be noted that these algorithms truncate the variance decomposition after 2nd order. If higher-order interactions are present in the model, these algorithms may yield incorrect estimates.

2.1.3. Morris elementary effects

Morris screening computes the sensitivity of the model output to each parameter θ_i by sampling the elementary effects [27,38]

$$d_i(\Theta )=\frac{f\left(\Theta +\Delta e_i\right)-f(\Theta )}{\Delta },$$

from a grid with points ${\Gamma }_{\ell }=\left\{0,\frac{1}{\ell -1},\dots ,1\right\}$ in the parameter distribution range Γ = {0, 1}. The number of points (in each dimension) of the grid and their value Δ is important as they determine the sampling of the parameter space. By choosing ℓ to be even and $\Delta =\frac{\ell }{2(\ell -1)}$, it is possible to ensure an even probability sampling from Γ_ℓ that guarantees equal probability for all points in the grid [38]. Similar to the Sobol indices f denote the scalar-valued nonlinear model given in (4).

By estimating the mean and variance of the distribution |d_i(θ)|, for which

$$\begin{array}{lll}\hfill u_i^{*}& =\hfill & \frac{1}{r}\sum _{j=1}^r|d_i^j(\theta )|,\hfill \\ \hfill {\sigma }_i^{2*}& =\hfill & \frac{1}{r-1}\sum _{j=1}^r{\left(d_i^j(\theta )-{\mu }_i\right)}^2,\hfill \\ \hfill {\mu }_i& =\hfill & \frac{1}{r}\sum _{j=1}^r{d}_i^j(\theta )\hfill \end{array}$$

we obtain an estimate of the sensitivity noting large local changes attributed to parameter θ_i.

The implementation presented in Algorithm 2 creates a trajectory through the grid in the parameter space, such that for each step in the trajectory, just one parameter is changed. Creating such a trajectory one elementary effect sampling

Algorithm 1. Sobol indices.

Using 2M model evaluations for each of the p parameters:

1. Create two M × p sample matrices

   $$A=\left[\begin{array}{ccccc}{\theta }_1^1& \cdots & {\theta }_i^1& \cdots & {\theta }_p^1\\ ⋮& & & & ⋮\\ {\theta }_1^M& \cdots & {\theta }_i^M& \cdots & {\theta }_p^M\end{array}\right],B=\left[\begin{array}{ccccc}{\widehat{\theta }}_1^1& \cdots & {\widehat{\theta }}_i^1& \cdots & {\widehat{\theta }}_p^1\\ ⋮& & & & ⋮\\ {\widehat{\theta }}_1^M& \cdots & {\widehat{\theta }}_i^M& \cdots & {\widehat{\theta }}_p^M\end{array}\right],$$

   where ${\theta }_i^j$ and ${\widehat{\theta }}_i^j$ are parameter samples drawn from the distribution corresponding to parameter i, for each sampled parameter configuration j = 1,…,M. In addition, create the matrix

   $$C={[AB]}^T,$$

   where T denotes transpose.
2. Create two M × p matrices

   $$A_B^i=\left[\begin{array}{ccccc}{\theta }_1^1& \cdots & {\widehat{\theta }}_i^1& \cdots & {\theta }_p^1\\ ⋮& & & & ⋮\\ {\theta }_1^M& \cdots & {\widehat{\theta }}_i^M& \cdots & {\theta }_p^M\end{array}\right],B_A^i=\left[\begin{array}{ccccc}{\widehat{\theta }}_1^1& \cdots & {\widehat{\theta }}_i^1& \cdots & {\widehat{\theta }}_p^1\\ ⋮& & & & ⋮\\ {\widehat{\theta }}_1^M& \cdots & {\widehat{\theta }}_i^M& \cdots & {\widehat{\theta }}_p^M\end{array}\right],$$
3. Compute the model output vectors

   $$y_A=f(A),y_B=f(B),y_{A_B^i}=f\left(A_B^i\right),y_{B_A^i}=f\left(B_A^i\right),$$

   nothing that

   $$y_C={\left[y_A{y}_B\right]}^T.$$
4. Estimates of the first order sensitivity indices can be computed by

   $$\begin{array}{lll}{S}_i^{*}\hfill & =\frac{\frac{1}{M}{\sum }_{j=1}^M{y}_A^j{y}_{B_A^i}^j-y_A^j{y}_B^j}{\frac{1}{2M}{\sum }_{j=1}^M{y}_C^j{y}_C^j-{\left(\frac{1}{M}{\sum }_{j=1}^M{y}_C^j\right)}^2}\hfill & =\frac{\frac{1}{M}{\left(y_A\right)}^T\left(y_{B_A^i}-y_B\right)}{\frac{1}{2M}{\left(y_C\right)|}^T{y}_C-{\left(\frac{1}{M}{\sum }_{j=1}^M{y}_C^j\right)}^2},\hfill \end{array}$$

   $$\begin{array}{lll}{S}_{T_i}^{*}\hfill & =\frac{\frac{1}{M}{\sum }_{j=1}^M{\left(y_A^j{y}_{A_b^i}^j\right)}^2}{\frac{1}{2M}{\sum }_{j=1}^M{y}_C^j{y}_C^j-{\left(\frac{1}{M}{\sum }_{j=1}^M{y}_C^j\right)}^2}\hfill & =\frac{\frac{1}{M}{\left(y_A-y_{A_B^i}\right)}^T\left(y_A-y_{A_B^i}\right)}{\frac{1}{2M}{\left(y_C\right)}^T{y}_C-{\left(\frac{1}{M}{\sum }_{j=1}^M{y}_C^j\right)}^2}.\hfill \end{array}$$

can be obtained for each of the p parameters using p + 1 function evaluations [27, 38].

It should be noted that both the Morris- and Sobol based methods are based on the assumption of mutually independent parameters and can yield inaccurate results for correlated parameters. Whereas extensions to Sobol theory exist for correlated parameters, they require knowledge of the underlying distribution and can yield negative indices that are difficult to interpret.

2.2. Parameter interactions

We employ two methods for constructing parameter subsets with no parameter interactions using the Structural Correlation Method (SCM) and the Orthogonal Sensitivities Method (OSM). These methods are investigated. Both methods are

Algorithm 2. Morris Indices.

For each j = 1,…,r:

1. Sample an initial parameter vector θ^* from the uniform distribution $\mathcal{U}{[0,1]}^p.$

2. Create a (p + 1) × p permutation matrix B^* where each row represents a parameter configuration. A completely deterministic permutation matrix is given by

   $$B_j^{*}=J_{p+1,p}{\theta }^{*}+\Delta B,$$

   where J_p+1,p is a (p + 1) × p matrix of ones, Δ is the chosen grid step size, and B is a lower triangular matrix of ones. Each row in B^* differs in exactly one column, corresponding to the parameter for which the step is made. A random permutation matrix can be obtained by applying the orientation matrix

   $$B_j^{*}=\left(J_{p+1,p}{\theta }^{*}+\frac{\Delta }{2}\left[\left(2B-J_{p+1,p}\right)D^{*}+J_{p+1,p}\right]\right)P^{*},$$

   where the p × p matrix D^* is diagonal with elements randomly chosen from the set {−1, 1}, and the p × p matrix P^* is created by perturbing the columns of a p × p identity matrix.
3. Evaluate the model output for each of these p + 1 configurations, in the (p + 1) × 1 vector

   $$y_j=f\left(B_j^{*}\right).$$
4. Through analysis of B^* determine which two consecutive rows k and k + 1 that corresponds to each parameter i, and calculate the corresponding elementary effect,

   $$d_i^j=\frac{y_{j,k+1}-y_{j,k}}{\Delta }.$$
   Finally, the sampling mean ${\mu }_i^{*}$ and variance ${\sigma }_i^{2*}$ for each parameter i can be estimated by

   $$u_i^{*}=\frac{1}{r}\sum _{j=1}^r\left|d_i^j(\theta )\right|,$$

   $${\sigma }_i^{2*}=\frac{1}{r-1}\sum _{j=1}^r{\left(d_i^j(\theta )-{\mu }_i\right)}^2,$$

   $${\mu }_i=\frac{1}{r}\sum _{j=1}^r{d}_i^j(\theta ).$$

based on the linearization of model output, which can be obtained without significant computational cost. Other methods, including the profile likelihood approach by Raue et al. [31], can provide more insight into parameter interactions, but crequire a significantly higher computational cost. Both the SCM and the OSM methods are based on analysis of the Fisher Information Matrix (F) given by

$$F=S^{T}S,$$ (8)

where S is the sensitivity matrix defined in (2), evaluated at a nominal parameter value.

For least-squares optimization problems, the goal is to determine a parameter vector θ^* that minimizes the objective function

$$\text{SS}=\frac{1}{2}\sum _{i=0}^{n-1}{r}_i^2,r_i=v_i-y_i,$$ (9)

where r_i are the residuals, i.e. v_i the measured data and y_i = f(t_¡) the ith output of the function. For this optimum, the gradient vector is zero; i.e. g(θ^*) = 0. The second-order Taylor expansion of the objective function at the optimum point θ^* in the parameter space is given by

$$SS(\theta )=SS\left({\theta }^{*}\right)+{\Delta }^{T}g+{\Delta }^{T}H\Delta +\mathcal{O}\left({\Delta }^3\right),$$ (10)

where Δ = (θ − θ*), g is the gradient

$$g_j=\frac{\partial SS}{\partial {\theta }_j}=\sum _{i=0}^{n-1}{r}_i\frac{\partial r_i}{\partial {\theta }_j}$$

and H is the Hessian – a symmetric matrix with second-order partial derivatives – of the objective function with entries

$$H_{jk}=\frac{\partial g_j}{\partial {\theta }_k}=\sum _{i=0}^{n-1}\frac{\partial r_i}{\partial {\theta }_k}\frac{\partial r_i}{\partial {\theta }_j}+r_i\frac{{\partial }^2{r}_i}{\partial {\theta }_j\partial {\theta }_k}.$$

Ignoring higher-order terms of the Taylor expansion in (10), and re-arranging terms yields

$$\text{SS}(\theta )=SS\left({\theta }^{*}\right)+{\Delta }^{T}H\Delta$$

since g(θ^*) ≈ 0. If H is singular there exists some Δ ≠ 0, such that HΔ = 0, which in turn means that SS(θ) = SS(θ*), and thereby, that the minimizer of the objective function is not unique and some parameters are unidentifiable.

When r is small or when r varies linearly near a minimum of (9), a reasonable approximation to the Hessian is obtained by neglecting the second-order term [47]

$$H_{jk}\approx \sum _{i=0}^{n-1}\frac{\partial r_i}{\partial {\theta }_k}\frac{\partial r_i}{\partial {\theta }_j}.$$

Since r_i = v_¡− y_¡ the derivative of the residual is given by

$$\frac{\partial r_i}{\partial {\theta }_j}=-\frac{\partial y_i}{\partial {\theta }_j}=-S_{ij}$$

and the Hessian approximation can be formulated as matrix product

$$H\approx S^{T}S=F$$

and the FIM can be used in place of the Hessian for local identifiability analysis. Whereas the previous presentation is formulated using the absolute sensitivities, similar arguments can be made for relative sensitivities

$${\tilde{S}}_{ij}=\frac{\partial \ln y_i}{\partial \ln {\theta }_j}.$$

Defining ${\tilde{y}}_i=\ln y_i$ and ${\tilde{\theta }}_j=\ln {\theta }_j$, the Hessian of the relative objective function is then formulated as

$$\tilde{H}={\tilde{S}}^T\tilde{S},$$

which can be approximated by the FIM using the relative sensitivities.

As noted previously, parameter identifiability requires that the Hessian H ≈ S^TS is non-singular. An immediate consequence of this is that the columns of S need to be nonzero and cannot be linearly dependent.

2.2.1. Structural Correlation Method

Correlations between parameters can be used to describe how the parameter values depends on each other when fitting experimental data from the same system. With a series of estimates for parameters a and b, the covariance and correlation are given by

$$\begin{array}{c}\text{Cov}[a,b]=\text{E}[(a-\text{E}[a])(b-\text{E}[b])],\\ \text{cor}[a,b]=\frac{\text{Cov}[a,b]}{\sqrt{\text{Cov}[a,a]\text{Cov}[b,b]}}.\end{array}$$

While this makes it clear how to understand what correlations are, the computational cost associated with this method is high.

For problems with constant measurement variance σ, the parameter covariance is the inverse of the FIM (given in (8)), i.e.

$$C=F^{-1},$$

which enables the calculation of correlations by

$$c_{ij}=\frac{C_{ij}}{\sqrt{C_{ii}{C}_{jj}}}.$$

In the structural correlation method, a pair of parameters with a large correlation – their uncertainty is strongly coupled – cannot both be estimated [19]. For practical applications, a correlation threshold γ < 1 is set to determine pairs of parameters that are correlated. For each pairwise correlated parameters, the least sensitive is identified as described in Algorithm 3.

2.2.2. Orthogonal sensitivities

The orthogonal sensitivities method developed by Li, Henson and Kurtz [22] combines two of the methods presented by Miao et al. [26]: the principal component analysis (PCA) and the orthogonal method. Whereas the structural correlation method is based on removing pair-wise correlation, the orthogonal sensitivities method constructs a subset by

Algorithm 3. Structural Correlation Method.

1. For each parameter, compute the norm of the sensitivity of the model output to this parameter. We choose the 2-norm

   $${\overline{S}}_j=\sqrt{\sum _{i=0}^{N-1}{S}_j^2\left(t_i\right)}.$$

2. Fix insensitive parameters for which ${\overline{S}}_j<\sqrt{\epsilon },$ where ε is the precision used when calculating the sensitivities.

3. Repeat the following steps.

   1. Calculate the covariance matrix by inverting the FIM, C = (F)⁻¹ = (S^TS)⁻¹, and determine correlations as described in Section 2.2.1. If the largest correlation is smaller than some value γ, the subset reduction procedure is complete. If the matrix is invertible, some parameters are perfectly correlated, and analysis can only be continued when one of these fixed.
   2. For the correlation with the largest absolute value, fix the parameter with the lowest overall sensitivity, determined by the norm of the sensitivities ${\overline{S}}_j.$
   3. The parameter to be fixed is noted, and the column of S corresponding to this parameter are removed from S, before correlations are calculated again, and the process is repeated until the stopping criteria in (a) is satisfied.

adding one parameter at the time. For each parameter, an importance index e is calculated using PCA and each time the subset of already selected parameters is updated, we compute the orthogonality index d for the remaining parameters. At each iteration the parameter with the highest product I = ed is added to the subset [22].

3. Results

In this section, we illustrate challenges associated with the various identifiability concepts using the algorithms summarized in Section 2. This will be accomplished using four examples illustrating different causes of unidentifiability: problem structure, availability of data, and a mixture of both. Note, that for the global sensitivity methods, the parameter values are mapped from [0, 1] to relevant intervals when the model is evaluated.

3.1. Structural unidentifiability

Consider a distensible blood vessel with a volume given by x. Blood flow into the vessel q_in = bu(t) is proportional to u(t) with proportionality b, while the flow out is proportional to the volume q_out = −ax. Here a is the constant of proportionality. Assume data y is proportional to the blood volume, y = cx, and treat the initial blood volume as a parameter x₀. For this system, the change in blood volume x can be quantified by the differential equation

$$\begin{array}{r}\hfill \dot{x}(x,t)\\ \hfill y\text{(}x\text{)}\end{array}\begin{array}{l}\text{=}-ax\text{+}bu\text{(}t\text{),}x\text{(0)}\text{=}{x}_0\hfill \\ =cx,\hfill \end{array}$$ (13)

Algorithm 4. Orthogonal Sensitivities Method.

1. For each parameter j calculate the Importance Index

   $$e_j=\frac{{\sum }_{i=1}^m{\lambda }_i{Q}_{ji}}{{\sum }_{i=1}^m{\lambda }_i},$$ (11)

   where γ_i is the i’th eigenvector of the matrix $F={\tilde{S}}^T\tilde{S}$ with the corresponding eigenvector being the i’th column of the matrix Q.

2. Select the parameter with the highest value of e as the first parameter, ${\theta }_{k_1}.$

3. Repeat the following steps until no more parameters can be added to the subset.

   1. Calculate the projection of the sensitivity vector ${\tilde{S}}_{{\Theta }_{l_{k+1}}}$ for each parameter j that has not been selected, onto the space spanned by the sensitivities of the k already selected parameters ${\theta }_{{\mathcal{l}}_1},\dots ,{\theta }_{{\mathcal{l}}_k},{\tilde{S}}_s=Span\left({\tilde{S}}_{{\theta }_{{\mathcal{l}}_1}},\dots ,{\tilde{S}}_{{\theta }_{{\mathcal{l}}_k}}\right),$

      $${\tilde{s}}_j=\sum _{i={\theta }_{{\mathcal{l}}_1}}^{{\theta }_{{\mathcal{l}}_k}}{\alpha }_i{\tilde{S}}_i.$$
      This is accomplished by finding the coefficients α_i that solves

      $$\underset{{\alpha }_i}{\text{min}}\frac{1}{2}{\left({\tilde{S}}_j-{\tilde{s}}_j\right)}^T\left({\tilde{S}}_j-{\tilde{s}}_j\right),$$

      which is equivalent to solving the linear system

      $$\left[\begin{array}{lllll}{\tilde{s}}_{{\theta }_{{\mathcal{l}}_1}}^T\hfill & {\tilde{s}}_{{\theta }_{{\mathcal{l}}_1}}\hfill & \cdots \hfill & {\tilde{s}}_{{\theta }_{{\mathcal{l}}_k}}^T\hfill & {\tilde{S}}_{{\theta }_{{\mathcal{l}}_1}}\hfill \\ \hfill & ⋮\hfill & \ddots \hfill & ⋮\hfill & \hfill \\ {\tilde{s}}_{{\theta }_{{\mathcal{l}}_1}}^T\hfill & {\tilde{s}}_{{\theta }_{{\mathcal{l}}_k}}\hfill & \cdots \hfill & {\tilde{s}}_{{\theta }_{{\mathcal{l}}_k}}^T\hfill & {\tilde{s}}_{{\theta }_{{\mathcal{l}}_k}}\hfill \end{array}\right]\alpha =\left[\begin{array}{c}{\tilde{S}}_j^T{\tilde{S}}_{{\theta }_{{\mathcal{l}}_1}}\\ ⋮\\ {\tilde{S}}_j^T{s}_{{\theta }_{\mathcal{l}}}\end{array}\right],$$ (12)

      as long as the columns corresponding to the already selected parameters are linearly independent.
   2. The linear independence of each parameter j is determined by considering the sine of the angle between the projection $\tilde{s}$ and ${\tilde{S}}_j.$ This yields the orthogonality index,

      $$d_j=\sin \left[\arccos \left(\frac{S_j^T\tilde{s}}{\Vert {\tilde{S}}_j\Vert \Vert \tilde{s}\Vert }\right)\right].$$
   3. Select the parameter j that has the highest Identifiability Index,

      $$I=e_j{d}_j,$$

   Note: Relative sensitivities were used in [22] but absolute sensitivities may be used as well.

where a, b, c and x₀ are model parameters. The model output is taken to be

$$y(t)=cx(t)=c{x}_0{e}^{-at}+cb{\int }_0^t{e}^{-a(t-s)}u(s)ds.$$ (14)

Note that the parameter c only appears in combination either with the initial condition parameter x₀ or with the parameter b. Hence, c, b and x₀ are not mutually identifiable since they are not uniquely determined by the data.

The local sensitivities can be calculated analytically by way of the ODE system

$$\begin{array}{lll}\frac{\partial y}{\partial a}(t,\theta )\hfill & =\hfill & -ct{x}_0{e}^{-at}-cb{\int }_0^t(t-s)e^{-a(t-s)}u(s)ds,\hfill \\ \frac{\partial y}{\partial b}(t,\theta )\hfill & =\hfill & c{\int }_0^t{e}^{-a(t-s)}u(s)ds,\hfill \\ \frac{\partial y}{\partial c}(t,\theta )\hfill & =\hfill & x_0{e}^{-at}+b{\int }_0^t{e}^{-a(t-s)}u(s)ds,\hfill \\ \frac{\partial y}{\partial x_0}(t,\theta )\hfill & =\hfill & c{e}^{-at}.\hfill \end{array}$$ (15)

Using the input function u(x) = sin(x), parameters a = 1.5, b = 2, c = 3 and x₀ = 1 for t_¡ ∈ [0, 7] yield the sensitivities shown in Fig. 2.

Fig. 2.

Sensitivities expressed in (15) for the linear input example given in (13–14).

When all parameters are included, or when only a is excluded, the matrix S^TS is singular due to linear dependence between the sensitivities to the parameters b, c and x₀. This is indicated by condition numbers on the order of 10¹⁶, whereas fixing of b, c or x₀ reduces the condition number to 10¹. We note that this matches well with the seemingly linear dependence of the sensitivities shown in Fig. 2.

Using the orthogonal sensitivities (OSM) in Algorithm 4, we obtained the ranking compiled in Table 1. We note that:

Table 1.

(a) Ranking obtained using the Orthogonal Sensitivities Method (OSM). Columns are Importance (PCA) score (e), Orthogonality (d) and total Identifiability (I = ed). (b) Sobol first order ($S_i^{*}$) and total effect ($S_{Ti}^{*}$), and Morris elementary effects mean and variance (${\mu }_i^{*}$ and ${\sigma }_i^{*}$).

(a) OSM
Rank	Parameter	e	d	I
1	a	0.66
2	b	0.45	0.53	0.24
3	x0	0.18	0.97	0.17
4	c	0.39	0.00	0.00
(b) Global
Parameter	Si*	$S_{Ti}^{*}$	${\mu }_i^{*}$	${\sigma }_i^{*}$
a	0.28	0.29	297.81	162.62
b	0.01	0.01	76.35	67.97
c	0.31	0.42	374.46	150.44
x0	0.24	0.32	322.08	120.32

1. a is ranked as most influential. The sensitivity plot in Fig. 2 confirms this, yet b and c are almost as influential.

2. b is ranked 2^nd. The orthogonality of b is only 0.45 which is lower than that of x₀, but b is chosen since the importance index is higher.

3. x₀ is ranked 3^rd. The importance index of x₀ is lower (0.18) than that of c (0.39), which is understandable considering the magnitude of the sensitivities in Fig. 2. However, since the sensitivity to x₀ has a larger projection (0.97) to those already selected, x₀ is chosen over c.

4. c is ranked last. When a, b, x₀ have already been chosen, the orthogonality index of c is 0.00. This confirms the observation that the sensitivities are linearly dependent.

Since there is a linear dependence in S, the FIM (8) is not invertible. Hence, it is impossible to calculate statistical correlations of the parameters using the Cramer-Rao bound [26]. As suggested by the Structural Correlation Method (SCM) one have to remove the least sensitive parameter if the sensitivity matrix is singular and so on until a non-singular sensitivity matrix is obtained. Removing one of the parameters b, c or x₀ the correlation matrix can be calculated. This results in off-diagonal elements with an absolute value significantly lower than 1, indicating that all remaining parameters are identifiable when b, c or x₀ is fixed. This problem with linear dependence could also be remedied by non-dimensionalization, here employed by scaling the model with respect to the initial condition, e.g. letting $\tilde{x}=x/x_0$, ${\tilde{x}}_0=1$, which in turn would give the output $\tilde{y}y(t)=c\tilde{x}(t)=c{e}^{-at}+c\tilde{b}{\int }_0^t{e}^{-a(t-s)}u(s)ds$, where $\tilde{b}=b/x_0$.

One important final remark is that fixing parameters reduces the degrees of freedom of the model and changes the model. For this example, the interaction between b, c and x₀ makes it impossible to determine which two parameters are identifiable. Since fixing one parameter effectively changes the model, it does not make sense to compare parameter values for x₀ in an optimization where a is fixed with another value obtained with b fixed. These two estimates of x₀ should not be expected to be similar as they are de facto parameters of different models. In other cases, it may be that noninfluential parameters are fixed. In contrast to cases with parameter interactions, this does not introduce a bias, as the noninfluential parameters have no impact on model behavior, and therefore also no impact on parameter estimates.

For the global sensitivity analysis methods the parameters are sampled from a uniform distribution ranging from $\frac{3}{4}$ to $\frac{5}{4}$ times the nominal value used for the local sensitivities. The model is evaluated for the times t_¡ = i/100, i ∈ [0,700], and the output considered is the sum of squares,${\sum }_{i=0}^{700}{y}^2\left(t_i\right)$. The total sensitivity index for the Sobol method and the elementary effects of the Morris method yields the same ordering of the parameters: c, x₀, a, b. These are different from the results obtained with the local methods, and also from what one would expect observing the local sensitivities in Fig. 2.

3.2. Practical unidentifiability

For practical unidentifiability, we consider two cases. The first case illustrates how parameters can be unidentifiable due to practical restrictions in model output measurements. This example shows how such restrictions can render a parameter noninfluential or fail to highlight differences in model effects. The second case is a model of a viscous material using a Voigt body model. While it is structurally identifiable, like the example in the first case, it is not clear whether the parameters are practically identifiable.

3.2.1. Case A: Aliasing

Consider the function

$$f(t,\theta )=\sum _{k=1}^p{\theta }_k^2{f}_k(t),f_k(t)=\sin (2\pi kt).$$ (16)

If the function is restricted to a certain rate, one may experience aliasing - appearing when an oscillating signal is recorded at a rate that make the signal appear to be oscillating at a lower frequency or not at all. This example is a nonlinear version of the aliasing example presented by Smith [38].

Assume that the function f can be evaluated at n evenly spaced values $t_i=i\Delta t,i=0,\dots ,n-1$, $\Delta t=\frac{1}{n-1}$, and that the parameters are restricted to the interval (0, 1). For example, for n − 1 = 4 the function can be observed/evaluated at the points t = 0, ¼, ½, ¾, 1.For p = 4 and n − 1 = {4, 8, 16, 256} the results are shown in Fig. 3(a)-(d), respectively. Panel (a) shows that for n − 1 = 4, there are only two non-zero basis functions and hence it is impossible to estimate all four parameters. Furthermore, the basis functions, f₁ and f₃ are linear dependent, implying that all possible model outputs can be represented by either of these functions. In (b) three basis function are non-zero, but no basis functions are linearly dependent. Hence, the parameters determining the weight of the non-zero basis functions are identifiable. For (c-d) all basis functions are non-zero and linearly independent, implying that all parameters are identifiable. The configurations corresponding to panel (a-c) will be analyzed using the methods of in Section 2.2.1.

Fig. 3.

Basis functions for the aliasing Example (3.2.1) evaluated at $t=0,\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$n$}\right.,\raisebox{1ex}{$2$}\!\left/ \!\raisebox{-1ex}{$n$}\right.,\dots ,\raisebox{1ex}{$n-1$}\!\left/ \!\raisebox{-1ex}{$n$}\right.,1$ (plotted with linear interpolation between the points) for the three measurement resolutions and full time series.

The local sensitivities can be calculated analytically. They are given by

$$\frac{\partial f}{\partial {\theta }_k}(t,\theta )=2{\theta }_k{f}_k(t).$$ (17)

Since it has already been shown that the basis functions are linearly dependent if evaluated at n − 1 = 4, 8, the sensitivities will also be linearly dependent. Therefore, it is known in advance that it is not possible to estimate all parameters uniquely for these configurations. Since the basis functions are linearly independent for n − 1 = 16 so are the sensitivities, and all parameters are identifiable.

Structural Correlation Method

Since there is a linear dependence between the four basis functions (two of them being zero), the FIM F = S^TS is not invertible, and therefore a linear combination of columns is trivially zero. Calculating the condition numbers, the F = S^TS-matrix for different values of n reveals that there is linear dependence in the sensitivities for n − 1 = 4, 8, but not for n − 1 = 16. As the FIM is invertible for n − 1 = 16, correlations can be calculated. This results in the correlation matrix with all zero entries indicating that no parameters are correlated.

Orthogonal Sensitivities Method

Table 2 shows the rankings obtained using the Orthogonal Sensitivities Method for n − 1 = 4,8,16. For n − 1 = 4 only one parameter is found to be identifiable, whereas three are found identifiable for n − 1 = 8, and all parameters are identifiable for n − 1 = 16.

Table 2.

(a) Ranking obtained using the orthogonal sensitivities method for n — 1 = {4, 8, 16}. Columns denotes the Importance (PCA) score (e), Orthogonality (d), and total Identifiability (I = ed). (b) Sobol and Morris indices for the parameters in Case A: Aliasing (Section 3.2.1) with m = 5 and n — 1 = 4, 8, 16 parameters. Here $S_i^{*}$ and S_Ti*· denote the first order sensitivity indices, while ${\mu }_i^{*}$ and ${\sigma }_i^{*}$ are the Morris indices.

(a) OSM
	n – 1 = 4				n – 1 = 8				n – 1 = 16
Rank	Par	e	d	I	Par	e	d	I	Par	e	d	I
1	θ3	0.71			θ1	0.55			θ3	0.38
2	θ1	0.71	0.00	0.00	θ3	0.51	1.00	0.51	θ2	0.33	1.00	0.33
3	θ2	0.00	0.00	0.00	θ2	0.48	1.00	0.48	θ1	0.33	1.00	0.33
4	θ4	0.00	0.00	0.00	θ4	0.00	0.00	0.00	θ4	0.25	1.00	0.25
(b) Global
	n – 1 = 4				n – 1 = 8				n – 1 = 16
Par	$S_i^{*}$	$S_{Ti}^{*}$	${\mu }_i^{*}$	${\sigma }_i^{*}$	$S_i^{*}$	$S_{Ti}^{*}$	${\mu }_i^{*}$	${\sigma }_i^{*}$	$S_i^{*}$	$S_{Ti}^{*}$	${\mu }_i^{*}$	${\sigma }_i^{*}$
θ1	0.05	0.85	1.10	1.24	0.29	0.32	1.15	1.33	0.20	0.29	1.31	1.56
θ2	0.00	0.00	0.00	0.00	0.41	0.36	1.12	1.31	0.18	0.25	1.36	1.62
θ3	0.15	0.90	1.10	1.25	0.39	0.37	1.17	1.34	0.23	0.29	1.33	1.61
θ4	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.16	0.25	1.33	1.61

For the global sensitivity analysis methods we assume that parameters are independent and uniformly distributed,

$${\theta }_i~\mathcal{U}(0,1).$$

From the local analysis and arguments determining the dimension of the spanned space, it is expected that parameters θ₂ and θ₄ are noninfluential for n − 1 = 4, that θ₄ is noninfluential for n − 1 = 8, and that all parameters are influential for n − 1 = 16. In addition, it is expected that the effect of all influential parameters on the model output are similar in magnitude.

Sobol indices

The estimated first-order Sobol indices $S_i^{*}$ and total indices $S_{Ti}^{*}$ are listed in Table 2. As expected, for n − 1 = 4 only θ₁ and θ₃ have an effect on the model output, while θ₂ and θ₄ have no effect. Also, note that the total effect estimates $S_{Ti}^{*}$ are similar for θ₁ and θ₃.

Likewise, for n − 1 = 8 and n − 1 = 16 the estimated indices match the expected behavior from the analysis of local sensitivities and the basis functions of the model.

Morris indices

All parameters are expected to have equal sensitivities, due to equal weights and same mean value. Parameter sets reducing to the same basis functions (n − 1 = 4) are expected to show some second- or higher-order effects. The resulting indices are shown in Table 2. The ordering predicted by the two global methods are consistent.

Next, we expand this aliasing example to p = n − 1 = 10 with results shown in Table 3. Both Sobol indices and Morris elementary effects suggests that all parameters are equally influential except for θ₅ and θ₁₀, which have negligible impact on the model output. This is confirmed by Fig. 4 where the graphs of the corresponding basis functions are plotted for the given resolution, and three higher resolutions. When n − 1 = 10, all samples of basis functions corresponding to θ₅ and θ₁₀ are zero. This result is consistent with the rankings obtained using OSM, also shown in Table 3. As indicated by these tables, there are two parameters θ₅ and θ₁₀ that do not contribute to the model output. For these subsets, the FIM is singular, and correlations cannot be calculated by inverting the FIM. Here consideration of which parameters’ fixation reduces the FIM condition number the most is required.

Table 3.

Sensitivity and identifiability analysis results for the aliasing example (Section 3.2.1) with p = n — 1 = 10. (a) Sobol indices, computed using Jansen’s implementation, and Morris elementary effects. Here $S_i^{*}$ and S_Ti*· denote the first order sensitivity indices, while $S_i^{*}$ and $S_{T{i}^{*}}$ are the Morris indices. (b) Ranking obtained using OSM. Here (e) denote the Importance (PCA) score (e), (d) Orthogonality, and (I = ed) total Identifiability.

(a) Global
	θ1	θ2	θ3	θ4	θ5	θ6	θ7	θ8	θ9	θ10
$S_i^{*}$	0.05	0.05	0.06	0.04	0.00	0.05	0.05	0.06	0.05	0.00
$S_{Ti}^{*}$	0.19	0.18	0.18	0.19	−0.02	0.17	0.18	0.17	0.20	−0.02
${\mu }_i^{*}$	9.78	8.83	9.25	8.90	0.00	8.82	9.96	8.88	9.22	0.00
${\sigma }_i^{*}$	112.00	105.67	116.31	93.10	0.00	103.31	134.31	102.61	117.50	0.00

(b) OSM
Rank	Parameter	e	d	I
1	θ2	0.36
2	θ8	0.36	0.98	0.35
3	θ4	0.30	1.00	0.30
4	θ6	0.30	0.98	0.30
5	θ1	0.25	0.97	0.24
6	θ9	0.25	0.98	0.24
7	θ7	0.23	0.92	0.21
8	θ3	0.23	0.98	0.22
9	θ10	0.00	0.00	0.00
10	θ5	0.00	0.00	0.00

Fig. 4.

Basis functions for the expanded aliasing Example (3.2.1) evaluated at $t=0,\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$n$}\right.,\raisebox{1ex}{$2$}\!\left/ \!\raisebox{-1ex}{$n$}\right.,\dots ,\raisebox{1ex}{$n-1$}\!\left/ \!\raisebox{-1ex}{$n$}\right.,1$ (plotted with linear interpolation between the points) for n − 1 = 10 and two other measurement resolutions and full resolution time series.

For the problem with p = 4 and n − 1 = 8, due to the resolution the parameter θ₄ is noninfluential, while θ₁, θ₂, θ₃ are linearly independent, and are therefore all identifiable. Adding to the sum an interaction term θ₁ θ₄ sin(2πt), gives the response-function

$$f(t,\theta )={\theta }_1{\theta }_4\sin (2\pi t)+\sum _{k=1}^4{\theta }_k^2\sin (2\pi kt).$$ (18)

One would expect that θ₄ now has influence on the model output, but that a large part of the influence is through interaction. Table 4 shows the ordering from OSM using parameter values θ_¡ = 1, as well as Sobol indices and Morris elementary effects. From OSM, we observe that the parameter θ₄ has an impact (the parameter e = 0.18)], but it is does not contribute any dynamics not already represented by the other parameters, namely θ₁ that it is linearly dependent on. Again, since there is a linear dependence in the sensitivities, correlations cannot be calculated from FIM. For the global methods, we note that all parameters are found to be influential, confirming that these methods measure sensitivity and not identifiability.

Table 4.

Results for linear aliasing with interaction term with p = 4, n — 1 = 8. Left: Global measures. Right: OSM Ranking.

Global						OSM
	θ1	θ2	θ3	θ4		Rank	Par	e	d	I
$S_i^{*}$	0.61	0.09	0.09	0.08		1	θ1	0.53
$S_{Ti}^{*}$	0.73	0.13	0.13	0.20		2	θ2	0.22	1.00	0.22
${\mu }_i^{*}$	19.95	11.85	11.06	4.00		3	θ3	0.22	1.00	0.22
${\sigma }_i^{*}$	298.03	87.54	93.07	18.81		4	θ4	0.18	0.00	0.00

3.2.2. Case B: Voigt body model

The previous aliasing example illustrated a case of practical unidentifiability. Unfortunately, for most models identifiability issues are more complex, as illustrated in this example.

Consider the two-element Voigt body model, shown in Fig. 5, which models the viscoelastic strain of the baroreceptor neurons in large arteries [2, 8, 24]. The deformation of the components can be described by the differential equations

$$\begin{array}{lll}\frac{d{\epsilon }_1}{dt}\hfill & =\hfill & -\left({\alpha }_1+{\alpha }_2+{\beta }_1\right){\epsilon }_1+\left({\beta }_1-{\beta }_2\right){\epsilon }_2+\left({\alpha }_1+{\alpha }_2\right){\epsilon }_w\hfill \\ \frac{d{\epsilon }_2}{dt}\hfill & =\hfill & -{\alpha }_2{\epsilon }_1-{\beta }_2{\epsilon }_2+{\alpha }_2{\epsilon }_w,\hfill \end{array}$$ (19)

where α_¡ = K_N/B_¡ and β_¡ = K_¡/B_¡. Let ε₁ be the model output, the Heaviside function be the model input ε_w = H, and nominal parameter values be α₁ = 2, α₂ = 1, β₁ = 2, β₂ = 0.4. For the global sensitivity analysis methods, we sample the parameter values with uniform probability on the intervals of 75 – 125% of the nominal value; e.g. α₁ ∈ 2 × (0.75, 1.25). We take the output variable to be ${\int }_0^5{\epsilon }_1^2(t)dt$. The model is solved for t ∈ (0, 5), with the Heaviside function changing at t = 0.5. Parameters α₁, α₂ determine how the strain is distributed between the two Voigt bodies, whereas β₁, β₂ determine the relaxation time for each Voigt body.

Fig. 5.

The Voigt body model used in the baroreceptor reflex model to describe viscoelastic strain of the baroreceptor neurons.

The local sensitivities are calculated using sensitivity equations and are plotted in Fig. 6 whereas analysis results are presented in Table 5.

Fig. 6.

Sensitivities for the Voigt body model.

Table 5.

Sensitivity and identifiability analysis results for the Voigt body model, (a) Sobol Indices (M = 50,000) and Morris elementary effects (r = 40) from Global analysis, (b) OSM ranking, (c) Initial correlations from SCM.

(a) Global
	α1	α2	β1	β2
$S_i^{*}$	0.19	0.50	0.14	0.11
$S_{Ti}^{*}$	0.15	0.53	0.16	0.17
μ	0.05	0.09	0.05	0.05
σi	0.01	0.01	0.01	0.00
(b) OSM ranking
Rank	Par	e	d	I
1.00	β2	0.97
2.00	α2	0.18	0.42	0.08
3.00	α1	0.04	0.23	0.01
4.00	β1	0.05	0.17	0.01
(c) SCM correlations
	α1	α2	β1	β2
α1	-	−0.90	−0.56	−0.91
α2	−0.90	-	0.85	0.98
β1	−0.56	0.85	−	0.75
β2	−0.91	0.98	0.75	-

Mahdi, Meshkat and Sullivant [23] found that the model is structurally identifiable, when regarding wall strain ε_w as model input and ε₁ as model output. Our analysis suggests that some parameters may be difficult to determine in practice. OSM yields that α₁ and β₁ have low identifiability indices based on low importance indices, while SCM yields large correlations between parameters in the pairs (α₁, α₂), (α₁, β₂) and (α₂, β₂). The global sensitivity analysis methods yield similar influence for all parameters.

3.3. Application to a physiological model

For the final application in mathematical biology, we consider a model of the cardiovascular system that describes the blood pressure in the systemic circulation of the human body. The model is derived from an electrical circuit analogy with a forcing function that models the pumping heart. It is described by a system of linear differential equations with an explicit forcing function. While the model is fairly simple, the relationship between, and identifiability of, the model parameters is not obvious due to the nonlinear dependence of model output on the parameters. We will analyze the model, build subsets and test the ability of the subsets to fit experimental data.

The model lumps the circulation into 5 compartments; Upper body arteries (au), lower body arteries (al), lower body veins (vl), upper body veins (vu) and the heart (h). For a more thorough description of the model see [46]. Figure 7 shows a schematic of the model, where components are depicted with the symbol of their electrical equivalent.

Fig. 7.

Overview of the cardiovascular model. All compartments represent a given volume and pressure, compartments are compliant modeled via a capacitor linked to ground, and flow between compartments are resistive. The heart compliance is time-varying representing pumping of the heart. Valves on both sides of the heart are modeled as diodes.

The model dynamics can be described from equations relating pressure (analogous to voltage) p_¡, flow (analogous to current) q_¡ and volume (analogous to charge) V_¡. As for most compartment models, the change in volume V is given by

$$\frac{dV}{dt}=q_{\text{in}}-q_{\text{out}},$$

and the flow and pressure related as described by Ohm’s law; i.e.

$$q=\frac{p_{\text{in}}-p_{\text{out}}}{R},$$

where R is the resistance, and finally pressure and volume is related as

$$p-p_{\text{ext}}=C\left(V-V_{\text{un}}\right).$$ (20)

Here C is the compartment compliance (analogous to capacitance), V_un the compartment unstressed volume, and p_ext = 0 is the tissue pressure.

The left ventricular pressure is predicted similar to (20), except it is formulated using a time varying elastance function E_lv(t) (inverse of compliance, E = 1/C) given by

$$\begin{array}{l}{p}_{lv}=E_{lv}(t)\left(V-V_{\text{un}}\right),\hfill \\ E_{lh}\left(t\right)=\{\begin{array}{l}\frac{E_M-E_m}{2}\left(1-\cos \left(\pi \frac{t}{T_M}\right)\right)+E_{m}0\le t\le T_M\hfill \\ \frac{E_M-E_m}{2}\left(1+\cos \left(\pi \frac{t-T_M}{T_R}\right)\right)+E_m{T}_M\le t\le T_M+T_R,\hfill \\ E_m{T}_M+T_R<t.\hfill \end{array}\hfill \end{array}$$

Figure 8 shows experimental blood pressure data and model output from the upper body arteries, computed using nominal parameter values. Since the blood pressure model does not have any description of the fluid dynamics of the blood, it is not able to predict the second peak in each cycle, as it originates from a reflective wave in the arteries. In attempt to encapsulate the modeled dynamics, we have selected two points in each cycle where the arterial blood pressure is not affected by the reflected wave for optimization during systole and diastole. Subsequently, for SA we selected five points centered around each of these, as illustrated in Fig. 8, resulting in the output times t = t_i, i ∈ [1, 5 × 2 × N] for N cycles. Inspired by Williams et al. [46], we added total blood volume and cardiac output (CO) – the total blood flow – to the model output, but only evaluate these at systole; that is, t = t_8+10(j−1), j ∈ [1, N],

$$y^m=\left[p_m\left(t_1\right),\dots ,p_m\left(t_{10N}\right),{\text{CO}}_m\left(t_8\right),\dots ,{\text{CO}}_m\left(t_{8+10(N-1)}\right),V_m\left(t_8\right),\dots ,V_m\left(t_{8+10(N-1)}\right)\right].$$

Fig. 8.

Zoom of experimental blood pressure data (red solid), and model output for nominal (blue long dashed) and optimized parameter values (green dash-dotted and black dotted, short dashed and dashed). The red dots show the time points used for SA and optimization and the vertical lines (gray) the start of each contraction cycle of the heart. The optimized curves are colored green for the combined method, and black for the individual SA methods. The Structural Correlation Method (SCM) and Orthogonal Sensitivities Method (OSM) solutions are virtually indistinguishable for large parts of the plot.

To balance the importance between matching pressure, blood volume, and cardiac output, blood volume and cardiac output are scaled by a constant factor α = 5. The resulting residual is given by

$$r=\frac{1}{10N+\alpha 2N}\left[\frac{y_1^m-y_1^d}{y_1^d},\dots ,\frac{y_{10N}^m-y_{10N}^d}{y_{10N}^d},\alpha \frac{y_{10N+1}^m-y_{10N+1}^d}{y_{10N+1}^d},\dots ,\alpha \frac{y_{12N}^m-y_{12N}^d}{y_{12N}^d}\right].$$

Initial values for parameter values and state variables (given in Table 8) are calculated from steady state data, as described by Williams et al. [46]. These values are used to calculate local sensitivities via sensitivity equations. For the global sensitivity analysis methods, to restrict parameters to positive values, the parameters are assumed to be log­normally distributed. Parameter values are sampled using a uniform distribution for the Morris and Sobol methods, and translated to corresponding percentile on log-normal distribution for the parameters. As the Morris methods sample from the extreme points on ℓ-grid, the sampled points have been scaled to fall within the 1 and 99 percentiles, since the 0 percentile corresponds to 0, and the 100 percentile does not have a well-defined value on the log-normal distribution. The global methods consider the least squares error J = r^Tr for the analysis.

Table 8.

Nominal and optimized parameter values for the cardiovascular model. Nominal values are calculated as described in the study by Williams et. al [46].

			Optimized
Parameter	Nominal	Common	SCM	OSM	Global
Cau	0.951	_	_	0.784	0.290
Cal	1.04	0.015	0.849	0.492	0.043
Cvl	42.2	-	29.7	-	0.746
Cvu	32.8	-	-	28.5	89.1
Raual	0.009	-	1.4e-13	-	-
Rauvu	1.66	-	-	-	16.5
Ralvl	0.799	3.78	1.05	1.99	4.10
Rvlvu	0.003	-	1.3e-9	-	-
Rvuh	0.001	-	-	-	-
Rhau	0.001	-	-	-	-
V0	10.0	-	-	0.014	-
Em	0.023	0.034	0.023	0.027	0.045
EM	2.31	1.16	1.13	1.01	1.17
tM	0.200	-	0.176	0.165	-
tR	0.150	-	-	-	-

Model data include upper body arterial pressure and heart rate measured over 32 cycles for a subject resting in supine position. Heart rate is used as a model input to set the length of each cardiac cycle, and blood pressure is to be compared to the model output. The fact that data is only available from one compartment means that the model is likely overparametrized. As common from electrical circuits, it can be observed that the two paths from the arterial to the venous side becomes almost equivalent. For this reason, interactions are expected between corresponding parameters in the upper and lower part of the circulation.

Figure 9 shows the ordering when parameters are ranked by the two-norm of sensitivities. Fixing the three least influential parameters, and then iteratively fixing the least influential parameter for the largest correlation pair until no correlation exceed 0.9 yields the subset

$${\theta }_{\text{SCM}}=\left\{E_{\text{M}},E_{\text{m}},t_{\text{M}},R_{\text{alvl}},C_{\text{al}},C_{\text{vl}},R_{\text{vlvu}},R_{\text{aual}}\right\},$$

using the SCM method.

Fig. 9.

Logarithm of two-norms of local relative sensitivities, estimated using sensitivity equations and scaled by largest two-norm.

Considering parameters with an I-value lower than 5 × 10⁻³ as unidentifiable, the OSM method yields the parameters

$${\theta }_{\text{OSM}}=\left\{E_{\text{M}},E_{\text{m}},t_{\text{M}},R_{\text{alvl}},C_{\text{al}},C_{\text{vu}},C_{\text{au}},V_0\right\}$$

as shown in Table 6.

Table 6.

(a) Results and ordering due to the OSM method for the blood pressure circulation model. The parameters below V₀ are considered unidentifiable due to having I < 10⁻⁴. (b-c) Ordering by the Sobol Total effects (5,000 model evaluations) and Morris elementary effects (480 evaluations) for the blood pressure circulation model.

(i) OSM				(ii) Sobol			(iii) Morris
Par	e	d	I	Par	μ	σ2	Par	μ	σ2
EM	0.49			Ralvl	0.64	1.21	Ralvl	0.47	0.48
Em	0.42	0.99	0.42	Rauvu	0.16	0.31	Rauvu	0.23	0.25
tM	0.46	0.63	0.28	Cal	0.12	0.18	Cal	0.18 0.19
Ralvl	0.29	0.47	0.14	Cau	0.066	0.14	Cau	0.15	0.16
Cal	0.10	0.20	0.021	Em	0.044	0.086	Em	0.12	0.13
Cvu	0.017	0.72	0.013	EM	0.003	0.015	EM	0.058	0.057
Cau	0.089	0.050	0.0044	Cvu	0.002	0.0012	Cvu	0.015	0.019
V0	0.12	0.014	0.0016	Cvl	0.00028	0.00087	Cvl	0.012	0.015
Rav	0.0067	0.11	0.00076	tM	0.00097	0.00079	tM	0.010	0.011
Rvlvu	0.0094	0.042	0.00039	Rvlvu	0.00002	0.00020	V0	0.0054	0.00581
tR	0.00079	0.14	0.00011	V0	0.00052	0.00015	Rvlvu	0.0046	0.0062
Raual	0.0056	0.015	0.000086	Raual	0.00008	0.00003	Raual	0.0020	0.0025
Rauvu	0.14	0.00019	0.000027	Rmv	0.00000	0.00001	Rmv	0.0011	0.0013
Cvl	0.0077	0.00063	0.000005	tR	0.00000	0.00000	tR	0.00057	0.00061
Rmv	0.0045	0.0011	0.000005	Rav	0.00000	0.00000	Rav	0.00016	0.00032

Table 6 also shows the sensitivity indices obtained for the Sobol and Morris methods. The two methods agree remarkably well, yielding just two parameters that differ in order. Choosing eight parameters, as with the local methods, the global sensitivity analysis methods yields the subset

$${\theta }_{\text{ Sobol }}={\theta }_{\text{ Morris }}=\left\{R_{\text{ alvl }},R_{\text{ auvu }},C_{\text{ al }},C_{\text{ au }},E_{\text{m}},E_{\text{M}},C_{\text{vu}},C_{\text{vl}}\right\}.$$

The three subsets have four parameters in common,

$${\theta }_{\text{ common }}=\left\{E_{\text{M}},E_{\text{m}},R_{\text{alvl}},C_{\text{al}}\right\},$$

which we will consider for optimization.

We used the Levenberg-Marquardt optimization method [21] to optimize the different subsets minimizing the least squares error, fixing all parameters not included in the subsets at their nominal values. Table 7 summarizes the optimization results: listing the number of estimated parameters, the least squares error, and the number of steps required by the optimization routine. Figure 8 shows the upper arterial pressure, simulated using the optimized parameter values for each parameter set and Table 8 shows the estimated parameter values. The optimization was run over the same time points used in the SA, and the plot shows a zoom over one cardiac cycle. From Fig. 8 it is clear that more than one parameter set can be used to obtain a reasonable fit. As expected, the estimated parameter values vary reflecting the bias associated with fixing certain parameters at specific values.

Table 7.

Optimization results for the different subsets.

Subset	# Parameters	LSE	Steps
Unoptimized*	15	67.8
Common	4	1.17	14
SCM	8	0.950	11
OSM	8	0.901	16
Global	8	1.14	27

^*Unoptimized parameters are fixed at their nominal values.

^**LSE = least squares error

To determine the variability of the model output, we sampled 5,000 points from each subset parameter space and calculated the least squares error at each point. Figure 10 shows the least squares error distribution for each subset using Gaussian kernel density estimation [37], as well as for the full subset with all parameters included. It is evident that the OSM, SCM and global subsets align well with the nominal distribution. As the common subset aligned well, but slightly worse than the others, we investigated the distribution when reducing the subset even more. As the figure shows, the distribution is fairly similar when using the common subset and removing E_m, however when removing E_M the distribution deviates.

Fig. 10.

Least squares error distribution with random sampling for different parameter sets. Each curve represents the distribution of 5,000 sampled configurations smoothed using kernel density estimation (KDE).

4. Discussion

In this investigation, we presented methods and algorithms for quantifying model sensitivity to parameters and consistently identifiable parameter subsets. We used partial derivatives of the model output with respect to parameters [26, 28, 32, 34] to locally estimate the parameter influence, and Sobol Indices [41, 38] and Morris Measures [27, 38] for global sensitivity analysis.

Whereas the latter allowed us to determine which parameters were noninfluential, they do not specify if any parameters are functionally related or the model over-parameterized. To remedy this challenge we considered two identifiability analysis methods: The Structural Correlation Method (SCM) [28] and the Orthogonal Sensitivities Method (OSM) [22]. The SCM constructs subsets by initially fixing noninfluential parameters, and then subsequently fixing parameters one by one until no linear pairwise correlations are left [28].

The OSM ranks the parameters by first selecting the most influential parameters based on a principal component analysis (PCA) of the local sensitivities, subsequently adding one parameter at the time to the ranking. The parameter added is the most identifiable found from analysis of its influence and the linear independence of its sensitivities to the space spanned by the sensitivities of the already ranked parameters [22]. Other methods aiming at using linear independence in sensitivities include the orthogonal method by Yao et al. [26, 48], SVD-QR [29, 12].

For the examples in Section 3.1 and 3.2, the global sensitivity analysis methods agreed, they both correctly specified parameters designated as noninfluential. While both OSM and SCM where able to determine structural and practical unidentifiability in these examples, the FIM is not invertible. If the FIM is not invertible it is necessary to first remove parameters for which local sensitivities are linearly dependent on others as described in the SCM algorithm. Moreover, in Section 3.1 we showed how non-dimensionalization may be used to reformulate the model and reduce the parameter space for structural unidentifiability. This method is not specific to the example presented here but can often help reduce complexity of any given model. The downside is that they may be more difficult to interpret directly as they need to be re-dimensionalized to have physical units.

A more complex and practically applied example was shown in Section 3.2.2 – a Voigt body is used to describe the viscoelastic properties of arterial wall. For this example, the SA methods showed that all parameters were influential, agreeing with the previous study by Mahdi, Meshkat and Sullivant [23], which showed that they are structural identifiable, yet both the OSM and the SCM suggested that given available input and output data, the model parameters are practically unidentifiable.

Finally, the analysis of the cardiovascular model in Section 3.3 showed that global sensitivity to parameters may be different from the local sensitivity, which is expected for a nonlinear model [32, 35]. Since OSM and SCM relies heavily on the two-norm of the local sensitivities, it is likely that subsets built using local sensitivities may be different for other model configurations. This may warrant an additional analysis after optimization to ensure that a better subset does not exist, and underlines the importance of good initial parameter configuration for optimization. Moreover, we noticed that the local subsets predicted by SCM and OSM appeared to be more efficient than the global set, with SCM reaching an optimum in fewer optimization steps, while OSM reaches a slightly better fit. In this regard, it should be noted that Levenberg-Marquardt, the optimization routine used, is a gradient based method [21], and as such relies on linear independence of the sensitivities, and thereby favors methods that seek to optimize linear independence of the linearized system. The parameter sets with eight parameters did slightly better than the set of the four common parameters. In general none of the subsets generated by the individual methods did much better than the smaller common set. One may use model comparison methods such as the Akaike Information Criterion [1] to quantify the (dis)advantage of including additional parameters for a slightly better fit. In terms of maintained variability of model output, the suggested methods for subset selection all produced similar results. But we also saw that the parameter set could be reduced even more without losing the original variability. Moreover, as observed in Table 8 the estimated parameters vary dependent on the chosen subset, this is expected as here we fix noninfluential parameters biasing the prediction. Therefore, care must be taken if influential parameters are fixed and results should be interpreted relative to the physiological problem in question.

These tasks are by no means special to physiological modeling, but may be seen in many other areas of mathematical modeling. Other obvious applications involves Chemical reaction models, or any other area where ODE or PDE models are constructed to match experimental data.

It is important to be aware that this is not the only nor the typical application of SA, and that other aspects of the analysis might have greater importance in other applications. A good general introduction to many aspects of SA is given by Saltelli et al. [34]. We have used SA as a tool to reduce the number of parameters in what Saltelli et al. [35] calls factors’ fixing in hyper-parameterized models. Another use is Factors’ prioritization where SA is applied to pinpoint which parameter to measure to reduce the model output uncertainty the most [35], and is perhaps the most frequent application of SA.

Depending on the problem under consideration, different strategies for subset selection may be viable. Based on the optimization of the cardiovascular model considered here, the local methods appear to superior in terms of computational effort and model fit. This is, however, based on having a reasonable initial parameter configuration and using a gradient based optimization routine. For models where a good initial configuration is not available, or where the exploration of potential behavior is prioritized, the global methods might be a better choice. If computational efficiency is essential, Morris Elementary Effects yields similar results to Sobol Indices at a fraction of the computational cost. One may also consider methods such as FAST [36] for more efficient calculation of Sobol Indices, or moment independent SA methods [44, 7] that consider the model output probability density function rather than the variance. [18] provides a nice overview of global SA methods. In addition Markov Chain Monte Carlo (MCMC) methods such as DRAM [15] or DREAM [43] could be considered for estimating parameter uncertainties and correlations. However, these methods are typically applied after identifiable parameters have been isolated, and as such are beyond the scope of this study.

In addition to considering which method(s) to apply, it is important to remember that fixing parameters and reducing parameter subsets may introduce a bias in the model in question. In practice this implies that a parameter value estimated using one subset, cannot be compared (or expected to be similar) to values obtained using other subsets, as the parameter has different possibilities to influence the model output through interactions with other parameters.