Navigating the landscape of parameter identifiability methods: A workflow recommendation for model development

Abstract

In pharmacometric modeling, it is often important to know whether the data is sufficiently rich to identify the parameters of a proposed model. While it may be possible to assess this based on the results of a model fit, it is often difficult to disentangle identifiability issues from other model fitting and numerical problems. Furthermore, it can be of value to ascertain identifiability beforehand. This paper compares four methods for parameter identifiability, namely Differential Algebra for Identifiability of SYstems (DAISY), the sensitivity matrix method (SMM), Aliasing, and the Fisher information matrix method (FIMM). We discuss the characteristics of the methods and apply them to a set of applications, consisting of frequently used PK model structures, with suitable dosing regimens and sampling times. These applications were selected to validate the methods and demonstrate their usefulness. While traditional identifiability analysis provides a categorical result [PLoS One, 6, 2011, e27755; CPT Pharmacometrics Syst Pharmacol, 8, 2019, 259; Bioinformatics, 30, 2014, 1440], we argue that in practice a continuous scale better reflects the limitations on the data and is more informative. The methods were generally consistent in their evaluation of the applications. The Fisher information matrix method seemed to provide the most consistent answers. All methods provided information on the parameters that were unidentifiable. Some of the results were unexpected, indicating identifiability issues where none were foreseen, but could be explained upon further analysis. This illustrated the usefulness of identifiability assessment.

Study highlights.

• WHAT IS THE CURRENT KNOWLEDGE ON THE TOPIC?

In pharmacometric modeling, it is important to know whether the data is sufficiently rich to identify the parameters of a proposed model. Parameter identifiability analysis can clarify this, but existing methods seem underused.

• WHAT QUESTION DID THIS STUDY ADDRESS?

How do selected identifiability methods (DAISY, aliasing, SMM, and FIMM) compare regarding correctness, interpretability, usefulness, and ease of use? What is the added benefit of continuous identifiability indicators? How to incorporate identifiability analysis in a modeling workflow?

• WHAT DOES THIS STUDY ADD TO OUR KNOWLEDGE?

FIMM provided the clearest and most useful answers. Both categorical and continuous indicators were valuable, with continuous indicators providing additional detail on the level of identifiability and hard‐to‐identify parameters. A workflow was recommended for identifiability analysis at various stages of model development.

• HOW MIGHT THIS CHANGE DRUG DISCOVERY, DEVELOPMENT, AND/OR THERAPEUTICS?

Parameter identifiability may be added to the toolbox of the modeler to diagnose over‐parameterization and can be used as a formal decision process to assess a model's suitability in relation to study design. The software is available for download [http://www.lapp.nl/lapp‐software/parid.html].

INTRODUCTION

In pharmacometric modeling, the goal is to develop a fit‐for‐purpose model. Typically, the range of models that can be used is constrained by the characteristics of the data set. In part, these constraints have to do with the adequacy of the model in describing the data and with numerical problems in fitting the model, and so the constraints cannot be made fully explicit. Another aspect is whether the data contains sufficient information to identify the model parameters at an adequate level of confidence. This issue can be addressed by parameter identifiability methods. In this paper, we address this topic.

Identifiability analysis can fit into various stages of the modeling process. It can be relevant before parameter estimation, to check whether a proposed model could in principle be determined by the data, or after parameter estimation, to help ascertain the quality of the fit. While it is superfluous to perform an identifiability analysis at every iteration step, it is of value to do so at least at the start and end of the process, and optionally at key steps in‐between when the model structure changes substantially.

We will first define what we mean by parameter identifiability. In this context, a model is considered as a black box, with the model parameters as input vector and the observed variables (e.g., a plasma drug concentration) as output vectors. The model parameters are formally identifiable, or in short, the model is formally identifiable, if two different parameter vectors lead to two different outputs. Mathematically, this can be formulated as follows: consider the model $M$ as a function from the parameter space $P$ to the output space $Y$, that is, $M:P\to Y$. Then, the model is formally identifiable if $p,q\in P, p\ne q\Rightarrow M\left(p\right)\ne M\left(q\right)$. This treats identifiability as a categorical outcome. Some methods provide a continuous outcome instead, which characterizes the level of identifiability. While this is more open to interpretation, in practice it can be more informative on the consequences of design choices. The applications will demonstrate this difference.

Identifiability can be qualified. As written here, the parameter vectors $p$ and $q$ can be anywhere in the parameter space. This corresponds to global identifiability. In contrast, for local identifiability at $p$ it is sufficient if the formula holds for $q$ in some small neighborhood of $p$.

Identifiability depends on the richness of the model output. Usually, the output consists of one or more time‐dependent quantities, such as concentration and/or biomarker levels. Some identifiability methods assume that these quantities are known at all timepoints. While this is not a realistic assumption in practice, it can indicate whether identifiability is possible in theory. Other methods restrict the observed quantities to user‐selected timepoints, which is more useful if one wants to consider a specific data set. We will use the terms structural and practical identifiability to distinguish these two types.

Pharmacometric models are often mixed‐effects models, that is, they have population parameters describing fixed effects and variability parameters describing the random effects. Some identifiability methods can handle population parameters only, while others, like log‐likelihood profiling (LLP) and the Fisher information matrix method (FIMM), can deal with random effects as well.

Another difference between methods is whether they can be applied before a model fit has been obtained (a priori) or afterward (a posteriori). In the latter case, the method relies on the ability to fit the model to the actual data. In the first case, the actual data is not used.

In this paper, we will consider four a priori methods for assessing identifiability: Differential Algebra for Identifiability of SYstems (DAISY), Aliasing, the sensitivity matrix method (SMM), and FIMM. They have been selected from a large collection of methods ¹ , ² , ³ , ⁴ , ⁵ , ⁶ , ⁷ , ⁸ , ⁹ , ¹⁰ , ¹¹ to cover a range of choices listed above, as well as on the availability of software implementations (without the need for proprietary software) and applicability to a large range of models. The first two methods were taken from the literature and the last two were developed by the authors. ¹² For SMM and FIMM, the natural identifiability criterion is sensitive to numerical approximation errors, and therefore, we developed alternative identifiability indicators that overcome this. ¹² All methods make assumptions on the smoothness of the model. In practice, the model will be defined by a set of ordinary differential equations and mild regularity conditions on these equations will imply smoothness. ¹³ As the methods employ differentiation of the state variables, discontinuities cannot be handled in general. There is no limit in theory to the degree of nonlinearity.

Earlier comparisons between identifiability methods ¹ , ³ considered structural categorical identifiability, apart from LLP. We consider a wider range, also including practical and continuous identifiability.

The methods will be compared against each other on different applications, based on models and datasets from the literature and form a cascade of increasing complexity. They were selected to highlight issues that may play a role in identifiability analysis, such as dependence on parameter values, dose level, or observation times and frequency.

The present work is linked to its companion paper ¹² but can be read independently and has a different purpose. The companion paper ¹² focuses on the development and technical aspects of the SMM and FIMM methods, and validates them with an example that is meant to be illustrative rather than a practical application. In contrast, the present analysis focuses on the application of these methods and studies them on more realistic examples. In addition, SMM and FIMM are compared with two other existing methods, and a workflow recommendation is provided to give guidance how to select and use the appropriate parameter identifiability methods in practice.

METHODS

Identifiability methods

DAISY performs structural identifiability analysis, using differential algebra to provide an exact answer to the question of global or local identifiability. ¹⁴ , ¹⁵ It assumes the output variables are known at all timepoints and can be applied to all models that can be written as a system of rational ordinary differential equations. Random‐effect parameters are not considered.

The other three methods engage in local, practical identifiability analysis. The Aliasing score, ranging from 0 to 100%, is a continuous identifiability indicator that characterizes the similarity of two derivatives $\frac{\mathit{dy}}{d{p}_1}$ and $\frac{\mathit{dy}}{d{p}_2}$, where $y$ is the output and $p_1$ and $p_2$ are two parameters. ¹⁶ If the two derivatives have similar profiles over time, then there is a relation between these two parameters and the model is locally unidentifiable. The output is evaluated at a finite set of timepoints.

SMM analyzes the sensitivity matrix (SM) $\frac{\mathit{dy}}{\mathit{dp}}$, consisting of the derivatives of the output $y$, evaluated at a finite set of timepoints, with respect to all input parameters $p$. Local unidentifiability is formally characterized by a non‐trivial null space of the SM, and there are several continuous indicators, namely the skewing angle, M‐norm, and L‐norm.

FIMM computes the Fisher information matrix (FIM) for a given parameter point and set of observation times. Local unidentifiability is formally characterized by a zero curvature of the log‐likelihood surface, corresponding to a zero eigenvalue of the FIM. Continuous indicators are the curvatures and the relative parameter changes.

SMM and FIMM have been extensively described in our companion paper, ¹² and further details are given in Supplement S1. They provide both categorical and continuous indicators. Of the four methods, FIMM is the only one that can handle random‐effects parameters.

Characteristics of the four methods are summarized in Table 1. Our implementations in R ¹⁷ of SMM, FIMM, and Aliasing are available for download. ¹⁸ DAISY was downloaded. ¹⁴ , ¹⁵

TABLE 1.

Overview of the characteristics of the identifiability methods.

Method	Global or local	Indicator type	Structural or practical	Mixed effects	Version
DAISY	Both	Categorical	Structural	No	1.9
Aliasing	Local	Continuous	Practical	No	0.5/R 4.1.1
SMM	Local	Both	Practical	No	0.5/R 4.1.1
FIMM	Local	Both	Practical	Yes	0.5/R 4.1.1

Note: The columns show if the method can characterize global and/or local identifiability, has categorical or continuous indicators, evaluates structural or practical identifiability, whether both fixed and random effects can be handled, and the software version number.

Applications

We selected a set of applications from the literature. Details on the models and scenarios are provided in Supplement S2. Figure 1 shows their time profiles. The applications are as follows:

1. A one‐compartmental linear pharmacokinetic (PK) model with first‐order absorption (oral dosing). This example illustrates various identifiability issues and highlights the difference between global and local, and categorical and continuous outcomes. Values of the clearance ($\mathit{CL}$) and volume ($V$) parameter, dosing and observation times were based on the literature. ¹⁹ Four scenarios were considered, namely scenario A1, where parameters were set such that the model was identifiable; A2, where the absorption rate (parameterized by $k_{\mathrm{a}}$) was lower to create an identifiability problem; A3, that was like A1 but with a bioavailability parameter ($F$) to make the model formally unidentifiable; and A4 that was like A3, except that the model was reparameterized to use an elimination rate $k_{\mathrm{el}}$ rather than $\mathit{CL}$. All scenarios employed an additive error $\epsilon ~N\left(0,{\sigma }^2\right)$.

2. A two‐compartmental linear PK model with intravenous (IV) dosing. This case was chosen to validate the methods. The model, dosing, observation times and values of the clearance ($\mathit{CL}$), central volume ($V$), intercompartmental clearance ($Q$), and peripheral volume ($V_{\mathrm{p}}$) parameters were based on the literature, ¹⁹ and an additive error $\epsilon ~N\left(0,{\sigma }^2\right)$ was included. Two scenarios were considered, namely B1 where the model was identifiable; and B2 where an identifiability problem was created by reducing the dose. With a lower dose, the model reached the lower limit of quantification (reflected by the residual error) at an earlier time, and to reflect this observation times were restricted.

3. A quasi‐equilibrium (QE) approximation to a target‐mediated drug disposition (TMDD) model, with IV dosing. This case was selected to illustrate that the methods can be applied to a more complicated, nonlinear example. The model, dosing, observation times and values of the elimination rate constant ($k_{\mathrm{el}}$), central volume ($V_{\mathrm{c}}$), tissue distribution rate constant ($k_{\mathrm{pt}}$), dissociation constant ($k_{\mathrm{D}}$), baseline receptor concentration ($B_{\max }$), degradation rate constant ($k_{\mathrm{deg}}$), and internalization rate constant ($k_{\mathrm{int}}$) parameters were based on the literature. ²⁰ Four scenarios were considered. The first three (C1, C2, and C3) each employed a single dose level from the literature, ²⁰ while scenario C4 combined these three dose levels. All scenarios used a proportional error $\epsilon ~N\left(0,{\sigma }^2\right)$.

FIGURE 1.

Time profiles for all applications and scenarios. (a) The one‐compartmental linear PK model with first‐order absorption (oral dosing), with scenarios A1: all parameters identifiable; A2: lower absorption rate to create an identifiability problem; A3: addition of bioavailability parameter (F = 1) to make the model formally unidentifiable; A4: like A3 but reparameterized by using an elimination rate instead of clearance. Scenarios A3 and A4 have the same time profile as A1; (b) the two‐compartmental linear PK scenarios with IV dosing. B1: identifiable; B2: identifiability problem created by reducing the dose; (c) the three dose levels of the TMDD‐QE scenarios with IV dosing. C1: dose 12.5; C2: dose 100; C3: dose 750; C4: combination of all three dose levels. The dots indicate the sampling times. Dosing is at time 0 in each case. Units have been left out as they are not relevant for identifiability analysis.

Inter‐individual variability (IIV) is not included in any of the cases, as some of the methods cannot handle it. Residual error is required for FIMM that include ${\sigma }^2$ as parameter and ignored for the other methods.

Data sets

All methods require dosing schemes. SMM, FIMM, and Aliasing require a sampling scheme, representing the timepoints of dependent variable measurements. DAISY just needs the output variables. Other data, like observed values, are not needed.

RESULTS

One‐compartmental linear PK model

DAISY classified scenario A1 as locally identifiable but not globally. This is because of the flip‐flop phenomenon: The model output is invariant under the parameter transformation $\left(\mathit{CL},V,k_{\mathrm{a}}\right)⟼\left(\mathit{CL},\frac{\mathit{CL}}{k_{\mathrm{a}}},\frac{\mathit{CL}}{V}\right)$. DAISY cannot distinguish between different parameter values, and so scenario A2 had the same outcome. Scenarios A3 and A4 were both unidentifiable because the model output is invariant under $\left(\mathit{CL},V,k_{\mathrm{a}},F\right)⟼\left(c\mathit{CL},cV,k_{\mathrm{a}},cF\right)$ for A3 or $\left(k_{\mathrm{el}},V,k_{\mathrm{a}},F\right)⟼\left(k_{\mathrm{el}},cV,k_{\mathrm{a}},cF\right)$ for A4, for any positive scalar $c$.

The SMM (Table 2) showed the scenarios A1 and A2 to be formally identifiable in the sense that the null space dimensions were zero, but A2 had lower values than A1 for the continuous indicators skewing angle (0.29 vs. 0.77), M‐norm (0.038 vs. 0.26), and lowest L‐norm (0.088 vs. 0.59, both for $V$). This showed that A2 had a lower level of identifiability than A1. The parameter $V$ had the lowest L‐norm in scenario A2, suggesting this was the hardest parameter to identify, while the MPR vector had a large component in each direction, indicating all parameters were unidentifiable. Scenarios A3 and A4 were formally unidentifiable, each having a one‐dimensional null space, corresponding to one unidentifiable parameter direction. The continuous indicators reflect this, having small values for the skewing angle, M‐norm, and smallest L‐norm. Both MPR and L‐norms indicated that $F$, $V,$ and $\mathit{CL}$ (for A3) were unidentifiable. For A4, $k_{\mathrm{el}}$ can be identified and indeed has a large L‐norm and no contribution in MPR.

TABLE 2.

SMM results for scenarios A1–A4.

Scenario	Null space dim	Skewing angle	M‐norm	MPR $\left(\mathit{CL},V,k_{\mathrm{a}}\right)$ (%)	L‐norm
A1	0	0.77	0.26	(−42, 34.3, 84)	$V$: 0.59, $k_{\mathrm{a}}$: 0.64, $\mathit{CL}$: 0.85
A2	0	0.29	0.038	(−32.8, 61.9, 71.3)	$V$: 0.088, $k_{\mathrm{a}}$: 0.1, $\mathit{CL}$: 0.27
A3	1	5.6e‐06	5.4e‐09	$\left(\mathit{CL},V,k_{\mathrm{a}},F\right)$ = (57.5, 57.5, 0, 58.1)	$F$: 3.7e‐16, $V$: 5.7e‐16, $\mathit{CL}$: 1.3e‐15, $k_{\mathrm{a}}$: 0.64
A4	1	0.0022	0	$\left(k_{\mathrm{el}},V,k_{\mathrm{a}},F\right)$ = (0, 70.7, 0, 70.7)	$F$: 2e‐16, $V$: 2.8e‐16, $k_{\mathrm{el}}$: 0.63, $k_{\mathrm{a}}$: 0.64

Note: MPR, minimum parameter relation, that is, the least identifiable direction in the parameter space, reported as relative changes in the parameters (normalized to 100%). The L‐norm column shows the parameters in increasing order of their L‐norm.

The aliasing score for $\left(k_{\mathrm{a}},V\right)$ (Figure 2) hints at the same issue for scenarios A1 and A2. All scores were low for scenario A1, up to 11%, consistent with identifiability. In scenario A2, there was a considerably larger correlation between $V$ and $k_{\mathrm{a}}$ of 67%, suggesting that these two parameters were more difficult to distinguish than in A1. Scores of 100% (indicating categorical unidentifiability) would be expected for the formally unidentifiable scenarios A3 and A4. This indeed happened for A4, but not for A3, where the highest score was 60%, showing that Aliasing did not unequivocally detect the unidentifiability.

FIGURE 2.

Aliasing scores (0–100%) for the four scenarios of the one‐compartmental linear PK model. Scenario A1 (top‐left): all parameters identifiable; A2 (top‐right): lower absorption rate to create an identifiability problem; A3 (bottom‐left): addition of bioavailability parameter (F = 1) to make the model formally unidentifiable; A4 (bottom‐right): like A3 but reparameterized by using an elimination rate instead of clearance. High scores indicate a correlation between these parameters and a lack of identifiability.

FIMM (Table 3) showed formal identifiability for scenarios A1 and A2 in the sense that all curvature values were strictly positive. However, for scenario A2 the first curvature was considerably closer to zero than for scenario A1 (0.0035 vs. 0.86), indicating that A2 was less identifiable. This was confirmed by the relative parameter changes: for scenario A1, all these changes were 10% or lower, while for scenario A2 and the lowest curvature, changes of over 80% were recorded for $V$ and $k_{\mathrm{a}}$. This was consistent with the results from SMM and Aliasing, which also found that $V$ and possibly $k_{\mathrm{a}}$ were hardest to identify. Scenarios A3 and A4 both reported a negative curvature (this was a numerical approximation error; the curvatures should be zero), corresponding to formal unidentifiability. The corresponding relative parameter changes show that the unidentifiable direction is $\left(\mathit{CL},V,k_{\mathrm{a}},{\sigma }^2,F\right)=\left(58\%,58\%,0\%,0\%,58\%\right)$ for A3 and $\left(k_{\mathrm{el}},V,k_{\mathrm{a}},{\sigma }^2,F\right)=\left(0\%,71\%,0\%,0\%,71\%\right)$ for A4, consistent with the theoretical invariance.

TABLE 3.

FIMM results for scenarios A1‐A4.

Scenario	Curvature	$\mathit{CL}$ or $k_{\mathrm{el}}$ (%)	$V$ (%)	$k_{\mathrm{a}}$ (%)	${\sigma }^2$ (%)	$F$ (%)
A1	0.86	5.4	−5.2	−10	0
	34	8.4	0.083	0.6	0
	83	−0.17	−0.068	8.5	0
	180	0	0	0	15
A2	0.0035	−44	83	95	0
	14	13	0.07	4.2	0
	180	0	0	0	15
	6200	−0.012	−5.1e‐04	10	0
A3	−5.9e‐13	58	58	7.6e‐13	0	58
	11	13	−0.13	−9.9	0	4.2
	61	2.7	−0.026	9.1	0	−1.2
	180	0	0	0	15	0
	5700	−0.031	−0.0013	0.076	0	2.6
A4	−2.1e‐13	0	71	−2.6e‐12	0	71
	33	−8	0.0025	13	0	−4
	180	0	0	0	15	0
	3300	7.2	−0.0021	0.17	0	3.3
	56,000	8.1	1.1e‐04	−8.0e‐04	0	−0.18

Note: Scenarios A1 and A2 each have four curvatures, and A3 and A4 each have five. Each curvature value is listed on a separate row, followed by columns containing the corresponding relative parameter changes. The “$\mathrm{CL}$ or ${\mathrm{k}}_{\mathrm{el}}$” column contains $\mathrm{CL}$ for scenarios A1‐A3 and ${\mathrm{k}}_{\mathrm{el}}$ for scenario A4. Relative changes in $\mathrm{F}$ are listed only for scenarios A3 and A4.

In summary, the categorical indicators did not distinguish between scenarios A1 and A2, but the continuous indicators showed a lower level of identifiability for scenario A2 than for A1. However, there is no theoretical cutoff value for any of these indicators to decide between identifiable and unidentifiable—this is a judgment call that the analyst must make. Based on the aggregated results of all the methods, we considered A1 to be clearly identifiable, while A2 had a low level of identifiability. DAISY, SMM, and FIMM classified scenarios A3 and A4 unequivocally as unidentifiable, while Aliasing did so only in case of A4.

Two‐compartmental linear PK model

Details are in Supplement S3. In brief, DAISY found both scenarios B1 and B2 to be globally identifiable, and this was confirmed by the categorical indicators (null space and lack of non‐positive curvatures) of SMM and FIMM. These indicators failed to detect the monophasic nature of the time profile of B2 (Figure 1). The continuous indicators of SMM and FIMM did detect this and found B2 to be less identifiable than B1. The SMM and FIMM showed that $V$ could be identified but all other parameters were badly identifiable. Aliasing results were less conclusive and showed that $\mathit{CL}$ and $Q$ had the largest identifiability problems in B2.

TMDD‐QE

Details are in Supplement S3. This model was originally analyzed for sensitivity and identifiability by the authors of the model, ²⁰ but in less detail. For this more elaborate model, DAISY did not provide an answer, see Supplement S4 for a discussion. The categorical indicators of SMM and FIMM showed all scenarios C1–C4 to be identifiable, but the continuous indicators of SMM and FIMM revealed the single dose level scenarios C1–C3 to have a considerably lower level of identifiability than the combined scenario C4. Aliasing results pointed in the same direction but were less clear. SMM and FIMM showed most of the parameters to be badly identifiable. These results could not easily be gleaned from the time profiles of Figure 1 that rather suggest that the highest dose scenario was identifiable, or at least that the other dose levels would not add relevant information.

DISCUSSION

Categorical yes/no answers to the identifiability question are valuable in case of formal unidentifiability, as in scenarios A3 and A4 that included an unidentifiable bioavailability parameter: If a model is categorically unidentifiable, then the identifiability problem is likely independent of the observation times, parameter values or dosing, and so the model needs to be redefined to resolve this. If categorical indicators indicate that the model is identifiable, there can still be identifiability issues, due to limitations caused by the choice of observation times, parameter values, or dosing. These can be identified by continuous indicators, as demonstrated by all discussed applications. The added value of continuous indicators is that they point to identifiability issues that the modeler may encounter in practice, even for formally identifiable models. Categorical indicators are provided by DAISY, SMM, and FIMM, and continuous ones by Aliasing, SMM, and FIMM. The next paragraphs discuss the methods one by one.

Concerning continuous indicators, FIMM provided the clearest and most useful answers across all analyzed applications. Besides the curvature metrics, FIMM has a natural cutoff between identifiability and unidentifiability in the size of the relative parameter changes: One could consider parameter changes above $p$ percent (for some value $p$) to be indicative of unidentifiability, reasoning that a parameter is not identifiable if it can change by $p$ percent without significant change in OFV (at 95% significance level). The level $p$ can be interpreted as the precision with which a parameter can be estimated. While the value of $p$ is somewhat arbitrary and depends on the number of subjects (a parameter that is not needed anywhere else in the analysis), the choice $p=50\%$ would classify all analyzed scenarios correctly and could be a reasonable choice in general.

SMM is an intuitive method that provides several continuous indicators. It is computationally more efficient than FIMM, scaling linearly in the number of outputs (like FIMM), and linearly in the number of parameters, in contrast to FIMM, which scales quadratically in the number of parameters. Therefore, SMM may be preferred for larger systems. A definition of specific cutoff values appears less obvious for SMM and cannot be justified from the theory. ¹² The method did allow comparison between different scenarios to assess their relative level of identifiability. Looking at all three applications, the following cutoff values could be used: 0.4 for the skewing angle, 0.05 for the M‐norm, and 0.1 for the lowest L‐norm. Smaller cutoffs (e.g., 10⁻¹⁰ for the M‐norm) may be used for the formally unidentifiable cases A3 and A4. There is however no guarantee that these values would also work for other applications and no ready interpretation in terms of parameter estimation.

Aliasing succeeded in distinguishing between scenarios A1 and A2 but had difficulties with the other two applications. A natural cutoff value does not exist, and no cutoff value would give the desired conclusion for all scenarios. Another limitation is that this method can only detect identifiability issues between pairs of parameters. It does not detect issues with a single parameter or with more than two, as illustrated by scenario A3, where the unidentifiability involved $\mathit{CL}$, $V$, and $F$. This contrasts with the other methods that did detect this issue.

DAISY provided a theoretical, exact answer that did not consider limitations on parameter values or observation times, assuming observations are given as functions of time. It was the only method that could identify the flip‐flop phenomenon, but it had difficulty with non‐polynomial models. In one instance (a variant of the TMDD‐QE model), the calculation ran for more than 5000 min on an Intel core i7‐7500U CPU at 2.7GHz without convergence, while convergent analyses typically took less than a minute. While it is possible to extend the use of DAISY beyond the polynomial models for which it was designed, ²¹ this was not successful in our case.

All four methods require a certain level of proficiency, in the sense that the analyst must interpret the results, weigh the different outcome indicators, and work around the method's limitations (in case of DAISY, which can handle rational models only). This applies mainly to continuous indicators, while categorical (formal) unidentifiability can be detected unequivocally by DAISY, SMM, and FIMM, as illustrated by scenarios A3–A4. The methods can be used in the model development process to separate identifiability problems from other modeling issues. Moreover, a successful fit is not equivalent to identifiability. ²² The methods are generally quite easy to use, requiring as input a model definition, and are freely available. An example application is provided in Supplement S6.

We recommend the following workflow for identifiability analysis, with different processes for three phases of model development, see Figure 3.

1. Before modeling starts: perform a structural identifiability analysis on the initial model(s). If a model fails this test, it either needs to be simplified or additional analytes need to be measured to support the model. In the second step, perform a practical identifiability analysis, which takes the study design into account. Failure indicates a need for model simplification or changes to the study design (e.g., additional sampling times). These steps help to determine a modeling strategy by providing an a priori check to see which models could be supported by the data and inform which biologically relevant parameters can be identified from the data. This is especially relevant if those parameters inform critical decisions (e.g., intervention or inhibition strategies) and may be critical in case of many parameters with limited sampling points. It also informs which model simplifications may be necessary for successful estimation, and whether identifiability issues are structural or related to design limitations.

2. While modeling, perform an identifiability analysis whenever relevant changes in the structural model are made, especially in case of instability or numerical issues in the parameter estimation, to assess whether they are caused by unidentifiability. This will point out potential sources of estimation problems and indicate ways to resolve them (i.e., highlight parameters that may have identifiability issues). The process is similar to that before modeling starts, except that structural identifiability analysis only needs to be performed if the model was made more elaborate. Also, adapting the study design is likely not an option at this stage, and so identifiability issues can only be resolved by model simplification.

3. After a candidate final model has been obtained, an identifiability analysis may help to assess model quality, relate observation error to estimation error (for SMM), and help suggesting additional experiments. Typically, one would perform only a practical identifiability analysis at this stage. The candidate model may be reconsidered if it turns out to be unidentifiable.

FIGURE 3.

Workflow for identifiability analysis. Left: the process to be followed for the initial model (before model development has started). Center: the process at key steps during model development. Right: the process after a candidate final model has been obtained. For structural identifiability analysis, we recommend using DAISY, with FIMM or SMM as fallback. For practical identifiability analysis, we recommend using FIMM, with SMM as fallback. cmplx ↑, increase in model complexity; N, no; pract idble, practically identifiable; struct idble = structurally identifiable; vars, variables; Y, yes.

The preferred tool for structural identifiability analysis is DAISY, as it provides an exact, global answer. If DAISY fails to provide an answer, FIMM or SMM, with a sufficiently large number of sampling times to minimize the influence of sampling limitations, are reasonable alternatives. They consider the actual study design but may suffer from numerical approximation errors. For practical identifiability analysis, we recommend FIMM, optionally combined with SMM, using dosing and observation times as in the study design. If the categorical indicators show identifiability, then continuous indicators can be analyzed to confirm this. If the categorical indicators show unidentifiability, then the model is unidentifiable, and continuous indicators inform on unidentifiable directions. A combination of methods may help to highlight potential issues. Our applications indicate that Aliasing results are hard to interpret, and so this method seems less useful.

Another alternative would be the $DESIGN option in NONMEM, ²³ or similar tools like PopED ²⁴ or PFIM. ²⁵ These tools assess the RSEs of the parameters of a model for a proposed design. These RSEs can be used as a measure of identifiability. This can however sometimes be misleading and unidentifiable cases may be classified as identifiable, see Supplement S5.

We recommend performing identifiability analysis without IIV, mainly because the identifiability of IIV parameters can often be evaluated directly and would impact the performance of the identifiability methods. For the same reason, inclusion of covariate effects is not recommended. FIMM is the only method of these four that can handle IIV. ¹²

In highly complex models like Physiologically Based Pharmacokinetics and Quantitative Systems Pharmacology, many parameters are a priori fixed to literature values and remaining parameters are estimated. Identifiability analysis can be used to justify the choice of estimated versus fixed parameters and the maximum influence of fixed parameters with potentially misspecified values.

For larger systems, the computational cost of the methods will increase. Aliasing and SMM scale linearly in the number of outputs and the number of parameters, while FIMM scales linearly in the number of outputs and quadratically in the number of parameters.

In conclusion, we think that parameter identifiability analysis may help model developers to investigate model stability issues or numerical errors, especially when the cause is unclear. Both categorical and continuous identifiability indicators have their uses and highlight different types of issues. Moreover, such an analysis can provide detailed information on the parameters that are hard to identify.

AUTHOR CONTRIBUTIONS

M.N., M.R., J.D., E.Ma., R.B., and N.S. wrote the manuscript and designed the research. M.N., M.R., J.D., E.Ma., and E.Me. performed the research and analyzed the data.

FUNDING INFORMATION

This work was conducted and funded by LAP&P Consultants BV.

CONFLICT OF INTEREST STATEMENT

The authors declared no competing interests for this work.

Supporting information

Data S1.

van Noort M, Ruppert M, DeJongh J, et al. Navigating the landscape of parameter identifiability methods: A workflow recommendation for model development. CPT Pharmacometrics Syst Pharmacol. 2024;13:1170‐1179. doi: 10.1002/psp4.13148