A Pharmacometric Workflow for Resolving Model Instability in Model Use‐Reuse Settings

ABSTRACT

The development of fit‐for‐purpose pharmacokinetic‐pharmacodynamic (PKPD) models based on clinical and pre‐clinical data is a critically important process in model informed drug development. This process is often hampered by modeling stability issues that are often multifactorial in nature and difficult to overcome, leading to protracted model building and arbitrary simplification of the model. This tutorial provides a heuristic workflow to help identify and resolve issues relating to model instability. The approach is centered on analyses undertaken using NONMEM, but the concepts can be generalized to other software used for population analysis of pharmacokinetics (PK) or PKPD data.

1. Introduction

Population PK and PKPD models are part of the fundamental building blocks of model informed drug development [1] and model informed precision dosing [2, 3]. Their accurate characterization provides the basis to understand the influence of dosing and various patient factors on the response profile of interest. These models pave the way for either the design of new clinical pharmacology studies or for optimizing therapy in clinical practice, both requiring extrapolation to different settings. A key requirement for extrapolation is their ability to capture the important biological mechanisms in order to predict outcomes under scenarios that have not yet been studied.

The aim of this tutorial and hence our target readers is twofold: (1) to raise awareness of model stability issues and ways to approach these problems and (2) to provide a detailed series of approaches that can be used to resolve these issues. While every effort has been made to ensure this tutorial is accessible to all readers, we concede that not all stability issues will allow this. Therefore, we hope that our coverage of aim 1 will provide a necessary background for the novice modeler and aim 2 will meet the needs of the expert whose role it will be to resolve the problem. We also would like to stress that this tutorial is by no means a complete treatment of stability issues, and further works are needed to support our modeling community.

While mechanistic appropriateness is critical, this feature can often be considered implicit for a single response population PK model for compounds whose kinetic profile is unaffected by the disease target (the most common scenario). In this case, a standard compartmental model created from the data or from reducing a PBPK model would be expected to arrive at about the same structure, viz. a two‐compartment disposition model with an appropriately conceived input process. Since the absorption process may not materially affect the important characteristics of the kinetic profile at steady state, this can often be relegated to a study‐specific nuance effect (i.e., not important for all settings but not ignorable for the current setting). However, more complicated population PK models, for example, those that include mono‐ or bi‐specific TMDD or multiple response types such as parent‐metabolite models or antibody‐drug conjugates require a priori sound (explicit) mechanistic understanding. This explicit mechanistic requirement is also necessary for PKPD models due to their often nonlinear relationship between the PD response and the PK profile.

Models for explicit mechanisms can be developed from two perspectives [4]: (1) developing the model from the lens of the data and applying it to the system and (2) developing the model from the lens of the system and applying it to the data. Both methods are used commonly in pharmacometrics, and both have their benefits and losses (Table 1).

TABLE 1.

Comparison of models developed from a data lens versus systems lens.

Feature	Data lens	Systems lens
Mechanism capture	Only as the data speaks to the mechanism a	Based on biology
Development time	+	+++
Application to data	+++	+
Extrapolation to other settings	+/++	+++

^a Care must be taken not to rely on the data to fully elucidate the mechanism. Data that arises from feedback systems may at the surface provide misleading interpretation of the mechanism [5].

Models developed from a data lens are therefore necessarily limited by the information content of the data and extension beyond this may cause modeling instability requiring a step back in mechanistic quality. For example, compounds that are expected to have TMDD properties may be modeled with varying levels of approximation depending on the data content,

• linear PK model (when the TMDD is minor in comparison with the PK profile at the proposed doses),

• linear time‐varying (when there is evidence of time‐varying clearance but insufficient to display nonlinearity),

• simplified equilibrium nonlinear binding (often where drug and target are both measured), and

• full kinetic binding.

The choice of model is therefore a trade‐off between data (Fisher‐type) information content and model complexity (readers may refer to Monteleone [6] for an introduction or Mentré [7] for a full treatment of Fisher information for the population PK setting). Here information content essentially reflects how much the data can tell us about the parameter values of the model. If the model complexity exceeds the data information content then instability in modeling may occur because the parameters will be imprecise.

Notwithstanding the natural application of a data‐lens approach to modeling data, that is, developing a model based on the observed data, there remain often complicated issues relating to intrinsic model‐based elements [8] and indeed settings where the data, although theoretically of good quality, may not behave as anticipated. For example, in the setting of PKPD model development, a turnover model is commonly applied to capture the delayed time course of a PD biomarker relative to the PK profile. This model is inherently mechanistic in flavor, usually incorporating parameters for the zero‐order production of the PD marker as well as its first‐order elimination [9]. Provided there is suitably informative data for both the PK and PD components, a data‐lens approach would be expected to produce a stable model and a reasonable approximation of the time course and magnitude of drug effects on the system. We explore a stability scenario later in a case study.

In contrast, models that were developed from a system lens are based, in theory, largely on mechanism without regard to how future data may arise under the system and indeed whether sufficient quality data are likely to be available from any single experiment. These models may be large and unwieldly [10] or may be fit‐for‐projected‐purpose. When the projected purpose is estimation they are also referred to as minimal models [11]. Again fit‐for‐projected‐purpose is theoretical since no model can be fit for all projected purposes even if the system is the same between minimal model development and application. We therefore distinguish fit‐for‐(current)‐purpose which relates to current purpose only from fit‐for‐projected‐purpose which is a generalisable claim. Application of system lens models are therefore likely to incur an additional overhead due to the potential for a mismatch of the information content of the data to the model complexity. Despite this mismatch, the obvious benefit of these models are (1) they are relatively plentiful in the literature and (2) if the system is relevant to your current setting then why not take advantage of what is already known and available?

As an example, it might be useful to re‐purpose an existing minimal model for a specialized development programme, such as vaccine development. In this setting developing a data‐lens model based on existing data may not allow extrapolation to future settings, for example, predicting from animal exposure to humans or from the first dose to the second or subsequent booster doses. It would be desirable to re‐purpose an existing immunostimulation‐immunodynamic (ISID) model for an existing vaccine such as the model developed by Rhodes and colleagues [12] for TB and applying this to a different disease setting. Such reuse would allow the system parameters relating to effector and memory T‐cells and the relationship with IFN‐γ to be retained while allowing the specifics of the antigen system to be explored.

1.1. Model Stability

We use the term model stability in a general sense to incorporate both model reliability (confidence in the model) and stability (resistance of model to change) [13], since we believe that lack of model stability is a cause of model unreliability. In concept, model stability refers to an acceptable level of variability in the performance of a model. Model instability, by corollary, is the situation in which the variability in model‐performance exceeds some reasonable expectation and the term is used here as a collective diagnosis that covers a constellation of issues that all pharmacometricians will have experienced. Some common issues include (we have chosen NONMEM as our exemplar, but the concepts below hold for other software):

• Runs where sometimes minimization will converge and produce standard errors of the parameters, and other times converge with no standard errors or not converge at all.

• Failure to converge, producing a termination error.

• Zero gradients during parameter search or poorly mixing chains.

• Different minima from different initial estimates.

• Different parameter estimates under different run settings.

• Biologically unreasonable parameter estimates.

• A change of platform results in very different modeling results.

• Large condition number.

• Failed standard errors.

• Run does not stop.

• Various complicated numerical errors occurring on minimization.

Although the overall diagnosis of “unstable model” is generally obvious, the term does not offer any insight into the root cause and therefore possible solution to the problem.

In this tutorial, we take a simplifying approach and propose that model instability is a combination of two discrete factors that may be teased apart and resolved separately.

• The balance of model complexity and data information content (= Design quality).

• Data quality.

In our exploration of model instability we are both supporting and challenging the common sentiment “your model is over‐parameterised”.

1.2. Workflow

There is no explicit workflow that will yield a solution to all problems of model instability. For instance, this workflow does not consider the presence of non‐syntactical coding errors (we assume syntactical coding errors will yield a syntax error and hence be identified by the user prior to a model‐based analysis). Non‐syntactical coding errors are not uncommon and require careful attention to code detail and would usually be identified during a code verification check (e.g., incorrectly numbering THETAs). We therefore make the assumption that prior to proceeding down the path of evaluating reasons for model instability that all reasonable measures have been undertaken to ensure that the intended model is confirmed and verified for its intended purpose. We use the engineering analysis framework described by Schlesinger [14] that defines confirmation as the process of confirming the schematic of the system for the current analysis is an appropriate representation of the system (e.g., confirming we are talking about a vaccine immunostimulatory‐immunodynamic system and this is the model you have located from the literature) and that your model code is a verified representation of the schematic (e.g., the minimal model extracted from the literature has been faithfully reproduced in your hands).

Other causes of model instability may relate to optimization features of the software and choice of the optimization algorithm. These important considerations are outside the scope of this tutorial.

The proposed workflow has been developed with users of NONMEM and R in mind as these are the most common industry standard platforms in pharmacometrics. We will be using specific NONMEM features throughout to illustrate the approach. Other software can be substituted. Although we introduce a workflow for considering model stability that we have found to be fairly robust it should not be considered that the workflow is neither complete nor accommodates all issues and indeed further work is needed to help modelers.

The approach is divided into two parts: (1) a workflow for disentangling the design quality from data quality and (2) considerations of data quality.

A schematic of the workflow is provided in Figure 1.

FIGURE 1.

Workflow for evaluating and resolving model instability.

1.2.1. Step 1: Structural Identifiability

The first step of any workflow for evaluating and resolving model instability is to evaluate whether the model you have at hand is structurally identifiable given the expected input (i.e., dosing) and output (i.e., response measures). If a model is not structurally identifiable, then the parameters of the model cannot be estimated because there is more than 1 set of parameter values (perhaps an infinite number of sets) that solve for the same input–output relationship. Modeling software does not check for structural identifiability and often will estimate parameters even for models that are not structurally identifiable but will do so with high instability. Readers are referred to Lavielle [15] for a description of identifiability in relation to population analyses.

The issue of structural identifiability is equally important for data‐lens models as for those minimal models that may be extracted from the literature. Although the process of developing a model based on data (e.g., extending from 1‐ to 2‐compartment or adding random effects) should, on the whole, be straightforward and without inherent issues there may be unobvious risks. For instance, adding a between‐subject random effect (ETA) for $F1$ (while fixing the typical value) for an orally administered drug is identifiable for a 1‐cpt model that is parameterized in terms of $\mathit{CL}$ and $V$ but not if parameterized as $k$ and $V$ [16].

The local structural identifiability of any model can be evaluated in NONMEM without the need for access to additional specialized software. We use local here because the identifiability analysis is performed at a specific set of parameter values and hence we can only assess its identifiability at these parameter values (hence local). This contrasts with various software that can distinguish local from global identifiability, the latter does not need a pre‐specified set of parameter values. However, due to the widespread issue of flip‐flop [17] most if not all PK and PKPD models are ever only locally identifiable making the distinction of global versus local of minimal practical importance.

Structural identifiability can be evaluated using the Fisher Information Matrix (FIM), based on the method of Shivva [18]. If the determinant of this matrix is zero then the model is not identifiable. To remove the possible influence of the underlying data design we can consider a (clinically) impossibly rich design, such that we can say it is the model which is not identifiable rather than the data design does not support all parameters. NONMEM version 7.5 can be used to calculate the determinant of the FIM using $DESIGN (see Bauer [16]) for a detailed description of this feature where the usual extended least squares objective function value (OFV) is replaced by $\mathit{OFV}=-\mathit{log}\left(\mathit{det}\left(\mathit{FIM}\right)\right)$. If the model is not identifiable the OFV value will be missing as the log of zero is not possible, but is also sometimes reported as a very large positive number, for example, 1.34E+154 or + Inf (meaning that the determinant was negative which is most likely a matrix algebra error).

There are three steps needed to perform a structural identifiability analysis in NONMEM:

1. Generate a single‐subject data template for a very rich design (e.g., a blood sample every half an hour for a week) from the largest dose of interest and including all types of dependent variables (DV) appropriate to your intended study. The DV items can be “.”.

2. Read this data template into your NONMEM model file* using $INPUT and $DATA, then remove (or comment out) both $EST and $COV and add $DESIGN GROUPSIZE = n FIMDIAG = 1 MAXEVAL = 1, and fix $SIGMA initial values to be small, for example, 1E‐3.

3. The model file is then run model in the command shell using nmfe75 <modelname> <resultfilename>.

   (example syntax: nmfe75 run1.mod run1.lst)

*In NONMEM the $ indicates a control statement that introduces characteristics of the NONMEM (NM‐TRAN) code

$INPUT = variable names relating to the data file

$DATA = data file name and any pre‐processing requirements on the data set

$EST = defines the estimation method and settings

$COV = defines the covariance step (standard error calculation) method and settings

$DESIGN = defines the Fisher information matrix calculation and settings

$COV & $EST are not used in the same problem as $DESIGN

Readers are referred to Bauer (tutorial part 1) [19]

Your choice of n for GROUPSIZE will depend on whether you want to consider only your structural parameters (e.g., THETAs) or your THETAs and OMEGAs, set n = 1 or n = 100 respectively. Fixing the $SIGMA values to be small improves the signal to noise of the design. Covariate effects will need to be turned off as the covariate values will not be in your simulated data file as these are seldom the cause of a structural identifiability issue.

If the model is not structurally identifiable then 1 or more parameter(s) will need to be fixed. This approach, however, does not tell the user which parameter is not identifiable. This can only be determined heuristically with additional run(s) in which 1 or more parameters are fixed. Additionally, if the model is not structurally identifiable then it is not appropriate to use $PRIOR since traditional maximum likelihood and Bayesian approaches (e.g., Markov chain Monte Carlo (MCMC) method and the Hamilton non‐U turn sampler) require the model to be identifiable.

It is also of limited value to perform a sensitivity analysis at this stage since the response variable may be either very sensitive or insensitive to a non‐identifiable parameter (i.e., not identifiable does not mean not influential) and hence will provide no information as to the choice of value. Where possible, choose a biologically plausible and justifiable value, for example, fixing $\mathrm{F1}=1$.

An example evaluation of structural identifiability analysis is provided in Supporting Information S1.

1.2.2. Step 2: Deterministic (Practical) Identifiability Analysis

Once parameters that are not structurally identifiable are fixed, it is now possible to consider the quality of the data design. Essentially, we are now looking at the parameter standard error values and if, for example, a % relative standard error (RSE) was 500%, then we can conclude that even though this parameter is, in theory, structurally identifiable, it is not deterministically identifiable given the current data design. Note we consider RSE, but this also remains a function of the parameter value, and care should be taken when considering this if the parameter is in the transform space, in which case standard error may be more appropriate.

The RSE values of the parameters can be calculated for the current model with the current parameter values based on the actual study data without the need to do an estimation run in NONMEM. Again, we will make use of the $DESIGN feature of NONMEM in which it calculates the FIM rather than the usual extended least‐squares OFV. The RSE for the ith parameter can be calculated from the FIM as $\%\mathrm{RS}{\mathrm{E}}_i=\frac{\text{sqrt}\left({\left(\mathrm{FI}{\mathrm{M}}^{-1}\right)}_{i,i}\right)}{P_i}\bullet 100$, fortunately NONMEM calculates the SE automatically and populates these into the usual location in the NONMEM output file (usually with the “.lst” file extension) and you can use any post‐processing package to obtain the RSE.

It is a two‐step process to perform a deterministic identifiability analysis

1. Remove (or comment out) $EST and $COV and add $DESIGN GROUPSIZE = 1 MAXEVAL = 1 FIMDIAG = 1

2. The model file is then run in the command shell using nmfe75 <modelname> <resultfilename> (example syntax: nmfe75 run2.mod run2.lst)

Since you will be using your current analysis data set as‐is, then all subjects, covariate effects, etc. can be retained in your model. There is no need to adjust $DATA or $INPUT since this method does not require a data template to be generated. Care should be taken with the initial values of $SIGMA. These should be close to expected. Note, the FIM is weighted by $SIGMA, so having an initial value of the variance of a proportional error term =1 (i.e., CV = 100%) would most likely yield high %RSE values for most parameters. Often, a more appropriate value might be 0.0225 (CV% = 15%). Similarly, other parameter values ($THETA and $OMEGA) should be reasonably close to what you anticipate from the analysis. In some instances, the parameter values may not be known with any certainty. Where possible, plausible values should be chosen (e.g., from prior information). Otherwise, this step could be repeated with several values of the parameters to evaluate the robustness of the design to the unknown parameter values.

When run, NONMEM will then return the expected standard errors based on your current executed design but ignoring your actual dependent variable values. It is now possible to disentangle the quality of the design from the quality of the data. For example, if all parameters had small RSE values, then it can be concluded that any model instability is caused by a data “feature” or an $EST algorithm “feature”. This finding will allow the modeler to focus their concerns outside of the model and design.

Any parameter that has a high RSE value should be considered further in the workflow.

An example evaluation of deterministic identifiability analysis is provided in Supporting Information S2.

1.2.3. Step 3: Global Sensitivity Analysis

Any parameter that was found to be not deterministically identifiable (e.g., RSE > 100%) from step 2 will ultimately need to be fixed or a prior incorporated on its value. Fixing the parameter may in many cases be the simplest and most appropriate step forward in which case the modeler may move forward simply to documenting this choice. The previous steps 1 and 2 will provide the necessary reasoning for fixing the parameter; however, the choice of value will depend on what is known.

A global sensitivity analysis (GSA) provides a method to understand the influence of the parameter(s) of interest on the model‐based inference. Readers are referred to the tutorial on Sobol, a version of GSA [20]. A GSA differs from an local sensitivity analysis (LSA) as it considers all relevant parameters, and their second and higher order interactions, whereas an LSA only considers one parameter at a time. The GSA method also calculates the contribution of any given parameter (or set of parameters) to the overall variability. This has the same interpretation as a partial $R^2$, that is, the $R^2$ attributable to that parameter, and hence provides a natural way to scale the apparent influence of a parameter. Note, if there is only 1 parameter that has a high RSE value then a simple local sensitivity analysis can be performed.

When applying GSA (or LSA) in this workflow it is only necessary to consider those parameters that were found not to be deterministically identifiable (from Step 2). Hence this is a much smaller dimensional problem than considering all parameters which is the usual case for GSA. In addition, it is often reasonable to only consider up to the second‐order interaction terms (the parameter in question paired with each of the other parameters). It is required to determine a summary measure of the response profile (e.g., maximum concentration, trough concentration, average concentration, …) in order to compare variability. Once selected the analysis can be performed in R using the package (see case study 2).

Any parameter where both itself and second order terms contribute < 20% of the overall variability may be considered non‐influential and therefore fixed to the initial value. If no influential parameters remain, then the workflow can be exited and both the reason why the parameter was fixed and evidence for the value to which it was fixed can be documented.

If, however, the parameter was found to be influential, then the choice of what value to fix the parameter to requires further consideration.

1.2.4. Influential but Not Estimable Parameters

An influential parameter that cannot be estimated (but is structurally identifiable) poses a general problem. It should not be fixed to any arbitrary value but rather a value must be selected that does not adversely influence the primary inference of the intended use of the model.

In this setting, the choice is whether to apply a Bayes prior or an empirical prior. The distinction between these two types is a matter of how the prior is determined. If the prior arises from previous work (i.e., work that excludes the current work) then this is a Bayes prior, whereas if the prior was determined empirically based on analysis that incorporated the current data, then this is an empirical prior. The latter lets the data speak to both the prior and the estimation conditioned on the prior (i.e., the data speaks twice). While this may seem like an inappropriate use of a prior, it is the most common scenario in data analysis.

The choice of type of prior will depend on either the intended analysis technique based on the modeling and simulation analysis plan or the particulars of the setting. For instance, it might be appropriate to consider an informative Bayes prior for settings in which the compound is anticipated to behave similarly to others in its class. In the case of monoclonal antibodies (mAbs), there is good evidence that they share similar disposition characteristics, meaning that the application of an informative Bayes prior may be deemed entirely appropriate [21]. Indeed, Haraya proposes that a subjective (informative) prior be used for mAbs to act as the reference intravenous disposition profile such that a bioavailability study need only test the subcutaneous dose.

The extreme case of applying a prior (Bayes or empirical Bayes) may involve fixing the parameter. Here fixing a parameter is really a special case of applying a prior in which the prior density is fixed on a single value rather than distributed across a range of plausible values. In the majority of population PK and PKPD analyses, a point density prior (= fixed parameter) is chosen, and the value is one that appears to provide reasonable estimation performance. Whether the value is chosen based on a prior study or this study is a matter of choice.

In contrast, to fixing the parameter it is also reasonable to consider a subjective empirical prior that therefore allows the data to speak to the value of the parameter as much as it can, while providing support for parameter regions where the data is not informative. The choice of the mean of the prior may be based on previous fixed trial values from prior runs with the current model and data set or from a previous analysis of the compound using earlier data sets. The value of the variance should be chosen to be informative without being a point density. Again this will have an element of trial and error. Using empirical priors, it is possible to access even a slight amount of information on the value of the parameter. The information content provided by the data can be evaluated by comparing the prior variance (the user defined value) of the parameter against the posterior variance ($=\mathrm{S}{\mathrm{E}}^2$). If the posterior variance is smaller than the prior variance then application of the prior leveraged some information from the data that would otherwise have been lost if the parameter had been fixed.

Finally, it is important to note that application of a (Bayes or empirical) prior in NONMEM may be performed either within a fully Bayesian or a frequentist approach. NONMEM allows the user to specify whether the full density of the posterior distribution is determined (such as using a Markov‐chain Monte‐Carlo sampling approach) or whether a maximum a posteriori like approach is used (e.g., FOCEI). A general description of Bayesian approaches for PKPD is provided in Johnston [22].

An example of application of an empirical prior is illustrated in Supporting Information S3.

2. Case Studies

Two case studies are included in this tutorial.

It is not possible within a workflow to accommodate all possible sources of model instability (excluding user error). The case studies presented here have been selected to raise additional concepts and awareness of sources of model instability and propose solutions to these problems that are extensions of techniques already applied in this tutorial. In the first case study a workflow is illustrated that applies new features of NONMEM using $DESIGN and sobol (in R) to help eliminate model instability. In the second example we explore a setting in which $DESIGN for local deterministic identifiability analysis or structural identifiability analysis did not elucidate the problem and hence the need for SSE in these settings.

2.1. Case Example 1: TMDD Model Instability

2.1.1. Modeling Context

This case explores the instability of a target‐mediated drug disposition (TMDD) model for a monoclonal antibody (mAb). TMDD models are inherently complicated due to their distinct PK and PD characteristics, typically involving four concentration‐time phases: an initial phase, an apparent linear phase, a transition phase, and a terminal phase [23]. Accurate parameter estimation relies on adequate sampling across these phases, but sparse sampling, common in mAb clinical trials, can lead to instability in those mechanistic TMDD models with numerous parameters.

In this example, the TMDD model performed well with a complete sampling dataset but became unstable with a reduced sampling design. For the complete sampling dataset, a total of 640 free drug concentrations and 720 total target concentrations (sum of free target and drug–target complex) were collected from 80 subjects. For the reduced sampling dataset, 400 free drug concentrations and 480 total target concentrations were collected from the same number of subjects. Model simplification was applied to mitigate instability for the reduced design. The model was adapted from a previously published TMDD example in the tutorial for $DESIGN in NONMEM, and NONMEM (v7.5.1) [24].

2.1.2. Model Instability Signs and Symptoms

• Varying parameter estimates under different run settings: It was observed that varying the initial value of $\mathit{Km}$ led to different model outputs. For example, when the initial value of $\mathit{Km}$ was set to 0.1 or 0.01 (nM), minimization terminated due to a rounding error, when set to 1, minimization was successful but the standard error calculation failed. Conversely, when $\mathit{Km}$ was initialized at 0.001, the minimization was successful, but the RSE for $\mathit{Km}$ was extremely high (530%). Additionally, if the between‐subject variance (BSV) of $\mathit{Km}$ was not fixed, it became inflated, with an RSE of 10,367% and a shrinkage of 100%.

• Failed standard errors: The TMDD model either had inflated percent relative standard errors (%RSE > 100%) or the standard error step ($COV) failed.

• Numerical errors during minimization: Various numerical errors occurred during minimization, including termination due to rounding error.

2.1.3. Workflow: Step 1 and Step 2

Structural and deterministic identifiability analyses were conducted using the $DESIGN function in NONMEM. The results of these analyses are presented in Table 2. A hypothetical rich sampling design (0, 0.01, 0.2, 0.6, 3, 50, 64, 81, and 91 h post‐dose) was used as the template data set for structural identifiability analysis. The evaluation of this design, based on the OFV being negative and the RSE of all parameters was acceptable, indicates that the model was structurally identifiable. However, when the model was evaluated using the design from the reduced dataset sampling points (0, 0.5, 3, 5, 12, and 24 h post‐dose), the parameter $\mathit{Km}$ had a high RSE (1495.65%), suggesting that this parameter was not deterministically identifiable.

TABLE 2.

Summary of the evaluation results and final simplified model parameters.

	Nominal value	Structural identifiability using hypothetical rich design	Deterministic identifiability using actual data design	Simplified model
		RSE %	RSE %	Nominal value	RSE %
D‐OFV/OFV		−87.111*	−70.224*	−88.826
$\mathit{CL}$ (L/h)	4.13	4.60	8.31	4.15	6
$\mathit{Vc}$ (L)	76.7	2.96	7.14	68.4	9
$\mathit{Vp}$ (L)	70	2.54	5.89	75.8	5
$Q$ (L/h)	44.7	6.06	21.90	59.9	11
$\mathit{\text{ksyn}}$ (nM/h)	1.09	4.41	11.18	1.16	6
$\mathit{\text{kdeg}}$ (/h)	0.349	3.47	10.92	0.373	4
$\mathit{Km}$ (nM)	0.13	38.12	1495.65	—	—
$\mathit{\text{kint}}$ (/h)	11.1	3.05	7.74	11.5	5
IIV‐ $\mathit{CL}$ (CV%)	25	17.44	21.28	21.9	15
IIV‐ $\mathit{Vc}$ (CV%)	25	16.80	18.40	31.1	17
IIV‐ $\mathit{\text{ksyn}}$ (CV%)	25	16.80	21.12	32.6	14
IIV‐ $\mathit{\text{kint}}$ (CV%)	25	17.76	24.16	28.7	13

Note: $\mathit{CL}$, the clearance of mAbs; D‐OFV = *D‐optimality objective function value; IIV, the inter‐individual variability; $\mathit{\text{kdeg}}$, the elimination of the target; $\mathit{\text{kint}}$, the elimination of drug‐target complex; $\mathit{Km}$, Michaelis–Menten constant; $\mathit{\text{ksyn}}$, the production of the target rate constant; $Q$, the inter‐compartment clearance; RSE%, relative standard error %; $\mathit{Vc}$, the central compartment distribution volume; $\mathit{Vp}$, the peripheral compartment distribution volume. Shaded areas represent very high values of RSE.

2.1.4. Step 3: Sobol Sensitivity Analysis

A sensitivity analysis was performed after identifying the deterministically unidentifiable parameters. The TMDD model was constructed using the rxode2 package (Version 2.1.3) [25], with the 24 h area under the concentration‐time curve (AUC_24h) as the response index to compare variability. The Sensobol package (Version 1.1.5) [26] was employed to conduct the sensitivity analysis, and the results shown that compared to other parameters (e.g., $\mathit{\text{ksyn}}$ and $\mathit{Vc}$), $\mathit{Km}$ was not sensitive to the model output (Figure 2). Consequently, a simplified version of the TMDD model was considered.

FIGURE 2.

The global sensitivity analysis of all the TMDD model parameters. Si (first‐order sensitivity index) indicates how much of the AUC variance is explained by varying one parameter alone, while averaging out the others. Ti (total‐order sensitivity index) captures the overall contribution of a parameter, including both its individual (first‐order) effect and interactions with other parameters. The vertical axis shows each parameter's share of the total model variance (AUC). A higher value indicates greater influence on the AUC, whereas a lower value implies less impact on the model output.

2.1.5. Step 4: Model Simplification

The model assumes a two‐compartment structure, with a single intravenous bolus dose administered into the central compartment. By modeling the drug‐target interactions exclusively within the central compartment, the ODEs (Equations (1), (2), (3)) describe the reaction mechanism.

$$\frac{d{L}_C}{\mathit{dt}}=-\left(k_e+k_{12}\right)L_C+k_{21}{L}_T-\frac{k_{\mathrm{int}}{L}_C{R}_{\mathrm{tot}}}{L_C+\mathit{Km}\mathit{Vc}}$$ (1)

$$\frac{d{L}_T}{\mathit{dt}}=k_{12}{L}_C-k_{21}{L}_T$$ (2)

$$\frac{d{R}_{\mathrm{tot}}}{\mathit{dt}}=-\frac{\left(k_{\mathrm{int}}-k_{\mathrm{deg}}\right)L_C{R}_{\mathrm{tot}}}{L_C+\mathit{Km}\mathit{Vc}}-k_{\mathrm{deg}}{R}_{\mathrm{tot}}+k_{\mathrm{syn}}\mathit{Vc}$$ (3)

Notes: $L_C$ represents the ligand (drug) amount in central compartment; $L_T$ represents the ligand (drug) amount in peripheral compartment; $R_{\mathrm{tot}}$ represents the total amount of the receptor and receptor‐drug complex; $k_e$ represents the clearance rate constant of the ligand in the central compartment; $k_{12}$ represents the transit rate constant of the ligand from the central compartment to peripheral compartment; $k_{21}$ represents the transit rate constant of the ligand from the peripheral compartment to central compartment; $V_C$ represents the distribution volume of the central compartment.

Based on the ODEs 1–3, we performed a simulation of the TMDD model using rxode2. As shown in Figure 3, $\mathit{Km}\times V_C$ (blue line) was much lower than the amount of free drug in the central compartment (red line), which provided the rationale for the model simplification.

FIGURE 3.

The amount‐time profile of the drug and $\mathit{Km}$. The blue line represents the $\mathit{Km}$ value, the red line represents the amount of free drug in central compartment; the shadows represent the 95% confidence interval.

During the first 24 h, the drug amount in the central compartment exceeded the $\mathit{Km}\times V_C$ value by almost 100 fold. Under these conditions, the saturation term $\left(\frac{L_C}{L_C+\mathit{Km}\bullet V_C}\right)$ approximates 1, the model could be simplified as ODEs Equations ((2), (4), (5)):

$$\frac{d{L}_C}{\mathit{dt}}=-\left(k_e+k_{12}\right)L_C+k_{21}{L}_T-k_{\mathrm{int}}{R}_{\mathrm{tot}}$$ (4)

$$\frac{d{R}_{\mathrm{tot}}}{\mathit{dt}}=-k_{\mathrm{int}}{R}_{\mathrm{tot}}+k_{\mathrm{syn}}{V}_C$$ (5)

$$L_C\left(t=0\right)=0,L_T\left(t=0\right)=0,R_{\mathrm{tot}}\left(t=0\right)=k_{\mathrm{syn}}{V}_C/k_{\mathrm{deg}}$$

The final simplified model parameters estimation is summarized in Table 2, with great improvement in the model stability, as exemplified below:

• Successfully minimization and convergence: the simplified model successfully used for parameter estimation, achieving full convergence, with both the estimation minimization (S) and covariance (C) steps completed.

• Reasonable RSE value: all parameters were estimated precisely with RSE for all parameters being well below 30%, demonstrating reasonable precision.

2.2. Case Example 2: Turnover Model Instability

2.2.1. Modeling Context

This case example presents a scenario where a mechanistically reasonable turnover model was unstable despite suitable design quality. Allopurinol is a urate‐lowering drug used mainly for the long‐term management of gout. Its pharmacological action is driven by an active metabolite, oxypurinol that inhibits of the production of uric acid. Oxypurinol therefore reduces serum uric acid concentrations and leads to a reduction in the number of acute gout flares experienced by patients [27]. A turnover model with a delayed urate‐lowering response relative to the oxypurinol plasma concentration profile was mechanistically supported by evidence in the literature [27, 28] and by observed hysteresis in the data [29]. Data from 648 subjects receiving allopurinol doses of 50‐900 mg daily, 5344 serum uric acid concentrations and 2764 plasma oxypurinol concentrations collected under 6 different study designs [5, 30, 31, 32, 33, 34] were available. The model was developed using an IPP structure where the individual oxypurinol pharmacokinetic parameters (empirical Bayes estimates) were first estimated then fixed to the individual values in the subsequent pharmacokinetic‐pharmacodynamic model [35].

2.2.2. Model Stability: Signs and Symptoms

Model building did not result in a satisfactory turnover model for allopurinol. The signs and symptoms of model instability primarily included:

• Varying parameter estimates under different run settings: Different and unreasonable parameter estimates occurred under different run settings. This was particularly evident for the estimated $k_{\mathit{\text{out}}}$ parameter and it's between subject variance (See Equation 7). Estimates of $k_{\mathit{\text{out}}}$ ranged from 0.006–2.97 h⁻¹ under different conditions, including: different initial estimates, different model parametrisations and whether other model parameters were fixed or estimated. Note that the expected value for $k_{\mathit{\text{out}}}$ according to the literature is about 0.027 h⁻¹ [20, 36]. The between subject variance of $k_{\mathit{\text{out}}}$ was either grossly inflated (e.g., 257 CV%) or collapsed to zero.

• Failed standard errors: The case of the allopurinol turnover model, the percent relative standard errors (%RSE) were either inflated (> 100%) or close to zero. In many cases, standard errors were not successful unless the “Unconditional” command was included which allowed standard errors to be calculated (the covariance step) even if the estimation step has terminated.

2.2.3. Workflow: Step 1 and 2

A deterministic identifiability analysis was conducted using the $DESIGN function in NONMEM (v7.5.1, ICON) to explore whether the model instability issues outlined above were the result of the design or data quality. The results of structural and deterministic identifiability analyses are presented in Table 3. For comparison, the parameter estimates for a typical NONMEM run for the allopurinol turnover model are also included in column 3. The D‐OFV ($-\log \left(\det \left(\mathrm{FIM}\right)\right)$) for the allopurinol PKPD model was—35.17 indicating that the model is locally structurally identifiable. The RSE of all parameters were found to be < 50% suggesting that the design used to develop the model contained sufficient information to estimate the turnover parameters with reasonable precision. We therefore conclude that the model, given our data, was locally deterministically identifiable. The subsequent steps in the workflow outlined in Figure 1, that is, a global sensitivity analysis etc., were therefore abandoned in favor of the bespoke exploration of “Internal deterministic identifiability”.

TABLE 3.

The parameters estimates for a typical Allopurinol PKPD model (using a turnover model).

Parameter	Parameter value	Estimated RSE (%) from NONMEM run	FIM derived RSE (%) from $DESIGN
$E_{\max }$	0.71 (fixed)	0.71 (fixed)	Fixed
${\text{urate}}_{\mathrm{BL}}$ (mmol/L)	0.56	12	17.0
$C_{50}$ (μmol/L)	93.1	10	7.0
$k_{\text{out}}$ (1/h)	0.074	125	39.3
${\theta }_{\text{diuretic}}$	1.7	61	5.5
${\omega }_{{\text{urate}}_{\mathrm{BL}}}$ (CV%)	17.9	461	15.7
${\omega }_{{\mathrm{C}}_{50}}$ (CV%)	78.1	715	12.3
${\omega }_{{\mathrm{k}}_{\text{out}}}$ (CV%)	257	239	33.8

Note: $E_{\max }$, maximum urate‐lowering effect; $C_{50}$, drug concentration which results in 50% of maximum response; ${\text{urate}}_{\mathrm{BL}}$, baseline urate concentrations; $k_{\text{out}}$, urate elimination rate constant; ${\theta }_{\mathrm{d}}$, fractional effect of diuretic; ${\omega }^2$, variance for between subject variability. Shaded areas represent large imprecision in parameter estimates.

2.2.4. Bespoke Workflow: Internal Deterministic Identifiability

Internal deterministic identifiability (IDI) refers to scenarios where an otherwise structurally and deterministically identifiable model is unstable because of an “internal” aspect of the model [6]. It was hypothesized that the turnover PKPD model for allopurinol may not always be deterministically identifiable because the time course of serum urate and plasma oxypurinol are similar and may be “flipped” in some individuals. This could manifest between people as a reversal of the oxypurinol elimination rate ($k_e$) and the elimination parameter for urate ($k_{\text{out}}$) that is, when $k_e$ > $k_{\text{out}}$, a PD time delay will be observed relative to the pharmacokinetic profile (as expected for a turnover model) and when $k_e$ < $k_{\text{out}}$ the time course of the PD biomarker is driven by the pharmacokinetic profile. We note that in the local deterministic identifiability analysis summarized above (Steps 1 and 2) $k_e$ and $k_{\text{out}}$ were not flipped and the design was considered locally on the current parameter set. When the parameters are estimated using FOCEI however the population may well be a mix of $k_e$ > $k_{\text{out}}$ and $k_e$ < $k_{\mathit{\text{out}}}$ or perhaps flip between each local state.

To test this hypothesis, a stochastic simulation‐estimation (SSE) analysis was conducted using a PKPD model for a hypothetical drug and turnover model for PD response. One hundred datasets were simulated, each with 100 virtual subjects under a turnover model. The simulation method was based on IPP, that is, the PK parameters were included in the data set and only the PD parameters were estimated.

Three settings were considered, (1) where $k_e>k_{\text{out}}$, (2) $k_e\approx k_{\text{out}}$ and (3) $k_e<k_{\text{out}}$. The simulation scenario was characterized by a drug displaying a one‐compartment pharmacokinetic first‐order elimination and a turnover model linked to an inhibitory $E_{\max }$ model on the rate of the production of the biomarker, given by;

$$\frac{\mathit{dR}}{\mathit{dt}}=R_{\mathit{\text{in}}}\left(1-E_{\max }\frac{C}{C_{50}+C}\right)-k_{\text{out}}R;R_0=\frac{R_{\text{in}}}{k_{\text{out}}}$$ (6)

where, $R_{\text{in}}$ is the zero‐order rate of synthesis, $k_{\text{out}}$ is the first‐order elimination rate constant, and $R_0$ is the baseline response. $E_{\max }$ is the maximum effect of the drug, $C$ is the drug concentration, $C_{50}$ is the drug concentration which results in 50% of maximum response.

In the simulated dataset, the drug was administered by an extravascular route at a dose of 100 mg daily for 30 days. Between‐subject variability was assumed to be CV 20% on all parameters. The residual unexplained variability (RUV) for PK data was 5% for the proportional and 0.1 units for the additive component. An additive RUV of 0.05 units was considered for the PD data. Data for the PK and PD variables were generated under an intensive geometric sampling design on days 1, 2, and 3, and day 30. In addition, a trough (just prior to the next dose) values were generated on Days 4–29.

Simulation was implemented in MATLAB vR2023a (MathWorks Inc) and estimation was performed in NONMEM (v7.5.1, ICON), using the first‐order conditional estimation method with interaction.

Summary results presented in Table 4.

• Scenario 1 ( ${\mathit{k}}_{\mathit{e}}$ (0.1/h) > ${\mathit{k}}_{\mathbf{\text{out}}}$ (0.025/h)): all parameters were precisely estimated (RSE%, 0.4%–22%) for all runs and all parameter estimates were close to their nominal values.

• Scenario 2 ( ${\mathit{k}}_{\mathit{e}}$ (0.1/h) ≈ ${\mathit{k}}_{\mathbf{\text{out}}}$ (0.1/h)): $k_{\text{out}}$ was negatively biased by 0.058/h (58%, 5th and 95th percentiles 0.060–0.056/h) and %RSE was unrealistically low (< 0.1%) in some runs.

• Scenario 3 ( ${\mathit{k}}_{\mathit{e}}$ (0.1/h) < ${\mathit{k}}_{\mathbf{\text{out}}}$ (0.5/h)): resulted in negatively biased estimates in $C_{50}$ by 15 mg/L (15%, 5th and 95th percentiles 45 – > 1000 mg/L), negatively biased $k_{\text{out}}$ by 0.446/h (90%, 5th and 95th percentiles 0.0003–0.057/h), positively biased ${\omega }_{C_{50}}$ (33 versus 20 CV%, 5th and 95th percentiles 0.2 – > 1000%) and negatively biased ${\omega }_{\text{kout}}$ (0.3 versus 5 CV%, 5th and 95th percentiles 0.2 – > 1000%). RSE estimates were > 50% RSE or < 0.1% RSE in 12%–74% of runs in all parameters with the exception of $R_0$ (0%–5% RSE).

TABLE 4.

Summary of the range of RSE (%) values (min‐max) from the 100 simulation‐estimation results for the three scenarios.

Parameter	Expected RSE ($DESIGN)	RSE $\left(k_{\mathrm{e}}>k_{\text{out}}\right)$	RSE $\left(k_{\mathrm{e}}\approx k_{\text{out}}\right)$	RSE $\left(k_{\mathrm{e}}<k_{\text{out}}\right)$
OFV	−35.17	—	—	—
$E_{\max }$	Fixed	Fixed	Fixed	Fixed
$\mathrm{B}{\mathrm{L}}_{\text{urate}}$	17.0	0.4–0.6	0–0.6	0 – > 1000
$C_{50}$	7.0	3–5	0–19	0 – > 1000
$k_{\text{out}}$	39.3	2–3	0–2.4	0 – > 1000
${{\omega }_{\mathrm{BL}}}_{\text{urate}}$	15.7	11–17	0.2–17	0 – > 1000
${{\omega }_{\mathrm{C}}}_{50}$	12.3	10–17	0–18	0 – > 1000
${\omega }_{{\mathrm{k}}_{\text{out}}}$	33.8	4–7	0–35	0 – > 1000

Note: D‐OFV, D‐optimality objective function value; $E_{\max }$, maximum urate‐lowering effect; $C_{50}$, drug concentration which results in 50% of maximum response; ${\text{urate}}_{\mathrm{BL}}$, baseline urate concentrations; $k_{\text{out}}$, urate elimination rate constant; ${\theta }_{\mathrm{d}}$, fractional effect of diuretic; ${\omega }^2$, variance for between subject variability; RSE%, relative standard error %. Units are provided in Table 4.

Overall, the conclusion was that the turnover model was not internally deterministically identifiable when $k_e$ and $k_{\text{out}}$ are close to the same value or that when $k_e$ < $k_{\text{out}}$, that is, when the time‐course of the drug is similar to the biomarker, which is the case for the oxypurinol & urate setting.

2.2.5. A Workable Solution

In the case of allopurinol PKPD model the most reasonable workable solution was to simplify the model structure using an immediate effects PD model since no additional delay is anticipated when $k_e$ and $k_{\text{out}}$ are similar values [37]. We note that other authors have taken a similar approach [38, 39].

3. Concluding Remarks

Evaluating model stability and establishing processes to deal with instabilities is complicated, and no single approach will be able to accommodate all elements. In this tutorial, we describe a reductionist approach in attempting to delineate the root cause as either a function of design or data, and then based on this, some possible approaches to identifying and resolving the issues. We have provided 2 case studies that both highlight the potential use of the proposed framework as well as providing an example that does not entirely fit with the approach but was still resolvable. The benefits of formalizing the workflow are the ease of subsequent documentation of why a particular approach was taken (often why a parameter was fixed).

Even though this is a tutorial, we do not claim that knowledge of the domain or the approach is final. Further work is needed on almost every aspect of what we have presented, and although forums are useful, they are often not practical due to either the complexity of the underlying setting or the confidentiality of some or all of the work. We recommend that pharmacometrics conferences dedicate sessions to understanding and resolving model stability issues.

Conflicts of Interest

The authors declare no conflicts of interest.

Supporting information

Data S1. Structural identifiability analysis example.

Data S2. Deterministic identifiability analysis example.

Data S3. Empirical prior example.

Duffull S. B., Wright D. F. B., Zhu X., Liu X., Abulfathi A., and Hishe H., “A Pharmacometric Workflow for Resolving Model Instability in Model Use‐Reuse Settings,” CPT: Pharmacometrics & Systems Pharmacology 14, no. 10 (2025): 1544–1555, 10.1002/psp4.70049.