Table 1.
Various evaluation graphs in nonlinear mixed effect model a and proposal for a core set of evaluation graphs
| Graphs | In core set | What to expect if the model is correct? | What to do if the graph does not fulfill the requirements? |
|---|---|---|---|
| Basic (prediction‐based) evaluation | |||
| Population‐based graphs | |||
| OBS vs. xPRED (x= ∅, C, and P b ) |
Yes NB: CPRED or PPRED c |
Data points are scattered around the identity line (but not necessarily evenly) | Trends may suggest a modification of structural model, residual error model or interindividual variability model. NB: Trends can also appear in absence of model misspecifications for models highly nonlinear with respect to random effects and large interindividual variability, especially when using PRED. |
| xWRES (x= ∅, C, and P b ) vs. time or xPRED |
Yes NB: CWRES or PWRES c |
Data points are scattered around the horizontal zero‐line (more or less evenly) | Trends may suggest a modification of structural model, residual error model, or interindividual variability model. Trends by conditioning on covariates suggest including covariates. NB: Trends can also appear in absence of model misspecifications for models highly nonlinear with respect to random effects and large interindividual variability, especially when using WRES. |
| xWRES (x= ∅, C, and P b ) vs. covariates | Yes if covariates are considered | No substantial correlation appears | Trends suggest including covariates or changing the covariate model. |
| Individual‐based graphs d | |||
| Individual fits |
Yes NB: at least for some representative individuals |
Observations are distributed evenly around the individual predicted curve | A substantial discordance between observations and predictions suggests a modification of the structural model, parameter variability model, or the residual error model. NB: This diagnostic is not useful for sparse data. |
| OBS vs. IPRED | Yes | Data points are scattered evenly around the identity line. Points cluster closer to the line than with observations vs. PRED, especially when interindividual variability is large | Trends may suggest a modification of the structural model or the residual error model. A lack of trend may not necessarily be associated with absence of model misspecification if data are sparse. |
| IWRES vs. time or IPRED |
Yes NB: Graphs of absolute IWRES vs. IPRED are also informative |
Data points are scattered evenly around the horizontal zero‐line. Most of the points lie within (−1.96 to 1.96) | Trends suggest a modification of structural model or residual error model. A cone‐shaped graph of IWRES vs. IPRED suggests a change in the error model. A lack of trend may not be necessarily be associated with absence of model misspecification if data are sparse. |
| Correlation between EBEs | Yes | No trend is expected in model without correlation between random effects if data are rich (i.e., low eta‐shrinkage) | Correlation between EBE suggests including correlation between random effects unless data are sparse. |
| EBEs vs. covariates | Yes if covariates are considered | No substantial correlation appears between EBE and covariates | Trends between EBE and covariates suggest, including covariates or changing the covariate model. |
| Simulation‐based graphs | |||
| VPC or pcVPC |
Yes NB: The choice between the two depends on the importance of covariates and use of adaptive designs |
Observed percentiles are not systematically different from the corresponding predicted percentiles and are within the corresponding confidence interval | Trends may suggest a modification of the structural model, the residual error model, or the parameter variability model. Trends when conditioning on covariates suggest including covariates or changing the covariate model. |
| NPC coverage | Observed percentiles are within the confidence interval of the corresponding predicted percentiles | Trends may suggest a modification of the structural model, the residual error model, or the interindividual variability model. Trends when conditioning on covariates suggest including covariates. | |
| npd (NPDE) vs. time or PPRED e | Yes | Data points are scattered evenly around the horizontal zero‐line. Most of the points lie within (−1.96 to 1.96). For graphs with observed and predicted percentiles and the confidence intervals: observed percentiles are not systematically different from the corresponding predicted percentiles and are within the corresponding confidence interval | Trends may suggest a modification of structural model, residual error model, or interindividual variability models. A cone‐shaped graph if npd vs. PPRED suggest a change in the residual error model. Trends when conditioning on covariates suggest including covariates. |
| npd (NPDE) vs. covariates | Yes if covariates are considered | No substantial correlation appears | Trends suggest including covariates or changing the covariate model |
CPRED, conditional population predictions; EBE, empirical Bayes estimate; FO, first order; IPRED, individual prediction; IWRES, individual weighted residuals; NB, nota bene; NPC, numerical predictive check; npd, normalized prediction distribution; NPDE, normalized prediction distribution error; OBS, observations; pcVPC, prediction‐corrected visual predictive check; PPRED, simulation‐based population predictions; PRED, FO population predictions; VPC, visual predictive check; WRES, weighted residuals.
aSee text or Supplementary Table S3 for definition of terms. bPPRED and PWRES are denoted EPRED and expectation weighted residuals in NONMEM, respectively. cDepending on the method used for parameter estimation (see text). dCaution in interpretation in case of high shrinkage. enpd is preferred over NPDE for graphical use.