Abstract
The main objective of any modeling exercise is to provide a rationale for effective decision making during drug development. The aim of the current simulation experiment was to evaluate the properties of predictive check as a covariate model qualification technique and, more importantly, to introduce and evaluate alternative criteria to qualify models.
Original concentration-time profiles (yod) were simulated using a 1-compartment model for an intravenous drug administered to 25 men and 25 women. The typical clearance for male subjects (TVCLm) was assumed to be 5-fold higher than that for female subjects (TVCLf). Fifty such trials under the same design were generated randomly. Predictive check was used as the model qualification tool to study predictive performance of true (males ≠ females) and false (males=females) models in the context of maximum likelihood estimation. For each yod, 200 replications were generated to study the properties of a discrepancy variable, a statistic that depends on the model, and a test statistic, a statistic that does not depend on the model. Several qualification criteria were evaluated in assessing predictive performance, such as, predictive p-value (Pp), probability of equivalence (peqv), and probability of rejecting the null hypothesis (data=model) using the Kolmogorov-Smirnov test (pks).
The Pp value was calculated using sum of squared errors as a discrepancy variable. For both of the models, the Pp values uniformly ranged between 0 and 1. The pattern of Pp values suggests that qualification of the false model is unlikely. For both of the models, the range of peqv is about 0.95 to 1.0 for concentration at 0.5 hours. However, this is not the case for the concentration at 4 hours, which is primarily dependent on the clearance. The false model (0.35 to 0.50) has poor predictive performance compared with the true model (0.65 to 0.80) using peqv. The pks suggests no difference in the distributions of replicated and original concentrations at all of the time points for both of the models.
Discrepancy variables cannot aid in rejecting false models, whereas the use of a test statistic can aid in rejecting false models. However, selection of an informative test statistic is challenging. As far as the qualification criteria are considered, the equivalence-based comparison of a test statistic is more informative than a significance-based comparison. No convincing evidence exists in the literature demonstrating the added advantages of predictive check as a routine model qualification tool over the existing tools, such as diagnostic plots or mechanistic reasoning. However, when a model is to be used for designing a trial, it should at least be able to regenerate the data used to build the model. In such cases, predictive check might offer insights into potential inconsistencies.
Keywords: model validation, model qualification, posterior predictive check, test statistic, discrepancy variable
Full Text
The Full Text of this article is available as a PDF (311.2 KB).
Footnotes
The views expressed in this article are those of the authors and do not necessarily reflect the official views of FDA.
References
- 1.Sheiner LB, Steimer JL. Pharmacokinetic/pharmacodynamic modeling in drug development. Annu Rev Pharmacol Toxicol. 2000;40:67–95. doi: 10.1146/annurev.pharmtox.40.1.67. [DOI] [PubMed] [Google Scholar]
- 2.Gobburu JV, Marroum PJ. Utilisation of pharmacokinetic-pharmacodynamic modelling and simulation in regulatory decision-making. Clin Pharmacokinet. 2001;40:883–892. doi: 10.2165/00003088-200140120-00001. [DOI] [PubMed] [Google Scholar]
- 3.Holford NHG, Hale M, Ko HS, Steimer JL, Sheiner LB. Simulation in Drug development: good practices.Center for Drug Development Science. Available at: http://cdds.georgetown.edu. Accessed January 1, 2004.
- 4.Wahlby U, Jonsson EN, Karlsson MO. Assessment of actual significance levels for covariate effects in NONMEM. J Pharmacokinet Pharmacodyn. 2001;28:231–252. doi: 10.1023/A:1011527125570. [DOI] [PubMed] [Google Scholar]
- 5.FDA. Guidance: Population Pharmacokinetics.FDA guidance page. Available at:http://www.fda.gov/cder/guidance. Accessed January 13, 2004.
- 6.Rubin D. Bayesianly justifiable and relevant frequency calculations for the applied statisitcian. Ann Stat. 1984;12:1151–1172. doi: 10.1214/aos/1176346785. [DOI] [Google Scholar]
- 7.Gelman A, Carlin JB, Stern HS, Rubin DB. Model checking and sensitivity analysis. In: Gelman A, editor. Bayesian Data Analysis. London: Chapman & Hall; 1995. pp. 161–189. [Google Scholar]
- 8.Gelman A, Meng XL. Model checking and model improvement. In: Gilks WR, Richardson S, Spiegelhalter DJ, editors. Markov Chain and Monte Carlo in Practice. London: Chapman & Hall; 1996. pp. 189–202. [Google Scholar]
- 9.Yano Y, Beal SL, Sheiner LB. Evaluating pharmacokinetic/pharmacodynamic models using the posterior predictive check. J Pharmacokinet Pharmacodyn. 2001;28:171–192. doi: 10.1023/A:1011555016423. [DOI] [PubMed] [Google Scholar]
- 10.Gelman A, Meng XL, Stern HS. Posterior predictive assessment of model fitness via realized discrepancies. Statistica Sinica. 1996;6:733–807. [Google Scholar]
- 11.Guttman I. The use of a concept of a future observation in the goodness-of-fit problems. J R Statist Soc B. 1967;1:83–100. [Google Scholar]
- 12.Tiao GC, Xu D. Robustness of maximum likelihood estimates for multi-step predictions: the exponential smoothing case. Biometrika. 1993;80:623–641. doi: 10.1093/biomet/80.3.623. [DOI] [Google Scholar]
- 13.Gelman A, Goegebeur Y, Tuerlinckx F, Van Mechelen I. Diagnostic checks for discrete data regression models using posterior predictive simulations. J Royal Stat Soc: C. 2000;49:247–268. doi: 10.1111/1467-9876.00190. [DOI] [Google Scholar]
- 14.Duffull SB, Aarons L. Development of a sequential linked pharmacokinetic and pharmacodynamic simulation model for ivabradine in healthy volunteers. Eur J Pharm Sci. 2000;10:275–284. doi: 10.1016/S0928-0987(00)00085-3. [DOI] [PubMed] [Google Scholar]
- 15.Duffull SB, Chabaud S, Nony P, Laveille C, Girard P, Aarons L. A pharmacokinetic simulation model for ivabradine in healthy volunteers. Eur J Pharm Sci. 2000;10:285–294. doi: 10.1016/S0928-0987(00)00086-5. [DOI] [PubMed] [Google Scholar]
- 16.Friberg LE, Freijs A, Sandstrom M, Karlsson MO. Semiphysiological model for the time course of leukocytes after varying schedules of 5-fluorouracil in rats. J Pharmacol Exp Ther. 2000;295:734–740. [PubMed] [Google Scholar]