Predictive Performance of a Gentamicin Population Pharmacokinetic Model in Neonates Receiving Full-Body Hypothermia

Abstract

Background

Population pharmacokinetic (popPK) models derived from small PK studies in neonates are often underpowered to detect clinically important characteristics that drive dosing. External validation of such models is crucial. In this study, the predictive performance of a gentamicin popPK model in neonates receiving hypothermia was evaluated.

Methods

A previously published gentamicin popPK model was developed in neonates with hypoxic ischemic encephalopathy undergoing hypothermia using a retrospective single-institution (UCSF) dataset. The predictive performance of this model was evaluated in an external retrospective dataset from UCSF (Validation A) and another from Duke University (Validation B). Both institutions used the same hypothermia protocol and collected similar clinical and PK data. Gentamicin dosing and samples were collected per routine care. Predictive performance was evaluated by quantifying the accuracy and precision of model predictions and using simulation-based diagnostics to detect bias in predictions.

Results

41 neonates (18 Validation A, 23 Validation B) with median (range) gestational age of 40wks (33–42) and birth weight of 3.3kg (1.9–4.6) and 76 samples (55% troughs, 33% and 28% drawn at 24 and 36h post dose, respectively) were analyzed. The model adequately predicted gentamicin concentrations from the same institution (Validation A; median average fold error [AFE]=1.1 and numerical prediction distribution error [NPDE] p-value>0.05) but under-predicted concentrations from the outside institution (Validation B; median AFE=0.6 and NPDE p-value<0.05).

Conclusion

The model demonstrated adequate predictive performance for an external dataset in the same institution but not from an outside institution. Larger sample sizes, use of data from multiple institutions, and external evaluation in development of popPK models in neonates may improve generalizability of dosing recommendations arising from single-institution studies.

BACKGROUND

Characterizing the pharmacokinetics (PK) of drugs in neonates is exceedingly important. Neonates have rapidly maturing physiologic processes (such as renal function and hepatic metabolism) relevant to drug disposition.^1,2 In addition, most drugs used in neonates have not been adequately studied or approved for use in this population.³ In recognition of this scarcity of information, the 2012 Food and Drug Administration Safety and Innovation Act requires inclusion of neonates in studies unless the disease does not occur in neonates or studies are not safe or feasible in that population.⁴

PK studies in neonates are scarce because conducting early-phase studies in this population is challenging. This is due to limited access to the population of interest, limited blood volume available for PK sampling, low parental consent rates associated with perceived risk of the investigation, and a scarcity of neonatal pharmacology investigators available to design the study and analyze the PK data.^5–7 Due to these limitations, population PK (popPK) models for neonates are often developed from a single medical center using a small sample with sparse PK sampling and are evaluated using the same information used to develop the model.^8–10

PopPK models derived from small PK studies in the highly variable neonatal population are likely often underpowered to determine clinically important characteristics that drive dosing. Although this approach provides meaningful information about the relationship between drug disposition and neonatal development, such popPK models can result in a limited characterization of the true variability in PK parameters and an inability to detect and properly characterize the relationship between covariates and PK parameters. Thus external validation approaches are especially important in this vulnerable population. In addition, because of the current pediatric drug development framework, evidence-based dosing in neonates lags well behind the advent of new agents, leading to pronounced regional differences in therapeutic approaches. Therefore, extrapolation of results beyond the range of data used to develop the models might not be appropriate, and there is a risk for lack of generalizability of results to the patient population of interest.

Gentamicin is commonly used for empiric treatment of presumed infection in neonates with hypoxic ischemic encephalopathy (HIE) undergoing hypothermia.¹¹ To optimize dosing in this setting, a popPK model was developed using retrospective medical record data from a single institution.⁸ This study suggested a need for dose adjustment of gentamicin in these neonates due to decreased gentamicin clearance—a finding that increases the importance of external validation prior to widespread adoption of new dosing guidelines. The objective of our study was to evaluate the predictive performance of this gentamicin popPK model.

MATERIALS AND METHODS

Study Design

To evaluate the predictive performance of the popPK model in neonates with HIE, two test datasets for model evaluation were generated, one at the University of California–San Francisco (UCSF) (Validation A) and one at Duke University Medical Center (Validation B). The studies were approved by the institutional review boards at both sites and did not require informed consent.

UCSF investigators collected retrospective data for neonates with presumed HIE who underwent whole-body hypothermia (33.5°C) and received gentamicin between 2011 and 2012 (Validation A dataset). They excluded neonates with congenital renal or cardiac disease, and those requiring extracorporeal membrane oxygenation. They only included the first measured peak and trough concentration per neonate. Gentamicin samples were measured using an immunoassay method. Serum creatinine concentrations were determined using the Jaffé rate method. Details of the study are previously described.¹²

We collected retrospective data for neonates admitted to the Duke intensive care nursery between January 1, 2006, and December 31, 2008, with presumed HIE who underwent whole-body hypothermia (33.5°C for 72 hours, initiated within 6 hours of birth¹³) and received gentamicin (Validation B dataset). We collected the following clinical data from each neonate: birth weight (BW), gestational age (GA), postnatal age (PNA), sex, serum creatinine (SCR), base deficit, lactate, and arterial or capillary pH, and the following PK data: dose amount, dose start date/time, infusion duration, sampling date/time, and gentamicin concentrations collected as part of therapeutic drug monitoring during routine medical care. Gentamicin samples were measured in the Clinical Laboratory Improvement Amendments-certified Duke University Medical Center laboratory using an immunoassay method run on an Abbott AxSym (until January 2008) or Beckman Coulter Synchron (after January 2008) instrument. The linear range of the assay was 0.5–12.0 µg/mL, with a between-run precision of ≤6.9%. Serum creatinine concentrations were measured using enzymatic (until January 2008) and Jaffé (after January 2008) methods.

Statistical Analysis

We assessed for differences in baseline patient characteristics between the Validation A and Validation B datasets using the two-sample test of proportions for categorical variables and Wilcoxon rank-sum test for continuous variables.

Pharmacokinetic Analysis

A gentamicin popPK model in neonates with HIE was developed by UCSF investigators and was described previously.⁸ Briefly, 29 neonates (GA and PNA median [range] of 40 weeks [36–42] and 2 days [1–4], respectively) and 47 gentamicin plasma concentrations were included to build the popPK model. Gentamicin PK was characterized with a one-compartment model, including BW as a covariate for volume of distribution (V), and BW and SCR as covariates for clearance (CL) as follows: CL(L/h) = 0.111 × [BW (kg)/3.3]^0.75 × [1/SCR(mg/dL)]^0.566 and V(L) = 1.56 × [BW(kg)/3.3]. The estimate of inter-individual variability in CL was 16.1% and residual variability (proportional error model) was 16.2%.

We evaluated the predictive performance of the popPK model using different methods. First, we assessed prediction performance at the individual level. We used the dosing information and clinical data from the external datasets to simulate gentamicin concentrations. To evaluate the accuracy and precision of model predictions, we calculated the average fold error (AFE = 10^{[1/N Σlog(predicted/observed)]}) and absolute average fold error (AAFE = 10^{[1/N Σ|log(predicted/observed|)]}) of predicted vs. observed gentamicin concentrations, respectively, for each neonate. We set a prediction acceptance criteria of 2 for AAFE.^14,15 As we were estimating accuracy with sparse sampling, we used a flexible acceptance criteria of 0.5–1.5 (50% under- or over-prediction) for AFE.

Second, we assessed prediction performance at the population level by simulating 1000 gentamicin concentrations per time point using the popPK model parameters for fixed and random effects. We plotted the observed data in the external datasets, along with the 5^th, 50^th, and 95^th percentiles of the simulated data (visual predictive checks [VPC]) to assess the degree of overlap between observed and simulated data for each time point. We set prediction acceptance criteria of 85% of observed concentrations falling within the 90% prediction interval.

We also conducted a numerical assessment of the predictive performance at the population level. Using the simulated data of the VPCs for each dataset, we computed normalized prediction distribution errors (NPDE) for the 1000 simulated vs. observed datasets.^16,17 We set prediction acceptance criteria of a normal distribution of NPDEs with a mean of zero (t test) and variance of one (Fisher test for variance). A p-value of <0.05 was considered inadequate prediction of the datasets. We also plotted NPDEs vs. predicted concentration and time after last dose to visually assess systematic bias in predictions.

Lastly, we used the model to estimate popPK parameters for each of the test datasets and calculated relative errors of these parameters compared with the published model estimates (relative error of estimate = [published – test] / published × 100%). We also evaluated goodness of fit and model stability using standard popPK goodness-of-fit plots, precision-of-parameter estimates, shrinkage, and condition number.¹⁸

As the start and stop date/time of cooling was not consistently documented in the medical record, we limited the analyses to gentamicin PK samples collected within 96 hours of birth to approximate the duration of hypothermia under standard-of-care protocols.¹³ Model predictive performance using the Validation B dataset was evaluated using all gentamicin concentrations collected during those 96 hours, as well as with a limited subset (Validation B limited dataset); in the Validation B limited dataset, post-therapy samples (referring to one or more samples drawn after the trough and before the next dose) were dropped. The Validation B limited dataset was created to approximate the study design that was used to build the popPK model.

We used Stata 12 (College Station, TX, USA) for plotting and calculation of AFE, AAFE, and relative errors. We used NONMEM 7.2 software (ICON, Ellicott City, MD, USA) to perform PK parameter estimation and model simulations. We used the NPDE add-in package for R software (http://www.r-project.org/) for computation and statistical analysis of NPDEs.

RESULTS

Neonates in the two external datasets had statistically different gentamicin doses, dosing intervals, GA, PNA, and base deficits (Table 1).

TABLE 1.

Demographics and clinical characteristics of study populations

	Validation A	Validation B	Validation B limited	p value	All*
PNA <96 h
Neonates (N)	18	23	23		41
Samples (N)	33	43	27		76
Female, N (%)	11 (61)	15 (65)		0.79	63
Median dose (mg/kg) during the study	5.0 (4.5–5.1)	3.5 (3.2–3.7)		<0.001	3.6 (3.2–5.1)
Dosing interval (hours) during the study	36 (30–36)	28 (4.5–68)		0.002	36 (4.5–68)
GA (weeks)	40 (38–42)	39 (33–41)		0.002	40 (33–42)
PNA (days) at the time of the 1st gentamicin sample	3 (1–3)	2 (1–3)		<0.001	2 (1–3)
PNA (days) at the time of the 1st SCR sample	0 (0–2)	0 (0–0)		0.11	0 (0–2)
Birth weight (kg)	3.4 (1.9–4.0)	3.3 (2.5–4.6)		0.66	3.3 (1.9–4.6)
SCR (mg/dL)	1.0 (0.6–1.3)	1.0 (0.4–1.9)		0.94	1.0 (0.4–1.9)
1st arterial or capillary pH	7.0 (6.6–7.2)	6.9 (6.4–7.3)		0.75	7.0 (6.4–7.3)
Base deficit (mmol/L)	14 (3–24)	21 (4–40)		<0.001	17 (3–40)
Vasopressor use, N (%)	9 (50)	13 (57)		0.68	22 (54)

Data are median (range), unless otherwise indicated. Laboratory values are at the time of the first gentamicin sample, unless otherwise indicated.

Values for N in the “All” column equal the sum of Validation A and Validation B.

Model-predictions were symmetric about the line of unity for the Validation A dataset, indicating a lack of bias (Figure 1A). However, Validation B and Validation B limited concentrations were systematically under-predicted (Figure 1B–C).

Figure 1.

Prediction of gentamicin concentrations in test datasets using the popPK model. A, diamonds = Validation A; B, triangles = Validation B; C, squares = Validation B limited.

Median fold errors were near one for the Validation A dataset but showed model under-prediction (AFE<1) for the Validation B and Validation B limited datasets (Table 2 and Figure 1). Nearly all Validation A neonates had AFE and AAFE values within acceptance criteria, while only 70% and 87% of Validation B and Validation B limited neonates, respectively, met acceptance criteria (Table 2). AFE and AAFE ranges were less wide in the Validation A vs. the Validation B datasets (maximum AAFE value of 1.9 vs. 4.7, respectively, Table 2).

TABLE 2.

Prediction errors

	Validation A	Validation B	Validation B limited
AFE	1.1 (0.6–1.9)	0.6 (0.2–2.2)	0.7 (0.4–1.3)
Neonates with AFE between 0.5 and 1.5 (n [%])	17 (94)	16 (70)	20 (87)
AAFE	1.3 (1.1–1.9)	1.9 (1.0–4.7)	1.8 (1.2–2.3)
Neonates with AAFE <2 (n [%])	18 (100)	16 (70)	20 (87)

Values are median (range), unless otherwise indicated.

The fraction of observed concentrations falling within the simulated 90% prediction intervals for VPCs were 88% for Validation A, 50% for Validation B, and 71% for Validation B limited (Figure 2). As evaluated using NPDEs, the model performed well with the Validation A dataset (observations randomly distributed around the zero line and p > 0.05) (see figure, Supplemental Digital Content 1A–B, which shows NPDEs). However, NPDEs for the Validation B datasets were not normally distributed (p < 0.05) and showed bias within the range of predicted concentration and time after dose (see figure, Supplemental Digital Content 1C–F).

Figure 2.

Visual predictive checks. Lines = 5^th, 50^th, and 95^th percentiles of 1000 simulations using the pop PK model; A, diamonds = Validation A; B, triangles = Validation B; C, squares = Validation B limited.

When the popPK model was fitted to the Validation A dataset, the relative error of the population CL estimate compared with the published model estimate was 5.4%; relative errors were higher using the Validation B datasets (−17.1 for both; see table, Supplemental Digital Content 2, which shows parameter estimates). The data were uninformative in the estimation of inter-individual variability in CL for the Validation B limited dataset as evidenced by high shrinkage (>30%).

DISCUSSION

We used external data from two institutions to evaluate a popPK model for gentamicin in neonates with HIE who underwent hypothermia, using several graphical and numerical methods. According to all model evaluation methods, the Validation A dataset was well-predicted. The Validation B and limited datasets were not well-predicted by the model using any of the evaluation methods.

The popPK model evaluated in this study was derived from a small number of neonates and gentamicin concentrations. As a result, the model only included inter-individual variability for one of two primary PK parameters (CL). These results suggest that the predictive performance of popPK models developed with small sample sizes and sparse sampling in neonates are very sensitive to the data used to generate the model. This observation is supported by the differences in PK parameter estimates between the published model and test datasets (see table, Supplemental Digital Content 2). A larger sample size for model development will enable more reliable and precise characterization of popPK parameters and covariate relationships (e.g., between SCR and CL). Combining the data and refining the model may also enable detection of additional covariate effects that would increase model generalizability.

It is possible that the predictive performance of the popPK model was affected by differences in the data structure used to generate and evaluate the model. One important difference is that the model was derived from a dataset that was limited to the initial peak and trough concentration for each neonate, while the Validation B dataset contained a total of two peak concentrations in the entire dataset and multiple post-therapy samples. When post-therapy samples were removed from the Validation B dataset (Validation B limited dataset), model performance was improved. However, the pre-specified model acceptance criteria were not met for the Validation B limited dataset, and the model continued to under-predict concentrations for most neonates.

There were several baseline differences between neonates in the two external datasets (Table 1). Gentamicin dose and dosing interval were expected to differ, as institutional gentamicin dosing for Validation A was 5 mg/kg every 36 hours and 3.5–4 mg/kg every 24 hours or 36 hours for Validation B. Differences in GA and PNA were statistically, but not clinically, significant. Although we did not detect differences in SCR, it is possible that differences in fluid management or renal function were responsible for differences in base deficit.

It is difficult to estimate the effect of SCR on gentamicin CL in neonates. SCR is not a good predictor of the glomerular filtration rate in newborns and is largely reflective of maternal SCR concentrations.^19,20 As the timing of SCR collection and median SCR concentrations did not vary between validation datasets, these factors did not likely affect predictive performance. In addition, the analytical method used to measure SCR can influence the modeling results; however, for neonates, there is no validated conversion factor for comparison of SCR concentrations measured using the enzymatic and Jaffé methods.²¹ As the model development and Validation A datasets used the Jaffé method, and the Validation B dataset used both methods over the course of data collection, it is possible that the analytical method for SCR quantification influenced the predictive performance of the model.

PopPK models for two widely used and often studied drugs in neonates have been externally evaluated. The predictive performance of a popPK model for morphine and its metabolites was evaluated using six external datasets from four institutions containing data for neonates and infants up to one year old (total of 120 neonates and infants, 705 morphine samples, and >660 samples for each of two metabolites). The model was developed using data from 248 postoperative or artificially ventilated children <3 years old who contributed 728 morphine samples, and >600 samples for each of two metabolites. Using observed vs. predicted concentration plots, NPDE analysis, and using the model to estimate popPK parameters for each dataset, this study found that the model had adequate predictive performance for external datasets from neonates and infants who were postoperative, artificially ventilated, or on extracorporeal membrane oxygenation with continuous venovenous hemofiltration, but under-predicted concentrations for a dataset from infants on extracorporeal membrane oxygenation without continuous venovenous hemofiltration.¹⁸

Another study evaluated six popPK models for vancomycin in neonates using one external dataset of 112 vancomycin concentrations from 78 neonates. The models were developed from datasets collected in separate institutions (all different countries) containing a median of 114 neonates and infants (range 19–374) and median 648 vancomycin samples (range 88–1103) per dataset. Using VPCs and NPDE analysis, the study found that there were significant differences in predictive performances of the models in the independent dataset depending on the analytical method used to measure SCR and vancomycin, and stated that recommending one of the models was beyond the scope of the analysis.²¹

PopPK models are infrequently externally validated.²² As a result, caution should be used in extrapolating popPK models beyond the patient population and sampling scheme available in the model development dataset. As the present study indicates, a model may not generalize well to very similar patients from a different institution. Previous external evaluation studies have been conducted for popPK studies in neonates. But none to date have evaluated a model developed from very sparse retrospective data collected as part of clinical care as was done in the present study. Greater use of external evaluation approaches are needed for popPK models derived from small sample sizes with sparse sampling designs. Extreme caution should be applied [0]when changing clinical practice (i.e., dosing recommendations) in the setting of sparse sampling and prior to adequate evaluations in prospective clinical trials, particularly in vulnerable populations (e.g., critically ill neonates).

CONCLUSION

A published popPK model of gentamicin in neonates with HIE receiving hypothermia demonstrated adequate predictive performance in an external dataset within the same institution, but not in an outside institution. Combining the data and exploring the inclusion of additional covariates, such as analytical methods for gentamicin and SCR, may improve predictive performance across institutions. External evaluation is recommended for assessment of popPK models developed in neonates and is critical to support dosing changes based on such analyses.

Supplementary Material

1. Supplemental Digital Content 1.tif.

Normalized distribution prediction errors (NPDE). A–B, diamonds = Validation A; C–D, triangles = Validation B; E–F, squares = Validation B limited.