Assessment of the validity of a population pharmacokinetic model for epirubicin

Abstract

Aims

The aim of this study was to evaluate a population model for epirubicin clearance using internal and external validation techniques.

Methods

Jackknife samples were used to identify outliers in the population dataset and individuals influencing covariate selection. Sensitivity analyses were performed in which serum aspartate transaminase (AST) values (a covariate in the population model) or epirubicin concentrations were randomly changed by ±10%. Cross-validation was performed five times, on each occasion using 80% of the data for model development and 20% to assess the performance of the model. External validation was conducted by assessing the ability of the population model to predict concentrations and clearances in a separate group of 79 patients.

Results

Structural parameter estimates from all jackknife samples were within 7.5% of the final population estimates and examination of log likelihood values indicated that the selection of AST in the final model was not due to the presence of outliers. Alteration of AST or epirubicin concentrations by ± 10% had a negligible effect on population parameter estimates and their precision. In the cross-validation analysis, the precision of clearance estimates was better in patients with AST concentrations >150 U l⁻¹. In the external validation, epirubicin concentrations were over-predicted by 81.4% using the population model and clearance values were also poorly predicted (imprecision 43%).

Conclusions

The results of internal validation of population pharmacokinetic models should be interpreted with caution, especially when the dataset is relatively small.

Introduction

A population model for epirubicin pharmacokinetics has previously been developed and used to suggest new dosage guidelines based on the measurement of serum aspartate transaminase (AST) [1]. Current recommendations for epirubicin dosing are based principally on serum bilirubin concentration. Food and Drug Administration guidelines state that the predictive performance of a model should be assessed if it is used to recommend new dosage guidelines [2]. The validity of a population model can be tested by measuring its predictive performance using the same dataset (internal validation) or a different dataset (external validation).

External validation provides the most compelling evidence for the validity of a population pharmacokinetic model [3]. However, a limitation of this approach is that if predictive performance is unsatisfactory, it is not possible to establish whether this is due to invalid model parameters or differences between the datasets [4].

In many cases, an external dataset is not available for validation purposes, and thus internal techniques have to be used. One method is to split the dataset (data-splitting) into a training set and a validation set before commencing the analysis. The disadvantage of this method is that, because the accuracy of the model is a function of sample size, removal of a proportion of the data (usually one-third) may influence the reliability of the model parameter estimates. Furthermore, the results obtained are highly variable and dependent on the split of the data [5]. One refinement of this method is termed cross-validation, and involves repeated data-splitting. This may be advantageous because the results are not dependent on a single data split and a larger proportion of patients can be included in the estimation process with each split. In addition, with cross-validation all subjects contribute to both the estimation process and the validation. Sensitivity analysis can be used to determine the robustness of the model by making small changes to the values of covariates or plasma concentrations [6]. Finally, case deletion diagnostics, such as jackknifing, can be used to identify outliers or influential individuals [7].

In this study, both internal and external validation techniques were used to assess the validity of a previously developed population model for epirubicin clearance.

Methods

Patient data

Model development and internal validation dataset

The development of the model was previously described in a population analysis of epirubicin pharmacokinetics [1]. The dataset used to develop the model was obtained from the study of 105 women with advanced breast cancer, 69 of whom were known to have liver metastases. They were treated at Guy's Hospital, London, UK, with epirubicin as a single agent at doses ranging from 12.5 to 120 mg m⁻², administered as an intravenous injection, with a median infusion duration of 3 min (ranging from 0.4 to 19 min). Eight to 18 blood samples were collected from each patient following the first cycle of treatment, generally up to 48 h postdose.

External validation dataset

Data were available from 79 patients treated at Uppsala University Hospital, Sweden, with epirubicin in combination with fluorouracil and cyclophosphamide (FEC) for early (n = 28) or advanced breast cancer (n = 51) [8]. Sixteen of the patients with advanced disease had liver metastases. FEC was administered at doses of 600/60/600 mg m⁻² or 600/75/900 mg m⁻² of fluorouracil, epirubicin and cyclophosphamide, respectively. Generally, cyclophosphamide was administered as a 15-min intravenous infusion, followed by a bolus dose of fluorouracil and then an intravenous infusion of epirubicin (median infusion time 1 h; range 5 min to 2.3 h). A median of three blood samples was collected from each patient (range 2–6) between 4 min and 25 h after the start of the infusion. Eighty-seven blood samples were collected during the infusion and 148 after the end of the infusion.

The clinical characteristics of the patients used for model development and validation were compared using a two-sided t-test. For characteristics that displayed skewed distributions, logged values were used. Differences in mean values between the data from model development and validation patients were considered to be statistically significant if P < 0.05.

Internal validation

Internal validation techniques were applied to the model development dataset as follows.

Jackknife analysis

One hundred and five new datasets were produced so that each excluded the data from one patient, a different patient being excluded in each dataset. These were termed jackknife samples. Each of these was analysed with NONMEM (FOCE-INTER) using the final population model. Population estimates from the jackknife samples were compared with the final population estimates to identify any individuals that had a large influence on the parameter values.

Likelihood-based method

Individuals influencing covariate (AST) selection were identified using the likelihood-based method previously described by Sadray and coworkers [7]. The influence of an individual was assessed by comparing the difference in the objective function values (OFV) between the basic model (without covariates) and final model (including covariates) when calculated using all the data and when data from the individual in question had been removed:

$$\begin{array}{c}{\text{OFV}\Delta }_{\text{jackknife,i}}=({\text{OFV}}_{\text{final,n}}-{\text{OFV}}_{\text{basic,n}})\\ -({\text{OFV}}_{\text{final,n}-\text{i}}-{\text{OFV}}_{\text{basic,n}-\text{i}})\end{array}$$

where OFV_final,n and OFV_basic,n are the OFVs of the final and basic models, respectively, using data from all individuals, and OFV_final,n–i and OFV_basic,n–i are the OFVs of the final and basic models, respectively, when subject i is excluded. Individuals that support the model have a negative OFVΔ_{jackknife,i’}, whereas individuals that do not support the model have a zero or positive value.

Cross-validation analysis

The dataset was randomly split into five groups of 21 patients. A new dataset was formed, referred to as ‘training data set 1’, comprising 84 patients from four groups. This was used to estimate population parameters using the basic model (excluding AST) and final model (incorporating AST) as described previously [1]. The remaining 21 patients were used for validation and referred to as ‘validation dataset 1’. This process was repeated four times, forming training datasets 2–5, each with a different group removed for validation purposes (validation datasets 2–5).

The basic models and final models derived from each training dataset were used to predict the clearance of epirubicin for patients in the corresponding validation datasets. Individual Bayesian estimates of model parameters were estimated for the validation datasets. The ability of the basic and final population models derived from the training datasets to estimate clearance in the corresponding validation datasets was assessed by calculating prediction errors (P_ei) as follows:

$$\begin{array}{c}{\text{P}}_{\text{ei}} (\%)=(\text{Population CL}-\text{Bayesian CL})/\\ \text{Bayesian CL}\times 100\end{array}$$

where CL is clearance. The root mean squared prediction error (rmse), which describes the imprecision of the population estimate relative to the Bayesian clearance estimate, was calculated from the square-root of the mean squared P_ei (%) values [9].

A paired t-test was performed on the squared prediction errors to assess whether differences in the imprecision of clearance estimated using the basic (without AST) or final model (with AST) was statistically significant (P < 0.05).

Sensitivity analysis

AST concentrations were randomly changed by ±10% of the measured value. Population analysis of the altered data was performed using NONMEM and the population parameters using the altered dataset were compared with those obtained using the original data. In a similar manner, a population analysis was performed using epirubicin plasma concentration data that were randomly changed by ±10%.

External validation

Prediction of concentrations in the validation dataset

Using the population model developed from the original dataset, predicted concentrations (PREDs) were calculated for each patient in the validation dataset at the available blood sampling times, given the dosage history and AST concentration. These predictions were obtained by entering the parameters of the structural model into NONMEM and fixing both interindividual variability in the pharmacokinetic parameters and residual error to zero. The $ESTIMATION command was set as MAXEVAL = 0 and NONMEM was then run. Prediction errors (P_e) were calculated for each PRED and expressed as a percentage of the measured value (DV) as follows:

$${\text{P}}_{\text{e}}=(\text{PRED}-\text{DV})/\text{DV}\times 100$$

A mean prediction error (MPE) was then calculated for each individual and the imprecision and bias determined as described previously.

Prediction of clearance values in validation dataset

The population model was used to predict clearance (CL_POP) for the validation patients, given their measured AST concentration. NONMEM was then used to obtain MAP Bayesian estimates of clearance (CL_Bayesian) for the validation patients using the available concentration measurements with the original population parameter estimates as priors. The program was run with MAXEVAL = 0. Prediction errors (P_ei) were calculated for each predicted clearance and expressed as a percentage of the MAP Bayesian estimate as follows:

$${\text{P}}_{\text{ei}}=({\text{CL}}_{\text{POP}}-{\text{CL}}_{\text{Bayesian}})/{\text{CL}}_{\text{Bayesian}}\times 100$$

The accuracy and imprecision of clearance predicted from the population model were assessed by calculation of rmse and me values (as described above).

Results

A summary of the clinical characteristics of the patients used for model validation and development is given in Table 1. All characteristics were approximately normally distributed with the exception of bilirubin, AST and creatinine. Logarithmically transforming these factors resulted in normal distributions, and these data were used for statistical comparisons. The mean dose for patients in the validation group was 30 mg higher than that in the model development group. Patients in the validation group were, on average, 7 cm taller than those in the model development group and 5 kg heavier; differences which were statistically significant. The mean AST and bilirubin values for patients in the validation group were significantly lower than those in the model development group. Approximately 6% and 20% of the validation group patients had bilirubin levels and AST concentrations, respectively, above the upper reference limit. This is a smaller proportion than in the model development dataset, where approximately 21% and 69% of patients had bilirubin and AST concentrations above the normal range, respectively. The mean creatinine clearance (calculated according to Cockcroft and Gault [10]) in validation group patients was 11 ml min⁻¹ higher than that of the patients in the model development group. The mean ages of patients in the model development and validation groups were similar.

Table 1.

Summary of clinical characteristics in the patients used for validation and model development

	Reference	Validation(n = 79)		Model development (n = 105)
	range	Mean	Range	Mean	Range	Difference
Dose (mg)		118	(30–120)	87	(20–228)	*P < 0.05
Age (years)		53	(32–83)	56	(35–79)	NS
Height (cm)		165	(154–177)	158	(132–175)	*P < 0.05
Weight (kg)		68	(35–102)	63	(37–89)	*P < 0.05
Body surface area (m2)		1.74	(1.3–2.1)	1.64	(1.25–2.2)	*P < 0.05
Bilirubin (µmol l−1)	<23	9	(3–63)	26	(1–282)	†*P < 0.05
AST (U l−1)	<43	58	(11–511)	125	(7–815)	†*P < 0.05
ALT (U l−1)	<34	42	(7–258)	–	–	–
Albumin (g l−1)	30–46	40	(24–49)	37	(25–54)	*P < 0.05
Creatinine (µmol l−1)	50–130	79	(53–197)	82	(47–167)	†NS
Creatinine clearance (ml min−1)		77	(17–137)	66	(21–127)	*P < 0.05
		n		n
Early disease/locally advanced		28		0
Metastatic disease		51		105
Liver metastases		16		69

^*Statistically significant; NS, not statistically significant;

^†t-test performed using logged data.

The structural parameter estimates from the jackknife samples were all within 7.5% of the population estimates. For the interindividual variability parameters, a maximum difference of −17% was observed for Q₂ when patient 402 was removed. The covariance parameters were most sensitive to removal of individual patients with a maximum difference of 19%, which was observed for covariance between V₁ and Q₂ when patient 143 was removed.

The OFVΔ_jackknife,i values are illustrated in Figure 1. Patient 225 had a high positive value for OFVΔ_jackknife,i indicating that this subject was masking the relationship between AST concentration and clearance. The strongest positive influences on the inclusion of AST in the model were caused by patients 413, 228 and 419. These individuals had the highest AST concentrations in the dataset (815, 530 and 489 U l⁻¹, respectively), but were not outliers because they were not set apart from the other individuals. Data from 73 patients produced negative OFVΔ_jackknife,i values and thus were better described by the final model (with AST), whereas data from 32 patients generated positive values and consequently were better described by the basic model (without AST).

Figure 1.

Change in jackknife objective function values calculated using likelihood-based method vs. patient identification number

Population estimates from the five training datasets were similar (within 10%) to those obtained using the complete dataset (Table 2). The rmse indicated an improvement in the imprecision of the clearance estimates in four of the five validation datasets if AST was used compared with estimates using the basic model (Table 3). However, the differences were not statistically significant. Prediction errors obtained using the final model showed only a statistically significant improvement compared with the basic model (P < 0.01) in those patients with AST concentration measurements > 150 U l⁻¹.

Table 2.

Population estimates from the complete dataset and from the training datasets

		Final		Training 1		Training 2		Training 3		Training 4		Training 5
	THETA	Estimate	RSE (%)	Estimate	RSE (%)	Estimate	RSE (%)	Estimate	RSE (%)	Estimate	RSE (%)	Estimate	RSE (%)
*CL (l h−1)	1	74.9	12.3	75.2	8.1	77.9	9.1	71.9	8.1	70.0	9.6	68.0	9.9
V1 (l)	2	9.99	8.7	10.5	9.9	9.84	8.9	10.7	8.7	10.4	9.6	10.3	8.7
Q2 (l h−1)	3	29.5	13.8	30.6	14.8	29.4	14.6	30.3	13.7	29.1	15.0	31.4	13.3
V2 (l)	4	36.0	18.6	36.1	19.1	35.9	18.3	35.9	17.4	33.1	18.9	37.6	17.0
Q3 (l h−1)	5	60.6	6.2	62.2	7.3	58.5	6.9	63.7	6.5	61.7	7.1	61.6	6.6
V3 (lL)	6	790	6.4	780	6.2	752	6.4	785	5.9	751	6.1	795	6.0
Ln AST (CL)	7	0.139	6.4	0.137	4.2	0.140	4.3	0.135	4.8	0.131	6.5	0.130	6.1
ωCL (%)		42.5	15.4	38.9	17.1	38.2	15.5	39.2	16.2	38.9	15.6	41.8	15.6
ωQ3 (%)		31.6	19.5	34.8	19.3	34.8	20.3	35.4	21.2	34.8	19.8	32.7	17.8
ωV3 (%)		43.1	22.1	39.7	23.8	43.8	25.3	43.6	25.7	45.8	22.7	39.0	27.9
σ (%)		23.1	9.29	22.8	10.1	23.4	10.6	23.3	10.1	23.2	10.5	23.1	10.5

^*CL = 74.9 − (74.9 × 0.139 × LnAST).

Table 3.

Imprecision (rmse percentage) of population clearance (CL) estimates calculated with the basic and final models

Validation dataset	Basic populationCL model, rmse (%)	Final populationCL model, rmse (%)	deltarmse (%)	t-testP-value
1	45.1	43.6	−1.53	0.891
2	49.7	61.6	11.9	0.270
3	56.9	44.4	−12.5	0.505
4	94.1	47.1	−47.0	0.151
5	71.1	30.9	−40.2	0.120
Combined	65.8	46.6	−19.2	0.062

Altering AST values by ±10% had a negligible effect on the population parameter estimates and their imprecision (Table 4). In particular, there was no change in Theta 7, which describes the influence of AST on clearance. Likewise, there was little change in the population parameter estimates or their imprecision following alteration of epirubicin concentrations by ±10%.

Table 4.

Population parameter estimates following sensitivity analysis

	Original data		±10% error in AST		±10% error in concentrations
	Estimate	RSE (%)	Estimate	RSE (%)	Estimate	RSE (%)
Pharmacokinetic parameter
θ1	72.9	8.0	73.7	7.9	73.0	8.4
θ2	10.3	8.2	10.3	8.2	10.5	9.0
θ3	30.2	12.7	30.1	12.7	32.1	13.2
θ4	35.7	16.1	35.6	16.0	39.8	17.1
θ5	61.5	6.1	61.6	6.1	61.2	6.4
θ6	772	5.5	772	5.5	782	6.1
θ7	0.135	4.5	0.136	4.4	0.135	4.7
Inter-individual variability
η1	0.16	14.3	0.16	14.3	0.16	14.6
η2	0.12	17.8	0.12	17.9	0.12	17.1
η3	0.18	22.4	0.18	22.4	0.20	22.0
η1–2	0.07	24.9	0.07	24.4	0.07	25.7
η1–3	0.08	23.9	0.08	24.4	0.08	27.7
η2–3	0.10	29.8	0.10	30.3	0.10	29.4
Residual error
σ	0.05	9.3	0.05	9.3	0.06	7.9

RSE, Relative standard error.

Model:

TVCL (l h⁻¹) = θ₁ × (1 − θ₇ × (LnAST)

TVV₁ (l) = θ₂

TVQ₂ (l h⁻¹) = θ₃

TVV₂ (l) = θ₄

TVQ₃ (l h⁻¹) = θ₅

TVV₃ (l) = θ₆

CL = TVCL × EXP η₁

Q₃ = TVQ3 × EXP η₂

V₃ = TVV3 × EXP η₃

where TV is typical value.

Predicted epirubicin concentrations in the validation group using the population model and measured AST concentrations are plotted vs. time in Figure 2. The population model predicted epirubicin concentrations in the validation patients poorly, with an imprecision (rmse) of 102% [95% confidence interval (CI) 96.4, 108]. Concentrations were significantly overestimated (me) in the validation patients by 81.4% (95% CI 68.5, 94.2). This over-prediction was most evident at epirubicin concentrations <100 ng ml⁻¹, which corresponded to the later time points in the concentration—time profile, i.e. the elimination phase.

Figure 2.

Predicted (□) and measured (•) plasma epirubicin concentrations obtained using the population model in the validation patients plotted vs. time

Individual clearance estimates were poorly predicted using the population model with imprecision (rmse) estimated to be 43% (95% CI 37.8, 48.0). Clearance in the validation patients was underestimated by 27.9% and this bias (me) was statistically significant (95% CI −35.2, −20.5). Figure 3 shows clearance values estimated from the population model plotted against MAP Bayesian clearance estimates in validation patients with normal and elevated AST concentrations.

Figure 3.

Clearance estimates for validation patients obtained using the population model plotted against Bayesian estimates. Patients with normal aspartate transaminase (AST) (<43 U l⁻¹) are represented by filled circles and patients with elevated AST (>43 U l⁻¹) by open squares. The dotted line represents the line of unity

Discussion

The aim of this analysis was to validate a previously developed population model for epirubicin pharmacokinetics using internal and external techniques. Although internal validation of the model was successful, major discrepancies arose with external validation. The findings raise important questions regarding the optimal approach for model validation.

Internal validation was undertaken first. The sensitivity analysis that involved ±10% changes in AST or epirubicin concentrations had a negligible influence on parameter estimates and their associated relative standard errors, suggesting that the model was relatively robust and was not sensitive to minor measurement or recording errors. Likewise, the results of the jackknife analysis showed that removal of any one individual did not result in any major changes to the population parameter estimates. As covariance parameters are generally more difficult to characterize than pharmacokinetic parameters, it was not unexpected that these parameters were the most sensitive to the removal of individual data. The likelihood method, which was used to identify individuals influencing covariate selection, indicated that the selection of AST in the final model was not due to the presence of outliers. One individual was identified as masking the relationship between AST and clearance, as clearance was much higher than expected for the AST concentration of 179 U l⁻¹. It is possible that this patient could have had an inaccurate AST value, possibly through haemolysis of the sample, as both bilirubin and albumin concentrations were within the reference ranges. Three individuals were identified as having the strongest influence on the selection of AST in the model. Although these individuals had the greatest influence on the final covariate model, they were not outliers. These results provided evidence that the model was stable and was not influenced by spurious data.

Given the relatively small size of the dataset, splitting the data into training and validation datasets was not undertaken, as it may have compromised model development. Cross-validation was used to test the model for its ability to predict epirubicin clearance. Accurate and precise estimation of this parameter is important because AUC (Dose/Clearance) has previously been shown to be a useful predictor of toxicity and therapeutic outcome [11, 12]. As the dataset contained rich data, the individual MAP Bayesian clearance estimates were likely to be accurate enough to be taken as the ‘true’ values. The results of the cross-validation were disappointing. Although the prediction of clearance was better with the final model (including AST) compared with the basic model (no covariates), the improvement was not statistically significant. Further investigation revealed that a statistically significant improvement in clearance predictions using the final model occurred only in patients with AST concentrations >150 U l⁻¹. This was probably due to the wide variability in clearance that was observed in patients with AST concentrations <150 U l⁻¹. Less variability in clearance was seen in patients with higher AST concentrations [1].

Many other additional techniques are available for internal validation, including bootstrapping [13] and posterior predictive check [14]. However, the most stringent test of a validation model is to assess predictive performance using an external dataset. The results obtained with external validation indicated that the predictive performance of the population model was poor. Predicted concentrations in the validation dataset were almost double the measured concentrations. Likewise, clearance values in the validation patients, estimated using measured AST values and the population model, were not predictive of the MAP Bayesian clearance estimates. Therefore, the question arises as to whether the population model is invalid or whether there are genuine differences between the two populations. As the results from the internal cross-validation indicated that the population model for clearance improved the precision of its estimates only in patients with AST > 150 U l⁻¹ and the validation dataset contained only six patients with AST above this value, the poor precision of the model was not unexpected. However, such pronounced bias was not anticipated and suggests that there are genuine differences in pharmacokinetics between the two datasets. The median clearance value for validation patients with normal liver function (62 l h⁻¹) was consistent with previously reported values (47–87 l h⁻¹) [15, 16], whereas the median clearance value for the model development dataset (38 l h⁻¹) was lower than those previously reported.

The two datasets differed in terms of the length of infusion, the dose, the number and timing of blood samples taken, the proportion of patients with liver dysfunction and the laboratory performing the epirubicin analysis. Furthermore, the population model was developed in patients receiving single agent epirubicin, whereas the validation patients received epirubicin in combination with cyclophosphamide and flurouracil. Co-administration of verapamil [17] or paclitaxel [18] has previously been shown to affect the pharmacokinetics of anthracyclines. Although a relationship between systemic exposures to cyclophosphamide, 5-FU and epirubicin has not previously been found [19], the possibility of a drug interaction cannot be ruled out. A cross-over study would need to be conducted to investigate this possibility. There could also be differences in the general health status of the patients. The model development dataset included patients with advanced disease, a poor prognosis and in general poor health, whereas in the validation dataset approximately half of the patients were receiving epirubicin as adjuvant therapy and therefore were likely to be in better health overall. Small differences in height, weight, albumin and creatinine clearance were observed between the model development and validation patients, but these would not be expected to result in such marked differences in the pharmacokinetics of epirubicin. Other physiological differences between the validation and model development patients, such as genetic, diet or other environmental factors, cannot be excluded. In addition, it is not possible to rule out differences in the bioanalytical assay between the two sites, since a cross-validation study was not performed.

The true clearance values in the validation patients were not known, and as estimates of clearance were obtained using sparse data, they may be unreliable. However, potential inaccuracies in the MAP Bayesian clearance estimates in the validation dataset do not account for the over-predictions observed in the concentration data.

The validation of the population model may have been clearer if the validation dataset had included patients on single-agent epirubicin therapy and similar health status, liver function and dosage regimens as observed in the model development dataset. However, a favourable outcome with such a dataset may have led to the erroneous conclusion that the population model was valid for all patients, despite the reality of heterogeneity between patients and treatment schedules.

This analysis has underlined some of the difficulties in using an external dataset to validate a population model and also the limitations of extrapolating results from a relatively small patient group to the population as a whole. Furthermore, it draws into question the value of internal validation techniques. The differences in the pharmacokinetics of epirubicin observed for the model development and validation datasets indicate that more work is required to identify the many factors that influence the pharmacokinetics of epirubicin, so that more safe and effective dosage guidelines can be developed.

The authors thank Quintiles Ltd for supporting L.D.R.