Evaluation of model‐integrated evidence approaches for pharmacokinetic bioequivalence studies using model averaging methods

Abstract

Conventional approaches for establishing bioequivalence (BE) between test and reference formulations using non‐compartmental analysis (NCA) may demonstrate low power in pharmacokinetic (PK) studies with sparse sampling. In this case, model‐integrated evidence (MIE) approaches for BE assessment have been shown to increase power, but may suffer from selection bias problems if models are built on the same data used for BE assessment. This work presents model averaging methods for BE evaluation and compares the power and type I error of these methods to conventional BE approaches for simulated studies of oral and ophthalmic formulations. Two model averaging methods were examined: bootstrap model selection and weight‐based model averaging with parameter uncertainty from three different sources, either from a sandwich covariance matrix, a bootstrap, or from sampling importance resampling (SIR). The proposed approaches increased power compared with conventional NCA‐based BE approaches, especially for the ophthalmic formulation scenarios, and were simultaneously able to adequately control type I error. In the rich sampling scenario considered for oral formulation, the weight‐based model averaging method with SIR uncertainty provided controlled type I error, that was closest to the target of 5%. In sparse‐sampling designs, especially the single sample ophthalmic scenarios, the type I error was best controlled by the bootstrap model selection method.

Study Highlights.

• WHAT IS THE CURRENT KNOWLEDGE ON THE TOPIC?

Non‐compartmental analysis (NCA) is the conventional method used in bioequivalence (BE) analysis for oral drug products, while bootstrap NCA serves as the conventional approach in the BE analysis for ophthalmic drug products. Both of these suffer from low power in many cases. Model‐integrated evidence (MIE) approaches exist, but may suffer from inflated type I error.

• WHAT QUESTION DID THIS STUDY ADDRESS?

Can model averaging be leveraged to improve the type I error control and achieve high power in MIE approaches for BE analysis of oral and ophthalmic formulations?

• WHAT DOES THIS STUDY ADD TO OUR KNOWLEDGE?

Model averaging MIE BE methods can adequately control type I error and achieve high power in oral and ophthalmic studies. Accurately estimating and integrating parameter uncertainty is crucial to MIE BE methods. Bootstrap model selection and weight‐based model averaging with sampling importance resampling (SIR) uncertainty are promising methods for MIE BE analysis.

• HOW MIGHT THIS CHANGE DRUG DISCOVERY, DEVELOPMENT, AND/OR THERAPEUTICS?

Our novel MIE approach may provide an alternative to traditional BE approaches, especially in studies where only sparse sampling is possible, such as studies of ophthalmic formulations.

INTRODUCTION

To receive approval for marketing a generic product, applicants must demonstrate the absence of a significant difference in the rate and extent of absorption of the active ingredient compared with a reference listed drug (RLD). For drug products that are systemically absorbed, bioequivalence (BE) of a test product to the RLD is traditionally determined by comparing geometric means of maximum concentration (C _max) and area under the concentration–time curve (AUC) between the two formulations, as calculated by non‐compartmental analysis (NCA). ¹ , ² However, conventional BE approaches using NCA typically require rich pharmacokinetic (PK) sampling and demonstrate low power in situations of sparse PK sampling. ³ Model‐integrated evidence (MIE) approaches for BE evaluation proposed by Hu et al. ³ and Dubois et al. ⁴ , ⁵ show promise, especially in PK studies with sparse sampling, but have shown inflated type I error. ⁵ Using methods with inflated type I error will cause non‐bioequivalent formulations to be accepted as bioequivalent at higher‐than‐expected rates. ³

In these previously proposed MIE BE methods, a single population PK model that contains treatment effects, and period and sequence effects for crossover designs, is fitted to the data. The estimated treatment effects on primary PK parameters (and their uncertainty) are converted into treatment effects for the secondary PK parameters (and their uncertainty) used for BE determination (AUC, C _max) by means of a bootstrap ³ or the delta method. ⁵ A conclusion on whether the test product is BE to the RLD can then be reached with two one‐sided t‐tests (TOST). ¹

There has recently been progress made in controlling the type I error of MIE methods ⁶ , ⁷ , ⁸ through improvement in testing methods and improvement in parameter uncertainty estimation. However, all of the MIE methods referenced above rely on a single model to describe the system and assume that model to be true. Methods based on a single model are subject to model misspecifications and model selection bias and may significantly underrepresent the uncertainty. Model averaging can alleviate some of these problems by allowing multiple alternative descriptions of the system. ⁹ , ¹⁰

Model averaging has been used to alleviate problems caused by the selection of a single model. ⁹ , ¹⁰ , ¹¹ Any pharmacometric model has misspecifications compared with the biological system and population that it intends to describe, and any predictions from a single model assume that the model is true. Traditional model building examines different models and selects the best model for the data at hand, estimating the parameter values and parameter uncertainty under that model. Using a single model in this fashion carries the misspecifications of that model into the model predictions and onward to any downstream analyses. This can be counteracted by model averaging, whereby several models in a (predefined) model pool contribute to predictions, which are weighted based on their respective goodness‐of‐fit. ⁹ The misspecifications in one feature of an otherwise good model can thus be balanced by complementary features in other models. This can be thought of as accounting for the uncertainty of the model structure within the scope of the model pool.

Since MIE BE shows particular promise in sparse data situations, ⁴ , ⁸ it is interesting to examine BE testing using model averaging approaches in these situations. In this work, we examine two simulated scenarios with sparse PK data: (1) oral formulations with sparse sampling; and (2) ophthalmic drug formulations with extremely sparse sampling (1–2 samples per subject). With such sparse data, complex (and perhaps more mechanistic) models may not fit the data, but model averaging enables different models in a model pool to capture different features that are not simultaneously identifiable in a single model.

Ophthalmic formulations present particular challenges as they often do not cause quantifiable systemic concentrations of the drug. Current FDA guidance on studies to determining BE through PK studies for generic ophthalmic products, outlined by Shen and Machado, ¹² uses PK samples from the aqueous humor of the eye at different timepoints after administration. These studies use either parallel designs with a single sample in each subject, or crossover designs with one sample from each eye. The single‐sample nature of these trials excludes traditional NCA methods. Instead, a bootstrapped NCA method ¹² , ¹³ , ¹⁴ is employed, where NCA is performed on a mean concentration curve of bootstrapped datasets. Our hypothesis is that the model averaging approaches presented in this paper may be significantly better than the bootstrapped NCA method for BE assessment of this type of data.

This work presents MIE BE analysis procedures that use model averaging and parameter uncertainty estimates to control type I error while maintaining high power. It evaluates them through simulation experiment of BE studies for ophthalmic and oral formulations simulated under bioequivalent and non‐bioequivalent scenarios, respectively, and compares the power and type I error behavior of these methods to conventional BE methods recommended by the FDA.

METHODS

We have developed MIE BE analysis procedures that employ two of the model averaging approaches described by Aoki et al. ⁹ These procedures produce estimates of model parameters and uncertainty, which in turn produce predictions of metrics of interest for BE. In weight‐based model averaging, we examined three different methods for estimating parameter uncertainty: (1) sandwich covariance matrix, (2) bootstrap, and (3) sampling importance resampling (SIR). Bootstrap model selection captures parameter uncertainty and sampling uncertainty in the bootstrap step of the procedure.

Model averaging methods

In the presented work, we explored two approaches of model averaging: (1) bootstrap selection, and (2) relative weights.

Model averaging by bootstrap selection

In bootstrap model selection, ⁹ visualized in Figure 1, the study subject level data of each BE study were bootstrapped with replacement into N _Boot datasets stratified to maintain the same proportions of treatments, sampling schedules, and treatment sequences (for crossover studies) as the original study (N _Boot = 500 in the presented simulation experiments). All eligible models in the model pool were fitted to each of the bootstrapped datasets. For each bootstrapped dataset, the model with the lowest AIC was selected (subject to each model passing a practical identifiability test, see below), producing N _Boot selected models with associated parameter estimates. These were used for simulations, presented in section “Model‐averaged simulations” below. See Data S1 for code examples.

FIGURE 1.

Overview of the model averaging by bootstrap selection bioequivalence procedure.

Model averaging by relative weights

In model averaging by relative weights, visualized in Figure 2, all models in the model pool were fitted to the BE study data. Each model was then assigned a weight based on its AIC according to Equation 1, ¹⁷

$$w_m=\frac{e^{\frac{-{\mathrm{AIC}}_m}{2}}}{\sum _{i=1}^M{e}^{\frac{-{\mathrm{AIC}}_i}{2}}}$$ (1)

where w _m is the relative weight for model m, AIC_m is the AIC for model m, AIC_i is the AIC for the ith candidate model, and M is the number of models in the model pool. Parameter uncertainty was estimated using either the sandwich covariance matrix from a NONMEM covariance step, SIR, ¹⁸ or nonparametric bootstrap. See Data S1 for code examples. For each model that passed a practical identifiability test (see section “Model validation for model averaging,” below), sets of parameter values from these uncertainty distributions were used for simulations to determine BE (see section “Model‐averaged simulations” below). Due to extensive runtimes and relatively poor performance, the weight‐based model averaging with bootstrap uncertainty was excluded from the two ophthalmic scenarios for type I error evaluation and the parallel design ophthalmic scenario for power evaluation.

FIGURE 2.

Overview of the model averaging by relative weights procedure for bioequivalence evaluation.

Model validation for model averaging

To be included in model‐averaged simulations, each model had to pass a practical identifiability test using saddle‐reset. ¹⁹ The estimation was deemed acceptable if the difference in objective function value (OFV) before and after saddle‐reset was less than 1, and no parameter estimate changed by more than 10%. For nested candidate models, the larger model was excluded unless a likelihood‐ratio test indicated a significant improvement (p = 0.05) in model fit. The smaller model was included regardless of inclusion or exclusion of the larger model. In weight‐based model averaging, models that receive a relative weight of less than 5% were excluded from the BE determination in order to reduce the number of models.

Model‐averaged simulations

In order to produce test‐to‐reference geometric mean ratios of the PK metrics, each included model was used for population simulation (for a total of N = 500 simulations per dataset, for this work). In weight‐based model averaging, this corresponded to using the model to produce a number of simulations proportional to its relative weight (w _m × N simulations for model m). Parameter uncertainty distributions were then generated for each included model by the respective uncertainty method (sandwich covariance matrix, bootstrap, or SIR), and parameter vectors for simulation were sampled from these distributions. In bootstrap model selection, each included model was used to simulate a portion of populations corresponding to the number of times out of 500 that the model was selected for each bootstrapped dataset. The selected model for each bootstrap sample, and the respective final parameter values from the estimation of that model in that bootstrap sample were used to simulate a population of n = 1000 subjects. Individual values of AUC_last, AUC_∞, and C _max were calculated for both formulations using closed‐form model‐based solutions (since such solutions exist for all models in the investigated model sets). The simulation design was a single‐dose, two‐way crossover design with one sequence. In order to isolate the formulation effects, the period and sequence effects were disregarded in the simulation for BE determination. Inter‐occasion variability was also ignored, but the inter‐individual variance was increased by the inter‐occasion variance.

Model‐averaged bioequivalence test

BE parameters in a population of n = 1000 subjects were simulated. The individual test‐to‐reference ratios, and then the geometric means of these ratios across the population were calculated. Uncertainty distributions of the geometric means were generated as described in the previous section. The nonparametric 90% confidence intervals of these AUC_last, AUC_∞, and C _max geometric mean ratios, across the N = 500 population simulations, were then computed. The 5th and 95th percentiles were compared with the pre‐determined ratio limits for BE, that is, 80% and 125%. The BE assessment was performed using the TOST method with the null hypothesis that the test product is not bioequivalent, that is, $H_0:\log \left({\mathrm{par}}_{\text{test}}/{\mathrm{par}}_{\mathrm{ref}}\right)<\log \left(0.8\right)\mathit{\text{or}}\log \left({\mathrm{par}}_{\text{test}}/{\mathrm{par}}_{\mathrm{ref}}\right)>\log \left(1.25\right)$, where $\mathrm{par}$ was AUC_∞, AUC_last, or C _max. The two products were deemed bioequivalent If the 90% confidence intervals of the geometric mean ratios for all examined parameters fell entirely within the allowed confidence interval as outlined by regulatory guidance, ²⁰ $\left[\log \left(0.8\right);\log \left(1.25\right)\right]$.

Simulation experiments

Simulation models (“True” models)

A one‐compartment model with first‐order absorption and first‐order elimination was adapted from the theophylline PK model implementation in Dubois et al. ⁵ The model predicts concentration y for individual i at observation j and occasion k:

$$\begin{array}{c}{y}_{\mathrm{ijk}}=\frac{F_{\mathrm{i}\kappa }{D}_i{k}_{a,\mathrm{ik}}}{C{L}_{\mathrm{ik}}-V_{i,\kappa }{k}_{a,\mathrm{ik}}}\left(e^{-k_{a,\mathrm{ik}}{t}_{\mathrm{ijk}}}-e^{-\frac{C{L}_{\mathrm{ik}}}{V_{\mathrm{ik}}}{t}_{\mathrm{ijk}}}\right)\left(1+{\epsilon }_{\mathrm{ijk}}\right)\end{array}$$ (2)

where F is relative bioavailability, D is dose, k _a is the absorption rate constant, V is the volume of distribution, CL is clearance, and $t_{\mathit{ijk}}$ is time after dose. The proportional residual error, ${\epsilon }_{\mathit{ijk}}$, is assumed to be normally distributed around zero, with a variance of σ ²:

$$\begin{array}{c}{\epsilon }_{\mathrm{ijk}}\sim N\left(0,{\sigma }^2\right)\end{array}$$ (3)

Parameter values for Individual i and occasion k were given by:

$$\begin{array}{c}{F}_{\mathrm{ik}}={\theta }_F·{\beta }_{\mathrm{TRT},F}·{\mathrm{e}}^{{\mathrm{\eta }}_{\mathrm{F},\mathrm{i}}}·e^{{\mathrm{\kappa }}_{\mathrm{F},\mathrm{ik}}},{\eta }_{F,i}\sim N\left(0,{\mathrm{\omega }}_F^2\right),{\mathrm{\kappa }}_{\mathrm{F},\mathrm{ik}}\sim N\left(0,{\mathrm{\gamma }}_F^2\right)\end{array}$$ (4)

$$\begin{array}{c}{k}_{\mathrm{a},\mathrm{ik}}={\theta }_{{\mathrm{k}}_{\mathrm{a}}}·e^{{\eta }_{k_a,i}}·e^{k_{k_a,\mathrm{ik}}},{\eta }_{k_a,i}\sim N\left(0,{\omega }_{k_a}^2\right),{\kappa }_{k_a,\mathrm{ik}}\sim N\left(0,{\gamma }_{k_a}^2\right)\end{array}$$ (5)

$$\begin{array}{c}{V}_{\mathrm{ik}}={\theta }_V·e^{{\eta }_{V,i}}·e^{k_{V,\mathrm{ik}}},{\eta }_{V,i}\sim N\left(0,{\omega }_V^2\right),{\kappa }_{V,\mathrm{ik}}\sim N\left(0,{\gamma }_F^2\right)\end{array}$$ (6)

$$\begin{array}{c}C{L}_{\mathrm{ik}}={\theta }_{\mathrm{CL}}·e^{{\eta }_{\mathrm{CL},i}}·e^{k_{\mathrm{CL},\mathrm{ik}}},{\eta }_{\mathrm{CL},i}\sim N\left(0,{\omega }_{\mathrm{CL}}^2\right),{\kappa }_{\mathrm{CL},\mathrm{ik}}\sim N\left(0,{\gamma }_{\mathrm{CL}}^2\right)\end{array}$$ (7)

where $\theta$ is the population parameter, $\eta$ is the individual deviation from the typical subject with variance ${\omega }^2$, $\kappa$ is the occasion deviation from the individual value with variance ${\gamma }^2$, and ${\beta }_{\mathrm{TRT},F}$ is the treatment effect on bioavailability and set as 1 for the reference product.

The population parameter values used to simulate the BE data were bioavailability, ${\theta }_F=100\%$, absorption rate constant, ${\theta }_{k_a}=1.48{\mathrm{h}}^{-1}$, volume of distribution, ${\theta }_V=480\mathrm{mL}$, and clearance, ${\theta }_{\mathrm{CL}}=40.36\mathrm{mL}/h$. Variance for inter‐occasion variability in any parameter was ${\gamma }^2=0.0225$ (~15% CV), variance for inter‐individual variability in any parameter was set to ${\omega }^2=0.25$ (approximately 50% CV) and the variance of residual error is set to σ² = 0.01 (10% CV). The true model was altered in the oral and parallel ophthalmic scenarios by removing the inter‐individual and inter‐occasion variability on some parameters (setting their variances to zero). The true model in the oral scenarios included inter‐individual variability on k _a, V, and CL, and inter‐occasion variability on F (${\gamma }_{k_a}^2$ = ${\gamma }_V^2$ = ${\gamma }_{\mathrm{CL}}^2$ = ${\omega }_F^2$ = 0). The true model in the parallel ophthalmic scenario included inter‐individual variability on all parameters, but no inter‐occasion variability (${\gamma }^2$ = 0).

Scenarios

Single‐dose BE studies of oral and ophthalmic formulations were simulated under different study designs, detailed in Table 1. For oral formulation scenarios, BE data were simulated using a single‐dose, two‐treatment, two‐period crossover design, simply referred to as a crossover design in this work, with either a rich sampling design (24 subjects, 10 samples/subject), or a sparse‐sampling design (40 subjects, 3 samples/subject), the intermediate and sparse designs used by Dubois et al. ⁵ For ophthalmic formulation scenarios, sample(s) were collected once for each subject and each period and simulations were performed for both a parallel design with 480 subjects (a single sample collected from one eye for each subject) and a crossover design with 120 subjects (two samples collected from each subject, one from each eye).

TABLE 1.

Overview of studied simulation scenarios.

Description	True mean‐ratio a (%)	Subjects	Sampling times (hours after dose)
Oral formulation
Power analysis
Crossover design with rich sampling	90	24	0.25, 0.5, 1, 2, 3.5, 5, 7, 9, 12, 24
Crossover design with sparse sampling	90	40	0.25, 3.35, 24
Type I error analysis
Crossover design with rich sampling	125	24	0.25, 0.5, 1, 2, 3.5, 5, 7, 9, 12, 24
Crossover design with sparse sampling	125	40	0.25, 3.35, 24
Ophthalmic formulation
Power analysis
Parallel design	90	480	0.25, 1.5, 5, 15, 24 b
Crossover design	90	120	0.25, 1.5, 5, 15, 24 b
Type I error analysis
Parallel design	125	480	0.25, 1.5, 5, 15, 24 b
Crossover design	125	120	0.25, 1.5, 5, 15, 24 b

^a β _TRT,F value used for test formulation in simulations.

^b For ophthalmic formulations: In parallel trials, each subject is sampled once, in crossover trials, each subject is sampled twice at the same time after the dose, once per study period. Subjects are assigned evenly across timepoints.

For each study design, simulations with different expected BE outcomes were carried out to evaluate type I error and the power of the studied methods. The expected BE outcome was imposed by simulating with a proportional test treatment effect on bioavailability (β _TRT,F , relative bioavailability, see Equation 4). The power of the compared methods was examined under bioequivalent scenarios, with a relative bioavailability of 90% (β _TRT,F  = 0.9) used to simulate BE datasets. The type I error rates were examined using non‐bioequivalent scenarios with a treatment effect on the relative bioavailability of 125% (β _TRT,F  = 1.25). This change in relative bioavailability is equivalent to a change of 125% in AUC_∞ (where applicable), AUC_last, and C _max between test and reference formulations for the simulation models described above. Additional type I error scenarios with a treatment effect of 80% were excluded due to long runtimes. Power and type I error was calculated for each method and each BE metric. In addition, an overall power or type I error was also obtained where BE was concluded when all required metrics approved bioequivalence (AUC_∞, AUC_last, and C _max for oral formulations, and AUC_last and C _max for ophthalmic formulations). For each scenario, 500 BE study datasets were simulated and the applicable BE methods for the scenario were applied to each simulated BE study. Type I error was considered controlled if it fell within 3.2%–7.0%, which is the 95% binomial proportion confidence interval around the expected 5% type I error rate for 500 trials (calculated using the Cornish‐Fisher expansion ²¹ implementation in R ²² ). Type I error rates above 7.0% were considered not adequately controlled, and below 3.2% were considered overly conservative.

Model pools

The pools of candidate models for the model averaging approach were different for the three different BE study design scenarios (see Table 2). The three model pools were based on the simulation (“true”) model, and thus made up of one‐compartment models with first‐order absorption, first‐order elimination, and proportional residual error. The study designs did not enable the estimation of all simulation model parameters. For example, the simulated study designs assumed an oral dose of the studied drug products, and therefore total bioavailability (${\theta }_F$) could not be estimated in any of the designs and was fixed to a value of 1. All relative changes in bioavailability based on treatment can be estimated in the standard BE study designs investigated in this work.

TABLE 2.

Overview of the model pools used in the simulation experiments.

Ophthalmic formulation parallel design model pool							Ophthalmic formulation crossover design model pool							Oral formulation model pool
Model number	Inter‐individual variability				Treatment effects		Model number	Inter‐individual variability				Treatment effects		Model number	Inter‐individual variability				Treatment effects		Inter‐occasion var.	Sequence effects		Period effects
	F	Ka	V	CL	F	Ka		F	Ka	V	CL	F	Ka		F	Ka	V	CL	F	Ka	F	F	Ka	F	Ka
1			✓	✓	✓	✓	1	✓	✓	✓	✓	✓	✓	1		✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
2		✓		✓	✓	✓	2		✓	✓	✓	✓	✓	2		✓	✓	✓	✓	✓	✓			✓	✓
3	✓			✓	✓	✓	3	✓		✓	✓	✓	✓	3		✓	✓	✓	✓	✓	✓	✓
4				✓	✓	✓	4	✓	✓		✓	✓	✓	4		✓	✓	✓	✓	✓	✓
5			✓		✓	✓	5	✓	✓	✓		✓	✓	5		✓	✓	✓	✓		✓
6		✓			✓	✓	6			✓	✓	✓	✓	6		✓	✓	✓		✓	✓
7	✓				✓	✓	7		✓		✓	✓	✓	7			✓	✓	✓		✓
8					✓	✓	8	✓			✓	✓	✓	8		✓		✓	✓		✓
9			✓	✓	✓		9				✓	✓	✓	9		✓	✓		✓		✓
10		✓		✓	✓		10			✓		✓	✓	10				✓	✓		✓
11	✓			✓	✓		11	✓				✓	✓	11	✓	✓	✓	✓	✓		✓
12				✓	✓		12	✓	✓	✓	✓	✓		12	✓	✓	✓	✓	✓
13			✓		✓		13		✓	✓	✓	✓
14		✓			✓		14	✓		✓	✓	✓
15	✓				✓		15	✓	✓		✓	✓
16					✓		16	✓	✓	✓		✓
							17			✓	✓	✓
							18		✓		✓	✓
							19	✓			✓	✓
							20				✓	✓
							21			✓		✓
							22	✓				✓

Note: Rows indicate numbered models, and the features of each model are indicated by checkmarks in the applicable columns. All models used were one‐compartment models with first‐order absorption, first‐order elimination, and proportional error. Shaded rows indicate that the model was excluded from model averaging by bootstrap model selection to reduce runtimes.

Treatment effects were added to all absorption parameters of the simulation model (${\beta }_{\mathrm{TRT},F}$ and ${\beta }_{\mathrm{TRT},k_a}$), and for crossover scenarios sequence and period effects were also added to the absorption parameters. Based on the resultant full model evaluating those design elements, a series of reduced candidate models were generated that included different combinations of features: covariate effect (treatment, sequence, and period effects on absorption parameters) and random effects (inter‐individual variability and inter‐occasion variability). The selection of candidate models was determined based on model identifiability, which was investigated using PopED, ¹⁶ or based on estimation success rate in a preliminary simulation experiment. For example, models estimating >2 random effect parameters were not identifiable and excluded for the ophthalmic parallel study scenario. Since there is only a single observation per individual in the ophthalmic parallel scenario, inter‐individual variability is essentially an extension of the residual error in those models, and the two may not be independently identifiable. Inter‐occasion variability cannot be estimated in ophthalmic scenarios, and it was not included in either of the ophthalmic scenario model pools.

To reduce runtimes of simulation experiments, certain models that were expected to have little chance to impact the final model averaging results were removed. For example, the models without treatment effect on F were not included in ophthalmic BE studies. The model pool for bootstrap model selection, the method with the longest runtime, was further reduced to the 10 models with the highest average weights in weight‐based model averaging based on preliminary simulation results. The features of the models in each model pool are detailed in Table 2.

Standard NCA‐based BE evaluation

For the oral product formulations, the model averaging approaches were compared with the NCA‐based method outlined for BE studies using a single‐dose, two‐way crossover design in the FDA guidance. ²⁰ In this method, individual AUC_∞, AUC_last, and C _max were calculated on log scale (using NCA in the NCAPPC package ¹⁵ ) and fitted to a linear mixed‐effects model according to Equation 8.

$$\begin{array}{c}\log \left({\theta }_{m,i,p}\right)={\alpha }_m+{\beta }_{m,\mathrm{TRT}}TR{T}_{i,p}+{\beta }_{m,\mathrm{SEQ}}SE{Q}_i+{\beta }_{m,\mathrm{PER}}PE{R}_p+{\eta }_{m,i}+{\epsilon }_{m,i,p}\end{array}$$ (8)

where ${\theta }_{m,i,p}$ is the value of metric m (AUC_∞, AUC_last, or C _max) of subject i during period p, ${\alpha }_m$ is the expected value of the metric at the reference levels of all design covariates. TRT _i,p , SEQ _i , and PER _p are the treatment, sequence, and period indicators, and ${\beta }_{m,\mathrm{TRT}}$, ${\beta }_{m,\mathrm{SEQ}}$, and ${\beta }_{m,\mathrm{PER}}$ are the corresponding coefficients for metric $m$. The random effects are η _m,i , for inter‐individual variability in metric m, and ε _m,i,p is the residual error. The 90% parametric confidence interval of the treatment effect coefficient for each metric, ${\beta }_{m,\mathrm{TRT}}$, was then calculated, and if it fell entirely within the 80%–125% range the test treatment was accepted as bioequivalent.

For the ophthalmic product formulations, the model averaging approaches were compared with a nonparametric bootstrapped NCA method for aqueous humor PK data described by Shen and Machado, ¹² which is currently recommended by the FDA in relevant product‐specific guidances. ¹⁴ In the simulated scenario, the drug was administered at a set time before cataract surgery, with a single sample of aqueous humor taken during the surgery. Crossover designs are possible, where two samples are taken at the same sample time after administration, one from each eye in two surgeries on separate occasions. At each sampling time, all individuals measured at that time were bootstrapped with replacement 10⁵ times. In parallel studies, this was done separately for the two treatments, while for crossover studies both test and reference measurements were included when a subject was selected by the bootstrap. Values for C _max and AUC_last were calculated using NCA for each treatment in each bootstrapped dataset from the geometric mean concentrations at each sampling time, obtaining 10⁵ test‐to‐reference ratios of C _max and AUC_last. AUC_∞ is not calculated in this method. ¹² Nonparametric confidence intervals for each metric ratio were formed from the 5th and 95th percentiles of the ratios from all bootstraps, and the test formulation passed the BE test if that interval fell within 80%–125%, see Figure S1 in Data S1.

Software

Estimation, including parameter uncertainty estimation, was performed using NONMEM 7.4, ²³ facilitated by Perl speaks NONMEM (PsN). ²⁴ Run management and statistical calculations were performed in R. ²² The NCA metrics were calculated using the NCAPPC package. ¹⁵

RESULTS

Type I error

Type I error results for each PK metric and the overall BE determination are visualized in Figure 3. Several model averaging approaches displayed inflated type I errors. This behavior was apparent for weight‐based model averaging with bootstrap and covariance matrix uncertainty in all crossover scenarios, for model averaging by bootstrap model selection in the crossover design with rich sampling for oral formulations, and for weight‐based model averaging with SIR uncertainty in the parallel design for ophthalmic formulations. In scenarios with sparse sampling and higher numbers of subjects, that is, the ophthalmic scenarios and the sparse crossover oral scenario, the type I error remained best controlled with the bootstrap model selection method. In the oral scenario with sparse sampling, the bootstrap model selection method had exactly 7.0% type I error, which is the upper limit of the acceptable interval. Where this method failed to adequately control type I error, however, was in the oral scenario with rich crossover design and few subjects. Weight‐based model averaging with SIR uncertainty successfully controlled type I error in both oral formulation scenarios.

FIGURE 3.

Method comparison of type I error for bioequivalence tests of AUC_∞, AUC_last, C _max, and all metrics combined (overall) in the ophthalmic product scenarios (top) and oral product scenarios (bottom). Type I error results within the 95% binomial proportion confidence interval (top and bottom dashed lines) are not statistically significantly different from the expected 5% (middle dashed line). Please note that not every method was examined in every scenario. The bootstrapped NCA method does not produce AUC_inf. N represents the number of subjects and n represents the number of samples per subject and period in each scenario.

The type I errors of NCA‐based methods fell within the expected range for all individual metrics in all scenarios except the parallel ophthalmic scenario, where C _max was overly conservative. However, the NCA BE result was not consistent across metrics within each simulated study, meaning that different metrics failed different studies. This led to the overall NCA BE result being overly conservative in all scenarios (0.0%–3.0%). See Figure S2 in Data S1 for a density plot of the GMR and its 5th and 95th percentiles to visualize how the GMR is distributed in the different methods of the oral formulation, rich sampling, and crossover design scenario.

Power

Power results for each PK metric and the overall BE determination are visualized in Figure 4. Model averaging approaches demonstrated considerably higher power over NCA methods when the BE data were simulated with 90% test‐to‐reference relative bioavailability (bioequivalence). The improvement in power is strongest in AUC_∞ and C _max. In the parallel ophthalmic scenario, C _max power for model averaging methods ranged between 33.6% and 50.8%, compared with the 4.2% power of the bootstrap NCA method. In the oral scenarios, AUC_∞ power for model averaging methods was 8.6%–18.0% higher than the standard NCA method.

FIGURE 4.

Method comparison of power for bioequivalence tests of AUC_∞, AUC_last, C _max, and all metrics combined (overall) in the ophthalmic product scenarios (top) and oral product scenarios (bottom). Error bars represent the 95% binomial proportion confidence interval of the power. Please note that not every method was examined in every scenario. The bootstrapped NCA method does not produce AUC_inf. N represents the number of subjects and n represents the number of samples per subject and period in each scenario.

For ophthalmic formulations, the crossover design with 120 subjects enabled all methods to achieve higher power than in the parallel design with 480 subjects. Similar to the type I error results, the power of model averaging methods was more consistent across metrics, leading to overall BE results close to the results of the individual metrics, while NCA‐based methods had lower overall power. The parallel ophthalmic scenario was severely underpowered for the bootstrap NCA method, with an overall power near 0%.

Model weights and selection frequency in model averaging approaches

To provide additional context for the comparison between model averaging approaches, we compared the weights assigned to each model in weight‐based model averaging (Equation 1) and the weights, or rather the relative selection frequencies, for each model in bootstrap model selection. The weights for the 16 models (listed in Table 2) and 500 simulated BE studies in the parallel ophthalmic simulation experiment are shown in Figure 5. This comparison reveals that the weight‐based model averaging method favored assigning high weight to one model in each study, while bootstrap model selection selected a wider range of models with lower frequencies to describe each dataset.

FIGURE 5.

Weights of the weight‐based model averaging method (top) and relative selection frequencies of the bootstrap model selection method (bottom) for the 16 models and 500 simulated BE studies in the parallel ophthalmic BE study example.

DISCUSSION

The model averaging methods presented here vastly outperformed the power characteristics of the NCA‐based methods for BE assessment, while two of the methods, model averaging by bootstrap model selection and weight‐based model averaging with SIR uncertainty, maintained acceptable type I error in respective designs.

MIE approaches can account for the variability and uncertainty of different sources (statistical model, parameter uncertainty, etc.), which leads to more precise estimation of BE metrics. More precise BE metrics produce narrower geometric mean ratio confidence intervals than those from NCA methods. Narrower confidence intervals are more likely to fit within the BE limits, and thus the MIE methods have higher power. However, the uncertainty must be accurately estimated and accounted for in MIE methods, as they may not adequately control type I error otherwise.

MIE approaches can evaluate C _max at any timepoint, while NCA methods are restricted to the observed timepoints based on the study design. MIE approaches also have a more elaborate and constrained time‐dependency, providing more accurate AUC extrapolation to infinity. ²⁵ , ²⁶ This gives MIE methods C _max and AUC_∞ power advantages, especially striking for C _max in the ophthalmic scenarios, and for AUC_∞ in the oral scenarios (see Figure 4). The MIE methods also produced more consistent BE results across the three PK metrics in each dataset. The inconsistency across metrics in the NCA methods can be seen in Figures 3 and 4 as the overall BE results were lower than the individual results for each PK metric. It is worth noting that the MIE approaches allow AUC_∞ comparisons for ophthalmic formulations, which is not performed in the currently used bootstrapped NCA method. For AUC_last the different BE methods produced much more similar results, with the NCA methods displaying higher power and higher type I error in AUC_last compared with the other metrics. The designs studied here are not ideal for performing NCA analyses, and NCA would achieve higher power in designs with richer and longer sampling. They are, however, still relevant designs that enable method comparisons.

Two methods stood out as the best performers in the studied scenarios: model averaging by bootstrap model selection and weight‐based model averaging with SIR uncertainty. Weight‐based model averaging with SIR uncertainty controlled the type I error in the oral formulation scenarios but failed to adequately control type I error in the ophthalmic formulation scenarios. Model averaging by bootstrap model selection best controlled the type I error in the ophthalmic formulation scenarios, with a slightly overly conservative type I error and a slight reduction in power compared with the weight‐based model averaging methods, but it failed to control the type I error rate in the oral formulation scenario with rich sampling in a study with relatively few subjects (24 subjects). These results reflect the strengths and weaknesses of the respective uncertainty methods. The bootstrap is robust even in sparse‐sampling scenarios but requires enough subjects to provide an adequate representation of the population. The SIR method is robust with fewer subjects but struggles to accurately estimate parameter uncertainty in very sparse data.

It is worth noting that the bootstrap model selection method is much more time‐consuming than the weight‐based model averaging methods. We observed overly conservative type I error rates for the bootstrapped NCA and model averaging by bootstrap model selection methods in the parallel design ophthalmic scenario (Figure 3 top left). This behavior is the result of the nature of performing two one‐sided tests when the variability is large due to extremely sparse data, and other factors such as an inadequate sample size, or using a parallel design. The high variation can produce cases in the type I error scenarios where the 95th percentile is within the 80%–125% range, but the 5th percentile falls below 80%, which leads to the observed low type I error, that is, the type I error (or more precisely the type I error of the upper limit test minus the type II error of the lower limit test) is overly conservative compared with the expected 5%. It can even lead to a result where none of the 500 cases have both the 5th and the 95th percentiles within 80%–125%. In addition, the high variation caused low power (Figure 4 top left), with bootstrap NCA having 0% overall power (AUC_last and C _max combined) in the parallel design ophthalmic scenario. Improved properties may be observed with an adequately powered study.

Compared with single‐model MIE approaches, model averaging approaches allow for fewer model assumptions. For example, several plausible formulation effects and statistical models can be tested within the scope of a single analysis. The inevitable misspecifications of any single model, compared with the biological system it describes, may cause uncontrolled type I error. ⁴ , ⁹ Taking multiple models into consideration will mitigate that risk. ⁹ , ¹⁰

Another interesting note regarding the traditional single‐model MIE methods is that even if it would be possible to find a model close to the true model, that model may not be practically identifiable with the available data. This is particularly relevant for ophthalmic formulations, where the extremely sparse data is unlikely to support the estimation of the true model. The proposed model averaging methods include a check for practical identifiability with saddle‐reset to reduce the risk of using a non‐identifiable model in the analysis. They can then average over practically identifiable models that cover different features of a more accurate, but non‐identifiable, model. MIE approaches also open the door for multiple ways of examining the final treatment effect. For example, in a linear system with a proportional treatment effect on bioavailability, BE can be tested directly on the parameter estimates. In the proposed method, we have chosen to simulate concentration–time profiles, which means that any PK model can be included in the model pool, including nonlinear models and models with complex treatment effects.

The main question that remains for the model averaging methods is how the model pool should be compiled. It seems that bootstrap model selection can handle a larger model pool because one model is selected for each bootstrapped dataset, rather than assigning a weight to each model in the pool. Using different strategies for compiling and validating a model pool can undoubtedly have a major impact on the results. In this work, we have employed PopED optimal design investigations to qualify models for inclusion in the model pool, and then likelihood‐ratio tests for nested models and saddle‐reset to qualify each estimation. It is recommended to constrain the model pool to a relatively small number of plausible models, but the strategies for selecting these model pools should be further investigated. One approach that we have identified as promising is to test the ability of each model to describe the study dataset. For this purpose, the posterior predictive check (PPC) of the R package NCAPPC ¹⁵ can be used, excluding any model that cannot predict the observed data. However, no models were excluded when the method was applied in this study.

We also acknowledge that this work is based on simulated data, which generally allows for a much more thorough examination and comparison than using real‐world data. The presented BE analysis methods using model averaging will be further evaluated on clinical trial data in future efforts.

CONCLUSION

The model averaging approaches presented here showed high power while maintaining controlled type I error rates for BE analysis. They should be further evaluated as potential alternative BE approaches in scenarios where NCA‐based methods are not expected to perform well (e.g., sparse sampling, few individuals, high variability, incomplete washout, etc.). In this work, model‐informed approaches seemed to perform especially well in the extremely sparse data scenarios of ophthalmic formulations. Model averaging by bootstrap model selection performed best in scenarios with sparse sampling in large numbers of subjects, such as those for ophthalmic products, while the weight‐based model averaging with SIR uncertainty performed best in the rich sampling scenario with fewer subjects.

AUTHOR CONTRIBUTIONS

H.B.N., X.C., M.D., L.F., L.Z., M.O.K., and A.C.H. wrote the manuscript. H.B.N., X.C., M.D., L.F., L.Z., M.O.K., and A.C.H. designed the research. H.B.N. and X.C. performed the research. H.B.N. and X.C. analyzed the data.

FUNDING INFORMATION

No funding was received for this work.

CONFLICT OF INTEREST STATEMENT

M.D., L.Z., and L.F. are employed by the U.S. Food and Drug Administration. H.B.N. is a former employee of Uppsala University and is currently employed with Pharmetheus AB. A.C.H. and M.O.K. have received consultancy fees from, and own stock in Pharmetheus AB, all unrelated to this manuscript. The opinions expressed in this manuscript are those of the authors and should not be interpreted as the position of the U.S. Food and Drug Administration. All other authors declared no competing interests for this work. As an Associate Editor for CPT: Pharmacometrics & Systems Pharmacology, Andrew Hooker was not involved in the review or decision process for this paper.

Supporting information

Data S1.

Bjugård Nyberg H, Chen X, Donnelly M, et al. Evaluation of model‐integrated evidence approaches for pharmacokinetic bioequivalence studies using model averaging methods. CPT Pharmacometrics Syst Pharmacol. 2024;13:1748‐1761. doi: 10.1002/psp4.13217