Beating the Placebo in HIV Prevention Efficacy Trials: The Role of the Minimal Efficacy Bound

Abstract

Objective

To produce an effect on the HIV epidemic, preventive interventions need to achieve a minimum level of efficacy in order to offset potential indirect effects such as an increase in risky behavior. The current generation of HIV prevention trials on oral pre-exposure prophylaxis and on vaginal microbicides were designed using different set points for minimum individual-level efficacy (MIE). Some trials were designed not only to show superiority over placebo but also to rule out lower efficacies. The MIE has a substantial impact on the size and cost of a trial. Ideally, the MIE should be chosen to reduce uncertainty in the estimation of population level effects. In this paper, we investigate the effect of MIE on estimates of population level impact in order to better inform trial design.

Methods

We used mathematical model simulations assuming various rates of efficacy obtained from trials and different MIEs to study the impact of wide-scale interventions on two public health indicators.

Results

Implementation factors were the main drivers of uncertainty in public health indicators for an intervention, although MIE also contributed. The level of uncertainty introduced by the MIE was substantially lower than that of the other factors.

Conclusions

Investigators in clinical trials have set the MIE solely on the basis of potential public health impact. However, the substantial increase in trial costs associated with a large MIE is unlikely to be justified. These additional funds would be better spent in evaluating more critical implementation factors that cannot be assessed in clinical trials.

Introduction

According to the latest UNAIDS Report on the global AIDS epidemic, the number of new HIV infections has declined by 19% since 1999 [1]. Part of this decline can be attributed to expanded access to treatment. New and more effective HIV preventive interventions are still urgently needed, as the overall number of people living with HIV has increased worldwide. Antiretroviral (ART)-based HIV preventives, such as oral pre-exposure prophylaxis (PrEP) and vaginal microbicides (VMB) have produced encouraging results in several phase IIb/III efficacy prevention trials [2-5]. However, negative results were obtained in the Pre-exposure Prophylaxis Trial for HIV Prevention amongAfrican Women (FEM-PrEP) [6] and are awaiting confirmation in ongoing trials where one or more arms have been discontinued [7].

One key feature in trial design is the choice of the null hypothesis for statistical inference on efficacy. Typically, to demonstrate that an intervention prevents the acquisition of HIV, one needs to show that it is more effective than a matched placebo – in other words, to demonstrate that the observed efficacy (defined as 1 - hazard ratio) is significantly larger than 0%. The minimum individual-level efficacy (MIE) is the set-point level chosen for the null hypothesis. In many trials of HIV prevention intervention, the investigator will set the MIE to values larger than 0%, thus powering the trial to detect a fixed level of efficacy and to rule out efficacies beneath that threshold. In practice, this means that for trials designed with an MIE of 0%, statistical significance will be achieved when the lower bound of the 95% confidence interval (CI) is above 0% whereas for trials with an MIE of 30%, significance will be achieved only when the lower bound of the 95% CI is above 30%. Note that in the context of product registration, it is not necessary to specify an MIE above 0%. The only requirement for seeking licensure is to show that the intervention is superior to placebo. More importantly, the choice of MIE will have a substantial impact on sample size. Indeed, a trial with an MIE set at 10% and 20% will be 50% and 130% larger than a trial with an MIE of 0%, respectively (see Supplemental Digital Content, Figure S1). Thus, designing trials with larger MIEs has substantial cost implications.

One concern has been that wide-scale prevention interventions with partial efficacy, if used alone, might have only a modest impact on the HIV epidemic [8]. In addition, many factors such as the varied populations of users, reductions in condom use (condom replacement), and other risk compensation may reduce the population level impact of an intervention proven efficacious in the controlled environment of a clinical trial [9-13]. Therefore, to offset any negative indirect effects potentially introduced by an increase in risky behaviour, imperfect adherence or other factors, promising interventions need to achieve a minimum level of efficacy well above 0% in clinical trials [14]. Mathematical modeling suggests that to have a meaningful impact at the population level, the efficacy may need to exceed 30% for an HIV vaccine [15, 16] and 25-50% for oral or topical PrEP [9, 10, 17]. Based on these considerations, several ongoing or completed oral PrEP and VMB trials have been powered with MIEs of up to 30%. For instance, the recently completed Pre-exposure Prophylaxis Initiative (iPrEx) and Partners PrEP trials were powered to detect an efficacy of 60% or more with an MIE set at 30% [3, 4] while the ongoing Vaginal and Oral Interventions to Control the Epidemic (VOICE) trial detects an efficacy of 55% using an MIE of 25% [7]. As the cost of HIV prevention efficacy trials is roughly proportional to the number of HIV endpoints targeted in a trial, setting an MIE at 30% substantially increases the cost of conducting a trial. For example, for the iPrEx trial, about 90 HIV endpoints were required to achieve 80% power with a one-sided alpha of 5%, as compared to 30, 43, and 59 endpoints for an MIE of 0%, 10%, and 20%, respectively. Thus, using an MIE of 30% instead of 0% has tripled the required number of endpoints. As the larger number of endpoints increases the precision of the efficacy estimate, an observed efficacy of 57% in a trial with 90 HIV endpoints will yield a 95% CI lower bound of approximately 33% as compared to 6% with an MIE of 0%, even if the results are significant with respect to their appropriate null hypothesis. Ideally, the MIE should be chosen to help identify interventions that will be of public health relevance but also to minimize the human and economic costs of the study[14]. Selecting MIEs that are too large will utilize resources that could have been better spent to evaluate other interventions or study factors associated with efficacy at the population level (that is, outside the clinical trial setting).

The overall objective of this study was to assess the potential public health impact of an HIV prevention intervention using various MIEs to establish confidence intervals. We used compartmental models to simulate wide-scale microbicide and oral PrEP interventions and to evaluate the precision of predictions of impact at the population level, when products were tested in trials designed with MIEs ranging from 0% to 20%. In addition, we investigated the likelihood of predicting a small or negative impact for different MIEs and sought to establish conditions that would justify the use of large vs. small MIEs in oral PrEP and VMB efficacy trials.

Methods

Mathematical model simulations were carried out to investigate the impact of different implementation factors on public health indicators. Estimates of efficacy were obtained with several trial design scenarios using varying MIEs and sample sizes.

Models

We modified previously developed models [18] to simulate the HIV epidemic and the introduction of oral PrEP and VMB in heterosexual populations using parameters representative of sub-Saharan Africa. Populations were stratified by gender, HIV status (susceptible, infected with wild-type HIV, infected with drug-resistant HIV, and AIDS), and VMB or PrEP use. Since none of the products were as yet widely available, we estimated maximum resistance rates (in case of systemic absorption of the product) from existing studies of ART resistance in developing countries [19], U.S. [20], and Europe [21-23]. Drug-resistant HIV was assumed to be less transmissible due to its reduced fitness relative to wild-type HIV [24]. Control measures that discouraged HIV-positive individuals from initiating or continuing use of VMB or oral PrEP were included in the model structure. Each model incorporated different features with regard to product introduction, coverage, resistance development and adherence, specific to the characteristics of the modeled intervention (oral PrEP or VMB). A complete description of the models is given in the Supplemental Digital Content.

Trial design scenarios

We evaluated three trial designs set to detect an efficacy of at least 50%, with an MIE of 0%, 10%, and 20%, respectively, each with 80% power and a two-sided alpha of 5%. To achieve the desired power and alpha levels, we calculated that between 50 and 150 HIV endpoints would be required. In fact, the HIV endpoints in completed and ongoing oral PrEP and VMB trials are within this range. For each trial design, we assumed three possible outcomes (33%, 50%, and 75% observed efficacy), for a total of 9 hypothetical trials. Efficacy was defined as 1 minus the relative risk (RR) of HIV infection in the intervention group as compared to the control group. As RR approximates a lognormal distribution, we used the lognormal distribution for the RR associated with the trial results to determine the distribution for the efficacy (see Supplemental Digital Content, Figure S2). Hypothetical trial designs are presented in Table 1.

Table 1. Design of hypothetical clinical trials.

Trial Design	Small			Medium			Large
Null hypothesis: MIE (No. of endpoints)1	0% (66)			10% (97)			20% (153)
Observed HIV endpoints: intervention/placebo	26/40	22/44	13/53	39/58	32/65	19/78	61/92	51/102	31/122
Size of trial1	2200	2400	2900	3200	3600	4300	5100	5700	6800
Observed efficacy [95% CI]3	33% [−7%,60%]	50% [16%,70%]	75% [55%,87%]	33% [−1%,55%]	50% [25%,68%]	75% [60%,85%]	33% [8%,52%]	50% [30%,64%]	75% [62%,83%]
p-value (2-sided)	8.6%	0.7%	<0.1%	5.4%	<0.1%%	<0.1%	1.2%	<0.1%	<0.1%

¹All trials were designed to detect 50% efficacy with a false positive error rate of 2.5%. MIE, minimum individual-level efficacy.

²Assuming baseline HIV incidence of 4% per year, 12 months of follow-up per participant, and 90% retention rate.

³The observed MIE is the lower bound of the 95% confidence interval (CI).

Implementation scenarios

We explored various ranges of expected adherence, coverage, resistance. These parameters were identified in previous studies [10, 25-27] as major implementation factors that influence the population level impact of oral PrEP and VMB interventions. Recognizing that coverage and adherence depend on product availability and affordability as well as on the readiness of target groups to consistently use particular preventive options, we analyzed pessimistic and optimistic ranges for each of the intervention scenarios (Table 2). For the purposes of this study, we assumed that the risk of systemic absorption of VMB was relatively low, within a range of 1% to 10%. Efficacy clinical trials to date have not been designed to look for reductions in the infectiousness of the HIV-positive individual protected by VMB or oral PrEP, a possibility that seems plausible in light of the positive results of the treatment-as-prevention trial (HPTN 052) [28]. We therefore investigated two distinct efficacy mechanisms: 1) one-directional, in which PrEP or VMB reduces only the susceptibility of the user, and 2) bi-directional, in which PrEP or VMB reduces susceptibility and infectiousness of the user. When present, the reduction in infectiousness was sampled from the same distribution as the reduction in susceptibility. A 20% reduction in condom use (condom replacement) was assumed for VMB and PrEP users in all scenarios.

Table 2. Intervention scenarios.

Param.	VMB model			PrEP model
Scenario	Coverage	Resistance1	Adherence2	Coverage	Resistance1	Adherence2
Sc1	40%-50%	0.33-0.5	50%-60%	40%-50%	0.33-0.5	70%-80%
Sc2	40%-50%	0.33-0.5	70%-80%	40%-50%	0.33-0.5	90%-100%
Sc3	40%-50%	1-2	50%-60%	40%-50%	1-2	70%-80%
Sc4	40%-50%	1-2	70%-80%	40%-50%	1-2	90%-100%
Sc5	70%-80%	0.33-0.5	50%-60%	70%-80%	0.33-0.5	70%-80%
Sc6	70%-80%	0.33-0.5	70%-80%	70%-80%	0.33-0.5	90%-100%
Sc7	70%-80%	1-2	50%-60%	70%-80%	1-2	70%-80%
Sc8	70%-80%	1-2	70%-80%	70%-80%	1-2	90%-100%

¹Annual rate of resistance development (r_max).

²The adherence metric assesses the proportion of sex acts in which each user is protected by VMB or the relative PrEP efficacy due to imperfect adherence compared to perfect adherence.

VMB, vaginal microbicides; PrEP, oral pre-exposure prophylaxis.

Public health impact indicators

To evaluate the effects of wide-scale implementation at the population level, we used two public health indicators (PHI): the cumulative fraction of infections prevented over T years after the start of the intervention F_C(T); and the relative decrease in the incidence of HIV T years after the start of the intervention F_I(T). The intervention was introduced in populations with mature HIV epidemic, i.e. at equilibrium. Estimates of PHI over time were compared across different implementation and trial design scenarios to investigate the influence of each factor on expected population benefits.

Simulations

For each hypothetical trial, we simulated 8 implementation scenarios (Table 2), assuming both uni- and bi-directional protection by PrEP or VMB. All pre-intervention parameters were initially sampled uniformly from ranges representative of sub-Saharan Africa and later filtered to meet target criteria that ensured endemic conditions and HIV prevalence similar to those observed in that region [1]: 1) basic reproductive number R₀>1 in the absence of any intervention, and 2) equilibrium HIV prevalence less than 35% in the simulated population. The selection process was repeated until 10,000 sets were attained per implementation scenario. For each simulated intervention, VMB and oral PrEP efficacies were sampled from distributions associated with the results of the hypothetical trials, while the remaining intervention parameters were uniformly drawn from their experimental ranges (Supplemental Digital Content, Table S2). Multivariate sensitivity analysis was conducted to determine the effect of the intervention parameters on the PHI.

Results

The results of simulations for VMB and oral PrEP are summarized in Figures 1 and 2, respectively. As expected, oral PrEP interventions had a larger impact on the epidemic than VMBs, as a VMB is used by the woman only and can prevent HIV acquisition by only one route of transmission (vaginal intercourse). Increasing the MIE improved precision of PHI, visible as the shorter box plots when the MIE was set to 20%. However, the implementation factors induced more variation in PHI than the choice of MIE (comparing Sc1 to Sc8). For example, at 50% observed VMB efficacy, F_C(10) median values were about 20% for optimistic values of adherence, resistance, and coverage (Figure 1B, Sc6 and Sc8) as compared to 7% for the more pessimistic scenarios (Figure 1B, Sc1 and Sc3), with little variation across MIE. Similarly, the lower bounds of the expected values for F_C(10), assuming an MIE of 20%, varied between 5% and 14% across implementation scenarios while the variation introduced by MIE was only 2-4%. Similar results were obtained (see Supplemental Digital Content, Figures S5 and S6) with the other PHI (F_I) and for periods of 5 and 20 years. Furthermore, the choice of efficacy mechanism (uni- or bi-directional) introduced a level of uncertainty in the PHI estimates equal to that of the change in MIE, although an intervention that (partially) prevented acquisition as well as transmission provided a larger benefit (Figures 1, 2B vs. Figures 1, 2D). Thus, implementation factors were more critical than MIE in determining the level of uncertainty on PHI of an intervention. While MIE also contributed, the added variability exerted by the MIE was substantially lower than that of the other factors.

Figure 1.

Model predictions of the cumulative fraction of infections prevented over a period of 10 years (F_C(10)) for vaginal microbicides in clinical trials with minimum individual-level efficacy (MIE) set at 20% (light gray boxes), 10% (white boxes), and 0% (dark gray boxes). Various observed efficacies are illustrated: A) 33%, B) and D) 50%, C) 75%. Box plots show 5^th, 25^th, 50^th, 75^th, and 95^th percentiles of the predictions generated by 10,000 simulations per scenario. Scenarios Sc1-Sc8 are defined in Table 2.

Figure 2.

Model predictions of the cumulative fraction of infections prevented over a period of 10 years (F_C(10)) for oral pre-exposure prophylaxis (PrEP) in clinical trials with minimum individual-level efficacy (MIE) set at 20% (light gray boxes), 10% (white boxes), and 0% (dark gray boxes). Various observed efficacies are illustrated: A) 33%, B) and D) 50%, C) 75%. Box plots show 5th, 25th, 50^th, 75^th, and 95^th percentiles of the predictions generated by 10,000 simulations per scenario. Scenarios Sc1-8 are defined in Table 2.

The influence of the risk of resistance increased over time due to the lower transmissibility of drug-resistant HIV; higher prevalence of drug-resistant HIV with a relatively low transmissibility meant more infections prevented. However, its effect on PHI was limited (compare Sc1 and Sc3, Sc2 and Sc4, Sc5 and Sc7, Sc6 and Sc8 in Figures 1 and 2). Notably, the risk of systemic absorption for VMB use is small, and the time during which infected users are assumed to continue using PrEP (3 to 6 months) is not long enough to support drug resistance in the presence of frequent monitoring. Periodic HIV testing in PrEP users also reduces the differences in population level impact due to uni-vs. bi-directional protective mechanisms of PrEP, because susceptible individuals exposed to infected users have a slim chance of also benefitting from the additional protection accorded by the reduced infectivity of the drug-resistant virus (Figures 2B, 2D). Although small, the uncertainty in PHI due to resistance risk factors was similar to that introduced by different MIEs.

The proximity to zero of the lower bounds and quartiles of the PHI box plots in Figures 1 and 2 can be used to assess the risk that an intervention may yield a very small or even a negative impact on the epidemic, given the efficacy estimates provided by clinical trials with different MIEs. For most of the scenarios, these bounds were positive; therefore the interventions were potentially beneficial. However, some impacts were marginal and even negative, particularly under more pessimistic implementation scenarios assuming lower efficacy (Figures 1A and 2A). To explore this further, we evaluated the likelihood of a small or negative PHI over a period of 10 years for the various trial designs. Some modeling studies [10-12] have predicted negative PHIs for interventions of relative low efficacy, assuming significant levels of condom replacement or increased-risk sexual behavior. Our analysis suggests that if 75% efficacy was observed in a trial, then oral PrEP intervention would be very unlikely to have a negative impact at the population level, regardless of the MIE used to design the trial, as no simulation predicted less that 5% impact over 10 years.

Assuming 20% condom replacement, it is unlikely (likelihood 0.1%-0.5%) that an intervention with bi-directional protection which had demonstrated 50% efficacy in a trial with 0% MIE would result in more HIV infections over 10 years, while the same likelihood for interventions with uni-directional protection was slightly higher 0.2%-0.7% (Supplemental Digital Content, Figure S7). Similarly, the likelihood of small population impact (below 5%) from 50% efficacious PrEP intervention with bi-directional protection remains below 2% regardless of the levels of MIE, coverage, adherence and resistance risk assumed (Figure 3A, C). Note that MIEs above 0% were more important for products with low efficacy (~30%) to be able to rule out products with no benefit or even with negative benefits (Figure 3B, D). However, the likelihood of small population impact was equally sensitive to the implementation factors. For instance, the increase in MIE from 0% to 20% reduced the likelihood of small PHI by 2- to 3-fold in all intervention scenarios but similar uncertainty was associated with variation in the drug-resistance rate.

Figure 3.

Model predictions of the likelihood of a small public health impact (below 5%) for oral pre-exposure prophylaxis (PrEP) with bi-directional protection, in clinical trials with minimum individual-level efficacy (MIE) set at 20% (light gray bars), 10% (white bars), and 0% (dark gray bars). The cumulative fraction of infections prevented in 10 years, F_C(10), is shown in A), B); the relative decrease in the incidence of HIV 10 years after intervention began, F_I(10), is shown in C), D); at observed efficacies of 50% and 33%, respectively. Results were generated by 10,000 simulations per scenario. Scenarios Sc1-8 are defined in Table 2.

Discussion

Investigators have used various levels of MIE to power individual-based randomized clinical trials (I-RCT) to ensure not only substantial individual benefit but also a positive impact once an intervention is rolled out at the population level. One key concern of investigators would be to implement and roll out an intervention using a low-efficacy product where the effectiveness is barely above 0% and any negative indirect effect of the implementation (risk disinhibition and condom replacement) could negate any potential population level impact. Our analysis suggests that although the use of larger MIEs increases the precision of the predicted PHI, the implementation factors are equally if not more important in determining the level of uncertainty in PHI. Estimates of PHI are very sensitive to the level of coverage and personal adherence assumed. Therefore, good measures of potential uptake or coverage as well as real world adherence would be more critical in predicting the PHI than having more precise estimates of efficacy in I-RCTs. Unfortunately, I-RCTs are not well suited to estimating uptake, coverage and even adherence outside their well-controlled environment. These critical factors would need to be estimated in implementation studies following the completion of the I-RCTs.

The magnitude of the MIE chosen by investigators has a substantial impact on the cost of the trial. The iPrEx trial, using an MIE of 30%, increased the number of HIV endpoints by a factor of 3 as compared to a design aiming to simply show superiority over placebo. Typically, clinical trial costs are roughly proportional to the targeted number of HIV infections and the total costs are often above $40-$50 million U.S. [29]. Therefore, an increase of 2- or 3-fold in trial size has substantial cost implications. Note that while some investigators have designed trials with large MIEs (≥25%), others have decided otherwise even if the interventions being evaluated were similar. For instance, the FEM-PrEP and Partners in PrEP trials evaluated the efficacy of tenofovir/emtricitabine (Truvada) using an MIE of 0% and 30%, respectively [6, 30]. Other ongoing PrEP trials such as VOICE and the vaginal ring study (MTN-020) are using MIEs of 25% or more [7, 31]. In light of our results, these additional costs may not be justified, as more important factors will impact the PHI. Given the scarcity of resources, further discussions on the rationale for powering I-RCTs with MIEs above 0% are warranted. We found that large MIE may be important when the efficacy estimated in I-RCT is relatively low (~30%). However, such products are not likely to be used in programs for eradicating HIV in a population and therefore their path toward licensure is questionable.

To our knowledge, larger MIEs have been used in the context of a single trial without considering the licensure process, and this practice may need to be revisited. Typically, a VMB will seek regulatory approval, while an oral PrEP will seek a change of label. In either case, the strength of evidence from two independent trials is required. In some contexts,the evidence from a single pivotal trial may be sufficient, but approval by the regulators is needed prior to initiation of the clinical evaluation program. If approved, a single pivotal trial will typically be larger than the two independent trials that would otherwise have been conducted (see discussion on the topic in [32]). In brief, negotiations with regulators will focus around selecting the false-positive error rate (alpha level) of the trial and not the MIE. Note that two successful independent trials designed to show superiority over placebo, with each trial MIE set to 0%, will greatly reduce the uncertainty of the efficacy estimates. For instance, if 50% efficacy was observed in each of the two trials, with a lower bound in the 95% CI of 1% in each, the lower bound for the 95% CI in the combined trials is around 19% (assuming the trial populations are similar). Thus, in selecting the MIE for a trial, it is important to consider the overall strength of evidence that will be accumulated prior to registration, because the combined precision on the estimates of efficacy will be greater than the one from a single trial. However, as oral PrEP and VMB interventions are being evaluated in very different populations, combining evidence in order to gain precision might be unwarranted, especially in situations where positive and negative trial outcomes are observed [2-6].

The size of a trial is driven not only by the precision needed on the estimates of efficacy. Sufficient amount of follow-up and length of product use are also required in order to evaluate the safety profile of the intervention. Typically, these requirements are set by the regulatory agencies. However, the MIEs in trials have been set solely on the basis of the potential public health impact and the needed precision on the estimate on the lower bound of efficacy. Our results suggest that the substantial increase in costs to run trials with large MIEs is unlikely to be justified in the context explored, especially when more than one trial is required for registration. Additional funds would be better spent in assessing more critical implementation factors that cannot be assessed in clinical trials. Other demographic and behavioral factors such as condom replacement which indirectly (negatively) affect the effectiveness of the intervention should also be considered. A concerted approach on the design of HIV prevention trials including discussions on the precision of efficacy estimates, the licensure strategy, and the implementation issues is needed given the scarcity of resources and the considerable costs associated with the design of the current generation of trials.