The Power of an Adaptive Trial Design for Endovascular Stroke Studies: Simulations Using IMS III Data

Abstract

Background and Purpose

Adaptive trial designs that allow enrichment of the study population through subgroup selection can increase the chance of a positive trial when there is a differential treatment effect among patient subgroups. The goal of this study is to illustrate the potential benefit of adaptive subgroup selection in endovascular stroke studies.

Methods

We simulated the performance of a trial design with adaptive subgroup selection and compared it to that of a traditional design. Outcome data were based on 90 day mRS scores, observed in IMS III, among patients with a vessel occlusion on baseline CTA (n=382). Patients were categorized based on two methods: 1) according to location of the arterial occlusive lesion (AOL) and onset-to-randomization time, and 2) according to onset-to-randomization time alone. The power to demonstrate a treatment benefit was based on 10,000 trial simulations for each design.

Results

The treatment effect was relatively homogeneous across categories when patients were categorized based on AOL and time. Consequently, the adaptive design had similar power (47%) compared to the fixed trial design (45%). There was a differential treatment effect when patients were categorized based on time alone, resulting in greater power with the adaptive design (82%) than the fixed design (57%).

Conclusion

These simulations, based on real-world patient data, indicate that adaptive subgroup selection has merit in endovascular stroke trials as it substantially increases power when the treatment effect differs among subgroups in a predicted pattern.

Introduction / Background

The overall goal of adaptive designs is to increase the efficiency and/or the likelihood of a positive clinical trial compared to non-adaptive trial designs.^{1, 2} To achieve this goal, adaptive trial designs allow modification of the trial during the course of the trial based on data accumulated in the trial. In order to maintain the integrity of the trial, it is critical that the adaptive procedures are specified prior to study initiation and are not decided upon after results of an interim analysis have become available. In this article we provide a brief overview of commonly used adaptive trial designs in stroke research and we provide an example of a novel adaptive design that may be useful in future clinical stroke trials.

Adaptive designs can allow modification of various aspects of a study, including the sample size, the treatment dose, the randomization algorithm, and the inclusion/exclusion criteria. One type of adaptation, which is implemented in almost all phase 3 stroke trials, is the group sequential design.¹ This design specifies rules for early stopping (and hence, a reduction of the final sample size) when there is clear evidence of benefit, harm or futility. Sample-size re-estimation is another adaptation that is commonly implemented in stroke trials.^{1, 3} It specifies a change in the sample size based on, for example, the overall observed event rate in the trial. The IRIS trial is a recent example of a trial that increased its sample size because of lower than expected rates of recurrent stroke.³ Another commonly used adaptive procedure is covariate adaptive randomization.¹ This method adjusts the randomization algorithm throughout the trial to maintain optimal balance between baseline covariates in the treatment and control arms. Modern stroke trials may include not just one, but multiple adaptation procedures, in the statistical design.

Enrichment designs are adaptive trial designs that aim to enrich the study population by restricting enrollment to patients likely to show a beneficial treatment effect, based on interim data.¹ They have not been widely used due to the statistical complexities associated with controlling Type I error rate. The main motivator behind enrichment designs is to increase the chance of a positive trial when heterogeneity of treatment effects is anticipated. This is, for example, the case with endovascular stroke treatment which is believed to be more effective in patients who are treated earlier.^{4, 5} Our group developed a novel adaptive trial design that restricts selection criteria to the subgroup with the best chance of showing a treatment effect if the results of an interim or final analysis demonstrate futility in the overall sample.^{6, 7} In the event that the enrollment criteria are restricted (adapted), both subsequent recruitment and the population in which the primary hypothesis is tested are limited to the selected subgroup. (Figures 1 and 2)

Figure 1.

The study cohort stratified into six categories based on two categorization schemes and the five subgroups that can be selected by the adaptive design. The trial cohort is split into six categories of approximate equal size. Under classification scheme #1, patients are categorized based on the location of the arterial occlusive lesion (three categories: ICA, MCA-M1 and MCA-M2+) and the time from symptom onset to randomization (two categories: ≤144 min and >144 min). Under classification scheme #2, patients are categorized solely based on time from symptom onset to randomization. The full cohort includes patients from all six categories (C_1–6). Five subgroups were defined that gradually increase in size and include one (C₁) to five categories (C_1–5). The adaptive design is based on the premise that the effect of endovascular treatment may differ between categories, with a predicted gradual decrease of the effect from C₁ to C₆ (and hence also a predicted gradual decrease of the effect in subgroups from C₁ to C_1–5)

Figure 2.

The adaptive design flow chart illustrates the various decision points and possible outcomes of the adaptive design analyses. Statistical testing using the Wilcoxon Rank sum test statistic (indicated with open circles) can lead to one of several conclusions (indicated by blue arrows). Each conclusion dictates the next step in the adaptive study design (black arrows). For example, if the results of the Wilcoxon Rank Sum test during the first interim analysis indicates ‘futility in the overall cohort’, the adaptive design dictates that the optimal subgroup is selected (white square) and that testing is repeated in this subgroup. If, on repeat testing in the optimal subgroup, neither the futility nor the benefit boundary is crossed, the trial continues but subsequent enrollment (and statistical testing) is limited to this subgroup.

A design with adaptive subgroup selection is not beneficial when the optimal subgroup is already known. In that case, a traditional trial limited to the optimal subgroup will be the most efficient approach. On the other hand, when there is reasonable suspicion, but also considerable uncertainty, regarding the presence and magnitude of a differential treatment effect, a design with adaptive subgroup selection should be considered. Under that circumstance, adaptive subgroup selection allows the trial to start with relatively broad inclusion criteria that can be narrowed if, during the course of the study, it becomes apparent that there is futility in the overall population. Without the benefit of adaptive subgroup selection, trials run the risk of being designed with overly restrictive or overly inclusive patient selection criteria based on insufficient prior knowledge. Trials with overly restrictive criteria are at risk of slow enrollment and their results may lack generalizability. For example, because the ECASS trials excluded patients over 80, many countries adapted this age-criterion as a contra-indication for tPA in routine clinical practice.^8–10 It took many years to demonstrate that tPA is equally effective in the elderly and to lift this contra-indication.¹¹ On the other hand, trials with overly inclusive enrollment criteria are at risk of being underpowered to demonstrate a treatment effect, due to dilution of the effect size. For example, the IMS III trial might have been positive had enrollment been limited to patients selected with more stringent imaging and/or clinical criteria.^{4, 12–15}

The aim of this study is to demonstrate, through trial simulations based on IMS III data, how a design with adaptive subgroup selection can impact the chance of success (power) of endovascular stroke trials.

Methods

This study was exempt from IRB review because it does not qualify as human subjects research involving human subjects; it does not use individually identifiable information.

Subjects and Subgroups

The simulations were based on data from subjects enrolled in IMS III who had an arterial occlusive lesion (AOL) of the ICA or MCA on their baseline CT or MR angiogram (n = 382). These subjects were grouped into 6 categories based on AOL location and/or time from symptom onset to randomization. These two variables were chosen because prior studies have suggested that they may modify the treatment effect (reduced efficacy in subjects who have more distal AOLs and/or are treated later).^{16, 17} The IMS III classification system was used to define the AOL. (see Table 1) Two categorization schemes were studied. (Figure 1) For our primary analysis (scheme #1), the cohort was categorized according to the location of the AOL (ICA vs MCA-M1 vs MCA-M2+) and time from symptom onset to randomization (early vs late based on a median time of 144 minutes). For our secondary analysis (scheme #2), the cohort was categorized only according to time from symptom onset to randomization. (Figure 1)

Table 1.

Arterial occlusive lesion (AOL) categories and criteria

AOL Category	Criteria based on CT angiography
ICA	Occlusion of the internal carotid artery (ICA) at (a) origin (b) cervical; (c) petrous; (d) lacerum; (e) cavernous; (f) clinoidal; (g) ophthalmic artery to posterior communicating artery; or (h) posterior communicating artery to terminus. Subjects with single ICA occlusions as well as subjects with tandem occlusions involving the ICA and a more distal cerebral vessel were included in the ICA category.
MCA-M1	Occlusion of the first segment of the middle cerebral artery (MCA-M1) (a) proximal to the striate or (b) distal to the striate vessels.
MCA-M2+	Occlusion of (a) a single secondary branch of the middle cerebral artery (MCA-M2); (b) multiple M2s; (c) a single M3; (d) multiple M3s; (e) a single M4; or (f) multiple M4 branches.

Simulations and statistical analyses

A detailed description of the adaptive design procedures has been published.^{6, 7} Briefly, in the case that the trial adapts its patient selection criteria midway, subsequent recruitment and the primary efficacy analysis are limited to patients who are part of the selected subgroup. Subjects who have already been enrolled but are not part of this subgroup are excluded from the primary efficacy analysis. The criterion for deciding which subgroup has the best chance of showing a benefit from endovascular therapy combines both the estimated magnitude of the effect in, and the sample size of, the subgroup. The subgroup with the largest Wilcoxon rank sum test statistic is chosen, a criterion that optimally balances the sample and effect sizes in the subgroups. Note that there are 6 subgroups (including the whole sample) of increasing size (Figure 1). The adaptive design employs two biological assumptions to limit the inflation of sample size due to the multiple comparisons underlying the adaptive design: 1) that the effect is largest in the subjects with the most proximal AOL and/or the shortest time to randomization, and 2) that the effect declines monotonically in the specified order of categories. (Figure 1)

The adaptive trial design was compared to a trial design with a fixed sample size. The conventional fixed sample design was chosen as the comparator in this analysis because it is more powerful, and hence yields more conservative results, than the group sequential counterpart with early futility stopping rules that was used in IMS III. For each design, 10,000 trials were simulated. Simulation parameters were based on the IMS III trial. For the adaptive trial design simulations, we used the mean modified Rankin Scale (mRS) outcomes within each category and treatment group, their variances, and the proportion of subjects in each of the six categories as observed in IMS III. (Table 2) The fixed trial design performance depends only on the "pooled parameters" (the overall mean mRS outcomes and variances in the two treatment groups), which are functions of the stratified parameters and thus depend on the categorization scheme (AOL and time vs time alone). To perform consistent simulations for the adaptive design and the fixed sample design, we used the identical stratified parameters for both, deriving the “pooled parameters” for the fixed sample design from the stratified parameters. This ensures that, within each categorization scheme, the population effect is the same for the adaptive and the fixed sample designs and allows a direct comparison of the simulated power between the two study designs. Between categorization schemes, the stratified parameters, and hence the population effects, differ. Consequently, the simulated power also differs between categorization schemes, both for the adaptive as well as for the fixed sample design.

Table 2.

Treatment effect by category

		C1		C2		C3		C4		C5		C6
		Inv.	Con.	Inv.	Con.	Inv.	Con.	Inv.	Con.	Inv.	Con.	Inv.	Con.
Cat. #1	n (%)	76 (19.9%)		62 (16.2%)		75 (19.6%)		74 (19.4%)		53 (13.9%)		42 (11.0%)
	n per arm	47	29	31	31	42	33	53	21	35	18	27	15
	Mean mRS	3.57	4.03	4.06	3.71	2.57	2.82	2.89	3.33	2.43	2.72	3.04	3.13
	Delta mRS (SD)	−0.46 (1.92)		0.35 (2.08)		−0.25 (2.00)		−0.44 (1.81)		−0.29 (1.87)		−0.09 (1.80)
Cat. #2	n (%)	52 (13.6%)		76 (19.9%)		65 (17.0%)		76 (19.9%)		49 (12.8%)		64 (16.8%)
	n	30	22	38	38	48	17	47	29	33	16	39	25
	Mean mRS	3.13	3.41	2.37	3.37	3.23	2.88	3.21	2.93	3	3.88	3.41	3.68
	Delta mRS (SD)	−0.28 (2.07)		−1.00 (1.96)		0.35 (1.92)		0.28 (2.12)		−0.88 (1.67)		−0.27 (1.96)

Cat #1 indicates categorization scheme #1 in which patients are categorized based on time and location of the arterial occlusion. Cat #2 indicates categorization scheme #2 in which patients are categorized only based on time. C_x indicates a patient category. See figure 1 for the exact criteria that define each patient category in categorization scheme #1 and #2. Inv. indicates the investigational treatment arm in the IMS III trial (intravenous tPA plus endovascular therapy); Con. indicates the control arm (intravenous tPA alone); n indicates number of subjects; mRS indicates modified Rankin Scale; and SD indicates standard deviation.

The treatment effect in the simulations was assessed with the normal approximation for the nonparametric Wilcoxon Rank Sum Test. The statistical significance criterion was set at a one-sided type 1 error rate (alpha) of 0.025, and the power at 90%. The maximum randomized sample size was 900 subjects. This was the intended sample size of IMS III, which was chosen to be able to detect an absolute treatment difference of at least 10%. For the adaptive design simulations, interim analyses were conducted after half (n=450) and three quarters (n=675) of the maximum sample size. All simulations were run using R for MacOS (v 0.98.1103).

Results

When patients were categorized based on AOL location and time (categorization scheme #1), the assumptions that the adaptive design was based upon were not met. Specifically, the treatment effect did not decline from C1 to C6, but instead was relatively homogeneous across categories. (Table 2) Consequently, the power of the adaptive design (47%) was comparable to that of the fixed trial design (45%). In contrast, when patients were categorized based on time alone, there was a differential treatment effect and the adaptive design had greater power than the fixed design (82% vs 57%).

Table 3 lists the proportions of adaptive trial simulations that terminated for benefit and for futility at each of the two interim analyses and at the final analysis. Because the adaptive trial simulations generally resulted in subgroup selection (a subgroup was selected in 75% of simulations under scheme #1 and in 67% of simulations under scheme #2, table 4), the mean number of subjects that were included in the final efficacy analysis was substantially lower than the mean number of subjects randomized. Under categorization scheme #1, the mean number of patients randomized was 778 (SD 169), of whom an average of 554 (SD 219) were included in the ultimate analysis. Under categorization scheme #2, an average of 762 (SD 167) patients were randomized, of whom 490 (SD 202) were included in the ultimate analysis.

Table 3.

Percentages of adaptive trial simulations that terminate for efficacy / futility by analysis stage

		Simulations (%) that end for efficacy by subgroup						Total % of simulations that end for efficacy	Total % of simulations that end for futility	Total % of simulations that end
	Analysis stage at study completion	C1	C1–2	C1–3	C1–4	C1–5	C1–6
Cat #1	1st interim (n=450)	3.0%	0.0%	0.0%	0.2%	0.3%	3.9%	7.4%	8.4%	15.8%
	2nd interim (n=675)	4.4%	0.0%	0.0%	0.2%	0.5%	5.6%	10.8%	12.1%	22.9%
	Final (n=900)	11.8%	0.1%	0.1%	0.6%	1.2%	15.1%	28.9%	32.5%**	61.4%
	Total	19.2%	0.1%	0.2%	1.0%	2.0%	24.6%	47.0%*	53.0%	100.0%
Cat #2	1st interim (n=450)	0.1%	7.5%	0.1%	0.0%	0.1%	5.2%	12.9%	2.7%	15.6%
	2nd interim (n=675)	0.1%	14.5%	0.2%	0.0%	0.2%	10.0%	24.9%	5.3%	30.2%
	Final (n=900)	0.2%	25.9%	0.3%	0.0%	0.4%	17.9%	44.7%	9.5%**	54.2%
	Total	0.4%	47.8%	0.6%	0.0%	0.7%	33.0%	82.5%*	17.5%	100.0%

Cat #1 indicates categorization scheme #1 in which patients are categorized based on time and location of the arterial occlusion. Cat #2 indicates categorization scheme #2 in which patients are categorized only based on time. C_1-y indicates a subgroup that consists of consecutive patient categories. See figure 1 for the exact criteria that define each patient category in scheme #1 and #2. C_1–6 includes all 6 patient categories and thus indicates that the trial completes without subgroup selection

^*The overall total corresponds to the power of the adaptive trial design

^**The percentage at the final analysis (n=900) reflects the number of simulations that neither stopped for efficacy or futility at an interim analysis nor demonstrated efficacy at the final analysis.

Table 4.

Percentages of adaptive trial simulations that demonstrated futility in the overall cohort and proceed to subsequent subgroup selection

	Analysis stage at subgroup selection	Total % of simulations that result in subgroup selection by stage	Subgroup selected
			C1	C1–2	C1–3	C1–4	C1–5
Cat #1	1st interim	34.6%	20.5%	1.1%	1.2%	4.1%	7.8%
	2nd interim	16.9%	8.2%	0.2%	0.3%	2.1%	6.1%
	Final	23.9%	8.6%	0.1%	0.3%	2.5%	12.5%
	Total	75.4%	37.3%	1.4%	1.8%	8.7%	26.3%
Cat #2	1st interim	29.4%	1.6%	24.6%	1.0%	0.0%	2.2%
	2nd interim	15.1%	0.1%	13.3%	0.4%	0.0%	1.3%
	Final	22.5%	0.0%	19.8%	0.4%	0.0%	2.2%
	Total	67.0%	1.8%	57.8%	1.8%	0.0%	5.7%

Cat #1 indicates categorization scheme #1 in which patients are categorized based on time and location of the arterial occlusion. Cat #2 indicates categorization scheme #2 in which patients are categorized only based on time. C_1-y indicates a subgroup that consists of consecutive patient categories. See figure 1 for the exact criteria that define each patient category in scheme #1 and #2.

Discussion

Our simulations, based on IMS III data, suggest that a trial design with adaptive subgroup selection can be a highly efficient method to test the effect of endovascular stroke treatment. The adaptive design yields a substantial increase in power compared to a fixed trial design when the treatment effect differs among subgroups in a predicted pattern. When the treatment effect does not vary according to prediction, and is instead relatively homogeneous across subgroups, the adaptive design has similar power compared to a fixed trial design. Given these results, the adaptive trial design can be viewed as a cost-efficient insurance policy; one that yields substantial gains in power when the predicted pattern of treatment effects is realized and has a relatively low premium (paid as a modest loss in power when the treatment effect is homogenous across subgroups).

An important feature of the adaptive design is that prior to the first analysis, patient categories need to be defined and rank-ordered according to the predicted treatment effect in each category. In a prospective trial there is no assurance that this prediction will come true. To emulate this scenario, we defined and rank-ordered patient categories prior to running our simulations. The results of our secondary analysis (scheme #2: patients categorized only based on time-to-randomization) demonstrate that the observed ranking of treatment effects across categories does not have to match the predicted pattern exactly in order to achieve a gain in statistical power with the adaptive design. When patients were categorized only according to time from symptom onset to randomization (scheme #2), the benefit of treatment was concentrated among patients in the two categories with the shortest times (<126 min). In patients with longer times from symptom onset to randomization, the treatment effect fluctuated: treatment was associated with worse outcomes in the middle time categories (126–163 min), substantial benefit in the penultimate category (163–181), and minimal benefit in the latest time category (181–275 min). Despite these fluctuations in treatment effects across the later time-categories, the concentration of treatment effect in the first two time-categories was sufficient to substantially increase the power of the study from 57% with a fixed trial design to 82% with an adaptive design. Had the observed ranking matched the predicted pattern more closely, with a gradual decline in treatment effects from the earliest to the latest treatment category, the gain in power would have been much more substantial.⁶ Had we chosen different boundaries for the time-categories, the number of patients and the observed treatment effect in each category would have been slightly different, which, in turn, would have led to small gains or losses in the power estimate.

The results of our primary analysis (scheme 1: patients categorized according to both time-to-randomization and location of the vessel occlusion) demonstrate the impact on study power when the treatment effect does not follow the predicted pattern but is, instead, relatively homogeneous across categories. Under this circumstance, the adaptive and fixed designs had similar power (47 vs 45%). In other words, the statistical price paid for implementing the adaptive design is small when treatment effects are relatively homogeneous across categories. There would, however, be a substantial loss of power if the observed pattern of treatment effects is opposite from what was predicted.⁶ For example, had there been a gradual decline in treatment effect, with the greatest benefit in subjects with distal vessel occlusions who were treated late and the greatest harm in patients with proximal vessel occlusions treated early, then the adaptive design would have had substantially less power than a fixed sample size study. While this may seem like an unlikely scenario given the variables chosen in this example, the possibility that adaptive subgroup selection can decrease the power of a study is the main risk of this trial design and should be kept in mind when it is being considered for future studies.

There are numerous variations possible of the adaptive design described in this manuscript. Theoretically it is possible to create an adaptive design that does not require pre-specification of the direction of the treatment modification. Such a design could allow any combination of categories (i.e. cells from Figure 1) to be selected as a subgroup. This, however, has two important drawbacks. First, the extreme flexibility in subgroup selection is associated with an increase in statistical cost (i.e. loss of power) due to the need to correct for much larger multiplicity of testing. Second, it would allow for the possibility of the trial to limit recruitment to a biologically illogical subgroup of patients. For example, after adaption of selection criteria, the trial could be limited to patients with ICA occlusions who are treated early and patients with MCA-M2 occlusions who are treated late. Thus, the advantages of the adaptive design that we tested, which requires pre-specification of the direction of treatment effects across categories, are that it has a relatively low associated statistical cost and that it eliminates the possibility of a biologically implausible subgroup selection. Another possible variation is to create an adaptive design with more or fewer than the 6 categories used in these simulations. An increase in the number of categories increases the “statistical cost” (a greater loss of power if the treatment effect is homogeneous across categories), whereas a decrease reduces the ‘statistical cost’.

There are important differences between the simulations in this study and the IMS III trial. First, categorization based on AOL and time relies on the ability to select patients based on the location of their arterial occlusive lesion on a CT or MR angiogram. This could not have been implemented, because IMS III did not require a CT or MR angiogram at baseline, and this type of imaging was therefore only available in a subset of the IMS III population. Second, the comparator in our simulations was a fixed sample size design, while the IMS III study had a group sequential design with early stopping rules. We chose a fixed sample size conventional (non-sequential) design as the comparator because it is more powerful than a group sequential design with the same sample size and treatment effect. Consequently, our estimate of the power gained with adaptive subgroup selection when patients were categorized based on time to randomization alone (scheme #2) is conservative and would have been greater if we had compared our results to a group sequential study of the same size.

There are some limitations to this study and to the adaptive design itself. A limitation of this simulation is that the effect sizes of endovascular treatment for the 6 categories may not reflect the true effect sizes in nature because they were based on relatively small numbers of subjects per category. Limitations of adapting (narrowing) patient selection criteria include: 1) increased length of recruitment and trial cost; 2) changes to study forms and recruitment materials to reflect the updated criteria; and 3) reduced generalizability of the trial results, which is a drawback if the adaptive design excludes categories in which a treatment benefit exists (ie, a type II error for that specific subgroup), but would be an advantage if categories are excluded in which the investigational treatment has no effect or is harmful. These limitations should be viewed in the context of the limitations of trials without adaptive subgroup selection. Such trials stop completely when futility is declared and, when a post-hoc analysis demonstrates benefit in a subgroup, often lead to the design of an entirely new trial in that subgroup. This two-step approach is more time-consuming and costly than the adaptive design with its seamless transition from overall population to subgroup. Another limitation of the design with adaptive subgroup selection (and of all other designs with adaptation based on interim outcome analyses, including now-standard group sequential designs) is that some alpha is spent on the interim analyses. This has already been taken into account for the ultimate efficacy analysis that determines whether treatment is beneficial at a one-sided alpha of 0.025. However, methods to produce valid estimates of the treatment effect size and its confidence interval, are still under development. Finally, all designs with interim outcome-based adaptation have the potential to reveal some information about effects at the interim, and this adaptive design shares that feature.

In summary, this study provides proof-of-principle, demonstrating that a study design with adaptive subgroup selection can result in important gains in the efficiency of endovascular stroke trials. Therefore, the results of this study support the implementation of adaptive subgroup selection in future endovascular stroke trials. One such study (DEFUSE 3), which was launched in the spring of 2016, employs a design with adaptive subgroup selection based on time from symptom onset to randomization and volume of the ischemic core.