Ready or Not, Here They Come: Randomized Trials Evaluating New Endovascular Aneurysm Therapies

Abstract

SUMMARY: Randomized trials evaluating endovascular therapy are currently underway. The results of these trials will present us with both new answers and new questions.

We, along with many in the interventional neuroradiology community, have been advocating prospective randomized trials comparing new coil technology with bare platinum coils. Such trials are inspired by the fact that recent registries, including ACTIVE¹ and HEAL,² have failed to show compelling benefits against historic controls of platinum coil cohorts. Randomized trials, including HELPS³ and the Cerecyte coil trial,⁴ comparing HydroCoil and Cerecyte coils, respectively, with bare platinum coils, are well underway, and Boston Scientific has recently announced that Matrix will be compared with bare platinum in a randomized trial (MAPS).⁵ The goal of these trials will be to demonstrate a decrease in aneurysm recurrence rates without patients’ incurring a significant increase in complications.

Even though completion of these trials will almost certainly advance our field, it remains unclear whether the resulting data will be easily interpreted or immediately relevant to clinical practice. There are basic concepts to consider as we move forward with randomized trials of new endovascular aneurysm therapies for cerebral aneurysms: These are 1) outcome measures, 2) sample size, and 3) bias, and each of these is discussed here.

Outcome Measures

One of the first considerations in developing trials of new endovascular aneurysm therapies is how outcome will be assessed. This may seem like a simple issue at first glance, but really it is somewhat complex. The 3 most relevant clinical outcome measures for patients with cerebral aneurysm are neurologic function, angiographic recurrence, and aneurysm retreatment. Each of these end points has strengths and weaknesses, which we will discuss.

Neurologic Function.

The most clinically relevant outcome of any cerebral aneurysm intervention is the avoidance of future stroke. Thus, for trials focused on aneurysm treatment, the most logical outcome would be long-term neurologic function. Neurologic function really addresses 2 separate outcome variables: the neurologic complication of the treatment procedure and the neurologic complication of future aneurysm hemorrhage. Risk of future hemorrhage, unfortunately, cannot be reasonably applied as an outcome because even with bare platinum coils, rehemorrhage of a previously ruptured aneurysm^6–8 and rupture of an unruptured aneurysm are rare events. Thus, massive numbers of enrolled subjects and impractically long follow-up periods would be required to establish an improvement in future hemorrhage rates by using proper statistical methods. Procedure-related morbidity and mortality certainly could be a primary outcome, but most observational studies suggest that these rates vary little, if any, among coil types. Because rehemorrhage rates and procedure-related morbidity and mortality are unlikely to differ very much among coil types, the outcome most important to the patients (ie, long-term neurologic function) has limited value as a primary outcome in the trials of new endovascular cerebral aneurysm therapies. Even so, neurologic outcome remains important because it is crucial to establish that new therapies do not adversely affect it.

Angiographic Aneurysm Recanalization or Recurrence.

Aneurysm recurrence following coil therapy was noted not long after the practice of coil therapy was first implemented clinically.⁹ Aneurysms recur in a substantial percentage of patients, with some relationship to factors such as aneurysm size and rupture status. These recurrences have been a clinical concern because it is generally presumed that an aneurysm that recurs following coil embolization has a higher risk of future hemorrhage than an aneurysm that remains completely occluded after coil therapy. This assumption drives the practice of follow-up imaging that is generally performed during the years following endovascular treatment.

We routinely perform follow-up imaging after coil embolization in an effort to find recurrences before they hemorrhage. Although we now have data that indicate that ruptured aneurysms rarely rehemorrhage following endovascular therapy,^6–8 we know that they occasionally do rehemorrhage, and we hope that by identifying recurrences, we can prevent most of them. One might argue that angiographic recurrences of cerebral aneurysms are a minor problem because such recurrences are rarely associated with hemorrhage. However, recurrences are undoubtedly a significant problem because the patient will have anxiety about future rupture and may be advised to undergo an additional therapeutic procedure. Indeed, angiographic recurrences drive the pursuit of improved endovascular therapies specifically targeted to reducing recurrences. Recurrence is a problem even for a patient whose aneurysm was coiled and who has not yet had a recurrence because the patient will be told that life-long imaging surveillance will be needed to look for a possible recurrence.

It may seem deceptively simple to identify aneurysm recurrence on high-quality angiograms. However, quantitative recurrence outcomes measures that are used currently are crude. Most practitioners rely on ordinal scales (eg, complete occlusion, neck remnant, aneurysm remnant; complete, near complete, incomplete; 100%, >90%, <90%). These scales have not been validated. Using these scales limits sensitivity because interval changes within a grade, such as interval worsening of an incomplete embolization, would be catalogued as “no change.” Furthermore, progression from any 1 grade to another is considered a failure, yet slight compaction with resultant change from “complete” to “near complete” is likely of less clinical relevance than progression from “near complete” to “incomplete.”

Another method of assessment of recurrence would be to assess any worsening, which might improve sensitivity but would assign equal value to a slight compaction of coils and a large recurrence. Some practitioners separate minor and major recurrences, but these seem to be simply euphemisms for retreatment, which we discuss. Such scales are rather subjective, with resulting interobserver and intraobserver variability that is a significant but not overwhelming problem.¹⁰ Although these scales are widely applied and rarely criticized, further analysis is needed before we accept their validity.

We treat cerebral aneurysms to prevent hemorrhage. Angiographic recurrence is only an indirect measure of potential for hemorrhage. Even so, angiographic recurrence has taken on a life of its own as an outcome. Our knowledge of the potential for angiographic recurrence is actually the primary motivation for developing devices that result in fewer recurrences. If current endovascular cerebral aneurysm therapies had the same durability (ie, low recurrence rate) as surgical clipping, we would not be looking for more durable alternatives to platinum coils. To move away from angiographic recurrences as an end point would be a major paradigm shift, and to do so would be to lose sight of the very effect that most new coils are being designed to address. We have to continue to study rates of angiographic recurrences because both patients and physicians consider recurrence to be a suboptimal outcome. Because the issue of angiographic recurrence has developed such an importance, it must continue to be a major focus of future trials, despite our inability to make completely objective and reliable assessments of this outcome.¹⁰

Aneurysm Retreatment.

Because retreatment usually follows immediately after discovery of an angiographic recurrence, it might be looked at as an indicator of a more severe form of angiographic recurrence than a recurrence that is not retreated. This outcome seems quite clinically relevant because it serves to define a serious recurrence necessitating exposing patients to additional procedures.

Yet, retreatment is actually even more confusing as an outcome variable than angiographic recurrence. “Was an aneurysm retreated?” appears to be a single objective question, but it actually incorporates 2 rather subjective questions. That is, the subjective questions “Can an aneurysm be retreated?” and “Should an aneurysm be retreated?” are rolled into 1 seemingly objective question.

“Can the recurrence be treated?” is terribly subjective and seems to be what most physicians refer to when defining a “major recurrence.” It is unscientific to consider recurrence of an aneurysm with a narrow neck that is retreated to be major simply because it underwent retreatment, whereas a recurrence of similar magnitude but with unfavorable anatomy for retreatment would be considered minor because it was not retreated. Variable use of assisting devices such as balloons or stents by physicians can lead to different answers to the “Can an aneurysm be retreated?” question.

Whether an aneurysm recurrence should be retreated is difficult or impossible to objectively determine because rehemorrhage rates in recurrent aneurysms are quite low and no firm guidelines are available anywhere for decision. We are aware of no study that attempted to determine whether any type of consensus among experts could be gleaned regarding the need for retreatment in a series of recanalized aneurysms. Even the 2 of us, trained in the same program and in practice together for many years, disagree in some cases about the need for retreatment. The question “Should an aneurysm be retreated?” also takes into account inconsistencies and biases inherent in human nature. Perhaps the physician had bad experiences with the first coil procedure and lacks enthusiasm for recommending retreatment, or perhaps the patient had bad experiences with the first coil procedure and lacks enthusiasm for following recommendations for retreatment. The subjectivity and variability in recommending and performing retreatment make retreatment a dubious outcome at this time.

Sample Size

Once an appropriate outcome measure is chosen, a trial must be designed so that enough patients will be enrolled in the trial to 1) allow a high probability of detection of a statistically significant result if there truly is a difference and 2) to allow a high level of confidence that there is truly no difference between groups if the data indicate no statistically significant difference. That is, the sample size must be large enough that the final results of the trial can be reasonably assumed to represent the truth. These sample-size estimates are mathematically simple to perform, but they are based on many assumptions that amount to educated guesses.

Neurologic decline, aneurysm recurrence, and aneurysm retreatment all occur in a minority of patients. New therapies will likely offer only an incremental improvement in these outcome variables rather than a complete elimination of such events. Detecting incremental changes in these variables will require relatively large sample sizes. The smaller the difference in outcome, the larger the required sample size becomes. The Table demonstrates the number of patients needed (ie, sample size) to reliably identify a statistically significant difference for various outcome estimates. Some differences that we would consider clinically significant, such as a 10% versus 5% recurrence rate or a 15% versus 10% recurrence rate, would require a larger sample size than is planned for the current trials. The HELPS, Cerecyte, and MAPS trials are each planning to enroll 500–630 patients.

Estimated samples sizes based on predictions of aneurysm recurrence rates

Recurrence Rates	Sample Size Needed
5% vs. 1%	530
10% vs. 1%	200
10% vs. 5%	870
15% vs. 5%	282
15% vs. 10%	1372
20% vs. 5%	152
20% vs. 10%	398
20% vs. 15%	1812
25% vs. 20%	2188
25% vs. 15%	500
25% vs. 10%	200

Note:—Calculations assume 0.05 significance level (alpha) and 0.80 power.

Estimating the expected recurrence rate in patients treated with platinum coils in a trial is not as straightforward as it might seem. There is an abundance of confusing literature regarding the subject of aneurysm recurrence. A wide range of recurrence rates for aneurysms treated with platinum coils has been reported in the literature. In large heterogeneous groups of patients with aneurysm, overall recurrence rates of 15%–34% have been reported in published series.^11–13 There is much variation in methods used to assess and report recurrence rates, but some clinically relevant trends are apparent:

• Recurrence rates clearly vary with aneurysm size and neck size.

  • Murayama et al¹¹ reported an overall recurrence rate of 5% for aneurysms of 4–10 mm with a neck <4 mm. For aneurysms of 4–10 mm with a neck of >10 mm, the recurrence rate increased to 20%.
  • Large aneurysms (10–25 mm) have been reported to have a recurrence rate of 35%–50%.^11,12
  • The recurrence rate for giant aneurysms has been reported to be 59%–87%.^11,14

• Recurrence rates are affected by rupture status. Unruptured aneurysms <9 mm have been reported to have a recurrence rate of 7%, whereas ruptured aneurysms <9 mm had a recurrence rate of 17%.¹⁵

• Degree of completeness of aneurysm occlusion at time of initial treatment is related to recurrence.¹²

• Packing attenuation of aneurysms affects recurrence rate.^16–18

• Length of the follow-up period is an important variable affecting the recurrence rate¹² (ie, the longer the follow-up period, the more recurrences that are identified).

• Use of adjunctive techniques such as stent placement might also impact recurrence rates.¹⁹

In a randomized trial, the proportion of patients affected by these variables would theoretically not differ greatly between the 2 treatment arms. On the other hand, random variations might actually turn out to be significant and confound the interpretation of the final results, especially with a small overall sample size.

Estimating the recurrence rate for aneurysms treated with a new type of coil in a trial is really accomplished by making a guess. Some data from registries or case series might be available to allow an estimation of recurrence rate. Another approach would be to use an estimate of a reduction in recurrence rate relative to platinum coils that is considered clinically significant. For example, decreasing the recurrence rate from 15% to 14% would not be considered clinically relevant by most physicians, but decreasing the recurrence rate from 15% to 10% would be a 33% relative reduction in recurrences and thus would probably be considered a clinically significant improvement by many practitioners. Note in the Table that 1372 patients would need to be randomized to prove such a statistically significant difference between a 15% and a 10% recurrence rate. The past attempts to establish reduced recurrence rates with small registries of patients treated with new coils exemplify investigator and manufacturer hubris; attempting to prove a significant improvement in recurrence rates with a randomized trial of only 500 patients might also indicate a significant degree of overconfidence of the investigators and manufacturers.

Estimates of recurrence rates and sample sizes for trials in the Table and in the current trials assume mathematically that all patients have the same disease and the same future risks. It is an oversimplification to assume that 500 patients with aneurysms all have basically the same disorder. One could argue that because ruptured and unruptured aneurysms have very different natural histories and different recurrence rates, they represent 2 very different clinical problems. An analogous clinical scenario would be the differences between symptomatic and asymptomatic carotid stenosis. Thus, it is potentially very problematic to include both ruptured and unruptured aneurysms in the same trial. Additionally, small aneurysms have very different natural histories and recurrence rates than large aneurysms, so it is wrong to ignore these different disease states. Of course, if only patients with a very specific type of aneurysm (eg, a small, ruptured aneurysm treated without adjunctive techniques) were eligible for enrollment in a trial, the trial would progress more slowly. Yet, it is clearly possible to conduct a large trial of a somewhat homogeneous patient population because this was demonstrated in the ISAT trial, which enrolled 2143 patients with ruptured aneurysms, of which 93% were ≤10 mm.⁶ A balance must be struck between practicalities of enrollment rate and an excessively heterogeneous patient population.

The number of subjects needed to show a statistically significant difference can be minimized by logical selection of aneurysm types. The ideal aneurysm type would be one that recurred frequently with bare platinum coils but infrequently with the new coils. Small aneurysms represent the most frequent aneurysm type but may be a poor choice for randomized trials because they are so effectively treated with platinum coils, unless they have a wide neck or they are ruptured.^11,12,15 Large and giant aneurysms recur consistently with bare platinum coils, so theoretically they might be great candidates for new coils. However, these aneurysms are relatively uncommon, meaning a trial focused on these aneurysms would take a long time to complete. Also, it is quite likely that even modified coils would fail to durably occlude large and giant aneurysms, so showing a difference compared with bare platinum coils may be difficult.

Given the shortcomings of enrolling small and large aneurysms, one might conclude that medium-sized aneurysms, on the order of 6–12 mm, might be the ideal type to enroll or that perhaps simply enrolling patients with ruptured aneurysms <10 mm in diameter and with good clinical grade (ie, Hunt and Hess grade 1–3) would be a reasonable approach. This was the population of patients enrolled in the ISAT trial, which enrolled 2143 patients,⁶ so it would certainly be possible to enroll an adequate number of patients by using this approach. Ruptured aneurysms are particularly attractive for a trial aimed at reducing aneurysm recurrence because 1) they recur at a higher rate than unruptured aneurysms¹⁵ and 2) we should probably be more concerned about recurrences of aneurysms that have previously ruptured than those that have not previously ruptured.

To our knowledge, current and pending trials do not limit enrollment except to exclude giant aneurysms. It may be possible to do post hoc subgroup analyses focused on particular types of aneurysms, but no trial has yet been empowered to allow such analyses to be performed convincingly. We fear that aneurysm heterogeneity will confound the interpretation of results of trials unless taken into account in the initial trial design; therefore, sample size issues may be the Achilles’ heel of the first wave of trials of next-generation coils.

Bias

For results of a trial to be valid, the trial must be conducted in an unbiased manner. A number of potential sources of bias exist and must be considered. Bias of the treating physician may impact trials of new endovascular aneurysm therapies. For example, if a practitioner is hopeful that a new coil will improve outcomes, he may behave differently in the way he uses the device and thus may affect outcomes. Ideally, the studies would be blinded to the type of device being implanted so that operator bias is minimized or eliminated. With devices that are obviously physically different from bare metal, blinding would be difficult or impossible. However, on the basis of our experience, some products such as in the Cerecyte trial could readily be compared in blinded fashion with bare platinum.

Clinical outcome also might be assessed in a biased manner by the treating physician. To detect differences in device safety as defined changes in the neurologic conditions of patients, independent neurologists should examine the patients after the procedure and at a predetermined later end point. Independent neurologic examination was a key component to other trials of invasive therapy, such as the NASCET²⁰ and ACAS²¹ trials. Another validated unbiased method of assessment of neurologic status is to have the patients assess their own outcome through the modified Rankin Scale, as was done in the ISAT trial.⁶ Physicians and patients care deeply about the complication rates. Without rigorous unbiased assessment and recording of the neurologic condition of the patient, it will be impossible to detect subtle but clinically significant differences in device-related complications.

Blinded interpretation of angiographic outcomes by a central reader is possible and is expected to occur in ongoing coil trials. Physicians treating patients in trials tend to look at their own results with rose-colored glasses, so they cannot be trusted to assess their own outcomes reliably. The angiographic outcomes will likely be the most significant measure in the ongoing trials, so it is reassuring to us that these interpretations can be performed in a blinded unbiased manner.

Sponsor bias is an important issue because trials of devices tend to be sponsored by the device manufacturer. The device manufacturer has inherent bias because of interest in demonstrating that the product is superior, to profit financially. Active physician participation in the trial can help to alleviate some of the biases that might arise during the design or conduction of the trial. Sponsor bias can also manifest itself after the data collection is completed through data suppression (ie, data that the sponsors view as unfavorable to their cause may never be subjected to peer review and publication).

Registries, Trials, and the Future

The manufacturers of Matrix, Cerecyte, and HydroCoil were in a hurry to get these products to market and managed to achieve their goal of quick product release with little data collection by using the 510(k) approval process of the US Food and Drug Administration. This was obviously not an unreasonable business strategy because they were successful in getting 510(k) approvals and they were then able to sell large numbers of these coils to physicians despite substantial proof of any improvement of outcomes relative to platinum. Next, the manufacturers all performed registries that were meant to show a decreased recurrence rate relative to platinum. However, because of a lack of a suitable historical control group treated with platinum coils and because of recurrence rates in the registries that were substantially above zero, these registries failed to achieve the manufacturers’ goal of convincing proof of superiority. In retrospect, it now seems silly to have expected the registries to show a major improvement in outcome, especially because the sample size was so low (191 for HEAL² and 100 for ACTIVE¹) that only an astoundingly good improvement in recurrence rates (ie, essentially no recurrences) would have been convincing evidence of a real benefit (see the Table). As time has passed, physicians have become more skeptical and have begun to wonder which, if any, of the next-generation coils will truly reduce aneurysm recurrence. Finally, the manufacturers have realized that only a randomized trial will convince most physicians that a particular coil reduces recurrence rates relative to platinum. It is essential that these trials be performed with valid outcome measures, appropriate sample size, and minimal bias.

We applaud the efforts to conduct randomized prospective trials of next-generation coils, but expectations about what these trials can and cannot show need to be realistic, given the inherent limitations. It is likely that these trials will teach us important lessons about the treatment of cerebral aneurysms, but we may yet have to learn the hard way. The current randomized trials comparing different coil therapies will not lead us to an end of debate but will instead open a new era of incremental progress based on knowledge rather than on theory and speculation.