Observer Agreement Regarding the Necessity of Retreatment of Previously Coiled Recurrent Cerebral Aneurysms

Do interventional neuroradiologists agree among each other regarding retreatment of previously coiled aneurysms? Because the need for retreatment is also used as a primary outcome measure in randomized studies, this observation extends well beyond individual patients. The well-known group from the Mayo Clinic addresses that question in this article. They identified 27 previously treated intracranial aneurysms and asked 5 readers to independently rate each on a 5-point scale, and in those cases needing retreatment to indicate which type of therapy would be best. There was only fair agreement regarding retreatment and a wide variation in the type of therapy chosen for the retreatment. The study demonstrates substantial variability among observers not only in whether to retreat a recurrent aneurysm but also in how to treat it. These findings suggest that patient management varies widely across treating physicians and also calls into question the use of “retreatment” as an objective end point in clinical trials.

Abstract

BACKGROUND AND PURPOSE:

The decision regarding whether or not to retreat a previously treated aneurysm not only directly impacts patient care but also serves as a primary outcome measure in numerous, leading randomized controlled trials of modified coils. Our aim was to determine the degree of interobserver variability regarding the need and type of treatment for recurrent aneurysms following coil embolization.

MATERIALS AND METHODS:

Twenty-seven previously treated recurrent aneurysms were identified. Five independent readers rated each aneurysm on a 5-point scale: 1, definitely do not retreat; 2, probably do not retreat; 3, unsure; 4, probably retreat; and 5, definitely retreat. The readers noted, in grades 2–5, the type of retreatment preferred, including simple coiling, balloon- or stent-assist coiling, or surgical clipping. Intraobserver agreement κ was calculated. Retreatment recommendations were evaluated between observers by using a Wilcoxon signed rank comparison. Descriptive statistics were performed for categoric treatment-type comparisons.

RESULTS:

At least 2- or 3-point differences between 2 readers were present in 17 (63%) and 11 (41%) of 27 cases, respectively. The median κ was 0.27 (range, 0.04–0.43), which indicates fair agreement. Differences between readers varied, with readers 4 and 5 more often recommending retreatment compared with reviewers 1–3 (P < .05). Wide variation was noted in treatment approach, with recommendations for surgical clipping ranging from 2 (7%) to 18 (67%) of 27 cases between readers 1 and 5.

CONCLUSIONS:

The current study demonstrates substantial variability among observers not only in whether to retreat a recurrent aneurysm but also how to treat it. These findings suggest that patient management varies widely across treating physicians and also calls into question the use of “retreatment” as an objective end point in clinical trials.

The use of endovascular embolization of cerebral aneurysms is widespread and has become the standard of care for treatment of many intracranial aneurysms,^1–5 based on favorable data from large randomized clinical trials.^2,6 Following endovascular therapy, concern for recurrence and the decision to retreat aneurysms have important ramifications.⁷ Patients must return for ongoing follow-up, with resultant anxiety and expense. Additionally, retreatment procedures, either with open surgery or via endovascular means, are not without risk.⁸ Furthermore, the “need to retreat” is a primary outcome in many previous case series as well as ongoing randomized clinical trials comparing coil types.⁹

Our understanding of the natural history of aneurysm remnants and recurrences continues to evolve. Even though aneurysm retreatment is entrenched as a widely accepted routine procedure for recurrent aneurysms,^8,10,11 the angiographic criteria that define a “need to retreat” have received little or no attention in the literature.¹² As best we can tell, the ability to place additional coils into a recurrent aneurysm cavity likely is 1 criterion, especially for endovascular retreatment considerations.^3–5,13 However, we are unaware of any specific size or morphology criterion that indicates that an aneurysm should be retreated. The use of adjunctive techniques such as balloon remodeling or stent-assisted coiling probably impacts the definition of “need to retreat,” especially in centers not prone to open neurosurgical re-intervention.

Given the clinical and research relevance of aneurysm-retreatment categorization, it should be reproducible across multiple practitioners or researchers. To our knowledge, there exists no formal study of the interobserver variability associated with retreatment following coil embolization. In the current study, we determined the degree of interobserver variability regarding the need and type of treatment required for recurrent or residual aneurysms following coil embolization.

Materials and Methods

Following institutional review board approval, 27 cases of previously coiled aneurysms with documented recurrence were compiled. Most had been previously identified in a prospective trial of contrast-enhanced MR angiography in the follow-up of coiled intracranial aneurysms,¹⁴ and the remainder were supplemented by recurrences identified during routine follow-up of previously coiled aneurysms. Aneurysm recurrence was the criterion for inclusion in the study series to prevent morphologic bias. These images were then compiled, de-identified, and placed into a 27-page PDF file (Adobe Systems, San Jose, California). Each page included a DSA of the initial uncoiled aneurysm if available, the final postcoiling working-projection DSA, and the follow-up DSA demonstrating the recurrence. The file was then distributed to 5 practitioners, including 2 endovascular neurosurgeons with 3 and 7 years' experience, respectively, and 3 interventional neuroradiologists, 2 with 13 years' experience and 1 with 12 years' experience. Two of the neuroradiologists were trained in the same fellowship and have been in practice together for 7 years, and 1 of the neurosurgeons was a fellow with these 2 neuroradiologists.

For the purposes of the current study, to diminish heterogeneity among responses based on clinical rather than imaging findings, we told the reviewers to make the following assumptions for all 27 patients: 1) The patient was a 50-year-old women who had an SAH from the index aneurysm, 2) she made a full clinical recovery, and 3) her follow-up was at 6 months.

These same observers independently reviewed each set of images and graded the need for retreatment, if any, for each case. We used a 5-point graded response in which grade 1 was defined as “definitely do not retreat,” grade 2 was “probably do not retreat,” grade 3 was “unsure regarding need for retreatment,” grade 4 was “probably should retreat,” and grade 5 was “definitely should retreat.” The reviewer was then asked to specify the exact mode of retreatment, if retreatment was to be undertaken. Modes of retreatment included coil embolization without adjunct, coil embolization with balloon assist, coil embolization with stent assist, or surgical clipping of the recurrent or residual aneurysm.

Statistical Analysis

All statistics were performed by using JMP 7.0.1 software (SAS Institute, Cary, North Carolina). κ scores were calculated between reviewers and are reported as the median and range. Plots were made for individual reviewer treatment ratings, and analysis was made between pairs of reviewers on the recommendation to retreat, by using a Wilcoxon signed rank test. Additional evaluation of adjunctive measures used to treat the recurrence was analyzed by using a Bowker test.

Results

Interobserver Variability for Retreatment

Representative cases are shown in Figs 1 and 2. The calculated agreement κ of the 5 observers over the 27 cases presented was κ = 0.27 (range, 0.04–0.43) with a κ of 0.0 being equal to random chance and 1.0 being perfect agreement. A 3- or 4-point difference between at least 2 readers was present in 10 (37%) and 2 (7%) cases, respectively. Twenty-three (85%) of 27 cases showed at least a 1-point difference between at least 2 readers, while 17 (63%) of the 27 cases had at least a 2-point difference between at least 2 readers. Figure 3 demonstrates the distribution of each reviewer's responses on the 5-point scale. Analysis between reviewers demonstrates that reviewers 4 and 5 were more likely to recommend retreatment than reviewers 1–3 (P < .05). When comparing recommendations to retreat, we found no difference based on the reviewers' training (ie, neurosurgery versus neuroradiology) in the current series.

Fig 1.

DSA in the anteroposterior plane showing the initial aneurysm conformation (Pre), the initial postcoiling final angiogram (Post), and the follow-up angiogram (Follow-Up) demonstrating recurrence. Case 9 of the series demonstrates a carotid terminus aneurysm for which all the reviewers were in complete agreement. Each reviewer scored this aneurysm as a grade 5, definitely retreat.

Fig 2.

DSA in the anteroposterior plane showing the initial aneurysm conformation (Pre), the initial postcoiling final angiogram (Post), and the follow-up angiogram (Follow-Up) demonstrating recurrence. Case 21 demonstrates a large basilar caput aneurysm that had a complete occlusion with the initial treatment and a recurrence with recanalization of the base. The average score for this aneurysm was a 2.6 ± 1.8, with scores of 1, 1, 5, 4, and 2. This case represents the highest variability of the 27 rated cases.

Fig 3.

Relative tendency to recommend retreatment across readers. Readers responses on the need to retreat by using the 5-point retreatment scale. Readers 4 and 5 were more likely to recommend retreatment than observation compared with readers 1–3 (asterisk indicates P < .05).

Fig 4.

Treatment adjunct per reviewer. Plot of the recommended adjunctive treatment technique for each of the 5 reviewers by the total number of cases. No significant difference between reviewers and adjuncts was found by using the Bowker test for significance. There was a trend toward significance for readers 1 and 5 (P < .07).

Recommended Retreatment Technique

The adjunct recommended for retreatment varied, depending on the strength of the retreatment. For cases that were scored a 2 or 3 (probably do not retreat and unsure), surgical clipping was most commonly recommended (53% of 2s and 83% of 3s). When retreatment recommendations were a 4 or 5 (probably retreat and definitely retreat), surgical clipping was recommended in 29% and 28% of cases, respectively, whereas primary coiling was recommended in 25% and 39%, respectively. Stent placement in this group was recommended in 25% and 7%, respectively. Because retreatment was more strongly recommended, primary coiling was more often recommended over surgical clipping or stent placement. No difference was found when comparing the retreatment technique on the basis of reviewers' training (neurosurgery versus neuroradiology).

Discussion

In this study, we demonstrated substantial variability among experienced operators when deciding, on morphologic features alone, whether a series of aneurysms should be retreated and how. This finding is important for several reasons. First, it suggests that patient management following aneurysm recurrence may vary widely among operators and centers. Second, these findings suggest that retreatment may not be an ideal outcome in a clinical trial, given wide interobserver variability. Third, it sheds light on a previously neglected aspect of patient care—that is, whom should we retreat and why?

The underlying reasons behind the wide variation among our readers remain unknown but likely are multifactorial. Issues such as ease of retreatment probably impacted individual ratings—that is, rounded narrow-necked recurrences, irrespective of size, might have prompted some reviewers to recommend treatment. Difficult wide-necked cases might have prompted those who favor stents to recommend retreatment. This tendency is shown by the fact that in cases in which readers were uncertain about retreatment, surgery was recommended, while in cases with high degrees of certainty, readers were amenable to coil embolization without the need for a stent.

In our study, roughly two-thirds of cases showed differences of ≥2 points. A 2-point difference indicates a difference of “not sure” to “definitely treat” or “definitely do not treat”, or a difference of “probably treat” to “probably not treat.” Thus, even a 2-point difference might lead to differences in patient management. Thirty-seven percent of cases varied by 3 points. A 3-point variation indicates a difference between “definitely treat” to “probably not treat” or vice versa, again indicating substantial differences in patient management.

One might theorize that treating physicians may be more likely to recommend retreatment for ruptured recurrent aneurysms over unruptured recurrences in certain cases. To prevent this possible bias in the current study, we told the reviewers to assume that all patients were 50-year-old women who had SAHs from the index aneurysm and who had made full clinical recoveries, and that their follow-up imaging was performed at their 6-month return visit. This information was included to prevent retreatment bias based on rupture status, age, and recovery status; however, these likely play a strong role in a provider's decision about recurrent aneurysm retreatment. More research is needed in this area. Likewise, we cannot be certain that our small cohort is typical of all recurrent aneurysms. It remains possible that these cases were particularly challenging. Future studies are needed to address this issue.

Several studies have been performed evaluating the occlusion rates of aneurysms and their recurrence.^13,15–19 Previous evaluations of intraobserver agreement in the assessment of aneurysm recurrence have found that agreement among observers expectedly improved as the number of response options decreased.¹² Likewise studies have demonstrated that recurrence depends on the initial size of the aneurysm as well as previous rupture status.^13,17 A large meta-analysis by Ferns et al²⁰suggested that approximately 20% of all coiled aneurysms recur and half of these go on to be retreated, but there are no good guidelines that can help a clinician decide whether observation or retreatment is more appropriate. Much of our decision-making about retreatment of recurrent aneurysms relies on experience and is likely affected by previous cases and poor outcomes.

This study has several limitations. First, 4 of the readers had overlapping training often with similar training styles. Nevertheless, the reviewer most likely to recommend observation and the reviewer most likely to recommend retreatment worked together. Second, previous studies from our group¹² have suggested, as is evident in our current data, that the increase in number of rating options leads to a decrease in the intraobserver agreement (κ = 0.27) as seen in our data. Likewise, the current study was not designed to look at which part of aneurysm morphology is more likely to evoke a response advocating retreatment. Additional studies are needed to further elucidate morphometric differences that would aid in defining the need for retreatment.

Abbreviations

DSA

digital subtraction angiography

SAH

subarachnoid hemorrhage

NCRR

National Center for Research Resources

NIH

National Institutes of Health

Post

after

Pre

before