Differences in the Angiographic Evaluation of Coiled Cerebral Aneurysms between a Core Laboratory Reader and Operators: Results of the Cerecyte Coil Trial

Abstract

BACKGROUND AND PURPOSE:

Independent evaluation of angiographic images is becoming widely applied in the assessment of treatment outcomes of cerebral aneurysms. In the current study, we assessed the agreement between an independent core laboratory and the operators regarding angiographic appearance in a recent randomized, controlled trial.

MATERIALS AND METHODS:

Data were derived from the Cerecyte Coil Trial. Angiographic images of each coiled aneurysm, taken immediately after embolization and at 5- to 7-month follow-up, were evaluated by the operator at the treating center and by an independent neuroradiologist at the core laboratory. For the purpose of this study, images were interpreted on a 3-point scale to provide uniformity for analysis; grade 1: complete occlusion, grade 2: neck remnant; and grade 3: sac filling. “Unfavorable angiographic appearance” was defined as grade 3 at follow-up or interval worsening of grade between the 2 time points.

RESULTS:

The study included 434 aneurysms. Immediately after embolization, grade 3 was reported by operators in 39 (9%) compared with 52 (12%) by the core laboratory (P = .159). On follow-up, grade 3 was reported by operators in 44 (10%) compared with 81 (19%) by the core laboratory (P < .0001). Overall, operators noted unfavorable angiographic appearance in 78 (18%) compared with 134 (31%) by the core laboratory (P < .0001). At every time point, agreement between the core laboratory and the operators was slight.

CONCLUSIONS:

Unfavorable angiographic appearance was noted almost twice as frequently by an independent core laboratory as compared with the operators. Planning of trials and interpretation of published studies should be done with careful attention to the mode of angiographic appearance interpretation.

Angiographic imaging represents the principal metric for evaluating endovascular treatment of cerebral aneurysms and comparing the efficacy of different coil types. The relationship between complete or stable aneurysm occlusion and incomplete occlusion and the risk of delayed aneurysm re-bleeding remains uncertain. Angiographic results are also used to decide on whether further treatment is considered. Treatment success and indications for further management, including the possibility of an additional intervention, are highly dependent on angiographic appearance in the belief that patients with incomplete occlusion are at risk of hemorrhage, though the size of this risk is unknown. Most previous literature included assessments of angiographic appearance on the basis of readings performed in the treating center(s).^1–3 However, other investigators,^4–11 especially in the setting of randomized, controlled trials,^9–11 have used an independent core laboratory for angiographic assessment. Independent core laboratories represent a centralized approach and can provide a standard operating procedure for imaging review. In this respect, they are used to provide a more objective assessment within a trial setting than may be possible at treating centers. This suggested objectivity may also result from the core laboratory being blinded to treatment settings, using the same prespecified systematic reviewing protocols, and usually consisting of more experienced reviewers. On the other hand, operators, being aware of the details of the patient status and the procedure, may have a hands-on sense of “success” when considering the risk-benefit ratio of adding more coils to the treated aneurysm. Also, there is currently no study addressing the degree of objectivity of operators judging their own procedural results at the end of prolonged risk procedures.

Reported angiographic appearance after endovascular treatment has varied widely in previous studies.^12–15 Many factors are known to influence angiographic appearance, including aneurysm size, neck diameter, initial rupture status, and follow-up duration.^10,16–18 However, the influence of core laboratory interpretation on outcomes for research purposes, compared with operator evaluations at the treating centers, remains poorly studied.^19,20 In the current study, we assessed the agreement between an independent core laboratory and the operators at the treating centers regarding angiographic outcomes in a recent randomized, controlled trial.

Materials and Methods

All data were derived from the Cerecyte Coil Trial.^9,21 The trial compared patients with either a ruptured or unruptured aneurysm treated with bioactive Cerecyte coils (Micrus Endovascular, San Jose, California) or bare platinum coils. Details about patient enrollment and inclusion/exclusion criteria are described elsewhere.^9,21 Briefly, patients with 1 aneurysm deemed suitable for endovascular treatment were enrolled into the trial and randomly assigned to either receiving Cerecyte or bare platinum coils. There were 23 participating centers worldwide. Aneurysms were imaged by DSA immediately after embolization and by either DSA or MR angiography at 5–7 months of follow-up. Each imaging session was evaluated by the operator and by an independent core laboratory. The core laboratory consisted of a single, experienced neuroradiologist who did not have access to the center evaluations and was blinded to the type of coil used. In a similar fashion, the operators at the treating centers were blinded to the core laboratory assessment. Images of the studied aneurysms were sent to the core laboratory reader on compact discs for assessment without further details about the procedure other than vessel location of the target aneurysm. More details about the process of sending images are described elsewhere.⁹

Angiographic appearance was evaluated at the core laboratory and each of the treating centers by use of predefined scales that are summarized in Table 1. Operators evaluated angiographic images immediately after embolization and at follow-up by use of a 4-point scale, including 1) complete occlusion; 2) subtotal occlusion or neck remnant; 3) incomplete occlusion/aneurysm filling; and 4) failed embolization. The core laboratory used a slightly modified 4-point scale immediately after embolization and another one at follow-up. The core laboratory scale immediately after embolization included 1) apparently complete occlusion; 2) overlapping coils/neck; 3) neck remnant; and 4) inflow/incomplete occlusion (body filling). The core laboratory scale at follow-up included 0, no change compared with immediate postembolization DSA; 1, complete occlusion; 2, stable neck appearance/neck remnant; and 3, residual filling. For the purposes of this study, and after confirming with the core laboratory reader about the definitions of the scale points, we considered “complete occlusion” from operator readings to be equivalent to “apparently complete,” “overlapping coils/neck,” or “complete occlusion” from core laboratory readings (grade 1 for the purposes of the current study); “subtotal occlusion or neck remnant” to be equivalent to “neck remnant” or “stable neck appearance/neck remnant” (grade 2 for the purposes of the current study); and “incomplete occlusion/aneurysm filling” to be equivalent to “inflow/incomplete (body filling)” or “residual filling” (grade 3 for the purposes of the current study). When the core laboratory reported that there was “no change” in the angiographic appearance on follow-up, the aneurysm was given the same grade of the core laboratory readings in the immediate postembolization assessment. “Failed embolization” cases were excluded from all analysis as well as cases with incomplete imaging data. Last, we defined “unfavorable angiographic appearance” as either grade 3 at follow-up or interval worsening of grade between the immediate postembolization angiogram and the follow-up imaging.

Table 1:

Definitions of the scales of assessment used

Angiographic Assessment Scale in the Current Study	Operator Angiographic Assessment Scalea	Core Laboratory Angiographic Assessment Scale Immediately after Embolization	Core Laboratory Angiographic Assessment Scale at Follow-Up
Grade 1	Complete occlusion	Apparently complete occlusion or overlapping coils/neck	Complete occlusion
Grade 2	Subtotal occlusion or neck remnant	Neck remnant	Stable neck appearance/neck remnant
Grade 3	Incomplete occlusion/aneurysm filling	Inflow/incomplete occlusion (body filling)	Residual filling

^aCases assessed as “failed embolization” by the operators were excluded from the analysis of this study.

Statistical Analysis

A simple κ coefficient and a 95% CI were used to assess agreement between the core laboratory reader and the operators in evaluating angiographic appearance. Kappa scores were interpreted as follows²²: <0 as “less than chance agreement,” 0.01–0.20 as “slight agreement,” 0.21–0.40 as “fair agreement,” 0.41–0.60 as “moderate agreement,” 0.61–0.80 as “substantial agreement,” and 0.81–0.99 as “almost perfect agreement.”

The McNemar test of symmetry was used to assess the marginal homogeneity of the proportion of grade 3 immediately after embolization and at follow-up as well as the proportion of aneurysms identified as having had an “unfavorable appearance” for both the operators and core laboratory readings. All statistical analysis was performed with the use of SAS software (v9.3, SAS Institute, Cary, North Carolina).

Results

Patients (n = 497) with 1 confirmed aneurysm suitable for coil embolization were enrolled in the trial. Of these 497 target aneurysms, 49 (9.9%) were excluded from the angiographic analysis because of incomplete imaging data and 14 (2.8%) were also excluded as the operators deemed them as “failed embolizations.” Thus, the current analysis included 434 aneurysms (87.3%) of 497 treated aneurysms that were reviewed immediately after embolization and at 5–7 months of follow-up by both the operators and core laboratory reader.

Angiographic appearance of the treated aneurysms immediately after embolization is presented in Table 2. Of the 395 (91%) of 434 aneurysms graded either 1 or 2 by the operators, 49 were graded as 3 by the core laboratory reader. The overall number of aneurysms graded as 3 by the operators was 39 (9%) of 434 aneurysms compared with 52 (12%) graded as 3 by the core laboratory reader (P = .159).

Table 2:

Immediate postembolization angiographic appearance on the 3-point scale

Core Laboratory Readings	Operator Readings
	Grade 1, n (%)a	Grade 2, n (%)	Grade 3, n (%)	Total
Grade 1	108 (48)	41 (24)	18 (46)	167
Grade 2	97 (43)	100 (59)	18 (46)	215
Grade 3	21 (9)	28 (17)	3 (8)	52
Total	226	169	39	434

^aPercentages follow columnar order.

Angiographic appearance of the treated aneurysms at follow-up is presented in Table 3. Of the 390 (90%) of 434 aneurysms that were graded as either 1 or 2 by the operators, 57 (16%) were graded as 3 by the core laboratory reader. The overall number of aneurysms graded as 3 by the operators was 44 (10%) of 434 aneurysms compared with 81 (19%) graded by the core laboratory reader (P < .0001).

Table 3:

Follow-up angiographic appearance on the 3-point scale

Core Laboratory Readings	Operator Readings
	Grade 1, n (%)a	Grade 2, n (%)	Grade 3, n (%)	Total
Grade 1	105 (43)	24 (16)	1 (2)	130
Grade 2	124 (51)	80 (55)	19 (43)	223
Grade 3	14 (6)	43 (29)	24 (55)	81
Total	243	147	44	434

^aPercentages follow columnar order.

Of the 356 aneurysms that had “favorable angiographic appearance” according to the operators, 86 (24%) had “unfavorable angiographic appearance” according to the core laboratory reader. The overall “unfavorable angiographic appearance” according to the operators was 78 (18%) of 434 aneurysms compared with 134 (31%) according to the core laboratory reader (P < .0001).

Agreement between the core laboratory reader and the operators by use of the simple κ coefficient is presented in Table 4. At every time point, agreement between the core laboratory and the operators was slight.

Table 4:

Agreement between operators and the core laboratory reader on different time point assessments

	Simple κ Coefficient (95% CI)	Interpretation
Immediately after embolization on 3-point scale	0.138 (0.067–0.209)	Slight agreement
Follow-up on 3-point scale	0.189 (0.121–0.258)	Slight agreement
Favorable versus unfavorable outcomes	0.121 (0.042–0.199)	Slight agreement

Discussion

In the current study, we have shown that interpretation angiographic appearance for the same treated aneurysms can significantly change when reviewed by an independent core laboratory versus the treating operators. At follow-up, the rates of both unfavorable angiographic appearance, as well as those deemed as grade 3, were nearly twice as high for the core laboratory compared with the operator assessments. Agreement between the core laboratory and operators was only “slight” after embolization and at follow-up. This appears to be far worse than the “good” agreement among readers in a single-center study.²³ These findings suggest not only that single-center studies with angiographic appearance assessed by operators are fundamentally different from those assessed by independent core laboratories but also that outcomes assessed at the treating centers are likely to be systematically more favorable than those from core laboratories, even for the same angiograms. The core laboratory recognized that some post-coiling angiograms appear complete but have coils overlapping aneurysm necks due to the view needed to separate parent vessel from coils not being available. An example is some anterior communicating aneurysms best viewed with a basal submental vertex view, but not done at the time of coiling.

The impact of core laboratory evaluation of outcomes has previously been studied in cardiology and its superiority over operator readings suggested.^24–28 A recent systematic review and meta-analysis of the endovascular literature for cerebral aneurysms¹⁹ found that studies that used an independent core laboratory reported higher numbers of unfavorable angiographic outcomes as compared with treating center evaluations, but the quality of evidence was low for the heterogeneity of the literature studies. The current study, which allowed direct comparison of individual cases from a randomized cohort instead of relying on literature reviews, adds additional evidence for the impact of core lab readings on reported outcomes. Another group of investigators²⁰ has shown that core laboratory evaluation of immediate angiographic images resulted in doubling the number of incompletely occluded aneurysms when compared with treating center evaluation, similar to our own findings.

The current study had some limitations. First, to achieve uniformity of the reviews as much as possible between core laboratory reader and operators at the treating centers, we had to exclude 13% of the aneurysms for the various reasons that were listed earlier. We could not standardize the level of experience and former training of the core laboratory reader with the individual operators and also account for the variations between readers from different centers. Also, the Cerecyte Coil Trial was not set up to answer the specific question of comparing the core laboratory and operator readings. One of the resulting challenges of the differently oriented design was the ambiguity of the boundaries between the grades of each scale that was used. The possible misperception of the scale definitions among readers can, in part, explain some of the resulting differences. The core laboratory reader of the Cerecyte Coil Trial included, for every case, a derivation of “millimeter” measurements and measured neck remnant depth establishing 2 mm as an attempted measured boundary between grade 2 and grade 3 instead of distinguishing them in the eye of the beholder. This strategy was not applied by the operators in the treating centers. The core laboratory reader received a limited amount of information about the individual cases on compact discs, which may affect the overall understanding of the procedure's details and possibly affect the subsequent judgment. Finally, this study assesses 1 core laboratory with a single reader, which reduces the generalizability of the conclusions. A future study would ideally recruit different core laboratory readers to assess for interreader variability.

Conclusions

Unfavorable angiographic appearance was noted almost twice as frequently by an independent core laboratory as compared with the operators at the treating centers for aneurysms treated with coil embolization. The planning of trials and interpretation of published studies should be done with careful attention to the mode of angiographic appearance interpretation. More specific attention to the consistent use of the categories of less-than-complete aneurysm obliteration may reduce the discrepancies.