Aggregate Analysis of the Literature for Unruptured Intracranial Aneurysm Treatment

Abstract

BACKGROUND: Publication bias and/or true heterogeneity can skew aggregate impressions from scientific literature. To better determine aggregate measures for unruptured intracranial aneurysm (UIA) treatment, we analyzed adverse outcome rates of surgical clipping and endovascular coil embolization.

METHODS: Two independent reviewers searched MEDLINE for studies publishing adverse outcome rates for endovascular coiling and surgical clipping between January 1990 and July 2003. Studies were classified as single-center, multicenter, or community-based. We defined adverse outcome rates as combined all-cause early or in-hospital morbidity and mortality. We determined cumulative adverse outcome rates by plotting precision measure (sample size) against trial-specific effect (adverse outcome rate).

FINDINGS: We included 4 endovascular coiling multicenter/community-based studies (1019 patients) and 13 single-center studies (810 patients) and 5 surgical clipping multicenter/community-based studies (10,541 patients) and 23 single-center studies (1759 patients). Cumulative adverse outcome rates for endovascular coiling and surgical clipping were 8.8% (95% confidence interval [CI] 7.6%–10.1%) and 17.8% (95% CI 17.2%–18.6%).

INTERPRETATION: Scattergram distribution illustrated the magnitude of bias in current literature reporting UIAs. Major parts of the literature may have underestimated surgical clipping morbidity and mortality, which can be attributed to bias from smaller retrospective studies. Neuroradiologic coiling studies were less likely to include factors contributing to inaccurate adverse outcome rates.

For many physicians, clinical practice is based on scientific literature that often lacks data from prospective, randomized trials. In these circumstances, an aggregate impression is formed from a dataset that consists primarily of single-center case series and retrospective series. Although not immediately apparent, influences such as publication bias and dataset heterogeneity may collectively skew the aggregate impression and thus have a profound impact on clinical management.

Currently, no independently-blinded, randomized clinical trials address whether coiling or clipping is optimal in the treatment of unruptured intracranial aneurysms (UIAs). Although prospective, nonrandomized data now available from the International Study of Unruptured Intracranial Aneurysms (ISUIA) provide an important assessment of treatment options and address many issues with data heterogeneity by providing uniform definitions of starting points, aneurysmal characteristics, end points, and follow-up of patients, it remains important to place these new results in context with previously published data to address unexpected outcomes and understand how new information may influence our thinking.

One way to frame the body of literature concerning the treatment of UIAs is to adapt previously described scattergram techniques (1), which allow us to use a simple graphical test to outline the summed magnitude of publication bias, data set heterogeneity, and other confounding factors that, though not immediately apparent, may significantly skew our decisions for therapy and cause conflict and controversy in our recommendations for treatment.

Here we systematically reviewed literature reporting clipping and coiling adverse outcome rates (AORs) and performed funnel plot analysis to further understand how the magnitude of biases affected reported AORs. Understanding the magnitude of these biases is important for determining the optimal treatment strategies for specific patient populations, guiding future research, and highlighting the need for more rigorous clinical trials.

Methods

We searched MEDLINE for studies by using keywords intracranial aneurysm, embolization, therapeutic, surgery, and treatment outcome. We reviewed bibliographies for citations until no further citations were found. Included studies fulfilled several criteria: publication between January 1990 and July 2003, UIA treatment with coiling or clipping, UIA treatment outcome data, separate reporting of ruptured intracranial aneurysms and UIA treatment outcomes, >10 patients, and publication in English. Two reviewers (T.L., M.B.) independently extracted data, and an adjudicator (J.P.-S.) resolved discrepancies. Data were separated by study design, baseline characteristics, and clinical outcome. We recorded mean age, time interval between procedure and outcome assessment by specialty, patient number, definition of morbidity, AOR, and procedural mortality. Studies were classified as single-center, multicenter, or community-based. We defined AOR as combined all-cause early or in-hospital morbidity and mortality. We recalculated AORs by using data from the modified Rankin scale, Glasgow Outcome Scale, or patient outcome data. Authors’ AOR criteria were used if data recalculation was not possible. We used the time interval for follow-up closest to 3 months. Data were recalculated to include patients with UIAs only. To evaluate bias, risks of clipping and coiling were tabulated by plotting precision measure (sample size) on the Y axis and trial-specific effect (AOR) on the X axis. Linear regression of AOR with time was performed by using the median study date.

Results

Coiling of UIAs

Nineteen coiling studies met selection criteria. From a total of 1925 patients, 4 multicenter or community-based studies described treatment of 1019 patients (2, 3). Two publications focused on basilar bifurcation aneurysms exclusively (4, 5). Baseline characteristics and AOR data appear in Tables 1 and 2.

TABLE 1:

Baseline study design and aneurysm data

	Clipping	Coiling
Studies
Retrospective single center	21	10
Prospective single center	1	5
Multicenter or community-based	8	4
Total	30	19
Patient baseline information
Mean age (y)	57.0	53.1
Aneurysm size	UIA No.*	%	UIA No.‡	%	UIA No.	%
Small	1399	35.7	297	36.1	1696	35.8
Medium	1394	35.6	236	28.7	1630	34.4
Large	898	22.9	219	26.6	1117	23.6
Giant	224	5.7	70	8.5	294	6.2
Total	3915		822		4737
Aneurysm site	UIA No.†	%	UIA No.§	%	UIA No.	%
Cavernous carotid artery	102	2.4	99	10.3	201	3.8
ICA and Ophthalmic	1542	36.1	269	27.9	1811	34.6
ACom, ACA, and pericollosal	676	15.8	173	18.0	849	16.2
MCA	1340	31.3	117	12.1	1457	27.8
PCom	183	4.3	71	7.4	245	4.9
Vertebrobasilar system	239	5.6	81	8.4	320	6.1
Basilar tip	194	4.5	153	15.9	347	6.6
Total	4276		963		5239

Note.—ICA indicates internal carotid artery; ACom, anterior communicating artery; ACA, anterior cerebral artery; PCom, posteror communicating artery. UIA nos. were based on the following references:

^*, 11, 14, 18–24;

^†, 3, 11, 14, 18–21, 23, 24, 25–27;

^‡, 3, 5, 7, 28–31;

^§, 3, 7, 28, 32.

TABLE 2:

Coiling data

Study No.	Reference	Study Type	Mean Age	Patient No.	Definition of Morbidity	AOR (%)	AOE (No.)	95% CI (%)	Mortality (%)
1	Cognard et al, 1998 (33)	PS	46	58	GOS ≤ 3	1.7	1	0.7–4.4	1.7
2	Collice et al, 1998 (8)	RS	57	10	Severe disability/death	30.0	3	4.7–17.9	0.0
3	Debrun et al, 1998 (32)	RS	49	89	GOS ≤ 3	10.1	9	7.4–13.7	7.9
4	Eskridge and Song, 1998 (4)	PM	52	67	Presented morbidity/mortality data	14.0	9	9.8–18.1	12.0
5	Keuther et al, 1998 (34)	RS	Not reported	41	Death/dependent status	19.5	8	14.1–26.4	9.8
6	Leber et al, 1998 (35)	RS	53	45	Morbidity/mortality data	11.5	6	9.0–19.2	Not reported
7	Bavinzski et al, 1999 (5)	RS	47	11	Neurologic deficit, unable to work, needed assistance	18.0	2	9.4–32.7	9.0
8	Johnston et al, 1999 (36)	RM	55	255	In-hospital mortality, D/C to nursing home/rehabilitation hospital	10.6	27	8.8–12.7	0.4
9	Lot et al, 1999 (37)	PS	47	83	Moderate to severe deficit, death	4.8	4	3.0–7.7	Not reported
10	Murayama et al, 1999 (28)	RS	51	115	Morbidity/mortality data	8.3	10	6.5–11.7	Not reported
11	Johnston, 2000 (38)	RS	57	62	≥2 change in mRS	8.1	5	5.3–12.2	1.6
12	Johnston et al, 2001 (2)	RM	56	287	In hospital death or D/C to nursing home/rehabilitation hospital	9.7	28	8.2–11.6	0.5
13	Qureshi et al, 2001 (6)	RS	Not reported	80	Author report	2.0	2	1.3–4.9	0.0
14	Roy et al, 2001 (30)	PS	51	116	mRS ≥3	0.9	1	0.3–2.2	0.0
15	Goddard et al, 2002 (29)	RS	56	62	Moderate to severe deficit, death	5.5	3	2.8–8.3	0.0
			median
16	Ng et al, 2002 (39)	RS	Not reported	58	GOS ≤3	1.7	1	0.7–4.4	1.7
17	Wanke et al, 2002 (31)	PS	48	39	GOS not “good recovery”	4.8	2	2.6–10.0	0.0
18	Raftopoulos et al, 2003 (7)	PS	49	38	mRS ≥3	0.0	0	0.0–2.5	0.0
19	Wiebers et al, 2003 (3)	PM	54	451	mRS change of ≥3 and/or impaired cognition	9.3	38	8.0–10.8	2.0

Note.—RS indicates retrospective single center; RM, retrospective multicenter; PM, prospective multicenter; AOR, adverse outcome rate; AOE, patient adverse outcome event.

More than half of the studies addressed only UIAs, and all reported AORs or data allowing for recalculation of AORs. Classification of AOR varied widely, and some studies omitted baseline information for age, aneurysm size, and site distribution. One study presented total morbidity without defining outcome criteria (6). Follow-up time interval spanned from the immediate postoperative period to 16.3 months. AORs varied from 0.0% (7) to 30.0% (8). Mortality varied from 0.0% to 12.0%. Peak procedural mortality was 9.7%. In most studies, procedural mortality was reported as 0.0%. Figure 1 shows coiling AORs versus sample size. The cumulative coiling AOR is 8.8% (95% confidence interval [CI] 7.6%–10.1%). Coiling AOR for single-center retrospective studies is 8.1% (95% CI 6.2%–10.4%). Coiling AORs have declined since coils were introduced in the early 1990s (Fig 2). This trend occurs consistently when AORs are plotted against median study date.

Fig 1.

Coiling AOR scattergram showing 19 prospective and retrospective studies. The average AOR was 8.8% (95% CI 7.6%–10.1%). Numbers correlate with study number given in Table 2.

Fig 2.

Coiling AORs versus time by median study date. Coiling, a relatively new procedure, demonstrates decreasing AORs between 1993 and 2000.

Clipping of UIAs

Thirty clipping studies met selection criteria. Meta-analyses were excluded (9, 10). Two continued follow-up with the same patient cohort (3, 11). Eight multicenter or community-based studies included 9579 of 11,363 patients. Twenty-one were retrospective, single-center studies, and only one was a prospective, single-center study. One study reported only giant intracranial aneurysms (12) whereas another discussed anterior circulation aneurysms exclusively (13). Fourteen studies presented UIA size or site distribution characteristics independently (Table 3), and all presented AORs or allowed for recalculation of AOR data. Clipping AORs varied from 0.0% (14) to 25.1% (2). Plotting clipping AORs against median study date shows increasing morbidity and mortality in the 1990s (Fig 3). This trend is largely due to recent publication of multicenter and community-based data.

TABLE 3:

Clipping data

Study No.	Reference	Study Type	Mean Age	Patient No.	Definition of Morbidity	AOR (%)	AOE (No.)	95% CI (%)	Mortality (%)
1	Charbel et al, 1991 (40)	RS	43	22	Persistent deficits	18.2	4	11.4–22.7	0.0
2	Awad and Little, 1991 (41)	RS	47	10	Requires occasional assistance	10.0	1	3.9–23.3	0.0
			Median
3	Rabinowicz et al 1991 (42)	RS	Not reported	21	Permanent perioperative complications	19.1	4	12.0–28.9	0.0
4	Taylor et al, 1991 (43)	RS	Not reported	21	Moderate disability to vegetative state	0.0	0	0.0–4.5	0.0
5	Chehrazi, 1992 (13)	RS	52	18	GOS moderate deficit to PVS/death	5.6	1	2.2–13.6	0.0
6	Symon, 1992 (12)	RS	Not reported	34	Author report	8.8	3	5.1–14.9	5.9
7	Inagawa et al, 1992 (18)	RS	Not reported	52	Author report	5.8	3	3.3–9.9	Not reported
8	Nakagawa and Hashi, 1994 (20)	RS	55	20	Author report	5.0	1	1.9–12.3	0.0
9	Dickey et al, 1994 (44)	RS	57	44	Author report	4.6	2	2.3–8.8	0.0
			Median
10	Solomon et al, 1994 (21)	RS	Not reported	202	New neurologic deficits	12.4	25	10.3–14.9	6.9
11	Asari and Ohmoto, 1993 (45)	RS	Not reported	69	Worsened after surgery	7.3	5	4.7–11.0	0.0
12	Mizoi et al, 1995 (14)	RS	57	90	Cannot perform normal activities	0.0	0	0.0–1.1	0.0
13	Dix et al, 1995 (46)	RS	Not reported	66	Cannot perform former activities	3.0	2	1.5–5.9	Not reported
14	Deruty et al, 1996 (25)	RS	Not reported	62	Author report	6.5	4	4.0–10.3	Not reported
15	Khanna et al, 1996 (47)	RS	Not reported	172	Inability to work, no self-care	9.9	17	7.8–12.4	Not reported
16	Leber et al, 1998 (35)	RS	Not reported	16	GOS ≤3	6.3	1	2.4–15.2	6.3
17	Lot et al, 1999 (37)	RS	Not reported	25	Moderate to severe deficit, death	4.8	3	6.9–20.0	0.0
18	Orz et al, 2000 (24)	RM	62	310	Dependent life	5.8	18	4.6–7.3	0.3
19	Johnston et al, 2000 (22)	RS	53	68	mRS change ≥2	25.0	17	20.2–30.6	1.5
20	Chyatte and Porterfield, 2001 (27)	RS	55	366	mRS change ≥2	6.0	22	4.9–7.4	3.8
21	Kashiwagi et al, 2000 (23)	RS	73	96	Dependent/dead	8.3	8	5.9–11.6	5.2
22	Raaymakers, 2000 (26)	PM	44	18	mRS change of ≥2	22.2	4	14.1–33.3	0.0
23	Rinne et al, 1996 (48)	CB	49	33	GOS moderate disability/death	12.1	4	7.6–18.9	0.0
24	Yamashita et al, 1997 (49)	CB	Not reported	349	GOS moderate disability/death	23.8	83	21.6–26.1	Not reported
25	Wiebers et al, 2003 (3)	PM	Not reported	995	mRS ≥3	15.2	151	14.1–16.4	1.8
26	Johnston et al, 1999 (36)	RM	52	2358	D/C to nursing home/rehabilitation hospital	18.5	436	17.7–19.3	2.3
27	Johnston et al, 2001 (2)	CB	54	1699	D/C to nursing home/rehabilitation hospital	25.1	426	24.0–26.1	3.5
28	Barker et al, 2003 (50)	RC	64	3498	Death/discharge to other than home	20.4	714	19.7–21.1	2.1
29	Raftopoulos et al, 2003 (7)	PS	49	33	mRS ≥3	3.0	1	1.2–7.6	0.0
30	Wiebers et al, 2003 (3)	PM	52	1917	mRS 3–5 and/or impaired cognition	13.2	254	12.9–14.6	Not reported

Note.—RS indicates retrospective single center; RM, retrospective multicenter; PM, prospective multicenter; CB, community-based; RC, retrospective cohort; AOR, adverse outcome rate; AOE, adverse outcome event; PVS, persistent vegetative state; GOS, Glasgow Outcome Scale; mRS, modified Rankin Scale.

Fig 3.

Clipping AORs versus time by median study date. Clipping AORs increased between 1984 and 1999 after publication of larger, prospective multicenter studies.

In 11,363 subjects, the cumulative clipping AOR is 17.8% (95% CI 17.2%–18.6%). For prospective, multicenter, and community-based studies, the cumulative AOR is 19.7% (95% CI 18.9%–20.5%), including only prospective ISUIA data (3). The cumulative AOR in retrospective studies is 7.9% (95% CI 6.7%–9.3%). The clipping AOR scattergram is shown in Figure 4.

Fig 4.

Clipping AOR scattergram showing 30 prospective and retrospective studies. The average AOR was 17.8% (95% CI 17.2%–18.6%). Numbers correlate with study number given in Table 3.

Discussion

The optimal treatment strategy for UIAs is currently unknown, because of the absence of large, randomized clinical trials. Although a clinical trial for ruptured intracranial aneurysms has been completed in the form of the International Subarachnoid Aneurysm Trial, (15) and prospective comparative data for unruptured aneurysms based on a standardized approach to patient entry and outcomes is available from the ISUIA, (3) current recommendations for management of UIAs still depend on data from heterogeneous series. AORs vary widely, and lack of comparability between studies hinders accurate aggregate impressions of the literature. Previous reports have discussed publication bias in surgical clipping studies (16). In Figs 1 and 4, we used a simple scattergram technique, adapted from previously published methods (1), to demonstrate the magnitude by which literature biases have influenced our outlook on UIAs. This technique estimates bias by plotting a measure of precision, such as the number of subjects, against a trial specific effect, such as AOR. In the absence of bias, the plot resembles an inverted symmetrical funnel, with small studies scattering at the bottom of the plot and more precise studies clustering around the true treatment effect. The presence of bias is suggested by visually apparent asymmetry (1).

Our results demonstrated significant asymmetry in the scattergram of clipping AORs, attributable to a substantial difference in cumulative AORs between the retrospective series (7.9%; 95% CI 6.7%–9.3%) and the remaining prospective, multicenter, and community-based studies, inclusive of the ISUIA (19.7%; 95% CI 18.9%–20.5%). The causes underlying this discrepancy are difficult to determine, but it seems likely that they would include publication bias—ie, a tendency to publish studies with positive outcomes—and true heterogeneity stemming from differences in patient population, such as age, aneurysm size, aneurysm site, and predisposing social and genetic factors, or from differences in the definitions of measured outcomes (17).

Other possibilities that may explain the large discrepancy seen in the retrospective series include differences in practitioner experience, procedure difficulty, and patient selection. Arguments that this may be the case, however, are difficult to substantiate by examining the data from the various studies. With respect to practitioner experience and procedure difficulty, one might surmise that the preponderance of lower AORs is attributable to centers with neurosurgeons particularly adept at clipping aneurysms or due to selection of patient subpopulations less likely to have significant morbidity and mortality; however, because these studies did not have homogeneous definitions of patient outcomes or selection criteria a priori, data supporting this reasoning is incomplete. Nevertheless, it is interesting to note that few authors publishing the results of the smaller of these series later report their clinical experience as the number of reportable cases increases, which raises the question of whether such reports may have been censored because of publication bias. Also of note, several studies reported outcomes for groups having subpopulations with prior SAH—a practice that would tend to increase overall AOR rather than decrease it.

The scattergram of coiling AORs appears significantly less asymmetric. Although this does not exclude the presence of publication bias or heterogeneity in the published literature for the coiling of UIAs, it does appear that the cumulative AORs for coiling cluster around a central tendency, making it more likely that they approximate a single treatment effect (1). This appears reasonable, in light of the fact that populations with UIAs may be more homogeneous than a randomly selected group from the general population in that they are often asymptomatic, healthy, middle-aged individuals with relatively few confounding comorbidities. Even among groups with UIAs these individuals are more likely to have similarities among themselves in that they are often selected for coiling after screening for surgical clipping candidates has taken place and tend to include the subset of the population likely to have increased periprocedural complication rates (3). With respect to end points, some variation in the definition of adverse outcomes may have been offset by the fact that the adverse outcomes being measured are often severe, such as severe neurologic deficit or death, and would have been recorded regardless of what definition was being used.

As suggested above, the presence of heterogeneity or bias cannot always be detected on the basis of asymmetry in the scattergram. As seen Fig 3, there appears to be a positive correlation between AOR and median publication date for clipping. This result is not unexpected, in view of the later publication dates of the prospective and multicenter or community-based studies. What is being demonstrated here is likely the same effects that skew the scattergram seen in Fig 4. In contrast, there appears to be a negative correlation between AOR and median publication date for coiling. This trend reflects the tendency for earlier studies to report higher AORs, which may reasonably be attributed to improvements in technique, refinement of practitioner skill, or better selection of treatment candidates. As we saw in the discussion of the clipping data, however, determining the cause of this trend proves problematic, again because of difficulty in standardizing starting points and outcomes. This highlights the distinct possibility that other undetected and unanticipated biases may be underlying the current data for both coiling and clipping, bringing out another limitation in aggregate data in comparison to data from well-controlled, randomized prospective studies.

Comparison between coiling and clipping results by using data from a review of this type is problematic. As evidenced by recent data from the ISUIA, adverse outcome rates vary widely due to patient age, aneurysm characteristics, and site distribution (3). Variability in the definitions and assessment of outcomes and the reporting of baseline aneurysm and patient characteristics in retrospective series makes analysis of the nature of the differences in reported AORs difficult, because the cumulative data are often incomplete. Furthermore, the degree to which differences such as these affect reported values in individual series is also difficult to assess. Prospective data now available from the ISUIA offer the first insight into the relationship baseline characteristics and clinical end points by offering a standardized prospective assessment of the procedures with uniform definitions of starting points, aneurysmal characteristics, end points, and follow-up of patients. Although the endovascular and surgical groups are not comparable, the ISUIA data do make it possible to outline the magnitude of the differences and adjust for them statistically.

Aggregate analysis as performed with the scattergram techniques demonstrated here is unable to correct for biases that may be present in individual studies, but it does allow us to see the cumulative effect of these biases as they influence the body of literature as a whole. Our data clearly demonstrate that there is a significant process at work skewing our impression of adverse outcomes associated with surgical clipping. It is important to note that the number of subjects in many of the retrospective series on surgical clipping is not insignificant. Thus, the number of patients in such series cannot necessarily be relied on as a measure of reliability of the result without closer scrutiny of the underlying methodology. As suggested by the prior discussion regarding the variability in outcomes with respect to patient and aneurysm characteristics, any estimate of outcomes by using aggregate data must be treated with caution. Indeed, even though cumulative aggregate AOR is appreciably higher than that reported by the ISUIA, it is still possible that these results are consistent, especially considering the older cohort in the aggregate data.

Conclusion

Efforts to distinguish and identify forms of bias, such as publication bias, have increased in the past 2 decades. In this scattergram analysis, we visually illustrate factors that implicate biases in the UIA literature. This aggregate analysis indicates that bias may influence reported clipping AORs to a larger extent than it does in coiling studies. Although we obtained corrected AOR estimates for both treatments, quantifying factors to account for the wide AOR variability was difficult. Ultimately, our results highlight the need for a well-designed prospective, randomized trial to avoid these biases and compare both types of treatment more accurately.