Skip to main content
NCBI home page
JAMA Network logoLink to JAMA Network
. 2023 Sep 1;6(9):e2331798. doi: 10.1001/jamanetworkopen.2023.31798

Outcomes Associated With Intracranial Aneurysm Treatments Reported as Safe, Effective, or Durable

A Systematic Review and Meta-Analysis

Victor Volovici 1,✉, Iris S Verploegh 1, Djaina Satoer 1, Noëlle J M C Vrancken Peeters 1, Yasmin Sadigh 1, Mervyn D I Vergouwen 3, Joost W Schouten 1, Gavin Bruggeman 1, Dana Pisica 1,2, Gizem Yildirim 1, Ayca Cozar 1, Femke Muller 1, Ana-Maria Zidaru 1, Kelsey Gori 1, Nefeli Tzourmpaki 1, Esther Schnell 1, Mbaye Thioub 4, Kimberly Kicielinski 5, Pieter-Jan van Doormaal 6, Nikolay Velinov 7, Mahjouba Boutarbouch 8, Michael T Lawton 9, Giuseppe Lanzino 10, Sepideh Amin-Hanjani 11, Ruben Dammers 1, Torstein R Meling 12
PMCID: PMC10474558  PMID: 37656458

Key Points

Question

Is there a difference in disease-specific outcomes between studies on intracranial aneurysm (IA) treatment with a positive, uncertain, or negative conclusion regarding safety, effectiveness, or durability, and how are safety, effectiveness, and durability defined?

Findings

In this systematic review and meta-analysis of 1356 studies with 410 993 patients, disease-specific outcomes did not differ between studies with a positive, uncertain, or negative conclusion regarding safety, effectiveness, or durability. Less than 2% of studies reported a definition of safety, effectiveness, or durability.

Meaning

Studies investigating the safety, effectiveness, and durability of IA treatments should follow the highest methodological rigor; the findings of this study suggest that best methodological practices should be improved field-wide.

This systematic review and meta-analysis examines studies investigating devices and procedures used in intracranial aneurysm treatment for methods and completeness of reporting and compares the results of studies with positive, uncertain, and negative conclusions.

Abstract

Importance

Testing new medical devices or procedures in terms of safety, effectiveness, and durability should follow the strictest methodological rigor before implementation.

Objectives

To review and analyze studies investigating devices and procedures used in intracranial aneurysm (IA) treatment for methods and completeness of reporting and to compare the results of studies with positive, uncertain, and negative conclusions.

Data Sources

Embase, MEDLINE, Web of Science, and The Cochrane Central Register of Clinical Trials were searched for studies on IA treatment published between January 1, 1995, and the October 1, 2022. Grey literature was retrieved from Google Scholar.

Study Selection

All studies making any kind of claims of safety, effectiveness, or durability in the field of IA treatment were included.

Data Extraction and Synthesis

Using a predefined data dictionary and analysis plan, variables ranging from patient and aneurysm characteristics to the results of treatment were extracted, as were details pertaining to study methods and completeness of reporting. Extraction was performed by 10 independent reviewers. A blinded academic neuro-linguist without involvement in IA research evaluated the conclusion of each study as either positive, uncertain, or negative. The study followed Preferring Reporting Items for Systematic Reviews and Meta-Analyses guidelines.

Main Outcomes and Measures

The incidence of domain-specific outcomes between studies with positive, uncertain, or negative conclusions regarding safety, effectiveness, or durability were compared. The number of studies that provided a definition of safety, effectiveness, or durability and the incidence of incomplete reporting of domain-specific outcomes were evaluated.

Results

Overall, 12 954 studies were screened, and 1356 studies were included, comprising a total of 410 993 treated patients. There was no difference in the proportion of patients with poor outcome or in-hospital mortality between studies claiming a technique was safe, uncertain, or not safe. Similarly, there was no difference in the proportion of IAs completely occluded at last follow-up between studies claiming a technique was effective, uncertain, or noneffective. Less than 2% of studies provided any definition of safety, effectiveness, or durability, and only 1 of the 1356 studies provided a threshold under which the technique would be considered unsafe. Incomplete reporting was found in 546 reports (40%).

Conclusions and Relevance

In this systematic review and meta-analysis of IA treatment literature, studies claiming safety, effectiveness, or durability of IA treatment had methodological flaws and incomplete reporting of relevant outcomes supporting these claims.

Introduction

In recent decades, the practice of medicine has witnessed an unprecedented rise in the number of medical devices being developed for various specialties. The history of medicine, however, is laden with examples in which the mechanistic logic1 that underpins the development of a device does not translate to better outcomes for patients and might, instead, be harmful. A MEDLINE search reveals that “safe and effective” is a mantra used by many reports, with an excess of 150 000 hits boasting these claims in the abstract alone. Even if devices are granted authorization, the evidence may often be of questionable quality or even lacking.2,3,4

Aneurysmal subarachnoid hemorrhage (aSAH) is a severe type of hemorrhagic stroke caused by the rupture of an intracranial aneurysm (IA) that affects 6.1 per 100 000 people per year,5 of whom more than 30% die within 3 months after ictus.6 A vast majority of survivors experience severe cognitive and functional disabilities and less than one-third returns to work.7,8 One of the mainstays of treatment is occlusion of the aneurysm, either in an unruptured state to prevent aSAH and increase life-years with good quality of life or after rupture to prevent rebleeding and increase the likelihood of a good outcome.7 Complete aneurysmal exclusion from circulation is the main objective of treatment, provided that the risk of treatment complications is lower than the risk of (re)rupture. In terms of endovascular treatment, numerous devices and modifications have been developed and tested in the past 2 decades, several of which have gained US Food and Drug Administration (FDA) approval. This increase of treatment modalities has led to an exponential growth of the number of scientific reports evaluating the safety, effectiveness, and durability of these techniques.

Given the ongoing discussion within the stroke community regarding the best treatment of IAs and proper patient selection, and the paucity of substantial, properly generalizable randomized evidence, we aimed to critically evaluate published claims of safety, effectiveness, and durability of IA treatments. Unsubstantiated claims of safety and effectiveness in biomedical science constitute a potential public health hazard and may influence health policy negatively, leading to suboptimal treatments for patients. We hypothesized that there was a difference in the incidence of disease-specific outcomes between studies with a positive, uncertain, and negative conclusion regarding claims of safety, effectiveness, and durability.

Methods

The study protocol was registered with PROSPERO, registration number CRD42020169592. The analysis plan was predefined and updated twice (eAppendix 8 in Supplement 1). The study followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) reporting guidelines in all aspects except data synthesis, as our objective was not to pool data but to compare the distributions of the reported outcomes.

Literature Search

We defined the search strategy with the help of a medical information specialist. Using a thrice-refined search algorithm (eAppendix 1 in Supplement 1), studies were retrieved from Embase, Ovid MEDLINE, Web of Science, and The Cochrane Central Register of Clinical Trials. Grey literature was retrieved from Google Scholar. The search was conducted for all records published between January 1, 1995, and the October 1, 2022.

Inclusion and Exclusion Criteria, Selection Process, and Data Extraction

Studies included had to contain more than 25 patients, as this constitutes in the authors’ opinion a minimum case series, and had to deal with the endovascular or microsurgical treatment of IAs by any device, means, or technique, either individually or combined. We excluded (1) articles that only described a minor modification of an already existing approach or technique, such as a surgical approach; (2) studies dealing with mycotic, traumatic, or purely dissecting aneurysms; and (3) articles solely dealing with aneurysms that recur after treatment. All articles had to mention either safety, effectiveness, or durability in one form or another in the conclusion.

We enrolled a team of 10 master’s degree students with specific interest in interventional neuroradiology, neurology, or neurosurgery and provided formal systematic review training and methods of study design for safety, effectiveness, and durability claims, upon which they were first split into 3 teams and later into 5 teams for independent sifting of studies, extraction of data, and assessment of the outcome parallel to the academic linguist. The process is detailed in eAppendix 8 in Supplement 1.

A professional, academic neuro-linguist, blinded for the type of study (endovascular or microsurgical), with expertise in regulatory assessment of devices and medical products but no extensive knowledge of the field of IA or any involvement in IA research (D.S.), assessed the wording of the conclusions of the included studies. A parallel assessment was conducted by the MSc students who extracted the data. Lexical fields were built for the 3 domains safe, effective, and durable. Every study or study group was labeled positive, uncertain, or negative for each of the 3 domains.

Definition of the Outcome

A positive conclusion meant the technique was described by the investigators to be safe, effective, or durable in the conclusion without any other comments; uncertain meant that wording was chosen to indicate that a technique might be safe, effective, or durable but only under certain circumstances or with potential caveats; and negative meant that a technique was described as being unsafe, ineffective, or not durable. The neuro-linguist also assessed whether the definition in the abstract was more positive, more negative, or the same as the conclusion in the text. A random sample of 10% of the conclusions were discussed with the first author (V.V.). For studies dealing with unruptured aneurysms, an extra assessment was performed by the second or third author together with the first author. We calculated the concordance rate of these assessments.

Variables and Data Extraction

For every study and study group (when different therapeutic modalities were compared within one study), we extracted data regarding the context and type of patients included (number of patients and aneurysms, number of centers, aneurysm location, rupture status of aneurysms). We defined (eAppendix 8 in Supplement 1) and extracted domain-specific outcomes. Safety outcomes were in-hospital mortality; poor functional outcome at discharge, defined as either modified Rankin Scale (mRS) 3 to 6 or Glasgow Outcome Score (GOS) 1 to 3; incidence of total complications; and incidence of thromboembolic complications, which were defined as complications leading to ischemia or requiring treatment. For effectiveness, we extracted the rate of complete and adequate occlusion, classified according to the Raymond-Roy Occlusion Classification Scale (RROCS), at discharge and final follow-up.9 Any other variation of adequate occlusion, ie, near complete, near total, almost near total, as well as RROCS grade 2 or equivalent were considered adequate. The durability outcome was the proportion of aneurysms with complete occlusion at last follow-up. The follow-up time and proportion of patients available for follow-up were also extracted. Definitions of safety, effectiveness, and durability were also extracted. We recorded whether our predefined domain-specific outcomes were reported at all.

During data extraction, we noticed that some of the studies used various strategies to increase or decrease reported percentages. For example, some reported the proportion of patients with good functional outcome of all living patients at discharge (instead of all included patients). Sometimes the numbers in the various tables did not add up but no explanation was given in the text for the discrepancies. Finally, in some cases, established clinical scales, such as the RROCS, were adjusted to give the appearance of better results. In these cases, we flagged the study under improper reporting.

Statistical Analysis

The rationale and approved analyses are detailed in eAppendix 8 in Supplement 1. In the main analysis, we compared the proportion of domain-specific outcomes of safety, effectiveness, or durability between studies with a positive, uncertain, or negative conclusion using the Kruskal-Wallis test or the Mann-Whitney U test when 1 group was small and only 2 groups were compared. When the Kruskal-Wallis test indicated a statistically significant result, we used Dunn post hoc test to pinpoint which comparisons were statistically significant (positive vs uncertain, uncertain vs negative, positive vs negative). For the rates of complete aneurysmal occlusion at follow-up, we tested whether the 3 groups (positive, uncertain, and negative conclusions) had comparable follow-up periods. When this was not the case, we performed sensitivity analyses by selecting a subset of studies with a follow-up period longer than 9 months, which maximized the number of studies included but kept the distribution of follow-up time between groups comparable.

Second, we summarized the published definitions of safe, effective, and durable together with the characteristics of studies using them. Third, we determined the proportion of studies not reporting our predefined domain-specific outcomes. Fourth, we studied which variables were associated with a higher likelihood of claiming safety, effectiveness, and durability using logistic regression models. Despite the initial analysis plan in terms of the regression analysis, when examining the data, we decided that a simple logistic regression model would be sufficient and provide the best interpretability.

Finally, we performed a sensitivity analysis by reporting the results of intervention groups involving unruptured aneurysms only. These conclusions were independently assessed by 2 of us (I.S.V. or Y.S.) as well as the first author (V.V.). We also extracted for this group whether independent neurological or radiological assessment of patients was performed and the incidence of major complications (stroke, ischemic or hemorrhagic). Statistical significance was set at α = .05, and tests were 2-tailed. R version 4.2.1 (R Project for Statistical Computing) was used for all analyses.

Results

We included 1356 studies comprising 410 993 patients, of whom 261 675 underwent endovascular aneurysm treatment and 149 300 microsurgical aneurysm treatment (Figure 1 and eAppendix 9 in Supplement 1). Of the included studies, 1055 were single-group studies with 129 882 patients, and 301 were multigroup studies with 281 111 patients. Of the multigroup studies, 287 were 2-group studies, 11 were 3-group studies, and 3 were 4-group studies.

Figure 1. Flowchart of the Included Studies.

Figure 1.

Open in a new tab

AVM indicates arteriovenous malformations.

Baseline Characteristics

Of all 1674 intervention groups included in this analysis, 1330 (79%) were endovascular interventions, 343 (20%) were microsurgical interventions, and 1 was a combined endovascular and microsurgical approach. Of the endovascular groups, in 554 (42%) the intervention was simple coiling, in 342 (26%) the intervention was stent- or balloon-assisted coiling, in 266 (16%) the intervention was a flow diverter, and in 61 (3%) a WovenEndoBridge device or another intrasaccular device. The rest of the endovascular interventions were classified as other (mainly combined endovascular interventions). Microsurgical interventions included different clip reconstruction modalities, with or without bypass flow replacement.

The 1674 intervention group included a median (IQR) of 63 (37-126) patients harboring 67 (38-130) aneurysms per group. Most studies were carried out in the United States (311 studies [23%]), China (199 studies [15%]), and South Korea (109 studies [8%]). The comparative studies were carried out mostly in the United States (81 studies [26%]), Northwestern Europe (52 studies [16%]), and China (47 studies [15%]). The single-group studies included 172 (16%) prospective observational studies, with the rest being retrospective observational ones. The comparative studies included 35 (12%) prospective observational studies and 9 (6%) randomized clinical trials (RCTs). Three reports included RCT subgroup analyses; 2 reports, periprocedural outcomes; 2 reports, long-term follow-up; and 1 report, safety assessments.

The concordance rate between the linguist and the other extractors was 87%, which was considered very good. In the sensitivity analysis for unruptured aneurysms only the concordance rate between the linguist and the clinicians was 92% (eAppendix 4 in Supplement 1 and eTable in Supplement 2).

Main Analysis: Safety

Of the 1060 intervention groups mentioning safety in the conclusion, 821 (77%) were deemed safe, 235 (22%) were categorized as uncertain, and 4 (0.4%) were categorized as not safe. There was no difference in the incidence of mortality at discharge between studies with a positive, uncertain, or negative conclusion (median [range]: positive, 0.7% [0%-50.9%]; uncertain, 1.6% [0%-34.4%]; negative, 2.2% [0%-18.0%], P = .46) (Table 1 and eAppendix 7 in Supplement 1). Similarly, there was no difference in poor functional outcome at discharge between studies with a positive, uncertain, or negative conclusion (Table 1).

Table 1. Main Analysis.

Outcome Median (IQR) [range] P value
Positive Uncertain Negative
Safety
Intervention groups, No. 821 235 4 NA
Total complications, % 10.5 (6.0-16.0) [0-100] 13.5 (10-19.6) [0-78.1] 21.8 (15.6-33.4) [9.3-45] <.001
Poor outcome at discharge, % 7.2 (2.4-17.0) [0-86.7] 9.1 (2.5-26.8) [0-100] 1.8a .13
Thromboembolic complication, % 4.8 (2.4-8.1) [0-74.1] 6.0 (3.2-9.1) [0-35.9] 17.5a .02
In-hospital mortality, % 0.7 (0.0-3.0) [0-50.9] 1.6 (0-4.6) [0-34.4] 2.2 (1.4-6.5) [0.35-18] .46
Effectiveness
Intervention groups 826 195 4 NA
Complete occlusion at discharge, % 61.4 (41.8-80.9) [0.0-100] 62.0 (42.8-76.9) [0.5-100] 51.8 (49.4-54.3) [46.9-56.7] .82
Complete occlusion at final FU, % 78.8 (68.4-88.0) [3.0-100] 75.0 (61.8-87.2) [3.5-98.8] 61.9 (59.9-63.8) [58.0-65.7] .08
Adequate occlusion at discharge, % 23.1 (13.4-35.8) [0.0-96.2] 24.0 (12.5-34.0) [0.0-94.0] 37.7 (37.6-37.7) [37.5-37.8] .32
Adequate occlusion at final FU, % 13.6 (7.6-25.0) [0-89.1] 11.3 (6.9-22.6) [0-69.0] 23a .59
Durability
Intervention groups, No. 94 64 6 NA
Complete occlusion at final FU, % 83.1 (72.7-90.1) [42.0-100] 76.9 (64.7-89.9) [35-100] 70.3 (64.2-72.3) [58-74.2] .09
FU time, mo 21.3 (12.0-38.8) [5.9-132] 12.4 (9.2-23.9) [6.0-101] 20.5 (15.2-25.5) [10-30] .04
Open in a new tab

Abbreviations: FU, follow-up; NA, not applicable.

a

Only 1 study was considered, so there is no IQR or range.

A definition of safety was available in 27 reports (2%) in the methods section (Table 2).10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36 Most (25 of 27 [93%]) detailed the outcomes used to claim safety and the follow-up time but not an incidence threshold used to support the safety claim. Of the 27 reports, 5 (19%) included safety definitions that were available in a published protocol. Two studies (7%) included an incidence threshold, under which the intervention would not be considered safe, either under 15% or under 25% of included patients after a predefined follow-up. The outcome to which the threshold applied was defined as “procedural complications, mortality, or morbidity,”16,36 but it was unclear which of the 3 should demonstrate an incidence below the threshold for safety to be claimed.

Table 2. Definitions of Safety Available in 27 Studies .

Definitions Study type Devices Rupture status Predefined in protocol Follow-up moments (No. studies) Incidence of outcomes deemed safe, (No. studies)
Rate of procedural complications, mortality, and morbidity (7 studies)10,11,12,13,14,15,16 3 RO, 3 PO, 1 RCT 3 coil, 1 WEB, 1 FD, 1 clip vs coil, 1 SAC vs SAC 1 aSAH, 6 both 0 At discharge (1), 12 mo (2), 3-6 mo (1) <25% (1)
Neurological death or major (ipsilateral) stroke (6 studies)17,18,19,20,21,22 1 RO, 5 PO 2 FD, 4 SAC 2 elective, 4 both 2 studies 6 mo (1), 12 mo (4) <15% (1)
Neurological death or major (ipsilateral) stroke or deterioration of the clinical condition (mRS 3-6, NIHSS 4 points) (14 studies)23,24,25,26,27,28,29,30,31,32,33,34,35,36 3 RO, 10 PO, 1 RCT 2 WEB, 6 FD, 3 SAC, 2 SAC vs coil, 1 coil vs coil 1 aSAH, 4 elective, 9 both 3 studies 1 mo (1), 3 mo (1), 6 mo (2), 12 mo (6) NA
Open in a new tab

Abbreviations: aSAH, aneurysmal subarachnoid hemorrhage; FD, flow diverter; mRS, modified Rankin Scale; NIHSS, National Institutes of Health Stroke Scale; PO, prospective observational; RCT, randomized clinical trial; RO, retrospective observational; SAC, stent-assisted coiling; WEB, WovenEndoBridge.

Main Analysis: Effectiveness

Of the 1025 intervention groups evaluated for effectiveness, 826 (81%) were deemed effective, 195 (19%) were categorized as uncertain, and 4 (0.4%) as not effective. There was no difference between the incidence of complete occlusion at discharge between studies with positive, uncertain, and negative conclusions (median [range]: positive, 61.4% [0%-100%]; uncertain, 62.0% [0.5%-100%]; negative, 51.8% [46.9%-56.7%]; P = .82). Similarly, there was no difference between the incidence of complete occlusion at final follow-up between studies with positive, uncertain, and negative conclusions (median [range]: positive, 78.8% [3.0%-100%]; uncertain, 75.0% [3.5%-98.8%]; negative, 61.9% [58.0%-65.7%]; P = .09). Results were similar when we restricted follow-up time between 9 months and 5 years, thereby keeping groups comparable in terms of follow-up period (eAppendix 2 in Supplement 1).

A definition of effectiveness was available in 2 studies (0.1%) (eAppendix 6 in Supplement 1). There were no thresholds available under which an intervention would be considered ineffective. Twenty-five other reports included definition of effectiveness at the aneurysm level, meaning when an aneurysm would be considered effectively treated, but these were excluded as we were only interested in group-level definitions.

Main Analysis: Durability

Of the 164 intervention groups evaluated for durability, 94 (58%) were considered durable, 64 (39%) were categorized as uncertain and 6 (2%) were not durable. The median number of aneurysms with complete occlusion at final follow-up did not differ between studies with positive, uncertain, or negative conclusions (median [range]: positive, 83.1% [42.0%-100%]; uncertain, 76.9% [35.0%-100%]; negative, 70.3% [58.0%-74.2%]; P = .10) (Table 1). The studies with positive conclusions had a higher median (range) proportion of patients available for follow-up (positive, 84.0% [15.4%-100%]; uncertain, 69.9% [33.0%-100%]; negative, 77.3% [73.5%-96.9%]; P = .02). Results were similar when we restricted follow-up time to 9 months to 5 years, thereby keeping groups comparable in terms of follow-up period (eAppendix 2 in Supplement 1). No studies reported a definition of durability.

Subgroup Analysis

When analyzing only studies including patients with unruptured aneurysms, there was a statistically significant difference in terms of median (range) in-hospital mortality between studies with a positive and those with an uncertain conclusion, but not between studies with a positive and those with a negative conclusion or between those with an uncertain conclusion and those with a negative conclusion (positive, 0% [0%-5.6%]; uncertain, 0.2% [0%-20.1%]; negative, 1.0% [0.4%-1.7%], P = .009). There was no difference in terms of median (range) poor outcome at discharge between studies with a positive, uncertain, or negative conclusion (positive, 2.0% [0%-27.5%], uncertain: 2.5% [0%-100%], negative: 1.8%, P = .79) (eAppendix 2 in Supplement 1). There were no differences in terms of effectiveness or durability outcomes between studies with a positive, negative, or uncertain conclusion (eAppendix 2 in Supplement 1).

The sensitivity analysis revealed that the incidence of major complications ranged between 0% and 28%. There was no difference between studies with a positive, negative, or uncertain conclusion (median [range]: positive, 3.8% [0%-21.0%]; uncertain, 7.2% [0%-21.5%]; negative, 24.1% [20.0%-28.0%]; P = .50) Of the intervention groups, 45 (21%) had an incidence rate of major complications greater than 10%. Thirty-two intervention groups (14%) had independent neurological assessment, and 16 (7%) had independent neurological assessment of patients (eAppendix 4 in Supplement 1).

Risk of Bias

Ten of the 17 RCT reports (59%) showed a high risk of bias (Figure 2).35,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52 We found improper reporting in 546 of the included studies (40%) (eAppendix 3 in Supplement 1). In 68 studies (5%), the conclusion in the abstract was more positive than the conclusion in the text.

Figure 2. Risk of Bias Analysis of Intracranial Aneurysm Treatment Randomized Clinical Trials and Related Publications .

Figure 2.

Open in a new tab

DSA indicates digital subtraction angiography; mRS, modified Rankin Scale.

Results of Regression Analyses

Regression analysis indicated that studies on stent-assisted coiling and with a lower incidence of total complications were associated with a higher likelihood of claiming safety in the conclusion (stent-assisted coiling: adjusted odds ratio [aOR], 1.9 [95% CI, 1.1-3.8]; complication incidence: β = −0.027; SE, 0.01). Similarly, studies on stent-assisted coiling (aOR, 2.0; 95% CI, 1.1-3.9]) and with a higher incidence of complete occlusion at last follow-up (β= 0.016; SE, 0.007) were associated with a higher likelihood of claiming effectiveness in the conclusion (eAppendix 5 in Supplement 1).

Discussion

In this systematic review and meta-analysis of IA treatment, we found that approximately 80% of the included papers had a positive conclusion on either safety or effectiveness. Most of these papers were single-group and retrospective in nature. In terms of safety, there were no differences in the proportions of patients with poor outcomes between studies with a positive, uncertain, or negative conclusion. When limiting the results to studies on unruptured aneurysms, only the rate of in-hospital mortality was different between the studies with a positive and uncertain conclusion regarding safety. In studies claiming a technique was safe to use in unruptured aneurysms, the maximum proportion of patients with in-hospital mortality was 5%; poor functional outcome, 27.5%; and major complications, 28%. There was no difference in the proportion of aneurysms with a complete occlusion at last follow-up between studies with a positive, uncertain, or negative conclusion neither in terms of effectiveness nor in terms of durability. More than half of all articles had improper reporting. All but 1 RCT showed high or concerning risk of bias in essential domains.

Proving safety and effectiveness of a new technique is important prior to the widespread implementation of such a technique in clinical practice, and its prerequisite is common sense: a new technique should be compared with the current standard of care. Ideally, this should be done in a multicenter RCT, with predefined time points, with blinded, uniform assessment of clinically and radiologically relevant outcomes, with predefined analyses and predefined “safety” and “effectiveness” outcomes. These desiderates are currently almost entirely absent in published studies. If the results of such a study fall within the predefined noninferiority margin compared with current practice, a technique may be adopted in clinical practice. Comparing a new technique with current practice in a prospective manner should be considered the bare minimum for any safety or effectiveness claim to be allowed in a report. Of course, safety may also be claimed when the incidence of complications is low, but without a control group one cannot infer whether the technique should be implemented in practice.

Single-group studies lack a critical benchmark, in that the results obtained cannot be reflected against the results of current standard clinical practice, which makes any claim of safety, effectiveness, or durability weak or unjustifiable. Investigators may be reluctant to label techniques as ineffective or unsafe, as evidenced by the very low number of negative conclusions. Only 0.002% of studies included any kind of incidence threshold for safety, effectiveness, or durability in their definition.

New medical devices must undergo evaluation and approval by regulatory agencies, such as the FDA. This can be done either by providing proof of safety and effectiveness from clinical trials or by the 510(k) clearance process, in which a new device is expected to fare as well as an already existing and approved device. Although this process accelerates innovation, there is evidence to suggest that the latter process is less robust and may lead to more recalls. In addition, the scientific literature backing this process may be methodologically less sound and sometimes even lacking.4,53,54,55 A thorough, critical evaluation of the evidence used to back the FDA approval of these devices is necessary. In our view, the current literature reflects a failure of the peer-review process. In a field with a very high level of technological advance, and with techniques that all carry a certain morbidity and mortality risk, reviewers and editors should be critical and hesitant to accept manuscripts with poor methods and reporting and make sure that conclusions are supported by study results.

Limitations

This study has limitations, including the missing data in the studies; the different definitions of, for example, thrombo-embolic complications or ischemia; the differences in case-mix; and the fact that in many studies results were not reported stratified for patients with aSAH and those with unruptured aneurysms. Another limitation was the fact that the second assessor of the conclusions was not always the same, which may have affected the uniformity of the assessment. We attempted to mitigate the effects of potentially unreliable assessment through training of the assessor and through the assessor’s experience and knowledge.

Conclusions

In this study, we found that studies claiming safety, effectiveness or durability of IA treatment often had methodological flaws and incomplete reporting of relevant outcomes supporting these claims. The consequences for patients with IA can hereby form a potential public health hazard. The field of IA treatment can be improved by applying proper methods and reporting and performing comparative studies in which new technologies are compared with current best clinical practice.

Supplement 1.

eAppendix 1. Search Strategy

eAppendix 2. Main Analysis: Sensitivity Analyses and Stratified Results for Effect Modifiers

eAppendix 3. Frequency of Essential Domain-Specific Outcomes Not Reported in the Included Studies

eAppendix 4. Sensitivity Analysis Including Only Studies With More Than 100 Patients

eAppendix 5. Regression Results

eAppendix 6. Correlations and Cross-Tables

eAppendix 7. Supplementary Figures

eAppendix 8. Statistical Analysis Plan and Data Dictionary

eAppendix 9. Characteristics of Included Studies

eReferences.

Click here for additional data file. (2.9MB, pdf)
Supplement 2.

eTable. Sensitivity Analysis: Studies or Study Arms Reporting Results of Unruptured Aneurysms Only

Click here for additional data file. (66.5KB, xlsx)
Supplement 3.

Data Sharing Statement

Click here for additional data file. (14.7KB, pdf)

References

  • 1.Howick J, Glasziou P, Aronson JK. Evidence-based mechanistic reasoning. J R Soc Med. 2010;103(11):433-441. doi: 10.1258/jrsm.2010.100146 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.Reynolds IS, Rising JP, Coukell AJ, Paulson KH, Redberg RF. Assessing the safety and effectiveness of devices after US Food and Drug Administration approval: FDA-mandated postapproval studies. JAMA Intern Med. 2014;174(11):1773-1779. doi: 10.1001/jamainternmed.2014.4194 [DOI] [PubMed] [Google Scholar]
  • 3.Somberg JC, McEwen P, Molnar J. Assessment of cardiovascular and noncardiovascular medical device recalls. Am J Cardiol. 2014;113(11):1899-1903. doi: 10.1016/j.amjcard.2014.03.024 [DOI] [PubMed] [Google Scholar]
  • 4.Zheng SY, Dhruva SS, Redberg RF. Characteristics of clinical studies used for US Food and Drug Administration approval of high-risk medical device supplements. JAMA. 2017;318(7):619-625. doi: 10.1001/jama.2017.9414 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Etminan N, Chang HS, Hackenberg K, et al. Worldwide incidence of aneurysmal subarachnoid hemorrhage according to region, time period, blood pressure, and smoking prevalence in the population: a systematic review and meta-analysis. JAMA Neurol. 2019;76(5):588-597. doi: 10.1001/jamaneurol.2019.0006 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6.GBD 2019 Stroke Collaborators . Global, regional, and national burden of stroke and its risk factors, 1990-2019: a systematic analysis for the Global Burden of Disease Study 2019. Lancet Neurol. 2021;20(10):795-820. doi: 10.1016/S1474-4422(21)00252-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Lawton MT, Vates GE. Subarachnoid hemorrhage. N Engl J Med. 2017;377(3):257-266. doi: 10.1056/NEJMcp1605827 [DOI] [PubMed] [Google Scholar]
  • 8.Neifert SN, Chapman EK, Martini ML, et al. Aneurysmal subarachnoid hemorrhage: the last decade. Transl Stroke Res. 2021;12(3):428-446. doi: 10.1007/s12975-020-00867-0 [DOI] [PubMed] [Google Scholar]
  • 9.Roy D, Milot G, Raymond J. Endovascular treatment of unruptured aneurysms. Stroke. 2001;32(9):1998-2004. doi: 10.1161/hs0901.095600 [DOI] [PubMed] [Google Scholar]
  • 10.Deshaies EM, Adamo MA, Boulos AS. A prospective single-center analysis of the safety and efficacy of the hydrocoil embolization system for the treatment of intracranial aneurysms. J Neurosurg. 2007;106(2):226-233. doi: 10.3171/jns.2007.106.2.226 [DOI] [PubMed] [Google Scholar]
  • 11.Fargen KM, Blackburn S, Deshaies EM, et al. Final results of the multicenter, prospective Axium MicroFX for Endovascular Repair of IntraCranial Aneurysm Study (AMERICA). J Neurointerv Surg. 2015;7(1):40-43. doi: 10.1136/neurintsurg-2013-011049 [DOI] [PubMed] [Google Scholar]
  • 12.Taschner CA, Vedantham S, de Vries J, et al. Surpass flow diverter for treatment of posterior circulation aneurysms. AJNR Am J Neuroradiol. 2017;38(3):582-589. doi: 10.3174/ajnr.A5029 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Kim BM, Kim DJ, Jeon P, et al. Endovascular embolization of intracranial aneurysms using bare platinum axium detachable coils: immediate and short-term follow-up results from a multicenter registry. Neurointervention. 2012;7(2):85-92. doi: 10.5469/neuroint.2012.7.2.85 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Cortez GM, Akture E, Monteiro A, et al. Woven EndoBridge device for ruptured aneurysms: Perioperative results of a US multicenter experience. J Neurointervent Surg. 2021;13(11):1012-1016. doi: 10.1136/neurintsurg-2020-017105 [DOI] [PubMed] [Google Scholar]
  • 15.Li W, Ye M, Cimpoca A, et al. Avenir vs Axium coils for the treatment of intracranial aneurysms: results of a multicenter randomized controlled trial with short-term follow-up. Front Neurol. 2022;12:817989. doi: 10.3389/fneur.2021.817989 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Daou BJ, Palmateer G, Linzey JR, et al. Stent-assisted coiling of cerebral aneurysms: Head to head comparison between the Neuroform Atlas and EZ stents. Interv Neuroradiol. 2021;27(3):353-361. doi: 10.1177/1591019921989476 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Hanel RA, Kallmes DF, Lopes DK, et al. Prospective study on embolization of intracranial aneurysms with the pipeline device: the PREMIER study 1 year results. J Neurointerv Surg. 2020;12(1):62-66. doi: 10.1136/neurintsurg-2019-015091 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Fiorella D, Boulos A, Turk AS, et al. ; LVIS investigators . The safety and effectiveness of the LVIS stent system for the treatment of wide-necked cerebral aneurysms: final results of the pivotal US LVIS trial. J Neurointerv Surg. 2019;11(4):357-361. doi: 10.1136/neurintsurg-2018-014309 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Jankowitz BT, Hanel R, Jadhav AP, et al. Neuroform Atlas Stent System for the treatment of intracranial aneurysm: primary results of the Atlas Humanitarian Device Exemption cohort. J Neurointerv Surg. 2019;11(8):801-806. doi: 10.1136/neurintsurg-2018-014455 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Zaidat OO, Hanel RA, Sauvageau EA, et al. ; ATLAS Investigators . Pivotal trial of the neuroform atlas stent for treatment of anterior circulation aneurysms: one-year outcomes. Stroke. 2020;51(7):2087-2094. doi: 10.1161/STROKEAHA.119.028418 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Möhlenbruch MA, Kizilkilic O, Killer-Oberpfalzer M, et al. Multicenter experience with FRED Jr flow re-direction endoluminal device for intracranial aneurysms in small arteries. AJNR Am J Neuroradiol. 2017;38(10):1959-1965. doi: 10.3174/ajnr.A5332 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Poncyljusz W, Biliński P, Safranow K, et al. The LVIS/LVIS Jr. stents in the treatment of wide-neck intracranial aneurysms: multicentre registry. J Neurointerv Surg. 2015;7(7):524-529. doi: 10.1136/neurintsurg-2014-011229 [DOI] [PubMed] [Google Scholar]
  • 23.Fiorella D, Molyneux A, Coon A, et al. ; WEB-IT Study Investigators . Demographic, procedural and 30-day safety results from the WEB Intra-saccular Therapy Study (WEB-IT). J Neurointerv Surg. 2017;9(12):1191-1196. doi: 10.1136/neurintsurg-2016-012841 [DOI] [PubMed] [Google Scholar]
  • 24.Arthur AS, Molyneux A, Coon AL, et al. ; WEB-IT Study investigators . The safety and effectiveness of the Woven EndoBridge (WEB) system for the treatment of wide-necked bifurcation aneurysms: final 12-month results of the pivotal WEB Intrasaccular Therapy (WEB-IT) Study. J Neurointerv Surg. 2019;11(9):924-930. doi: 10.1136/neurintsurg-2019-014815 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Yue J, Xie Y, Zhang X, et al. The safety and outcomes of acutely ruptured intracranial aneurysms with incomplete occlusion after coiling: a case-control study. BMC Neurol. 2020;20(1):412. doi: 10.1186/s12883-020-01991-y [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26.Petrov A, Rentsenkhuu G, Nota B, et al. Initial experience with the novel p64MW HPC flow diverter from a cohort study in unruptured anterior circulation aneurysms under dual antiplatelet medication. Interv Neuroradiol. 2021;27(1):42-50. doi: 10.1177/1591019920939845 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27.Roh H, Kim J, Bae H, et al. Comparison of stent-assisted and no-stent coil embolization for safety and effectiveness in the treatment of ruptured intracranial aneurysms. J Neurosurg. 2019;133(3):1-7. [DOI] [PubMed] [Google Scholar]
  • 28.Kan P, Mohanty A, Meyers PM, et al. Treatment of large and giant posterior communicating artery aneurysms with the Surpass treamline flow diverter: results from the SCENT trial. J Neurointerv Surg. 2023;15(7)679-683. doi: 10.1136/neurintsurg-2021-018189 [DOI] [PubMed] [Google Scholar]
  • 29.Becske T, Brinjikji W, Potts MB, et al. Long-term clinical and angiographic outcomes following pipeline embolization device treatment of complex internal carotid artery aneurysms: five-year results of the pipeline for uncoilable or failed aneurysms trial. Neurosurgery. 2017;80(1):40-48. doi: 10.1093/neuros/nyw014 [DOI] [PubMed] [Google Scholar]
  • 30.Hanel RA, Cortez GM, Lopes DK, et al. Prospective study on embolization of intracranial aneurysms with the pipeline device (PREMIER study): 3-year results with the application of a flow diverter specific occlusion classification. J Neurointerv Surg. 2023;15(3)248-254. doi: 10.1136/neurintsurg-2021-018501 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Lefevre PH, Schramm P, Kemmling A, et al. ; ATLAS EU PMCF Investigators . Multi-centric European post-market follow-up study of the Neuroform Atlas Stent System: primary results. J Neurointerv Surg. 2022;14(7):694-698. doi: 10.1136/neurintsurg-2021-017849 [DOI] [PubMed] [Google Scholar]
  • 32.Aguilar Pérez M, Henkes H, Kurre W, et al. Results of the pToWin Study: using the pCONUS Device for the treatment of wide-neck intracranial aneurysms. J Clin Med. 2022;11(3):884. doi: 10.3390/jcm11030884 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 33.Zhang H, Li L, Zhang H, et al. Small and medium-sized aneurysm outcomes following intracranial aneurysm treatment using the pipeline embolization device: a subgroup analysis of the PLUS registry. Front Neurol. 2022;13:881353. doi: 10.3389/fneur.2022.881353 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34.Hanel RA, Yoon N, Sauvageau E, et al. Neuroform Atlas stent for treatment of middle cerebral artery aneurysms: 1-year outcomes from Neuroform Atlas stent pivotal trial. Neurosurgery. 2021;89(1):102-108. doi: 10.1093/neuros/nyab090 [DOI] [PubMed] [Google Scholar]
  • 35.Raymond J, Klink R, Chagnon M, et al. ; PRET Collaborative Group . Patients prone to recurrence after endovascular treatment: periprocedural results of the PRET randomized trial on large and recurrent aneurysms. AJNR Am J Neuroradiol. 2014;35(9):1667-1676. doi: 10.3174/ajnr.A4035 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 36.McDougall CG, Diaz O, Boulos A, et al. Safety and efficacy results of the Flow Redirection Endoluminal Device (FRED) stent system in the treatment of intracranial aneurysms: US pivotal trial. J Neurointerv Surg. 2022;14(6):577-584. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 37.White PM, Lewis SC, Gholkar A, et al. ; HELPS trial collaborators . Hydrogel-coated coils versus bare platinum coils for the endovascular treatment of intracranial aneurysms (HELPS): a randomised controlled trial. Lancet. 2011;377(9778):1655-1662. doi: 10.1016/S0140-6736(11)60408-X [DOI] [PubMed] [Google Scholar]
  • 38.Coley S, Sneade M, Clarke A, et al. Cerecyte coil trial: procedural safety and clinical outcomes in patients with ruptured and unruptured intracranial aneurysms. AJNR Am J Neuroradiol. 2012;33(3):474-480. doi: 10.3174/ajnr.A2836 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 39.Molyneux A, Kerr R, Stratton I, et al. ; International Subarachnoid Aneurysm Trial (ISAT) Collaborative Group . International Subarachnoid Aneurysm Trial (ISAT) of neurosurgical clipping versus endovascular coiling in 2143 patients with ruptured intracranial aneurysms: a randomised trial. Lancet. 2002;360(9342):1267-1274. doi: 10.1016/S0140-6736(02)11314-6 [DOI] [PubMed] [Google Scholar]
  • 40.Li XY, Li CH, Wang JW, Liu JF, Li H, Gao BL. Safety and efficacy of endovascular embolization of ruptured intracranial aneurysms within 72 hours of subarachnoid hemorrhage. J Neurol Surg A Cent Eur Neurosurg. 2022;83(3):265-274. doi: 10.1055/s-0041-1731752 [DOI] [PubMed] [Google Scholar]
  • 41.Spetzler RF, McDougall CG, Zabramski JM, et al. Ten-year analysis of saccular aneurysms in the Barrow Ruptured Aneurysm Trial. J Neurosurg. 2019;132(3):771-776. doi: 10.3171/2018.8.JNS181846 [DOI] [PubMed] [Google Scholar]
  • 42.Taschner CA, Chapot R, Costalat V, et al. GREAT—a randomized controlled trial comparing HydroSoft/HydroFrame and bare platinum coils for endovascular aneurysm treatment: procedural safety and core-lab-assessed angiographic results. Neuroradiology. 2016;58(8):777-786. doi: 10.1007/s00234-016-1693-y [DOI] [PubMed] [Google Scholar]
  • 43.White PM, Lewis SC, Nahser H, Sellar RJ, Goddard T, Gholkar A; HELPS Trial Collaboration . HydroCoil Endovascular Aneurysm Occlusion and Packing Study (HELPS trial): procedural safety and operator-assessed efficacy results. AJNR Am J Neuroradiol. 2008;29(2):217-223. doi: 10.3174/ajnr.A0936 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 44.Zhao R, Duan G, Yang P, et al. ; CATCH Investigators . Endovascular aneurysm treatment with the numen coil embolization system: a prospective randomized controlled open-label multicenter noninferiority trial in China. World Neurosurg. 2022;160:e23-e32. doi: 10.1016/j.wneu.2021.11.067 [DOI] [PubMed] [Google Scholar]
  • 45.Molyneux AJ, Birks J, Clarke A, Sneade M, Kerr RSC. The durability of endovascular coiling versus neurosurgical clipping of ruptured cerebral aneurysms: 18 year follow-up of the UK cohort of the International Subarachnoid Aneurysm Trial (ISAT). Lancet. 2015;385(9969):691-697. doi: 10.1016/S0140-6736(14)60975-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 46.Brinjikji W, White PM, Nahser H, et al. HydroCoils reduce recurrence rates in recently ruptured medium-sized intracranial aneurysms: a subgroup analysis of the HELPS trial. AJNR Am J Neuroradiol. 2015;36(6):1136-1141. doi: 10.3174/ajnr.A4266 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 47.Brinjikji W, White PM, Nahser H, et al. HydroCoils are associated with lower angiographic recurrence rates than are bare platinum coils in treatment of “difficult-to-treat” aneurysms: a post hoc subgroup analysis of the HELPS trial. AJNR Am J Neuroradiol. 2015;36(9):1689-1694. doi: 10.3174/ajnr.A4349 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 48.Bendok BR, Abi-Aad KR, Ward JD, et al. ; HEAT Study Investigators . The Hydrogel Endovascular Aneurysm Treatment Trial (HEAT): a randomized controlled trial of the second-generation hydrogel coil. Neurosurgery. 2020;86(5):615-624. doi: 10.1093/neuros/nyaa006 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 49.McDougall CG, Spetzler RF, Zabramski JM, et al. The Barrow Ruptured Aneurysm Trial: clinical article. J Neurosurg. 2012;116(1):135-144. doi: 10.3171/2011.8.JNS101767 [DOI] [PubMed] [Google Scholar]
  • 50.Raymond J, Klink R, Chagnon M, et al. Hydrogel versus bare platinum coils in patients with large or recurrent aneurysms prone to recurrence after endovascular treatment: a randomized controlled trial. AJNR Am J Neuroradiol. 2017;38(3):432-441. doi: 10.3174/ajnr.A5101 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 51.You L, Huang J, Zhang J, Jiang Z. Multiple overlapping stent-assisted coiling improves efficacy and safety of treatment for complex intracranial aneurysms: a randomized trial. Biomed Eng Online. 2021;20(1):100. doi: 10.1186/s12938-021-00936-x [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 52.Wadd IH, Haroon A, Habibullah, et al. Aneurysmal subarachnoid hemorrhage: outcome of aneurysm clipping versus coiling in anterior circulation aneurysm. J Coll Physicians Surg Pak. 2015;25(11):798-801. [PubMed] [Google Scholar]
  • 53.Feldman MD, Petersen AJ, Karliner LS, Tice JA. Who is responsible for evaluating the safety and effectiveness of medical devices? the role of independent technology assessment. J Gen Intern Med. 2008;23(Suppl 1)(suppl 1):57-63. doi: 10.1007/s11606-007-0275-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 54.Kadakia KT, Dhruva SS, Caraballo C, Ross JS, Krumholz HM. Use of recalled devices in new device authorizations under the US Food and Drug Administration’s 510(k) pathway and risk of subsequent recalls. JAMA. 2023;329(2):136-143. doi: 10.1001/jama.2022.23279 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 55.Hwang TJ, Sokolov E, Franklin JM, Kesselheim AS. Comparison of rates of safety issues and reporting of trial outcomes for medical devices approved in the European Union and United States: cohort study. BMJ. 2016;353:i3323. doi: 10.1136/bmj.i3323 [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplement 1.

eAppendix 1. Search Strategy

eAppendix 2. Main Analysis: Sensitivity Analyses and Stratified Results for Effect Modifiers

eAppendix 3. Frequency of Essential Domain-Specific Outcomes Not Reported in the Included Studies

eAppendix 4. Sensitivity Analysis Including Only Studies With More Than 100 Patients

eAppendix 5. Regression Results

eAppendix 6. Correlations and Cross-Tables

eAppendix 7. Supplementary Figures

eAppendix 8. Statistical Analysis Plan and Data Dictionary

eAppendix 9. Characteristics of Included Studies

eReferences.

Click here for additional data file. (2.9MB, pdf)
Supplement 2.

eTable. Sensitivity Analysis: Studies or Study Arms Reporting Results of Unruptured Aneurysms Only

Click here for additional data file. (66.5KB, xlsx)
Supplement 3.

Data Sharing Statement

Click here for additional data file. (14.7KB, pdf)

Articles from JAMA Network Open are provided here courtesy of American Medical Association

RESOURCES