Patent Foramen Ovale Closure and Medical Treatments for Secondary Stroke Prevention A Systematic Review of Observational and Randomized Evidence

Abstract

Background and Purpose

Patients discovered to have a patent foramen ovale in the setting of a cryptogenic stroke may be treated with percutaneous closure, antiplatelet therapy, or anticoagulants. A recent randomized trial (CLOSURE I) did not detect any benefit of closure over medical treatment alone; the optimal medical therapy is also unknown. We synthesized the available evidence on secondary stroke prevention in patients with patent foramen ovale and cryptogenic stroke.

Methods

A MEDLINE search was performed for finding longitudinal studies investigating medical treatment or closure, meta-analysis of incidence rates (IR), and IR ratios of recurrent cerebrovascular events.

Results

Fifty-two single-arm studies and 7 comparative nonrandomized studies and the CLOSURE I trial were reviewed. The summary IR of recurrent stroke was 0.36 events (95% CI, 0.24–0.56) per 100 person-years with closure versus 2.53 events (95% CI, 1.91–3.35) per 100 person-years with medical therapy. In comparative observational studies, closure was superior to medical therapy (IR ratio=0.19; 95% CI, 0.07–0.54). The IR for the closure arm of the CLOSURE I trial was higher than the summary estimate from observational studies; there was no significant benefit of closure over medical treatment (P=0.002 comparing efficacy estimates between observational studies and the trial). Observational and randomized data (9 studies) comparing medical therapies were consistent and suggested that anticoagulants are superior to antiplatelets for preventing stroke recurrence (IR ratio=0.42; 95% CI, 0.18–0.98).

Conclusions

Although further randomized trial data are needed to precisely determine the effects of closure on stroke recurrence, the results of CLOSURE I challenge the credibility of a substantial body of observational evidence strongly favoring mechanical closure over medical therapy.

The optimal management strategy for treating patients with cryptogenic stroke (CS) who are discovered to have a patent foramen ovale (PFO) remains to be defined. The advent of percutaneously implantable devices has offered a widely applicable closure approach not requiring open heart surgery. Case series of patients undergoing closure have repeatedly suggested that percutaneous closure is safe and results in low recurrence rates. ^1–3 Such observations, together with the plausible pathophysiologic rationale that supports PFO closure, have resulted in the rapid clinical diffusion of this approach, with devices for septal defects being used on an “off-label” basis in the United States. ⁴ However, early results from the first completed randomized clinical trial (RCT), CLOSURE I, comparing closure with medical therapy for secondary prevention did not identify a significant difference in the risk of recurrent cerebrovascular events between the 2 arms. ⁵ There has not been any recent comprehensive summary providing quantitative estimates of the comparative effectiveness of closure versus medical therapy, and the preferred medical therapy (antiplatelets versus anticoagulants) also remains unclear.

Thus, to assess the totality of available evidence on secondary stroke prevention strategies in patients with PFO who have experienced a CS, we systematically reviewed and appraised the relevant randomized and observational data. We obtained quantitative summaries of the absolute risk of recurrence, the relative efficacy and safety of closure and medical treatment, and we explored modifiers of these estimates.

Materials and Methods

Literature Search and Study Selection

Previous systematic reviews on percutaneous closure or medical management of PFO for the secondary prevention of cerebrovascular events have not examined quantitatively the comparative effectiveness of available treatment options but have systematically searched the relevant literature. ^1–3,6,7 Thus, we perused the lists of studies analyzed in the previously published systematic reviews and updated the most comprehensive literature search strategies ^1,7 from their last date of inclusion until August 2010 using the MEDLINE database. Reference lists of included publications and selected review articles were also manually screened to identify additional potentially relevant citations. We considered only studies published in full text and in the English language. Eligible studies were longitudinal in design, included at least 10 patients with presumed paradoxical embolism (including CS and transient ischemic attack [lsqb]TIA[rsqb]), investigated medical treatment or percutaneous PFO closure, and reported the incidence of recurrent cerebrovascular events (stroke or TIA) over a minimum mean follow-up period of 12 months. We considered both comparative (randomized or nonrandomized) and single-arm studies of medical and closure treatment.

Data Extraction

From each study, we extracted information on patient characteristics (including demographics, the proportion of patients with cardiovascular risk factors and atrial septal aneurysms), types of medical or percutaneous closure treatment used, study design variables, and outcome data for recurrent cerebrovascular events (including stroke, TIA, or all ischemic cerebrovascular events). For each outcome, we extracted or calculated the number of recurrent events and the total duration of the follow-up experience in each cohort expressed in person-years (calculated from raw data or extracted from Kaplan-Meier curves). ⁸ From closure treatment studies, we also extracted information on periprocedural adverse events. For studies reporting data on mixed cohorts of patients with various types of right-to-left atrial shunts, we extracted data only for the subgroup of patients with PFO when available. In the case of comparative studies involving both medical and closure treatment arms, we extracted data separately from each arm and described data for these groups as distinct strata. Baseline characteristics and outcome data were extracted in duplicate. For those studies that had been included in published systematic reviews, we also cross-validated our extractions with the previously reported data. Discrepancies were resolved by consensus.

Assessment of Study Validity

In view of the diverse study designs we planned to consider and the well-recognized limitations of quality scores, ⁹ we did not use a scale to assign quality scores to studies. Instead, we assessed the following study-level characteristics: study design (prospective or retrospective); the use of imaging modalities for the detection of PFO; duration of follow-up; and the methods for outcome ascertainment. Specifically, the robustness of outcome ascertainment was assessed based on whether a structured screening instrument for stroke detection was used, whether a neurologist ascertained recurrent events, and whether events were documented by neuroimaging.

Statistical Analyses

For each stratum of medical or closure treatment (single-arm studies or respective arms of comparative studies), we estimated the IR of recurrent events as events per person-years of follow-up, and the associated 95% exact Poisson confidence interval (95% CI). To obtain summary estimates of IR of recurrent events from these single-arm strata of medical or closure treatment, we performed meta-analyses using a generalized linear multilevel model for count data with the exact Poisson likelihood (random effects Poisson regression). ¹⁰ Meta-analysis using generalized linear regression models avoids many of the problems encountered in meta-analysis of rare events, such as violations of distributional assumptions (non-normally distributed outcomes) attributable to small sample sizes or the presence of zero cells.

For comparative studies, we estimated the incidence rate ratio (IRR) comparing the incidence densities (events over person-time) of the outcomes of interest between PFO closure and medical treatment arms. We obtained summary IRR using random-effects Poisson regression, as described. For studies that estimated the relative effect of closure versus medical treatment after adjusting for potential confounders, we synthesized the reported maximally adjusted effect sizes using a random-effects model ¹¹ as a sensitivity analysis. We compared the summary estimate from this analysis to the one derived from Poisson regression.

Although comparative studies represent the most informative design for examining treatment effects (allowing the direct estimation of IRR), we also aimed to incorporate information from the large number of uncontrolled (single-arm) studies that have investigated either PFO closure or medical treatment. ¹² Thus, we obtained an estimate for the treatment effect (IRR) incorporating all available evidence, including both comparative and single-arm studies in a common random-effects model. ^13,14

We performed a subgroup analysis of studies including only younger patients (younger than 60 years old). To explore further whether between-study differences in IR could be explained by patient characteristics, such as patient mean age, or the proportion of patients with atrial septal aneurysm, hypertension, diabetes, smoking, hyperlipidemia, or stroke (versus TIA) as index event, we included these study-level variables as covariates in meta-regression models and also examined their interactions with the treatment effect. ¹⁵ We used the same approach to explore whether study-level indicators of robustness in outcome ascertainment (use of a structured screening instrument for stroke detection, outcome ascertainment by a neurologist, use of neuroimaging to document recurrent events) explained between-study variability. Furthermore, for medical treatment arms, we performed subgroup analyses stratified by type of medical treatment (antiplatelets versus anticoagulants). For closure treatment arms, we recorded the types of devices used in each study; however, the inclusion of multiple devices in each of the studies and lack of comparative studies with reporting of outcome data by device precluded analyses comparing device types.

Based on the currently available information from the CLOSURE I trial that compared closure with the STARFlex device versus best medical therapy (aspirin or warfarin or combination thereof), ⁵ we extracted baseline demographic information, long-term outcome, ⁸ and periprocedural adverse event rates. We juxtaposed these data with the synthesized data of observational studies. For single-arm IR estimates, we formally tested for differences between the observational summary and the trial estimates by including the trial data in the random-effects Poisson regression model with an indicator variable for study design. For the efficacy estimate of closure, we compared the summary relative effect size from observational studies (IRR) to the effect size extracted from CLOSURE I (hazard ratio) with a test for interaction. ¹⁶

Statistical analyses were conducted using Stata version 11.1/SE (Stata Corp) and Meta-analyst (version 3.0beta; Tufts Medical Center). ¹⁷ Statistical significance was defined as a 2-sided P[lt]0.05. We did not perform adjustments for multiple comparisons. ¹⁸

Results

Search Results

The selection of studies is depicted in Figure 1. Merging the search results of the 2 updated search strategies yielded 511 unique abstracts. A total of 115 full-text articles (89 after abstract screening and 26 articles included in previous reviews) were evaluated against our inclusion criteria, and finally 57 articles were included. ^19–75 Of these, 7 studies were comparative (thus contributing data to both medical and closure treatment arms), 2 studies provided data on 2 different cohorts of patients, and 1 study 19 provided information on the IR for a group of patients observed before and after closure treatment; the person-time corresponding to medical treatment was included only in sensitivity analyses. Thus, in the main analysis a total of 66 meta-analytic strata were considered (52 strata from single-arm studies and 14 strata from the 7 comparative studies).

Figure 1.

Flow chart of the study selection process.

Eligible Studies

Summary and study-level characteristics of the included studies are shown in Table 1 and Supplemental Tables 1 and 2 (http://stroke.ahajournals.org). Data from a total of 8916 patients with PFO (1903 receiving medical treatment and 7013 undergoing percutaneous closure) were included in the main analysis. The majority of studies were prospective in design; the median of the average ages of participants was [dsim]47 years. Patient characteristics in medical and closure treatment arms were generally similar; however, there was some evidence that the median proportion of smokers or of patients who had a stroke as the index event was higher in medical than in closure studies. In medical treatment arms, patients were treated with either antiplatelets (aspirin, clopidogrel, or ticlodipine) or anticoagulants (warfarin or heparinoids). The proportion of patients under each type of treatment varied considerably across studies. For closure treatment arms, we observed that the majority of studies (70%) used multiple closure devices (Supplemental Table 2). Considering the cumulative number of patients enrolled in studies of various designs over time (Figure 2), it becomes evident that the majority of patients have been enrolled in single-arm closure studies, with a rapid pace of enrollment in the past 10 years, followed by enrollment in nonrandomized comparative studies.

Table 1.

Summary Characteristics of Studies Examining Medical Treatment and Percutaneous Closure

	Medical Treatment Studies	Percutaneous Closure Studies
No. of studies	17	49
Total no. of included patients	1903	7013
Prospective design	76%	75%
Mean age of patients, median (25th–75th percentile), n	47.0 (43.0–51.1), n=17	46.4 (43.0–50.5), n=48
% Male, median (25th–75th percentile), n	58 (51–60), n=16	53 (48–56), n=45
% Atrial septal aneurysm, median (25th–75th percentile), n	23 (19–36), n=14	31 (22–37), n=43
% Hypertension, median (25th–75th percentile), n	24 (20–31), n=14	23 (17–34), n=31
% With diabetes, median (25th–75th percentile), n	6 (3–9), n=14	6 (4–8), n=27
% Hyperlipidemia, median (25th–75th percentile), n	18 (12–29), n=10	25 (15–32), n=23
% Smoking, median (25th–75th percentile), n	33 (25–40), n=13	22 (15–32), n=28
% With stroke as the index event, median (25th–75th percentile), n	76 (63–84), n=17	63 (48–73), n=42
Use of structured screening instrument for recurrent stroke detection	88%	47%
Recurrent events ascertained by neurologist	82%	55%
Recurrent events documented by neuroimaging	59%	45%

N indicates number of studies providing relevant information (reported when some studies did not present data on a variable of interest).

Figure 2.

Cumulative number of patients recruited in studies of different designs over a period of 16 years for the comparison of closure vs medical treatment. RCT indicates randomized clinical trial.

Meta-Analysis Results for Cerebrovascular Outcomes

The results of the synthesis of IR from single-arm studies are shown in Table 2 and in Figure 3. Among the 49 studies examining patients treated with closure, 26 reported no recurrent cerebrovascular events among patients treated with closure. The summary IR of recurrent stroke in closure studies was 0.36 events (95% CI, 0.24–0.56) per 100 person-years, whereas the summary IR derived from the 18 medical studies was much higher at 2.53 events (95% CI, 1.91–3.35) per 100 person-years. Both meta-analytic estimates were characterized by significant statistical heterogeneity (P[lt]0.001). Medical treatment was also associated with higher IR of TIA and the composite outcome of all ischemic cerebrovascular events.

Table 2.

Summary Estimates Derived From Meta-Analyses of Observational Studies and Estimates Extracted From the CLOSURE I Trial

Outcome		Observational Studies			Randomized Clinical Trial (CLOSURE I)
	Closure Arm IR (95% CI), per 100 Person-Years	Medical Arm IR (95% CI), per 100 Person- Years	IRR (95% CI) Closure vs Medical (Comparative Studies Only)	IRR (95% CI) Closure vs Medical (All Studies)	Closure Arm IR (95% CI), per 100 Person- Years	Medical Arm IR (95% CI), per 100 Person- Years	HR (95% CI) Closure vs Medical
Total events	0.80 (0.55–1.18)	4.73 (3.41–6.56)	0.22 (0.07–0.64)	0.17 (0.10–0.28)	2.79 (1.81–4.13)	3.25 (2.19–4.63)	0.76 (0.45–1.29)
Stroke events	0.36 (0.24–0.56)	2.53 (1.91–3.35)	0.19 (0.07–0.54)	0.14 (0.08–0.24)	1.34 (0.69–2.34)	1.41 (0.74–2.41)	0.89 (0.41–1.95)
TIA events	0.46 (0.29–0.74)	1.93 (1.16–3.20)	0.15 (0.02–1.35)	0.24 (0.12–0.47)	1.45 (0.77–2.49)	1.83 (1.10–2.94)	0.73 (0.35–1.50)

For observational studies, results are presented separately for meta-analyses of incidence rates of events (stroke and transient ischemic attack) from single-arm studies and for meta-analyses of incidence rate ratios from comparative studies (with or without considering the evidence from single arm studies).

CI indicates confidence interval; HR, hazard ratio; IR, incidence rate; IRR, incidence rate ratio; TIA, transient ischemic attack.

Figure 3.

Forest plots for the meta-analyses of incidence rates of recurrent strokes (per 100 person-years) for the closure and medical arms, respectively. Results from comparative (2-arm) studies are shown in the shaded area; these studies contributed data to the meta-analysis of incidence rates of both closure and medical treatment arms. Incidence rate estimates for each arm of the CLOSURE I trial are shown in red.

The IR of recurrent stroke in closure and medical treatment arms for the subgroup of studies exclusively enrolling patients younger than 60 years were similar to the overall analysis of 9 closure arms (IR=0.45; 95% CI, 0.13–1.54) and 6 medical arms (IR=2.30; 95% CI, 1.43–3.68). In meta-regression analyses, differences in study-level averages of patient characteristics (mean age, atrial septal aneurysm, hypertension, diabetes, smoking, hyperlipidemia, or stroke as index event) were not associated with heterogeneity in the estimated IR in either the closure or in the medical arms. Similarly, these variables did not appear to modify (interact with) the treatment effect at the study level.

The meta-analysis of comparative studies (n=7) showed that closure was associated with an 81% reduction in the IR of stroke compared with medical treatment (IRR=0.19; 95% CI, 0.07–0.54). Including single-arm studies in the model did not substantially change the point estimate of the treatment effect but resulted in improved precision (IRR=0.14; 95% CI, 0.08–0.24).

Periprocedural Adverse Events With Closure

Adverse events that occurred in PFO patients treated with closure during the periprocedural period were reported by 39 studies. Studies did not report adverse events in a consistent and comprehensive manner that covers all procedure-related complications, but rather provided data on the exact complications that occurred in each study cohort (Supplemental Table 3). It was not considered appropriate to quantitatively synthesize this information.

Comparing Alternative Medical Treatments and Closure Devices

Within medical treatment arms, recurrent stroke outcome data stratified by the specific medical treatment administered (anticoagulation versus antiplatelets) were available for 7 comparative studies; of those, 6 were observational and 1 was a subgroup analysis of CS patients with PFO (n=98) from the randomized PFO in Cryptogenic Stroke Study (PICSS). Two additional studies were single-arm studies of antiplatelet treatment. By synthesis of results of all 9 studies, treatment with anticoagulation had a lower IR of recurrent stroke (1.27; 95% CI, 0.44–3.64) than antiplatelets (3.17; 95% CI, 1.94–5.18), and this difference was statistically significant (IRR=0.42; 95% CI, 0.18–0.98). Considering the 7 comparative studies only, this difference was of similar magnitude but not statistically significant (IRR=0.41; 95% CI, 0.16–1.01). However, analyses for total cerebrovascular events and for TIA alone showed consistently statistically significant results favoring anticoagulation (Supplemental Table 5).

Within closure treatment arms, outcome data by device type were commonly unavailable. Amplatzer Occluders (9 studies) and the Cardioseal device (4 studies) were the most commonly used devices in studies that used a single type of device ([mt]90% of all procedures performed in the study) for the majority of patients. By synthesizing the IR of recurrent events in this subset of noncomparative studies, we observed that Amplatzer Occluders were associated with lower rates of events (Supplemental Table 6). However, further comparative analyses to evaluate specific types of devices were not performed because of the small number of comparative studies and incomplete reporting in studies that used multiple devices (Supplemental Table 2).

Sensitivity Analyses and Assessment of Bias

The study by van de Wyngaert et al ¹⁹ provided information on the IR for a group of patients observed before and after closure treatment. In the main analysis, we excluded the retrospective follow-up period (ie, pre-PFO closure) from this study (and only included events and person-time from the post-PFO closure prospective follow-up period) to eliminate within-subject correlation. Including the retrospective portion of this study in a sensitivity analysis of medical treatment studies did not materially change the summary IR (data not shown).

Three studies reported adjusted hazard ratios for the effect of closure versus medical treatment obtained using Cox proportional hazards models. The random effects summary hazard ratio based on these studies was 0.32 (95% CI, 0.15–0.71), which is consistent with analyses based on events and person-time from all studies (main analysis), as well as the results of unadjusted IRR meta-analysis of comparative studies (Supplemental Figure 1).

The correlation between sample size and reported IR estimates for stroke among the medical treatment arms was not strong or statistically significant (Spearman rank correlation coefficient, rho=[minus]0.05; P=0.851); however, the correlation coefficient between IR and sample sizes of closure arms was positive and statistically significant (rho=0.351; P=0.013), providing some indication of “small study” effects (ie, possible evidence of publication bias).

The meta-regression analysis for study-level indicators of robustness in outcome ascertainment showed that the use of a structured screening instrument for cerebrovascular event detection was associated with increased IR of stroke (P=0.016; Supplemental Table 4). This increased detection of strokes with the use of such instruments (questionnaires) was true for both closure and medical treatment arms (P for difference [mt]0.05); because such instruments were more commonly used by medical treatment studies (Table 1), this factor may explain, in part, the higher IR of stroke with medical treatment.

Comparison of Observational Summary Estimates With the RCT Results

By comparing the reported baseline characteristics of patients enrolled in the closure and medical arms of the CLOSURE I trial to the average patient characteristics in the observational studies, we observed that the mean age of patients and the proportion of males were similar. However, patients randomized to the closure arm of the trial were more likely to have experienced a stroke (not a TIA) as the qualifying event (73%) and to have experienced an atrial septal aneurysm (38%) compared to patients treated with closure in observational studies (across studies, the corresponding median proportions were 63% and 31%).

Assuming no censoring over the 2-year follow-up period of the CLOSURE I trial, we calculated IR of recurrent stroke equal to 1.34 (95% CI, 0.69–2.34) and 1.41 (95% CI, 0.74–2.41) events per 100 person-years for the closure and medical arms, respectively. The hazard ratio of closure versus medical treatment from time to recurrent stroke analyses based on the trial was 0.89 (95% CI, 0.41–1.95; Table 2). The juxtaposition of the results from observational studies and the randomized trial are in stark disagreement: the IR of recurrent stroke in patients randomized to closure was substantially larger (almost 4-fold) than the summary IR in observational closure studies (1.34; 95% CI, 0.69–2.34) versus 0.36 (95% CI, 0.24–0.56) events per 100 person-years, respectively (P for difference [lt]0.001). Furthermore, patients randomized to medical treatment fared better than those receiving medical treatment in observational studies, although this difference was not statistically significant (IR=1.41 and 95% CI, 0.74–2.41 versus 2.53 and 95% CI, 1.91–3.35, respectively; P for difference=0.079). Overall, randomized and observational studies provided dramatically different estimates of closure efficacy (P for difference=0.002; Table 2). This difference remained significant even when only comparative observational studies were considered (P for difference=0.018).

Differences in reporting of adverse events between observational studies and the CLOSURE I trial did not allow direct comparisons of the various types of adverse events between study designs. Overall, it was evident that atrial fibrillation and major vascular complications were much more common events in CLOSURE I compared to the observational studies (Supplemental Table 3).

To explore for differences in the efficacy of different types of medical treatment for stroke prevention between observational and randomized data, we compared the IRR of anticoagulants versus antiplatelets for stroke prevention estimated from the subgroup analysis for CS patients with PFO of the PICSS RCT (0.53; 95% CI, 0.18–1.70) to the summary estimate of 8 observational studies (0.33; 95% CI, 0.10–1.11). No statistically significant difference was found (P for difference=0.585). Although the PICSS subgroup result was not statistically significant, its point estimate was consistent with the summary of observational studies; by considering the totality of available evidence, a significant result favoring anticoagulants was found (IRR=0.42; 95% CI, 0.18–0.98). Similar results were obtained for other outcomes (total cerebrovascular events and TIA, data not shown).

Discussion

This study summarizes quantitatively the observational evidence on the safety and efficacy of PFO closure and medical management for the secondary prevention of paradoxical embolism and puts observational results in context with the available randomized data. The findings are notable for the sheer volume of observational data: currently, the number of patients included in observational studies is 10-fold larger than the number of patients enrolled in the CLOSURE I trial (Figure 2). Outcome rates of ischemic events among patients treated with mechanical closure were [dsim]4-fold higher in the randomized study compared to the observational data. More importantly, observational and randomized evidence point to different conclusions regarding treatment efficacy of closure. Case series and nonrandomized comparisons have long suggested that closure is a highly efficacious procedure and have led to the rapid adoption of this intervention by some clinical practitioners. In contrast, the randomized trial failed to identify any statistically significant difference between closure and medical therapy.

The discrepancy between the findings of these 2 types of studies may be a reflection of inherent limitations of observational evidence. It is well-appreciated that confounding by indication can create noncomparable populations in the medical and closure treatment arms, in terms of the underlying risk of the outcomes of interest for the patients. Thus, nonrandomized groups of patients may vary in the distribution of factors that determine both the likelihood that the index event was PFO-related and also the risk of a recurrent event. ⁷⁶ Further, traditional methods of multivariate adjustment are not expected to be useful in these studies because outcomes are exceedingly rare; analyses based on 3 studies that reported adjusted treatment effects did not lead to different inferences compared to unadjusted analyses of all studies. More robust methods of adjustment for confounders (such as propensity score-based approaches) ⁷⁷ were not applied in the observational studies included. Although our meta-regressions did not reveal any heterogeneity of treatment effects based on patient characteristics, the ability of meta-analyses to examine effect modification using study-level averages of patient characteristics is limited because of ecological biases.⁷⁸

In addition to confounding by indication, several other biases may affect the observational studies and likely played a role in: differential follow-up intensity or dropout rates of invasively versus medically treated patients may result in informative censoring; a nonuniform start time (for example, some delay for the procedure during which a vascular event might occur and be credited to medical therapy) can lead to immortal time bias; ⁷⁹ and differences in the quality and thoroughness of follow-up examinations would result in differential outcome ascertainment. Finally, information bias may operate in this field, if interventionists are less likely to publish unfavorable results, particularly from single-center investigations. In our meta-analysis, we found evidence both of differential outcome ascertainment (higher probability of being followed-up by a neurologist in medically treated compared to closure-treated patients) and of publication bias among closure studies.

However, RCT may fail to reject the null hypothesis for numerous reasons, with limited statistical power as a primary concern. The outcome rate in the medical therapy arm of the CLOSURE I trial was lower compared to previously published observational studies of medical therapy, suggesting that the baseline risk in the trial population may have been lower. However, outcome rates in the closure arm were 4-times higher compared to previous estimates from observational studies. Although this may be attributable to biased or unrealistic estimates in the observational studies, it may also point to a failure to select patients with PFO-related events in the CLOSURE I trial. This may, in part, be attributable to a reluctance to randomize those patients with CS who appear most likely to have had a PFO-related event ^80,81 (ie, patients with clinical indicators of paradoxical embolism, such as large shunt, associated atrial septal aneurysm, Valsalva maneuver at onset of stroke, or absence of any conventional stroke risk factors) and preference to treat these patients with percutaneous closure off-protocol. Idiosyncratic effects of the STARFlex device, such as in situ thrombosis or other mechanical complications, may have also increased the outcome rates in the intervention arm. Inconsistent and inadequate reporting of periprocedural complications and of device-specific outcome data in the observational literature limits our ability to evaluate this possibility. Finally, the benefits of closure may become apparent only with longer follow-up, because this intervention has an increased risk of periprocedural complications and is expected to prevent the occurrence of relatively infrequent outcome events. ⁸² The 2-year follow-up of CLOSURE I may be inadequate to fully capture the benefit of the intervention and reassessment after additional person-time has accumulated may be warranted.

Whereas previous meta-analyses and systematic reviews of treatment strategies for patients with PFO who have experienced CS have been performed, these were completed before the majority of the data included in this analysis were available and therefore were based on a much less substantial body of evidence. Additionally, previous studies drew inferences on the effectiveness of closure based on indirect comparisons of univariate pooled estimates of IR, 3, 6, have included small numbers of studies, 2, 6, have pooled subjects across studies, which by combining groups of unequal samples can potentially lead to flawed results (Simpson paradox), 2 and have ignored the distinction between comparative and noncomparative studies or used statistical approximations 7 that are inappropriate for analyses of rare events. Our work used more appropriate statistical methods, thus providing updated direct and indirect estimates on the effectiveness of closure, enabling a direct juxtaposition of estimates of closure efficacy from observational and randomized studies, which were shown to be discrepant. Nonetheless, our meta-analysis is limited by the level of detail reported in the primary studies, and the fact that we did not have access to individual patient data. We therefore could not, for example, analyze adverse events or examine the effects of particular closure devices. However, we were able to examine the effect of type of medical treatment on outcomes within a small subset of studies with available data; treatment with anticoagulation was consistently associated with lower IR of recurrent events compared to treatment with antiplatelets.

This comprehensive evidence synthesis of studies investigating secondary stroke prevention strategies in patients with CS and PFO highlights a striking difference between what was inferred for the effect of closure versus medical treatment from observational studies and what was found in the first RCT designed to address this question. Although CLOSURE I does not provide a definitive answer (the CI of the treatment effect was wide and cannot exclude effect sizes that may be clinically relevant), it does call into question the validity of the large body of observational evidence. Based on the CLOSURE I trial results, we believe that closure cannot be currently considered the treatment of choice for presumed paradoxical embolism. Our results reinforce the clinical equipoise for closure versus medical treatment and suggest that randomization in ongoing trials in the field (RESPECT, PC-Trial, REDUCE, CLOSE) should be encouraged. ²

In contrast to the results regarding the comparative effectiveness of device closure versus medical therapy, the observational and randomized data for the comparative effectiveness of anticoagulants (mostly warfarin) versus antiplatelets were more consistent. The only relevant randomized study identified was a subgroup analysis in patients with CS and PFO from the PICSS study, ⁵⁹ which found a nonsignificant benefit of warfarin over aspirin. This result was statistically consistent with the meta-analysis of 8 observational studies on the same comparison; combining observational and randomized evidence resulted in a significant result in favor of anticoagulants. Although there are data showing the differential selection of anticoagulants over antiplatelets in clinical studies ^2,85 and, consequently, confounding by indication may operate in this field, RCT and observational efficacy data appear to be consistent. Thus, whereas the RCT evidence alone would be clearly insufficient to draw any conclusion, the totality of the evidence (ie, including observational studies) suggests that anticoagulants may be superior to antiplatelets for secondary prevention (although the strength of evidence would be considered low to moderate by conventional criteria for assessing the overall strength of a body of evidence). ⁸⁶

Despite methodological challenges, high-quality observational studies are a critical component of comparative effectiveness research because they can address issues that are otherwise difficult or impossible to study in randomized trials; empirical evidence supports that observational studies typically agree with the results of randomized trials. ⁷⁷ For the comparison of anticoagulants versus antiplatelets for secondary prevention in CS and PFO, observational studies, together with limited randomized data, offer at least some support for anticoagulation. Yet, the case of PFO closure highlights the risks of over-reliance on observational evidence, particularly when RCT are feasible and ethically justifiable. Although the [mt]7000 CS patients who had their PFO closed in the included observational trials represent only a small fraction of patients so treated, if this sample instead had been recruited into well-designed RCT, then it is likely that the best way to prevent recurrent strokes in this population would now be known.