Are “Treatment” Bare Metal Stents Superior to “Control” Bare Metal Stents? A meta-analytic approach

Abstract

Background

It has been suggested that the benefits of drug-eluting stents compared to bare metal stents (BMS) have been over-estimated in part because target lesion/vessel revascularization (TLR/TVR) rates in the BMS control group of these trials were spuriously high.

Methods

We used meta-analytic techniques to systematically compare clinical event rates among patients treated with BMS in trials where BMS were the experimental (BMS_experimental) rather than the control (BMS_control) intervention. MEDLINE searches were performed to identify eligible randomized trials comparing either drug-eluting stents with BMS_control, or BMS_experimental with balloon angioplasty in patients with non-acute coronary artery disease. Trial characteristics and 6 to 12 month rates for death, myocardial infarction, TLR/TVR and major adverse cardiac events (MACE) were extracted and assessed.

Results

Eligible trials yielded 50 BMS cohorts: 19 in the BMS_control group (4 046 patients) and 31 in the BMS_experimental group (5 068 patients). Summary death and infarction rates did not differ between groups. The summary TLR/TVR rates were 16.2% (95% confidence interval, CI: 13.5, 19.3) versus 13.8% (95% CI: 12.0, 15.7) in BMS_control versus BMS_experimental groups, respectively (p=0.15). Among 39 BMS cohorts with ≤250 patients, TLR/TVR rates were significantly higher in BMS_control versus BMS_experimental groups (18.9% [95% CI: 16.0, 22.2] versus 13.7% [95% CI: 11.5, 16.3], p=0.01). There were no between-group differences among larger BMS cohorts (p=0.98).

Conclusions

While overall clinical event rates did not differ in the BMS_control and the BMS_experimental groups, a higher rate of TVR/TLR was seen in the BMS_control group among smaller trials.

Introduction

Over the last two decades, there have been several major technological innovations in the management of non-acute coronary artery disease. Over this period, percutaneous balloon angioplasty has been shown to improve outcomes compared to medical therapy alone;¹ bare metal stents (BMS) have been shown to improve outcomes compared to balloon angioplasty;² and, in turn, drug eluting stents (DES) have been shown to improve outcomes compared to BMS.³

However, the apparent progress has been largely based on non-uniform and composite outcomes that typically combine hard endpoints (such as death and myocardial infarction) with softer endpoints (such as need for revascularization). Indeed, meta-analyses have largely ruled out substantial improvements in myocardial infarction or mortality rates in the aforementioned succession of intervention comparisons.^2,4,5

In the case of stents for coronary disease, technological innovations have been adopted mainly on the basis of softer endpoints: Benefits have been limited to a sequential decrease in the need for target lesion/vessel revascularization (TLR/TVR), an outcome much more common than death or infarctions. TLR/TVR is used in the context of protocol-driven angiography in clinical trials. It does not directly reflect patient response to clinical care, and is sensitive to physician judgment (and potentially to bias). A recent narrative review suggested that the TLR/TVR rate in the BMS control group of several DES trials may have been higher than expected compared to some other contemporary BMS trials.⁶

Such an observation might signal a more general, previously undescribed tendency toward “control rate inflation” with soft outcomes – something that might bias trial results in favor of the experimental intervention. We sought to systematically examine this possibility the rate of TLR/TVR in trials in which BMS were the experimental (BMS_experimental) intervention, compared to trials in which they were the control (BMS_control) intervention.

Methods

Database formation and study eligibility

We utilized two comprehensive meta-analyses performed by the same team^2,3 as a starting basis for our search. The meta-analyses evaluated randomized controlled trials comparing BMS with balloon angioplasty², and DES with BMS³ in patients with coronary artery disease who were not treated for acute coronary syndromes. These two meta-analyses were updated from the end of their coverage period onwards, using similar MEDLINE search strategies (last search 08/21/2006) and the same eligibility criteria. Later-published meta-analyses from different teams were also searched for additional eligible randomized trials.^5,7

Eligible trial reports randomized more than 10 patients per arm and evaluated the clinical outcomes of death, myocardial infarction, TLR/TVR, and major adverse cardiac events (MACE, defined as the first occurring of the aforementioned outcomes) at follow up times between 6 and 12 months. Outcomes from the latest available time point (within the window of 6 to 12 months) were collected.

Data extraction

From each trial, we extracted information describing patient characteristics (proportion of males, mean age and age range, and proportion with diabetes); trial characteristics and design (follow-up time; year that enrollment started; presence of protocol-driven co-treatments with antiplatelet agents; presence and timing of protocol-driven follow-up angiography; and type of bare metal stents); and the aforementioned clinical outcomes, including whether MACE was based on target lesion or target vessel revascularization.

Meta-analyses

TLR (or, where not available, TVR) was the primary endpoint of this study. Trial arms that used BMS were considered as individual prospective cohorts, and were classified in two cohort groups, BMS_control and BMS_experimental. Summary event rates per group were calculated with random effects meta-analyses using the DerSimonian and Laird method⁸ after a logit transformation, and were compared with a z-score, as previously described.⁹ We tested for heterogeneity with Cochran’s Q (considered significant at p<0.10), and quantified it’s extent with I².¹⁰ I² represents the proportion of between-study variability that is attributable to heterogeneity rather than chance.¹⁰

Subgroup and Sensitivity Analysis

Based on prior literature suggesting greater treatment effects in smaller versus larger trials,^9,11,12 subgroup analysis was used to test the hypothesis that smaller trials might be more vulnerable to a control rate inflationary bias, should one exist. We arbitrarily defined as smaller BMS cohorts with ≤250 patients (i.e., we defined a trial of 500 people or less as being small, and assumed 1:1 patient allocation). We also assessed whether blinding of the patients or the outcome assessors differentiated the findings. In sensitivity analyses we excluded BMS cohorts from trials specifically targeting patients with difficult lesions, i.e. complex lesions or lesions in small arteries (≤3.0mm diameter), since these might have especially high revascularization rates.

Meta-regression

We used random-effects¹³ and fixed effects meta-regressions to adjust for the following trial-level characteristics: follow-up length, initial year of patient enrollment (as a proxy for evolution in patient care), whether the trial specifically targeted patients with difficult (i.e. complex, long or small vessel) lesions (yes/no) and trial size (square-root-transformed).

The authors had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Characteristics of the treatment and control cohorts were compared with Mann-Whitney tests. All p-values are two-tailed and considered significant at the 0.05 level. Analyses were conducted in Intercooled Stata 8.2 (Stata Corporation, College Station, TX, US).

Results

Based on the selected meta-analyses and the updated searches 74 papers were retrieved in full text, and 50 were deemed eligible: Thirty-one describing BMS_experimental cohorts (published from 1994 to 2005, 5 068 patients) and 19 describing BMS_control cohorts (published from 2001 to 2006, 4 046 patients) (see Supplementary Table). Cohort sample sizes, proportion of male enrollees and proportion of enrollees with diabetes did not differ beyond chance across the two groups (Table 1). On median, the average patient age was 62 years versus 60 years in the BMS_experimental and BMS_control cohorts, respectively. BMS_control cohorts had longer median follow-up times (median 9 versus 6 months), and belonged to trials that started enrolling patients in more recent years (median 2001 versus 1995) (Table 1). All trials had protocol-driven angiography.

Table 1.

Comparison of trial and arm characteristics of the two cohort groups

Characteristic	BMScontrol	BMSexperimental	p
Number of cohorts (patients)	19 (4 046)	31 (5 068)	NA
Patients randomized to BMS	140 (50, 263)	125 (57, 204)	0.69
Publication year, median (IQR)	2004 (2003, 2005)	2000 (1998, 2000)	<0.01
Enrollment initiation year, median (IQR) *	2001 (2000, 2002)	1995 (1992, 1996)	<0.01
Follow up (months)	9 (9, 12)	6 (6, 12)	0.01
Proportion of males, (%)	73 (70, 80)	77 (72, 82)	0.53
Proportion of diabetics, (%)†	21 (16, 25)	17 (11, 21)	0.10
Mean participant age (years)‡	62 (62, 63)	60 (58, 61)	<0.01
Protocol driven angiography	19/19	31/31	1.00
Stent strut thickness ≤0.1mm||	1/11	17/27	<0.01

^*five BMS_control and six BMS_experimental cohorts did not provide this information

^†three BMS_experimental cohorts did not provide this information

^‡one BMS_control and two BMS_experimental cohorts did not provide this information

^||could not be ascertained in eight BMS_control and four BMS_experimental cohorts

BMS: bare metal stents; IQR: interquartile range

We were able to determine strut thickness on 38 of the 50 trials. In the BMS_control group 1 of 11 trials (9%) used only stents with strut thickness <0.1mm. The corresponding proportion in the BMS_experimental group was 17 out of 27 trials (63%) (p<0.01 for this comparison).

Twelve of the 50 trials presented data for both TLR and TVR rates (five in the BMS_experimental and seven in the BMS_control group). Among them, the median difference (TVR minus TLR) in the outcome rate was 1.1% (IQR: 0.0 to 3.0%). TLR was not available in 18 of 31 BMS_experimental trials and 2 of 19 BMS_control trials, and was substituted with TVR.

There was extensive between trial (within group) heterogeneity in the rate of TLR/TVR (See Supplementary Figure) (p<0.001, I²>67% for both cohort groups), indicating much greater between study variability than would be expected by chance alone. Despite a trend toward more frequent outcomes in BMS_control cohorts, the summary rates did not differ significantly between the two groups: 16.2% (95% confidence interval, CI: 13.5, 19.3%) in the BMS_control and 13.8% (95% CI: 12.0, 15.7%) in the BMS_experimental group (p = 0.15) (Figure 1). MACE rates, which are grossly determined by TLR/TVR rates, were 20.1% (95% CI: 17.0, 23.5%) in the BMS_control cohorts and 17.5% (95% CI: 15.4, 19.9%) in the BMS_experimental cohorts (p=0.20) (Figure 1), again with extensive between study heterogeneity (p<0.001, I²>69% for both groups).

Figure 1.

Summary outcome rates in BMS_control versus BMS_experimental cohorts

BMS: bare metal stents; MACE: major cardiac adverse events; MI: myocardial infarction; TLR/TVR: target lesion/vessel revascularization. The figure illustrates the summary event rates (random effects meta-analysis) in the two groups (BMS_control versus BMS_experimental) for death, MI, TLR/TVR, and MACE. Dots stand for the point estimate of the summary rates and horizontal lines represent the 95% confidence intervals. P-values are for between group differences and are unadjusted for multiple comparisons.

Subgroup and sensitivity analysis

In the 9 larger cohorts (>250 patients in BMS arm), there was no difference in TLR/TVR rates in experimental compared to control treatment groups (p=0.98) (Table 2; Figure 2). However, in the 39 smaller cohorts a substantial and statistically significant effect was found, with almost 40% higher outcome rates in BMS_control compared to BMS_experimental (18.9% versus 13.7%; p=0.01). At the same time, small BMS_control cohorts had significantly higher rates that larger BMS_control cohorts (18.9% vs 13.1%, p=0.04). Results for the overall analysis and the subgroup analysis remained consistent when we excluded BMS cohorts from trials targeting patients with small artery or complex lesions (Table 2).

Table 2.

Target Lesion/Vessel Revascularization: subgroup and sensitivity analyses

Comparison or subgroup	BMScontrol		BMSExperimental		p
	Studies (Pts)	TLR/TVR, (%) [95% CI]	Studies (Pts)	TLR/TVR, (%) [95% CI]
Overall	19 (4 046)	16.2 (13.5, 19.3)	31 (5 068)	13.8 (12.0, 15.7)	0.15
Excluding difficult lesions*	15 (2 958)	14.8 (11.8, 18.4)	22 (3 903)	13.0 (11.0, 15.3)	0.35
Cohort sample size
N>250	5 (2 528)	13.1 (9.4, 18.0)	4 (1 836)	13.1 (11.3, 15.2)	0.98
N≤250	14 (1 518)	18.9 (16.0, 22.2)	25 (3 071)	13.7 (11.5, 16.3)	0.01
N≤250 and excluding difficult lesions*	11 (1 006)	17.4 (14.4, 20.9)	18 (2 067)	12.7 (10.0, 16.1)	0.04

BMS: bare metal stents; CI: confidence interval; Pts: number of patients; TLR/TVR: Target lesion/vessel revascularization

^*Trials specifically enrolling patients with complex lesions or small artery lesions, as described in the supplementary Table.

Figure 2.

Summary outcome rates for TLR/TVR in BMS_control versus BMS_experimental cohorts according to sample size. Layout is similar to Figure 1.

None of the 31 trials in the BMS_experimental group blinded outcome assessors. Summary TLR/TVR rates did not differ beyond chance between the 10 BMS_control cohorts where outcome assessment was blinded and the remaining 9 (16.1% versus 16.4%, respectively; p=0.94). The same was true for the comparison of each subgroup against BMS_experimental cohorts. Among small BMS cohorts (≤250 patients), blinded BMS_control cohorts still had higher summary TLR/TVR rates from BMS_experimental cohorts, but the difference was not formally significant (p=0.09).

Meta-regression

Meta-regression yielded a similar, statistically non-significant estimate of the overall BMS_control versus BMS_experimental effect on TLR/TVR rates (p=0.16). Length of follow-up and cohort sample size did not have a significant effect on the rate of TLR/TVR in the meta-regressions (Table 3) using random effects analyses. TLR/TVR rates increased with later enrollment initiation and when patients with difficult lesions were studied. However, controlling for these trial-level characteristics did not reveal any differences in outcome rates between the BMS_control and BMS_experimental groups overall (Table 3). The only significant effect in the random-effects meta-regression confirmed the aforementioned group-by-cohort size interaction (i.e. that different between group [BMS_control versus BMS_experimental] outcome rates are observed between smaller, but not between larger cohorts).

Table 3.

Meta-regressions of target lesion/vessel revascularization on various covariates

Effect of Group (BMScontrol/BMSExperimental) on TLR/TVR adjusting for	Covariate* effect direction	p for covariate* effect REM (FEM)	p between subgroups REM (FEM)
unadjusted	NA	NA	0.16 (0.11)
Follow-up duration (continuous)	↑ TLR/TVR	0.72 (0.82)	0.27 (0.25)
Initial year of enrollment†	↑ TLR/TVR	0.04 (0.05)	0.45 (0.42)
Presence of difficult lesions‡ (yes/no)	↑ TLR/TVR	0.05 (0.04)	0.15 (0.13)
Cohort sample size (square root transformation)	↓ TLR/TVR	0.16 (0.01)	0.10 (0.03)
Cohort sample size, >250 patients (yes/no)	↓ TLR/TVR	0.18 (0.02)	0.10 (0.04)
Cohort sample size, >250 patients (yes/no) and interaction of cohort size with Group	NS	NS	0.05 (0.02)

NA: Not applicable; NS: not shown; REM/FEM: Random/Fixed effects meta-regression; TLR/TVR: Target lesion/vessel revascularization.

Direction of effects in the meta-regression that incorporated the interaction terms are not shown because the interpretation is more complex.

^*Refers to independent variables other than subgroup participation

^†Missing values for year of initial enrollment were imputed using information from the year of publication

^‡Trials specifically enrolling patients with complex lesions or small artery lesions, as described in the supplementary Table

Discussion

We did not find evidence for control rate inflation affecting the rates of TLR/TVR in our main analyses. Although a visual inspection of the summary estimates in Figure 1 across the BMS groups might imply higher TLR/TVR or MACE rates in the BMS_control group of cohorts, this trend was relatively small compared to the substantial between trial outcome rate heterogeneity. However, a clinically substantial and statistically significant effect was seen among BMS cohorts from smaller trials, where TVR/TVL outcome rates were approximately 40% higher in the BMS_control cohorts compared to the BMS_experimental cohorts. This could have a meaningful influence on the estimated treatment effect in these trials.

There are several potential explanations for this apparent inflation of BMS_control rates among smaller trials, including chance. We used an arbitrary, non-data-driven cutoff in our subgroup analysis. However, we acknowledge that findings from subgroup analyses often result from the interplay of chance or confounding, and should be considered with due caution.¹⁴ Second, because patients were not randomized between the BMS_control and BMS_experimental groups, the apparent control rate inflation may be ascribed to clinical heterogeneity across the different populations (e.g. higher risk lesions or patients in the smaller BMS_control compared to BMS_experimental groups),¹⁵ or differences in design features of the trials. Most notably, DES trials (BMS_control) had on average longer follow-up. However, on average, the magnitude of the effect of follow-up time on outcome rate was both negligible and not statistically significant. It is possible that the effect of follow up is masked by the ubiquity of protocol driven angiography, which actively detects lesions that may need revascularization. Moreover, other differences would have biased toward lower event rates in the BMS_control versus the BMS_experimental group: In the former, the modal duration of dual-antiplatelet therapy was 6 months versus only 1 month; and there was more frequent use of the more restrictive endpoint of target lesion instead of target vessel revascularization (90% versus 42% of trials, in the two groups respectively).

Third, there was heterogeneity in the type of BMS stents used. For example, we found a higher frequency of stents with thick struts among the BMS_control cohorts. This observation has been made before, when it was argued that the DES were compared against a “straw man”⁶ (namely the thick strut BMS) rather than thin strut BMS (which might be more effective¹⁶). However, strut thickness alone seems unlikely to account for the difference in TLR/TVR rates among smaller BMS_control cohorts, since all larger BMS_control cohorts used thick strut stents and had very similar TLR/TVR rates to the BMS_experimental cohorts.

Finally, it is possible that biases (systematic errors) may have influenced outcomes rates. TLR/TVR is a physician-determined outcome, and it is not impossible that it might be differentially ascertained by assessors with strong prior beliefs on the effectiveness of different types of stents. In our case, blinding did not appear to influence outcome rates in the BMS_control cohorts, suggesting that systematic errors are not the most plausible explanation. Nevertheless, the potential for biases should always be considered when evaluating clinical research.

Whatever its cause, control rate inflation may provide a much-needed opportunity to demonstrate yet another advancement, another step in a sequence of ever successful interventions. Small trials were found to provide the wider margin of opportunity in our evaluation. Small trials are more susceptible to the interplay of chance and systematic errors, as several lines of empirical evidence suggests. Estimates from small trials differ beyond chance from those of larger trials 10–25% of the time,^11,12 and usually claim stronger and more impressive effects.^9,11 Inflated control rates in some of the small DES trials may have biased toward larger, more positive effect sizes, as the dependency of effect sizes on control rates is well appreciated in the methodological literature.¹⁷

Continual innovation leads to progressively diminishing control rates and diminishing control rates leads to diminishing benefits for new therapeutic advances. Indeed, sustained continual innovation theoretically requires an inexhaustible control rate; one could hypothesize that this might exert subtle pressure for higher event rates in control interventions that might manifest in trial design or trial conduct, resulting in an exaggeration of the degree of true progress. The existence and importance of a general tendency toward control rate inflation deserves systematic (not just anecdotal) study. Our evaluation did not find evidence that the control rate was inflated when all DES trials were assessed. However, the accumulated evidence suggests a higher than expected TLR/TVR control rate in smaller trials. While these results do not challenge the overall superiority of DES for reducing TLR/TVR compared to BMS, they add to others that caution against early enthusiasm for new technologies based on small trials, especially when this is based on soft rather than hard endpoints.