Meta-analyses of moving targets

This editorial refers to ‘Long-term follow-up after ultrathin vs. conventional 2nd-generation drug-eluting stents: a systematic review and meta-analysis of randomized controlled trials’, by M. V. Madhavan et al., doi:10.1093/eurheartj/ehab280.

Percutaneous coronary intervention (PCI) is indicated for haemodynamicallysignificant coronary artery disease in the presence of limiting angina or angina equivalent when symptoms are inadequately controlled by optimised medical therapy,¹ and second generation drug eluting stents (DES) with a strut thickness >70 μm are the current standard of care. Whilst PCI is an effective treatment, patients may experience recurrent ischaemic events, including angina or an acute coronary syndrome.² Although there is a wide differential diagnosis,² target lesion failure due to neointima formation and, rarely, stent thrombosis, is an avoidable complication. The stent can be causally-implicated, and advances in stent technologies, include reducing the strut thickness, are intended to limit these adverse events. Ultrathin-strut thickness (≤70 μm) represents one of the latest advances in stent technology, and evidence of clinical effectiveness compared to second generation DES is necessary to support a change in clinical practice.

In 2018, a study-level meta-analysis of 10 clinical trials involving 11,658 patients reported that newer-generation ultrathin strut DES were associated with a 16% reduction in target lesion failure defined as a composite of cardiovascular death, target vessel MI, or ischemia-driven target lesion revascularization (TLR) evaluated at 1-year follow-up (relative risk, 0.84; 95% CI, 0.72–0.99) driven by less myocardial infarction (relative risk, 0.80; 95% CI, 0.65–0.99).³ Since then, the clinical follow-up in these trials has been extended and new data have been reported.

Now, Madhavan et al  ⁴ have reported an updated systematic review and study-level meta-analysis. The authors followed PRISMA group guidelines.⁵ The prespecified primary endpoint was long-term target lesion failure, a composite of cardiac death, myocardial infarction (MI), or clinically-driven target lesion revascularisation (CD-TLR). Secondary endpoints included the components of target lesion failure, stent thrombosis, and all-cause mortality. There were 16 eligible trials including 20,701 randomised patients. The median duration of follow-up was 2.5 years. The primary outcome of TLF was reduced in the ultrathin-strut DES. However, the ultrathin-strut DES group was associated with a numerical increase in the risk of all-cause mortality at latest follow-up (relative risk 1.11 [0.98, 1.26], P = 0.114), that was not statistically significant.

The peer review journey followed an interesting course. The article was submitted as a FastTrack publication linked to a Late Breaking Clinical Trial presentation at EuroPCR 2021. In the initial version of the manuscript there was a statistically significant increase in all-cause mortality associated with the ultrathin-strut DES group. This was an unexpected finding that at first sight was difficult to explain. Feedback from the reviewers highlighted some new trial data that had not been included in the initial analysis i.e., the 2-year follow-up results from the TALENT⁶ and BIOSTEMI⁷ trials. Other limitations included the use of summary results drawn from trial publications rather than using individual patient data, the limited number of studies included in the analysis, their variable follow-up duration of some of the trials, the inability to distinguish between the type of MI (e.g., STEMI vs. NSTEMI), and the lack of information on the composite outcome of ‘death or MI’. These limitations are consistent with what might be expected in this form of analysis. The authors responded with additional analyses to address the reviewers’ comments, and the final results are described in the manuscript.

However, in the final analysis, the signal on mortality had changed. There was no statistically significant association between the ultrathin-strut group and all-cause mortality. The authors explained that inclusion of the longer term follow-up data from TALENT⁶ and BIOSTEMI⁷ trials were relevant, and only a few studies in their cohort reported deaths beyond one year. Further, in the TALENT trial, all-cause mortality in the control stent arm (0.6% at 12 months) was lower than expected.⁸ Specifically, 14/720 deaths occurred in the ultrathin-strut DES group and 4/715 deaths occurred in the thin strut DES group. However, when the two-year results for the TALENT trial were included, all-cause mortality occurred in 18/720 patients in the ultrathin-strut DES group and 21/715 in the thin strut DES group. Therefore, all-cause mortality was numerically lower in the ultrathin-strut group at 2 years. Finally, with no statistically significant difference in all-cause mortality between the groups the reader might conclude the results are clear. Or are they? Two trials (PRISON-IV (3-year follow-up)⁹ and BIOFLOW IV (4 year follow-up)¹⁰) were not included, however, their results are only available in a summarized (abstract) form, hence, not fully peer reviewed.

In the final analysis,⁴ including two new studies with longer term follow-up,^{6  ,  7} there is no longer a statistically significant association between ultrathin-strut DES type and all-cause mortality. Nonetheless, there is a directionally distinct association between the ultrathin stent group and mortality (cardiac, non-cardiac and all-cause). The lower limit of the 95% confidence interval for relative risk of all-cause death is 0.98. This result is statistically fragile. In other words, just a few more deaths in the DES group, perhaps only 4, would shift the confidence interval to meet or cross unity. However, calculation of the fragility index, or reverse fragility index, is not obvious since the included trials should have 1:1 randomisation between the arms,¹¹ which was not consistently the case here. In my opinion, the authors appropriately conclude that the directional association with all-cause death cannot be dismissed as a chance finding. It deserves further study with longer term follow-up from the present trials, and ideally further randomised trials.

What is the big picture? This case highlights some fundamental limitations of meta-analysis. Firstly, the validity of the analysis is dependent on including all relevant studies. Negative publication bias by trialists, or inadvertent error by those leading the meta-analysis, may lead to omission of relevant data. Secondly, researchers undertaking study-level meta-analysis are not using source data that they have generated (and take responsibility for), rather, they are including the results that other investigators have reported. Thirdly, clinical trials are a moving target. A primary endpoint is defined in time, but extensions to the follow-up period may, or may not be predefined, and even if predefined, additional factors e.g., funding, may influence future delivery and publication. The results of meta-analyses are, therefore, updateable, and the clinical implications may change over time.

Clinical practice guidelines cite meta-analyses as underpinning a Level A classification of evidence.¹ The limitations of meta-analyses outlined here should give pause for thought on what form (or standard) of meta-analysis can be accepted as contributing to a Level A recommendation. What might the criteria be? Guideline committees should consider whether the authors are experts in the field, whether there are multiple meta-analyses published on a given subject, and if so, are the results coherent? Are patient-level analyses available? Have all of the relevant trials been included? For a given subject, have the clinical trials completed their pre-defined follow-up, or are the targets still moving? At least practice guidelines are updatable to take account of these issues.

Funding

CB acknowledges research support from the British Heart Foundation (PG/17/2532884; FS/17/26/32744; RE/18/6134217) and Medical Research Council (MR/S005714/1).

Conflicts of interest: C.B. is employed by the University of Glasgow which holds consultancy and/or research agreements with companies that have commercial interests in the diagnosis and treatment of ischaemic heart disease. The companies include Abbott Vascular, AstraZeneca, Boehringer Ingelheim, GSK, HeartFlow, Menarini Farmaceutica, Neovasc, and Siemens Healthcare.