Reporting data from meta-analysis: snapshot of a moving target

This commentary refers to ‘Meta-analyses of moving targets’, by C. Berry, https://doi.org/10.1093/eurheartj/ehab359; and the discussion piece ‘Strengths and limitations of meta-analyses’, by C. Berry, https://doi.org/10.1093/eurheartj/ehab682.

We agree with Dr Berry¹ that the results of meta-analyses are a moving target; that their validity is dependent on including all relevant studies; and that publication bias may prejudice towards a positive outcome. However, we disagree with his characterization from the confidential peer review process that the reviewers highlighted and prompted inclusion of new trial data not previously identified, described as author error. BIOSTEMI 2-year data² had not been published or presented at the time of our original submission and none of the reviewers recommended its inclusion. It was identified during an updated systematic search and thus added during revision. TALENT 2-year data were mentioned in peer review but at initial submission had only been published as an abstract, including only two endpoints. We, therefore, relied on 1-year data for most analyses, including mortality. When the 2-year TALENT results were published,³ we updated our analysis accordingly. PRISON IV 3-year data (described as missing in the editorial) were in fact included (reference 21) and utilized for longer follow-up analyses. BIOFLOW IV 4-year data were not used because only a single endpoint was published in the abstract.

The change in statistical significance for all-cause mortality between submissions highlights an important lesson in interpreting borderline findings. To dichotomize an effect as significant or non-significant based on a 0.05 P-value cut-off discards important information from the point estimate and its 95% confidence intervals (CI) on the effect size and precision, as well as issues of heterogeneity. Between revisions, the relative risk (95%CI) for all-cause mortality changed from 1.14 (1.01–1.30), P = 0.034, to 1.11 (0.98–1.16), P = 0.11. A reasonable interpretation is that these effect sizes are similar with overlapping confidence intervals; drawing firm conclusions regarding mortality from either dataset purely based on P-value thresholds should be avoided. We noted in our manuscript⁴ that ‘… the number of events required to shift the P-value to beyond the threshold for statistical significance is estimated to possibly only be 4’ and ‘Longer-term follow-up from the present trials, and ideally additional randomized studies are necessary to clarify this uncertainty’ as the analysis was underpowered for rare events such as death despite its sample size of 20 701 patients. Finally, while a call for more data to clarify this uncertainty is conservative and ‘academic’, we believe it unlikely that ultrathin-strut drug-eluting stents (DES) result in greater mortality given the numerically fewer MIs, stent thrombosis, and target-lesion revascularization observed with these devices, i.e. an increase lacks mechanistic plausibility.

The editorial raises broader questions about the place of study-level meta-analysis for informing practice guidelines and clinical decision-making. We believe that meta-analyses are useful to examine low-frequency outcomes that are under-powered from individual trials. Meta-analytic methods also increase the precision and confidence of effect sizes from trials which are appropriate to pool together (a frequently abused prerequisite). Individual-patient-data pooled analyses are optimal for assessing relationships between the time course and predictors of events but are resource intensive and time consuming.

Finally, just because targets are moving does not mean that efforts should not be made to repeatedly appraise the ever-expanding totality of available evidence to provide the best overall estimates of therapeutic safety and effectiveness, as demonstrated from the present outcomes from serial meta-analyses of ultra-thin strut DES.

Conflict of interest: M.V.M. was supported by a grant from the National Institutes of Health/National Heart, Lung, and Blood Institute to Columbia University Irving Medical Center (T32 HL007854). G.W.S. has received speaker honoraria from Cook and Terumo; has served as a consultant to Valfix, TherOx, Vascular Dynamics, Robocath, HeartFlow, Gore, Ablative Solutions, Miracor, Neovasc, V-Wave, Abiomed, Ancora, MAIA Pharmaceuticals, Vectorious, Reva, and Matrizyme; and owns equity/options in Ancora, Qool Therapeutics, Cagent, Applied Therapeutics, the Biostar family of funds, SpectraWave, Orchestra Biomed, Aria, Cardiac Success, the MedFocus family of funds, and Valfix. All other authors have reported that they have no relationships relevant to the contents of this paper to disclose.

Data availability

No new data were generated or analysed in support of this research.