Risk of Bias in Randomized Clinical Trials Comparing Transcatheter and Surgical Aortic Valve Replacement: A Systematic Review and Meta-analysis

Fabio Barili; James M Brophy; Daniele Ronco; Patrick O Myers; Miguel Sousa Uva; Rui M S Almeida; Mateo Marin-Cuartas; Amedeo Anselmi; Jacques Tomasi; Jean-Philippe Verhoye; Francesco Musumeci; John Mandrola; Sanjay Kaul; Stefania Papatheodorou; Alessandro Parolari

doi:10.1001/jamanetworkopen.2022.49321

. 2023 Jan 3;6(1):e2249321. doi: 10.1001/jamanetworkopen.2022.49321

Risk of Bias in Randomized Clinical Trials Comparing Transcatheter and Surgical Aortic Valve Replacement

A Systematic Review and Meta-analysis

Fabio Barili ^1,^2,^✉, James M Brophy ³, Daniele Ronco ⁴, Patrick O Myers ⁵, Miguel Sousa Uva ^6,⁷, Rui M S Almeida ⁸, Mateo Marin-Cuartas ⁹, Amedeo Anselmi ¹⁰, Jacques Tomasi ¹⁰, Jean-Philippe Verhoye ¹⁰, Francesco Musumeci ¹¹, John Mandrola ¹², Sanjay Kaul ¹³, Stefania Papatheodorou ¹, Alessandro Parolari ⁴, for the International Evidence Grading Research Initiative Targeting Transparency and Quality (INTEGRITTY)

¹Department of Epidemiology, Harvard T. H. Chan School of Public Health, Boston, Massachusetts

²Department of Cardiac Surgery, S. Croce Hospital, Cuneo, Italy

³Department of Medicine, McGill Health University Center, Montreal, Quebec, Canada

⁴Department of University Cardiac Surgery, IRCCS Policlinico San Donato, Milan, Italy

⁵Division of Cardiac Surgery, CHUV–Lausanne University Hospital, Lausanne, Switzerland

⁶Department of Cardiac Surgery, Hospital Santa Cruz, Carnaxide, Portugal

⁷Department of Cardiac Surgery and Physiology, Porto University Medical School, Porto, Portugal

⁸University Center Assis Gurgacz Foundation, Cascavel, Paraná, Brazil

⁹University Department of Cardiac Surgery, Leipzig Heart Center, Leipzig, Germany

¹⁰Department of Thoracic and Cardiovascular Surgery, University Hospital of Rennes, Rennes, France

¹¹Department of Cardiac Surgery and Heart Transplantation, San Camillo Forlanini Hospital, Rome, Italy

¹²Baptist Health Louisville, Louisville, Kentucky

¹³Department of Cardiology, Cedars-Sinai Medical Center, Los Angeles, California

Accepted for Publication: November 10, 2022.

Published: January 3, 2023. doi:10.1001/jamanetworkopen.2022.49321

^✉

Corresponding Author: Fabio Barili, MD, PhD, Department of Epidemiology, Harvard T. H. Chan School of Public Health, 677 Huntington Ave, Boston, MA 02115 (fbarili@hsph.harvard.edu).

Author Contributions: Dr Barili and Prof Parolari had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Barili, Myers, Sousa Uva, Almeida, Marin-Cuartas, Anselmi, Tomasi, Verhoye, Musumeci, Kaul, Papatheodorou, Parolari.

Acquisition, analysis, or interpretation of data: Barili, Brophy, Ronco, Tomasi, Mandrola, Kaul, Papatheodorou.

Drafting of the manuscript: Barili, Mandrola, Kaul.

Critical revision of the manuscript for important intellectual content: Barili, Brophy, Ronco, Myers, Sousa Uva, Almeida, Marin-Cuartas, Anselmi, Tomasi, Verhoye, Musumeci, Kaul, Papatheodorou, Parolari.

Statistical analysis: Barili, Kaul, Papatheodorou, Parolari.

Administrative, technical, or material support: Ronco, Myers, Marin-Cuartas, Verhoye.

Supervision: Myers, Sousa Uva, Marin-Cuartas, Anselmi, Tomasi, Parolari.

Conflict of Interest Disclosures: Dr Myers reported serving as secretary general elect for the European Association for Cardio-Thoracic Surgery and secretary for the Cardiothoracic Surgery Network. Dr Anselmi reported receiving personal fees from Abbott Cardiovascular outside the submitted work. Dr Kaul reported receiving personal fees from Boehringer-Ingelheim, NovoNordisk, Bayer, GSK, Abbott Consulting, and AstraZeneca outside the submitted work; and holding equity interest in Johnson & Johnson. No other disclosures were reported.

Group Information: The International Evidence Grading Research Initiative Targeting Transparency and Quality (INTEGRITTY) Collaborators are listed in Supplement 3.

Data Sharing Statement: See Supplement 4.

^✉

Corresponding author.

PMCID: PMC9857525 PMID: 36595294

Key Points

Question

Does randomization protect randomized clinical trials (RCTs) comparing transcatheter aortic valve implantation (TAVI) and surgical aortic valve replacement (SAVR) from biases other than nonrandom allocation?

Findings

This systematic review and meta-analysis of 8 RCTs including 8849 participants and comparing TAVI vs SAVR found substantial overall proportions of deviation from assigned treatment, loss to follow-up, additional procedures, and additional myocardial revascularization together with a systematic selective imbalance in the same direction characterized by significantly lower proportions among participants undergoing TAVI.

Meaning

This study suggests that RCTs comparing TAVI and SAVR show serious methodological imbalances with a common selective pattern, and should be considered at high risk of performance and attrition bias that may affect internal validity.

Abstract

Importance

Recent European Society of Cardiology/European Association for Cardio-Thoracic Surgery (ESC/EACTS) guidelines highlighted some concerns about the randomized clinical trials (RCTs) comparing transcatheter aortic valve implantation (TAVI) and surgical aortic valve replacement (SAVR) for aortic stenosis. Quantification of these biases has not been previously performed.

Objective

To assess whether randomization protects RCTs comparing TAVI and SAVR from biases other than nonrandom allocation.

Data Sources

A systematic review of the literature between January 1, 2007, and June 6, 2022, on MEDLINE, Embase, and Cochrane Central Register of Controlled Trials was performed. Specialist websites were also checked for unpublished data.

Study Selection

The study included RCTs with random allocation to TAVI or SAVR with a maximum 5-year follow-up.

Data Extraction and Synthesis

Data extraction was performed by 2 independent investigators following the PRISMA guidelines. A random-effects meta-analysis was used for quantifying pooled rates and differential rates between treatments of deviation from random assigned treatment (DAT), loss to follow-up, and receipt of additional treatments.

Main Outcomes and Measures

The primary outcomes were the proportion of DAT, loss to follow-up, and patients who were provided additional treatments and myocardial revascularization, together with their ratio between treatments. The measures were the pooled overall proportion of the primary outcomes and the risk ratio (RR) in the TAVI vs SAVR groups.

Results

The search identified 8 eligible trials including 8849 participants randomly assigned to undergo TAVI (n = 4458) or SAVR (n = 4391). The pooled proportion of DAT among the sample was 4.2% (95% CI, 3.0%-5.6%), favoring TAVI (pooled RR vs SAVR, 0.16; 95% CI, 0.08-0.36; P < .001). The pooled proportion of loss to follow-up was 4.8% (95% CI, 2.7%-7.3%). Meta-regression showed a significant association between the proportion of participants lost to follow-up and follow-up time (slope, 0.042; 95% CI, 0.017-0.066; P < .001). There was an imbalance of loss to follow-up favoring TAVI (RR, 0.39; 95% CI, 0.28-0.55; P < .001). The pooled proportion of patients who had additional procedures was 10.4% (95% CI, 4.4%-18.5%): 4.6% (95% CI, 1.5%-9.3%) in the TAVI group and 16.5% (95% CI, 7.5%-28.1%) in the SAVR group (RR, 0.27; 95% CI, 0.15-0.50; P < .001). The imbalance between groups also favored TAVI for additional myocardial revascularization (RR, 0.40; 95% CI, 0.24-0.68; P < .001).

Conclusions and Relevance

This study suggests that, in RCTs comparing TAVI vs SAVR, there are substantial proportions of DAT, loss to follow-up, and additional procedures together with systematic selective imbalance in the same direction characterized by significantly lower proportions of patients undergoing TAVI that might affect internal validity.

This systematic review and meta-analysis assesses whether randomization protects randomized clinical trials comparing transcatheter aortic valve implantation and surgical aortic valve replacement from biases other than nonrandom allocation.

Introduction

The literature on the comparison between transcatheter and surgical approaches for aortic valve stenosis comprises 7 randomized clinical trials (RCTs) conducted since 2007 in progressively lower-risk populations that demonstrated noninferiority and, in some cases, superiority of transcatheter aortic valve implantation (TAVI) compared with the standard of care.^{1,2,3,4,5,6,7,8,9,10,11,12,13,14} These RCTs form the basis of guidelines on the management of heart valve disease; nevertheless, several concerns on study design remain.^15,16,17,18 Although randomization allows control for confounding on admission, exclusion after randomization and deviations from protocol related to additional procedures can still introduce bias.^17,18 Exclusion after randomization first occurs when participants do not adhere to or receive the assigned treatment and can be summarized as the difference between intention-to-treat (ITT) and as-treated cohorts. This difference is also known as deviations from random assigned treatment (DAT).¹⁹ The erosion of the unbiased ITT groups generated by nonrandom loss of participants produces biases related to the violation of the principle of randomization, particularly if nonrandom loss of participants is selectively represented in the treatments.^19,20,21 In RCTs designed for evaluating outcomes at follow-up, the main source of exclusion is loss to follow-up, as participants might withdraw from the study or might not be located.^19,22,23 Each loss impairs internal validity, especially if it is differential among treatment groups. The imbalance related to patients’ exclusion for nonadherence to treatments or loss to follow-up could lead to attrition bias,^22,23 which might undermine the advantages of randomization, thereby challenging the comparability of the treatment groups.

Another major issue that emerged in the 2021 European Society of Cardiology/European Association for Cardio-Thoracic Surgery (ESC/EACTS) guidelines’¹⁸ quality assessment by version 2 of the Cochrane risk-of-bias tool for randomized trials (RoB 2 tool) is the risk of performance bias that occurs if additional treatments are provided preferentially to a single group, potentially leading to biased outcomes.^22,23 An imbalance in concomitant procedures (revascularization and other additional procedures) between patients who underwent TAVI and those who underwent surgical aortic valve replacement (SAVR) is present in almost all the RCTs. Although some of these procedures cannot be defined as protocol deviation because they are allowed according to the protocol, each treatment other than isolated aortic valve replacement may affect short- and long-term outcomes, and differential rates of additional procedures may selectively affect the results.

Although the assessment of methodological quality of RCTs by the RoB 2 tool is mostly qualitative,²⁴ we performed a quantitative assessment of DAT, loss to follow-up, and receipt of additional treatments for the TAVI vs SAVR trials and an evaluation of their differential rates between treatments.

Methods

Search Strategy and Selection Criteria

This meta-analysis study is exempt from ethics approval as we collected and synthesized data published from previous clinical trials in which informed consent had already been obtained by the trial investigators. The study protocol adhered to the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) reporting guideline.²⁵ The protocol has been registered in PROSPERO.²⁶

We searched publications from MEDLINE, Embase, and the Cochrane Central Register of Controlled Trials from January 1, 2007, to June 6, 2022. We also checked websites^{27,28,29,30,31} for unpublished data. The search algorithm has been published¹⁶ and is detailed in the eTable in Supplement 1. We included RCTs with random allocation to undergo TAVI or SAVR with a maximal follow-up of 5 years.

Outcomes Measures

The primary outcomes were the proportion of DAT, loss to follow-up, patients who underwent additional interventions, and additional myocardial revascularization. DAT was calculated as the difference between the ITT and as-treated cohorts. Loss to follow-up was extracted at all available follow-up times.

Proportion meta-analysis was used to calculate the pooled overall proportion of the primary outcomes. Risk ratio (RR) was used as effect size for evaluating differential rates of the primary outcomes between TAVI and SAVR (with SAVR as the control group).

Data Extraction

Two independent investigators (F.B. and D.R.) identified trials that fulfilled the prespecified inclusion criteria. Eligible trials were then reviewed in duplicate and disagreement was resolved by a third investigator (A.P.). Extracted data from the text and appendixes of eligible trials were trial characteristics, patients’ baseline data and comorbidities, device type, and implantation access.

Risk of Bias and Quality Assessment

This meta-analysis quantitatively evaluated 2 domains of the RoB 2 tool. The risk of bias among trials included in the ESC/EACTS 2021 Guidelines for Management of Heart Valve Disease by the 4 delegates¹⁸ was then updated by 2 researchers (F.B. and A.P.), with a third researcher (A.A.) designated to resolve potential disagreements using the RoB 2 tool,²⁴ incorporating information provided by outcomes of this meta-analysis.

Statistical Analysis

Random-effects proportion meta-analysis was used to estimate the pooled overall proportions of the primary outcomes.³² The between-study variance was estimated with the Hartung-Knapp-Sidik-Jonkman method, while the Cochran Q test and the I² statistics were applied to evaluate between-study heterogeneity.

Random-effects meta-analysis was used to pool the RR of primary outcomes in the TAVI vs SAVR groups.³³ Between-study variance was estimated with the Hartung-Knapp-Sidik-Jonkman method, while the Cochran Q test and the I² statistics were applied to evaluate between-study heterogeneity. The association between loss to follow-up and follow-up time was explored with random-effects meta-regression. Analyses were performed with R, version 4.2.0 (R Group for Statistical Computing).³⁴ All P values were from 2-sided tests and results were deemed statistically significant at P < .05.

Results

Trials Characteristics

Nine trials were checked for further assessment.^{1,2,3,4,5,6,7,8,10,11,12,13,15,16,35,36} The STACCATO (Transapical Transcatheter Aortic Valve Implantation vs Surgical Aortic Valve Replacement in Operable Elderly Patients With Aortic Stenosis) trial³⁵ was excluded as it was prematurely terminated by the data safety monitoring board. The remaining 8 trials (PARTNER [Placement of Aortic Transcatheter Valve Trial] 1A, PARTNER 2A, PARTNER 3, CoreValve US Pivotal High Risk Trial, SURTAVI [Surgical Replacement and Transcatheter Aortic Valve Implantation] Trial, Evolut Low Risk Trial, NOTION [Nordic Aortic Valve Intervention] Trial, and UK TAVI [Transcatether Aortic Valve Implantation] Trial) fulfilled the prespecified inclusion criteria and were included in the meta-analysis (eFigure 1 in Supplement 1).^{1,2,3,4,5,6,7,8,10,11,12,13,15,16,36}

Baseline characteristics of the included RCTs are detailed in the eTable in Supplement 2. All studies were multicenter RCTs. The primary analysis was ITT analysis in PARTNER 1A, PARTNER 2A, and the UK TAVI Trial; modified ITT (patients who had undergone randomization and an attempted procedure) analysis in SURTAVI; and as-treated analysis in PARTNER 3, CoreValve US Pivotal High Risk Trial, Evolut Low Risk Trial, and NOTION Trial. Overall, 8849 patients were randomly assigned to undergo TAVI (n = 4458) or SAVR (n = 4391). In the 8 trials, both balloon-expanding and self-expanding TAVI devices were evaluated. Different approaches were used for TAVI; however, the most common access was transfemoral.

Analysis of DAT

The pooled proportion of DAT was 4.2% (95% CI, 3.0%-5.6%) (eFigure 2 in Supplement 1), with significant heterogeneity among the studies (τ² = 0.0017; Q test P < .001; I² = 86%). DAT showed a selective pattern, being 6.2-fold lower in the TAVI group compared with the SAVR group (pooled RR, 0.16; 95% CI, 0.08-0.36; P < .001) (eFigure 3 in Supplement 1).

The pooled proportion of DAT in the subgroup of 5 studies that did not perform ITT analysis was 3.6% (95% CI, 2.1%-5.5%) (eFigure 4 in Supplement 1). Also, this subset showed a differential rate of DAT, with a 6.2-fold lower proportion of DAT in the TAVI group compared with SAVR group (RR, 0.16; 95% CI, 0.04-0.63; P = .008) (Figure 1). Only the NOTION Trial did not show a selective DAT.

Figure 1. — There is a selective pattern characterized by a lower proportion of DAT in the TAVI group resulting in a 6.2-fold lower proportion of DAT in the TAVI group compared with the SAVR group. Only the NOTION (Nordic Aortic Valve Intervention) Trial does not show a selective DAT. The size of the solid squares is proportional to the weight of each study, the horizontal bars indicate the 95% CI for each study, and the diamond represents the pooled estimate with 95% CI. PARTNER 3 indicates Placement of Aortic Transcatheter Valve Trial 3; RE, random effect; and SURTAVI, Surgical Replacement and Transcatheter Aortic Valve Implantation.

Analysis of Loss to Follow-up

Figure 2 shows a forest plot presenting the proportional meta-analysis of loss to follow-up by year of follow-up in the 8 included RCTs. The pooled proportion of loss to follow-up was 4.8% (95% CI, 2.7%-7.3%), indicating that, on average, 4.8% of patients at risk were missing at each follow-up time. A subgroup analysis on different follow-up times demonstrated that the prevalence of loss to follow-up increased concordantly with follow-up time, increasing from 1.4% (95% CI, 0.5%-2.6%) at 1 year to 8.9% (95% CI, 3.3%-16.9%) at 5 years. Meta-regression confirmed a significant association between the proportion of participants lost to follow-up and follow-up time (slope, 0.042; 95% CI, 0.017-0.066; P < .001) (Figure 3).

Figure 3. — There is a significant association between proportion of patients lost to follow-up and follow-up time. Shaded area indicates 95% CI; diagonal line indicates the linear association between loss to follow-up and follow-up time; circles indicate the rates of loss to follow-up at follow-up time for each randomized clinical trial; and circle size indicates the weight of single data.

The analysis of selective loss to follow-up by follow-up years is depicted in Figure 4. The pooled RR of loss to follow-up in the TAVI group vs the SAVR group was 0.39 (95% CI, 0.28-0.55; P < .001), indicating that TAVI had a significant 2.56-fold lower risk of patients lost to follow-up. The selective loss to follow-up appears to decrease with increasing follow-up time, although meta-regression does not confirm the significance of this association (slope, 0.166; 95% CI, −0.034 to 0.366; P = .10) (eFigure 5 in Supplement 1).

Analysis of the Proportion of Patients Who Received Additional Treatments

Data on additional procedures and additional revascularization are available for the CoreValve US Pivotal High Risk Trial, PARTNER 2A, SURTAVI, UK TAVI Trial, PARTNER 3, Evolut Low Risk Trial, and NOTION Trial.^{2,3,4,5,6,7,8,10,11,12,13,15,16,36}

The pooled proportion of patients who underwent additional procedures was 10.4% (95% CI, 4.4%-18.5%) (eFigure 6 in Supplement 1). There was significant heterogeneity among studies (τ² = 0.0241; Q test P < .001). There were differential rates of receipt of additional treatments in the 2 groups; the pooled proportion in the TAVI group was 4.6% (95% CI, 1.5%-9.3%) and in the SAVR group was 16.5% (95% CI, 7.5%-28.1%) (eFigure 7 in Supplement 1). This imbalance was statistically significant (RR, 0.27; 95% CI, 0.15-0.50; P < .001), indicating that TAVI has a 3.7-fold lower proportion of participants who undergo additional procedures (Figure 5). Only the NOTION Trial showed no significant difference between groups in the rate of patients who had additional procedures.

Figure 5. — There is a selective pattern similar to deviation from assigned treatment and loss to follow-up, resulting in a 3.7-fold lower proportion of participants with additional procedures in the TAVI group. The size of the solid squares is proportional to the weight of each study, the horizontal bars indicate the 95% CI for each study, the diamond represents the pooled estimate with 95% CI, and the vertical dotted line indicates a risk ratio equal to 1. NOTION indicates Nordic Aortic Valve Intervention Trial; PARTNER, Placement of Aortic Transcatheter Valve Trial; RE, random effect; and SURTAVI, Surgical Replacement and Transcatheter Aortic Valve Implantation.

The pooled proportion of additional revascularizations was 7.5% (95% CI, 3.0%-13.9%), with significant heterogeneity among studies (τ² = 0.0189; Q test P < .001) (eFigure 8 in Supplement 1). Again, there was a differential rate of additional myocardial revascularization in the 2 groups; the pooled proportion in the TAVI group was 4.5% (95% CI, 1.4%-9.0%) and in the SAVR group was 10.8% (95% CI, 4.8%-18.9%) (eFigure 9 in Supplement 1). The imbalance between groups was confirmed by the RR of patients with additional revascularization (RR, 0.40; 95% CI, 0.24-0.68; P < .001) (eFigure 10 in Supplement 1), indicating that TAVI has a 2.5-fold lower proportion of additional revascularization.

Discussion

This study has 3 main findings. First, there are substantial proportions of DAT, loss to follow-up, and receipt of additional procedures in the seminal TAVI vs SAVR trials. Second, there is systematic imbalance in these parameters that trend in the same direction—with significantly lower proportions of DAT, loss to follow-up, and receipt of additional procedures in the TAVI groups. Therefore, despite randomization, there may remain biases that are critically associated with internal validity.

Third, attrition bias may be the most substantial threat to internal validity. Loss to follow-up threatens the ITT principle for guaranteeing an unbiased treatment comparison and should be minimized to preserve proper randomization. Although only 0% of losses ensures the benefit of randomization, 5% is the proposed cutoff for differentiating between little risk and intermediate risk of bias, while loss exceeding 20% could pose critical threats to validity.^19,22 In the RCTs comparing TAVI and SAVR, there is a progressive increase of loss to follow-up reaching 5% at 2 years and 9% at 5 years that may progressively bias results with increasing follow-up time, resulting in noninformative data on durability and effectiveness at midterm and long term. The selective loss to follow-up is even more critical than the overall loss to follow-up, as it is not random¹⁹ and can potentially lead to informative censoring. Almost all the pivotal RCTs have a high imbalance of loss to follow-up, with a constant pattern favoring TAVI, reaching a loss of 10% in the TAVI groups and more than 20% in the SAVR groups at 5 years. This imbalance between treatments challenges internal validity. In addition, as the clinical events contributing to the primary outcomes (ie, death and stroke) are relatively rare in the study populations, the attrition bias introduced by significant loss to follow-up can be potentially amplified. Incompleteness of follow-up data is somehow inevitable; however, for minimizing incompleteness, the sample size should be estimated taking into account the length of follow-up and potential rates of loss, and the protocol should include methods for reducing and handling losses, including selection of centers with a history of excellent retention and use of statistical methods for analysis of missing data, such as imputation.²⁴

Both the absolute rate of DAT and selective DAT can be associated with the ITT principle and undermine unbiased comparison groups at baseline.^20,21,22,24 To obviate this risk, the Cochrane Rob 2 tool recommends estimating the effect of assignment to the intervention group by ITT analysis, which is a de facto standard for a clinical trial, and when as-treated analysis is performed.³⁷ The use of ITT analysis, where all patients are counted according to their randomized group, ensures comparability between groups as obtained through randomization, maintains sample size, and eliminates bias. This approach measures both the efficacy of the intervention as assigned and overall adherence. In contrast, as-treated analysis refers to inclusion in the analysis of only patients who received the assigned intervention and per-protocol analysis refers to inclusion in the analysis of only patients who strictly adhered to the protocol. Although per-protocol analysis provides an estimate of the true efficacy of an intervention, it can be subject to bias as randomization has been violated. Poor trial quality (eg, due to poor protocol adherence or missing data) might bias results toward the null. In superiority studies, ITT analysis is therefore conservative, favoring the null hypothesis of no difference. However, in noninferiority trials (7 of 8 pivotal trials were designed as noninferiority trials with the NOTION Trial being the only superiority trial), where the alternative hypothesis is that of no difference, ITT analysis is no longer conservative as it can lead to false-positive results by favoring the alternative hypothesis. Consequently, noninferiority trials often consider both ITT and per-protocol analyses concurrently. Randomized clinical trials comparing TAVI and SAVR have chosen different approaches; nonetheless, in this case, the selective imbalance of DAT between TAVI and SAVR might raise concerns regarding internal validity in trials that primarily report as-treated or per-protocol analyses.

Performance bias is another critical issue in RCTs comparing TAVI and SAVR.²² Although most of the additional treatments are allowed by the protocol design and the formal definition of protocol deviation does not apply, selective application of additional procedures biases outcomes, as it expresses imbalanced baseline myocardial disease or systematic differences in the provided care. Myocardial revascularization strategies should be equally applicable to the 2 groups and patients requiring coronary revascularization should be included unless only surgical revascularization is considered appropriate,³⁶ to avoid selective administration of adjunctive treatments. The high difference in rates of concomitant myocardial revascularization in TAVI groups vs SAVR groups is inconsistent with the equality of treatments; the trialists of PARTNER 2A acknowledge this point as they attribute such imbalanced rates to the undertreatment of significant coronary artery disease in the TAVI group.⁴

Similar considerations may be made with respect to the concomitant performance of thoracic aortic surgery or nonaortic valve procedures among patients undergoing SAVR; these procedures occurred at discrete, nonnegligible rates in PARTNER 3 and the Evolut Low Risk Trial. In other words, the administration of different therapeutic strategies to the same disease is in itself a selective protocol deviation with an intrinsic high risk of performance bias. The ESC/EACTS delegates interpreted the different proportion of myocardial revascularization in favor of SAVR and the different proportion of other additional procedures in favor of TAVI.¹⁸ This position is debatable based on the results of the ISCHEMIA (International Study of Comparative Health Effectiveness with Medical and Invasive Approaches) trial, which demonstrated that medical therapy holds a lower risk of adverse events in the short term while invasive treatment seems to yield a morbidity or mortality advantage only in the midterm and long term, confirming that each invasive maneuver holds an intrinsic risk of invasiveness-related adverse events.³⁸ The more invasive the procedure, the higher the short-term risk of complications, while the potential advantage of an invasive treatment is likely to emerge in the midterm to long-term follow-up.^39,40 As it is well expressed by EuroSCORE (the European System for Cardiac Operative Risk Evaluation) and STS PROM (Society of Thoracic Surgeons–Predicted Risk of Mortality) score, the adjunct of coronary artery bypass graft as well as other additional procedures significantly increases perioperative risk and represents an intrinsic baseline gap vs the isolated procedure, as also confirmed by the US Food and Drug Administration exploratory analysis of PARTNER 1A, where adding concomitant procedures to SAVR nearly doubled the risk of mortality (RR, 1.9; 95% CI, 1.3-2.7), biasing short-term results in favor of TAVI.⁴¹ The potential unfavorable effect of selective undertreatment reported in the TAVI groups will likely be unmasked only in the midterm, well beyond 1- to 2-year follow-up of the RCTs as evaluated in primary or secondary outcome measures.

The quantitative estimation of overall rates and selective imbalance of DAT, loss to follow-up, and receipt of additional procedures of our meta-analysis may help to revise at least some domains of the RoB 2 tool (eFigure 11 and eAppendix in Supplement 1). The imbalance of both proportion of additional procedures and myocardial revascularization favors TAVI as it increases the short-term risk from SAVR, while the beneficial association of a higher rate of myocardial revascularization may be evident in the long term. Hence, only the NOTION Trial is at low risk of performance bias, while the other 7 RCTs should be considered at high risk of performance bias. The high rate of exclusions after randomizations and the significant discrepancy between TAVI and SAVR suggest that PARTNER 1A, the CoreValve US Pivotal High Risk Trial, PARTNER 2A, PARTNER 3, SURTAVI, the UK TAVI Trial, and the Evolut Low Risk Trial are at high risk of attrition bias. These observations should lead clinicians to consider almost all RCTs comparing TAVI and SAVR to be at high risk of biases, and it might reduce their internal validity, although the direction of the biases may be only hypothesized without individual patient data.

Limitations

This meta-analysis has some limitations. The main limitation is that we cannot predict how these imbalances act in the magnitude and direction of bias and overall benefit. Although it is described that selective rates of withdrawal may favor the treatment compared with the control, only analyses of individual patient data can confirm it. At this point of evidence, we can simply highlight that RCTs show serious methodological imbalances with a common selective pattern that can increase the risk of bias. The association of the risk of performance and attrition biases with trials’ primary outcomes is far beyond the aims of the present meta-analysis, which is focused on methodological issues and has been designed to quantify the key factors that might inflate performance and attrition biases. Only data sharing and an independent evaluation may overcome the existing limitations, permitting a deep understanding of the association between risk of bias and results.

Conclusions

This systematic review and meta-analysis found that the RCT design does not protect from biases other than nonrandom allocation. In RCTs comparing TAVI vs SAVR, there were systematic imbalances in the proportion of DAT, loss to follow-up, and receipt of additional procedures and additional myocardial revascularization that can pose a serious threat to internal validity due to high risk of performance and attrition biases. The potential associations of this risk of bias with trials’ primary outcomes are beyond the aims of the present meta-analysis and need data sharing to be evaluated.

Supplement 1.

eTable. Updated Search Algorithms

eFigure 1. Systematic Review—Flow Chart

eFigure 2. Forest Plot Representing Pooled Proportion of Deviation From Assigned Treatment (DAT)

eFigure 3. Forest Plot of Risk Ratio of Deviation From Assigned Treatment (DAT) in TAVI vs SAVR (Selective DAT)

eFigure 4. Forest Plot Representing Pooled Proportion of Deviation From Assigned Treatment (DAT) in RCTs That Performed AT or Modified ITT Analysis

eFigure 5. Meta-regression of Relationship Between RR of Loss to Follow-up in TAVI/SAVR and Follow-up Time

eFigure 6. Forest Plot Representing Pooled Rate of Provided Additional Treatments

eFigure 7. Forest Plot Representing Pooled Proportion of Provided Additional Treatments in TAVI and SAVR Groups

eFigure 8. Forest Plot Representing Pooled Rate of Provided Additional Myocardial Revascularization

eFigure 9. Forest Plot Representing Pooled Proportion of Provided Additional Myocardial Revascularization in TAVI and SAVR Groups

eFigure 10. Forest Plot Presenting the Risk Ratio of Patients With Provided Additional Myocardial Revascularization in TAVI vs SAVR

eFigure 11. Risk of Bias Evaluation Using the RoB 2.0 Tool

eAppendix. Revised Cochrane Risk of Bias Tool for Randomized Trials (RoB 2 Tool) Comparing TAVI and SAVR

Click here for additional data file.^{(1.1MB, pdf)}

Supplement 2.

eTable. Baseline Characteristics of the 8 Trials Included in the Meta-analysis

Click here for additional data file.^{(24.8KB, xlsx)}

Supplement 3.

Nonauthor Collaborators. International Evidence Grading Research Initiative Targeting Transparency and Quality (INTEGRITTY) Collaborators

Click here for additional data file.^{(148KB, pdf)}

Supplement 4.

Data Sharing Statement

Click here for additional data file.^{(14.9KB, pdf)}

References

1.Mack MJ, Leon MB, Smith CR, et al. ; PARTNER 1 trial investigators . 5-Year outcomes of transcatheter aortic valve replacement or surgical aortic valve replacement for high surgical risk patients with aortic stenosis (PARTNER 1): a randomised controlled trial. Lancet. 2015;385(9986):2477-2484. doi: 10.1016/S0140-6736(15)60308-7 [DOI] [PubMed] [Google Scholar]
2.Gleason TG, Reardon MJ, Popma JJ, et al. ; CoreValve U.S. Pivotal High Risk Trial Clinical Investigators . 5-Year outcomes of self-expanding transcatheter versus surgical aortic valve replacement in high-risk patients. J Am Coll Cardiol. 2018;72(22):2687-2696. doi: 10.1016/j.jacc.2018.08.2146 [DOI] [PubMed] [Google Scholar]
3.Deeb GM, Reardon MJ, Chetcuti S, et al. ; CoreValve US Clinical Investigators . 3-Year outcomes in high-risk patients who underwent surgical or transcatheter aortic valve replacement. J Am Coll Cardiol. 2016;67(22):2565-2574. doi: 10.1016/j.jacc.2016.03.506 [DOI] [PubMed] [Google Scholar]
4.Makkar RR, Thourani VH, Mack MJ, et al. ; PARTNER 2 Investigators . Five-year outcomes of transcatheter or surgical aortic-valve replacement. N Engl J Med. 2020;382(9):799-809. doi: 10.1056/NEJMoa1910555 [DOI] [PubMed] [Google Scholar]
5.Leon MB, Smith CR, Mack MJ, et al. ; PARTNER 2 Investigators . Transcatheter or surgical aortic-valve replacement in intermediate-risk patients. N Engl J Med. 2016;374(17):1609-1620. doi: 10.1056/NEJMoa1514616 [DOI] [PubMed] [Google Scholar]
6.Thourani V. PARTNER 2: 5 year outcomes. European Association for Cardio-Thoracic Surgery. Accessed June 4, 2022. http://medialibrary.eacts.cyim.com/mediatheque/media.aspx?mediaId=76635&channel=10233
7.Reardon MJ, Van Mieghem NM, Popma JJ, et al. ; SURTAVI Investigators . Surgical or transcatheter aortic-valve replacement in intermediate-risk patients. N Engl J Med. 2017;376(14):1321-1331. doi: 10.1056/NEJMoa1700456 [DOI] [PubMed] [Google Scholar]
8.TCTMD.com. SURTAVI: five-year results from a randomized trial of TAVR vs. SAVR in patients at intermediate surgical risk. Accessed June 4, 2022. https://www.tctmd.com/slide/surtavi-five-year-results-randomized-trial-tavr-vs-savr-patients-intermediate-surgical-risk
9.Mack MJ, Leon MB, Thourani VH, et al. ; PARTNER 3 Investigators . Transcatheter aortic-valve replacement with a balloon-expandable valve in low-risk patients. N Engl J Med. 2019;380(18):1695-1705. doi: 10.1056/NEJMoa1814052 [DOI] [PubMed] [Google Scholar]
10.Mack MJ, Leon MB. Two-year clinical and echocardiographic outcomes from the PARTNER 3 low-risk randomized trial. American College of Cardiology. Accessed June 4, 2022. https://www.acc.org/~/media/Clinical/PDF-Files/Approved-PDFs/2020/03/24/ACC20/29Mar-Sun/9amET-PARTNER-3-acc-2020.pdf
11.Leon MB, Mack MJ, Hahn RT, et al. ; PARTNER 3 Investigators . Outcomes 2 years after transcatheter aortic valve replacement in patients at low surgical risk. J Am Coll Cardiol. 2021;77(9):1149-1161. doi: 10.1016/j.jacc.2020.12.052 [DOI] [PubMed] [Google Scholar]
12.Thyregod HGH, Ihlemann N, Jørgensen TH, et al. Five-year clinical and echocardiographic outcomes from the Nordic Aortic Valve Intervention (NOTION) randomized clinical trial in lower surgical risk patients. Circulation. 2019;139:2714-2723. doi: 10.1161/CIRCULATIONAHA.118.036606 [DOI] [PubMed] [Google Scholar]
13.Popma JJ, Deeb GM, Yakubov SJ, et al. ; Evolut Low Risk Trial Investigators . Transcatheter aortic-valve replacement with a self-expanding valve in low-risk patients. N Engl J Med. 2019;380(18):1706-1715. doi: 10.1056/NEJMoa1816885 [DOI] [PubMed] [Google Scholar]
14.TCTMD.com. Primary results from the Evolut low risk trial. Accessed June 4, 2022. https://www.tctmd.com/slide/primary-results-evolut-low-risk-trial
15.Barili F, Freemantle N, Parolari A. Five-year outcomes with transcatheter aortic-valve replacement. N Engl J Med. 2020;383(6):594. doi: 10.1056/NEJMc2018853 [DOI] [PubMed] [Google Scholar]
16.Barili F, Freemantle N, Musumeci F, et al. ; Latin European Alliance of CardioVascular Surgical Societies (LEACSS) and with the endorsement of the Latin American Association of Cardiac and Endovascular Surgery (LACES), LEACSS members are the Italian Society of Cardiac Surgery (FB FM MR MdM AP), the Portuguese Society of Cardiac Surgery (MSU), the French Society of Cardiac Surgery (JFV, AA) and the Spanish Society of Cardiac Surgery (JRR) Institutions . Five-year outcomes in trials comparing transcatheter aortic valve implantation versus surgical aortic valve replacement: a pooled meta-analysis of reconstructed time-to-event data. Eur J Cardiothorac Surg. 2022;61(5):977-987. doi: 10.1093/ejcts/ezab516 [DOI] [PubMed] [Google Scholar]
17.Jadad AR, Enkin MW. Randomised Controlled Trials: Questions, Answers and Musings. 2nd ed. BMJ Books; 2007. doi: 10.1002/9780470691922 [DOI] [Google Scholar]
18.Vahanian A, Beyersdorf F, Praz F, et al. ; ESC/EACTS Scientific Document Group . 2021 ESC/EACTS guidelines for the management of valvular heart disease. Eur Heart J. 2022;43(7):561-632. doi: 10.1093/eurheartj/ehab395 [DOI] [PubMed] [Google Scholar]
19.Schulz KF, Grimes DA. Sample size slippages in randomised trials: exclusions and the lost and wayward. Lancet. 2002;359(9308):781-785. doi: 10.1016/S0140-6736(02)07882-0 [DOI] [PubMed] [Google Scholar]
20.Ranganathan P, Pramesh CS, Aggarwal R. Common pitfalls in statistical analysis: measures of agreement. Perspect Clin Res. 2017;8(4):187-191. doi: 10.4103/picr.PICR_123_17 [DOI] [PMC free article] [PubMed] [Google Scholar]
21.Smith VA, Coffman CJ, Hudgens MG. Interpreting the results of intention-to-treat, per-protocol, and as-treated analyses of clinical trials. JAMA. 2021;326(5):433-434. doi: 10.1001/jama.2021.2825 [DOI] [PMC free article] [PubMed] [Google Scholar]
22.Mansournia MA, Higgins JPT, Sterne JAC, Hernán MA. Biases in randomized trials: a conversation between trialists and epidemiologists. Epidemiology. 2017;28(1):54-59. doi: 10.1097/EDE.0000000000000564 [DOI] [PMC free article] [PubMed] [Google Scholar]
23.Jüni P, Altman DG, Egger M. Systematic reviews in health care: assessing the quality of controlled clinical trials. BMJ. 2001;323(7303):42-46. doi: 10.1136/bmj.323.7303.42 [DOI] [PMC free article] [PubMed] [Google Scholar]
24.Sterne JAC, Savović J, Page MJ, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ. 2019;366:l4898. doi: 10.1136/bmj.l4898 [DOI] [PubMed] [Google Scholar]
25.Moher D, Liberati A, Tetzlaff J, Altman DG; PRISMA Group . Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Int J Surg. 2010;8(5):336-341. doi: 10.1016/j.ijsu.2010.02.007 [DOI] [PubMed] [Google Scholar]
26.Analysis of imbalances inflating performance and attrition biases in randomized controlled trials comparing trans-catheter and surgical aortic valve replacement: systematic review and meta-analysis. PROSPERO identifier: CRD42022335718. June 15, 2022. Accessed June 15, 2022. https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=335718
27.ClinicalTrials.gov . Accessed November 21, 2022. http://www.clinicaltrials.gov
28.American College of Cardiology . Accessed November 21, 2022. http://www.acc.org
29.European Society of Cardiology . Accessed November 21, 2022. http://www.escardio.org
30.TCTMD.com . Accessed November 21, 2022. http://www.tctmd.com
31.PCR online. Accessed November 21, 2022. http://www.pcronline.com
32.Barker TH, Migliavaca CB, Stein C, et al. Conducting proportional meta-analysis in different types of systematic reviews: a guide for synthesisers of evidence. BMC Med Res Methodol. 2021;21(1):189. doi: 10.1186/s12874-021-01381-z [DOI] [PMC free article] [PubMed] [Google Scholar]
33.Whitehead A. Meta-analysis of Controlled Clinical Trials. John Wiley & Sons Ltd; 2002. doi: 10.1002/0470854200 [DOI] [Google Scholar]
34.R Development Core Team . R: A language and environment for statistical computing. R Foundation for Statistical Computing. Accessed November 21, 2022. http://www.R-project.org/
35.Nielsen HH, Klaaborg KE, Nissen H, et al. A prospective, randomised trial of transapical transcatheter aortic valve implantation vs. surgical aortic valve replacement in operable elderly patients with aortic stenosis: the STACCATO trial. EuroIntervention. 2012;8(3):383-389. doi: 10.4244/EIJV8I3A58 [DOI] [PubMed] [Google Scholar]
36.Toff WD, Hildick-Smith D, Kovac J, et al. ; UK TAVI Trial Investigators . Effect of transcatheter aortic valve implantation vs surgical aortic valve replacement on all-cause mortality in patients with aortic stenosis: a randomized clinical trial. JAMA. 2022;327(19):1875-1887. doi: 10.1001/jama.2022.5776 [DOI] [PMC free article] [PubMed] [Google Scholar]
37.Higgins JPT, Savović J, Page MJ, Elbers RG, Sterne JAC. Chapter 8: Assessing risk of bias in a randomized trial. Accessed August 14, 2022. https://training.cochrane.org/handbook/current/chapter-08
38.Maron DJ, Hochman JS, Reynolds HR, et al. ; ISCHEMIA Research Group . Initial invasive or conservative strategy for stable coronary disease. N Engl J Med. 2020;382(15):1395-1407. doi: 10.1056/NEJMoa1915922 [DOI] [PMC free article] [PubMed] [Google Scholar]
39.Stone GW, Kappetein AP, Sabik JF, et al. ; EXCEL Trial Investigators . Five-year outcomes after PCI or CABG for left main coronary disease. N Engl J Med. 2019;381(19):1820-1830. doi: 10.1056/NEJMoa1909406 [DOI] [PubMed] [Google Scholar]
40.Fearon WF, Zimmermann FM, De Bruyne B, et al. ; FAME 3 Investigators . Fractional flow reserve–guided PCI as compared with coronary bypass surgery. N Engl J Med. 2022;386(2):128-137. doi: 10.1056/NEJMoa2112299 [DOI] [PubMed] [Google Scholar]
41.US Food & Drug Administration. Summary of safety and effectiveness data. Accessed November 28, 2022. https://www.accessdata.fda.gov/cdrh_docs/pdf11/P110021b.pdf

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials