Generalisability of randomised trials evaluating perioperative β-blocker therapy in noncardiac surgery

Abstract

Background

The limited applicability of evidence from RCTs in real-word practice is considered a potential bottleneck for evidence-based practice but rarely systematically assessed. Using our failure to recruit patients into a perioperative beta-blocker trial, we set out to analyse the restrictiveness and generalisability of trial eligibility criteria in a real-world cohort.

Methods

We prospectively included adult patients (≥18 yr) scheduled for elective noncardiac surgery at an academic tertiary care facility who were screened for inclusion in a planned perioperative beta-blocker RCT, which was terminated owing to recruitment failure. The primary outcome was the proportion of screened patients who matched the eligibility criteria of 36 published RCTs included in a large Cochrane meta-analysis on perioperative beta-blocker therapy. The pragmatic/explanatory level of each RCT was assessed using the PRagmatic–Explanatory Continuum Indicator Summary 2 (PRECIS-2) score, which ranges from 9 points (indicating a very explanatory study) to 45 points (indicating a very pragmatic study).

Results

A total of 2241 patients (54% female, n=1215; 52 [standard deviation, 20] yr) were included for the assessment of trial eligibility between October 2015 and January 2016. Only a small proportion of patients matched the inclusion and exclusion criteria for each of the 36 RCTs, ranging from 53% to 0%. The average proportion of patients who did match the eligibility criteria of all 36 RCTs was 6.5% (n=145; 95% confidence interval, 6.3–6.6). A higher PRECIS-2 score was associated with a higher proportion of matching patients (P<0.001).

Conclusions

Trial eligibility criteria in perioperative beta-blocker therapy trials are overly restrictive and not generalisable to a real-world surgical population.

Clinical trial registration

EudraCT#: 2015-002366-23.

Editor's key points.

• •Clinical trials are more useful if they provide meaningful results that can be broadly applied.
• •Explanatory trials should have high internal validity, testing an intervention under ideal, homogenous conditions.
• •Pragmatic trials should have high external validity (generalisability), testing interventions under a broad range of patients and settings.
• •The PRECIS-2 tool can be used to assess the level of pragmatism, thus the generalisability, of a given trial.

Well-conducted RCTs are the gold standard for medical evidence and the basis for clinical decision-making. The applicability of evidence from RCTs in real-word practice may be limited but is rarely systematically assessed.^1,2 Patients enrolled in RCTs are commonly highly selected using restrictive inclusion and exclusion criteria to reduce subject variability and bias.³ Internal validity refers to the accuracy of a trial's findings and the degree of a causal effect relationship between treatment and outcome. Sample homogeneity improves the chance of internally valid study results. However, restrictive inclusion and exclusion criteria may impair a study's generalisability.

Generalisability refers to the external validity of study inferences and represents the degree to which study results can be extrapolated to different circumstances and populations. Knowledge of the external validity of a trial is vital as it determines its applicability to heterogeneous measures, persons, settings, and times.⁴ Real-world evidence is defined by the degree of pragmatism. However, reporting of the determinants of external validity in scientific publications is often limited.⁵

The PRagmatic–Explanatory Continuum Indicator Summary 2 (PRECIS-2) tool has been developed to support the assessment of the pragmatic or explanatory level of a given RCT.⁶ Whereas explanatory trials seek for efficiency and internal validity, pragmatic trials focus on effectiveness and external validity.^7,8 The pragmatism of a given trial is essential if its results are to be translated adequately into clinical practice but rarely assessed by investigators.⁵ Systematic assessment of the applicability of current evidence from RCTs may allow clinicians for a better interpretation of RCT evidence to real-world practice.

The current study was driven by our inability to recruit adult patients scheduled for noncardiac surgery into an RCT because of restrictive inclusion and exclusion criteria. The RCT aimed to determine efficacy and safety of perioperative beta-blocker therapy in patients with a high risk of cardiovascular mortality. In the trial's screening phase, we noticed that the potentially eligible trial population was very small (only one patient out of more than 2200 patients screened could be enrolled within a 3 month period). The inability to recruit patients at one of the largest tertiary care centres in Europe raised the concern that the generalisability of beta-blocker trial populations in a real-world cohort of patients may be limited.

We hypothesised that published RCTs on perioperative β-blocker therapy in noncardiac surgery are largely exploratory and thus applicable only to a minority of patients in clinical practice. The current study aimed to test this hypothesis by systematically assessing the proportion of patients of the screening cohort who matched the inclusion and exclusion criteria of 36 RCTs included in a large Cochrane meta-analysis⁹ and by assessing the pragmatism/explanatory level of each RCT using the PRECIS-2 tool.

Methods

Study setting and patients

This single-centre prospective cohort study was conducted at the Department of Anaesthesia, Medical University of Vienna at the General Hospital of Vienna, Austria, one of the largest tertiary care facilities in Europe. We screened adult patients (≥18 yr), who were scheduled for elective noncardiac surgery (including trauma and orthopaedic surgery, thoracic surgery vascular surgery, abdominal surgery, urologic and gynaecologic surgery) for eligibility for a planned perioperative beta-blocker RCT (details on surgery types which were eligible for inclusion in the RCT are shown in Supplementary Table S1). The RCT was registered at clinicaltrialsregister.eu (EudraCT#: 2015-002366-23). Inclusion and exclusion criteria were chosen based on literature available by 2015. Inclusion criteria were: age ≥45 yr, no history of beta-blocker treatment within 30 days before enrolment, increased cardiovascular risk because of coronary artery disease or a combination of two or more of the following risk factors: ≥70 yr, congestive heart failure (New York Heart Association [NYHA] ≤II), chronic renal failure, smoking history, hypertension, diabetes, lipid-lowering drug treatment or hypercholesterolaemia, and severe obesity (BMI ≥35 kg m⁻²). Exclusion criteria comprised: history of stroke, severe asthma or chronic obstructive pulmonary disease, congestive heart failure NYHA class III/IV, and cardiogenic shock. Table 1 shows the inclusion and exclusion criteria of the failed trial along with matching proportions of the screening cohort.

Table 1.

Inclusion and exclusion criteria of the terminated trial and proportions of patients of the screening cohort fulfilling the criterion. Data are total numbers (n) and percentage (%) of patients fulfilling the criterion. COPD, chronic obstructive pulmonary disease; AVB, atrioventricular block; SAB, sinoatrial block; NYHA, New York Heart Association.

	n (%)
Inclusion criteria
Age ≥45 yr	1510 (67.4)
Scheduled for elective major noncardiac surgery (>1 h)	1657 (73.9)
General anaesthesia	1875 (83.7)
Anticipated hospital stay >2 days	2117 (94.5)
No treatment with β-receptor blocking drugs >30 days before inclusion	426 (19.0)
Coronary artery disease OR	119 (5.3)
Combination of ≥2 cardiovascular risk factors (≥70 yr, congestive heart failure [NYHA ≤II], chronic renal failure, smoking, arterial hypertension, type 1 or 2 diabetes mellitus, high cholesterol or lipid-lowering medication, severe obesity [BMI ≥35])	585 (26.0)
Exclusion criteria
Spinal or epidural anaesthesia	366 (16.3)
History of stroke	72 (3.2)
Severe asthma or COPD	241 (10.8)
Preoperative heart rate <61 beats min−1	44 (2.0)
Cardiac pacemaker	39 (1.7)
Second- or third-degree AVB or SAB	14 (0.6)
Sick sinus syndrome	7 (0.3)
Cardiogenic shock	9 (0.4)
Congestive heart failure (NYHA III or IV)	39 (1.7)
Pulmonary oedema or pulmonary hypertension	20 (0.9)
Untreated pheochromocytoma	0
Vulnerable patients, unable to consent	155 (6.9)
Pregnant or breastfeeding women	4 (0.2)
Intake of calcium channel blockers, cardiac glycosides (digoxin, digitoxin) or If inhibitors (such as ivabradine) <48 h before surgery	231 (10.3)
Liver disease	112 (5.0)
Intolerances or hypersensitivity to study drug or any of its inactive ingredients	0
Previous participation in the study	0
Participation in another, possibly interfering trial	5 (0.2)

Our primary endpoint was the proportion of screened patients matching the inclusion and exclusion criteria of 36 RCTs included in a large Cochrane meta-analysis. Secondary endpoints included the relation of eligibility proportions to the sample size and the PRECIS-2 score of each RCT.

The study was approved by the Ethics Committee of the Medical University of Vienna and conducted in accordance with Helsinki declarations.

RCTs included in the Cochrane meta-analysis

A large Cochrane meta-analysis was used as reference for high-quality evidence on the effect of perioperative beta-blockers in patients undergoing surgery.⁹ Two authors (MT, CK) independently extracted inclusion and exclusion criteria of each RCT (n=36,10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45 Table 2) and systematically assessed the eligibility proportions of our screening cohort.

Table 2.

RCTs on noncardiac surgery patients included in the Cochrane meta-analysis (n=36). Detailed study characteristics are available at https://www.cochranelibrary.com/.⁴⁶ n.a., not available; PMID, PubMed ID.

First author, year of publication	Inclusion criteria (n)	Exclusion criteria (n)	Enrolled patients (n/screened)	PMID
Apipan,10 2010	4	6	60/n.a.	20006165
Bayliff,11 1999	3	13	99/242	10086546
Burns,12 1988	4	2	86/n.a.	2972306
Coleman,13 1980	5	0	42/n.a.	7004251
Cucchiara,14 1986	3	15	74/n.a.	2877599
Juul,15 2006	4	6	921/2066	16793810
Gibson,16 1988	2	11	40/n.a.	2904310
Gupta,17 2011	4	8	66/n.a.	21712869
Inada,18 1989	3	7	40/n.a.	2697239
Jakobsen,19 1992	4	0	40/n.a.	1600972
Jakobsen,20 1997	3	0	35/n.a.	9422300
Kawaguchi,21 2010	5	13	56/n.a.	20118792
Lai,22 2006	2	0	60/n.a.	16687084
Lee,23 2010	4	8	60/n.a.	20877702
Liu,24 1986	4	6	30/n.a.	3768764
Liu,25 2006	2	0	30/n.a.	16706126
Magnusson,26 1986	4	4	30/309	3511930
Mangano,27 1996	6	0	200/n.a.	8929262
Marwick,28 2009	2	6	400/n.a.	19332211
Miller,29 1990	6	8	45/n.a.	n.a.
Miller,30 1991	6	9	548/n.a.	1683818
Moon,31 2011	4	8	54/n.a.	22117987
Neary,32 2006	6	7	38/2351	16764198
Oxorn,33 1990	4	11	48/n.a.	1968784
Brady,34 2005	2	8	103/151	15874923
Devereaux,35 2008	6	10	8351/n.a.	18479744
Raby,36 1999	3	2	26/n.a.	10071990
Sandler,37 1990	4	14	45/n.a.	n.a.
Shukla,38 2010	2	7	60/n.a.	n.a.
Stone,39 1988	3	7	128/n.a.	2895596
Suttner,40 2009	2	6	75/n.a.	19336536
Wallace,41 1998	5	3	200/n.a.	9447850
Whitehead,42 1980	2	0	60/n.a.	7004258
Yang,43 2006	2	7	496/n.a.	17070177
Yang,44 2008	2	0	102/n.a.	18953854
Zaugg,45 1999	2	8	63/n.a.	10598610

Statistical methods

Data are presented as mean (standard deviation [sd]) or median with the 25–75% inter-quartile range (IQR), as appropriate. We determined proportions of screened patients matching the inclusion and exclusion criteria for each available RCT together with 95% Jeffreys intervals. To estimate the average proportion of matches across all studies, we used Poisson regression with a robust cluster variance estimator to calculate 95% confidence intervals. The pragmatic–exploratory continuum PRECIS-2 tool was used to assess the pragmatic/explanatory level of each RCT. The PRECIS-2 tool gives an estimate of the pragmatism/explanatory level of a trial ranging from a minimum of 9 points (very explanatory study) to a maximum of 45 points (very pragmatically conducted study). It contains nine domains, which address the most relevant features of a trial, and was developed to support researchers in planning trial designs and to evaluate the impact of design decisions on applicability.^6,47,48 The domains include eligibility criteria, recruitment, setting, organisation, flexibility delivery, flexibility adherence, follow-up, primary outcome, and primary analysis.^48,49

We plotted proportions of matches vs RCT's sample sizes and their PRECIS-2 scores, and used regression analysis to quantify the association. We report P-values from the Wald test. Stata Statistical Software: Release 14 (2017; StataCorp. LLC, College Station, TX, USA) was used for data analysis, and Prism 8 was used for OS X Version 8.3.0 to draw the figures. We considered a two-sided P-value <0.05 statistically significant.

Results

A total of 2241 patients (mean age, 52 [standard deviation, 20] yr; 54% female; n=1215) were included for the assessment of trial eligibility between October 2015 and January 2016 (Table 3). Only a small proportion of patients matched the eligibility criteria for each of the 36 RCTs, ranging from 53% to 0%. The average proportion of patients matching the eligibility criteria for all 36 studies was 6.5% (95% confidence interval, 6.3–6.6; Fig 1). Less than 10% of patients were eligible for 31 studies. For 11 published studies, not a single patient from our cohort met the eligibility criteria. Supplementary Table S2 lists the most common inclusion and exclusion criteria of each of the 36 studies contributing most to trial ineligibility of the screening cohort.

Table 3.

Baseline characteristics of the study cohort assessed for trial eligibility. Data represent mean (standard deviation) or n (%). The study cohort includes all adult patients (≥18 yr) who were scheduled for elective noncardiac surgery during the study period. COPD, chronic obstructive pulmonary disease.

Age (yr)	52 (20)
Patient sex (female)	1215 (54)
Weight (kg)	76 (21)
Height (cm)	169 (12)
Comorbid diseases
Coronary artery disease	119 (5.1)
Congestive heart failure	39 (1.7)
Atrial fibrillation	110 (4.9)
Atrioventricular block	14 (0.6)
Bronchial asthma/COPD/bronchospasm	241(10.8)
Kidney disease	156 (7.0)
Liver disease	112 (5.0)
Cardiovascular disease risk factors
Hypertension	675 (30.1)
Type 1 and 2 diabetes mellitus	227 (10.1)
Smoking history	981 (43.8)
Severe obesity	155 (6.9)
Hyperlipidaemia	297 (13.3)
Regular beta-blocker intake	426 (19.0)

Fig. 1.

Proportions (with 95% Jeffreys intervals) of screened patients matching the eligibility criteria for each available RCT. The average proportion of matching patients was 6.5% (95% confidence interval, 6.3–6.6; red horizontal line). X-axis: individual RCTs in descending order of matches. Y-axis: proportion of matching patients (%).

The 10 most common criteria substantially contributing to trial ineligibility of the screening cohort across all 36 RCTs were type of surgery (used in 26 studies), current beta-blocker treatment (16 studies), coronary artery disease or cardiovascular risk profile (13 studies), asthma/COPD (13 studies), patient age (used as single criterion in 13 studies), congestive heart failure (10 studies), ASA physical status (nine studies), renal disease (six studies), arterial hypertension (used as single criterion in six studies), and diabetes mellitus (used as single criterion in four studies).

These factors contributed to trial ineligibility on average by: type of surgery: 95% (min–max, 16–100), coronary artery disease or cardiovascular risk profile: 76% (8–100), arterial hypertension: 50% (30–70), age: 35% (6–72), ASA physical status: 26% (7–26), current beta-blocker treatment: 19% (19–19), diabetes mellitus: 11% (11–89), asthma/COPD: 11% (11–11), renal disease: 7% (7–7), and congestive heart failure: 5% (5–5) (Supplementary Table S3).

The median PRECIS-2 score of the RCTs was 18 points (IQR, 17–20), indicating that the majority of RCTs were quite explanatory, less pragmatic studies. Figure 2 shows the nine domains of the PRECIS-2 tool and the PRECIS-2 scores of four representative RCTs included in the Cochrane meta-analysis. A higher PRECIS-2 score was associated with a higher proportion of matching patients (P<0.001; Fig 3). Individual study sample sizes were not related to trial eligibility (P=0.84; Fig 4).

Fig. 2.

PRECIS-2 scores of four representative studies included in the Cochrane meta-analysis. (a) The nine PRECIS-2 domains represent the explanatory/pragmatic level of a trial on the pragmatic to explanatory continuum, ranging from a minimum of 9 points (very explanatory level) to a maximum of 45 points (very pragmatic level). (b) PRECIS-2 scores of four RCTs, exemplifying the explanatory, less pragmatic design of studies included in the Cochrane meta-analysis. The median PRECIS-2 score of all 36 RCTs was 18 points (inter-quartile range, 17–20). PRECIS-2, PRagmatic–Explanatory Continuum Indicator Summary 2.

Fig. 3.

Proportions of matching patients vs PRECIS-2 scores of RCTs. A higher PRECIS-2 score was associated with a higher proportion of matching patients (P<0.001). X-axis: PRECIS-2 scores of RCTs. Y-axis: proportion of matching patients (%). PRECIS-2, PRagmatic–Explanatory Continuum Indicator Summary 2.

Fig. 4.

Eligibility proportions vs sample size of RCTs. Individual study sample sizes were not related to trial eligibility (P=0.84). X-axis: sample size of RCTs on a logarithmic scale. Y-axis: proportion of matching patients (%).

Discussion

This study was based on the inability to recruit patients into a perioperative beta-blocker trial at one of the largest tertiary care centres in Europe. The recruitment failure raised the concern that the generalisability of beta-blocker trial populations in a real-world cohort of patients may be limited. We analysed the restrictiveness and generalisability of the trial eligibility criteria of a large Cochrane meta-analysis on perioperative beta-blockers in patients undergoing noncardiac surgery in a real-world cohort. The proportions of matching patients were related to the sample size and the PRECIS-2 score of each RCT to determine the patients' detail matching.

Only a small number of patients matched the inclusion and exclusion criteria for each of the 36 RCTs. The sample size of individual studies was not related to trial eligibility. It may be questioned whether the clinical impact of a trial should mainly be based on its sample size, as frequently observed in scientific conversations, or additionally include a tool to rate the pragmatic and explanatory characteristics of a clinical trial design.

Using the PRECIS-2 tool we found that very pragmatically conducted studies, which are designed to be externally valid, are missing in the field of perioperative beta-blocker therapy. The focus on exploratory studies is a suitable explanation for the lack of applicability of available evidence to our screening cohort.

Current evidence suggests a possible increase in all-cause mortality and stroke rate with the perioperative use of beta-blockers in patients undergoing noncardiac surgery. Although more data are needed to draw a final conclusion, the conduct of future explanatory studies on the effect of perioperative beta-blocker use including only high-risk patients seems warranted. Even more important will be the standardised assessment and reporting of the PRECIS-2 score or equivalent tools^50,51 by trial investigators to support readers in appraising a trial's degree of pragmatism and help clinicians to estimate the applicability of study findings to their patients in clinical practice.⁵² Unfortunately, sufficient details on a study's pragmatic/explanatory level are rarely provided, which masks critical information on a study's validity and provokes the risk of misinterpretation.

The ratio of screened to included patients indicates the representativeness of study participants and is thus relevant to external validity (Consolidated Standards of Reporting Trials [CONSORT] 2010). A standardised reporting of a trial's screening phase, as suggested by STROBE (Strengthening the Reporting of Observational Studies in Epidemiology)⁵³ or CONSORT Statements 1996 and 2010,^54,55 however, is rarely given or even criticised as of low interest.^56,57 Only four of 36 trials included into the Cochrane meta-analysis report numbers of screened patients in the main text. Reports on screening failures are likewise rarely seen, which may generate distorted estimates owing to the selection of patient cohorts in studies with effects that cannot be directly quantified. This problem remains present throughout the conduct, analysis and interpretation of a trial. Sophisticated analytical models may attract much attention, further concealing limited generalisability of study results because of the cohort selection processes. Checklists have been shown to assess multiple types of validity improving internal consistency and generalisability of study results.⁵⁸ There is, however, an ongoing unmet need of standardised reporting on screening and screening failures in general and finding a feasible strategy which guarantees high quality of collected data by giving all available information concerning their origin.

Our study has several limitations. Our data suggest a substantial limitation in the generalisability of existing evidence for the use of beta-blockers in noncardiac surgery patients. It is obvious that this may apply to all clinical research to some extent. However, by looking at one sample topic, we cannot generalise our findings to other perioperative interventions and medical fields.

Furthermore, it should be noted that our findings are limited by the definition of eligibility criteria used for the failed RCT. A precise definition of inclusion and exclusion criteria is essential when planning a trial, to avoid unintentional exclusion of relevant patient groups and missing of possible treatment effects. In the failed trial, we applied the NYHA functional classification system to grade heart failure severity, for example. Use of NYHA is common in clinical practice. Its suitability as eligibility criterion, however, may be challenged into question given its limited objectivity⁵⁹ and poor discriminating ability across the spectrum of heart failure.⁶⁰ Applying a more differentiated phenotyping, as given by the 2016 European Society of Cardiology (ESC) guidelines,⁶¹ may allow including all relevant patient groups and limit bias of treatment effect estimates.

In addition, we used the Cochrane meta-analysis published by Blessberger and colleagues⁶² in 2014 as reference for the current evidence on the effect of perioperative beta-blockers in noncardiac surgery. Since 2018, an updated version of the analysis is available. However, the update excluded only one trial,⁴¹ which added only two outcome events to a total of 304 all-cause mortality events and thus did not strongly influence the results of the meta-analysis.

Finally, we did not systematically assess matches between our screening cohort and the sample populations actually included in the RCTs. Possible mismatches between patients who were eligible and those who were actually included, however, need to be considered when interpreting the eventual applicability of a given trial (Supplementary Fig. S1).

Conclusions

We found that trial eligibility criteria in perioperative beta-blocker therapy trials are overly restrictive and not generalisable to a real-world population. Despite the availability of high-quality evidence from a large Cochrane meta-analysis, the applicability of results from perioperative beta-blocker trials may be limited. This may be partly explained by a lack of pragmatic studies in this field. Systematic assessment and reporting of the applicability of trial results may allow clinicians for a better interpretation of data generalisability.

Authors' contributions

Study concept and design: MT, CK, AD, PN, HH.

Collection of patient data: MT, CK, DR, AD, PN.

Data analysis: MT, CK, AD, PN, HH.

Data interpretation: MT, MS, NB, CS, HH.

Statistical analysis: HH.

Writing of the first draft of the manuscript, and drawing of figures and tables: MS, NB, CS.

All authors critically revised the manuscript for important intellectual content and approved its current version. All authors agree to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The manuscript has not been previously published and is not under consideration for publication in the same or substantially similar form in any other peer-reviewed media.

Declarations of interest

The authors declare that they have no conflicts of interest.

Handling editor: Paul Myles

Appendix A. Supplementary data

The following is the Supplementary data to this article:

Multimedia component 1