Impact of differences in acute respiratory distress syndrome randomised controlled trial inclusion and exclusion criteria: systematic review and meta-analysis

Abstract

Background

Control-arm mortality varies between acute respiratory distress syndrome (ARDS) RCTs.

Methods

We systematically reviewed ARDS RCTs that commenced recruitment after publication of the American–European Consensus (AECC) definition (MEDLINE, Embase, and Cochrane central register of controlled trials; January 1994 to October 2020). We assessed concordance of RCT inclusion criteria to ARDS consensus definitions and whether exclusion criteria are strongly or poorly justified. We estimated the proportion of between-trial difference in control-arm 28-day mortality explained by the inclusion criteria and RCT design characteristics using meta-regression.

Results

A literature search identified 43 709 records. One hundred and fifty ARDS RCTs were included; 146/150 (97.3%) RCTs defined ARDS inclusion criteria using AECC/Berlin definitions. Deviations from consensus definitions, primarily aimed at improving ARDS diagnostic certainty, frequently related to duration of hypoxaemia (117/146; 80.1%). Exclusion criteria could be grouped by rationale for selection into strongly or poorly justified criteria. Common poorly justified exclusions included pregnancy related, age, and comorbidities (infectious/immunosuppression, hepatic, renal, and human immunodeficiency virus/acquired immunodeficiency syndrome). Control-arm 28-day mortality varied between ARDS RCTs (mean: 29.8% [95% confidence interval: 27.0–32.7%; I²=88.8%; τ²=0.02; P<0.01]), and differed significantly between RCTs with different Pao₂:FiO₂ ratio inclusion thresholds (26.6–39.9 kPa vs <26.6 kPa; P<0.01). In a meta-regression model, inclusion criteria and RCT design characteristics accounted for 30.6% of between-trial difference (P<0.01).

Conclusions

In most ARDS RCTs, consensus definitions are modified to use as inclusion criteria. Between-RCT mortality differences are mostly explained by the Pao₂:FiO₂ ratio threshold within the consensus definitions. An exclusion criteria framework can be applied when designing and reporting exclusion criteria in future ARDS RCTs.

Editor's key points.

• •In this systematic review and meta-analysis, the authors identified modifications to acute respiratory distress syndrome definitions that are used to specify trial inclusion criteria.
• •Variation in mortality between RCTs is accounted for by differences in selected Pao ₂:FiO₂ ratio thresholds.
• •Exclusion criteria between trials vary greatly, but can be adjudicated based on the rationale for selection.
• •This framework can be used to select exclusion criteria in future RCTs and when deciding whether results from an RCT are useful in patients excluded from the RCT.

Inclusion criteria in acute respiratory distress syndrome (ARDS) RCTs are usually based either on the American–European Consensus Conference (AECC) definition¹ or the Berlin definition,² which superseded the former. It is recognised that the difference in control-arm mortality between ARDS RCTs is related to the severity of hypoxaemia³ within the ARDS consensus definitions.^{1 , 2} However, there has been limited assessment of how components of the inclusion criteria that are not specified within the consensus definitions^{1 , 2} contribute towards the observed mortality differences between ARDS RCTs. These refinements of ARDS inclusion criteria are considered important for improving certainty of ARDS diagnosis. For example, the use of standardised ventilatory settings or confirmatory time periods before definitive diagnosis of ARDS has been proposed to select patients more likely to have ‘true’ ARDS.^{4 , 5}

Furthermore, exclusion criteria vary between ARDS RCTs, excluded patients do not explain the difference in control-arm outcomes, and justification⁶ of exclusion criteria used in ARDS RCTs has not been assessed to date.

In this context, we tested the hypotheses that variations in inclusion criteria not specified within the ARDS consensus definitions could contribute to the differences in control-arm mortality in ARDS RCTs, and that developing an exclusion criteria justification framework for ARDS RCTs would inform design of future RCTs. Using a systematic review and meta-analysis of ARDS RCTs, we assessed the concordance of the inclusion criteria reported in RCTs with the ARDS consensus definitions,^{1 , 2} and estimated the proportion of between-trial variance in control-arm 28-day mortality explained by differences in the inclusion criteria and RCT characteristics. We then used the justification framework reported by Van Spall and colleagues⁶ to assess the exclusion criteria reported in ARDS RCTs.

Methods

Review protocol

This systematic review and meta-analysis was prospectively registered with the International Prospective Register of Systematic Reviews (PROSPERO) (CRD42019089703) and conducted and reported in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) recommendations.⁷ The PRISMA checklist is available in the Supplementary data. We did not receive external funding.

Information sources

We searched MEDLINE, Embase, and the Cochrane central register of controlled trials from January 1, 1994 to October 31, 2020 with no language restrictions. We used subject headings and text-word terms to search for RCTs on ARDS and adults (Cochrane, McMaster, Robinson, and Dickersin clinical trial filters). The full MEDLINE electronic search strategy is presented in Supplementary Methods S1. We manually searched reference lists from published ARDS systematic reviews. Citations were saved in EndNote (Philadelphia, PA, USA) and duplicated records removed.

RCT selection

All citations were independently reviewed against our RCT selection criteria by at least two authors (RS, BA, and GM) using Rayyan QCRI.⁸ Potentially relevant full-text articles were reviewed and disagreements were resolved by consensus (RS, BA, and GM). We included RCTs in adult patients (>16 yr) with ARDS that commenced recruitment after publication of the AECC definition in March 1994. RCTs with a factorial design that were reported separately were considered as distinct RCTs. Crossover RCTs, where control-arm mortality could not be quantified, and articles published only in abstract form or foreign language were excluded. RCTs in patients with COVID-19 were also excluded. Further details of RCT selection criteria are available in Figure 1 .

Fig 1.

Preferred Reporting Items for Systematic Reviews and Meta-Analyses flowsheet showing RCT selection process. ARDS, acute respiratory distress syndrome.

Data items

A preliminary data extraction form was piloted on 20 randomly selected RCTs by RS and BA. Based on feedback from this process, variables in the final data collection form were then amended for inter-observer reliability. Data were independently extracted by RS, BA, and GM from the following domains: RCT design, patient characteristics, inclusion criteria, exclusion criteria, intervention tested, and all reported mortality outcomes. RCT design characteristics included type of intervention, sponsorship and funding, number of participating centres, World Bank country income group (first author), year of publication, and number of patients in control group.

Inclusion criteria were defined as any variable used for the ARDS case definition by the RCT; these included consensus ARDS definition cited, radiographic criteria, assessment of cardiac involvement, Pao ₂:FiO₂ (P:F) ratio threshold, PEEP threshold, and inclusion of invasively or noninvasively ventilated patients. All time criteria that specified inclusion into the RCT based on the duration of ARDS or ventilation were also extracted (time since intubation, time since onset of ARDS, time since symptom onset, and time since admission). In RCTs, where ARDS inclusion criteria were not explicitly listed, we assumed that criteria corresponded to the ARDS definition cited in the body of the text or references. Criteria limiting RCT eligibility of patients were treated as exclusion criteria and were extracted from the RCTs.

Risk of bias

RCTs were assessed for risk of bias for control-arm mortality outcomes using the Cochrane risk-of-bias tool.⁹ A funnel plot of standard error against control-arm mortality was used to assess for evidence of publication bias related to control-arm mortality (Supplementary Fig. S1). The analysis was not subsequently adjusted for bias.

Synthesis of results

We extracted 28-day control-arm mortality using GraphClick¹⁰ from uncensored cumulative mortality/survival curves, if not reported. In addition, 30-day mortality, reported in two RCTs, was combined with 28-day mortality for meta-analysis.

We assessed concordance of every element of the ARDS inclusion criteria reported to the corresponding consensus definitions cited by the authors. A summary of clinical criteria used to diagnose ARDS in consensus definitions is available in the Supplementary data. The P:F ratio thresholds reported in RCTs were regrouped into two categories: 26.6–39.9 and <26.6 kPa.^{1 , 2}

To inform an exclusion criteria framework for future ARDS RCTs, all exclusion criteria were categorised, as described previously by Van Spall and colleagues.⁶ Additional categories were specified for acute illness severity, and barotrauma, and an exclusion criteria category based on lower age limited was removed (in keeping with our RCT selection criteria). The category ‘related to female gender’ was recoded to ‘pregnancy related’. They were then classified into ‘strongly justified’ and ‘poorly justified’ based upon the rationale for their selection, as described by Van Spall and colleagues.⁶ Further details of the classification scheme are available in Supplementary Tables S3 and S4.

Meta-analysis model

To study the variability in control-arm mortality between RCTs, we used a random-effects meta-analysis model with control-arm 28-day mortality as the dependent variable and a random intercept for each RCT. Distribution of control-arm mortality was assessed for normality using a quantile–quantile plot (Supplementary Fig. S3). As control-arm mortality was not normally distributed, proportions were transformed using the Freeman–Tukey double arc-sine method. Each RCT was weighted by the inverse of the sampling variance. A maximum likelihood estimator was used to estimate mean mortality (random-effects pooled estimate), between-trial standard deviation attributable to heterogeneity (τ), and the percentage of variance attributable to heterogeneity rather than chance (I ²). Subgroup analyses were conducted for all ARDS RCT inclusion criteria (listed in Table 1 ) and specific RCT characteristics (year of publication, single vs multicentre, first author World Bank country income group, and sample size). To estimate the proportion of between-trial variance (R ²) explained by ARDS inclusion criteria, they were all included as predictors in a meta-regression model. The RCT characteristics were treated as categorical predictors (as specified in Table 1) and were added to estimate additional impact.¹¹ Variance inflation factors were calculated for all predictors in the model to exclude multicollinearity. Model robustness was tested using a permutation test.

Table 1.

RCT design characteristics and specified inclusion criteria. ARDS, acute respiratory distress syndrome; CXR, chest radiograph; LA, left atrial; PAWP, pulmonary artery wedge pressure; P:F, Pao₂:FiO₂. ∗Inclusion criteria were not assessed individually for the four RCTs that used the composite Murray score to include patients.

A. RCT design characteristics	Number (%) of RCTs, n=150
Type of intervention
Mechanical ventilation strategies and respiratory care	63 (42.0)
Pharmacological RCTs	80 (53.3)
Fluid management strategies	7 (4.7)
Sponsorship and funding
Government or institutional funding	80 (53.3)
Partial/complete industry funding	36 (24.0)
Unknown	34 (22.7)
Number of participating centres
Single	67 (44.7)
Multiple	83 (55.3)
National	31 (20.6)
International	52 (34.7)
World Bank country income group (first author)
Middle	37 (24.7)
High	113 (75.3)
Year of publication
1996–2000	15 (10.0)
2001–2005	24 (16.0)
2006–2010	33 (22.0)
2011–2015	38 (25.3)
2016–2020	40 (26.7)
Patients in control group
0–25	61 (40.7)
26–50	33 (22.0)
51–100	16 (10.7)
101–200	20 (13.3)
>200	20 (13.3)
ARDS definition
American–European Consensus Conference	118 (78.7)
Berlin	28 (18.7)
Murray score	4 (2.6)
Invasively ventilated patients only
Yes	142 (94.7)
No	6 (4.0)
Unclear	2 (1.3)
B. Inclusion criteria∗
P:F ratio (maximum threshold; kPa)
26.6–39.9	64 (43.8)
<39.9	4 (2.7)
<250	1 (0.7)
<225	63 (42.5)
<26.6	1 (0.7)
<26.6	11 (8.2)
<22.6	1 (0.7)
<20.0	1 (0.7)
<6.7 for >3 h or P:F <10.6 for >6 h
<26.6 or 39.9 depending on PEEP
Minimum P:F ratio threshold specified	10 (6.9)
Minimum PEEP (cm H2O) specified
5	40 (24.7)
>5	10 (7.5)
Description of radiographic findings
Bilateral infiltrates on CXR	112 (76.7)
In three or four quadrants	6 (4.1)
Bilateral infiltrates on CXR or CT	27 (18.5)
Non-aerated lung parenchyma on CT	1 (0.7)
Exclusion of cardiac involvement
No clinical evidence of LA hypertension or PAWP <18 mm Hg	121 (82.9)
Not explained by cardiac failure or overload	25 (17.1)
Illness duration before enrolment specified
Time since symptom onset	34 (23.3)
Time since ARDS onset	66 (45.2)
Time since intubation (maximum)	29 (19.9)
Time since intubation (minimum)	7 (4.8)
Time since admission	3 (2.1)
Confirmatory time period specified	20 (13.7)
FiO2 specified	12 (8.2)

Sensitivity analysis

We report three sensitivity analyses. First, to assess whether lung-protective strategy was independently associated with mortality, we included this variable in our meta-regression model. As RCTs did not always explicitly state the use of a lung-protective strategy, we assumed that RCTs commencing recruitment after the publication of the ARDS Network lower vs higher tidal volume study used a lung-protective strategy and vice versa. Second, we repeated the meta-regression with alternative mortality time points (i.e. ICU, hospital, and 60-day mortality) as an independent variable. Third, we excluded RCTs that did not specifically use a P:F ratio inclusion threshold of 26.6 or 39.9 kPa.

For all analyses, a P-value of <0.05 was considered significant. Analyses were performed in R version 3.4.2¹² (R Foundation for Statistical Computing, Vienna, Austria) using the tidyr,¹³ dplyr,¹⁴ meta,¹⁵ and forestplot ¹⁶ packages.

Results

RCT selection

The bibliographic database search identified 43 709 records, as of October 31, 2020. After excluding duplicates, amongst the 37 756 records screened, 304 records were eligible for full-text evaluation. After full-text evaluation, we excluded 157 records and included 147 RCTs from the database search and three further RCTs from hand searching published review articles, resulting in 150 unique ARDS RCTs published between 1994 and 2020 that met our selection criteria (Fig. 1). A summary of RCT characteristics is reported in Table 1.

Risk of bias within RCTs

One hundred and four (69.3%) of 150 RCTs, including all RCTs of mechanical ventilation and respiratory care, were at high risk of bias attributable to inadequate blinding of participants or personnel. Although we included only RCTs in our study, 58 (38.7%) of 150 RCTs either did not or inadequately described the methods for allocation concealment and random sequence generation. Risk of bias was unclear in these RCTs. Lack of blinding of outcome assessors was not deemed to introduce bias, as mortality is an objective outcome. Twenty-one (14.0%) RCTs that did not report mortality outcomes were adjudicated to have a high risk of reporting bias, whereas 13 (8.7%) RCTs that did not specify the mortality time point were felt to be at unclear risk (Fig. 2 ).

Fig 2.

Risk of bias for mortality outcomes in ARDS RCTs. Domains were specified and adjudicated as per the Cochrane risk-of-bias tool. RCTs were divided by intervention type and ordered by year of publication. ARDS, acute respiratory distress syndrome.

Assessment of concordance of inclusion criteria to ARDS consensus definitions

The AECC definition was used in 117 of 150 (78.0%),17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133 the Berlin definition in 29 of 150 (19.3%),134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162 and the composite Murray score¹⁶³ in four RCTs164, 165, 166, 167 as ARDS inclusion criteria, resulting in 146 RCTs, where components of inclusion criteria were assessed further. Only three of 32131, 132, 133 RCTs that commenced recruitment of patients after publication of the Berlin definition did not use it to specify RCT inclusion criteria. The most commonly reported P:F ratio inclusion thresholds, below which patients were included in the RCT, were ≤26.6 kPa (63/146; 43.2%) and ≤39.9 kPa (64/146; 43.8%) (Table 1). The P:F ratio maximum threshold was concordant with the Berlin definition in 26/29 (89.7%) RCTs134, 135, 136 ^, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149 ^{, 151 , 152 ,} 154, 155, 156, 157, 158, 159, 160, 161 compared with 101/117 (86.3%) RCTs18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34 ^{, 36 ,} 38, 39, 40, 41, 42, 43, 44 ^{, 46 ,} 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74 ^{, 76 ,} 78, 79, 80, 81, 82, 83, 84, 85, 86 ^, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109 ^{, 111 , 112 ,} 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130 ^{, 133 , 137} that used the AECC definition. Bilateral chest infiltrates on the chest radiograph (CXR) were used in 118/146 (80.8%) RCTs.17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112 ^, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133 ^{, 137 , 138} The CXR or CT findings (as per the Berlin definition) were used in 27/146 (18.5%) of RCTs.134, 135, 136 ^, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162 Exclusion of cardiac involvement conformed to the AECC definition (absence of clinical evidence of left atrial hypertension or pulmonary arterial wedge pressure >18 mm Hg) in 121/146 (82.9%) RCTs,17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133 ^, 137, 138, 139 ^{, 143} and was consistent with the Berlin definition (not explained by cardiac failure or overload) in the remaining 25/146 (17.1%) RCTs.134, 135, 136 ^, 140, 141, 142 ^, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162 In accordance with the Berlin definition, a minimum PEEP of 5 cm H₂O was specified in all 29 RCTs using this definition. The AECC definition does not specify a minimum PEEP value, and 96/117 (82.1%) RCTs17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34 ^, 39, 40, 41, 42, 43 ^, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60 ^, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76 ^, 78, 79, 80, 81, 82, 83, 84, 85, 86 ^, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102 ^{, 106 , 107 , 109 ,} 111, 112, 113, 114, 115, 116, 117, 118, 119 ^, 121, 122, 123, 124 ^, 127, 128, 129, 130 ^{, 132 , 133} that used this definition did not specify a minimum PEEP value. Time since symptom onset was specified in 34/146 (23.3%) RCTs,^{22 , 27 , 42 , 75 , 80 ,} 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162 including all 29 RCTs, where the Berlin definition was used (median: 7 days; range: 3–28 days).

Assessment of deviation of inclusion criteria from ARDS consensus definitions

Deviations from the consensus criteria for P:F ratio were P:F <20.0 kPa (11/146; 7.5%),^{35 , 37 , 45 ,} 47, 48, 49 ^{, 77 , 87 , 113 , 150 , 153} P:F <33.3 kPa (4/146; 2.7%),^{75 , 110 , 132 , 162} P:F <26.6 kPa with PEEP >5 cm H₂O or P:F <39.9 with PEEP >10 cm H₂O (1/146; 0.7%),⁴⁶ P:F <29.9 kPa (1/146; 0.7%),¹⁸ P:F <22.6 kPa (1/146; 0.7%),¹⁰⁵ and P:F <6.7 kPa for >3 h or P:F <10.6 kPa for >6 h (1/146; 0.7%).¹³¹

Ten of 146 (6.8%) RCTs specified an additional minimum P:F ratio threshold below which patients were excluded.^{25 , 53 , 57 , 88 , 104 , 106 , 128 , 139 , 145 , 157} This minimum threshold varied from 8.0 to 26.6 kPa. Twelve (8.2%) RCTs specified standardised FiO₂ settings that ranged between 0.5 and 1.0.^{36 , 42 , 47 , 48 , 53 , 61 , 62 , 110 , 125 , 131 , 139 , 158}

Deviation from radiological criteria was noted in six (4.1%) RCTs^{17 , 18 , 52 , 53 , 67 , 81} that stipulated the presence of infiltrates in more than three or four CXR quadrants, and one (0.7%) RCT¹¹³ that used CT criteria only. Four (2.7%) RCTs137, 138, 139 ^{, 143} that used the Berlin definition instead used criteria from the AECC definition to exclude cardiac involvement. Amongst RCTs using the AECC definition, 22/117 (18.8%)35, 36, 37 ^, 45, 46, 47, 48, 49 ^{, 61 , 77 , 87 , 103 , 104 , 110 , 120 , 125 , 126 , 131} specified a minimum value for PEEP. In RCTs using the Berlin definition, 3/29 (10.3%)^{147 , 150 , 158} specified PEEP >5 cm H₂O. One hundred and seventeen of 146 RCTs (80.1%) specified inclusion criteria related to illness duration before enrolment. Time since ARDS onset was specified in 66/146 (45.2%; median: 2 days; range: 1–7) RCTs.17, 18, 19, 20, 21 ^, 28, 29, 30, 31, 32, 33, 34 ^{, 36 ,} 42, 43, 44, 45 ^{, 48 , 49 , 52 , 53 , 61 , 62 ,} 64, 65, 66, 67 ^, 71, 72, 73, 74, 75, 76, 77, 78 ^{, 81 , 84 , 85 , 87 , 89 , 92 , 93 , 96 ,} 102, 103, 104 ^{, 106 , 107 , 110 , 112 , 118 , 119 , 122 , 123 , 130 ,} 135, 136, 137, 138 ^{, 141 , 142 , 147 , 148 , 150 , 153 , 161} Maximum time since intubation was specified in 29/146 (19.9%; median: 3 days; range: 1–10) RCTs.^{24 , 35 , 44 , 47 , 53 , 54 , 60 , 61 , 63 , 64 , 90 , 92 , 93 , 105 ,} 113, 114, 115, 116 ^{, 124 , 126 , 127 , 131 , 132 , 134 , 136 , 141 , 145 , 150 , 152 , 162} Minimum time since intubation was specified in seven (4.8%; median: 1 day; range: 0.33–1) RCTs.^{35 , 69 , 70 , 91 , 105 , 113 , 140} Time since admission was specified in three (2.1%; median: 2 days; range: 2–2) RCTs.^{51 , 56 , 59}

Twenty (13.7%) RCTs^{24 , 25 , 31 , 37 , 42 , 48 , 49 , 90 , 94 , 98 , 105 ,} 114, 115, 116, 117 ^{, 120 , 131 , 132 , 153 , 158} stipulated a confirmatory time period, between 30 min and 24 h, during which ARDS needed to persist for enrolment into the trial. Eight of these RCTs reported a confirmatory time period between 12 and 24 h. Although 94.7% (142/150) of RCTs were conducted in invasively ventilated patients, six (4.0%) included patients receiving noninvasive ventilation^{33 , 57 , 128 , 139 , 151} and two (1.3%) did not clearly state how patients were being ventilated.^{79 , 133}

Assessment of between-trial variance (I²) in control-arm 28-day mortality and impact of differences in inclusion criteria

Mortality outcomes were reported in 125 of 146 (85.6%) RCTs (Supplementary Fig. S2). The 28-day mortality was reported in 73 (50.0%) RCTs, 30-day mortality in two, and extrapolated from a further nine RCTs using GraphClick, resulting in 84 RCTs with primary outcome for meta-analysis. There was significant variance in control-arm mortality at 28 days between RCTs, with a random effects estimated mean mortality of 29.8% (95% confidence interval [CI]: 27.0–32.7%; range: 3.6–69.7%; I ²=88.8% (84.4–92.3%); τ²=0.015 (0.011–0.023); P<0.01) (Fig. 3 a; Supplementary Fig. S4).

Fig 3.

(a) Univariate, weighted analysis of the impact of inclusion criteria and RCT characteristics on 28-day mortality. (b) Results of a meta-regression model of the impact of inclusion criteria and RCT characteristics on 28-day mortality. Total variance is made up of between- and within-study variance. Within-study variance cannot be explained using study-level factors. I² is the proportion of variance in ARDS RCTs that is attributable to between-trial variance. The proportion of between-trial variance that can be explained (R²) is 30.6%; inclusion criteria (28.8%), and RCT design characteristics (1.8%); 69.4% of between-trial variance remained unexplained. ARDS, acute respiratory distress syndrome; CI, confidence interval; CXR, chest radiograph; P:F, Pao₂:FiO₂; LA, left atrial; PAWP, pulmonary artery wedge pressure.

The 28-day mortality in RCTs that used a P:F ratio inclusion threshold of ≤26.6 kPa (35.1%; 95% CI: 31.8–38.4%) was higher compared with RCTs using a P:F ratio inclusion threshold between 26.6 and 39.9 kPa (25.7%; 95% CI: 21.9–29.7%). Mean mortality did not significantly differ between RCTs with other additional inclusion criteria differences: specification of minimum P:F ratio threshold below which patients were excluded, definition of imaging findings, definition for exclusion of cardiac involvement, time windows for inclusion (symptom onset, ARDS onset, intubation, admission, and confirmatory time period), and specification of PEEP >5 cm H₂O or FiO₂ (Fig. 3a).

Within-trial variance that cannot be explained by study-level variables accounted for 11.4% (95% CI: 7.7–16.0%) of total variance. Therefore, 88.6% (95% CI: 84.0–92.3%) of total variance was a result of between-trial variance (I ²). A meta-regression model, including all inclusion criteria variables and RCT design characteristics, accounted for a total of 30.6% (95% CI: 17.7–43.5%) of the between-trial variance (R ²).

The P:F ratio inclusion threshold was the only variable in the model significantly associated with control-arm 28-day mortality (P<0.01). Therefore, 69.4% (95% CI: 56.5–82.3%) of the between-trial variance remained unexplained by inclusion criteria differences (Fig. 3b; Supplementary Tables S5 and S6).

Assessment of exclusion criteria and justification framework

Of 150 RCTs, 141 (94.0%) reported exclusion criteria. Of the nine RCTs that did not report exclusion criteria, five were published after the Consolidated Standards of Reporting Trials (CONSORT) 2010 guidelines.¹⁶⁸ Definitions of exclusion criteria varied greatly between RCTs (Supplementary Table S4). For example, hepatic disease was variably defined within RCTs based on specific factors, including bilirubin level, transaminase concentration, Child–Pugh grade, and evidence of cirrhosis, or non-specifically as chronic liver disease or acute liver failure (Supplementary Table S4).

The exclusion criteria reported in ARDS RCTs were then categorised based on the justification framework proposed by Van Spall and colleagues⁶ (Fig. 4 ). To do this evaluation, we had to modify this framework in four ways: (i) by including two additional but important ARDS specific categories: acute illness severity and barotrauma; (ii) changing the domain ‘related to female gender’ to ‘pregnancy related’ to avoid conflating the two issues; (iii) by not including a lower age limit category, as we were assessing adult ARDS RCTs; and (iv) by not including the ‘potentially justified’ category. As defined by Van Spall and colleagues,⁶ potentially justified criteria relate to potential patient non-adherence to intervention or follow-up that cannot be classified as poorly/strongly justified. We did not identify any potentially justified criteria in ARDS RCTs, and therefore opted to simplify the justification framework.

Fig 4.

Categories of exclusion criteria reported in ARDS RCTs. ARDS, acute respiratory distress syndrome; HIV/AIDS, human immunodeficiency virus/acquired immunodeficiency syndrome.

The exclusion criteria in RCTs with strong justification were comorbidities neurological (73/78; 93.6%), respiratory (70/70; 100%), cardiac (55/55; 100%), life expectancy (long-term prognosis [33/33; 100%]; short-term prognosis [41/41; 100%]), acute illness related (acute illness severity [58/58; 100.0%]; barotrauma [34/34; 100%]), participation in another trial (55/55; 100%), inability to obtain consent (46/46; 100%), and medication related (45/45; 100%) (Fig. 4; Supplementary Table S3). The exclusion criteria in RCTs with poor justification were pregnancy related (76/83 [80.5%]; upper age limit [17/17; 100%]) and the following comorbidities: infectious/immunosuppression (50/50; 100%), hepatic (49/61; 80.3%), renal (27/34; 79.4%), and human immunodeficiency virus/acquired immunodeficiency syndrome (15/16; 93.8%) (Fig. 4; Supplementary Table S4).

Sensitivity analysis

Between-trial and overall relationships between inclusion criteria and mortality remained consistent and unchanged after the inclusion of lung-protective strategy as a variable in the meta-regression model. Similarly, meta-regression with mortality time points (28-day/60-day/ICU/hospital mortality) as an independent categorical variable, or exclusion of RCTs that used a P:F ratio inclusion threshold other than 26.6 or 39.9 did not alter findings. Univariate and multivariate models for these analyses are reported in Supplementary Figures S5–S7.

Discussion

Although 97% of ARDS RCTs use consensus definitions for inclusion criteria, we observed deviations from the consensus definitions that were primarily aimed to increase the certainty of ARDS diagnosis, such as duration of hypoxaemia after ventilation, in more than 80% of ARDS RCTs. Importantly, the transition from the AECC definition to the Berlin definition has improved overall concordance between individual elements of the RCT inclusion criteria and the consensus definition. As predicted, there was significant variance in control-arm mortality. However, these additional refinements of ARDS inclusion criteria did not explain the observed variance, over and above the effect of severity of hypoxaemia on control-arm mortality. We illustrate that the justification framework reported by Van Spall and colleagues⁶ is a feasible tool to assess the exclusion criteria reported in ARDS RCTs, the tool requires modification, and that many exclusion criteria in ARDS RCTs were poorly justified. We provide a novel assessment of the concordance of inclusion criteria in ARDS RCTs to the ARDS consensus definitions and its impact on control-arm mortality.

For assessing the impact of inclusion criteria differences, we explored how consensus definitions were modified as ARDS RCT inclusion criteria and assessed individual contributions to outcome. Our finding that there is significant heterogeneity in control-arm mortality is consistent with a previous meta-analysis that included RCTs between 1984 and 2006.³ Importantly, deviations from the consensus criteria, such as use of standardised ventilator settings for PEEP and FiO₂ when defining ARDS,^{5 , 169 , 170} delayed reassessment of patients to confirm ARDS diagnosis (confirmatory period),^{171 , 172} and classification based on time of onset into early or late ARDS¹⁷³ was thought to improve ARDS diagnosis or enrich for adverse outcome. These had negligible impact on 28-day mortality over and above the P:F ratio strata. Our analyses highlight that severity of hypoxaemia is the only inclusion criterion that reliably stratifies ARDS patients by risk of death, which is consistent with the predictive validity analyses reported in the Berlin ARDS definition.

We also highlight that the justification framework proposed by Van Spall and colleagues⁶ requires modification for use in ARDS RCTs. Based on our analysis, we propose two classes of exclusion criteria, namely, strongly justified and poorly justified, and that exclusion criteria could be linked to the intervention tested. This proposal has validity, as exclusion criteria that we categorised as poorly justified in ARDS RCTs have also been highlighted in non-critical care RCTs.^{6 , 174 , 175} For example, instead of blanket exclusion of women of reproductive age attributable to pregnancy-related risk, the framework we propose would consider whether excluding pregnant patients was strongly or poorly justified for the specific intervention that is being trialled. If the intervention does not have teratogenic properties (e.g. mechanical ventilation), then pregnancy need not be an exclusion. Extending this argument, we could also consider whether results from a trial are useful when managing patients with an exclusion criterion in that trial. This issue is seldom considered during clinical practise.

Aside from eligibility criteria, heterogeneity in mortality outcomes between RCTs can be attributed to setting and design characteristics, such as geographic and socio-economic population differences,176, 177, 178 unreported exclusion of patients,¹⁷⁹ and differences in post-randomisation care between RCTs.¹⁸⁰

Strengths and limitations

We report the first assessment of ARDS RCT eligibility criteria as a reason for between-trial differences in control-arm mortality. We pre-registered with PROSPERO and reported in accordance with PRISMA recommendations. Our exclusion criteria framework could be used for RCT design and prospective reporting. This would inform whether the results of a specific ARDS RCT would apply to patients with one or more of the RCT exclusion criteria, but without a known risk of harm from the intervention.

We applied the Cochrane risk-of-bias tool to explore bias between RCTs, but did not alter our analysis to account for this. This was because the primary outcome, control-arm mortality, is unlikely to be influenced by risk of bias. As we do not know which exclusion criteria were specifically met by patients excluded from RCTs, we were unable to assess their impact on mortality. We used year of publication rather than year of enrolment as the time variable in our analysis because the latter was not always reported. In a proportion of RCTs, where individual components of the inclusion criteria were not specified, inclusion criteria were assumed to correspond to the stated definition. We specifically focused on 28-day mortality, which was not available in 66 RCTs, and grouped RCTs by P:F ratio inclusion threshold (26.6–39.9 and <26.6) to assess concordance with consensus definitions. Both issues are addressed within our sensitivity analyses, which were consistent with our main analysis.

We need to ascertain whether our modified exclusion criteria framework is implementable in future ARDS RCTs. Furthermore, it will be important to consider how to record multiple exclusion criteria in a single patient, within the CONSORT framework, which can be explored using a clinical database. Various definitions of exclusion criteria can be simulated to examine their impact upon an RCT population.

Conclusion

In most ARDS RCTs, the consensus definitions are modified to use as inclusion criteria. Most of the between-RCT differences in mortality are accounted for by the P:F ratio threshold within the consensus definitions. Exclusion criteria definitions were different between RCTs. We provide a simplified exclusion criteria framework to be applied when designing and reporting exclusion criteria in future ARDS RCTs.

Authors' contributions

Study conception/design: RS, MS-H

Development of search strategy: RS, MS-H

Literature review: RS, BA, GM

Data extraction: RS, BA, GM

Data analysis: RS

Data interpretation: all authors

Drafting of paper: RS, MS-H

Critical revision and approval of paper: all authors

All authors confirm the accuracy and integrity of the work. RS takes responsibility for the integrity of the work as a whole, from inception to published paper.

Handling editor: Jonathan Hardman

Declarations of interest

The authors declare that they have no conflicts of interest.

Funding

Wellcome Trust to CS; Medical Research Council to CS; National Institute of Health Research Cambridge Biomedical Research Centre to CS; National Institutes of Health to CS; British Heart Foundation to CS; GlaxoSmithKline to CS; AstraZeneca to CS; Bristol Myers Squibb to CS; National Institute of Health Research Clinician Scientist Award (NIHR-CS-2016-16-011) to MS-H.

Appendix A. Supplementary data

The following is the Supplementary data to this article: