Acute Respiratory Distress Syndrome after Trauma: Development and Validation of a Predictive Model

Abstract

Objective

To determine early clinical predictors of Acute Respiratory Distress Syndrome (ARDS) after major traumatic injury and characterize the performance of this ARDS prediction model, and two previously published ARDS prediction models, in an independent cohort of severely injured patients.

Design

Prospective cohort study

Setting

University-affiliated level I trauma center in Seattle, WA, and nine hospitals participating in the Inflammation and Host Response to Injury Consortium.

Patients

Model derivation utilized data from 224 patients participating in a randomized controlled trial. All models were validated in an independent cohort of 1,762 trauma patients.

Measurements and Main Results

Variables strongly associated with ARDS in bivariate analysis (p<0.01) were entered into a multiple logistic regression equation to generate an ARDS predictive model. We evaluated the performance of all models using the area under the receiver operator characteristic (ROC) curve. ARDS occurred in 79 subjects (35%) belonging to the development cohort and in 423 subjects (24%) from the validation cohort. Multivariable predictors of ARDS after trauma included subject age, Acute Physiology and Chronic Health Evaluation (APACHE) II Score, injury severity score, and the presence of blunt traumatic injury, pulmonary contusion, massive transfusion, and flail chest injury (area under the ROC curve 0.79 [95% C.I. 0.73, 0.85]). Validation of the prediction model resulted in an area under the ROC curve of 0.71 (95% C.I. 0.68, 0.74). Our model's performance in the validation cohort was superior to that of two other published ARDS prediction models (0.65 [95% C.I. 0.63, 0.68] and 0.66 [95% C.I. 0.64, 0.69], p<0.01 for all comparisons).

Conclusions

Using routinely available clinical data, our prediction model identifies patients at high risk for ARDS early after severe traumatic injury. This predictive model could facilitate enrollment of subjects into future clinical trials designed to prevent this serious complication.

Introduction

Injury, a leading cause of death across all age groups, remains the number one cause of death in U.S. citizens age 1 – 44 years old (1). Deaths directly attributable to severe trauma often occur within hours of injury, even before hospitalization (2-4). Trauma victims who survive their initial injuries to hospitalization in the intensive care unit (ICU) face the possibility of life-threatening complications such as multiple organ failure (MOF) (2-4), the leading cause of death in these patients (5). Acute respiratory distress syndrome (ARDS) is the most frequent manifestation of MOF after trauma (6), occurring in 12-25% of injured patients (7-9). While studies differ on the mortality attributable to ARDS in trauma patients (7, 10, 11), injured patients with ARDS and MOF have mortality rates as high as 50-80% (6, 10, 12). Moreover, ARDS is independently associated with longer hospital stays, increased costs, and worse long-term health related quality of life in trauma patients (13-15). Interventions which could prevent ARDS carry the potential to reduce the substantial morbidity, mortality, and resource utilization associated with this syndrome (9).

Progress towards identifying potentially causal and modifiable factors that could lead to the development and testing of preventative ARDS therapies has been slow in part because of an incomplete understanding of which patients are likely to develop ARDS after major trauma. Altogether, few published models predicting trauma-associated ARDS exist, with early predictive models suffering from the application of inconsistent ARDS definitions (16-19). Recently, Miller et al. studied blunt trauma victims surviving to 24 hours after injury and hospitalized in the ICU at a single center (20). The authors identified several ARDS predictors including an injury severity score (ISS) greater than 25, age greater than 65 years, and the presence of hemorrhagic shock (admission systolic blood pressure < 90 mm Hg), a pulmonary contusion, and a massive red blood cell (RBC) transfusion requirement (>10 RBC units over the first 24 hours of hospitalization). They reported an area under the receiver operating characteristic (ROC) curve of 0.80. Another study by Navarrete-Navarro and colleagues focused on identifying early predictors of ARDS among trauma victims hospitalized at several centers in Spain. Using a stepwise approach, Acute Physiology and Chronic Health Evaluation (APACHE) II Score on ICU admission, the number of RBC units required in the first 24 hours of hospitalization, and presence of a femur fracture, and major chest trauma (rib/sternal fractures) emerged as significant predictors (21). These authors reported an area under the ROC curve of 0.76. Although both models displayed promising discriminatory abilities, neither model was validated using an independent cohort of patients (22-27).

In this study, we sought to develop and validate a predictive instrument for ARDS after major trauma and compare our models performance to two other published predictive models (20, 21) using the same independent validation cohort. We hypothesized that objective clinical factors available early after trauma could discriminate between who would and would not develop ARDS, and that the model would maintain robust predictive ability when validated in an independent cohort of trauma patients.

Methods

Study Design, Setting, and Description of the Derivation Cohort

The derivation cohort consisted of severely injured trauma patients who participated in a randomized controlled trial, conducted between 2003-2004, designed to evaluate the effect of leukoreduced versus standard blood transfusions on post-traumatic infection (28). The trial demonstrated no effect of treatment on the development of infections or ARDS (28, 29). All study subjects were located at Harborview Medical Center, a level I trauma center affiliated with the University of Washington. Inclusion criteria were age >17 years and RBC transfusion within 24 hours of injury; full details regarding trial methods are otherwise provided elsewhere (28, 29). Of the available 268 subjects, we included those surviving without documented ARDS to 24 hours after injury and requiring hospitalization in the ICU in the derivation cohort.

Description of the Validation Cohort

The validation dataset consisted of an independent cohort of severely injured trauma patients prospectively assembled as part of a Large-Scale Collaborative Project Award from the National Institutes of Health, National Institute of General Medical Sciences (U54-GM62119, Inflammation and the Host Response to Injury). Eligibility criteria were: blunt traumatic injury in all age groups not exclusive to the head with evidence of shock within 60 minutes of emergency department arrival (systolic blood pressure <90 or base deficit < -6), a requirement for blood transfusion within 12 hours of injury, at least a partially-intact cervical spinal cord, absence of severe traumatic brain injury (Abbreviated Injury Scale for the head < 4 or Glasgow Coma Score [GCS] motor > 3 within 24 hours of injury), and arrival to the hospital within 6 hours of injury. Subjects were enrolled at 9 participating hospitals, all of which are level I trauma centers, with all data reviewed and validated by a central data processing core and subsequently made publically available in a de-identified fashion. Additional details regarding participating centers and the aims of this research study are provided at www.gluegrant.org. As the study is ongoing, we used all available de-identified subject data from study onset (2003) to December, 2010. To ensure model validation was performed in the at-risk cohort of interest, we excluded subjects < 18 years of age, those not surviving to 24 hours after injury or with documented ARDS within this timeframe, and those not requiring ICU care. There was no overlap of enrolled subjects between the validation and derivation cohorts.

Data Collection, Quality, and Variable Definitions

Both datasets contained prospectively collected data abstracted from hospital medical records including demographic, comorbidity, physical exam, resuscitation, and a priori outcome data. Data quality was ensured by the training of research nurses in the recognition of ARDS, as well as the use of trained data entry personnel and regular internal audits. ARDS was defined according to the American-European consensus conference criteria (30). The ISS, GCS, and APACHE II Score, calculated at the time of emergency department presentation, were available in each dataset. Specific injury characteristics were defined using International Classification of Diseases, Ninth Revision (ICD-9) diagnosis codes present in each database. The following case definitions were used: femur fracture (820.x and/or, 821.x), pelvic fracture (808.x excluding 808.2 [stable pelvic fracture]), flail chest (807.4), and pulmonary contusion if the subject had an ICD-9 code representing a direct chest injury including 3 or more rib fractures (807.13 - 807.19), and/or flail chest injury (807.4), and/or a lung contusion/laceration (861.x). Finally, we collapsed exposure to platelet, fresh frozen plasma, and/or cryoprecipitate transfusions into a single categorical exposure term, a high plasma volume component (HPVC) transfusion, as recent evidence suggests HPVCs are an important ARDS risk factor (31). Variables in the models developed by Miller et al. and Navarrete-Navarro et al. were defined according to their published definitions (20, 21).

Model Development and Validation

A multivariable model predicting trauma-associated ARDS was constructed in three steps: assessment and appropriate modeling of each candidate variable, construction of a parsimonious model, and internal validation of the developed model using a bootstrap approach followed by external validation in an independent validation cohort. Candidate predictor variables were selected with their best format (continuous, categorical) determined after graphical analysis, balancing their utility according to published evidence, generalizability, and clinical availability (32, 33). We performed descriptive bivariate comparisons between all potential predictors of interest and ARDS outcome using chi² t tests, or Wilcoxon analysis as appropriate. To avoid introducing predictive optimism, we maintained a predictor to outcome ratio at 1:10 (24, 34) in the development cohort. Accordingly, after determining best format, candidate predictors with a p-value ≤ 0.01 in bivariate analysis were included in the initial model (32). We constructed the multivariable logistic regression model using a backward selection approach that minimized the Akaike Information Criterion (AIC). The AIC is a likelihood-based measure of model fit penalizing models with large numbers of variables in an attempt to reduce overfitting (35). We considered adding interaction terms between massive transfusion and blunt injury as well as between subject age and APACHE II score, age and ISS, and age and massive transfusion status using the likelihood ratio test to determine statistical importance to model fit. After the model was derived, we used the bootstrap to internally validate the final model by sampling with replacement for 1,000 iterations. The model was fit on each bootstrap sample by repeating the stepwise selection algorithm followed by evaluation on the original cohort to estimate the degree of predictive deterioration ascribed to sampling bias when fit to the validation cohort (36). We quantified model discrimination using the area under the ROC curve (37). Model calibration was assessed several ways. First, we used the Hosmer-Lemeshow (HL) goodness-of-fit statistic, with p <0.10 indicating fit was inadequate (32, 38). After internal validation, calibration was also assessed across models by generating plots of the observed compared to the expected probability of ARDS across deciles of predicted ARDS risk. Lastly, we compared the aggregate number of observed and predicted ARDS cases in the validation cohort by dividing observed by mean predicted ARDS cases to calculate the standardized ARDS ratio, using a chi-square test to statistically determine their respective equivalence. After fitting our new study model to the validation cohort, and those developed by Miller (20) and Navarrete-Navarro (21), we compared the areas under the ROC curves using the method of Delong et al. and adjusted each p value for the effect of multiple comparisons using Bonferroni's method (39). Finally, we considered the models predicted probability over a range of thresholds and calculated the corresponding sensitivity, specificity, percent correctly classified, and positive/negative likelihood ratios. All unadjusted tests of significance used a two-sided p < 0.05. All statistical analyses were performed using Stata 11.0 statistical software (StataCorp, College Station, TX, 2003). This study was approved by the Institutional Review Board for the University of Washington.

Results

Among 268 subjects available for inclusion in the derivation cohort, we excluded 37 (14%) subjects not requiring ICU care and 7 (3%) not surviving to 24 hours after injury, leaving 224 subjects in the final derivation cohort. ARDS was identified in 79 (35%) of these subjects. As seen in Table 1, subjects in the derivation cohort developing ARDS were older, had higher ISS and APACHE II scores, and were more likely to have a blunt injury mechanism, chest trauma (flail chest and pulmonary contusion), and present to the emergency department in shock (systolic blood pressure < 90mmHG) compared with those not developing ARDS. Those developing ARDS were also more likely to receive a massive RBC transfusion (≥ 10 RBC units within 24 hours of injury) and were more often exposed to a HPVC than those not developing ARDS (Table 1). There were no meaningful differences in sex, medical comorbidities, initial hematocrit, lactate levels, coagulopathy markers, and percentage with femur and pelvic fractures between subjects with and without ARDS.

Table 1.

Subject demographics, injury, and resuscitation characteristics in the derivation cohort by ARDS development status

Variable	ARDS Absent (N = 145)	ARDS Present (N = 79)	p-value
Age (years)	40 (18)	47 (19)	0.009
Male Sex	99 (68%)	52 (66%)	0.708
Injury Severity Score	23 (10)	30 (11)	<0.001
APACHE II Score	17 (7)	21 (6)	<0.001
Glasgow Coma Score	8 (6)	7 (6)	0.327
Lowest Recorded ED Hematocrit	27 (7)	27 (7)	0.497
ED Lactate	3.9 (3.0)	4.5 (2.6)	0.116
ED INR	1.3 (0.39)	1.4 (0.73)	0.140
ED Platelet Count (1000/μL)	237 (79)	227 (78)	0.362
ED Fibrinogen	222 (99)	236 (116)	0.366
Shock Diagnosis in EDa	48 (33%)	38 (48%)	0.027
Diabetes	9 (6%)	9 (11%)	0.173
Alcohol use	13 (9%)	9 (11%)	0.560
Blunt Injury Mechanism	112 (77%)	74 (94%)	0.002
Femur Fracture	30 (21%)	23 (29%)	0.156
Pelvic Fracture	39 (27%)	25 (32%)	0.452
Flail Chest	(1%)	6 (8%)	0.005
Pulmonary Contusion	33 (23%)	33 (42%)	0.003
Crystalloid (Liters/24h)	12 (8)	14 (8)	0.134
Massive RBC Transfusion (≥10 RBC units/24hours)	12 (8%)	17 (22%)	0.005
Moderate RBC Transfusion (6 – 10 RBC units/24 hours)	28 (19%)	21 (27%)	0.208
Red Blood Cells (Units/24h)c	3 (2, 6)	5 (2, 9)	0.002
Fresh Frozen Plasma (Units/24h)c	2 (0, 6)	6 (2, 10)	<0.001
Cryoprecipitate (Units/24h)c	0 (0, 0)	0 (0, 6)	0.301
Platelets (mL/24h)c	0 (0, 250)	150 (0, 287)	0.019
Received a High Plasma Volume Componentb	84 (58%)	63 (80%)	0.001

^aShock defined as systolic blood pressure less than 90 during ED phase of care

^bHigh plasma volume component (HPVC) transfusion defined as a transfusion with platelets, fresh frozen plasma, or cryoprecipitate within 24 hours of injury Continuous data presented as mean (standard deviation) except

^clisted as Median (IQR). Categorical data listed as N (%)

1,950 subjects available in the Inflammation and the Host Response to Injury Consortium study were considered for inclusion in the validation cohort (Figure 1). The primary reasons for exclusion were death within 24 hours of injury (6%) followed by age under 18 years (4%), an ARDS diagnosis within 24 hours of admission (1%), and lack of a requirement for ICU care (1 subject). This left 1,762 subjects in the validation cohort. Due to the enrollment criteria in the parent study, the validation cohort had a higher percentage with a blunt injury mechanism and with shock compared to the derivation cohort (Table 2). Those in the validation cohort had a slightly higher ISS and APACHE II scores and were more likely to have a pelvic fracture and pulmonary contusion as compared to the derivation cohort (Table 2). In general, subjects in both cohorts were of a similar age and sex, carried similar comorbidities, and had similar coagulation profiles at presentation. While those in the validation cohort were more likely to require a massive RBC transfusion and received slightly more crystalloid within the first 24 hours of injury, the percentage of those exposed to a HPVC was identical across the derivation and validation cohorts. Fewer patients in the validation cohort developed ARDS as compared with the derivation cohort (24% versus 35% respectively).

Figure 1.

Subject Accrual into the validation cohort.

Table 2.

Subject demographics, injury and resuscitation characteristics, and outcomes for derivation and validation cohorts

Variable	Derivation Cohort (N = 224)	Validation Cohort (N = 1,762)
Age (years)	43 (19)	44 (18)
Male Sex	151 (67%)	1,158 (66%)
Injury Severity Score	25 (11)	32 (13)
APACHE II Score	19 (7)	28 (7)
Glasgow Coma Score	8 (6)	9 (6)
Lowest Recorded ED Hematocrit	27 (7)	24 (6)
ED Lactate	4.1 (2.9)	4.4 (2.7)
ED INR	1.4 (0.5)	1.4 (0.7)
Shock Diagnosis in EDa	86 (38%)	1,120 (64%)
Diabetes	18 (8%)	126 (7%)
Alcohol use	22 (10%)	226 (13%)
Blunt Injury Mechanism	186 (83%)	1,762 (100%)
Femur Fracture	53 (24%)	518 (29%)
Pelvic Fracture	64 (29%)	694 (39%)
Flail Chest	(3%)	114 (6%)
Pulmonary Contusion	66 (30%)	637 (36%)
Crystalloid (Liters/24h)	12 (8)	14 (8)
Massive RBC Transfusion (≥10 RBC units/24hours )	29 (13%)	457 (26%)
Moderate RBC Transfusion (6 – 10 RBC units/24 hours)	49 (22%)	409 (23%)
Red Blood Cells (Units/24h)c	4 (2, 7)	5 (3, 9)
Fresh Frozen Plasma (Units/24h)c	4 (0, 8)	2 (0, 6)
Cryoprecipitate (Units/24h)c	0 (0, 6)	0 (0, 5)
Platelets (mL/24h)c	0 (0, 250)	0 (0, 300)
Received High Plasma Volume Componentb	147 (66%)	1,168 (66%)
Required Mechanical Ventilation	203 (91%)	1,634 (93%)
ARDS Development	79 (35%)	423 (24%)
ARDS Diagnosis Day	5 (3, 6)	3 (2, 5)
Death	36 (16%)	203 (12%)
Total number of days hospitalized in ICUc	5 (2, 13)	10 (5, 18)
Total number of days hospitalizedc	14 (8, 25)	19 (11, 13)

^aShock defined as systolic blood pressure less than 90 during ED phase of care

^bHigh plasma volume component (HPVC) transfusion defined as a transfusion with platelets, fresh frozen plasma, or cryoprecipitate within 24 hours of injury Continuous data presented as mean (standard deviation) except

^clisted as Median (IQR). Categorical data listed as N (%)

Using the development cohort, age, ISS, and APACHE II score (modeled as continuous variables), as well as blunt injury mechanism, flail chest, pulmonary contusion, requirement for a massive transfusion, and exposure to a HPVC (modeled as binary categorical variables) were considered as candidate ARDS predictors. After employing a backward selection approach using the AIC to populate the model, all variables except exposure to a HPVC were identified as predictors of ARDS in the final model (Table 3). The interaction terms between massive transfusion and blunt injury as well as between subject age and APACHE II score, age and ISS, and age and massive transfusion status did not significantly improve model fit and were not included (p >.10 for all). Fitting the model to the derivation cohort demonstrated an area under the ROC of 0.79 (95% confidence interval [C.I.] 0.73, 0.85). The HL goodness-of-fit test demonstrated no statistical evidence of lack of fit (χ_df=8² = 11, p=0.20). After internal validation using the bootstrap approach, the area under the ROC curve for the final model was maintained at 0.79 (95% C.I. 0.73, 0.85).

Table 3.

Results of multivariable logistic regression for final ARDS prediction model from derivation cohort (N=224)

Variable	Regression Coefficient (β)	Standard Error
APACHE II Score	0.067	0.025
Injury Severity Score	0.028	0.018
Blunt Injury	1.300	0.543
Pulmonary Contusion	0.599	0.361
Massive Transfusion	0.847	0.538
Flail Chest	1.877	1.172
Age (Years)	0.022	0.010
Constant	-5.087	0.815

$\text{Predicated ARDS risk}={\frac{1}{1+e}}^{-\left(\mathrm{logit}\right)}$

After application of the final model to the independent validation cohort, model discrimination assessed by area under the ROC curve was 0.71 (95% C.I. 0.68, 0.74). Applying the predictive model developed by Navarrete-Navarro et al.(21) to the identical validation cohort resulted in an area under the ROC curve of 0.66 (95% C.I. 0.64, 0.69), while the model developed by Miller et al.(20) resulted in an area under the ROC curve of 0.65 (95% C.I. 0.63, 0.68). In comparing our model's area under the ROC curve to the two other models (Figure 2), our model's performance was significantly greater than that of the Navarrete-Navarro model (Bonferroni adjusted p=0.001) and the Miller model (Bonferroni adjusted p<0.001). In the validation cohort, the HL goodness-of-fit test demonstrated statistical evidence of insufficient fit (p<0.10 for all models). The observed and mean predicted number of ARDS cases were 24.0% and 58.4% respectively for a standardized ARDS ratio of 0.41 (p < 0.001). As demonstrated in the calibration plot analyses, our model (Figure 3, Panel A) systematically overestimated ARDS risk. By comparison, the model by Miller et al. (Figure 3, Panel B) consistently underestimated ARDS risk except at the highest decile of predicted risk where fit appeared visually adequate. On the other hand, the model by Navarrete-Navarro et al. (Figure 3, Panel C) appeared reasonably well calibrated at the two lowest risk deciles but then, like our model, consistently overestimated ARDS risk. Finally, we calculated sensitivity, specificity, and diagnostic likelihood ratios across a range of predicted probability thresholds for our model (Table 4). Using a cutoff of ≥ .60, sensitivity was 0.72 and specificity was 0.59. The positive likelihood ratio of 1.7 suggests that patients developing ARDS are almost twice as likely to have a predicted probability of ARDS of ≥ 60% as calculated by our predictive instrument early after traumatic injury.

Figure 2.

Model performance as assessed by area under the receiver operating characteristic (ROC) curves for the new ARDS study prediction model and two comparison models in the validation cohort. The area under the ROC (AUC), specific to each model, is presented under the figure.

Figure 3.

Calibration plots of observed (black) and expected (gray) probabilities of ARDS across equally-sized deciles of predicted ARDS risk in the validation cohort (deciles 1-10 represent a continuum of lowest to highest predicted risk). Presented calibration plots are specific to the new ARDS prediction model (panel A), and those developed by Miller et al. (Panel B) and Navarrete-Navarro et al. (Panel C).

Table 4.

Operating characteristics for increasing thresholds of the model's predicted probability of ARDS in the validation cohort

Probability threshold	Sensitivity	Specificity	% Correctly classified	Likelihood ratio +	Likelihood ratio -
0.30	0.97	0.17	36.4	1.2	0.14
0.40	0.91	0.30	44.3	1.3	0.30
0.50	0.84	0.45	54.3	1.5	0.36
0.60	0.72	0.59	61.9	1.7	0.48
0.70	0.57	0.72	68.4	2.0	0.59
0.80	0.39	0.83	72.7	2.3	0.73
0.90	0.19	0.92	74.9	2.5	0.87

Discussion

Using routinely-available clinical variables, we developed a model that performs well in predicting the development of ARDS in at-risk patients with severe trauma requiring RBC transfusion and hospitalization in the ICU. The predictive instrument incorporates objective and commonly-measured clinical variables available early after injury including age, APACHE II Score, ISS, and the presence of blunt traumatic injury, pulmonary contusion, massive transfusion, and flail chest injury. Our model maintained its predictive ability upon validation in a large, independent population of critically injured trauma patients. Moreover, our model's discriminatory performance was superior to two other published ARDS predictive models when discrimination was compared in the identical validation cohort.

We developed this model in response to recent calls for studies that target those at high risk of ARDS for research studies aimed at understanding biologic mechanisms of ARDS and in testing interventions directed towards ARDS prevention (40, 41). Our instrument, designed to accurately predict incident ARDS in at-risk trauma patients, could serve several important functions in future research settings of this type. First, a well-validated predictive model will enhance resource utilization through the targeted sampling of biologic specimens (i.e. cells, DNA/RNA) from those at highest risk of ARDS. By sampling patients most likely to develop ARDS, investigators can decrease patient heterogeneity and optimize the signal to noise ratio in laboratory-based ARDS mechanistic studies, potentially improving the probability of detecting informative changes in measured biologic factors (40). Second, a minority of patients with traumatic injury ultimately develop ARDS. Our predictive instrument could be used to identify patients most likely to develop this serious complication, enriching future randomized controlled trials (42). Those with severe trauma, compared to other at-risk populations such as sepsis, represent an attractive population in whom to test preventative ARDS therapies because a clear temporal distinction between injury and a defined ARDS outcome is present. A study evaluating the clinical risks of ARDS noted this feature, as well as the comparably large time interval between risk identification and the development of subsequent ARDS (10).

Our prediction model is not the first to target ARDS as a primary outcome. Recently, a validated ARDS prediction model targeted those with diverse ARDS risk factors (i.e. sepsis, pneumonia, aspiration) identified at time of hospital admission (43). While we were not able to compare the performance of our prediction model to this model, our instrument focuses entirely on ARDS prediction in those with severe trauma, a group collectively representing less than 10% of the above at-risk cohort (43). Those with trauma-associated ARDS represent an important group to study as these patients carry clinical outcomes and biomarker profiles different from those with ARDS linked to non-traumatic risk factors (44), suggesting the possibility of unique underlying clinical risk factors and causal biologic mechanisms. Unfortunately, trauma populations also typically represent the minority of those enrolled in ARDS epidemiology and intervention studies (9, 45-48), preventing generalizability of successful and failed ARDS interventions to trauma patients with ARDS. As we demonstrate that ARDS onset can be predicted early after injury, our model might aid in risk stratification specific to trauma patients. Our study builds on prior work in this area through the assessment and comparison of ARDS predictive model performance in an independent cohort of at-risk patients.

An expected consequence of externally validating a prediction model is a decrease in model performance compared to that in the derivation cohort. A portion of this decreased performance is explained by optimistic bias, which stems from several factors inherent to the model's construction. We attempted to avoid introducing this bias by limiting candidate predictor variables relative to the number of ARDS cases (24, 34), employing the use of AIC in model building, and by estimating the models sampling variability using an internal bootstrap simulation method (25, 37). Despite these attempts, factors including the derivation cohort's small sample size, the relatively high percentage of ARDS cases relative to the validation cohort, and the fact that the trauma patients in the derivation cohort uniquely represented subjects enrolled in a single center clinical trial all contributed to the decreased model performance on validation (23, 24, 37, 49). While these factors contributed to only a minor decrease in discriminatory performance in our model, they had a substantial effect on model calibration. After fitting our model to the validation cohort, we observed a relatively weak agreement between predicted and actual ARDS event rates with our model systematically overestimating ARDS risk (Figure 3, panel A). Calibration was equally poor in the comparison models (Figure 3, Panels B/C). Overall, our model's predictive performance, with thresholds for ARDS probability set at 50-60%, resulted in reasonable ARDS sensitivity at the expense of lower specificity (Table 4).

Elements of population variability between the derivation and validation cohorts not only explain the challenges in accuracy with respect to calibration, but also explain differences in performance measures across the compared ARDS predictive models upon validation. For example, the blunt injury predictor variable in our model was lost when applied to the validation cohort as all subjects had a blunt injury mechanism (enrollment criteria of the Inflammation and Host Response to Injury Consortium). Furthermore, ARDS event rates in the validation cohort (24%) were lower than those in our derivation cohort (35%), yet higher than the reported ARDS event rates in the derivation cohorts of Miller et al. (4.5%) (20) and Navarrete-Navarro et al. (6.9%) (21), altogether highlighting the variability between derivation and validation populations. Interestingly, the decrease in ARDS events in the validation cohort occurred in a population that appeared more severely injured (i.e. higher ISS and APACHE II, increased percentage with shock, more RBC use) compared to the derivation population. While ARDS cases were lower than expected, the finding parallels that found in other contemporary studies of ARDS in trauma patients (50, 51). Moreover, the adoption of improved supportive care practices over the study period, including the use of lower transfusion thresholds and smaller tidal volume settings in those at risk for or with Acute Lung Injury (ALI) and/or ARDS, likely accounts for this finding(52, 53). Although impossible to ascribe the lower rate of ARDS cases observed in the validation cohort to one select intervention, the use of standard operating procedures across the hospitals participating in the Inflammation and Host Response to Injury Consortium is a plausible explanation. These were introduced in a stepwise fashion and address broad supportive care measures considered best practice for trauma patients based on published evidence and expert consensus (54-63) . Whatever the causes explaining the differences present between derivation and validation populations, the value of validating our new ARDS prediction model, and two other published models for comparison, using a large and independent cohort of trauma patients to provide a realistic measure of its true performance cannot be overstated. This validation represents one step to demonstrate the generalizability of using our model, and two other published models specific to ARDS after trauma, in a research setting (64). While our model overpredicts ARDS in the validation cohort (i.e. model calibration suffers), it retains its ability to discriminate well despite the important differences between the derivation and validation cohorts highlighted above.

We recognize several limitations of the current study. We were unable to evaluate other potentially important predictors of ARDS (i.e. aspiration of gastric contents (10, 43), plasma-derived ARDS biomarkers (65, 66)) as these were not documented in our data sources. Although these factors might improve the model's discriminatory performance, they would also introduce challenges in objectivity (defining gastric aspiration) and practicality (specialized testing of biologic samples), complicating the simplicity of using this tool in the clinical setting (27). We acknowledge that some overlap exists between selected predictors of interest (i.e. APACHE II score and age, definitions for pulmonary contusion and flail chest); however, our aim was not to establish independent associative or causal relationships between “risk factors” and ARDS outcome. Rather, we sought to identify the combination of variables that best predicted trauma-associated ARDS. While calibration suffered on validation, it is difficult to develop models with both excellent discrimination and calibration (67). The fact that our model maintained adequate discriminatory performance may be the most important element of our model's overall accuracy given the intended use of this model in future settings (i.e. as a tool to identify high risk populations for further research) (64). Finally, several studies demonstrate unique ARDS risk factors and outcome measures according to the timing of ARDS development after injury, early- versus late-onset ARDS (20, 68, 69). This suggests that a level of heterogeneity may exist within the specific clinical syndrome of trauma-associated ARDS, with early- and late-onset ARDS potentially linked to equally diverse inflammatory/coagulopathic mechanistic pathways. The size of our derivation cohort prevented the identification of specific predictors of early versus late-onset ARDS, reducing these potentially heterogeneous ARDS definitions in trauma into a single entity.

Conclusions

In summary, we developed a model to identify trauma patients at greatest risk for ARDS which demonstrated acceptable discrimination upon validation in an independent cohort of patients. The model outperformed two separate published ARDS prediction models and provides a generalizable set of criteria for future research studies aimed at investigating ARDS mechanisms or prevention strategies.