Development and Validation of a Mortality Prediction Model for Patients Receiving 14 Days of Mechanical Ventilation

Abstract

Objectives

The existing risk prediction model for patients requiring prolonged mechanical ventilation is not applicable until after 21 days of mechanical ventilation. We sought to develop and validate a mortality prediction model for patients earlier in the ICU course using data from day 14 of mechanical ventilation.

Study Design

Multi-center retrospective cohort study.

Patients

Adult patients receiving at least 14 days of mechanical ventilation at 5 medical centers (development cohort) or enrolled in the ARDS Network FACTT trial (validation cohort).

Measurements and Main Results

Predictor variables were measured on day 14 of mechanical ventilation in the development cohort and included in a logistic regression model with one-year mortality as the outcome. Variables were sequentially eliminated to develop the ProVent 14 model. This model was then generated in the validation cohort. A simplified prognostic scoring rule (ProVent 14 Score) using categorical variables was created in the development cohort and then tested in the validation cohort. Model discrimination was assessed by the area under the receiver-operator characteristic curve (AUC).

491 patients and 245 patients were included in the development and validation cohorts, respectively. The most parsimonious model included age, platelet count, requirement for vasopressors, requirement for hemodialysis, and non-trauma admission. The AUC for the ProVent 14 model using continuous variables was 0.80 (95% CI, 0.76–0.83) in the development cohort and 0.78 (95% CI, 0.72–0.83) in the validation cohort. The ProVent 14 Score categorized age at 50 and 65 years old, and categorized platelet count at 100 × 10⁹/L, and had similar discrimination as the ProVent 14 model in both cohorts.

Conclusion

Using clinical variables available on day 14 of mechanical ventilation, the ProVent14 model can identify patients receiving prolonged mechanical ventilation with a high risk of mortality within one year.

For an increasing number of patients, critical illness or injury is neither self-limited nor imminently fatal¹. As survival from life-threatening illness improves, pre-existing disease and newly-acquired organ dysfunction may conspire against recovery, leaving patients dependent on life supporting therapies for extended time periods. This syndrome, termed “chronic critical illness”, is commonly typified by persistent respiratory failure that requires prolonged mechanical ventilation (PMV). Long-term mortality is high, approaching rates of 40% to 60% at one-year in inclusive cohorts^2–10. Patients have a very high symptom burden during the weeks of prolonged ventilation^11,12 and chances of living at home with functional independence at the end of the year are as low as 10%^9,13.

Most PMV patients are unable to participate in their own clinical decision making and must therefore rely on surrogates to guide their goals of care. These surrogate decision makers report that physicians rarely discuss prognosis, and studies have shown that physician and family estimates of one year outcome are highly discordant^14,15. Traditional approaches to ICU mortality prediction, such as APACHE, are ill-suited to predicting longer term outcomes, particularly in this population¹⁶. Our research group developed and validated the ProVent mortality prediction model which estimates one-year mortality for patients receiving at least 21 days of mechanical ventilation to address prognostic uncertainty for these unique patients^2,3,17.

However, many important decisions regarding the care of PMV patients occur before 21 days have elapsed, including electing to perform tracheotomy¹⁸ or transfer care to a long-term acute care hospital (LTACH), both of which in are in some form decisions to continue life sustaining therapies^19,20. If compelling prognostic data were available sooner for the patient whose goals of care are known, these decisions might be made more easily and earlier in the course of illness^21–23. Building on our previous work, we hypothesized that a simple mortality prediction model constructed from factors clinically available by day 14 in the course of mechanical ventilation could discriminate between PMV patients at high and low risk of death at one year. Using a heterogeneous multicenter cohort of PMV patients in addition to a contemporaneous cohort of patients with acute lung injury (ALI) who were enrolled in a randomized, controlled trial, we sought to develop and validate a new model using data collected on day 14 of mechanical ventilation rather than on day 21 as in our previous model.

METHODS

Development Cohort

Patients were enrolled from 5 tertiary care medical centers in geographically diverse regions of the U.S. (Seattle, WA, Denver, CO, San Francisco, CA, Philadelphia, PA, and Durham, NC). Patients who were discharged in 2005 and received at least 14 days of mechanical ventilation uninterrupted by more than 48 hours were eligible. Patients were excluded if they were less than 18 years old; diagnosed with acute or chronic neuromuscular disease, had extensive burns, or required chronic mechanical ventilation prior to hospital admission. Commitment to continued care with life-sustaining therapies was not an inclusion or exclusion criterion. Consecutive samples of patients were included from 4 centers, and a random sample of 176 patients was included from the largest center. The overall cohort included 260 patients who received 21 days or more of mechanical ventilation with whom we validated our previous model³.

Data on demographic variables, potential risk factors for death, and outcomes were abstracted from medical records by 2 investigators at each site. Risk factors were measured on day 14 of mechanical ventilation. The investigator abstracting data on risk factors was blinded to outcomes. The principal investigator at each site reviewed the first 10 charts and a random sample of 10 additional charts to confirm accuracy of data and identify errors that would prompt review and correction.

Potential risk variables for the model were chosen a priori based upon previous literature, clinical judgment and reliability of measurement. Each variable was measured on day 14 of mechanical ventilation. These included the four primary risk variables from the original ProVent model, including age, platelet count, requirement for vasopressors, and requirement for hemodialysis^2,3. Requirement for hemodialysis was defined as renal replacement therapy provided on or within 48 hours before or after day 14 of mechanical ventilation. Additional variables included gender, a primary or secondary diagnosis of trauma, PEEP level, serum glucose, white blood cell count, and hemoglobin. The primary outcome, one year vital status, was obtained from medical records if available, or alternatively the National Death Index and Washington State Death Database. Work on the development cohort was approved by Institutional Review Boards (IRB) at all participating institutions, with waiver of informed consent for this minimal risk observational study.

Validation Cohort

The validation cohort was derived from patients enrolled in the NHLBI ARDS Network Fluids and Catheter Treatment Trial (FACTT)^24,25 who consented to be concomitantly enrolled in a prospective economic outcomes study (EA-PAC)²⁶ and who received at least 14 days of mechanical ventilation. Patients were enrolled in FACTT from 40 centers across the U.S. if they met consensus criteria for acute lung injury and were willing to consent to the trial and consent to a central venous catheter. Patients requiring hemodialysis or those having very advanced comorbid diseases such as severe cirrhosis at admission were excluded. While imminent plan to withdraw life support was an exclusion for enrollment in FACTT, there was no encouragement or enforcement of continued commitment to life-sustaining therapies beyond the enrollment window²⁷. A subgroup of patients in FACTT was enrolled in EA-PAC when surrogate decision makers consented to both studies. Detailed characteristics of patients in both studies have been described elsewhere^24–26. Overlap between patients in development and validation cohorts was not measurable due to de-identified datasets, however if it occurred at all, it was less than 2% due to differences in study periods and non-consecutive enrollment in the FACTT trial. Day 14 risk variables were extracted from the FACTT database. Missing risk variable data were assumed to be normal. One-year mortality was extracted from the FACTT and EA-PAC databases; all patients in the validation cohort had known vital status at one year.

Informed consent was obtained for participation in FACTT and EA-PAC studies, and IRB approval was obtained at all participating institutions. Additionally, IRB approval was obtained at the University of Washington for use of FACTT/EA-PAC data for this validation cohort, with waiver of additional informed consent.

Analysis

Descriptive statistics are presented using mean and standard deviation for continuous variables with normal distribution or median and interquartile range for variables with non-normal distribution. Categorical variables are presented as proportions. Using the derivation cohort, all of the preselected predictor variables were included in a logistic regression equation with one-year mortality as the outcome variable. The area under the Receiver Operator Characteristic curve (AUC) was used to assess model discrimination. Individual variables were dropped from the model in a stepwise fashion with a subsequent assessment of the AUC. Variables were returned to the model if the AUC changed by more than 0.02 from the original value³. Calibration of the final model was assessed by the Hosmer and Lemeshow goodness of fit statistic, comparing observed mortality to predicted mortality for each decile of risk. A bootstrap method repeating 1000 random samples consisting of 60% of the cohort was used to obtain a 95% confidence interval for the AUC of the final model.

We performed 2 additional steps to develop a points-based clinical scoring rule. Continuous variables from the final model above were categorized and included in a second logistic regression model using categorical variables only. Next, points were assigned to each variable based upon the beta coefficients in that model. To assess the performance of the scoring rule, cumulative scores (ProVent 14 Score) were tabulated for each patient, and a third logistic regression model for one-year mortality was generated using the cumulative score as the independent variable.

Using the validation cohort, we then fit a logistic regression model using the beta values of the risk variables from the derivation scoring rule model. Assessments of model discrimination and calibration for the validation cohort were performed similarly to the development set by calculating the AUC and Hosmer Lemeshow goodness of fit statistics (using observed levels of score). We compared ROC discrimination of the ProVent14 continuous model to APACHE III²⁸ for prediction of one-year mortality using the DeLong method²⁹. We created Kaplan-Meier plots to assess differential survival by ProVent 14 Score in both development and validation cohorts.

Analyses were performed using Stata version 11.0 (Stata, College Station, Texas), MedCalc Statistical Software version 14.10.2 (MedCalc Software, Ostend, Belgium) and SAS/STAT software, Version 9.4 of the SAS System for Windows (Cary, NC, USA).

RESULTS

Model Development

A total of 491 patients were included in the development cohort. Patient characteristics and hospital outcomes are presented in Table 1. The mean age of patients was 54 ± 17 years old, and 40% were female. Median (IQR) duration of mechanical ventilation was 22 (17–33) days. Hospital mortality was 29%, and one-year mortality was 45%. Univariate analysis of associations between predictor variables and one-year mortality are presented in Table 2. The AUC for the logistic regression model with all predictor variables was 0.80. Stepwise elimination of gender, glucose, white blood cell count, hemoglobin, and PEEP level resulted in a final model containing age, platelet count, vasopressors, hemodialysis, and non-trauma diagnosis. Enrollment site was not an independent predictor when added to this model and did not affect the AUC. The AUC for the final parsimonious model was 0.80 (95% CI, 0.76–0.83), and the Hosmer and Lemeshow goodness of fit statistic was 3.32 with 8 degrees of freedom, p=0.91. In contrast, the AUC using APACHE III to predict one-year mortality in this cohort was 0.60 (95% CI, 0.55–0.64), which differed significantly from the ProVent 14 model (p<0.0001).

TABLE 1.

Patient Characteristics and Outcomes

Variable	Development (n = 491)	Validation (n = 245)	P
Age, yr, mean ± SD	54±17	53±16	0.2090
Female sex, n (%)	185 (40)	107 (44)	0.3230
Non white race, n (%)	182 (37)	94 (38)	0.9754
Trauma, n (%)	118 (24)	29 (12)	0.0001
Acute Physiology and Chronic Health Evaluation III score, mean ± SD	81±28	101±28	< 0.0001
Platelets day 14, mean ± SD	316±199	297±191	0.2303
Platelets < 100,000/mL, n (%)	63 (13)	32 (13)	0.9770
Vasopressors/inotropes day 14, n (%)	78 (16)	35 (16)	0.9634
Hemodialysis day 14, n (%)	51 (10)	34 (14)	0.2027
Mechanical ventilation days, median (IQR)	22 (17–33)	23 (17–31)	0.9277
ICU length of stay, median (IQR)	28 (21–38)	25 (19–33)	0.0015
Hospital length of stay, median (IQR)	38 (27–55)	34 (26–47)	0.0102
Hospital mortality, n (%)	141 (29)	121 (49)	< 0.0001
1-yr mortality, n (%)	221 (45)	145 (59)	0.0004

IQR = interquartile range.

TABLE 2.

Univariate Analysis of Risk Variables and 1-Year Mortality in Development Cohort

Variable	Alive, n = 270	Dead, n = 221	P
Age, yr
Mean (SD)	48.9 (16.9)	60.9 (14.6)	< 0.0001
Median (IQR)	49 (38–61)	63 (52.8–72)	< 0.0001
Gender (male), n (%)	172 (64.7)	111 (55.0)	0.0421
Race (nonwhite),a n (%)	137 (50.7)	64 (29.0)	0.2411
Platelets (day 14), × 109/L
Mean (SD)	380 (211)	238 (150)	< 0.0001
Median (IQR)	347 (215–507)	234 (109.3–342)	< 0.0001
Vasopressors (day 14), n (%)	18 (6.7)	60 (27.1)	< 0.0001
Hemodialysis (day 14), n (%)	12 (4.4)	39 (17.6)	< 0.0001
Nontrauma, n (%)	174 (64.4)	199 (90)	< 0.0001
Positive end-expiratory pressure (day 14)
Mean (SD)	7.1 (3.5)	7.2 (3.6)	0.8572
Median (IQR)	5 (5–8)	5 (5–8)	0.7508
Glucose (day 14), mg/dL
Mean (SD)	145 (46)	167 (64)	< 0.0001
Median (IQR)	134 (112.5–171.5)	156 (124–197.5)	0.0001
WBC (day 14), × 109/L
Mean (SD)	14.3 (7.5)	15.1 (12.3)	0.4216
Median (IQR)	12.7 (9.7–17.3)	12.5 (9.2–18.7)	0.9188
Hemoglobin (day 14), g/dL
Mean (SD)	9.9 (7.3)	9.5 (2.4)	0.3610
Median (IQR)	9.1 (8.2–10.1)	9.3 (8.6–10)	0.1685

IQR= interquartile range.

^aNonwhite category includes race coded as unknown.

Model Validation

Three hundred forty-two patients from FACTT received 14 or more days of mechanical ventilation, of whom 245 had known one-year outcomes through EA-PAC and were included in the validation cohort. Table 1 compares patient characteristics and outcomes between the development and validation cohorts. Patients in the two cohorts were similar in age, gender, and race, and had similar median days of mechanical ventilation. Patients in the validation cohort had higher acute illness severity at hospital admission as represented by APACHE III score, and they had higher hospital and one-year mortality. The AUC for the Provent 14 model in the validation cohort was 0.78 (95% CI 0.72–0.83), and the Hosmer Lemeshow statistic was 9.39, p=0.31. The AUC using APACHE III to predict one-year mortality in the validation cohort was 0.62 (95% CI, 0.55–0.68), which differed significantly from the ProVent 14 model (p<0.0001).

ProVent 14 Score

To develop the simplified prognostic scoring rule, age was cut at 50 and 65 years old as in the original ProVent model³. Platelet count was cut at 100 × 10⁹, which was associated with a higher risk of mortality when measured at day 14 than a cutpoint of 150 × 10⁹ as in the original 21 day model. In the development cohort, a logistic regression model was fit with these categorized variables as independent variables and one-year mortality as the dependent variable. Points assigned to each predictor according to the beta coefficients from the categorical model are shown in Table 3.

TABLE 3.

Model in Development Cohort With Categorized Risk Variables to Derive Simplified Scoring Rule (ProVent 14 Score)

Categorical Variable	n (%)	OR (95% Cl)	β	Points
Age, ≥ 65 yr	144 (29)	6.5 (3.8, 11.2)	1.86	2
Age, 50–64 yr	162 (33)	2.7 (1.6, 4.4)	0.95	1
Platelets ≤ 100×109/L	63 (13)	3.0 (1.5, 6.1)	1.11	1
Vasopressors	78 (16)	3.8 (2.0, 72)	1.32	1
Hemodialysis	51 (10)	2.5 (1.1, 5.4)	0.93	1
Nontrauma	373 (76)	2.6 (1.5, 4.6)	0.94	1

OR = odds ratio.

Cumulative points based upon the number of predictor variables present for a patient constitute the ProVent 14 Score. We combined the seven possible scores into five categories: 0, 1, 2, 3 and 4 or greater. Figure 1 contains Kaplan Meier plots with long-term survival by ProVent 14 Score in the development and validation cohorts. ProVent 14 Score models performed well both in the development cohort (AUC 0.78, 95% CI 0.74–82; Hosmer-Lemeshow statistic 9.57, p=0.02) and in the validation cohort (AUC 0.76, 95% CI 0.70–0.81; Hosmer-Lemeshow statistic 1.45, p=0.69).

Figure 1.

A, Kaplan-Meier plots of survival for development cohort by ProVent 14 Score. B, Kaplan-Meier plots of survival for validation cohort by ProVent 14 Score.

DISCUSSION

In this two-cohort multi-center study, we successfully developed and validated a long-term mortality prediction model for patients requiring 14 days of mechanical ventilation after acute illness or injury. The model performed well for discrimination and calibration in the 5 centers constituting the development cohort, and the model has similar discrimination and calibration in the validation cohort. Calibration was best in the validation cohort for patients who are at highest and lowest risk for death—the patient subgroups groups for whom objective mortality prediction is likely to have the most impact on decision making³⁰.

The good performance of the model in the validation cohort is notable given that the cohort is a distinctly different group of patients derived from 40 hospitals in the U.S. who were enrolled in a clinical trial for patients with ARDS. This compares to the development cohort which enrolled consecutive patients with any diagnosis who received 14 days of mechanical ventilation. The validation cohort had higher overall hospital and one-year mortality, likely reflecting higher acute illness severity associated with the presence of ARDS in all patients. Overall mortality also was increased because vital status in the EA-PAC database was known for all patients who died in the hospital, whereas unknown vital status due to loss to follow up affected hospital survivors only. Other factors differentiating the validation cohort from the development cohort include various exclusion criteria of the FACTT trial such as low expected 6-month survival, morbid obesity, and requirement for hemodialysis at trial enrollment. Despite the overall differences between patients in the 2 cohorts and any variations in management, the mortality prediction model had good performance in the external validation cohort, which supports the broad based value of the model^31,32.

Multiple studies have shown that family members of patients in the ICU desire prognostic information, including families of patients requiring PMV, but this information is not always forthcoming^33,34. This deficiency results in significant discordance between expectations of clinicians and surrogate decision-makers, with typically overly optimistic expectations by families¹⁴. There are numerous barriers to sharing of prognostic information, including uncertainty and worries about sharing incorrect prognoses, concerns about upsetting family members or defeating hope, and available clinician time^35,36. Like the MELD score in common use for patients with cirrhosis³⁷, the ProVent 14 model uses a small number of readily available clinical variables that can be collected in minutes at the bedside or remotely from the electronic medical record. Surrogate decision makers for patients with very high likelihood of death can be prepared for that eventuality and prompted to consider the role of ongoing life support in the context of the patient’s values and wishes. For patients at very low risk of long-term mortality, family members and surrogate decision makers can be reassured that their loved one is unlikely to meet the usual grim outcome of the average patient requiring prolonged mechanical ventilation. Even for surrogates of patients with intermediate risk (e.g. 30% to 70% mortality estimate), typically overly optimistic family expectations may be modified¹⁴, and families can be better prepared for the potential of poor outcome.

Over twenty years ago, the SUPPORT trial failed to show an impact of providing objective prognostic information to clinicians, perhaps because clinicians did not share the information in 80% of cases³⁸. Yet, when incorporated into formal decision aids, data from prognostic models can facilitate the presentation of objective prognostic information to surrogates, reduce decisional conflict, reduce use of low value health care, and better align choice with values^21–23. Preliminary studies incorporating the ProVent model in an innovative decision aid to inform surrogate decision makers of PMV patients indicate that significant discordance in expectations of outcome between clinicians and decision-makers can be minimized³⁹. The ProVent model also differs from SUPPORT in its simplicity and timing, intending to provide information more than a week after the onset of critical illness—a time when surrogates remain uncertain of prognosis and anxious for information. However, objective prognostic information should not replace clinical judgment that incorporates unique patient characteristics, but such models are useful in the clinical setting to support clinical judgment and anchor it in objective data⁴⁰.

Our model has several limitations. The ProVent 14 model was developed in cohorts of patients requiring PMV at tertiary care research centers and may not be generalizable to other settings. However the majority of patients requiring PMV are managed in tertiary centers which have a high number of patients at risk for PMV and accept transfers of complicated patients from smaller centers. Additionally, some of the hospitals in the validation cohort were smaller community hospitals affiliated with research centers. The development cohort consisted of patients who received PMV in 2005, however there have been no empirical data published in the intervening period that suggests that long-term outcomes of PMV patients have improved in any appreciable way. In this study, variables were chosen for analysis as predictors based upon associations with outcomes in acute or chronic critical illness, ease of measurement, and relatively complete availability in medical records. We deliberately limited the number of variables assessed in order to create a simple, usable model that would stand up to external validation. It is possible that inclusion of additional variables could have improved the discrimination of the model, but this would be a subject for future prospective studies. Finally, the ProVent 14 model predicts one-year mortality, but it does not provide information on long-term physical or cognitive function, which would also be of significant importance in decision making and should be an objective of future research⁴¹.

Conclusion

The ProVent 14 mortality prediction model has good discrimination and calibration for one-year mortality in patients who require mechanical ventilation for at least 14 days. This simple prognostic model can support clinical judgment of prognosis and can inform clinical decision aids to facilitate discussions of goals of care in the setting of PMV, where long-term outcomes are patient specific and often not immediately evident.

TABLE 4.

ProVent 14 Score and Observed 1-Year Mortality

ProVent 14 Score	Development Cohort		Validation Cohort
	n	Observed Mortality, % (95% Cl)	n	Observed Mortality, % (95% Cl)
0	70	4 (0, 9)	17	24 (3, 44)
1	99	28 (19, 37)	68	32 (21, 44)
2	142	43 (35, 51)	66	62 (50, 74)
3	117	61 (52, 70)	62	81 (71, 90)
4–6	63	92 (84, 100)	32	88 (75, 100)

The PnoVent 14 Score is calculated by summing the point values assigned according to the presence of risk variables listed in Table 3 when measured on day 14 of mechanical ventilation.