Development and Validation of the Neonatal Risk Estimate Score for Children Using Extracorporeal Respiratory Support (Neo-RESCUERS)

Abstract

Objective

To develop and validate the Neonatal Risk Estimate Score for Children Using Extracorporeal Respiratory Support (Neo-RESCUERS), which estimates the risk of in-hospital death for neonates prior to receiving respiratory extracorporeal membrane oxygenation (ECMO) support.

Study design

We used an international ECMO registry (2008–2013); neonates receiving ECMO for respiratory support were included. We divided the registry into a derivation sample and internal validation sample, by calendar date. We chose candidate variables a priori based on published evidence of association with mortality; variables independently associated with mortality in logistic regression were included in this parsimonious model of risk adjustment. We evaluated model discrimination with the area under the receiver operating characteristic curve (AUC) and we evaluated calibration with the Hosmer-Lemeshow goodness-of-fit test.

Results

During 2008–2013, 4,592 neonates received ECMO respiratory support with mortality of 31%. The development dataset contained 3,139 patients treated in 2008–2011. The Neo-RESCUERS measure had an AUC of 0.78 (95% confidence interval: 0.76–0.79). The validation cohort had an AUC=0.77 (0.75–0.80). Patients in the lowest risk decile had an observed mortality of 7.0% and a predicted mortality of 4.4%, and those in the highest risk decile had an observed mortality of 65.6% and a predicted mortality of 67.5%.

Conclusions

Neo-RESCUERS offers severity-of-illness adjustment for neonatal respiratory failure patients receiving ECMO. This score may be used to adjust patient survival to assess hospital-level performance in ECMO-based care.

Extracorporeal membrane oxygenation (ECMO) decreases neonatal respiratory failure mortality when mechanical ventilation cannot support gas exchange or when the degree of support required is injurious (1, 2). However, the effect of ECMO may differ among ECMO centers. Hospitals’ neonatal ECMO mortality rates span a wide interquartile range of 18–50% (3). The neonatal respiratory ECMO mortality rate is also increasing across the three most common primary diagnoses: meconium aspiration syndrome, congenital diaphragmatic hernia and persistent pulmonary hypertension (4). We do not know why neonatal ECMO mortality rates have increased or why they vary between centers, but the absence of a pre-ECMO risk adjustment tool has impeded the study of these questions (4).

A pre-ECMO risk adjustment tool can facilitate three functions. First, it enables hospitals to compare their risk-adjusted performance to peer institutions (5, 6). Second, it can facilitate efficient adjustment between groups, allowing use of observational data to assess whether interventions are efficacious (7, 8). Third, pre-ECMO severity of illness score could predict the mortality risk for similar patient groups (9, 10), which can assist physicians and families in weighing the risks and benefits of ECMO. Currently, no severity of illness score exists for these purposes. In this study, we develop the Neonatal Risk Estimate Score for Children Using Extracorporeal Respiratory Support (Neo-RESCUERS). Neo-RESCUERS will estimate the pre-ECMO risk of in-hospital death for neonates receiving respiratory ECMO support.

Methods

This study was designed in accordance with Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) (11). We developed Neo-RESCUERS utilizing an ECMO registry, Extracorporeal Life Support Organization (ELSO).

Neonates (≤ 28 days at the time of ECMO cannulation) were eligible if they received ECMO for respiratory support from 2008–2013. We chose 2008 because advances in ECMO technology made ECMO support safer and easier thereafter (3, 12). The ELSO dataset was divided by calendar date (11) into a two-thirds development and a one-third validation subset. The development dataset spanned January 1, 2008, through December 31, 2011; the validation dataset included January 1, 2012, through December 31, 2013. Patients with a primary diagnosis of congenital heart disease, heart failure, cardiomyopathy, and myocarditis were excluded because Neo-RESCUERS is designed to predict neonatal respiratory ECMO mortality.

We selected candidate variables based on two factors: (1) variable missing in <10% of ELSO observations; and (2) published evidence that the variable was independently associated with mortality. We systematically searched PubMed for neonatal respiratory ECMO articles published on or after January 1, 2000 using the search line (neonatal respiratory ECMO mortality) OR (((“Extracorporeal Membrane Oxygenation”[Mesh]) AND “Infant, Newborn”[Mesh]) AND “Mortality”[Mesh]). We also included variables in neonatal mortality risk adjustment models (13) published after January 1, 2000: Score for Neonatal Acute Physiology-II (SNAP-II) (14, 15), SNAP Perinatal Extensions-II (SNAPPE-II) (14, 15), Vermont Oxford Network-Risk Adjustment (VON-RA) (15) and Agency for Healthcare Research and Quality (AHRQ) Neonatal Indicator (16).

The following candidate variables were identified: patient age (17, 18), gestational age (15, 19), birth weight (14, 17, 19–21), sex (15, 19), primary diagnosis (17–23), pre-ECMO renal failure (24), comorbid conditions (15, 16), pre-ECMO cardiac arrest (17, 22) and pre-ECMO measures including arterial blood pH (19, 21) the arterial partial pressure of carbon dioxide (PaCO₂) (21), and the ratio of arterial partial pressure of oxygen to fraction of inspired oxygen (PF ratio) (14), oxygenation index (OI) (25), Apgar (14, 15, 19, 21), mean arterial pressure (14), and pre-ECMO use of inhaled nitric oxide (22) or surfactant (18).

Primary diagnoses were divided into categories using International Classification of Diseases-9-Clinical Modification (ICD-9-CM) codes (Table I; available at www.jpeds.com). Categories were chosen based on previous reports (1, 18, 19, 25). If a neonate had a primary diagnosis of acute respiratory failure or respiratory arrest, the second diagnosis was used to categorize the patient. Renal failure and cardiac arrest were identified by ICD-9-CM diagnostic codes and were only included if they were not listed as a complication of ECMO (Table II; available at www.jpeds.com), suggesting the renal failure or cardiac arrest occurred prior to ECMO (3). Comorbidity was defined using ICD-9-CM codes described by AHRQ neonatal indicator (16). The only modification is the exclusion of codes used to define the primary diagnosis of congenital diaphragmatic hernia.

The ELSO registry contains observations with data that is missing at random. Logistic regression models with missing at random data should not limit analysis to patients with complete data because this can bias results (11, 26). We addressed missingness through multiple imputation with iterative chained equations (27). Multiple imputation predicts an observation’s missing data by using the partial information available in the observation and the data contained in other observations (27). We used predictive mean matching with 10 nearest neighbors for nonparametric continuous variables (28), and we used the logit function to impute sex.

We performed a logistic regression for mortality with all candidate variables. Variables independently associated with mortality p ≤ 0.05 were selected. We tested for interactions between gestational age and birth weight as well as pH and PaCO₂ on mortality. Interactions were not significant and were not included in the model. Neo-RESCUERS is the sum of each selected variable (x_i) weighted by its regression coefficient (β_i) plus the intercept (β₀):

$$\text{Neo-RESCUERS}={\mathrm{\beta }}_1{\mathrm{x}}_1+{\mathrm{\beta }}_2{\mathrm{x}}_2+\dots {\mathrm{\beta }}_{\mathrm{i}}{\mathrm{x}}_{\mathrm{i}}+{\mathrm{\beta }}_0+\mathrm{\mu }.$$

To test the robustness of our variable selection to missingness we performed a sensitivity analysis, and selected variables based only on those observations with complete data.

Discrimination was assessed by the area under the receiver operating characteristic curve (AUC), and model calibration was assessed with the Hosmer-Lemeshow goodness-of-fit (H-L) test. The score was internally validated using the ELSO registry data from 2012–2013. Our primary validation was performed on patients with complete data, but we also validated Neo-RESCUERS against those with imputed data.

OI was not included in Neo-RESCUERS because it was missing in >10% of cases. However, clinically, many neonatologists use OI. Consequently, we derived and validated a supplementary score in an identical manner. The supplementary score considered the previously listed candidate variables, but substituted OI for PF ratio. Analyses were performed in Stata 14.

Results

During the study years, 4,787 neonates received respiratory ECMO. We excluded 103 neonates with a primary cardiac diagnosis and 92 neonates with >1 ECMO run leaving 4,592. Most neonates (65%) were placed on ECMO after a venoarterial cannulation. Overall, most patients received pre-ECMO support of high frequency oscillator ventilation (59%), inhaled nitric oxide (86%), neuromuscular blockade (55%), and were on at least one vasoactive infusion (62%). Neonates receiving ECMO support from 2008–2011 were similar to those receiving support from 2012–2013, with the exception that after 2011, fewer patients received vasoactive infusions or neuromuscular blocking agents (Table III).

Table 3.

Demographic and pre-extracorporeal membrane oxygenation (ECMO) characteristics of patients, by dataset

Variable	Derivation, 2008–2011 (n=3,139)	Validation, 2012–2013 (n=1,453)	p-value
	Median (Interquartile Range)
pH†	7.20 (7.08–7.30)	7.17 (7.07–7.27)	<0.001
PaCO2,† mm Hg	59 (45–76)	60 (47–79)	0.04
SaO2,† %	68 (46–82)	68 (47–82)	0.36
PF Ratio†	37 (28–48)	36 (28–46)	0.49
Oxygenation Index†	47 (33–67)	48 (35–67)	0.19
Apgar 1 minute of life	5 (2–7)	4 (2–7)	0.11
Mean Arterial Pressure,† mm Hg	42 (35–50)	43 (36–51)	0.30
Pre-ECMO hours of mechanical ventilation	29 (14–58)	30 (15–54)	0.77
Birth Weight, kilograms	3.2 (2.8–3.6)	3.2 (2.8–3.6)	0.92
Gestational Age, (weeks)	39 (38–40)	39 (37–40)	0.03
Age, days	1 (1–3)	1 (1–3)	0.63
	Number (Percent)
Female	1,291 (41.1)	645 (44.4)	0.03
Primary Diagnosis
Meconium Aspiration Syndrome	789 (25.1)	346 (23.8)	0.33
Congenital Diaphragmatic Hernia	920 (29.3)	417 (28.7)	0.67
Respiratory Distress Syndrome	62 (2.0)	15 (1.0)	0.02
Persistent Pulmonary Hypertension	650 (20.7)	312 (21.5)	0.55
Sepsis	235 (7.5)	102 (7.0)	0.58
Other	483 (15.4)	261 (18.0)	0.03
Comorbidity‡	259 (8.3)	151 (10.4)	0.02
Pre-ECMO Renal Failure*	41 (1.3)	17 (1.2)	0.7
Pre-ECMO Cardiac Arrest*	43 (1.4)	15 (1.0)	0.34
Ventilator Mode
Conventional Mechanical Ventilator	826 (26.3)	387 (26.6)	0.82
Oscillator	1,832 (58.4)	897 (61.7)	0.03
Other or missing	481 (15.3)	169 (11.6)	0.01
Inhaled Nitric Oxide	2,739 (87.3)	1,231 (84.7)	0.02
Surfactant	894 (28.5)	411 (28.3)	0.89
Neuromuscular Blockade	1,801 (57.4)	738 (50.8)	<0.001
Vasoactive Infusions	2,126 (67.7)	697 (48)	<0.001

PaCO₂=arterial partial pressure of carbon dioxide, SaO₂ =percent oxygen saturation of hemoglobin in arterial blood, PF ratio=ratio of arterial partial pressure of oxygen to the fraction of inspired oxygen, and pre-ECMO hours of mechanical ventilation=number of hours of mechanical ventilation prior to ECMO. For p-values continuous, nonparametric variables were compared using Mann-Whitney U test, and categorical variables were compared using Pearson’s chi-square test.

^†Most abnormal value recorded within 6 hours of ECMO support.

^‡Comorbidity is defined using Agency for Healthcare Research and Quality definitions (16).

^*Pre-ECMO renal failure and pre-ECMO cardiac arrest are defined by ICD-9-CM codes plus the respective absence of renal failure or cardiac arrest as an ECMO complication.

Neo-RESCUERS Development

In the development dataset, four candidate variables were excluded because they were missing in >10% of the case files: OI (missing in 19.4%), mean arterial pressure (43.8%), Apgar at 1 minute (10.3%) and Apgar at 5 minutes (11.5%). After logistic regression, ten variables independently associated with mortality were selected for Neo-RESCUERS (Table IV).

Table 4.

Multivariate analysis of pre-extracorporeal membrane oxygenation (ECMO) factors associated with death prior to hospital discharge

Variable	Odds Ratio (95% CI)	β-coefficient (se)	p-value
pH† (per 0.1 change in pH)	0.85 (0.81–0.90)	NA	<0.001
pH† (per 1 change in pH)	NA	−1.589 (0.265)	<0.001
PaO2†/FiO2 (mm Hg)	0.997 (0.993–0.99996)	−0.003 (0.002)	0.047
Birth Weight (kilograms)	0.72 (0.61–0.85)	−0.333 (0.086)	<0.001
Gestational Age (weeks)	0.94 (0.88–0.99)	−0.067 (0.029)	0.02
Age (days)	1.04 (1.02–1.06)	0.042 (0.010)	<0.001
Female	1.26 (1.06–1.50)	0.234 (0.089)	0.008
Primary Diagnosis
Meconium Aspiration Syndrome	referent	referent
Congenital Diaphragmatic Hernia	11.05 (8.01–15.24)	2.402 (0.164)	<0.001
Respiratory Distress Syndrome	2.40 (1.15–5.03)	0.877 (0.376)	0.02
Persistent Pulmonary Hypertension	3.31 (2.33–4.69)	1.195 (0.179)	<0.001
Sepsis	7.84 (5.19–11.83)	2.059 (0.210)	<0.001
Other	6.29 (4.40–8.97)	1.838 (0.181)	<0.001
Comorbidity‡	1.81 (1.35–2.41)	0.592 (0.148)	<0.001
Pre-ECMO Renal Failure*	3.35 (1.64–6.84)	1.209 (0.365)	0.001
Pre-ECMO Inhaled Nitric Oxide*	0.69 (0.54–0.88)	−0.375 (0.127)	0.003
intercept	NA	12.793 (2.130)	<0.001

95% CI= 95% Confidence Interval, se=standard error, PF ratio=the ratio of arterial partial pressure of oxygen to fraction of inspired oxygen.

^†Most abnormal value recorded within 6 hours of receipt of ECMO support

^‡Comorbidity is defined using Agency for Healthcare Research and Quality (AHRQ) definitions (16).

^*Pre-ECMO renal failure and pre-ECMO cardiac arrest are defined by ICD-9-CM codes plus the respective absence of renal failure or cardiac arrest as an ECMO complication.

In the final multivariable model, higher arterial pH, PF ratio, birth weight, and gestational age are all associated with a lower risk of mortality. In contrast, older postnatal age, female sex, pre-ECMO renal failure, presence of comorbid conditions, and a primary diagnosis other than meconium aspiration syndrome are all associated with a higher risk of mortality. Among specific diagnoses, patients with congenital diaphragmatic hernias had the worst outcome.

The Neo-RESCUERS model had moderate discrimination, the ability to identify patients who were likely to die, as measured by the AUC=0.78 (95% confidence interval: 0.76–0.79) (Figure 1, A; available at www.jpeds.com). Model fit was assessed with the H-L test, which tests if there is a statistically significant difference between the rates of predicted and observed mortality. In a perfectly calibrated model, the predicted risk will match the observed risk. In our model, the predicted mortality risk was not statistically significantly different from the observed percent mortality (Table V and Figure 2, A). To calculate Neo-RESCUERS, use β-coefficients in Table IV or visit www.neo-rescuers.com. The predicted probability of mortality =e^Neo-RESCUERS/(1+e^Neo-RESCUERS).

Figure 1.

Predicted and observed model fit in the derivation and validation datasets.

Table 5.

Discrimination and calibration of models of extracorporeal membrane oxygenation (ECMO) mortality among neonates, in derivation and validation datasets

Dataset	Patients	Discrimination	Calibration
			Hosmer-Lemeshow
		AUC	Chi-Statistic	p-value
Derivation (2008–2011)	3,139	0.78 (0.76–0.79)	7.67	0.47
Validation (2012–2013), complete data*	1,283	0.77 (0.75–0.80)	8.64	0.37
Validation (2012–2013)	1,453	0.77 (0.74–0.79)	8.10	0.42

AUC = area under the curve of the receiver operating characteristic curve

^*complete data = only those neonates in the dataset with complete data for retained variables are included

Figure 2.

The Neonatal Risk Estimate Score for Children Using Extracorporeal Respiratory Support (Neo- RESCUERS) predicted mortality.

To test the robustness of our model we also developed the model using only observations with complete data (no imputation). This method selected the same set of variables except that it excluded sex and gestational age. Selected variables had similar odds ratios. The model had similar discrimination AUC=0.78 (0.76–0.80) and calibration.

In the validation dataset there were 1,453 patients. The validation among patients with complete data had an AUC=0.77 (0.75–0.80) and similar goodness-of-fit as the development model (Table V and Figures 1, B and 2, B). Those in the lowest decile of risk had an observed mortality of 7.0% and predicted mortality of 4.4%. In contrast, those in the highest decile of risk had an observed mortality of 65.6% and a predicted mortality of 67.5%. The model performed with similar discrimination and calibration (Table V) among patients with imputed data.

As a supplementary analysis, we substituted OI for PF ratio. Aside from OI, the supplementary multivariable logistic regression identified the same variables (Table VI; available at www.jpeds.com). OI had a small independent association with mortality adjusted odds ratio=0.003 (0.001–0.006) (Table 6; online). The development AUC=0.78 (0.76–0.79), validation AUC=0.77 (0.74–0.80) and calibration were not substantively different (Table VII; available at www.jpeds.com) from Neo-RESCUERS.

Discussion

This study developed and validated a novel risk adjustment score for neonates receiving respiratory ECMO. Neo-RESCUERS is a well-calibrated and sufficiently discriminatory model to predict mortality. For reference, SNAP-II, SNAPPE-II and VON-RA are neonatal mortality risk prediction scores to compare risk-adjusted outcomes between NICUs (13). SNAP-II has an AUC=0.86 and H-L test p=0.34; SNAPPE-II has an AUC=0.89 and H-L test p=0.26; and VON-RA has an AUC=0.94 and H-L test p=0.11 (15). Neo-RESCUERS has less discrimination with an AUC=0.77, but equivalent calibration H-L test p=0.37.

To avoid a bias, we imputed missing data (11, 26). The model performed similarly when the validation was applied to neonates with complete or imputed data. Additionally, if we developed the model from observations with complete data the odds ratios of selected variables were similar; the models discrimination and calibration were similar, and the selected variables were the same except for the exclusion of gestational age and sex. The imputed model’s selection of gestational age and sex is consistent with previous neonatal mortality risk scores (15). The similarity of results speaks to a robust model.

Primary diagnosis was one of the strongest contributing factors to the predictive capacity of Neo-RESCUERS. Patients with congenital diaphragmatic hernia had an 11-fold higher adjusted odds of mortality compared with those with meconium aspiration syndrome. Neonates with pre-ECMO renal failure had a much higher (3-fold higher) odds of mortality. This may represent an association between pulmonary hypoplasia and renal failure or it may stem from the difficulty of supporting neonates in renal failure.

Neo-RESCUERS cannot be used to predict if someone should receive ECMO because it is developed from a cohort of patients who all received ECMO. Additionally, even though Neo-RESCUERS was developed and validated using ELSO data, it may not generalize to patients cared for at non-ELSO centers. This tool may, nonetheless, help inform families and physicians of the mortality risk for neonates with similar clinical circumstances who have received ECMO.

Like many tools designed to adjust for severity of illness, this tool was derived from a clinical registry. The registry provides strength through the large number of patients treated at numerous international centers. However, the ELSO registry has limitations. Some physiologic variables that are a part of other severity-of-illness models are not available in ELSO. For example, there are no physiologic measures of renal function as measured by urine output (14) or creatinine (29). In addition, physiologic measures of neurologic status such as pupillary response (30) or seizures (14) are absent. It is possible that collecting additional variables in the future may be required to improve model discrimination and calibration.

The use of ICD-9-CM codes relies on the accurate entry, and consequently the reported codes may not represent the patient’s complete list of diagnoses. For example, Neo-RESCUERS defines pre-ECMO renal failure and pre-ECMO cardiac arrest based on ICD-9-CM codes plus the absence of an ECMO complication of renal failure or cardiac arrest. If providers do not enter the ICD-9-CM for a pre-ECMO cardiac arrest, then our approach would underestimate pre-ECMO cardiac arrest. Alternatively, if some providers entered the ICD-9-CM code for an arrest that occurred on ECMO, but did not indicate that the cardiac arrest was an ECMO complication, then we would overestimate pre-ECMO cardiac arrest. However, ELSO has been internally audited and found to have 1% error in reported fields (31).

We were not able to evaluate neurocognitive outcomes. Previous randomized control studies have suggested that patients supported with ECMO do not have more neurocognitive disability than similarly ill neonates or adults (1, 32). However, neonates with the degree of pre-ECMO hypoxia experienced by many patients in this study are at risk for cerebral injury (33). Future studies should characterize how pre-ECMO severity of illness affects subsequent neurocognitive development.

We believe this work has three potential applications. First, the Neo-RESCUERS tool can help facilitate intra- and inter-institutional benchmarking. Previous studies (34–36) have suggested that regular receipt of risk-adjusted outcomes can motivate self-examination at the institutional level and subsequently improve outcomes. ELSO already reports outcomes to centers benchmarked against other ELSO-participating institutions, but risk adjusting those outcomes may enhance comparisons and thus further motivate improvements in ECMO.

Second, neonatal ECMO management is currently based on consensus protocols (2, 37). With application of Neo-RESCUERS as a risk adjustment model, future studies will have greater discriminatory power to compare clinical outcomes for specific management strategies in ventilation, anticoagulation, and cannulation, which in turn may inform evidence-based guidelines for ECMO management.

Third, although this tool should not be used as a decision tool to select which patients should or should not receive ECMO, it may help to quantify the risk for similar patients receiving ECMO. This can serve as a helpful point of reference for families to discuss the risks and benefits of ECMO with their physicians.

Neo-RESCUERS offers a novel approach to risk adjustment for neonatal respiratory failure patients receiving ECMO. It demonstrates satisfactory discrimination and calibration that is validated in internal samples. Future studies may augment this score by considering additional variables that would add to the discriminatory ability of the model, although more exhaustive data collection may require several years to achieve sufficient case counts.

Table 1.

International classification of diseases-9- clinical modification (ICD-9- CM) definitions of primary diagnosis

Diagnosis	ICD-9-CM	Description
Meconium Aspiration Syndrome	770.1–770.18	fetal and newborn aspiration
Congenital Diaphragmatic Hernia	552.3	diaphragmatic hernia with obstruction
	553.3	diaphragmatic hernia without obstruction
	756.6	anomalies of diaphragm
Respiratory Distress Syndrome	769	respiratory distress syndrome
Persistent Pulmonary Hypertension	416.0	primary pulmonary hypertension
	747.83	persistent fetal circulation
	747.89	specified anomalies of the circulatory system
Sepsis	008.62	enteritis due to adenovirus
	033–033.9	whooping cough
	036.2	meningococcemia
	038–038.9	septicemia
	040.82	toxic shock syndrome
	040.89	other specified bacterial diseases
	041.02	streptococcus, group B
	041.11, 041.12	staphylococcus aureus infection
	041.2	pneumococcus infection
	041.7	pseudomonas infection
	052.1	varicella pneumonitis
	054–054.0, 054.5, 054.7–054.72, 054.74–054.9	herpes simplex virus
	079.0	adenovirus
	079.3	rhinovirus
	079.6	respiratory syncytial virus
	079.99	unspecified viral infection
	112.4, 112.5	candidiasis
	117.3	aspergillosis
	466.1–466.19	bronchiolitis
	480–488.89	pneumonia and influenza
	510–510.9	empyema
	511.1	pleurisy with effusion and bacterial cause
	513–513.1	abscess of lung and mediastinum
	770.0	congenital pneumonia
	771–771.4	infections specific to the perinatal period
	771.7–771.89	infections specific to the perinatal period
	785.52	septic shock
	790.7, 790.8	bacteremia, viremia
	995.9–995.94	systemic inflammatory response syndrome and sepsis

Table 2.

International classification of diseases-9- clinical modification (ICD-9-CM) definitions of cardiac arrest, renal failure

Category	ICD-9-CM	Description
Cardiac Arrest	427.5	cardiac arrest
	779.85	cardiac arrest of newborn
	v12.53	personal history of cardiac arrest
Renal Failure	584–584.9	acute kidney failure
	585–585.9	chronic kidney disease
	586	renal failure unspecified

Table 6.

Multivariate analysis of oxygenation index and pre-ECMO factors associated with death prior to hospital discharge

Variable	Odds Ratio (95% CI)	β-coefficient (se)	p-value
pH† (per 0.1 change in pH)	0.86 (0.81–0.91)	NA	<0.001
pH† (per 1 change in pH)	NA	−1.478 (0.269)	<0.001
Oxygenation Index (per 1 cmH2O/mm Hg)	1.003 (1.001–1.006)	0.003 (0.001)	0.004
Birth Weight (kilograms)	0.71 (0.60–0.84)	−0.339 (0.086)	<0.001
Gestational Age (weeks)	0.93 (0.88–0.99)	−0.069 (0.029)	0.02
Age (days)	1.04 (1.02–1.06)	0.041 (0.010)	<0.001
Female	1.25 (1.05–1.49)	0.221 (0.047)	0.01
Primary Diagnosis
Meconium Aspiration Syndrome	referent	referent
Congenital Diaphragmatic Hernia	11.64 (8.41–16.11)	2.455 (0.166)	<0.001
Respiratory Distress Syndrome	2.48 (1.18–5.19)	0.908 (0.377)	0.02
Persistent Pulmonary Hypertension	3.36 (2.37–4.78)	1.213 (0.179)	<0.001
Sepsis	7.72 (5.11–11.67)	2.044 (0.211)	<0.001
Other	6.35 (4.44–9.08)	1.849 (0.182)	<0.001
Comorbidity‡	1.79 (1.34–2.39)	0.584 (0.147)	<0.001
Pre-ECMO Renal Failure*	3.30 (1.62–6.74)	1.194 (0.364)	0.001
Pre-ECMO Inhaled Nitric Oxide*	0.71 (0.55–0.90)	−0.346 (0.125)	0.006
intercept	NA	11.734 (2.160)	<0.001

95% CI= 95% Confidence Interval, se=standard error

^†Most abnormal value recorded within 6 hours of receipt of ECMO support

^‡Comorbidity is defined using Agency for Healthcare Research and Quality (AHRQ) definitions (16).

^*Pre-ECMO renal failure and pre-ECMO cardiac arrest are defined by ICD-9-CM codes plus the respective absence of renal failure or cardiac arrest as an ECMO complication.

Table 7.

Discrimination and calibration of oxygenation index models for ECMO mortality among neonates

Data	Patients	Discrimination	Calibration
			Hosmer-Lemeshow
		AUC	Chi-Statistic	p-value
Derivation (2008–2011)	3,139	0.78 (0.76–0.79)	7.93	0.44
Validation (2012–2013), complete data*	1,109	0.77 (0.74–0.80)	3.93	0.86
Validation (2012–2013)	1,453	0.76 (0.73–0.79)	2.96	0.94

AUC = area under the curve of the receiver operating characteristic curve

^*complete data = only those neonates in the dataset with complete data for retained variables are included