Simultaneous Prediction of New Morbidity, Mortality, and Survival without New Morbidity from Pediatric Intensive Care: A New Paradigm for Outcomes Assessment

Murray M Pollack; Richard Holubkov; Tomohiko Funai; John T Berger; Amy E Clark; Kathleen Meert; Robert A Berg; Joseph Carcillo; David L Wessel; Frank Moler; Heidi Dalton; Christopher J L Newth; Thomas Shanley; Rick E Harrison; Allan Doctor; Tammara L Jenkins; Robert Tamburro; J Michael Dean

doi:10.1097/CCM.0000000000001081

. Author manuscript; available in PMC: 2016 Aug 1.

Published in final edited form as: Crit Care Med. 2015 Aug;43(8):1699–1709. doi: 10.1097/CCM.0000000000001081

Simultaneous Prediction of New Morbidity, Mortality, and Survival without New Morbidity from Pediatric Intensive Care: A New Paradigm for Outcomes Assessment

Murray M Pollack ¹, Richard Holubkov ², Tomohiko Funai ³, John T Berger ⁴, Amy E Clark ⁵, Kathleen Meert ⁶, Robert A Berg ⁷, Joseph Carcillo ⁸, David L Wessel ⁹, Frank Moler ¹⁰, Heidi Dalton ¹¹, Christopher J L Newth ¹², Thomas Shanley ¹³, Rick E Harrison ¹⁴, Allan Doctor ¹⁵, Tammara L Jenkins ¹⁶, Robert Tamburro ¹⁷, J Michael Dean ¹⁸

¹Department of Pediatrics, Children’s National Medical Center and the George Washington University School of Medicine and Health Sciences, Washington DC and Phoenix Children’s Hospital and the University Of Arizona School of Medicine Phoenix

²Department of Pediatrics, University of Utah School of Medicine, Salt Lake City, UT

³Department of Pediatrics, University of Utah School of Medicine, Salt Lake City, UT

⁴Department of Pediatrics, Children's National Medical Center, Washington DC

⁵Department of Pediatrics, University of Utah School of Medicine, Salt Lake City, UT

⁶Department of Pediatrics, Children's Hospital of Michigan, Detroit, MI

⁷Department of Pediatrics, Children’s Hospital of Philadelphia, Philadelphia, PA

⁸Department of Critical Care Medicine, Children's Hospital of Pittsburgh, Pittsburgh, PA

⁹Department of Pediatrics, Children's National Medical Center, Washington DC

¹⁰Department of Pediatrics, University of Michigan, Ann Arbor, MI

¹¹Department of Child Health, Phoenix Children’s Hospital and University of Arizona College of Medicine-Phoenix, Phoenix, AZ

¹²Department of Anesthesiology and Critical Care Medicine, Children's Hospital Los Angeles, Los Angeles, CA

¹³Department of Pediatrics, University of Michigan, Ann Arbor, MI

¹⁴Department of Pediatrics, University of California at Los Angeles, Los Angeles, CA

¹⁵Departments of Pediatrics and Biochemistry, Washington University School of Medicine, St. Louis, MO

¹⁶Pediatric Trauma and Critical Illness Branch, Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD), the National Institutes of Health (NIH), Bethesda, MD

¹⁷Pediatric Trauma and Critical Illness Branch, Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD), the National Institutes of Health (NIH), Bethesda, MD

¹⁸Department of Pediatrics, University of Utah School of Medicine, Salt Lake City, UT for the Eunice Kennedy Shriver National Institute of Child Health and Human Development Collaborative Pediatric Critical Care Research Network (CPCCRN)

^✉

Correspondence to: Murray M. Pollack, MD, Children’s National Medical Center, 111 Michigan Ave, NW, Washington, DC 20010, mpollack@childrensnational.org, telephone: 202.476.2203, fax: 202.476.5868

PMCID: PMC4657566 NIHMSID: NIHMS737434 PMID: 25985385

Abstract

Objective

Assessments of care including quality assessments adjusted for physiological status should include the development of new morbidities as well as mortalities. We hypothesized that morbidity, like mortality, is associated with physiological dysfunction and could be predicted simultaneously with mortality.

Design

Prospective cohort study from December 4, 2011 to April 7, 2013.

Setting and Patients

General and cardiac/cardiovascular pediatric intensive care units at 7 sites.

Measurements and Main Results

Among 10,078 admissions, the unadjusted morbidity rates (measured with the Functional Status Scale (FSS), and defined as an increase of ≥ 3 from pre-illness to hospital discharge) were 4.6% (site range 2.6% to 7.7%) and unadjusted mortality rates were 2.7% (site range 1.3% – 5.0%). Morbidity and mortality were significantly (p<0.001) associated with physiological instability (measured with the PRISM III score) in dichotomous (survival, death) and trichotomous (survival without new morbidity, survival with new morbidity, death) models without covariate adjustments. Morbidity risk increased with increasing PRISM III scores and then decreased at the highest PRISM III values as potential morbidities became mortalities. The trichotomous model with covariate adjustments included age, admission source, diagnostic factors, baseline FSS and the PRISM III score. The three-level goodness of fit test indicated satisfactory performance for the derivation and validation sets (p>0.20). Predictive ability assessed with the volume under the surface (VUS) was 0.50 ± 0.019 (derivation) and 0.50 ± 0.034 (validation) (versus chance performance = 0.17). Site-level standardized morbidity ratios were more variable than standardized mortality ratios.

Conclusions

New morbidities were associated with physiological status and can be modeled simultaneously with mortality. Trichotomous outcome models including both morbidity and mortality based on physiological status are suitable for research studies, and quality and other outcome assessments. This approach may be applicable to other assessments presently based only on mortality.

Keywords: severity of illness, morbidity, functional status, functional status score, pediatrics, outcome prediction, critical care, pediatric critical care, intensive care; pediatric intensive care

INTRODUCTION

Mortality adjusted for physiological status and other case mix factors has been the core methodology of adult, pediatric and neonatal intensive care assessments for decades. Users of these methods have been early proponents of standardized mortality ratios for quantitative, unit-based quality assessments for both internal and external benchmarking.^1–9 Case-mix adjusted survival and death rates are primary outcomes for national databases beyond critical care medicine.^10–15 For example, the Agency for Healthcare Research and Quality and the Centers for Medicare and Medicaid publish hospital mortality rates for common conditions including acute myocardial infarction, stroke, congestive heart failure, pneumonia, hip fractures, and gastrointestinal hemorrhage.^16–19

Mortality for most pediatric critical illnesses has decreased since these methods were developed, and medical therapies are increasingly focused on reducing morbidity in survivors.^20,21 Therapeutic initiatives such as hypothermia, prevention of secondary injury following head trauma, rapid resuscitation of shock, and early thrombolysis therapy are aimed at reducing survivors’ morbidity as well as improving survival rates.^22–28 However, most quantitative outcome assessment methods continue to focus on the dichotomous outcome of survival versus death. We hypothesized that morbidity affecting functional status, like mortality, is significantly associated with physiological dysfunction in pediatric intensive care unit (PICU) patients and could be predicted simultaneously with mortality to provide quantitative outcome prediction for morbidity, mortality, and survival without new morbidity (intact survival). This paper describes the development and validation of a prediction model from 10,078 patients in the Trichotomous Outcomes in Pediatric Critical Care (TOPICC) study.

METHODOLOGY

This investigation was performed in the Collaborative Pediatric Critical Care Research Network (CPCCRN) of the Eunice Kennedy Shriver National Institute of Child Health and Human Development.²⁹ Patients from newborn to less than 18 years were randomly selected and stratified by hospital from December 4, 2011 to April 7, 2013. The study had daily limits on the number of patients enrolled at each center. To ensure that patients enrolled in TOPICC were randomly selected from all eligible PICU admissions, a random number sequence was generated by the Data Coordinating Center for each calendar day. During enrollment days when a site had more eligible patients than the daily limit, this number sequence was used to randomly select those patients to be enrolled, based on the trailing digits of their medical record number. Patients from both general/medical and cardiac/cardiovascular PICUs were included. There were no separate general surgical or neurological PICUs. Moribund patients (vital signs incompatible with life for the first two hours after PICU admission) were excluded. Only the first PICU admission during a hospitalization was included. Researchers, research coordinators, and research assistants were trained in data collection in-person during quarterly network meetings and during biweekly conference calls. All sites had electronic medical records. Data were collected daily although information available in the medical records may have been accessed retrospectively. The protocol was approved by all Institutional Review Boards. Descriptive publications on partial samples have occurred.^20,21,30

Data included descriptive and demographic information (Table 1). Interventions included surgery and interventional catheterization. Cardiac arrest included closed chest massage within 24 hours prior to hospitalization or after hospital admission but prior to PICU admission. Admission source was classified as emergency department, inpatient unit, or post intervention unit from the same hospital or another institution. Diagnosis was classified by system of primary dysfunction based on the reason for PICU admission; cardiovascular conditions were classified as congenital or acquired. Potential predictors of morbidity and/or mortality were identified a priori and included gender, age, admission source, admission status (elective vs. emergency), post-intervention status and type of intervention, cardiac arrest, diagnosis, baseline functional status, and physiological status.

Table 1.

PRISM III Sampling Intervals for Cardiac Patients Receiving an Intervention. The admission time interval refers the period of the 2 hours prior to admission to 4 hours after admission for laboratory data and the first 4 hours of PICU care for other physiological variables. The post-intervention time interval refers to the first 4 hours of PICU care after a cardiac intervention (surgery or interventional catheterization, but not diagnostic catheterization).

Age at Admission	ICU length of stay prior to Cardiac Intervention	PRISM III Collection Time Interval
< 24 hours	≤12 hours	Admission
< 24 hours	12 hours – 10 days	Post-Intervention
24 hours to 10 days	0–10 days	Post-Intervention
24 hours to 10 days	> 10 days	Admission
11 days to 30days	≤48 hours	Post-Intervention
11 days to 30days	> 48 hours	Admission
31 days to 90 days	≤48 hours	Post-Intervention if cardiac surgery Admission if cardiac catheterization
31 days to 90 days	> 48 hours	Admission
> 90 days	All	Admission

Open in a new tab

Outcomes

Morbidity, mortality, and survival without new morbidity were assessed at hospital discharge. Morbidity affecting a significant decrement in functional status was assessed with the Functional Status Scale (FSS) and was recorded for the pre-acute illness (baseline) and at hospital discharge.³¹ The FSS is an age-independent assessment of pediatric functional status suitable for large studies. It was developed specifically for this project as well as to provide a new functional status assessment instrument suitable for large pediatric outcome studies. The domains, domain items, and data collection process were designed to be used in this study and the validation process was constructed to be similar to the data collection process used in this study. It is composed of 6 domains (mental status, sensory, communication, motor function, feeding, respiratory) with domain scores ranging from 1 (normal) to 5 (very severe dysfunction). The operational definitions and manual for the classifications have been published.³¹ It was determined from the medical records and/or discussions with the health care providers. Newborns never achieving a stable baseline were assigned an FSS score of 6; this was operationalized by assigning a FSS of 6 to admissions to the study sites from 0 – 2 days of age and to transfers from another facility from 3–6 days of age. Baseline FSS scores were categorized as 6–7 (good), 8–9 (mildly abnormal), 10–15 (moderately abnormal), 16–21 (severely abnormal), and > 21 (very severely abnormal).²⁰ New morbidity was defined as an increase in the FSS score of ≥ 3 points from baseline to hospital discharge; changes of this magnitude indicate very significant worsening of functioning. Previous analysis on those children with FSS score changes of ≥ 3 points revealed that over 95% of these children had a change of 2 or more points in a single domain, a clearly significant functional change.^20,21,31 Morbidity occurs in essentially all ages and types of patients, in relatively equal proportions, and involves all FSS domains.²¹

Measurement of Physiological Status

Physiological status was measured with the Pediatric Risk of Mortality (PRISM) III score with a shortened time interval (2 hours prior to admission to 4 hours after admission for laboratory data and the first 4 hours of PICU care for other physiological variables).^5,30 For model building, the PRISM components were separated into cardiovascular (heart rate, systolic blood pressure, temperature), neurological (pupillary reactivity, mental status), respiratory (arterial PO2, pH, PCO2, total bicarbonate), chemical (glucose, potassium, blood urea nitrogen, creatinine), and hematological (white blood cell count, platelet count, prothrombin and partial thromboplastin time) components, and the total PRISM III was also separated into neurological and non-neurological components.

Congenital Cardiac Conditions

The timing interval for assessing PRISM III data was modified for cardiac patients < 91 days of age because some institutions admit infants to the PICU prior to a cardiac intervention to “optimize” the clinical status but not for intensive care; in these cases, the post-intervention period more accurately reflects intensive care.⁵ However, in other infants for whom the cardiac intervention is delayed after PICU admission, the intervention is a therapy required due to failed medical management of the acute condition; in these infants, the routine PRISM data collection time interval is an appropriate reflection of critical illness. Therefore, while blinded to outcome status, we identified infants for whom it would be more appropriate to utilize data from the 4 hours after the cardiac intervention (post-intervention time interval) and those for whom using the admission time interval was more appropriate. We operationalized this decision on the conditions likely to present within the first 90 days, the time period when the vast majority of these conditions present. This is shown in Table 1.

Statistical Methods

Statistical analyses utilized SAS 9.2^® for descriptive statistics, model development, and fit assessment, and R 3.0.2 for analytic and graphical evaluation of predictive ability. Patient characteristics were descriptively compared and evaluated across sites using the Kruskal-Wallis test for continuous variables, and the Pearson chi-squared test for categorical variables. The statistical analysis was under the direction of R.H.

Model Building

The data set was randomly divided into a derivation set (75%) for model building and a validation set (25%) stratified by study site.

We tested the hypothesis that both new morbidity and mortality were associated with physiological status by investigating this relationship in dichotomous and trichotomous (3 outcome) logistic regression models without other covariates. Trichotomous models were constructed using the generalized logit model, which as parametrized simultaneously estimates odds ratios for mortality and for new morbidity versus discharge alive without new morbidity. Separate coefficients were fit for log odds of mortality and of morbidity because assuming proportionality of odds was not tenable. Descriptor variables having significance levels below 0.10 with respect to either morbidity or mortality odds ratios in the univariate trichotomous models were considered candidate predictors for the final trichotomous outcome model. A non-automated (examined by biostatistician and clinician at each step) backward stepwise selection approach was used to determine factors in the final reported model. Multi-categorical factors (e.g., diagnostic categories) had factors combined when appropriate per statistical and clinical criteria. Statistical criteria for factor inclusion and for determining the number of factor categories included the likelihood ratio test for nested models and the Akaike Information Criterion (AIC) for comparing general models as well as satisfactory overall goodness of fit. Clinician input was included (and paramount) in this process to ensure the model fit overall and within subgroups was relevant and consistent with clinical information. Construction of a clinically relevant, sufficiently predictive model using predictors readily available to the clinician took precedence over inclusion based solely on statistically significance.

Final Model Evaluation

Final candidate models were evaluated on the derivation and validation sets with respect to consistency of estimated coefficients, predictive ability, and goodness of fit. Predictive ability was assessed by two-dimensional receiver operating characteristic (ROC) curves for dichotomized outcomes), and by three-dimensional volume under the surface (VUS) for the modeled three-level outcome. Overall model goodness of fit was assessed for both the derivation and validation sets using a three-level extension of the Hosmer-Lemeshow test.³² For the entire dataset, goodness of fit with respect to key subgroups was assessed by examining standardized mortality and morbidity ratios for descriptive and diagnostic categories not utilized in the final model, as well as within the individual study sites. Only categories with at ≥ 10 outcomes in observed and expected cells were utilized.

Two-dimensional ROC curves were generated, and their variability estimated, using R package pROC.³³ Three-dimensional ROC surfaces were constructed by an algorithm varying a grid of two predicted probability cutpoints that gave priority to prediction of deaths, and VUS was estimated using a triplet-classification rule minimizing Euclidean distances.^34,35 The average dichotomized c-index (the average of the areas under the curve considered over all possible ordered dichotomizations of the outcome) is reported as an alternate summary measure of multidimensional model discriminatory ability.³⁶ While asymptotically exact formulas were used to assess standard error of two-dimensional areas under the curve, one thousand bootstrap replications(generated with outcome proportions fixed) were generated to estimate variability of the multidimensional VUS and c-index estimates.³⁷

RESULTS

There were 10078 patients from the seven sites with each site contributing from 1252 to 1617 patients (Table 2). Two patients remained in the hospital after completion of the study and were not included; information on all predictive variables in the model was available for all included patients. The distribution of all patient characteristic except cardiac arrest varied significantly between sites (p<0.001). Overall, the median age was 3.7 years, the predominant organ systems of primary dysfunction were respiratory (33.5%), cardiovascular (24.1%), and neurological (20.1%) and most patients were non-interventional (62.3%) and emergency (63.6%) admissions. Of the patients discharged alive, 174 (1.7%) were discharged to other acute care hospitals and 500 (5.0%) were discharged to other inpatient care facilities (rehabilitation, chronic care, skilled nursing, psychiatric). The unadjusted mortality rate was 2.7% (site range 1.3% – 5.0%) and the unadjusted new morbidity rate was 4.6% (site range 2.6% to 7.7%).

Table 2.

Site and Overall Sample Characteristics. All characteristics except cardiac arrest were significantly different among the sites (p<0.001).

Site	A	B	C	D	E	F	G	Overall
Sample Size	1252	1404	1617	1498	1347	1547	1413	10078
Median Age in Years (IQR)	3.2 (0.7,10.4)	3.5 (0.7,10.9)	3.9 (1.0,10.4)	4.0 (1.0,11.0)	3.9 (1.3,10.9)	4.1 (0.7,11.1)	3.3 (0.6,11.1)	3.7 (0.8,10.8)
Insurance¹ Commercial/Government/Other (%)	25.8/72.9/1.3	50.1/41.0/8.8	49.9/44.2/5.9	41.1/54.1/4.9	61.8/34.1/4.0	25.9/69.7/4.3	34.3/61.4/4.2	41.4/53.8/4.9
Race: Caucasian/Black/-Other (%)	45.8/48.0/6.2	70.5/10.5/18.9	47.8/27.8/24.4	45.7/44.3/10.0	76.8/15.7/7.5	44.0/6.9/49.1	30.2/8.2/61.6	51.2/22.8/26.0
Primary System of Dysfunction (n (%))
Respiratory	396 (31.6)	422 (30.1)	627 (38.8)	614 (41.0)	580 (43.1)	449 (29.0)	288 (20.4)	3376 (33.5)
Cardiovascular disease	347 (27.7)	450 (32.1)	321 (19.9)	264 (17.6)	192 (14.3)	316 (20.4)	540 (38.2)	2430 (24.1)
Neurologic	225 (18.0)	218 (15.5)	348 (21.5)	268 (17.9)	309 (22.9)	373 (24.1)	281 (19.9)	2022 (20.1)
Other	284 (22.7)	314 (22.4)	321 (19.9)	352 (23.5)	266 (19.7)	409 (26.4)	304 (21.5)	2250 (22.3)
Admitted for Post-Intervention Care (n (%)²	459 (36.7)	687 (48.9)	580 (35.9)	408 (27.2)	456 (33.9)	504 (32.6)	703 (49.8)	3797 (37.7)
PICU admission status
Elective (n (%)	417 (33.3)	611 (43.5)	544 (33.6)	443 (29.6)	453 (33.6)	458 (29.6)	741 (52.4)	3667 (36.4)
Emergency n (%)	835 (66.7)	793 (56.5)	1073 (66.4)	1055 (70.4)	894 (66.4)	1089 (70.4)	672 (47.6)	6411 (63.6)
Cardiac Arrest Prior to PICU Admission (n (%)³	27 (2.2)	14 (1.0)	17 (1.1)	24 (1.6)	15 (1.1)	19(1.2)	26 (1.8)	142 (1.4)
Median (IQR) Baseline FSS Score	6.0 (6.0,8.0)	6.0 (6.0,7.0)	6.0 (6.0,9.0)	6.0 (6.0,9.0)	6.0 (6.0,9.0)	6.0 (6.0,7.0)	6.0 (6.0,8.0)	6.0 (6.0,8.0)
Median (IQR) PRISM Score	2.0 (0.0,5.0)	3.0 (0.0,7.0)	0.0 (0.0,4.0)	2.0 (0.0,6.0)	2.0 (0.0,5.0)	0.0 (0.0,4.0)	3.0 (0.0,7.0)	2.0 (0.0,5.0)
Median (IQR) Hospital Length of Stay (days)	5.2 (2.8,10.4)	5.7 (2.9,13.8)	4. 5(2.3,11.1)	4.7 (2.2,10.7)	4.2 (2.2,8.7)	4.0 (1.9,7.7)	7.0 (3.3,16.8)	4.9 (2.5,11.0)
Outcome at hospital discharge (n (%))
New Morbidity⁴	60 (4.8)	47 (3.3)	80 (4.9)	115 (7.7)	35 (2.6)	48 (3.1)	78 (5.5)	463 (4.6)
Death	39 (3.1)	41 (2.9)	37 (2.3)	36 (2.4)	17 (1.3)	34 (2.2)	71 (5.0)	275 (2.7)

Open in a new tab

Other includes unknown

Interventions included operations and interventional catheterizations

Cardiac arrest occurring within 24 hours prior to hospital admission or during the hospitalization prior to the PICU admission

⁴

New Morbidity = Increase of FSS of ≥ 3

IQR = interquartile range

Without covariate adjustments, there was a significant association of morbidity (Figure 1a) and mortality (Figure 1b) with physiological instability measured with the PRISM III score (p<0.001 for likelihood ratio test for both outcomes). Increasing PRISM III scores were associated with increasing new morbidity and mortality risks for the dichotomous outcomes. In the trichotomous relationship, the relationship of PRISM III with mortality changes very little after accounting for simultaneous morbidity risk. However, the relationship of morbidity with PRISM III changes substantially after accounting for mortality risk; morbidity risk initially increased with higher PRISM III scores, but then decreased among children with the highest PRISM III scores whose mortality risk is high. The mortality and morbidity univariate odds ratios for the PRISM III score and each of its components of cardiovascular, respiratory, neurological, chemical, and hematologic are also significantly associated with both morbidity and mortality (Supplemental Digital Content - Table 3). Physiological status had a stronger influence on mortality than morbidity as evidenced by the slope in the figures and the significantly different magnitudes of the odds ratios (Supplemental Digital Content - Table 3, p<0.001 in all instances for Wald tests comparing PRISM coefficients for mortality versus morbidity).

Morbidity (Figure 1A) and Mortality (Figure 1B) for Dichotomous (dashed lines) and Trichotomous (solid lines) Relationships. The graph illustrates the relationships in the development set (n = 7650). There were 113 patients with PRISM III scores > 20. There was a significant association of morbidity and mortality with physiological instability measured with the PRISM III score (p<0.001 for likelihood ratio test of significance) for both outcomes.

The univariate odds ratios and confidence intervals of potential predictor variables demonstrated that factors considered were significantly associated with both morbidity and mortality, only one outcome, or neither outcome (Supplemental Digital Content - Table 3). Potential predictor variables associated with morbidity and/or mortality odds ratios were cardiac arrest status, age, primary system of dysfunction, intervention category, cancer status, trauma status, admission source, baseline FSS, and the PRISM score components.

Considering candidate variables identified in the univariate models, the final multivariate trichotomous outcome model included age, admission source, cardiac arrest, a diagnosis of acute (non-primary) or chronic cancer, trauma, primary system of dysfunction, baseline FSS, and PRISM III score divided into the neurological and non-neurological components (Table 4). All included variables except acute and chronic diagnoses of cancer, neurological disease, and the baseline FSS score were significant independent predictors of both morbidity and mortality; cancer diagnoses and baseline FSS score were significant, independent predictors of mortality only, while neurological disease was significant for morbidity only. The validation set had 112 morbidities observed and 113.6 predicted, and 61 deaths observed and 67.1 deaths predicted. Trichotomous Hosmer-Lemeshow tests (Table 5) found acceptable fit for both the derivation (p=0.22) and validation (p=0.32) sets. Figure 2 shows the discrimination for the derivation set using the three-dimensional surface of proportions of each outcome correctly simultaneously predicted using varying probability cutpoints. The estimated VUS for this three-way outcome relationship is 0.50 ± 0.019 where 0.019 is the standard error of the estimate. For the validation set, the estimated VUS is 0.50 ±0.034. (VUS chance performance indicated by the shaded area in Figure 2 is 0.167 (one sixth) in the three-dimensional setting). The area under the curve for the most clinically relevant 2-dimensional ROC curves for the derivation and validations sets was 0 .89 ± 0.012 and 0.89 ± 0.020 for the survival vs. death dichotomy, 0.79 ± 0.011 and 0.80 ± 0.018 for the death or new morbidity vs. survival without new morbidity dichotomy, and 0.72 ± 0.014 and 0.74 ± 0.024 for the new morbidity vs. death or survival without new morbidity dichotomy (Supplemental Digital Content Figure 3). The average dichotomized c-index³⁶ (chance performance = 0.50) achieves values of 0.84 ± 0.009 and 0.85 ± 0.016 for the derivation and validation sets, respectively.

Table4.

Final Trichotomous Outcome Model for Simultaneous Prediction of Morbidity and Mortality.

Predictors	Morbidity Coefficients (SE)	Odds Ratios: New Morbidity vs. No New Morbidity (95% CI)	Mortality Coefficients (SE)	Odds Ratios: Death vs. No New Morbidity (95% CI)
Intercept	−3.92 (0.17)	NA	−5.51 (0.27)	NA
Age at PICU Admission
0 day to < 14 days	0.80 (0.23)	2.23 (1.43,3.49)	1.64 (0.27)	5.14 (3.00,8.79)
14 days to < 1 month	0.47 (0.44)	1.61 (0.68,3.79)	1.26 (0.56)	3.53 (1.19,10.50)
1 month to < 12 months	0.39 (0.14)	1.48 (1.13,1.93)	0.42 (0.21)	1.52 (1.02,2.28)
>12 months	Reference	Reference	Reference	Reference
Admission Source
Direct admission: Referral Hospital	0.76 (0.15)	2.15 (1.59,2.90)	1.09 (0.24)	2.96 (1.87,4.70)
Inpatient Unit: Same Hospital	0.87 (0.18)	2.38 (1.67,3.39)	1.70 (0.25)	5.46 (3.33,8.95)
Emergency Department: Same Hospital	0.11 (0.16)	1.12 (0.81,1.53)	0.64 (0.25)	1.90 (1.16,3.14)
OR/PACU for Postoperative Care	Reference	Reference	Reference	Reference
Cardiac Arrest (1)	0.97 (0.33)	2.63 (1.38,5.00)	1.52 (0.33)	4.56 (2.40,8.66)
Acute (non-Primary) or Chronic Diagnosis of Cancer (1)	0.25 (0.28)	1.28 (0.74,2.21)	0.89 (0.30)	2.44 (1.36,4.40)
Trauma (1)	1.18 (0.19)	3.26 (2.23,4.77)	0.81 (0.35)	2.26 (1.13,4.51)
Primary System of Dysfunction
Cardiovascular/Respiratory	Reference	Reference	Reference	Reference
Cancer	0.73 (0.28)	2.07 (1.20,3.59)	0.90 (0.43)	2.47 (1.06,5.74)
Low Risk (DKA, Hematologic, Musculoskeletal, Renal)	−0.93 (0.31)	0.39 (0.21,0.72)	−1.69 (0.61)	0.18 (0.06,0.61)
Neurologic	0.38 (0.15)	1.46 (1.08,1.98)	−0.07 (0.25)	0.93 (0.57,1.54)
Other	−0.21 (0.23)	0.81 (0.52,1.28)	0.11 (0.31)	1.11 (0.61,2.03)
Baseline FSS Score Categorized as Good (1,2)	−0.23 (0.13)	0.80 (0.61,1.03)	−0.66 (0.19)	0.52 (0.36,0.74)
PRISM III Neurological Score (3, 4)	0.11 (0.02)	1.12 (1.08,1.16)	0.21 (0.02)	1.24 (1.19,1.29)
PRISM III Non-Neurological Score (4)	0.09 (0.01)	1.09 (1.07,1.12)	0.18 (0.01)	1.19 (1.16,1.23)

Open in a new tab

NA = not applicable

Reference is absence of the factor.

Baseline FSS score = 6 or 7.

PRISM III neurological components are pupillary reactions and mental status.

⁴

For each one point change.

Table 5.

Trichotomous Hosmer Lemeshow Goodness of Fit Tests for the Development and Validation Sets.

	Derivation					Validation
Risk Decile	n	Deaths		New Morbidities		n	Deaths		New Morbidities
		O	E	O	E		O	E	O	E
1	816	0	1.5	8	9.9	261	0	0.4	4	3.2
2	739	1	2.7	14	13.5	252	0	0.9	2	4.4
3	724	2	3.2	19	16.3	258	0	1	3	5.8
4	745	11	4.6	13	18.7	237	1	1.4	2	5.8
5	763	4	5.6	19	23.4	252	1	1.7	3	7.6
6	750	8	7.8	25	27.1	252	6	2.7	15	8.7
7	829	9	12.0	41	37.4	261	3	3.5	13	11.5
8	704	14	13.5	52	41.7	251	3	4.4	15	14.5
9	735	27	25.8	56	57	244	10	8.3	20	18.3
10	755	138	137.2	104	106.1	250	37	42.8	35	33.8
Total	7560	214	214.0	351	351	2518	61	67.1	112	113.6
	Chi-squared=20.0, p=0.22 (16 df)					Chi-squared=22.3, p=0.32 (20 df)

Open in a new tab

O = Observed

E = Expected

df = degrees of freedom

Volume Under the Surface (VUS) for the Trichotomous Predictor. Data are shown for the Derivation Set. The three edges of this surface are the two two-dimensional ROC curves for the prediction of each pair of outcomes. In general, vertical “slices” of this surface in any dimension may be viewed as conditional ROC curves showing the ability of the model to predict a pair of outcomes conditional on correctly predicting the third outcome in a given proportion of subjects. Estimated VUS (proportion of the “cube” that is under the prediction surface) was 0.50 ± 0.019 for the derivation set and 0.50 ± 0.034 for the validation set. The shaded triangular space indicates chance performance analogous to the diagonal line in a two-dimensional ROC curve which in three dimensions is a VUS of 0.167. Average dichotomized c-index (chance performance=0.50) was 0.84 ± 0.009 and 0.85 ± 0.016 for the derivation and validation sets.

Standardized morbidity and mortality ratios for factors not included in the model (Supplemental Digital Content - Table 6) including PICU type (general medical and cardiac/cardiovascular), elective/emergency admission status, post-intervention category including cardiac interventions, and specific diagnoses such as sepsis, respiratory disease, neurological trauma, and congenital heart disease representing over 50% of the sample had standardized ratios not significantly different from unity. Only the commercial payer type but not government payer type had a standardized mortality ratio less than predicted. Model performance within individual study sites revealed significantly more variability (Figure 4). When the standardized ratios are evaluated independently, four sites had standardized morbidity ratios significantly different from unity (p<0.05), three less than 1.0 indicating significantly less morbidity than predicted by the model and one greater than unity. Two sites had standardized mortality ratios different from unity, one below 1.0 and one significantly greater than 1.0. When both standardized ratios were considered simultaneously in an overall assessment of model fit within each site, 3 sites were significantly different than predicted, one with a higher number of deaths than predicted by the model, a second with a higher number of morbidities and a trend toward fewer deaths than predicted, and a third with number of both mortalities and morbidities significantly lower than predicted.

Standardized Morbidity (Clear Bars) and Mortality (Solid Bars) Ratios with 95% Confidence Intervals For The Individual Sites. Site order does not correspond to Table 2. * = p<0.05 for overall model fit at the indicated site.

DISCUSSION

We demonstrated that new morbidities significantly affecting functional status at hospital discharge were associated with many of the same factors as mortality, including physiological status measured by the PRISM III score, age, admission source, and diagnostic factors, and that morbidity can be modeled simultaneously with mortality. Trichotomous modeling uncovered the phasic association of morbidity risk with physiological status and produced a well-performing model for simultaneous prediction of both morbidity and mortality suitable for risk adjustment in research, quality and other studies.

We found that the association of new morbidity with physiological status was similar to that of mortality, increasing as physiological dysfunction increased and only decreasing as the physiological dysfunction became sufficiently large to change potential morbidities into mortalities. Critical care mortality is usually associated with physiological abnormalities in the cardiovascular, respiratory, neurological and hematological systems.^5,38–42 Our findings are consistent with the hypothesis that new morbidity significantly affecting functional status is often an event along the path toward mortality as both outcomes are strongly associated with the degree of physiological alterations. In pediatric critical care, new morbidity assessed by change in functional status is almost twice as common as mortality and could serve as a new, clinically relevant and important outcome for clinical trials and quality studies to supplement the relatively low rate of mortality.²¹

Our findings have wide implications for research trials and quality programs, especially those currently based on internal or external benchmarking of standardized mortality ratios. First, since pediatric morbidity is more common than mortality and many critical care therapies are aimed at reducing morbidity risk, care assessments that focus on morbidity as well as mortality will have wide appeal and relevance.^43–46 Second, although limited in scope, there was more variability among the participating sites in the standardized morbidity ratios than in the standardized mortality ratios. This suggests quality factors beyond those associated with mortality may influence morbidity and, therefore, the investigation of the variability in standardized morbidity ratios could identify new, important quality factors. Potentially, evaluations of, and improvements in, the structure and process of care analogous to those resulting from the investigations of the variability of standardized mortality ratios could result.^46–51 Third, although this study was conducted in pediatric intensive care units, it is likely that patients in neonatal and adult intensive care units have a similar relationship between morbidity and physiological dysfunction, enabling similar models of morbidity and mortality risks based on the physiological dysfunction scores currently available for those patient populations.^45,52,53 Fourth, non-ICU initiatives that monitor standardized mortality ratios could find relevance and applicability in the simultaneous morbidity and mortality outcome model developed in this study.^{15–19,54–56}

The most important limitation to our study is the lack of long term follow-up to correlate with hospital discharge morbidity. There is continued recovery of function after discharge although the incidence and severity of long-term morbidity is correlated with the discharge condition.^43,57,58 We did find that measuring functional status with the FSS as a morbidity outcome was practical, relevant, and worked well even in this large study. Several other limitations are important. First, the data abstractors were not blinded to the study hypothesis and this has the potential to introduce bias. Second, although the FSS was designed and validated for this project, we did not formally re-assess the accuracy of the data collection.

In conclusion, trichotomous outcome models for pediatric intensive care based on physiological status were developed with performance suitable for use in research trials and quality and other assessments. This approach is likely applicable to other disciplines that are currently dependent on adjusted mortality alone. Given the decreasing PICU mortality rates, the ability to more finely assess outcomes among surviving children in terms of morbidity allows the opportunity to distinguish between different care practices at a more refined level, thereby furthering the opportunity to improve patient outcomes.

Supplementary Material

Figure 3 Suppl Digital. Supplemental Digital Content Figure 3.

The Area Under The Curve For the Most Clinically Relevant 2-Dimensional ROC Curves. The areas under the curves for the derivation and validation sets were 0.89 ± 0.012 and 0.89 ± 0.020 for the survival vs. death dichotomy, 0.79 ± 0.011 and 0.80 ± 0.018 for the death or new morbidity vs. survival without new morbidity dichotomy, and 0.72 ± 0.014 and 0.74 ± 0.024 for the new morbidity vs. death or survival without new morbidity dichotomy.

NIHMS737434-supplement-Figure_3_Suppl_Digital.tif^{(170.9KB, tif)}

Table 3. Supplimental Data. Supplemental Digital Content - Table 3.

Univariate Odds Ratios for Morbidity (N=351) and Mortality (N=214) in the Development Set (N=7560).

NIHMS737434-supplement-Table_3___Supplimental_Data.docx^{(18.4KB, docx)}

Table 6. Supplimental Data File _.doc_ .tif_ pdf_ etc._. Supplemental Digital Content - Table 6.

Standardized Morbidity and Mortality Ratios for Diagnostic and Descriptive Categories Not Included in the Final Model for the Total Sample. Variables are included only for those with at least 10 observed and expected outcomes for both morbidity and mortality.

NIHMS737434-supplement-Table_6__Supplimental_Data_File___doc___tif__pdf__etc__.docx^{(17.2KB, docx)}

Individuals Acknowledged and Roles

Teresa Liu, MPH, CCRP; University of Utah (project management, Data Coordinating Center)

Jean Reardon, MA, BSN, RN; Children’s National Medical Center (institutional project management, data collection)

Elyse Tomanio, BSN, RN; Children’s National Medical Center (institutional project management, data collection)

Morella Menicucci, MD, CCRP; Children’s National Medical Center (data collection)

Fidel Ramos, BA; Children’s National Medical Center (institutional project management, data collection)

Aimee Labell, MS, RN; Phoenix Children’s Hospital (institutional project management, data collection)

Courtney Bliss, BS, DTR; Phoenix Children’s Hospital (data collection)

Jeffrey Terry, MBA; Children’s Hospital Los Angeles (data collection)

Margaret Villa, RN; Children’s Hospital Los Angeles and Mattel Children’s Hospital UCLA (institutional project management, data collection)

Jeni Kwok, JD; Children’s Hospital Los Angeles and Mattel Children’s Hospital (institutional project management, data collection)

Amy Yamakawa, BS; Children’s Hospital Los Angeles and Mattel Children’s Hospital UCLA (data collection)

Ann Pawluszka, BSN, RN; Children’s Hospital of Michigan (institutional project management)

Symone Coleman, BS, MPH; Children’s Hospital of Michigan (data collection)

Melanie Lulic, BS; Children’s Hospital of Michigan (data collection)

Mary Ann DiLiberto, BS, RN, CCRC; Children’s Hospital of Philadelphia (institutional project management, data collection)

Carolann Twelves, BSN, RN; Children’s Hospital of Philadelphia (data collection)

Monica S. Weber, RN, BSN, CCRP; University of Michigan (institutional project management, data collection)

Lauren Conlin, BSN, RN, CCRP; University of Michigan (data collection)

Alan C. Abraham, BA, CCRC; Children’s Hospital of Pittsburgh of University of Pittsburgh Medical Center (institutional project management, data collection)

Jennifer Jones, RN; Children’s Hospital of Pittsburgh of University of Pittsburgh Medical Center (data collection)

Jeri Burr, MS, RN-BC, CCRC; University of Utah (project management, Data Coordinating Center)

Nichol Nunn, BS, MBA; University of Utah (project management, Data Coordinating Center)

Alecia Peterson, BS, CMC; University of Utah (project management, Data Coordinating Center)

Carol Nicholson, MD (former Project Officer, Eunice Kennedy Shriver National Institute of Child Health and Human Development, National Institutes of Health, Department of Health and Human Services, for part of the study period)

Funding Source: Supported, in part, by the following cooperative agreements from the Eunice Kennedy Shriver National Institute of Child Health and Human Development, National Institutes of Health, Department of Health and Human Services: U10HD050096, U10HD049981, U10HD049983, U10HD050012, U10HD063108, U10HD063106, U10HD063114 and U01HD049934.

Data Access, Responsibility, and Analysis: Dr. Murray M Pollack and Dr. Richard Holubkov had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Copyright form disclosures: Dr. Pollack received grant support from the NICHD, has planned patents through Children's National Medical Center (his not-for-profit employer), and received support for article research from the National Institutes of Health (NIH). Dr. Holubkov served as a board member for Pfizer, the National Burn Association, and Fibrocell, Inc. (Data Safety Board Membership); consulted for St. Jude Medical, Inc., Physicians Committee for Responsible Medicine, and Covidien, Inc. (Biostatistical Consulting); received support for article research from the NIH. Dr. Holubkov and his institution received grant support (Salary support [biostatistician]) and support for travel (Steering Committee Meeting Travel) from the NIH. Dr. Funai received support for article research from the NIH. Dr. Funai and his institution received grant support from the NIH (salaries support Biostatistician). Dr. Berger received support for article research from the NIH. His institution received grant support from the NIH and the Association for Pediatric Pulmonary Hypertension. Dr. Clark received support for article research from the NIH. Her institution received grant support from the NIH. Dr. Meert received support for article research from the NIH. Her institution received grant support from the NIH. Dr. Berg received grant support from the NICHD and received support for article research from the NIH. His institution received Dr. Carcillo received support for article research from the NIH. His institution received grant support and support for travel. Dr. Wessel received support for article research from the NIH. His institution received grant support from the NIH (ongoing). Dr. Moler received support for article research from the NIH. His institution received grant support and support for travel from the NICHD. Dr. Dalton lectured for rEVO Biologics, received royalties from the Society of Critical Care Medicine (SCCM) for Pediatric Multidisciplinary Board Review Book, and received support for article research from the NIH. Her institution received grant support from the NIH. Dr. Newth received support for article research from the NIH. His institution received grant support from the NIH / NICHD (Collaborative Pediatric Critical Care Research Network). Dr. Shanley received support for article research from the NIH and served as a board member for International Ped Res Foundation (SPR Representative on Exec Comm). His institution received grant support from the NICHD (CPCCRN Grant) and the NIH (CTSA Grant) and received support for travel from the NICHD (CPCCRN Grant). Dr. Harrison lectured for the SCCM (Board review lectures for my professional society) and received support for article research from the NIH. His institution received grant support from the NIH (CPCCRN funding and THAPCA trial funding via NIH).

Dr. Doctor consulted for the NICHD/CPCCRN (honorarium to serve as SC chairperson), Terumo, BCT, iNO Therapeutics, Novartis, and Galleon Pharmaceuticals; received support for travel from the NICHD/CPCCRN; and lectured for Terumo, BCT. His institution received grant support from the NIH, AHA, and the Children's Discovery Institute. Dr. Jenkins received support for article research from the NIH and disclosed government work. Dr. Tamburro received grant support from the US FDA Office of Orphan Product Development Grant to study calfactant in pediatric stem cell transplant patients (No longer co-PI on the trial secondary to accepting a new position), received royalties from Springer Publisher (received royalties for serving as an associate editor on a pediatric critical care textbook and study guide), received support for article research from the NIH, and disclosed government work. Dr. Dean received support for article research from the NIH. His institution received grant support from the NIH and HRSA.

Footnotes

Data Presentation: Results will be presented, in part, at the Annual Society of Critical Care meeting in January, 2015 and have been submitted to the Academic Pediatric Societies Meeting for April 2015.

Contributor Information

Murray M. Pollack, Department of Pediatrics, Children’s National Medical Center and the George Washington University School of Medicine and Health Sciences, Washington DC and Phoenix Children’s Hospital and the University Of Arizona School of Medicine Phoenix.

Richard Holubkov, Department of Pediatrics, University of Utah School of Medicine, Salt Lake City, UT.

Tomohiko Funai, Department of Pediatrics, University of Utah School of Medicine, Salt Lake City, UT.

John T. Berger, Department of Pediatrics, Children's National Medical Center, Washington DC.

Amy E. Clark, Department of Pediatrics, University of Utah School of Medicine, Salt Lake City, UT.

Kathleen Meert, Department of Pediatrics, Children's Hospital of Michigan, Detroit, MI.

Robert A. Berg, Department of Pediatrics, Children’s Hospital of Philadelphia, Philadelphia, PA.

Joseph Carcillo, Department of Critical Care Medicine, Children's Hospital of Pittsburgh, Pittsburgh, PA.

David L. Wessel, Department of Pediatrics, Children's National Medical Center, Washington DC.

Frank Moler, Department of Pediatrics, University of Michigan, Ann Arbor, MI.

Heidi Dalton, Department of Child Health, Phoenix Children’s Hospital and University of Arizona College of Medicine-Phoenix, Phoenix, AZ.

Christopher J. L. Newth, Department of Anesthesiology and Critical Care Medicine, Children's Hospital Los Angeles, Los Angeles, CA.

Thomas Shanley, Department of Pediatrics, University of Michigan, Ann Arbor, MI.

Rick E. Harrison, Department of Pediatrics, University of California at Los Angeles, Los Angeles, CA.

Allan Doctor, Departments of Pediatrics and Biochemistry, Washington University School of Medicine, St. Louis, MO.

Tammara L. Jenkins, Pediatric Trauma and Critical Illness Branch, Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD), the National Institutes of Health (NIH), Bethesda, MD.

Robert Tamburro, Pediatric Trauma and Critical Illness Branch, Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD), the National Institutes of Health (NIH), Bethesda, MD.

J. Michael Dean, Department of Pediatrics, University of Utah School of Medicine, Salt Lake City, UT for the Eunice Kennedy Shriver National Institute of Child Health and Human Development Collaborative Pediatric Critical Care Research Network (CPCCRN).

REFERENCES

1.Pollack MM, Ruttimann UE, Getson PR. Accurate prediction of the outcome of pediatric intensive care. A new quantitative method. N Engl J Med. 1987 Jan 15;316(3):134–139. doi: 10.1056/NEJM198701153160304. [DOI] [PubMed] [Google Scholar]
2.Knaus WA, Zimmerman JE, Wagner DP, Draper EA, Lawrence DE. APACHE-acute physiology and chronic health evaluation: a physiologically based classification system. Critical Care Medicine. 1981 Aug;9(8):591–597. doi: 10.1097/00003246-198108000-00008. [DOI] [PubMed] [Google Scholar]
3.Teres D, Lemeshow S, Avrunin JS, Pastides H. Validation of the mortality prediction model for ICU patients. Critical Care Medicine. 1987 Mar;15(3):208–213. doi: 10.1097/00003246-198703000-00005. [DOI] [PubMed] [Google Scholar]
4.Richardson DK, Gray JE, McCormick MC, Workman K, Goldmann DA. Score for Neonatal Acute Physiology: a physiologic severity index for neonatal intensive care. Pediatrics. 1993 Mar;91(3):617–623. [PubMed] [Google Scholar]
5.Pollack MM, Patel KM, Ruttimann UE. PRISM III: an updated Pediatric Risk of Mortality score. Critical Care Medicine. 1996 May;24(5):743–752. doi: 10.1097/00003246-199605000-00004. [DOI] [PubMed] [Google Scholar]
6.Zimmerman JE, Kramer AA, McNair DS, Malila FM. Acute Physiology and Chronic Health Evaluation (APACHE) IV: hospital mortality assessment for today's critically ill patients. Critical Care Medicine. 2006 May;34(5):1297–1310. doi: 10.1097/01.CCM.0000215112.84523.F0. [DOI] [PubMed] [Google Scholar]
7.Horbar JD, Carpenter JH, Badger GJ, et al. Mortality and neonatal morbidity among infants 501 to 1500 grams from 2000 to 2009. Pediatrics. 2012 Jun;129(6):1019–1026. doi: 10.1542/peds.2011-3028. [DOI] [PubMed] [Google Scholar]
8.Le Gall JR, Loirat P, Alperovitch A, et al. A simplified acute physiology score for ICU patients. Critical Care Medicine. 1984;12(11):975–977. doi: 10.1097/00003246-198411000-00012. [DOI] [PubMed] [Google Scholar]
9.Shann F, Pearson G, Slater A, Wilkinson K. Paediatric index of mortality (PIM): a mortality prediction model for children in intensive care. Intensive Care Medicine. 1997 Feb;23(2):201–207. doi: 10.1007/s001340050317. [DOI] [PubMed] [Google Scholar]
10.Horbar JD, Soll RF, Edwards WH. The Vermont Oxford Network: a community of practice. Clin Perinatol. 2010 Mar;37(1):29–47. doi: 10.1016/j.clp.2010.01.003. [DOI] [PubMed] [Google Scholar]
11.Wetzel RC. The virtual pediatric intensive care unit. Practice in the new millennium. Pediatr Clin North Am. 2001 Jun;48(3):795–814. doi: 10.1016/s0031-3955(05)70340-0. [DOI] [PubMed] [Google Scholar]
12.Cook SF, Visscher WA, Hobbs CL, Williams RL Project ICIC. Project IMPACT: results from a pilot validity study of a new observational database. Critical Care Medicine. 2002;30(12):2765–2770. doi: 10.1097/00003246-200212000-00024. [DOI] [PubMed] [Google Scholar]
13.Elixhauser A, Pancholi M, Clancy CM. Using the AHRQ quality indicators to improve health care quality. Jt Comm J Qual Patient Saf. 2005 Sep;31(9):533–538. doi: 10.1016/s1553-7250(05)31069-5. [DOI] [PubMed] [Google Scholar]
14.Daily OP, Kauffman HM. Quality control of the OPTN/UNOS Transplant Registry. Transplantation. 2004 Apr 27;77(8):1309–1310. doi: 10.1097/01.tp.0000120943.94789.e4. [DOI] [PubMed] [Google Scholar]
15.Hussey PS, Burns RM, Weinick RM, Mayer L, Cerese J, Farley DO. Using a hospital quality improvement toolkit to improve performance on the AHRQ quality indicators. Jt Comm J Qual Patient Saf. 2013 Apr;39(4):177–184. doi: 10.1016/s1553-7250(13)39024-2. [DOI] [PubMed] [Google Scholar]
16.Stausberg J. The best period for mortality rates associated with hospital stay: hospital mortality performs well for nonsurgical diagnostic groups. Qual Manag Health Care. 2011 Jul-Sep;20(3):198–206. doi: 10.1097/QMH.0b013e318223d00a. [DOI] [PubMed] [Google Scholar]
17.Forthman MT, Gold RS, Dove HG, Henderson RD. Risk-adjusted indices for measuring the quality of inpatient care. Qual Manag Health Care. 2010 Jul-Sep;19(3):265–277. doi: 10.1097/QMH.0b013e3181eb143d. [DOI] [PubMed] [Google Scholar]
18.Shahian DM, Wolf RE, Iezzoni LI, Kirle L, Normand S-LT. Variability in the measurement of hospital-wide mortality rates. N Engl J Med. 2010 Dec 23;363(26):2530–2539. doi: 10.1056/NEJMsa1006396. [Erratum appears in N Engl J Med. 2011 Apr 7;364(14)1382]. [DOI] [PubMed] [Google Scholar]
19.Borzecki AM, Christiansen CL, Loveland S, Chew P, Rosen AK. Trends in the inpatient quality indicators: the Veterans Health Administration experience. Med Care. 2010 Aug;48(8):694–702. doi: 10.1097/MLR.0b013e3181e419e3. [DOI] [PubMed] [Google Scholar]
20.Pollack MM, Holubkov R, Funai T, et al. Relationship between the functional status scale and the pediatric overall performance category and pediatric cerebral performance category scales. JAMA Pediatrics. 2014 Jul;168(7):671–676. doi: 10.1001/jamapediatrics.2013.5316. [DOI] [PMC free article] [PubMed] [Google Scholar]
21.Pollack MM, Holubkov R, Funai T, et al. Pediatric Intensive Care Outcomes: Development of New Morbidities During Pediatric Critical Care. Pediatr Crit Care Med. 2014;15(9):821–827. doi: 10.1097/PCC.0000000000000250. [DOI] [PMC free article] [PubMed] [Google Scholar]
22.Marion DW, Penrod LE, Kelsey SF, et al. Treatment of traumatic brain injury with moderate hypothermia. N Engl J Med. 1997 Feb 20;336(8):540–546. doi: 10.1056/NEJM199702203360803. [DOI] [PubMed] [Google Scholar]
23.Nielsen N, Wetterslev J, Cronberg T, et al. Targeted temperature management at 33degreeC versus 36degreeC after cardiac arrest. N Engl J Med. 2013 Dec 5;369(23):2197–2206. doi: 10.1056/NEJMoa1310519. [DOI] [PubMed] [Google Scholar]
24.Shankaran S, Pappas A, McDonald SA, et al. Childhood outcomes after hypothermia for neonatal encephalopathy. N Engl J Med. 2012 May 31;366(22):2085–2092. doi: 10.1056/NEJMoa1112066. [Erratum appears in N Engl J Med. 2012 Sep 13;367(11)1073]. [DOI] [PMC free article] [PubMed] [Google Scholar]
25.Bulger EM, May S, Brasel KJ, et al. Out-of-hospital hypertonic resuscitation following severe traumatic brain injury: a randomized controlled trial. Jama. 2010 Oct 6;304(13):1455–1464. doi: 10.1001/jama.2010.1405. [DOI] [PMC free article] [PubMed] [Google Scholar]
26.Mentzelopoulos SD, Malachias S, Chamos C, et al. Vasopressin, steroids, and epinephrine and neurologically favorable survival after in-hospital cardiac arrest: a randomized clinical trial. Jama. 2013 Jul 17;310(3):270–279. doi: 10.1001/jama.2013.7832. [DOI] [PubMed] [Google Scholar]
27.Jauch EC, Saver JL, Adams HP, Jr, et al. Guidelines for the early management of patients with acute ischemic stroke: a guideline for healthcare professionals from the American Heart Association/American Stroke Association. Stroke. 2013 Mar;44(3):870–947. doi: 10.1161/STR.0b013e318284056a. [DOI] [PubMed] [Google Scholar]
28.Dellinger RP, Levy MM, Rhodes A, et al. Surviving sepsis campaign: international guidelines for management of severe sepsis and septic shock, 2012. Critical Care Medicine. 2013 Feb;41(2):580–637. doi: 10.1097/CCM.0b013e31827e83af. [DOI] [PubMed] [Google Scholar]
29.Willson DF, Dean JM, Meert KL, et al. Collaborative pediatric critical care research network: looking back and moving forward. Pediatr Crit Care Med. 2010;11(1):1–6. doi: 10.1097/PCC.0b013e3181c01302. [DOI] [PMC free article] [PubMed] [Google Scholar]
30.Pollack MM, Dean JM, Butler J, et al. The ideal time interval for critical care severity-of-illness assessment. Pediatr Crit Care Med. 2013 Jun;14(5):448–453. doi: 10.1097/PCC.0b013e31828a7270. [DOI] [PMC free article] [PubMed] [Google Scholar]
31.Pollack MM, Holubkov R, Glass P, et al. Functional Status Scale: new pediatric outcome measure. Pediatrics. 2009 Jul;124(1):e18–e28. doi: 10.1542/peds.2008-1987. [DOI] [PMC free article] [PubMed] [Google Scholar]
32.Fagerland MW, Hosmer DW, Bofin AM. Multinomial goodness-of-fit tests for logistic regression models. Stat Med. 2008 Sep 20;27(21):4238–4253. doi: 10.1002/sim.3202. [DOI] [PubMed] [Google Scholar]
33.Robin X, Turck N, Hainard A, et al. pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics. 12:77. doi: 10.1186/1471-2105-12-77. [DOI] [PMC free article] [PubMed] [Google Scholar]
34.Li J, Fine JP. ROC analysis with multiple classes and multiple tests: methodology and its application in microarray studies. Biostatistics. 2008 Jul;9(3):566–576. doi: 10.1093/biostatistics/kxm050. [DOI] [PubMed] [Google Scholar]
35.Mossman D. Three-way ROCs. Med Decis Making. 1999 Jan-Mar;19(1):78–89. doi: 10.1177/0272989X9901900110. [DOI] [PubMed] [Google Scholar]
36.Waegeman WDBB, Boullart L. ROC analysis in ordinal regression learning. Pattern Recognition Letters. 2008;29(1):1–9. [Google Scholar]
37.DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988 Sep;44(3):837–845. [PubMed] [Google Scholar]
38.Cook R, Cook D, Tilley J, Lee K, Marshall J, Canadian Critical Care Trials G. Multiple organ dysfunction: baseline and serial component scores. Critical Care Medicine. 2001 Nov;29(11):2046–2050. doi: 10.1097/00003246-200111000-00002. [DOI] [PubMed] [Google Scholar]
39.Ferreira AMP, Sakr Y. Organ dysfunction: general approach, epidemiology, and organ failure scores. Semin. 2011 Oct;32(5):543–551. doi: 10.1055/s-0031-1287862. [DOI] [PubMed] [Google Scholar]
40.Mayr VD, Dunser MW, Greil V, et al. Causes of death and determinants of outcome in critically ill patients. Crit Care. 2006;10(6):R154. doi: 10.1186/cc5086. [DOI] [PMC free article] [PubMed] [Google Scholar]
41.Proulx F, Joyal JS, Mariscalco MM, Leteurtre S, Leclerc F, Lacroix J. The pediatric multiple organ dysfunction syndrome. Pediatr Crit Care Med. 2009 Jan;10(1):12–22. doi: 10.1097/PCC.0b013e31819370a9. [DOI] [PubMed] [Google Scholar]
42.Wilkinson JD, Pollack MM, Ruttimann UE, Glass NL, Yeh TS. Outcome of pediatric patients with multiple organ system failure. Critical Care Medicine. 1986 Apr;14(4):271–274. doi: 10.1097/00003246-198604000-00002. [DOI] [PubMed] [Google Scholar]
43.Fiser DH, Long N, Roberson PK, Hefley G, Zolten K, Brodie-Fowler M. Relationship of pediatric overall performance category and pediatric cerebral performance category scores at pediatric intensive care unit discharge with outcome measures collected at hospital discharge and 1- and 6-month follow-up assessments. Critical Care Medicine. 2000 Jul;28(7):2616–2620. doi: 10.1097/00003246-200007000-00072. [DOI] [PubMed] [Google Scholar]
44.Marshall JC, Vincent J-L, Guyatt G, et al. Outcome measures for clinical research in sepsis: a report of the 2nd Cambridge Colloquium of the International Sepsis Forum. Critical Care Medicine. 2005 Aug;33(8):1708–1716. doi: 10.1097/01.ccm.0000174478.70338.03. [DOI] [PubMed] [Google Scholar]
45.Donnino MW, Salciccioli JD, Dejam A, et al. APACHE II scoring to predict outcome in post-cardiac arrest. Resuscitation. 2013 May;84(5):651–656. doi: 10.1016/j.resuscitation.2012.10.024. [DOI] [PMC free article] [PubMed] [Google Scholar]
46.Burstein DS, Jacobs JP, Li JS, et al. Care models and associated outcomes in congenital heart surgery. Pediatrics. 2011 Jun;127(6):e1482–e1489. doi: 10.1542/peds.2010-2796. [DOI] [PMC free article] [PubMed] [Google Scholar]
47.Pollack MM, Cuerdon TT, Patel KM, Ruttimann UE, Getson PR, Levetown M. Impact of quality-of-care factors on pediatric intensive care unit mortality. Jama. 1994 Sep 28;272(12):941–946. [PubMed] [Google Scholar]
48.Pollack MM, Alexander SR, Clarke N, Ruttimann UE, Tesselaar HM, Bachulis AC. Improved outcomes from tertiary center pediatric intensive care: a statewide comparison of tertiary and nontertiary care facilities. Critical Care Medicine. 1991 Feb;19(2):150–159. doi: 10.1097/00003246-199102000-00007. [DOI] [PubMed] [Google Scholar]
49.Checkley W, Martin GS, Brown SM, et al. Structure, process, and annual ICU mortality across 69 centers: United States Critical Illness and Injury Trials Group Critical Illness Outcomes Study. Critical Care Medicine. 2014 Feb;42(2):344–356. doi: 10.1097/CCM.0b013e3182a275d7. [DOI] [PMC free article] [PubMed] [Google Scholar]
50.Shortell SM, Zimmerman JE, Rousseau DM, et al. The performance of intensive care units: does good management make a difference? Med Care. 1994 May;32(5):508–525. doi: 10.1097/00005650-199405000-00009. [DOI] [PubMed] [Google Scholar]
51.Kerlin MP, Small DS, Cooney E, et al. A randomized trial of nighttime physician staffing in an intensive care unit. N Engl J Med. 2013 Jun 6;368(23):2201–2209. doi: 10.1056/NEJMoa1302854. [DOI] [PMC free article] [PubMed] [Google Scholar]
52.Kramer AA, Higgins TL, Zimmerman JE. Comparison of the Mortality Probability Admission Model III, National Quality Forum, and Acute Physiology and Chronic Health Evaluation IV hospital mortality models: implications for national benchmarking*. Critical Care Medicine. 2014 Mar;42(3):544–553. doi: 10.1097/CCM.0b013e3182a66a49. [DOI] [PubMed] [Google Scholar]
53.Zupancic JAF, Richardson DK, Horbar JD, et al. Revalidation of the Score for Neonatal Acute Physiology in the Vermont Oxford Network. Pediatrics. 2007 Jan;119(1):e156–e163. doi: 10.1542/peds.2005-2957. [DOI] [PubMed] [Google Scholar]
54.Borzecki AM, Christiansen CL, Chew P, Loveland S, Rosen AK. Comparison of in-hospital versus 30-day mortality assessments for selected medical conditions. Med Care. 2010 Dec;48(12):1117–1121. doi: 10.1097/MLR.0b013e3181ef9d53. [DOI] [PubMed] [Google Scholar]
55.Black N. Assessing the quality of hospitals. Bmj. 2010;340:c2066. doi: 10.1136/bmj.c2066. [DOI] [PubMed] [Google Scholar]
56.Shahian DM, Iezzoni LI, Meyer GS, Kirle L, Normand S-LT. Hospital-wide mortality as a quality metric: conceptual and methodological challenges. Am J Med Qual. 2012 Mar-Apr;27(2):112–123. doi: 10.1177/1062860611412358. [DOI] [PubMed] [Google Scholar]
57.Knoester H, Bronner MB, Bos AP. Surviving pediatric intensive care: physical outcome after 3 months. Intensive Care Medicine. 2008 Jun;34(6):1076–1082. doi: 10.1007/s00134-008-1061-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
58.Graf J, Koch M, Dujardin R, Kersten A, Janssens U. Health-related quality of life before:1 month after, and 9 months after intensive care in medical cardiovascular and pulmonary patients. Critical Care Medicine. 2003 Aug;31(8):2163–2169. doi: 10.1097/01.CCM.0000079607.87009.3A. [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Figure 3 Suppl Digital. Supplemental Digital Content Figure 3.

NIHMS737434-supplement-Figure_3_Suppl_Digital.tif^{(170.9KB, tif)}

Table 3. Supplimental Data. Supplemental Digital Content - Table 3.

Univariate Odds Ratios for Morbidity (N=351) and Mortality (N=214) in the Development Set (N=7560).

NIHMS737434-supplement-Table_3___Supplimental_Data.docx^{(18.4KB, docx)}

Table 6. Supplimental Data File _.doc_ .tif_ pdf_ etc._. Supplemental Digital Content - Table 6.

NIHMS737434-supplement-Table_6__Supplimental_Data_File___doc___tif__pdf__etc__.docx^{(17.2KB, docx)}

[R1] 1.Pollack MM, Ruttimann UE, Getson PR. Accurate prediction of the outcome of pediatric intensive care. A new quantitative method. N Engl J Med. 1987 Jan 15;316(3):134–139. doi: 10.1056/NEJM198701153160304. [DOI] [PubMed] [Google Scholar]

[R2] 2.Knaus WA, Zimmerman JE, Wagner DP, Draper EA, Lawrence DE. APACHE-acute physiology and chronic health evaluation: a physiologically based classification system. Critical Care Medicine. 1981 Aug;9(8):591–597. doi: 10.1097/00003246-198108000-00008. [DOI] [PubMed] [Google Scholar]

[R3] 3.Teres D, Lemeshow S, Avrunin JS, Pastides H. Validation of the mortality prediction model for ICU patients. Critical Care Medicine. 1987 Mar;15(3):208–213. doi: 10.1097/00003246-198703000-00005. [DOI] [PubMed] [Google Scholar]

[R4] 4.Richardson DK, Gray JE, McCormick MC, Workman K, Goldmann DA. Score for Neonatal Acute Physiology: a physiologic severity index for neonatal intensive care. Pediatrics. 1993 Mar;91(3):617–623. [PubMed] [Google Scholar]

[R5] 5.Pollack MM, Patel KM, Ruttimann UE. PRISM III: an updated Pediatric Risk of Mortality score. Critical Care Medicine. 1996 May;24(5):743–752. doi: 10.1097/00003246-199605000-00004. [DOI] [PubMed] [Google Scholar]

[R6] 6.Zimmerman JE, Kramer AA, McNair DS, Malila FM. Acute Physiology and Chronic Health Evaluation (APACHE) IV: hospital mortality assessment for today's critically ill patients. Critical Care Medicine. 2006 May;34(5):1297–1310. doi: 10.1097/01.CCM.0000215112.84523.F0. [DOI] [PubMed] [Google Scholar]

[R7] 7.Horbar JD, Carpenter JH, Badger GJ, et al. Mortality and neonatal morbidity among infants 501 to 1500 grams from 2000 to 2009. Pediatrics. 2012 Jun;129(6):1019–1026. doi: 10.1542/peds.2011-3028. [DOI] [PubMed] [Google Scholar]

[R8] 8.Le Gall JR, Loirat P, Alperovitch A, et al. A simplified acute physiology score for ICU patients. Critical Care Medicine. 1984;12(11):975–977. doi: 10.1097/00003246-198411000-00012. [DOI] [PubMed] [Google Scholar]

[R9] 9.Shann F, Pearson G, Slater A, Wilkinson K. Paediatric index of mortality (PIM): a mortality prediction model for children in intensive care. Intensive Care Medicine. 1997 Feb;23(2):201–207. doi: 10.1007/s001340050317. [DOI] [PubMed] [Google Scholar]

[R10] 10.Horbar JD, Soll RF, Edwards WH. The Vermont Oxford Network: a community of practice. Clin Perinatol. 2010 Mar;37(1):29–47. doi: 10.1016/j.clp.2010.01.003. [DOI] [PubMed] [Google Scholar]

[R11] 11.Wetzel RC. The virtual pediatric intensive care unit. Practice in the new millennium. Pediatr Clin North Am. 2001 Jun;48(3):795–814. doi: 10.1016/s0031-3955(05)70340-0. [DOI] [PubMed] [Google Scholar]

[R12] 12.Cook SF, Visscher WA, Hobbs CL, Williams RL Project ICIC. Project IMPACT: results from a pilot validity study of a new observational database. Critical Care Medicine. 2002;30(12):2765–2770. doi: 10.1097/00003246-200212000-00024. [DOI] [PubMed] [Google Scholar]

[R13] 13.Elixhauser A, Pancholi M, Clancy CM. Using the AHRQ quality indicators to improve health care quality. Jt Comm J Qual Patient Saf. 2005 Sep;31(9):533–538. doi: 10.1016/s1553-7250(05)31069-5. [DOI] [PubMed] [Google Scholar]

[R14] 14.Daily OP, Kauffman HM. Quality control of the OPTN/UNOS Transplant Registry. Transplantation. 2004 Apr 27;77(8):1309–1310. doi: 10.1097/01.tp.0000120943.94789.e4. [DOI] [PubMed] [Google Scholar]

[R15] 15.Hussey PS, Burns RM, Weinick RM, Mayer L, Cerese J, Farley DO. Using a hospital quality improvement toolkit to improve performance on the AHRQ quality indicators. Jt Comm J Qual Patient Saf. 2013 Apr;39(4):177–184. doi: 10.1016/s1553-7250(13)39024-2. [DOI] [PubMed] [Google Scholar]

[R16] 16.Stausberg J. The best period for mortality rates associated with hospital stay: hospital mortality performs well for nonsurgical diagnostic groups. Qual Manag Health Care. 2011 Jul-Sep;20(3):198–206. doi: 10.1097/QMH.0b013e318223d00a. [DOI] [PubMed] [Google Scholar]

[R17] 17.Forthman MT, Gold RS, Dove HG, Henderson RD. Risk-adjusted indices for measuring the quality of inpatient care. Qual Manag Health Care. 2010 Jul-Sep;19(3):265–277. doi: 10.1097/QMH.0b013e3181eb143d. [DOI] [PubMed] [Google Scholar]

[R18] 18.Shahian DM, Wolf RE, Iezzoni LI, Kirle L, Normand S-LT. Variability in the measurement of hospital-wide mortality rates. N Engl J Med. 2010 Dec 23;363(26):2530–2539. doi: 10.1056/NEJMsa1006396. [Erratum appears in N Engl J Med. 2011 Apr 7;364(14)1382]. [DOI] [PubMed] [Google Scholar]

[R19] 19.Borzecki AM, Christiansen CL, Loveland S, Chew P, Rosen AK. Trends in the inpatient quality indicators: the Veterans Health Administration experience. Med Care. 2010 Aug;48(8):694–702. doi: 10.1097/MLR.0b013e3181e419e3. [DOI] [PubMed] [Google Scholar]

[R20] 20.Pollack MM, Holubkov R, Funai T, et al. Relationship between the functional status scale and the pediatric overall performance category and pediatric cerebral performance category scales. JAMA Pediatrics. 2014 Jul;168(7):671–676. doi: 10.1001/jamapediatrics.2013.5316. [DOI] [PMC free article] [PubMed] [Google Scholar]

[R21] 21.Pollack MM, Holubkov R, Funai T, et al. Pediatric Intensive Care Outcomes: Development of New Morbidities During Pediatric Critical Care. Pediatr Crit Care Med. 2014;15(9):821–827. doi: 10.1097/PCC.0000000000000250. [DOI] [PMC free article] [PubMed] [Google Scholar]

[R22] 22.Marion DW, Penrod LE, Kelsey SF, et al. Treatment of traumatic brain injury with moderate hypothermia. N Engl J Med. 1997 Feb 20;336(8):540–546. doi: 10.1056/NEJM199702203360803. [DOI] [PubMed] [Google Scholar]

[R23] 23.Nielsen N, Wetterslev J, Cronberg T, et al. Targeted temperature management at 33degreeC versus 36degreeC after cardiac arrest. N Engl J Med. 2013 Dec 5;369(23):2197–2206. doi: 10.1056/NEJMoa1310519. [DOI] [PubMed] [Google Scholar]

[R24] 24.Shankaran S, Pappas A, McDonald SA, et al. Childhood outcomes after hypothermia for neonatal encephalopathy. N Engl J Med. 2012 May 31;366(22):2085–2092. doi: 10.1056/NEJMoa1112066. [Erratum appears in N Engl J Med. 2012 Sep 13;367(11)1073]. [DOI] [PMC free article] [PubMed] [Google Scholar]

[R25] 25.Bulger EM, May S, Brasel KJ, et al. Out-of-hospital hypertonic resuscitation following severe traumatic brain injury: a randomized controlled trial. Jama. 2010 Oct 6;304(13):1455–1464. doi: 10.1001/jama.2010.1405. [DOI] [PMC free article] [PubMed] [Google Scholar]

[R26] 26.Mentzelopoulos SD, Malachias S, Chamos C, et al. Vasopressin, steroids, and epinephrine and neurologically favorable survival after in-hospital cardiac arrest: a randomized clinical trial. Jama. 2013 Jul 17;310(3):270–279. doi: 10.1001/jama.2013.7832. [DOI] [PubMed] [Google Scholar]

[R27] 27.Jauch EC, Saver JL, Adams HP, Jr, et al. Guidelines for the early management of patients with acute ischemic stroke: a guideline for healthcare professionals from the American Heart Association/American Stroke Association. Stroke. 2013 Mar;44(3):870–947. doi: 10.1161/STR.0b013e318284056a. [DOI] [PubMed] [Google Scholar]

[R28] 28.Dellinger RP, Levy MM, Rhodes A, et al. Surviving sepsis campaign: international guidelines for management of severe sepsis and septic shock, 2012. Critical Care Medicine. 2013 Feb;41(2):580–637. doi: 10.1097/CCM.0b013e31827e83af. [DOI] [PubMed] [Google Scholar]

[R29] 29.Willson DF, Dean JM, Meert KL, et al. Collaborative pediatric critical care research network: looking back and moving forward. Pediatr Crit Care Med. 2010;11(1):1–6. doi: 10.1097/PCC.0b013e3181c01302. [DOI] [PMC free article] [PubMed] [Google Scholar]

[R30] 30.Pollack MM, Dean JM, Butler J, et al. The ideal time interval for critical care severity-of-illness assessment. Pediatr Crit Care Med. 2013 Jun;14(5):448–453. doi: 10.1097/PCC.0b013e31828a7270. [DOI] [PMC free article] [PubMed] [Google Scholar]

[R31] 31.Pollack MM, Holubkov R, Glass P, et al. Functional Status Scale: new pediatric outcome measure. Pediatrics. 2009 Jul;124(1):e18–e28. doi: 10.1542/peds.2008-1987. [DOI] [PMC free article] [PubMed] [Google Scholar]

[R32] 32.Fagerland MW, Hosmer DW, Bofin AM. Multinomial goodness-of-fit tests for logistic regression models. Stat Med. 2008 Sep 20;27(21):4238–4253. doi: 10.1002/sim.3202. [DOI] [PubMed] [Google Scholar]

[R33] 33.Robin X, Turck N, Hainard A, et al. pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics. 12:77. doi: 10.1186/1471-2105-12-77. [DOI] [PMC free article] [PubMed] [Google Scholar]

[R34] 34.Li J, Fine JP. ROC analysis with multiple classes and multiple tests: methodology and its application in microarray studies. Biostatistics. 2008 Jul;9(3):566–576. doi: 10.1093/biostatistics/kxm050. [DOI] [PubMed] [Google Scholar]

[R35] 35.Mossman D. Three-way ROCs. Med Decis Making. 1999 Jan-Mar;19(1):78–89. doi: 10.1177/0272989X9901900110. [DOI] [PubMed] [Google Scholar]

[R36] 36.Waegeman WDBB, Boullart L. ROC analysis in ordinal regression learning. Pattern Recognition Letters. 2008;29(1):1–9. [Google Scholar]

[R37] 37.DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988 Sep;44(3):837–845. [PubMed] [Google Scholar]

[R38] 38.Cook R, Cook D, Tilley J, Lee K, Marshall J, Canadian Critical Care Trials G. Multiple organ dysfunction: baseline and serial component scores. Critical Care Medicine. 2001 Nov;29(11):2046–2050. doi: 10.1097/00003246-200111000-00002. [DOI] [PubMed] [Google Scholar]

[R39] 39.Ferreira AMP, Sakr Y. Organ dysfunction: general approach, epidemiology, and organ failure scores. Semin. 2011 Oct;32(5):543–551. doi: 10.1055/s-0031-1287862. [DOI] [PubMed] [Google Scholar]

[R40] 40.Mayr VD, Dunser MW, Greil V, et al. Causes of death and determinants of outcome in critically ill patients. Crit Care. 2006;10(6):R154. doi: 10.1186/cc5086. [DOI] [PMC free article] [PubMed] [Google Scholar]

[R41] 41.Proulx F, Joyal JS, Mariscalco MM, Leteurtre S, Leclerc F, Lacroix J. The pediatric multiple organ dysfunction syndrome. Pediatr Crit Care Med. 2009 Jan;10(1):12–22. doi: 10.1097/PCC.0b013e31819370a9. [DOI] [PubMed] [Google Scholar]

[R42] 42.Wilkinson JD, Pollack MM, Ruttimann UE, Glass NL, Yeh TS. Outcome of pediatric patients with multiple organ system failure. Critical Care Medicine. 1986 Apr;14(4):271–274. doi: 10.1097/00003246-198604000-00002. [DOI] [PubMed] [Google Scholar]

[R43] 43.Fiser DH, Long N, Roberson PK, Hefley G, Zolten K, Brodie-Fowler M. Relationship of pediatric overall performance category and pediatric cerebral performance category scores at pediatric intensive care unit discharge with outcome measures collected at hospital discharge and 1- and 6-month follow-up assessments. Critical Care Medicine. 2000 Jul;28(7):2616–2620. doi: 10.1097/00003246-200007000-00072. [DOI] [PubMed] [Google Scholar]

[R44] 44.Marshall JC, Vincent J-L, Guyatt G, et al. Outcome measures for clinical research in sepsis: a report of the 2nd Cambridge Colloquium of the International Sepsis Forum. Critical Care Medicine. 2005 Aug;33(8):1708–1716. doi: 10.1097/01.ccm.0000174478.70338.03. [DOI] [PubMed] [Google Scholar]

[R45] 45.Donnino MW, Salciccioli JD, Dejam A, et al. APACHE II scoring to predict outcome in post-cardiac arrest. Resuscitation. 2013 May;84(5):651–656. doi: 10.1016/j.resuscitation.2012.10.024. [DOI] [PMC free article] [PubMed] [Google Scholar]

[R46] 46.Burstein DS, Jacobs JP, Li JS, et al. Care models and associated outcomes in congenital heart surgery. Pediatrics. 2011 Jun;127(6):e1482–e1489. doi: 10.1542/peds.2010-2796. [DOI] [PMC free article] [PubMed] [Google Scholar]

[R47] 47.Pollack MM, Cuerdon TT, Patel KM, Ruttimann UE, Getson PR, Levetown M. Impact of quality-of-care factors on pediatric intensive care unit mortality. Jama. 1994 Sep 28;272(12):941–946. [PubMed] [Google Scholar]

[R48] 48.Pollack MM, Alexander SR, Clarke N, Ruttimann UE, Tesselaar HM, Bachulis AC. Improved outcomes from tertiary center pediatric intensive care: a statewide comparison of tertiary and nontertiary care facilities. Critical Care Medicine. 1991 Feb;19(2):150–159. doi: 10.1097/00003246-199102000-00007. [DOI] [PubMed] [Google Scholar]

[R49] 49.Checkley W, Martin GS, Brown SM, et al. Structure, process, and annual ICU mortality across 69 centers: United States Critical Illness and Injury Trials Group Critical Illness Outcomes Study. Critical Care Medicine. 2014 Feb;42(2):344–356. doi: 10.1097/CCM.0b013e3182a275d7. [DOI] [PMC free article] [PubMed] [Google Scholar]

[R50] 50.Shortell SM, Zimmerman JE, Rousseau DM, et al. The performance of intensive care units: does good management make a difference? Med Care. 1994 May;32(5):508–525. doi: 10.1097/00005650-199405000-00009. [DOI] [PubMed] [Google Scholar]

[R51] 51.Kerlin MP, Small DS, Cooney E, et al. A randomized trial of nighttime physician staffing in an intensive care unit. N Engl J Med. 2013 Jun 6;368(23):2201–2209. doi: 10.1056/NEJMoa1302854. [DOI] [PMC free article] [PubMed] [Google Scholar]

[R52] 52.Kramer AA, Higgins TL, Zimmerman JE. Comparison of the Mortality Probability Admission Model III, National Quality Forum, and Acute Physiology and Chronic Health Evaluation IV hospital mortality models: implications for national benchmarking*. Critical Care Medicine. 2014 Mar;42(3):544–553. doi: 10.1097/CCM.0b013e3182a66a49. [DOI] [PubMed] [Google Scholar]

[R53] 53.Zupancic JAF, Richardson DK, Horbar JD, et al. Revalidation of the Score for Neonatal Acute Physiology in the Vermont Oxford Network. Pediatrics. 2007 Jan;119(1):e156–e163. doi: 10.1542/peds.2005-2957. [DOI] [PubMed] [Google Scholar]

[R54] 54.Borzecki AM, Christiansen CL, Chew P, Loveland S, Rosen AK. Comparison of in-hospital versus 30-day mortality assessments for selected medical conditions. Med Care. 2010 Dec;48(12):1117–1121. doi: 10.1097/MLR.0b013e3181ef9d53. [DOI] [PubMed] [Google Scholar]

[R55] 55.Black N. Assessing the quality of hospitals. Bmj. 2010;340:c2066. doi: 10.1136/bmj.c2066. [DOI] [PubMed] [Google Scholar]

[R56] 56.Shahian DM, Iezzoni LI, Meyer GS, Kirle L, Normand S-LT. Hospital-wide mortality as a quality metric: conceptual and methodological challenges. Am J Med Qual. 2012 Mar-Apr;27(2):112–123. doi: 10.1177/1062860611412358. [DOI] [PubMed] [Google Scholar]

[R57] 57.Knoester H, Bronner MB, Bos AP. Surviving pediatric intensive care: physical outcome after 3 months. Intensive Care Medicine. 2008 Jun;34(6):1076–1082. doi: 10.1007/s00134-008-1061-4. [DOI] [PMC free article] [PubMed] [Google Scholar]

[R58] 58.Graf J, Koch M, Dujardin R, Kersten A, Janssens U. Health-related quality of life before:1 month after, and 9 months after intensive care in medical cardiovascular and pulmonary patients. Critical Care Medicine. 2003 Aug;31(8):2163–2169. doi: 10.1097/01.CCM.0000079607.87009.3A. [DOI] [PubMed] [Google Scholar]

PERMALINK

Simultaneous Prediction of New Morbidity, Mortality, and Survival without New Morbidity from Pediatric Intensive Care: A New Paradigm for Outcomes Assessment

Murray M Pollack, MD

Richard Holubkov, PhD

Tomohiko Funai, MS

John T Berger, MD

Amy E Clark, MS

Kathleen Meert, MD

Robert A Berg, MD

Joseph Carcillo, MD

David L Wessel, MD

Frank Moler, MD

Heidi Dalton, MD

Christopher J L Newth, MD, FRCPC

Thomas Shanley, MD

Rick E Harrison, MD

Allan Doctor, MD

Tammara L Jenkins, MSN, RN

Robert Tamburro, MD, MSc

J Michael Dean, MD

Abstract

Objective

Design

Setting and Patients

Measurements and Main Results

Conclusions

INTRODUCTION

METHODOLOGY

Table 1.

Outcomes

Measurement of Physiological Status

Congenital Cardiac Conditions

Statistical Methods

Model Building

Final Model Evaluation

RESULTS

Table 2.

Figure 1.

Table4.

Table 5.

Figure 2.

Figure 4.

DISCUSSION

Supplementary Material

Individuals Acknowledged and Roles

Footnotes

Contributor Information

REFERENCES

Associated Data

Supplementary Materials

ACTIONS

PERMALINK

RESOURCES

Similar articles

Cited by other articles

Links to NCBI Databases