Severity of Illness Measures for Pediatric Inpatients

Abstract

Background

Severity of illness (SOI) measures are commonly used in adults for comparison of treatment and outcomes in similar populations. Less is known about the psychometric properties of measures available to providers and health care systems caring for pediatric patients.

Purpose

To (a) identify SOI measures used for pediatric patients admitted to acute care hospitals, and (b) compare the ability of two SOI measures to predict mortality and length of stay (LOS).

Methods

Twelve instruments were identified through literature search and one, the Pediatric Chronic Complex Condition (CCC), was retained. The CCC and the Charlson/Deyo comorbidity score (CCI) were applied to an 8-year retrospective, multi-institutional dataset using logistic and zero-truncated negative binomial regression models.

Results

Records from 199,001 children were examined. The CCC performed better for predicting mortality (OR=3.36; 95% CI: 3.20–3.53) and LOS (IRR=2.24; 95% CI: 2.22–2.26).

Conclusions

The CCC may be preferable for predicting outcomes among pediatric inpatients.

Implications

Pediatric SOI measures are not extensively developed and tested nor widely and freely available. Use of the CCC can predict mortality and LOS to guide care, resource allocation and research for the pediatric population.

Introduction

Background

Rapid deterioration and improvements in physiologic condition are a hallmark of the pediatric population, with different stressor responses and fewer physiologic reserve pathways than those found in adult patients.^1–4 Understanding the complex interplay of care processes and patient physiologic variables is imperative for optimal outcomes. Disentangling modifiable features of both provider performance and pediatric variables from those that are not modifiable may support efficient and effective allocation of resources.^2–5 One such indicator is severity of illness (SOI), the “physiologic complexity comprising the extent and interactions of a patient’s diseases presented to healthcare personnel.”² SOI measures are patient-centric, typically physiologic indicators that can guide resource allocation, not a measure of therapeutic interventions or treatments rendered.

Problem

While SOI measures are commonly used in adults to allow quantification of patient variables, and in turn, allow comparison of treatment and outcomes in similar populations, less is known about the measures available to providers and health care systems caring for pediatric patients.² The development and testing of SOI tools is burgeoning in recent years; since the year 2000 over 400 articles using SOI measures have been published, 25% of those since 2013. This reflects, in part, the increased focus on reducing unnecessary hospital admissions, length of stay (LOS) and costs associated with admissions, and improving the safety and efficiency of outpatient and in-hospital care. Substantial heterogeneity in these measures exists, however, and this confounds the researcher and clinician to select the most appropriate tool.^5–7

Purpose

Therefore, the aims of this project were to (a) identify SOI measures used for pediatric patients admitted to acute care hospitals, and (b) compare the ability of two severity of illness measures to predict mortality and LOS outcomes among pediatric inpatients using a retrospective, multi-institutional dataset. The purpose of this manuscript is to inform healthcare quality professionals of measures available to quantify SOI in pediatrics, study findings related to mortality and LOS, and implications for practice.

Methods

Design

This is a literature review followed by a retrospective analysis of existing data from patients. This study addresses the research questions: a) what SOI measures are available for use for pediatric patients admitted to acute care hospitals, and (b) can severity of illness measures predict mortality and LOS outcomes among pediatric inpatients using a retrospective, multi-institutional dataset?

Ethical Approvals

This study was approved by our organizations Institutional Review Board (AAK4050).

Sample and Setting

This is a retrospective analysis of data from patients <18 years of age admitted to three hospitals in the New York City metropolitan area between 2006 and 2014.

Procedures

Following institutional review board approval electronically stored data were extracted from a clinical data warehouse for all hospitalized in-patients; the development of this database has been described elsewhere.⁸ The database includes comprehensive information from a variety of sources: (1) International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) diagnosis and procedure codes; (2) clinical records including medication and device use; (3) laboratory reports, and (4) administrative records including demographics of age, sex, admission and discharge dates, and previous hospitalizations within the system. LOS was defined as number of days from admission to discharge date and morbidity was defined as all cause, same stay.

Data Analysis

The association between indices and in-hospital mortality using separate logistic regression models for each measure was assessed. To evaluate performance of the SOI measures, C-statistics and associated 95% confidence intervals were conducted. The C-statistic, sometimes called the “concordance” statistic or C-index, is a measure of goodness of fit for binary outcomes in a logistic regression model, such as these data, and was calculated to assess the absolute fit of the models. It is comparable to the area under the Receiver Operating Characteristic (ROC) curve and ranges from 0.5 to 1. A C-statistic value of 0.5 means that the model is no better at predicting an outcome than random chance and a value of 1 means that the model perfectly predicts those group members who will experience an outcome and those who will not. Models are typically considered good when the C-statistic is higher than 0.7 and strong when it exceeds 0.8.²³

Zero-truncated negative binomial regression was used to assess the association between each SOI measure and hospital LOS. This method is used to model count data for which the value zero cannot occur and when there is evidence of over dispersion. Two separate zero-truncated negative binomial regression models were fitted: one for LOS and one for mortality.

We also sought to provide an absolute measure of model fit for count data. Several Pseudo R² have been developed for that purpose.²⁴ We calculated the adjusted McFadden’s pseudo R² (ρ²), which is conceptually similar to the traditional R² measure in Ordinary Least Square (OLS) regression. Pseudo R² is appropriate for count data and can be interpreted like R², but its value is considerably lower than that of the R² index.²⁴

Odds ratios (OR) and associated 95% confidence intervals from logistic regression model and incidence rate ratio (IRR) and associated 95% confidence intervals from zero-truncated negative binomial regression were reported to assess the strength and direction of the effects of each SOI tool on both outcome measures. The OR is the exponential of the regression coefficient from logistic regression models. In the zero-truncated negative binomial regression model, the outcome measure LOS is treated as a count data, and thus the exponential of the regression coefficient of the log-linear model is called the IRR. All statistical analyses were performed using SAS version 9.3.^24,25

Results

Demographics

Our dataset included records from 199,001 children, of whom 104,425 (52%) were male and 94, 576 (48%) were female. Mean age was 2.8 years (range 0–17 years). Average LOS was 6.35 days (range 2–395 days) and 1,134 (0.6%) died while hospitalized.

Findings

Selection of Comparator SOI Indices

To address the first research question “what SOI measures are available for use for pediatric patients admitted to acute care hospitals,” we searched of three databases (CINAHL, PubMed and Ebmase) and Google Scholar for English language, peer-reviewed journal articles published between 2006–2016 using the search terms: “pediatrics”, “inpatient” “hospital”, “acute care”, “measures”, “severity scores “, “severity of illness” and “severity index”. Titles and abstracts were scanned for identification of SOI measures. Measures were excluded if they are related to pediatric early warning system or “trigger tools”. Though there is some conceptual overlap, early warning system and trigger tools are designed primarily to screen for clinical deterioration and response and SOI measures are generally used for research and administrative purposes, not as a guide for clinical management.^5,7,9 Additionally, measures were excluded if they were disease- or condition-specific, (e.g., sickle cell, pediatric inflammatory bowel disease, multi-organ failure, trauma), related to behavioral or psychological health, or limited to neonates.

To select the most appropriate SOI measures associated with mortality and LOS in the pediatric population, the following characteristics of studies were summarized: the purpose of the tool, its dimensions and items, psychometric properties, and the study setting and sample. Twelve tools were identified and assessed; measure and study characteristics are shown in Table 1. ^2,5,10–16 SOI measures were included for further analysis if they were publically available, not proprietary, available free of charge, and included variables that would be readily available using electronically collected hospital data.

Table 1.

Summary of Severity of Illness (SOI) Measures and Articles

Tool	Author/Year	Instrument Purpose	Dimensions & Items	Reliability Validity	Tested Sample/Setting	Comments
Neonatal therapeutic intervention scoring system (NTISS)	Gray, J. E., et al. (1992).10	Measure therapeutic intensity as an indicator of SOI and resource utilization in neonates independent of birthweight	Measures 8 clinical subscales (respiratory, cardiovascular, drug therapy, monitoring, metabolic/nutrition, transfusion, procedural and vascular access) using 62 item, 4 point Likert scale based on weights of therapeutic intensity and complexity.	Internal consistency demonstrated α = 0.84. Criterion validity, predictive of LOS and total hospital charge. Scores correlated with illness severity and mortality risk estimates by physicians.	1643 Neonates 3 hospitals, Neonatal Intensive Care Units (NICUs)	Modification of the76 item Adult Therapeutic Intervention Scoring System (TISS). Suggested use in conjunction with physiology based SOI measure (SNAP). Exclude: limited to neonates
Score for Neonatal Acute Physiology, Version II. (SNAP-II)	Richardson, D. K., et al. (2001).11	SNAP-II as measure of newborn illness severity SNAPPE-II as measure of mortality risk applicable to all birthweights.	SNAPP-II: 6 weighted variables: mean blood pressure, lowest temperature, PO2/FIO2 ratio, serum pH, seizures and urine output. SNAPPE-II: Items above plus birthweight, Apgar score and small gestational score (9items total).	Demonstrated construct validity by Pearson correlation coefficients using SNAP-PE and S NAPE-II, predictive discrimination by ROC and model goodness-of –fit statistics. Primary outcome in-hospital mortality	SNAP-II 10,819 neonates; SNAPPE-II 14,610 neonates. SNAPII and SNAPPE-II 30 NICU’s in Canada and U.S.	Study purpose to recalibrate SNAP tool using current population data and reduce the number of items using empirically a weights of mortality risk (not physician estimates). Exclude: limited to neonates
Score for Neonatal Acute Physiology (SNAP)	Richardson, D. K., et al. (1993).12	Measure of neonatal severity of illness by physiologic derangements in each organ system in first 24 hours.	Physiologic deviations, 26 items on 5 point Likert scale.	Construct Validity by Pearson correlation coefficient using SNAP and neonatal mortality.	1643 neonates, 3 NICUs (U.S).	Correlates with other SOI indicators: nursing workload, therapeutic intensity, LOS and physician estimates of mortality risk. Exclude: limited to neonates
Pediatric Risk of Admission (PRISA)	Chamberlain, J. M., et al. (1998).13	Predict risk of admission for pediatrics in ED	Pediatric Risk of Admission (PRISA) score, included the following: 3 components of the medical history, 3 chronic disease factors, 9 physiologic variables, 2 therapies, and 4 interaction terms.	Construct validity by prediction of admission, GOF and ROC statistics demonstrate good discrimination. Not all items predicted risk of admission.	2,683 ED pediatric patients (80% development and 20% validation samples). 1 pediatric hospital (U.S.)	Accuracy better for sicker patients, suggested modifying score/deleting items. Excluded not: publically available, nonproprietary, free of cost, or included variables that would be not be readily available using electronically collected hospital data or current data set.
The Pediatric Risk of Admission Second generation (PRISA –II)	Chamberlain, J. M., et al. (2005).1	SOI measure to predict admission for pediatric ED patients, use for benchmarking, controlling for SOI when enrolling in clinical trials	score uses components of acute and chronic medical history, physiology, and ED therapies Final SOI model includes 10 historical and 29 physiologic variables associated with mandatory admission.	Calibration and discrimination very good by GOF and ROC, Construct validity by correlation of proportion of patients in predicted risk intervals and outcomes of admission, mandatory admission and ICU admission (different models).	11,664 ED pediatric patients, random selection. (75% development and 25% validation sample). 16 block-random selected ED (U.S).	PRISA score was previously developed in a single hospital, recalibrated and validated in 2, studies in from academic pediatric hospitals. This study aim was to develop and validate a score in a larger sample of diverse hospitals. Excluded not: publically available, nonproprietary, free of cost, or included variables that would be not be readily available using electronically collected hospital data or current data set.
The Pediatric Risk of Admission (PRISA)	Chamberlain, J. M., et al. 2004.14	SOI measure of descriptive, physiologic and diagnostic variables to indicate probability of admission for pediatric patients.	PRISA score consists of 18 variables, including 3 acute history variables, 3 chronic disease variables, 9 physiologic variables, and 3 therapies to model the risk of the primary outcome, hospital admission.	Construct validity with increase PRISA score and hospital admission, excellent calibration and discrimination by GOF and ROC statistics.	2000 pediatric ED patients, including those with minor injuries/illness (80% development and 20% validation samples). 5 pediatric hospitals.	Expands original PRISA testing to more than one ED and inclusion of all patients (not just sickest), this recalibration performs better than original scale. Excluded not: publically available, nonproprietary, free of cost, or included variables that would be not be readily available using electronically collected hospital data or current data set.
The Pediatric Risk of Mortality (PRISM) - IV	Pollack, M. M., et al. (2016).15	Physiologically based score used to quantify physiologic status, compute mortality and expected morbidity risk.	PRISM-IV includes the original PRISM 17 physiologic variables with the subcategories of neurologic and non-neurologic PRISM scores, age, admission source, cardio-pulmonary arrest within 24 hours before admission, cancer, and low-risk systems of primary dysfunction.	Model fit and validation by GOF and ROC statistics demonstrates excellent predictive performance.	10,078 newborn to 18 years of age inpatients	physiologically based severity of illness measure using 17 commonly measured physiologic variables and their ranges (5). The PRISM score is a quantification of physiologic status using predetermined physiologic variables and their ranges that use categorical variables to facilitate accurate estimation of mortality risk (5). PRISM is commonly used to control for severity of illness in studies and to assess quality of care through standardized mortality ratios (SMRs). Excluded not: publically available, nonproprietary, free of cost, or included variables that would be not be readily available using electronically collected hospital data or current data set.
The Pediatric Risk of Mortality III—Acute Physiology Score (PRISM III-APS):	Pollack, M. M., et al. (1997).16	Measures physiologic instability for pediatric patients more sensitive to small changes in status (includes first 24 hour abnormal values).	21 physiologic variables, with 59 ranges for these items, broad severity scale 0–356 where higher values indicate higher instability.	Construct validity with increase PRISM-III-APS scores and increase mortality by MLR analyses and very god/excellent discrimination by GOF and ROC statistics.	111,165 consecutive PICU admissions	More sensitive than PISM-III to small changes in physiologic status. Excluded not: publically available, nonproprietary, free of cost, or included variables that would be not be readily available using electronically collected hospital data or current data set.
Comprehensive Severity Index (CSI)	Horn, S. D., et al. (2002).2	Severity score using disease specific patient variables, includes patient historical factors, physiologic and laboratory results. Predict “common outcomes” LOS, cost, mortality.	“A comprehensive set of clinical or historical findings, i.e., physiologic signs and symptoms of a disease, laboratory values, radiologic findings, and physical findings, and not treatments, was assigned to age-appropriate pediatric disease criteria sets, with severity criteria given an integer value from 1 to 4. Unclear number of items.	Admission CSI score predicted mortality b, good discrimination by ROC and GOF). Construct validity by Maximum CSI and relationship with LOS and cost.	16,495 pediatric admissions randomly selected.	Critique of PRISM and mortality, model based measures not including all patient variation and factors as well as outdated treatment effects. on of adult CSI. CSI had better predictability than Pediatric Risk of Mortality. Excluded not: publically available, nonproprietary, free of cost, or included variables that would be not be readily available using electronically collected hospital data or current data set.
Inpatient triage, assessment and treatment (ITAT)	Olson, D., et al. (2013).5	Identify high risk children after hospital admission.	Four equally weighted variables, yielding a cumulative score between 0 and 8. Variables included oxygen saturation, temperature, and age-adjusted heart and respiratory rates.	Construct validity with higher scores correlate with death within 2 days.	Nested case control1615 children <15 years old.	Developed for resource constrained hospitals in developing countries, research needed in developing countries, Aim was to develop a simplified pediatric early warning system score to continually monitor and triage patients throughout hospitalization. Excluded not: publically available, nonproprietary, free of cost, or included variables that would be not be readily available using electronically collected hospital data or current data set.
Pediatric Complex Chronic Conditions (CCC)	Feudtner, C., et al. (2000).3	ICD-9 codes categorized into 9 categories to examine particular CCC categories, identify patients with multiple categories and study patterns of pediatric mortality and end-of0life care	Physiologic, malignancies, genetic and congenital anomalies that comprise 9nine CCC categories (cardiovascular, respiratory, neuromuscular, renal, gastrointestinal, hematologic or immunologic, metabolic, other congenital or genetic, and malignancy) and corresponding ICD-9 codes	Construct and criterion validity established by sensitivity testing, single underlying cause of death (87%) sensitive detecting a CCC.	Retrospective cohort (1980–1997) study in one state including 21,617 child deaths aged 0–19 from death certificate data.	CCC accounted for increasing proportion of deaths in this cohort, though many die in infancy the need for supportive services in this population is needed. RETAINED

Of the twelve measures identified, the Pediatric Chronic Complex Condition (CCC), used variables that are commonly available in most medical records and was available at no cost. Because of these characteristics, we considered the CCC to have the greatest potential for generalizability and broader availability and for that reason, we selected the CCC for further analysis. For data analysis, this measure was compared to a commonly used measure of mortality and SOI in adults the Charlson/Deyo comorbidity score (CCI). This tool was selected as it is commonly applied in children though not validated in pediatrics. Both the CCC and the CCI are comprised of the International Classification of Disease, 9th revision, Clinical Modification (ICD-9-CM) categories and codes as shown in Table 2.^3,17 The range of scores for the CCC is 0–9 and for the CCI is 0–33. Each tool has been widely used, extensively tested and demonstrated sufficient psychometric properties.^18–22,28 We then applied the selected measures to our data to examine model fit and predictive capabilities.

Table 2.

Severity of Illness Measures, Conditions and ICD-9 Codes

Condition	Pediatric Complex Chronic Conditions (CCC)		Charlson Comorbidity Index (CCMI)
Cardiovascular	Heart and great vessel malformations	745.0–747.4	Myocardial infarction	410.x, 412.x
	Cardiomyopathies	425.0–425.4, 429.1	Congestive heart failure	425.4–425.9, 428.x, 404.13, 404.91, 404.93, 404.01, 404.03, 404.11, 398.91, 402.01, 402.11, 402.91
	Conduction disorders	426.0–427.4	Peripheral vascular disease	093.0, 437.3, 440.x, 441.x, 443.1–443.9, 47.1, 557.1, 557.9, V43.4
	Dysrhythmias	427.6–427.9	Cerebrovascular disease	362.34, 430.x–438.x
Respiratory	Respiratory malformations	748.0–748.9	Chronic pulmonary disease	416.8, 416.9, 490.x–505.x, 506.4, 508.1, 508.8
	Chronic respiratory disease	770.7
	Cystic fibrosis	277.0
Neuromuscular	Brain and spinal cord malformations	740.0–742.9	Dementia	290.x, 294.1, 331.2
	Mental retardation	318.0–318.2
	CNS degeneration and disease	330.0–330.9, 334.0–334.2, 335.0–335.9
	Infantile cerebral palsy	343.0–343.9
	Muscular dystrophies and myopathies	359.0–359.3
Renal	Congenital anomalies	753.0–753.9	Renal disease	403.01, 403.11, 403.91, 404.02, 404.03, 404.12, 404.13, 404.92, 404.93, 582.x, 583.0–583.7, 585.x, 586.x, 588.0, V42.0, V45.1, V56.x
	Chronic renal failure	585.0
Gastrointestinal	Congenital anomalies	750.3, 751.1–751.3, 751.6–751.9	Moderate or severe liver disease	456.0–456.2, 572.2–572.8
	Chronic liver disease and cirrhosis	751.4, 571.9	Mild liver disease	070.22, 070.23, 070.32, 070.33, 070.44, 070.54, 070.6, 070.9, 570.x, 571.x, 573.3, 573.4, 573.8, 573.9, V42.7
	Inflammatory bowel disease	555.0–556.9	Peptic ulcer disease	531.x–534.x
Hematologic or immunologic	Sickle cell disease	282.5–282.6	Rheumatic disease	446.5, 710.0–710.4, 714.0–714.2, 714.8, 725.x
	Hereditary anemias	282.0–282.4
	Hereditary immunodeficiency	279.00–279.9, 288.1–288.2, 446.1
Metabolic	Acquired immunodeficiency	0420–0421	HIV/AIDS	042.x–044.x
	Amino acid metabolism	270.0–270.9	Diabetes without chronic complication	250.0–250.3, 250.8, 250.9
	Carbohydrate metabolism	271.0–271.9	Diabetes with chronic complication	250.4–250.7
	Lipid metabolism	272.0–272.9
	Storage disorders	277.3, 277.5
	Other metabolic disorders	275.0–275.3, 277.2, 277.4, 277.6, 277.8–277.9
Malignancy	Malignant neoplasms	140.0–208.9, 235.0–239.9	Any malignancy, including lymphoma and leukemia, except malignant neoplasm of skin	140.x–172.x, 174.x–195.8, 200.x–208.x, 238.6
			Metastatic solid tumor	196.x–199.x
Other congenital or genetic defect	Chromosomal anomalies	758.0–758.9	Hemiplegia or paraplegia	334.1, 342.x, 343.x, 344.0–344.6, 344.9
	Bone and joint anomalies	259.4, 737.3, 756.0–756.5
	Diaphragm and abdominal wall	553.3, 756.6–756.7
	Other congenital anomalies	759.7–759.9

Secondary Analyses

Findings addressing the second research question “can severity of illness measures predict mortality and LOS outcomes among pediatric inpatients using a retrospective, multi-institutional dataset?” are shown in Tables 3 and 4. Table 3 presents parameter estimates and model fit for the two binary logistic regression models assessing the relationship between each of the SOI measures and hospital mortality. Mortality was positively associated with CCC (OR=3.36; 95% CI: 3.20–3.53): with one unit increase in the CCC sore, the odds of mortality tripled. The mortality was also positively associated with CCI score (OR=1.48; 95% CI: 1.44–1.51): with one unit increase in the CCI score, the odds of mortality increased about 50%. The CCC outperformed the CCI score, with the higher C-statistics (0.80 vs. 0.67)). According to McFadden (1973), CCC is strong predictor of mortality and CCI is a moderate predictor of mortality.

TABLE 3.

Parameters of Model Performance in Predicting Pediatric Inpatient Mortality using Binary Logistic Regression Models 2006–2014 (n=199001)

	Parameter Estimates		Model fit
Model	Odds Ratio(OR)	95% Confidence Interval	C-statistics (95% CI)
Number of complex chronic conditions	3.36	3.20–3.53	0.803(0.790–0.816)
Charlson/Deyo comorbidity score	1.48	1.44–1.51	0.668(0.653–0.683)

TABLE 4.

Parameters of Model Performance in Predicting Pediatric Inpatient Length of Stay using Zero-Truncated Negative Binomial Regression Models 2006–2014 (n=199001)

	Parameter Estimate		Model fit
Instrument Model	Incidence Rate Ratio(IRR)	95% Confidence Interval	Pseudo R2(ρ2) (%)*
Pediatric complex chronic conditions (CCC)	2.24	2.22–2.26	4.6
Charlson/Deyo comorbidity (CCI)	1.26	1.25–1.27	0.5

^*calculated based on $R^2=1-\frac{ln\widehat{L}(M_{\mathit{Full}})}{ln\widehat{L}(M_{\mathit{Intercept}})}$, where L(M_Interpect) denotes the maximized likelihood value from the current fitted model, and L(MFull) denotes the corresponding value but for the null model - the model with only an intercept and no covariates.

Table 4 presents parameter estimates and model performance from zero-truncated negative binomial regression models assessing the relationship between hospital LOS and each SOI measure. LOS is positively associated with CCC (IRR=2.24; 95% CI: 2.22–2.26): with one unit increase in the CCC score, the LOS doubled. LOS is also positively associated with the CCI (IRR=1.26; 95% CI: 1.25–1.27): with one unit increase in the CCI score, the LOS increased about a quarter. The CCC was superior to the CCI score in predicting hospital LOS based on higher McFadden’s pseudo R² (5% vs. 0.5%). According to McFadden (1973)²⁹, pseudo R² values of 5% and 0.5% are about equivalent to 10% and 1% R² values in OLS linear regression. This indicates CCI is a weak predictor of pediatric inpatients’ LOS, which explained about 1% of variance of LOS in OLS regression, whereas CCC is a better predictor by explaining about 10% of variance of LOS in OLS linear regression.

Limitations

As a secondary analysis of existing data this study has inherent limitations. The quality of the data is subject to coding quality and capture by computer systems designed for clinical use, not research purposes. Despite this limitation, with routine data integrity checks and use of this dataset for multiple research purposes we are confident these data have acceptable levels of reliability and validity and mimic the type and quality of data available in other acute care institutions. Other institutions are likely to have similar deficits in ICD coding and these findings suggest they would still be able to utilize this measure to predict LOS and mortality. Finally, other measures not available in this retrospective dataset may also predict mortality and LOS in the pediatric population, but to test additional measures not captured electronically, prospective studies would be needed.

Discussion

Although we identified numerous severity measures for pediatric patients admitted to acute care hospitals, many were early warning or trigger tools used to rapidly identify and guide clinical care of deteriorating patients, or were condition specific. Twelve SOI tools were identified as conceptually congruent with our SOI definition. Each tool assessed has a clear purpose, is psychometrically sound, and a few, such as the PRISM and PRISA are extensively used in the literature.^13–15 However, to enhance the potential for external validity we selected for analysis and comparison two tools that used only data readily available electronically within most organizations and freely available and accessible (i.e., not proprietary nor available only with a fee).

To meet our second aim, comparing predictive capabilities of selected measures on mortality and LOS outcomes among pediatric inpatients using a retrospective, multi-institutional dataset, we selected another known measure, the CCI. The CCI is one of the most commonly used comorbidity measures that is based on physiologic measures and employs ICD coding, and such is readily available at most hospitals. To our knowledge this is the first such comparison using this measure in the pediatric population.

Applying both these measures to our data validated the use of the tools with external data, that is, we did not build the model by identifying the best variables to select. The CCC demonstrated better predictive ability of mortality and LOS than the CCI. According to McFadden (1973), pseudo R² as 5% and 0.5% are about 10% and 1% as R² in OLS linear regression.²³ The CCC, or number of complex chronic conditions, is a better predictor of LOS with R² equivalent to 10% in OLS linear regression compared to the CCI with R² equivalent to 1% in OLS linear regression. In examining the crosswalk of ICD-9 codes used in each measure (Table 2), while major condition categories are identical, sub-conditions differ in subtle but important ways. For example, the CCC includes codes in the respiratory and neuromuscular categories that will detect cystic fibrosis, respiratory malformations and neuromuscular conditions found in pediatric patients, such as muscular dystrophy, brain and spinal cord malformations, cerebral palsy; these are not included in the CCI and therefore these conditions were not detected by the CCI.

Conclusions

Pediatric SOI measures are not extensively developed and tested nor widely and freely available. This study identified SOI tools that are accessible and can be used with existing data for classification of pediatric patients. Two measures, the CCC and the CCI, demonstrated external validity when applied to our data in predicting LOS and mortality, but the CCC was a better predictor of LOS and mortality for pediatrics admitted to three hospitals over an eight-year period. These study findings provide healthcare quality professionals with additional evidence and tools to examine SOI in inpatient pediatric populations and guide the provision of targeted services to reduce pediatric mortality and LOS.

Implications

The validation of the CCC in this pediatric population is important for several reasons. In the past decade, there has been a steady increase in the number and complexity of children with complex, chronic medical conditions.^9,10 Some estimates are as high as 11 million in the U.S.^27,28 Increasingly, healthcare quality professionals and administrators need to have the tools and knowledge to advocate for this vulnerable population. Healthcare quality professionals can use the CCC to predict mortality and LOS, both important quality of care indicators, and in turn, direct care, education and resource allocation for the pediatric population in acute care settings.

SOI measures that are sensitive to pediatric characteristics are also needed in health services research to examine effects of demographics, processes of care, and organizational capabilities, such as pediatric hospital designation, on survival and related outcomes.⁶ For example, use of the CCC tool can allow for benchmarking to establish comparative standards for performance of provider groups within a healthcare system, or across multiple systems. The healthcare quality professional can use these data analytics to identify issues, areas of opportunity and track and trend performance over time to guide interventions. Findings from this study, and future research using these measures, can inform healthcare policy internally and externally. For example, policies can be developed to guide resource allocation, requisite therapeutic interventions, and guide administrators developing policy to support the provision of safe, effective models of care delivery. In summary, findings from this study have important practice, policy and research implications for healthcare quality professionals.

Biographies

Amanda J. Hessels, PhD, MPH, RN, CIC, CPHQ, FAPIC is an Associate Research Scientist at Columbia University School of Nursing, NY, NY with a joint appointment as a Nurse Scientist at Hackensack-Meridian Health, Neptune, NJ. She has extensive experience in Infection Prevention and Healthcare Quality administration. Dr. Hessels’ research endeavors center on advancing the science of infection prevention practices, particularly standard precautions.

Jianfang Liu, PhD is an Assistant Professor of Quantitative Research in Nursing at Columbia University Medical Center. She has more than eight years of professional experience as a statistician/data analyst providing statistical analyses, instruction, and consultation for research projects and grants. Dr. Liu has expertise in big data management and processing.

Bevin Cohen, MS, MPH, is the Program Director, Center for Interdisciplinary Research to Prevent Infections, at Columbia University School of Nursing, and a doctoral candidate in the Mailman School of Public Health, New York, NY. As program director, Ms. Cohen has supported the research activities and training of numerous pre and post-doctoral fellows. Her research interests center on antimicrobial resistance in acute care settings.

Jingjing Shang, PhD, RN is an Assistant Professor of Nursing at Columbia University, School of Nursing, NY, NY. Her research interests include oncology nursing, nursing workforce, quality of care, patient safety, symptom management, exercise adherence among cancer patients.

Elaine Larson, RN, PhD, FAAN is the Associate Dean for Research and Anna C. Maxwell Professor of Nursing Research, School of Nursing and Professor of Epidemiology, Mailman School of Public Health, Columbia University, New York, N.Y. Dr. Larson is the Principal Investigator for the Center for Interdisciplinary Research to Prevent Infections, at Columbia University School of Nursing, supporting 26 pre- and postdoctoral trainees funded by two NIH training grants to receive didactic and practical training on how to work collaboratively in interdisciplinary teams to address complex research topics. She has made exceptional contributions to the field of infection prevention research and is designated a ‘Living Legend’ by the American Academy of Nursing.