Abstract
Objective
Determine if the shortest sampling interval for laboratory variables used to estimate baseline severity of illness in pediatric critical care is equivalently sensitive across multiple sites without site-specific bias, while accounting for the vast majority of dysfunction compared to the standard 0 hour to 12 hour Pediatric Risk of Mortality (PRISM) III score.
Design
Prospective, random patient selection.
Setting
General/Medical and Cardiac/Cardiovascular pediatric intensive care units (PICUs) in 8 hospitals.
Patients
Patients < 18 years admitted to the PICU.
Interventions
None.
Measurements and Main Results
A total of 376 patients were included. Measurements for PRISM III laboratory variables (pH, PCO2, total CO2, PaO2, glucose, potassium, blood urea nitrogen (BUN), creatinine, total (WBC) count, platelet count, PT/PTT) were recorded from 2 hours prior to the PICU admission through 12 hours of the PICU care except for data in the operating room. Decreasing the observation period from the 0 hour to 12 hours post-PICU admission resulted in progressive decreases in the PRISM III laboratory variables measured. However, allowing the observation period to start 2 hours prior to PICU admission to 4 hours reduced this loss to only 3.4%. Similar trends existed for each of the individual laboratory PRISM III variables. There was a nearly identical distribution of laboratory PRISM III points within the −2 hour to 4 hour period compared to the standard period. We did not detect any institutional bias using the −2 hour to 4 hour time period compared to the baseline.
Conclusions
Prognostically important laboratory physiologic data collected within the interval from two hours prior to PICU to admission through four hours after admission account for the vast majority of dysfunction that these variables would contribute to PRISM III scores. There was no institutional bias associated with this sampling period.
Keywords: severity of illness, pediatrics, outcome prediction, critical care, pediatric critical care, intensive care, scoring systems, Pediatric Risk of Mortality Score, PRISM Score
INTRODUCTION
The assessment of severity of critical illness using physiological-based profiles is a balancing act of choosing a measurement period that is long enough to include all appropriate measurements, sufficiently short to minimize the effects of therapy on the variable values in order to represent “true” severity on ICU admission, and choosing a time period that does not impose institutional bias.1–5. First, the assessment period should be sufficiently long to enable the assessment of prognostically important physiological variables. For some variables such as heart rate or blood pressure, measurement is so frequent that short assessment periods will include these measurements. However, some laboratory variables with proven prognostic information are measured relatively infrequently and this can lengthen the measurement period necessary to capture these variables. At the same time, though, the assessment period should be as short as possible to minimize the effects of therapy on the initial estimation of illness severity, especially if the purposes of the severity method include assessing quality of care. Finally, the time period should not impose biased estimates because of institutional practice patterns of laboratory testing. For example, some PICUs routinely use measurements done prior to admission as their admission labs and other routinely repeat these measurements after PICU admission.
The Collaborative Pediatric Critical Care Research Network (CPCCRN) undertook a study of the appropriate sampling time period for physiological variables used to estimate baseline severity of illness in pediatric critical care. Our aim was to determine the shortest time period for collection of the laboratory variables that would be equivalently sensitive across all study sites without evidence of site-specific bias while accounting for the vast majority of dysfunction, compared to the 0 hour to 12 hour PRISM III score for laboratory variables. This study was preparatory to a current study evaluating a new paradigm of quality assessment of critical care based on using initial severity of illness to predict functional status (other than survival) at PICU discharge or later.
METHODOLOGY
Patient Population
The CPCCRN is composed of 7 sites and 8 PICUs, and admits approximately 17,000 patients per year.6 Patients from newborn to less than 18 years were selected according to a randomization scheme developed at the data coordinating center, for a total of approximately 50 subjects per PICU during a one month period. For enrollment days when more than the protocol-allocated number of patients was eligible at a center, the required number was selected using this pre-specified random selection method. Both General/Medical PICUs and Cardiac/Cardiovascular PICUs were included. There were no separate surgical or neurological PICUs. The protocol was approved by the Institutional Review Boards at all participating institutions.
Selected Physiologic Variables
Physiological variables included in this assessment were the laboratory components of PRISM III because each had been previously associated with mortality in univariate and multivariate analyses.7 These included pH, PCO2, total CO2, PaO2, glucose, potassium, blood urea nitrogen (BUN), creatinine, total (WBC) count, platelet count, and PT/PTT. The measurement time was assessed as the time stamp associated with the measurement. All measurements were recorded from 2 hours prior to the PICU admission through 12 hours of the PICU care except for data in the operating room. That is, pre-PICU laboratory data were included from the emergency department, other care areas of the hospital, the post-anesthesia care unit, and/or from outside care facilities if available up to 2 hours prior to the PICU admission.
While the data effort focused primarily on the laboratory variables, the most abnormal non-laboratory variables in PRISM III, systolic blood pressure, heart rate, mental status/GCS, and temperature, were also collected with the time of occurrence from 0 hour to 12 hours.
Data Analyses
The analysis focused on determining the shortest measurement interval for laboratory-based variables that would be sufficiently sensitive (overall as well as across all study sites without evidence of site-specific bias), measuring a comparable amount of dysfunction compared to the 0 hour – 12 hour PRISM III score for laboratory variables (“gold standard”).7 We examined a variety of intervals beginning 2 hours prior to admission (−2 hours), 1 hour prior to admission (−1 hour), or at the time of admission (0 hour), and extending to post-admission times of 2 hours, 4 hours, 6 hours, 8 hours, and 12 hours. We evaluated the number of patients with any measurement available during the time interval, the number of patients with specific variables measured during the interval, and the PRISM III points generated from the laboratory variables (laboratory PRISM III) in the various candidate intervals. These were compared to the period of 0 hours to 12 hours, which is the standard period for data collection for PRISM III scoring.
Finally we compared the availability of each of the laboratory variables during the intervals of interest at each institution. This comparison allowed us to assess if there was institutional bias due to including or excluding a disproportionate number of laboratory values across institutions. This analysis used chi-squared and Fisher’s exact testing to compare proportions of patients at each institution for whom laboratory value availability status changed when the interval was modified. This analysis (which is not reported in detail in the text) found no significant institutional bias after accounting for the multiple (12 laboratory value) comparisons that were performed.
Statistical analysis used the chi-square test for comparison of categorical variable distributions between centers, and the Kruskal-Wallis test for comparing distributions of continuous variables between centers.
Results
A total of 376 patients were included from the 8 PICUs. Table 1 displays the patient characteristics in the participating sites. The patients came from both medical PICUs (73.1%) and cardiovascular PICUs (26.9%). This differed significantly across the sites with as few as 4.3% and as many as 44.9% of institutional samples from cardiovascular units (p < .001). Median age was 4.8 years and while it did not differ across the sites, the distribution of ages did vary. Median length of stay was 4.4 days and differed between the institutions (p<.05). Overall, 44.7% of patients were postoperative and this varied from 28.0% to 59.2% (p<.01). The most common organ system of dysfunction based on the admitting diagnosis was the cardiovascular system followed by the neurological and the respiratory systems. Table 2 shows the number of individuals with a measurement of any PRISM III laboratory variable and the number of individuals with a measurement of each of the PRISM III variables in representative candidate intervals. Overall, 82 patients or 22% did not have any PRISM III laboratory measurement in the 0 hour to 12 hour (standard) period. A total of 6.5% more patients had laboratory variables measured in the −2 hour to 12 hour period than the standard period.
Table 1.
Patient Characteristics
| Factor | Overall | Site Ranges |
|---|---|---|
| Median Age (years) a | 4.8 | 1.65–8.3 |
| 0–1 month (%) | 4.8% | 0%–8.2% |
| >1–3 months (%) | 4.5% | 0%–8.3% |
| >3–6 months (%) | 6.1% | 2.0%–16.3% |
| >6–12 months (%) | 8.5% | 6.0%–16.7% |
| >12 months – 6 years (%) | 30.9% | 19.4%–46.7% |
| >6 years–12 years (%) | 17.3% | 10.0%–36.7% |
| >12 years (%) | 27.9% | 10.2%–36.2% |
| Diagnoses b | ||
| Respiratory | 20% | 8%–40% |
| Neurological | 28% | 24%–26% |
| Cardiovascular (n, (%)) | 35% | 24%–57% |
| Miscellaneous | 18% | 6%–30% |
| Operative Status c | ||
| non-operative (n, (%)) | 208 (55.3%) | 40.8%–72.0% |
| post-operative (n, (%)) | 168 (44.7%) | 28.0%–59.2% |
| Cardiac Surgery d | ||
| non-operative (n, (%)) | 301 (80.1%) | 57.1%–93.3% |
| cardiac surgery (n, (%)) | 75 (19.9%) | 6.7%–42.9% |
| Type of ICU d | ||
| medical (n, %) | 275 (73.1%) | 55.1%–95.7% |
| cardiovascular (n, (%)) | 101 (26.9%) | 4.3%–44.9% |
| Median Length of Stay (days) e | 4.4 | 3.2–5.2 |
P>.50; Kruskal Wallis test comparing distributions across sites
P<.01, chi-square test
P<.01, chi-square test
P<.001, chi-square test
P<.05, Kruskal Wallis test comparing distributions across sites
Table 2.
Measurement of PRISM III Laboratory Variables. The number of patients with a measurement of any PRISM III laboratory variable and the number with a measurement of each of the PRISM III variables are shown as a percent of the baseline period (0 hour to 12 hours) for selected candidate intervals. The absolute number of patients with measurements is shown in the baseline period.
| Time Interval (hours) | Any PRISM III Laboratory Measurement | WBC | Acidosis | pH | PCO2 | Total CO2 | PaO2 | Platelets | Glucose | Potassium | PT/PTT | BUN | Creatinine |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| −2 to 12 | 106.5 | 117.6 | 107.4 | 111.3 | 111.3 | 110.9 | 109.1 | 117.6 | 110.9 | 110.2 | 116.1 | 111 | 110.8 |
| 0 to 12 | |||||||||||||
| (Baseline) | 100 (n = 294) | 100 (n = 222) | 100 (n = 282) | 100 (n = 194) | 100 (n = 194) | 100 (n = 247) | 100 (132) | 100 (n = 221) | 100 (n = 247) | 100 (n = 254) | 100 (n = 137) | 100 (n = 246) | 100 (n = 249) |
| 0 to 8 | 91.8 | 86.0 | 90.1 | 97.4 | 97.4 | 83.4 | 96.2 | 86.0 | 84.6 | 83.9 | 91.2 | 82.9 | 83.1 |
| 0 to 6 | 88.4 | 81.1 | 86.5 | 94.8 | 94.8 | 77.3 | 90.9 | 81.0 | 77.3 | 77.2 | 87.6 | 76 | 76.3 |
| 0 to 4 | 84.4 | 76.6 | 83.0 | 91.2 | 91.2 | 71.3 | 84.1 | 76.5 | 68.8 | 71.3 | 77.4 | 69.9 | 70.3 |
| 0 to 2 | 72.4 | 63.5 | 70.6 | 78.4 | 78.4 | 56.7 | 70.5 | 63.3 | 56.7 | 57.5 | 68.6 | 56.5 | 56.6 |
| −1 to 4 | 89.5 | 89.2 | 86.9 | 95.4 | 95.4 | 81.0 | 90.2 | 89.1 | 77.3 | 79.9 | 89.1 | 78.9 | 79.1 |
| −2 to 4 | 96.6 | 101.8 | 95.0 | 105.7 | 105.7 | 89.9 | 95.5 | 101.8 | 86.2 | 89.4 | 98.5 | 87.4 | 87.6 |
Decreasing the observation period from the standard period to 0 hour to 8 hours, 0 hour to 6 hours, 0 hour to 4 hours, and 0 hour to 2 hours resulted in substantial decreases in the number of variables measured (Table 2). For example, reducing the time period to 0 hour to 8 hours resulted in a reduction of 8.2% of patients having any PRISM III laboratory variable measured. Reducing the laboratory observation period by 50% to 6 hours from 0 hour to 6 hours resulted in an 11.6% decrease. However, allowing the observation period to start 2 hours prior to PICU admission to 4 hours reduced this loss to 3.4%. Therefore, further analysis focused on the suitability of the −2 to 4 hour period compared to the standard 0 hour to 12 hour period.
Similar trends existed for each of the individual laboratory PRISM III variables as the overall measurement prevalence (Table 2). In 10 of the 13 laboratory variables, the time period most closely reflecting the standard was the −2 hour to 4 hour period. For the other 3 laboratory variables (pH, PCO2, PO2), the −2 hour to 4 hour period actually captured more measurements than the standard period while 0 hour to 8 hour and 0 hour to 6 hour candidate time intervals were more comparable to the standard period. WBCs and platelets were also measured in more individuals in the −2 hour to 4 hour period compared to the standard period.
Figure 1 shows the very similar distributions of laboratory PRISM III points within the −2 hour to 4 hour period compared to the standard period. Some patients lost and some patients gained PRISM III laboratory points (Figure 2) by using the −2 hour to 4 hour interval compared to the standard interval, but the mean laboratory PRISM III score only changed from 3.8 in the standard period to 3.6 in the −2 hour to 4 hour interval. Seventy percent of patients did not change their PRISM Score at all; 13% of patients lost points and 18% of patients gained PRISM III points.
Figure 1.
Distribution of PRISM III Laboratory Points in the −2 Hour to 4 Hour and the 0 Hour to 12 Hour (Baseline) Intervals.
Figure 2.
Distribution of the Change in PRISM III Laboratory Points When the Observation Interval is Changed from the 0 Hour to 12 Hour (Baseline) to the −2 Hour to 4 Hour Interval.
The most abnormal non-laboratory PRISM III values in the first 12 hours were also assessed. The maximum dysfunction for systolic blood pressure, heart rate, temperature, and GCS/mental status occurred in 80%, 92%, 98%, and 96% of the patients, respectively, within the first 4 hours of PICU admission.
We did not detect any institutional bias using the −2 hour to 4 hour time period compared to the baseline. Table 3 shows the change in the PRISM III laboratory score from the standard period to the −2 to 4 hour interval, for all sites and the individual sites. There was no significant institutional bias present (p=0.42 for Kruskal-Wallis test comparing distributions of this change between centers). For some sites, variable measurement and PRISM III scores slightly increased and for other, they slightly decreased.
Table 3.
PRISM III Laboratory Score Change From the Standard Period to the −2 Hour to 4 Hour Time Period by Site.
| Site | Score increased by > 4 points | Score increased between 1 and 4 points | Score remained the same | Score decreased between 1 and 4 points | Score decreased by > 4 points |
|---|---|---|---|---|---|
| 1 | 2% | 12% | 73% | 10% | 2% |
| 2 | 2% | 11% | 70% | 13% | 4% |
| 3 | 0% | 12% | 76% | 8% | 4% |
| 4 | 0% | 12% | 70% | 10% | 8% |
| 5 | 4% | 4% | 65% | 22% | 4% |
| 6 | 2% | 4% | 78% | 14% | 2% |
| 7 | 6% | 11% | 56% | 28% | 0% |
| 9 | 2% | 20% | 64% | 7% | 7% |
| Total | 2% | 11% | 70% | 14% | 4% |
DISCUSSION
We have demonstrated that prognostically important laboratory and non-laboratory physiologic data collected within the interval from two hours prior to PICU to admission through four hours after admission account for almost all of the dysfunction that these variables would contribute to PRISM III scores compared to the standard 0 hour to 12 hour time period. Data collection within this interval was not associated with significant inter-institutional bias, and this shortened interval (compared to the standard period of time of admission through 12 hours after admission) will minimize the effects of therapy on the initial estimation of disease severity. However, the PRISM III score using this revised time interval should not be used to represent risk of mortality or morbidity until formal validation studies are conducted.
The CPCCRN undertook this study in preparation for the Trichotomous Outcome Prediction in Critical Care study (TOPICC), prospectively enrolling 10,000 patients admitted to CPCCRN PICUs, and assessing the functional status at the time of PICU discharge with the goal of deriving and validating a statistical prediction model relating initial status to ultimate outcome. PRISM III and the Pediatric Index of Mortality (PIM) have been used to assess and compare the quality of care received in PICUs by comparing observed and predicted mortality.7–10 However, the significant decrease in PICU mortality during the past several decades and the greater emphasis on preventing morbidity has reduced the relevance of mortality based quality assessments. Therefore, quality of care assessments and comparisons should consider morbidities such as functional status impairment as well as mortalities.
The first step in accomplishing the goals of the TOPICC study was to develop a parsimonious instrument for assessing functional outcome of children at the time of PICU discharge. The CPCCRN developed the Functional Status Score for this purpose.11 The second step was the subject of the current report: to reassess the appropriate time interval for the measurement of severity of illness variables. Practice patterns have changed since the development of the PRISM III score in 1996. Initially, the 12 hour period for PRISM III was chosen because that time period was required to capture 90% of the variables that would be measured in the first 24 hours.12 This was done to assure that all 16 centers participating in the initial validation of PRISM III would have at least one measurement for variables, minimizing the potential for institutional bias. As part of this national study, PRISM III will be recalibrated to mortality as well as morbidity, which will enable adjustment for changes induced by the new sampling period from −2 hours to 4 hours post PICU admission.
This study found that the time period of 2 hours prior to PICU admission to 4 hours after ICU admission provides a time interval resulting in sampling frequencies very similar to the original 12 hour time period used in the development of PRISM III, and did not bias the laboratory PRISM III variables of any of the PICUs participating in the study.12 This time period accounts for the common practices of variable measurement, including not routinely repeating laboratory data on admission. It reduces the potential for lead time bias by reducing the PICU observation time from 12 hours to 4 hours, which is expected to more effectively separate the effects of therapy on physiological functioning. Therefore, the TOPICC study investigating the relationship of severity of illness to the development of morbidity as well as mortality is using this shortened sampling interval.
Acknowledgments
This work was supported by a grant from the Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD), Bethesda, Md
The Authors wish to acknowledge the contributions of the following individuals:
Teresa Liu, University of Utah
Jean Reardon, Children’s National Medical Center
Aimee Labell, Phoenix Children’s Hospital
Jeffrey Terry, Children’s Hospital Los Angeles
Ann Pawluszka, Children’s Hospital of Michigan
Mary Ann DiLiberto, Children’s Hospital of Philadelphia
Moni Weber, University of Michigan
Lauren Conlin, University of Michigan
Alan Abraham, University of Pittsburgh Medical Center
Jeri Burr, University of Utah
Tammara Jenkins, National Institutes of Health (NIH)
Contributor Information
Murray M. Pollack, Department of Child Health, Phoenix Children’s Hospital and University of Arizona College of Medicine-Phoenix, Phoenix, AZ.
J. Michael Dean, Department of Pediatrics, University of Utah School of Medicine, Salt Lake City, UT.
Jerry Butler, Department of Pediatrics, University of Utah School of Medicine, Salt Lake City, UT.
Richard Holubkov, Department of Pediatrics, University of Utah School of Medicine, Salt Lake City, UT.
Allan Doctor, Departments of Pediatrics and Biochemistry, Washington University School of Medicine, St. Louis, MO.
Kathleen L. Meert, Department of Pediatrics, Children’s Hospital of Michigan, Detroit, MI.
Christopher J. L. Newth, Department of Anesthesiology and Critical Care Medicine, Children’s Hospital Los Angeles, Los Angeles, CA.
Robert A. Berg, Department of Pediatrics, Children’s Hospital of Philadelphia, Philadelphia, PA.
Frank Moler, Department of Pediatrics, University of Michigan, Ann Arbor, MI.
Heidi Dalton, Department of Child Health, Phoenix Children’s Hospital and University of Arizona College of Medicine-Phoenix, Phoenix, AZ.
David L. Wessel, Department of Pediatrics, Children’s National Medical Center, Washington DC.
John Berger, Department of Pediatrics, Children’s National Medical Center, Washington DC.
Rick E. Harrison, Department of Pediatrics, University of California at Los Angeles, Los Angeles, CA.
Joseph A. Carcillo, Department of Critical Care Medicine, Children’s Hospital of Pittsburgh, Pittsburgh, PA.
Thomas P. Shanley, Department of Pediatrics, University of Michigan, Ann Arbor, MI.
Carol E. Nicholson, The Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD), Bethesda, Md.
References
- 1.Nielsen MS, Woodcock TE, Nolan KM, Jonas MM. Lead time bias and standardised mortality ratios in intensive care patients. Anaesthesia. 1999 Apr;54(4):399. doi: 10.1046/j.1365-2044.1999.00869.x. [DOI] [PubMed] [Google Scholar]
- 2.Richardson D, Tarnow-Mordi WO, Lee SK. Risk adjustment for quality improvement. Pediatrics. 1999 Jan;103(1 Suppl E):255–265. [PubMed] [Google Scholar]
- 3.Tunnell RD, Millar BW, Smith GB. The effect of lead time bias on severity of illness scoring, mortality prediction and standardised mortality ratio in intensive care--a pilot study. Anaesthesia. 1998;53(11):1045–1053. doi: 10.1046/j.1365-2044.1998.00566.x. [DOI] [PubMed] [Google Scholar]
- 4.Vincent J-L, Moreno R. Clinical review: scoring systems in the critically ill. Crit Care. 2010;14(2):207. doi: 10.1186/cc8204. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Harrison DA, Rowan KM. Outcome prediction in critical care: the ICNARC model. Curr Opin Crit Care. 2008 Oct;14(5):506–512. doi: 10.1097/MCC.0b013e328310165a. [DOI] [PubMed] [Google Scholar]
- 6.Willson DF, Dean JM, Meert KL, et al. Collaborative pediatric critical care research network: looking back and moving forward. Pediatr Crit Care Med. 2010;11(1):1–6. doi: 10.1097/PCC.0b013e3181c01302. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Pollack MM, Patel KM, Ruttimann UE. PRISM III: an updated Pediatric Risk of Mortality score. Critical Care Medicine. 1996;24(5):743–752. doi: 10.1097/00003246-199605000-00004. [DOI] [PubMed] [Google Scholar]
- 8.Czaja AS, Scanlon MC, Kuhn EM, Jeffries HE. Performance of the Pediatric Index of Mortality 2 for pediatric cardiac surgery patients. Pediatr Crit Care Med. 2011;12(2):184–189. doi: 10.1097/PCC.0b013e3181e89694. [DOI] [PubMed] [Google Scholar]
- 9.Shann F, Pearson G, Slater A, Wilkinson K. Paediatric index of mortality (PIM): a mortality prediction model for children in intensive care. Intensive Care Medicine. 2000;23(2):201–207. doi: 10.1007/s001340050317. [DOI] [PubMed] [Google Scholar]
- 10.Slater A, Shann F, Pearson G Paediatric Index of Mortality Study G. PIM2: a revised version of the Paediatric Index of Mortality. Intensive Care Medicine. 2003;29(2):278–285. doi: 10.1007/s00134-002-1601-2. [DOI] [PubMed] [Google Scholar]
- 11.Pollack MM, Holubkov R, Glass P, et al. Functional Status Scale: new pediatric outcome measure. Pediatrics. 2009 Jul;124(1):e18–28. doi: 10.1542/peds.2008-1987. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Pollack MM, Patel KM, Ruttimann U, Cuerdon T. Frequency of variable measurement in 16 pediatric intensive care units: influence on accuracy and potential for bias in severity of illness assessment. Critical Care Medicine. 24(1):74–77. doi: 10.1097/00003246-199601000-00013. [DOI] [PubMed] [Google Scholar]