PICU Volume and Outcome: A Severity-Adjusted Analysis

Abstract

Objective

To determine the relationship between Pediatric Intensive Care Unit (PICU) volume and severity-adjusted mortality in a large, national dataset.

Design

Retrospective cohort study.

Setting

The VPS database (VPS, LLC), a national multicenter clinical PICU database.

Patients

All patients with discharge dates between September 2009 to March 2012 and valid PIM2 and PRISM III scores, who were not transferred to another ICU and were seen in an ICU that collected at least three quarters of data.

Interventions

none

Measurements and Main Results

Anonymized data received included ICU mortality, hospital and patient demographics, PIM2 and PRISM III scores. PICU volume/quarter was determined (VPS sites submit data quarterly) per PICU and was divided by 100 to assess the impact per 100 discharges per quarter (Volume). A mixed effects logistic regression model, accounting for repeated measures of patients within ICUs was performed to assess the association of volume on severity-adjusted mortality, adjusting for patient and unit characteristics. Multiplicative interactions between volume and severity of illness were also modeled.

We analyzed 186,643 patients from 92 PICUs, with an overall ICU mortality rate of 2.6%. Volume ranged from 0.24 to 8.89 per ICU per quarter; the mean Volume was 2.61. The mixed effects logistic regression model found a small, but non-linear relationship between volume and mortality that varied based on severity of illness. When severity of illness is low, there is no clear relationship between volume and mortality up to a PIM2 risk of mortality of 10%; for patients with a higher severity of illness, severity of illness adjusted mortality is directly proportional to a unit’s volume.

Conclusions

For patients with low severity of illness, ICU volume is not associated with mortality. As patient severity of illness rises, higher volume units have higher severity of illness-adjusted mortality. This may be related to differences in quality of care, issues with unmeasured confounding, or calibration of existing severity of illness scores.

Introduction

The relationship between institutional volume – the number of patients treated per unit time – and outcome, has been a source of great interest and study in numerous fields within healthcare. Many, but not all, studies have suggested that higher volumes translate to better outcomes, whether at the physician, procedure, disease, or hospital level. The primary mechanism invoked in explaining these relationships is simply that “practice makes perfect,” and the more patients with a certain condition that a physician or hospital treats, the better they will become at treating them. On an intuitive level, few of us would allow our gall bladders to be removed by a surgeon who only does this operation once or twice a year; if we had to choose one, we would go to one that does dozens of this procedure yearly.

The existing literature seems to support an association between higher volume and improved health outcomes, in the hospital (1) and adult intensive care unit (2), with various methods of risk adjustment. In a recent systematic review on this subject in adult ICUs, the majority of studies reviewed supported such a relationship (3).

The existing literature on this issue in pediatric intensive care corroborates the findings in the other areas of healthcare, but the two existing studies considered a relatively small number of ICUs (4,5). We sought to use the largest US pediatric ICU specific database (VPS, LLC), to examine the relationship between PICU volume and mortality based on a patient level, severity-adjusted analysis, accounting for the nested effect of patients seen within the same hospital. Some of this work has been previously presented in abstract form (6,7).

Materials and Methods

The VPS database (VPS, LLC) is a national, multicenter, clinical database of PICU patients with robust quality controls, which is currently being used for quality improvement and benchmarking by over 100 PICUs in the USA. It was queried for all US patients discharged from September 2009 through March 2012 with valid Pediatric Index of Mortality 2 (PIM2) and Pediatric Risk of Mortality III (PRISM III) severity of illness scores. We chose PIM2 and PRISM III scores as these are the widely validated severity of illness scores for PICU patients, and a large number of patients in the VPS database have these scores. Anonymized data received included ICU mortality, age, gender, race/ethnicity, PIM2 and PRISM III scores, and characteristics of the hospital that the patient was discharged from (pediatric critical care fellowship program, 24/7 in-house attending physicians and if the unit admitted patients after open heart surgery). PICU volume/quarter was determined by the number of discharges per quarter (VPS sites submit data quarterly) per PICU. To be included, a PICU had to have submitted at least 3 quarters of data during the time period studied, to be consistent with a repeated measures model framework. Volume/quarter was then divided by 100 to assess the association of each additional 100 discharges per quarter. All references to volume henceforth refer to individual quarters of PICU volume/quarter/100 discharges (Volume). The volume of the individual PICU in the quarter the patient was discharged from that PICU was assigned as the volume variable for the individual patient. Patients who were transferred to another ICU were excluded from analysis, but their admission to the unit counted towards the unit’s volume for that quarter.

PIM2 based Standardized Mortality Ratios (SMRs) were calculated per unit per quarter, and were plotted against the volume of the unit per quarter for all patients, and for the subset of patients with higher PIM2 risk of mortality (≥ 4%, ~ 10% of the total patients). This was done to explore if there was an obvious univariate relationship between volume and severity of illness-adjusted mortality for all patients and the more severely ill patients.

To assess the relationship between volume and outcome at the patient level, we built a mixed effects logistic regression model, accounting for the individual ICU as the random effect, and other variables as the fixed effects. Variables considered for inclusion were: whether the unit included open heart surgery patients, had a fellowship program, had 24/7 in-house attending physician coverage, patient demographics and patient severity of illness. Variables were ultimately kept in the model if they retained an association with mortality or were deemed a significant confounder with the volume term (by changing the estimate of the coefficient by > 20%). We then looked to see if any of the variables in the model had a significant interaction with volume by building multiplicative terms. If the interaction term was significant, it was kept in the final model. Finally, to account for the fact that volume may not have a linear relationship with outcome, fractional polynomial analysis was explored, as well as the addition of a squared volume term was added to the model.

Model fit was evaluated by assessing (a) that the distribution of the random effects is normal; (b) the area under the curve of the Receiver Operating Characteristic Curve (as a test of discrimination); and (c) model calibration using a Hosmer-Lemeshow test extended for multilevel models (based on ten deciles of risk; additional details about this method are provided in the electronic supplement). Variance inflation factors (derived for multilevel models) were calculated to evaluate for potential colinearity of variables in the final model (8). The primary analyses are based on PIM2 as the metric for severity of illness, but an additional analysis using PRISM III is presented in the electronic supplement.

Analysis was performed in SPSS version 22 (IBM), R (9-11), and Stata version 10 (StataCorp LP). Statistical significance of results was evaluated at the 5% significance level.

Results

From September 2009 to March 2012, 194,400 patients across 107 PICUs were discharged. There were 15 units with 5,965 patients excluded because units provided less than three quarters of data. The remaining 188,435 patients were used to calculate volume per quarter per 100 patients. An additional 1,792 patients were excluded due to being transferred to another ICU, leaving 186,643 patients, across 92 units (with 721 quarters of collected data) for analysis.

Table 1 demonstrates the basic demographics and outcomes of the population studied, and are stratified based on quintiles of average PICU volume per quarter. The overall mortality rate was low (2.6%). Figures 1 and 2 graph PIM2 based Standardized Mortality Ratios (SMRs) per unit per quarter, against the volume of the unit per quarter for all patients (Figure 1; supplementary Figure E1 plots this relationship eliminating quarters with an SMR of 0), and patients with a PIM2 risk of mortality ≥ 4% (Figure 2). For all patients, as well as the more severely ill patients, there appears to be no obvious relationship between volume and mortality or SMR. The PIM2 SMR for the entire population is 1.04 with an AUCROC of 0.884 (95% CI 0.879 – 0.889). The PRISM III SMR for the entire population is 1.03 with an AUCROC of 0.915 (95% CI 0.910 – 0.919).

Table 1.

Demographics per quintile (stratified by volume bins)

Number of PICUs	19	18	18	18	19
Volume/quarter range	29 to 136	136 to187	187 to 264	264 to 389	389 to 876
Total n	12,167	25,408	30,437	49,434	69.197
Mortality (%)	2.7	2.6	2.6	2.6	2.6
Female (%)	44.7	43.9	44	44.1	44
24/7 attendings (%)	15.8	44.4	72.2	61.1	57.9
Fellowship (%)	21.1	33.3	27.8	55.6	84.2
Postoperative (%)	31.6	38.2	39.0	33.2	33.9
SMR PIM2	0.90	0.90	1.00	1.20	1.10
SMR PRISM III	1.00	0.90	1.10	1.10	1.00
African American (%)	13.3	8.7	8.2	17.9	16.5
Caucasian (%)	49.4	39.1	21.4	46.1	38.2
Hispanic (%)	13.4	10.8	11	11.3	16.1

Figure 1.

PICU SMR as a function of volume per quarter. Quarters with an SMR of ≥ 5 (n=4) were removed for better visualization.

Figure 2.

PICU SMR for higher risk (PIM2 ROM ≥ 4%) patients vs. volume per quarter. Quarters with an SMR of ≥ 5 (n=4) were removed for better visualization.

However, when determining the SMR for low vs. high risk patients (PIM2 ROM < or ≥ 4%), differences emerge. The actual mortality rate for patients with a PIM2 ROM of ≥ 4% was 12.7% (n=3453 of 27,293 patients). The average PIM2 ROM for this population was 13.8%. Therefore the SMR was 1.09. Of the 159,350 patients with a PIM2 ROM < 4%, 1369 died (0.9%). The average PIM2 ROM in this group was 0.5%; the SMR was therefore 1.8. This suggests a calibration variance in the PIM2 score between low and high risk patients.

Variables which had a univariate association with volume or mortality considered for inclusion in the multilevel model included: 24/7 in-house attendings, fellowship programs, open heart surgery, patient severity of illness (PIM2 and then PRISM III – in a separate model; see electronic supplement Table E1), race, and gender. PIM2 and PRISM III were not included in the same model out of concerns for colinearity. Variables retained in the final model using PIM2 (PRISM III included in Table E1): volume, a squared term for volume, open heart surgery, PIM2 score, gender, and race. While fractional polynomial analysis did not demonstrate that there was a univariate polynomial relationship between volume and outcome, this squared volume term was kept in the final model because it became significant when adjusting for severity of illness and other variables. We also found there was a multiplicative interaction between volume and severity of illness which was included in the multivariate model. Table 2 shows the coefficient estimates of the model’s fixed effects.

Table 2.

Model coefficients with PIM2 score as severity of illness metric

Variable	Coefficients	Standard Error	p-values
Volume	0.22	0.03	<0.0001
Volume2	-0.01	0.01	0.018
Open heart = no	−0.78	0.04	<0.0001
Open heart = yes	−0.80	0.07	<0.0001
PIM2 score	0.83	0.02	<0.0001
Male	−0.07	0.03	0.005
Caucasian	−0.03	0.03	0.281
Hispanic	−0.01	0.03	0.861
Other race	0.15	0.03	<0.0001
Unspecified race	−0.01	0.03	0.833
Volume × PIM2 score	0.04	0.01	<0.0001

Because of the multiplicative interaction between volume and severity of illness, a single odds ratio cannot be displayed summarizing the relationship between volume and outcome. However, this interaction is displayed in Figure 3, where it is clear that if PIM2-based severity of illness is low, there is no clear relationship between volume and mortality (all lines superimposed), up to a risk of mortality of approximately 10%; for patients with a higher severity of illness, mortality is proportional to a unit’s volume (lines diverge). The relationship between volume and outcome was similar when using PRISM III instead of PIM2, with a similar multiplicative interaction (Table E1).

Figure 3.

Interaction of PIM2 and PICU volume and its effects on patient mortality

Model diagnostics showed that the model was valid, with normally distributed random effects across unit IDs (Shapiro-Wilk p = 0.27); Variance Inflation Factors less than 2; 9 out of 92 hospitals have a Hosmer Lemeshow statistics < 0.05 (per the methodology developed and discussed in the electronic supplementary material); and an AUCROC of 0.89.

Discussion

Our study is the first report from a PICU clinical database of this size that does not demonstrate an inverse relationship between PICU volume and risk adjusted mortality. Our analysis demonstrates that the relationship between volume and outcome is a function of patient severity of illness, with similar severity of illness adjusted mortality among units of different volume when severity of illness is low. However, as severity of illness rises, patients admitted to higher volume units have higher severity of illness adjusted mortality than patients admitted to lower volume units. This holds for both PIM2 and PRISM III scores.

Most published reports examining the association of ICU volume and outcome have shown an inverse relationship. Kahn et al. studied 20,241 adult, mechanically ventilated patients at 37 institutions in the APACHE information database from 2002-2003 (12). They observed a 37 percent reduction in the adjusted mortality rate for patients admitted to hospitals in the highest quartile (> 400 patients ventilated per year) compared to those admitted to the lowest quartile (≤ 150 patients per year). In the systematic review by Khanere et al. of adult ICU volume-outcome studies, although all 13 included trials demonstrated an inverse relationship between volume and mortality, only 8 retained a significant effect for the entire cohorts after risk adjustment (2), and only 3 of 5 studies evaluating mechanically ventilated patients demonstrated significant risk-adjusted reductions in mortality with increasing volume. Similar results were seen when looking at the UK’s Intensive Care National Audit & Research Centre data with higher volume centers associated with improved outcome among 122,000 mechanically ventilated patients (13). Gaieski et al. found significantly improved outcomes for adults with sepsis comparing the highest volume to the lowest volume hospitals over a 7 year period in over 900,000 patients in the National Inpatient Sample (14). However, several large adult critical care studies have shown no relationship between risk-adjusted mortality and volume for non-postoperative mechanically ventilated patients (15), adults with sepsis (16), or all patients (17). In the most recent systematic review of adult ICU patients analyzed by patient type (e.g., cardiovascular, sepsis, respiratory, etc.), Nguyen et al. found 63% of 46 studies reported improved outcomes with higher volumes (3). Interestingly, studies that controlled for ICU or hospital factors were less likely to demonstrate this relationship.

Studying 16 PICUs in a single calendar year (1993), Tilford et al. found an adjusted odds ratio of 0.95 (per 100 patients/year; 95% CI 0.91 – 0.99) in a model including a number of variables, such as PRISM, age, and various high risk categories, such as oncologic disease or congenital anomalies (4). Pearson et al. compared the outcomes of children admitted to multiple smaller PICUs in Trent, UK, to a single large PICU in Victoria, Australia and found a dramatic increased risk of severity-adjusted mortality in Trent (OR 2.09, 95% CI 1.37 – 3.19)(18). Marcin et al. then reported an adjusted odds ratio (per 100 patients/year) of 0.68 (95% CI 0.52 – 0.89) in a regression model with 23 additional variables (5). Interestingly, when volume was analyzed as a quadratic term, they identified a range of volume (992 – 1,491/year) that was associated with optimal outcomes.

Unlike most of the other studies, our study specifically identified an interaction between severity of illness and volume that is associated with mortality. That is, severity of illness adjusted mortality is similar among units of different volume when patient severity of illness is low. However, as patient severity of illness rises, higher volume units have higher severity of illness adjusted mortality.

There are several potential explanations for this finding which may relate to (a) differences in quality of care, (b) issues with unmeasured confounding, or (c) calibration of severity of illness scores particularly with higher severity of illness. First, it may be that indeed lower volume units take better care of more acutely ill patients. The existing literature would not support this explanation, given that most other studies show the relationship is in the opposite direction. Of course, there is theoretical validity to this explanation, as a low volume unit may have more human resources to devote to acutely ill patients since they may have fewer other patients to manage. Indeed there is some concern that excessive numbers of patients per intensivist may be associated with worse outcomes (18). We did not have data on staffing ratios; as a result this concept would be a speculative explanation for our findings. In addition, Sasabuchi et al. describe better ICU outcomes with increased volume only in hospitals with relatively high ICU bed to total hospital bed ratios, again suggesting a possible “saturation” effect with too many critically ill patients if the relative number of ICU beds is too small for the hospital size (20). We did not evaluate ICU bed total to hospital bed ratios. We have eliminated patients from analysis who were transferred to another ICU; hence, the explanation is unlikely to be related to sicker patients being transferred out of smaller ICUs to larger ICUs.

Secondly, there certainly could be other variables associated with both volume and mortality (confounders) which are not routinely accounted for in our model or severity of illness scores. While both PIM2 and PRISM III have high and low risk diagnoses associated with them, it may be that, for example, the types of patients cared for in high volume units are fundamentally different than those taken care of in low volume units. For example, it is clear that diagnoses like stem cell transplant, or complex chronic conditions (22), are independently associated with mortality, even after controlling for severity of illness. Certainly these types of diagnoses are more likely to be present in higher volume units, and may be part of the explanation for the increased mortality. There are likely numerous other variables like this, which we were not able to account for in our model. Levy et al. examined the relationship in adult ICUs between patients managed by a critical care physician and those who were not; they attributed their finding that despite adjustment for severity of illness, patients managed by critical care physicians had a higher mortality rate due to unrecognized confounders (22).

Lastly, because the effect of volume and mortality is really only present for those with a high severity of illness, this finding may be an artifact of the calibration of severity of illness scores for higher risk patients. In general, both PIM2 and PRISM III are designed to function on the overall population in the ICU, and were meant to be parsimonious. That is, only a few variables are chosen, to help discriminate the sick from the less sick patient. Overall, both of these models perform well across all patients, with SMRs near 1 and AUCROCs between 0.88 (PIM2) and 0.92 (PRISM III). However, they may not perform as well at differentiating the moderately sick from the severely sick patient, for example. Indeed, we found the PIM2 SMR for low risk patients (PIM2 ROM < 4%) to be significantly different (1.8) than high risk (PIM2 ROM ≥ 4%) patients (1.09).

Interestingly, Straney et al. recently reported the development and validation of PIM3, an updated severity of illness tool from 60 PICUs in Australia, New Zealand and the UK (23). When comparing the SMR with this new tool across 4 different volume categories of PICUs (in admissions/year: 0-600, 601-1200, 1201-1800, 1801-2477), there appeared to be modest differences in predictive value, with SMR’s ranging from 0.93 to 1.08.

There are several limitations to our study, despite the relatively large sample size. First, although the VPS database is the largest PICU database available, it does not capture all PICUs in the United States, and there may be a selection bias towards more academic focused or resource rich ICUs who can pay to join VPS. Second, we did not consider all potential confounding variables on the relationship between volume and outcome, particularly those not captured in VPS, or more specifically in the data elements we obtained from VPS. We chose to be selective on the number of variables included to maximize sample size. Many variables which could have been considered as potential confounders are not required data elements in VPS, and would have led to “missing” data, dropping the sample size, or biasing the sample towards units that provide all data. Third, while PIM2 and PRISM III are accepted severity of illness scores, they were developed and first validated over a decade ago, and may not perform as well in today’s ICUs. We chose not to attempt to recalibrate these scores (through use of the raw data elements of the scores) and re-do our analysis (using the recalibrated scores) to retain generalizability in interpretation. Fourth, we kept patients in the dataset who were transferred from other ICUs. It is possible that the volume of the ICU in which they received their initial care may be relevant, and that is not accounted for in this model. We believe this point is unlikely to bias the results as this represents approximately 2% of the patients, and their PIM 2 SMR is 1.1, almost identical to the entire cohort (analysis not shown). Finally, there are few available model diagnostics for multilevel mixed effects models in the literature. As such, we proposed what we believe to be a natural extension of the Hosmer-Lemeshow test to evaluate model fit for multilevel mixed effects models with binary outcomes; our approach has not been previously validated in the literature.

Conclusions

In this very large dataset of PICU patients, we found that the relationship between volume and outcome is a function of patient severity of illness, with similar severity of illness adjusted mortality among units of different volume when severity of illness is low. However, as severity of illness rises, patients admitted to higher volume units have higher severity of illness adjusted mortality than patients admitted to lower volume units. This may be related to differences in quality of care, issues with unmeasured confounding, or calibration of existing severity of illness scores.

Supplementary Material

Supplemental Data File _.doc_ .tif_ pdf_ etc.__1

Supplementary Figure E1: Volume per quarter by unit

Supplemental Data File _.doc_ .tif_ pdf_ etc.__2

Supplementary Figure E2: PICU SMR as a function of volume per quarter with quarters of SMR=0 or ≥ 5 removed.