Variation in Adjusted Mortality for Medical Admissions to Pediatric Cardiac Intensive Care Units

Abstract

Objective:

Pediatric cardiac intensive care units (CICU) should be adept at treating both critical medical and surgical conditions for patients with cardiac disease. There are no case mix adjusted quality metrics specific to medical CICU admissions. We aimed to measure case mix adjusted CICU medical mortality rates and assess variation across CICUs in the Pediatric Cardiac Critical Care Consortium (PC4).

Design:

Observational analysis

Setting:

PC4 clinical registry

Patients:

All CICU admissions that did not include cardiac surgery.

Interventions:

None

Measurements and Main Results:

The primary endpoint was CICU mortality. Based on multivariable logistic regression accounting for clustering, we created a case mix adjusted model using variables present at CICU admission. Bootstrap resampling (1000 samples) was used for model validation. We calculated a standardized mortality ratio (SMR) for each CICU based on observed-to-expected mortality from the fitted model. A CICU was considered a statistically significant outlier if the 95% confidence interval around the SMR did not cross 1.

Of 11,042 consecutive medical admissions from 25 CICUs (8/2014–5/2017) the observed mortality rate was 4.3% (N=479). Final model covariates included age, underweight, prior surgery, time of and reason for CICU admission, high-risk medical diagnosis or comorbidity, mechanical ventilation or ECMO at admission, and pupillary reflex. The c-statistic for the validated model was 0.87 and it was well calibrated. Expected mortality ranged from 2.6–8.3%, reflecting important case mix variation. SMRs ranged from 0.5–1.7 across CICUs. Three CICUs were outliers; two had lower-than-expected (SMR <1) and one had higher-than-expected (SMR >1) mortality.

Conclusions:

We measured case mix adjusted mortality for CICU patients with critical medical conditions, and provide the first report of variation in this quality metric within this patient population across PC4 CICUs. This metric will be used by PC4 CICUs to assess and improve outcomes by identifying high-performing (low-mortality) centers through collaborative learning.

Introduction

Pediatric cardiac intensive care units (CICU) initially evolved from general pediatric ICUs into specialized critical care units to provide postoperative care for children undergoing increasingly complex cardiac surgical interventions. Those admissions that do not include cardiac surgery – medical admissions – result from a diverse set of indications for critical care: routine post-procedural care (e.g. catheterizations or non-cardiac surgery), evaluation of structural heart disease, and for treatment of acute medical conditions in surgical patients who develop complications while convalescing on the non-ICU ward, patients with decompensated heart failure, and patients with primary heart disease who present with general critical illness, among others. Caring for medical patients is inherently different from providing postoperative care, and CICU quality may differ across these admission types. Focused assessment of quality of care in medical admissions can offer important insights to hospitals looking to improve outcomes for children with critical cardiovascular disease.

However, CICU quality metrics and case mix adjustment methods specific to medical admissions have not been developed to date. Previous efforts to study CICU quality care have focused on postoperative admissions.(1) Analyses to derive case mix adjustment models have been applied to general pediatric ICU populations, in some cases inclusive of cardiovascular medical admissions, but none of these analyses have been adequately tested within pediatric CICUs.(2, 3) Existing evidence suggests that these previous models do not perform well for patients with critical cardiovascular disease,(4, 5) and some predictor variables (e.g. PaO₂) lack face validity for the heterogeneous physiologies seen in a CICU, necessitating new approaches tailored to this clinical setting. As such, it remains unknown whether the quality of care for medical patients varies across CICUs. This uncertainty hinders attempts to uncover key drivers of performance and initiate potential quality improvement initiatives.

In this context, we sought to measure CICU quality of care in medical admissions using adjusted mortality as the outcome metric, and to describe variation in adjusted mortality across the CICUs participating in the Pediatric Cardiac Critical Care Consortium (PC4). To do so we created a CICU-specific case mix adjustment model from the PC4 registry. We hypothesized 1) that important variation in the characteristics of medical admissions exists across CICUs, underscoring the need for a validated case mix adjusted model, and 2) that we could identify outlying CICUs with statistically significant higher-than and lower-than-expected mortality.

Methods

Data source

PC4 is a quality improvement collaborative that collects data on all patients with primary cardiac disease admitted to the CICU attending service at participating hospitals.(6) PC4 maintains a clinical registry to support research and quality improvement initiatives. Each participating center has a trained data manager who has completed a certification exam. The data managers collect and enter data in accordance with the standardized PC⁴ Data Definitions Manual. The PC4 registry shares common terminology and definitions with applicable data points from the International Pediatric and Congenital Cardiac Code (IPCCC), Society of Thoracic Surgeons (STS) Congenital Heart Surgery Database, and American College of Cardiology Improving Pediatric and Adult Congenital Treatment (IMPACT) Registry, as previously described.(6) Participating centers are audited on a regular schedule and audit results suggest complete, accurate and timely submission of data across centers, with the published results demonstrating a major discrepancy rate of 0.6% across 29,476 fields.(7) The University of Michigan Institutional Review Board provides oversight for the PC⁴ Data Coordinating Center; this study was reviewed and approved with waiver of informed consent.

Inclusion and exclusion criteria

The study period spanned from August 1, 2014 through May 31, 2017; the start date corresponds to the current version of the PC4 registry when specific variables were added for the purpose of performing this analysis. Twenty-five CICUs contributed at least one medical admission to the registry over this period.

CICU admissions were included if no cardiac surgery occurred either immediately prior to (e.g. patient not admitted directly from the cardiac operating room or post-anesthesia unit) or during the CICU admission; this defines a medical CICU admission in the registry. Each CICU admission ends with discharge or transfer from the CICU. Repeat CICU admissions (~1%) were eligible for analysis and were treated as independent events in the analysis. Though this statistical assumption is a potential source of bias, these admissions are infrequent and they are high-risk so were deemed clinically important to include. Exclusion criteria included admission for hospice care and/or admissions where a patient had a do not resuscitate order on arrival to the CICU (N=54), and PICU overflow admissions (patients without cardiovascular disease, N=172). One site had submitted only two medical admissions to the registry at the time of analysis and was dropped from comparative analyses.

Outcome and predictor variables

CICU mortality was the primary outcome. We assessed candidate predictors of mortality including patient characteristics (e.g. age, non-cardiac comorbidities), admission characteristics (e.g. source of admission, time of day, reason for admission), acute conditions present on arrival, and a set of variables related to illness-severity captured specifically in the registry for evaluation in this model that included laboratory values and cardiac-specific disease modifiers (Appendix 1). The candidate predictors were selected by consensus and were considered independent of CICU quality of care by the study investigators.

We characterized high-mortality admission diagnoses based on empirical associations with mortality. Briefly, each CICU admission for acute medical condition has one medical diagnosis assigned. We assessed mortality rates for each diagnosis and characterized those in the top quartile as high-risk. Our method classifying each patient as high-risk or low-risk based on the combination of diagnoses and cardiovascular specific comorbidities is described in detail in Appendix 2.

We analyzed acute conditions present at or shortly after CICU admission as potential predictors of mortality. We assessed these variables relative to the CICU time of admission at four separate points (see Appendix 1): present in a previous encounter during the hospitalization, present on arrival to CICU, present on arrival or <1 hour after CICU admission, and present on arrival or <2 hours after CICU admission. There was a high degree of collinearity between these variables, and most were significantly associated with mortality. We operated under a conceptual model that events occurring within the first hour of admission were not related to CICU quality of care, but likely reflected the patients’ severity of illness at the time of arrival. Therefore, we selected acute conditions present on arrival or within the first hour after CICU admission as covariates in the case mix adjustment model, and ignored the other time points. We did include those conditions present in previous encounters if they were significantly associated with mortality on univariate analysis, but the present on arrival variable was not.

Statistical analysis

Our analysis was designed to measure CICU quality of care using mortality as the metric while adjusting for case mix differences across CICUs. We first created a case mix adjusted model (PC4 CICU Medical Mortality Model) to account for differences across hospitals in patient characteristics and illness severity on arrival to the CICU. This model was used to predict mortality at the population level; it was not intended for predicting mortality in individual patients.

We evaluated all candidate predictor variables in univariate analyses using chi-square, Fisher’s exact test, or Wilcoxon rank sum test as appropriate. Variables associated with mortality at p<0.1 in the univariate analyses were included in a multivariable logistic regression model. We used a hierarchical (multilevel) model with a hospital-specific random effect to account for clustering of patients within hospitals. Additionally, a hospital level covariate included in the analysis was whether or not ventricular assist device and/or heart transplant are offered by the institution. Our rationale for this variable was that hospitals that do not offer advanced heart failure therapies may have lower mortality rates as a result of transferring out this high-mortality population; 5 hospitals in the cohort do not offer these advanced heart failure therapies. We used the variance inflation factor test to assess multicollinearity, which showed no significant collinearity among variables included in the model. We then performed bootstrapping (using 1000 resamples) to obtain bias-corrected 95% confidence intervals (CI) of the odds ratio for each covariate in the final model. All covariates whose CI did not cross 1 were included in the final model.

Laboratory values – serum creatinine, lactate, and beta-natriuretic peptide – were recorded if they were drawn within 2 hours before or after admission. Inclusion of laboratory values was complicated by a high rate of missing data; no lab variable was collected on >50% of the admissions and <25% had all three labs collected. We assessed the lab value variables continuously and categorically and included a category for “not obtained.” We imputed the missing lab values for every admission based on other covariates and included these in subsequent multivariable models for predicting mortality. However, inclusion of the imputed lab variables negatively affected stability of the significant predictors of mortality in the final model, with no improvement in model performance. On the basis of these empirical findings we elected to exclude the lab variables from the final case mix adjusted model.

From the bootstrapping we calculated an optimism-corrected c-statistic for the final model. We assessed calibration of the final model using the Hosmer-Lemeshow X² test (p-value > 0.05 demonstrating adequate fit of the model). Goodness of fit was also assessed qualitatively across 10 groups rank-ordered by predicted probability of mortality. Maximum vasoactive-inotropic score (VIS)(8, 9) in the first 2 hours after admission was associated with mortality in the final model. However, the investigative team decided that this variable was susceptible to differences in CICU practice patterns. Therefore, we compared results from models with and without VIS and found the results to be similar. VIS was subsequently dropped from the final model.

We then used the final model to calculate a standardized mortality ratio (SMR) for each hospital. The SMR is calculated as the observed CICU medical mortality divided by the expected CICU medical mortality; expected mortality was derived from our final case mix adjusted model. From our original bootstrapping we obtained bias-corrected 95% CIs around the SMR estimate for each CICU. CICUs whose CIs did not cross 1 were considered to demonstrate statistically significant lower-than-expected (SMR < 1) or higher-than-expected (SMR > 1) mortality. Analyses were performed using SAS Version 9.4 (SAS Institute, Cary, NC) or STATA Version 14 (Stata Corp, College Station, TX), with statistical significance at a p-value of less than 0.05.

Results

The final study cohort included 11,042 CICU medical admissions; this represents 37% of all admissions in the registry (pc4http:\\quality.org; accessed March 10, 2018). CICUs contributed between 2 and 1139 (median 401, interquartile range 301–576). Overall unadjusted mortality was 4.3% (N=479). Observed mortality ranged from 2.0–7.5% among the 24 CICUs in comparative analyses.

Selected patient and admission characteristics are shown in Supplemental Digital Content - Table 1, as are the univariate associations with mortality. Patients with structural heart disease accounted for 70% of admissions. The most common reasons for admission were acute medical condition (72%) and post-cardiac catheterization (14%). Only 9% had cardiac surgery during the same hospitalization and 16% of admissions were readmissions to the CICU.

PC4 CICU medical mortality model

Results of the multivariable analysis to identify predictors of mortality are shown in Table 1. Younger age, ECMO or cardiac arrest within one hour of CICU admission, dialysis, previous diagnosis of sepsis during the hospitalization, and high-risk diagnosis as reason for admission were the strongest predictors of increased mortality. Routine post-procedural care after non-cardiothoracic surgery or catheterization was strongly associated with decreased odds of mortality. The final case mix adjusted model (PC4 CICU Medical Mortality Model) includes the variables in Table 1. The optimism-corrected c-statistic was 0.87 and the Hosmer-Lemeshow X² statistic was 28.4 (p=0.21) demonstrating excellent discrimination and calibration. The corresponding goodness-of-fit plot is shown in a Supplemental Digital Content – Figure 1. The model predicted mortality accurately for both low- and high-risk admissions. Model coefficients are shown in Appendix 3.

Table 1–

Multivariable analyses results from PC4 CICU Medical Mortality Model

Variable	Adjusted Odds Ratio	95% Confidence Interval	β coefficient
Age (Reference: Child)
Preterm neonate	7.3	4.5–11.8	1.99
Term neonate	3.5	2.5–5.0	1.25
Infant	1.5	1.2–2.0	0.41
Adult	1.3	0.9–2.0	0.26
Underweight (Reference: normal)	1.5	1.1–1.9	0.41
Weekend daytime admission (Reference: weekday day)	1.7	1.3–2.2	0.53
Reason for encounter non-cardiac post-operative care or post-catheterization (Reference: all other)	0.4	0.2–0.8	−0.92
Cardiac arrest <1 hour after admission	5.4	2.6–11.4	1.69
Mechanical ventilation at admission	2.8	2.0–3.8	1.03
ECMO <1 hour after admission	5.5	2.5–11.9	1.70
Dialysis on admission	141	6.0–3247	4.95
Pulmonary hypertension during hospitalizationa	3.1	1.4–6.8	1.13
Stroke during hospitalizationa	3.6	1.8–7.1	1.28
Sepsis during hospitalizationa	3.9	1.9–8.2	1.36
Pupillary reflex other than both reactive (Reference: both reactive)	2.9	2.0–4.4	1.06
Cardiac surgery during hospitalizationa	0.5	0.3–0.9	−0.69
High risk diagnosis	5.7	4.2–7.9	1.74
Constant			−5.45

^aprior to CICU medical admission

Assessing variation in CICU quality of care

We used the PC4 CICU Medical Mortality Model to calculate a SMR for 24 CICUs in the cohort. Expected mortality from the model ranged from 2.6–8.3%, reflecting important case mix differences across these CICUs. The SMR varied from 0.5–1.7 (Figure 1). Two CICUs had lower-than-expected mortality (CICUs 3 and 4; SMRs 0.6 and 0.8, respectively) and one was higher-than-expected (CICU 24, SMR 1.7). Figure 2 shows the expected mortality for each CICU arranged from lowest SMR to highest. There was no relationship between SMR and expected mortality (linear correlation coefficient, R² = 0.05), suggesting that the model provides a measure of adjusted mortality that is independent of the underlying case mix.

Figure 1 –

Standardized mortality ratio across CICUs with 95% confidence intervals

Figure 2 –

Expected mortality by standardized mortality ratio

Discussion

This is the first report demonstrating variation in adjusted mortality across pediatric CICUs related to patients admitted for medical conditions, which may represent differences in quality of care. We observed that some CICUs have important outlying standardized mortality rates in this patient population. Developing case mix adjusted quality metrics for CICU medical admissions may stimulate quality improvement initiatives for this growing patient population.

Our analysis revealed significant case mix variation across CICUs; expected mortality varied almost 4-fold across the CICUs in PC⁴. These data imply that hospitals differentially utilize their cardiac critical care beds, and indirectly may reflect differences in resources available in non-ICU settings. The observed variation in expected mortality (i.e. case mix) underscores the importance of developing and applying a clinically valid case mix adjustment model for pediatric CICUs, as we did in this analysis. While not intended or statistically structured to directly compare the performance of two or more CICUs, this model allows each CICU to assess its quality of care relative to what is expected for its case mix. Given the differences in case mix across hospitals, observed outcomes alone would be misleading as a quality indicator for many CICUs. Previous risk adjustment methods have included cardiac medical patients in the derivation and validation cohorts (2, 3), but none of these have been evaluated specifically within this population. Informed by this previous work, we included some of the well-known predictors of mortality, and extended it by adding granular information on cardiac diagnoses, surgical palliation status, and other markers of illness severity known to impact outcomes in this population.

We identified 3 of 24 CICUs whose mortality was not as expected, rather they had outlying performance (higher- or lower-than-expected). It is important to note that some CICUs had relatively small sample sizes (<200 medical admissions) during the study period. This may in part be due to differences in CICU capacity and surgical volume, but also because each hospital began submitting data to the clinical registry at different time points. It is possible that additional hospitals may be identified as statistical outliers as more cases accrue. Identification of these outliers is crucial to quality improvement activities. PC4 has developed a transparent reporting platform where CICUs can identify peers in the Consortium who achieve better-than-expected performance across a variety of quality metrics. The metric of adjusted CICU medical mortality has been implemented onto this reporting platform, and as such CICUs looking to improve their care to patients admitted for medical indications can access the knowledge and experience at these high performing sites. Further investigation may identify characteristics specific to high performing CICUs and improve the care of patients with medical cardiovascular disease.

Our focus on the medical subpopulation was driven by our interest in providing CICUs with tools to assess and improve quality. Examining overall mortality may not provide the requisite insight to improve because performance could differ within a CICU between surgical and medical populations. It is possible that systems factors and practices associated with performance in surgical admissions do not impact the care of medical patients, and vice versa. Our approach to case mix adjustment can be applied to quality measurement for other outcomes in the medical CICU population such as acquired complication rates, failure-to-rescue,(10) and CICU length of stay, which should allow CICUs to evaluate their performance across a wide array of quality metrics. Coupled with models developed specifically for post-operative admissions (would reference companion article submitted by Tabbutt et al if accepted in this issue of the Journal), CICUs will be able to identify areas of strength and weakness within these discrete patient populations and identify the specific resources necessary to improve care. Future work will examine differences in CICU outcomes across surgical and medical patients, and what structure and process factors may relate to differential performance.

There are potential limitations to our analysis. First, we chose to evaluate predictors of mortality up to the first hour after CICU admission. While we believe that events in that time period like cardiac arrest and ECMO cannulation are more likely related to patient illness severity than CICU quality, in some instances this may not be true. This approach was endorsed by consensus among a multi-institutional group of CICU clinician-scientists from PC⁴ and has been accepted by the greater community within the Consortium. Second, we did not analyze a full complement of CICU and hospital characteristics associated with SMR to identify whether certain organizational, system, or practice variables may associate with quality. This is certainly an area of intense interest and will be assessed in follow-up analyses. Third, because of the way medical and surgical admissions are defined in the registry, some patients, particularly those with heart failure, present with an acute medical condition and then have surgery. These patients are not included in medical population outcome assessments such as this analysis, but they do require treatment of acute medical conditions. Fourth, mortality as an outcome does not account for varying institutional philosophies regarding withdrawal of life-sustaining therapies and the balance between functional survival and death when measuring quality. Finally, this analysis was conducted among large, dedicated pediatric CICUs from North America, so it remains unclear how the results would generalize to other care settings.

Conclusion

We demonstrated variation in adjusted mortality for medical admissions across a group of 24 pediatric CICUs. We must identify other domains where CICU quality of care varies in this patient population and determine whether important features of high performing CICUs can be disseminated to other institutions. The findings from this analysis have already been implemented into a tool that allows CICUs in PC⁴ to monitor their outcomes and engage in collaborative learning with peers at CICUs who excel in the care of medical patients. Future work will focus on quality improvement that stems from this process.