Hospital Variation in Risk-adjusted Pediatric Sepsis Mortality

Abstract

Objective

With continued attention to pediatric sepsis at both the clinical and policy level, it is important to understand the quality of hospitals in terms of their pediatric sepsis mortality. We sought to develop a method to evaluate hospital pediatric sepsis performance using 30-day risk-adjusted mortality and to assess hospital variation in risk-adjusted sepsis mortality in a large state-wide sample.

Design

Retrospective cohort study using administrative claims data

Setting

Acute care hospitals in the state of Pennsylvania from 2011 through 2013

Patients

Patients between the ages of 0 to 19 years admitted to a hospital with sepsis defined using validated ICD-9-CM diagnosis and procedure codes

Interventions

None

Measurements and Main Results

During the study period, there were 9013 pediatric sepsis encounters in 153 hospitals. After excluding repeat visits and hospitals with annual patient volumes too small to reliably assess hospital performance, there were 6468 unique encounters in 24 hospitals. The overall unadjusted mortality rate was 6.5% (range across all hospitals: 1.5% to 11.9%). The median number of pediatric sepsis cases per hospital was 67 (range across all hospitals: 30 to 1858). A hierarchical logistic regression model for 30-day risk-adjusted mortality controlling for patient age, gender, emergency department admission, infection source, presence of organ dysfunction on admission, and presence of chronic complex conditions showed good discrimination (C-statistic = 0.80) and calibration (slope and intercept of calibration plot: 0.95 and -0.01 respectively). The hospital specific risk-adjusted mortality rates calculated from this model varied minimally, ranging from 6.0 to 7.4%.

Conclusions

Although a risk-adjustment model for 30-day pediatric sepsis mortality had good performance characteristics, the use of risk-adjusted mortality rates as a hospital quality measure in pediatric sepsis is not useful due to the low volume of cases at most hospitals. Novel metrics to evaluate the quality of pediatric sepsis care are needed.

INTRODUCTION

Pediatric sepsis is a common cause of morbidity and mortality in the United States. It accounts for over 75,000 pediatric inpatient admissions annually, is associated with a mortality rate of 5 to 20%, and results in decreased functional status in approximately one-third of survivors (1–4). Several treatment strategies can improve outcomes in children with sepsis, including early recognition and treatment with appropriate fluid resuscitation and timely source control (2, 5–9). Based on these strategies, many hospitals have initiated quality improvement efforts aimed at improving initial pediatric sepsis care (10–13). Adding to these quality efforts are policy initiatives designed to improve sepsis outcomes by encouraging similar guideline implementation. The first of these policies, enacted in New York State in 2014, requires all hospitals to develop and implement an evidence-based protocol for the recognition and treatment of sepsis for all patients, including children (14).

With increasing attention to improving pediatric sepsis outcomes, it is crucial to be able to assess hospital quality in terms of pediatric sepsis care. Accurate and reliable outcome measurement is necessary to measure the success of quality improvement programs and policy initiatives, as well as to drive further improvements in care by helping to identify high quality providers in order to learn from their practices. Risk-adjusted mortality is a direct and patient-centered method of assessing quality of care at hospitals (15–17). Mortality rates are also widely available in the state and national administrative datasets that are commonly used for performance assessment. This stands in contrast to other pediatric outcome measures, including functional morbidity scores such as the Functional Status Score (FSS) or Patient Overall Performance Category (POPC) (18–20), which are often used in clinical research but require clinical data elements and therefore are not useful in quality measurements using administrative data to compare hospitals broadly. Accordingly, we sought to develop a method using administrative claims data to assess hospital performance based on 30-day risk-adjusted mortality and to apply this method to a large state-wide sample to evaluate variation in risk-adjusted mortality.

MATERIALS AND METHODS

Study design, data, and patients

We performed a retrospective cohort study using administrative data obtained from the Pennsylvania Healthcare Cost Containment Council (PHC4) from 2011-2014. PHC4 is an independent state agency that collects information on all inpatient hospital discharges from Pennsylvania hospitals. Each record contains de-identified data from the administrative claim, including patient demographics, dates of service, and International Classification of Diseases, 9^th revision, Clinical Modification (ICD-9-CM) diagnosis and procedure codes. We identified patient encounters for all patients aged 0 to 19 with ICD-9-CM codes consistent with sepsis, using both the “explicit”’ diagnosis codes for severe sepsis (995.92) and septic shock (785.52) and the “implicit” coding framework based on validated codes for infection and organ dysfunction (3, 21). This approach was chosen because it identifies clinical sepsis with the best mix of sensitivity and specificity compared to other administrative methods (22). Although these codes are limited in terms of their ability to accurately identify sepsis, clinical case ascertainment is not yet feasible at the state level and we considered these codes to be sufficiently accurate for making comparisons across hospitals.

We included only patients admitted to general acute care hospitals, excluding skilled nursing facility and long-term acute care hospital admissions as well as primary maternity hospitals. We defined primary maternity hospitals as hospitals with more than 70% of pediatric admissions stemming from a live birth. We also excluded hospitals with less than ten pediatric sepsis cases per year in order to reduce random variation from small sample sizes. In the event of multiple claims of sepsis per patient, we randomly selected one episode in order to prevent interdependence of observations.

Study Variables

Patient characteristics including age, gender, race, payer, admission source, intensive care unit admission, length of stay, and discharge location were obtained directly from the claims. Complex chronic conditions and organ dysfunctions present on admission were identified using ICD-9-CM codes as previously described with the addition of present-on-admission septic shock (23, 24). We used previously-validated present-on-admission indicators to determine which codes were present on admission, as has been performed in prior risk-adjustment models (25). To identify the type of infection, we grouped validated ICD-9-CM infection codes into mutually exclusive categories based on observed frequency and organ system affected (21). We assigned infection types hierarchically so that each patient was assigned the infection type associated with the highest average mortality, such that each patient was assigned only one type. Mortality at 30-days from hospital admission was determined using death certificate records from the Pennsylvania Department of Health, linked to the PHC4 data using direct identifiers.

Hospital characteristics included hospital size, hospital type, teaching status and pediatric capabilities. We based hospital size on the total number of beds, using data from the 2012 Healthcare Cost Report Information System (HCRIS). We based teaching status on the ratio of resident full-time equivalents to total beds, also using data from HCRIS. We defined “non-teaching hospitals” as those with no residents, “small teaching hospitals” as those with a ratio above zero but under 0.2, and “large teaching hospitals” as those with a ratio of 0.2 or above. Pediatric capabilities were obtained from the 2011 American Hospital Association’s annual survey and included the presence or absence of a dedicated pediatric ED, a dedicated pediatric ward, and a pediatric intensive care unit.

Statistical Analysis

We used hierarchical logistic regression models to generate hospital-specific risk-adjusted 30-day mortality rates. To do this we fit a mixed-effects logistic regression model using 30-day mortality as the dependent variable and patient gender, age, infection type, the presence of complex chronic conditions (as indicator covariates), an indicator for the emergency department as the admission source, and the presence of organ dysfunctions on admission (as indicator covariates), as independent variables. These variables were chosen based on their a priori hypothesized relationship to increased mortality. Emergency department admission source was included as an indicator for admission source, which has been associated with variation in mortality (26). Based on these models we calculated predicted and expected mortality rates and then derived risk-adjusted mortality rates as a ratio of predicted and expected mortality rates for each hospital multiplied by the overall unadjusted mortality rate.

Predicted mortality, as opposed to observed mortality, is the number of deaths among a hospital’s patients accounting for both patient-level factors as well as unmeasured hospital-level effects—it is the mortality at that hospital factoring in hospital quality while also acknowledging that at small hospitals there is uncertainty about that quality. Expected mortality is the mortality among a hospital’s patients accounting for only patient-level factors—it is the mortality at that hospital that would be expected if all hospitals were of the same quality. Dividing these two numbers and multiplying by the state average gives mortality rates that are both risk-adjusted and reliability-adjusted. They are risk-adjusted in that the rates account for variation in baseline risk across hospitals and they are reliability-adjusted in that for small hospitals with low reliability the model moves the adjusted rates toward the state-wide mean. This process is identical to that used in multiple previous studies which calculate risk-adjusted mortality rates (27–29). We evaluated model discrimination using the area under the receiver-operator curve and calibration using slope and intercept values from a calibration plot.

We also performed a sensitivity analysis to evaluate the impact of neonatal sepsis on the model by repeating our analyses after excluding patients less than one month of age.

We performed all analysis using STATA version 14.1 (StataCorp, College Station, TX). In accordance with the policies of the University of Pittsburgh Hospital Institutional Review Board, our use of this de-identified data set was considered exempt human subjects research.

RESULTS

Patient and Hospital Characteristics

A total of 9013 records for patients age 0 to 19 with a diagnosis of sepsis were identified in 153 hospitals in Pennsylvania from 2011 through 2013. After exclusions, the final cohort contained 6468 unique patient encounters in 24 hospitals (Figure 1). There were 106 (69.2%) hospitals (727 patients) excluded based upon an annual volume of patients <10.

FIGURE 1.

Cohort Inclusion and Exclusion Flow Diagram

Patient demographic characteristics by survival status are shown in Table 1. The median patient age was 3 years (Interquartile range: 0 to 13 years) with 35.8% being less than one year old. The majority of patients (74.0%) had at least one complex chronic condition, with the mean number of complex chronic conditions being 1.5± 1.3. The overall unadjusted mortality rate for the entire cohort was 6.5%. Almost two-thirds (62.9%) of patients were admitted to a dedicated children’s hospital, and nearly all (94.5%) patients were admitted to a large teaching hospital.

TABLE 1.

Patient Characteristics for Cohort Compared by Survivorship

Characteristic	All patients (n=6468)	Survivors (n=6050)	Non-survivors (n=418)	p-value
Age in year (median, IQR)	3 (0, 13)	3 (0, 14)	0 (0, 10)	<0.001
Male gender	3531 (54.6%)	3295 (54.5%)	236 (56.5%)	0.41
Race				<0.001
White	3386 (52.4%)	3200 (52.9%)	186 (44.5%)
Black	1381(21.4%)	1306 (21.6%)	75 (17.9%)
Other	778 (12.0%)	711 (11.8%)	67 (16.0%)
Unknown	923 (14.3%)	833 (13.8%)	90 (21.5%)
Payer information				0.28
Commercial	2683 (41.5%)	2498 (41.3%)	185(44.3%)
Medicaid	3558 (55.0%)	3342 (55.2%)	219 (51.7%)
Uninsured	163 (2.5%)	150 (2.5%)	13 (3.1%)
Other	64 (1%)	60 (1%)	4 (1%)
Number of CCC (mean ± SD)	1.5 ± 1.3	1.5 ± 1.3	2 ± 1.2	<0.001
Any CCC	4805 (74.3%)	4420 (73.1%)	385 (92.1%)	<0.001
Mechanical ventilation	3615 (55.9%)	3288 (54.4%)	327 (78.2%)	<0.001
Sepsis identification
Explicit codes only	473 (7.3%)	427 (7.1%)	46 (11.0%)	<0.001
Implicit codes only	5220 (80.7%)	4974 (82.2%)	246 (58.9%)	<0.001
Both codes	775 (12.0%)	649 (10.7%)	126 (30.1%)	<0.001
Hospital LOS in days (median, IQR)	13 (5, 33)	13 (5, 34)	17 (5, 42)	0.14
ICU LOS in days (median, IQR)	6 (2, 14)	6 (2, 13)	10 (3, 23)	<0.001

IQR = interquartile range, CCC = chronic complex condition, SD = standard deviation, ED = emergency department, ICU = intensive care unit, LOS = length of stay

Hospital characteristics are shown in Table 2. The median number of sepsis cases per hospital over the three-year study period was 68 with a total range from 35 to 1861 cases. Of the included hospitals, 19 (79.2%) were large teaching hospitals and the median bed size was 330 (Interquartile range: 217 to 369). Only 6 (25%) hospitals had pediatric intensive care unit capacity. Overall unadjusted hospital-specific mortality ranged from 1.5% to 11.9%.

TABLE 2.

Hospital Characteristics of Final Cohort

Characteristic	Hospitals (n= 24)
Total number of hospital beds
>250 beds	23 (92%)
100-250 beds	2 (8.3%)
Pediatric ward	17 (70.8%)
Pediatric emergency department	10 (41.7%)
Pediatric intensive care unit	6 (25.0%)
Dedicated children’s hospital	3 (12.5%)
Total volume of pediatric sepsis cases
>1000	2 (8.3%)
100-1000	5 (20.8%)
30-99	17 (70.8%)
Teaching Hospital
Large teaching	19 (79.2%)
Small teaching	4 (16.7%)
Non-teaching	1 (4.2%)

Risk-adjusted hospital specific mortality

The final risk-adjustment model is shown in Table 3. The model showed good discrimination and calibration, with a C-statistic of 0.80 and calibration plot with intercept of -0.01 (p=0.62) and slope of 0.95 (p=0.44). Notable factors associated with a statistically significant increased odds of death included diagnostic coding of septicemia; neurologic, cardiovascular, hematologic/immunologic, malignancy or neonatal-related chronic complex conditions; as well as presence of respiratory failure, liver failure or neurologic dysfunction on admission. Overall hospital-specific risk-standardized mortality rates based on this model varied from 6.2% to 7.0% (Figure 2). In the sensitivity analysis excluding neonates under the age of 1 month (n=1474), the overall mortality rate was 5.8% and risk-adjusted hospital-specific mortality rates did not vary substantially.

TABLE 3.

Results of the Logistic Regression Model for Risk-Adjustment of Pediatric Sepsis Mortality

Variable	Odds ratio	95% CI	p-value
Age < 1 year	0.54	0.36-0.81	<0.01
Age 1-5 years	0.91	0.77-1.06	0.22
Age 5-12 years	1.07	0.98-1.18	0.15
Age >12 years	0.97	0.89-1.06	0.49
Female	0.90	0.73-1.12	0.35
ED admission source	0.72	0.55-0.92	<0.01
Infection type
Bacteremia	1.75	0.72-4.26	0.22
Central nervous system	1.96	0.85-4.53	0.12
Fungal	2.08	0.85-4.60	0.07
Gastrointestinal	1.60	0.20-13.09	0.66
Genitourinary	2.24	0.94-5.34	0.07
Cardiovascular	3.12	0.81-12.07	0.10
Respiratory	1.70	0.88-3.28	0.27
Peritoneal	2.27	1.06-4.86	0.04
Sepsis, general	5.42	2.91-10.09	<0.01
Soft tissue	0.13	0.02-1.03	0.05
Complex chronic condition
Neuromuscular	1.85	1.39-2.46	<0.01
Cardiovascular	2.01	1.59-2.55	<0.01
Respiratory	1.26	0.93-1.71	0.14
Renal	1.57	1.08-2.28	0.02
Gastrointestinal	0.65	0.47-0.90	<0.01
Hematologic/Immunologic	1.97	1.45-2.69	<0.01
Metabolic	1.48	1.09-2.01	<0.01
Genetic	1.37	0.98-1.91	0.07
Malignancy	1.70	1.19-2.44	<0.01
Neonatal	2.69	2.02-3.57	<0.01
Technology dependence	0.74	0.54-1.01	0.06
Transplant	1.63	1.03-2.57	0.04
Present on admission indicator
Respiratory failure	1.47	1.13-1.92	<0.01
Cardiovascular failure	0.92	0.60-1.41	0.51
Renal injury	1.67	1.17-2.39	<0.01
Liver failure	3.21	1.87-5.52	<0.01
Neurologic injury	2.92	1.82-4.70	<0.01
Acid/base disturbance	0.90	0.59-1.36	0.61
Coagulopathy	1.55	1.16-2.07	<0.01
Septic shock	0.63	0.37-1.06	0.08

CI = confidence interval, ED = emergency department

FIGURE 2.

Distribution of Variation in Risk-Adjusted Mortality Rates by Hospital

DISCUSSION

In a large state-wide sample, we found that most cases of pediatric sepsis were admitted to a relatively small number of hospitals. Unadjusted pediatric sepsis mortality rates varied widely across these hospitals. However, when adjusted for differences in case-mix and reliability due to small numbers of patients at some hospitals, mortality rates varied only a small amount.

Risk-adjusted mortality rates are valuable to benchmark hospitals on their quality because they are relatively easy to measure using widely available claims data and mortality is an extremely patient-centered outcome. Indeed, risk-adjusted mortality is used in many fields of medicine as a quality metric to benchmark hospitals (27, 29, 30). However, the low patient volumes and minimal variation seen with risk-adjusted mortality in pediatric sepsis makes this approach less useful for understanding hospital quality for pediatric sepsis care. Low volumes affected our models in two ways. First, only 17% of Pennsylvania hospitals met our minimum volume threshold for inclusion in the study, reducing the number of hospitals that we were able to study. Although low case volumes themselves may be a marker of poor quality, the fact that no estimates could be made at 83% of Pennsylvania hospitals is concerning (31). Second, among those remaining, our hierarchical modeling approach, which is a standard approach to benchmarking (32), substantially shrunk the mortality rate toward the overall average. In one sense, this approach may obscure poor quality at small hospitals leading to false inference (33). However, this approach also ensures that small hospitals are not penalized for random variation in the number of sepsis deaths per year. A similar problem in detecting meaningful variation has been described in the surgical literature where case-load and mortality rates for most hospitals are also low (34).

The inability to use risk-adjusted mortality as an outcome in measuring quality of pediatric sepsis care poses a significant problem. Increasingly health policy-makers are implementing novel policy-oriented efforts designed to improve sepsis outcomes. Although New York is at present the only state to have implemented these policies, Illinois, Pennsylvania, and Wisconsin are following suit (14). However, there is currently no method to measure success of these programs broadly using a relevant, patient-centered outcome measure. While functional outcomes measures such as the FSS and POPC are useful in clinical trials and other clinical research, at this time these scores are not yet useful for benchmarking due to their lack of availability in large administrative data sets. As more states develop and implement their own policy initiatives and as hospitals devote resources to improving the quality of their sepsis care, incorporating functional outcome assessments into routine care and making electronic health record (EHR) data available for quality measurement may aid the evaluation of these efforts.

Indeed, as more hospitals develop their EHR systems, there will be significant opportunity to develop risk-adjustment models using granular clinical data, thus overcoming many of the problems inherent in administrative-data based performance assessment. These include not only limitations due to risk-adjustment but also limitations with case identification of sepsis seen with use of administrative codes (35). Clinical data would also allow for a more robust risk-adjustment model using validated severity of illness scores such as the Pediatric Index of Mortality or the Pediatric Logistic Organ Dysfunction score (36, 37). We acknowledge that the lack of granular clinical data is a potential source of bias in our study.

Until EHR based measures can be developed and tested, alternative approaches to quality measurement are needed. One strategy would be the use of non-outcome based performance measures, such as the process of care (e.g. the percentage of adherence to an early treatment bundle (38)) or the structure of care (e.g. whether or not the hospital has the equipment needed to resuscitate pediatric sepsis patients (39) or employs trained pediatric intensivists at night (40)). Process and structure measures are useful because they are less sensitive to low volumes than outcome measures. However, they are also less relevant to patients and their families. Another strategy would be to estimate risk-adjusted outcomes other than mortality. Derivation of a functional morbidity score similar to the FSS or POPC for administrative data would be a valuable tool for quality measurement in pediatric care. A third option is the use of composite performance measures. Composite measures combine several weighted factors into an aggregate score to increase the precision of mortality assessments and create a wider window into hospital quality (43). In sepsis, these factors may take into account hospital volume, hospital-specific mortality of sepsis, as well as other related events. This approach, which in part considers low volumes themselves as markers of poor performance, has been used to measure quality in other areas of medicine with low patient volumes and low mortality such as the surgical field (34, 43).

CONCLUSION

With small volumes of cases at most centers and a low overall mortality rate in pediatric sepsis, hospital-level risk-adjusted mortality is not useful as a quality measure. Given the pressing need to understand hospital quality for pediatric sepsis, novel approaches that build off our results by developing composite measures and other approaches to measure pediatric sepsis quality are urgently needed.