Evaluating inpatient adverse outcomes under California's Delivery System Reform Incentive Payment Program

Abstract

Objective

The California Delivery System Reform Incentive Payment Program (DSRIP) provided incentive payments to Designated Public Hospitals (DPHs) to improve quality of care. We assessed the program's impact on reductions in sepsis mortality, central line–associated bloodstream infections (CLABSIs), venous thromboembolisms (VTEs), and hospital‐acquired pressure ulcers (HAPUs).

Data Sources

We used 2009‐2014 discharge data from California hospitals.

Study Design

We used a pre‐post study design with a comparison group. We constructed propensity scores and used them to assign inverse probability weights according to their similarity to DPH discharges. Interaction term coefficients of time trends and treatment group provided significance testing.

Data Extraction

We used Patient Safety Indicators for CLABSI, HAPU, and VTE, and constructed a sepsis mortality measure.

Principal Findings

Discharges from DPHs and non‐DPHs both saw decreases in the four outcomes over the DSRIP period (2010‐2014). The difference‐in‐difference estimator (DD) for sepsis was only significant during two time periods, comparing 2010 with 2012 (DD: −2.90 percent, 95% CI: −5.08, −0.72 percent) and 2010 with 2014 (DD: −5.74, 95% CI: −8.76 percent, −2.72 percent); the DD estimator was not significant comparing 2010 with 2012 (DD: −1.30, 95% CI: −3.18 percent, 0.58 percent) or comparing 2010 with 2013 (DD: −3.05 percent, 95% CI: −6.50 percent, 0.40 percent). For CLABSI, we did not find any meaningful differences between DPHs and non‐DPHs across the four time periods. For HAPU and VTE, the only significant DD estimator compared 2014 with 2010.

Conclusions

We did not find that DPHs participating in DSRIP outperformed non‐DPHs during the DSRIP program. Our results were robust to multiple sensitivity analyses. Given multiple concurrent inpatient safety initiatives, it was challenging to assign improvements over time periods to DSRIP.

1. What is Known on This Topic

• California implemented a pay‐for‐performance initiative known as the Delivery System Reform Incentive Payment (DSRIP) Program from 2010 to 2015 aimed at improving health care delivery and inpatient safety.

• Given the upfront costs associated with intense quality improvement and public reporting efforts, some have expressed concern that public hospitals could underperform in pay‐for‐performance initiatives.

2. What This Study Adds

• This study evaluates the effect of DSRIP on reducing inpatient adverse outcomes (severe sepsis mortality, central line–associated bloodstream infections, venous thromboembolisms, and pressure ulcers) given the potential for pay‐for‐performance programs to support patient safety initiatives.

• We did not find that hospitals participating in DSRIP outperformed nonparticipating hospitals over the evaluation time period (2010‐2014).

• The inpatient adverse events included in the DSRIP program were and continue to be the focus of a variety of national and state initiatives that were in effect contemporaneously with DSRIP; it is possible that these other initiatives encouraged improvement in these measures in many health systems.

1. INTRODUCTION

Under a Section 1115 Medicaid Demonstration Waiver, an avenue that allows states to test innovative ways to pay for and deliver health care, California implemented a pay‐for‐performance initiative known as the Delivery System Reform Incentive Payment (DSRIP) Program from November 2010 to October 2015. ¹ This initiative provided up to $3.4 billion in federal funds to 17 Designated Public Hospital (DPH) systems to increase access to care, build quality improvement programs, create population health programs, and implement processes to reduce inpatient adverse outcomes. All DPH systems participated in DSRIP, including five University of California (UC) hospital systems and 12 county‐owned systems. Under DSRIP, DPHs implemented five categories of projects to: (1) improve ambulatory care infrastructure, (2) improve ambulatory care delivery processes, (3) develop capacity for data capture and reporting in ambulatory and outpatient settings, (4) improve inpatient care, and (5) improve access to care for persons with HIV. The funds were provided to support quality improvement activities and were separate from funds provided to the health systems for direct care provision. Projects and milestones were approved by the Centers for Medicare and Medicaid Services (CMS).

For the inpatient care category, hospitals were required to participate in two required projects and two additional projects selected from a broader set of options; choosing from the optional projects allowed DPHs to focus on high‐value projects. The required projects included severe sepsis detection and management and central line–associated blood stream infection (CLABSI) prevention. The optional projects included venous thromboembolism (VTE) prevention and treatment, hospital‐acquired pressure ulcer (HAPU) prevention, stroke management, hospital fall prevention, and surgical site infection (SSI) prevention. DPHs were required to meet specific milestones within each project in order to receive funds. Milestones included implementing evidence‐based inpatient safety bundles (such as sepsis, CLABSI, VTE bundles), reporting data on bundle compliance to the state, participating in quality improvement learning sessions, leveraging electronic health records to document compliance, and documenting improvement (Table 1).

TABLE 1.

Activities and milestones for inpatient safety measures included in DSRIP

Inpatient safety measure	Types of milestones and activities conducted by Designated Public Hospitals (DPHs)
Severe sepsis	The sepsis management project required DPHs to report sepsis mortality and bundle compliance adherence measures. DPHs also participated in collaboratives and shared learning opportunities to reduce severe sepsis mortality.
Central line–associated bloodstream infection (CLABSI)	The CLABSI prevention project required DPHs to report the overall rate of CLABSI in the DPH and the rate of CLABSI in the DPH’s ICU, neonatal ICU, and acute care units. CLABSI prevention also measured adherence to the Central Line Insertion Practices (CLIP) bundle. DPHs also participated in collaboratives and shared learning opportunities to reduce CLABSIs.
Venous thromboembolism prevention and management	VTE prevention milestones included reporting process measures that assessed the rate of patients that received: VTE prophylaxis, intensive care unit VTE prophylaxis, anticoagulation overlap therapy, unfractionated heparin with dosages/platelet count monitoring by protocol, VTE discharge instructions.
Hospital‐acquired pressure ulcer (HAPU) prevention	Activities included establishing processes to: ensure timely placement of patients on appropriate pressure redistribution surfaces, ensure timely mobilization, and communicate patients with “skin at risk.” DPHs also reported HAPU prevalence rates to the state and participated in shared learning opportunities.

We focused on evaluating the effect of DSRIP on reducing inpatient adverse outcomes given the importance of reducing hospital‐acquired infections and never events in hospitals and the availability of state discharge data to construct outcome measures for discharges from DPHs and from other hospitals in California. In addition, given the upfront costs associated with intense quality improvement and public reporting efforts, some have expressed concern that public hospitals could underperform in pay‐for‐performance initiatives. ² Several studies have found that hospitals with a large proportion of Medicaid patients perform worse on quality‐of‐care measures, patient experience, and 30‐day readmissions than those with low proportions of Medicaid patients, ³ , ⁴ , ⁵ , ⁶ , ⁷ while others have suggested that quality improvement efforts in safety net settings may be short‐lived. ⁸ Thus, we were interested in examining whether funding intense quality improvement efforts in health systems defined as DPHs would be associated with increased reductions in inpatient adverse outcomes compared with similar health systems not receiving these incentives.

1.1. Inpatient adverse outcomes

In this paper, we focused on the two required DSRIP inpatient care projects (severe sepsis management and CLABSI reduction) and two of the three most frequently selected optional projects (VTE prevention and management and HAPU prevention and management). The measures evaluated in this analysis are of particular public health significance in the United States. Severe sepsis is a leading cause of death and one of the primary drivers of mortality in the noncoronary intensive care unit (ICU) setting. ⁹ CLABSIs are associated with high mortality rates among ICU patients and have been found to significantly increase a patient's length of stay and costs in the hospital. ¹⁰ , ¹¹ , ¹² More than 2.5 million people each year develop pressure ulcers in hospitals in the United States. ¹³ HAPUs are largely preventable and estimated to cost $11 billion in excess hospital costs annually. ¹⁴ An estimated 300, 000 individuals are affected by VTE each year, defined as either deep vein thrombosis or pulmonary embolism, and approximately 50 percent of VTE events are hospital‐associated. ¹⁵ VTE is one of the leading causes of maternal mortality in the United States and one of the greatest causes of preventable deaths in hospitals. ¹⁶

1.2. Patient safety initiatives prior to and concurrent with DSRIP

Our previous evaluation of the DSRIP program, which included a survey of the status of various quality improvement initiatives prior to the program, indicated that DPHs had ongoing initiatives aimed at reducing inpatient adverse outcomes. ¹³ , ¹⁷ For example, 17 health systems had prior efforts to reduce CLABSIs, 9 systems had prior efforts to improve care for sepsis, 11 systems had SSI prevention programs, and 12 systems had ongoing HAPU prevention programs. However, we found that key health system stakeholders noted that the DSRIP program provided resources to expand quality initiatives to additional departments or set new goals. Furthermore, key stakeholders at DPHs noted that DSRIP projects, such as the sepsis initiative, helped galvanize the organization into action, as sepsis outcomes were not national patient safety goals at program outset.

DPHs also reported clinical process improvements attributable specifically to DSRIP. These process improvements included increased adherence to compliance to bundles such as the sepsis bundle, with adherence to the bundle increasing from an average of 59.9 percent to 73.4 percent, and adherence to Central Line Insertion Practices, with adherence increasing from 95.4 percent to 98.7 percent. ¹⁸ Other clinical process improvements included increased use of evidence‐based practices, such as providing warfarin therapy discharge instructions and implementing prophylaxis programs to reduce VTEs.

Several national initiatives, including the CMS Meaningful Use incentive program, aimed at improving electronic health record use, and CMS hospital quality initiatives, focused on reducing hospital‐acquired infections, were occurring at the same time as the DSRIP program. ¹⁹ , ²⁰ In our evaluation, DPHs reported that these initiatives were synergistic with DSRIP projects. For example, DPHs reported that Meaningful Use participation contributed to DSRIP implementation by expanding DPH capacity to monitor and report on patient safety outcomes. ¹⁸ Non‐DPHs were also eligible for the CMS Meaningful Use incentive program and other CMS quality initiative programs. In addition, California required reporting of specific Patient Safety Indicators, including CLABSIs, since 2005, which would have been applicable to both DPHs and non‐DPHs throughout the DSRIP time period. ²¹

1.3. Study objectives

The objectives of this study were to assess whether increased incentives provided by DSRIP for training, staff, and quality improvement efforts (such as bundle implementation), as well as participation in statewide collaboratives, were associated with reduced adverse inpatient outcomes in health systems participating in the DSRIP program. To compare the DSRIP program's performance over time and in context with several co‐occurring national and state patient safety efforts, we constructed a propensity score–matched group of non‐DPH discharges. Discharges from this group included individuals discharged from a mix of private and noncounty public hospitals in California. We used the Donabedian conceptual framework to guide our study design, as we hypothesized that improving quality program infrastructure and processes of care delivery would result in decreased inpatient adverse outcomes. ²² We hypothesized that DSRIP provided the resources and incentives for DPHs, above and beyond existing initiatives, to improve performance on inpatient adverse outcome measures in contrast to non‐DPHs. For example, we hypothesized that initiatives such as increasing training for clinical staff on evidence‐based CLABSI bundles and implementing electronic health record configurations to track bundle adherence (structure‐related changes) increased adherence to the bundles (process‐related changes), which would result in reduced CLABSIs (outcomes). ¹³ , ¹⁷

2. METHODS

2.1. Study design

To examine the impact of DSRIP on inpatient adverse outcomes, we used a pre‐post study design with a comparison group (discharges from non‐DPHs) and specified separate logistic regression models for each outcome measure. We used this study design to account for other federal, state, and private initiatives that may have affected the outcome measures in both DPHs and non‐DPHs before and during the DSRIP period. For the required sepsis and CLABSI projects, all discharges from DPHs participating in DSRIP were compared with the comparison group. For the optional HAPU and VTE projects, only discharges from DPHs participating in each project were included in these analyses.

2.2. Setting

As noted earlier, the 17 DPH health systems included 12 county‐owned and county‐operated health systems, with a total of 15 hospitals within these systems, and five University of California (UC) systems, with a total of six hospitals within the UC systems. These health systems serve most of the state's low‐income population, including the remaining uninsured and Medicaid beneficiaries. ¹

2.3. Data sources

We used the confidential California Office of Statewide Health Planning and Development (OSHPD) patient discharge data and OSHPD financial and statistical reports for this analysis. We constructed outcome measures at the patient level and used 2009‐2014 OSHPD annual patient discharge data to compare adverse outcomes and changes in rates over time in DPHs and comparison hospitals. We ran two sets of analyses: one with the full sample and another with a sample restricted to discharges where Medicaid was the primary payer. We elected to include both because: (a) the Medicaid‐only group has a reduced sample size, which decreases our precision; and (b) the Medicaid population is younger, which reduces the probability of the outcomes. Thus, by including both models, we were able to estimate the population‐level effect and confirm the trends in the Medicaid population, the target of the 1115 Demonstration Waiver.

2.4. Timeline

We used 2010 as the baseline period. We treat 2011‐2014 as the full DSRIP implementation period.

2.5. Propensity scores

The role of DPHs within California's health care system made achieving a balanced comparison group challenging. DPH patients were distributed differently across a number of demographic variables in OSHPD data, even in comparison with private hospitals in California that were otherwise similar in terms of size, payer mix, ownership type, and teaching status. We opted to use inverse probability weights to address discharge‐level demographic and comorbidity characteristics for our primary models. To do this, we first ran a logistic regression model, using the demographic variables present in OSHPD discharge records to predict likelihood of being an individual discharge occurring at a DPH. Variables included in this model reflected patient clinical information and discharge characteristics, including Elixhauser score, ²³ gender, age, race, indicator variables for chronic conditions (congestive heart failure (CHF), depression, chronic obstructive pulmonary disease (COPD), hypertension, diabetes, and obesity), admission source and type, and Major Diagnostic Code (MDC). For the model not restricted to Medicaid patients, we included a categorical payer variable. We generated inverse probability weights for each outcome separately and assigned these to all patients in the OSHPD discharge records included in both samples. Inverse probability weights were used as weights in regression models and also as covariates.

We compared the propensity score–weighted and propensity score–unweighted DPH and Non‐DPH samples using linear regression to examine differences in continuous variables and chi‐squared tests to examine differences in categorical variables. This allowed us to assess whether the treatment (DPH) and control (non‐DPH) groups were balanced across various demographic and diagnostic characteristics in both the full sample and the Medicaid‐only samples (Table 2 and Table S2). We used the Stata margins command to report the predicted probability of continuous descriptive variables before and after weighing using the inverse probability weights.

TABLE 2.

Discharge‐level demographic and comorbidity characteristics by inpatient safety measure, full sample

	Severe sepsis mortality				Central line–associated bloodstream infections				Hospital‐acquired pressure ulcers				Venous thromboembolism
	Weighted		Unweighted		Weighted		Unweighted		Weighted		Unweighted		Weighted		Unweighted
	Non‐DPH	DPH	Non‐DPH	DPH	Non‐DPH	DPH	Non‐DPH	DPH†	Non‐DPH	DPH	Non‐DPH	DPH	Non‐DPH	DPH	Non‐DPH	DPH
Hospital level
Teaching hospital	0.09	0.62***	0.08	0.84***	0.01	0.47***	0.01	0.48***	0.09	0.70***	0.09	0.85***	0.10	0.99***	0.10	0.93***
Percent Medicare	0.36	0.32	0.38	0.20***	0.35	0.35	0.35	0.34	0.35	0.30	0.36	0.23***	0.36	0.25***	0.37	0.13***
Discharge level
Elixhauser Score	15.95	16.76	16.18	13.65***	4.15	4.09	4.17	4.07	9.00	8.46	9.18	7.34*	4.25	4.87	4.28	2.75***
CHF	0.30	0.31	0.31	0.22***	0.13	0.13	0.13	0.13	0.22	0.21	0.23	0.14***	0.09	0.08*	0.09	0.05***
Depression	0.11	0.11	0.12	0.10*	0.13	0.13	0.13	0.12	0.16	0.19	0.17	0.14***	0.09	0.09	0.09	0.05***
COPD	0.27	0.26	0.28	0.17***	0.2	0.20	0.20	0.19	0.28	0.26	0.29	0.18***	0.16	0.15	0.16	0.08***
Hypertension	0.57	0.55	0.59	0.48***	0.44	0.43	0.44	0.43	0.59	0.56	0.60	0.47***	0.52	0.47*	0.53	0.35***
Diabetes	0.37	0.34*	0.39	0.33***	0.22	0.21	0.22	0.22	0.30	0.30	0.31	0.24***	0.22	0.22*	0.23	0.20**
Obesity	0.13	0.10*	0.14	0.10*	0.13	0.13	0.13	0.13	0.13	0.12	0.14	0.08***	0.17	0.13*	0.17	0.13
Male	0.52	0.52	0.52	0.57***	0.36	0.36	0.36	0.37	0.49	0.48	0.48	0.55***	0.46	0.45	0.46	0.52
Severe sepsis mortality	0.24	0.28	0.24	0.25	0.01	0.02	0.01	0.02	0.04	0.04	0.04	0.03***	0.01	0.02**	0.01	0.02
Age
<35	0.07	0.14	0.05	0.11	0.27	0.27	0.27	0.27	0.10	0.13	0.09	0.16	0.11	0.17	0.09	0.23
35‐45	0.05	0.04	0.05	0.09	0.12	0.12	0.12	0.12	0.08	0.08	0.07	0.12	0.10	0.11	0.10	0.17
45‐55	0.11	0.10	0.11	0.19	0.12	0.12	0.12	0.13	0.14	0.14	0.13	0.20	0.16	0.18	0.16	0.23
55‐65	0.19	0.16	0.19	0.26	0.13	0.13	0.13	0.14	0.18	0.16	0.17	0.23	0.20	0.20	0.21	0.23
>65	0.57	0.56	0.60	0.36***	0.36	0.36	0.36	0.34***	0.51	0.49	0.53	0.30***	0.42	0.34***	0.44	0.14***
Race
White	0.52	0.51	0.53	0.38	0.52	0.53	0.52	0.51	0.57	0.57	0.58	0.47	0.6	0.61	0.62	0.26
Black	0.10	0.11	0.10	0.12	0.09	0.09	0.09	0.09	0.10	0.10	0.10	0.12	0.07	0.05	0.07	0.10
Native American/Eskimo/Aleut	0.24	0.25	0.23	0.35	0.27	0.26	0.26	0.30	0.21	0.22	0.21	0.28	0.22	0.23	0.21	0.52
Other	0.11	0.09	0.11	0.11	0.09	0.09	0.09	0.06	0.08	0.07	0.08	0.09	0.07	0.06	0.07	0.10
Unknown	0.03	0.04	0.03	0.04***	0.04	0.04	0.04	0.04***	0.03	0.03	0.03	0.04***	0.04	0.05	0.04	0.02***
Payer
Private	0.16	0.18	0.16	0.14	0.30	0.31	0.31	0.27	0.20	0.22	0.20	0.20	0.37	0.46	0.38	0.10
Medicare	0.61	0.58	0.63	0.39	0.39	0.38	0.39	0.37	0.56	0.52	0.57	0.36	0.43	0.33	0.44	0.13
Medicaid	0.17	0.18	0.16	0.34	0.22	0.22	0.22	0.25	0.16	0.15	0.15	0.28	0.12	0.12	0.10	0.45
Self‐pay	0.03	0.02	0.02	0.05	0.04	0.04	0.04	0.06	0.04	0.04	0.03	0.06	0.05	0.04	0.05	0.09
Indigent	0.01	0.01	0.01	0.05	0.02	0.02	0.02	0.03	0.02	0.03	0.02	0.07	0.02	0.03	0.01	0.20
Other	0.02	0.03	0.01	0.03***	0.02	0.02	0.02	0.02***	0.02	0.05	0.02	0.03***	0.02	0.03	0.02	0.03***

Models also controlled for major diagnostic categories (MDCs) and California counties. Source: Authors’ Analysis of California Office of Statewide Health Planning and Development Discharge Date 2009‐2014. *** P < .001, ** P < .01, *P < .05. For the sepsis measure, N = 67 791, and there were 348 hospitals included in the analysis. For CLABSI, N = 2 433 126 and there were 423 hospitals included in the model. For HAPU, N = 626 989 and there were 409 hospitals included in the model. For VTE, N = 461 660 and there were 364 hospitals included in the model.

Abbreviations: CHF, congestive heart failure; COPD, chronic obstructive pulmonary disease; DPH, Designated Public Hospital.

2.6. Outcomes

We used the available AHRQ Patient Safety Indicators (PSI) for CLABSI (PSI #07), HAPU (PSI #03), and VTE (PSI #12). The AHRQ PSIs have been used for hospital‐level surveillance, public reporting, and for hospital performance rewarding. ²⁴ For severe sepsis, where no AHRQ measure was available, our analysis sample was all patients with a severe sepsis diagnosis not present upon admission and our outcome was death during the inpatient stay. Outcome variable construction details are in Table S1.

2.7. Statistical analysis

We developed logistic regression models estimating likelihood of adverse outcomes and used the Stata margins command to calculate the predicted probability of each outcome. We estimated the predicted probability of sepsis mortality, HAPU, VTE, and CLABSI for DPH and non‐DPH discharges during the baseline and postimplementation periods. We employed cluster‐robust standard errors in the logistic models to account for hospital‐level variance. In all models, we controlled for hospital random effects.

We specified alternative models in order to test the suitability of our comparison group. These models interacted the group effect with all year dummy variables in order to control for group‐specific time trends that may have been occurring prior to the start of DSRIP. The parallel trends assumption requires that in the absence of DSRIP, the difference in outcomes between DPHs and Non‐DPHs is constant over time. ²⁵ We examined evidence of the parallel trends assumption in two ways. First, we compared two years of the preperiod, 2009 and 2010, and interacted hospital type (DPH) and year for evidence. We used a nonsignificant result as an indication of the parallel trends assumption. We also graphed the propensity score–weighted outcomes over time to visually assess parallel trends (Figure 1). ²⁶

FIGURE 1.

Adjusted Predicted Probabilities of Adverse Outcomes at Baseline and During the DSRIP Program for Designated Public Hospital Discharges and Non‐DPH Discharges. The x‐axis is the propensity score‐weighted rate of the adverse outcome measure, the y‐axis is the year. Abbreviations: DPH: Discharges from Designated Public Hospitals. Non‐DPH: Discharges from Non‐Designated Public Hospitals. CLABSI: Central‐Line Associated Bloodstream Infection; HAPU: Hospital‐Acquired Pressure Ulcers; VTE: Venous Thromboembolisms. DSRIP: Delivery System Reform Incentive Payment Program. Source: Authors' Analysis of California Office of Statewide Health Planning and Development Discharge Data, 2009‐2014. [Color figure can be viewed at wileyonlinelibrary.com]

For our main difference‐in‐difference (DD) estimators, we examined differences between the groups at four time periods—2010 vs 2011, 2010 vs 2011, 2010 vs 2013, and 2010 vs 2014—using coefficients for interaction terms for year and treatment group, and we verified significant findings in postestimation using Stata's lincom command to test significance of differences in time trends. The interaction between treatment group and time should only show significant coefficients after the intervention, allowing any observed effect to be attributed to the intervention and not some group‐specific time trend. This test draws upon the insight of the Granger causality test and is standard econometric practice in the difference‐in‐difference literature. ²⁵ , ²⁷ We used Stata version 16 for all analyses and calculated the predicted probabilities of adverse outcomes for ease of interpretation.

We included controls for discharge‐level risk for each outcome, including age, gender, race, and payer (as a proxy for socioeconomic status and access to care), as well as highly prevalent comorbidities such as CHF, depression, COPD, hypertension, diabetes, and obesity. We also included an individual Elixhauser Comorbidity Index score. ²³ We included controls for hospital‐level variables that might be associated with inpatient rates, including the percentage of patients with Medicare, a proxy for hospital payer mix, and an indicator for whether the hospital was a teaching hospital.

2.8. Sensitivity analyses

In addition to our inverse probability weight method, we estimated three additional models to demonstrate that our results were robust to model specification and not driven by differences between DPHs and non‐DPHs: (1) a model with an “intervention” group (hospitals that participated in DSRIP) and a control group of hospitals (hospitals that did not participate in DSRIP), matched at the hospital level; (2) a second hospital‐level matched model that excluded the UC hospitals; and (3) a propensity score hybrid model. The first model used all DSRIP‐participating hospitals compared with a sample of other hospitals in California matched on hospital characteristics using a combination of Gower's distance and exact matching. We matched on the following characteristics: license category (general acute care, acute psychiatric, psychiatric health facility, chemical dependency hospital), licensed emergency department level at the end of the year, case mix based on Medicare Severity Diagnosis Related Groups (MS‐DRG) and their associated weights created by OSHPD, ²⁸ number of nonpediatric beds, ratio of outpatient admissions to inpatient admissions, ratio of ICU beds to general acute care beds, trauma level, teaching hospital status, and principal service type (general medical/surgical, physical rehabilitation, long‐term care, orthopedic or pediatric orthopedic, psychiatric, developmentally disabled, chemical dependency, pediatric, and other) (Table S7). Table S8 illustrates the discharge‐level demographic and clinical characteristics at DPHs and non‐DPHs using this matching approach.

The second sensitivity analysis used an identical specification, but we excluded UC system hospitals from the DPH group to explore the effect of this distinct subgroup on DPH performance trends. While most DPHs are county‐owned and focus on serving a safety net population, the UC system hospitals are large academic medical centers with significant financial resources and larger proportion of privately insured patients than other hospitals in the DPH group. We excluded the UC hospitals after matching. Both matched models used a random‐effects specification to control for hospital‐level variance. Table S9 illustrates the discharge‐level demographic and clinical characteristics at DPHs and non‐DPHs using this matching approach.

Finally, we specified a propensity score hybrid model using the same propensity score matching approach as our main model. ²⁹ , ³⁰ , ³¹ The propensity score hybrid model decomposes within‐ and between‐cluster variance in order to limit the potential biases that can arise due to the violation of assumptions made by random‐effects models. We used this model to address concerns about the potential correlation of observed predictor variables and unobserved hospital‐level heterogeneity. Results for the propensity score hybrid model are in Table S15. A complete discussion of the propensity score hybrid model and all sensitivity analyses is included in the Appendix S1.

3. RESULTS

3.1. Discharge‐level characteristics

Unweighted and propensity score–weighted discharge‐level demographic and comorbidity characteristics for the full sample are presented in Table 2. Discharges from non‐DPHs had higher Elixhauser scores and higher predicted probabilities of comorbidities such as CHF, depression, COPD, hypertension, diabetes, and obesity. DPH discharges were more likely to be nonwhite and less likely to have private insurance. We found the same patterns in our Medicaid‐only sample (Table S2).

3.2. Changes in rates of adverse events in DPH and comparison hospitals over time

Predicted probabilities and difference‐in‐difference estimates are presented in Table 3a for the full sample and Table 3b for the Medicaid‐only sample. Full regression results are presented in Tables S5 and S6. With only two pre‐intervention time periods to observe (2009 and 2010), one of which served as a reference category (2009), we found a statistically significant interaction between treatment group (DPH) and year for CLABSI and VTE in the full sample and CLABSI in the Medicaid‐only model (Tables S3 and S4). We also examined interaction terms between year and treatment group in our other three sensitivity analysis models (Tables S11, S13, S15, and S17). Figure 1 illustrates the propensity score–weighted outcome measures from 2009 to 2014, which demonstrates weighted trends for all four measures.

TABLE 3.

Adjusted Predicted Probabilities, 95% Confidence Intervals, and Difference‐in‐Difference Estimates of Inpatient Adverse Outcomes at Baseline and during DSRIP, by Outcome Measure (a) Full Sample and (b) Medicaid‐Only Sample

(a)
Predicted probability (95% CIs)	Baseline	Year 1	Year 2	Year 3	During DSRIP	DD (2011 vs 2010) a	DD (2012 vs 2010) a	DD (2013 vs 2010) a	DD (2014 vs 2010) a
	2010	2011	2012	2013	2014
Severe sepsis mortality
DPH discharges	31.23% (27.04%, 35.41%)	29.31% (25.35%, 33.27%)	26.47% (22.74%, 30.2%)	25.34% (21.58%, 29.09%)	21.23% (18.14%, 24.31%)	−1.30% (−3.18%, 0.58%)	−2.90%** (−5.08%, −0.72%)	−3.05% (−6.50%, 0.40%)	−5.74%*** (−8.76%, −2.72%)
Non‐DPH discharges	26.41% (25.45%, 27.37%)	25.8% (24.8%, 26.8%)	24.56% (23.56%, 25.56%)	23.57% (22.6%, 24.54%)	22.15% (21.14%, 23.16%)
Central line–associated bloodstream infections (CLABSIs)
DPH discharges	1.6% (1.24%, 1.95%)	1.58% (1.22%, 1.95%)	1.6% (1.24%, 1.97%)	1.5% (1.15%, 1.86%)	1.39% (1.05%, 1.72%)	0.05% (−0.03%, 0.13%)	0.14%* (0.03%, 0.25%)	0.09% (−0.02%, 0.2%)	0.07% (−0.03%, 0.17%)
Non‐DPH discharges	1.66% (1.49%, 1.83%)	1.6% (1.43%, 1.76%)	1.52% (1.36%, 1.68%)	1.47% (1.32%, 1.63%)	1.38% (1.23%, 1.53%)
Hospital‐acquired pressure ulcers (HAPUs)
DPH discharges	4.84% (3.76%, 5.92%)	4.48% (3.63%, 5.34%)	4.29% (3.55%, 5.03%)	4.35% (3.62%, 5.08%)	3.47% (2.91%, 4.04%)	−0.22% (−0.66%, 0.23%)	−0.25% (−0.75%, 0.25%)	−0.10% (−0.61%, 0.41%)	−0.79%* (−1.52%, −0.05%)
Non‐DPH discharges	3.96% (3.74%, 4.18%)	3.82% (3.61%, 4.03%)	3.67% (3.46%, 3.88%)	3.57% (3.37%, 3.78%)	3.39% (3.19%, 3.58%)
Venous thromboembolism (VTE)
DPH discharges	2.44% (1.7%, 3.17%)	2.49% (1.76%, 3.23%)	2.32% (1.57%, 3.07%)	2.23% (1.48%, 2.97%)	1.88% (1.28%, 2.47%)	0.11% (−0.05%, 0.26%)	−0.04% (−0.27%, 0.2%)	−0.02% (−0.34%, 0.3%)	−0.32%* (−0.59%, −0.06%)
Non‐DPH discharges	1.63% (1.54%, 1.72%)	1.58% (1.49%, 1.67%)	1.54% (1.45%, 1.63%)	1.44% (1.35%, 1.52%)	1.39% (1.31%, 1.47%)

(b)
	Baseline	Year 1	Year 2	Year 3	Year 4	DD (2011 vs 2010) a	DD (2012 vs 2010) a	DD (2013 vs 2010) a	DD (2014 vs 2010) a
	2010	2011	2012	2013	2014
Severe sepsis mortality
DPH discharges	27.78% (24%, 31.55%)	26.19% (22.4%, 29.98%)	24.97% (20.72%, 29.23%)	21.85% (18%, 25.71%)	18.52% (15.23%, 21.8%)	−2.06% (−5.08%, 0.97%)	−1.76% (−5.81%, 2.28%)	−3.97% (−8.49%, 0.55%)	−5.18%* (−9.44%, −0.92%)
Non‐DPH discharges	24.98% (23.61%, 26.35%)	25.45% (23.96%, 26.94%)	23.94% (22.57%, 25.31%)	23.03% (21.81%, 24.25%)	20.9% (19.67%, 22.13%)
Central line–associated bloodstream infections (CLABSIs)
DPH discharges	0.92% (0.71%, 1.14%)	0.93% (0.69%, 1.16%)	0.95% (0.73%, 1.17%)	0.84% (0.62%, 1.06%)	0.74% (0.56%, 0.93%)	0.00% (−0.10%, 0.10%)	0.07% (−0.04%, 0.18%)	0.00% (−0.11%, 0.12%)	0.02% (−0.1%, 0.14%)
Non‐DPH discharges	0.94% (0.83%, 1.06%)	0.94% (0.83%, 1.06%)	0.9% (0.79%, 1.01%)	0.86% (0.75%, 0.96%)	0.74% (0.65%, 0.83%)
Hospital‐acquired pressure ulcers (HAPUs)
DPH discharges	4.71% (3.55%, 5.88%)	4.5% (3.52%, 5.49%)	4.1% (3.24%, 4.96%)	3.87% (2.98%, 4.75%)	2.85% (2.18%, 3.51%)	−0.12% (−0.6%, 0.36%)	−0.39% (−1.13%, 0.36%)	−0.40% (−0.89%, 0.1%)	−1.05%* (−1.9%, −0.2%)
Non‐DPH discharges	3.71% (3.39%, 4.02%)	3.62% (3.31%, 3.92%)	3.48% (3.18%, 3.78%)	3.26% (2.96%, 3.55%)	2.89% (2.63%, 3.15%)
Venous thromboembolism (VTE)
DPH discharges	2% (1.39%, 2.6%)	2% (1.47%, 2.53%)	1.86% (1.43%, 2.29%)	1.67% (1.08%, 2.26%)	1.4% (0.99%, 1.8%)	−0.06% (−0.4%, 0.28%)	−0.12% (−0.45%, 0.22%)	−0.13% (−0.45%, 0.19%)	−0.25% * (−0.49%, 0.01%)
Non‐DPH discharges	1.81% (1.66%, 1.95%)	1.87% (1.71%, 2.03%)	1.79% (1.64%, 1.93%)	1.61% (1.46%, 1.77%)	1.45% (1.33%, 1.58%)

(a) Full Sample: For the sepsis measure, N = 338 101 and there were 374 hospitals included in the analysis. For CLABSI, N = 10 828 081 and there were 436 hospitals included in the model. For HAPU, N = 3 972 574 and there were 442 hospitals included in the model. For VTE, N = 4 090 717 and there were 385 hospitals included in the model.

(b) Medicaid‐Only Sample: For the sepsis measure, N = 59 667 and there were 348 hospitals included in the analysis. For CLABSI, N = 2 433 126 and there were 423 hospitals included in the model. For HAPU, N = 626 989 and there were 409 hospitals included in the model. For VTE, N = 461 660 and there were 364 hospitals included in the model.

Abbreviations: DD, difference‐in‐difference estimator; DPH, Designated Public Hospital; DSRIP, Delivery System Reform Incentive Payment Program.

^a***P < .001; **P < .01; *P < .05.

Sepsis mortality for both DPH and non‐DPH discharges declined over the DSRIP program period (2010‐2014). The adjusted predicted probability of sepsis mortality for DPH discharges decreased from 31.23 percent in 2010 to 21.23 percent in 2014. The adjusted predicted probability of sepsis mortality for non‐DPH discharges decreased from 26.41 percent in 2010 to 22.15 percent in 2014. As noted earlier, we examined differences between the treatment groups at four time periods (2010 vs 2011, 2010 vs 2011, 2010 vs 2013, and 2010 vs 2014). For sepsis, the difference‐in‐difference estimator (DD) was only significant during two time periods, from 2010 to 2012 (DD: −2.90 percent, 95% CI: −5.08, −0.72 percent) and 2010 to 2014 (DD: −5.74, 95% CI: −8.76 percent, −2.72 percent). For the Medicaid‐only sample, the adjusted predicted probability of sepsis mortality for DPH discharges decreased from 27.78 percent in 2010 to 18.52 percent in 2014. The adjusted predicted probability of sepsis mortality for non‐DPH discharges in the Medicaid‐only sample decreased from 24.98 percent in 2010 to 20.9 percent in 2014. However, the DD was only significant comparing 2014 with 2010 (DD: −5.18 percent; 95% CI: −9.44 percent, −0.92 percent). These results were robust to our sensitivity analyses; in all models, the DD comparing 2014 with 2010 was significant, but only two models (our hospital match model and the hospital match model excluding the UCs) showed DDs that were significant at all time periods (Table S10a).

For CLABSI, DPH and non‐DPH discharges started at similar levels and declined to the same levels. Using the full sample, the adjusted predicted probability of developing a CLABSI for DPH discharges decreased from 1.60 percent in 2010 to 1.39 percent in 2014. For non‐DPH discharges, the adjusted predicted probability of developing a CLABSI decreased from 1.66 percent to 1.38 percent. With the exception of the DD estimator from 2010 vs 2012, the DDs at all the time points were not significant. For the Medicaid‐only sample, the adjusted predicted probability of developing a CLABSI for DPH discharges decreased from 0.92 percent to 0.74 percent from 2010 to 2014. The adjusted predicted probability of developing a CLABSI for the Medicaid‐only, non‐DPH discharges decreased from 0.94 percent in 2010 to 0.74 percent in 2014, demonstrating nearly identical improvement across both DPHs and Non‐DPHs. Our results were robust to our sensitivity analyses, although the propensity score hybrid model results and magnitude were higher and significant for three out of the four time periods (Table S10b).

The probability of developing a HAPU was slightly higher at baseline for DPH discharges than for non‐DPH discharges, but both groups ended at nearly the same rates by 2014. Using the full sample, the adjusted predicted probability of developing a HAPU for DPH discharges decreased from 4.84 percent in 2010 to 3.47 percent in 2014. For non‐DPH discharges, the adjusted predicted probability of developing a HAPU decreased from 3.96 percent to 3.39 percent during the same time period. For the Medicaid‐only sample, the adjusted predicted probability of developing a HAPU for DPH discharges decreased from 4.71 percent to 2.85 percent; for non‐DPH discharges, the adjusted predicted probability decreased from 3.71 percent to 2.89 percent. For both the full sample and Medicaid‐only sample, the only significant DD estimator compared 2014 with 2010 (Full sample: DD: −0.79 percent, 95% CI: −1.52 percent, −0.05 percent; Medicaid‐only sample: DD: −1.05 percent, 95% CI: −1.90 percent, −0.20 percent). The DD estimators comparing 2010 with other DSRIP years were not significant. These results were robust to our sensitivity analyses; in all models, the DD comparing 2014 with 2010 was significant, but only two models (our hospital match model and the hospital match model excluding the UCs) showed found significant DDs for three out of four time periods (Table S10c). The magnitude of the effect across all time periods was higher in our hospital match model excluding the UCs.

The adjusted probability of developing a VTE decreased slightly for both DPD and non‐DPH discharges from 2010 to 2014. Using the full sample, the adjusted predicted probability of developing a VTE for DPH discharges decreased from 2.44 percent in 2010 to 1.88 percent in 2014. For non‐DPH discharges, the adjusted predicted probability of developing VTE decreased from 1.63 percent to 1.39 percent during the same time period. For the Medicaid‐only sample, the adjusted predicted probability of developing a VTE for DPH discharges decreased from 2.00 percent to 1.4 percent from 2010 to 2014; for non‐DPH discharges, the adjusted predicted probability decreased from 1.81 percent to 1.45 percent. For both the full sample and Medicaid‐only sample, the only significant DD estimator compared the two groups from 2014 to 2010 (full sample: DD: −0.32 percent, 95% CI: −0.59 percent, −0.06 percent; Medicaid‐only sample: DD: −0.25 percent, 95% CI: −0.49 percent, −0.01 percent). Our results were robust to our sensitivity analyses (Table S10d).

4. DISCUSSION

Adverse outcomes in all four measures decreased over the study period for both DPH and non‐DPH discharges in both our full sample and our Medicaid‐only sample. Overall, we did not find that DPHs participating in DSRIP outperformed non‐DPHs across all four time periods and we did not find significant DDs across the four time periods. Our results were robust to multiple sensitivity analyses.

The inpatient adverse events included in the DSRIP program were and continue to be the focus of a variety of national and state initiatives that were in effect contemporaneously with DSRIP. It is possible that these other initiatives encouraged improvement in these measures in both the DPH and non‐DPH health systems. For example, CLABSIs have been a focus of the National Healthcare Safety Network since 2006, a nationwide initiative to track health care–acquired infections. ³² Starting in 2008, health systems no longer received higher DRG payments to patients with pressure ulcers that developed during hospitalization. ³³ Additionally, health systems may have begun preparing for rules that required reporting of sepsis outcomes and processes for patients discharged on or after October 1, 2015. ³⁴ Prior to DSRIP, CMS, the National Quality Forum, the Joint Commission, and AHRQ developed performance measures, incentive programs, public reporting initiatives, and other programs aimed at improving VTE prevention. ³⁵ These various nationwide initiatives likely drove quality improvement initiatives in hospitals across California to improve compliance and avoid reductions in reimbursements.

Our analyses have several limitations. Challenges of concurrent quality incentive programs such as Meaningful Use and other programs and unique conditions in the state of California may limit generalizability of the results. Incomplete reporting, misclassification of health care–associated infections, and lack of detection of conditions present on admission have all been found to be sources of error in the PSIs. ³⁶ Sepsis coding practices changed over this time period as a result of pressure to improve sepsis mortality, making this measure difficult to interpret. ³⁷ , ³⁸ Additionally, DPHs and non‐DPHs have fundamental differences in terms of characteristics such as teaching status and patient population, making the establishment of a balanced comparison group difficult. We created propensity scores to match on a variety of discharge‐level characteristics, including demographic and comorbidity characteristics. However, our discharge data lacked detailed information available in the electronic health record, which would have allowed us to better characterize severity of the condition and other complicating factors, meaning our comparison group may be significantly different in terms of important demographic and clinical factors. The significant differences in preintervention time trends in VTE and CLABSI indicate how this limitation affected our ability to evaluate DSRIP’s effectiveness. However, DD estimators that use propensity scores result in lower estimation bias and type II errors compared with specifications that attempt to adjust for differences in preintervention trends. ³⁹ Ongoing incentive programs at the national level also complicate isolating the effect of DSRIP projects. DPHs were participating in other CMS and non‐CMS projects relating to the outcomes measured in this study during the observation period, and therefore, improvement on patient safety measures cannot be attributed solely to DSRIP.

Despite study limitations, our analysis benefits from using baseline pre‐DSRIP data and several years of data during the DSRIP program. Using these multiple years of data allowed us to control for historical and concurrent trends, including other inpatient safety initiatives at the state and national level that could impact the outcomes we studied at both the DPHs and comparison groups.

In conclusion, we found that patients discharged from both DPHs and non‐DPHs experienced important reductions in adverse outcomes such as sepsis mortality, HAPUs, CLABSIs, and VTEs over the study time period. However, we did not find that DPHs participating in DSRIP outperformed non‐DPHs during this time.

Supporting information

Author Matrix

Appendix S1

Keller MS, Chen X, Godwin J, Needleman J, Pourat N. Evaluating inpatient adverse outcomes under California's Delivery System Reform Incentive Payment Program. Health Serv Res.2021;56:36–48. 10.1111/1475-6773.13550