Abstract
Background
Surgical checklists are increasingly used to improve compliance with evidence-based processes in the perioperative period. Although enthusiasm exists for using checklists to improve outcomes, recent studies have questioned their effectiveness in large populations.
Objective
We sought to examine the association of Keystone Surgery, a statewide implementation of an evidence-based checklist and Comprehensive Unit Based Safety Program (CUSP), on surgical outcomes and healthcare costs.
Methods
We performed a study using national Medicare claims data for patients undergoing general and vascular surgery (n = 1,002,241) from 2006-2011. A difference-in-differences approach was used to evaluate whether implementation was associated with improved surgical outcomes and decreased costs when compared to a national cohort of non-participating hospitals. Propensity score matching was used to select 10 control hospitals for each participating hospital. Costs were assessed using price-standardized 30-day Medicare payments for acute hospitalizations, readmissions and high-cost outliers.
Results
Keystone Surgery implementation in participating centers (N=95 hospitals) was not associated with improved outcomes. Difference-in-differences analysis accounting for trends in non-participating hospitals (N=950 hospitals) revealed no differences in adjusted rates of 30-day mortality [Relative Risk (RR):1.03, 95%CI:0.97-1.10]; any complication (RR:1.03, 95%CI:0.99-1.07); reoperations (RR:0.89, 95%CI:0.56-1.22); or readmissions (RR:1.01, 95%CI:0.97-1.05). Medicare payments for the index admission increased following implementation ($516 average increase in payments, 95%CI:$210 to $823 increase), as did readmission payments ($564 increase, 95%CI:$89 to $1,040 increase). High-outlier payments ($965 increase, 95%CI:$974decrease to $2904 increase) did not change.
Conclusions
Implementation of Keystone Surgery in Michigan was not associated with improved outcomes or decreased costs in Medicare patients.
Keywords: administrative data, cost analysis, effectiveness, observational studies, outcomes research, quality improvement, surgery
INTRODUCTION
Checklists have been reported to substantially decrease rates of adverse outcomes in surgical patients in controlled settings. Initial studies of safety checklist use in the operating room reported a 36% reduction in surgical complications (from 11% to 7%) and a 47% reduction in mortality (from 1.5% to 0.8%).(1) Numerous studies (2-4) and systematic reviews(5-7) have since corroborated these findings. Given the potential benefits of use, and the perceived ease of implementation, surgical safety checklists have since been adopted in more than 1500 hospitals around the world, and have been nationally implemented in multiple countries.(8)
Despite this enthusiasm, the real world effectiveness of surgical checklists when implemented across broad areas and large populations remains unclear. A recent study of surgical checklist use in Ontario, Canada failed to show a significant improvement in surgical outcomes following mandatory adoption throughout the province.(9) However, this study has been criticized for a short follow-up interval, low event rates, and lack of a unified and comprehensive program for implementation.(10, 11) Moreover, no studies have directly evaluated the influence of surgical checklists on healthcare utilization or costs. Given recent concerns regarding the effectiveness of checklist-based quality improvement programs in surgery, a better understanding of their financial impact will be critical for healthcare leaders considering broad surgical checklist implementation.
In this study, we sought to investigate the impact of a checklist-based intervention, “Keystone Surgery,” on surgical outcomes, utilization, and costs in a large, high-risk population. The Keystone Surgery program was a surgical checklist-based intervention implemented within specialty-specific surgical teams at participating Michigan hospitals between April 2008 and January 2010. The implementation process was standardized, and supported by a comprehensive program to improve safety culture. To evaluate Keystone Surgery, we used national Medicare data to examine surgical outcomes and Medicare payments in patients undergoing surgery at participating Michigan hospitals, compared to patients undergoing surgery at national control hospitals not implementing the program. To create a similar cohort of control hospitals, we used propensity score matching.
METHODS
Data Source and Study Population
This study utilized claims data from the Center for Medicare and Medicaid Services (CMS) for the years 2006 to 2011. The Medicare Provider Analysis and Review (MEDPAR) file, which contains hospital discharge records for all fee-for-service acute care hospitalizations of Medicare beneficiaries not enrolled in managed care plans, was used to create the main dataset for analysis, and the Medicare Denominator file was used to determine vital status of patients 30 days after surgery. This study was approved by the Institutional Review Board at the University of Michigan.
We used appropriate procedure codes from the International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) to identify all patients between 65 and 99 years old who underwent one of eleven general and vascular surgery procedures: esophagectomy, pancreatic resection, colon resection, gastrectomy, liver resection, ventral hernia repair, cholecystectomy, appendectomy, abdominal aortic aneurysm repair, lower extremity bypass, and carotid endarterectomy (Supplemental Digital Content 1). These procedures were selected because they best represent general and vascular surgery, the two surgical specialties most impacted by the Keystone Surgery program. Moreover, within each specialty, we chose a spectrum of operations from common, low-risk procedures (e.g., cholecystectomy and appendectomy), to rare, high-risk procedures (e.g., pancreatectomy). Including this range of procedures allows us to explore the potential impact across different types of procedures. To minimize confounding and increase cohort homogeneity, patients who underwent abdominal aortic aneurysm repair were excluded if they had accompanying codes suggesting presence of a rupture, thoracoabdominal aneurysm, or dissection (1292 patients, < 2% of patients undergoing aneurysm repair). We also excluded patients who underwent simultaneous procedures (e.g., colon and liver resection), which accounted for less than 1% of all cases.
Keystone Surgery Program
The Keystone Surgery Program was a prospective cohort intervention implemented within specialty-specific surgical teams at participating Michigan Health & Hospital Association hospitals (N=95 hospitals), with a goal of improving surgical care throughout the state.(12) Implementation within these hospitals occurred over a two-year period using a stepped-wedge design (different hospitals and teams completed implementation at different times). The majority of hospitals implementing the program did so on May 1st, 2008. Two additional hospitals implemented the program on June 1st 2009, one on December 1st, 2009, and one on January 1st, 2010.
Like the Keystone ICU Patient Safety program,(13, 14) the Keystone Surgery program utilized two main components: a novel model to translate evidence into practice,(15) and the Comprehensive Unit-based Safety Program (CUSP) to improve safety culture.(16) The evidence-based practice component included a surgical checklist tool focused on six CMS Surgical Care Improvement Program process measures: appropriate prophylactic antibiotic use (selection, timing, and discontinuation), appropriate hair removal, maintenance of perioperative normothermia, and glucose control.(17-20) Participating operative teams were encouraged to use the surgical checklist during briefings and debriefings surrounding every procedure, to monitor use, and to adapt the tool to local needs. As such, the specific surgical checklist used at each institution was distinct, however the common domains of the SCIP measures were included in each. Examples of checklist questions would include “Has antibiotic prophylaxis been given within the last sixty minutes?”, and “What is the anticipated blood loss of the case?”. The CUSP is an iterative 5-step process, described elsewhere, that has been previously validated to improve teamwork and safety culture.(16, 21) This program was uniformly used within the Keystone Surgery program to support initial implementation and ongoing participation at participating hospitals.
For this analysis, Michigan hospitals participating in the Keystone Surgery program were identified, and the date of implementation at each hospital was determined. For Keystone hospitals, patients undergoing surgery prior to that hospital’s date of implementation were considered “pre-implementation,” while patients undergoing surgery after that date were considered “post-implementation.” Because the vast majority of patients underwent surgery at hospitals implementing the program on May 1st, 2008, patients undergoing surgery at Non-Keystone hospitals were considered “post-implementation” if they received surgery after May 1st, 2008.
Primary Outcomes
This analysis evaluated the influence of Keystone Surgery implementation on six important clinical outcomes: mortality, any complication, serious complications, reoperation, readmission, and length of stay. Mortality was defined as the rate of death within 30-days of the index procedure. Complications were defined using a list of primary and secondary ICD-9-CM diagnostic and procedure codes from the index hospitalization (Supplemental Digital Content 2).(22, 23) A complication was considered serious if the hospitalization included an extended length of stay (≥75th percentile for each procedure).(24, 25) Reoperations were identified using ICD-9-CM procedure codes indicating secondary procedures during the index hospitalization (Supplemental Digital Content 3).(26) Readmission was defined as admission (all causes included) to any hospital within 30-days after discharge from the index procedure.(27) Length of stay was identified using the time from date of admission for the index procedure to date of discharge.
Medicare Payments
We also evaluated the influence of Keystone Surgery implementation on Medicare payments. Though payments are not a direct measure of hospital costs, they represent a metric of resource utilization that can be compared across hospitals.(28, 29) In addition, these payments represent the perspective of Medicare, who is responsible for a majority of the excess costs associated with poor quality care.(30) In this analysis, we used Medicare facility payments from the MedPAR file, which include all payments related to the index hospitalization, readmissions, and high-cost outliers. Given that Medicare payments vary across hospitals (e.g., payments may include those for disproportionate share of low income patients or graduate medical education) and geographic regions (e.g., payments are indexed to reflect differences in wages), we used previously described methods to “price-adjust” Medicare payments.(29, 31)
Statistical Analysis
The purpose of this analysis was to evaluate whether implementation of the Keystone Surgery program in participating Michigan hospitals was associated with improved outcomes and decreased costs for Medicare patients, compared to similar hospitals that did not participate in the program. To do this, we used a difference-in-differences approach, which is an econometric technique commonly used to evaluate the impact of policy changes.(32-34) This approach utilizes a control group to isolate changes in outcomes associated with an intervention (e.g., implementation of Keystone Surgery) apart from changes observed in controls. Our control group included a national cohort of hospitals not participating in the program, as they were likely exposed to all other factors driving improved outcomes during the time period except implementation of Keystone Surgery.
We used propensity score matching to select 10 similar control hospitals for each Keystone hospital. We used multivariable logistic regression to estimate propensity scores for hospital matching, with Keystone Surgery participation as the dependent variable. Hospitals were matched based a propensity score that included annual surgical volume, hospital teaching status, for profit-status, and the availability of hospital technology(35). Hospital bed size and total surgical admissions were not included due to strong collinearity with surgical volume. Hospitals were matched on the logit of the propensity score using a caliper width of 0.1 times the standard deviation of the logit of the propensity score, using the nearest neighbor method without replacement. To check for covariate imbalance before and after matching, we calculated t-tests for equality of means, the standardized percentage bias, and the pseudo-R2, in order to achieve the greatest percentage reduction in absolute bias. A 1:10 match was used because it was empirically determined to result in the greatest bias reduction.
To perform the difference-in-differences analysis, we used the appropriate multivariable regression model (i.e. logistic regression for dichotomous clinical outcomes or linear regression for continuous Medicare payments) to evaluate the relationship between the dependent variable of interest and implementation of the Keystone Surgery program. The difference-in-differences model included a variable indicating whether the patient had surgery at a participating hospital (Keystone vs. control), a variable indicating whether the surgery occurred before or after Keystone Surgery implementation (pre-post), and the interaction of these two variables (Keystone*pre-post). The coefficient from this interaction term can be interpreted as the independent relationship between Keystone Surgery implementation and the outcome of interest.(33, 36, 37) In all models, we included a continuous year of surgery variable to account for linear time trends during the study period. We also adjusted for age, sex, race, procedure type, and coexisting conditions. Coexisting conditions were identified using the Elixhauser method, which uses ICD-9-CM codes to classify secondary diagnoses into 30 different comorbid conditions.(38, 39)
The main clinical findings of our analyses are presented as relative risks instead of odds ratios, as the latter are not an accurate representation of the risk ratio for common outcomes. (40) We used logistic regression models to estimate the marginal effects, which were then used to calculate relative risks. For all statistical tests, 95% confidence intervals were calculated using robust variance estimates, p-values were two-tailed, and alpha was set at 0.05. All analyses were conducted using STATA version 12.0 (StataCorp, College Station, TX).
RESULTS
A total of 1,002,241 Medicare patients underwent general or vascular surgery during the study period. Details of patient and operative characteristics are shown in Table 1. The average age of patients was 72.7 years old, 50.5% were female, and 12.4% were non-white race. A majority of the procedures performed were general surgery operations (73.7%). The most common general surgery operations were colon resection, cholecystectomy, and ventral hernia repair, while the most common vascular surgery operation was carotid endarterectomy. Although there were multiple statistically significant differences between the Keystone and control cohorts, there were few clinically significant differences (Table 1).
Table 1.
Characteristics of patients undergoing surgery at hospitals participating in the Keystone Surgery program, compared to a national cohort of control hospitals, during the years 2006 to 2011 (values represent percentages unless otherwise noted; SD: standard deviation).
| Patient Characteristics | Patients at Keystone Surgery Hospitals N= 91,317 |
Patients at Control Hospitals N= 910,924 |
|---|---|---|
| Patient Age, mean (SD) | 72.5 (10.8) | 72.7 (10.7)* |
| Female | 50.2 | 50.5 |
| Non-white race | 11.8 | 12.4* |
| General surgery cases | ||
| All general surgery | 71.4 | 73.7* |
| Esophagectomy | 1.6 | 1.3* |
| Pancreatic resection | 0.8 | 1.0* |
| Colon resection | 23.4 | 22.9* |
| Gastrectomy | 1.8 | 2.0* |
| Liver resection | 1.2 | 1.7* |
| Ventral hernia repair | 14.0 | 13.5* |
| Cholecystectomy | 21.7 | 24.3* |
| Appendectomy | 6.9 | 7.0 |
| Vascular surgery cases | ||
| All vascular surgery | 28.6 | 26.3* |
| Abdominal aortic aneurysm repair | 6.3 | 6.3 |
| Lower extremity bypass | 7.0 | 6.9 |
| Carotid endarterectomy | 15.3 | 13.1* |
| Comorbid diseases | ||
| Hypertension | 58.2 | 57.0* |
| Diabetes w/o chronic complications | 19.8 | 19.0* |
| Chronic pulmonary disease | 20.0 | 18.2* |
| Peripheral vascular disease | 11.4 | 10.7* |
| Hypothyroidism | 10.1 | 8.9* |
| Congestive heart failure | 8.9 | 7.8* |
| Chronic anemia | 2.1 | 1.5* |
| Renal failure | 8.8 | 8.6 |
| Disseminated cancer | 6.0 | 6.6* |
| Depression | 4.8 | 4.5* |
| Neurological disorder | 3.5 | 3.5 |
| Coagulopathy | 2.8 | 3.0* |
| Liver disease | 1.8 | 2.2* |
| Pulmonary vascular disease | 1.2 | 1.5* |
| Psychoses | 1.9 | 1.8 |
Statistically significant difference (P<.05) between Keystone Surgery & Control hospitals.
Table 2 displays the characteristics of the 1045 study hospitals. Overall, the Keystone (N=95) and control (N=950) cohorts were well matched, as there were no significant differences between the groups outside of geographic location. A majority of hospitals were non-profit (90.5%), performed <175 cases per year (67.7%), and had <200 beds (66.3%). Only 42.7% were considered high-technology, and less than 10% were teaching hospitals (Table 2).
TABLE 2.
Characteristics of hospitals participating in the Keystone Surgery program, compared to a national cohort of control hospitals, during the years 2006-2010.
| Hospital Characteristics | Keystone Surgery Hospitals N=95 |
Control Hospitals N=950 |
|---|---|---|
| Surgical volume (annual cases) | ||
| <175 | 67.4 | 67.7 |
| 175-374 | 18.9 | 17.9 |
| 375-574 | 9.5 | 10.1 |
| ≥ 575 | 4.2 | 4.3 |
| Teaching hospital | 8.4 | 8.4 |
| Technology hospital | 43.2 | 42.7 |
| Geographic region | ||
| Northeast | 0.0 | 17.9* |
| Midwest | 100.0 | 33.1* |
| South | 0.0 | 30.7* |
| West | 0.0 | 18.3* |
| Bed size | ||
| <200 | 66.3 | 66.3 |
| 200-349 | 15.8 | 15.6 |
| 350-499 | 9.5 | 10.1 |
| ≥ 500 | 8.4 | 8.0 |
| Baseline outcomes (first year) | ||
| Mortality | 4.7 | 4.6 |
| Complications | 17.4 | 17.8 |
| Serious complications | 8.3 | 8.4 |
| Reoperation | 0.6 | 0.5 |
| Readmissions | 13.0 | 13.7 |
| Other characteristics | ||
| Nurse ratio | 8.6 | 7.7 |
| % Medicaid days | 18.0 | 17.7 |
| Profit status | ||
| For-profit | 1.1 | 1.1 |
| Non-profit | 90.5 | 90.5 |
| Other | 8.4 | 8.4 |
| Licensed Beds | 219 | 247 |
| Total Admissions | 9,277 | 9,582 |
| Full-Time Equivalent Employees | 1,301 | 1,230 |
| Total Annual Surgical Operations | 7,826 | 7,312 |
| Number of Operating Rooms | 8.5 | 10.2 |
Statistically significant difference (P<.05) between Keystone Surgery & Control hospitals.
Risk adjusted rates of adverse outcome at Keystone hospitals are shown in Table 3. In Keystone hospitals, rates of 30-day mortality (4.7% before implementation vs. 4.5% after, p<0.001), any complication (18.4% vs. 20.3%, p<0.001), serious complications (8.8% vs. 9.1%, p<0.001), reoperations (0.6% vs. 0.5%, p=0.51), readmissions (13.1% vs.13.3%, p<0.001), did not decrease following implementation of the program. Difference-in-differences analysis accounting for trends in non-participating hospitals revealed no differences in adjusted rates of 30-day mortality [Relative Risk (RR): 1.03, 95% CI: 0.97-1.10]; any complication (RR: 1.03, 95% CI: 0.99-1.07); reoperations (RR: 0.89, 95% CI: 0.56-1.22); and readmissions (RR: 1.01, 95% CI: 0.97-1.05), and revealed a slightly increased risk of serious complications (RR: 1.06, 95% CI: 1.01-1.11). Mean length of stay (7.2 days vs 6.6 days, p<.001) did decrease slightly following implementation. However when taking into account the secular trends using the difference-in-differences technique this was not significant; the relative change in length of stay after Keystone implementation versus control hospitals was .17 days (95% CI -.06 to .41).
Table 3.
Adjusted rates and relative risks of adverse outcomes, at hospitals participating in Keystone Surgery, and in the entire cohort, after Keystone Surgery implementation compared to before (values are percentage unless otherwise noted; CI: confidence interval)
| Adverse Outcome | Hospitals Participating in Keystone Surgery | Entire Cohort | ||
|---|---|---|---|---|
| Before enrollment in Keystone Surgery* N = 37,393 |
After enrollment in Keystone Surgery* N = 53,924 |
Relative Risk of adverse outcome after enrollment vs. before (95% CI)** N = 91,317 |
Relative Risk of adverse outcomes after enrollment vs. control hospitals (95% CI)*** N = 1,002,241 |
|
| Mortality | ||||
| Overall | 4.66 | 4.51 | 0.97 (0.81, 1.13) | 1.03 (0.97, 1.10) |
| General | 5.38 | 5.17 | 0.95 (0.77, 1.12) | 1.02 (0.95, 1.09) |
| Vascular | 2.39 | 2.28 | 1.16 (0.87, 1.45) | 1.10 (0.94, 1.26) |
| Complications | ||||
| Overall | 18.35 | 20.33 | 1.06 (0.99, 1.13) | 1.03 (0.99, 1.07) |
| General | 20.19 | 22.45 | 1.07 (1, 1.14) | 1.03 (0.98, 1.07) |
| Vascular | 12.52 | 13.23 | 0.99 (0.85, 1.13) | 1.05 (0.98, 1.13) |
| Serious complications | ||||
| Overall | 8.79 | 9.05 | 1.02 (0.92, 1.12) | 1.06 (1.01, 1.11) |
| General | 9.56 | 9.83 | 1.03 (0.93, 1.13) | 1.05 (0.99, 1.10) |
| Vascular | 7.17 | 7.25 | 0.96 (0.74, 1.19) | 1.10 (1.00, 1.19) |
| Reoperations | ||||
| Overall | 0.59 | 0.54 | 0.88 (0.34, 1.42) | 0.89 (0.56, 1.22) |
| General | 0.78 | 0.69 | 0.89 (0.32, 1.46) | 0.89 (0.56, 1.21) |
| Vascular | 0.15 | 0.13 | 0.75 (0.21, 2.43) | 0.97 (0.09, 1.86) |
| Readmissions | ||||
| Overall | 13.10 | 13.34 | 1.03 (0.96, 1.11) | 1.01 (0.97, 1.05) |
| General | 13.57 | 13.81 | 1.01 (0.91, 1.1) | 0.99 (0.95, 1.04) |
| Vascular | 11.77 | 11.96 | 1.11 (0.98, 1.24) | 1.04 (0.99, 1.09) |
Adjusted for year of surgery.
Adjusted for year of surgery, patient characteristics, procedure type, and coexisting conditions.
The independent impact of Keystone Surgery was ascertained using a differences-in-differences analysis, which compared adverse outcomes after vs. before enrollment after controlling for trends towards improved outcomes in control hospitals. These models also adjusted for year of surgery, patient characteristics, procedure type, and coexisting conditions.
The influence of Keystone Surgery implementation on Medicare payments is shown in Table 4. In Keystone hospitals, there were no significant differences in total Medicare payments, or payments for index admissions, readmissions, or high-outliers following implementation. After accounting for trends in non-participating hospitals, Medicare total payments ($516 average increase, 95% CI: $210 to $823 increase, p<0.001), index admission payments ($439 average increase, 95% CI: $168 to $711 increase, p=0.002) and payments for readmission ($564 average increase, 95% CI: $ 62 to $1,216 increase, p=0.02) were higher in participating hospitals compared to controls, while payments for outliers ($965 average increase, 95% CI: $1974 decrease to $2904 increase, p=0.44) did not change. As shown in Table 4, the increases in total payments and index admission payments likely resulted from increased payments for vascular surgery procedures during the study period.
Table 4.
Medicare facility payments at hospitals participating in Keystone Surgery, and in the entire cohort, after Keystone Surgery implementation compared to before (CI: confidence interval).
| Payment as Outcome in 2010 dollars | Hospitals Participating in Keystone Surgery | Entire Cohort | ||
|---|---|---|---|---|
|
| ||||
| Before enrollment in Keystone Surgery* N = 37,393 |
After enrollment in Keystone Surgery* N = 53,924 |
Difference in Medicare payments after enrollment vs. before (95% CI)** N = 91,317 |
Difference in Medicare payments after enrollment vs. control hospitals (95% CI)*** N = 1,002,241 |
|
|
| ||||
| Total payments | ||||
| Overall | $17,817 | $15,119 | $846 ($155, $1,538) | $516 ($210, $823) |
| General | $19,107 | $16,185 | $813 ($151, $1,476) | $566 ($175, $958) |
| Vascular | $14,758 | $12,367 | $879 ($-414, $2,173) | $378 ($109, $646) |
|
| ||||
| Payments for index admission | ||||
| Overall | $16,159 | $13,687 | $689 ($83, $1,296) | $439 ($168, $711) |
| General | $17,387 | $14,689 | $649 ($61, $1,237) | $476 ($123, $830) |
| Vascular | $13,243 | $11,098 | $780 ($-337, $1,897) | $343 ($130, $556) |
|
| ||||
| Payments for readmissions | ||||
| Overall | ||||
| % with payment | 14.7 | 14.2 | ||
| Payment when present | $11,277 | $10,072 | $268 ($-914, $1,450) | $564 ($89, $1,040) |
| General | ||||
| % with payment | 15.3 | 14.7 | ||
| Payment when present | $11,223 | $10,158 | $351 ($-991, $1,692) | $624 ($64, $1,184) |
| Vascular | ||||
| % with payment | 13.3 | 12.9 | ||
| Payment when present | $11,426 | $9,821 | $-118 ($-2,334, $2,098) | $250 ($-657, $1,158) |
|
| ||||
| Payments for outliers | ||||
| Overall | ||||
| % with payment | 4.8 | 4.7 | ||
| Payment when present | $20,328 | $18,700 | $-4,638 ($-9,825, $550) | $965 ($-974, $2,904) |
| General | ||||
| % with payment | 5.4 | 4.7 | ||
| Payment when present | $21,466 | $20,221 | $-4,322 ($-10,195, $1,551) | $820 ($-1,895, $3,535) |
| Vascular | ||||
| % with payment | 3.5 | 4.4 | ||
| Payment when present | $16,179 | $14,506 | $-3,406 ($-7,756, $944) | $1,188 ($-1,186, $3,563) |
Adjusted for year of surgery.
Adjusted for year of surgery, patient characteristics, procedure type, and coexisting conditions.
The independent impact of Keystone Surgery was ascertained using a differences-in-differences analysis, which compared adverse outcomes after vs. before enrollment after controlling for trends towards improved outcomes in control hospitals. These models also adjusted foryear of surgery, patient characteristics, procedure type, and coexisting conditions
Table 5 demonstrates the adverse outcomes as well as change in payment for each operation individually, to further clarify whether specific procedures types were impacted differently by the Keystone program. The findings were similar for each procedure, with no evidence of improved outcomes or reduced payments following Keystone implementation. The relative risk for mortality (RR 1.31, 95% CI 1.09-1.53), and the overall length of stay (.46 day increase, 95% CI .05-.87) increased slightly for appendectomy, and the length of stay for carotid endarterectomy increased by .17 days (95% CI .04-.31).
Table 5.
Relative risks of adverse outcomes and change in total payment by procedure type, at hospitals participating in Keystone Surgery, vs non-participating controls (CI: confidence interval)
| Procedure: | Relative Risk of Adverse Outcome (95% CI) | Change in Total payment (95% CI) | ||||
|---|---|---|---|---|---|---|
| Mortality | Complications | Serious complications | Reoperations | Readmissions | ||
| Esophagectomy | 1.02 (0.65, 1.4) | 1.15 (0.98, 1.31) | 1.07 (0.45, 1.69) | 0.7 (0.2, 2.48) | 0.81 (0.41, 1.21) | $203 ($-5,933, $6,338) |
| Pancreatic resection | 1.24 (0.84, 1.65) | 0.83 (0.54, 1.13) | 0.78 (0.29, 1.28) | 0.61 (0.12, 3.04) | 1.09 (0.88, 1.3) | $-1447 ($-4,237, $1,344) |
| Colon resection | 1 (0.9, 1.1) | 1.04 (0.99, 1.09) | 1.05 (0.97, 1.12) | 1.1 (0.81, 1.49) | 0.96 (0.89, 1.02) | 557 ($-68, $1,182) |
| Gastrectomy | 1.13 (0.9, 1.1) | 1.09 (0.98, 1.2) | 1.11 (0.83, 1.38) | 0.58 (0.22, 1.51) | 0.88 (0.64, 1.11) | $212 ($-1,926, $2,350) |
| Liver resection | 0.81 (0.07, 1.56) | 1.03 (0.6, 1.46) | 1.15 (0.77, 1.53) | 0.73 (0.04, 12.47) | 1.11 (0.87, 1.35) | $1333 ($-1,797, $4,463) |
| Incisional hernia repair | 1.08 (0.92, 1.24) | 1.07 (1.00, 1.15) | 1.12 (1.02, 1.21) | 1 (0.59, 1.7) | 1.03 (0.94, 1.12) | $599 ($23, $1,175) |
| Cholecystectomy | 0.91 (0.74, 1.07) | 0.98 (0.93, 1.02) | 1 (0.93, 1.08) | 0.57 (0.33, 0.98) | 1.03 (0.98, 1.09) | $346 ($-29, $720) |
| Appendectomy | 1.31 (1.09, 1.53) | 0.98 (0.89, 1.08) | 1.02 (0.89, 1.14) | 0.89 (0.4, 1.98) | 0.97 (0.8, 1.14) | $916 ($162, $1,669) |
| Abdominal aortic aneurysm repair | 1.26 (1.00, 1.52) | 1.02 (0.89, 1.15) | 0.97 (0.77, 1.16) | 1.19 (0.48, 2.97) | 0.99 (0.84, 1.13) | $347 ($-437, $1,130) |
| Lower extremity bypass | 1.03 (0.80, 1.26) | 1.06 (0.96, 1.16) | 1.16 (1.02, 1.31) | 0.91 (0.31, 2.67) | 1.02 (0.93, 1.11) | $689 ($153, $1,225) |
| Carotid endarterectomy | 1.03 (0.74, 1.32) | 1.07 (0.92, 1.21) | 1.1 (0.96, 1.25) | * | 1.09 (0.98, 1.20) | $177 ($-59, $412) |
Insufficient sample size to run regression model for this outcome measure
DISCUSSION
In this study, we found that implementation of a checklist-based intervention across Michigan was not associated with improved surgical outcomes (mortality, complications, or serious complications) in Medicare patients. When stratifying by type of procedure, including a spectrum of low- and high-risk operations, there was still no evidence of improvement in outcomes. In addition, we found no evidence that the program decreased resource utilization (reoperations or readmissions) or costs (Medicare payments). When compared to a national cohort of similar hospitals not participating in the program, these findings persisted.
Studies evaluating the effectiveness of surgical checklists are mixed. While numerous studies have reported significant improvements in surgical outcomes and safety culture following surgical checklist implementation,(1, 2, 4-6, 41) the majority of these studies have occurred in small populations or single institutions with controlled environments, and did not include a control group. On the other hand, multiple recent population-based studies of real world checklist implementation in surgical patients have failed to report significant associations with improved outcomes,(9, 42) but have been criticized due to methodological concerns. The evaluation of mandatory checklist adoption in Ontario, Canada by Urbach and colleagues,(9) for example, only evaluated outcomes 3 months following checklist implementation and included a substantial proportion of low-risk outpatient procedures. Moreover, though checklist-adoption was mandatory, the implementation strategies used to support adoption were variable.(10, 11)
The methods used in this study overcome many of the limitations of prior work. First, rather than include low-risk and outpatient procedures only, we evaluated a cohort comprised of both low- and high-risk patients. Second, we incorporated a much longer follow-up period, as we assessed outcomes up to 3 years following implementation. Third, we evaluated a program that was supported by a standardized and comprehensive implementation strategy, unlike the variable implementation support seen in Ontario. And finally, to limit confounding, we included a cohort of similar, non-participating hospitals as a control group. Given these additional strengths, we believe the findings of this study demonstrate strong evidence that surgical checklist implementation in the real world, even when supported by a robust and comprehensive implementation strategy, may have a limited impact on surgical outcomes.
The lack of an association between Keystone Surgery implementation and reduced costs is a new and interesting finding. To our knowledge, no studies have directly evaluated the impact of surgical checklists on hospitals costs or Medicare payments. A previous economic evaluation of the Keystone ICU project reported that $3,375 was saved with each medical infection averted, and estimated that more than $50,000 in additional health care costs could ultimately be saved by the program.(43) Studies estimating potential savings following use of surgical checklists, such as those by Semel and colleagues,(44) assume program effectiveness will be similar to the results reported in the Safe Surgery Saves Lives Study,(1) which has not been consistently replicated across all studies of surgical checklist implementation. Although estimated cost savings in their analysis were robust to variations in the cost of implementation, the authors did not consider the scenario presented here: a comprehensive implementation strategy was used to broadly implement a checklist that did not improve outcomes. In other words, substantial resources (financial and otherwise) were used to implement the program, but the intervention did not appear to generate cost savings.
This study has several limitations. First, given this study only includes Medicare patients, it may not generalizable to patients younger than 65 years old. Second, the use of administrative data may limit robust risk-adjustment, due to miscoding and imprecision, and the lack of granular clinical detail.(45, 46) Despite these limitations, Medicare data was purposely chosen for this study for multiple reasons. First, Medicare patients comprise a substantial number of all patients undergoing the operations in question, and represent a cohort with increased risks of morbidity and mortality for whom these results may be most applicable. Second, the use of Medicare data allows for a novel evaluation of healthcare costs associated with surgical checklist implementation through an analysis of Medicare payments, which has not been done previously. And most importantly, the use of Medicare data allows for evaluation of both participating and non-participating hospitals, which limits concern for confounding from competing quality improvement programs or organizations. A third limitation of our study is a lack of detail regarding implementation and program compliance at participating hospitals. Multiple prior studies have reported associations between these details and checklist effectiveness.(47, 48) Though compliance data would help explain why Keystone Surgery program implementation was not effective, the purpose of the present study was to evaluate the impact of the program on clinical outcomes, rather than assess why it did or did not work. A detailed assessment of program implementation, including a qualitative assessment of the barriers to implementation, is beyond the scope of the present study.
This study has important implications for organizations considering surgical checklist implementation. Many argue that checklists should be universally adopted because they are easy, inexpensive, and not likely to be harmful. While we agree that checklists can be beneficial, especially to prevent rare events (e.g., wrong-site surgery) and to improve communication among operating room teams, our present study suggests that organizations should not expect large improvements in outcomes after implementation. Much of the literature supporting checklists suggests front-line provider buy-in and comprehensive implementation strategies are essential to achieve true checklist compliance, which is associated with program effectiveness.(47, 49) Yet the results of this and other recently published studies of checklist utilization across broad populations suggest that successful implementation is more difficult to achieve in the real world. Given the potential resources and costs associated with a comprehensive implementation strategy, organizations wishing to use surgical checklists must design their intervention carefully, in order to maximize both clinical effectiveness, and cost savings.
Supplementary Material
1. Identification of Cases, from International Classification of Diseases, Ninth Revision (ICD-9)
2. Complication Codes, from International Classification of Diseases, Ninth Revision (ICD-9)
3. International Classification of Diseases, Ninth Revision (ICD-9), Diagnoses and Procedure Codes Used to Determine Reoperations after Surgery
Acknowledgments
Funding/Support: Dr. Reames and Dr. Scally are supported by a grant from the National Cancer Institute (5T32CA009672-23). Dr. Dimick is supported by grant from Blue Cross/Blue Shield of Michigan Foundation.
References
- 1.Haynes AB, Weiser TG, Berry WR, et al. A surgical safety checklist to reduce morbidity and mortality in a global population. N Engl J Med. 2009;360:491–499. doi: 10.1056/NEJMsa0810119. [DOI] [PubMed] [Google Scholar]
- 2.Weiser TG, Haynes AB, Dziekan G, et al. Effect of a 19-item surgical safety checklist during urgent operations in a global patient population. Ann Surg. 2010;251:976–980. doi: 10.1097/SLA.0b013e3181d970e3. [DOI] [PubMed] [Google Scholar]
- 3.de Vries EN, Prins HA, Crolla RM, et al. Effect of a comprehensive surgical safety system on patient outcomes. N Engl J Med. 2010;363:1928–1937. doi: 10.1056/NEJMsa0911535. [DOI] [PubMed] [Google Scholar]
- 4.Haugen AS, Softeland E, Almeland SK, et al. Effect of the World Health Organization Checklist on Patient Outcomes: A Stepped Wedge Cluster Randomized Controlled Trial. Ann Surg. 2014 doi: 10.1097/SLA.0000000000000716. [DOI] [PubMed] [Google Scholar]
- 5.Ko HC, Turner TJ, Finnigan MA. Systematic review of safety checklists for use by medical care teams in acute hospital settings--limited evidence of effectiveness. BMC Health Serv Res. 2011;11:211. doi: 10.1186/1472-6963-11-211. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Bergs J, Hellings J, Cleemput I, et al. Systematic review and meta-analysis of the effect of the World Health Organization surgical safety checklist on postoperative complications. Br J Surg. 2014;101:150–158. doi: 10.1002/bjs.9381. [DOI] [PubMed] [Google Scholar]
- 7.Borchard A, Schwappach DL, Barbir A, et al. A systematic review of the effectiveness, compliance, and critical factors for implementation of safety checklists in surgery. Ann Surg. 2012;256:925–933. doi: 10.1097/SLA.0b013e3182682f27. [DOI] [PubMed] [Google Scholar]
- 8.Patient Safety: Surgical Safety Web Map. [June 3rd, 2014];2012 Mar 26; Available at: http://maps.cga.harvard.edu:8080/Hospital/
- 9.Urbach DR, Govindarajan A, Saskin R, et al. Introduction of surgical safety checklists in Ontario, Canada. N Engl J Med. 2014;370:1029–1038. doi: 10.1056/NEJMsa1308261. [DOI] [PubMed] [Google Scholar]
- 10.Haynes AB, Berry WR, Gawande AA. Surgical safety checklists in ontario, Canada. N Engl J Med. 2014;370:2350. doi: 10.1056/NEJMc1404583. [DOI] [PubMed] [Google Scholar]
- 11.Weiser TG, Krummel TM. Surgical safety checklists in ontario, Canada. N Engl J Med. 2014;370:2349–2350. doi: 10.1056/NEJMc1404583. [DOI] [PubMed] [Google Scholar]
- 12.MHA Keystone: Surgery. [March 28th, 2014]; Available at: http://www.mhakeystonecenter.org/collaboratives/surgery.htm.
- 13.Pronovost P, Needham D, Berenholtz S, et al. An intervention to decrease catheter-related bloodstream infections in the ICU. N Engl J Med. 2006;355:2725–2732. doi: 10.1056/NEJMoa061115. [DOI] [PubMed] [Google Scholar]
- 14.Berenholtz SM, Pham JC, Thompson DA, et al. Collaborative cohort study of an intervention to reduce ventilator-associated pneumonia in the intensive care unit. Infect Control Hosp Epidemiol. 2011;32:305–314. doi: 10.1086/658938. [DOI] [PubMed] [Google Scholar]
- 15.Pronovost PJ, Berenholtz SM, Needham DM. Translating evidence into practice: a model for large scale knowledge translation. BMJ. 2008;337:a1714. doi: 10.1136/bmj.a1714. [DOI] [PubMed] [Google Scholar]
- 16.Timmel J, Kent PS, Holzmueller CG, et al. Impact of the Comprehensive Unit-based Safety Program (CUSP) on safety culture in a surgical inpatient unit. Joint Commission journal on quality and patient safety / Joint Commission Resources. 2010;36:252–260. doi: 10.1016/s1553-7250(10)36040-5. [DOI] [PubMed] [Google Scholar]
- 17.Bratzler DW, Hunt DR. The surgical infection prevention and surgical care improvement projects: national initiatives to improve outcomes for patients having surgery. Clin Infect Dis. 2006;43:322–330. doi: 10.1086/505220. [DOI] [PubMed] [Google Scholar]
- 18.Clancy CM. SCIP: making complications of surgery the exception rather than the rule. AORN journal. 2008;87:621–624. doi: 10.1016/j.aorn.2008.02.007. [DOI] [PubMed] [Google Scholar]
- 19.Stulberg JJ. Adherence to Surgical Care Improvement Project Measures and the Association With Postoperative Infections. Jama. 2010;303:2479. doi: 10.1001/jama.2010.841. [DOI] [PubMed] [Google Scholar]
- 20.Details for Demonstration Project Name: Medicare Acute Care Episode (ACE) Demonstration. [November 14, 2013]; Available at: http://go.cms.gov/19oWDYZ.
- 21.Sexton JB, Berenholtz SM, Goeschel CA, et al. Assessing and improving safety climate in a large cohort of intensive care units. Crit Care Med. 2011;39:934–939. doi: 10.1097/CCM.0b013e318206d26c. [DOI] [PubMed] [Google Scholar]
- 22.Iezzoni LI, Daley J, Heeren T, et al. Identifying complications of care using administrative data. Med Care. 1994;32:700–715. doi: 10.1097/00005650-199407000-00004. [DOI] [PubMed] [Google Scholar]
- 23.Weingart SN, Iezzoni LI, Davis RB, et al. Use of administrative data to find substandard care: validation of the complications screening program. Med Care. 2000;38:796–806. doi: 10.1097/00005650-200008000-00004. [DOI] [PubMed] [Google Scholar]
- 24.Dimick JB, Nicholas LH, Ryan AM, et al. Bariatric surgery complications before vs after implementation of a national policy restricting coverage to centers of excellence. JAMA. 2013;309:792–799. doi: 10.1001/jama.2013.755. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Livingston EH. Procedure incidence and in-hospital complication rates of bariatric surgery in the United States. Am J Surg. 2004;188:105–110. doi: 10.1016/j.amjsurg.2004.03.001. [DOI] [PubMed] [Google Scholar]
- 26.Morris AM, Baldwin LM, Matthews B, et al. Reoperation as a quality indicator in colorectal surgery: a population-based analysis. Ann Surg. 2007;245:73–79. doi: 10.1097/01.sla.0000231797.37743.9f. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Tsai TC, Joynt KE, Orav EJ, et al. Variation in surgical-readmission rates and quality of hospital care. N Engl J Med. 2013;369:1134–1142. doi: 10.1056/NEJMsa1303118. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Birkmeyer JD, Gust C, Baser O, et al. Medicare payments for common inpatient procedures: implications for episode-based payment bundling. Health services research. 2010;45:1783–1795. doi: 10.1111/j.1475-6773.2010.01150.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Miller DC, Gust C, Dimick JB, et al. Large variations in Medicare payments for surgery highlight savings potential from bundled payment programs. Health Aff (Millwood) 2011;30:2107–2115. doi: 10.1377/hlthaff.2011.0783. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Dimick JB, Weeks WB, Karia RJ, et al. Who pays for poor surgical quality? Building a business case for quality improvement. J Am Coll Surg. 2006;202:933–937. doi: 10.1016/j.jamcollsurg.2006.02.015. [DOI] [PubMed] [Google Scholar]
- 31.Gottlieb DJ, Zhou W, Song Y, et al. Prices don’t drive regional Medicare spending variations. Health Aff (Millwood) 2010;29:537–543. doi: 10.1377/hlthaff.2009.0609. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Colla CH, Wennberg DE, Meara E, et al. Spending differences associated with the Medicare Physician Group Practice Demonstration. JAMA. 2012;308:1015–1023. doi: 10.1001/2012.jama.10812. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Volpp KG, Rosen AK, Rosenbaum PR, et al. Mortality among hospitalized Medicare beneficiaries in the first 2 years following ACGME resident duty hour reform. JAMA. 2007;298:975–983. doi: 10.1001/jama.298.9.975. [DOI] [PubMed] [Google Scholar]
- 34.Wooldridge JM. Introductory econometrics : a modern approach. Mason, Ohio: South-Western Cengage Learning; 2009. [Google Scholar]
- 35.Ghaferi AA, Osborne NH, Birkmeyer JD, et al. Hospital characteristics associated with failure to rescue from complications after pancreatectomy. J Am Coll Surg. 2010;211:325–330. doi: 10.1016/j.jamcollsurg.2010.04.025. [DOI] [PubMed] [Google Scholar]
- 36.Donald SG, Lang K. Inference with difference-in-differences and other panel data. Rev Econ Stat. 2007;89:221–233. [Google Scholar]
- 37.Ryan AM. Effects of the Premier Hospital Quality Incentive Demonstration on Medicare Patient Mortality and Cost. Health Services Research. 2009;44:821–842. doi: 10.1111/j.1475-6773.2009.00956.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.Elixhauser A, Steiner C, Harris DR, et al. Comorbidity measures for use with administrative data. Med Care. 1998;36:8–27. doi: 10.1097/00005650-199801000-00004. [DOI] [PubMed] [Google Scholar]
- 39.Southern DA, Quan H, Ghali WA. Comparison of the Elixhauser and Charlson/Deyo methods of comorbidity measurement in administrative data. Med Care. 2004;42:355–360. doi: 10.1097/01.mlr.0000118861.56848.ee. [DOI] [PubMed] [Google Scholar]
- 40.Zhang J, Yu KF. What’s the relative risk? A method of correcting the odds ratio in cohort studies of common outcomes. JAMA. 1998;280:1690–1691. doi: 10.1001/jama.280.19.1690. [DOI] [PubMed] [Google Scholar]
- 41.Russ S, Rout S, Sevdalis N, et al. Do safety checklists improve teamwork and communication in the operating room? A systematic review. Ann Surg. 2013;258:856–871. doi: 10.1097/SLA.0000000000000206. [DOI] [PubMed] [Google Scholar]
- 42.Reames BN, Krell RW, Campbell DA, et al. A Checklist-based Intervention to Improve Surgical Outcomes in Michigan: Evaluation of the Keystone Surgery Program. JAMA surgery. 2014 doi: 10.1001/jamasurg.2014.2873. In Press. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43.Waters HR, Korn R, Jr, Colantuoni E, et al. The business case for quality: economic analysis of the Michigan Keystone Patient Safety Program in ICUs. American journal of medical quality : the official journal of the American College of Medical Quality. 2011;26:333–339. doi: 10.1177/1062860611410685. [DOI] [PubMed] [Google Scholar]
- 44.Semel ME, Resch S, Haynes AB, et al. Adopting A Surgical Safety Checklist Could Save Money And Improve The Quality Of Care. U S Hospitals Health Affairs. 2010;29:1593–1599. doi: 10.1377/hlthaff.2009.0709. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 45.Iezzoni LI, Foley SM, Daley J, et al. Comorbidities, complications, and coding bias. Does the number of diagnosis codes matter in predicting in-hospital mortality? JAMA. 1992;267:2197–2203. doi: 10.1001/jama.267.16.2197. [DOI] [PubMed] [Google Scholar]
- 46.Fisher ES, Whaley FS, Krushat WM, et al. The accuracy of Medicare’s hospital claims data: progress has been made, but problems remain. Am J Public Health. 1992;82:243–248. doi: 10.2105/ajph.82.2.243. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 47.Conley DM, Singer SJ, Edmondson L, et al. Effective surgical safety checklist implementation. J Am Coll Surg. 2011;212:873–879. doi: 10.1016/j.jamcollsurg.2011.01.052. [DOI] [PubMed] [Google Scholar]
- 48.Hannam JA, Glass L, Kwon J, et al. A prospective, observational study of the effects of implementation strategy on compliance with a surgical safety checklist. BMJ quality & safety. 2013;22:940–947. doi: 10.1136/bmjqs-2012-001749. [DOI] [PubMed] [Google Scholar]
- 49.Saturno PJ, Soria-Aledo V, Da Silva Gama ZA, et al. Understanding WHO surgical checklist implementation: tricks and pitfalls. An observational study World J Surg. 2014;38:287–295. doi: 10.1007/s00268-013-2300-6. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
1. Identification of Cases, from International Classification of Diseases, Ninth Revision (ICD-9)
2. Complication Codes, from International Classification of Diseases, Ninth Revision (ICD-9)
3. International Classification of Diseases, Ninth Revision (ICD-9), Diagnoses and Procedure Codes Used to Determine Reoperations after Surgery