Effects on Clinical Outcomes of a 5-Year Surgical Safety Checklist Implementation Experience: A Large-scale Population-Based Difference-in-Differences Study

Abstract

The adoption of a surgical checklist is strongly recommended worldwide as an effective practice to improve patient safety; however, several studies have reported mixed results and a number of issues are still unresolved. The main objective of this study was to explore the impact of the first 5-year period of a surgical checklist-based intervention in a large regional health care system in Italy (4 500 000 inhabitants). We conducted a retrospective longitudinal study on 1 166 424 patients who underwent surgery in 48 public hospitals between 2006 and 2014. The adherence to the checklist was measured between 2011 and 2013 through a computerized database. The effects of the intervention were explored through multivariable logistic regression and difference-in-differences (DID) approaches, based on current administrative data sources. In-hospital and 30-days mortality, 30-days readmissions and length-of-stay (LOS) ⩾8 days were the observed outcomes. Adherence to the checklist showed marked variations across hospitals (0%-93.3%). A pre/post analysis detected statistically significant differences between surgical interventions performed in hospitals with higher adherence to the checklist (⩾75% of the surgeries) and those performed in other hospitals, as for the 30-days readmissions rate (odds ratio [OR]: 0.96; 95% confidence interval [CI]: 0.94-0.98) and LOS ⩾ 8 days rate (OR: 0.88; 95% CI: 0.87-0.89). These findings were confirmed after risk adjustment and DID analysis. No association was observed with mortality outcomes. On the whole, our study attained mixed results. Although a protective effect of the surgical checklist use could not be proved over the first 5 years of this regional implementation experience, our research offers some methodological insights for practical use in the evaluation process of large-scale implementation projects.

Introduction

Surgical care is delivered through high-risk clinical processes and accounts for a large proportion of hospital care and in-hospital adverse events (AEs).^1–3 Temporal trends of clinical outcomes after surgery have been described in large populations, at the national or regional levels.^4,5 Since up to 40%-50% of in-hospital AEs are associated with a surgical specialty or an operation,^2,6 with variable proportions of preventability,³ the perioperative setting is greatly worth considering and exploring with respect to patient safety measurement and implementation issues.^7,8

Among interventions aimed to improve safety in surgery, the effectiveness of surgical checklists, particularly the WHO surgical safety checklist (WHO SSCL), in reducing perioperative mortality and morbidity due to surgical procedures, was first described in 2009,^9,10 and subsequently analyzed by several systematic reviews and meta-analyses, updating the available evidence of the effectiveness of compliance with and critical factors involved with adopting surgical checklists in routine surgical practice.^11–16 Across the reviewed studies, the use of a surgical checklist seems consistently and significantly associated with a reduction in postoperative complications^9,17 with more significant results when all three components of the checklist were completed,¹⁷ whereas increased rates of complications were also reported.¹⁵ The association with a reduction in mortality appears less robust, varying between favorable risk ratios^11,13,15 and no significant results,^12,18 whereas increased rates of mortality following the implementation of the checklist were also observed.¹⁵ More precisely, among published large scale experiences, Urbach et al.¹⁸ failed to detect any significant decrease in post-operative mortality or complications associated with the implementation of the surgical checklist in Ontario, Canada; conversely, Haynes et al¹⁹ described a reduction in deaths after inpatient surgery in South Carolina within the context of a surgical safety collaborative based on a structured, voluntary implementation process.²⁰ Internationally, the reported use of surgical checklists, while showing wide variations across countries, was associated with lower mortality.²¹ As for Italy, a study carried out in a tertiary care hospital reported an association of the checklist with a 27% reduction in the adjusted 90-days mortality rate, as well as a reduction in the adjusted LOS.²² Other specific outcomes, such as wound infections^12,13,15 and blood loss,^12,15 seem to be on the overall positively influenced by the use of a surgical checklist, while a non-significant association was detected for pneumonia¹² and mixed results were described for unplanned returns to the operating room.^12,15 Regarding length of admission, a statistically significant decrease was observed, although probably clinically insignificant.¹⁵

Generally speaking, most reviews of available evidence on the effectiveness of the SSCL were based on a limited number of studies—between 7^12,13 and 22¹¹—and only one included 25 studies.¹⁵ Moreover, individual studies showed marked methodological heterogeneity and contradictory observations. Recently, and remarkably, de Jager et al.¹⁵ emphasized the overall inconsistent effects of the SSCL on postoperative outcomes.

In summary, over the last decade, the WHO SSCL has been strongly recommended for adoption internationally, therefore gradually and widely introduced, sometimes as a mandatory practice, by several national or regional governments.^{10,17–18,20,23,24} However, since mixed results have been reported, a number of questions remain unresolved, concerning both the real impact on patient safety and practical issues raised by local implementation processes.^16–18,25

Since 2009, the Italian Ministry of Health has recommended the use of a modified WHO SSCL as a component of more extensive guidelines on safety in the operating room.²⁶ In the Emilia-Romagna region—in central Italy—the 2009 regional average death rates after surgery (in-hospital 1.03%; 30-days from discharge 0.51%) were comparable with or even lower than those reported in many European countries.^4,5 On the basis of international and national recommendations, implementing the SSCL was promptly set as a priority and a regional project was started in 2010, supported by the Regional Agency for Health and Social Care (RAHSC). The main purpose of the project was to build a regional network of hospitals committed to the local implementation of the SSCL, also developing appropriate monitoring of its compliance and impact in routine surgical practice.

This paper describes the observed temporal trends of surgical outcomes in hospitalized patients who underwent surgery in Emilia-Romagna hospitals between 2006 and 2014. In this retrospective cohort study, we performed a population-based evaluation of a region-wide checklist-based intervention (the SOS.net project), implementing a modified WHO surgical checklist and exploring the impact on four clinical outcomes measured through current administrative data sources. Ultimately, the first 5-year experience of implementing the SSCL is explored, with a focus on variations in rates of adherence across hospitals and evaluation of the impact on selected outcomes following a wide range of surgical procedures.

In comparison with other studies, our research rests on three main strengths: a large-scale population-based perspective, a concurrent control cohort and a long observation period. Similar characteristics were reported by few other studies,^17–19,27 but to our knowledge, our research project is the first one combining them all.

Methods

Setting and study period

The checklist-based intervention, as well as the evaluation study, were conducted in the Emilia-Romagna region of Italy (Supplemental, Figure 1), where a publicly funded healthcare system exists, serving about 4 500 000 inhabitants. Overall, more than 300 000 surgical procedures are performed yearly, more than 200 000 done in ordinary admissions and about 100 000 as day-surgeries. The region is divided in three administrative areas: North, Centre, Romagna (Supplemental, Figure 2).

The implementation process evolved between 2010 and 2014, as is described below. The rate of adoption of the surgical checklist was surveyed over 3 years, between 2011 and 2013. The impact of the checklist based regional intervention on surgical outcomes was evaluated along a 9-year study period, between January 1, 2006, and December 31, 2014. The period from January 2006 to December 2010 was defined as “pre-implementation (or baseline) period,” whereas the period from January 2011 to December 2014 was defined as “post-implementation period,” also including the “pilot” year 2011.

Intervention

The WHO-SSCL was introduced in Emilia-Romagna with a multifaceted intervention (Table 1), targeting from the beginning the entire regional healthcare system. The choice of such a large scale perspective was based on the assumption that our health care system, with long-standing experience in patient safety and risk management, was likely to be culturally mature to react positively and diffusely to a promising and scientifically grounded innovation such as the use of a SSCL.

Table 1.

Emilia-Romagna Region (Italy). SOS.net (Sale Operatorie Sicure, Safe Operating Rooms) project: concise description of the threefold intervention, related actions and support provided to the implementation process.

Actions	Support
1) Regional adaptation of the national recommendations for safety in the operating room and the checklist tool
Development of an adapted 20-item WHO SSCL (form A) Development of a Form B aimed to collect data on a defined set of most common flaws detectable through the form A.	Regional workshops to launch the project Regional Coordinating Multidisciplinary Group (RCMG) established Dissemination of national and regional recommendations Official invitation from the Regional Agency for Health and Social Care (RAHSC) to the hospitals to voluntarily participate in the SOS.net project
2) Development of a regional computerized SOSnet database
A computerized regional information system allowing documentation of the use of the SSCL in routine practice was developed in 2010. Hospitals provide regular input—both form A and form B—of all checklists actually used.	Computerized SOS.net database hosted by one of the regional university hospitals (Policlinico di Modena), also providing a helpdesk, regular recalls for transfer of local data and periodical feedback reports to the participating hospitals and local teams. Data analysis and regional reports provided by the RAHSC.
3) Regional educational intervention
Managers of healthcare services, risk managers, key professionals in the surgical wards (nurses, surgeons and anesthesiologists) were involved in an educational “cascade”.	Coordinating activity of the RCMG Educational materials provided Regional educational events, theory and practice Onsite training with simulation and debriefing Local multidisciplinary supporting teams identified Annual daylong regional workshops (2011, 2012, 2013) to discuss the state-of-the art of the project Setting up of a dedicated website

First, in March 2010, a regional coordinating multidisciplinary group (anesthesiologists, surgeons, operating room nurses, healthcare managers, and support personnel from the Regional Agency) was established for the purpose of developing region-wide adaptation of the national recommendations on safety in the operating room, to be disseminated and implemented in all hospitals performing surgery. A revised WHO-SSCL was then drafted and split into two forms: form A, which included the traditional items of the WHO-SSCL, and form B, which collected all observed deviations from the standard behavior, as outlined by form A.

Second, all hospitals performing surgery were encouraged to participate in the regional project and were asked to form a local multidisciplinary support team (anesthesiologist, surgeon, operating room nurse, risk manager, health services physician). Participating hospitals and their surgical wards volunteered to be enrolled in the project and did not receive any direct financial support for their work. The regional project and the network supporting it were called, respectively, SOS.net (Italian acronym standing for “Sale Operatorie Sicure” —Safe Operating Rooms).

From December 2010, the enrollment took place gradually; by the end of 2011, 29 public hospitals had joined the project, increasing to 40 (out of 50, 80%) by the end of 2012 (although not necessarily involving the totality of surgical units or providing regular input to the SSCL database (see below, Data)). The implementation process was initially undertaken through a regional educational intervention; subsequently, all hospitals were asked to offer the same program at the local level, with a cascade process. By December 2011, more than 2000 health professionals had been trained. The pilot phase of the SOS.net project started at the end of 2010 and continued throughout 2011; feedback on compliance with the SSCL was sent quarterly to the participating hospitals from the beginning of the pilot phase until the end of 2013.

Study design

The effects of the regional implementation project were evaluated through process and outcome measures, the compliance with the SSCL and four clinical outcomes, respectively.

Professional attitudes and factors influencing compliance were also investigated in 2012 through a 30-item semi-structured questionnaire administered to the local hospital teams. The survey provided a halfway range of information on the local context and attitudes of the multidisciplinary teams; general attitudes toward the project, local barriers, professional resistance, and time issues were particularly explored. The results of this survey are not discussed in this paper and will be reported elsewhere.

Data

Project database

All items of the SSCL, as included in both forms A and B, were translated into as many variables of a computerized, specifically designed, regional database (the SOS.net database) which supported a systematic monitoring process and a periodical feedback—mostly quarterly and always yearly—to all participating hospitals and surgical teams. The access to the database was made available in each hospital, although its use was not made compulsory.

Population

The study population for the 9-year descriptive analysis of surgical outcomes was defined by all ordinary admissions (either electively or as an emergency) between 2006 and 2014. Inclusion criteria for patients were residence in Emilia-Romagna, age ⩾18 and ⩽85. Inclusion criteria for procedures were any surgical Diagnosis Related Groups (DRGs) registered in a public hospital that performs at least 50 surgical DRGs yearly. Percutaneous coronary interventions (PCI), biopsies and interventions on varicose veins (ligation and stripping) were excluded, as well as procedures of interventional radiology. During the study period a major revision occurred, at the national level, for some groups of codes (referring to both diagnoses and procedures) of the International Classification of Diseases, 9th Revision (ICD-9-CM), including cardiac surgery, plastic surgery, orthopedics and pain therapy. Since an in-depth analysis showed that cardiovascular procedures were specifically affected by this change in terms of volumes and related rates, we chose to exclude them, in order to prevent any possible biases.

All selected surgical procedures were then classified according to the ICD-9-CM procedure codes as: nervous, endocrine, eye, ear, nose, respiratory, lymphatic, gastrointestinal, urinary, male genital, female genital, obstetric, musculoskeletal, breast and skin.

Measurement

Table 2 summarizes and clarifies in terms of definition, rationale, and methodological details the entire set of measures adopted throughout the threefold evaluation process carried out to monitor the implementation of the checklist-based intervention and its impact on surgical outcomes in the study population.

Table 2.

Emilia-Romagna Region (Italy). SOS.net Project: concise description of the three-fold evaluation process, with related measures, rationale and essential methodological details.

Measures	Rationale	Methodological details
1) Adherence to the checklist (study period 2011-2013)
Rate, hospitala Rate, surgical warda	Measures the formally declared adherence to the surgical checklist regional project by the organizational components of the regional healthcare system	Source of data: denominators: Emilia-Romagna Regional Computerized Hospital Discharge Database (HDD) numerators: SOSnet computerized regional database
Surgical operations covered by the SSCL (%)	Measures the use of the surgical checklist in the routine clinical practice
N/frequency of deviations from the standarda	Describes the potential of the SSCL in detecting problems
2) Impact of the SOS.net project (study period 2006-2014)
In-hospital mortality rate 30-days (from discharge) mortality rate	Measures largely used in the literature on the SSCL	Ordinary admissions only Source of data: Emilia-Romagna HDD
Length of stay (LOS): ⩾ 8 days	Proxy of complications occurred during index hospitalization. The 8-days cut off was opted for since most surgical patients without complications are discharged within 6-7 days.	Source of data: Emilia-Romagna HDD
30-days (from index discharge) readmissions rate	Proxy of post-surgery “any complications”	Source of data: Emilia-Romagna HDD
3) Professional attitudes (study period 2012)
Issues related to the local contexts and teamsa	Local context and professional attitudes are influential factors in quality improvement projects16	Source of data: half-structured questionnaire (30 items) to the local hospital teams.

^aData not reported in the article.

Compliance with the checklist

Adherence to the SSCL in every surgical intervention was determined by a SSCL which was filled out and registered in the regional computerized project database. The database allowed the measurement of several indicators: the proportion of surgeries covered by the SSCL, the completeness of the registered SSCLs, the number and frequency of deviations from the standard behavior, as explored by each item of the SSCL. The degree of completion or accuracy of the SSCL was not considered in this study.

The “percentage of coverage” of surgical operations by the checklist was obtained as a ratio between the number of SSCLs registered in the SOS.net database and the number of surgeries eligible for use of the SSCL, in the same study period (source: regional computerized Hospital Discharge Database—HDD). Throughout 2011-2013, variations in compliance were measured across hospitals and Local Health Units and over time, in order to describe the pattern of changes that occurred during the implementation process.

According to the levels of compliance, two populations of surgeries were identified and compared: those performed in hospitals with higher levels (best performer) and those performed in other hospitals. The threshold value between the two groups of surgeries was set at 75%. The reason behind the choice of a 75% threshold coverage for a best performer hospital was twofold: although only 100% coverage is expected to be effective, it is also a challenging target to achieve and maintain and is not frequently observed, whereas a 75% coverage, attained by a sufficient number of hospitals, seemed acceptable and realistic. A comparable value has been reported as mean performance in systematic reviews.¹¹

Impact on surgical outcomes

The association between the use of the SSCL and improved outcomes in adult surgical patients was investigated within the study period 2006-2014. Four measures of surgical outcomes were selected: in-hospital and 30-days (from discharge) mortality rate, percentage of patients with a length of stay (LOS) ⩾8 days (as a proxy of complications occurred during the index hospitalization), 30-days (from index discharge) readmissions rate. The 8-day threshold of the LOS indicator was based on the 75th percentile of the mean LOS distribution. Data were obtained from the regional computerized HDD provided by the Emilia-Romagna Regional Health and Social Care Information System, which includes demographic details (age and sex), main discharge diagnosis, up to 15 secondary diagnosis codes as well as 15 procedure codes from the ICD-9-CM, and discharge status. Information on death status was also retrieved from the same source, as well as from the Regional Healthcare Population Register and the Regional Mortality Register.

Statistical analysis

The impact of the whole SOS.net project on the defined surgical outcomes was evaluated in the observation period 2006-2014 with a population-based, quasi-experimental retrospective before/after study, using a pre/post analysis based on logistic regression. A difference-in-differences (DID) analytic approach²⁸ was also used, isolating changes of outcomes associated with the investigated intervention while taking into account (and removing) the effect of any secular trends observed in a comparison group not directly exposed to the same organizational change (Supplemental Appendix 1). The group of surgeries performed in hospitals with a ⩾75% rate of adherence to the SSCL achieved in 2012 or 2013 (best performers) was compared with the group of surgeries performed in all other hospitals, assuming that both groups were likely to be equally exposed to several factors possibly impacting on surgical outcomes over time (ie, advancement of surgical technology, reorganization of surgical processes, dissemination of scientific knowledge, professional education, case-mix modifications), with the exception of the surgical checklist.

Risk adjustment procedures were guaranteed, including a list of comorbidity conditions based on the Gagne index,²⁹ in addition to sex, age, and geographical area of discharge. The results of the analyses are expressed as odds ratios (OR) with 95% confidence interval (95% CI) and P values.

Results

Our study cohort consisted of 1 166 424 patients who underwent surgery in 48 hospitals performing at least 50 surgical DRGs yearly. Considerable variations were observed in the rates of adherence to the SSCL; in 2011 to 2013, the percentage of surgical operations in which the use of the checklist was registered in the SOS.net database was 47.3% on average, ranging between 0.0% and 93.3% across Local Healthcare Units (LHUs) and related hospitals (Table 3).

Table 3.

Emilia-Romagna Region (Italy). SOS.net project: surgical interventions “covered” by the surgical checklist, Local Healthcare Units and related hospitals. Absolute numbers and %, 2011-2013.

Local Health Care Unit	Year 2011					Year 2012					Year 2013
	Measure of adherence			Hospitals		Measure of adherence			Hospitals		Measure of adherence			Hospitals
	Dena	Num.b	%	Tot c	Part d	Den.a	Num.b	%	Tot.c	Part d	Den.a	Num.b	%	Tot.c	Part d
P	13 348	191	1.4	3	1	12 881	242	1.9	3	1	12 627	7229	57.3	3	1
R	5725	1761	30.8	2	2	5943	2500	42.1	2	2	5980	2486	41.6	2	2
E	10 949	4106	37.5	4	2	10 625	7580	71.3	5	4	10 474	8601	82.1	4	4
M	24 392	2230	9.1	8	3	21 180	3483	16.4	8	2	22 404	3994	17.8	7	3
B	23 401	849	3.6	8	1	23 099	9332	40.4	9	8	21 840	18 742	85.8	9	8
I	6467	3087	47.7	2	1	5875	5352	91.1	2	2	5967	5565	93.3	2	2
F	8071	1615	20.0	5	2	7098	3665	51.6	3	2	7126	5590	78.4	3	3
V	17 425	605	3.5	3	1	16 714	2986	17.9	3	3	16 230	8026	49.5	3	3
L	8194	438	5.3	1	1	8136	1142	14.0	1	1	8206	1686	20.5	1	1
C	9466	1783	18.8	1	1	9476	1131	11.9	1	1	9289	1149	12.4	1	1
N	17 081	2882	16.9	5	2	15 716	6865	43.7	5	5	15 928	9281	58.3	5	5
A	18 704	4910	26.3	1	1	18 746	11 142	59.4	1	1	17 333	9093	52.5	1	1
O	15 696	518	3.3	1	1	15 060	1240	8.2	2	1	14 637	2149	14.7	2	1
D	16 162	9257	57.3	1	1	14 390	10 239	71.2	1	1	15 644	8990	57.5	1	1
G	20 116	288	1.4	1	1	20 133	145	0.7	1	1	19 888	−	-	1	0
U	13 691	4241	31.0	1	1	10 896	2986	27.4	1	1	10 813	8506	78.7	1	1
H	12 322	775	6.3	1	1	13 222	1535	11.6	2	2	13 385	6756	50.5	2	2
TOTAL	241 210	39 536	16.4	48	23	229 190	71 565	31.2	50	38	227 771	107 843	47.3	48	39

^aDenominator: total number of ordinary admissions with surgical DRGs registered in the year of observation (Source: Regional Computerized HDD provided by the Emilia-Romagna Region health-and-social-care information system).

^bNumerator: number of surgical interventions “covered” by the surgical safety checklist (Source: regional SOS.net database).

^cTotal: public hospitals performing surgery in Emilia-Romagna region. The number changes slightly across years due to formal institutional changes involving multiple hospitals located in the same Local Healthcare Organizations.

^dParticipant: hospitals with surgical safety checklists registered in the regional SOS.net database.

Among the hospitals included in our research, 21 were defined as best performers, accounting for a surgical population of 386 167 patients (33.1%), while other hospitals (n = 27) accounted for 780 257 patients (66.9%). The total number of admissions was lower and less variable in best performer hospitals (mean: 19 308, median: 12 361; min: 2716, max: 76 897) when compared with other hospitals (mean: 28 898, median: 25 100, min: 368, max: 105 065). As for the subset of university hospitals (not specifically analyzed in this paper), the total number of admissions in best performer hospitals was half the number observed in the other group (137 877 vs 296 602).

Table 4 shows the characteristics of patients who underwent surgery in either defined group, according to the two time intervals, preceding (2006-2010) or following (2011-2014) the implementation of the SOS.net project in Emilia-Romagna (the latter including the “pilot year” 2011), respectively.

Table 4.

Emilia-Romagna Region (Italy). Patients who underwent surgery, cohort selected for the analysis (N = 1 166 424). Characteristics of the patients by time interval (pre- and post- implementation of the SSCL) and group of adherence to the surgical checklist (best performer hospitals, other hospitals).

Patients’ characteristics	Pre-implementation, 2006-2010 n = 659 757 (56.6%)		Post-implementation, 2011-2014 n = 506 667 (43.4%)
	Best performer hospitals	Other hospitals	Best performer hospitals	Other hospitals
Number of patients, n (%)	225 687 (34.2)	434 070 (65.8)	160 480 (31.7)	346 187 (68.3)
Age (mean ± SD)	54.8	54.7	55.4	55.7
Age class (%)
18-35	19.4	19.7	17.2	16.8
35-65	45.3	45.0	47.1	47.0
>65	35.3	35.0	35.7	36.2
Gender (% female)	58.1	56.1	58.3	55.6
Gagne index29 (%)
(%>=1)	26.7	27.5	26.9	27.9
Co-morbiditiesa (%)
Any tumor	20.7	20.8	20.8	21.1
Hypertension	15.8	15.6	15.2	15.3
Metastatic cancer	5.0	5.5	4.9	5.6
Pulmonary circulation disorders	4.3	4.3	3.7	3.8
Cardiac arrhythmias	3.8	3.9	3.8	4.0
Congestive heart failure	2.4	2.5	2.5	2.4
>=2 co-morbidities (%)	17.1	17.7	16.3	17.4
Procedure typea (%)
Musculoskeletal	22.2	26.0	21.4	26.3
Gastrointestinal	22.0	20.4	21.2	20.6
Female genital	11.6	11.1	11.0	9.3
Obstetric	8.8	8.1	8.5	7.3
Urinary	6.9	7.1	7.9	7.6
Geographical area of discharge (%)b
North (Emilia)	32.1	47.2	34.1	47.0
Center	67.9	17.8	65.9	17.7
Romagna	–	35.0	–	35.3

^aHigher frequency in the total cohort.

^bSee Supplemental, Figure 2.

Across the two time intervals and in both groups of hospitals, patients were substantially similar in all characteristics—demographics, procedures, comorbidities, and geographical area of discharge—except for very small differences in age distribution and female genital procedures in other hospitals. As for a comparison between best performer and other hospitals, in the former group the proportion of females was slightly higher and the proportion of musculoskeletal procedures was moderately lower. Noticeably, among the best performer hospitals, a considerably higher proportion of surgical patients was discharged in the central area of the region (Centre); on the other hand, hospitals located in the Romagna area were all included in the other group.

In the entire cohort of patients, an overall 6% reduction of surgical volumes was observed over the 9-year study period, mostly explained by the 15.5% reduction in the central area of the region. Some variations could be observed across different types of surgical procedures, with higher reductions in male and female genital categories and more moderate reductions in breast, ear, skin, obstetric and nervous procedures. The only positive trend, with a 35.5% increase in surgical volumes, was observed in lymphatic procedures. Three main groups of surgeries—musculoskeletal, gastrointestinal, and female genital—overall accounted for 56.3% of the surgeries performed in the total cohort.

At the regional level both in-hospital and 30-days mortality rates showed annual fluctuations either in best performer and other hospitals, with a 9-year overall slightly decreasing trend, −0.05% and −0.09%, respectively. A more stable, albeit slightly decreasing, trend was observed in regional rates of admissions with a LOS⩾8 days, decreasing from 26.4% in 2006 to 23.61% in 2014, with an overall 2.8% reduction in the cohort; the observed differences between the two groups appeared more pronounced in the post-implementation years. Finally, regional rates of 30-days readmissions show an overall 0.9% reduction, higher in the post-implementation period. Temporal trends for the selected outcomes in the two contrasted populations of surgeries performed by the two groups of hospitals are shown in Figure 1.

Figure 1.

Emilia-Romagna Region, patients who underwent surgery (2006-2014). Temporal trends for the selected outcomes, best performer and other hospitals.

Risk-adjusted ORs for the selected outcomes are shown in Table 5. At the pre-post analysis, no association was observed between the implementation of the SSCL and either crude or adjusted ORs of mortality outcomes (in-hospital and 30-days). The same analysis showed a consistent statistically significant difference between best performer and other hospitals in LOS⩾8 days rate (crude OR: 0.88; 95% CI: 0.87-0.89; adjusted OR: 0.87; 95% CI: 0.86-0.88) and 30-days readmissions rate (crude OR: 0.96; 95% CI: 0.94-0.98; adjusted OR: 0.95; 95% CI: 0.93-0.97). The DID models, accounting for competing time trends during the study period, did not substantially change these results. The stratified analysis including the main groups of operations in terms of frequency and mortality—musculoskeletal, gastrointestinal, urinary—and focusing on 30-days mortality, LOS⩾8 days and 30-days readmissions, only partially confirmed the previously described results (Table 6), showing a higher consistency in the LOS⩾8 days rate.

Table 5.

Emilia-Romagna Region (Italy). Evaluation of the impact of the SOS.net project: adjusted ORs for the four analyzed outcomes. Pre–post analysis (only best performer hospitals) and difference-in-differences (DID) analyses (entire study cohort).^a Absolute numbers, rates, 95% confidence intervals, P value.

Outcomes	Adjusted ORs (95% CI)
	Pre–post analysis (best performer hospitals)	DID analysis (entire study cohort)
In-hospital mortality rate (n = 11 106)	1.046 [0.978-1.119] 0.1923	1.040 [0.972-1.113] 0.2524
30 days mortality rate (n = 16 962)	1.036 [0.980-1.096] 0.2112	1.033 [0.977-1.002] 0.2544
LOS⩾8 days rate (n = 293 735)	0.873 [0.858-0.888] <.0001	0.867 [0.789-0.806] 0.0001
30-days readmissions (n = 134 551)	0.947 [0.926-0.968] <.0001	0.946 [0.925-0.968] <.0001

^aThe covariates used in this analysis are the same as reported in Table 4.

Table 6.

Emilia-Romagna Region (Italy). Evaluation of the impact of the SOS.net project: adjusted ORs for three analized outcomes and three groups of surgical procedures. Pre-post analysis (only best performer hospitals) and difference-in-differences (DID) analysis (entire study cohort). Rates, 95% confidence intervals, P value.

Outcomes (groups of surgical procedures)	Adjusted ORs (95% CI)
	Pre–post analysis (best performer hospitals)	DID analysis (entire study cohort)
Musculoskeletal (N = 288 442)
30 days mortality rate	0.901 [0.757-1.072] 0.2391	0.901 [0.759-1.070] 0.2361
LOS⩾8 days rate	0.869 [0.842-0.898] <.0001	0.868 [0.840-0.896] <.0001
30-days readmissions	0.878 [0.839-0.919] <.0001	0.877 [0.838-0.917] <.0001
Gastrointestinal (n = 243 425)
30 days mortality rate	1.021 [1.018-1.235] 0.0205	1.115 [1.012-1.228] 0.0277
LOS⩾8 days rate	0.925 [0.894-0.957] <.0001	0.925 [0.894-0.956] <.0001
30-days readmissions	0.995 [0.950-1.042] 0.8177	0.997 [0.951-1.044] 0.8852
Urinary (n = 85 601)
30 days mortality rate	1.030 [0.760-1.395] 0.8511	1.047 [0.773-1.418] 0.7668
LOS⩾8 days rate	0.886 [0.837-0.937] <.0001	0.886 [0.838-0.937] <.0001
30-days readmissions	0.921 [0.859-0.988] 0.0213	0.923 [0.861-0.990] 0.0256

Discussion

In this retrospective cohort study, we performed a population-based evaluation of a region-wide checklist-based intervention (the SOS.net project), implementing a modified WHO surgical checklist and exploring the impact on four selected clinical outcomes measured through current administrative data sources.

In a regional context of gradually decreasing trends of surgical mortality, LOS and readmissions rates over the 9-year observation period, we detected small but statistically significant differences between the population of surgeries performed by best performer hospitals—with better adherence to the SSCL—and those performed by other hospitals, all of them involved in the first 5-year period of the implementation experience. The observed differences in LOS⩾8 days and 30-days readmissions rate in adult patients who had undergone surgery were robust across pre-post and DID analyses, whereas a stratified analysis across three large groups of surgical procedures showed mixed results. No significant association was observed between the use of the checklist and improvement in postoperative mortality outcomes.

In our view, our study adds to the current literature for several reasons, notably from a study design point of view. Main strengths of our experience are explained below.

First, as pointed out by de Jager et al,¹⁵ very few published studies that investigated the effect of the surgical checklist had a concurrent control group. We were able to compare two subpopulations of surgeries, those performed in best performer hospitals, with a “coverage” by the SSCL ⩾75%, and those performed in other hospitals, then valuing the importance of a comparison cohort when assessing the effectiveness of complex organizational interventions.^30,31

Second, once again only few studies were conducted on a large scale and followed implementation projects targeting regional, state or even national healthcare systems.^17–18,20 Our study targeted a large regional population of patients who had surgery in many hospitals of different sizes, academic statuses and affiliations. In addition, a period of 9 years was covered and the 5 years that have elapsed since 2010, when the surgical checklist was first introduced in Emilia-Romagna, represent a time interval long enough to allow the detection of a reasonably stable post-intervention effect, potentially approaching a routine practice.

Third, drawing upon both risk adjustment and DID approach, our study was able to take into account possible confounding due to baseline characteristics, comorbidities, and geographical area of discharge of analyzed patients, and to isolate the effect of the intervention from underlying secular trends toward improvement of the selected outcomes in the study period.²⁸

Finally, our set of surgical outcomes was entirely based on current administrative data sources; although these measures can be viewed as being rather rough and less reliable when compared to prospective surveys based on chart review or on site data collection, they are low cost and time saving in large studies. Moreover, the outcomes we studied are less susceptible to misclassification in administrative data.¹⁸ As for mortality, we did not limit our observations to in-hospital measures, but also investigated 30-days mortality, whether occurring inside or outside hospital. As for morbidity, we explored it through 30-days readmissions as a proxy for any complications (except death), serious complications and reoperations. In addition, we measured the rate of compliance with the SSCL (in terms of “coverage” of performed surgical procedures) through a specifically designed database that allowed us to routinely collect and disseminate data among participant teams.

In fact, our study rests on three main strengths: a large-scale population-based perspective, a concurrent comparison cohort, and a long observation period. Similar characteristics were reported by few other studies,^17–19,27 but to our knowledge, our research project is the first one combining them all.

Our study has a number of limitations. First, the retrospective design has well-known drawbacks; nonetheless, assessing compliance with the checklist through routinely collected data from operating teams who are unaware that they are participating in a research study might prove to be a suitable choice.³²

Second, recorded levels of compliance with the checklist might lack some precision. Participating hospitals remarkably differed in the way they entered data into the SOS.net database, ranging from a postponed manual input of a paper checklist to a real-time computerized data entry through fully integrated information technologies in the operating room; in addition, a certain amount of checklists might be used without any concurrent or subsequent input into the regional database. Consequently, the observed rates of compliance might be affected by some underestimation; nevertheless, with regard to this possible bias, our study should be viewed as highly conservative. On the other hand, it was not feasible to measure actual checklist use in the operating room on a regional basis (although this task was carried out in some individual hospitals). Therefore, reported rates of adherence, as registered by the project database, might not reflect, notably overestimate, true levels of use in clinical practice, an issue already raised in previous studies.^18,19,33 One might reasonably suppose that over and underestimation counterbalance each other and are equally distributed across hospitals; nevertheless, future analyses will need to accurately scrutinize this subject.

Third, our selection of outcomes, besides the above-mentioned pros, might suffer from some cons in terms of comparability with previous studies. We measured complications through proxy indicators based on administrative sources of data, precisely 30-days readmissions and LOS⩾8 days. Previously published studies referred to “any complications”, as defined by the American College of Surgeons’ National Surgical Quality Improvement Program,^9,17 whereas other studies used 30-day AEs.³⁴ In terms of data collection, some studies carried out ad hoc surveys,⁹ while others used administrative sources.¹⁸ In addition, “any complications” often included death.^9,17 As a matter of fact, the effects of the surgical checklist on post-operative complications were evaluated through a rather variable range of methods. Our choice was based on a certain rationale. Thirty-days readmissions were similarly analyzed in previously published research.¹⁸ We postulate that these events reasonably account for major complications, mostly requiring hospital treatments (such as serious anemia, blood loss, surgical site infection, pneumonia, need of reoperation, etc.). As a matter of fact, described results seem plausible with published research on determinants of surgical readmissions, some of them (such as bleeding or anemia, wound complication, sepsis/shock, venous thromboembolism)³⁵ are de facto covered by specific items of the checklist. As for the LOS, it was deemed to be an acceptable proxy for in-hospital complications, also examined in other studies.^15,18,27 The rationale behind our choice to focus on admissions exceeding the 8 days threshold was twofold. First, the association between SSCL coverage and LOS as continuous variables was also measured but results were not significant; second, circumscribing the evaluation to longer hospital stays, more likely to reflect more serious surgical and clinical complications, sounded reasonable in order to improve sensitivity. Overall, it must be recognized that published literature seems to suggest that the effects of the SSCL on length of admission are inconsistent and of little significance, either statistical or clinical.¹⁵

Fourth, we did not include the completeness of the checklist as a determinant in the evaluation model, as suggested by some authors^17,31; although we were able to measure it³⁶ we could not obtain reliable data for the whole study period.

Other factors should be considered when interpreting our results. While voluntary participation of the hospitals may have introduced some biases,²¹ mandatory adoption does not guarantee full compliance either.^37,38 However, the internal validity of this study is not to be undermined, as our research was aimed to evaluate the impact of a project as it was implemented rather than implementation itself. Our inclusion criteria for surgical discharges and patients were rather narrow; nevertheless, urgent surgeries, day or ambulatory surgery and pediatric specialties should be better targeted by and compared with specific implementation and evaluation studies.¹⁶ Similarly, investigating AEs in patients aged >85 is a complex issue, necessarily requiring a purposeful focus on appropriateness. The low baseline mortality at the regional level could at least partially explain the lack of any effect observed on this outcome. Lastly, residual confounding cannot be excluded and our observations may indirectly reflect the overall quality of perioperative care²¹ or local policies targeting the reduction of LOS and hospital readmissions. In this regard, a potential source of bias might be the common exposure to the same intervention of both groups of hospitals, best performers and other: in spite of the differences observed in their compliance with the checklist, their involvement in the same implementation process could have produced unmeasured changes (eg, a better attention to some aspects of the perioperative care) with positive effects on the outcomes and consequent underestimation of the effect size.

Indeed, a more in-depth analysis of results deserves some additional considerations. First, best performer hospitals are a smaller, apparently more homogeneous and “less academic” group, as suggested by the number of admissions, the status of the hospitals and their geographical location (see Results). This factor might reflect some selection bias, given that risk adjustment did not include any variables concerning hospital characteristics, except for the geographical area of discharge. Similarly, as regards the analysis of 30-days readmissions, a persisting difference between surgeries from best performer and other hospitals is evident in Figure 1, notwithstanding the statistically significant results described. Therefore, to some degree, our findings might be at least partially explained on the basis of confounding by institutional characteristics. Had this proposition been proved true, we would conclude that hospitals with specific organizational or structural characteristics are more likely to successfully implement the SSCL. Although this hypothesis is plausible and deserves to be properly explored, this was not an objective of the present study, which was mainly aimed at evaluating the overall impact of the checklist-based intervention according to a population-based design.

Second, time trends of both mortality outcomes show a peak in the post-implementation period (Figure 1), which might be attributed to a special cause, high mortality variation. Again, a more in-depth examination should be performed in order to clarify this issue, inasmuch our results might prove to be different, once the special cause has been removed from the analysis.

Third, the DID analysis pointed out an increase in adjusted odds ratio of 30-days mortality rate for gastrointestinal procedures from both pre-post and DID analyses (see Table 6). It is hard to interpret these observations beyond the overall inconclusive findings on mortality attained by our study. However, it is worth mentioning that similar effects have been already reported by other authors.¹⁵

Implications for clinical practice and future research

Overall, our study attained not conclusive results. Although we were able to detect some small but statistically significant differences between the two populations of surgeries performed by best performer and other hospitals, respectively, their clinical relevance is uncertain and we cannot conclude that the surgical checklist had a protective effect on the selected outcomes over the first 5 years of our regional implementation experience. What we have learned makes a contribution in three directions: further research at regional level, future choices at the national level and existing knowledge on this topic at the international level.

At the regional level, our study intended to take stock of the situation after 5 years from the outset of the SOS.net project, in order to give general feedback and orientation for future, long term strategies, especially addressing both regional government and management of the local healthcare organizations. Most of the limitations previously discussed are not likely to be easily overcome in the context of such a large population-based study design, but need to be tackled with a more composite perspective, based on proper geographical granularity and diversified methodologies, possibly including quantitative and qualitative approaches. Therefore, forthcoming evaluations will be directed to drawing a sharper regional map of within-hospitals differences in surgical outcomes and compliance with the checklist, as well as thoroughly investigating local processes of implementation. Work is underway at two levels, beyond the timeframe reported in this paper: while updating and refining policies and research at the regional level, local experiences will be examined more extensively, drawing on the presently available knowledge on the role of several factors such as continuity of training and commitment, institutional support and clinical engagement, particularly regarding surgeons.^39–43 These issues were initially explored by the questionnaire administered in 2012; this work is worth to be resumed and developed in order to achieve a full understanding of barriers and facilitators underlying observed performances. In the meantime, the monitoring system of the compliance with the checklist is being strengthened and data on coverage show a progressive improvement (79% on average in 2017, unpublished data). At the same time, a direct observation study conducted on a sample of surgeries showed variable discrepancies between data on coverage recorded in the regional database and data collected on site.⁴⁴ These observations will be valuable in supporting the implementation of a mixed approach, combining the systematic use of administrative data with effective and targeted quality controls.

At the national level, given the recent trend in Italy toward a mandatory use of the surgical checklist, it is essential that both regional and central governments make appropriate efforts to adopt a standardized and repeatable monitoring system. Our study makes a methodological proposal in that direction, also suggesting that the feasibility and replicability of our approach is mainly determined by the existence of a reliable registration system of the actual use of the checklist (which might be integrated, for example, in the computerized information system of the operating room or in the computerized HDD). On the other hand, this perspective implies a broader view on the repeatedly reported gap between recorded adherence and real compliance with the SSCL.³³ While a merely formal fulfillment of an institutional duty^37–39 should be prevented, given the sometimes disappointing effects of a regulatory strategy,^18,33 a robust improvement process for the purpose of reducing the burden of surgical harm⁴⁵ will be needed, first including (but not limited to) the use of the SSCL and carefully considering the encouraging results achieved through a collaborative approach.^19,46

Taking all this into account, the described methodology, based on current administrative data, could take on a practical value—at both national and international level—as a possibly convenient, yet not exhaustive, preliminary approach in large-scale implementation projects supported by high-quality computerized information systems and providing centralized support to checklist based interventions.

As for the contribution of our research to the large existing literature on the same topic, our results could complement current knowledge highlighting methodological issues when introducing the SSCL in surgical practice, particularly in large-scale and centralized interventions.¹⁵ In fact, in spite of a meaningful burden of studies reporting the effectiveness of the SSCL, substantial inconsistencies still persist when real-world practice is examined, particularly on a large scale.¹⁵ Moreover, rigorous evaluation of experiences remains a challenge, since the implementation of the surgical checklist is undoubtedly a complex intervention and its role should be appropriately emphasized among a wider range of best practices for patient safety in surgery.^40–43,46

Conclusion

In conclusion, this study offers an example of controlled, region-wide, population-based evaluation of the effects of a 5-year SSCL implementation experience, as a first step in a necessarily longer and more complex evaluation process, which is now in progress in the Emilia-Romagna healthcare system.

Surgical checklists were introduced as an apparently easy method promising to cut complications and save lives, but in routine practice, they may not be so simple to be used and often even fail to produce benefits.^15,47 Although administrative data are a priceless resource for monitoring and evaluation in large-scale interventions, their use would benefit from being part of a more comprehensive set of tools and methods. Moreover, it must be remembered that future research on SSCL, as well as on other safety practices, should more explicitly and extensively focus on complexity and variability of organizational interventions, highly influenced by socio-cultural attitudes and increasingly requiring a better understanding of the role of adaptive human and social practices in safety efforts.⁴⁸