Impact of the World Health Organization's Surgical Safety Checklist on safety culture in the operating theatre: a controlled intervention study

Abstract

Background

Positive changes in safety culture have been hypothesized to be one of the mechanisms behind the reduction in mortality and morbidity after the introduction of the World Health Organization's Surgical Safety Checklist (SSC). We aimed to study the checklist effects on safety culture perceptions in operating theatre personnel using a prospective controlled intervention design at a single Norwegian university hospital.

Methods

We conducted a study with pre- and post-intervention surveys using the intervention and control groups. The primary outcome was the effects of the Norwegian version of the SSC on safety culture perceptions. Safety culture was measured using the validated Norwegian version of the Hospital Survey on Patient Safety Culture. Descriptive characteristics of operating theatre personnel and checklist compliance data were also recorded. A mixed linear regression model was used to assess changes in safety culture.

Results

The response rate was 61% (349/575) at baseline and 51% (292/569) post-intervention. Checklist compliance ranged from 77% to 85%. We found significant positive changes in the checklist intervention group for the culture factors ‘frequency of events reported’ and ‘adequate staffing’ with regression coefficients at −0.25 [95% confidence interval (CI), −0.47 to −0.07] and 0.21 (95% CI, 0.07–0.35), respectively. Overall, the intervention group reported significantly more positive culture scores—including at baseline.

Conclusions

Implementation of the SSC had rather limited impact on the safety culture within this hospital.

Editor's key points.

• The World Health Organization's Surgical Safety Checklist was introduced to improve perioperative morbidity, mortality, and adherence to clinical protocols.

• The role of changes in safety culture in the positive effects of this checklist was assessed in a prospective controlled intervention survey in operating theatre personnel.

• Successful checklist implementation had limited impact on patient safety culture in this single-site study, for unclear reasons that require further study.

An estimated 234 million major surgical operations are performed annually worldwide.¹ As volume and importance of surgery in global healthcare increase, patient safety and quality in surgical care gain more attention.^2,3 Nearly one in 10 in-hospital patients experience iatrogenic events and more than half of them occur within perioperative care.⁴

In 2008, the World Health Organization (WHO) launched the Safe Surgery Saves Lives campaign and produced the ‘Surgical Safety Checklist’ (SSC) designed to reduce complications and deaths associated with surgery.⁵ In an international pilot study, the SSC intervention resulted in a decrease in mortality (1.5–0.8%) and morbidity (17–11%).⁶ Similar effects were found after implementing the more comprehensive Surgical Patient Safety System (SURPASS) checklist on patient outcomes in the Netherlands.⁷ An important purpose of introducing the WHO SSC was to improve basic clinical processes as shown by the increase in appropriate antibiotic use from 56% to 83%, correct site marking from 54% to 92%, and overall clinical safety processes from 34% to 57%, suggesting improved reliability in clinical care.⁶

Within the healthcare and other industries, checklists are more than a simple intervention. At a basic level, they function as reminders, which ensure basic care processes are adhered to (assuming whichever checklist is in place is used correctly). At a broader level, checklists and their usage have implications for team working, team cohesion, and safety culture. Checklists require people to change their work routines—for example, the Time Out phase of the WHO SSC requires the entire operating theatre team to gather and pause for a few seconds before proceeding with a procedure. Given that the healthcare industry was rarely using such interventions until recently, it has been argued that checklists are not a panacea that will fix every safety problem—rather they are likely to interact with the team and safety culture of the local team and wider organization.⁸ If significant wider problems exist within an organization, the likely outcome is that a checklist will not have a positive benefit, and indeed, it may be reduced to a tick box exercise.⁹

Along these lines, checklist-driven improvements have been hypothesized to impact positively on team and safety culture and, in turn, to drive decreases in patient mortality and morbidity.⁶ Safety culture relates to personnel's attitudes, common thoughts, and behaviours within an organization.¹⁰ Although not easy to measure, a number of surveys that assess safety culture have been published¹¹—alongside studies that investigate culture via ethnographic approaches and observation.^12,13 Survey instruments typically investigate a range of facets of culture, including team working,^14,15 communication,^16,17 and attitudes to safety.¹⁸ Studies to date have linked occurrence of patient safety incidents with safety culture and hence tools to monitor culture within hospitals have been implemented.^11,19

To date, the effects of the WHO SSC have been evaluated regarding compliance,²⁰ communication,^21,22 staff attitudes, and partly safety culture.^18,22,23 Published studies are typically pre-/post-implementation designs without control groups. The primary aim of this study was to measure the effects of the WHO SSC on operating theatre personnel perceptions of safety culture using a controlled study design. We hypothesize that implementation of the SSC is associated with positive changes in safety culture.

Methods

The study was reviewed by the Regional Committee for Medical and Health Research Ethics (Ref: 2009/561) and the hospital privacy Ombudsman, who approved it (Ref: 2010/413). Written informed patient consent was waived. Operating theatre personnel gave consent by responding to the surveys.

Study design

This was a prospective controlled intervention study using pre- and post-intervention surveys with the intervention and control groups. The primary outcome was the changes of safety culture perceptions in operating theatre personnel after implementation of the Norwegian version of the WHO Surgical Safety Checklist, introduced after WHO guidelines.⁵ A randomized stepped wedge design²⁴ was utilized to determine the order of intervention introduction across three surgical specialities (orthopaedic, thoracic, and neurosurgery—see the following section for details) in the intervention site of the hospital. Compliance with checklist usage was the secondary outcome.

Study population

The study took place in Haukeland University Hospital, a 1100-bed tertiary university hospital in the western part of Norway. The perioperative setting comprised 10 surgical departments and the accompanying departments of anaesthesia and intensive care administering anaesthesia and perioperative care. The target population of perioperative personnel included all eligible surgeons, anaesthetists, operating theatre nurses, nurse anaesthetists, and ancillary personnel (unit assistants, clerks, and cleaning assistants) located at two separate sites. The intervention group comprised personnel from orthopaedic surgery, thoracic surgery, and neurosurgery placed at the central hospital site. The control group comprised personnel from ear, nose, and throat; maxillofacial; plastic; endocrine; urology; gastrointestinal; obstetric; and gynaecological surgery specialities located at the peripheral hospital site. Within the hospital, operating theatre clinical and other personnel work in the separate sites without rotation, except for a few anaesthetists. Inclusion was based on work list information. A census approach was taken for recruitment—with the entire target population (as described above) invited to take part in the study. A total of 349 participants responded at baseline and 292 responded at post-intervention.

Study procedure

The study was carried out over 9 months from October 2009 to July 2010. Baseline and post-Checklist intervention survey data were collected during two 4 week periods in October 2009 and June 2010 (Fig. 1). The surveys were forwarded to the operating theatre personnel using both hospital electronic mail and the internal mail system (i.e. hardcopies). Identification numbers were assigned to or printed on each questionnaire to match individuals for the pre- and post-intervention surveys. Compliance with the Checklist was prospectively recorded (i.e. Checklist ‘used’ or ‘not used’) via the computer-based operating planning system within the operating theatres of the hospital. Nurse anaesthetists and theatre nurses also checked manually whether the paper versions of the Checklist had been completed for every case.

Fig 1.

Study procedure for the SSC intervention and the pre-intervention and post-intervention surveys at Haukeland University Hospital, Bergen, Norway, in 2009–2010. N, subjects included; n, subjects responded.

Checklist intervention

The Norwegian version of the SSC was introduced using a randomized sequential roll-out of the intervention.²⁴ In a joint venture between the Norwegian National Unit for Patient Safety, the Health Trust of Førde, and the Surgical Safety Study Group of Bergen, the Checklist was translated and adapted to meet Norwegian surgical flow of care. The Checklist consisted of 20 items orally confirmed by operating theatre personnel aimed at ensuring patient safety during anaesthesia and surgery. It was performed at three critical junctures in care: before induction of anaesthesia (Sign In), immediately before incision or start of treatment (Time Out), and before the leading surgeon left the operating theatre after surgery (Sign Out).⁵ The Sign In part before anaesthesia induction was led by the nurse anaesthetist. The Time Out and Sign Out parts were led by the circulating nurse. A completed checklist form was included into the patient's notes.

With management leaders support, the Surgical Safety Study Group of Bergen introduced the SSC to all specialities/professional groups in the intervention group, using an educational programme consisting of lectures, the NHS (UK) videos on how to perform and not to perform the checklist,²⁵ information disseminated via e-mails, and WHO guideline material⁵ translated into Norwegian. The SSC was piloted during the two first weeks of implementation resulting in a few minor adjustments—including that the Sign In should be led by the nurse anaesthetist with anaesthetist and operating theatre nurse present before induction, and the Time Out should be performed by the operating theatre nurse as the surgeon was ready to start the operation—pausing the whole team. Further feedback was received by end-users 2 weeks and also 2 months post-initial implementation. The randomized stepped wedge implementation started with orthopaedic surgery followed by thoracic and neurosurgery at 4 week intervals in the intervention site of the hospital.

Outcome measures

The primary outcome was post-implementation changes in safety culture measured by the Norwegian version of the Healthcare Research and Quality Hospital Survey on Patient Safety Culture (Hospital SOPS).^19,26–28 The survey instrument measures hospital staff perceptions of safety culture using 42 items that cover 12 factors, or elements of culture: ‘overall patient safety’, ‘frequency of events reported’ (including near misses in theatres), ‘unit manager/leader promoting safety’, ‘organizational learning—continuous improvement’, ‘teamwork within units’, ‘communication openness’, ‘feedback on error reported’, ‘non-punitive response to errors’, ‘adequate staffing’ (to handle difficult situations in theatre), ‘hospital manager/leader promoting safety’, ‘teamwork across units’, and ‘quality of information handoffs and transitions of care’.²⁸ The first nine factors address culture at clinical unit level, whereas the last three factors address culture at the wider hospital level. Items are scored on five-point agreement scales (1, strongly disagree, to 5, strongly agree) or frequency scales (1, never, to 5, always) as appropriate.^19,28 The Hospital SOPS instrument had previously not been used within this hospital. The instrument was selected based on its very good psychometric properties^27–29 and also because we could compare our findings with previous findings from similar populations assessed using the same tool.^27,28

Statistical analyses

Statistical analyses were carried out using SPSS version 20 (SPSS, Chicago, IL, USA). The reliability of the Norwegian Hospital SOPS instrument in the form of internal consistency was assessed using Cronbach's α coefficients. Descriptive statistics quantified sample characteristics and compliance data. Each of the 12 patient safety culture factors was based on three or four items, which were aggregated to produce a mean score for the factor. Negatively worded items were reversed to ensure that higher scores overall indicate better safety culture. A mean sum score was calculated across all 12 factors. The intervention and control groups were compared using analysis of variance (anova). We used a hierarchical mixed linear model (MLM) based on multiple regression analysis to calculate effects of the SSC intervention. Following subjects responding both at baseline and post-intervention, the MLM test allows for inclusion of subjects responding only at baseline or post-intervention.³⁰ The regression model is detailed in the Appendix. Variations between responders and non-responders were assessed with the Pearson χ² test. Statistical significance was set at two-tailed P≤0.05.

Results

Sample characteristics

A total of 641 participants took part in the study. Overall response rates for the two phases of the study were 61% (349/575) at the baseline/pre-intervention survey and 51% (292/569) at the post-intervention survey. Subjects responding in both surveys represented 67% (432/641) of the respondents. Detailed sample characteristics are displayed in Table 1. We performed a χ² analysis of non-responders to establish possible differences with responders regarding gender, groups, and profession and found a significant variation (P<0.01) for professions in both surveys, with fewer non-responding nurse anaesthetists in the pre- and post-intervention surveys and more surgeons and ancillary personnel as non-responders in the post-intervention survey. Checklist compliance for the study period was 85% of all cases (elective and emergency surgery) for the Sign In, 84% for the Time Out, and 77% for the Sign Out (Table 2).

Table 1.

Characteristics of the intervention (WHO SSC) and control groups (n=641)

	Pre-intervention survey						Post-intervention survey
	Checklist		Control		Total	P-value	Checklist		Control		Total	P-value
	n	%	n	%	n		n	%	n	%	n
Occasion and groups	146	41.8	203	58.2	349	–	140	47.9	152	52.1	292	0.21
Gender
Male	64	43.8	80	39.4	144	0.44	65	46.4	52	34.2	117	0.04
Female	82	56.2	123	60.6	205		75	53.6	100	65.8	175
Profession						<0.01						<0.01
Surgeon	44	30.1	83	40.9	127		32	22.9	60	39.5	92
Operating theatre nurse	35	24.0	42	20.7	77		36	25.7	35	23.0	71
Anaesthetist	24	16.4	20	9.9	44		31	22.1	10	6.6	41
Nurse anaesthetist	43	29.5	33	16.3	76		41	29.3	30	19.7	71
Ancillary personnel	—	—	25	12.3	25		—	—	17	11.2	17
Patient contact						0.02						0.21
Yes	140	98.6	182	90.5	322		128	96.2	140	92.2	269
No	2	1.4	19	9.5	21		5	3.8	12	7.8	17
Weekly working hours
<20	2	1.4	15	7.5	17		1	0.7	5	3.3	6
20–37	59	40.7	80	39.8	139		53	38.4	63	41.9	116
>37	84	57.9	106	52.7	190		84	60.9	83	55.0	167
Hospital experience (yr)						0.97						0.43
<1	7	4.8	9	4.5	16		9	6.5	7	4.7	16
1–5	34	23.4	50	25.3	84		25	18.1	42	28.2	67
6–10	28	19.3	36	18.2	64		28	20.3	28	18.8	56
11–15	28	19.3	38	19.2	66		26	18.8	21	14.1	47
16–20	20	13.8	22	11.1	42		19	13.8	17	11.4	36
21–40	28	19.3	43	21.7	71		31	22.5	34	22.8	65

Table 2.

Compliance with the WHO SSC in orthopaedic, thoracic, and neurosurgical operations (N=2367) at Haukeland University Hospital, Bergen, Norway, in 2009–2010

Surgery	Use of SSC
	N	Sign In		Time Out		Sign Out		All parts
		n	%	n	%	n	%	n	%
Orthopaedic	1579	1414	90	1386	88	1307	83	1264	80
Thoracic	393	337	86	338	86	300	76	287	73
Neuro	395	264	67	257	65	225	57	216	55
All	2367	2015	85	1981	84	1832	77	1767	75
P-value		<0.01		<0.01		<0.01		<0.01

Norwegian ‘Hospital SOPS’ reliability

Reliability was assessed at baseline (n=349) with lowest Cronbach's α of 0.60 and 0.64 for ‘adequate staffing’ and ‘organizational learning and continuous improvement’ and with the α ranging between 0.67 and 0.85 for the remaining factors. At post-intervention (n=292), the lowest α was 0.60 for ‘non-punitive response’ and ranged from 0.66 to 0.85 for the other factors. Overall, these are acceptable to very good levels of reliability for research purposes.

Checklist intervention effects on safety culture

Detailed descriptive analyses across the two groups (control vs intervention) and the two time-points (pre-intervention vs post-intervention) are presented in Table 3.

Table 3.

Descriptive statistics analyses in the intervention (checklist) compared with control groups for the pre-intervention (n=349) and post-intervention (n=292) phases of the study. se, standard error; CI, confidence interval

Safety factors (scale 1–5)	Pre-intervention			Post-intervention
	Mean	se	95% CI	Mean	se	95% CI
Overall patient safety
Intervention	3.63	0.05	3.53, 3.73	3.69	0.04	3.60, 3.77
Control	3.51	0.04	3.42, 3.60	3.57	0.05	3.48, 3.66
Frequency of events reported
Intervention	2.93	0.07	2.80, 3.06	2.77	0.06	2.66, 2.89
Control	2.72	0.05	2.62, 2.82	2.80	0.07	2.67, 2.93
Unit manager promoting safety
Intervention	3.78	0.06	3.66, 3.90	3.70	0.06	3.56, 3.82
Control	3.56	0.06	3.44, 3.67	3.52	0.07	3.38, 3.65
Organizational learning
Intervention	3.43	0.05	3.34, 3.53	3.50	0.05	3.41, 3.60
Control	3.27	0.05	3.18, 3.37	3.33	0.06	3.22, 3.45
Teamwork within units
Intervention	3.66	0.05	3.55, 3.76	3.72	0.05	3.62, 3.81
Control	3.55	0.04	3.46, 3.63	3.54	0.05	3.44, 3.64
Communication openness
Intervention	3.67	0.05	3.56, 3.78	3.61	0.06	3.50, 3.72
Control	3.52	0.04	3.43, 3.61	3.57	0.06	3.46, 3.68
Feedback/communication on error
Intervention	3.33	0.06	3.20, 3.45	3.21	0.06	3.08, 3.33
Control	3.07	0.05	2.98, 3.17	2.98	0.06	2.85, 3.10
Non-punitive response to error
Intervention	3.88	0.05	3.78, 3.98	3.89	0.04	3.80, 3.98
Control	3.68	0.05	3.57, 3.78	3.70	0.06	3.59, 3.82
Adequate staffing
Intervention	3.44	0.05	3.34, 3.54	3.58	0.05	3.48, 3.67
Control	3.35	0.05	3.26, 3.45	3.29	0.06	3.17, 3.40
Hospital management promoting safety
Intervention	2.80	0.06	2.69, 2.93	2.90	0.06	2.78, 3.02
Control	2.86	0.05	2.76, 2.96	2.95	0.06	2.83, 3.07
Teamwork across units
Intervention	3.06	0.04	2.97, 3.14	3.03	0.04	2.94, 3.11
Control	3.08	0.04	3.00, 3.15	3.13	0.04	3.05, 3.21
Quality of handoffs and transitions
Intervention	3.03	0.05	2.93, 3.12	3.05	0.05	2.96, 3.15
Control	3.05	0.04	2.97, 3.13	3.17	0.05	3.08, 3.26
Sum mean score (12 factors)
Intervention	3.39	0.03	3.32, 3.45	3.39	0.03	3.32, 3.45
Control	3.27	0.03	3.20, 3.33	3.29	0.04	3.22, 3.36

The multivariate analysis with MLM demonstrated a significant effect (P<0.01) of the SSC intervention on the two factors ‘frequency of events (near misses) reported’ and ‘adequate staffing’ (Fig. 2). The effect is described by regression coefficients for the interaction as −0.25 [95% confidence interval (CI), −0.47 to −0.07] and 0.21 (95% CI, 0.07–0.35), respectively (Table 4). For instance, for ‘frequency of events reported’, there was an increase of 0.11 from pre to post in the control group, but a decrease in the Checklist group (0.11–0.25=−0.14). For ‘adequate staffing’, there is hardly any change (b₂=−0.07) in the control, but an increase in the Checklist group (−0.07+0.21=0.14). For the safety culture measured on the overall hospital level, the MLM analysis also showed a significant effect of the Checklist intervention for the factors ‘hospital management promotes safety’ and ‘quality of information handoffs and transitions of care’ in both groups. The regression coefficients for the differences were 0.12 (95% CI, 0.04–0.20) and 0.08 (95% CI, 0.02–0.14), respectively. The same pattern was obtained when we adjusted the analyses using profession, gender, level of patient contact, and work experience as covariates. Subgroup analyses with covariates did not change the results.

Fig 2.

WHO SSC impact on safety culture perceptions of ‘frequency of events reported’ (near misses in theatres) and ‘adequate staffing’ (to be able to handle any difficult situation in theatre), before and after the SSC intervention at Haukeland University Hospital, Bergen, Norway in 2009–2010. CI, confidence interval.

Table 4.

Effects on safety culture factors of the intervention (checklist) compared with control groups and pre-intervention (n=349) vs post-intervention (n=292) survey phases estimated by the linear mixed model. ^†β_i (i=1, 2, 3), estimated regression coefficients; β₀, estimated mean/constant; CI, confidence interval; NS, not significant and not included; ^†With interaction: y=β₀+β₁·checklist group+β₂·post intervention survey+β₃·checklist group×post intervention survey; without interaction: y=β₀+β₁·checklist group+β₂·post intervention survey+0. *P≤0.05; **P≤0.01

Predictors		Constant		Differences for group checklist vs control		Overall change post- vs pre-intervention survey		Checklist effect for group×survey (pre/post) interaction
Safety factors (scale 1–5)	Items	β0	95% CI	β1	95% CI	β2	95% CI	β3	95% CI
Overall patient safety	4	3.49	3.43, 4.57	0.14	0.03, 0.24*	0.06	−0.01, 0.13	NS	—
Frequency of events reported	3	2.71	2.61, 2.81	0.20	0.04, 0.35*	0.11	−0.02, 0.23	−0.25	−0.43, −0.07**
Unit manager promoting safety	4	3.54	3.44, 3.64	0.22	0.08, 0.36**	−0.05	−0.13, 0.03	NS	—
Organizational learning	3	3.27	3.19, 3.35	0.16	0.04, 0.27**	0.07	−0.00, 0.15	NS	—
Teamwork within units	4	3.54	3.47, 3.62	0.14	0.03, 0.25**	−0.01	−0.08, 0.06	NS	—
Communication openness	3	3.52	3.43, 3.60	0.11	−0.01, 0.23	0.02	−0.06, 0.09	NS	—
Feedback/communication on error	3	3.03	2.97, 3.16	0.24	0.10, 0.37**	−0.08	−0.16, 0.01	NS	—
Non-punitive response to error	3	3.65	3.57, 3.74	0.21	0.09, 0.33**	0.01	−0.07, 0.09	NS	—
Adequate staffing	4	3.33	3.25, 3.42	0.10	−0.04, 0.23	−0.07	−0.17, 0.03	0.21	0.07, 0.35**
Hospital management promoting safety	3	2.84	2.75, 2.94	−0.04	−0.17, 0.10	0.12	0.04, 0.20**	NS	—
Teamwork across units	4	3.09	3.02, 3.15	−0.05	−0.14, 0.05	0.03	−0.03, 0.09	NS	—
Quality of handoffs and transitions	4	3.06	2.98, 3.13	−0.04	−0.15, 0.07	0.08	0.01, 0.14*	NS	—
SUM score (mean)	42	3.26	3.20, 3.31	0.12	0.04, 0.20**	0.02	−0.02, 0.06	NS	—

Across both baseline and post-intervention, we found significant group differences between the Checklist intervention group and the control group, in favour of the intervention group, for the factors ‘overall patient safety’, ‘frequency of events reported’, ‘manager promoting safety’, ‘organizational learning-continuous learning’, ‘teamwork within units’, ‘feedback/communication about errors’, ‘non-punitive response to error’, and the ‘sum score’ (i.e. overall safety culture scale mean score)—details of these differences are presented as regression coefficients with 95% CI in Table 4.

Discussion

In this prospective controlled intervention study of the WHO SSC in Norway, the introduction of the Checklist was associated with rather small impact on patient safety culture (measured by the ‘Hospital SOPS’ scale). Overall, the intervention group scored higher on a number of baseline culture factors—but even taking this into account, we only found positive effects on two dimensions of patient safety culture: a significant decrease in ‘frequency of events reported’ and a significant improvement in perceptions of ‘adequate staffing’ in the Checklist intervention group. The decrease in events reported could be associated with a real mitigation of near misses or errors after the introduction of the WHO Checklist in the intervention group. The SSC effects change in theatre routines, such that team members may eventually be better prepared for anaesthesia and surgery, hence leading to fewer near misses. Improved safety processes in the operating theatre have been seen after SSC implementation, such as raised awareness in the operating team and foreseeing any errors or problems.^6,31 In fact, the SURPASS study quantified incidence of errors caught to 40.6% (2562/6313) of checklists, supporting the assumption that checklists prevent near misses and errors.³² Direct observational evidence would be required to further validate this finding.

The improvement in perceptions of having adequate staffing to handle difficult situations in theatre is more difficult to account for. During the study period, there was no objective increase in staffing as an explanatory variable. According to data from the hospital personnel system, the number of active health personnel was constant during the study period and even the numbers on sick leave were not significantly different. It is possible that this effect is entirely subjective—staff's perceptions of teamwork have been shown to be associated with measures of safety and quality in patient care,³³ and Bõhmer and colleagues²² found that the team introductions during the Time Out contributed to improved staff satisfaction. The use of the SSC, including team introductions, might have enhanced team cohesion and thus affected subjective perceptions of staffing. This finding clearly requires further investigation.

Checklist and safety culture

Is it possible for operating theatre teams to adopt a practice which seems important to them but without broader improvements in their attitudes to and perceptions of safety? The safety culture factors that did not improve post-implementation of the SSC in this study are somewhat different from findings in other studies, especially regarding teamwork^15,16 and communication.^21,22,34

For checklist implementation to be effective, a concurrent cultural change within organizations has been suggested to be crucial^6,35,36—indeed, it has been argued that poor organizational culture and deeper-running problems can undermine the effectiveness of interventions like checklists.⁹ Interestingly, the compliance rates with the SSC in this study were rather high (85%, 84%, and 77% for the Sign In, Time Out, and Sign out phases, respectively) which indicate fairly successful early implementation. Anecdotal evidence observed by and also relayed to the research team also concurred that there were no major problems. This compares favourably with findings from other countries like the UK pilot implementation of the SSC, which was met with some resistance and compliance ranged from 42% to 80%.³¹ Strategies for successful implementation of the SSC have included education (training and materials), champions, organizational leadership, clear roles in the team, regular audits, feedback, and local adaptation^5,31,36—which are all elements that we used during implementation.

We thus have a rather paradoxical effect of a reasonably successful introduction of the SSC intervention but no major cultural impact. A number of explanations could be put forward here—all of which are amenable to further study. A first possibility is that culture and SSC are unrelated—but in the light of previous evidence, this is not the likeliest possibility. Secondly, the baseline culture levels of these services were already high—hence a ceiling effect prevented further improvements. A third, related possibility is that the timeline was too short to obtain such an improvement—after all introducing a new procedure is fairly quick, whereas a shift in experienced professionals' mind sets regarding their organization and practice might require a longer gestation period. Both of these explanations require longitudinal ongoing evaluations of culture and its fluctuations—and linking these with Checklist utilization. Cross-sectional studies between different countries and healthcare systems currently using the Checklist (e.g. Norway and UK) would also be useful in this respect. Further, observational assessments of how the SSC is actually used within the pressurized theatre environment are also required—culture measures are useful, but they cannot account for people reporting one thing yet doing another. A fourth explanation is that people can change their behaviour without necessarily visibly changing their underlying attitudes. Psychological theory suggests that this cannot hold for a very long time, as people strive to be consistent between their attitudes (i.e. perceptions of culture) and their behaviour³⁷ (i.e. usage of checklist)—which makes more compelling the longitudinal evaluation of both behaviour and culture perceptions.

Limitations and strengths

The response rate at baseline (61%) and at post intervention (51%) might be a limitation for sample representativeness. There were differences in professional backgrounds between responders and non-responders (but not for other patient characteristic factors). The significant differences within groups and variations within professions could indicate study weaknesses, thus the MLM analysis³⁰ adjusted for these—and indeed inclusion of the covariates in the analyses did not influence the results. Finally, information about the Checklist intervention and local enthusiasm could have been transferred to individuals in the control group and biased results (‘spill-over’ effect)—which is something that could not be controlled. In balance, key strengths of this study are the use of a carefully controlled design and matched assessments of safety culture pre- and post-intervention at the individual participant level (rather than group level).

Implications

Our findings, and overall experience with the study, have implications for the introduction of safety interventions, like the SSC, and for further research on the effectiveness of such interventions. In this study, seven of the safety culture factors showed overall significant differences between the intervention and control groups, with the intervention group being significantly more positive. Following WHO advice, one could advocate that implementation of an intervention should begin with healthcare teams or professionals who are positive towards the intervention—hence the concept of ‘champions’. This, however, might be a challenge when designing evaluation studies that include a control group, as entire units or operating theatres that are more positive towards the intervention might show a ‘ceiling effect’—that is, the size of the improvement triggered by the intervention is smaller in these groups precisely because they are more positively predisposed to the intervention to start with. Pre-/post-intervention designs are not the most suitable to tease out such effects—and indeed, we would argue that a deeper understanding of how exactly a healthcare organization moves across dimensions of culture over time cannot be gauged by such studies. We would thus advocate periodic and systematic assessments of an organization's culture using a well-validated instrument. This will allow longitudinal, time-series-based evaluation of whether the organization (or parts of it) moves in a certain direction, and whether interventions are causing such shifts.

Further, feedback of such measures within the organization can provide better self-insight and allow clinical units to self-evaluate and to compare themselves with their peers. We would hypothesize that such organization-wide assessments are an intervention in themselves and that a positive relationship should be expected between them and organizational readiness to improve safety and quality and to adopt novel interventions. These hypotheses await further study.

Declaration of interest

S.H. is the leader of the advisory board for the Norwegian Patient Safety Campaign hosted by the governmental and non-commercial Norwegian Knowledge Centre for the Health Services.

Funding

This work was supported by research grants from the Western Regional Health Trust of Norway (grant numbers 911635, 911510). N.S. and C.A.V. are affiliated with the Imperial Centre for Patient Safety and Service Quality, which is funded by the National Institute for Health Research, UK. Funding for open access charge: the Western Regional Health Trust of Norway.

Appendix

The aim of the MLM regression analysis is to describe the effect of the checklist intervention on primary outcome; the safety culture factors of the Hospital SOPS (dependent variables) with intervention and control groups at baseline and post-intervention surveys (independent variables).

A hierarchical model was formed for the interaction and expressed as:

(A1)

where is the model constant/intercept, the regression coefficients (estimated by restricted maximum likelihood methods), Group the independent variable as intervention (=1) or control (=0) group, and Survey the independent variable as baseline survey (=1) and post-intervention (=0) survey.

The models assume covariance type CSR (compound symmetry with correlation parameterization) for repeated response from subjects at baseline and at post-intervention. The analyses also include respondents replying only at baseline or post-intervention. Independent variables were fixed. To adjust for co-variables, we included profession, gender, patient contact, and work experience in the hospital in the models:

(A2)

For safety culture factors without significant interaction effects of the checklist intervention, we used an equation for assessing the variations between groups and between surveys:

(A3)

Equations (A1) and (A3) are used in Table 4.