Improving the WHO Surgical Safety Checklist sign-out

Abstract

Background

The WHO Surgical Safety Checklist has been shown to reduce perioperative morbidity and mortality worldwide. There is evidence to suggest that sign-out is the most poorly performed phase of the checklist as it coincides with a period of high workload for team members. This study aimed to see whether modification of this process might result in greater compliance.

Methods

A controlled longitudinal (before and after) study was performed to evaluate the effect of a modified checklist sign-out on compliance in a single surgical department. Checklist quality was evaluated by measurement of checklist completion, active participation, and team member presence. Workload assessment was performed to identify the optimal moment for the sign-out process. The sign-out process was modified through an iterative multidisciplinary approach, informed by results from the workload assessment. Feedback was obtained through staff surveys.

Results

A total of 185 operations were used, with an intervention group in vascular surgery and a control group in orthopaedics. The optimal timing for sign-out was identified as after final wound closure. The modified sign-out process improved active participation of team members (21 of 34 versus 31 of 34; P = 0.010). In the control group, complete compliance improved (48 of 76 versus 30 of 41; P = 0.041). However, active participation decreased (53 of 76 versus 19 of 41; P = 0.022). No differences were noted between groups in team member presence. Eighteen of 21 staff questioned viewed the modifications positively.

Conclusion

The optimal sign-out timing was identified as immediately after final wound closure prior to undraping the patient.

In an analysis using a workload-based approach, it was found that the optimal WHO checklist sign-out timing was during final wound closure. This finding was used to reiteratively modify the WHO checklist sign-out process, leading to improved active participation between team members.

Introduction

The WHO Surgical Safety Checklist (SSC) was launched in June 2008 to standardize perioperative care and optimize patient safety¹. There is robust evidence to suggest that it reduces inpatient mortality². The checklist works by providing pauses during three crucial time points during any surgical procedure, allowing team members opportunities to communicate key issues regarding patient safety before proceeding. Over the years, significant benefits of implementing the checklist have been reported, showing reductions in complications and deaths associated with surgery³. As such, the WHO checklist has become a mandatory process adopted by most hospitals internationally.

Direct observational studies^3,4 have shown that compliance with the WHO SSC can be suboptimal, in stark contrast to hospital audits reporting absolute compliance⁵. The sign-out process was associated with the poorest compliance rates, regardless of surgical subspecialties^3,5–8. This process occurs at the end of the surgery to provide a final checkpoint for the surgical team to raise any unresolved issues before the patient leaves the room. This phase often coincides with a period of increased workload for nursing and anaesthetic staff. The design of the sign-out phase of the SSC contributes to the pressure faced by these team members, and may lead to disengagement and inadequate communication.

Conscientious adoption of the checklist is an integral part of its effectiveness⁹. In times of high workload, it may prove invaluable in providing a framework for performing required activities. The present study sought to evaluate a redesigned sign-out component of the checklist, based on workload assessment, in the hope that this might address ergonomic issues and improve team dynamics. The objectives of this study were to determine an optimal time for performing the sign-out process, modify the sign-out component of the WHO SSC, and assess its effectiveness in improving quality of compliance, as measured by checklist completion, presence of every team member, and their active participation.

Methods

The study was approved by the Oxford University Hospitals Research and Development Ethics Committee (20032017-NOTSS-TULLY) and registered within the Trust data management system, Datix (Datix, London, UK).

Before initial baseline observations, six clinical observers underwent a training session and scoring process to ensure interobserver reliability. This involved a written assessment where observers were presented with several scenarios relating to use of the WHO SSC. Each observer scored the quality of compliance with the WHO SSC with respect to three parameters during all three phases: checklist completion (total, partial, or incomplete), presence of every team member (yes or no), and their active participation (yes or no), as established previously⁴.

The protocol for recording observational data has been described previously⁴. Baseline results were obtained by direct observation of a variety of elective surgical procedures from two surgical departments (orthopaedics and vascular surgery), located separately in two tertiary referral hospitals within a single National Health Service (NHS) Trust, for 3 months (April to June 2017). All theatre staff included in the study were informed before data collection, but were not told which part or components of the WHO SSC process were being assessed.

In the 2-month planning period (July to August 2017), two teams, each comprising one clinically trained and one human factors (HF)-trained observer, undertook seven workload assessments that were used to inform the redesign process. A HF-trained observer is an individual with Masters level qualification in HF methodolgies who was trained to undertake specific HF observations. The assessments were conducted during the final phase of the surgical procedure, in the department in which intervention was to be implemented (vascular surgery).

To determine the optimal time at which all team members were most able to participate in sign-out, a workload assessment was performed using a nine-point integrated workload scale (IWS). An IWS score of more than 4 denoted that individuals were not able to participate fully in the sign-out process¹⁰. The workload assessments identifed the optimal time to perform sign-out as after final wound closure but prior to undraping the patient.

Together with the results of the workload assessment, the checklist process was modified during discussions in multidisciplinary meetings and modified iteratively through trial runs during the planning period. During this process it was decided that the sign-out be broken into two phases, one in theatre and the other in recovery to promote direct communication between the surgeon, anaesthetist and recovery nurse regarding immediate considerations for post-operative management.

A further 3-month postintervention collection period (September to December 2017) was subsequently carried out using a similar protocol, comparing the final design of the checklist process in the intervention group (vascular surgery) with the standard WHO SSC process in the control group (orthopaedics).

Anonymous surveys, detailing self-reported perceptions of the modified checklist, were filled out by staff members in the intervention group. Three interviews were conducted, representing the views of a surgeon, an anaesthetist, and one recovery nurse who were involved in the intervention group.

Statistical analysis

Both intergroup and intragroup comparisons were performed using χ² tests. P < 0.050 was considered statistically significant. All analyses were undertaken using SPSS^® version 22 (IBM, Armonk, New York, USA). Survey results are presented descriptively.

Results

Interobserver reliability

There was good interobserver agreement for all six clinical observers in scoring checklist completeness and team member presence (82 per cent, κ = 0.65 (95 per cent c.i. 0.60 to 0.66) and 93 per cent, κ = 0.67 (0.52 to 0.72) respectively). There was moderate interobserver correlation for the parameter pertaining to active participation (90 per cent, κ = 0.48 (0.46 to 0.49)).

Workload assessment

Before data collection, HF observers underwent a marking validation process based on 121 min of observed activity. There was good interobserver agreement between the two HF observers (ρ = 0.868, P < 0.005).

Seven sign-out processes were observed within the proposed intervention group. Clinically trained observers identified three key consistent events that could be used as process milestones on which to base observations: nursing count complete (CC), which occurred before superficial wound closure, final wound closure complete (FWCC), and transfer of patient to trolley (TOP).

Four-minute time blocks around each of these key events were analysed using the IWS-9. As there were seven sign-out processes observed, a total of 28 min was analysed for each of the three events. All team members were available for 3 min of the 28-min interval in CC, 10 min for FWCC, and 2 min for TOP. Considering only both surgeon and scrub nurse availability, the availability was 5 min of the 28-min interval for CC, 11 min for FWCC, and 12 min for TOP. Therefore, the optimum time to perform sign-out was during FWCC.

Evolution of the modified WHO Surgical Safety Checklist

The results of the workload assessment were presented to a multidisciplinary group, which agreed unanimously that FWCC was the best time to commence sign-out. The modified sign-out phase was further subdivided into two phases: one in the operating theatre, and the other in recovery (Fig. 1), allowing division of labour in two separate locations as well as reducing communication breakdown during transfer of the patient to the trolley.

Fig. 1.

Finalized version of the modified sign-out component of the WHO Surgical Safety Checklist

In the operating theatre component of the modified checklist, the circulating nurse initiates sign-out at the point of wound closure, after the needle holder or needle has been returned to the scrub nurse, but before undraping. The circulating nurse now has to engage specifically with the surgeon, followed by a verbal read-back by the surgeon. In the recovery room or ICU component, the surgeon, scrub nurse, and anaesthetist not only discuss but also document the recovery and management plan for the patient.

Compliance quality of sign-out before and after intervention

A total of 185 operations were observed across two NHS sites, between April 2017 and December 2017 (elective orthopaedics 117, vascular surgery 68) (Fig. 2).

Fig. 2.

Breakdown of quality of sign-out process in two surgical departments of one National Health Service Trust

The quality of checklist compliance is summarized in Fig. 3 and Table 1. After the implementation, there was a non-significant increase in complete compliance with the checklist (23 of 34 versus 29 of 34; P = 0.220). There was a statistically significant improvement in complete compliance in the control group (48 of 76 versus 30 of 41; P = 0.041). The presence of all team members improved significantly in the intervention group (28 of 34 versus 34 of 34 operations; P = 0.033), but not in the intervention group (55 of 76 versus 33 of 41; P = 0.456). The intervention group had improved active participation (21 of 34 versus 31 of 34; P = 0.010), whereas the control group had decreased active participation (53 of 76 versus 19 of 41; P = 0.022).

Fig. 3.

Frequency and quality of sign-out process before and after implementation of modified checklist

Table 1.

Sign-out quality parameters before and after implementation of modified checklist in vascular surgery (intervention) and elective orthopaedics (control)

		No. of procedures		P *
		Before implementation	After implementation
Vascular (intervention)		34	34
Compliance	Complete	23	29	0.220
	Partial	8	4
	Not done	3	1
All team members present	Yes	28	34	0.033
	No	6	0
Active participation	Yes	21	31	0.010
	No	13	3
Orthopaedics (control)		76	41
Compliance	Complete	48	30	0.041
	Partial	25	6
	Not done	3	5
All team members present	Yes	55	33	0.456
	No	21	8
Active participation	Yes	53	19	0.022
	No	23	22

^*χ² test.

Analysis of outcomes in interventional group

The effectiveness of the modified checklist was evaluated in its two new phases: in the operating theatre and in recovery. Compliance improved in both areas but this was not statistically significant in either (23 of 34 versus 29 of 34 (P = 0.153) and versus 30 of 34 (P = 0.079) respectively). Presence of all team members improved in both operating room and recovery stages but was only statistically significant in the operating room (28 of 34 versus 34 of 34; P = 0.033). Active participation significantly improved in both stages (21 of 52 versus 31 of 52 (P = 0.010) and versus 32 of 34 (P = 0.003) respectively) (Table 2).

Table 2.

Subgroup analysis of the two subphases of the modified checklist comparing sign-out quality parameters before and after implementation in vascular surgery

	Before implementation, theatre	After implementation, theatre phase 1	P *	After implementation, recovery phase 2	P *
Complete compliance	23 of 34	29 of 34	0.153	30 of 34	0.079
Team members	28 of 34	34 of 34	0.033	30 of 34	0.732
Active participation	21 of 52	31 of 52	0.010	32 of 34	0.003

^*χ² test.

Survey results

Surveys were completed by 21 of 36 staff members from the intervention group. A positive response corresponded to a score of 5 and above (Fig. 4). Of these respondents, 18 reported that they had a positive impression of the checklist modification, whereas 16 rated the modified checklist positively as a valuable addition as well as registering a self-reported improvement in team communication after implementation. Two-thirds were likely to use the modified checklist in future, but only 12 were actually aware of the safety benefits the modified checklist brought to the department.

Fig. 4.

Survey respondent breakdowns and self-reported results of modified checklist

(a) respondents by position (b) staff satisfaction with the modified WHO sign-out (c) staff perception of the value of the modified WHO sign-out (d) staff perception of increased safety to the patient since the introduction of the modified WHO sign-out (e) staff reported perceptions of change in communication post introduction of the modified WHO sign-out (f) staff reported likelihood of using the modified WHO sign-out in future.

Discussion

Use of this modified WHO SSC led to improvements in the quality of compliance during the sign-out process, compared with the standard WHO SSC. There was significantly improved team member presence, and active participation in the intervention group increased significantly, whereas the latter decreased in the control group. The absence of significant improvement in completing the checklist in the intervention group, compared with a significant improvement in the control group, might reflect insufficient training, the initial learning curve, a type II error owing to observer drop-out, and the timing of the second phase (lull period for elective procedures). Although there was an improvement in total completion of the checklist for both groups in the second phase, there was a decrease in attempted completion (total completion and partial completion) only in the control group, suggesting that there were more checklists left uncompleted in this group than at baseline (Fig. 3 and Table 1). There was positive feedback overall from staff who used the new checklist process.

The present study featured a novel approach using workload assessments to determine the optimal timing of the sign-out phase based on staff pressure^10–12. Data collection using the IWS-9 scale demonstrated high variability and unequal workload demands among operating theatre team members during sign-out, but also suggested that an optimal time to initiate WHO sign-out was after final wound closure. This was in contrast to a previous study¹³ that also highlighted the difficulty of finding the right timing for checks; the authors found that sign-out performance may be affected if left until after wound closure as senior surgeons may have departed. Senior surgeons may leave the operating theatre early when wound closure is left to a junior colleague. This may lead to difficulty in initiating the sign-out process as the anaesthetist is now the only senior member of the team at a time when they are focused on actively managing the patient after the wound has been closed. The sign-out process was therefore modified, with delegation of the task of initiating this phase to a scrub nurse who is least busy at that point^14,15. Subgroup analysis of the two newly introduced subphases during sign-out showed that active participation had improved significantly, implying that the workload assessment had identified a period during which all staff members were not too distracted with their tasks. Only the theatre subcomponent showed a statistically significant improvement in team member presence that might be explained by the addition of a new requirement for surgeons to be present to confirm the plan. Neither phase led to significant improvement in complete compliance, although this might again relate to staff training.

One of the strengths of the present study design was implementation of real-time direct observation to assess the quality of the checklist meaningfully. The results of this approach are in stark contrast to the 2018 audit outcomes in the same Trust, which reported 90–100 per cent compliance with the checklist¹⁶. A wider issue is the tick-box culture of such retrospective administrative audits, which has persisted possibly owing to the disciplinary and financial consequences of not meeting set standards, at the expense of active communication¹⁷. Another advantage was the co-design process that allowed wider involvement of all staff, including management and support staff. This fostered a closer working relationship between the research and clinical teams. The process of modifying the WHO SSC using workload assessments and an iterative co-design process may be generalized and adapted to other phases of the WHO SSC, and for use in other contexts where there are clinical workload pressures¹⁸.

A significant limitation of the present study was the Hawthorne effect, whereby the scrutiny of team members by direct observers might have led to better adherence than when unobserved. However, this bias might have been blunted by not informing the team members of the actual objectives of the study, and observers staying throughout the entire procedure. Another limitation is the interobserver reliability of clinical observers during all phases of data collection. Attempts were made to reduce this by introducing mandatory training sessions and tests before any data collection, although this was significantly offset by observer drop-out in the preintervention period. This was because medical students, who were used for their ability to blend into the operating theatre environment, had other clinical commitments that took priority¹⁹. The use of video analysis may overcome these issues. Another barrier was the difficulty in training all relevant staff because of variations in staff rostering and locum professionals. Although the surgeon often remained the same, the scrub nurse and anaesthetist were often different, which may have affected team dynamics^18,19. There was also an assumption that the sign-out process of the WHO SSC is the phase most affected by HF issues, based on previous studies in five hospitals across three NHS Trusts⁴. The current literature, however, indicates great variability in methods of measuring checklist compliance, rendering it difficult to determine whether the sign-out process is indeed the least completed section.

 

Presented in part to an international conference of the Association of Surgeons in Training, Edinburgh, UK, 2018