Abstract
Background
A Surgical Patient safety Observation Tool (SPOT) was developed and tested in a multicentre observational pilot study. The tool enables monitoring and benchmarking perioperative safety performance across departments and hospitals, covering international patient safety goals.
Methods
Nineteen perioperative patient safety observation topics were selected from Dutch perioperative patient safety guidelines, which also cover international patient safety goals. All items that measured these selected topics were then extracted from available local observation checklists of the participating hospitals. Experts individually prioritized the best measurement items per topic in an initial written Delphi round. The second (face to face) Delphi round resulted in consensus on the content of SPOT, after which the measurable elements (MEs) per topic were defined. Finally, the tool was piloted in eight hospitals for measurability, applicability, improvement potential, discriminatory capacity and feasibility.
Results
The pilot test showed good measurability for all 19 patient safety topics (range of 8–291 MEs among topics), with good applicability (median 97 (range 11·8–100) per cent). The overall improvement potential appeared to be good (median 89 (range 72·5–100) per cent), and at topic level the tool showed good discriminatory capacity (variation 27·5 per cent, range in compliance 72·5–100 per cent). Overall scores showed relatively little variation between the participating hospitals (variation 13 per cent, range in compliance 83–96 per cent). All eight auditors considered SPOT a straightforward and easy‐to‐use tracer tool.
Conclusion
A comprehensive tool to measure safety of care was developed and validated using a systematic, stepwise method, enabling hospitals to monitor, benchmark and improve perioperative safety performance.
Introduction
Improving patient safety remains a major public health concern1. Almost two‐thirds of in‐hospital adverse events are associated with perioperative care2. Improvement in perioperative patient safety can be achieved by observing and measuring safety performance, and by using the results to improve the perioperative process3, 4, 5 in a plan–do–check–act cycle. Benchmarking of perioperative safety performance may be used to compare departments, hospitals and methods, to improve patient safety.
National audits by the Dutch Health Care Inspectorate (Inspectie voor de Gezondheidszorg; IGZ) in 2007–2009 showed that perioperative care in the Netherlands could be improved by information transfer, clinical documentation, teamwork and coordination6, 7, 8. In response to the recommendations of the IGZ, national evidence‐based perioperative safety guidelines9, 10, 11 were developed between 2010 and 2012, applicable to both surgical and non‐surgical interventions in hospitals. Adherence to evidence‐based guidelines is associated with safer perioperative care and improved outcome12, 13, 14.
In 2012, the IGZ reaudited perioperative care and concluded that, although the perioperative process was better structured, there was still room for improvement. Furthermore, safety performance still varied considerably between hospitals. For example, although preoperative time‐out and postoperative sign‐out procedures were performed in all hospitals, approximately half of these procedures were incomplete. In addition, the IGZ announced a zero tolerance policy in cases of wrong‐side and wrong‐site surgery15.
There is a growing need for an observation tool that enables monitoring of guideline compliance, follow‐up and comparison between different specialties and hospitals, and identification of best practices. Until now, such a condensed and easy‐to‐use observation tool, developed and used by professional caregivers themselves, has not been available. The tool under development was designed to cover all 19 patient safety observation topics used by the IGZ for auditing Dutch hospitals (IGZ framework 2012) (Table 1)15. IGZ and guideline developers agreed that these topics represent the main safety observation topics. The topics are also in line with the six international patient safety goals highlighted by the Joint Commission International (JCI)16 as areas of concern in healthcare (Table 2).
Table 1.
Perioperative patient safety observation criteria used by the Dutch Health Care Inspectorate15
| Criterion | Topic | Details |
|---|---|---|
| 1 | Patient identification | Identification of the patient should be done using at least two out of three characteristics (patient name, date of birth, patient identification number). Identification is based on three independent sources: the (wakened) patient or legal representative, the patient's medical dossier and identification bracelet(s) |
| 2 | Handover from wards | It is strongly recommended to use structured checklists for handover of patients during the entire perioperative healthcare process |
| 3 | Pre‐time‐out before the operation | A pre‐time‐out is performed as an additional check when invasive preparations (e.g. regional anaesthetic block) for the operation are done in the preanaesthesia care unit of the OR. The attending anaesthetist together with a second attending person checks the following items: patient identification; type of operation; side and/or location of operation; possible allergies of the patient; the actual parameters of (anti)coagulation; and presence of appropriate materials. If possible, these checks are done with involvement of the conscious patient. The anaesthetist is responsible for performing and recording this safety verification hold |
| 4 | Time‐out before the operation | A time‐out is performed in the presence of the patient before induction of anaesthesia. The attending anaesthetist, surgeon and OR staff check the following items: correct patient; correct type of operation; correct side and/or location of the operation; the actual parameters of (anti)coagulation; necessity for administration of antibiotics; possible allergies of the patient; co‐morbidity of the patient; positioning of the patient during the operation; presence of appropriate personnel and materials; and other relevant details. The surgeon is responsible for performing and recording this safety verification hold |
| 5 | Intraoperative team communication | Communication between attending surgeon and anaesthetic team member(s) present is obligatory at the start and finish of the surgical procedure and at all points that are of interest to the condition or safety of the patient, or that will interfere largely with the activity of the other attending specialty |
| 6 | Handover at shift change during the operation | A handover is performed at all shift changes during the operation |
| 7 | Sign‐out after the operation | Under responsibility of the surgeon a sign‐out is performed in the OR, before the patient leaves the OR, in the presence of the full attending team. At least the following should be discussed and recorded: all essential information with respect to the operation performed; the results of the counting of used operation instruments and materials; and all items necessary for adequate postoperative care |
| 8 | Patient transport | During transport of the patient, bedrails are in the upright position or the patient is secured sufficiently |
| 9 | Handover to recovery room | The person responsible for patient transfer ensures the patient's vital parameters and leaves only once the patient has been adequately reconnected to monitoring equipment and shows stable vital signs |
| 10 | Discharge from recovery room | The postanaesthesia recovery score and pain score are documented in the patient's dossier at the time of discharge from the recovery room. Discharge of the patient from the recovery room is done once predetermined discharge criteria have been checked, recorded and met. The person responsible for discharge is documented in the patient's file |
| 11 | Discharge from the surgical ward | In the ward the postoperative pain score is measured and documented. The patient is discharged only once predetermined discharge criteria have been met and recorded. The person responsible for discharge is documented in the medical file. The patient is informed about the procedure performed and follow‐up treatment. Documentation on the surgical procedure is recorded in the medical file within 2 days after discharge from hospital |
| 12 | Safety of medical equipment | A date‐valid sticker concerning the support state of all medical equipment is visible |
| 13 | Counting used surgical equipment | Used surgical meshes, needles, instruments and disposables are counted by two people and the results are recorded in the patient's file |
| 14 | Behaviour with respect to infection prevention | The wearing of OR suits, clogs, surgical masks and caps, the handling of personal jewellery of attending team members, the performance of hand hygiene, OR door movements and preoperative removal of the patient's hair must be done according to existing national guidelines |
| 15 | Air ventilation and conditioning characteristics of the operating room | Adequacy of ventilation pressure in the OR must be displayed, with the facility to generate an alarm if malfunctioning. Positioning of the OR light and of the patient under the plenum must be done in accordance with optimal air‐conditioning characteristics for the prevention of wound infection |
| 16 | Prevention of transmission of hepatitis B and MRSA | Each institution has an assured procedure (in accordance with existing directives) for the prevention of transmission of hepatitis B and MRSA |
| 17 | Double‐check of medication | The preparation and administration of parenteral drugs are double‐checked |
| 18 | Propofol handling | Handling of propofol must be according to the directives of the Dutch Healthcare Inspectorate and manufacturer instructions |
| 19 | Drug storage | No date‐expired medication is stored in any local place that is in use for drug storage |
OR, operating room; MRSA, methicillin‐resistant Staphylococcus aureus.
Table 2.
Definition of the Joint Commission international patient safety goals16
| IPSG | Goal | Standard |
|---|---|---|
| 1 | Identify patients correctly | The hospital develops and implements a process to improve the accuracy of patient identification |
| 2 | Improve effective communication | The hospital develops and implements a process to improve the effectiveness of verbal and/or telephone communication among caregivers |
| 3 | Improve the safety of high‐alert medications | The hospital develops and implements a process to improve the safety of high‐alert medications |
| 4 | Ensure correct site, correct procedure, correct patient surgery | The hospital develops and implements a process for ensuring correct site, correct procedure, correct patient surgery |
| 5 | Reduce the risk of healthcare‐associated infection | The hospital adopts and implements evidence‐based hand hygiene guidelines to reduce the risk of healthcare‐associated infection |
| 6 | Reduce the risk of patient harm resulting from falls | The hospital develops and implements a process to reduce the risk of patient harm resulting from falls |
IPSG, international patient safety goal.
Several methods are available to measure patient safety and identify risks. Record review is the most frequently used and a widely accepted method of measuring incidence rates of adverse events17, 18. However, the reliability of record review is often considered only moderate, and there is no evidence that record review really detects all adverse events19. Direct observation methods of patient care capture valuable (real‐time), accurate and precise patient data20, 21. A recent study22 of unannounced JCI accreditation surveys showed that changes in practice during periods of surveyor observations in US hospitals decreased patient mortality significantly. Patient tracer observations are seen internationally as a robust method of measuring patient safety performance in daily practice23. In patient tracer methodology auditors follow representative patients from a specific hospital's patient population to evaluate compliance with standards16, 23. During each tracer procedure, the auditor(s) systematically observe and document care aspects according to: the course of care, treatment and services provided to in‐hospital patients; the interrelationships between and among disciplines and departments; and potential concerns in the patient care process.
The aim of the present multicentre observational pilot study was to develop a comprehensive and easy‐to‐use Surgical Patient safety Observation Tool (SPOT) to measure perioperative safety performance in daily clinical practice and to identify safety improvement areas. The development procedure included pilot testing of SPOT in eight hospitals to assess the clinimetric characteristics in line with previous studies24, 25, 26, 27. Feasibility was added to these characteristics (Table 3).
Table 3.
| Criterion | Definition | Score |
|---|---|---|
| Measurability | Measurable elements are measurable by observation | Good: at least 80 per cent of elements within the topics |
| Moderate: more than 20 to less than 80 per cent of elements within the topics | ||
| Poor: 20 per cent or less of items within the topics | ||
| Applicability | Measurable elements are applicable to the selected patients/procedures. | Good: at least 80 per cent of elements within the topics |
| Poor: less than 80 per cent of elements within the topics | ||
| Improvement potential | Room for improvement of current practice (topic level) | Good: compliance with the standard less than 90 per cent |
| Poor: compliance with the standard at least 90 per cent | ||
| Discriminatory capacity for comparison | Discrimination of practice performance (compliance with the standards) between different topics and between departments or hospitals | Good: more than 20 per cent variation between lowest and highest scores |
| Poor: 20 per cent or less variation between lowest and highest scores | ||
| Feasibility | Easy to use and applicable without the help of others (after instruction) | Good: at least 90 per cent of the auditors |
| Moderate: 50–90 per cent of the auditors | ||
| Poor: less than 50 per cent of the auditors |
Methods
No ethical approval was considered necessary, as no intervention was performed for a specific patient care process. A RAND‐modified Delphi procedure28, 29 was performed to reach consensus on the content of a standardized SPOT. The procedure consisted of the following steps.
Recruiting hospitals and collecting surgical patient safety observation checklists
The SPOT study was introduced in 2014 to all eight Dutch academic hospitals associated with the Netherlands Federation of University Medical Centres (NFU) and to all 14 member hospitals of the Dutch Safe Curative Care Association (Vereniging Veilige Curatieve Zorg; VVCZ), which includes one NFU academic hospital. VVCZ member hospitals help one another to improve patient safety. All 21 hospitals (8 academic, 7 tertiary care and 6 regional care hospitals) were invited and agreed to participate in the study. The hospitals were asked to share locally used surgical patient safety observation tools and checklists.
First Delphi round: item selection by experts
From the collected hospital checklists, items covering one or more of the 19 perioperative patient safety observation topics based on national and international perioperative guidelines, and used by the IGZ, were selected. These items, grouped per topic, were digitally presented to a multidisciplinary group of experts (professionals selected by the participating hospitals with specific interest and qualifications in perioperative patient safety, and locally identified as experts in this area). In total, 13 perioperative experts participated in a first Delphi round: five physicians, five operating room (OR) staff members, two ward nurses and one medical technician (Table S1, supporting information). These experts received a preselected list of topics depending on their specific expertise and relevance to their function. They ranked the observation items by making a personal top three list per topic. As well as ranking, the experts were also invited to comment on the content and phrasing of the items. The results of this step were used for reaching consensus on content and phrasing of the items for the final observation tool.
Second Delphi round: discussing the results of the first Delphi round
The scores and suggestions for phrasing of the items by the expert panel were processed and analysed. Observation items rated in the top three ranking received 10, 5 or 2 points respectively. The number of experts eligible to score topic items ranged from four to seven per topic. Thus, the potential maximum score of a topic item ranged from 40 to 70 points (Table S1, supporting information). All items that scored at least 50 per cent of the maximum were presented in a face‐to‐face consensus meeting. To create broad support, a second expert panel of ten experts in perioperative patient safety from VVCZ member hospitals (3 OR managers, 1 hygiene and infection adviser, 2 quality and safety managers, 1 anaesthesia nurse, 1 OR nurse, 1 physician and professor in perioperative patient safety, and 1 senior researcher – NFU member hospitals were not able to participate in this second Delphi round) discussed the results in order to achieve consensus regarding the content of the final SPOT tracer list.
Pilot testing of the Surgical Patient safety Observation Tool
The SPOT tracer list was pilot‐tested in one academic, four tertiary care and three regional care VVCZ member hospitals between September 2015 and April 2016 to assess the clinimetric characteristics of measurability, applicability, improvement potential, discriminatory capacity and feasibility according to predefined criteria (Table 3). Eight experienced auditors (3 physicians and 5 non‐physicians) assessed the feasibility and applicability of SPOT. For this last assessment, auditors responded to two statements – ‘SPOT is easy to use’ and ‘I am able to use SPOT independently’ – using a 6‐point answering scale (1, totally disagree; 6, totally agree). To facilitate interpretation, the answer categories of 5 (agree) and 6 (totally agree) were combined to express good feasibility.
Results of the observations are presented in a descriptive way, as numbers and percentages. The various scoring definitions of the clinimetric characteristics are presented in Table 3.
Results
Participating hospitals and local surgical patient safety observation checklists
Twenty‐one hospitals joined the study and delivered their local perioperative observation checklists covering a large variety of observation items related to perioperative patient safety. Four hospitals (2 academic, 1 tertiary care and 1 regional care hospital) did not use a perioperative patient safety observation checklist, but were performing specific in‐depth audits on different patient safety topics such as ‘handover’, ‘medication safety’ and ‘infection prevention’. The other 17 hospitals (6 academic, 6 tertiary care and 5 regional care hospitals) did use a perioperative patient safety observation checklist for internal auditing (Table S2, supporting information). The topics ‘behaviour with respect to infection prevention’ (15 hospitals), ‘time‐out before the operation’ (14) and ‘double‐check of medication’ (13) were most frequently represented in the obtained checklists. Least represented topics were ‘intraoperative team communication’ (3) and ‘discharge from the surgical ward’ (3) (Table S2, supporting information).
First Delphi round: item selection
In the first Delphi round, 13 experts independently and anonymously prioritized the observation items by means of a personal top three per topic. Items in two topics, ‘safety of medical equipment’ and ‘counting used surgical equipment’, failed to reach the 50 per cent criterion (the cut‐off point for discussion in the second Delphi round) because of large variation in experts' ratings. For these topics, considered too important to exclude, items having at least 20 per cent of the maximum score were taken into the second Delphi round. In total, 24 items (range 1–6 items per topic) were presented to the expert panel for discussion.
Second Delphi round: face‐to‐face meeting
The second Delphi round resulted in consensus on the content of observation items per topic. Based on the results, a tracer list of 134 measurable elements (MEs), with a range of 1–16 MEs per topic, was designed (Appendix S1, supporting information). For practical reasons related to use of the SPOT inside or outside the physical OR complex, SPOT was divided into two parts: part A for tracing the preoperative and peroperative phase (including holding area or preanaesthesia care unit (22 MEs), OR (65 MEs) and recovery room (39 MEs)), and part B for tracing the postoperative discharge process on the surgical ward (8 MEs). Some MEs were measured twice in different locations of the perioperative patient path, and therefore given a unique number to be reviewed and scored. This was the case for the topics ‘patient transport’ (measured before and after the operation), ‘behaviour with respect to infection prevention’, ‘safety of medical equipment’ and ‘double‐check of medication’ (all measured in both the operating and recovery room). In addition, a dichotomous scoring scale, ‘yes’ or ‘no’ (respectively complied or not complied to the standard) per measurable item, was added to the list, facilitating an overall score calculation per topic. Finally, an instruction guide was developed and added to the SPOT.
Results of the pilot test
A total of eight hospitals participated in the SPOT pilot test, resulting in 2085 observations and scores of measurable elements related to 36 surgical procedures over ten surgical specialties: general surgery, cardiothoracic surgery, obstetrics and gynaecology, neurosurgery, ophthalmic surgery, plastic surgery, urology, oral and maxillofacial surgery, orthopaedic surgery, and ear, nose and throat surgery. All 19 patient safety topics were found to be measurable (adequate information available by observation), ranging from eight (‘discharge from the surgical ward’) to 291 (‘time‐out before the operation’) MEs among topics (Table 4).
Table 4.
Overall results of the pilot Surgical Patient safety Observation Tool scores per topic
| Compliant† | ||||
|---|---|---|---|---|
| Perioperative patient safety topic | Observations* | Applicable | Yes | No |
| Communication and handover | ||||
| 1 Patient identification | 127 | 127 (100) | 118 (91·5) | 11 (8·5) |
| 2 Handover from wards | 95 | 92 (97) | 73 (74) | 25 (26) |
| 3 Pre‐time‐out before the operation | 204 | 24 (11·8) | 23 (96) | 1 (4) |
| 4 Time‐out before the operation | 291 | 288 (99·0) | 275 (92·9) | 21 (7·1) |
| 5 Intraoperative team communication | 49 | 41 (84) | 41 (100) | 0 (0) |
| 6 Handover at shift change during the operation | 48 | 18 (38) | 18 (100) | 0 (0) |
| 7 Sign‐out after the operation | 142 | 127 (89·4) | 120 (93·8) | 8 (6·2) |
| 8 Patient transport | 33 | 33 (100) | 28 (85) | 5 (15) |
| 9 Handover to the recovery room | 190 | 180 (94·7) | 161 (88·5) | 21 (11·5) |
| 10 Discharge from recovery room | 179 | 172 (96·1) | 157 (91·3) | 15 (8·7) |
| 11 Discharge from the surgical ward | 8 | 8 (100) | 8 (89) | 1 (11) |
| Medical equipment | ||||
| 12 Safety of medical equipment | 33 | 33 (100) | 30 (86) | 5 (14) |
| 13 Counting used surgical equipment | 114 | 108 (94·7) | 95 (87·2) | 14 (12·8) |
| Infection prevention | ||||
| 14 Behaviour with respect to infection prevention | 262 | 258 (98·5) | 227 (81·4) | 52 (18·6) |
| 15 Air ventilation and conditioning characteristics of the OR | 18 | 18 (100) | 15 (83) | 3 (17) |
| 16 Prevention of transmission of hepatitis B and MRSA | 36 | 32 (89) | 26 (81) | 6 (19) |
| Medication | ||||
| 17 Double‐check of medication | 116 | 116 (100) | 100 (72·5) | 38 (27·5) |
| 18 Propofol handling | 49 | 29 (59) | 28 (97) | 1 (3) |
| 19 Drug storage | 91 | 88 (97) | 83 (94) | 5 (6) |
| Total (mean) | 2085 | 1792 (86·9) | 1626 (88·6) | 232 (11·4) |
| Median (range) | 95 (8–291) | 97 (11·8–100) | 89 (72·5–100) | 11 (0–27·5) |
Values in parentheses are percentages unless indicated otherwise.
Observed measurable elements.
Yes and No numbers combined are sometimes higher than the number of observations per topic, because both answer options were sometimes scored by the auditors. OR, operating room; MRSA, methicillin‐resistant Staphylococcus aureus.
The topics showed good applicability (median 97 (range 11·8–100) per cent). Poorly applicable topics were ‘pre‐time‐out before the operation’ (applicable in only 12 per cent of procedures), ‘handover at shift change during the operation’ (applicable in 38 per cent of procedures) and ‘propofol handling’ (a strict prescription of how to handle the intravenous administration of propofol to prevent bacterial contamination30, used in only 59 per cent of procedures) (Table 4).
The overall improvement potential of the topics was good: median 89 (range 72·5–100) per cent compliance per topic. Ten patient safety topics scored less than 90 per cent compliance, including ‘handover from wards’ (compliance 74 per cent) and ‘double‐check of medication’ (compliance 72·5 per cent) (Table 4).
The perioperative patient safety compliance score per topic (range 72·5–100 per cent) also indicated good discriminatory capacity of the tool (variation 27·5 per cent), with greatest variation for topics 2 and 17 (‘handover from wards’ and ‘double‐check of medication’) (Table 4).
The overall patient safety performance with respect to the participating hospitals showed lower variation (range 13 per cent), with greatest variation between hospitals 3 and 5, which scored 83 and 96 per cent respectively (Table 5).
Table 5.
Total ‘yes’ scores for compliance per hospital per topic
| Participating hospitals | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Surgical patient safety topic | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | Range (%) | Total ‘yes’ |
| Communication and handover | ||||||||||
| 1 Patient identification | 70 (92) | 12 (93) | 8 (86) | 7 (100) | 3 (100) | 6 (92) | 5 (100) | 7 (93) | 86–100 | 118 (91·5) |
| 2 Handover from wards | 39 (77) | 9 (90) | 2 (20) | 5 (100) | 4 (100) | 5 (60) | 5 (100) | 4 (80) | 20–100 | 73 (74) |
| 3 Pre‐time‐out before the operation | – | – | – | 9 (90) | 5 (100) | – | 9 (100) | – | 90–100 | 23 (96) |
| 4 Time‐out before the operation | 181 (96·3) | 29 (94) | 26 (82) | 16 (81) | 12 (100) | 11 (100) | – | – | 81–100 | 275 (92·9) |
| 5 Intraoperative team communication | 28 (100) | – | 5 (100) | 3 (100) | 2 (100) | 3 (100) | – | – | 100 | 41 (100) |
| 6 Handover at shift change during the operation | 9 (100) | – | 3 (100) | 3 (100) | 3 (100) | – | – | – | 100 | 18 (100) |
| 7 Sign‐out after the operation | 90 (93) | 7 (100) | 7 (100) | 7 (100) | 5 (100) | 4 (88) | – | – | 88–100 | 120 (93·8) |
| 8 Patient transport | 16 (80) | 2 (100) | 3 (100) | 2 (100) | 2 (100) | 2 (100) | – | 1 (100) | 80–100 | 28 (85) |
| 9 Handover to the recovery room | 99 (83) | 12 (100) | 10 (100) | 12 (100) | 9 (100) | 11 (100) | – | 8 (88) | 83–100 | 161 (88·5) |
| 10 Discharge from recovery room | 114 (89·3) | 15 (100) | – | 14 (93) | – | – | – | 14 (100) | 89·3–100 | 157 (91·3) |
| 11 Discharge from the surgical ward | – | – | – | – | – | – | – | 8 (89) | 89 | 8 (89) |
| Medical equipment | ||||||||||
| 12 Safety of medical equipment | 17 (81) | 3 (100) | 2 (67) | 2 (100) | 2 (100) | 2 (100) | 1 (100) | 1 (100) | 67–100 | 30 (86) |
| 13 Counting used surgical equipment | 62 (86) | – | 13 (93) | 7 (100) | 6 (86) | 7 (93) | – | – | 86–100 | 95 (87·2) |
| Infection prevention | ||||||||||
| 14 Behaviour with respect to infection prevention | 145 (84·1) | 15 (68) | 21 (81) | 14 (93) | 13 (85) | 14 (86) | 3 (83) | 2 (67) | 67–93 | 227 (81·4) |
| 15 Air ventilation and conditioning characteristics of the OR | 9 (75) | 1 (100) | 2 (100) | 1 (100) | 1 (100) | 1 (100) | – | – | 75–100 | 15 (83) |
| 16 Prevention of transmission of hepatitis B and MRSA | 17 (92) | 2 (50) | 2 (50) | 2 (100) | 2 (100) | 1 (100) | – | – | 50–100 | 26 (81) |
| Medication | ||||||||||
| 17 Double‐check of medication | 68 (78) | 8 (100) | 3 (29) | 8 (88) | 4 (100) | 6 (83) | 2 (75) | 1 (25) | 25–100 | 100 (72·5) |
| 18 Propofol handling | 18 (96) | 3 (100) | 2 (100) | 3 (100) | – | 2 (100) | – | – | 96–100 | 28 (97) |
| 19 Drug storage | 57 (97) | 7 (83) | 5 (100) | 6 (100) | 1 (100) | 2 (100) | 2 (67) | 3 (100) | 67–100 | 83 (94) |
| Total (mean) | 1039 (88·6) | 125 (91·4) | 114 (82·5) | 121 (94·3) | 74 (96) | 77 (92) | 27 (90) | 49 (87) | 1626 | |
Values in parentheses are percentages unless indicated otherwise. OR, operating room; MRSA, methicillin‐resistant Staphylococcus aureus.
All eight auditors considered SPOT easy to use, with scores of 5 or 6 points (maximum 6). Seven of the eight auditors stated that they were able to use SPOT independently. One auditor suggested the addition of a ‘partly complied’ option to the dichotomous scoring system.
Discussion
A comprehensive easy to use surgical patient safety observation tool, SPOT, was developed and pilot‐tested for prospective risk analysis, monitoring, benchmarking and improving perioperative safety. SPOT is currently used in all VVCZ member hospitals as part of their intrahospital and yearly interhospital auditing. The audit team consists of an audit leader together with two expert professionals in perioperative care (one physician and one non‐physician). The hospitals use SPOT results to support internal perioperative patient safety improvement initiatives. Results are anonymized and then used for benchmarking between VVCZ member hospitals.
A variety of local in‐hospital observation tools were found, and so a RAND‐modified Delphi technique was used to reach consensus on the content of a comprehensive and easy‐to‐use tracer and observation tool28, 29. The present study resulted in a measurement instrument for perioperative patient safety performance in daily clinical practice. All 19 national perioperative patient safety topics appeared to be easily measurable with SPOT, and seem generally applicable to various surgical procedures.
The overall improvement potential appeared to be good; in particular, topics such as ‘handover from wards’ and ‘double‐check of medication’ showed much room for improvement. At topic level, good discriminatory capacity of the tool was shown. Overall scores showed less variation between the hospitals that participated in the pilot.
The chosen cut‐off point of 90 per cent indicating good or poor compliance, as used in previous studies24, 25, 26, 27, may need revision in the future. It could be argued that 100 per cent compliance should be the goal for all patient safety topics.
In the absence of direct observation of daily practice, true compliance of standard perioperative procedures is unknown31. Auditing is thus an important activity of quality management, used to explore whether daily care is consistent with evidence‐based guidelines. Observation in daily practice by healthcare professionals themselves with the use of SPOT is considered a simple and therefore attractive method to stimulate patient safety guideline compliance. Insight in actual provided patient safety is a crucial step in the plan–do–check–act cycle before selecting and realizing patient safety improvement activities32. Furthermore, detailed monitoring and feedback about current non‐optimized patient safety performance contribute to employee awareness of the risky nature of their actions and motivate them in changing behaviour33.
Worldwide, patient safety programmes are designed to measure and improve safety in order to control risks and minimize potentially avoidable patient harm, by systems and teamwork approaches34. Supplementary to the legal frameworks and professional guidelines, the Dutch basic requirements for a hospital Safety Management System (SMS) are nationally established and described in the Dutch Technical Agreement (DTA)35. The DTA contributes to national uniformity of the SMS and aims to create transparency in patient safety in hospitals. Dutch hospitals have to meet the need for transparency by having their SMS evaluated and audited by internal and external parties35. In that way, the public and policy‐makers emphasize taking action to ensure compliance with safety guidelines. However, external pressure may also be associated with negative consequences, and may be more focused towards better administration than actually improving patient safety. It is therefore important to assess what really matters and focus on those patient safety issues that actually improve patient safety and motivate frontline caregivers36.
No perioperative patient safety observation tool similar to SPOT is currently available to characterize the safety of daily clinical practice. Perioperative experts from various hospitals participated in the present study, which resulted in broad support for one comprehensive and condensed observation tool. It offers a structured method to measure and monitor perioperative risks to improve patient safety. From the pilot test it was shown that SPOT was a suitable easy‐to‐use tool to identify patient safety risks in the perioperative process. The SPOT concept can be transferred easily to a broad range of medical disciplines and activities with an interventional character.
The instrument is currently set up with ‘yes’ and ‘no’ as possible answers for included items. This turned out to be insufficient for some observations in clinical practice. One auditor suggested that ‘partly complied’ be added as a third answer category; this should be taken into account in further development. A limitation of SPOT might be the subjective judgement of a single observation if SPOT is used by only one auditor. Preferably, the observations are done by at least two auditors, although this might interfere with a maximum allowable number of professionals being in the OR. For this reason, during an audit, the results of an individual auditor should be discussed with the audit team. In addition, the presence of an observer may influence the normal daily practice behaviour, which may result in an overestimation of the actual performance (the Hawthorn effect), although the size of this effect is not known37. Another concern, despite the simplicity of SPOT, is the potential time‐consuming nature of processing and analysing the data, which may be a barrier to use. To facilitate this, SPOT should be transformed from a paper version into a digital ‘SPOT tracer and monitor application’, as has already been realized in Radboud University Medical Centre. Although SPOT is based on international goals, national guidelines and the perioperative patient safety framework of the Dutch Health Care Inspectorate15, the IGZ framework did not cover the latest developments, on such topics as ‘hand hygiene’ and ‘air ventilation and conditioning characters of the operating room’, so new items should be addressed, while other items will need reframing.
Best practice performances of departments and hospitals should enable them to share experiences and learn from one another. The anticipated users of SPOT are internal auditors, such as front‐line professionals, and external auditors, such as peer colleagues from other hospitals. Based on these initial experiences with SPOT in the Netherlands, a regular update of the content of the tool is recommended. This may be done in cooperation with national guideline developers and with use of international standards, such as the JCI accreditation standards. Because SPOT is based on international patient safety guidelines, the tool could be widely applicable. Evaluation of SPOT outside the Netherlands should therefore be undertaken to assess whether this is the case. Further work should include evaluation of the effect of using this condensed and standardized tool on patient safety outcomes.
Supporting information
Table S1 Number of perioperative experts who judged perioperative patient safety topics
Table S2 Patient safety observation topics covered in local observation tools of the participating hospitals
Appendix S1 Surgical Patient safety Observation Tool (SPOT)
Acknowledgements
The authors thank B. van Acker, Process Improvement and Innovation Consultant, Radboud University Medical Centre, for valuable support in managing the data.
Disclosure: The authors declare no conflict of interest.
Funding information:
No funding
References
- 1. James JT. A new, evidence‐based estimate of patient harms associated with hospital care. J Patient Saf 2013; 9: 122−128. [DOI] [PubMed] [Google Scholar]
- 2. de Vries EN, Ramrattan MA, Smorenburg SM, Gouma DJ, Boermeester MA. The incidence and nature of in‐hospital adverse events: a systematic review. Qual Saf Health Care 2008; 17: 216−223. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3. de Vries EN, Hollmann MW, Smorenburg SM, Gouma DJ, Boermeester MA. Development and validation of the SURgical PAtient Safety System (SURPASS) checklist. Qual Saf Health Care 2009; 18: 121−126. [DOI] [PubMed] [Google Scholar]
- 4. Greenberg CC, Regenbogen SE, Studdert DM, Lipsitz SR, Rogers SO, Zinner MJ et al Patterns of communication breakdowns resulting in injury to surgical patients. J Am Coll Surg 2007; 204: 533−540. [DOI] [PubMed] [Google Scholar]
- 5. Griffen FD, Stephens LS, Alexander JB, Bailey HR, Maizel SE, Sutton BH et al The American College of Surgeons' closed claims study: new insights for improving care. J Am Coll Surg 2007; 204: 561−569. [DOI] [PubMed] [Google Scholar]
- 6. Dutch Health Care Inspectorate (IGZ) . [Preoperative Process Lacks a Multidisciplinary and Standardized Approach and Teambuilding.] IGZ: The Hague, 2007. [Google Scholar]
- 7. Dutch Health Care Inspectorate (IGZ) . [Standardization is Essential for Risk Reduction in the Operative Process.] IGZ: The Hague, 2008. [Google Scholar]
- 8. Dutch Health Care Inspectorate (IGZ) . [Postoperative Care in Hospitals is Partly Sufficient, But Still Involves Too Many Risks.] IGZ: The Hague, 2009. [Google Scholar]
- 9. Nederlandse Vereniging voor Anesthesiologie (NVA), Nederlandse Vereniging voor Heelkunde (NVvH) . [Guideline: The Preoperative Care Process.] NVA, NVvH: Utrecht, 2010. [Google Scholar]
- 10. Nederlandse Vereniging voor Anesthesiologie (NVA), Nederlandse Vereniging voor Heelkunde (NVvH) . [Guideline: The Peroperative Care Process.] NVA, NVvH: Utrecht, 2011. [Google Scholar]
- 11. Nederlandse Vereniging voor Anesthesiologie (NVA), Nederlandse Vereniging voor Heelkunde (NVvH) . [Guideline: The Postoperative Care Process.] NVA, NVvH: Utrecht, 2012. [Google Scholar]
- 12. Baines R, Langelaan M, de Bruijne M, Spreeuwenberg P, Wagner C. How effective are patient safety initiatives? A retrospective patient record review study of changes to patient safety over time. BMJ 2015; 24: 561−571. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13. de Vries EN, Prins HA, Crolla RM, den Outer AJ, van Andel G, van Helden SH et al Effect of a comprehensive surgical safety system on patient outcomes. N Engl J Med 2010; 363: 1928−1937. [DOI] [PubMed] [Google Scholar]
- 14. van Klei WA, Hoff RG, van Aarnhem EE, Simmermacher RK, Regli LP, Kappen TH et al Effects of the introduction of the WHO ‘Surgical Safety Checklist’ on in‐hospital mortality: a cohort study. Ann Surg 2012; 255: 44−49. [DOI] [PubMed] [Google Scholar]
- 15. Dutch Health Care Inspectorte (IGZ) . Operatief Proces Beter Gestructureerd, Scherpere Handhaving op Achterblijvende Uitvoering Blijft Noodzakelijk. IGZ: Utrecht, 2013. [Google Scholar]
- 16. Joint Commission International (JCI) . Accreditation Standards for Hospitals (5th edn). JCI: Oakbrook Terrace, 2014. [Google Scholar]
- 17. Murff HJ, Patel VL, Hripcsak G, Bates DW. Detecting adverse events for patient safety research: a review of current methodologies. J Biomed Inform 2003; 36: 131−143. [DOI] [PubMed] [Google Scholar]
- 18. Zegers M, de Bruijne MC, Spreeuwenberg P, Wagner C, Groenewegen PP, van der Wal G. Quality of patient record keeping: an indicator of the quality of care? BMJ 2011; 20: 314−318. [DOI] [PubMed] [Google Scholar]
- 19. Hanskamp‐Sebregts M, Zegers M, Vincent C, van Gurp PJ, de Vet HC, Wollersheim H. Measurement of patient safety: a systematic review of the reliability and validity of adverse event detection with record review. BMJ 2016; 6: e011078. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20. WHO . Patient Safety: Rapid Assessment Methods for Estimating Hazards. WHO: Geneva, 2002. [Google Scholar]
- 21. Michel P. Strengths and Weaknesses of Available Methods for Assessing the Nature and Scale of Harm Caused by the Health System: Literature Review. WHO: Geneva, 2003. [Google Scholar]
- 22. Barnett ML, Olenski AR, Jena AB. Patient mortality during unannounced accreditation surveys at US hospitals. JAMA 2017; 177: 693−700. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23. Joint Commission International (JCI) . Survey Process Guide for Hospitals (5th edn). JCI: Oakbrook Terrace, 2014. [Google Scholar]
- 24. McGlynn EA. Choosing and evaluating clinical performance measures. Jt Comm J Qual Improv 1998; 24: 470−479. [DOI] [PubMed] [Google Scholar]
- 25. Rubin HR, Pronovost P, Diette GB. From a process of care to a measure: the development and testing of a quality indicator. Int J Qual Health Care 2001; 13: 489−496. [DOI] [PubMed] [Google Scholar]
- 26. Hermens RP, Ouwens MM, Vonk‐Okhuijsen SY, van der Wel Y, Tjan‐Heijnen VC, van den Broek LD et al Development of quality indicators for diagnosis and treatment of patients with non‐small cell lung cancer: a first step toward implementing a multidisciplinary, evidence‐based guideline. Lung Cancer 2006; 54: 117−124. [DOI] [PubMed] [Google Scholar]
- 27. Emond YE, Stienen JJ, Wollersheim HC, Bloo GJ, Damen J, Westert GP et al Development and measurement of perioperative patient safety indicators. Br J Anaesth 2015; 114: 963−972. [DOI] [PubMed] [Google Scholar]
- 28. Boulkedid R, Abdoul H, Loustau M, Sibony O, Alberti C. Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review. PLoS One 2011; 6: e20476. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29. Ogden SR, Culp WC Jr, Villamaria FJ, Ball TR. Developing a checklist: consensus via a modified Delphi technique. J Cardiothor Vasc Anesth 2016; 30: 855−858. [DOI] [PubMed] [Google Scholar]
- 30. Bennett SN, McNeil MM, Bland LA, Arduino MJ, Villarino ME, Perrotta DM et al Postoperative infections traced to contamination of an intravenous anesthetic, propofol. N Engl J Med 1995; 333: 147−154. [DOI] [PubMed] [Google Scholar]
- 31. Leape LL. The checklist conundrum. N Engl J Med 2014; 370: 1063−1064. [DOI] [PubMed] [Google Scholar]
- 32. Grol R, Wensing M. Effective implementation of change in healthcare: a systematic approach In Improving Patient Care: The Implementation of Change in Health Care, Grol R, Wensing M, Eccles M, Davis D. (eds). John Wiley: Chichester, 2013; 40−63. [Google Scholar]
- 33. Grol R, Wensing M. What drives change? Barriers to and incentives for achieving evidence‐based practice. Med J Aust 2004; 180(Suppl): S57−S60. [DOI] [PubMed] [Google Scholar]
- 34. McCulloch P, Morgan L, New S, Catchpole K, Robertson E, Hadi M et al Combining systems and teamwork approaches to enhance the effectiveness of safety improvement interventions in surgery: the Safer Delivery of Surgical Services (S3) program. Ann Surg 2017; 265: 90–96. [DOI] [PubMed] [Google Scholar]
- 35. Netherlands Standardization Institute (NSI) . NTA 8009 Safety Management System for Hospitals and Organizations which Administer Hospital Care. NSI: Delft, 2014. [Google Scholar]
- 36. Thomas EJ, Classen DC. Patient safety: let's measure what matters. Ann Intern Med 2014; 160: 642−643. [DOI] [PubMed] [Google Scholar]
- 37. McCambridge J, Witton J, Elbourne DR. Systematic review of the Hawthorne effect: new concepts are needed to study research participation effects. J Clin Epidemiol 2014; 67: 267−277. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Table S1 Number of perioperative experts who judged perioperative patient safety topics
Table S2 Patient safety observation topics covered in local observation tools of the participating hospitals
Appendix S1 Surgical Patient safety Observation Tool (SPOT)