UK national audit of safety checks for radiology interventions

Abstract

Objective:

To reaudit the use of safety checklists in radiology departments in NHS departments throughout the UK.

Methods:

This audit was performed on behalf of The Royal College of Radiologists Audit Committee in 2016 and was sent to radiology audit leads at every NHS department in the UK to determine the use of safety checks in various modalities and subspecialties. Free-form text boxes gathered data on problems with checklist implementation.

Results:

109/177 (62%) trusts responded. 48% of respondents used safety checklists for all radiological procedures in all modalities. 50% used checklists for some procedures. 2% did not use a checklist. Checklist use had increased since the previous audit (98% 2016, compared to 94% in 2012) but implementation for different procedures remains variable. For example, in ultrasound-guided fine needle and breast stereotactic procedures (49%), use has not increased since 2012.

Conclusion:

Reasons for not using checklists include a perception that intervention suite checklists were not appropriate for minor procedures and the limited flexibility of radiology information systems. The limitations of checklists are discussed.

Advances in knowledge:

Our reaudit shows that in spite of increased implementation, use of safety checks is variable. Local ownership and radiology information system flexibility are needed to support the culture of safety processes in radiology departments.

Introduction

Human error is an inevitable cause of patient harm. Safety checks are a simple method of reducing the risk of harm using a periprocedural routine to reduce the likelihood of error.

Safety checklists have been in use in the aviation industry since the late 1930s. In 1935, a prototype Boeing B-17 bomber crashed during a test flight, killing two experienced test pilots, leading to the development of more formalised cockpit safety checklist.¹

‘At the controls of the Model 299 this day were two experienced Army pilots….

The aircraft made a normal taxi and takeoff. It began a smooth climb, but then suddenly stalled. The aircraft turned on one wing and fell, bursting into flames upon impact…..

The investigation found “Pilot Error” as the main cause of the accident. Hill, unfamiliar with the aircraft, had neglected to release the elevator lock prior to take off. Once airborne, Tower evidently realized what was happening and tried to reach the lock handle but by that time it was too late.

The pilots sat down and put their heads together. What was needed was some way of making sure that everything was done properly; that nothing was overlooked. What resulted was a pilot’s checklist. In the end, four checklists were developed – take off, flight, before landing, and after landing. As it turned out, the Model 299 was not “too much airplane for one man to fly,”––it was simply too complex for any one male’s memory. These checklists for both the pilot and the co-pilot made sure that nothing was forgotten.

With these new checklists, careful planning and rigorous training, the twelve aircraft managed to fly 1.8 million miles without a serious accident….The B-17 went on to become the most widely used aircraft in WWII.’

Surgical checklists in the operating room have been established practice in the UK for several years. A World Health Organisation (WHO) study published in 2009 showed that the use of a surgical safety checklist resulted in statistically significant improvements in patient morbidity and mortality.^2,3 In-patient complication rates fell from 11 to 7% (p < 0.001), and mortality fell from 1.5 to 0.8% (p = 0.003) after the introduction of the safety checklist.²

In 2009, the National Patient Safety Agency (NPSA) adapted the WHO checklist for use in the UK, and since then it has been a requirement for every patient undergoing a surgical procedure.⁴

Following this landmark change in practice in the operating room, there has been evolving advice for the application of safety checks in the radiology department.

In 2009 and then again in 2010, the NPSA and Royal College of Radiologists (RCR) gave advise on the use of safety checks including a modification of the surgical checklist to be used for radiological interventions (Figure 1).^5,6 The RCR advised that the implementation of the safe surgical checklist should be applied “to all diagnostic and interventional procedues”.⁵ An “intervention” was defined by the NPSA & RCR as any procedure which is “dependent on the penetration of skin, other than the placement of intravenous access under local or general anaesthesia”.

Figure 1.

Modified WHO safety checklist.

The document mentions the possibility of using local clinical governance processes for local adaptation or to decide to omit some radiological procedures from the process.

A further document was published by the RCR in 2013 offering guidance on the implementation of surgical safety checklists for radiological procedures.⁷ This was in response to concerns expressed by Fellows about the use of a “version of the checklist for use in operating theatre environments”. This document stated that “adaptation of the checklist to suit local needs is to be encouraged” and gave an example of an adapted checklist.

As in previous documents, this advice focussed on interventions more likely to be performed in the intervention suite but does include a brief comment on “minor” procedures undertaken by “Sole Practitioners”.

“Where a Fellow is undertaking procedures alone [usually relatively minor image-guided interventions such as fine needle aspiration (FNA)], the formal completion and scanning of a checklist for each patient might appear overburdensome and perverse. However, the RCR recommends that sole practitioners run through the checklist (ideally locally adapted) before each intervention and record that they have done so in the procedure report”.

The frequency of never events did not decline as hoped following the introduction of the surgical safety checklist. The surgical services patient safety expert group commissioned a Surgical Never Events Taskforce report into never events in the operating room.⁸ This report published in 2014 documented that 32⁹ never-events had taken place in England in 2012/13. Possible causes for this, such as variation in implementation of different facets of the checklist, were discussed. The report comments that “there are formidable problems evidencing that the checklist is of value specifically in preventing never events and other serious incidents in the NHS”.

This report recommended that the NHS in England develop national standards for surgical procedures and that “National standards require all providers of NHS funded care to develop and maintain local standards consistent with the national standards”.⁸

The resulting document, “National Safety Standards for Invasive Procedures” (NatSSIPs)⁹ was published in 2015. The purpose was to provide guidance for the implementation of Local Safety Standards for Invasive Procedures (LocSSIPs).

Safety checks performed at the time of a procedure are considered only a part of the LocSSIP. Documentation of a process within a LocSSIP document should “start at the point at which the patient is admitted to the procedure area and end at the point at which the patient is discharged form the procedure area”.

All NHS providers are required to follow this guidance and provide documented LocSSIPs for all invasive procedures.

This document defines an invasive procedure as surgical, endoscopic and thoracic interventions and specifically mentions “radiological intervention procedures” and “biopsies and other invasive tissue sampling”. The document excludes from NatSSIPs and LocSSIPs “procedures that involve the simple penetration of the skin or entry in to a body cavity such as the insertion of an intravenous line or a urinary catheter”. The foreword states that local standards for major surgery will not be the same as for chest drain insertion, though the document focuses on prevention of never events and processes around surgery.

Standards for documentation are discussed and regular audit is required to assess compliance with LocSSIPs.

In 2012, the RCR undertook a UK audit of safety checklist compliance¹⁰ with a response rate of 34%. At that time 87% of units had implemented a safety checklist. Many comments on the process were positive, some noting that adverse events had been averted by the use of a checklist. Negative comments were generally related to the length of the checklist process and resulting increased time for procedures.

This study reports on a reaudit undertaken by the RCR in 2016. The purpose was to determine whether radiology departments have improved implementation and met national standards for safety checklists across the UK. A second objective was to examine difficulties in implementation.

Methods and materials

Survey design

The RCR clinical audit committee designed and co-ordinated this national audit. Questions were based upon a previous audit in 2012, presented at the RCR Audit Forum in 2013¹⁰ but several new areas of questioning were included to investigate difficulties with the implementation of safety checklists across the UK. As a consequence of this wider objective, the data collected in this audit are not identical to those in 2012 but provide some comparisons.

The survey was then sent to all radiology audit leads at hospital Trust and Health Boards in the UK. The survey was in Survey Monkey format. Data about the following were requested:

• Extent of checklist use for a variety of interventions in the intervention suite and general department.

• Methods of consent.

• Patient information provided prior to a procedure.

• Whether there is a local policy for use of checklists.

• Which procedures are included.

• Whether a national form or local modification of checklists is used.

• Use of a checklist with different forms of anaesthesia.

• Whether checklists are completed by hand or are electronic.

• Whether a permanent record is kept.

• Recording of consent and complications.

• Perceived obstacles to implementation

• Opinions about important components for a checklist for procedures in the general radiology department.

Audit standards

The standards set for this audit are based on the original NPSA requirements regarding the WHO intervention safety checklist (Table 1). A 2009 Patient Safety Alert mandated that all organisations within the NHS use safety checklists and required a safety checklist for every patient undergoing invasive procedures.

Table 1.

Questions included in the audit showing 100% audit standards

Question	Audit standard
Home nation?	N/A
Number of beds at your trust?	N/A
Does your unit provide the following image guided procedures? (CT/USS/Fluoro/Breast/IR/Other)	N/A
Does your unit provide information sheets prior to interventions in CT/USS/Fluoro/Breast/IR/Other?	N/A
Does your department use procedural checklists for invasive procedures?	100%
Is the use of a procedural checklist supported by a written departmental protocol?	100%
Is there a local written policy stating that checklists need not be used for some procedures?	N/A
For which procedures is a checklist NOT required by local policy	N/A
Does your department routinely perform a procedural checklist for invasive procedures in the following areas	100%
Does the routine performance of a procedural checklist for invasive procedures apply to all procedures performed with: no anaesthetic/local anaesthetic/general anaesthetic	N/A
What form does your department’s checklist take in the intervention suite?	N/A
Does your department have another local modified checklist for minor interventions?	N/A
Is written or verbal consent or neither used for the following interventions?	N/A
Is the checklist performed by all radiologists or radiographers undertaking a procedure in the department?	100%
Does your department use a checklist that is: handwritten/electronic/both handwritten and electronic forms?	N/A
Are paper records scanned onto an information system or stored in paper format?	N/A
Does your RIS allow you to add an electronic checklist?	N/A
Does the RIS require the checklist to be completed before the examination can be completed?	N/A
Is it departmental policy to document consent in the examination report?	N/A
Is it departmental policy to document complications discussed in the examination report?	N/A
Does your department undertake audit of the checklist process and its recording? How often?	N/A
To what extent are the following (inadequate time/resources/feeling it is superfluous/different levels of commitment by individuals or staff groups/feeling the checklist is too long and not all data are necessary/other) a problem to implementing checklists where you work?	N/A

RIS, radiology information systems.

Data collection

Survey responses were collated on the Survey Monkey system and data transferred to Microsoft Office Excel.

Statistical analysis

No statistical analysis was performed. Where possible, direct comparison is made with data from the previous audit.

Results

109 out of 177 (62%) units responded to the 2016 survey (34% in 2012).

The vast majority of these units provide interventions in CT, ultrasound, fluoroscopic, breast and radiology intervention suites.

The results for the standards comparable between 2012 and 2016 are shown in Table 2.

Table 2.

Comparison of audit results from 2012 and 2016

	2012		2016
Standard 1	Have you already implemented or PARTLY implemented the checklist anywhere in the Department?		Does your department use procedural checklists for invasive procedures?
	Yes	93	All	48
	No	7	Some	50
			None	2
Standard 2	Please indicate which interventional procedures (do not include siting venflons) are utilising the checklist in your department:		Does your department routinely perform a procedural checklist for invasive procedures in the following areas:
	CT yes	78	CT FNA yes	66
	CT no	22	CT FNA no	34
			CT core yes	79
			CT core no	21
			CT drainage yes	79
			CT drainage no	21
	Ultrasound yes	85	Ultrasound FNA yes	49
	Ultrasound no	15	Ultrasound FNA no	51
			Ultrasound core yes	66
			Ultrasound core no	34
			Ultrasound drainage yes	79
			Ultrasound drainage no	21
	Fluoro yes	96	Fluoro yes	58
	Fluoro no	4	Fluoro no	42
	Stereotactic yes	53	Stereotactic yes	49
	Sterotactic no	47	Sterotactic no	51
	IR suite yes	98	IR suite yes	95
	IR suite no	2	IR suite no	5
Standard 3	Are there any areas, individuals or procedures excluded from using the checklist?		Is there a local written policy stating that checklists need not be used for some procedures?
	Yes	69	Yes	6
	No	31	No	87
			Don't know	6
Standard 4	Is the checklist used in this audit a local adaptation of the NPSA & RCR checklist?		Does your department have another local modified checklist for minor interventions?
	Yes	53	Yes—modified WHO	25
	No	47	Yes—CIRSE	1
			Yes—Local modification used in IR	5
			Yes—Simplified local modification	23
			No	46

FNA, fine needle aspiration; RIS, radiology information systems; WHO, World Health Organisation.

Patient information

The proportion of units providing patient information sheets varies from 73% in intervention suite procedures to 65% in fluoroscopic interventions (Figure 2).

Figure 2.

Provision of intervention and information leaflets by modality.

Uniformity of checklist use

• 48% provide procedural checklists for all invasive procedures in all of the modality areas: CT, Ultrasound, fluoroscopic, breast and radiology intervention suites.

• 50% use checklists for some procedures.

• 2% do not use checklists.

This shows an increase in the frequency of checklist used from 93 to 98%.

Departmental policy

• 58% of units which use a checklist have a supporting written departmental protocol.

• 29% do not.

• 13% of responders were not aware of whether a supporting written protocol existed.

• 2% of units have a policy excluding particular procedures from use of checklists. These included Ultrasound FNA, Ultrasound musculoskeletal (MSK) injections, breast FNA/core biopsy/localisation/stereotactic procedures.

Form of checklist for different procedures

The use of a checklist varied with the procedure undertaken (Figure 3).

Figure 3.

Checklist use for invasive procedures by department.

In the Intervention suite, checklists were routinely performed in 95% of units (2012, 98%),

CT biopsy and drainages 79%, CT-guided MSK 66% and CT-guided FNA 62% (mean 2013, 67%)

Ultrasound-guided FNA 49%, Ultrasound core biopsy (non-breast) 66% and Ultrasound-guided drainage (79%). (Mean 2012 for non-breast Ultrasoundguided procedures 58%).

Breast stereotactic procedure 49% (2012, 50%).

Form of the checklist in the intervention suite

74% use the modified WHO checklist, 22% a local modification and 1% the Cardiovascular and Interventional Radiological Society of Europe checklist.

Anaesthesia

60% routinely use checklists for local anaesthesia (LA). 84% use a checklist for procedures under general anaesthesia (GA).

This compares with routine use in 68% of LA procedures and 81% for GA procedures in 2012.

Form of the checklist for minor procedures

54% of units have a modified checklist for minor interventions compared with 53% in 2012.

This 54% comprises a modified WHO checklist in 25%, CIRSE in 1% and a simplified local modification in 23%.

Consent

This varied widely with procedure (Figure 4).

Figure 4.

Use of written or verbal consent.

92% of interventional units used written consent for all procedures (other units verbal or written depending on procedure).

Oral consent is used in the majority of minor interventions, e.g. FNA including thyroid (66%).

Documentation and preservation of the record

The vast majority of departments use a handwritten record (87%), some a combination of handwritten and electronic forms (12%) and only 1% electronic only.

Only 26% of departments have a RIS enabling adding an electronic checklist.

In 89%, the RIS allowed completion of the examination without completion of the checklist (i.e. the examination can be completed without a record of the completed checklist).

Handwritten records were scanned in 73% of cases and stored in paper form.

Audit

72% of departments undertook annual audit of the checklist process.

Difficulties in implementation

There was variable compliance with 33% of departments reporting lack of uniformity of checklist completion by different staff members.

Units were asked about a variety of obstacles to implementation.

The data suggest that the process is generally perceived as valuable but that the greatest problems are commitment of different individuals and the feeling the checklist is too lengthy and not all data are necessary. Time pressure and resources to scan paper checklists are also factors (Table 3).

Table 3.

Problems with checklist implementation

To what extent are the following a problem to implementing checklists where you work?
	No problem	Small problem	Moderate problem	Large problem	Very large problem	Total	Weighted average
Inadequate time to complete it within appointment lots	28 (31%)	32 (35%)	21 (23%)	9 (10%)	1 (1%)	91	2.15
Resources to scan paper checklists onto an information system	32 (35%)	25 (27%)	25 (27%)	8 (9%)	1 (1%)	91	2.13
A feeling it is superfluous to patient care	43 (47%)	31 (34%)	15 (16%)	2 (2%)	0 (0%)	91	1.74
Different levels of commitment by individuals or staff groups	20 (22%)	35 (38%)	27 (30%)	8 (9%)	1 (1%)	91	.29
A feeling the checklist used is too lengthy and not all data are necessary	31 (34%)	21 (23%)	27 (30%)	12 (13%)	0 (0%)	91	2.22
Other problems	69 (76%)	11 (12%)	7 (8%)	4 (4%)	0 (0%)	91	1.41

What are the essentials for a checklist in radiological procedures outside the intervention suite?

Respondents were asked for their views about the value of different components of a safety check. The responses reflect the view that important safety checks depend on the particular procedure (Figure 5). For example, most respondents felt that routine blood tests were appropriate prior to liver biopsy but not thyroid FNA or Ultrasound-guided MSK injection.

Figure 5.

User recommendations for inclusion in intervention records.

Discussion

The culture of safety checks is now almost universal in intervention suites, but there is variable use of safety checks in other radiology interventions.

Direct comparison between the current audit and the 2012 audit is limited because of changes in national standards and a wider remit of the current study. There are four comparable standards between the two audits (Table 2).

In 2012, 93% of responders had implemented or partly implemented a checklist process. This improved in the 2016 audit, 98% of trusts had implemented a checklist for all or some interventions. However, 95% use intervention suite checklists in 2016 compared with 98% in 2012.

The 2016 survey differentiated between core biopsy, drainage and FNA, whereas the 2012 survey did not. As FNAs are often excluded from checklist use, it may be more useful to compare the 2012 overall CT and ultrasound data with the 2016 data for core biopsies and drainages. This gives a similar checklist usage between the two audits in CT. However, even when excluding the Ultrasound FNA data, the usage rate for ultrasound checklists is lower in 2016.

There was lower checklist use in fluoroscopy and stereotactic biopsies compared to 2012.

The 2016 data suggest that although overall checklist use is lower than in 2012, there are fewer areas and procedures excluded from checklist use.

Checklist uptake is lower for MSK and breast procedures. Factors contributing to this may include perception of lower risk associated with these procedures and the potential to slow relatively rapid procedures and interrupt workflow. Another factor may be that, unlike the intervention suite, no standardised process exists.

Overall, the data show lower use of checklists for interventional procedures in several areas. This may reflect the greater response rate in the 2016 audit or perhaps decreasing impact following the original NPSA alert. Whatever the reason, use of checklists remains variable.

Would increasing checklist compliance improve patient safety?

Though there is no study of radiology specific never events or near misses prior to and after the intervention suite checklist, the stalling of implementation of checks is reminiscent of the difficulties in embedding the surgical checklist.

The recognition that surgical checklists have not prevented never events in the operating room was an important driver for the NatSSIPs document. “Experience with its (the surgical checklist) use has suggested that the benefits of a checklist approach can be extended beyond surgery. Experience with it has also made it evident that checklists in themselves cannot be fully effective in protecting patients from adverse incidents”.⁹

Although no national register is available specifically for radiological intervention adverse events, some could occur in the updated list of never events. The revised never event policy published in 2015 includes:

• ….interventions that are considered surgical but may be done outside of a surgical environment e.g. ….biopsy, interventional radiology procedures, … drain insertion and line insertion.

• Surgical placement of the wrong implant or prosthesis where the implant/prosthesis placed in the patient is other than that specified.¹¹

In the year April 2016 to March 2017, 424 never events were recorded and declared by NHS Improvement.¹² This compares with 32⁹ in 2012/13.⁸ Though this may reflect better reporting and changed definitions of never events, the evidence suggests that never events are stubbornly persistent despite the checklist process.

In the operating room, lack of conviction about the value of the process by operating team members, failure to read the checklist out loud or absence of key members at the time of checks or failure of sign out have been found to be frequent in the UK⁸ and in other countries.¹³

Limits in the effectiveness of checklists are recognised. A study of the attempt in the UK to emulate the Michigan Keystone project which reduced intensive care central line infections, “Matching Michigan”, found that the introduction of the checklists on which the project was based in another context was of limited success. The success or failure of any safety initiative is a complex process with a variety of factors beyond the initiative itself, such as perception of the initiative by staff, the context of its implementation, and the culture of various institutions.¹⁴

We await evidence as to whether NatSSIPs, with the emphasis on processes from admission till discharge from the procedure area, will succeed where the surgical checklist did not.

The limits of checklists have long been recognised in aviation.¹⁵ A report on checklist design the US Federal Aviation Administration in 1991 comments that:

As the overall accident rates have declined over the years and flight safety has shown significant improvement, the percentage of accidents that result from flight crew action or inaction has remained relatively constant in the majority of fatal air carrier accidents.¹⁵

The Federal Aviation Administration recognised then that “human factors” are a source of error, however, rigorous the checklist process:

While the NTSB and ASRS reports suggest that flight crews need to place more emphasis on checklist usage, they also suggest other areas that require attention with due consideration for human factors. Such factors as fatigue, crew reliance on working or short term memory, crew interruption or distraction, and complacency or failure to visually verify aircraft configuration, are factors that may affect crew performance and have the potential to cause checklist error.¹⁵

Should safety checks be uniform?

The comments in the current study indicate a perception that the checklist for intervention suite processes is not well-suited to minor invasive procedures. This is not specifically addressed in the NatSSIPS document though this document includes the statement:

Neither is it intended that every detail of the NatSSIPs be transposed into LocSSIPs for single-operator, ward-based procedures such as bone marrow aspiration, pleural biopsy and tapping of ascites’ and appropriate local adaptation of safety checks is recommended.

It will be necessary to address this to enable appropriate and practical safety processes to be widely adopted for minor procedures. An example of a locally modified checklist for ultrasound-guided procedures is shown (Figure 6). The respondents regarded various components of a safety checklist as having variable importance depending on the procedure. This suggests that uniformity of checklists is not a useful objective.

Figure 6.

Example electronic Radiology Information System modified WHO checklist.

This approach needs to be acknowledged explicitly if those developing checklists locally are to take ownership of patient safety. We need to accept, in the absence of evidence to the contrary, that simply extending the length of a checklist in the hope of covering every possible error that might occur is not evidence based, and is likely to be counterproductive.

Are information systems adequate for safety check processes?

An obstacle to safety processes in radiology is the lack of flexibility of IT systems. Without a flexible RIS, it is difficult to tailor checklists to particular procedures and recording and storing the data is slow and unreliable. Until this flexibility is more widely available, compliance will be difficult.

The ability to integrate a compulsory checklist before performing the procedure would be an example of the needed flexibility. Audit is unlikely to be reliable if paper is saved in filing cabinets. The practice of scanning safety documentation perpetuates the impression of a bureaucratic process that increases work. Current versions of RIS are not fit for purpose in this regard.

Checklists are only one aspect of intervention safety as proposed by the NatSSIP/LocSIPP process. Another important aspect of this in all interventions is the provision of information before a procedure, a component of informed consent. About two-thirds of units are now providing information to patients prior to procedures.

There are limitations to this audit. There are frequently variations in practice within an area in a particular department and it is difficult to reflect this within a survey. Due to the changing landscape in safety process, there are limited comparisons between the 2012 and 2016 audits.

In conclusion, this survey demonstrates widespread adoption of the safety check process in interventional radiology units but variable use in other areas of the radiology department. There is also variable use of patient information sheets, departmental protocols and audit.

The proportion using checklists in the intervention suite is very high but slightly lower than in the audit of 2012. This could reflect a larger sample. There is now a wider range of interventions for which checklists are used but not improved uptake in minor interventions. The data suggest this is due to a perception that checklists based on the intervention suite checklist are inappropriate.

There is a requirement under NatSSIPs guidance for uniformity in safety checks though the details of processes for procedures outside the intervention suite are not prescribed. It is likely that locally devised safety processes that are appropriate and proportionate to the procedure will result in local ownership of and greater improvements in patient safety.

Experience from other contexts suggests that safety gains from increasing checklist use will be limited and that cultural, contextual and human factors around these processes need to be considered if safety is to improve.

It is difficult to foresee effective implementation of these processes including permanent record and audit without flexible RIS systems in which these processes can be embedded.