Aviation’s Normal Operations Safety Audit: a safety management and educational tool for health care? Results of a small-scale trial

Abstract

Background

A National Health Service (NHS) contingent liability for medical error claims of over £26 billion.

Objectives

To evaluate the safety management and educational benefits of adapting aviation’s Normal Operations Safety Audit (NOSA) to health care.

Methods

In vivo research, a NOSA was performed by medical students at an English NHS Trust. After receiving training from the author, the students spent 6 days gathering data under his supervision.

Results

The data revealed a threat-rich environment, where errors – some consequential – were made (359 threats and 86 errors were recorded over 2 weeks). The students claimed that the exercise improved their observational, investigative, communication, teamworking and other nontechnical skills.

Conclusion

NOSA is potentially an effective safety management and educational tool for health care. It is suggested that 1) the UK General Medical Council mandates that all medical students perform a NOSA in fulfillment of their degree; 2) the participating NHS Trusts be encouraged to act on students’ findings; and 3) the UK Department of Health adopts NOSA as a cornerstone risk assessment and management tool.

Introduction

The Institute of Medicine¹ has argued for a better understanding of the systemic causes of medical error. The Department of Health² has argued for a wider appreciation of the value of the systems approach in preventing medical error. The National Patient Safety Agency promotes systems-thinking:

The best way of […] reducing error rates is to target the underlying systems failures rather than take action against individual members of staff […]. A much wider appreciation of the value of the systems approach in preventing, analyzing and learning from patient safety incidents [is required].³

Despite these exhortations, medicine’s safety praxis has been little influenced by the systems approach:

While being widely championed in patient safety, where factors related to individuals, technology and the wider organization are afforded equal consideration […] there is […] evidence that the systems approach […] is still underexploited and could be taken much further.⁴

The National Health Service’s (NHS’s) failure to embrace the systems approach has occurred against the backdrop of a growing contingent liability for medical error:

With potential legal claims, mostly for clinical negligence, now totalling more than £26 billion, the NHS is facing unsustainable liabilities […]. There is no sign of any improvement in reducing the incidence of harm being caused to patients […].⁵

Against this backdrop of limited progress in the application of systems-thinking and a growing contingent liability, a proactive, systems-thinking-inspired risk management tool – the Normal Operations Safety Audit (NOSA) – was trialed at an English NHS Trust.

Organised by a university medical school, this research had three objectives:

1. to evaluate the safety benefits of conducting a NOSA in various clinical settings;

2. to assess whether fifth-year medical students could conduct a NOSA;

3. to assess the educational benefits for students of conducting an audit.

The project was developed and supervised by the paper’s author (referred to as “Convenor”). The author is certificated to work on the flight-deck. He has spent 1470 h on the jump seat, these hours being accumulated as follows: 232 sectors (a sector being an airport-to-airport flight) on the A319; 66 sectors on the A320; 62 sectors on the A321; 82 sectors on the B737; 181 sectors on the B757 and 7 sectors on the A300. The author has flown gliders and has performed a landing in a 737-300 simulator.

Regarding the research described in this paper, ethical permission was granted by the University of Leicester, England’s University Ethics Sub-Committee for Medicine and Biological Sciences. The agreed project title was “Exploring the value of holistic observations of clinical practice: developing a student observational learning tool”. The observers were full-time medical students who volunteered. The ethics sub-committee did not require the study to obtain observers’ or observees’ consent. This was an integral part of the observers’ medical degree.

Systems-thinking – its meaning and application in aviation

Systems-thinking draws on ethnography, participant observation, action research, oral history and mass observation. To paraphrase Waterson and Catchpole,⁴ systems-thinking is not so much about applying the “right” type of knowledge to a problem, but about applying the right approach. Systems- thinking is a frame of reference with a simple premise – that human error can be induced. For example, a badly designed display may cause a pilot to misread an instrument.⁶

Systems-thinking in aviation

Aviation has pioneered the systems-thinking approach to risk management and accident investigation. Watershed moments include Moshansky’s 1992⁷ analysis of the 1989 Dryden accident and Haddon-Cave’s 2009⁸ analysis of the 2006 Nimrod loss. Complex systems – prone to dynamic events such as emergence and practical drift and subject to social, economic and political pressures – are difficult to manage.^9–14 Lagadec¹⁵ and Perrow¹⁶ associate complexity with vulnerability. Understanding how in reality systems work is the sine qua non of successful system management.

The 1972 Florida Tri-Star disaster¹⁷ (101 dead) and the 1977 Tenerife disaster¹⁸ (583 dead) convinced the industry that it needed:

1. a better understanding of routine flight operations and

2. improved teamworking, both on and around aircraft.

Human factors tools were developed. First, crew resource management improved teamwork and resource utilization.^19,20 Second, NOSA documented the reality of flight operations.

NOSA

Recognizing the mutability of the system as designed, NOSA documents the system as found. Systems-thinking tools, such as NOSA, assume system behavior to be an emergent property of complex, hard-to-discern interactions between human and nonhuman components (e.g., personnel, equipment, resourcing, rules, regulations, personal ambition, corporate aspirations and the law). Systems-thinking challenges the false certainties of reductionism.¹⁴

Executed by trained observers familiar with flight operations, a NOSA reveals the lived reality – the verité – of flight-deck labor. Observers’ freedom to roam and probe reflects NOSA’s grounding in actor-network theory, specifically Latour’s²¹ exhortation that researchers must “follow the actors”.

A NOSA is sensitive to phenomena such as practical drift and emergence, where “simple entities, because of their interaction […] can produce far more complex behaviors as a collective […] ”.²² A NOSA describes:

1. the threat environment (e.g., substandard air traffic control);

2. the number and type of errors made by flight crew (e.g., intentional noncompliance with a rule);

3. coping mechanisms:

   [R]outine threats to the safety […] of the system are constantly being managed by the system before they lead to serious outcomes […] this information is often not captured […] by the organization. [NOSA] provides a means by which this can be achieved;²³

4. good practice (e.g., safety innovations introduced by personnel).²⁴

   NOSA meets Hollnagel’s²⁵ Safety-II standard. Specifically:

   1. safety management should be proactive;
   2. safety initiatives should be tailored through topographic research;
   3. because of their local knowledge, workers should be at the center of risk management processes.

The NOSA methodology is promoted by the International Civil Aviation Organization.

Methodology

An example of action research, the project tested claims that a NOSA can – by producing a topographic account – help managers understand the lived reality of a labor process. A literature search conducted in early 2016 by the medical school found no references to NOSA in the medical safety literature.

Potential impacts – pedagogic

With reference to theories of immersive/experiential learning²⁶ and action learning,^27,28 the project offered students the opportunity to:

1. conduct in vivo research into a complex, politically charged and difficult to solve problem (patient harm);

2. be within a bespoke problem-solving team;

3. use a research instrument that demands of the user solid nontechnical skills;

4. with the possibility that findings would inform policy and action.

Potentially the project would improve students’ team-working, observational, communication and problem-solving skills.

Potential impacts – organizational

Denscombe²⁹ observed, “Early on, action research was […] seen as research specifically geared to changing matters […] this has remained a core feature […]”. Lewin³⁰ characterized action research as a “spiral of steps […] composed of a circle of planning, action and fact-finding about the result of the action”.

Drawing on Lewin’s³⁰ and Denscombe’s²⁹ formulation, the Convenor intended the data to provoke change within the research setting (the Trust). To this end, he:

1. organized a feedback session for participating medical practitioners and academics;

2. wrote a journal paper;

3. sought funding for a larger project.

Research instrument

Unlike an aircraft flight-deck, a medical facility (e.g., an accident and emergency department) is a permeable workspace open to actors with varied roles (e.g., doctors, nurses, ambulance crew, porters, cleaners, police officers). Consequently, the standard University of Texas Human Factors Research Project NOSA Threat and Error Management Worksheet³¹ was simplified to create the more functional Threat and Error Assessment and Management Worksheet (TEAM-W; Figure 1).

Figure 1.

Blank Threat and Error Assessment and Management Worksheet.

The TEAM-W coding system (Table 1) was developed by a clinician with a working knowledge of NOSA. It is reproduced in the “Quantitative analysis” section.

Table 1.

Threat and error codes

Threats (Code 100–499)
Code	Scored n times
100 Human in origin	5
101 Infection, introduced by:	2
1011 Visitors	0
1012 Staff	19*
1013 Patients	1
102 Exhausted staff	14*
103 Temporary staff	0
104 Insufficient staff	32*
105 Insufficient range of skills among duty staff	12*
106 Lack of leadership – junior staff making high-level decisions	4
107 Mandatory training naïve	1
108 Health and safety issue – staff, e.g., alcohol, blood-borne virus, mental health	3
109 Health and safety issue – patients	4
1091 Physical health	4
1092 Mental health	6
1093 Violence	2
110 Inadequate patient notes available	31*
111 Wrong patient labels in notes	0
112 Wrong investigation results filed	0
113 Similar patient names in a clinical arena	2
114 Copying and pasting from one software program to another	2
115 Not logging out in clinical area	1
116 Mislabeling specimens	2
117 Mixing biopsies/smears/aspirates	0
118 Laterality	0
119 Leaving swabs in situ	0
120 Privacy and dignity	6
1201 Staff-related, e.g., discussing patient in corridor, leaving patient lying in urine and so on	24*
1202 Institution-related, e.g., no private room	20*
121 Abnormal result not acted upon	0
1211 Not escalated	0
1212 Escalated, but did not reach frontline team in time, e.g., message not passed on	0
122 Risk of pressure sores	0
1221 Lying on trolleys awaiting a bed	1
123 Risk of deep-vein thrombosis/pulmonary embolism	0
124 Too many patients on theater list or clinic	6
125 Smoking on site	0
126 Slipping	2
127 Manual handling	3
128 Cultural issues	0
1281 Insensitivity to cultural norms	0
1282 Imperfect knowledge of spoken tongue	14*
129 Information blindness (useful knowledge overlooked)	2
1291 Willful	0
1292 Accidental	1
130 Team dynamics	13*
Total	239

Threats
Code
200 Technological in origin	20*
201 Slow computers	6
202 Software misadventure	8
203 Lack of Wi-Fi	1
204 Electronic prescribing pitfalls	4
205 Dual electronic and hard-copy prescribing on the same patient	2
206 Awaiting import of digital radiology images from elsewhere	3
207 Confidentiality	13*
Total	57

Threats
Code
300 Building	1
301 Layout design	40*
302 Air conditioning	15*
303 Cleanliness	2
304 Seating and posture in offices	5
Total	63

Errors (Code 500–1300)
Code
500 Prescribing	6
501 Drug due to handwriting	0
502 Drug due to drug interaction	0
503 Drug due to comorbidity, for example, renal impairment, liver failure, age	0
504 Controlled drug error	0
505 Allergy	4
506 Dose	2
507 Frequency	1
508 Units	0
509 Duplicate digital and hard-copy prescribing	1
510 Blood transfusion	0
511 Radiotherapy	0
512 Inadequate or absent thromboprophylaxis leading to DVT or PE	0
513 Antibiotic prophylaxis not prescribed prior to procedure, leading to sepsis	0
514 Failure to bridge/stop anticoagulant appropriately prior to procedure, leading to bleeding	0
515 Stolen prescription pad	1
Total	15

Errors
Code
600 Ordering investigations	2
601 Wrong patient	0
6011 Radiology	0
6012 Non-radiology	0
602 Wrong investigation	2
Total	4

Errors
Code
700 Wrong interpretation of investigation	1
701 Radiology, for example, missed fracture, missed tumor	2
702 Blood result	1
Total	4

Errors
Code
800 Appropriate investigation not performed (e.g., missed fracture)	7
Total	7

Errors
Code
900 Patient given wrong information (e.g., given cancer diagnosis in error, leading to psychological distress, or cancer sufferer told s/he is cancer-free)	6
Total	6

Errors
Code
1000 Equipment	16*
1001 Energy	0
10011 Fire	1
10012 Skin burn	0
10013 Incorrect setting	2
10014 Incorrect result	0
10015 Organ injury	0
1002 Anesthetic	0
1003 Radiotherapy	0
1004 Wrong prosthesis	1
1005 Others, for example, catheter issue due to “bad batch”	3
Total	23

Errors
Code
1100 Mental capacity	0
1101 Failure to properly determine, leading to inappropriate decision	0
Total	0

Errors
Code
1200 Intervention	2
1201 Never-event occurs	5
Total	7

Errors
Code
1300 Failing to follow prescribed procedures	9
1301 Guidelines not followed	11*
1302 Unaware of skill-set	0
Total	20

Note:

^*The Convenor judged that threat and error subcodes scored 10 or more times merited case studies.

Reflections on the methodology

Reflection³² revealed potential pitfalls:

1. The data could be skewed by the Hawthorne effect: persons subject to observation may modify their behavior.³³

2. The data could be skewed by experimenter bias: observees’ identification with the observer may cause them to modify their behavior.

3. The data could be skewed by observer bias: observers’ preconceptions may influence the choice of scenario and interpretation of same.

4. The data could be skewed by TEAM-W’s coding structure: to a degree, coding systems focus researchers’ attention.

5. Observer cognitive overload could cause data to be misinterpreted or lost: information overload and prioritization errors can result in observer task saturation, reducing situation awareness.³⁴ Medical settings can appear chaotic.

6. Knowledge deficit could reduce accuracy: student-observers might lack the knowledge and experience required to make accurate observations.

7. TEAM-W could be dismissed as derivative: “[R]esearch is never conducted without reference to other studies”.³⁵

Research team

The researchers were university medical school fifth-year students due to progress to foundation training in 2017. In the United Kingdom, medical graduates who elect to work in the National Health Service spend up to nine years as a Junior Doctor. Two years are spent as a Junior Doctor foundation trainee, then either three years as a GP trainee, or seven years as a Hospital Speciality trainee.³⁷

Students volunteered to join the team in fulfillment of one of the student selected components (SSCs) of their degree (The General Medical Council mandates that SSCs must constitute a minimum of 10% of course time. SSCs allow students to demonstrate mandatory competencies.). The project commenced with a day’s introduction to TEAM-W. The training mixed a video presentation – the award-winning educational video “Recognizing risk and improving patient safety – Mildred’s Story”³⁸ re-presented through a NOSA lens – with PowerPoint presentations and a question-and-answer session.

The 11 volunteers (7 females, 4 males) were divided into five mixed-gender groups (four groups of 2 students and one group of 3). They were given a timetable of appointments with clinicians and told that they would return for an interim wash-up (debrief) at the end of Week 1 and a final wash-up at the end of Week 2. They were told that:

1. each group would make a case study-based PowerPoint presentation in the final wash-up;

2. each student would complete a Competence Log Book (to be signed off by the Convenor), as shown in Figure S1 (only the first two pages are shown);

3. each student would complete their own TEAM-Ws, to be given to the Convenor at the final wash-up in either electronic or hard-copy form.

During his visits with the various teams, the Convenor made notes, some of which he later transcribed onto TEAM-Ws.

Fieldwork

Ethical permission was granted by the NHS about a month before the project commenced. Although the NHS permit did not require that participants’ informed consent be secured, on request, observers discussed the purpose of their observations/questions with the observees/interviewees. The five groups rotated through a number of NHS clinical settings over a period of 2 weeks during summer. Settings included a mental health facility, a fracture clinic (acute), a urology ward and general and vascular surgery theaters. The Student Roster is reproduced in Table S1.

At liberty to talk to anyone (e.g., staff, patients, relatives) and observe any intervention or procedure, the students were able to describe:

1. subjects’ actions and

2. systemic influences on behavior.

To facilitate completion of TEAM-Ws, the students were given clipboards. Appropriate ID was displayed. The days could be long and busy, with few breaks. The Convenor rotated between medical settings offering support and making his own TEAM-W notes (although not incorporated into the final data set, one observation is reproduced in Table S2 and Figure S2). There were few issues. Most staff members were receptive (despite occasionally not having been told about research). The major difficulty was locating staff and students in sprawling facilities. A representative sample of threats and errors recorded ten or more times by the students is produced in Table 2. The narratives are the students’ own words.

Table 2.

Qualitative analysis (case studies)

(A) Threat: 1012 infection, introduced by: staff
Event	Threat X	Error X	How the event was managed and the outcome(s) (M&O)	Lessons learned by the observer (LL)	Event code (EC)
The Dr did not wash hands before going to see and examine the patient (occurred eight times)	X		The Dr washed hands after each patient, but not before	It is not sufficient to wash hands only after seeing a patient. Hands must be washed again before seeing the next patient	1012
A Charge Nurse entered the patient side-room without the required PPE. There was a notice on the door which says to use aprons and gloves before entering	X	X		Patients are inside the room due to infectious disease. The Charge Nurse risks contracting the infection himself by entering without PPE, and also passing the infection to other patients	1012
Failure of medical staff to change PPE (apron and gloves) when moving between patients in an isolation bay	X	X	A nurse pointed out that they should be changing their PPE in between each patient, and after that, they did	Staff may be aware of the guidelines, but may choose to follow them only when reminded	1012
Doctor did not wash hands before or after examining patients. Wore gloves Doctor washed hands once during a clinic where he saw 10 patients On one occasion, the consultant entered and examined a patient. He neither wore gloves nor washedhis hands. He washed his hands afterwards		X	The error was not acknowledged	Possibly, doctors believe that if they use gloves, they do not need to wash By not washing, the consultants may help normalize such behavior	1012

(B) Threat: 102 exhausted staff
Event	Threat	Error	M&O	LL	EC
[Fracture clinic] Overworked medical staff and overbooked clinic. Dr should have 15 patients instead of 20. Other Dr should have 20 patients instead of 25. Nurse should have 8 patients instead of 16	X		All patients were seen, but higher potential to miss information, to make incorrect decisions, causing patient harm	Not enough staff or resources for the demands placed on the department [by 2016, NHS finances were precarious39]	102
[Operating theater] Staff member overcome by heat. Felt faint. Had to step away from the operating table. Very hot. Had not had time for lunch	X		Unscrubbed. Sat. Drank water	Aircon and heat problematic	102
[Operating theater] Surgeon has been on feet for 2.5 h. No rest. Complains of feeling uncomfortable. Shortly after he nicks artery	X		Artery clamped and sutured	No chairs available for surgeons Provision for longer surgery (breaks, chairs) not made	102

(C) Threat: 104 Insufficient staff
Event	Threat	Error	M&O	LL	EC
One HCA monitoring a four-bed bay plus three side-rooms. Two patients just back from angio suite and theater require 15 min observations	X			Competing priorities for understaffed bay Observation times not achieved because of competing priorities	104
HCA monitoring post-op patient unaware of what he is meant to be doing. He says his English is not great, so he was finding it difficult. Another member of staff kept having to tell him what to write	X			Inadequate supervision Other members of staff having to support HCA Lack of staff meant there was little support for the new staff member	104 and 105

(D) Threat: 105 Insufficient range of skills among duty staff
Event	Threat	Error	M&O	LL	EC
[Flexible cystoscopy list, Urology] Patient with an artificial urethral sphincter does not know how to deactivate it (which must be done before the procedure)	X		Several unsuccessful attempts were made to deactivate it. The balloon would not stay empty. The scope would not pass The Dr called the consultant and asked the patient to return to the waiting room. He wanted the consultant to have a go		105
A new device was being used in theater to move patients from a supine to a prone position. Some staff had little experience with the equipment	X		One member of staff familiar with the device was able to instruct others	Staff are not always formally trained when a new device is introduced The new device was meant to reduce the risk to staff of manual-handling injuries, but [the manner of the device’s introduction] seemed to increase the risk of injury to the patient	105

(E) Threat: 110 Inadequate patient notes available
Event	Threat	Error	M&O	LL	EC
Information on a board in the Staff Office relating to patients was out of date	X		By the end of the shift, a member of staff had been identified as being responsible for updating the board	Information on patient review dates is not always kept up to date Once the problem was identified, action was quickly taken	110
[Vascular clinic] Assuming the hospital would have a record of their medications, a patient brought neither their medications nor a list of their medications	X		The GP’s letter (seen by the Dr) did not list the medications the patient was describing. The patient did not know the names of the medications, but knew vaguely how they worked. The patient was given a prescription and asked to check the names of the medications in the prescription	Overreliance on hospital notes No coordination between the GP and the hospital, that is, no shared program or information No request to patient asking them to bring medications to clinic	110

(F) Threat: 1201 Privacy and dignity: staff-related, e.g., discussing patient in the corridor, leaving patient lying in urine and so on
Event	Threat	Error	M&O	LL	EC
Consultant talked loudly about patient in the Ward Reception Area		X	Consultant encountered colleague involved in the care of a patient who was due in theater. The consultant loudly proclaimed to staff present in the Ward Reception Area: “[patient’s name deleted] scrotum is all falling apart”	There is insufficient respect for privacy when discussing patients with the staff Information about patients is not always transmitted to other members of staff in appropriate settings	1201
[Operating theater] Patient had been covered for dignity. Surgeon uncovered patient before scrub. Unnecessary		X	HCA re-covered patient	Patient only exposed when time comes to drape	1201
Consultation between doctor and patient occurred with the door wide open This occurred for every consultation observed, except one. The only time the door was closed was when the doctor noticed he was being observed		X	The entire consultation was done with the door open	The room had not been designed for clinical consultations. It had no windows and no aircon. It was a warm day. Opening the door was the only way to keep the room cool and comfortable for both doctor and patient	1201
Consultant staff heard calling junior doctors “communist”. [In 2016, the government imposed a new contract on junior doctors. Strikes followed. NHS England postponed nearly 13,000 routine operations, cancelled study leave and holidays and redeployed consultants36]		X	Not challenged	All work environments are professional environments	1201

(G) Threat: 1202 Privacy and dignity: institution-related, for example, no private room
Event	Threat	Error	M&O	LL	EC
Curtains not closed during ward round. This happened with the majority of the patients		X	Ward round continued – curtains not drawn	Poor compliance by all members of the team regarding confidentiality	1202
[Morning vascular ward round] Patient dignity: examined femoral pulses and groin in front of at least nine people with the curtains open		X	Examination continued. Patient appeared mildly embarrassed. This can undermine trust and the doctor–patient relationship, resulting in the patient not opening up, not consenting for other examinations and the Dr unable to do her job effectively	Routine nature of job/workload means little things – for example, explaining the examination – are forgotten/not done Lack of appreciation of patient dignity in some situations by health care professionals	1202
Shouting names of patients with appointments in clinic’s reception	X		The calling of names was to direct patients to their areas when it was their turn to see a health care professional	This is common practice A ticket system that uses numbers rather than names is better and maintains confidentiality, but this requires money that the department does not have	1202

(H) Threat: 1282 Imperfect knowledge of spoken tongue
Event	Threat	Error	M&O	LL	EC
Translator not available, therefore a staff member had to be used to give patient information during the ward round	X		Staff member was used to receive and give patient information about their care. Unclear if translation was correct	Translators are not prebooked by junior doctors for ward rounds. Over-the-phone translators are either not used or not known about High risk of misinterpretation Ad hoc NHS translators then liable?	1282
A patient was missed out during ward round because no one spoke the language	X		It was decided to return when an interpreter was present		1282
[Flexi-cystoscopy] A Slovak woman with no English arrived for a procedure. As there was no translator, the patient could not be consented			The patient was sent back to the waiting room A translator was requested No one arrived A translator could not be found. Consequently, the clinic cancelled the woman’s appointment. No one told the woman (who was waiting for further information)		1282

(I) Threat: 130 Team dynamics
Event	Threat	Error	M&O	LL	EC
Junior doctor talking to consultant. Junior doctor had not asked sufficient questions to determine whether a patient could be discharged. Nevertheless, he told the consultant that he thought the patient could be discharged: “I think so”, opined the junior doctor		X	Junior doctor subsequently returned to the patient to ask the relevant questions	Junior doctors fear telling consultants that they have missed something	130
Administrators failed to pass on a message from a patient who left 2 days ago explaining they could not make an appointment		X	Clinical staff were unaware of the cancellation Because the patient had given >24 hours notice, had clinical staff been aware of the cancellation, they could have booked another patient into the slot	Patient-initiated cancellations are often lost in the system Clinical staff believe there is a divide between those administrators who did the bare minimum and those who did more. They said it was unjust that administrators who did more could not be rewarded	130
A consultant discovered that his appointments to see patients and relatives – made days earlier – had been changed	X		This meant that some time slots had to be rushed In some cases, other parties, such as family members, interpreters and police officers, are left waiting	Planning is sometimes overturned Despite some patients and third parties having to wait, none complained	130

(J) Threat: 200 Technological in origin
Event	Threat	Error	M&O	LL	EC
Shortage of dictaphones in clinic			Consultant had purchased one	Equipment shortages are the norm	200

(K) Threat: 207 Confidentiality
Event	Threat	Error	M&O	LL	EC
A patient handover list with identifiers was left unattended by a computer and another list was left on a desk		X	The lists were not moved by staff Because nobody took ownership of the lists, I shredded both	Confidentiality is not considered important on wards	207
A new patient was called in. The doctor still had the X-ray and clinical details of the previous patient on the screen		X	The doctor did not see this as a problem and continued		207
A consultant with no ID visible entered the ward and began reading patient notes		X	The consultant was challenged by a nurse The consultant apologized for not making his ID visible		207

(L) Threat: 301 Layout design
Event	Threat	Error	M&O	LL	EC
Long queue to check in to fracture clinic delays care and obstructs those with broken limbs, who need more space, for example, persons on crutches	X		Previously, patients and relatives have had to sit on the floor	Overbooking exacerbates the overcrowding problem The clinic’s waiting area is too small	301
Cables and cords lying all over the floor of the theater Possible to trip and disrupt surgery, for example, by desterilizing surgery or an open area, creating an infection risk	X		Staff stepped around the cables	Floor sockets could reduce the trip/infection hazard	301

(M) Threat: 302 Air conditioning
Event	Threat	Error	M&O	LL	EC
No air-con in the windowless room where the manipulation was taking place, so the door was left open	X		The only time the door was closed was when an X-ray was being done	Poorly designed rooms lead to failures in maintaining confidentiality. [Was the room fit for purpose? This case demonstrates how systems theory can help explain rule violations – in this case, the doctor’s failure to maintain patient confidentiality]	302
[Vascular ward round] No air conditioning	X		Doctors were sweating profusely, as was I. Patients were obviously uncomfortable and had to drink more	Potential for dehydration and infection spread, leading to longer stays in hospital and higher rates of morbidity and mortality	302

(N) Threat: 1000 Equipment
Event	Threat	Error	M&O	LL	EC
[Trauma theater] Only one K-wire set left. It was used in the procedure	X			System unable to ensure adequate surgical supplies Surgery may have been delayed if second K-wire set was required	1000

(O) Threat: 1301 Guidelines not followed
Event	Threat	Error	M&O	LL	EC
A patient was brought into theater, then exposed from the waist down. The theater doors were left open. Passing staff could see the patient	X	X	Staff entered and exited through the doors, but no one closed them They were only closed when the next patient (under local anesthetic) was brought into theater	At times, the dignity of the patient was compromised It appeared that in the case of patients under general anesthetic, the doors were left open In the case of patients under local anesthetic, the doors were closed Staff might have left the doors open because of the heat. Their response to an environmental threat [heat] compromised patient dignity	1301
[Theater] Theater commenced without the team briefing being completed This could have led to confusion over theater patients’ care plans	X	X	It seemed the plan was communicated only when the World Health Organization checklist had been completed	When questioned, staff were unsure why the team briefing had not been completed It seemed that practices had emerged to deal with situations where briefings had not been completed [systems theory10,11,40,41 suggests reasons for partial/absent team briefings]	1301
Two intensive care unit (ICU) patients were judged ready to be moved to a ward. There were no beds available. A bed became available out-of-hours. Contraryto guidelines, one of the patients was transferred out of the ICU in the middleof the night, resulting in an elevated levelof risk to the patient. In this case, it also caused the doctor and the patient’s family a degree of confusion and inconvenience	X	X	Staff claimed they had raised the issue of bed shortages with managers	Only hospital managements can resolve resourcing issues Staff should follow guidelines	1301 and 300

Abbreviations: F1, Charge Nurse; PPE, personal protective equipment; NHS, (UK) National Health Service; HCA, Health care assistant.

Presentations and sign-off

On the final day of the SSC, each group made a presentation. Although invitations were sent to clinicians involved in the research, none attended. The Convenor judged the presentations to be of exceptional quality. The case studies revealed numerous issues. Students reflected on NOSA’s suitability to health care. Students’ reflections are listed in Box S1. The Convenor judged that all students had passed the SSC. Competence Log Books were signed-off.

Student and convenor insights into the methodology

These are as described in Box S1. Generally, the students encountered few difficulties, although the staff could occasionally be suspicious of the study. Students attributed the majority of adverse reactions to staff not being told in advance. Without exception, students claimed that the study had improved their awareness of patient safety issues. They also claimed it had improved their situation awareness, observation and communication skills and self-confidence.

Although students considered the coding system a good first attempt, all stated that it required further development. Use of a paper-based recording system (TEAM-W) made record-keeping and data analysis laborious, time-consuming and demanding. All felt that consideration should be given to developing an electronic TEAM-W for a modern digital platform, such as an iPad. In the opinion of the Convenor, this would significantly reduce the time required to mine the data for trends and patterns.

Ideally, group assessments of common settings would have been cross-checked for consistency. A number of factors made cross-checking difficult, most notably, the fact that the Convenor’s budget did not include monies for the development of an electronic TEAM-W (that, as discussed, could have been stored on a portable electronic device such as an iPad). The use of a hard-copy TEAM-W significantly complicated data analysis (including cross-checking). The large volume of data collected from multiple sites and scenarios meant it was impossible to cross-check within the time available. There were no funds allocated for the post-trial creation of an electronic database. Monies are being sought for a second field trial, supported by an iPad-hosted TEAM-W developed in consultation with the School of Medicine and NHS.

Data

The student observers (henceforth referred to as observers) generated a large volume of data. Subjected to quantitative (frequency) and qualitative (case study) analysis, the data showed health care to be a threat-rich environment with errors (some consequential) a commonplace.

Quantitative analysis

Where no suitable subcode could be found, the observers scored against the main code. For example, regarding threats that were judged human in origin, 5 were scored against “100 Human in origin”. Regarding threats that were judged technological in origin, 20 were scored against “200 Technological in origin” (Table 1).

Analysis

Pedagogic impact

The School of Medicine summarized student feedback in a pie chart (Figure 2). Although operationalizing NOSA as an educational tool in live settings was not unproblematic (refer observers’ comments in Box S1), observers’ feedback on the educational benefits of the SSC was overwhelmingly positive. Regarding “new skills gained”, typical Competence Log Book comments were:

• improved observational skills. One observer wrote: “I have developed my skills in observing a scenario from more of an objective view, whereas previously I watched just to gain clinical knowledge”

• improved listening skills;

• improved communication skills. One observer wrote:

• “I had to explain the project and make sure people did not feel threatened by us”;

• ability to empathize;

• better at maintaining situational awareness;

• ability to critique medical practice;

• tenacity (desire to “get to the bottom of things”);

• better at uncovering the truth of a situation;

• better at record-keeping;

• better at putting staff at their ease;

• methodical skepticism;

• objectivity;

• ability to “stand apart” (disinterestedness);

• capacity for reflection;

• diplomacy;

• confidence. One observer wrote: “[I have learned] how to question NHS staff […] about the system […] without fearing intimidation, or being criticised for [asking questions]”;

• restraint (the importance for safety of not blaming).

Figure 2.

Student feedback pie chart.

Abbreviation: SSC, student selected component.

Insights into the human factors aspects of health care provision

The following claims should be considered against the potential biases described in the “Reflections on the methodology” section:

1. The elements of health care provision observed during the study presented a threat-rich environment.

2. Errors, some consequential, were made by medical professionals.

3. Systems theory posits that the origins of error are complex. Frequently, they are the product of individual and organizational failings: “[H]uman mistakes […] rarely have a single underlying contributory factor. Error is the product of design, procedures, training and/or the environment”.²⁰ This study confirms the systems theory view of error: some errors resulted from willful neglect. For example, the persistent failure to gel (sterilize) hands. One observer wrote, “[C]onsultant washed hands total of 5 times for approx 30 patients – juniors didn’t wash hands at all”; “Hands not gelled by doctors throughout the ward round”; “HCA touched bin and then touched patient without washing hands […] HCA did not sanitize hands during time on ward and touched 3 further patients”.

   Others, such as the guideline-flouting night-time transfer from ICU of a patient, were induced by circumstance (a bed shortage).

4. Regarding basic safety procedures such as hand sterilization, there appeared to be a subculture among consultants and doctors of ignoring advice (e.g., signage reminding staff to gel). Referencing Hatch’s⁴² work, Bennett and Stewart⁴³ observed: “Organizational culture is seldom monolithic. Organizations often consist of numerous subcultures, constituted in part through workers’ shared interests, beliefs, skills and profession”. Subcultures produce “inconsistencies”.⁴⁴ Several questions occur. For example:

   • Why did some consultants and doctors ignore hygiene guidelines?
   • To what degree does consultants’ behavior influence doctors’ behavior (e.g., in regard to hygiene)? Armstrong⁴⁴ noted how an organization’s culture is shaped by senior management. Consultants’ behavior could be normative. Janis⁴⁵ notes a proclivity for “concurrence-seeking behavior” in tight-knit groups. He claims the members of such groups are subject to “conformity pressures” and that group behavior may exhibit “derangement”.
   • Is it reasonable to conclude that health care fosters subcultures that harbor deviant behaviors?
5. NHS England⁴⁶ defines never-events as

   [S]erious incidents that are wholly preventable […]. Each Never Event […] has the potential to cause serious patient harm or death. However, serious harm or death is not required to have happened […] for [an] incident to be categorized as a Never Event.

   Observers recorded five never events (code 1201). As the nomenclature implies, a never event has the potential to harm or kill. Never events include:

   • wrong-site surgery;
   • retained foreign object postprocedure;
   • wrong-route administration of medication;
   • scalding of patients.⁴⁷

   The data should be considered against a background of public concern about patient safety.

Conclusion

Regarding the project’s first objective (see Introduction section), the safety benefits of conducting a NOSA in health care include:

• identification of bad and good practice;

• reasons for work-arounds (expedients);

• through the production of thick description, documenting the lived reality of medical labor;

• provision of information in support of informed policy-making by the government and trusts.

Regarding the project’s second objective, under the mentorship of the Convenor and clinicians (Table S1), the fifth-year students completed a large number of TEAM-Ws to a high standard. The forthright nature of the comments evidenced a lack of inhibition (suggesting confidence in the methodology and a desire to contribute). The Convenor’s promise that data would be anonymized helped secure observers’ and observees’ commitment. Anonymous reporting within a just culture encourages flight crew commitment to NOSA.²³ Regarding the project’s third objective, a School of Medicine survey confirmed the project’s educational benefits.

Of course, the above claims should be considered against the possible research biases discussed in the “Reflections on the methodology” section. To reprise one potential source of bias, it is always possible that observees “performed” for the observers. Given the difficulty of quantifying the Hawthorne effect, data, inferences and conclusions should be tested. It is also possible that observers cloaked failings by indulging their own research interests (e.g., by focusing on a narrow range of clinical care issues). As with knowledge generated in the natural sciences, knowledge generated in the social sciences is potentially refutable.

The results of the study suggest three policy developments.

First, the UK General Medical Council should mandate that all medical students perform a NOSA in fulfillment of their degree. As discussed, the 11 students claimed to have benefited in various ways from their participation in the trial. Three said they would like to be involved in further patient safety research.

Second, participating NHS Trusts should be encouraged to act on students’ NOSA findings. NOSA-derived insights should be considered a useful supplement to insights derived from established patient safety systems (such as in-house confidential error reporting systems).

Third, in light of the continuing high level of avoidable deaths in NHS hospitals, the UK Department of Health should adopt NOSA as a cornerstone risk assessment and management tool. It is hypothesized that, other things being equal, groups are capable of producing superior analyses than individuals working alone. Shaw⁴⁸ observes:

This effect can be accounted for by the increased number of judgments in the group […] the wider range of knowledge in the group […] and the influence of the more confident (and more accurate) individuals in the group […].

The NOSA tool should be used by mixed teams. NOSA teams should possess a range of expertise and experience. A team might consist of a radiologist, junior doctor, nurse, paramedic and pharmacist.

NOSAs should be notified to all staff in good time, comprehensively planned and adequately resourced. NHS Trusts must ensure that funds are available to implement suggested remediations. Failure to implement remediations will undermine confidence in NOSA, reduce safety audit buy-in and, possibly, undermine staff commitment to the organization’s goals. Trust is a fragile resource. Taking a long time to build, it can be destroyed in an instant.⁴⁹

Supplementary materials

Figure S1

Competence Log Book.

Figure S2

The clinic’s layout.

Table S1.

Student roster

Wednesday 31st August
Attend	0800 h Outpatients	0755 h Junior doctor handover and emergency Urology ward round	1100 h onward: Urology ward	0845 h Mental health clinic	0800 h Fracture clinic
Observer team (OT)	D	C	E	A	B

Thursday 1st September
	0755 h Theater	1300 h Theater	08:30 h Day Case Theaters	1230–1300 h Meet up with Urology consultant 1330–1445 h Local Urology MDT 1500 h onward: Urology triage – On-call junior doctors	0800 h Fracture clinic
OT	E	D	B	C	A

Monday 5th September
	0755 h Students’ theater	1300 h Students’ theater	0830 h Main theaters	1300 h Flexible cystoscopy list, Urology	0845 h Mental health clinic	0800 h Trauma theater
OT	E	C	A	A	B	D

Tuesday 6th September
	0900 h Ward with FY1 doctors	0755 h Junior doctor handover and emergency Urology ward round	1330 h Urology outpatients	0845 h Mental health clinic	0800 h Trauma theater
OT	D	C	A	E	B

Wednesday 7th September
	0800 h Ward with Matron and HCA	1300 h Outpatients	0845 h Lithotripsy (ESWL) list	1300 h Urology ward, junior doctors	0845 h Mental health clinic	0800 h Fracture clinic
OT	E	C	B	A	D	C

Thursday 8th September
	0900 h Ward round	1300 h Matron/HCA on ward in the afternoon	0845 h TRUS biopsy of prostate list, Urology	0800 h Fracture clinic
OT	B	C	E	D

Note: The students were divided into 5 groups: A, B, C, D, E.

Abbreviations: HCA, Health-care assistant; ESWL, Extracorporeal shock-wave lithotripsy; MDT, Multi-disciplinary team; TRUS, trans-rectal ultrasound.

Table S2.

Convenor-generated observational data

Afternoon urology clinic in a large city hospital.

Consulting room	In	Out	Minutes	In	Out	Minutes	In	Out	Minutes
6	1347 h	1410 h	23	1419 h	1439 h	10	1448 h	1512 h	24
8	1408 h	1415 h	7	1422 h	1426 h	4	1436 h	1442 h	6
9	1416 h	1425 h	9	1428 h	1438 h	10	1440 h	1455 h	15

Note: No scheduled breaks for staff (although some were able to take an informal break). Timings for early part of clinic.

Box S1.

During their end-of-SSC presentations, the observers made the following comments:

• Observers must be knowledgeable.

• It is tempting for observees to “teach” observers.

• It is tempting for observees to use observers as a resource (e.g., asking them to fetch patient files).

• Compounding the problem of enrollment, observers felt obligated to help when colleagues seemed under pressure (e.g., by closing the curtains around beds to maintain patient dignity).

• It can be difficult to sustain disinterestedness.

• The “filter-down” approach to informing staff of a NOSA is too haphazard. All staff likely to be observed should be informed individually.

• There is a risk that observers could distract clinical staff. “In extremis”, errors could be induced.

• Medical scenarios are often actor and action-rich. Sometimes there were so many observees that it was difficult to maintain situation awareness. The faster the observees worked, the more difficult it was to record actions, interactions, outcomes and personnel changes.

• Because machines must be operated “as directed”, machine-centric work is easier to interpret than patient-centric work (where there may be n ways of achieving a goal/n solutions to a problem).

• The fact that some consultants have their own “modus operandi” adds to the problem of keeping an accurate record.

• The observers received positive feedback from clinicians, mainly because the observers were trying to understand the circumstances behind human error.

• The coding system needs development.

• A NOSA helps staff appreciate that a health care system is a network of interdependent actants.

• Work-arounds recorded by observers should be propagated.

• From a patient safety standpoint, the observers felt that this type of holistic, system-focused audit is more useful than a Care Quality Commission audit.