Effectiveness of systems changes suggested by simulation of adverse surgical outcomes

Abstract

Background

Simulation of adverse outcomes (SAO) has been described as a technique to improve effectiveness of root cause analysis (RCA) in healthcare. We hypothesise that SAO can effectively identify unsuspected root causes amenable to systems changes.

Methods

Systems changes were developed and tested for effectiveness in a modified simulation, which was performed eight times, recorded and analysed.

Results

In seven of eight simulations, systems changes were effectively utilised by participants, who contacted anaesthesia using the number list and telephone provided to express concern. In six of seven simulations where anaesthesia was contacted, they provided care that avoided the adverse event. In two simulations, the adverse event transpired despite implemented systems changes, but for different reasons than originally identified. In one case, appropriate personnel were contacted but did not provide the direction necessary to avoid the adverse event, and in one case, the telephone malfunctioned.

Conclusions

Systems changes suggested by SAO can effectively correct deficiencies and help improve outcomes, although adverse events can occur despite implementation. Further study of systems concepts may provide suggestions for changes that function more reliably in complex healthcare systems. The information gathered from these simulations can be used to identify potential deficiencies, prevent future errors and improve patient safety.

Introduction

What is already known on this subject:

• Previous studies have demonstrated the feasibility of using simulation for the analysis of adverse outcomes (SAO) and have demonstrated that SAO identifies deficiencies missed by conventional root cause analysis (RCA).

• The deficiencies identified by SAO in these studies were amenable to systems changes. However, the effectiveness of error correction by these changes has not been established.

What this study adds:

• Systems deficiencies identified by SAO can suggest changes that may be implemented to successfully limit the potential for recurrence of an adverse event in simulated exercises.

Root cause analysis (RCA) is defined by the Joint Commission as a process for identifying the basic or causal factor(s) underlying variation in performance, including the occurrence or possible occurrence of a sentinel event (involving death, serious physical or psychological injury or the risk thereof).¹ The US Department of Veterans Affairs (VA) and the Joint Commission introduced RCA to the medical community in the mid-1990s and the Joint Commission now requires that healthcare organisations perform an RCA for every sentinel event.² Percarpio and Watts in a 2013 cross-sectional study on the relationship between utilisation of RCA and patient safety at 139 VA medical centres noted that large, high-spending centres conduct more RCAs per year than smaller, low-spending facilities; facilities that perform more RCAs develop more corrective actions, and those which complete fewer than four per year have higher rates of postoperative complications.³ The authors noted that it was unclear if RCAs are associated with a functional patient safety programme or directly improve patient safety. Indeed, traditional RCA in healthcare has significant limitations including recall bias^4–6 and has not been shown to reduce the number of errors.^7–9 Additionally, these limitations often lead to individual human error being the perceived cause of the event instead of the system as a whole.

To remove many of the limitations of a retrospective analysis other industries utilise accident reconstruction to assist in the evaluation of adverse events.^10–12 Similar methods had not been reported in healthcare until recently when the authors developed and tested a method of utilising simulation for the analysis of adverse outcomes (termed ‘simulation of adverse outcomes (SAO)’, formerly referred to as ‘simulation-based RCA’).¹³ The pilot study was developed using a single sentinel event that occurred at the authors’ primary institution and demonstrated that the adverse event could be reproduced, but not consistently. When the adverse event did occur, the SAO identified deficiencies that were missed by conventional RCA. These deficiencies were amenable to systems changes.

In the second study three cases were chosen to increase the variability of the cases and to determine if the SAO method was generalisable.¹⁴ A case of an unrecognised duodenal injury, a case of postoperative bleeding and a case of inappropriate assessment of non-biliary abdominal pain were evaluated. Similar to the pilot study, the adverse event in each case could be replicated but additional causes of error were identified that had not been noted by traditional RCA. This exemplifies the inherent complexity of the healthcare system and the need to evaluate errors within the overall context of the event.

While the previous studies have demonstrated the feasibility of SAO and have demonstrated that SAO identifies different potential causes of error, the effectiveness of error correction has not been established. We hypothesise that systems changes identified through SAO can be implemented to correct deficiencies and may improve patient safety and outcomes.

Methods

To determine the effectiveness of an intervention identified by SAO for error correction a single, previously reported SAO event was chosen.¹³ The case chosen was a missed postprocedural haemorrhage resulting in death, that demonstrated the ability of SAO to identify not only the individual-focused root cause determined by traditional methods, but also an additional systemic root cause. Traditional RCA as performed by the hospital determined that the root cause of the adverse event was negligence of the nursing staff—individuals not checking vital signs consistently. SAO identified that the adverse event also likely stemmed from the fact that identifiable responsible physicians were not readily accessible in the proximity of the unit—a systemic deficiency. The nursing staff did not know who to contact regarding the condition of the patient, or experienced difficulty contacting physicians after noting the deteriorating condition of the patient.

In preparation for re-simulation of the selected case, the authors devised systems changes to address the deficiencies noted in the original series of simulations. In order to compensate for the identified lack of a responsible physician in the proximity, a physician contact list with telephone numbers of the responsible physician was added as a prop in the form of a printed paper list next to the telephone. Nursing staff participants were informed of the presence of this list and that anaesthesia was the designated primary physician team for the simulated unit. In addition, no prompting was given to the participants to address the individual-focused root cause of the nursing staff not checking vitals consistently in this modified simulation.

All other aspects of simulation development and implementation remained identical to the preintervention series. A simulated paper and electronic chart was re-created from extensive review of the hospital records and scripts were developed for the key healthcare personnel and the patients (confederates). The confederates were given pertinent information, such as history and healthcare status at the time of the adverse outcome, but were not informed of the ultimate outcome of the event. An appropriate environment, including equipment, was chosen at our simulation centre. To minimise the focused attention on a single event, the scenario included two patients: the primary patient experiencing the adverse event, a missed haemorrhage and a secondary patient experiencing severe pain while in preoperative holding waiting for a laparoscopic cholecystectomy.

Thirty-two individuals participated in the eight simulated exercises: eight nursing staff test subjects, 16 volunteer patient actors, and 8 residents portraying the ‘on call’ anaesthesia resident in charge of the simulated unit. The hospital nursing staff members all had similar experience to those in the original in-house adverse event but naïve to the original simulations and the actual event. Residents were selected by scheduling availability and included first and second year general surgery residents, as well as second year anaesthesia residents, all deemed by the research team to be of an appropriate clinical level to address the medical issues presented by the simulated exercise. Patient actors were medical students who volunteered their time and were trained with a standardised script during a training session held prior to the simulated exercise. Similar to the work described by Nestel et al,¹⁵ the standardised trained students as patient actors in these simulated exercises served to preserve the fidelity of the scenario.

As the ‘real’ adverse event occurred over 2 h, the simulation exercise had to be compressed. Therefore, ‘simulation time’ was used. Using simulation time, non-critical time can be eliminated so events occurring over several hours can be simulated in minutes. Such non-critical time includes time spent waiting for laboratory results. Using the simulated clock, a lab may be drawn at 10:00 but instead of waiting 30 min for it to be reported, the clock is advanced 30 min.

All simulations were recorded using CAE LearningSpace (CAE Healthcare, Sarasota, Florida, USA) for review and analysis. Immediately after completing each simulation, the test subjects and confederates were ‘de-briefed’ in another recorded session, where they were asked about their decision-making processes during the simulation. They also completed questionnaires regarding fidelity of the simulation and suggestions for improvement. Once this information was obtained, they were informed of the details and outcomes of the actual event and asked for any further comments. All debriefings were recorded, transcribed, and analysed. The results were compared to those of the original series of simulations for this case.

Results

Development, performance and analysis of the original series of simulations required a total of 165 person-hours.¹³ Development of the revised series of simulations required 12 person-hours, including time for meetings and editing of the original script in the context of the suggested changes. Each revised simulation exercise took approximately 2 h including the debriefing. Thus, eight simulations required approximately 16 h of performance time, or 96 person-hours when accounting for all involved individuals. Analysis of results for the revised simulation required 10 additional hours of labour. The revised simulations required a total of 118 person-labour hours, from development to performance and analysis.

In seven of the eight simulated exercises, system changes were effectively applied by the nurse participants. They contacted the responsible physician, the anaesthesia resident, using the telephone list to inform the physician about the concerning condition of the patient. In six out of seven scenarios, the anaesthesia resident responded adequately by visiting the patient, identifying the bleeding and avoiding the adverse event. In one scenario, the nursing subject responded appropriately by calling the resident but the simulated resident (first year surgical resident) failed to address the condition of the patient correctly, causing the adverse event. In scenario number 2 the telephone malfunctioned when the nurse tried to contact the anaesthesia resident. The simulation technicians intervened by allowing the nurse to use their mobile phone but the participant was overwhelmed by this problem, ultimately used the phone but called the wrong discipline. These results are summarised in table 1.

Table 1.

Summary of results of eight simulations performed to evaluate effectiveness of systems changes suggested by previous simulation of adverse outcomes

Simulation number	Adverse outcome avoided?	Key communication or factor contributing to simulation outcome
1	Yes	Nurse: Hey, I need you to get back out over here to outpatient holding. Our pressure is still dropping, the patient's very nauseous and he's saying he has doubts. I called urology, but I think it's—just he's not feeling really good right now. I want both of you to come over please and check him. Thanks
2	No	Malfunctioning telephone
3	Yes	Nurse: Doctor, I know you're next door, do you think you can come over and look at this patient I have at OPS for urology? They can't get here right now and I think she needs someone to look at her…Her blood pressure's dropping a bit…she's really anxious
4	No	Systems change utilised however, anaesthesia confederate did not properly address situation
5	Yes	Nurse: She has some changes in vital signs—she's becoming hypotensive, tachycardic, and tachypneic. At one point she was saying that she wanted to cancel her surgery. I called to speak to the urologist but he's scrubbed in to another procedure, so I was trying to see if you could possibly come over and take a look at her
6	Yes	Nurse: I just paged you about a patient going for lithotripsy, she's an obese patient and she's going for surgery…She had a nephrostomy tube placed, and since she's gotten back her blood pressure has dropped, her heart is racing, and she's disoriented, so I need someone to come see her right away
7	Yes	Nurse: I called a rapid response, and gave her a fluid bolus. She's not responding to the fluid bolus. I would assume I have people at the bedside since I called a rapid response, but I don't know what else we're doing, because I'm still not getting a blood pressure on her. She doesn't look good. She may be bleeding. She's very pale…Someone needs to see her ASAP
8	Yes	Nurse: I have a urology patient here that's supposed to be going back for surgery. They are crashing on me and I can't get anyone to respond from urology. Can you come see them?

Discussion

The study shows that systems deficiencies identified in SAO can suggest changes which may be implemented to successfully limit the potential for recurrence of an adverse event. In the study, those systems changes were utilised successfully in nearly all of cases (seven out of eight), and the adverse outcome was avoided in six of eight simulations. Equally important is the finding that using simulation to test the systems change can identify potential areas for breakdown of the system prior to implementation in the hospital setting. In this study, one breakdown was the apparent malfunction of the phone system which subsequently redirected the attention of the test subject to such an extent that the adverse event was duplicated. This identified the need for redundancy in the system. The second breakdown was not in the new system but in the knowledge base of the individual responding to the crisis. This breakdown highlights the importance of standardised team training, both for simulated teams and in real life—a concept which been elaborated on by Nestel et al,¹⁵ who described the use of medical students and trainees in a vascular unit to portray members of a simulated endovascular suite for carotid stenting and reported that a simulated interventional team proved feasible with these resources, and that authentic psychological fidelity complemented the physical fidelity of the simulated suite. While the test subject nurse did react and call for assistance, the confederate playing the role of the anaesthesiologist did not identify the problem in a timely manner leading to the adverse event occurring. As the crisis was considered ‘basic’, haemorrhagic shock, and our main goal was to evaluate the response of the nurse, we used confederates with experience that did not match that of the expected responder. In the simulation in question, the responding confederate was a first year surgical resident. This may have played a significant role in the lack of identification and correction of the problem. Future work will try to match experience level of not only the test subjects but also the confederates.

There are several limitations to this study. The simulation represents a small sample size as the simulation was only repeated eight times. However, given the relative uniformity in the results we do believe the sample was sufficient to prove the hypothesis. If in future studies significant variability exists, an increase in the number of times the simulation is run may be required. Standardisation of simulations was promoted to the greatest extent possible, but there was variability among the simulations, depending on unscripted reactions to questions posed by participants and also on the experience level or background of the confederates and participants.

The fact that the participants’ actions were being monitored and recorded is also a potential source of bias and inherent drawback to simulation. Participants were consented and aware that they would be recorded during the simulation and debriefing exercises, though videos were reviewed only by the research team. We attempted to minimise this effect by having a second patient involved. This is similar to the work load of the nurses in outpatient surgery during preoperative preparation. By further increasing the verisimilitude of future simulations, this bias may be minimised further.

Although hand-off communications are recognised as a critical patient safety issue and defective hand-offs are known to cause problems beyond adverse events, such as delays in treatment, inappropriate treatment and increased length of stay,¹⁶ the simulations included a standardised patient hand-off written by the simulation team and did not focus on such interventions as the utilisation of situation, background, assessment, recommendation (SBAR) or other communication tools so as to eliminate additional variables not present in the adverse event. Future work may also address this topic.

Hu et al¹⁷ state that as we continue to work to improve safety and efficiency in healthcare, we must simultaneously work to build systems that will help us avert deviations and train providers to anticipate and deal with those that are unavoidable. As our limited experience shows, the medium of simulation may answer the training needs of a medical culture increasingly promoting the concepts of systems of care. Pronovost and Bo-Linn¹⁸ recently suggested that the time has come for healthcare delivery to ‘mature and embrace systems engineering’ such that clinicians may systematically address all known harms that patients may experience in order to improve healthcare and reduce costs. A given series of simulations does not address all known harms that patients may experience to the extent of the studies of a systems engineer, but provides a reasonable medium for an institution to experiment with the principles of systems science. Institutions should consider SAO not only as a medium to improve on traditional RCA processes and test the effectiveness of suggested changes as described in this study, but also to answer the increasing need for novel education and training techniques in the systems and human factors-based approaches to patient safety.

Conclusions

Systems changes suggested by SAO can effectively correct deficiencies and help improve outcomes, although adverse events can occur despite implementation. Simulation may be effective as a teaching tool for training healthcare providers how to react to the realities of complex systems, in addition to serving as a medium for evaluation of adverse outcomes and changes proposed to address root causes of error.