Application of a Systems Theory-Based Accident Analysis Technique to Perioperative Safety Reports from the COVID-19 Pandemic

Abstract

Objectives:

Non-linear retrospective analytic techniques can allow for in-depth understanding of accidents and their causes, yet they are infrequently used in healthcare. The purpose of this study was to provide an example, using Causal Analysis based on Systems Theory (CAST) together with an inductive thematic analysis to understand the contextual factors contributing to one hospital’s perioperative safety events.

Methods:

We created a hierarchical control structure of the hospital’s perioperative system with input from a multidisciplinary group. We then analyzed safety events that were self-reported during a COVID surge (April 2020) using CAST to understand their contributing factors. Next, we analyzed the contributing factors using inductive qualitative thematic coding to identify system-level safety risks. We mapped each system-level safety risk to a recommendation for future mitigation.

Results:

We screened 122 safety reports and found 19 safety events that met inclusion criteria. The analysis revealed 245 contributing factors represented by 22 subthemes corresponding to 3 major themes: 1) vulnerable processes, being problems with workflows or communication channels; 2) personnel challenges including challenges with staff redeployment as well as cognitive and behavioural challenges; and 3) poorly designed or unavailable equipment. Each subtheme corresponded to a prevention strategy, such as creation of a central protocol hub.

Conclusion:

Using a non-linear accident analysis technique together with thematic analysis, we were able to identify system-wide contributing factors to safety events. These contributing factors led to recommendations for future pandemics or crises characterized by scarce resources, limited data, and a rapidly changing environment.

Introduction

Retrospective analysis of safety events is critical to our understanding of how accidents occur and is a vital component of accident prevention and safety culture. Traditionally in medicine we use analytic techniques based on linear models like Domino Theory¹ and Swiss Cheese Model.² While these models assume that accidents occur due to linear causality and independent failures of system components, in healthcare systems with interacting human and software components, these assumptions become overly simplistic.

Complex models, such as the Systems-Theoretic Accident Model and Processes (STAMP),³ are increasingly being adopted within healthcare.^4–7 These models consider non-linear interactions amongst components and the impact of the system beyond clinicians, resulting in the identification of more systemic weakness than linear techniques.⁵ Important factors, like time pressure, are not linear contributors to accidents, yet they impact everyone in the system.³

Causal Analysis based on System Theory (CAST) is an analysis technique that uses STAMP as the causation model. This model assumes that safety arises from the ability of the system to control the behavior of the controlled process it is managing (see Table 1). When behavior escapes those bounds, the system moves into a hazardous state, where an accident might occur. Control is maintained through a control loop, where a controller (e.g. an anesthesiologist) sends control actions (e.g. increase the vasopressor dose) via an actuator (e.g. the medication infusion pump) to the process that it is trying to control (e.g. the patient’s hemodynamics). The controller receives feedback from the controlled process about its current state via a sensor (e.g. the blood pressure reading on the arterial line). The controller uses this feedback to update its mental model, or its understanding of the process that it is trying to control. A system can be modeled as a hierarchical structure, comprised of stacks of control loops. Each control loop layer is analyzed to determine how the system shifted to an unsafe state, encouraging a non-linear view of the system and accident.

Table 1.

Common terminology and definitions from the Systems Theory-based Accident Modelling Process

Term	Definition
Accident	Loss to the system (e.g., financial, inability to achieve mission goals, injury, death). Occurs when a hazard combines with an environmental factor.
Hazard	System state that can predispose the system to an accident. A hazard must be in the control of the system.
Controller	Person or piece of software that issues commands to constrain the behaviour of the pieces of the system below it in the hierarchy
Control Action	Commands given by the controller or actions taken by the controller to act upon the controlled process it is responsible for
Actuator	Hardware or equipment that is used to enact the control action onto the controlled process
Controlled Process	Component of the system that the controller is trying to maintain within safe bounds (e.g., patient physiology)
Sensor	Equipment or workflow in a system designed to provide feedback about the controlled process to the controller responsible
Mental Model	Human controller’s understanding of the state of the process under their control
Control Loop	Construct comprised of the controller issuing control actions which impact the controlled process via the actuator. The controlled process then provides feedback to the controller via the sensor, which allows the controller to update its mental model and issue new control actions.
Hierarchical Control Structure	Graphical representation of a system involving multiple control loops stacked upon each other in layers of control

The purpose of this study was to demonstrate the use of CAST in a healthcare setting on multiple accidents and combine the model results with an inductive thematic analysis of the contributing factors to extract broader lessons about the system from the separate event analyses. While thematic analysis has not been previously used with CAST in a healthcare setting, we hypothesized that this multimodal technique would result in comprehensive lessons learned and generalizable system improvements. We chose to analyze perioperative safety reports during the April 2020 COVID-19 surge because the surge involved significant non-linear pressures and too little time at the height of the crisis to learn lessons from the safety reports in real time.

Materials and Methods

Study Site

This mixed methods study was completed at a 1,000-bed academic medical centre that performed over 54,000 anesthetics per year. The study site used an electronic safety reporting system that prompted clinicians to report safety events at the end of each anesthetic case. The Mass General Brigham Institutional Review Board evaluated the study protocol and exempted it from review after determining that it did not involve human subjects research (Protocol #2021P002310). The study protocol was additionally reviewed and approved by hospital quality and safety leadership. This manuscript adheres to the Standards for Reporting Qualitative Research guidelines.⁸

Safety Report Screening

We obtained safety reports that were submitted to our hospital’s anesthesia department beginning April 28, 2020, which corresponded with the peak of the COVID-19 surge. Two analysts (ASW and KCN) screened the reports chronologically to include only those that 1) represented a safety event (an event which resulted in, or could have resulted in, unintended harm to a patient that is not due to the underlying medical condition of the patient)⁹ and 2) involved the main operating rooms, preoperative evaluation, or postoperative recovery units. Events occurring elsewhere, such as in the interventional radiology suite or the obstetric floor, were excluded as those environments had different workflows, requiring separate hierarchical control structures. We obtained the safety reports in one-week batches to minimize the number of safety reports released and protect patient and clinician privacy. We pulled safety reports in batches of two-week time periods, stopping upon reaching data saturation,¹⁰ defined as the point where we noted no further thematically unique contributing factors after the analysis of three consecutive safety events.

Causal Analysis Based on Systems Theory

According to procedures described by Leveson,³ two analysts (ASW and KCN) completed CAST analyses for each of the safety events. See Table 1 for definitions of commonly used terms in CAST. We first created a hierarchical control structure to represent the components of the perioperative system during the study period by interviewing a multidisciplinary team of ten clinicians (anesthesiologists, surgeons, nurse anesthetists, pharmacists, and perioperative nurses) and four patient safety and perioperative leaders who worked in the operating rooms during the surge. Interview notes were transcribed and used to iteratively revise the structure.

To identify factors contributing to the safety events, we relied on two sources of information: 1) the original safety report, which provided the perspective of the person who filed it; and 2) the hierarchical control structure, which allowed us to identify additional system-wide contributing factors. In using the hierarchical control structure, we first identified the key role groups (e.g., anesthesia clinician, pharmacist, perioperative leadership, etc.) that may have been involved in the safety event. For each role group, we walked through three analytic steps: (1) identifying the control action(s) that could have allowed the system to move into an unsafe state, (2) exploring the possible mental model of the person involved, and (3) identifying factors that could have contributed to the unsafe control action or flawed mental model. Use of the hierarchical control structure was critical to achieve a full understanding of the possible contributing factors because we did not have access to the clinicians who were involved in the safety events. The contributing factors were assessed for plausibility by the same multidisciplinary team that assessed the hierarchical control structure. Table 2 provides an example of a CAST analysis done on a hypothetical safety report. Due to the nature of the data and the need to protect the confidentiality of patients and clinicians involved in safety events, data are presented in aggregate and/or de-identified.

Table 2.

Hypothetical safety report demonstrating the typical level of detail of our raw safety report data and the corresponding CAST analysis. This process was done for all safety reports included in the study. The safety report and analysis presented here are hypothetical to protect the privacy of the patients and clinicians involved in the events analyzed in this study.

Hypothetical Safety report: During induction, I meant to give rocuronium but I accidentally gave succinylcholine to this burn patient who had been admitted for the last week in the hospital. I don’t know how it happened, but I realized the error when I saw the fasciculations. I intubated the patient and then gave rocuronium when twitches returned. I checked a potassium that was normal and watched the EKG which remained normal throughout the case. The beginning of the case was busy, and the medications looked similar.
Controller	Unsafe Control Action	Contributing Factors
Anesthesiologist and/or the Anesthesia Clinician in the OR	Gave the wrong medication	Actuator: Succinylcholine and rocuronium prefilled syringes look similar Sensor: there is no immediate feedback to the provider about a medication that is about to be pushed (e.g. there is no point-of-care clinical decision support with auditory feedback on the medication about to be given) Control algorithm: Cognitive burden is high at induction, especially with a COVID-positive patient and there are new protocols to consider, as in this case
Perioperative Leadership	Stocked medications that looked similar	Actuator: Stressed supply chains may have limited the options for non-look-alike medications in the operating room Sensor: May have been unaware at the point of ordering that the medications have similar packaging
Hospital Incident Command Structure	Created protocols for the care of COVID patients that created a high cognitive load on clinicians	Sensor: Due to delays in the safety reporting system, they may not have been aware of challenges associated with the protocols they created Control algorithm: Protocols may have represented the best possible balance between provider and patient safety. Trade-offs exist when there are two competing priorities.

Thematic Coding of Accident Analyses

We performed an inductive thematic analysis of the contributing factors identified by CAST to capture larger themes and subthemes running through multiple safety events. See Table 2 for an example of the contributing factors used in this analysis. We assembled a multidisciplinary team of four perioperative clinicians - an anesthesiologist (ASW), surgical trainee (RDS), pharmacist (LLT), and anesthesia technician (JRF)- to review the CAST output. All data were coded by two independent reviewers, using an inductive approach - analyzing the data with an open mind, looking for patterns and themes, and interpreting these for meaning.^11,12 The reviewers met weekly with two experts in qualitative research (KCN and RM) to iteratively discuss codes and emerging themes. Disagreements were resolved by consensus. Ultimately the consensus-based decision was recorded, and therefore a kappa statistic between reviewers was not assessed.

Recommendations for System Changes

Finally, from our thematic analysis, we generated recommendations to prevent similar events in the future. A multidisciplinary group consisting of three anesthesiologists (ASW, KCN, RM), one surgical trainee (RDS), one pharmacist (LLT), and one anesthesia technician (JRF) mapped each subtheme from the qualitative thematic analysis to a mitigating recommendation. When a subtheme pointed to multiple candidate recommendations, disagreements were resolved by consensus with prioritization based on clinical experience and relevant literature. This was done to ensure a broad set of high impact and quality recommendations.

Statistical Plan

We used descriptive statistics, including counts with medians and interquartile ranges to describe the findings from the safety analyses. The final sample size of safety reports was determined by data saturation as discussed above.

Results

Hierarchical Control Structure

The hierarchical control structure is shown in Figure 1 (simplified version) and Supplemental Figure 1 (full version). The patient is considered the “controlled process” with layers of clinicians and management structured above. We did not include higher layers of the system, such as governmental agencies, because their actions are outside the scope of our control. Additionally, our hierarchical control structure emphasizes detail at the frontline level and show less detail through the managerial structure because we were working with safety reports filed by frontline clinicians, which naturally represented a greater richness of detail about the safety events at the frontline level. This decision does limit the later analysis in that potential contributing factors at higher system levels, such as the pressure from payors that drives staffing shortages, are not considered fully.

Figure 1.

Hierarchical control structure of the perioperative system. The patient is the controlled process at the base of the structure. Just above the patient are role groups that provide direct patient care or immediately support patient care through providing equipment or medication. The next tier includes clinicians who make decisions about anesthesia but with less direct anesthesia care, including the surgeon and the anesthesiologist (if the anesthesiologist is supervising a nurse anesthetist or trainee). Perioperative leadership, including nursing, surgery, and anesthesia leadership, creates schedules, interprets infection control guidance for use in the operating room, communicates changes in protocols, and gathers safety reports to understand system performance. The Hospital Incident Command System interfaces with the government and creates hospital policies and protocols. For simplicity, all sensors and actuators except for the equipment and medication component have been omitted, as have the detailed control actions and feedback sources. High-level control actions are given in blue, and high-level feedback is given in red. Not all control and feedback pathways are highlighted here. For the fully detailed model structure, please see Supplemental Figure 1.

Safety Report Screening for Selection for Analysis

We screened 122 safety reports from the two-week period with the highest COVID-19-related patient volume and workflow disruption (April 28-May12, 2020). These safety reports were not able to be investigated at the time of reporting due to resource constraints in the setting of the pandemic. Of these, 39 were excluded because they were not safety events (e.g., reports of COVID-positive patients or subsequently resolved controlled substance discrepancies), and 61 were excluded for occurring outside the operating room. Of the remaining 22 safety reports, 3 described the same safety event, leaving a final sample of 19 safety events with 22 safety reports (see Figure 2). Thematic saturation was reached at this point, with three consecutive safety reports yielding no new contributing factors, and so no further safety reports were screened.

Figure 2.

Schematic diagram of safety report screening

Safety Event Analyses

Each safety event involved a median of 3 controllers (interquartile range 2–4) and a median of 12 contributing factors (interquartile range 9–17), for a total of 245 contributing factors to the 19 safety events.

Qualitative Thematic Analysis

We identified 3 themes describing factors contributing to the safety events: 1) vulnerable processes, 2) personnel challenges, and 3) poorly designed and/or unavailable equipment. These themes are not mutually exclusive. For example, a poor interpersonal relationship (personnel challenge) might impact a handoff (vulnerable process). Thus, there were times when, given the detail available in the safety reports, we made a decision to place an event in one theme when it might have also been appropriate under another theme. While process, people, and equipment are intricately interconnected, the division into themes helps organize the lessons to be learned from these events.

The direct quotes reported are from interviews that were completed to build the hierarchical control structure, not from safety reports. Data are presented in aggregate to protect privacy.

Vulnerable processes

Vulnerable processes involved problems with workflows, protocols, and system communication channels. Some were COVID-19-specific challenges, while others were latent vulnerabilities. There were frequent descriptions of uncertainty around new pandemic protocols, policies, or workflows. For example, confusion around use of a viral filter led to possible clinician exposure to SARS-CoV-2 during an equipment exchange.

Multiple communication problems contributed to pandemic-specific events. For example, communication pathways were often unclear as multiple nurses described getting so many emails that it was challenging to stay up-to-date on important policy emails, which contributed to a contamination event in the operating room pharmacy. The rate of change of policies being communicated was also challenging. In an event involving contamination of a clean equipment space with dirty equipment, the analysis showed that procedures and policies were changing faster than could be clearly communicated.

While some vulnerable processes were pandemic-related, others were pre-existing. For example, in an event involving a patient with underlying organ dysfunction who accidentally received an antibiotic overdose, contributing factors included a lack of available information at the point of decision-making, such as lab results and last doses of medications.

Personnel Challenges

As with vulnerable processes, personnel challenges involved both COVID-19-specific problems and pre-existing interpersonal and cognitive behaviours that may have been exacerbated by the pandemic.

A key COVID-19-specific personnel issue was redeployment and its associated challenges with training, unfamiliar work environments, and new clinical work. While many of these incidents were related to workflows and protocols, the contributing factors involved new staff and their unfamiliarity with protocols and workflows that were otherwise intact, putting these incidents into the personnel theme, as opposed to the vulnerable processes theme. One safety report described confusion about the order of patient procedures in the preoperative holding area. Contributing factors included challenges such as a new work environment with unfamiliar clinical workflows and training gaps. For example, a nurse manager described that clinicians were floated to new sites, but “training and on-boarding were disorganized and frequently cut short.” This contributed to inadequate staffing levels, as staff balanced clinical care and on-boarding of new clinicians.

While some personnel challenges arose from the pandemic, others were pre-existing. In the case of the antibiotic overdose described above, anesthesia and surgery clinicians described an inadequate preoperative handoff with the primary inpatient team, which was driven by interpersonal relational challenges. Another challenge was difficulty speaking up in the perceived hierarchy of healthcare teams. This was a contributing factor to the exposure of a junior member of the anesthesia team to COVID-19 because they had not been fit-tested for personal protective equipment and a senior clinician asked them to assist with induction of a known COVID-19 positive patient. The junior team member did not feel comfortable declining to assist.

Personnel challenges also involved typical human limitations, including slip errors, lapse errors and cognitive overload. One safety report described a patient who received an antibiotic to which they had an allergy. An anesthesiologist noted that “the peri-induction period is a time of high cognitive workload, so [medications] may get less thought and mindfulness than a medication given later in the case.” Additionally, clinicians from a variety of role groups noted that additional protocols and decreased staffing during the COVID-19 surge put an extra cognitive strain on providers.

Poorly Designed and/or Unavailable Equipment

Poorly designed equipment involved personal protective equipment and infection prevention tools (e.g., protective plastic coverings), which changed the way equipment functioned. For example, when analyzing a safety event involving a hematoma after attempted line placement, an anesthesia resident noted “it can be difficult, especially at this time with enhanced cleaning protocols and everything coated in plastic, to obtain an ultrasound, leading to more pressure to place lines by landmark.” In another event, a patient had inadvertently been positioned on a small electrode, leading to a pressure injury. One contributing factor noted by clinicians was that the required personal protective equipment for COVID-19 can decrease their field of vision and tactile abilities. Both the arterial hematoma and the pressure injury also involved patient risk factors such as obesity and the use of anticoagulants. While these patient factors are not immediately modifiable, they serve to highlight the importance of improving equipment design.

Unavailable equipment involved supply chain disruptions and increased demand for virus-related products that led to an inability to source an adequate number of viral filters for the operating room, which was a contributing factor identified by an anesthesia technician in a case of a missing viral filter on the anesthesia machine.

Recommendations for System Changes

Twenty-two recommendations were generated, one for each subtheme in our thematic analysis. For example, to address protocol uncertainty due to unclear communication channels, we recommended the standardization of communication of updates, including the timing, format, and medium of communication. See Table 3 for additional examples of recommendations; a complete list of recommendations is available in Supplemental Table 1.

Table 3.

Qualitative analysis themes with a sample of subthemes, recommendations and examples of how they might be tailored to the local context. A full list of qualitative analysis subthemes and recommendations are available in Supplemental Table 1.

Theme	Subtheme	Recommendation	Example Local Implementation of Recommendation
Vulnerable Processes	Protocol, policy, and workflow uncertainty due to lack of accessible protocols	Create a central hub, easily accessible by all employees, with the most up to date information organized by role	Perioperative COVID repository with videos, flow charts, checklists, and full policies available
	Absent feedback	Create clear and rapid channels for feedback from frontline workers to management and, during times of crisis, incident command	Digital tag that clinicians can add to safety reports to tag them as COVID-related and expedite them for rapid review
Personnel Challenges	Cognitive overload	Adopt cognitive aids, like checklists, to assist at times of high cognitive workload and make them readily available at the point of care	“COVID-positive patient induction checklist” displayed on the anesthesia machine
Personnel Challenges	Staff redeployment resulting in unfamiliar work environments	Orient all staff to the new work environment, even if they are being deployed to perform the same clinical work as their pre-deployment duties	Obtain frontline staff input into the setup of newly repurposed locations, and arrange time for staff to familiarize themselves with the space prior to the start of their clinical duties
Equipment	Poor design of equipment leading to difficulty understanding its use	Include instructions for equipment use at the point-of-care	Instruction manuals and/or short videos available by QR code at the point-of-care
Equipment	Poor design of equipment leading to difficulty in use	Consider device design and usability in your purchasing decisions	Complete usability testing prior to ordering new equipment
	Equipment availability	Do not rely on just in time supply for mission-critical devices due to fragile supply chains	In non-crisis times, stock a one-month supply of essential equipment

Discussion

We created a hierarchical control structure to model the hospital system during the peak of the initial COVID-19 pandemic wave in April 2020. Using this model, we then analyzed 19 perioperative safety events with a CAST analysis and a qualitative thematic analysis of the CAST outputs. We found 245 factors that may have contributed to these safety events, which clustered into three overall themes: vulnerable processes, personnel challenges, and poorly designed or unavailable equipment.

While somewhat resource-intensive, this multimodal technique offered several advantages over traditional linear models such as the “5 whys”.¹³ First, it allowed us to use the safety report data in the context of the hierarchical control structure to understand factors that contributed or may have contributed to the safety events. While scant safety report data may not allow for confirmation of contributing factors in a given safety event, CAST allowed us to use the scant data coupled with a deep knowledge of the system to uncover weaknesses that might contribute to similar events in the future. These potential system weaknesses are not easily recognized without the hierarchical control structure. While the safety reports used in this study were not deeply investigated due to the system strain at the time, CAST can become an even more powerful tool when the details gained in a deep investigation are combined with the hierarchical control structure. In an ideal setting, we would recommend that safety reports be investigated thoroughly, with CAST then being used to analyze the data uncovered from both the safety report and the investigation.

Second, this non-linear technique allowed us to better understand the whole system (e.g., both frontline and managerial controllers) in the same model, as well as nonlinear elements, such as interpersonal interactions and time pressure. Finally, the analysis of each safety event using the hierarchical control structure facilitated identification of repeated themes and common contributing factors, which were further elucidated with the inductive thematic analysis on the group of results. For example, dysfunction of the control arrow down from the OR leadership tier responsible for distributing policies and protocols was identified as a contributing factor in multiple different incidents and emerged as a subtheme in the thematic analysis, signaling a key system weakness that should be a target for system improvements.

Using a complex accident model to retrospectively understand the weaknesses of our system revealed specific and actionable recommendations for improvements in the context of the COVID-19 surge. Several groups early in the pandemic recognized the importance of safety reports and created systems to use them as real-time data for policy and protocol changes.^14,15 However, they were necessarily limited to less complex analyses of these events, typically looking only for the most obvious contributing factor at the time of initial analysis, because of the need for fast turnaround of the analyses.

The themes that we identified align with previous COVID-19 work, which has identified the need for improved data management and information sharing,¹⁶ the challenges inherent in scaling capacity into new care locations,¹⁷ the challenges around staff redeployment and shortages,¹⁷ and the need for centralized information repositories.¹⁸ Other institutions described challenges with obtaining appropriate personal protective equipment and other supply chain disruptions.¹⁹ The agreement of our findings with those of other institutions adds validity to the insights we gained from our model. However, using CAST, we further identified factors such as the rapid rate of protocol change and the increased cognitive burden of the COVID-19 protocols that threatened safety during this period. The breadth of systemic weaknesses that can be identified is a strength of the non-linear models and can help justify the increased effort in building them.

This study has several limitations. First, as with any qualitative analysis, our results were impacted by the biases of the research team, which we limited through the use of rigorous qualitative methodologies. This is additionally true of any system model where analysts make decisions about how and what to model. There are trade-offs between detail and complexity, and we made these decisions jointly as a research team to check our inherent biases. Second, in generating recommendations, there were instances of multiple candidate recommendations for a single subtheme. We narrowed these down to one recommendation through discussion and consensus. Further research should focus on prioritizing recommendations for testing and implementation. Our department is using these analyses to prioritize improvement initiatives. Third, this is a single-centre study from a large, tertiary-care academic medical centre, possibly limiting the generalizability of our COVID-19-specific findings. However, the use of CAST is broadly applicable to retrospective analysis of any safety event, including outside of the context of a crisis. Fourth, this study used a relatively small set of safety reports for analysis compared to other types of safety report research. However, our sample size was determined by data saturation, which is consistent with qualitative research methodology, and can be reached with much smaller sample sizes than in quantitative research because of the depth of information available in qualitative data.²⁰ Finally, this study is set up as a case study to show an application of a non-linear technique for accident analysis. CAST, with its basis in systems theory, represents a theoretical improvement in the number of contributing factors that can be identified relative to an analytic technique grounded in a linear chain of events accident causation model, but future work exists in showing this empirically.

Conclusion

In conclusion, we showed that a complex accident analysis technique applied to perioperative safety events during the COVID-19 2020 surge resulted in wide-reaching yet concrete lessons for health systems continuing to face ongoing surges, as well as other crises marked by acute staffing shortages, rapidly changing environments, and demand for critical resources. Rigorous nonlinear analyses of safety reports are critical to identify the causes of errors and work to fix system failures. CAST is a tool that can generate a broad range of recommendations for future accident prevention.