Multi-institutional SBRT incident learning: evaluation of safety barriers using a human factors analysis and classification system

Abstract

Objectives:

Stereotactic body radiation therapy (SBRT) can improve therapeutic ratios and patient convenience but delivering higher doses per fraction increases the potential for patient harm. Incident learning systems (ILS) are being increasingly adopted in radiation oncology to analyze reported events. This study used an ILS coupled with a Human Factor Analysis and Classification System (HFACS) and barriers management to investigate the origin and detection of SBRT events, and to elucidate how safeguards can fail allowing errors to propagate through the treatment process.

Methods:

Reported SBRT events were reviewed using an in-house ILS at four institutions over 2014–2019. Each institution used a customized care path describing their SBRT processes including designated safeguards to prevent error propagation. Incidents were assigned a severity score based on the American Association of Physicists in Medicine Task Group Report 275. A HFACS system analyzed failing safeguards.

Results:

160 events were analyzed with 106 near misses (66.2%) and 54 incidents (33.8%). 50 incidents were designated as low-severity, with 4 considered medium-severity. Incidents most often originated in the treatment planning stage (38.1%) and were caught during the pre-treatment review and verification stage (37.5%) and treatment delivery stage (31.2%). HFACS revealed safeguard failures were attributed to human error (95.2%), routine violation (4.2%), and exceptional violation (0.5%), and driven by personnel factors 32.1% of the time, and operator condition also 32.1% of the time.

Conclusion:

Improving communication and documentation, reducing time pressures, distractions, and high workload should guide proposed improvements to safeguards in radiation oncology.

Introduction

Stereotactic body radiation therapy (SBRT) is becoming increasingly popular due to its efficacy^1–4, availability⁵, convenience, and cost-effectiveness⁶. However, the complexity and high dose-per-fraction associated with SBRT can increase the risk of patient harm. Recognizing this risk, many professional societies^1,7–10 and countries^2,3 provide guidelines regarding things like patient selection, staff training, equipment requirements, patient-specific quality assurance (QA), and treatment delivery processes.

Generally, individual departments adapt these guidelines into their own processes. Given that SBRT patients undergo many of the same steps as conventional radiation therapy patients, introducing additional SBRT-specific steps into the existing treatment delivery process offers the simplest means of delivering SBRT clinically. However, the application of prospective risk mitigation strategies offers the opportunity to perform a more comprehensive evaluation of the treatment process. In this way, resources can be preemptively directed to focus on error pathways with the highest risk to the patient.

A common prospective risk mitigation strategy used in radiation oncology is failure mode and effects analysis (FMEA)^11–14. Alternatively, barrier management techniques also aim to prospectively identify specific hazards and identify controls both to prevent the hazard occurring and to mitigate its effect should it occur^15,16. In this context, the concept of a barrier is used to describe a control that an organization relies on to protect against harm. Barriers can come in many forms such as physical, safety checklists, independent software checks, and some combinations of such, among others. Indeed, it is recognized that many of these approaches are imperfect, and thus many systems imbed multiple safeguards, each imperfect, but increases the probability that errors will be detected¹⁷. Specifically, in barrier management particular controls within processes are designated as safeguards or barriers (all conditions met) if they possess the following:

• Have ownership (i.e. an individual or group is responsible for the use and maintenance of the control);

• are traceable (i.e. the control’s existence is able to be tied to a specific policy or procedure within the organization’s management system);

• are auditable (i.e. the control itself can have its performance evaluated);

• are specific (i.e. the control is designed to be sensitive at detecting the threat it is put in place to prevent);

• are independent (i.e. the person/software performing the control is not involved in the preceding steps and is checking items not covered by other controls in the process);

• and are effective at catching the event it is designed for¹⁵.

Barrier management approaches have also been recently introduced into healthcare¹⁶, and radiation oncology specifically¹⁸. We selected barrier management versus FMEA to focus our analysis on the reliability of the controls (safeguards or barriers) included in the SBRT system. As such, instead of focusing on failure modes and how to prevent them, we focused on what controls are present and how reliably they work to prevent failures.

An increasingly common method to monitor a system’s performance is to report incidents to a centralized registry (often termed an incident learning system; ILS), that can then be retrospectively analyzed. This approach has been widely used in radiation oncology at both the institutional and societal level^19–23. Importantly, there is recent work combining both an ILS and FMEA to complement the strengths of both approaches^13,24.

We herein review linear accelerator based SBRT events from four independent institutions. A multi-institutional approach was taken to generate information that may be more translatable to other centers. In this work, we consider events that are both near misses (also known as a good catch, near hit, close call) in which the reported event did not reach the patient; and incidents for ones that do reach the patient. This is consistent with nomenclature proposed by Ford et al²⁵, in the upcoming American Association of Physicists in Medicine (AAPM) Task Group 288 and the American Society for Therapeutic Radiology and Oncology (ASTRO)-AAPM RO-ILS platform. Each event was analyzed to uncover where in the treatment process it originated, where it was caught, and what controls failed. All events were further analyzed using a Human Factor Analysis and Classification System (HFACS) to uncover any patterns in the underlying causes of events. We chose HFACS to analyze contributing factors as it allows for a systems approach to accident investigation. With a HFACS model, human error is viewed as a symptom of a system problem in the organization, not the cause of the accident (e.g., assumes a no-blame culture). As such, organizations develop and implement controls to prevent adverse events at all relevant levels (e.g., organization, supervisory, preconditions of unsafe acts, and unsafe acts). Therefore, these conceptual underpinnings establish a strong link between barrier management and the HFACS model.

The overarching goal of this work was to assess if an ILS system based on barrier management concepts in conjunction with a HFACS can provide insights into the origins, and detection of events in radiation oncology as well as better identify and understand underlying factors that enabled events to occur.

Methods and Materials

A retrospective study was performed involving four institutions with active SBRT programs each using their own ILS participated in this study. Since 2013, Institution 1 and Institution 2 have been using the ImprovementFlow^™ (Communify Health LLC, Greenwich, CT) ILS to drive quality improvement efforts for all treatment modalities²⁶, analyzing over 3200 user-submitted events. Institution 3 uses an in-house system for initial reporting of incidents which was modified from that of Mutic et al²⁷ and RO-ILS for event analysis. Institution 4 utilizes a customized radiation oncology-specific ILS that was implemented in 2016 in the hospital-level ILS software (RL Solutions, Toronto, ON) with a data field structure patterned after the national RO-ILS database, but customized to capture the nomenclature, workflow, and process map of the institution.

Each institution recreated their individual SBRT treatment process, known as a care path, as a series of steps (e.g. patient registration, initial consult, through to treatment completion and follow up) within the ImprovementFlow™ platform. Certain collections of steps were grouped into higher-level categories such as patient selection and assessment, imaging, treatment planning, treatment delivery and on-treatment management (medical and chart reviews occurring after the patient has started treatment) to account for differences between institution care paths and simplify event staging. The categories are those detailed in the AAPM’s consensus recommendations for incident learning in radiation oncology²⁵. The period of analyzed events differed for all institutions and were as follows: January 2016 to January 2019 (Institution 1), August 2014 to April 2019 (Institution 2); January 2016 to March 2019 (Institution 3); April 2017 to January 2019 (Institution 4).

No controls in the institutions’ care paths met the formal criteria of a barrier. However, several met 5 of the 6 criteria such as the fully automatic data transfer between the linear accelerator and treatment management system and audits of machine quality assurance records performed by radiation safety officials. For this study, if an institution deemed a control important and it also met at least 3 of the 6 barrier criteria, it was designated as a safeguard. Examples of safeguards meeting this standard included documentation of the patient prescription, physics pre-treatment check, and image review of patient position prior to initiating radiation. Figure 1 shows one institution’s care path with the treatment planning stage specifically expanded in detail.

Figure 1.

Example care path, with the Treatment Planning Section expanded to highlight all specific steps and safeguards within the care path stage. CMD = certified medical dosimetrist, PHY = medical physicist, MD = attending/resident physician, RTT = radiation therapist, RN = registered nurse, Admin = administration specialist, MQ = Elekta MOSAIQ® treatment management system, QCL = quality checklist (a Mosaiq specific list of tasks assigned to certain working groups), DOS = dosimetry working group within Mosaiq, TPS = treatment planning system.

Every SBRT-related staff-reported event (e.g. suboptimal patient setup, equipment failure/limitations, communication challenges leading to delays/deviations) was analyzed within each institution’s care path in two stages. The first stage identified the step in the institution care path the event originated, at which step it was caught, and how often a relevant safeguard designed to catch the event was crossed. All events considered as incidents were assigned a severity score based on the 1–3 initial scoring scale (representing low-, medium-, or high-severity) used as part of the American Association of Physicists in Medicine Task Group Report 275²⁸ and by one of the collaborating institutions²⁴. Other scales also exist, such as the French Nuclear Safety Authority. We chose our scale based on its foundation in TG-100 and FMEA. The second stage, used a HFACS^29,30 embedded in the ImprovementFlow™ platform to assign an unsafe act and one or more contributing factors to events that crossed relevant safeguards. Unsafe acts are classified as either human error or violations. Human errors are unintentional mistakes and can be skill-based, decision-based, or perceptual-based. Skill-based errors are mistakes made when executing routine, highly practiced tasks relating to a procedure such as failing to mark an item on a checklist as complete. Decision-based errors occur when the process proceeds as intended yet the chosen plan proves inadequate to achieve the desired end-state and results in an unsafe situation. For example, a planner moving the treatment isocenter to achieve better beam geometry but increasing the risk of collision between the patient and treatment machine. Perceptual-based errors occur when the operator bases their actions on incorrect sensory information, such as looking at a color scheme for dose visualization that differs from the standard leading to a mistake regarding the dose to the target or organs at risk. Violations differ from human-errors as they are intentional decisions made by operators and are separated into routine and exceptional violations. Routine violations refer to automatic and sometimes-unconscious behaviors, or other workarounds, that are often tolerated by the organization. For example, therapy staff pre-loading imaging fields before a patient enters the room to save time. Such behavior is tolerated but may lead to re-work or imaging the incorrect patient if a different patient is brought into the room due to coordination issues between team members. Exceptional violations are for rare intentional deviations from rules, for example, intentionally disabling or over-riding an alarm or machine inhibit to treat a patient.

The contributing factors assigned to each unsafe act could include: a) preconditions such as the condition of the operator (inattention/distraction, fatigue, workload); personnel factors (incomplete/missing/incorrect documentation, communication between staff groups, inexperience or training); environmental factors (noise/lighting/technological interfaces); b) unsafe supervision (lack of management support/failure to correct known issues/willful disregard for rules), and c) organizational influences (resource management, processes/policies, or staff hierarchy). Unsafe supervision could include failing to train personnel on new equipment or procedures. Organizational influences could include a scheduling too many patients into a day, forcing overtime or causing staff to rush, or failing to budget for appropriate equipment needed to treat certain treatment sites or using the best treatment technique for a certain patient(s).

Finally, safeguards that were common across all institutions were further investigated to uncover which failed the most often, and what specific HFACS factors were assigned to these safeguard failures to provide input to maximize the impact of future quality improvement initiatives.

Results

163 events were reported across the four institutions for the period covering August 2014 – April 2019. At one institution, 3 events originated outside their care path and were excluded from the analysis. The analysis of 160 reported events revealed 106 near-misses (66.2%) and 54 incidents (33.8%). Fifty incidents were considered low-severity and 4 were considered medium-severity (2.5%). Low-severity incidents included lack of equipment availability, setup shifts not applied prior to imaging, collisions with immobilization devices, and scheduling issues causing patient delays. Details for the four medium-severity incidents included: (1) Treatment on consecutive days vs every-other-day as prescribed; (2) extreme weight gain between planning and treatment with one fraction of their SBRT course delivered before re-planning; (3) an incorrect planning target volume (PTV) margin used for 1 out of 5 liver SBRT fractions; (4) a patient positioned with sub-optimal alignment for 1 of 5 multilevel spine SBRT treatments.

Figure 2 shows the flow of events that originate, travel through, and are caught across the 9 care path stages. It shows most events originate at the treatment planning stage (38%), and includes things like incorrect prescription specification, incorrect scheduling of patients in the treatment management system, and incorrect documentation of patient shifts. Figure 2 also shows that the most events are caught at the pre-treatment verification (38%) stage and treatment delivery stage (31%).

Figure 2.

Pipe diagram indicating the origin and path of the 160 pooled events from all institutions across all stages of the collaborative care path.

Using the numbers in Figure 2, the cumulative effectiveness of safeguards at each care path stage was estimated. For example, for the pre-treatment review and verification stage, there were 97 events flowing into the stage, with another 9 generated at the same stage for a total of 106 “active” events. 60 of these events were caught for a detection rate of 60/106 = 57%. Similarly, for the treatment delivery stage, there are a total of 61 active events and 50 caught for a detection rate of 82%. Note, for this analysis we only consider a safeguard to have failed if it was designed to catch the specific event being analyzed. Therefore, the detection rates calculated above should be a close approximation to their effectiveness but ultimately is related to all safeguards in that stage, rather than any specific safeguard.

Across the 160 reported events, there were 189 instances where a safeguard expected to prevent an event occurring instead allowed that event to pass through. The number of failed safeguards is greater than the reported incidents as some events passed through multiple safeguards. Table 1 shows most failures were attributable (at least in part) to human error. Reassuringly, there were very few routine violations (8 events, 4.2%) or exceptional violations (1 event, 0.5%).

Table 1.

Reported numbers for unsafe acts leading to a relevant safeguard failure pooled across all four institutions

Actions:	Number (%)
Human error	180 (95.2%)
Routine violations	8 (4.2%)
Exceptional violations	1 (0.5%)

For the unsafe acts listed in Table 1, the underlying contributing factors assigned to each safeguard failure are shown in Figure 3. Nearly 80% of contributing factors fall into the preconditions category that broadly covers personnel factors, conditions of the operator, and environmental factors. Secondary to this are organizational issues, with inadequate supervision being a minor contributing factor.

Figure 3.

Breakdown of the contributing factors of the unsafe acts of Table 1. With most (95.2%) unsafe acts being human error, the categories in the chart therefore mostly represent the contributing factors that increase the chances of a human error occurring.

The specific safeguards able to be identified as common across the four institutions and which failed the most often by volume were the physics pre-treatment check (n=29 failures) and therapy pre-treatment check (n=21). Table 2 shows the results of a subset analysis of the contributing factors leading to events passing through these two safeguards. The two leading preconditions for unsafe acts related to these two safeguards were condition of the operator and personnel factors.

Table 2.

A subset of contributing factors for the unsafe acts specific to when the physics and therapy pre-treatment check safeguards failed. These safeguards were chosen as they are common to all institutions and had the highest number of failures. Note, multiple HFACS categories may be given to a single safeguard failure. No supervisory contributing factors were observed for these two safeguards

	Specific safeguard
HFACS categories assigned to safeguard failures	Physics pre-treatment check (29 safeguard failures)	Therapy pre-treatment check (21 safeguard failures)
Preconditions for Unsafe Acts
Condition of the operator	20	16
Personnel factors	17	14
Environmental	4	2
Organizational Influences
Operational process	11	6

Discussion

The vast majority of radiation therapy treatments are performed with very low rates of errors^31–36. However, practice changes towards using fewer, higher-dose fractions increase the risk for patient harm when an incident occurs. The complexity of newer technologies, including those enabling SBRT, has been well recognized in our field for decades, including the Thera-25 accidents occurring in 1985–87³⁷ and more recently the medical events detailed in a series of New York Times reports^38,39. The continuing evolution of technologies in our industry suggests that the risk of future events due to complex interactions will likely persist^40,41.

There are three main findings of this study. First, we identified where events start and where they are caught. Our collective data show that 38% of events were generated in the treatment planning phase. This is consistent with one review of SBRT-specific RO-ILS data where 36.6% of reported events were related to treatment planning⁴², most commonly involving incorrect shifts/alignment of the patient, collision/clearance issues, incorrect sites, or incorrect contours. Many of the events analyzed in our study were of a similar nature. Our study is also consistent with the RO-ILS 3^rd quarter 2018 report where 30% of aggregated events were generated in the treatment planning phase⁴³. Also in our study, most events were detected during pre-treatment review and verification stage (38%) and the treatment delivery stage (31%). These are again in line with the 2018 RO-ILS aggregate data of 23% and 30%, respectively⁴³.

Our second main finding in this study was identifying factors that contribute to safeguard failures. Our analysis showed that 33.8% of reported events reached the patient, and thus were deemed incidents. However, this seemingly high rate does not necessarily mean the chosen safeguards are ineffective. In fact this 33.8% consists mostly of very minor incidents resulting in small delays or repeated work rather than any patient harm. It is more likely the 2.5% of medium-severity incidents are similar to incidents used to calculate rates previously reported in the literature^32,34–36. The higher 33.8% incident rate may also be influenced by the reporting practices of each institution. We found 95% of the time, human error was the unsafe act attributed to a safeguard failure and the most common contributing factors were condition of the operator, such as inattention or distraction, workload, and time-pressures; and personnel factors such as documentation and communication issues. This finding is similar to that reported by others^34,36. Previous work has also identified hazards associated with handoffs, quality assurance, and workflows⁴⁴. The nature of the observed events, and the similarity in origin and detection stages suggest that safeguards for SBRT treatments are defeated in similar ways as conventional radiation therapy.

Importantly, the HFACS component of this work did reveal opportunities for improvement. Given most events are generated at the treatment planning phase, which involves communication between multiple sources, and intense human-computer interactions, improvement efforts may be best directed at improving communication pathways, and human-computer interfaces. To this end, some of the participating institutions have initiated efforts to improve the clarity and availability of policy and procedure documents to improve organizational effectiveness in communication and coordination between staff groups and to standardize treatment planning templates. Further efforts in improving checklists used in the pre-treatment review and verification stages, aiming to reduce cognitive workload and alleviate inattention, distraction, and complacency have also been undertaken, based on prior work in aviation⁴⁵, healthcare^46,47, and radiation therapy^28,48. Maintaining a work environment free of distractions (the “sterile cockpit” concept) is also of utmost importance²⁸.

Thirdly, no control in any institutions’ care paths met all formal criteria of a barrier and only a few controls met 5 of the 6 criteria. This finding presents an opportunity to improve both conventional and SBRT treatment modalities by improving shared safeguards. For example, finding methods to audit existing controls would upgrade many into barriers. More comprehensively, the need for introducing or improving a control may be informed prospectively with an FMEA analysis to identify events with high severity, high occurrence, or low detectability, with the control then designed using barrier management principles. Importantly, there may be situations where there is no practical method of upgrading specific controls to barriers. Therefore, devoting resources to, and maintaining an underlying culture of continuous quality improvement remains necessary to improve existing controls. Root cause analysis of reported incidents using a barrier management philosophy, and associated tools such as Bowtie analysis may offer opportunities in these situations.

This study has several limitations. First, it is nearly certain that not all events were reported nor completely characterized or understood. Prior studies have estimated reporting rates as low as 10%⁴⁹ and the reporting of these events can be incomplete. Given the emphasis that our institutions have placed on ILS, we are confident that our reporting rate is >10%. Further, the retrospective nature of the analysis introduces difficulties in identifying the exact event origin and associated HFACS factors. Second, computing the safeguard effectiveness is complicated by two main factors. One, incidents submitted in the early years are being evaluated against the latest version of the institution’s care path. In other words, specific safeguards may have been implemented in response to certain prior events, which results in the safeguard appearing less effective. This is less of a concern as most participating institutions had mature care paths but cannot be avoided. The other factor complicating the calculation of safeguard effectiveness is that a safeguard detection rate is based on known, reported events. A more complete picture would also compute the detection rate using the total number of times each safeguard was tested. Unfortunately, this information was unavailable from all institutions. Prospectively designing, implementing, and evaluating controls may provide better estimates of safeguard effectiveness but is beyond the scope of this work. Third, classification of incidents as low-, medium-, and high-severity is imperfect. There are many classification/categorization methods, and certainly some of these alternative approaches may increase the reported severity of the incidents. This would not have altered our overall results as we have grouped all incidents together for the analysis. A formal analysis limited to only the most-concerning incidents might be of interest, but the number of such events is fortunately modest. Given the high risk of potential harm with high doses per fraction, it seemed most reasonable to group all incidents together since it is at least theoretically possible that any of them, left uncorrected, could have led to a clinically meaningful error. Fourth, the number of events analyzed is relatively modest. Nevertheless, this multi-institutional study represents the largest of its kind to date focusing on SBRT. Thus, while recognizing the above issues, we believe that the results are both useful, and may be broadly generalizable to other centers.

Conclusions

Across four institutions, events occur most commonly during the treatment planning stages, and most are caught at either the pre-treatment review and verification or treatment delivery stages. Most errors were attributable to human error with the top three contributing factors being personnel factors involving communication and documentation, staff conditions including inattention, workload and time pressures, and suboptimal organizational processes such as unclear policy and procedures.