Mitigation of Intensity Modulated Radiation Therapy and Stereotactic Body Radiation Therapy Treatment Planning Errors on the Novel RefleXion X1 System Using Failure Mode and Effect Analysis Within Six Sigma Framework

Abstract

Purpose

The aim of this study was to apply the Six Sigma methodology and failure mode and effect analysis (FMEA) to mitigate errors in intensity modulated radiation therapy (IMRT) and stereotactic body radiation therapy (SBRT) treatment planning with the first clinical installation of RefleXion X1.

Methods and Materials

The Six Sigma approach consisted of 5 phases: define, measure, analyze, improve, and control. The define, measure, and analyze phases consisted of process mapping and an FMEA of IMRT and SBRT treatment planning on the X1. The multidisciplinary team outlined the workflow process and identified and ranked the failure modes associated with the plan check items using the American Association of Physicists in Medicine Task Group 100 recommendations. Items with the highest average risk priority numbers (RPNs) and severity ≥7 were prioritized for automation using the Eclipse Scripting Application Programming Interface (ESAPI). The “improve” phase consisted of developing ESAPI scripts before the clinical launch of X1 to improve efficiency and safety. In the “control” phase, the FMEA ranking was re-evaluated 1 year after clinical launch.

Results

Overall, 100 plan check items were identified in which the RPN values ranged from 10.2 to 429.0. Fifty of these items (50%) were suitable for automation within ESAPI. Of the 10 highest-risk items, 8 were suitable for automation. Based on the results of the FMEA, 2 scripts were developed: Planning Assistant, used by the planner during preparation for planning, and Automated Plan Check, used by the planner and the plan checker during plan preparation for treatment. After 12 months of clinical use of the X1 and developed scripts, only 3 errors were reported. The average prescript RPN was 138.0, compared with the average postscript RPN of 47.8 (P < .05), signifying a safer process.

Conclusions

Implementing new technology in the clinic can be an error-prone process in which the likelihood of errors increases with increasing pressure to implement the technology quickly. To limit errors in clinical implementation of the novel RefleXion X1 system, the Six Sigma method was used to identify failure modes, establish quality control checks, and re-evaluate these checks 1 year after clinical implementation.

Introduction

Biology-guided radiation therapy (BgRT) has the potential to revolutionize current radiation therapy practice, specifically in the treatment of oligometastatic disease.^1,2 Unlike conventional image guided radiation therapy, which uses orthogonal or cone beam images for target and patient alignment, BgRT uses the tumor's physiological signature as a homing beacon for targeting.^1,3, 4, 5, 6, 7 F-18–labeled fluorodeoxyglucose (FDG) is the diagnostic standard for tumor staging owing to its ability to measure metabolic activity. When F-18 FDG is combined with near real-time positron emission tomography (PET) imaging, this allows for potential real-time tracking and targeting of the tumor without the need of invasive surrogates such as implanted fiducials. In addition to F-18 FDG, there is an increasing number of novel, tumor-specific radiotracers that have a potential to increase the therapeutic ratio when treating with BgRT.8, 9, 10

The novel RefleXion X1 (RefleXion Medical Inc, Hayward, CA) is the first commercial realization of BgRT. The X1 system consists of a fast-rotating ring gantry design containing a 6-MV flattening-filter-free linear accelerator, on-board fan-beam kilovoltage computed tomography (kVCT), and on-board dual 90° detectors for PET imaging.^3,11, 12, 13, 14 Although the X1 system provides the tools (ie, hardware and software) for BgRT, the BgRT capability is still under investigation and not approved by the US Food and Drug Administration (FDA) for clinical use. However, the X1 system has been cleared by the FDA for intensity modulated radiation therapy (IMRT) and stereotactic body radiation therapy (SBRT) treatments. The first clinical installation of the X1 system was performed at our institution in early 2021 and has been used for IMRT and SBRT treatments since May 2021. Acceptance testing, commissioning of the X1 and its treatment planning system (TPS), and treatment planning studies with the X1 have been published previously by our group.^4,11, 12, 13^,15 Much of the previous literature on the X1 has focused on physics or dosimetry-specific aspects of the system, including underlying physics principles, beam or treatment planning system commissioning, treatment planning investigations, or dosimetry studies, rather than clinical workflow and/or integration of the X1 into current practice.^3,5, 6, 7^,11, 12, 13^,16

As with any new technology, treatment modality, or procedure in the clinical environment, there exists the opportunity for errors, which could directly or indirectly affect patient care. The potential or likelihood for these errors to occur increases with increasing pressure to implement the technology quickly and with decreasing resources to properly establish workflows and quality assurance (QA) checkpoints during the treatment process.¹⁴ To aid in establishing workflows and minimizing potential errors for IMRT and SBRT treatments on the RefleXion X1 system, we applied a Six Sigma method that provides a structured framework to measure and reduce defects in the process and has been successfully used in other radiation oncology settings.17, 18, 19, 20 Failure mode and effect analysis (FMEA) is another tool that can be used to establish a general workflow for any process, prospectively identify possible failure modes, and rank identified failure modes in terms of a risk priority number (RPN).²¹ The results of such an analysis can be used to revise the workflow or introduce QA checkpoints to reduce the RPN of the identified failure modes. In this study, an FMEA analysis was performed for the IMRT and SBRT workflows as part of the Six Sigma framework to mitigate the errors stemming from treatment planning before clinical launch of the X1 system. The results of the FMEA analysis were used to guide the implementation of QA checkpoints into the workflow, using, where applicable, scripting within the Eclipse Scripting Application Programming Interface (ESAPI; Varian Medical Systems, Palo Alto, CA).

Methods and Materials

As part of this improvement project, we followed the Six Sigma approach using 5 phases—define, measure, analyze, improve, and control (DMAIC):

• • •
    Define: Set the goal of reducing the reported treatment-planning incidents
  • •
    Measure: Perform FMEA to examine the process map, define the possible failure modes, and rank them according to risk priority
  • •
    Analyze: Identify items eligible for automation and prioritize them using Pareto-sorted failure modes
  • •
    Improve: Develop ESAPI tools to mitigate the errors and streamline the workflow
  • •
    Control: Create a feedback loop from users to further improve the ESAPI scripts and evaluate failure modes 1 year after launch of ESAPI tools

Define, measure, and analyze phases: Process map and FMEA analysis before clinical launch

Before clinical launch of the RefleXion X1 system, the process map detailing the steps in the RefleXion workflow process was outlined and discussed during a series of multidisciplinary meetings including radiation oncologists, physicists, dosimetrists, therapists, administration, and information technology personnel. The RefleXion X1 IMRT and SBRT workflow is depicted in Fig. 1. Fusion and contouring are not available in the X1 TPS; hence, these tasks are performed in Eclipse, version 15.6 (Varian Medical Systems), and MIM, version 7.1.2 (MIM Software Inc, Cleveland, OH), at our institution. Owing to the lack of plan comparison or plan summation tools, the treatment plans are reviewed both in Eclipse and the X1 TPS. Scheduling, billing, and image guided radiation therapy offline review are performed in the Aria Record and Verify System, version 15.6 (Varian Medical Systems). The prescription that is set in Aria is regarded as a physician's written directive in this process.

Figure 1.

The RefleXion X1 image guided radiation therapy (IGRT) and stereotactic body radiation therapy treatment workflow. Currently, fusion and contouring are not available in the X1 treatment planning system (TPS); hence, these tasks are performed in Eclipse and MIM at our institution. The plans are reviewed both in Eclipse and in the X1 TPS, and scheduling, billing, and IGRT offline review are performed in Aria.

An itemized list of the individual treatment plan check steps was compiled using the American Association of Physicists in Medicine (AAPM) Task Group 275 checklist,²² departmental procedures, and items directly inspired by errors and near misses reported to a web-based departmental incident learning system (ILS). A multidisciplinary quality-improvement team composed of radiation oncologists, physicists, dosimetrists, and therapists ranked the failure modes associated with these identified plan check items using AAPM Task Group 100 recommendations.²¹ The team ranked the following: severity of the effect on a patient's radiation therapy if the error is not caught (S) and detectability dormancy (D), or the probability of the error going undetected. Occurrence (O) was determined using the records from the departmental ILS based on our Eclipse planning experience and on RefleXion preclinical evaluation. The RPN was calculated for each plan check item using the FMEA formalism (Equation 1).

$$RPN=Severity\left(S\right)\times Detectabilitydormancy\left(D\right)\times Occurrence\left(O\right)$$ (1)

Improve phase: Eclipse API scripting for failure mode mitigation

Implementation of QA or quality control (QC) checks is a natural response to identified failure modes in a workflow or process.²¹ The aim of such QA checks is to reduce the RPN of the failure modes by decreasing the likelihood of occurrence or decreasing the probability of the error going undetected. Implementation of policies or procedures incorporating QA checks is an effective tool for mitigating these errors, and such checks are well suited for automation because these QA items involve checking similar parameters for each patient.²³ Several investigators have demonstrated success in improving safety and efficiency by automating QC checks using treatment planning system API scripting as part of the physics initial check process.^20,23, 24, 25, 26, 27, 28, 29 Therefore, the identified failure modes from the FMEA analysis in this work were Pareto-sorted in order of decreasing RPN score to determine the highest-priority items to be addressed first. These Pareto-sorted items and items with severity ≥7 were then evaluated for eligibility for either full or partial automation using API scripts. For this work, the ESAPI was used for development of 2 scripts used at 2 different time points in the treatment planning process: the Planning Assistant (PA) tool, used by the planner during preparation for planning, and the Automated Plan Check (APC) tool, used by the planner and the plan checker during plan preparation for treatment. These are described in detail in the following subsections.

Planning Assistant tool

The PA tool was developed to streamline the process of computed tomography (CT), structure set preparation for RefleXion treatment planning, and mitigate the errors arising in the beginning of the treatment planning process. During commissioning,¹² several failure modes were identified that could result in treatment errors or in lost time correcting the issues. The amount of time lost depended on the error and ranged from minor inconvenience (ie, <10 minutes) for missing or required structures in the X1 TPS for planning to major failure (ie, >2 hours) if, for example, a treatment isocenter is selected in the TPS that is not physically achievable in the X1 system, thus requiring replanning. The included QC checks before exporting the CT images and structure set to the X1 TPS for planning included cropping all structures protruding from the body, removing all empty structures from the structure set, inserting an artificial couch structure to aid the X1 TPS in identifying the correct location on the image where the RefleXion couch structure should be placed (performed automatically by the X1 TPS), check the proposed treatment isocenter position against the X1 TPS and measured collision zones for the X1 system, and so on. To facilitate wider adoption of the X1 system, the PA tool has been made open source and is freely available via GitHub (link: https://github.com/esimiele/PlanningAssistant). Although the PA tool was developed and refined during commissioning of the X1 TPS, it was thoroughly tested for presence of false positives and false negatives before clinical implementation.

Automated Plan Check tool

The APC tool was adapted from the previously developed ESAPI script that focused on mitigating failures identified using the Six Sigma DMAIC method combined with an FMEA analysis of our institution's planning and treatment practice.²⁰ Liu et al²⁰ found the developed APC to significantly reduce planning errors before they reached patient treatment while simultaneously improving the efficiency of physics plan checking. Given the positive results of our previous work, this tool is still in use at our institution today and is routinely updated based on user feedback and newly identified failure modes (eg, workflow changes). The present study built on the success of the APC tool and incorporated checks for identified failure modes from the FMEA analysis of the X1 planning workflow. The APC tool was modified and refined during the commissioning process of the X1 and was thoroughly tested for false positives and false negatives before clinical implementation.

Control phase: Re-evaluation of FMEA 1 year after clinical launch

One year after clinical launch and introduction of the PA and APC tools, all 100 plan check items were re-evaluated to update the FMEA O (occurrence) and D (detectability dormancy) values. Reported treatment planning errors at the time of physics plan check, therapy plan check, and treatment were used to determine O. Reported treatment planning errors at the time of physics plan check and therapy plan check were normalized to the total number of plans completed in the time frame. The Six Sigma defect rate per opportunity (DPO) was calculated using

$$\begin{array}{c}\text{DPO}=\text{Errors}\hfill \\ (\text{Opportunities}\text{for}\text{errors}\text{in}\mathrm{a}\text{plan}\times \text{Number}\text{of}\text{plans})\hfill \end{array}$$ (2)

where the opportunities for errors in plans were estimated as an average number of plan elements checked during the physics plan check of 5 representative cases. This DPO value was compared with the Six Sigma goal of 3.4 × 10⁻⁶, which was determined to be both acceptable and achievable.

Results

Process map and FMEA analysis

Overall, 100 physics plan check elements were identified. The list of plan check elements Pareto-sorted by RPN is shown in blue in Fig. 2. The RPN values ranged from 10.2 to 429.0. Fifty elements out of 100 (50%) were assigned as potentially suitable for either full or partial automation within the ESAPI environment. Among the 10 highest-risk items, 8 were suitable for automation, including treatment isocenter within the collision-safe zone (RPN = 429.0), isocenter-shift instructions provided to therapists (RPN = 357.1), correct RefleXion plan imported to Eclipse (RPN = 314.2), cumulative items checking Aria prescription and RefleXion plan match (dose per fraction, number of fractions, total dose, and frequency; RPN range, 248.0-279.7), and accounting for cardiac device (RPN = 288.0). The plan check items with high-severity scores (≥7) are shown in Table 1.

Figure 2.

Pareto-sorted list of failure modes of all plan check elements ranked by risk priority number (RPN) value. The RPN values evaluated without and with the assistance of the Application Programming Interface (API) scripts are shown in blue and red, respectively, for the identified failure modes. The RPN values that did not change with the assistance of the API scripts indicate these items were not suitable for automation or were not accessible from the Eclipse API.

Table 1.

Plan check elements with the highest severity (S ≥ 7)

Plan check element	Automated	Severity
Contouring: target(s)	-	8.8
Patient assessment: cardiac device is taken into consideration	Mostly	8.0
Patient assessment: previous RT is taken into consideration	-	8.0
Rx in Aria vs Rfx Tx plan: dose/fraction	Fully	7.6
Rx in Aria vs Rfx Tx plan: total dose	Fully	7.6
Rx in Aria vs Rfx Tx plan: number of fractions	Fully	7.6
CT SIM: consistency between orientation of CT scan and Tx plan	Fully	7.6
Rfx plan optimization and calculation: number of fractions	Fully	7.4
Isocenter shifts: treatment isocenter is within the collision-safe zone	Fully	7.3
Contouring: organs at risk	-	7.0

Abbreviations: CT = computed tomography; CT SIM = computed tomography simulation; Rfx = RefleXion X1; RT = radiation therapy; Rx = prescription; Tx = treatment.

Eclipse API scripting for error mitigation

Planning Assistant

The graphical user interface (GUI) for the PA script is shown in Fig. 3A, where users could prepare a structure set for planning in the X1 TPS by simply following the buttons on the Plan Preparation tab from top to bottom. During the X1 TPS commissioning process, it was discovered that the X1 TPS prevented optimization and dose calculation if structures extended beyond the body contour or overlap with the treatment couch. However, the X1 TPS did not identify the offending structure. This caused a tedious and time-wasting effort to identify which structure was the problem in Eclipse, fix it, export and import the new structure, and adjust the plan setup. This failure mode with high occurrence (O = 8; RPN = 199.8) is the first item checked during plan preparation using the PA to avoid lost time during planning. The greatest time saved using the PA tool was in selection of a treatment isocenter inside the collision-safe zone, which was identified as the failure mode with the highest RPN and high severity (S = 7.3; RPN = 429.0).

Figure 3.

(A) The Planning Assistant user interface. The various functions included in the Planning Assistant script are grouped by category on separate tabs (plan preparation, X1 treatment planning system [TPS] commissioning, etc). (B) The collision safety zones (x = lateral distance from the center of the couch to the treatment isocenter; z = vertical distance from the top of the couch to the treatment isocenter) for the X1 TPS machine in International Electrotechnical Commission (IEC) coordinates. “Measured” indicates the collision zone was measured on the X1 TPS machine during commissioning, “hard-coded” indicates the collision zone was used in the X1 TPS, and “processed” indicates the intersection (with additional margin) of the measured and hard-coded collision zones was used in the Planning Assistant script to evaluate the proposed treatment isocenter. (C) If the selected treatment isocenter is outside the collision safety zone, the Planning Assistant script will contour the safety zone (shown in green colorwash) on the patient image so the planner can immediately see the allowed isocenter positions.

In the X1 TPS, the treatment isocenter can be assigned to the geometric center of a structure or at the viewing plan intersection. However, at the time of this writing, the isocenter itself cannot be visualized in the X1 TPS, even after assignment. To enable unambiguous isocenter assignment that can be visualized in the X1 TPS and Eclipse, a marker structure (of Digital Imaging and Communications in Medicine (DICOM)-type marker) is added by the planner in Eclipse at the proposed isocenter location before exporting the structure set to the X1 TPS. The treatment isocenter is then assigned to this marker structure (ie, a point that can be visualized). The X1 TPS has a hard-coded safety zone (ie, allowable positions for the treatment isocenter); optimization is not permitted if the placed isocenter is outside this safety zone. During commissioning of the X1 machine,^11,12 this collision zone was also measured on the machine, and it was observed that the TPS and measured collision zones disagreed. The TPS collision zone was generally more conservative than the measured collision zone for a large range of couch positions (Fig. 3B). However, the TPS collision zone allowed isocenter placement in regions that were not physically achievable on the machine (ie, a collision would occur). This represents a significant failure mode because this error would not be caught until patient setup, which would cause significant delays to the patient treatment to fix the problem. To eliminate this failure mode, the proposed treatment isocenter location is verified by the PA script against the intersection of these 2 collision zones (measured and hard-coded), which would immediately notify the planner if a new isocenter location would need to be selected (see “Processed” curve in Fig. 3B-C).

Automated Plan Check tool

Because final plan evaluation was performed in Eclipse, several checks needed to be executed for high-severity items, such as that the exported and imported dose distribution was the intended dose distribution (RPN = 314.2) and the prescription entered by the planner in the X1 TPS matched the prescription entered into Aria (RPN range, 272.3-279.7). An example of the physics second check report produced from the modified APC is shown in Fig. 4A. Similar to Liu et al,²⁰ the APC verifies several basic planning parameters such as prescription approval, planned dose per fraction, and that the number of fractions matches the prescription. However, only the dose distribution, not the plan and dose, can be exported from the X1 TPS as a DICOM object to be imported into Eclipse for evaluation. This limitation introduces a severe failure mode: the wrong dose distribution could be exported from the X1 TPS (eg, from an alternate plan) and used for evaluation, which could guide clinical decision making for that patient.

Figure 4.

(A) Physics second check report printed from the modified automated plan check (APC) for an example RefleXion X1 treatment plan. The first half of the check items focus on the plan parameters entered in Eclipse (prescription approved, plan matches prescription, etc). The second half of the check items focus on comparing the results printed in the X1 treatment plan report (generated by the X1 treatment planning system [TPS]) with the parameters entered into Eclipse (eg, the plan name in X1 TPS matches the plan name in Eclipse). Furthermore, to ensure the correct dose distribution was imported into Eclipse, the dose-volume histogram (DVH) metric results in the PDF report are parsed and compared with the same DVH metrics generated in Eclipse (extracted using the APC). (B) Plan quality metrics report generated by the APC for an example RefleXion X1 plan. The APC can automatically determine the plan type and site, then extract relevant DVH information and compare it with our institutional guidelines and recommended constraints from reports of the Radiation Therapy Oncology Group, American Association of Physicists in Medicine, and so on.

Because limited information could be extracted from the DICOM dose file once imported, additional checks were added to the APC to verify that the physician's intent was carried out during planning. Most of this information is contained within the PDF report generated by the X1 TPS for each plan. Therefore, the APC was modified to read the PDF plan report upon launch for the patient of interest and parse relevant data to verify several planning parameters, including but not limited to the following: a plan name in the X1 TPS that matched the plan name in Eclipse, a prescription in the X1 TPS that matched the prescription in Eclipse, agreement between the dose-volume histogram (DVH) doses reported by the X1 TPS and the DVH doses reported by Eclipse, an isocenter selected for treatment that was within the collision zone and agreed with the marker originally placed in Eclipse, and confirmation that all approvals in the plan report generated by the X1 TPS were performed by the appropriate parties. To ensure the correct dose distribution was imported into Eclipse, the APC parses DVH metrics for multiple structures from the report and compares them with the DVH metrics calculated in Eclipse (extracted using the APC) for the imported dose distribution. If the difference between the report DVH metrics and the APC-calculated DVH metrics exceeds a predefined tolerance, the check fails and advises the user to verify that the correct dose distribution is being used for analysis.

An example of the plan quality metrics report produced from the APC is shown in Fig. 4B. For each plan, the APC determines the type (eg, IMRT, SBRT, total body irradiation (TBI), etc) and site, then extracts relevant DVH information and compares it with our institutional guidelines and with recommended DVH metrics from relevant reports from the Radiation Therapy Oncology Group, AAPM, NRG Oncology, and so on. If a DVH metric is outside tolerance, that metric is reported as a failure and flags the user to inspect the plan for accuracy to ensure an error was not made during planning.

Impact of API scripts on RPN scores

After 12 months of clinical use of the RefleXion X1 and 12 months after script implementation, the occurrence and detection dormancy scores were re-evaluated. Occurrence values at 12 months were based on actual errors occurring during this period. After treating 80 patients and 1747 fractions, only 3 errors were reported during this period: omitting bolus placement for treatment, incorrect laterality specified in the RefleXion plan prescription name but the correct site planned and treated (nomenclature error), and an Eclipse tracking plan that was not approved after the physics plan check.

The average difference between postscript and prescript RPN values was –90.2 (range, –407.0 to 0). The average RPN prescript value was 138.0 compared with the average postscript RPN value of 47.8 (P < .05). Among the tests with the largest decrease in RPN were treatment isocenter in a collision-free zone (RPN difference, –407.0); isocenter-shift instructions (RPN difference, –350.7); special consideration for RT, such as a cardiac device (RPN difference, –272.0); a prescription in Aria that matched the RefleXion Plan in dose per fraction, total dose, and number of fractions (RPN difference, –264.9 to –272.1); and a correct RefleXion plan dose imported into Eclipse (RPN difference, –253.0) (Fig. 2). The estimated DPO value of 1.01 × 10⁻⁴ did not achieve the 6σ level but achieved the 5.2σ level process with 99.99 yield for a σ shift of 1.5.

Discussion

Using the FMEA framework, numerous failure modes were identified in the treatment planning process for the novel RefleXion X1 system. These failure modes ranged from inconvenience to serious error as determined from the RPN ranking from a multidisciplinary radiation oncology team (Fig. 2). Previous studies have also used the FMEA formalism to evaluate the associated risk when implementing new/experimental technologies30, 31, 32 or when reviewing current clinical practice.^20,33,34 Furthermore, many of these studies implemented some form of automation into the QC process as part of the FMEA analysis to improve efficiency and accuracy over manual checks.^20,30,33,34

In this study, 100 physics plan check elements were identified as part of the FMEA, half of which were eligible for either full or partial automation (Fig. 2). Two scripts were written to address the high-RPN and high-severity items from this analysis: the PA tool, which increases the efficiency and safety of the preparation process for treatment planning with the X1 TPS, and the Automatic Plan Check tool, which enhances the efficiency and safety of the plan check process. The average prescript RPN was 138.0 compared with the average postscript RPN of 47.8 (P < .05), indicating a safer practice. Liu et al²⁰ reported a decrease in the FMEA RPN ranking from 129.2 to 83.7 at 9 months after introduction of the APC script, which resulted in a decrease in the reported treatment-planning error rate from 16.1% to 4.1%. Holdsworth et al³⁵ reported an overall decrease in total plan revisions from 18% to 11.2% after introduction of their in-house automatic plan checking software. Covington at al²⁴ reported a 60% reduction in the number of patient delays in the 6 months after implementation of their in-house plan check tool.

Our study focused on the IMRT and SBRT treatment process for the X1 system but will be extended to BgRT when it obtains FDA approval and is enabled for clinical use. Before installing the X1 system in their institution, Hwang et al¹⁴ described a detailed clinical workflow and process map for BgRT and SBRT treatments on the X1 system. They found that 74 of the 133 total steps identified for treatment on the X1 system were unique to BgRT delivery, which indicates a significant departure from conventional workflows used in radiation oncology.¹⁴ These steps originate primarily from acquiring PET images of the patient on the X1 system, creating special biological treatment volumes for planning, and plan optimization using the acquired pretreatment PET images.¹⁴ While the study by Hwang et al represents an important first step toward implementation of BgRT into current clinical practice, additional work needs to be performed before widespread adoption of X1 system. In addition to process mapping, an FMEA analysis needs to be performed with a particular focus on BgRT treatments, as the results of such an analysis can be used to improve treatment workflow to minimize any identified errors, similar to the present work.

The present study is not without limitations. As the FMEA was conducted prospectively, the occurrence (O) values were determined based on the records from the departmental ILS based on Eclipse planning experience and on RefleXion preclinical evaluation. For occurrence values at 12 months after script implementation, incidence reporting on RefleXion X1 was used. Of particular note, only 3 errors were caught and reported in the 12 months after initial clinical implementation of the X1 system, all of which were not eligible for automation. However, error reporting cannot be assumed to be consistent or complete throughout the period. Another limitation is that only specific failure modes could be coded into the API for automation, either owing to restrictions in the API or to tasks that were unsuitable for scripting. These failure modes that could not be automated required changing clinical workflows or policies and procedures. In addition to these uncertainties, FMEA is only a semiquantitative analysis and is highly dependent on the users’ assessment of the risk factors and their impact in the clinic.

Conclusion

Introduction of novel technology in the clinic can be highly error prone, especially in the environment of tight timelines, increased pressure for quick implementation, and scarce resources to properly establish workflows and quality assurance checkpoints. To limit the risk of this error-prone process, a prospective FMEA analysis of RefleXion X1 IMRT and SBRT treatment planning was performed in the Six Sigma DMAIC framework. The failure modes with high severity and high RPN values were then prioritized for automation with the ESAPI. The average difference between postscript and prescript RPN values was –90.2, whereas the average prescript RPN was 138.0 compared with the average postscript RPN of 47.8, indicating a significant improvement in the safety of the planning practice. After treating 80 patients and 1747 fractions over 12 months with the X1, only 3 errors were reported, none of which were eligible for automation. The identified plan check items in this work will require periodic re-evaluation for changes in risk associated with, for example, changes in clinical practice. Future work includes implementation of BgRT-specific check items in the scripts and modifying the APC to force the user to correct any detected errors, not merely to inform the user. Furthermore, in addition to the rule-based QC methods currently used, knowledge-based QC methods will be implemented in the scripts to detect outliers and raise warnings about suboptimal plans.³⁶