Failure mode and effect analysis-based quality assurance for dynamic MLC tracking systems

Abstract

Purpose: To develop and implement a failure mode and effect analysis (FMEA)-based commissioning and quality assurance framework for dynamic multileaf collimator (DMLC) tumor tracking systems.

Methods: A systematic failure mode and effect analysis was performed for a prototype real-time tumor tracking system that uses implanted electromagnetic transponders for tumor position monitoring and a DMLC for real-time beam adaptation. A detailed process tree of DMLC tracking delivery was created and potential tracking-specific failure modes were identified. For each failure mode, a risk probability number (RPN) was calculated from the product of the probability of occurrence, the severity of effect, and the detectibility of the failure. Based on the insights obtained from the FMEA, commissioning and QA procedures were developed to check (i) the accuracy of coordinate system transformation, (ii) system latency, (iii) spatial and dosimetric delivery accuracy, (iv) delivery efficiency, and (v) accuracy and consistency of system response to error conditions. The frequency of testing for each failure mode was determined from the RPN value.

Results: Failures modes with RPN≥125 were recommended to be tested monthly. Failure modes with RPN<125 were assigned to be tested during comprehensive evaluations, e.g., during commissioning, annual quality assurance, and after major software∕hardware upgrades. System latency was determined to be ∼193 ms. The system showed consistent and accurate response to erroneous conditions. Tracking accuracy was within 3%–3 mm gamma (100% pass rate) for sinusoidal as well as a wide variety of patient-derived respiratory motions. The total time taken for monthly QA was ∼35 min, while that taken for comprehensive testing was ∼3.5 h.

Conclusions: FMEA proved to be a powerful and flexible tool to develop and implement a quality management (QM) framework for DMLC tracking. The authors conclude that the use of FMEA-based QM ensures efficient allocation of clinical resources because the most critical failure modes receive the most attention. It is expected that the set of guidelines proposed here will serve as a living document that is updated with the accumulation of progressively more intrainstitutional and interinstitutional experience with DMLC tracking.

INTRODUCTION

Real-time tumor tracking using a dynamic multileaf collimator (DMLC) represents an attractive solution to manage intrafraction motion during radiotherapy delivery to thoracic and abdominal tumors. In several recent studies, DMLC tracking has been empirically demonstrated to provide good geometric and dosimetric accuracy for a variety of delivery techniques including 3D conformal delivery, step-and-shoot intensity modulated radiotherapy (IMRT), dynamic or sliding window IMRT, and intensity modulated arc therapy.^{1, 2, 3, 4, 5, 6, 7} In order to translate this (or any other) promising technology into clinical practice, it is necessary to set up reliable clinical quality management (QM) guidelines. We define clinical QM for a tracking system as a set of procedures that includes (i) comprehensive evaluation of the system such as that performed during commissioning, annual testing, and major hardware∕software upgrades, as well as (ii) more frequent quality assurance (QA), performed monthly, weekly, or daily.

In this work, we examine the clinical QM requirements for a prototype DMLC tracking system. Toward this, we propose a failure mode and effect analysis (FMEA)-based approach to formulate a QM protocol for a DMLC tracking system. The present QM protocol pertains to a specific platform: Electromagnetic (EM) transponder-based position monitoring (Calypso Medical Technologies, Seattle, WA) and real-time beam adaptation using a DMLC (Millennium-120, Varian Medical Systems, Palo Alto, CA). However, the analytical methodology and the guidelines formulated herein may be implemented with appropriate modifications for other similar DMLC tracking systems. Appendix A describes the equipment needs for performing these tests, while Appendix B provides example worksheets for recording the commissioning and QA measurements. FMEA survey instrument,³³ available at: http://www.aip.org/pubservs/epaps.html.

BACKGROUND

Failure mode and effect analysis

FMEA is an industrial engineering technique for risk management and safety improvement of complex processes.^{8, 9, 10, 11} This technique has been used extensively in a variety of industries such as chemical engineering, pharmaceuticals, and medical device manufacturing. In the context of radiotherapy, the forthcoming AAPM Task Group (TG) 100 (Ref. ¹²) recommends FMEA as the framework of choice for setting up clinical QM protocols.

The FMEA technique consists of the following steps: (i) identifying each step in a process and charting a process tree, (ii) identifying potential modes of failure for each step, (iii) identifying the potential causes and the local as well as downstream effects of each failure and, (iv) assessing the overall risk of each failure using three independent variables: The probability of occurrence (O), the severity of effect (S), and the detectibility, i.e., probability of failure to detect (D). Typically, each variable ranges from 1 to 10, where 1 represents the best-case scenario (no risk) and 10 represents the worst-case scenario (certain failure). The overall risk probability number (RPN) for each failure mode is calculated as

$$\text{RPN}=O\times S\times D.$$ (1)

When developing a QM protocol, failure modes corresponding to the highest RPNs will command the highest priority. In contrast to a prescriptive QM approach (i.e., test everything), the FMEA-based approach aims to achieve a high level of safety and operational consistency, while utilizing minimal additional clinical resources. Detailed discussions on various considerations for implementing FMEA-based QM for radiotherapy can be found in Rath.¹³ Implementation of FMEA-based QM for the external beam radiotherapy has been described by Ford et al.¹⁴ Finally, Huq et al.¹² discussed the FMEA-based QM framework under development for the forthcoming AAPM Task Group 100, which is examining the QM needs of modern radiotherapy.

DMLC tracking-based delivery process

In the present study, we use the FMEA technique to develop QM procedures for a real-time dynamic MLC tracking system. As discussed in Sec. 2A, the first step in such a study is to chart a process tree. Figure 1 shows a process tree for real-time DMLC tracking-based delivery. In the context of the present discussion, we define “real-time” as a time duration much smaller than that observed for the motion of the tumor target. The shaded portion in Fig. 1 highlights process steps specific to DMLC tracking. All other functional blocks are identical to current motion-managed image-guided radiotherapy (IGRT). Therefore, the delivery scenario discussed in this work assumes that the steps upstream of tracking-based delivery, i.e., the creation of the treatment plan, definition of setup margins, pretreatment setup and localization, etc., are performed in a manner similar to nontracking-based IGRT delivery (e.g., gating).

Figure 1.

Process flow of real-time DMLC tracking-based radiotherapy. The steps highlighted by the shaded region are specific to tracking. All other steps are common to current motion-managed IGRT.

During tracking-based delivery, target position is continuously monitored by a position monitoring system. In the present case, this task is performed by a research version of the Calypso system, which provides information about the 3D translation (x, y, z) of the target centroid at 25 Hz. This information is passed on to the MLC tracking software module, which modifies the original fluence map by recalculating MLC leaf positions based on the treatment plan as well as the target position. For tumor motion that is periodic or semiperiodic (e.g., lung and liver) and of a sufficiently high frequency relative to the temporal latency of the system, a kernel density prediction algorithm is used to account for the latency.¹⁵ For aperiodic motion (e.g., prostate motion), prediction is disabled. The recalculated leaf positions are transmitted to the MLC controller via Ethernet which, in the case of the Varian Millennium 120-leaf MLC, leaf positions are updated at 20 Hz. The MLC controller actuates the motion of individual MLC leaves. Throughout the tracking process, a variety of safety checks are performed to verify the integrity of communication between the position monitoring system and the MLC tracking software to identify errors and∕or anomalous conditions such as the target moving outside the field or rapid∕erratic motion of the target. When such anomalous conditions are detected, the radiation beam is held off, but the MLC leaves continue to follow the target motion so as to enable quick resumption of treatment once the erroneous condition has passed or has been rectified. A detailed system description and characterization of the MLC tracking algorithm and the integrated system may be found in Refs. ^{1, 3}, respectively.

Note that there is ongoing research on using all phases of a 4D CT scan to generate a 4D treatment plan that calculates leaf positions by explicitly accounting for the state of the anatomy in each respiratory phase.^{16, 17} Such methods, while promising, are not yet close to clinical implementation and, therefore, not addressed in the present work. Thus, example delivery scenarios covered by this document include DMLC tracking of prostate motion based on a 3D CT-based plan, or DMLC tracking of a thoracic or abdominal tumor based on a plan from a single phase of the 4D CT scan, or a plan based on an expected volume derived from multiple phases of a 4DCT.

Scope of MLC tracking QM

The introduction of any new technology into the clinic requires the formulation of reliable procedures to ensure safe, consistent, and accurate operation. In practice, the frequency and the complexity of these procedures have to be balanced with the availability of resources toward these tasks.^{12, 18} In order to achieve a proper balance, we limit the scope of additional QM for tracking-based delivery to the testing of tracking-specific processes and components of the integrated system. We assume that individual subsystems within the integrated tracking system have undergone and passed prescribed commissioning and QA procedures. Such procedures have been widely described in the literature. For example, a detailed commissioning and QA protocol for electromagnetic transponder-based position monitoring using the Calypso system has been described by Santanam et al.¹⁹ Procedures for general linac QA may be found in the AAPM TG 40 (Ref. ¹⁸) and the more recent TG 142.²⁰ QA issues related to the Varian MLC, including S-IMRT and D-IMRT delivery using the Varian platform, have been discussed in detail by LoSasso.²¹ Commissioning guidelines for IMRT are detailed in AAPM TG 119.²² Finally, Ling et al.²³ have described QA procedures for RapidArc delivery.

Considerations for MLC tracking QM

From a functional standpoint, an integrated DMLC tracking system has to fulfill three requirements.

• i.Spatial and dosimetric accuracy. We define spatial and dosimetric error of tracking-based delivery to a moving target as the difference in the dose map with respect to delivery of the same beam to a static target. Accurate tracking-based delivery requires accurate transformation from patient coordinates to MLC coordinates, accurate characterization of system latency where predictive algorithms are used, and accurate leaf-fitting at each control point so as to optimally account for dose fraction and the instantaneous position of the target.
• ii.Efficient dose delivery. We define delivery efficiency as the ratio of beam-on time to the total time required to deliver a given beam. Higher delivery efficiency is desirable as it results in reduced treatment time, reduced patient discomfort, and reduced probability of patient motion. Tracking-based delivery has the potential to achieve high efficiency; significantly higher than, for example, respiratory gating.
• iii.Detection of and response to anomalous conditions. During the course of tracking-based delivery, a variety of anomalous conditions can occur. In each such situation, the system initiates a beam-hold. Where possible, the MLC leaves continue to follow the target motion so that delivery can be resumed as soon as the target is in a treatable position.¹ For the system to pass quality requirements, it should successfully detect each anomalous condition and trigger a beam-hold.

The goal of the tests developed in the present work is to ensure that these requirements are consistently and accurately satisfied. We assume that the integrated system has undergone acceptance testing according to vendor guidelines. It is thus assumed that the system is found to perform to specifications, software and hardware upgrades have been tested, patient and room coordinate transformations have been verified, and the system has built-in mitigators such as interlocks and backup systems for handling software errors and∕or crashes, mechanical failures, data transmission failure, etc.

METHODS

A detailed FMEA was performed for the tracking components of the process tree (shaded portion in Fig. 1). For each functional block in the shaded portion, possible tracking-specific failure modes were identified. A “failure” was defined as an event where an error occurs but does not trigger the corresponding mitigator. A tracking-specific failure was defined as a failure that occurs when the system is used in the MLC tracking mode. Say, for example, the patient was positioned prone while the treatment plan had been developed for a supine position. For the Calypso system, this would mean that the beacons are positioned 180° rotated through the superior-inferior (SI) axis. This is a nontracking failure that should be identified and rectified during the established QM∕QA of the position monitoring system. (In this case, the position monitoring system should communicate with the treatment delivery system, check patient orientation, and assert an interlock.) Say, however, that the communication between the PM system and MLC tracking software is lost, but instead of holding the beam, the system continues to treat at the last target position received. This is an example of a tracking-specific error that lies within the scope of the present analysis and QM∕QA procedures.

The impact of each failure was quantified by assigning a score between 1 and 10 for (i) the probability of occurrence of the failure (O), (ii) the severity of the failure (S), and (iii) the probability of failure to detect (D). A common scale, shown in Table 1, was established for scoring the three variables.

Table 1.

Common scale used in assigning O, S, and D values for MLC tracking FMEA. ADL: Activities of daily living.

Value	Probability of occurrence (O)	Severity of effect (S) a	Detectibility (D) (probability of failure to detect)
1	Very unlikely (<0.01%)	No adverse event (AE)	Very unlikely (i.e., always detected) (<0.01%)
2–3	Low probability (0.02%–0.04%)	Grade 1: Mild; asymptomatic or mild symptoms; clinical or diagnostic observations only; intervention not Indicated	Low probability (0.2%–0.5%)
4–5	Some probability (0.05%−0.4%)	Grade 2: Moderate; minimal, local or noninvasive intervention indicated; limiting age-appropriate instrumental activities of daily living (ADL)b	Some probability (1%–2%)
6–7	Moderate probability (0.5%–1%)	Grade 3: Severe or medically significant but not immediately life-threatening; hospitalization or prolongation of hospitalization indicated; disabling; limiting self-care ADLc	Moderate probability (5%–10%)
8–9	High probability (2%–5%)	Grade 4: Life threatening consequences. Urgent intervention indicated.	High probability (15%–20%)
10	Certain failure (>5%)	Grade 5: Patient death related to AE	Certain failure (impossible to detect) (>20%)

^aSource: National Cancer Institute Common Terminology Criteria for Adverse Events v4.0 (2009).

^bInstrumental ADL: Preparing meals, shopping for groceries or clothes, using the telephone, managing money, etc.

^cSelf-care ADL: Bathing, dressing and undressing, feeding self, using the toilet, taking medications, and not bedridden.

While occurrence and detectibility were scored based on estimated probabilities, the severity of effect was scored based on the National Cancer Institute’s Common Terminology Criteria for Adverse Events.²⁴ The overall impact of each failure was then quantified by the RPN [Eq. 1].

In the present study, a short survey was sent out to seven physicists from five different institutions. Each physicist in the survey was chosen because they were closely involved in the development and∕or use of the tracking system. (Note that the relatively small number of respondents was due to the early nature of the MLC tracking work.) The survey included an explanation of the FMEA methodology, the MLC tracking process tree (Fig. 1), the common scale (Table 1), failure mode charts for two scenarios (a conventionally fractionated treatment and a hypofractionated treatment), and an example (nontracking) FMEA to illustrate the assignment of O, S, and D. The survey also included an open-ended field for each participant to enter failure modes not identified by the authors. The average O, S, and D values from the responses were used to calculate the RPN [Eq. 1] for each failure mode.

Tracking-specific tests

The insights obtained from the FMEA were used in order to develop procedures to test each failure mode (see Table 2, second column). These tests were performed at Stanford University on the prototype DMLC tracking system with the linac operated in Service mode. Note that the test set described under “commissioning” is also to be used for annual system testing and for testing after major software∕hardware upgrades. Furthermore, in Sec. 3A1, the term “target” is used interchangeably to refer to the tumor target and the centroid marked by the electromagnetic transponders. It is of course clearly recognized that when implanted in a patient, the transponders merely serve as surrogates and are distinct from the tumor target.

Table 2.

FMEA for conventionally fractionated and hypofractionated DMLC tracking-based delivery. For each failure mode, numeric values are assigned to the probability of occurrence (O), severity of effect (S), and detectibility of failure (D), and a RPN is calculated.

Process step	Failure mode	Possible causes of failure	Effect of failure	Conventional fractionation				Hypofractionation
				O	S	D	RPNC	O	S	D	RPNH
1. Position Monitoring (PM) system estimates real-time target position	i. Target moves outside spatial tolerance set by PM system, but beam-hold is not asserted	i. (a) Beam-hold not sent by monitoring system to tracking system	Dose error >15%	3.4	5.7	4.0	78	3.3	7.1	3.6	84
2. Tracking system receives real-time target position and recalculates MLC leaf positions as a function of dose fraction and target position	ii. Communication with PM system is lost, i.e., target position is no longer current but beam-hold not asserted	ii. (a)PM system failure	Dose error >30%	3.7	7.1	2.6	68	3.7	8.4	2.6	81
		ii. (b) Data transfer cable(s) physically disconnected
	iii. Error in coordinate system conversion	iii. (a) System installation error	Dose error >60%	1.7	8.6	4.3	63	1.7	9.3	3.9	61
		iii. (b) Hardware∕software changes
	iv. Optimal leaf-fitting not achieved	iv. (a) Complex motion+highly modulated field	Dose error >10%	5.4	4.0	6.7	146	5.3	5.0	6.7	177
3. Tracking system checks if fluence map is completely within field	v. Beam-hold not asserted when fluence map is partially or completely under one or more jaws	v. (a) Software crash or failure	Dose error >20%	2.9	4.9	3.7	52	2.6	6.1	3.9	61
		v. (b) Jaws outside tolerance
4. MLC controller actuates leaf motion. If leaves are within tolerance, linac delivers dose, else beam is held off.	vi. System latency outside expected range	vi. (a) Hardware and∕or software changes	Dose error >5%	4.0	3.0	6.0	72	4.0	4.1	6.3	104
		vi. (b) Changes in network connection speed
	vii. Too many beam-holds. Efficiency drops below desired threshold	vii. (a) MLC leaves cannot keep up with target motion	Efficiency <70%	7.0	2.3	1.4	23	6.0	2.9	1.6	27

Experimental setup for commissioning

Figure 2a shows the experimental setup for the commissioning of a DMLC tracking system. The arrangement consists of a research version of the Calypso system for real-time 3D position monitoring (at 25 Hz) and a Varian 120-leaf DMLC running a research software for real-time beam adaptation. A high-precision (∼100 μm), three-axis, programmable motion platform was used to simulate target motion.²⁵ A 2D ion chamber array (PTW Seven29, PTW, Freiburg, Germany) was fixed to the arm of the three-axis phantom, with 20 mm water equivalent buildup and backscatter. An aluminum foil was wrapped around the dosimetric array in order to minimize electromagnetic interference between the PTW electronics and Calypso electronics. The source-to-detector distance was 100 cm. Real-time position was monitored by exciting and reading three electromagnetic transponders embedded in a circular polystyrene disk (10 mm thick and 40 mm diameter) which was securely fastened to the dosimetric array [Fig. 2a]. The three-axis platform was programed with mathematically defined sinusoidal as well as patient-derived motion traces and the instantaneous 3D position calculated by the Calypso system was transmitted via an Ethernet connection to the DMLC tracking computer. The tracking algorithm recalculated MLC leaf positions as a function of dose fraction and target position.^{1, 26} The recalculated leaf positions were sent at 20 Hz via a separate Ethernet connection to the MLC controller which actuated the mechanical movement of the MLC leaves.

Figure 2.

Measurement setup to perform (a) comprehensive (commissioning∕annual QA) and (b) frequent (monthly QA) tests for an electromagnetic position monitoring based DMLC tracking system

Experimental setup for monthly QA

The setup used for monthly QA tests [Fig. 2b] was a simplified version of that used for commissioning. The ion chamber array assembly described above was fixed to a motion phantom capable of 1D sinusoidal motion. A standard Calypso QA fixture block was placed at a fixed position on the phantom. The remaining operation of the tracking system was identical to that described above. Note that all of the tests that are performed using the setup in Fig. 2b can be performed using the setup in Fig. 2a, but not vice versa. The following procedures were developed to test each failure mode:

• (1)Target moves outside a predefined 3D tracking volume. A tracking volume of 5×5×5 cm³ was defined for a target positioned and localized at the isocenter using the Calypso system. Continuous position monitoring was enabled and DMLC tracking-based beam delivery using a 10 cm diameter circular field was performed on the static target. Subsequently, the patient couch was shifted by 6 cm in each direction (i.e., outside the predefined tracking volume). In each case, beam delivery with DMLC tracking was attempted and the system response (i.e., beam-hold) was recorded.
• (2)Communication failure. System response to communication failure between the position monitoring system and the DMLC tracking system was verified by manually switching off the Calypso system during beam delivery and checking the status of the beam-on.
• (3)Error in coordinate conversion. The three-axis motion platform was programed with 1D sinusoidal motion (30 mm peak-to-peak, 6 s period) in each of the three directions: SI, anterior-posterior (AP), and left-right (LR). Each motion trajectory was tracked via a circular MLC field (100 mm diameter at isocenter, jaws 14×14 cm², and field light turned on) for gantry angles 90°, 180°, and 270° and collimator angles 45° and 270°. At each gantry and collimator angle combination, it was visually verified that the MLC field projected on the phantom by the field light moved in the same direction as the phantom in each of the three directions. For motion along the beam axis, this meant magnification (demagnification) of the aperture as the target moved toward (away from) the source.
• (4)Optimal leaf-fitting not achieved (dosimetric error). The spatial and dosimetric accuracy of tracking-based delivery to a moving target was quantified by comparing it to delivery of the same field to a static target. For an ideal system, the two dose distributions should match exactly, though in reality system latency, finite leaf velocity, and width, position monitoring system uncertainties and algorithmic limitations introduce errors. The differences in the dose distributions indicate the spatial and dosimetric delivery errors introduced during DMLC tracking-based delivery in the presence of motion. Following the guidelines proposed by the AAPM Task Group 119 on IMRT commissioning,²² we quantified dosimetric error using a 3%–3 mm gamma index criterion normalized to the maximum dose.

For comprehensive testing, such as used during commissioning, we chose to characterize delivery accuracy over a representative set of patient-derived motion trajectories. For the present study, the setup shown in Fig. 2a was used. Dosimetric accuracy was measured for a highly modulated sliding window IMRT lung stereotactic body radiotherapy (SBRT) field [6 MV, gantry 20°, collimator 270°, and 753 monitor units (MU) at 600 MU∕min]. The collimator angle was chosen so as to align the direction of leaf motion in the SI direction. Four representative motion trajectories (Fig. 3) recorded from lung cancer patients using the Synchrony system (Accuray, Sunnyvale, CA) were programed into the motion platform.²⁷ These trajectories were chosen so as to be representative of the wide variety of respiratory motion observed clinically and were comprised of typical motion (predominantly SI), high frequency motion due to rapid breathing, predominantly left-right motion, and baseline shifts. In each case, measurements were obtained for (a) static target, (b) moving target, no tracking, and (c) moving target with DMLC tracking.

Figure 3.

Three-dimensional motion trajectories recorded from lung cancer patients using the Synchrony system used as patient representative motion in this study (Suh et al., Ref. 27).

For monthly QA, we chose a simplified setup [Fig. 2b] and measurement procedure. A 1D motion platform was used to simulate periodic sinusoidal motion. A dynamic delivery picket fence test pattern (Fig. 4) was defined so as to test the MLC leaves overlying a central region in the isocentric plane, where the tracking volume is typically defined. Dose was delivered in dynamic mode and was measured using the PTW array fixed on the 1D platform programed with sinusoidal motion (30 mm peak-to-peak, 4 s period) along the SI direction and parallel to the MLC leaf motion. Linac parameters were: 6 MV, X-Y=14.2×26.5 cm², gantry 180°, collimator 270°, and 400 MU at 600 MU∕min. Measurements were obtained for (a) static target, no tracking, (b) moving target, no tracking, and (c) moving target with DMLC tracking.

Figure 4.

(a) Snapshot of picket fence pattern proposed for monthly QA. The thin horizontal and vertical lines drawn across the frame depict the positions of the X and Y jaws, respectively. (b) Simulated, integrated image of entire delivery

(5) Fluence pattern under one or more jaws. The gantry and collimator were set to 180° and 270°, respectively. A 13×13 cm² field was defined using the X and Y jaws. A 10 cm diameter circular field tracked a large-amplitude target motion (sinusoidal, 40 mm peak-to-peak), such that the field was partially under a jaw at each peak. The beam-hold response was recorded. The process was repeated after rotating the collimator by 90° in order to test system response to target motion under the second set of jaws.

(6) Latency. We define latency as the elapsed time between target motion and MLC response. System latency was measured using a previously developed methodology,³ briefly outlined as follows. The motion platform was programmed with a sinusoidal trajectory in the SI direction (20 mm peak-to-peak, 4 s∕cycle). A plastic block embedded with electromagnetic transponders and a spherical 2 mm diameter tungsten ball bearing served as the target and was fixed to the arm of the platform. The motion of the target was tracked by a 10 cm diameter circular MLC field (jaws 13×13 cm², gantry 90°, and collimator angle 270°). The tracking was recorded via continuous EPID imaging (∼13 frames∕s). The EPID images were segmented offline to locate the position of the target and the centroid of the MLC aperture in each frame. These positions were plotted as a function of time and the motion trajectories thus obtained were fitted with sine curves. System latency was calculated from the time difference between the two trajectories. The measured latency value was used as input to a kernel-based prediction algorithm and the experiment was repeated with prediction enabled in order to estimate the residual latency.

(7) Delivery efficiency. The efficiency of dose delivery was calculated for each delivery from the ratio of the beam-on time (B) to the elapsed time (E) values which are displayed on the linac console. The relative efficiency of tracking-based delivery η_relative was calculated from

$${\eta }_{\text{relative}}=\frac{B_{\text{tracking}}∕E_{\text{tracking}}}{B_{\text{static}}∕E_{\text{static}}},$$ (2)

where the subscripts “tracking” and “static” indicate delivery in the presence and absence of MLC tracking, respectively. A failure mode was defined as η_relative<70%.

RESULTS

Failure mode and effect analysis

Table 2 outlines each step in the process of DMLC tracking-based delivery and shows the proposed FMEA-based framework. For each potential failure, the average values of O, S, and D (rounded to the first decimal place) are shown, along with the corresponding RPN. The open-ended field in the FMEA table was not filled by any of the participants. The perceived severity of effect and therefore the RPN values for hypofractionated delivery (RPN_H) were systematically higher than those for conventionally fractionated delivery (RPN_C). In both cases, optimal leaf-fitting during tracking-based delivery was the highest concern (RPN_C=146; RPN_H=177), while delivery efficiency was the lowest (RPN_C=23; RPN_H=27).

FMEA-based QM∕QA tests

The frequency of performing each QM∕QA test was determined by the corresponding RPN value. In the present study, we (somewhat arbitrarily) chose a threshold RPN=125 (⇒O=S=D=5). We determined that for failure modes with RPN≥125, tests be performed as part of monthly QA. Due to the fact that MLC tracking is still a relatively new technology, we do recommend that failure modes with RPN<125 be tested annually as well as during commissioning and after major hardware∕software upgrades.

• (1)Target moves outside a predefined 3D tracking volume (RPN_C=78; RPN_H=84). Treatment was correctly interrupted by a system-initiated beam-hold when the target moved outside a predefined tracking volume of 5×5×5 cm³.
• (2)Communication failure (RPN_C=68; RPN_H=81). Treatment was interrupted by a system-initiated beam-hold when the communication between the Calypso system and the tracking system was lost.
• (3)Error in coordinate conversion (RPN_C=63; RPN_H=61). The directionality of tracking was visually observed to be correct for each of the six gantry and collimator angle combinations, indicating that the online coordinate transformation of target motion from patient coordinates to MLC coordinates was being performed correctly.
• (4)Optimal leaf-fitting not achieved (RPN_C=146; RPN_H=177). Figure 5 shows isodose maps for the lung SBRT field delivered to a static target [Fig. 5b] and the same field delivered to a target moving with the 3D “typical motion” trajectory (Fig. 3) in the absence [Fig. 5a] and presence [Fig. 5c] of MLC tracking. The square symbols indicate points on the dosimetric array that failed a 3%–3 mm gamma index criterion normalized to maximum dose. Figure 6 shows percentage points failing a 3%–3 mm gamma criterion for delivery to a moving target compared to delivery to a static target. Figure 6a shows results for a sliding window lung SBRT field in the presence and absence of MLC tracking for the four representative lung tumor motion trajectories shown in Fig. 3. Tracking-based delivery exhibited a failure rate of 0% for all of the four representative motion trajectories. In contrast, failure rates in the absence of DMLC tracking ranged from 4.2% (baseline shifts) to 57% (typical motion). Similar results were observed for the picket fence pattern delivered to a sinusoidally moving target (30 mm peak-to-peak, 4 s period). As shown in Fig. 6b, ten consecutive measurements of tracking-based delivery exhibited an average failure rate of ∼4% (±2.8%) compared to ∼53% failure without tracking. For the monthly QA, tracking-based delivery (picket fence+sinusoidal motion trajectory) was performed ten times in order to be conservative. However, from the results [Fig. 6b], we estimate that for routine monthly QA, three to five consecutive measurements should be sufficient.
• (5)Fluence pattern under one or more jaws (RPN_C=52; RPN_H=61). Treatment was correctly interrupted by a system-initiated beam-hold when any part of the desired fluence pattern moved under a jaw. As expected, treatment delivery resumed once the fluence pattern was completely back into the open field.
• (6)Latency (RPN_C=72; RPN_H=104). Figure 7a shows a segmented EPID image frame depicting the center of the aperture (crosshairs) and the tungsten ball bearing (center of small circle). Motion trajectories derived from these images are shown in Fig. 7b. As expected, the MLC aperture lags behind the target because of system latency. The average latency of the system was determined to be 193 ms. This value was used as input to a kernel-based prediction algorithm. The residual latency with prediction enabled was ±4 ms, which is within the bounds of experimental error. Note that during tracking-based delivery, prediction is used only for periodic or semiperiodic motion such as respiratory motion. Thus, for example, prediction would typically be turned off during DMLC tracking of prostate motion.
• (7)Delivery efficiency<70% (RPN_C=23; RPN_H=27). Table 3 shows the delivery efficiency of MLC tracking relative to a static delivery for five different motion trajectories: The four representative respiratory motion trajectories (Fig. 3) and a sinusoidal trajectory for the picket fence delivery (Fig. 4). The system passed the test criterion (η_relative≥70%) in all but one case, predominantly LR motion. The low efficiency in the LR case is likely due to the fact that the principal axis of target motion was perpendicular the motion of the MLC leaves. It has been shown previously that such perpendicular target motion can cause significant degradation of delivery efficiency.¹

Figure 5.

Isodose maps for the lung SBRT field delivery for a target undergoing typical motion (Fig. 3). Dose maps are shown for three cases: (a) Moving target, no tracking, (b) static target, and (c) moving target with MLC tracking. The solid squares indicate points on the PTW array that failed a 3%–3 mm gamma test with respect to the dose map shown in (b), corresponding to the static delivery.

Figure 6.

Percentage points failing a 3%–3 mm gamma index criterion in the absence and presence of DMLC tracking-based delivery for (a) a lung SBRT field (commissioning∕annual QA) and (b) ten measurements using a picket fence pattern (monthly QA)

Figure 7.

(a) EPID image of target (small circle) and aperture (crosshairs) and (b) the corresponding motion trajectories (red, target; blue, aperture), used to calculate the total temporal latency of the integrated system.

Table 3.

Relative efficiency of MLC tracking with respect to static delivery.

Motion	ηrelative (%)
Typical	95
High frequency breathing	92
Predominantly L-R	59
Baseline shifts	74
Sinusoidal	80

Time required for performing QM∕QA tests

For monthly QA, the total time required for setup and measurements was ∼35 min. In the case of the commissioning tests, setup and measurements required ∼2.5 h. Offline analysis of latency measurements comprising transfer of EPID images to a standalone PC workstation (3 GHz Pentium Core-2, 4 GB RAM) and image segmentation and analysis using an in-house MATLAB routine required an additional one hour.

DISCUSSION AND CONCLUSION

Our objective was to develop a quality management framework for DMLC tracking systems that maximizes system diagnostics while minimizing additional demands on clinical resources in terms of personnel time and equipment. Fortunately, for a tracking system, a large portion of the work is already performed as part of routine QM of each subsystem. The use of FMEA helped further streamline this process by ensuring that the most critical failure modes receive the most attention and vice versa.

Given the early stage of the implementation of the MLC tracking system, there are a few limitations to this study. First, the number of potential participants was limited to 11 physicists and one physician (individuals who have used the system). Of these, only seven physicists responded. We expect that as MLC tracking becomes increasingly widespread, a greater number of medical physicists can contribute toward the analysis. Furthermore, we believe that the present analysis, as well as any radiotherapy FMEA, will greatly benefit from input given by allied professionals: Physicians, dosimetrists, and therapists. In particular, physician input for scoring severity of effect can prove to be an invaluable resource. Second, due to the limited number of users, the dosimetric error for each failure mode was based on our estimates as expert users of the tracking system. Ideally, these errors should be cumulatively recorded and quantified from repeated measurements performed by several users. Third, the failure mode tests in this study did not explicitly check for false positives, i.e., a failure detected when it has not actually occurred.

In addition to the use of FMEA, we also leveraged the quality management know-how from a closely related, clinically established technology (stereotactic robotic radiosurgery)²⁸ to inform our recommendations of the frequency of testing various failure modes. For example, we do not recommend daily QA for MLC tracking. This decision is in line with recommended procedures for robotic radiosurgery systems, where the robotic targeting accuracy is tested on a monthly basis.²⁸

While the issue of patient-specific QA is beyond the scope of the present study, the experimental setup and methodology described for the commissioning measurements may be used as a good starting point for implementing such protocols. Admittedly, the experimental setup for the commissioning∕annual measurements is relatively complex and resource-intensive. This more conservative approach was felt necessary due to the fact that MLC tracking is a new technology. As clinical experience increases, it may be possible to simplify the experimental setup and measurements for commissioning and annual testing.

While the dosimetric tests in the present study were performed using a 2D ion chamber array, they can easily be performed with 2D diode arrays or, as described by Smith et al.,⁴ using dosimetric film (preferably from the same batch). For all of the dosimetric tests developed in this work, we chose to compare tracking-based delivery with static delivery to a static target rather than to a treatment plan. Our method ensures that the errors observed in tracking-based delivery are clearly decoupled from other sources of error such as geometric setup uncertainties, finite resolution of the MLC leaves, etc. Errors that occur or could occur during delivery to a static target are outside the scope of tracking QM. Such nontracking errors should be identified and rectified by existing institutional QM procedures, based on recommendations from the manufacturer, appropriate Task Group reports, and institutional experience.

Finally, it is important to understand that there are limitations to applying a classical RPN-based FMEA to radiotherapy, where catastrophic events can have, quite literally, life and death consequences. In the classical approach (RPN=O×S×D), the “perceived criticality” of catastrophic, low probability failures may be equal or even less than that of moderate, higher frequency failures. In such situations, the FMEA provides an incomplete characterization of the actual risk. Several researchers have proposed more sophisticated FMEA methodologies in order to overcome the limitations of classical RPN-based FMEA.^{29, 30, 31, 32} While a full discussion of this issue is beyond the scope of the present work, we recommend that users carefully understand both the benefits as well as the limitations of RPN-based FMEA before implementing it in a radiotherapy setting.

In conclusion, the FMEA-based framework is well-suited for the clinical incorporation of new technologies, as it not only focuses resources to where they are most needed but also serves as a living document. In this context, it is important to note that the values of O, S, and D presented in Table 2 are not absolute, but rather, starting points. These values will be updated as collective user experience with MLC tracking increases within and across institutions. Finally, it is expected that the accumulation of progressively more failure mode data and the implementation of more sophisticated FMEA methodologies will further enhance the role of this QA∕QM framework in guiding design and operational aspects of future tracking systems.

APPENDIX A: TOOLS AND EQUIPMENT

The following equipment is required in order to conduct commissioning and quality assurance for Calypso-based DMLC tracking

Test phantom

Monthly QA

The QA fixture which comes as part of the standard Calypso package (Fig. 8), may be used for monthly QA. This phantom contains three embedded EM transponders and has scribe lines on the surface, which can be aligned with respect to the room lasers so that the center of the phantom is positioned at the machine isocenter.

Figure 8.

Schematic illustration (not to scale) of the Calypso QA fixture showing horizontal and vertical scribe lines which can be used to align the phantom with respect to the room lasers.

Commissioning

For commissioning, we recommend a phantom with a smaller vertical dimension in order to have sufficient clearance between the top of the phantom and the electromagnetic array. For the measurements reported in the present work, a circular disk with embedded EM transponders was used.

Motion platform

Monthly QA

For monthly QA, we recommend a motion platform capable of 1D sinusoidal motion up to 30 mm peak-to-peak and 4 s period. It is desirable, but not required, that the range of motion and the speed of motion be adjustable.

Commissioning

In order to test the performance of tracking with patient-derived motion traces and to efficiently test coordinate system accuracy, we recommend a three-axis motion platform capable of moving ±30 mm in the horizontal plane and ±20 mm along the vertical axis, with a speed of 1.5 cm∕s. We recommend a minimum load carrying capacity of 5 kg in order to be able to carry the weight of the phantom and the dosimeter (with appropriate buildup and backscatter) without significant flex.

MV imager (Commissioning only)

A MV x-ray imager capable of acquiring portal images in cine-mode (≥7 frames∕s) is required during commissioning. This imager serves as an independent recording system of the tracking process and is required to determine system latency.³, ⁵

Planar dosimetric detector (commissioning and monthly QA)

A 2D planar dosimeter is required to perform dosimetric measurements. The dosimeter should be mountable on the motion platform, again with minimal flex of the loading arm. For user convenience, it is preferable to perform these tests using an electronic dosimetric array. Such arrays, based on ion chambers (PTW seven29, PTW, Freiburg, Germany) or diodes (MapCheck, Sun Nuclear Corp, Melbourne, FL), are commercially available. Alternately, dosimetric films (preferably from the same batch) may also be used. As with standard dosimetric measurements, the 2D array∕film should be sandwiched between solid water slabs of appropriate thickness to provide build-up and backscatter.

APPENDIX B: EXAMPLE WORKSHEETS FOR COMMISSIONING AND MONTHLY QA