Examining credentialing criteria and poor performance indicators for IROC Houston’s anthropomorphic head and neck phantom

Abstract

Purpose:

To analyze the most recent results of the Imaging and Radiation Oncology Core Houston Quality Assurance Center’s (IROC-H) anthropomorphic head and neck (H&N) phantom to determine the nature of failing irradiations and the feasibility of altering credentialing criteria.

Methods:

IROC-H’s H&N phantom, used for intensity-modulated radiation therapy credentialing for National Cancer Institute–sponsored clinical trials, requires that an institution’s treatment plan agrees within ±7% of measured thermoluminescent dosimeter (TLD) doses; it also requires that ≥85% of pixels pass ±4 mm distance to agreement (7%/4 mm gamma analysis for film). The authors re-evaluated 156 phantom irradiations (November 1, 2014–October 31, 2015) according to the following tighter criteria: (1) 5% TLD and 5%/4 mm, (2) 5% TLD and 5%/3 mm, (3) 4% TLD and 4%/4 mm, and (4) 3% TLD and 3%/3 mm. Failure rates were evaluated with respect to individual film and TLD performance by location in the phantom. Overall poor phantom results were characterized qualitatively as systematic errors (correct shape and position but wrong magnitude of dose), setup errors/positional shifts, global but nonsystematic errors, and errors affecting only a local region.

Results:

The pass rate for these phantoms using current criteria was 90%. Substituting criteria 1–4 reduced the overall pass rate to 77%, 70%, 63%, and 37%, respectively. Statistical analyses indicated that the probability of noise-induced TLD failure, even at the 5% criterion, was <0.5%. Phantom failures were generally identified by TLD (≥66% failed TLD, whereas ≥55% failed film), with most failures occurring in the primary planning target volume (≥77% of cases). Results failing current criteria or criteria 1 were primarily diagnosed as systematic >58% of the time (11/16 and 21/36 cases, respectively), with a greater extent due to underdosing. Setup/positioning errors were seen in 11%–13% of all failing cases (2/16 and 4/36 cases, respectively). Local errors (8/36 cases) could only be demonstrated at criteria 1. Only three cases of global errors were identified in these analyses. For current criteria and criteria 1, irradiations that failed from film only were overwhelmingly associated with phantom shifts/setup errors (≥80% of cases).

Conclusions:

This study highlighted that the majority of phantom failures are the result of systematic dosimetric discrepancies between the treatment planning system and the delivered dose. Further work is necessary to diagnose and resolve such dosimetric inaccuracy. In addition, the authors found that 5% TLD and 5%/4 mm gamma criteria may be both practically and theoretically achievable as an alternative to current criteria.

1. INTRODUCTION

Since 2001, the Imaging and Radiation Oncology Core Houston Quality Assurance Center (IROC-H, formerly the RPC) head and neck (H&N) phantom has been used to credential institutions wishing to participate in National Cancer Institute (NCI)-sponsored clinical trials utilizing intensity-modulated radiation therapy (IMRT) delivery techniques. The credentialing process ensures that the participating sites are capable of delivering complex treatment plans as intended. This limits the variability of data in the clinical trial and increases the quality of the study.¹ This process has also helped expose and resolve IMRT delivery errors (among other inaccuracies), which in turn helps improve an institution’s treatment delivery as a whole.^1,2

According to Molineu et al.,² since the introduction of the H&N phantom, the annual pass rate has increased from an initial low of 66% in 2001 to 88.5% in 2012 using criteria of ±7% of the planned dose and ±4 mm distance to agreement (DTA). This improvement is attributed to the growing competency of advanced delivery techniques, modeling accuracy, and IROC-H feedback.² However, a substantial number of institutions still fail to meet the minimum criteria, thus warranting further investigation to determine the root cause. Previous works like that of McVicker et al.³ have used the H&N phantom to determine the detectability of potential commissioning errors. While this work exposes some of the limitations of the phantom, it does not address the prevalence and detection of errors in multi-institutional performance. To date, no study has evaluated the predominant factors of failure in the H&N phantom credentialing results. Furthermore, as the pass rate continues to climb, it is possible that current acceptance criteria may be deemed unsuitably lax to correctly reflect the present accuracy of dose delivery.

The goal of the current study was to analyze the most recent results of IROC-H’s IMRT H&N phantom irradiations with respect to thermoluminescent dosimeter (TLD) and film measured doses. Alternative criteria were applied to determine their effects on the pass rate, and failures were identified and categorized. Additionally, TLD measurement variations were analyzed to determine the probability of noise-induced failures at different criteria. This study also delineated the greatest contributions to institutional failure and examined the feasibility of revising IROC-H’s acceptance criteria for credentialing.

2. MATERIALS AND METHODS

IROC-H’s IMRT H&N phantom holds six double-loaded TLDs within two target structures (Fig. 1). Radiochromic films are placed in a sagittal plane through the primary planning target volume (PTV) and in an axial plane through both PTVs and the organ at risk. The specific design and features of the H&N phantom have been described previously.^2,4,5

FIG. 1.

IROC-H H&N phantom with block dosimetry insert.

Participating institutions are instructed to treat the phantom as they would a patient; this means the institution images the phantom, develops a treatment plan according to clinical practice, and treats the phantom. A dose of 6.6 Gy is to be administered to at least 95% of the primary PTV and 5.4 Gy is to be delivered to at least 95% of the secondary PTV. The organ at risk is to receive no more than 4.5 Gy.

IROC-H then analyzes the radiochromic films and TLDs as described by Molineu et al.⁴ The TLD-100 capsules (Quantaflux, LLC, Oregonia, OH) are analyzed using the same technique as that used by IROC-H’s mail-out dosimetry service, with a precision of 1%.⁶ The radiochromic films (Gafchromic EBT2, Ashland, Wayne, NJ) are processed according to TG-55 recommendations⁷ and have a localization accuracy of 1 mm and dosimetric uncertainty of 2.6%–3.6%.⁸ For congruence, the planar dose is normalized to the TLD dose in the primary PTV. For acceptance, an irradiation must pass both TLD and film evaluations: all six TLDs must agree within a given percentage of the planned dose for each location, and both axial and sagittal films must meet the specified gamma analysis criteria.

In the current study, a total of 156 phantom irradiations between November 1, 2014 and October 31, 2015 were evaluated. Of those that failed, seven institutions repeated the phantom irradiation and six subsequently passed, and a single institution irradiated three times without passing in the timeframe of this study. All irradiations, including repeats, were included in this analysis. Uncertainties in the analyses, which were used for comparison between alternate criteria, were calculated using a binomial approximation of the variance.

Current acceptance criteria are that the measured TLD dose be within ±7% of the planned absolute dose to the PTVs and that film measurements undergo gamma analysis for ±7% dose and ±4 mm DTA with ≥85% of pixels passing. These criteria, originally developed in conjunction with the National Clinical Trial Network, are based on the results of the first ten institutions to irradiate the H&N phantom. The current criteria were suitable such that 90% of the initial institutions could meet the criteria, and have since remained the standard for which IROC-H evaluates institution performance.² The phantoms considered in this study were re-evaluated using the standard IROC-H workflow but with the following more stringent criteria: (1) 5% TLD and 5%/4 mm, (2) 5% TLD and 5%/3 mm, (3) 4% TLD and 4%/4 mm, and (4) 3% TLD and 3%/3 mm. All gamma analyses were performed by comparing the measured film plane to the corresponding dose plane from the treatment planning system (TPS) and, unless otherwise specified, were performed with an acceptance criterion of at least 85% of pixels passing. Pass rates were also compared with respect to percentage of passing pixels required for gamma analyses. Failure rates for varying criteria were also evaluated with respect to individual film and TLD performance by location in the phantom (e.g., film plane or primary versus secondary PTV location) in order to better elucidate the types of error manifestation.

Failure attributes were qualitatively assessed for the current acceptance criteria and for criteria 1 (5%/4 mm) by visually inspecting the dose profiles of these cases. Reports were characterized by consensus of IROC-H personnel in terms of systematic dosimetric errors (systematic dose shift of the delivered dose distribution by 3% or more on average), setup errors (systematic positional shift of the delivered dose distribution by >3 mm), global but nonsystematic errors (large-scale, nonsystematic deviation of the delivered dose distribution), and errors affecting only a local region that included irradiations in which only one TLD failed (small-scale, nonsystematic dose deviation of the delivered dose distribution). Phantom results were relatively clearly categorized into these four distinct categories.

3. RESULTS

Of the 156 phantom irradiations in this study, 140 (90%) met current acceptance criteria (Table I). Values expressed in Table I are subdivided into the percentages of irradiations that passed both TLD and film criteria and at least one of the individual criteria. These values are shown with standard errors to facilitate comparison between the different criteria. The overall pass rate dropped 13% for criteria 1 (5%/4 mm) and continued to decline when tighter criteria were applied. A majority of participating institutions (63%) were still able to meet the 4%/4 mm acceptance criteria, for which 109 irradiations passed.

TABLE I.

Institutional percentage pass rates for overall and individual criteria.

Criteria	Overall passa	TLD pass	Gamma passb
7% TLD, 7%/4 mm	90 ± 2	93 ± 2	92 ± 2
5% TLD, 5%/4 mm	77 ± 3	80 ± 3	86 ± 3
5% TLD, 5%/3 mm	70 ± 4	80 ± 3	75 ± 3
4% TLD, 4%/4 mm	63 ± 4	67 ± 4	79 ± 3
3% TLD, 3%/3 mm	37 ± 4	49 ± 4	48 ± 4

^aOverall pass rate describes the ratio of institutions that passed both gamma index and TLD criteria, thus fulfilling the requirements for acceptance.

^bGamma criterion requires ≥85% of pixels pass at the specified criteria.

Most (>44% for all criteria) of the failing irradiations resulted from both TLD and film (Table II). Because the film dose is normalized by the TLD dose, this relationship is not surprising—if the TLD disagreed by >7%, the film would typically fail the 7%/4 mm gamma criteria when normalized to the measured TLD dose. The TLD point dosimeters identified 66%–90% of the failing cases for the changing criteria (found as the total failing cases identified by TLD, including irradiations failing both criteria). Gamma criteria did identify 55%–82% of failing cases, with the number of failures being higher when the DTA requirement was tightened to 3 mm.

TABLE II.

Comparison of failures by test.

Criteria	Both gamma and TLD (%)	Only TLD (%)	Only gamma (%)
7% TLD, 7%/4 mm	7 (44)	4 (25)	5 (31)
5% TLD, 5%/4 mm	17 (47)	14 (39)	5 (14)
5% TLD, 5%/3 mm	23 (49)	8 (17)	16 (34)
4% TLD, 4%/4 mm	26 (45)	26 (45)	6 (10)
3% TLD, 3%/3 mm	61 (62)	18 (18)	20 (20)

A total of 1560 gamma evaluations were calculated for this study (using both axial and sagittal films for each of the alternative criteria) and compared for different percentages of pixels required for acceptance. Increasing the percentage of pixels required yielded decreased acceptance for the tighter criteria (Table III). Most of the institutions that we reviewed passed more than 90% of pixels at criteria 1 (5%/4 mm), and almost half of the institutions reviewed passed 85% pixels at the stringent criteria 4 (3%/3 mm). Pass rates dropped more than 10% when the DTA was restricted from 4 to 3 mm for the 5% criteria.

TABLE III.

Institutional gamma percentage pass rates for increasing percentage of pixels required for acceptance.

Criteria	≥85%	≥90%	≥95%
7%/4 mm	92 ± 2	90 ± 2	82 ± 3
5%/4 mm	86 ± 3	79 ± 3	58 ± 4
5%/3 mm	75 ± 3	62 ± 4	39 ± 4
4%/4 mm	79 ± 3	62 ± 4	42 ± 4
3%/3 mm	48 ± 4	28 ± 4	10 ± 2

Because each gamma analysis required both the axial and sagittal films to pass the phantom, we investigated rates of failure for the individual films. Despite the differences in film orientation, the results for the axial and sagittal films were not statistically different from one another for any of the criteria (p ≥ 0.453, McNemar test), meaning that for a given poor-performing irradiation, it was equally likely for an error to be exhibited in either film. Additionally, most irradiations that did fail the film criteria failed both films. These observations are of interest because for most typical plans with no additional isocenter shift or collimator angle, the axial and sagittal films capture slightly different information: the axial film provides a detailed description of the dose distribution from a single leaf-pair, whereas the sagittal distribution provides less information on a given leaf pair, but includes information on all leaf pairs. It is interesting that neither approach shows a clear benefit over the other.

When TLD results were analyzed by location in the phantom, the majority of failures were related to the primary PTV (Table IV). However, the mean ratios (TLD dose to TPS-predicted dose) for the primary PTV and the secondary PTV TLDs were correlated (r = 0.768), and this relationship was statistically significant (p < 0.001, 2-sided). This means that TLD results were consistent across the two targets, thus signaling a systematic issue.

TABLE IV.

Classification of institution TLD criteria failures by location in the phantom.

Criteria	Total fail (%)	At least PTV (%)	At least 2PTV (%)	Both PTVs (%)
7%	11 (7)	9 (6)	5 (3)	3 (2)
5%	31 (20)	24 (15)	18 (12)	11 (7)
4%	52 (33)	45 (29)	35 (22)	28 (18)
3%	79 (51)	68 (44)	61 (39)	50 (32)

Note: PTV = planning target volume; 2PTV = secondary PTV.

Qualitative observations of the planar dose distributions highlight predominant attributes found from failing institutions (Table V). Data are presented only for 7%/4 mm and 5%/4 mm since these two criteria were found to be most practically achievable. An overwhelming majority of the failures at either criterion were determined to be due to systematic errors, with a greater extent being systematically low compared with the dose profile predicted by the TPS. That is, these profiles had the correct shape and position, but the magnitude of the dose was incorrect. Setup/positioning errors occurred in only 2 and 4 cases for the 7%/4 mm and 5%/4 mm criteria we analyzed, respectively (11%–13% of failing cases). These profile shifts were measured to be between 3 and 5 mm from the expected distribution. When further explored, it was determined that the four phantom irradiations designated as setup errors at the 5%/4 mm criteria were all cases that failed the gamma criteria only, which consisted of a total of five cases (Table II). These irradiations were still within tolerance for the 5% TLD criterion. That is, almost all of the irradiations that failed only the gamma criteria but not the TLD criteria were setup/positioning errors. The third category of errors — local errors — were not observed at the 7%/4 mm criteria but did constitute a major contribution to failures at the 5%/4 mm criteria, consisting of eight failing cases (22%). Finally, we identified three global, nonsystematic failures as irradiations that either did not follow the TPS-predicted dose profile or exhibited multiple errors. Figures 2 and 3 depict comparisons of film and calculated dose planes exhibiting global and systematic errors, respectively.

TABLE V.

Consensus description of causes of failure.

Attribute	7% TLD, 7%/4 mm (%)	5% TLD, 5%/4 mm (%)
Systematic	11 (69)	21 (58)
Low	9 (56)	17 (47)
High	2 (13)	4 (11)
Setup/position	2 (13)	4 (11)
Locala	0 (0)	8 (22)
Globalb	3 (19)	3 (8)

^aLocal errors included tests in which local phenomena caused only one TLD to fail, typically by a small margin (∼1%).

^bGlobal errors were those that resulted from multiple errors or irregular dose distributions.

FIG. 2.

Sample global error. These plots of (a) a right–left profile taken from an axial film and (b) a superior–inferior profile taken from a sagittal film are from an institution trial deemed to exhibit global nonsystematic errors in treatment planning and delivery. This plan has poor dose uniformity through both the primary PTV and the secondary PTV. No additional error patterns (rotation, shifting, etc.) were evidenced in this irradiation. PTV = planning target volume.

FIG. 3.

Sample systematic error. These plots of (a) a superior–inferior profile taken from a sagittal film and (b) an anterior–posterior profile taken from an axial film are from an institution trial deemed to exhibit systematic errors in treatment planning and delivery, as there is a systematic difference of ∼8% between the measured and calculated dose distributions in the high-dose region. Such errors constitute the majority of errors observed in failing plans. PTV = planning target volume.

It is critical that tests such as the phantom irradiation test have a negligible probability of failing due to random noise in the dosimeters. This is a particular concern for those irradiations presenting local errors, which could potentially be manifestations of such an effect. Based on the known TLD variation⁸ and assuming that phantom results are normally distributed (and that TLDs are not correlated), we calculated the statistical probability that at least one TLD result would fall outside of either the 7% or 5% criterion. The worst-case probability for a random failure of a perfectly delivered treatment (the ratio of IROC-H measured dose to TPS-planned dose being unity) was approximately 0.002% and 0.4% for the 7% and 5% criteria, respectively. Should the average measured dose be 2% different from the calculated dose, the probability of at least one random TLD failure would not exceed 9% for the 5% criterion. This rate is less than half the proportion of failures classified as local phenomena in Table V (22%), meaning these events cannot be described by dosimeter variability alone, and indicating a genuine difference between the measured and calculated doses. Furthermore, because this probability is conservative, we expect the true rate of noise-induced failures should be less than that reported here.

Nearly 70% (109/156) of the irradiations expressed a mean TLD ratio less than 1, meaning that the measured dose was less than the institution treatment plan prediction. The overall mean TLD ratio for all phantom irradiations was determined to be 0.98 (s.d. 0.03), and this mean was statistically different from unity (p < 0.001, t-test). Given that systematic biases have not been demonstrated in the TLD protocol,⁹ these data indicate that the TPSs, in general, overestimated the dose that was actually delivered to the target. This interpretation is consistent with the observation that most phantom failures were classified as systematically low (Table V).

4. DISCUSSION

Considerable improvements have been made in the pass rate for the IMRT H&N phantom; yet, phantom irradiation failures still occur with current acceptance criteria. Therefore, it is imperative to identify and describe the causes of these failures so that corrective actions can be taken. Our study highlights that, in general, systematic differences exist between TPS calculations and the actual delivered dose, and that other errors are less prominent causes of credentialing failure. Tightening the passing criteria facilitated visualization of errors that could not be discerned at current criteria, especially local phenomena and less extreme cases of systematic discrepancies. Together these results assist in the diagnosis of potential causes of suboptimal performance in phantom credentialing.

The prominent cause of phantom failures was likely dosimetric, and inspection of failing cases likewise indicated such inaccuracy, especially underdosing (Table V). Dosimetric errors may result from inaccuracies such as those in beam-modeling within the TPS software, inadequate commissioning for IMRT, or incorrect output factors (or the corresponding calibration). These faults have been identified previously in credentialing for clinical trials,^1,2 especially in accurate TPS modeling of the relative output of IMRT segments, where the multileaf collimator defines a small opening in a larger jaw field.^10,11 Consistently and systematically, the output from such an “IMRT segment” is overestimated by the TPS. This would be expected to cause an overestimation of the delivered dose from an IMRT treatment, consistent with the trends seen in this study.

One of the major challenges of the IMRT phantom program is that the phantom irradiation serves as an end-to-end quality assurance (QA) check. As such, it is limited in its ability to identify specific contributions to irradiation failure. Because the current work can only conjecture the scope of possible errors, further work is necessary to develop methodologies to properly diagnose the source of these dosimetric errors.

For film measurements, Table III demonstrates that the current rate of agreement displayed in planar dose distributions is reasonable, that is, more than half of all irradiations passed even under the stringent ≥95% requirement for the 5%/4 mm criteria. This demonstrates the increasing ability of institutions to achieve accurate setup and to deliver dose to the regions of interest, which may be attributable to better image guidance. We found 85% to be sensitive enough to intercept gross errors without excessive specificity. To increase the passing pixel requirement beyond 90% may be feasible if current criteria are maintained, given that both the QA standards and possible clinical trial participation must be considered in the development of reasonable criteria.

While in many cases the TLD dose measurement can accurately determine errors in phantom irradiation, the film data are useful in the diagnosis of gross setup errors. As discussed previously, the cases in which setup error was presumed to be the mode of failure, film analyses indicated such. For these instances, film analysis was the sole failing criterion: the setup errors could not be accurately detected by TLD. This suggests that, for the criteria investigated here, when an accurate setup was achieved, the TLDs captured the extreme majority of the true irradiation conditions. This result is initially surprising (considering the seemingly large tolerances of both TLD and film criteria currently) but is actually consistent with IMRT QA results that have found point dosimeters to do a very good job of capturing the entire picture of an IMRT treatment and expose dosimetric errors that may not be discovered through planar dosimetry.^12–15 This result was also simulated in work by McVicker et al., where gamma analyses comparing plans calculated with and without commissioning errors could not reveal clinically severe effects, even with stringent 2%/2 mm criteria.³ These data suggest that film dosimetry may have limited applicability in error detection for IMRT.

We expect that the introduction of new acceptance criteria may further improve the quality of results found from clinical trials. However, the decision to implement new criteria must also consider its impact on the cohort of institutions that would be permitted to participate (and consequently the number of patients who can be enrolled) in clinical trials. Although the credentialing process is meant to reduce the variability between institutions, the goal of study quality must not supersede the necessity for adequate participation.

Of the criteria considered in the current study, criteria 1 (5%/4 mm) appears to be the most plausible alternative due to its increased sensitivity relative to current standards, as well as its projected pass rate. Several visually detectable errors were identified at the ±5% criterion that could not be distinguished with an action level of ±7%. According to Molineu et al.² the current acceptance criteria had been criticized previously for being too lax, but adjusting the criteria was considered impractical because so many institutions still could not meet the standard. Now that more than 90% success for the H&N phantom has been achieved in the past few years, consideration of tighter acceptance criteria may be plausible. Here we demonstrate that 77% of all irradiations tested could meet criteria 1. In addition, criteria 1 would better adhere to the clinical threshold of ±5% originally proposed by ICRU Report No. 24,¹⁶ which is closer to the action level that is commonly accepted in clinical use.

Another concern in considering the adoption of a new acceptance standard is the possibility of noise-induced failure, especially as the criteria approach the uncertainty limit for current measurement techniques (e.g., film and TLD). As previously determined, the rates of failure due to variability for the 7% and 5% criteria are very low compared with the observed proportions. Likewise, the response of EBT2 film is well-understood and should yield dose uncertainty of no more than 2.8%, considering all uncertainties.¹⁷ Together, these findings suggest that the dosimetric differences observed were more likely a result of poor TPS calculations than of random variation, and differences should be discernable at a 5% action level. However, more work is necessary to determine the causes linked to poor performance, especially considering the degree of dosimetric errors observed.

5. CONCLUSIONS

We have investigated the effect of tightening criteria on pass rates and have found ±5% TLD and 5%/4 mm to be both theoretically and practically achievable, with 77% of institutions currently able to meet these criteria. We have also explored results from the different dosimeters and regions of interest. According to our observations, approximately half of all failures seen in credentialing tests are due to systematic underdosing, and these inaccuracies are identified primarily by TLD measurements. Other failure modes contributed to a lesser degree. While film analysis was less effective at detecting dosimetric discrepancies, it could diagnose gross setup errors. Local phenomena could only be determined at the 5% criteria, and global errors were uncommon.

Although errors are still widely present in radiotherapy credentialing and warrant attention, the extent of errors at a >7% level is slowly declining. Tightening the criteria has the potential to increase the quality of clinical trials and to more closely reflect the precision to which institutions can currently perform. This study also highlighted that further work is warranted in identifying and resolving the continuing dosimetric errors in treatment planning.

CONFLICT OF INTEREST DISCLOSURE

The authors have no COI to report.