Interobserver Variability in Radiotherapy Plan Output: Results of a Single-Institution Study

Abstract

Purpose

To investigate the sources of variability in radiotherapy treatment plan output between planners within a single institution.

Materials/Methods

40 treatment planners across 5 campuses of the same institution created a plan on copies of the same thoracic esophagus patient CT and structure set. Plans were scored and ranked based on the planner’s adherence to ordered list of target dose coverage and normal tissue evaluation criteria. A runs test was used to identify whether any of the studied planner qualities influenced the ranking. Spearman’s rank correlation was used to investigate whether plan score correlated with years of experience or planned MU.

Results

The distribution of scores, ranging from 80.24 to 135.89, was negatively skewed (mean = 128.7, median = 131.5). No statistically significant relationship between plan score and campus (p=0.193), job title (p=0.174), previous outside experience (p=0.611), or number of gantry angles (p=0.156) exists. No statistical correlation between plan score and MU or years of experience was found.

Conclusion

Despite clear and established critical organ dose criteria and well documented planning guidelines, planning variation still occurs, even among members of the same institution. As plan consistency does not seem to significantly correlate with experience, career path, or campus, investigation into alternate methods beyond additional education and training to reduce this variation, such as knowledge based planning or advanced optimization techniques, is necessary.

INTRODUCTION

Radiotherapy treatment plan quality and consistency is of interest to all members of the radiation oncology community. Suboptimal treatment plans could influence the results of national and international protocols[1, 2] and for individual patients could be associated with unnecessary morbidity and mortality as well as suboptimal tumor control[3, 4]. There are two main sources of variability in external beam radiotherapy (EBRT): delineation of targets and organs-at-risk (OARs) and in the planning process. For the latter, items such as beam arrangement, optimization parameters, and relative tradeoffs in target coverage and OAR sparing are all sources of variability.

Most of the plan quality studies available in the literature are inter-institutional in nature. Williams et al. described a study where 3 Australian radiotherapy centers, each with a different treatment planning system (TPS), created plans for the same segmented head and neck (HN) CT dataset. They found large variation between each center’s interpretation of the prescription and application of planning priorities[8]. The QUASIMODO IMRT verification project group in Europe had 11 plans created at 9 different centers on an anthropomorphic pelvic phantom[6]. These centers employed different combinations of TPS, IMRT delivery techniques, beam energies, and linear accelerators and observed a large variation in number of monitor units, treatment time, and acceptability of the submitted dose distributions. Nelms et al. analyzed 125 plans voluntarily submitted from centers throughout the world on the same segmented prostate CT scan[7]. They found that plans ranked in terms of a plan quality metric (PQM) had wide variability which exhibited little to negligible correlation with TPS, IMRT versus VMAT, planner education, certification status, years of experience, self-reported confidence, number of beam angles, or total monitor units.

One intra-institutional study of plan quality was published by Batumalai et al., where 6 dosimetrists created an IMRT plan for the same segmented HN CT dataset[5]. Despite the small sample size, the authors concluded that increased planning experience results in better quality plans with better delivery efficiency.

We present an analysis of treatment plan quality in an intra-institutional setting. Our sample size (40) is much larger than Batumalai, allowing us to utilize a scoring methodology and statistical analysis similar to that of Nelms[7]. In the context of conflicting results in the literature regarding planner experience, we are interested in determining whether plan quality correlates with plan complexity or the experience, career path, or campus for our planner population. Participation in our study was mandatory, an important distinction from the voluntary nature of Nelms, removing the potential bias of self-selected data. Further, we only use 1 TPS to design the plan for 1 particular linac, treatment technique, and beam energy, removing the potentially confounding variables found in the inter-institutional studies.

METHODS AND MATERIALS

Forty treatment planners, a mix of physicists and dosimetrists, across 5 campuses of a single institution were required to create sliding window IMRT treatment plans for an anonymized, segmented, planning CT scan for a patient who had been treated with radiotherapy at our institution. The patient had limited stage small cell carcinoma of the distal esophagus and was treated with a simultaneous integrated boost (SIB) to 4500 cGy and 5000 cGy in 25 fractions with 6 MV x-rays. This exercise was performed in the context of a planning competency demonstration, where each planner was required to individually confirm that they are able to appropriately and efficiently use the TPS. Esophagus, featuring a target surrounded by a number of OAR’s, which at our institution have detailed, clearly prioritized, and consistently used planning goals (Table 1) was chosen as the site of disease. This particular patient (Figure 1) was chosen because it was an example of SIB planning and the target/OAR geometry represented a moderate level of difficulty. Each planner had 1 week to create the plan. The plans were submitted over a 13 month period, from January 2014 to January 2015. All planners used the same treatment planning system, Eclipse® (Varian Medical Systems, Palo Alto, CA), and optimization engine (Varian Dose Volume Optimizer), although due to an upgrade during that time period 9 plans were completed in version 10 and 31 in version 11. Further, due to changing practice over this time 34 plans were completed with a dose grid resolution of 0.2 cm and 6 with 0.125 cm.

Table 1.

Institutional evaluation criteria for esophageal IMRT planning, listed in order from highest to lowest priority. Columns 3-5 represent the average and range of values over the 40 plans created on the dataset.

Organ / Criteria	Clinical Goal	Average ± Std Dev	Minimum	Maximum
Lungs V20Gy (%)	≤20	19.21±2.54	15.93	32.82
Lungs V30Gy (%)	≤15	6.29±0.92	4.69	10.41
Lungs V40Gy (%)	≤10	2.72±0.51	0.92	4.50
Cord Max Dose (cGy)	≤4000-4500	3977.08±192.20	3569	4376
PTVLow Dose D95% (cGy)	≥4500	4537.68±120.79	4415	5047
PTVLow Dose Max Dose (cGy)	≤5300	5262.58±160.13	5104	5838
PTVHigh Dose D95% (cGy)	≥5000	5064.20±141.80	4897	5637
PTVHigh Dose Max Dose (cGy)	≤5600	5355.08±155.93	5169	5909
Heart V30Gy (%)	≤20-30	20.35±4.00	15.48	33.46
Heart Mean Dose (cGy)	≤3000	2290.95±135.76	2051	2664
Lungs V10Gy (%)	≤35-65	56.57±4.36	47.40	71.68

Figure 1.

CT scan used for this study. PTV_{Low Dose} (bold contour) and PTV_{High Dose} (thin contour) are superimposed on the image.

All planners had access to the same institutional procedures for esophageal IMRT planning, which include the recommended number and arrangement of gantry angles as well as a template containing our default starting point dose/volume (D/V) optimization objectives and corresponding weights. However, consistent with our clinical practice, the final decisions about beam geometry and population of the objective function are left to the discretion of the individual planner. D/V objectives are typically adjusted as the optimization process is on-going, and relative weights are modified if the planner judges that the optimizer does not respond sufficiently to a D/V adjustment. All planners also had access to the same target coverage and critical organ dose criteria, which included a detailed and prioritized ranking of planning goals (Table 1). Neither the procedures nor the planning goals were modified over the course of the 13 months.

Upon submission of the plan, the dose volume histogram (DVH) data was extracted from Eclipse^® and analyzed in Excel (Microsoft Corp., Redmond, WA). A score was assigned to each plan using a point system similar to Nelms’ PQM (Figure 2)[7]. Higher priority clinical goals, the V20_Gy, V30_Gy, and V40_Gy of the lungs for example, have a greater number of points at stake. While a few points may be awarded for increased sparing of high priority OARs, the number of points is less than that lost for exceeding a lower priority OAR goal, ensuring that a planner will not be unduly rewarded for neglecting lower priority OARs. There is a penalty for both PTV under-coverage and over-coverage. A plan’s score is the sum of the points acquired at each of the individual evaluation points in Figure 2. A total of 148 points were available.

Figure 2.

Graphical illustration of point system used to assign scores to each submitted plan based on how well it met the various clinical criteria for this anatomical site.

The following binary information was recorded for each planner: job title (physicist or dosimetrist), campus (main or regional), treatment planning as their primary job responsibility (yes or no), and prior Eclipse experience at another institution (yes or no). Also, the number of gantry angles (=5 or >5), grid size (0.125 cm or 0.2 cm), and Analytical Anisotropic Algorithm (AAA) dose calculation algorithm version (10.0.28 or 11.0.31) were recorded. Statistical runs tests[9] were used to determine whether the sequence of each of these classifications when ordered from lowest to highest plan score, was non-random, defined as having a p-value less than or equal to 0.05 on a two-tailed test. Also recorded was: total plan monitor units (MU), date of submission, and years of experience at time of submission. Scatter plots of plan score versus each of these categories were plotted and Spearman’s rank correlation coefficients, and associated two tailed p-values, were calculated in MATLAB (Mathworks, Natick, MA). Finally, a scatter plot of total plan MU versus planner experience was plotted and the Spearman’s rank correlation coefficient was calculated.

RESULTS

As seen in Table 2 and Figure 3, the plan score distribution was negatively skewed and ranged from 80.24 to 135.89. The 2 scores below 100 represented plans submitted for a prescription dose of 5040 cGy rather than 4500 cGy, which is a more typical prescribed dose for IMRT esophagus at our institution. The score of 110.5 was planned to the correct prescription dose but had a lung V20_Gy of 32.82%, greatly exceeding the clinical goal of 20%. The remainder of the plans had scores bunched over an 11.5 point range.

Table 2.

Each row consists of the demographic information for a particular planner and the score that they achieved, ordered from lowest to highest score.

Plan Score	Physicist vs. dosimetrist	Main vs Regional	> or = 5 gantry angles	Grid Size (cm)	Algorithm	Prior Eclipse Experience	Planning is Primary role
80.24	P	R	E	0.2	11.0.31	N	N
88.52	D	R	E	0.2	11.0.31	Y	Y
110.56	P	M	E	0.2	11.0.31	N	Y
124.48	D	R	E	0.2	10.0.28	N	Y
126.27	P	M	G	0.125	11.0.31	Y	Y
126.83	P	M	E	0.2	11.0.31	N	Y
127.43	D	R	G	0.125	11.0.31	Y	Y
128.04	D	M	G	0.2	11.0.31	Y	Y
128.54	P	M	E	0.2	10.0.28	Y	Y
128.64	P	M	E	0.2	11.0.31	Y	Y
129.09	P	M	E	0.2	10.0.28	N	Y
129.14	P	M	E	0.2	10.0.28	N	Y
129.58	P	R	E	0.2	11.0.31	N	Y
130.03	D	M	E	0.2	10.0.28	N	Y
130.56	D	M	E	0.2	10.0.28	Y	Y
130.60	P	R	E	0.20	11.0.31	N	N
130.64	D	R	E	0.2	11.0.31	N	Y
130.69	P	R	E	0.2	11.0.31	N	Y
131.31	D	M	E	0.2	11.0.31	Y	Y
131.48	P	R	E	0.2	11.0.31	N	Y
131.60	D	M	E	0.2	11.0.31	N	Y
132.79	P	R	E	0.2	11.0.31	N	Y
133.00	D	M	E	0.2	11.0.31	Y	Y
133.06	D	R	E	0.2	11.0.31	Y	Y
133.06	D	M	E	0.125	11.0.31	Y	N
133.09	P	R	E	0.2	11.0.31	N	N
133.19	P	R	G	0.2	11.0.31	N	N
133.28	D	R	G	0.125	11.0.31	Y	Y
133.36	P	M	E	0.2	10.0.28	Y	Y
133.50	D	R	E	0.2	11.0.31	N	Y
133.69	P	R	E	0.2	11.0.31	N	Y
133.85	P	M	E	0.2	11.0.31	N	Y
134.01	D	R	E	0.2	11.0.31	N	Y
134.04	P	R	E	0.2	11.0.31	N	N
134.06	P	M	E	0.2	10.0.28	Y	Y
134.67	P	M	E	0.2	10.0.28	Y	Y
134.70	D	R	E	0.2	11.0.31	N	Y
134.91	D	M	G	0.125	11.0.31	Y	Y
135.31	D	R	E	0.125	11.0.31	Y	Y
135.89	P	R	E	0.2	11.0.31	N	Y
Runs test p-value	0.174	0.193	0.156	0.897	0.365	0.611	0.039

The runs test p-value indicates whether the resulting pattern in each column is sufficiently non-random (p ≤ 0.05).

Note: P = Physicist, D = Dosimetrist, M = Main, R = Regional, E = equal to 5 gantry angles, G = >5 gantry angles, N = No, Y = Yes.

Figure 3.

Distribution of plan scores for all 40 submitted plans

From the runs tests shown in Table 2, there is no statistically significant relationship between plan score and job title, campus, number of gantry angles utilized, grid size, AAA version, or whether they had prior experience planning with Eclipse. There was a statistically significant pattern in the relationship between score and planning as a primary job responsibility but it does not appear that the pattern has any clinically discernible or actionable result.

For the purposes of identifying possible correlations, the two lowest scores have been designated as outliers and removed from the scatter plot analysis. Those two scores were the result of prescription misinterpretation which would not be related to MU, date of submission, or planner years of experience. As seen in Figure 4, there are no statistically significant Spearman rank correlations for plan score versus MU, date of submission, or years of experience, nor for MU versus years of experience.

Figure 4.

Scatter plots of plan score versus (a) total MU, (b) date of submission, (c) self reported years of experience and (d) MU versus self reported years of experience. For the Spearman rank correlation calculation the two plans created for the incorrect prescription dose, marked as “x’s”, were omitted as outliers.

DISCUSSION

Most traditional dose-volume based optimizers rely on a trial-and-error optimization process where the planner iteratively adjusts parameters until an appropriate dose distribution is constructed, tying the planner’s judgment and ability to the final results. If an actionable correlation between general planner qualities or demographics can be identified, corresponding changes in the management of personnel can be made to improve plan output for poorly performing cohorts. For example, if a particular geographical campus is identified as correlating with poor plan quality, an action plan to improve that quality could be to rotate planners periodically through different campuses. On the other hand, if plan quality is truly just a function of individual differences and abilities, modifications to the planning process itself may be more important, for example, utilizing an optimization algorithm that requires less user interaction.

The plan scoring approach that we adopted followed methodology originally presented by Nelms[7]. While it is not the only conceivable distribution of points, it is based on a well defined prioritization of planning goals and represents an objective method of comparing different plans on the same anatomy to one another. The fact that a major planning goal violation had an obvious impact on the plan’s score is an indication that this approach to quantifying plan quality may be beneficial.

Since the data was collected over the course of 13 months, there could be some concern that subtle changes in practice may contribute to differences in plan quality. A software upgrade and a smaller default grid resolution were introduced over this time. However, our results indicate that these possible confounding variables are not individually correlated with the plan scores. Similarly, there was no statistical relationship between the date that the plan was submitted and the plan score. This indicates that it is valid to compare the plans despite the elapsed time.

Our findings are more similar to those of Nelms[7] than Batumalai[5] and indicate that additional professional training, i.e. physicist rather than dosimetrist, would not be expected to yield better quality plans. Similarly, beyond a nominal amount of experience, there does not appear to be any obvious advantage to having many years on the job or to having used the same planning system at a previous institution. Therefore, continuing education and training need to be supplemented by engineered changes to the planning process itself[10] in order to further improve plan quality and consistency. Those changes could include the adoption of advanced optimization techniques, such as Prioritized Optimization (PO)[11, 12] and Multi-Criteria Optimization (MCO)[13, 14], or the use of Knowledge Based Planning (KBP) approaches[15-18]. The average planning result for this study (Table 1) closely matched our institutional norms, indicating that a KBP prediction could potentially be used to effectively highlight the planner outliers.

Increased plan complexity does not translate into increased plan quality. The highest MU was 2.2 times greater than the lowest MU, but both were in the middle of the spectrum of plan scores. The average MU for all plans was 864.63. The second through fifth highest scoring plans delivered less than the average, two of them greater than one standard deviation below. Four of the top 5 scoring plans were limited to 5 gantry angles. These results are encouraging in that less complex deliveries are more likely to be robust to minor delivery errors and treatment uncertainties[19].

Ninety percent of our plans had scores within 10 points of one another, indicating that the plan output of our institution is generally consistent. However, the presence of the three lowest scores is an indication that there is a risk that sub-optimal plans will be produced from time to time. The plan with a lungs V20Gy of 32.82% could have a greater risk for radiation pneumonitis[20-22]. It is unclear whether a prescription dose of 4500 cGy versus 5040 cGy would result in different levels of tumor control[23, 24], but the two plans created for 5040 cGy had the highest heart V30Gy and mean doses, which could increase heart toxicity[25]. One must ensure that the planning quality control process, e.g. physician plan review and physicist chart check, is designed to catch problems such as prescription misinterpretation and planning goal violation before they would reach a patient.

This study focused on one anatomical treatment site, esophagus. Future investigation would be helpful in determining whether the lack of correlations identified here would persist over the range of anatomical sites.

Quantifying and analyzing plan quality in this manner would be a beneficial exercise for any institution looking to increase the consistency of their planning results. Further, it should be an essential aspect of the quality assurance in multi-institutional trials where judgments about the efficacy of experimental treatment interventions should not be influenced by variability introduced from dose distributions of differing quality levels.

CONCLUSIONS

We found no correlation between plan quality and job title, years of experience, campus, or plan complexity. This indicates that although basic training and continuing education have a role in ensuring good plan quality, alternate optimization approaches, such as PO and MCO, or evaluation approaches, such as KBP, warrant further investigation as potential tools to reduce the role of individual differences in planner ability on an institution’s plan quality output.