Toward Systematic Assessment and Improvement of Radiotherapy Plan Quality of Cooperative Group Trial Submissions: A Report from the Children’s Oncology Group

Abstract

Purpose:

This study evaluates the quality of plans used for treatment of patients on the Children’s Oncology Group (COG) study, ACNS1123. Plan quality is quantified based on a scoring system specific to the protocol. In this way, the distribution of plan quality scores is determined which can be used to identify plan quality issues for this study and for future plan quality improvement.

Methods and Materials:

ACNS1123 stratum 1 patients (70) were evaluated. This included 50 photon and 20 proton plans. DICOM structure and dose data was obtained from the COG. A commercially available plan quality scoring algorithm was used to create a scoring system we designed using the protocol dosimetric requirements. The whole ventricle and boost PTVs could earn a maximum of 70 points while the OARs could earn 30 points (total maximum score of 100 points). The scoring algorithm adjusted scores based on the difficulty in achieving the structure dose requirements which depended on the proximity of the PTVs and the dose gradients achieved relative to the organs at risk. The distribution of plan scores was used to determine the mean, median, and range of scores.

Results:

The median adjusted plan quality scores for the 20 proton and 50 photon plans was 83.3 and 86.9, respectively. The range of adjusted scores (maximum-minimum) was 50 points. The average score adjustment was 7.4 points. Photon and proton plans performed almost equally. Average plan quality by individual structure revealed that the brainstem, PTV boost, and cochlea lost the most points.

Conclusions:

This report is the first to systematically analyze overall radiation therapy plan quality scores for an entire cohort of patients treated in a cooperative group clinical trial. The methodology demonstrated a large variation in plan quality in this trial Future clinical trials could potentially use this method to reduce plan quality variability which may improve outcomes.

Introduction

Radiation therapy dosimetric plan quality is generally considered to be a measure of how well the plan maximizes the therapeutic ratio. Plan quality can be highly variable even for the same disease and tumor site within the same institution and even more so across institutions^1–4. In the setting of a clinical trial, it is critical that every patient enrolled to receive radiation therapy is planned and treated per the guidelines provided in the protocol. The dosimetric guidelines for organs-at-risk are written based on generally accepted normal organ tolerances. Target dose guidelines are typically standardized to the desired prescribed dose with a 5-10% margin for both dose and volumetric coverage. The clinical trial quality assurance (QA) group receives the DICOM RT data for each patient, imports those data into their treatment review software, and determines whether each plan follows the protocol dosimetric guidelines. This is generally a manual and time-consuming process. This review can be a rapid pretreatment review where plan approval must be given before the start of treatment, or it can be at some point after treatment. Participants in the trial may design their plan to meet the dosimetric guidelines or they may go beyond these requirements to further improve the quality of the plan even though it is not required. Plan quality depends on the experience of the planner, time constraints, and the complexity of the case. There is evidence that patients treated per protocol guidelines have better outcomes than those treated with protocol deviations⁵. Theoretically, outcomes should be better when patients are treated with high quality plans, where target coverage is maximal and OAR sparing is better than the protocol requirement.

This study aims to quantitatively characterize the RT plan quality of patients enrolled in a recently closed Children’s Oncology Group (COG) intracranial germ cell brain tumor trial, ACNS1123 (Phase 2 Trial of Response-Based Radiation Therapy for Patients with Localized Central Nervous System Germ Cell Tumors (CNS GCT))⁶. This study was chosen because of the complexity of the target geometry, the presence of a variety of OARs, the recent completion, and the relatively good accrual. Such an analysis to determine the plan quality heterogeneity for a clinical trial has not been previously published to our knowledge. This plan quality analysis will inform the COG Radiation Oncology Discipline Committee regarding the opportunities for plan quality improvement for future trials and could potentially be used as an upfront mechanism for data submission evaluation for trials requiring rapid plan evaluation (before or within 3 days of start of RT), and to “crowd-source” plan quality standards for subsequent participants in the same study based on the first 20, for example.

Methods and Materials

Patients

ACNS1123 (NCT01602666) had two stratums; stratum 1 was for patients with central nervous system nongerminomatous germ cell tumors who had a complete response to induction chemotherapy and required 30.6 Gy to the whole ventricles plus a 23.4 Gy boost to the primary site for a total of 54 Gy; stratum 2 was for pure germinoma patients who received induction chemotherapy followed by response based radiotherapy and required 18 Gy or 24 Gy to the whole ventricles with a boost of 12 Gy to the primary site for a total dose of 30 to 36 Gy. The protocol was open from 2012 to 2018 and enrolled a total of 262 patients. After IRB approvals and signing of data use agreements with the COG and the institutions of two authors (AJO and MG), the quality assurance center for COG trials, IROC Rhode Island, provided the DICOM RT image, plan, structure and dose data for 194 patients enrolled on the trial for which these data were available. Stratum 1 was chosen for plan quality evaluation over stratum 2 due to the higher dose prescription which would be more challenging to plan considering the various OAR tolerances. One hundred eleven patients were enrolled on stratum 1. Four were found ineligible and 37 did not receive radiotherapy, leaving 70 patients who had a complete response to induction chemotherapy, were eligible for reduced radiotherapy on stratum 1, and who actually received radiotherapy. Of note these patients were treated with either photon or proton therapy. Also, the whole ventricle volume can be a challenging target to contour; the radiation oncologists treating patients on these trials were referred to a published atlas (https://www.qarc.org/cog/ACNS1123_Atlas.pdf) which was developed from a consensus project. Two plans, one for the whole ventricle and one for the local boost, were summed to make the plan evaluated in this study.

Plan quality scoring method

PlanIQ software (Sun Nuclear Corporation, Melbourne, Florida) was used to evaluate plan quality. The DICOM RT structure sets and dose grids were imported into PlanIQ for each patient. After DICOM data were imported, DVHs were re-calculated using a high-accuracy algorithm⁷.

Within this software, a plan quality scoring template was created which specified a number of points to allocate on a sliding scale to each target and OAR depending on specified dose limits⁴. The sum of all scores over all the metrics yields a total Plan Quality Metric (PQM) score.

A total of 100 points can be awarded to a plan, with 70 points total coming from the two targets (whole ventricles and boost) and 30 points from the 9 OARs (right and left cochlea, brainstem, spinal canal, and right and left eyes, hypothalamus, optic chiasm and pituitary) (table 1). This point scoring system for targets and OARs were assigned based on a consensus among the physician and physicist authors, but certainly different scoring rules could be applied which could be equally valid.

Table 1.

Plan Quality scoring algorithm for targets and OARs. The total number of possible points and the dose or % required are shown along with the dose or percent volume range over which points are assigned between 0 and the maximum number of points. The total possible points is 100.

Metric name	Metric Description	Maximum Score (points)	Dose or Percent Range (Gy)
Dv Boost	Dose covering 99.5% of boost PTV	25 for ≥ 54Gy	> 48.6Gy [≥54Gy]
Vd Boost	Volume % of the boost PTV covered by 59.4Gy (110% of Rx)	10 for ≤ 5%	< 15% [≤ 5%]
Dv WVPTV	Dose covering 99.5% of Whole Ventricle PTV	20 for ≥ 30.6Gy	> 27.54Gy [≥ 30.6Gy]
Vd WVPTV	Volume % of the Whole Ventricle PTV covered by 59.4Gy (110% of Rx)	15 for ≤ 5%	< 15% [≤ 5%]
Dv Cochlea_R	Dose covering 50% of Cochlea_R	3.33 for ≤ 20Gy	< 30.1Gy [≤ 20Gy]
Dv Cochlea_L	Dose covering 50% of Cochlea_L	3.33 for ≤ 20Gy	< 30.1Gy [≤ 20Gy]
Dv Brainstem	Dose covering 0.1cc of Brainstem	7 for ≤ 50Gy	< 55Gy [≤ 50Gy]
Dv Spinal canal	Dose covering 0.1cc of Spinal canal	2 for ≤ 50Gy	< 54.01Gy [≤ 50Gy]
Dv Eye_R	Dose covering 0.1cc of Eye_R	2 for ≤ 25Gy	< 45.01 Gy [≤ 25Gy]
Dv Eye_L	Dose covering 0.1cc of Eye_L	2 for ≤ 25Gy	< 45.01 Gy [≤ 25Gy]
Dv Hypothalamus	Dose covering 0.1cc of Hypothalamus	1.67 for ≤ 20Gy	< 30Gy [≤ 20Gy]
Dv Pituitary	Dose covering 0.1cc of Pituitary	1.67 for ≤ 20Gy	< 30Gy [≤ 20Gy]
Dv Optic Chiasm	Dose covering 0.1cc of Optic Chiasm	7 for ≤ 53Gy	< 55Gy [≤ 53Gy]

The two targets receive points for both prescribed dose coverage (25 points for the boost PTV and 20 points for the whole ventricular PTV) and maximum dose (10 points for the boost PTV and 15 points for the whole ventricular PTV)). The dose coverage limits for the whole ventricle and boost targets were taken from the COG protocol major violation rules. For both targets, maximum points are assessed if the protocol goal doses were achieved, and zero points were assessed for a major protocol violation. We assessed the dose covering 99.5% of each target to avoid issues with assessment at the 100% level due to the shape of the DVH curve. A protocol major deviation was assessed for coverage less than 90% of the prescribed dose and in such instances, we assigned 0 points. Both target maximum dose limits were derived from increasing the percent volume allowed by the minor violation rule by 5% (there was no major violation rule for hot spot). A protocol major deviation was assessed if more than 15% of the target received more than 110% of the prescribed dose, so 0 points was assigned in that scenario. Each OAR was assigned a maximum number of points consistent with the dose limits from the protocol and the relative severity of the possible late effect. The point assignments were chosen as a reasonable measure of the intent of the protocol and the clinical implications of not meeting target dose specifications. In the event a required structure was not contoured and thus the dose could not be determined, those structures were given 0 points. For all targets and OARs, points were assigned from 0 to the maximum number of allowed points by a linear function of dose or volume. For the brainstem for example, the maximum possible number of points was 7 if the dose to 0.1 cc was less than or equal to 50 Gy while 0 points were assigned for a dose of 55 Gy (7 points linearly assigned over a 5 Gy range). For a volume dose of 53 Gy (2 Gy less than the maximum allowed dose), then 2/5 of 7, or 2.8 points were assigned. In cases where the protocol guidelines were met, better plans can be identified by high plan scores. For example, for the boost PTV, the protocol requires at least 48.6 Gy to 99.5%, the case in figure 2b had 49.1 Gy, barely above the limit, earning it just 2.6 points but the case for figure 2a had 54.9 Gy, well above the requirement which earned it all 25 points. Similarly, for the cochlea where both cases got less than the 30 Gy protocol limit, the case in figure 2a received about 14 Gy vs. the case in figure 2b which received about 26 Gy, earning case 2a more points than 2b. All OARs limits except hypothalamus and pituitary were taken from the stated limits in the protocol (table 1). The hypothalamus and pituitary were mentioned in the protocol as OARs but had no protocol dose limits so we chose to apply generally accepted clinical constraints. Of note, germ cell tumors often arise in the pituitary stalk and may involve the pituitary gland or hypothalamus, which was one reason that dose limits were not included.

Figure 2.

Example scoresheet for cases showing where some structure scores are adjusted based on patient geometry and feasibility of achieving the objectives. a) a high scoring plan, b) a lower scoring plan with missing ROIs.

Calculation of benchmark dose and adjusted plan quality metric (APQM)

For different patient datasets, the relative difficulty in meeting goals and achieving a high PQM score can vary due to differences in the anatomic and geometric relationships of targets and OARs. To account for this, an APQM was calculated that uses a proprietary algorithm⁸ to make adjustments that, in effect, levels the playing field for plans that would be harder than the average plan due to proximity of the OARs to the targets and their dose limits compared to the required prescription dose level(s).

The process of computing the APQM starts with calculating a synthetic 3D dose grid (benchmark dose) for each patient. The user specifies the targets and their respective prescription dose levels which are applied to 100% of each target volume, then an algorithm adds photon physics-based, maximum dose gradients around the target surface, and finally a low dose periphery region (figure 1). This benchmark dose is not based on any particular beam arrangement and depends only on the patient anatomy and target(s) and their dose levels as well as the anticipated nominal photon beam energy to be used (e.g., 6 MV in this study). This resulting benchmark dose represents the best-case dose sparing for any individual OAR given 100% target dose coverage(s).

Figure 1.

a) Coronal plane showing the whole ventricular PTV (green) and boost PTV (red). b) Benchmark dose distribution showing the 100% target dose coverage regions (30.6 Gy to the whole ventricular PTV and 54 Gy to the boost PTV) and the predicted maximal dose fall off region (blue) adjacent to normal brain.

Once calculated, the benchmark dose can be used to assess the achievability of DVH-based OAR goals (metrics) for a particular patient (e.g., determine what would be impossible to achieve given 100% target coverage). Additionally, the benchmark dose was used to adjust per-metric scores for the case when metric objectives become very difficult, if not impossible, to achieve in reality. Score adjustment starts with a metric-by-metric process where the benchmark dose’s metric result is itself assessed by the scoring algorithm. If the benchmark dose’s metric result earns less than the maximum points for that metric, it means that it is very difficult or impossible to achieve the maximum achievable points for that metric for that specific patient. In this case, the clinical plan’s score for that metric will be increased by normalizing to the benchmark dose’s score result rather than the score function’s maximum score. This effectively grades “on a curve” due to patient-specific anatomy challenges. In cases where the clinical plan achieves better than the benchmark dose – which can happen only if target dose is sacrificed in the clinical plan – then the adjustment normalizes to the achieved score rather than the benchmark score, thus avoiding relative metric score greater than 100%.

The APQM% score is defined as the absolute PQM score divided by the sum of the normalizing per-metric scores rather than the sum of the score function maximum scores (figure 2). By definition, the APQM% (achieved score / sum of benchmark scores) will always be greater than or equal to the PQM% (achieved score / sum of score function maximum scores). If the APQM% is the same as the PQM%, it means no adjustments were made for difficulty. However, the degree to which the APQM% is higher than the PQM%, indicates the relative difficulty to achieve the plan objectives for that given patient anatomy.

For example, in figure 1b, suppose the brainstem were located adjacent to the 54 Gy volume, and the benchmark predicted lowest possible dose to 0.1 cc was 53 Gy. Our scoring algorithm for the brainstem will assign a maximum of 7 points if that dose is less than or equal to 50 Gy and 0 if equal to or greater than 55 Gy. But 50 Gy is predicted to be impossible by the benchmark dose algorithm due to the proximity of the target and OAR. In this case the scoring algorithm would scale the 7 points possible based on 53 Gy instead of 50 Gy as the lowest possible dose while 55 Gy still earns 0 points (thus 7 points are assigned over a 2 Gy range). If the 0.1 cc dose to the brainstem for this patient were actually 54 Gy, then points would be assigned on a linear scale from 53 to 55 (((55-54)/(55-53))=1/2 of 7), giving 3.5 points instead of 1.4 points if no adjustment were made (((55-54)/(55-50)) =1/5 of 7).

If the benchmarked 0.1 cc brainstem dose were 55 Gy for a given plan, then it is deemed impossible to do better for that plan, but instead of the plan receiving 0 points out of 100 as the scoring algorithm dictates, the total possible plan points (which normalizes the score percentage) would be reduced by the maximum brainstem points (7 points) to 93. In effect this gives points back to the plan on a percentage basis because of the smaller total score denominator. Similarly, for the example in figure 2b, to the degree the cochlea was very close to the PTV, the total plan score would be increased, so the remaining percent score difference between cases in figure 2a and 2b is due to planning procedures rather than case-specific geometric features.

In this way, plans based on more difficult geometries are not penalized relative to easier geometries by virtue of the score being increased with the normalizing adjustment strategy such that the resulting distribution of adjusted scores is more indicative of variation in plan quality.

After each plan PQM and APQM score is calculated, the total scores are added to a score distribution histogram where mean, maximum, and minimum score statistics are calculated (figure 3).

Figure 3.

PQM and APQM histograms.

Results

Twenty patients were treated with protons (double scattered × 5, uniform scanning × 7, pencil beam scanning × 7, unknown × 1), and 50 with photons. The 20 proton plans analyzed in this study were either directly planned on PTV or optimized to CTV but confirmed by the institutions to meet the protocol specified PTV coverage criteria. Therefore, all 70 patients were analyzed with the same PQM algorithm. The PlanIQ calculated benchmark dose represents the upper limit to the dose falloff that any photon plan can achieve, so proton plans should generally be able to perform well against it. Photon plans which have difficult geometries will receive score adjustments putting them on an even playing field with the proton plans (which can also have score adjustments).

The minimum, maximum, median, mean, and standard deviation of scores of both the PQM and APQM are shown in figure 3. The range of scores (maximum-minimum) is 46.7 and 50.0 points, and the standard deviation is 10.1 and 10.8, for unadjusted and adjusted scores, respectively.The mean and median scores for the adjusted group are 7-8 points higher than for the unadjusted group (range 1.5-12.5 for photons, 4.3 −10.8 for protons), and this was true for both the proton and photon groups, indicating a modest score adjustment was made. The median APQM for the 20 proton and 50 photon plans was 83.3 and 86.9, respectively. To test the significance of statistical difference in median APQM scores for photon versus proton plans, a Mann Whitney test was carried out for the 2 data sets which produced a p-value of 0.13622 showing that the results are not significant at p<0.05.

The adjusted scores for the individual targets and OARs were also plotted as histograms (figure 4). These scores varied widely across the PTVs, cochlea, and brainstem. The structures with the greatest mean percentage point losses in order of amount lost were brainstem (−50%), PTV boost (−32%), cochlea (−30%), WVPTV (−11%), spinal cord (−9%), chiasm (−9%) and eyes (−5%). Losses for all structures were nearly identical for proton and photon cases.

Figure 4.

Histograms of percent of fully adjusted scores for the two targets and the OARs in this study. The hypothalamus and pituitary are not shown as few plans received any points for those structures. The diversity of plan scores is apparent in the PTV boost and brainstem, while the other OARs were largely highly scored.

A correlation matrix depicting the relationship between individual structure adjusted quality metric scores and the cumulative plan APQM scores was calculated as another way to ascertain which of these structures had the biggest impact on APQM score. Only the brainstem and chiasm had negative correlation scores, −0.25 and −0.2 respectively, indicating that as their quality score increased, overall APQM decreased. The corresponding p-values were 0.0275 and 0.0320, respectively. This can possibly be explained by their close proximity to the PTV whereby driving the OAR dose down to meet constraints necessarily reduced PTV coverage which decreased the APQM. On the contrary, the eyes show significantly positive correlation with plan quality. However, these structures are farther away from the PTV so dose sparing could be achieved without impacting PTV coverage. The rest of the structures did not reach a statistically significant p-value (supplemental table 1). Bold coding shows significant p-values with negative correlation coefficients.

There were 4 protocol violations in the 70 cases, 2 minor and 2 major⁶. The minor violations were for dose and for uniformity, with an APQM of 81.3 and 48.5, respectively. The minor dose violation was due to marginal target coverage and this case correspondingly lost the majority of its points for that reason. The major violations were for dose and for volume, with APQM scores of 55.3 and 86.6, respectively. The major dose violation was for a patient with a nongerminomatous germ cell tumor that received RT for pure germinoma. The major volume violation was for an inadequate target volume, but what was contoured by the institution, although inadequate, was covered well by their plan and thus achieved a high APQM score.

Discussion

To the best of our knowledge, this report is the first to systematically analyze radiation therapy plan quality scores across an entire cohort of patients within a cooperative group clinical trial. Beyond determining if the plan submission is protocol compliant with appropriate target volumes and prescription doses, currently no evaluation is made of the plan quality by the QA center, i.e., is this the best plan that can be created with the given dose constraints and PTV and OAR geometry. Not only does our proposed methodology score each plan based on the protocol requirements using a defined point system but also applies score adjustments based on geometric features of the patient’s target and OARs to level the playing field across all simple as well as difficult cases. The proposed scoring system not only detects whether a protocol requirement is met or not, but the application of a point system serves to discover which plans were better even when all protocol requirements were met. Thus, instead of just stating the percentage of radiotherapy compliant cases as can be done now, we can in addition quantify the mean and range of plan quality across the submissions. We found a large range of plan quality scores, even after systematically correcting for difficult geometries, indicating a large degree of heterogeneity in planning considerations for optimizing OAR and target doses even after protocol rules were followed. There was little difference in scores between the proton and photon plans. The mean magnitude of scoring adjustments based on the intrinsic difficulty level of each case was about 7 points out of 100. This study methodology also allowed the determination of which OARs and targets were more difficult to accrue points relative to the other structures; the brainstem, boost PTV, and cochlea were the most challenging.

This type of analysis can be performed on other completed trials as well, but more importantly, can be performed in real time as cases are being submitted to the QA center, either for rapid or post treatment review. Once a reasonable number of cases have been processed (perhaps 20), a mean APQM score can be used to assess the plan quality of subsequent submissions. ACNS1123 was an “on treatment review” study, meaning plans had to be submitted and receive review and approval before treatment could commence (or within first 3 days). This window could be used to intervene and request the institution to replan the patient based on the relative scoring algorithm results, within practical time constraints. Institutions and QA centers could use the plan quality score distribution from the first approximately 20 submissions as a guide for plan improvement. This process could provide insights for future trials and protocols on the achievable plan quality metrics and DVH’s. There are other software solutions available which can also provide feedback about plan quality improvements such as knowledge-based planning systems. Geng et al.⁹ described a study comparing PlanIQ to a commercial knowledge-based planning system for the ability to predict the possible plan quality improvements to original RTOG protocol plan submissions. They generally found the two systems to perform similarly in most cases.

There have been several efforts in the past to assess individual patient plan quality using automated methods. It has been well understood that there are variations in plan quality within a radiation oncology department even for a given type of treatment. These variations can be minimized by understanding and addressing the source of this variation, resulting in an overall increase in average plan quality.

Wang et al.¹⁰ described a semi-automated plan quality analysis method for RTOG trials which extracts the dosimetric values from the plan dose grid and compares them to the protocol requirements. This saves time for the QA center personnel who would have to do the same thing manually in the current workflow, however, this was only to determine if protocol requirements were achieved or not. Assessment of plan quality beyond adhering to protocol requirements was not performed nor was there an attempt to understand the range of plan quality across cases.

Reducing variation in plan quality has been studied by several groups. Cui et al.¹¹ described their effort to apply key plan quality metrics to individual plans to reduce SRS plan quality variation at their institution. Comparing metrics and associated determinants across patients, they found plans with relatively low quality that could be improved. Roy et al.¹² used control charts with various risk factors including distance from target to OAR. Plans were improved by replanning with this knowledge. In a Canadian study of 235 prostate patients from 4 large centers, fourteen anatomic features were found to account for patient population differences. Despite uniform guidelines, statistically significant plan quality differences were found². One report quantified the quality deficiencies of the RTOG 0126 clinical trial, comparing the plans from 219 patients to a benchmarked high-quality plan. They did not account for patient-specific geometric differences but did replan a random selection of cases and found plan quality improvements could be achieved¹.

PlanIQ has been used by several groups for plan quality improvements. Perumal et al.¹³ compared plans made using RTOG dose constraints to those using PlanIQ with its feasibility analysis and found that while all plans met the RTOG dose constraints, those based on the benchmark doses calculated by PlanIQ had better OAR sparing without compromise to target coverage. Alves et al.¹⁴ and Duffy et al.¹⁵ found dosimetric variability was reduced for head and neck and other sites when PlanIQ methods were used. The feasibility tool in PlanIQ was also investigated for lung tumor planning and better lung sparing was found¹⁶. Dose-volume predictions from the feasibility tool in PlanIQ were used to assess 114 plans from 3 RTOG trials. Significant reductions in parotid and heart doses were found compared to the submitted plans¹⁷. Sasaki et al.¹⁸ used PlanIQ feasibility software for 148 prostate cases to improve plan quality and identified areas of improvement in treatment plans, allowing inferior plans to be discovered and replanned.

A limitation of our work is that plan quality evaluation software only evaluates the volumes segmented by the institution, volume review will still need to be performed by the radiation oncology investigator or the QA center. If volumes are modified as a result of the rapid pretreatment review, then the PlanIQ evaluation can be rerun. Also, our approach to defining the plan score metrics was based on the protocol guidelines and our understanding of the clinical goals of the study but there is no standard way to decide these scoring metrics and other systems could be just as valid. In addition, it remains to be proven that outcomes improve with improved plan quality beyond meeting the protocol constraints. Although we have demonstrated the feasibility and utility of our methodology on a single COG study, before cooperative group implementation, further study of feasibility and effectiveness should be performed.

Conclusions

We present a methodology for assessing plan quality for radiotherapy clinical trials using ACNS1123 as an example. Although most cases analyzed did not result in a protocol deviation, there was about a 50-point (out of 100 maximum points) spread of plan quality scores even after adjusting for variations in geometric relationship between targets and OARs, demonstrating that there are opportunities to improve plan quality for many protocol plan submissions. This approach could be applied to any radiotherapy clinical trial with the goal of improving average plan quality and narrowing the quality variability which may serve to improve outcomes.

Data availability:

DICOM data used in this study is housed in the IROC-Rhode Island database, access must be approved by the Children’s Oncology Group. The full set of results of our analysis can be provided upon request to the corresponding author.