Improving Quality and Consistency in NRG Oncology RTOG 0631 for Spine Radiosurgery via Knowledge-Based Planning

Abstract

Purpose

Consistency and standardization of radiotherapy plan quality in multi-institutional clinical trials are always challenging, and non-protocol compliant radiotherapy plans have been shown to impact patient outcomes. This study aimed to use knowledge-based planning (KBP) as a method of producing high quality, consistent, protocol-compliant treatment plans, in a complex setting of spine SBRT on NRG Oncology RTOG 0631.

Methods

An internally developed KBP model was applied to an external validation cohort of 22 anonymized cases submitted under NRG Oncology RTOG 0631. The original and KBP plans were compared via their protocol compliance, target conformity and gradient index, dose to critical structures, and dose to surrounding normal tissues.

Results

The KBP model generated plans meeting all protocol objectives in a single optimization when tested on both internal and protocol-submitted NRG Oncology RTOG 0631 cases. Two submitted plans that were considered to have a protocol-unacceptable deviation were made protocol compliant through the use of the model. There were no statistically significant differences in protocol spinal cord metrics (D10% and D0.03cc) between the manually optimized plans and the KBP plans. The volume of PTV receiving prescription dose increased from 93.3 ± 3.2% to 98.3 ± 1.4% (P=0.01) when using KBP. High-dose spillage to surrounding normal tissues (V105%) showed no significant differences (2.1 ± 7.3 cc for manual plans to 1.8 ± 0.6 cc with KBP), and dosimetric outliers with large amounts of spillage were eliminated through the use of KBP. KBP plans were also found to be significantly more consistent in several metrics, including target coverage and high dose outside of the target. KBP applied to NRG Oncology RTOG 0631

Conclusion

Incorporation of KBP models into the clinical trial setting may have a profound impact on the quality of trial results due to the increase in consistency and standardization of planning, especially for treatment sites or techniques that are non-standard.

Introduction

Spinal stereotactic body radiation therapy (sSBRT) and spinal stereotactic radiosurgery (sSRS) have been developed to improve upon the suboptimal local control outcomes achieved from classical doses of palliative conventional external beam radiotherapy.^1,2 Radiation Therapy Oncology Group (RTOG), now part of NRG Oncology, initiated a Phase II/III study (RTOG 0631) to examine whether or not the higher biologically effective doses delivered with sSRS could lead to improved quality of life and pain control compared with standard palliative radiation therapy. The planning and delivery of single fraction sSRS is technically challenging, and investigation within the clinical trial setting required extensive credentialing, including phantom irradiation, image guidance review, and pre-treatment plan reviews.³

At the time of reporting the Phase II results,³ 65 academic and community practices were credentialed to enroll patients on the study. Each institution had variable experience with the planning of sSRS. Reflecting the complexity of planning for high dose sSRS, nearly half of pre-treatment plan reviews failed on the first attempt after review by the principal investigator, and 11% of cases required a third submission before approval. The most common reason for failure was high-dose spillage outside of the target volume. For all plans approved at pre-treatment review and subsequently treated on the study, 26% percent of cases had minor variations in target coverage and 26% percent had variations in normal tissue doses.

A multitude of previous studies have shown that variations in plan quality and trial compliance can adversely affect patient outcome^4–7 and some studies have found that accrual volume is associated with outcome.^8,9 The lack of standardization and potentially sub-optimal plan quality can limit the efficacy of clinical trials, making it difficult to prove that advanced radiation therapy-based interventions are improvements on current techniques, and may have potential permanent deleterious side effects (e.g. myelopathy).

Knowledge-based planning (KBP), as applied to radiation therapy, has shown to be a promising technique for improving plan quality, standardization, and efficiency in a variety of treatment sites^10–18 and across multiple treatment centers.^19–21 KBP has also been applied in the context of clinical trials as a quality assurance tool.^4,22,23 In sSBRT and sSRS, knowledge-based models permit automation and standardization of planning, and save substantial planning time, even for a dosimetrist skilled in planning sSBRT.²⁴

In the setting of NRG Oncology RTOG 0631, we sought to test the hypothesis that application of a validated KBP model for NRG Oncology RTOG 0631 could be used as both a retrospective clinical trial quality control tool and potentially a prospective tool to aid in plan quality improvement and standardization.

Methods

This study was approved by the NRG Oncology RTOG 0631 scientific committee. NRG Oncology RTOG 0631 requires a prescription dose of 16 or 18 Gy (note, only 16 Gy plans were analyzed in this study) to be delivered to at least 90% of the target volume with a strict dose fall-off/spillage requirement as well as both volumetric and point dose constraints on the spinal cord. The primary target and OAR objectives that must be considered in plan evaluation for this protocol are shown in Table 1. Note that this is not a complete list of planning goals that must be considered when evaluating NRG Oncology RTOG 0631 plans, as this protocol requires all OARs within 10cm of the target to be contoured. All plans used for this study were evaluated in absolute dose as submitted or optimized and did not use any plan normalization.

Table 1.

Selected NRG Oncology RTOG 0631 protocol constraints/objectives. For structures with objectives in parentheses, plans are considered variation acceptable (score 2) if the specified dose level or volume falls between the two limits and deviation unacceptable (score 3) if the dose or volume falls outside of the secondary limit.

Structure	Description	Objective
NonPTV1600	All normal tissue not included in the PTV	V16.8Gy < 2cc and within 1cm of PTV (V16.8 Gy < 3cc and within 1.5cm of PTV)
		D0.03cc < 17.6 Gy (D0.03cc < 18.4 Gy)
SpinalCord_Prt	Truncated version of spinal cord, 5–6mm superior/inferior of PTV	D10% < 10 Gy
SpinalCord	Spinal cord as defined on MRI/CT fusion	D0.03cc < 14 Gy D0.35 < 10 Gy
PTV_1600	Target structure receiving 16 Gy prescription dose	D90% > 16 Gy (D80% > 16 Gy)

PTV = planning target volume, MRI = magnetic resonance imaging, CT = computed tomography

Our NRG Oncology RTOG 0631-specific KBP model was created using an in-house training set of 40 manually optimized cases and validated with a further 11 cases. Modeling was done using the commercial Rapidplan KBP software (Varian Medical Systems). See the Supplementary Materials for further information on the in-house validation of our NRG Oncology RTOG 0631 KBP model.

Beyond the in-house validation of our KBP model as a means of creating high quality, protocol compliant treatment plans, an external validation cohort was obtained containing 22 cases submitted under the NRG Oncology RTOG 0631 protocol from NRG institutions (henceforth referred to as submitted plans). We included only submitted plans that used volumetric modulated arc therapy (VMAT) exclusively, were planned to 16 Gy, and were comprised of a single target. Planning system was not used as an exclusion factor. Multi-target cases were excluded from the validation to simplify cross comparison of non-target doses. Contours for all protocol structures were first reviewed and adjusted if necessary to be protocol compliant (very minor adjustments, none of which affected protocol compliance, were required for six cases). See the results section for a complete description of these minor edits. A high dose normal tissue (HDNT) ring structure was added as required for optimization with our model. This is a ring around the planning target volume (PTV) extending from 1mm to 5mm beyond the PTV.

New plans were then created for each case and optimized using the KBP model with a single optimization (henceforth referred to as KBP plans). For these optimizations, we used Eclipse version 13.6 in the NRG/IROC cloud environment with the Photon Optimizer set to fine (1.25mm) resolution (this is the dose calculation resolution employed during the optimization itself). Final calculations were done with AAA on a 1 mm grid. The KBP plans were VMAT with two 360 degree arcs with fixed collimator angles of 10 and 85 degrees.

The submitted plans used a mixture of unknown, standard definition (5 and 10mm leaf width), and high definition (2.5 and 5mm leaf width) multi-leaf collimators (MLCs). The MLC model was matched when possible, and the standard definition MLC was used if the original MLC model was unknown. The data made available by NRG for this study was anonymized such that we did not have access to the DICOM-RT files or institutional origin to request information on the MLC used.

All 44 plans (submitted and KBP) were compared against the protocol for compliance and target and normal tissue dose metrics for the two sets of plans were compared. For conformity index (CI), we used the Paddick index,²⁵ given by:

$$CI=\frac{{TV}_{PI}^2}{\mathit{PIV}\times TV}$$

Where TV_PI is the target volume encompassed by the prescription isodose surface, PIV is the prescription isodose volume, and TV is the target volume. The gradient index is defined as

$$GI=\frac{{PI}_{50\%}}{\mathit{PIV}}$$

Where PI_50% is the volume encompassed by the 50% isodose surface. Statistical significance in the comparison between the two sets of plans was determined using the two-tailed Student’s t-test. In comparing variances, a likelihood ratio test was performed for testing the null hypothesis of equal variances in the submitted and KBP groups. We assumed the samples followed a bivariate normal distribution to account for correlation between the groups and compared a null model where variances were constrained to be equal between the two groups with an alternative model that allowed unequal group variances.

Table 2 shows the parameters of our in-house RapidPlan model for NRG Oncology RTOG 0631 that were used for this study. The structure HDNT is not part of the protocol and was added to meet the normal tissue dose requirements as described earlier. We found that employing this structure produced the best dose fall-off around the target, however, another option would be to use a normal tissue objective in the optimization. The model was designed to prioritize target coverage and conformity, as long as the specified OAR dose limits were still met. When using RapidPlan, line objectives are created based on dose volume histogram (DVH) estimations, and consist of an entire series of point objectives as opposed to single point objectives.²⁶ The term “model generated” means that the RapidPlan model creates the objective automatically based on the model training cases. Note that the model does not include all structures that are included in NRG Oncology RTOG 0631. The normal tissue objective (NTO) penalizes dose outside of the target volume based on the distance from the target²⁷ and therefore aids in limiting dose to any structures not explicitly defined in the model.

Table 2.

RapidPlan NRG Oncology RTOG 0631 Model Parameters

Structure	Objective Type	Volume (%)	Dose (Gy)	Weight
PTV_1600	Upper	0.1	18	100
	Lower	100	16.2	125
CaudaEquina	Upper	0	15	100
	Upper	10	12	100
Esophagus	Line	Model Generated	Model Generated	Model Generated
HDNT	Upper	0	16.4	100
Larynx	Upper	5	10	50
SpinalCord	Upper	0	12	100
SpinalCord_Prt	Upper	0	12	100
	Upper	1	9	100
	Line	Model Generated	Model Generated	Model Generated
Trachea	Upper	5	10	Model Generated
NTO (Automatic)				70

PTV = planning target volume, HDNT = high dose normal tissue, NTO = Normal Tissue Objective

Results

Minor edits were made to the submitted structure sets in six of the cases used in this analysis. In three cases, the partial spinal cord (SpinalCord_Prt, see Table 2) included contours on one or two extra slices, which affects the D10% metric for this structure. For a fair comparison across all cases, a modified contour of the correct length was used for evaluation in each case, however, all of these cases were protocol compliant no matter which contour was used. In one case, the required NonPTV structures had not been contoured and were added. In two cases, there was a 1mm gap between the PTV and NonPTV structures which should not exist. In these cases, the NonPTV dose was evaluated on the institution’s submitted contours for fairness, however, in neither case did this change whether the plan was compliant. The variation in contours seen in these cases provides further motivation for incorporating automation into clinical trials, not just in optimization but also in plan preparation.

Table 3 shows the distribution of vertebral levels and the MLC types in each of the cases examined here. All of the submitted plans met target and spinal cord objectives, however, two of the submitted plans had an unacceptable deviation for NonPTV dose (one due to both NonPTV1600 V16.8 Gy > 3cc and NonPTV1600 D_max > 18.4 Gy and one due to NonPTV1600 D_max > 18.4 Gy only). This is similar to what was seen in the credentialing results described in the introduction where the majority of non-compliant plans failed due to dose outside the target volume. All KBP plans were protocol compliant. However, there were noticeable differences in the planning priorities between the KBP plans and the submitted plans, where KBP plans tended to increase PTV coverage at the expense of higher (although still protocol-compliant) dose to the NonPTV1600 structure.

Table 3.

Vertebral body level and MLC type used for the 22 NRG Oncology RTOG 0631 cases used in this study. The * indicates submitted cases that were non-protocol compliant due to dose in the NonPTV1600 structure.

Patient #	Level	MLC Type
A	T9	HD/HD
B	T7	M/M
C	L2	?/M
D*	L4	HD/HD
E	L2-L3	HD/HD
F	L4-L5	?/M
G	L5	M/M
H	L1	HD/HD
I	L2	M/M
J	T8	HD/HD
K*	T2-T3	HD/HD
L	C3-C4	HD/HD
M	C3-C4	HD/HD
N	T8-T9	?/M
O	T1-T2	?/M
P	L3-L4	?/M
Q	T10-T11	?/M
R	T2-T3	HD/HD
S	L3-L4	HD/HD
T	T6	?/M
U	L1-L2	?/M
V	T9	?/M

MLC = multileaf collimator, M = millennium MLC, HD = high-definition MLC, OAR = organ at risk, PTV = planning target volume

In Table 4, we compare the means and standard deviations of several dosimetric parameters between the submitted plans and the KBP plans. KBP plans had significantly increased target coverage and overlap of the PTV with the prescription isodose surface volume (PIV). Spinal cord metrics were similar across the two sets of plans and below the protocol limit in all cases. Conformity index was significantly increased via KBP although this result may not be clinically significant. Other dosimetric parameters were not statistically significantly different. In comparing the variances of the results, several were significantly different as shown in the table.

Table 4.

Comparison of protocol-relevant dosimetric parameters and standard deviations between submitted plans and KBP plans. PTV_1600 & PIV describes the Boolean overlap of the target with the prescription isodose volume.

Structure	Parameter	Submitted plans	KBP plans	P-value (comparing means)	P-value (comparing variances)
SpinalCord_Prt	D10%	7.6 ± 1.4 Gy	7.6 ± 0.5 Gy	0.95	<0.0001
SpinalCord	D0.03 cc	6.7 ± 3.9 Gy	7.1 ± 3.8 Gy	0.28	0.767
NonPTV1600	V16.8 Gy	2.1 ± 7.3 cc	1.8 ± 0.6 cc	0.88	<0.0001
NonPTV1600	V17.6 Gy	0.6 ± 2.8 cc	0.1 ± 0.1cc	0.34	<0.0001
PTV_1600	D90%	16.2 ± 0.3 Gy	16.5 ± 0.2 Gy	0.007	0.005
PTV_1600	V16Gy	93.3 ± 3.2 %	98.3 ± 1.4 %	0.01	0.0004
PTV_1600 & PIV	Volume	52.2 ± 33.1 cc	56.1 ± 36.0 cc	<0.001	<0.0001
CI		0.80 ± 0.11	0.84 ± 0.07	0.04	0.007
GI		4.91 ± 2.0	4.65 ± 1.0	0.38	<0.0001

PTV = planning target volume, PIV = prescription isodose surface volume, CI = conformity index, GI = gradient index

Figures 1 and 2 show two examples of comparisons between a protocol-submitted plan and the corresponding KBP plan. In the example in Figure 1, the submitted case was non-protocol compliant due to NonPTV1600 dose. The RapidPlan model made noticeable improvements to the dose distribution. The CI and GI for the submitted plan are 0.53 and 9.3, respectively, compared to 0.83 and 5.3 for the KBP plan. NonPTV1600 V16.8Gy was 34.6cc and 1.3 cc in the submitted and KBP plans, respectively.

Figure 1.

Axial and sagittal slices through the PTV for a protocol submitted plan (left) and corresponding KBP plan (right) in a case where dose conformity was improved with the use of KBP. PTV is contoured in pink and cauda equina is contoured in yellow. These images are from Plan D in Table 3.

Figure 2.

Axial and sagittal slices through the PTV for a protocol submitted plan (left) and corresponding KBP plan (right). The protocol submitted plan prioritized cord sparing while the KBP plan prioritized target coverage. PTV is contoured in red and spinal cord is contoured in blue. These images are from Plan B in Table 3.

In Figure 2, the comparison shows an example of a case where both plans were protocol compliant, but different trade-offs were made regarding target versus OAR dose. The CI and GI for the submitted plan are 0.85 and 3.8, respectively, compared to 0.85 and 4.4 for the KBP plan. Partial spinal cord D10% was 5.5 Gy and 8.0 Gy, and spinal cord D0.03cc was 5.8 Gy and 8.7 Gy in the submitted and KBP plans, respectively. NonPTV1600 V16.8 Gy was increased from 0.1cc in the submitted plan to 1.0cc in the KBP plan, whereas PTV coverage with the prescription dose (V16 Gy) also increased from 94.4% to 99.1% between the two plans.

Discussion

Knowledge based planning automated production of high quality, protocol-compliant plans in a single optimization for 100% of the NRG Oncology RTOG 0631 external validation cases. In some cases, particularly those in which the submitted plan was not protocol compliant (e.g. Figure 1), using the KBP model was able to meaningfully improve the dose to the target and nearby normal tissue. In the plan shown in Figure 1, the volume of normal tissue receiving the prescription dose or higher was considerably reduced, coverage was improved, and hot spots near the spinal cord were reduced. Using a KBP model such as the one studied in this work within the context of clinical trials has the ability to prevent outlier plans from being included in the study, with the potential to greatly facilitate the interpretation of study results and reduce unnecessary normal tissue complication risks.

Table 4 illustrates the difference in the dosimetric parameters between the plans submitted under the protocol and the KBP plans. PTV coverage was significantly greater in the KBP plans as was spinal cord maximum dose. This may indicate that in some cases, institutions prioritized cord sparing to too great an extent over coverage of the target. Additionally, as illustrated by comparing the variances across the two sets of cases, the KBP plans were observed to be significantly more consistent in the majority of metrics evaluated, including both PTV coverage as well as the amount of high dose outside of the PTV. The benefits in consistency gained through KBP in the amount of high dose outside of the PTV may be especially important in the context of the current trial, as this was one of the most frequently violated objectives noted in the Phase II results.³

Not all clinics participating in a clinical trial may be experienced in the type of planning required to produce near-optimal quality treatment plans subject to the dose constraints of the trial. For these institutions, using KBP to make an initial plan or as a retrospective evaluation of a previously optimized plan can reduce the probability of submitting a plan that either does not meet protocol requirements, or that could be improved within the constraints of the protocol. As noted previously, nearly half of the pre-treatment reviews in the NRG Oncology RTOG 0631 Phase II analysis failed and had to be replanned and resubmitted, causing substantial work for the sites and clinical trial staff, as well as potentially delaying patient treatment.

The time savings of using knowledge-based planning has been pointed out in the literature in other studies as well.^12,14,24 Spine SBRT is often planned and treated on an expedited time scale, making the potential for creating a protocol-compliant plan after a single optimization highly desirable. In the majority of cases in this study, dose distributions in the submitted plan and the KBP plan were already quite similar. However, if needed after the first optimization, the planning objectives may be further manipulated as desired to adjust the dose distribution per institutional preference, still saving time over creating the plan from scratch.

In a few cases, there was a clear difference in the prioritization of PTV coverage versus spinal cord sparing (Figure 2) as well as sparing of other normal tissue. PTV coverage was prioritized in this model as long as all protocol constraints were still met, however, we note that all submitted and KBP plans studied here met the protocol objectives for PTV coverage. When there is flexibility allowed in a protocol, the exact planning priorities will be institution- and physician-dependent and will vary across cases submitted under any protocol. The fact that KBP plans will consistently choose one or the other, depending on how the model was designed, is both an asset and a limitation. Reducing variation among protocol cases may improve the ability to interpret of the outcome of the trial results and reduce adverse effects.^4–7 The use of KBP could add considerable value to a protocol by essentially enforcing this reduced variability, especially if the use of KBP were required. However, this does limit the ability of the treating physician to select the type of dose distribution that is most desired. This preference will depend on multiple factors, such as the given institution’s immobilization equipment, imaging capabilities, and physician comfort level with each type of dose distribution. It is not a straightforward process to reproduce a plan with a different level of OAR sparing compared to the model being used (e.g. as in Figure 2). Additionally, here we have given an example of a KBP model created using single-institution data. The origin of the treatment plans included in a KBP model is less important than that all included plans represent the planning goals embodied by the protocol. Including data from multiple institutions in a model would more closely reflect the natural planning variability present across institutions, however, this does begin to negate to the consistency benefit derived from using KBP. Some trials may also benefit from having multiple available KBP models if planning flexibility is important in the protocol, as just described.

In the future, further incorporation of knowledge-based planning into clinical trials will have considerable benefits in ensuring compliance with protocol requirements and consistency in radiotherapy planning, allowing an increasing number of institutions to participate in trials and accrue additional patients. This incorporation would involve the sharing of trial-specific knowledge-based planning models that would be freely available to institutions participating in the trials. The model studied in this work was planning system-specific, however, in the future one could imagine a planning system-neutral solution capable of both generating an optimization cost function compatible with all or the majority of treatment planning systems in use, as well as generating estimated DVH’s for comparison to manually optimized treatment plans. This comparison could be performed either by the participating institution and/or the clinical trial quality assurance team.

Conclusion

We have shown that a KBP model for NRG Oncology RTOG 0631 is capable of producing consistently planned, protocol-compliant treatment plans with a single optimization. The KBP plans prioritized target coverage provided that all normal tissue objectives were met, whereas manual plans showed more variability in planning strategy. Given that consistency in planning is critical to the success of clinical trials, the application of KBP in this setting should continue to be investigated and expanded.

Summary.

Using NRG Oncology RTOG 0631 as an example clinical trial, we use knowledge-based planning (KBP) to improve trial treatment plan quality and consistency. KBP enables the generation of trial-compliant treatment plans in a single optimization. Incorporation of KBP models into the clinical trial setting may have a profound impact on the quality of trial results due to the increase in consistency and standardization of planning, especially for treatment sites or techniques that are non-standard.