Real-Time Pre-Treatment Review Limits Unacceptable Deviations on a Cooperative Group Radiotherapy Technique Trial: Quality Assurance Results of RTOG 0933

Abstract

Purpose/Objectives

RTOG 0933 was a phase II trial of hippocampal avoidance during whole-brain radiotherapy for patients with brain metastases. Results demonstrated improvement in short-term memory decline, as compared to historical controls, and preservation of quality of life. Integral to the conduct of this trial were quality assurance processes inclusive of pre-enrollment credentialing and pre-treatment centralized review of enrolled cases.

Methods and Materials

Prior to enrolling patients, all treating physicians and sites were required to successfully complete a “dry-run” credentialing test. Treating physicians were credentialed based on accuracy of MRI-CT fusion and hippocampal and normal tissue contouring, and sites were credentialed based on protocol-specified dosimetric criteria. Using the same criteria, pre-treatment centralized review of enrolled cases was conducted. Physicians enrolling 3 consecutive patients without unacceptable deviations were permitted to enroll further patients without pretreatment review, although their cases were reviewed post-treatment.

Results

113 physicians and 84 sites were credentialed. 8 physicians (6.8%) failed hippocampal contouring on 1^st attempt; 3 were approved on 2^nd attempt. 8 sites (9.5%) failed IMRT planning on 1^st attempt; all were approved on 2^nd attempt. 113 patients were enrolled on RTOG 0933; 100 were analyzable. 87 cases were reviewed pre-treatment. 5 (5.7%) violated eligibility criteria, and 21 (24%) had unacceptable deviations. With feedback, 18 cases were approved on 2nd attempt, 2 cases on 3^rd attempt, and 1 case was treated off protocol. 22 cases were reviewed post-treatment. 1 (4.5%) violated eligibility criteria, and 5 (23%) had unacceptable deviations.

Conclusions

While >95% passed pre-enrollment credentialing, pre-treatment centralized review disqualified 5.7% of reviewed cases, prevented unacceptable deviations in 24% of reviewed cases, and limited the final unacceptable deviation rate to 5%. Thus, pre-treatment review is deemed necessary in future hippocampal avoidance trials and potentially useful in other similarly challenging radiotherapy technique trials.

Introduction

Conformal avoidance of the hippocampus during whole-brain radiotherapy (HA-WBRT) has been proposed as a technique for preventing memory decline following cranial irradiation [1]. Multiple intensity-modulated radiotherapy (IMRT) techniques have been developed to safely deliver WBRT while minimizing dose to the hippocampus [2,3]. These techniques involve a detailed anatomic understanding of the hippocampal dentate gyrus, where radiosensitive neural stem cells that subserve memory function upon maturity are believed to reside [4,5], and a technical understanding of the IMRT nuances required to conformally avoid two small structures (bilateral hippocampal dentate gyri) centrally located within a larger target volume (whole-brain parenchyma).

RTOG 0933 was a phase II study of HA-WBRT for patients with brain metastases. The primary endpoint of this study was the Hopkins Verbal Learning Test-Revised (HVLT-R) at 4 months. The study built in a pre-specified statistical comparison to a historical control of brain metastasis patients treated with WBRT without hippocampal avoidance. Results have demonstrated relative prevention of HVLT-R decline and preservation of quality of life with the use of HA-WBRT in patients with brain metastases [6,7].

Integral to this radiotherapy technique trial was the mandatory centralized quality assurance involving two components: 1) pre-enrollment credentialing for all enrolling sites and treating physicians, and 2) centralized review of all hippocampal contouring and HA-WBRT plans. These quality assurance processes were developed specifically in cognizance of the inherent difficulty in hippocampal contouring and HA-WBRT planning. This manuscript describes the infrastructure and mechanisms supporting these processes and illustrates the importance of robust centralized pre-treatment case review for future planned phase III trials of hippocampal avoidance and other similar trials testing novel but technically challenging radiotherapy techniques.

Methods

Study design and patients

Patients with brain metastases outside a 5-mm margin around either hippocampus, pathologically proven diagnosis of non-hematopoetic malignancy other than small cell lung cancer or germ cell malignancy, RTOG recursive partitioning analysis (RPA) class I or II, and English proficiency were eligible for inclusion. Patients younger than 18 years and those with leptomeningeal metastasis, radiographic evidence of hydrocephalus, prior radiation to the brain, planned upfront radiosurgery or surgical resection, contraindication to MR imaging, serum creatinine >1.4 mg/dl ≤30 days prior to study entry, or non-small cell lung cancer-associated brain metastases with 2 or more organ sites of extracranial metastases, were excluded. All eligibility criteria matched those for the historical control, which comprised brain metastases patients treated with WBRT without hippocampal avoidance on the PCI-P-120-9801 phase III trial [8].

For hippocampal contouring and HA-WBRT planning, all patients required a three-dimensional spoiled gradient, magnetization-prepared rapid gradient echo or turbo field echo T1-weighted axial MRI scan of the brain with axial slice thickness ≤1.5mm, fused to a radiotherapy planning head CT scan with axial slice thickness ≤2.5mm. Bilateral hippocampal contours were manually generated on the fused MRI-CT image set and expanded by 5mm to generate the hippocampal avoidance regions. The clinical target volume (CTV) was defined as the whole-brain parenchyma, and the planning target volume (PTV) was defined as the CTV excluding the hippocampal avoidance regions. IMRT was delivered to 30 Gy in 10 fractions to cover the PTV while avoiding the hippocampus.

All patients provided written informed consent. The study was approved by the National Cancer Institute and participating centers’ institutional review boards.

Pre-enrollment credentialing

Prior to enrolling patients, all sites were required to meet specific technology requirements and provide baseline physics information for the use of IMRT on this study (“RTOG IMRT Credentialing”). In addition, all treating physicians and sites were required to successfully complete a “dry-run” quality assurance test involving fusion of MRI and radiotherapy-planning CT, hippocampal contouring, and development of an IMRT plan for HA-WBRT for a sample patient chosen from a test group of 5 patients whose MRI and CT imaging were provided electronically. To train on the techniques of hippocampal contouring and IMRT planning for HA-WBRT, multiple didactic workshops were held by the trial’s Principal Investigators (V.G and M.P.M.) during RTOG semi-annual meetings, and a contouring atlas for hippocampal delineation was made available electronically on the RTOG website [9].

Credentialing for RTOG 0933 was both physician- and site-specific. Treating physicians were credentialed based on meeting compliance criteria for MRI-CT fusion and accuracy of hippocampal and normal tissue contouring. If either of these criteria were not met, the physician would be asked to partake again in the “dry run” quality assurance testing using a second image set. Enrolling sites were credentialed for their specific IMRT modality based on meeting pre-specified dosimetric compliance criteria. Violation of these compliance criteria at the Deviation Unacceptable level would require repeat IMRT planning using a second image set. When compliance criteria were not met, detailed instructive and corrective feedback was provided electronically by the trial’s Medical Physics Co-Chair (D.M.) and Principal Investigators (V.G. and M.P.M.) to assist in repeat contouring and/or planning on the second image set. Each site was permitted to have more than one treating physician so long as each treating physician was separately credentialed.

Centralized review

Central pre-treatment review of hippocampal contours and HA-WBRT planning was conducted within a 3 working-day period prior to initiating treatment. If an Unacceptable Deviation was scored for any of the compliance criteria upon pretreatment review, the treating physician and site were provided instructive and corrective feedback from the Principal Investigators (V.P. and M.P.M.) and required to make recommended corrections prior to treatment (Figure 1). At the discretion of the Principal Investigators, resubmission of the revised treatment plan was requested. Sites that enrolled 3 consecutive patients without unacceptable deviations were permitted to enroll further patients without pretreatment central review. However, all of these treatment plans were reviewed post-treatment.

Figure 1. Example of pre-treatment review preventing unacceptable deviation.

A) On initial case submission, surface rendering of minimal distance between treating physician contour and Principal Investigator contour demonstrated a Hausdorff distance of 9.57mm. B) Instructive and corrective feedback was electronically provided to the treating physician by the Principal Investigators (X.X. and X.X.). Treating physician’s contour (Yellow) demonstrates that hippocampal dentate gyrus was not extended to the uncal recess as demonstrated by Principal Investigator contour (Blue). C) On case resubmission with feedback, surface rendering of minimal distance between treating physician contour and Principal Investigator contour demonstrated a Hausdorff distance of 4.51mm.

Compliance Criteria

For the 5 sample image sets used for pre-enrollment credentialing, a “gold standard” hippocampal contour was collectively developed by the trial’s Neuro-Radiology Co-Chair (H.R.) and Principal Investigators (V.P and M.P.M). For each enrolled patient undergoing centralized review, a hippocampal contour was generated by the trial’s Principal Investigators (V.G and M.P.M.). The treating physician’s hippocampal contour was volumetrically compared to the Principal Investigators’ hippocampal contour through the calculation of a Hausdorff distance, defined as the greatest of all the distances from a point on one contour to the closest point on the other contour (supremum infimum). The Hausdorff distance was calculated by overlaying the DICOM structure set containing the physician’s hippocampal contour onto the credential case image set in the Computational Environment for Radiotherapy Research (CERR), run through a specialized MATLAB software program that permitted the computation of Hausdorff distance in three-dimensional space. A Hausdorff distance of greater than 7mm or errors in MRI/CT fusion was considered an unacceptable deviation. Per protocol, Variation Acceptable and Deviation Unacceptable scoring criteria are summarized in Table 1.

Table 1.

Summary of dosimetric data from all treatment plans that passed pre-enrollment credentialing.

Dosimetric parameter	Mean value	Range	Protocol-Specified Compliance Criteria
			Per Protocol	Variation Acceptable	Deviation Unacceptable
PTV V95%	96.4%	94.0–99.7%
PTV D98%	26.3Gy	18.3–36.1Gy	≥25Gy	<25Gy
PTV D90%	30.9Gy	30.0–36.6Gy			<30.0Gy
PTV D2%	36.0Gy	31.7–39.8Gy	≤37.5Gy	>37.5Gy, ≤40Gy	>40.0Gy
Hippocampal maximum dose	14.8Gy	10.6–17.0Gy	≤16Gy	≤17Gy	>17Gy
Hippocampal D100%	8.6Gy	6.3–10.0Gy	≤9Gy	≤10Gy	>10Gy
Optic nerve maximum dose	32.8Gy	11.4–37.2Gy	≤37.5 Gy	≤37.5Gy	>37.5Gy
Optic chiasm maximum dose	34.4Gy	31.0–37.4Gy	≤37.5 Gy	≤37.5Gy	>37.5Gy

Infrastructure for conducting centralized quality assurance

The infrastructure for conducting centralized quality assurance was developed through the RTOG and the Image-Guided Therapy Center (ITC). Complete data, including the fused planning MRI/planning CT image set with hippocampal contours and other target/organ at risk volumes and associated treatment plan with dose-volume histogram, were submitted by each site through the ITC before centralized review was initiated. All data were housed at RTOG headquarters. The designated trial investigators conducted centralized quality assurance review through remote Citrix access. Communication between the trial investigators and treating physician/site was facilitated electronically by RTOG staff (D.M.). Real-time pre-treatment reviews were conducted within 3 business days of completed data submission and prior to treatment initiation.

Statistical Analysis

For physicians who were successfully credentialed, the dosimetric impact of hippocampal contouring variation was evaluated by comparing the following hippocampal dosimetric parameters between the credentialed physician’s contours and the “gold standard” contour: hippocampal maximum dose, minimum hippocampal dose (D100%), and dose to 40% of the hippocampus (D40%). D40%, converted to equivalent dose in 2-Gy fractions assuming an α/β ratio of 2.0, has been validated in a prospective trial as being significantly associated with cognitive decline [10]. Comparisons were made using paired samples T-test.

Results

113 physicians and 84 sites were credentialed in the techniques for HA-WBRT. 8 physicians failed hippocampal contouring for the dry-run case with 3 approved with first resubmission and 5 not pursuing further resubmission. 8 sites failed IMRT planning for the dry-run case with all 8 approved with 1^st resubmission. The mean of the Hausdorff distance distribution was 5.47mm (range: 3.12–15.3mm) (Figure 2). Table 1 provides a summary of dosimetric data for all treatment plans approved during the credentialing component. Table 2 compares hippocampal dosimetry between the treating physician’s contours submitted during the credentialing component and the “gold standard” contour. Variations in hippocampal contouring significantly impacted maximum hippocampal dose (p<0.001), with a more modest but significant impact on hippocampal D40% (p<0.001). Variations in hippocampal contouring did not impact hippocampal D100%.

Figure 2. Distribution of Hausdorff distance results during credentialing and case review.

A Hausdorff distance of greater than 7mm was deemed an unacceptable deviation.

Table 2.

Comparison of hippocampal dosimetry between the treating physician’s contours and the Principal Investigators’ contour. EQD₂ : equivalent dose in 2-Gy fractions assuming an α/β ratio of 2.0.

Mean (Range)	Hippocampal contour used to evaluate IMRT plan		p value
	Credentialing physician	Gold standard
Hippocampal maximum dose (Gy)	14.9 (10.6–17.0)	20.3 (12.3–32.2)	<0.001
% of cases with hippocampal maximum dose >17Gy	0%	69%
Hippocampal D100% (Gy)	8.6 (6.3–10.0)	8.6 (6.3–10.0)	NS
% of cases with hippocampal D100% >10 Gy	0%	0%
Hippocampal D40% (Gy)	10.7 (7.3–13.3)	11.0 (7.4–15.4)	<0.001
% of cases with hippocampal D40% >7.3 Gy (EQD2)	77.5%	77.5%

From March 31, 2011 to November 7, 2012, a total of 113 patients were accrued from 36 different institutions. Pre-treatment central review led to the discovery of 5 patients who did not meet eligibility criteria: 2 patients were noted to have hydrocephalus on submitted MRI, 2 patients did not undergo the required pretreatment MRI sequence, and 1 patient was noted to have a metastasis within 5mm of the hippocampus. Prior to quality assurance review, 2 patients did not receive protocol treatment, 1 patient died, and 1 patient withdrew consent.

Of the 82 cases reviewed prior to treatment as part of the real-time rapid review process, 21 cases (25%) from 12 different institutions received a score of Deviation Unacceptable: 11 cases had unacceptable deviations of contouring only, 5 cases had unacceptable deviations of IMRT planning, and 5 cases had unacceptable deviations of both contouring and IMRT planning. The mean of the Hausdorff distance distribution was 5.65mm (range: 3.66–10.10mm) (Figure 2). Reasons for unacceptable contouring deviations were Hausdorff distance for hippocampal contour in excess of 7mm (11 cases), incorrect contouring of optic chiasm/nerves (7 cases) and incorrect MRI/CT fusion (2 cases). Reasons for unacceptable deviations of IMRT planning were PTV D90% <30 Gy (8 cases), hippocampal maximum dose >17 Gy (1 case), and hippocampal D100% >10 Gy (1 case). With feedback provided to the treating physician on all 21 cases, 18 cases were approved on second attempt, 2 cases were approved on third attempt, and 1 case was treated off protocol.

Of the 22 cases reviewed following protocol treatment completion, 1 patient was disqualified due to radiographic evidence of hydrocephalus, and 5 (24%) cases from 3 different institutions received a score of Unacceptable Deviation: 2 cases had unacceptable deviations of contouring only and 3 cases had unacceptable deviations of IMRT planning only. The mean of the Hausdorff distance distribution was 6.13mm (range: 4.37–9.26mm). Reasons for unacceptable contouring deviations were Hausdorff distance for hippocampal contour in excess of 7mm (1 case) and incorrect MRI sequence used for hippocampal contouring (1 case). Reasons for unacceptable deviations of IMRT planning were PTV D90% <30 Gy (2 cases) and hippocampal D100% >10 Gy (1 case).

Figure 3 demonstrates the distribution of accrual and unacceptable deviations across treating sites. 100 patients were deemed eligible for statistical analysis. With all cases receiving a score of Unacceptable Deviation on pre-treatment review appropriately corrected or treated off protocol, and post-treatment review scoring 5 cases as Unacceptable Deviation, the final Unacceptable Deviation rate was 5%.

Figure 3. Distribution of accrual and unacceptable deviations across treating sites.

Cases that were deemed ineligible during case review or that died, withdrew consent or did not receive protocol treatment prior to quality assurance review are not included.

Discussion

RTOG 0933 demonstrated that a robust centralized quality assurance infrastructure could be developed for a cooperative group trial testing a novel radiotherapy technique. Over 95% of physicians and 100% of institutions interested in enrolling patients on RTOG 0933 eventually passed the requisite credentialing exercise, with the vast majority passing on the first attempt. However, in spite of this credentialing effort, 24% of cases submitted for pre-treatment centralized review demonstrated unacceptable deviations of contouring and/or IMRT planning and required re-planning prior to initiating protocol therapy. In addition, amongst treating physicians who enrolled three consecutive patients without unacceptable deviations on pre-treatment review and who were permitted to enroll patients without requisite pre-retreatment review, post-treatment review of these cases demonstrated unacceptable deviations in 23%, not including one patient who was disqualified due to radiographic evidence of an exclusion criterion.

In sum, pre-treatment centralized review of cases enrolled on RTOG 0933 led to the disqualification of 5.7% of reviewed cases and significantly lowered the unacceptable deviation rate from approximately 24% to a more acceptable protocol-predicted rate of 5%. These results underscore the importance of including real-time pre-treatment centralized review of cases, in addition to pre-enrollment credentialing, in future hippocampal avoidance trials, as well as other similar trials testing novel and technically challenging radiotherapy techniques.

The impact of protocol deviations in terms of radiotherapy technique on treatment outcomes has been previously demonstrated. Ohri and colleagues performed a meta-analysis of cooperative group clinical trials and observed that radiotherapy protocol deviations were associated with increased risk of treatment failure and overall mortality [11]. Importantly, this study was limited only to trials using two-or three-dimensional radiotherapy techniques and did not include trials using IMRT. With greater conformality of IMRT, however, the impact of radiotherapy protocol deviations would be expected to be larger. In the setting of HA-WBRT, where a 3–4 cc bilateral structure centrally located within a larger target volume is being selectively spared of significant doses of radiotherapy, protocol deviations can confound the ability to reliably test the hypothesis of hippocampal sparing, especially with the knowledge that a steep-dose response gradient may exist for this structure. The positive results of this phase II trial, in which the unacceptable deviation rate was limited to 5% due to pre-treatment case review, highlight the importance of carrying these centralized quality assurance processes forward to planned phase III trials of hippocampal avoidance. In the setting of declining NIH funding and in order to scale these efforts to much larger planned phase III trials, RTOG is considering more cost-effective and less labor-intensive approaches to these centralized quality assurance processes.

The disparity in unacceptable deviation rate between credentialing exercises and pre-treatment centralized case review was larger than expected. On first attempt of the credentialing exercise, more than 90% of physicians and sites passed. This high pass rate was in part facilitated by a contouring atlas made available online and workshops held by the trial’s Primary Investigators (V.G and M.P.M.) at biennial RTOG meetings. However, pre-treatment review identified a much higher unacceptable deviation rate of 24%. The difference may be attributable to the shorter turnaround time required for contouring and planning brain metastasis patients requiring whole-brain radiotherapy, as opposed to planning a “test case,” where such time pressures do not exist. A similar unacceptable deviation rate was observed amongst investigators who underwent three consecutive pretreatment reviews without any unacceptable deviations and were thereafter permitted to enroll patients without pre-treatment review, suggesting that past performance did not predict for lower unacceptable deviation rate and that such challenging techniques might require significantly greater experience prior to reaching an error-free or low-error state.

Related to this high unacceptable deviation rate is the concern that the promising results of this phase II single-arm trial may lead to a potentially unsafe off-protocol utilization of HA-WBRT for patients with brain metastases, if this approach is widely adopted without appropriate credentialing. On RTOG 0933, in which the unacceptable deviation rate was limited to 5%, two treatment-related grade 3 adverse events of fatigue and headache were reported; there were no grade 4 or higher adverse events. In the absence of proper credentialing and/or expertise, off-protocol utilization of HA-WBRT has the potential to be associated with inappropriate treatment delivery, potentially compromising treatment efficacy and raising the risk of unexpected toxicity. As an example, 37% of unacceptable contouring deviations involved inaccurate contouring of the optic chiasm or nerves. Depending on IMRT modality, HA-WBRT can be associated with significant plan heterogeneity. RTOG 0933 permitted dose to 2% of the whole brain PTV up to 40 Gy in 10 fractions as an acceptable deviation. However, to mitigate risk of optic neuritis, maximum point dose to the optic chiasm/nerves was limited to 37.5 Gy. Inaccurate contouring of the optic chiasm or nerves has the potential to lead to maximum point doses on the optic structures well in excess of 40 Gy in 10 fractions, which approximates a 60 Gy equivalent dose in 2-Gy fractions (assuming alpha/beta of 2.0).

Hausdorff distance in excess of 7mm was the most common unacceptable deviation. We sought to determine whether the 7mm Hausdorff distance threshold was appropriate for evaluating discrepancies in hippocampal contouring. For this analysis, all plans approved during the credentialing process were reviewed, since all of these cases had a Hausdorff distance of 7 mm or less. Hippocampal dosimetry was then compared between using the treating physician’s hippocampal contour versus the “gold standard” hippocampal contour developed by the trial’s Neuroradiology Co-Chair and Principal Investigators. This comparison demonstrated that hippocampal contouring discrepancies with Hausdorff distance ≤7 mm significantly impacted maximum hippocampal dose, but only modestly impacted dose to hippocampal D40% and had no effect on hippocampal D100%. Analysis of RTOG 0933 demonstrated an association between higher hippocampal D100% and greater memory decline [6,7]. No such association was observed for maximum hippocampal dose. These data therefore suggest Hausdorff distance of 7 mm may be a clinically appropriate threshold for assessing accuracy of hippocampal contouring.

In conclusion, pre-treatment case review on RTOG 0933 significantly limited the unacceptable deviation rate, which may have played a role in the positive results as compared to historical controls. These data strongly favor the use of pretreatment case review in future planned hippocampal avoidance trials and other similar trials testing novel but technically challenging radiotherapy techniques.

Summary.

RTOG 0933 demonstrated improvement in short-term memory decline, as compared to historical controls, and preservation of quality of life following conformal avoidance of the hippocampus during whole-brain radiotherapy for patients with brain metastases. Centralized quality assurance included pre-enrollment credentialing and pre-treatment case review. This article summarizes the results of these quality assurance processes and demonstrates the value of pre-treatment case review in future hippocampal avoidance trials and potentially other trials testing similarly challenging radiotherapy techniques.