Insights From Image Guided Radiation Therapy Credentialing for the NRG Oncology RTOG 1112 Liver Stereotactic Body Radiation Therapy Trial

Abstract

Purpose:

NRG Oncology trial RTOG 1112 is a randomized phase 3 study of sorafenib with or without stereotactic body radiation therapy for locally advanced hepatocellular carcinoma. Image guided radiation therapy (IGRT) credentialing is essential for this study because of the high doses, respiratory motion, and variety of delivery technologies. This analysis presents the IGRT credentialing experience.

Methods and Materials:

Credentialing of volumetric IGRT requires submission of planning and localization images, planning structures, and resulting IGRT shifts for a patient treated according to the study requirements. A study reviewer uses these data to repeat the registrations and compare to the actual clinical registrations. Agreement within 5 mm was considered acceptable for credentialing.

Results:

Volumetric images of 130 fractions from 42 institutions between June 2013 and January 2018 were reviewed. The median agreement between clinical registrations and study reviewer was 3 mm, with 95% of all fractions within 5 mm. A subanalysis identified a statistically significant difference between the use of low-contrast soft tissue and high-contrast surrogates (eg, implanted fiducial markers, surgical clips, metallic stents) for registration. Soft tissue and high-contrast surrogate registrations both agreed within 3 mm in 50% of fractions. However, soft tissue registrations exceeded 10 mm in 3% of fractions, while no high-contrast surrogate registrations exceeded 5 mm.

Conclusions:

The RTOG 1112 credentialing experience suggests that most institutions perform liver IGRT with sufficient accuracy to deliver stereotactic body radiation therapy safely, as assessed by expert reviewers. Both soft tissue and high-contrast surrogates appear adequate for consistent registration in most instances; however, some disagreements were observed when using soft-tissue registration targets. The use of high-contrast surrogates appears to reduce the small risk of substantial geographic miss owing to mis-registration in liver IGRT.

Introduction

NRG Oncology trial NRG/RTOG 1112 (NCT01730937) is a randomized phase 3 study of sorafenib with or without stereotactic body radiation therapy (SBRT) for hepatocellular carcinoma. This was one of the most technology-intensive studies initiated by RTOG at the time it was launched. Image guided radiation therapy (IGRT) credentialing is essential for this study because of high doses, respiratory motion, and a variety of delivery and imaging technologies. During the period of the study, there was rapid adoption of many IGRT technologies, including 2-dimensional (2D) orthogonal imaging, 3D cone beam computed tomography (CBCT), breath hold CBCT, and 4D CBCT.

Several studies have demonstrated the value of quality assurance activities in multi-institutional clinical trials.^1–5 These suggest that a failure to meet protocol expectations may affect toxicity and disease control, as well as confound the ability to answer the question the study was designed to address. This analysis examines the IGRT credentialing experience of NRG/RTOG 1112. We report the agreement observed for submitted IGRT registrations and attempt to identify factors that may be associated with deviations.

Methods and Materials

Informed consent was obtained from all patients before enrollment in this study.

Two-dimensional and 3D IGRT were both permitted in RTOG 1112; however, 2D IGRT employed a different credentialing process and is excluded from this current analysis. Volumetric IGRT credentialing followed the previously reported processes in place at RTOG at the inception of the study.⁶ Credentialing of volumetric IGRT for RTOG 1112 required submission of planning and localization images, planning structures, and resulting IGRT images. These were required for 2 to 5 fractions for a patient receiving IGRT and treated “as if” they were on the study (the number of required submitted fractions changed over the course of the study). Institutions were also asked to provide additional detail regarding motion management and IGRT registration strategy; however, this was not required and was not universally submitted.

All submitted data were imported into MIM (MIM Software Inc, Cleveland, OH) for analysis. A study reviewer independently repeated the registrations and compared with the actual submitted clinical registrations. The independent registration included a review of the reference and localization images and regions of interest, including the gross tumor volume and planning target volume. The independent registration attempted to register the images as well as possible in the region of the gross tumor volume and followed the strategy of the submitting institution when possible. Agreement within 5 mm in each orthogonal axis (as determined from the 3D translation between the reference and localization images) was considered acceptable for credentialing. All submissions deemed potentially unacceptable by the study reviewer were rereviewed with the principal investigator and discussed with the submitting institution before rendering a final decision. Institutions were encouraged to continue to pursue credentialing and were supported in a second submission when the first was deemed unacceptable. The physics co-chair that performed the independent reviews and the study’s principal investigator that rereviewed potentially unacceptable submissions had approximately 10 and 15 years’ experience, respectively, with liver radiation therapy and image guidance at the beginning of the study.

A subanalysis was performed to determine whether any available factors were related to variations in agreement. This subanalysis is considered hypothesis-generating because of the limitations of the available data. Two-tailed t tests were used to determine whether the mean agreement was different based on number of patients accrued by a center as of January 2019 (zero vs greater than zero), IGRT technology (CBCT vs other), or choice of registration surrogate (soft tissue vs high-contrast surrogate). Motion management was not assessed because of the small number of institutions reporting this information in detail.

Results

Fifty-six IGRT credentialing submissions were submitted between June 2013 and January 2018. Twelve 2D IGRT submissions (6 2D kV fluoroscopy with a robot-mounted linear accelerator, 4 2D kV IGRT with a C-arm linear accelerator, and 2 2D kV with a proton gantry) were excluded because this analysis considers volumetric IGRT only. Images of 130 fractions from 42 institutions remained for analysis. Forty-three were initial submissions, and 2 were repeat submissions. One institution submitted 2 initial submissions for 2 different volumetric IGRT modalities. The institution of the principal investigator and physics cochair was excluded from this analysis to avoid potential bias.

The submission details are provided in Table 1. The majority used CBCT, with some CT-on-rails and megavoltage computed tomography. Most submissions did not provide motion management details. For those who did, abdominal compression was the most common motion management strategy, with assisted breath hold, gating, and free breathing also used. The soft tissue visible in the image was the most common registration surrogate, with high-contrast features such as implanted fiducial markers, stents, surgical clips, and lipiodol also used.

Table 1.

Characteristics of submissions for NRG Oncology RTOG 1112 IGRT credentialing

Variable	Number of submitting institutions (%)	Number of submitted fractions (%)
Submissions
Initial	43 (96%)	126 (97%)
Repeat	2 (4%)	4 (3%)
Country
United States	40 (89%)	108 (83%)
Canada	2 (4%)	7 (5%)
Hong Kong, China	2 (4%)	10 (8%)
Australia	1 (2%)	5 (4%)
IGRT technology
CBCT	41 (91%)	117 (90%)
CT-on-rails	3 (7%)	11 (8%)
MVCT	1 (2%)	2 (2%)
Motion management
Not stated	25 (56%)	65 (50%)
Compression	10 (22%)	31 (24%)
Breath hold	6 (13%)	20 (15%)
Gating	2 (4%)	7 (5%)
Free breathing	2 (4%)	7 (5%)
Registration surrogate
Soft tissue	24 (53%)	63 (48%)
Clips	7 (16%)	18 (14%)
Fiducial markers	6 (13%)	13 (10%)
Stent	4 (9%)	19 (15%)
Lipiodol	4 (9%)	17 (13%)

Abbreviations: CT = computed tomography; CBCT = cone beam computed tomography; IGRT = image guided radiation therapy; MVCT = megavoltage computed tomography; RTOG = Radiation Therapy Oncology Group.

Agreement between institution-reported registration and study-reviewer registration are reported as the largest value in any of 3 orthogonal axes. Over all submitted fractions, the median agreement between submitted registrations and study reviewer was 3 mm, with 95% of all fractions within 5 mm (Fig. 1). The maximum difference in agreement was 21 mm. The standard deviation was 2.5 mm over all fractions. No trends were evident in the axis or direction of large disagreements.

Figure 1.

Frequency of agreement between institution’s reported registration and study reviewer registration for 130 fractions from 42 institutions.

On the subanalysis, only registration surrogate was identified as a statistically significant factor (P < .01) for agreement between the submitting institution and study reviewer. The number of patients accrued by a center and IGRT technology were not significant factors.

Differences between use of low-contrast soft tissue (n = 63 imaged fractions) or high-contrast surrogates (n = 67) (eg, implanted fiducial markers, surgical clips, metallic stents) for registration are shown in Fig. 2. As shown in Table 2, soft tissue registrations agreed within 3 mm in 50% of fractions and 9 mm in 95% of fractions, while high-contrast surrogate registrations agreed within 3 mm in 50% of fractions and 5 mm in 95% of fractions. Soft tissue registrations exceeded 10 mm in 3% of fractions, while no high-contrast surrogate registrations exceeded 5 mm. The standard deviation was 3.3 mm for soft tissue and 1.4 mm for high contrast surrogates. Figure 3 illustrates examples of registrations with large disagreements.

Figure 2.

Frequency of agreement between institution-reported registration and study reviewer for (A) 67 fractions using high contrast features (eg, implanted fiducial markers, lipiodol, metallic stents, or surgical clips) for registration or (B) 63 fractions using soft tissue for registration.

Table 2.

Agreement between submitting institution and study reviewers as a proportion of submitted fractions and registration surrogate

Metric	High contrast	Soft tissue
25% of fractions within	2 mm	2 mm
50% of fractions within	3 mm	3 mm
75% of fractions within	4 mm	4 mm
90% of fractions within	5 mm	5 mm
95% of fractions within	5 mm	9 mm

Figure 3.

Cone beam computed tomography (CBCT) images and overlaid target and liver contours for submitted fractions from different institutions that were scored as unacceptable. Target and liver contours would be overlaid on the CBCT target and liver for an ideal match. (A) Coronal image with contours representing the treated position, indicating the treatment was inferior to the intended location. (B) Sagittal image with contours representing the treated position, indicating the treatment was anterior to the intended location.

Discussion

The RTOG 1112 credentialing experience suggests that most institutions perform liver IGRT with sufficient accuracy to deliver SBRT safely. Ninety-five percent of all submitted fractions were within the acceptable threshold of 5 mm.

There are few reports of the accuracy of IGRT performance being assessed for multi-institutional credentialing activities in the literature. Our observed median 3-mm disagreement is similar to the average displacements seen for Radiation Therapy Oncology Group (RTOG) lung (0915, 0813), spine (0631), and head and neck (0920) studies of around 2 mm.⁶ Our values may be expected to be slightly larger because of greater challenges with target contrast and respiratory motion in liver SBRT. The fact that liver SBRT is an uncommon treatment at most institutions may also increase the variation in performance. Our credentialing process does not examine machine performance, such as examined in a Japan Clinical Oncology Group study.⁷ Their analysis suggests uncertainty associated with machine performance is very small when appropriate quality control processes are in place.^8,9

Our finding that infrequent but large mis-registrations were observed when registering to soft tissue, but not high-contrast objects, appears consistent with inter- and intraobserver IGRT studies. Moseley et al¹⁰ had observers perform registrations of CBCT images of patients with prostate cancer with fiducial markers and repeated the activity with a specialized image reconstruction to remove the fiducial markers. This resulted in a unique data set with a low- and high-contrast target registration analyzed for each patient on each fraction. Interpretation of prostate soft tissue on CBCT was associated with a random geometric uncertainty with a standard deviation of approximately 2 mm, while this uncertainty was only approximately 1 mm when comparing fiducial markers on 2D MV and fiducial markers on CBCT. Morrow et al¹¹ evaluated registrations performed for prostate soft-tissue registration using kV fan-beam CT, kV CBCT, MV fan-beam CT, and MV CBCT. They found the standard deviation of agreement was related to image quality, with a range of about 1 mm for kV fan-beam CT to 2 to 3 mm for MV CBCT. Levegrun et al¹² found that interobserver variation for IGRT spine registrations was slightly better for CBCT versus megavoltage computed tomography because of differences in image quality and resolution. Fiandra et al¹³ evaluated interobserver uncertainty for ultrasound-based prostate IGRT. They observed a difference of 1 to 2 mm in root-mean-square error between users with more than 1 year experience with ultrasound IGRT versus those with less. The authors speculate this is because of experience in interpreting the poor image quality in the superior-inferior direction. These publications, and our current work, are consistent with the intuitive expectation that image registration is more accurate and precise when the target is unambiguous. However, we also need to consider that the median agreement was similar for soft tissue and high-contrast registration in our work. Figure 2 highlights that infrequent large mis-registrations for the soft tissue registration are the only substantial difference between the 2 approaches. High-contrast surrogates are also not all equal: the high-contrast object must be located close to the clinical target volume to be a good surrogate,¹⁴ fiducial markers may not be fixed and migrate during treatment,^15,16 and stents may flex or move over a course of treatment.

From the perspective of submitting institutions, 95% of institutions had 5-mm agreement or better for all submitted fractions. There are some common features of institutions that submitted unacceptable submissions. They were all from the United States, used CBCT, and did not report their motion management approach. However, none of these features were statistically significant and do not explain their results. Target lesion locations included mid-liver, dome of liver, and a malignant thrombus extending into the heart, which does not suggest location strongly determined accuracy. Because all the unacceptable fractions came from only 2 institutions, some hypotheses about IGRT practice may be formed. One may be that most institutions perform very accurate liver IGRT, and a small fraction have some systemic issues. Alternatively, it may be that all institutions, regardless of the quality of their IGRT processes, are at a small risk of mis-registration. It may even be that both hypotheses are true; however, our data do not allow us to explore this question. Encouragingly, it should be noted that all institutions that submitted unacceptable fractions did eventually pass IGRT credentialing.

The results of this analysis should be considered in context. This work is a retrospective report of the IGRT credentialing experience for a clinical trial; it was not conceived as a prospective multi-institutional assessment of IGRT performance. The statistical power to assess associations with failed results is limited because there were only 5 fractions from 2 institutions that failed to meet the criterion for credentialing. Also, although we observed a difference in the mean agreement for high-contrast surrogates of 2.9 versus 4.1 mm for soft tissue registration, the statistical significance associated with this finding is strongly associated with the largest mis-registration of 21 mm. The difference in means is no longer significant if this single value is censored. A larger data set that prospectively collected more detailed data could validate or refute these results. We would encourage future studies by NRG Oncology and other groups to collect more planning, delivery and imaging details, such as motion management information, to allow deeper investigation of these interesting questions.

Additionally, the fact that this was a credentialing exercise with a 5-mm threshold for acceptability may affect the data. For example, if the study reviewer determined a registration that differed from the institution-reported value by exactly 5 mm, that submission would typically be accepted as a pass, and there would be no further analysis. However, if the study reviewer determined a registration that differed by 6 mm, an investigation process would be initiated, potentially including the reviewer reassessing their own registration, a review of the institution’s registration, rereview by the study principal investigator, and discussion with the institution to ensure the reviewers had all relevant information and their approach to the registration was understood. Therefore, some resampling of shifts greater than 5 mm occurred. This is possibly evident in Figs. 1 and 2, with the histograms demonstrating a sharp drop in observations from 5 to 6 mm. Finally, the study reviewer is treated as a surrogate for a true correct registration in this activity; however, their independent registrations are imperfect and contribute some uncertainty themselves.

These results suggest high-contrast target surrogates that are coincidentally available, such as stents, surgical clips, and lipiodol, should be used in the IGRT image registration process whenever possible. In these cases, using the high-contrast target appears to reduce the risk of substantial mis-registration at no cost. The question of whether fiducial marker implantation should be routinely performed is more nuanced. Our results suggest this would benefit IGRT accuracy, but there are requirements and costs, such as the need for an interventional radiology procedure and the risk of toxicity^15,16 associated with the implantation. Our results also suggest soft tissue based IGRT is sufficient in most cases. Considering that we do not have toxicity information from fiducial implantation for IGRT credentialing, that fiducial markers were used in a relatively small fraction of submitted patients, that we do not have comprehensive motion management information, and improvements in IGRT technology, it is difficult for our study to make a broad recommendation to implant fiducial markers or not. However, we can recommend that each institution considering implanting fiducial markers should carefully weigh these factors and the benefits and cost in their environment.

Conclusion

NRG Oncology RTOG 1112 credentialing suggests most institutions perform liver IGRT with sufficient accuracy for safe liver SBRT as assessed by expert reviewers. Both soft tissue and high-contrast surrogates appear adequate for registration in most instances; however, some disagreement was observed when using soft-tissue surrogates for liver tumor IGRT. High-contrast surrogates appear to reduce the small risk of geographic miss owing to mis-registration. These results highlight the valuable role of IGRT credentialing in ensuring high quality RT in liver SBRT studies.