The Radiotherapy Quality Assurance Gap Among Phase III Cancer Clinical Trials

Abstract

Purpose:

Quality assurance (QA) practices improve the quality level of oncology trials by ensuring that the protocol is followed and the results are valid and reproducible. This study investigated the utilization of QA among randomized controlled trials that involve radiotherapy (RT).

Methods and Materials:

We searched ClinicalTrials.gov in February 2020 for all phase III oncology randomized clinical trials (RCTs). These trials were screened for RT-specific RCTs that had published primary trial results. Information regarding QA in each trial was collected from the study publications and trial protocol if available. Two individuals independently performed trial screening and data collection. Pearson’s Chi-square tests analyses were used to assess factors that were associated with QA inclusion in RT trials.

Results:

Forty-two RCTs with RT as the primary intervention or as a mandatory component of the protocol were analyzed; the earliest was started in 1994 and one trial was still active though not recruiting. Twenty-nine (69%) trials mandated RT quality assurance (RTQA) practices as part of the trial protocol, with 19 (45%) trials requiring institutional credentialing. Twenty-one (50%) trials published protocol deviation outcomes. Clinical trials involving advanced radiation techniques (IMRT, VMAT, SRS, SBRT) did not include more RTQA than trials without these advanced techniques (73% vs. 65%, p=0.55). Trials that reported protocol deviation outcomes were associated with mandating RTQA in their protocols as compared to trials that did not report these outcomes (100% vs. 38%, p<0.0001).

Conclusions:

There is a lack of RTQA utilization and transparency in RT clinical trials. It is imperative for RT trials to include increased QA for safe, consistent, and high-quality RT planning and delivery.

INTRODUCTION

Oncology is an innovative multi-disciplinary field that continuously adapts, utilizing new therapies and technologies to improve outcomes. For research findings to translate into improved patient outcomes, clinical trials must be performed with proper design and high quality. Quality assurance (QA) practices help oncology trials reach an acceptable and consistent quality level by ensuring that the trial is performed according to protocol, and that the reported data are valid and reproducible.^1,2 Specifically, QA is an integral component of clinical trials that involve radiotherapy (RT) due to the complex technologies associated with RT delivery, and risks associated with inaccurate or poor-quality RT.^3–7

Calls for radiotherapy quality assurance (RTQA)⁸ began as early as 1968 with the formation of the Radiological Physics Center (RPC), now part of Imaging and Radiation Oncology Core (IROC), which analyzed data validity in RT clinical trials.⁹ Since then, various groups, including the Radiation Therapy Oncology Group (RTOG, now part of the NRG Oncology cooperative group), European Organization for Research and Treatment of Cancer (EORTC), American College of Radiology (ACR), and National Cancer Institute (NCI), have sponsored conferences and studies that highlight the need for RTQA and made recommendations for trials to involve RTQA practices.^6,9–13 Experts from several clinical trials have also encouraged greater utilization of RTQA after post-hoc analyses of some trials showed that lapses in RTQA was associated with a detriment in local control and survival.^8,14–16 Despite these actions, a standardized set of required RTQA practices does not exist for clinical trials that involve the use of RT, and the patterns of RTQA in clinical trials are unknown. Herein, we investigate the utilization of RTQA practices in RT phase III cancer RCTs.

MATERIALS AND METHODS

Phase III oncologic trials were identified through a search of the ClinicalTrials.gov registry using the following advanced parameters: Terms: “cancer”; Status: excluded: “Not yet recruiting”; Phase: Phase 3; and Study Results: “With Results.” Performed in February 2020, this search yielded 1,877 trials. We then identified 42 trials that used RT as the primary intervention or as a mandatory part of the treatment protocol (Figure 1). We searched for information regarding any RTQA practice that was conducted for each trial using ClinicalTrials.gov, primary and secondary publications of results, cooperative group and IROC online records, and the Clinical Trials Support Unit (CTSU) database. A majority of studies (37 trials, 88%) had publicly-available study protocols, which were also used for data extraction. Two individuals independently performed trial screening and data collection. According to Donabedian’s structure-process-outcome QA theory,^17,18 which has been widely used to formulate RTQA practices,^2,3 and guidance from the Global QA of Radiation Therapy Clinical Trials Harmonization Group (GHG),¹² the following characteristics were identified as important for reporting in RT clinical trials and were extracted from each study: RT modality, energy, technique, image guidance, target volume, field margins, motion control, total dose, and fractionation schedule. Advanced RT techniques were defined as volume-modulated arc therapy (VMAT), intensity-modulated radiation therapy (IMRT), stereotactic radiosurgery (SRS), and stereotactic-body radiation therapy (SBRT). Studies were assessed for mandating institutional credentialing requirements and for reporting protocol deviation outcomes in their trial-associated publications. Additionally, we assessed each trial for inclusion of a RTQA protocol that mandated QA practices. Inclusion of a RTQA protocol was a binary variable (specification of any RTQA practice in the protocol versus no RTQA mentioned in the protocol). It was noted whether the QA was done before the start of treatment or during real-time, “pre-treatment RTQA”, or after RT completion, “post-treatment RTQA”.

Figure 1.

Flowchart of trial screening and eligibility.

Pre-treatment RTQA included:

• Protocol-specific phantom run

• Review of simulation images, contours, or dose distribution

• Machine calibration measurements

• Review of portal film or other image-guided RT images

Post-treatment RTQA practices included:

• Review of case contours or dose distribution

• Review or portal film or other image-guided RT images

• Verification of patient set-up and protocol-derived plan delivery

Descriptive statistics were used to characterize each trial and its RTQA practices. Statistical significance was defined as p<0.05. Pearson’s Chi-square test was used to analyze the association between trial-related factors and the use of RTQA protocols (SPSS, Version 26.0).

RESULTS

Forty-two phase III trials with RT as either the primary intervention (21 trials) or as a mandatory component of the treatment protocol (21 trials) were identified (Supplementary Table). Thirty-two (76%) trials had NIH funding and 33 (79%) had cooperative group sponsorship. The majority of RCTs analyzed non-small-cell lung cancer, prostate cancer, central nervous system cancer, or head-and-neck cancer (6 trials [18%] each). Forty-one (98%) trials evaluated the adult population and one (2%) trial evaluated pediatric patients. Nineteen trials (45%) used advanced radiation technologies (VMAT, IMRT, SRS, or SBRT). There was increased use of advanced radiation technologies in trials which began in 2005 and onward. Of trials that did not use advanced radiation technologies, 16 (38%) used 2D/3DRT. Four trials used brachytherapy (10%), two used total body irradiation (5%), and one used selective-internal radiation therapy (2%). Trial characteristics are highlighted in Table 1.

Table 1:

Trial characteristics and their association with the inclusion of RTQA protocols in radiotherapy phase III cancer RCTs.

	N/Ntotal (%)	RTQA Protocol Inclusion N/Ntotal (%)a	P-Valueb
RTQA protocol inclusiona
Yes	28/42 (66.7%)	-	-
No	14/42 (33.3%)	-	-
Year of enrollment initiation
1991–2000	5/42 (11.9%)	5/5 (100%)	0.24
2001–2010	32/42 (76.2%)	20/32 (62.5%)
2011–2020	5/42 (11.9%)	3/5 (60.0%)
Trial enrollment
0–499 participants	17/42 (40.5%)	9/17 (52.9%)	0.28
500–999 participants	18/42 (42.9%)	14/18 (77.8%)
1,000+ participants	7/42 (16.7%)	5/7 (71.4%)
Trial geographyc
Multi-national	32/42 (76.2%)	21/32 (65.6%)	0.80
Uni-national	10/42 (23.8%)	7/10 (70%)
Disease staged
Non-metastatic	26/42 (61.9%)	20/26 (76.9%)	0.08
Metastatic	7/42 (16.7%)	3/7 (42.9%)
Industry funding
No	28/42 (66.7%)	19/28 (67.9%)	0.81
Yes	14/42 (33.3%)	9/14 (64.3%)
Cooperative Group sponsorship
No	9/42 (21.4%)	4/9 (44.4%)	0.11
Yes	33/42 (78.6%)	24/33 (72.7%)
Radiotherapy intervention
Primary intervention	21/42 (50.0%)	13/21 (61.9%)	0.51
Non-randomized interventione	21/42 (50.0%)	15/21 (71.4%)
Advanced radiation techniquef
No	23/42 (54.8%)	15/23 (65.2%)	0.83
Yes	19/42 (45.2%)	13/19 (68.4%)
Protocol deviation reportedg
No	21/42 (50.0%)	8/21 (38.1%)	<0.001
Yes	21/42 (50.0%)	19/21 (90.5%)
Trials with PEP success
No	28/42 (66.7%)	16/28 (57.1%)	0.06
Yes	14/42 (33.3%)	12/14 (85.7%)
Primary intervention radiotherapy trials with PEP success
No	15/21 (71.4%)	8/15 (53.3%)	0.21
Yes	6/21 (28.6%)	5/6 (83.3%)

Abbreviations: RTQA, radiotherapy quality assurance; RCT, randomized controlled trial; VMAT, volume-modulated arc therapy; IMRT, intensity-modulated radiation therapy; SRS, stereotactic radiosurgery; SBRT, stereotactic-body radiation therapy; PEP, primary end-point.

^aRTQA protocols include: phantom analysis, simulation image review, dosimetry/prescription review, machine calibration measurements, real-time review, treatment plan analysis.

^bFor all included trials, the p-value represents the results of a Pearson’s Chi-square test for RTQA protocol inclusion in each trial factor.

^cTrial geography was defined as the geographic distribution of centers accruing to the trial.

^dCentral nervous system and hematological malignancy trials were excluded in the disease stage analysis.

^eNon-randomized intervention was defined as trials that included RT as a mandatory part of the protocol, but not part of the randomization scheme.

^fAdvanced radiation techniques include: IMRT, VMAT, SRS, and SBRT.

^gTrials that reported protocol deviation outcomes in their trial-associated publications.

We assessed the RT characteristics reported by each trial and whether trials required institutional credentialing, included a RTQA protocol that mandated any QA as part of the trial design, and presented protocol deviation outcomes (Table 2). All 42 trials reported the total dose and fractionation schedule used. Of the trials studying cancer sites that warrant motion management (n=18), 13 trials (72%) reported the use of motion control techniques. Notably, RTQA data or associated RTQA results were not publicly available for any study. Of all 42 trials, nineteen (45%) required institutional credentialing to allow for participation in the trial. All four trials that included brachytherapy required credentialing. Of the 19 trials that used advanced radiation technologies, 11 (58%) required institutional credentialing. The most common components of institutional credentialing were phantom analysis and facility questionnaire (16 trials [84%] each). Two-thirds of included trials (29 trials, 69%) reported that they mandated RTQA protocols. Of these, 5 trials (17%) included pre-treatment RTQA alone, 12 (41%) included post-treatment RTQA alone, 10 (34%) used both pre- and post-treatment RTQA, and 2 (7%) were unknown. All 13 trials that included motion control techniques required RTQA. Of all 42 trials, three (7%) mandated that a phantom run of the prescribed treatment was completed and successful before initiating RT for the first trial patient at an institution. Finally, 50% of all trials reported protocol deviations and associated outcomes in their primary or secondary publication.

Table 2:

RTQA in radiotherapy phase III cancer RCTs stratified by (a) general RT characteristics reported, (b) RTQA protocol requirements, and (c) institutional credentialing requirements.

(a):

RT Characteristics Reported	N/Ntotal (%)
Source
No	4/42 (9.5%)
Yes	38/42 (90.5%)
Energy
No	12/42 (28.5%)
Yes	29/42 (69.0%)
Image guidance
No	26/42 (61.9%)
Yes	13/42 (31.0%)
Target volume
No	5/42 (11.9%)
Yes	34/42 (81.0%)
Field margins
No	7/40 (17.5%)
Yes	32/40 (80.0%)
Not applicable	2/42 (4.8%)
Motion control
No	5/18 (27.8%)
Yes	13/18 (72.2%)
Not applicable	24/42 (57.1%)
Total dose
No	0/42 (0.0%)
Yes	42/42 (100.0%)
Fractionation
No	0/42 (0.0%)
Yes	42/42 (100.0%)

Abbreviations: RTQA, radiotherapy quality assurance; RCT, randomized controlled trial.

For all 42 trials, we examined factors associated with including a mandated RTQA protocol (Table 1). Utilization of RTQA protocols did not significantly change over time (100% in trials initiated from 1991–2000, 62.5% in trials from 2001–2010, 60% in trials from 2011–2020; p=0.24). Trials that used advanced radiation technologies were not more likely to implement RTQA protocols when compared to trials that did not use advanced techniques (73% vs. 65%, p=0.55). Trials that reported protocol deviation outcomes were associated with having mandated RTQA protocols (100% vs. 38% for trials that did not report deviation outcomes, p<0.0001). Trials that studied non-metastatic disease also included RTQA more frequently (81% vs. 43% for trials that studied metastatic disease, p=0.05). The 14 (33%) positive trials (trials where PEP was met) tended to include RTQA protocols in their trial design more often than negative trials (86% vs. 61%, respectively, p=0.06).

DISCUSSION

This study provides an overview of the RTQA profile of phase III cancer trials where RT was an integral treatment component. We found that two-thirds of trials implemented RTQA protocols into their study design. In these trials, RTQA most commonly included post-treatment QA through a central review; half of trials lacked pre-treatment QA. Trials utilizing advanced radiation techniques did not incorporate RTQA more frequently into their protocols, and institutional credentialing was required in slightly more than half of trials using advanced techniques. Finally, half of all included RT trials did not report information about protocol deviations in any trial-associated publication.

While limited in number, other studies have corroborated the suboptimal use of QA in radiotherapy research.^19–21 One study reviewed RTQA in lymphoma RCTs and found that only 38% of trials described the delineated target volume and 20% of trials used a RTQA plan.¹⁹ Another study on prostate cancer RCTs similarly found that a majority of trials (60%) neither used RTQA nor adequately reported the RT process.²⁰ Insufficient RTQA reporting has also been found in preclinical RT research, as one study found that even though radiation energy, source, and dose were well-described in these studies, there were insufficient descriptions of physics and dosimetry QA, such as calibration and geometry.²¹ The ability to utilize QA to safely and successfully treat patients with RT is a guiding principle of radiation oncology; thus, the lack of RTQA information available both in studies and publicly is sobering.

Utilization of and adherence to RTQA can lead to better patient and trial outcomes.^{6,7,15,22–25} Zhong et al. found that RTQA was associated with better patient outcomes in head and neck cancer phase III RCTs.¹⁶ Additionally, mandatory RTQA has been associated with studies having a lower number of major violations in treatment plans and delivery,^26,27 especially through pre-treatment QA.^6,28–30 The majority of mistakes leading to violations have been found to occur in the planning process,³¹ which emphasizes the importance of pre-treatment RTQA and creates concern that our study showed that many trials lack pre-treatment QA. This may increase the risk that patients do not receive the protocol-directed radiation plan, potentially leading to worse outcomes and/or increased toxicity.⁶ In turn, this may diminish the capacity of a clinical trial to reliably and accurately answer a particular question, which may increase the number of patients required for trial enrollment and drive up trial costs. Thus, pre-treatment RTQA not only improves patient outcomes, but is also advantageous for trial design and outcomes.²⁵ Another aspect of RTQA that trials should consider is the use of a central QA review. Not only does a central QA review ensure all clinical trial sites are following protocol, but it also promotes dissemination and proper use of new radiation technologies for more patients. While many may acknowledge that RTQA is a meaningful correlate of outcomes, until we comprehensively utilize RTQA and publish information about these practices, we cannot ensure proper RT delivery and reproducibility in clinical trials.⁶

In the past 20 years, RT has expanded to routinely use VMAT, IMRT, SRS, SBRT, and proton therapy. With more advanced technology comes more complex imaging, RT plans and delivery,⁶ which consequently increases the risk of adverse events. One study found that QA violations occurred more frequently in patients requiring more complex radiation plans, for instance in patients with greater treatment volumes or with more advanced disease.²⁶ Moreover, the accurate delivery of advanced RT has been repeatedly shown as suboptimal by IROC,^32,33 including suboptimal beam modeling in the treatment planning system.³⁴ As such, QA is of the utmost importance in trials assessing advanced radiation and imaging technologies. Once institutions demonstrate the ability to complete adequate basic QA for advanced technologies, radiation modality-specific QA should be considered to optimize the treatment process given the complex nature of advanced technologies. However, our results showed that published trials using advanced radiation technologies did not include more protocol RTQA. Furthermore, our study showed that 42% of trials using advanced radiation techniques did not require institutional credentialing despite prior reports demonstrating the association of credentialing with less treatment variability.^6,7 Current RTQA in trials studying advanced radiation techniques is inadequate, which is worrisome for future, and potentially more complex, trials. There is a clear opportunity to improve trials using advanced radiation techniques by mandating institutional credentialing and radiation modality-specific RTQA in trial protocols.

Protocol deviations are another critical aspect of clinical trials that warrant reporting. Some protocol deviations are a result of complex tumor volumes and dosimetry constraints, however other deviations are due to errors in RT planning and delivery, which are associated with suboptimal RT quality and may have a negative effect on patient outcomes.^{7,22–21, 35,36} Consequently, it is concerning that our results and others³⁷ have demonstrated that only a minority of trials publish protocol deviation details. As much as possible, investigators should provide protocol deviation and QA information that may have contributed to patient harm to provide learning opportunities for future trials.

To improve protocol compliance and QA transparency, clinical trial teams should better integrate physicists as key members in RCTs.^6,7 Physicist collaboration is essential for RCT design, completion of machine benchmark tests, and accurate data preparation and submission.⁷ Physicists are also essential for clinical trial RTQA to ensure that participating institutions are using the protocol technology accurately and consistently.³⁸ Moreover, innovations in medical physics have greatly contributed to the advances in radiation oncology over the past twenty years, further supporting the close collaboration between physicists and radiation oncologists.³⁹ Better RTQA along with close physics collaboration could help investigators quantify their protocol reliability, decrease the number of protocol deviations, and improve research quality, consequently improving confidence in trial results and ultimately patient outcomes.^6,7,22,30

Even though evidence supports the need for RTQA, several challenges exist for investigators to implement RTQA into their clinical trial protocols. First, the purpose of RTQA is to identify errors and deficiencies in RT planning and delivery; however, resolving these issues takes time and expense.^22,40 Specifically, pre-treatment centralized plan review may create challenges with time constraints and delay of treatment. However, IROC has a three-day turn-around policy when reviewing pre-treatment plans to mitigate the potential for treatment delay. Additionally, RT nuances exist for each cancer site, such as daily target localization for prostate cancer and post-implant dosimetric analysis for brachytherapy,³ which make standardizing RTQA procedures across RCTs difficult.³ Finally, adequate funding is essential for trials to perform RTQA. Upfront funding on RTQA is not only important for proper QA capabilities and adherence, but it also serves as a “return-on-investment” by saving future expenses that may be needed to overcome trial failures. Soon et al. showed that RCTs sponsored by cooperative groups were more likely to include RTQA.²⁰ As cooperative groups continue to support RTQA, both private and public sponsoring bodies should strongly consider the benefits of RTQA and prioritize funding RTQA to ensure optimal trial quality and reproducibility.^6,7

Currently, RTQA differs between trials with no required set of practices for RT RCTs. The Global QA of Radiation Therapy Clinical Trials Harmonization Group (GHG) is one organization that aims to improve clinical trial validity by centralizing the RTQA process.^4,41,42 The GHG published 10 “harmonized RTQA procedures” that could be done with every trial to encourage collaboration, international consistency, reproducibility, and greater confidence of findings. The 10 procedures compromise a thorough set of pre- and post-treatment QA practices: facility questionnaire, beam output audit, benchmark case, dummy run, complex treatment dosimetry check, virtual phantom, individual case review, review of patients’ treatment records, and protocol compliance and dosimetry site visit.¹² Yet, our study found that only one-third of RCTs use both pre- and post-treatment RTQA. Moreover, we found that none of the studies have made their QA or results data publicly available. Thus, the GHG should further disseminate their harmonized RTQA procedures and future investigators should consider transparency of their methods and results to create collaborative, reproducible, and higher-quality RT RCTs.⁶ Not only does this represent a critical unmet need in RT trials, but it also represents a potential for patient harm unless more resources are devoted to improving the quality of RT RCTs.

Our study has several limitations. First, we analyzed only those RCTs with posted results on ClinicalTrials.Gov. Even though posting results to ClinicalTrials.Gov is mandatory for trials that accrue in the United States, our search strategy missed any trials that that did not accrue in the United States or did not post their results to ClinicalTrials.Gov. However, despite this limitation and that our study did not examine ongoing trials, our analysis included trials using advanced radiation techniques, which demonstrates that utilization of RTQA methodology should be improved among modern RT trials to ensure rigorous and high-quality trials. While trials using advanced radiation techniques are newer and may have future secondary analyses discussing RTQA, we believe that QA information should be made available in each trial’s primary publication or supplementary materials for reasons as described above. Additionally, phase I-II trials were not captured with our search strategy, and may also be important in understanding utilization patterns of RTQA and how RTQA (or lack thereof) in earlier phase trials may impact subsequent late-phase study results and reliability. Finally, only publicly-available information was identified and captured, and a minority of trials (12%) did not have publicly-available protocols. Trials with RTQA descriptions, processes, or components that were not made public may not have been captured.

CONCLUSIONS

RTQA plays a critical role in validating radiotherapy RCTs, which allows for advances in clinical practice. Some RCTs have both pre- and post-treatment RTQA, and provide detail about their RTQA practices. However, a majority of RTQA is inadequate as many trials do not include pre-treatment RTQA, institutional credentialing, or information about protocol deviations. Additionally, with increased utilization of advanced and more complex radiation techniques, it is imperative that clinical trials mandate robust RTQA and involve medical physicists in the RTQA design and planning process. Finally, investigators should be encouraged to make their trial RTQA publicly available to allow for repetition and successful translation of trial results into clinical practice. Through further analyses and consensus trial recommendations, improving clinical trial RTQA and increasing the reporting of RTQA methodology and outcomes are necessary to improve the integrity of clinical trials and, ultimately, improve patient care.

(b):

RTQA Protocol Requirements	N/Ntotal (%)
RTQA protocol
No	13/42 (30.1%)
Yes	29/42 (69.0%)
RTQA protocol typea
Pre-treatment	5/29 (17.2%)
Post-treatment	11/29 (37.9%)
Both	10/29 (34.4%)
Unknown	2/29 (6.9%)
Pre-treatment RTQA componentsb
Simulation image review	11/15 (73.3%)
Dosimetry/prescription review	11/15 (73.3%)
Phantom analysis	3/15 (20.0%)
Machine calibration measurements	2/15 (13.3%)
Portal film review	10/15 (66.7%)
Post-treatment RTQA componentsc
Dosimetry review	19/21 (90.5%)
Treatment set-up review	18/21 (85.7%)
Unknown	1/21 (4.7%)
RTQA protocol reviewa
Local institution review	4/29 (13.8%)
Central review	23/29 (79.3%)

^aPercentage breakdown in these categories were based on the number of trials which included RTQA protocols.

^bPercentage breakdown in this category was based on the number of trials which included pre-treatment RTQA protocol requirements. Trials may have contained more than one pre-treatment RTQA component.

^cPercentage breakdown in this category was based on the number of trials which included post-treatment RTQA protocol requirements. Trials may have contained more than one post-treatment RTQA component.

(c):

Institutional Credentialing Requirements	N/Ntotal (%)
Institutional credentialing
No	24/42 (57.1%)
Yes	18/42 (42.9%)
Credentialing componentsa
Phantom analysis	15/18 (83.3%)
Facility questionnaire	15/18 (83.3%)
Knowledge assessment	13/18 (72.2%)
Radiation source registration	3/18 (16.7%)
Treatment planning system verification	2/18 (11.1%)

^aPercentage breakdown in this category was based on the number of trials which required institutional credentialing.

Highlights:

• Two-thirds of all radiotherapy (RT) randomized controlled trials (RCTs) mandated RT quality assurance (RTQA).

• Less than half of trials required institutional credentialing.

• Clinical trials involving advanced radiation techniques were not associated with utilization of RTQA.

• Half of radiotherapy RCTs published protocol deviation outcomes.

• There is a need for greater use of RTQA in RCTs to ensure protocol-specified RT is being delivered safely.

Abbreviations:

QA

Quality assurance

RT

Radiotherapy

RTQA

Radiotherapy quality assurance

RCT

Randomized controlled trial