Challenges in Credentialing Institutions and Participants in Advanced Technology Multi-institutional Clinical Trials

Abstract

The Radiological Physics Center (RPC) has functioned continuously for 38 years to assure NCI and the cooperative groups that institutions participating in multi-institutional trials can be expected to deliver radiation treatments that are clinically comparable to those delivered by other institutions in the cooperative groups. To accomplish this, the RPC monitors the machine output, the dosimetry data utilized by the institutions, the calculation algorithms used for treatment planning, and the institutions’ quality control procedures. The methods of monitoring include on-site dosimetry review by an RPC physicist, and a variety of remote audit tools. The introduction of advanced technology clinical trials has prompted several study groups to require participating institutions and personnel to become credentialed, to assure their familiarity and capability with techniques such as 3DCRT, IMRT, SBRT and brachytherapy. The RPC conducts a variety of credentialing activities, beginning with questionnaires to evaluate an institution’s understanding of the protocol and their capabilities. Treatment planning benchmarks are used to allow the institution to demonstrate their planning ability, and to facilitate a review of the accuracy of treatment planning systems under relevant conditions. The RPC also provides mailable anthropomorphic phantoms to verify tumor dose delivery for special treatment techniques. While conducting these reviews, the RPC has amassed a large amount of data describing the dosimetry at participating institutions. Representative data from the monitoring programs will be discussed and examples will be presented of specific instances in which the RPC contributed to the discovery and resolution of dosimetry errors.

Introduction

The RPC was established in the late 1960s as a resource in radiation dosimetry and physics for cooperative clinical trial groups and the radiation therapy facilities that deliver radiation treatment to patients entered into cooperative group protocols. The primary responsibility of the RPC is to assure NCI and the cooperative groups that participating institutions have adequate quality assurance procedures and no major systematic dosimetry discrepancies, so that they can be expected to deliver radiation treatments that are clinically comparable to those delivered by other institutions in the cooperative groups. To accomplish this, the RPC monitors the basic machine output and brachytherapy source strengths, the dosimetry data utilized by the institutions, the calculation algorithms used for treatment planning, and the institutions’ QA procedures. The methods of monitoring consist of on-site dosimetry reviews by an RPC physicist and a variety of remote auditing tools. The remote auditing tools include mailable anthropomorphic phantoms to verify tumor dose delivery for special treatment techniques; and credentialing of institutions and personnel wishing to participate in clinical trials utilizing advanced technologies.

Several cooperative groups have determined that the technologies used in certain clinical trials are sufficiently advanced to warrant credentialing of institutions that wish to participate in these trials. To conduct credentialing activities, the RPC works with all of the NCI-sponsored cooperative groups, either directly or in collaboration with the Advanced Technology QA Consortium (ATC, http://atc.wustl.edu), a consortium of QA offices funded by the NCI. [1,2]

Trials Requiring Credentialing

In recent years, several study groups have recognized that the introduction of advanced technologies into clinical trials can jeopardize the success of the trial unless the ability of institutions to perform these advanced technologies is evaluated [3,4]. The Radiation Therapy Oncology Group (RTOG) has led the way in clinical trial credentialing by instituting requirements for a number of recent clinical trials.

Purpose and Benefits of Credentialing

Credentialing offers a number of benefits. Chief among these is the education of staff at the participating institution to ensure they understand the protocol and its goals. As an example, the knowledge assessment questionnaire that is required for RTOG 0413/NSABP B-39 requires physicians to complete a short ten-question questionnaire. Questions ask about details of the prescription, whether or not heterogeneity corrections are to be used, and limitations of doses to normal tissues, etc. The knowledge assessment questionnaires can be completed online using web-based forms, the results of which are submitted digitally and automatically. The questionnaires are reviewed by dosimetrists at the RPC. Questionnaires with incorrect responses are returned to the physician with instructions to review the protocol.

Other credentialing requirements, such as a facility questionnaire, enable the collection of data from institutions, particularly the types of equipment to be used for treating patients and the QA procedures that will be followed by institutional staff. Treatment planning benchmarks and anthropomorphic phantom irradiation experiments evaluate the institution’s ability to deliver the treatment modality required by the protocol. It should be noted that a benchmark is an evaluation of the treatment planning system’s capabilities, and the institution’s ability to use them, but does not evaluate the institution’s imaging capabilities, nor their ability to correctly deliver the treatment to a patient. The facility questionnaires are reviewed by RPC dosimetrists and benchmark plans are verified by independent calculations performed at the RPC.

The primary goal of credentialing is to reduce the deviation rate for data submitted to clinical trials. Cooperative groups have experienced deviation rates that sometimes amount to as much as 17% of the cases submitted, according to a study conducted by the RPC.[5] An elevated number of deviations reduces the quality of the study, and increased rates of major deviations may limit accrual to the trial. Credentialing evaluations result in feedback to the institution, to explain the results of the procedure and suggestions for improving those results in the future.

Specific Requirements

3D CRT

Several years ago, the NCCTG began requiring that all members become credentialed using a 3D CRT treatment planning benchmark developed and made available jointly by RPC and the Quality Assurance Review Center (QARC, http://www.qarc.org) under the ATC umbrella. The credentialing procedures require that institutions complete questionnaires requesting demographic and equipment, QA procedures, and an evaluation of the physician’s understanding of the protocol. A benchmark treatment plan allows institutions to demonstrate their capability to perform 3D treatment plans. The RPC reviews these treatment plans, and independently confirms the accuracy of dose calculation.

IMRT

Protocols such as RTOG 0521 allow an institution to become credentialed by submitting an IMRT treatment planning benchmark. In this case, the benchmark must specifically demonstrate the institution’s use of IMRT and inverse planning, and the ability of the institution to generate a dose distribution conforming to a complex structure with a concave surface and a nearby organ at risk. A treatment planning benchmark developed by QARC is used for this purpose.[6] However, most RTOG trials evaluate an institution’s use of IMRT by requiring the irradiation of a standardized anthropomorphic phantom, such as the RPC head and neck phantom shown in Figure 1 [7–10]. Irradiation of the standardized phantom provides a comprehensive evaluation of the institution’s ability to perform appropriate imaging, accurate treatment planning, and precise treatment delivery.

Figure 1.

The RPC’s head and neck phantom.

SRS/SBRT

Several recent RTOG protocols have required evaluation not only of the institution’s ability to image, plan, and deliver a stereotactic treatment, but also an evaluation of the institution’s procedures for immobilizing patients and confirming correct repositioning of the patient for daily treatment. For these trials, such as RTOG 0236, the use of an anthropomorphic phantom allowed the institution to demonstrate the appropriateness of the imaging, planning and treatment delivery procedures as well as the accurate placement of the dose distribution and accurate delivery of dose [11–13]. The institution’s imaging capabilities and procedures for confirming patient position were evaluated separately through the use of questionnaires and imaging data. For another recent SBRT protocol, RTOG 0438, institutions were required to image and treat a phantom that moved in a manner that mimicked respiratory motion. Institutions that had the capability to gate treatment delivery or track tumor motion during delivery could demonstrate the accuracy of those techniques with the RPC moving phantom.

Brachytherapy

Several recent protocols required evaluation of each participating institution’s ability to deliver LDR or HDR brachytherapy. RPC collaborated with QARC to develop a credentialing mechanism that has been implemented by both RTOG and the American College of Surgeons Oncology Group. Questionnaires and benchmarks verified that the institution performed adequate quality assurance procedures, confirmed that their source calibration procedures were acceptable and were traceable to a national standard, tested the institution’s calculation accuracy for a single source, and evaluated their calculation of a benchmark treatment plan.

Results of credentialing

The RPC’s credentialing and retrospective review programs have revealed several important characteristics of institutions participating in clinical trials.

• Most institutions completed credentialing procedures relatively quickly and without undue effort. Credentialing questionnaires, especially those completed on-line, must be completed before they are submitted, and require participants to resolve their questions before submission, thus satisfying the educational component of the credentialing process.

• Treatment planning benchmarks occasionally demonstrate software errors and have revealed incorrect data entry. The RPC’s brachytherapy benchmark requirements have detected misunderstandings in the application of the updated TG-43 protocol [14] and in the implementation of corrections to the NIST 1999 standard for some seed sources [15].

• Irradiation of an anthropomorphic phantom is a comprehensive test of an institution’s ability to image, plan and treat a patient, and the process frequently demonstrates failings in the institution’s own QA procedures. As of this date, roughly 30 % of institutions failed to deliver a dose distribution to the H&N phantom that agrees with their own treatment plan to within 7 % or 4 mm [9, 10, 12, 16] (See Table 1). Similar experience, although with somewhat higher pass rates, has been seen with the other RPC phantoms. Phantom irradiations as part of credentialing have identified the following errors:

  • Incorrect output factors and percent depth dose data entered by the institution.

  • Inadequate modeling of the penumbra at MLC leaf ends [17]

  • Incorrect application of QA calculations or measurements

  • Inadequate QA of multileaf collimator

  • Incorrect patient positioning, including couch indexing errors with serial tomography system

  • Errors in treatment planning software

• Credentialing has been shown to result in reduced deviation rates in clinical trials. The RPC recently evaluated the deviation rates of several trials that required credentialing for some or all participants [5, 18, 19]. The results, shown in Table 2, indicated that trials requiring credentialing experienced low deviation rates. Three protocols for which credentialing was required of all participants had rates of deviation between 0% and 4% while two protocols that had limited credentialing requirements had rates of deviation on the order of 7% to 17%. In another study some institutions were credentialed for one technique but not for another. Credentialed institutions received no deviations on the protocol, while the remaining institutions had a deviation rate of 16%. Similar findings have been reported elsewhere.[20]

Table 1.

Institution passing rates with the RPC phantoms

Phantom	H&N	Prostate	Thorax	Liver
Irradiations	250	64	24	4
Pass	179	55	17	3
Fail	71	9	7	1
Year introduced	2001	2004	2004	2005

Challenges to Credentialing

While the benefits of credentialing have been demonstrated, resistance to the imposition of credentialing requirements is sometimes exhibited by study groups, study chairs, and participating institutions. The reasons given include the time required for completing the requirements by an institution, and the loss from the pool of contributing institutions of those that fail.

The decision to require credentialing demands that a compromise be struck between the spectrum of institutions that will be permitted to participate in the protocol, and the uniformity and quality of the data to be collected. By its nature, credentialing will limit the participating institutions to those who can pass the requirements, concomitantly reducing the number of patients likely to be enrolled. At the same time, data described above indicate that credentialed institutions are more likely to comply with the requirements of the protocol, and to submit patient data that will be useful for the study.

Another criticism is that institutions often have to wait to receive a phantom, when irradiation of a phantom is one of the credentialing requirements. This is regrettable, but unavoidable, as it is impractical to construct a large enough number of phantoms to always be able to provide them on demand. However, between the construction of additional phantoms, and the establishment of criteria for prioritizing institutions, the RPC has been able to provide phantoms within a few days to institutions that meet the following criteria:

• The protocol has been approved by the institution’s IRB,

• The institution has requested and received an account with the ITC,

• The institution is a study group member and is otherwise qualified to participate in the protocol for which they seek credentialing.

In addition, institutions that are expected to contribute significiantly to the trial, based on their past participation, are given higher priority. Institutions that have previously received an RPC phantom and either delayed returning it, or returned it unirradiated (for reasons other than equipment failure) are moved to lower priority.

Conclusions

Several NCI-funded study groups have instituted credentialing requirements for specific clinical trials. This action was prompted in part by guidelines published by the NCI and in part by the groups’ desire to assure that advanced treatment technologies are used effectively in these trials. Credentialing demands additional effort by the participating institutions over the existing requirements imposed by clinical trials. It also challenges quality assurance offices such as the RPC to design meaningful credentialing procedures that are not unduly onerous.

The benefits of credentialing have been demonstrated. Institutions that were required to undergo a credentialing process were better prepared to comply with the requirements of the protocol. This may be because the credentialing process provides feedback on how better to comply with the treatment protocol prior to submitting a patient onto the study. Therefore, the frequency of deviations can be reduced for institutions that go through the credentialing process.

Table 2.

The effect of credentialing on the deviation rate of several trials.

Study	Major Deviations	Minor Deviations	Number of Patients
GOG 165, HDR Cervix
Credentialed Inst	0	15	70
Non-credentialed	57	87	275
RTOG 95–17, HDR & LDR Breast(all)	0	4	100
RTOG 0019, LDR Prostate	0	6	117