ACRIN—lessons learned in conducting multi-center trials of imaging and cancer

Abstract

The American College of Radiology Imaging Network (ACRIN) is a US National Cancer Institute-funded clinical trials cooperative group charged with conducting multi-center clinical trials of diagnostic imaging and image-guided treatment technologies as they are employed in the detection, diagnosis and staging, treatment, and evaluation of treatment for cancer. Operating since 1999, ACRIN involves participating institutions around the world and hundreds of radiologists, methodologists, and scientists in the 22 trials it has been working on to date, including several large screening trials. The experience with ACRIN has elucidated the unique requirements that must be fulfilled by imaging trials if they are to be successful, particularly in such areas as trials design, definition of technologies and their findings, quality assurance, and ensuring sufficient accrual. ACRIN is now pursuing several courses of action that will disseminate the products devolving from ACRIN trials into the public domain, so that they may benefit other investigators.

Introduction

Almost certainly, the most frequent applications of medical imaging are to either ‘rule out’ or help manage cancer. Indeed, much of the medical imaging literature details the use of imaging technology for these purposes or evaluates its performance. However, the vast majority of this literature is comprised of single institutional studies that frequently fail to fulfill high standards of validity, reliability, and most critically, generalizability. Such studies suffer in credibility for lack of institutional resources, the lack of multi-disciplinary input, and the fact that samples of both subjects and interpreters inadequately reflect the breadth of patients and practitioners encountered in the world of clinical practice. The result is poor guidance from our research on what represents the most appropriate imaging care for cancer.

In the United States, for nearly a half century, cancer treatment researchers have combated these concerns by collaborating in National Cancer Institute (NCI)-funded multi-center clinical trial cooperative groups. These groups leverage the input of diverse, multi-disciplinary experts and the accruing power of many institutions’ participation in a trial to generate definitive and generalizable assessments of new treatments that can guide the actions of practitioners. As examples, some of the better known therapeutic cooperative groups include the Cancer and Leukemia Group B (CALGB), the Southwest Oncology Group (SWOG) and the Eastern Oncology Group (ECOG). Indeed, there are cooperative groups that focus on medical, surgical, and radiation oncology, and some that combine trials in all of these disciplines. However, until recently, there was no cooperative group that focused on imaging.

In 1997, recognizing the rising importance of imaging to cancer, then Deputy Director of the NCI, Robert Wittes, MD, directed the newly conceived Biomedical Imaging Program (BIP, later to become the Cancer Imaging Program, or CIP, under the direction of Daniel Sullivan, MD) to develop a request for applications (RFA) to create a new cooperative group that would fulfill the perceived need for high quality imaging clinical trials. The resultant competitive grant was awarded to the American College of Radiology Imaging Network (ACRIN), which began operations in March 1999 [1]. ACRIN has since had its core funding renewed through 2007 and received a number of administrative supplements to conduct specific trials that it either has proposed to NCI or been asked to consider by other cooperative groups. Expended moneys to date and promised funds through 2009 exceed US$200 million.

ACRIN’s mission and strategy

ACRIN’s overarching mission is, through clinical trials of diagnostic imaging and image-guided treatment to:

• improve the length and quality of the lives of cancer patients;

• promote the earlier detection of cancer so as to improve the likelihood of a cure;

• reduce the anxiety of those who believe they might have cancer but do not.

To facilitate achieving these objectives, ACRIN developed five key hypotheses to which it addresses all of its trials [2]:

• Imaging screening can reduce cancer mortality.

• Image-guided treatment can provide local control of cancer and perhaps extend life.

• Molecular and functional imaging can improve the diagnosis and staging of cancer and hence improve treatment.

• Imaging endpoints can serve as reliable proxies for patient outcomes and hence guide appropriate therapy.

• Imaging informatics can improve cancer detection and diagnosis.

Within this context, ACRIN’s disease site committees contributed to a written strategy that outlines ACRIN’s research agenda for the next 4 years ^[3]. These committees also determine which specific trials have the highest priority for immediate protocol development. A Steering Committee makes the final decision on what trials ACRIN will pursue, within the constraints of its budget.

For trials that are approved for development, the disease site committee appoints a trial principal investigator, who works with ACRIN’s Headquarters (American College of Radiology Research Offices, Philadelphia, Pennsylvania) and Biostatistical and Data Management Center (Brown University Center for Statistical Sciences, Providence, Rhode Island) to appoint a multi-disciplinary trial team. Typically, such a team will encompass all of the trial functions, including: additional imaging expertise; medical, surgical, and radiation oncologists as dictated by the subject material of the trial; two or more statisticians; other methodologists, such as a sociometrician or economist, as dictated by the goals of the trial; a regulatory expert, data manager, auditor, and administrative support from Headquarters; a research associate; and a patient advocate representing ACRIN’s Patient Advocacy Committee to provide the perspective of potential subjects. Thus, an ACRIN trial team, along with ACRIN’s informatics resources, constitutes a complete infrastructure for the successful development, accrual, analysis, and dissemination of the results of imaging clinical trials.

All ACRIN operations and all of its trials are conducted in a virtual environment. Regular committee and trial team meetings are conducted by phone. All subjects are registered and, if necessary, randomized over the Web. All data and images are transmitted by the Web and archived at ACR Headquarters.

Lessons learned

Since its initiation in 1999, ACRIN has worked on 22 trials. These are in various stages of completion, from that of concept development, through full protocol and data forms design, to accruing subjects, to the writing and publishing of manuscripts. The smallest trials have involved as few as five institutions, while the largest have had over 30. Among these 22 trials are two of the largest and highest profile clinical trials in the US: the National Lung Screening Trial (NLST), evaluating whether imaging screening reduces lung cancer-specific mortality; and the Digital Mammographic Imaging Screening Trial (DMIST), which compares the effectiveness of conventional and digital mammography. Each of these trials accrued approximately 50000 subjects, who are in stages of clinical and imaging follow-up.

From this considerable experience, several truisms have manifested:

• The ACRIN group structure must differ from the membership type group that has been traditional for therapeutic trials because of the large number of technologies that must be addressed and variability in technology across sites.

• Diagnostic trials are different from therapeutic trials, requiring different scientific approaches and expertise.

• The broad range of technologies that might be employed in trials, variability in technology across sites, and the nature of image-derived information require a different approach to quality assurance than is generally employed in therapeutic trials.

Cooperative group structure

The therapeutic clinical trial cooperative groups have uniformly been ‘membership’ organizations. Although rules differ among the groups, and some have rules that are quite arcane, simplistically, institutions become members based upon their scientific expertise and retain their membership based on the numbers of patients they accrue to their group’s clinical trials. For a number of reasons, this approach would not work well for a cooperative group focused on imaging. The main reason is that there is much more diversity across institutions in what technologies are actually available. Many ACRIN trials can only be performed at selected institutions. Thus, ACRIN adopted an ‘open’ structure, which allows institutions (referred to as ‘participants’, rather than members) to participate in trials for which it has the technology and expertise and to de-participate when it does not.

Diagnostic trials methods

The therapeutic cooperative groups almost exclusively perform phase III randomized controlled trials (RCT) of a new drug or combination treatment. RCTs are the gold standard of research for therapeutic trials and clearly have applications that are important to diagnostic trials. However, RCTs tend to be ponderous and take a long time to complete. Because new imaging technologies are emerging so rapidly, and often change dramatically as they disseminate, the RCT approach would be fraught with problems if ACRIN focused on this methodology exclusively. Rather, ACRIN has adopted a tiered approach to its trials, matching each trials goals and methods to the level of maturity and the extent of dissemination of the technology. In many instances, ACRIN uses the traditional paired design of diagnostic trials which dictates that each subject receive imaging examinations by all the technologies under study. Only when the goals of the trial demand a randomized design—such as when we are interested in the impact of using a technology on the mortality rate—do we resort to RCT methodology.

Quality assurance

While the drugs being tested differ, therapeutic trials generally deal with only one or a few administration technologies. Thus, the therapeutic cooperative groups usually have a manual that deals extensively with these technologies. The purview of ACRIN is all imaging technologies at all disease sites subject to cancer. Clearly, this proffers an overwhelming number of permutations that might be instituted in a trial—far more than might be covered in a quality assurance manual. Rather, ACRIN devised a quality assurance ‘skeleton’, that defines the quality assurance considerations that must be undertaken in the design of a trial. The specifics of the quality assurance program are written into the protocol.

Similarly, mechanisms for auditing therapeutic trials differ from diagnostic ones. Therapeutic trials rely on the information in the patient’s record, whereas ACRIN’s data often exists only in the forms used to transmit data to Headquarters for archival. Hence, it was necessary for ACRIN to design unique audit procedures to insure the integrity of trial data. All ACRIN sites are audited no less than every 2 years.

Specific peculiarities of diagnostic imaging trials for cancer and how ACRIN has addressed them

There are a number of issues that devolve from the foregoing, which are specific to trials of diagnostic imaging, and that have necessitated ACRIN developing specific responses to ensure that trials are successful.

Fast developing technologies

As noted above, diagnostic imaging is characterized by rapidly emerging technologies that are rapidly disseminated, and that may change dramatically during their dissemination. To address the problem of designing a study the result of which may have no value in the context of this ‘moving target’, ACRIN has adopted several approaches:

• As mentioned, ACRIN’s open, non-member network facilitates easy entry and easy exit from the organization for individuals and institutions that may have special expertise or technology. This facilitates rapid ramp-up of trials.

• As appropriate, ACRIN conducts small, rapid pilot trials of emerging technologies to determine whether a technology is ready for a more definitive trial and to gain insight as to what are the appropriate goals for such a trial.

• The trial design and endpoints are dictated by the stage of development and dissemination of the technology. It makes little sense to perform a randomized trial to assess the cost-effectiveness of a rapidly morphing technology, whereas patient outcomes and/or cost-effectiveness may be the most appropriate endpoints for a trial of a technology for which reader performance (i.e. accuracy) has been well established.

• For large trials that will take a long time to accrue subjects, we generally allow the technology to ‘float’—that is change consistent with general use—during the accrual of subjects.

Quality assurance

Again, as mentioned in preceding paragraphs, precise quality control measures are essential to the conduct of a reliable trial and may be unique to each trial. Specific steps taken to ensure that ACRIN trials are rigorously performed include:

• ACRIN participants designed a quality assurance ‘skeleton’ that lays out all of the considerations that should go into each trial’s specific quality assurance plan.

• The specifics of a quality assurance plan are spelled out in exquisite detail in each protocol.

• Each trial requires continuous quality monitoring, usually by a trial-specific quality assurance committee comprised of the investigative team. ACRIN’s unique electronic infrastructure facilitates monitoring of image and interpretive quality by facilitating the retransmission of data and images over the Web from the ACRIN archive to quality assurance reviewers around the world.

• Possible adverse events—which differ considerably from those of therapeutic trials—are defined in prospect; a rigorous reporting system is enforced throughout accrual and follow-up.

• Each protocol defines what constitutes ‘source material’, which will be reviewed and compared to transmitted data when participating sites are audited.

Operator dependence

Compared with administering chemotherapy, or even radiation therapy, there is a great deal more operator dependence in the use of imaging technology and that must be accounted for in the design of ACRIN trials. In reality, there are really two samples of concern—a sample of subjects and a sample of image interpreters—that must be considered. Both are important to the generalizability of the results of any clinical trial. To accommodate this:

• We build the expectation of variability among readers into our sample size calculations.

• Participating sites are taken into the trial with an eye to providing the level of generalizability consistent with the goals of the trial. If the trial is one that queries the performance of an imaging technology in general practice, clearly, the sample of image interpreters must reflect that broad range. Conversely, if the trial asks how an emerging technology performs in the hands of subspecialist radiologists, a smaller, more focused sample of image interpreters is possible.

• Particularly when ACRIN studies newly emerging technologies, even subspecialist readers may have very different levels of experience and expertise. There also may be differences of opinion on definitions of disease or what constitutes the imaging signs of disease presence. All ACRIN trials feature either virtual or face-to-face ‘kickoff meetings’ where such issues are confronted and a consensus develops. Image interpreters agree to abide by these consensus agreements at least for the purposes of the trial. Depending on the trial, prospective participants may be required to undertake formal training programs designed by ACRIN and pass a set of test cases before they can begin accruing subjects at their site.

‘Upstream’ recruitment

Particularly for trials that investigate imaging applications to diagnosis and staging, as well as for therapeutic trials, an unbiased sample often requires that subjects be recruited before the decision is made to perform imaging. This means that ties must be developed between radiologists and referring clinicians—primarily oncologists—to facilitate successful accrual and timely trial completion. To accommodate this need ACRIN has:

• Developed broad relationships with a number of the therapeutic cooperative groups to collaborate on clinical trials. This generally occurs by:

  • – ACRIN proposing a trial and reaching an agreement with a therapeutic cooperative group to offer its sites funds and ‘credit’ for continuation in their group for each subject accrued.
  • – A therapeutic group asking ACRIN to develop a ‘correlative’ imaging trial nested in a therapeutic trial it wishes to undertake. Again, oncologists are offered funds and credit for each subject accrued.

• The ACRIN Steering Committee specifically considers the likelihood of successful accrual as a major criterion in determining whether or not to pursue a clinical trial. To a considerable extent, their decision may depend on the track record of timely completion of trials by the collaborating therapeutic group.

ACRIN’s accomplishments and the future

• ACRIN has been and is continuing to conduct clinical trials of imaging technologies that promise to provide information that will lead to better, more appropriate cancer care.

• For the first time, imaging researchers may draw upon the ACRIN clinical trials infrastructure for continuing support in the conduct of rigorous, multi-center clinical trials.

• With over 130 US and international institutions now qualified to participate in ACRIN trials, and the involvement of hundreds of researchers, it can be said that ACRIN is leading a cultural revolution among imagers in recognizing the importance of multi-center trials.

• ACRIN is disseminating standards of quality for imaging trials, such that other researchers may draw upon ACRIN’s methods, forms, and other instruments to improve the quality of imaging in clinical trials.

• That ACRIN exists has raised the consciousness of non-imaging oncology researchers as to the importance of well-performed imaging in clinical trials.

ACRIN welcomes participating institutions and individuals from around the world. Much can be learned about ACRIN by consulting the ACRIN Web site, www.acrin.org, or by contacting the Network Chair, Bruce J. Hillman, MD, at bjh8a@virginia.edu.