Achieving Sufficient Accrual to Address the Primary Endpoint in Phase III Clinical Trials from U.S. Cooperative Oncology Groups

Abstract

Purpose

Assessing impact of poor accrual on premature trial closure requires a relevant metric. We propose defining accrual sufficiency on apparent ability to address primary endpoints (PE) rather than attaining accrual targets.

Experimental Design

All phase III trials open January 1, 1993, to December 31, 2002, by five U.S. oncology Clinical Trials Cooperative Groups (CTCG) were evaluated for accrual sufficiency and scientific results. Sufficient accrual included meeting accrual target, CTCGs documentation attesting adequate accrual, or conclusive results at interim analysis; insufficient accrual included poor accrual as cited closure reason or other reasons rendering a trial unable to address its primary endpoints. Closure rates based on our accrual sufficiency definition are compared with rates of meeting accrual targets and addressing the primary endpoints. A percentage of target accrual above which trials commonly answer the intended scientific question was identified to serve as an alternative to meeting full target accrual in designating accrual success.

Results

Of 238 eligible trials, 158 (66%) closed with sufficient accrual. Among 80 trials with insufficient accrual, 70 (29%) closed specifically because of poor accrual. Inadequate accrual rates are overemphasized when defining accrual success solely by meeting accrual targets. Nearly 75% of trials conclusively addressed the primary endpoints with positive results in 39% of trials. Exceeding 80% of target accrual serves as a reliable proxy for answering the intended scientific question.

Conclusions

Approximately one third of phase III trials closed with insufficient accrual to address the primary endpoints, primarily due to poor accrual. Defining accrual sufficiency broader than meeting accrual targets represents a fairer account of trial closures.

Introduction

The Institute of Medicine (IOM) report “A National Cancer Clinical Trials System for the 21st Century: reinvigorating the NCI Cooperative Group Program” released in April 2010 called attention to early trial closure. The report cited that 40% of trials approved by the National Cancer Institute's (NCI) Cancer Therapy Evaluation Program (CTEP) close without meeting 100% of target accrual (1). This statistic, considered by many to be one of the most disconcerting findings in the report, referred to 419 trials of all phases (1, 2). Among the subset of 48 phase III trials in this study, 71% were considered as having poor accrual (2). This was defined as not meeting 100%of the accrual goal for trials closed to accrual and not meeting 75% or more of the accrual goal for trials still open to accrual, with other possible causes for early trial closure not known (2). In contrast, a subsequent study of 149 adult phase III trials conducted by the NCI-sponsored Clinical Trials Cooperative Groups (CTCG) reported 27% of trials were estimated to close because of inadequate accrual (3). Trials were defined as having successful accrual if closed with 90% or more of target accrual. This cutoff value was selected prospectively by the authors based on a belief that trials having accrued 90% or more of target accrual will typically have accumulated enough events to attain sufficient statistical power for analysis (3). Adjustments were made in the estimated closure rate secondary to inadequate accrual to account for open trials, and closures due to reasons other than accrual were considered separately in this study (3). The disparity in these 2 results highlights the effect of not having a common metric for reporting early trial closure. Basing early trial closure rates solely on not meeting accrual targets renders an incomplete picture of early trials closures. However, selecting a criteria not based on meeting accrual targets requires a rationale and a reproducible rule. Early closure can also be precipitated by events other than inadequate accrual, such as an interim analysis, unacceptable toxicity, or newly released results obviating the need to continue an ongoing trial. Such events present an ethical imperative to stop a trial early. A common metric is essential to appreciating the actual impact of poor accrual and evaluating the impact of any changes prompted by the IOM report or other critical evaluations of the clinical trials enterprise.

To clarify early phase III trial closures in the cooperative group setting, we conducted the Oncology Clinical Trial Accrual Study (OCTAS). OCTAS entails a systematic evaluation of combined phase III trial experiences from 5 NCI CTCGs conducted over a 10-year time period. Principal goals of OCTAS include identifying trial factors associated with accrual success present prior to trial opening and to determine tested, optimal benchmarks for early trial closure. To conduct this research, an accrual sufficiency metric as the outcome measure first needed to be established and is presented here.

Rather than defining accrual sufficiency solely on attaining target accrual or by some predesignated percentage thereof, we proposed to define sufficient accrual by whether, at the time of accrual closure, it seems that a trial will be able to address its primary endpoint. Ultimately, answering the intended scientific question is the most important goal of trial. Closure rates based on the OCTAS definition of sufficient accrual, as well as rates based on meeting target accrual and on reasons unrelated to accrual, are presented. Finally, these accrual sufficiency measures are compared with rates of generating scientific output, namely, resulting in publication and actually answering the scientific question relative to the primary endpoint. In addition, this information allows a percentage of target accrual to be defined above which trials commonly answer the intended scientific question and which could serve as a reliable alternative to meeting target accrual in designating accrual success.

Experimental design

The study sample includes all phase III trials open to accrual between January 1, 1993, and December 31, 2002, by at least 1 of 5 CTCGs. This 10-year time frame was selected to include a cohort of relatively contemporary trials that had been closed long enough to produce scientific publications. Trials were excluded if still open to accrual at analysis or if available trial documentation was inadequate for abstraction. Participating CTCGs were Cancer and Leukemia Group B (CALGB), Eastern Cooperative Oncology Group (ECOG), North Central Cancer Treatment Group (NCCTG), National Surgical Adjuvant Breast and Bowel Group (NSABP), and Southwest Oncology Group (SWOG; Seattle, WA). Participation was limited to CTCGs offering therapeutic trials for adult cancer patients to reduce variability in accrual experiences.

Trials were categorized as having sufficient or insufficient accrual based on information available at accrual closure indicative of the ability to address the primary endpoint. Target and actual accrual data were available for each trial. Sufficient accrual was defined as any of the following: (i) meeting target accrual, (ii) a CTCG closure letter or other documentation stating the trial had closed with complete or adequate accrual, (iii) closure at interim analysis with conclusive results, or (iv) closure due to toxicity. Target accrual comprised either the original sample size calculated to satisfy the primary endpoint as documented in the original protocol or a revised sample size if the trial underwent a major revision during its course affecting the statistical considerations. Statistically significant positive or negative findings relevant to the primary endpoint constituted conclusive results. Insufficient accrual was defined as any of the following: (i) CTCG documentation indicating closure due to poor accrual or (ii) closure due to factors external to the trial rendering it unlikely to address the primary end-point such as loss of equipoise resulting from new data or discontinuation of a test agent.

Accrual sufficiency designation followed a consistent algorithm (Fig. 1). Information on trial closure reason was abstracted by the primary author (A.T. Schroen) and research assistant (M.J. Thielen) from closure letters, technical reports, and other documentation available through the CTCG, with permission from CTCG leadership. If target accrual was not reached and no closure documentation was available, then the principal publication, defined later, was evaluated for an explanation of trial closure circumstances. These publications were separately evaluated for evidence of addressing the primary endpoint.

Figure 1.

Algorithm for designating accrual sufficiency status.

The principal publication was identified through a PubMed or Medline search or through bibliographies posted on the CTCG website. A principal publication of trial was considered the earliest peer-reviewed article reporting results pertaining to the primary endpoint, as specified in the trial protocol and used for the sample size calculation of trial. Both the principal publication search and the results categorization as conclusive or inconclusive were conducted independently by 2 authors (A.T. Schroen and M.J. Thielen). In cases where trial results seemed inconclusive or where discordance arose between closure documents and principal publications, 4 authors (A.T. Schroen, M.J. Thielen, G.R. Petroni, and H. Wang) determined accrual sufficiency status and final scientific trial results by consensus through joint review of trial protocol, CTCG documentation and publications. This process first involved voicing separate opinions on accrual sufficiency or publication results, then establishing consensus determination during a group discussion. A total of 13 trials underwent consensus review. Seven trials warranted discussion on accrual sufficiency status and 9 trials on publication results. These discussions mostly involved distinguishing between negative and inconclusive results. Statistical considerations outlined in the original protocol were precisely followed in determining if a predefined bound had been crossed, producing a negative result, or if a futility criteria had been met, producing an inconclusive result.

Rates of answering the scientific question pertinent to the primary endpoint and of resulting in publication are reported. Association of attaining a particular percentage of target accrual and answering the scientific question is also shown. At analysis, trials closed to accrual greater than 7 years with no identified principal publication were classified as not addressing the primary endpoint, whereas trials closed less than 7 years with no identified principal publication were categorized as unknown for addressing the primary endpoint. Seven years was selected prospectively as a cutoff value, allowing 5 years after accrual closure for patient follow-up and 2 additional years for publication. This exceeds the median years to publication reported in the few studies of time to publication in clinical trials (4). Results are reported for all trials and separately by trial type, categorized as therapeutic and nontherapeutic. Therapeutic trials are defined here as trials testing therapies directed at treating cancer. Trials testing strategies for cancer prevention, treatments for cancer-related or treatment-related symptoms, behavior modification, and quality of life are classified as nontherapeutic.

Proportions are reported for trial closure rates, closure reasons, and publication rates. The association of percentage of target accrual reached and the outcome of answering intended scientific questions are displayed by box plots. The fence on these box plots is set at 1.5 times the interquartile range with observations outside of the upper and lower fences marked. Statistical analyses were conducted with SAS Version 9.2 software (SAS Institute Inc.). Although OCTAS was conducted with CTCG cooperation, OCTAS study design, data collection, analysis, and results interpretation were conducted independently of the CTCGs by study personnel at the University of Virginia (Charlottesville, VA). This study was approved by the University of Virginia Institutional Review Board (IRB-HSR # 12582).

Results

A total of 248 phase III trials were open by 5 CTCGs between January 1, 1993, and December 31, 2002. After excluding 3 trials still open to accrual, 5 trials with missing or incomplete protocols not allowing abstraction, and 2 trials not deemed to actually be phase III, 238 trials remained for analysis. Of these 238 trials, 181 trials (76%) were classified as therapeutic. Accrual sufficiency status was assigned on the basis of meeting target accrual or on CTCG documentation of closure reason for 221 trials (93%). The remaining 17 trials were coded as insufficient accrual on the basis of overall low accrual, including 15 trials with less than 50% of target accrual and 10 trials with less than 20% of target accrual at closure. Definitive determination of answering the intended scientific question was possible for 218 trials (92%) by having a principal publication (n = 172) or having been closed for more than 7 years without a principal publication (n = 46). Among these 46 trials, 45 met less than 50% of target accrual and 34 met less than 20% of target accrual.

Closure with sufficient accrual was found in 66% of phase III trials overall. This rate is similar for therapeutic and nontherapeutic trials (Table 1). Specific reasons for trial closure are presented in Table 1. Only 5 trials had more than one closure reason cited; each was categorized with the most compelling closure reason. Among 80 trials closed with insufficient accrual, 70 were closed specifically due to poor accrual. In the 10 trials closed because of external factors, the range of completed accrual was 11% to 83% with 5 trials having accrued less than 50% of target accrual at closure. Closure reasons external to a trial may not be directly related to accrual. The early trial closure rate was, therefore, also calculated excluding these 10 trials, leaving 31% of overall trials closed because of inadequate accrual. When evaluating only trials initially opened after January 1 1993, the closure rate with sufficient accrual remains at 65%. Finally, the closure rate with sufficient accrual across the 5 CTCGs ranges from 60% to 83%.

Table 1. Phase III trial closures by accrual sufficiency status.

	Phase III trials, n (%)
	All (N = 238)	Therapeutic (n = 181)	Nontherapeutic (n = 57)
Sufficient accrual	158 (66)	121 (67)	37 (65)
Documented closure with complete accrual	134 (56)	101 (56)	33 (58)
Closed at interim analysis with positive results	9 (4)	8 (4)	1 (2)
Closed at interim analysis with negative results	12 (5)	10 (6)	2 (3)
Closed because of toxicity	3 (1)	2 (1)	1 (2)
Insufficient accrual	80 (34)	60 (33)	20 (35)
Documented closure due to inadequate accrual rate	70 (29)	52 (29)	17 (30)
Closed because of external data obviating need for trial	6 (3)	6 (3)	0
Closed because of trial agent shortage	2 (1)	1 (0.5)	1 (2)
Closed because of manufacturer withdrawing support for trial agent	2 (1)	1 (0.5)	1 (2)

Scientific output, measured by trials resulting in publication and answering the intended scientific question, is presented in Table 2. In assessing scientific output, 20 trials that had been closed less than 7 years were excluded, as their ability to produce publications was considered indeterminate. Among the remaining 218 trials, 172 (79%) had resulted in a principal publication related to the primary endpoint and 46 (21%) had been closed greater than 7 years without a publication. Conclusively answering the intended scientific question was noted in 159 (73%) trials, whereas inconclusive results specific to the primary end-point were noted in 13 (6%) of trials. The other 46 trials closed more than 7 years without a publication were deemed unlikely to produce future publications based on low overall accrual, as described earlier. Among trials with sufficient accrual closed to accrual greater than 7 years, all but one had resulted in publication. This one trial evaluated therapies for an acute cancer treatment–associated side effect; therefore, 7 years presented ample time for patient follow-up and publication. Of 159 trials conclusively addressing the primary endpoint, positive results relative to the primary endpoint were published for 62 trials (39%) and negative results for 97 trials (61%). These proportions are not statistically different between all trials and the subsets of therapeutic and nontherapeutic trials (Table 2). Finally, 6 trials had abstracts only that were not counted in the presented results. These abstracts either reported preliminary results from a trial closed less than 7 years, results unrelated to the primary endpoint, or the text of the abstract could not be found.

Table 2. Scientific productivity for phase III trials, excluding trials closed less than 7 years without a principal publication to date (n = 20).

	Phase III trials, n (%)
	All N = 218	Therapeutic n = 167	Nontherapeutic n = 51
Resulted in principal publication	172 (79)	135 (81)	37 (73)
Answered intended scientific question	159 (73)	125 (75)	34 (67)
Positive results	62 (39)	52 (42)	10 (29)
Negative results	97 (61)	73 (58)	24 (71)
Inconclusive	13 (6)	10 (6)	3 (6)
Closed ≥ 7 years without a principal publication	46 (21)	32 (19)	14 (27)

Defining accrual sufficiency solely on meeting 100% of target accrual resulted in lower rates of accrual success than our primary definition. Among all trials, 116 (49%) would be counted as having successful accrual if meeting 100% of target accrual defined accrual success. In the therapeutic and non-therapeutic trial subsets, only 90 (50%) and 26 (46%) trials, respectively, would be considered successful. However, more than 90% of trials are able to address the primary endpoint once at least 80% of target accrual is attained (Fig. 2). The numbers of trials accruing between 30% and 80% of target accrual were small and exceeding target accrual was common. The accompanying box plots take this into account and support defining accrual success by percentage achieved/target accrual at less than 100%. Defining accrual success at 90% of target accrual or even 80% to 85% of target accrual for therapeutic trials is supported by these results (Fig. 3).

Figure 2.

Proportion of trials addressing their intended scientific question by decile of percentage of target accrual achieved. Each decile is inclusive of the number, for example, 10 represents 0% to 10% of target accrual achieved.

Figure 3.

Median and interquartile range values of percentage of target accrual achieved for trials answering (Y) or not answering (N) their respective scientific question. Dashed lines represent 80%, 90%, and 100% of target accrual.

Discussion

Trial closure with sufficient accrual to address the primary endpoint occurs in approximately 66% of phase III trials in the CTCG setting, leaving about 1 in 3 trials closing with insufficient accrual. Reasons for early trial closure external to the trial do occur but are distinctly less common than early closures directly attributed to poor accrual. As stated in the IOM report, trials which close due to inadequate accrual represent a great inefficiency in terms of judicious use of scarce resources and failed opportunities to advance the science of clinical cancer care (1). The scope of the problem is overemphasized, however, when trials are labeled as unsuccessful based strictly on not meeting accrual targets. The assumptions upon which target accrual calculations are based are subject to error. More frequently, predictions underpinning the feasibility of attaining target accrual are flawed (5). Therefore, judging the success of a clinical trial should be weighted heavily by whether the trial is able to address its primary endpoint. If this information is unknown, our results support reaching 90% of target accrual or even 80% to 85% for therapeutic trials as an alternate definition for accrual success.

Our results reflect a markedly lower percentage of phase III trials having insufficient accrual than recently reported by Cheng and colleagues in a smaller sample of phase III trials conducted in a similar setting (2). Our results more closely approximate those released by Korn and colleagues (3). Other early closure rates have been reported in British trials, showing that about half of trials do not reach target accrual (6, 7). A study of 41 randomized trials listed in the 1979 NIH inventory showed 41% of trials did not achieve at least 75% of target accrual (8). An accrual sufficiency definition that is broader than meeting accrual goals represents a fairer account of trial closures and better reflects complexities inherent in trial conduct.

The publication rate in this series of CTCG phase III trials (74%) was high. This compares favorably to a study of trials registered with ClinicalTrials.gov in which 59% of cooperative group-sponsored and 6% of industry-sponsored phase III trials were published (9). A bias towards publishing positive studies was not apparent in our study, where 39% of published results were positive. In contrast, 75% to 85% of publications from industry-sponsored trials were deemed positive in 2 studies of trials registered in ClinicalTrials.gov (9, 10). Scientific results were carefully scrutinized in our study to ensure that conclusions were presented without distortion and that alternate endpoints were not substituted for original ones. Deflecting focus from the primary results has been identified in trial publications with statistically inconclusive results (11). A high and unbiased publication rate for CTCG trials speaks to the CTCGs' important and unique role in U.S. cancer clinical trial conduct.

Steps to address systematic issues with trial design, prioritization, and activation processes that may impact accrual have been used or are currently under study. New guidelines dictating a maximum allowable development time for phase III trials were a primary focus of the recent NCI Operational Efficiency Working Group report (12). Operational studies have illustrated the steps and time involved in phase III trial progression from concept submission to trial activation. An association between longer trial development times and decreased likelihood of achieving target accrual has been shown (2, 13). Effective trial prioritization presents another challenge in that tools are lacking to measure the clinical relevance of a trial and to compare trial concepts over disparate areas in clinical oncology. Scientific steering committees established by recommendation of the Clinical Trials Working Group were tasked with phase III trial concept prioritization (14). Finally, prior work suggests there are certain trial design factors that reduce the likelihood of attaining sufficient accrual (15, 16). Our ongoing work examining trial-based factors associated with accrual sufficiency challenges the generally accepted impact of some of these trial factors such as presence of nontreatment arms, number of arms, and greater number of eligibility criteria (17, 18). Forthcoming results using our accrual sufficiency definition include accrual benchmarks for early trial closure or redesign and trial-based factors identifiable prior to trial activation that are associated with accrual success.

Study limitations include CTCG documentation not always stating trial closure reason. Formal closure documentation was not found for 22% of trials, but 93% of all trials either met target accrual or had closure documentation. The remaining 7% of trials, coded as insufficient accrual, had low accrual. Some closure documents may have attributed closure to inadequate accrual while not citing other influential factors, thereby possibly underrepresenting other closure reasons. Second, our definition of insufficient accrual included closure due to external factors; however, these may not actually reflect accrual problems. Our definition of accrual sufficiency hinged strongly on the apparent ability of trials to answer the scientific question. Trials closed to external reasons are unlikely to be able to address the intended primary endpoint. As such, an argument can be made for either including or excluding trials closed because of external reasons. Given the emphasis on answering the scientific question in our definition of accrual sufficiency, trials closed to external causes were retained in our study. Furthermore, external closure reasons were uncommon, and our analysis was not appreciably different when these trials were excluded. Third, some trials may require more than 7 years after closure to produce final results and thus been mislabeled as not addressing the primary endpoint. Because all but one trial closed more than 7 years and lacking a principal publication also had poor accrual, future publications seem improbable. The 7-year time frame from end of patient enrollment to publication, therefore, represents a reasonable cutoff value for this particular trial cohort. Limited studies available on time to publication of clinical trials suggest that most trials take less than 7 years from accrual completion to publication (4, 19). Nonetheless, future studies should give consideration to anticipated patient follow-up requirements based on disease course and primary endpoints to select a reasonable time frame for expected publication. Finally, our study time frame, which was selected to allow time for producing scientific results includes some older trials. No change in closure rates was seen when excluding trials open prior to 1993. However, ongoing changes in trial conduct may impact future accrual success rates and require evaluation to verify their impact on accrual.

The recent IOM report helped draw attention to early trial closures as a primary challenge facing the U.S. cancer clinical trials enterprise. In quantifying the impact of poor accrual, a shift to metrics reflecting an ability of trials to address the primary endpoint offers a more meaningful representation of CTCG trial efforts than metrics based on meeting accrual targets alone. Nonetheless, our finding that roughly a third of phase III trials close due to poor accrual provides a compelling reason to improve clinical trial processes both before and after trial activation. This new metric could also facilitate future evaluations of the impact current changes in the cooperative groups have on successful clinical trial conduct. Identifying an acceptable trial failure rate will also be needed, thereby establishing realistic goals for the clinical trials enterprise while not suppressing scientific innovation. Next, urgently needed steps include further improvements in accrual prediction accuracy, reliable identification of trials with higher likelihood of accrual problems before trial activation, and optimization of prioritization processes to select the most compelling treatment questions. Meeting accrual goals alone does not suffice. Efficiently delivering scientific results for clinically relevant treatment questions is required for optimal realization of the potential in today's cancer care.

Translational Relevance.

Inadequate clinical trial accrual poses a significant and ongoing obstacle to advancing the science of clinical cancer care. Recent reports, including from the Institute of Medicine, have focused attention on the issue of poor clinical trial accrual. This article presents a metric for insufficient accrual in phase III clinical trials based on the ability of trial's to address its primary endpoint rather than solely on meeting target accrual, which has served as the basis for most previous reports on trial accrual. This new metric is useful in appreciating the actual impact of poor accrual and in evaluating the impact of any changes prompted by recent critical evaluations of the clinical trials enterprise.