Stopping or Reporting Early for Positive Results in Randomized Clinical Trials: The National Cancer Institute Cooperative Group Experience From 1990 to 2005

Abstract

Randomized clinical trials are designed with stopping boundaries to guide data monitoring committees with their decision making concerning ongoing trials. In particular, when extremely positive results are seen and a boundary is crossed, the data monitoring committee may recommend releasing the results earlier to the public than at the definitive final analysis time specified in the protocol. For trials that are still accruing, this also means stopping accrual. Because the information about treatment efficacy is more limited in an early analysis than in a final analysis, questions have been raised about the appropriateness of incorporating early stopping for positive results in trial designs. In particular, there are concerns that treatment effects seen early may not be real or may be overly optimistic. To examine this issue, we collected information about treatment efficacy on National Cancer Institute Cooperative Group trials that were stopped early for positive results (information both at the time the trial was stopped/released and at times of further follow-up). Twenty-seven such trials were located. For 17 of 18 of these trials with sufficient follow-up information, the treatment effect was similar or only slightly smaller at last follow-up compared with the stopping/release time. We critically evaluate reasons why one might be concerned about early stopping for positive results. We conclude that for trials with well-designed interim monitoring plans, the ability to stop early for positive results is an important component of the trial design, allowing the public to benefit as soon as possible from the study conclusions.

INTRODUCTION

Interim monitoring plans, including formal guidelines for stopping trials early (stopping accrual and/or releasing results early) for compelling results, are a standard part of the designs of randomized clinical trials (RCTs). The monitoring guidelines are designed to limit the probability of a false-positive result (type I error) while allowing trials to stop early. Although the benefits to the public in releasing compelling positive trial results early are obvious, there have been concerns expressed^1–5 about the correctness of early stopping of RCTs for positive results. To address the concerns raised, we performed a review of all treatment RCTs performed by the National Cancer Institute (NCI) Cooperative Groups and published from 1990 to 2005, thus providing empiric data on the issues as has been suggested.⁶ We also examined frequently cited reasons why one should be cautious about using early stopping for positive results and found some of these reasons to be correct but others lacking in statistical validity.

NCI COOPERATIVE GROUP TRIALS THAT STOPPED OR RELEASED RESULTS EARLY FOR POSITIVE RESULTS

We located NCI Cooperative Group phase III treatment RCTs whose accrual was stopped early or whose results were released early for positive results with the first publication appearing in the years 1990 to 2005; we included Canadian National Cancer Institute of Canada (NCIC) Clinical Trials Group trials partially supported by NCI. Using a list from PDQ (http://www.cancer.gov/cancertopics/pdq/cancerdatabase), all trial publications appearing in Journal of Clinical Oncology, Journal of the National Cancer Institute, New England Journal of Medicine, The Lancet, or Blood or as an American Society of Clinical Oncology or American Society of Hematology abstract were examined for evidence of early stopping/release for positive results. We located 27 such trials (Table 1);^7–60 it is possible that a small number of trials were missed if they were not published in the searched publications or if no mention was made of the early stopping/release in the publications. (Once a relevant trial was located, all publications associated with that trial were examined.) The end points that crossed interim monitoring boundaries for the 27 trials were overall survival (OS; seven trials), progression-free survival (PFS; six trials), disease-free survival (DFS; six trials), event-free survival (EFS; three trials), complete response rate (two trials), failure-free survival (one trial), continuous complete remission rates (one trial), and response rate (one trial). End point definitions can be found in Appendix Table A1 (online only). Accrual was complete at the time of stopping/release for 70% of the trails (19 of 27 trials) and ranged from 66% to 97% complete for the other eight trials.

Table 1.

Descriptions of NCI Cooperative Group Phase III Treatment Trials That Stopped or Released Results Early for Positive Results, With Primary Publication Appearing Between 1990 and 2005

Trial Identifier (PDQ)	Trial Description and Reference Numbers*	Trial End Point†	Year Trial Was Stopped/Reported	% of Patients Accrued When Trial Stopped/Reported
NCCTG-844652	Phase III pilot of adjuvant therapy with levamisole v levamisole plus 5-fluorouracil for resectable adenocarcinoma of the colon7,8	OS	1989	100
RTOG-8501	Phase III comparison of radiotherapy alone v radiotherapy plus combination chemotherapy with CACP/FU in patients with localized carcinoma of the esophagus9,10	OS	1990	86
POG-9006	Phase III comparison of intensification with MP/MTX v alternating MP/MTX, VM-26/ARA-C, and DNR/ARA-C/VCR/PRED/PEG-ASP following induction with PRED/VCR/ASP/DNR in children with higher risk ALL11,12	CCR	1994	97
EST-3189	Phase III randomized comparison of CAF (CTX/ADR/FU) v a 16-week multidrug regimen (CTX/ADR/VCR/MTX/FU) as adjuvant therapy in node-positive patients with receptor-negative breast cancer13,14	OS	1994	100
CLB-9011	Phase III comparison of CLB v FAMP in previously untreated patients with intermediate- and high-risk (Rai stage I-IV) B-cell CLL15,16	CR	1995	100‡
SWOG-8892	Phase III study of radiotherapy with v without concurrent CDDP followed by CDDP/FU for previously untreated stage III/IV carcinoma of the nasopharynx17–19	PFS	1995	71
E-2491	Phase III randomized trial of tretinoin v ARA-C/DNR as induction therapy and of tretinoin v observation as maintenance therapy for patients with previously untreated promyelocytic leukemia20	DFS	1995	94
CCG-1882	Phase III treatment of poor-prognosis childhood ALL (excluding infants and patients with lymphoma-leukemia and FAB L3 blasts), including a randomized comparison of standard v augmented BFM in late responders21	EFS	1996	100
SWOG-8814	Phase III randomized comparison of adjuvant therapy with TMX v CAF (CTX/DOX/FU) plus concurrent or delayed TMX in postmenopausal women with node- and receptor-positive breast cancer22,23	DFS	1997	100
SWOG-8797	Phase III randomized comparison of radiotherapy with v without continuous-infusion FU/bolus CDDP after radical hysterectomy and node dissection in high-risk patients with stages IA2, IB, and IIA carcinoma of the cervix24,25	OS	1998	100
CLB-9344	Phase III randomized study of adjuvant cyclophosphamide/doxorubicin comparing standard- v intermediate- v high-dose doxorubicin, with v without subsequent paclitaxel, in women with node-positive breast cancer26–28	DFS	1998	100
RTOG-9001	Phase III comparison of pelvic irradiation with concurrent CDDP/5-FU v pelvic and para-aortic irradiation without chemotherapy in patients with high-risk carcinoma of the cervix29,30	OS	1998	100
CCG-5942	Phase III study of adjuvant low-dose involved-field radiotherapy v no adjuvant therapy in children with Hodgkin's disease in CR after chemotherapy assigned by clinical stage31	EFS	1998§	77
SWOG-9133	Phase III randomized trial of subtotal irradiation with v without DOX/VBL in patients with stage IA/IIA Hodgkin's disease32,33	FFS	2000	83
RTOG-9413	Phase III randomized study of whole pelvic irradiation followed by a cone-down boost to the prostate v prostate irradiation only and of neoadjuvant v adjuvant FLUT/ZDX for adenocarcinoma of the prostate34–36	PFS	2001	100
SWOG-S9701	Phase III randomized trial of 12 months v 3 months of paclitaxel in patients with advanced ovarian, fallopian tube, or primary peritoneal cancer in complete remission after platinum/paclitaxel–based chemotherapy37,38	PFS	2001	66
NCCTG-N9741	Phase III randomized study of combinations of oxaliplatin, fluorouracil, leucovorin calcium, and irinotecan as initial therapy in patients with advanced adenocarcinoma of the colon and rectum39,40	PFS	2002	100
NCIC-MA17	Phase III randomized study of letrozole v placebo in postmenopausal women with primary breast cancer who have completed at least 5 years of adjuvant aromatase inhibitor41,42	DFS	2003	100
E-1496	Phase III randomized study of standard therapy followed by maintenance therapy with rituximab (IDEC-C2B8 monoclonal antibody) or observation in patients with stage III or IV low-grade non-Hodgkin's lymphoma43,44	PFS	2003	100
E-E1A00	Phase III randomized study of dexamethasone with or without thalidomide in patients with newly diagnosed multiple myeloma45,46	RR	2003	100
CCG-1961	Phase III randomized study of treatment based on response to induction chemotherapy in patients with higher risk childhood acute lymphocytic leukemia: standard v augmented BFM regimen with standard v prolonged intensification for rapid early responders and doxorubicin v idarubicin and cyclophosphamide with delayed intensification for slow early responders47,48	EFS	2003	100
E-3200	Phase III randomized study of oxaliplatin, fluorouracil, and leucovorin calcium with or without bevacizumab v bevacizumab only in patients with previously treated advanced or metastatic colorectal adenocarcinoma49–51	OS	2004	100
ECOG-2997	Phase III randomized study of fludarabine with or without cyclophosphamide in patients with previously untreated chronic lymphocytic leukemia52,53	CR	2004	100
NSABP-B-31/NCCTG-N9831	Phase III randomized study of doxorubicin and cyclophosphamide followed by paclitaxel with or without trastuzumab in women with node-positive breast cancer that overexpresses HER-2/Phase III randomized study of doxorubicin plus cyclophosphamide followed by paclitaxel with or without trastuzumab in women with HER-2–overexpressing node-positive or high-risk node-negative breast cancer54,55	DFS	2005	81
ECOG-4599	Phase II/III randomized study of paclitaxel and carboplatin with or without bevacizumab in patients with advanced, metastatic, or recurrent non–squamous cell non–small-cell lung cancer56	OS	2005	100
ECOG-2100	Phase III randomized study of paclitaxel with or without bevacizumab in patients with locally recurrent or metastatic breast cancer57,58	PFS	2005	100
NCIC-MA21	Phase III randomized study of adjuvant cyclophosphamide, epirubicin, and fluorouracil v cyclophosphamide, epirubicin, filgrastim (G-CSF), and epoetin alfa followed by paclitaxel v cyclophosphamide and doxorubicin followed by paclitaxel in premenopausal or early postmenopausal women with previously resected node-positive or high-risk node-negative stage I-IIIB breast cancer59,60	DFS	2005	100

Abbreviations: NCI, National Cancer Institute; NCCTG, North Central Cancer Treatment Group; OS, overall survival; RTOG, Radiation Therapy Oncology Group; CACP, cisplatin; FU, fluorouracil; POG, Pediatric Oncology Group; MP, mercaptopurine; MTX, methotrexate; VM-26, teniposide; ARA-C, cytarabine; DNR, daunorubicin; VCR, vincristine; PRED, prednisone; PEG, pegylated; ASP, asparaginase; ALL, acute lymphoblastic leukemia; CCR, continuous complete remission; CTX, cyclophosphamide; ADR, doxorubicin; CLB, chlorambucil; FAMP, fludarabine; CLL, chronic lymphocytic leukemia; CR, complete response; SWOG, Southwest Oncology Group; CDDP, cisplatin; PFS, progression-free survival; DFS, disease-free survival; CCG, Children's Cancer Group; FAB, French-American-British; BFM, Berlin-Frankfurt-Muenster; EFS, event-free survival; TMX, tamoxifen; DOX, doxorubicin; VBL, vinblastine; FFS, failure-free survival; FLUT, flutamide; ZDX, goserelin; NCIC, National Cancer Institute of Canada; RR, response rate; ECOG, Eastern Cooperative Oncology Group; NSABP, National Surgical Adjuvant Breast and Bowel Project; HER-2, human epidermal growth factor receptor 2; G-CSF, granulocyte colony-stimulating factor.

^*Reference numbers are given for references that were used to complete the trial information for Tables 1 to 3. Additional information on the trials was obtained from personal communications, as follows: EST-3189, E-1496, E-3200, ECOG-2997 (R.J. Gray, personal communication, July 2008); CLB-9011 (B. Peterson, personal communication, August 2008); SWOG-8892 (M. LeBlanc, personal communication, July 2008); SWOG-8814 (W. Barlow, personal communication, July 2008); SWOG-8797 (P.Y. Liu, personal communication, June 2008); RTOG-9001 (K. Winter, personal communication, July 2008); CCG-5942 (R. Sposto, personal communication, July 2008, of interim report on study 5942 prepared for the CCG Data and Safety Monitoring Committee, October 12, 1998); CCG-1961 (N.L. Seibel, personal communication, May 2008); and CCG-1882 (M. Devidas, personal communication, August 2008).

^†Complete definitions of the trial end points are given in Appendix Table A1.

^‡This is the percentage of the original target sample size; the primary end point was changed and the sample size was increased after the interim analysis when the results crossed the boundary in 1993. At that time, the percentage of accrual was 59%.

^§Random assignment stopped in 1998; as per protocol, results were reported later (2002).

Table 2 lists efficacy statistics on these trials at their times when they crossed their interim monitoring boundary, they were first published, and further follow-up information was published (when available). Efficacy data are given as reported in the publications (eg, hazard ratio or the 3-year survival in each arm), but all P values given here are two-sided. The follow-up information is an attempt to assess, in hindsight, how accurate or inaccurate the early positive results were. Regardless of these findings, there is no intended implication that the data monitoring committees making these particular stopping decisions made incorrect decisions given the protocol interim monitoring guidelines and the information they had at the time. We focus here on the results at the time of interim monitoring boundary crossing and at the last follow-up available (which could be when the results were first published if there was no additional follow-up). When the trial results crossed the boundary, the ratio of observed events to events required at the final analysis (information fraction) ranged from 15% to 90%, with a median of approximately 60%.

Table 2.

Statistics on NCI CTEP-Sponsored Trials Whose Accrual Was Stopped or Whose Results Were Released Early for Positive Results From 1990 to 2005

Trial Identifier	Results When Trial Crossed Boundary			Results When Trial First Published			Follow-Up Results of Trial
	% Information	Treatment Effect	P	% Information	Treatment Effect	P	% Information	Treatment Effect	P
NCCTG-844652*	60	HR = 0.67	.0064	Same as when trial stopped			89	HR = 0.67	.0007
RTOG-8501†	∼60	HR = ∼0.49	.0045	87	2-year OS: 38% v 10%	< .001	108	2-year OS: 36% v 10%	< .001
POG-9006‡	∼38	2-year CCR: 82% v 70.8%	.0016	∼54	2-year CCR: 84% v 75%	.006	77	4-year CCR: 70.6% v 64%	.22
EST-3189§	62	3-year OS: 84% v 73%	.0050	Same as when trial stopped			90	4-year OS: 78.1% v 71.4%	.10
CLB-9011‖	34	CR rates: 30% v 2%	.00014	78	CR rates: 33% v 8%	< 10−5	117	CR rates: 20% v 4%	< 10−5
SWOG-8892	26	HR = 0.26	< .0001	Same as when trial stopped			50	HR = 0.31	< .001
ECOG-2491¶	50	1-year DFS: 92% v 57%	< .0001	Same as when trial stopped			> 100	5-year DFS: 69% v 29%	< .0001
CCG-1882	68	4-yr EFS: 75.4% v 57.2%	.0013	80	5-year EFS: 75.0% v 55.0%#	< .001	—	—	—
SWOG-8814	81	HR = 0.66	.002	Same as when trial stopped			120	HR = 0.76	.002
SWOG-8797	∼60	HR = 0.45**	0.006**	∼60-63††	HR = 0.50	.01	63	HR = .51‡‡	.007‡‡
CLB-9344§§	25	HR = 0.79	.013	Same as when trial stopped			59	HR = 0.83	.0023
RTOG-9001	58	5-year OS: 73% v 58%	.0027	59	5-year OS: 73% v 58%	.004	81	5-year OS: 73% v 52%	< .0001
CCG-5942	34‖‖	HR = 0.27	.0048	64	HR = 0.59	.057¶¶
SWOG-9133	46	3-year FFS: 93% v 81%	< .001	Same as when trial stopped			48	3-year FFS: 94% v 81%	< .001
RTOG-9413##	90	4-year PFS: 56% v 46%	.014	Same as when trial stopped			156	4-year PFS: 54% v 54%	NS
SWOG-S9701	15	HR = 0.43	.0046	Same as when trial stopped			62	HR = 0.70	.008
NCCTG-N9741***	81	HR = 0.71	.0009	Same as when trial stopped			98	HR = 0.74	.0014
NCIC-MA17	40	HR = 0.57	.00008	Same as when trial stopped			48	HR = 0.58	< .001
E-1496	49	HR = 0.42	.00016	59	HR = 0.5	.00006	83†††	HR = 0.38	< 10−5
E-E1A00	59	RR: 80% v 53%	.0046	Same as when trial stopped			108	RR: 63% v 41%	.0034
CCG-1961‡‡‡	84	5-year EFS: 78.8% v 69.8%	.0198	91	5-year EFS: 80% v 71%	.01	124	5-year EFS: 81.2% v 71.7%	< .001
E-3200§§§	90	HR = 0.74	.0024	Same as when trial stopped			111	HR = 0.75	.0011
ECOG-2997	79	CR rates: 22.9% v 5.8%	.0008	98	CR rates: 22.4% v 5.8%	.0002	107	CR rates: 23.4% v 4.6%	< 10−5
NSABP-B-31/NCCTG-N9831‖‖‖	55	HR = 0.48	< .0001	Same as when trial stopped			87	HR = 0.49	< .0001
ECOG-4599	72	HR = 0.78	.0076	100	HR = 0.79	.003	—	—	—
ECOG-2100	65	HR = 0.50	< .001	114	HR = 0.60	< .001	—	—	—
NCIC-MA21¶¶¶	58	HR = 0.60	.0006	Same as when trial stopped			—	—	—

Abbreviations: NCI, National Cancer Institute; CTEP, Cancer Therapy Evaluation Program; NCCTG, North Central Cancer Treatment Group; HR, hazard ratio; RTOG, Radiation Therapy Oncology Group; OS, overall survival; POG, Pediatric Oncology Group; CCR, continuous complete remission; CR, complete response; SWOG, Southwest Oncology Group; ECOG, Eastern Cooperative Oncology Group; DFS, disease-free survival; CCG, Children's Cancer Group; EFS, event-free survival; FFS, failure-free survival; PFS, progression-free survival; NS, not significant; NCIC, National Cancer Institute of Canada; RR, response rate; NSABP, National Surgical Adjuvant Breast and Bowel Project.

^*NCCTG-844652: We consider only the analysis of the stage C patients here (which was prespecified). The trial was stopped because of differences in the levamisole plus fluorouracil arm compared with the observation arm, which are the results reported here.

^†RTOG-8501: Information percentage for when the trial crossed the boundary was estimated based on length of accrual/follow-up and observed survival rates. Treatment effect HR for when the trial crossed the boundary was estimated based on P value and estimated number of events.

^‡POG-9006: Information percentage for when trial crossed boundary and when trial was first published was estimated based on accrual/follow-up and observed CCR. In the follow-up period, treatment effect 2-year CCR rates were 85.2% and 80.4%.

^§EST-3189: OS was one of the two coprimary end points and the one that crossed the interim monitoring boundary. In the follow-up period, the treatment effect– estimated 3-year OS rates from Figure 2 are 83% v 77%. Regarding the P value, the protocol for this trial specified a one-sided 5% level test, which corresponds to a two-sided 10% level test.

^‖CLB-9011: This was initially a three-armed trial, with the experimental fludarabine plus chlorambucil arm later dropped. The results presented here are for the comparison of fludarabine v chlorambucil (control). The P value used for the interim analysis for when trial crossed boundary was P = .0005 based on a comparison of both experimental arms combined compared with the control arm.

^¶ECOG-2491: The results reported here are for the induction therapy comparison (there was also a maintenance treatment randomization in this trial). For the follow-up period, the estimated 1-year DFS rates from Figure 1 of Tallman et al²⁰ are 88% v 57%.

^#CCG-1882: Regarding treatment effect results when the trial was first published, the estimated 4-year EFS rates from Figure 1 of Nachman et al ²¹ are 77% v 60%.

^**SWOG-8797: HR and P value were unadjusted for stratification variables. The values adjusted for the random assignment stratification variables are 0.49 and P = .02.

^††SWOG-8797: The information fraction is not given in the first publication but must be between when the trial was stopped and the later follow-up.

^‡‡SWOG-8797: HR and P value were adjusted at follow-up for random assignment stratification factors.

^§§CLB-9344: This trial had a factorial design; comparison of ± paclitaxel was the one that led to the release of the data and is the one reported here. HR for when trial crossed boundary was estimated based on P value and percent information.

^‖‖CCG-5942: Per protocol, the random assignment was stopped when the boundary was crossed, but the results were not released until there was further follow-up.

^¶¶CCG-5942: This result would be considered statistically significant because the trial was designed with a one-sided type I error of 0.10, which corresponds to a two-sided type I error of 0.20.

^##RTOG-9413: This trial had a factorial design; comparison of whole-port v prostate-only radiation was the one that crossed the interim monitoring boundary and is the one reported here. The percent information for when trial crossed boundary was estimated from interim monitoring boundary. The 4-year PFS results in the follow-up period were estimated from curves in Figure 2A of Lawton et al.³⁵

^***NCCTG-9741: This trial had three arms; fluorouracil/leucovorin + oxaliplatin v fluorouracil/leucovorin + irinotecan is the comparison that crossed the interim monitoring boundary and is the one considered here.

^†††E1496: Information is estimated based on CI width for the HR.

^‡‡‡CCG-1961: We are considering the analysis of the intensity of treatment question in the rapid early responder subgroup, which is the one that crossed an interim analysis boundary.

^§§§E-3200: Arms being discussed here are the infusional fluorouracil, leucovorin, and oxaliplatin (FOLFOX4) arm (control arm) and FOLFOX4 + bevacizumab arm; the bevacizumab-alone arm closed early for negative results.

^‖‖‖NSABP-B-31/NCCTG-N9831: The results reported here are from the combined analysis of the trials NSABP-B-31 and NCCTG-N983, which are considered one trial for the purposes of discussion here.

^¶¶¶NCIC-MA21: This was a three-arm trial; results are given for epirubicin-cyclophosphamide/paclitaxel v doxorubicin-cyclophosphamide/paclitaxel as experimental and control treatments, respectively.

Although all 27 trials met their prespecified study objectives when their positive results crossed their boundaries, to provide a summary characterization of Table 2, we first focused on the 18 trials that had follow-up information of at least 80% (an arbitrary figure representing trials for which planned final analysis results could be considered available). For 14 of the 18 trials (North Central Cancer Treatment Group NCCTG-844652; Radiation Therapy Oncology Group RTOG-8501; Cancer and Leukemia Group B CLB-9011; Eastern Cooperative Oncology Group ECOG-2491; Children's Cancer Group CCG-1882; RTOG-9001; NCCTG-9741; Eastern Cooperative Oncology Group E1496; Eastern Cooperative Oncology Group E-E1A00; CCG-1961; E3200; E2997; National Surgical Adjuvant Breast and Bowel Project B NSABP-B-31/NCCTG-N9831; ECOG-4599), the treatment effect was similar at early stopping/release and last follow-up. For Southwest Oncology Group SWOG-8814 and Eastern Cooperative Oncology Group ECOG-2100, the treatment effect became slightly smaller, although with the same statistical significance. For Eastern Cooperative Oncology Group EST-3189, the treatment effect became slightly smaller with the statistical significance much weaker (although still statistically significant based on the protocol specification). For RTOG-9413, the treatment effect disappeared; at the time of early release, 90% of the required events were observed, and the 4-year PFS rates were 56% v 46% (P = .014). When long-term follow-up became available (156% of the required events), the 4-year PFS rates were 54% in both arms. Because in this trial the results were released so close to the protocol-specified final analysis, one can argue that this is an illustration of a study with the final analysis reversed by longer follow-up.

Although not one of the 18 trials with at least 80% information follow-up, the release of data in POG-9006 (with accrual 97% complete) may seem problematic. The trial reported 2-year EFS rates of 84% v 75% (P = .006) with 54% information (when the data crossed the interim monitoring boundary, the 2-year EFS rates were 82% v 70.8%, P = .0016, with 38% information). With further follow-up and 77% information, the reported 4-year EFS rates were 70.6% v 64% (P = .21). We will return to discussion of this trial in the next section. For the other eight trials that did not have at least 80% information follow-up, one trial had no further follow-up (NCIC-MA21), and the other seven trials retained statistical significance, with two having a smaller treatment effect (CCG-5942 and SWOG-S9701) and five showing a similar treatment effect (SWOG-8892, SWOG-8797, CLB-9344, SWOG-9133, and NCIC-MA17).

STATISTICAL AND DESIGN ISSUES INVOLVING EARLY STOPPING/RELEASE

We discuss a number of reasons that have been used to suggest that early stopping for extremely positive results may not be appropriate.

Choice of Primary End Point

The designated primary end point of an RCT is the one that is used to make the definitive statement concerning treatment effectiveness. There can be controversy about the appropriate primary end point, with different end points yielding different required sample sizes for the trial. For example, a trial that demonstrates a positive treatment effect at its conclusion for PFS may not have a sufficient number of deaths at that time to evaluate conclusively OS benefits. The possibility of early stopping exacerbates this potential problem in that there may be little information available about the nonprimary end points if the trial is stopped early. For example, Cannistra⁶¹ questioned the decision to close and report early the results of SWOG-S9701 and NCIC-MA17 based on PFS and DFS end points, respectively, when the OS data were immature. If one believes that an improvement in a non-OS end point results in direct patient benefit, then there should be little argument against stopping a trial early based on extremely positive results for that end point. For example, the NCIC-MA17 investigators⁶² considered DFS an important clinical end point for their adjuvant breast cancer setting. However, sometimes a non-OS end point is used not because it directly represents patient benefit, but because it is a surrogate for OS. As a surrogate for OS, it may have more statistical power because events accumulate faster and because it may be less susceptible to potential confounding by treatment crossovers to the experimental arm after non-OS events in the control arm. In this case, it is not as clear that one would need to stop a trial early for positive non-OS treatment effects, unless the surrogate is uniformly accepted in the clinical community.

When a non-OS primary end point does not directly represent clinical benefit, a reasonable strategy is to use OS for the interim analysis for positive effects even though the primary end point is different. To do this, one would have to be comfortable continuing a trial that showed extremely positive results in the non-OS primary end point provided that OS differences were not large. Other possibilities include requiring extremely positive results for early stopping/release or starting the interim monitoring for positive effects at a late enough time point that accrual will be complete or almost complete. This may allow evaluation of the OS effect with further follow-up. For example, in Table 3, we see that seven of 16 trials that stopped based on non-OS end points in Table 2 eventually achieved OS treatment effects that were large enough and precise enough to attain statistical significance (P < .05); four trials did not have OS results available. A potentially useful strategy to ameliorate the dilution of treatment effect as a result of crossovers is to censor the OS data of the control arm patients at the time when positive results are released; see Bukowski et al⁶³ for an example.

Table 3.

Latest OS Results for Trials in Table 2 Whose Primary End Point Was Not OS

Trial Identifier	Treatment Effect	95% CI for HR or Difference in Rates	P
CLB-9011	Median OS: 66 v 56 months	Not applicable	.10
SWOG-8892	5-year OS: 67% v 37%	Not available	.001
E2491	5-year OS: 69% v 45%; difference = 24%	13.4% to 34.6%	.0001
SWOG-8814	HR = 0.83	0.69 to 0.99	.04
CLB-9344	HR = 0.82	0.71 to 0.95	.0064
CCG-5942	3-year OS: 98% v 99%; difference = −1%	−3.3% to 1.3%	.90
RTOG-9413	4-year OS: 84.7% v 84.3%; difference = 0.4%	−4.6% to 5.4%	.94
SWOG-S9701	HR = 0.84	0.61 to 1.16	.30
NCCTG-9741	HR = 0.66	0.54 to 0.82	.0001
NCIC-MA17	HR = 0.82	0.57 to 1.19	.3
ECOG-1496	HR = 0.51	0.25 to 1.04	.06
CCG-1961	HR = 0.64	0.47 to 0.87	.005
E2997	2-year OS: 79% v 80%; difference = −1%	−14.3 to 12.3*	.69
NSABP-B-31/NCCTG-N9831	HR = 0.63	0.49 to 0.81	.0004
ECOG-2100	HR = 0.88	0.74 to 1.05	.16
NCIC-MA21	47 v 65 deaths	Not applicable	.09†

NOTE. No OS results are available for POG-9006, CCG-1882, E-E1A00, or SWOG-9133.

Abbreviations: OS, overall survival; HR, hazard ratio; SWOG, Southwest Oncology Group; CCG, Children's Cancer Group; RTOG, Radiation Therapy Oncology Group; NCCTG, North Central Cancer Treatment Group; NCIC, National Cancer Institute of Canada; ECOG, Eastern Cooperative Oncology Group; NSABP, National Surgical Adjuvant Breast and Bowel Project.

^*Based on a binomial approximation using a Peto effective sample size.

^†Derived using a Poisson approximation for the numbers of deaths.

Crossing Hazards

Depending on the shape of the experimental and control treatment survival curves, early release of data can lead to different conclusions than with additional follow-up. Figure 1 displays hypothetical curves, with the experimental treatment being better on average. Note that although the control treatment curve drops faster than the experimental treatment curve during the first 5 years, the opposite is true for years 6 to 10. This is known as a case of crossing hazards and does not imply that the experimental treatment is worse than the control treatment in the later years (see the Appendix, online only). An important implication of crossing hazards for a conventionally designed RCT is that the trial may have more power to reject the null hypothesis with less follow-up. For example, if the true survival curves are as displayed in Figure 1, then an RCT randomly assigning 750 patients per arm accruing uniformly over 4 years with 2 years of follow-up would have 85% power to reject the null hypothesis (one-sided type I error = 0.025, log-rank statistic). (All power calculations are derived from simulation of 10,000 data sets.) The same trial with 5 years of follow-up would have 54% power. (In special circumstances where one expects the survival curves to come together, alternatives to the log-rank test that weight the earlier data more heavily may be appropriate.)

Fig 1.

Theoretical survival curve data with crossing hazards at 5 years.

An implication of crossing hazards to the topic of early stopping is that an early highly statistically significant result leading to stopping the trial may become less statistically significant (or even not statistically significant) with further follow-up. (This could also happen with additional long-term follow-up of a trial that was not stopped early.) For example, suppose the RCT described earlier with 5 years of follow-up had an interim analysis after 2 years of follow-up that would release the trial results early if P < .0025. This would happen 57% of the time if the true survival curves were as in Figure 1, and with 3 years of additional follow-up, 22% of these times the results would no longer be statistically significant (P > .025). Note that these occurrences would not be false positives because the null hypothesis that the survival curves are identical is not true. However, if one believed that it would be misleading to the clinical community to see only the first 6 years of the curves in Figure 1, then releasing the results early would be a mistake regardless of the statistical significance of the early results. In practice, one will unlikely know beforehand whether the curves will come back together as in Figure 1 or keep separating. This suggests that interim monitoring is appropriate, but additional follow-up after the early release of extremely positive results is advisable.

Empirical evidence of crossing hazards is suggested by Figures 2 and 3, which display the OS curves for EST-3189 and the complete continuous remission curves for POG-9006, respectively. For EST-3189, the P value went from .0025 at the time of interim monitoring boundary crossing to .10 with 2 years of additional follow-up. For POG-9006, the P value went from .0016 at the time of interim monitoring boundary crossing to .22 with 4 years of additional follow-up.

Fig 2.

Overall survival curves from EST-3189 (control treatment = cyclophosphamide, doxorubicin, and fluorouracil [CAF]; experimental treatment = 16-week regimen; P = .10). (Reprinted from Fig 2 with permission.¹⁴)

Fig 3.

Complete continuous remission curves from Pediatric Oncology Group trial 9006. (Adapted from Fig 2¹² by permission from Macmillan Publishers Ltd.)

Type I Errors

Although RCTs are designed to answer definitively a treatment question, they are not perfect. In particular, they will infrequently lead to a rejection of the null hypothesis when it is true (a type I error) or the nonrejection of the null hypothesis when the treatments are truly different (a type II error). A design parameter for RCTs is the type I error rate, which is frequently set at 0.05. This means that if the null hypothesis is true, then there is, at most, a 5% chance that the trial will result in a statistically significant outcome. It is important to note that, in a properly designed trial, the type I error rate encompasses both type I errors that occur when the trial is stopped early for positive results as well as type I errors that occur with a positive conclusion at the regularly scheduled trial end. Therefore, there is not an excess of type I errors as a result of the possibility of early stopping with appropriately designed interim monitoring boundaries.

An alternative way to consider type I errors vis-à-vis concerns about early stopping is to calculate the probability that a positive conclusion is a false positive, given that the trial stopped early. A standard application of Bayes' theorem allows this calculation as a function of the prior probability that the null hypothesis is true.⁶⁴ Such calculations show that, for a positive trial, the probability that the trial is a false positive is lower if the trial crossed an interim monitoring boundary than if it did not. As a simple example, consider a trial designed with 90% power for a specified alternative, with one-sided type I error of 0.025, and where the true treatment effect is null 80% of the time and equal to the specified alternative 20% of the time. Without the possibility of early stopping, the probability that a trial with a positive outcome (one-sided P < .025) is a false positive is 10%. If an O'Brien-Fleming interim monitoring boundary⁶⁵ is used with two equally spaced interim looks, then the probability that a trial that crosses this boundary at the first interim look (33% information) is a false positive is 1.2%, and the probability that a trial that first crosses the boundary at the second interim look (67% information) is a false positive is 4.4%; the overall false-positive rate remains at 10%.

Biased or Implausible Positive Interim Results

It has been noted^66,67 that a treatment effect observed for a trial that stops early for positive results will be, on average, higher than the true treatment effect (ie, is biased upward). Some^2,3,5 use this to argue against stopping trials early for positive effects. However, it is also true that the observed treatment effect for a trial that concludes at its regularly scheduled end with a significantly positive result will be biased upward (although not as high as one that has stopped early).⁶⁸ In particular, for interim analyses occurring with half or more of the total planned events, the upward bias as a result of early stopping is comparable to the upward bias seen in similarly positive trials not stopped early.⁶⁹ It is important to note that even though the treatment effect is biased upward when estimated when a trial stops early for positive results, there is only a small probability (the type I error) that the treatment effect is not positive. Therefore, concerns about treating future patients with the best treatment may outweigh concerns about not knowing exactly how much better the better treatment is. The empirical data in Table 2 suggest that the potential bias as a result of early stopping is not a major problem.

It has been suggested⁷⁰ that if the magnitude of the treatment effect at an interim monitoring look is implausible, then one should not stop the trial at this point (implying that one should stop for a smaller observed effect). However, not stopping a trial for extremely positive results but stopping it for less extreme positive results runs counter to both common sense and statistical thinking⁷¹; see Clayton and Wheatley⁷² for an alternative point of view. Whether or not a trial is stopped early, if one has prior information about the magnitude of the treatment effect, then a Bayesian analysis⁷³ may be useful in providing an attenuated estimator of an extremely positive treatment effect.

DISCUSSION

The vast majority of NCI Cooperative Group phase III trials that crossed an interim monitoring boundary for positive results led to the early release of treatment effect data to the public that, in retrospect, was appropriate and beneficial. Concerns about excess false positives as a result of the early stopping are not supported by statistical theory or the empirical evidence presented here. Concerns about biased treatment effects as a result of the early stopping are statistically valid but may not be practically important; the bias may not be much larger than would be seen for a positive trial not stopped early, and releasing information early about an effective treatment may be more important than knowing the exact magnitude of the benefit. However, concerns about early stopping/release limiting the ability to estimate long-term survival curves (and potentially identify crossing hazards) or to estimate OS curves (when the stopping is based on a non-OS end point) are statistically valid and practically important. An important consideration in this situation is whether the survival curves can be accurately estimated with additional follow-up after the early stopping/release. If the accrual was not complete at the time of early stopping or many patients could be expected to cross over to the experimental treatment when the positive results are released, then it may be impossible even with additional follow-up to estimate what the survival curves would have looked like if there had been no early stopping/release. In this situation, the interim monitoring plan could be conservative during accrual if the monitoring end point is not OS or there is strong interest in the long-term survival curves.

The NCI Cooperative Group trials that we have considered had well-designed interim monitoring plans. The choice of end point and monitoring plan needs to be carefully considered before a trial starts; trial investigators should be comfortable with the predictable stopping and not stopping decisions that will occur under different accruing data scenarios. The ability to stop a trial and release positive data early is an important component of phase III trial design, allowing the public to benefit as soon as possible from the study conclusions.

Appendix

Crossing Hazards

We give a heuristic explanation of how crossing hazards, as in Figure 1, can occur. Consider 100 patients treated with the control treatment. One would expect 30 of these patients to die in the first 5 years (hazard = 30/100 = 30%) and 20 to die in the last 5 years (hazard = 20/(100 – 30) = 29%). Suppose that, compared with the control treatment, the experimental treatment delays for 5 years one third of the deaths (ie, n = 10) that would have occurred in the first 5 years and has no effect on deaths that would have occurred in the last 5 years. One would expect, for 100 patients treated with the experimental therapy, 20 deaths in the first 5 years (hazard = 20/100 = 20%) and 30 deaths in the last 5 years (hazard = 30/(100 – 20) = 37.5%). Notice that the control treatment hazard is higher than the experimental treatment hazard in the first 5 years and vice versa in the last 5 years.

Table A1.

Definitions of End Points Used in Table 1

Trial ID	Primary End Point As Defined in the Primary Publication
NCCTG-844652	Overall survival
RTOG-8501	Overall survival
POG-9006	Continuous complete remission rate: the time from achievement of a complete remission to failure (death, relapse, or second malignancy)
EST-3189	Overall survival
CLB-9011	Complete response: absence of constitutional symptoms and of lymphadenopathy, splenomegaly, and hepatomegaly on physical examination; an absolute neutrophil count of at least 1,500/μL, a platelet count of at least 100,000/μL, a hemoglobin level higher than 11 g/dL (without transfusion), and an absolute lymphocyte count of less than 4,000/μL; and bone marrow of normal cellularity, with less than 30% lymphocytes and no lymphoid nodules
SWOG-8892	Progression-free survival: time from registration to the date of first observation of progressive disease or death as a result of any cause
ECOG-2491	Disease-free survival: the time from the beginning of complete remission to relapse or death as a result of any cause
CCG-1882	Event-free survival: the time from random assignment to relapse at any site, a second malignant neoplasm, or death during remission
SWOG-8814	Disease-free survival: the time from random assignment to recurrence or death (the definition is from the study protocol; primary publication is an abstract)
SWOG-8797	Overall survival
CLB-9344	Disease-free survival: the time from study entry to first locoregional recurrence, first distant metastasis, or death as a result of any cause
RTOG-9001	Overall survival
CCG-5942	Event-free survival: the time to disease relapse, progression, occurrence of a second malignant neoplasm, or death from any cause
SWOG-9133	Failure-free survival: the time from random assignment to the date of disease progression or death
RTOG-9413	Progression-free survival: time to the first occurrence of local progression, regional nodal failure, distant failure, biochemical (PSA) failure, or death as a result of any cause
SWOG-S9701	Progression-free survival: the time from registration to the date of first recurrence or death
NCCTG-N9741	Progression-free survival: the time from study entry to disease progression. Deaths occurring within 30 days of treatment discontinuation were considered disease progression. Without contradictory data, patients who died or were lost to follow-up were assumed to have experienced progression at the time they were last known to be progression free. (This end point was referred to as “time to progression” in the primary publication)
NCIC-MA17	Disease-free survival: the time from random assignment to the recurrence of the primary disease (in the breast, chest wall, or nodal metastatic sites), or the development of a new primary breast cancer in the contralateral breast; secondary cancer or death without a recurrence, or a diagnosis of contralateral breast cancer were not included as events
E-1496	Progression-free survival: the time from maintenance random assignment to progression or death (the definition is from the study protocol; primary publication is an abstract)
E-E1A00	Response rate: best response within four cycles of treatment (4 months from the start of treatment). Standard ECOG response criteria were used. An objective response was defined as a 50% or higher decrease in the serum and urine monoclonal protein levels from baseline. Patients with measurable disease only in the urine needed to have a greater than 90% reduction in 24-hour urine monoclonal protein excretion to be considered as having a response
CCG-1961	Event-free survival: time to relapse at any site, death during remission, or a second malignant neoplasm
E-3200	Overall survival
ECOG-2997	Complete response: response was evaluated according to NCI Working Group Criteria (Cheson BD et al, Blood 87:4990-4997, 1996)
NSABP-B-31/NCCTG-N9831	Disease-free survival: time to local, regional, and distant recurrence; contralateral breast cancer, including ductal carcinoma in situ; other second primary cancers; and death before recurrence or a second primary cancer
ECOG-4599	Overall survival
ECOG-2100	Progression-free survival: the time from randomization to disease progression or death from any cause
NCIC-MA21	Disease-free survival: the time from random assignment to the time of recurrence of the primary disease. Local or nodal recurrence and metastatic disease are considered a recurrence of the primary tumor. Patients who have contralateral breast cancer or a second primary malignancy or die as a result of some cause other than disease will be censored as event free at the time of death (the definition is from the study protocol; primary publication is an abstract)

Abbreviations: NCCTG, North Central Cancer Treatment Group; RTOG, Radiation Therapy Oncology Group; POG, Pediatric Oncology Group; CLB, Cancer and Leukemia Group B; SWOG, Southwest Oncology Group; ECOG, Eastern Cooperative Oncology Group; CCG, Children's Cancer Group; PSA, prostate-specific antigen; NCIC, National Cancer Institute of Canada; NCI, National Cancer Institute; NSABP, National Surgical Adjuvant Breast and Bowel Project.

Note Added in Proof

After this article was accepted for publication, another trial came to our attention that was stopped early for positive results.⁷⁴ The trial was not identified in our search because the abstract reporting the initial results⁷⁵ made no mention that the trial was stopped early.

AUTHORS' DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST

The author(s) indicated no potential conflicts of interest.

AUTHOR CONTRIBUTIONS

Conception and design: Edward L. Korn, Boris Freidlin, Margaret Mooney

Collection and assembly of data: Edward L. Korn, Boris Freidlin

Data analysis and interpretation: Edward L. Korn, Boris Freidlin

Manuscript writing: Edward L. Korn, Boris Freidlin, Margaret Mooney

Final approval of manuscript: Edward L. Korn, Boris Freidlin, Margaret Mooney