Randomization and Statistical Power: Paramount in Trial Reproducibility (Even for Rare Cancers)

Abstract

This article comments on the importance of randomization and statistical power in clinical trials.

Research in medicine and biology is not dictated by the laws of physics. The location of a celestial body can be determined with great accuracy with relatively simple series of equations, but multiple consistent observations are needed in medicine and biology to achieve what is considered a standard practice or a biological inference. Probability and statistics are the engines that guide these inferences. There is no escaping statistics, be it in the identification of the Higgs boson [1] or the identification of one treatment as better than another.

Examples in the recent literature underscore the loss of focus on this most basic issue in the sciences in our rush to find new treatments and to further academic careers through publications. The American Society of Clinical Oncology (ASCO) published a list of the 58 studies reported in 2011 that significantly altered the way cancer is understood or had a direct effect on patient care. Of those citations, 33 were abstracts, 29 were reported at an ASCO meeting, and 9 were announcements or press releases from the U.S. Food and Drug Administration [2]. Should an abstract or a press release serve as sufficient evidence for clinical decision-making? Arguably not.

In a fascinating study of preclinical cancer research, Begley and Ellis reported in a commentary that only 6 of 53 publications cited as landmark publications could be replicated in an independent laboratory [3]. Should a single manuscript serve as sufficient evidence to draw a scientific conclusion? Again, arguably not. In this spirit, this commentary is offered on the manuscript of Pautier et al., which compares the benefit of gemcitabine alone versus the doublet of gemcitabine and docetaxel in patients with leiomyosarcoma (LMS) [4].

The interest in gemcitabine-docetaxel in soft-tissue sarcoma stems from a phase II clinical trial led by Martee Hensley, reporting that 18 of 34 patients with metastatic leiomyosarcoma (mostly uterine) achieved a Response Evaluation Criteria In Solid Tumors partial response as best result [5]. Later, Leu et al. reported the synergy of gemcitabine and docetaxel, offering a possible biological explanation for the clinical report of Hensley of two “inactive” drugs [6–9] becoming effective when combined in a specific manner [10]. Written as a phase III study, the Sarcoma Alliance for Research through Collaboration (SARC) study 002 demonstrated that gemcitabine and docetaxel were superior in terms of progression-free overall survival compared to gemcitabine alone in patients with metastatic soft-tissue sarcoma [11].

The new publication of the TAXOGEM study from Pautier et al. arrives at different conclusions than SARC002 [4]. The main conclusions are as follows:

1. Both regimens were efficacious, and gemcitabine alone yielded comparable results with less toxicity.

2. LMS in uterus responds differently (and better) than nonuterine LMS.

3. Durable stable disease is an important endpoint for patients with LMS (progression-free survival >40%).

The use of gemcitabine-docetaxel in sarcoma was last discussed in The Oncologist in 2007 [12]. We have updated tables from that review that cite response and survival data of gemcitabine alone and/or in combination with docetaxel (see Table 1). In Table 2, we offer a literature review of these two therapies according to histologic type, and in the case of leiomyosarcoma, primary site of disease, examining both sarcomas of soft tissue and bone. These data suggest that the combination of gemcitabine and docetaxel has significant activity and provide context for the present study; in randomized and nonrandomized trials alike, the activity of the combination appears greater than single-agent gemcitabine.

Table 1.

Overall response rates and overall survival times for gemcitabine alone and/or in combination with docetaxel in metastatic/unresectable soft tissue and bone sarcomas

Abbreviations: ASPS, alveolar soft part sarcoma; G, gemcitabine; G+D, gemcitabine + docetaxel; HGUPS, high-grade undifferentiated pleomorphic sarcoma; LMS, leiomyosarcoma; MFH, malignant fibrous histiocytoma; MPNST, malignant peripheral nerve sheath tumor; NA, not available; nos, not otherwise specified; PD, progressive disease; RMS, rhabdomyosarcoma; STS, soft tissue sarcoma.

Table 2.

Histologic subtypes of responders

^aSome reported studies were not sufficiently detailed to permit inclusion of denominator information.

Abbreviations: MFH, malignant fibrous histiocytoma; MPNST, malignant peripheral nerve sheath tumor; NOS, not otherwise specified.

The TAXOGEM study also used a creative design, consisting of two parallel randomized phase II trials. The text identified Simon as a source for the trial design, albeit without a specific reference. As a metric to declare a significant difference in the study, a “selection” paradigm based on response was used, not a comparative paradigm. Thus, the probabilities the study cites do not represent the power of the study, but rather the probability that the arm with the higher true response rate comes out on top at the end of the trial, a weaker comparative design. This weaker design should only be used (if ever) to choose which among competing experimental regimens deserves further testing, not to declare superiority, equivalence, or noninferiority. To be true to this spirit would mean the authors should conclude that gemcitabine-docetaxel is the winner in the uterine arm (and vice versa).

The true power calculation of this study depends on the type I error chosen, which was not included in the study design. Assuming a liberal one-sided alpha of 20%, which is favored by the National Cancer Institute, yields a power for the uterine stratum (40% vs. 50% overall response rate) of 30%, far below the 80% power used in most screening studies. For the nonuterine stratum (20% vs. 40% overall response rate), the power is 55%. In other words, the parallel phase II studies are not powered to make a determination of which treatment is better. Although randomization is important, without sufficient power, the data of this study cannot be used to judge the relative benefit of gemcitabine and docetaxel versus gemcitabine alone in metastatic leiomyosarcoma.

It is also important to note a common potential bias in studies using progression-free survival study designs. If the more toxic treatment leads to slower and later assessment of disease, the more toxic therapy will appear to be better just on the basis of repeating evaluations (scans) less frequently than in the standard arm. In this study, images were repeated at 6-week intervals in one group and 8-week intervals in another, which clouds interpretation of the data as well.

We would also like to correct what appear to be misconceptions of the statistical design of SARC002. The discussion of the TAXOGEM study expresses a concern for the Bayesian study design of SARC002. We concur that the frequentist foundation of statistics is far more common and even conventional as described in the TAXOGEM study. The authors indicated that the SARC study was imbalanced for LMS histology. However, this is the precise intention of an outcome-adaptive Bayesian randomized clinical trial design [27]. SARC chose a Bayesian-based “play the winner” strategy for several reasons:

1. It was hoped that such a design would reduce the number of subjects treated with the inferior regimen [28, 29].

2. The above-mentioned phase II study was successful in reducing the number of subjects needed in each stratum.

3. We wanted to explore new ways to make improvement in a rare set of diseases occur more quickly.

We applaud the efforts of the French Sarcoma Group to address the question of therapy for rare diagnoses such as metastatic leiomyosarcoma. As noted in the first part of this commentary, reproducibility is the only way we can make firm treatment recommendations. However, the data that contribute to those recommendations must be of sufficient quality, even in rare diseases, to use as guides to treatment. The lack of statistical power is as important as a concern as randomization itself in drawing conclusions about the quality of data we examine.

See the accompanying article on pages 1213–1220 of this issue.