Implementing Historical Controls in Oncology Trials

Abstract

Drug development in oncology has broadened from mainly considering randomized clinical trials to also including single‐arm trials tailored for very specific subtypes of cancer. They often use historical controls, and this article discusses benefits and risks of this paradigm and provide various regulatory and statistical considerations. While leveraging the information brought by historical controls could potentially shorten development time and reduce the number of patients enrolled, a careful selection of the past studies, a prespecified statistical analysis accounting for the heterogeneity between studies, and early engagement with regulators will be key to success. Although both the European Medicines Agency and the U.S. Food and Drug Administration have already approved medicines based on nonrandomized experiments, the evidentiary package can be perceived as less comprehensive than randomized experiments. Use of historical controls, therefore, is better suited for cases of high unmet clinical need, where the disease course is well characterized and the primary endpoint is objective.

Implications for Practice

Incorporating historical data in single‐arm oncology trials has the potential to accelerate drug development and to reduce the number of patients enrolled, compared with standard randomized controlled clinical trials. Given the lack of blinding and randomization, such an approach is better suited for cases of high unmet clinical need and/or difficult experimental situations, in which the trajectory of the disease is well characterized and the endpoint can be measured objectively. Careful pre‐specification and selection of the historical data, matching of the patient characteristics with the concurrent trial data, and innovative statistical methodologies accounting for between‐study variation will be needed. Early engagement with regulators (e.g., via Scientific Advice) is highly recommended.

Short abstract

Drug development in oncology has broadened from mainly considering randomized clinical trials to also including single‐arm trials tailored for very specific subtypes of cancer. They often use historical controls, and this article discusses benefits and risks of this paradigm and provide regulatory and statistical considerations.

Introduction

In oncology, nonrandomized trials have been considered in some circumstances as a valid option to investigate treatment efficacy by regulators. A survey of European Medicines Agency (EMA) and U.S. Food and Drug Administration (FDA) approvals granted between 1999 and 2014 reported that 66% (49/74) of approvals based on nonrandomized experiments were in oncology, including hematological malignancies [1]. In particular, FDA has regularly given approvals based on single‐arm oncology trials (SATs) [2]. In Europe, between 1995 and 2014, more than 20% of oncology approvals were based on nonrandomized experiments [3].

The current paradigm in oncology drug development has indeed broadened from randomized clinical trials run on a large and heterogeneous population aiming at proving treatment benefit on overall or progression‐free survival, to include single‐arm trials tailored for very specific subtypes with observed response rates as primary endpoint [2]. Baskets trials are an example of this shift and, although they can involve randomization, are often planned as several independent single‐arm trials conducted with Simon's two‐stage design in each of the conditions in which the drug is investigated [4, 5, 6, 7, 8].

When and How to Implement Historical Controls

Single‐arm trials do not include concurrent controls, and therefore counterfactual information needs to be found in external sources. The E10 guideline of the International Conference on Harmonisation of Technical Requirements for Registration of Pharmaceuticals for Human Use (ICH) on the choice of the control arm [9] provides guidance on when to use historical controls and emphasizes that they are preferable when used to avoid difficult or unethical experimental situations, such as when the disease is rare. In particular, it recommends that substantial efforts have to be devoted to reducing the potential lack of comparability between the active treatment arm and the historical control group because of the lack of blinding and randomization. Adequate covariate adjustment will therefore be needed in the statistical analysis of the trial data ([10] and references therein). In particular, concurrent and historical data should be as similar as possible in regard to the key characteristics defining the trial logistics, inclusion criteria, and treatment confounders, such as age, gender, clinical history, response to former treatments, stage of the disease, genomic subtype, length of follow‐up, etc. Matching can, however, be challenging, as the relevant information might not be routinely collected in the available sources of historical data.

When resorting to disease registries as sources of historical controls, additional problems can also be encountered. For example, there might be a bias of selection because in a randomized controlled trial (RCT), patients who must give consent to receive an intervention might be unrepresentative of those who might be recorded in a disease registry, for which patients’ information is sometimes recorded spontaneously at diagnosis. Furthermore, in time‐to‐event trials, as for most of the oncological conditions, the problem of guaranteed survival needs to be addressed [11]. Indeed, in order to be recorded in a disease registry, patients need to be alive at the time of diagnosis, whereas patients in a trial need to be alive at the time of randomization, meaning they had to survive longer: quite apart from the bias in selection that might result, this raises the issue of an appropriate time origin [12, 13, 14, 15].

In summary, the cornerstone of the evaluation of the treatment effect when resorting to external information is the appraisal of the shift from historical controls to concurrent ones. In order to assess the validity of a trial based on the inclusion of external controls, it is fundamental to evaluate how relevant the outcome obtained from historical data is, as compared with the one that would have been obtained with concurrent controls.

As a consequence, all these caveats have restricted the use of single‐arm trials to situations where the rarity of the disease or of the subtype investigated has rendered randomization impossible or unethical. Loss of equipoise and a strong belief of the superiority of the experimental therapy as compared with other alternatives are also often key prerequisites [9]. In order to attempt to disentangle the treatment effect from regression to the mean, which cannot be controlled for because of the absence of randomization, single‐arm trials are better suited for a setting where the disease has a predictable course. Moreover, the mechanism of action of the drug investigated needs to be well understood and characterized. The ICH E10 guideline also encourages the choice of an objective endpoint. In oncology, for example, the objective response rate is measured as tumor size changes using radiological imaging, often in reference to RECIST, which is regarded a standard measure of tumor response [16]. Simon et al. [2] and Casali et al. [17] proposed a discussion of its appropriateness as an endpoint in single‐arm trials and gave some suggestions of improvements of its sensitivity.

Accessing Historical Data

Historical data can take the shape of aggregated response rates or individual patients’ data and can nowadays be accessed in many different sources. For example, the ‘Historical Trial Data Sharing Initiative’ of TransCelerate enables drug developers to share anonymized information from placebo and standard of care control arms of completed clinical trials [18]. Project Data Sphere allows searching and selecting online individual patient data from completed phase III cancer clinical trials and even proposes a SAS built‐in analysis tool [19]. Other institutions like the European Organisation for Research and Treatment of Cancer or collaborative initiatives like Clinical Study Data Request [20] or the Yale University Open Data Access project [21] have also put efforts to give access to clinical data upon submission and validation of a research proposal.

Once the potential sources of information have been identified, the studies constituting the historical benchmark have to be carefully selected. In particular, the inclusion criteria have to be specified prior to the completion of the concurrent data in order to avoid any advantageous cherry‐picking. Some recommendations on this matter can be found in [10, 18, 22].

Statistical Analysis

Several important reviews exposed a spectrum of methods for incorporating historical controls, ranging from the simplest ones combining the observed response rates of previous studies to the most complex ones relying on Bayesian methodologies such as power priors [12, 15, 18] and the meta‐analytic predictive prior of Schmidli et al. [13] The main drawback of these methods is a potential inflation of the type I error rate when the response rate in the historical data sets is much lower than the response rate that would have been observed had the trial included a concurrent placebo arm [12]. Moreover, the importance of accounting for the between‐trial heterogeneity has been raised by several authors and could be addressed by using a proper hierarchical model [10, 15, 23]. As explained above, important efforts will have to be devoted to match the historical placebo patients to the concurrent trial in terms of inclusion/exclusion criteria, demographics, and potential modifiers of the treatment effect.

A stimulating example of the reuse of historical data in oncology is the construction of a benchmark for planning metastatic pancreatic cancer trials [24].

Regulatory Interactions and Benefit/Risk Assessment

Although their use in early phase trials is becoming frequent, the implementation of historical controls in confirmatory trials is still rare.

Indeed, confirmatory single‐arm phase II trials based on response rate are generally well received, provided all the caveats about data selection and integration outlined here are properly addressed. In this respect historical controls provide a yardstick, however imperfect, to put the results of the experimental arm in perspective. However, when supplementing a randomized clinical trial, the potential conflict between historical and concurrent controls brings a supplementary issue. In case of discrepancies, regulators are sometimes not able to assess which control arm to take as true benchmark. If the concurrent controls are in general taken as real evidence, the historical studies can be downweighed to a certain extent [13]. This often necessitates the implementation of Bayesian methods and therefore the choice of prior distributions, which might differ on both sides of the regulatory divide (although regulators are starting to be more open to Bayesian developments [25]). Another issue is that these methods tend to not guarantee control of the type I error rate [12], which is a regulatory mandate in confirmatory trials [26, 27].

Therefore, when resorting to historical controls in confirmatory trials, early interaction with regulators is recommended. In Europe, Scientific Advice is an ideal forum to seek agreement on the choice of historical studies, construction of the historical benchmark, and design of the statistical analyses (including the potential choice of the prior distributions).

In terms of level of evidence required for drug registration, approval of a drug whose efficacy is shown in a single‐arm trial mainly through historical comparisons would require a dramatic and durable treatment effect in order to offset the different logistic and methodological biases described above. Indeed, according to the ICH E10 guideline, externally controlled trials tend to overestimate treatment efficacy. In a context of high unmet clinical need, regulators need to face the patients’ urgency to access new treatments and to deal with a trade‐off between, on one side, the impossibility to run a gold‐standard RCT and, on the other side, an evidence of treatment efficacy potentially tarnished by the limitations of SATs. Ironically, in such a context where drug development is demanding, the required regulatory standard might be thought to be lowered by the use of SATs, whereas, on the contrary, in order to make up for all the biases inherited from the lack of concurrent controls, the level of observed treatment effect required to convince regulators would probably be much higher than what would have been needed with an RCT [9]. Regulatory guidance on relevant clinical effect to target in such a setting is sometimes pledged by drug developers. However, in a context where drug development programs can be very different in terms of indication, availability of alternative treatments, quality of the historical controls and of their integration, etc., it seems challenging and hazardous to advocate for clear‐cut thresholds to define clinical efficacy, even for a given specific indication. The reader is referred to [19] for a discussion on this matter.

In order to accelerate access to medicines in certain areas of high unmet clinical need, the EMA introduced in 2006 the possibility of granting a conditional marketing authorization (CMA) to products demonstrating a positive benefit/risk profile but for which supplementary information is required [3, 28, 29, 30]. To do so, the benefit for public health in placing the product earlier on the market has to outweigh the risk born by the absence of more comprehensive data. Sponsors have an obligation to provide the required data in specified timelines to confirm the benefit/risk profile granting the CMA (e.g., longer‐term survival, other clinical studies, etc.). The fulfilment of this obligation is annually reviewed, and the CMA is ultimately converted into a regular approval if the updated benefit/risk profile is positive.

Because of all the limitations of SATs outlined above, submissions based on an SAT are, provided that a positive benefit/risk profile can be observed, more likely to be granted a CMA than a full approval. Between the introduction of the CMA in 2006 and October 2016, 68 initial applications in oncology received an approval—whether standard marketing authorization application (MAA), CMA, or MAA under exceptional circumstances—among which 18 were based on an SAT and 50 were based on an RCT as source of evidence for efficacy. Out of these, 5/18 (28%) and 8/50 (16%) were granted a CMA for SATs and RCTs, respectively [3].

In 2017 the EMA published a 10‐year report on its experience with CMA (https://www.ema.europa.eu/documents/report/conditional-marketing-authorisation-report-ten-years-experience-european-medicines-agency_en.pdf) showing that for all the drugs granted a CMA before 2010 (17 oncology products), the CMA was converted into a full MAA by the end of 2016. We completed the data published in this report by collecting all CMA granted until the end of December 2019 and by extending the follow‐up of those granted before. By the end of 2019, the CMA was converted for 15 out of 26 oncology products.

Interestingly, 10 applications out of 26 (38%) were exclusively based on one or several SATs. These submissions all occurred after 2011, and restricting the analysis to this period of time, the percentage of CMA subsequently converted into a full MAA was slightly lower when supported only by an SAT (4/10, 40%) than when including at least one randomized experiment (6/11, 55%) in the evidence package. Using the same post‐2011 data, the median time from CMA to conversion to regular approval was 4 years.

For the first time a CMA was revoked in 2019. In this case, an RCT (ANNOUNCE) failed to show an improvement in time‐to‐event endpoints in sarcoma after a CMA had been granted in 2016 based on superiority of olaratumab in combination with doxorubicin compared with doxorubicin alone [31]. Despite this result, the CMA is still being advocated as a useful tool to accelerate access to innovative medicines for patients in situations of unmet clinical need, provided that uncertainties are clearly communicated to patients and physicians [31].

Conclusion

Leveraging the information brought by historical controls could shorten development time and, even more importantly, reduce the number of patients enrolled in clinical trials. Although planning single‐arm trials for regulatory purposes using a historical benchmark is not rare in oncology, especially with the advent of basket trials, this is less frequent in other therapeutic areas, especially for supplementing a confirmatory randomized clinical trial. In any case, this will require a careful selection of the past studies and a statistical analysis accounting for the heterogeneity between studies. This will need to be prespecified, and early engagement with regulators will therefore be key to success. From the regulatory point of view, although both EMA and FDA have already approved medicines on the basis of a positive benefit/risk profile supported by an SAT, such submissions can be perceived as a less comprehensive piece of evidence than those based on randomized experiments. Confirmatory trials resorting to historical data as the sole source of counterfactual information may therefore be better suited for cases of high unmet clinical need, where the disease course is well characterized and the primary endpoint is objective.

Author Contributions

Conception/design: Olivier Collignon, Anna Schritz, Riccardo Spezia, Stephen J. Senn

Provision of study material or patients: Olivier Collignon, Anna Schritz, Riccardo Spezia, Stephen J. Senn

Collection and/or assembly of data: Olivier Collignon

Data analysis and interpretation: Olivier Collignon

Manuscript writing: Olivier Collignon, Anna Schritz, Riccardo Spezia, Stephen J. Senn

Final approval of manuscript: Olivier Collignon, Anna Schritz, Riccardo Spezia, Stephen J. Senn

Disclosures

The authors indicated no financial relationships.