An Overview of the Adaptive Designs Accelerating Promising Trials Into Treatments (ADAPT-IT) Project

Abstract

Randomized clinical trials, which aim to determine the efficacy and safety of drugs and medical devices, are a complex enterprise with myriad challenges, stakeholders, and traditions. While the primary goal is scientific discovery, clinical trials must also fulfill regulatory, clinical, and ethical requirements. Innovations in clinical trials methodology have the potential to improve the quality of knowledge gained from trials, the protection of human subjects, and the efficiency of clinical research. Adaptive clinical trial (ACT) methods represent a broad category of innovations intended to address a variety of long-standing challenges faced by investigators, such as sensitivity to prior assumptions and delayed identification of ineffective treatments. The implementation of ACT methods, however, requires greater planning and simulation compared to a more traditional design, along with more advanced administrative infrastructure for trial execution. The value of ACT methods in exploratory phase (phase II) clinical research is generally well accepted, but the potential value and challenges of applying ACT methods in large confirmatory phase clinical trials is relatively unexplored, particularly in the academic setting. In the Adaptive Designs Accelerating Promising Trials Into Treatments (ADAPT-IT) project, a multidisciplinary team is studying how ACT methods could be implemented in planning actual confirmatory phase trials in an established, NIH funded clinical trials network. The overarching objectives of ADAPT-IT are to identify and quantitatively characterize the ACT methods of greatest potential value in confirmatory phase clinical trials, and to elicit and understand the enthusiasms and concerns of key stakeholders that influence their willingness to try these innovative strategies.

Introduction

The use of randomized clinical trials to determine the efficacy and safety of drugs and medical devices is a complex and often cumbersome enterprise with myriad challenges, stakeholders, and traditions. While the primary goal is scientific discovery, clinical trials must also fulfill multiple regulatory, clinical, and ethical requirements. Innovations in clinical trials methodology have the potential to improve the quality of knowledge gained from confirmatory phase trials, the protection of human subjects, and the efficiency of clinical research. Whether traditional or innovative, poorly designed clinical trials can increase the risk to subjects and the likelihood of error. In traditional designs these risks are familiar – although sometimes poorly understood, but innovative methods may introduce risks in ways that are initially unexpected or difficult to interpret.

Adaptive clinical trial (ACT) methods represent a broad category of innovations intended to address a variety of challenges that investigators have faced for a long time, including sensitivity to prior assumptions and delayed identification of ineffective treatments. ACTs have the potential to accelerate both the evaluation of effective treatments and the identification of ineffective therapies, and are now generally well accepted in exploratory phase trials but not in pivotal, confirmatory trials. The implementation of ACT methods requires greater preparatory statistical analysis and simulation, along with more advanced administrative infrastructure. Since traditional designs typically have a closed form solution to the statistical test used for the primary hypothesis, simulation is not required for the estimation of sample size – although in many cases it would be advisable as it would allow the trial planners to more comprehensively evaluate the impact of varying assumptions used in the sample size calculations. In addition, many statisticians in current practice do not have experience or are uncomfortable with ACT methods.¹ These considerations may be contributing to slower adoption of adaptive methods in larger, multi-center, confirmatory phase trials. Assessing the potential benefits of ACT methods in confirmatory phase trials, ensuring their validity and safety, and promoting careful acceptance of innovation when warranted has the potential to accelerate the development and evaluation of new therapeutic drugs and devices.

The NIH and FDA jointly funded research proposals aiming to evaluate the applicability of novel approaches to the development of and testing of medical products and fostering the development and evaluation of new tools to accelerate scientific discovery which improves the health of the population.² In the Adaptive Designs Accelerating Promising Trials Into Treatments (ADAPT-IT) project, a multidisciplinary group of researchers are applying ACT principles and studying the design process within an existing clinical trials cooperative group, the Neurological Emergencies Treatment Trials network (NETT). NETT’s purpose is to perform confirmatory clinical trials of promising drugs and devices for the treatment of patients with acute neurological illness or injury. NETT therefore provides a “real world” laboratory in which to study the process of collaboration necessary to design and implement ACTs.³ This paper describes some of the current limitations in the contemporary clinical trials and the opportunities adaptive designs may present to improve the efficiency of research. The approach used by the ADAPT-IT project will be briefly summarized to illustrate key, potentially high yield adaptive design concepts.

The problem: limitations of traditional clinical trials

Drug and medical device development in the United States is cumbersome, expensive, and inefficient.^4–6 The reasons for this are complex and to some extent, differ by indication and drug/device class. The extraordinary costs of development often discourage sponsors and investigators from pursuing new drugs or exploring additional indications, especially for time limited acute illnesses. In 2004, the Food and Drug Administration (FDA) issued a Critical Path report “to address the growing crisis in moving basic discoveries to the market where they can be made available to patients” and noted a “stagnation” in drug development, manifested by a drop in new drug and biologic entities submitted to regulatory agencies for approval worldwide.⁴ The FDA report also noted the stifling effect of development costs on new therapies. The 2006 FDA Critical Path Opportunities Report highlighted streamlining clinical trials as an important step towards accelerating and reinvigorating drug and device evaluation.⁷ The recent Institute of Medicine (IOM) Report “A National Cancer Clinical Trials System for the 21st Century: Reinvigorating the NCI Cooperative Group Program” highlighted immense obstacles to be surmounted in moving ideas to successful clinical trials.⁸

One potential contributor to the current state of affairs with respect to the pace and efficiency of discovery is the fundamental structure of most confirmatory phase, contemporary clinical trials. In such trials, most if not all key design characteristics (e.g., the randomization probabilities used to allocate patients to each of the treatment arms, the number of treatment arms, the maximum sample size, eligibility criteria) are held constant during the conduct of the trial. While this approach makes it straightforward to determine the operating characteristics of the trial and provides an intuitive safeguard against bias and error, it may leads to financial inefficiencies.^9–12

One of the most concerning and expensive issues in drug development is the increasing rate of failure during phase III confirmatory testing.¹³ While the root causes of this are multifactorial, one contributor is the limited information available during the design of the confirmatory trial, especially residual uncertainty in the optimal dose, timing of the treatment intervention, rates of side effects, and the likely treatment effect size.¹⁴ Importantly, some failed trials may not represent bad treatments, but rather false negative results attributable to design characteristics. In short, traditional confirmatory phase testing requires investigators to commit, prior to the collection of trial data, to a fixed set of study parameters. Historically, these parameters are typically informed by data from much smaller exploratory phase (pilot/phase 2) studies. These pilot studies are often based upon small numbers of subjects which can limit the number of parameters that can be explored and the accuracy and precision of those estimates. It is common for confirmatory phase studies to fail, in part, because of such problems as misestimates of the optimal dose, or overly optimistic or pessimistic estimates in placebo group primary outcome event rate.

Adaptive Designs Offer Potential Solutions

During the conduct of a trial that is adaptive by design, key clinical trial characteristics may be altered according to predefined rules, over the course of the trial and in response to information that accumulates within the trial itself (Table 1).^{9, 10} At least five possible types of adaptations have significant promise for confirmatory trials, but further experience and research is needed to better assess the advantages and disadvantages of these techniques in specific situations (Table 2). These approaches may lead to reduced exposure of trial subjects to ineffective experimental arms, more efficient and accurate dose finding, and more rapid identification of drugs and devices lacking clinically important benefit, so that resources and future subjects may be allocated to other promising interventions.^{11, 15} The FDA draft guidance on adaptive designs notes that “Compared to non-adaptive studies, adaptive design approaches may lead to a study that (1) more efficiently provides the same information, (2) increases the likelihood of success on the study objective, or (3) yields improved understanding of the treatment’s effect” but warns against “the possible introduction of bias and the increased possibility of an incorrect conclusion.”⁹ Adaptive designs add complexity, require additional planning, and may be more difficult for unfamiliar clinicians and statisticians to understand.

Table 1.

Four types of rules governing clinical trial designs

Rule	Traditional trials - examples	Innovative trials - examples
1. Allocation Rule: How are subjects allocated to different arms of the trial?	Randomization ratio remains same throughout trial	Randomization ratio changes based on data accrued during trial (covariates or outcomes)
2. Sampling Rule: How many subjects are sampled at the next stage? (What is sample size of next stage of trial?)	Group sequential designs with fixed interim analysis point(s)	Internal pilot designs Information based designs; sample size re-estimation
3. Stopping Rule: When does trial end? (efficacy, harm, futility (or low probability of efficacy), safety, “graduation” from Phase II to Phase III, maximum sample size reached)	Trial without any rule for early termination that will always enroll to maximum sample size (uncommon in confirmatory phase)	Stopping rules based on Bayesian predictive probabilities; conditional power; rules
4. Decision Rule: Design change not covered in previous three rules.	Ad hoc changes; such changes in fixed designs may occur based on data safety and monitoring board (DSMB) recommendations - however formal decision rules for DSMBs are not currently common	Changing patient population to pre-specified sub-population based on a predefined rule to improve chance of trial success

Table 2.

Adaptations of potential high yield for confirmatory trials

Potential Adaptations	Examples of Adaptive Techniques
1. Stopping accrual early for high probability of demonstrating efficacy or for futility	Decision rules based on Bayesian predictive probabilities including those from longitudinal or hierarchical modeling;
2. Altering the randomization probabilities between different arms, or adding/dropping treatment arms;	Response adaptive randomization
3. Transitioning between phases or trial goals (e.g. addressing safety then efficacy).	Seamless phase II/III designs; decision rules based on Bayesian predictive probabilities
4. Altering enrollment criteria to change the treatment window or focus on pre-specified sub-populations identified within trial as responsive to therapy;	Decision rules based on Bayesian predictive probabilities or hierarchical Modeling;
5. Evaluating multiple doses, therapies or diseases simultaneously in a single model while adaptively focusing on better performing strategies	Dose response modeling; indication finding designs29; hierarchical modeling

Despite increasing interest in the potential of adaptive designs to accelerate the evaluation of new therapies, important questions remain regarding optimal approaches to the design and implementation of such trials, the degree with which theoretical improvements in efficiency can be realized in actual practice, and methods to avoid or mitigate threats to validity.⁹ It is important to emphasize that an ACT, as defined within this overview, includes only adaptations that are predefined, both in terms of the changes that may be made in the trial parameters and in terms of the decision rules dictating when these changes would be made. When both the range of possible adaptations and the rules that dictate these adaptations are defined a priori, the study design is completely defined and its operating characteristics may be rigorously determined by numerical simulation.^16–18 In contrast, changes in a trial design that are conceived on an ad hoc basis, such as protocol revisions suggested by a data monitoring committee, lead to difficult-to-define type I error rates that may compromise the validity and integrity of the trial results.⁹

The ADAPT-IT Project and NETT: a laboratory to explore confirmatory adaptive clinical trial designs

In the ADAPT-IT project, we will apply and characterize adaptive strategies in planning large scale NIH sponsored confirmatory studies, a setting in which such strategies are infrequently used. Specifically, the ADAPT-IT team will design innovative ACTs to address important questions regarding the efficacy of treatments for patients with acute neurological diseases and injuries and, at the same time, measure attitudes about the innovations and qualitatively study the development process, with the goal of generating new and generalizable information regarding the use of adaptive approaches in confirmatory clinical trials conducted in the academic setting.

The Neurological Emergencies Treatment Trials network (NETT) was funded by NINDS to perform phase III (confirmatory) clinical trials of new treatments for patients with neurological emergencies.¹ The purview of NETT covers all neurological conditions in which emergency treatment may improve outcomes. The trial development process within NETT is outlined in Figure 1. At present, NETT is simultaneously conducting three NIH funded confirmatory clinical trials; it recently completed its first trial. To date, all of NETT’s studies have incorporated a group sequential design with early stopping for efficacy and/or futility and have incorporated blinded sample size re-estimation plans. In addition to the conduct of ongoing clinical trials, NETT is actively developing four new clinical trials, involving both drugs and medical devices, at varying stages of development. These trial design activities provide an opportunity to explore the application of ACT designs and assess the barriers to the acceptance and implementation of these designs. Using actual proposed clinical trials enhances this process by incorporating both practical and theoretical considerations. Adaptive trial experts are working closely with the established NETT and clinical teams to plan alternative innovative designs. The design team presents the adaptive design including its advantages, disadvantages, and logistical requirements to each trial’s PI, collaborators and the NETT team of clinicians and biostatisticians who evaluate the design, provide feedback, and potentially incorporate the proposal or its elements into the clinical trial protocol.

Figure 1.

Trial development process within NETT

Selecting High Yield Adaptive Solutions in ADAPT-IT

The major design goal of ADAPT-IT is to develop, and then test through simulation, adaptive clinical trial features that best address the challenges facing a set of real confirmatory clinical trials. Broad examples of the types of ACTs that may be used to solve these problems are summarized here, as well as the collaborative process that is being used to develop and refine the adaptive elements of these new trial designs.

The adaptive design statisticians on the ADAPT-IT team use Bayesian methods for the majority of the models used in clinical trial design. A full review of Bayesian statistics and the relative advantages and disadvantages of Bayesian versus frequentist designs is beyond the scope of this review, but other works address these questions in some depth.^19–22 It is important to note, however, that both frequentist and Bayesian methods are used in both fixed and adaptive designs; not all Bayesian clinical trials are adaptive and not all adaptive trials are Bayesian. One of the advantages of Bayesian approaches is that relatively complex models can be estimated (such as predicting the overall likelihood of trial success when only short term response outcomes are available from some patients) using all of the data accrued from subjects. Advances in computing power have made allowed biostatisticians to feasibly fit these more complicated models and incorporate them into the operation of clinical trials.

Six key design features or elements characterize the approach to ACTs that are used in this project (Table 3). All elements have not necessarily been used in every trial, as a feature of the adaptive approach is tailoring the design to trial-specific research questions. In general, the approach hinges on frequently-updating Bayesian probability distributions for key trial parameters used to predict future events, which in turn support ongoing decision-making according to pre-planned rules. Parameters whose probability distributions are updated during the course of a trial include measures of treatment effects, rates of adverse events, characteristics of dose-response relationships, or the strength of relationships between the primary and intermediate endpoints. Examples of using predictive probabilities for ongoing decision making include predicting whether the current trial will ultimately demonstrate a clinically-important treatment effect, whether the current experimental treatment would be successful in a separate phase III trial, or whether a particular dose meets defined efficacy requirements.^{14, 23, 24} As a specific example, consider a trial to determine the optimum duration of therapeutic hypothermia after cardiac arrest. Traditionally, a Phase II trial would first examine the safety of several durations (i.e. 12, 24, 36, and 48 hours) and then two would be selected to go forward to a Phase III trial. An adaptive approach allows the investigators to actually estimate the shape of the “dose” (in this case duration) response function and estimate the duration most likely to improve outcomes. Response adaptive randomization could be used to allocate subjects in such a trial to maximize the learning in the part of the duration response curve that appears to have the most efficacy. This would simultaneously improve the precision of the estimate of the treatment effect and minimize the number of patients who are exposed to ineffective durations.

Table 3.

Key elements of adaptive design approaches used by ADAPT-IT

Element	Description
Frequent Interim Analyses Using All Currently Observed Information	Interim analyses typically involve assessing evidence of treatment efficacy, rates of adverse events, and patient safety. These analyses may be done with the goal of terminating a trial early for futility or success, or may consider other design alterations or transitions. For example, in a seamless phase II/III design, a design rule would define the circumstances under which the trial would transition from phase II to phase III, with the goal changing from dose-finding to definitively establishing the efficacy of the dose identified as most promising. Contemporary trials often incorporate multiple looks to stop early for futility or overwhelming efficacy; ACTs can have more frequent looks to make pre-specified additional alterations beyond stopping early. Modeling strategies can allow for greater use of information which has accrued within the trial.
Use of predefined decision rules at each interim analysis, based on predictive probabilities	One may calculate the predictive probability that the trial will demonstrate a clinically important treatment effect if the trial is continued to completion, to determine whether a trial should be terminated early for success or futility. In addition, for a trial that requires the long-term evaluation of patient outcomes, one may determine the predictive probability that the trial will successfully demonstrate a treatment effect under the assumption that no new subjects are enrolled but the existing subjects are followed for the planned duration. In this way, patient resources may be saved when it is highly probable that the currently enrolled sample size is sufficient to achieve trial success.
Longitudinal modeling to improve interim estimation and decision-making	This makes maximal use of even partial information acquired from each subject. For example, although the final outcome after a serious injury or illness may not be known for several months, more proximate outcomes (e.g., changes in neurologic status, the presence of multisystem organ dysfunction) may be available. These more proximal outcomes are often correlated with the primary outcome. Thus, longitudinal modeling can be used to learn about the relationship between proximal and primary outcomes during the course of the study. This is not to develop clinical prediction models for use outside the trial, but allows the use of all accumulating information when making decisions regarding trial design (such as early termination given a low predicted probability of declaring efficacy at the end of trial), but without the need to make permanent, strong assumptions regarding the existence or strength of these relationships. These models can and should be flexible and may contribute very little to decision making when the relationship between proximal and final outcomes is weak.
Response-adaptive randomization	The ongoing modification of the probabilities used to allocate patients to different treatment arms or to different doses of an active agent. For example, patients may be preferentially randomized to the treatment arm that appears most promising, based on the accumulating data, to improve the outcomes of patients treated in the trial. This may be a particularly important goal if the disease being studied leads to significant mortality or to long-term morbidity, or if patients are enrolled in the trial under an emergency exception from informed consent (21 CFR §50.24). In a trial that includes a dose-finding objective, patients may be preferentially randomized to doses in areas of the dose-response curve that are particularly relevant. This may lead to substantial improvements in efficiency, especially when the location of the optimal dose was not well-known at the beginning of the trial.
Dose response modeling	Using a dose-response model, assumed to be a continuous, but not necessarily linear, function of dose, rather than attempting to learn about each dose’s effect in isolation can increase the efficiency with which we gain information about the effect of each dose by increasing the precision of estimates of dose-specific treatment effects,
Extensive trial simulation	ACTs are fundamentally more complex than traditional trials, making it more difficult to characterize their performance. For example, a typical adaptive trial may include multiple interim analyses and, at each of these analyses, the trial may be stopped to declare success, stopped for futility, the randomization proportions may be adjusted, or the goals of the trial may be shifted from a phase II dose-finding goal to a phase III efficacy goal. Nonetheless, it is critical that key performance characteristics (e.g., type I error rates, power, and accuracy in parameter estimation) be well-characterized over the full wide range of possible true treatment effects, patient population characteristics, and adverse event rates. In order to precisely and fully characterize the performance characteristics of a proposed design, extensive simulations are conducted, typically with thousands of trials simulated for each of a wide variety of possible scenarios. Each scenario consists of a defined treatment effect size or dose-response profile, patient accrual and drop-out rates, adverse event rates, and other parameters.

Probability distributions are used to represent the current state of knowledge for each parameter. As the trial progresses, these distributions are updated with accumulating results.^{25, 26} These methods naturally lead to an estimation of trial parameters and future patient outcomes, in a manner analogous to multiple imputation, incorporating multiple sources of uncertainty and using all available information. A key component of the Bayesian approach is that the ‘act of looking at the data’ or ‘taking apossible design action’ does not alter future probability calculations, making the Bayesian approach ideal for adaptive trials.

Design activities are occurring as follows. First, an initial face-to-face “kick off” meeting is held for each trial where the participants discuss the goals and requirements of the trial, the originally proposed design, potential efficiencies to be realized from an adaptive design, and possible barriers to implementation. Next, an initial design approach is proposed by the adaptive design consultants and presented to the broader group. This design is then subjected to further refinement and simulations, incorporating a wide range of assumptions includingvarying treatment effect, accrual rates, and covariate distributions. Operating characteristics of the proposed ACT designs are estimated using trial simulation. Additional discussions occur as necessary to iterate the design adjustments and re-simulation. In many cases, simulations are programmed in open source statistical packages, including R (cran.r-project.org) and OpenBugs (http://www.openbugs.info/w/) allowing other interested researchers and regulatory bodies to verify the design assumptions that were used in trial planning. The final designs are presented at summative face-to-face meetings.

Understanding Stakeholder Acceptance of Innovation Involving Adaptive Features in Confirmatory Phase Clinical Trials

Innovation is a double-edged sword in scientific research. While innovation is valued and widely recognized as a fundamental driver of scientific progress, initially it can be thought to entail additional risk compared to well-established and familiar methodology. This concern is particularly relevant in confirmatory phase clinical trials because the results are not just intended to inform a subsequent design but instead are meant to influence patient care outside the trial.

ADAPT-IT employs a mixed-methods evaluation of the design and implementation process for confirmatory phase ACTs. Mixed methods research, which uses both quantitative and qualitative methods, has come to be recognized in the social and health sciences as a useful model for conducting research to improve the understanding of observations.^{27, 28} The evaluation includes mini-focus group evaluations of meeting participants; unstructured observations during meetings; quantitative visual analog scale attitude assessments and qualitative short-answer written assessments about the strengths and limitations of each proposed trial; interviews with study section members and journal peer reviewers; summative evaluations and content analysis of the summary statements returned after study section peer review of the proposed trials.

Conclusions

Through better awareness of what adaptive innovations can and cannot offer in confirmatory clinical trials, and what may or may not be acceptable to stakeholders, the ADAPT-IT project will advance our understanding of how to improve this critical phase of the drug development enterprise. At the same time, the ADAPT-IT project will have a very pragmatic and concrete payoff in the form of detailed and innovative ACT designs for several important neurological emergency conditions, including stroke, status epilepticus, spinal cord injury, and post-cardiac arrest encephalopathy that can be immediately implemented. The designs will either be entirely implemented in the clinical trials, or will help inform important improvements in the final protocols of the trials.

We hope to expand the understanding of circumstances in which adaptive designs for confirmatory trials offer meaningful advantages and in which they do not. In addition, we will gain important insights into the barriers and challenges that may be inhibiting the more rapid adoption of these designs in situations where they could be beneficial. Finally, we hope to be able to offer a model for discussing, considering, and communicating adaptive trial designs for the confirmatory setting across the entire clinical trial process – from the initial idea, to the grant proposal, the trial conduct, the reporting of the trial results, and the interpretation of the results by the broader clinical community.

The ADAPT-IT team will explore important questions regarding the challenges of adaptive designs. Adaptive designs add complexity and cost to designing and implementing a trial and require more resources prior to grant submission – when a project is at high risk of being terminated. Optimal methods will be developed to express the potential efficiencies and costs of different types of clinical trial designs and which situations are suitable to more complicated, adaptive designs. Such designs can behave unpredictably if not carefully simulated, and the designs themselves are more opaque given broad lack of current understanding of statistical simulation in the clinical community. Since it will be crucial for adaptive trials to be credibly interpreted and translated into clinical practice, our process evaluation and data collection process will gain important insights into the pathway of clinical discovery which uses adaptive designs in confirmatory phase trials.