Operational Challenges and Solutions with Implementation of an Adaptive Seamless Phase 2/3 Study

Abstract

A wide variety of operational issues were encountered with the planning and implementation of an adaptive, dose-finding, seamless phase 2/3 trial for a diabetes therapeutic. Compared with a conventional design, significant upfront planning was required, as well as earlier, more integrated cross-functional coordination. The existing infrastructure necessitated greater flexibility to meet the needs of the adaptive design. Rapid data acquisition, analysis, and reporting were essential to support the successful implementation of the adaptive algorithm. Drug supply for nine treatment arms had to be carefully managed across many sites worldwide. Details regarding these key operational challenges and others will be discussed along with resolutions taken to enable successful implementation of this adaptive, seamless trial.

Introduction

An adaptive, dose-finding, inferentially seamless phase 2/3 trial was utilized in the development of dulaglutide (dula; LY2189265), a glucagon-like peptide-1 analog.¹ The trial is divided into two stages based on two randomization schemes: a Bayesian response adaptive scheme (stage 1) and a fixed scheme (stage 2). The objective of stage 1 is to identify up to two doses of dula for further study in the trial or to terminate the study early due to futility. If dose selection occurs, the trial will progress into stage 2, where additional patients will be enrolled to meet the sample size requirements for the primary objective. Should this trial continue to completion, it will serve as a confirmatory phase 3 registration trial. This approach differs from the traditional development paradigm in which two trials would be conducted separately and there would be a lag time of approximately 9 to 12 months between them.

This trial design required significant upfront planning, especially with regards to the statistical methodology and operational considerations. While successful implementation of any trial requires coordination across many functions, including biostatistics, clinical operations, data manage-ment, and drug supply, this study required even earlier and more integrated cross-functional integration. The infrastructure normally used to support traditional trials, such as the double-blind, randomized, parallel-group designs, had to be modified because the adaptive design necessitated greater flexibility than what existed. Rapid data acquisition and analysis was required to support updates to the algorithm every 2 weeks. Drug supply for nine treatment arms at all sites during the adaptive stage had to be managed carefully to meet the changing needs, and the commercial formulation of dula had to be established much earlier than in a traditional development paradigm to support this as a phase 3 confirmatory trial. These and other operational issues encountered in the execution of this adaptive, seamless study will be discussed as well as the resolutions applied to ensure successful implementation of this design.

Implementation Challenges and Solutions

The key operational challenges encountered in the design and execution of this adaptive, seamless trial are listed in Table 1 and are compared with those if two trials were conducted separately in a traditional development paradigm.

Table 1.

Operational Considerations: Adaptive, Seamless Trial versus Two Separate Studies

	Adaptive, seamless phase 2/3 study	Separate phase 2 and phase 3 studies
Resources	More upfront planning and cross functional integration	Traditional planning
Documentation	Design sufficiently described in protocol All adaptations had to be prespecified and documented Additional design documentation needed for clarity: ERB supplement, trial simulation report, DMC charter	Designs sufficiently described in separate protocols
Communication plans	Prespecified to manage adaptations and seamless transition	Additional plans not required
Enrollment rate	Enrollment rate targeted to optimize performance of adaptive algorithm Need to monitor and manage enrollment rate	Enrollment may proceed as quickly as feasible
Seamless design feature	Initiating regulatory approval processes for new sites at risk (for stage 2) Uncertainty of timing of these activities Need to minimize operational bias	Studies are conducted separately Potential lag time of 9 to 12 months between studies
Data acquisition and management	Rapid data entry needed Frequent data transfers and data validation to ensure quality of data Rapid data integration, extraction, and reporting required Need for highly integrated data flow system	Standard time frames for data acquisition and transfers
Randomization	Flexible system needed to accommodate adaptive and fixed randomization schemes Developed jointly with data analyses processes	Traditional systems for fixed dose designs may suffice
Drug supply and management	Accelerated formulation needed Plans needed for managing adaptations Greater quantity and more dosage strengths needed Significant material wastage encountered	Acceleration of formulation not necessary Less quantity and material wastage
DMC	Required statistical expertise to monitor adaptive algorithm Responsible to oversee patient safety Involved in decision-making processes Proposed role for limited sponsor involvement Document and implement restrictive firewalls	May be utilized for a phase 3 study, depending on the study objectives, to oversee patient safety

Clinical Operations

Documentation

Challenge

Compared with a traditional study design, the adaptive, seamless design necessitated significantly more docu-mentation for purposes of clarity, transparency, and integrity, given the number of decisions that had to be preplanned. How the trial conduct might change and how those decisions would be made had to be documented. In addition, action plans on how such changes might be executed (for example, dose selection), the timing of interim analyses, what data are needed at these interims and the quality of that data, data flow processes, data unblinding plans, site communication plans, and roles and responsibilities in the decision/termination-making processes had to be prespecified. These details were critical to facilitate both internal review as well as external review by regulatory authorities, investigators, and ethics review boards (ERBs).

Solutions

Protocol

A justification of why an adaptive approach was taken in lieu of conventional methodologies was discussed. A description of the adaptive design features along with an overview of the decision-making processes and committees were described. Specifics related to control of the type 1 error and sensitivity analyses to assess operational bias of the primary analyses were noted in the protocol, but full details were provided in the simulation report and statistical analysis plan (SAP). To minimize operational bias, additional details of the adaptive allocation process (including an example of how adaptive allocation might occur) and dose-selection criteria were not provided in the protocol but were provided to ERBs in a separate document termed “an ERB supplement.” Because many ERBs have little experience with adaptive designs, let alone adaptive, seamless designs, it was critical to ensure an understanding of the adaptations planned to demonstrate how changes potentially impacted the safety of participants.

Informed Consent Document

To ensure trial participants were adequately informed about the trial design, it was necessary to obtain variances from our corporate standard operating procedures (SOPs) in the creation of the informed consent document. For example, no randomization probabilities could be provided since randomization in stage 1 is adaptive; therefore, an overview of the adaptive allocation process was described. Similarly, because the sample size was not fixed, only an approximate range of the number of subjects to be included in the study could be provided. These variances were well documented to maintain internal and external compliance standards.

Statistical Analysis Plan and Simulation Report

A traditional SAP was written and finalized prior to first unblinding. An unblinding plan was included in the SAP that described all planned interim analyses based on the adaptions in the study to prevent introduction of bias and scientific integrity. Access to treatment assignment was permitted only to preauthorized individuals [such as the Data Monitoring Committee (DMC)]. The SAP was finalized prior to first unblinding to maintain study integrity. A separate simulation report was authored prior to the first patient visit, describing in depth the adaptive design features, decision criteria, operating characteristics of the trial, control of type 1 error, and simulations with resulting measures used to compare the adaptive design to a conventional fixed-dose design.

Communication Plans

To ensure swift execution of any number of modifications in this study, such as discontinuing a dula arm or dose selection and initiation of stage 2, plans were developed prior to study initiation regarding how the study team would implement each possible change. This was done to ensure rapid implementation of a decision and communication to sites.

Enrollment Rate During Stage 1

Challenge

Adaptive designs are most efficient when information about effect is available soon after treatment. When end points of interest are not available early, fast enrollment relative to enrollment rate reduces the information that can be learned from the accruing data. Simulations of this trial were conducted to assess the impact of different enrollment rates on the performance of the algorithm, meaning the probability it would select the therapeutically optimal dula doses efficiently or stop the study for futility. An enrollment rate of no fewer than five and no more than eight patients per week yielded the optimal performance of the algorithm.2 The challenge was how to manage enrollment at this rate across a global trial with a lead-in period that could vary between 4 to 11 weeks and how to educate sites about the need for a controlled enrollment rate when they are usually asked by sponsors to expedite enrollment.

Solution

Site personnel were educated at study startup meetings on the goal of enrolling an average of five to eight patients per week during stage 1. Simulation results were shared to exemplify the impact of differing enrollment rates on the performance of the adaptive algorithm.

During stage 1, there were more than 50 sites active in ⁶countries. Site readiness was staggered as a consequence of the timing of regulatory and ERB approvals, which helped to manage enrollment. A cap was placed on the number of patients each site could enroll. In addition, each week, sites were asked to project the number of patients to be screened and enrolled in the subsequent ²weeks (via facsimile). Bayesian statistical methods were used to synthesize this information with the enrollment data collected through the electronic treatment assignment system and to provide estimates about screening, randomization, and screen failure rates. Future enrollment was projected using a nonparametric bootstrap method and predictive probabilities.3 The study team regularly reviewed these data. If the enrollment rate was averaging five to eight patients per week, no action was needed. If the rate exceeded these specifications, the operations team would intervene by temporarily suspending screening at selected sites or by asking sites to defer randomizing a patient for a few days or up to a week. This was a labor-intensive process for both clinical operations and site personnel. With all these measures combined, the overall enrollment rate was successfully maintained between five to eight patients per week.

Seamlessly Transitioning to Stage 2

Challenge

If dose selection occurred at the decision point (DP; i.e., the time when the algorithm either selects up to two dula doses or stops the trial for futility), the timing of when to initiate new country/site approval processes for stage 2 was more difficult to predict. The goal was to minimize the potential lag time between the DP and the addition of new countries/sites needed to attain the required total sample size. Although simulations had been conducted to predict the earliest times (200 subjects enrolled) and latest times (400 subjects enrolled) when the DP²might be reached, the team had to rely upon real-time data to yield the best estimate.

Solution

To accomplish this, the same Bayesian statistical methods used to monitor enrollment were used to predict the earliestand latest time points (i.e., when 200 and 400 patients would be enrolled) and when the DP could be reached; thus this estimate was based on real-time data enroll-ment. These data were then used to decide when to initiate the regulatory approval processes for each of the new countries and sites intended to be added to stage 2. Of course, these preparations were done at risk, as it was not known if the algorithm would select doses or terminate the study early.

Data Acquisition and Management

A plan needed to be in place to minimize the time to extract data, create analysis data sets, and perform data analyses to inform decisions, such as the dula treatment allocation probabilities in stage 1.

Challenge 1

Data accumulating during stage 1 of the trial, specifically the four clinical utility index measures [heart rate, diastolic blood pressure, weight, and hemoglobin A1c (HbA1c)],2 would be used to adapt the randomization to the seven dula doses every 2 weeks. Successful implementation of the response adaptive algorithm would require rapid data extraction, quality data (such as no missing data on the case report form), and timely data analysis. It would also require a computer system capable of running the programming in Fortran for the adaptive algorithm and then outputting instructions for treatment adaptations to be used by the randomization system. The entire process needed to be automated as much as possible. Expertise was needed in computer science to run the adaptive algorithm and in Bayesian statistics to interpret the diagnostics of the algorithm and ensure it was functioning appropriately.

Challenge 2

The execution of this study would require a randomi-zation system that had the flexibility to execute the periodic adaptations to treatment. The system also had to accommodate a fixed randomization scheme duringthe “burn-in” period2 and then switch to the adaptive randomization scheme for which the allocation probabilities would change every 2 weeks. Finally, if dose selection occurred, the system would have to be able to transition back to a fixed randomization scheme for the remainder of the trial.

Challenge 3

The Statistical Analysis Center (SAC) was responsible for monitoring the performance of the adaptive algorithm and data analysis. The DMC was responsible for overseeing patient safety and review of the Bayesian analyses performed for the four variables used in the adaptive algorithm. A means to extract, analyze, and report unblinded data to these committees was necessary so they could monitor the performance of the algorithm and safety data every 2 weeks.

Challenge 4

All of these systems to extract, process, and output data had to be fully integrated.

Solutions: Data Flow

A fully integrated and validated process for data flow (Figure 1) was created by Lilly and several external vendors, defining the SAC. Three companies form the SAC: (1) a consulting firm with expertise in information technology (IT) applications to run the adaptive algorithm; (2) a pharmaceutical services organization with expertise in creating interim DMC reporting, producing the DMC reports, and providing statistical support and project management for the DMC; and (3) a statistical consulting group with expertise in adaptive clinical trial designs and Bayesian methodologies to oversee the performance of the adaptive algorithm and provide statistical support for the DMC during the adaptive portion of the trial.

Figure 1.

Data flow through the adaptive stage of the trial. eCRF, electronic case report form; eDC, electronic data capture; IVRS, interactive voice response system; Lilly, Eli Lilly and Company

Where appropriate and possible, existing internal systems were modified to accommodate the needs of this and future adaptive designs. Other requirements were outsourced to vendors with demonstrated capabilities. The entire data flow process was tested extensively before the first patient visit and later confirmed with actual study data accrued during the burn-in period. A detailed data extraction schedule, a data-flow risk mitigation plan, a roles and responsibilities document, and a data flow communication plan were created to support the system and overall data flow. Contingency plans were prepared for issues that could arise with data transfer, and appropriate firewalls were put in place (e.g., blinding) to ensure the data flow would not compromise the trial integrity.

Solutions: Process and System Infrastructure

The data flow process is briefly outlined as follows (Figure 1): sites enter data into the electronic data capture system, these data are integrated with laboratory data, and blinded data sets are extracted and transferred to the SAC every 2 weeks along with the actual treatment information from the randomization system. The SAC prepares and analyzes unblinded data, calculates the dula randomization probabilities using the adaptive algorithm, automatically sends a file containing the updated treatment allocation probabilities to the randomization system, andcreates interim reports for the DMC. The IT firm’s involvement in the SAC would cease if dose selection occurred and the trial proceeded to stage 2, and data extraction and transfer to the SAC would assume a more traditional frequency (e.g., quarterly).

Solutions: Data Extraction, Analysis, and Reporting

To ensure optimal performance of the adaptive algorithm, timely acquisition of data was necessary. Sites were instructed to enter data into the electronic case report forms within 48 h of the study visit. Data for measures used in the adaptive algorithm² were reviewed and validated daily to ensure data was of high quality. There were many automatic queries that “fired” upon electronic data entry by the site, and manual queries were sent when necessary (e.g., for missing data), with timely site resolution expected within 48 h. Laboratory data were transferred from the central vendor weekly (monthly is more typical for traditional fixed designs), because HbA1c was used in the adaptive algorithm. Hence, the best available data were used to calculate the adaptive randomization probabilities.

Solutions: Randomization System

The existing internal system for randomized treatment assignment and drug management was enhanced to increase flexibility and meet the needs of this study. A new system module (flexible to use for this and future adaptive designs) was created and integrated to adapt randomization in real time based on (1) custom software to integrate adaptive design processing to other core system functions such as dispensing and call flows and (2) a standardized data set that includes treatment allocation probabilities and other necessary data to link to the trial, stage, treatment status, and randomization state to upload and perform quality checks. The new module was developed to support versatile trial randomization designs, such as this trial’s transition from predefined, list-based treatment assignments (e.g., for the burn-in period) to a dynamic, adaptive randomization scheme. It allows multiple randomization states to be configured in advance to facilitate enacting them throughout the course of the trial when needed (e.g., to support the transition to stage 2). It also allows for disabling a treatment arm if randomization to a given arm is stopped due to safety concerns or for other reasons, thus blocking the potential for new patients to be assigned to that treatment. The system can also automatically adjust a treatment-demand predictor value when the randomization adapts, which triggers reevaluation of all sites’ study drug inventory.

Drug Supply and Management

Challenge 1

The planned commercial formulation of dula, and its attendant processes and standards for its use as a studydrug in a confirmatory phase 3 trial, had to be established earlier than in a traditional fixed-dose program.

Challenge 2

There are nine treatment arms in the adaptive dose-finding stage: seven dula doses, active comparator, and placebo. A plan was required to manage drug supply at each site so there would be sufficient stock for each of the nine treatment arms and additional provisioning (supply already on hand) to respond immediately to the adaptive changes in allocation without incurring extensive material waste. This scenario, with its potential treatment variability, along with the proactive operational plan to implement the adaptations without a post-adaptation restocking period, would inherently increase the quantities of study drug needed. Furthermore, the forecasting and management of drug supply was reliant upon system tools based on traditional fixed-dose designs and lacking the integrations that could optimize “responsiveness” during the trial, as demand uncertainty lessened with each adaptation.

Challenge 3

A limited number of study personnel were required to be involved in the communication regarding what dose(s) were in demand (based on the treatment probabilities) to provide oversight and manage drug supply. These same individuals also needed to be apprised of the dynamics of the trial, specifically management of the enrollment rate that led to variances in the frequency with which patients were randomized at a given site. Another challenge was how to appropriately “firewall” these individuals from the rest of the blinded study personnel while still remaining informed about the dynamics of the trial. With more than 50 sites active worldwide during stage 1, this added to the complexity of managing study drug supply across the trial.

Solution 1

The commercial formulation process was accelerated, which represented a significant corporate investment very early in dula development and required proper planning of formulation, manufacturing, and packaging processes.

With seven dula doses being studied, it was not feasible to produce seven different concentrations so that a single injection volume could be used in the study; nor was it appropriate to produce one concentration, because this would necessitate seven different injection volumes, which would undermine the study blind. Instead, three concentrations were generated, and using these concentrations, three injection volumes were identified that yielded the seven different dula doses. At randomi-zation, the patient was assigned study medication and a volume to inject. To maintain the study blind, patients assigned a placebo injection were also randomized to one of these three volumes, thus no correlations with dose could be made.

Solution 2

A plan to manage drug supplies was devised that took into consideration geographic regions, distribution and label strategies, and shipping. This plan also included countries intended to be added in stage 2. Country selection was made well in advance of study start to prepare study drug labeling and regulatory documents. Countries were selected where drug waste could be minimized (e.g., countries with fewer temperature excursions or shorter shipping times). Central distribution of study material and the ability to label material for use in multiple countries was considered advantageous, but this was not always feasible based on differing regulations for the different geographies involved. Adaptive designs can translate into additional supply/demand issues and increased material waste. To address changes in demand, the study drug occasionally had to be shipped urgently to sites with low inventory.

A forecasting simulation system was used; however, it was not capable of simulating the treatment adaptations. The initial supply plan for stage 1 was forecasted by assuming the maximum proportion of patients from the maximum sample size (400) could be allocated to any of the seven treatments. This was increased by a percentage factor to cover other potential supply chain and distribution losses. The supply plan was broken into several resupply orders because of capacity limitations at the third-party filling operation. This allowed adjustments to the orders during trial execution.

Solution 3

Study personnel responsible for study drug supplies needed to understand potential changes to enrollment forecasts to adjust supply inventory or other influences on supply needs, such as dropping a treatment arm due to safety concerns. These same individuals were also knowledgeable of the treatment probabilities. To keep them informed of enrollment, they had to be appropriately firewalled from the remainder of the study team so as not to influence the integrity of the trial or introduce operational bias. To resolve this issue, a “listen-only” telephone line was created to allow those responsible for drug supply to be silent participants during team meetings.

Considerations Regarding the Data Monitoring Committee

The composition of the DMC reviewing the interim data to decide upon adaptations was important. Expertise in various therapeutic areas was required, including endo-crinology, cardiology, and gastroenterology. In addition, the statistical consulting group, which was part of the SAC, had to have expertise in adaptive clinical trial designs and Bayesian methodologies to oversee the performance of the adaptive algorithm and provided statistical support for the DMC during the adaptive portion of the trial.

The adaptive stage of this trial entailed interim monitoring. Access to these data and the results of the ongoing trial were restricted to maintain trial integrity and avoid the possibility of bias being introduced into the study results, as discussed previously.

For this study, the roles and responsibilities of the DMC extended beyond the traditional role of oversight of patient safety to include frequent assessment of analyses performed for the response variables used in the adaptive algorithm. The DMC was also involved in the trial decision-making processes, specifically, they were charged to render an opinion at the DP as to whether or not to continue the trial with a subset of doses based on interim results or terminate the study. Given the complexities of this decision, it was not reasonable to place this decision solely in the hands of the DMC, as the details and ramifications of the decision would not have been known to the development team for some time. Therefore, limited sponsor involvement in the interim review and decision-making processes was predefined. A Lilly internal review committee (IRC) served this role, and the composition of the IRC was restricted to the smallest number of management representatives who could provide the necessary perspectives to make decisions and who were not involved with the dula development program. The IRC would review the same data the DMC had reviewed at the DP to render an opinion, and the IRC could be conveyed if the DMC recommended stopping the trial for futility. It is recognized that this approach conflicts with current practices in nonadaptive settings.

To accomplish all of this, the DMC was assembled early in the trial planning stages and in-depth discussions regarding their roles and responsibilities, and those of the IRC, were held. These decisions were well documented in the DMC charter prior to study initiation, and the restrictive firewalls were prospectively defined. Input from regulatory authorities was sought in advance of study initiation to garner acceptability of the plans.

Conclusions

There are many considerations in planning and executing an adaptive, seamless trial.4,5 The design and successful implementation of this study for the development of dula required extensive upfront planning and marked cross-functional integration. Designing the trial was an iterative process based on the results of trial simulation and input from external consultants, including regulatory authorities. Team members had to be flexible to accom-modate these changes. Study processes were developed in parallel, and decisions and actions were often executed at risk while the design was still being finalized.

All adaptations had to be prespecified. More extensive documentation of the adaptations and the consequences of those changes were required. This was critical to facilitate an understanding of the design, given the current limited experience of investigators, ERBs, and regulatory authorities with these designs. To meet this need, existing study document templates had to be modified or variances sought where the SOPs were not updated or new document types were created such as a simulation report. This generated robust internal discussions and heightened the recognition for greater flexibility and modifications in our documentation operating procedures to accommodate the needs of adaptive designs. Since initiation of this trial, the U.S. Food and Drug Administration has issued a draft guidance on adaptive design clinical trials and detailed expectations regarding adequate documentation of an adaptive design study.⁶

A complex, fully integrated data flow process, including data collection, validation, integration, and data transfer processes, was created utilizing both internal and external resources to ensure optimal performance of the adaptive algorithm. Therefore, existing processes and systems should be assessed early to determine what work can be managed internally (including modifying or replacing existing technologies) and what work should be outsourced to a third-party partner. Data flow and process were automated and standardized as much as possible. Extensive testing of the data flow process was conducted to ensure proper execution of changes made by the adaptive design. Continual oversight, communication, and resolution of any data-flow issues were necessary to ensure success. Although this approach worked, the ultimate solution would be to develop a fully integrated data flow where data from the collection system(s) are automatically extracted and routed directly into the adaptive analysis program, with analysis output randomi-zation probabilities feeding directly into the interactive randomization system.

Given that this trial is intended to be a confirmatory phase 3 trial, the commercial formulation of dula, and its attendant processes and standards for use as a study drug in phase 3, had to be established earlier than in a traditional paradigm. This required a significant initial investment and was begun at risk, as it remained to be determined if dose selection would occur and the trial would proceed as planned.

Managing drug supply for nine treatment arms and ensuring adequate supplies while minimizing waste was very challenging. The ability to model and reassess the dynamic changes in the treatment allocation using simulation software would allow quantification of the likelihood of the extreme case scenarios and aid supply planning. Similarly, linking the supply forecast with the treatment allocation patterns simulated during trial design would further refine the forecast by linking planning with execution. Centralizing distribution of study drug and utilizing a label for use in multiple countries would also be advantageous. If on-demand study drug labeling becomes a reality, this too may ease some drug supply and management issues.

Overall, a number of operational and logistical challenges were encountered in the implementation of this adaptive, dose-finding, seamless phase 2/3 study. Creative solutions were identified, resulting in improvements in our internal processes and procedures. Several modifications were made to our infrastructure to support implementation of this study, which will also benefit future adaptive design trials within our company. The experience and knowledge gained with this adaptive trial should translate into greater efficiencies in the planning, approval, and execution of future adaptive design trials and in the drug development process overall.

Glossary

(DMC)

Data Monitoring Committee

(DP)

decision point

(dula)

dulaglutide

(ERB)

ethics review board

(HbA1c)

hemoglobin A1c

(IRC)

internal review committee

(IT)

information technology

(SAC)

Statistical Analysis Center

(SAP)

statistical analysis plan

(SOP)

standard operating procedure

Funding

This work is sponsored by Eli Lilly and Company. Additional details of this study, entitled “A Study of LY2189265 Compared to Sitagliptin in Patients with Type 2 Diabetes Mellitus on Metformin,” can be found at http://clinicaltrials.gov as NCT00734474.

Disclosures

Kimberly Spencer, Kelly Colvin, Brad Braunecker, Marcia Brackman, Joyce Ripley, Paul Hines, Zachary Skrivanek, Brenda Gaydos, and Mary Jane Geiger are full-time employees of and own stock or stock options for Eli Lilly and Co.