EFFICACY-TO-EFFECTIVENESS CLINICAL TRIALS

Abstract

Efficacy trials, which assess treatments in optimally selected patients under advantageous conditions for relatively short time periods, are necessary to gain regulatory approval for marketing. In contrast, effectiveness trials, which test treatments across a spectrum of patients in real-world conditions with follow-up periods that match typical treatment regimens, provide critical information on drug effects in those patients who may ultimately receive the treatment. We previously proposed a study design that integrates efficacy and effectiveness trials into a 2-component “efficacy-to-effectiveness (E2E) trial,” in which if the initial efficacy trial component is positive, then the trial immediately and seamlessly transitions to the effectiveness trial component.

However, we believe that total study duration could be even further shortened by simultaneously addressing efficacy and effectiveness too (EE2). An EE2 trial rigorously demonstrates efficacy, but uses broad inclusion characteristics of effectiveness trials. An example of a study using EE2 design, the IMMEDIATE (Immediate Myocardial Metabolic enhancement During Initial Assessment and Treatment in Emergency Care) trial, is provided.

INTRODUCTION

The joint contributions of researchers, pharmaceutical developers, regulators, clinicians, and research participants have led to groundbreaking therapeutics for many illnesses. The drug development pipeline, which mitigates illness by bringing these medications to the market, is a major social asset. However, there are increasing concerns that the drug development process may not be functioning in the best interests of the widest range of patients.

Among a diverse set of well-documented concerns is that clinical trials take too long and are too expensive (1,2) and that, in many cases, they do not provide a clear understanding of for whom in the general public a treatment will really work — and thus for whom it should be prescribed. Given the expense and delays inherent in conventional trials, innovations in trial design should address the need to improve efficiency, timeliness, and successful completion (3,4). A major reason for the uncertainty of who will be the best beneficiaries of a new drug is that the drugs are typically tested in a far narrower group of patients than will ultimately be treated with the drug (5). It is typical, as reviewed further below, that of the wide range of patients who ultimately receive a treatment, only a fraction would have met the inclusion criteria for the selective trials that led to its approval for marketing. Accordingly, in many cases, more than half of patients are being treated based on heuristics and extrapolation of narrow trials rather than direct evidence of the effectiveness of the treatment on patients with their characteristics.

There is a significant difference between study populations targeted for trials intended for regulatory approval and the wider population that ultimately will use a treatment (6,7). This is because the efficacy and safety of a drug is easier to show in a population carefully selected to be most likely to benefit from the treatment, and less likely to have complications (8). An alternative approach would be to test a new treatment in the broad range of patients and setting that will represent its ultimate use. But in that case, confounding variables may make it more difficult to show efficacy and safety. The distinction between these two approaches has been posed as the difference between “efficacy trials” and “effectiveness trials.” Efficacy trials are used to assess a new drug’s impact in optimally selected patients and conditions. Such trials typically are conducted for short periods, to obtain regulatory approval and subsequent marketing to the public. However, to understand the likely impact of a new treatment when used in a typical patient population, these trials should be supplemented by effectiveness trials — trials that test treatments under usual practice conditions across the spectrum of patients for whom it ultimately will be used, and with follow-up that more closely matches the durations that patients use medications (9).

Unfortunately, such effectiveness trials are infrequently performed, and when they are, they are typically conducted years after the efficacy trial. Thus, opportunities are missed to improve the net health impact of a treatment through better-informed patient selection, and money is spent on ineffective or suboptimal treatments, many of which have scant safety data (10,11).

The current nonlinkage of efficacy and effectiveness trials contributes to a slow and inefficient drug development process, and reduces understanding of a drug’s true clinical value once it reaches the market. What typically occurs is, following identification of a potentially promising new drug, after basic safety and dosing are determined, early clinical trials and efficacy trials are performed. If the results of these trials are positive, approval and publication may follow. In addition, the sponsor may conduct required post-approval (phase IV) studies that often are designed to study drug effects in selected patient populations (e.g., pediatrics) or under specific conditions of use, but not generally formally testing effectiveness. Then, potentially years later, upon reading the literature, those with interest in the treatment and with experience and infrastructure for running effectiveness trials might apply for and might receive funding for a generalizable effectiveness trial. If investigators initiate such an effectiveness trial, they might or might not be able to secure the involvement of the original investigators, and they might or might not think to, or have the opportunity to, collect rich data to allow understanding of the heterogeneity of treatment effects in various subgroups. Moreover, these investigators may or may not have involvement of those who could provide predictive models and other forms of decision support to facilitate translation and dissemination into widespread use.

To address the delays and inefficiencies related to having efficacy trials and effectiveness trials unlinked, we previously, as part of a group with interest in clinical trial design representing academia, industry, and governmental regulatory agencies, proposed a study design approach that integrates both efficacy and effectiveness trials, which we referred to as an “efficacy-to-effectiveness (E2E) trial” (12). Based on the E2E approach, if during the efficacy trial portion a positive effect is seen and ratified (e.g., by a pre-specified analyses and/or the Data Safety Monitoring Board), then the trial seamlessly and immediately transitions to the effectiveness trial portion, even while the US Federal Drug Administration (FDA) approval process is underway and the regulatory decision is pending. Thus, rather than disassembling the trial infrastructure and operations to assess effectiveness in more typical care, the study enrollment criteria are widened, more usual-care study sites are added, and plans are made for longer treatment and follow-up. Thereby, the effectiveness trial could proceed without delay, potentially being complete near the time of FDA approval.

Besides providing a broader understanding of the treatment by enrolling more diverse study participants, this approach would allow better understanding of the heterogeneity of treatment effects for special patient groups. During the effectiveness trial component, plans for enhanced use and dissemination of the trial results could be devised, such as multivariable predictive models to support the treatment’s optimal use in those patients most likely to benefit. The net result should be a better translation of research results into an impact on the public (13).

Both efficacy and effectiveness trials provide important contributions to the development and optimal use of a treatment, and both have limitations. As a practical matter, in a healthcare ecosystem in which regulatory approval of new medications for marketing is critical, optimizing the success of efficacy trials is key. And, once accomplished, the cost of running an effectiveness trial, with its attendant risk of undermining the established findings, makes the concept unattractive to manufacturers. Thus, effectiveness trials are infrequently performed, especially with support by the manufacturer.

There are countervailing forces to this situation, however. Without effectiveness trial evidence, patients and clinicians are left uncertain about whether the efficacy results apply to care in the broader ranges of illness severity, comorbidities, age, and durations of treatment beyond those represented in the efficacy trials. Also, payers are left uncertain as to whether treatments are truly worth supporting when they are being used on individuals for whom the treatment was never directly tested. Although the patients have the most need for resolving this information gap, finances are also involved. Post-marketing information that would limit the use of a drug would presumably limit sales, which would be adverse to the manufacturer, but potentially beneficial to payers and patients if this limited costs for potentially ineffective treatments.

Resolving this information gap and the different interests in ways that benefit all stakeholders is not straightforward. There is a general understanding of the need to integrate the needed approaches — including most importantly among the public and patients — and in recent years, there has been a flurry of attempts to improve various aspects of the pharmaceutical ecosystem (14). The E2E trial design is one such approach. However, although seen as a significant improvement, even at its best, the E2E trial approach requires the time, logistics, and financing of two clinical trials. Could this be improved upon without losing the important objectives? There are important advantages of the seamless transition in an E2E trial, but could the need to have two trials be eliminated without a loss in evidence generation? Could we continue to accrue the benefits of broader study enrollment, but reduce the time and expense for generating efficacy and effectiveness evidence by half? We think so. We believe that a possible avenue for this would be simultaneous efficacy and effectiveness trials: “efficacy and effectiveness too (EE2)” trials. These would be trials that provide both the clinical evidence of an efficacy trial, necessary to meet regulatory requirements for approval, as well as the more generalized and similarly valuable information of an effectiveness trial.

The drug development ecosystem is beginning to acknowledge that there are evidence gaps in the traditional drug development pathway. The increasing interest in seamless and adaptive trials by the pharmaceutical industry, FDA, European Medicines Agency, and other stakeholders (15,16) shows an interest in improving the efficiency of drug development. However, these types of trials do not always prioritize effectiveness and generalizability. Seamless trials are most often used in oncology drug development, and combine separate trials, delineated by interim looks at the data. Seamless trials also typically add cohorts to a first-in-human trial to investigate dose-response in a variety of cancers (17). As outlined below, whereas there are similarities between EE2 trials and adaptive trials, there are important distinctions.

Similarities Between Adaptive and EE2 Trials

• Both have prospectively defined hypotheses, eligibility criteria, and sample size for the inclusion of new cohorts.

• Both should use an independent DSMB to provide oversight of safety and efficacy data from existing cohorts, and ensure the statistical integrity of the trial.

• Both maintain a randomized, controlled trial (RCT) design approach throughout the trial to generate robust data.

Differences Between Adaptive and EE2 Trials

• Besides efficacy, EE2 trials focus on effectiveness and generalizability to the spectrum of patients for which the treatment ultimately is likely to be used in practice.

• EE2 trials begin with broad inclusion criteria and maintain them throughout to identify heterogeneity of treatment effect. In contrast, adaptive trials may expand or narrow the inclusion criteria based on interim analysis.

• EE2 trials include longer follow-up.

• EE2 trials address patient heterogeneity of treatment by emphasizing real-world implementation as part of the trial.

Another type of trial that addresses generalizability is the “pragmatic” RCT, which is designed to supplement RCT efficacy (“explanatory”) trial data by generating real-world effectiveness data. The Salford Lung Study, which compared the real-world effectiveness of a novel inhaler compared to existing therapy for chronic obstructive pulmonary disease and asthma, was one of the first pragmatic RCTs. This phase 3 trial randomized 7,000 participants and followed-up according to the investigating physician’s usual practice, using electronic health records (18). One of the limitations of typical pragmatic trials is that study design and data collection are not sufficiently controlled to address important efficacy questions, thus making interpretation of the trial results difficult and applicability to regulatory approval uncertain (19,20). The EE2 design allows for generation of statistically and clinically meaningful efficacy and effectiveness data.

AN EXAMPLE OF AN EE2 TRIAL: THE IMMEDIATE TRIAL

To illustrate the principles of EE2 trials, we review the IMMEDIATE trial of intravenous glucose-insulin-potassium (GIK) for acute coronary syndrome (ACS) (21,22). This trial was done under an FDA investigation new drug study to show efficacy, but using an effectiveness framework. Thus, it needed to have a study design that properly tested the pathophysiological model intended to be addressed by GIK — and also show this in the wide span of patients and settings in which the treatment is ultimately intended.

There is a long history of experimental studies showing that GIK myocardial metabolic support (23) started immediately in cardiac ischemia followed by reperfusion does the following:

• Improves glucose, glycogen, and energy metabolism, and maintains cellular adenosine triphosphate levels

• Supports cardiac function and delays necrosis

• Decreases plasma and cellular free fatty acid (FFA) levels (FFAs damage membranes, cause arrhythmias, and waste oxygen)

• Preserves myocyte potassium (anti-arrhythmic)

Based on these effects, GIK resulted in:

• Prevention of arrhythmias and cardiac arrest that occur very early in ACS/acute myocardial infarction (AMI), associated with elevated FFAs

• Protection of the myocardium from ischemia, limiting progression to AMI, or at least the ultimate infarct size

• Based on both of these effects, reduced mortality

Although these experimental studies have shown the greatest benefit of GIK is when started very early in ischemia, before IMMEDIATE, trials gave GIK at the hospital only after documentation of AMI or ST elevation myocardial infarction (STEMI), typically on the order of 6 hours after ischemia onset. This was even though most cardiac arrests occur in the first hour of ACS, one of the targets of GIK therapy, and that reperfusion interventions are highly dependent on their being very soon after ischemic symptom onset (aiming for the very first hours). Indeed, prior trials typically started GIK after reperfusion was initiated, even though a major posited mechanism for impact of GIK myocardial metabolic support is to extend the time-window for cardiac salvage until reperfusion. Additionally, the prior GIK trials for AMI/STEMI were not placebo controlled.

To show efficacy, the IMMEDIATE trial investigators believed that replicating the experimental approach of very early treatment would be critical. A way to do this could be by having paramedics give GIK to patients with presumed ACS immediately after they call 9-1-1 with ischemic symptoms, in the community. Accordingly, the IMMEDIATE trial was designed to have much earlier administration of GIK than prior studies, and to be a classic double-blinded placebo-controlled study to show efficacy (Figure 1).

Fig. 1.

Administration of GIK in IMMEDIATE and other trials. Abbreviations: GIK, glucose-insulin-potassium; EMS, emergency medical service; CREATE-ECLA, The Clinical Trial of Metabolic Modulation in Acute Myocardial Infarction Treatment Evaluation-Estudios Cardiologicos Latinoamerica; DIGAMI, Diabetic Patients with Acute Myocardial Infarction.

In implementing the trial in emergency medical services (EMS) settings to test GIK in a way that would be applicable to the wide group of patients and setting, EMS systems were selected to include different organizational structures in a wide variety of settings, which included a wide variety of patients and ethnic distribution — a clinical effectiveness trial approach. The study sites are listed in Table 1.

TABLE 1.

IMMEDIATE Trial Site Selection

Site	EMS Type	Population
Albuquerque, NM	Private, Fire	1,166,096
Anchorage, AK	Fire	258,455
Bellingham, WA	Fire	166,814
Brockton, MA	Private	110,000
Concord, MA	Hospital	63,306
Dallas, TX	Fire	1,200,000
El Paso, TX	Fire	676,365
Macon, GA	Hospital	293,447
Milwaukee, WI	Fire	604,477
New Haven, CT	Private, Fire	408,288
Sioux Falls, SD	Private	158,424

In implementing the trial in usual care settings, it must address in a widely applicable way the fact that of every 100 patients with chest pain or other symptoms suggestive of ACS who call 9-1-1 for EMS care, or who arrive directly at emergency departments (EDs) without calling 9-1-1 with such complaints, approximately 75% will not have ACS (24). Not having the target condition, they would not benefit from GIK. This means that the testing of GIK, and its ultimate use in practice, would be hampered by not being able to accurately identify those with ACS who could benefit from immediate treatment. To address this, the investigators used a previously developed electrocardiograph (ECG)–based cardiac predictive instrument, the acute cardiac ischemia time-insensitive predictive instrument (ACI-TIPI) (25) and thrombolytic predictive instrument (TPI) (26) that provide decision support for identifying ACS and STEMI in EDs, and they were tested for use in EMS settings (27). They assisted EMS paramedics in accurately identifying the right patients (Figure 2). Thereby, it was possible to conduct an efficacy trial of the effect of GIK used in very early care of patients with suspected ACS in an effectiveness trial context to represent the ultimate target patients and settings.

Fig. 2.

Ambulance ECG with ACI-TIPI and TPI decision support. Abbreviations: ECG, electrocardiogram; ACI-TIPI, acute cardiac ischemia time-sensitive predictive instrument; TPI, thrombolytic predictive instrument.

Accordingly, the trial’s inclusion criteria were as intended for use once GIK was widely available:

• Age 30 years or older and seen by EMS for symptoms consistent with ACS

• Paramedic judgment that clinical picture suggests ACS/AMI and pre-hospital 12-lead ECG has at least one of the following:

• ACI-TIPI predicted probability of ACS of 75% or more

• TPI or equivalent ECG detection of STEMI

• STEMI identified based on local EMS practice protocols

Generally, clinical trial patient enrollment represents a significant barrier to study initiation and contributes to the growing costs and lengthening durations of clinical trials. Of particular importance to the objective of the effectiveness approach is that a trial enroll an unbiased sample of those envisioned as ultimately being candidates for a treatment. According to a 2001 FDA review (28), the inadequate representation of certain populations is shown by the fact that less than 5% of adults with cancer participate in clinical trials (29), and that minority groups collectively represented less than 10% of participants in clinical trials for cancer drugs between 1995 and 1999. Similarly, whereas nearly two-thirds of cancer patients are age 65 years old or older, this age group accounts for less than one-third of clinical trial enrollees (30). Narrowing the range of eligible patients increases the risk of enrollment bias. The most obvious and uncomplicated cases are the ones that are most likely to be enrolled, which may be a biased sample. This means that the ultimate study sample may be nonrepresentative of those for whom the treatment is intended, which undermines the intent of an effectiveness trial.

Undermining efforts to counter this risk is that, in most trials, it is difficult to know the proportion of eligible patients who are ultimately enrolled; the denominator for calculating the enrollment rate is hard to identify. To address this in the IMMEDIATE trial, because the point-of-entry for inclusion was the performance of an ECG in the EMS setting, the ECGs could be used as a screen to determine the proportion of all eligible patients who were enrolled. The denominator of all likely eligible patients would be those who had an ECG performed for the evaluation of ACS and who in fact had ECG ischemic abnormalities that met the criteria for enrollment listed above. The ECG scans stored in the computerized ECG could be checked for all patients who met the enrollment criteria (i.e., those presenting with symptoms consistent with ACS with qualifying ECG changes), which then defined the denominator of eligible patients. The way this was implemented is shown in the CONSORT (Consolidated Standards of Reporting Trials) enrollment diagram in Figure 3, and the system for assuring capture of all candidate ECGs is depicted in Figure 4. Using this approach, ultimately 52% of all eligible patients were enrolled using this approach, on the order of 10-fold that typical of efficacy trials, and in a very challenging EMS setting.

Fig. 3.

CONSORT enrollment diagram. Adapted from Selker et al (21). Abbreviations: ECG, electrocardiogram; EMS, emergency medical service; GIK, glucose-insulin-potassium; ITT, intent-to-treat.

Fig. 4.

IMMEDIATE Trial ECG-based enrollment monitoring. Abbreviations: EMS, emergency medical service; ECG, electrocardiogram; ACI-TIPI, acute cardiac ischemia time-sensitive predictive instrument; TPI, thrombolytic predictive instrument; STEMI, ST elevation myocardial infarction.

The results of the IMMEDIATE trial are shown in Tables 2 through 5 and in Figures 5 and 6. These show important signals of efficacy, and because the trial was also an effectiveness trial, the results should be generalizable to the real-world treatment of ACS. Thus, the IMMEDIATE trial represents the EE2 approach in practice. As this was the first placebo-controlled trial of GIK registered at FDA, the FDA is asking for a second trial, and they have again approved the same EE2 approach. If the second trial is positive, approval of GIK for wide-use for ACS is expected.

TABLE 2.

IMMEDIATE Trial Participants

	GIK (n=411)	Placebo (n=460)
Age (mean, yrs)	64	63
Men	73%	70%
White/Black/Hispanic	82/13/11%	87/9/13%
Chest pain chief complaint	87%	85%
Shortness of breath chief complaint	4%	4%
Pre-hospital HR (mean, BPM)	87	87
History of DM	29%	26%
History of HF	17%	17%
History of MI	37%	35%

Reprinted from Selker et al (21). Abbreviations: GIK, glucose-insulin-potassium; HR, hazard ratio; BPM, beats per minute; DM, diabetes mellitus; HF, heart failure; MI, myocardial infarction.

TABLE 5.

IMMEDIATE Trial Results: Patients With ST Elevation and ACS; 30-Day Intent-to-Treat Endpoints

Reprinted from Selker et al (21). Abbreviations: ACS, acute coronary syndrome; ITT, intent to treat; GIK, glucose-insulin-potassium; CI, confidence interval; MI, myocardial infarction; HF, heart failure.

Fig. 5.

IMMEDIATE trial: biological mechanism cohort. Abbreviations: GIK, glucose-insulin-potassium; FFA, free fatty acid; IQR, interquartile ratio.

Fig. 6.

IMMEDIATE trial metabolic effects of GIK in EMS. Adapted from Selker et al (41). Abbreviations: GIK, glucose-insulin-potassium; EMS, emergency medical service; FFA, free fatty acid.

TABLE 3.

IMMEDIATE Trial Results: GIK Versus Placebo

Reprinted from Selker et al (21). Abbreviations: EMS, emergency medical service; GIK, glucose-insulin-potassium; IQR, interquartile ratio; PCI, percutaneous coronary intervention.

TABLE 4.

IMMEDIATE Trial Results: Patients With Suspected ACS; 30-day Intent-to-Treat Endpoints

Reprinted from Selker et al (21). Abbreviations: ITT, intent to treat; ACS, acute coronary syndrome; GIK, glucose-insulin-potassium; MI, myocardial infarction; HF, heart failure.

DISCUSSION

The most important objective of the EE2 approach is to generate rigorous evidence of drug efficacy that is generalizable to the widest span of patients and settings. Currently there is a large gap between the patients evaluated in efficacy trials and the target patient population that will ultimately receive the treatment. As a result, treatment of the wider population proceeds with little or no data on the drug’s effect on these patients. This should be of concern to all members of the healthcare community, including patients, prescribers, payers, regulators, and developers.

This gap in knowledge about treatment effects in broad patient populations appears to be widespread in medicine, and it exists with the most commonly prescribed drugs. A recent article examining the external validity of RCTs in the fields of cardiology, mental health, and oncology found that more than 70% of patients included in these three types of RCTs were not representative of patients encountered in routine clinical practice and/or that population difference may have a relevant impact on external validity of RCT findings (5). Compared with patients enrolled in major cardiology RCTs, patients encountered in routine clinical practice were more likely to have high-risk characteristics, as they were older, more likely to be female, and more often had clinical impairment and comorbidities. Masoudi et al (31), assessing the generalizability of RCT results of common prescription drugs, applied the inclusion and exclusion criteria of three major chronic heart failure trials to 20,388 representative patients in the United States from the Medicare database. Only 18%, 13%, and 25%, respectively, of “real-world” patients met the enrollment criteria of the Studies of Left Ventricular Dysfunction (SOLVD), Metoprolol CR/XL Randomised Intervention Trial in Congestive Heart Failure (MERIT-HF), and Randomized Aldactone Evaluation Study (RALES) trials, respectively. (The SOLVD trial tested the efficacy of enalapril, MERIT-HF tested metoprolol, and RALES tested spironolactone.) Martin et al compared the characteristics of real-world patients in France treated with simvastatin (for hyperlipidemia), tacrine (for Alzheimer’s), and celecoxib (for arthritis) to the characteristics and eligibility criteria of participants in clinical trials used for regulatory approval of these drugs (32). Of those patients being assessed who were being administered these treatments in real-life practices, 37% to 43% would have been excluded by the trials’ protocols, and the proportions of women receiving these medications in real-life significantly over-represented the study samples. Iyngkaran (33) reviewed 28 RCTs of six beta blockers for ischemic and nonischemic cardiomyopathies and found gaps in recruitment related to ethnic diversity, comorbid conditions of diabetes, renal impairment, and generally sicker patients.

Emphasizing the importance of addressing this gap is the fact that the results of narrow efficacy trials can become instantiated in clinical practice guidelines. Among hypertension trials, Uijen et al (34) found substantial differences in characteristics between patients treated for hypertension in general practices and participants of trials on which recommendations for their treatment were based. When the inclusion and exclusion criteria of each trial were applied to patients from the general practices database (N = 5,494), less than half would have been eligible for inclusion in the specific trial. Patients with dementia, depression, and/or substance abuse were excluded in 11 trials, and patients with a recently diagnosed cardiovascular disease were excluded in 19 trials. As hypertension medications make up 3 of the top 10 most commonly prescribed medications in the United States, these results show a need for better methods of evidence generation to strengthen definitive individual treatment decisions in general practice. This is not limited to US trials; a Scottish study looked at the proportion of people with type 2 diabetes living in Scotland who met eligibility criteria for inclusion in seven major RCTs of glycemic control that had informed guidelines (35). A maximum of 51% of people with newly diagnosed type 2 diabetes in the Scottish population were eligible for inclusion in the UKPDS33 trial for metformin, the seventh most prescribed drug in the United States. Reasons for exclusion were patient age (trial participants had an average age of 53 years compared to the general patient population of 66 years) and renal impairment (patients were excluded for elevated plasma creatinine levels). For another very common condition, a study of the proportion of people with asthma who would be eligible for the major randomized controlled trials on which the Global Initiative for Asthma guidelines were based found that only 4% of a cohort of 749 patients were eligible for one or more trials (36).

These studies strongly suggest that an effectiveness trial approach should be embedded in the drug development process. The EE2 approach would further advance the objectives of the E2E model, described above, by requiring that the link of efficacy to ultimate practice be addressed at the very onset of drug testing, and by generating both efficacy and effectiveness data in half the time of, and presumably at much less expense than, the E2E sequence. We think this could have important benefits for all members of the ecosystem for the testing and use of new drugs.

In illustrating the EE2 approach, the IMMEDIATE trial has a feature that facilitates and shows the intent of this approach to include all qualified study candidates: the use of the ECG, which is central to the evaluation to all potential trial candidates, as real-time decision support for clinicians to properly enroll candidate patients. Additionally, because of the importance of being able to identify the denominator for assessing the rate of enrollment, by having the ECG data available electronically, the device also was used for monitoring the completeness of enrollment at a site. More generally, for EE2 trials, we believe that this approach also could be applied to the use of diagnostic devices in trials of other medical problems, such as the use of computed tomographic scanners for patients with acute neurological problems, oximetry for those with respiratory problems, and others. We also believe this approach also could be embedded in electronic health records, including the use of predictive instrument decision support for shared decision-making for patients and clinicians, based on clinical trial enrollment based on “mathematical equipoise” (37).

An additional important advantage of the EE2 design is that it has the potential to provide critical insights regarding the heterogeneity of treatment effect. Although there will be the impulse to avoid heterogeneity in a study sample and settings, as this could be expected to attenuate the observed average treatment effect, heterogeneity can offer opportunities to detect how patient characteristics influence treatment effects. Moreover, insights might go in both directions. For example, in an overall negative trial, heterogeneity might help detect individuals or groups who benefited from treatment, and in an overall positive trial, those for whom benefits did not outweigh risks (38,39). As we outlined for E2E trials (12), it would not be warranted to compromise regulatory approval that would be based on efficacy data by the diluted signal by broader inclusion of those not benefiting, but insights from the broader sample could inform care and reimbursement decisions. Moreover, the heterogeneity of treatment effects reflected in trial data could be used to generate multivariable predictive models, to aid individualized treatment decisions in patient care.

In implementing the EE2 design, it will be important to be mindful of the key characteristics of these trials, which would include, but not be limited to, the following:

• The study design and endpoints must be sufficient to show efficacy, of which FDA approval of the protocol would be one indicator.

• To the extent possible, the use of the treatment should be in patients and settings for which it ultimately would be intended, including those often kept out of efficacy trials because of age, comorbidities, and diversity, and should be for a length of treatment as desired for ultimate use.

• Sufficient data should be included to allow analysis of heterogeneity of treatment effects.

• Enrollment should be done in ways that are compatible with usual clinical care, and with mechanisms for assessing the denominator for reporting enrollment rates of eligible patients. This may be facilitated by device-based enrollment and monitoring approaches.

In summary, by combining the functions of efficacy trials, which rigorously assess treatment effects in a specified group of patients to meet regulatory requirements for approval, and of effectiveness trials, which test treatments across the spectrum of patients likely to be treated in real-world conditions, the EE2 trial design has important advantages. EE2 builds upon the advantages of our earlier E2E trial design, but with the additional and important advantage of requiring less time and fewer resources, as shown in the IMMEDIATE trial. EE2 trials, by having broader inclusion criteria, may also prove to achieve greater and more efficient enrollment. This would address one of the biggest challenges facing clinical researchers and sponsors, with 11% of sites in a trial not recruiting a single participant, and 37% of sites under-enrolling (40). Because of these potential benefits, we believe that the EE2 approach deserves further consideration and refinements.

DISCUSSION

Abel, Iowa City: That’s a very nice talk. So, you’re very much aware of the controversies surrounding GIK trials and the diverse outcomes in other trials. So, how is your trial fundamentally different from the other ones, and how do you reconcile that with the debates in the field?

Selker, Boston: What you are alluding to is that large trials, such as CREAT-ECLA, with as many as 20,000 people, did not show a clear effect from GIK. I remember when I first presented these results of the IMMEDIATE trial, the question was: did the other trials get it wrong? And I hate to say it, probably so. We know that in experimental studies if you clamp off the coronary artery of any animal and you don’t give GIK for 6 hours, the animal has a large myocardial infarction and may well die. As you well know, cardiac arrest happens largely within the first hour of cardiac ischemia, and most death is within the first couple of hours — especially the first hour. So if, as in the prior GIK trials, you treat people at 6 hours after the onset of infarction, when you’ve documented infarction using troponin or other biomarkers at the hospital, it’s completely sensible that there would be no effect. That is the biggest difference. The other thing is, in CREATE-ECLA, which was done internationally, in many cases GIK was not started before coronary reperfusion, so GIK had no chance to provide myocardial support prior to reperfusion. Moreover, the reperfusion used in India, China, and other places, primarily streptokinase, was probably of inconsistent effectiveness, as potency had not been very well regulated there, so we don’t even know that reperfusion was accomplished.

What we tried to do is to translate the animal laboratory research into the real world. That means that once a patient has ischemic symptoms, you need to treat right away, in the first hours. And that’s why we did it in field following 9-1-1 calls, even before the diagnosis was definitely made. That’s the fundamental difference between our trial and the previous GIK trials. Also, our trial was placebo controlled, whereas none of the other GIK trials were, and so the level of evidence is different. GIK is unfortunately not patentable, so there’s not a lot of interest in running expensive placebo controlled trials by industry, so we had to go to NIH for support. Prior GIK trials were basically just add-on trials to other drug trials, and so they were not really specifically designed to test GIK.

Abel, Iowa City: Thank you. As someone who’s studied ischemic metabolism, I think it does make sense to increase glucose in the heart, so it’s actually nice to actually see a trial done in a way that supports decades of basic science.

Selker, Boston: Thank you!

Mushlin, New York: Harry, that was a great talk and you know I think the advantages of incorporating effectiveness trials within the context of proven efficacy has great appeal, for a lot of reasons that you elucidated. This can really be something that should take off. But you didn’t discuss very much some of the tradeoffs when one thinks about going from efficacy to effectiveness trials, such as sample size. The sample sizes required, as you know for effectiveness trials, can be significantly higher and the duration of the trial may need to be quite a bit longer. And maybe you could give us a little bit of an idea of the calculus that goes into deciding whether the design of the trial would lean toward the efficacy side or the effectiveness side, and how to rationalize a major movement toward increasing sample sizes for randomized control trials and lengthening the period of time.

One could also add to that the ethical dilemma that people would argue that after efficacy has been proven, the ethics of continuing further randomization could be called into question. I know you’ve thought a lot about these issues and maybe you can extend some of your comments to the things that I brought up.

Selker, Boston: A great question from an excellent clinical epidemiologist! You’ve hit on two really important things. I’ll address the second question first; the question of ethics arises when you transition from the first efficacy component of an E2E (efficacy-to-effectiveness) trial to the second effectiveness component. If, at the finish of the efficacy trial portion, you find a mortality benefit, in most cases it would not be ethical to continue with a randomized controlled trial for the effectiveness trial component. So, I think ideal candidates for these trials will not be ones where you’re looking for a mortality benefit, unless there is intent to broaden and look for special heterogeneous effects. Otherwise, I think mortality is not an ideal endpoint for E2E trials. And that’s one of the reasons we went to the EE2 design instead of E2E — because basically, if you’re looking for the mortality effect, as we did the example I presented, then you can avoid the ethical problem of doing the effectiveness component after completion of the efficacy trial in which there was a demonstrated mortality benefit, as by then, the disruption of equipoise would preclude randomization.

On the first question, about the signal-to-noise ratio in an effectiveness trial: while you’re doing the primary trial, there’s a tension between pharmaceutical companies, payors, and the public. It turns out that the dominant stakeholders, the pharmaceutical manufacturers, want to have efficacy tested, not effectiveness, because with the broad inclusion criteria for an effectiveness trial has a less favorable signal-to-noise ratio. On the other hand, payors would like to know if they’re paying for effective treatments in real-world patients. So, there’s a balance of interests there. And regulatory agencies have actually gotten quite supportive of addressing this. They have been telling pharmaceutical companies: you’re going to have to cope with a little bit of real-world effectiveness if you’re going to get approval. So, we think that there’s a dynamic that has to be played out, and just because it’s an effectiveness trial, you can’t be sloppy. I should add, and I want to be clear — these are not what you would call “pragmatic” trials. I think pragmatic trials are, in many cases, are not practical as they may leave key questions unanswered. But these are the kinds of balances that we have to explore.

Wolf, Boston: I was surprised you didn’t present data of the efficacy of people treated in the first 4 to 6 hours versus the efficacy of people who were treated after 6 hours.

Selker, Boston: You’re surprised I didn’t show those data? Is that what you’re asking?

Wolf, Boston: Yes.

Selker, Boston: Oh, of course.....

Wolf, Boston: I mean, one would have thought if your hypothesis is correct, you would have efficacy and the people treated early, and relatively less efficacy, if any, in the people treated late?

Selker, Boston: You clearly looked at the forest plots in the appendix of our JAMA report of the trial, where we parsed it by 1 hour to 3 hours to 6 hours. And you’re absolutely right that the earlier you treat, the better the outcomes. However, those details were beyond the limit for this presentation. Also, the trial was not powered to formally address those intervals as primary endpoints — but yes, that effect was present. Again, it reinforces the primacy of timing as why the first GIK trials didn’t work, because they were treating patients too late.