Challenges in the Design and Analysis of Non-Inferiority Trials: A Case Study

Abstract

Background:

The literature on the design, conduct and analysis of non-inferiority (NI) trials is continuously evolving. Several design issues continue to be researched including the choice of active control, choice of NI margin and optimal analytic approaches. To date, there has been relatively little in the literature documenting actual experiences with implementing available methodology for NI trials.

Purpose:

This article serves as a case study and highlights some of the challenges encountered in the design of a Phase III NI trial in status epilepticus that is being conducted under an FDA Investigational New Drug (IND) application.

Methods:

The IND was put on clinical hold by the FDA due to concerns with the design. Specifically, support for the active control, NI margin and overall interpretability of trial results were questioned, and a recommendation was made to consider a superiority design. The authors highlight their interactions with the FDA and their application of available methods and approaches to address these concerns.

Results:

The investigators’ response to the clinical hold provided detailed information to support the conduct of an NI trial. The study team received FDA approval to initiate the trial in October 2008. The trial enrollment began in June 2009 and is being conducted by roughly 800 paramedic units in over 40 Emergency Medicine Service systems across the United States.

Limitations:

There is still a great deal of methodology research needed to fully understand the application and impact of the NI trial design.

Conclusions:

It is evident that NI trials have an important place in the lexicon of clinical trial design and analysis. These trials may be the only way and only opportunity to answer certain questions, so they must be designed and conducted with rigor. This case study is an attempt to share our experiences in implementation.

BACKGROUND

There has been a great deal of research in the design and analysis of non-inferiority (NI) trials. Audiences have been introduced to the concepts and terminology of this study design [1–5] as well as to the more detailed issues of conduct and analysis [6–13]. Regulatory agencies have provided guidance [14–16] and the Food and Drug Administration (FDA) recently issued a draft guidance to highlight their current recommendations on acceptable approaches to the design and analysis of NI trials [17]. Although there is not an established ‘optimal’ approach or design, all parties in the clinical trial community agree that a rigorous NI trial requires three key elements: appropriate active control, appropriate NI margin and overall assay sensitivity. Nonetheless, several important challenges in the approach to NI trials still exist in addition to these key elements, including utility of early stopping guidelines [18,19], appropriate analysis population [20,21], and optimal analytical approach [12,22,23].

To date, there has been relatively little in the literature documenting actual experiences with NI trial design, particularly in the context of life-threatening diseases. Our currently ongoing NIH-funded NI trial conducted under an Investigational New Drug (IND) application provides the opportunity to highlight the application of available methods and approaches to handle some of the known challenges, and therefore serves as a case study in NI trial design and analysis. The following is an attempt to share our experiences in implementation of a Phase III non-inferiority trial and highlights the authors’ interactions with the FDA.

PURPOSE

Status epilepticus (SE) is a true neurologic emergency associated with substantial morbidity and mortality. It has been estimated that there are between 120,000 and 200,000 cases of SE in this country each year resulting in as many as 55,000 deaths. SE is traditionally defined as seizure activity persisting for greater than 30 minutes or lack of return to baseline function between seizures within a 30 minute interval, but more recently it has been suggested that duration of even 5 minutes or more is deleterious and should be considered indicative of SE [24]. This is a life-threatening emergency which requires immediate intervention to prevent serious sequelae. Benzodiazepines are an effective first line pharmacotherapy for SE and studies have been conducted to assess differences between various benzodiazepines as well as doses [25,26], but the optimal agent and route of administration varies between the pre-hospital setting and the Emergency Department (ED) setting. The preferred treatment for SE in the ED is intravenous (IV) lorazepam which is labeled by the FDA at a dose of 4mg for treatment of SE in adults based on data that shows superior efficacy to active comparators including lower doses of lorazepam and diazepam (a benzodiazepine commonly used in the pre-hospital setting). IV lorazepam also has been shown to be the most effective at stopping seizures in the pre-hospital setting [27,28]. However, lorazepam requires refrigeration or restocking after every 60 days, which makes it impractical and prohibitive for use in a pre-hospital setting in most Emergency Medical Services (EMS) systems. In addition, because convulsions can be violent, IV administration in a patient with seizures in the field is often technically difficult. Placement of an IV catheter may not be possible or may take additional time that delays treatment. A practical alternative mode of administration is intramuscular (IM), which allows for immediate injection upon EMS arrival at the scene of a patient with ongoing seizure activity. Lorazepam, however, is relatively poorly absorbed when given intramuscularly or across mucus membranes as compared to more lipophilic benzodiazepines [29,30]. Increasingly, many EMS systems have begun to use an alternative benzodiazepine, midazolam, for the treatment of SE because it does not require refrigeration, is more economical, and can be given IM. Midazolam is a highly lipophilic agent that is rapidly absorbed intramuscularly with subsequent very rapid distribution in the central nervous system. In observational studies, midazolam appears to be highly effective at terminating seizures when given as an initial agent for SE and a recent meta-analysis of six clinical trials comparing non-IV midazolam to non-IV and IV diazepam concluded that IM midazolam is safe and effective at stopping seizures in SE patients [31]. However, the safety and efficacy of IM midazolam has never been studied in a randomized controlled manner for the termination of seizures in children or adults in the pre-hospital setting [32].

METHODS

RAMPART: An Overview

The Rapid Anticonvulsant Medication Prior to Arrival Trial (RAMPART) conducted by the Neurological Emergencies Treatment Trial Network (NETT) is designed to determine if the efficacy of IM midazolam is not inferior, i.e., no less than some clinically acceptable amount, to the efficacy of IV lorazepam. RAMPART is a double-blind, two-armed double-dummy NI trial. In summary, adults and children with estimated weights of 13kg or greater are eligible; if they are seizing on paramedic arrival, or if unresponsive on paramedic arrival and have a qualifying generalized seizure without regaining consciousness; and, if they are being transported to a participating hospital. Eligible patients are randomized to receive either active IV and placebo IM, or placebo IV and active IM. All subjects enrolled are treated with the IM autoinjector first, followed by IV. Clinical efficacy is defined as the proportion of subjects with seizure termination at arrival to the ED after a single dose of study medication and without use of rescue medication (provided at 10 minutes for continued seizure activity). A conclusion of non-inferiority is based on a one-sided hypothesis test using Dunnett and Gent’s method at a significance level of 0.025 [6]. Several secondary analyses are planned, and will be conducted in a superiority framework. These include comparison of the two study treatments in: the time interval between paramedic arrival to the termination of clinically evident seizure; the time interval between initiation of treatment and the termination of clinically evident seizure; the frequency of endotracheal intubation; and, the frequency and duration of hospitalization and of ICU admission.

The study is powered to assure greater than 80% likelihood of identifying less than a 10% difference in success rates between the two treatment groups at a one-sided type I error rate of 0.025. Sample size estimation is based on the comparison of independent proportions with a 1:1 randomization scheme and two planned interim analyses for futility. The maximum sample size required for randomization is 696 subjects (348 per treatment group). A small subset of patients with poorly controlled seizure disorders are frequently treated by EMS for SE, and are anticipated to be at risk for multiple enrollments in this trial. The total sample size is inflated by 15% to account for multiple enrollments of a study subject. Subjects will be independent for the primary analysis.

The trial is being conducted under the FDA’s investigational new drug application (IND) and due to the acute nature of the disease it also is conducted under the Exception From Informed Consent for Emergency Research provision [33,34]. After initial submission of the IND, the FDA placed the study on clinical hold primarily due to the following issues: effectiveness of the proposed active control, choice of NI margin and the overall interpretability of the trial results. The FDA suggested a detailed discussion of all relevant previous trials to support the active control’s minimal treatment effect and a detailed justification for the NI margin. The agency also suggested the investigators consider designing a superiority trial on a different outcome such as time to seizure cessation, but said that they would consider ‘an evidence-based reason’ as to why an NI trial was required. The investigators responded to the FDA’s concerns as described in detail below and were allowed to proceed with the proposed clinical trial.

Results

Rationale for Choosing a Non-inferiority Design over a Superiority Design

There are inherent weaknesses to the NI design. The assumed effect of the active control, choice of NI margin and inclusion of a placebo arm are key components of the design that when incorrectly chosen will result in reduced trial validity. A superiority design does not share the same vulnerability and is considered to have readily interpretable results since the intent is to show that the experimental treatment is better than the control treatment (i.e., placebo, standard of care or active control). Therefore careful consideration was given to the FDA’s suggestion of making “time to seizure cessation” the primary outcome for a superiority trial design. Time was originally proposed as a secondary outcome with a superiority analysis plan. The measure provides information that may help in understanding clinical performance of the study treatments. However the primary known driver of clinical outcome (and thus the clinical question of greatest interest) is whether initial therapy stops the seizure. Time to seizure cessation in the pre-hospital setting may or may not impact clinical outcomes at all, and is of far less clinical interest. Although there are qualitative data to demonstrate that more rapid termination of seizures in SE in general is likely to be clinically superior to more prolonged seizures, there are no data to suggest exactly how much time to termination represents a clinically relevant difference within the 10 minute window being studied.

The investigators also considered a superiority trial of IM in terms of the chosen primary endpoint (seizure termination). The FDA guidance suggests alternative study designs in cases where the validity of an NI design is questionable, including add-on studies, subpopulation (e.g., medication resistant populations) studies and rescue treatment design. All of these designs suggest a superiority comparison to placebo which is an important piece of evidence of the efficacy of the new drug. However in the case of RAMPART, superiority of IM administration of benzodiazepines to placebo is not of clinical interest since it has been proven that benzodiazepines are superior to placebo/no treatment in stopping seizures. Superiority of IM midazolam to the most effective IV benzodiazepine is also not of primary interest since it has been consistently shown that IV lorazepam is highly effective (over other benzodiazepines) at stopping seizures. The overall goal of RAMPART is to provide results that can be used by EMS medical directors to determine the treatment protocol for the pre-hospital setting. In the current clinical environment, the EMS medical directors are faced with a problem in planning the pre-hospital care of SE. The preferred treatment of seizures provided to patients in the ED is not being provided to patients by paramedics in the field due to a potential difficulty in administration as well as storage as described above. EMS directors are not looking for a safer or more effective treatment than what currently is available in EDs; what they want to know is whether a practical alternative is as safe and effective. The pragmatic benefits of IM midazolam (ease of use, stability without refrigeration, and economy) are all well established, so a superiority trial that fails to reject the null hypothesis would not inform decision making. Indeed, the benefits of IM with the potential for non-inferiority are already driving many EMS directors to choose this off-label use of IM midazolam as an alternative to IV lorazepam based on preliminary evidence that this agent and route of administration can be just as effective as IV even though this has not been proven in a controlled setting. The clinically important question is whether IM works well enough to forego IV in the pre-hospital setting in order to improve ease of administration. RAMPART is designed to readily provide an answer to the clinical question posed by these decision makers in the current environment: if the proposed practical IM treatment works as well in terms of termination of seizure and as safely as the FDA-approved, preferred ED treatment (IV lorazepam), then EMS medical directors may wish to adopt it due to ease of administration and other practical advantages. If it does not, then an alternative effort is needed to provide emergency care in the field that is as effective as that provided in the ED.

Choice of Active Control and Inclusion of a Placebo Arm

Published guidelines suggest that the overall NI trial be designed to ensure the ability to distinguish an effective treatment from a less effective or ineffective treatment [16]. The concept is referred to as assay sensitivity and is critical for insuring trial integrity. The most challenging aspect of assay sensitivity is proving that the chosen active control was truly effective in the NI trial i.e., if a placebo was included in the design then the active control would exhibit a benefit over this placebo that is as least as much as what was shown in past comparative studies. The active control should be an established, widely used treatment that can withstand the constancy assumption, meaning it has been consistently shown to have the same treatment effect across similar studies in the patient population of interest and it will continue to show this effect in the NI trial [15]. This requires that the chosen active control and its treatment effect, primary endpoint, patient population and conduct of the NI trial be similar to the historical settings that proved superiority of the active control. It is impossible to provide proof of assay sensitivity in an NI trial, unless a placebo control is included in the proposed study design. However, the inclusion of a placebo arm in an NI trial is rare since the chosen active control often has been proven to be superior to placebo and other active agents. Ethical conflicts become a particularly important issue in life-threatening scenarios such as SE. Thus the investigator is forced to make assumptions about constancy and overall assay sensitivity. As the recent FDA Guidance highlights, supporting evidence that these assumptions are correct often does not occur until the planned NI trial is complete.

To support the choice of active control (IV lorazepam) in RAMPART, the investigators did an extensive search of the existing literature to examine the treatment effect of IV lorazepam. Twelve published trials were identified that included a mix of observational and randomized studies that total 1,439 SE patients. The studies were similar in study population (clinically evident generalized convulsive seizures), dose of lorazepam (2–4mg), treatment in early stages of SE, and definition of efficacy (seizure termination consistently defined as termination of seizure within a specified short period of time, averaging 10 minutes). Figure 1 was submitted as part of the response to the FDA’s clinical hold, specifically to illustrate the performance of IV lorazepam in past studies. Results from 1,253 subjects in randomized clinical trials against placebo (one trial) and other active comparators (six trials) represent 87% of these data. Non-randomized studies were included in the assessment to provide the FDA with all existing information on the performance of the active control in stopping seizures in patients with SE. The individual studies consistently illustrate IV lorazepam’s level of performance in terms of stopping seizures. Based on these data and the proposed design features, the investigators were confident in their choice of active control. However the evidence of overall assay sensitivity still needed to be addressed.

Figure 1:

Effect of IV Lorazepam, Placebo or Active Control in Acute Seizures Studies (marker (●) size indicates sample size)

The recommendation by the FDA to include a placebo arm in RAMPART was likely driven by the recognition that the best or only approach to ensuring assay sensitivity is to include a placebo arm [16, 35]. This arm can be considered a calibration arm, used to ensure that both treatment arms (active and new intervention) are superior to placebo. This would appear viable since RAMPART has a 10-minute window for rescue treatment. However, the ethical justification for inclusion of a placebo arm becomes challenging once there is a known efficacious treatment that has been tested in one or several placebo controlled trials, and when the condition being treated is life threatening or can cause serious sequelae. In the case of RAMPART, only one placebo controlled pre-hospital trial was conducted to test (and showed) the superiority of IV administration of benzodiazepines (lorazepam and diazepam) compared to placebo control [27,28]. The trial concluded that IV administration of benzodiazepines is safe and effective compared to placebo for the pre-hospital treatment of SE patients. Seizures were stopped prior to ED arrival in 59.1% (95% CI: 47%, 71%) of the population randomized to lorazepam, 42.6% (95% CI: 31%, 54%) of those randomized to diazepam and 21.1% (95% CI: 12%, 31%) in the placebo arm. Due to the results of this trial and the standard use of benzodiazepines in the field, it was impossible to convince the participating clinicians associated with RAMPART that a placebo arm would be ethically permissible, and it was widely perceived that this view would be even more strongly held by local Institutional Review Boards (IRBs). Even with the 10-minute rescue treatment, investigators could not rationalize randomization to placebo/no treatment. The ethical constraint on including a placebo arm made it impossible for the investigators to prove their assumption of assay sensitivity in the design of RAMPART.

Designing the study such that it mirrors the settings (in terms of study population, definition of primary outcome, study procedures) of the previous studies that determined the efficacy of IV lorazepam provides at least some support towards the goal of assay sensitivity. However, trial conduct and incorrect assumptions still have the potential to reduce assay sensitivity and the ability to interpret the results. To address the possibility of poor assay sensitivity and to protect the type I error rate (falsely claiming non-inferiority) in RAMPART, the conclusion of non-inferiority at the final analysis requires that the absolute difference in the primary outcome between the IM and IV groups be within the pre-specified NI margin. If and only if this criterion is met, then it also must be shown that the active control, IV lorazepam, maintains superiority over the historical placebo (using the single trial results) as determined by a 2-sided 95% confidence interval for the point estimate of the proportion of seizure termination in the IV lorazepam arm. If the lower limit of the interval is less than the lower limit of IV lorazepam seen in the historical trial (i.e., less than 47%), then non-inferiority of the IM arm cannot be claimed for RAMPART due to poor assay sensitivity.

Justification for the Chosen NI margin

Regulatory authorities have attempted to provide guidance regarding the choice of the NI margin in various therapeutic areas [14,16,17], and researchers have conducted much exploration into the best approach [36,37]. The recent FDA guidance on NI trials specifically states that determining the NI margin is the ‘single greatest challenge’ as it ties closely into assay sensitivity and is based on an assumption of the active control’s effect in the NI trial. Both the FDA guidance and D’Agostino et al [10] present a concise approach to the choice of margin that can be easily understood by general audiences, involving both statistical reasoning and clinical judgment. The RAMPART investigators chose to use a ‘fixed margin approach’ which requires specification of two variables: the past effect of the active control (compared to placebo or proven standard) and the amount of the past effect that should be retained based on clinical judgment. The FDA guidance specifically addresses the NI margin from a single study and the dilemma of not being able to assess the variability of the active control’s effect over placebo. In this situation, the guidance suggests using the lower bound of the confidence interval of the effect size (of the active control compared to placebo/no treatment) seen in that single trial. Although conservative, this minimum treatment effect helps to protect against an overestimation of the assumed active control’s effect and the wrong choice of the NI margin. The NI margin for RAMPART was heavily dependent on the single placebo-controlled trial where the absolute difference in the proportion of termination of seizure prior to ED arrival between the IV lorazepam and placebo control arms was 38% (95% CI: 23%, 52%). Using the lower limit of this confidence interval and applying a commonly used guideline of retaining at least 50% of the active control’s effect over placebo [38], the clinical investigators were presented with a margin of 12%. After discussion with the clinicians regarding the perceived real-world relevance of this margin and how much efficacy physicians were willing to potentially give up in exchange for more practical administration, it was decided that the NI margin should be set to 10%.

Conclusions

The RAMPART investigators received FDA approval to initiate the study in October 2008. The trial enrollment began in June 2009 and is being conducted by roughly 800 paramedic units in over 40 EMS systems across the United States. Patient recruitment was anticipated to be 22 patients per month. Participating sites have exceeded this expected accrual rate and RAMPART anticipates completing recruitment in late 2010 with results presented in early 2011. The investigators will abide by the CONSORT guidelines for NI trials to give their audience the opportunity to assess the trial design and conclusions.

As several researchers correctly point out, not all active controlled trials need to be NI trials and not all NI trials need to only be two-armed studies. However there is a current need for a trial design that allows the conclusion of ‘not worse than some pre-specified amount’ or ‘almost the same as’. The increase in drug/device development and the desire for comparative effectiveness research requires researchers to continue to develop innovative trial design and analysis methods. Life-threatening or rare illnesses are likely candidates for NI trial designs. Yet these are the illnesses that often have few if not only one historically controlled trial available and have ethical questions regarding the addition of a placebo control arm. There have been great advancements in the design of NI trials with a much needed focus on assay sensitivity including estimating the effectiveness of the active control and setting the NI margin. However the design continues to be under much scrutiny, which likely will continue until there is more application of the proposed methodology. RAMPART investigators faced the design challenges and present in this paper a summary of the trial’s statistical analysis plan as well as the responses specific to the FDA clinical hold. Unfortunately the FDA draft document was not available during the development of RAMPART. Nonetheless, several of the approaches taken by the RAMPART investigators comply with those proposed in the guidance. The FDA draft document summarizes ‘best practices’ based on available methodology, and although the presented information is not necessarily new to researchers experienced in NI design it does provide recommendations to the key questions and serves as a helpful guide to all investigators regardless of whether regulatory approval is being sought. The call to researchers now should be to move forward with implementing the recommendations and focus on advancements in less studied areas of the NI design such as interim analysis and stopping guidelines, and the utility of NI trials in the adaptive design setting (beyond switching from non-inferiority to superiority).