Principles of designing a clinical trial: optimizing chances of trial success

Abstract

Purpose of Review

Clinical trials are essential to advance health care and develop new therapies. In this review we discuss the underlying principles of clinical trial design with an emphasis on assessing design risks that lead to trial failure as well as negative trials. While of general interest, this is perhaps particularly timely for the neuromodulation community, given the paucity of well-designed trials in the field. We give some examples from the phantom limb pain (PLP) literature.

Recent Findings

It is critical to gather as much preliminary data as possible and to know how to interpret it in order to choose an appropriate trial design. Therefore, the investigator needs to effectively assess the likely trial design risk/benefit ratio with a view to maximizing the chance of a meaningful outcome, whether this outcome rejects or fails to reject the null hypothesis. This analysis is especially important in a complex and heterogeneous disorder such as PLP, which has had many negative trials.

Summary

We discuss the factors pertaining to a strong trial design benefit/risk assessment, how late trial phases require greater support from preliminary data, how to design trials to minimize risks, maximize benefits, and optimize internal validity as well as the chances of a positive outcome. We highlight the need for investigators to incorporate best practice in trial design to increase the chances of success, to always anticipate unexpected challenges during the trial.

Introduction

Clinical trials investigate new therapeutic interventions in a controlled environment, aiming to minimize biases and build on truth (1). Although experimentation has marked progress throughout human history, James Lind’s “A treatise of the scurvy” published in 1753 pioneered the methodology of modern controlled trials (1,2). Further progress on statistical and other methodological techniques (such as better blinding methods) together with improved ethical guidelines helped advance the science of clinical trial methodology. Therefore modern medicine has seen great advances as trial design and quality improved over the past 60 years, leading to better internal validity, efficiency and reporting (3).

Poorly designed trials often lead to inconclusive or negative results, usually at significant human and financial cost. Clinical trials can only be ethically justified when the potential benefits of a new intervention (whether it is a drug, device, surgery, etc.) for the target population outweighs its risks (4,5). Benefits are “the positive results of a given treatment for an individual or a population (i.e. efficacy, convenience, or even quality of life)”, while risks are “the unfavorable negative results (adverse outcomes) of a given treatment for an individual or a population” (6). The expected risk/benefit ratio will depend on the trial phase and supporting data, as we discuss below. It is the role of the investigator to optimize trial design and thereby maximize the potential benefits of the scientific investigation.

In this review we introduce the underlying principles of clinical trial design, giving examples from the phantom limb pain (PLP) literature. We chose PLP given our experience designing trials for it as well as other neuropathic pain syndromes. Furthermore, PLP clinical trials have tested a variety of interventions including drugs (7,8), non-invasive brain stimulation (NIBS) (9–11), and invasive procedures (12). In addition, PLP is a disorder whose pathophysiology and optimum management remain elusive despite numerous studies. Several of the challenges of clinical trial design discussed here are applicable to PLP research.

Overview of clinical trial phases

Clinical trials aim to advance science by deepening our understanding of the mechanisms of action and therapeutic effects of one or more interventions. However, the final purpose of most clinical trials is to gain regulatory approval of a specific agent, or to compare interventions within a target population (13).

Progressive trial phases require higher levels of evidence and sophistication, depending on the preliminary data and research question. While conducting trials in progressive phases increases trial challenges and short-term costs, this method is important to enhance safety and decrease long-term costs. In fact, late trials (e.g. phase III trials), which have a large number of subjects and are more costly, must have a strong signal from early less costly clinical trials in order to enhance chances of success.

Trial phases are generally divided into: phase I, phase II, phase III and phase IV (optional). See Table 1 for the main goals and typical design of each phase (design can vary at times). This framework is used by and was developed for pharmacological trials. For medical device trials, including those for NIBS, phases are usually divided into feasibility (preliminary or pilot) trials followed by pivotal trials (post-marketing trials are optional here too). There is a less established framework for procedure trials (though they tend to follow device phases), especially given the relative lack of FDA regulations for them (14). More details and examples for each phase will be discussed below.

Table 1.

Phases of clinical trials

	Pharmacological Trials				Medical Device Trials
	Phase I	Phase II	Phase III	Phase IV	Feasibility (preliminary or pilot)	Pivotal
Main goal	Safety	Safety, efficacy	Effectiveness, efficacy, safety	Post-marketing approval, long-term effects	Feasibility and safety	Effectiveness
Design	Open-label	Randomized, controlled, often blinded	Randomized, controlled, typically blinded	Open-label	Open-label and randomized controlled	RCT most common and desirable; possible open-label and non-randomized
Target population	HV and patients	Mostly patients, sometimes HV	Mostly patients	Only patients	Usually patients, possibly HV	Patients
Sample size	Small (n = 20 to 100)	Medium (n = up to several hundred)	Large (n = usually around 300-3000, could be larger)	Large (n = usually more than 1000)	Relatively small (n = usually 10-50)	Larger but smaller than drug phase III (n = usually around 100-300)
Duration	Short	Short (up to 1 year)	Moderate (around 2 years, including follow-ups)	Long (can go up to 3-4 years or an even longer follow-up)	Short (weeks to months)	Usually not as long as drug trials (months)

Abbreviations: HV: healthy volunteers

Clinical trial phases: pharmacological studies

Pharmacological Trials

It is important that the investigator clearly understands the stage of development of the drug or device being tested. Phases of development are classic for drug trials, and are described as phases I-IV as we explain below.

Phase I

A phase I trial primarily aims to evaluate the safety and early efficacy of an intervention or approach (15). It also helps to determine aspects such as pharmacodynamics, pharmacokinetics, adverse event rates and risk/benefit ratio following dose titration. Phase I trials require a small sample size (n = 20 to 100), usually composed healthy volunteers (patients are included when testing more toxic drugs such as chemotherapy). The protocols are often more flexible than in subsequent phases, and are frequently open-label and non-controlled.

Phase II

A phase II trial’s main goal is to demonstrate efficacy and provide additional safety information. It generates data for power calculations and the intervention’s effect size, but the sample is often limited (n = up to several hundred). As a result, phase II trials often utilize surrogate markers (in place of or in addition to clinical endpoints) in a homogeneous sample to maximize the signal to noise ratio and consequently increase power (13,16).

Phase II trials are also important to collect additional safety data, determining drug dosing ranges, routes and timing for phase III trials, as well as common short-term adverse events. There are numerous phase II trials evaluating the efficacy of pharmacological agents in PLP, for example, gabapentin (17), ketamine (18), memantine (19), and calcitonin combined with ketamine (20).

Phase III

A phase III trial aims to evaluate an intervention’s effectiveness in a broader population. It is typically a randomized, controlled, double-blind study comparing an intervention to a control (placebo or standard treatment(s)) on a large sample (n = around 300 to 3000). It is considered a later stage of development for a low risk, high benefit intervention before being put to the market.

Phase III trials provide additional information about effectiveness, efficacy and safety, giving a clearer picture of the intervention’s risk-benefit ratio with a potential to change clinical practice (21). We are not aware of any published phase III drug clinical trials on PLP; however, there is an editorial on the design of the ongoing phase III PLATA trial (Prevention of Phantom Limb Pain After Transtibial Amputation) comparing intravenous pain control vs. optimized intravenous pain control in addition to regional anesthesia (22).

Phase IV

A phase IV trial aims to evaluate the long-term therapeutic and adverse effects of an approved intervention, and to investigate it on a larger scale (n = usually more than 1000) or in new populations, e.g. children, the elderly, etc. Approved interventions often do not undergo phase IV trials; however, these trials can be very useful to deepen our knowledge of a therapy’s optimal use patterns, different adverse event profiles (especially long-term or rare events) and effects on morbidity and/or mortality.

Clinical trial phases: medical device studies

For medical device and diet/procedure studies, the phases are typically divided into the following: Feasibility (preliminary or pilot) and pivotal studies (15).

Feasibility (preliminary or pilot) study

The main goal is to evaluate feasibility, but other relevant goals include safety, design/parameter improvement and to define the limits of tolerability in a small sample (n = around 10 – 50). The sample is usually composed of patients (potentially healthy subjects) in an open-label and randomized controlled trial format.

One such trial applied transcutaneous electrical nerve stimulation as analgesic for PLP in 5 adult amputees (23). Furthermore, Bolognini et al. evaluated the effects of 5-day transcranial direct current stimulation (tDCS) on the motor cortex of 8 amputees with PLP, based on the theory of maladaptive plasticity (10). These trials found promising effects of neuromodulation to alleviate PLP (low risk approach with high potential benefits). Other pilot studies concerned the application of sympathetic blocks (24) and cryoablation (12) for the treatment of PLP. In order to translate these techniques to clinical practice, the next step would be to conduct a pivotal trial.

Pivotal Study

The main goal is to evaluate the intervention’s effectiveness in a larger population of patients (n = around 100 – 300). This is typically an RCT of shorter duration than comparable drug trials (lasting months). One example is a factorial trial assessing transcranial direct current stimulation (tDCS) and mirror therapy in PLP patients (25).

Choosing the appropriate phase design

Although the phase design framework helps the investigator to choose the appropriate phase and thus design the study based on its main objectives, the choice of phase is not always easy to determine. For example, the amount of phase II data needed in order to move to a phase III trial is not always clear. Below, we created a framework based on the risk that the trial will fail vs. the chance that results will be beneficial to help the investigator choose the appropriate phase design.

Understanding clinical trial failure

A successful clinical trial answers its primary research question in a valid way to advance the science. It is important to underscore that failure to reject the null hypothesis (or failure to show positive results, i.e. a negative trial) does not mean trial failure (or that the trial did not advance science). Indeed, failing to reject the null hypothesis also teaches us something important. Well-designed negative trials can generate useful data, and have been published in high-impact journals (26). Trial failure means that results are not valid and so we are unable to either reject or not reject the null hypothesis.

Although a positive outcome is frequently seen as the only indicator of a trial’s success and publish-worthiness, a positive trial could be invalid if associated with significant biases. Therefore, significance does not necessarily mean validity of results. Accordingly, the investigator must assess two important components when designing a trial:

1. Factors that would lead to invalid results (e.g., biases);

2. Whether the trial (assuming it is valid) would advance science regardless of whether it is positive or negative. If a valid trial with negative results would not advance science, then the investigator needs to assess the likelihood his/her hypothesis will be confirmed given the current evidence.

Failed (invalid) clinical trials vs. negative (but valid) clinical trials

One way to see the importance of this problem is by analyzing the attrition rate from early to late phases. Attrition means the drop in the number of drugs that make it to market compared to those studied in preclinical and clinical trials. Most drugs studied (about 90%) never make it to market, as human trials do not show efficacy (27–29). This attrition may happen at all phases and may result from differences between animal models and humans, disparities in designing basic science studies vs. clinical trials, lack of mechanistic data and also by building on invalid early positive clinical data that are not confirmed in later phases (27,28,30–33). Attrition is highly costly and frustrating, yet such studies help by showing us what does not work – “validation using known failures” (32) – assuming they are published.

The problem of failed trials may also be seen indirectly by the large rate of unpublished trials. Chen et al. demonstrated that only 29% of the trials were published in journals and 13% reported on ClinicalTrials.gov (upon examining over 4300 ClinicalTrials.gov registered trials conducted in 2007-2010 across 51 US institutions). By 2014, only two thirds of the trials were either reported or published (34). This was consistent with previous studies showing that 25-50% of trials went unpublished for years (35–39). Trials may not be published due to lack of time (or interest) for the researcher to write up results (40); however, it is likely that many go unpublished due to methodological shortcomings such as unexpected unblinding, large attrition rates, inability to recruit, etc.

Avoiding designing failed clinical trials: main issues that lead to invalid data

RCT design, despite being the gold standard in clinical research, may yet be associated with significant biases leading to invalid results. Investigators must understand the important factors that may lead to invalid results such as randomization, allocation concealment, blinding, and multiple outcomes.

Randomization has a major impact on trial validity. Randomization should balance known and unknown factors between intervention groups (41), such that different results can be causally attributed to the intervention rather than to confounders. To maintain this balance, all randomized subjects should be analyzed (intention-to-treat analysis). Adherence to the trial protocol also has a significant impact on internal validity, and measures should be taken to maintain adherence, e.g. checklists, and careful selection of exclusion criteria.

Related to randomization, allocation concealment (which should always be implemented), also protects against selection bias, e.g. such that investigators cannot encourage only patients they think will benefit to enroll in the trial. Meanwhile, blinding protects against ascertainment bias (changing the outcome on knowing what intervention was received). Unblinded assessors might score outcomes differently; unblinded subjects may report symptoms differently, become less adherent, or drop out at higher rates (if they get sham treatment). However, not all studies can be blinded feasibly or ethically; e.g. many surgical interventions cannot be blinded for these reasons.

Multiple outcomes in a study can also negatively impact validity. A single outcome should be defined a priori, with sample size and power calculations based on that outcome. Not having a clear predefined primary outcome may lead to an underpowered study, and having multiple outcomes may lead to biased results. Importantly, multiple outcome analyses at a set p value have an increasing risk of type I error (false positives), so the multiplicity should be adjusted for statistically or the analysis will be invalid.

Avoiding designing negative (but valid) clinical trials

After eliminating or significantly decreasing potential biases, the investigator still needs to design a strong trial with a high likelihood of confirming the main hypothesis. The strength of a trial will be based on the previous clinical and mechanistic evidence supporting its design. The design should be led by this foundation, rather than the investigator retroactively trying to find data to support a faulty foundation. The greater the supporting data, the more complex and expensive the trial design can be, and vice-versa.

Poor design is probably the top reason for trial failure. Success in a well-executed trial relates to good trial design, which relates to a good research question and a strong specific hypothesis, which in turn depends on a deep understanding of the subject matter and clinical research methods. For ethical reasons, any research question requires clinical equipoise (i.e. it is unclear if the intervention is better than the control) (42). No trial is thus guaranteed success, but a strong design can mitigate potential risks and augment its benefits and feasibility.

Designing studies with a strong foundation I: preliminary clinical data

A trial’s ability to answer its research question is enhanced by having its hypothesis built on coherent preliminary data. Knowledge gaps should not be filled with assumptions – if data is lacking, the hypothesis and research question should be broken down to something simpler that can be supported and answered within a clinical trial.

Some preliminary data should be available to support most of the trial’s design; not only to support the hypothesis but also to support trial feasibility. It is essential to learn as much as possible about the target disease’s epidemiology (e.g. prevalence), and about recruitment and adherence in similar trials. Otherwise, the amount of preliminary data necessary will depend on the trial phase. For instance, in the early phases of a tDCS trial, the investigators tested the device in 8 PLP subjects with one session only, looking for immediate effects and testing different parameters (10). This, combined with further data on brain stimulation in pain, led the same group to then design a large pivotal trial using optimized parameters (25).

Designing studies with a strong foundation II: preliminary mechanistic data

Understanding the step-by-step process by which a disease occurs and by which an intervention will interact with the patient’s body aids the development of a strong hypothesis. In fact, mechanistic data is essential to determine the main aspects of the research question, known as PICOT: Population, Intervention, Control, Outcome, and Timeline. Understanding the mechanisms of the disease as well as those of the intervention being tested is essential to define the population, type of outcome, timeline, and other factors. This understanding for instance led to the design of a trial on tDCS in PLP including follow-ups months after treatment was completed (25), because tDCS mechanisms of action putatively lead to neuroplastic changes lasting several months (43).

Lacking mechanistic data leads to an increased risk of unexpected results even if the trial design eliminated all or most sources of bias. In fact, a limited understanding of the disease process is often highlighted by an abundance of negative trials, e.g. in PLP. It is much harder to design a clinical trial effectively if one doesn’t understand the interactions of genetic, environmental and/or pathophysiologic factors leading to a disease, especially one that is heterogeneous or complex (44).

One tool to overcome this problem is demonstrated by Griffin and Tsao, who proposed a “mechanism-based classification of phantom limb pain” based on theorized mechanisms of action. In this way they aimed to help target patient subpopulations that might benefit from specific interventions, rather than group different PLP subsets to receive a therapy that might only works on a small number of them (44). Such a mechanism-based approach may help direct clinical trials toward positive outcomes. For this reason, the NIH proposed a mechanistic data model to design psychiatry trials – the RDoC (Research Domain Criteria) framework (45).

Designing studies with a strong foundation III: preliminary safety data

Clinical research should ultimately improve population health. Therefore, any intervention must provide significantly more benefit than harm. Safety is defined by how much an intervention increases patients’ risks of having adverse events (especially serious ones such as death or hospitalization). A solid understanding of the literature and putative mechanisms of action leads to an enhanced appraisal of the intervention’s safety and probable future adverse events.

A study’s risk-benefit ratio should always include safety outcomes, whether they are clinical, surrogate or both. Monitoring adverse events in a study is critical. Patient safety must be prioritized over trial completion, and a well-designed clinical trial is less likely to be terminated early for safety reasons.

The last component: maximizing benefits from the information collected from a valid trial

Clinical trials, when successful, advance science by providing important insights into disease and treatment mechanisms, eventually changing clinical practice. The results of a trial should be interpretable, whether they are positive or negative; if they are not, then the preliminary data should be reassessed and a new research question considered. Preliminary data may come from models, observational studies or the endpoints of related studies.

A well-designed trial’s primary endpoint (outcome) should teach you something about the therapeutic process, how well it works (if at all) and its underlying mechanisms. Secondary endpoints help generate hypotheses for future trials. In this way researchers can build on models and different types of data (clinical, surrogate, etc.) from progressive trial phases, contextualizing them to improve clinical practice.

How to increase the impact of a clinical trial

Clinical trials consume tremendous human and financial resources. Thus, it is important that to maximize the chances that the trial will be Feasible, Interesting, Novel, Ethical and especially Relevant (FINER criteria).

It is critical to assess how the results of a trial will better the field if its aims are achieved – before selecting these aims. An investigator must carefully consider the problem the study is trying to solve. The more prevalent and devastating the problem is, and the greater the potential benefits, the easier it is to justify performing the trial. Conversely, if the study is on, for example, a very rare genetic disorder or a common but minor cosmetic issue, then the benefits of the trial may not justify the costs (especially for trials funded by taxpayers).

The impact of improving medical knowledge is conditional on this knowledge changing clinical practice. Thus, surrogate outcomes should not stand alone but be clinically validated, and clinical outcomes should reach the minimum clinically important difference. Eventually, study outcomes should lead to cures and treatments for previously untreatable conditions, improved therapeutic options for different patient subgroups, more efficient or less costly healthcare delivery, or similarly meaningful improvements to healthcare.

Balancing risk and benefit

The researcher needs to perform a benefit/risk assessment when choosing a study design. Trial results can never be guaranteed but it may be reasonable to expect benefit based on previous studies; this benefit may occur at the individual or group level, with subsets of patients improving dramatically and others not at all.

Assessing the risk/benefit ratio does not mean to automatically reject high risk or low benefit proposals, but rather to adjust the design to minimize cost to subjects and society. We develop and discuss a basic framework with four scenarios (see Table 2).

Table 2.

Benefit/Risk Assessment and related clinical phases

Risks	Benefits	Next Step
+	+	Pilot, phase I or small mechanistic phase II trial to evaluate the risks
+	−	Revise research question
−	−	If feasible (costs and practicability) run a small trial to optimize treatment
−	+	Ideal scenario – phase II/III, phase III or pivotal trial

High risk, high benefit trial

This type of trial can be run if necessary. A trial is only ethical if the proportion of benefit clearly outweighs the risk, especially when there are more than minimal risks. However, trials with higher potential benefits are often highly risky to patients, but may be the only existing options to help populations with devastating diseases. For instance, cancer patients undergoing investigational chemotherapy may suffer from toxic side effects, but their therapeutic options are so limited that running the study is justified. The risks inherent in such trials must be minimized, e.g. by cautiously choosing eligibility criteria and stringently monitoring for adverse events.

When both potential benefits and risks are high, we recommend running pilot, phase I, or small mechanistic phase II studies to better evaluate and quantify the risks. If the risks have been previously quantified and there are no better options, then one may consider running late phase II or III trials, e.g. the PLATA trial investigating the effects of anesthetics combined with intravenous pain medication to prevent PLP development (22), where the authors did not find evidence of positive outcomes with anesthetics in the preliminary data, but instead cited an RCT supporting stringent perioperative control of pain (46). But researchers need to be careful as investigations in this category can fail more often than in less risky trials. One example is a phase III trial investigating tDCS for neuropathic pain that was not based on strong preliminary data. Indeed, this trial failed and thus patients were unnecessarily exposed to risk (47).

High risk, low benefit trial

One should not run this study, but rather revise the research question to reverse the benefit/risk ratio. For example, in a poorly studied intervention that is not likely to lead to a minimum clinically important difference, it is too early to design the trial for such an outcome. It would be better to revise the design to better understand mechanisms, risks and benefits, or to explore other therapeutic options altogether.

Low risk, high benefit trial

This is the ideal type of study to run, assuming there is enough preliminary data to effectively evaluate risks and benefits (and that the benefit/risk assessment is not based on wishful thinking or assumptions). Depending on the robustness of previous data, phase II, phase III or pivotal trials with clinically relevant outcomes can be run. For example, in Bolognini et al., 5 days of tDCS (a technique thought to be safe (48) that has been studied in neuropathic pain) led to a significant and sustained reduction of pain levels in PLP patients, along with an improved ability to move their limbs (10). Following such positive findings, a larger phase III trial was designed (25).

Low risk, low benefit trial

If risks are very low and the study is easy (and not too expensive) to conduct, a pilot trial can help quantify and better evaluate the benefits, helping optimize them for future trial designs. For instance, a study evaluating the analgesic effects of gabapentin enrolled 24 PLP patients (17). However, pain level differences in the active group (even if over half the participants had a meaningful decrease in pain) did not reach significance compared to placebo. A pilot study aiming to optimize benefits may have been a better option.

In summary, a benefit-risk assessment is always necessary when designing a trial, and the benefits and risks to research subjects vs. those for the target population are crucial to identify.

Conclusions

Well-designed clinical trials are essential for the advancement of modern medicine. Developing an in-depth understanding of existing preliminary data and a risk-benefit analysis can lead to an optimal research question and a solid trial design. The investigator needs to be cautious and conservative in his/her choices given the risk of losing time and resources in an invalid or valid negative trial. Progressive trial phases require more preliminary data to increase the potential benefits and reduce the risks of the trial.

Even upon following best practices in trial design, the investigator needs to be prepared for unexpected roadblocks such as lack of funding, challenges in training personnel, recruitment, and other factors diminishing trial feasibility. It is important to actively anticipate major challenges and ways to deal with them while still in the design phase.