Progressive Staging of Pilot Studies to Improve Phase III Trials for Motor Interventions

Abstract

Based on the suboptimal research pathways that finally led to multicenter randomized clinical trials (MRCTs) of treadmill training with partial body weight support and of robotic assistive devices, strategically planned successive stages are proposed for pilot studies of novel rehabilitation interventions Stage 1, consideration-of-concept studies, drawn from animal experiments, theories, and observations, delineate the experimental intervention in a small convenience sample of participants, so the results must be interpreted with caution. Stage 2, development-of-concept pilots, should optimize the components of the intervention, settle on most appropriate outcome measures, and examine dose-response effects. A well-designed study that reveals no efficacy should be published to counterweight the confirmation bias of positive trials. Stage 3, demonstration-of-concept pilots, can build out from what has been learned to test at least 15 participants in each arm, using random assignment and blinded outcome measures. A control group should receive an active practice intervention aimed at the same primary outcome. A third arm could receive a substantially larger dose of the experimental therapy or a combinational intervention. If only 1 site performed this trial, a different investigative group should aim to reproduce positive outcomes based on the optimal dose of motor training. Stage 3 studies ought to suggest an effect size of 0.4 or higher, so that approximately 50 participants in each arm will be the number required to test for efficacy in a stage 4, proof-of-concept MRCT. By developing a consensus around acceptable and necessary practices for each stage, similar to CONSORT recommendations for the publication of phase III clinical trials, better quality pilot studies may move quickly into better designed and more successful MRCTs of experimental interventions.

The recent statement from the Cumberland Consensus Working Group on the “Future of restorative neurosciences in stroke” looked for answers to the nagging question of why the traditional model of a translational research pipeline that transforms basic science into novel clinical practice has generally failed to improve rehabilitation practice for people after stroke.¹ The group suggested that much translational restorative research has been limited to small-scale studies and that patient recruitment for larger clinical trials has been difficult. One offered solution is to develop collaborations among multiple sites to optimize pilot data for the design of multi-center randomized clinical trials (MRCTs).

An MRCT is difficult to develop, manage, and finance. This gold standard for the scientific testing of, for example, interventions to enhance motor recovery, requires solutions for many experimental details.^2–6 MRCTs differ from the far less restrained approach of the pilot studies that precede them. Pilot studies often evolve opportunistically for a select group of participants who are given an intervention that is a work in progress. The exploratory data may be massaged to find some statistical inference of benefit, followed by a hope for publication and grant to pursue the idea further. This early feasibility pathway of discovery, more often than not, can lead to results that cannot be replicated and may be misleading in regard to the likely real-world strength of the intervention.

The Food and Drug Administration (FDA) established the phases of clinical trials primarily for the pharmaceutical industry to test medications. These regulatory-related classifications, however, may obscure the line of experiments necessary to test rehabilitation interventions that involve physical and cognitive interventions. FDA phase I studies examine the safety and chemical actions of a medication in healthy participants. Phase II tests the safety and develops preliminary efficacy data in patients with the target of interest and, usually, no other medical problems.⁷ Dose-response studies may accompany this phase. In phase III, a prospective, randomized controlled trial, often involving a placebo therapy, is carried out in a planned number of participants to assess efficacy.

The phases of rehabilitation trials of physical and cognitive therapies tend to move forward in random increments compared to the FDA process for drug approval. This problem is apparent for studies of body weight-supported treadmill training (BWSTT) and robotic assistive devices. Despite many pilot studies and randomized clinical trials (RCTs) of various forms of treadmill training with body weight support for stroke, paraparesis after spinal cord injury (SCI), cerebral palsy, multiple sclerosis, and Parkinson’s disease, this technique cannot yet be considered a standard of care.⁸ The same limitation holds for the growing number of robotic assists employed to improve ambulation and functional use of an upper extremity.^9,10 At best, these are still adjunct interventions for patients who have not improved to their potential after routine task-related skills training, task modification, selective strengthening, and conditioning.¹¹ After 20 years of studies of treadmill use and 10 years of robotics, the rehabilitation community and patients may wonder why clinicians still cannot define a subgroup of patients who may benefit.¹² A nest full of problems arise from the erratic flight pattern of rehabilitation pilot studies of novel strategies. Can specific stages of pilot studies be developed that will converge into MRCTs of high value to patients?

Problems

A key aim of clinical research in neurologic rehabilitation is to develop pragmatic, validated, valued, and robust therapeutic interventions. Therapies under consideration usually arise from experience, intuition, fortuitous opportunities, and attempts to translate an idea from basic research into clinical practice. Clinical pilot experiments usually start out as quasi-experimental, underpowered studies with or without a control group and randomization.^13,14 They later fall along a continuum between an observational case series and a single site RCT, always with manifest and latent biases.

The literature covering neurorehabilitation has been an alluring minefield of pilot studies of potential therapies. Interventions sound wonderful when applied to a half-dozen selected participants who are pretested and posttested with multiple outcome measures, until statistical significance arises somewhere in the heap of calculations. If published, the authors hope that their study will be cited and lead to larger trials. However, as of 2006, about 40% of the 15 million articles in PubMed had never been cited and most of the rest were cited fewer than 3 times.¹⁵ For example, using the ISI Web of Knowledge to find articles under “stroke” and “rehabilitation,” only 38% of the 7662 articles written as of November 2008 had been cited more than 5 times and half had been cited less than 3 times. One expects that self-citations account for at least several of every article’s total citations. This academic black hole of uncited and never-to-be-followed-up publications may be especially deep for reports of pilot studies in neurological rehabilitation.

If a pilot study creates a buzz and is promoted at symposia that seek a touch of novelty, it begets more pilot studies by the same and new investigators. This small pilot-on-pilot scenario moved slowly forward for the 2 most novel interventions of the 1990s, BWSTT and constraint-induced motor training (CIMT). BWSTT was developed first for SCI² and stroke,¹² but 16 years passed before an MRCT for SCI was completed.^16,17 For SCI, the technique turned out to improve walking no better than early conventional rehabilitation.¹⁷ For patients who walk slowly after stroke, the best-designed trial of BWSTT, called Locomotor Experience Applied Post-Stroke (LEAPS), will be completed in late 2009.⁴ For all those years, uncontrolled and small trials prevailed using varied training techniques in patients with a range of levels of impairment and walking ability. Commercial treadmill systems were developed and sold, despite the lack of evidence that the technique would be better than conventional gait retraining. Interventions that employ robotic assists for walking seem to be following the same pattern of pilot-on-pilot studies that cannot prove efficacy. This pattern of pilot studies is not unique to rehabilitation interventions that seek to employ technologies. CIMT followed a similar course, until the Extremity Constraint-induced Therapy Evaluation (EXCITE) MRCT was published a dozen years later.¹⁸

Authors of pilot studies tend to inflate the qualitative interpretation of their results in the discussion section or conclusion of a publication. Reviewers and editors of journals read many more of these marketing-like statements that never make it to publication, but may appear in a poster or presentation at a scientific meeting. The authors often focus on selected data about motor recovery that may include unwarranted extrapolations and overstate the generalizability of the results. Conclusions may include a statement of “likely efficacy” that should “lead to a larger randomized controlled” trial. Authors may downplay the limitations of the pilot trial and avoid mentioning caveats. Often, they use multiple statistical comparisons without specifying in advance the endpoint, which is the focus of the intervention that, if improved, will enhance the motor functioning of patients. The search for confirmational P values may reveal positive results, but hide or downplay the negative results that do not make it to print. One significant result out of many, however, soon leads to the conclusion that an effect exists when a type 1 error is likely, that is, no real effect. Rehabilitation pilot trials are not alone in their culpability. In much of biomedicine, the sparse success of clinical translation from basic science studies suggests that the over-interpretation of highly restricted experiments is common.

Most pilot studies and even some of the best MRCTs for motor-related rehabilitation have not employed a proper control group. One excuse is that the usual therapy, which is no therapy if a patient no longer is receiving rehabilitation, serves as a real-world control. A nonactive placebo may be the only therapy in a drug trial if no standard drug therapy exists, but comparing a specific rehabilitation treatment to nothing is no longer acceptable. The World Medical Association, which wrote the Declaration of Helsinki for ethical medical research, recently proposed (October, 2008) an amendment that states, “a new intervention must be tested against … the best current proven intervention.” A placebo is acceptable “where no current proven intervention exists.” Motor recovery studies have repeatedly gone beyond the once necessary demonstration that an active therapy for walking or use of the upper extremity is likely to lessen impairment or disability more than no therapy.¹⁹ Indeed, a low-tech and conventional exercise approach may be as efficacious as more complex strategies such as BWSTT and robotics.²⁰ The well-performed EXCITE trial compared the CIMT intervention for 2 weeks to no intervention, because of NIH-imposed cost restraints,²¹ so some may interpret this MRCT as a published precedent for an inactive control arm. Others may interpret EXCITE as showing only that a well-managed task-specific intervention is better than no therapy.²² Unfortunately, pilot studies of CIMT against an active control intervention had been infrequent or too small to interpret to inform the need for a comparison therapy at the time EXCITE was approved for funding.

A select review of recent pilot studies and MRCTs to improve walking and functional use of the upper extremity reveals additional problems that have solutions.

Studies of Treadmill Walking

Techniques for BWSTT without and with robotic assistive devices were initially developed to enable patients with profound paraparesis from SCI to take steps. Approaches were built on the preclinical reports of stepping on a moving TM belt generated by cats and other mammals after low thoracic spinal transection and tail stimulation.²³ The conceptual basis related to lumbar spinal cord pattern generation and learning was fascinating, but tenuous when transferred to patients with incomplete SCI or stroke who had varied degrees of descending and ascending supraspinal actions. The technique made sense from perhaps a more important angle for patients with modest residual motor control. BWSTT allows task-oriented practice of walking with the assistance of therapists who move the legs and provide cues. The moving belt induces automatic hip extension and rhythmicity for stepping and weight support, and limits the need to fully load a leg that may buckle from paresis. As encouraging as that seems, it was up to investigators to determine how hemiparetic and paraparetic patients who walked poorly should be trained. Parameters for consideration included levels of maximal limb loading, particular speeds that enabled practice to lessen identified gait deviations and produce faster over-ground velocities, forms of feedback, and others.^24,25

The outcome measure for the spinalized animal was that it could take steps at a variety of speeds on the moving belt, but not over ground. The outcome measure for patients, then, was level of independence in walking, walking speed over a short distance, or the distance walked in 2 to 6 minutes.^26,27 Pilot studies tended to not consider a change in a speed-based classification that could measure a clinically important change in capability.^26,28,29 Whereas participants given BWSTT in uncontrolled pilot studies often improved in their over-ground walking speeds from 10% to 30%, the final speed that might allow them to move about safely or in the community was generally not enough to lead to community walking. One confounder in interpreting clinical pilot studies (and many MRCTs) is that the outcome measures employed may not reveal the threshold of a gain in function that could have value for patients.³⁰ Instead, authors report percent changes or absolute gains (sometimes questionably using an ordinal scale), but no direct connection to effective walking or use of the arm is evident.

Pilot studies had shown likely efficacy of BWSTT compared to no controls or historical controls, which usually overestimates the effectiveness of a new treatment. A total of 4 RCTs with 222 patients subsequently found no statistically significant effect of locomotor training on walking with BWSTT in a Cochrane analysis, or with the addition of functional electrical stimulation or robotic assists after SCI.³¹ The same lack of efficacy was reported in a Cochrane analysis of stroke.⁸ Pilot studies offered clues about the potential of combining BWSTT with co-interventions, but this information has not been incorporated into subsequent pilot studies.^32–38 The literature on cerebral palsy and other diseases includes only small trials and modest, if any, changes in walking speed or function. Not included in the Cochrane analysis of MRCTs are several recent trials. These studies continue the pattern of aiming to show that an active intervention is better than no intervention. For example, in a randomized trial of 70 patients with chronic stroke, 6 months of progressive treadmill training compared to no training (just stretching) led to improved aerobic fitness (51% vs 11% increase) and walking speed (19% increase vs 8% gain). Thus, aerobic training is better than no aerobic training if the goal is to improve aerobic fitness. Walking speed did not improve enough to alter functional walking ability. Cerebral adaptations to training were noted by functional magnetic resonance imaging (fMRI), but that is not a surrogate for functional gains.³⁹

Robotic Assisted Stepping

A spin-off from BWSTT has been robotic assistive training for patients with SCI, stroke, and multiple sclerosis. The Lokomat (Hocoma)⁴⁰ and the Gait Trainer (Reha-Stim)⁴¹ were the first commercially available devices to provide repetition of guided leg movements. Pilot studies with uncontrolled test-retest, crossover, and A-B-A designs suggested possible efficacy, as well as cerebral adaptations associated with training.^42,43 These pilot studies, however, did not manipulate ways to optimally employ the devices in terms of adjusting stepping speeds, unloading body weight, duration and intensity of training, and most important, just what the participants were meant to practice and learn. As with testing BWSTT, little effort during pilot studies went into defining the best ways to deliver the new training paradigm. The utility of physical and verbal cues for instruction and feedback were not examined. No standard approach was reported and tested in regard to what the patient was supposed to practice on the device to correct gait deviations and carry over to walking.^44,45 The SCILT and LEAPS trials did define a mix of BWSTT and over-ground training, although the combination was based more on consensus about the best approach rather than drawing on the results from pilot studies that had established the most efficacious strategy. For robotic assists, the results of RCTs parallel those of BWSTT to date. A Cochrane analysis concluded that electromechanical and robotic-assisted gait training in a variety of paradigms across 8 trials and 414 participants prior to 2008 could increase the proportion of patients who walked independently, but had no clinically meaningful effect on walking distance or speed.⁹ Patients who did walk independently required over-ground training as well.

Not discussed in the Cochrane review were 2 recent trials of interest to the development of more rational sequences of pilot studies. Lo and Triche carried out a crossover trial of 13 participants with multiple sclerosis who could walk over ground.⁴⁶ They were assigned to 12 sessions of BWSTT versus BWSTT with the Lokomat robotic device. At the end of training, no significant differences in walking speed, distance walked, or change score on the Kurtzke Expanded Disability Status Scale between the 2 approaches were found. However, walking speed and distance did significantly increase after both interventions by 30%. Without a control intervention, it would have appeared that either of the interventions was better than no intervention. Some large within-subject gains were found, but the lack of a phase-in of therapy for walking to wean out early rapid gainers may account for some of these outliers.

Hidler and colleagues⁴⁷ reported an MRCT that also offers lessons. This well-designed trial took 5 years to complete, despite reasonably broad inclusion and exclusion criteria. For example, the target population had a self-selected walking speed over ground of 0.35 m/s after a recent stroke. The control intervention was task-related, with over-ground gait training that emphasized walking speed, stability, step symmetry, and endurance. The Lokomat arm was told to move their legs with the device and was given progressive training for speed, unweighting, endurance, and less guidance assistance from the device. The Lokomat provided visual feedback about hip and knee kinematics to be matched to the Lokomat. Given 24 training sessions, 63 of 72 participants completed the trial at 2 sites. Correcting for days poststroke, gains at posttraining were 0.12 m/s for the Lokomat group versus 0.25 m/s for the conventional group (P = .002). At 3 months after entry, the differences were 0.15 m/s versus 0.30 m/s (P = .08) with similar changes for distance walked in 6 minutes. The authors concluded that the diversity of conventional gait training was more effective than robotic assisted training to facilitate walking ability.

No single MRCT, let alone series of pilot studies, makes for a final statement about efficacy, but this MRCT from Hidler and colleagues using the Lokomat does set a standard for replication studies. Of interest, if one only looks at pretesting versus posttesting walking outcomes for either intervention alone, the differences are statistically significant. Thus, if no control arm had been included, as happens in most pilot studies, the investigators would have shown only that task-related therapy of enough intensity is better than no therapy. In addition, prior robotic pilot studies did not offer information about the optimal number of training sessions, whether the relatively slow treadmill speed of the Gait Trainer and Lokomat^48,49 might limit their effectiveness for gains in walking speed, and whether concordant over-ground training could have been used to determine what ought to be practiced on the Lokomat to help TM training generalize to over-ground gait. What is evident is that whatever the participant practices is less observable when strapped into a mechanical device and so the amount of participant involvement and subsequent task-related learning is uncertain.^50,51 Other devices may be less restrictive in their assistance for stepping, but future experiments will need to tackle these same potential confounders.

Upper Extremity Robotic Assists

The CIMT strategy that was employed for the EXCITE trial was preceded and followed by many pilot studies that modified the duration and frequency of therapy, time of initiation after stroke onset, reliance on constraining the unaffected hand, and sometimes included active control participants and functional imaging-related outcomes.^52–56 It stretches the imagination to consider just how all of these unconnected pilot studies will be developed as a comparison intervention to the one used in EXCITE. This diversity of small trials that do not build on each other has also characterized the development of robotic assistive devices that might enable task-related practice with greater duration of therapy, assist the onset of a limited movement, and offer feedback that provides greater motivation than usual physical and occupational therapy activities.^10,57 Across 11 trials of varying design and 328 participants, however, a Cochrane review found that electromechanical and robot-assisted arm training improved strength in some proximal muscles, but did not improve use of the arm in daily activities.⁵⁸ Subsequent quasi-experimental studies have provided more pilot data on a variety of devices, but do not set the stage for an MRCT.^59,60

Of interest, the MIT-Manus robotic assist, which had been thoroughly studied in a number of uncontrolled pilots, was compared recently in a randomized study of 21 patients to a more conventional task-related treatment that matched the intensity and emphasis of the robotic assist on shoulder/elbow movements. Participants in each group realized a 3-point increase in the Fugl-Meyer motor score after 18 hours of therapy.⁶¹ Again, an active control group altered the interpretation of the results of the experimental intervention. The data pointed to the need to develop a more flexible MIT-Manus type of robotic device—one that assisted more than tabletop reaching in a horizontal plane and included activity of the wrist and perhaps hand. The Veterans Administration Cooperative Study 558, “Robotic assisted upper-limb neurore-habilitation in stroke,” is testing a 3-part system in an MRCT. Potential concern is that no phase-in of arm therapy is planned, so one may see outliers who get much better quickly after starting their therapy, beyond any expectation for a change in motor control. The devices being used are quite new, so an optimal training paradigm may not yet be quite ready for a primetime MRCT. The trial uses the Fugl-Meyer as its primary outcome measure.⁶² Pilots often use this scale to evaluate motor impairment, but the clinical meaning of an increase of 3 to 5 points, which is in the range of the minimal detectable change, has uncertain clinical meaning for multijoint movements needed in daily activities.⁶³

Summary of the Problems

Most rehabilitation MRCTs have failed to show efficacy of an experimental motor-related therapy or only found it to be better than no intervention or demonstrated a marginal gain in terms of clinical significance. If pilot studies that test hypotheses about a rehabilitation therapy progressed forward by a more systematic approach in definable stages of development, perhaps more optimal MRCTs could be designed that lead to better treatments for patients. By describing 4 general stages that seem to characterize past pilot studies, I intend to challenge the clinical trials community to develop systematic guidelines for pilot studies, perhaps in parallel to those embodied in the CONSORT recommendations for RCT publications.⁶⁴

Solutions

Staging Pilot Studies

Stage 1: Consideration-of-Concept Trials

The primary endpoints in an FDA phase I trial relate to safety of the maximally tolerated dose in healthy participants. The rationale is to maximize the therapeutic potential of the test medication and find a safe dose range for subsequent phases that will not be subtherapeutic. The consideration-of-concept stage for neurological rehabilitation studies includes elements of a phase I trial, but also looks for information to support or quickly refute the investigator’s opinion about going beyond initial empirical studies. Although healthy participants may be involved in developing the basis for the new intervention, affected patients are the usual participants.

Most stage 1 clinical rehabilitation pilot studies of physical and cognitive interventions start with a convenience sample of 6 to 12 participants. A series of questions ought to be considered as investigators reach beyond their usual therapeutic strategies. The conceptual basis for the intervention should be definable—why and how might this approach work? Motor interventions often aim to enable the patient to use spared neural pathways of the distributed sensorimotor and cognitive systems. Is the therapeutic strategy combined with practice at a high enough intensity to augment skills learning and improve outcomes that are relevant to the practiced skill? What ought to be practiced? How much therapy each day for how many days? Who, in terms of degree of impairment or disability, ought to receive the intervention? Should patients who already possess good motor control (enough selective movement to reach and open and close the hand or to walk >0.6 m/s) be included? These higher-level participants already have many practice options. In addition, can combinational strategies that act on related aspects of the targeted impairment and disability be tested to create a more formidable intervention?

This stage often draws on patients who completed their formal rehabilitation. Authors then state, “We used a stable chronic group of participants to lessen the possibility of spontaneous recovery.” Chronic impairment and disability, however, does not mean unchanging impairment or disability any more than chronic disease suggests that symptoms and signs do not improve or worsen or fluctuate over short intervals. If the investigator believes that the population has an unchanging clinical measure of the target of the new intervention, that measure ought to be repeated weekly for a few weeks before the start of the intervention. Even better, pilot studies that do, and certainly those that do not, include a control group might consider giving all participants a modest conventional intervention for perhaps 6 sessions as a phase-in to the new treatment.²² If the outcome measure improves, then this serves as a new baseline and the investigators can consider providing another round of 6 sessions, until the participants reach a plateau by conventional therapy. This approach lessens the likelihood of a rapid gain in participants who have a greater latent capacity to improve than expected and simply needed some additional routine therapy, rather than a new form of treatment. The investigator also diminishes the chance that a few outlier participants who improve beyond expectations (an increase >30% in walking speed or >5 points on the Fugl-Meyer score for the arm, for example) will lead down the road to impressive, but unrealistic group results. Thus, as nonuse has become the basis for the utility of CIMT in patients who have fair motor control of the hand and wrist, surely the investigator wants to reduce the likelihood of gains from any active intervention of the same intensity, not just from the new one being tested for inclusion in a phase III trial.

The value of a stage 1 pilot study with only a preintervention and postintervention outcome measure is limited to learning how well the intervention can be applied and to whom it ought to be given, its safety, the utility of the outcome measures chosen, and the variation in patient responses. The conclusions of these small studies, however, are perhaps too often interpreted by the researcher with confirmation bias, which is a belief that the experimental data are important, fit into the web of rehabilitation science expectations, and do not contradict best hopes for the experimental therapy.

Stage 2: Development-of-Concept Trials

This next development-of-concept stage should standardize the new intervention and add a control group, randomization, and masked outcomes. At this point, pilot studies must pay attention to chance findings and deal accordingly. Indeed, uncontrolled pilot studies of motor retraining strategies should be abandoned as soon as the protocol has been refined enough for genuine testing. In addition, a task-related active intervention should serve as the control, even if the investigator can only enter 15 participants.

This stage is also an opportunity to test enrichment strategies that lead to the best possible experimental treatment for an MRCT.²² Regarding research design, investigators can combine 2 therapies^65,66 compared to one by a factorial design⁶⁷ or test for the therapeutic equivalence between the new intervention and the usual one by employing a therapeutic equivalence design.⁶⁸ Stage 2 can be used to consider multiple treatments in 1 trial by a multiple crossover-periods/multiple parallel-groups design. Other insight can be gained by adjusting change scores for baseline levels through multivariate adjustment. These approaches, even in a well-powered trial, however, increase the risk of type 1 and type 2 errors and the loss of validity criteria. As stage 2 trials are completed, they should offer insight into the biggest-bang-for-the-buck, new experimental therapy.

The conceptual basis for a new strategy at this stage increasingly falls under the rubric of a manipulation of motor learning-related neuroplasticity.^69–72 Pilot studies must not, however, rely on training-related cortical remapping to make the case for a step toward an MRCT. Changes assessed by fMRI and transcranial magnetic stimulation (TMS) do not serve as a surrogate for gains in motor control. The great majority of costly fMRI studies include fewer than the 15 or more participants in a treatment and a control group that may be needed to reveal statistical significance without a type 1 error.⁷³ Although functional imaging may reveal changes in voxel counts and signal intensity and TMS may reveal an evolution in cortical excitability over time, the changes do not reveal brain-behavior causality, let alone serve as a substitute for effectiveness. Any mechanistic exploration for physiological and behavioral correlations is likely to have to include serial measures, for example, 2 preintervention activation and behavioral studies, several during treatment, one at completion, and one at 3 to 6 months later.^69,74

Even underpowered stage 2 trials can have great value within the context of staging successive pilot studies. A series of pilots can examine the practicality of delivering and monitoring the effects of the new intervention.⁷⁵ This stage can assess other outcome measures and help decide whether a measure of impairment, disability or quality of life, for example, best compliments the intervention. Stage 2 is especially appropriate to test different doses of the new therapy^66,76 or to incorporate an additional component to the primary intervention that might increase motor skills learning, such as augmented feedback. The pilot can examine any number of questions that must be settled prior to the MRCT. Was it feasible to obtain a representative sample selected from a relevant population? Entry criteria may need to be loosened if participants are few and far between. Were participants available who could be entered at a similar point in their impairment or disability? What descriptors ought to be collected for baseline measurements? Were the interventions and data collections successfully operationalized? What is the shortest duration of therapy and longest follow-up time necessary to observe a plateau in gains? Is blinding of participants and of observers who assess outcomes feasible? Can outcomes be assessed objectively and reliably?⁷⁷ Which statistical tests and models offer the best approach to the hypothesis and data? Can variability in the primary and secondary outcome measures be estimated? A fork in the road at stage 2 must be taken, to go or not to go toward an MRCT. Thus, stage 2 trials that aim to build on a prior success but fail to replicate positive findings ought to be published.⁷⁸

Stage 3: Demonstration-of-Concept Trials

This stage aims to find compelling arguments for the potential utility of the new intervention. Within the context of FDA phases, the demonstration-of-concept stage falls within phase II trials. Now that the interventional technique has been operationally defined for the participants who are most likely to benefit, stage 2 calls for better designs and larger, but probably underpowered trials.

An active control intervention is especially critical here. From a statistical view, a P value does not measure the weight of evidence against the null hypothesis relative to an alternative hypothesis, because no alternative is included in the statistical procedure. Comparing something to nothing at this stage will not estimate the potential efficacy of a new motor therapy. This recommendation also applies even if the investigators show improvement that at least equals either the minimal detectable change (MDC) or the minimal clinically important difference (MCID), as derived from other studies of the measurement tools.⁷⁹ These measures of change from baseline function, which are often substituted for a control in rehabilitation pilot studies, may be fool’s gold. The MDC is the smallest amount of change that likely reflects a reliable change, rather than an error of measurement inherent in the score. Thus, it is the safest threshold for showing statistically detectable individual changes. It is derived from the z score, standard error of measurement (SEM), and standard deviation. Just how to best measure and interpret the MCID is uncertain.^80,81 For a within-person change, the MCID value should be greater than the MDC to indicate that the measure is precise enough to indicate a meaningful clinical change. Some authors have used coefficients of the SEM, ranging from 1 to 2.77.⁸² The problem here is that even if the magnitude of change on a scale were a clinically meaningful difference, the scale may still not inform the investigator about what the gain means to patients.⁷⁹ For outpatient walking and upper extremity trials in stage 3, hierarchical levels of function ought to be considered within the context of clinical value, such as the ability to walk at home versus with limitations in the community versus without limitations, based on a specific demonstration of that functional level or by usual walking speed.^4,20

Stage 3 trials should enable the calculation of a potential effect size, that is, the magnitude of the real effect to be detected. The effect size is the mean result of the experimental group, minus the mean of the control group, and divided by the standard deviation of the control group for the dependent variable, which is the primary outcome measure. A moderate effect is 0.5. The higher the effect, the smaller the number of participants needed to treat to achieve the statistically significant gain of less impairment or disability. At a typical power of 0.8 for a 2-tailed alpha = 0.05, the number of well-matched participants needed in each arm of an RCT will be about 30 to 40. That is not a lot of participants compared to trials of medications to manage most medical diseases. Consider the recent report played as a remarkable achievement by the media in which 100 participants would have to be treated with a statin for a year to prevent 1 stroke or myocardial infarction in adults who have an elevated C-reactive protein. It took 17 800 participants to find this advantage.⁸³ The system of payment for rehabilitation services seems unlikely to fund interventions with an effect size much below 0.4. For complex, novel neurologic rehabilitation strategies aimed at a serious disability or impairment, an intervention that offers an effect size of 0.4 (50 participants per arm of the RCT) to 0.6 (25 participants per arm of the RCT) makes practical sense. Indeed, the EXCITE, SCILT, and Hidler et al MRCTs found very clear answers when they entered this range of participants.

The demonstration-of-concept stage can be used to establish the best dose of therapy for the highest plateau in response.⁷⁶ Apparently robust interventions drawn from stage 2 can be tested in similar groups of at least 15 experimental participants, each assigned to 2 largely different doses of the new therapy, and compared with 12 to 15 affected control participants who receive an active intervention. This is a tall order for most single sites, so collaboration with another site may enable this process.

An investigator can also consider this stage, with its less than optimal sample size, as the equivalent of the number of entries necessary to complete an interim futility analysis during a fully powered RCT for an intervention that has at least a moderate effect size. Most phase III MRCTs include a plan for one or more interim analyses to ascertain whether the arms of the interventions are equally safe, whether one is clearly superior, and whether it will take many more participants than planned to detect a difference between the 2 arms because the results to date are so similar. Thinking in terms of an interim futility analysis for the first 15 participants in each arm assumes that 50 participants in each arm for an anticipated effect size of 0.4 would have provided adequate power, if a more liberal P value than the 1-sided .0025 used in drug trial futility analyses were employed. Of course, this approach may not detect a potential for efficacy of the experimental therapy in an underrepresented subgroup of the sample.

An investigator could also set up a trial in which the objective is to identify candidate interventions that show no promise of superiority against an inactive or active control therapy. One can assume that an active treatment will have a greater effect than one that does not aim to promote walking or upper extremity function. The futility test then serves as a test of nonsuperiority.⁸⁴ Within the context of an effect size of at least 0.4, two active experimental treatment arms can be compared to choose the superior one for a phase III trial design, a so-called selection trial. This paradigm, which has been applied to drug studies,⁸⁵ differs from a conventional phase II drug trial in that it does not test a specific null hypothesis, so no type 1 error can be committed. If all treatments were equally efficacious, then it would not matter which one is selected. Of course, a type 2 error may result in this scenario for a stage 3 rehabilitation study. Although the rehabilitation investigator is at risk for a type 1 or 2 error in a pilot study that enters 15 participants in each arm, a careful post hoc analysis of the data and a repeat study that accounts for any prior problems stands a reasonable chance to provide an estimate of the effect size. That information may be enough to enable a decision about going forward with an MRCT.

In addition to the already discussed goals of other stages (who to treat, how to deliver the therapy, value of a phase-in of conventional therapy to test the stability of baseline measures of function, etc), complete and transparent reporting of all results is very valuable to all who may contribute to future trials. The raw data from each subject about key baseline and outcome measures ought to be published to allow the reader to assess the variations in the data. Standard deviations of change scores, standard errors of mean changes, or 95% confidence intervals for change scores relevant to each stipulated contrast are better than group standard deviations, because these data would facilitate computation of appropriate standardized effect sizes. Also, in preparing for a possible MRCT, the investigator should look for explanations about why some participants did very well compared to those who changed little. Outliers—participants who worsened and participants who improved far more than most—are of special interest. In a pilot study, these participants can distort the results of a small group analysis.

A phase II rehabilitation study can merge into a phase III RCT with some preplanning. Phase II drug trials have been integrated into a phase III study, which shortens development time without compromising the chance of success.⁸⁶ Seamless phase II/III trials involve complex adaptations at the interim analysis, such as treatment selection, sample size reassessment, and stopping for futility. Bayesian methods can support these interim adaptations.

Trials for stage 3 require staff and funding. The Cumberland Conference¹ call for standing regional or national consortiums that have some structural support would best enable several sites to quickly complete demonstration-of-concept trials. Medicare may be another resource in the United States for covered patients. A September 2008 statement from the Medicare Learning Network Matters (Number SE0822), titled “Clarification of Medicare Payment for Routine Costs in a Clinical Trial,” discussed payment by Medicare for routine services associated with clinical trials. Certain costs of institutionally approved trials may be eligible for coverage of conventional and experimental treatments.

Stage 4: Proof-of-Concept MRCT

Phase III drug trials are a proof of concept. An MRCT, as opposed to a single site RCT, establishes generalizable efficacy of the novel intervention across sites and enables faster recruitment of participants. Stage 4 is built on the information from stages 1 to 3. That information about design, feasibility of entering enough participants within the time frame that funding allows, providing the experimental and control interventions, and measuring a relevant primary outcome and interesting secondary outcomes ought to be assured from what was learned. The MRCT can also test more than one intensity or combination of experimental and control interventions drawn from evidence in stage 3. Most important, the results of well-honed prior stages should make it reasonably likely that the experimental therapy can offer more than a small incremental increase in function. Rodent studies and drug interventions for common diseases can afford small gains. Motor-related rehabilitation interventions, however, ought to aim to find an effect size that will require no more than 50 participants with complete data per arm, as discussed. Finally, validity-of-concept is developed from the replication of the results of an MRCT. Replication offers a wider base of evidence on which to make inferences about what is true and detect biases that confound the truth.

Conclusion

Clinical investigators want to optimize their pilot studies so that the best possible MRCTs for the most likely to succeed new motor interventions are developed for disabled patients. Pilot studies can advance in stages that each acquire the information necessary to decide whether to go forward to an MRCT. By planning for the needs of these successive stages, setting goals that will signal a go versus no-go for a subsequent stage, and reporting each study fully, investigators and collaborators can advance the effort to bring the most likely-to-succeed, novel therapies for motor rehabilitation into evidence-based practice.