Rethinking phase 2 trials in amyotrophic lateral sclerosis

Abstract

There is a long history in amyotrophic lateral sclerosis (ALS) of promoting therapies based on phase 2 data, which then fail in phase 3 trials. Experience suggests that studies of 6 months in duration are too short, especially with function-based outcome measures. Multiplicity poses a serious threat to data interpretation, and strategies to impute missing data may not be appropriate for ALS where progression is always expected.

Emerging surrogate markers of clinical benefit such as reduction of neurofilament light chain levels may be better suited to phase 2 go/no-go decisions. Over-interpretation of phase 2 data, and overly optimistic communication of exploratory analyses must be avoided to ensure optimal prioritization for the investment needed for definitive phase 3 trials and to minimize the harm of false hope for people living with ALS. Delivering on advances in understanding of the neurobiology of ALS requires urgent attention to phase 2 design and implementation.

There is a long history of ALS therapies being promoted on the basis of Phase 2 data, which then fail to translate into meaningful benefits in Phase 3 trials. Benatar et al. argue that to deliver on the therapeutic promise of advances in our understanding of ALS neurobiology, we must rethink our approach to Phase 2 trials.

Introduction

Clinical trials typically progress sequentially through early, mid and late stages, often referred to as phase 1, phase 2 and phase 3. Phase 2 trials, broadly speaking, aim to gather information about safety, tolerability and dosing of an experimental treatment, and to determine whether the therapeutic agent is sufficiently promising to warrant further investigation in a large-scale, randomized, controlled phase 3 study. In the field of amyotrophic lateral sclerosis (ALS), there is a long history of identifying ‘promising’ candidates in phase 2, only for these to fail in phase 3. Moreover, depending on the design of phase 2 trials and how decisions are made whether or not to proceed to phase 3, the potential risk of discarding a drug too early should also be considered. Much has been written about the host of potential reasons why a large number of trials has led to so few effective treatments and how we, as a field, might learn from this experience.^1,2 This introspection notwithstanding, lessons from the past may not have been fully absorbed, and with a risk of stumbling into new ways to err.

Duration

Historically, phase 2 trials in ALS have varied in duration, but a 6-month placebo-controlled phase followed by an open label extension (OLE) appears to have become a common standard, e.g. the Healey platform trial.³ Whether or not a 6-month duration is sufficient depends on the chosen outcome measures, the therapeutic mechanism of action, and the anticipated efficacy of the experimental agent. The phase 3 VALOR study of tofersen, an SOD1 antisense oligonucleotide (ASO), in SOD1 ALS offers useful insights. In this 6-month trial, meaningful reductions in both CSF SOD1 protein levels (a marker of target engagement) and plasma neurofilament light chain (NfL) (a marker of the rate of axonal degeneration⁴) were observed at 12 weeks. While changes in the revised ALS functional rating scale (ALSFRS-R) directionally favoured tofersen at this time point, a significant clinical benefit was not apparent until 12 months in an integrated analysis of double-blind and OLE data.⁵ The long latency is noteworthy, given that the SOD1 ASO targets an upstream biological mechanism that is known to drive disease pathogenesis. This study showed that clinical benefit may take time to be detectable using current functional scores, even when experimental therapeutics directly target disease-causing mechanisms; longer treatment periods may be needed for clinical effects to become apparent for therapeutics that target more downstream biological mechanisms. The VALOR study also showed that a significant reduction in NfL was detected over a shorter timeframe than significant clinical effects, providing an early confidence signal, later reflected in the US Food and Drug Administration (FDA)’s ground-breaking recognition of NfL as a ‘reasonably likely surrogate marker of clinical benefit in ALS’ in their decision to grant accelerated approval to tofersen.⁶

The issue of a long latency to show clinical benefit is not unique to the ALSFRS-R. For example, the delayed separation of survival curves observed in the riluzole⁷ and pentoxifylline⁸ trials, albeit with riluzole conferring a survival advantage and the opposite for pentoxifylline, similarly illustrate that a clinical benefit or harm may only become apparent after a significant period of time (>6 months) following treatment initiation. A short study duration, therefore, risks missing potential delayed clinical effects, increasing the likelihood of a false negative result (type II error). This risk can be partially mitigated by implementing an OLE, which allows time for delayed effects to become evident during the extension phase. The analysis of OLE data, however, should be pre-specified, with an early versus delayed start paradigm permitting comparison of outcomes between those who initiated investigational product during the double-blind phase and those who first initiated during the OLE.⁹

A lengthy OLE, however, cannot fully compensate for a short placebo-controlled period. Since both randomized groups receive the same treatment during the OLE, the duration of the extension phase directly impacts the estimated between-group differences. The longer the OLE period, the more similar the groups become, which dilutes a potentially efficacious treatment effect and eventually increases the risk of a type II error. Statistical strategies have been proposed to estimate the hypothetical (counterfactual) trajectory of the originally randomized placebo arm had no active treatment been given during OLE. An example includes the rank-preserving structural failure time model,¹⁰ which estimates the overall survival for the placebo group had there been no treatment switching. Although these methods are of interest to make better use of OLE information, they are prone to bias if underlying assumptions are not met, which risks either increasing type I or II errors. These biases are, among other things, directly related to censoring (i.e. incomplete follow-up), differences between study completers and non-completers, and selection bias in those who are eligible to switch—challenges that are often encountered in ALS clinical trials and also jeopardize the value of OLE data in general.

These observations should warrant careful reflection on the optimal duration of phase 2 trials, both the placebo controlled and OLE periods. While optimal duration is unknown and will almost certainly vary based on the selected primary and secondary outcome measures, we would venture that phase 2 trials of 6 months’ duration are likely sufficient when go/no-go decisions will be driven primarily by biomarker data, but that it may be prudent to extend the duration of the double-blind phase beyond 6 months if the intent is to rely primarily on clinical outcome measures such as the ALSFRS-R. In addition, selection biases and loss to follow-up occurring in the OLE period could be minimized by combining the double-blind and extension periods into a single study protocol, removing the need to reconsent patients after switching treatments.

Outcome measures

The ALSFRS-R is an evaluator-administered instrument that assesses patient functional independence with an array of activities that reflect bulbar, fine motor, gross motor and respiratory muscle function.¹¹ It is a measure of how patients are functioning, and a slower rate of decline, or a higher score, correlates with longer survival (5). Heterogeneity in the rate of disease progression across patients, manifesting as enormous variation in the rate of change of the ALSFRS-R, typically renders phase 2 trials that rely on the ALSFRS-R, or any other current clinical outcome, under-powered. As such, changes in the ALSFRS-R are better suited to phase 3 trials that aim to address questions of clinical efficacy. Moreover, selection of the ALSFRS-R as the primary outcome measure for a phase 2 trial suggests a desire to demonstrate clinical efficacy. This is particularly challenging, given that there is no agreed upon standard as to what constitutes a minimal clinically important difference (MCID) in the ALSFRS-R,¹² or how treatment-related changes in the ALSFRS-R translate into long-term gain in overall survival. Over-reliance on the ALSFRS-R as the primary outcome measure may also tempt investigators to seek drug approval based on a single, small, phase 2 trial, as was the case for AMX0035.¹³ To the extent that directionally favourable effects on the ALSFRS-R are to be incorporated into phase 2 trials as supportive of biomarker-based readouts, strategies to control for known heterogeneity, including incorporation of blood baseline NfL and the European Network for the Cure of ALS (ENCALS) prediction score as baseline covariates, should be employed.^14,15

Judicious use of biomarkers

Since traditional clinical outcome measures such as the ALSFRS-R are invariably underpowered in phase 2, they have great potential to mislead. Perhaps not surprisingly, they have, thus far, been poorly predictive of phase 3 outcomes. There is also a growing recognition of the potential value of response biomarkers (such as NfL) as tools to aid therapy development, build a more compelling biological rationale, and better prioritize drug candidates for phase 3 trials.^4,6 While there is much that is still not known about NfL, it is undoubtedly one of the most promising candidates to date to help prioritize drugs for the significant participant burden that phase 3 study entails. With certain caveats—for example, the proposed mechanism of action of the drug, potential impact of the drug on NfL clearance mechanisms¹⁶ and potential safety effects—a significant reduction in NfL in response to an experimental agent should be taken as a promising sign. It is a matter of urgency that the definition of significant reduction be agreed upon by international consensus and pre-specified for future studies. The absence of such an effect on NfL level, or any relevant pre-specified marker of target engagement, should then inject a note of caution at least. In such situations, first critically evaluating other criteria such as drug availability at site of action and suitability of the selected dose seems more appropriate prior to proceeding to phase 3.

Missing data

Related to both trial duration and selection of the primary outcome measure, is the question of how best to deal with missing data, whether due to death, treatment interruption or loss to follow-up. Strategies such as last observation carried forward (LOCF) used in the masitinib phase 2/3,¹⁷ edaravone phase 3 trials¹⁸ and in post hoc analyses of the CENTAUR trial AMX0035 data are prone to bias given the assumption that outcome is constant following withdrawal; this is difficult to justify for diseases such as ALS, which are progressive over time,¹⁹ especially when instruments such as the ALSFRS-R are used to measure outcome. Moreover, exclusive reliance on the ALSFRS-R (when it is used as the primary outcome) is problematic insofar as analysis plans should address the potential impact of treatment on function, while accounting for mortality. This issue was well illustrated by the FDA's analysis of the CENTAUR trial data, where differential handling of deaths with accompanying missing data, led to a loss of statistical significance.²⁰

The dangers of multiplicity

Multiplicity is an important consideration in the analysis of clinical trial data. It arises when multiple significance tests are carried out, increasing the likelihood of false positive discovery (type I error). Multiplicity may arise from the use of multiple outcomes, measurement of outcomes at several time points, use of multiple doses, or reference to multiple study populations and subgroups. The problem of multiplicity is magnified when, for example, one primary, four secondary, and fifteen exploratory outcomes are specified. A total of 20 outcomes are then evaluated across multiple dosing arms, study populations and subgroups. Considering a threshold for significance of P < 0.05, conducting one statistical test has a 5% chance of false positive discovery, which increases to 40% and 64% when conducting 10 and 20 statistical tests, respectively. The fact of ‘pre-specifying’ this multitude of significance tests does not detract from the real risk of false positive discovery and erroneously advancing drugs to phase 3 clinical trials. Neither does reporting ‘nominal P-values’ when outcomes are not specified hierarchically or when there is no adjustment for multiplicity.

Interpretation and communication

Failure to adequately control for multiplicity, with the attendant risk of false positive discovery, has the potential to result in overly favorable interpretation of phase 2 trial data. Missing the primary and all secondary outcomes in the principal study population but finding a ‘hit’ in one (or several) exploratory outcomes in a sub-population, even if pre-specified, may not represent a real effect of the experimental agent. Especially when influenced by financial considerations, there is a real risk of overly optimistic interpretation and unbalanced communication of results. The dissemination of results of several recent trial results, serve as useful recent examples (Table 1^21-23). Not only is there a risk that false positive discovery will encourage large, necessarily expensive and time-consuming phase 3 trials, but patient demand for expanded access programs (EAPs) that are predicated upon the promise of nascent therapeutics. Furthermore, ‘advocacy’-led congressional appropriation of significant funds for such purposes may siphon valuable research funding away from more promising therapeutic candidates and clinical trials (a type III error).²⁴ There is also the risk that the community living with ALS is given false hope for the potential clinical benefits of these experimental compounds based on minimal data.

Table 1.

Examples of recent trial result communications

Company	Quote from press release (with context)
PharmAust21	‘The 58% slowing in ALSFRS-R decline […] clearly demonstrates the potential to provide meaningful clinical benefit to people living with MND/ALS’—In an open-label study of 12 patients.
Clene Nanomedicine22	‘Prolonged life with 49% decreased risk of death for participants […] compared to PRO-ACT matched placebo over long-term follow-up’—phase 2 data compared to historical controls
Prilenia23	‘Survival benefits from post-hoc analyses […] showed a prolongation of median survival time from ∼300 to 600 days in these participants compared to the delayed-start placebo participants’—post-hoc subgroup analysis in patients with definite or probable ALS who were also early in the course of the disease.

ALS = amyotrophic lateral sclerosis; ALSFRS-R = revised ALS Functional Rating Scale; MND = motor neuron disease; PRO-ACT = Pooled Resource Open-Access ALS Clinical Trials Database.

Go/no-go decision-making

A well-designed phase 2 trial should ideally pre-specify ‘go/no-go’ rules that will be used to decide whether there is sufficient rationale to proceed to a larger phase 3 study. Such rules are typically tied to pre-specified hypotheses that the phase 2 trial aims to evaluate. While subsidiary analysis of phase 2 trial data might legitimately also lead to the generation of new hypotheses that could be tested in a future study, a distinction should be drawn between phase 3 trials that are predicated upon a hypothesis that was tested in phase 2 versus a hypothesis that was generated based on phase 2 data. Failure to triage drugs that do not meet ‘go’ criteria at the end of phase 2 risks undermining the triage value of the phase 2 endeavour. The risks associated with predicating phase 3 trials on the results of post hoc exploratory analyses of phase 2 data are well illustrated by experience with dexpramipexole,^25,26 reldesemtiv^27,28 and NurOwn.^29,30

The over-reliance on clinical endpoints, short duration of placebo-controlled follow-up, and multiplicity, combined with overly optimistic communication, have jeopardized the ‘go’ criteria and the predictive value of phase 2 results for phase 3 outcomes. This issue was starkly illustrated by the topline negative phase 3 outcomes for edaravone,³¹ AMX0035³² and tauroursodeoxycholic acid.³³ Each of these studies was preceded by positive clinical results in short, small trials,^13,18,34 which primarily drove the decision to advance to phase 3. Whether there were signs of ‘biological efficacy or target engagement’ remained unclear, playing a seemingly subordinate role in the decision-making process at the end of phase 2. Although edaravone, in a small phase 2 study, showed highly significant reductions in CSF 3-nitrotyrosine, a marker of oxidative stress,³⁵ these results were not reported in the later studies that led to licensing. Consequently, it remains unclear whether this potential druggable target for ALS should now be abandoned. It is not clear if the negative results are due to targeting the wrong mechanism, or whether the drugs simply did not (sufficiently) interact with their intended target. The lack of biological and mechanistic information on many of the failed drugs severely limits the lessons that can be learned.

Shifting the ‘go’ decision at the end of phase 2 from a clinically oriented criterion to a more biological and mechanistic consideration may significantly de-risk phase 3 initiation. While there remains a risk that ‘go’ criteria based on biological markers, such as NfL⁴ or other mechanistic markers, will not guarantee translation to clinically meaningful phase 3 outcomes, this approach would at least provide some insights into how modifying certain pathophysiological parameters affects clinical outcomes. Ultimately, this could help to better triage drugs and prioritize viable disease mechanisms for therapeutic targeting.

Premature drug approval

There are both risks and benefits associated with drug approval based on phase 2 data. There was significant public discourse around the FDA's decision to approve AMX0035 for the treatment of ALS based on the results of the small phase 2 trial,³⁶ with the pressing need for meaningful therapeutics highlighted by patient testimony at the FDA's Advisory Committee Meeting.³⁶ A good case can certainly be made for early access to a drug with a favourable safety and tolerability profile while a confirmatory phase 3 trial is ongoing. Drug approval based on limited evidence, however, should not be conflated with proof of efficacy. One unintended consequence of prematurely concluding that a drug has clinical benefit, despite lack of effect on a promising surrogate marker such as NfL, might be to undermine the potential utility of such a biomarker for future drug screening.³⁷ Apart from approval by regulatory authorities such as the FDA, however, EAPs provide one mechanism for patients who are ineligible for ongoing trials to access experimental agents,³⁸ with an alternative being for phase 3 trials to adopt broader eligibility criteria while focusing the principal analysis on a more restricted (and homogeneous) population in which it is possible to demonstrate clinically meaningful benefit in a statistically robust manner^39-41.

Conclusions

It is often said in the ALS community that we need ‘more shots on goal’ (i.e. more clinical trials). In truth, we need ‘better shots on goal’ (i.e. clinical trials more likely to identify an effective therapeutic). This will undoubtedly require a better understanding of the underlying biology of disease, which in turn will yield more informed selection of therapeutic targets. In addition, however, we need to make better use of the triage value of phase 2 trials, selecting for phase 3 investigation only the most promising candidates. To this end, phase 2 trials should rely more heavily on biomarkers rather than clinical instruments to measure outcome. To the extent, however, that sponsors and investigators continue to rely on clinical outcome measures, it may be prudent to extend the duration of phase 2 trials. Primary and secondary outcomes should be pre-specified, alongside statistical analysis plans that more seriously consider the risk that multiplicity may lead to false positive discovery (Box 1). Those living with ALS need hope, but false hope has moral harms.⁴² The hope offered must firmly be founded in rigorous, unbiased peer-reviewed scientific and clinical trial data.

Box 1. Action plan for phase 2 trials.

1. Incorporate the following design elements in phase 2 trials:

   1. Place greater reliance on biomarkers of target engagement and pharmacodynamic effect (e.g. NfL) as primary study objectives instead of clinical measures of efficacy such as the revised Amyotrophic Lateral Sclerosis Functional Rating Scale (ALSFRS-R).

   2. To the extent that phase 2 trials continue to rely on clinical outcome measures, the duration of the double-blind phase should be extended beyond 6 months, with optimal use of prognostic clinical [e.g. European Network for the Cure of ALS (ENCALS) prediction score] and biomarkers (e.g. NfL) to adjust for known sources of heterogeneity.

   3. Evaluate the appropriateness of dose, potentially including additional randomized arm(s).

   4. Protocolize double-blind and open-label extension phases into a single protocol.

2. Clearly pre-specify go/no-go decisions for phase 3 initiation:

   1. Pre-specified primary and secondary measures, with appropriate adjustment for multiplicity when used for decision-making.

   2. Pre-specified statistical analysis plan before unblinding of data and the exploration of subgroup analyses when used for decision-making.

3. Provide the ALS community with more measured descriptions of phase 2 trial results to minimize the potential for false hope and maintain trust through an objective, data-driven appraisal of trial results.

Funding

M.B. is supported by U54-NS092091 from the US National Institutes of Health (NIH).

Competing interests

M.B. reports consulting fees from Alector, Alexion, Annexon, Arrowhead, Biogen, Cartesian, Denali, Eli Lilly, Horizon, Immunovant, Novartis, Roche, Sanofi, Takeda, UCB, and uniQure. The University of Miami has licensed intellectual property to Biogen to support design of the ATLAS study. C.J.M. reports consulting fees from Novartis, Biogen, Verge, PTC Therapeutics, Amylyx, Ferrer. M.R.T. reports salary support from the Motor Neurone Disease Association; royalties or licenses from Oxford University Press, Oneworld, and Karger; speakers’ honoraria from University of Miami; and participation in advisory board for Biogen and Novartis. R.v.E. reports no conflicts.