Unfounded Claims of Improved Functional Outcomes Attributed to Follistatin Gene Therapy in Inclusion Body Myositis

Main Text

A recent study by Mendell et al.¹ in Molecular Therapy claims to have demonstrated clinical and biomarker efficacy for inclusion body myositis (IBM) from follistatin gene therapy. Although the authors are to be congratulated for performing a long and difficult study, its design could not possibly support this claim. Additionally, the publication reports a different primary outcome measure than the ClinicalTrials.gov registered primary outcome measure, uses post hoc analyses that bias efficacy evidence, presents safety data in a confusing manner, and misrepresents published IBM literature.

The study is an analysis of selected data obtained in one clinical trial² combined with data obtained from a neuromuscular clinical practice and analyzed using a post hoc-defined primary outcome measure (Figure 1A). The clinical trial is a phase 1A, open-label, single group assignment (there was no comparator or “control” group) study of 15 patients, 9 with IBM and 6 with Becker muscular dystrophy (BMD). Three IBM patients received unilateral quadriceps dosing and are not discussed. Analysis of 6 patients with BMD participating in the trial was previously published in Molecular Therapy.³

Figure 1.

Study Design Does Not Allow for Conclusions of Efficacy for Follistatin Gene Therapy

(A) Study design and its relationship to clinical trial NCT01519349. (B) Hypothetical design in which 3 of the 4 interventions are controlled through blinding and use of a comparator group. 6MWT, 6-min walk test

This new study reports on the remaining 6 IBM patients.¹ They received at least 4 potentially therapeutic interventions: follistatin gene therapy (AAV-FS344) into bilateral quadriceps muscles, high-dose prednisone for approximately 60 days, a prescribed and monitored exercise program, and the well-known placebo effects that come from both participation in a clinical study alone and the receipt of open-label candidate therapies with intended clinical efficacy. In addition, the authors use aggregate 6 min walk test data from 8 IBM patients drawn from a neuromuscular clinic as a comparator to make the claim that follistatin gene therapy has clinical efficacy.

However, it is impossible for the authors to make that conclusion. Because the “treated” group received 4 possibly therapeutic interventions and the “control” patients were not matched for any of these interventions, it is impossible to attribute any outcome differences between the two groups to any specific intervention. A hypothetical design that might have allowed such a conclusion is outlined in Figure 1B; such a design is typically reserved for phase 2 studies. The authors present circumstantial arguments as to why they attribute the apparent clinical efficacy to follistatin gene therapy rather than prednisone or exercise therapy: “A question could be raised regarding efficacy entirely related to exercise, but we believe this to be highly unlikely given the failure of exercise alone (including 10-m and 30-m walk, timed-up-and-go, stair climbing) to improve function in the absence of follistatin therapy.” However, they neglected to cite a study⁴ that did show statistically significant benefits from exercise for patients with IBM using 30-min walk time and stair climbing outcome measures.

Furthermore, the study did not control for placebo and related effects. Participants (patients and investigators) in this clinical trial were aware of the use of an intended therapeutic candidate based on cutting-edge science in an otherwise relentless progressive disease. Patient performance and its measurement, theoretically enhanced by placebo effects, was compared with patients from a neuromuscular clinic who had expectations of continued decline and whose performance and its measurement were theoretically reduced by nocebo responses. Empirically, placebo responses in IBM are readily apparent in several published IBM double-blind randomized clinical trials, and their magnitiude may exceed that seen in the current study (A.A. Amato et al., 2016, American College of Rheumatology Annual Meeting, abstract).5, 6

The publication states that “The primary outcome for this trial was distance traveled for the 6-min walk test”, yet the trial registration indicates its primary outcome measure is “Safety trial based on development of unacceptable toxicity defined as the occurrence of any Grade III or higher treatment-related toxicities Time Frame: 2 years.” Another publication regarding the BMD patients in this trial makes a similar contradicted claim (“The distance walked on the 6MWT was the primary outcome measure”).³ The publication of clinical trial post hoc outcome measures reported to be clinical trial primary outcome measures is a common practice in the medical literature that undermines study validity.7, 8, 9, 10, 11 The ClinicalTrials.gov database was in part created to discourage such outcome measure alterations.¹² Peer-reviewers often do not recognize,¹³ and manuscripts appear to not be more likely to be rejected for, outcome measure alteration.¹⁴

The investigators used a post hoc created “annualized 6MWT” outcome measure. Whereas a fixed time-point outcome measure, such as the 6-min walk test (6MWT) distance at 52 weeks, is a real objectively defined measure, an “annualized 6MWT” is an imaginary one that will change based on what time-point to be annualized is chosen for each patient. Analysis of data from the 251-person randomized, double-blind, placebo-controlled phase 2/3 bimagrumab IBM trial (A.A. Amato et al., 2016, American College of Rheumatology Annual Meeting, abstract) demonstrates that an “annualized 6MWT” is an invalid measure, critically dependent on the time-point used for annualization. For example, the results of an “annualized 6MWT” for the 3 mg/kg cohort are 43 meters in favor of bimagrumab 3 mg/kg based on 6-month annualization, but 3 meters in favor of placebo based on 16-month annualization.

The handling of subject 3 illustrates an additional pitfall in the use of an “annualized 6MWT.” His 23-m 6MWT improvement at 2 months was “annualized” (multiplied by 6.5), projecting into the future an imaginary 6MWT improvement of 138 m attributed to follistatin gene therapy and used as part of a circular argument for AAV-FS344 efficacy. The use of the imaginary value of 138 m assumes that AAV-FS344 would have had continuing efficacy from months 2–12 for this subject, while the resulting value of 138 m forms part of a dataset that is then used to conclude AAV-FS344 has clinical efficacy.

Additionally, the analysis of the annualized 6MWT compared data from 6 patients with IBM with a post hoc chosen comparator group of 8 patients from a neuromuscular clinic and reported a p value of 0.01. The p value for this comparison depends on the size of this comparator group and the justification for choosing N = 8 to compare to the N = 6 group is not provided. A smaller comparator group would generally result in a larger, less significant p value.

The frequency of adverse outcomes (AEs) data presented in the text and Table S2 is misleading. The text states “Unrelated AEs occurred in two subjects (6%), characterized as falls, and one subject (3%) had a biking accident,” yet 2 of 6 IBM patients is 33%, not “6%,” and 1 of 6 IBM patients is 17%, not “3%.” Similar percentages of other adverse events in Table S2 do not reflect the frequency of those adverse events among patients in this trial. These rates appear to be the frequency of a particular adverse event among all adverse events. Thus, the “6%” fall rate appears to have been calculated with a numerator of 2 (reflecting the number of adverse event falls) and a denominator of approximately 32 (reflecting the total number of adverse events of all types). This is an unprecedented method for reporting frequencies of safety data and highly misleading. The frequency of falls in this trial was 33% of the patients, and, without data indicating this was not excessive, the publication’s conclusion that follistatin gene therapy is “unequivocally safe” is premature. The published analysis is an unstated interim analysis at a post hoc chosen time-point. The trial is stated to have safety and efficacy endpoints at 2 years, but only 2 subjects have 6MWT data presented at 2 years in this publication. Three subjects (4, 5, and 6) appear to have 8–12 more months of ongoing safety and efficacy data collection. FDA guidance has noted that “any interim analysis that is not planned appropriately…may flaw the results of a trial.”¹⁵

The publication presents biomarker data comparing pre- and post-AAV-FS344 injection muscle biopsies interpreted as demonstrating that AAV-FS344 reduces muscle fibrosis and related molecular markers. First, attribution of biomarker efficacy to AAV-FS344 is impossible because of trial design for reasons indicated above. Second, the injection sites, and hence the post-treatment biopsy sites located at injection sites, were purposefully selected based on magnetic resonance imaging (MRI) guidance as areas of reduced fibrosis. The data simply confirm that there was less fibrosis in post-treatment muscle that was selected ahead of time to have less fibrosis. The supplemental methods indicate that, in the analysis of follistatin copy number in post-treatment muscle biopsies, subject 2 had no detectable FS344 DNA present. The publication explains that this was “because of sampling error.” What is the basis for the definitive conclusion that the lack of FS344 DNA was due to sampling error, as opposed to failure of the pharmacological mechanism (e.g., drug delivery, myofiber transduction)?

The publication indicates a “benefit of a combined gene delivery of follistatin with a muscle contraction exercise program” based on the ad hoc observation that “all patients in this study improved in outcome measures, but the difference was striking between the exercise cohort (subjects 2, 4, 5, and 6) and the non-exercise cohort (subjects 1 and 3) while the dose of rAAV1.CMV.huFS344 remained the same for all participants.” This is an error based on assessing greatest improvement by final 6MWT (which has varied time points, from 2–24 months), not by the “annualized 6MWT.” In fact, subject 4 in the “exercise cohort” had the second smallest “annualized 6MWT” (50 m), and subject 3 in the “non- exercise cohort” has the second largest “annualized 6MWT” (138 m).

Lastly, the publication concludes with the claim that “… this is the first clinical trial to show clear evidence of a treatment benefit in sIBM.” Not only does the NCT01519349 trial design preclude demonstration of a “clear” benefit of follistatin gene therapy, but this claim neglects to note other publications that have, invalidly positioning itself as the “first.” Two high-quality randomized double-blind placebo-controlled trials (oxandrolone¹⁶ and bimagrumab⁶) and one open-label randomized controlled trial (anti-thymocyte globulin plus methotrexate compared to methotrexate alone¹⁷) have previously shown evidence of treatment benefit in IBM. The proper positioning of the reported results¹ is with other open-label, single group assignment clinical trial studies that lack adequate comparator groups for claims of efficacy and, instead, incorporate non-trial data as comparators.