ALS/SURV: A modification of the CAFS statistic

Abstract

We present a composite endpoint that can be used in amyotrophic lateral sclerosis (ALS) trials, which combines functional status (via the ALS functional rating scale) and survival, denoted ALS/SURV. ALS/SURV modifies and extends the combined assessment of function and survival (CAFS) score and assigns rankings to participants that withdraw or are lost to follow up in a way that does not disproportionately lower and skew ranks for those participants that reach study endpoint (either death or study completion). ALS/SURV has properties of: (1) ordering participants that completed the study from the shortest surviving participant to the last observed death followed by worst function to best function; (2) ordering participants withdrawing at time of withdrawal by their decline in functional status relative to all the participants still in the study; and (3) then maintaining this ordering at time of withdrawal relative to participants still in the study. These properties allow ALS/SURV to better account for participant drop out compared to CAFS. We derive and compare the rankings of participants from the ceftriaxone treatment trial for ALS/SURV and CAFS and demonstrate that ALS/SURV does not modify the ordering of participants that complete a study by the results of participants who withdraw. Additionally, ALS/SURV can be summarized as either median functional status or median survival along with interquartile range, thereby adding clinical meaning to the statistic. Finally, by applying normal deviates, confidence intervals can be computed and used to estimate power for future studies. In summary, the above properties support the role for ALS/SURV as a new ALS composite statistic.

Introduction

Survival is an important outcome in amyotrophic lateral sclerosis (ALS) trials, but studies that use survival as the primary outcome can be long and costly. The Revised ALS Functional Rating Scale (ALSFRS-R)¹ offers an alternative endpoint to enable shorter duration studies.² Moreover, combining survival and functional status into a single statistic may increase the power to detect a treatment effect, thereby reducing trial duration and/or sample size and cost.³ One such measure in ALS is the combined assessment of function and survival (CAFS),⁴ used in ceftriaxone⁵ and dexpramipexole⁶ studies, and derived from Finkelstein and Schoenfeld.⁷ CAFS enables analysis of participant data when ALSFRS-R scores are not available due to death and has increased power, compared to an analysis of ALSFRS-R alone, in the setting of a marked survival benefit.

We propose an alternative to CAFS that combines ALSFRS-R and survival (ALS/SURV) in a manner that is easy to calculate and interpret clinically, and uses study windows, to calculate confidence intervals. A critical distinction of ALS/SURV from CAFS is that ALS/SURV assigns a score to participants that are lost to follow up at the time of withdrawal in a manner that the withdrawals do not affect the ordering of the participants who complete the study (completers), where withdrawals in CAFS can cause a reordering of completers. We recently used ALS/SURV in a post-hoc analysis of outcomes from Phase 1 and 2 participants who underwent intraspinal stem cell transplantation compared to historical datasets.⁸ In this report, we present a comparison of CAFS to ALS/SURV and show that these two summary statistics yield similar results except for participants who have a short follow up; i.e., withdrew from the study or were lost to clinical follow up. In the development of ALS/SURV, we also considered two revisions to the CAFS statistic that reduce this disparity between the two statistics for early withdrawals. Ultimately, we show that ALS/SURV is similar to CAFS, but by incorporating visit windows, it yields clinically interpretable results and allows the calculation of confidence intervals and estimation of power.

Methods

Development of ALS/SURV

The ALS/SURV composite statistic is calculated using a set of windows which would normally correspond to clinical visits up to the end of the trial. It is computed from either (1) ALSFRS-R decline if the participant is alive in the window or lost-to-follow-up, or (2) time to death. A ranking procedure assigns scores to all the participants at each window. ALS/SURV is visually illustrated in Figure 1.

Figure 1. Overview of ALS/SURV.

ALS/SURV ranks and scores each participant, relative to all other participants, at the last observation in a visit window. Ranks are assigned from 1 (worst) to N (best) and can be performed at each visit window. Scores are ordered numerically and are calculated as follows: the participant’s rank (r) is divided by N+1 to bound the scores from 0 to 1. For a missed visit, ALSFRS-R can be interpolated if performed at the visits before and after. In this example, all participants can be scored at visit 1. At visit window 2, participants B, C, and D cannot be scored due to missing ALSFRS-R without a measure at the next time window; ALSFRS-R interpolation is performed for participants E and F; and participant’s A death is scored. At visit window 3, participants E, F, and G completed the study and participants’ A and D deaths are scored. The table shows scores and ranks of each participant at each window. Shaded cells indicate a carried forward rank due to the participant missing a visit window. Although this results in duplication of scores, it ensures that the ranks of completers are not skewed by withdrawing participants.

The ALS composite statistic was developed using the following criteria: (A) If N is the number of participants who complete the study [D who die and (N-D) who have a final assessment], they are ordered from 1 to D for those who died, where 1 is the first to die and D is the last to die, and from D+1 to N where D+1 has the largest decline in ALSFRS-R and N the least decline, similar to CAFS. (B) Participants who withdraw at the same time period are ordered by their functional status, either ALSFRS-R score or decline in ALSFRS-R. (C) Since ALSFRS-R declines over time, participants followed for longer periods are expected, on average, to have larger declines in ALSFRS-R than participants observed for shorter periods. Therefore, those withdrawing in time window t are ordered with respect to results of all other participants in that window (as the window was the end of the study).

The simplicity of the calculation comes from creating a combined scale of time to death and ALSFRS-R. In this example, time to death (TD) is recorded in years (with 3 decimals, enabling differentiation between participants). In each window, X is the decline in ALSFRS-R or −100+TD if the participant died prior to the visit (100 is selected so that values do not overlap with decline in ALSFRS-R). Therefore, every participant who has not withdrawn has a value for X. The values of X for those who died will be very small (−100 to −90) and ordered by time of death. The values of X for those still alive (ALSFRS-R decline or ALSFRS-R) are ordered from worst to best and are greater than those who died. Therefore, rankings in each window are: first to die will have the smallest X and receive the rank 1; last to die will have a lower rank than the participant with the most negative ALSFRS-R decline; and least negative ALSFRS-R decline will have the highest rank.

If a participant died before the end of the window, then death was the outcome even if ALSFRS-R had been assessed earlier in the window. If a participant missed a visit but had a subsequent evaluation, the value for the missed window was estimated by linear interpolation of the adjacent ALSFRS-Rs. A score was not assigned when the participant missed the visit and died after the window closed; the participant would be considered a withdrawal for this visit. The participant would then be scored at subsequent visits after the time of death. Since the scores of ALS/SURV are evaluated in every window, groups can be compared at any study window.

The number of participants ranked at each visit will change due to withdrawals. Therefore, instead of using the rank (r) as the assigned value, it is replaced by a score: r/(N+1) where N is the number of participants ranked at the visit (+1 in the denominator makes 1 and N equidistant from 0.5). Over time, N will decrease as withdrawals increase but the scores will remain in the range (0,1) and symmetric about 0.5.

At each window and at study conclusion, the last assigned score is used for each participant. Withdrawals are given the assigned score at the last visit at which they were evaluated. Consider if the assigned score to a withdrawal was ¼; at the time of withdrawal approximately 25% of the participants had worse outcomes and 75% had better outcomes. Since the participant remains with the score ¼, the participant has the same ordering relative to all the participants as he/she had when withdrawn.

At each window and at study conclusion, all participants receive a rank score. Using these rank scores, the treatment groups can be compared and confidence intervals can be computed. The rank scores corresponding to the end points of the confidence intervals can be converted back to either decline in ALSFRS-R or survival time.⁸

Although we are using r/(N+1) as the score for ALS/SURV, other functions of the rank can also be used. One alternative is to use the normal deviate (z-score) corresponding to r/(n+1). When the z-score is used, the difference between groups can be used as the expected difference to design future studies; i.e., to estimate power for different sample sizes.

Comparison of ALS/SURV to CAFS

Data from the ceftriaxone trial (provided by Northeast ALS Consortium)⁵ were used to compare ALS/SURV and CAFS. This study occurred between 2006–2012 with primary outcomes of survival and difference in ALSFRS-R slope at 1 year; there was no treatment benefit.⁵ To simplify the comparison we defined clinical visits as occurring at 6-month intervals with a maximum follow up of two years. We extracted ALSFRS-R values that fell within a window of ±0.1 months from the visit date, the smallest interval of time recorded in the data file. Duration to death was recorded to an accuracy of 0.1 years. Therefore, a 6-month window was constructed as 6±0.15 months (5.9 to 6.1 months). We used the same assessments at these visits to evaluate CAFS. We evaluated both CAFS and ALS/SURV as if the study duration was planned to be 24 months. Therefore, ALS/SURV and CAFS⁴ were evaluated at 6, 12, 18, and 24 months. For the purpose of comparing statistics, we computed the score and then ranked the scores from worst (1) to best (N). We demonstrate the similarities and differences between the rankings of participants utilizing ALS/SURV and CAFS both graphically and via Spearman correlations.⁹

As CAFS does not specify handling of missing data and analysis of data in visit windows, we also considered two modifications to CAFS. Under the assumption that CAFS compares participants in time periods (visit windows), as opposed to time points, we derived alternative 1, denoted as CAFS-alt1, which eliminates differential loss to follow up bias by comparing the decline of ALSFRS-R between participants at the same time intervals. CAFS is altered as follows: T1 and T2 represent the time intervals that two participants are followed and participants are compared at time T, the smaller of T1 and T2. If one of the participants died at time T, they are assigned a −1 and the surviving participant a +1. Otherwise, the two participants are compared at time T based on their declines from baseline. This eliminates bias due to unequal time of follow up in instances of trials with varying follow up times and for participants who withdraw or are lost to follow up. This modification is challenging to implement when participants do not have ALSFRS-R measurements at the same time intervals but interpolation can be used to estimate the ALSFRS-R at each time point.

Alternative 2, denoted CAFS-alt2, addresses the criterion that all the participants who complete the study should be ranked relative to their last observation (either time to death or ALSFRS-R decline). To calculate CAFS-alt2, all completers are scored relative to each other. Thus, if C participants complete the trial, the scores -(C-1) to +(C-1) are assigned to the completers. Next, withdrawals are compared to completers at the time point of withdrawal, but the scores of the completers remain unchanged. That is, the withdrawals receive a score of +1, 0 or −1 for each comparison to the C completers using the method of comparison described for CAFS-alt1, but the score for the completer is not changed. As a result, the withdrawals can receive a score from -C to +C, although the extremes are unlikely. Since completers are not affected by the scores given to withdrawals, the ordering of the completers is unchanged and fulfill criterion (A) above, that all participants completing the study are ranked from 1 to D for those who die and D+1 to N for those who complete the study. Withdrawals at the same time point are ordered correctly since the one who has a larger decline receives a lower (or possibly equal) score to the one with a lesser decline; Equality occurs only if there are no observed values for the completers between or equal to one of the withdrawals; since withdrawals are scored only against completers, two withdrawals have equal scores when there are no ALSFRS-R declines for completers ranked between them.

Conceptually, ALS/SURV works sequentially through the visit windows to the end of study, assigning rank scores to all available participants at each time and carrying forward the rank scores of withdrawals. CAFS, on the other hand, uses the last values, either survival or ALSFRS, depending on the comparison. CAFS-alt1 performs partial computations in each window comparing all participants to withdrawals in that time interval and keeps summing over windows until the end of the study. CAFS-alt2 first calculates the CAFS statistic for all the completers including those who died; it then goes back to all the windows to compute the statistic for withdrawals relative to completers at the time of withdrawal.

Results

Participants

Nine participants who did not die and did not have any ALSFRS-R values within a visit window were deleted, leaving data on 504 participants. The scores obtained by each method were then ranked from 1 to 504.

Comparison of CAFS to ALS/SURV

The rankings for CAFS, CAFS-alt1, and CAFS-alt2 compared to ALS/SURV are shown in Figures 2a, 2b, and 2c, respectively, stratified by withdrawals and completers. Comparing CAFS to ALS/SURV (Figure 2a), the rankings for participants withdrawing at 6 months by CAFS are much higher than ALS/SURV, resulting from the shorter available time for ALSFRS-R to decline compared to participants withdrawing at 12 and 18 months. In turn, rankings of completers are lower in CAFS versus ALS/SURV. This is further visually illustrated by the lack of a monotonic line relating the ranks of the ALS/SURV to CAFS completers, which indicates the differing rankings between these methods.

Figure 2. Comparison of ALS/SURV to CAFS and its alternatives.

For all comparisons, an observation falling on line with a slope = 1 indicates that the two composite statistics ranked participants equally; an observation falling above a line with slope = 1 indicates that the comparison ranked participants higher compared to ALS/SURV; finally, an observation below the line with slope = 1 indicates that the ALS/SURV ranked participants higher. (A) CAFS versus ALS/SURV. Participants withdrawing at 6 months (blue circles) are ranked higher in CAFS compared to ALS/SURV as these participants had less observation time compared to other participants, but were ranked based on data at the end of the trial. This is also the case for participants withdrawaling at 12 and 18 months, but not as extreme. Participants completing the trial were on a monotonic line until about ALS/SURV rank = 200 and then deviate below this line as the withdrawals were assigned extreme scores compared to completers under CAFS. (B) CAFS-alt1 versus ALS/SURV. Using CAFS-alt1, a withdrawing participant is compared to all others at their time of withdrawal. Thus completers are more monotonic, but the spread about line with slope = 1 for completes is scattered as withdrawals are interspersed with completers. (C) CAFS-alt2 vs ALS/SURV. Under CAFS-alt2, ranks of completers are not modified by withdrawals, therefore, the completers line is monotonic with most falling on a line with slope = 1. (D) ALS-R/SURV versus ALS/SURV. In this case, the variability of those participants withdrawing is related to using the ALSFRS-R score instead of the ALSFRS-R decline.

mo, month; WD, withdrawal.

Comparing CAFS-alt2 to ALS/SURV rankings (Figure 2c) shows a more monotonic line, which indicates that the ranks of completers are not modified by the withdrawals. This finding contrasts to the comparison of CAFS-alt1 to ALS/SURV rankings (Figure 2b) showing a more linear relationship between the two statistics, but more spread around the line as a result of the rankings of the withdrawing participants being interspersed with the completers which disproportionately affects the rankings for the completers in CAFS-alt1 versus CAFS. Also, withdrawing participants have lower relative ranks in CAFS-alt1 and CAFS-alt2 than for CAFS because CAFS-alt1 and CAFS-alt2 perform comparisons at time of withdrawal.

Comparison of ALS/SURV to ALS-R/SURV

Added flexibility of this method allows the absolute ALSFRS-R score to be used in place of the ALSFRS-R decline--denoted as ALS-R/SURV--used in an analysis of ALS participants from an open-label Phase 1 and 2 trial of intraspinal stem cell transplantation.⁸ ALS-R/SURV plotted against ALS/SURV is shown in Figure 2d. As expected, all the deaths are ranked similarly and therefore the line is monotonic for the lower rankings. However, ALSFRS-R score are not ranked in the same order as declines in ALSFRS-R and therefore the withdrawals are spread above and below the line for completers.

Spearman Correlations

Spearman correlations between rankings of each composite statistic for all the participants (N=504), for participants who reached the endpoint (had a value at 24 months or died, N=353), and for participants who withdrew (N=151) are shown in Table 1. Correlations for all participants (Table 1A, top panel) are highest between CAFS-alt1 and CAFS-alt2 since they only differ by whether the withdrawals affect the scores assigned to the completers. Correlations for completers (Table 1B, middle panel) are close to 1; correlation between ALS/SURV and CAFS-alt2 is 1 because both methods rank (score) the outcomes of the completers without adjusting the completers’ scores by their relationship to the withdrawals. Although CAFS has a high correlation with ALS/SURV among the withdrawals (Table 1C, bottom panel), there is a large difference in the rankings that are associated with the withdrawals (Figure 2a).

Table 1.

Spearman correlations of ranks utilizing the 5 composite methods

A. All observations, N = 504
	ALS/SURV	ALS-R/SURV	CAFS	CAFS-alt1	CAFS-alt2
ALS/SURV	1	0.949	0.827	0.891	0.922
ALS-R/SURV	0.949	1	0.823	0.843	0.883
CAFS	0.827	0.823	1	0.556	0.640
CAFS-alt1	0.891	0.843	0.556	1	0.977
CAFS-alt2	0.922	0.883	0.640	0.977	1
B. Participants who completed the study, N = 353
	ALS/SURV	ALS-R/SURV	CAFS	CAFS-alt1	CAFS-alt2
ALS/SURV	1	0.987	0.999	0.992	1
ALS-R/SURV	0.987	1	0.986	0.981	0.987
CAFS	0.999	0.986	1	0.995	0.999
CAFS-alt1	0.992	0.981	0.995	1	0.992
CAFS-alt2	1	0.987	0.999	0.992	1
C. Participants who withdrew, N = 151
	ALS/SURV	ALS-R/SURV	CAFS	CAFS-alt1	CAFS-alt2
ALS/SURV	1	0.812	0.917	0.824	0.843
ALS-R/SURV	0.812	1	0.784	0.684	0.695
CAFS	0.917	0.784	1	0.624	0.643
CAFS-alt1	0.824	0.684	0.624	1	0.998
CAFS-alt2	0.843	0.695	0.643	0.998	1

Mann-Whitney two-sample test comparing active to placebo in ceftriaxone shows all non-significant p-values (0.48 for CAFS-alt2 to 1.00 for CAFS), as expected.

Discussion

A composite statistic for ALS trials is valuable because it incorporates all available and meaningful clinical data,⁴ including the “net benefit”¹⁰ of survival and functional outcomes, addresses missing data and multiple comparisons concerns,^{3, 4} and increases the data points or events available for analysis.^{3, 11} Importantly, a composite statistic is more powerful than using only survival or only decline in ALSFRS-R when both survival and decline are affected by treatment because there is a positive correlation between the two measures. We show that ALS/SURV extends CAFS by assigning rankings to withdrawing participants in a way that does not skew ranks of completers [those participants that reach the study endpoint (either death or study completion)]. Given the high correlation of the methods discussed in this paper, all these methods will have a similar power profile as described for CAFS.⁴

Unlike ALS/SURV, CAFS and CAFS-alt1 skew ranks since the last observation is used when participants withdraw, without adjusting for differences in observation time. To illustrate, consider two participants who die at the same time, but the last ALSFRS-R of one showed a minimal decline and of the other declined to close to zero. If a participant who withdrew has an ALSFRS-R that falls between the last ALSFRS-R scores of the participants who completed the study, the CAFS scores of the completers will be modified, creating greater distance between the two scores. If enough withdrawals fill this gap, completers will have greatly different scores and likely be ranked differently relative to other completers. Essentially this means that the scores of participants who complete the study are modified by their comparison to withdrawals. CAFS-alt2 prevents the ranks of completers from being affected by results of withdrawing participants, but it may be difficult to calculate due to the need to compute the statistic for each withdrawing participant at the time of withdrawal.

The ALS/SURV has certain advantages compared to CAFS: (1) it orders the expected outcomes from worst (shortest time to death) to best (lowest ALSFRS-R decline) for all participants who complete the study; (2) rankings of withdrawals are compared against all the other participants at the time of the last observation of the withdrawal and this ranking is carried forward, thereby adjusting for observation time; (3) participants are analyzed using a rank score instead of computing pairwise comparisons; (4) groups can be compared at every time point; and (5) the results (when z-scores replace the rank score) can be used to estimate power for future studies.

These distinctions are important because trials have a fixed follow up time or a fixed end date. For a fixed end date, participants recruited initially have a longer follow up duration, versus those recruited last, and therefore have a higher likelihood of dying or experiencing a larger decline in ALSFRS-R compared to those recruited later. Randomization should distribute this bias to all treatment groups, thereby not affecting the validity of the data analysis of the study. However, if differential loss to follow up occurs, such as participants in one group discontinuing due to side effects or competing risks from disease-related dropouts,¹² the unbalanced early terminations can create bias since those who terminate early will have, on average, a smaller decline in ALSFRS-R than those who did not withdraw. This bias favors the group with more withdrawals, and leads to higher rankings in this group with CAFS.

ALS/SURV and ALS-R/SURV differ in whether ALSFRS-R decline or the absolute ALSFRS-R score is used to rank the participants. While ALSFRS-R represents the functional status of a participant and provides context to the patient’s current condition, the decline in ALSFRS-R is a marker of progression and may be more sensitive to therapeutic change. Therefore, the decision of whether to use the ALS/SURV or the ALS-R/SURV will be study dependent, will vary by study design and whether the desired outcome represents progression/survival or functional status/survival. Factors that may influence this choice include study duration, baseline mean ALSFRS-R score standard deviation, and presence of floor effects in the population. ALS-R/SURV also provides a clinically meaningful interpretation since the median ALSFRS-R score can be provided at each visit window. In the analysis of ALS participants from an open-label Phase 1 and 2 trial of intraspinal stem cell transplantation,⁸ the rationale to use ALS-R/SURV was that death is associated with ALSFRS-R values rather than with the decline in ALSFRS-R values. Therefore, the ranking of withdrawals would implicitly reflect disease severity at that time point rather than the decline in values to the time point. Additionally, the composite statistic provided meaningful estimates of the status of the study population at each window. Since the statistic (rank) can easily be converted back (reversed) into decline of ALSFRS-R or survival time, it and its 95% confidence interval can be converted into clinically interpretable outcomes.

The two statistics, CAFS and ALS/SURV, are highly correlated and have similar properties with respect to the evaluation of clinical trial outcomes when there are few subjects who are lost to follow up. However, ALS/SURV compares results from withdrawals to those of subjects in the study for the same length of time; CAFS comparisons for withdrawals may be biased by the length of the study. ALS/SURV also provides comparisons between groups at each time point within a study where CAFS provides only a final comparison. Another major difference between the two statistics is that ALS/SURV can easily provide power estimates to plan future studies; this is not possible with the CAFS statistic. Thus, we contend that ALS/SURV extends the utility of CAFS.

As noted in the simulations performed for CAFS,⁴ a statistic that combines ALSFRS-R and survival will have more power than a test based on only ALSFRS-R or survival when the intervention affects both survival and progression. This is the desired effect for most interventions--to improve both survival and functional status; however, while initial ALSFRS-R scores and/or slopes are associated with survival status,^13–15 future ALSFRS-R scores do not always correlate with survival during disease course.¹⁶ Thus, a potential limitation of the CAFS and ALS/SURV is that the composite statistic preferentially uses ALSFRS-R over survival even though it may not be the most instantaneous significant outcome. The recently introduced use of joint modeling for ALS studies,¹⁷ compared to ALS/SURV and CAFS, better accounts for the simultaneous effects of survival and function, thereby addressing this limitation in CAFS and ALS/SURV.

In summary, the ALS/SURV, a modification of CAFS, is an easy to calculate ALS composite statistic that provides a clinically interpretable result, maintains the rankings of completers in the presence of withdrawing participants, and enables the calculation of confidence intervals and estimation of power. ALS/SURV is a new composite statistic available for ALS trials.

Biography

Stephen A Goutman, MD, MS / Department of Neurology / 1500 E Medical Center Dr / Ann Arbor, MI 48109-5223 / Phone: 734-936-8586 / Fax: 734-936-5185 / e-mail: sgoutman@med.umich.edu / ORCID ID: 0000-0001-8780-6637

Dr. Goutman is a clinical associate professor of neurology at University of Michigan, director of the multidisciplinary ALS clinic, and associate director of University of Michigan’s ALS Center of Excellence

Morton B. Brown, PhD / M4039 SPH II / 1415 Washington Heights / Ann Arbor, Michigan 48109-2029 / e-mail: mbb@umich.edu

Dr. Brown is a professor emeritus of biostatistics at University of Michigan

Merit Cudkowicz, MD / Massachusetts General Hospital / Neurological Clinical Research Institute (NCRI) / 165 Cambridge street, suite 600, Boston, MA 02114 / e-mail: mcudkowicz@mgh.harvard.edu

Dr. Cudkowicz is the Chief of Neurology, and Director of the Neurological Clinical Research Institute at Massachusetts General Hospital and the Julieanne Professor of Neurology at Harvard Medical School.

Nazem Atassi, MD, MMSc / Massachusetts General Hospital / Neurological Clinical Research Institute (NCRI) / 165 Cambridge street, suite 600, Boston, MA 02114 / e-mail: NATASSI@mgh.harvard.edu

Dr. Atassi is an Associate Professor of Neurology at Harvard Medical School, Co-Director of the Neurological Clinical Research Institute at Massachusetts General Hospital.

Eva L. Feldman, MD, PhD / University of Michigan / 109 Zina Pitcher Place / 5017 AAT-BSRB / Ann Arbor, MI 48109-2200 / Phone: 734-763-7274 / Fax: 734-763-7275 / Email: efeldman@umich.edu

Dr. Feldman is the Russell N. DeJong Professor of Neurology at the University of Michigan and Director of the Neurology Program for Research and Discovery and the ALS Center of Excellence.