Longitudinal Changes in Cognitive Test Scores in Patients With Relapsing-Remitting Multiple Sclerosis

Abstract

Background and Objectives

Cognitive impairment is a common and impactful symptom of relapsing-remitting multiple sclerosis (RRMS). Cognitive outcome measures are often used in cross-sectional studies, but their performance as longitudinal outcome measures in clinical trials is not widely researched. In this study, we used data from a large clinical trial to describe change on the Symbol Digit Modalities Test (SDMT) and the Paced Auditory Serial Addition Test (PASAT) over up to 144 weeks of follow-up.

Methods

We used the data set from DECIDE (clinicaltrials.gov identifier NCT01064401), a large randomized controlled RRMS trial to describe change on the SDMT and PASAT over 144 weeks of follow-up. We compared change on these cognitive outcomes with change on the timed 25-foot walk (T25FW), a well-established physical outcome measure. We investigated several definitions for clinically meaningful change: any change, 4-point change, 8-point change, and 20% change for the SDMT, any change, 4-point change, and 20% change for the PASAT, and 20% change for the T25FW.

Results

DECIDE included 1,814 trial participants. SDMT and PASAT scores steadily improved throughout follow-up: the SDMT from a mean 48.2 (SD, 16.1) points at baseline to 52.6 (SD 15.2) at 144 weeks and the PASAT from 47.0 (SD 11.3) at baseline to 50.0 (SD 10.8) at 144 weeks. This improvement in scores is most likely due to a practice effect. Throughout the trial, participants were more likely to experience improvement than worsening of their SDMT and PASAT performance, whereas the number of worsening events on the T25FW steadily increased. Changing the definition of clinically meaningful change for the SDMT and PASAT or using a 6-month confirmation changed the overall number of worsening or improvement events but did not affect the overall behavior of these measures.

Discussion

Our findings suggest that the SDMT and PASAT scores do not accurately reflect the steady cognitive decline that people with RRMS experience. Both outcomes show postbaseline increases in scores, which complicates the interpretation of these outcome measures in clinical trials. More research into the size of these changes is needed before recommending a general threshold for clinically meaningful longitudinal change.

Cognition is affected in all forms of multiple sclerosis (MS), and cognitive dysfunction in MS can occur from disease onset, including at the clinically isolated syndrome stage.¹ Cognitive dysfunction in adults with MS is common; its prevalence depends on the definition of impairment and cognitive tests used and varies between 43% and 65%.²

Clinical trials in all forms of MS often include measures of cognitive function, but most studies examining cognition in MS are cross-sectional. Cross-sectional analyses reflect differences in outcome across the entire spectrum of disease duration and participant ages, whereas longitudinal studies describe change over a range of only a few years. It is unclear whether cognitive measures as currently assessed are useful longitudinal outcomes over the two- or three-year duration of a typical clinical trial. In addition, measures taken at a single time point may fail to identify patients with progressive cognitive decline who have yet to reach a threshold defined as clinically meaningful. Two commonly used cognitive trial outcomes are the 3-second Paced Auditory Serial Addition Test (PASAT) and the Symbol Digit Modalities Test (SDMT). The PASAT is a measure of processing speed and executive function and requires patients to perform a series of mental additions under time pressure. The PASAT is often experienced as stressful,³ and therefore, many trial participants do not tolerate it well. The PASAT also has a substantial practice effect, which limits its usefulness as a trial outcome and makes it difficult to estimate change over the trial period.⁴ The SDMT was originally developed as a screening test⁵ and predominantly measures processing speed. The SDMT requires trial participants to match symbols to numbers in a set period of time. The SDMT is generally seen as more patient-friendly and was recommended as the standard longitudinal outcome for clinical trials in MS.^6-8 Initially, a 4-point decrease in SDMT performance was understood to be clinically relevant as this cutoff is associated with the loss of employment⁹; more recently, an 8-point decrease has been proposed as a reliable threshold of cognitive decline.¹⁰ Validated cutoff scores do not exist for the PASAT.

In an analysis of a large clinical trial data set, we found that people with secondary progressive MS showed a steady improvement on the SDMT throughout 2 years of follow-up.¹¹ This improvement in SDMT scores occurred in contrast to steady worsening on physical outcome measures and may be due to a practice effect. Other possibilities include a placebo effect and the net effects of selective dropouts. In this investigation, we use patient-level data from DECIDE, a large phase 3 randomized controlled trial in relapsing-remitting MS (RRMS), to study change in SDMT and PASAT scores over 3 years of follow-up.

Methods

Standard Protocol Approvals, Registrations, and Patient Consents

The DECIDE data set was obtained from Biogen (Biogen, Cambridge, MA), the pharmaceutical company that conducted and oversaw this trial. Ethical approval for the original trial is described in the original publication.¹² Ethical approval for this analysis was sought and granted by the University of Calgary Conjoint Health Research Ethics Board.

Data Set

DECIDE was a randomized controlled phase 3 study comparing subcutaneous daclizumab, administered every 4 weeks, with IM interferon beta-1a, administered once weekly, in people with RRMS for up to 144 weeks. Entry criteria included a confirmed diagnosis of RRMS, age 18–55 years, a score of 0–5.0 inclusive on the Expanded Disability Status Scale (EDSS),¹³ and clinical relapses in the 2 years before inclusion.

Statistical Analyses

In DECIDE, the SDMT was measured every 24 weeks, and the PASAT was measured every 12 weeks. As a measure of physical disability, we chose the Timed 25-foot Walk (T25FW),¹⁴ which we previously showed to be a reliable and robust measure of physical disability in RRMS.¹⁵ The T25FW was performed every 12 weeks in DECIDE. For this analysis, we focused on the measurements at baseline, 24, 48, 72, 96, 120, and 144 weeks.

We determined the proportion of participants with unconfirmed and 6-month confirmed worsening and improvement on these outcome measures by comparing baseline and follow-up measures. Individuals missing a score at baseline, the time point of interest, or the corresponding 6-month confirmation assessment were excluded from the analysis.

We explored different definitions for clinically meaningful change for the SDMT and PASAT. For the SDMT we used (1) any change from baseline, (2) the established threshold⁹ of a 4-point change from baseline, (3) the recently suggested¹⁰ threshold of an 8-point change from baseline, and (4) 20% or more change from baseline. For the PASAT, we used (1) any change from baseline, (2) 4-point change from baseline, and (3) 20% or more change from baseline. For the T25FW, we defined clinically meaningful change as 20% or more change from baseline.¹⁶

We compared cognitive outcome scores and change in cognitive outcome scores between the treatment arms with Student t tests. Statistical significance was taken to be at the two-sided 0.05 level. We used the R statistical software package for Windows version 4.2.2 for all statistical analyses.¹⁷

Change in Cognitive Scores by Cognitive Performance at Baseline

Because the change in cognitive outcomes could vary depending on the degree of cognitive dysfunction at baseline,¹⁸ we investigated the mean change from baseline in SDMT and PASAT scores in individuals with below-average vs average-and-above scores at baseline. For these analyses, we used the average scores at baseline in the DECIDE data set as cutoffs: 49 for the SDMT and 50 for the PASAT (Table 1).

Table 1.

Baseline Characteristics

Baseline Factors Associated With Change in Cognitive Scores

To investigate which baseline patient factors may be associated with the change in cognitive scores, we built linear regression models with the change in SDMT and PASAT scores between baseline and 96 weeks as the dependent (outcome) variable and age, sex, disease duration, treatment arm, and the cognitive score at baseline as independent (predictor) variables.

Estimate of Regression to the Mean

To illustrate the possible influence of regression to the mean on our results, we first determined the Pearson correlation coefficient (CC) of the first (baseline) and second (24 weeks) SDMT and PASAT measurements. We then calculated Z-scores for the first and second SDMT and PASAT measurements and selected cases with a baseline Z-score ≤ −1.282, which represent the lowest 10% of normally distributed test scores. We then calculated the mean Z-score for the 24-week measurement in these selected cases. We used the following formula to estimate the size of the effect of regression to the mean in the selected cases:

(1 – CC) * (mean Z-score 24 weeks – mean Z-score baseline)

We multiplied the result of this formula with the SD of the baseline measurement in the whole cohort to arrive at an estimate of regression to the mean in SDMT or PASAT units.

Data Availability

The data used in this study are available on request from Biogen. Individual participant data collected during the trial will be shared after anonymization and on approval of a research proposal and data sharing agreement. Research proposals can be submitted online (biogenclinicaldatarequest.com).

Results

DECIDE Data Set

The DECIDE data set contained patient-level data from 1,814 trial participants. Their baseline characteristics are given in Table 1. DECIDE was designed to follow the participant cohort until the last participant had finished the 96-week visit to a maximum of 144 weeks. As a consequence of this design, there were substantially fewer participants at the 120- and 144-week follow-up visits. At 96 weeks, there were 1,311 participants (72.2% of the baseline cohort) with SDMT measurements in the cohort, compared to 699 (38.5% of the baseline cohort) at 120 weeks and 383 (21.1% of the baseline cohort) at 144 weeks. For the PASAT, there were 1,316 (72.5% of the baseline cohort) measurements at 96 weeks, 703 (38.8% of the baseline cohort) at 120 weeks, and 388 (21.4% of the baseline cohort) at 144 weeks (Table 2).

Table 2.

Raw Outcome Scores Throughout Follow-up

Change in Cognitive Outcome Scores

Mean SDMT and PASAT scores steadily increased during follow-up (Figure 1 and Table 2). At 144 weeks of follow-up, mean SDMT scores had increased by 4.9 points (SD 12.5), and mean PASAT scores had increased by 3.3 points (SD 8.1) compared with baseline.

Figure 1. Mean Cognitive Outcome Scores During Follow-up.

Mean SDMT (A) and PASAT (B) scores throughout follow-up. Error bars represent the SD. Mean scores on both instruments steadily increase throughout follow-up, likely due to a practice effect of the repeated application of the measures. There is little difference between the treatment arms in cognitive scores.

There were significant differences in SDMT and PASAT scores between treatment arms from baseline to 144 weeks of follow-up, with significantly higher scores in the daclizumab arm (Figure 1 and Table 3). In both treatment arms, SDMT and PASAT scores increased over time, significantly more in the daclizumab arm than in the interferon arm: at 144 weeks, mean SDMT scores increased by 6.8 (SD 12.2) in the daclizumab arm, compared with 2.9 (SD 12.5) in the interferon arm; however, at 48 weeks, the significant differences were in the opposite direction with the interferon beta-1a arm showing greater change. At 144 weeks, mean PASAT scores increased by 4.0 points (SD 8.1) in the daclizumab arm, compared with 2.6 points (SD 8.1) in the interferon arm (Table 3). Although these differences were statistically significant, the overall difference in raw scores was low, for example, at 96 weeks of follow-up, the mean SDMT score was 52.4 (SD 15.1) in the daclizumab arm, compared with 51.0 (SD 15.1) in the interferon arm. Similarly, the mean PASAT score at 96 weeks of follow-up was 50.2 (SD 10.6) in the daclizumab arm and 49.8 (SD 10.6) in the interferon arm (Figure 1 and Table 3).

Table 3.

Mean Outcome Scores Throughout Follow-up in the 2 Treatment Arms

Clinically Meaningful Change in Cognitive Outcomes

Throughout the trial, participants were more likely to experience improvement than worsening of their SDMT and PASAT performance, and improvement events increased steadily over follow-up, whereas there was little change in worsening events for the SDMT and PASAT (Figure 2 and Table 4). These findings for cognitive outcomes stand in contrast to change on the T25FW, a physical outcome measure. On the T25FW, the number of worsening events steadily increased over time, and worsening events always outnumbered improvement events. Using 6-month confirmation yielded lower estimates than for unconfirmed change but did not alter the overall behavior of the measures (Table 4). Changing the definition of clinically meaningful change for the SDMT and PASAT similarly changes the overall number of worsening or improvement events but does not affect the overall behavior of these measures (Table 4).

Figure 2. Worsening and Improvement Events During Follow-up.

Participants with 6-month confirmed ≥20% worsening or improvement on the T25FW (A), SDMT (B), and PASAT (C). As expected, worsening events on the T25FW steadily increase throughout the trial, and worsening events always outnumber improvement events. In contrast, on the SDMT and PASAT, worsening events remain low and stable throughout follow-up, whereas improvement events steadily increase. Improvement events always outnumber worsening events on the SDMT and PASAT.

Table 4.

Percentages of Participants With Worsening and Improvement of Cognitive Outcomes Throughout Follow-up

Change in Cognitive Scores by Cognitive Performance at Baseline

SDMT and PASAT scores increased in participants with below-average cognitive performance and in those with average-and-above cognitive performance at baseline (Table 5). Scores increased more in participants with below-average performance at baseline for both measures (mean change from baseline at 96 weeks: 7.3 SDMT points, 5.2 PASAT points) compared with those with average-and-above performance at baseline (mean change from baseline at 96 weeks: 0.4 SDMT points, 0.5 PASAT points, Table 5).

Table 5.

Mean Raw Scores and Change From Baseline in Cognitive Outcomes by Baseline Cognitive Performance

Baseline Factors Associated With Change in Cognitive Scores

For the SDMT, age, treatment arm, and SDMT score at baseline were significantly associated with the change in scores between baseline and 96 weeks in the linear regression model, whereas sex and disease duration showed no significant association after adjustment for the other covariates. The regression coefficients were −0.17 per year increase in age (p < 0.0001), −0.38 per point increase in SDMT at baseline (p < 0.0001), and 1.67 for participants in the treatment arm (p = 0.01, adjusted R² = 0.22). For the PASAT, only age and PASAT score at baseline were significantly associated with the change in scores between baseline and 96 weeks, whereas sex, treatment arm, and disease duration showed no significant associations. The regression coefficients were −0.08 per year increase in age (p = 0.001) and −0.28 per point increase in the PASAT at baseline (p < 0.0001, adjusted R² = 0.17).

Estimate of Regression to the Mean

Studies of cognitive outcome often select for participants with low cognitive performance. In DECIDE, 182 participants had a baseline SDMT Z-score, and 242 participants had a baseline PASAT Z-score of −1.282 or below. Our calculation showed an estimated regression to the mean of 1.52 SDMT and 0.41 PASAT points between the first and second measurements in the selected cases, which suggests a moderate effect of regression to the mean on the results.

Discussion

In this investigation of people with active RRMS, similar to our previous study in SPMS,¹¹ the cognitive outcomes SDMT and PASAT show a steady improvement over time. Because the clinical impression is that cognitive function in RRMS steadily declines, it would be reasonable to assume that a useful measure of cognitive function in RRMS would show worsening over time at the group level, even if only a portion of the trial cohort experienced cognitive decline. Contrary to this expectation, our investigation of the DECIDE data set shows that SDMT and PASAT scores steadily increased throughout the 3 years of follow-up, probably due to a practice effect. We cannot exclude that the increase in SDMT and PASAT scores is due to a true treatment effect of both interferon and daclizumab; however, we deem this possibility unlikely because the increase in SDMT and PASAT scores is roughly equal in both trial arms, although daclizumab is the more efficacious treatment. In our previous investigation on cognitive outcome scores in ASCEND, a large placebo-controlled trial in SPMS,¹¹ the increases over time were also seen in the placebo arm; thus while not ruling out a treatment effect, it adds to the minimization of such an effect. Further research into SDMT and PASAT scores in the placebo arms from placebo-controlled clinical trials and in real-world patient cohorts is necessary to clarify this point.

Another possible explanation of our findings would be regression to the mean, in the sense that the baseline measurements of the cognitive outcomes were unusually low and that the subsequent regression to the mean at later measurements explains the increase in scores over the duration of the trial. Our exploration of this question suggests a moderate amount of regression to the mean in DECIDE. This effect does not appear strong enough to explain the increase in scores on its own. The observed increase in cognitive scores is most likely a combination of a practice effect, regression to the mean, and possibly also a true treatment effect, with the practice effect as the largest contributor.

To get a better understanding of this issue, we contrasted change on the SDMT and PASAT by several definitions with change on the T25FW, a responsive and reliable measure of physical disability in RRMS.¹⁵ Our intention here was not to suggest that cognitive performance and physical disability are closely correlated, but to compare the SDMT and PASAT with a reliable outcome measure that consistently behaves in keeping with the clinical impression and patient experience of disability worsening. As expected, worsening events on the T25FW steadily increase, whereas improvement events change little over the 3 years of follow-up. On the T25FW, worsening events always outnumber improvement events. In contrast to this, both the SDMT and PASAT show a steady increase in improvement events throughout follow-up, no matter what definition of worsening is used, and improvement events always outnumber worsening events. This suggests that the SDMT and PASAT scores do not accurately reflect the steady cognitive decline that people with RRMS experience.

Practice effects can be defined as improvements in performance on a cognitive outcome due to repeated exposure.^19,20 A practice effect on the SDMT has been reported in healthy adults,²¹ in people with mild cognitive impairment,¹⁸ and in patients with stroke.²² More recently, an investigation of 264 individuals with self-declared MS using a smartphone application also found a substantial practice effect for repeated SDMT testing over a 5-week interval.²³ These findings are important to put some reports from clinical trials into perspective. The long-term extension of the BENEFIT trial of interferon beta-1b treatment in people with clinically isolated syndrome reports a 0.9-PASAT point difference between the immediate and delayed treatment groups after 11 years of follow-up and concludes that early treatment had a long-lasting beneficial effect on cognitive outcomes.²⁴ An analysis in the EXPAND trial of siponimod treatment for secondary progressive MS showed an increase in SDMT scores from baseline of about 1.5 points at 24 months and a decrease in the placebo group of about 1.5 points and concludes that treatment with siponimod beneficially affects cognitive functioning.²⁵ We warn against a potential overinterpretation of such results, as these reported differences fall well within the range of change from baseline observed in our investigation.

To explore a possible treatment effect of daclizumab on cognitive outcomes, we investigated SDMT and PASAT scores by treatment arm. Throughout follow-up, there was little difference in cognitive performance scores between treatment arms. For example, at 96 weeks, daclizumab-treated trial participants reached a mean SDMT score of 52.4 (SD 15.1) points compared with 51.0 (SD 15.1) points for interferon beta-1a treated trial participants (Table 3), a mean 1.4-point difference. A previous publication on cognitive outcomes in the DECIDE data set using a different methodology interpreted these differences as evidence of daclizumab efficacy when compared with interferon beta-1a for cognitive processing speed improvement.²⁶ Given the small differences in cognitive performance observed in this study, we again caution against the overinterpretation of such results.

The 4-point threshold for clinically meaningful change on the SDMT that is now generally used is based on a study in 97 people with MS who underwent SDMT measurements at 2 time points 1 or more years apart.⁹ In that study, a 4-point raw score decrease was associated with a loss of employment. More recently, another investigation in 166 patients sought to validate thresholds for reliable change and suggested an 8-point cutoff as more meaningful.¹⁰ Neither of these studies corrected for the practice effect of the SDMT. Our investigation shows that the issue of steady improvement on the SDMT persists regardless of the threshold used. Thus, a 4-point decline may underestimate a change attenuated by a practice effect and may explain why an 8-point decline arose as an alternative cut point. Given the inherent issues of postbaseline increases in scores, regardless of the reasons, more research into the size of the practice effect of the SDMT and PASAT should be performed before recommending a general threshold for clinically meaningful longitudinal change.

Although practice effects are usually understood as a source of error and imprecision, it has also been proposed that the practice effect itself could be seen as a cognitive achievement and may therefore deliver meaningful information.^19,20 Our exploration of the change in cognitive scores by baseline cognitive performance shows that this interpretation may not be relevant in RRMS. In the DECIDE data set, both participants with below-average and with average-and-above cognitive performance at baseline showed increasing cognitive scores during follow-up. For both outcome scores, there was greater improvement in those with below-average performance at baseline compared with those with average-and-above performance. This may be an artifact, in the sense that those with high baseline SDMT and PASAT scores can improve only marginally on their already high baseline score, whereas those with low baseline scores had more room to improve with practice. Our analysis of baseline factors associated with the change in cognitive outcome scores showed that higher age and higher cognitive performance at baseline were associated with a decrease in cognitive scores during follow-up. Sex or disease duration showed no such associations.

Our study has several limitations: although it includes a large cohort of patients with RRMS, it is worth remembering that the DECIDE participants were selected using strict entry criteria. The change in cognitive function of DECIDE participants may therefore differ from unselected patients with RRMS. DECIDE also, in part due to its design, had substantial numbers of individuals without later follow-up visits. The later follow-up time points contained only a fraction of the baseline trial cohort and few of those recruited earlier into the study, which could affect the measurements and introduce bias.

Cognitive decline in all MS phenotypes remains a challenging and patient-relevant issue and an important treatment goal. Further research into longitudinal change in cognitive outcomes in both clinical trial and real-world cohorts is needed to better characterize the performance of such measures and to test and develop new outcome measures to detect true change in clinical trials.

Glossary

CC

correlation coefficient

CELs

contrast-enhancing lesions

EDSS

Expanded Disability Status Scale

IQR

interquartile range

MS

multiple sclerosis

PASAT

Paced Auditory Serial Addition Test

RRMS

relapsing-remitting multiple sclerosis

SDMT

Symbol Digit Modalities Test

T25FW

timed 25-foot walk

Appendix. Authors

Study Funding

The authors report no targeted funding.

Disclosure

N. Castrogiovanni and J. Mostert report no disclosures. P. Repovic received consulting and/or speaking honoraria from Alexion, Biogen, Celgene, Roche, Sanofi Genzyme, Viela, and EMD Serono. J.D. Bowen received honoraria from serving on the scientific advisory board and speaker's bureau of Biogen, Celgene, EMD Serono, Genentech, and Novartis. He has received research support from AbbVie Inc, Alexion, Alkermes, Biogen, Celgene, Sanofi Genzyme, Genentech, Novartis, and TG Therapeutics. B. Uitdehaag received consultancy fees and/or research support from Biogen, Sanofi Genzyme, EMD Serono, Novartis, Roche, Teva, and Immunic Therapeutics. E. Strijbis reports no disclosures. G. Cutter served on Data and Safety Monitoring Boards: Applied Therapeutics, AI Therapeutics, AMO Pharma, Astra-Zeneca, Avexis Pharmaceuticals, Biolinerx, Brainstorm Cell Therapeutics, Bristol Meyers Squibb/Celgene, CSL Behring, Galmed Pharmaceuticals, Green Valley Pharma, Horizon Pharmaceuticals, Immunic, Karuna Therapeutics, Mapi Pharmaceuticals LTD, Merck, Mitsubishi Tanabe Pharma Holdings, Opko Biologics, Prothena Biosciences, Novartis, Regeneron, Sanofi-Aventis, Reata Pharmaceuticals, Teva Pharmaceuticals, NHLBI (Protocol Review Committee), University of Texas Southwestern, University of Pennsylvania, and Visioneering Technologies, Inc. Consulting or Advisory Boards: Alexion, Antisense Therapeutics, Biogen, Clinical Trial Solutions LLC, Entelexo Biotherapeutics, Inc., Genzyme, Genentech, GW Pharmaceuticals, Immunic, Immunosis Pty Ltd, Klein Buendel Incorporated, Merck/Serono, Novartis, Perception Neurosciences, Protalix Biotherapeutics, Regeneron, Roche, and SAB Biotherapeutics. Dr. Cutter is employed by the University of Alabama at Birmingham and President of Pythagoras, Inc., a private consulting company located in Birmingham AL. M.W. Koch received consulting fees and travel support from Biogen, Novartis, Roche, Sanofi Genzyme, and EMD Serono. Go to Neurology.org/N for full disclosures.