Novel computer-based testing shows multi-domain cognitive dysfunction in patients with multiple sclerosis

Short abstract

Background

Although cognitive dysfunction is a leading cause of disability and poor quality of life in patients with multiple sclerosis (MS), it is infrequently tested in routine clinical evaluation. Development of a cognitive testing paradigm that captured MS-related cognitive dysfunction and could be obtained in a routine clinical setting may increase surveillance and recognition of cognitive dysfunction.

Objectives

This was a pilot study to determine if Cognivue could find cognitive performance differences between patients with MS and healthy controls (HC).

Methods: A total of 24 patients with MS and 12 HCs between 18 and 50 years old were enrolled. Baseline testing included an Expanded Disability Scale (EDSS), paced auditory serial additions test (PASAT), symbol digit modalities test (SDMT) and Cognivue. Subjects then had repeat testing every 1–2 months for a maximum of three tests.

Results

Significant differences were found between MS and HC on SDMT, PASAT, and Cognivue Total score. Most Cognivue subtests showed significant differences between MS and HC. Cognivue scores correlated with both SDMT and PASAT and had high test-retest reliability in HCs.

Conclusion

Cognivue was able to detect multi-domain cognitive dysfunction in MS. Further studies to determine validity of Cognivue in MS with comparison with neuropsychological testing and sensitivity to clinical change are still needed.

Introduction

Cognitive impairment is a leading cause of diminished quality of life in patients with multiple sclerosis (MS). It is associated with loss of employment and inability to perform activities of daily living (ADL) and instrumental activities of daily living (IADLs).^1–5 Despite the substantial impact of cognitive impairment on quality of life, routine clinical assessment of cognition is rarely performed.⁶ Identification of cognitive dysfunction with testing that could be implemented in the routine clinical setting, thereby increasing cognition dysfunction surveillance, may marshal social supports, guide cognitive rehabilitation, target drug therapies, and serve as a biomarker for evaluation of disease modifying therapies.

MS-related cognitive dysfunction is heterogeneous. Cognitive impairment is seen early in the disease course prior to a clinical relapse in a disease status known as radiologically isolated syndrome (RIS). In RIS, patients have been found to have common deficits in information processing speed and verbal memory, in common with patients with clinically definite relapsing MS.⁷ A study by Rao et al. in 1991 in 100 patients with MS showed cognitive deficits in recent memory, abstract reasoning, attention/concentration, and visuospatial skills.⁸ A study by Benedict et al. further supported the heterogeneity of cognitive dysfunction in validating the Minimal Assessment of Cognitive Function in MS (MACFIMS) by showing impairments in verbal fluency, visuospatial perception, verbal memory, visual memory, working memory, processing speed and concept formulation.⁹ Cognitive dysfunction is related to both patient demographics, education, as well as disease-related factors including disease duration, disease type, disease-related disability and disease burden (lesion volume, brain atrophy, etc.).^7,10–16 Therefore, evidence of cognitive impairment is seen early in the disease course and worsens with clinical and pathological features of the disease.

Formal neuropsychological testing (NPT) in patients with MS shows a broad spectrum of cognitive dysfunction.^17,18 However, NPT may take several hours to perform and is infrequently used in the clinical setting. Instead, the Symbol Digit Modalities Test (SDMT) and Paced Auditory Serial Addition Test (PASAT) have been developed for use as tools for evaluation of cognitive dysfunction for use in clinics and clinical trials.^19–21 Both the SDMT and PASAT focus on the known common impairment of processing speed and take fewer than 5 min to administer. While the SDMT and PASAT evaluate other cognitive domains in addition to processing speed, the single outcome score does not allow for specific domain assessment; thus, they may not capture the heterogeneity of MS-related cognitive dysfunction.

The ideal cognitive test would be easy to use, of brief duration, and control for motor and visual impairment. It would be desirable for such tests to be autonomously administered, test multiple domains, and provide performance scores that facilitate tracking of cognitive dysfunction in MS. In 2015, the Food and Drug Administration (FDA) gave Cognivue, a computer-based cognitive test of perception and memory, de novo approval for the assessment of cognitive dysfunction in individuals over the age of 55 years. Cognivue has many features of the ideal cognitive screening test for patients with MS, including autonomous administration, short duration (10 min), calibration for dominant hand and vision impairment and perception abilities for memory testing. In this study, we explored the ability of Cognivue to find differences in cognition in patients with MS and healthy controls (HCs), and thus evaluate preliminary validity of Cognivue as a screening measure for cognition in MS in comparison to validated measures of the SDMT and PASAT.

Materials and methods

Cognivue

Cognivue is a computer-based cognitive test that received de novo FDA approval for the assessment of early signs of cognitive impairment in individuals over the age of 55 years in 2015 based on a validation study with the St. Louis University Mental Status (SLUMS) test (FDA De Novo Number 130033). Cognivue testing is based on findings of perceptual deficits in patients with Alzheimer’s disease, including letter, word, and motion perception.^22–24

The Cognivue system is a computer-on-wheels with rotatory mandipulum (joystick) for item selection. First, the patient engages in visuomotor testing for calibration of dominant hand motor and visual function (Figure 1). The first visuomotor test is the “Adaptive Motor” test. During the adaptive motor test, dots on the screen move in a circular pattern with random changes in direction, speed. The subject is cued to keep the dots selected by moving the rotatory mandipulum. During testing the sensitivity of rotatory mandipulum also changes requiring the subject to modifying the extent of mandipulum movement. This test is used to calibrate for dominant hand motor impairment. Next, the subject performs the “Visual Salience” test in which the patient again tracks the dots as they move in a circular pattern and change direction and also change contrast – starting with higher contrast and then becoming dim and then becoming bright (Figure 1). The overall time that the patient can keep the dots selected is reflected in a higher score on a scale of 0–100.

Figure 1. Cognivue testing. Samples of each Cognivue test.

Following visuomotor testing, perception and memory tests are calibrated such that patients who perform well on adaptive motor testing will be given less time to make selections and those who perform well on visual salience testing will have reduced visual presentation of words, letters, and symbols in both perception and memory testing (Figure 1). Next, the patient is tested in perception of letters, words, symbols, and direction of movement. In each task the subject is prompted to identify the correct letter, word, symbol, and direction of movement (Figure 1). Perception tasks then calibrate for memory testing in letter, word, symbol, and movement (Figure 2). In memory testing, the subject is prompted to remember the first item and shown several items in sequence dependent on task, e.g. letters, words, symbols, or direction of movement dependent on task. The subject is then presented several items, only one of which is the correct response.

Figure 2.

Cognivue testing algorithm. Sequential intra-testing calibration schematic for motor (dominant hand), visual disability, and perceptual testing prior to memory testing. MD/CAS Access: treating clinician and Cerebral Assessment Systems (CAS) access to Cognivue testing.

Lastly, a total score is calculated based on the mean of the perception and memory tests. Subtest scores are also calculated. All scores are percentage of correct responses on a scale of 0–100.

Subjects

We enrolled 24 patients with clinically-definite MS, by the 2010 revised McDonald criteria with relapsing phenotype, and 12 HCs s were enrolled. Patients with MS were recruited from the Rochester MS Center at the University of Rochester Medical Center, Rochester, NY. HCs were frequently spouses or acquaintances of the MS patients enrolled. Inclusion criteria for both groups were age between 18 and 50 years. Exclusion criteria for both groups included a history of cognitive impairment related to another cause, Hamilton Depression Rating Scale (HAM-D) score of > 20, and inability to complete the informed consent process. The HAM-D is a 17-item questionnaire to assess symptoms and severity of depression. The upper age limit of the study was used to reduce the influence cognitive changes that could be due to age-related neurodegenerative diseases. The study was approved by the University of Rochester Research Subjects Review Board. All subjects provided written informed consent. All MS patients were being treated with natalizumab to reduce the risk of clinical or MRI relapse and promote availability for monthly testing around the time of their infusions. Additional exclusion criteria for MS patients included a clinical or radiological relapse in the prior 3 months.

Testing

Subjects were scheduled for a total of three study visits scheduled at an interval of 1–2 months. At each study visit the subjects completed the Multiple Sclerosis Functional Composite (MSFC; timed 25-foot walk T25W, 9-hole peg test, and PASAT), SDMT, and Cognivue testing. At the first visit all subjects were evaluated and scored on the Expanded Disability Severity Scale (EDSS).

Statistical analysis

Due to the small number of subjects and concern for non-normal distributions, nonparametric analyses were performed for comparisons between subjects with MS and HCs. Mann–Whitney tests were used for comparisons of age, education (years), HAM-D, and EDSS. A Chi-squared test was used to compare gender proportions between groups. The first trials of MSFC and SDMT were compared between MS and HC again using Mann–Whitney tests. The first trial results of Cognivue total and subtask scores were compared using Mann–Whitney tests with conservative correction for multiple comparisons by multiplying the p value for each analysis by the total number of comparisons (p′ = p * # comparisons, while keeping alpha = 0.05.). A Bland–Altman plot, which plots the mean score of trial 1 and 2 (x-axis) and the difference between trials 1 and 2, was used to assess reliability. Finally, correlation analysis between first trials of Cognivue total score, SDMT, PASAT, and 9-hole peg test and Cognivue Adaptive Motor test were made with combined MS + HC data as well as MS only using nonparametric Spearman correlation analysis. The p value was adjusted for multiple comparisons (p′ = p * # comparisons), while keeping alpha = 0.05.

Results

Demographics

All subjects (MS, n = 24; HC, n = 12) completed a total of two study visits, 9 MS subjects and 5 HCs completed all three study visits. Demographic data, including age, sex, and level of education, were similar between HC and MS (Table 1). Depression severity, as measured by the HAM-D was not different between HC and MS (median interquartile range (IQR), MS = 3.5 IQR 1–7, HC = 4 IQR 2.75–5.75, p = 0.7). The mean disease duration in the MS patients was 9.5 years (SD 6.19).

Table 1.

Demographics and baseline characteristics.

	HC median (SD, IQR)	MS median (SD, IQR)	p value
Age (SD)	39 (8.19, 28.75–42.0)	35.5 (6.16, 32.75–38.25)	0.52
Gender – % women	50%	62.5%	0.47a
Education (SD)	16 (2.61, 13.5–17.25)	14 (2.59, 12–16)	0.23
Hamilton depression (SD)	4.33 (3.23, 2.75–5.75)	4.88 (5.63, 1.0–7.0)	0.7
EDSS (SD)	0 (0.89, 0–0)	2.25 (1.38, 1.5–3.0)	<0.00001

HC: Healthy control; IQR: interquartile range; MS: multiple sclerosis; EDSS: expanded disability scale.

^aChi square.

Baseline disability

MS had a median EDSS of 2.25 (SD 1.38, IQR 1.5–3.0, Table 1). The T25W was, as expected, higher in MS than HC (Table 2). Additional MSFC components of 9-hole peg test and PASAT and SDMT scores were also different between MS and HC (Table 2).

Table 2.

Multiple sclerosis functional composite and SDMT.

	Multiple sclerosis	Healthy controls
	Mean (SD, IQR)	Mean (SD, IQR)	p
Timed 25-foot walk (s)	4.62 (1.56, 3.75–5.02)	3.50 (0.60, 3.09–3.84)	.01
9-hole peg test (s)	26.47 (18.64, 19.71–24.60)	18.17 (2.22, 17.12–18.63)	.001
PASAT	38.75 (12.18, 31.00–50.25)	50.42 (6.44, 45.00–55.25)	.005
SDMT	45.67 (12.55, 39.75–50.50)	61.67 (8.32, 56.50–65.75)	.0005

IQR: interquartile range; PASAT: paced auditory serial additions test; SDMT: Symbol digit modalities test.

Cognivue testing

MS patients performed significantly worse on Cognivue testing than HCs (Figure 3). Total scores, reflecting average scores across perception/discrimination and memory testing, were 80.71 and 93.00 for MS and HC, respectively (p = 0.001). Both visuomotor calibration tests, adaptive motor and visual salience, were significantly different between MS and HC. Of the cognitive battery components, motion discrimination and letter memory tasks were significantly different between MS and HC (p < 0.05). The test with largest magnitude difference between MS and HC was motion discrimination (74.50 SD 13.61 4.22 vs 97.92 SD 4.95).

Figure 3.

Cognivue scores. Mean first Cognivue testing scores for both healthy controls (blue) and patients with multiple sclerosis (orange). Perception and memory tasks are identified with labelled brackets. Error bars = SD. Analysis with nonparametric Mann–Whitney tests with correction for multiple comparisons. p < 0.05. HC: Healthy controls; MS: multiple sclerosis.

Cognivue reliability

We evaluated intra-subject Cognivue reliability in subjects who had completed two or more study visits. Test-retest stability in both HC and MS subjects were analyzed with a Bland–Altman plot (Figure 4), which plots the mean Cognivue Total score vs the difference between the scores on Test 1 and Test 2 for each subject.²⁵ HCs had lower test-retest variability compared to MS.

Figure 4.

Bland–Altman plot. Total Cognivue scores from the first and second tests are plotted for healthy controls (blue) and patients with multiple sclerosis (orange). 95% confidence interval for all scores is noted by red lines. SE: standard error.

Convergent validity

Cognivue testing was compared to both SDMT and PASAT by calculating Spearman correlation coefficients (Figure 5, Table 3). There were significant positive correlations between Cognivue Total scores and both SDMT and PASAT scores for all subjects (MS + HC) and MS alone (Table 3). The Cognivue Total score was more strongly with SDMT than PASAT. Additionally, there was a significant positive correlation between Cognivue Adaptive Motor and the 9-hole Peg test, tests of dominant hand function (Table 3).

Figure 5.

Spearman correlation of cognitive testing. Correlation analysis of (a) SDMT and PASAT, (b) total Cognivue score and PASAT, and (c) total Cognivue score and SDMT for healthy controls (blue) and patients with multiple sclerosis (orange). R and p values are in Table 3. Correction for multiple comparisons. SDMT: symbol digit modalities test; PASAT: paced auditory serial additions test.

Table 3.

Spearman correlation analysis of cognition testing.

Combined MS + HC	r	p
SDMT–PASAT	0.63	0.0002
Cognivue–SDMT	0.82	1 × 10−8
Cognivue–PASAT	0.55	0.004
9-hole peg – Adaptive Motor	0.57	0.002
MS only	r	p
SDMT–PASAT	0.48	0.1
Cognivue–SDMT	0.79	<0.0001
Cognivue–PASAT	0.61	0.1
9-hole peg – Adaptive Motor	−0.15	3.9

MS: multiple sclerosis; HC: Healthy control; SDMT: Symbol digit modalities test; PASAT: paced auditory serial additions test.

We also compared MS patient’s scores on SDMT and Cognivue. A total of 12 MS subjects had a SDMT Z score of less than –1.0, and all 12 of these subjects had a Cognivue Total score that was greater than 1 SD below the HC mean. No HC subject had a SDMT Z score <1.0 SD or a Cognivue Total score < 1.0 SD below the HC mean.

Discussion

This is the first study to evaluate cognitive dysfunction in patients with MS using the computer-based testing system Cognivue. Based on our baseline data and demographics, MS subjects in this study reflect a population that would be typical of a MS population with stable, relapsing disease of significant disease duration, and moderate disease-related disability EDSS. Not unexpectedly, subjects with MS did significantly worse on all validated motor and cognition tests, which again confirms that these patients have signs of MS-related disability. With this established, we explored the cognitive phenotype of this sample population with respect to Cognivue testing.

Cognivue total scores were significantly worse in subjects with MS (Figure 3). Additionally, subcomponent tasks of visuomotor performance, motion discrimination and letter memory were significantly different between HC and MS. The fact that visuomotor tasks were significantly different between the groups highlights the importance of calibrating cognitive testing for dominant hand and visual impairment, which is something that Cognivue attempts to perform. The exact implication of motion discrimination and letter memory impairment in patients with MS is uncertain. However, a prior study has shown that motion perception is impaired with patients with optic neuritis following recovery of visual acuity.²⁶ We did not obtain clinical data with regard to history of optic neuritis as a part of this study. Letter memory testing may reflect previously reported verbal memory impairment in patients with MS, but it is unclear why Cognivue letter but not word memory is impaired.^7,9 This study was not designed to answer these questions, but to determine whether Cognivue has the potential to be a screening tool for cognitive dysfunction in patents with MS. Larger studies that include neuropsychological testing and brain imaging (quantitative MRI analyses) are needed to further explore links between Cognivue, cognitive impairment, and MS-related pathology.

The primary limitations of this study are the small sample size and study completion failure (14 of 24 subjects completed all three study visits). However, even with that small sample size, significant differences between HC and MS were seen. A major strength of this study was our matched age, sex, and education matched HC group. Findings of correlation between SDMT and PASAT and differences between MS and HC groups in these validated tests further strengthen this study.

Evaluation of test–retest variability showed greater variability in MS patients compared to HCs. This finding may reflect MS patients’ greater variability across a wide range of functional assessments.^27–30 Subjects with MS tended to have improved scores on the second test, which may reflect a learning or practice effect in combination with a ceiling effect seen in HC. Though not yet reported, similar practice/learning effects with Cognivue have been seen in individuals over the age of 55 years.

Further evaluation of Cognivue in MS patients is needed. First, obtaining a large Cognivue dataset in HC will allow for creation of total Cognivue and Cognivue subtest score normative quantiles based on age and education, similar to the SDMT. This will allow clinicians to consider both the total score and the subtest scores with respect to established normative data in this population. Second, Cognivue may provide a useful clinical biomarker of disease progression. In this regard, it will be important to understand the sources of variability in performance in routine testing to optimize sensitivity to changes in individual patient’s cognition. Morrow et al. showed that the SDMT detected a worsening in cognitive function during a MS relapse with near recovery to baseline 2 months later.³¹ We will attempt a similar study using Cognivue to determine if it is sensitive to acute cognition changes seen during MS relapses. Additionally, a longitudinal study is needed to evaluate for progressive cognitive changes in patients with relapsing and progressive forms of MS.

In conclusion, this was first study of cognition in patients with MS using Cognivue, an FDA approved computer-based cognitive test. We were able to detect significant differences in cognition between age, gender and education-matched MS and HC subjects. Cognivue may present a practical alternative to traditional cognitive screening tests. Strengths of Cognivue testing include its automated administration, short duration, multiple domain assessments, and adaptive approach. These technical features and results from this pilot study suggest that Cognivue may capture MS-related cognitive dysfunction and could be implemented in the clinical setting. This preliminary study lays the groundwork for future studies in Cognivue reliability, validity, and sensitivity to disease-related change in MS.

Conflicts of interest

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: ADS received fellowship training support from the National MS Society Sylvia Lawry Physician Grant and personal compensation for consulting from Sanofi Genzyme. CD is the CEO and founder of Cognivue, Inc., formerly Cerebral Assessment Systems, manufacturer of Cognivue. ADG received personal compensation for consulting from the following commercial entities: Abbvie, Acorda Therapeutics, Adamas, Sanofi Genzyme, Mylan, Novartis. ADG's employer, the University of Rochester, received research support for conducting clinical trials from the following commercial entities: Acorda Therapeutics, Avanir, Biogen-Idec, Novartis, Genetech Roche, Sanofi Genzyme, Sun Pharma, Teva.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.