Promising New Approaches to Assess Cognitive Functioning in People with Multiple Sclerosis

Abstract

Cognitive impairment has a major impact on the lives of people with Multiple Sclerosis (MS). Yet, it is often under-diagnosed, and more effective assessment methods are needed. In particular, brief measures that focus on cognitive functioning in daily life situations, are sensitive to modest change over time, and do not require a highly skilled assessor merit exploration. The purpose of this exploratory study was to investigate the performance of individuals with MS on three relatively new measures: the PROMIS Cognitive Concerns and Abilities Scales and the Everyday Problems Test (EPT), and to compare scores on these measures with scores on neurocognitive performance measures typically used to assess cognitive functioning in people with MS. Twenty-nine individuals with MS who reported cognitive concerns participated in the study. Most were non-Hispanic White women, with relapsing-remitting MS, diagnosed approximately 18 years ago. All three measures yielded reliability coefficients of .80 or above and also demonstrated sensitivity to change following an educational intervention. Scores on the revised EPT were moderately correlated with scores on five standard neuropsychological measures. Compared with the PROMIS Cognitive Concerns scale, scores on the self-reported PROMIS Cognitive Abilities scale tended to correlate more highly with the neurocognitive performance measures, although the correlations were generally small. While results of this exploratory study are promising, future research should be conducted with larger and more diverse samples of people with MS to determine the broader utility of these measures.

Impairments in cognitive abilities are among the most disturbing side effects of Multiple Sclerosis (MS). Benedict¹ estimated that approximately half of those diagnosed with MS have cognitive deficits, particularly in the areas of processing speed and episodic memory. These deficits impact all areas of life, and frequently preclude employment.^2–4 Because cognitive impairment may occur early in the course of MS,⁵ timely assessment of cognitive functioning in clinical settings is critically important. As Benedict and his colleagues pointed out,⁶ however, cognitive impairment in people with MS has been under-diagnosed, and more effective assessment methods are needed.

Multiple measures, including both self-report and performance tests, have been used to assess cognitive functioning in people with MS. One of the most widely accepted neuropsychological assessment batteries is the MACFIMS.¹ The MACFIMS battery⁷ is based on recommendations from an internal MS consensus conference. It consists of seven well–established neuropsychological tests covering five cognitive domains (language, spatial processing, new learning and memory, processing speed and working memory, and executive function). While there is strong evidence to support the reliability and validity of the MACFIMS,⁷ the battery is costly, because the tests must be administered by highly trained testers and take approximately two hours to complete. Moreover, the MACFIMS assesses functioning in a highly controlled testing situation and may be less useful in reflecting day-to-day cognitive functioning outside the standardized testing environment. “There is the question of the extent to which the laboratory ability and processing tasks traditionally studied by psychologists represent the mechanics underlying the pragmatic tasks of daily living.” (p. 69).⁸ It is also not clear that the neuropsychological tests used to diagnose impairment are sensitive to the change that might result from psycho-educational interventions designed to build cognitive skills to improve the daily lives of people with MS. Consequently, performance measures that focus on cognitive functioning in daily life situations, are sensitive to modest change over time, and that do not require a highly skilled assessor or specialized equipment merit investigation. One such measure is the Everyday Problems Test, ⁹ which was developed to test cognitive abilities in daily living situations among older adults.

In addition to performance measures that can be easily administered and reflect everyday activities, brief and psychometrically sound self-report measures of perceived cognitive functioning are needed to complement performance tests. Such measures are particularly useful for rapid screening in clinical settings. While a number of self-report measures exist and have been used with persons with MS (e.g. MSNQ, Perceived Deficits Questionnaire), more recent measures have capitalized on contemporary advances in psychometric theory to produce relatively short instruments that discriminate well throughout the underlying cognitive abilities continuum. Two such “new generation” self-report measures are the PROMIS Cognitive Concerns and Cognitive Abilities Scales derived from the Patient-Reported Outcomes Measurement Information System (PROMIS).¹⁰ Therefore the purpose of this exploratory study was to investigate the performance of individuals with MS on the PROMIS scales and the Everyday Problems Test, and to compare scores on these measures with scores on neuropsychological performance tests typically used to assess cognitive functioning in people with MS.

Methods

Recruitment

Following Institutional Review Board Approval, participants for this study were recruited from among those who had recently participated in a cognitive rehabilitation intervention study and had agreed to participate in future studies. They had been enlisted via notices in the research page of the National Multiple Sclerosis Society website and contacts with local neurologists.¹¹ To be eligible for the study, participants had to have physician confirmation of their MS diagnosis, and have been diagnosed for at least six months. In addition, participants had to score at least 20 on the Perceived Deficits Questionnaire,¹² which was administered in a telephone screening.

Test Procedure

All procedures were approved by the University institutional review board. Measures were administered in a university research setting. The tests were administered according to the standardized protocols provided by the instrument developers. The tester was trained to administer the neuropsychological tests by an experienced neuropsychologist and his licensed psychological associate. The instrument battery took 90 minutes on average to administer.

Instruments

Developed in a longitudinal study of older adults, the 42-item Everyday Problems Test (EPT) assesses the cognitive ability to reason and solve problems encountered in daily living.⁹ Performance is assessed in seven areas: Meal Preparation/Nutrition, Medications, Phone Use, Shopping, Financial Management, Transportation, and Household Management. The person being tested is presented with directions, charts, or forms, and asked written questions about how to use them. Separate norms are provided for men and women in different age and education levels. Internal consistency reliability coefficients exceeding .80 and a test/retest correlation of .83 have been reported in samples of older adults. Construct validity was established by comparing scores to actual performance of household tasks (.67) and convergent validity was established by comparing EPT performance with performance on other self-report measures of functioning. Significant performance differences were found between elders diagnosed with Alzheimer’s and those without.

The Cognitive Concerns and Cognitive Abilities scales were derived from the Patient-Reported Outcomes Measurement Information System (PROMIS). An initiative of the National Institutes of Health, the aim of PROMIS is to capitalize on recent advances in measurement theory to develop a dynamic and valid patient-reported outcomes system (http://www.nihpromis.org). PROMIS consists of a large item bank that provides researchers with a common item repository that can be administered in print or as computerized adaptive tests. Based on the World Health Organization framework of physical, mental, and social health, nearly 7,000 items available from patient-reported outcome measures in areas such as pain, emotional distress, and physical functioning were reviewed. The items were subjected to quantitative analysis using Item Response Theory and qualitative analysis using cognitive interviewing procedures.¹⁰ A key feature of PROMIS has been to ensure that accessibility for people with disabilities was addressed in its development. The PROMIS Cognitive Abilities Scale consists of 8 items, such as “My thinking has been as fast as usual”. The PROMIS Cognitive Concerns is also an 8-item scale, with items such as “I have had to work harder than usual to keep track of what I was doing.” Items on both scales utilize 5-pt. rating scales from “not at all” to “very much”. Items are summed to create a total score.

Scores on the EPT, Cognitive Concerns, and Cognitive Abilities measures were compared with scores on the following five tests from the MACFIMS battery. These tests have been used extensively to diagnose cognitive impairment in people with MS:

1. The Controlled Oral Word Association Test (COWAT) assesses verbal fluency and word finding.¹³ The number of correct words on three one-minute word-naming trials is combined to yield a total score.

2. The California Verbal Learning Test, second edition, (CVLT-II) assesses verbal memory.¹⁴ Examiners read 16 words and ask participants to repeat as many words as possible. After a 25-minute interval participants are asked to recall the information again without further exposure. Scores on this measure include total recall across the 5 trials and delayed recall.

3. The Brief Visuospatial Memory Test – Revised (BVMT-R) tests nonverbal learning and memory.¹⁵ Participants are asked to reproduce a page with six figures they are shown for 10 seconds on three separate trials. Designs are scored based on accuracy and location scoring criteria. The three free-recall trials are summed, followed by a 25 minute delayed recall trial.

4. The Paced Auditory Serial Addition Test (PASAT) assesses auditory information processing speed and flexibility as well as calculation abilities.¹⁶ One of the most commonly used and most sensitive measures of cognitive function in MS, the PASAT includes 60 trials presented at inter-stimulus intervals of 3 and 2 seconds as recommended by Rao.¹⁷ The total number of correct responses for 3 and 2-minute intervals are reported separately.

5. The Symbol Digit Modalities Test (SDMT) assesses complex scanning and visual tracking.¹⁸ Participants are presented with a series of symbols and digits and instructed to then verbalize the digit associated with each symbol. The number of correct responses in 90 seconds constitutes the score.

Results

After data entry was double-checked, data analyses were conducted using SPSS Version 19. Descriptive statistics and correlations were then computed.

Sample Description

Twenty-nine individuals participated in this exploratory study. The sample was 90% female and 83% indicated they were non-minority White (see Table 1). They had been diagnosed 18 years ago, on average. The average age was 50 years. Sixty-nine percent had at least a college degree. Thirty-one percent were working, but 48% reported being unemployed due to their disabilities. Sixty-nine percent indicated they had relapsing-remitting MS. The average score on the Self-Administered Expanded Disability Status Scale¹⁹ was 5.3.

Table 1.

MS Cognitive Sample Background Characteristics (n=29)

Characteristic	N (%)
Gender
Male	3 (10)
Female	26 (90)
Ethnicity
Hispanic	3 (10)
Non Hispanic	26 (90)
Race
Black	2 (7)
White	26 (90)
Multiple	1 (3)
Education
High School/GED	8 (28)
College	9 (31)
Graduate	12 (41)
Employment
Part/Full Time	9 (31)
Unemployed	20 (69)
Type of MS
Relapsing-Remitting	20 (69)
Progressive	7 (23)
Don’t Know	2 (7)
Average Age	Mean = 49.6, SD = 7.56, range = 33 – 61
Average Years Diagnosed	Mean = 18.2, SD= 8.5, range = 3 – 38
EDSS Score	Mean = 5.3, SD = 1.34, range = 3.5 – 8.5

Characteristics of EPT, Cognitive Concerns, and Cognitive Abilities Scores

Means, standard deviations, and ranges for the EPT, PROMIS Cognitive Concerns, and PROMIS Cognitive Abilities Scales are shown in Table 2. Scores were approximately normally distributed. Cronbach alpha coefficients for the PROMIS Cognitive Concerns and PROMIS Cognitive Abilities were each .94. Two-month test/retest correlations with a subset of the sample were .80 for Cognitive Abilities and .83 for Cognitive Concerns (n=14). The correlation between Cognitive Concerns and Cognitive Abilities was −.80.

Table 2.

Baseline Descriptive Statistics for the Revised Everyday Problems Test, PROMIS Cognitive Concerns, and PROMIS Cognitive Abilities Scales (n=29)

Scale	Mean	S.D.	Range
Revised Everyday Problems Test	23.24	4.79	11–30
PROMIS Cognitive Abilities	24.34	6.60	12–38
PROMIS Cognitive Concerns	25.10	6.95	11–39

Initial internal consistency reliability analysis for the EPT revealed that a number of EPT item/total correlations were low, or could not be computed because some items were answered correctly by all, or nearly all respondents. Consequently, permission was granted by the instrument developer to create a shortened form of the EPT that eliminated 12 of the items. (Willis, personal communication). The reliability coefficient for the revised 30-item version was .83 and the 2-month test/retest reliability was .86 for the 14 individuals tested a second time. The correlation between total scores on the original EPT and the revised EPT was r =.99. The analyses presented here utilize the revised version of the EPT (the EPT-R).

As shown in Table 3, scores on the EPT-R had moderately strong positive correlations with all the neuropsychological tests. The strongest correlations were between the EPT-R and the 3-second PASAT (r=.60) and 2-second PASAT (R=.56). The only correlation that did not reach the .05 level of statistical significance was between the EPT-R and the BVMT delayed recall (r=.31, p=.10). By contrast, EPT-R scores were not correlated with the self-report measures.

Table 3.

Correlations among Revised Everyday Problems Test, PROMIS Cognitive Concerns, PROMIS Cognitive Abilities, and Other Cognitive Tests (n=29)

	Original EPT	Revised EPT	Cognitive Abilities	Cognitive Concerns
Original EPT	1	.99**	.10	−.09
Revised EPT	.99**	1	.11	−.09
CVLT Total Recall	.57**	.50**	.07	−.03
CVLT Delayed Recall	.61**	.55**	.01	−.07
BVMT Total Recall	.42*	.40*	.23	−.15
BVMT Delayed Recall	.33	.31	.31	−.29
Symbol Digit	.51**	.51**	.27	−.16
PSAT 3 seconds	.62**	.60**	.04	−.15
PSAT 2 seconds	.54**	.56**	.34	−.19
COWAT	.46*	.40*	.20	−.18

^*p<.05;

^**p<.01

CVLT=California Verbal Learning Test, BVMT=Benton Visual Motor Test, Symbol Digit=Symbol Digit Modalities Test, PSAT=Paced Auditory Serial Addition Test, COWAT=Controlled Oral Word Association Test.

An interesting pattern emerges with respect to the PROMIS Scales. Scores on the PROMIS Abilities scale were somewhat more highly correlated than PROMIS Concerns scores with neuropsychological test performance, particularly the BVMT, the Symbol Digit, and the 2-second PASAT.

Sensitivity to Change

A subset of this sample (n=14) was retested after using a computer program for eight weeks designed to build their cognitive skills. Paired-t test analyses revealed statistically significant change from pre to post test on the EPT-R and both PROMIS scales (see Table 4). The corresponding effect sizes were moderate to large (Cohen’s d-value of .53 for the EPT, 1.19 for PROMIS Cognitive Concerns, and 1.25 for Cognitive Abilities). These changes should be interpreted cautiously, however, because of the small sample size.

Table 4.

Change in Revised Everyday Problems Test, PROMIS Cognitive Concerns, and Cognitive Abilities Scores Following Computer Practice (n=14)

	Mean	S.D.	T-Value	Cohen D
Time 1 EPT Test	22.14	5.55	2.29*	.53
Time 2 EPT Test	23.93	5.43

	Mean	S.D.	T-Value	Cohen D
T1 Cognitive Concerns	25.00	6.33	−4.45**	1.19
T2 Cognitive Concerns	20.71	5.73

	Mean	S.D.	T-Value	Cohen D
T1 Cognitive Abilities	23.50	5.45	4.69**	1.25
T2 Cognitive Abilities	27.86	5.63

^*p <.05

^**p <.01

Discussion

While previous research has examined the cognitive performance of people with MS using neuropsychological batteries such as the MACFIMS and self-report measures, such as the MSNQ, to our knowledge, performance on the PROMIS and EPT have not previously been reported in people with MS. All three measures demonstrated acceptable psychometric properties in this sample of community dwelling individuals with MS.

The results suggest that the Every Day Problems Test-R may complement the standard neuropsychological tests by assessing cognitive functioning in every day activities in a simple to administer format. While standard neuropsychological testing may still needed to diagnose cognitive impairment, tools such as the EPT may prove a useful adjunct for assessing cognitive performance in day-to day-settings. The revised 30-item version of the EPT yielded reliability coefficients above .80 and was also moderately correlated with standard neuropsychological tests. The EPT-R takes less time to administer than a standard neuropsychological battery thereby decreasing the potential for patient fatigue. Other tests have been developed to test participants’ ability to carry out basic day-to-day functional activities, such as the Direct Assessment of Functional Status or the Rivermead Behavioral Memory Test, but many require specialized equipment and trained administrators. The fact that the EPT-R is a paper and pencil test that can be administered with little formal training makes it more feasible to administer in many settings, where a more formal assessment is not needed.

Our ability to shorten the EPT by approximately 30% while retaining acceptable reliability and validity increases its feasibility as a clinical data collection tool for MS patients. The original 42-item EPT took approximately half an hour on average to complete. Decreasing its length by 30% might shorten the administration markedly, thereby lessening patient fatigue and burden.

The PROMIS Cognitive Abilities and Cognitive Concerns self-report measures also show “promise” as short, easy to administer measures of self-reported cognitive functioning. Their scores were correlated in the expected direction with the neurocognitive tests and show initial evidence of sensitivity to change following a computer intervention designed to build cognitive skills. The 8-item Cognitive Concerns and Cognitive Abilities measures demonstrate good reliability, reflecting the careful item calibration process underlying PROMIS. These scales provide researchers and clinicians alike with brief measures of self-reported cognitive function that minimize the data collection burden on people with MS. Because they are part of the NIH Patient-Reported Outcomes Measurement Information System, using these scales enable comparisons with research investigating outcomes for patients with a variety of chronic conditions.

Although the results must be interpreted cautiously because of the small sample size, results do suggest that the EPT-R and the PROMIS Cognitive Abilities and Cognitive Concerns scales may be sensitive to change following a cognitive intervention. This finding has great significance for researchers who require cognitive measures that can detect meaningful improvement in cognitive functioning following an intervention.

Future investigations of these measures should be conducted with larger and more diverse samples of people with MS. This sample was recruited from individuals in one community, not a clinic population. Sixty-nine percent of them reported that they had relapsing/remitting MS, and they had been diagnosed an average of 18 years ago. While everyone in this study self-reported at least some level of cognitive impairment, this volunteer sample of community dwelling individuals may not be representative of those individuals seeking medical treatment for cognitive impairment. It will also be important to examine the performance of these measures in a sample of individuals with more progressive forms of MS and those more newly diagnosed. Future studies might also incorporate additional exclusion criteria that could impact cognitive functioning, such as certain medications, psychiatric diagnoses, substance use, or other comorbid neurological, medical, or orthopedic conditions. Moreover, future studies should investigate the sensitivity of the EPT-R, Cognitive Abilities, and Cognitive Concerns measures to meaningful change in cognitive functioning following various interventions.

Practice Points.

• The Revised Everyday Problems Test and PROMIS Cognitive Scales are psychometrically sound measures that are brief and feasible for administration in clinical settings.

• The Revised Everyday Problems Test and PROMIS Cognitive Scales demonstrate sensitivity to change following interventions designed to build cognitive skills in people with MS.