Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2020 May 1.
Published in final edited form as: J Clin Exp Neuropsychol. 2019 Jan 14;41(4):331–340. doi: 10.1080/13803395.2018.1562048

Validity of the Verbal Concept Attainment Test in Multiple Sclerosis

Ryan Mulligan 1, Michael R Basso 2, Lily Lau 3, Bradley Reynolds 4, Douglas M Whiteside 5, Dennis Combs 6, Robert A Bornstein 7
PMCID: PMC6428607  NIHMSID: NIHMS1520221  PMID: 30642223

Abstract

Objective.

As many as 70% of people with multiple sclerosis (MS) have clinically significant cognitive impairment, and most of these individuals exhibit executive dysfunction. Most research concerning executive dysfunction in MS has focused upon non-verbal measures. The Verbal Concept Attainment Test (VCAT) has demonstrated construct validity as an executive function measure in people infected with HIV and in people with focal brain lesions, but its validity among people with MS is unknown. The current study evaluated the VCAT’s criterion, diagnostic, and ecological validity in people with MS.

Participants and Methods.

A comprehensive neuropsychological battery was administered to 44 healthy individuals and 97 people with MS. Based on existing norms, they were classified as impaired or unimpaired, resulting in 65 people with MS categorized as unimpaired and 32 as impaired. They were administered a battery assessing neuropsychological impairment and disability status.

Results.

The VCAT correlated with most measures of neuropsychological function, but its largest correlations occurred with measures of executive function, working memory, and verbal memory. Regarding classification accuracy, the VCAT achieved satisfactory sensitivity and specificity in identifying neuropsychological impairment in people with MS. The VCAT achieved moderate correlations with measures of disability status.

Conclusions.

The data provide evidence for an optimal VCAT cutoff score for establishing neuropsychological impairment in people with MS, and they demonstrate that the VCAT possesses acceptable criterion, diagnostic, and ecological validity. As such, these data support the inclusion of the VCAT in research and clinical practice involving people with MS.

Keywords: Executive Function, Verbal Abstract Reasoning, Multiple Sclerosis, Criterion Validity, Ecological Validity, Classification Accuracy

In multiple sclerosis (MS), demyelination and lesions to gray matter may result in disabling cognitive dysfunction (Benedict et al., 2017; Bobholz & Rao, 2003). As many as 40 to 70% of people with MS have clinically significant cognitive impairment (Chiaravalloti & DeLuca, 2008) that diminishes ability to manage activities of daily living (Sumowksi et al., 2018). Cognitive deficits involving processing speed, episodic memory, visuospatial perception, and executive function are especially common (Chiaravalloti & DeLuca, 2008).

Executive function is a multifaceted construct, and deficits involving concept formation, cognitive flexibility, ideational fluency, and abstract reasoning are commonly observed (cf. Arnett et al., 1997; Zakzanis, 2000). Abstract reasoning concerns the ability to integrate categories, concepts, rules, and principles and to derive meaning, logic, or organization from experience (Davies & Piovesana, 2015). Such reasoning permits individuals to discern relationships between concepts involving complex information (Ropper, Samuels, & Klein, 2014).

Most investigations concerning abstract reasoning in people with MS have employed non-verbal stimuli. In particular, people with MS perform more poorly than healthy individuals on abstract reasoning measures (cf. Feinstein, O’Connor, Gray, & Feinstein, 1995), such as the Wisconsin Card Sorting Test (WCST; Heaton, Chelune, Tallen, Kay, & Curtiss, 1993) and Category Test (CT; DeFilippis & McCampbell, 1997). The clinical utility of abstract reasoning tests notwithstanding, their reliance upon non-verbal stimuli may diminish their effectiveness, especially among people presenting with abstract verbal reasoning deficits. Most verbally-mediated executive functioning measures place few demands upon complex skills such as abstract reasoning, concept formation, or hypothesis testing. Rather, oral fluency tasks tend to form the basis of verbally-mediated executive function measures in this population (cf. Basso et al., 1996; Benedict et al., 2002; Rao, 1990; Rao et al., 1991). Such ideational fluency measures may fail to detect higher-order deficits involving abstract reasoning.

Furthermore, although MS may present with multiple cerebral lesions, at times the disease may manifest with a greater lesion burden involving one hemisphere more than the other. In circumstances in which the left hemisphere is more strongly affected, MS may yield more impairment involving verbal than non-verbal reasoning (e.g., Stuss & Knight, 2002). Consequently, the usefulness of non-verbal abstract reasoning measures among such patients may be attenuated, and the need for verbally-mediated measures of abstraction in MS may be emphasized.

These concerns notwithstanding, relatively little research has examined verbal concept formation among people with MS. This may be due to the aforementioned absence of verbally-mediated executive function measures in MS, and, more generally, may reflect the general dearth of verbal abstract reasoning measures in clinical neuropsychological batteries (cf. Davies & Piovesana, 2015). Among the few studies employing verbally-mediated abstract reasoning tests, Beatty et al. (1995) administered the Shipley-Hartford Institute of Living Scale (SILS; Zachary, 1986), WCST, and California Card Sorting Test (CCST; Delis, Bihrle, & Massman, 1992) to people with MS and a healthy comparison group. The patients with MS performed worse on verbal abstraction indices from the SILS and had more perseverative errors on the WCST and CCST than the healthy group. The SILS did not correlate with the WCST, but did correlate with CCST perseverations. Notably, the CCST (ultimately supplanted by the Delis Kaplan Executive Function System Card Sorting Test) involves verbal and non-verbal stimuli. The deficits observed by Beatty et al. imply that verbal concept formation is prone to impairment in MS, and this impairment may not be readily detected by the WCST. Although the CCST involves both verbal and non-verbal stimuli, it is not a homogenous verbal abstract reasoning test, and its specificity in detecting verbally-mediated abstract reasoning deficits in MS may be lacking.

The Verbal Concept Attainment Test (VCAT; Rosen, 1962) is an abstract reasoning test that relies exclusively upon verbal stimuli. Each of its 23 items is comprised of three to six lines of words. Each line contains three to five words. Examinees are instructed to select one word from each line that is alike in some way. For example:

CAT  BALL  BUS
PLAY  CAR  CORN
MILK  BIRD  TRAIN
Open in a new tab

For this item, the correct choices are BUS, CAR, and TRAIN; the three words represent modes of transportation. No other combination of words from each line yields a coherent verbal concept. Examinees have 30 minutes to complete the test. Bornstein and Leason (1985) asserted that the test involves verbally-mediated concept formation, problem-solving, and hypothesis testing, rendering it a verbal analogue to non-verbal tasks such as the WCST.

Over a series of studies, Bornstein and colleagues demonstrated that the VCAT possesses satisfactory criterion and convergent validity (Bornstein, 1982; 1983; Bornstein & Leason, 1985). In clinical samples comprised of varied diagnoses, the VCAT correlated moderately and positively with intelligence, WCST, and the Category Test. Notably, Bornstein (1986) administered the VCAT, WCST, and Category Test to groups of patients with circumscribed lesions and found that the VCAT had the best sensitivity and specificity to left frontal lobe lesions. It also has demonstrated construct validity as an executive function measure among people infected with HIV (Bornstein et al., 1993) and in people with mild cognitive impairment (Aretouli, Tsilidis, & Brandt, 2013).

As a measure of executive function, these findings reveal that the VCAT possesses satisfactory criterion validity among a variety of patient populations. As of yet, however, no investigation has evaluated its validity in people with MS. Hence, this study seeks to address this limitation in the literature. To evaluate criterion validity, the VCAT was correlated with measures of executive function and overall level of neuropsychological impairment. Classification accuracy was evaluated by comparing VCAT scores of people with MS to those from a group of healthy individuals, and optimal cutoff scores for identifying neuropsychological impairment with the VCAT were determined. Ecological validity was examined by correlating scores on the VCAT with measures of adaptive function.

Method

Participants

Participants were recruited from support groups and advertisements in the local National Multiple Sclerosis Society Newsletter. Individuals were paid an honorarium for their participation. They were informed that their data would be de-identified and embargoed from release. Thus, their data could not be used for clinical purposes. A board certified neurologist made diagnoses in accordance with the revised McDonald et al. (2005) criteria (Polman et al., 2005). Patients were excluded if they had a psychiatric disorder that preceded onset of MS, current or past substance use disorder, history of learning or developmental disorders, or any neurological disease or injury besides MS. All healthy control group participants were free of neurological, substance, or psychiatric disorder, and none had a history of learning or developmental disorder. The study was approved by the local Institutional Review Board. Participants included 44 healthy individuals and 97 people diagnosed with MS. Demographic information concerning the sample appears in Table 1.

Table 1.

Demographics for Sex, Ethnicity, Age, and Education

Healthy MS Total MS Unimpaired MS Impaired p1
Sex .004**
Female 30 (68.2%) 76 (78.4%) 57 (87.7%) 19 (59.4%)
Male 14 (31.8%) 21 (21.6%) 8 (12.3%) 13 (40.6%)
Ethnicity .26
White 40 (90.9%) 87 (89.7%) 58 (89.2%) 29 (90.6%)
African American 1 (2.3%) 4 (4.1%) 3 (4.6%) 1 (3.1%)
Asian 2 (4.5%) 0 (0%) 0 (0%) 0 (0%)
Hispanic 0 (0%) 1 (1.0%) 0 (0%) 1 (3.1%)
American Indian 0 (0%) 4 (4.1%) 3 (4.6%) 1 (3.1%)
Other 1 (2.3%) 0 (0%) 0 (0%) 0 (0%)
Age .28
Mean (SD) 42.89 (10.54) 45.29 (10.61) 44.54 (11.04) 46.81 (9.68)
Education .08
Mean (SD) 14.32 (2.22) 14.41 (2.29) 14.77 (2.16) 13.69 (2.43)
MS Subtype
62 Relapsing Remitting 47 Relapsing Remitting 15 Relapsing Remitting
9 Secondary Progressive 6 Secondary Progressive 3 Secondary Progressive
2 Primary Progressive 2 Progressive Relapsing 2 Primary Progressive
3 Progressive Relapsing 10 Uncertain 1Progressive Relapsing
21 Uncertain 11 Uncertain
Open in a new tab

Note:

1

Between group differences for the healthy group, unimpaired MS group, and impaired MS Group.

Measures

The Verbal Concept Attainment Test (VCAT; Rosen, 1962).

The VCAT is a 23 item test that assesses abstract reasoning, hypothesis testing, and concept formation. It relies exclusively upon verbal stimuli. Examinees are instructed to select combinations of words that are all alike in some way. Examinees have 30 minutes to complete the test. A total raw score was examined in analyses.

The Wisconsin Card Sorting Test (WCST; Heaton, Chelune, Tallen, Kay, & Curtiss, 1993).

The WCST is a non-verbal measure of abstract reasoning and concept formation that is sensitive to frontal lobe dysfunction. Examinees match stimulus cards to a criterion card according to color, shape, or number of shapes stimulus card. After achieving ten consecutive correct sorts, the correct matching principle changes without warning. Examinees must determine the new matching principle and adjust their responses accordingly. Number of perseverative errors and conceptual level responses were examined.

The Delis Kaplan Executive Function System-Verbal Fluency (DKEFS; Kaplan et al., 2001).

The Verbal Fluency subtests of the DKEFS were administered. It consists of three components, phonemic (i.e. identifying words that start with a specified letter), semantic (i.e. sorting words by a given category), and semantic category switching abilities. Age corrected scaled scores for letter fluency, category fluency, and category switching accuracy were analyzed.

The Paced Auditory Serial Addition Test (PASAT; Rao, 1990).

The PASAT is a measure of information processing speed and auditory working memory. The PASAT presents a series of 60 digits at 3- and 2-second intervals. Examinees attempt to state consecutive sums of two digits, and the total number of correct sums is assessed. Norm-referenced T-scores for Trial 1 were examined.

The Symbol Digit Modalities Test (SDMT; Smith, 1991).

The SDMT assesses speed of information processing and visual working memory. Participants view a set of nine abstract symbols that are uniquely associated with digits. Beneath this set of stimuli, a matrix of symbols without digits appears, and participants write digits associated with the symbols over a 90 second interval. Afterward, participants perform the same task, but they verbalize the number associated with each symbol rather than write it. Number of oral responses was recorded. Norm referenced T-scores were computed.

The Action Fluency Test (Piatt et al., 1999a).

The Action Fluency Test is a verbal fluency measure in which participants name as many action words as possible in a one-minute interval. It possesses satisfactory reliability and validity as an executive function measure (Piatt et al., 1999b; Woods et al., 2005). The total raw score was analyzed.

Wechsler Adult Intelligence Scale IV-Digit Span (WAIS-IV; Wechsler, 2008).

The Digit Span subtest of the WAIS-IV assessed auditory working memory. Sequences of digits are read aloud at a rate of one per second, and participants recited the sequences. An age corrected scaled score was analyzed.

The Boston Naming Test (BNT; Kaplan, Goodglass, & Weintraub, 2000).

BNT assesses word retrieval abilities, and is sensitive to language impairment. Participants view illustrations of common objects and attempt to name them aloud. Total number of correct responses was evaluated.

The California Verbal Learning Test-2 (CVLT-2; Delis, Kramer, Kaplan, & Ober, 2000).

CVLT-2 assesses memory for a 16-item word list that is repeatedly read aloud to the examinee. Norm referenced z-scores for total recall were assessed.

Impairment Index.

An overall impairment index was calculated to quantify the degree of neuropsychological deficit. Scores from the following indices comprised the impairment index: Digit Span Age Corrected Scaled Scores, WCST Percent Preservative Errors and Percent Conceptual Level Responses, BNT Total Raw Score, DKEFS Letter Fluency, Category Fluency, and Total Switching Accuracy Scaled Scores, CVLT-II Total Recall T-Score, PASAT Total Correct T-Score, Symbol Digit Modalities Test Oral Scale Z-Score, and Action Fluency Total Raw Score. Scores were compared to relevant test norms for each of the 11 neuropsychological indices. Performance at or below the 16th percentile was considered impaired (e.g., Heaton et al. 1991). The number of impaired scores was summed. If a participant scored in the impaired range on one third or more of the index scores (i.e., 4 or more), they were classified as impaired. A similar criterion has been widely used in clinical practice (cf. Reitan & Wolfson, 1993) and in research concerning people with MS (cf. Baughman et al., 2015; Rao et al., 1991).

Multiple Sclerosis Functional Composite Scale-Timed 25-Foot Walk (MSFC; Cutter et al., 1999; Fischer et al., 1999; Schwid et al., 1997).

The Timed 25-Foot Walk, a component of the Multiple Sclerosis Functional Composite Scale (MSFC; Cutter et al., 1999; Fischer et al., 1999) assesses disability severity. The time for a participant to walk 25 feet was measured.

The Perceived Deficits Questionnaire (PDQ; Sullivan, Edgley, & DeHouse, 1990).

The PDQ is a self-report measure of cognitive dysfunction among people with MS. It contains 20 questions regarding impaired attention, memory, and organizational skills. The total score was computed and evaluated.

The 36-Item Short Form Health Survey (SF-36; Ware & Sherbourne, 1992).

The SF-36 is a self-report measure that assesses ability to manage activities of daily living. It contains eight subscales, measuring physical functioning, pain, perceptions on one’s health, vitality, limitations on social roles due to both physical, role limitations due to emotional problems, functioning in social settings, and mental health.

The Environmental Status Scale (ESS; Mellerup et al., 1981).

The ESS measures the extent to which MS-related disability affects daily functioning. The ESS consists of 5 items evaluating aspects of work status, financial and economic status, personal residence or home, personal assistance required, transportation, community services, and social activity. Each item is rated on a 5 point scale based on level of functioning. The sum of these scores is calculated. Higher scores indicate a higher impairment to daily functioning.

The Incapacity Status Scale (ISS; Kurtzke, 1981).

The ISS is a twenty item, self-report measure of physical impairment due to MS-related disability. The questions are rated on a 4 point Likert scale and span a variety of topics such as ability to perform daily tasks, vision, speech, fatigue, incontinence, and sexual functioning.

The Chicago Multiscale Depression Inventory (CMDI; Nyenhuis et al., 1998).

The CMDI is a self-report measure of depression. The CMDI consists of 50 items evaluating aspects of depression such as mood, evaluation of self, and vegetative tendencies, and examinees rate how severely they have experienced the symptoms using a 5 point Likert scale. The mood scale was examined in the study.

Procedure

After providing informed consent, participants answered questions concerning medical and psychiatric history. After completing the neuropsychological battery, self-report measures were completed.

Results

Among the patients with MS, 65 were classified as unimpaired and 32 were classified as impaired. Table 1 depicts the demographic characteristics of the groups. One-way analyses of variance (ANOVA) and chi-square tests showed that the healthy group, unimpaired MS group, and impaired MS group were not significantly different in ethnicity, age, or education. A significant difference for gender existed (p = .004), and more women were in both of the MS groups than in the healthy group, consistent with a higher disease prevalence of MS among women than men (Bove & Chitnis, 2014). However, a point-biserial correlation revealed that gender did not correlate with the VCAT (r = −.054, p = .52), suggesting that no systematic bias in VCAT performance existed because of sex differences. Additionally, although education was not significantly different between the groups, the p-value = .08. Thus, we correlated VCAT scores with education, and found no significant relationship (r=.11, p=.22). We also compared the two groups of participants with MS with respect to disease course, and the groups did not differ. The means and standard deviations for the neuropsychological measures and disability indices appear in Table 2. The means and standard deviations for the disability indices appear in Table 3.

Table 2.

Means and Standard Deviations for Neuropsychological Tests

Healthy MS Total MS Unimpaired MS Impaired p1 η2 HSD
VCAT2 <.001 .25 A>B>C
Mean (SD) 21.03 (1.42) 18.76 (2.95) 19.52 (2.67) 17.16 (2.88)
Imp. Index <.001 .68 A&B<C
Mean (SD) .93 (1.42) 2.51 (2.30) 1.15 (1.03) 5.35 (1.47)
Digit Span <.001 .20 A>B>C
Mean (SD) 11.63 (3.04) 9.55 (2.34) 10.17 (2.31) 8.26 (1.95)
WCST PE <.001 .20 A&B<C
Mean (SD) 12.28 (6.14) 15.36 (9.52) 12.35 (6.34) 21.66 (11.87)
WCST CR <.001 .19 A&B>C
Mean (SD) 70.85 (15.77) 64.68 (20.99) 71.4 (16.51) 50.59 (22.60)
BNT <.001 .16 A&B>C
Mean (SD) 56.23 (2.91) 55.55 (3.04) 56.49 (2.15) 53.58 (3.66)
DKEFS LF <.001 .27 A&B>C
Mean (SD) 10.48 (3.05) 9.19 (3.19) 10.45 (2.77) 6.55 (2.29)
DKEFS CS <.001 .30 A&B>C
Mean (SD) 11.80 (2.93) 10.81 (3.54) 12.28 (2.82) 7.74 (2.89)
DKEFS CF <.001 .27 A&B>C
Mean (SD) 12.50 (2.77) 10.56 (3.76) 11.91 (3.09) 7.74 (3.51)
CVLT 2 <.001 .17 A&B>C
Mean (SD) 49.93 (9.80) 44.68 (10.25) 47.57 (9.33) 38.61 (9.52)
PASAT <.001 .24 A&B>C
Mean (SD) 50.80 (9.23) 42.55 (11.92) 46.09 (9.83) 35.13 (12.64)
SDMT Oral <.001 .25 A&B>C
Mean (SD) .95 (1.26) -.39 (1.77) .12 (1.55) -1.46 (1.76)
AF <.001 .26 A&B>C
Mean (SD) 19.95 (4.84) 16.71 (5.00) 18.48 (4.60) 13.00 (3.63)
Open in a new tab

Note:

1

Between group differences for the healthy group, unimpaired MS group, and impaired MS Group.

2

VCAT raw scores are reported here to provide the reader a sense of averages for the groups. However, because of non-normality, VCAT data were transformed using a square root transformation. P-values, effect size estimates, and group contrasts are based on the transformed scores instead of the raw values.

A: Healthy Group, B: MS Unimpaired Group, C: Impaired Group, Imp. Index: Impairment Index, the Wisconsin Card Sorting Test Percent Preservative Errors: WCST PE, The Wisconsin Card Sorting Test Percent Conceptual Level Responses: WCST: CR, The Boston Naming Test Total Raw Score: BNT, the Delis Kaplan Executive Function System Letter Fluency: DKEFS LF, the Delis Kaplan Executive Function System Category Fluency: DKEFS CF, the Delis Kaplan Executive Function System Total Category Switching Accuracy Scaled Scores: DKEFS CS, California Verbal Learning Test-2 Total Recall T-Score: CVLT 2, the Paced Auditory Serial Addition Test Composite Total Correct 1 T-Score: PASAT 1, Symbol Digit Modalities Test Oral Scale Z-Score: SDMT Oral, and Action Fluency Total Raw Score: AF.

Table 3.

Means and Standard Deviations for Disability Measures

Healthy MS Total MS Unimpaired MS Impaired p1 η2 HSD
25 Foot Walk .001 .09 A&B<C
Mean (SD) 4.97 (1.14) 24.00 (50.37) 15.45 (36.70) 41.24 (67.95)
ISS <.001 .35 A<B<C
Mean (SD) .08 (.35) 1.46 (1.18) 1.22 (1.10) 1.97 (1.20)
ESS <.001 .36 A<B<C
Mean (SD) .15 (.48) 2.57 (1.93) 2.22 (1.84) 3.28 (1.92)
SF36 Pain <.001 .12 A>B&C
Mean (SD) 71.55 (25.62) 51.59 (26.75) 54.54 (26.47) 45.59 (26.71)
SF36 Health <.001 .19 A>B&C
Mean (SD) 66.83 (11.84) 51.18 (15.66) 51.54 (15.48) 50.47 (16.23)
SF36 Vitality <.001 .30 A>B&C
Mean (SD) 63.50 (19.49) 33.20 (22.06) 33.31 (22.75) 32.97 (20.94)
SF36 Social <.001 .21 A>B&C
Mean (SD) 89.06 (18.66) 64.43 (25.54) 67.69 (26.14) 57.81 (23.28)
SF36 Emotional <.001 .15 A>B&C
Mean (SD) 94.17 (16.69) 63.92 (38.69) 65.13 (39.72) 61.46 (37.01)
SF36 Mental .002 .09 A>B&C
Mean (SD) 82.40 (12.68) 70.39 (19.58) 71.44 (19.72) 68.25 (19.44)
SF36 Physical <.001 .44 A>B>C
Mean (SD) 88.13 (13.90) 42.22 (30.59) 48.92 (31.16) 28.59 (24.67)
SF36 Role Physical <.001 .24 A>B&C
Mean (SD) 85.63 (31.15) 42.53 (38.72) 46.15 (40.56) 35.16 (34.11)
PDQ Total <.001 .27 A<B&C
Mean (SD) 17.48 (10.24) 35.62 (15.22) 34.86 (14.99) 37.16 (15.82)
CMDI Mood .001 .10 A&C<B
Mean (SD) 46.49 (6.66) 55.58 (14.22) 56.09 (14.73) 54.56 (13.29)
Open in a new tab

Note:

1

Between group differences for the healthy group, unimpaired MS group, and impaired MS Group. A: Healthy Group, B: MS Unimpaired Group, C: Impaired Group, Timed 25 Foot Walk: 25 Foot Walk, The Mental Health Inventory: The Incapacity Status Scale: ISS, the Environmental Status Scale: ESS, The 36-Item Short Form Health Survey: SF36, the Perceived Deficits Questionnaire: PDQ, the Chicago Multiscale Depression Inventory: CMDI.

To control for Type I error, a conservative p<.01 was employed as a criterion for statistical significance. When relevant, Tukey honestly significant difference (HSD) contrasts were employed after significant ANOVA results to delineate differences between groups while protecting against Type I error. One-way ANOVAs showed that significant differences between the three groups occurred on all neuropsychological and self-report measures. As expected because of our post-hoc method of assigning participants to groups, Tukey HSD contrasts showed that the impaired MS patients performed worse than the healthy group (p’s < .01) and unimpaired MS patients (p’s < .01) on all neuropsychological measures (See Table 2). The two MS groups reported more dysfunction on all of the self-report disability measures than the healthy group (p’s < .01). The two MS groups endorsed equivalent levels of symptoms and disability (p’s > .10) except on the ISS, the ESS, and the SF Physical Functioning Subscale (p’s < .01). On these scales, the MS impaired group reported worse outcomes than the MS unimpaired group (See Table 3).

VCAT Performance

VCAT performance of the entire sample ranged from 11 to 23 (M=19.41, SD=2.85). Scores ranged from 18 to 23 (M=21.0, SD=1.4) for the healthy group, 12 to 23 (M=19.5, SD=2.7) for the MS unimpaired group, and 11 to 22 (M=17.0, SD=3.0) for the MS impaired group. Kurtosis and skew values were examined to determine violations of normality. For the overall sample, kurtosis (.57) and skew (−0.99) revealed a mesokurtic distribution with a negative skew. In contrast, kurtosis and skew for the healthy group (kurtosis=−.−.64; skew=−.36), MS unimpaired group (kurtosis=.35; skew=−.87), and MS impaired group (kurtosis=−.52; skew=−.35) failed to reveal non-normality. Although each individual group achieved a normal distribution on the VCAT, when the entire sample was examined, a significantly negative skew which violated normality was observed. This might be expected because of the inclusion of a clinically-impaired group. Nonetheless, to correct for this issue, the VCAT was transformed using a square root transformation. This yielded values ranging from 1.00 to 3.61 (M=2.04, SD=0.65) and normal values for kurtosis (−.47) and skew (.37). This square root transformed variable was analyzed in subsequent analyses of variance and correlations.

Criterion Validity

To assess criterion validity, correlations of the VCAT with other measures of neuropsychological function were computed. To control for Type I error, a conservative criterion of p < .01 was chosen to determine significance. The VCAT correlated highly with the overall impairment index (r = −.50, p < .001). The VCAT correlated moderately with WCST Percent Perseverative Errors and Percent Conceptual Response, CVLT-2 Total Recall, Boston Naming Test Total Raw Score, SDMT, and Action Fluency Total Raw Score with values ranging from .31 to .38 (p’s < .001). The VCAT achieved marginally smaller correlations with PASAT, Digit Span, and DKEFS Category Fluency and Switching (r’s ranged from .22 to .27, p’s < .01 to p=.001). Fisher’s Z-transformations of dependent correlations were computed and reported in Table 4. None was significant. Thus, the magnitude of difference among the correlations is insufficiently large to suggest a unique relationship of the VCAT with executive function measures. Rather, the VCAT correlated significantly with executive function measures and other indices in the battery.

Table 4.

Criterion Validity: Correlations with the Verbal Concept Attainment Test

Measure r
Impairment Index −0.50**
Digit Span Test 0.26*
WCST Percent Perseverative Errors −0.32**
WCST Percent Conceptual Level Response 0.38**
Boston Naming Test 0.32**
DKEFS Letter Fluency 0.17
DKEFS Category Fluency 0.22*
DKEFS Switching 0.25*
CVLT-2 Total Recall 0.37**
PASAT Total Correct 0.27*
SDMT Oral 0.31*
Action Fluency 0.31**
Open in a new tab
*

p<.01

**

p<.001

Note. The Wisconsin Card Sorting Test: WCST, the Delis Kaplan Executive Function System: DKEFS, California Verbal Learning Test-2: CVLT 2, the Paced Auditory Serial Addition Test: PASAT, Symbol Digit Modalities: SDMT. Because of non-normality, VCAT scores were transformed using a square root transformation. Correlations are based on the transformed scores instead of raw values.

Classification Accuracy

To identify an optimal cutoff score for determining neuropsychological impairment, VCAT scores were subjected to a Receiver Operating Characteristic (ROC) analysis. In particular, VCAT scores were used to predict participant classification as neuropsychologically impaired or unimpaired. The VCAT accounted for 80% of the area under the curve with a 95% confidence interval of .72 to .89. In identifying a cutoff score for the VCAT, sensitivity, specificity, and the Youden index were evaluated. To optimize the balance between sensitivity and specificity, a score of 18 emerged as the best cutoff score for determining neuropsychological impairment. With this value, the VCAT achieved 72% sensitivity, 79% specificity, and a .51 Youden index. This value was 2.13 standard deviations below the healthy group’s mean and fell at approximately the 2nd percentile of that group. This information is found in Table 5.

Table 5.

Classification Accuracy: Cut Score for the Verbal Concept Attainment Test

Cut Score Youden Sensitivity Specificity
22 .11 1.00 .11
21 .28 .94 .34
20 .41 .88 .54
19 .45 .78 .67
18* .51 .72 .79
17 .40 .53 .87
16 .26 .34 .92
15 .23 .28 .95
14 .19 .22 .97
13 .17 .19 .98
12 .07 .09 .98
11 .06 .06 1.00
Open in a new tab
*

Optimal cut-off for impairment on the Verbal Concept Attainment Test

Note: Youden: Youden Index

Ecological Validity

To assess ecological validity, the VCAT was correlated with measures of disability. In particular, the VCAT was correlated with the Incapacity Status Scale, Environmental Status Scale, SF-36 Physical Functioning, PDQ Attention, PDQ Retrospective and Prospective Memory, PDQ Planning, and PDQ Total. The VCAT achieved significant correlations with these disability measures (p’s < .01) but was uncorrelated with measures of emotional distress (i.e., CMDI Mood Subscale). These correlations appear in Table 6.

Table 6.

Ecological Validity: Correlation of Disability Measures with the VCAT

Measure r
Timed 25-Foot Walk −.26**
The Incapacity Status Scale −.43***
The Environmental Status Scale −.32***
SF36 Bodily Pain −.12
SF36 General Health Perceptions .10
SF36 Vitality .18*
SF36 Social Role Function .14
SF36 Emotional Role Functioning .15
SF36 Mental Health .12
SF36 Physical Functioning .41***
SF36 Physical Role Functioning .16*
PDQ Attention −.27***
PDQ Retrospective Memory −.28***
PDQ Prospective Memory −.26**
PDQ Planning −.24**
PDQ Total −.28***
CMDI Mood Score −.12
Open in a new tab
***

: p≤.001

**

: p<.01

*

p<.05

Note: Timed 25 Foot Walk: 25 Foot Walk, The 36-Item Short Form Health Survey: SF36, PDQ: the Perceived Deficits Questionnaire, the Chicago Multiscale Depression Inventory: CMDI. The square-root transformed VCAT scores were included in these correlations.

Discussion

In assessing executive function in people with MS, previous research has emphasized the use of non-verbal measures. The current research reveals that the VCAT, a verbally-mediated measure of abstract reasoning, hypothesis testing, and concept formation, possesses compelling criterion, diagnostic, and ecological validity. Indeed, the data show that the VCAT correlated with most measures of neuropsychological function, but its largest correlations occurred with measures of executive function, working memory, confrontation naming, and verbal memory. Thus, as a measure of verbally-mediated executive function in people with MS, the VCAT appears to possess satisfactory criterion validity. Although it correlated with executive function measures, the degree of shared variance was insufficient to suggest that the VCAT is a redundant measure. Correlations with the WCST did not exceed 0.40, indicating that unique variance in executive dysfunction was captured by the VCAT. Thus, these data reveal evidence of convergent validity for the VCAT, but it retained relative uniqueness.

Regrettably, the VCAT did not achieve convincing evidence of divergent validity. In particular, it correlated significantly with measures of verbal learning, confrontation naming, and auditory working memory. The magnitude of these correlations was not significantly different from those between the VCAT and measures of executive function. To some extent, this may reflect an underlying aspect of shared verbal reasoning skill among the VCAT, CVLT-2, Boston Naming Test, and oral version of the SDMT. Perhaps inclusion of a visual reasoning or visual memory test in our battery would have better demonstrated divergent validity for the VCAT. This might be useful to address in future research.

In addition to criterion validity, the VCAT demonstrated acceptable classification accuracy. With a cut score of 18 and lower, the VCAT achieved satisfactory sensitivity and specificity in accurately distinguishing between MS patients with and without neuropsychological impairment. In particular, it achieved sensitivity and specificity of at least .7. The value of 18 fell well below the average range of performance for the healthy group (over 2 standard deviations below the mean) and below-average for the unimpaired MS group (approximately 1 standard deviation below the mean). As such, the VCAT seems to be a valid instrument to assess cognitive impairment in MS.

In addition to criterion validity and classification accuracy, our data reference the ecological validity of the VCAT. In particular, the VCAT achieved moderate correlations with measures of disability status. Higher VCAT scores corresponded with better activities of daily living and diminished disability burden. This was demonstrated across an array of commonly employed measures such as the SF-36, PDQ, ESS, and ISS. As such, VCAT performance appears to be a valid correlate of how well a patient with MS is managing their daily affairs.

Despite the satisfactory criterion, diagnostic and ecological validity demonstrated in this study, certain limitations should be acknowledged. Consistent with established prevalence rates (Bove & Chitnis, 2014), more women than men appeared in the MS groups than in the healthy comparison group. Hence, these data may not generalize robustly to samples comprised of mostly male patients with MS. This possibility notwithstanding, sex failed to correlate with VCAT performance, implying negligible bias according to sex. Additionally, performances at or below the 16th percentile were chosen as the criterion for identifying neuropsychological impairment in the sample, and this may be criticized as an excessively liberal criterion. Nonetheless, the 16th percentile is commonly used in the literature to designate impairment (cf. Heaton et al., 1991), and our data are commensurate with this criterion. Moreover, the VCAT seems to have limited evidence of test-retest validity and there is no alternate form of the instrument, attenuating its utility for serial evaluation. Individuals are permitted to take as long as 30 minutes to complete the measure, and this may render it prohibitive for some clinical applications. Furthermore, these data represent performance of a relatively young to middle-aged sample, and may not generalized to older people with MS. Regardless, the data imply that the VCAT effectively identified individuals who performed poorly on the neuropsychological test battery.

For future research, it might be useful to determine whether lateralized burden load correlates with specific impairment on the VCAT. For instance, inasmuch as patients with greater left hemisphere lesions perform worse on the VCAT than those with relatively more right hemisphere lesions, construct validity would be elaborated for the instrument. Additionally, it would be interesting to more fully evaluate the construct validity of the VCAT in MS. The relationship of the VCAT with other executive function measures (e.g., Iowa Gambling Test, Trail Making Tests A and B, D-KEFS Sorting Test) could be delineated.

The aforementioned limitations and unanswered questions notwithstanding, these data imply that the VCAT may serve as a useful instrument in assessing executive dysfunction in people with MS. Previous literature on abstract reasoning has focused on the instruments which emphasize visual processing (e.g., WCST) (Heaton, Chelune, Tallen, Kay, & Curtiss, 1993). The VCAT may serve to diversify methods of assessing executive function in people with MS. Indeed, because of its emphasis on verbally-mediated concept formation, problem-solving, and hypothesis testing (cf. Bornstein & Leason, 1985), the VCAT holds a relatively unique place among measures of executive function. Although it shared variance with measures of executive function (e.g., WCST), it was not redundant with such measures. As such, the current data imply that aspects of executive function are assessed by the VCAT which are left unmeasured by more commonly-used instruments. Furthermore, the current findings concur with previous studies involving other populations. For instance, The VCAT achieved satisfactory criterion validity and classification accuracy in people with HIV, stroke, and mild cognitive impairment (Aretouli, Tsilidis, & Brandt, 2013; Bornstein et al., 1993). The current data showed the VCAT achieved similarly satisfactory validity in people with MS. The concordance of these findings provides support for the inclusion of the VCAT as a useful measure of executive function in research and clinical practice involving people with MS.

Contributor Information

Ryan Mulligan, University of Tulsa.

Michael R. Basso, University of Tulsa

Lily Lau, University of Tulsa.

Bradley Reynolds, University of Tulsa.

Douglas M. Whiteside, University of Iowa

Dennis Combs, University of Texas at Tyler.

Robert A. Bornstein, The Ohio State University

References

  1. Anderson CV, Bigler ED, & Blatter DD (1995). Frontal lobe lesions, diffuse damage, and neuropsychological functioning in traumatic brain injured patients. Journal of Clinical & Experimental Neuropsychology, 17, 900–908. [DOI] [PubMed] [Google Scholar]
  2. Anderson SW, Damasio H, Jones RD, & Tranel D (1991). Wisconsin Card Sorting Test performance as a measure of frontal lobe damage. Journal of Clinical & Experimental Neuropsychology, 13, 909–922. [DOI] [PubMed] [Google Scholar]
  3. Arnett PA, Rao SM, Grafman J, Bernardin L, Luchetta T, Binder JR, & Lobeck L (1997). Executive functions in multiple sclerosis: An analysis of temporal ordering, semantic encoding, and planning abilities. Neuropsychology, 11(4), 535–544. doi: 10.1037/0894-4105.11.4.535 [DOI] [PubMed] [Google Scholar]
  4. Aretouli E, Tsilidis KK, & Brandt J (2013). Four-year outcome of mild cognitive impairment: The contribution of executive dysfunction. Neuropsychology, 27(1), 95–106. doi: 10.1037/a0030481 [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Basso MR, Beason-Hazen S, Lynn J, Rammohan K, & Bornstein RA (1996). Screening for cognitive dysfunction in multiple sclerosis. Archives of Neurology, 53, 980–984. [DOI] [PubMed] [Google Scholar]
  6. Baughman BC, Basso MR, Sinclair RR, Combs DR, & Roper BL (2015). Staying on the job: the relationship between work performance and cognition in individuals diagnosed with multiple sclerosis. Journal of Clinical and Experimental Neuropsychology, 37(6), 630–640. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Beatty WW, Hames KA, Blanco CR, Paul RH, & Wilbanks SL (1995). Verbal abstraction deficit in multiple sclerosis. Neuropsychology, 9(2), 198–205. Doi: 10.1037/0894-4105.9.2.198 [DOI] [PubMed] [Google Scholar]
  8. Benedict RH, DeLuca J, Enzinger C, Geurts JJ, Krupp LB, & Rao SM (2017). Neuropsychology of Multiple Sclerosis: Looking Back and Moving Forward. Journal of the International Neuropsychological Society, 23, 832–842. doi: 10.1017/S1355617717000959 [DOI] [PubMed] [Google Scholar]
  9. Benedict RHB, Fischer JS, Archibald CJ, Arnett PA, Beatty WW, et al. (2002). Minimal Neuropsychological Assessment of MS Patients: A consensus approach. The Clinical Neuorpsychologist, 16, 381–397. [DOI] [PubMed] [Google Scholar]
  10. Bobholz JA, & Rao SM (2003). Cognitive dysfunction in multiple sclerosis: a review of recent developments. Current Opinion in Neurology, 16(3), 283–288. doi: 10.1097/01.wco.0000073928.19076.844 [DOI] [PubMed] [Google Scholar]
  11. Bornstein RA (1983). Verbal Concept Attainment Test: cross-validation and validation of a booklet form. Journal of Clinical Psychology, 39(5), 7. [DOI] [PubMed] [Google Scholar]
  12. Bornstein RA, Podraza AM, Para MF, Whitacre CC, Fass RJ; Rice RR Jr., Nasrallah HA (1993). Effect of minor head injury on neuropsychological performance in asymptomatic HIV–2 infection. Neuropsychology, 7(2), 228–234. Doi: 10.1037/0894-4105.7.2.228 43-745. [DOI] [Google Scholar]
  13. Bornstein RA, & Leason M (1985). Effects on localized lesions on the verbal concept attainment test. Journal of Clinical and Experimental Neuropsychology 4, 421–429. [DOI] [PubMed] [Google Scholar]
  14. Borkowska A, Drożd W, Jurkowski P, & Rybakowski JK (2009). The Wisconsin Card Sorting Test and the N-back test in mild cognitive impairment and elderly depression. World Journal of Biological Psychiatry, 10, 870–876. doi: 10.1080/15622970701557985 [DOI] [PubMed] [Google Scholar]
  15. Bove R, & Chitnis T (2014). The role of gender and sex hormones in determining the onset and outcome of multiple sclerosis. Multiple Sclerosis Journal, 20(5), 520–526. Doi: 10.1177/1352458513519181 [DOI] [PubMed] [Google Scholar]
  16. Chiaravalloti ND, DeLuca J (2008). Cognitive Impairment in Multiple Sclerosis. The Lancet Neurology, 7, 1139–51 [DOI] [PubMed] [Google Scholar]
  17. Cutter GR, Baler ML Rudick RA, Cookfair DL, Fischer JS, Petkau J, et al. (1999). Development of a multiple sclerosis functional composite as a clinical trial outcome measure. Brain: Journal of Neurology, 122, 871–972. [DOI] [PubMed] [Google Scholar]
  18. Davies G, & Piovesana A (2015). Adult Verbal Abstract Reasoning Assessment Instruments and their Clinimetric Properties. The Clinical Neuropsychologist, 29(7), 1010–1033, DOI: 10.1080/13854046.2015.1119889 [DOI] [PubMed] [Google Scholar]
  19. Davidson PR, Fu Qiang G, Mason WP, Winocur G, & Anderson ND (2008). Verbal fluency, Trail Making, and Wisconsin Card Sorting Test performance following right frontal lobe tumor resection. Journal of Clinical & Experimental Neuropsychology, 30(1), 18–32. Doi: 10.1080/13803390601161166 [DOI] [PubMed] [Google Scholar]
  20. DeFilippis NA, & McCampbell E (1997). The Booklet Category Test professional manual (2nd ed.). Lutz, FL: Psychological Assessment Resources. [Google Scholar]
  21. Delis DC, Kaplan E, & Kramer JH (2001). Delis-Kaplan Executive Function System: Technical Manual San Antonio, TX: Harcourt Assessment Company. [Google Scholar]
  22. Delis DC, Kramer JH, Kaplan E, & Ober BE (2000). California Verbal Learning Test – Second Edition Manual Psychological Corporation, San Antonio, TX [Google Scholar]
  23. Delis DC, Squire LR, Bihrle A, & Massman P (1992). Componential analysis of problem-solving ability: Performance of patients with frontal lobe damage and amnestic patients on a new sorting test. Neuropsychologia, 30, 683–697. [DOI] [PubMed] [Google Scholar]
  24. Demakis GJ (2003). A meta-analytic review of the sensitivity of the Wisconsin Card Sorting Test to frontal and lateralized frontal brain damage. Neuropsychology, 17, 255–264. [DOI] [PubMed] [Google Scholar]
  25. Donders J (2008). A confirmatory factor analysis of the California Verbal Learning Test—Second Edition (CVLT-II) in the standardization sample. Assessment, 5(2), 123–31. [DOI] [PubMed] [Google Scholar]
  26. Feinstein A, O’Connor P, Gray T, & Feinstein K (1999). Pathological laughing and crying in multiple sclerosis: a preliminary report suggesting a role for the prefrontal cortex. Multiple Sclerosis, 5(2), 69–73. [DOI] [PubMed] [Google Scholar]
  27. Fischer JS, Rudick RA, Cutter GR, Reingold SC, & The National M.S. Society Clinical Outcomes Assessment Task Force (1999). The Multiple Sclerosis Functional Composite Measure: An integrated approach to M.S. clinical outcome assessment. Multiple Sclerosis, 5, 244–250. [DOI] [PubMed] [Google Scholar]
  28. Grace J, Stout JC, & Malloy PF (1999). Assessing frontal behavioral syndromes with the Frontal Lobe Personality Scale. Assessment, 6, 269–284. [DOI] [PubMed] [Google Scholar]
  29. Gronwall D, & Wrightson P (1974). Delayed recovery of intellectual function after minor head injury. Lancet, 2(7881), 605–609. [DOI] [PubMed] [Google Scholar]
  30. Heaton RK, Chelune GJ, Tallen JL, Kay GG, Curtiss G (1993). Wisconsin Card Sorting Test Manual Revised and Expanded Odessa, FL: Psychological Assessment Resources. [Google Scholar]
  31. Heaton RK, Miller SW, Taylor MJ, & Grant I (2004). Revised comprehensive norms for an expanded Halstead-Reitan Battery: Demographically adjusted neuropsychological norms for African American and Caucasian adults scoring program Odessa, FL: Psychological Assessment Resources, Inc. [Google Scholar]
  32. Lezak M, Howieson DB, & Loring DW (2004). Neuropsychological assessment (4th ed.). New York: Oxford University Press. [Google Scholar]
  33. Mellerup E, Fog T, Raun N, Colville P, De Rham B, Hannah B, et al. (1981). The socio-economic scale. Acta Neurologica Scandinavica, 64, 130–138. [Google Scholar]
  34. Milner B (1963). Effects of different brain lesions on card sorting. Archives of Neurology, 9, 100–111. [Google Scholar]
  35. Kao C (2014). Exploring the relationships between analogical, analytical, and creative thinking. Thinking Skills And Creativity, 1380–88. doi: 10.1016/j.tsc.2014.03.006 [DOI] [Google Scholar]
  36. Kaplan E, Goodglass H, & Weintraub S (2000). The Boston Naming Test - Second Edition (2nd ed.). Philadelphia: Lea & Febiger. [Google Scholar]
  37. Kurtzke JF (1981). A proposal for a uniform minimal record of disability in multiple sclerosis. Acta Neurologica Scandinavica, 64, 40–47. doi: 10.1111/j.1600-0404.1981.tb05548.x [DOI] [Google Scholar]
  38. Nyenhuis DL, Luchetta T, Yamamoto C, Terrien A, Bernardin L, Rao M, & Garron D (1998). The Development, Standardization, and Initial Validation of the Chicago Multiscale Depression Inventory. Journal of Personality Assessment, 70(2), 386. [DOI] [PubMed] [Google Scholar]
  39. Polman CH, Reingold SC, Edan G, Filippi M, Hartung H, Kappos L, Lublin FD, Metz LM, McFarland HF, O’Connor PW, Sandberg-Wollheim M, Thompson AJ, Weinshenker BG, & Wollinsky JS (2005). Diagnostic criteria for multiple sclerosis: 2005 revisions to the “McDonald Criteria.” Annals of Neurology, 58, 840–846. [DOI] [PubMed] [Google Scholar]
  40. Piatt AL, Fields JA, Paolo A, & Troster AI (1999a). Action (Verb Naming) fluency as a unique executive function measure: Convergent and divergent evidence of validity. Neuropsychologia, 37(13), 1499–1503. [DOI] [PubMed] [Google Scholar]
  41. Piatt AL, Fields JA, Paolo A, Koller WC, & Troster AI (1999b). Lexical, semantic, and action fluency in Parkinson’s disease with and without dementia. Journal of Clinical and Experimental Neuropsychology, 21(4), 435–443. [DOI] [PubMed] [Google Scholar]
  42. Piras MR, Magnano I, Canu EG, Paulus KS, Satta WM, Soddu A, Conti M, Achene A, Solinas G, & Aiello I (2003). Longitudinal study of cognitive dysfunction in multiple sclerosis: Neuropsychological, neuroradiological, and neurophysiological findings. Journal of Neurology, Neurosurgery & Psychiatry, 74(7), 878–885. doi: 10.1136/jnnp.74.7.878 [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Rao SM (1990). A Manual for the Brief Repeatable Battery of Neuropsychological Tests in Multiple Sclerosis. Cognitive Function Study Group of the National Multiple Sclerosis Society National Multiple Sclerosis Society, New York. [Google Scholar]
  44. Rao SM, Leo GJ, Bernadin L, & Unversagt F (1991). Cognitive dysfunction in multiple sclerosis: I, Frequency, patterns, and prediction. Neurology, 41, 685–691. [DOI] [PubMed] [Google Scholar]
  45. Reitan MR & Wolfson D (1993). The Halstead-Reitan neuropsychological test battery: Theory and clinical interpretation (2nd ed.). Neuropsychological Press, Tucson, AZ. [Google Scholar]
  46. Ropper AH, Samuels MA, & Klein J (2014). Principles of Neurology McGraw-Hill Education. [Google Scholar]
  47. Schwid SR, Goodman AD, Mattson DH, Mihai C, Donohoe KM, Petrie MD, Scheid EA, Dudman JT, & McDermott MP et al. (1997). The measurement of ambulatory impairment in multiple sclerosis. Neurology, 49, 1419–1424. [DOI] [PubMed] [Google Scholar]
  48. Smith A (1991). Symbol Digit Modalities Test Los Angeles, CA: Western Psychological Services. [Google Scholar]
  49. Stuss DT, Levine B, Alexander MP, Hong J, Palumbo C, Hamer L, Murphy KJ, & Izukawa D (2000). Wisconsin Sorting Card Test performance in patients with focal frontal and posterior brain damage: Effects of lesion location and test structure on separable cognitive processes. Neuropsychologia, 38, 388–402. [DOI] [PubMed] [Google Scholar]
  50. Stuss DT, & Knight RT (2002). Principles of Frontal Lobe Function Oxford University Press. [Google Scholar]
  51. Sullivan M, Edgley K, & DeHouse E (1990). A survey of multiple sclerosis, part 1: perceived cognitive problems and compensatory strategy use. Canadian Journal of Rehabilitation, 4, 99–105. [Google Scholar]
  52. Sumowski JF, Benedict R, Enzinger C, Filippi M, Geurts JJ, Hamalainen Paivi, Hulst H, Inglese M, Leavitt VM, Rocca MA, Rosti-Otajarvi EM, & Rao S (2018). Cognition in multiple sclerosis: State of the field and priorities for the future. Neurology, 90, 1–10. doi: 10.1212/WNL.0000000000004977 [DOI] [PMC free article] [PubMed] [Google Scholar]
  53. Tracy VL, Basso MR, Marson DC, Combs DR, & Whiteside DM (2017). Capacity for financial decision making in multiple sclerosis. Journal of Clinical and Experimental Neuropsychology, 39(1), 46–57. doi: 10.1080/13803395.2016.1201050 [DOI] [PMC free article] [PubMed] [Google Scholar]
  54. Veit C, & Ware J (1983). The structure of psychological distress and wellbeing in general populations. Journal of Consulting and Clinical Psychology, 51, 730–742. [DOI] [PubMed] [Google Scholar]
  55. Ware JE Jr., & Sherbourne CD (1992). The MOS 36 ItemShort Form Health Survey (SF 36). 1. Conceptual framework and item selection. Medical Care, 30, 473–483 [PubMed] [Google Scholar]
  56. Wechsler D (2008). WAIS-IV technical and interpretive manual San Antonio, TX: Pearson. [Google Scholar]
  57. Woods SP, Scott JC, Sires DA, Grant I, Heaton RK, Troster A, and the HIV Neurobehavioral Research Center Group (2005). Action (verb) fluency: Test-retest reliability, normative standards, and construct validity. Journal of the International Neuropsychological Society, 11, 408–415. [PubMed] [Google Scholar]
  58. Zachary R (1986). Shipley Institute of Living Scale Revised Manual Los Angeles: Western Psychological Services. [Google Scholar]
  59. Zakzanis KK (2000). Distinct neurocognitive profiles in multiple sclerosis subtypes. Clinical Neuropsychology, 15(2), 115–136. [PubMed] [Google Scholar]
  60. Zhao M, Meng H, Xu Z, Du F, Liu T, Li Y, & Chen F (2011). The neuromechanism underlying verbal analogical reasoning of metaphorical relations: An event-related potentials study. Brain Research, 142562–74. doi: 10.1016/j.brainres.2011.09.041 [DOI] [PubMed] [Google Scholar]

RESOURCES