Incremental Validity of Montreal Cognitive Assessment Index Scores in Mild Cognitive Impairment and Alzheimer’s Disease

Abstract

Background/Aims

To evaluate incremental validity of the MoCA index scores and the MoCA total score in differentiating persons with normal cognition vs. mild cognitive impairment or Alzheimer’s disease (AD).

Methods

Effect sizes were calculated for ADNI research participants with normal cognition (n=295), MCI (n=471), or AD (n=150).

Results

Effect sizes for the total score were large (> 0.80) and exceeded the index scores in differentiating those with MCI vs. normal cognition, MCI vs. AD, and AD vs. normal cognition. A combined score incorporating the Memory, Executive, and Orientation indexes also improved incremental validity for all three group comparisons.

Conclusion

Administration of the entire MoCA is more informative than the index scores, especially in distinguishing normal cognition vs. MCI. A combined score has stronger incremental validity than the individual index scores.

INTRODUCTION

The Montreal Cognitive Assessment[1] (MoCA) is a popular test used for cognitive screening in dementia clinics. Traditional interpretation of the MoCA is based upon the summed points across all items, with a maximum of 30 points. Recently, Julayanont and colleagues developed MoCA index scores for the domains of memory, executive function, visuospatial, language, attention, and orientation.[2] The investigators found that both the traditional total score and the Memory index Score, consisting of performance on delayed free, cued, and recognition conditions of the word list, were strong predictors of conversion from mild cognitive impairment (MCI) to Alzheimer’s disease (AD) over an average follow-up of 18 months. Conversion to AD was 90.5% in patients with scores below the cutoff for impairment on both the total score (<20/30 points) and the Memory Index score (<7/15 points), 74.5% when one score was below the cutoff, and 53% when both scores were above the cutoff. While not as strong as the Memory index (area under the curve (AUC) =.66)) in predicting conversion to AD, Julaynont et al. reported that with the exception of the Language Index score (AUC=.58), the AUCs of the other Index scores (range=.61–.63) were also significant predictors of conversion from MCI to AD.

In the current study, we investigated the incremental validity of the MoCA index scores vs. the MoCA total score in differentiating cognitively normal persons and those with MCI or AD. Our goal was to determine the clinical utility of domain-based scoring [2] by calculating effect sizes. Effect sizes reflect the strength of statistically significant differences in performance between groups and can provide a metric for comparing the diagnostic utility of various measures. We expected that raw scores of the derived index measures would significantly differ among the three groups of participants (Cognitively Normal > MCI > AD) and that there would be differences in the strength of the effect sizes.

METHODS

Participants

Data used in the preparation of this article were obtained from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) database (adni.loni.usc.edu). The ADNI was launched in 2003 as a public-private partnership, led by Principal Investigator Michael W. Weiner, MD. The primary goal of ADNI has been to test whether serial magnetic resonance imaging (MRI), positron emission tomography (PET), other biological markers, and clinical and neuropsychological assessment can be combined to measure the progression of MCI and early AD. Inclusion criteria for ADNI require an age range between 55–90 years old, a minimum of 6 years of formal education, fluency in English or Spanish, Hachinski Ischemic Scale [3] scores <4 points (out of a possible 18), and Geriatric Depression Scale[4] Short Form scores <6 points (out of a possible 15). Subjects are excluded if they were taking any medications with anticholinergic properties or if they regularly use narcotic analgesics (>2 doses per week). Additional inclusion/exclusion criteria are available at http://www.adni-info.org/Scientists/ADNIStudyProcedures.aspx.

Diagnosis of cognitively intact, amnestic MCI, or AD in ADNI is based on a cognitive screening battery that includes the Mini-Mental State Examination (MMSE) [5], immediate and delayed recall of the first Logical Memory story (Anna Thompson) from the Wechsler Memory Scale-Revised [6], and the Clinical Dementia Rating (CDR) interview [7] conducted with each subject and their program partner. Based upon performance guidelines in the ADNI protocol, controls are defined as having no significant memory complaints beyond those expected for age, a specified education adjusted cutoff score on Logical Memory delayed recall, a MMSE score between 24–30 points, a CDR score of 0 (including a 0 on the Memory Box score), and intact instrumental activities of daily living. MCI subjects have a memory complaint or a memory problem noted by their partner, a specified education adjusted cutoff score on Logical Memory, a MMSE score between 24–30, a CDR score (including the Memory Box score) of 0.5, and relatively preserved instrumental activities of daily living. Finally, participants with AD have a memory complaint or memory problem noted by their study partner, an education adjusted cutoff score on Logical Memory delayed recall, a MMSE score between 20–26, a CDR score between 0.5–1.0, and meet NINCDS/ADRDA [8] criteria for probable AD.

Procedure

We used the scores from participants’ first administration of the MoCA during their baseline study visit. The ADNI database includes item level data, thereby allowing for calculation of a total score and index scores. MoCA scores were converted into six index scores based on the item combinations used by Julayanont et al. [2]. The Memory Index Score (MIS) consisted of the number of words recalled in delayed free, category-cued, and multiple choice conditions, multiplied by 3, 2 and 1 respectively (0–15 points). The Executive Index Score (EIS) included Trail-Making, Clock, Digit Span, Letter A Tapping, Serial 7 Subtraction, Letter Fluency, and Abstraction (0–13 points). The Visuospatial Index Score (VIS) consisted of Cube Copy, Clock, and Naming (0–7 points). The Language Index Score (LIS) included Naming, Sentence Repetition, and Letter Fluency (0–6 points). The Attention Index Score (AIS) was comprised of Digit Span, Letter A Tapping, Serial 7 Subtraction, Sentence Repetition, and Words Recalled in Both Immediate Recall Trials (0–18 points). Finally, the Orientation Index Score (OIS) included all the Orientation items (0–6 points).

We also calculated a combined score by summing the raw scores on the Memory, Executive, and Orientation indexes (0–34 points).

Statistical Analyses

Analyses of variance and chi-square tests were conducted to evaluate the presence of group differences in demographic variables. Analyses of covariance were used to compare the MoCA total score vs. each index score in differentiating between diagnostic groups (Cognitively Normal vs. MCI; Cognitively Normal vs. AD; MCI vs. AD). Bonferroni posthoc analyses were performed. Effect sizes (Cohen’s d) were calculated as the difference between the means of two diagnostic groups divided by their weighted pooled standard deviations. Effect sizes were judged to be either small (0.2–0.3), medium (0.5) or large (>0.8) according to recommended conventions.[9, 10]

Statement of Ethics

All participants provided written informed consent using forms approved by the institutional review boards at each ADNI site.

RESULTS

The sample included 916 participants (295 cognitively normal, 471 MCI, and 150 AD). Participants with MCI were significantly younger (mean age=71.6, SD=7.5) than those with normal cognition (mean=72.9, SD=6.0) and AD (mean=74.7, SD=8.2), p<.001, although the effect sizes were small (MCI/Normal Cognition: d=.20; MCI/AD: d=0.40). The latter two groups did not significantly differ from each other. There was also a significant difference in years of completed education (p<.01), with cognitively normal controls achieving higher levels (mean=16.6, SD=2.5) than both MCI (mean=16.1, SD=2.6) and AD (mean=15.8, SD=2.7) participants who, in turn, did not differ from each other, although effect sizes again were small (Normal Cognition/MCI: d=0.18; Normal Cognition/AD: d=0.32). There was a smaller percentage of female participants in the cognitively normal group (45.1%) than in the MCI (54.6%) and AD (58.7%) groups, p<.01. All three groups had comparable distributions of race, with Whites being most often represented (>90%), p=.42.

ANCOVAs controlling for age, education, and gender revealed significant differences (p<.001) among the groups in total MoCA scores as well as for each index score. Table 1 shows the demographically unadjusted means and SDs for the groups. Posthoc Bonferroni tests indicated that for all comparisons, the scores of the cognitively normal participants were higher than the scores of both the MCI and AD groups. In turn, the MCI participants consistently outperformed the AD participants.

Table 1.

Means, SDs, and Ranges for MoCA Total Score and Index Scores

	Normal Cognition (n=295)	MCI (n=471)	AD (n=150)
Measure (Maximum Score)
Total Score (30)	25.62 ± 2.51, 16–30	23.23 ± 3.22, 13–30	16.93 ± 4.51, 4–25
Memory Index (15)	10.44 ± 3.52, 0–15	7.56 ± 3.86, 0–15	4.06 ± 2.43, 0–13
Executive Index (13)	11.98 ± 1.20, 8–13	11.24 ± 1.65, 5–13	8.71 ± 2.89, 1–13
Visuospatial Index (7)	6.28 ± 0.90, 2–7	6.01 ± 1.00, 0–7	4.81 ± 1.48, 0–7
Language Index (6)	5.45 ± 0.74, 2–6	5.20 ± 0.96, 1–6	4.18 ± 1.54, 0–6
Attention Index (18)	16.96 ± 1.36, 7–18	16.23 ± 1.90, 8–18	13.51 ± 3.39, 3–18
Orientation Index (6)	5.94 ± 0.25, 4–6	5.67 ± 0.62, 0–6	4.00 ± 1.52, 0–6

Values are means ± standard deviations and ranges. MCI, Mild Cognitive Impairment; AD, Alzheimer’s disease.

p values are <.001 for all scores; all posthoc Sheffé tests indicate significant group differences (Normal Cognition>MCI>AD, p<.001)

The effect sizes for the MoCA total score and for the index scores are displayed in Table 2. As seen, the effect sizes of the MoCA total score were large across all three group comparisons. In addition, these values were larger than the effect sizes of the index scores for every group comparison. For the MCI/AD and AD/Cognitively Normal group comparisons, most index scores had large effect sizes, with the Language index for the MCI/AD group comparison approaching a large effect size (d=0.79). In contrast, for the MCI/Cognitively Normal group comparison, the effect sizes for the index scores ranged from small (Visuospatial d=0.28, Language d=0.29, Attention d=0.44) to medium (Executive d=0.51, Orientation d=0.57), and approached a large effect size for Memory (d=0.78).

Table 2.

Effect Sizes (Cohen’s d) Comparing Diagnostic Groups on the MoCA Total Score and Index Scores

	MCI vs Normal Cognition	MCI vs. AD	AD vs. Normal Cognition
Measure
Total Score	0.83	1.61	2.38
Memory Index	0.78	1.09	2.11
Executive Index	0.51	1.08	1.48
Visuospatial Index	0.28	0.95	1.20
Language Index	0.29	0.79	1.05
Attention Index	0.44	0.99	1.34
Orientation Index	0.57	1.44	1.78
Memory, Executive, and Orientation Index Combined	0.91	1.61	2.65

MCI, Mild Cognitive Impairment; AD, Alzheimer’s disease.

Combined Indexes

Due to the finding that the Memory, Executive, and Orientation indexes had larger effect sizes than the other MoCA indexes, we added the raw scores and calculated new effect sizes to determine if the combined score (out of 34 maximum points) improved diagnostic utility (Table 2, last row). As expected, the combined score in the cognitively normal subjects (mean=28.36, SD=3.80) was significantly higher than the MCI (mean=24.45, SD=4.70), and AD (mean=16.77, SD=4.89) groups who, in turn, significantly differed from each other as well, p<.001. The combined effect size was now large (d=0.91) for the MCI/Cognitively Normal comparison, and it exceeded the effect size of the total score (d=0.83). For the MCI/AD group, the combined effect size (d=2.65) also exceeded the total score (d=2.38). The effect sizes of the total score and combined score were identical for the AD/MCI group comparison (d=1.61).

Table 3 shows the sensitivity, specificity, positive and negative predictive values, and areas under the curve of various cutoffs in detecting MCI vs. Normal Cognition, and AD vs. Normal Cognition. A combined point value of < 23 points for the Memory, Executive, and Orientation Indexes provided optimal sensitivity (92%, 95% CI 86–96) and specificity (90%, 95% CI 86–93) for detecting AD vs. normal cognition, with 138 of 150 AD patients being correctly classified. A combined point value of < 29 points yielded reasonable sensitivity (82%, 78–86) but low specificity (41%, 35–47) for MCI vs. normal cognition, with 388 of 471 MCI patients being correctly classified.

Table 3.

Diagnostic Accuracy of Combined Scores on the Memory, Executive, and Orientation Indexes in Detecting Mild Cognitive Impairment or Alzheimer’s Disease vs. Normal Cognition

Mild Cognitive Impairment
MoCA Cutoff	Sensitivity	Specificity	PPV	NPV	AUC
	95% CI	95% CI	95% CI	95% CI	95% CI
≤27	73 (68–77)	62 (56–67)	75 (72–78)	59 (54–63)	71 (66–76)
≤28	78 (74–81)	50 (44–56)	71 (69–74)	59 (53–63)	73 (69–78)
≤29	82 (78–86)	41 (35–47)	69 (67–71)	59 (53–65)	73 (69–78)
Alzheimer’s Disease
≤23	92 (86–96)	90 (86–93)	83 (77–87)	96 (93–97)	87 (81–94)
≤24	95 (91–98)	84 (79–88)	75 (70–79)	97 (94–99)	89 (85–94)
≤25	97 (93–99)	76 (71–81)	68 (63–72)	98 (96–99)	91 (88–95)

PPV, Positive Predictive Value; NPV, Negative Predictive Value; AUC, Area Under Curve; CI, Confidence Interval

DISCUSSION

The results of the current study replicate prior findings demonstrating the utility of the MoCA total score as an initial screen for MCI and AD.[1, 11–14] The total score had large effect sizes, and these values increased in distinguishing MCI vs. Cognitively Normal (d=0.83), AD vs. MCI (d=1.60), and AD vs. Cognitively Normal (d=2.38) groups. The large effect sizes in the current study, based on a research sample screened for MCI and AD, replicates the findings in clinic samples. For example, Larner [12] reported large effect sizes for the MoCA in distinguishing persons with dementia vs. no dementia (d=1.80) and MCI vs. no dementia (d=1.45). Since the diagnosis of MCI requires evidence of relatively preserved instrumental activities of daily living,[15] it is not surprising that the MCI group would be hardest to distinguish from those with normal cognition who also have preserved IADLs.

Although the point values of the total score and the index scores were significantly different at p<.001 for all comparisons, examination of the effect sizes provided information about the clinical relevance of these statistical differences. These analyses revealed that the effect sizes of the total score were larger than every index score. In addition, while the Memory index approached a large effect size for the MCI vs. Cognitively Normal groups, the remaining effect sizes ranged from small to medium. In the initial cognitive screening of individuals with subtle deficits, the administration of the entire MoCA may be more informative than the index scores, especially in distinguishing those with normal cognition vs. MCI.

We evaluated whether combining the scores on the MoCA indices improved incremental validity relative to relying on the separate indices. We combined the raw scores on the Memory, Executive, and Orientation indices since individually, these had the highest effect sizes for all three groups, and moreover, the items comprising these indices did not overlap. We found that the effect size of a combined score was larger than their separate index scores, and it was either comparable to the total score (MCI vs. AD, d=1.61) or exceeded the effect sizes for the total score (MCI/Cognitively Normal, d=0.91 vs. 0.83; AD vs. Cognitively Normal, 2.65 vs. 2.38). In our study, the combined MoCA score had good sensitivity and specificity for distinguishing AD and normal cognition. However, while sensitivity was high for distinguishing MCI and normal cognition, specificity was weak. The latter indicates that the combined index score is sensitive in detecting cases of mild cognitive impairment but will also incorrectly classify persons with normal cognitive functioning as being impaired. Overall, the results of this investigation suggest that if a person’s cognitive status is unknown and the deficits are subtle such as is often the case in persons with MCI, the administration of the entire MoCA may be prudent. The shortened version combining items from the Memory, Executive, and Orientation Indices may be substituted, but will only eliminate three items (naming, sentence repetition, and cube copy), thus saving a few minutes of administration time at best.

Limitations of the current study include the evaluation of the MoCA index scores in a highly selected group of participants in ADNI who have amnestic single domain or multidomain MCI. It is therefore not surprising that the MoCA items measuring memory and orientation are especially sensitive to cognitive difficulty. Future studies should include a broader representation of MCI subtypes to see if the effect sizes show a different pattern. In addition, our study is limited to an evaluation of the index scores in a group screened for Alzheimer’s disease as the primary etiology. Future studies should include a variety of etiologies for cognitive impairment such as frontotemporal or vascular dementia. Finally, Julayanont and colleagues[2] derived the index scores for the purpose of tracking those persons vulnerable to conversion from MCI to AD. This may be the most useful indication for deriving the index scores and should be replicated in other longitudinal samples.