Is the Montreal Cognitive Assessment Culturally Valid in a Diverse Geriatric Primary Care Setting? Lessons from the Bronx

Abstract

Background:

Efficacy and validity of the MoCA for cognitive screening in ethnoculturally and linguistically diverse settings is unclear. We sought to examine the utility and discriminative validity of the Spanish and English MoCA versions to identify cognitive impairment among diverse community-dwelling older adults.

Methods:

Participants aged ≥65 with cognitive concerns attending outpatient primary care in Bronx, NY. MoCA and neuropsychological measures were administered in Spanish or English, and a neuropsychologist determined cognitive status (normal with subjective cognitive concerns [SCC], mild cognitive impairment [MCI], dementia). One-way ANOVA compared cognitive statuses. ROC analyses identified optimal MoCA cutpoints for discriminating possible cognitive impairment.

Results:

There were 231 participants, mean age 73, 72% women, 43% Hispanic; 39% Black/African American; 113 (49%) completed testing in English and 118 (51%) in Spanish. Overall MoCA mean was 17.7(SD=4.3). Neuropsychological assessment identified 90 as cognitively normal/SCC, average MoCA 19.9(SD=4.1), 133 with MCI, average MoCA 16.6(SD=3.7), and 8 with dementia average MoCA 10.6(SD=3.1). Mean English MoCA average was 18.6(SD=4.1) versus Spanish 16.7(SD=4.3). The published cutpoint ≤23 for MCI yielded a high false positive rate (79%). ROC analyses identified ≤18.5 as the score to identify MCI or dementia using the English MoCA (65% sensitivity;77% specificity;) and ≤16.5 for the Spanish MoCA (64% sensitivity;73% specificity) in this sample of older adults with cognitive concerns.

Conclusions:

Current MoCA cutpoints were inappropriately high in a culturally/linguistically diverse urban setting, leading to a high false-positive rate. Lower Spanish and English MoCA cutpoints may improve diagnostic accuracy for identifying cognitive impairment in this group, highlighting the need for the creation and validation of accurate cognitive screeners for ethnoculturally and linguistically diverse older adults.

Introduction:

Cognitive impairment disproportionately impacts racially, ethnically, and culturally diverse older adults.^1–4 Common cognitive screening tools used in primary care include the Montreal Cognitive Assessment (MoCA).^5–8 A frequently noted shortcoming of the MoCA is the significant influence of education and cultural factors.^9–11 The MoCA developers added a one-point education correction, though that may not sufficiently mitigate limitations.^6,9 While MoCA cutpoints generally differentiate well between levels of cognitive impairments, individuals with limited education and/or those in traditionally underserved populations tend to perform worse on the MoCA, even in the absence of cognitive impairments.^9,12 Researchers have indicated lower MoCA cutpoints may be more appropriate, particularly in diverse populations.^10,13 To date, only a few studies have evaluated the utility of the MoCA in groups of culturally and ethnically diverse older adults in the US, and few studies have tested the utility and current cutpoints in community-based Spanish speaking/Hispanic individuals in the US. ^9,10,14

In this study, we evaluated whether the currently published English and Spanish MoCA cutpoints were appropriate in a racially/ethnically diverse community-based older-adults with cognitive concerns in Bronx, NY. We hypothesized that currently published MoCA cutpoints would poorly discriminate between those with and without cognitive impairment, as identified by gold standard neuropsychological testing. We secondarily aimed to identify new cutpoints for possible cognitive impairment (i.e., MCI or dementia) in culturally/racially diverse community-based older adults with cognitive concerns.

Methods:

Participants & Study Design

Participants were a subset from the 5-Cog Study, enrolled from May 2019 to March 2020. The 5-Cog Study prospectively enrolled community-dwelling adults aged ≥65 with cognitive concerns who were English or Spanish speaking and attended outpatient primary care visits at an academic medical center in Bronx, NY. Briefly, the 5-Cog Study is a randomized, controlled, single-blind study evaluating whether a 5-minute assessment can improve dementia care by primary care physicians (PCP) versus an active control (Clinicaltrials.gov NCT03816644).¹⁵ Exclusion criteria were prior dementia/MCI, nursing home residency, and visual or auditory impairments limiting ability to complete assessments.

Following enrollment, participants in both study arms had the opportunity to complete neurocognitive testing. Evaluation was carried out in-person by a research assistant under the supervision of a clinical neuropsychologist. All protocols were double scored and neuropsychological measures were normatively scored with appropriately selected English or Spanish norms. The study neuropsychologist (EFW) reviewed test results (but not the MoCA) in the context of questionnaires of daily living activities and mood, and labeled participants as “normal cognition with subjective cognitive concerns (SCC),” “mild cognitive impairment (MCI),” or “dementia,” depending on the participants normative performance relative to estimates of premorbid ability. Participant health characteristics were not collected as part of study protocol.

Cognitive Measures

MoCA version 7.2 was administered in Spanish or English (per participants’ preference) by bilingual research assistants. The MoCA briefly evaluates six cognitive domains and is scored out of 30 points with an additional point given for individuals with twelve or fewer years of education. It has demonstrated good diagnostic accuracy for MCI and dementia. ⁶ While the developers initially suggested a cutpoint of ≤26 to identify individuals at risk of MCI or dementia, researchers including Milani et al. have indicated that lower MoCA cutpoints such as of 23 or 24 for MCI, and 16 to 19 for dementia may be more appropriate, particularly in diverse populations.^10,13

Neuropsychological testing included an approximately 1-hour battery administered in English or Spanish (Wechsler Test of Adult Reading/Word Accentuation Test, Hopkins Verbal Learning Test– R, Boston Naming– Short Form, Controlled Oral Word Association Test, Animals, Symbol Digit Modalities Test, Clock Drawing Test). Additionally, assessments included instrumental activities of daily living questionnaire, AD8 Dementia Screening Interview, The Quick Dementia Rating System (QDRS), and self-report mood measures (Geriatric Depression Scale, Patient Health Questionnaire– 2).^15–26 Cognitive impairment was defined >1.5SD below published normative data and functional impairment was necessary for identification of dementia. MoCA test results were not used in the determination of cognitive impairment.

Data Analysis

Analyses were performed using IBM SPSS Statistics for Macintosh v27.0. Independent sample t-test or Pearson Chi-Square test were used for comparisons between English and Spanish MoCA groups. Pearson correlation compared years of education, age, and MoCA score. One-way ANOVA compared cognitive status groups on the MoCA with post-hoc Bonferroni adjustment. We did not adjust for study arm as analyses were based on baseline measurements. Effect sizes were calculated for outcome variables where appropriate (Cohen’s d for independent sample t-test and eta squared for one-way ANOVA). All inferential testing was two-tailed with alpha set at 0.05. Receiver operating characteristic (ROC) curve analyses were used to determine area under the curve (AUC), and sensitivity and specificity of MoCA cutpoint scores for cognitively normal with SCC versus MCI/dementia for both Spanish and English versions of the MoCA.

Ethics

This study complies with ethical rules for human experimentation, stated in the Declaration of Helsinki and was approved by Albert Einstein College of Medicine institutional review board (2018–9140).

Results:

Participants

Of 1,913 potential participants during this period of the parent study, 815 were approached and assessed for cognitive complaints and 618 endorsed having cognitive complaints. All potential participants who screened positive were offered enrollment but 162 were unable to complete the 5-Cog study on the day of their appointments. Thus, 456 participants were randomized into the parent study and 234 of those individuals agreed to complete a neuropsychological battery. Three participants were excluded due to poor effort and/or the neuropsychologist’s inability to provide a cognitive label. The final sub-study population included 231 older adults, mean age= 72.8 (SD=5.9); 72% women; average education= 11.2 years (SD=4.2). This study population self-identified ethnicity as 42.4% Hispanic and racially as 38.5% Black/African American. Participant demographics, stratified by MoCA language (Spanish or English), are presented in Table 1. There were no significant differences between groups except the English group had significantly higher education (mean 12.6 years) versus the Spanish group (mean 9.8 years) (p<.001). The MoCA was positively correlated with education r(229)=.43, p<.001.

Table 1:

Participant Demographics

Variable	Full Sample (n= 231)	Spanish MoCA (n= 118)	English MoCA (n= 113)
Age, y (mean, SD)	72.8 (5.9)	72.6 (5.8)	72.9 (6.1)
Age, y (range)	65–93	65–93	65–86
Sex, (n female, % female)	167 (72.3%)	85 (72.0%)	82 (72.6%)
Education, y (mean, SD)	11.2 (4.2)	9.8 (4.7)	12.6 (3.1)
Education, y (range)	1-20	1-20	4-20
Race/Ethnicity
Hispanic-only* (n, %)	98 (42.4%)	85 (72.0%)	13 (11.5%)
Black/AA (n, %)	89 (38.5%)	10 (8.5%)	79 (69.9%)
White (n, %)	20 (8.7%)	13 (11.0%)	7 (6.2%)
Other (n, %)	24 (10.4%)	10 (8.5%)	14 (12.4%)
Primary Language
Spanish (n, %)	122 (52.8%)	118 (100.0%)	4 (3.5%)
English (n, %)	107 (46.3%)	0 (0.0%)	107 (94.7%)
Other (n, %)	2 (0.9%)	0 (0.0%)	2 (1.8%)
MoCA Score (mean, SD)	17.7 (4.3)	16.7 (4.3)	18.6 (4.1)
MoCA Score (range)	6-28	6-28	8-28

^*Participants who identified themselves as Hispanic only and no other ethnicity/race.

Language, Race, and Ethnicity

Majority of self-identified Hispanic individuals (87%) chose to complete testing in Spanish. Most self-identified African American/Black individuals (89%) chose English testing. All individuals whose primary language was English completed testing in English (n= 107). Four primary Spanish speakers (4%) chose to complete their evaluations in English (Table 1).

Cognitive Status and MoCA

Overall mean MoCA score was 17.7 (SD=4.3), range 6–28. Ninety (39%) participants were classified by neuropsychological testing as “cognitively normal/SCC,” 133 (58%) as “MCI,” and 8 (3%) as “dementia” (Table 2). The MoCA differentiated significantly between levels of cognitive status F(2,228)=33.902, p<.001 and was significant at all levels of cognitive classification (normal/SCC versus MCI; MCI versus dementia; normal/SCC versus dementia) at p<.001. Individuals identified as cognitively normal/SCC had an average MoCA of 19.9 (SD=4.1), those classified as MCI had an average score of 16.6 (SD=3.7), and those identified as dementia had an average score of 10.6 (SD=3.1).

Table 2:

MoCA Mean Comparisons

Variable	Sample Size	MoCA Mean (SD)	F	p-value	Effect size
Cognitive Status by NPT*			33.9	<.001	0.23
Cognitively Normal/SCC	90	19.9 (4.1)
Mild Cognitive Impairment	133	16.6 (3.7)
Dementia	8	10.6 (3.1)
Race/Ethnicity			3.26	<.001	0.44
Hispanic-only	141	16.9 (4.3)
Non-Hispanic	90	18.8 (4.2)
Primary Language			3.30	0.001	0.44
Spanish	122	16.8 (4.3)
English	107	18.7 (4.1)
MoCA version			3.34	<.001	0.44
Spanish	118	16.7 (4.3)
English	113	18.6 (4.1)

NPT=neuropsychological testing; SCC=subjective cognitive concerns

When identifying cognitive statuses using the Milani et al. MoCA cutpoints, only 23 (10%) participants were classified as cognitively normal (cutpoint of ≤23) and 88 (38%) participants were classified as dementia (cutpoint of ≤16). These rates were discrepant to the number of individuals classified as cognitively normal/SCC (N=90; 39%) or dementia (N=8; 3%) when utilizing neuropsychological tests (Figure 1). The number of people correctly identified as “cognitively normal” on the MoCA when utilizing a cutpoint of ≥24 was 19/90 (21%) with a consequently high false positive rate for MCI/dementia (79%).

Figure 1:

Diagnostic Accuracy of Established MoCA Cutpoints versus Neuropsychological Testing. Legend: Bars represent number of participants; Milani et al. MoCA cutpoints: Dementia= 0 to 16; MCI= 17 to 23; Cognitively Normal= 24+

Language of MoCA

The overall English MoCA average was 18.6 (SD=4.1) and higher than the Spanish MoCA average of 16.7 (SD=4.3); t(229)=3.337, p<.001. Similar findings were apparent when comparing MoCA scores between primary Spanish-speaking as compared to primary English-speaking participants and when comparing self-identified Hispanic versus non-Hispanic participants (Table 2). After controlling for education, language of MoCA test was no longer significant, suggesting education was a primary mediating factor in our sample (p=.205).

ROC Curve Analyses

Due to sample size, MCI and dementia were combined into one “cognitive impairment” group for the ROC analyses. AUC for the English MoCA to identify cognitive impairment was adequate/good (AUC= .785); a cutpoint of 18.5 yielded .646 sensitivity and .774 specificity for cognitive impairment. AUC for the Spanish MoCA was also adequate/good (.789); a cutpoint of 16.5 yielded .644 sensitivity and .729 specificity for cognitive impairment. Comparatively, using previously published cutpoints had good sensitivity but poor specificity for identifying cognitive impairment on the English MoCA and especially poor specificity on the Spanish MoCA (Supplementary Table S1).

Discussion:

In this study, we found both the English and Spanish MoCA had adequate diagnostic accuracy in a population of culturally, educationally, and linguistically diverse older adults presenting to primary care with baseline cognitive concerns. Similar to prior studies, the MoCA adequately differentiated between cognitive groups (i.e., cognitively normal with subjective cognitive concerns [SCC] versus MCI versus dementia).¹⁰ However, as hypothesized, currently published cutpoints poorly discriminated cognitively impaired individuals from cognitively normal/SCC individuals in our ethnically/racially diverse sample, with a very high false positive rate (79%).¹⁰

In busy urban clinics where there may be limited access to subspecialty referrals and/or other resources, overidentifying cognitive impairment is problematic, leading to longer wait times for patients seeking dementia specialty service and causing potentially unnecessary stress to patients and families.^27–29 Given that published cutpoints were quite inconsistent to the number of individuals actually classified as cognitively normal (with SCC) or MCI/dementia by neuropsychological testing, particularly the Spanish MoCA, these cutpoints appear to be inappropriate for multicultural and diverse populations.

As previously reported, we found a positive correlation between years of education and MoCA performance.^9,10 Educational and cultural factors, along with other social determinants of health (e.g., quality of education, poorer employment opportunities, limited opportunities for physical activity, experience of discrimination) may have influenced performance on the MoCA in this diverse population compared to others.^6,30 This appears particularly true for participants who completed the Spanish MoCA, and on average had nearly three years less education than English MoCA participants (mean of 9.8 years versus 12.6 years). Consequently, in many of these individuals a lower performance on the MoCA may be capturing educational or sociocultural factors rather than true cognitive impairment.

Given the observed shortcomings of current cutpoints, the present study sought to evaluate alternative cutpoints for use in diverse urban healthcare systems, particularly those with large Hispanic and Black/African American populations. We found that a cutpoint of 18 on the English MoCA and 16 on the Spanish MoCA may be more appropriate to identify possible cognitive impairment (MCI or dementia) in such a diverse population. These proposed cutpoints are notably imperfect, well-below the current suggested cutpoints, with only adequate sensitivity and specificity. In our Bronx sample a “normal” MoCA score was just above 50% of the total score, indicating that there may be irrelevant or redundant items on this screener, consistent with prior findings that some MoCA items (i.e., cube drawing) are frequently missed in these populations⁹. Cultural and language factors also contribute to the approach and completion of cognitive tests, even when culturally appropriate normative data are used, as tests may be measuring different constructs in different cultural/linguistic groups.^31,32 Thus, clinicians working in culturally/ethnically diverse communities, such as the Bronx, should be cautious when administering and interpreting the MoCA. While researchers attempt to translate or adapt Western cognitive tasks to other cultures and languages, our diverse patients would likely benefit from the creation of culturally appropriate measures, such as the 5-Cog.¹⁵

This study has important strengths. This was a diverse, community-based population. An extensive neurocognitive battery was included, and diagnoses was confirmed by study neuropsychologist. There are also important limitations. Few individuals had dementia (n=8), so unique cutpoints for MCI and dementia could not be separately determined. The study included mostly women which may impact generalizability. Participants were allowed to choose test administration language which can be problematic as participants may overestimate language fluency; however, this was a notably small number of individuals (i.e., four primary Spanish-speakers chose testing in English). We did not take age into consideration for cutpoints due to limited sample size, though age was only weakly correlated to the MoCA in our sample. Finally, all participants had cognitive concerns at baseline, limiting applicability of the determined cutpoints beyond this population. However, generally cognitive assessments are only administered to patient in the clinic who voice cognitive concerns (or have family/caregiver with concerns).^33,34

This study highlights the importance of reconsidering current cutpoints for the MoCA when evaluating culturally and linguistically diverse older adults. As reported, even lower cutpoints had only adequate sensitivity and specificity, suggesting clinicians should be cautious when utilizing Spanish and English MoCAs in diverse populations. Creation of alternative or additional cognitive screening tools attentive to sociocultural considerations may be advantageous for ethnoculturally and linguistically diverse older adults.

Key Points:

• The English and Spanish MoCA have considerable shortcomings as cognitive screening tools for older adults with cognitive concerns in a multiethnic urban primary care setting and clinicians should be cautious when utilizing and interpreting the MoCA in diverse patients.

• The currently published MoCA cutpoints were inappropriately high for this linguistically and ethnoculturally diverse setting (Bronx, NY), with a high false-positive rate for detecting cognitive impairment.

• Utilizing lower cutpoints on the English and Spanish MoCA may yield better diagnostic accuracy for identifying diverse older adults at risk of cognitive impairment.

Why does this matter?

This study highlights the utility and shortcomings of the MoCA as a cognitive screener in community-based ethnically and linguistically diverse older adults and cautions clinicians use and interpretation of this tool in diverse populations.