Montreal Cognitive Assessment as a screening tool

Abstract

Background

We evaluated Montreal Cognitive Assessment (MoCA) performance in a veteran traumatic brain injury (TBI) population, considering performance validity test (PVT) and symptom validity test (SVT) data, and explored associations of MoCA performance with neuropsychological test performance and self-reported distress.

Methods

Of 198 consecutively referred veterans to a Veterans Administration TBI/Polytrauma Clinic, 117 were included in the final sample. The MoCA was administered as part of the evaluation. Commonly used measures of neuropsychological functioning and performance and symptom validity were also administered to aid in diagnosis.

Results

Successively worse MoCA performances were associated with a greater number of PVT failures (ps < 0.05). Failure of both the SVT and at least 1 PVT yielded the lowest MoCA scores. Self-reported distress (both posttraumatic stress disorder symptoms and neurobehavioral cognitive symptoms) was also related to MoCA performance.

Conclusions

Performance on the MoCA is influenced by task engagement and symptom validity. Causal inferences about neurologic and neurocognitive impairment, particularly in the context of mild TBI, wherein the natural course of recovery is well known, should therefore be made cautiously when such inferences are based heavily on MoCA scores. Neuropsychologists are well versed in the assessment of performance and symptom validity and thus may be well suited to explore the influences of abnormal performances on cognitive screening.

The Montreal Cognitive Assessment (MoCA) is a cognitive instrument developed to screen for mild cognitive impairment and dementia.¹ Several studies have demonstrated its psychometric superiority^2,3 over other screening instruments. The MoCA does not comprehensively measure cognition; rather, it provides an estimate of global cognition, assisting clinicians when deciding whether to make referrals for neuropsychological evaluation.¹ Research has demonstrated utility of the MoCA across many neurologic populations.^4–6 Nevertheless, evidence suggests that the MoCA should be not be interpreted solely as a measure of cognitive ability, as doing in many instances would lead to excessive false-positive diagnoses because of noncognitive factors.³

Considerable research in mild traumatic brain injury (mTBI)⁷ suggests that cognitive, physical, and emotional symptoms resolve rapidly in most individuals, with full return to baseline within days to weeks.⁸ Given high rates of reporting cognitive symptoms among those with histories of mTBI^9,10 in the context of evidence that persistent cognitive deficits are unlikely to be directly related to neurologic injury,¹¹ the importance of assessing validity has been highlighted within this population.¹² Establishing both validity of cognitive data via tests specifically designed to assess performance validity and validity of symptom self-report via measures designed to assess symptom validity is crucial to drawing conclusions regarding impairment.¹³

This study was designed to evaluate the influence of suboptimal performance validity test (PVT) and symptom validity test (SVT) performances on scores derived from the MoCA. We hypothesized that indicators of invalid response style (i.e., failure on PVTs and SVTs) would incrementally influence total MoCA scores, in that as the number of such indicators increased, MoCA scores would decrease. We also anticipated a negative association of self-reported distress (e.g., posttraumatic stress disorder [PTSD] symptoms and pain), poor sleep, and cognitive symptoms with MoCA performance.

Methods

Standard protocol approvals, registrations, and patient consents

This study was approved by the Veterans Administration (VA) institutional ethical standards committee and institutional review board as a retrospective analysis of data collected in a clinical context.

Participants

One hundred ninety-eight consecutively referred outpatients at a VA tertiary care medical center TBI/Polytrauma Clinic undergoing comprehensive TBI evaluations over a 2-year period (October 2011 to October 2013). Patients were excluded if they had a TBI of greater than mild severity, as determined by the self-reported duration of loss of consciousness (LOC; moderate-severe TBI = LOC > 30 minutes; n = 11), or if there was insufficient self-reported data about the injury to determine a diagnosis or severity of TBI (n = 9). The mild TBIs were sustained at least months before our evaluations and on average years before their TBI evaluation. Of the patients who reported LOC <30 minutes and/or posttraumatic amnesia duration of <24 hours (n = 178), 123 were diagnosed with mTBI by a physical medicine and rehabilitation physician (i.e., physiatrist) during the evaluation, and 39 were clinically determined not to have TBI (6 cases were “diagnostically unclear,” and 10 cases were missing a physician's diagnosis). One hundred seventeen individuals completed all 4 PVTs and thus comprised the final sample. Most were male (95.7%) and most self-identified ethnically as white (74.4%; black: 6.8%; Latino: 2.6%; Pacific Islander: 0.9%; and biracial/multiracial: 1.7%; 13.7% declined to respond). Patients ranged in age from 21 to 70 years (mean age = 31.33 years, SD = 8.48), with a mean of 13.13 years of education (SD = 1.66; range = 8–18). Nearly half of the patients (48%) reported no LOC. Slightly more than half (53%) reported 1 or fewer episodes of disorientation related to head injury, whereas 35% reported 2–4 episodes, and 7% reported 5 or more episodes.

Cognitive measures

Montreal Cognitive Assessment

The MoCA¹ is a brief cognitive screen comprising 30 scorable items that assess the attention, working memory, short-term memory, delayed memory, visuospatial abilities, executive functioning, language, and orientation to time and place. The MoCA has been shown to be sensitive and specific in detecting cognitive impairment¹⁴ and has been demonstrated to have good criterion and convergent validity.¹⁵

Peabody Picture Vocabulary Test–Fourth Edition

The Peabody Picture Vocabulary Test–Fourth Edition assesses vocabulary acquisition and contains 228 test items, each consisting of 4 full-color pictures as response options on a page,¹⁶ and thus is a useful estimate of IQ. For each item, the test administrator says a word, and the respondent selects the picture that best illustrates the word's meaning. Raw scores for this test were standardized, correcting for age and education.¹⁶ Analyses were conducted using standardized scores.

Trail Making Test (Parts A and B; TMT-A and TMT-B)

The Trail Making Test (TMT) ¹⁷ provides information on visual search, mental flexibility, processing speed, and executive functioning. The TMT consists of 2 parts—A and B. TMT-A requires test takers to draw lines sequentially to connect 25 encircled numbers (numbers 1 through 25) on a sheet of paper. TMT-B is similar except that test takers alternate between sequential numbers and letters (1, A, 2, B, 3, C, etc.). The TMT has been shown to be sensitive to a variety of neurologic impairments and processes and reflects ability to engage in higher-order cognitive functioning.¹⁸ The raw score on each part represents the amount of time used to complete the task.

Distress measures

Neurobehavioral Symptom Inventory

The Neurobehavioral Symptom Inventory (NSI) is a 22-item self-report measure of postconcussive symptoms comprising 4 subscales–sensory, physical, affective, and cognitive.¹⁹ For this investigation, only the 4 cognitive items were used. Respondents rate the extent to which symptoms have bothered them over the past 2 weeks (e.g., difficulty making decisions, poor concentration, and cannot pay attention) using a 5-point Likert-type scale from 0 (none) to 4 (very severe). The NSI has been shown to be a reliable measure of self-reported postconcussive symptoms.²⁰ Cronbach alpha for cognitive symptoms of the NSI was 0.867 in this sample.

Posttraumatic Stress Disorder Checklist–Military Version

The Posttraumatic Stress Disorder Checklist–Military Version (PCL-M)²¹ is a 17-item self-report scale that measures PTSD symptoms (e.g., re-experiencing, hypervigilance, and emotional numbing) that correspond to the PTSD symptoms in the fourth version of the Diagnostic and Statistical Manual of Mental Disorders.²² Each item is rated on a 5-point Likert scale from 1 (not at all) to 5 (extremely) and indicates the degree to which respondents were bothered by symptoms in the past month. This scale has been demonstrated to be a well-validated measure that shows good temporal stability, internal consistency, test-retest reliability, and convergent validity.²³

Pain experience

Current experience of pain was assessed using an item within the Brief Pain Inventory²⁴: e.g., Please rate your pain by circling the one number that tells how much pain you have right now. Respondents rated the question from 0 (“no pain”) to 10 (“pain as bad as you can imagine”).

Hours of sleep

Hours of sleep during the previous night was collected via self-report. This approach has been used by this group previously.²⁵

Performance and symptom validity measures

Wechsler Adult Intelligence Scale–Fourth Edition Digit Span subtest

The Wechsler Adult Intelligence Scale–Fourth Edition Digit Span subtest²⁶ assesses auditory attention and working memory through the repetition and sequencing of digits and consists of 3 individual parts (Digit Span Forward, Digit Span Backward, and Digit Span Sequencing). The Digit Span Scaled Score was derived from converting the total number of correct trials to age-adjusted norms.²⁶ The cutoff score of Digit Span Scaled Score as a PVT for inadequate effort was ≤6, which has been shown to have adequate sensitivity and specificity.^27,28 Reliable Digit Span is an embedded measure of effort calculated by summing the highest number of digits successfully repeated across 2 consecutive same span-length trials of Digit Span Forward (repeating digits in the same order as administered) and Digit Span Backward (repeating the digits in reverse order).²⁹ The cutoff score of Reliable Digit Span as a PVT for inadequate effort was ≤7, which has been shown to have fair sensitivity and good specificity.²⁸ Timed Digit Span is also an embedded measure of effort, calculated by summing the duration (in seconds) of both trials of four-span Digit Span Forward.²⁷ The cutoff score of Timed Digit Span as a PVT for inadequate effort was >3.5 seconds, which has been shown to have adequate sensitivity and specificity.²⁷

Test of Memory Malingering

The Test of Memory Malingering (TOMM) is a forced-choice visual recognition task consisting of pictures of common objects.³⁰ The TOMM is a commonly administered, well-validated test that has been shown to produce excellent classification and diagnostic statistics for poor task engagement in veterans with mTBI³¹ in addition to also having good specificity and sensitivity.³² Patients in this study were administered trial 1 of the TOMM as a PVT. The use of trial 1 of the TOMM has been demonstrated to have high diagnostic accuracy for screening insufficient engagement.^32,33 The cutoff score of TOMM trial 1 as a PVT used in this study was <45 to maximize sensitivity.

Modified Somatic Perceptions Questionnaire

The Modified Somatic Perceptions Questionnaire (MSPQ)³⁴ is a 13-item self-report measure that assesses symptoms of somatization associated with psychological distress (e.g., “mouth becoming dry” and “sweating all over”). Each item is rated on a Likert-type scale from 1 (not at all) to 4 (extremely) and indicates how respondents felt during the past week regarding their somatic complaints. The MSPQ has been shown to have good sensitivity and specificity for detecting exaggerated reports of pain (i.e., symptom validity).³⁵ The cutoff score of the MSPQ as an SVT was >11 (scores of 12 or higher indicate an exaggerated somatization report³⁶).

Procedure

Patients were referred for TBI evaluation as the result of a positive score on a 5-item TBI screen administered during a previous episode of care in the VA system to all returning veterans who served in Operation Enduring Freedom, Operation Iraqi Freedom, and Operation New Dawn. As part of routine care, during the TBI evaluation, patients were administered a neuropsychological screen by trained research assistants under the supervision of licensed clinical psychologists. Demographic and injury-related data were collected during the screen and from review of available medical records.

Data analyses

Data were analyzed using Statistical Package for Social Sciences Version 20.0. See Table 1 for descriptive statistics for MoCA, estimated IQ, emotional distress, and other variables of interest. Missing data were excluded pairwise in the analyses. A t-test was used to examine the differences in MoCA performance between patients with and without TBI diagnoses. Correlational analyses were used to examine relationships between MoCA and self-report measures for the whole sample and for patients who passed PVTs (table 2). Analysis of variance (ANOVA) was used to examine the differences in MoCA performance based on failure of PVTs and SVTs. Nonparametric tests were used to examine relationships between non-normally distributed variables (e.g., age and pain experience).

Table 1.

Descriptive statistics for variables of interest (N = 117)

Table 2.

Bivariate (N = 117) correlations between MoCA and self-report symptom measures (above diagonal); correlations between MoCA and self-report symptom measures in patients who passed PVTs (N = 57) below diagonal

Data availability

Deidentified data are kept on a secure server and will be made available on request for the purposes of replication of findings.

Results

When controlling for performance validity using a Digit Span Scaled Score cutoff of >6 for inclusion in analyses, the MoCA total score was not significantly different in patients with a TBI diagnosis (X = 25.27, SD = 2.12) compared with patients without a TBI diagnosis (X = 25.22, SD = 2.51; t(131) = −0.12, p = 0.90). Controlling again for performance validity, the MoCA total score was found to be associated with estimated IQ (r_s = 0.25, p < 0.01), but not with age (r_s = −0.07, p > 0.05) or years of education (r_s = −0.11, p > 0.05). Similarly, performance on the MoCA was also associated with performance on TMT-A (r = 0.28, p < 0.05) and TMT-B (r = 0.39, p < 0.01).

Of 117 individuals who completed all 4 PVTs, 57 (49%) passed all PVTs; 21% failed 1 PVT, 23% failed 2 PVTs, and 8% failed 3 PVTs; no one failed all 4 PVTs. An ANOVA demonstrated differences in MoCA performance stratified by the number of PVTs failed (F(115, 3) = 9.67, p < 0.01; table 3), with successively worse MoCA performance as the number of PVT failures increased (figure).

Table 3.

Mean differences in the MoCA total score by number of PVTs failed

Figure. Mean MoCA total score by number of PVTs failed and mean MoCA total score by number of PVTs and SVTs failed.

Error bars represent standard error of the mean. MoCA = Montreal Cognitive Assessment; PVT = performance validity test; SVT = symptom validity test.

A similar relationship was found between MoCA performance and validity tests when an SVT (i.e., MSPQ total score) was included in the analysis. Patients who failed both the SVT and at least 1 PVT performed worse on the MoCA (X = 21.11, SD = 4.68) than those who failed 1 or more PVTs but passed the SVT (X = 24.54, SD = 2.16), those who passed the PVTs but failed the SVT (X = 25.04, SD = 2.51), and those who passed both SVTs and PVTs (X = 25.72, SD = 2.0; F(115,3) = 9.67, p < 0.01; table 4, figure).

Table 4.

Mean differences in the MoCA total score by number of PVTs and SVTs failed

Correlational analyses in the overall sample controlling for higher-order cognitive functioning (i.e., TMT-B; partial correlation) demonstrated relationships between the MoCA score and PCL-M total score (r = −0.39, p < 0.01), as well as cognitive symptom reports on the NSI (r = −0.30, p < 0.01). Correlations between estimated hours of sleep and MoCA (r = 0.154) and report of pain and MoCA (r = −0.051) were not significant. Equivalent results were obtained in the adequate PVT performance subsample (MoCA and PCL-M r = −0.31, p < 0.01; cognitive symptom report on the NSI r = −0.30, p < 0.01); based on a Fisher r-to-z comparison, there were no significant differences in the correlations (between MoCA and PCL-M) between the 2 samples (ps = 0.341 and 0.245, respectively).

Discussion

The finding of an association between performance on the PPVT (i.e., an estimate of general intelligence) and MoCA is consistent with previous research that demonstrated an association between overall IQ and MoCA performance in a mixed clinical sample.³⁷ Thus, general intellectual ability, regardless of neurologic impairment, influences MoCA performance. Task engagement, commonly linked to performance validity, and validity of self-report also significantly affected MoCA scores, with cumulative PVT and SVT failures resulting in the lowest MoCA scores.

When only valid data were examined, MoCA scores were associated with tests sensitive to brain injury (e.g., TMT Parts A and B). Therefore, the MoCA is likely an appropriate choice of screening instrument in this context, so long as examiners also assess relevant aspects of validity of the presentation. Overall MoCA scores among valid performers were generally consistent with population norms. Notably, although sleep and pain experience were not associated with MoCA scores, PTSD symptoms and self-reported cognitive problems were. This pattern was evident in both the overall sample and in the adequate PVT performance subsample as well. Thus, even among patients who pass PVTs, psychological factors including illness perception (i.e., subjectively viewing oneself as impaired or distressed) may play a role in performance on objective cognitive tests.

As expected, MoCA scores were affected by performance and symptom invalidity, with lower MoCA scores obtained in individuals who failed performance and symptom validity indicators, with greater decrements in MoCA scores associated with an increasing number of validity test failures. Given previous research that has found that neuropsychological test data are affected more by performance invalidity/suboptimal effort than by neurologic variables,^38,39 this finding is not surprising.

Notably, numerous studies have shown greater symptom report and poorer performance on PVTs in mTBI vs moderate to severe TBI populations. As such a pattern is not consistent with neuropathologic findings, the conclusion reached by specialists in the field is that this apparent paradox reflects symptom magnification in a number of cases. Consideration of that possibility should occur in all clinical settings to ensure appropriate treatment (i.e., management of general distress and concern vs medical treatment).

Clinicians who regularly administer the MoCA as a screening instrument should be aware of the potential influence of both failed performance validity tests and overendorsement of cognitive, physical, and emotional problems on SVTs on cognitive test performances. Low MoCA performance may, in reality, represent poor effort or task engagement rather than impairment or “brain damage”. Without clinician awareness of this potential, substantial iatrogenesis, misdiagnosis (i.e., neurologic vs emotional) and misallocation of resources may occur, for reasons well established in the social science literature (i.e., diagnosis threat and attribution bias).⁴⁰ Thus, the findings of this study support that interpretation of cognitive screening data be tempered by consideration of non-neuropathologic contributions in providing appropriate and adequate care to the veteran and other mTBI populations.

Limitations

The present study has several limitations that merit discussion. We acknowledge that PVTs and SVTs that are more precise in their ability to detect an invalid response style exist and would like to note that our selection of PVTs was based on clinical and logistical demands and available resources in our particular setting (a Midwestern VA medical center). Each of the PVTs chosen, however, is well validated and commonly used. Despite the fact that 3 of the indicators stemmed from the same cognitive subtest (i.e., Digit Span), each indicator (overall Scaled Score, Reliable Digit Span, and Timed Digit Span) taps a separate cognitive function (i.e., overall engagement, consistency within the subtest, and speed of responding). Thus, we believe that our approach to this investigation was appropriate, given our time and resource limitations within a clinical setting.

The generalizability of our results remains to be established through cross-validating research because this sample was comprised exclusively of US military veterans with limited field data available to objectively identify the presence of concussion. The possibility of the experience of multiple concussions and incremental effect on functioning was not the focus of the present study but likely merits future investigation. The implications of our results are likely to be of greater importance in compensation-seeking vs clinical populations, although all evaluations in the VA system can be considered as potentially influential with regard to potential future compensations and should be considered as such with regard to the importance of using PVTs and SVTs.

In sum, the MoCA is a useful and valid cognitive ability screening tool to assess for the possibility of neuropsychological dysfunction. Cautious clinical use of the instrument, however, taking into consideration overall baseline intellectual functioning, distress, and motivation to perform well (or poorly) on cognitive testing, is strongly recommended to minimize erroneous conclusions.

Author contributions

B. Waldron-Perrine developed the study hypotheses, directed the research investigation, authored major portions of the manuscript, and completed some statistical analyses. N. Gabel authored portions of the manuscript, provided editorial assistance, assisted with statistical analyses, and assisted with completion of the tables. K. Seagly assisted with manuscript preparation including editing for scientific content and reference management. A.Z. Kraal assisted with manuscript preparation including preparation of data tables. P. Pangilinan assisted with manuscript preparation from a physician perspective and data collection and analysis. R. Spencer assisted with manuscript preparation and data collection and analysis. L. Bieliauskas oversaw the clinical evaluations and research investigation as PI of the parent study and participated in manuscript preparation.

Study funding

No targeted funding reported.

Disclosure

The authors report no disclosures relevant to the manuscript. Full disclosure form information provided by the authors is available with the full text of this article at Neurology.org/cp.

TAKE-HOME POINTS

• → The MoCA is a useful and valid cognitive ability screening tool to assess for the possibility of neuropsychological dysfunction.

• → Cautious clinical use of the instrument, however, is recommended, particularly in a concussion population in which full resolution of symptoms is expected in days to weeks after the event.

• → Interpretation of performance on the MoCA should take into consideration overall baseline intellectual functioning, psychological distress (including pain and sleep disturbance), and motivation to perform well (or poorly) on cognitive testing.

• → Consideration of these factors in interpretation is essential to minimize erroneous and potentially iatrogenic conclusions.