The MoCA 5-min protocol is a brief, valid, reliable and feasible cognitive screen for telephone administration

Abstract

Background and Purpose

The NINDS-CSN vascular cognitive impairment (VCI) Harmonization working group proposed a brief cognitive protocol for screening of VCI. We investigated the validity, reliability and feasibility of the Montreal Cognitive Assessment 5-minute protocol (MoCA 5-min protocol) administered over the telephone.

Methods

Four items examining attention, verbal learning and memory, executive functions/language and orientation were extracted from the MoCA to form the MoCA 5-min protocol. One hundred and four patients with stroke or TIA, including 53 with normal cognition (CDR 0) and 51 with cognitive impairment (CDR 0.5 or 1), were administered the MoCA in clinic and a month later, the MoCA 5-min protocol over the telephone.

Results

Administration of the MoCA 5-min protocol took five minutes over the telephone. Total score of the MoCA 5-min protocol correlated negatively with age (r=-0.36, p<0.001) and positively with years of education (r=0.41, p<0.001) but not with gender (rho=0.03, p=0.773). Total scores of the MoCA and MoCA 5-min protocol were highly correlated (r=0.87, p<0.001). The MoCA 5-min protocol performed equally well as the MoCA in differentiating patients with cognitive impairment from those without (AUC for MoCA 5-min protocol=0.78; MoCA=0.74, p>0.05 for difference; Cohen's d for group difference 0.801.13). It differentiated cognitively impaired patients with executive domain impairment from those without (AUC=0.89, p<0.001; Cohen's d=1.7 for group difference). 30-day test-retest reliability was excellent (Intraclass correlation coefficient=0.89).

Conclusions

The MoCA 5-min protocol is a free, valid and reliable cognitive screen for stroke and TIA. It is brief and highly feasible for telephone administration.

Background

Vascular cognitive impairment (VCI) refers to cognitive dysfunction with an underlying vascular etiology. It is common in the aging population and is particularly prevalent in patients with stroke.^{1, 2} Cognitive assessment is integral in the diagnosis and management of VCI. Screening instruments for VCI should be sensitive to mild impairment and neuropsychological features of VCI. In 2005, the National Institute of Neurological Disorders and Stroke-Canadian Stroke Network (NINDS-CSN) VCI Harmonization Working Group recommended a set of three neuropsychological protocols to serve different purposes in VCI assessment. The 60-minute protocol and the 30-minute protocols were designed for detailed cognitive profiling and clinical screening, whereas a very brief protocol (∼5 minutes) was proposed to serve as a screen at bedside, busy clinics and over the telephone.¹ The latter protocol aims to increase access for cognitive assessment for patients living in remote areas or for those not able to attend clinical follow up for various reasons (e.g. homebound); and to support telemedicine stroke service and large epidemiological and clinical research.

The Montreal Cognitive Assessment (MoCA) has been recommended as a clinical screening instrument for VCI. It is valid and reliable in the patients with VCI including stroke, subarachnoid hemorrhage and stroke-free persons with vascular risk factors.^3-6 However, its paper-and-pencil test format requires the examinee be physically present for examination. It is also too lengthy (∼15 minutes) to be used as a very brief screen. There are at least three abbreviated versions of the MoCA that has been developed and examined in stroke patients (table 1).^{4, 7} In accordance to the structure of the 5-minute protocol proposed by the NINDS-CNS Harmonization workshop, we constructed the MoCA 5-min protocol by extracting four subtests from the MoCA. The objective of this study is to examine the validity and reliability of the MoCA 5-min protocol administered over the telephone.

Table 1. Brief description of abbreviated versions of MoCA examined in stroke/TIA patients.

Author	Mai et al.7	Pendlebury et al.4		Wong et al. (current study)
Name	Short MoCA	T-MoCA	T-MoCA Short	MoCA 5-min Protocol
Study sample	25 Stroke; 24 TIA; 16 Migraine; 10 Cognitive Impairment; 6 Epilepsy; 21 ‘Others’	Stroke and TIA: 41 no MCI; 15 Single domain MCI; 12 Multiple domain MCI		Stroke/TIA: 53 Cognitively normal; 51 Cognitively impaired (CDR 0.5-1)
Subtests	Clock Drawing	Verbal Fluency	Verbal Fluency	5-word learning (as attention measure)
	Abstraction	Orientation	Orientation	Verbal Fluency
	5-word recall	Digit span	5-word recall	Orientation
		Attention		5-word recall
		Calculation
		Repetition
		Abstraction
		5-word recall
Maximum Score	10	22	12	30
Ability for telephone use	No	Yes	Yes	Yes

Abbreviation: MCI=Mild Cognitive Impairment

Methods

Participants are patients with stroke or TIA recruited in the ongoing STroke Registry Investigating Cognitive DEcline (STRIDE) study,⁸ which is an ongoing 5 year longitudinal study to evaluate the rate and predictors of delayed cognitive decline in a consecutive cohort of 1,013 patients (mean age 69.6 [SD=11.7]; 44% female) admitted to a major regional hospital in Hong Kong for stroke and TIA between 2009 and 2010. Patients with prestroke dementia were excluded from the STRIDE study. In the current substudy, participants were selected through stratified random sampling by cognitive status based on Clinical Dementia Rating (CDR).⁹ Patients with moderate or severe dementia, as defined by CDR ≥2, were excluded. Patients had adequate sensorimotor and language capacity to complete cognitive testing, as determined at entry into the STRIDE study. Approval from the CUHK-NTEC Ethics Committee was obtained for the STRIDE study and all patients gave written informed consent for participation. Proxy consents (adult children for four patients, spouse for one) were obtained in patients with dementia who lacked capacity to give informed consent. Study assessments were administered in the Cantonese language.

Description of the MoCA 5-min protocol

The MoCA 5-min protocol consists of four subtests examining five cognitive domains, including attention, verbal learning and memory, executive functions/language, and orientation. In this study the MoCA 5-min protocol was derived from the Hong Kong version of the MoCA.⁵ These items are selected for domain specificity for VCI, brevity, and feasibility for administration over the telephone. The stimuli of four test items remained the same in the MoCA 5-min protocol. Several modifications were made with regard to the scoring of these items. First, the number of words recalled in the first immediate recall trial of the 5-word learning was adopted as a measure of immediate auditory attention span. Second, outputs in verbal fluency test were scored incrementally instead of categorically using a single predetermined cut off. Note that while phonemic fluency was proposed in the English version,¹ semantic fluency (animal) was used in study because the Cantonese language is based on a non-alphabetic system.³ Third, different weightings were given to free recall and cued recall or recognition to reflect different types of memory failure (i.e. encoding vs. retrieval deficits). Total scores of the MoCA 5-min protocol range between 0 and 30. Table 2 shows a description of the MoCA 5-min protocol.

Table 2. Description of MoCA 5-min protocol.

Cognitive Domain	Item	Description	Scoring	Maximum Score
Attention	1	Immediate recall of 5 words	1 point for each word correctly recalled in first trial	5
Executive Functions/Language	2	1-minute verbal fluency (animal category)	0.5 point for each correct output	9
Orientation	3	6-item date and geographic orientation	1 point for each correct response	6
Memory	4	Delayed recall and recognition of 5 words learned in item 1	2 points for each of the word spontaneously recalled; 1 point for each word by cued recall or recognition but not spontaneously recalled	10
			Total	30

Clinical assessment of cognitive functions

As part of their annual clinical assessment in the STRIDE study, participants were administered the Hong Kong version of the MoCA⁵ and the Cantonese Mini-Mental State Examination (MMSE)¹⁰ for psychometric evaluation of cognitive functions. The Clinical Dementia Rating (CDR) was assessed using a semi-structure interview format with informants to provide a global rating of cognitive status based on clinical history.⁹ Based on the CDR score, participants were classified as Cognitively Normal (CDR 0) or Impaired (CDR 0.5 or 1).

Administration of MoCA 5-min protocol over telephone

Patients who agreed to participate were administered the MoCA 5-min protocol over the telephone one month post clinical visit. Test order was the same for all participants. Before the administration on the MoCA 5-min protocol, participants were advised to turn off the radio and TV and to go to a quiet area for the test. Whenever possible, family members were explained on the phone the purpose of the test and were encouraged to remove any distractions and aids (e.g. wall calendars) before the test administration. Performance on the MoCA 5-min protocol was not considered in the rating of the CDR.

Statistical Analysis

Bland-Altman analysis was used to calculate the mean difference and 95% limits of agreement (LoA) between the total scores of the MoCA and MoCA 5-min protocol. Interscale agreement was indexed by the mean difference and LoA (i.e. the smaller the better agreement). Pearson correlation coefficient (r) was also used to examine the correlations between the total scores of the MoCA 5-min protocol with age, education and total MoCA scores. Relationship between MoCA 5-min protocol and gender was examined with Spearman correlation coefficient (rho). Analysis of Covariance (ANCOVA) was used to compare scores of the MMSE, MoCA and MoCA 5-min protocol between the cognitively normal and impaired groups with age and education adjusted. The Cohen's d statistic was used as a measure of effect size of group difference. Cohen's d ≥0.8 is classified as large effect size.¹¹ To examine the external validity of the MoCA 5-min protocol in detecting cognitive impairment, Receiver Operating Characteristics Curve (ROC) was used to examine the ability of the two tests to differentiate the cognitively impaired group from the cognitively normal group. Areas under ROC curve (AUC) of total scores of the MoCA and MoCA 5-min protocol were compared.¹²

We also calculated the executive and memory domain scores of the MoCA according to the method by Lam et al.¹³ To determine the ability of the 5-min protocol in screening for executive impairment, cognitively impaired patients (i.e. CDR ≥0.5) was further categorized based on the presence of executive impairment, which is defined as 1.5 standard deviation below the mean of MoCA executive domain score in 50 healthy controls with comparable demographics (mean age 69.2 years, 50% female, education 7.3 years) in our previous study.⁵ We compared the 5-min protocol score between cognitively impaired patients with and without executive impairment using ANCOVA with age and education adjusted. Furthermore, we performed a ROC analysis to examine the ability of the 5-min protocol to differentiate patients with executive impairment from those without. Moreover, to evaluate the advantage of the weighted scoring for delayed memory, Pearson correlation was calculated between the delayed memory score and spontaneous recall (without adding score for correct cued recall and recognition) of the MoCA 5-min protocol with the memory domain score of the MoCA.¹³

Thirty participants completed a second MoCA 5-min protocol over the telephone 30 days after the first administration to assess the test-retest reliability indexed by the intraclass correlation coefficient (ICC). Internal consistency of the four items of the MoCA 5-min protocol was measured by the Cronbach's α.¹⁴ Statistical analyses were performed using SPSS Statistics version 21.

Results

One hundred and four patients with ischemic stroke or TIA participated in the study. Mean number of days between index hospital admission and clinical assessment was 39.4 (SD=7.6) months. Table 3 shows the demographic and clinical features and cognitive performance of the participants. Approximately half of the participants (n=51; 49%) had cognitive impairment. Cognitively normal and impaired groups were comparable in age, years of education, gender and stroke severity, as measured by the National Institute of Health Stroke Scale.

Table 3. Clinical and cognitive performance of study participants.

		Normal	Cognitively Impaired	p
N		53	51
Age in years		68.9 (10.1)	70.8 (9.2)	0.341
Female		27 (51%)	19 (37%)	0.16
Education in years		6.3 (4.4)	6.0 (4.5)	0.77
NIHSS at admission*		2.0 (4.0)	3.0 (4.3)	0.148
MMSE		27.8 (2.0)	24.6 (4.8)	<0.001	Cohen's d
MoCA		24.1 (3.8)	19.2 (6.3)	<0.001	0.97
Mini-MoCA	Total	26.0 (3.2)	20.8 (6.0)	<0.001	1.13
	Memory	4.2 (0.8)	3.4 (1.2)	<0.001	0.80
	Verbal Fluency	7.1 (1.9)	5.4 (2.3)	<0.001	0.81
	Orientation	5.8 (0.4)	4.9 (1.5)	<0.001	0.95
	Delayed Memory	8.9 (1.3)	7.1 (2.4)	<0.001	0.97

Cohen's d is a measure of effect size of test scores compared between the cognitively normal and impaired groups

Scores of the MMSE, MoCA and MoCA 5-min protocol between the cognitively normal and impaired groups were compared with age and education adjusted

^*Data shown in median (interquartile range) and compared using Mann-Whitney U Test

In general, administration of the MoCA 5-min protocol and the MoCA took five and 12 minutes, respectively. Total score of the MoCA 5-min protocol negatively correlated with age (r=-0.36, p<0.001) and positively with years of education (r=0.41, p<0.001) but not with gender (rho=0.03, p=0.773). Bland-Altman analysis showed that the total scores were on average 1.8 points higher on the MoCA 5-min protocol than the MoCA (figure 1). The 95% LoA between the MoCA and MoCA 5-min protocol (calculated as MoCA total - MoCA 5-min protocol total) were between -7.4 and 3.8. Figure 2 shows a scatter plot of total scores of MoCA and 5-min protocol for the whole sample. The total scores of the two tests were highly correlated (r=0.87, p<0.001). Compared to the cognitively normal group, the cognitively impaired group performed more poorly in the MoCA and MoCA 5-min protocol as well as in all items scores of the MoCA 5-min protocol (p<0.001 for all). Cohen's d ranged between 0.80 and 1.13 for the total and item scores of the MoCA 5-min protocol and 0.97 for the MoCA total score. AUCs for the MoCA and MoCA 5-min protocol were 0.74 (SE=0.05) and 0.78 (0.05), respectively and not significantly different from each other (z=0.64, p>0.05). Excluding patients with dementia (CDR 1; n=5) resulted in a slightly lower AUC (0.75 [0.05]). Eighteen (35%) of 51 patients with cognitive impairment had impaired executive domain score on the MoCA. Those with executive domain impairment on the MoCA had significantly lower scores on the MoCA 5-min protocol (MoCA 5-min protocol mean=15.6 [SD=5.2]) than those without (23.6 [4.2], p=0.001, Cohen's d=1.7). AUC for the MoCA 5-min protocol to differentiate the group with executive domain impairment was 0.89 (SE=0.045, p<0.001). The delayed memory score of the MoCA 5-min protocol had slightly higher correlation (r=0.73, p<0.001) than the spontaneous recall score (r=0.68, p<0.001) with the MoCA memory domain score.

Figure 1.

Bland-Altman plot showing mean difference between total scores of the MoCA and MoCA 5-min protocol. The two dotted line covers the 95% limits of agreement between the two tests.

Figure 2.

Scatter plot of total scores of MoCA and 5-min protocol for the whole sample.

Thirty participants completed a second MoCA 5-min protocol 31.7 (6.7) days after the first administration of the MoCA 5-min protocol. Test-retest reliability was excellent (ICC=0.89, 95% confidence interval 0.77 to 0.95, p<0.001). Cronbach's α measuring internal consistency of the four items of the MoCA 5-min protocol was 0.79.

Discussion

In this study we showed that the MoCA 5-min protocol administered over telephone was equally useful as the clinically administered MoCA in detecting cognitive impairment in patients with stroke or TIA. Excluding patients with mild dementia (n=5) only resulted in a minimal reduction in diagnostic accuracy. Inter-scale agreement between the two tests was also good. Effect size between the cognitively normal and impaired groups was even slightly larger for the MoCA 5-min protocol than MoCA (Cohen's d 1.15 [MoCA 5-min protocol] vs. 0.97 [MoCA]). In addition, although it appears that the MoCA 5-min is weighted toward memory and orientation, we showed that it is highly accurate in detecting patients with executive impairment. The MoCA 5-min protocol also exhibited excellent test-retest reliability and acceptable internal consistency.

Telephone-based cognitive tests serve to increase accessibility for testing and minimize missing data due to inability to attend clinical visits, thus offering multiple benefits for clinical and research purposes. A number of telephone-based tests have been developed for clinical use and epidemiological studies.^{4, 15, 16} The MoCA was recommended by the NINDS-CSN VCI working group as the choice of test for VCI screening. It has demonstrated good psychometric properties across a wide range of VCI related conditions and is free for clinical and research use (at the time of this writing). The MoCA 5-min protocol was derived from the MoCA by extracting items with domain-specificity for VCI, brevity and feasibility for telephone administration. Several abbreviated versions of the MoCA have been developed (Table 1). For example, Mai et al. showed that a combination of clock drawing, delayed recall and abstraction items performed better than the combination proposed by the NINDS-CSN working group in predicting impaired MoCA performance.⁷ However, their version precludes the possibility for telephone administration as drawing is required. Pendlebury et al. developed two telephone versions of MoCA (T-MoCA) of different lengths, including a short version that is composed of the set of items recommended by NINDS-CSN working group. This short version included phonemic fluency, word recall and orientation with a maximum score of 12. It was shown to be valid in detecting stroke or TIA patients with mild cognitive dysfunction.⁴ The MoCA 5-min protocol exhibits several strengths over other abbreviated MoCA versions. First, the MoCA 5-min protocol capitalizes on the performance of immediate word recall as an extra measure of auditory attention span. Second, verbal fluency performance in the MoCA 5-min protocol is scored incrementally instead of dichotomously, which enables a continuous and more representative weighting of this item in the overall score. Despite the fact that executive functioning was assessed based on one item, namely category fluency, it is a sensitive test to frontal lesions¹⁷ and our data showed that the MoCA 5-min protocol is effective in detecting executive impairment. Third, by retaining category cueing and recognition in the delayed memory, the MoCA 5-min protocol offers important information regarding the type of memory failure that may help to differentiate subtypes of VCI. We showed that when compared to the spontaneous recall score,^{4, 7} this modified scoring method correlated better with the MoCA memory domain score. Last, the MoCA 5-min protocol retains a 30-point scoring which allows a broader range of scores for more precise clinical measurement and for statistical analysis. This 30-point scoring is also in line with other common cognitive screens such as the MoCA and MMSE. Nevertheless, it important to keep in mind that scores between the MoCA 5-min protocol and the MoCA are not equivalent for interpretative purpose because of differences in test make up and scoring methods. The T-MoCA of Pendlebury et al. takes longer to administer but might offer an advantage over the MoCA 5-min protocol in longitudinal studies where direct comparison of item performance can be made with the paper-and-pencil MoCA.⁴ The MoCA 5-min protocol best serves as a very brief clinical screen that offers additional clinical information such as type of memory failure (i.e. encoding vs. retrieval) that may help more accurate diagnosis.

The MoCA 5-min protocol is highly feasible for telephone administration. From our experience, it is important to keep administration time as brief as possible in order to minimize environmental factors such as distractions from TV and family members that may invalidate test performance. We found it useful to explain to family members the purpose of the test and to seek their cooperation in test preparation, for example, to remove small children from the room and calendars from the wall. Family members are also asked not to help the patient in answering the test questions. As a limitation inherent to all telephone-based tests, individuals with hearing impairment may not be suitable for testing with the MoCA 5-min protocol, and hence screening of hearing problem is suggested before administering the test.

There are a number of limitations in our study. First, participants' performance on the MoCA 5-min protocol may be biased by the previous exposure to the MoCA, and the order of the MoCA and MoCA 5-min protocol was not counter-balanced. Yet, we tried to minimize this influence by using a longer interval of 30 days between the administrations of MoCA and the MoCA 5-min protocol. Second, the inclusion of a few participants with mild dementia (n=5) may have slightly inflated the performance of the MoCA 5-min protocol in group comparison and ROC analysis.¹⁸ However, we showed that the reduction in diagnostic accuracy was only minimal when patients with dementia were excluded. Likewise, we did not include patients with more severe dementia and therefore the psychometric properties of the MoCA 5-min protocol towards this end of cognitive spectrum and in other etiologies would need to be further investigated. Third, undetected subtle hearing problem might have lowered the MoCA 5-min score and contributed to reduced concordance between performance in MoCA and MoCA 5-min protocol in our sample. Fourth, in this study we did not derive any single cutoff score or education adjustment for the total score of the MoCA 5-min protocol because a much larger sample is needed to generate age- and education-corrected norms. Future studies are needed for standardization of norms for the MoCA 5-min protocol, and to compare its performance with other telephone-based cognitive tests such as the Telephone Interview for Cognitive Status (TICS).¹⁵ Last but not least, despite the participants in study were recruited from a large cohort of consecutive patients admitted for stroke and TIA, results of this study may not generalize to patients with different cultural and language backgrounds.

In conclusion, our study shows that the MoCA 5-min protocol is a brief, valid and reliable cognitive test that performed equally well as the MoCA in screening patients with stroke or TIA having cognitive impairment. It is highly feasible for administration over the telephone.