Abstract
Background
The predictive ability of National Institute of Neurological Disease and Stroke-Canadian Stroke Network (NINDS-CSN) 5-minute protocol and Montreal Cognitive Assessment (MoCA) administered sub-acutely and at the convalescent phase after stroke for significant vascular cognitive impairment (VCI) at 1 year is unknown. We compared prognostic values of these tests.
Methods
Patients with ischemic stroke and transient ischemic attack (TIA) received MoCA sub-acutely (within 2 weeks) and 3–6 months after stroke followed by a formal neuropsychological evaluation at 1 year. The total score of NINDS-CSN 5-minutes protocol was derived from MoCA. Moderate-severe VCI was defined as neuropsychological impairment in ≥ 3 domains. Area under the receiver operating characteristic curve (AUC) analyses were conducted to establish the optimal cutoff points and discriminatory properties of the MoCA and NINDS-CSN 5-minute protocol in detecting moderate-severe VCI.
Results
Four hundre patients were recruited at baseline. Of these, 291 received a formal neuropsychological assessment 1 year after stroke. 19 % patients had moderate-severe VCI. The MoCA was superior to the NINDS-CSN 5-minute protocol [sub-acute AUCs: 0.89 vs 0.80, p < 0.01; 3–6 months AUCs: 0.90 vs 0.83, p < 0.01] in predicting for moderate-severe VCI at 1 year. At respective cutoff points, MoCA had significantly higher sensitivity than the NINDS-CSN 5-minute protocol at baseline (p = 0.01) and 3–6 months (p = 0.04).
Conclusions
MoCA administered sub-acutely and 3–6 months after stroke is superior to the NINDS-CSN 5-minute protocol in predicting moderate-severe VCI at 1 year.
Electronic supplementary material
The online version of this article (doi:10.1186/s12883-016-0570-y) contains supplementary material, which is available to authorized users.
Keywords: Vascular dementia, Acute stroke, Predictive value of tests, Cognition
Background
Post-stroke vascular cognitive impairment (VCI) at 3–6 months after stroke is common (57.7 %) in Asian patients with mild stroke and transient ischemic attack (TIA) [1]. Approximately 25 % of stroke/TIA patients had moderate-severe VCI of sufficient severity to place them at a higher risk for cognitive decline [1, 2]. Therefore, early detection of patients with moderate-severe VCI in the subacute stroke phase (initial 2 weeks after stroke) is required for intensive reduction of vascular risk factors and improve prognosis [3].
We have previously established that the commonly used brief screening instruments, i.e., the Mini-Mental State Examination (MMSE) and Montreal Cognitive Assessment (MoCA), administered at the subacute stroke phase can predict moderate-severe VCI 3–6 months after stroke with good sensitivity, as determined by formal neuropsychological evaluations [1], which has been replicated [4]. Note that patients with the highest education level of primary school or below would award 1-point for MoCA. However, both the MoCA and MMSE take approximately 10–12 min to administer, and may not be practical for bedside screening, especially during acute stroke admission. By contrast, the National Institute of Neurological Disease and Stroke-Canadian Stroke Network (NINDS-CSN) 5-minute protocol drawn from the MoCA takes less time and has been recommended for bedside screening [5]. Two studies examined the NINDS-CSN 5-minute protocol for post-stroke VCI screening in person or over the phone and showed conflicting results [6, 7]. One study reported that NINDS-CSN 5-minute protocol performed less well than the MoCA [6], while the other reported both tests to be equivalent in detecting post-stroke VCI [7]. These inconsistent findings could be due to different scoring methods (total score of 12 and 30 respectively) and diagnosis of VCI (modified Petersen’s criteria for mild cognitive impairment and cognitive impairment defined by Clinical Dementia Rating scale respectively). Additionally, the predictive ability of NINDS-CSN 5-minute protocol for post-stroke moderate-severe VCI remains unknown. We therefore aimed to: 1) examine whether the MoCA and NINDS-CSN 5-minute protocol administered at subacute stroke phase and 3–6 months later could predict moderate-severe VCI at 1 year; and 2) compare discriminatory abilities of these two tests using their respective optimal cutoff points.
The MoCA administered at baseline has been reported to be predictive of moderate-severe VCI at 3–6 months after stroke/TIA [1]. Additionally, the NINDS-CSN 5-minute protocol was drawn from the MoCA items. Therefore, we hypothesized that the MoCA and NINDS-CSN 5-minute protocol administered at baseline and 3–6 months after stroke would be equivalent, sensitive, and predictive of moderate-severe VCI at 1 year.
Methods
Subjects
We recruited 400 consecutive patients (≥21 years old) with a recent ischemic stroke or TIA during the sub-acute stroke phase at the National University Hospital, Singapore. Patients were excluded if they had a major physical disability or an active psychiatric disorder that would impede cognitive testing. Eligible patients were assessed at baseline, i.e., within 2 weeks after their cerebrovascular event, 3–6 months and 12 months after stroke/TIA. At baseline their demographic and clinical outcome measures were collected along with cognitive status measured by brief cognitive test the MoCA. At 3–6 months and 12 months after stroke/TIA, their clinical outcome measures and neuropsychological performance on a formal test battery were collected. The availability of an informant was required for a patient to be included in the study.
Standard protocol approvals and patient consent
This study was approved by the National Healthcare Group Domain-Specific Review Board (DSRB), and was conducted in conformity with the Declaration of Helsinki. Written informed consent was obtained from all participants and/or their legally acceptable representatives.
Procedures
Baseline characteristics
Demographic information (age, gender, ethnicity, years of education), clinical information, vascular risk factors, and neurological status i.e. National Institute of Health Stroke Score (NIHSS) and modified Rankin Scale (mRS) were collected.
Clinical outcome measures
Patients were followed up at 3–6 months and 1 year after their index cerebrovascular event for their neurological status, clinical outcomes and interval events. Medical records were used to verify these events (strokes or TIA, peripheral artery disease, intracranial hemorrhage, cardiovascular risk factor, cardiac events or deaths from any of the above).
Cognitive measures
Patients’ cognitive status was assessed by the MoCA at baseline and 3–6 months after stroke. The MoCA has 12 items with a maximum score of 30. It consists of seven subtests: visuo-executive, naming, attention, language, abstraction, delayed recall and orientation. The MoCA was modified for Singaporean population such as replacing a test of letter fluency with semantic fluency (animals). Patients with a highest education level of primary school or below would receive an additional 1-point as education adjustment in MoCA scores. Details of MoCA modification can be found in our previous paper [8]. The NINDS-CSN 5-minute test protocol consists of three individual test items, i.e., 5-word recall (5 points), 6-item orientation (6 points), and verbal fluency [1 point if >10 words (animals) are generated in 60 s], drawn from the MoCA with a maximum score of 12 [5]. Brief description of these two tests were covered in [9]. Patients also received a formal neuropsychological battery covering seven domains locally validated for Singaporean elderly at 12 months after stroke [10], which was administered by trained research psychologists blinded to the baseline MoCA scores. The formal neuropsychological battery (Additional file 1: Table S1) has been locally validated for Singaporean and was used in stroke clinical research [1, 2, 10]. It consists of domains in executive function, attention, language, visuoconstruction, visuomotor speed, verbal and visual memory. This formal neuropsychological battery was used to define cognitive impairment. Age and education-adjusted cutoffs of 1.5 standard deviations (SD) below the established norms were used on individual tests. A result of at least half the tests failing in a domain defines domain impairment. The neuropsychological findings were interpreted by a doctoral level neuropsychologist (YD) to establish cognitive impairment. Patients with vascular cognitive impairment no dementia (VCIND) were those with objective cognitive domain impairment determined by the neuropsychological test battery without meeting the criteria for dementia. VCIND patients were further classified into VCIND mild (≤ 2 impaired domains) and VCIND moderate (≥ 3 impaired domains) [2]. VCI patients were dichotomized into no–mild VCI (no cognitive impairment (NCI) and VCIND mild) and moderate–severe VCI (VCIND moderate and dementia). Diagnoses of dementia were made according to the Diagnostic and Statistical Manual of Mental Disorders [11]. The Bayer-Activities of Daily Living Scale [12] was administered to the informant to ascertain functional disabilities primarily due to cognitive impairment.
Statistical analyses
Data analysis was conducted using Statistical Package R version 3.2.0. Receiver operating characteristic (ROC) curve analyses were conducted to establish the optimal cutoff points and discriminatory properties of the MoCA and NINDS-CSN 5-minute protocol in detecting moderate-severe VCI. The optimal cutoff points were chosen by Youden Index [13]. Area under the operating characteristics curves (AUCs) of these tests were compared statistically [14]. Additionally, McNemar’s test was employed to compare sensitivities at the optimal cutoff points of the MoCA and NINDS-CSN 5-minute protocol.
Results
Subject characteristics
Recruited patients were mostly Chinese (70.3 %) and male (69.8 %), with a mean age of 59.8 ± 11.6 years and a mean of 7.7 ± 4.3 years of formal education. A majority of subjects had mild stroke (NIHSS ≤ 2, n = 233, 58.5 %) and minimal disability (mRS ≤ 2, n = 251, 63 %). Of 319 ischemic stroke patients classified using the TOAST criteria [15], most (n = 179, 44.8 %) patients had small artery occlusion, with a further 15.0 % (n = 60) with large artery atherosclerosis, 14.8 % (n = 59) with cardio-embolism, 5.3 % (n = 21) with undetermined or other determined etiology.
Two hundred ninety one out of 400 stroke/TIA patients completed a formal neuropsychological assessment at 1 year after stroke. Of these, 56 (19 %) had moderate-severe VCI (Table 1). Patients with moderate-severe VCI were older with lower education level and had higher NIHSS scores. Patients who were lost to follow-up at 1 year (n = 109) did not have significantly different baseline characteristics from those who completed follow-up.
Table 1.
Characteristics | No-mild VCI (n = 235) | Moderate-severe VCI (n = 56) | Univariate Analysis p | ||
---|---|---|---|---|---|
Age, mean (SD) | 57.46 | (10.20) | 68.43 | (11.23) | <0.01 |
Gender, female | 64 | (27 %) | 21 | (38 %) | 0.18 |
Education, y (mean, SD) | 8.33 | (4.15) | 5.52 | (4.02) | <0.01 |
Ethnicity (no., %) | 0.51 | ||||
Chinese | 171 | (73 %) | 36 | (64 %) | |
Malay | 43 | (18 %) | 14 | (25 %) | |
Indian | 19 | (8 %) | 6 | (11 %) | |
Others | 2 | (1 %) | 0 | (0 %) | |
Stroke classification (no., %) | 0.39 | ||||
SAO | 112 | (48 %) | 23 | (41 %) | |
LAA | 30 | (13 %) | 10 | (18 %) | |
CE | 33 | (14 %) | 11 | (20 %) | |
UND | 8 | (3 %) | 4 | (7 %) | |
OC | 1 | (0.43 %) | 0 | (0 %) | |
TIA | 51 | (22 %) | 8 | (14 %) | |
NIHSS (median) | 1.00 | 4.00 | <0.01 | ||
Baseline mRS (median) | 2.00 | 3.00 | <0.01 | ||
Mean interval (Days) between stroke/TIA and assessment (mean, SD) | 2.62 | (2.09) | 4.02 | (3.05) | <0.01 |
Cognitive screening test | |||||
Baseline MoCA (mean, SD) | 22.36 | (4.26) | 14.75 | (4.13) | <0.01 |
Baseline NINDS-CSN 5-minute protocol (mean, SD) | 9.29 | (2.25) | 6.41 | (2.46) | <0.01 |
Number of cardiovascular risk factors (median) | 3 | 3 | 0.14 | ||
Recurrent vascular events 3–6 months (no., %) | 10 | (4 %) | 2 | (4 %) | 1.00 |
Recurrent vascular events 1-year (no., %) | 4 | (2 %) | 2 | (4 %) | 0.54 |
VCI vascular cognitive impairment, SD standard deviations, SAO small artery occlusion, LAA large artery atherosclerosis, CE cardioembolism, UND undetermined etiology, OC other determined etiology, TIA transient ischemic attack, NIHSS National Institute of Health Stroke Score, mRS modified rankin scale, MoCA Montreal Cognitive Assessment
Comparison of the predictive ability of the MoCA and NINDS-CSN 5-minute protocol
Figures 1 and 2 shows that the MoCA had statistically larger AUCs than the NINDS-CSN 5-minute protocol sub-acutely [AUC (95 % CI): 0.89 (0.85–0.93) vs 0.80 (0.74–0.87), p <0.01] and 3–6 months after stroke [AUC (95 % CI): 0.90 (0.86–0.94) vs 0.83 (0.77–0.89), p <0.01] for detecting patients with moderate-severe VCI at 1 year. ROC analyses repeated using age- and education-adjusted scores did not alter the results.
Table 2 shows the discriminatory properties of the MoCA and NINDS-CSN 5-minute protocol sub-acutely and at 3–6 months for predicting moderate-severe VCI at 1 year after stroke. At respective optimal cut-off points at baseline, the MoCA (<20) was superior to NINDS-CSN 5-minute protocol (<8) (p < 0.01). McNemar’s tests showed that the sensitivities of MoCA (<20) was significantly higher than NINDS-CSN 5-minute protocol (<8) (p = 0.01). At the respective optimal cutoff points at 3–6 months after stroke, the MoCA (<21) was equivalent to NINDS-CSN 5-minute protocol (<8) (p = 0.07). Additionally, the MoCA (<21) had significantly higher sensitivity than NINDS-CSN 5-minute protocol (<8) (p = 0.04). Moreover, we excluded participants who experienced at recurrent vascular event between baseline and 1 year follow up, and repeated the ROC analysis, however, results remained the same.
Table 2.
MoCA | NINDS-CSN 5-minute protocol | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
CUT-OFF | SEN | SPEC | PPV | NPV | ACC | CUT-OFF | SEN | SPEC | PPV | NPV | ACC |
(A) Baseline | |||||||||||
21/22 | 0.95 | 0.62 | 0.37 | 0.98 | 0.68 | 9/10 | 0.79 | 0.54 | 0.29 | 0.91 | 0.59 |
20/21 | 0.93 | 0.69 | 0.42 | 0.98 | 0.74 | 8/9 | 0.77 | 0.66 | 0.35 | 0.92 | 0.68 |
19/20a | 0.88 | 0.75 | 0.46 | 0.96 | 0.78 | 7/8a | 0.73 | 0.75 | 0.41 | 0.92 | 0.75 |
18/19 | 0.80 | 0.81 | 0.50 | 0.95 | 0.81 | 6/7 | 0.63 | 0.83 | 0.47 | 0.90 | 0.79 |
17/18 | 0.70 | 0.84 | 0.51 | 0.92 | 0.81 | 5/6 | 0.43 | 0.95 | 0.69 | 0.88 | 0.85 |
(B) 3–6 months | |||||||||||
22/23 | 0.91 | 0.68 | 0.40 | 0.97 | 0.72 | 9/10 | 0.81 | 0.64 | 0.35 | 0.93 | 0.67 |
21/22 | 0.89 | 0.74 | 0.45 | 0.97 | 0.77 | 8/9 | 0.74 | 0.77 | 0.43 | 0.92 | 0.76 |
20/21a | 0.83 | 0.80 | 0.50 | 0.95 | 0.81 | 7/8a | 0.70 | 0.83 | 0.49 | 0.92 | 0.80 |
19/20 | 0.68 | 0.87 | 0.55 | 0.92 | 0.83 | 6/7 | 0.49 | 0.94 | 0.65 | 0.89 | 0.85 |
18/19 | 0.64 | 0.91 | 0.64 | 0.91 | 0.86 | 5/6 | 0.25 | 0.98 | 0.72 | 0.84 | 0.84 |
aOptimal cut-off score
MoCA Montreal Cognitive Assessment, NINDS-CSN National Institute of Neurological Disease and Stroke-Canadian Stroke Network, SEN sensitivity, SPEC specificity, PPV positive predicted value, NPV negative predicted value, ACC correctly classified rate
Discussion
The principal finding of this study is that MoCA administered in the subacute stroke phase and 3–6 months later, is more sensitive and superior to the NINDS-CSN 5-minute protocol in predicting moderate-severe VCI at 1 year, as determined by formal neuropsychological evaluations.
Our finding, that the MoCA administered at baseline is predictive of moderate-severe VCI at 1 year, is consistent with previous studies [1, 4]. Our finding that the MoCA at baseline and 3–6 months after stroke are more sensitive and superior to NINDS-CSN 5-minute protocol in detecting moderate-severe VCI at 1 year later is consistent with a previous study [6], although that study did not compare the AUCs of MoCA to NINDS-CSN using inferential statistics. Our finding differs from a recent study [7] due to different scoring method (the initial scoring method of 12 points in the present study vs. 30 points in the previous study). Additionally, the NINDS-CSN 5-minute protocol at its optimal cutoff point (<8) at baseline or 3-6 months had similar sensitivity to the findings reported by Bocti et al [16] (0.73, 0.70, and 0.72 respectively) [16]. However the specificity of the NINDS-CSN 5-minute protocol in the present study is somewhat lower than the previous findings by Bocti et al (0.75, 0.83, and 0.91 respectively). This could be due to different gold standard criteria to define cognitive impairment, i.e., a formal neuropsychological evaluation was used in the present study while MoCA < 26 was used to define cognitive impairment by Bocti and colleagues. The use of MoCA cutoff points to define cognitive impairment may have inflated the value of specificity. In view of a higher sensitivity and negative predictive value (NPV) of the MoCA relative to NINDS-CSN 5-minute test at optimal cutoff points (baseline sensitivity: 0.88 vs 0.73, NPV: 0.96 vs 0.92; 3–6 months sensitivity 0.83 vs 0.70, NPV: 0.95 vs 0.92), MoCA administered at baseline and 3–6 months after stroke would be better than the NINDSCSN 5-minute protocol in predicting for moderate-severe VCI at 1 year.
One limitation of the present study is that our sample consisted of mild stroke/TIA patients, therefore results may not be generalizable to all stroke populations. Nevertheless, approximately 1/5 of this milder stroke group had moderate-severe VCI at 1 year after stroke who may escape clinical attention without early screening at baseline or 3–6 months after stroke. Additionally, the NINDS-CSN 5-minute test protocol was developed based on expert opinion without empirical evidence [5]. The MoCA test items sensitive to cognitive pattern of “subcortical deficits” (bradyphrenia, frontal-executive and visuospatial deficits), such as visuo-executive items (Trails B, Cube, Clock) and working memory item such as Serial 7 s, that may improve screening accuracy are not included in the NINDS-CSN 5-minute test. Therefore future studies should develop an abbreviated MoCA test based on empirical evidence and validate it through independent administration of this abbreviated MoCA against a gold standard neuropsychological battery for bedside screening of VCI. Furthermore, the utility of longitudinal changes of the MoCA, NINDS-CSN 5 minute test and the empirically derived short MoCA may be examine for clinical practice.
Conclusion
In conclusion, MoCA administered sub-acutely during stroke admission and 3–6 months after stroke has significantly higher sensitivity and is superior to the NINDS-CSN 5-minute protocol in predicting moderate-severe VCI at 1 year.
Acknowledgments
This study is funded by a Center Grant from the National Medical Research Council (NMRC/CG/NUHS/2010). Y. Dong was a recipient of the NUHS National Clinician Scientist Program award during this study. The authors thank all participants for their involvement, the research team from NUHS Memory Ageing and Cognition Centre for data collection and the NUHS Medical Publication Support Unit for its editorial support.
Additional file
Footnotes
Competing interests
The authors declare that they have no competing interests.
Authors’ contribution
YHD designed the study, obtained the funding, collected data, supervised statistical analysis, interpreted the data, and drafted the manuscript. JX performed statistical analysis, drafted statistical and results sections. PLC collected data, recruited patients and revised the manuscript. CSS collected data, recruited patients and revised the manuscript. NV collected data, recruited patients and revised the manuscript. HLT collected data, recruited patients and revised the manuscript. VKS collected data, recruited patients and revised the manuscript. LHC designed the study and revised the manuscript. All authors read and approved the final manuscript.
Contributor Information
YanHong Dong, Phone: +65 66011977, Email: yanhong_dong@nuhs.edu.sg.
Jing Xu, Email: phcxuj@nus.edu.sg.
Bernard Poon-Lap Chan, Email: Bernard_CHAN@nuhs.edu.sg.
Raymond Chee Seong Seet, Email: raymond_seet@nuhs.edu.sg.
Narayanaswamy Venketasubramanian, Email: drnvramani@gmail.com.
Hock Luen Teoh, Email: Hock_Luen_TEOH@nuhs.edu.sg.
Vijay Kumar Sharma, Email: Vijay_Kumar_SHARMA@nuhs.edu.sg.
Christopher Li-Hsian Chen, Email: christopher_chen@nuhs.edu.sg.
References
- 1.Dong Y, Venketasubramanian N, Chan BP-L, Sharma VK, Slavin MJ, Collinson SL, et al. Brief screening tests during acute admission in patients with mild stroke are predictive of vascular cognitive impairment 3–6 months after stroke. J Neurol Neurosurg Psychiatry. 2012;83:580–5. doi: 10.1136/jnnp-2011-302070. [DOI] [PubMed] [Google Scholar]
- 2.Narasimhalu K, Ang S, De Silva DA, et al. Severity of CIND and MCI predict incidence of dementia in an ischemic stroke cohort. Neurology. 2009;73:1866–72. doi: 10.1212/WNL.0b013e3181c3fcb7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Gorelick PB, Scuteri A, Black SE, et al. Vascular contributions to cognitive impairment and dementia: A statement for healthcare professionals from the American Heart Association/American Stroke Association. Stroke. 2011;42:2672–713. doi: 10.1161/STR.0b013e3182299496. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Salvadori E, Pasi M, Poggesi A, Chiti G, Inzitari D, Pantoni L. Predictive value of MoCA in the acute phase of stroke on the diagnosis of mid-term cognitive impairment. J Neurol. 2013;260:2220–7. doi: 10.1007/s00415-013-6962-7. [DOI] [PubMed] [Google Scholar]
- 5.Hachinski V, Iadecola C, Petersen RC, et al. National Institute of Neurological Disorders and Stroke–Canadian Stroke Network vascular cognitive impairment harmonization standards. Stroke. 2006;37:2220–41. doi: 10.1161/01.STR.0000237236.88823.47. [DOI] [PubMed] [Google Scholar]
- 6.Pendlebury ST, Welch SJ, Cuthbertson FC, Mariz J, Mehta Z, Rothwell PM. Telephone assessment of cognition after transient ischemic attack and stroke: modified telephone interview of cognitive status and telephone Montreal Cognitive Assessment versus face-to-face Montreal Cognitive Assessment and neuropsychological battery. Stroke. 2013;44:227–9. doi: 10.1161/STROKEAHA.112.673384. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Wong A, Nyenhuis D, Black SE, Law LS, Lo ES, Kwan PW, et al. Montreal Cognitive Assessment 5-minute protocol is a brief, valid, reliable, and feasible cognitive screen for telephone administration. Stroke. 2015;46:1059–64. doi: 10.1161/STROKEAHA.114.007253. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Dong Y, Sharma VK, Chan BP-L, Venketasubramanian N, Teoh HL, Seet RCS, et al. The Montreal Cognitive Assessment (MoCA) is superior to the Mini-Mental State Examination (MMSE) for the detection of Vascular Cognitive Impairment after Acute Stroke. J Neurol Sci. 2010;299:15–8. doi: 10.1016/j.jns.2010.08.051. [DOI] [PubMed] [Google Scholar]
- 9.Dong Y, Koay WI, Yeo L, et al. Rapid Screening for Cognitive Impairment in Parkinson’s Disease: A Pilot Study. Parkinson’s Dis. 2015;348063:1–6. doi: 10.1155/2015/348063. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Yeo D, Gabriel C, Chen C, et al. Pilot validation of a customized neuropsychological battery in elderly Singaporeans. Neurol J South East Asia. 1997;2:123. [Google Scholar]
- 11.American Psychiatric Association . Diagnostic and Statistical Manual of Mental Disorders. 4. Washington, DC: American Psychiatric Association; 1994. [Google Scholar]
- 12.Hindmarch I, Lehfeld H, de Jongh P, et al. The Bayer Activities of Daily Living Scale (B-ADL) Dement Geriatr Cogn Disord. 1998;9:20–6. doi: 10.1159/000051195. [DOI] [PubMed] [Google Scholar]
- 13.Youden WJ. Index for rating diagnostic tests. Cancer. 1950;3:32–5. doi: 10.1002/1097-0142(1950)3:1<32::AID-CNCR2820030106>3.0.CO;2-3. [DOI] [PubMed] [Google Scholar]
- 14.DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988;44:837–45. doi: 10.2307/2531595. [DOI] [PubMed] [Google Scholar]
- 15.Adams HP, Jr, Bendixen BH, Kappelle LJ, et al. Classification of subtype of acute ischemic stroke. Definitions for use in a multicenter clinical trial. TOAST. Trial of Org 10172 in Acute Stroke Treatment. Stroke. 1993;24:35–41. doi: 10.1161/01.STR.24.1.35. [DOI] [PubMed] [Google Scholar]
- 16.Bocti C, Legault V, Leblanc N, Berger L, Nasreddine Z, Beaulieu-Boire I, Yaneva K, Boulanger J-M. Vascular Cognitive Impairment: Most Useful Subtests of the Montreal Cognitive Assessment in Minor Stroke and Transient Ischemic Attack. Dement Geriatr Cogn Disord. 2013;36:154–62. doi: 10.1159/000351674. [DOI] [PubMed] [Google Scholar]