Accuracy of Two Cognitive Screening Tools to Detect Mild Cognitive Impairment in Parkinson's Disease

Abstract

Background

Patients with Parkinson's disease (PD) who have mild cognitive impairment (PD‐MCI) are at increased risk of developing PD dementia (PDD). Therefore, it is important to identify PD‐MCI in a reliable way.

Objectives

We evaluated the accuracy of the Parkinson's Disease‐Cognitive Rating Scale (PD‐CRS) and the Mattis Dementia Rating Scale‐2 (MDRS‐2) for detecting PD‐MCI. Data from healthy subjects were used to correct for demographic influences.

Methods

We compared the accuracy of the two instruments using ROC analysis. The gold standard was level II diagnosis of PD‐MCI according to consensus criteria of the International Parkinson and Movement Disorder Society.

Results

Seventy‐five healthy subjects and 125 PD patients were included. Education level, age and sex correlated with the PD‐CRS, but only age correlated with the MDRS‐2. Twenty‐seven percent of the patients had PD‐MCI. Areas under the curve (AUCs) for raw scores of PD‐CRS and MDRS‐2 were 0.83 and 0.81, respectively. At the optimal cut‐off for the PD‐CRS (101/102), sensitivity was 88% and specificity was 64%. For the MDRS‐2 (139/140) sensitivity and specificity were 68% and 79%, respectively.

AUCs for demographically corrected scores of PD‐CRS and for age‐corrected scores of MDRS‐2 were 0.80 and 0.78, respectively. At the optimal cut‐off for the PD‐CRS, sensitivity was 79% and specificity was 72%, while for the MDRS‐2 these were 77% and 67%, respectively.

Conclusions

Both cognitive screening tools are suitable for distinguishing PD‐MCI patients from cognitively intact PD patients. Demographical correction of scores did not improve sensitivity and specificity.

Introduction

Parkinson's disease (PD) patients may have cognitive impairments, ranging from mild cognitive impairment (PD‐MCI) to dementia (PDD). These cognitive impairments affect quality of life,1 may result in caregiver distress,2 and are associated with nursing home admission.3 PD‐MCI patients have an increased risk of developing dementia compared to cognitively intact PD patients.4, 5 Hence, it is essential to diagnose PD‐MCI in order to tailor clinical interventions, and hopefully in the future, to delay or prevent further cognitive decline.

PD‐MCI occurs in approximately 27% of PD patients (range: 19%‐38%).6 The impaired domains are typically of a frontal‐subcortical nature, with decreased attention and executive function, while there may also be impairment in visuospatial skills and memory.7, 8 If at all, language is affected later in the disease course.9

An International Parkinson and Movement Disorder Society (MDS) task force has proposed diagnostic criteria for PD‐MCI.10 The criteria consist of two levels of diagnostic certainty, depending on the comprehensiveness of the assessment. Level I diagnosis of PD‐MCI is based on an abbreviated assessment, while level II diagnosis is based on a comprehensive neuropsychological assessment.10

The task force proposed four cognitive screening tools for identifying PD‐MCI at level I. The aim of the current study was to evaluate the validity of two of these tools, viz. the Parkinson's Disease‐Cognitive Rating Scale (PD‐CRS)11 and the Mattis Dementia Rating Scale‐2 (MDRS‐2),12 in distinguishing PD‐MCI as diagnosed with level II criteria. Both scales were recently recommended by another MDS task force.13 We also wanted to explore the effect of correcting the raw scores for demographic influences. This is not always done when evaluating the validity of cognitive screening tools. However, these corrections may improve sensitivity and specificity, because they remove score variance that is unrelated to the disease.

Since the MDRS‐2 (unlike the PD‐CRS) is not a PD‐specific screening tool, and since its tasks are easier than those of the PD‐CRS, we expected that the PD‐CRS is a more sensitive screening tool for identifying PD‐MCI than the MDRS‐2.

Methods

Subjects

The PD patients were candidates for deep brain stimulation (DBS). They had been diagnosed and treated for several years by one or more neurologists before they were referred for DBS to the Academic Medical Center. Since PDD is an exclusion criterion for DBS,14 this group did not include patients with dementia.

Healthy control (HC) subjects consisted of volunteers who were recruited from the community and through word of mouth. HC subjects were at least 40 years old, had Dutch as first language, were free of psychiatric or neurological disorders, and did not take medication known to influence cognitive abilities. Also, subjects were excluded if they scored more than two standard deviations below the average on both cognitive screening tools; these subjects appeared to not actually meet our inclusion criteria. See Table 1 for demographic and clinical characteristics of control subjects and patients.

Table 1.

Demographic and Clinical Characteristics of all Subjects (Mean ± Standard Deviation)

	HC subjects (n = 75)	All PD patients (n = 125)	PD‐CogInt (n = 91)	PD‐MCI (n = 34)
Age (yr.)	60.3 ± 9.3	62.3 ± 6.9	61.4 ± 6.9	64.7 ± 6.5
Education level (ISCED)	4.5 ± 1.0	4.2 ± 1.1	4.3 ± 1.1	4.0 ± 1.2
Male (%)	45.3	62.4	60.4	67.6
UPDRS ‘on’ state	‐	19.9 ± 9.0	18.6 ± 8.7	23.6 ± 9.0
		(n = 106)	(n = 78)	(n = 28)
Subjective complaints patient (%)	‐	38.4	18.7	91.2
Subjective complaints friends/family (%)	‐	33.3	17.5	76.2
		(n = 78)	(n = 57)	(n = 21)
PD‐CRS total score	107.6 ± 12.3	97.4 ± 15.4	101.5 ± 14.2	86.2 ± 12.9
PD‐CRS demographically corrected z‐score	0 ± 1	−0.5 ± 1.3	−0.2 ± 1.2	−1.3 ± 1.1
MDRS‐2 total score	141.2 ± 2.6	139.4 ± 4.6	140.7 ± 3.2	135.8 ± 5.8
MDRS‐2 demographically corrected z‐score	0 ± 1	−0.4 ± 1.9	0.1 ± 1.4	−1.8 ± 2.5

Abbreviations: HC subjects, healthy control subjects; MDRS‐2, Mattis Dementia Rating Scale‐2; PD, Parkinson's disease; PD‐CogInt, cognitively intact PD patients; PD‐CRS, Parkinson's Disease‐Cognitive Rating Scale; PD‐MCI, mild cognitive impaired PD patients; UPDRS, Unified Parkinson's Disease Rating Scale.

The ethical review board of the Psychology department of the University of Amsterdam approved the study and all subjects gave informed consent.

Procedure

PD Patients

From January 2013 until September 2015 the PD‐CRS and the MDRS‐2 were administered as part of a comprehensive neuropsychological assessment for DBS screening. Patients were tested in the on‐phase. Level II diagnosis of PD‐MCI was established following the MDS PD‐MCI criteria10 if either the patient or an informant reported cognitive decline, and if the patient obtained abnormal scores (more than 1.0 SD below the demographically corrected mean) on at least two tests in one domain or in two or more separate domains10 (see Table 2 for tests used). Patients who did not satisfy PD‐MCI criteria were designated as cognitively intact (PD‐CogInt). None of the patients met PDD criteria.15

Table 2.

Neuropsychological Tests (and Associated Corrections) Used for Each Cognitive Domain to Diagnose PD‐MCI with Level II Diagnostic Criteria of the MDS Task Force

Domains	Tests	Performance corrected for
Executive domain	Letter fluency	Education
	Trail Making Test part B	Age, education, sex and TMT part A
Speed/attention domain	Trail Making Test part A	Age and education
	STROOP test (color naming)	Age, education and sex
Memory domain	Rivermead BMT prose recall	Age, education and sex
	Rey AVLT delayed recall	Age, education and sex
Language domain	Boston Naming Test	Age and education
	WAIS IV Similarities	Age
Visuospatial doman	GIT visuospatial reasoning	Age
	Judgment Of Line Orientation	Age and sex

Abbreviations: AVLT, Auditory Verbal Learning Test; GIT, Groninger Intelligence Test; RBMT, Rivermead Behavioral Memory Test; TMT, Trail Making Test; WAIS IV, Wechsler Adult Intelligence Scale IV. See Lezak et al.²⁴ for more information regarding the tests.

Healthy Control Subjects

Test administration was done at the homes of the HC subjects. After collection of demographic data, the MDRS‐2 was administered first. The MDRS‐2 took approximately 20‐30 minutes to administer, with poor performance resulting in longer administration time. Subsequently, the PD‐CRS was administered, which took approximately 15 minutes to complete.

Education was scored on the UNESCO ISCED (1997) scale, which ranges from 0 to 6; a higher score indicates a higher education level.

Statistical Analysis

A stepwise multiple regression analysis was performed on the HC data to determine which demographic variables significantly predicted scores on the PD‐CRS and the MDRS‐2. The variables age, age squared, sex, and educational level were included. Group differences on demographic variables and cognitive test scores were tested by univariate t‐tests or nonparametric tests.

The accuracy of the PD‐CRS and the MDRS‐2 was evaluated by the area under the curve (AUC) of receiver operating characteristic (ROC) analyses. The 90% confidence intervals (CI) of the AUC were used to determine whether the PD‐CRS or the MDRS‐2 was a better cognitive screening tool for identifying PD‐MCI. Performances were expressed both as raw scores and as demographically corrected z‐scores, based on the results of the HC subjects.

Data were analyzed using SPSS version 23.0 statistical software for Windows.

Results

One‐hundred and twenty‐five PD patients completed the comprehensive neuropsychological assessment. The PD patients, 78 men and 47 women, had a mean age of 62.3 years (SD = 6.9) and a median education level of 4 (mean = 4.2, SD = 1.1; range 1–6). All patients were questioned on subjective cognitive complaints, and for 78 patients (62.4%) an informant provided additional information regarding possible cognitive decline. Thirty‐four patients were diagnosed with PD‐MCI, and 91 were cognitively intact (PD‐CogInt). Therefore, the prevalence of PD‐MCI was 27.2% in this sample.

Of the 90 HC subjects tested, data of two subjects were excluded due to considerable below average performance. Additionally, data of 13 HC subjects (highly educated, young subjects) were removed from the dataset in order to match this group better to the PD patients. This resulted in 75 HC subjects—34 men and 41 women—for the analyses. They were between 44 and 84 years old (mean age = 60.3, SD = 9.3) and had a median education level of 5 (mean = 4.5, SD = 1.0; range 1–6). HC subjects and PD patients did not differ significantly in age. However, HC subjects were more often female (χ², p = 0.03), and were slightly higher educated (M‐W test, p = 0.04; see Table 1).

PD‐MCI patients were significantly older than the PD‐CogInt patients (t‐test, p = 0.02), but the patient subgroups did not significantly differ in level of education (M‐W test, p = 0.24) or gender (χ², p = 0.54).

See Table 1 for the demographic and clinical characteristics. Figure 1 shows the raw score distributions of both scales in the three groups. PD‐CRS scores follow an approximately normal distribution, whereas the MDRS‐2 scores show a ceiling effect, except in the PD‐MCI group. Figure 2 shows the relationship between both scales (see also Text S1).

Figure 1.

(A) Distributions of raw PD‐CRS scores in cognitively intact PD patients, PD‐MCI patients, and healthy controls. (B) Distributions of raw MDRS‐2 scores in cognitively intact PD patients, PD‐MCI patients, and healthy controls.

Abbreviations: HC, healthy control subjects; MDRS‐2, Mattis Dementia Rating Scale‐2; PD‐CogInt, cognitively intact PD patients; PD‐CRS, Parkinson's Disease‐Cognitive Rating Scale; PD‐MCI, PD patients with mild cognitive impairment.

Figure 2.

Scatterplot of raw scores of MDRS‐2 and PD‐CRS in cognitively intact PD patients, PD‐MCI patients, and healthy control subjects. Continuous line: regression line describing the relation between both scales (R² = 0.39); dashed lines: 90% individual confidence interval.

Abbreviations: HC, healthy control subjects; MDRS‐2, Mattis Dementia Rating Scale‐2; PD‐CogInt, cognitively intact PD patients; PD‐CRS, Parkinson's Disease‐Cognitive Rating Scale; PD‐MCI, PD patients with mild cognitive impairment.

Demographic Correction

The results of the stepwise regression analysis for the PD‐CRS indicated that education level (p < 0.001), age squared (p = 0.001), and sex (p = 0.013) significantly predicted the PD‐CRS score (adjusted R² = 0.36). Only age‐squared significantly predicted performance on the MDRS 2 (p = 0.001; adjusted R² = 0.13) and education and sex were not significant predictors (p = 0.17 and 0.67, respectively). The performance of the patients on both scales was expressed as demographically corrected z‐scores based on the regression results shown above. See Text S1 for details.

Comparison Between the PD‐MCI and PD‐CogInt Groups

The PD‐MCI patients performed significantly worse than the PD‐CogInt patients on the PD‐CRS and the MDRS‐2, both with and without demographic correction (t‐test, all p < 0.001; see Table 1).

Diagnostic Accuracy and Optimal Cut‐off Scores for the PD‐CRS and MDRS‐2

We conducted ROC analyses to examine the validity of the raw scores and the demographically corrected z‐scores for differentiating between PD‐MCI and PD‐CogInt patients (see Table 3). The raw scores of the PD‐CRS had an AUC of 0.83 and for the MDRS‐2 this was 0.81. The PD‐CRS z‐score and MDRS‐2 z‐score had a slightly lower AUC (0.80 and 0.78, respectively) than the raw score. None of the differences in AUCs was significant.

Table 3.

Areas Under the Curve of Raw and Demographically Corrected z‐scores of the PD‐CRS and Mattis DRS‐2 for Distinguishing PD‐MCI Patients from PD‐CogInt Patients

Test	AUC	90% CI	Optimal cut‐off	sensitivity	specificity
PD‐CRS raw score	0.83	0.77 – 0.89	101/102	88%	64%
MDRS‐2 raw score	0.81	0.74 – 0.88	139/140	68%	79%
PD‐CRS demographically corrected z‐score	0.80	0.73 – 0.86	−0.60	79%	72%
MDRS‐2 demographically corrected z‐score	0.78	0.70 – 0.85	−0.1	77%	67%

Abbreviations: AUC, Area under the curve; CI; Confidence Interval; Mattis DRS‐2, Mattis Dementia Rating Scale‐2; PD‐CRS, Parkinson's Disease‐Cognitive Rating Scale.

The optimal cut‐off score (Youden index) for the raw PD‐CRS score was 101/102, with sensitivity of 88% and specificity of 64%; for the raw MDRS‐2 score, the optimum was 139/140, with sensitivity of 68% and specificity of 79%. The optimal demographically corrected cut‐off score for the PD‐CRS had a sensitivity of 79% and a specificity of 72%. For the demographically corrected z‐score of the MDRS‐2 the optimum had a sensitivity of 77% and specificity of 67% (Table 3).

Discussion

We evaluated the PD‐CRS and the MDRS‐2 as cognitive screening tools for differentiating PD‐MCI patients from cognitively intact patients. Both the PD‐CRS and the MDRS‐2 can be considered good screening tools (AUC between 0.78 and 0.83). Contrary to our expectation, PD‐CRS was not superior to MDRS‐2, even though the latter had a clear ceiling effect in cognitively intact people. Apparently, the MDRS‐2 remains equally sensitive to detect cognitive decline. Nevertheless, in clinical practice we prefer applying the PD‐CRS, as its administration time is slightly shorter (taking about 15 min for cognitively intact people in our study, which is comparable to other reports).11

Correcting raw scores for demographic influences did not make much difference. Theoretically, one may expect that this correction would increase sensitivity and/or specificity, as it removes variance that is not due to the disease. However, age and education are also risk factors for cognitive decline. Apparently, the theoretical gain in validity is undone by the removal of information on risk factors.

Recently, both the PD‐CRS and the MDRS‐2 were recommended by an MDS task force over other cognitive screening tools.13 In a study similar to ours, Marras et al.16 examined the accuracy for detecting PD‐MCI of the Mini‐Mental State Examination (MMSE), the Montreal Cognitive Assessment (MoCA), and the Scales for Outcomes of Parkinson's Disease‐Cognition (SCOPA‐Cog), using the level II criteria as gold standard. These screening tests had a low combined sensitivity and specificity for identifying PD‐MCI (AUC was 0.68 for the MMSE, 0.71 for the MoCA, and 0.72 for the SCOPA‐Cog). Fernández de Bobadilla et al.17 evaluated the accuracy of the PD‐CRS, using the Clinical Dementia Rating (CDR = 0.5), presence of cognitive complaints, and abnormal MDRS‐2 scores as gold standard of PD‐MCI. The PD‐CRS showed more promising results (AUC = 0.85), comparable to our findings.

For the MDRS‐2 we found an optimal cut‐off of 139/140 for PD‐MCI, with a sensitivity of 68% and a specificity of 79%. Others report similar findings.18, 19, 20, 21 However, the cut‐off value for the PD‐CRS is not identical across studies.11, 17 The authors of the scale (Pagonabarraga et al.)11 found 80/81 as the optimal cut‐off score, with sensitivity of 73% and specificity of 84% for PD‐MCI defined as a score of 0.5 on the CDR (which is a level I diagnosis of PD‐MCI). In a later study of the same group, Fernández de Bobadilla et al.17 used the CDR and the MDRS‐2 for level I diagnosis of PD‐MCI. They found a sensitivity of 79% and a specificity of 80% of the PD‐CRS, again at cut‐off of 80/81. We found an optimal cut‐off of 101/102. However, we used a comprehensive neuropsychological assessment applying level II criteria for PD‐MCI, which might explain why the optimum in our study was considerably higher. Additionally, although the sensitifity in our study is higher than in other studies (88%), the corresponding specificity is lower (64%). This is to be expected with a higher optimal cut‐off score.

There are some limitations to our study. First, we relied on the control subjects' judgments whether or not they were cognitively healthy. Therefore, it is possible that not all control subjects were cognitively intact. Some control subjects indeed scored below the cut‐off for PD‐MCI, as can be seen in Figure 1. Second, we analyzed a sample of patients who were candidates for DBS screening. Therefore, this is a biased sample of PD patients. DBS patients are generally younger than the average PD patient. Our PD‐MCI sample is also rather small, with only 34 PD‐MCI cases. One needs large samples to show statistically significant differences between diagnostic instruments.22 Third, the approach we used to diagnose PD‐MCI might need some improvement. It has not yet been established which neuropsychological tests are optimal for determining decline in each cognitive domain.18 Additionally, we chose the cut‐off of 1.0 SD below average as definition of abnormal performance. Although this is in line with the cut‐off for diagnosing PD‐MCI proposed by the MDS task force (i.e. ‘approximately 1 to 2 SDs below appropriate norms’), we might have overestimated the presence of PD‐MCI. However, we found a mean prevalence of 27.2% of PD‐MCI, which is comparable to what has been reported in the literature (range: 19%‐38%).6 Finally, there was not always an informant to provide information on the patient's cognitive decline.

Conclusions

In sum, the PD‐CRS and the MDRS‐2 are both good cognitive screening tools to detect MCI in PD patients. However, the MDRS‐2 shows ceiling effects in unimpaired persons, and it has a slightly longer administration time. Demographic correction of the scores does not increase the validity of the scales.

Author Roles

1. Research Project: A. Conception, B. Organization, C. Execution; 2. Statistical Analysis: A. Design, B. Execution, C. Review and Critique; 3. Manuscript Preparation: A. Writing the First Draft, B. Review and Critique.

E.W.K.: 1B, 1C, 2A, 2B, 3A

B.S.: 1A, 2C, 3B

G.J.G.: 1A, 2A, 2C, 3B

Disclosures

Ethical Compliance Statement: The authors confirm that the approval of an institutional review board was not required for this work. We confirm that we have read the Journal's position on issues involved in ethical publication and affirm that this work is consistent with those guidelines.

Funding Sources and Conflicts of Interest: No specific funding was received for this work and the authors declare that there are no conflicts of interest relevant to this work.

Financial Disclosures for the previous 12 months: Emmie W. Koevoets reports no disclosures. Ben Schmand received royalties from Pearson Assessment, Hogrefe Publishing; and received an honorarium from Philips Research. Gert J. Geurtsen received research grants from the Michael J. Fox Foundation and royalties from Cogstate Clinical Trials. Additionally, Dr. Schmand received salary from the Academic Medical Center and the University of Amsterdam and Dr. Geurtsen received salary from the Academic Medical Center.

Supporting information

Supporting information may be found in the online version of this article.

Text S1. Demographic correction formulas for the PD‐CRS and MDRS‐2 and estimation of PD‐CRS score based on MDRS‐2 score and vice versa.