Selection of normative group affects rates of mild cognitive impairment in Parkinson disease

Abstract

Objective

Examine the impact of different methods of standardizing cognitive data in the Parkinson’s Progression Marker Initiative.

Methods

Cognitive data from 423 participants with Parkinson Disease were included [Age=61.7(9.7), Education=15.6(3.0)]. Internal norms were calculated using the group mean and standard deviation of the healthy control group. Published norms were compared to the overall group mean of and to age-stratified norms from healthy controls for each neuropsychological test over four visits. Rates of mild cognitive impairment were calculated using established criteria.

Results

The use of internal norms resulted in lower standardized scores than published norms on all tests with the exception of memory and processing speed (p≤.001). Individuals were 1.5–2.1 times more likely to be diagnosed with Mild Cognitive Impairment using internal norms than published norms.

Conclusions

Standardization approaches with cognitive data are not interchangeable. Selection of a normative comparison group impacts research and clinical interpretations of cognitive data.

Introduction

Many cognitive abilities change with age [1–3]. Therefore, raw scores from neuropsychological tests are often compared to a demographically representative (e.g., age, education, and/or ethnicity) normative sample. Interpretation of standardized scores depends upon the demographics and characteristics of the normative sample [2–3]. Some researchers have argued for the use of raw scores in longitudinal research rather than norm-adjusted scores [4]. Including age as a covariate can control for the effect of age within a model, but it does not provide information about the score relative to what would be the expected performance for that age group (reviewed by [2, 5]). In a study with Alzheimer’s disease, younger and older patients did not differ significantly in raw scores on neuropsychological testing, but when age-based standardized scores were used, the younger group performed worse on tasks of executive functioning, visuospatial skills, and memory than the older group [6].

The Parkinson’s Progression Markers Initiative (PPMI) study includes widely-used neuropsychological tests to assess participants with early-stage Parkinson disease (PD) and age- and sex-matched healthy controls (HCs) annually. There are three primary approaches that have been used to analyze cognitive data from the PPMI. First, each of the neuropsychological tests in the PPMI includes published norms comparing the participant’s performance to a normative mean of a community sample based upon age [7] and, for some tests, level of education. Second, internal norms have been created by transforming the raw score of the PD group into a z-score based upon the mean and standard deviation of the HC group for each cognitive test [8]. This approach does not take into consideration the age of the individual participants with PD; however, it is possible to create age-based norms utilizing the HC sample. Lastly, cognitive data from PPMI may be analyzed using raw scores. [10]

There is inconsistency in the literature regarding the use of cognitive data as a clinical outcome. Differences in the standardization of cognitive data may result in incorrect conclusions or contradictory findings. The purpose of this study was to examine the impact of different methods of determining normative or standardized cognitive data on cognitive outcomes in PD.

Methods

Study Cohort

The PPMI is a multi-site, longitudinal study of de novo PD and HCs. Information about the aims of the PPMI study, collection sites, and methodology have previously been published and are available on the PPMI website (http:/www.ppmi-info.org/study-design)[11]. The PPMI cohort includes 423 PD subjects and 196 healthy controls. Data for the present study were obtained in April 2017.

Standard Protocol Approvals, Registrations, and Patient Consents

Each participating PPMI site received approval from an ethical standards committee on human experimentation before study initiation. Written informed consent was obtained from all study participants.

Procedures

As a part of the PPMI study, all participants are administered the Montreal Cognitive Assessment (MoCA) [12] and detailed neuropsychological tests annually (baseline and 12, 24, and 36-month follow-up). Published age-norms are available for the neuropsychological tests included in PPMI and were utilized as the first normative method. The neuropsychological tests in PPMI assess verbal learning/memory (Hopkins Verbal Learning Test-Revised; HVLT-R) [13], verbal fluency (Animals) [2], processing speed (Symbol Digit Modalities Test; SDMT) [14], working memory (Letter Number Sequencing; LNS) [15], and visuospatial ability (Judgment of Line Orientation; JLO) [16]. For the current study, HC overall group mean internal norms were calculated by creating a z-score for each participant using the group mean and standard deviation of the entire PPMI control group at each time point as the second normative method. The third normative method involved creating age-based norms from the healthy control group using the following age ranges; 30–45 (n=23), 46–60 (n=69), 61–75 (n=91), and 76–90 (n=13). The z-scores were then converted to T-scores or scaled scores for direct comparison with published norms, with the exception of SDMT, which is typically presented as a z-score.

Based on the MDS Task Force Level I guidelines for classifying PD-MCI, [17] participants in the current study were classified as MCI if they scored at least ≤1.5SD from the normative mean on 2 or more neuropsychological tests (per guidelines, HVLT immediate and delay count as one test). For the purposes of comparing the differences between impairment rates based on different normative samples, MoCA was not used as criteria for MCI in this study.

Analyses

Repeated measures one-way ANOVAs were conducted for each of the neuropsychological measures comparing the three normative methods at baseline and 12, 24, and 36-month follow-up. Post hoc tests using Tukey’s HSD test were also conducted. Relative risk of with impairment (at least 1.5 SD below the mean) using the published norms versus overall group mean internal norms and age-based internal norms was calculated. In order to compare the effect of different cutoffs, this analysis was repeated examining ≤1SD and ≤2SD as the cutoff for each test. Finally, the percentage of participants meeting Level I criteria for MCI based on the Litvan et al. criteria [17] was compared using the three normative approaches and a <1.5SD cutoff.

Results

Baseline data and demographic variables for PD participants and HCs were analyzed. There were no significant differences between the PD or control group in terms of age or sex; however, the HC group (M = 16.04, SD = 2.89) had a significantly higher level of education than the PD group (M = 15.54, SD = 2.99; p = .05). At baseline, the mean disease duration for the PD group was 5.88 months (SD = 3.54) and mean UPDRS-III “off” score was 20.91 (SD = 8.87).

Mean test performances and ANOVA results comparing the three normative methods are included in Table 1. There was no significant effect of normative methods on HVLT immediate recall (F_{(2, 915)} = 1.65, p=0.193). However, there was a significant effect of normative method on HVLT delayed recall (F_{(2, 914)} = 3.29, p=.038), JLO (F_{(2, 910)} = 169.71, p≤.001), LNS (F_{(2, 914)} = 231.89, p≤.001), VF (F_{(2, 915)} = 13.62, p≤.001), and SDMT (F_{(2, 882)} = 3.86, p=.021). The use of internal norms resulted in lower standardized scores than published norms on all tests with the exception of memory and processing speed (p≤.001). Tables S1–S3 demonstrate the percentages of individuals with ≤1SD, ≤1.5SD, and ≤2SD the mean for each test using the three different normative groups.

Table 1.

Comparison of PPMI Internal Norms and Published Age-Normative Data in Participants with Parkinson Disease

	Raw Scores M ± SD	Test Performance Based on Published Norms M ± SD	Test Performance Based on Internal HC Overall Group- Norms M ± SD	Test Performance Based On Internal HC Age-Norms M ± SD	df	F	p
HVLT-R Immediate Recall T-score
Baseline Form 1	24.44 ± 4.98	46.34 ± 10.99	46.43 ± 11.06	46.51 ± 10.72	2	0.10	.902
12-month Form 2	23.72 ± 5.34	46.73 ± 11.57	44.38 ± 11.44	45.04 ± 11.61	2	3.34	.036*a
24-month Form 3	23.67 ± 5.51	45.54 ± 11.22	45.61 ± 10.90	44.65 ± 11.21	2	0.65	.523
36-month Form 4	26.50 ± 5.27	49.48 ± 12.18	46.56 ± 11.64	46.58 ± 13.14	2	3.76	.024*ab
HVLT-R Delayed Recall T-score †
Baseline Form 1	8.36 ± 2.52	46.83 ± 11.75	45.99 ± 10.85	46.58 ± 10.60	2	0.70	.497
12-month Form 2	8.12 ± 2.79	47.07 ± 12.83	46.02 ± 11.13	45.78 ± 11.21	2	0.60	.547
24-month Form 3	8.18 ± 2.97	48.02 ± 12.60	45.92 ± 11.75	45.87 ± 12.19	2	3.95	.020*b
36-month Form 4	9.03 ± 2.73	49.11 ± 12.55	47.40 ± 11.15	45.70 ± 12.73	2	5.33	.005*b
LNS Scaled Score †
Baseline	10.59 ± 2.66	11.49 ± 2.68	9.67 ± 3.10	9.84 ± 3.01	2	35.73	<.001*ab
12-month	10.41 ± 2.63	11.38 ± 2.75	9.44 ± 2.96	9.70 ± 3.00	2	41.43	<.001*ab
24-month	10.28 ± 2.82	11.34 ± 2.88	9.23 ± 3.37	9.59 ± 3.25	2	38.15	<.001*ab
36-month	11.02 ± 2.77	11.32 ± 3.12	9.10 ± 3.27	9.30 ± 3.47	2	1538.78	<.001*abc
JLO Scaled Score †
Baseline Odd Numbered Items	12.77 ± 2.13	12.79 ± 2.76	9.47 ± 3.23	9.66 ± 3.11	2	121.99	<.001*ab
12-month Even Numbered Items	12.39 ± 2.37	12.37 ± 2.90	9.69 ± 2.90	9.11 ± 3.38	2	104.90	<.001*abc
24-month Odd Numbered Items	12.78 ± 2.28	12.88 ± 2.74	9.59 ± 3.29	9.72 ± 3.11	2	125.78	<.001*ab
36-month Even Numbered Items	12.54 ± 2.28	12.61 ± 2.86	10.08 ± 2.99	9.10 ± 3.60	2	95.32	<.001*abc
Semantic Fluency T-Score †
Baseline	20.96 ± 5.34	50.79 ± 9.84	47.90 ± 9.95	47.73 ± 9.60	2	9.92	<.001*ab
12-month	20.94 ± 5.40	50.96 ± 10.45	47.03 ± 10.43	47.95 ± 9.98	2	11.70	<.001*ab
24-month	20.92 ± 5.76	51.03 ± 10.87	47.28 ± 10.58	48.10 ± 10.93	2	10.48	<.001*ab
36-month	22.46 ± 5.29	51.19 ± 10.49	46.94 ± 10.32	47.49 ± 10.22	2	15.72	<.001*ab
SDMT z-score †
Baseline Version 1	48.18 ± 9.73	−0.49 ± 0.86	−0.53 ± 0.92	−0.54 ± 0.87	2	0.44	.647
12-months Version 2	40.90 ± 10.13	−0.46 ± 0.92	−0.58 ± 0.90	−0.51 ± 0.96	2	1.26	.284
24-months Version 1	39.85 ± 11.13	−0.50 ± 0.97	−0.90 ± 1.00	−0.55 ± 1.02	2	13.81	<.001*ab
36-months Version 2	47.85 ± 11.05	−0.47 ± 1.03	−0.67 ± 1.01	−0.70 ± 1.11	2	4.42	.012*ab

Notes: Raw Scores are provided for reference only. Repeated Measures ANOVA was conducted between the three normative groups. PPMI = Parkinson’s Progressive Marker Initiative; HC = Healthy Control; HVLT-R = Hopkins Verbal Learning Test Revised; LNS = Letter Number Sequencing, JLO = Judgment of Line Orientation; SDMT = Symbol Digit Modalities Test; Internally Normed z-scores were derived from the mean of the participant score minus the group mean of the healthy control group, divided by the standard deviation of the group mean of the healthy control group for each measure. The HC Overall Group Norms are based on the mean and standard deviation of the entire HC sample. The HC Age-Based Norms utilized norms from the following age ranges: 30–45, 46–60, 61–75, and 76–90. The z-scores were then converted to T-scores or Scaled Scores for direct comparison with published norms, with the exception of SDMT. T-scores have a mean of 50 with a standard deviation of 10; Scaled Scores have a mean of 10 with a standard deviation of 3; z-scores have a mean of 0 with a standard deviation of 1.

^†indicates the between subjects model was significant, p≤.05

^*indicates a significant difference between normative methods, p≤.05

^aindicates a significant difference between published norms and HC overall group mean-based norms, p≤.05

^bindicates a significant difference between published norms and HC age-based norms, p≤.05

^cindicates a significant difference between HC age-based norms and HC overall group mean-based norms, p≤.05

Table 2 demonstrates the percentages of individuals who meet Level I MDS Task Force criteria [17] for MCI based upon the normative sample using 1.5SD below the mean as the cutoff. The relative risk of MCI using internal HC overall group-based norms was 1.5 to 2.1 times higher than published norms. The use of HC age-based norms increased the risk of MCI diagnosis between 1.5 to 1.8 times compared to published norms. There was little to no increased risk of MCI using HC age-based vs. overall group-based norms (relative risk = 1.0–1.3).

Table 2.

Percentage of individuals identified as meeting criteria for MCI based upon Litvan et al. Level I criteria using <1.5 SD on two or more tests

Impaired %				Impaired %
	Published Norms	Internal HC Overall Group-Based Norms	Relative Risk of MCI diagnosis	Published Norms	Internal HC Age-Based Norms	Relative Risk of MCI diagnosis
Baseline	8.3%	17.7%	2.13	8.3%	13.51%	1.63
12-month	9.8%	18.0%	1.84	9.8%	17.21%	1.76
24-month	12.9%	24.7%	1.91	12.9%	18.96%	1.47
36-month	13.3%	20.4%	1.53	13.3%	20.56%	1.55

Notes: MCI = Mild Cognitive Impairment; HC = Healthy Control; ; Internally Normed z-scores were derived from the mean of the participant score minus the group mean of the healthy control group, divided by the standard deviation of the group mean of the healthy control group for each measure. The HC Overall Group Norms are based on the mean and standard deviation of the entire HC sample. The HC Age-Based Norms utilized norms from the following age ranges: 30–45, 46–60, 61–75, and 76–90.

Discussion

Among participants with PD in PPMI, there are differences in standardized cognitive scores depending upon the comparison group that is used. The use of HC internal norms, even when stratified by age, resulted in lower standardized scores than published norms with the exception of memory tests. The use of HC internal norms was also associated with an approximately 1.5 to 2-fold greater risk of participants being classified as having MCI compared to published norms.

Differences between the inclusion and exclusion criteria for the PPMI HC group and normative samples for published tests may explain some of the current results. For example, the JLO published norms are from a community sample and have less stringent criteria than PPMI. The significant difference in JLO performances may also be the result of the way the PPMI utilizes the normative data (e.g., doubling the raw score from the short version and comparing the participants to individuals who were administered the full version).

Given the limitations of utilizing cognitive norms from different sources within a battery of tests, utilizing norms from one sample for all cognitive tests may be advantageous; however, this depends upon the characteristics of the normative sample. The HC group in PPMI is comprised of individuals who scored ≥27 on the MoCA at screening, although a score of 26 is considered to be within normal limits [12]. Furthermore, the HC group has a higher education than the PD group. Therefore, it is possible that the PPMI healthy control group is comprised of an enriched sample of adults with above average cognitive abilities and comparison to this group may over-pathologize the participants with PD.

Using published norms for neuropsychological data in research allows for more direct comparisons when applying research to clinical settings. However, each published test has a different normative sample and there can be substantial differences between the samples [18]. Published norms for LNS [15] have relatively strict inclusion/exclusion criteria for the sample, whereas the JLO norms utilize a community-based sample [19]. Strict inclusion/exclusion criteria for normative groups tend to result in a higher proportion of individuals diagnosed with cognitive impairment, particularly among older adults [20]. Therefore, in PPMI, the difference between the average LNS and JLO performances using published norms may potentially be an artifact of the normative sample, rather than differences in those specific cognitive abilities.

Although there are several strengths related to the cognitive data in the PPMI, one limitation is the absence of an estimate of baseline intellectual functioning for participants. The participants in PPMI average approximately 16 years of education and it is possible that scores within the average range may actually represent a decline.

Future studies should examine the impact of premorbid functioning on longitudinal cognitive studies in participants with PD. Future studies should also focus on the psychometric properties of the tests included in PPMI (e.g., practice effects and equivalency of alternate forms), which may increase measurement error and impact the results of longitudinal research. Although the participants are within the early stages of PD, functional impairment was not evaluated in this study and it is possible that some of the participants met criteria for dementia. Differences in normative scores may be more pronounced in later stages of the disease and should also be explored in participants with more advanced PD.

This study highlights the importance of understanding the characteristics of the comparison group when using standardized cognitive data. Selection of normative comparison groups requires careful consideration, as such decisions impact both research and clinical interpretations of cognitive data.