Skip to main content
NCBI home page
Archives of Clinical Neuropsychology logoLink to Archives of Clinical Neuropsychology
. 2024 Feb 13;39(6):732–746. doi: 10.1093/arclin/acae005

Development and Validity of Norms for Cognitive Dispersion on the Uniform Data Set 3.0 Neuropsychological Battery

Andrew M Kiselica 1,✉, Alyssa N Kaser 2, Daniel S Weitzner 3, Cynthia M Mikula 4, Anna Boone 5, Steven Paul Woods 6, Timothy J Wolf 7, Troy A Webber 8,9
PMCID: PMC11345113  PMID: 38364295

Abstract

Objective

Cognitive dispersion indexes intraindividual variability in performance across a battery of neuropsychological tests. Measures of dispersion show promise as markers of cognitive dyscontrol and everyday functioning difficulties; however, they have limited practical applicability due to a lack of normative data. This study aimed to develop and evaluate normed scores for cognitive dispersion among older adults.

Method

We analyzed data from 4,283 cognitively normal participants aged ≥50 years from the Uniform Data Set (UDS) 3.0. We describe methods for calculating intraindividual standard deviation (ISD) and coefficient of variation (CoV), as well as associated unadjusted scaled scores and demographically adjusted z-scores. We also examined the ability of ISD and CoV scores to differentiate between cognitively normal individuals (n = 4,283) and those with cognitive impairment due to Lewy body disease (n = 282).

Results

We generated normative tables to map raw ISD and CoV scores onto a normal distribution of scaled scores. Cognitive dispersion indices were associated with age, education, and race/ethnicity but not sex. Regression equations were used to develop a freely accessible Excel calculator for deriving demographically adjusted normed scores for ISD and CoV. All measures of dispersion demonstrated excellent diagnostic utility when evaluated by the area under the curve produced from receiver operating characteristic curves.

Conclusions

Results of this study provide evidence for the clinical utility of sample-based and demographically adjusted normative standards for cognitive dispersion on the UDS 3.0. These standards can be used to guide interpretation of intraindividual variability among older adults in clinical and research settings.

Keywords: Assessment, Elderly/Geriatrics/Aging, Norms/normative studies, Dementia with Lewy bodies

INTRODUCTION

Although mean-level test performance is typically utilized in neuropsychological research and clinical practice, alternate approaches have been shown to provide additional diagnostic information (e.g., process approach; Kaplan, 1988). One way of indexing subtle neuropsychological problems in different clinical conditions is to assess cognitive intraindividual variability (IIV; Tractenberg&Pietrzak, 2011). Although there tends to be natural variability in cognitive performance even among healthy adults, pronounced difficulties maintaining consistent performance over time on the same task (i.e., inconsistency) and across tasks (i.e., dispersion) are thought to be pathological (Costa,et al., 2019; Hultsch,et al., 2000; Stuss,et al., 2003). In this study, we focus on cognitive dispersion across a battery of standardized tests, which is arguably the aspect of IIV with the most direct clinical applicability because dispersion scores can be calculated from test scores obtained during routine neuropsychological testing without administration of additional tests (Stuss,et al., 2003; Webber,et al., 2023).

Dispersion in neuropsychological test scores can arise from a variety of different sources, including characteristics of the tests, the scores, and the examinee (Hill,et al., 2013). However, a higher-than-expected dispersion given an examinee’s demographic background may be suggestive of the presence of current or incipient pathology (Hilborn,et al., 2009; Rapp,et al., 2005). Specifically, high dispersion is thought to reflect failure of top–down regulation processes needed to maintain attention over the course of extended testing (i.e., cognitive dyscontrol; Hultsch,et al., 2000; Stuss,et al., 2003; West,et al., 2002). Supporting this notion, measures of cognitive dispersion show negative associations with performance on tasks of executive functions (Sullivan,et al., 2018) and other tasks that involve strategic monitoring over time (Mustafa,et al., 2023), likely regulated by fronto-parietal brain networks (Costa,et al., 2019).

Dispersion has shown great promise as a clinical tool. Measures of dispersion have distinguished between healthy and neurologically compromised individuals (Hultsch,et al., 2000; Strauss,et al., 2002), and helped differentially identify severity and nature of neurological impairment. For example, greater dispersion was found in those with Alzheimer’s disease (AD) compared to Parkinson’s disease (PD), and greater dispersion was found in both groups compared to a healthy comparison group (Burton,et al., 2006). In addition, dispersion scores have been shown to accurately differentiate those with dementia with Lewy bodies from healthy participants, even after controlling for global cognitive performance (Webber,et al., 2022). Measures of dispersion are also an important prognostic indicator over and above other relevant clinical, demographic, and genetic information. They have been shown to predict emergence of AD when controlling for demographic information and APOE E4 allele status (Anderson,et al., 2016). Furthermore, dispersion scores significantly predict AD dementia and mild cognitive impairment diagnoses above Alzheimer’s biomarkers (Gleason,et al., 2018). Finally, among participants with mild cognitive impairment, higher dispersion at baseline is associated with increased likelihood of transitioning to dementia (Roalf,et al., 2016).

Measures of dispersion have also demonstrated ecological validity through their associations with everyday functioning. In samples of healthy older adults, greater dispersion is associated with functional decline over-and-above demographic information (Christensen,et al., 1999) and mean-level cognitive performance (Rapp,et al., 2005). Similarly, in participants with confirmed hepatitis C, higher dispersion was associated with an increased risk of unemployment in those with higher overall mean total scores (Morgan,et al., 2012). And in those with mild traumatic brain injury, unemployment status was associated with higher dispersion but not total mean scores (Sakamoto,et al., 2021). Finally, increased dispersion was associated with worse daily functioning in individuals with dementia with Lewy bodies over-and-above global cognition, dementia status, demographic variables, informant-rated cognitive fluctuations, and clinician-rated cognitive fluctuations (Webber,et al., 2023).

In summary, measures of dispersion have great potential as clinical tools to assist with diagnosis, prognosis, and clinical decision-making. However, interpretation of dispersion indices is complicated because variability in test performance during a lengthy neuropsychological assessment is common, even among healthy adults (Binder,et al., 2009; Kiselica,et al., 2020b; Schretlen,et al., 2003). Thus, it is hard to ascertain the degree to which a given level of dispersion is normal versus abnormal. Interpretation of dispersion measures is also made difficult by conflicting findings regarding the relationship between dispersion scores and age. Some studies suggest that dispersion increases with advancing age (De,Felice&Holland, 2018; Hilborn,et al., 2009), whereas others have reported a non-significant association (Christensen,et al., 1999; Rapp,et al., 2005). Furthermore, to our knowledge, there have been few studies whose goal was to investigate sex and racial/ethnic differences in dispersion. Therefore, it is unclear how test takers’ dispersion scores might be influenced by demographic factors. As a result of these gaps in the literature, measures of dispersion are difficult to integrate into clinical and research practice. These issues can be resolved by provision of demographically adjusted normative scores for measures of dispersion (Kiselica,et al., 2023), which would allow for interpretation of dispersion scores as reflective of a percentile rank within a reference population (Lezak,et al., 2012).

Therefore, the aims of the current study were to (a) establish unadjusted normed scaled scores for dispersion indices; (b) explore the relationship between dispersion scores and demographics factors; (c) derive demographically adjusted normed scores for dispersion indices; (d) provide a user-friendly calculator for obtaining demographically adjusted normed scores in practice; and (e) evaluate the diagnostic utility of these scores. Because dispersion has commonly been applied to better understand aging and neurodegenerative disease, we sought to develop normed scores using a well-defined cohort of older adults from the National Alzheimer’s Coordinating Center Uniform Data Set (UDS) 3.0. This battery is employed at every Alzheimer’s Disease Research Center and has been investigated as a clinical tool (Devora,et al., 2019; Kiselica,et al., 2020; Kiselica,et al., 2020a; Sachs,et al., 2020; Wang,et al., 2021; Weintraub,et al., 2018). Given the relevance of intraindividual cognitive variability to cognitive fluctuations and functional outcomes in Lewy body disease (Webber,et al., 2022; Webber,et al., 2023), we evaluated the ability of normed dispersion scores to differentiate those with cognitive impairment due to Lewy body disease from cognitively normal individuals.

MATERIALS AND METHODS

Normative Sample

Data from the UDS were requested from the National Alzheimer’s Coordinating Center (NACC) portal on September 17, 2022. This data file included participant information collected at 39 Alzheimer’s Disease Research Centers from the commencement of data collection to the June 2022 data freeze. Participants provided written informed consent as part of the enrollment and data collection process at their designated centers. All data were recorded via electronic or printed UDS forms using a standardized evaluation process. Further details on the UDS sample and data collection process have been outlined elsewhere (Beekly,et al., 2004; Besser,et al., 2018; Morris,et al., 2006; Weintraub,et al., 2018).

Figure 1 displays our normative sample selection process in a STROBE diagram (Von,Elm,et al., 2007). Our initial sample included individuals receiving the latest version of the UDS (version 3.0) at their initial visit because of our interest in the most recent version of the neuropsychological battery. Next, we limited our sample based on demographic factors. First, we restricted our analyses to participants aged 50 and over. Second, we excluded individuals who did not speak English as their primary language.

Fig. 1.

Fig. 1

Open in a new tab

STROBE diagram for normative sample selection. UDS3NB = Uniform Data Set 3.0 Neuropsychological Battery. CDR = Clinician Dementia Rating.

Then to create a normative sample, we excluded all individuals who were not classified as cognitively normal at baseline as measured by the Clinical Dementia Rating® (CDR) Scale or NACC clinical diagnosis. The CDR provides a global rating of cognitive impairment (0 = cognitively normal, 0.5 = mild cognitive impairment [MCI], 1 = mild dementia, 2 = moderate dementia, 3 = severe dementia) and has well-established reliability and validity (Fillenbaum,et al., 1996; Morris, 1993, 1997; Morris,et al., 1997). In addition, the cognitive status of each NACC participant is determined at every UDS visit by a single clinician or consensus panel using form D1 (available at: https://naccdata.org/data-collection/forms-documentation/uds-3). Diagnoses are based on available interview, cognitive, behavioral, biomarker, imaging, and genetic data with the following ratings: 1 = cognitively normal, 2 = cognitively impaired without meeting criteria for MCI, 3 = MCI, 4 = dementia. The normative sample used in the present study featured individuals with a CDR score of zero and an NACC clinical diagnosis score of one.

Finally, we removed cases wherein there was missing data on either a neuropsychological test used in the calculation of the dispersion indices (due to physical problems, verbal refusal, or other problems) or a demographic variable of interest (age, sex, education, and race/ethnicity). These steps were taken to ensure a uniform normative sample and because all neuropsychological test scores were desirable in the calculation of dispersion (Webber,et al., 2022). In contrast, participants with missing data due to cognitive or behavioral issues were retained by using established winsorization methods (Benge,et al., 2020). Specifically, individuals whose data were missing due to a cognitive or behavioral problem were coded as having scored at the lowest possible value on the missing test. For most tests, this meant the raw score was coded as 0. The exception was the Trail Making Test, wherein a score of 150 was assigned on Part A and a score of 300 was assigned on Part B. Following all steps resulted in a final sample of 4,283 participants (see Table 1 for demographic characteristics and raw cognitive test performance at baseline).

Table 1.

Demographic and cognitive characteristics of the normative sample

Demographic variables Total sample (n = 4283)
Age M (SD), range 69.42 (7.83), 50–101
Education M (SD), range 16.42 (2.39), 1–20
Sex % female 66.54%
Race %
 White 77.37%
 Black/African American 18.79%
 Asian 1.78%
 American Indian/Alaska Native 1.27%
 Native Hawaiian/Pacific Islander 0.09%
 Other 0.70%
Ethnicity % Hispanic/Latino 5.16%
Raw neuropsychological test scores (possible score range) M (SD)
Benson Copy (0–17) 15.51 (1.36)
Benson Recall (0–17) 11.35 (2.90)
Animal Naming (0-infinite) 21.60 (5.44)
Vegetable Naming (0-infinite) 15.02 (4.08)
Trail Making Part A (0–150) 31.55 (12.04)
Trail Making Part B (0–300) 81.45 (39.32)
Letter Fluency (0-infinite) 28.55 (8.02)
Craft Story IR (0–44) 22.01 (6.41)
Craft Story DR (0–44) 19.17 (6.47)
Number Span Forward (0–14) 8.34 (2.30)
Number Span Backward (0–14) 7.09 (2.15)
MINT (0–32) 30.12 (2.04)
Open in a new tab

Note: Male coded as 0, female coded as 1. Non-Hispanic White coded as 0, non-White/Hispanic coded as 1. Numbers Forward = Number Span Forward; Numbers Backward = Number Span Backward; IR = immediate recall; DR = delayed recall; MINT = Multilingual Naming Test.

Clinical Sample

To assess the validity of normed dispersion scores, we evaluated whether they could differentiate cognitively normal individuals from those with mild cognitive impairment or mild dementia due to Lewy body disease. To obtain this clinical sample, we limited our sample based on demographic factors and data availability as earlier. Next, we selected individuals who had a CDR global score of 0.5 (MCI) or 1 (mild dementia) and a clinical diagnosis of 2 (impaired, not MCI), 3 (MCI), or 4 (mild dementia). Finally, we selected individuals who had a primary etiological diagnosis of Lewy body disease. These steps yielded a sample of 282 individuals with cognitive impairment due to Lewy body disease. Characteristics of this sample are outlined in Table 2. Of note, this sample has overlap with samples investigated in prior work on dispersion (Webber,et al., 2022; Webber,et al., 2023).

Table 2.

Demographic and cognitive characteristics of the clinical sample

Cognitively normal (n = 4283) Impaired-LBD (n = 282) t or chi, p-value
Demographic variables
Age, M (SD) 69.42 (7.83) 70.46 (8.01) 2.16, *
Education, M (SD) 16.42 (2.39) 16.68 (2.56) 1.78, p = .07
Sex % Female 66.54% 16.67% 281.69, ***
Race/Ethnicity % Non-White/Hispanic 26.87% 3.90% 72.38, ***
Raw neuropsychological test scores M (SD)
Benson Copy 15.51 (1.36) 13.30 (3.74) −9.89, ***
Benson Recall 11.35 (2.90) 7.35 (3.93) −16.79, ***
Animal Naming 21.60 (5.44) 14.89 (5.93) −18.48, ***
Vegetable Naming 15.02 (4.08) 8.89 (3.88) −25.66, ***
Trail Making Part A 31.55 (12.04) 66.85 (38.64) 15.29, ***
Trail Making Part B 81.45 (39.32) 199.25 (93.32) 21.08, ***
Letter Fluency 28.55 (8.02) 21.40 (9.24) −12.70, ***
Craft Story IR 22.00 (6.41) 14.51 (7.41) −16.58, ***
Craft Story DR 19.17 (6.47) 11.55 (6.97) −17.87, ***
Numbers Forward 8.34 (2.30) 7.39 (2.35) −6.58, ***
Numbers Backward 7.09 (2.15) 5.04 (2.14) −15.57, ***
MINT 30.11 (2.04) 28.27 (4.97) −6.19, ***
Open in a new tab

Note: Male coded as 0, female coded as 1. Non-Hispanic White coded as 0, non-White/Hispanic coded as 1. LBD = Lewy Body disease; Numbers Forward = Number Span Forward; Numbers Backward = Number Span Backward; IR = immediate recall; DR = delayed recall; MINT = Multilingual Naming Test. Two-sample t-tests were used for continuous variables, whereas chi-square tests were used for categorical variables. *p < .05, **p < .01, ***p < .001.

Measures

Cognitive assessments

The Uniform Data Set 3.0 Neuropsychological Battery (UDS3NB) measures have been thoroughly described in prior literature (Besser,et al., 2018; Weintraub,et al., 2018). The current study utilized 12 core scores from six neurocognitive tests (Kiselica,et al., 2020a, 2020b), including (a) the Craft Story, a test of immediate and delayed recall for information presented in a story format (verbatim recall scores used; Craft,et al., 1996); (b) the Benson Figure, a measure of visuoconstruction and delayed visual recall abilities (Possin,et al., 2011); (c) Number Span Forward and Backward Test, a digit repetition task that measures attention and working memory; (d) the Multilingual Naming Test (MINT), a measure of confrontation naming (Gollan,et al., 2012; Ivanova,et al., 2013); (e) verbal fluency, measured by letter (F- and L-) and semantic (animals and vegetables) fluency tasks; and (f) Trail Making Test Parts A (TMT A) and B (TMT B), which assess simple number sequencing and alternating letter–number sequencing, respectively (Partington&Leiter, 1949). Measures of dispersion on this battery of tests have been shown to be related to diagnostic status within the UDS sample, as well as cognitive fluctuations and functional outcomes among individuals with dementia with Lewy bodies (Webber,et al., 2022; Webber,et al., 2023).

Analyses

Dispersion indices

Dispersion was estimated using two different indices. The first was the intraindividual standard deviation (ISD; Holtzer,et al., 2008; Morgan,et al., 2011; Webber,et al., 2022; Webber,et al., 2023). To calculate ISD, raw scores from the 12 UDS3NB indicators were transformed into demographically adjusted z-scores (adjusting for the demographic variables age, sex, and education) using published normative data (Weintraub,et al., 2018). These z-scores were then transformed into T-scores, using the following formula: (10*z-score) + 50. Next, the ISD was calculated by taking the square root of the average squared deviation of each test T-score from the individual’s UDS3NB normative mean score. The normative mean is the average of the demographically adjusted T-scores for all 12 neuropsychological test variables. In addition to the ISD, we calculated the coefficient of variation (CoV), which provides an adjustment for global cognitive performance. CoV is calculated by dividing the ISD by the participant’s USD3NB normative mean score (i.e., the average of the demographically adjusted T-scores for all 12 neuropsychological test variables). This transformation controls for systematic between-subject confounds in terms of level of global cognitive abilities (Stawski,et al., 2017).

Normed score calculation

We provided normed scores in two ways. First, we used the raw2scaled function from the test2norm R package (Umlauf&Umlauf, 2022) to map the raw ISD and CoV scores onto a normalized distribution of scaled scores, following the recommendations of Heaton,and,colleagues,(2003) and procedures used in similar norming studies with the UDS3NB (e.g., Marquine,et al., 2023). This step also allowed for us to meet assumptions of normality required for ordinary least squared regression when calculating regression-based normed scores. Scaled scores were coded such that increased dispersion was associated with a lower scaled score, given that higher dispersion is usually associated with worse clinical outcomes, and neuropsychologists typically assume lower normed scores indicate worse performance. This step resulted in a table of unadjusted scaled dispersion scores.

Second, we used the score2adjust function to derive a best-fitting regression equation for creating adjusted scaled scores created in the previous step. This function uses a multivariable fractional polynomial regression approach to test inclusion of linear and polynomial terms (i.e., investigation of different non-linear associations between predictor and outcome) for each predictor variable with backward elimination of non-significant terms based on a prespecified alpha value for inclusion (Heinze, 2022; Royston&Sauerbrei, 2008). We selected an alpha value of 0.05. The function also identified transformations that could be applied to the scaled scores for ease of calculation. Predictor variables included age, years of education (recoded such that the highest possible value was 20), sex (male = 0, female = 1), and race/ethnicity (White/non-Hispanic = 0 vs. non-White/Hispanic = 1)1. Coefficients from the final model were extracted to create regression equations for calculating demographically adjusted z-scores, following methods outlined by Shirk,and,coworkers,(2011). These formulas were then applied in an Excel calculator to create a freely available tool for use in clinical and research contexts.

Validation

We evaluated the ability of six different measures of dispersion to distinguish between cognitively normal individuals and those with cognitive impairment due to Lewy body disease: (a) raw unadjusted ISD scores, (b) raw unadjusted CoV scores; (c) unadjusted ISD scaled scores; (d) unadjusted CoV scaled scores; (e) demographically adjusted ISD z-scores; and (f) demographically adjusted CoV z-scores. Receiver operating characteristic (ROC) curves were constructed using the ROCit package in R (Khan, 2019). Areas under the curves (AUCs) were used as measures of overall classification accuracy and interpreted according to established rules of thumb (Hosmer,et al., 2013), which describe AUC values under 0.70 as unacceptable, those between 0.7 and 0.8 as acceptable, those between 0.8 and 0.9 as excellent, and those ≥0.9 as outstanding. Cutpoints that best balanced sensitivity and specificity for detection of cognitive impairment were assessed with Youden’s J statistic (Youden, 1950) using the cutpointr package in R (Thiele&Hirschfeld, 2020). Sensitivities and specificities at optimal cutpoints are presented in Table 5.

Table 5.

Optimal cutpoints for cognitive dispersion measures with associated sensitivity and specificity values

Variable Optimal cutpoint Youden Index (J) Sensitivity Specificity
Raw ISD 10.8669 0.5239 0.6206 0.9033
Raw CoV 0.2244 0.5898 0.7021 0.8877
Scaled ISD 8.7506 0.5171 0.6137 0.9033
Scaled CoV 9.2253 0.5986 0.7111 0.8875
Dem. Adj. ISD z-score −0.5124 0.5298 0.6029 0.9269
Dem. Adj. CoV z-score −0.3239 0.6362 0.7296 0.9066
Open in a new tab

Note: ISD = intraindividual standard deviation; CoV = coefficient of variation; Dem = demographically; Adj = adjusted.

RESULTS

Intraindividual Standard Deviation

The distribution of the raw, unadjusted ISD scores had a mean of 8.01 and a standard deviation of 2.43. It was highly non-normal (skewness = 1.88, kurtosis = 13.95). Use of the raw2scaled function mapped the raw, unadjusted ISD scores onto a normalized distribution of unadjusted scaled scores as shown in Table 3.

Table 3.

Raw unadjusted intraindividual standard deviation score to unadjusted scaled score conversion table

Raw score minimum Raw score maximum Scaled score
1.75 1.7736 21
1.7737 2.6495 20
2.6496 3.1331 19
3.1332 3.5855 18
3.5856 4.1264 17
4.1265 4.5212 16
4.5213 5.0721 15
5.0722 5.5946 14
5.5947 6.1472 13
6.1473 6.7459 12
6.746 7.3499 11
7.35 8.0337 10
8.0338 8.7816 9
8.7817 9.5855 8
9.5856 10.5356 7
10.5357 11.5315 6
11.5316 12.7658 5
12.7659 14.5849 4
14.585 17.2418 3
17.2419 20.6735 2
20.6736 35.1 1
Open in a new tab

Note: Because higher dispersion is usually associated with worse clinical outcomes, scaled scores were coded such that increased dispersion corresponded with a lower scaled score.

Use of the score2adjust function with ISD scaled scores as the outcome and demographic variables as predictors yielded a best-fitting regression model without sex. There was a linear association of race/ethnicity with ISD. Non-White/Hispanic individuals had lower scaled ISD scores than White/non-Hispanic individuals (B = −0.82, p < .001). There were non-linear associations of education and age with ISD. The education term was raised to the power of −0.5 (B = −2.67, p < .001). Examination of the top-left panel in Fig. 2 indicates that increasing levels of education were associated with decreased dispersion; this relationship was stronger at lower levels of education and weaker at higher levels of education. Next, the age term was cubic (B = −2.41, p < .001). Examination of the top-right panel in Fig. 2 indicates that dispersion increased slightly from age 50 to around age 55, then plateaued, before beginning to increase dramatically in the 80s. The final best-fitting equation was as follows:

Fig. 2.

Fig. 2

Open in a new tab

Top-left panel: Relationship between education and intraindividual standard deviation (ISD) scaled scores. Top-right panel: Relationship between age and ISD scaled scores. Bottom panel: Relationship between age and coefficient of variation scaled scores. Note that lower scaled scores indicate greater dispersion.

graphic file with name DmEquation1.gif

Applying this equation resulted in a near normal distribution of demographically adjusted z-scores for ISD from which percentile ranks can be derived (Shapiro–Wilk normality test: w = 0.99, p = .7726; see Fig. 3 top panel). This equation was used to construct a user-friendly Excel calculator for estimating demographically adjusted z-scores and percentile ranks for ISD (see supplemental materials).

Fig. 3.

Fig. 3

Open in a new tab

Top panel: Frequency distribution of demographically adjusted z-scores for the intraindividual standard deviation. Bottom panel: Frequency distribution of demographically adjusted z-scores for the coefficient of variation.

Coefficient of Variation

The distribution of raw, unadjusted CoV scores had a mean of 0.17 and a standard deviation of 0.06. It was highly non-normal (skewness = 2.79, kurtosis = 21.25). Use of the raw2scaled function mapped the raw, unadjusted CoV scores onto a normalized distribution of unadjusted scaled scores as shown in Table 4.

Table 4.

Raw unadjusted coefficient of variation score to unadjusted scaled score conversion table

Raw score minimum Raw score maximum Scaled score
0.03 0.0337 21
0.0338 0.0498 20
0.0499 0.0564 19
0.0565 0.0683 18
0.0684 0.0784 17
0.0785 0.0885 16
0.0886 0.0989 15
0.099 0.1104 14
0.1105 0.1217 13
0.1218 0.1345 12
0.1346 0.1482 11
0.1483 0.1643 10
0.1644 0.181 9
0.1811 0.2 8
0.2001 0.2215 7
0.2216 0.2508 6
0.2509 0.2854 5
0.2855 0.347 4
0.3471 0.4249 3
0.425 0.5184 2
0.5185 0.86 1
Open in a new tab

Note: Because higher dispersion is usually associated with worse clinical outcomes, scaled scores were coded such that increased dispersion corresponded with a lower scaled score.

Use of the score2adjust function with CoV scaled scores as the outcome yielded a best- fitting regression without sex or education in the model. There was a linear association of race/ethnicity with CoV. Non-White/Hispanic individuals had lower scaled CoV scores than White/non-Hispanic individuals (B = −1.55, p < .001). There was a complex non-linear relationship between age and CoV, which included a cubic term (B = −2.22, p < .001) and a cubic * logarithmic term (B = −15.14, p = .001). Examination of the bottom-right panel of Fig. 2 indicates a slowly accelerating increase in dispersion with increasing age. The final best-fitting equation was as follows:

graphic file with name DmEquation2.gif

Applying this equation resulted in a near-normal distribution of demographically adjusted z-scores for CoV from which percentile ranks can be derived (Shapiro–Wilk normality test w = 0.99, p = .2928; see Fig. 3 bottom panel). This equation was used to construct a user-friendly Excel calculator for estimating demographically adjusted z-scores and percentile ranks for CoV (see supplemental materials).

Validity of Dispersion Measures

We evaluated the ability of six different measures of dispersion to distinguish between cognitively normal individuals and those with cognitive impairment due to Lewy body disease using ROC curves and associated AUC values. As can be seen in Fig. 4, all AUC values fell in the excellent range (values ranged from 0.81 to 0.88), with the demographically adjusted CoV z-scores yielding the best absolute performance. Optimal cutpoints for each dispersion measure were defined using the J statistic and are shown in in Table 5, along with associated sensitivity and specificity values.

Fig. 4.

Fig. 4

Open in a new tab

Receiver operating characteristic (ROC) curves and associated area under the curve (AUC) and 95% confidence interval values for dispersion measures predicting group membership for cognitively normal individuals versus individuals with cognitive impairment due to Lewy body disease. ISD = intraindividual standard deviation, CoV = coefficient of variation.

DISCUSSION

Measures of cognitive intraindividual variability dispersion show promise as indicators of central nervous system dysfunction and poor everyday functioning. Application of these metrics has been limited by a dearth of available norms. Normative values of dispersion are essential for determining if clinical dispersion scores are greater in magnitude than what would be expected for an individual’s demographic background. Therefore, we provided methods for calculating two dispersion indices, ISD and CoV. Furthermore, we developed tools to calculate both unadjusted and demographically adjusted normed scores for these metrics, including a user-friendly Excel calculator. Finally, we assessed the validity of these scores for differentiating cognitively normal individuals from those with cognitive impairment due to Lewy body disease.

Normative Scaled Scores for ISD and CoV

First, we provided tables for mapping raw ISD and CoV scores onto a normal distribution of scaled scores, following the recommendations of Heaton,and,colleagues,(2003). Scaled scores were coded such that increased dispersion was associated with a lower scaled score, given that higher dispersion is usually associated with worse clinical outcomes. These tables are useful for summarizing degree of dispersion quickly for an older adult without making demographic adjustments. Practitioners may wish to use these tables in cases wherein demographic adjustment might not be warranted. For example, it has been argued that when there is potential for denial of services or financial compensation, it may be inappropriate to adjust for race/ethnicity (Kiselica,et al., 2021; Manly, 2005; Possin,et al., 2021).

It is important to note differences in interpretation when using ISD versus CoV as the dispersion metric. ISD is a raw measure of the degree to which scores vary from the overall test battery mean. CoV adjusts ISD for global cognitive performance by dividing ISD by the overall test battery mean. As a result, two people can have the same ISD score, but two different CoV scores. Imagine that Person 1 has an overall test battery mean T-score of 40 and an ISD of 10, while Person 2 has an overall test battery mean T-score of 60 and an ISD of 10. These individuals show the same degree of dispersion when indexed by ISD. However, the CoV for Person 1 and Person 2 would be 0.25 and 0.167, respectively. In this case, CoV adjusts for the fact that Person 2, who has a higher overall test battery mean, will have ISD scores that are more extremely affected by a few low scores. The presence of a few low scores is common even among individuals high in intelligence (Heyanka,et al., 2013; Jeffay,et al., 2021), such that using CoV instead of ISD may avoid the problem of overpathologizing scattered low scores for these individuals.

Demographically Adjusted Normative Scores for ISD and CoV

Of course, unadjusted scores are less likely to be specific to an individual participant or patient, and there are many cases in which demographic adjustments may be advantageous. To investigate the potential value of demographic adjustments for ISD and CoV, we evaluated relationships of age, sex, education, and race/ethnicity with dispersion using multivariable fractional polynomial regression. First, age demonstrated a complex non-linear relationship with both ISD and CoV, wherein there was an increasing influence of age on dispersion at older age ranges. This finding suggests that there is nuance to the interpretation of prior results indicating a positive linear correlation between age and dispersion (Hilborn,et al., 2009), such that dispersion is most influenced by age among those in the oldest age range. Second, sex was not associated with either ISD or CoV. This result replicates prior research, which suggested that initial sex differences in dispersion in middle age were no longer significant in older age (Bielak,et al., 2014). Third, our results indicated that lower education was associated with more dispersion as indexed by the ISD, particularly at the low end of the education spectrum. This finding is consistent with prior work on dispersion (Christensen,et al., 2005), as well as broader research suggesting education is protective against cognitive problems (Wilson,et al., 2019). Notably, education was not significantly related to CoV. This result makes sense, given that CoV adjusts the ISD by the overall mean test battery score, thus limiting the impact of higher academic achievement on dispersion. Finally, non-White/Hispanic individuals showed increased dispersion in comparison to White/non-Hispanic participants. To our knowledge, no prior studies have examined the relationship of race/ethnicity with measures of dispersion. Our novel results fit with the expansive body of literature suggesting that non-White and Hispanic individuals may evidence worse cognitive performance due to a variety of structural and social factors associated with racism, even when accounting for differences in educational attainment (Adkins-Jackson,et al., 2023).

Results of multivariable fractional polynomial regression were used to construct equations for calculating demographically adjusted z-scores for dispersion measures, which formed the basis for a user-friendly Excel calculator (see supplemental materials). Examining hypothetical values in this calculator shows the extent to which demographic adjustments may alter interpretation of dispersion measures. Supplementary material online, Fig. S1 shows hypothetical data for a 70-year-old Black woman with 10 years of education. The raw CoV in this case is 0.25, corresponding with an unadjusted scaled score of 6 (~9th percentile; below average range). This score indicates a relatively rare, high degree of dispersion, possibly concerning for executive dyscontrol. Applying demographic adjustment results in a z-score of −0.66 (~25th percentile; average range). This score is fairly typical for this individual and is likely not concerning for executive dyscontrol.

Of course, it must be noted that interpretation of the CoV variable is not always this straightforward: “While … [CoV] represents a quantitatively-simple approach to take mean into account when examining … [dispersion], the index itself is a ratio—an interaction (ISD multiplied by the inverse of the mean). Interpreting this ratio is not straightforward as any observed association may reflect a relationship between ISD, the (inverse of the) mean, or their interaction” (Stawski,et al., 2019, p. 406). In our analyses, ISD and CoV exhibited similar relationships with outcomes, suggesting they can be interpreted similarly. However, users are cautioned against interpreting CoV without reference to its relationship with ISD.

Practical Application of the Calculator

There are several potential practical applications of this calculator, which will require investigation in future research. First, the methods provided in this manuscript and the associated calculator can be used as a template for generating normed scores for other batteries and populations. Second, the normed scores for ISD and CoV might be applied to assist in diagnosis and differential diagnosis. Prior research suggests that measures of dispersion are useful in differentiating cognitively unimpaired individuals from those with cognitive impairment due to infection (Morgan,et al., 2012; Morgan,et al., 2011), brain injury (Merritt,et al., 2018), or neurodegenerative disease (Webber,et al., 2022). Furthermore, increased dispersion may be suggestive of one etiology versus another (e.g., Alzheimer’s disease vs. Parkinson’s disease; Burton,et al., 2006). Third, normed scores for dispersion measures may be used in prognostication, as higher dispersion has been associated with transitioning from mild cognitive impairment to dementia (e.g., Roalf,et al., 2016). And fourth, normed dispersion indices may be useful in informing decisions about functional capacity, as higher dispersion has been repeatedly related to poorer everyday functioning (Bangen,et al., 2019; Costa,et al., 2019; Webber,et al., 2023).

Validity of Dispersion Scores

Our results suggested that raw, unadjusted normed, and demographically adjusted normed scores all show excellent ability to differentiate cognitively normal individuals from those with cognitive impairment due to Lewy body disease. Results are consistent with prior literature on this topic (Webber,et al., 2022). These findings suggest that when distinguishing cognitively normal individuals from those with cognitive impairment due to Lewy body disease, any of the six cognitive dispersion measures are appropriate for clinical classification purposes. However, there may be a slight advantage to using demographically adjusted CoV scores, which outperformed other measures slightly.

Optimal cutpoints and associated sensitivity and specificity values for each measure are provided in Table 5 to facilitate their use in clinical practice. Optimal cutpoints yielded modest sensitivity but excellent specificity. Of note, even a modestly low z-score for ISD (−0.5124) or CoV (−0.3239) was highly specific for cognitive impairment due to Lewy body disease. This finding may be surprising at first glance, given that scores ≤−1 SD below the mean are more typically associated with cognitive impairment. However, one interpretation of these results is that the demographically adjusted scores for ISD and CoV are simply extremely specific indicators of cognitive impairment due to Lewy body disease. Indeed, demographically adjusted regression-based normed scores enhance specificity (Kiselica,et al., 2023) and measures of dispersion may reflect cognitive fluctuations (Webber,et al., 2022), a highly specific cardinal feature of Lewy body disease (McKeith,et al., 2017). Stated another way, low sensitivities but high specificities reflect the fact that while not all persons with Lewy disease would be expected to demonstrate elevated dispersion, those who do are highly suspect for the disorder. Thus, cutpoints should be employed to confirm suspicion of cognitive impairment due to Lewy body disease and not for broad detection purposes. And, of course, future research is needed to evaluate diagnostic accuracy of these dispersion measures in other clinical groups.

Limitations and Future Directions

The tools provided in this manuscript should be applied with certain limitations in mind.

First, care must be taken in the interpretation of dispersion measures. Some research suggests that higher dispersion scores are indicative of brain dysfunction in fronto-parietal regions (Costa,et al., 2019). However, there are many different potential interpretations of within-person variability in test performance. For example, extreme inconsistencies may provide evidence of performance validity concerns (Sherman,et al., 2020). Furthermore, contextual factors, such as poor sleep, pain, stress, and life complexity may also affect test performance and moderate the relationship of test scores with outcomes (Suchy,et al., 2020). Thus, there is a degree of clinical expertise and judgment needed to incorporate dispersion indices into clinical practice.

Second, it should be noted that data for all measures on the UDS3NB are needed to calculate ISD and CoV, limiting application in cases where there are missing data. In cases where the cause of missingness is the presence of cognitive impairment (e.g., Trail Making Part B discontinued due to confusion), we recommend that scores be winsorized to allow for use of the calculator. However, when missingness is due to other causes, the calculator should not be used. It is likely possible to allow for calculation of ISD and CoV when there are missing data, but the number of possible patterns of missingness is quite high. This obstacle might be overcome in the future with sophisticated computer programming.

Third, users of the calculator are cautioned to consider the makeup of this sample when applying results to an individual participant or patient. For certain characteristics, there is good representation within the UDS. Strengths include availability of data for both sexes, a range of education levels, a large age range (50–101), and a solid representation of participants identifying as Black (nearly 20%). Availability of data for other groups (e.g., Asian, Native American) is limited, however. Furthermore, participants in the UDS may be more representative of research cohorts than persons from the community at large (Arce,Rentería,et al., 2023; Wang,et al., 2021). Norms for dispersion indices should be expanded to include more diverse groups, as well as clinical and community samples.

Fourth, we did not explore interaction effects among demographic variables in regressions. This choice was made given that current software packages for multivariable fractional polynomial analyses do not support systematic investigation of interactions, requiring individual analyses to be conducted for each interaction term. In our case, there would be nearly 500 potential two-way interactions to model, given the number of demographic variables and potential polynomial terms. Further research on this topic is warranted.

Fifth, it must be noted that the UDS does not include independent measures of performance validity. There are several potential raw and age-adjusted score embedded performance validity measures that can be examined in these data. However, it would be difficult to ascertain whether below cutoff performances on these measures reflect validity concerns versus true low scores (Erdodi&Lichtenstein, 2017). Past research in the same cohort suggests that rates of below-cutoff performance on these measures in the UDS are typically quite low (i.e., <10%; Hromas,et al., 2022). Thus, the data are unlikely to be significantly affected by performance validity problems. In a real-world assessment, however, it would be important to interpret IIV scores in light of performance validity scores and clinical context, as elevated dispersion may reflect “compelling inconsistencies” associated with performance validity failures (Sherman,et al., 2020).

CONCLUSIONS

This is the first known study to provide transparent, replicable methods for calculating normed scores for dispersion indices among older adults. A user-friendly Excel calculator is included in the supplemental materials to facilitate use of these normed scores in practice. Our findings suggest that the normed scores may have particular value in differentiating cognitively normal individuals from those with cognitive impairment due to Lewy body disease. However, more work is needed to provide evidence for the validity of these scores in other populations and for other purposes (e.g., prognostication). Such advances will provide confidence in the application of normed dispersion indices in research and clinical practice.

Supplementary Material

UDS_3_0_IIV_Normed_Score_Calculator_FINAL_CORRECTED_acae005
uds_3_0_iiv_normed_score_calculator_final_corrected_acae005.xlsx (18.5KB, xlsx)

ACKNOWLEDGEMENTS

We would like to thank Anya Umlauf for her guidance in using the test2norm package. The NACC database is funded by NIA/NIH Grant U24 AG072122. NACC data are contributed by the NIA-funded ADRCs: P30 AG062429 (PI James Brewer, MD, PhD), P30 AG066468 (PI Oscar Lopez, MD), P30 AG062421 (PI Bradley Hyman, MD, PhD), P30 AG066509 (PI Thomas Grabowski, MD), P30 AG066514 (PI Mary Sano, PhD), P30 AG066530 (PI Helena Chui, MD), P30 AG066507 (PI Marilyn Albert, PhD), P30 AG066444 (PI John Morris, MD), P30 AG066518 (PI Jeffrey Kaye, MD), P30 AG066512 (PI Thomas Wisniewski, MD), P30 AG066462 (PI Scott Small, MD), P30 AG072979 (PI David Wolk, MD), P30 AG072972 (PI Charles DeCarli, MD), P30 AG072976 (PI Andrew Saykin, PsyD), P30 AG072975 (PI David Bennett, MD), P30 AG072978 (PI Neil Kowall, MD), P30 AG072977 (PI Robert Vassar, PhD), P30 AG066519 (PI Frank LaFerla, PhD), P30 AG062677 (PI Ronald Petersen, MD, PhD), P30 AG079280 (PI Eric Reiman, MD), P30 AG062422 (PI Gil Rabinovici, MD), P30 AG066511 (PI Allan Levey, MD, PhD), P30 AG072946 (PI Linda Van Eldik, PhD), P30 AG062715 (PI Sanjay Asthana, MD, FRCP), P30 AG072973 (PI Russell Swerdlow, MD), P30 AG066506 (PI Todd Golde, MD, PhD), P30 AG066508 (PI Stephen Strittmatter, MD, PhD), P30 AG066515 (PI Victor Henderson, MD, MS), P30 AG072947 (PI Suzanne Craft, PhD), P30 AG072931 (PI Henry Paulson, MD, PhD), P30 AG066546 (PI Sudha Seshadri, MD), P20 AG068024 (PI Erik Roberson, MD, PhD), P20 AG068053 (PI Justin Miller, PhD), P20 AG068077 (PI Gary Rosenberg, MD), P20 AG068082 (PI Angela Jefferson, PhD), P30 AG072958 (PI Heather Whitson, MD), P30 AG072959 (PI James Leverenz, MD).

Footnotes

1

Due to small cell sizes in a number of racial/ethnic groups, we took the approach of dichotomizing race/ethnicity as White/non-Hispanic versus non-White/Hispanic. This approach has been used in prior research: Kiselica, A. M., Kaser, A., Webber, T. A., Small, B. J., & Benge, J. F. (2020). Development and preliminary validation of standardized regression-based change scores as measures of transitional cognitive decline Archives of Clinical Neuropsychology, 35(7), 1168–1181.

Contributor Information

Andrew M Kiselica, Department of Health Psychology, University of Missouri, Columbia, MO, USA.

Alyssa N Kaser, Department of Psychiatry, University of Texas Southwestern Medical Center, Dallas, TX, USA.

Daniel S Weitzner, James A. Haley Veterans Hospital, Tampa, FL, USA.

Cynthia M Mikula, Institute of Human Nutrition, Columbia University, New York, NY, USA.

Anna Boone, Department of Occupational Therapy, University of Missouri, Columbia, MO, USA.

Steven Paul Woods, Department of Psychology, University of Houston, Houston, TX, USA.

Timothy J Wolf, Department of Occupational Therapy, University of Missouri, Columbia, MO, USA.

Troy A Webber, Mental Health Care Line, Michael E. DeBakey VA Medical Center, Houston, TX, USA; Department of Psychiatry and Behavioral Sciences, Baylor College of Medicine, Houston, TX, USA.

FUNDING

Dr Kiselica is supported by a career development award from the National Institute on Aging (NIA) of the National Institutes of Health under Award Number U54AG063546, which funds NIA Imbedded Pragmatic Alzheimer’s Disease and AD-Related Dementias Clinical Trials Collaboratory (NIA IMPACT Collaboratory). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

CONFLICT OF INTEREST

None declared.

AUTHOR CONTRIBUTIONS

Andrew Kiselica (Conceptualization, Data curation, Formal analysis, Methodology, Project administration, Software, Supervision, Visualization, Writing—original draft, Writing—review & editing), Alyssa Kaser (Data curation, Methodology, Writing—original draft, Writing—review & editing), Daniel Weitzner (Writing—original draft, Writing—review & editing Cynthia Mikula Data curation) Anna Boone (Writing—original draft), Steven Woods (Conceptualization, Data curation, Writing—review & editing), Timothy Wolf (Writing—review & editing), and Troy Webber (CRediT contribution not specified)

References

  1. Adkins-Jackson, P. B., George, K. M., Besser, L. M., Hyun, J., Lamar, M., Hill-Jarrett, T. G., et al. (2023). The structural and social determinants of Alzheimer's disease related dementias. Alzheimer's & Dementia: The journal of the Alzheimer's Association, 19(7), 3171–3185. 10.1002/alz.13027. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Anderson, E. D., Wahoske, M., Huber, M., Norton, D., Li, Z., Koscik, R. L., et al. (2016). Cognitive variability—A marker for incident MCI and AD: An analysis for the Alzheimer's Disease Neuroimaging Initiative. Alzheimers Dement (Amst), 4(1), 47–55. 10.1016/j.dadm.2016.05.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Arce Rentería, M., Mobley, T. M., Evangelista, N. D., Medina, L. D., Deters, K. D., Fox-Fuller, J. T., et al. (2023). Representativeness of samples enrolled in Alzheimer's disease research centers. Alzheimers Dement (Amst), 15(2), e12450. 10.1002/dad2.12450. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Bangen, K. J., Weigand, A. J., Thomas, K. R., Delano-Wood, L., Clark, L. R., Eppig, J., et al. (2019). Cognitive dispersion is a sensitive marker for early neurodegenerative changes and functional decline in nondemented older adults. Neuropsychology, 33(5), 599–608. 10.1037/neu0000532. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Beekly, D. L., Ramos, E. M., Van Belle, G., Deitrich, W., Clark, A. D., Jacka, M. E., et al. (2004). The national Alzheimer's coordinating center (NACC) database: an Alzheimer disease database. Alzheimer Disease & Associated Disorders, 18(4), 270–277. [PubMed] [Google Scholar]
  6. Benge, J. F., Artz, J. D., & Kiselica, A. M. (2020). The ecological validity of the Uniform Data Set 3.0 neuropsychological battery in individuals with mild cognitive impairment and dementia. The Clinical Neuropsychologist, 36(6), 1453–1470. 10.1080/13854046.2020.1837246. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Besser, L., Kukull, W., Knopman, D. S., Chui, H., Galasko, D., Weintraub, S., et al. (2018). Version 3 of the National Alzheimer’s Coordinating Center’s Uniform Data Set. Alzheimer Disease and Associated Disorders, 32(4), 351–358. 10.1097/WAD.0000000000000279. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Bielak, A. A., Cherbuin, N., Bunce, D., & Anstey, K. J. (2014). Intraindividual variability is a fundamental phenomenon of aging: evidence from an 8-year longitudinal study across young, middle, and older adulthood. Developmental Psychology, 50(1), 143–151. 10.1037/a0032650. [DOI] [PubMed] [Google Scholar]
  9. Binder, L. M., Iverson, G. L., & Brooks, B. L. (2009). To err is human: “Abnormal” neuropsychological scores and variability are common in healthy adults. Archives of Clinical Neuropsychology, 24(1), 31–46. 10.1093/arclin/acn001. [DOI] [PubMed] [Google Scholar]
  10. Burton, C. L., Strauss, E., Hultsch, D. F., Moll, A., & Hunter, M. A. (2006). Intraindividual variability as a marker of neurological dysfunction: a comparison of Alzheimer's disease and Parkinson's disease. Journal of Clinical and Experimental Neuropsychology, 28(1), 67–83. 10.1080/13803390490918318. [DOI] [PubMed] [Google Scholar]
  11. Christensen, H., Dear, K. B., Anstey, K. J., Parslow, R. A., Sachdev, P., & Jorm, A. F. (2005). Within-occasion intraindividual variability and preclinical diagnostic status: is intraindividual variability an indicator of mild cognitive impairment? Neuropsychology, 19(3), 309–317. 10.1037/0894-4105.19.3.309. [DOI] [PubMed] [Google Scholar]
  12. Christensen, H., Mackinnon, A. J., Korten, A. E., Jorm, A. F., Henderson, A. S., & Jacomb, P. (1999). Dispersion in cognitive ability as a function of age: A longitudinal study of an elderly community sample. Aging, Neuropsychology, and Cognition, 6(3), 214–228. 10.1076/anec.6.3.214.779. [DOI] [Google Scholar]
  13. Costa, A. S., Dogan, I., Schulz, J. B., & Reetz, K. (2019). Going beyond the mean: Intraindividual variability of cognitive performance in prodromal and early neurodegenerative disorders. The Clinical Neuropsychologist, 33(2), 369–389. 10.1080/13854046.2018.1533587. [DOI] [PubMed] [Google Scholar]
  14. Craft, S., Newcomer, J., Kanne, S., Dagogo-Jack, S., Cryer, P., Sheline, Y., et al. (1996). Memory improvement following induced hyperinsulinemia in Alzheimer's disease. Neurobiology of Aging, 17(1), 123–130. 10.1016/0197-4580(95)02002-0. [DOI] [PubMed] [Google Scholar]
  15. De Felice, S., & Holland, C. A. (2018). Intra-individual variability across fluid cognition can reveal qualitatively different cognitive styles of the aging brain [original research]. Frontiers in Psychology, 9, 1973. 10.3389/fpsyg.2018.01973. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Devora, P. V., Beevers, S., Kiselica, A. M., & Benge, J. F. (2019). Normative data for derived measures and discrepancy scores for the Uniform Data Set 3.0 Neuropsychological Battery. Archives of Clinical Neuropsychology, 35(1), 75–89. 10.1093/arclin/acz025. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Erdodi, L. A., & Lichtenstein, J. D. (2017). Invalid before impaired: an emerging paradox of embedded validity indicators. The Clinical Neuropsychologist, 31(6–7), 1029–1046. 10.1080/13854046.2017.1323119. [DOI] [PubMed] [Google Scholar]
  18. Fillenbaum, G., Peterson, B., & Morris, J. (1996). Estimating the validity of the Clinical Dementia Rating scale: the CERAD experience. Aging Clinical and Experimental Research, 8(6), 379–385. 10.1007/BF03339599. [DOI] [PubMed] [Google Scholar]
  19. Gleason, C. E., Norton, D., Anderson, E. D., Wahoske, M., Washington, D. T., Umucu, E., et al. (2018). Cognitive variability predicts incident Alzheimer's disease and mild cognitive impairment comparable to a cerebrospinal fluid biomarker. Journal of Alzheimer's Disease, 61(1), 79–89. 10.3233/jad-170498. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Gollan, T. H., Weissberger, G. H., Runnqvist, E., Montoya, R. I., & Cera, C. M. (2012). Self-ratings of spoken language dominance: A Multilingual Naming Test (MINT) and preliminary norms for young and aging Spanish–English bilinguals. Bilingualism: Language and Cognition, 15(3), 594–615. 10.1017/S1366728911000332. [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Heaton, R. K., Taylor, M. J., & Manly, J. (2003). Demographic effects and use of demographically corrected norms with the WAIS-III and WMS-III. In Clinical interpretation of the WAIS-III and WMS-III, (pp. 181–210). Cambridge, MA: Academic Press. [Google Scholar]
  22. Heinze, G. (2022). mfp: Multivariable Fractional Polynomials. https://cran.r-project.org/web/packages/mfp/index.html
  23. Heyanka, D. J., Holster, J. L., & Golden, C. J. (2013). Intraindividual neuropsychological test variability in healthy individuals with high average intelligence and educational attainment. The International Journal of Neuroscience, 123(8), 526–531. 10.3109/00207454.2013.771261. [DOI] [PubMed] [Google Scholar]
  24. Hilborn, J. V., Strauss, E., Hultsch, D. F., & Hunter, M. A. (2009). Intraindividual variability across cognitive domains: investigation of dispersion levels and performance profiles in older adults. Journal of Clinical and Experimental Neuropsychology, 31(4), 412–424. 10.1080/13803390802232659. [DOI] [PubMed] [Google Scholar]
  25. Hill, B. D., Rohling, M. L., Boettcher, A. C., & Meyers, J. E. (2013). Cognitive intra-individual variability has a positive association with traumatic brain injury severity and suboptimal effort. Archives of Clinical Neuropsychology, 28(7), 640–648. 10.1093/arclin/act045. [DOI] [PubMed] [Google Scholar]
  26. Holtzer, R., Verghese, J., Wang, C., Hall, C. B., & Lipton, R. B. (2008). Within-person across-neuropsychological test variability and incident dementia. JAMA, 300(7), 823–830. 10.1001/jama.300.7.823. [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Hosmer, D. W., Lemeshow, S., & Sturdivant, R. X. (2013). Applied logistic regression, (Vol. 398). Hoboken, NJ: John Wiley & Sons. [Google Scholar]
  28. Hromas, G., Rolin, S., & Davis, J. J. (2022). Racial differences in positive findings on embedded performance validity tests. Applied Neuropsychology: Adult, 1–9. 10.1080/23279095.2022.2146504. [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Hultsch, D. F., MacDonald, S. W., Hunter, M. A., Levy-Bencheton, J., & Strauss, E. (2000). Intraindividual variability in cognitive performance in older adults: comparison of adults with mild dementia, adults with arthritis, and healthy adults. Neuropsychology, 14(4), 588–598. 10.1037/0894-4105.14.4.588. [DOI] [PubMed] [Google Scholar]
  30. Ivanova, I., Salmon, D. P., & Gollan, T. H. (2013). The multilingual naming test in Alzheimer's disease: clues to the origin of naming impairments. Journal of the International Neuropsychological Society, 19(3), 272–283. 10.1017/S1355617712001282. [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Jeffay, E., Binder, L. M., & Zakzanis, K. K. (2021). Marked intraindividual cognitive variability in a sample of healthy graduate students. Psychological Injury and Law, 14(3), 171–183. 10.1007/s12207-021-09417-x. [DOI] [Google Scholar]
  32. Kaplan, E. (1988). A process approach to neuropsychological assessment. In Clinical neuropsychology and brain function: Research, measurement, and practice, (pp. 127–167). Washington, D.C.: American Psychological Association. [Google Scholar]
  33. Khan, M. R. A. A. (2019). Rocit-an r package for performance assessment of binary classifier with visualization. https://cran.r-project.org/web/packages/ROCit/index.html.
  34. Kiselica, A. M., Johnson, E., Lewis, K. R., & Trout, K. (2021). Examining racial disparities in the diagnosis of mild cognitive impairment. Applied Neuropsychology. Adult, 30(6), 749–756. 10.1080/23279095.2021.1976778. [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Kiselica, A. M., Karr, J. E., Mikula, C. M., Ranum, R. M., Benge, J. F., Medina, L. D., et al. (2023). Recent advances in neuropsychological test interpretation for clinical practice. Neuropsychology Review. 10.1007/s11065-023-09596-1. [DOI] [PubMed] [Google Scholar]
  36. Kiselica, A. M., Kaser, A., Webber, T. A., Small, B. J., & Benge, J. F. (2020). Development and preliminary validation of standardized regression-based change scores as measures of transitional cognitive decline. Archives of Clinical Neuropsychology, 35(7), 1168–1181. 10.1093/arclin/acaa042. [DOI] [PMC free article] [PubMed] [Google Scholar] [Retracted]
  37. Kiselica, A. M., Webber, T. A., & Benge, J. F. (2020a). The Uniform Data Set 3.0 Neuropsychological Battery: Factor structure, invariance testing, and demographically adjusted factor score calculation. Journal of the International Neuropsychological Society., 26(6), 576–586. 10.1017/S135561772000003X. [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Kiselica, A. M., Webber, T. A., & Benge, J. F. (2020b). Using multivariate base rates of low scores to understand early cognitive declines on the uniform data set 3.0 Neuropsychological Battery. Neuropsychology, 34(6), 629–640. 10.1037/neu0000640. [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Lezak, M., Howieson, D., & Loring, D. (2012). Neuropsychological assessment (5th ed.). Oxford, UK: Oxford University Press. [Google Scholar]
  40. Manly, J. J. (2005). Advantages and disadvantages of separate norms for African Americans. The Clinical Neuropsychologist, 19(2), 270–275. 10.1080/13854040590945346. [DOI] [PubMed] [Google Scholar]
  41. Marquine, M. J., Parks, A., Perales-Puchalt, J., González, D. A., Rosado-Bruno, M., North, R., et al. (2023). Demographically adjusted normative data among Latinos for the version 3 of the Alzheimer's Disease Centers' Neuropsychological Test Battery in the Uniform Data Set. Alzheimers Dement, 19(9), 4174–4186. 10.1002/alz.13313. [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. McKeith, I. G., Boeve, B. F., Dickson, D. W., Halliday, G., Taylor, J.-P., Weintraub, D., et al. (2017). Diagnosis and management of dementia with Lewy bodies: Fourth consensus report of the DLB Consortium. Neurology, 89(1), 88–100. 10.1212/WNL.0000000000004058. [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Merritt, V. C., Clark, A. L., Crocker, L. D., Sorg, S. F., Werhane, M. L., Bondi, M. W., et al. (2018). Repetitive mild traumatic brain injury in military veterans is associated with increased neuropsychological intra-individual variability. Neuropsychologia, 119, 340–348. 10.1016/j.neuropsychologia.2018.08.026. [DOI] [PubMed] [Google Scholar]
  44. Morgan, E. E., Woods, S. P., Delano-Wood, L., Bondi, M. W., Grant, I., & The HIV Neurobehavioral Research Program (HNRP) Group (2011). Intraindividual variability in HIV infection: evidence for greater neurocognitive dispersion in older HIV seropositive adults. Neuropsychology, 25(5), 645–654. 10.1037/a0023792. [DOI] [PMC free article] [PubMed] [Google Scholar]
  45. Morgan, E. E., Woods, S. P., Rooney, A., Perry, W., Grant, I., Letendre, S. L., et al. (2012). Intra-individual variability across neurocognitive domains in chronic hepatitis C infection: elevated dispersion is associated with serostatus and unemployment risk. The Clinical Neuropsychologist, 26(4), 654–674. 10.1080/13854046.2012.680912. [DOI] [PMC free article] [PubMed] [Google Scholar]
  46. Morris, J. C. (1993). The Clinical Dementia Rating (CDR): Current version and scoring rules. Neurology, 43(11), 2412. 10.1212/WNL.43.11.2412-a. [DOI] [PubMed] [Google Scholar]
  47. Morris, J. C. (1997). Clinical dementia rating: a reliable and valid diagnostic and staging measure for dementia of the Alzheimer type. International Psychogeriatrics, 9(S1), 173–176. 10.1017/S1041610297004870. [DOI] [PubMed] [Google Scholar]
  48. Morris, J. C., Ernesto, C., Schafer, K., Coats, M., Leon, S., Sano, M., et al. (1997). Clinical dementia rating training and reliability in multicenter studies: the Alzheimer's Disease Cooperative Study experience. Neurology, 48(6), 1508–1510. 10.1212/WNL.48.6.1508. [DOI] [PubMed] [Google Scholar]
  49. Morris, J. C., Weintraub, S., Chui, H. C., Cummings, J., DeCarli, C., Ferris, S., et al. (2006). The Uniform Data Set (UDS): Clinical and cognitive variables and descriptive data from Alzheimer Disease Centers. Alzheimer Disease & Associated Disorders, 20(4), 210–216. 10.1097/01.wad.0000213865.09806.92. [DOI] [PubMed] [Google Scholar]
  50. Mustafa, A. I., Woods, S. P., Loft, S., & Morgan, E. E. (2023). Lower prospective memory is associated with higher neurocognitive dispersion in two samples of people with HIV: A conceptual replication study. Journal of the International Neuropsychological Society, 29(7), 677–685. 10.1017/S1355617722000698. [DOI] [PMC free article] [PubMed] [Google Scholar]
  51. Partington, J. E., & Leiter, R. G. (1949). Partington pathways test. Psychological Service Center Journal, 1, 11–20. [Google Scholar]
  52. Possin, K. L., Laluz, V. R., Alcantar, O. Z., Miller, B. L., & Kramer, J. H. (2011). Distinct neuroanatomical substrates and cognitive mechanisms of figure copy performance in Alzheimer's disease and behavioral variant frontotemporal dementia. Neuropsychologia, 49(1), 43–48. 10.1016/j.neuropsychologia.2010.10.026. [DOI] [PMC free article] [PubMed] [Google Scholar]
  53. Possin, K. L., Tsoy, E., & Windon, C. C. (2021). Perils of race-based norms in cognitive testing: The case of former NFL players. JAMA Neurology, 78(4), 377–378. 10.1001/jamaneurol.2020.4763. [DOI] [PMC free article] [PubMed] [Google Scholar]
  54. Rapp, M. A., Schnaider-Beeri, M., Sano, M., Silverman, J. M., & Haroutunian, V. (2005). Cross-domain variability of cognitive performance in very old nursing home residents and community dwellers: relationship to functional status. Gerontology, 51(3), 206–212. 10.1159/000083995. [DOI] [PubMed] [Google Scholar]
  55. Roalf, D. R., Quarmley, M., Mechanic-Hamilton, D., Wolk, D. A., Arnold, S. E., & Moberg, P. J. (2016). Within-individual variability: An index for subtle change in neurocognition in mild cognitive impairment. Journal of Alzheimer's Disease, 54(1), 325–335. 10.3233/jad-160259. [DOI] [PMC free article] [PubMed] [Google Scholar]
  56. Royston, P., & Sauerbrei, W. (2008). Multivariable model —Building: A pragmatic approach to regression anaylsis based on fractional polynomials for modelling continuous variables. Hoboken, NJ: Wiley. [Google Scholar]
  57. Sachs, B. C., Steenland, K., Zhao, L., Hughes, T. M., Weintraub, S., Dodge, H. H., et al. (2020). Expanded demographic norms for Version 3 of the Alzheimer Disease Centers’ Neuropsychological Test Battery in the Uniform Data Set. Alzheimer Disease & Associated Disorders, 34(3), 191–197. 10.1097/WAD.0000000000000388. [DOI] [PMC free article] [PubMed] [Google Scholar]
  58. Sakamoto, M. S., Delano-Wood, L., Sorg, S. F., Schiehser, D. M., & Merritt, V. C. (2021). Unemployment status is associated with greater cognitive intraindividual variability in veterans with a history of remote mild TBI. Journal of Head Trauma Rehabilitation, 36(6), E391–e396. 10.1097/htr.0000000000000693. [DOI] [PubMed] [Google Scholar]
  59. Schretlen, D. J., Munro, C. A., Anthony, J. C., & Pearlson, G. D. (2003). Examining the range of normal intraindividual variability in neuropsychological test performance. Journal of the International Neuropsychological Society, 9(6), 864–870. 10.1017/s1355617703960061. [DOI] [PubMed] [Google Scholar]
  60. Sherman, E., Slick, D. J., & Iverson, G. L. (2020). Multidimensional malingering criteria for neuropsychological assessment: A 20-year update of the malingered neuropsychological dysfunction criteria. Archives of Clinical Neuropsychology, 35(6), 735–764. 10.1093/arclin/acaa019. [DOI] [PMC free article] [PubMed] [Google Scholar]
  61. Shirk, S. D., Mitchell, M. B., Shaughnessy, L. W., Sherman, J. C., Locascio, J. J., Weintraub, S., et al. (2011). A web-based normative calculator for the Uniform Dataset (UDS) neuropsychological test battery. Alzheimer's Research & Therapy, 3(6), 32. 10.1186/alzrt94. [DOI] [PMC free article] [PubMed] [Google Scholar]
  62. Stawski, R. S., MacDonald, S. W. S., Brewster, P. W. H., Munoz, E., Cerino, E. S., & Halliday, D. W. R. (2017). A comprehensive comparison of quantifications of intraindividual variability in response times: A measurement burst approach. The Journals of Gerontology: Series B, 74(3), 397–408. 10.1093/geronb/gbx115. [DOI] [PMC free article] [PubMed] [Google Scholar]
  63. Stawski, R. S., MacDonald, S. W. S., Brewster, P. W. H., Munoz, E., Cerino, E. S., & Halliday, D. W. R. (2019). A comprehensive comparison of quantifications of intraindividual variability in response times: a measurement burst approach. The Journals of Gerontology. Series B, Psychological Sciences and Social Sciences, 74(3), 397–408. 10.1093/geronb/gbx115. [DOI] [PMC free article] [PubMed] [Google Scholar]
  64. Strauss, E., MacDonald, S. W., Hunter, M., Moll, A., & Hultsch, D. F. (2002). Intraindividual variability in cognitive performance in three groups of older adults: cross-domain links to physical status and self-perceived affect and beliefs. Journal of the International Neuropsychological Society, 8(7), 893–906. 10.1017/s1355617702870035. [DOI] [PubMed] [Google Scholar]
  65. Stuss, D. T., Murphy, K. J., Binns, M. A., & Alexander, M. P. (2003). Staying on the job: the frontal lobes control individual performance variability. Brain, 126(11), 2363–2380. 10.1093/brain/awg237. [DOI] [PubMed] [Google Scholar]
  66. Suchy, Y., Ziemnik, R. E., Niermeyer, M. A., & Brothers, S. L. (2020). Executive functioning interacts with complexity of daily life in predicting daily medication management among older adults. The Clinical Neuropsychologist, 34(4), 797–825. 10.1080/13854046.2019.1694702. [DOI] [PubMed] [Google Scholar]
  67. Sullivan, K. L., Woods, S. P., Bucks, R. S., Loft, S., & Weinborn, M. (2018). Intraindividual variability in neurocognitive performance is associated with time-based prospective memory in older adults. Journal of Clinical and Experimental Neuropsychology, 40(7), 733–743. 10.1080/13803395.2018.1432571. [DOI] [PMC free article] [PubMed] [Google Scholar]
  68. Thiele, C., & Hirschfeld, G. (2021). cutpointr: Improved estimation and validation of optimal cutpoints in R. Journal of Statistical Software, 98(11), 1–27. 10.18637/jss.v098.i11. [DOI] [Google Scholar]
  69. Tractenberg, R. E., & Pietrzak, R. H. (2011). Intra-individual variability in Alzheimer's disease and cognitive aging: definitions, context, and effect sizes. PLoS One, 6(4), e16973. 10.1371/journal.pone.0016973. [DOI] [PMC free article] [PubMed] [Google Scholar]
  70. Umlauf, A., & Umlauf, M. A. (2022). Package ‘test2norm’. https://cran.r-project.org/web/packages/test2norm/index.html.
  71. Von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., & Vandenbroucke, J. P. (2007). The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. The Lancet, 370(9596), 1453–1457. 10.1016/S0140-6736(07)61602-X. [DOI] [PubMed] [Google Scholar]
  72. Wang, C., Katz, M. J., Chang, K. H., Qin, J., Lipton, R. B., Zwerling, J. L., et al. (2021). UDSNB 3.0 neuropsychological test norms in older adults from a diverse community: Results from the Einstein Aging Study (EAS). Journal of Alzheimer's Disease, 83(4), 1665–1678. 10.3233/JAD-210538. [DOI] [PMC free article] [PubMed] [Google Scholar]
  73. Webber, T., Kiselica, A. M., Mikula, C., & Woods, S. P. (2022). Dispersion-based cognitive intra-individual variability in dementia with Lewy bodies. Neuropsychology, 36(8), 719–729. 10.1037/neu0000856. [DOI] [PMC free article] [PubMed] [Google Scholar]
  74. Webber, T., Lorkiewicz, S., Kiselica, A. M., & Woods, S. (2023). Ecological validity of cognitive fluctuations in Lewy Body Disease. Journal of the International Neuropsychological Society, 30(1), 35–46. 10.1017/S1355617723000255. [DOI] [PMC free article] [PubMed] [Google Scholar]
  75. Weintraub, S., Besser, L., Dodge, H. H., Teylan, M., Ferris, S., Goldstein, F. C., et al. (2018). Version 3 of the Alzheimer Disease Centers’ Neuropsychological Test Battery in the Uniform Data Set (UDS). Alzheimer Disease and Associated Disorders, 32(1), 10–17. 10.1097/WAD.0000000000000223. [DOI] [PMC free article] [PubMed] [Google Scholar]
  76. West, R., Murphy, K. J., Armilio, M. L., Craik, F. I., & Stuss, D. T. (2002). Lapses of intention and performance variability reveal age-related increases in fluctuations of executive control. Brain and Cognition, 49(3), 402–419. 10.1006/brcg.2001.1507. [DOI] [PubMed] [Google Scholar]
  77. Wilson, R. S., Yu, L., Lamar, M., Schneider, J. A., Boyle, P. A., & Bennett, D. A. (2019). Education and cognitive reserve in old age. Neurology, 92(10), e1041–e1050. 10.1212/WNL.0000000000007036. [DOI] [PMC free article] [PubMed] [Google Scholar]
  78. Youden, W. J. (1950). Index for rating diagnostic tests. Cancer, 3(1), 32–35. 10.1002/1097-0142(1950)3:132::AID-CNCR28200301063.0.CO;2-3. [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

UDS_3_0_IIV_Normed_Score_Calculator_FINAL_CORRECTED_acae005
uds_3_0_iiv_normed_score_calculator_final_corrected_acae005.xlsx (18.5KB, xlsx)

Articles from Archives of Clinical Neuropsychology are provided here courtesy of Oxford University Press

RESOURCES