Quantification of five neuropsychological approaches to defining mild cognitive impairment

Abstract

Objectives

Operational definitions of cognitive impairment have varied widely in diagnosing mild cognitive impairment (MCI). Identifying clinical subtypes of MCI has further challenged diagnostic approaches, since varying the components of the objective cognitive assessment can significantly impact diagnosis. Therefore, we investigated the applicability of diagnostic criteria for clinical subtypes of MCI in a naturalistic research sample of community elders and quantified the variability in diagnostic outcomes that results from modifying the neuropsychological definition of objective cognitive impairment.

Design

Cross-sectional and longitudinal study

Setting

San Diego, CA, Veterans Administration Hospital

Participants

90 nondemented, neurologically normal, community-dwelling older adults were initially assessed and 73 were seen for follow-up approximately 17 months later.

Measurements

Participants were classified via consensus diagnosis as either normally aging or having MCI via each of five diagnostic strategies, which varied the cutoff for objective impairment as well as the number of neuropsychological tests considered in the diagnostic process.

Results

A range of differences in the percentages identified as MCI versus cognitively normal were demonstrated, depending on the classification criteria employed. A substantial minority of individuals demonstrated diagnostic instability over time as well as across diagnostic approaches. The single domain non-amnestic subtype diagnosis was particularly unstable (e.g., prone to reclassification as normal at follow up).

Conclusion

Our findings provide empirical support for a neuropsychologically derived operational definition of clinical subtypes of MCI and point to the importance of using comprehensive neuropsychological assessments. Diagnoses, particularly involving non-amnestic MCI, were variable over time. The applicability and utility of this particular MCI subtype warrants further investigation.

1. Objective

Because of its potential importance for early identification and intervention in those at risk for the development of dementia, the concept of mild cognitive impairment (MCI) has received considerable research attention. However, the definition itself has changed considerably over time. Historically, MCI was characterized primarily as an amnestic disorder representing an intermediate stage between normal aging and Alzheimer's dementia.¹ More recently, broader conceptualizations of MCI have emerged delineating clinical subtypes that commonly include amnestic and non-amnestic forms, and single and multiple cognitive domains. ^2-5 With the advent of these broader classification schemes, diagnostic challenges related to MCI have also increased.

Multiple factors can vary in the operational definition of objective cognitive impairment. Performances falling 1 standard deviation (SD), ⁶ 1.5 SD, ⁷ and 1.96 SD ⁸ have all been suggested as cutoffs demarcating impairment. Others have suggested different cutoffs for different MCI subtypes ⁹ or advocated for the use of clinical judgment of memory impairment informed by psychometrics. ^10-12 While most studies of MCI have used verbal recall measures to document objective memory impairment, ⁷ often further limited to paragraph recall, ¹³ a sizeable minority of amnestic MCI (aMCI) cases are likely to be missed if visual memory is not examined. ¹⁴ Failing to fully examine cognitive domains other than memory has also likely led to improper characterization of so-called aMCI samples. ¹⁵ Isolated amnestic presentation in many early studies of MCI was not always clear or accurate given that memory was the only cognitive domain formally assessed. Other studies have employed larger neuropsychological test batteries that formally assessed multiple cognitive domains, but still lacked some rigor because they defined impairment as a deficit (1.5 SD) on only one test in a particular domain. ^{5, 16}

Demonstration of objective neuropsychological impairment in the classification of MCI is perhaps the most variable and ill-defined component of the diagnosis. ¹⁷ The lack of a universally accepted approach to the objective identification of cognitive impairment and a wide-range of conceptual and diagnostic approaches to MCI have led to highly variable prevalence rates from 1% to 30%, and annual rates of conversion from MCI to dementia that vary drastically from 1% to 72%. ⁷ This has also resulted in variable estimates of the stability of the MCI diagnosis, ¹⁸ with rates of those initially diagnosed with MCI reverting back to normalcy on reassessment ranging from 10-40%, ^{8, 19} with evidence that single domain subtypes are less stable over time than multi-domain subtypes. ⁸

Many researchers studying MCI have highlighted the problems associated with the use of different operational criteria across studies, but empirical evaluation of the impact of these criteria-related differences on diagnosing subtypes of MCI is scarce. Such an evaluation is critically important since the accuracy of classification of MCI subtypes can influence clinical and research outcomes, particularly with burgeoning evidence suggesting subtypes differ in diagnostic outcome and likelihood of conversion to AD. ^{6, 20}

In light of these concerns, we sought to better characterize how different diagnostic approaches, based on neuropsychological methods widely employed in the extant MCI literature, impact classification of individuals as normal or MCI. The consequence of altering variables such as the cut point at which performance is considered impaired, or the number of tests on which one needs to demonstrate impaired performance, was of primary interest. We predicted that (a) varying statistically based cutoff points would impact internal consistency of the MCI diagnosis, and that (b) requiring impairment on more than one test within a cognitive domain would increase the stability of the MCI diagnosis.

2. Methods

2.1 Participants

Participants were community dwelling volunteers selected from a larger group of individuals enrolled in a longitudinal study of normal aging. Ninety nondemented older adults, ages 62-92 (mean age = 77) were initially assessed (Time 1) and 73 were seen for follow-up (Time 2). Based on year 1 data (N=90), participants were determined to be non-demented based on consensus diagnosis utilizing all available neurological, neuropsychological, and functional data. All participants were free of functional impairment (Independent Living Scales [ILS]²¹ T-scores > 40), and exhibited normal global cognitive functioning at Time 1 and, if follow-up was available, at Time 2 (Dementia Rating Scale²² [DRS] ≥ 127, mean = 140) (See table 1). This cutoff was selected statistically to ensure that no participant scored more than 1.5 SD below the published normative mean on the DRS, ²² consistent with cutoffs for impairment in other cognitive domains as described below. Those with a history of alcoholism, drug abuse, learning disability, neurological, or psychiatric illness were excluded. The Geriatric Depression Scale (GDS)²³ identified a range of depressive symptomatology in the sample that varied depending on the diagnostic strategy applied. Applying the comprehensive criteria revealed that the MCI group was significantly more depressed than the cognitively normal group (t=2.2, p= .03), though the mean GDS score was 7, well below the suggested clinical cutoff of 11 for depression. ²³ Conversely, the normal control group had higher levels of depression than the MCI group when the historical approach was applied (t=2.1, p=.05). No significant differences in levels of depressive symptomatology were noted when any of the other diagnostic strategies were applied (all p-values > .08).

Table 1.

Demographics of Baseline Sample (N=90)

	Range	Mean	SD
Age	62-92	77	7.4
Gender (M/F)	42/48	-	-
Education	12-20	16	2.4
ANART VIQ	102-128	120	5.8
DRS Total	127-144	140	3.8
ILS Health and Safety T-score	42-63	54.7	6.3
ILS Managing Money T-score	41-61	55.6	4.8
Geriatric Depression Scale	0-21	5.7	5.0

On average, duration between assessments was 17 months (SD = 7.0 months). There were no significant differences between those with follow-up data and those with no follow-up on age, years of education, level of depressive symptoms, daily functioning, or global cognitive functioning (all p-values >.10). Proportions of the sample characterized as MCI did not differ for any of the diagnostic approaches (all p-values > .07), except the most conservative criteria, in which 35% of the group with no follow up was initially identified as MCI versus only 12% of those with follow up [χ² (1, N=90) = 5.24, p=.02]. Of the 17 individuals who were not available for follow up, nine withdrew, and the remaining eight were lost to follow up (moved, non-working phone numbers, etc.).

2.2 Assessment

All participants underwent a comprehensive neuropsychological assessment. The tests of interest included measures from five cognitive domains (memory, attention, language, visuospatial functioning, and executive functioning), with at least three measures from each domain (see table 2).

Table 2.

Neuropsychological Tests Administered

Memory	Attention	Language	Visuospatial Functioning	Executive Functioning
WMS-R Logical Memory (immediate, delayed free recall)	DRS attention	BNT	WISC-R Block Design	WCST-48-card version (categories achieved and perseverative errors)
WMS-R Visual Reproduction (immediate, delayed free recall)	WAIS-R Digit Span	Letter fluency	D-KEFS Visual Scanning	TMT, Part B
CVLT Trials 1-5 total, long delay free recall	TMT Part A	Category fluency	D-KEFS Design Fluency (empty and filled dot conditions)	D-KEFS Color-Word Interference Test (inhibition and inhibition/switching)
			DRS construction	D-KEFS fluency switching (visual and verbal)
			Clock drawing

Wechsler Memory Scale - Revised²⁷ (WMS-R), California Verbal Learning Test²⁸ (CVLT), Wechsler Adult Intelligence Scale-Revised ²⁹ (WAIS-R), Trail Making Test ³⁰ (TMT), Part A, Boston Naming Test ³¹ (BNT), Wechsler Intelligence Scale for Children-Revised ³² (WISC-R), Delis-Kaplan Executive Function System³³ (D-KEFS), modified Wisconsin Card Sorting Test³⁴ (WCST-48-card version), Trail Making Test ³⁰ (TMT), Part B,

Normative data was drawn from Mayo's Older Americans Normative Studies (MOANS)³⁵ or from other published norms ²⁷-²⁹, ³¹-³⁴, ³⁶-³⁸ except for block design, which used age and education adjusted norms drawn from local unpublished data derived from the UCSD Alzheimer Disease Research Center.

2.3 MCI Classification

Each participant was classified as normal or MCI on the basis of five sets of neuropsychologically based criteria for MCI that differed in their characterization of objective cognitive impairment. For all criteria, participants were labeled as Single Domain Amnestic MCI if only the memory domain was impaired, as Single Domain Non-Amnestic MCI if only one non-memory domain was impaired, as Multiple Domain Amnestic MCI if memory and at least one other domain showed impairment, and as Multiple Domain Non-Amnestic MCI if more than one non-memory domain was impaired. Variations on cutoffs for impairment and number of tests required to be in the impaired range are delineated below.

Historical criteria

The “historical criteria” operationalized the original Petersen criteria¹. Specifically, individuals were identified with MCI if objective memory performance fell more than 1.5 SD below their age appropriate norms on the WMS-R LM subtest. This was an entirely dichotomous designation, classifying individuals as either “normal” or “amnestic MCI,” and mirrored the dominant method by which objective memory impairment was defined in the early MCI literature as well as in more recent large clinical trial studies. ³

Typical criteria

The “typical criteria,” adapted from the most recent criteria outlined by Petersen and Morris (2005), ³ operationally-defined objective cognitive impairment for multiple subtypes of MCI. Individuals were classified as normal if no neuropsychological measure fell greater than 1.5 SD below age-appropriate norms in any cognitive domain. Impairment required scores to fall more than 1.5 SD below age appropriate norms on only one test within a domain.

Comprehensive criteria

A “comprehensive criteria” approach was developed for identification of multiple subtypes of MCI, and required that at least two performances within a cognitive domain fall below the established cutoff in order for that domain to contribute to the MCI classification. In order to strike a balance between reliability and sensitivity to detect mild impairment, a lower cutoff for impairment of 1 SD below normative data was adopted ^{11, 12}. Individuals were classified as normal if, at most, performance on one measure within one or two cognitive domains fell more than 1 SD below age-appropriate norms.

Liberal criteria

To better cover the range of possible diagnostic approaches, a “liberal criteria” was developed that required impairment only on one test within a domain, with impairment identified at the level of 1 SD below normative data. When applying these criteria, individuals were classified as normal if no neuropsychological measure fell greater than 1 SD below age-appropriate norms in any cognitive domain.

Conservative criteria

Finally, to cover the other end of the spectrum of diagnostic approaches, a “conservative criteria” was developed that required impairment on two measures within a domain, with impairment identified as 1.5 SD below normative data. Individuals were classified as normal if, at most, performance on one measure within one or two cognitive domains fell greater than 1.5 SD below age-appropriate norms.

3. Results

At Time 1, 10-74% of participants were characterized as having MCI, depending on the diagnostic strategy applied. Subtype classification rates were similarly variable. Among those identified as having MCI, single domain amnestic MCI was the resulting subtype in 8.9% (comprehensive criteria) to 12.2% (conservative criteria) of cases. Single domain non-amnestic MCI was the clinical subtype in 2.2% (conservative) to 31.1% (liberal) of those with MCI. Multidomain amnestic MCI was identified in 1.1% (conservative) to 23.3% (liberal) of those with MCI. Finally, the multidomain non-amnestic subtype was identified in 0% (conservative) to 10% (liberal) of those with MCI.

An analysis of agreement using the Kappa statistic was performed to determine consistency of diagnosis between different diagnostic criteria on the full sample at Time One (N=90). Examination of the correspondence in classifications provided by the comprehensive and typical criteria showed that 74% of individuals were given the same dichotomous classification (either normal or MCI) by both procedures (Kappa=.49, p <.0001). Of the remaining 26% of the sample on which there was not agreement, 20% of subjects were categorized as normal by the comprehensive criteria but as MCI by the typical criteria. In addition, 11% of those categorized globally as MCI by both procedures were categorized with different MCI subtypes depending on the criteria used.

Comparison of the comprehensive and historical criteria showed that 76% of individuals received the same dichotomous classification (either normal or MCI) by both procedures (Kappa=.35, p <.0001). The 24% of individuals in whom the classifications did not correspond were all classified as normal by the historical criteria and MCI by the comprehensive criteria. Forty-one percent of these latter individuals were categorized as amnestic (single or multiple domain) MCI by the comprehensive criteria, suggesting that the historical criteria were not simply missing non-amnestic presentations. In fact, for all these individuals, impairment on memory tests other than Logical Memory led to the amnestic classification via the comprehensive criteria.

Comparison of the comprehensive and liberal criteria showed that 60% of individuals received the same dichotomous classification (either normal or MCI) by both procedures (Kappa=.31, p <.0001). The 40% of individuals in whom the classifications did not correspond were all classified as normal by the comprehensive criteria and MCI by the liberal criteria.

Comparison of the comprehensive and conservative criteria showed that 81% of individuals received the same dichotomous classification (either normal or MCI) by both procedures (Kappa=.52, p <.0001). The 19% of individuals in whom the classifications did not correspond were all classified as normal by the conservative criteria and MCI by the comprehensive criteria.

Comparison of the typical and historical criteria showed 61% correspondence between the dichotomous classifications (normal or MCI) provided by the two procedures (Kappa=.21, p =.001). The remaining 39% of individuals were categorized as MCI by the typical criteria but normal by the historical criteria. Of these latter individuals, 30% were identified as single or multiple domain amnestic MCI by the typical criteria, suggesting that the historical criteria were missing amnestic subjects identified by memory tests other than paragraph recall

Comparison of the typical and liberal criteria showed 74% correspondence between the dichotomous classifications (normal or MCI) provided by the two procedures (Kappa=.49, p >.0001). The remaining 26% of individuals were categorized as MCI by the liberal criteria but normal by the typical criteria.

Comparison of the typical and conservative criteria showed 67% correspondence between the dichotomous classifications (normal or MCI) provided by the two procedures (Kappa=.32, p >.0001). The remaining 33% of individuals were categorized as MCI by the typical criteria but normal by the conservative criteria.

Comparison of the historical and liberal criteria showed only 36% correspondence between the dichotomous classifications (normal or MCI) provided by the two procedures (Kappa=.07, p =.06). The remaining 64% of individuals were categorized as normal by the historical criteria but MCI by the liberal criteria.

Comparison of the historical and conservative criteria showed 94% correspondence between the dichotomous classifications (normal or MCI) provided by the two procedures (Kappa=.75, p >.0001). The remaining 6% of individuals were categorized as normal by the historical criteria but MCI by the conservative criteria.

Comparison of the liberal and conservative criteria showed only 41% correspondence between the dichotomous classifications (normal or MCI) provided by the two procedures (Kappa=.12, p =.02). The remaining 59% of individuals were categorized as MCI by the liberal criteria but normal by the conservative criteria.

The stability over time of the classification provided by each diagnostic approach was examined by comparing the classification they provided at Time 1 versus Time 2. When the comprehensive criteria were applied, 70% of subjects exhibited diagnostic stability. That is, at a strictly dichotomous level, 70% of individuals maintained either their normal or MCI status, irrespective of subtype, from Time 1 to Time 2. Fifteen percent declined (changed from normal to MCI) and 15% improved (reverted from MCI to normal). Of those who improved, 73% were initially classified as single domain non-amnestic MCI, 18% were single domain amnestic MCI, and only 9% were multi-domain (amnestic). Eight percent of those who were classified globally as MCI at both time points changed MCI subtype over time. Thus, 28 of 73 participants with follow-up data either changed category or MCI subtype (i.e., there was a 38% overall instability rate).

When the typical criteria were applied, the global classifications of 74% of the participants maintained either a normal or MCI diagnosis over time. 12% changed from normal to MCI, and 14% changed from MCI to normal (80% of these were initially classified with single domain non-amnestic MCI and the remaining 20% were single domain amnestic MCI). In addition, 18% of those who were diagnosed as MCI at both time points changed MCI subtype resulting in a 44% overall instability rate for the typical criteria.

When the historical criteria were applied, 98% of individuals maintained their dichotomous designation of either normal or MCI over time, and 2% declined from normal to MCI. No one reverted from MCI to normal when the historical criteria were applied.

When the liberal criteria were applied, the classifications of 81% of the participants maintained either a normal or MCI status (regardless of subtype) over time. Seven percent changed from normal to MCI, and 12% changed from MCI to normal (56% of whom were initially classified with single domain non-amnestic MCI, 11% as single domain amnestic MCI, 11% as multi-domain amnestic, and 22% as multiple domain non-amnestic MCI). However, 33% of those who were diagnosed as MCI at both time points changed MCI subtype resulting in a 52% overall instability rate for the typical criteria.

Finally, when the conservative criteria were utilized, 93% of participants were stable overtime (remaining either normal or MCI), 4% declined, and 3% improved, all of whom were initially characterized as single domain amnestic MCI. Only 1% of those diagnosed as MCI at both timepoints changed MCI subtype. This subtype change paired with both improvers and decliners resulted in an overall instability rate of only 8%.

4. Comment

Supporting the overall validity of the concept of MCI, the majority of diagnoses in the present study was consistent across diagnostic schemes and remained stable over time. However, a sizeable minority of diagnoses differed across diagnostic schemes and were unstable over the follow-up interval (when variability in diagnosis of MCI subtypes was included). The diagnoses resultant from application of the liberal criteria were particularly likely to deviate from other diagnostic strategies and to be unstable over time. These results are generally consistent with previous reports of the instability of the MCI diagnosis, ^{6, 19, 24} but expand on existing knowledge comparing stability over time of multiple diagnostic approaches, as opposed to those studies offering only a “one-test, 1.5 SD” perspective. The results also highlight the potential for vastly different outcomes depending on the diagnostic strategies applied.

Individuals who revert from MCI to normal over time are of particular interest. One concern would be that their change over time suggests that normal cognitive variability can be misinterpreted as impairment or decline and lead to unreliability in the MCI diagnosis. However, another hypothesis would be that these individuals do represent a unique risk group for future decline, though do not follow a linear trajectory. The tendency to revert from MCI to normal over time was most acute in those individuals identified as single domain non-amnestic MCI, irrespective of the diagnostic strategy applied, which is consistent with Busse and colleagues ⁶. While it is possible that depression played a role in the instability of this subtype, post-hoc tests showed that the GDS scores of the single domain non-amnestic group were not significantly higher than any other subtype. Further investigation of this subtype of MCI is certainly warranted to determine its validity and utility as a predictor of future decline.

The historical criteria provided the greatest stability over time (98%). It is likely, however, that the historical criteria missed a substantial sector of individuals with both amnestic and non-amnestic MCI since it utilized only paragraph recall to define memory impairment and failed to assess non-memory cognitive domains, even though development of Alzheimer's disease does not necessarily begin with a memory deficit. ²⁵ Thus, the historical criteria represent a conservative approach with high specificity for amnestic MCI, but limited sensitivity for distinguishing between normal individuals and those with other forms of MCI. This classification approach might be most appropriate for intervention trials where a high likelihood of negative side effects warrants the avoidance of false positive diagnoses to protect patient safety. The conservative criteria performed similarly to the historical criteria in terms of its stability over time. As its moniker implies, it also presents a conservative choice, similar to that of the historical criteria, but with the added benefit of the ability to identify non-amnestic forms of MCI.

The comprehensive criteria for MCI were developed in consideration of the fact that the interpretive value of an isolated impaired score is often limited. Heaton et al. ^{11, 12} reported that the majority of neurologically normal adults administered an expanded Halstead-Reitan battery obtained at least one “abnormal” test score. Similarly, Palmer and colleagues found that a sizable minority of healthy older adults earned one impaired score in two different cognitive domains, but very few (approximately 5% or less) had two or more impaired scores within the same cognitive domain.²⁶ It is perhaps not surprising that the comprehensive criteria exhibited greater stability over time than did the typical or liberal criteria, while maintaining the ability to detect non-amnestic and amnestic presentations that were not detected with the historical criteria.

The comprehensive criteria may also be the better choice for researchers and clinicians looking for a relatively stable diagnostic strategy; it offers a potentially better balance of sensitivity and specificity to detect impairment than the conservative or historical criteria present. It is also likely that the comprehensive and conservative criteria better approximate clinical decision-making processes. Clinicians typically examine multiple scores within a cognitive domain and base decisions on consistency of findings and patterns of impairment rather than isolated deficit scores. Thus, the comprehensive or conservative criteria may represent more valid strategies than the other approaches that only consider one test in identifying impairment (i.e., typical or liberal) since the one-test approaches are likely to lead to a higher proportion of false positive diagnoses based on statistical chance alone. These data argue for use of the comprehensive criteria over the conservative criteria in most cases given the likelihood that the stringent requirements of the conservative criteria result in a high false negative rate. Along the same lines, the historical criteria may lead to a higher proportion of false negative diagnoses because they are under-inclusive and do not identify non-amnestic presentations. Indeed, application of the comprehensive criteria classified a percentage of individuals as MCI (34%) that was the median value among the diagnostic approaches. Furthermore, the typical and liberal criteria may not be ideal for characterizing MCI subtypes, since the requirement of impairment on only one test within a cognitive domain led to a relatively high percentage (18-33%) of individuals who changed MCI subtype over time.

Because there is no “gold standard” operational criterion for diagnosis of MCI subtypes, and none of our sample has yet converted to AD, it is impossible to determine with certainty which diagnostic strategy is the most valid or has the highest sensitivity or specificity. From the standpoint of allowance for possible conversion to dementia, the follow-up interval was short. However, the relatively short follow-up interval is perhaps better suited for a methodological examination of diagnostic stability. Additional limitations of the study include the fact that, despite being a community sample, the group is highly educated and therefore may not generalize to lower education levels, although the use of age- and education-corrected norms in the current study represents an improvement from most decision-making methods in the existing literature. Finally, selective attrition may have influenced these results. However, only limited differences between those with and without follow-up were present and were unique to the groups created by the conservative criteria.

Despite these limitations, the present results quantify the vast variability arising from current diagnostic practices and add to our understanding of potential factors that explain the wide range of prevalence and conversion rates for MCI reported in the literature. The present findings clearly illustrate the importance of understanding the criteria used to identify cognitive impairment when making the MCI diagnosis and the value of using comprehensive neuropsychological assessment when diagnosing MCI subtypes. The results also highlight the instability of the single domain non-amnestic MCI subtype and point to the need for further exploration of MCI subtypes.