Developing a brief neuropsychological battery for early diagnosis of cognitive impairment

Abstract

Objectives:

Early diagnosis of cognitive impairment is increasingly emphasized in the literature to facilitate timely preventive interventions. While bedside cognitive tests such as the Montreal Cognitive Assessment (MoCA) is widely-used for such early diagnostic purposes, they may not have comparable performance to a full neuropsychological battery (FNB) in diagnosing early cognitive impairment. This study investigated whether a small subset of neuropsychological tests can be added-on to MoCA to match its performance to that of the FNB in discriminating mild cognitive impairment and dementia (MCI/dementia) from normal cognition.

Design:

Cross-sectional diagnostic study.

Setting:

Alzheimer’s Disease Centers across the United States.

Participants:

Older participants (≥50 years) who completed MoCA and the FNB (n=9,187).

Measures:

The study sample was split into two - the derivation sample (n=1,837) was used to develop a brief neuropsychological battery that best discriminated MCI/dementia (using the best-subset approach with tenfold-cross-validation); while the validation sample (n=7,350) verified its actual performance in discriminating MCI/dementia.

Results:

A 3-item neuropsychological battery was identified, comprising MoCA, Benson Complex Figure Recall, and Craft Story 21 Delayed Recall. It had excellent performance in discriminating MCI/dementia from normal cognition (AUROC 90.0%, 95%CI 89.2–90.7%), which was comparable to that of the FNB (AUROC 88.4%, 87.6–89.2%). By contrast, MoCA alone had significantly worse AUROC (86.9%, 95% CI 86.0–87.7%) than that of the FNB.

Conclusions and Implications:

Using rigorous methods, this study developed a brief neuropsychological battery that maintained the brevity of a bedside cognitive test, while rivaling the diagnostic performance of a FNB in early cognitive impairment. This brief battery offers a viable alternative when the full neuropsychological battery is needed but cannot be feasibly administered in non-specialty clinics. It can have a wider health-systems effect of improving patients’ access to accurate diagnosis in early cognitive impairment and facilitating timely interventions to delay the progression of cognitive impairment.

Brief summary:

This study developed a brief neuropsychological battery that can facilitate early diagnosis of cognitive impairment, and improve patients’ access to timely preventive interventions.

INTRODUCTION

Early diagnosis of cognitive impairment has many tangible benefits, and has increasingly been emphasized in the literature.¹ When diagnosed early, patients with mild cognitive impairment (MCI) or early dementia can receive timely interventions to delay disease progression, such as those related to cognitive training² and risk factor modifications.³ They may also be given the opportunity to participate in preventive trials for dementia, and contribute to the development of disease-modifying drugs for dementia. In the foreseeable future when the disease-modifying drugs become available, patients who have received early diagnosis will be more likely to benefit from the newer drugs to preserve available brain functions, at a much earlier time before any irreversible neuronal cell death has occurred.⁴

While the newer biomarkers of cognitive impairment (such as those related to amyloid protein, tau protein and neuronal injury)⁵ are increasingly used for early diagnostic purposes, they are often not accessible to general clinicians outside of specialized memory clinics. Consequently, much of the diagnostic process of cognitive impairment to date still relies on full neuropsychological batteries or bedside cognitive tests,⁶ of which the Montreal Cognitive Assessment (MoCA)⁷ is one of the most widely-used bedside cognitive tests for such purpose.⁸ The widespread use of MoCA for early diagnostic purposes is understandable, considering its many appealing features to general clinicians - MoCA can be completed in 10–15 minutes, is freely available at www.mocatest.org, covers the key cognitive domains relevant to MCI and dementia (including several robust measures of visuospatial and executive function), and has demonstrated acceptable performance in detecting early cognitive impairment.^7,8

Notwithstanding the appeal of bedside cognitive tests such as MoCA, they are much briefer than a full neuropsychological battery (FNB) and may not provide as much information as a FNB in the various cognitive domains.⁹ As a result, they may not be as good in detecting more subtle deficits in cognition, and hence may not necessarily be comparable to a FNB in identifying early cognitive impairment.⁹ Not surprisingly, the FNB is often still administered in clinical practice when there is diagnostic uncertainty about early cognitive impairment,⁹ on top of routinely-used bedside cognitive tests such as the MoCA. While the practice of adding the FNB to a bedside cognitive test is understandable, it does not take into account the information that has already been provided by the bedside cognitive test, or the presence of redundancies between the bedside cognitive test and the FNB. With a FNB requiring about 2–4 hours to complete, such redundancies potentially translate into much waste of clinical resources as well as the increase of burden of administration to the patients.

The current study sought to trim the redundancies between a bedside cognitive test (specifically MoCA) and a FNB, and produce a brief battery - comprising MoCA, plus a small subset of neuropsychological tests - which can rival the diagnostic performance of a FNB in discriminating MCI and dementia from normal cognition.

METHOD

Study population

This cross-sectional diagnostic study is based on the National Alzheimer’s Coordinating Center (NACC) database which involves participants recruited from the Alzheimer’s Disease Centers across the United States.¹⁰ It included participants who: (1) were recruited from March 2015 (the date when MoCA was first introduced in the NACC database) to May 2018; (2) aged ≥50 years; and (3) completed MoCA and the FNB in at least one of the study visits (only the first set of data was used when the participants completed these tests on multiple different visits). Research using the NACC database was approved by the University of Washington Institutional Review Board.

Measures

MoCA⁷ is a widely-used cognitive assessment tool. It assesses cognitive functions in the following seven domains: Visuospatial/Executive, Naming, Attention, Language, Abstraction, Delayed recall and Orientation. The FNB in NACC database included 11 nonproprietary cognitive tests¹¹ which covered the domains of immediate memory (Craft Story 21 Immediate Recall), visuospatial abilities (Benson Complex Figure Copy), delayed memory (Craft Story 21 Delayed Recall and Benson Complex Figure Recall), language (Multilingual Naming Test, and Verbal Fluency-Animal and L-words), attention (Number Span Test Forward and Backward), processing speed (Trail Making Test Part A) and executive function (Trail Making Test Part B). The 11 cognitive tests are further described in Supplementary Material 1.

The Z-scores for MoCA and the 11 cognitive tests in the FNB were computed using the age-, sex- and education-adjusted normative calculator that was recently published for the NACC cohort.¹¹ Additionally, the global Z-score for the FNB was also computed by averaging the Z-scores of all the 11 cognitive tests in the battery.

The diagnoses of MCI or dementia were made based on all available data, with majority of the diagnoses made via consensus conference (in 84.2% of the participants) and the remainder made by single clinicians. MCI was diagnosed using modified Petersen criteria,¹² while dementia was diagnosed with McKhann (2011) criteria.¹³

Statistical analyses

The study sample was randomly split into two - 20% of the sample was set apart for derivation of the brief neuropsychological battery that can best discriminate the diagnoses of MCI or dementia (MCI/dementia) from normal cognition, while 80% of the sample was used for validation of the actual performance of this brief battery.

The best-subset approach¹⁴ with tenfold cross-validation was employed in the derivation sample, to select items from the 12 potential test items (MoCA and the 11 cognitive tests in the FNB) which can best discriminate MCI/dementia from normal cognition. The best-subset approach is a computationally-intensive method of variable selection.¹⁵ It uses logistic regression to exhaustively evaluate all possible combinations among the 12 test items (MoCA and the 11 cognitive tests in the FNB), and narrows down to a list of top models which have the lowest prediction errors. It then selects the best model using tenfold cross-validation - by randomly dividing the sample into 10 folds of equal size, cross-validating the prediction error within the 10 folds, and selecting the most parsimonious model which is within one standard error of the best model (the ‘one-standard error’ rule is commonly applied to avoid selecting an overfitted model and ensure the replicability of the findings even in other independent samples).¹⁵ The selected model would then constitute the new, brief battery (henceforth referred to as MoCA+).

In the validation sample, the area-under-the-receiver-operating-characteristic-curve (AUROC) of MoCA+ was plotted against all the top models as well as against that of the FNB, to confirm that the selected MoCA+ was indeed the most parsimonious model even in a separate sample. AUROC of each model was evaluated with its global Z-score (that is, the averaged Z-scores of the items within each model). Following the confirmation of MoCA+ as the best model, its AUROC was further compared to that of the FNB via a non-parametric approach,¹⁶ with p-values of ≤0.05 indicating significant difference between the two AUROC.

Additionally, seven sensitivity analyses were conducted in the validation sample to re-evaluate the diagnostic performance of MoCA+ in discriminating:

1. MCI from normal cognition;

2. dementia from MCI;

3. dementia from non-dementia;

4. MCI/dementia from normal cognition among participants with >12 years of education;

5. MCI/dementia from normal cognition among participants with ≤12 years of education;

6. MCI/dementia from normal cognition among participants aged >65 years;

7. MCI/dementia from normal cognition among participants aged ≤65 years;

Best-subset approach was performed with the ‘bestglm’ package¹⁴ in R (version 3.5.1). The other analyses were conducted in Stata (version 14).

RESULTS

The total sample size was 9,187, comprising 62.3% normal cognition, 21.4% MCI and 16.3% dementia. The flow diagram related to participant selection is presented in Supplementary Material 2, while the participant characteristics are shown in Supplementary Material 3.

In the derivation sample (n=1,837), the exhaustive search method identified a list of top models as presented in Table 1. As shown in Figure 1, the tenfold cross-validation then selected the 3-item model as the most parsimonious model. This new brief battery (MoCA+) comprised: (1) MoCA; (2) Benson Complex Figure Recall; and (3) Craft Story 21 Delayed Recall.

Table 1.

The top models which best discriminate the diagnoses of mild cognitive impairment or dementia from normal cognition, as identified by the best-subset approach in the derivation sample (n=1,837).

Items in the brief neuropsychological battery	The identified top models (arranged by the number of items in the new battery)
	1	2	3	4	5	6	7	8	9	10	11	12
1. MoCA	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
2. Benson Complex Figure Copy								✓	✓	✓	✓	✓
3. Benson Complex Figure Recall			✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
4. Craft Story 21 Immediate Recall												✓
5. Craft Story 21 Delayed Recall		✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
6. Multilingual Naming Test										✓	✓	✓
7. Verbal Fluency (Animals)				✓	✓	✓	✓	✓	✓	✓	✓	✓
8. Verbal Fluency (L-words)											✓	✓
9. Number Span Test (Forward)							✓	✓	✓	✓	✓	✓
10. Number Span Test (Backward)									✓	✓	✓	✓
11. Trail Making Test (Part A)						✓	✓	✓	✓	✓	✓	✓
12. Trail Making Test (Part B)					✓	✓	✓	✓	✓	✓	✓	✓

MoCA, Montreal Cognitive Assessment.

Figure 1.

Model selection in the derivation sample (n=1,837), using tenfold cross-validation. Models with lower prediction error are considered better. Among which, the model with 10 items had the lowest values and was considered the best model. However, based on the ‘one- standard error’ rule, the model with 3 items was selected to constitute the new brief battery because it was the most parsimonious model that still fell within one standard error of the best model (indicated by the two horizontal dotted lines in the plot).

In the validation sample (n=7,350), the AUROC of MoCA+ was plotted against the AUROC of all the top models (from Table 1) as well as against that of the FNB, and were presented in Supplementary Material 4. Among the top models, the models with 3 to 7 items had AUROC which were higher than that of the FNB, with no overlap between their 95% CI and that of the FNB. Although the 5-item model had the highest AUROC, its 95% CI overlapped with those of the 3-item model, confirming that the 3-item model (MoCA+) was still the most parsimonious choice even in the validation sample.

The diagnostic performance of MoCA+ was then compared to that of the FNB in the validation sample, with the results presented in Table 2 (results related to the 4-item and 5-item models are separately presented in Supplementary Material 5 for readers who may still be interested in these two models). As shown in Table 2, MoCA+ had excellent performance in discriminating MCI/dementia from normal cognition (AUROC 90.0%); and was significantly better (p<0.001) than that of the FNB (AUROC 88.4%), albeit by a small margin. By contrast, MoCA alone had significantly worse AUROC (86.9%) than that of the FNB. MoCA+ had 78.2% sensitivity and 86.0% specificity at its optimal cut-off (Z-score≤−0.75), while the FNB had similar sensitivity (78.5%) but lower specificity (81.1%) at its optimal cut-off (Z-score≤−0.50).

Table 2.

Performance of MoCA and MoCA+ in discriminating the diagnoses of mild cognitive impairment or dementia from normal cognition in the validation sample (n=7,350), and a comparison of the performance with the full neuropsychological battery. Bold-faced values indicate the sensitivities and specificities at the optimal cut-off scores.

Global Z-scorea	Full neuropsychological batteryb		MoCA		MoCA+c
	Se, %	Sp, %	Se, %	Sp, %	Se, %	Sp, %
≤ −0.00	95.4	47.5	91.2	54.5	93.7	52.9
≤ −0.25	88.5	66.8	88.0	64.4	89.9	66.0
≤ −0.50	78.5	81.1	83.9	72.6	84.5	78.0
≤ −0.75	64.9	91.1	79.2	79.3	78.2	86.0
≤ −1.00	51.6	96.0	73.4	84.9	71.2	91.6
≤ −1.25	37.3	98.7	67.3	89.3	64.0	95.1
≤ −1.50	25.6	99.6	61.6	92.4	57.4	97.3
≤ −1.75	17.3	99.9	55.4	94.2	49.5	98.4
≤ −2.00	11.4	100.0	49.6	95.9	43.8	99.2
AUROC,% (95% CI)	88.4 (87.6–89.2)		86.9 (86.0–87.7)d		90.0 (89.2–90.7)e

MoCA, Montreal Cognitive Assessment; Se, sensitivity; Sp, specificity; AUROC, area under the receiver operating characteristics curve; CI, confidence interval.

^aZ-score standardizes the raw score relative to the mean and standard deviation of the population with normal cognition. It describes how much a person deviates from the population average. For example, a Z-score of −1 indicates that a person has an overall score which is 1 standard deviation below the population average.

^bThe full neuropsychological battery was based on the averaged Z-scores of 11 cognitive tests, namely Benson Complex Figure Copy and Figure Recall, Craft Story 21 Immediate Recall and Delayed Recall, Multilingual Naming Test, Verbal Fluency Test (Animal and L-words), Number Span Test (Forward and Backward), and Trail Making Test (Part A and B).

^cMoCA+ was based on the averaged Z-scores of MoCA, Benson Complex Figure Recall, and Craft Story 21 Delayed Recall.

^dThe AUROC of MoCA was significantly lower than that of the full neuropsychological battery (p<0.001).

^eThe AUROC of MoCA+ was significantly higher than that of the full neuropsychological battery (p<0.001).

The results remained consistent in the sensitivity analyses, with MoCA+ demonstrating comparable or slightly better performance to that of the FNB even across age and education subgroups (Supplementary Material 6–12).

DISCUSSION

By trimming the redundancies between a bedside cognitive test (MoCA) and a FNB, this study produced a brief neuropsychological battery (MoCA+) which capitalized on the strengths of both the bedside cognitive test and the FNB. This brief battery - comprising the widely-used MoCA and two nonproprietary neuropsychological tests (Benson Complex Figure Recall and Craft Story 21 Delayed Recall) - maintained the brevity of a bedside cognitive test, while rivalling the diagnostic performance of a FNB in early cognitive impairment. Notably, the brief battery was developed from a large sample (n=9,187) using rigorous methods - through exhaustive search of all possible combinations of items from the FNB, and following the well-established processes of derivation, cross-validation and independent validation.

While there can be other alternative methods to derive a brief battery, the best-subset approach (which exhaustively searched for the best subset in the FNB)¹⁴ has the strength of producing the most efficient brief battery that has the least number of items while remaining comparable to the FNB in its diagnostic performance. The product of this approach, MoCA+, offers a viable alternative the FNB is needed but cannot be feasibly administered due to practical reasons such as the limited availability of neuropsychologists, the high costs associated with the FNB, or the patients’ difficulties to complete 2–4 hours of neuropsychological assessments. It may be especially useful in non-specialty clinics (such as the primary care, geriatric and stroke prevention clinics), where there can be a pressing need to detect early cognitive impairment among high-risk patients with multimorbidity.

MoCA+ comprises nonproprietary tests and can be adopted by general clinicians without additional costs. The two add-on neuropsychological tests require only another 25 minutes to complete (which involves copying of a complex figure and registering a verbal story; follows by recalling the figure after 10 minutes and recalling the story after 20 minutes). They have clearly prescribed instructions that can be easily followed, and may potentially be administered by trained non-neuropsychologists such as nurses and allied-health professionals. After the test administration, their raw scores can be easily converted into age-, sex- and education-adjusted Z-scores, using a convenient normative calculator that has recently been published.¹¹ Essentially, MoCA+ provides general clinicians with the flexibility to extend beyond MoCA when there is still uncertainty about the presence of cognitive impairment. It can have a wider health-systems effect of improving patients’ access to neuropsychological testing and accurate diagnosis in the evaluation of early cognitive impairment. Potentially, it may narrow the treatment gap in routine clinical practice (often related to the lack of accurate diagnosis in early cognitive impairment) and facilitate timely interventions to delay the progression of cognitive impairment.

Several limitations should be considered. First, the findings were based on participants who volunteered at the Alzheimer’s Disease Centers, and may not necessarily apply to other settings with different disease profiles and comorbidities. As such, they will require further validation in other independent samples, in particular the confirmation that the 3-item model (MoCA+) is indeed the most parsimonious choice in detecting early cognitive impairment. Second, the diagnoses of MCI and dementia were made primarily by single clinicians in a small proportion of the participants (15.8%). They may not necessarily be as accurate as those made via consensus conference. Third, MoCA+ is intended to expand the armamentarium of general clinicians in confirming the presence of early cognitive impairment. It may be not be a replacement for other investigations which are still relevant in evaluating the etiology of cognitive impairment, including the use of neuroimaging, cerebrospinal fluid, or even the full battery of neuropsychological tests (when the detailed profile of cognitive deficits is still needed).

CONCLUSIONS AND IMPLICATIONS

In conclusion, this study developed a 3-item brief neuropsychological battery that maintained the brevity of a bedside cognitive test, while rivalling the diagnostic performance of a FNB in early cognitive impairment. The brief battery was developed using rigorous methods, and offers a viable alternative when the FNB is needed but cannot be feasibly administered in non-specialty clinics. It can have a wider health-systems effect of improving patients’ access to neuropsychological testing and accurate diagnosis in early cognitive impairment, which in turn can facilitate timely interventions to delay the progression of cognitive impairment.