Computer Assessment of Mild Cognitive Impairment

Abstract

Many older individuals experience cognitive decline with aging. The causes of cognitive dysfunction range from the devastating effects of Alzheimer’s disease (AD) to treatable causes of dysfunction and the normal mild forgetfulness described by many older individuals. Even mild cognitive dysfunction can impact medication adherence, impair decision making, and affect the ability to drive or work. However, primary care physicians do not routinely screen for cognitive difficulties and many older patients do not report cognitive problems. Identifying cognitive impairment at an office visit would permit earlier referral for diagnostic work-up and treatment. The Computer Assessment of Mild Cognitive Impairment (CAMCI) is a self-administered, user-friendly computer test that scores automatically and can be completed independently in a quiet space, such as a doctor’s examination room. The goal of this study was to compare the sensitivity and specificity of the CAMCI and the Mini Mental State Examination (MMSE) to identify mild cognitive impairment (MCI) in 524 nondemented individuals > 60 years old who completed a comprehensive neuropsychological and clinical assessment together with the CAMCI and MMSE. We hypothesized that the CAMCI would exhibit good sensitivity and specificity and would be superior compared with the MMSE in these measures. The results indicated that the MMSE was relatively insensitive to MCI. In contrast, the CAMCI was highly sensitive (86%) and specific (94%) for the identification of MCI in a population of community-dwelling nondemented elderly individuals.

Introduction

Many diseases of aging are associated with a decline in cognitive function. Early detection of cognitive decline in older individuals is crucial. It provides the opportunity to identify potentially treatable causes of cognitive loss, and in cases of early Alzheimer’s disease (AD), the opportunity to make future plans at a time when symptoms are mild and patients are able to make informed decisions, participate in their own care, and begin treatment to slow disease progression.¹ Identifying mild cognitive loss regardless of etiology also helps the family and physician determine whether the individual can safely manage complex activities such as financial decisions, handling medication regimens, or the ability to drive. This will become increasingly important with the aging of the population as it is estimated that 5 million people in the United States currently have AD.^2,3 An additional 5 million people have milder cognitive impairment that does not reach the threshold of dementia and may progress to AD.⁴

Current trends in health care suggest that the majority of elderly individuals will obtain their health care solely from their family physician; however, many do not report memory loss and, among those who do, few are referred for specialist evaluation.⁵ Many primary care physicians (PCPs) are hesitant to diagnose cognitive dysfunction, leading to substantial rates of under-detection of dementia in the primary care setting.^5–7 Boustani et al¹ found that up to 60% of all cases of dementia went unrecognized by PCPs and concluded that screening older patients could double the number of individuals whose cognitive loss is identified.

Although comprehensive neuropsychological assessments can identify individuals with impaired cognition, they are expensive, time-consuming, and not adaptable for use as screening tools in physicians’ offices. Tests such as the Mini Mental State Examination (MMSE) are widely used⁸ but may not be sensitive to mild cognitive changes because they are designed to identify the presence of dementia and have ceiling effects when applied to nondemented patients.⁹ The Montreal Cognitive Assessment, a brief paper-and-pencil test, has been shown to be more sensitive to MCI than the MMSE¹⁰ but it is not always adequately specific¹¹ and still has to be administered and scored by a clinician. More recently, several computer cognition tests have been developed, most of which are designed for use with a standard personal computer (responses are made either via the keyboard or mouse).^12–16 See Wild et al¹⁷ for a review of related tests. Most of these tests, whether paper-and-pencil or computer-based, have to be administered and scored by a clinician and few PCPs have the time to administer even a brief cognitive screening test.

We present data on the Computer Assessment of Mild Cognitive Impairment (CAMCI), a newly developed test designed to address the challenges of testing for cognitive impairment in a clinic setting.^18,19 The CAMCI is self-administered, automatically scored, and provides the physician with an indication as to whether the patient’s score would be determined by expert review to be within the range of MCI. We present data comparing performance on the CAMCI with results from a comprehensive neuropsychological evaluation and expert review. We hypothesize that the CAMCI would have good sensitivity and specificity for MCI and would be superior in these respects compared with the MMSE.

Materials and Methods

Participants

A total of 581 individuals ≥ 60 years were recruited for the study. Three hundred sixty completed the CAMCI and the comprehensive evaluation as part of a larger study of the utility of memory screening in PCP offices (PCP sample). Two hundred twenty-one individuals were recruited from 9 senior community centers in the urban and suburban Pittsburgh area (community sample) and also completed the CAMCI and the comprehensive evaluation. Exclusionary criteria included: diagnosis of alcohol abuse, stroke, mental health disorder, presence of neurological disease, and significant sensory deficit or physical limitation precluding performance on cognitive tests. Three subjects from the original sample were excluded from analysis because subsequent chart review revealed a prior diagnosis of dementia, 27 because they were classified as demented during the adjudication process, and 2 because of a score of ≤ 18 on the MMSE,⁸ indicating the presence of significant cognitive impairment. Three additional subjects were excluded because English was not their first language, 2 because of examiner error, 3 because they did not complete the evaluation, and data from 12 subjects were lost due to software error. The final sample size was 524 (PCP sample = 323, community sample = 201). The gender and racial demographics of the sample closely mirrored those of Allegheny County and the PCP offices. The study was approved by the University of Pittsburgh and Copernicus Group Institutional Review Boards and was conducted in accordance with the Helsinki Declaration, with all participants providing written informed consent.

Procedure

Participants were first approached by their PCP regardless of the reason for their office visit. Participants who consented were tested when not acutely ill, either at their home or PCP office. Community participants were recruited from their community center through advertisements, word-of-mouth, or direct contact with the examiner. Testing was performed in a quiet room at a community center.

Assessment

Each participant completed the MMSE⁸ and an extensive neuropsychological battery from the University of Pittsburgh Alzheimer Disease Research Center (ADRC) as well as the computerized test. The Appendix at the end of the article provides a full description of the CAMCI subtests. The ADRC battery includes multiple tests grouped into 5 cognitive domains: memory for verbal and visual material, attention/psychomotor speed, language abilities, spatial abilities, and executive functioning.^20,21 Participants also completed a comprehensive demographic/medical form, a modified version of the Center for Epidemiological Studies-Depression (mCES-D) scale,²² an assessment of instrumental activities of daily living (IADLs) (modified Alzheimer’s Disease Functional Assessment and Change Scale),²³ and a standardized questionnaire of subjective memory complaints.²⁴

Diagnostic Adjudication Procedure

All functional, medical, and cognitive data, with the exception of the results of the computerized test, were reviewed at consensus meetings that included 3 licensed neuropsychologists and the clinical research staff who interviewed the participant. These procedures conform to the Pittsburgh ARDC protocol which has been reported elsewhere.^25–27 The adjudication procedure followed the 2-step process described below.

1. Neuropsychological Classification

Participants were first classified as having cognitive test scores within the range of normal, MCI, or dementia using the following definitions from the University of Pittsburgh ADRC:^20,21

1. Cognition within the dementia range: individuals with test scores > 2 standard deviation (SD) below age-corrected norms on 2 cognitive domains, one of which must be memory.

2. Cognition within the range of MCI: individuals with at least 2 test scores between 1 to 2 SD below age-corrected norms.

3. Normal cognition: individuals not meeting either a or b.

2. Review of Medical and Functional Status

Demographic information including age, education level, prior occupation, medical history, subjective memory complaints, IADLs, and depression scores were reviewed to determine if these factors were consistent with the neuropsychological diagnosis.

The final clinical classification was based on a number of factors including cognitive test performance and information generated from research interviews and clinical data according to previously published criteria used at the Pittsburgh ADRC,^25–27 which are very similar to recent guidelines.^20,28 In accordance with these guidelines,²⁹ the panel considered whether individuals with impaired neuropsychological test performance also had subjective memory complaints and/or deficits in IADLs but did not require caregiver ratings of functioning as informants were not typically available for this study. For the majority of cases the panel reached a final classification without much discussion. However, in cases with impaired test scores that the expert panel felt could be attributed to, for example, low education level, significant depression, medical factors, or extreme old age (≥ 90 years), the neuropsychological classification was reassigned. Twenty-seven participants met the criteria for dementia and were not included in subsequent analyses because the purpose of this study was to determine the sensitivity and specificity of the CAMCI to detect MCI that does not rise to the level of dementia.

CAMCI Design

Data from the CAMCI were not used in the adjudication process. Because the test is designed for elderly individuals who may not be comfortable with computers, the CAMCI uses a response format that is no more complex than choosing which keys to hit on a telephone. The test runs on a tablet computer allowing greater portability and flexibility than desktop computers. Two major approaches were used in the development of the CAMCI tests. First, we modified standard neuropsychological tests of attention, executive abilities, working memory, and verbal and visual memory to allow administration and response via a computer format. All of the modified tests were scored for both accuracy and reaction time. Second, we used virtual reality technology to develop a test in which the individual moves through a virtual world on a shopping trip that resembles an everyday experience. The CAMCI takes about 20 minutes to complete and all tests are presented both visually and orally.

CAMCI Test-Retest

For test-retest reliability, 39 participants who had completed the assessment as part of the community sample were asked to return to retake the CAMCI 2 to 4 weeks after the initial session.

Results

Table 1 presents the demographic characteristics for the total sample, the PCP and community samples, and the 2 clinical subgroups: MCI and normal. Of the sample, 296 (54.5%) participants were identified as having normal cognition and 228 (45.5%) as having cognition within the range of MCI. Significant differences were observed in age and years of education between the 2 subject samples with the community sample being older and less well educated (P < 0.01). Significant differences were also observed in age, education, and MMSE score between the cognitively normal subgroup and the MCI subgroup (P < 0.01). There were also significant differences in 24 of the 25 CAMCI variables (P < 0.01), with the MCI subgroup having lower scores on every CAMCI subtest than the cognitively normal subgroup.

Table 1.

Demographic Characteristics of the Total Group, PCP and Community Samples, and the Normal and MCI Subgroups

	Total Group n = 524 (100%)	PCP n = 323 (61.6%)	Community n = 201 (38.4%)	Normal n = 296 (56.5%)	MCI n = 228 (43.5%)
Age, years	73.30	72.71	74.24	71.84	75.18
(Mean SD)	(6.52)	(5.96)	(7.25)a	(5.95)	(6.76)a
Education, years	13.46	13.81	12.90	13.74	13.10
(Mean SD)	(2.67)	(2.82)	(2.31)a	(2.69)	(2.61)a
Gender (% male)	34.9	39.6	27.4	32.8	37.7
Race (% white)	94.3	97.8	88.6	95.9	92.1
MMSE score	28.12	28.13	28.09	28.67	27.41
Mean (SD)	(1.66)	(1.76)	(1.48)	(1.39)	(1.71)a

Abbreviations: MCI, mild cognitive impairment; MMSE, Mini Mental State Examination; PCP, primary care physician; SD, standard deviation.

^aSignificant at the P < 0.01 level.

We performed classification and regression trees (CRT) to determine: 1) the importance of each variable to the classification of a subject within the 2 categories (normal or MCI) and 2) the sensitivity and specificity of the CAMCI in classifying subjects into one of the 2 categories (normal or MCI).

Using the CAMCI tasks described in the Appendix, the CRT grew a tree to a depth of 13 levels and identified the number of words recalled (out of 5) on the Word Recall task as the most important variable in the classification of normal or MCI. Two tests were identified at the second level: Prospective Memory on the shopping trip and percent correct from the Picture Recognition (Visual Memory) test. At the third level the CRT identified: the number of correct choices made at the road intersections on the Shopping Trip Choice Points (out of 18); time taken to complete the first trial of the Forward Digit Span (Attention) task; and time taken to identify the correct word on the Word Recognition (Verbal Memory) test and the correct picture on the Picture Recognition (Visual Memory) test. Age, education, and gender were also entered into the model but only age appeared in the tree and then not until level 7.

The CAMCI was 86% sensitive in classifying MCI participants as MCI (ie, the probability of a positive test among patients whose cognition is within the range of MCI). Furthermore, the CAMCI was 94% specific in classifying normal subjects as normal (ie, the probability of a negative test among patients whose cognition is normal). We derived the sensitivity and specificity from the final tree model which was created using 10-fold cross-validation. The 10-fold cross-validation method divides the sample into subsamples and then generates trees excluding 1 subsample at a time and is superior to other validation methods such as the split-sample approach because all observations are used.³⁰ The CRTs were completed in SPSS statistical software (Version 16.0, SPSS Inc., Chicago, IL) using the CRT growth method, Gini impurity measure, tree depth of 13, 10 minimum cases in parent node, 1 minimum case in child node, and minimum change in improvement was set to a Gini impurity level of 0.0001.

Table 2 shows the sensitivity and specificity for identifying MCI in this sample using the CAMCI and 5 different MMSE cut-off scores. The CAMCI was 86% sensitive and 94% specific for MCI in the whole group of 524 subjects. Similar results were obtained when the community and PCP groups were considered individually. The accuracy of classification is shown separately for each group in Table 3 and Table 4.

Table 2.

Sensitivity and Specificity of the MMSE for 4 Cut-off Scores and for the CAMCI

MMSE Cut-off	25	26	27	28	CAMCI
Sensitivity	4%	13%	28.5%	45%	86%
Specificity	99%	97%	92%	80%	94%

Abbreviations: CAMCI, Computer Assessment of Mild Cognitive Impairment; MMSE, Mini Mental State Examination.

Table 3.

Comparison of Adjudication and CAMCI Classifications for Community Subjects

	Adjudication Classification
CAMCI		Normal	MCI
Classification	Normal	92 (46%)	14 (7%)
	MCI	6 (3%)	89 (44%)

Abbreviations: CAMCI, Computer Assessment of Mild Cognitive Impairment; MCI, mild cognitive impairment.

Table 4.

Comparison of Adjudication and CAMCI Classifications for PCP Subjects

	Adjudication Classification
CAMCI		Normal	MCI
Classification	Normal	185 (57%)	13 (4%)
	MCI	13 (4%)	112 (35%)

Abbreviations: CAMCI, Computer Assessment of Mild Cognitive Impairment; MCI, mild cognitive impairment.

CAMCI Test-Retest

Pearson correlations were performed between initial testing and follow-up testing (mean interval = 20.7 days) on a subset of 39 subjects (18 = normal; 21 = MCI). Correlations ranged from 0.30 to 0.74 and were significant for 8 of the variables at the P < 0.01 level. Three variables were significant at the P < 0.05 level and 1 variable (shopping trip, Incidental Recall) approached significance (P = 0.06). These results indicated that the CAMCI was highly stable over a relatively short time.

Discussion

Our goals in developing the CAMCI were to use current cognitive criteria for MCI²⁹ to develop a test that is self-administered, user-friendly, scores automatically, and can be completed in a quiet space (eg, doctor’s examination room) to provide a score that represents the probability that the individual’s cognitive performance would fall within the range of MCI if tested on a comprehensive neuropsychological test battery and reviewed by expert neuropsychologists. Our results indicated that the CAMCI successfully identified elderly nondemented patients with MCI and was more effective compared with the MMSE.

Decline in everyday memory is an indicator of MCI and impaired performance on standard neuropsychological tests of memory was one of the criteria used to identify MCI. Therefore, it is not surprising that the CRT analysis identified performance on tests of verbal and visual memory (both free recall and recognition) as 2 of the most important variables. Reaction time was also identified as an important variable, suggesting that individuals with MCI may make correct responses but take longer to do so. Also, performance on 2 of the shopping trip tasks entered into the CRT analysis at an early stage, levels 2 and 3, indicated that performance on this ecologically valid assessment can also be used as an indicator of MCI.

A MMSE cut-off of 28 provided the best sensitivity and specificity (45% and 80%, respectively) for identification of MCI but this was considerably poorer than the rates observed using the CAMCI. Our findings suggest that the MMSE is insensitive at the upper end of the range of cognitive abilities and does not identify individuals with the very mildest, earliest signs of cognitive problems.

One important limitation of this study is the absence of a reliable informant to determine the classification of MCI. The use of proxies to provide information about the participant’s cognitive and functional abilities is helpful because self-reporting may not be reliable, particularly among cognitively impaired individuals. Another limitation is the possibility that individuals > 65 years have limited experience using computers and may be reluctant to try the test. However, that was not our experience in this study. Only 3 of the original samples of 581 individuals recruited for the study failed to complete the test. The majority of elderly patients were enthusiastic in their response to using the computer. The current literature on computer use by the elderly suggests that older Americans are increasingly relying on computers for communication, purchases, and searching for information and suggests that print size must be adequate, they must be able to self-pace, have the opportunity to practice, and the presentation of material must be highly organized with visual displays that are simple and relevant to the task.^31–33 The CAMCI satisfies these conditions and the patient both sees and hears the instructions to increase comprehension and reduce the possibility of errors from sensory impairment. The touch-screen tablet computer is no more complex than a phone or automatic teller machine (ATM) keypad. As the baby-boom generation ages, more elderly individuals will have had routine experience with computers in the workplace and at home.

The community sample was older and less well educated than the PCP sample. The demographic differences in the 2 samples represent the normal range of variance in the elderly population and ensure that the total sample in this study was more representative of the community at large than either sample would be individually. Furthermore, each individual was clinically adjudicated as either having normal or impaired cognition taking into account demographic characteristics.

Our sample included a higher percentage of individuals with MCI than typically found in the general population. For example, Meguro et al³⁴ reported a 30.2% overall frequency of MCI. There are 2 possible reasons for this finding. First, it was not an epidemiological study. Primary care physicians were more likely to refer individuals who they had concerns about than those they thought had normal cognition, and the senior citizens centers from which the community sample was recruited probably included a disproportionately large number of frail elderly patients. Second, currently accepted criteria for differentiating MCI from dementia and normal cognition from MCI are evolving. They are imperfect and the neuropsychological cut-off for cognitive impairment used in this study, 1 SD below age norms, is more liberal than suggested by some researchers.^29,35–38 This may have resulted in more individuals with normal cognition being included in the cognitively impaired group than would be the case using other criteria. However, application of the neuropsychological criteria was only the first stage of the review process and many individuals with neuropsychological test scores within this range would not have been given a final classification of MCI once the expert panel reviewed the demographic, medical, psychiatric, and functional data. Using this higher cut-off point allowed us to include individuals who are often missed from studies employing lower cut-off points (ie, those highly educated, high-functioning individuals who have subjective complaints and may have declined from a previously higher level but not met a lower cut-off). As with the results of a comprehensive neuropsychological evaluation, performance on the CAMCI alone does not determine the etiology of a cognitive impairment and performance on the CAMCI is only 1 indicator. The physician should determine the validity of the score, the need for further investigation, and the final etiology of the deficit.

We recognize that many clinicians are reluctant to subject patients to tests of cognitive functioning fearing that some will become despondent, leading to increased rates of depression. With no currently available effective screening test for determining MCI leading to AD, widespread screening in PCP offices has not been proven to be cost-effective. However, proponents of screening believe that a quick and easy test would capture a baseline assessment against which future cognitive changes could be compared, assuming an effective treatment will become available. Some propose only screening patients who are reporting noticeable memory problems in everyday life. However, not all impaired elderly patients report cognitive decline.³⁹

Our goal in developing the CAMCI was not to develop a test to diagnose early AD or other cognitive disorders, but rather to develop a test that reliably reproduces the results of an expert interpretation of a comprehensive neuropsychological test battery without the attendant time burden or the need for expertise of administration, scoring, and interpretation. We envisage that the CAMCI will be an addition to the currently available tests and that PCPs will be able to use it for determining the presence or absence of cognitive difficulties as readily as they test for the presence of vascular risk factors, heart disease, or metabolic disease.

Conclusion

We have reported the results of a relatively simple test that identifies the presence of mild cognitive difficulties and is suitable for use in PCP offices. The CAMCI is highly specific in reliably identifying older individuals with normal cognition. The test is also highly sensitive compared with the MMSE and other currently available measures, but may misidentify a small number of individuals with cognitive difficulties as normal. High specificity is essential for a test of MCI because those individuals who are classified as having “normal cognition” can be reassured that their scores are indeed within the normal range and the small number of patients with progressive cognitive decline who may be misclassified will be identified at a later date, but still sooner than is possible with current screening methods.

Appendix

CAMCI Subtests

The CAMCI takes about 20 minutes to complete and is comprised of 7 subtests that are computerized versions of standard pencil-and-paper tests and a virtual reality shopping trip during which the subject must remember to perform various tasks.

Modified Pencil-and-Paper Tests for Computer Presentation

Star Task (Attention)

The participants were shown a star, circle, square, or triangle and told to tap the screen as quickly as possible only when the star appeared.

Forward Digit Span (Attention)

A series of 3 to 6 numbers was presented, 1 per second. After presentation, participants were told to recall the numbers in the correct order using a display at the bottom of the screen.

Word Recognition (Verbal Memory)

A list of 6 words was shown on the screen, 1 at a time. The participant was instructed to remember each word and warned that they would be asked to recall it later. After a delay, 6 sets of 4 words (3 distracters and the target word) were displayed and the task was to tap the target word.

Word Recall (Verbal Memory)

Participants were presented with a series of five 3-letter words one at a time and told to remember them. The words were presented 3 times. Approximately 10 minutes later participants were asked to recall the 3-letter words by typing them on a keyboard that appeared at the bottom of the screen.

Picture Recognition (Visual Memory)

The participant was shown a fixed series of pictures. Some had been shown previously and some were new. The participant was told to tap “yes” if the picture had been shown previously and “no” if it had not.

Go/No-Go Test (Executive Function)

In Part 1 of this task, participants were asked to tap the screen twice when they heard 1 beep and once when they heard 2 beeps. In Part 2 the rules were changed and participants were told to tap twice when they heard 1 beep and do nothing when they heard 2 beeps.

Digit Reverse Span (Working Memory)

This task is identical to Forward Digit Span except that participants were told to recall the numbers in reverse order.

CAMCI Shopping Trip/Virtual Reality

Participants were asked to navigate through a virtual world on the tablet computer. They were told they would be driving to “Sullivan’s Market” to purchase several items and on the way they must run several errands such as stopping at the bank and the post office. This approach to cognitive assessment is more ecologically valid and better tolerated by older participants than traditional pencil-and-paper tests. This approach also offers the opportunity to test abilities not usually assessed in laboratory test batteries but which are relevant to everyday functioning, such as prospective memory (remembering to do something in the future), incidental memory (remembering material that you did not expect to have to remember), and decision-making.

Recognition Memory

The participant was shown a list of grocery items (ie, bread, bananas, donuts, and shampoo) and told that these are the items he/she wants to purchase at the store. At the end of the shopping trip the participant was shown a series of pictures including the target items and told to tap the picture of the item they were told to purchase.

Incidental Recall

During the shopping trip the participant passed several objects including a sedan car, a city bus, and a boy on a bicycle, but was not warned ahead of time to remember them. After the trip he/she was presented with a series of pictures and told to tap on the ones that appeared during the test.

Prospective Memory

At the beginning of the shopping trip the participant was told that on the way to the market they must stop at the bank to transfer money and at the post office to mail a letter. The participant was told to tap on the image of the bank and the post office when they appeared on the screen to indicate that they had remembered to perform these tasks.

Shopping Trip Choice Points

At the beginning of the shopping trip the participant is provided with directions to Sullivan’s Market. The directions include statements such as “Make a left on Fir Street, then make a right on Ash Street.” The directions remained on the screen throughout the test so the participant could refer to them at any time. At each intersection the participant had to decide which direction to go.

Bank Machine

When the participant arrived at the bank he/she saw a standard bank automatic teller machine (ATM). The participant was told to transfer $250.00 from the savings account to the checking account. Each step of the transaction was scored for correctness and time taken to complete the task. If the participant did not remember to stop at the bank, the CAMCI automatically “drove” to the bank so that this portion of the test was completed by all participants.