Picture-Based Memory Impairment Screen for Dementia

Abstract

OBJECTIVES

To develop and validate a picture-based memory impairment screen (PMIS) for the detection of dementia.

DESIGN

Cross-sectional.

SETTING

Outpatient clinics, Baby Memorial Hospital, Kozhikode city in the southern Indian state of Kerala.

PARTICIPANTS

Three hundred four community-residing adults aged 55 to 94 with a mean education level of 8 years; 65 were diagnosed with dementia.

MEASUREMENTS

PMIS: a culture-fair picture-based cognitive screen designed to be administered by nonspecialists. Diagnostic accuracy estimates (sensitivity, specificity, positive and negative predictive power) of PMIS cut-scores in detecting dementia (range 0–8).

RESULTS

PMIS scores were worse in participants with dementia (1.5) than in controls (7.7, P < .001). At the optimal cut-score of 5, PMIS had a sensitivity of 95.4% (95% confidence interval (CI) = 90.3–100.0%) and a specificity of 99.2% (95% CI = 98.0–100.0%) for detecting dementia. In the 167 participants with <10 years of education, PMIS scores of five or less had a sensitivity of 97.8% (95% CI = 93.6–100.0%) and specificity of 99.2% (95% CI = 97.6–100.0%). The PMIS had better specificity than the Mini-Mental State Examination in detecting dementia, especially in older adults with low education.

CONCLUSION

The PMIS is a brief and reliable screen for dementia in elderly populations with variable literacy rates.

The prevalence of dementia is predicted to increase three to four times more in developing than developed regions.¹ A recent review noted that only 20% to 50% of older adults with dementia are recognized in primary care in high-income countries.² Underdetection of dementia is expected to be greater in low- and middle-income countries. A study in India suggested that 90% of dementia cases remain unidentified.³ It is estimated that as many as 28 million of the world's 36 million people with dementia have yet to receive a diagnosis and are therefore not receiving appropriate treatment, information, and care.² More research is needed for developing countries to estimate the extent of and risk factors for dementia and to make confident predictions of future needs.^4–6

A major limiting factor in identifying dementia in developing countries is the lack of cognitive screens that account for cultural differences and variable literacy rates.^4,5,7–9 Furthermore, many cognitive screens are lengthy and not designed for nonspecialist use.^10,11 The Memory Impairment Screen (MIS) is a brief, four-item delayed free- and cued-recall memory test.¹² It has been translated into other languages,¹³ correlates highly with Alzheimer's pathology,¹⁴ and is recommended for dementia screening,^10,11 but it requires minimum literacy levels that restrict its use in low-education populations.¹³

The primary aim of the current study was to develop a picture-based version of the MIS (PMIS) to screen for dementia that would account for limitations of the MIS and other screens.¹¹ The diagnostic accuracy of the PMIS was examined in a clinic-based sample of 304 individuals aged 55 to 94, 65 with dementia, participating in the Kerala-Einstein Study (KES).¹⁵ Feasibility of administration of the PMIS by nonspecialists was examined as a secondary aim.

METHODS

Participants

The goals of the KES were to build research infrastructure and to identify risk factors for dementia in the southern Indian state of Kerala. KES is based in Kozhikode and Thiruvananthapuram cities, as well as a rural site in Thrissur district.¹⁵ The PMIS validation study was conducted at Kozhikode and the feasibility study at Thrissur. Consecutive patient referrals to the neurology clinics at Baby Memorial Hospital, Kozhikode, between October 2008 and October 2010 were prospectively recruited.¹⁵ Reasons for referral to clinic included cognitive complaints or other neurological symptoms such as headache, strokes, or neuropathy. Additional controls were recruited from relatives of clinic patients. Inclusion criteria were aged 55 and older and willing to complete study procedures. Exclusion criteria included severe audiovisual loss or presence of active medical, neurological, or psychiatric illness that in the judgment of the study clinician (MLN) would interfere with completion of study procedures. During the 25-month study period, 360 individuals were identified, of whom 304 met study criteria. Informed consent was obtained before enrollment, and study evaluations followed protocols approved by the Baby Memorial Hospital and the Albert Einstein College of Medicine institutional review boards.

Clinical Evaluations

All interviews and tests were conducted in the local language (Malayalam). A study clinician (MN) obtained medical history, conducted a neurological examination, completed the Clinical Dementia Rating scale (CDR),¹⁶ and administered the PMIS. Another clinician (BJ) with behavioral neurology expertise assessed the following cognitive domains using locally validated tests: Addenbrooke Cognitive Examination (ACE, general cognition),^8,9 Rey Auditory Verbal Learning Test (memory),¹⁷ Digit Span Test (attention and executive function),¹⁸ and 15-item Geriatric Depression Scale (mood).¹⁹ The ACE incorporates the Mini-Mental State Examination (MMSE).⁹ Clinicians administering PMIS and cognitive tests were blinded to each other's evaluations and to evaluations done at the clinic before enrollment.

PMIS Development and Administration

A pool of 30 color digital photographs was created based on items with high naming and familiarity scores on a locally validated line drawing series.²⁰ The photographs were shown to 25 volunteers (40% women) aged 18 to 76 who did not participate in the validation study. The list was narrowed down to 12 photographs that all volunteers easily recognized. The photographs were randomly assigned to three sets of four pictures each. Each picture in the set belonged to a different category (Figure 1).

Figure 1.

Picture-based memory impairment screen picture set example. Items with category cues clockwise from top left are bicycle (mode of transportation), feet (body part), monkey (animal), and bangle (ornament).

PMIS administration procedures followed those for MIS.¹² Participants were tested individually and were randomly administered one of three sets. Participants were asked to identify the pictures shown one by one on a 13.3-inch computer screen. The computer program did not allow the examiner to change the picture set or retake the set. Participants were then asked to name each item (e.g., feet) when the tester said its category cue (e.g., body part). After an interference task lasting approximately 2 minutes (repeated counting from 1 to 20 and back or questions about activities of daily living in participants unable to count), the individual was asked to recall the pictures in any order. Category cues were presented to elicit recall only of items that were not retrieved by free recall. Two points were given for each freely recalled picture and 1 point for pictures recalled with cues (range 0–8).

Diagnostic Procedure

All clinical and neuropsychological information (except PMIS) was reviewed at consensus case conferences that two or more study clinicians attended. Dementia was diagnosed according to Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition criteria.²¹ The clinician (MN) administering the PMIS did not assign the final diagnosis.

Statistical Methods

Analysis of variance for continuous variables and chi-square tests for categorical variables were used to test differences in baseline characteristics according to dementia status.²² The three PMIS forms were compared using the Kruskal-Wallis test.²² Internal consistency of PMIS was determined using the Cronbach coefficient alpha.²² Interrater reliability was examined in 20 participants assessed by the study clinician and a nurse independently. Twenty participants were administered the MIS, and Pearson correlations with PMIS were examined.

Discriminative validity was assessed by calculating the sensitivity and specificity of the PMIS for detecting dementia for various cut-scores.²² The base rate of dementia in the sample was 21% (65/304). Because base rate of dementia varies with setting and age, positive predictive value (PPV) and negative predictive value (NPV) are presented under different assumptions of dementia prevalence (%) using Bayes Theorem.

The discriminative validity of the PMIS was examined in participants with less than a high school-level education. Six participants could not complete the ACE (but did the PMIS).⁹ The discriminatory validity of the MMSE and the PMIS were compared at optimal cut-scores²³ to detect dementia in the overall and low-education samples. Sensitivity analyses were conducted to account for the influence of mild cognitive impairment (MCI)²⁴ on the PMIS. To increase generalizability, participants with depression were not excluded. All statistical analyses were performed using SAS 9.2 (SAS Institute, Inc., Cary, NC).

Power

Initial power estimates, based on the MIS study,¹² indicated that 100 participants without and 50 with dementia would provide 85% power to test PMIS validity. The KES recruited a larger sample to serve other aims.¹⁵ Because all participants readily accepted the PMIS, testing was extended to the larger sample to increase power (>90%), enabling estimates of sensitivity with a standard error of 5% or less and specificity with a standard error of 3% or less.

RESULTS

Of the 304 eligible KES participants, 160 (52.6%) were men, and 144 (47.4%) were women. Mean age was 68.3. Mean education was 8 years, with 167 (54.9%) participants with less than a high school–level education (<10 years) and 16 (5.3%) without formal schooling. Sixty-five participants met dementia criteria (34 Alzheimer's disease (AD), 17 vascular dementia, 12 mixed dementia, and one unspecified).

All participants completed the PMIS in 4 minutes. Mean PMIS score was 6.4 ± 2.8. PMIS scores showed good correlation with the ACE (Pearson correlation coefficient (r) = 0.79, P < .001) and the MMSE (r = 0.80, P < .001). PMIS scores were weakly correlated with age (r = −0.36), education (r = −0.21) and depressive symptoms (r = −0.15). PMIS scores were not different in men (6.6 ± 2.5) and women (6.3 ± 3.0, P = .35). PMIS scores were worse in individuals with dementia (1.5) than in controls (7.7, P < .001), as were other cognitive test scores (Table 1). The 44 participants with Geriatric Depression Scale scores of 10 or greater (cut-score for major depression) had worse MMSE (20.8 ± 2.9 vs 25.8 ± 2.8, P < .001) but not PMIS scores (6.7 ± 1.1 vs 7.6 ± 0.4, P = .35) than the other participants.

Table 1.

Demographic Variables and Cognitive Performance Overall and According to Dementia Status

Variable	Overall, n = 304	Dementia, n = 65	Controls, n = 239	P-Value
Age, mean ± SD	68.3 ± 6.3	72.2 ± 7.2	67.3 ± 5.7	<.001
Female (%)	47.4	55.4	45.1	.09
Education, years, mean ± SD	8.0 ± 3.8	6.5 ± 3.5	8.4 ± 3.8	<.001
Cognitive tests, mean ± SD
Picture-based Memory Impairment Screen (range 0–8)	6.4 ± 2.8	1.5 ± 2.0	7.7 ± 0.5	<.001
Addenbrooke Cognitive Examination (range 0–100)	69.5 ± 22.2	35.9 ± 16.7	78.4 ± 13.1	<.001
Mini-Mental State Examination (range 0–30)	24.9 ± 7.0	13.5 ± 6.1	27.4 ± 3.2	<.001
Rey Auditory Verbal Learning Test 20-minute recall	3.8 ± 3.5	0.1 ± 0.6	4.8 ± 3.2	<.001
Digit span forward	5.5 ± 3.4	3.9 ± 1.1	5.9 ± 1.1	<.001
Digit span backward	3.4 ± 2.8	1.3 ± 1.4	3.9 ± 2.8	<.001
Geriatric Depression Scale (range 0–15)	4.9 ± 3.4	6.1 ± 3.1	4.6 ± 3.4	.001

SD = standard deviation.

P-values are for comparisons between participants with dementia and controls using chi-square and t-tests.

Reliability and Feasibility

No effect of picture set was found on PMIS (P = .77), indicating high alternate forms reliability. The Cronbach alpha for all participants was 0.90. Interrater reliability on PMIS scores between the study clinician (MN) and nurse in 20 participants was excellent (intraclass correlation (ICC) = 0.91). Concordance between PMIS and MIS scores in 20 participants was excellent (ICC = 0.88).

To demonstrate feasibility of administering PMIS by nonspecialists, two healthcare workers with 10 years of education were trained in one session to administer the PMIS. Individual picture prints were used. The healthcare workers administered the PMIS to 21 individuals (aged 60–84, 52% women) at the rural KES site. Mean PMIS score was 7.2 ± 0.5. A psychiatrist (KSS) blinded to PMIS scores diagnosed all participants as cognitively normal (ICC = 1.00). The psychiatrist did not participate in diagnostic conferences for the primary study. The site and participants were not part of the validation study.

Validity

The optimal PMIS cut-score is 5 (sensitivity 95.4%, 95% confidence interval (CI) = 90.3–100.0%; specificity 99.2%, 95% CI = 98.0–100.0%). The association of this cut-score with dementia remained significant even after adjustments for age, sex, education, and depressive symptoms (P < .001). In a subgroup analysis matching the 65 individuals with dementia to 103 controls according to median age, sex, and low education, the PMIS cut-score retained high sensitivity (95.0%) and specificity (98.1%).

Table 2 shows the PPV and NPV of PMIS scores at different dementia base rates. The PPV for a PMIS score of 5 or less for detecting dementia varies from 53.5% at a 1% dementia base rate to 96.6% at a 20% rate. NPV varied from 85.7% to 98.8% at different dementia rates.

Table 2.

Discriminative Validity of Picture-Based Memory Impairment Screen (PMIS)

			Prevalence of Dementia
			1%		5%		10%		20%
PMIS Score	Sensitivity	Specificity	PPV	NPV	PPV	NPV	PPV	NPV	PPV	NPV
0	0.523	1.000	1.000	0.995	1.000	0.976	1.000	0.950	1.000	0.893
1	0.615	1.000	1.000	0.996	1.000	0.980	1.000	0.959	1.000	0.912
2	0.723	1.000	1.000	0.997	1.000	0.986	1.000	0.970	1.000	0.935
3	0.815	1.000	1.000	0.998	1.000	0.990	1.000	0.980	1.000	0.956
4	0.862	0.996	0.675	0.999	0.916	0.993	0.958	0.985	0.981	0.966
5	0.954	0.992	0.535	1.000	0.857	0.998	0.927	0.995	0.966	0.988
6	0.985	0.946	0.155	1.000	0.488	0.999	0.668	0.998	0.819	0.996
7	0.985	0.824	0.054	1.000	0.228	0.999	0.384	0.998	0.583	0.995
8	1.000	0.000	0.010	—	0.050	—	0.100	—	0.200	—

Sensitivity and specificity were calculated based on results obtained from the study sample. Positive predictive value (PPV) and negative predictive value (NPV) were calculated at different hypothetical dementia base rates (see Methods).

The mean PMIS score was 1.7 ± 2.1 in 34 participants with AD and 1.4 ± 2.0 in 31 with dementia other than AD. The PMIS score of 5 or less detected AD with a sensitivity of 94.1% (95% CI = 86.2–100.0%) and a specificity of 99.2% (95% CI = 98.0–100.0%). CDR score¹⁶ in the dementia cases was 0.5 in four participants, one in 12, and two or greater in 49. The sensitivity of a PMIS score of 5 or less to detect 16 mild dementia cases (CDR score of 0.5 or 1) was 82.4%, specificity was 99.2%, PPV was 87.5%, and NPV was 98.8%.

The established MMSE dementia cut-score of 24 had a sensitivity of 96.9% (95% CI = 92.7–100.0%) and a specificity of 83.3% (95% CI = 78.5–88.0%). The sensitivity of the MMSE, hence, was similar to that of the PMIS, although the specificity of the PMIS was higher. When restricted to 167 participants with low or no education, a PMIS score of 5 or less had a sensitivity of 97.8% (95% CI = 93.6–100.0%) and a specificity of 99.2% (95% CI = 97.6–100.0%) for detecting dementia. In contrast, the MMSE cut-score of 24 had a sensitivity of 100% and a specificity of 71.1% (95% CI = 63.0–79.2%). A PMIS score of 5 or less had high sensitivity (89.5%) and specificity (99.2%) in the 137 participants with high school–level education or higher.

Twenty-seven participants met criteria for MCI (74% amnestic).²⁴ Mean PMIS score in participants with MCI was 7.2 ± 1.0. Sensitivity (95.4%) and specificity (99.5%) for a PMIS score of 5 or less remained high after excluding MCI cases.

DISCUSSION

The PMIS is a quick and reliable screen for dementia that can be used in older adults with little or no education. It discriminated cognitively normal older adults from those with dementia regardless of age, sex, education, severity of dementia, or presence of depression. All participants completed the PMIS, which took approximately 4 minutes including the interference task. High interrater reliability was demonstrated between administration of PMIS by the clinician and the nurse. The high alternate-forms reliability supports use of different PMIS sets for repeated testing. Professionals and nonprofessionals successfully administered the PMIS in urban and rural clinic settings. The PMIS was administered on a computer screen and using cards. These findings support the feasibility of using the PMIS in various settings and by personnel with different levels of expertise.

High concordance was seen as expected between the MIS and the PMIS,¹² although a main limitation of the MIS is that it cannot be applied to low-education populations. Low education and cultural differences can result in cognitively unimpaired people screening positive for dementia.⁶ Most screens have not been validated in samples with low education or illiterate individuals.^6,7 More than half of the participants in the current study had low education, including one-third who had 5 years or less of schooling or were illiterate. The PMIS was successfully administered to all of these participants (six could not do ACE) and had high diagnostic accuracy. Most of these participants would not be able to complete the MIS. In a study from Spain, 16% of the sample with low education could not complete the MIS.¹³ The PMIS had better specificity than the MMSE, especially in the low-education group. Dementia screens that are designed to be self-administered²⁵ or rely on reliable and educated informants²⁶ may have limited generalizability in populations with low or no literacy.

Pictures are remembered better than words. The visual component of the PMIS involves deeper or additional layers of cognitive processing; enhancing learning and recall. Poor visual memory predicts dementia,²⁷ suggesting an alternate mechanism (not tapped by the MIS) for PMIS validity in detecting dementia. Other pictorial tests such as the visual association test,²⁸ phototest,¹³ and scenery picture memory test²⁹ have reliably detected dementia in developed countries, supporting the approach of the current study.

Table 2 shows sensitivity and specificity based on the sample and PPV and NPV at hypothetical dementia base rates that can guide choice of cut-scores. A sensitive PMIS cut-score can be used to monitor cognitive function in older adults and guide timing of follow-up visits, whereas a specific cut-score with high PPV can be used to avoid false-positive cases when recruiting candidates for clinical trials involving potentially toxic medications.

As with any volunteer sample, selection biases limit generalization of results. The sample was less selective than samples excluding individuals with MCI, depression, or low education.^10–12,25 As with any cognitive test, cross-validation in other settings and populations is necessary. Pictures that are locally relevant can be selected when using the PMIS in other parts of the world, although test principles and procedures will remain the same. Although some but not all previous studies have used larger samples,^11,25,26 absence of power estimates is a major limitation of most reports of diagnostic tests.³⁰ Depression can lead to false-positive dementia diagnoses.³¹ Depressive symptoms are common in older adults and often coexist with cognitive impairment.³¹ Hence, individuals with depression were not excluded, which would reduce generalizability. PMIS scores were only weakly correlated with depressive symptoms.

The initial step in evaluating older adults presenting with cognitive complaints is to detect or exclude dementia regardless of subtype. Because the PMIS is a cognitive screening test, individuals who fail it should be referred for definitive diagnostic assessment. Alternatively, individuals scoring in the normal range could be asked to return for a repeat screen at a longer interval.