The Sensitivity and Specificity of Cognitive Screening Instruments to Detect Cognitive Impairment in Older Adults With Severe Psychiatric Illness

Abstract

Background

Older adults with severe psychiatric illness are often treated at community mental health centers (CMHCs) and these individuals commonly have numerous risk factors for cognitive impairment (CI). Brief cognitive screening instruments are frequently used to evaluate cognitive functioning in CMHCs, but the validity of these measures for detecting CI has not been adequately evaluated in this patient population.

Objectives

To determine the sensitivity and specificity of 2 cognitive screening measures (the Mini-Mental Status Examination [MMSE] and the Stroop Color and Word Test [SCWT]) for detecting CI in a sample of older adults with severe psychiatric illness.

Methods

Data were collected from 52 older adults receiving services at a CMHC. Diagnosis of CI was made by a neuropsychologist. Sensitivity and specificity coefficients for 2 cutoff scores for the MMSE and the SCWT were calculated.

Results

A cutoff score of 25 on the MMSE yielded a sensitivity of 43.3% and a specificity of 90.4% for detecting CI, whereas a cutoff score of 21 yielded sensitivity of 13.1% and 100% specificity. Using an age- and education-corrected scaled score (SS) on the SCWT falling at or below 7 as the criterion the SCWT had 88.8% sensitivity and 36.8% specificity, whereas a cutoff score of 5 or below yielded sensitivity of 59.2% and specificity of 57.8%.

Conclusions

Overall, the MMSE was found to be the more clinically useful cognitive screening tool for use in CMHC. Yet, because of the poor sensitivity of the MMSE for detecting CI in this patient population, alternative screening methods should be explored.

Introduction

Community mental health centers (CMHCs) represent an important avenue for the detection of cognitive impairment (CI) in older adults with severe psychiatric illness. Older adults who receive treatment at CMHCs often have severe chronic psychiatric illnesses, substance abuse histories, and significant medical comorbidies,^1,2 and each of these factors represent significant risk factors for CI in this patient population.^3–6 Additionally, as the risk of CI associated with many neurodegenerative diseases such as cerebrovascular disease and Alzheimer disease often increases with age,^7,8 the CMHC also offers a significant opportunity for early detection and treatment of CI associated with neurodegenerative diseases for older adults receiving care at these treatment settings.

The impact of CI on mental health interventions for older adults with severe psychiatric illness that are receiving treatment at CMHCs is not well understood but represents a critical area of future research. CMHCs commonly offer a variety of mental health interventions to older adults including psychotherapy, pharmacotherapy, case management, vocational training, and substance abuse treatments,^9–12 and each of these interventions may be negatively affected by CI. However, to date, determining the relationship of CI to treatment outcomes in this patient group has been difficult to evaluate, largely due to difficulties in the accurate diagnosis and documentation of CI in this patient population. One recent study by our research group suggests that CI was present in 60% of an elderly CMHC patient sample; however, CI was recognized or documented in less than a third of cases.¹³ Given successes in modifying treatment approaches for individuals with CI in other patient populations,^14–17 the improved recognition of CI represents a critical avenue to more effective mental health services in CMHCs.

Unlike specialty neurology clinics or geriatric psychiatry clinics, CMHCs do not typically have the resources, time, or trained staff to use comprehensive neuropsychological assessments to evaluate the cognitive functioning of their patients. Given these constraints, CMHCs often rely on brief cognitive screening measures, such as the Mini-Mental State Examination (MMSE),¹⁸ to detect CI. Unfortunately, the use of these brief cognitive screening measures to identify CI in older adults receiving treatment at CMHCs is not well known. The MMSE is one of the most commonly used screening measures of cognitive functioning, can be administered quickly, and has established psychometric properties including good interrater and test–retest reliabilities (ranging from r = .82 to r = .98).^18,19 Estimates of the degree to which the MMSE can be used to accurately identify CI vary depending on the population assessed and the cutoff score used. When using a commonly used cutoff score of 25, the sensitivity of the MMSE has been reported to be 82% with a specificity of 80% among older community dwelling adults.²⁰ With a lower cutoff score of 21, past researchers found sensitivity of 80% and specificity of 98%²¹ for detecting CI in community dwelling older adults. Yet, the use of the MMSE for detecting CI among individuals with severe psychiatric illness at CMHCs has not been explored, despite the widespread use of this measure in these settings. Furthermore, several studies have shown that the MMSE is largely insensitive to identifying mild CI^22–24 and that the MMSE may not be as useful for detecting CI in the presence of mood disorders.²⁵ As CI patients in CMHCs are often mild and many patients with CMHCs have mood disorders,¹³ each of these issues may influence the use of the MMSE in detecting CI in CMHCs.

Although not as commonly used as the MMSE, the Stroop Color and Word Test (SCWT)²⁶ has also been shown to be a sensitive measure of CI in older adults with neurodegenerative disease,^27,28 psychiatric disorders,^29–33 and substance abuse disorders.^34,35 Furthermore, results from a recent study suggest that among schizophrenia outpatients, performance on an abbreviated SCWT was superior to performance on the MMSE for detecting CI in this patient population.³⁶ In this investigation, the SCWT demonstrated a sensitivity coefficient of 74% and a specificity coefficient of 54% at optimal cutoff points in comparison to performance on the MMSE of 72% sensitivity and 56% specificity at optimal cutoff points. Given that the SCWT also has good test–retest reliability coefficients (r = .80)³⁷ and a relatively short administration time (approximately 4 minutes), these results suggest that the SCWT may be useful as a more widely adopted cognitive screening tool in CMHC settings.

The current study was conducted in order to evaluate the utility of using the MMSE and the SCWT to identify CI in an ethnically diverse sample of older adults with severe psychiatric illness receiving treatment at a CMHC. Furthermore, the administration time for the MMSE was assessed to evaluate the time burden of this measure for routine use in CMHCs as the MMSE, unlike the SCWT, has no limit on administration time.

Methods

Participants

Participants included older adults (ages 60 and older) recruited from a large community mental health agency in San Francisco. Of approximately 92 patients participating in day programming on a consistent basis over a 1-year period, 52 participants volunteered to participate in this study. Twenty-five participants were male; 64% were Caucasian, 14% were Asian, 14% were African American, 2% were Pacific Islanders, and 6% of participants identified as belonging to “other” ethnic groups. The mean age of the sample was 69.4 years (SD = 7.4) and the mean level of education was 13.4 years (SD = 3.1). Sixty-percent of the sample lived independently, 28% lived in board and care facilities, and 12% lived in supportive senior centers.

Study Procedures

All study procedures were approved by an Institutional Review Board (IRB) committee for human research. Participation in this study was voluntary. Patients were provided information about the study by posting fliers in the lobby of the community mental health facility and through brief announcements given by CMHC staff at the beginning of day programming. Interested individuals discussed the project with CMHC staff members and assessment appointments with research staff were subsequently made for individuals interested in participating in the study. All participants provided informed consent for their participation in this project. Neuropsychologists, psychologists, or trained research assistants administered an assessment battery including measures of cognitive functioning, depression, and substance abuse history. All assessments were conducted at the community mental health agency facility. Information obtained during research evaluations was not included in patients’ clinical record. Two identified staff members from the CMHC conducted medical chart reviews for participants to obtain current psychiatric diagnoses. Demographic information (age, gender, education, and marital status) were obtained through participant self-report. Not all participants completed all measures.

Measures

Mattis Dementia Rating Scale-2 (DRS-2)³⁸

The DRS-2 was utilized in this study to evaluate for the presence of clinically significant CI. The DRS-2 is a comprehensive measure of overall cognitive functioning for use with older adults and provides indices of cognitive functioning for the following cognitive domains: attention, initiation/perseveration (executive functioning), memory, construction (visuospatial processing), and conceptualization. The DRS-2 has established psychometric properties for each cognitive domain scale referenced to age-matched peers³⁹ and takes approximately 30 minutes to administer. Age- and education-corrected scaled scores (SSs) for the DRS total score are also available⁴⁰ and were used to calculate standard scores for the DRS total score for this study. The DRS has good psychometric properties with test–retest reliability of .97, split-half reliability of .90, and good construct and convergent validity.³⁸ The DRS has documented sensitivity and specificity for detecting CI for individuals with mild CI⁴¹ and dementia⁴² and is commonly used in studies of cognitive functioning in psychiatric populations.^43–45

Mini-Mental State Examination¹⁰

The MMSE is a commonly used 19-item assessment screening measure of mental status. The MMSE evaluates orientation, recall and recent memory, abstract thinking, attention and calculation, and objection identification. The MMSE has a maximum score of 30, with high scores representing better performance. Administration time for the MMSE is dependent on how quickly the respondent answers each question, and there is no time limit for this test. The administration time for the MMSE was recorded for each participant.

Stroop Color and Word Test⁴⁶

The SCWT is a measure of response inhibition and speed of information processing which has been shown to be a sensitive measure of cognitive dysfunction.^27,28 The SCWT measures the ability to selectively attend to the color of a printed word while ignoring the printed word itself. An age-corrected SS for the total number of correct responses on the color word trial was utilized as the outcome measure for this test.⁴⁷ The SCWT has a standardized administration time of 45 seconds for each trial regardless of performance, thus administration time for the SCWT was approximately 3 to 4 minutes, which includes time for providing instructions to participants.

Data Analytic Procedure

Diagnosis of CI was made by a neuropsychologist. CI was diagnosed for individuals with DRS-2 performance falling below the 10th percentile when referenced to age- and education-matched peers.⁴⁰ Descriptive statistics for all demographic and cognitive variables were calculated. Next, we identified the sensitivity and specificity of 2 cutoff scores for both of our cognitive screening instruments. For the MMSE, we used cutoff scores of 25 and 21, and for the SCWT, 2 cutoff scores based on age-corrected SSs of 7 and 5 were used.

Results

As reported previously, 92% of this sample had a primary psychiatric diagnosis and CI was exhibited by 60% of the sample.¹³ Participants included individuals with a diagnosis of major depressive disorder (35%), schizophrenia (20%), schizoaffective disorder (17%), bipolar disorder (12%), delusional disorder (6%), obsessive compulsive disorder (2%), and no primary diagnosis specified in the medical chart (8%). The mean DRS-2 score for the sample was 6.6 (SD = 3.8), the mean MMSE score was 26.1 (SD = 3.2), and the mean SCWT SS was 5.3 (SD = 3.2). The average administration time for the MMSE was 7.3 minutes (SD = 3.6). When using a cutoff score of 25, the MMSE had a sensitivity of 43.3% and a specificity of 90.4%, whereas a cutoff score of 21 yielded 13.1% sensitivity and 100% specificity for detecting CI. Using an age- and education-corrected SS on the SCWT falling at or below 7 as the criterion; the SCWT had 88.8% sensitivity and 36.8% specificity, whereas a cutoff score of 5 or below yielded sensitivity of 59.2% and specificity of 57.8%. Table 1 summarizes the results of sensitivity and specificity coefficients and positive and negative predictive values for each cognitive screening measure.

Table 1.

Comparison of Sensitivity and Specificity for the MMSE to the SCWT Using Different Cutoff Scores (n = 52)

	Sensitivity (%)	Specificity (%)	Positive Predictive Value (%)	Negative Predictive Value (%)
MMSE < 25	43.3	90.4	29.4	70.5
MMSE < 21	13.1	100	7.8	92.1
SCWT SS < 7	88.8	36.8	78.2	21.7
SCWT SS < 5	59.2	57.8	52.1	47.8

Abbreviations: MMSE, Mini-Mental State Examination; SCWT, Stroop Color and Word Test; SS, scaled score.

Discussion

The MMSE is often considered the gold standard of cognitive screening instruments and has been validated in many patient populations and research settings. Furthermore, the MMSE is generally considered to be a measure that can be administered by individuals with a wide range of professional training, and administration time is typically short. For these reasons, the MMSE is a widely utilized cognitive screening tool commonly employed in CHMC settings. In our sample, the MMSE demonstrated inadequate sensitivity, but good specificity, to detect CI, using both cutoff scores. When using a cutoff score of 25 on the MMSE as a cognitive screening tool in CMHC, clinicians would likely identify only approximately one third of individuals that have CI correctly but would also categorize 70% of cognitively intact individuals correctly. Using a cutoff score of 21 on the MMSE would result in 8% of patients with CI and 92% of cognitive intact participants being correctly identified. Therefore, using a cutoff score of 25 on the MMSE would be a better indicator of CI in a CMHC setting; however, clinicians should also be aware that individuals with CI would frequently be missed by the MMSE. These results are consistent with other studies that have found that when using cut scores on the MMSE in isolation, this measure is not a sensitive indicator of CI, particularly among individuals with milder cognitive deficits.^22–24 Furthermore, our results suggest that the relatively short administration time for the MMSE (7 minutes) would enable this measure to be used on a consistent basis in CMHC. Overall, given the ease of administration, wide availability, and generally short administration time, the MMSE possesses several advantages for use in CMHC. Yet, knowledge of the strengths and limitations of the MMSE, that is, the trade-off between sensitivity and specificity inherent in all screening measures, should be incorporated into training programs for staff at CMHC to ensure that maximal benefit is derived when using this measure to make clinical decisions.

Our results also suggest that the SCWT had good sensitivity for detecting CI when using a cutoff score of 7, but for both cutoff scores, the SCWT had inadequate specificity for identifying CI. When using a cutoff score of 7 on the SCWT, the measure correctly identified nearly 80% of individuals with CI but yielded too many false positives (nearly 80%) to be clinically useful in this sample. For this reason, we found that the SCWT is not likely a suitable cognitive screening instrument for CMHC settings; however, further study would be warranted to determine whether our results are replicated in other CMHC samples. In comparison to other previously published studies for which an abbreviated SCWT was used as a indicator of CI for individuals with schizophrenia,³⁶ our results yielded a higher sensitivity coefficient (89% vs 74%) but also a lower specificity coefficient (37% vs 56%), and we attribute these differences to both methodological differences between the 2 studies as well as differences with respect to the clinical characteristics of the sample. More specifically, in the Camozzato and Chaves study, an abbreviated version of the SCWT was used with a time to complete 10 rows as the outcome variable. In our own sample, we used an age-corrected SS as the outcome variable, using the standard clinical version of the SCWT that we believe yielded a higher sensitivity coefficient because of presumed improved psychometric properties of the longer, clinically validated measure. However, a direct comparison of the results between the 2 studies is also confounded by the difference in ages of the 2 samples (33 vs 69 years) and because our study included a much more heterogeneous sample of psychiatric disorders. We would attribute these differences as being significant factors in the reduced specificity of the SCWT in our sample. Although our results did not suggest that the SCWT offered significant advantages when compared to the MMSE to identify CI in this sample, future research is necessary to determine whether the SCWT can be used for this purpose in other clinical samples given the short administration time and good psychometric properties of this measure. Furthermore, given the sensitivity of the SCWT in identifying CI, it would appear that using this measure in combination with other brief screening measures may hold promise for use as one component of a short cognitive screening battery in CMHC populations.

The current study does not go without limitations. The most significant of these limitations is that our sample was relatively small and as such may not be representative of the larger CMHC population. Similarly, given the study design, we were not able to evaluate the clinical characteristics of individuals who declined to participate in this study. As such, we recognize that individuals experiencing cognitive difficulties or individuals with specific types of psychiatric illness may have been less likely to participate in this study, which further limits the generalizability of our results. Additionally, as psychiatric diagnoses were obtained from a medical chart review, we acknowledge that some individuals may have diagnosed incorrectly, which would further limit the generalizability of our findings to other patient groups. Similarly, our sample consisted of individuals with a variety of psychiatric diagnoses, including some individuals without a clearly documented mental health diagnosis, and as such our ability to comment on the sensitivity and specificity of cognitive screening measures within a specific diagnostic group is limited. Of note, we elected to retain the individuals without a specified psychiatric diagnosis in our sample as we felt that such individuals were representative of the CMHC population because they were eligible to receive services at the CMHC, despite the lack of a clearly documented diagnosis. However, despite these limitations, as our intent was to evaluate the clinical use of these measures in a specific treatment setting, that is, CMHC, we feel that the heterogeneity of the sample is representative of this patient group and can also be viewed as a strength of the study. An additional limitation of the study is that our diagnosis of CI was largely based on performance on another cognitive instrument and did not include a detailed clinical history or more extensive neuropsychological testing that would be necessary for a clinical diagnosis of CI or dementia. Our rationale for using the DRS-2 as a criterion measure for CI has been described previously,¹³ but the abbreviated assessment approach that was used in our research design does pose a significant methodological concern. Nevertheless, this criterion measure was selected due to the excellent normative data for age- and education-matched peers and validity for detecting CI in other populations including severe psychiatric illness. Furthermore, the DRS has been shown to be less influenced by ethnicity than other commonly used measures of cognitive functioning samples,⁴⁸ an important consideration given the ethnically diverse nature of our sample.

These limitations notwithstanding, to our knowledge, this study is the first to evaluate the sensitivity and specificity of cognitive screening instruments among individuals with severe psychiatric illness at a CMHC. We believe that our results are significant because despite the widespread use of the MMSE in CMHCs, very little is currently know about the clinical utility of this measure to identify CI in this patient population. Given the high rates of CI at CMHCs¹³, and the potential for CI to affect mental health treatment outcomes,^14,49–51 improving the accurate diagnosis of CI in CMHCs is a significant priority for future research. As CMHCs often do not have the resources to conduct comprehensive neuropsychological evaluations, identifying clinically useful screening instruments for CI in CMHC populations is, therefore, critical. Our results suggest that the MMSE can be used by CMHC staff to identify individuals at risk of CI and to make appropriate referrals, but using the MMSE in isolation for this purpose will only identify approximately one half of individuals with CI. As such, we feel that it is important that CMHC staff receive training on the inherent tension of using screening measures to correctly identifying individuals with CI relative to the risk of a false-negative diagnosis. Our findings are particularly salient now given that the MMSE is no longer available in the public domain, and as a result, many CMHCs are looking for alternatives to the MMSE to screen for CI.