The Six-Item Screener and AD8 for the Detection of Cognitive Impairment in Geriatric Emergency Department Patients

Abstract

Study objective

We evaluate the diagnostic test characteristics of the Six-Item Screener and the AD8 to detect cognitive dysfunction in adults older than 65 years and using the emergency department (ED) for any reason.

Methods

We conducted an observational cross-sectional cohort study at a single academic urban university-affiliated hospital. Subjects were consenting, non-critically ill, English-speaking adults older than 65 years and receiving care in the ED. We quantitatively assessed the diagnostic test characteristics of the Six-Item Screener and AD8 by using the Mini-Mental State Examination score less than 24 as the criterion standard for cognitive dysfunction.

Results

The prevalence of cognitive dysfunction was 35%, but only 6% of charts noted a pre-existing deficit. The Six-Item Screener was superior to either the caregiver-administered AD8 or the patient-administered AD8 for the detection of cognitive dysfunction.

Conclusion

The Six-Item Screener was superior to the caregiver- or patient-administered AD8 to identify older adults at increased risk for occult cognitive dysfunction.

INTRODUCTION

Background and Importance

Aging baby boomers will cause emergency departments (EDs) to care for unprecedented numbers of geriatric adults during the next 3 decades.^1,2 The modern emergency care model is often poorly equipped to recognize and manage the unique constellation of aging pathology called geriatric syndromes.^3,4 Cognitive dysfunction is one such geriatric syndrome and includes mild cognitive impairment, delirium, and dementia. Multiple observational trials have demonstrated that emergency physicians and nurses often fail to recognize cognitive dysfunction.^5-9 Inpatient and outpatient physicians also fail to recognize impaired cognition.^9-11 In fact, more than 70% of ED patients with cognitive dysfunction lack a previous diagnosis of dementia.⁸

Prompt recognition of cognitive dysfunction is essential for high-quality geriatric emergency care for several reasons. First, the prevalence of cognitive impairment is 23% to 40% in older ED patients.^6,8,12-16 In the United States, 5.3 million persons experience Alzheimer’s dementia, at a health care cost of $148 billion per year.¹⁷ By 2050, 1 in 85 persons will live with Alzheimer’s dementia. If interventions could delay disease progression by just 1 year, 9.2 million fewer patients would require the expense of higher-level care.¹⁸ Second, unrecognized cognitive impairment may impede effective emergency care. Already, older adults use emergency care more frequently than younger populations and have longer lengths of stay while undergoing more diagnostic testing and more frequent admissions.^19-22 Despite this disproportionate allocation of resources, geriatric adults have higher rates of ED recidivism and frequently report not understanding their emergency care instructions.^19,22,23 Cognitive impairment may impede medical comprehension and accurate elicitation of details germane to the chief complaint.^20,23-25

Goal of This Investigation

The importance of cognitive dysfunction screening within emergency medicine has recently been underscored by the development of geriatric quality indicators that include testing for cognitive dysfunction as a minimal standard of care.²⁶ Furthermore, multiple stakeholders have also defined the recognition of cognitive impairment in older adults as a minimal core competency for emergency medicine residents.²⁷ However, one of the barriers to effective cognitive screening is the lack of brief, ED-valid screening instruments.^16,28 The Six-Item Screener (SIS) (Figure 1) is one instrument that can be administered to patients without the need for extra materials or equipment.²⁹ Unfortunately, previous multicenter trials have failed to validate the SIS as a sufficiently sensitive instrument in the ED.¹⁵ The AD8 (Figure 2) is another paperless instrument developed to screen for dementia through 8 questions administered to the patient or the caregiver but which has not been validated in ED settings.³⁰ The objective of this study was to evaluate the diagnostic test characteristics of the SIS and the AD8 in the ED to detect cognitive dysfunction in adults older than 65 years.

Figure 1.

Six-Item Screener.

Figure 2.

AD8.

MATERIALS AND METHODS

Study Design

This was a prospective, cross-sectional, convenience-sampling study in the ED of one urban academic medical center.

Setting

Barnes Jewish Hospital is a Level I trauma center academic teaching hospital in St. Louis, MO, with more than 90,000 total visits annually, 20% of which are by patients aged 65 years or older.

Selection of Participants

According to the availability of 4 research assistants from July 1, 2008, to April 20, 2009, all ED patients aged 65 years or older were approached for enrollment in a convenience sampling. Enrollment occurred in the ED on weekdays and weekends during equally distributed day, evening, and overnight shifts. Potential participants were approached by a research assistant with the approval of the emergency physician caring for the patient. After potential participants received a scripted description of the study that did not include details of the study hypothesis, written consent was obtained. When subjects were physically or mentally incapacitated as judged by caregivers or clinicians, proxy consent was obtained from informed caregivers. We excluded patients who received medications that may have affected their mental status during the testing period (narcotics, benzodiazepines, antiemetics), were too critically ill to participate, as judged by the attending emergency physician, were unable to consent or cooperate with data acquisition, did not speak English, or refused to complete the questioning. Psychoactive medications administered before or during the cognitive testing interval were noted by a research assistant review of the electronic medical record. Our institution uses a computer physician order entry system, so all medications prescribed are electronically documented, including a nursing time stamp when the agents or interventions are actually administered. The study was approved by the Barnes-Jewish Hospital Institutional Review Board, with informed consent.

Data Collection and Processing

For eligible and consenting patients, the physician evaluating the patient (either senior resident or attending physician) administered the SIS at least 30 minutes before or after trained research personnel had administered the Mini-Mental State Examination (MMSE), using a standardized data collection form. The 30-minute delay was to provide a washout period for the 3-item recall, although the SIS used different items for the recall than the MMSE (apple, table, penny). In the majority of cases, the MMSE was administered first because physicians were not readily available to complete the SIS. The order of administration of MMSE and SIS was not randomized, nor was it recorded for subsequent analysis.

During their evaluation, physicians rated the patient’s level of consciousness as alert and attentive (normal), inattentive, hyperalert, lethargic, stuporous, or comatose. The physicians also performed and scored the SIS during their routine history and examination. On completion of the study sheet, the physician placed it in an opaque envelope, sealed it, and recorded the time on the outside of the envelope. Trained research personnel performed the MMSE according to standard instructions.³¹ When a caregiver was present at the bedside, the caregiver was asked to complete the AD8 (cAD8) while the research assistant administered the MMSE. If no caregiver was present at enrollment, the patient was asked to complete the AD8 (pAD8). Both the physicians performing the SIS and the research personnel completing the MMSE were blinded to the result of the other test. After completion of the MMSE and SIS, the ED medical record was reviewed for documentation of Alzheimer’s and other types of dementia in the medical history section.

Primary Data Analysis

Analysis was conducted according to Standards for Reporting of Diagnostic Accuracy criteria with SPSS (version 16.0; SPSS, Inc., Chicago, IL) and MEDCALC (version 11.2.1.0; MedCalc Software, Mariakerke, Belgium).³² The criterion standard for cognitive dysfunction was an MMSE score less than or equal to 23. Standard operating characteristics of diagnostic tests were calculated, including sensitivity, specificity, likelihood ratios, and receiver operating characteristics curves with area under the curve and 95% confidence intervals (CIs). In addition to computing the sensitivity and specificity at each score for the SIS and AD8, the receiver operating characteristics curve was visually inspected to determine the optimal cutoff point for each instrument that would simultaneously maximize sensitivity and specificity. The SIS sensitivity in the preliminary validation study was 94%.¹³ Verifying this sensitivity with 5% range for sensitivity (ie, if the point estimate were 94%, then the true value would reside somewhere between 89% and 99% or SD 5%) and a baseline prevalence of cognitive dysfunction of 23% would require 377 subjects to be enrolled with complete data collection.³³

RESULTS

Characteristics of Study Subjects

Between June 2008 and April 2009, we approached 957 patients, excluded 586, and enrolled 371 subjects (Figure 3). A total of 52 patients did not complete the MMSE, SIS, or AD8 because they were discharged or admitted before the completion of data collection, so 319 subjects were included in this analysis.

Figure 3.

Flow diagram for patient enrollment.

Enrolled subjects had mean age of 76 years, and 58% were women. Whites represented 42% of the cohort, whereas blacks were 58%. The prevalence of cognitive dysfunction was 35.4% (113/319; 95% CI 31% to 41%). Although cognitive dysfunction may represent mild cognitive impairment, dementia, or delirium, only 6% (19/319) of subjects had dementia listed in the ED medical records. The AD8 was administered to caregivers in 41% of cases and to patients in 59% of cases.

Main Results

As demonstrated in Table 1, the sensitivity of the SIS was superior to that of the cAD8 or pAD8 (74% versus 63% versus 37%, respectively). In addition, the SIS area under the curve was superior to that of the cAD8 and pAD8 (Figure 4A and B). The SIS receiver operating characteristics curve in Figure 4A and B differ because the respective curves represent the subset with (n=132) and without (n=187) a caregiver present at the bedside when cognitive testing occurred. Evaluation of the receiver operating characteristics did not identify a cutoff point by which to define abnormal cognitive function superior to those previously reported (SIS ≥2 errors, AD8 ≥2 affirmative responses). In isolation, neither the positive-nor the negative-likelihood ratios for the SIS, cAD8, or pAD8 would significantly alter the posttest probability of cognitive dysfunction.³⁴ However, the combination of either an abnormal SIS or an abnormal cAD8 score demonstrated an increased sensitivity and specificity for cognitive dysfunction (Table 1). No combination of normal or abnormal SIS or cAD8 scores demonstrated better ability to discriminate normal from abnormal MMSE patients than the SIS alone (Table 2), although the combination of an abnormal SIS score and an abnormal cAD8 score had a positive likelihood ratio of 19.9 (95% CI 9.8 to 74.4). When compared with documented medical history, the MMSE, SIS, and cAD8 each identified an increased number of subjects with potential cognitive impairment (Table 3).

Table 1.

Diagnostic test characteristics of SIS and AD8.

Cognitive Test	Sensitivity, % (95% CI)	Specificity, % (95% CI)	Positive Likelihood Ratio (95% CI)	Negative Likelihood Ratio (95% CI)	Receiver Operating Characteristics Area Under the Curve (95% CI)
SIS	74 (68–80)	77 (74–80)	3.3 (2.5–4.1)	0.33 (0.25–0.44)	0.83 (0.78–0.87)
cAD8	63 (53–72)	79 (73–85)	3.0 (1.9–4.6)	0.44 (0.31–0.62)	0.74 (0.65–0.81)
pAD8	37 (28–46)	82 (77–86)	2.0 (1.1–3.3)	0.77 (0.63–0.93)	0.67 (0.60–0.74)
SIS+cAD8*	89 (80–95)	70 (63–73)	3.0 (2.3–3.6)	0.16 (0.07–0.30)

^*Abnormal SIS or abnormal cAD8 result.

Figure 4.

A, Receiver operating characteristic curve for pAD8* compared with SIS (n=187). B, Receiver operating characteristic curve for cAD8* compared with SIS (n=132).

*The ROC curves are significantly different (P = 0.002).

Table 2.

Diagnostic accuracy for various combinations of SIS and cAD8 (number of patients).

Cognitive Test	Abnormal MMSE* Result	Normal MMSE* Result
SIS *
Abnormal	86	48
Normal	29	159
cAD8 †
Abnormal	34	17
Normal	18	63
SIS or cAD8 abnormal
Yes	45	28
No (both normal)	6	53
SIS abnormal cAD8 normal
Yes	13	3
No	38	78
SIS normal cAD8 abnormal
Yes	7	15
No	44	66
SIS abnormal and cAD8 abnormal
Yes	25	2
No	26	79
SIS normal and cAD8 normal
Yes	6	61
No	45	20

^*N = 319.

^†N = 132.

Table 3.

Diagnostic accuracy for MMSE, SIS, and AD8 compared with documented medical history of dementia (number of patients).

Cognitive Test Result	Dementia Documented	No Dementia Documented
MMSE *
Abnormal	15	98
Normal	4	202
SIS *
Abnormal	15	115
Normal	4	185
cAD8 †
Abnormal	10	39
Normal	2	81

^*N = 319.

^†N = 132.

LIMITATIONS

This study has several limitations. First, the MMSE is an imperfect criterion standard for cognitive impairment. The MMSE may be less specific in poorly educated and lower socioeconomic strata and less sensitive among highly educated populations.^35-37 In addition, the MMSE is insensitive to mild cognitive impairment and does not differentiate dementia from delirium.^38,39 If the SIS or AD8 were in fact superior to the MMSE as a criterion standard for cognitive dysfunction, then false-positive and false-negative rates would be erroneously inflated, misleadingly diminishing the diagnostic properties of the SIS or AD8.^40-42 Furthermore, to differentiate dementia from delirium, a validated delirium screening tool such as the Confusion Assessment Method or the Confusion Assessment Method-ICU would need to be coadministered with the MMSE.^43,44 Alternative screening instruments such as the Montreal Cognitive Assessment are superior tools to identify mild cognitive impairment.³⁹ Future trials should assess the ability of brief screening instruments such as the SIS and cAD8 to detect mild cognitive impairment and delirium.

Second, this study was conducted at a single urban academic medical center. We excluded a substantial portion of patients who had received potentially sedating medications before enrollment, in addition to critically ill and non-English speaking subsets. We had a nonconsecutive convenience sampling of patients. Our results may not reflect the diagnostic performance of the SIS or AD8 in these excluded populations, although our analysis did not identify any significant differences in age or sex between enrolled and nonenrolled subjects.

Third, we failed to complete data collection on the a priori 377-subject sample size. However, the CIs for the SIS are within the preplanned 5% range. Our findings are also consistent with previous results with the SIS.¹⁵

Finally, we did not assess reliability of the SIS or AD8. Specifically, we cannot ascertain whether individual patients’ baseline cognitive status during their ED evaluation was confounded by their acute illness. Although the MMSE was administered before the SIS and AD8 in almost all cases, we cannot exclude a learning phenomenon for the 2 distinct 3-item recalls (MMSE and SIS) tested between the 2 instruments, which could affect observed diagnostic performance.^45,46 In addition, the 30-minute washout period between the MMSE and SIS may have permitted cognitive deterioration or improvement in acutely ill patients, which would consequently increase false-positive or false-negative rates, respectively. We also did not assess for potential selection or ascertainment biases between research assistants by analyzing patient characteristic differences across research assistants. However, previous research has suggested that ED-based testing for cognitive function is reproducible.⁴⁷ Furthermore, we have ongoing 3-week follow-up assessments under way to measure test-retest reliability for cognitive function in our population.

DISCUSSION

The sensitivity and specificity of the SIS to detect cognitive dysfunction in the ED are lower than initially reported.^13,29 A previous multicenter trial had suggested significant variability in the SIS sensitivity between sites, with the possibility of a learning phenomenon when the same 3-item recall was used for the MMSE and the SIS.^15,45,46 Our ED was site 2 in the previous multicenter SIS validation trial and had demonstrated SIS sensitivity of 60%. Our site had been the only one of 3 that used the identical 3-item recall for the SIS and MMSE. Because the MMSE was administered before the SIS, we offered a referenced hypothesis that subjects at our site learned the 3-item recall during criterion standard MMSE testing, resulting in better recall the second time with the SIS.¹⁵ If the hypothesis is correct, this learning phenomenon would increase false-negative results and decrease false-positive results, leading to decreased sensitivity and increased specificity, respectively. In support of this hypothesis, our site had the lowest sensitivity and highest specificity. In the current report, using unique 3-item recall items, the specificity is identical to that of the 2 other sites in the previous multicenter trial, and the sensitivity approaches their mean. The current study eliminates this bias by using distinct 3-item recall lists, although the sensitivity of the SIS would still fail to identify 25% of cognitively impaired patients.

The AD8 was developed to differentiate normal cognitive aging from mild dementia, using an informant-based interview.⁴⁸ In outpatient clinic populations, this instrument has demonstrated reasonable sensitivity (74%), specificity (86%), and intrarater reliability.^30,48,49 Combining the AD8 with performance-based measures has increased the sensitivity for the detection of cognitive impairment in patients referred to a dementia clinic.⁵⁰ The AD8 has not previously been tested in ED populations and offers several potential advantages over other screening instruments. The AD8 consists of 8 brief questions that can be read to or by subjects. It can be administered to either patients or their caregivers when patients are incapacitated or unavailable. The results from this single-center trial do not support the use of the cAD8 or pAD8. The cAD8 is significantly inferior to the SIS, with nonoverlapping CIs for the receiver operating characteristics area under the curve, although the cAD8 is superior to the pAD8 in the ED setting.

An important finding of this study is the underrecognition of cognitive dysfunction in ED patients. According to our chart review, we found that only 6% of subjects had recognized dementia before their screening, whereas 35.4% had an abnormal MMSE result. This confirms previous research demonstrating that emergency nurses and physicians often fail to recognize cognitive impairment.^5,8,9 Most do not routinely screen for cognitive dysfunction with validated instruments.⁵¹ Furthermore, inpatient and outpatient physicians often fail to identify cognitive impairment.^9,10

In assessing diagnostic instruments, researchers and clinicians must face the reality that the perfect test does not exist. For a diagnostic test to be clinically useful, identification of patients with disease (true-positive results) should provide an opportunity to treat illness. Because false-positive and false-negative results also occur with diagnostic testing, clinicians must weigh the potential harm related to more invasive confirmatory testing or treatment of these erroneous results.^52,53 How then do we define an acceptable level of diagnostic inaccuracy to appropriately balance the risks and harms associated with testing? For cognitive dysfunction, ED personnel fail to document recognition in the majority of cases, and only 6% of subjects in our trial presented with a history of dementia. If clinicians were to apply the SIS alone, 74% of patients with MMSE-defined cognitive impairment would be correctly identified within 1 minute, with a 23% false-positive rate. Although our research did not assess the potential benefits or adverse effects of cognitive screening, it is reasonable to surmise that with no cost-prohibitive criterion standard testing available in the ED, the “risks” of false-positive test results are minimal. In promptly recognizing a higher-risk subset of patients with the ED-friendly SIS, diagnostic good may outweigh the risk of harm while clinicians await an alternative cognitive dysfunction screening test for an increasingly geriatric society.⁵⁴

As demonstrated in Table 1, the combination of a normal SIS and cAD8 result significantly reduces the probability of cognitive dysfunction (negative likelihood ratio=0.16) compared with either instrument in isolation. However, using both instruments requires more time and training, with the additional need to find consenting caregivers to complete the AD8. More important, adding the AD8 to the cognitive evaluation of geriatric ED patients does not improve one’s ability to increase the probability of dementia or delirium (positive likelihood ratio 3.0 for the combination compared with 3.3 for SIS alone). Therefore, the cAD8 may not be worth the added effort because the goal is to recognize (not necessarily exclude) those at increased risk for cognitive dysfunction.

Screening tools and clinical decision aids always require external validation in populations distinct from those used to derive them.⁵⁵ Numerous examples of methodologically compelling tools that subsequently fail to be validated in unique patient groups have been recognized.^56-58 Screening tools are statistical models that may not apply to alternative populations because of overfitting data to the derived model or individual variable instability.⁵⁹ Several potential reasons could explain why the SIS and AD8 failed to be validated in our ED population. Patients and their caregivers in the ED present with acute or decompensated illnesses entailing labile emotions and chaotic information exchange. The increased prevalence of emotional and cognitive distracters, combined with diminished attentiveness associated with illness-related suffering, could reduce the accuracy and reliability of the patient and caregiver’s SIS and AD8 responses. This is precisely why such instruments need to be validated within the environments in which they could be used.^60,61 Diagnostic tests are therefore subjected to a hierarchic outcomes approach progressing from technical value to diagnostic accuracy to clinical outcome efficacy and societal efficacy.⁶²

Because the SIS evaluates only 2 domains (recall, orientation) of cognitive dysfunction, future assessment of screening tools in the ED should evaluate different or additional domains. Validating cognitive screening instruments that can be quickly administered without the need for paper, pencil, score cards, or specific equipment remains essential for ED screening efforts. A number of such instruments have been developed, but all await validation in the hectic ED environment.^16,63

The SIS is more sensitive than the AD8 in identifying geriatric ED patients at increased risk for cognitive dysfunction, using the MMSE as the criterion standard. Future trials should assess equipment-free screening instruments that incorporate additional domains while maintaining ED-appropriate brevity. In addition, dementia and delirium should be distinguished, as should mild cognitive impairment and early dementia, using appropriate criterion standards.

Editor’s Capsule Summary.

What is already known on this topic

Emergency physicians often fail to recognize cognitive dysfunction among emergency department (ED) patients older than 65 years.

What question this study addressed

Three hundred nineteen geriatric patients were screened for cognitive dysfunction, using the Six-Item Screener (SIS), the AD8 (caregiver-completed AD8 or patient-completed AD8), and the Mini-Mental State Examination (MMSE). The MMSE score was used as the criterion standard.

What this study adds to our knowledge

Cognitive dysfunction was observed in 35% of patients. Only 6% had pre-existing dysfunction noted in the medical record. The SIS was more sensitive than either AD8 but carried a 23% falsepositive rate.

How this is relevant to clinical practice

The SIS is a brief tool that may help physicians identify cognitive dysfunction among geriatric ED patients.