Validation of the 6-Item Cognitive Impairment Test and the 4AT test for combined delirium and dementia screening in older Emergency Department attendees

Abstract

Background

screening for cognitive impairment in Emergency Department (ED) requires short, reliable tools.

Objective

to validate the 4AT and 6-Item Cognitive Impairment Test (6-CIT) for ED dementia and delirium screening.

Design

diagnostic accuracy study.

Setting/subjects

attendees aged ≥70 years in a tertiary care hospital’s ED.

Methods

trained researchers assessed participants using the Standardised Mini Mental State Examination, Delirium Rating Scale-Revised 98 and Informant Questionnaire on Cognitive Decline in the Elderly, informing ultimate expert diagnosis using Diagnostic and Statistical Manual of Mental Disorders (DSM-V) criteria for dementia and delirium (reference standards). Another researcher blindly screened each participant, within 3 h, using index tests 4AT and 6-CIT.

Result

of 419 participants (median age 77 years), 15.2% had delirium and 21.5% had dementia. For delirium detection, 4AT had positive predictive value (PPV) 0.68 (95% confidence intervals: 0.58–0.79) and negative predictive value (NPV) 0.99 (0.97–1.00). At a pre-specified 9/10 cut-off (9 is normal), 6-CIT had PPV 0.35 (0.27–0.44) and NPV 0.98 (0.95–0.99).

Importantly, 52% of participants had no family present. A novel algorithm for scoring 4AT item 4 where collateral history is unavailable (score 4 if items 2–3 score ≥1; score 0 if items 1–3 score is 0) proved reliable; PPV 0.65 (0.54–0.76) and NPV 0.99 (0.97–1.00). For dementia detection, 4AT had PPV 0.39 (0.32–0.46) and NPV 0.94 (0.89–0.96); 6-CIT had PPV 0.46 (0.37–0.55) and NPV 0.94 (0.90–0.97).

Conclusion

6-CIT and 4AT accurately exclude delirium and dementia in older ED attendees. 6-CIT does not require collateral history but has lower PPV for delirium.

Introduction

It is reported that 20–30% of older hospitalised patients have dementia, often undiagnosed, and frequently with superimposed delirium [1, 2]. Delirium is associated with longer hospital stays, increased morbidity and mortality, and higher discharge rate to long-term care [3, 4], but is unrecognised in acute settings in up to 76% of cases [3]. Of concern, missed delirium in the Emergency Department (ED) is associated with high risk of adverse outcomes [5].

The importance of cognitive assessment of older people in ED is recognised [6]. Some countries use financial incentives to encourage cognitive screening at the point of hospital admission. Validated, short cognitive screening tools could allow ED staff to identify older attendees with cognitive vulnerability, triggering appropriate care pathways and urgent assessment of those with possible delirium. A single tool to reliably identify possible delirium and dementia would be ideal. However, illness acuity in ED hinders cognitive assessment [4, 7, 8] and tool performance may be suboptimal here [9, 10]. Our study therefore aimed to assess the diagnostic accuracy of two candidate tools, 4AT and 6-Item Cognitive Impairment Test (6-CIT), for dementia and delirium detection in ED.

Methods

This study adheres to reporting standards for studies of diagnostic test accuracy in dementia (STARDdem [11]).

Setting/participants

We recruited a prospective, non-consecutive sample of adults ≥70 years, attending ED in a medium-sized, tertiary care hospital in Cork, Ireland (90 ED attendees per day; 14 per day ≥70 years). Researchers attended the ED Monday–Friday, 8AM–6PM, monitoring the real-time electronic ED database for potential participants. All ED attendees during these times were eligible. Exclusion criteria were: refusal, inability to consent and no family member to give assent, being actively drunk, severe intellectual disability, requiring medical isolation, poor English, medically unstable (resuscitation room or 1:1 nursing care) and prior study recruitment. Written consent was obtained, supplemented by proxy consent from nearest family member when patients could not give informed consent but assented to cognitive testing. Ethical approval was granted from the appropriate research ethics committee (Clinical Research Ethics Committee, University College Cork, Ireland).

Assessments

Our index tests were 4AT and 6-CIT. 4AT [12] is commonly used clinically, in the UK particularly, at point of access to hospital. It has been endorsed in Ireland for delirium screening in ED [13], but not yet validated in an ED population. It screens for delirium and cognitive impairment (CI), advantageously generating a score in ‘untestable’ patients. Item 1 assesses alertness, item 2 orientation (AMT-4) and item 3 attention (reciting ‘months of the year backwards’, MOTYB). Item 4 requires family or carer collateral information. Scores 1–3 suggest CI; scores ≥4 suggest delirium (i.e. defined cut-off 3/4); ‘0’ (with collateral information) indicates delirium or significant CI are unlikely [14].

6-CIT [15, 16] contains a logical memory item (5-item address), two attention items (counting backwards 20–1, and MOTYB) and three orientation questions (year, month, time). It thus has obvious potential as a combined delirium/ CI screening tool. 6-CIT can ‘quickly and reliably’ detect cognitive impairment in ED [8, 17]. It has been shown to detect prevalent delirium and predict incident delirium [18, 19]. It takes approximately 2 min to complete and, importantly, does not require collateral information. Values range from 0 to 28; higher scores indicate greater CI [16]. We used a 9/10 cut-off for dementia screening in this study, based on reported studies [15, 17, 20, 21].

Our reference standard was expert (geriatrician with special interest in delirium/dementia) delirium and dementia diagnosis using Diagnostic and Statistical Manual of Mental Disorders (DSM-V) criteria, using researcher-collected Standardised Mini Mental State Examination (SMMSE), Delirium Rating Scale-Revised 98 (DRS-R98) and Informant Questionnaire on Cognitive Decline in the Elderly (IQCODE) data, and demographic data, presenting complaint, and information from the GP referral letter or hospital notes about dementia diagnosis. The expert was blind to index test results.

SMMSE is a validated cognitive assessment tool. Administration and scoring guidelines were followed [22]. DRS-R98 is a validated delirium assessment tool, with 13 severity and 3 diagnostic items (maximum score: 46) [23]. Following the tool creator’s guidance, a total score ≥ 18 represented delirium, 12–17 represented sub-syndromal delirium, and 10–11 borderline sub-syndromal delirium.

Family members provided information on cognitive decline over the preceding 10 years, using the IQCODE tool, which rates cognitive changes from 1 (‘much improved’) to 5 (‘much worse’) [24]. Scores are averaged across the 16 items; scores ≥ 3.3 (community) to ≥3.8 (hospitalised, delirium) indicate possible dementia [25, 26]. Where collateral history was unavailable, to maximise usable data, the following assumption was made: SMMSE scores ≥27/30 implied ‘no dementia’ (n = 26), as a previous hospital-based study showed only 2% of older patients in hospital scoring ≥27/30 have dementia [2].

The two researchers were rigorously trained in assessments by the expert (S.T.), including simulated cases and directly observed patient interactions. Difficult-to-score items were recorded in detail and the expert advised on scoring.

Procedure

One researcher assessed participants for dementia and delirium using SMMSE and DRS-R98 (‘assessment’). Medical notes were reviewed for demographics, presentation time, existing dementia diagnosis and presentation route. The second researcher blindly screened each participant, within 3 h, using 4AT and 6-CIT, performed in random order (‘screening’). Assessment-screening order was also random. IQCODE information was collected by either researcher, after face-to-face cognitive testing was completed.

Analysis

Analyses were carried out in SPSS version 22 and MedCalc. Demographic and prevalence data were analysed using appropriate parametric or non-parametric tools. Diagnostic accuracy was assessed using receiver operating characteristic (ROC) curves, calculating Area Under the Curve (AUC), and 2 × 2 cross-tabulations of expert diagnosis and index tests to yield sensitivity, specificity, positive and negative predictive values, with 95% confidence intervals. As we used a priori index test cut-offs, we did not correct for multiple analyses. Post hoc exploratory cut-offs are clearly identified as such.

Results

During the study period, June to November 2015, 706 older patients presented to ED (Figure 1); 419 were recruited. Patients were assessed within a median 1.8 h of presentation (IQR = 7.7 h). Approximately half were female (50.8%); median age was 77 years; 91.1% presented from home; 24.6% presented by ambulance. Just 5.6% (21/400) had a dementia diagnosis documented in their referral letter or hospital notes. The detailed assessment was carried out first in 49.4% of cases, with a median 15 min between assessment and screening (range: 5 min to 2.8 h). All instruments were completed in 330 cases. The following numbers were attained for individual instruments: SMMSE (n = 405), DRS-R98 (attempted n = 411; usable n = 388 cases (collateral available)), 4 AT (attempted n = 415; 360 had collateral available), 6-CIT (n = 405) and IQCODE (n = 351). Missing direct patient testing data was due to refusal to continue (n = 6), medical status (n = 11) or discharge (n = 5). Family were present for collateral history in only 201 cases (48%). Follow-up telephone IQCODE data were missing (n = 68) due to relatives not answering the telephone (n = 17), incorrect number (n = 24) or patient refusal to allow contact (n = 27). Of note, 18.8% of relatives were only contactable that evening or the following day, sometimes only after repeated attempts.

Figure 1.

4AT (index test) compared to expert diagnosis (reference standard).

Dementia status was determined in 381 cases (91%), of whom 21.5% (n = 82) had dementia. Delirium status was determined in 389 (93%), and 15.2% (n = 59) had delirium. Of those with dementia, 49.4% had superimposed delirium; 72.2% of all delirium was superimposed on dementia. Ambulance cases had significantly more dementia (35.2%, P < 0.001) and delirium (28.1%, P < 0.001). Of those aged ≥80, 28.4% had dementia (P < 0.01 compared to < 80 years), and 23.5% delirium (P < 0.001).

Dementia screening

4AT

4AT was performed in 415 patients, of whom expert dementia diagnosis was available in 381 (Figure 1). The AUC for dementia detection was 0.83. 4AT score was ‘0’ in 211 cases (50.8%); 13/200 with 4AT ‘0’ had dementia (6.5%). 4AT score was 1–3 in 101 cases, of whom 22.8% had dementia. Only 10% scoring ‘1’ had dementia, versus 15/23 scoring 2/3. Using a ‘sensitive’ 0/1 cut-off (0 = normal, ≥1 = dementia), sensitivity for dementia detection was 0.84 (95% confidence interval: 0.74–0.91) and specificity 0.63 (0.57–0.69). A second cut-off of 1/2 was explored (i.e. 0 or 1 was normal, Table 1). This reduced the number potentially requiring formal dementia or delirium assessments from 50% to 30%, with PPV and NPV, at 0.61 and 0.92, respectively. Using a ‘specific’ 4/5 cut-off, PPV improved (0.67) but NPV suffered (0.87) (Table 1).

Table 1.

4-AT diagnostic accuracy for dementia and delirium detection

Cut-off	Sensitivity (confidence intervals)	Specificity (confidence intervals)	PPV (confidence intervals)	NPV (confidence intervals)
4AT dementia detection (n = 415)
0/1	0.84 (0.74–0.91)	0.63 (0.57–0.69)	0.39 (0.32–0.46)	0.94 (0.89–0.96)
1/2	0.74 (0.64–0.83)	0.87 (0.82–0.90)	0.61 (0.51–0.71)	0.92 (0.87–0.95)
4/5	0.52 (0.41–0.64)	0.93 (0.89–0.96)	0.67 (0.54–0.78)	0.87 (0.83–0.91)
4AT delirium detection (cut-off 3/4; n = 415 but item 4 (collateral) missing in 55 cases)
4AT: only cases with collateral history (n = 350)	0.93 (0.83–0.98)	0.91 (0.88–0.94)	0.68 (0.57–0.78)	0.99 (0.96–1.00)
4AT cases with collateral: dementia cohort (n = 78)	0.92 (0.79–0.98)	0.79 (0.64–0.91)	0.82 (0.67–0.92)	0.91 (0.76–0.98)
4AT using algorithm* for missing collateral (n = 378)	0.93 (0.84–0.98)	0.91 (0.88–0.94)	0.65 (0.54–0.76)	0.99 (0.97–1.00)

*Algorithm:

a) If 4AT items 1–3 are all scored as 0, and there are no documented or observed hallucinations, fluctuations, etc., during assessment, then item 4 is scored as 0.

b) If 4AT items 2–3 are ≥1, item 4 is scored as 4 (item 1 if abnormal is scored as 4, which is above the threshold for screening positive for delirium).

6-CIT

For dementia screening, a pre-specified 6-CIT 9/10 cut-off was used. In total, 405 patients completed assessments (supplemental data), of whom 36.8% screened positive. Of 368 patients with 6-CIT data and expert diagnosis, 6-CIT mis-categorised dementia in 22.8% of cases; 70 were incorrectly categorised as dementia, and 14 cases missed. Thus 6-CIT had sensitivity of 0.81 (0.70–0.89) and specificity of 0.76 (0.71–0.81). In this cohort, a slightly better cut-off for dementia screening was 8/9 (Table 2).

Table 2.

6-CIT diagnostic accuracy for dementia and delirium detection at various cut-offs

6-CIT cut-off	Sensitivity (confidence intervals)	Specificity (confidence intervals)	PPV (confidence intervals)	NPV (confidence intervals)
6-CIT dementia detection (n = 368)
3/4	0.93 (0.85–0.98)	0.39 (0.34–0.45)	0.28 (0.22–0.34)	0.96 (0.91–0.99)
4/5	0.88 (0.78–0.94)	0.55 (0.49–0.61)	0.33 (0.26–0.40)	0.95 (0.90–0.98
5/6	0.87 (0.78–0.94)	0.56 (0.50–0.62)	0.35 (0.27–0.41)	0.95 (0.90–0.98)
6/7	0.86 (0.77–0.93)	0.67 (0.61–0.72)	0.40 (0.32–0.48)	0.95 (0.91–0.98)
7/8	0.86 (0.77–0.93)	0.67 (0.62–0.73)	0.40 (0.32–0.48)	0.95 (0.91–0.98)
8/9a	0.84 (0.73–0.91)	0.76 (0.71–0.81)	0.47 (0.38–0.55)	0.95 (0.91–0.97)
9/10b	0.81 (0.70–0.89)	0.76 (0.71–0.81)	0.46 (0.37–0.55)	0.94 (0.90–0.97)
10/11	0.77 (0.66–0.86)	0.82 (0.77–0.86)	0.52 (0.42–0.61)	0.93 (0.90–0.96)
11/12	0.73 (0.61–0.83	0.83 (0.79–0.87)	0.52 (0.42–0.62)	0.92 (0.89–0.95)
12/13	0.69 (0.57–0.79)	0.81 (0.75–0.87)	0.59 (0.50–0.66)	0.87 (0.83–0.91)
13/14	0.68 (0.56–0.78)	0.89 (0.85–0.92)	0.61 (0.52–0.69)	0.92 (0.87–0.94)
6CIT delirium detection (n = 378)
7/8	0.94 (0.84–0.99)	0.64 (0.59–0.69)	0.30 (0.23–0.38)	0.99 (0.96–1.00)
8/9	0.91 (0.79–0.97)	0.73 (0.67–0.77)	0.35 (0.27–0.44)	0.98 (0.95–0.99)
9/10b	0.89 (0.77–0.96)	0.74 (0.68–0.78)	0.35 (0.27–0.44)	0.98 (0.95–0.99)
10/11	0.87 (0.75–0.95)	0.79 (0.74–0.83)	0.40 (0.31–0.50)	0.97 (0.95–0.99)
11/12	0.85 (0.72–0.93)	0.81 (0.77–0.85)	0.42 (0.33–0.52)	0.97 (0.94–0.99)
12/13	0.83 (0.70–0.92)	0.86 (0.81–0.89)	0.48 (0.38–0.59)	0.97 (0.94–0.99)
13/14a	0.83 (0.70–0.92)	0.87 (0.83–0.91)	0.51 (0.40–0.62)	0.97 (0.94–0.99)
14/15	0.74 (0.60–0.85)	0.90 (0.86–0.93)	0.53 (0.41–0.65)	0.95 (0.92–0.98)

^aPost hoc optimal cut-off for dementia and delirium.

^bPre-specified cut-off for dementia and delirium.

To summarise, 6-CIT and 4AT perform similarly for dementia screening in an older ED population, using optimal cut-offs based on this data.

Delirium screening

4AT

4AT was commenced in 415 patients, but completed in only 360, despite exhaustive attempts to contact relatives, and indicated possible delirium in 23.1% (83/360) (Figure 1). In total, 350 patients had complete 4AT data plus expert delirium diagnosis; of these, 68.4% screening positive had delirium (i.e. 4AT≥4). The sensitivity was 0.93 (0.83–0.98) and specificity was 0.91 (0.88–0.94). In the cohort with dementia (n = 78), sensitivity and specificity remained strong (Table 1).

We also assessed accuracy where collateral history was unavailable, using the following algorithm: if items 1–3 were normal and the patient had no hallucinations, delusions or fluctuations during assessment, or reported by ED staff, then item 4 was scored ‘0’; if items 2–3 were abnormal (total score 1–3), then item 4 was scored ‘4’ (to avoid missed diagnosis). Using this algorithm (n = 415 now with usable data), 4AT indicated possible delirium in 24.8% (103/415). Of the 386 participants with expert diagnosis and algorithm-aided 4AT data, 65.5% screening positive had delirium. The sensitivity and specificity for delirium detection was unaffected by the algorithm (Table 1). DRS-R98 scores in discordant cases are presented as Supplementary data.

6-CIT

6-CIT showed good diagnostic accuracy for delirium detection (n = 405 with complete data; 378 of these had expert diagnosis). Using the same pre-specified cut-off of 9/10 as for dementia diagnosis, 133/378 (35.2%) screened positive for delirium and 35.3% of these had delirium. 6-CIT sensitivity was 0.89 (0.77–0.96) and specificity 0.74 (0.68–0.78) (Table 2). However, the optimal cut-off for delirium detection was 13/14 (Table 1), with NPV 0.97 and PPV 0.51. DRS-R98 scores in discordant cases are presented as Supplementary data.

To summarise, 4AT effectively rules out delirium (NPV 0.99). In the 23% of older people who screen positive and require formal assessment, almost 70% have delirium, supporting its use in delirium screening. 6-CIT also performs well in delirium screening, again effectively ruling out delirium at a range of cut-offs (NPV 0.97). Using the optimal cut-off of 13/14, 22.8% screen positive; of these, just over 50% have delirium.

Discussion

Our study demonstrates that dementia and delirium are prevalent in older people presenting to ED, at 21.5 and 15%, respectively. Similar to our previous finding in 606 hospitalised older people across six regionally clustered hospitals [1], approximately half of older people in ED with dementia have delirium, and most delirium is superimposed on dementia [1]. Other studies report similar dementia-delirium dependencies. Thus, delirium screening tools have to perform well in people with dementia.

We found that 4AT scores of 0 or 1 effectively ‘rule out’ dementia; only 30% of 4AT-screened older ED attendees require further cognitive assessment, mainly for suspected delirium (7.5% screened positive for CI). Reducing the cut-off to 0/1 improves sensitivity, but increases the workload (50% now require further testing) and reduces the PPV. The benefit of higher detection rates may be attenuated if clinical staff find it onerous to perform additional assessments for little perceived gain. 4AT has little prior investigation for CI screening, despite clinical use across diverse settings. In 111 acute stroke patients, sensitivity and specificity were 0.86 and 0.78 respectively, compared to a Montreal Cognitive Assessment score < 27/30 [27]. We found that 6-CIT with cut-off 8/9 or 9/10 can also exclude dementia (NPV 0.95 and 0.94, respectively) but PPV is low. A 13/14 cut-off seems better in clinical practice to reduce unnecessary workload from formal assessment of false negative cases. This cut-off has equal accuracy to 4AT (both PPV 0.61, NPV 0.92).

Our results support the clinical use of 4AT to quickly and accurately detect delirium (PPV 0.68; NPV 0.99). Our results reflect the stroke patient study, where 4AT demonstrated excellent diagnostic accuracy (Confusion Assessment Method (CAM) used as reference standard for delirium diagnosis), PPV 0.43, NPV 1.0 [27]. In a mixed acute and rehabilitation older hospital population (n = 234), using DSM-IV-TR criteria (incorporating CAM data), 4AT’s sensitivity was 0.89 and specificity 0.84; AUC 0.92 [14]. More recent studies report 4AT sensitivity as 0.87 [28] and specificity 0.69–0.8 [29] in geriatric and orthopaedic populations. Dementia adversely affects 4AT accuracy, in our study and others [14, 28].

We explored 6-CIT accuracy for delirium detection, initially using 9/10 cut-off, as for dementia screening, leading to 92/378 miscategorised cases. Many ‘false positive’ cases had borderline or definite subsyndromal delirium, indicating 6-CIT may not discriminate between syndromal and sub-syndromal delirium. However, patients with sub-syndromal delirium are at high risk of developing delirium. In a separate cohort of 191 older medical in-patients, we found that a 6-CIT 8/9 cut-off on admission had NPV 91.2%, and PPV 59.2% for prevalent delirium [18]. Admission 6-CIT score predicted incident delirium [19], while 6-CIT remained stable in a study subset of 20 non-delirious in-patients, assessed twice daily for 1 week [30], suggesting a potential role for 6-CIT in repeated cognitive testing during hospitalisation, where altered 6-CIT score suggests delirium onset.

As 6-CIT has not previously been studied in delirium assessment outside our group, we explored several cut-offs in our ED population. 6-CIT performed best at a 13/14 cut-off, effectively ruling out delirium (NPV 0.97); in the 24% screening positive, only 50% had delirium. This is inferior to 4AT accuracy for delirium. Thus 4AT out-performs 6-CIT for delirium screening, if properly used (i.e. collateral history is sought and available). In a recent Italian study, 6-CIT items suggestive of delirium were highly correlated with in-hospital mortality (P < 0.0001) among acute medical/surgical in-patients (n = 2521)[14]. 6-CIT accuracy for delirium detection was not evaluated. We recommend further studies establish 6-CIT cut-offs for delirium.

In this study, measures to reduce bias included blinding of the second researcher and expert, random order of assessment versus screening tests, and performance in rapid succession, limiting the effect of fluctuations. The large sample size generated tight confidence intervals for tool performance measures. Importantly, this study used expert diagnosis, superior to using another cognitive test as reference standard, although this relied on cognitive and collateral informant data collected by highly trained researchers. This study did not explore tool accuracy for detecting MCI, which is common in hospitalised older patients, including those with delirium [25].

There are challenges to screening for cognitive problems when patients are acutely unwell, worried or anxious. The ED environment presents particular issues such as a lack of privacy and noise, potentially affecting cognitive test performance. Nevertheless, detecting CI is important to initiate timely treatment plans, including enhanced delirium prevention measures. Most importantly, cognitive assessment is vital to detect prevalent delirium. We present here important accuracy data on cognitive screening in ED, including a potential 4AT algorithm for use when collateral history is unavailable. Future research needs to explore accuracy when ED staff perform the cognitive screening, particularly exploring the effect of unavailable collateral on 4AT accuracy.

Key points.

• Dementia and delirium are common in older people attending Emergency Departments (ED), and are frequently co-morbid.

• 4AT and 6-Item Cognitive Impairment Test (6-CIT) can both accurately exclude delirium, but 6-CIT generates more false-positives.

• A novel algorithm for scoring 4AT in the absence of collateral history (a potential problem with 4AT) performed well in this emergency department (ED).

• 4AT and 6-Item Cognitive Impairment Test (6-CIT) exclude dementia accurately in an older Emergency Department (ED) population; ‘resource efficient’ cut-offs are 1/2, and 13/14.

Supplementary data

Supplementary data mentioned in the text are available to subscribers in Age and Ageing online.

Conflict of interest

None.

Funding

None.