Validation of the Confusion Assessment Method for the Intensive Care Unit in Older Emergency Department Patients

Abstract

Objectives

In the emergency department (ED), health care providers miss delirium approximately 75% of the time, because they do not routinely screen for this syndrome. The Confusion Assessment Method for the Intensive Care Unit (CAM-ICU) is a brief (<1 minute) delirium assessment that may be feasible for use in the ED. The study objective was to determine its validity and reliability in older ED patients.

Methods

In this prospective observational cohort study, patients aged 65 years or older were enrolled at an academic, tertiary care ED from July 2009 to February 2012. Research assistants (RAs) and an emergency physician (EP) performed the CAM-ICU. The reference standard for delirium was a comprehensive (~30 minutes) psychiatrist assessment using the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, Text Revision criteria. All assessments were blinded to each other and were conducted within 3 hours. Sensitivities, specificities, and likelihood ratios were calculated for both the EP and the RAs using the psychiatrist’s assessment as the reference standard. Kappa values between the EP and RAs were also calculated to measure reliability.

Results

Of 406 patients enrolled, 50 (12.3%) had delirium. The median age was 73.5 years old (interquartile range [IQR] = 69 to 80 years), 202 (49.8%) were female, and 57 (14.0%) were nonwhite. The CAM-ICU’s sensitivities were 72.0% (95% confidence interval [CI] = 58.3% to 82.5%) and 68.0% (95% CI = 54.2% to 79.2%) in the EP and RAs, respectively. The CAM-ICU’s specificity was 98.6% (95% CI = 96.8% to 99.4%) for both raters. The negative likelihood ratios (LR–) were 0.28 (95% CI = 0.18 to 0.44) and 0.32 (95% CI = 0.22 to 0.49) in the EP and RAs, respectively. The positive likelihood ratios (LR+) were 51.3 (95% CI = 21.1 to 124.5) and 48.4 (95% CI = 19.9 to 118.0), respectively. The kappa between the EP and RAs was 0.92 (95% CI = 0.85 to 0.98), indicating excellent interobserver reliability.

Conclusions

In older ED patients, the CAM-ICU is highly specific, and a positive test is nearly diagnostic for delirium when used by both RAs and EPs. However, the CAM-ICU’s sensitivity was modest, and a negative test decreased the likelihood of delirium by a small amount. The consequences of a false-negative CAM-ICU are unknown and deserve further study.

Delirium is an underrecognized public health problem that occurs in 8% to 10% of older emergency department (ED) patients, affecting approximately 1.5 million older ED patients each year in the United States alone.^1–4 This form of acute brain failure is a significant threat to their quality of life and is associated with higher death rates⁵ and accelerated functional and cognitive decline.^6–11 Furthermore, delirium may compromise patient safety as these patients are less likely to provide an accurate reason why they are in the ED.¹² This may lead to inadequate diagnostic workups and delays in the diagnosis of their underlying medical illness.¹³ If discharged, they are less likely to comprehend their discharge instructions, and this may lead to noncompliance.^12,14

Emergency physicians (EPs) miss delirium in 57% to 83% of cases,^{1–3,15–18} because they do not routinely screen for this syndrome.^3,19 This has been described as a medical error, and the lack of recognition may have downstream effects on clinical care.²⁰ A significant barrier to recognizing delirium in the ED is the absence of brief (<2 minutes) delirium assessments tailored for the busy and fast-paced ED environment. Most currently available delirium assessments take longer than 5 minutes to complete and may not be feasible for the ED.²¹

The Confusion Assessment Method for the Intensive Care Unit (CAM-ICU, Figure 1) is a brief delirium assessment that takes less than 1 minute to perform and may have utility in the ED.²² It incorporates brief neuropsychiatric assessments that limit the need for clinical judgment and increase its ease of use. In an ICU population, the CAM-ICU has excellent sensitivity (93% to 100%), specificity (89% to 100%), and inter-rater reliability between physicians and nurses (kappa = 0.84 to 0.96).^23,24 Although the CAM-ICU has already been used in several ED studies,^3,12,25–28 its validity in older ED patients remains unknown. Older ED patients, in general, have lower severities of illness than an ICU population. As a result, we sought to determine the diagnostic performance and reliability of the CAM-ICU performed by physicians and nonphysicians for older ED patients while using a psychiatrist’s assessment as the reference standard.

Figure 1.

CAM-ICU flow sheet. Courtesy of Vanderbilt University, Nashville, TN, 2002. Used with permission. CAM-ICU = Confusion Assessment Method for the Intensive Care Unit; RA = research assistant.

METHODS

Study Design

This was a prospective observational study. The local institutional review board (IRB) reviewed and approved this study. Informed consent was obtained whenever possible, but a waiver of consent was provided for patients who lacked capacity and a power of attorney was not immediately available in the ED or by telephone. Although this was not required by the IRB, the research team usually notified the treating physician if the patient was delirious as diagnosed by the psychiatrist, especially in those with subtle symptoms or if discharge was likely. The sponsors of this study had no role in the design, methods, subject recruitment, data collection, analysis, or preparation of this paper.

Study Setting and Population

This study was conducted at a tertiary care, academic ED with an annual census of approximately 57,000 visits. A convenience sample of patients was enrolled from July 2009 to February 2012, Monday through Friday between 8 a.m. and 4 p.m.; the enrollment window was based on the psychiatrists’ availability. Because the psychiatrists’ assessments were extensive, enrollment was limited to one patient per day. Patients were included if they were aged 65 years or older, in the ED for less than 12 hours at the time of enrollment, and not in hallway beds. The 12-hour cutoff was set to include patients who presented in the evening and early morning hours. Patients in the hallway beds were not included because of the high level of ambient noise in these areas. Performing these cognitive assessments, especially lengthy psychiatric evaluations, would have been difficult in this setting. Patients were excluded if they refused consent, were not English-speaking, were previously enrolled, were deaf or blind, were comatose, were nonverbal or unable to follow simple commands prior to their acute illness, or did not complete all of the study assessments. Comatose patients were excluded because a patient must be arousable to verbal stimuli to assess for delirium.²⁹ Patients who were non-verbal or unable to follow simple commands prior to their acute illnesses were identified by surrogate interview or medical record review. These patients were considered to have end-stage dementia and diagnosing delirium in these patients can be challenging, even for a psychiatrist. During the enrollment window, research assistants (RAs) screened patients for the inclusion criteria using the ED electronic whiteboard, which provided each patient’s age, location, and length of stay.³⁰ The RAs approached those who met inclusion criteria and then determined the presence of any exclusion criteria. The first patient who met eligibility criteria was enrolled for that day.

Study Protocol

Research assistants performed the prospective data collection and served as one of the CAM-ICU raters. They were college graduates, emergency medical technicians, and paramedics. Prior to the study, the RAs underwent a training session that took approximately 6 to 8 hours given by the principal investigator. They were given didactic lectures on delirium epidemiology, the study protocol, the informed consent process, and the CAM-ICU. They also studied the CAM-ICU training manual, watched live patient demonstrations of the CAM-ICU, and practiced assessment administration using simulated and actual patients.

The RAs performed the ED screening and determine patient eligibility. After eligible patients were enrolled, the RA and an EP (principal investigator JHH) performed the CAM-ICU at the same time. One person performed the assessment while the other person observed the patient; both raters completed the CAM-ICU case report forms independently. This method of interobserver reliability was chosen to avoid learning on the part of the patient, which would have confounded the delirium assessments. This also minimized other problems intrinsic to test–retest reliability comparisons, such as a sudden change in the patient’s mental status if significant time occurred between ratings.³¹ One of three consultation-liaison psychiatrists served as the reference standard for delirium. These psychiatrists had an average of 11 years of clinical experience, and diagnosing delirium was a routine part of their daily clinical practice. The psychiatrist’s reference standard evaluation was performed within 3 hours of the CAM-ICU. All assessors were blinded to each other.

Measurements

The CAM-ICU (Figure 1) is based on the Confusion Assessment Method developed by Inouye et al.,³² but uses objective assessments with prespecified cutoffs to determine the presence of inattention and disorganized thinking. To test for inattention (Feature 2), the CAM-ICU uses the Attention Screening Examination (ASE), which has auditory and visual components. The auditory component uses the Vigilance A letter test, which asks the patient to squeeze every time he or she hears the letter “A”;³³ a series of 10 letters (“SAVEAHAART”) is given every 3 seconds. The visual component uses a picture recognition test; the patient is initially shown five simple pictures at 3-second intervals. Then the patient is shown 10 pictures and must identify which pictures were seen previously. If a patient makes three or more errors on the ASE letter or picture component, then the patient is considered to be inattentive. If the patient is unable or refuses to perform either ASE component, then the patient is also considered to be positive for inattention. To assess for altered level of consciousness (Feature 3), the CAM-ICU uses the Richmond Agitation Sedation Scale (RASS) to quantify altered level of consciousness.³⁴ This scales ranges from –5 (comatose) to +4 (combative). A patient with a RASS other than 0 (alert, normal level of consciousness) is considered to be Feature 3 positive. To test for disorganized thinking (Feature 4), the rater asks four yes or no questions and asks the patient to perform a simple command. A patient who makes two or more errors is considered to have disorganized thinking. The CAM-ICU is positive if a patient has both altered mental status or fluctuating course (Feature 1) and inattention (Feature 2) and either altered level of consciousness (Feature 3) or disorganized thinking (Feature 4).

The reference standard for delirium was a consultation-liaison psychiatrist’s assessment using Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, Text Revision (DSM-IV-TR) criteria.³⁵ Their patient evaluations were comprehensive, and they used all means of patient evaluation and testing, as well as data gathered from those who best understood the patient’s current mental status (e.g., the patient’s surrogates, physicians, and nurses). They routinely performed an extensive battery of cognitive tests at the bedside; performed a focused neurologic examination; and evaluated for affective lability, hallucinations, and arousal level. Confrontational naming, proverb interpretation or similarities, and assessments for apraxias were also performed at the discretion of the reference psychiatrists, especially if the diagnosis of delirium was inconclusive.

Medical record review was performed to measure the Charlson Comorbidity Index, which is a weighted index that takes into account the number and seriousness of preexisting comorbid conditions.³⁶ To quantify severity of illness, the Emergency Severity Index (ESI) was recorded from the nurse’s triage note. The ESI is five-level triage algorithm that stratifies patients from level 1 (most urgent) to level 5 (least urgent) on the basis of acuity and resource needs.³⁷ We also calculated the Acute Physiology Score (APS) portion of the Acute Physiology and Chronic Health Evaluation II.³⁸ The APS is a continuous variable based on the initial values of 12 routine physiologic measurements, and higher scores indicate higher severities of illness. Dementia was also collected from the electronic medical record. The medical record review was initially performed by an RA, but was double-checked by the principle investigator for accuracy.

Data Analysis

Measures of central tendency and dispersion for continuous variables were reported as medians and interquartile ranges (IQRs). Categorical variables were reported as absolute numbers and proportions. Sensitivities, specificities, positive likelihood ratios (LR+), and negative likelihood ratios (LR–) with their 95% confidence intervals (CIs) were calculated for both the EP and the RA using the psychiatrist’s DSM-IV-TR assessment as the reference standard.³⁹ Kappa statistics were also calculated to measure interobserver reliability between the EP and RA.

We performed a secondary analysis to determine if age, dementia documented in the medical record, and severity of illness affected the RAs’ CAM-ICU diagnostic performance using multivariable logistic regression.⁴⁰ Based on our review of the literature and expert opinion, we hypothesized that these variables were most likely to cause spectrum bias.^41–43 We chose the RAs’ CAM-ICU to maximize the generalizability of our secondary analysis. Previous validation studies conducted in non–critically ill patients used RAs and resident physicians to perform the CAM-ICU.^42,43 In the clinical setting, the CAM-ICU is also usually performed by bedside nurses.

Dementia documented in the medical record was used instead of a formal cognitive assessment for three reasons. First, we wanted to mimic real-world conditions where dementia screening is not routinely performed in the ED. Second, many delirium studies rely on assessments such as the Informant Questionnaire on Cognitive Decline in the Elderly to assess for premorbid cognition. However, this could not completed in all patients because surrogates were absent in approximately 30% of the subjects. Finally, because delirium causes an acute loss of cognition, bedside cognitive testing such as the Montreal Cognitive Assessment would not have accurately reflected a delirious patient’s premorbid cognition. The APS was used to quantify severity of illness. To calculate the CAM-ICU’s diagnostic performance, age, dementia, and severity of illness, as well as two-term interactions between the variables of interest and psychiatrist-diagnosed delirium (age × delirium, dementia × delirium, severity of illness × delirium), were incorporated into the multivariable logistic regression model. The CAM-ICU’s sensitivity was computed as the predicted probability of a multivariable logistic regression model where the patient had delirium diagnosed by the psychiatrist. Specificity was computed as 1 minus the predicted probability of a multivariable logistic regression model where the patient did not have delirium diagnosed by the psychiatrist. The reliability of the multivariable regression model was internally validated using the bootstrap method.⁴⁴ Two-thousand sets of bootstrap samples were generated by resampling the existing data. The optimism was estimated to determine the degree of overfitting, and an optimism value greater than 0.90 indicated no evidence of overfitting.⁴⁴ All statistical analyses were performed using open-source R statistical software, version 2.15.1 (http://www.r-project.org/).

RESULTS

A total of 953 patients were screened. A total of 365 patients were excluded because they refused consent, 65 left the ED before the assessments could be completed, 26 were previously enrolled, 20 were deaf or blind, 19 were unable to follow simple commands prior to their acute illnesses, 15 did not have both assessments completed within the 3-hour window, 12 refused the psychiatrists assessments, 11 were unresponsive to verbal stimuli, nine were not English-speaking, and five were excluded for unknown reasons. As a result, 406 patients met enrollment criteria, and their characteristics can be seen in Table 1. The median age was 73.5 years old (IQR = 69 to 80), 202 (49.8%) were female, and 57 (14.0%) were nonwhite race. During the study period, 22,168 potentially eligible ED patients who were 65 years or older presented to the ED. Enrolled and potentially eligible patients were similar in age and sex. However, enrolled patients were more likely to have ESI scores of 2, be admitted to the hospital, and have a chief complaint of chest pain (Data Supplement S1, available as supporting information in the online version of this paper).

Table 1.

Patient Characteristics

Characteristic	Enrolled Patients (n = 406)
Age (yr), median (IQR)	73.5 (69–80)
Female sex	202 (49.8)
Nonwhite race	57 (14.0)
Residence
Home	366 (90.1)
Assisted living	24 (5.9)
Rehabilitation/postacute care	5 (1.2)
Nursing home	11 (2.7)
Education
Elementary or below	9 (2.2)
Middle school	48 (11.8)
High school	163 (40.2)
College	118 (29.1)
Graduate school	67 (16.5)
Missing	1 (0.3)
Dementia in medical record	24 (5.9)
Charlson, median (IQR)	2 (1–4)
APS, median (IQR)	2 (1–4)
Given a benzodiazepine or opioid in the prehospital or ED setting	95 (23.4)
ESI
1	1 (0.3)
2	264 (65.0)
3	135 (33.3)
4	5 (1.2)
5	0 (0.0)
Unknown	1 (0.3)
ED chief complaint
Abdominal pain	17 (4.2)
Altered mental status	23 (6.2)
Chest pain	67 (16.5)
General weakness	40 (9.9)
Shortness of breath	46 (11.3)
Syncope	23 (5.7)
Admitted to the hospital	294 (72.4)

Data are reported as n (%) unless otherwise noted.

APS = Acute Physiology Score; ESI = Emergency Severity Index; IQR = interquartile range.

Of those enrolled, 50 (12.3%) were diagnosed with delirium by the psychiatrist. The median time between the CAM-ICU and psychiatrist assessments was 70 minutes (IQR = 31 to 120 minutes). The CAM-ICU’s diagnostic performance by the EP and RAs with their interobserver reliabilities can be seen in Table 2. Of the 14 physician CAM-ICU false-negatives that occurred, 12 were Feature 2 (inattention) negative. The kappa between the EP and RAs was 0.92 (95% CI = 0.85 to 0.98), indicating excellent interobserver reliability.

Table 2.

Diagnostic Performance of the CAM-ICU

Rater	Sensitivity (95% CI)	Specificity (95% CI)	LR+ (95% CI)	LR– (95% CI)
EP	72.0% (58.3–82.5)	98.6% (96.8–99.4)	51.3 (21.1–124.5)	0.28 (0.18–0.44)
RA	68.0% (54.2–79.2)	98.6% (96.8–99.4)	48.4 (19.9–118.0)	0.32 (0.22–0.49)
	κ = 0.92 (95% CI = 0.85–0.98)

CAM-ICU performed by the EP and RAs. The reference standard for delirium was a psychiatrist assessment using DSM-IV-TR criteria.

CAM-ICU = Confusion Assessment Method for the Intensive Care Unit; DSM-IV-TR = Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, Text Revision; EP = emergency physician; LR+ = positive likelihood ratio; LR– = negative likelihood ratio; RA = research assistant.

A secondary analysis was performed to determine how age, dementia documented in the medical record, and severity of illness affected the RAs’ CAM-ICU sensitivity and specificity using multivariable logistic regression (Figure 2). The RAs’ CAM-ICU sensitivity significantly increased as the severity of illness increased and in patients with dementia documented in the medical record. Age and severity of illness had no effect on the RAs’ CAM-ICU specificity, but there was a significant decrease in its specificity in patients with dementia. The multivariable logistic regression model was internally validated, and the optimism was 0.93, indicating that there was no evidence of substantial overfitting. For a patient who was 67 years old, had an APS of 4, and had no dementia, the RA CAM-ICU would have been 47.5% sensitive (95% CI = 25.4% to 70.6%) and 99.4% specific (95% CI = 95.6% to 99.9%). For an 87-year-old male with an APS of 1 and dementia documented in the medical record, the RA CAM-ICU would have been 93.7% sensitive (95% CI = 66.8% to 99.1%) and 83.3% specific (95% CI = 40.6% to 97.3%).

Figure 2.

The effect of age, severity of illness, and dementia on the RA CAM-ICU’s diagnostic performance. Severity of illness was measured by the Acute Physiology Score and dementia was obtained from the medical record. CAM-ICU = Confusion Assessment Method for the Intensive Care Unit; RA = research assistant.

DISCUSSION

Emergency physicians miss approximately 1.2 million of cases of delirium each year in the United States alone,^{1–3,15–18} because there is a dearth of brief (<2 minutes) and easy-to-use delirium assessments available for the fast-paced ED environment. The CAM-ICU possesses these desirable characteristics and may be useful for this challenging setting. We observed that the CAM-ICU had modest sensitivity, but excellent specificity compared with a psychiatrist’s DSM-IV-TR delirium assessment. A negative CAM-ICU modestly decreased the likelihood of delirium, but a positive CAM-ICU was nearly diagnostic as indicated by its very high LR+ (~50.0). It can also be performed by nonphysicians such as patient care technologists, paramedics, or nurses, who usually have more exposure to the patient and could potentially play an instrumental role in ED delirium surveillance.

To our knowledge, only two other studies have investigated the CAM-ICU’s criterion validity in a less critically ill patient population. Neufeld et al.⁴³ investigated its diagnostic performance in 117 non–critically ill patients admitted to the medical oncology ward and whose median age was 56 years old. They observed that the CAM-ICU performed by an RA was 18% sensitive and 99% specific. Mitasova et al.⁴² also investigated the CAM-ICU in 129 patients with acute stroke whose median age was 72.5 years old. In this cohort, the CAM-ICU’s sensitivity and specificity were 76 and 98%, respectively; the CAM-ICU was performed by a junior resident.

We hypothesized that this wide variability in sensitivity was secondary to the presence of spectrum bias. The CAM-ICU uses objective measurements with prespecified cutoffs to determine inattention and disorganized thinking. While this enhances ease of use, its diagnostic performance may be more likely to be influenced by patient characteristics. We observed that the RAs’ CAM-ICU sensitivity improved in patients with higher severities of illness and dementia, and this may explain the variation in its sensitivity observed between studies conducted in non–critically ill patients. It may also explain why the CAM-ICU’s sensitivity in non–critically ill patients is lower than what has been observed in an ICU population who most likely have much higher severities of illness. We also observed that the CAM-ICU’s specificity decreased in those with dementia. Although the specificities reported by others have been consistently above 90%, specificities as low as 71.4% have been reported.⁴¹ Because we had a small number of patients with dementia documented in the medical record (n = 24), the 95% CIs for sensitivity and specificity were relatively wide in this subgroup. Larger studies are needed to better estimate the extent in which spectrum bias exists.

The CAM-ICU had modest sensitivity in older ED patients. The Brief Confusion Assessment Method (bCAM) is a modification of the CAM-ICU; the primary difference is that the bCAM asks the patient to recite the months backwards from December to July to assess for inattention (Feature 2) instead of the ASE letters and pictures. With this modification, the bCAM was 84% sensitive and 96% specific when performed by the EP and 78% sensitive and 97% specific when performed by the RA.⁴⁵ The bCAM is also brief (<1 minute) and may be a feasible mechanism to screen for delirium older ED patients.

This investigation is the preliminary step for future delirium investigations in the ED, a setting in which delirium has been understudied. Most of the CAM-ICU’s false-negatives occurred because they were able to successfully perform the ASE letter and picture tasks; the consequences of missing these cases deserve further study. Conversely, it is possible that the CAM-ICU may identify delirious patients with more severe impairments and at higher risk for adverse outcomes. Previous investigations have reported that a positive CAM-ICU in older ED patients is an independent predictor of 6-month mortality and prolonged hospitalizations.^26,28 Future studies should elucidate which delirium assessment (CAM-ICU, bCAM, etc.) best identifies those at highest risk for adverse outcomes. Identification of high-risk delirious patients may also help identify those who may most benefit from delirium interventions.

LIMITATIONS

Because of psychiatrists had limited availability, we enrolled a convenience sample, which possibly introduced selection bias. Although our enrolled cohort was similar in age and sex compared with all potentially eligible older ED patients, they were more likely to be admitted, indicating higher severities of illnesses. Given the fluctuating course of delirium, as well as the administration of psychoactive medications (e.g., opioid and benzodiazepine medications) in the ED, time may have caused some discrepant observations between the research team’s and psychiatrist’s assessments. This can both underestimate and overestimate the CAM-ICU’s diagnostic accuracy. Despite being one of the larger delirium validation cohorts to be enrolled, the precision of the CAM-ICU’s sensitivity estimate was relatively wide. To assess the effect of dementia on the CAM-ICU’s sensitivity and specificity, we chose to use the medical record to ascertain the patient’s dementia status. However, dementia is poorly documented in the medical record, and misclassification likely occurred.²⁵ It is unclear if this misclassification magnified or attenuated the relationship between dementia and the CAM-ICU’s sensitivity or specificity.

This study was performed in a single ED located at an urban, tertiary care, academic hospital in patients who were 65 years and older; our findings may not be generalizable to other settings and especially in those who are less than 65 years of age. The CAM-ICU was also performed by the research staff and a single EP. The CAM-ICU’s diagnostic accuracy and reliability may be lower when used by health care personnel in real world settings.

CONCLUSIONS

The CAM-ICU is highly specific in older ED patients when used by both research assistants and an emergency physician. A positive test is nearly diagnostic for delirium. However, the CAM-ICUt’s sensitivity was modest and a negative test decreased the likelihood of delirium by a small amount. While previous studies have shown the CAM-ICU to predict adverse outcomes in older ED patients, the consequences of a false-negative CAM-ICU are unknown and deserve further study. Severity of illness and dementia may affect the CAM-ICU’s diagnostic accuracy and may help explain why its diagnostic performance varies between studies. However, given the relatively small sample size of demented patients, larger studies are needed to precisely determine their true effect.