3D-CAM: Derivation and Validation of a 3-Minute Diagnostic Interview for CAM-defined Delirium

Abstract

Background

Delirium is common, morbid, and costly, yet remains often unrecognized in most clinical settings. The Confusion Assessment Method (CAM) is the most widely used diagnostic algorithm, and operationalizing its features would represent a substantial advance for clinical care.

Objective

To derive the 3D-CAM, a new 3-minute diagnostic assessment for CAM-defined delirium, and to validate it against a clinical reference standard.

Design

Diagnostic test study

Setting

4 general medicine units in an academic medical center

Participants

201 inpatients aged ≥ 75 years old

Measurements

We identified 20 items that best operationalized the 4 CAM diagnostic features to create the 3D-CAM. For prospective validation, 3D-CAM assessments were administered by trained research assistants. Independently, clinicians performed an extensive assessment that included patient interviews, family interviews, and review of the medical record. These data were considered by an expert panel to determine the presence or absence of delirium and dementia (reference standard). We compared the 3D-CAM delirium determination to the reference standard in all patients and in subgroups with and without dementia.

Results

The 201 participants in the prospective validation study had mean age (SD) of 84 (5.5) years, and 27% had dementia. The expert panel identified delirium in 21%. Median administration time for 3D-CAM was 3 minutes (inter-quartile range: 2–5 minutes). The sensitivity [95% CI] of 3D-CAM was 95% [84%, 99%] and the specificity was 94% [90%, 97%]. The 3D-CAM performed well in patients both with dementia (sensitivity=96% [82%, 100%], specificity=86% [67%, 96%]) and without dementia (sensitivity=93% [66%, 100%], specificity=96% [91%,99%]).

Limitations

Limited to single center, cross-sectional, and medicine patients only

Conclusion

The 3D-CAM operationalizes the CAM algorithm using a 3-minute structured assessment with high sensitivity and specificity relative to a reference standard and could be an important tool for improving recognition of delirium.

Introduction

Delirium is common, morbid, and costly in hospitalized elders (1–3). Despite increasing awareness of its importance, most delirium, particularly hypoactive delirium and delirium superimposed on dementia, still goes unrecognized (1–3). Prompt recognition of delirium is the first key step in its appropriate management, which involves careful review for reversible contributors, preventing complications (including ensuring patient safety), and instituting cognitive and physical rehabilitation (1). Evidence suggests that such an approach can shorten the duration of delirium and improve its associated adverse outcomes (1, 3).

The Confusion Assessment Method (CAM), developed in 1990 (4), has been widely adopted, and a recent comparison of diagnostic methods suggests the CAM is the best performing bedside delirium assessment tool (5). While the CAM is widely used to define delirium in the literature (6), it can be challenging to operationalize in the clinical setting, requiring cognitive assessment and substantial interviewer training. Moreover, there is still a great deal of variability in how the CAM is applied, which can lead to differential performance in detecting delirium (5).

A brief, structured mental status assessment that operationalizes the CAM algorithm would be extremely helpful to accelerate widespread ascertainment of delirium in high risk patients (4, 5). Therefore, our overall goal was to develop and validate the 3D-CAM, the 3-Minute Diagnostic Assessment for Delirium using the CAM algorithm. Our current aims were: 1) to create the 3D-CAM using model selection methods to finalize items and to determine thresholds for the presence or absence for each of the 4 CAM diagnostic features, and 2) to prospectively validate the 3D-CAM by comparing it to a reference standard that included an extensive clinical evaluation in a new population of older general medicine patients with a high burden of baseline cognitive impairment and comorbidity.

METHODS

Derivation of the 3D-CAM (for details, see eAppendix 1)

We started with a dataset of 4598 structured delirium assessments from a previously completed multi-site trial of the Delirium Abatement Program (DAP) conducted in 8 post-acute facilities (7). In previously published 3D-CAM derivation work, we mapped over 120 items from this assessment to the four CAM diagnostic features (8), and used item response theory (IRT) (9) to identify the 36 most informative items for the identification of each of these features (10). For more details, see eAppendix 1.

For the current 3D-CAM derivation work, we further reduced this set of 36 items using logistic regression and assembled the most useful subset of items from each of the 4 CAM diagnostic features to create the 3D-CAM. We used regression coefficients to determine weights of each item and thresholds for determining the presence or absence of each of the features: 1) acute change and fluctuating course, 2) inattention, 3) disorganized thinking, and 4) altered level of consciousness. For each feature, the best performing approach weighted each cognitive testing item, patient symptom question, and interviewer observation equally. Moreover, the CAM feature was rated as present if any one of the items (cognitive test result, reported symptom, observation item) was rated as positive. Once each feature is rated, the presence of delirium is determined by the CAM diagnostic algorithm, which requires the presence of features 1 and 2, and either 3 or 4 (see Figure 1 for 3D-CAM items and scoring algorithm).

Figure 1. Overview of 3D-CAM Assessment.

This figure depicts the CAM diagnostic algorithm, with the 3D-CAM items and scoring summarized under each CAM diagnostic feature.

Once we selected the items and defined the scoring algorithm for the 3D-CAM, we made a preliminary assessment of the 3D-CAM’s diagnostic accuracy. Using the DAP dataset of 4598 assessments, we used only the 3D-CAM items and algorithm described above to score the CAM algorithm, and compared the presence or absence of delirium generated from this approach with the results from the full 160-item structured delirium assessment (11). In this initial derivation work, the 3D-CAM achieved 92% sensitivity, 95% confidence interval (C.I.) [90%, 94%] and 93% specificity, 95% C.I. [92%, 93%] relative to the full assessment, which met our goal and allowed us to proceed with the prospective validation.

Prospective Validation Study

Study Population

We enrolled participants from a large urban teaching hospital in Boston, Massachusetts. Inclusion criteria were: 1) age ≥ 75 years old, 2) admitted to general medicine or geriatric medicine services, 3) able to communicate effectively in English, 4) without terminal conditions, 5) expected hospital stay of ≥ 2 days, and 6) not a previous study participant. Experienced clinicians (clinical psychologists and advanced practice nurses) screened for eligible patients. After obtaining approval from the attending physician, each eligible patient was approached for informed consent. If the patient lacked capacity to provide consent, the designated surrogate decision-maker was contacted. The study protocol and informed consent procedures were approved by the Institutional Review Board.

Reference Standard Delirium Assessment

The operational reference standard delirium diagnosis was based on an extensive face-to-face patient interview (45 minutes), medical record review, and input from the patient’s nurse and available family members. This assessment included: 1) reason for hospital admission and hospital course, 2) presence of cognitive concern, both prior to and during the hospitalization, 3) family, social, and functional history, 4) Montreal Cognitive Assessment (MoCA), a 30-item assessment that takes approximately 20 minutes to administer (12), 5) Geriatric Depression Scale (GDS) to evaluate for presence of depressive symptoms (13), 6) medical record review, including quantification of comorbidities using the Charlson index (13), diagnosis of dementia or mild cognitive impairment (MCI) prior to hospitalization, determination of functional status using the basic and Instrumental Activities of Daily Living scales (15, 16), and a list of psychoactive medications administered. If the assessment indicated potential cognitive impairment, (MoCA score≤23) (12), the clinical assessor conducted a proxy interview to assist with determining the patient’s baseline mental status relevant to a possible diagnosis of dementia versus a history of lifelong, developmentally based cognitive limitations. The proxy interview included: 1) ascertainment of a cognitive concern, prior to and during the hospitalization; 2) ascertainment as to whether specific cognitive deficits evident on testing existed prior to hospitalization; 3) confirmation of functional status obtained from the medical record, and 4) a proxy-based screening questionnaire for dementia (the AD-8) (17).

The final delirium diagnoses were adjudicated by a study panel including the clinical assessor (psychologist or advanced practice nurse), the study PI (Marcantonio), a geriatrician, and a board-certified neuropsychologist (O’Connor), using DSM-IV criteria (18) For patients not meeting delirium criteria, the panel adjudicated the presence or absence of subsyndromal delirium (19), defined by the presence of acute change/fluctuating course, plus inattention or disorganized thinking or an altered level of consciousness. The panel was blinded to the results of subsequent 3D-CAM testing (see below). A geropsychiatrist (Metzger) subsequently re-adjudicated a 10% subsample (10 randomly selected participants with delirium and 10 without delirium) blinded to the original results to verify the panel adjudication process. In addition to determining delirium status, the panel adjudicated the presence or absence of cognitive impairment at baseline, including dementia or mild cognitive impairment (MCI) using the National Institute on Aging – Alzheimer’s Association criteria (20, 21) (for details of data used for adjudication of dementia and MCI, see eAppendix 2).

3D-CAM Assessments

Following the reference standard assessment, the 3D-CAM was administered by research assistants (RA’s) who were blinded to the reference standard results. A total of 8 RA’s participated in the validation study, and each evaluated between 4–49 participants, based on participant and RA availability. Before the start of the study, each RA underwent a 1–2 hour training session on the use of the 3D-CAM, including practice administering the instrument to each other and to actual patients. To assess inter-rater reliability, based on a random number sequence 50% of the participants were selected to undergo a second 3D-CAM assessment, blinded both to the reference standard and the first 3D-CAM. All 8 RA’s participated in the reliability study, representing 18 distinct pairs of raters, each evaluating between 1–19 participants, again based on participant and RA availability. To assure temporal proximity, all assessments (the reference standard and the one or two 3D-CAM assessments) were completed within a 2-hour period during the hours of 11 AM and 2 PM (Figure 2).

Figure 2. Prospective Validation Study Flow Diagram.

*Presence or absence of delirium as determined by first 3D-CAM assessment was compared to the reference standard delirium diagnosis to assess validity, and to the second 3D-CAM assessment to assess inter-rater agreement

All assessments were completed within 2 hours of each other, and results were strictly blinded from the other assessors

Statistical Analyses

We calculated sensitivity, specificity, and 95% confidence intervals for the 3D-CAM delirium determination compared to the reference standard. We performed subset analyses to determine diagnostic test characteristics of the 3D-CAM stratified by the patient’s baseline cognitive status (normal/MCI vs. dementia). We assessed inter-rater agreement of the two independent 3D-CAM assessments using simple descriptive statistics. We used SAS software, Version 9.3 (SAS Institute, Inc., Cary, NC) for all data analyses.

Role of Funding Agency

The National Institute on Aging had no role in the design, conduct, and analysis, nor in the decision to submit this manuscript for publication.

RESULTS

Prospective Validation Study

Patient Characteristics

A total of 201 patients met inclusion criteria and provided informed consent. The mean age [standard deviation (SD)] was 84 (5.4) years, 62% were women, and 88% self-reported to be white (Table 1). Approximately half had some college education, and 5% had English as a second language but were sufficiently fluent to be enrolled. The mean Charlson comorbidity score (SD) was 3.0 (2.3), indicating a moderate to high level of medical comorbidity (14). Fifty-five percent had dependencies in one or more basic Activities of Daily Living (15), and 81% had dependencies in one or more Instrumental Activities of Daily Living (16). Based on the expert panel operational reference standard assessment, 27 percent of the population was found to have baseline dementia.

Table 1.

Characteristics of the Prospective Validation Study Population (N=201)

Characteristic
Age, mean (SD)	84 (5.4)
Sex, N (%) female	125 (62)
Race–White, N (%)	177 (88)
Education*, N (%)
Less than High School	20 (10)
High School Graduate	75 (38)
College or More	100 (49)
English second language N (%)	10 (5)
Severe sensory Impairment–Vision, N (%)	5 (2)
Severe sensory Impairment–Hearing, N (%)	18 (9)
Charlson Comorbidity Index (14), mean (SD)	3.0 (2.3)
Activities of Daily Living Dependence (15), N (%)	110 (55)
Instrumental Activities of Daily Living Dependence (16), N (%)	163 (81)
Baseline Cognition—Mild Cognitive Impairment, N (%)	50 (25)
Baseline Cognition–Dementia, N (%)	55 (27)
Delirium by Reference Standard, N (%)	42 (21)
Delirium Psychomotor Features, N (%)
Hypoactive or Normal	37 (19)
Hyperactive or Mixed	5 (2)

^*Education status was missing in 6 (3%) participants

Overall Study Flow, Reference standard and 3D-CAM assessments

Figure 2 depicts the study flow of assessments, which included the reference standard clinical evaluation for delirium, the 3D-CAM assessment, and a second 3D-CAM in a random one half of participants. All of the face-to-face components of the assessments were completed within the desired 2 hour time window. The reference standard assessment for delirium took approximately 1.5 hours to complete including patient interviews, medical record reviews, and proxy interviews. Based on these assessments, 42 participants (21% of the sample) were diagnosed with delirium. Of these, 37 (88% of delirium cases) had either hypoactive or normal psychomotor features with only 5 showing hyperactive or mixed features. In the 20 patients (10 with delirium, 10 without delirium) whose records were re-reviewed by the blinded geropsychiatrist, there was 100% concordance with the expert panel diagnosis. In comparison to the reference standard assessment, the 3D-CAM was completed in a median of 3 minutes (interquartile range 2–5 minutes, overall range 1–15 minutes). Forty-nine participants (24%) were determined to have delirium based on the 3D-CAM.

Diagnostic Test Characteristics of the 3D-CAM

Compared to the reference standard delirium diagnosis, the sensitivity of the 3D-CAM was 95% with 95% confidence intervals (C.I.) of [84%, 99%], while the specificity of 3D-CAM was 94%, with 95% C.I. [90%, 97%] (Table 2). These result in a likelihood ratio positive of 16.8, 95% C.I. [8.9, 31.8], and a likelihood ratio negative of 0.05, 95% C.I. [0.01, 0.20]. Notably, of the 9 “false positives” identified by 3D-CAM, 6 were adjudicated to have subsyndromal delirium based on the reference standard assessment (19). In post-hoc analyses, we examined the effect of regrouping subsyndromal cases with the “positives” based on the reference standard. This resulted in the sensitivity of the 3D-CAM increasing to 46/48 (96%) and specificity increasing to 150/153 (98%).

Table 2.

Diagnostic Test Characteristics of the 3D-CAM Compared to the Reference Standard Delirium Assessment

Delirium Diagnosis (− or +)	Reference Standard (+)	Reference Standard (−)	3D-CAM Totals
3D-CAM (+)	40	9*	49
3D-CAM (−)	2	150	152
Reference Standard Totals	42	159	201
Test Characteristic†	Sensitivity=40/42	Specificity=150/159
% [95% Confidence Intervals]	95% [84%,99%]	94% [90%,97%]

^*Of the 9 3D-CAM “false positives”, 6 had sub-syndromal delirium based on the reference standard (19)

^†The sensitivity and specificity result in a likelihood ratio positive of 16.8 [8.9, 31.8], and a likelihood ratio negative of 0.05 [0.01, 0.20] (for more details, see eAppendix).

Diagnostic Test Characteristics Stratified by baseline cognition

We examined the diagnostic test characteristics of 3D-CAM stratified by participants’ baseline cognitive function (Table 3). In the group with either normal baseline cognition or MCI, the sensitivity was 93%, with specificity of 96%. In the dementia subgroup, the sensitivity was excellent at 96%, with slightly lower specificity of 86% (for 95% CI’s and likelihood ratios stratified by baseline cognition, see Table 3).

Table 3.

Diagnostic Test Characteristics of the 3D-CAM Stratified by Baseline Cognitive Function

3D-CAM Test Characteristics in Patients with Normal Baseline Cognition or MCI (N=145)
Delirium Diagnosis (− or +)	Reference Standard (+)	Reference Standard (−)	3D-CAM Totals
3D-CAM (+)	13	5	18
3D-CAM (−)	1	126	127
Reference Standard Totals	14	131	145
Test Characteristic†	Sensitivity = 13/14	Specificity = 126/131
%, [95% C.I.]	93% [66%,100%]	96% [91%,99%]

3D-CAM Test Characteristics in Patients with Dementia (N=56)
Delirium Diagnosis (− or +)	Reference Standard (+)	Reference Standard (−)	3D-CAM Totals
3D-CAM (+)	27	4	31
3D-CAM (−)	1	24	25
Reference Standard Totals	28	28	56
Test Characteristic†	Sensitivity = 27/28	Specificity = 24/28
%, [95% C.I.]	96% [82%,100%]	86% [67%,96%]

^†The sensitivity and specificity result in a likelihood ratio positive of 24.3 [10.2, 58.2], and a likelihood ratio negative of 0.07 [0.01, 0.49] for patients with normal baseline cognition or MCI, and a likelihood ratio positive of 6.8 [2.7, 16.8], and a likelihood ratio negative of 0.04 [0.01, 0.29] for patients with dementia.

Inter-rater Agreement

Finally, we examined agreement across raters in the subset of 100 participants who underwent a second 3D-CAM assessment blinded to the first. This demonstrated that the 3D-CAM had excellent inter-rater agreement of 95% (see eTable3 for details).

DISCUSSION

Delirium is an important clinical syndrome to detect. Currently there is no brief instrument well-suited for widespread use across clinical settings. We therefore sought to develop and evaluate the 3D-CAM, a short, structured diagnostic assessment for delirium that can be administered by healthcare delivery staff with minimal additional training. The 3D-CAM demonstrated strong performance characteristics in our study. It was completed in a median of 3 minutes and had excellent sensitivity and specificity relative to a reference standard delirium diagnosis, even in patients with dementia. The slightly lower specificity in the dementia subgroup is attributable to a higher likelihood of false positives inherent in the challenging process of distinguishing symptoms and signs of delirium from dementia (22). The 3D-CAM also had excellent inter-rater agreement. With its brevity and ease of use, 3D-CAM may provide a useful tool for improving widespread detection of delirium in clinical settings.

Hospitalized patients are rarely formally assessed for delirium (1, 3). Studies performed over the past 20 years suggest that the clinical recognition rate has not significantly changed, and remains in the 12–35% range (23–27). Moreover, delirium cases identified tend to be agitated patients who are disruptive to patient care, while hypoactive patients remain unrecognized. Studies have shown that hypoactive patients with delirium have either similar or somewhat worse outcomes compared with agitated patients with delirium (28, 29). The 3D-CAM demonstrated excellent sensitivity in our sample, even though 88% of delirium cases had either hypoactive or normal psychomotor features.

To improve delirium recognition in hypoactive patients, it is important to incorporate results from direct questioning of patients and mental status testing into the delirium assessment and not just rely on interviewer observations. In the initial validation studies, the CAM developers performed a structured mental status assessment consisting of the Mini-Mental State Examination (MMSE) (30) plus recall of a story or digit span before operationalizing the CAM algorithm (4). In a subsequent publication, nurses were interviewed about the CAM diagnostic features on a daily basis using observations from routine clinical care without formal mental status assessment (31). When nurse CAM ratings were compared to researcher CAM ratings following a structured mental status assessment, the sensitivity of the nurse CAM ratings was only 20% per interview, and 33% over the course of the entire hospitalization (31). Thus, in the absence of mental status testing, the CAM algorithm alone did not substantially improve the rate of delirium detection. The 3D-CAM demonstrates that excellent sensitivity for delirium can be achieved with brief mental status testing focused on attention and orientation.

The CAM-ICU is an example of a structured assessment that incorporates a specific set of questions to operationalize each CAM diagnostic feature (32). For instance, attention is assessed using 2 items: the vigilance “A” task, and the picture recognition task from the Attention Screening Examination (33). Because the CAM-ICU was designed to assess delirium in intubated ICU patients, all questions are answerable using non-verbal responses, such as a nod of the head (yes or no). With its brevity and ease of use, the CAM-ICU has greatly enhanced the assessment of delirium in the ICU (34). Yet, recent studies suggest that the CAM-ICU may not be optimized to non-ICU populations, where it demonstrates lower sensitivity relative to a verbal delirium assessment (35–37). For instance, in a recent validation study of 406 people evaluated in an Emergency Department (mean age 73.5 years, dementia prevalence of 5.9%) the CAM-ICU demonstrated a sensitivity of 68–72% (37). The recently developed and validated B-CAM provides an alternative that incorporates verbal testing using items very similar to the CAM-ICU (38). Notably, both the CAM-ICU and the B-CAM put major emphasis on the Richmond Agitation and Sedation Scale (RASS) (39), which detects altered levels of consciousness. Yet, the recently published DSM-5 delirium definition de-emphasizes the importance of altered level of consciousness in delirium and focuses instead on assessment of attention and orientation (40), both key components of the 3D-CAM. Moreover, altered level of consciousness is much less prevalent in delirium outside of the ICU; in our sample, it was present in only 8 of 42 delirium cases (19%).

While physicians and nurses will need to be trained to use the 3D-CAM optimally, its brevity and algorithmic structure should simplify the process. There are clear instructions mapping specific questions to each CAM feature. The 3D-CAM instrument and training manual (available free of charge at www.hospitalelderlifeprogram.org) provide clear explanations of how to code the presence or absence of each feature based on patient responses. These characteristics of the 3D-CAM reduce the amount of judgment required by the assessor and facilitate reproducibility across assessors, consistent with the extremely high inter-rater agreement from this study. The structured nature of 3D-CAM also makes it very amenable to administration via an electronic platform, such as mobile technology. Notably, since only one “positive” item is required to trigger the presence of each feature, the assessment could potentially be shortened further by incorporating skip patterns, an approach that we did not test in our validation study. Finally, the 3D-CAM is entirely compatible with the short form version of the recently published CAM-S delirium severity measure (41). We are developing an algorithm to score the CAM-S using 3D-CAM items, to be presented in future work.

Our approach has several important strengths. First, the 3D-CAM was created using items rigorously selected using IRT to be maximally informative for determining the presence or absence of the 4 CAM diagnostic features. These items were further reduced using model selection techniques in a dataset of 4598 delirium assessments. Our prospective validation study enrolled a sample of over 200 patients with a mean age of over 80 years, substantial comorbidity, and a high burden of baseline cognitive impairment, representing a purposeful “challenge population” for 3D-CAM validation. The 3D-CAM is equally applicable in younger populations with a lower prevalence of dementia, and will likely perform even better since delirium assessment in these patients is more straightforward. Importantly, our study employed a clinical reference standard and a design in which all delirium assessments were administered close in time while the results of each test were kept strictly blinded from the other assessors. Finally, we assessed and confirmed excellent inter-rater agreement for the 3D-CAM ratings and also independently validated a subset of reference standard delirium diagnoses.

Our approach does have several limitations. First, our prospective validation study enrolled only general medicine patients and was conducted at a single site. Thus, our findings should be confirmed in other settings in which delirium is also common and morbid, such as non-ICU surgical wards, palliative care, post-acute care, and other types of hospitals. Second, our prospective validation study assessed patients only on a single hospital day. Additional studies should examine the performance and acceptability of the 3D-CAM when repeated on a daily basis for delirium screening. Third, all of our evaluations were performed during the day shift, and performance during evening and night shifts should also be evaluated. Fourth, the 95% confidence intervals for some test characteristics in our stratified analyses are wide, and should be interpreted accordingly. Finally, we used a paper and pencil, static 3D-CAM assessment, and future studies should examine use of a dynamic assessment employing automated skip patterns on an electronic platform, which might further enhance efficiency.

In conclusion, in our study of general medicine patients with advanced age and a high prevalence of underlying cognitive impairment, the 3D-CAM proved to be a brief, highly reproducible, and valid method for diagnosing delirium using the CAM algorithm. Given these characteristics, the 3D-CAM could be an important component of future efforts to improve systematic case-finding of delirium in high-risk older adults. Further research will focus on developing the optimal strategies for translating the 3D-CAM into routine care, and determining whether improved detection of delirium can result in improved outcomes for vulnerable hospitalized elders.