Comparative Accuracy and Efficiency of Four Delirium Screening Protocols

Abstract

Background/Objectives:

Systematic screening can improve detection of delirium, but lack of time is often cited as why such screening is not performed. We investigated the time required to implement four screening protocols that use the Ultra-Brief 2-item screener for delirium (UB-2) and the 3-Minute Diagnostic Interview for CAM-defined Delirium (3D-CAM), with and without a skip pattern that can further shorten the assessment. Our objective was to compare the sensitivity, specificity, and time required to complete four protocols: (1) full 3D-CAM on all patients, (2) 3D-CAM with skip on all patients, (3) UB-2, followed by the full 3D-CAM in ‘positives,’ and (4) UB-2, followed by the 3D-CAM with skip in ‘positives.’

Design:

Comparative efficiency simulation study using secondary data.

Setting:

Two studies (3D-CAM and READI) conducted at a large academic medical center (3D-CAM and READI) and a small community hospital (READI only).

Participants:

General medicine inpatients age 70 and older (3D-CAM, n= 201; READI, n= 330).

Measurements:

We used 3D-CAM data to simulate the items administered under each protocol, and READI data to calculate median administration time per item. We calculated sensitivity, specificity, and total administration time for each of the 4 protocols.

Results:

The 3D-CAM and READI samples had similar characteristics and all four protocols had similar simulated sensitivity and specificity. Mean administration times were 3 minutes 13 seconds for 3D-CAM, 2 minutes 19 seconds for 3D-CAM with skip, 1 minute 52 seconds for UB-2 + 3D-CAM in positives, and 1 minute 14 seconds for UB-2 + 3D-CAM with skip in positives, which was 1 minute and 59 seconds faster than the 3D-CAM (p<.001).

Conclusion:

The UB-CAM, consisting of the UB-2, followed in positives by the 3D-CAM with skip pattern, is a time-efficient delirium screening protocol that holds promise for increasing systematic screening for delirium in hospitalized older adults.

INTRODUCTION:

Delirium, an acute confusional state, is a common, morbid and costly condition in older adults, affecting one third of general medical inpatients age 70 and older¹. It is associated with longer lengths of stay, increased rates of institutionalization, and higher mortality². Despite these potentially devastating consequences, less than half of all delirium cases are identified in routine clinical care³. Systemic screening can improve detection, but clinicians often cite lack of time as a major reason why such screening is not performed.

Several tools have been created for delirium diagnosis and screening. The most widely used is the Confusion Assessment Method (CAM). The CAM diagnostic algorithm assesses four core features: 1) acute change or fluctuating course, 2) inattention, 3) disorganized thinking, and 4) altered level of consciousness, with features 1, 2, and either 3 or 4 needed to fulfill criteria for delirium⁴. Brief structured instruments have been developed for specific settings, for example the Brief Confusion Assessment Method (bCAM) for emergency department patients⁵, the CAM-ICU for intensive care patients⁶, and the Pediatric Confusion Assessment Method (pCAM-ICU) for pediatric intensive care patients⁷. For general medicine inpatients, briefer delirium assessment tools include the 3D-CAM, a three-minute structured assessment that operationalizes the CAM diagnostic algorithm⁸, and the 4-AT, an assessment of alertness, cognition (orientation and attention), and acute change in mental status that is not based on the CAM algorithm⁹.

The 3D-CAM consists of 20 items in total, ten of which are administered directly to patients, including seven items assessing orientation and attention, and three patient symptom probes⁸. The remaining ten items are observation items assessing all four CAM diagnostic features, and are completed by the assessor at the conclusion of the interview. The 3D-CAM takes approximately three minutes to complete and was previously demonstrated to have 95% sensitivity and 94% specificity compared to a reference standard delirium assessment performed by a clinician involving detailed patient interview, family interview, and medical record review⁸. A 3D-CAM skip pattern that stops assessment of each CAM feature if a single cognitive item mapped to that feature is incorrect, a single symptom probe is endorsed, or a single observation item of the feature is present (any of these is sufficient to trigger presence of the feature) can further shorten the 3D-CAM assessment.

Because time is a barrier to widespread screening even with a three-minute tool, shorter “ultra-brief” screening tools have also been developed to quickly distinguish patients who warrant further evaluation for delirium from those in whom delirium can be ruled out quickly. We previously developed the Ultra-Brief 2-item (UB-2)¹⁰ screen for delirium using two items from the 3D-CAM: “Recite the months of the year backwards,” and “What is the day of the week?” Incorrect responses to either of these items constitutes a positive screen, which has 93% sensitivity, but only 64% specificity for the presence of delirium¹⁰. Based on this high sensitivity, negative screens can be used to quickly rule out delirium, whereas positive screens require further assessment, such as by completing the rest of the 3D-CAM to accurately identify delirium.

The specific aim of the present study was to use simulations to compare the sensitivity, specificity, and time required to complete four delirium identification protocols: (1) full 3D-CAM (all 20 items) on all patients, (2) 3D-CAM with skip pattern (as described above) on all patients, (3) UB-2, followed by full 3D-CAM in ‘positives,’ and (4) UB-2, followed by 3D-CAM with skip pattern in ‘positives.’ We hypothesized that delirium screening protocols (2), (3) and (4) would be faster than the full 3D-CAM, and that protocol (4) would be the fastest.

METHODS:

Study Design:

Secondary data from two studies, 3D-CAM: Derivation and Validation of a 3-Minute Diagnostic Interview for CAM-defined Delirium (3D-CAM)⁸, and Researching Efficient Approaches to Delirium Identification (READI)¹¹, were used to simulate protocol administration time under the four protocols described above. The 3D-CAM study data included 201 full 3D-CAM assessments, which were used to determine the total number of items and specific items administered under each of the four protocols using computer simulations. In the READI study, assessments were prompted by the “Delirium-READI” App, which recorded the time each item response was entered. We used data from the first 330 READI assessments to calculate the median administration time for each of the 20 3D-CAM items.

Study Populations:

The sample population for the 3D-CAM validation study has been previously described in detail⁸. It consisted of 201 inpatients aged 75 years or older on general medicine units at a large academic medical center in Boston, Massachusetts. The READI study is designed to assess the feasibility, test characteristics, time required, and cost of administration of a 2-step protocol for delirium identification consisting of the UB-2 followed by the 3D-CAM in ‘positives,’ as administered by doctors, nurses, and nursing assistants¹¹. It prospectively enrolled participants from the same medical center as 3D-CAM, and from a smaller rural community hospital in Pennsylvania. Inclusion criteria were age 70 years and older, admission to general medicine service, expected length of hospital stay of 3 or more days, and ability to communicate adequately in English. Exclusion criteria were imminently terminal condition, severe deafness or blindness, and nonverbal condition. Nursing assistants administered only the UB-2 in READI, and therefore were not used for our timing analysis. Doctors and nurses had similar administration times for individual items and data from both disciplines were used in our timing calculations. For both parent studies (3D-CAM and READI), eligible patients were approached for informed consent by study staff after approval from the attending physician, and study protocol and informed consent procedures were approved the institutional review boards of all involved institutions. As the present study is a secondary data analysis, no informed consent procedures were required for this analysis.

Statistical Analysis:

Using data from each of the 201 3D-CAM administrations in the 3D-CAM study, we simulated the total number of items and the identity of the items administered under each of the four delirium identification protocols. This procedure generated 804 simulations. For the full 3D-CAM protocol, all of the items in the 3D-CAM would be administered. For 3D-CAM with skip protocol, we dropped any of the items skipped once a CAM Feature was triggered. However, we did not apply skips between features, such as skipping Feature 4 if Feature 3 was present. This is due to the clinical importance of recognizing Altered Level of Consciousness. In the two protocols that begin with the UB-2, if the participant got both items in the UB-2 correct, then only 2 items would have been administered. If they got one or both of the UB-2 items wrong, then either the full 3D-CAM or the 3D-CAM with skip would have been administered. For each simulated protocol, we determined whether the protocol would have resulted in the presence or absence of delirium, and calculated sensitivity and specificity relative to the clinical reference standard for each participant.

We then calculated median administration time for each 3D-CAM item using READI study data from the 330 assessments described above. Times for the UB-2 screener items were calculated from all READI assessments; times for the remaining 3D-CAM items were calculated only from READI assessments in which the patient failed the screener. We calculated mean time for each delirium identification protocol by applying the median item times to the items administered in each simulation described above, then summing over the entire protocol.

We used the mean administration time for the 3D-CAM as our reference standard to complete pairwise analyses for the time required for the remaining assessment protocols using the Wilcoxon rank-sum test for nonparametric data. We completed analyses stratified by presence of delirium and baseline dementia status for the UB-2 followed by 3D-CAM with skip pattern protocol by simulating the number and identity of items and time per item in these subgroups. We used SAS, version 9.4 (SAS Institute, Inc., Cary, NC) for all data analyses.

RESULTS:

Patient Characteristics:

Demographic characteristics of the 3D-CAM and READI cohorts are described in Table 1. The two study samples had similar demographic characteristics: the mean age for 3D-CAM patients was 84 years and 62% were women, while for READI the mean age was 81 years and 57% were women. For race/language, a similar percentage of patients in both studies were non-white and spoke English as a second language but were sufficiently fluent to be enrolled. Participants in both studies were comparable on education level, mean Charlson comorbidity score, dependencies on activities of daily living and instrumental activities of daily living. Similar percentages of patients had dementia (3D-CAM 28%, READI 34%) and delirium (3D-CAM 21%, READI 20%).

Table 1:

Characteristics of the Study Populations^h

Characteristic	3D-CAM Participants (n=201)	READI Participants (n=330)
Mean age (SD), years	84 (5.4)	81 (6.9)
Female	125 (62)	191 (58%)
White Race	177 (88)	283 (86)
Educationi
Less than high school	20 (10)	43 (13)
High school graduate	75 (38)	116, 2 GED (36)
College or more	100 (49)	170 (51)
English as a second language	10 (5)	8 (2)
Charlson comorbidity index score (SD)j	3.0 (2.3)	3.0 (2.4)
Dependence in activities of daily living	100 (55)	121 (36)
Dependence in instrumental activities of daily living	163 (81)	197 (59)
Dementiak	56 (28)	112 (34)
Delirium by reference standard	42 (21)	65 (20)

^hValues reported as numbers (percentages) unless otherwise indicated

ⁱEducation status was missing in 3(1) participants for READI, 6(3) participants in 3D-CAM

^jFrom reference¹⁷

^k3D-CAM dementia status determined by an expert panel’s assessment of operational reference standard; 3D-CAM includes 1 participant with lifelong developmental cognitive limitations. READI dementia status based on a combination of medical record review, patient and proxy self-report, and AD8 dementia screening instrument¹⁸ administered to proxies.

Comparative Accuracy:

The sensitivity and specificity of the full 3D-CAM was 0.95 and 0.94, as previously reported⁸. Compared to the full 3D-CAM in all patients, the new delirium screening protocols performed well. Under the simulation scenarios, the skip pattern does not change the CAM Features being triggered, or the final delirium determination, and is not expected to impact sensitivity or specificity. Simulations of the 3D-CAM with skip therefore had the same sensitivity and specificity as the 3D-CAM. Simulations of protocols using the UB-2 screener had slightly decreased sensitivity (0.93 with 95% confidence intervals [CI] of 0.81 – 0.99 compared to 0.95 with CI of 0.84 – 0.99), but slightly improved specificity (0.95 with CI of 0.90 – 0.98 compared to 0.94 with CI of 0.90 – 0.97) relative to the 3D-CAM.

Time Per Question:

We calculated median administration time for each item in the 3D-CAM assessment (Table 2). Items 1-10 are administered directly to the patient and reflect time that the clinician spends with the patient (Face-to-face Items). Administration times for these items ranged from a minimum of 6 seconds, “Please tell me what type of place this is? [hospital]” to a maximum of 32 seconds, “Please tell me the months of the year backwards.” Items 11-20 are observation items completed by the clinician. Because there was not a separate keystroke on the app to account for the transition time from the patient interview to observation rating items, we estimated the time for Question 11a from Question 11b, which is a very similar observational question for altered level of consciousness. The remaining time we called “transition time,” and included this in all simulations that included both patient interview and observation questions. All observation items took between 2 to 4 seconds to complete.

Table 2:

Administration Times of Individual 3D-CAM Items

Item	Question	Median, s [Q1, Q3]	Mean, s (SD)
1	Please tell me the day of the week (UB-2 Item 1)	8 [6,13]	14.3 (20.9)
2	Please tell me the months of the year backwards, starting with December (UB-2 Item 2)	32 [24,49]	41.2 (34.0)
3	Please tell me the year we are in right now	9 [6,14]	12.5 (12.3)
4	Please tell me what type of place is this? (hospital)	6 [5,9]	8.81 (7.8)
5	I am going to read some numbers. I want you to repeat them in backwards order from the way I read them to you. For instance, if I say “5-2,” you would say “2-5.” OK? The first one is “7-5-1” (1-5-7)	26 [22,34]	30.48 (14.1)
6	The second is “8-2-4-3” (3-4-2-8)	15 [11,21]	17.7 (11.4)
7	Please tell me the days of the week backwards, starting with Saturday	20 [15,34]	27.8 (22.0)
8	During the past day have you felt confused?	9 [7,13]	11.8 (9.7)
9	During the past day did you think that you were not really in the hospital?	8 [6,10]	10.4 (9.9)
10	During the past day did you see things that were not really there?	7 [6,9]	8.7 (9.3)
--	lTransition time--interview to observational items	20 [8,43]	83.8 (592.4)
11am	Was the patient sleepy during the interview?	m2 [2,3]	m 3.1 (2.4)
11b	Was the patient stuporous or comatose during the interview?	2 [2,3]	3.1 (2.4)
12	Did the patient show excessive absorption with ordinary objects in the environment (hypervigilant)?	3 [2,4]	4.2 (5.4)
13	Was the patient’s flow of ideas unclear or illogical, for example tell a story unrelated to the interview (tangential)?	3 [2,5]	4.7 (5.7)
14	Was the patient’s conversation rambling, for example did he/she give inappropriately verbose and off-target responses?	3 [2,4]	4.3 (10.1)
15	Was the patient’s speech unusually limited or sparse (e.g. yes/no answers)?	3 [2,4]	3.7 (3.3)
16	Did the patient have trouble keeping track of what was being said during the interview?	4 [3,6]	5.3 (4.3)
17	Did the patient appear inappropriately distracted by environmental stimuli?	3 [2,4]	3.6 (2.8)
18	Did the patient’s level of consciousness fluctuate during the interview, for example, start to respond appropriately and then drift off?	3 [2,4]	13.3 (47.1)
19	Did the patient’s level of attention fluctuate during the interview, e.g., did the patient’s focus on the interview or performance on the attention tasks vary significantly?	3 [2,6]	5.2 (6.5)
20	Did the patient’s speech/thinking fluctuate during the interview, for example, patient spoke slowly, then spoke very fast?	3 [2,5]	4.0 (3.5)

^lTransition time from the patient interview to observational rating items was included in all simulations (with or without skip) that included both patient interview and observation questions.

^mTimes for question 11a were estimated from question 11b.

Comparative Efficiency:

Average time for the entire assessment, and for face-to-face administration time not including the observation items for each protocol are reported in Table 3. Average total time for the full 3D-CAM was 3 minutes and 13 seconds, with a face-to-face time of 2 minutes 20 seconds. Total times for the 3D-CAM with skip pattern, UB-2 followed by full 3D-CAM in ‘positives,’ and UB-2 followed by 3D-CAM with skip pattern in ‘positives,’ were significantly shorter than the full 3D-CAM total time. The UB-2 followed by 3D-CAM with skip pattern protocol was fastest, with a total administration time of 1 minute 14 seconds, and a face-to-face administration time of 56 seconds. This approach leads to a reduction in total administration time of 1 minute and 59 seconds over the full 3D-CAM (p<.001).

Table 3:

Mean Time and Comparative Efficiency of 4 Delirium Assessment Protocols

Protocol Times
Assessment Protocol	Mean Number of Items Administered	Mean Total Time, minutes:seconds (Q1, Q3)	Mean Face-to-face Items Time, minutes:seconds (Q1, Q3)
3D-CAM in all patients	20	3:13 (2:20, 4:56)	2:20 (1:48, 3:26)
3D-CAM with skip pattern in all patients	15	2:19 (1:43, 3:33)	1:48 (1:23, 2:42)
UB-2 plus 3D-CAM in ‘positives’	10.5	1:52 (1:22, 2:53)	1:27 (1:07, 2:10)
UB-2 plus 3D-CAM with skip pattern in ‘positives’	6	1:14 (0:52, 1:56)	0:56 (0:42, 1:24)
Comparative Efficiency
3D-CAM in all patients	Reference Assessment	Mean Time Saved	P-Valuen
3D-CAM with skip pattern in all patients	3D-CAM	54 seconds	<.001
UB-2 plus 3D-CAM in ‘positives’	3D-CAM	1 minute 21 seconds	<.001
UB-2 plus 3D-CAM with skip pattern in ‘positives’	3D-CAM	1 minute 59 seconds	<.001

ⁿP-values generated from non-parametric Wilcoxon Rank-Sum pairwise analysis

Stratified Analyses:

To further evaluate the most efficient delirium screening approach, the UB-2 followed by 3D-CAM with skip pattern in ‘positives,’ we completed a subset analysis stratified by dementia and delirium status. We found that total administration time was the shortest for patients with no dementia or delirium at 1 minute, 4 seconds. Patients with dementia only, delirium only, or both delirium and dementia required longer to assess at 1 minute 30 seconds for all three groups. The shorter administration time in patients without delirium or dementia may be attributable to these patients passing the UB-2 screen more often and not requiring the additional questions in the 3D-CAM.

DISCUSSION:

Detecting delirium is clinically important, but the time required to administer screening tools may be a major barrier toward implementation in clinical care. The 3D-CAM is a validated instrument for assessing CAM-defined delirium and takes about three minutes to administer. The present study sought to further reduce the time required to administer a delirium identification protocol by introducing an ultra-brief screener, the UB-2¹⁰, and a skip pattern for the 3D-CAM. Both of these reduced the total number of items administered, and the total assessment time. The UB-2 followed by 3D-CAM with skip pattern in ‘positives’ protocol was fastest, reducing the clinician time required for delirium assessment by 1 minute and 59 seconds. With a total administration time of 1 minute and 14 seconds, this approach may be a quick and useful protocol for time-conscious clinicians aiming to improve delirium detection in their patients.

The UB-2 followed by 3D-CAM with skip protocol performs well compared to other tools for brief delirium screening. The 4AT, a four-item screening tool that includes measures of alertness, cognition via the Abbreviated Mental Test 4 (AMT-4), attention via months of the year backwards, and acute change or fluctuation in mental status, takes on average <2 minutes complete and has a sensitivity of 89.7% and specificity of 84.1%⁹. Tools commonly used in other settings perform similarly to the UB-2, followed by 3D-CAM with skip pattern. The CAM-ICU takes an average of 55 seconds¹² to complete and has sensitivity of 100% and specificity of 83% when performed by an intensivist in the ICU setting⁶. The b-CAM, developed from the CAM-ICU, has sensitivity of 84% and specificity of 96% when administered by an Emergency Department physician following a positive Delirium Triage Screen (DTS)⁵. It is important to note that performance characteristics reported here and in the literature are unique to their study populations and will vary across settings. Of the above tools, only the 4AT and 3D-CAM are designed for general medical patients.

Our results suggest that use of the UB-2 followed by 3D-CAM with skip pattern in ‘positives’ consolidates clinician time on patients who warrant the most thorough clinical evaluation. The UB-2 quickly screens out cognitively intact patients who get both questions correct, such that more detailed assessments are administered only to patients who demonstrate some degree of cognitive impairment. For those who fail the UB-2 screen and move onto the 3D-CAM, incorporation of the skip pattern quickly identifies delirium in patients who are very impaired, since questioning for a particular feature stops once the patient gets one item incorrect. Thus, in the UB-2 followed by 3D-CAM with skip pattern protocol, patients who are neither completely intact nor very impaired receive the most administration items. Spending more time with these persons allows clinicians to develop a better sense of their cognitive status, which will assist them in completing the observation items, and ultimately making the determination of the presence of delirium.

Administration of the UB-2 and 3D-CAM with skip pattern is an example of adaptive testing, which has been used to improve the efficiency of other assessments in older patients^13–16. Such testing is facilitated by use of a computer-based platform where automatic skip patterns can be pre-programmed, and only the relevant items presented to the clinician to ask the patient. The “Delirium-READI” iPad App that we developed for the READI study is an example of a platform that facilitates adaptive testing.

Our approach has several limitations. Most importantly, the items administered under the various protocols were based on computer simulations. Future research is needed to prospectively validate our results. During actual administration, reducing the time clinicians spend with patients in skip pattern protocols may affect how clinicians rate observation items. For example, a clinician may miss a transient symptom or fluctuating course if he or she is in the room only briefly. It is also possible that earlier questions may influence a patient’s ability to answer later questions through a priming effect (e.g. completing digit span 3 backwards may improve performance on digit span 4 backwards). If such an effect exists, using results from administration of the full 3D-CAM to estimate performance in protocols that use skip patterns might not be entirely accurate. This further emphasizes the need to prospectively validate our results, which we are currently doing.

The item administration times that we report may be overestimates if these approaches were used to broadly screen a hospitalized population. Our study samples were purposefully older than the general hospitalized population and had a high proportion of persons with delirium and dementia. As demonstrated in our stratification analysis, assessing patients with delirium, dementia, or both takes longer than assessing cognitively intact patients. Therefore, younger, more cognitively intact general medicine populations may complete the delirium identification protocol faster than our study results reflect. Finally, the results may vary in different populations, such as where the delirium rates are higher or lower, or the spectrum of delirium is different than the general medicine wards (e.g. post-anesthesia recovery unit).

Conclusion:

A delirium screening protocol consisting of the UB-2, followed in “positives” by the 3D-CAM with skip pattern, or Ultra-Brief CAM (UB-CAM) has high sensitivity and specificity and can be completed in slightly over 1 minute. If prospectively validated, this approach holds promise for increasing implementation of systematic screening and improving detection of delirium in hospitalized older adults. Future research prospectively assessing the speed and accuracy of different types of clinicians administering this protocol will provide an important evidence base for implementation of systematic hospital-wide delirium identification.

Figure 1:

Study Design, Items Administered, and Mean Times of 4 Delirium Assessment Protocols

Impact Statement:

We certify that this work is novel. This research specifically adds to the literature by providing data supporting new delirium identification protocols. These protocols demonstrate that it is possible to decrease the time required for screening while maintaining accuracy. Adoption of these protocols may increase systematic screening and delirium detection in hospitalized older adults.