Abstract
BACKGROUND:
Recently, the Ultra-Brief Confusion Assessment Method (UB-CAM), designed to help physicians and nurses to recognize delirium, showed high, but imperfect, accuracy compared with Research Reference Standard Delirium Assessments (RRSDAs). The aim of this study is to identify factors associated with disagreement between clinicians’ app-based UB-CAM assessments and RRSDAs.
METHODS:
Design:
Secondary analysis of a prospective diagnostic test study
Setting:
2 Hospitals
Participants:
527 inpatients (≥70 years old), 289 clinicians (53 physicians, 236 nurses)
Measurements:
Trained research associates performed RRSDAs and determined delirium presence using the CAM. Clinicians administered the UB-CAM using an iPad app. Disagreement factors considered were clinician, patient, and delirium characteristics. We report odds ratios and 95% confidence intervals.
RESULTS:
1795 clinician UB-CAM assessments paired with RRSDAs were administered. The prevalence of delirium was 17%. The rate of disagreement between clinician UB-CAM assessments and RRSDAs was 12%. Significant factors associated with disagreement between clinician UB-CAM assessments and RRSDAs (OR [95% CI]) included: presence of dementia (2.7 [1.8–4.1]), patient education high school or less (1.9 [1.3–2.9]), psychomotor retardation (2.5 [1.4–4.2]), and the presence of mild delirium or subsyndromal delirium (5.5 [3.5–8.7]). Significant risk factors for false negatives were patient age less than 80 (2.2 [1.1–4.3]) and mild delirium (3.5 [1.6–7.4]). Significant risk factors for false positives were presence of dementia (4.0 [2.3–7.0]), subsyndromal delirium (5.1 [2.9–9.1]), and patient education high school or less (2.0 [1.2–3.6]). Clinician characteristics were not significantly associated with disagreement.
CONCLUSIONS:
The strongest factors associated with disagreement between clinician UB-CAM screens and RRSDAs were the presence of dementia and subsyndromal delirium as risk factors for false positives, and mild delirium and younger age as a risk factor for false negatives. These disagreement factors contrast with previous studies of risk factors for incorrect clinician delirium screening, and better align screening results with patient outcomes.
Keywords: delirium, screening, accuracy
INTRODUCTION:
Delirium has a profound impact on patient outcomes by increasing the risk of cognitive decline, functional decline, and mortality1,2,3. Accurate detection of delirium improves patient management and outcomes, yet more than half of cases are missed in routine care.4,5 Structured tools help clinicians screen for and detect delirium. The widely-used Confusion Assessment Method (CAM) diagnostic algorithm requires acute onset or fluctuating course, inattention, and either disorganized thinking or an altered level of consciousness.6 Several cognitive assessments to score the CAM diagnostic algorithm have been created7–9, including the UB-CAM, which uses a 2-step approach, with an ultra-brief 2-item screen (the UB-2), followed in “positives” by the 3D-CAM with skip pattern9,10.
We previously found that clinicians administering the UB-CAM facilitated by an iPad app was feasible (97% completion on eligible days), quick (completed in slightly over 1 minute) and accurate (88% relative to a reference standard)10. The current study extends that work, with the aim of identifying factors associated with disagreement between clinicians’ (physicians and nurses) UB-CAM assessments and Research Reference Standard Delirium Assessments (RRSDAs).
METHODS
Using data from the Researching Efficient Approaches to Delirium Identification (READI) study10, we compared the UB-CAM assessments performed by 289 physicians and nurses with RRSDAs performed by trained research associates on the same day in hospitalized adults aged 70 and older. Details on the READI study design and sampling and test methods have been published and are found in the supplementary material10. The study protocol and informed consent were approved by the Institutional Review Boards of participating hospitals.
Factors we examined for association with disagreement between clinician UB-CAM assessments and RRSDA included patient, delirium, and clinician characteristics. Patient characteristics included: age, gender, higher comorbidity determined by a Charlson comorbidity score11 ≥ 2, presence of dementia (presence of Alzheimer’s Disease or Related Dementias (ADRD)), ethnicity/race, education level, and native English speaker status. Delirium characteristics, determined by RRSDA ratings, included: mild delirium (CAM positive delirium with CAM-S severity score12 less than 8), subsyndromal delirium (presence of CAM Feature 1, plus one of the other major features (2, 3, or 4) plus one or more minor features (5–10), and the presence of psychomotor retardation (feature 9 on long CAM). Finally, we examined clinician characteristics including provider type (nurse vs. physician), gender, and clinician years of service at the hospital (excluding training).
Statistical Analysis and Sample Size
Using the clinician UB-CAM assessments paired with RRSDAs as the unit of analysis, we examined whether the factors noted above were associated with disagreement on the presence or absence of delirium. The RRSDA determination was used as the criterion standard for the diagnosis of delirium. Further, we examined factors associated with clinician false negatives and false positives.
Statistical modeling was performed using generalized linear models with logit link and binomial error distribution with the outcome being clinician-RRSDA disagreement and the independent variables being the patient, delirium, and clinician characteristics described above. We report the adjusted odds ratios and 95% confidence intervals. We took patient cluster effect into account (due to repeated assessments for each patient) by treating patients as random effects.13 We addressed potential cluster effect by clinician (multiple assessments performed by the same clinician) by using fixed effects. Given the high degree of RRSDA inter-rater reliability (kappa=0.96 in 89 paired assessments), we found no clustering by reference standard assessor; thus, this was not further considered in the modeling.
The target sample size for READI was 500 patients and 900 RRSDAs (500 Day 1 and 400 Day 2 assessments), which was needed to provide precise estimates of clinician UB-CAM sensitivity, specificity, and overall accuracy, relative to RRSDAs, accounting for clustering as described above. We used SAS statistical analysis software version 9.4, Cary NC, specifically the GENMOD procedure, for all analyses.
RESULTS
Patient and Clinician Characteristics:
527 patients were enrolled, and their demographic characteristics are described in Supplementary Table S1. The mean age was 80 years with a 35% prevalence of preexisting ADRD. Additionally, 53 physicians and 236 nurses were enrolled, and their demographic characteristics are reported in Supplementary Table S2. Despite excluding trainees, the clinicians were relatively young (mean age physicians=36.2 years, nurses=30.7 years). Few were certified in geriatrics or gerontology, but the majority reported experience caring for patients with delirium.
Diagnostic Accuracy of Clinician Assessments:
All 527 patients had a “day 1” RRSDA, and 397 had a “day 2” RRSDA, totaling 924 RRSDAs. Most patients were assessed by both a physician and a nurse on the same RRSDA day; 902 nurse and 893 physician assessments were performed over the 924 days—for a total of 1795 paired clinician UB-CAM - RRSDA assessments. A flow diagram describing patient enrollment and clinician UB-CAM assessments paired with RRSDAs is shown in Figure 1.
Figure 1. STARD Flowchart of Clinician Ultra-Brief Confusion Assessment Method (UB-CAM) Delirium Assessments paired with Research Reference Standard Delirium Assessments (RRSDA).
Flow diagram of recruitment and diagnostic classification of study subjects enrolled in the READI study according to the Standards for Reporting of Diagnostic Accuracy (STARD) studies.
1795 Clinician UB-CAM assessments (902 by nurses, 893 by physicians) paired with RRSDA delirium assessments were completed over 924 days. 1578 (87.9%) of the clinician UB-CAM delirium assessments agreed with the RRSDAs. 191 of the concordant delirium assessments were true positives and 1387 were true negatives. 217 (12.1%) of the clinician UB-CAM assessments disagreed with the RRSDAs. 106 of the discordant assessment were false negatives, and 111 were false positives.
Figure Abbreviations: RRSDA= Research Reference Standard Delirium Assessment; UB-CAM= Ultra-Brief Confusion Assessment Method
Color Codes: Blue boxes=screening, enrollment, and assessment completion data; Green boxes=agreement between clinician UB-CAM screens and RRSDAs; Red Boxes=disagreement between clinician UB-CAM screens and RRSDAs.
Among the 1795 paired clinician-RRSDA assessments, 297 (16.5%) of the RRSDAs were delirium positive. 1578 (87.9%) of clinician UB-CAM assessments agreed with the RRSDAs, while 217 (12.1%) disagreed. Of the 1578 concordant assessments, 191 were true positives, while 1387 were true negatives. Of the 217 discordant assessments, 106 were false negatives, and 111 were false positives. Clinician UB-CAM assessments had sensitivity of 63.2% and specificity of 92.9% for detecting delirium. The positive predictive value was 64.3% and the negative predictive value was 92.5% (Supplementary Table S3). Clinicians performed varying numbers of assessments with a varied percent being RRSDA delirium positive. However, Supplementary Figure S1 shows that neither of these factors influenced accuracy.
Factors Associated with Overall Disagreement:
Table 1 presents descriptive statistics of the 1795 paired clinician UB-CAM-RRSDA assessments by clinician, patient, and delirium characteristics, overall, and by agreement-disagreement categories (concordant, discordant, false negative, false positive). Table 2 presents the modeling results examining significant factors of disagreement, both overall, and within false negative and false positive categories. Clinician characteristics, such as provider type, gender, and years of service were not significantly associated with disagreement. However, patient characteristics and delirium characteristics were significantly associated with disagreement.
Table 1.
UB-CAM Assessments Based on Clinician, Patient, and Delirium Characteristics
| Factor | Total Assessments N=1795b | Concordant UB-CAM Assessments N=1578 | Discordant UB-CAM Assessments N=217 | False Negatives N=106 | False Positives N=111 | |
|---|---|---|---|---|---|---|
| Clinician Factors, n (%) | Type: Nurse | 902 (50.3) | 802 (88.9) | 100 (11.1) | 52 (5.8) | 48 (5.3) |
| Type: Physician | 893 (49.7) | 776 (86.9) | 117 (13.1) | 54 (6.0) | 63 (7.1) | |
| Gender: Female | 1043 (58.1) | 930 (89.2) | 113 (10.8) | 58 (5.5) | 55 (5.3) | |
| Gender: Male | 752 (41.9) | 648 (86.2) | 104 (13.8) | 48 (6.4) | 56 (7.4) | |
| Years of Service ≥5 | 992 (55.3) | 872 (88) | 120 (12) | 60 (6.0) | 60 (6.0) | |
| Years of Service <5 | 803 (44.7) | 706 (87.9) | 97 (12.1) | 46 (5.7) | 51 (6.4) | |
| Patient Factors, n (%) | Charlson Score 0 or 1a | 805 (44.8) | 716 (88.9) | 89 (11.1) | 40 (5.0) | 49 (6.1) |
| Charlson Score ≥ 2a | 990 (55.2) | 862 (87.0) | 128 (13.0) | 66 (6.7) | 62 (6.3) | |
| Without ADRD | 1164 (64.8) | 1079 (92.7) | 85 (7.3) | 47 (4.0) | 38 (3.3) | |
| Presence of ADRD | 631 (35.2) | 499 (79.1) | 132 (20.9) | 59 (9.3) | 73(11.6) | |
| Gender: Male | 1019 (56.8) | 909 (89.2) | 110 (10.8) | 51(5.0) | 59 (5.8) | |
| Gender: Female | 776 (43.2) | 669 (86.2) | 107 (13.8) | 55 (7.1) | 52 (6.7) | |
| Age ≥80 | 823 (45.8) | 711 (86.4) | 112 (13.6) | 42 (5.1) | 70 (8.5) | |
| Age <80 | 972 (54.2) | 867 (89.2) | 105 (10.8) | 64 (6.6) | 41 (4.2) | |
| Native English Speaker, n(%)c | 1597 (91.3) | 1408 (88.2) | 189 (11.8) | 90 (5.6) | 99 (6.2) | |
| English as a Second Languagec | 152 (8.7) | 130 (85.5) | 22 (14.5) | 10 (6.6) | 12 (7.9) | |
| Education High school graduate or lessc | 825 (46.3) | 702 (85.1) | 123 (14.9) | 60 (7.3) | 63 (7.6) | |
| Education Some college education or morec,d | 956 (53.7) | 863 (90.3) | 93 (9.7) | 46 (4.8) | 47 (4.9) | |
| Ethnicity/race: non-Hispanic, Whitec | 1569 (87.5) | 1381 (88.0) | 188 (12.0) | 90 (5.7) | 98 (6.3) | |
| Ethnicity/race: Hispanic/ Non-whitec | 224 (12.5) | 195 (87.1) | 29 (12.9) | 16 (7.1) | 13 (5.8) | |
| Delirium Factors, n (%) | No Delirium Features/ Severe Delirium | 1482 (82.6) | 1386 (93.5) | 96 (6.5) | 38 (2.6) | 58 (3.9) |
| Mild/Subsyndromal Delirium | 313 (17.4) | 192 (61) | 121 (39) | 68 (22) | 53(17) | |
| Presence of Psychomotor retardation | 237 (13.2) | 175 (73.8) | 62 (26.2) | 53 (22.4) | 9 (3.8) | |
| Absence of Psychomotor retardation | 1558 (86.8) | 1403 (90.1) | 155 (9.9) | 53 (3.4) | 102(6.5) |
Abbreviations: UB-CAM= Ultra-Brief Confusion Assessment Method (performed by clinicians); ADRD= Alzheimer’s Disease and Related Dementias
Table Footnotes:
Charlson score calculated without Alzheimer’s Disease since Alzheimer’s considered separately in the analysis.
1498 of the assessments were delirium negative and 297 were delirium positive based on Research Reference Standard Delirium Assessments (RRSDAs)
Number of missing data points: Patient Language=46, Patient Education=14, Patient ethnicity/race= 2
Some college education or more includes; some college, associate degree, bachelor’s degree, graduate degree, master’s degree, and doctoral degree.
Percentages in the second column use the total paired assessments as the denominator (N=1795) (Factors with missing assessments use the total numbers of assessments with data as the denominator for the percentage in the second column). Percentages in the remaining columns (3–6) use the n from column 2 as the denominator (total assessments performed within a specific factor).
Table 2.
Factors associated with Disagreement, False Negative and False Positive
| Factors | Disagreement Adjusted OR (95% CI) | False Negative Adjusted OR (95% CI) | False Positive Adjusted OR (95% CI) |
|---|---|---|---|
| Patient age | |||
| <80 | 1.1 (0.7–1.6) | 2.2 (1.1–4.3) | 0.6 (0.3–1.0) |
| >=80 | Reference | Reference | Reference |
| Patient gender | |||
| Male | 1.3 (0.9–2.0) | 1.1 (0.6–2.2) | 1.2 (0.7–2.2) |
| Female | Reference | Reference | Reference |
| Alzheimer’s disease and related dementias | |||
| Yes | 2.7 (1.8–4.1) | 0.7 (0.3–1.3) | 4.0 (2.3–7.0) |
| No | Reference | Reference | Reference |
| Charlson score without Alzheimer’s disease | |||
| Yes | 1.4 (0.9–2.2) | 1.8 (0.9–3.5) | 1.2 (0.7–2.1) |
| No | Reference | Reference | Reference |
| Patient Language | |||
| English as a Second Language | 0.8 (0.5–1.4) | 0.6 (0.3–1.6) | 0.9 (0.5–1.8) |
| Native English Speaker | Reference | Reference | Reference |
| Patient Education | |||
| High School Graduate or less | 1.9 (1.3–2.9) | 1.1 (0.6–2.2) | 2.0 (1.2–3.6) |
| Some college or more | Reference | Reference | Reference |
| Patient Ethnicity/Race | |||
| Hispanic and/or Non-white | 0.9 (0.5–1.7) | 1.3 (0.4–4.4) | 0.6 (0.2–1.5) |
| White, Non-Hispanic | Reference | Reference | Reference |
| Borderline Delirium Status (Mild Delirium, Subsyndromal Delirium) | |||
| Yes | 5.5 (3.5–8.7) | 3.5 (1.6–7.4) | 5.1 (2.9–9.1) |
| No | Reference | Reference | Reference |
| Psychomotor Retardation | |||
| Yes | 2.5 (1.4–4.2) | 1.7 (1.0–3.0) | 1.0 (0.3–3.4) |
| No | Reference | Reference | Reference |
| Clinician gender | |||
| Male | 1.4 (0.9–2.1) | 1.1 (0.5–2.1) | 1.5 (0.9–2.6) |
| Female | Reference | Reference | Reference |
| Clinician years of service | |||
| <5 | 1.1 (0.8–1.5) | 0.9 (0.5–1.7) | 1.2 (0.8–1.9) |
| >=5 | Reference | Reference | Reference |
| Clinician type | |||
| Physician | 1.0 (0.7–1.5) | 1.1 (0.6–1.9) | 1.0 (0.7–1.7) |
| Nurse | Reference | Reference | Reference |
Table Abbreviations: OR= Odds Ratio; CI= Confidence Interval
Specifically, we found the following factors increased the odds of disagreement between clinician UB-CAM assessments and RRSDAs: the presence of ADRD with an adjusted odds ratio of 2.7 (95% CI=1.8–4.1), lower education level with an adjusted odds ratio of 1.9 (1.3–2.9), psychomotor retardation with an adjusted odds ratio of 2.5 (1.4–4.2), and mild or subsyndromal delirium with an adjusted odds ratio of 5.5 (3.5–8.7).
Factors Associated with False Negatives and False Positives:
Among the RRSDA positives, two factors had statistically significant associations with the probability of clinicians obtaining false negative delirium assessments: age less than 80 with an adjusted odds ratio of 2.2 (95% CI=1.1–4.3), and mild delirium with an adjusted odds ratio of 3.5 (1.6–7.4). Among RRSDA negatives, three factors had statistically significant associations with the probability of clinicians obtaining false positive delirium assessments: patient with high school education or less with an adjusted odds ratio of 2.0 (95% CI=1.2–3.6), presence of ADRD with an adjusted odds ratio of 4.0 (2.3–7.0), and presence of subsyndromal delirium with an adjusted odds ratio of 5.1 (2.9–9.1).
DISCUSSION
In this study of 1795 paired assessments, we found a 12% rate of disagreement between clinicians using the app-directed UB-CAM protocol and a RRSDA delirium diagnosis. We identified factors associated with disagreement, with the strongest risk factors being mild delirium and age under 80 for false negatives and subsyndromal delirium, presence of dementia, and low educational status for false positives. Our findings point out the challenges of clinician delirium screening with ultra-brief assessments, and have important implications for improving their accuracy.
Two prior studies examined nurse CAM ratings with reference standard assessments by researchers14,15. In both studies nurses performed no cognitive testing, but rated the presence or absence of CAM diagnostic features based on observations from routine care and clinical judgment. Results showed very low sensitivity (30%) but high specificity (over 95%) with risk factors for under-recognition being hypoactive delirium, age ≥80, dementia, vision impairment, and prolonged hospital length of stay. Thus, poor sensitivity occurred in the very patients most likely to have bad outcomes following delirium-- those over age 80 and with dementia. These findings may represent an inherent clinician bias that confusion (or worsened confusion) in the hospital is “normal” in such patients.
In contrast, the app-directed UB-CAM protocol used in READI employs a structured approach, with cognitive testing items built into assessment of each of the CAM features (e.g. months of the year backwards for inattention, day of the week for disorganized thinking) and fixed thresholds (1 item incorrect) for triggering the presence of each CAM feature. Such an approach, also used in assessments like the CAM-ICU7, B-CAM8, and 4AT16, improves the sensitivity of clinician assessments relative to unstructured screening, with a modest reduction in specificity. More importantly, and supported by the risk factors for disagreement identified in the current analysis, the UB-CAM better aligns incorrect clinician screening results with patient risk, with risk factors associated with false negatives, (age under 80 and mild delirium) being associated with better overall patient outcomes, and factors associated with false positives (dementia and low education) being associated with worse outcomes.
Additionally, our results identify opportunities to improve the accuracy of delirium screening. Using simpler attention items (e.g., days of the week backwards instead of months of the year backwards) could improve accuracy of CAM Feature 2 in persons with dementia or low education. Integrating a proxy interview using tools such as the Family Confusion Assessment Method (FAM-CAM)17, or even a single screening item for acute change, could improve assessment of CAM Feature 1 in persons with dementia. Finally, as noted above, individuals with dementia, lower education, and subsyndromal delirium are at risk for poor outcomes in the setting of acute illness even in the absence of delirium18–20. Thus, the added clinician attention warranted by a positive delirium screen in these more vulnerable individuals is appropriate, whether or not it is a “true positive” or “false positive”.
Strengths of our study included the large sample size of 527 patients and 1795 paired clinician UB-CAM-RRSDA delirium assessments. The enriched patient sample with advanced age and ADRD, enhances our ability to examine the impact these factors have on clinician screening accuracy. The RRSDA performed by research associates was rigorous involving direct patient cognitive testing, interview of a proxy, and detailed medical record review. Moreover, the RRSDAs had very high inter-rater reliability (99% for delirium presence), providing further evidence of validity of the reference standard.
The study had limitations. Each clinician completed a variable number of interviews and assessed patients with varying delirium characteristics, but we found neither affected accuracy (Supplementary Figure 1). The research was performed at two hospitals with primarily English-speaking patients, which may limit generalizability. Additionally, the only delirium screening tool used was the UB-CAM assessment, also limiting generalizability. Although interviewers had access to pocket amplifiers for patients with difficulty hearing, we did not analyze the impact of vision or hearing impairment on delirium screening accuracy, which are factors that have been previously shown to be related to under-recognition of delirium14. Finally, nurses and physicians enrolled in READI had only a 20–30-minute training session on the UB-CAM app. More detailed specific training in administration of delirium assessments could lead to improved clinician sensitivity for subtle delirium signs and symptoms.
CONCLUSION:
In our study of 1795 clinician UB-CAM screens paired with RRSDAs, clinicians disagreed with RRSDAs on the presence or absence of delirium in 12% of assessments. Nurses were as accurate as physicians, and no other clinician factors influenced accuracy. Patient and delirium characteristics associated with false positives: dementia, low education, and subsyndromal delirium, are generally associated with worse outcomes regardless of delirium status, while characteristics associated with false negatives: younger age and mild delirium, are associated with better outcomes. Thus, the clinician UB-CAM screening results are aligned with identifying the patients most at risk for bad hospital outcomes. Our findings suggest that brief and accurate delirium identification by clinicians is achievable. This is a key first step toward appropriate management, with a goal of improving outcomes for vulnerable hospitalized older adults.
Supplementary Material
Key Points.
When administering the UB-CAM clinicians disagreed with reference standard delirium assessments on the presence or absence of delirium in 12% of assessments.
The strongest factors associated with disagreement were the presence of dementia, subsyndromal delirium, and low education as risk factors for false positives, while younger age and mild delirium status were both risk factors for false negatives.
These disagreement factors better align with patient outcomes than previously reported disagreement factors for clinician delirium screening, with risk factors for false positives occurring in more vulnerable individuals (those with advanced age and dementia), and risk factors for false negatives occurring in less vulnerable individuals.
Why Does this Paper Matter?
This research specifically adds to the literature by identifying factors associated with disagreement between clinician delirium assessments and research reference standard delirium assessments. These findings demonstrate that a structured approach to delirium screening such as the UB-CAM with integrated cognitive testing and fixed thresholds for positive screens may be superior to screening methods that rely on clinician judgment. Additionally, the identified disagreement factors provide opportunities to further improve accuracy of delirium screening in the clinical setting.
Funding Sources:
NIA Grants: 5T35AG038027 (Dhliwayo), R01AG030618 (Marcantonio and Fick), K24AG035075 (Marcantonio), R33AG071744 (Inouye), and R01AG044518 (Inouye).
Dr. Inouye holds the Milton and Shirley F. Levy Family Chair at Hebrew SeniorLife/Harvard Medical School.
Sponsor’s Role:
The National Institute on Aging had no role in the design, conduct, analysis, or decision to submit the manuscript for publication.
Footnotes
Conflict of Interest: The authors have no conflict.
REFERENCES
- 1.Mattison MLP. Delirium. Ann Intern Med 2020; 173:ITC49–ITC64. [DOI] [PubMed] [Google Scholar]
- 2.Marcantonio ER. Delirium in hospitalized older adults. N Engl J Med 2017; 377:1456–1466. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Kinchin I, Mitchell E, Agar M, Trépel D. The economic cost of delirium: A systematic review and quality assessment. Alzheimer’s Dement 2021; 17:1026–1041. [DOI] [PubMed] [Google Scholar]
- 4.Lemiengre J, Nelis T, Joosten E et al. Detection of delirium by bedside nurses using the confusion assessment method. J Am Geriatr Soc 2006; 54:685–689. [DOI] [PubMed] [Google Scholar]
- 5.Gustafson Y, Brännström B, Norberg A, Bucht G, Winblad B. Underdiagnosis and poor documentation of acute confusional states in elderly hip fracture patients. J Am Geriatr Soc 1991; 39:760–765. [DOI] [PubMed] [Google Scholar]
- 6.Inouye SK, Van Dyck CH, Alessi CA et al. Clarifying confusion: the confusion assessment method. A new method for detection of delirium. Ann Intern Med 1990; 113: 941–948. [DOI] [PubMed] [Google Scholar]
- 7.Ely EW, Inouye SK, Bernard GR et al. Delirium in mechanically ventilated patients: validity and reliability of the confusion assessment method for the intensive care unit (CAM-ICU). JAMA 2001; 286:2703–2710. [DOI] [PubMed] [Google Scholar]
- 8.Han JH, Wilson A, Vasilevskis EE et al. Diagnosing delirium in older emergency department patients: validity and reliability of the delirium triage screen and the brief confusion assessment method. Ann Emerg Med 2013; 62:457–465. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Marcantonio ER, Ngo LH, O’Connor M et al. 3D-CAM: Derivation and validation of a 3-minute diagnostic interview for CAM-defined delirium: a cross-sectional diagnostic test study. Ann Intern Med 2014; 161:554–561. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Marcantonio ER, Fick DM, Jung Y et al. Comparative implementation of a brief app-directed protocol for delirium identification by hospitalists, nurses, and nursing assistants: a cohort study. Ann Intern Med 2022; 175:65–73. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Charlson ME, Pompei P, Ales KL, MacKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chronic Dis 1987; 40:373–383. [DOI] [PubMed] [Google Scholar]
- 12.Inouye SK, Kosar CM, Tommet D et al. The CAM-S: development and validation of a new scoring system for delirium severity in 2 cohorts. Ann Intern Med 2014; 160:526–533. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Kahan BC, Morris TP. Assessing potential sources of clustering in individually randomised trials. BMC Med Res Methodol 2013; 13:58. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Inouye SK, Foreman MD, Mion LC, Katz KH, Cooney LM. Nurses’ recognition of delirium and its symptoms: comparison of nurse and researcher ratings. Arch Intern Med 2001; 161:2467–2473. [DOI] [PubMed] [Google Scholar]
- 15.Rice KL, Bennett M, Gomez M, Theall KP, Knight M, Foreman MD. Nurses’ recognition of delirium in the hospitalized older adult. Clinical Nurse Specialist 2011; 25:299–311. [DOI] [PubMed] [Google Scholar]
- 16.Bellelli G, Morandi A, Davis DH et al. Validation of the 4AT, a new instrument for rapid delirium screening: a study in 234 hospitalised older people. Age Ageing 2014; 43:496–502. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Steis MR, Evans L, Hirschman KB et al. Screening for delirium using family caregivers: convergent validity of the Family Confusion Assessment Method and interviewer-rated Confusion Assessment Method. J Am Geriatr Soc 2012; 60:2121–2126. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Jones RN, Yang FM, Zhang Y, Kiely DK, Marcantonio ER, Inouye SK. Does educational attainment contribute to risk for delirium? A potential role for cognitive reserve. The Journals of Gerontology 2006; 61:1307–1311. [DOI] [PubMed] [Google Scholar]
- 19.Fick DM, Agostini JV, Inouye SK. Delirium superimposed on dementia: a systematic review. J Am Geriatr Soc 2002; 50:1723–1732. [DOI] [PubMed] [Google Scholar]
- 20.Serafim RB, Soares M, Bozza FA et al. Outcomes of subsyndromal delirium in ICU: a systematic review and meta-analysis. Crit Care 2017; 21:179. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.