Adaptation and Validation of a Chart‐Based Delirium Detection Tool for the ICU (CHART‐DEL‐ICU)

Abstract

OBJECTIVE

To adapt and validate a chart‐based delirium detection tool for use in critically ill adults.

DESIGN

Validation study.

SETTING

Medical‐surgical intensive care unit (ICU) in an academic hospital.

MEASUREMENTS

A chart‐based delirium detection tool (CHART‐DEL) was adapted for use in critically ill adults (CHART‐DEL‐ICU) and compared with prospective delirium assessments (i.e., clinical assessments (reference standard) by a research nurse trained by a neuropsychiatrist and routine delirium screening tools Confusion Assessment Method (CAM‐ICU)) and (Intensive Care Delirium Screening Checklist (ICDSC)). The original CHART‐DEL tool was adapted to include physician‐reported ICDSC score (for probable delirium) and Richmond‐Agitation Sedation Scale score (for altered level of consciousness and agitation). Two trained chart abstractors blinded to all delirium assessments manually abstracted delirium‐related information from medical charts and electronic medical records and rated if delirium was present (four levels: uncertain, possible, probable, definite) or absent (no evidence).

RESULTS

Charts were manually abstracted for delirium‐related information for 213 patients who were included in a prospective cohort study that included prospective delirium assessments. The CHART‐DEL‐ICU tool had excellent interrater reliability (kappa = 0.90). Compared to the reference standard, the sensitivity was 66.0% (95% CI = 59.3–72.3%) and specificity was 82.1% (95% CI = 78.0–85.7%), with a cut‐point that included definite, probable, possible, and uncertain delirium. The AUC of the CHART‐DEL‐ICU alone is 74.1% (95% CI = 70.4–77.8%) compared with the addition of the CAM‐ICU and ICDSC (CAM‐ICU/CHART‐DEL‐ICU: 80.9% (95% CI = 77.8–83.9%), P = .01; ICDSC/CHART‐DEL‐ICU: 79.2% (95% CI = 75.9–82.6%), P = .03).

CONCLUSION

A chart‐based delirium detection tool has improved diagnostic accuracy when combined with routine delirium screening tools (CAM‐ICU and ICDSC), compared to a chart‐based method on its own. This presents a potential for retrospective detection of delirium from medical charts for research or to augment routine delirium screening methods to find missed cases of delirium.

1. INTRODUCTION

Delirium is a neuropsychiatric syndrome that is common in critically ill adults during an intensive care unit (ICU) stay. ¹ Delirium is associated with adverse short‐term (longer ICU or hospital stay, increased odds of mortality) and long‐term outcomes (post‐intensive care syndrome). ² The Society of Critical Care Medicine recommends regular screening for delirium using a validated tool. ³ Regular screening is important for the early recognition of delirium symptoms which may prompt the ICU care team to use nonpharmacological and pharmacological strategies to prevent and manage delirium. ³ Studies report that routine delirium screening is associated with reduced in‐hospital mortality ⁴ and reduced patient anxiety. ⁵ Two validated tools are commonly used for routine delirium screening in the ICU: the Confusion Assessment Method for the Intensive Care Unit (CAM‐ICU) ⁶ , ⁷ and the Intensive Care Delirium Screening Checklist (ICDSC). ⁸ However, delirium is a clinical diagnosis and the sensitivity and specificity of these screening tools (compared to clinical diagnosis) vary across studies. ⁹ , ¹⁰ , ¹¹ , ¹² , ¹³

Key Points.

• Chart‐based ICU delirium detection alone has low sensitivity as not all delirium symptoms are recorded in the medical chart.

• A chart‐based ICU delirium detection tool has improved diagnostic accuracy when used in combination with routine delirium screening.

• Chart‐based ICU delirium detection can be used for retrospective detection of delirium from medical charts for research or to augment routine delirium screening.

Why Does this Paper Matter?

• A chart‐based delirium detection tool has not yet been validated in an adult intensive care unit setting. The current study demonstrates the utility of a chart‐based delirium detection tool for research or to augment routine delirium screening in critically ill adults.

A chart‐based delirium detection instrument (CHART‐DEL) was developed and validated in older hospitalized adults to either ¹ add to the delirium detection process by augmenting routine delirium screening and provide a 24 hour a day perspective; or ² simplify the process of delirium detection through the use of chart‐based reporting to identify delirium. ¹⁴ , ¹⁵ , ¹⁶ This chart‐based method for delirium detection includes reviewing a patient's chart for diagnoses synonymous with delirium or a change to the patient's baseline mental status as indicated by trigger words or phrases that are associated with a high positive predictive value for delirium. Chart‐based delirium detection has applicability to the ICU, where it may complement routine delirium screening as part of standard care. Moreover, the chart‐based method for delirium detection has the potential to augment ICU delirium research with critically ill patients where research staff can complete delirium assessments in addition to routine delirium screening that is already part of patient care. The present study aimed to adapt and validate the CHART‐DEL detection tool for use in a critically ill adult population in the ICU.

2. METHODS

2.1. Study Setting

The protocol followed the STAndards for the Reporting of Diagnostic accuracy studies (STARD) guidelines ¹⁷ (Figure 1). This validation study is nested in a cross‐sectional study (Clinicaltrials.gov: NCT03379129), where exposure (ICU) and outcome (delirium) were measured at the same time, although serial assessments were conducted and multiple observations per patient were included. ¹⁰ , ¹⁸ Patients were recruited from a 28‐bed, general systems ICU at Foothills Medical Center (FMC ICU) in Calgary, Canada (catchment population: 1.8 million) between November 2017 and March 2019. Patients were eligible for inclusion in the study if they were ≥18 years of age, had no primary direct brain injury (e.g., severe traumatic brain injury), were able to provide consent or surrogate consent, understood English and were expected to remain in the ICU for at least 24 hours. Patients were screened daily and were excluded on a given day if they had a Richmond Agitation‐Sedation Scale (RASS) of −4 or −5 ¹⁹ , ²⁰ or Glasgow Comma Scale of ≤9. ²¹ The study was approved by the Conjoint Health Research Ethics Board at the University of Calgary (Reference number: REB16‐2060).

Figure 1.

STARD (Standards of Reporting of Diagnostic Accuracy Studies) flow chart. FN, false negative; FP, false positive; TN, true negative; TP, true positive.

2.2. Adaptation of the CHART‐DEL Tool for the ICU

The CHART‐DEL tool was adapted to maximize sensitivity to improve the detection of delirium in critically ill adults. An extensive list of trigger words and phrases for delirium detection is listed in the CHART‐DEL training guide. ¹⁵ Before the tool was adapted, two chart abstractors (licensed physician assistant (ICU delirium research experience, clinical judgment, medical chart experience) and senior research associate (ICU delirium research experience, medical chart experience)), who were blinded to the delirium status of the patient, including clinician delirium assessment (i.e., RN‐documented ICDSC score), independently reviewed 10% (22/218) of the medical charts (e.g., progress notes, nursing notes in the electronic medical record (EMR), consult notes) and recorded evidence of acute confusion (i.e., delirium, agitation, disorientation, hallucinations, etc.) using the CHART‐DEL abstraction tool. The chart abstraction was overseen by the principal investigator (KF) and intensive care physician (HTS). The chart abstractors made notes about any extra information that was not included in the CHART‐DEL appendix of trigger words and phrases (e.g., medications initiated for the treatment of delirium). The chart abstractors met to discuss discrepancies between their abstractions and assessments, and if required, returned to the medical chart to review. They also met with ICU delirium experts (ICU physician, ICU pharmacist, ICU registered nurse, delirium researchers) via a working group to discuss potential modifications to the CHART‐DEL tool. Lastly, the study team met with a geriatrician (SI, developer of the CHART‐DEL tool) via teleconference to discuss the proposed changes that were incorporated into the ICU version of the CHART‐DEL tool: CHART‐DEL‐ICU.

2.3. Evaluation of the Validity of the CHART‐DEL Tool in the ICU

To evaluate the validity of the adapted CHART‐DEL‐ICU tool, we followed several steps. First, face validity was assessed through discussion with an expert panel (as detailed above). Second, the CHART‐DEL‐ICU was piloted in a subset of 20 randomly selected charts, where chart abstractors (physician assistant, senior research associate) independently abstracted and rated each chart as no evidence (<10% probability), uncertain (10–40% probability, ambiguous statements (e.g., patient confused)), possible (40–60% probability, two or more CAM features reported), probable (60–85% probability, positive ICU delirium screening tool) or definite delirium (>85% probability, confirmed diagnosis by experienced reference standard rater) (Supplementary Table S1). Chart abstractors were blinded to the delirium status of the patient, including clinician delirium assessment (i.e., RN‐documented ICDSC score). Charts where consensus was not reached were discussed or, if necessary, reviewed by a third assessor (KF). Lastly, two chart abstractors abstracted data for the remaining 198 charts. This included manual abstraction of data from the medical chart (e.g., physician progress notes) and electronic medical records (e.g., bedside registered nurses' daily synopsis and free text areas for comments on RASS from the neurological assessment) from ICU admission to ICU discharge. The chart abstractors recorded any mention of the trigger words (e.g., confus* (e.g., confusion, confused), alert and oriented < 3 (i.e., A + O x1 or A + O x2), hallucinations, disorient* (e.g., disoriented, disorientation)), and phrases, and coded delirium as yes/no (for a specific day) and classified their confidence of delirium being present based on the information contained in the medical chart as no evidence/uncertain/possible/probable/definite (Supplementary Table S1). The chart abstractors also recorded other possible supporting words (not included in the assessment of the CHART‐DEL‐ICU tool) such as antipsychotics (e.g., haloperidol, dexmedetomidine, olanzapine, quetiapine, zopiclone). To assess inter‐rater reliability, 10% of the charts were randomly selected (20/198) to be abstracted and rated independently and in duplicate. Concurrent (against reference standard ratings) validity was evaluated and included CHART‐DEL‐ICU alone or in combination with routine delirium screening tools (CAM‐ICU and ICDSC).

2.4. Clinical Delirium Assessments

Patients enrolled in the study were evaluated for delirium using routine ICU delirium screening tools and a reference standard delirium assessment, as described below.

2.4.1. Routine ICU Delirium Screening Tools

1. The CAM‐ICU was administered twice daily, morning (9:00–11:00 a.m.) and afternoon (2:00–4:00 p.m.), by trained research assistants (while the participant remained in the ICU for a maximum of 5 days). ¹⁰ , ¹⁸ The CAM‐ICU includes questions and/or commands to test for inattention and disorganized thinking, which are completed along with the RASS. ¹⁹ , ²⁰ The presence of acute onset or fluctuating course and inattention with either disorganized thinking or altered level of consciousness indicates the patient has delirium. The CAM‐ICU has a pooled sensitivity among nine studies of 80.0% (95% CI = 77.1–82.6%) and a pooled specificity of 95.9% (95% CI = 94.8–96.8%). ¹² The trained research assistants were blinded to the CHART‐DEL‐ICU delirium rating and the chart abstractors were blinded to the CAM‐ICU assessments.

2. The ICDSC, was completed twice daily, morning (6:00 a.m.) and afternoon (6:00 p.m.), as per standard patient care by a bedside registered nurse (during the entire ICU stay). Each item is rated based on the patient's symptoms over the previous 12 hours. A score of ≥4 (i.e., the presence of four or more items) indicates the patient has delirium. The ICDSC has a pooled sensitivity among four studies of 75% (95% CI = 65.3–81.5%) and a pooled specificity of 81.9% (95% CI = 76.7–86.4%). The bedside registered nurses were blinded to the CHART‐DEL‐ICU rating and the chart abstractors were blinded to the ICDSC delirium rating.

2.4.2. Reference Standard

• 3 A research nurse trained by a neuropsychiatrist assessed for delirium according to the Diagnostic and Statistical Manual of Mental Disorders (DSM‐5) ²² once daily, afternoon (2:00–4:00 p.m.) using a standardized form (while the participant remained in the ICU, for a maximum of 5 days). ¹⁸ The trained research nurse was blinded to the CHART‐DEL‐ICU rating, CAM‐ICU and ICDSC scores, and the chart abstractors were blinded to the DSM‐5 delirium rating.

2.5. Other Data and Clinical Outcomes

Baseline demographics such as age, gender, biological sex, severity of illness (Acute Physiology and Chronic Health Evaluation (APACHE) II, Sequential Organ Failure Assessment (SOFA) score upon ICU admission) and receipt of mechanical ventilation were collected during the study period.

2.6. Statistical Analyses

Descriptive statistics were examined for all study variables. Continuous variables with a normal distribution were presented as mean ± standard deviation (SD). The CHART‐DEL‐ICU (definite/probable, definite/probable/possible or definite/probable/possible/uncertain), CAM‐ICU, ICDSC, and reference standard were scored dichotomously (ever/never delirium) for daily delirium. The first reference standard on each patient‐day (that aligned with the time of the CAM‐ICU and ICDSC assessments) were used to evaluate validity. The criterion validity of CHART‐DEL‐ICU compared to the reference standard (DSM‐5) was evaluated by calculating sensitivity, specificity and area under the curve (AUC) and their accompanying 95% confidence intervals (95% CIs). To determine the value of the CHART‐DEL‐ICU tool in enhancing routine delirium screening, we compared the diagnostic accuracies between CHART‐DEL‐ICU alone versus CAM‐ICU/CHART‐DEL‐ICU or ICDSC/CHART‐DEL‐ICU combined. A random sample of the data was abstracted (10%, 20/198 remaining after the pilot) independently, in duplicate, to calculate interrater reliability, wherein overall agreement (as measured by the kappa) was interpreted as fair (0.21–0.40), moderate (0.41–0.60), substantial (0.61–0.80), and almost perfect (0.81–1.00). ²³ Logistic regression analysis was used to evaluate the relationship between trigger words or phrases and antipsychotic use with delirium identified by the reference standard. All analyses were performed using Stata, version 14.0 (StataCorp, College Station, TX, USA).

3. RESULTS

3.1. Population Characteristics

From November 2017 to March 2019, 910 patients were screened, 356 patients were eligible and 218 patients were enrolled, with a participation rate of 61% (218/356). The majority of patients were ineligible because they were not expected to remain in the ICU for at least 24 hours (n = 231), had a RASS less than −3 (n = 118), did not understand English (n = 67), were unable to provide consent or surrogate consent (n = 40) or were less than 18 years of age (n = 4). All reasons for ineligibility or exclusion are shown in Supplementary Figure S1. Of these 218 patients, 213 had complete CAM‐ICU, ICDSC and DSM‐5 assessments across multiple days, for a total of 628 assessments. Patient characteristics are presented in Table 1. Most patients were male (128/213, 60.1%) with a mean age of 56.7 years (SD 15.6). Most patients were admitted with a medical diagnosis (110/213, 51.6%), had a median APACHE‐II score of 20 (interquartile range (IQR) 11) and a median SOFA score of 7 (IQR 4). Most patients received mechanical ventilation (159/213, 74.6%) during their ICU stay.

Table 1.

Characteristic of Participants (n = 213)

Characteristic	All Patients (n = 213)	Patients with DSM‐5 Delirium (n = 92)
Age, mean (standard deviation)	56.7 (15.6)	58.2 (14.8)
Female sex, n (%)	85 (39.9)	29 (31.5)
Gender identity, woman, n (%)	85 (39.9)	29 (31.5)
ICU length of stay, median (IQR)	8.8 (10.5)	12.3 (11.2)
APACHE‐II, median (IQR)	20 (10)	23 (10)
SOFA, median (IQR)	7 (4)	8 (4.5)
Receipt of mechanical ventilation, n (%)	159 (74.6)	80 (87.0)
Admitting diagnosis category, n (%)
Medical	110 (51.6)	46 (50.0)
Neuroscience	39 (18.3)	22 (23.9)
Surgical	33 (15.5)	11 (12.0)
Trauma	31 (14.6)	13 (14.1)

Abbreviations: APACHE‐II, Acute Physiology and Chronic Health Evaluation II score; ICU; intensive care unit; IQR, interquartile range; SOFA, Sequential Organ Failure Assessment.

3.2. Adaptation of CHART‐DEL

Few modifications were made to the CHART‐DEL tool to adapt it for a critically ill adult population (Supplementary Table S1). This included the addition of ICU physicians as experienced reference standard raters for definite delirium. Few charts included a physician‐reported ICDSC score in the physician progress notes (18/628, 2.9%). As such, the CHART‐DEL‐ICU tool includes an ICDSC score of ≥4 as a measure of “probable delirium” when physician‐reported ICDSC scores are included in the physician progress notes but does not include nurse‐reported ICDSC scores reported in the EMR. The last modification was to include RASS ²⁰ as an indicator of altered level of consciousness and agitation (trigger word). Ten percent of the charts were randomly selected for the CHART‐DEL‐ICU tool assessment in duplicate by two independent raters, who had excellent interrater reliability (kappa = 0.90) for ever/never delirium and substantial interrater reliability for daily delirium (kappa = 0.79). Chart abstracters took a median time of 28 minutes (interquartile range (IQR) 29) to abstract and rate delirium for each patient chart (i.e., all records for an individual hospitalization).

3.3. CHART‐DEL‐ICU Versus DSM‐5

The sensitivity, specificity, and AUC for each level of confidence in delirium detection (no evidence/uncertain/possible/probable/definite) are displayed in Table 2. For definite delirium, sensitivity was 29.8% (95% CI = 23.7–36.4%) and specificity was 92.0% (95% CI = 89.0–94.4%), when compared with the reference standard. When the cut‐point included definite, probable and possible delirium, the sensitivity was 50.7% (95% CI = 43.8–57.6%) and specificity was 87.7% (95% CI = 84.1–90.7%), when compared with the reference standard. When the cut‐point was expanded to include uncertain delirium, the sensitivity was 66.0% (95% CI = 59.3–72.3%) and specificity was 82.1% (95% CI = 78.0–85.7%), when compared with the reference standard. When considering the cut‐point that includes definite/probable/possible/uncertain delirium ratings, all false negatives occurred when there was no documentation of delirium symptoms in the physician progress notes or nursing notes in the EMR (73/628, 11.6%). There were 74 (74/628, 11.8%) false positives, which included when delirium was indicated as an issue in the physician progress notes (33/74, 44.6%) or if there was a physician reported ICDSC score (6/74, 8.1%). False positives also occurred when trigger words for altered level of consciousness/fluctuation (14/74, 18.9%), disorientation or confusion (each 4/74, 5.4%) or multiple trigger words (6/74, 8.1%) were recorded. There were no false positives when the trigger words for hallucinations were recorded. The remaining false positives (7/74, 9.5%) occurred when the chart abstractor was uncertain if delirium was present, but no trigger words were recorded.

Table 2.

Criterion validity of delirium detection tools (paired assessments) compared to the reference standard assessment (n = 628)

	Delirium Prevalence % (95% CI)	Sensitivity % (95% CI)	Specificity % (95% CI)	AUC % (95% CI)	AUC P‐value
Reference standard
DSM‐5	34.2 (30.6–38.0)	—	—	—	—
Routine screening
ICDSC	24.3 (21.2–27.9)	58.1 (51.2–64.8)	93.2 (90.4–95.4)	75.7 (72.1–79.2)
CAM‐ICU	33.4 (29.8–37.2)	68.8 (62.2–75.0)	85.0 (81.2–88.3)	76.9 (73.4–80.5)
Chart‐based (cut point)
CHART‐DEL‐ICU (def/prob)	22.9 (19.8–26.4)	45.1 (38.3–52.0)	88.6 (85.2–91.5)	66.9 (63.2–70.5)	—
CHART‐DEL‐ICU (def/prob/poss)	25.4 (22.2–29.0)	50.7 (43.8–57.6)	87.7 (84.1–90.7)	69.1 (65.5–72.9)	—
CHART‐DEL‐ICU (def/prob/poss/unc)	34.4 (30.8–38.2)	66.0 (59.3–72.3)	82.1 (78.0–85.7)	74.1 (70.4–77.8)	—
Chart based + routine screening (cut point)
CHART‐DEL‐ICU + CAM‐ICU (def/prob)	42.5 (38.7–46.4)	80.9 (75.0–86.0)	77.5 (73.1–81.4)	79.2 (75.9–82.5)	(vs CHART‐DEL‐ICU) <0.001
CHART‐DEL‐ICU + CAM‐ICU (def/prob/poss)	43.9 (40.1–47.9)	83.3 (77.6–88.0)	76.5 (72.1–80.5)	79.9 (76.7–83.1)	(vs CHART‐DEL‐ICU) <0.001
CHART‐DEL‐ICU + CAM‐ICU (def/prob/poss/unc)	48.2 (44.3–52.2)	88.8 (83.8–92.7)	72.9 (68.3–77.1)	80.9 (77.8–83.9)	(vs CHART‐DEL‐ICU) <0.001
CHART‐DEL‐ICU + ICDSC (def/prob)	34.9 (31.2–38.7)	71.2 (64.6–77.1)	84.0 (80.1–87.4)	77.6 (74.1–81.1)	(vs CHART‐DEL‐ICU) <0.001
CHART‐DEL‐ICU + ICDSC (def/prob/poss)	35.5 (31.9–39.3)	72.1 (65.6–78.0)	83.5 (79.6–87.0)	77.8 (74.3–81.3)	(vs CHART‐DEL‐ICU) <0.001
CHART‐DEL‐ICU + ICDSC (def/prob/poss/unc)	41.1 (37.3–45.0)	79.5 (73.5–84.7)	78.9 (74.7–82.8)	79.2 (75.9–82.6)	(vs CHART‐DEL‐ICU) <0.001

Abbreviations: AUC, area under the receiver operating characteristic curve, CAM‐ICU, Confusion Assessment Method for the ICU; CHART‐DEL‐ICU, Chart‐based Delirium Screening for the ICU; def, definite; ICDSC, Intensive Care Delirium Screening Checklist; poss, possible; prob, probable; unc, uncertain.

3.4. CHART‐DEL‐ICU as an Adjunct for Routine Delirium Screening

The added value of routine delirium screening to the CHART‐DEL‐ICU tool for identifying delirium was assessed by comparing the sensitivity, specificity, and AUC of the CHART‐DEL‐ICU alone and combined with CAM‐ICU or ICDSC (Table 2). The sensitivity of CAM‐ICU/CHART‐DEL‐ICU (cut‐point: definite/probable), compared with the reference standard was 80.9% (95% CI = 75.0–86.0%), with a specificity of 77.5% (95% CI = 73.1–81.4%) and AUC of 79.2% (95% CI = 75.9–82.5%). The sensitivity of ICDSC/CHART‐DEL‐ICU (cut‐point: definite/probable), compared with the reference standard was 71.2% (95% CI = 64.6–77.1%), with a specificity of 84.0% (95% CI = 80.1–87.4%), and AUC of 77.6% (95% CI = 74.1–81.1%). There was a significant difference between the AUC of the CHART‐DEL‐ICU alone and in combination with routine delirium screening tools for all cut‐points (definite/probable, definite/probable/possible, definite/probable/possible/uncertain) (Figure 2).

Figure 2.

Receiver operating characteristic (ROC) curves for the CHART‐DEL‐ICU tool (alone and combined with the CAM‐ICU and ICDSC) compared to the reference standard.

3.5. Frequency of Trigger Words and Antipsychotics

Table 3 includes the frequency of trigger words and antipsychotics mentioned in the chart and their association with delirium identified by the reference standard. There was an increased odds of delirium when confus* (odds ratio (OR) 5.0, 95% CI = 2.7–9.1), disorientation (i.e., disor* or AO < 3, OR = 5.7, 95% CI = 3.1–10.4), altered level of consciousness defined by a RASS fluctuation (OR = 6.8, 95% CI = 4.2–10.9) or hallucinations/delusions (OR = 5.2, 95% CI = 1.4–20.1) was recorded in the paper chart or EMR. There was an increased odds that the patient had delirium if haloperidol (OR = 4.1, 95% CI = 1.8–9.3) or quetiapine (OR = 2.9, 95% CI = 1.8–4.5) was recorded in the paper chart or EMR. Records of zopiclone in the medical chart were also associated with delirium (OR = 1.9, 95% CI = 1.3–3.0).

Table 3.

Frequency of Mention of Trigger Words or Antipsychotics in the Chart (n = 628)

Category	n (%)	Association with DSM‐5 Delirium, Odds Ratio (95% CI)
Trigger word
Confus*	55 (8.8)	5.0 (2.7–9.1)
Disor* (including AO < 3)	56 (8.9)	5.7 (3.1–10.4)
RASS fluctuation (e.g., RASS −2 to +2)	102 (16.2)	6.8 (4.2–10.9)
Hallucinations/delusions	11 (1.7)	5.3 (1.4–20.1)
Antipsychotic
Dexmedetomidine	3 (0.5)	1.0 (0.9–10.6)
Haloperidol	27 (4.3)	4.1 (1.8–9.3)
Olanzapine	3 (0.5)	1.0 (0.9–10.6)
Quetiapine	93 (14.8)	2.9 (1.8–4.5)
Sleep aid/antipsychotic/sedative
Zopiclone	108 (17.2)	1.9 (1.3–3.0)

4. DISCUSSION

The current study adapted and validated a chart‐based method for identifying delirium in critically ill adults. In this study of 213 critically ill adults and 628 paired delirium assessments, the sensitivity of the CHART‐DEL‐ICU tool was 66.0% (95% CI = 59.3‐72.3%) and the specificity 82.1% (95% CI = 78.0‐85.7%) when the cut‐point of uncertain delirium was included. Combining CHART‐DEL‐ICU tool with routine delirium screening tools was significantly better than the use of CHART‐DEL‐ICU alone.

The CHART‐DEL‐ICU tool performed similarly to the CHART‐DEL tool. The initial study validating the CHART‐DEL tool reported a sensitivity of 74% (95% CI = 65–81%) and specificity of 83% (95% CI = 80–86%), ¹⁴ which is similar to the CHART‐DEL‐ICU in the present study. The sensitivity of the CHART‐DEL‐ICU was maximized by including the cut‐point of uncertain delirium, but similar to the CHART‐DEL tool, there were still a substantial number of false negatives (34.4% for CHART‐DEL‐ICU vs 26% for CHART‐DEL). Another study developed and validated a chart‐based method using the DSM‐IV classification for delirium and a consensus panel with three geriatricians and a psychiatrist rated each diagnosis as unlikely, possible or probable delirium. ²⁴ The overall accuracy was comparable to the CHART‐DEL‐ICU with a sensitivity of 89% (95% CI = 82–91%) and specificity of 75% (95% CI = 71–79%) for possible delirium and sensitivity of 58% (95% CI = 53–62%) and specificity 93% (95% CI = 90–95%) for probable delirium. In all cases, a chart‐based delirium detection should be paired with routine delirium screening when used in clinical practice.

The CAM‐ICU and ICDSC have been validated in a research setting, ¹¹ , ¹² , ²⁵ but their accuracy for routine delirium screening was reported to be lower in previous clinical studies ⁹ , ²⁶ , ²⁷ , ²⁸ , ²⁹ and in the current study (Table 2). Given the different presentations of delirium (hyperactive vs hypoactive) and its often fluctuating course, the CHART‐DEL‐ICU tool will be a useful addition in delirium research or in routine clinical practice. In the case of delirium research, this may include the study team using the routine assessments and augmenting them with chart review in place of additional delirium assessments. In the case of routine clinical practice, this may include healthcare professionals reviewing patient charts for trigger words which, in combination with routine delirium assessment, may find missing cases of delirium. However, there are some factors associated with incorrect detection of delirium (i.e., false positives or false negatives) that should be considered before a chart‐based tool is considered for delirium‐related patient care.

When considering the cut‐point that maximizes sensitivity for case identification (i.e., including uncertain delirium ratings), all false negatives occurred when there was no documentation of delirium symptoms in the chart either because the episode of delirium was not charted or recognized. However, there were several false positives (74/628, 11.8%). A common trigger word used in the chart was confus*. In some charts, confusion was written with no other details. This was categorized as uncertain delirium by the abstractor and accounted for four (5.4%) of the false positives. Other trigger phrases such as altered level of consciousness and fluctuating course, often described as a fluctuating RASS (e.g., RASS −2 to +2), were identified as uncertain delirium. This accounted for 14 (18.9%) of the false positives. The most common reason for a false positive was when delirium was listed as a problem or issue in the physician's progress notes (33/74, 44.6%). Sometimes this was followed with sufficient detail to indicate delirium was present such as identifying it as hypoactive delirium or describing features of delirium such as disorientation. However, in some cases it was listed as an issue with no other details. The missing detail resulted in a chart abstractor indicating delirium was present, when it may not have been. For future studies using the CHART‐DEL‐ICU tool and including a cut‐point of definite/probable/possible/uncertain, it is recommended when delirium is listed as a problem in the chart (with no other details), the chart abstractor should look for other evidence (e.g., antipsychotic use, symptoms of delirium) before rating it as definite delirium.

4.1. Strengths and Limitations

There are several strengths of the study. First, the preregistered protocol was co‐designed with patients, researchers, and ICU care providers and tested in a pilot study. ¹⁰ , ¹⁸ Second, inter‐rater reliability was assessed in a random subset of patients. Third, chart abstractors were blinded to the results of the CAM‐ICU and ICDSC assessments (and CAM‐ICU and ICDSC assessors to the results of the chart abstraction). Last, the reference standard rater used DSM‐5 criteria to evaluate all participants for delirium. There are several limitations to this study that are important to discuss. First, the cross‐sectional nature of the data means we examined patients with prevalent delirium and were unable to distinguish patients with incident delirium. Second, the study was conducted in a single academic ICU, which limits the generalizability of the CHART‐DEL‐ICU for other ICU settings. However, the FMC ICU provides healthcare services to a diverse population of over 2 million people, and results may readily generalize to other centers with similar health systems and patient populations. Moreover, generalizability will also depend on the accuracy and breadth of information recorded in paper charts and EMRs at other hospitals. Third, varied delirium assessments may be attributed to several factors. Though delirium assessments were compared at the same time points (i.e., afternoon assessments), delirium assessments were not conducted concurrently and it is possible that the assessments varied due to the fluctuating course of delirium. Fourth, sedative and analgesic medications, both risk factors for delirium, were time‐varying and not controlled for in this study. Fifth, no adjustments were made to account for within‐patient correlations. Lastly, the mean age of the study population is lower than a typical medical/surgical ICU population, which is attributed to recruitment patient population with younger and lower morality (i.e., less severe illness given requirements to be able to respond for delirium assessments).

5. CONCLUSIONS

A chart‐based delirium detection tool (CHART‐DEL‐ICU) can be used to gather chart‐based information on delirium. The CHART‐DEL‐ICU has improved diagnostic accuracy when used in combination with routine delirium screening (CAM‐ICU and ICDSC) and thus, may be useful for both clinical and research applications. However, on its own, the CHART‐DEL‐ICU has lower sensitivity since not all episodes of delirium are charted or recognized. Due to the false‐positive rate, a chart‐based tool for delirium detection is not recommended for use alone in patient care or diagnostic purposes. However, the CHART‐DEL‐ICU brings considerable strengths to ICU delirium research with critically ill adults, bringing 24‐hour perspective and bedside assessments (CAM‐ICU and ICDSC) improve the overall sensitivity of CHART‐DEL‐ICU.

Supporting information

Supplementary Table S1: Comparison between the Chart‐based Delirium Detection Instrument (CHART‐DEL) and Intensive Care Unit CHART‐DEL (CHART‐DEL‐ICU) scoring.

Supplementary Figure S1: Participant flow diagram.