Factors Influencing CAM-ICU Documentation and Inappropriate “Unable to Assess” Responses

Abstract

Background

Detecting delirium with standardized assessment tools such as the Confusion Assessment Method for the Intensive Care Unit (CAM-ICU) is important, but such detection is frequently hampered by poor documentation and inappropriate “unable to assess” responses (in noncomatose patients).

Objective

To identify patient, clinical, and workplace factors that may impede or facilitate appropriate delirium assessment through use of the CAM-ICU, specifically documentation and inappropriate “unable to assess” responses.

Methods

An electronic health record–based data set was used to quantify CAM-ICU documentation and inappropriate “unable to assess” responses during 24 months. Associated patient (eg, age), clinical (eg, diagnosis), and workplace (eg, geographic location within the ICU, shift) factors were evaluated with multivariable regression.

Results

Of 28 586 CAM-ICU documentation opportunities, 66% were documented; 16% of documentations in alert or lightly sedated patients had inappropriate “unable to assess” responses. Night shift was associated with lower CAM-ICU documentation rates (P = .001), whereas physical restraints and location on side B (rather than side A) of the ICU were associated with higher documentation rates (P < .05 for both). Age older than 80 years, non-White race, intubation, and physical restraints were associated with more inappropriate “unable to assess” responses (all P < .05), as was infusion of propofol, midazolam, dexmedetomidine, or fentanyl (all P < .05).

Conclusion

Data from electronic health records can identify patient, clinical, and workplace factors associated with CAM-ICU documentation and inappropriate “unable to assess” responses, which can help target quality improvement efforts related to delirium assessment.

Delirium, an acute fluctuating alteration in mental status characterized by inattention and disorganized thinking, is prevalent during critical illness; it affects up to 80% of patients in intensive care units (ICUs).¹ Studies have demonstrated that delirium in the ICU is independently associated with numerous adverse short- and long-term outcomes, including prolonged ICU and hospital lengths of stay and disabling cognitive, physical, and mental health impairments.^2–6 As a consequence, delirium is estimated to cost the US health care system up to $150 billion per year.⁵ Although prevalent, delirium is potentially modifiable, with risk factors such as oversedation and immobility representing key targets for preventive efforts.

Despite interest in delirium in the ICU, up to 72% of episodes go unrecognized by health care professionals, possibly because most do not involve consistent hyperactivity that is easily identifiable at the bedside, but rather involve hypoactivity with intermittent short bouts of hyperactivity (the mixed subtype).^7–10 Fortunately, easy-to-use bedside tools such as the Confusion Assessment Method for the Intensive Care Unit (CAM-ICU)¹¹ and the Intensive Care Delirium Screening Checklist¹² have been validated to aid in the detection of delirium in critically ill patients. These tools are fundamental for delirium management, and recent clinical practice guidelines strongly recommend their use on a daily basis.¹³

Despite established guidelines and widespread efforts to prevent delirium in the ICU, completion of delirium assessments by ICU staff (usually bedside nurses) is highly variable, with reported rates as low as 38% in usual care settings and as high as 84% to 95% following rigorous intervention efforts.^14–17 Even when documented, CAM-ICU responses can be scored incorrectly, particularly in patients with hypoactive delirium, those with neurological or neurosurgical diagnoses, and those hospitalized in ICUs that lack an established delirium detection and prevention infrastructure.⁸ Specifically, inappropriate “unable to assess” (IUTA) responses, defined as an “unable to assess” response despite a patient being noncomatose (ie, alert or lightly sedated), are particularly common, accounting for 19% to 30% of documented CAM-ICU scores.^18,19

Prior studies have evaluated barriers to delirium assessment, focusing on staff-related barriers such as lack of education and perceived difficulty of use of validated delirium assessment tools; however, patient-related factors such as age, primary language, and receipt of sedative infusions may also be significant barriers that have been difficult to evaluate through staff surveys.^20–22 To evaluate these issues further, we analyzed patient, clinical, and workplace factors associated with CAM-ICU documentation and IUTA responses using an electronic health record (EHR)–based data set.

Methods

Study Design and Data Source

As part of a delirium quality improvement project in our academic medical ICU (MICU), we teamed with the Office of Health Informatics and Analytics at the University of California, Los Angeles, to design an EHR-derived data set comprising various patient (eg, age, primary language), clinical (diagnosis, sedation, physical restraints), and workplace (eg, shift, location within the ICU) variables that could potentially inform and improve practices surrounding delirium. Our delirium data set included every patient present at 12 am in any of the 24 beds in the closed MICU. In this MICU, nursing staff began CAM-ICU documentation on March 1, 2013, the day a new EHR system was introduced containing a CAM-ICU documentation flowsheet. After nurses received appropriate training, their monthly CAM-ICU documentation rates rose quickly, plateauing by late 2014. Hence, for this analysis we chose the 24-month period spanning January 1, 2015, to December 31, 2016, during which CAM-ICU documentation rates remained consistent at 65% to 70%.

The 24-bed MICU involved in this analysis has a nurse-to-patient ratio of 1:2, with nurses on the day shift working 7 am to 7 pm and nurses on the night shift working 7 pm to 7 am. Notably, during the time frame of this analysis, individual nurses tended to work in 1 of 2 connected V-shaped halves of the ICU—side A, containing beds 1 through 12, or side B, containing beds 13 through 24; patients were randomly assigned to the first available bed and cared for by 1 of 2 separate ICU teams. As our hospital also has separate neurological, surgical/trauma, cardiothoracic, and liver ICUs, our MICU population consisted primarily of medically complex nonneurological, nonsurgical patients with respiratory failure, shock, or both.

CAM-ICU Assessments

Nursing staff assessed delirium twice daily (once per shift, usually at 8 am and 8 pm) using the CAM-ICU. In our MICU, this assessment was performed in English and involved bedside performance of the 4 features of the CAM-ICU; this assessment includes a level-of-consciousness evaluation (feature 3) using a modified Riker Sedation-Agitation Scale (SAS), with scores of −3 to 3 replacing the traditional 1 to 7 ordinal scale.²³ The Supplemental Table compares the traditional and modified Riker SAS. On the basis of this modified scale, patients with scores of −1 or higher (paralleling a traditional Riker SAS score ≥3) were deemed “noncomatose” and were subsequently evaluated for delirium with the CAM-ICU. Although the Richmond Agitation-Sedation Scale²⁴ is often used to evaluate the level-of-consciousness component of the CAM-ICU, the Riker SAS has been shown to be comparable to the Richmond Agitation-Sedation Scale for assessing delirium and, with the Richmond Agitation-Sedation Scale, is recommended in clinical practice guidelines for monitoring the quality and depth of sedation.^25,26

Outcomes

Our primary outcome was CAM-ICU documentation (ie, a “positive,” “negative,” or “unable to assess” response). Among documented CAM-ICU assessments, we also evaluated the proportion of IUTA responses, defined as an “unable to assess” response despite a modified Riker SAS score of −1 or higher (when the patient was alert or sedated, ie, not comatose). Specifically, we calculated this proportion by dividing the total number of IUTA responses by the number of documented CAM-ICU assessments in noncomatose patients (ie, should be able to assess with a CAM-ICU score of “positive” or “negative”). Hence the denominator used to calculate the proportion of IUTA responses excluded “unable to assess” responses in comatose (modified Riker SAS score of −2 or −3) patients whose “unable to assess” scores were appropriate.

On the basis of our clinical experience and prior research, we selected patient- and ICU-level factors that could influence CAM-ICU nondocumentation and IUTA responses. We analyzed factors that possibly influence nursing practices, such as shift (weekend vs weekday, night vs day) and geographic location of the patient in the ICU (side A vs side B); we also included month of the year (July vs all other months) as a factor to evaluate whether the presence of new residents influenced CAM-ICU performance. We analyzed demographic factors including patient age, race, and primary language (the CAM-ICU was conducted only in English at our institution). Clinical characteristics included diagnosis category at admission, intubation status, presence of physical restraints, and use of sedative infusions.

Statistical Analysis

We used descriptive statistics to report CAM-ICU documentation rates and patient and ICU characteristics. Using multivariable logistic regression, we evaluated the adjusted odds ratio (OR) of patient (eg, age category, primary language), clinical (admission diagnosis type, sedation), and workplace (eg, shift, geographic side of the ICU) factors with CAM-ICU documentation (primary outcome) and IUTA responses (secondary outcome). In all models, we used generalized estimating equations to account for repeated time-varying measures (eg, CAM-ICU assessments). For patients who were admitted multiple times during the data collection period, we confined our analysis to each patient’s first admission. We used variance inflation factors to evaluate for multicollinearity and detected none. We analyzed data using Stata 15.1 software (StataCorp).

Within the context of ongoing quality improvement efforts in our MICU, our institutional review board deemed this project to be a quality improvement project and thus exempt from full review.

Results

Patient Characteristics

The data set included 1942 patients. The patients’ demographic characteristics are shown in Table 1; most were White, male, and spoke English as their primary language. Clinical factors are also described in Table 1, including diagnosis at admission, need for intubation and physical restraints, receipt of sedative infusions, and length of stay. Only 22% of patients screened positive for delirium during their ICU stay.

Table 1.

Demographic and clinical variables for 1942 patients

Variable	No. (%) of patientsa
Demographic variables
Age, mean (SD), y	63 (18)
Sex
Male	1074 (55)
Female	868 (45)
Ethnicity
Hispanic	383 (20)
Non-Hispanic	1537 (79)
Unknown/refused	22 (1)
Race
White	1194 (61)
Black	229 (12)
Asian/Pacific Islander/Hawaiian	208 (11)
Otherb	311 (16)
Primary language
English	1641 (85)
Spanish	151 (8)
Otherc	150 (8)
Clinical variables
Diagnosis category at admission
Sepsis	590 (30)
Otherd	373 (19)
Gastrointestinal/hepatologic	249 (13)
Cardiac	243 (13)
Respiratory	214 (11)
Hematologic/oncologic	138 (7)
Renal/endocrine/rheumatologic	135 (7)
Intubated	707 (36)
Ever needed physical restraints	557 (29)
Ever received a sedative infusion	594 (31)
Fentanyl	461 (24)
Propofol	360 (19)
Midazolam	321 (17)
Dexmedetomidine	178 (9)
Ever delirious	428 (22)
Length of stay, median (IQR), d
ICU	5 (3–8)
Hospital	10 (5–20)
Death
In ICU	339 (17)
In hospital	410 (21)

Abbreviations: ICU, intensive care unit; IQR, interquartile range.

^aUnless indicated otherwise in the first column.

^bIncludes unknown, patient refused, and unrecorded.

^cIncludes Amharic, Arabic, Armenian, Bulgarian, Cantonese, Chinese, Farsi, French, Hebrew, Japanese, Korean, Mandarin, Russian, unknown, Urdu, Vietnamese, and other.

^dIncludes neurological disorders, nonsepsis infectious disease, toxicology, dermatologic disorders.

CAM-ICU Documentation

During the 24-month analysis, there were 28 586 CAM-ICU documentation opportunities. Overall, 66% (n = 18 779) of these opportunities were indeed documented (monthly range, 64%−69%); documentation did not differ substantially on the basis of reported sedation-agitation score (ie, alert, sedated, or comatose).

In the multivariable model (Table 2), CAM-ICU documentation was higher during the day shift (vs night shift, P = .001) and for patients requiring physical restraints (P = .001), those from side B of the ICU (P = .001), and those receiving an infusion of dexmedetomidine (P = .001). We found no significant differences in documentation rate based on patient age, sex, race, ethnicity, or primary language (Table 2).

Table 2.

Multivariable analysis of factors associated with documented CAM-ICU assessments and inappropriate “unable to assess” responses

	Documented CAM-ICU			IUTA response
Factor	Odds ratioa	95% CI	P	Odds ratioa	95% CI	P
Patient related
Age, y
<65	Reference			Reference
65–80	0.95	0.88–1.02	.16	1.28	0.99–1.65	.06
>80	0.96	0.87–1.06	.46	1.72	1.25–2.35	.001
Sex
Male	Reference			Reference
Female	1.06	0.99–1.13	.09	1.19	0.94–1.51	.16
Primary language
English	Reference			Reference
Spanish	1.09	0.95–1.24	.22	0.48	0.27–0.87	.02
Other	0.94	0.83–1.06	.31	1.49	0.94–2.37	.12
Ethnicity
Non-Hispanic	Reference			Reference
Hispanic	1.00	0.91–1.10	>.99	1.44	0.99–2.11	.06
Race
White	Reference			Reference
Non-White	0.95	0.88–1.01	.11	1.46b	1.13–1.89	.005
Clinical
Admission diagnosis category
Respiratory	Reference			Reference
Sepsis	1.04	0.93–1.16	.53	2.37	1.55–3.64	<.001
Other	0.90	0.81–1.00	.05	1.32	0.87–2.02	.18
Not intubated	Reference			Reference
Intubated	1.06	0.97–1.15	.20	2.18	1.74–2.74	<.001
Not requiring physical restraints	Reference			Reference
Physical restraints	1.20	1.09–1.32	<.001	2.15	1.64–2.82	<.001
No propofol infusion	Reference			Reference
Propofol infusionb	1.14	0.99–1.30	.06	1.93	1.41–2.64	<.001
No midazolam infusion	Reference			Reference
Midazolam infusionb	0.90	0.76–1.05	.17	1.82	1.05–3.16	.04
No dexmedetomidine infusion	Reference			Reference
Dexmedetomidine infusionb	1.38	1.18–1.62	<.001	1.69	1.12–2.56	.01
No fentanyl infusion	Reference			Reference
Fentanyl infusionb	0.92	0.81–1.04	.18	1.98	1.50–2.62	<.001
Workplace
Day shift	Reference			Reference
Night shift	0.86	0.81–0.91	<.001	0.99	0.93–1.06	.89
Side A	Reference			Reference
Side B	1.38	1.30–1.48	<.001	0.78	0.60–1.01	.07
Weekend	Reference			Reference
Weekday	1.02	0.97–1.08	.44	0.91	0.84–0.99	.04
Not July	Reference			Reference	Reference
July	1.06	0.94–1.19	.32	0.51	0.22–1.18	.11

Abbreviations: CAM-ICU, Confusion Assessment Method for the Intensive Care Unit; IUTA, inappropriate “unable to assess.”

^aCalculated by using multivariable logistic regression, with generalized estimating equations to account for repeated time-varying measures (eg, CAM-ICU). For patients admitted multiple times during the data collection period, we confined our analysis to each patient’s first admission. Variance inflation factors were used to evaluate for multicollinearity; none was detected. Higher documentation and inappropriate “unable to assess” response odds ratios represent factors associated with a greater odds of completion of CAM-ICU documentation or inappropriate “unable to assess” scores, respectively.

^bAdministered at any time during a 12-hour CAM-ICU documentation period.

IUTA Responses on the CAM-ICU

Of 18 779 documented CAM-ICU assessments, 3776 (20%) were done in patients without a Riker SAS score or who were comatose and thus were excluded from analysis. Of the remaining 15 003 CAM-ICU assessments in alert or lightly sedated patients, 2368 (16%) were documented as “unable to assess” and therefore deemed as an IUTA response. Notably, when patients were intubated or receiving propofol, midazolam, or fentanyl infusions, more than 50% of all documented CAM-ICU assessments fell into the IUTA category.

In the multivariable model (Table 2), factors associated with an IUTA response included age older than 80 years (P = .001), non-White race (P = .005), intubation (P < .001), use of physical restraints (P < .001), the infusion of any sedative (P < .05), sepsis (P < .001), and weekday (P = .04).

Discussion

Prior studies involving focus groups or surveys of delirium stakeholders (ie, physicians, nurses, and pharmacists) identified common barriers to the detection of delirium in the ICU; these barriers included the time required to perform an assessment, intubation, and a lack of nurse education.^15,27–29 Although those staff believed that delirium was a problem, they expressed concern that delirium detection efforts were undermined by providers (eg, physicians) who did not prioritize delirium and a general lack of effective treatments.^17,30,31 Although informative, these survey-based studies were vulnerable to recall and selection bias, along with the Hawthorne effect.^20,21

As a novel approach, we applied a multivariable regression model using a large EHR-based data set, which we compiled using data from 28 586 CAM-ICU opportunities across 1942 patients in a MICU. We aimed to understand the patient, clinical, and workplace factors associated with completion of delirium assessments (ie, CAM-ICU documentation) and with IUTA responses. Nearly 2 years after implementation of the CAM-ICU, we observed that only two-thirds of CAM-ICU assessments were documented. Only 22% of our patients ever tested positive for delirium—far lower than the 47% to 80% prevalence reported in other MICU populations—suggesting that delirium may have gone undetected in many cases.^1,32–34 Consistent with the published literature, intubation and sedative infusions posed significant barriers to completely or accurately assessing delirium. As a novel finding, older age, non-White race, night shift, and the patient’s geographic location in the ICU also were associated with incomplete or inaccurate CAM-ICU assessments.

Our primary finding was an overall CAM-ICU documentation rate of 66%. This rate is consistent with those from prior ICU studies, which demonstrated 40% to 60% completion rates preceding a dedicated delirium intervention and 65% to 90% afterward.^15–17 We also found an overall rate of IUTA responses (“unable to assess” CAM-ICU scores in alert or mildly sedated patients) of 17%; this value is comparable to those from 2 prior studies reporting IUTA rates (19% and 24%).^18,19 Notably, one of those studies reported an IUTA rate as a proportion of only “unable to assess” responses; had all evaluations of noncomatose patients been used as a denominator, the IUTA rate would have been 9% rather than 24%.¹⁹

To our knowledge, this study is the first to describe both geographic and shift-based imbalances in delirium assessment within a single unit. Previous studies involving handwashing demonstrated interunit variation within the same hospital, most likely due to a combination of staff and leadership issues.^35,36 The lower CAM-ICU documentation rates, however, along with likely higher IUTA rates for the same geographic ICU area and lower CAM-ICU documentation rates for specific shifts, suggest that clusters of staff may have lacked motivation or knowledge regarding delirium assessment. The main difference between geographic locations and shifts is the staff performing delirium assessments (as opposed to physicians and leaders), suggesting strongly that practice variations may be due to differences in age, seniority (ie, years of ICU experience), certifications, and/or collegial relationships, factors that may influence collective interest and engagement in evidence-based practices.²⁰ Because all staff received the same delirium-related training, future studies can evaluate staff demographics, attitudes, and beliefs to determine causes of practice variability.

Consistent with prior studies suggesting mechanical ventilation as the single largest factor influencing IUTA responses, our EHR-based analysis demonstrated a significant association between intubation and CAM-ICU accuracy.^28,29 That association was unexpected, however, because intubated patients had CAM-ICU documentation rates no worse than the rates in patients who were not intubated, but they were more likely to have IUTA responses.

Similar to results in intubated patients, we expected to observe lower CAM-ICU completion and higher IUTA rates in patients receiving a sedative infusion, a well-known barrier to CAM-ICU assessment completion.²⁸ Dexmedetomidine was the only infusion associated with higher rates of CAM-ICU documentation, whereas midazolam, propofol, and fentanyl had no association. Our findings suggest that dexmedetomidine might facilitate delirium assessment, whereas sedatives such as fentanyl might have the opposite effect. Whether and how specific sedative infusions affect nurses’ ability to evaluate delirium requires further investigation.

Finally, we found that older age (>80 years old) and non-White race were associated with higher rates of IUTA responses. Although IUTA and age have not, to our knowledge, been studied previously, studies of patients outside an ICU have suggested difficulties detecting delirium in older patients because of sensory impairments and dementia.^37,38 More IUTA responses for non-White patients is a unique finding, as prior research did not highlight race as a factor influencing the documentation of delirium.^38–40 The influence of patient race and primary language pose an intriguing area of investigation, and for ICUs with diverse populations such as ours, these factors represent important areas for dramatic improvements in detecting delirium in the ICU.

Although the results of our analysis highlight novel targets for improving delirium assessment, the method we used is broadly applicable. Qualitative methods (ie, surveys and focus groups) are valuable, but they are limited by bias and the inability to quantify the extent of reported problems. As our EHR-based technique pinpointed otherwise unrecognized performance factors (ie, staff on one side of the ICU documenting more than staff on the other side), we believe that such a data-driven technique could be used to inform subsequent qualitative evaluations as a method to streamline institutional efforts to improve care.

Strengths and Limitations

Strengths of our study include use of a large EHR-based data set and a multivariable model to evaluate factors previously described using qualitative methods. To our knowledge, it includes more patients and a longer observational period than any prior study. Because it was retrospective, there was no risk of a Hawthorne effect. As a result, we were able to highlight targeted areas of improvement to help develop and inform ongoing efforts to detect delirium.

Despite these strengths, our analysis has some limitations. First, given the large data set, the analysis was vulnerable to unmeasured confounding and missing or inaccurate data. Our data set did, however, have minimal missing data; items such as sedation scores were more than 90% complete. Second, we did not access nursing notes, which could have provided specific reasons for CAM-ICU nondocumentation or an IUTA response, such as patient refusal or neurological impairment. Implementation of this field in an EHR may provide a significantly improved understanding of barriers to delirium assessment. Compared with interviews and surveys, through which one can try to assess health care professionals’ intentions or feelings, EHRs do not allow such reasoning to be easily understood. Furthermore, because our data were limited to those in EHRs, we did not have access to nursing demographic data (years of experience, certifications, etc); inclusion of such data is a consideration for future quality improvement work. Last, because we obtained our data from a medically complex nonneurological and nonsurgical patient population in a single academic MICU, our findings may not be generalizable to populations in other ICUs, particularly those with mixed patient populations. Future analyses could involve multiple centers, acknowledging the challenge of harmonizing queries from EHR systems that have been customized to the specific needs of each ICU and health system.

Conclusions

We used a large EHR-derived data set to evaluate patient, clinical, and workplace factors affecting CAM-ICU documentation and IUTA responses. Only two-thirds of CAM-ICU opportunities had been documented; of those, one-sixth had an IUTA response. Unsurprisingly, intubation and sedative infusions posed barriers to documentation, whereas novel factors such as geographic side of the ICU, night shift, older patient age, and non-White patient race were identified as specific targets for delirium improvement efforts. Our EHR-based approach provides an automated, objective, and generalizable method for evaluating barriers to and facilitators of delirium detection on a large scale, the results of which can subsequently be used to jumpstart and inform improvement efforts. Because of the limitations of EHR-abstracted data in allowing an understanding of nurses’ knowledge and intent, which are often better assessed with surveys and interviews, this analytic process provides a complementary tool to improve delirium assessment in an ICU.