Implementation, reliability testing, and compliance monitoring of the Confusion Assessment Method for the Intensive Care Unit (CAM-ICU) in trauma patients

Abstract

Objective

To implement delirium monitoring, test reliability, and monitor compliance of performing the Confusion Assessment Method for the Intensive Care Unit (CAM-ICU) in trauma patients.

Design and setting

Prospective, observational study in a Level 1 trauma unit of a tertiary care, university-based medical center.

Patients

Acutely injured patients admitted to the trauma unit from February 1, 2006–April 16, 2006.

Measurements and Results

Following web-based teaching modules and group in-services, bedside nurses evaluated patients daily for depth of sedation with the Richmond Agitation-Sedation Scale (RASS) and for the presence of delirium with the CAM-ICU. On randomly assigned days over a 10-week period, evaluations by nursing staff were followed by evaluations by an expert evaluator of the RASS and the CAM-ICU, in order to assess compliance and reliability of the CAM-ICU in trauma patients. Following the audit period, the nurses completed a post-implementation survey.

One thousand and eleven random CAM-ICU assessments were performed by the expert evaluator, within 1 hour of the bedside nurses’ assessments. Nurses completed the CAM-ICU assessments in 84% (849 of 1011) of evaluations. Overall agreement (κ) between nurses and the expert evaluator was 0.77 (0.721, 0.822; p<0.0001). In TBI patients κ was 0.75 (0.667, 0.829; p<0.0001), while in mechanically-ventilated patients κ was 0.62 (0.534, 0.704; p<0.0001). The survey revealed nurses were confident in performing the CAM-ICU, realized the importance of delirium, and were satisfied with the training they received. The survey also acknowledged obstacles to implementation including nursing time and failure of physicians/surgeons to address treatment approaches for delirium.

Conclusions

The CAM-ICU can be successfully implemented in a university-based trauma unit with high compliance and reliability. Quality improvement projects seeking to implement delirium monitoring would be wise to address potential pitfalls including time complaints and the negative impact of physician indifference regarding this form of organ dysfunction.

Introduction

Delirium is defined as an acute change or fluctuation in cognitive function, accompanied by inattention and either disorganized thinking or an altered level of consciousness [1]. It is one of the most frequent complications experienced in the intensive care unit (ICU), associated with prolongation of hospital length of stay [2], higher costs [3] and is an independent predictor of mortality [4, 5]. The clinical practice guidelines of the Society of Critical Care Medicine (SCCM) [6] now recommend routine monitoring for the presence of delirium using validated tools such as the Confusion Assessment Method for the Intensive Care Unit (CAM-ICU) [7, 8] and the Intensive Care Delirium Screening Checklist (ICDSC) [9]. The CAM-ICU has been validated in ventilated and non-ventilated critically ill patients, and has been implemented successfully with compliance and reliability testing in different medical centers on more than one continent [5–8, 10], confirming that it is easy to implement and can be performed by the bedside nurse with a high degree of accuracy.

To our knowledge, there are no published studies reporting the feasibility and reliability of delirium monitoring in the trauma population. This is particularly challenging given that there are a significant number of trauma patients with traumatic brain injury (TBI), a group excluded from previous ICU delirium studies.

Materials and Methods

Vanderbilt University’s Institutional Review Board approved the study as a quality improvement project, and informed consent was waived as no identifying data were collected and because delirium monitoring was introduced as standard of care within all our ICUs at Vanderbilt University.

Patients

Vanderbilt University Medical Center (VUMC) is a 631-bed academic hospital that includes a level 1 trauma center admitting approximately 3000 acutely injured patients annually. In an effort to introduce delirium screening in the trauma ICU, unit-wide educational efforts began in December 2005. Routine CAM-ICU monitoring was implemented in January 2006, and all patients admitted to the 31-bed trauma unit from February 1, 2006 to April 16, 2006 were included in the compliance and reliability phase of this study. In order to facilitate implementation of the CAM-ICU and to ensure compliance, patients were included regardless of the presence of traumatic brain injury (TBI), a history of dementia, or neurologic disease. Non-English speaking patients were excluded, however, due to their inability to understand the auditory component of the CAM-ICU. All 96 nurses employed in the trauma unit during the study period were involved in the training and implementation.

Delirium Monitoring Instrument

The delirium-monitoring instrument that was incorporated into standard practice was the Confusion Assessment Method for the ICU (CAM-ICU), which is a well-validated delirium-monitoring instrument, already being routinely utilized in the surgical and medical ICUs at VUMC. Delirium assessment using the CAM-ICU consists of four key features derived from the Diagnostic Statistical Manual IV definition of delirium [1]. The patient is diagnosed as CAM-ICU positive, or delirious, if both features 1 (Acute onset of mental status change or fluctuation in course) and 2 (Inattention) and either feature 3 (Disorganized thinking) or feature 4 (Altered level of consciousness) are present [7].

Implementation of CAM-ICU

Implementation of the CAM-ICU in the trauma unit was performed in four phases: education (4 weeks), implementation (2 weeks), compliance and reliability testing (10 weeks), and post-implementation follow up (8 weeks).

Education phase

First, the expert evaluators trained selected nursing champions and the nurse educator on the CAM-ICU. Inter-rater reliability was assessed between the expert evaluators and the nurse educators during study staff meetings, to ensure that all assessments were standardized and in accordance with the original validation studies of the CAM-ICU [7, 8]. The education phase for the bedside nurses was then performed by the expert evaluators (study co-investigators) along with the nursing champions and the nurse educator. The bedside nurses initially completed a mandatory online in-service developed by the study co-investigators. The in-service included background information on delirium and its importance, detailed descriptions of the Richmond Agitation Sedation Scale (RASS) and the CAM-ICU, and a number of case vignettes that covered commonly encountered patient cognitive assessments. This was followed by mandatory 30-minute group in-services for the nurses and one-on-one bedside demonstrations performed by the expert evaluators and the nurse educator. Educational handouts and pocket cards regarding the steps involved in CAM-ICU administration were placed at the bedside for easy reference. Education via informal in-services continued throughout the remaining three study phases by the nursing champions and the nurse educator.

Implementation phase

Following completion of the initial education, the bedside nurses were required to document the CAM-ICU in the patient medical record once per shift at 8am and 8pm. As a paper documentation system was utilized at this time, modification of the daily nursing flow-sheet was necessary to incorporate the CAM-ICU score adjacent to the RASS score, which is standard of care for targeted sedation in our trauma ICU. The CAM-ICU was recorded as + (positive), − (negative), or UTA (unable to assess in cases of RASS −4 and −5 in which patients are unresponsive to verbal stimulation) based on the definitions in the original validation cohort of the CAM-ICU [7]. In addition, unscheduled follow-up evaluation and informal in-services were performed by the nursing leaders and by study group members on a weekly basis. The intensity and duration of these sessions was determined by the nursing leadership according to their perception of bedside nurse knowledge of the CAM-ICU and how well each nurse actually performed using this tool.

Compliance and reliability testing

In the third phase, compliance and reliability of performing the CAM-ICU were assessed on day shifts, night shifts, and weekend shifts using a random pre-determined audit schedule. Delirium evaluations by the nursing staff were followed within one hour by evaluations by an expert evaluator, trained by members of the ICU Delirium and Cognitive Impairment Study Group (www.icudelirium.org). The expert evaluator performed her CAM-ICU assessment independently, but utilized the patient medical record to ascertain fluctuations in the patient’s level of consciousness and was permitted to question the bedside nurse regarding the same, if needed, to obtain Feature 1 of the CAM-ICU. In the event of disagreement between the nurse CAM-ICU assessment and that of the expert evaluator, the expert evaluator took the time to re-educate and clarify any misinterpretations by the bedside nurse. However, the initial nursing documentation of CAM-ICU was recorded for evaluation of reliability.

Post implementation phase

During the post-implementation phase, surveys were distributed to the nursing staff to determine their understanding of delirium and their comfort and satisfaction with the CAM-ICU. Barriers and educational needs were also identified. Six weeks after the compliance and reliability phase, spot checks were performed once again in order to ensure ongoing compliance.

Statistical Analysis

Patient characteristics were summarized using percentages for binary data and medians and interquartile ranges for continuous data. Compliance was determined by the number of observations (patient-shifts) with at least one CAM-ICU assessment, taking the total number of assessments performed and dividing this by the total number of assessments expected. Reliability, or the agreement of CAM-ICU scores between the bedside nurses and the expert evaluators, was calculated using the kappa statistic. Kappa values were computed to demonstrate reliability in all patients, in patients with TBI, and in patients receiving mechanical ventilation. Positive, negative, and UTA assessments were included. Kappa values of 0.81–0.99 were considered to indicate very high agreement, 0.61–0.80 high agreement, 0.41–0.60 moderate agreement, and 0.21–0.40 fair agreement [11]. Statistical analyses were performed using STATA 7.0 (College Station, TX) by an independent biostatistician.

Results

Patient Characteristics

Demographic data for all patients admitted to the trauma unit from February 1, 2006 to April 16, 2006 were obtained from the institution’s Trauma Registry of the American College of Surgeons and are shown in Table 1. This study was designed to valuate the compliance and inter-rater reliability of patient assessments, and was performed with a waiver of consent. Hence, patient identifiers and data specific to individual study patients were not collected.

Table 1.

Demographic Data of Patients Admitted to the Trauma ICU during Study Period

Variable	N= 347
Age, years	39 (25–52)
Male	250 (72%)
Blunt injury	298 (86%)
Weighted revised trauma score, RTS	7.84 (5.97–7.84)
Injury severity scale (ISS) score	17 (12–29)
Probability of survival, TRISS	0.97 (0.82–0.99)
Hospital length of stay, days	4 (2–7)
ICU length of stay, days	2 (1–5)
Mortality	18 (5.2%)

Data are presented as median (interquartile range) or n (%).

Observational Characteristics

There were a total of 1011 CAM-ICU assessments made by the expert evaluator during the study period. Of these, 585 (58%) were performed on day shift, 274 (27%) on night shift, and 152 (15%) on the weekend. Thirty-one percent of the observations (309/1011) were on patients who were mechanically ventilated. Thirty-five percent of the observations (352/1011) were on patients with traumatic brain injury (TBI).

Compliance

Compliance was defined as the completion of the CAM-ICU assessment in concert with a RASS score by the bedside nurse prior to the assessment by the expert evaluator. Overall compliance was 84% (849 of 1011 observations). Compliance was 83% (485/585) during day shift, 86% (235/274) during night shift, and 85% (129/152) during the weekend shift. In 12% of observation periods, the CAM-ICU was performed more often than the minimum requirement of once per 12-hour shift.

Of the 849 total observations in which compliance was observed, the CAM-ICU was UTA by the expert evaluator in 149 (18%) and UTA by the nurse in 112 (13%). Of the 293 observations in TBI patients in which compliance was observed, the CAM-ICU was UTA by the expert evaluator in 88 (30%) and UTA by the nurse in 69 (24%).

Reliability

Reliability was defined as the agreement of CAM-ICU scores between the bedside nurse and the expert evaluator. Overall inter-rater agreement (κ) (confidence interval; p value) was 0.77 (0.721, 0.822; p<0.0001). The kappa statistic was 0.62 (0.534, 0.704; p<0.0001) in mechanically ventilated patients and 0.75 (0.667, 0.829; p<0.0001) in TBI patients.

Post-implementation

Of the 96 nurses employed in the trauma unit, 42 (44%) completed the post-implementation survey (Table 2). These nurses had a mean of 5.8 ± 4.2 years of nursing experience. Fifty percent of the responders worked day shift and 50% worked night shift. Fifty-seven percent (24/42) stated that the combined RASS and CAM-ICU assessments could be completed in 1 to 2 minutes. The majority of nurses reported that they understand delirium, received adequate education on the CAM-ICU, and can assess delirium confidently and accurately. However, a large number of respondents indicated a belief that in their unit, implementation of the CAM-ICU did not enhance patient management. The survey also asked the nurses to identify potential barriers to routine delirium monitoring in a Level I trauma ICU. The most commonly identified barriers, in order of frequency, were time (15/72; 21%), lack of feedback on performance (14/72; 19%), and knowledge (11/72; 15%).

Table 2.

Post Implementation Survey of Trauma ICU Nurses (n=42)

Question	Disagree/Strongly Disagree, %	Neither Agree or Disagree, %	Agree/Strongly Agree, %
I understand what delirium is and the types	9.5	23.8	66.7
I received adequate education on the CAM-ICU and delirium assessments	26.2	11.9	61.9
The CAM-ICU is easy to administer	26.2	31	42.8
I feel confident in administering the CAM-ICU	14.3	26.2	59.5
Patient care is enhanced by the CAM-ICU	47.6	35.7	16.7
My delirium assessments are accurate	11.9	23.8	64.3
The physicians value the CAM-ICU data	54.6	28.6	16.7

Six weeks following the compliance and reliability phase, spot checks performed over a 2-week period revealed continued compliance at an overall rate of 92% (Figure 1). Compliance was 85% (45/53) during day shift, 94% (248/264) during night shift, and 92% (252/274) during the weekend shift.

Figure 1.

Compliance Over the Study Period Including Post-Implementation Spot-Checks. Nursing compliance with performing the CAM-ICU once per shift remained high over the entire study period including the post implementation period when no active compliance monitoring was being performed

Compliance and reliability testing: February 1 – April 16, 2006; Post-implementation follow-up: June 1 – June 16, 2006.

Discussion

Acute brain dysfunction or delirium is one of the most common forms of organ dysfunction experienced by critically ill patients [7, 12–16]. Despite its high prevalence, delirium is often unrecognized by treating physicians and nurses [17]. This difficulty in diagnosis is partly a result of the predominance of the “quiet” or hypoactive form of delirium, which is characterized by negative symptoms including lethargy and inattention, rather than the overt positive symptoms of hyperactive delirium such as agitation and pulling out lines and tubes that may cause harm to patients or staff [18]. Trauma ICUs have been resistant to implement delirium monitoring because of the concern that these instruments have not been studied to date in multiple trauma patients. This investigation and quality improvement project, therefore, helps to address the need for data on delirium monitoring in this large and important facet of critical care.

In this study, we have shown that with a well-structured start up plan and continued education and support, delirium monitoring using the CAM-ICU is feasible and reliable in the trauma population. Nursing compliance was high, with improvement observed and sustained over time even in spite of nurses’ frustration at physician indifference. We observed a compliance increase from approximately 85% during the data collection period to greater than 90% during the post-implementation phase despite the fact that there was no active monitoring by study staff during this phase. In addition, we also found that 1 in 10 nursing shifts actually had more CAM-ICU recordings than were required. Rates were similar between day shift, night shift, and weekend shift, and our compliance rate of 84% is consistent with rates of 84% and 90% seen in a previous study [10]. Reliability was also demonstrated, with substantial agreement between the bedside nurse and the expert evaluator (κ=0.77). Furthermore, inter-rater agreement was sustained in mechanically ventilated and TBI patients.

Our finding that the CAM-ICU is reliable in mechanically ventilated patients has been supported by two previous studies [7, 10] but this is the first report in trauma patients, including those with TBI. While it is clear that the CAM-ICU has not been tested for its validity in diagnosing delirium in patients with TBI, it may serve as an excellent tool to supplement the Glasgow Coma Scale (GCS) in monitoring neurological status in this patient population. The high inter-rater reliability in patients with TBI attests to the fact that the CAM-ICU should be performed in all trauma patients, in order to facilitate implementation of routine monitoring for delirium without having to consider degree of neurological trauma and performing delirium monitoring only in a subset of patients. Such an approach would result in much lower compliance and make it necessary to have detailed algorithms to determine which patients should be monitored for delirium and which should not. By making it mandatory in all patients, delirium monitoring can easily be incorporated into standardized assessments, and changes in the CAM-ICU assessment in patients with TBI should draw the attention of the bedside nurses and medical teams to a change in neurological status that might warrant further investigation.

Our post-implementation survey revealed that the majority of nurses understand delirium, feel they received adequate education, have confidence in administering the CAM-ICU, and believe their assessments are accurate. About 1 in 5 reported that time was a limiting factor in completing “yet another” monitoring point for their patients. To that point, we now teach that this two-step method of monitoring consciousness with the RASS and CAM-ICU actually simplifies and organizes their previous assessments, and brings brevity and efficiency in communication to interdisciplinary rounds. We also found that 12% of the time, the tests were valued enough to be performed more often than required. The survey data also helped us identify misconceptions and opportunities for education. Approximately 10% of responders were not confident in performing the CAM-ICU despite the extensive educational endeavors; this information prompted us in instituting further in-services and also incorporating delirium assessment into the orientation packages for new nurses, as well as the 6-month competency assessments for the ICU nurses. Despite the high compliance rate, a large percentage (47.6%) of respondents also expressed the belief that the CAM-ICU results did not enhance patient care, a feeling derived from the fact that physicians rarely modified treatment plans or medication regimens based on the additional knowledge gained by the diagnosis of delirium. Several nurses questioned the purpose of delirium assessment, since at the time there was no consistent protocol in place for the treatment of a delirious patient. While studies are underway, current best practice for ICU delirium involves systematic strategies for prevention and treatment in conjunction with delirium monitoring. The authors have developed a working protocol that is being adapted for use in our different ICUs and updated as new data become available (see Delirium Management section of www.icudelirium.org). In the absence of large-scale interventional trials, awareness of the presence of delirium should, at a minimum, prompt a variety of non-pharmacological interventions, including patient reorientation, appropriate attempts to restore normal sleep/wake cycles, and a review of the medication profile to remove drugs known to be associated with delirium and allow substitution of potentially less psychoactive medications.

There are several limitations of this study that must be considered. First, all observations were completed by a single expert evaluator, the presence of whom may have led to increased compliance. However, we feel that a single evaluator increased the consistency among the expert CAM-ICU assessments. As clinical pharmacists are now playing a leading role in delirium-related guideline development and educational efforts [6, 19–21], a pharmacist successfully served as the expert evaluator in this study. Second, as some evaluations were likely performed in the same patients by the same nurses, the 1011 evaluations may not have been completely independent, thereby affecting the kappa estimations. Third, the CAM-ICU has not been validated in trauma patients. However, because trauma patients share a lot of commonality with other critically ill patients, we believe that with the exception of patients with neurological injury (including TBI), the CAM-ICU can be used to diagnose delirium in trauma patients. In patients with TBI, the CAM-ICU can be used as an adjunct to the GCS for a more detailed neurological exam given that the instrument had a high agreement between evaluators. Fourth, utilization of the individual CAM-ICU features was not included in our data collection, an important consideration given that many trauma patients, particularly those with TBI, could be classified as delirious simply by having an altered level of consciousness. Finally, physician teams that rotate in the trauma unit were not formally trained in delirium monitoring and, consequently, acknowledgement and treatment of delirium varied, which led to frustration among the nursing staff. This has been addressed by resident and faculty education as well as development and adoption of a management algorithm to help guide physicians on intervention and treatment decisions when provided with RASS and CAM-ICU data from the nursing staff. This apparent lack of physician “buy-in” and nurse valuation may also have been a result of the fact that the CAM-ICU has not been validated in trauma patients. In addition, the low response rate (44%) to the post-implementation survey introduces selection bias that is present in most survey data.

In conclusion we have shown that CAM-ICU monitoring is both feasible and reliable in the trauma population, including those patients on mechanical ventilation and with traumatic brain injury. The bedside nurses were highly compliant in performing the CAM-ICU as standard of care and believed they could perform delirium assessments accurately. In patients with neurological injury, the CAM-ICU can still be reliably used as an adjunct to the Glasgow Coma Scale and may detect subtle neurological changes that would warrant attention. Routine monitoring of delirium should therefore be performed in trauma patients, in accordance with the published guidelines by the SCCM.