Validity and reliability of the Thai version of the Confusion Assessment Method for the Intensive Care Unit (CAM-ICU)

Abstract

Purpose

The purpose of this study was to determine the validity and reliability of the Thai version of the Confusion Assessment Method for the Intensive Care Unit (CAM-ICU), when compared to the diagnoses made by delirium experts.

Patients and methods

This was a cross-sectional study conducted in both surgical intensive care and subintensive care units in Thailand between February–June 2011. Seventy patients aged 60 years or older who had been admitted to the units were enrolled into the study within the first 48 hours of admission. Each patient was randomly assessed as to whether they had delirium by a nurse using the Thai version of the CAM-ICU algorithm (Thai CAM-ICU) or by a delirium expert using the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, Text Revision.

Results

The prevalence of delirium was found to be 18.6% (n=13) by the delirium experts. The sensitivity of the Thai CAM-ICU’s algorithms was found to be 92.3% (95% confidence interval [CI] =64.0%−99.8%), while the specificity was 94.7% (95% CI =85.4%−98.9%). The instrument displayed good interrater reliability (Cohen’s κ =0.81; 95% CI =0.64−0.99). The time taken to complete the Thai CAM-ICU was 1 minute (interquatile range, 1−2 minutes).

Conclusion

The Thai CAM-ICU demonstrated good validity, reliability, and ease of use when diagnosing delirium in a surgical intensive care unit setting. The use of this diagnostic tool should be encouraged for daily, routine use, so as to promote the early detection of delirium and its rapid treatment.

Introduction

Delirium is a disturbance in a person’s consciousness and a change in cognition that develops over a short period and usually fluctuates over time.¹ Delirium is a complication found commonly in intensive care units (ICUs); its incidence in ICUs varies from between 20%−80%, depending on the types of screening tool used and the populations involved.^2–15 Patients who suffer from delirium incur higher medical costs,¹⁶ require longer hospital stays,^4,6,17 and have higher mortality rates than those who do not.^4,6,14,15 In addition, delirium is associated with long-term cognitive impairment.¹⁷

A diagnosis of delirium is quite a challenge for nonpsychiatric medical personnel. Many critically ill patients often require mechanical ventilation, meaning that the symptoms normally presented by delirious patients cannot be diagnosed using standard tools normally used with patients who can communicate verbally. Delirium can go unrecognized when diagnosis relies on clinical presentation alone;¹³ therefore, delirium screening tools are required in order to detect symptoms early and provide for immediate treatment.

In Thailand, delirium screening is not routinely performed by medical staff in ICUs due to a lack of available, standardized tools, meaning that most diagnoses rely on clinical symptoms only. The Confusion Assessment Method algorithm (CAM algorithm) has been available for clinical diagnosis of delirium by nurses or physicians who are non-delirium experts; however, its poor reliability limits the use of this tool among patients in the ICU.^7,18 The Confusion Assessment Method for the Intensive Care Unit (CAM-ICU) is a highly sensitive and reliable tool for the diagnosis of delirium in the ICU.⁴ The CAM-ICU has been translated into several languages, including German,³ Dutch,¹⁹ Swedish,²⁰ Greek,⁸ Portuguese,¹⁰ Spanish,²¹ Chinese,²² and Korean.⁹ A Thai version of the CAM-ICU (Thai CAM-ICU) has already been produced, however, it has not yet been validated. The objective of this study was to determine the validity and reliability of the Thai CAM-ICU when compared to a standard reference diagnostic criteria, as used by delirium experts, in a surgical intensive care unit (SICU).

Materials and methods

This cross-sectional study was approved ethically by the independent Institutional Review Board at the Faculty of Medicine, Chiang Mai University. Three well-trained, critical-care study nurses informed and obtained consent from the participants or their surrogate decision-makers prior to the assessment. The clinicaltrials.gov registration number of this study is NCT 01267630.

Participants and procedures

Based on the prevalence of delirium within ICUs (50%)³ and the acceptable levels of sensitivity and specificity for the CAM-ICU (about 90%),²³ 70 participants were recruited. Those recruited were elderly people 60 years of age and older, who had been admitted within the previous 48 hours to a 14-bed SICU and sub-SICU at Maharaj Nakorn Chiang Mai Hospital. Recruitment took place from February 2011 to June 2011. The participants were excluded if they had severe visual or auditory impairment, were likely to pass away within 24 hours, had a poor command of Thai, were unwilling to participate, or were comatose.

A research nurse and a delirium expert were responsible for assessing selected admissions within the first 48 hours of arrival at the ICU between the times of 8:30 am and 5 pm on weekdays. The nurse carried out the assessments using the Thai CAM-ICU, while the delirium expert used criteria from the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, Text Revision (DSM-IV-TR). The assessments were randomly assigned at interrater intervals of no more than 30 minutes. Each participant was evaluated twice, once by a nurse (SM or ND) and once by a delirium expert (NW or PD), independently.

Before screening for delirium, each participant was assessed using the Richmond Agitation and Sedation Scale (RASS) to determine the level of sedation.²⁴ RASS is a ten-point scale that ranges from +4 (combative) to −5 (deeply sedated or unresponsive). If participants are alert, agitated, or combative, they are given a score of +4 to 0. If participants are drowsy or awake only briefly, but are able to respond to verbal stimuli, they are given a score of −1 to −3. If patients do not respond to verbal stimuli at all and do not respond to any physical stimuli, they are scored −4 to −5. In this study, all participants needed to score −3 or better using RASS in order to take part. A positive mental status change was defined as one where a participant showed evidence of an acute onset of mental status change when compared to the baseline upon admission, or any fluctuation in their RASS score, within 24 hours of being initially assessed.

The Attention Screening Examination (ASE) tool was used to assess inattention. The ASE consists of visual and auditory components through the recognition of ten letters. Inattention was diagnosed when patients scored less than eight on any component. To test disorganized thinking, participants were asked to answer a set of four questions correctly. An alteration of consciousness was rated when the RASS score was not zero.

Development of the Thai version of the CAM-ICU

The Thai version of the CAM-ICU was developed by the third author (SM) of this paper, following the “Principles of Good Practice for the Translation and Cultural Adaptation Process for Patient-Reported Outcomes Measures.”²⁵ First, SM translated the CAM-ICU into Thai. Then a bilingual translator, who had never been exposed to the original CAM-ICU, did a back translation. Both translators reached a consensus after comparing and adapting the instrument so that its meaning corresponded to Thai culture.

The CAM-ICU tool assesses four features: (1) the acute onset of mental status changes, or a fluctuating course, (2) inattention, (3) disorganized thinking, and (4) altered levels of consciousness. Delirium is diagnosed when a participant manifests features 1 and 2, plus either feature 3 or 4. The training manual for the Thai CAM-ICU is currently available from the website www.icudelirium.org.

CAM-ICU training and interrater reliability

To ensure the reliability of the CAM-ICU, two critical care nurses (SM and ND) who are authors of this study performed Thai CAM-ICU assessments on ten patients. A gold standard evaluation was also performed by two delirium experts (NW and PD) on another ten patients using DSM-IV-TR criteria. There was found to be perfect agreement in each group of researchers (Cohen’s κ =1 for both).

Statistical analysis

Demographic variables are presented as mean ± standard deviation for continuous data, and frequency and percentage for categorical data. The performance of the Thai version of CAM-ICU algorithms was calculated by using sensitivity, specificity, and predictive values (positive predictive value and negative predictive value). The likelihood ratio of the test (sensitivity/1 − specificity) including 95% confidence intervals (CIs) were estimated. A receiver-operating characteristic (ROC) curve was presented to demonstrate the performance of the Thai version of the CAM-ICU.²⁶ Cohen’s κ coefficient was used to demonstrate interrater reliability between critical care nurses and delirium experts. All statistics were analyzed using STATA 11.0 (StataCorp LP, College Station, TX, USA). A P-value <0.05 was considered statisticaly significant.

Results

Patient characteristics

Over a 5-month period, 92 patients were enrolled in the study. Three were excluded because they did not want to participate, or their relatives did not want them to. Seventeen patients were discharged and two patients died before an assessment could take place. Figure 1 shows the details of the study flow.

Figure 1.

Study patient selection process.

Abbreviation: n, number.

A total of 70 patients completed the study. The baseline patient characteristics are presented in Table 1. The majority of the study patients were male (64.3%) and most had also graduated from primary school (64.7%). The percentage of delirious patients who required mechanical ventilation was 22.7% and the mortality rate during hospitalization was 18.2%.

Table 1.

Baseline characteristics of the study participants

Characteristics	Frequency (n=70)
Age (years), median (interquartile range)	71.5 (62.0−78.0)
Male, n (%)	45 (64.3%)
Body Mass Index (kg/m2), mean ± SD	20.54±3.32
Level of education, n (%)
Primary school	44 (64.7%)
Secondary school	8 (11.8%)
University	6 (8.8%)
Unknown	10 (14.7%)
APACHE II, median (interquartile range)	13 (10−17)
Glasgow Coma Score, mean ± SD	14.3±1.29
Delirium, n (%) by DSM-IV-TR	13 (18.6%)
Delirium, n (%) by CAM-ICU	12 (17.1%)
Mechanically ventilated, n (%)
Delirium	5 (22.7%)
Causes of ICU admission, n (%)
Sepsis	9 (12.8%)
Respiratory failure	7 (10.0%)
Myocardial infarction/congestive heart failure	5 (7.1%)
Cardiac arrest/shock	3 (4.3%)
Hepatic or renal failure	3 (4.3%)
Other causes	43 (61.4%)
Hospital stay (days), median (interquartile range)	17 (11−25)

Notes: Data are presented as number and percent, mean ± SD, or median (25th−75th percentiles).

Abbreviations: APACHE II, Acute Physiologic and Chronic Health Evaluation II; CAM-ICU, The Confusion Assessment Method for the Intensive Care Unit; DSM-IV-TR, Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, Text Revision; n, number; SD, standard deviation; ICU, intensive care unit.

Validity and interrater reliability of the Thai CAM-ICU

The assessments made by the two critical care nurses and the delirium experts are shown in Table 2. The prevalence of delirium in SICU patients during the first 48 hours after admission was found to be 18.6%. The sensitivity of the Thai CAM-ICU’s algorithms was 92.3% (95% CI =64.0%−99.8%), while their specificity was 94.7% (95% CI =85.4%−98.9%). The positive and negative predictive values were 80.0% (95% CI =51.9%−95.7%) and 98.2% (95% CI =90.3%−100.0%), respectively. The likelihood ratio of Thai CAM-ICU was 17.5 (95% CI =5.8−53.3). The results of the ROC analysis showed the area under the curve was 0.93 (95% CI =0.85−1.00). Cohen’s κ between the study nurses and the delirium experts was 0.81 (95% CI =0.64−0.99), with a P-value of <0.001.

Table 2.

Comparison of delirium assessments carried out by the critical care study nurses and the delirium experts

Delirium by CAM-ICU (nurses)	Delirium	No delirium
Delirium by DSM-IV-TR (experts)
Delirium	12	3
No delirium	1	54

Abbreviations: CAM-ICU, Confusion Assessment Method for the Intensive Care Unit; DSM-IV-TR, Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, Text Revision.

Table 3 presents the results of previous CAM-ICU validations, as shown in the literature. In this study, four CAM-ICU rating misclassifications were made by the two critical care nurses, while three false-positive assessments and one false negative assessment were identified. Among the three patients who gave false positives, two had hypovolemic shock and one developed septic shock during the ICU admission. One patient who gave a false negative was diagnosed with delirium plus a hypoalert condition by a delirium expert.

Table 3.

Validation of CAM-ICU versions in the literature

CAM-ICU version	Total number (n)	Age (years)	Sensitivity (95% CI)	Specificity (95% CI)	PPV (95% CI)	NPV (95% CI)	Cohen’s κ (95% CI)
Ely (original CAM-ICU)30	96	ND	(93.0%−100.0%)	98.0%−100.0%	92.0%−100.0%	98.0%−100.0%	0.97* (0.92−0.99)
German3	54	ND	(88.0%−92.0%)	100.0%	100.0%	91.0%−94.0%	0.96*
Spanish21	129	ND	79.4% (63.2%−89.7%)	97.9% (92.6%−99.4%)	86.7% (62.1%−96.3%)	86.7% (62.1%−96.3%)	0.79* (0.66−0.92)
Portuguese10	119	>18	72.5% (ND)	96.2% (ND)	90.6% (ND)	87.4% (ND)	0.96**
Greek8	71	ND	95.2% (74.0%−98.0%)	85.0% (72.0%−93.0%)	76.0% (58.0%−88.0%)	91.7% (84.0%−99.0%)	0.75* (0.59−0.91)
Korean9	22	ND	(77.4%−89.8%)	(72.4%−75.8%)	ND	ND	0.81*
Chinese22	126	≥50	(91.8%−93.4%)	(87.7%−90.8%)	(87.7%−90.3%)	(92.2%−93.4%)	0.92* (0.88−0.97)
Present study	70	≥60	92.3% (64.0%−99.8%)	94.7% (85.4%−98.9%)	80.0% (51.9%−95.7%)	98.2% (90.3%−100.0%)	0.82*

Notes:

^*Correlation between CAM-ICU and standard reference (DSM-IV-TR).

^**Correlation between CAM-ICU and CAM-ICU Flowsheet.

Abbreviations: CAM-ICU, Confusion Assessment Method for the Intensive Care Unit; CI, confidence interval; DSM-IV-TR, Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, Text Revision; n, number; ND, not defined in the study; NPV, negative predictive value; PPV, positive predictive value.

Time spent by nurses using the Thai CAM-ICU algorithms

The average time taken by the nurses to perform the Thai CAM-ICU algorithm was significantly less than the time taken by the psychiatric staff to do the full psychiatric assessments using DSM-IV-TR (1.59±0.86 minutes versus 8.48±4.81 minutes) (P<0.001). The minimum and maximum times used to calculate the Thai CAM-ICU algorithms were 0.5 minutes and 5 minutes, while the minimum and maximum times used for the DSM-IV-TR calculations were 2 minutes and 25 minutes.

Discussion

This study demonstrated that the Thai version of CAM-ICU has a high validity and reliability, and that it takes less time to complete than the full psychiatric assessments. The validity of the Thai CAM-ICU’s algorithms is comparable to the original CAM-ICU developed by Ely et al⁴ showing sensitivities of between 95%−100%, and specificities of between 89%−93%. Previous studies have validated the CAM-ICU in different settings, including in an SICU, where Guenther et al³ reported the validity of the CAM-ICU Flowsheet in a SICU. In their study, the CAM-ICU Flowsheet was modified from the original, which meant changing the numbering for Features 3 and 4 to allow rapid assessments to take place. Their results show a high sensitivity, specificity, and reliability (88%−92% versus 100%, Cohen’s κ =0.96).³ In this study, and when used by the two critical care nurses, the sensitivity and the specificity of the Thai CAM-ICU algorithm were 92.3% and 94.7%, respectively. The Thai CAM-ICU algorithms revealed a high diagnostic value when taking into account the value of the likelihood ratio (LLR) of the test when given by the two nurses (LLR =11.8 and 30.0) and the area under the ROC. When comparing the performance of the test, the LLR of the test was superior to the positive and negative predictive values and it was not affected by the prevalence of disease.²⁷

In addition, the area under the ROC confirmed the power of the Thai CAM-ICU algorithm in terms of discriminating between delirious and nondelirious patients.²⁶ The present study demonstrated that the Thai CAM-ICU had a higher sensitivity and specificity than other validated versions of the CAM-ICU algorithms, including the Portuguese¹⁰ (sensitivity =72.5% and specificity =96.2%), Korean⁹ (sensitivity =77.4% for rater 1 and 89.8% for rater 2 and specificity =72.4% for rater 1 and 75.8% for rater 2), and Greek versions⁸ (sensitivity =95.2% and specificity =85.0%). In addition, the validity of the Thai version of CAM-ICU was comparable to the Chinese version.²² A good agreement was shown between the two critical care nurses when diagnosing for delirium (Cohen’s κ =0.82). The level of agreement between the two sets of raters (critical care nurses and psychiatrists) was comparable to that found with the Korean version⁹ (Cohen’s κ =0.81 and 0.75, respectively), but lower than the Greek and Spanish versions.^8,21

The present study revealed some misclassifications within the Thai CAM-ICU ratings (three false positives and one false negative). In the three cases of patients showing false positives with the Thai CAM-ICU, delirium was diagnosed based on a positive test for Features 1, 2, and 4. For the patient with a false negative, a “nondelirious” state was diagnosed due to a positive result for Feature 1 and a negative result for Feature 2, while the psychiatrist gave positive results for all the reference standard domains. There are two possible explanations for these misclassifications. First, the use of sedative drugs may have impaired participants’ consciousness levels, which would have increased the chances of delirium being diagnosed by the CAM-ICU.²⁸ Haenggi et al²⁸ recommended that sedation should be discontinued, and the level of sedation should be re-evaluated before any assessment for delirium was made. One previous study demonstrated that a patient’s level of education may have an influence on the results for Feature 2: ASE.²²

The present study found that 64.7% of patients had graduated from primary school. Wang et al²² concluded that the reliability of the ASE test is lower for illiterate patients (Cohen’s κ =0.61) and patients with only a primary school education (Cohen’s κ =0.70). They stated that giving an explanation of the test methods before doing the ASE auditory evaluation is very important for illiterate patients. Use of the Thai CAM-ICU in routine, clinical practice will improve the accuracy of delirium diagnosis for critically ill patients, and particularly in ICU settings, providing a standardized tool and serving as a benchmark when comparing against CAM-ICU tools from other countries.²⁹

This study had a number of strengths. First, we compared the delirium diagnosis performance of the Thai CAM-ICU against a standard reference used by psychiatrists, and second, evaluations were carried out by both critical care nurses and delirium experts in order to reduce individual variability during the assessment process. Also, the evaluations carried out by the two sets of raters (study nurses and psychiatrists) on the same patient were never more than 30 minutes apart. This approach reduced the risk of variability in terms of diagnosis due to fluctuations in the condition.

However, the study also had a few limitations. First, the two nurses used in the study were not able to evaluate all the patients over the study period, due to the manpower situation in the SICU. As a result, we could not fully demonstrate the differences between the nurses’ and psychiatrists’ assessments. However, the authors did perform an interrater reliability assessment of the two critical care nurses, until perfect agreement had been achieved, before recruiting the study patients. Next, this study did not demonstrate motoric subtypes of delirium, which can be categorized by the level of RASS found. The presence of delirium symptoms is more important than the diagnosis of delirium, because patients with different subtypes of delirium, such as hypoactive or hyperactive delirium, require different treatments and prognoses.

Conclusion

The Thai CAM-ICU demonstrated good validity and reliability, and proved easy to use when diagnosing delirium in a surgical intensive care unit setting. The use of this diagnostic tool in daily practice should therefore be promoted, in order to facilitate the early detection and rapid treatment of delirium.