Evaluation of PRE-DELIRIC (PREdiction of DELIRium in ICu patients) delirium prediction model for the patients in the intensive care unit

Abstract

Introduction.

Delirium not only compromises patient care, but is also associated with poorer outcomes: increased duration of mechanical ventilation, higher mortality, and greater long-term cognitive dysfunction. The PRE-DELIRIC model is a tool used to calculate the risk of the development of delirium. The classification of the patients into groups by risk allows efficient initiation of preventive measures. The goal of this study was to validate the PRE-DELIRIC model using the CAM-ICU (The Confusion Assessment Method for the Intensive Care Unit) method for the diagnosis of delirium.

Materials and methods.

Patients admitted to the University Hospital of Vilnius during February 2015 were enrolled. Every day, data were collected for APACHE-II and PRE-DELIRIC scores. Out of 167 patients, 38 (23%) were included and screened using the CAM-ICU method within 24 hours of admission to the ICU. We defined patients as having delirium when they had at least one positive CAM-ICU screening or haloperidol administration due to sedation. To validate the PRE-DELIRIC model, we calculated the area under receiver operating characteristic curve.

Results.

The mean age of the patients was 69.2 ± 17.2 years, 19 (50%) were male, APACHE-II mean score 18.0 ± 7.4 points. Delirium was diagnosed in 22 (58%) of 38 patients. Data used for validation of the PRE-DELIRIC model resulted in an area under the curve of 0.713 (p < 0.05, 95% CI 0.539–0.887); sensitivity and specificity for the patients with 20% risk were, accordingly, 77.3% and 50%; 40% risk – 45.5% and 81.3%, 60% – 36.4%, and 87.5%.

Conclusions.

The PRE-DELIRIC model predicted delirium in the patients within 24 hours of admission to the ICU. Preventive therapy could be efficiently targeted at high-risk patients if both of the methods are to be implemented.

INTRODUCTION

Delirium is defined as an acute, usually reversible, cerebral dysfunction that manifests in a wide range of neuropsychiatric symptoms. The main characteristics are a sudden onset, behavioural changes (from aggression to apathy), and inattention. Even to this day, the aetiology and pathophysiology of this condition remain unclear. However, it is known that delirium is associated with such negative outcomes as prolonged stay in the hospital, decreased physical functionality, and increased health care costs (1, 2).

The incidence of delirium in different departments ranges from 19% to 82%, the highest being in the intensive care unit (ICU) (3). The patients who develop delirium in the ICU have a two-to-four-times increased risk of death both in and out of hospital compared to the patients who do not (4, 5). As a result, delirium has attracted attention of many researchers worldwide for the past decade. This has led to the development of diagnostic screening tools, prognostic risk models, and effective prevention methods, which are now becoming more and more part of routine in the care of critically ill patients.

Despite the fact that the development of delirium has definite organic causes, there are no laboratory investigations or imaging tests that could confirm this condition. Thus, the diagnosis of delirium is based on clinical criteria, which are often unrecognized and overlooked. In the view of the fluctuating nature of delirium, it has been acknowledged that cognitive functions of the patients have to be monitored in the same manner as other vital functions (heart rate, blood pressure, etc.). For this reason, various screening tools for early detection of delirium in critically ill patients were developed.

Seven tests have been developed to evaluate delirium in the ICU. Only four tests have been validated against DSM-IV criteria (Diagnostic and Statistical Manual of Mental Disorders IV) and only two are useful for intubated patients (6–10).

The most widely used delirium screening method is CAM-ICU (The Confusion Assessment Method for the Intensive Care Unit), which has been validated in high-quality studies, with sensitivity of 79–100%, specificity of 87–100% (6–11). Even though the effectiveness of these instruments is clearly proven, the implementation is difficult due to the lack of staff members, a misunderstanding regarding delirium risk factors, outcomes, and disbelief that this screening approach may benefit (7, 12).

Prevention is an important delirium intervention. According to the Pain, Agitation and Delirium Guidelines (13), non-pharmacological measures such as early mobilization, patient orientation, sleep, and friendly environment are highly recommended for the prevention while the pharmacological approach is still lacking clear evidence. However, the implementation of such interventions requires considerable time and increases the workload significantly. Therefore, various prediction models that could help identify high-risk patients have been developed.

One of the models to predict delirium according to risk factors – the PRE-DELIRIC model – was internationally validated, though its true predictive value can vary from one ICU to another (14). The PRE-DELIRIC model was created in 2013 in the Netherlands based on a recent systemic review of delirium risk factors (15). This model includes ten objectively and clearly defined risk factors known within 24 hours of admission to the ICU: age, Acute Physiology and Chronic Health Evaluation II (APACHE II) score, coma, urgent admission (unplanned ICU admission), admission category (surgical, medical, trauma, neurology/neurosurgical), infection, coma, use of sedatives, morphine use (three dosage groups), serum urea, and metabolic acidosis (Table 1). Neither alcohol and drug abuse nor dementia were included in the model, because these factors were considered as being of high risk for development of delirium.

Table 1.

Formula for the PRE-DELIRIC model (14)

Formula for the PRE-DELIRIC model

Risk of delirium = 1/(1+exp–(–4.0367)

+0.0183 × age

+0.0272 × APACHE-II score

+0 for non-coma, or 0.2578 for drug-induced coma, or 1.0721 for miscellaneous coma, or 1.3361 for combination coma

+0 for surgical patients, or 0.1446 for medical patients, or 0.5316 for trauma patients, or 0.6516 for neurology/neurosurgical patients

+0.4965 for infection

+0.1378 for metabolic acidosis

+0 for no morphine use, or 0.1926 for 0.01–7.1 mg/24 h morphine use, or 0.0625 for 7.2–18.6 mg/24 h morphine use, or 0.2414 for >18.6 mg/24 h morphine use

+0.6581 for the use of sedatives

+0.0141× urea concentration (mmol/L)

+0.1891 for urgent admission

The intercept of the scoring system is expressed as –4.0369; the other numbers represent the shrunken regression coefficients (weight) of each risk factor.

The values were recalibrated during an international study for the PRE-DELIRIC model (14).

Our aim was to test the PRE-DELIRIC model and compare the results with other studies. The main goal of the study was to validate the PRE-DELIRIC model for our ICU patients by using novel diagnostic tools. Indirectly, we aimed to spread the up-to-date knowledge about delirium to our staff members.

METHODS

This prospective observational study was performed at the 32-bed mixed medical-surgical ICU at the Centre for Anaesthesiology, Reanimatology, and Critical Care Medicine of the Vilnius University Hospital.

Fig. 1.

Delirium assessment tool, Confusion Assessment Method for the ICU (CAM-ICU) flow sheet

All consecutive 167 patients older than 18 years of age admitted to the ICU from 1 to 28 February 2015, were enrolled to the study. In order to reliably detect delirium, we used a validated delirium assessment tool, the CAM-ICU (Fig. 1). Out of 167 patients, 38 (23%) were included and screened using the CAM-ICU method within 24 hours of admission to the ICU. Patients were not included if they were: comatose during the ICU stay, the length of stay in the ICU less than one day, unable to understand Lithuanian, suffered from receptive aphasia or mental disability, or had a history of alcohol or drug abuse (Fig. 2). To learn the method, we used freely accessible manual with detailed guidelines and cases analyses as well as video materials (16). After training, we carried out a pilot study of inter-rater reliability of the Lithuanian version of the CAM-ICU. The inter-rater reliability of two assessors was above 0.80 Cohen’s kappa, indicating a very high agreement. To ensure if there was no change in the level of consciousness during the shift, we asked the nurses to describe patient’s consciousness during the night and checked the medication list for haloperidol administration due to sedation.

Fig. 2.

Flowchart of inclusion criteria

Delirium was defined as at least one positive CAM-ICU screening during a patient’s complete stay and/or a patient was treated with haloperidol. Additionally, we distinguished three different types of delirium according to psychomotor symptoms indicated by the Richmond Agitation Sedation Scale (RASS) score: hyperactive (RASS > 0, hyperalert, agitated), hypoactive (RASS ≤ 0, hypoalert, lethargic), and mixed (RASS score varied) (17, 18). The ten predictors of the PRE-DELIRIC model were collected within the first 24 h of ICU admission. Additionally, we collected data about patient’s demographics, outcome, and duration of mechanical ventilation, the length of stay in the ICU and in the hospital. The bioethical agreement was not obtained because the design of the study was observational, without intervention, and had no influence on the treatment decision.

Statistical analysis

Quantitative normally distributed variables were presented as means ± standard deviation (SD) and non-normally distributed variables as median and the interquartile range (IQR). Categorical variables were expressed as actual numbers and percentages. Variables were assessed in the groups of the patients stratified by the delirium status: delirious (n = 22) and non-delirious (n = 16).

Statistical analysis was performed using SPSS 19 (SPSS, Inc., Chicago, IL, USA). Tests were chosen according to the sample size and data distribution. Quantitative and not normally distributed variables were compared using the Mann-Whitney test. For normally distributed variables, we applied the t-test. Categorical variables were compared by using χ² or a Fisher exact test. To assess the accuracy of the PRE-DELIRIC prediction model ROC curve was designed. All statistical tests were two-sided. P < 0.05 was considered significant for all tests.

RESULTS

The mean age of the patients was 69.2 ± 17.2 years, 19 (50%) were male; the mean APACHE II score was 18.0 ± 7.4 points. The majority of the patients were neurological/neurosurgical (45%), followed by surgical (24%), medical (16%), and trauma (16%). Overall, 21 (54%) patients had surgery one day prior or during their stay in the ICU.

We confirmed delirium in 22 of 38 (58%) of included patients by screening them with the CAMICU, eight of them had hyperactive delirium, 12 – hypoactive, two were mixed cases (Table 2). Haloperidol was administered due to sedation to six out of ten patients with hyperactive or mixed delirium. A median day to the onset of delirium was 1 (IQR 1–2) and delirium persisted for one day (IQR 1–2).

Table 2.

Clinical characteristics of ICU patients

Variable	N = 38
Age (years), median (IQR, range)	71 (65–81)
Sex (male, %)	19 (50%)
APACHE II score, mean ± SD	18.0 ± 7.4
Reasons for ICU admission:
neurological/neurosurgical	17 (45%)
surgical	9 (24%)
medical	6 (16%)
trauma	6 (16%)
Delirium	22 (58%)
hyperactive	8 (36%)
mixed	2 (9%)
hypoactive	12 (55%)
Days to the onset of delirium, median (IQR, range)	1 (1–2, 9)
Duration of delirium (days), median (IQR, range)	1 (1–2, 2)
Mechanical ventilation	25 (58%)
Sedation	15 (40%)
Length of stay in the ICU (days), median (IQR, range)	2 (2–4)
Median (IQR, range) length of stay in the hospital (days)	14 (8–19)

Data are medians with the 25th–75th quartiles (IQR) range and mean with standard deviation or number of patients (%).

The comparison of groups of delirious patients (n = 22) and non-delirious patients (n = 16) showed that delirium was associated with significantly more frequent mechanical ventilation (p = 0.002), longer duration of mechanical ventilation (p = 0.001), a higher RASS score (p = 0.001), and a PRE-DELIRIC score (p = 0.048) (Table 3). APACHE II scores, days with sedation, duration of the stay in the ICU, and mortality rate were not significantly different between the groups.

Table 3.

Comparison of delirious and non-delirious patients

	Delirious (n = 22)	Non-delirious (n = 16)	P value
Age (years), median (IQR, range)	73.5 (67.5–81, 73)	69 (54.75–81.75, 65)	0.407
APACHE-II score, median (IQR, range)	17.5 (12–23.5, 31)	17 (12.25–22.5, 26)	0.593
Sex (male, %)	13 (59%)	6 (38%)	0.189
Mechanical ventilation	19 (86%)	6 (38%)	0.002
Days on mechanical ventilation, median (IQR, range)	1 (1–4, 18)	0 (0–1, 4)	0.001
Sedation	11 (50%)	4 (25%)	0.120
Days with sedation	0.5 (0–1, 16)	0 (0–0.25, 5)	0.117
RASS score, median (IQR, range)	1.4 (1–2.5, 4.2)	0 (0–0.6, 3.5)	0.001
PRE-DELIRIC score, mean ± SD	0.398 ± 0.19	0.270 ± 0.19	0.048

The ability of the PRE-DELIRIC model to predict the prognosis correctly was tested by classification matrices (Table 4). The highest overall correct classification of 0.74 was obtained using a decision criterion of 28.4%, with the sensitivity of 72.7% and the specificity of 75.0%.

Table 4.

Correct classification rate, sensitivity, specificity, true positive and false positive rates for the PRE-DELIRIC model

Cut-off	Sensitivity	Specificity	Positive likelihood ratio	Negative likelihood ratio	Correct classification rate
20%	77.3%	50%	1.55	0.45	0.66
40%	45.5%	81.3%	2.42	0.67	0.61
50%	36.4%	87.5%	2.91	0.73	0.58

The ROC curve for the PRE-DELIRIC model equation applied to our data is shown in Fig. 3. The area under the ROC curve was 0.713 (standard error 0.088, 95% CI 0.539–0.887, p < 0.05) and confirmed the good discrimination of the PRE-DELIRIC model.

Fig. 3.

The ROC curve for the PRE-DELIRIC model (curved line). The relationship between true-positive (sensitivity) and false-positive (1 – specificity). The diagonal line is the line of chance performance. The area under the curve is 0.713 (p < 0.05, 95% CI 0.539–0.887). Red dots represent the cut-off values of 20%, 40% and 50%

DISCUSSION

We tested an internationally validated delirium prediction model (PRE-DELIRIC) for our ICU patients. The main goal of the study was to evaluate the PRE-DELIRIC model for delirium prediction after admission to the ICU. The model resulted in an area under the curve of 0.713, indicating a moderate prediction value. In the original study, the results were slightly better – the AUROC was equal to 0.76 (14). The biggest difference lies in the broader range of the confidence interval that can be explained by a smaller sample size in our study.

The original delirium prediction model was developed in 2012. The data for the model were derived from 1613 patients from a single hospital, 25.5% of whom developed delirium (19). Six original research papers, including five prospective cohort studies and one retrospective record analysis, were selected in the systematic review resulting in the identification of 25 risk factors (15). Of these potential risk factors, alcohol misuse, dementia, use of an epidural catheter, hyperamylasaemia, hyponatraemia, use of dopamine, and use of lorazepam were excluded because of a prevalence rate below 10%. After multivariate logistic regression analysis with the remaining risk factors, the PRE-DELIRIC model, which consisted of ten risk factors, was constructed. In the derivation cohort, the model had 87% accuracy rate for distinguishing between patients who did and did not develop delirium, whereas the predictions of physicians and nurses attending a subgroup of 124 patients were just 59% accurate. This team validated the model in a further 549 patients from the same hospital, finding that it had an accuracy of 89%.

In 2013, the model was tested in six other countries (Australia, Belgium, Germany, Spain, Sweden, and United Kingdom) (14). This resulted in the recalibration of the model. Despite differences of the predictors between the countries, the discriminative power of the PRE-DELIRIC model was not affected, indicating that the most important predictors for the development of delirium in the ICU are included in the model. In our study, the incidence of delirium was higher in comparison to other similar design studies (58% vs. 12–39%). One of the main reasons might be the fact that there were a lot of neurological and neurosurgical patients (45% vs. 2–20%). Brain injury is especially associated with an increased risk of delirium and this is reflected in the PRE-DELIRIC model by high coefficients given for the patients in coma states and neurological and neurosurgical pathology (respectively, 0.25–1.34 and 0.65). Otherwise, the data showed high variability in regard to patient demographics depending on specific case-mix: 42% vs. 12–46% for comatose patients, 32% vs. 6–58% for morphine use, 40% vs. 16–82% for use of sedation, 87% vs. 35–97% for urgent admission and 34% vs. 15–55% for presence of infection (14).

In proportion to risk factors, these differences reflect the need to recalibrate PRE-DELIRIC model as each ICU can have its own specific patient demographics or tendencies in treatment that might influence the predictive power of the model. Even though it was proved to be superior to prediction of the attending caregiver (19), it remains essential to test the model in order to refine the coefficients for each risk factor while also leaving a possibility to add new risk factors.

Additionally, in our study we compared delirious and non-delirious patients. The results were similar to the general trends in scientific literature: delirious patients had higher median RASS scores, spent a significantly longer time on mechanical ventilation and in the ICU (1, 2, 5, 20, 21).

High-risk patients can be easily and reliably identified and can benefit from various preventive measures. It is proven that non-pharmacological interventions such as early mobilization, keeping the normal circadian rhythm, cognitive stimulation, reduction of sedatives and narcotic analgesics reduce the incidence of delirium as well as its duration (3, 14, 19–21). It is even suggested that primary prevention of delirium is probably the most effective treatment strategy. Ideally, such measures should be provided for all ICU patients. However, it is not possible as they require a lot of resources and time. Moreover, there might be some benefits of haloperidol prophylaxis for the patients undergoing surgery (23). In order to avoid potential side effects of the drugs, only high-risk patients should be given drug prophylaxis. However, pharmacological preventive strategies still lack solid evidence and they have not been recommended yet (13). Lastly, the ability to stratify risk can help physicians explain risks to patients and their families, and can help the families to better understand the possible outcomes.

The only challenge left is the implementation of the model. Even though the risk factors of the model are easy to note down, it requires a significant effort to collect them. Thus, the workload of nurses increases and this adds a temporary barrier for the use of the model. However, an important part of this workload will be eliminated if the model is implemented in the electronic medical system calculating prediction values automatically for each patient.

Otherwise, we experienced a negative attitude from our staff towards application of the CAMICU and the need to use a significant amount of resources for delirium prevention. From the experience in other countries, the negative attitude is a common problem (7, 11). However, the situation tends to change significantly after the implementation. With a better understanding and experience, nurses and physicians started to value the tool.

We strongly believe that a better understanding of delirium consequences and new ways to manage it would lead to a positive change in the culture of the ICU. Thus, in order to ease the implementation of new diagnostic and prognostic methods for delirium in the ICU, the first step would be the education of staff members and understanding of their current knowledge and attitude towards this condition.

CONCLUSIONS

The PRE-DELIRIC model can reliably predict a manifestation of delirium within 24 hours of admission to the ICU. However, the true value of the model can only be evaluated after its implementation into clinical practice. Only then we could see if identifying high-risk delirium patients has an effect on the attitude of physicians and nurses towards prevention.

Gabrielė Linkaitė, Mantas Riauka, Ignė Bunevičiūtė, Saulius Vosylius