Predicting delirium risk using an automated “Mayo Delirium Prediction” tool – Development and Validation of a risk stratification model

Abstract

Objective:

To develop a delirium risk prediction tool that is applicable across different clinical patient populations and can predict the delirium risk at admission to hospital.

Methods:

A retrospective study that included 120,764 patients admitted to Mayo Clinic between January 1, 2012 and December 31, 2017, with age 50 and greater. The study group was randomized into a derivation cohort (n=80,000) and a validation cohort (n=40,764). Different risk factors were extracted and analyzed utilizing least absolute shrinkage and selection operator (LASSO) penalized logistic regression.

Results:

The area under the ROC curve (AUROC) for Mayo Delirium Prediction (MDP) tool using derivation cohort was 0.85 (95% C.I. 0.846 to 0.855). Using the regression coefficients obtained from derivation cohort, predicted probability of delirium was calculated for each patient in validation cohort. For the validation cohort, AUROC was 0.84 (95% C.I. 0.834 to 0.847). Patients were classified into one of the three risk groups based on their predicted probability of delirium: Low (≤5%), Moderate (6-29%) and High (≥30%). In the derivation cohort, observed incidence of delirium was 1.7%, 12.8%, and 44.8% (low, moderate, and high risk respectively) which is similar to the incidence rates in validation cohort of 1.9%, 12.7% and 46.3%.

Conclusion:

The Mayo Delirium Prediction tool was developed from a large, heterogeneous patient population with good validation results and appears to be a reliable automated tool for delirium risk prediction with hospitalization. Further prospective validation studies are required.

Introduction:

Delirium incidence varies across clinical settings and is noted to be between 30-50% in general care wards of hospital and as high as 80% in ICU¹. Delirium results in prolonged hospital stay, readmission, increased mortality, institutionalization, and long-term cognitive impairment². Despite the high incidence and association with adverse outcomes, the diagnosis is still missed a majority of the time in routine clinical care (60–80%) due to under-diagnosis and under-documentation¹. Delirium is a syndrome that is common in older adults, characterized by an acute onset of confusion associated with fluctuating course, inattention, and disorganized thinking or altered level of consciousness. The Confusion Assessment Method (CAM) tool developed by Sharon Inouye is a widely used clinical tool for diagnosis of delirium³. Variations of CAM and other diagnostic tools have been developed over the years for the diagnosis of delirium^{4, 5, 6}.

Delirium risk factors are broadly classified as predisposing risk factors and precipitating risk factors^{7, 8}. The interplay of a multitude of these risk factors, which varies on an individual basis, results in delirium occurrence⁷. Effective reduction in delirium incidence by 40% was achieved using a multicomponent intervention program in high-risk individuals^{9, 10}. Prevention efforts include a thorough understanding of the existing predisposing risk factors, as well as prevention and management of precipitating risk factors. Multicomponent delirium prevention programs have been shown to be cost effective in older adult patients on medical ward populations¹¹ and many components of delirium prevention such as encouraging the use of glasses or hearing aids in patients with sensory impairment should be considered standard of care for all patients in the hospital. However, some delirium prevention programs are resource intense and may not be feasible or practical for widespread application in all hospitalized patients^{12, 13, 14}

A reliable delirium prediction tool is needed to improve risk stratification, so that effective, resource-intense programs are appropriately targeted and prioritized^{15, 16}. Focusing prevention resources on patients most likely to benefit has the potential to reduce delirium incidence, improve patient outcomes, and improve the overall value and resource stewardship of the delirium prevention program¹⁶. A systematic review of delirium prediction modeling tools identified several limitations of current models, including limited study population heterogeneity, small sample size, and lack of prediction accuracy and reliability¹⁷. These models had poor validation results limiting broad clinical implication. No one particular delirium prediction tool is used widely in clinical practice, creating a gap for optimization in delirium care. We aimed to develop an accurate and reliable delirium prediction tool that is derived from a large, heterogeneous patient population.

Methods:

The Mayo Delirium Prediction (MDP) tool was developed using retrospective patient data at Mayo Clinic Hospitals, Rochester, MN. Patients included in the study were age 50 years and older and admitted to Mayo Clinic Hospital, Rochester during January 1, 2012 to December 31, 2017. Admissions related to acute substance use disorder (intoxication/ withdrawal) as admitting diagnosis were excluded from this analysis. The diagnosis of delirium was obtained either from physician documentation in problem lists or diagnosis sections of notes (diagnosis of encephalopathy/delirium only included) by electronic data extraction, or from nursing documentation of CAM-ICU assessments (routine screening) electronically, which are completed twice daily in all hospitalized patients including non-ICU patients. Any CAM-ICU positive documentation was counted as a delirium episode. Data informatics personnel performed data abstraction and statistical analysis was performed using Statistical Analysis System (version 9.4 (SAS Institute Inc.) and R statistical software version 3.4.1 (R Foundation for Statistical Computing). The Mayo Clinic Institutional Review Board approved this study. Electronic extraction of the data for the derivation study was verified for accuracy by random sampling by the primary author SRP.

A total of 120,764 patients were included in the study. This cohort was then randomly divided into a derivation cohort (n=80,000) and a validation cohort (n=40,764). Data from electronic medical records were obtained for all potential predictor variables and demographic data (Table 1). These variables were selected based on an extensive literature review of risk factors for delirium and in alignment with the practicality of automatic extraction from the electronic medical record. The variables along with other patient characteristics are summarized using mean ± standard deviation (SD), or median (25^th, 75^th) percentile for continuous variables and frequency counts and percentages for categorical variables (Supplemental Table 1). Some of the predictor variables abstracted from the medical record were further categorized as described below (Supplemental Table 1). Pressure ulcers, red blood cell transfusion, and fracture type were categorized as any or none. All of the activities of daily living, hearing impairment, and visual impairment were categorized into a scoring system from 1-3 (Supplemental Table 1). Laboratory test result and vital sign variables were categorized into binary variables with a cut point. All medical comorbidities are classified as either having the disease or not and medication classification as yes or no values for having any prescription of that type. Age, fall risk score, and the Charlson comorbidity scores were left as continuous variables. The overall population, which includes the derivation and validation cohorts for both the medical and surgical populations, had variables of interest extracted from the day of admission (from time of admission until midnight of that day). Missing data were imputed for the derivation datasets only using multiple imputations by fitting a regression model.

Table 1:

Demographics and risk factors abstracted from electronic medical records

Gender

Age

Race & Ethnicity

Functional status (Activity of Daily Living’s of Bathing, Toileting, Eating, Mobility, Oral cares)

Hearing Impairment

Visual Impairment

Charlson index score

Fall risk score

Medical comorbidities (Congestive heart failure, Chronic obstructive pulmonary disease, Atrial fibrillation, Diabetes, Chronic kidney disease, Coronary artery disease, Cerebrovascular accident, Cancer)

ICU admission

History of delirium

Dementia

Pressure ulcer

Psychiatric disorder

Depression/Anxiety Disorder

Sleep Disorder

Polypharmacy (Number of Medications)

Medication classification type

Surgical Procedure type

Emergency Surgery

Anesthesia Type

Fracture

Laboratory test results (Hemoglobin, Creatinine, Sodium, Potassium, White Blood Cell count)

Vital signs (Temperature, Hypoxia)

Red Blood Cell Transfusion

Predictive modeling was performed utilizing least absolute shrinkage and selection operator (LASSO) penalized logistic regression using 10-fold cross-validation. LASSO is a regression analysis method that performs variable selection and regularization. We assumed a priori that contribution of many risk factors may be different for medical versus surgical admissions. Therefore, estimation of regression coefficients was performed separately for medical and surgical patients based on the admission service which will enable the MDP tool to compute the prediction risk depending on the clinical presentation at admission. Regression coefficients for different variables were calculated and an area under the curve plotted for the tool.

Results:

The patients in the total study cohort were well represented based on gender (54.6% male) with an average age of 68. About 94% of the study population was White, and no race or ethnicity groups were excluded. Table 1 includes all the risk factor variables that were extracted from the electronic medical record. Data for each of these variables were represented as percentages (categorical) or value (continuous) of each risk factor retrieved as well as the number missing (Supplemental Table 1). The MDP tool utilizes information available on the first day of the hospital admission to calculate the predicted probability that a given patient will develop delirium during their hospital stay. The MDP tool uses the following variables (Figure 1) : age; history of delirium; dementia; psychiatric disorder; ICU level of care; narcotic analgesic or narcotic antagonist administration; emergency department admission for a surgical procedure as opposed to an elective procedural admit; any red blood cell transfusion, medical comorbidities of congestive heart failure; chronic obstructive pulmonary disease; atrial fibrillation; Charlson score; fall risk score; laboratory test result values of white blood cell count, hemoglobin, serum creatinine and serum sodium; pressure ulcer; and fracture. Odds ratios (OR’s) for all the included categorical variables and continuous variables (for every one unit increase) are represented in Figure 1.

Figure 1:

Odds ratio (OR) estimates for different risk factor variables incorporated into the Mayo Delirium Prediction model ^a

^a Substance use disorder (substance related intoxication/withdrawal) on admission needs to be categorized as high risk for delirium.

The final regression coefficients and OR’s for different risk factor variables included in the MDP tool are summarized (Supplemental Table 2). The area under the ROC curve (AUROC) for MDP tool using derivation cohort (composite of medical and surgical patients) was 0.85 (95% C.I. 0.846 to 0.855). Using the regression coefficients obtained from derivation cohort, predicted probability of delirium was calculated for each patient in validation cohort. For the validation cohort, AUROC was 0.84 (95% C.I. 0.834 to 0.847). A calibration plot of the observed versus expected frequency of delirium for validation cohort is presented (Supplemental Figure).

For both derivation and validation cohorts, patients were classified into one of the three risk groups based on their predicted probability of delirium: Low (≤5%), Moderate (6-29%) and High (≥30%) (Table 2). About 60% of the patients were classified as low risk and 5.5% were classified as high risk based on prediction probability stratification in both cohorts. In the derivation cohort, observed incidence of delirium was 1.7%, 12.8%, and 44.8% (low, moderate, and high risk respectively) which is similar to the incidence rates in validation cohort of 1.9%, 12.7% and 46.3%.

Table 2:

Summary of the Mayo Delirium Prediction tool for the derivation and validation cohorts

	Derivation (N=80000) AUC = 0.85, 95% CI = (0.846, 0.855)			Validation (N=40072b) AUC = 0.84, 95% CI = (0.834, 0.847)
		Delirium			Delirium
Risk groupa	N (% of total)	#	%	N (% of total)	#	%
Low (≤ 5%)	48153 (60.2%)	820	1.7%	24086 (60.1%)	455	1.9%
Medium (6% to 29%)	27240 (34.1%)	3499	12.8%	13794 (34.4%)	1754	12.7%
High (≥ 30%)	4607 (5.6%)	2065	44.8%	2192 (5.5%)	1015	46.3%

^aRisk groups were defined using cut-offs based on the predicted probability of delirium.

^bOf the 40764 patients originally assigned to the validation cohort, 692 (1.7%) were excluded because they had missing information for one or more of the predictor variables required for the MDP model.

Discussion:

Delirium prediction is immensely important because there is strong evidence that multi-factorial delirium prevention strategies, applied early to the high risk patient, can reduce the incidence of delirium by 44% ¹⁸. The aim of this study was to develop a reliable and accurate tool to predict delirium. The above results are reassuring that the MDP tool will enable health care providers to risk stratify patients effectively. The MDP tool derivation and validation cohorts are large and include both medical and surgical patient populations. To our knowledge, this is the largest cohort study for delirium prediction in United States including heterogeneous clinical groups as compared to any other prediction tool developed in United States. The results with retrospective validation cohort are very promising and reflect reliability of the tool as compared to prior delirium prediction tools^{16, 17}. The MDP tool predicts incident risk of delirium in hospitalized adults at admission. This can be built into the electronic medical record as an automated tool to facilitate behind the screen computation with a simplified risk stratification score being delivered to the care provider.

Risk stratification improves the value of delirium prevention efforts in several ways. While arguments exist for global application of basic delirium prevention strategies such as early mobility and adaptive aids for sensory impairment in all older adults, delirium risk stratification could help identify the highest risk group of patients most likely to benefit from more resource-intensive delirium prevention interventions (e.g. cognitive/social/recreational activities). Risk stratification may also improve medical decision-making, when clinicians are weighing the risks vs benefits of seemingly competing priorities, such as the need to promote uninterrupted sleep or vs the need to obtain frequent vital sign measurements, or weighing the potential benefits of intensifying pain management vs the risks of pain medications. Better prediction of delirium will also help both medical and surgical care providers consider the additional comorbid risks of delirium in the context of perioperative evaluation and/or discussions regarding the patient’s overall goals and preferences¹⁶. Awareness of risk will better enable the multidisciplinary care team to anticipate, prevent, identify, and mitigate the consequences of incident delirium¹⁶. Nursing teams are forefront caregivers both in assessing and caring for patients with delirium. An effective and automated prediction tool would facilitate easier identification of high risk patients in whom targeted or more intensive resource allocation is most likely to add value.

Given that the diagnosis of delirium was extracted electronically, based on either physician diagnosis of delirium or encephalopathy or nursing documentation of CAM-ICU screening as part of twice daily assessments, the validation or accuracy assessment of a delirium diagnosis is limited in this retrospective study. The prevalence rate of delirium in the overall cohort for this study is noted to be about 8.1%, which is lower than other published estimates on prevalence ^{19, 20}. This study was not designed to quantify or replicate the frequency of under-diagnosis or under-documentation of delirium. Clinically, missed delirium diagnoses are more likely hypoactive delirium and mild hyperactive delirium.. The MDP tool was validated based on nursing or clinician identification of delirium, and thus should be able to predict and risk stratify clinically significant cases of delirium. Hence, we expect that a high risk delirium prediction by MDP tool should hold good validity and further prospective studies are needed to better analyze the MDP tool risk prediction.

The relatively low prevalence of delirium in this population may also be due to the exclusion of substance use disorder related admissions as we believe that any admission related to intoxication or withdrawal of a substance is high risk for delirium and hence recommend considering substance use disorder related admissions as high risk of delirium irrespective of the MDP tool prediction. Other study limitations include missing data for some variables due to retrospective study design, limited practicality of the tool for bedside administration by the clinician, and uncertainty regarding tool performance in a younger patient population. A prospective validation study is further required to both assess and demonstrate effectiveness of the MDP tool.

Conclusions:

The Mayo Delirium Prediction tool was developed from a large, heterogeneous patient population with good validation results. Further prospective studies, both internally and externally will better delineate the strengths and weakness of this tool.

Abbreviations:

MDP

Mayo Delirium Prediction

ICU

Intensive Care Unit

CAM

Confusion Assessment Method

AUROC

Area Under the Receiver Operating Characteristics Curve

LASSO

Least Absolute Shrinkage and Selection Operator