REcognizing DElirium in geriatric Emergency Medicine: The REDEEM Risk Stratification Score

Abstract

Study objective:

To derive a risk score that uses variables available early during the ED encounter to identify high-risk geriatric patients who may benefit from delirium screening.

Methods:

This was an observational study of older adults age ≥ 75 years who presented to an academic ED and who were screened for delirium during their ED visit. Variable selection from candidate predictors was performed through a LASSO-penalized logistic regression. A risk score was derived from the final prediction model, and predictive accuracy characteristics were calculated with 95% confidence intervals (CIs).

Results:

From the 967 eligible ED visits, delirium was detected in 107 (11.1%). The area under the curve for the REcognizing DElirium in Emergency Medicine (REDEEM) score was 0.901 (95% CI 0.864 to 0.938). The REEDEM risk score included 10 different variables (7 based on triage information and 3 obtained during early history taking) with a score ranging from −3 to 66. Using an optimal cutoff of ≥ 11, we found a sensitivity of 84.1% (90 of 107 ED delirium patients, 95% CI 75.5% to 90.2%) and a specificity of 86.6% (745 of 860 non-ED delirium patients, 95% CI 84.1% to 88.8%). A lower cutoff of ≥ 5 was found to minimize false negatives with an improved sensitivity at 91.6% (98 of 107 ED delirium patients, 95% CI 84.2% to 95.8%).

Conclusion:

A risk stratification score was derived with the potential to augment delirium recognition in geriatric ED patients. This has the potential to assist on delirium targeted screening of high-risk patients in the ED. Validation of REDEEM, however, is needed prior to implementation.

INTRODUCTION

Delirium is an acute brain failure that commonly occurs in older adults presenting to the emergency department (ED).¹ Its diagnosis has been associated with decreased long-term functionality² and increased mortality³. As ED delirium is often hypoactive⁴ and frequently missed⁵, active screening for delirium has been recommended for all older adults by geriatric ED guidelines.⁶ Despite the frequent occurrence of delirium in geriatric patients and its negative consequences, there are several practical challenges precluding the implementation of universal screening of all geriatric patients for delirium in the ED. The increasing number of older adults presenting to the ED and the large amount of competing standardized care processes in the ED (e.g., several mandatory screenings) add a significant burden to providers and to the entire ED workflow. The identification of a subset of high-risk patients would allow for a targeted screening strategy, decreasing low yield or unnecessary screenings. In this context, the development of a delirium risk score that does not significantly increase nurse or physician workload is a priority of the Geriatric Emergency Care Applied Research (GEAR) network.⁷

Prediction models and risk stratification scores are most useful when clinicians fail to efficiently identify a condition through routine care, and when there are serious consequences associated with missing the diagnosis.⁸ Delirium is missed in up to 83% of cases in the ED; and because delirium has numerous prognostic implications,^9,10 there could be substantial benefits from improving detection. A stratification tool that flags patients who are high risk for delirium could augment providers’ ability to recognize delirium while increasing the feasibility of implementation of screening. The development of such system, however, is challenging because delirium is fluctuant and has multiple risk factors.^11,12 In the ED setting recognizing delirium is even more difficult as providers may have limited information about the patient (e.g., unreliable history and/or no caregivers present). While a systematic review identified 28 delirium prediction models in the inpatient setting, none of these models were built for the ED.¹³ Moreover, existing ED-specific risk stratification models had limited diagnostic accuracy and used variables not readily available to clinicians early in the ED course.^4,14,15

In this study, our objective was to develop a risk stratification system that uses variables available early during the ED encounter to identify high-risk patients who should be screened for delirium.

METHODS

This manuscript adheres to the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) guidelines.¹⁶ This study was approved by our Institutional Review Board and only patients who provided research authorization for medical records review were included.

Study design, setting, and participants

This was an observational study of adults aged 75 years or older who presented to an academic quaternary ED in Minnesota with approximately 80,000 patient visits per year. We included all patients who presented over a 14-month period (December 2^nd, 2019, through February 1^st, 2021) and who were screened for delirium during their ED visit. Delirium screening was recommended for all older adults age ≥ 75 years except those deemed not assessable for delirium (e.g., stuporous, or comatose patients). The decision to screen an individual patient, however, took place at the discretion of the bedside nurse.

Delirium measurement (outcome)

The presence of ED delirium was ascertained by ED nurses with the validated sequential 2-step approach: the Delirium Triage Screen (DTS) and the brief Confusion Assessment Method (bCAM).^17,18 The DTS tool has a sensitivity of 98% and can be used as a rule-out screening tool.¹⁷ The bCAM has a sensitivity of 84% and specificity of 96%.¹⁷ Patients had DTS performed first, and if negative, patients were ruled out of having delirium based on the high sensitivity of DTS. If DTS was positive, then bCAM was applied. A patient was considered positive for delirium if they had a positive bCAM because of the high specificity of bCAM. A positive DTS followed by a negative bCAM was considered negative for delirium. Patients without a DTS recorded but with a negative bCAM were considered negative for delirium. Delirium screening was available in the electronic health record (EHR). For patients who had more than one screening during the ED evaluation, if one of the screenings was positive, we classified them in the group of ED delirium.

For patients in which the delirium screening was unclear (e.g., positive DTS followed by an incomplete bCAM, or both DTS and bCAM incomplete), two physicians (one board-certified emergency physician [J.S.] and one physician-scientist [L.O.J.S.]) performed independent individual chart review and assessed for the presence of ED delirium through the chart-based method developed by Inouye and colleagues.¹⁹ This method has had reported sensitivity of 74%, specificity of 83%, and a likelihood ratio for a positive result of 4.4.¹⁹ Disagreements were resolved by a third reviewer (board certified emergency physician [F.B.]).

Risk factors (candidate predictors)

Our main goal was to create a risk stratification score that assists in the prioritization of delirium screening prior to the completion of the ED evaluation; therefore, to identify candidate predictors, we used findings from our previously published systematic review ^11,12 and other variables that that could be rapidly accessed early in the ED course. The following variables were extracted automatically from the EHR: age, sex, marital status, ethnic group, race, residence status (private residence, assisted living, skilled nursing facility, or unknown), means of arrival (ambulance or not), Emergency Severity Index (ESI) triage level,²⁰ initial vital signs (first-listed vitals in the EHR), chief complaint,²¹ nurse-based fall risk assessment,²² and presence of comorbidities such as visual or hearing impairment, history of dementia, history of stroke/TIA, history of previous delirium, history of depressive disorders, history of anxiety disorders, and history of seizure disorders. Comorbidities (prior history) were measured with previously validated lists of International Classification of Disease (ICD) codes (Appendix 1).^23,24 The inclusion of fall risk assessment and their components as potential candidate predictors was based on a recent study for prediction of inpatient delirium.²⁵ The Memorial ED Fall Risk Assessment Tool (MEDFRAT)²² was routinely applied by nurses early in the course of patients’ ED stay, transforming its results to information that could be used as potential predictors of delirium in the ED. Full details on the measurement of each of these variables are available in the Appendix 1. Initial vital signs were categorized based on extreme values. Patients in the lower 10% for oxygen saturation were flagged as having low levels, whereas patients in either the lower 10% or upper 10% for heart rate, respiratory rate, blood pressure, and temperature were flagged as extreme and abnormal.²⁶ Appendix 2 provides the cut-offs used to divide extreme vital signs from the normal range. All these extracted variables were deemed candidate predictors for the final model.

Missing data

Data was complete for all variables except for fall risk score (2.7% missing), temperature (3.4% missing), heart rate (0.7% missing), and respiratory rate (2.8% missing). Missing data for fall risk assessment elements were coded/scored as 0 when an item was not present, while missing data for temperature, heart rate, and respiratory rate were imputed to be the median vitals from the non-missing data.

Variable selection and risk score development

To prevent over-fitting, variable selection from candidate predictors was performed through a least absolute shrinkage and selection operator (LASSO) penalized logistic regression.^27,28 The penalization parameter lambda (λ) was chosen through 10-fold cross-validation on the data and any variable with model estimates not shrunk to 0 were chosen for the final model. Selected variables were weighted by their respective LASSO-penalized regression estimates. These weighted variables were then rounded to a decimal and multiplied by 10 to create a variable risk score, then summed to return the REcognizing DElirium in Emergency Medicine (REDEEM) risk score. The rounding to a decimal instead of an integer was done to allow for more granularity in the range of possible REDEEM scores.

Discrimination (how well the features in the model separated those with from those without ED delirium) was assessed by the area under the receiver operating characteristic curve (AUC), and calibration of our final model was assessed by the Hosmer-Lemeshow goodness-of-fit test.

To divide groups into low and high-risk patients, a risk score cutoff was determined using Youden’s index²⁹ to maximize the sensitivity and specificity of the risk score predictions. Also, we alternatively determined the risk score cut-off that would maximize sensitivity (i.e., favoring the detection of a higher proportion of delirium cases at the cost of more false positives) with the requirement that specificity could not drop below 70%. Accuracy, sensitivity, specificity, as well as positive and negative predictive values were calculated along with their 95% confidence intervals (CIs) using an asymptotic binomial approximation. Positive and negative likelihood ratios were calculated along with 95% CIs using a logarithmic transformation on binomial proportions.

All statistical analyses were performed with R software version 3.6.2 by a statistician. For descriptive statistics, continuous features were summarized as median and interquartile ranges (IQRs) or means and standard deviations (SD) as appropriate based on data distribution. Categorical features were summarized as counts and percentages.

RESULTS

There were 1,060 ED visits in which delirium screening was performed; 93 of these were excluded due to lack of research authorization, leaving a final sample size of 967 by 897 distinct patients. Median age of our cohort was 83 years (IQR 79–88), 54.4% were female, and 97.7% were White. Individual chart review to assess for the presence of delirium was required in 49 visits (5.1%). The inter-rater agreement for delirium assessment in these visits requiring chart review was substantial (kappa 0.80, 95% CI 0.63 to 0.96: overall agreement 89.8%). Overall, ED delirium was detected in 107 (11.1%). (Flowchart available in Appendix 3) Most delirium-positive episodes were hypoactive delirium (N=80, 74.8%). Description of the cohort stratified by delirium screening is detailed in Table 1.

Table 1.

Baseline characteristics of the REDEEM cohort.

	n (%) or median (IQR)
	Screened Negative for ED Delirium (N=860)	Screened Positive for ED Delirium (N=107)	Total (N=967)	Missing Data, N (%)
Age and sex
Age (years)	83 (79, 88)	85 (79, 90)	83 (79, 88)	-
Female	466 (54.2%)	60 (56.1%)	526 (54.4%)	-
Ethnicity
Hispanic, or Latino	6 (0.7%)	2 (1.9%)	8 (0.8%)	-
Not Hispanic or Latino	841 (97.8%)	104 (97.2%)	945 (97.7%)	-
Unknown Ethnicity	13 (1.5%)	1 (0.9%)	14 (1.4%)	-
Race
White	831 (96.6%)	104 (97.2%)	935 (96.7%)	-
African American	8 (0.9%)	1 (0.9%)	9 (0.9%)	-
Asian	8 (0.9%)	0 (0.0%)	8 (0.8%)	-
Other Race	11 (1.3%)	2 (1.9%)	13 (1.3%)	-
Unknown Race	2 (0.2%)	0 (0.0%)	2 (0.2%)	-
Residence status
Private residence	568 (66.0%)	51 (47.7%%)	619 (64.0%)	-
Assisted living	89 (10.3%)	20 (18.7%)	109 (11.3%)	-
Skilled nursing facility	64 (7.4%)	30 (28.0%)	94 (9.7%)	-
Unknown	139 (16.2%)	6 (5.6%)	145 (15.0%)	-
Prior history and comorbidities
History of dementia	226 (26.3%)	59 (55.1%)	285 (29.5%)	-
History of stroke or TIA	206 (23.9%)	37 (34.6%)	243 (25.1%)	-
History of delirium	119 (13.8%)	17 (15.9%)	136 (14.1%)	-
History of depression	233 (27.1%)	41 (38.3%)	274 (28.3%)	-
History of anxiety	247 (28.7%)	38 (35.5%)	285 (29.5%)	-
History of seizures	27 (3.1%)	11 (10.3%)	38 (3.9%)	-
Visual impairment	61 (7.1%)	11 (10.3%)	72 (7.4%)	-
Hearing impairment	112 (13.0%)	18 (16.8%)	130 (13.4%)	-
Fall risk assessment
ED Fall Risk Score	2 (0, 3)	8 (5, 9)	2 (0, 4)	26 (2.7%)
History of falling prior 3 months	313 (36.4%)	50 (46.7%)	363 (37.5%)	21 (2.2%)
Confusion or disorientation	97 (11.3%)	82 (76.6%)	179 (18.5%)	20 (2.1%)
Intoxication or sedation	4 (0.5%)	3 (2.8%)	7 (0.7%)	21 (2.2%)
Impaired gait	409 (47.6%)	78 (72.9%)	487 (50.4%)	21 (2.2%)
Mobility assistance device	444 (51.6%)	74 (69.2%)	518 (53.6%)	21 (2.2%)
Altered elimination	133 (15.5%)	54 (50.5%)	187 (19.3%)	22 (2.3%)
ED visit characteristics
First systolic blood pressure	142 (126, 160)	143 (119, 159)	142 (125,160)	-
First diastolic blood pressure	76 (67, 86.2)	76 (64, 92)	76 (66, 87)	-
First temperature (°F)	98.1 (97.9, 98.4)	98.1 (97.7, 98.6)	98.1 (97.8, 98.4)	33 (3.4%)
First heart rate	77 (67, 89)	78 (68, 94)	78 (67, 90)	7 (0.7%)
First respiratory rate	18 (16, 20)	18 (16, 22)	18 (16, 20)	27 (2.8%)
First oxygen saturation	100% (90%, 100%)	100% (90%, 100%)	100% (90%, 100%)	-
Arrival via EMS†	356 (41.4%)	70 (65.4%)	426 (44.1%)	-
Chief complaint of altered mental status	24 (2.8%)	42 (39.3%)	66 (7.0%)
ESI level 1	2 (0.2%)	1 (0.9%)	3 (0.3%)	-
ESI level 2	137 (15.9%)	31 (29.0%)	168 (17.4%)	-
ESI level 3	654 (76.0%)	75 (70.1%)	729 (75.4%)	-
ESI level 4	66 (7.7%)	0 (0.0%)	66 (6.8%)	-
ESI level 5	1 (0.1%)	0 (0.0%)	1 (0.1%)	-
Delirium Subtype *
Hyperactive	---	27 (25.2%)	---
Hypoactive	---	80 (74.8%)	---

^†Includes both ground and air ambulances.

^*We used the Richmond Agitation Sedation Scale (RASS) recorded along with the delirium screening to define delirium subtype. Patients with an initial RASS score between +1 and +4 were considered to have hyperactive delirium. Patients with a RASS score between 0 and −3 were considered to have hypoactive delirium.

Table 2 provides the list of selected variables along with the LASSO-penalized regression estimates. The strongest predictors were a triage chief complaint of altered mental status and the presence of confusion or disorientation identified during the nurse fall risk assessment. The REDEEM risk score included 10 different variables (7 based on triage information and 3 obtained during early history taking) with a score ranging from −3 to 66. (Table 2) A logistic regression model using this risk score found that a 10-unit increase in risk score was associated with more than 3 times the odds of ED delirium (odds ratio [OR] = 3.11, 95% CI: 2.63 to 3.69, p < .0001), while a 1-unit increase in risk score increased the odds of delirium by 12% (OR = 1.12, 95% CI: 1.10 to 1.14, p < .0001). Figure 1 illustrates the AUC for the REDEEM risk score, which was estimated at 0.901 (95% CI 0.864 to 0.938).

Table 2.

REDEEM and its selected variables from LASSO-penalized logistic regression with model estimates.

Predictor	Model Estimate†	Assigned Scores
Triage information
Arrival via EMS (ambulance)	0.036	+1
Triage chief complaint of altered mental status	1.804	+18
ESI level ≥ 3	−0.266	−3
Low oxygen saturation (< 92%)	0.205	+2
Low systolic blood pressure (< 111 mmHg)	0.235	+2
High diastolic blood pressure (>99 mmHg)	0.114	+1
Respiratory rate
Low respiratory rate (<16 breaths per minute)	0.299	+3
High respiratory rate (> 24 breaths per minute)	0.583	+6
Early history taking
Confusion or disorientation identified during fall risk assessment*	2.464	+25
Altered elimination identified during fall risk assessment*]	0.826	+8
History of seizure disorders	0.436	+4
REDEEM Risk Score: ranges from −3 (lowest) to +66 (highest)

ESI, Emergency Severity Index.

^†These estimates give an idea of relative variable importance within the data. Positive values indicate a positive relationship with ED delirium and negative values indicate a negative relationship. The absolute magnitude of the model estimate indicates the strength of the association and importance of the predictor.

^*These two variables were part of the ED fall risk assessment and can be interpreted as being part of early history taking by ED nurses right after patients were roomed in the ED. Altered elimination is flagged by nurses in the presence of urinary or fecal signs or symptoms.

Figure 1.

Receiver operating characteristics (ROC) curve for the REDEEM model that derives the risk score.

Youden’s index found the optimal cutoff to be 11; patients with a risk score of 11 or greater were classified as high-risk whereas patients with a risk score less than 11 were classified as low risk. Using this cut-off, we found an overall accuracy of 86.3% (835 of 967 patients, 95% CI 84.0 to 88.4%) with a sensitivity of 84.1% (90 of 107 ED delirium patients, 95% CI 75.5% to 90.2%) and a specificity of 86.6% (745 of 860 non-ED delirium patients, 95% CI 84.1% to 88.8%). Table 3 is a two-by-two contingency table and Table 4 summarizes prediction characteristics. A Hosmer-Lemeshow test for the goodness of fit of a logistic model using this risk score to predict ED delirium found that the model was a good fit for the data (p = 0.959).

Table 3.

Two-by-two contingency tables of the REDEEM risk score for the two selected optimal cutoffs.

	ED delirium assessment
Model Predictions	With ED Delirium	Without ED Delirium
Youden’s Index (cutoff ≥ 11)
High Risk (≥ 11)	90	115
Low Risk (< 11)	17	745
Favoring Sensitivity (cutoff ≥ 5)
High Risk (≥ 5)	98	235
Low Risk (< 5)	9	625

Table 4.

REDEEM risk score performance for the two selected optimal cutoffs.

Cut-offs	REDEEM ≥ 11	REDEEM ≥ 5
Accuracy (95% CI)	86.3% (84.0%, 88.4%)	74.8% (71.9%, 77.5%)
Sensitivity (95% CI)	84.1% (75.5%, 90.2%)	91.6% (84.2%, 95.8%)
Specificity (95% CI)	86.6% (84.1%, 88.8%)	72.7% (69.5%, 75.6%)
PPV (95% CI)	43.9% (37.0%, 51.0%)	29.4% (24.7%, 34.7%)
NPV (95% CI)	97.8% (96.4%, 98.7%)	98.6% (97.2%, 99.3%)
LR (+) (95% CI)	6.29 (5.21, 7.60)	3.35 (2.96, 3.79)
LR (−) (95% CI)	0.18 (0.12, 0.28)	0.12 (0.06, 0.22)

PPV, positive predictive value; NPV, negative predictive value; LR (+), likelihood ratio of a positive result; LR (−), likelihood ratio of a negative result.

Because of the important consequences of missing delirium, an alternative cut-off that favors detecting a higher proportion of delirium cases at the cost of more false positives would be 5. If patients with a risk score of 5 or greater were considered high-risk and patients with a score below 5 were considered low risk, the overall accuracy would be lower at 74.8% (723 of 967 patients, 95% CI 71.9% to 77.5%). However, sensitivity would be increased to 91.6% (98 of 107 ED delirium patients, 95% CI 84.2% to 95.8%) with a corresponding specificity of 72.7% (625 of 860 non-delirium patients, 95% CI 69.5% to 75.6%). (Tables 3 and 4) Accuracy characteristics of other alternative cutoffs are available in Appendix 4.

Figures 2 and 3 are a visual representation of these 2 cutoffs in a hypothetical scenario of 1,000 patients. Using a score of ≥ 5 means that among 1,000 older adults presenting to the ED, 338 patients would need to be screened to find 92 delirious patients. A total of 662 patients would be considered low risk and not screened, including 8 patients that would be delirious and therefore potentially missed. (Figure 2) Using a cutoff of ≥ 11 would translate to 205 screenings to find 84 delirious patients while erroneously labeling 16 delirious patients as low risk. (Figure 3)

Figure 2.

Pictogram of a hypothetical scenario of 1,000 older adults in which only those at high-risk (defined as REDEEM score ≥ 5) undergo targeted delirium screening in the ED.

The target sign along with the green and blue dummies represents the patients who would be targeted for screening.

Figure 3.

Pictogram of a hypothetical scenario of 1,000 older adults in which only those at high-risk (defined as REDEEM score ≥ 11) undergo targeted delirium screening in the ED.

The target sign along with the green and blue dummies represents the patients who would be targeted for screening.

LIMITATIONS

First, this was a retrospective study from a single academic ED and our findings may not be replicable at other centers. For example, our cohort was composed mostly of White Non-Hispanic patients, with a lack of racial and ethnic diversity. Second, the prediction model and score require external validation and evaluation of its impact on patient-oriented outcomes prior to adoption into clinical practice. Third, only a small proportion (8.6%) of all adults aged 75 years or older who presented to the ED were screened for delirium during the study period, and the selection is almost certainly not random despite nurses not being aware of the study, and for this reason selection bias cannot be ruled out. Nevertheless, our rate of delirium was similar to prior ED literature (around 10%).³⁰ Moreover, most delirium episodes in our study were hypoactive, which is also consistent with prior ED studies. Fourth, although the 2-step approach has been reported to have good diagnostic performance,¹⁷ its performance in daily practice when used by nurses may be different. However, ED nurses spend considerable time at the bedside and are in optimal position to recognize features of delirium.³¹ Fifth, the chart review method developed by Inouye and colleagues was originally developed to be used in the inpatient setting and its diagnostic accuracy might be different when applied to identifying delirium retrospectively in a particular point in time (i.e., the ED). Sixth, due to the use of data routinely collected for clinical purposes, variables such as history of dementia, for example, may have been underestimated if not included as a diagnosis in the medical history. Lastly, this study did not aim to evaluate independent risk factors for ED delirium but rather to derive the most optimized risk stratification system using variables available early in the ED course. For this reason, the fact that other important risk factors such as history of dementia or history of stroke, for example, are not included in the final model does not mean they are not important but rather means that the selected variables using LASSO-penalized logistic regression provided similar or better information for the prediction of ED delirium in our dataset.

DISCUSSION

We found that approximately one in ten adults 75 years of age or older presenting to the ED will screen positive for delirium. We derived the REDEEM risk score, a risk stratification tool that includes 10 easily obtained variables with relatively good accuracy to predict risk of ED delirium. The score ranges from −3 to 66, and two different cutoff scores can be used to define high-risk patients. All the variables included in this score are structured in the EHR and available early in the ED course. This will facilitate external validation and its potential implementation for prioritization of delirium screening in geriatric ED patients. REDEEM is not intended to “rule in” or “rule out” delirium, but rather it is a risk stratification tool to assist on a targeted screening strategy in the ED. Even patients deemed as very high risk by REDEEM will require a formal delirium assessment to confirm such diagnosis. REDEEM should not be used as a delirium-specific diagnostic or screening tool, and it does not replace the validated 2-step diagnostic approach for delirium (DTS followed by bCAM). Rather, it may be a useful system to risk stratify patients and avoid unnecessary delirium screenings by rapidly identifying high-risk patients. Screening all patients may not be feasible in the ED, but a risk stratification tool like REDEEM could allow us to focus our efforts on those who need the most. Also, the variables are simple enough that the score can be built into the medical record as an automated alert system.

Several prior studies have evaluated risk factors for ED delirium.^11,12 However, none have attempted to create a risk stratification system that could allow for targeted screening through the identification of a high-risk group early in the ED course. Han and colleagues created a 3-point risk score that included the following: history of dementia, Katz activities of daily living (ADL) index, and presence of hearing impairment.⁴ Kennedy and colleagues created a 17-point risk score that included age, history of dementia, history of stroke or TIA, respiratory rate, suspected infection, and ED diagnosis of intracranial hemorrhage.¹⁴ Lastly, Sri-on and colleagues created a prediction model that included history of dementia, hearing impairment, and ED diagnosis of metabolic derangement.¹⁵ Despite having relatively adequate predictive ability (AUCs between 0.77 and 0.82), these prediction models have selected variables that are often not available in the early ED course, making them difficult to stratify patients for targeted screening. An ADL assessment, for example, is rarely available in the ED, and the diagnoses of conditions such as intracranial hemorrhage or metabolic derangement will frequently be available after comprehensive work-up. ED diagnoses are mostly helpful for inpatient providers who are focused on predicting those who may develop delirium during hospitalization. The REDEEM score overcomes many of the limitations in previously derived models by using variables routinely available in the EHR at the beginning of the ED visit. This allows care providers to identify patients who are high risk for delirium while in the ED, to deploy early active targeted screening, and to initiate immediate prevention strategies.

The variables included in REDEEM are aligned with known delirium risk factors. For example, mode of arrival (via ambulance vs other), ESI triage level, and initial vital signs are all correlated with severity of illness, a classic delirium risk factor.^12,32,33 Also, a chief complaint of altered mental status in geriatric patients has previously been identified as a very strong risk factor for delirium during the ED stay.^12,21 History of seizures was also included in our score, and has been associated with a higher risk of ED delirium.¹² Similar to cognitive impairment and history of stroke, prior seizures likely represent brain insult, which increases the probability of delirium. Lastly, two elements of REDEEM came from information obtained through the fall risk assessment that was routinely performed by nurses including confusion/disorientation and altered elimination.^22,34 Documented history of dementia, a well-known strong risk factor for delirium,¹² was not retained in the final model. Its absence in REDEEM is partly explained by the fact that our cohort was exclusive of patients 75 years or older. Derivation of this score in a younger cohort might have retained dementia as a significant predictor. Nevertheless, the variable of “confusion and/or disorientation” captures most if not all patients with significant dementia as one would expect that these patients would inevitably have some sort of confusion and/or disorientation at baseline. For this reason, this variable is not only capturing patients with documented dementia but also other cognitive impairments that have not been previously identified in the medical records. A geriatric patient with confusion or disorientation (independent of how this information is obtained) is at high-risk of being delirious and should therefore be actively screened for delirium

To decipher an optimal cutoff to define the high-risk group (i.e., identify those who would receive targeted screening), we presented 2 options: a cutoff of ≥ 5 or ≥ 11. A cutoff of ≥ 5 has greater sensitivity and minimizes false negatives (patients who would not be screened but in fact are delirious), while a cutoff of ≥ 11 has greater specificity and minimizes false positives (patients who would be screened but in fact are non-delirious). Delirium is very important to diagnose and missing it can have important prognostic implications,¹⁰ so one may think that a lower cutoff is better because of its increased sensitivity. However, in a hypothetical scenario of 1,000 geriatric patients presenting to the ED, the balance between “false negatives” and “false positives” is not as straightforward. In aggregate, for every 1,000 patients, when using the lower cutoff (≥ 5), one would need to perform 133 more screenings to detect 8 more delirious patients. Independent of the approach chosen, targeted screening in general would decrease unnecessary screenings (654 and 778 patients assigned as low risk who would not have delirium in the scenarios of Figures 2 and 3, respectively). It is important to recognize that these figures assumed that delirium screening has excellent diagnostic accuracy, which is not true given limitations of existing diagnostic tools.^7,35

External validation is necessary to determine a prediction model’s reproducibility in different settings.^36,37 Despite the strengths of REDEEM including prior systematic review to inform data collection,¹² use of variables routinely available in the EHR during early ED course, and variable selection for the model using penalization methods, this was a derivation study, and external validation is required prior to its implementation into practice. This is important because there are several examples in the medical literature of models with good predictive ability in their derivation that were not confirmed to be adequate in subsequent validation studies. Further studies should also evaluate if a targeted screening strategy using REDEEM (or other risk stratification system) is feasible in the ED flow and if it improves patient-oriented outcomes, such as incidence of falls, functionality at discharge, and in-hospital mortality, when compared to screening without risk stratification (usual care). In the meantime, clinicians can use the findings of our study to augment their judgment for the recognition of high-risk patients who probably require delirium screening in the ED, and in whom early preventive interventions may be beneficial.