Measuring and Forecasting Emergency Department Crowding in Real Time

Abstract

Objective:

To quantify the potential for monitoring current and near-future emergency department (ED) crowding using four measures: the Emergency Department Work Index (EDWIN), the National Emergency Department Overcrowding Scale (NEDOCS), the Demand Value of the Real-time Emergency Analysis of Demand Indicators (READI), and the Work Score.

Methods:

We calculated the four measures at 10-minute intervals during an eight-week study period (6/21/2006 – 8/16/2006). Ambulance diversion status was the outcome variable for overcrowding, and occupancy level was the performance baseline measure. We evaluated discriminatory power for current overcrowding by the area under the receiver operating characteristic curve (AUC). To assess forecasting power, we applied activity monitoring operating characteristic (AMOC) curves, which measure the timeliness of early warnings at varying false alarm rates.

Results:

We recorded 7,948 observations during the study period. The ED was on ambulance diversion during 30% of the observations. The AUC was 0.81 for the EDWIN, 0.88 for the NEDOCS, 0.65 for the READI Demand Value, 0.90 for the Work Score, and 0.90 for occupancy level. In the AMOC analysis, only the occupancy level provided more than an hour of advance warning (median 1 hour, 7 minutes) prior to overcrowding with one false alarm per week.

Conclusion:

The EDWIN, the NEDOCS, and the Work Score monitor current ED crowding with high discriminatory power, although none of them exceeded the performance of occupancy level across the range of operating points. None of the measures provided substantial advance warning prior to overcrowding at low rates of false alarms.

Introduction

Background

Emergency department (ED) crowding is recognized to be a major, international concern that affects both patients and providers [1–10]. A recent report from the Institute of Medicine noted that the increasing strain caused by crowding is creating a deficit in quality of emergency care [11]. Crowding has been associated with reduced access to emergency medical services [12–15], delays in care for cardiac patients [16–18], increased patient mortality [19–22], extended patient transport time [23–24], inadequate pain management [25], violence of angry patients against staff [26], increased costs of patient care [27], and decreased physician job satisfaction [28].

Importance

As suggested by the principle “you can’t manage what you can’t measure,” the lack of a universal metric for ED crowding impedes efforts to alleviate the problem [29–30]. In an effort to address this, mathematical formulas have been proposed in the peer-reviewed literature to quantify crowding: the Emergency Department Work Index (EDWIN), the National Emergency Department Overcrowding Scale (NEDOCS), the Demand Value of the Real-time Emergency Analysis of Demand Indicators (READI), and the Work Score [31–35]. These four measures use simple operational variables to assess the present state of crowding in an ED.

There have been mixed reports in the literature about the usefulness of these measures to assess ED crowding [31–41]. Prior validation efforts have often used subjective assessments of crowding by physicians and nurses as the dependent variable [31–32,34,37,39,41]. The measures were intended for continuous monitoring of ED operations [31–35]; however, only the Work Score has been integrated with a clinical information system [35]. Furthermore, the measures have the potential to serve as an early warning system for overcrowding [31,35]. This capability, however, has not yet been established for any of the measures.

Goals of this Investigation

The objective of this study was to assess the usefulness of the EDWIN, the NEDOCS, the READI Demand Value, and the Work Score as monitoring instruments of ED crowding. To achieve this goal, we addressed three related questions. First, is it feasible to evaluate the measures in real time? Second, how accurately do the measures reflect present crowding? Finally, can the measures reliably forecast the future state of crowding?

Methods

Study Design

This was a prospective validation of four ED crowding measures during an eight-week period (6/21/2006 – 8/16/2006). The study did not involve any direct patient contact, and the local Institutional Review Board approved the study by expedited review.

Setting

The validation took place in the adult ED of a tertiary-care, academic medical center with a Level 1 trauma service. The adult ED provides care for more than 45,000 patients annually. It contains 41 licensed beds, four of which are trauma beds. In addition, four fast-track beds are available for low-acuity patients from 11 AM to 11 PM, and eight dedicated rooms are available for psychiatric patients. The ED staff were kept unaware of the study to avoid a potential source of bias. The validation site was independent of the development site for all measures considered.

Methods of Measurement

The EDWIN, the NEDOCS, the READI Demand Value, and the Work Score were calculated to assess the degree of crowding [31–35]. All four of these measures output a continuous variable, where a higher value denotes a greater degree of crowding.

The EDWIN [31] was calculated using the following formula:

$$\text{EDWIN}=\sum {\text{n}}_{\text{i}}{\text{t}}_{\text{i}}/\left({\text{N}}_{\text{a}}\ast \left({\text{B}}_{\text{t}}-{\text{P}}_{\text{board}}\right)\right)$$

where n_i = number of non-boarding patients in triage category i; t_i = reversed triage category i, where 5 denotes the sickest patients and 1 denotes the least sick patients; N_a = number of attending physicians on duty; B_t = number of licensed treatment beds in the ED; and P_board = number of boarding patients.

The NEDOCS [32] was calculated using the following formula:

$$\text{NEDOCS}=\left({\text{P}}_{\text{bed}}/{\text{B}}_{\text{t}}\right)\ast 85.8+\left({\text{P}}_{\text{admit}}/{\text{B}}_{\text{h}}\right)\ast 600+{\text{W}}_{\text{time}}\ast 5.64+{\text{A}}_{\text{time}}\ast 0.93+{\text{R}}_{\text{n}}\ast 13.4-20$$

where P_bed = number of patients in licensed beds and overflow locations, such as hallway beds or chairs, B_t = number of licensed treatment beds, P_admit = number of admitted patients, B_h = number of hospital beds, W_time = waiting time for the last patient put into bed, A_time = longest time since registration among boarding patients, and R_n = number of respirators in use, maximum of two. The respirator variable (R_n) did not generalize to the study setting, because patients ill enough to require mechanical ventilation are stabilized and transferred immediately to a critical care unit. As a surrogate, the number of trauma beds was used in place of the number of respirators.

The Demand Value of the READI score [33–34] was calculated using the following formulas:

$$\text{DV}=\left(\text{BR}+\text{PR}\right)\ast \text{AR}$$

$$\text{BR}=\left({\text{P}}_{\text{total}}+{\text{A}}_{\text{pred}}-{\text{D}}_{\text{pred}}\right)/{\text{B}}_{\text{t}}$$

$$\text{AR}=\sum {\text{n}}_{\text{i}}{\text{t}}_{\text{i}}/{\text{P}}_{\text{triage}}$$

$$\text{PR}={\text{A}}_{\text{hour}}/\sum \text{PPH}$$

where DV = Demand Value, BR = Bed Ratio, AR = Acuity Ratio, PR = Provider Ratio, P_total = number of ED patients, A_pred = number of predicted arrivals, D_pred = number of predicted departures, B_t = number of licensed treatment beds, n_i = number of patients in triage category i, t_i = reversed triage category i, P_triage = number of patients in the ED with an assigned triage category, A_hour = number of arrivals in the past hour, and PPH = average patients seen per hour for each attending physician and resident on duty. The predicted number of arrivals (A_pred) and departures (D_pred) for each hour of the day was calculated using nine months of ED data (9/1/2005 – 6/1/2006). The original READI instrument used a four-level triage system, so the five-level Emergency Severity Index (ESI) was condensed into four categories by combining the two least severe acuity levels [42]. The number of patients seen per hour was calculated for residents at each level of training and for attending physicians who saw patients without a resident, using nine months of ED data (9/1/2005 – 6/1/2006).

The Work Score [35] was calculated using the following formula:

$$\text{Work}\text{Score}=3.23\ast {\text{P}}_{\text{wait}}/{\text{B}}_{\text{t}}+0.097\ast \sum {\text{n}}_{\text{i}}{\text{t}}_{\text{i}}/{\text{N}}_{\text{n}}+10.92\ast {\text{P}}_{\text{board}}/{\text{B}}_{\text{t}}$$

where P_wait = number of waiting patients, B_t = number of licensed treatment beds, n_i = number of patients under evaluation in triage category i, t_i = triage category i, N_n = number of nurses on duty, and P_board = number of boarding patients.

The ED occupancy level was used as a control measure for baseline comparison. The occupancy level was calculated using the following formula:

$$\text{Occupancy}\text{level}=100\ast {\text{P}}_{\text{bed}}/{\text{B}}_{\text{t}}$$

where P_bed = number of patients in licensed beds and overflow locations, such as hallway beds or chairs; and B_t = number of licensed treatment beds.

Under extreme operating conditions, the original published formulas for the EDWIN and the Acuity Ratio of the READI score could generate mathematical errors. If the number of boarding patients in the ED matched or exceeded the number of licensed treatment beds, the denominator of the EDWIN would become zero or negative. If there were no patients in the ED with an assigned triage category, the denominator of the Acuity Ratio would become zero. However, these conditions have never been approached in the study setting, so no changes to compensate for this were deemed necessary for the present study.

Data Collection and Processing

To enable real-time monitoring of ED operations, a computer program was developed using Matlab (version 7.1, http://www.mathworks.com) and integrated with the ED information systems. At 10-minute intervals, the program queried the information systems for the data required to evaluate the four crowding measures and the occupancy level. The resulting values were recorded in a research database.

Outcome Measure

Ambulance diversion status was used as the outcome measure for overcrowding. Policy at our hospital allows for ambulance diversion when any of the following criteria apply and are not expected to be remedied within one hour: “1) all critical care beds in the ED are occupied, patients are occupying hallway spaces, and at least 10 patients are waiting; 2) an acuity level exists that places additional patients at risk; or 3) all monitored beds within the ED are full.” A committee reviews the appropriateness of all diversion episodes on a monthly basis. The hospital’s aeromedical service, which is responsible for maintaining diversion records, provided log files for the study period.

Primary Data Analysis

The ability of each crowding measure to discriminate current ambulance diversion status was analyzed using receiver operating characteristic (ROC) curves [43]. An ROC curve plots sensitivity against (1 - specificity) for all possible thresholds in a binary classification task. The area under an ROC curve (AUC) represents the overall discriminatory ability of a test, where a value of 1.0 denotes perfect ability and a value of 0.5 denotes no ability. To reduce the effect of serial correlation on ROC curve estimation, each measure series was down-sampled to an observation frequency of three hours. The AUC of each measure was calculated with 95% confidence intervals (CI). Pairwise tests for significant differences of AUC were conducted between each measure and occupancy level [44]. An alpha level of 0.05 / 4 = 0.0125, with the Bonferroni correction for multiple pairwise comparisons, was used for the tests of significance. All ROC analyses were performed using the ROCKIT software package (version 0.9.1, http://xray.bsd.uchicago.edu/krl/roc_soft.htm). The operating characteristics of each measure were calculated by fixing each measure’s threshold to achieve 90% sensitivity with respect to ambulance diversion status. At this fixed threshold, each measure’s specificity, predictive values, and likelihood ratios were calculated.

The ability of each crowding measure to forecast ambulance diversion status in the near future was analyzed following the Centers for Disease Control framework for evaluating biosurveillance systems [45]. Activity monitoring operating characteristic (AMOC) curves were developed to characterize the performance of early warning systems [46], and they have been previously applied to the problem of disease outbreak detection [47,48]. An AMOC curve plots timeliness scores against false alarm rates for all possible thresholds in an early warning system. The false alarm rate is generally normalized per unit time – in the present study, per week. The timeliness score may be interpreted here as the median warning time given prior to diversion, within a maximum specified time window. The time window was defined to be four hours for this study, and alarms were classified as 1) true alarms, if they occurred less than four hours before the start of a diversion episode; 2) false alarms, if they occurred more than four hours before the start of a diversion episode; or 3) redundant alarms, if they occurred during a diversion episode. Redundant alarms were not further considered, because they affect neither the timeliness nor the false alarm rate.

The standard method of generating AMOC curves would treat all false alarms as independent events, even when they occurred at consecutive 10-minute intervals [46]. From an ED operational perspective, we considered it more appropriate to treat consecutive alarms as a single, sustained alarm, since the only the first alarm would trigger an intervention. Thus, the AMOC framework was extended for the present study as follows. Each measure series was de-noised using cubic spline smoothing with the Matlab function csaps. A smoothing parameter of 0.99 was applied, where a value of 1.0 represents no smoothing and values below 0.95 resulted in excessive smoothing. Each sequence of consecutive alarms was counted as a single, sustained signal. However, when a trough in the smoothed signal occurred during a sustained false alarm, it was considered to be the beginning of a new false alarm, thus ensuring a monotonic relationship between the false alarm rate and timeliness. All AMOC analyses were performed using Matlab (version 7.1, http://www.mathworks.com).

The timeliness of the four crowding measures and occupancy level were compared by fixing the threshold such that each measure triggered one, two, and three false alarms per week, which was considered the maximum number likely to be tolerated by ED personnel. False alarm rates of were examined for each measure. The timeliness prior to every diversion episode was calculated, and a paired Wilcoxon rank-sum test was used to compare the median difference in timeliness between each measure and occupancy level. The Bonferroni-corrected 95% confidence intervals, equivalent to unadjusted 98.75% confidence intervals, were calculated using R (version 2.3.1, http://www.r-project.org).

Results

Characteristics of Study Period

During the study period, a total of 7,948 10-minute intervals were observed out of 8,064 possible (98.6%). Two incidents of computer system downtime accounted for all of the missed observations. Descriptive statistics for ED operational variables during the study period are listed in table 1. A total of 37 ambulance diversion episodes occurred during the study period, lasting an average of 11.7 hours per episode. There were no episodes of citywide diversion, such that the ED was forced to end its diversion, during the study period. The ED was on ambulance diversion during 30% of the intervals observed. To illustrate the response of each measure to ED overcrowding, figure 1 shows an eight-week time series plot of each crowding measure, superimposed on episodes of ambulance diversion.

Table 1.

ED operational variables (6/21/2006 – 8/16/2006)

Characteristic	No Diversion (n = 5,599)	Diversion (n = 2,349)
Patient Factors
Registrations in last hour (#)	6 (3 - 8)	6 (4 - 9)
Discharges in last hour (#)	5 (3 - 8)	7 (5 - 9)
Mean acuity level (ESI)	2.57 ± 0.16	2.57 ± 0.12
Occupancy level (%)	78 (61 - 88)	96 (91 - 100)
Average length of stay (h)	5.4 (3.9 - 8.3)	8.0 (6.3 - 9.6)
Waiting patients (#)	1 (0 - 4)	11 (5 - 16)
Average waiting time (min)	11 (0 - 31)	84 (52 - 115)
Boarding patients (#)	9 (4 - 15)	20 (15 - 23)
Average boarding time (h)	5.7 (2.5 - 10.2)	10.4 (7.0 - 12.6)
Provider Factors
Attendings on duty (#)	3.0 ± 0.9	3.5 ± 0.7
Residents on duty (#)	4.4 ± 0.5	4.7 ± 0.5
Nurses on duty (#)	13.5 ± 1.8	14.5 ± 1.5
Hospital Factors
Medical-surgical diversion	15%	26%
Critical care diversion	4%	13%

Observations were made at 10-minute intervals during the study period. Descriptions are presented as percentages for discrete variables, mean ± SD for normally distributed continuous variables, and median (IQR) for skewed variables.

Figure 1.

Time series plots of the crowding measures during the study period. The plots shown here are smoothed using cubic splines. Episodes of ambulance diversion are marked by the shaded areas.

Main Results

The ROC curves for the EDWIN, the NEDOCS, the READI Demand Value, the Work Score, and occupancy level are shown in figure 2. The AUC was 0.81 for the EDWIN (95% CI: 0.77-0.85), 0.88 for the NEDOCS (95% CI: 0.85-0.91), 0.65 for the READI Demand Value (95% CI: 0.60-0.71), 0.90 for the Work Score (95% CI: 0.86-0.92), and 0.90 for occupancy level (95% CI: 0.87-0.93). Pairwise tests for differences of AUC showed that occupancy level had greater discriminatory power for overcrowding than the EDWIN (p < 0.001) and the READI Demand Value (p < 0.001), while the NEDOCS and the Work Score did not differ significantly in discriminatory power from occupancy level (p = 0.190 and p = 0.769, respectively). The operating characteristics for each measure at a fixed sensitivity level of 90% are shown in table 2.

Figure 2.

Receiver operating characteristic curves of the crowding measures. The AUC of each measure is shown in parentheses.

Table 2.

Operating characteristics at fixed 90% sensitivity

	Spec	PPV	NPV	LR+	LR−
EDWIN	63%	50%	94%	2.42	0.15
NEDOCS	67%	53%	94%	2.75	0.15
READI Demand Value	32%	35%	88%	1.32	0.32
Work Score	71%	56%	94%	3.09	0.14
Occupancy level	70%	56%	95%	3.05	0.13

Spec = specificity; PPV = positive predictive value; NPV = negative predictive value; LR+ = positive likelihood ratio; LR- = negative likelihood ratio

The AMOC curves for the EDWIN, the NEDOCS, the READI Demand Value, the Work Score, and occupancy level are shown in figure 3. Only the occupancy level provided more than an hour of advance warning (median 1 hour, 7 minutes) prior to overcrowding at a rate of one false alarm per week. Note that the vertical distance between curves in figure 3 illustrates the difference between medians of timeliness; however, with non-parametric paired data, the median difference shown in table 3 may provide more reliable comparisons. As assessed by confidence intervals that do not overlap zero, the occupancy level gave more timely warnings of overcrowding than the EDWIN at rates of one, two, and three false alarms per week. When the false alarm rate was fixed at three per week, the READI Demand Value gave more timely warnings of overcrowding than occupancy level. All other pairwise comparisons of median timeliness to occupancy level were not statistically significant.

Figure 3.

Activity monitoring operating characteristic curves of the crowding measures. A higher value of timeliness denotes a greater amount of warning time prior to episodes of ambulance diversion.

Table 3.

Median difference in timeliness between crowding measures and occupancy level

	False alarm rate
	1 per week	2 per week	3 per week
EDWIN	−1:37 (−2:48, −0:09)	−2:06 (−3:20, −0:28)	−2:01 (−2:53, −0:35)
NEDOCS	−1:04 (−2:24, 0:19)	−1:06 (−2:26, 0:27)	−0:24 (−1:56, 0:50)
READI Demand Value	0:04 (−1:16, 1:14)	0:43 (−0:50, 1:42)	1:16 (0:00, 2:05)
Work Score	−1:17 (−2:37, 0:13)	−0:20 (−1:49, 1:20)	0:02 (−1:32, 1:39)

Differences in timeliness are presented as hours:minutes. A positive value indicates that the measure gave more timely warnings than occupancy level. Lower and upper bounds of the Bonferroni-corrected 95% CI for the median difference are shown in parentheses.

Limitations

A potential limitation of our study is the use of ambulance diversion status as a surrogate for overcrowding. While a clear, universal definition for ED overcrowding does not exist, an expert panel considered ambulance diversion status to be a practical, operational definition [49]. It has been used previously as an dependent variable to validate crowding measures [35–36,40]. The justifiability of using ambulance diversion status as an objective surrogate for overcrowding depends on the rigor of diversion policy at a given institution. As described previously, our institution has specified criteria by which ambulance diversion may be initiated, and regular reviews are conducted to ensure compliance. On these grounds, ambulance diversion status was considered to be the best available reference standard for overcrowding in this study.

A second limitation arises from the fact that the four crowding measures – the EDWIN, the NEDOCS, the READI Demand Value, and the Work Score – were originally developed for the purpose of measuring the present state of crowding [31–35]. The creators of the EDWIN and the Work Score discussed the potential use of the measures to forecast near-future overcrowding, without directly exploring this application [31,35]. As the creators of the NEDOCS and the READI did not explicitly describe this possibility, we acknowledge that validating these measures as early warning systems by AMOC analysis may have exceeded the authors’ intentions [32–34].

Lastly, the study was conducted at a single academic institution, and further research will be required to determine the generalizability of the findings to other emergency department settings. However, as this study represents an independent, prospective validation of all four crowding measures, some notion of their generalizability may be inferred from the findings.

Discussion

The findings demonstrate that the EDWIN, the NEDOCS, the READI Demand Value, and the Work Score may be evaluated in real time by integration with ED information systems [31–35]. Implementing the four measures as monitoring instruments requires the electronic availability of common ED operational variables, such as waiting room count, length of stay, and number of boarding patients.

We examined the ability of the four measures to reflect current ED crowding. The ROC curves and operating characteristics demonstrate that the EDWIN, the NEDOCS, and the Work Score all have high discriminatory power for predicting current ambulance diversion status. However, none of the measures performed better than the control measure, occupancy level. The READI Demand Value showed lower discriminatory power, which is consistent with an earlier report that found no significant association between the READI Demand Value and staff assessments of crowding [34].

We also examined the ability of the four crowding measures to forecast near-future ED overcrowding. Based on the AMOC curves and the timeliness at fixed false alarm rates, all measures had difficulty providing much advance notice at low rates of false alarms. None of the available crowding measures clearly exceeded the control measure, occupancy level, in forecasting performance. Although the READI Demand Value showed poor discriminatory power, it performed much better in the AMOC analysis. The time series plots in figure 1 suggest that, while the other measures tend to peak in the middle of diversion episodes, the READI Demand Value appears to peak close to the beginning of diversion episodes, lending credence to its timeliness.

Two points should be noted from the analysis of forecasting power. First, it is insufficient to consider only operating characteristics such as sensitivity, specificity, and discriminatory power when validating an early warning system. Good performance in terms of discriminatory power does not imply timely forecasts, and vice versa. The Centers for Disease Control recommended a careful analysis of timeliness when evaluating public health monitoring systems [45]. Second, the READI Demand Value is the only measure evaluated that predicts near-future operational changes based on historical data. The other three measures and occupancy level are all point estimates based on current operating status. It is plausible that the use of historical data to predict near-future patient arrivals and departures explains why the READI Demand Value fares relatively well in forecasting ED crowding.

Occupancy level was intended as a simple baseline measure for comparison in the present study. It was interesting to find that none of the four crowding measures clearly exceeded its performance across the range of operating points. This finding is near in spirit to Occam’s razor – roughly paraphrased, one should use the most parsimonious model possible that achieves the intended purpose, since more complex models may be prone to over-fitting.

Future efforts to validate ED crowding measures should focus on using objective endpoints to define crowding. Although not all institutions allow for ambulance diversion, researchers at any ED could define a rule involving the occupancy level, waiting room count, or other basic variables as the reference standard. The use of subjective assessment as the sole dependent variable when validating a crowding measure should be treated cautiously. For example, conflicting reports have been published regarding the utility of the NEDOCS to measure crowding, which may illustrate the difficulty of replicating findings based on a subjective dependent variable [32,37,39].

Future research should also focus on improving the forecasting power of crowding measures. The use of historical data to predict changes in the next few hours may allow for substantial improvements in the performance of an early warning system. Advanced modeling techniques such as neural networks, applied specifically for the purpose of forecasting, may result in improved forecasting power [40]. The development of a good forecasting model for ED crowding will pave the way to studying intervention policies, which may allow researchers to identify ways of sustaining health care quality and access in the face of overcrowding [50]. Other researchers have discussed strategies including the use of reserve physicians and nurses [51] and deferring care of low-acuity patients [52–53], either of which could be initiated given a few hours of advance warning prior to overcrowding.

In summary, the findings demonstrate the feasibility of implementing four measures for real-time monitoring of ED crowding. Occupancy level showed discriminatory power similar to or greater than the four other measures for measuring current ED crowding. In terms of timely forecasting, none of the measures showed a clear advantage over occupancy level. These findings suggest new directions for the measurement and management of ED overcrowding.