Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2017 Jul 11.
Published in final edited form as: J Patient Saf. 2016 Jan 11:10.1097/PTS.0000000000000242. doi: 10.1097/PTS.0000000000000242

Factors Influencing Time-dependent Quality Indicators for Suspected Acute Coronary Syndrome Patients

Daniel France 1, Scott Levin 2, Ru Ding 2, Robin Hemphill 2, Jin Han 4, Stephan Russ 4, Dominik Aronsky 4, Matt Weinger 1
PMCID: PMC4940339  NIHMSID: NIHMS715147  PMID: 26756723

Abstract

Objectives

Rapid risk stratification and timely treatment are critical to favorable outcomes for acute coronary syndrome (ACS) patients. Our objective was to identify patient and system factors that influence time-dependent quality indicators (QIs) for unstable angina (UA)/non-ST elevation Myocardial Infarction (NSTEMI) patients in the emergency department (ED).

Methods

A retrospective, cohort study was conducted over a 42-month period of all patients aged 24 years or older suspected of having ACS as defined by receiving an electrocardiogram (ECG) and at least one cardiac biomarker test. Cox regression was used to model the effects of patient characteristics, ancillary service utilization, staffing provisions, equipment availability and ED and hospital crowding on ACS QIs.

Results

ED adherence rates to national standards for ECG read out time and biomarker turn-around-time were 42% and 37%, respectively. Cox regression models revealed that chief complaints without chest pain and the timing of stress testing and medication administration were associated with the most significant delays.

Conclusions

Patient and system factors both significantly influenced QI times in this UA/NSTEMI cohort. These results illustrate both the complexity of diagnosing NSTEMI patients and the competing effects of clinical and system factors on patient flow through the ED.

Keywords: Acute Coronary Syndrome, Emergency Department, Delays, Systems Engineering

1.0 INTRODUCTION

1.1 Background

Chest pain prompts over 5.5 million emergency department (ED) visits. There are more than 1 million hospitalizations annually for acute myocardial infarction (AMI), the leading cause of death in the U.S [1]. Acute coronary syndrome (ACS) - an umbrella term that includes all clinical findings consistent with acute myocardial ischemia - is a strong predictor of future AMI. Clinical research has shown that rapid risk stratification and timely treatment are critical to favorable outcomes in ACS patients. However, the majority of this research has focused on ST-Elevated Myocardial Infarction (STEMI). Randomized clinical trials of ACS management during the 1990s yielded numerous diagnostic tools and treatment strategies. Based on these data, the American College of Cardiology (ACC), the American Heart Association (AHA), and others developed clinical practice guidelines for the diagnosis and treatment of ACS using an evidence-based framework for clinical decision making [2, 3].

Results from the “Can Rapid Risk Stratification of Unstable Angina Patients Suppress Adverse Outcomes with Early Implementation of the ACC/AHA Guidelines” (CRUSADE) study demonstrated a significant relationship between adherence to recommended guidelines in the ED and hospital and inhospital mortality. The CRUSADE study did not evaluate the association between the adherence to recommended time-based guidelines, such as time to electrocardiogram, and in-hospital mortality. For every 10% increase in composite guideline adherence score at a hospital, there was a 10% decrease in ACS patients’ likelihood of in-hospital mortality [4]. Despite the evidence favoring the adoption and adherence to these AHA/ACC guidelines, many EDs struggle to meet these criteria.

For many components of cardiac care, the timeliness of care delivery is nearly as important for safety outcomes as the care itself [2, 4–9]. Timely identification of very low risk patients could benefit ED quality by reducing the ED length of stay (LOS) for this population that generates a particularly high volume of clinical workload. The impact of early and accurate triage decisions is significant in light of mounting evidence that ED crowding adversely affects quality of care [10–15]. Quality indicators (QI) that emphasize the temporal dimension of care have been adopted for ED ACS patients.

In this study, we analyzed natural and artificial variability in completion times of diagnostic and therapeutic care events for suspected ACS patients presenting to a large tertiary care center accredited as a Chest Paint Center by the Society of Chest Pain Centers. We hypothesized that time-based ACS QIs are significantly influenced by ED and hospital system factors adjusted for patient factors (i.e., patient disease process and time of ED arrival). To test this hypothesis, we used multivariable regression modeling to assess the independent contributions of patient, ED, and hospital system factors on ED performance on five time-based QIs for UA/NSTEMI: (1) ECG readout time; (2) laboratory turnaround time (TAT); (3) therapeutic turn-around time (TTAT); (4) ED boarding time; and (5) ED LOS.

2.0 METHODS

2.1 Study Design and Methods

This was a retrospective cohort study of all patients presenting to an urban, academic, tertiary care hospital with suspected ACS between February 1, 2006 and August 31, 2009. We included all patients aged 24 years or older on whom an electrocardiogram (ECG) and at least one cardiac biomarker test was performed and documented in the hospital’s adult ED (see Table 1). Twenty-four years of age was selected as the cut-off age because it serves as a demarcation line for Vanderbilt’s adult and children’s hospitals. The 45-bed ED has 55,000 patient visits per year, a 29% admission rate, and is accredited as a Chest Pain Center by the Society of Chest Pain Centers. The study was approved by the Vanderbilt University Institutional Review Board.

Table 1.

Socioeconomic and clinical characteristics of the study cohort and exclusion cohorts

Variable Study Cohort*
Age >=24; ECG and Biomarker (N=12,544)		 Exclusion Cohort with one ACS-related Test† (N=25,564) Exclusion Cohort with No ACS-related Testing†† (N=173,646)
N % N % N %
Age (yr.)
Median(IQR) 56.8(46.0, 70.1) 52.7(40.8, 66.7) 35.1 (24.8, 49.6)
Mean/SD 58.1/16.2 54.0/17.5 39.5/18.5
English speaking 11,264 90% 21,655 85% 150,451 87%
Gender (Female)* 6,040 48% 13,962 55% 96,720 56%
RACE*
White 9,138 73% 17,067 67% 109,509 63%
Black 2,747 22% 7,300 29% 49,888 29%
Others 502 4% 945 4% 12,263 7%
Insurance Type*
Commercial 4,608 37% 7,573 36% 58,354 43%
Medicaid 1,567 12% 3,209 15% 35,346 26%
Medicare 5,119 41% 8,178 39% 19,754 14%
Self Pay 1,236 10% 2,102 10% 23,797 17%
ESI Level*
1 141 1% 297 1% 3,113 2%
2 10,128 81% 17,831 70% 55,240 36%
3–5 2,220 18% 7,318 29% 94,903 62%
Chief Complaint
Chest pain NOS (786.5) 5,202 41% 1,484 6% 19,364 11%
Shortness of Breath (786.05) 1,915 15% 6,490 25% 3,082 2%
Other Malaise & Fatigue (780.79) 931 7% 2,317 9% 6,168 4%
Syncope and Collapse (780.2) 508 4% 2,998 12% 2,792 2%
Abdominal Pain-Site NES (789.09) 404 3% 1,035 4% 872 1%
Musculoskeletal (710–739) 477 4% 725 3% 408 0%
Digestive (520–579) 388 3% 687 3% 1,008 1%
Other Cardiovascular(785) 367 3% 1,303 5% 11,846 7%
Circulatory (390–459) 314 3% 1,296 5% 28,363 16%
Skin (680–709) 310 2% 706 3% 8,556 5%
Injury (800–999) 275 2% 1,431 6% 23,769 14%
Respiratory (460–519) 264 2% 599 2% 11,260 6%
Nervous (320–459) 179 1% 905 4% 7,049 4%
Mental (290–319) 159 1% 832 3% 10,297 6%
General (all other ICD9) 851 7% 2,756 11% 38,812 22%
Admission to hospital 6,021 48% 7,554 30% 29,582 17%
Open in a new tab
*

Less than 1% missing value in study cohort.

†

Patients had either an ECG or cardiac enzyme tests but not both.

††

Patients had neither an ECG nor cardiac enzyme tests.

Time-stamped diagnostic and therapeutic care event data were obtained from the hospital’s clinical information systems to recreate the time course of care for each suspected ACS patient presenting to the ED during the study period. Data were obtained from the following systems and sources: provider computerized order entry (CPOE); ED Information Systems (EDIS); ED Nursing documentation; Laboratory information system (LIS); Admission, Transfer, Discharge (ADT) system; Perioperative Information Management System; the National Cardiovascular Data Registry (NDCR) ACTION Registry – Get With the Guidelines (GWTG); and the NCDR CATH PCI Registry. Patient socioeconomic and clinical data, ED and hospital resource utilization data, and concurrent staffing data were obtained. System queuing metrics, designed to capture system workload and ancillary resource congestion, were calculated from the hospital’s clinical and operational data systems. Patient variable, staffing levels, work process factors and system queuing metrics were used as potential independent variables for the multivariable analyses.

2.2 Outcome Measures

Table II shows the five ED process QI time intervals for UA/MSTEMI used as the dependent variables for all analyses. The ACC/AHA have adopted formal standards for ECG read out time and lab turn-around time. Standards have not been adopted for therapeutic turn-around-time, boarding time, or ED length of stay, however, as of January 2012, the Centers for Medicare and Medicaid Services requires hospitals to report hospital performance on the ED throughput measures for admitted patients. The adoption of formal standards is anticipated as a result of these new requirements.

Table 2.

Time-dependent Quality Indicators for UA/NSTEMI patients (N=12,544)

Quality Indicator Definition Performance Standard Time: Median (IQR) min Percent Adherence
1. ECG read out time Arrival time to ECG read out time ≤ 10 min24–26 13 (7 – 32) 42%
2. Lab turn-around-time Physician biomarker order to lab report time ≤ 60 min3, 27, 28 70 (53 – 96) 37%
3. Therapeutic turn-around-time (overall) Physician biomarker order to first anti- ischemic medication administration Not defined 65 (14 – 66) --
3b. Therapeutic turn-around-time (Anti-platelet) Physician biomarker order to first anti- platelet medication administration Not defined 173 (71 – 317) --
3c. Therapeutic turn-around-time (Anti-thrombin) Physician biomarker order to first anti- thrombin medication administration Not defined 30 (14 – 65) --
4. Boarding time Disposition decision time to inpatient admit time for admitted patients Not defined 198 (97 – 496) --
5. ED length of stay Arrival time to discharge (disposition decision for inpatients) Not defined 247 (156 – 392) --
Open in a new tab

2.3 Primary Data Analysis

Chi-square tests and analysis of variance (ANOVA), respectively, were used to compare categorical and continuous variables characterizing the socioeconomic and clinical characteristics of the study cohort. The median interval time, interquartile range (IQR), and the percentage adherence to defined ACS standards were computed for each QI (Table II). Cox proportional hazard regression modeling was used to test our hypothesis that system factors influence ACS QIs when adjusted for patient factors. For all tests, a 2-tailed probability of less than 5% was used for statistical significance. We modeled the statistical relationships between selected independent variables and care process interval times (i.e., dependent variables) that define the time-based QIs for UA/STEMI patients. For the dependent variables door-to-ECG readout, door-to-biomarker readout, and biomarker order-to-treatment, only the time to first event was used for patients who received serial biomarkers, cardiac enzyme tests, or treatments. Independent variables included in the analysis were measures of: ED, ancillary, and hospital operations, as well as patient characteristics (see Table II) hypothesized to influence the timeliness of cardiac care. ED operational variables measured ED workload, resource demands, and staffing levels. Ancillary operational variables measured the workload, resource demands (e.g., ECG), staffing levels and shift patterns of radiology and the clinical laboratory. Hospital operational variables reflected workload demand, resource availability (e.g., inpatient beds) and staffing levels of cardiology, the catheterization laboratory and the operating rooms. QIs that were the sum of multiple, serially correlated clinical processes were partitioned such that one regression model was created for each process. The assumption that each process time interval was independent of the previous interval (i.e., renewal process) was accounted for by including the previous time interval as a predictor variable.

Predictor variables were selected by first identifying a sub-set of operational, clinical, and patient demographic predictors that were hypothesized to influence each process time interval. A penalized likelihood approach was used to determine which predictor variables to include in the final models [16, 17].

The proportional hazard assumption was tested by examining plots of scaled Schoenfeld residuals and using the proportional hazard test described by Grambsch [18]. None of the predictor variables used in the final models had a non-proportional effect over time. Bootstrapping was used to assess model bias [19, 20].

Applying Cox regression within this framework raises several analytical issues. Our dataset had very little missing data (<1%) so bias from our complete-case analyses was negligible. We also included only one (i.e., the first) visit per patient (<5% had more than one visit) excluding data from subsequent repeat visits to simplify the handling of correlation within a patient over repeat visits. Correlated survival times for patients treated by the same provider during the same shift were expected. If correlated survival times follow a proportional hazards model then treating correlated survival times as independent gives consistent estimates of relative risk parameters, but may yield inconsistent estimates of asymptotic variance [21–23]. These effects were examined by creating clusters around ED provider and shift. The clusters included all cohort patients with the same ED attending (i.e., attending physician responsible for care upon patient arrival) during the same attending shift. Robust variance estimates [21–23] were compared to original variance estimates and were found to be consistent.

3.0 RESULTS

During the 42-month study period, 12,544 (6%) patient visits met the inclusion criteria for suspected ACS. A consort diagram of the study cohort is displayed in Figure 1. The socioeconomic and clinical characteristics of the final cohort are displayed in Table 1. Most patients meeting the inclusion criteria received a chest x-ray (N = 11,354; 91%) in the ED before a treatment or disposition decision was made. Overall, 48% (N=6,021) of cohort visits resulted in admission to the hospital as compared to a 19% rate (N = 37,136) for all other ED patients not meeting inclusion criteria. Of the admitted patients, 2,401 (40%) received anti-ischemic, anti-platelet, and/or anti-thrombin therapy during their ED stay.

Figure 1.

Figure 1

Open in a new tab

Emergency Department ACS Pathway Analyzed for Study Cohort

Table II presents the calculated median interval time, IQR and the percent adherence measured for each ACS QI. Performance standards have been defined for time-to-ECG readout ≤ 10 minutes [24–26] and laboratory TAT ≤ 60 minutes [3, 27, 28]. The ED adhered to these performance standards in less than 50% of visits comprising the cohort which is consistent with previously published adherence rates [29–31]. The median TTAT was much faster for anti-thrombin therapy than anti-platelet therapy. The median ED LOS was approximately 4 hours and the median boarding time was nearly 3.5 hours for patients admitted to the hospital in the cohort.

Cox proportional hazards regression models were created for each defined ACS interval (See Table III for Hazard Ratios (HR) and 95% Confidence Intervals). Predictor variable HRs greater than one are associated with shorter time intervals. HRs less than one are associated with longer time intervals. Table III also shows the influence of the independent variables on the baseline median interval for each QI model.

Table 3.

Cox Regression Models for Critical ACS Intervals

Outcome Variable Hazard Ratio (95% CI) Median Δ in Minutes
ECG Readout Time Baseline = 9.1
Age: >64 Reference -
Age: 24–34 0.89 (0.79, 0.96) 0.8
Age: 35–44 0.96 (0.89, 1.01) 0.3
Age: 45–54 0.94 (0.88, 0.99) 0.4
Age: 55–64 0.96 (0.90, 1.10) 0.3
Gender: Male Reference -
Gender: Female 0.94 (0.90, 0.98) 0.4
Language: English Reference -
Language: non English 1.16 (1.10, 1.22) −0.7
Mode of Arrival: Walk-in Reference
Mode of Arrival: Ambulance 0.94 (0.90, 0.99) 0.4
Mode of Arrival: Missing 1.00 (0.87, 1.12) −1.1
Emergency Severity Index (ESI): Level 3–5 Reference -
ESI Level 1 1.83 (1.70, 2.10) −2.4
ESI Level 2 1.36 (1.31, 1.42) −1.3
Chief Complaint: Cardiovascular (CV) with Chest Pain Reference -
Chief Complaint: CV without chest pain 0.58 (0.53, 0.63) 4.8
Chief Complaint: non CV-related 0.35 (0.30, 0.50) 16.3
Season: Summer Reference -
Season: Winter 0.95 (0.90, 0.99) 0.3
Physician Triage*: Yes compared to No 0.95 (0.90, 0.99) 0.3
Cardiologist in ED*: Yes compared No 1.16 (1.11, 1.20) −0.7
ED ACS/Stroke load: Number of chest pain and neurological patients admitted to ED within one hour of patient admission to ED 0.99 (0.98, 1.00) 0.2
Wait Room Count at Time of Admission 0.99 (0.98, 0.99) 0.4
Biomarker Readout Time Baseline = 66.3
Gender: Female 0.91 (0.87, 0.95) 2.5
Mode of Arrival: Ambulance 1.06 (1.02, 1.10) −1.3
Mode of Arrival: Missing 1.22 (1.11, 1.31) −14.2
Chief Complaint: CV without chest pain 0.89 (0.84, 0.94) 3.0
Chief Complaint: Non- CV-related 0.88 (0.83, 0.91) 3.6
Time of Day: Afternoon Reference -
Time of Day: Evening 0.89 (0.84, 0.94) 3.1
Time of Day: Night 0.84 (0.77, 0.90) 4.8
ED Trauma Load 0.95 (0.91, 0.99) 1.4
Cardiologist in ED*: Yes compared to No 0.74 (0.69, 0.80) 8.0
Physician Triage*: Yes/No 1.12 (0.08, 1.16) −2.5
ED Bed Occupancy (%) at Time of Biomarker Test Ordered 0.99 (0.99, 0.99) 2.3
Point-of-Care Testing: Yes compared to No 1.15 (1.08, 1.22) −3.1
ED ACS Lab Load: Number of biomarker tests sent to lab from ED within one hour of order 0.97 (0.95, 0.98) 0.9
Therapeutic TAT Baseline = 61.7
Chief Complaint: CV without chest pain 0.73 (0.66, 0.80) 35.3
Chief Complaint: Non CV-related 0.61 (0.53, 0.68) 64.3
ED Bed Occupancy (%) at Time of Biomarker Test Ordered 0.99 (0.99, 1.00) 3.8
Medication received before biomarker test 2.51 (2.31, 2.66) −17.4
Stress Test Prior to* Treatment: Yes/No 0.40 (0.20, 0.61) 160.1
Portable Chest X*-Ray prior to treatment: Yes compared to No 1.23 (1.11, 1.36) −30.0
ED Boarding Time Baseline = 218.2
Year Admitted: 2009 Reference -
Year Admitted: 2006 1.07 (0.96, 1.17) −0.7
Year Admitted: 2007 1.18 (1.11, 1.29) −22.7
Year Admitted: 2008 1.03 (0.94, 1.11) 7.3
Gender: Female 0.94 (0.88, 0.99) 13.9
Language: Non English 1.20 (1.11, 1.29) −33.1
Insurance: Private Reference -
Insurance: Medicaid 0.80 (0.71, 0.89) 39.9
Insurance: Medicare 0.88 (0.82, 0.93) 30.3
Insurance: Self Pay 0.84 (0.74, 0.95) 53.8
ESI Level 1 3.78 (3.57, 3.99) −142.1
ESI Level 2 1.15 (1.07, 1.23) −25.9
Day of Week: Weekday Reference -
Day of Week: Weekend 1.37 (1.29, 1.45) 0.0
Time of Day: Evening 0.97 (0.90, 1.03) 6.9
Time of Day: Night 0.82 (0.73, 0.92) 45.4
Holiday: Yes compared to No 1.36 (1.17, 1.56) −51.3
ED LOS: Minutes to Disposition Decision 1.00 (1.00, 1.01) −11.8
Treatment before Disposition Decision 0.75 (0.66, 0.85) 71.4
Cath Lab Open: Yes compared to No 0.87 (0.77, 0.97) 17.8
Telemetry Bed Occupancy (%) 0.31 (0.03, 0.60) 17.9
CVICU Bed Occupancy (%) 0.65 (0.43, 0.88) 6.3
Cardiology consult: Yes compared to No 1.37 (1.13, 1.43) −51.9
Portable Chest X- Ray*prior to disposition Decision 1.18 (1.13, 1.23) −30.2
ED LOS Baseline = 231.3
Year Admitted: 2009 Reference -
Year Admitted: 2006 1.23 (1.11, 1.36) 0.81
Year Admitted: 2007 1.30 (1.22, 1.38) −5.1
Year Admitted: 2008 1.16 (1.12, 1.20) 7.0
Age: 24–34 0.91 (0.82, 0.99) 11.4
Age: 35–44 0.80 (0.73, 0.86) 26.3
Age: 45–54 0.80 (0.74, 0.86) 26.0
Age: 55–64 0.91 (0.86, 0.97) 10.4
Gender: Female 0.95 (0.91, 0.99) 5.6
Mode of Arrival: Ambulance 0.92 (0.87, 0.97) 9.9
Mode of Arrival: Missing 2.12 (2.02, 2.31) −28.0
Chief Complaint: CV without chest pain 1.12 (1.06, 1.17) −10.7
Chief Complaint: Not CV-related 0.87 (0.81, 0.92) 16.9
Time of Day: Evening 0.89 (0.86, 0.96) 13.1
Time of Day: Night 0.94 (0.86, 1.01) 7.4
Cardiologist in ED*: Yes compared to No 0.97 (0.91, 1.03) 3.3
Point-of-Care Testing*: Yes compared to No 0.93 (0.82, 1.02) 9.4
ED ECG Load: Number of ECG performed in ED within one hour of admission 1.02 (1.01, 1.03) −1.7
ED ACS/Neuro Load: Number of chest pain and neurological patients admitted to ED within one hour of patient admission to ED 1.01 (1.00, 1.02) −2.1
ED OCCUPANCY (%) at ED Admission 0.99 (0.99, 1.00) 0.8
Admitted to Hospital: Yes compared to No 1.72 (1.68, 1.77) −50.8
Anti-platelet. Anti- ischemic, or anti- thrombin therapy provided: Yes/No 0.95 (0.90, 0.99) 6.3
Stress Test Prior to* Disposition Decision: Yes compared to No 0.35 (0.28, 0.42) 169.7
Portable Chest X*-Ray before disposition decision 1.16 (1.12, 1.20) −29.0
Open in a new tab

Forty-two percent of ECG TATs met the national guideline of 10 minutes or less. Performance on this QI was primarily influenced by patient factors. Patients who presented to the ED with chief complaints inconsistent with ACS, especially those without chest pain, experienced the longest delays. Conversely, median ECG TATS were 2.4 minutes shorter for patients stratified into the Emergency Severity Index (ESI) Level 1 at triage than for lower acuity patients. Age<65, female gender, and non-English language were all statistically significant in our multivariable model. Trauma activations (i.e., ED Trauma load in our models) did not influence ECG acquisition time. ED waiting room count and arrival by ambulance were associated with very small ECG acquisition delays, while having a dedicated cardiologist in ED was associated with small reductions in ECG TATs.

Cardiac biomarker results were reported within 60 minutes of the physician’s order in 37% of patients in the ACS cohort. The regression model for biomarker TAT indicates that ED performance was influenced by female gender, presenting with chief complaint not related to chest pain or any cardiovascular disease, evening or night arrival, and ED occupancy. The presence of a cardiologist in the ED at the time of admission was associated with an 8 minute increase in median lab TAT, whereas ambulance arrival, physician triage, and point-of-care testing were associated with faster TATs.

The median therapeutic TAT was 65 minutes. More than half of the patients in our cohort received anti-ischemic, anti-thrombin, or anti-platelet therapies before the biomarker test results were available. Patients presenting with non ACS-related chief complaints and/or received a stress test before receiving medication therapy experienced the longest TTAT. Reduced ED physician staffing levels were was also contributed to TTAT delays. The use of portable X-ray machines was associated with the greatest (i.e., 30 minutes) reduction in median TTAT.

Boarding time increased over the study period as ED visits increased. Female gender and insurance type (Self pay, Medicaid, and Medicare) were associated with longer boarding times, while high patient acuity and non-English language was associated with shorter boarding times. Temporal factors also influenced boarding times: weekends and holidays had shorter boarding times whereas evenings or nights had longer boarding times. Receiving medication therapy before disposition decision and increased census in the catheterization laboratory, cardiovascular intensive care unit (CVICU), and cardiac telemetry unit were also independently and significantly associated with increased ED of ACS patients boarding. Access and use of critical resources such portable X-ray machines and cardiology consults shortened ED boarding times.

The median ED LOS of 247 minutes was also influenced by numerous patient, ED, and clinical factors. Patient age (<65 years) and chief complaint (i.e., non cardiovascular, non chest pain-related) were associated with increased ED LOS. Female gender was also associated with increased LOS but the effect was small. Influential ED operational factors included point-of care testing, the use of portable x-rays machines, and the presence of a cardiologist in the ED. ED LOS increased each year of the study as ED volume increased. ED crowding factors significantly influenced ED LOS but hospital crowding factors did not. Factors affecting ED crowding and demand included: ED occupancy; the number of ED beds occupied by suspected ACS and neurology patients; and demand for ECGs. Year and evening or night admissions were the only temporal factors of significance in the model. Patients who received a stress test before disposition decision experienced the longest ED LOS, while patient admitted to the hospital had the shortest LOS.

4.0 DISCUSSION

Systems-based research and quality improvement have primarily focused on decreasing the time to interventions in patients with STEMI. The diagnosis or rule-out of UA/NSTEMI is equally important, especially in this era of ED crowding. With a more subtle presentation and more gradual clinical manifestation, clinical evaluation can be prolonged due to diagnostic uncertainty and complexity. Systems-based improvements are more difficult to design and implement for UA/NSTEMI because the evaluation period of these patients is longer, more resource intensive, and more dependent on intra-hospital factors. The duration of ED evaluation can be problematic because it makes the diagnosis and treatment of UA/NSTEMI patients more susceptible to fluctuations in ED and hospital occupancy, provider handovers, and time-varying demands on ED and hospital resources. This study measured and characterized variability in ACS care processes at an accredited chest pain center and used multivariable regression modeling to isolate the contributory effects of specific patient, ED, and hospital factors.

The results of the Cox regression models show that six classes of factors affected the timeliness of emergency ACS care in our cohort: (1) patient socioeconomic factors; (2) clinical factors; (3) ED and hospital process factors; (4) temporal factors; and (6) crowding and resource demand factors. The following socioeconomic factors significantly influenced each of our QIs: Age<65, female gender, non-English language, and insurance type. Female gender was significantly associated with delay in ECG TAT, biomarker TAT, boarding time, and ED LOS. Non-English language was the only patient factor associated with shorter QI intervals. Insurance type only influenced boarding time in our cohort. The results of previous studies on the effects of patient socioeconomic factors on ECG TATS have been largely inconclusive, although female gender has been consistently associated with delayed ECG TAT [30, 32–34]. By comparison, Diercks’ analysis of data from patients enrolled in the CRUSADE initiative found female gender, nonwhite race, insurance type, and off-hour presentation to be significantly associated with delayed ECG acquisition [30]. Research has shown that patient socioeconomic factors can affect care quality perhaps due to conscious or subconscious biases of healthcare providers [35].

Clinical factors associated with ACS QI intervals included ESI level, chief complaint, disposition decision, and the use of ancillary services. Patients triaged to ESI level I or II experienced significantly shorter QI intervals; the highest acuity patients were appropriately attended first. Yoon found that patients in intermediate triage levels III and IV had the longest wait times for physicians and nurses, and the longest ED LOS [36]. Our finding that ACS patients with non chest pain-related and non cardiovascular-related chief complaints had the longest QI intervals is a logical result since discordant chief complaints can lead clinicians down the wrong diagnostic pathway [37]. From a systems improvement perspective, the most interesting clinical factors were those involving ancillary services. Previous research has shown that diagnostic imaging, laboratory tests, and specialty consultations are associated with longer LOS [36]. Our results suggest that efficient resource management and the creative navigation of constraints related to ancillary services can facilitate timely, but not always optimal, ACS care. For example, patients received the fastest TTAT when physicians began medication therapy before biomarker tests were reported by the laboratory. TTAT, boarding times, and ED LOS were shorter for patients who received a chest x-ray in the ED via a portable device. However, patients receiving stress tests had among the longest QI intervals because the service was not available after-hours. Cardiology consults were associated with a 49 minute reduction in median boarding time in our study compared to a 2.5 hour (150 minute) mean prolongation in Yoon’s study [36]. Increasing access to cardiac stress testing, provided via 24-hour stress testing services, will likely reduce delays in TTAT and improve patient flow in high throughput EDs.

ED and hospital process factors included strategies designed to improve the quality of emergency ACS care. Our study period overlapped with the implementation of three QI interventions: (1) inclusion of a physician on the triage team; (2) point-of-care testing; and (3) placement of cardiologist in the ED. The physician triage system was initiated in January of 2006 but its hours of operation were modified substantially four times between January 2006 and July 2007. The effect of physician triage in the Cox models was likely muted since the majority of patients during our study period were evaluated under this system. Previously published research performed in our ED showed physician triage reduced ED LOS by 37 minutes [38]. The new STEMI response system (i.e., “One Call system”) uses a single call to the hospital’s LifeFlight team (i.e., air rescue) to activate the STEMI team and the catheterization laboratory. Since the One Call system emphasized urgency in obtaining an ECG in less than 10 minutes for suspected ACS patients, it has benefited all patients even those ruled out for STEMI.

Our ED implemented a variation of true point-of-care testing that uses a dedicated pneumatic tube system to deliver specimens to an ED-dedicated (i.e., 24/7/365) laboratory technician in the central laboratory. Time-of-day effects, which were significant in our model, persisted under the new point-of-care testing model. A recent multi-center cost-benefit study of true point-of-care testing in the ED found that its use increased the proportion of patients successfully discharged home and reduced the median hospital LOS but did not reduce costs [39]. These results contradict recent projected benefits of point of care testing in the ED [40]. Further research will be required to determine if our variant of point-of-care testing is clinically effective and cost-effective.

The presence of an in residence ED cardiologist was not associated with a systematic improvement UA/NSTEMI QI intervals. However, conclusions regarding this intervention must be made cautiously because the dichotomous variable used in our models represented whether or not a cardiologist was present at the time of patient admission and not whether the cardiologist personally evaluated that patient. Future research should evaluate how the cardiologist’s direct interaction with suspected ACS patients affects care quality.

Factors characterizing ED and hospital crowding were significantly associated with delays in ACS care. ED crowding and/or increasing demands on critical resources prolonged all QI intervals. Our finding that ED occupancy contributed to biomarker TAT delays is consistent with, although less pronounced, than the results of Hwang and colleagues [41–43] but contradict those of Hulstad et al. or Pines et al. [41–43]. Similar to McCarthy et al. [44], we found hospital crowding, rather than ED crowding, was associated with prolonged boarding times. Instead, high occupancy in the catheterization lab, telemetry unit, and CVICU delayed ACS care processes, consistent with previously published research by our team [45]. Increasing occupancy in the catheterization laboratory is a major driver of ED boarding for cardiac patients. However, neither expanding the catheterization lab from 16 to 19 beds in 2008 nor extending its hours of operation to include Saturday morning influenced any of our ACS QI intervals. Interestingly, concurrent (i.e., with ED admissions) cardiac surgery volume was not associated with QI intervals. This may reflect limitations of our models’ variables to measure accurately variation over time. For example, our ‘cardiac surgery load’ variable was simply a count (i.e., a snapshot) of cardiac surgical procedures occurring within a 1 hour period of the ACS care event being analyzed. In a previously published study, we demonstrated that variability in elective surgery schedules had more significant adverse effects on ED boarding times than sources of natural or random variability [46].

While individual patient visits to the ED are naturally (i.e., random) occurring, ED arrival patterns at the population level are not. In fact, ED visit patterns are fairly predictable by time of day, day of week, and season of year [46]. Nationally, ED occupancy typically increases from early afternoon until night, and it spikes on Friday and Saturday night and annually during influenza season [47]. Consistent with national findings, year and time of day were the most significant temporal factors in our ACS QI models. Year was a proxy for ED volume which increased steadily over the study period as is occurring nationwide. As ED volume increased QI intervals lengthened. Evening and night time ED admissions were associated with delays in ECG acquisition and biomarker TAT, and longer boarding time. Night time admission increased median boarding time by 45 minutes. Conversely, holiday admissions decreased median boarding by 51 minutes. Winter was associated with slower lab processing of cardiac biomarker tests. Knowledge of temporal patterns of ED utilization should be used to inform quality improvement strategies. For example, Bucheli added one additional physician during nights shifts to reduce mean ED LOS by approximately 35 minutes [48].

4.1 Limitations

This study has limitations. Since the data are from a single academic medical center, the results may not be applicable to other hospitals and chest pain centers. Secondly, the statistical models were developed from retrospective data collected from clinical information systems. The reliability of these data can be highly variable, especially when electronic documentation is influenced by many of the very factors we analyzed, including staffing levels and crowding. However, prior to our study we conducted over 200 hours of direct observations of ED physician and nurse work practices and workflow [49–51]. Through this work we were able to accurately measure the variability in clinical information system variables and used this information to guide subsequent data usage. The third limitation was that we limited our analysis to initial care events. The diagnosis of UA/NSTEMI often requires extended clinical evaluation and serial diagnostic tests so by ignoring the iterative processes of diagnosing complex ACS cases we missed an opportunity to explore the effects of clinical uncertainty on ED response. This study did not include physiological data (i.e., vitals sign data) which are certain to influence care processes (but are naturally occurring variables). Finally, we were unable to acquire reliable pre-hospital (i.e., EMS data) that would have potentially provided important insights on pre-hospital delays and clinical practices (i.e., ECG, administration of aspirin, etc.). Therefore, patients who received an ECG by EMS were excluded from our analysis if the test was not repeated and documented in the ED.

5.0 CONCLUSIONS

Wait time distributions associated with ACS QIs were significantly influenced by both artificial variability in ED and hospital work processes and by natural variability in ED patient arrivals and clinical factors. Clinical improvement strategies designed to enhance the management and coordination of resources between the ED and hospital showed the most potential for improving ACS care delivery.

The analysis and modeling of temporal patterns of ED utilization and process flow can make emergency departments more adaptive to time-varying demands on clinical resources. Systems modeling provides a valuable methodology for understanding clinical uncertainty and the factors affecting care quality. Good models facilitate evidence-based decisions about quality improvement interventions.

Acknowledgments

Source of Funding: This research was supported by National Heart, Lung, and Blood Institute Grant R21 HL091322 (Drs. France and Levin).

We thank the National Heart, Lung, and Blood Institute for supporting this research through Grant R21 HL091322 (PIs: Drs. France and Levin). We thank Brittany Cunningham, Quality Consultant, at the Vanderbilt Heart and Vascular Institute for her assistance with ACS registry data. We thank, RuAnn Schleicher, Manager of Laboratory Computer Services, for her assistance with obtaining and processing cardiac biomarker data. We thank Marilyn (Buffy) Key, Director of Clinical Laboratory Operations, for her explanation of laboratory processes as related to ACS. We thank Marie Hasselblad, Administrative Director of Invasive Cardiology and Director of Cardiothoracic Operating Rooms, for her explanations of clinical and operation processes associated with the catheterization laboratory and cardiac surgery.

Footnotes

Conflicts of interest: The authors have no conflicts of interests to report.

References

  • 1.Bhuiya FA, Pitts SR, McCaig LF. Emergency Department Visits for Chest Pain and Abdominal Pain: Unites States, 1999–2008. Vol. 43. Hyattsville, MD: DHHS; Sep, 10 A.D. pp. 1–8. [PubMed] [Google Scholar]
  • 2.Krumholz HM, Anderson JL, Brooks NH, et al. ACC/AHA clinical performance measures for adults with ST-elevation and non-ST-elevation myocardial infarction: a report of the American College of Cardiology/American Heart Association Task Force on Performance Measures (Writing Committee to Develop Performance Measures on ST-Elevation and Non-ST-Elevation Myocardial Infarction) Circulation. 2006 Feb;113(5):732–761. doi: 10.1161/CIRCULATIONAHA.106.172860. [DOI] [PubMed] [Google Scholar]
  • 3.Anderson JL, Adams CD, Antman EM, et al. ACC/AHA 2007 guidelines for the management of patients with unstable angina and non-ST-segment elevation myocardial infarction. A report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines (Committee on the Management of Patients With Unstable Angina) J Am Coll Cardiol. 2007 Aug;50(7):e1–e157. doi: 10.1016/j.jacc.2007.02.013. [DOI] [PubMed] [Google Scholar]
  • 4.Peterson ED, Roe MT, Mulgund J, et al. Association between hospital process performance and outcomes among patients with acute coronary syndromes. JAMA. 2006 Apr;295(16):1912–1920. doi: 10.1001/jama.295.16.1912. [DOI] [PubMed] [Google Scholar]
  • 5.Bradley EH, Roumanis SA, Radford MJ, et al. Achieving door-to-balloon times that meet quality guidelines: how do successful hospitals do it? J Am Coll Cardiol. 2005 Oct;46(7):1236–1241. doi: 10.1016/j.jacc.2005.07.009. [DOI] [PubMed] [Google Scholar]
  • 6.Bradley EH, Herrin J, Wang Y, et al. Strategies for Reducing the Door-to-Balloon Time in Acute Myocardial Infarction. N Engl J Med. 2006 Nov; doi: 10.1056/NEJMsa063117. [DOI] [PubMed] [Google Scholar]
  • 7.Bradley EH, Herrin J, Elbel B, et al. Hospital quality for acute myocardial infarction: correlation among process measures and relationship with short-term mortality. JAMA. 2006 Jul;296(1):72–78. doi: 10.1001/jama.296.1.72. [DOI] [PubMed] [Google Scholar]
  • 8.Bradley EH, Herrin J, Wang Y, et al. Door-to-drug and door-to-balloon times: where can we improve? Time to reperfusion therapy in patients with ST-segment elevation myocardial infarction (STEMI) Am Heart J. 2006 Jun;151(6):1281–1287. doi: 10.1016/j.ahj.2005.07.015. [DOI] [PubMed] [Google Scholar]
  • 9.Bradley EH, Curry LA, Webster TR, et al. Achieving rapid door-to-balloon times: how top hospitals improve complex clinical systems. Circulation. 2006 Feb;113(8):1079–1085. doi: 10.1161/CIRCULATIONAHA.105.590133. [DOI] [PubMed] [Google Scholar]
  • 10.Hwang U, Radford MJ, Krumholz HM. The association between emergency department crowding and time to antibiotic adminstration. Ann Emerg Med. 2004;44:S6–S7. [Google Scholar]
  • 11.Hwang U, Richardson LD, Sonuyi TO, et al. The effect of emergency department crowding on the management of pain in older adults with hip fracture. J Am Geriatr Soc. 2006 Feb;54(2):270–275. doi: 10.1111/j.1532-5415.2005.00587.x. [DOI] [PubMed] [Google Scholar]
  • 12.Pines JM, Hollander JE, Localio AR, et al. The association between emergency department crowding and hospital performance on antibiotic timing for pneumonia and percutaneous intervention for myocardial infarction. Acad Emerg Med. 2006 Aug;13(8):873–878. doi: 10.1197/j.aem.2006.03.568. [DOI] [PubMed] [Google Scholar]
  • 13.Schull MJ, Morrison LJ, Vermeulen M, et al. Emergency department overcrowding and ambulance transport delays for patients with chest pain. CMAJ. 2003 Feb;168(3):277–283. [PMC free article] [PubMed] [Google Scholar]
  • 14.Schull MJ, Vermeulen M, Slaughter G, et al. Emergency department crowding and thrombolysis delays in acute myocardial infarction. Ann Emerg Med. 2004 Dec;44(6):577–585. doi: 10.1016/j.annemergmed.2004.05.004. [DOI] [PubMed] [Google Scholar]
  • 15.Sprivulis PC, Da Silva JA, Jacobs IG, et al. The association between hospital overcrowding and mortality among patients admitted via Western Australian emergency departments. Med J Aust. 2006 Mar;184(5):208–212. doi: 10.5694/j.1326-5377.2006.tb00416.x. [DOI] [PubMed] [Google Scholar]
  • 16.Fan J, Runze L. Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties. J Am Stat Assoc. 2001;96(456):1348–1360. [Google Scholar]
  • 17.Fan J, Runze L. Variable Selection for Cox’s Proportional Hazards Model and Frailty Model. Ann Stat. 2002;30(1):74–99. [Google Scholar]
  • 18.Grambsch P, Therneau T. Proportional hazard tests in diagnostics based on weighted residuals. Biometrika. 1994;81:515–526. [Google Scholar]
  • 19.Efron B, Tibshirani R. Bootstrap methods for standard errors, confidence intervals, and other measures of statistical accuracy. Stat Sci. 1986;1(1):54–77. [Google Scholar]
  • 20.Harrell FE, Jr, Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med. 1996 Feb;15(4):361–387. doi: 10.1002/(SICI)1097-0258(19960229)15:4<361::AID-SIM168>3.0.CO;2-4. [DOI] [PubMed] [Google Scholar]
  • 21.Wei L, Lin D, Weissfeld L. Regression analysis of multivariate incomplete failure time data by modeling marginal distributions. Journal of the American Statistical Association. 1989;84:1065–1073. [Google Scholar]
  • 22.Lee E, Wei W, Amato D. Cox-type regression analysis for large numbers of small groups of correlated failure time observations. In: Klein J, Goel P, editors. Survival Analysis: State of the Art. Boston: Kluwer Academic Publishers; 1992. pp. 237–247. [Google Scholar]
  • 23.Lipsitz S, Parzen M. A Jackknife Estimator of Variance for Cox Regression for Correlated Survival Data. Biometrics. 1996;52:291–298. [PubMed] [Google Scholar]
  • 24.Diercks DB, Kirk JD, Lindsell CJ, et al. Door-to-ECG time in patients with chest pain presenting to the ED. Am J Emerg Med. 2006 Jan;24(1):1–7. doi: 10.1016/j.ajem.2005.05.016. [DOI] [PubMed] [Google Scholar]
  • 25.National Heart Attack Alert Program Coordinating Committee. Emergency Department: Rapid Identification and Treatment of Patients With Acute Myocardial Infarction. Annals of Emergency Medicine. 1994 Feb;23(2):311–329. [PubMed] [Google Scholar]
  • 26.Sagarin MJ, Cannon CP, Cermignani MS, et al. Delay in thrombolysis administration: causes of extended door-to-drug times and the asymptote effect. J Emerg Med. 1998 Jul;16(4):557–565. doi: 10.1016/s0736-4679(98)00054-7. [DOI] [PubMed] [Google Scholar]
  • 27.Bassand JP, Hamm CW, Ardissino D, et al. Guidelines for the diagnosis and treatment of non-ST-segment elevation acute coronary syndromes. Eur Heart J. 2007 Jul;28(13):1598–1660. doi: 10.1093/eurheartj/ehm161. [DOI] [PubMed] [Google Scholar]
  • 28.Morrow DA, Cannon CP, Jesse RL, et al. National Academy of Clinical Biochemistry Laboratory Medicine Practice Guidelines: clinical characteristics and utilization of biochemical markers in acute coronary syndromes. Clin Chem. 2007 Apr;53(4):552–574. doi: 10.1373/clinchem.2006.084194. [DOI] [PubMed] [Google Scholar]
  • 29.Diercks DB, Kirk JD, Lindsell CJ, et al. Door-to-ECG time in patients with chest pain presenting to the ED. Am J Emerg Med. 2006 Jan;24(1):1–7. doi: 10.1016/j.ajem.2005.05.016. [DOI] [PubMed] [Google Scholar]
  • 30.Diercks DB, Peacock WF, Hiestand BC, et al. Frequency and consequences of recording an electrocardiogram >10 minutes after arrival in an emergency room in non-ST-segment elevation acute coronary syndromes (from the CRUSADE Initiative) Am J Cardiol. 2006 Feb;97(4):437–442. doi: 10.1016/j.amjcard.2005.09.073. [DOI] [PubMed] [Google Scholar]
  • 31.Pines JM, Hollander JE. Association between cardiovascular complications and ED crowding. American College of Emergency Physicians 2007 Scientific Symposium; Oct. 11 A.D; Oct 8, 2007. [Google Scholar]
  • 32.Ding R, McCarthy ML, Desmond JS, et al. Characterizing waiting room time, treatment time, and boarding time in the emergency department using quantile regression. Acad Emerg Med. 2010 Aug;17(8):813–823. doi: 10.1111/j.1553-2712.2010.00812.x. [DOI] [PubMed] [Google Scholar]
  • 33.Takakuwa KM, Shofer FS, Hollander JE. The influence of race and gender on time to initial electrocardiogram for patients with chest pain. Acad Emerg Med. 2006 Aug;13(8):867–872. doi: 10.1197/j.aem.2006.03.566. [DOI] [PubMed] [Google Scholar]
  • 34.Yates RB, Hiestand BC. Effects of age, race, and sex on door-to-electrocardiogram time in emergency department non-ST elevation acute coronary syndrome patients. J Emerg Med. 2011 Feb;40(2):123–127. doi: 10.1016/j.jemermed.2008.01.024. [DOI] [PubMed] [Google Scholar]
  • 35.Institute of Medicine. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: National Academy Press; 2001. [PubMed] [Google Scholar]
  • 36.Yoon P, Steiner I, Reinhardt G. Analysis of factors influencing length of stay in the emergency department. CJEM. 2003 May;5(3):155–161. doi: 10.1017/s1481803500006539. [DOI] [PubMed] [Google Scholar]
  • 37.Swap CJ, Nagurney JT. Value and limitations of chest pain history in the evaluation of patients with suspected acute coronary syndromes. JAMA. 2005 Nov;294(20):2623–2629. doi: 10.1001/jama.294.20.2623. [DOI] [PubMed] [Google Scholar]
  • 38.Russ S, Jones I, Aronsky D, et al. Placing physician orders at triage: the effect on length of stay. Ann Emerg Med. 2010 Jul;56(1):27–33. doi: 10.1016/j.annemergmed.2010.02.006. [DOI] [PubMed] [Google Scholar]
  • 39.Fitzgerald P, Goodacre SW, Cross E, et al. Cost-effectiveness of point-of-care biomarker assessment for suspected myocardial infarction: the randomized assessment of treatment using panel Assay of cardiac markers (RATPAC) trial. Acad Emerg Med. 2011 May;18(5):488–495. doi: 10.1111/j.1553-2712.2011.01068.x. [DOI] [PubMed] [Google Scholar]
  • 40.Birkhahn RH, Haines E, Wen W, et al. Estimating the clinical impact of bringing a multimarker cardiac panel to the bedside in the ED. Am J Emerg Med. 2011 Mar;29(3):304–308. doi: 10.1016/j.ajem.2009.12.007. [DOI] [PubMed] [Google Scholar]
  • 41.Hwang U, Baumlin K, Berman J, et al. Emergency department patient volume and troponin laboratory turnaround time. Acad Emerg Med. 2010 May;17(5):501–507. doi: 10.1111/j.1553-2712.2010.00738.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 42.Kulstad EB, Kelley KM. Overcrowding is associated with delays in percutaneous coronary intervention for acute myocardial infarction. Int J Emerg Med. 2009;2(3):149–154. doi: 10.1007/s12245-009-0107-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 43.Pines JM, Pollack CV, Jr, Diercks DB, et al. The association between emergency department crowding and adverse cardiovascular outcomes in patients with chest pain. Acad Emerg Med. 2009 Jul;16(7):617–625. doi: 10.1111/j.1553-2712.2009.00456.x. [DOI] [PubMed] [Google Scholar]
  • 44.McCarthy ML, Zeger SL, Ding R, et al. Crowding delays treatment and lengthens emergency department length of stay, even among high-acuity patients. Ann Emerg Med. 2009 Oct;54(4):492–503. doi: 10.1016/j.annemergmed.2009.03.006. [DOI] [PubMed] [Google Scholar]
  • 45.Levin SR, Dittus R, Aronsky D, et al. Optimizing cardiology capacity to reduce emergency department boarding: A systems engineering approach. American Heart Journal. 2008 Dec;156(6):1202–1209. doi: 10.1016/j.ahj.2008.07.007. [DOI] [PubMed] [Google Scholar]
  • 46.Levin SR, Dittis RS, Aronsky D, et al. Evaluating the effects of increasing surgical volume on emergency department patient access. BMJ Qual Saf. 2011;20:146–152. doi: 10.1136/bmjqs.2008.030007. [DOI] [PubMed] [Google Scholar]
  • 47.Hoot NR, Aronsky D. Systematic review of emergency department crowding: causes, effects, and solutions. Ann Emerg Med. 2008 Aug;52(2):126–136. doi: 10.1016/j.annemergmed.2008.03.014. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 48.Bucheli B, Martina B. Reduced length of stay in medical emergency department patients: a prospective controlled study on emergency physician staffing. Eur J Emerg Med. 2004 Feb;11(1):29–34. doi: 10.1097/00063110-200402000-00006. [DOI] [PubMed] [Google Scholar]
  • 49.Dale K, Debuze R, Levin S, et al. Impact of Emergency Department Occupancy and Patient Boarding on Registered Nurse Work Patterns and Subjective Ratings of Workload and Quality. ACEP Research Forum. 2007;50(3):143. [Google Scholar]
  • 50.Levin S, France DJ, Hemphill R, et al. Tracking workload in the emergency department. Hum Factors. 2006;48(3):526–539. doi: 10.1518/001872006778606903. [DOI] [PubMed] [Google Scholar]
  • 51.Levin S, Aronsky D, Hemphill R, et al. Shifting toward balance: Measuring the distribution of workload among emergency physician teams. Annals of Emergency Medicine. 2007 Oct;50(4):419–423. doi: 10.1016/j.annemergmed.2007.04.007. [DOI] [PubMed] [Google Scholar]

RESOURCES