Measuring Emergency Department Acuity

Abstract

Background

Emergency department (ED) acuity is the general level of patient illness, urgency for clinical intervention, and intensity of resource use in an ED environment. The relative strength of commonly used measures of ED acuity is not well understood.

Methods

We performed a retrospective cross-sectional analysis of ED-level data to evaluate the relative strength of association between commonly used proxy measures with a full spectrum measure of ED acuity. Common measures included the percentage of patients with Emergency Severity Index (ESI) scores of 1 or 2, case mix index (CMI), academic status, annual ED volume, inpatient admission rate, percentage of Medicare patients, and patients-seen-per-attending-hour. Our reference standard for acuity is the percentage of high acuity charts (PHAC) coded and billed according to the Centers for Medicare and Medicaid Service’s Ambulatory Payment Classification (APC) system. High acuity charts included those APC 4, 5 or critical care. PHAC was represented as a fractional response variable. We examined the strength of associations between common acuity measures and PHAC using Spearman’s rank correlation coefficients (r_s) and regression models including a quasi-binomial generalized linear model and linear regression.

Results

In our univariate analysis, the percentage of patients ESI 1 or 2, CMI, academic status and annual ED volume had statistically significant associations with PHAC. None explained more than 16% of PHAC variation. For regression models including all common acuity measures, academic status was the only variable significantly associated with PHAC.

Conclusion

ESI had the strongest association with PHAC followed by CMI and annual ED volume. Academic status captures variability outside of that explained by ESI, CMI, annual ED volume, percentage Medicare patients, or patients-per-attending-per-hour. All measures combined only explained only 42.6% of PHAC variation.

INTRODUCTION

Emergency department (ED) acuity may be defined as the general level of patient illness, urgency for clinical intervention, and the intensity of resource use within an ED’s clinical care environment. In this context it is a characteristic encompassing the intensity of medical need and services delivered. Validated measures to compare ED acuity, as a cross-institutional measure, are limited despite it being an important metric to guide clinical resource allocation.

Here we explore acuity as a quality of the ED care environment for which comparisons can be made across facilities with 2 specific phenomena in mind. First, is the growing number of metrics making performance comparisons across a diversity of facilities all described as EDs. The function and performance of an ED may be different based on whether it is a relatively lower or higher acuity facility. Being able to identify these differences relies on comprehensively assessing acuity. EDs also range from 5,000 visit per year critical access centers to 250,000 visits per year facilities with care sub-units. Measures need to go beyond size and volume differences to include the diversity of patient care needs, resources available and applied to provide care, and the urgency with which ED care is needed or delivered (i.e. – the balance of primary/urgent care to emergency acute care requiring early involvement of specialist services such as stroke, emergency abdominal surgery, ST-segment elevation myocardial infarction, and impending labor). Second, is the drive to adopt practice innovations (e.g. - split flow, team triage, physician-in-triage, ED physician cath lab activation, chest pain protocols) from one ED to another. Doing so relies on being able to assess the transferability of a practice or policy between EDs. One important consideration is whether the intervention’s performance was tested in an ED of similar overall patient illness, urgent need for clinical intervention, and intensity of resources used for acute care. As a result, it is critical to understand the reliability of measures used to differentiate EDs by their acuity.

Common proxies for ED acuity include: the aggregated Emergency Severity Index (ESI) score, case mix index (CMI),^1,2 annual ED volume,^3,4,5 academic status,⁶ and inpatient admission rate (the percentage of ED patients admitted to the hospital with the intensity of care meeting “inpatient” as opposed to intense “observation” level admissions).^6,7 In addition, many national benchmarking organizations within emergency medicine use percentage of higher risk ED populations and resource-to-patient ratios. Examples include the percentage of Medicare patients and patients-per-attending-hour.^8,9,10

Despite the frequent use of these metrics, each has significant limitations in capturing the level of acuity for the full spectrum of an ED’s clinical environment. In this study we assess how commonly used proxies are related to the full spectrum of ED acuity as measured by the percentage of high acuity charts (PHAC). To our knowledge, a study of this kind has not been done in emergency medicine, because obtaining these measures for cross-institutional comparison is challenging. The ED Operations Study Group (EDOSG) is a non-profit research consortium dedicated to advancing research informing evidence-based clinical operations. The organization’s research network has the ability to obtain these measures for a sample of EDs, while representing the diversity of clinical practice in the United States (US). Despite this innovation and opportunity to explore ED acuity, we acknowledge there is no true reference standard. As a result, our analyses should be considered more suggestive than conclusive.

METHODS

Study Design

This was a retrospective cross-sectional study using ED-level EDOSG data collected via survey as reported by each member ED. The objective was to determine the strength of association between common proxies for ED acuity with a rationalized reference standard, PHAC. Independent variables included seven established proxies for ED acuity including the percentage of patients with an Emergency Severity Index (ESI) score of 1 or 2, CMI,^1,2 annual ED volume,^3,4,5 academic status,⁶ inpatient admission rate (the percentage of ED patients admitted to the hospital with the intensity of anticipated care designated as “inpatient” as opposed to “observation”), the percentage of Medicare patients, and patients-seen-per-attending-hour per day.

Study Population

With the ED as the unit of analysis, our study population included 62 facility members of the EDOSG summarizing the care delivered during 3,323,071 ED visits in 2014. The group represents the diversity of US EDs in terms of US geographic regions, infrastructure, patient acuity, patient volume, academic intensity, and clinical staffing differences. (Table 1)

Table 1.

Characteristics of Emergency Departments with and without Complete Data¹

	N	Complete Data N=27	Incomplete Data N=27	Full Study Sample N=54	P-value
Percentage high acuity charts	54	0.54 0.66 0.73 (0.65±0.15)	0.46 0.64 0.71 (0.58±0.21)	0.52 0.64 0.72 (0.61±0.19)	0.363
Percentage of ESI2 1 or 2	41	0.08 0.18 0.25 (0.17±0.11)	0.05 0.10 0.15 (0.12±0.089)	0.07 0.13 0.25 (0.16±0.11)	0.153
Case Mix Index	40	1.40 1.68 1.80 (1.63±0.28)	1.45 1.56 1.80 (1.71±0.49)	1.40 1.64 1.80 (1.65±0.35)	0.993
Annual ED Volume (per 10,000)	54	3.7 5.3 6.6 (5.6±2.6)	2.1 5.2 7.0 (5.6±4.6)	3.5 5.3 6.8 (5.6±3.7)	0.583
Academic Status	54				0.204
Non-academic		19% 5/27	41% 11/27	30% 16/54
AMC affiliated5		44% 12/27	33% 9/27	39% 21/54
Primary AMC ED		37% 10/27	26% 7/27	31% 17/54
Percentage Medicare Patients	45	20.4 23.7 28.0 (23.1±7.9)	22.8 28.0 30.0 (27.2±7.7)	20.9 26.1 29.0 (25.1±8.0)	0.093
Inpatient Admission Rate	53	15 18 24 (23±17)	13 19 23 (19±10)	14 18 24 (22±15)	0.893
Patients Per Attending hr/day	54	2.15 2.40 2.90 (2.62±0.78)	1.92 2.50 3.18 (2.45±0.83)	2.10 2.50 3.0 (2.54±0.8)	0.713
Region					0.144
Midwest		7% 2/27	26% 7/27	17% 9/54
Northeast		56% 15/27	41% 11/27	48% 24/54
South		30% 8/27	33% 9/27	31% 17/54
West		7% 2/27	0% 0/27	4% 2/54

a b c represent the lower quartile a, the median b, and the upper quartile c for continuous variables x ± s represents X ± 1 SD.

¹Complete Cases were those EDs without missing data. Yes: indicates no missing data. No: indicated any missing data.

² ESI is the emergency severity index. Tests used:

³Wilcoxon Rank Sum test;

⁴Pearson Chi Square test.

⁵AMC is an academic medical center.

Data Collection

We asked each site to provide demographic information for their facility, percentage of patients with ESI 1 or 2, CMI^1,2 annual ED volume,^3,4,5 academic status,⁶ inpatient admission rate,^6,7 percent of patient admitted under observation status, percentage of Medicare patients, patients-seen-per-attending-hour per day, and the aggregated percentage of billed charts falling into each ambulatory payment classification (APC) coding level category.^8,9,10 Each site PI agreed to accurate data collection and reporting to the EDOSG. Most frequently site-PIs are medical or clinical operations directors. The group, however, includes directors of research, nurse administrators, and research faculty with clinical operations interests. All were individuals with roles permitting data access. The data for this study were collected simultaneously with the EDOSG’s 2015 ED metric collection reflecting 2014 clinical care.

Primary Outcome

Our primary outcome was the PHAC coded and billed within each ED according to the Center for Medicare and Medicaid Service’s APC system as our measure of the acuity within each EDs care environment. Most ED visits are captured in a chart documenting patient symptoms, clinician observations, and medical care provided. The majority are then coded using the APC system. APC coding is a federal scaling system developed to guide Medicare and Medicaid reimbursements according to a hierarchy of patient illness and resources applied for ambulatory care.⁹ It consists of a scale of 1–5 where 5 represents a high level of care intensity and patient acuity.¹¹ Two levels of critical care can replace a level 5 chart if intensive-care unit level services are provided. Higher coding levels result in higher reimbursements for physician evaluation and management professional fees. Charts scored as levels 4, 5, or with associated critical care represent the highest acuity patients. Other payers have since adopted this system for reimbursement.

The US Office of the Inspector General (OIG) monitors fraud and abuse of this classification system, and is empowered to conduct audits when inappropriate billing is suspected. Periodic analyses of national trends are performed to monitor coding shift. Penalties can be applied to the institution for verified misconduct.¹¹ The APC system was not designed to measure ED acuity. However, the ability of the percentage of high acuity charts to retrospectively account for resource intensity and patient level of illness, and the wide spread use of the standardized and federally monitored APC system, makes it our rationalized reference standard.

We asked each ED to report the percent distribution of their coded and billed charts for 6 ranked categories: levels 1 through 5, and critical care. We then calculated the percentage of high patient acuity charts – levels 4, 5 and critical care – to obtain the PHAC for each ED. EDs with missing PHAC were excluded from the analysis. All unbilled patient charts were excluded, yet represented less than 5% of all ED visits.

Statistical Analysis

We reported data as aggregated counts, means, and percentages for measures of resource utilization, process, and performance. We used descriptive statistics to report summaries as percentages and medians with interquartile ranges (IQR). This is the standard for measuring clinical performance within emergency medicine.¹⁴ However, we simultaneously calculated means with one standard deviation (SD) to identify skewness that would otherwise be masked.

We evaluated associations between common proxies of acuity and PHAC in three ways: rank-based analyses, graphical assessment and regression analyses. First, we examined the strength of association between PHAC and common acuity measures using Spearman’s rho rank correlation coefficients for continuous acuity measures and the Kruskal-Wallis rank sum test for categorical acuity measures. Those rank-based methods avoid distributional assumptions and thus are less influenced by outliers. Second, we used scatterplots with loess smoothed lines or box plots to visualize the relationship of each common acuity measure with PHAC. Lastly, we conducted regression analysis to relate traditional acuity measures separately and combined with PHAC. PHAC ranges between 0 and 1 and represents a proportional quantity represented as a fractional response varable. In addition, we were concerned relationships with PHAC may not be linear. As a result we used a quasi-binomial generalized linear regression models (GLM) with logit link as our primary regression approach. We then repeated the univariate and multivariate analyses using a linear model to compare and assess whether assuming linearity in the measures relationship with PHAC was reasonable. Within both models, the univariate analyses quantified the relationship between each individual common acuity measure and PHAC and the multivariate analyses tested the degree to which the common measures could jointly predict PHAC.

Multiple imputation was used to handle missing common acuity measures to prevent a statistical power reduction and avoid potentially biased results from only including EDs with complete cases.¹² Specifically, missing data for independent variables were imputed 10 times using additive imputation models allowing for nonlinear transformations on the data to make optimal use of partial information recorded for each ED. We then summarize the regression results from all 10 multiple imputations.

Traditional likelihood based goodness of fit statistics such as the Akaike information criterion (AIC) do not apply to quasi-likelihood based GLM. Therefore, R-squared (R²) was used as a goodness-of-fit statistic for model comparisons. For models assessing combined effects of all seven acuity measures (full model), we tested a series of nested models and compared each with the full model using R² to avoid over-fit. The nested models excluded 1 or 2 variables explaining the least variance of PHAC. We observed minimal differences in R², the strength of associations or significance of remaining variables, so the full model is reported. We evaluated linearity in the GLM using residual plots based on standardized deviance residuals plotted against PHAC on the logit scale. We calculated variance inflation factors for each independent variable to ensure there is no collinearity problem among the common acuity measures.

We set significance a-priori at p<0.05, and present results as odds ratios (OR) with 95% confidence intervals (CI). Statistical analyses were performed by biostatisticians in R, version 3.2.3 (R Foundation for Statistical Computing, Vienna, Austria, 2015, https://www.R-project.org).

RESULTS

From the rank-based analyses, we found notable associations with PHAC for the following 5 common measures: percentage of patients with ESI 1 or 2, CMI, academic status, annual ED volume, and inpatient admission rate. We did not find statistically significant associations with the percentage of Medicare patients or patients-seen-per-attending-per-hour (Table 2).

Table 2.

Rank-Based Analyses: Strength of Association with the Proportion of High Acuity Charts in an Emergency Department¹, ²

(Spearman Correlation)
	Correlation with PHAC3	CI	P-value
Percentage of ESI 1 or 24	0.466	(0.187, 0.676)	0.000
Case Mix Index	0.383	(0.101, 0.607)	0.006
Academic Status – Non-Academic5	-	-
Academic Status – Primary AMC6	-	-	0.0297
Academic Status – AMC Affiliated	-	-
Annual ED Volume	0.378	(0.123, 0.586)	0.005
Inpatient Admission Rate	0.360	(−0.113, 0.420)	0.015
Percentage of Medicare Patients	0.171	(−0.031, 0.560)	0.226
Patient per Attending hr/day	0.127	(−0.142, 0.387)	0.366

¹Imputed complete outcome cohort (N= 54).

² High acuity chart as determined by Center for Medicare and Medicaid Services’ (CMS) Ambulatory Payment Classification (APC) System.

³Spearman correlation coefficient.

⁴ESI is the emergency severity index

⁵Reference for comparison

⁶AMC is an academic medical center.

⁷Kruskal-Wallis rank sum test was used for this categorical variable.

Scatterplots suggest the percentage of patients with ESI 1 or 2, CMI and the percentage of Medicare patients are linearly related with PHAC. Bell-shaped loess smooth curves from the scatterplots for annual ED volume, inpatient admission rates and patients-per-attending-hour suggest potential non-linear relationships with PHAC (Figure 2a–f).

Figure 2.

Scatterplots for the Associations between Common Acuity Measures (Continuous Variables) with the Proportion of High Acuity Charts

Results from univariate analyses using quasi-binomial GLM are consistent with findings from rank-based analyses. We found similar notable associations between the following 5 acuity measures and PHAC: percentage of patients with ESI 1 or 2, CMI, academic status, annual ED volume and inpatient admission rate (Table 3). When all acuity measures are included in one model, only academic status demonstrated a statistically significant association with PHAC, and annual ED volume is borderline (p=0.052). The R² from this full model accounts for less than half of the PHAC variability. No systematic patterns against randomness around 0 were observed from residual plots of each acuity measure from the full model. Results using linear regression are similar to quasi-binomial GLM, but with lower R² for both univariate analyses and multiple linear regression, which indicates quasi-binomial GLM is a more appropriate modeling approach.

Table 3.

Regression Analyses: Relationship between Common Measures of ED¹ Patient Population Acuity and the Proportion of High Acuity Charts (PHAC)²

3a – Generalized Linear Model3
	Univariate Analysis4 (N=54)				Multiple Regression (N=54)
	OR	CI	P-value	R2	OR	CI	P-value	R2
Percent ESI 1 or 26	1.03	(1.01, 1.05)	0.015	0.16	1.01	(0.99, 1.04)	0.263
Case Mix Index	2.21	(1.15, 4.25)	0.020	0.14	1.58	(0.79, 3.16)	0.204
Academic Status – Non-Academic	1.00	-	-		-	-	-	0.426
Academic Status – Primary AMC7	1.09	(1.03, 1.16)	0.007	0.15	1.07	(1.00, 1.14)	0.052
Academic Status – AMC Affiliated	1.73	(1.08, 2.79)	0.027		1.84	(1.15, 2.93)	0.014
Annual ED Volume	2.04	(1.23, 3.38)	0.008	0.16	1.49	(0.79, 2.81)	0.228
%Medicare Patients	1.02	(0.99, 1.04)	0.242	0.03	1.02	(0.99, 1.05)	0.193
Inpatient Admission Rate	1.01	(1.00, 1.02)	0.170	0.04	1.00	(0.99, 1.02)	0.615
Patient per Attending hr/day	1.11	(0.58, 1.45)	0.435	0.01	0.96	(0.73, 1.27)	0.777

3b – Linear Regression Results3
	Univariate Analysis4 (N=54)				Multiple Regression5 (N=54)
	Beta	CI	P-value	R2	Beta	CI	P-value	R2
Percent ESI 1 or 26	0.006	(0.001, 0.011)	0.013	0.13	0.003	(−0.003, 0.009)	0.27
Case Mix Index	0.178	(0.019, 0.336)	0.021	0.11	0.101	(−0.090, 0.929)	0.21
Academic Status – Non-Academic	-	-	-	-	-	-	-
Academic Status – Primary AMC7	0.019	(0.005, 0.032)	0.007	0.13	0.016	(0.001, 0.031)	0.04
Academic Status – AMC Affiliated	0.134	(0.02, 0.25)	0.025	0.15	0.142	(0.029, 0.256)	0.01	0.385
Annual ED Volume	0.170	(0.05, 0.292)	0.008	0.15	0.098	(−0.067, 0.264)	0.19
%Medicare Patients	0.004	(−0.003, 0.01)	0.242	0.03	0.004	(−0.002, 0.011)	0.20
Inpatient Admission Rate	0.002	(−0.002, 0.006)	0.174	0.03	0.001	(−0.003, 0.005)	0.62
Patient per Attending hr/day	0.025	(−0.038, 0.089)	0.435	0.01	−0.001	(−0.079, 0.058)	0.75

¹Emergency department (ED).

²High acuity chart as determined by Center for Medicare and Medicaid Services’ (CMS) Ambulatory Payment Classification (APC) System.

³Imputed complete outcome cohort (N= 54).

⁴The univariate analysis test each common measure for its individual association with PHAC.

⁵The multivariate analysis combines all measures to test their collective ability to explain variation in PHAC.

⁶ESI is the emergency severity index

⁷AMC is an academic medical center.

Through univariate quasi-binomial GLM, we quantified the change in PHAC associated with a unit of change in each measure. Specifically, PHAC increases by 3% with a 1% increase of patients categorized as ESI 1 or 2, increases by 8% for each 0.1 unit increase in CMI, and increases by 9% for each 10,000 patient increase in annual ED volume. The ratio of AMC affiliated ED and primary AMC ED PHAC to non-academic EDs was 1.73 and 2.04 respectively.

DISCUSSION

We found the percentage of patients with ESI 1 or 2 had the strongest association with PHAC followed by CMI and annual ED volume. Academic status captures variability outside of that explained by ESI, CMI, annual ED volume, percentage Medicare patients, or patients-seen-per-attending-per-hour as evidenced by retaining its significant association in the multiple regression analysis. We found no significant relationships between PHAC and the percentage of Medicare patients, inpatient admission rate, or patients-seen-per-attending-per-hour.

Emergency Severity Index (ESI) Scoring

The percentage of patients with an ESI score of 1 or 2 had the strongest, albeit moderate, correlation with PHAC. These are patients anticipated to need immediate attention for a potentially life or limb threatening condition. ESI is a prospectively applied score that considers illness urgency and intensity of anticipated services. It is calculated for the majority of individual ED patients shortly after arrival and prospectively estimates acuity using vital sign instability, signs of distress, symptoms associated with urgent medical needs, and the clinical judgement of emergency medical service personnel and triage providers. The ESI’s inclusion of objective findings, clinical judgement, predetermined symptoms of concern, and the majority of ED patients likely impart a significant advantage over the other measures which primarily rely on retrospectively assessed administrative data. Given these characteristics, we anticipated a stronger association in our regression analysis, and a more positive slope in Figure 2f. We found a more moderate correlation than we anticipated. This may be due to limitations of the measure as it scores anticipated needs and does not account for the actual illness level determined or resources used during the ED visit.

Case Mix Index (CMI)

CMI also had a moderate correlation coefficient. It is a complex measure designed to capture the intensity of in-hospital care provided to admitted Medicare or Medicaid patients. The adjusted association, however, was not statistically significant. Our loess curves of the relationship with PHAC (Figure 2a) showed a slope of nearly 1, but it appears to be significantly influenced by two outliers. The linear model demonstrates a more positive slope that may plateau in higher CMI facilities, thus reducing the strength of the association overall. However, this is unclear given there are few EDs with CMI >2.3. CMI’s major limitation is it excludes the majority of patients, those who are discharged from the ED or admitted to observation. The ability of CMI to stratify acuity within the admitted patient subgroup likely accounts for its discrimination of PHAC across EDs.

Annual Emergency Department (ED) Patient Volume

Annual ED volume had a moderate correlation with PHAC. This may be because, although it accounts for the full ED patient spectrum, it provides no information on services or illness level for individual patients. Nonetheless, prior work has demonstrated an ED will function differently and require an increase in the intensity of resources to meet care demand with every 20,000 patient increase in annual ED volume,^2–4 This increase in acuity with rising volume is seen in our data (Figure 2b). The relationship demonstrates a sharp rise until EDs reach 60,000 patients per year, then the trend levels off. A more linear trend in the higher volume EDs may be difficult to detect in this study due to there being only 7 EDs with annual volume >100,000 visits. Greater representation of larger volume EDs may demonstrate a stronger association.

Academic Status

There was a clear distinction in acuity between the academically-connected EDs and non-academic community practice sites illustrated by the medians of the academic groups being not only higher than non-academic EDs, but also falling outside of the non-academic ED interquartile range. In Figure 3, however, we did not observe a significant distinction in the distribution of acuity between primary academic medical center (AMC) EDs and academically-affiliated EDs. AMCs serve as teaching centers partly because they provide a mix of patient acuity adequate for trainee education. As a result, they tend to be urban, tertiary care referral centers, with high acuity patient populations. It is, however, possible that resident and attending physician documentation may comparatively increase the care captured in patient charts and augment the resultant APC coding. Academic-affiliated sites are typically the community practice experience locations for trainees, thus a step away from the tertiary care hub function of many primary academic centers. As a result, we anticipated a significant difference from primary AMC sites. It may be that academic-affiliated sites benefit from added resident staffing in a practice environment were attending physicians document for the majority of patients. This increased documentation may augment coding. This is in contrast to primary AMC EDs where attendings more typically review resident documentation for most patients and may be more engaged in bedside teaching for a larger proportion of trainees than charting.

Figure 3.

Histogram for the Association of Academic Status with the Proportion of High Acuity Charts

Inpatient Admission Rate

We found a moderate correlation coefficient and a no significant regression associations between inpatient admission rate and PHAC. It seems intuitive that a higher percentage of patients admitted under inpatient status would indicate a sicker patient population. Hospitals, however, have varied thresholds for admission and varied ED resources. In some cases, more intense ED care may obviate inpatient admission. This makes it difficult to compare one ED’s admission rate to another. In addition, this measure reveals little about the care provided during the ED stay, and does not represent the majority: those discharged or admitted to observation. The mean inpatient admission rate have been previously reported as 17.5%.¹⁶ This is similar to the 18% observed in our study. The scatterplot demonstrates a sharp rise until inpatient admissions reach approximately 18–20% with an IQR of 14–24% (Figure 2d). We anticipated this variability, within the interval with the steepest slope, would reliably discriminate acuity across EDs, as is often assumed in practice. However, the moderate correlation observed suggests other factors may contribute to the admission decision. Some EDs may provide more extensive diagnostic testing or treatment permitting discharge that, if deferred, could qualify a patient for inpatient admission. (i.g., cardioversion for atrial fibrillation with a rapid ventricular response;^17,18 or young patients with suspected infection who present with hemodynamically unstable vital signs, improve with resuscitation, and are discharged with home care). This may be due to most EDs lying within the plateau area of the loess curve in the scatter plot (Figure 2e). Additionally, this metric is vulnerable to variable leveling as inpatient versus observation status across institutions. This may create artificial differences between similar patient populations. For example, a 2013 OIG report found significant variation across hospitals in the relative mix of long outpatient stays and short inpatient stays.¹⁷ This may explain why the OR for inpatient admission rate were 1.0 in both our univariate and multivariate regression analyses.

Percentage of Medicare Patients

We found a weak association with PHAC and the percentage of Medicare patients. A larger percentage of ED Medicare beneficiaries would seem to introduce a larger patient cohort with higher risk for severe illness due to increased age, number of comorbidities, and frequency of healthcare utilization. This group, however, is generally a minority subset of the larger ED population and thus this measure does not account for the service or illness for the majority of patients. We observed a slight, but non-statistically significant, trend towards an increasing percentage of high acuity charts with an increasing Medicare population (Figure 2c). The lack of association in our correlation or regression analyses suggests any association with acuity is indirect.

Patients Seen Per Attending Hours Per Day

A high patient to attending physician ratio, measured as patients-per-attending-hour, is assumed to correlate with a lower acuity ED. The rationale is if one physician can see many patients, the intensity of care and level of illness of the broader patient population is likely low. We observed an initial rise in the association of patients-per-attending-hour, an inflection around 3 patients per hour, then a downturn (Figure 2e). The progression from 1–3 patients per hour may be reflective of higher acuity with ≥3 patients per hour is consistent with a low acuity environment. Nursing, resident, and physician extender staffing levels, in addition to facility design and other variables such as the use of scribes may impact the productivity of an attending physician. This efficiency effect is a difficult confounder to quantify.²⁰ Alternatively, physicians seeing ≥3 patients per hour may not have the time to provide the same attention to documentation details thus resulting in a lower billing acuity. This parabolic association, however, did not demonstrate significant influence in our study. Further investigation into the dynamics responsible for inflection at ~3 patients per hour warrants further investigation.

LIMITATIONS

There is no established reference standard for retrospectively measuring ED patient population acuity. As a result, we present our study as a thought provoking analysis exploring the validity of several traditional acuity measures for cross-facility comparisons. Participating EDs are members of a voluntary reporting organization motivated to share these metrics for the purpose of research. This likely introduces selection bias into the sample. To reduce this bias, the EDOSG strategically recruits EDs from all regions of the US with attention to the diversity of ED types (size, population served, urbanicity, and academic status) as is reflected in Table 1. The study uses an administrative data set, however, the data were collected with the intent of informing this specific study. The sample size is small with 27 of the 54 EDs sending complete data. However, 88% of the data requested was received, and we applied multiple imputations to our Complete Outcome Cohort (54) to reduce the bias introduced by a small data set (Figure 1).

Figure 1.

ED Inclusion/Exclusion Diagram

The objective of our modeling was to quantify each common acuity measure’s association with PHAC using this sample, rather than generating probabilities for unobserved EDs. As a result these findings should be considered suggestive given the limitations described above. Many multi-center studies of this type are limited to focal regions of the US or a particular type of ED. In addition, they often include academic centers with an interest and an incentive to perform research. This data set includes representation of the geographic and demographic diversity of EDs in the US including sites not typically involved in clinical research. There is less representation of the West and Midwest; however the proportional regional representation mirrors the population distribution of the US.

Our outcome variable is a percentage (or fraction) rather than a continuous variable, which involves loss of information on the acuity of individual patients within each ED. However, our primary objective was to capture the presence of a subset of truly acute patients within each ED. In addition, a fractional response outcome permits the use of a quasi-binomial linear model with a logit link allowing us to include variables into the analysis without major transformations that risk over-fitting the model to the idiosyncrasies of our sample. Despite our concerns over assuming linearity, our linear regression results mirrored that of the GLM. PHAC’s dependent on the translation of medical record data to acuity scoring for billing reimbursement may introduce information bias and selection bias. In a study examining the accuracy of coding-based billing compared to coding from direct observation, they found agreement amongst different coders in only 55% of patient visits.²¹ Other studies have documented similar agreement level with the use of similar methods.^22,23 However, under and over estimating the intensity of services provided was equivalent suggesting that aggregate summary measures of patient visits over time may be valid. In addition, much of this prior research was done at a time when our current coding and reimbursement system was being introduced to medicine. Contemporary provider practices have likely adjusted. However, longitudinal research using billing data should account for this confounder. Transitioning electronic medical record systems has been known to impact the quality of documentation and the accuracy of billing. However, none of the EDs in this sample were involved in an EMR system change during the study period.

Our unit of analysis is the ED, and PHAC is an aggregated measure for individual EDs. As a result we have not weighted the PHAC for number of patients included at each hospital. This was handled in our analysis with the use of the logit link. The use of a fractional response outcome prevents one from using our model to estimate the acuity for an ED that was not included in our sample. Even if we had weighted our outcome variable, the low odds ratios obtained would not yield reliable estimates of PHAC.²⁰ This was a study performed on an available dataset, and the sample size did not permit traditional internal validation procedures. However, we boot strapped our confidence intervals to validate the associations of acuity. As a result of this discrepancy, our analysis should be repeated in a larger sample to assure external validity before widespread adoption of our results.

CONCLUSION

The percentage of ESI 1 or 2 patients had the strongest association with PHAC followed by CMI and annual ED volume. Academic status captures variability outside of that explained by ESI, CMI, annual ED volume, percentage Medicare patients or patients-seen-per-attending-per-hour. The relative strengths of association observed in our study suggest a hierarchy of quality as measure for ED acuity, but should be examined in a larger data sample. All measures combined only explained 42.6% of PHAC variation so the validity of their common use as proxies for ED acuity should be further investigated.