Marginal analysis in assessing factors contributing time to physician in the Emergency Department using operations data

Abstract

Background: Standard Emergency Department (ED) operations goals include minimization of the time interval (tMD) between patients' initial ED presentation and initial physician evaluation. This study assessed factors known (or suspected) to influence tMD with a two-step goal. The first step was generation of a multivariate model identifying parameters associated with prolongation of tMD at a single study center. The second step was the use of a study center-specific multivariate tMD model as a basis for predictive marginal probability analysis; the marginal model allowed for prediction of the degree of ED operations benefit that would be affected with specific ED operations improvements. Methods: The study was conducted using one month (May 2015) of data obtained from an ED administrative database (EDAD) in an urban academic tertiary ED with an annual census of approximately 500,000; during the study month, the ED saw 39,593 cases. The EDAD data were used to generate a multivariate linear regression model assessing the various demographic and operational covariates' effects on the dependent variable tMD. Predictive marginal probability analysis was used to calculate the relative contributions of key covariates as well as demonstrate the likely tMD impact on modifying those covariates with operational improvements. Analyses were conducted with Stata 14MP, with significance defined at p < 0.05 and confidence intervals (CIs) reported at the 95% level. Results: In an acceptable linear regression model that accounted for just over half of the overall variance in tMD (adjusted r ² 0.51), important contributors to tMD included shift census (p = 0.008), shift time of day (p = 0.002), and physician coverage n (p = 0.004). These strong associations remained even after adjusting for each other and other covariates. Marginal predictive probability analysis was used to predict the overall tMD impact (improvement from 50 to 43 minutes, p < 0.001) of consistent staffing with 22 physicians. Conclusions: The analysis identified expected variables contributing to tMD with regression demonstrating significance and effect magnitude of alterations in covariates including patient census, shift time of day, and number of physicians. Marginal analysis provided operationally useful demonstration of the need to adjust physician coverage numbers, prompting changes at the study ED. The methods used in this analysis may prove useful in other EDs wishing to analyze operations information with the goal of predicting which interventions may have the most benefit.

Introduction

The time interval between patients' Emergency Department (ED) presentation and their initial evaluation by a physician is a key performance indicator in ED operations. Minimization of the interval between ED arrival and first physician evaluation, hereafter termed tMD, has ramifications for medical care quality, ED operations efficiency, and patient satisfaction. There are many other factors that influence ED operations (e.g., laboratory and radiology turnaround times), but tMD is one of the more important parameters. The medical advantages of seeing patients more quickly are self-evident, and it is clear that patients who are seen faster are less likely to leave before they are seen.^1,2

Even those studies focusing on other ED operations parameters often stress on the importance of tMD. This is exemplified by the literature focusing on “left without being seen” (LWBS) cases.^3,4,5 One reason LWBS continues to be a problem in many EDs is its consistent association with prolonged tMD. The best efforts to reduce LWBS have been met with limited success when tMD remains too long.^6,7,8,9

By using triage scales such as the Emergency Severity Index (ESI)¹⁰ to stratify patients, previous investigators have set tMD goals.⁸ Those cases in the acuity mid-range (ESI 3 on the 1–5 ESI scale) are recommended to have wait times of < 45 minutes; less acute ESI 4/5 patients' wait time target should be < 60 minutes.⁸ Another triage scale, the Canadian ED Triage and Acuity Scale (CTAS), has also been used as a basis for establishing tMD goals.¹¹

Whether emphasizing reduced LWBS or increased medical safety, the literature is clear on the importance of streamlining tMD. Given the impetus to minimize tMD, the current study was designed as a two-step process.

First, the investigators wished to assess at one hospital, Hamad General Hospital (HGH), ED operational data, in an attempt to generate a HGH-specific multivariate tMD model. Second, the study plan called for use of the study center-specific tMD model as a basis for predictive marginal probability analysis (“marginal” analysis).

Marginal analysis allows for fixing of values for each of the regression model's covariates, with subsequent calculation of the dependent variable's value, given those covariate values. In essence, marginal analysis executes prediction of the exact result for a dependent variable (in this study, tMD), given dictated values of a regression model's independent variables (covariates).¹² Marginal analysis is not novel to the ED operations literature, but despite its straightforward appeal for both analysis and illustration in multivariate modeling, the technique is not commonly encountered.^12,13

The current study aimed to first generate a multivariate model that describes the study center's tMD and then to use that model as a basis for marginal analysis to delineate tMD improvements accrued with operational changes. Specifically, given a priori concerns about fluctuating levels of physician coverage at the study ED, study planning called for marginal analysis to assess the effect of ensuring a minimum number of physicians per shift. This study describes execution of the results of model generation and marginal analysis. An important goal of the current study was demonstration of the details of methodology of marginal analysis, since these details can allow others to reproduce the methods in other settings. This study report therefore includes, as a mechanism to optimize the utility of the analysis tool to others, a detailed report of the statistical methodology. A planned analysis for the future will describe whether enactment of operational changes had the predicted effect on tMD.

Methods

This was a retrospective analysis of data, which were collected and entered into the HGH ED administrative database (EDAD). There was no analysis of patient identifiers, protected health information, or any clinical information that would identify individual cases. The institution's ethics review board exempted the study.

Setting and time frame

The study was conducted at HGH, the flagship hospital of the government-operated healthcare system of Qatar. As the sole tertiary-care general ED for a country with a population exceeding 2.5 million, HGH is busy. The annual ED census is approximately 500,000; during the study month of May 2015, the ED recorded 38,593 visits.

The ED operates in shifts for both medical and ancillary staff. There are three shifts: day shift (0600–1400), evening shift (1400–2200), and night shift (2200–0600).

The ED is operated with tiered physician staffing. Overall supervision comes from 40 EM consultants, while direct care is provided by approximately 150 EM specialists (i.e., post-residency, EM-boarded physicians), 20 EM fellows (first and second-year post-EM residency), and 40 residents in a four-year training program accredited by the Accreditation Council of Graduate Medical Education-International (ACGME-I).

During the study period, in the ED, the consultants' role was virtually always supervisory and administrative in nature. There were relatively few consultants in the ED at any one time (4, 6, and 3 during the day, evening, and night shifts respectively). Furthermore, the numbers of ED consultants did not change from one day to the next; there were always the same numbers of consultants in the ED at the same time of day, every day. The low numbers of consultants, the fact that consultants were not responsible for hands-on delivery of care, and the known consistency of consultant coverage numbers from day to day were responsible for an a priori decision to focus tMD analysis on numbers of “non-consultant” physicians (i.e., the specialists, fellows, and residents who were actually the first to see the patients and determine tMD). These “non-consultant” physicians are hereafter referred to as “ED physicians.”

All ED physicians work in 8-hour shifts along the previously iterated time frames for day, evening, and night duty. Each day, there are a few “mid-shifts” (e.g. 1000–1600) that supplement coverage during parts of the day, evening, and night shift time frames. The goal for the study ED is to achieve consistent physician coverage throughout the 24-hour day, but staffing issues result in slightly reduced coverage during the overnight time period. With fractional apportionment of mid-shift ED physicians to the three standard shifts (e.g., 10:00–16:00, ED physician is apportioned half to the day and half to the evening shift), their goal of 22 ED physicians on duty is usually met during day and evening shifts and usually not met during overnight shifts. The a priori plan for analysis was to determine whether physician n < 22 was associated with tMD, and if so to determine the potential effect of changing physician staffing to ensure a consistent and minimum number of 22 physicians during all shifts.

The ED is divided into separate areas delineated by acuity and demographics (sex and nationality). Patients at the highest acuity levels are seen in the more-urgent areas of the ED regardless of demographics. Women and children, as well as those of Qatari nationality, are seen in separate areas of the ED. With the exception of the highest-acuity patients, who do not stop at triage, the general patient flow starting with triage and then physician evaluation is the same in all parts of the ED. The clinical and operational approach to the separate parts of the ED is the same as that for the other areas of the ED, and the same physicians simultaneously cover both areas.

In particularly busy times, an ED physician may be positioned in the waiting area for the lower-acuity areas of the ED to facilitate nursing triage. During the study period, these physicians did not document on the medical record (and thus did not impact EDAD-recorded tMD).

Data collected: The ED administrative database (EDAD)

The study's data source was the hospital's EDAD, which has been in operation for over two years. Mechanisms surrounding the EDAD's data abstraction and entry methods did not change in the few months surrounding the study period.

The EDAD comprises information extracted by administrative personnel (mostly nurses who are not involved with clinical care), who read a copy of every ED chart on a daily real-time basis and populate the EDAD with relevant data. The workflow for recording the time data used in this study followed standard paper-based charting methods. At patient presentation, nursing triage, and physician initial evaluation, the ED staff document the times of interaction with the patient as part of the medical record. Data collection and entry proceeded under the supervision of an administrative physician (co-author JM) for whom EDAD management is a full-time effort.

During the study period, the hospital did not use an electronic medical record (EMR). Thus, for every patient visit to the study ED, a paper medical record was written. A copy of this record was forwarded to the EDAD team at the time of patient disposition. The EDAD team then reviewed the paper record to abstract the data elements.

EDAD data elements included times of presentation, triage, and initial ED physician evaluation. The EDAD also dichotomously categorizes arrival mode as to whether patients came to the ED via Emergency Medical Services (EMS). Also included for each visit was the presence or absence of referrals to the ED (from another physician).

The EDAD indicates whether patients were admitted to a hospital service outside of the Department of EM. However, due to situations unique to the study center, the EDAD does not indicate all admissions. This is because the Department of EM itself is responsible for well over half of all admissions, via its Short-Stay Unit (SSU). The SSU, staffed by Internal Medicine physicians working under EM auspices, comprises three separate areas of 30 beds in total; SSU admits dozens of cases per day and cares for patients for up to five days. The relevance of the SSU to the current study is limited to the fact that its existence skews downward the reported “admission percentage” since in the EDAD during the study period, this admission number included only those admitted outside the EM group.

The EDAD also records presenting chief complaints (CCs) that are grouped very broadly (e.g., “gastrointestinal” group includes anything from hematemesis to constipation). These CCs, recorded directly from patient-verbalized reason for ED visit, are a coarse reflection of case mix.

EDAD demographics data include age, sex, and nationality. By government direction, the ED's nationality focus rests on whether patients are Qatari nationals.

Analysis approach

Statistical analysis was performed with Stata 14MP (StataCorp, College Station, TX). Relevant 95% confidence intervals (CIs) were calculated using exact techniques. For hypothesis testing, significance was defined as the absence of overlapping 95% CIs or a p value of < 0.05.

Unit of analysis

The study's unit of analysis was the “ED shift.” Since the study month (May 2015) comprised 31 days, the study's number of shifts was 93 (i.e., three shifts per day).

Descriptive statistics

Measures of central tendency were assessed for each of the 93 shifts. For categorical data (e.g., nationality, sex), proportions are reported.

For data that were not categorical, the Stata skewness–kurtosis testing procedure was performed to assess normality. Data that were identified as normal had a central tendency value reported as mean ±  standard deviation (SD). Non-normal variables' central tendency is reported as a median value with interquartile range (IQR).

The study's primary endpoint of interest was tMD. Since time-to-event data are best assessed with medians, this was the approach used to generate the tMD estimate for a given shift.¹³ Since the pooled collection of all 93 shifts' tMD medians were normally distributed (skewness–kurtosis p = 0.56), the overall tMD measure of central tendency was the mean value.

Time to triage (tTriage) was incorporated into the analysis not as a covariate of focus, but rather to allow modeling to account for any triage delays that would prolong tMD. For descriptive statistics, since the overall group of shift medians (tTriage for the 93 shifts) were found to be non-normal (skewness–kurtosis p = 0.0004), central tendency was reported as median with IQR.

The shift census (i.e., number of patients that presented during a given shift) was included as a key covariate. Similarly important was the number of physicians who were on duty for each shift. By using information tracked on a daily basis at the time of the actual ED shift, the doctors' schedule served as the basis to determine each shift's physician n.

The patient case mix (CC) data for the entire study population (n = 38,593) were assessed and grouped per a priori-defined categories. Those CC categories that had at least 1000 cases during the study month were included in the analysis. This 1000-case minimum was met by eight CC categories that accounted together for 92.7% of all ED visits. For each of the 93 shifts, the proportions of cases constituted by each of the eight most common CC categories were tracked and incorporated into the analysis. For example, if in a shift, 100 cases were seen and 20 were GI, then that shift's CC number for GI would be entered as 20%.

Univariate analysis

Categorical data were assessed using Chi-square testing. Univariate linear regression and non-parametric (Kruskal–Wallis) testing were used to assess continuous variables.

For regression, estimates of covariates' effect of magnitude β (i.e., expected change in the dependent variable with one-unit change in the covariate) are reported with 95% CIs. A univariate analytic cutoff of p < 0.20 was used to define covariates to be assessed for significance in the overall multivariate model.¹⁴

Multivariate analysis

The multivariate analysis entailed execution of linear regression with the dependent variable tMD as the primary outcome of interest. A stepwise model-building approach was used, with addition of covariates identified (by univariate analysis) as potentially important. As model-building proceeded, the possible of effect modification was assessed using interaction terms. Potential confounding was checked by the reintroduction of covariates into the model to assess whether their inclusion (regardless of statistical significance) resulted in >20% change in the main effect β point estimate.¹⁴

The overall model performance was assessed using a variety of approaches. The adjusted r ² value was used to indicate the proportion of the overall tMD variance that was accounted for by the model. Stata's post-estimation information matrix test was used to evaluate model skewness, kurtosis, and heteroscedasticity. A plot of residuals versus fitted values was used to assess for reasonability of the assumptions of linearity (i.e., random “bounce” of residuals around a baseline of 0), equal error term variances (i.e., horizontal banding around 0 line), and few outliers. The link test was used to assess model specification.

After the model was generated, a marginal predictive probability analysis was executed using Stata algorithms, as outlined by Mitchell.¹⁵ The marginal analysis was used to assess and visually depict the impact of a single variable on the outcome of interest, while simultaneously adjusting for all other covariates in the model.

EDAD data validation and missing-records assessment

The reliability of the EDAD data used as the basis of the study was assessed in two fashions. The two-step approach was used to optimize the chances of detecting study bias related to errors or omissions in the EDAD.

The first step, EDAD data validation, was to assess reliability of the data in the EDAD. Study physician researchers re-reviewed 1% (n = 372) of the total month's charts that had been entered into the database by the EDAD team. Each of a dozen covariates were reassessed with the initial data entry categorized dichotomously as “correctly recorded” (i.e., exactly the same as determined by the study physician team) or “incorrectly recorded.” The proportions of correctly entered data were recorded for each variable to provide a measure of accuracy of the initial EDAD team's records.

The second step in EDAD validation entailed the use of nonparametric (Kruskal–Wallis) testing to search for potential effects of data loss (i.e., missed cases that were never entered into the EDAD). The initial approach dealt with assessing for association between the proportion of a shift's “missed EDAD cases” and the endpoint of interest (tMD). The next approach entailed hand-retrieval of charts for an actual review of the 1% of cases that had initially bypassed the EDAD entry.

With regard to “missed EDAD cases”, the first analysis was performed to assess for association between the median proportion of missed cases on a shift, and the shift's median tMD. In a second analysis, shifts for which at least 10% of cases were missed were defined a priori as “high file-loss” shifts; the “high file-loss” status was then assessed for association with tMD.

The final step in assessing the possible impact of cases missed by the EDAD entailed physical chart retrieval (from the hospital medical records department) of the 1% of cases that had been missed by the EDAD. EDAD data abstraction from these initially missed records was followed by analysis to assess whether this group's covariate central tendencies were similar to those of the overall EDAD covariate central tendencies.

Results

Descriptive statistics

Table 1 shows summary statistics for the study data as collated for the unit of analysis of study shift (n of 93).

Table 1.

Overall descriptive statistics for 93 ED shifts.

Parameter	Central tendency	IQR* or 95% CI* *
Median % of charts entered into EDAD	91.4	IQR 87.7–93.2
Median time to triage (min)	11	IQR 7–21.5
Median time to physician (tMD in min)	47	IQR 35–57
Mean % of patients admitted outside of ED	4.0	95% CI 3.8–4.3
Median n patients	498	IQR 135–587
Median n physicians	22	IQR 20–23
Median % of patients from Qatar	19.5	IQR 17.1–21.3
Mean % of male patients	64.8	95% CI 64.0–65.9
Mean age of patients	30.5	95% CI 30.2–30.8
Mean % of ambulance arrival	15.2	95% CI 14.6–15.9
Median % of patients with physician referral to the ED	3.1	IQR 2.1–4.1
Median % of chief complaint categories:***
Gastrointestinal	16.9	IQR 15.0–19.6
Soft tissue injury or infection	11.4	IQR 10.1–13.6
Musculoskeletal pain	9.7	IQR 7.5–11.7
Eye, ear, nose, or throat	9.5	IQR 7.8–10.6
Neurologic or psychiatric	8.2	IQR 7.3–9.3
Shock or trauma	7.6	IQR 6.4–9.7
Chest complaints	6.9	IQR 6.0–7.6
Fever	4.9	IQR 4.0–5.9

*IQR: interquartile range.

**CI: confidence interval.

***The remaining 24.9% of cases had “other” diagnoses.

Univariate analysis

Univariate analysis entailed assessment for association between the primary endpoint of a shift's tMD and individual variables being considered for inclusion in multivariate modeling.

The first parameter assessed was the study date. The study included the 31 days of the month of May 2015, and study date was assessed to determine whether there was a relationship between the advancing day of the month and the tMD variable. Kruskal-Wallis testing revealed no association (p = 0.48) between study date and tMD. Also not associated with tMD were the rates of referrals to the ED (p = 0.16) or the rates of admission of patients to non-EM services (p = 0.37).

Univariate analysis with Kruskal–Wallis testing revealed no significant association (p = 0.26) between tMD and the proportion of files entered into the EDAD. Similarly, when the shifts that were determined to have a “high proportion” (at least 10%) of cases that bypassed the EDAD entry were analyzed categorically, these “high file loss” shifts were not associated (p = 0.94) with a longer tMD.

Time to triage (tTriage) was correlated (p < 0.001) with tMD. tTriage was therefore included in the multivariate analysis along with the other covariates as outlined in Table 2.

Table 2.

Univariate associations between independent variables and time to physician (tMD).

Parameter	tMD mean (min)	95% CI*	p * *
Shift time of day			0.0001
Morning (0600–1400)	Mean 35.6	31.7–39.5
Evening (1400–2200)	Mean 49.8	45.0–54.7
Night (2200–0600)	Mean 53.1	49.2–57.1
Number of on-duty physicians	Mean 21.7	21.3–22.2	0.005
Shift weekday (Sunday = baseline)
Sunday	Mean 50.6	43.6–57.5	–
Monday	Mean 49.3	38.4–60.2	0.259
Tuesday	Mean 45.0	35.4–54.7	0.940
Wednesday	Mean 46.8	37.9–55.7	0.168
Thursday	Mean 48.5	38.8–58.1	0.436
Friday	Mean 38.2	30.9–45.5	0.041
Saturday	Mean 46.0	41.3–50.6	0.798
β-coefficient*** in regression vs. tMD
Shift census proportion of:
Patients from Qatar	1.1	0.48–1.8	0.001
Females	0.82	0.21–1.4	0.009
Patients arriving by ambulance	0.86	− 0.03–1.8	0.057
Median age of patients in shift	− 4.0	− 5.7 to − 2.2	< 0.001
Chief complaint (proportion in shift)
Gastrointestinal	− 1.4	− 2.3 to − 0.6	0.001
Soft tissue injury or infection	− 0.1	− 1.3 to 1.0	0.827
Musculoskeletal pain	− 0.1	− 1.2 to 0.9	0.787
Eye, ear, nose, or throat	− 0.7	− 2.0 to 0.6	0.317
Neurologic or psychiatric	− 0.4	− 2.3 to 1.5	0.660
Shock or trauma	1.5	0.4 to 2.7	0.009
Chest complaints	0.1	− 1.7 to 1.9	0.936
Fever	1.5	− 0.2 to 3.2	0.092

*CI: confidence interval.

**The p value indicates the significance level of testing for association between tMD and the parameter denoted in the first column.

***The β-coefficient indicates how strongly (indicated by an absolute value with zero as null value), and in which direction (indicated by the sign, with negative β meaning inverse correlation) the predictor variable influences the outcome of interest.¹⁴

Multivariate linear regression: Factors associated with time to physician (tMD)

After the univariate basic analysis revealed the candidates for model inclusion (i.e., those factors with univariate p < 0.20), multivariate regression was undertaken using the stepwise model-building fashion.¹⁴

The model commenced with inclusion of the fundamental covariates – shift census and time to triage – that were well-known to be required in the model. The other covariates were then added into the modeling process and the final model was generated with the independent variables and effect estimates as depicted in Table 3.

Table 3.

Multivariate regression model.

Parameter	β (95% CI*)	p
Time to triage	1.0 (0.6–1.7)	< 0.001
Shift census	0.07 (0.02–0.12)	0.008
Minimum of 22 physicians on duty (dichotomous)	− 6.5 ( − 10.8 to − 2.1)	0.004
Median % of cases arriving by ambulance	0.7 (0.1–1.4)	0.021
Median % of patients with gastrointestinal-related chief complaints	− 0.82 ( − 1.5 to − 0.1)	0.020
Shift time of day
Morning (0600–1400)	Baseline	–
Evening (1400–2200)	16.2 (7.7–24.7)	< 0.001
Night (2200–0600)	42.5 (17.2–67.9)	0.002

At this point, forcing parameters back into the model to assess for confounding failed to identify any such factors. For example, when the percentage of patients admitted to non-ED services was reassessed in the model, none of the parameters' point estimates of effect (as outlined in Table 3) changed by more than 1%; also, the p value for the admission percentage covariate was not significant (p = 0.9). Similarly, non-significant results were identified with the proportions of patients who were referred in from outside physicians (p = 0.4) or who were from Qatar (p = 0.9).

The diagnostic testing for the linear regression final model revealed that the model was acceptable in all tested respects. The model's adjusted r ² value of 0.51 indicates that the regression model explains just over half of the variation in tMD. Post-estimation information matrix testing failed to identify any concerns with an overall p value of 0.4 and non-significant findings for heteroscedasticity (p = 0.5), skewness (p = 0.2), and kurtosis (p = 0.7). The link test was consistent with good model specification (p = 0.9).

Figure 1 provides visual support for the conclusion that the model's residuals and fitted values are uncorrelated (i.e., as they should be in the form of a homoscedastic model with normally distributed errors). The model residuals are the differences between the observed value and the predicted (fitted) value for the dependent variable.¹⁴

Figure 1.

Plot of fitted values versus residuals for the model outlined in Table 3.

Database validation and missing-values analysis

The first component of EDAD validation was a re-review of 1% of the records entered into the month's database to assess EDAD errors in data abstraction or entry for a dozen covariates of interest. For this assessment, if there were any difference (no matter how minor) between the first and second chart assessments, the covariate was labeled as manifesting disagreement.

A review of 372 records yielded perfect agreement for five covariates: day of the month, day of the week, sex, Qatari status, and admission to a non-ED service. For the remaining seven covariates, there was disagreement on at least one chart re-review. Overall, for a dozen covariates on the 372 cases, there was disagreement on 60 items out of 4464 data points for an overall disagreement rate of 1.3% (binomial exact 95% CI 1.0–1.8%).

Covariate-specific results are shown in Table 4. The table shows the numbers of records in which there was anything other than perfect agreement for the covariates listed. The information for the five covariates with perfect agreement in all 372 cases is not shown in the table.

Table 4.

Database validation results for covariates with any disagreement.

Parameter	n records (% of 372, 95% CI*) with disagreement
Age	8 (2.2%, 0.9–4.2%)
Arrival by ambulance	3 (0.8%, 0.2–2.3%)
Referral to the ED from a physician	2 (0.5%, 0.07–1.9%)
Arrival time	6 (1.6%, 0.6–3.5%)
Triage time	18 (5.1%, 3.1–7.9%)
Initial physician evaluation time	8 (2.2%, 0.9–4.2%)
Chief complaint	12 (3.2%, 1.7–5.6%)

*CI: confidence interval.

For tMD, there were eight cases in which the EDAD-recorded time was incorrect. In three cases, EDAD-recorded tMD was too short and in the other five cases, it was too long. These eight cases, with a combined median error of 9.5 minutes (IQR − 17.5 to 19.5) are outlined in Table 5. Similar calculations for the time to triage (tTriage) datapoint's reassessments yielded an underestimation of the initial recording in six cases and an overestimation in 10 cases (median error on initial EDAD recording of 5.5 with IQR − 4 to 21).

Table 5.

Cases of incorrect data entry for the time-to-physician parameter.

Case	Initial EDAD data	Corrected data	Error magnitude
Case 1	21	8	13
Case 2	13	73	− 60
Case 3	42	63	− 21
Case 4	34	2	32
Case 5	2	16	− 14
Case 6	17	11	6
Case 7	14	1	13
Case 8	26	0	26

The overall results after this review of 1% of EDAD cases were that the tMD changed in only a few dozen cases (of nearly 40,000 patients). There was no basis for a conclusion that the analyses were flawed by incorrect information abstraction and entering of data into the EDAD.

The next EDAD validation approach entailed requesting records that had been missed on entry into the EDAD. A total of 49 records (1% of the missed records) were requested from medical records for hand-review. Assessment of data from these records produced the information shown in Table 6.

Table 6.

Descriptive statistics for 49 validation cases.

Parameter	Central tendency	IQR* or 95% CI* *
Median time to triage (min)	12	IQR 4–40
Median time to physician (tMD in min)	44	IQR 26–62
Percentage of patients admitted outside of the ED	8.2%	95% CI 2.3–19.6%
Percentage of patients from Qatar	30.6%	95% CI 18.3–45.4%
Percentage of male patients	71.4%	95% CI 56.7–83.4%
Median patient age	32	IQR 22–42
Percentage of ambulance arrival patients	24.4%	95% CI 13.3–38.9%
Percentage of patients with physician referral to the ED	4.1%	95% CI 0.5–14.0%
Median % of patients with chief complaints:
Gastrointestinal	18.4%	95% CI 8.8–32.0%
Soft tissue injury or infection	18.4%	95% CI 8.8–32.0%
Musculoskeletal pain	6.1%	95% CI 1.3–16.9%
Eye, ear, nose, or throat	8.2%	95% CI 2.3–19.6%
Neurologic or psychiatric	14.3%	95% CI 5.9–27.2%
Shock or trauma	10.2%	95% CI 3.4–22.2%
Chest complaints	10.2%	95% CI 3.4–22.2%
Fever	8.2%	95% CI 2.3–19.6%

*IQR: interquartile range.

**CI: confidence interval.

As shown in Table 6, in all cases, the 95% CIs for all covariates abstracted from the initially missed records that underwent subsequent review, overlapped the central tendencies of the extant EDAD data (see Table 1). Differences in the fashion with which the data were tabulated precluded formal statistical testing of Table 1 and Table 6 data (the Table 6 data are from a set of individual cases and the Table 1 data are central tendencies from entire shifts). However, the substantial overlap between the Table 1 central tendency point estimates and the 95% CIs for every covariate as calculated for Table 6 contribute to the judgment that there is no suggestion that inclusion of all possible missing-chart data (i.e., information that was never entered into the EDAD due to charts' not being abstracted at the time of the ED visit) would alter the results of the study.

Statistical testing identified no significant difference between the validation-set covariates and the full EDAD dataset for all of the following analyzed covariates: age (p = 0.75), time to triage (p = 0.40), time to physician (p = 0.48), or proportions of males (p = 0.41), those from Qatar (p = 0.10), referrals (p = 0.79), admissions (p = 0.27), or arrival by EMS (p = 0.19). By all methods of testing performed, there was no suggestion of selection bias in the results of the assessment of 1% of missing-chart cases.

Marginal predictive probability analysis

Based upon a priori planning, marginal analysis was executed to explore and portray the relationship between the presence of a specified n (22) of ED physicians and tMD. The marginal analysis approach details are presented in depth in this work to enable the focus on the study's secondary goal of teaching application of this potentially valuable tool.

Of interest was the prediction of tMD improvement associated with increasing physician n from < 22 to 22. Of additional interest was the degree of incremental improvement associated with moving beyond 22 physicians on duty.

Adjusting for all other variables, the marginal analysis suggested a noteworthy incremental gain in tMD values associated with assurance of 22 on-duty physicians (data shown in Table 7). A 5.6 minute tMD improvement was seen once there were at least 22 physicians present. Further staffing increase had a lesser impact on tMD, which improved only by 1.4 minutes with increasing physician coverage to beyond 22 (when compared to n = 22 physicians).

Table 7.

Predictive marginal analysis of the tMD* impact of changing physician n and shift time of day.

On-duty physician staffing level	tMD (min), adjusting for other covariates (margin with 95% CI* *)	tMD difference (95% CI)
Physicians on duty
< 22	49.6 (46.6–52.7)	6.5 (2.2–10.9) min longer than 22 or more physicians
22	44.0 (39.2–48.9)	1.4 ( − 4.7 to 7.6) min longer than >22 physicians
>22	42.6 (39.0–46.3)	–
Shift time of day
Day (0600–1400)	26.5 (15.0–37.9)	–
Evening (1400–2200)	43.2 (37.7–48.7)	16.7 (7.8–25.4) min longer than day shift
Night (2200–0600)	68.9 (53.6–84.2)	25.7 (6.1–45.2) min longer than evening shift; 42.4 (16.2–68.6) min longer than day shift

*tMD: time to physician (in min).

**CI: confidence interval.

For technical reasons related to margins calculations in Stata, an overlap of margins' 95% CIs (as depicted in Table 7) cannot be used to ascertain the significance of changes from one group to the next; thus the “tMD staffing curve” as shown in Figure 2 does not depict margins' 95% CIs.¹⁵ Instead, a separate Stata calculation (“contrast”) is appropriate.¹⁵ By using the contrast calculation and adjusting for all other covariates, there is a significant (6.5 minute) gain in staffing the ED with at least 22 physicians per shift when compared with having fewer than 22 physicians in the ED. This information is portrayed in Table 7, which also demonstrates the tMD prolongation associated with advancing time of day (day shift vs. evening shift vs. night shift).

Figure 2.

Decreasing time to physician associated with assuring physician staffing at n = 22 per shift.

The significant prolongations in tMD that were observed in moving from the day to the evening shift, as well as the further increases in tMD values demonstrated with movement from the evening to the night shift, were not unexpected (see the Discussion section). However, the sizeable magnitude of these changes, as depicted in Table 7 and Figure 3, were somewhat unexpected. This is because the shift-associated tMD prolongation remained operationally and statistically significant even after adjusting (as depicted in the table and figure) for critical covariates such as shift census and physician staffing. The results with regard to shift time of day and tMD were demonstrative of the fact that even though the linear regression model for tMD was acceptable, there was much – in fact, nearly half – variance in tMD values for which the model did not directly account (i.e., the factors related to “shift”, other than census, physician staffing, and other Table 3 covariates).

Figure 3.

Increasing time to physician associated with different ED shifts.

Discussion

Analysis of operations data includes focus on variables with impact on the medical outcome as well as economic and patient-satisfaction outcomes.^{1,16,17,18,19,20} One factor of particular importance is the time interval between the initial patient arrival at the ED and initial patient contact with a physician (tMD). Faster tMD indicates a higher likelihood of optima patient safety and care quality (due to more-rapid physician evaluation); patients who are attended to more quickly are also more satisfied and less likely to leave the ED before their evaluation.^6,7,8,9

There are a variety of factors that can potentially impact tMD. This study intended to adjust for a number of these known factors (e.g. patient census) while searching for other parameters that impact tMD at a single facility. The overarching goal was to identify parameters previously under-appreciated as mediators of prolonged tMD, in order to identify potential areas for operations interventions to improve tMD. A secondary study goal was to demonstrate the utility of predictive marginal probability analysis in ED operations. Demonstration of the application and assessment of the analysis tool was an important goal of the study, and meeting this goal is the reason for the detailed presentation of methodology and results.

It is important to emphasize that tMD is but one measure of ED operational performance. As noted previously, and as pointed out by numerous authors, there is a breadth of factors that should be considered as being at least as important as tMD.^21,22 This study's emphasis on tMD is not meant to imply that other (unmeasured) factors are not important.

The discussion of the study's findings and potential uses is divided into subcategories. First, the descriptive and univariate results (including EDAD data validation) are addressed. Multivariate results interpretation follows in the next section. Introduction of the application of predictive marginal probability analysis comprises the third section. The final section is an overview of some of the limitations that should be considered before drawing conclusions from this analysis.

Descriptive and univariate results

The first set of results were obtained from the descriptive and univariate calculations. The data in Table 1 demonstrates a busy ED with a reasonably high “capture” of paper charts (over 90% on an average shift) into the EDAD. Times to triage and initial physician evaluation were also reasonably fast when compared to existing operations literature.^3,8

The study ED was fairly typical in terms of operational performance in comparison to EDs' times worldwide. In the USA, for instance, the federal government Centers for Disease Control and Prevention (http://www.cdc.gov/nchs/ahcd.htm) provides a summary estimate for average tMD of 30 minutes. This overall average of about half-an-hour included community hospitals and major urban centers, and is within the range of the time identified in our center.

Times to physician have also been assessed in other countries, again with findings that are generally in line with those identified in the current study. In a setting in Brussels, for instance, a unique approach to physician triage was associated with times to physician of 48 minutes;²³ this time is similar to the tMD reported in the current study. The times to triage in our center, which like in other centers, are somewhat dependent on patient acuity levels, are in line with both reported triage times and triage-time goals from the literature from Europe, Australia, and Canada.²⁴

The proportion of patients admitted outside the ED represents only those patients who were admitted to other services that are not administered by the Department of Emergency Medicine. As previously noted, the ED operates a short-stay unit (SSU) that handles virtually all acute medicine admissions up to 2–3 days; the SSU admissions often run to 4–5 days. The “admission” rate of 4% in Table 1 does not include any of these “ED admissions” and should not be compared with other EDs.

Table 1 also outlines an ED that is dominated by the presence of males, by the relatively young, and by those who are not natives of the country in which the ED is located. These findings mirror the demographics of the country where the ED is located. Similarly, the proportion of patients arriving by ambulance is relatively low; this is consistent with the fact that at the study ED, many cases come from the various worksites around the city and from those worksites, buses come daily bringing many patients (literally, dozens) with minor illness and injury who would likely use ambulance services in other settings.

Another characteristic of the country in which the ED is located is the virtual absence of longitudinal care for the overwhelming majority of ED patients. Therefore, it is not surprising that the numbers of referrals from outside physicians are low. As with some of the other covariates, the utility of including this parameter in the study ED modeling was not for absolute comparison against other centers, but rather for assessing for possible association with the dependent variable; as no such association was identified, the covariate was dropped from further consideration.

The chief complaint information was included only to control for case mix. The complaints as recorded, have been recorded the same way for years. Thus, the data should be useful in their intended purpose of coarsely adjusting for diagnosis.

Multivariate results interpretation

The multivariate results included many expected findings. For example, the positive association between patient census and tMD was not surprising. Similarly, due to the direct contribution of time to triage to the tMD parameter, this covariate was also expected to be significant and required for inclusion in a tMD model.

The number of ambulance cases was a significant contributor to the tMD model in this study dataset. This is likely because of the inability of the study data to otherwise control for acuity. The higher proportion of ambulance cases broadly corresponds to fewer walk-in cases and therefore higher acuity. It is thus an expected finding that the ambulance case proportion would be positively associated with tMD – the more the number of higher-acuity patients, the faster those patients will be seen by physicians in the study ED as in most others. The a priori intent was to include ambulance proportion data solely for acuity adjustment. The chief complaint findings (i.e., patients with more straightforward complaints in the “gastrointestinal” category) were also included solely for acuity and resource-requirement adjustment. These findings are not intended to serve as a basis for any generalizable conclusions.

In addition to the main parameter of interest, the number of physicians, the data demonstrated that shift time of day (i.e., day, evening, or night) was strongly associated with tMD even after adjusting for census, physician coverage levels, and other covariates outlined in Table 3.

The shift-associated prolongation in tMD seems likely due to overall boarding and occupancy rates in the ED. These parameters, as well as the related covariate of hospital occupancy, have been demonstrated previously to be major drivers of ED operations performance.²⁰ As is the case in many EDs, the clinical areas tend to be less full at the beginning of the day and with increasing lengths of stay the ED becomes “more fully” independent of actual numbers of patients arriving to be seen. This finding will be explored in a future analysis at the study center, as the ability to track hospital occupancy is integrated into the EDAD.

Marginal analysis findings and utility

The utility of marginal analysis for the current dataset was that the technique could be used to assess the impact of changing one variable while adjusting for (integrating over) the values of the other model covariates.¹⁵ In this study, after the initial multivariate model generation, the focus of the analysis turned to physician staffing and shift time (i.e. day, evening, or night). Predictive marginal probability analysis was thus used to model estimated values for tMD for specified values of these parameters, while holding all other covariates the same.

The marginal analysis results demonstrated that there were operationally and statistically significant “costs” for the failure of maintaining physician staffing at the level of at least 22 on-duty doctors. The study data does not make a case that an exact number of 22 non-consultant doctors is an unwavering requirement for the HGH ED. Rather, the calculated number represents the best estimate of the minimum non-consultant physician staffing needed at our ED. (Incorporation of the consultant numbers would not change the study calculations, other than simply boosting the non-consultant numbers by the constant number of on-duty consultants.)

The clearly illustrated results of marginal analysis allowed for an evidence-based discussion with the study center doctors, who were easily able to understand the importance of optimizing staffing. The information obtained from the marginal model helped demonstrate to ED administration that our staffing needed to be increased to at least 22 physicians before there are demonstrable operational ramifications due to short-staffing. Even though the calculations were necessarily imperfect – not including nurses or other non-physician staff, whose numbers were not available for this study – the data were sufficient to make a clear case for minimum physician levels.

The large volume and the limited bed resources of the HGH ED mean that patients are essentially always queued and needing to be seen, thus minimizing the circadian impact of patient presentation differential. At 0300, for instance, the ED is still working through the “bolus” of patients who presented hours earlier.

The marginal analysis results for the time-of-day parameter (i.e. day, evening, or night shift) have spurred major efforts to identify which particular aspects of the later shifts are responsible for – and amenable to mitigation of – tMD prolongation. In this respect, marginal analysis has enabled communication with ED leadership and administration to try and identify (and fix) concrete factors that can be addressed to improve operations.

There are substantial risks to overextrapolation of the results (as discussed in the next section). However, the marginal analysis results were easily understood and useful in making the case for a need to change something to address prolonged tMD as shift times progressed from the day to the evening to the night.

Limitations

The current project's intent was to identify factors amenable to ED operational adjustment, with the goal of improving tMD. The plan was to first identify tMD-prolonging parameters and then to decide upon adjustments to those parameters and so to achieve reduction in tMD. After the implementation of operational adjustments (e.g. increased physician coverage), a planned follow-up analysis will be carried out using the same methodology, assessing the same tMD endpoint to adjudicate the success of ED operational changes.

The overall results suggest that the data and analytic methodology are suitable for the limited application of the study's intent. However, numerous limitations preclude overextension of the results or widespread generalization of the findings. This section reviews some of the major study shortcomings.

First, the study assessed data from an administrative database, the EDAD. Administrative databases are historically useful for ED operations analysis, but the use of these types of datasets come with well-characterized strengths and weaknesses.¹³

In the EDAD, there are advantages of objectivity of data entry, as well as accuracy of information that is optimized by the use of fairly simple and objective variables. The time-stamp data are nowhere near infallible, but there are few reasons to think that the data are biased; inaccuracies are most likely spread evenly across the independent variables assessed in this study. This assumption is, however, just that – an assumption that cannot be tested, given the available information. Thus, the hand-entered time data are a source of potential error and a study limitation.

There are other disadvantages that accrue from the simplicity of hand-entered time information. Data completeness and accuracy are obtained at the cost of reducing the overall amount of data that are entered into the EDAD. For example, diagnostic information is collapsed into broad categories and during the study, there was a lack of any direct measure of patient acuity (e.g. triage scoring and vital signs).

The study only assessed cases from a single month. There are advantages to this. Namely, the staff themselves did not have appreciable turnover and there was little change in external factors (e.g. no holidays or weather changes) that could affect ED operations. However, there remains the possibility that the results cannot be accurately generalized to other months of the year.

The chief complaint (CC) categories of EDAD are a noteworthy limitation to the study. These categories have been in use for an extended period of time and the entry of data into these categories has been consistent over the years. However, the CC data are necessarily imprecise, and in some cases there could be argument that a given case could fall under multiple CC categories. The CC data problem is minimized for this study, due to the fact that CC is incorporated into this analytical modeling solely to account for the case mix; no inferences should be drawn with regard to the actual corresponding diagnoses seen in the ED. As long as the same approach to categorizing CCs is used over time, the incorporation of these data into this study and in follow-up analyses is a reasonable method for coarse adjustment for diagnosis.

Another study shortcoming is the missed charts (i.e. those records that were not abstracted into the EDAD). While for an average EDAD 91.4% capture rate for paper charts in an ED seeing nearly 40,000 cases per month represents acceptable performance, it would be preferable to have 100% data capture. Familiarity with the EDAD led the investigators to believe that there is no systematic tendency for certain case types (e.g. admitted cases) to “miss” entry into the EDAD. However, in the absence of 100% data capture, there is always the risk of selection bias. The study's routes to address this shortcoming demonstrated no issues, but the possibility of selection bias always remains.

Previous preliminary work has suggested that goals for overall wait times should be set depending on the triage level; 45 minutes (for mid-level cases) or 60 minutes (for lower-level cases).⁸ Since the goals of many previous ED tMD studies have been stratified by various levels of triage acuity, the inability of the current data to adjust for a triage score is a substantial limitation. The study ED is moving to a new EMR, which will address many of the shortcomings of data entry into the EDAD. All ED nurses have now been trained in administration of the Canadian Triage and Acuity Score (CTAS), which will be available when the new EMR is implemented in 2016. The fact that the study center's times to triage and tMD may not fit other centers' times is a potential limitation depending on the comparator center. There are some locations (e.g. Ontario, Canada),²⁵ that have reported shorter tMD due to concentrated efforts, but the times identified in this analysis are consistent with the broader ED literature from around the world.²⁴

The study period EDAD does not account for the potential of brief pre-triage physician–patient interactions. Such interactions occur for only a small minority (well under a few percent) of ED cases at the study center, and are centered on physicians' attempts to ensure patient safety in an often-overloaded waiting room. At the time of the study – but no longer – these interactions were neither documented nor included in EDAD time-to-physician calculations. This practice has now changed and since ED operations performance can be significantly improved by placement of a physician or advanced practice provider at triage, the physician-out-front plan may be part of future operational (and tMD improvements).^19,26,27

Another study limitation is the focus on just one part of the ED operations timeline. Despite the undoubted importance of tMD, it would be ideal to be able to reliably assess the entire timeline from initial patient presentation to patients' leaving the ED. Such an approach of assessing the overall ED “throughput time” has been used in literature that demonstrates time reduction advantages with respect to various operational endpoints.¹⁹ For these study data, though, the ability to focus on the whole picture of ED length of stay was not within the capabilities of the EDAD. This is in part because of the fact that the ED itself functions as a “mini-hospital” with as many as 400–500 cases per month being “admitted to the ED” and therefore staying as long as five days under the care of the ED team. Rather than have data that were of questionable applicability to the design of ED operations, the investigators chose to follow the narrow time window of tMD. This maximized internal validity, but at a cost of external validity. Fortunately, there is evidence that focusing solely on tMD – while providing only a part of the overall picture – yields results of substantial operational importance.²⁶

As another methodological limitation, the study focused on just one “endpoint” parameter that was not clinical. The time to physician is important, but the study's impact is limited by failure to focus on any patient-related endpoints (e.g. errors due to prolonged wait times) or directly relevant operational endpoints (e.g. overall throughput time).

Finally, an operations study would ideally focus on patient-specific information. This level of granularity, while theoretically useful, would not be possible, given the data available for EDAD analysis. Therefore, the shift-based analysis was used as a surrogate, though imperfect, marker for operations performance.

Conclusions

This study provides clear evidence of a statistically and operationally significant association between time to physician evaluation and ED physician staffing levels as well as shift time of day despite the limitations as noted. In framing the results of this study's complicated analysis in the simplest terms, the data proves that there is indeed a level of physician staffing below which operations are threatened. The intuitive sense that there must be a minimum number of physicians on duty, to enable a minimum functionality of the ED, is demonstrated with data in this study. The use of marginal analysis facilitated practical interpretation of multivariate modeling, and this information contributed to a decision to modify physician coverage to address problems identified in the analysis.

The study methodology is presented in detail herein to allow for other operations' administrators to reproduce these methods locally. One of the most important conclusions of the current analysis is that the tools described in this study were in fact quite useful to drive change in the study hospital; these tools may also be useful in other settings. It is hoped that the detailed methodology as presented in this report can be of use to readers wishing to reproduce this analytical approach.

Follow-up reassessment in the coming year will focus on whether the operational implementation (e.g. changes in physician coverage) mitigated tMD problems.