Predicting emergency department admissions using a machine-learning algorithm: a proof of concept with retrospective study

Abstract

Introduction

Overcrowding in emergency departments (ED) is a major public health issue, leading to increased workload and exhaustion for the teams, resulting poor outcomes. It seems interesting to be able to predict the admissions of patients in the ED.

Aim

The main objective of this study was to build and test a prediction tool for ED admissions using artificial intelligence.

Methods

We performed a retrospective multicenter study in two French ED from January 1st, 2010 to December 31st, 2019.We tested several machine learning algorithms and compared the results.

Results

The arrival and departure times from the ED of 2 hospitals were collected from all consultations during the study period, then grouped into 87 600 one-hour slots. Through the development of two models (one for each location), we found that the XGBoost method with hyperparameter adaptations was the best, suggesting that the studied data could be predicted (mean absolute error) at 2.63 for Hospital 1 and 2.64 for Hospital 2).

Conclusions

This study ran the construction and validation of a powerful tool for predicting ED admissions in 2 French ED. This type of tool should be integrated into the overall organization of an ED, to optimize the resources of healthcare professionals.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12873-024-01141-4.

Introduction

The overcrowding of emergency departments (ED) is a major issue [1–3] for department-level management. It is mostly the result of a mismatch between a growing demand for ED consultations and a decrease in the overall resources of these services. On the one hand, the number of ED consultations is constantly increasing, with a twofold raise during the last decade in France and the United States [1, 4]. On the other hand, we observed that internal factors of the ED organization led to an increase in passage duration, especially the inadequacy between the staff and the number of patients. For example, in the United States, there is an average of one nurse for every four patients and one doctor for every ten patients, with extended medical care delay [5]. In addition, there are downstream problems with a permanent reduction in the number of hospital beds worldwide, resulting in delays in patient care due to longer boarding times [6].

This overcrowding is harmful on several levels. It increases the workload drudgery for the teams, leading to a decrease in the quality of care, particularly by increasing the risk of medication errors [5]. It also causes additional health care costs, estimated at $6,8 millions over three years in the United States. Finally, in an overcrowded ED, treatments are on average 30 min late, which has a considerable impact for acute care [7]. It is thus a risk factor for morbidity and increased mortality with approximately thirteen preventable deaths per emergency department per year [5, 8]. The impact of waiting times is also significant for elderly patients, resulting in excess mortality [9]. Some teams are trying to reduce this waiting time in our EDs [10]. However, to increase the effectiveness of these interventions, i to optimize resources, particularly in terms of manpower, it would be interesting to be able to predict ED attendance. Some teams have already identified some recurrences of overcrowding and routinely use these trends based mainly on calendar data [11]. At the country level, there is an increased risk of overcrowding for certain seasons, certain days of the week and certain times of the day [11]. It seems that the ED attendance is associated with the weather or the road traffic [12–14]. The number of factors influencing the number of consultations seems to be large enough to prevent all of them from being taken into account using simple statistical tools. Faced with this complex and multifactorial aspect, artificial intelligence (AI) tools such as machine learning or deep learning have started to show promising results, but still insufficient to be used in routine [15–19]. The models used and the variable selection could be further increased in order to provide a better prediction. AI is also playing an important role in ED prediction [20].

Some researchers have focused on the prediction of waiting times in ED. This prediction can be useful for patients. But, current systems have largely included rolling average estimates or median historical waiting times [21]. For example in prediction of waiting times, Q-lasso method reports higher accuracy than the rolling average estimator [22], or the efficiency of a quantile regression model [23].

We believe that predicting the number of patients could enable us to better size the number of healthcarers present in the ED and regulate the flow [24]. In addition, interdisciplinary efforts could potentially improve prediction performance [25]. That’s why we’ve brought together emergency and AI specialists.

The primary aim of this study was to build a performant model to predict the number of admissions per hour to the ED. The secondary aim was to predict the number of patients in attendance per hour in the ED.

Methods

Data collection

We conducted a retrospective multicenter study in two ED in France (Mercy and Bel Air Hospitals) and developed a location specific model. All admissions in each of the two services between January 1st, 2010 and December 31st, 2019 were included for analysis. From the extracted data, we constituted two samples per center: (i) a model training sample of each center from 2010 to 2018 included; (ii) a test sample with 2019 data on each center. In order to maintain patient anonymity, data was aggregated at source. The number of admissions per hour, per service and the mean length of stay per hour (average time from arrival in the ED to discharge) and its standard deviation were extracted.

Output

The primary endpoint was the number of admissions per hour to each ED (variable to be predicted). The secondary endpoint was the number of patients attending in each ED per hour (variable to be predicted).

Input

We extracted, from the history of the previous 4 weeks, 672 explanatory variables to predict the number of admissions per hour (4 weeks x 7 days x 24 h). Concerning the explanatory data of the number of patients attending per hour, the historical data also included: the average duration of ED consultations by pathology typology (abdominal-pelvic pathologies, cardiovascular, traumatology, weakness and others). In addition, we collected environmental data which could have an impact on ED attendance over 4 weeks: day of the week, meteorological data (temperatures, atmospheric pressure, cloudiness, humidity, precipitation, wind speed, and horizontal visibility), road traffic data, and epidemiological data from the epidemic disease monitoring network (influenza, diarrhea, varicella.), as well as calendar data such as vacations, holidays and celebrations, and ephemeris data [26–30]. In total, these data provided 712 explanatory variables to predict a datum (Fig. 1). Detailed description of variable were in supplementary data (Table S1).

Fig. 1.

Predicted variable and explanatory variables

Statistical analysis

Algorithm testing methodology

With the objective of finding the best model for each hospital, we used and tested different types of machine learning algorithms (Table 1) and compared the results for each hospital [31–34].

Table 1.

Tested algorithms

Algorithms	Types	References
Random forest	ensemble decision tree method	Ho, Tin Kam, « Random Decision Forests » (1995) [35]; Breiman, L. “Random Forests” (2001) [36]
XGBoost	Ensemble decision tree method	Chen, T. et al XGBoost: A Scalable Tree Boosting System (2016) [31]
LightGBM	ensemble decision tree method	Guolin Ke et al et al. LightGBM: a highly efficient gradient boosting decision tree (2017) [32]
Lasso	regression penalized	Robert Tibshirani, « Regression shrinkage and selection via the lasso » (1996) [37]

There are two types of missing data, those related to the target variable (number of patients) and those related to the features variables. For the target variable, there is no missing data. Slots without any patient are recorded with the value 0. The number of missing data related to features variables is very low (from 0,0% to 0,01%). Since the weather variables are continuous, and we get access to several weather channels, we selected data from three different weather channels close to the hospital and performed a linear extrapolation in order to fill the missing data. The other categories of features variables (mainly those related to traffic) were filled in by replication of the last known value, in order to standardize the dataset for all methods of the algorithms. Normalization for comparison purposes was performed for quantitative variables, and target encoding for qualitative variables [38] In a first comparison, we used the default values of the hyperparameters [39]. Then we optimized the hyperparameters on the selection of explanatory variables.

The hyperparameters are certain weights or values that determine the learning process of a machine learning algorithm. For instance, hyperparameters for XGBoost are divided into 4 classes:

• General parameters that guide the overall functioning of the model (number of parallel threads used to run the model, the maximum depth of a tree etc.).

• Booster parameters are 2 types (tree booster and linear booster). For instance, hyperparameter for tree booster are: the learning rate, the maximum delta step we allow each tree’s weight estimation to be etc. In the paper, maximum depth was fitted by systematically testing integers from 2 to 15.

• Learning task parameters are used to define the optimization objective i.e., the metric to be calculated at each step (e.g., objective could be regression with squared loss), the metric to be used for data validation (e.g., root mean square error),

• Command line parameters are only used in the console version.

Learning methods

Once the predictive model was chosen (the extreme gradient boosting), it was necessary to fix its hyperparameters and to select the explanatory variables in order to avoid overlearning. The risk of overlearning in time series models using XGBoost was mitigated through time series-specific cross-validation, ensuring the model did not have access to future data during training. Additionally, early stopping was employed to halt model training when performance on the validation set began to degrade. Limiting the maximum tree depth (max_depth) and reducing the number of estimators (n_estimators) further helped to prevent the model from learning overly specific details from the training data, thereby improving its generalization capacity.

The hyperparameter used was the maximum depth at which to stop the learning. We also conducted sensitivity analysis by modifying the other hyperparameters (initially fixed at the recommended default values), without any significant performance improvement. We then looked at how the learning evolved with the number of explanatory variables. Finally, to obtain the best learning, we performed a gluttonous approach. We then considered only the “best” of the explanatory variables (the one that best discriminated), and performed learning for a maximum depth ranging from 2 to 15. The best mean absolute error (MAE) learning score obtained corresponded to our reference, which should be lowered as new variables were added. We took the best of the remaining variables for the second explanatory variable, performed as many learnings as there were maximum depths between 2 and 15, and looked to see if we were thus able to improve the best score obtained with a single variable. In this case, we reproduced the process by starting with the two selected variables and by adding the best candidate (then tested on depths from 2 to 15), and if not we discarded this second variable and tested the next most promising one. At each iteration, we tested a new variable, which was added to the list provided to the model if there was a maximum depth that improved the current score. On the one hand, we saw the local fluctuations resulting from the addition of a new variable: some were improving the previous score, others degraded it. We then noticed that the MAE and the root mean squared error (RMSE) decrease during the iterations (tests of new variables) until reaching a plateau around 125, then an increase which indicates overlearning.

Descriptive analysis

Descriptive statistics were expressed as mean and median with standard deviation (SD). For the validation of the best performing models, we calculated the MAE and the RMSE, in order to have reference values for comparison.

Ethics

This retrospective study contains no health data and all data were aggregated. That’s why, this study did not require specific authorization according to the current French regulations. This research is outside the scope of clinical research and does not require an ethics committee or asked for consent (French public health code (art. L1110-1 to L6441-1).

Results

Characteristics of the admissions

ED arrival and departure times for Mercy and Bel Air hospitals were collected from January 1, 2010 to December 31, 2019, and then aggregated into 87 600 one-hour slots. At both locations, with the exception of the years 2012 and 2017, when slight decreases are observed, the year-to-date number of patients per hour has been steadily increasing, from around 47,000 to around 83,000 over the period 2010–2019 for Mercy, and from around 48,000 to around 63,000 for Bel Air (Fig. 2).

Fig. 2.

Annual increase in cumulative hourly emergency department patients between 2010 and 2019 at Mercy and Bel-Air hospitals

At the Mercy hospital, there was an average of 7.9 patient arrivals per hour, with a standard deviation of 4.4 patients, for a minimum of 1 patient/hour, and a peak of 36 patients At the Bel Air hospital, the average number of arrivals was slightly lower, with 7.1 patients/hour, but with a slightly lower standard deviation (around 4). The average number of patients in the ED was 15.9 and 13.3 per hour respectively for Mercy and Bel Air. All results were in Table 2. The general trend in the number of patients per hour over the past 10 years was upward in both cases, but with a more marked increase in the case of Mercy (Figure S1 and S2). These time series also showed the large standard deviation, as well as possible annual seasonality fluctuations.

Table 2.

Summary of data by hospital

Hospital	Number of observations	Mean (patients/hour)	Median (patients/hour)	Standard deviation (patients/hour)	Minimum (patients/hour)	Maximum (patients/hour)
Arrivals per hour at mercy	87,600	7,9	7	4,4	1	36
Patients attending per hour at mercy	87,600	15,9	14	9,5	1	105
Arrivals per hour at bel air	87,600	7,1	7	4	1	34
Patients attending per hour at bel air	87,600	13,3	12	8,4	1	96

The average number of arrivals to the ED varied greatly depending on the time of year. In August, the average number of arrivalswas ∼ 7 patients/hr for Mercy, and ∼ 6.5 patients hr for Bel-Air. (Figure S3). Early year, late spring and late year periods were associated with a significant increase of patient arrivals. However, the monthly profile was almost constant, except for a marked drop in August for both hospitals, showing the absence of seasonality. The weekly data (Figures S4 and S5) showed a greater load at the end of the day on Mondays and on weekends. In the middle of the week, attendancewas lower. Finally, the daily profilewas similar for the two ED, with an ascending phase from 7:00 am to 8:00 pm (with a slight plateau around 4:00 pm), followed by a descending phase (Figures S6-9).

Baselines

Before studying advanced machine learning models, we must first be sure that basic approaches do not provide reliable predictions, and that these predictions are much worse than those obtained by artificial intelligence. First of all, it can be only cyclic changes, the signal would correspond to its seasonal part in a decomposition of the same name. To check that this is not the case, we approximated the signal by its seasonal component (see Figure S10), to see how well such a predictor approximates the signal. The MAE obtained is 5.89 for Bel Air (6.73 for Mercy) which, as we will see, is a much larger error than what will be obtained using machine learning. A less naive baseline consists in having a time series approach to prediction, using recognized techniques such as autoregressive integrated moving average (ARIMA). This raises the question of whether such models would not produce equally good results without having to worry about managing such a collection of features. Indeed, there is a strong correlation between the series of patients and its version shifted by one time unit: the signal is strongly auto-correlated, and using the number of patients at time t, t-1 seems a good idea to predict the number at time t + 1. After calculation, the best AR model (with a lag of 4 days) leads to an MAE of 4.25 for Mercy (3.45 for Bel Air), and the ARIMA model does not do much better (3.96 for Mercy, 3.11 for Bel Air), for an optimal choice of parameters obtained from the autocorrelation and partial autocorrelation graphs in the context of an upward trend, of (2,1,2) We will see later that artificial intelligence approaches allow us to obtain better results. Finally, let us note that the optimal hyperparameters are equal for both hospitals: they may be different, but they have basically the same dynamics, because they are always patients going to the ED. As the same causes produce the same effects, we can hope that the method can be extended to other hospitals.

Learning

The MAE started by decreasing for Bel Air, up to a maximum depth of 5–7, and then started to increase sharply, independently of the number of explanatory variables: the models became too complex in view of the signal to be predicted, and we were clearly entering a phase of over-learning. On the other hand, the RMSE, measured obvious mistakes, and behaved in much the same way: first it decreased and then stagnated, before starting to increase strongly. In both cases, as well as for Mercy, it could be inferred that an optimal value for this hyperparameter was around 5–6. Changing the choice of hyperparameters did not result in a significant improvement in learning (Figures S11, S12). We found that the optimal was obtained for a max depth of 6, and that in this case the MAE decreased slightly but continuously with the number of variables. On the contrary, the worst results were obtained for a max depth greater or equal to 10, and the increase in the number of explanatory variables did not improve the learning.

Validation and selection of the best performing algorithm and model building

The dataset has been divided in two parts: 80% for learning (“training dataset”) and 20% for validation. The simulations showed that beyond 4 weeks, the history is less important for the prediction. The performance of the predictions evaluated on 80 h of the test set was illustrated in Fig. 3 (note that there is no seasonality anymore, as the hours of the test set are randomly drawn).

Fig. 3.

Predictions versus reality for all the tests

Overall, the models performed well, with a slight lead for the XGBoost model. This was explained on the one hand by the real predictability, to some degree, of the quantity of interest, and on the other hand by the choice made both in terms of explanatory variables and predictive models (and their parameters). Figure 4 showed the results for the first 6 days of the second week of 2019. Seasonality was found in this case. The errors, in the range of one to a few patients per hour, were small in the face of the variability and signal intensity.

Fig. 4.

Predictions versus reality on the test set, Mercy and Bel Air hospitals over the first 6 days of the second week of 2019 (x-axis hours and y-axis number of patients in the emergency department)

Performance was measured over the entire test set in terms of absolute error mean and square root quadratic error mean. In total, the most accurate prediction was obtained with XGBoost with hyperparameter optimization allowing a MAE of 2.63 for Mercy and 2.64 for Bel-Air (Table 3), allowing us to retain this algorithm as the best performing for both models. We use repeated k-fold cross-validation to validate the model. The notebook of our algorithms is available on GitHub to ensure the reproducibility of the results (available on: https://github.com/extome-ai/predicting_emergency_department_crowding).

Table 3.

Scores of the different prediction algorithms for the two models (ARIMA = autoregressive integrated moving average;MAE = Mean Absolute Error; RMSE = Mean Square Error)

Algorithms	MERCY MODEL		BEL-AIR MODEL
	MAE	RMSE	MAE	RMSE
Mean	11.92	15.06	12.13	15.20
Mean / hour	9.34	12.22	8.70	11.65
AR	4.25	5.65	3.45	4.55
ARIMA	3.96	5.39	3,11	4,13
Lasso	2.92	3.82	2.90	3.80
XGBoost	2.74	3.68	2.76	3.74
Random forests	2.71	3.62	2.73	3.67
LightGBM	2.66	3.56	2.66	3.59
XGBoost + hyperparameters selection	2.63	3.51	2.64	3.59

Discussion

Artificial intelligence, in particular by increasing the computing power and the number of combinations, has opened up many tracks of work for clinician-researchers in emergency medicine: triage tools, reducing medical errors [40–44]. For the first time to our knowledge, using a retrospective multicenter study, we were able to build several prediction models for 2 French ED centers admissions by selecting the most efficient one for each hospital and achieve a robust proof of concept. Our prediction is superior to other existing models described in the literature, with a MAE of 2.63 for the first model (Mercy) and 2.64 for the second location (Bel Air), using the XGBoost algorithm with hyperparameter optimization. In predictive models of ED severity and care consumption, Barak-Corren et al. have used XG-BOOST et showed that site-specific customization is a key driver of predictive model performance [45]. As in our study, the XGBoost model achieved its best performance using the uniform and site-specific approaches (AUC = 0.9–0.93) [45]. To address the generalizability concerns, our strategy involves fine-tuning models to the specific data of each new hospital. This includes the integration of additional context-specific variables, such as air quality, location, access to public transportation etc., which may influence patient outcomes. By tailoring the model to these local factors, we enhance its ability to perform effectively across varied healthcare environments.

Indeed, Berg J.et al. did a retrospective study and used an automatic prediction algorithm by exponential smoothing to predict the number of admissions per month with an absolute percentage error mean of 4.8 [15]. Despite this encouraging result, the error still seems too large and the monthly granularity does not seem precise enough to organize the EDs on this model alone. Several teams have used the ARIMA system designed for the analysis and prediction of time series data [16–19]. In particular Kadri et al. who presented a MAE of 3,79 [17]. The ARIMA system alone seems to provide an accuracy of 85% and when used with other tools such as a Fournier hybrid series, a performance up to 91.2% [16–19]. In addition to results superior to those found in the literature, the efficiency of the algorithm developed on two different datasets makes its highly probable to export to other hospitals. These results suggest that overly complex models do not necessarily improve performance with relatively low-dimensional ED data. We also find results of other authors on other types of predictive models for ED (hospitalization, or clinical decision support) suggest that models that are too complex perform less well [46].

An accurate prediction is interesting to allow the organization of EDs and to limit the risks of overcrowding. Today, hospital directors with an ED are advised to use tools to plan certain activities in order to better manage ED resources and adapt them to the flow [47]. Until now, flow predictions at the EDs have been made with multiple linear regression models based on calendar data allowing a flow prediction with an error margin of 11%, notably because taking into account special days such as public holidays has further refined the predictions [12]. Weather conditions are a better predictor than the day of the year for all trauma admissions allowing an accuracy prediction of 95% [13]. Finally, taking into account road traffic as a reflection of human activity was essential to bring power and accuracy to predictions [14]. Nevertheless, if these elements have a real impact on the ED attending, used separately they still do not provide sufficient predictive quality to be used in daily practice.

While predictive tools such as the one we have proposed could provide data for the organization of services, they do not solve the problem of overworked caregivers and the shortage of healthcare workers [48]. The organization of an ED is complex and multifactorial; in addition, there is probably no ideal way of functioning that would allow an implementation to all medical departments. This is why efficient and accurate flow prediction tools should be integrated into a global reflection at local, regional and national levels [48]. Adapting resources and thinking about better team agility seems to be a key point in the proposal made by flow prediction. However, the main challenge to encouraging this approach, is not easy today in the hospital system. In particular, by proposing that staff be more present during certain types of hours or on certain days of the week (weekends),, where we sometimes encounter significant resistance to change. The financing of these additional positions in certain time slots must also be weighed against the reduction of positions in other slots and the potential overall benefit to the quality of care. v Finally, a future possibility of our prediction tool would be the precise prediction of unscheduled hospitalizations. This axis would allow an adaptability of the hospital and an early anticipation of the needs in daily beds. This lever would allow an action on the downstream which is also a strong response to the congestion of the ED [49]. In fact, some authors have used A.I. to predict the need for hospitalization upon admission to the ED [50]. This would be a relevant development for our tool, to increase its impact for hospital managers. On all these points, an experiment on several emergency centers would be relevant. It could consist in studying the organization set up by emergency departments, guided by the tool’s prediction of the number of passages in the short (hour, day) and medium term (week, month). Emergency departments can then reorganize themselves with advance knowledge of fluctuations in throughput. They could, for example, adjust the size of their medical or paramedical staff, create advanced medical posts, or take preventive action on certain events that generate specific flows. This study would enable us to select organizational models that are more effective than others, depending on specific flows.

Our study has a number of limitations. First, to ensure anonymity and to avoid having to obtain patients’ consent, we worked with aggregated data; this level of precision allowed us to meet the main objective of the study but did not allow us to predict the types of ED consultations. This level of granularity could be relevant to anticipate skills in terms of technical facilities (specialists, imaging, specialty beds, etc.). A prospective study will make it possible to refine this point. Secondly, these data concern only two different emergency departments in the same region. However, the large volume of patients studied and the quality of prediction obtained by creating each model allows us to imagine a possible good extrapolation to other centers. Thirdly we have limited the choice of algorithms. XGBoost and LightGBM were chosen because they are very recent methods and consistently provide the best predictions in comparative studies. For the past five years, they have also been almost consistently the finalists in Kaggle prediction competitions. They are often tied with deep learning methods based on LSTM or CNN, but have a much more reasonable learning time and hardware requirements: hospitals that would like to use our proposal will not have to buy expensive GPUs. XGBoost and LightGBM belong to the decision tree ensemble methods, whose ancestor is random forests. So we thought it would be interesting to include these in the study, to see how much better the modern methods did than the old reference. Finally, LASSO was added because it has a reputation for performing well in the presence of large sets of variables. This information was added to the paper. Of course, we could have included even more models, but there would be no end to it, and this is only a first study (which already includes the most promising methods). Finally, before envisaging a use in everyday life, it will be necessary to ask the questions of safety, ethics and prospective clinical validation of this type of tool, in relation to the European regulation of medical devices in force and the recommendations of patient data management [51] The work undertaken must continue in a multidisciplinary manner.

Conclusions

We have developed two predictive models for ED admissions at two hospital locations, performing in an interdisciplinary manner and taking into account public health needs. It is possible to predict emergency admissions and we confirm that an XGBoost model performs well using both uniform and site-specific approaches. This type of tool should be integrated into the overall organization of an emergency department, to optimize the resources of healthcare professionals. A prospective phase of this work will consolidate the results and address the complex issue of ED organization, in particular, by studying the use of this tool in real time, and observing its impact and use by managers. Our tool could also consider including other data to anticipate the need for hospital beds.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Author contributions

C.B. and L.A.V. wrote the main manuscript text and M.A. and C.G. prepared figures. L.C. traduct in english version. All authors reviewed the manuscript.

Funding

This research received no external funding.

Data availability

The data generated or analyzed during this study are available from the corresponding author on reasonable request. Algorithm is on: https://github.com/extome-ai/predicting_emergency_department_crowding.

Declarations

Institutional review board statement

This retrospective study contains no health data and all data were aggregated. That’s why, this study did not require specific authorization according to the current French regulations. This research is outside the scope of clinical research and does not require an ethics committee or asked for consent (French public health code (art. L1110-1 to L6441-1).

Consent for publication

N/A.

Informed consent

This retrospective study contains no health data and all data were aggregated. That’s why, this study did not require specific authorization according to the current French regulations. This research is outside the scope of clinical research and does not require an ethics committee or asked for consent (French public health code (art. L1110-1 to L6441-1).

Conflict of interest

The authors declare no conflict of interest.