A meta-analysis of the diagnostic test accuracy of artificial intelligence predicting emergency department dispositions

Abstract

Background

The rapid advancement of Artificial Intelligence (AI) has led to its widespread application across various domains, showing encouraging outcomes. Many studies have utilized AI to forecast emergency department (ED) disposition, aiming to forecast patient outcomes earlier and to allocate resources better; however, a dearth of comprehensive review literature exists to assess the objective performance standards of these predictive models using quantitative evaluations. This study aims to conduct a meta-analysis to assess the diagnostic accuracy of AI in predicting ED disposition, encompassing admission, critical care, and mortality.

Methods

Multiple databases, including Scopus, Springer, ScienceDirect, PubMed, Wiley, Sage, and Google Scholar, were searched until December 31, 2023, to gather relevant literature. Risk of bias was assessed using the Prediction Model Risk of Bias Assessment Tool. Pooled estimates of sensitivity, specificity, and area under the receiver operating characteristic curve (AUROC) were calculated to evaluate AI’s predictive performance. Sub-group analyses were performed to explore covariates affecting AI predictive model performance.

Results

The study included 88 articles possessed with 117 AI models, among which 39, 45, and 33 models predicted admission, critical care, and mortality, respectively. The reported statistics for sensitivity, specificity, and AUROC represent pooled summary measures derived from the component studies included in this meta-analysis. AI’s summary sensitivity, specificity, and AUROC for predicting admission were 0.81 (95% Confidence Interval [CI] 0.74–0.86), 0.87 (95% CI 0.81–0.91), and 0.87 (95% CI 0.84–0.93), respectively. For critical care, the values were 0.86 (95% CI 0.79–0.91), 0.89 (95% CI 0.83–0.93), and 0.93 (95% CI 0.89–0.95), respectively, and for mortality, they were 0.85 (95% CI 0.80–0.89), 0.94 (95% CI 0.90–0.96), and 0.93 (95% CI 0.89–0.96), respectively. Emergent sample characteristics and AI techniques showed evidence of significant covariates influencing the heterogeneity of AI predictive models for ED disposition.

Conclusions

The meta-analysis indicates promising performance of AI in predicting ED disposition, with certain potential for improvement, especially in sensitivity. Future research could explore advanced AI techniques such as ensemble learning and cross-validation with hyper-parameter tuning to enhance predictive model efficacy.

Trial registration

This systematic review was not registered with PROSPERO or any other similar registry because the review was completed prior to the opportunity for registration, and PROSPERO currently does not accept registrations for reviews that are already completed. We are committed to transparency and have adhered to best practices in systematic review methodology throughout this study.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12911-025-03010-x.

Background

The emergency department (ED) of most hospital facilities serve as the frontline of medical care, its importance remains undeniable. Despite its crucial role, predicting the demand for emergency services is challenging, as it is both random and unpredictable. Compounded by finite medical staffing and resources, sudden influxes of patients can overwhelm the ED, making it difficult to meet medical needs promptly. This scenario often leads to ED crowding, where the demand for emergency services exceeds the ED’s capacity to provide timely care [1]. ED over-crowding has a significant negative impact on overall healthcare quality, such as a negative patient experience [2], reduced care quality [3], and unpleasant staff experiences [4]. Despite the adoption of various ED crowding intervention techniques, including technology-based, physical-based, or flow modification methods [5], ED crowding remains a considerable challenge for most EDs worldwide.

The use of artificial intelligence (AI) has been applied across various fields. Advancements promoted by AI have been especially felt in healthcare, for predictive purposes, to yield promising results. One such application is the utilization of machine learning in predicting ED dispositions [6–10], which has shown significant achievements. The logical next step is to systematically and objectively consolidate those study findings in order to provide reference for both medical practice and academia alike. However, existing reviews on the application of machine learning in predicting ED dispositions maintain certain shortcomings. For instance, most studies only analyze the performance of prediction models through systematic review, lacking more objective quantitative analysis through meta-analysis to consolidate the stated performance of those models. Additionally, there is a lack of comprehensive analysis regarding different ED dispositions (e.g., admission, critical care, and mortality). Dispositions in this study are limited to the outcomes of patients’ visits to the ED, specifically focusing on hospital admission, critical care, and mortality. Diagnoses or presenting conditions (e.g., sepsis) are not included in the scope of this analysis.

Therefore, this study aims to systematically evaluate the diagnostic performance of AI models in predicting key ED dispositions—hospital admission, critical care, and mortality—through a comprehensive meta-analysis. Specifically, the study seeks to: (1) Quantify the overall diagnostic accuracy of AI models in predicting ED dispositions, enabling a clearer understanding of their general capabilities, (2) Identify and analyze covariates (e.g., data characteristics, model types, and study settings) that contribute to heterogeneity in AI performance across studies, and (3) Provide actionable insights and practical recommendations for stakeholders—including clinicians, researchers, and administrators—on how AI applications can be better utilized to improve ED workflows and decision-making processes. The research questions of this study include: (1) What is the performance of AI applications in predicting admission, critical care, and mortality?; and, (2) What covariates can account for the heterogeneity between studies? Given the varied application settings of each predictive model, this meta-analysis is intended to present an overall view of AI applications rather than offering tailored recommendations for individual clinical situations. By synthesizing results from diverse contexts, this meta-analysis aims to highlight general trends, identify strengths and limitations, and outline key areas for future research. While this study offers insights into the overall status of AI applications in predicting ED dispositions, specific applications in particular settings would still require further development to meet unique contextual needs.

The primary contributions of this study are as follows: (1) Providing an objective, quantitative evaluation of AI performance in predicting ED dispositions, helping readers understand the general capabilities, advantages, and limitations of current AI applications in this context. (2) Identifying covariates from different data and technical perspectives that may influence AI performance to offer strategies to enhance the performance of AI predictive models. (3) Compiling and synthesizing existing knowledge on AI predictions in ED dispositions, creating a practical resource to guide clinicians, administrators, and researchers considering AI adoption to improve ED workflows.

Related works

To provide a comprehensive evaluation of the application of AI in predicting ED dispositions, this study adopts both a macro and a micro perspective. The macro perspective focuses on synthesizing findings from existing review studies, which offer a high-level understanding of trends, methodologies, and limitations in the field. Complementing this, the micro perspective examines original studies to provide a granular view of AI model performance, including key metrics such as sensitivity, specificity, and area under the receiver operating characterisitc curve. By integrating both perspectives, this study aims to bridge the gap between broad trends and specific evidence, offering a holistic understanding of AI’s role in predicting ED dispositions.

Macro perspective - Review studies

Previous literature has systematically reviewed and/or meta-analyzed the dispositions of ED patients, thereby contributing to a better understanding of the topic. As such, these studies also indicate areas for further research and improvement. For example, Shung et al. [11] conducted a systematic review analyzing machine learning’s use in predicting outcomes of acute gastrointestinal-bleeding patients, finding an area under the curve of approximately 0.84 for mortality, interventions, or re-bleeding prediction. However, the study lacked meta-analysis, limiting comprehensive evaluation, especially regarding ED disposition. Guo et al. [12] reviewed machine-learning applications in predicting heart failure diagnoses, readmissions, and mortality, affirming its effectiveness. Yet, the review summarized results without comprehensive statistical analyses, particularly regarding heart failure patients.

Kareemi et al. [13] reviewed machine-learning’s diagnostic and prognostic applications in ED patients, showing superior performance but lacking meta-analyses for a more objective assessment. Naemi et al. [14] reviewed studies predicting in-hospital mortality among ED patients using vital signs and noting reporting shortcomings. Despite proposing future directions, it lacked meta-analysis to objectively quantify predictive capabilities. Buttia et al. [15] focused on machine-learning predictions of COVID-19 outcomes, highlighting limitations in model generalizability. But, as noted, it didn’t deeply assess machine-learning performance.

Chen et al. [16] reviewed studies predicting ICU transfers among ED patients, demonstrating promising performance but lacking both a broader perspective and meta-analytic techniques. Issaiy et al. [17] reviewed machine-learning predictions for acute appendicitis, emphasizing high accuracy but lacking meta-analysis for comprehensive assessment. Larburu et al. [18] systematically reviewed studies predicting ED patient hospitalizations, noting logistic regression’s common usage but lacking meta-analytical synthesis. Olender et al. [19] reviewed studies predicting mortality among older adults using machine learning, conducting meta-analyses to quantify predictive abilities. While providing valuable insights into mortality prediction, the review’s focus on mortality and lack of specificity regarding in-hospital mortality limits its comprehensive assessment capability. Zhang et al. [20] systematically reviewed and meta-analyzed studies predicting sepsis patients’ mortality using machine learning, demonstrating superior predictive performance compared to existing scoring systems. Despite its comprehensive analysis, the review’s limitation lies in its focus solely on the combination of sepsis and mortality prediction.

Based on the existing review literature, there are notable areas for improvement in synthesizing studies that apply machine learning to predict ED dispositions. To begin with, there is a lack of comprehensive review studies on ED dispositions. Among the ten reviewed studies, only a few focused on specific aspects such as admission, mortality, or critical care, rather than providing a holistic view. Secondly, the meta-analytical approach is under-utilized, with only a minority of the reviewed studies employing them. Meta-analysis has the potential to provide a more objective evaluation of machine-learning predictive models, benefiting both practitioners and academics alike. Detailed information on existing reviews is shown in Table 1.

Table 1.

Emergency department disposition-related review studies

Source	ED disposition	Database	Analytic strategy	Articles included	Specific disease/condition	Major findings
Shung et al. [11]	Admission and mortality	Embase, Medline, Cochrane, Central, Web of Science, WHO COVID-19 Global Literature on Coronavirus Disease, and Google Scholar	Systematic review	14	Gastrointestinal bleeding	The median AUC for mortality was 0.84, with AI yielding higher AUCs. Machine learning demonstrated superior performance compared to clinical risk scores for mortality in cases of upper gastrointestinal bleeding.
Guo et al. [12]	Mortality	PubmMed	Systematic review	335	Heart failure	Machine learning helps identify heart failure patients and assess their risk for readmission and mortality accurately.
Kareemi et al. [13]	Admission and mortality	Medline, Embase, Central, and CINAHL	Systematic review	23	No	Machine learning might surpass standard care in predicting outcomes for emergency department patients in diverse clinical scenarios.
Naemi et al. [14]	Mortality	PubMed, Scopus, and Embase	Systematic review	15	No	Eight recommendations for future research to enhance the practical implementation of machine learning in various domains.
Buttia et al. [15]	ICU admission, intubation, high-flow nasal therapy, extracorporeal membrane oxygenation, and mortality	Embase, Medline, Cochrane Central, Web of Science, WHO COVID-19 Global Literature on Coronavirus Disease, and Google Scholar	Systematic review	314	COVID-19	Several clinical prognostic models for COVID-19, described in the literature, suffer from limited generalizability and applicability due to unresolved statistical and methodological concerns.
Chen et al. [16]	ICU admission	PubMed, Embase, Cochrane Library, and Web of Science	Systematic review	10	No	Machine learning excels in identifying and predicting critically ill patients in emergency department triage.
Issaiy et al. [17]	ICU admission	PubMed, Embase, Scopus, and Web of Science	Systematic review	29	Acute appendicitis	The artificial neural network exhibited high performance across the majority of cases.
Larburu et al. [18]	Admission	Scopus, PubMed, and Google Scholar	Systematic review	14	No	Artificial intelligence models improve emergency department care and ease healthcare system burdens.
Olender et al. [19]	Mortality	PubMed, Embase, Web of Science, Scopus, and Proquest	Systematic review and meta-analysis	37	Older adults (+ 65)	Machine learning models demonstrate strong discriminatory power in predicting mortality.
Zhang et al. [20]	Mortality	PubMed, Embase, Cochrane Library, and Web of Science	Systematic review and meta-analysis	50	Sepsis	Machine learning methods exhibit notably high accuracy in predicting mortality risk among sepsis patients.

Note: AI = Artificial Intelligence and AUC = Area Under Curve

Micro perspective - Original studies

From a micro perspective, in studies predicting admission disposition, most utilized private datasets, while only a few studies [21–24] developed prediction models using public datasets, such as the National Hospital and Ambulatory Medical Care Survey (NHAMCS) ED data and the Medical Information Mart for Intensive Care IV (MIMIC-IV) ED database. Most studies relied on structured features, with some [25–29] combining structured and unstructured features (e.g., free text), while others [30–32] used only unstructured features. Regarding model validation, the majority employed internal validation, with only a small number of studies [33] using external validation. Studies that applied cross-validation (e.g., K-fold cross-validation) outnumbered those that did not. Additionally, studies using traditional machine learning methods for ED disposition prediction slightly outnumbered those employing deep learning techniques [21–24, 26, 30–33]. A large portion of studies [21, 28, 29, 34–48] adopted ensemble methods for building predictive models.

In studies predicting critical care disposition, research utilizing public datasets [9, 21, 23, 24] remained limited, with NHAMCS and MIMIC-IV being the most commonly used public datasets. These studies predominantly relied on structured features, with only a few [49–57] combining structured and unstructured features, or solely using unstructured features [58]. Similarly, external validation was infrequently employed [51, 52, 59], with most studies relying on internal validation. The number of studies applying cross-validation was greater than the number of studies that did not. Most predictive models were built using traditional machine learning methods, while the use of ensemble methods in this context was less common [54, 57, 60].

For studies predicting mortality, research utilizing public datasets [6, 61] was significantly less prevalent compared to those using private datasets. These studies primarily relied on structured features, with fewer studies incorporating unstructured features, including free text and imaging data [52, 58, 62, 63]. Model construction predominantly relied on internal validation, with fewer studies adopting external validation [52]. Studies employing cross-validation still outnumbered those that did not. Traditional machine learning approaches were more commonly used than deep learning methods [52, 58, 61, 62, 64–67]. However, for mortality prediction, studies utilizing ensemble methods outnumbered those that did not.

From the above analysis, it is evident that studies predicting admission, critical care, and mortality dispositions differ significantly in terms of sample sources, feature structuredness, and algorithm choices, leading to varied performance outcomes. Therefore, conducting a meta-analysis to summarize the overall performance of these predictive models is essential. Furthermore, the differences in sample sources and methodologies may contribute to between-study heterogeneity, making it equally important to identify potential factors causing this heterogeneity.

Methods

This study adheres to the reporting guidelines outlined in the Preferred Reporting Items for Systematic Reviews and Meta-Analyses [68, 69] (see Additional file 1 and Additional file 2). Additionally, the research protocol for this study has the approval of E-Da Hospital (EMRP-109-158).

Search strategy and selection process

This study utilized a combination of keywords to search across seven electronic databases, including Scopus, SpringerLink, ScienceDirect, PubMed, Wiley, Sage, and Google Scholar, until December 31, 2023. The primary focus was on three types of ED dispositions: Admission, critical care, or mortality. ‘Admission’ refers to when patients seek treatment at the ED and then transfer to a general ward: ‘critical care’ involves patients with critical conditions requiring ICU transfer with or without the use of intubation or mechanical ventilation; and, ‘mortality’ refers to patients who expire before leaving the ED. Due to potentially diverse keywords for expressing these three ED dispositions, the study employed the keyword combination ‘emergency department’ AND (‘machine learning’ OR ‘deep learning’ OR ‘artificial intelligence’), followed by manual filtering by the researchers.

The inclusion criteria consisted of (1) studies focusing on ED dispositions, (2) studies reported in English, and (3) studies utilizing machine learning or deep learning methods. Exclusion criteria included (1) studies not employing machine learning or deep learning, (2) studies lacking sufficient information on outcome measures, and (3) studies not related to the prediction of ED dispositions. Following these criteria, 12,214 potential articles were identified. After excluding 156 duplicate records and the screening of titles and abstracts, 241 full-text articles remained. These were independently reviewed by two researchers, resulting in the exclusion of 153 articles not meeting the inclusion criteria. Ultimately, 88 articles [6–10, 21–67, 70–105]were selected for subsequent meta-analysis. The literature screening process is illustrated in Fig. 1. The studies included in this review are listed in Additional file 3 and Additional file 4.

Fig. 1.

Article selection process

Data extraction

For the included articles, this study extracted the following information: author(s), publication year, sample size, type of ED disposition (admission, critical care, or mortality), data source (private or public dataset), data structure for features (structured, unstructured, or combined), type of unstructured feature (free text or image), age group of samples (adult, mixed, youth, elder, or unclear), type of AI techniques adopted (machine learning or deep learning), whether cross-validation was used, and whether ensemble learning was adopted. Additionally, this study captured the numbers of true/false positives and true/false negatives. If not directly provided, this study performed conversions based on existing data located in the articles. As the same article may develop multiple ED disposition models simultaneously, this study treated them as distinct ED disposition models for purposes of inclusion.

Methodological analysis

This study assessed the risk of bias and applicability of the evidence based on the Prediction model risk of bias assessment tool (PROBAST) [106, 107]. It primarily focuses on four domains: participants, predictors, outcomes, and analysis.

Statistical analysis

This study followed recommendations from prior diagnostic test accuracy literature [108] to calculate the following measures for test accuracy: sensitivity, specificity, area under the receiver operating characteristic curve (AUROC), diagnostic odds ratio (DOR), positive likelihood ratio (+ LR), and negative likelihood ratio (-LR). Additionally, forest plots were utilized to depict the variability among the included literature, along with the hierarchical summary receiver operating characteristic curve (HSROC) with 95% confidence intervals (CI) and 95% prediction intervals. Further, to identify potential factors influencing heterogeneity, meta-regression analysis was conducted, incorporating variables such as type of ED disposition, data source, type of feature, type of unstructured feature, type of sample, type of AI techniques employed, whether cross-validation was undertaken, and whether ensemble learning was adopted. All analyses were conducted using R Statistical Software v4.3.2 [109] with the lime4 v1.1-35.1 [110] and the mada 0.5.11 [111] package. The MetaDTA was utilized to create the HSROC [112, 113].

Results

General study characteristics

As illustrated in Table 2, among the 88 included articles, there were a total of 117 models present (see Supplementary file C and D), with 39, 45, and 33 models predicting admission, critical care, and mortality, respectively. The majority of models sourced their data from private sources (88.89%). Features used to construct predictive models for ED dispositions were predominantly structured data (78.63%), while among models employing unstructured features, approximately 68% and 32% utilized free text and image data, respectively. Notably, free text data were processed using natural language processing (NLP) techniques to extract meaningful features for model development. Most models (94.02%) only adopted internal validation (the same dataset) instead of external validation (a completely independent dataset) to develop predictive models. Most models were based on samples of adult (70.94%), and the majority employed machine learning techniques (71.79%) rather than deep-learning techniques. Approximately 63% of models utilized cross validation during the training process, while about 38% employed ensemble learning.

Table 2.

Characteristics of included studies

Characteristic	Value	Frequency	%
Disposition	Admission	39	33.33
	Critical care	45	38.46
	Mortality	33	28.21
Data source	Private	104	88.89
	Public	13	11.11
Data structure for feature	Structured	92	78.63
	Unstructured	21	17.95
	Combined	4	3.42
Type of unstructured feature	Free text	17	68.00
	Image	8	32.00
Type of validation	External	7	5.98%
	Internal	110	94.02%
Age group of sample	Adult	83	70.94
	Mixed	13	11.11
	Youth	10	8.55
	Elder	6	5.13
	Unclear	5	4.27
Type of artificial intelligence techniques adopted	Deep learning	33	28.21
	Machine learning	84	71.79
Cross validation	No	43	36.75%
	Yes	74	63.25%
Ensemble	No	45	38.46
	Yes	72	61.54

This study further categorizes machine learning and deep learning approaches. From Table 3, it is evident that Random forest (RF) (19.66%) and eXtreme gradient boosting (XGB) (18.80%) are the most commonly used algorithms, followed by Gradient boosting machine (GBM) (11.97%) and LightGBM (5.13%). In the realm of deep learning, the Deep neural network (DNN) (18.80%) has a higher usage rate than the Convolutional neural network (CNN) (6.84%) and Recurrent neural network (RNN) (2.56%).

Table 4.

Performance of predicting ED dispositions by artificial intelligence

Metric	Admission	Critical care	Mortality
AUROC	0.866 (0.836–0.929)	0.928 (0.893–0.951)	0.932 (0.894–0.956)
Sensitivity	0.81 (0.74–0.86)	0.86 (0.79–0.91)	0.85 (0.80–0.89)
Specificity	0.87 (0.81–0.91)	0.89 (0.84–0.93)	0.94 (0.90–0.96)
DOR	17.3 (12.40–23.50)	44.5 (24.70–74.20)	74.6 (37.70–133.00)
+LR	4.76 (3.73–6.04)	7.73 (5.06–11.50)	12.8 (7.77–20.10)
-LR	0.277 (0.23–0.33)	0.18 (0.12–0.25)	0.177 (0.13–0.23)

Note: AUROC = Area Under Receiver Operating Characteristic curve, CI = Confidence Interval, DOR = Diagnostic Odds Ratio, ED = Emergency Department, +LR = Positive Likelihood Ratio, and –LR = Negative Likelihood Ratio

Table 3.

Type of artificial intelligence techniques adopted

Technique	Algorithm	Frequency	%
Machine learning	Random forest	23	19.66
	eXtreme Gradient Boosting	22	18.80
	Gradient boosting machine	14	11.97
	LightGBM	6	5.13
	Logistic regression	4	3.42
	Support vector machine	4	3.42
	Stacking (or other ensembles)	3	2.56
	CatBoost	2	1.71
	Decision tree	2	1.71
	Neural network	1	0.85
	INC2.5	1	0.85
	LASSO	1	0.85
	AutoScore	1	0.85
Deep learning	Deep neural network	22	18.80
	Convolutional neural network	11	6.84
	Recurrent neural network	3	2.56

Quality assessment

In terms of risk of bias, among the 87 included articles, approximately 70.11% were classified as having a high risk of bias regarding the predictors domain. For the other three domains—participants, outcomes, and analysis—98.85%, 100%, and 97.70% of articles, respectively, were assessed as having a low risk of bias. Overall, 96.55% of articles were assessed as having a low risk of bias, while 3.45% were classified as high risk.

Regarding the risk of applicability, nearly all articles were assessed as having a low risk across the three domains. Specifically, all articles demonstrated low risk concerning participant applicability, while 98.85% showed low risk for predictors and outcomes. Overall, 100% of the articles were assessed as having a low risk of applicability. This assessment of risk of bias and applicability based on the PROBAST tool is summarized in Fig. 2.

Fig. 2.

Quality assessment by PROBAST

Diagnostic accuracy

Among the three major types of ED disposition prediction models, those forecasting mortality achieved the highest area under the receiver operating characteristic curve (AUROC), followed by models predicting critical care, with admission prediction models exhibiting the lowest performance (see Table 4). The reported statistics for sensitivity, specificity, and AUROC represent pooled summary measures derived from the component studies included in this meta-analysis. The pooled summary AUROC for predicting admission, critical care, and mortality were 0.866 (95% CI 0.836–0.929), 0.928 (95% CI 0.893–0.951), and 0.932 (95% CI 0.894–0.956), respectively. In terms of sensitivity, admission prediction models showed the lowest sensitivity at 0.81 (95% CI 0.74–0.86), followed by critical care models at 0.86 (95% CI 0.79–0.91), and mortality models at 0.85 (95% CI 0.80–0.89). Regarding specificity, admission models exhibited the lowest specificity at 0.87 (95% CI 0.81–0.91), followed by critical care models at 0.89 (95% CI 0.84–0.93), and mortality models at 0.94 (95% CI 0.90–0.96). These statistics are primarily based on models utilizing internal validation, as only 5.98% of the included models performed external validation. In terms of sensitivity, critical care prediction models performed the best, closely followed by mortality prediction models, while admission prediction models showed the lowest sensitivity. Regarding specificity, mortality prediction models demonstrated the highest specificity, followed by critical care prediction models, with admission prediction models exhibiting the lowest specificity.

Analysis of the DOR revealed that models predicting mortality exhibited the highest discriminatory performance [114], while models predicting admission had the lowest DOR. Considering that the + LR, models predicting mortality were better at identifying true mortality cases due to their highest + LR, whereas models predicting critical care had a lower -LR, indicating their better ability to identify non-critical care patients [115]. Forest plots of sensitivity and specificity for models predicting admission, critical care, and mortality are illustrated in Figs. 3, 4 and 5.

Fig. 3.

Sensitivity and specificity of models predicting admission (n = 39)

Fig. 4.

Sensitivity and specificity of models predicting critical care (n = 45)

Fig. 5.

Sensitivity and specificity of models predicting mortality (n = 33)

Plausible covariates to explain between-study heterogeneity

Overall, machine learning models for predicting ED disposition demonstrate a sensitivity of approximately 0.84 (95% CI 0.80–0.87) and a specificity of around 0.90 (95% CI 0.87–0.92) (see Table 5), indicating their higher ability to correctly identify negative cases. When differentiating ED disposition into the categories of admission, critical care, and mortality, using admission as the reference category for comparison (as depicted in Table 5), the sensitivity of admission prediction models (0.81, 95% CI 0.74–0.86) is slightly lower than that of critical care and mortality prediction models (0.86, 95% CI 0.79–0.91 and 0.85, 95% CI 0.80–0.89, respectively), although these differences are not statistically significant. Similarly, the specificity of admission prediction models (0.87, 95% CI 0.81–0.91) is also slightly lower than that of critical care and mortality prediction models (0.89, 95% CI 0.84–0.93 and 0.94, 95% CI 0.90–0.96, respectively), with a statistically significant difference observed between admission prediction models and mortality prediction models’ specificity (0.87 vs. 0.94, p = 0.027).

Table 5.

Summary estimates for sensitivity and specificity

Covariate	Metric	n	Estimate	95%	C.I.	p value
Overall	Sens	117	0.84	0.80	0.87
Overall	Specs	117	0.90	0.87	0.92
Admission	Sens	39	0.81	0.74	0.86	[Reference]
Admission	Specs	39	0.87	0.81	0.91	[Reference]
Critical care	Sens	45	0.86	0.79	0.91	0.289
Critical care	Specs	45	0.89	0.84	0.93	0.503
Mortality	Sens	33	0.85	0.80	0.89	0.193
Mortality	Specs	33	0.94	0.90	0.96	0.027

Note: C.I. = Confidence Interval, Sens = Sensitivity, and Specs = Specificity

Plausible covariates for admission predictive models

This study compares the predictive abilities of different ED dispositions (admission, critical care, and mortality) to assess whether or not they are influenced by variables such as data characteristics, sample properties, or machine-learning methods.

Firstly, regarding the prediction of admission models, Table 6 shows that models using public datasets exhibit higher sensitivity (0.94) and specificity (0.90) when compared to those using private datasets (sensitivity = 0.80, specificity = 0.86), although the differences are not statistically significant. In terms of data structuring, both sensitivity (0.84) and specificity (0.88) of structured data models surpass those of unstructured data models (sensitivity = 0.72, specificity = 0.80) and those models combining unstructured and structured data (sensitivity = 0.74, specificity = 0.82). However, the differences among the three types of data are not statistically significant. Notably, among unstructured data, while some studies utilize image and free-text data, only eight models use free text as a feature for predicting admission, with no models using image data, thus this variable was not included for purposes of analysis. Among the eight models using free text data, the pooled sensitivity was 0.73 (95% CI 0.65–0.80), and the pooled specificity was 0.81 (95% CI 0.73–0.88), indicating moderate diagnostic accuracy.

Table 6.

Summary estimates for sensitivity and specificity of admission studies

Characteristic	Covariate	Metric	n	Estimate	95% C.I.		p value
Data	Public dataset	Sens	6	0.94	0.57	0.99	0.284
	Public dataset	Specs	6	0.90	0.70	0.97	0.581
	Private dataset	Sens	33	0.79	0.73	0.84	[Reference]
	Private dataset	Specs	33	0.86	0.79	0.91	[Reference]
	Unstructured	Sens	3	0.72	0.70	0.74	0.387
	Unstructured	Specs	3	0.80	0.74	0.86	0.521
	Combined	Sens	5	0.74	0.61	0.84	0.362
	Combined	Specs	5	0.82	0.68	0.91	0.510
	Structured	Sens	31	0.83	0.75	0.89	[Reference]
	Structured	Specs	31	0.88	0.81	0.93	[Reference]
Sample	Adult	Sens	21	0.81	0.72	0.87	0.308
	Adult	Specs	21	0.87	0.76	0.93	0.448
	Youth	Sens	6	0.82	0.69	0.90	0.218
	Youth	Specs	6	0.80	0.67	0.88	0.064
	Elder	Sens	3	0.76	0.74	0.79	0.541
	Elder	Specs	3	0.75	0.70	0.79	0.027
	Mixed	Sens	5	0.70	0.55	0.82	[Reference]
	Mixed	Specs	5	0.92	0.84	0.96	[Reference]
Artificial intelligence technique	Machine learning	Sens	28	0.80	0.72	0.86	0.535
	Machine learning	Specs	28	0.87	0.80	0.92	0.770
	Deep learning	Sens	11	0.85	0.71	0.93	[Reference]
	Deep learning	Specs	11	0.85	0.73	0.92	[Reference]
	Random forest	Sens	10	0.82	0.61	0.93	0.147
	Random forest	Specs	10	0.92	0.77	0.98	0.720
	eXtreme Gradient Boosting	Sens	8	0.77	0.69	0.84	0.022
	eXtreme Gradient Boosting	Specs	8	0.86	0.77	0.91	0.788
	Deep neural network	Sens	7	0.70	0.65	0.75	0.002
	Deep neural network	Specs	7	0.78	0.74	0.82	0.360
	Recurrent neural network	Sens	2	0.97	0.31	1.00	0.604
	Recurrent neural network	Specs	2	0.93	0.58	0.99	0.899
	Convolutional neural network	Sens	2	0.99	0.15	1.00	[Reference]
	Convolutional neural network	Specs	2	0.99	0.02	1.00	[Reference]
	Ensemble	Sens	26	0.78	0.71	0.84	0.301
	Ensemble	Specs	26	0.86	0.78	0.91	0.713
	No-ensemble	Sens	13	0.86	0.73	0.93	[Reference]
	No-ensemble	Specs	13	0.88	0.76	0.95	[Reference]
	Cross validation	Sens	26	0.84	0.76	0.90	0.170
	Cross validation	Specs	26	0.86	0.76	0.92	0.541
	No cross validation	Sens	13	0.74	0.65	0.82	[Reference]
	No cross validation	Specs	13	0.88	0.82	0.93	[Reference]

Note: C.I. = Confidence Interval, Sens = Sensitivity, and Specs = Specificity

Regarding sample properties, models using mixed samples (all age groups) demonstrate lower sensitivity compared to models using adult, youth, and elder samples, yet the specificity of models using mixed samples is higher than those using the other three types of samples. Sensitivity among the four different samples does not reach statistical significance, but the specificity of models using mixed samples (0.92 vs. 0.75, p = 0.027) is significantly higher than that of models using elderly samples.

In terms of machine-learning methods, models utilizing deep learning exhibit higher sensitivity (0.86) when compared to those generated using traditional machine-learning methods (sensitivity = 0.80), but the difference is not statistically significant (p = 0.541). The specificity of models built using traditional machine learning (0.87) is slightly higher than that of models generated using deep learning (0.86), also not statistically significant (p = 0.868). To further compare the performance of different machine-learning algorithms, models utilizing CNN show both higher sensitivity and specificity than those using RF, XGB, DNN, and RNN. Among these, the differences in sensitivity compared with XGB (p = 0.022) and DNN (p = 0.002) are statistically significant. Models built using ensemble learning demonstrate lower sensitivity (0.78) and specificity (0.86) when compared to models not using ensemble learning (sensitivity = 0.86, specificity = 0.88), but the differences are not statistically significant (p = 0.301 and 0.713). Additionally, models not using cross-validation exhibit a lower level of sensitivity (0.74) and a higher level of specificity (0.88) when compared to models using cross-validation (sensitivity = 0.84, specificity = 0.86), yet both sensitivity and specificity do not reach statistical significance (p = 0.170 and 0.541).

Plausible covariates for critical care predictive models

In predicting critical care models, the sensitivity (0.87) and specificity (0.90) of models using private datasets are higher than those using public datasets (sensitivity = 0.76 and specificity = 0.73), but the differences do not reach statistical significance (p = 0.316 and 0.103). As there is only one model in the critical care prediction category that solely uses unstructured data, comparisons are made only between models using structured and combined data types. From Table 7, it is evident that the sensitivity (0.86) and specificity (0.90) of models using structured data are higher than or equal to those of models using combined data (sensitivity = 0.86, specificity = 0.87), but none of the differences are statistically significant (p = 0.865 and 0.626). In these prediction models, some models combine image and free-text data. This study further compares the impact of these two formats on prediction models, revealing that models combining image have higher sensitivity (0.87) compared to those combining free text (0.83), while models combining image have lower sensitivity (0.86) as compared to those combining free text (0.87), but neither difference reaches statistical significance (p = 0.530 and 0.861).

Table 7.

Summary estimates for sensitivity and specificity of critical care studies

Characteristic	Covariate	Metric	n	Estimate	95%	C.I.	p value
Data	Public dataset	Sens	5	0.76	0.64	0.85	0.316
	Public dataset	Specs	5	0.73	0.62	0.82	0.103
	Private dataset	Sens	40	0.87	0.79	0.92	[Reference]
	Private dataset	Specs	40	0.90	0.85	0.94	[Reference]
	Combined	Sens	12	0.86	0.77	0.92	0.865
	Combined	Specs	12	0.87	0.80	0.92	0.626
	Structured	Sens	32	0.86	0.77	0.92	[Reference]
	Structured	Specs	32	0.90	0.83	0.95	[Reference]
	Image	Sens	5	0.87	0.76	0.94	0.530
	Image	Specs	5	0.86	0.80	0.90	0.861
	Free text	Sens	8	0.83	0.67	0.92	[Reference]
	Free text	Specs	8	0.87	0.74	0.94	[Reference]
Sample	Adult	Sens	38	0.86	0.78	0.91	0.664
	Adult	Specs	38	0.90	0.83	0.94	0.616
	Youth	Sens	2	0.85	0.73	0.92	0.762
	Youth	Specs	2	0.72	0.51	0.86	0.204
	Mixed	Sens	3	0.90	0.74	0.96	[Reference]
	Mixed	Specs	3	0.84	0.75	0.90	[Reference]
Artificial intelligence technique	Machine learning	Sens	31	0.88	0.80	0.93	0.205
	Machine learning	Specs	31	0.91	0.84	0.95	0.171
	Deep learning	Sens	14	0.78	0.69	0.85	[Reference]
	Deep learning	Specs	14	0.83	0.78	0.87	[Reference]
	Random forest	Sens	9	0.91	0.77	0.97	0.465
	Random forest	Specs	9	0.96	0.83	0.99	0.297
	eXtreme gradient boosting	Sens	11	0.95	0.85	0.99	0.302
	eXtreme gradient boosting	Specs	11	0.84	0.71	0.92	0.965
	LightGBM	Sens	3	0.69	0.29	0.92	0.439
	LightGBM	Specs	3	0.98	0.59	1.00	0.259
	Logistic regression	Sens	2	0.78	0.71	0.84	0.673
	Logistic regression	Specs	2	0.84	0.80	0.87	0.922
	Deep neural network	Sens	10	0.75	0.64	0.83	0.442
	Deep neural network	Specs	10	0.83	0.77	0.88	0.932
	Convolutional neural network	Sens	3	0.84	0.63	0.94	[Reference]
	Convolutional neural network	Specs	3	0.84	0.74	0.90	[Reference]
	Ensemble	Sens	25	0.91	0.82	0.95	0.032
	Ensemble	Specs	25	0.91	0.82	0.96	0.339
	No ensemble	Sens	20	0.76	0.69	0.82	[Reference]
	No ensemble	Specs	20	0.86	0.81	0.90	[Reference]
	Cross validation	Sens	27	0.86	0.76	0.92	0.952
	Cross validation	Specs	27	0.87	0.79	0.92	0.288
	No cross validation	Sens	18	0.87	0.75	0.93	[Reference]
	No cross validation	Specs	18	0.92	0.83	0.96	[Reference]

Note: C.I. = Confidence Interval, Sens = Sensitivity, and Specs = Specificity

Regarding sample properties, since there is only one data point for unclear and the elderly in the models predicting critical care, they were not included in the analysis. Only adult, youth, and mixed samples were compared. From Table 7, it is observed that the sensitivity (0.90) of models using mixed samples is higher than those using adult (0.86) and youth (0.85), but the differences are not statistically significant (p = 0.664 and 0.762). Furthermore, the specificity (0.84) of models using mixed samples is lower than that of models using adult (0.90) but higher than that of models using youth (0.72), yet none of the differences are statistically significant (p = 0.616 and 0.204).

Regarding machine-learning methods, the sensitivity (0.88) and specificity (0.91) of models using traditional machine learning are higher than those using deep learning (sensitivity = 0.78 and specificity = 0.83), but the differences are not statistically significant (p = 0.205 and 0.171). When comparing specific machine-learning algorithms, CNN algorithm achieves both a sensitivity and specificity of 0.84. Compared to RF models, which have a higher sensitivity (0.91) and specificity (0.96), CNN models perform slightly lower on both metrics, though these differences are not statistically significant. Similarly, CNN models outperform LightGBM and LR in sensitivity (LightGBM: 0.69, LR: 0.78) but are on par with LR in specificity (0.84) and below LightGBM (0.98), without statistically significant differences. When compared to DNN, CNN models achieve higher sensitivity (DNN: 0.75) but perform similarly in specificity (DNN: 0.83), with no statistically significant differences. Overall, CNN models show a balanced performance in sensitivity and specificity compared to other algorithms. Concerning the use of ensemble learning, the sensitivity (0.91) and specificity (0.91) of models using ensemble learning are higher than those not using it (sensitivity = 0.69 and specificity = 0.81), but only sensitivity reaches statistical significance (p = 0.032). Lastly, the sensitivity (0.86) of models using cross-validation is slightly lower than those not using it (0.87), and the specificity (0.87) of models using cross-validation is also lower compared to models not using it (0.92). Neither difference reaches statistical significance (p = 0.952 and 0.288, respectively).

Plausible covariates for mortality predictive models

In models predicting mortality, both sensitivity (0.90) and specificity (0.95) are higher for models using public datasets than those using private datasets (sensitivity = 0.85, specificity = 0.94), but none of the differences are statistically significant (see Table 8). Regarding data structuring, since no models solely use unstructured data, comparisons were made only between models using structured and combined data. From Table 8, it is observed that both sensitivity (0.86) and specificity (0.95) of models using structured data are higher than those using combined data (sensitivity = 0.82, specificity = 0.85), but none of the differences are statistically significant (p = 0.591, 0.189). Although some models using combined data incorporate image, only one model utilizes free text, achieving a sensitivity of 0.92 (95% CI: 0.89–0.94) and a specificity of 0.81 (95% CI: 0.80–0.81). Therefore, a comparison between these two types of unstructured data was not conducted.

Table 8.

Summary estimates for sensitivity and specificity of mortality studies

Characteristic	Covariate	Metric	n	Estimate	95%	C.I.	p value
Data	Public dataset	Sens	2	0.90	0.64	0.98	0.446
	Public dataset	Specs	2	0.95	0.70	0.99	0.799
	Private dataset	Sens	31	0.85	0.80	0.88	[Reference]
	Private dataset	Specs	31	0.94	0.90	0.96	[Reference]
	Combined	Sens	4	0.82	0.66	0.91	0.591
	Combined	Specs	4	0.85	0.74	0.92	0.189
	Structured	Sens	29	0.86	0.80	0.90	[Reference]
	Structured	Specs	29	0.95	0.91	0.97	[Reference]
Sample	Adults	Sens	24	0.85	0.79	0.89	0.853
	Adults	Specs	24	0.94	0.89	0.97	0.759
	Youths	Sens	2	0.99	0.00	1.00	0.663
	Youths	Specs	2	0.89	0.83	0.92	0.458
	Elders	Sens	2	0.88	0.77	0.94	0.689
	Elders	Specs	2	0.90	0.77	0.96	0.540
	Mixed	Sens	5	0.84	0.67	0.93	[Reference]
	Mixed	Specs	5	0.96	0.82	0.99	[Reference]
Artificial intelligence technique	Machine learning	Sens	25	0.86	0.80	0.90	0.709
	Machine learning	Specs	25	0.95	0.90	0.97	0.442
	Deep learning	Sens	8	0.83	0.74	0.90	[Reference]
	Deep learning	Specs	8	0.91	0.84	0.96	[Reference]
	Random forest	Sens	4	0.91	0.56	0.99	0.516
	Random forest	Specs	4	1.00	0.04	1.00	0.476
	eXtreme Gradient boosting	Sens	3	0.73	0.61	0.82	0.562
	eXtreme Gradient boosting	Specs	3	0.95	0.77	0.99	0.839
	LightGBM	Sens	3	0.91	0.78	0.96	0.205
	LightGBM	Specs	3	0.92	0.75	0.98	0.748
	Logistic regression	Sens	2	0.80	0.73	0.86	0.938
	Logistic regression	Specs	2	0.95	0.68	0.99	0.866
	Deep neural network	Sens	5	0.86	0.76	0.92	0.418
	Deep neural network	Specs	5	0.89	0.83	0.93	0.436
	Convolutional neural network	Sens	3	0.79	0.63	0.89	[Reference]
	Convolutional neural network	Specs	3	0.94	0.74	0.99	[Reference]
	Ensemble	Sens	21	0.88	0.82	0.92	0.095
	Ensemble	Specs	21	0.95	0.89	0.98	0.620
	No ensemble	Sens	12	0.79	0.71	0.86	[Reference]
	No ensemble	Specs	12	0.93	0.87	0.96	[Reference]
	Cross validation	Sens	21	0.85	0.78	0.90	0.926
	Cross validation	Specs	21	0.96	0.92	0.98	0.032
	No cross validation	Sens	12	0.85	0.78	0.90	[Reference]
	No cross validation	Specs	12	0.87	0.81	0.91	[Reference]

Note: C.I. = Confidence Interval, Sens = Sensitivity, and Specs = Specificity

Regarding sample properties, as there are no models classified as ‘unclear,’ comparisons were made among models with samples classified as mixed, adult, youth, and elder. The results show that sensitivity (0.84) of models using mixed samples is lower than those using the other three types of samples (0.85 for adult, 0.99 for youth, and 0.88 for elder), but none of the differences are statistically significant. However, specificity (0.96) of models using mixed samples is higher than those using the other three types of samples (0.94 for adult, 0.89 for youth, and 0.90 for elder), yet none of the differences are statistically significant.

Regarding machine-learning methods, both sensitivity (0.86) and specificity (0.95) of models using machine learning are higher than those using deep learning (sensitivity = 0.83, specificity = 0.91), but neither difference reaches statistical significance (p = 0.709 and 0.442). To further compare the performance of different algorithms, models using the RF and LR algorithms have both higher sensitivity and specificity than those using CNN, though these differences are not statistically significant. Models employing LightGBM and DNN have higher sensitivity but lower specificity compared to CNN, with none of these differences reaching statistical significance. Additionally, models using XGB exhibit lower sensitivity than CNN, but their specificity is higher, also without statistical significance. Models using ensemble methods have higher sensitivity (0.88) and specificity (0.95) than those not using ensemble methods (sensitivity = 0.79 and specificity = 0.93), but none of the differences reach statistical significance (p = 0.095 and 0.620).

Regarding the use of cross-validation in prediction models, sensitivity is the same for models with and without cross-validation (0.85). However, models using cross-validation have significantly higher specificity compared to those not using it (0.96 vs. 0.87, p = 0.032). The summary sensitivity and specificity performance of mortality prediction models are presented in Table 8.

Summarization of plausible covariates for three predictive models

Summarizing the performance of the three disposition prediction models (see Table 9), First off, regarding data sources, models utilizing public data sources perform better in predicting admission and mortality when compared to those using private data sources, but the opposite trend is observed for predicting critical care. Secondly, concerning data structuring, models using structured data outperform those using both structured and unstructured data in predicting admission, critical care, and mortality. In admission-prediction models, those solely using unstructured data exhibit the poorest performance. As for sample properties, no distinct pattern emerges favoring any particular sample combination among the four different types.

Table 9.

Summarization of the performance of predictive models for three emergency department dispositions

Disposition	Metric	Characteristic
		Data		Sample	Artificial-intelligence technique
		Source	Feature Type	Group	Approach	Top algorithm	Ensemble	Cross validation
Admission	Sens	Public > Private	S > B > U	A = Y > E > M	DL > ML	CNN	No ensemble > Ensemble	No CV > CV
	Spec	Public > Private	S > B > U	M > A > Y > E	ML > DL	CNN	No ensemble > Ensemble	No CV > CV
Critical care	Sens	Private > Public	S > B	M > A > Y	ML > DL	XGB	Ensemble > No ensemble	CV > No CV
	Spec	Private > Public	S > B	A > M > Y	ML > DL	LightGBM	Ensemble > No ensemble	No CV > CV
Mortality	Sens	Public > Private	S > B	Y > E > A > M	ML > DL	RF/LightGBM	Ensemble > No ensemble	CV = No CV
	Spec	Public > Private	S > B	A > M > Y > E	ML > DL	RF	Ensemble > No ensemble	CV > No CV

Notes:

1. Metric: Sens = Sensitivity and Spec = Specificity

2. Source: Public = public data source and Private = private data source

3. Type of feature: S = structured data, B = structured and unstructured data, and U = unstructured data

4. Sample: A = adults, Y = youths, E = elders, and M = mixed samples

5. Approach: ML = machine learning and DL = deep learning

6. CNN = Convolutional neural network, XGB = eXtreme gradient boosting, and RF = Random forest

7. Cross-validation: CV = cross-validation

In terms of machine learning methods, except for admission-prediction models where sensitivity is better with deep learning, both sensitivity and specificity in the other two prediction models favor machine learning. To further compare the performance of different machine-learning algorithms, CNN demonstrates superior sensitivity and specificity for admission prediction compared to other algorithms. For critical care prediction, XGB shows the highest sensitivity, while LightGBM excels in specificity. For mortality prediction, RF and LightGBM yield the best sensitivity, with RF also showing the highest specificity. However, when employing ensemble learning, the sensitivity and specificity of admission-prediction models are both lower as compared to models not using ensemble learning. For critical care and mortality-prediction models, the sensitivity and specificity of models without ensemble learning are higher than those with ensemble learning. Additionally, the use of cross-validation does not consistently guarantee better model performance; while admission-prediction models without cross-validation exhibit higher sensitivity and specificity, the sensitivity and specificity of critical care- or mortality-prediction models vary.

Finally, this study employed HSROC to evaluate model performance, with the HSROC curves for admission, critical care, and mortality depicted in Figs. 6 and 7, and 8, respectively. It is evident from these figures that the mortality-prediction model exhibits higher precision compared to the admission and critical care prediction models. This is indicated by the smaller 95% prediction-interval region and 95% confidence region for the mortality-prediction model, as observed in Figs. 6, 7 and 8.

Fig. 6.

Summary receiver operating-characteristic curve for models predicting admission

Fig. 7.

Summary receiver operating-characteristic curve for models predicting critical care

Fig. 8.

Summary receiver operating-characteristic curve for models predicting mortality

Discussion

Based on the meta-analysis of 117 models extracted from 87 articles included in this study, the overall sensitivity and specificity for predicting ED disposition patterns were determined to be 0.84 and 0.90, respectively. These results indicate that the utilization of machine learning in predicting the discharge disposition of ED patients shows acceptable predictive capabilities. This capability allows for the early acquisition of patient disposition information, which can greatly aid in the effective allocation of medical personnel and resources within modern healthcare institutions.

Type of ED dispositions predicted

Upon further examination, among the 117 predictive models, 39 are focused on admission, 45 on critical care, and 33 on mortality. The meta-analysis reveals that mortality-prediction models exhibit the highest AUROC, followed by critical-care prediction models, with admission prediction models demonstrating the lowest performance. Sensitivity analysis indicates that critical care-prediction models have the highest sensitivity, followed by mortality-prediction models, while admission-prediction models have the lowest. Similarly, regarding specificity, mortality-prediction models show the highest, followed by critical care-prediction models, with admission-prediction models again displaying the lowest specificity.

Notably, when admission prediction serves as the reference category, sensitivity and specificity among these three types of prediction models generally do not exhibit statistically significant differences, except for the specificity of mortality-prediction models, which is significantly higher than that of admission-prediction models. Additionally, the specificity of these prediction models tends to outweigh sensitivity, suggesting a stronger ability to correctly identify true negatives but potentially missing some true positives. Future research may necessitate an iterative refinement process to enhance the sensitivity of ED disposition models.

Clarification of research purpose and scope

While this meta-analysis provides a quantitative overview of AI performance in predicting ED patient dispositions, it is important to recognize the heterogeneity among the predictive models included. These models vary in terms of the disposition types predicted, data used, machine learning methods, and patient conditions, which limits the generalizability of our meta-analysis results to specific clinical situations. The primary goal of this meta-analysis is to offer insights into general trends, strengths, and challenges in AI applications for ED disposition prediction, rather than to provide tailored recommendations for individual contexts. Additional research and development tailored to the unique demands of each clinical setting may therefore be necessary.

Public or private data source

In terms of the data utilized, the majority of the data are proprietary rather than publicly available. However, the analysis results show that when publicly available data are used for predicting admission and mortality, both sensitivity and specificity are higher compared to predictive models using private data. Conversely, when publicly available data are used for predicting critical care, sensitivity and specificity are lower than those of private datasets. The meta-analysis results of this study indicate an improvement in the predictive ability of publicly available data for predicting admission and mortality. In the realm of machine learning for skin image recognition, Tschandl et al. [116] argue for the importance of making skin image data publicly available, suggesting that by doing so would enhance skin image recognition performance. Since the number of evidences included in this study from public datasets is limited (n = 13), whether this argument applies to non-image-based data necessitates further research for validation.

Data structure of features

Based on the findings of this study, predictive models utilizing solely structured data consistently demonstrate higher sensitivity and specificity across all prediction categories: admission, critical care, and mortality. Notably, the lowest sensitivity and specificity are observed in models predicting admission solely based on structured features. This observation may stem from the necessity of feature extraction in handling unstructured data, wherein differences in extraction methods could impact prediction performance variability, ultimately leading to less effective performance than models using structured features alone.

Although the integration of both structured and unstructured features theoretically offers better informative data for purposes of model development, this assertion remains unconfirmed by the current meta-analysis. Specifically regarding unstructured data, while the use of image data for critical care prediction yields higher sensitivity compared to free text, the opposite is observed for specificity. However, neither type of unstructured data significantly influences the predictive outcomes of ED disposition.

Sample type

Regarding sample selection, models utilizing adult samples consistently demonstrate higher sensitivity and specificity across all three ED dispositions—admission, critical care, and mortality— when compared to models using other sample types. Models employing mixed samples exhibit superior sensitivity and specificity in predicting admission and critical-care dispositions compared to other sample types. Furthermore, models utilizing youth samples demonstrate higher specificity in predicting mortality compared to other samples. However, there is no discernible pattern in the performance of prediction models based on sample utilization, suggesting that the choice of sample may not substantially impact prediction model performance.

Machine learning vs. deep learning

Among the included models, machine learning remains predominant. Generally, in predicting the three ED dispositions, models built using machine-learning methods demonstrate higher sensitivity and specificity compared to those employing deep learning methods, except for admission-prediction models, where sensitivity is higher in deep learning-based models. This study infers that since the included research data primarily consist of structured data rather than images, the complexity may be lower, thus making machine-learning methods adequate for handling the task. Conversely, deep-learning methods may not effectively leverage their image processing capabilities in this context. Further analysis of different machine learning algorithms reveals that Convolutional neural networks perform best for predicting ED dispositions related to admission. For critical care disposition predictions, eXtreme Gradient Boosting and LightGBM models show superior performance, while for mortality predictions, Random forest and LightGBM models demonstrate the highest performance.

Ensemble-learning technique

Using ensemble learning is generally believed to improve the predictive capability of models [117]. However, according to the results of this study, the situation is not entirely straightforward. For the prediction of admission, models not utilizing ensemble learning performed better, while for predicting critical care and mortality, models employing ensemble learning outperformed in both sensitivity and specificity. The discrepancy in performance may be attributed to the fact that the ensemble learning techniques employed in the study were not identical. This suggests that the selection of appropriate methods and parameter configurations is crucial when utilizing ensemble learning.

Cross-validation technique

Regarding the use of cross-validation, the results of this study show a mixed picture. For predicting admission, models without cross-validation demonstrated superior sensitivity and specificity compared to those with cross-validation. However, for predicting critical care and mortality, models utilizing cross-validation exhibited higher sensitivity and specificity than those without. There was no clear pattern indicating that adopting cross-validation consistently enhanced model performance across all prediction categories. An inference drawn from this study is that among the 51 models employing cross-validation, 26 did not undergo hyper-parameter tuning to find the optimal settings, potentially leading to sub-optimal performance improvements. Finally, our review highlights that most models predominantly relied on internal validation rather than external validation, raising concerns about potential overfitting.

Risk of bias assessment

Our study used PROBAST to assess the risk of bias across four domains: participants, predictors, outcomes, and analysis. Overall, the results show that most studies have a low risk of bias, indicating that these studies are well-designed with adequate sample sizes and appropriate handling of missing data. This finding differs from previous PROBAST-based assessments [118, 119], which often identified a high risk of bias in most prediction models. This discrepancy may be due to the fact that PROBAST was not specifically designed for AI applications, and some signaling questions may not fully apply in this context. This limitation could be addressed once the PROBAST + AI tool is officially released.

Future research directions

The analysis results of this study suggest that the specificity of the included models for predicting admission, critical care, and mortality is higher than their sensitivity. This implies that the models excel in correctly identifying non-cases of ED disposition, but may struggle to identify all relevant cases. While the sensitivity of these prediction models exceeds 80%, there is room for improvement in their predictive capabilities. Recommendations for enhancement could be explored in the following areas.

Define standard features for predicting ED disposition at various stages

Throughout the ED visitation process, a wide array of data is generated, encompassing physiological signs, injury records, diagnoses, laboratory findings, radiographic images, and far more. In the models analyzed in this meta-analysis, the utilization of data varies considerably across stages, making it challenging to identify over-arching patterns for comparison. Future research could explore constructing predictive models in stages based on the data generated during patients’ ED visits and evaluate the performance of these models at each stage. Additionally, recommendations for features and timeframes applicable to each stage of ED visits could be proposed to facilitate further model development. Early prediction of patient disposition in the ED holds significant potential for optimizing emergency medical resource management, service capabilities, and overall allocations.

Build a public dataset for predicting ED disposition

Once suggested features are identified, relevant data can be collected based on these features to construct predictive models. Another logical step is to establish a public dataset by leveraging collaborative efforts from EDs worldwide. This shared dataset aims to support hospitals in developing their own models for predicting ED patient disposition. Furthermore, with access to these shared datasets, different models become comparative, potentially enhancing predictive performance. The results of this meta-analysis also suggest that models using public datasets outperform those using private datasets in predicting admission and mortality.

Structure the nature of features for predicting ED disposition

In theory, unstructured data may contain more crucial information, suggesting that utilizing unstructured data could lead to better predictive model performance. However, the analysis results of this study do not support this argument. Instead, models built using structured data outperformed those using both structured and unstructured data in predicting admission, critical care, and mortality. Models solely based on unstructured data performed the least satisfactorily in predicting admission. One possible explanation for this finding may be that the unstructured data were completed using templates, resulting in uniform content and reducing the significance of the information contained within. It is suggested that future research prioritize structured data as they contain primary features, with the simultaneous use of structured and unstructured data as additional features.

Sample dataset for predicting ED disposition

Due to the possibility of incomplete physiological maturity in younger emergency department patients, such as with infants, their response to illness may differ significantly from that of adults, particularly the elderly. Therefore, it is suggested that future studies consider distinguishing between age groups when constructing predictive models for emergency department patient disposition. This approach would better cater to the clinical needs of emergency departments. In the studies included in this meta-analysis, certain models were specifically tailored for the elderly [25, 33, 45, 47, 48, 65] or for adolescents/infants [44, 82].

Utilize tailored artificial intelligence techniques for predicting ED disposition

Based on the results of this meta-analysis, it appears that the predictive performance of deep- learning models is generally lower than that of machine-learning models, contrary to the common belief among the general public that deep learning outperforms. Subsequent research should further investigate possible reasons for this discrepancy and subsequently enhance the predictive capabilities of models built using deep-learning methods. Additionally, ensemble learning in this meta-analysis demonstrated superior performance in predicting critical care and mortality compared to models that did not utilize ensemble learning. Future research may consider employing different types of ensemble learning to identify more effective model architectures. Moreover, while cross-validation theoretically aids in improving model predictive ability, it is recommended that future studies utilize hyper-parameter tuning alongside cross-validation to enhance model performance. Lastly, future studies are strongly encouraged to adopt external validation to minimize the risk of overfitting.

Limitations

This review has several limitations that warrant acknowledgment. Firstly, caution is needed when interpreting the pooled sensitivity and specificity of this study due to the presence of between-studies heterogeneity. Secondly, 71 articles were excluded due to insufficient quantitative information. It is recommended that future research on ED disposition using machine learning provide sufficient metric information to enhance profile study characteristics.

Conclusions

The main aim of this study is to meta-analyze the performance of artificial intelligence techniques used in predicting ED dispositions. Due to the lack of objective assessments in existing review literature on this topic, a comprehensive understanding of how artificial intelligence performs in predicting ED disposition is limited. This limitation may hinder the effective utilization of this technology, which could be crucial for optimizing emergency medical resources and for addressing vital issues such as ED overcrowding.

The primary findings of this study indicate that machine-learning techniques applied to predict different ED dispositions, including admission, critical care, or mortality, achieve AUROC scores ranging from 0.87 to 0.93. Models predicting mortality perform the best, with sensitivity and specificity ranging from 0.81 to 0.94. However, the specificity for each of the three ED dispositions is higher than sensitivity, suggesting room for improvement in predicting positive cases of ED disposition. Feasible approaches to address this matter include:

1. To establish standardized feature sets for predicting ED dispositions;

2. To create shared datasets for training predictive models, accessible to both emergency medical practitioners and researchers;

3. To integrate structured and unstructured datasets; and,

4. To leverage machine-learning techniques such as cross-validation with hyper-parameter tuning and ensemble learning to enhance performance.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Abbreviations

AUC

Area under curve

AUROC

Area under receiver operating characteristic curve

CI

Confidence interval

CNN

Convolutional neural network

DNN

Deep neural network

DOR

Diagnostic odds ratio

DT

Decision tree

ED

Emergency department

GBM

Gradient boosting machine

HSROC

Hierarchical summary receiver operating characteristic curve

ICU

Intensive care unit

-LR

Negative likelihood ratio

+LR

Positive likelihood ratio

LR

Logistic regression

NN

Neural network

PROBAST

The Prediction Model Risk of Bias Assessment Tool

RF

Random forest

RNN

Recurrent neural network

SVM

Support vector machine

XGB

EXtreme gradient boosting

Author contributions

KMK and CSC conceived of this study and participated in the design and administration of the study. KMK and CSC drafted the manuscript and performed the statistical analysis. All authors read and approved the final manuscript.

Funding

This study was supported by the National Science and Technology Council, Taiwan, under grant number MOST-110-2410-H-239-015. The funder had no role in the study design, data collection, analysis, interpretation, or manuscript preparation.

Data availability

The datasets utilized and analyzed in this study are provided in the Additional file 3 and Additional file 4.

Declarations

Ethics approval and consent to participate

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. This article does not contain any studies with animals performed by any of the authors. The experimental protocols, approved by the Institutional Review Board of E-DA Hospital (IRB No. EMRP-109-158), included waived informed-consent requirements.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.