Predicting Avoidable Emergency Department Visits Using the NHAMCS Dataset

Abstract

Despite the important role avoidable emergency department (ED) visits play in healthcare costs and quality of care, there has been little work in development of predictive models to identify patients likely to present with an avoidable ED visit. We use a conservative definition of ‘avoidable’ ED visits defined as visits that do not require diagnostic or screening services, procedures, or medications, and were discharged home to classify visits as avoidable. Models trained using data from emergency departments across the US yielded a training AUC of 0.723 and a testing AUC of 0.703. Models trained using the full dataset were tested on demographic groups (race, gender, insurance status), finding comparable performance between white/black patients and male/female with reductions in performance in Hispanic populations and patients with Medicaid. Predictors strongly associated with non-avoidable ED visits included increased age, increasing number of total chronic diseases, and general as well as digestive symptoms. Reasons for visit stated as injuries and psychiatric symptoms influenced the model to predict an avoidable visit.

Introduction

Avoidable emergency department (ED) visits put significant strain on the healthcare system by increasing overall cost and leading to ED overcrowding^1,2. This problem has received more attention in the US in recent years due to significant increases in annual ED visits and reductions in ED admission rates among the elderly, the Medicare-reimbursed, and patients with multiple comorbidities³. Previous studies suggest that diverting ED visits for nonurgent conditions that are treatable at retail clinics or urgent care facilities may lead to a projected saving of $4.4 billion annually⁴. Avoidable ED visits are also believed to compromise quality of care by contributing to excessive testing and treatment as well as compromising the longitudinal relationship between the patients and the primary care physicians^2,5. Yet despite the heightened interest in reducing potentially avoidable ED visits, only a handful of studies had explored the factors associated with avoidable ED visits such as reasons for visit, demographics features, and encounter characteristics. In one national study of ED visits and utilization, non-urgent ED visits were shown to be more prevalent in the older, non-Hispanic white, and Medicare-insured patients⁶. Another review showed that, based on limited evidence, younger age, convenience of the ED compared to alternatives, and referral to the ED by a physician all contributed to driving up non-urgent ED use².

To our knowledge, none of the studies had developed advanced models to classify and predict avoidable ED visits, the insights from which could be used to inform more efficient and cost-effective care management. Thus, the goal of this study is to use several machine learning algorithms to predict avoidable ED visits based on patient and visit characteristics known before the point of triage using a conservative definition of avoidable ED visit⁷.

Methods

Study Population

The emergency department visits data used for this study comes from the Center for Disease Control (CDC) National Hospital Ambulatory Medical Care Survey (NHAMCS), which is an annual survey tool to collect the utilization of ambulatory services in hospital emergency and outpatient departments⁸. NHAMCS surveys a sample of non-institutional and general hospitals across the United States⁸. For this study, data from the NHAMCS survey collected between the years 2014 to 2018 were included in our sample.

Variables of Interest

From the NHAMCS dataset, we included the following variables of interest: age, gender, race and ethnicity, arrival time, whether the patient was seen at the ED with the past 72 hours, alcohol abuse, substance abuse, comorbidities, including Alzheimer’s, Asthma, Cancer, Chronic Kidney Disease, Chronic Obstructive Pulmonary Disease, Chronic Heart Failure, Coronary Arterial Disease, Depression, Diabetes, End Stage Renal Disease, HIV, Hyperlipidemia, Hypertension, Obstructive Sleep Apnea, Osteoporosis, and Obesity, the total number of chronic conditions, time of day, day in the week of visit, month of visit, and reasons for visit codes. The NHAMCS dataset also contains patient stated reasons for visit, broken down into broad categories of chief complaints by organ system. Reasons for visit, in the form of organ system categories were used as variables in this study. In addition, patient time of arrival, which is encoded as a 24-hour time was recoded into arrival at daytime (between 7 AM and 5PM), night (5 PM and midnight), and overnight (midnight to 7AM).

Data Preprocessing and Avoidable ED Definition

Variables with more than 30% missing values (n=7 variables) and observations with missing data values (18.55%) were dropped. Training and testing datasets were created via a 7:3 split. Subsets of the dataset were created by stratifying by race, gender, and insurance type.

The avoidable ED visit definition of this study is based on a prior avoidable ED visit study by Hsia et al that uses NHAMCS data from earlier years⁷. An avoidable ED visit is defined as discharged ED visits not requiring any diagnostic tests, procedures or medications. Patients admitted for observation, hospitalized, transferred, died in the ED or were dead on arrival were considered to be non-avoidable ED visits and excluded (Figure 1).

Figure 1.

Flowchart for defining avoidable ED visit and removing data with missingness or errors

Modeling Methods & Statistical Analysis

To determine the best performing method for the classification task, multiple machine learning methods were experimented, including logistic regression, random forest, gradient boosted tree (XGboost), and multi-layer perceptron (MLP). Random forest and XGboost models were tuned to improve accuracy. After training 4 models with each machine learning method, we compared the area under curve (AUC) of each model. Modeling was performed using python 3.8.

To understand the performance of our model in various key demographic groups, we tested our full model on the demographic-specific testing sets. In addition, we created demographic-specific models, by retraining on demographic-specific training sets, and tested those models on the corresponding demographic-specific testing sets.

In addition, we examined the importance of variables used in the XGboost model. We obtained the Shapley Additive Explanations (SHAP) values from each model using the shap python package. Individual SHAP plots for the main model is shown in Figure 2 and SHAP plots for demographic subgroups are included as supplementary figures.

Figure 2.

Feature Importance Chart of Full XGBoost Model.

Results

There are a total of 77,714 individuals in our sample population. There are more females (55%) than males in the data and the mean age is 37.06. 57% of the population is white, 23% is black, and 16% is Hispanic. Private insurance makes up 18% of the population, with 27% of patients using Medicare and 35% using Medicaid. In comparison to the white and black population, the Hispanic population was significantly younger, had decreased usage of Medicare and increased usage of Medicaid (Table 1).

Table 1:

Baseline characteristics of the sample population

Data	Visits	Non-ED%	Age (mean)	# Comorbidities (mean)	Race (White %)	Gender (Female %)	Insurance
Full	77714	4.44	37.06	1.05	57%	55%	34.6 (Medicaid) 27.1 (Medicare) 17.8 (Private Insurance)
White	44402	3.75	41.26	1.18	-	55.13%	26.34 (Medicaid) 23.24 (Medicare) 32(Private Insurance)
Black	17688	4.85	32.52	1.01	-	55.7%	43.8 (Medicaid) 12.2 (Medicare) 20 (Private Insurance)
Hispanic	12689	4.89	29.00	0.69	-	54.5%	50.6 (Medicaid) 7.67 (Medicare) 19.0 (Private Insurance)

We tested four types of models XGBoost, Random Forest, Logistic Regression, and MLP. Out of these models XGBoost was found to have the best testing AUC (Table 2). The training AUC of our full model is 0.723 and the testing AUC is 0.703. The AUC of the full model tested on demographic-specific groups is shown in Table 3. The AUC of the full model is comparable between the female-only (AUC = 0.706) and male-only (AUC = 0.700) testing sets, as well as between white (AUC = .6984) and black patients (AUC = .7006) testing sets. There is a decrease in model performance among the Hispanic group (AUC = .6724). Within the insurance subgroups, the performance of the model is highest in the Medicare-only group (AUC = 0.7289) and lower in the private insurance-only group (AUC = 0.685) and Medicaid-only group (AUC = 0.677).

Table 2:

Accuracy metrics for best machine learning model experimented in this study

Model	Training AUC	Testing AUC	Precision	Recall	F-Measure
XGBoost	0.723	0.703	0.975	0.635	0.769

Table 3a:

Race Models

Race, test set N	Training Set, N	Training AUC	Testing AUC
White, 13345	Full	0.7234	0.6984
	White-only, 31057	0.7282	0.7006
Black, 5295	Full	0.7234	0.7050
	Black-only, 12393	0.7390	0.6899
Hispanic, 3777	Full	0.7234	0.6724
	Hispanic-only, 8912	0.7634	0.6733

We identified several variables to have strong influence on model prediction of non-avoidable or avoidable ED visits. (Figure 2). Age, increasing number of total chronic diseases, and general as well as digestive symptoms all strongly influenced the model to label a visit as avoidable. Reasons for visit stated as injuries, psychiatric symptoms, and younger age strongly influenced the model to predict an avoidable visit. The time of day and day of week of the visit were not considered to be particularly important by our model. We also noted several differences in the feature importance of models trained on specific demographic subgroups. Genitourinary symptoms were strong predictors of non-avoidable ED visits in women (Supplementary Figure 1a). The presence of Psychiatric symptoms was seen as important in the Medicaid population, but not in either Medicare or Private insurance (Supplementary Figure 3).

Figure 3a.

Supplementary figure 3a. Private-Insurance only model.

Discussion

In this study, we developed a classification algorithm using machine learning methods to predict avoidable ED visits prior to the patients arriving at the ED. Our results suggest a moderate ability to predict avoidable ED visits using variables known prior to the point of triage, with a max testing AUC just over 0.70. There was little difference in model performance between genders, and between black and white populations, suggesting that our model is not sensitive to racial differences between the two groups. Model performance was worse in Hispanic, as well as patients on Medicaid, suggesting that our model may be biased against these populations. Compared to the Black and White populations, the Hispanic population was significantly younger, used Medicaid more often, and had less comorbidities. Differences in the Hispanic population within the data in comparison to Black and White groups as well as confounding factors not represented in the data such as language or cultural barriers, which have been noted in machine learning work involving Hispanic groups, may explain reduced model performance in this group⁹.

Some of the variables identified in figure 2 as strong predictors of avoidable/non-avoidable determination are consistent with existing knowledge surrounding avoidable ED visits. It is noted that the emergency department setting is not well equipped to treat mental health disorders and substance abuse⁷, which may explain why psychiatric symptoms were associated with avoidable ED visits. Our finding that pediatric age is associated with increased likelihood of avoidable ED visits is consistent with previous findings^10,11, potentially due to a mismatch in perceived severity of a child’s symptoms between parents and healthcare providers. The presence of Medicare insurance was not considered an impactful metric, although PAYTYPER 1(Private Insurance) was associated with increased avoidability and 5(Self-Pay) and -8(Unknown Payment) were associated with decreased avoidability. The time-of-day variable was not considered to be important in the model, suggesting that there is no difference in avoidable visits between overnight and daytime admissions. The day of week variable was also not considered to be important. It is known that patient access to primary care is an important factor in ED utilization. It has been shown that patients seek ED care during windows when primary care physicians are unavailable, such as overnight arrivals and weekend visits in single site studies^12,13. As our data comes from the national level, the impact of primary care availability that may be visible at a local level may be obscured.

Some features were more important in the race, gender, and insurance specific models compared to the base model. Genitourinary symptoms were important in females but not males, possibly due to an increased abundance of urinary tract infections and other gynecological symptoms that present more commonly in females. Arrival by ambulance was important in females but not males and digestive symptoms were more important in females than males. Women are known to have higher rates of digestive primary complaints compared to men (and lower admissions, but same when adjusted for age and comorbidities)¹⁴. Pregnant women have been noted to use emergency department services non-urgently¹⁵.

Our work represents the first attempt to use machine learning in the classification of avoidable emergency visits. We identified several factors associated with avoidable and non-avoidable ED visits that may be useful for clinicians and patients in managing their care options. For example, in a primary care or community clinic setting, our model could be deployed to provide patient guidance on whether to schedule a meeting with their physician or go to the emergency department.

The addition of longitudinal care data from healthcare providers would likely improve the accuracy of our model in this setting. Future work should examine the effect of additional variables known prior to the point of triage to improve predictive performance, such as information regarding prior ED utilization.

Figures & Table

Figure 1a.

Supplementary figure 1a. Female only model.

Figure 1b.

Supplementary figure 1b. Male only model.

Figure 2a.

Supplementary figure 2a. Black only model.

Figure 2b.

Supplementary figure 2b. Hispanic only model.

Figure 2c.

Supplementary figure 2c. White only model.

Figure 3b.

Supplementary figure 3b. Medicare only model.

Figure 3c.

Supplementary figure 3c. Medicaid only model.

Table 3b:

Gender Models

Gender, test set N	Training Set, N	Training AUC	Testing AUC
Female, 13119	Full	0.7234	0.7064
	Female-only, 30535	0.7304	0.7064
Male, 10651	Full	0.7234	0.6995
	Male-only, 24926	0.7193	0.6928

Table 3c:

Insurance Models

Insurance, test set N	Training Set, N	Training AUC	Testing AUC
Private, 6509	Full	0.7234	0.6848
	Private, 15109	0.7613	0.6742
Medicare, 4242	Full	0.7234	0.7289
	Medicare, 9858	0.8163	0.7220
Medicaid, 8225	Full	0.7234	0.6770
	Medicaid, 19082	0.7235	0.6617