Abstract
Background
Previous studies examining high-frequency ED utilization have primarily used single-center data, potentially leading to ascertainment bias if patients visit multiple centers. The goals of this study were (1) to create a predictive model to prospectively identify patients at risk of high-frequency ED utilization for asthma, and (2) to examine how that model differed using state-wide versus single-center data.
Methods
To track ED visits within a state, we analyzed 2011-2013 data from the New York State Healthcare Cost and Utilization Project (HCUP) State Emergency Department Databases (SEDD). The first year of data (2011) was used to determine prior utilization; 2012 was used to identify index ED visits for asthma and for demographics; and 2013 was used for outcome ascertainment. High-frequency utilization was defined as 4+ ED visits for asthma within one year after the index visit. We performed analyses separately for children (age <21 years) and adults, and constructed two models: one included all state-wide (multi-center) visits, and the other was restricted to index hospital (single-center) visits. Multivariable logistic regression models were developed from potential predictors selected a priori. The final model was chosen by evaluating model performance using AIC scores, 10-fold cross validation, and ROC curves.
Results
Among children, high-frequency ED utilization for asthma was observed in 2,417/94,258 (2.56%) using all state-wide visits, compared to 1,853/94,258 (1.97%) for index hospital visits only. Among adults, the corresponding results were 7,779/159,874 (4.87%) and 5,053/159,874 (3.16%), respectively. In the multi-center visit model, the area under the curve (AUC) from 10-fold cross validation for children was 0.70 (95% CI: 0.69-0.72), compared to 0.71 (95% CI: 0.69-0.72) in the single-center visit model. The corresponding AUC results for adults were 0.76 (95% CI: 0.76-0.77) and 0.76 (0.75-0.77), respectively.
Conclusion
Data available at the index ED visit can predict subsequent high-frequency utilization for asthma with AUC ranging from 0.70-0.76. Model accuracy was similar regardless of whether outcome ascertainment included all state-wide visits (multi-center) or was limited to the index hospital (single-center).
Background
Asthma is a significant contributor to high-frequency emergency department (ED) utilization for both adults and children.1-3 Approximately 19 million adults and 7 million children in the United States have active asthma.4 Among adults, public insurance, absence of insurance, and markers of asthma severity have been associated with increased frequency of ED utilization.5 Among children, the rate of ED visits for asthma increased by 13.3% from 2001-2010, with children who were from racial/ethnic minorities or who had government insurance at the highest risk for an asthma-related ED visit.6
However, our current strategies for identifying patients at risk of high-frequency ED utilization for asthma are quite limited. Patients are primarily identified after the pattern of utilization has occurred, when opportunities for intervention may be more limited. Successful identification of patients at risk of high-frequency ED utilization for asthma could assist in intervention development by providing a “high-risk” cohort for researchers.
Previously reported predictive models for asthma-related healthcare utilization have used heterogeneous samples with varying amounts of clinical and administrative data. Multiple models have found that prior ED or hospital utilization is predictive of subsequent use.7-12 However, many models used information about previous prescriptions 7,8,12 or detailed prior history and laboratory testing 12 that are unlikely to be available to an ED provider, 7,8,12 limiting their clinical utility. In addition, several studies were only able to ascertain return visits to a single center, potentially leading to ascertainment bias if patients visited a different hospital.10,11 This is of particular importance for emergency care as up to 32% of return visits could be to a different institution.13
To the best of our knowledge, there have been no direct comparisons between the health system models (with data that may not be available to an individual provider at the bedside) and the individual hospital models (which may suffer from ascertainment bias). For both clinicians and researchers, understanding the relative utility of those two approaches is key for interpreting studies reporting risk factors for high-frequency utilization, and designing future research.
Therefore, the goals of this study were (1) to derive a predictive model for high-frequency ED utilization for asthma using data typically available to an ED clinician at the time of the visit, and (2) to compare the performance of predictive models generated using two different methods for outcome ascertainment: one using all state-wide visits (capturing index and non-index hospitals) versus another using single center (index hospitals only) visits.
Methods
Study design
We analyzed data from the Healthcare Cost and Utilization Project (HCUP) State Emergency Department Databases (SEDD). The SEDD contains all ED visits for a state that resulted in discharge, transfer, or death in the ED. Individual patients were tracked through episodes of care using the de-identified patient ID (visitlink) provided by HCUP. Our analytical dataset included three years (2011-2013) of New York State data. New York SEDD was chosen for analysis because it has the most pediatric visits among states with patient-linked SEDD data.14
Patients were eligible for inclusion if they had an asthma visit, defined by ICD-9 codes 493.00 through 493.92, in the index year (2012). For each unique patient, the index visit was defined as the first asthma visit to the ED (as found in the SEDD) during the index year, and the hospital in which that visit occurred was identified as the index hospital. We excluded patients with an ED disposition indicating death since achievement of the outcome would not be possible. All analyses were performed separately for children (age <21 years) and adults. We chose 21 years of age to best reflect age-related treatment and triage patterns based on prior data showing that 64% of pediatric EDs have an age limit of <21 years.15
Outcome and exposure variables
Previous utilization year assessed during the 365 days preceding the index visit., The primary outcome, high-frequency utilization of the ED for asthma, was defined as 4 or more visits within 365 days after the index visit. This structure reduces the potential for seasonal variation or misinterpretation of utilization patterns within a single calendar year. We chose to define high-frequency utilization as 4 or more visits to be consistent with the prior literature.1,16,17
Exposure variables were selected a priori to test for a possible association with our outcome. Exposure variables included age, sex, race/ethnicity, insurance status, prior ED utilization, index visit month, time of presentation, day of week, disposition on index visit and median household income by ZIP code. We created four variables describing prior ED utilization dichotomized into the groups <4 and ≥4 visits to ED: (1) ED visits for any cause to any location (2) ED visits for asthma to any location (3) ED visits for any cause to index hospital and (4) ED visits for asthma to index hospital. For pediatric patients, we additionally created a variable to identify the presence of any complex chronic condition using the classification scheme developed by Feudtner.18
Statistical analysis
All analyses were performed using Stata 14.1 (Stata Corp, College Station, TX). To evaluate the associations between patient factors and high-frequency ED utilization for asthma, we performed unadjusted analyses using chi-square, Fisher's exact test, or Wilcoxon rank sum test, as appropriate. Data are presented as counts, proportions, and medians with interquartile ranges (IQR). All P-values were two-tailed, with P<0.05 considered statistically significant.
The goal of the multivariable modeling was to investigate differences in optimal model performance between state-wide and single center data. Therefore, two separate models were derived for each age category (children and adults). The first model included state-wide (index and non-index hospital) visits; the second model included single-center (index hospital only) visits. Data from all three years were randomly split into two sets: 50% to develop the model (derivation set) and 50% to measure its performance (test set). Primary analyses were conducted with a derivation set-test set approach. We used a 3-step approach. In the first step, all candidate model predictors were selected a priori for possible inclusion in the model, regardless of significance in the univariate analysis. In the second step, Akaike's Information Criterion (AIC) was chosen to select predictors for inclusion. Variables were checked for inclusion manually, one at a time, and the set of variables with the lowest AIC score was chosen. In the third stage, we examined the discrimination/calibration measures for this set of predictors and made changes to the model to get the highest AUC. The model was adjusted for overfitting/underfitting and the most parsimonious model was selected (Please see Appendix for variable selection table).
Final covariate selection was based on clinical importance, statistical significance, and the Akaike information criterion (AIC) model scores. After the multivariable model was finalized using the derivation set, its performance was reevaluated in the test set. Model results are reported as odds ratios (ORs) with 95% confidence intervals (95% CIs). We compared model performance in both sets using the area under the ROC curve (AUC) and the Hosmer-Lemeshow (H-L) goodness-of-fit test. For a secondary analysis of model performance, we then used 10-fold cross validation to calculate overall model performance using all data (i.e., derivation and test sets combined). The optimal cut-point to maximize sensitivity and specificity was identified in the derivation set and evaluated in the test set. This work was determined to be exempt from IRB review by the Children's Hospital of Philadelphia IRB.
Results
In the full cohort, high-frequency ED utilization for asthma was observed in 2,417/94,258 (2.56%) pediatric patients using state-wide visits, compared to 1,853/94,258 (1.97%) for index hospital visits only. Among adults, the corresponding results were 7,779/159,874 (4.87%) and 5,053/159,874 (3.16%), respectively. For children, there were 50,604 return ED visits for asthma, of which 42,435 (83%) were to the index ED. For adults, the corresponding numbers were 118,489 return visits, of which 86, 549 (73%) were to the index ED.
Pediatric
Pediatric patient characteristics for state-wide (index and non-index hospitals) and single-center (index-hospital only) ED visits for asthma are shown in Table 1. The median age was similar between children with and without high-frequency utilization (8 and 9 years, respectively). Of the children with high-frequency ED utilization, 72% were publicly insured, as compared to 57% of those without high-frequency ED utilization (P<0.001). In the multivariable logistic regression model developed in the derivation set for state-wide visits, sex, race/ethnicity, presence of a complex chronic condition, prior state-wide ED visits for any cause, index visit month, index visit day of week, and disposition were significant predictors of high-frequency ED utilization for asthma in the outcome year (Table 2). H-L goodness-of-fit test showed that the model's estimates fit the data at an acceptable level in the derivation and the test set. The AUCs for the derivation and the test sets were 0.71 (95% Cl: 0.69-0.73) and 0.70 (95% CI: 0.69-0.72), respectively (Figure 1). Using 10-fold cross validation, the AUC was 0.70 (95% CI: 0.69-0.72). Using the optimal cut-point to maximize sensitivity and specificity (0.02), the model had 61.4% sensitivity and 66.4% specificity with a negative predictive value of 98.5.
Table 1. Characteristics of pediatric patients by high-frequency emergency department utilization for asthma in the derivation set (n=47,128).
| State-wide Visits (index and non-index hospitals) |
Single-center Visits (index hospitals only) |
|||||
|---|---|---|---|---|---|---|
| High-frequency utilization | High-frequency utilization | |||||
| Yes (n=1,202) |
No (n=45,926) |
Yes (n=922) |
No (n=46,206) |
|||
| n (%) | n (%) | p-value | n (%) | n (%) | p-value | |
| Age at index visit in years, median (IQR) | 8 (3-17) | 9 (4-15) | 0.89 | 7 (3-16) | 9 (4-15) | 0.03 |
| Sex | <0.001 | 0.01 | ||||
| Male | 594 (49) | 25,384 (55) | 470 (51) | 25,508 (55) | ||
| Female | 608 (51) | 20,542 (45) | 452 (49) | 20,698 (45) | ||
| Race/ethnicity | <0.001 | <0.001 | ||||
| White | 182 (15) | 11,520 (25) | 128 (14) | 11,574 (25) | ||
| Black | 512 (43) | 16,146 (35) | 372 (40) | 16,286 (35) | ||
| Hispanic | 381 (32) | 11,732 (26) | 317 (34) | 11,796 (26) | ||
| Other | 127 (11) | 6,528 (14) | 105 (11) | 6,550 (14) | ||
| Median household income by patient ZIP code | <0.001 | <0.001 | ||||
| <$39,000 | 702 (58) | 20,386 (44) | 544 (59) | 20,544 (45) | ||
| $39,000 - $47,999 | 196 (16) | 8,466 (18) | 156 (17) | 8,506 (18) | ||
| $48,000 - $62,999 | 167 (14) | 8,150 (18) | 133 (14) | 8,184 (18) | ||
| $63,000 and over | 137 (11) | 8,924 (19) | 89 (10) | 8,972 (19) | ||
| Primary expected payer | <0.001 | <0.001 | ||||
| Public Insurance (Medicare/Medicaid) | 863 (72) | 26,012 (57) | 682 (74) | 26,193 (57) | ||
| Private Insurance | 209 (17) | 14,013 (31) | 154 (17) | 14,068 (30) | ||
| Self-pay | 113 (9) | 4,810 (10) | 74 (8) | 4,849 (10) | ||
| No charge/other | 17 (1) | 1,091 (2) | 12 (1) | 1,096 (2) | ||
| Any co-morbidity | 39 (3) | 952 (2) | 0.01 | 33 (4) | 958 (2) | 0.002 |
| Prior state-wide ED visits for any cause (≥ 4 ED visits) | 415 (35) | 3,088 (7) | <0.001 | 321 (35) | 3,182 (7) | <0.001 |
| Prior state-wide ED visits for asthma (≥ 4 ED visits) | 143 (12) | 302 (1) | <0.001 | 117 (13) | 328 (1) | <0.001 |
| Prior single-center ED visits for any cause (≥ 4 ED visits) | 320 (27) | 2,173 (5) | <0.001 | 278 (30) | 2,215 (5) | <0.001 |
| Prior single-center ED visits for asthma (≥ 4 ED visits) | 115 (10) | 236 (1) | <0.001 | 105 (11) | 246 (1) | <0.001 |
| Index visit months by quarter | <0.001 | <0.001 | ||||
| January - March | 565 (47) | 13,823 (30) | 443 (48) | 13,945 (30) | ||
| April - June | 279 (23) | 11,672 (25) | 217 (24) | 11,734 (25) | ||
| July - September | 177 (15) | 8,931 (19) | 137 (15) | 8,971 (19) | ||
| October - December | 181 (15) | 11,500 (25) | 125 (14) | 11,556 (25) | ||
| Index visit time of presentation | 0.84 | 0.63 | ||||
| 1 am - 6 am | 123 (10) | 4,683 (10) | 89 (10) | 4,717 (10) | ||
| 7 am - 12 pm | 344 (29) | 12,617 (27) | 270 (29) | 12,691 (27) | ||
| 1 pm - 6 pm | 370 (31) | 14,347 (31) | 278 (30) | 14,439 (31) | ||
| 7 pm - 12 am | 365 (30) | 14,279 (31) | 285 (31) | 14,359 (31) | ||
| Day of index visit | 0.05 | 0.03 | ||||
| Monday - Friday | 893 (74) | 32,908 (72) | 691 (75) | 33,110 (72) | ||
| Saturday - Sunday | 309 (26) | 13,018 (28) | 231 (25) | 13,096 (28) | ||
| Disposition of patient at index visit | <0.001 | 0.01 | ||||
| routine discharge from hospital | 1,175 (98) | 44,899 (98) | 905 (98) | 45,169 (98) | ||
| transfers to other facility/hospital/home healthcare | 12 (1) | 820 (2) | - | - | ||
| left against medical advice | 15 (1) | 207 (0.5) | - | - | ||
| Other* | - | - | 17 (2) | 1,037 (2) | ||
| Time between visits from index to first visit in days | ||||||
| no visits after index | 0 (0) | 22,488 (49) | 0 (0) | 22,488 (49) | ||
| ≤ 3 | 141 (12) | 1,455 (3) | 110 (12) | 1,486 (3) | ||
| > 3 | 1,061 (88) | 21,983 (48) | 812 (88) | 22,232 (48) | ||
| Time between visits from index to first visit in days | ||||||
| no visits after index | 0 (0) | 22,488 (49) | 0 (0) | 22,488 (49) | ||
| ≤ 10 | 262 (22) | 2,713 (6) | 196 (21) | 2,779 (6) | ||
| > 10 | 940 (78) | 20,725 (45) | 726 (79) | 20,939 (45) | ||
Data presented as no. (%) unless otherwise noted. Percentage totals may not equal 100% because of rounding.
Includes transfers to other facility/hospital/home healthcare and left against medical advice. Categories combined in single-center dataset due to low cell counts for each group.
Abbreviations: ED, emergency department; IQR, inter-quartile range
Table 2. Multivariable logistic regression models predicting high-frequency emergency department utilization for asthma among pediatric patients for State-wide and Single-center Visits (n=94,258).
| State-wide Visits (index and non-index hospitals) |
Single-center Visits (index hospitals only) |
|||||||
|---|---|---|---|---|---|---|---|---|
| Derivation set | Test set | Derivation set | Test set | |||||
| OR (95%CI) | p - value | OR (95%CI) | p-value | OR (95%CI) | p-value | OR (95%CI) | p-value | |
| Female | 1.18 (1.05-1.33) | 0.01 | 1.14 (1.02-1.28) | 0.02 | 1.11 (0.97-1.27) | 0.12 | 1.03 (0.90-1.18) | 0.63 |
| Race/ethnicity | ||||||||
| White | Reference | Reference | Reference | Reference | ||||
| Black | 1.93 (1.62-2.29) | <0.001 | 1.62 (1.37-1.91) | <0.001 | 1.98 (1.62-2.43) | <0.001 | 1.52 (1.25-1.84) | <0.001 |
| Hispanic | 1.82 (1.52-2.19) | <0.001 | 1.61 (1.35-1.91) | <0.001 | 2.17 (1.76-2.68) | <0.001 | 1.65 (1.36-2.02) | <0.001 |
| Other | 1.17 (0.93-1.48) | 0.18 | 1.20 (0.97-1.49) | 0.10 | 1.39 (1.07-1.81) | 0.01 | 1.25 (0.98-1.59) | 0.07 |
| Any co-morbidity | 1.54 (1.10-2.15) | 0.01 | 1.26 (0.90-1.78) | 0.17 | 1.72 (1.20-2.48) | 0.003 | 1.45 (1.01-2.08) | 0.05 |
| Prior state-wide ED visits for any cause (≥ 4 ED visits) | 6.34 (5.58-7.20) | <0.001 | 6.91 (6.09-7.84) | <0.001 | 6.17 (5.35-7.12) | <0.001 | 6.71 (5.82-7.73) | <0.001 |
| Index visit months by quarter | ||||||||
| January – March | Reference | Reference | Reference | Reference | ||||
| April – June | 0.66 (0.57-0.77) | <0.001 | 0.70 (0.60-0.80) | <0.001 | 0.66 (0.56-0.78) | <0.001 | 0.68 (0.58-0.80) | <0.001 |
| July – September | 0.62 (0.52-0.74) | <0.001 | 0.58 (0.49-0.70) | <0.001 | 0.62 (0.51-0.75) | <0.001 | 0.54 (0.44-0.67) | <0.001 |
| October - December | 0.50 (0.42-0.60) | <0.001 | 0.49 (0.41-0.58) | <0.001 | 0.44 (0.36-0.55) | <0.001 | 0.45 (0.37-0.55) | <0.001 |
| Day of index visit | ||||||||
| Monday – Friday | Reference | Reference | ||||||
| Saturday - Sunday | 0.93 (0.81-1.06) | 0.28 | 0.84 (0.74-0.97) | 0.01 | - | - | ||
| Disposition of patient at index visit | ||||||||
| routine discharge from hospital | Reference | Reference | ||||||
| transfers to other facility/hospital/home healthcare | 0.59 (0.33-1.06) | 0.08 | 0.75 (0.45-1.27) | 0.29 | - | - | ||
| left against medical advice | 2.22 (1.29-3.85) | 0.004 | 0.76 (0.31-1.88) | 0.55 | - | - | ||
Abbreviations: OR, odds ratio; 95%CI, 95% confidence interval; ED, emergency department
Figure 1. ROC curves from State-wide and Single-center visit test sets predicting high-frequency ED utilization for asthma among pediatric patients.
a. State-wide visits (index and non-index hospitals)
b. Single-center visits (index hospitals only)
For the pediatric single-center model, race/ethnicity, presence of a complex chronic condition, prior state-wide ED visits for any cause, and index visit month were significant predictors of high-frequency ED utilization for asthma in the outcome year (Table 2). The AUCs for the derivation and the test sets were 0.71 (95% Cl: 0.70-0.73) and 0.70 (95% CI: 0.69-0.72), respectively (Figure 1). Using 10-fold cross validation, the AUC was 0.71 (95%CI: 0.69-0.72). Using the optimal cut-point to maximize sensitivity and specificity (0.02), sensitivity was 59.1%, specificity 69.5% and the negative predictive value was 98.8.
Adult
Adult patient characteristics for state-wide and single-center ED visits for asthma are shown in Table 3. Of the patients with high-frequency utilization, 69% were on Medicare or Medicaid, as compared to 51% of those without high-frequency utilization (P<0.001). In the multivariable logistic regression model developed in the derivation set for state-wide visits, age at index visit, sex, median income by patient ZIP code, primary insurance, prior state-wide ED visits for any cause, and index visit month were significant predictors of high-frequency ED utilization for asthma (Table 4). The AUCs for the derivation and the test sets were 0.76 (95% Cl: 0.76-0.77) and 0.76 (95% CI: 0.75-0.77), respectively (Figure 2). Using 10-fold cross validation, the AUC was 0.76 (95% CI: 0.76-0.77). Using the optimal cut-point to maximize sensitivity and specificity (0.04), sensitivity was 67.1%, specificity 70.7% and negative predictive value 97.7%.
Table 3. Characteristics of adult patients by high-frequency emergency department utilization for asthma in the derivation set (n=79,940).
| State-wide Visits (index and non-index hospitals) |
Single-center Visits (index hospitals only) |
|||||
|---|---|---|---|---|---|---|
| High-frequency utilization | High-frequency utilization | |||||
| Yes (n=3,918) |
No (n=76,022) |
Yes (n=2,564) |
No (n=77,376) |
|||
| n (%) | n (%) | p-value | n (%) | n (%) | p-value | |
| Age at index visit in years, median (IQR) | 40 (29-50) | 42 (29-54) | <0.001 | 40 (29-51) | 42 (29-54) | <0.001 |
| Sex | <0.001 | 0.01 | ||||
| Male | 1,396 (36) | 24,528 (32) | 896 (35) | 25,028 (32) | ||
| Female | 2,522 (64) | 51,494 (68) | 1,668 (65) | 52,348 (68) | ||
| Race/ethnicity | <0.001 | <0.001 | ||||
| White | 1,094 (28) | 27,227 (36) | 739 (29) | 27,582 (36) | ||
| Black | 1,593 (41) | 22,845 (30) | 1,005 (39) | 23,433 (30) | ||
| Hispanic | 836 (21) | 16,495 (22) | 570 (22) | 16,761 (22) | ||
| Other | 395 (10) | 9,455 (12) | 250 (10) | 9,600 (12) | ||
| Median household income by patient ZIP code | <0.001 | <0.001 | ||||
| <$39,000 | 2,175 (56) | 30,995 (41) | 1,454 (57) | 31,716 (41) | ||
| $39,000 - $47,999 | 653 (17) | 14,821 (20) | 438 (17) | 15,036 (19) | ||
| $48,000 - $62,999 | 570 (15) | 14,269 (19) | 375 (15) | 14,464 (19) | ||
| $63,000 and over | 520 (13) | 15,937 (21) | 297 (12) | 16,160 (21) | ||
| Primary expected payer | <0.001 | <0.001 | ||||
| Medicare | 617 (16) | 13,035 (17) | 431 (17) | 13,221 (17) | ||
| Medicaid | 2,062 (53) | 25,995 (34) | 1,332 (52) | 26,725 (35) | ||
| Private Insurance | 562 (14) | 22,751 (30) | 359 (14) | 22,954 (30) | ||
| Self-pay | 561 (14) | 11,479 (15) | 362 (14) | 11,678 (15) | ||
| No charge/other | 116 (3) | 2,762 (4) | 80 (3) | 2,798 (4) | ||
| Prior state-wide ED visits for any cause (≥ 4 ED visits) | 1,829 (47) | 7,814 (10) | <0.001 | 1,196 (47) | 8,447 (11) | <0.001 |
| Prior state-wide ED visits for asthma (≥ 4 ED visits) | 787 (20) | 828 (1) | <0.001 | 561 (22) | 1,054 (1) | <0.001 |
| Prior single-center ED visits for any cause (≥ 4 ED visits) | 1,196 (31) | 4,254 (6) | <0.001 | 937 (37) | 4,513 (6) | <0.001 |
| Prior single-center ED visits for asthma (≥ 4 ED visits) | 528 (13) | 480 (1) | <0.001 | 452 (18) | 556 (1) | <0.001 |
| Index visit months by quarter | <0.001 | <0.001 | ||||
| January - March | 1,973 (50) | 21,710 (29) | 1,344 (52) | 22,339 (29) | ||
| April - June | 913 (23) | 19,817 (26) | 585 (23) | 20,145 (26) | ||
| July - September | 586 (15) | 17,177 (23) | 362 (14) | 17,401 (22) | ||
| October - December | 446 (11) | 17,318 (23) | 273 (11) | 17,491 (23) | ||
| Index visit time of presentation | <0.001 | 0.02 | ||||
| 1 am - 6 am | 484 (12) | 8,200 (11) | 312 (12) | 8,372 (11) | ||
| 7 am - 12 pm | 1,167 (30) | 24,103 (32) | 800 (31) | 24,470 (32) | ||
| 1 pm - 6 pm | 1,236 (32) | 25,024 (33) | 787 (31) | 25,473 (33) | ||
| 7 pm - 12 am | 1,031 (26) | 18,695 (25) | 665 (26) | 19,061 (25) | ||
| Day of index visit | 0.98 | 0.92 | ||||
| Monday - Friday | 2,874 (73) | 55,751 (73) | 1,878 (73) | 56,747 (73) | ||
| Saturday - Sunday | 1,044 (27) | 20,271 (27) | 686 (27) | 20,629 (27) | ||
| Disposition of patient at index visit | <0.001 | <0.001 | ||||
| routine discharge from hospital | 3,766 (96) | 72,901 (96) | 2,486 (97) | 74,181 (96) | ||
| transfers to other facility/hospital/home healthcare | 37 (1) | 1,465 (2) | 14 (1) | 1,488 (2) | ||
| left against medical advice | 115 (3) | 1,656 (2) | 64 (3) | 1,707 (2) | ||
| Time between visits from index to first visit in days | ||||||
| no visits after index | 0 (0) | 33,974 (45) | 0 (0) | 33,974 (44) | ||
| ≤ 3 | 460 (12) | 2,935 (4) | 267 (10) | 3,128 (4) | ||
| > 3 | 3,458 (88) | 39,113 (51) | 2,297 (90) | 40,274 (52) | ||
| Time between visits from index to first visit in days | ||||||
| no visits after index | 0 (0) | 33,974 (45) | 0 (0) | 33,974 (44) | ||
| ≤ 10 | 1,049 (27) | 6,102 (8) | 668 (26) | 6,483 (8) | ||
| > 10 | 2,869 (73) | 35,946 (47) | 1,896 (74) | 36,919 (48) | ||
Data presented as no. (%) unless otherwise noted. Percentage totals may not equal 100% because of rounding.
Abbreviations: IQR, Inter-quartile range; ED, emergency department
Table 4. Multivariable logistic regression models predicting high-frequency emergency department utilization for asthma among adult patients for State-wide and Single-center Visits (n=159,874).
| State-wide Visits (index and non-index hospitals) |
Single-center Visits (index hospitals only) |
|||||||
|---|---|---|---|---|---|---|---|---|
| Derivation set | Test set | Derivation set | Test set | |||||
| OR (95%CI) | p-value | OR (95%CI) | p-value | OR (95%CI) | p-value | OR (95%CI) | p- value | |
| Age at index visit in years | 0.99 (0.992-0.997) | <0.001 | 0.99 (0.991-0.996) | <0.001 | - | - | ||
| Female | 0.81 (0.76-0.87) | <0.001 | 0.78 (0.73-0.84) | <0.001 | 0.85 (0.78-0.93) | <0.001 | 0.81 (0.74-0.88) | <0.001 |
| Race/ethnicity | ||||||||
| White | Reference | Reference | ||||||
| Black | - | - | 1.24 (1.12-1.38) | <0.001 | 1.20 (1.08-1.34) | 0.001 | ||
| Hispanic | - | - | 1.04 (0.92-1.17) | 0.54 | 1.15 (1.02-1.29) | 0.02 | ||
| Other | - | - | 0.88 (0.75-1.02) | 0.09 | 0.81 (0.69-0.95) | 0.01 | ||
| Median household income by patient ZIP code | ||||||||
| <$39,000 | Reference | Reference | Reference | Reference | ||||
| $39,000 - $47,999 | 0.69 (0.63-0.76) | <0.001 | 0.68 (0.62-0.74) | <0.001 | 0.73 (0.65-0.82) | <0.001 | 0.69 (0.61-0.77) | <0.001 |
| $48,000 - $62,999 | 0.68 (0.62-0.75) | <0.001 | 0.70 (0.64-0.78) | <0.001 | 0.69 (0.61-0.77) | <0.001 | 0.69 (0.61-0.78) | <0.001 |
| $63,000 and over | 0.63 (0.57-0.70) | <0.001 | 0.69 (0.62-0.76) | <0.001 | 0.56 (0.49-0.64) | <0.001 | 0.62 (0.54-0.70) | <0.001 |
| Primary expected payer | ||||||||
| Medicare | Reference | Reference | Reference | Reference | ||||
| Medicaid | 1.20 (1.07-1.33) | 0.001 | 1.24 (1.11-1.39) | <0.001 | 1.17 (1.04-1.32) | 0.01 | 1.22 (1.09-1.38) | 0.001 |
| Private Insurance | 0.61 (0.54-0.70) | <0.001 | 0.60 (0.53-0.68) | <0.001 | 0.62 (0.54-0.72) | <0.001 | 0.65 (0.56-0.76) | <0.001 |
| Self-pay | 0.99 (0.87-1.13) | 0.9 | 1.06 (0.93-1.22) | 0.38 | 1.00 (0.86-1.15) | 0.95 | 1.08 (0.93-1.26) | 0.31 |
| No charge/other | 0.87 (0.71-1.08) | 0.21 | 1.01 (0.81-1.24) | 0.95 | 0.94 (0.73-1.20) | 0.29 | 0.96 (0.74-1.24) | 0.76 |
| Prior state-wide ED visits for any cause (≥ 4 ED visits) | 5.79 (5.39-6.21) | <0.001 | 5.98 (5.57-6.42) | <0.001 | 5.31 (4.88-5.78) | <0.001 | 5.45 (5.00-5.94) | <0.001 |
| Index visit months by quarter | ||||||||
| January - March | Reference | Reference | Reference | Reference | ||||
| April - June | 0.61 (0.56-0.67) | <0.001 | 0.60 (0.55-0.65) | <0.001 | 0.58 (0.53-0.65) | <0.001 | 0.54 (0.49-0.60) | <0.001 |
| July - September | 0.48 (0.43-0.52) | <0.001 | 0.47 (0.42-0.52) | <0.001 | 0.44 (0.39-0.50) | <0.001 | 0.42 (0.37-0.48) | <0.001 |
| October - December | 0.40 (0.36-0.45) | <0.001 | 0.43 (0.38-0.47) | <0.001 | 0.37 (0.32-0.42) | <0.001 | 0.39 (0.34-0.45) | <0.001 |
| Index visit time of presentation | ||||||||
| 1 am - 6 am | Reference | Reference | ||||||
| 7 am - 12 pm | - | - | 0.90 (0.79-1.04) | 0.15 | 0.89 (0.77-1.02) | 0.09 | ||
| 1 pm - 6 pm | - | - | 0.85 (0.74-0.98) | 0.02 | 0.84 (0.73-0.96) | 0.01 | ||
| 7 pm - 12 am | - | - | 0.95 (0.82-1.09) | 0.46 | 0.89 (0.77-1.02) | 0.10 | ||
| Day of index visit | ||||||||
| Monday - Friday | Reference | Reference | ||||||
| Saturday - Sunday | 1.05 (0.97-1.13) | 0.25 | 1.07 (0.99-1.15) | 0.09 | - | - | ||
Abbreviations: OR, odds ratio; 95%CI, 95% confidence interval; ED, emergency department
Figure 2. ROC curves from State-wide and Single-center visit test sets predicting high-frequency ED utilization for asthma among adult patients.
a. State-wide visits (index and non-index hospitals)
b. Single-center visits (index hospitals only)
For the adult single-center model, sex, race/ethnicity, median income by patient ZIP code, primary insurance, prior state-wide ED utilization for any cause, index visit month, and index visit time were associated with high-frequency ED utilization for asthma (Table 4). The AUCs for the derivation and the test sets were 0.76 (95% Cl: 0.75-0.77) and 0.76 (0.75-0.77), respectively (Figure 2). Using 10-fold cross validation, the AUC was 0.76 (95% CI: 0.75-0.77). Using the optimal cut-point to maximize sensitivity and specificity (0.03), sensitivity was 66.5%, specificity 71.5% and negative predictive value 98.5.
Discussion
Administrative data available at the index ED visit predicted subsequent ED utilization for asthma with AUC ranging from 0.70-0.76. As compared to the pediatric models, the adult models were more influenced by age, income, and prior any-cause ED utilization, rather than asthma-specific utilization. As expected, in both the adult and pediatric populations, fewer patients were categorized as high-utilizers in the model that used single-center visits. However, both state-wide and single-center models performed similarly. Despite the under-identification of high-frequency utilization, single center data can still be useful for identifying risk factors.
Previously specified predictive models have reported a range of model performances from an AUC of 0.69, with sensitivity of 48% and specificity of 82%7 to sensitivity of 25% and specificity of 92%.8 Hanson et al. found that prior acute care visits for asthma predicted subsequent acute care visits with an AUC of 0.75, 10 after adjustment for age, sex, race, and insurance status; and the association between prior ED visits and subsequent ED visits for asthma has been shown elsewhere as well.11 One study using a clinical score found an AUC of 0.75 for predicting exacerbation, 9 and a very complex model utilizing multiple clinical risk factors had an AUC of 0.87 for predicting 4 or more asthma attacks in 2 years.12 Given that the current study was limited to administrative data, the AUCs of 0.70-0.76 are consistent with, or a slight improvement on prior results.
As compared to the pediatric models, the adult models were more influenced by age, income and primary insurance, potentially describing cost pressures in adults that are reduced in children. Qualitative interviews suggest that adult patients with asthma choose the ED for a number of reasons that are related to income—including concerns about insurance status, lack of medication and inability to access outpatient providers.19 The absence of an association with age in the pediatric models differs from prior reports. High-frequency ED utilization for any cause has been associated with younger age in multiple pediatric studies.1,16 However, the highest-utilization group in those studies is children < 1 year of age, who would not be diagnosed with asthma, potentially explaining why age was not significantly associated in our cohort.
From a clinical perspective, predictive models such as the one described here could be added to electronic medical record systems to flag potential high-risk patients for increased case management, primary care or specialty referral resources. For example, one study found that patient navigators reduced primary care related ED visits over 12 months, but not over the full 24 months of follow-up, for patients aged 18-65 with public insurance or who were uninsured.20 In that study, the savings were greater than the costs of the program.20 For patients with asthma, reported reasons for ED use include lack of asthma medications and inability to access outpatient providers because of time availability, referral of severe cases to the ED or insurance status, as well as perceptions of appropriate locations of care based on symptom severity.19 Interventions to address high-frequency utilization will need to consider the issues of access and availability, as well as unmet educational needs. One systematic review examining non-ED interventions to reduce ED visits found that educational interventions had the largest effect.21 A successful educational intervention will need careful attention to the role of limited health literacy,22,23 a driver of ED use for both adults and children.
For researchers, a pre-defined cohort at increased risk for high-frequency utilization, identified at their index visit, would be the ideal population in which to study interventions to reduce ED utilization for asthma and improve systems of care. In particular, this patient population could be targeted for efforts to ensure that guideline-recommended management was optimized, as adult patients with high-frequency utilization of the ED for asthma are often not receiving recommended therapies.5
Of additional importance for researchers is the finding that the model performance was similar between state-wide and single-center visits. As expected, in both the adult and pediatric populations, fewer patients were categorized as high-utilizers in the model that used single-center visits. A study of patients with high ED utilization in Maryland found that 18% of the individuals with more than 5 ED visits visited more than 3 hospitals, and only 59% made more than 5 visits to any one hospital, suggesting that the rest would be under-identified by single-center studies.24 Using HCUP data, Duseja et al. found that 32% of ED revisits (within 3 days of an index visit) were at a different institution.13 Surprisingly, however, in our study both state-wide and single-center models performed similarly. This may be because patients with asthma are more likely to return to the same institution, as it represents an acute exacerbation of a chronic disease for which the patient may be undergoing treatment at a particular center. Overall, these data suggest that, despite the potential for under-identification of high-frequency utilization, single center data can still be useful for identifying risk factors and developing models to identify high-risk groups. We hope that these results will encourage centers to investigate the creation and utilization of models to identify patients at high risk of ED utilization for asthma, even if only single-center data is available.
Limitations
The large sample size in this study allowed for robust modeling, including easily collectable and available covariates. However, the absence of clinical data in this administrative dataset likely limits the strength of the predictive model. For example, we were able to adjust for complex chronic conditions in the pediatric analyses, but not for Elixhauser comorbidities in the adult analyses, due to the absence of diagnosis related group (DRG) data in SEDD. We also did not have data on underlying severity or chronic treatment of asthma. Additional limitations of the dataset include minimal data on socioeconomic variables, resulting in the utilization of median income by patient ZIP code, a proxy measure of actual household income. Furthermore, we defined index hospital as the first hospital the patient visited for asthma in the index year, which could result in the index hospital not being their primary site of care (if that visit was to another hospital by chance, or if they move). Subsequent analyses could consider using claims data to define index hospital based on frequency of individual hospital use based on billing information. Although we investigated single-center and state-wide models separately, we used prior state-wide visits for any cause as a potential predictor, which is unlikely to be available to a single-center unless reported by the patient. Finally, the asthma care patterns in New York may not be fully generalizable to other states.
Conclusion
Data available at the index ED visit can predict subsequent high-frequency ED utilization for asthma with AUC ranging from 0.70-0.76. Model accuracy was similar regardless of whether outcome ascertainment included state-wide visits or was limited to index hospital visits. These data suggest that predictive models could be used to identify patients who may require additional resources, and create a high-risk cohort for subsequent research. In addition, single center outcome ascertainment of return visits appears adequate for identifying a cohort of patients at increased risk of high-frequency ED utilization for asthma.
Acknowledgments
We would like to thank Michael Kallan for assistance with the database preparation.
Financial support: Dr. Samuels-Kalow was supported by NIH K12 HL10900904 for the first year of the study.
Appendix.
Variable Selection Table
Approach To Model Selection
Step 1. All candidate model predictors were selected a priori for possible inclusion in the model.
Step 2 Akaike's Information Criterion (AIC) was chosen to select predictors.
Step 3. We examined at discrimination and calibration measures, and added or removed predictors to achieve the best predictive model based on discrimination/calibration results.
The model was adjusted for overfit/underfit when comparing training/test datasets and the most parsimonious model selected
| Predictor No. | Predictors | |
|---|---|---|
| Prior high-frequency ED utilization: | ||
| 1 | ED visits for any cause to index and non-index hospitals, | |
| 2 | ED visits for asthma to index and non-index hospitals, | |
| 3 | ED visits for any cause to index hospital, | |
| 4 | ED visits for asthma to index hospital | |
|
| ||
| 5 | Sex | |
|
| ||
| 6 | Race/ethnicity | |
|
| ||
| 7 | Primary expected payer | |
|
| ||
| 8 | Median household income by patient ZIP code | |
|
| ||
| 9 | Index visit months by quarter | |
|
| ||
| 10 | Disposition of patient at index visit | |
|
| ||
| 11 | Day of admission at index visit | |
| 12 | Any comorbidity | |
|
| ||
| 13 | Time of presentation/admission hour at index visit | |
|
| ||
| 14 | Age | |
|
| ||
| Model selection steps for state-wide visit pediatric model | ||
|
| ||
| Step 1 | All candidate model predictors (1-14) considered for possible inclusion in the model. | |
|
| ||
| Step 2 Results | AIC Score | Combination of candidate predictors (1-14) were manually checked to achieve model with lowest AIC score. For example, |
|
| ||
| 10432.8 | 1-2 | |
| 10279.8 | 1-3 | |
| 10280.4 | 1-4 | |
| 10282.3 | 1-5 | |
| 10274.5 | 1-6… | |
| selected model | 10097.8 | 1-12, 14 |
|
| ||
| Step 3 Results | AUC Score | We examined discrimination and calibration measures, and added or removed predictors to achieve the best predictive model based on discrimination/calibration results; adjusted for overfit/underfit; most parsimonious model selected |
| overfit | 0.734 | 1-12,14 |
| overfit | 0.732 | 1-12 |
| overfit | 0.732 | 1-3, 5-12 |
| overfit | 0.73 | 1-2, 5-12… |
| selected model | 0.71 | 1, 5, 6, 9-12 |
| Model selection steps for single-center visit pediatric model | ||
|
| ||
| Step 1 | All candidate model predictors (1-14) were considered for possible inclusion in the model. | |
|
| ||
| Step 2 Results | AIC Score | Combination of candidate predictors (1-14) were manually checked to achieve model with lowest AIC score. For example, |
|
| ||
| 8493.9 | 1-2 | |
| 8361.4 | 1-3 | |
| 8332.6 | 1-5 | |
| 8330.7 | 1-6 | |
| 8175.2 | 1-12,14… | |
| selected model | 8172.3 | 1-9, 11, 12, 14 |
|
| ||
| Step 3 Results | AUC Score | We examined discrimination and calibration measures, and added or removed predictors to achieve the best predictive model based on discrimination/calibration results; adjusted for overfit/underfit; most parsimonious model selected |
|
| ||
| overfit | 0.74 | 1-9, 11, 12, 14 |
| overfit | 0.74 | 1-9, 12, 14 |
| overfit | 0.73 | 1-9, 12 |
| overfit | 0.73 | 1, 5-9, 12… |
| selected model | 0.71 | 1, 5, 6, 9, 12 |
|
| ||
| Model selection steps for state-wide visit adult model | ||
|
| ||
| Step 1 | All candidate model predictors (1-11, 13, 14) were considered for possible inclusion in the model. | |
|
| ||
| Step 2 Results | AIC Score | Combination of candidate predictors (1-11, 13, 14) were manually checked to achieve model with lowest AIC score. For example, |
|
| ||
| 27246.7 | 1-2 | |
| 27236.1 | 1-3 | |
| 27216.1 | 1-5 | |
| 26545.3 | 1-10… | |
| selected model | 26523.3 | 1-10, 13, 14 |
|
| ||
| Step 3 Results | AUC Score | We examined discrimination and calibration measures, and added or removed predictors to achieve the best predictive model based on discrimination/calibration results; adjusted for overfit/underfit; most parsimonious model selected |
|
| ||
| overfit | 0.77 | 1-10, 13, 14 |
| overfit | 0.77 | 1-5, 6-13, 14 |
| overfit | 0.76 | 1-5, 6-13 |
| overfit | 0.76 | 1, 5-13… |
| selected model | 0.76 | 1, 5, 7-9, 11 |
|
| ||
| Model selection steps for single-center visit adult model | ||
|
| ||
| Step 1 | All candidate model predictors (1-11, 13, 14) were considered for possible inclusion in the model. | |
|
| ||
| Step 2 Results | AIC Score | Combination of candidate predictors (1-11, 13, 14) were manually checked to achieve model with lowest AIC score. For example, |
|
| ||
| 20019.8 | 1-2 | |
| 19854.7 | 1-3 | |
| 19810 | 1-5 | |
| 19352 | 1-9… | |
| selected model | 19328 | 1-10, 14 |
|
| ||
| Step 3 Results | AUC Score | We examined discrimination and calibration measures, and added or removed predictors to achieve the best predictive model based on discrimination/calibration results; adjusted for overfit/underfit; most parsimonious model selected |
|
| ||
| overfit | 0.77 | 1-10, 14 |
| overfit | 0.77 | 1-10 |
| overfit | 0.77 | 1, 3, 5-10 |
| overfit | 0.76 | 1, 3, 5-9… |
| selected model | 0.76 | 1, 5-9, 13 |
Footnotes
Presentation: This work was presented, in part, at the Society for Academic Emergency Medicine and Pediatric Academic Societies annual meetings in May 2017.
Conflicts of interest: The authors report no conflicts of interest.
References
- 1.Alpern ER, Clark AE, Alessandrini EA, et al. Recurrent and high-frequency use of the emergency department by pediatric patients. Acad Emerg Med. 2014;21(4):365–373. doi: 10.1111/acem.12347. [DOI] [PubMed] [Google Scholar]
- 2.Vinton DT, Capp R, Rooks SP, Abbott JT, Ginde AA. Frequent users of US emergency departments: characteristics and opportunities for intervention. Emerg Med J. 2014;31(7):526–532. doi: 10.1136/emermed-2013-202407. [DOI] [PubMed] [Google Scholar]
- 3.Sun BC, Burstin HR, Brennan TA. Predictors and outcomes of frequent emergency department users. Acad Emerg Med. 2003;10(4):320–328. doi: 10.1111/j.1553-2712.2003.tb01344.x. [DOI] [PubMed] [Google Scholar]
- 4.Asthma Facts—CDC's National Asthma Control Program Grantees. 2013 http://www.cdc.gov/asthma/pdfs/asthma_facts_program_grantees.pdf.
- 5.Hasegawa K, Sullivan AF, Tovar Hirashima E, et al. A multicenter observational study of US adults with acute asthma: who are the frequent users of the emergency department? The journal of allergy and clinical immunology In practice. 2014;2(6):733–740. doi: 10.1016/j.jaip.2014.06.012. [DOI] [PubMed] [Google Scholar]
- 6.Nath JB, Hsia RY. Children's Emergency Department Use for Asthma, 2001-2010. Acad Pediatr. 2015;15(2):225–230. doi: 10.1016/j.acap.2014.10.011. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Lieu TA, Quesenberry CP, Sorel ME, Mendoza GR, Leong AB. Computer-based models to identify high-risk children with asthma. Am J Respir Crit Care Med. 1998;157(4 Pt 1):1173–1180. doi: 10.1164/ajrccm.157.4.9708124. [DOI] [PubMed] [Google Scholar]
- 8.Schatz M, Nakahiro R, Jones CH, Roth RM, Joshua A, Petitti D. Asthma population management: development and validation of a practical 3-level risk stratification scheme. The American journal of managed care. 2004;10(1):25–32. [PubMed] [Google Scholar]
- 9.Forno E, Fuhlbrigge A, Soto-Quiros ME, et al. Risk factors and predictive clinical scores for asthma exacerbations in childhood. Chest. 2010;138(5):1156–1165. doi: 10.1378/chest.09-2426. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Hanson JR, Lee BR, Williams DD, et al. Developing a risk stratification model for predicting future health care use in asthmatic children. Annals of allergy, asthma & immunology : official publication of the American College of Allergy, Asthma, & Immunology. 2015;116(1):26–30. doi: 10.1016/j.anai.2015.10.014. [DOI] [PubMed] [Google Scholar]
- 11.Wu DJ, Hipolito E, Bilderback A, Okelo SO, Garro A. Predicting future emergency department visits and hospitalizations for asthma using the Pediatric Asthma Control and Communication Instrument - Emergency Department version (PACCI-ED) J Asthma. 2016;53(4):387–391. doi: 10.3109/02770903.2015.1115520. [DOI] [PubMed] [Google Scholar]
- 12.Blakey JD, Price DB, Pizzichini E, et al. Identifying Risk of Future Asthma Attacks Using UK Medical Record Data: A Respiratory Effectiveness Group Initiative. The journal of allergy and clinical immunology In practice. 2016 doi: 10.1016/j.jaip.2016.11.007. Epub ahead of print. [DOI] [PubMed] [Google Scholar]
- 13.Duseja R, Bardach NS, Lin GA, et al. Revisit rates and associated costs after an emergency department encounter: a multistate analysis. Ann Intern Med. 2015;162(11):750–756. doi: 10.7326/M14-1616. [DOI] [PubMed] [Google Scholar]
- 14.SEDD Overview. https://www.hcup-us.ahrq.gov/seddoverview.jsp.
- 15.Dobson JV, Bryce L, Glaeser PW, Losek JD. Age limits and transition of health care in pediatric emergency medicine. Pediatr Emerg Care. 2007;23(5):294–297. doi: 10.1097/01.pec.0000248701.87916.05. [DOI] [PubMed] [Google Scholar]
- 16.Neuman MI, Alpern ER, Hall M, et al. Characteristics of recurrent utilization in pediatric emergency departments. Pediatrics. 2014;134(4):e1025–1031. doi: 10.1542/peds.2014-1362. [DOI] [PubMed] [Google Scholar]
- 17.Peltz A, Samuels-Kalow ME, Rodean J, et al. Characteristics of Children Enrolled in Medicaid With High-Frequency Emergency Department Use. Pediatrics. 2017 doi: 10.1542/peds.2017-0962. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Feudtner C, Feinstein JA, Zhong W, Hall M, Dai D. Pediatric complex chronic conditions classification system version 2: updated for ICD-10 and complex medical technology dependence and transplantation. BMC pediatrics. 2014;14:199. doi: 10.1186/1471-2431-14-199. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Lawson CC, Carroll K, Gonzalez R, Priolo C, Apter AJ, Rhodes KV. “No Other Choice”: Reasons for Emergency Department Utilization Among Urban Adults With Acute Asthma. Acad Emerg Med. 2014;21(1):1–8. doi: 10.1111/acem.12285. [DOI] [PubMed] [Google Scholar]
- 20.Enard KR, Ganelin DM. Reducing preventable emergency department utilization and costs by using community health workers as patient navigators. Journal of healthcare management / American College of Healthcare Executives. 2013;58(6):412–427. discussion 428. [PMC free article] [PubMed] [Google Scholar]
- 21.Morgan SR, Chang AM, Alqatari M, Pines JM. Non-emergency department interventions to reduce ED utilization: a systematic review. Acad Emerg Med. 20(10):969–985. doi: 10.1111/acem.12219. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Griffey RT, Kennedy SK, McGownan L, Goodman M, Kaphingst KA. Is Low Health Literacy Associated With Increased Emergency Department Utilization and Recidivism? Acad Emerg Med. 2014;21(10):1109–1115. doi: 10.1111/acem.12476. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Morrison AK, Schapira MM, Gorelick MH, Hoffmann RG, Brousseau DC. Low Caregiver Health Literacy Is Associated With Higher Pediatric Emergency Department Use and Nonurgent Visits. Academic Pediatrics. 2014;14(3):309–314. doi: 10.1016/j.acap.2014.01.004. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Horrocks D, Kinzer D, Afzal S, Alpern J, Sharfstein JM. The Adequacy of Individual Hospital Data to Identify High Utilizers and Assess Community Health. JAMA Intern Med. 2016;176(6):856–858. doi: 10.1001/jamainternmed.2016.1248. [DOI] [PubMed] [Google Scholar]