Risk Stratification for Return Emergency Department Visits Among High-Risk Patients

Abstract

OBJECTIVES:

To compare 2 methods of identifying patients at high-risk of repeat emergency department (ED) use: high Care Assessment Need (CAN) score (≥90), derived from a model using Veterans Health Administration (VHA) data, and “Super User” status, defined as more than 3 ED visits within 6 months of the index ED visit.

STUDY DESIGN:

Retrospective cohort study.

METHODS:

Using McNemar’s test, we compared rates of high-risk classification between CAN score and Super User status. We examined differences in patient characteristics and healthcare utilization across 4 levels of risk classification: high CAN and Super User status (n = 198), CAN <90 and non–Super User (n = 622), high CAN and non–Super User (n = 616), or Super User and CAN score <90 (n = 106). We used logistic regression to identify associations between risk classification and any ED visit within 90 days.

RESULTS:

Of 1542 veterans, 52.8% (n = 814) had a CAN score ≥90 and 19.7% (n = 304) were Super Users (P <.0001), indicating discrepant rates of high-risk classification. However, we found no differences in patient characteristics. Rates of subsequent ED use were high: 63.1% of patients had 1 or more ED visits. No levels of risk classification were associated with subsequent ED use within 90 days (P = .25).

CONCLUSIONS:

Among the VHA users with multimorbidity and 3 or more prior ED visits or hospitalizations, subsequent ED use was high. Although CAN scores have demonstrated utility for predicting hospitalizations and deaths, prior utilization and multimorbidity without further risk classification identified a high-risk group for repeat ED use.

Emergency department (ED) utilization not resulting in hospital admission, referred to as outpatient ED visits, may be avoidable¹ and more costly than an outpatient clinic²; thus, it is considered potentially low-value care. To reduce low-value care, risk prediction models have been developed to identify the patients who account for a disproportionately large amount of healthcare utilization; the goal is to target these patients for interventions that can reduce avoidable utilization.³ Researchers often develop and validate risk prediction models for disease-specific populations3; however, the models may not be generalizable to broader and more medically complex populations.

Outpatient ED visits are common in the Veterans Health Administration (VHA), the largest integrated healthcare system in the United States, which serves more than 9 million veterans nationally.⁴ From 2007 to 2008, 80% of ED visits were outpatient; of these, 15% had a repeat ED visit within 30 days—a higher rate than Medicare beneficiaries.¹ The VHA has been at the forefront of predictive analytics in healthcare and has implemented Care Assessment Need (CAN) scores for all VHA users. CAN scores use complex multivariate modeling to generate a validated risk prediction of hospitalization and/or death within 90 days or 1 year, using available electronic health records (EHRs) and administrative data.⁵ CAN scores are utilized to optimize care coordination and resource allocation for high-risk veterans.⁵

However, it is unknown whether CAN scores identify patients at higher risk for repeat ED utilization—especially compared with simpler strategies, such as a previous history of high ED utilization.^5–11 Thus, in this exploratory study, we examined whether the CAN score provided further information on risk for repeat ED visits for a high-risk cohort of VHA-aftliated patients. First, we compared whether CAN scores and Super User status (ie, having 4 or more ED visits within the last year)¹² identified the same patients as high risk. Then, we assessed whether these risk classifications could predict repeat ED visits that occurred within 90 days of an index ED visit.

METHODS

Study Cohort

The study cohort met initial eligibility criteria for an ongoing randomized clinical trial at the Durham Veterans Affairs Medical Center (DVAMC), Discharge Information and Support for Patients Receiving Outpatient Care in the ED (DISPO ED), which took place from March 10 to September 30, 2014.¹³ DISPO ED examined the effectiveness of a nurse-led intervention to reduce repeat ED visits. In addition to having an index outpatient ED visit, inclusion criteria included: 1 or more visits to a DVAMC-aftliated primary care clinic within the previous 12 months (proxy for engagement with the VA system), 1 or more DVAMC ED visit or hospital admission in the 6 months prior to the index ED visit, and 2 or more chronic conditions.¹³ By the end of the study time period, 17% of all ED visits met these eligibility criteria. Exclusion criteria included current enrollment or previous refusal to participate in DISPO ED, residence in a nursing home, and death on date of the index ED visit.

Data Sources

We used VHA administrative data files, including the Vital Status Mini File,¹⁴ enrollment tables from the VHA’s Assistant Deputy Under Secretary for Health for Policy and Planning,¹⁵ Medical SAS datasets,¹⁶ and additional domains from the Corporate Data Warehouse.

Measures

Primary outcome: repeat VHA ED visit within 90 days.

We determined ED use within 90 days of the index visit through administrative stop codes for ED care at US Veterans Affairs medical centers. We determined outpatient ED encounters by using administrative codes for VHA ED visits and VHA inpatient care administrative datasets.

Key predictors (CAN score).

We extracted the CAN score predicting the percentile of risk of hospital admission in the 90 days closest to the index ED visit date and dichotomized CAN scores using the median split (<90 or ≥90). For example, a CAN score of 90 is associated with an average observed hospitalization rate (≤90 days) of 14% compared with an average of 2.7% in the general VHA population.¹⁷

ED Super User.

Veterans with more than 4 ED visits to the DVAMC within 6 months (including the index ED visit) were categorized as Super Users, based on prior studies and clinical experience.¹³

Covariates (sociodemographics).

Demographics included race, age, marital status, and gender. To indicate economic status, we determined whether the veteran was exempted from co-payments due to limited financial means and had unstable housing within the 12 months prior to the index ED visit.¹

Chronic conditions.

We used diagnosis codes associated with encounters in the year prior to the index ED visit to identify anemia, congestive heart failure, chronic lung disease, chronic renal failure, diabetes, hypertension, ischemic heart disease (IHD), peripheral vascular disease, and mental health conditions, including anxiety disorder, depressive disorder, posttraumatic stress disorder (PTSD), and substance abuse disorder, in accordance with the VHA definition of chronic conditions per the VHA Support Service Center Chronic Disease Registry Development Rules.^13,18

Medical complexity (Quan Charlson Comorbidity Index).

The Quan Charlson Comorbidity Index predicts mortality within 12 months using 17 comorbidities based on the original Charlson Comorbidity Index,¹⁹ but using updated weights identified by Schneeweiss et al.²⁰

Outpatient utilization in year prior to index ED visit.

We counted the number of VHA primary care, outpatient specialty services, and mental health clinic encounters.

Statistical Analysis

We first compared high-risk classification by CAN score of ≥90 and Super User status, using McNemar’s test. Second, we examined differences in demographics, chronic conditions, and utilization in the year prior to the index ED visit across the 4 classification groups: high-risk by both (CAN score ≥90 and identified Super User), high-risk by CAN score only (CAN score ≥90 and non–Super User status), high-risk by Super User status only (CAN Score <90 and identified Super User), or not considered high risk by both (CAN score <90 and non–Super User). For categorical variables, we used χ² analysis. Analysis of variance was used for continuous variables and Poisson regressions for count variables. Finally, we compared repeat ED visits within 90 days (yes/no) for these 4 groups, examining CAN score and Super User status in logistic models, adjusting for the aforementioned demographic, economic, comorbidity, and prior healthcare use covariates.

RESULTS

Study Cohort Characteristics

Fifty percent of participants were African American, and 46% were white. The majority (81%) were exempt from co-payments due to financial need. Moreover, 80% had hypertension, 50% had diabetes, and 28% had IHD.to financial need. Moreover, 80% had hypertension, 50% had diabetes, and 28% had IHD. Additionally, 66% had 1 or more mental health conditions: 31% were diagnosed with PTSD, 41% with depression, and 19% with anxiety. This cohort demonstrated a high level of engagement with the VHA in the year prior to their index ED visit, with 49% having had 1 or more mental health outpatient encounters and 93% having had 1 or more outpatient specialty service encounters. Veterans with mental health, outpatient specialty, or primary care use in the year prior to the index ED visit had an average of 11 mental health, 9 specialty, and 5 primary care outpatient clinic encounters for the year, respectively (eAppendix A [eAppendices available at ajmc.com]).

Risk Classification by CAN Score Versus Super User Status

Using McNemar’s test, the rate of classification between the 2 methods was discrepant (P <.0001) (Table 1). Based on a CAN score of ≥90, the rate of high-risk classification was 52.8% (n = 814) versus 19.7% (n = 304) for ED Super Users. Of those with a CAN score of 90 or above (n = 814), 76% were not identified as Super Users. Of Super Users (n = 304), 34.9% had a CAN score greater than 90. The 2 methods identified different sets of patients as high risk.

TABLE 1.

Risk Classification by CAN Score and Super User Status With 1542 Matched Pairs^a

CAN Score	Super Userb	Non–Super User	Total
CAN score ≥90	198 (12.8%)	616 (39.9%)	814 (52.8%)
CAN score <90	106 (6.9%)	622 (40.3%)	728 (47.2%)
Total	304 (19.7%)	1238 (80.3%)	1542

CAN indicates Care Assessment Need.

^aThe rate of risk determination is discrepant between the 2 high-risk identification methods (McNemar’s Test for Marginal Homogeneity; P <.0001).

^bVeterans with 3 or more emergency department (ED) visits within 6 months of the index ED visit were categorized as Super Users.

Characteristics and Utilization Outcomes by Risk Classification of CAN Score and Super User Status

In general, there were few differences in characteristics across the 4 groups based on cross-classification of CAN score of ≥90 and Super User status (eAppendix A). Significant differences included gender (P <.001), primary care utilization in the year prior to the index ED visit (P <.001), specialty care and mental health care utilization in the year prior to the index ED visit (P <.001), and number of ED visits in the year prior to the index ED visit (P <.05). Patients who were identified as Super Users and had a CAN score below 90 had the highest number of primary care visits in the year prior, and those identified as high risk by both methods had the highest number of mental health visits. There were no significant differences in proportion with chronic conditions, with the exception of anxiety (P <.05).

Repeat ED Visits

Overall, 63% (n = 973) of the cohort had 1 or more repeat ED visits within 90 days after the index ED visit (mean = 1.7 repeat ED visits within the observation period); 90% of which were outpatient ED visits. When examining only repeat outpatient ED visits, 59% (n = 906) of the cohort had more than 1 repeat encounter within 90 days. Veterans with a repeat outpatient ED visit had an average of 1.6 repeat ED visits. Six percent (n = 92) of patients died within 90 days of their index ED visit.

There were no statistically significant differences in the proportion of patients with 1 or more repeat ED visits within 90 days of the index encounter, the number of repeat ED visits, or outpatient ED visits across the 4 risk groups (eAppendix B).

Risk classification of the 4 groups was not associated with repeat ED visits within 90 days of the index encounter (P = .28). Adjustment for covariates yielded similar results. When limiting the outcomes to outpatient ED visits within 90 days, we found similar results (Table 2).

TABLE 2.

Logistic Regression Models of Risk Classification by CAN Score and Super User Status With 90-day Repeat VHA ED Visit

ED Visit	Unadjusted		Adjusted
	Odds Ratio (95% CI)a	P	Odds Ratio (95% CI)b	P
Risk classificationc
CAN score <90 and non–Super Userd	ref	–	ref	–
Super User and CAN score <90	0.8 (0.5−1.2)	.33	0.8 (0.5−1.3)	.38
CAN score ≥90 and non–Super User	1.1 (0.9−1.4)	.26	1.2 (0.9−1.5)	.19
CAN score ≥90 and Super User	0.9 (0.7−1.3)	.54	0.9 (0.7−1.3)	.64

CAN indicates Care Assessment Need; ED, emergency department; ref, reference; VHA, Veterans Health Administration.

^aC statistic = 0.53

^bWe controlled for age, gender, marital status, co-payment exemption, unstable housing, Quan Charlson index risk score, any mental health diagnosis, total mental health clinic stops, total primary care clinic stops, and total outpatient specialty services clinic stops within the year prior to the index ED visit (C statistic = 0.58).

^cThe P for the Type 3 test of the 4-level variable is P = .28 in the unadjusted model. The P for the Type 3 test of the 4-level variable is P = .25 in the adjusted model.

^dVeterans with 3 or more ED visits within the 6 months prior to the index ED visit were categorized as Super Users.

DISCUSSION

In our cohort of VHA ED patients with multimorbidity (defined as 2 or more chronic conditions) and history of a prior ED visit or hospitalization, nearly two-thirds had at least 1 repeat ED visit within 90 days. We found that the CAN score and ED Super User status identified different groups of patients. When we examined 4 risk stratification groups cross classifying by CAN score and Super User status, we found no association between the risk classification groups and repeat ED visits within 90 days of the index date.

Our findings of high repeat ED use in the cohort overall are consistent with previous studies of older veterans, which found that high rates of chronic conditions and prior ED and hospital use were independent predictors of repeat ED use.^6,21,22 Compared with the general VHA ED population, of which 15% of patients had a repeat ED visit within 30 days, our cohort had much higher repeat ED utilization.1 Considering these past studies, our cohort was more racially diverse^6,23,24 and had more mental health conditions.²⁵

The fact that these 2 methods of risk stratification did not improve prediction of ED returns in this already high-risk population has important clinical and research implications. First, new applications of existing risk prediction tools should be validated before being put into practice. CAN scores have been available to primary care providers throughout the VA system; however, more data about their clinical utility outside of recognizing patients at high risk for hospitalization and mortality are needed before they are repurposed. Second, although the CAN model, along with other EHR-based methods of risk stratification, incorporates diverse information on patient demographics, medical conditions, and previous utilization, it excludes potentially important data, such as socioeconomic, cultural, and other contextual factors that often play significant roles.²⁶ Third, considering the CAN score is a comprehensive model of health status predictors, improvements in prediction may be achieved through the use of alternative models more frequently seen in other disciplines, such as models based on machine learning techniques.^27–30

Although we did not identify subgroups of patients at higher risk of ED returns based on CAN scores, ED Super User status, or a combination of these variables, an examination of the cohort characteristics reveals possibilities for future study into clinical populations of interest and potential ways to improve care. First, we observed high engagement with VHA services, with an average of 5 primary care clinic encounters in the year prior to the index ED visit. Previously, lack of access and low engagement with PCPs and specialists have been highlighted as associated with increased ED use.^31–35 Our results suggest that high repeat ED visit rates may not be due exclusively to access barriers; other issues, such as inadequate care coordination, may also play a role.³⁶ If this is the case, increased use of strategies, such as telehealth, may be essential to reducing repeat ED visits. Moreover, the prevalence of mental health diagnoses in our cohort (49%; mean = 11 mental health outpatient encounters in the year prior) was much higher than other studies examining recurring ED use²⁵ and may have contributed to the higher rates of repeat ED visits. Future research should consider the high rates of mental health conditions explicitly. It is notable that the VHA has more extensive and available mental health services compared with many non-VHA systems,³⁶ and interventions directed at non-VHA patients may need to surmount additional barriers to access for mental health services.

Limitations

There are several limitations of this study. First, we identified VHA ED utilization and diagnoses of interest to veterans using VHA administrative claims data, which are subject to coding bias, errors in record keeping, and delayed records of utilization. Second, the single-site cohort potentially limited generalization. Third, we had limited information about other potentially relevant variables related to ED use, such as socioeconomic status data. Fourth, no gold-standard definition of the term “Super User” exists. We examined ED visits over a 9-month period, which does not address the issue of seasonality. We also only examined repeat ED visits at a single time point (within 90 days); findings may have been somewhat different with a longer time horizon (ie, 6, 9, and 12 months). To address these limitations, we extracted data after allowing suftcient time for records to be updated and relied on clinical expertise from senior researchers regarding best practices to determine diagnoses using claims data. We also relied on the literature, clinical expertise, and prior work on the distribution of ED utilization to define Super User status.¹² As an exploratory analysis of associations using secondary data, there was no power calculation for this study. However, the percentages of repeat ED use across the different categorizations were similar and generally above 60%. Thus, there were no indications of significant differences we were not powered to detect; the narrow widths of the confidence intervals are reasonable from the logistic models.

In conclusion, among DVAMC users with multimorbidity and more than 2 prior ED visits or hospitalizations, repeat ED use within 90 days was very high. Applying 2 methods of risk stratification in this population identified discrepant groups of patients, and classification of risk by these 2 measures was not associated with repeat ED use within 90 days. Identifying clinically relevant subgroups is important for future interventions to improve care and provide high-value services for high-risk groups defined by multimorbidity and utilization.

TAKEAWAY POINTS.

We compared 2 methods of identifying patients at high risk of subsequent emergency department (ED) use: 1) Care Assessment Need (CAN) score and 2) Super User status.

• Patients with 2 or more chronic conditions and 3 or more prior ED visits or hospitalizations were identified as a cohort at high risk for subsequent ED visits. Subsequent risk stratification through the CAN score or Super User status did not improve prediction of repeat ED use within 90 days of the index ED visit.

• Although the 2 methods had discrepant rates of classification, there were no statistically significant differences by patient characteristics or subsequent ED use.

eAppendix A.

Characteristics of Cohort by Risk Classification of CAN Scores and Super-user Status

	Overall Cohort	CAN score <90		CAN score ≥90
	N = 1542	Non–Super-usera (n = 622)	Super-user (n = 106)	Non–Super-user (n = 616)	Super-user (n = 198)
Sociodemographics
White race: n, %b	707 (46.2)	282 (45.6)	44 (41.5)	288 (47.1)	93 (47.5)
Male: n, %c	1370 (88.9)	567 (91.2)	95 (89.6)	528 (85.7)	180 (90.9)
Age, mean (SD)d	62.3 (12.7)	62.6 (12.7)	62.3 (14.1)	62.4 (12.5)	61.0 (12.3)
Married: n, %e	798 (51.8)	327 (52.6)	59 (55.7)	317 (51.5)	95 (48.0)
Exempt from healthcare co-pays due to financial need: n. %e	1245 (80.8)	507 (81.6)	82 (77.4)	496 (80.5)	160 (80.8)
Unstable housing within past year: n, %	117 (7.6)	56 (9.0)	5 (4.7)	37 (6.0)	19 (9.6)
Chronic Health Conditions: n, %
Anemia	214 (13.9)	94 (15.1)	19 (17.9)	74 (12.0)	27 (13.6)
Congestive heart failure	77 (5.0)	39 (6.3)	3 (2.8)	24 (3.9)	11 (5.6)
Chronic lung disease	337 (21.9)	129 (20.7)	19 (17.9)	147 (23.9)	42 (21.2)
Chronic renal disease	290 (18.8)	123 (19.8)	23 (21.7)	111 (18.0)	33 (16.7)
Diabetes	771 (50.0)	319 (51.3)	52 (49.1)	307 (49.8)	93 (47.0)
Hypertension	1235 (80.1)	496 (79.7)	82 (77.4)	496 (80.5)	161 (81.3)
Ischemic heart disease	429 (27.8)	180 (28.9)	29 (27.4)	181 (29.4)	39 (19.7)
Peripheral vascular disease	129 (8.4)	59 (9.5)	9 (8.5)	49 (8.0)	12 (6.1)
Dementia	177 (11.5)	80 (12.9)	10 (9.4)	72 (11.7)	15 (7.6)
Any mental health condition	1017 (66.0)	407 (65.4)	73 (68.9)	405 (65.8)	132 (66.7)
Anxiety disorderc	295 (19.1)	101 (16.2)	26 (24.5)	134 (21.8)	34 (17.2)
Depressive disorder	638 (41.4)	248 (39.9)	47 (44.3)	265 (43.0)	78 (39.4)
Post-traumatic stress disorder	471 (30.5)	188 (30.2)	37 (34.9)	184 (29.9)	62 (31.3)
Substance abuse disorder	290 (18.8)	119 (19.1)	22 (20.8)	106 (17.2)	43 (21.7)
Other mental health	146 (9.5)	54 (8.7)	8 (7.6)	60 (9.7)	24 (12.1)
Medical complexity (Quan Charlson Comorbidity Index)f n, %
0	342 (22.2)	128 (20.6)	23 (21.7)	145 (23.5)	46 (23.2)
1	604 (39.2)	259 (41.6)	39 (36.8)	229 (37.2)	77 (38.9)
2	465 (30.2)	180 (28.9)	33 (31.1)	194 (31.5)	58 (29.3)
3+	131 (8.5)	55 (8.8)	11 (10.4)	48 (7.8)	17 (8.6)
VHA utilization within 1 year prior to index date
Number of primary care clinic encounters for those with at least 1 visit, mean (SD)g,h	5.3 (4.8)	5.3 (4.9)	4.9 (3.7)	5.5 (5.1)	4.5 (3.6)
Any outpatient specialty servicesi: n, %	1436 (93.1)	572 (92.0)	100 (94.3)	578 (93.8)	186 (93.9)
Number of outpatient specialty services encounters, mean (SD)g,h	8.9 (8.6)	9.2 (9.2)	9.3 (7.8)	8.6 (8.1)	8.4 (8.6)
Any mental health outpatient encounter: n, %	759 (49.2)	300 (48.2)	53 (50.0)	303 (49.2)	103 (52.0)
Number of mental health encounters: mean, SDg,h	10.9 (18.1)	10.8 (15.7)	7.5 (10.4)	10.9 (19.9)	12.7 (21.9)
Any hospitalization: n %	606 (39.3)	248 (39.9)	38 (35.8)	243 (39.4)	77 (38.9)
Number of ED visits for those with ≥1 visit, mean (SD)g,h	4.4 (2.8)	4.6 (3.3)	3.9 (2.5)	4.4 (2.5)	4.2 (2.2)

CAN indicates Care Assessment Need; ED, emergency department; SD, standard deviation; VHA, Veterans Health Administration.

^aVeterans with 3 or more ED visits within 6 months prior to the index ED visit are categorized as Super-users.

^b10 missing.

^cP <.05.

^dAnalysis of variance (ANOVA)

^e1 missing

^fThe Quan Charlson Comorbidity Index includes updated weights identified by Schneeweiss et al for the weighted 17 comorbidities identified as predictors of mortality within 1 year as defined by the Charlson Comorbidity Index.

^gP <.001.

^hPoisson regression.

ⁱOutpatient specialty services include the major sub-specialties and services, such as laboratory, pharmacy, behavioral programs, and other specialty services.

eAppendix B.

VHA 90-Day Repeat ED Visit Post Index ED Visit by Risk Classification

Characteristic	CAN score <90		CAN score ≥90
	Non– Super-user n = 622	Super- user n = 106a	Non– Super-user n = 616	Super- usera n = 198
Utilization
ED visit: n, %	389 (62.5)	61 (57.6)	404 (65.6)	119 (60.1)
Number of ED visits, mean (SD)	1.6 (1.0)	1.8 (1.0)	1.7 (1.0)	1.5 (0.9)
Outpatient ED visit: n, %	364 (58.5)	56 (52.8)	374 (60.7)	112 (56.6)
Number of outpatient ED visits,    mean (SD)	1.6 (0.9)	1.6 (0.8)	1.7 (0.9)	1.5 (0.8)

CAN indicates Care Assessment Need; ED, emergency department; SD, standard deviation; VHA, Veterans Health Administration.

^aVeterans with 3 or more ED visits within the 6 months prior to the index ED visit were categorized as Super-users.