Graded Coronary Risk Stratification for Emergency Department Patients With Chest Pain: A Controlled Cohort Study

Abstract

Background

Resource utilization among emergency department (ED) patients with possible coronary chest pain is highly variable.

Methods and Results

Controlled cohort study amongst 21 EDs of an integrated healthcare system examining the implementation of a graded coronary risk stratification algorithm (RISTRA‐ACS [risk stratification for acute coronary syndrome]). Thirteen EDs had access to RISTRA‐ACS within the electronic health record (RISTRA sites) beginning in month 24 of a 48‐month study period (January 2016 to December 2019); the remaining 8 EDs served as contemporaneous controls. Study participants had a chief complaint of chest pain and serum troponin measurement in the ED. The primary outcome was index visit resource utilization (observation unit or hospital admission, or 7‐day objective cardiac testing). Secondary outcomes were 30‐day objective cardiac testing, 60‐day major adverse cardiac events (MACE), and 60‐day MACE‐CR (MACE excluding coronary revascularization). Difference‐in‐differences analyses controlled for secular trends with stratification by estimated risk and adjustment for risk factors, ED physician and facility. A total of 154 914 encounters were included. Relative to control sites, 30‐day objective cardiac testing decreased at RISTRA sites among patients with low (≤2%) estimated 60‐day MACE risk (−2.5%, 95% CI −3.7 to −1.2%, P<0.001) and increased among patients with non‐low (>2%) estimated risk (+2.8%, 95% CI +0.6 to +4.9%, P=0.014), without significant overall change (−1.0%, 95% CI −2.1 to 0.1%, P=0.079). There were no statistically significant differences in index visit resource utilization, 60‐day MACE or 60‐day MACE‐CR.

Conclusions

Implementation of RISTRA‐ACS was associated with better allocation of 30‐day objective cardiac testing and no change in index visit resource utilization or 60‐day MACE.

Registration

URL: https://www.clinicaltrials.gov; Unique identifier: NCT03286179.

Nonstandard Abbreviations and Acronyms

ACS

acute coronary syndrome

CDS

electronic clinical decision support

CPT

current procedural terminology

DID

difference‐in‐differences

ECG

electrocardiogram

ED

emergency department

EDACS‐ADP

Emergency Department Assessment of Chest pain Score Accelerated Diagnostic Protocol

HEART

History, Electrocardiogram, Age, Risk factors and Troponin

ICD‐10

International Classification of Disease, 10^th revision

IQR

Interquartile ratio

KPNC

Kaiser Permanente Northern California

LOS

length of stay

MACE

major adverse cardiac event

MACE‐CR

major adverse cardiac event, excluding coronary revascularization

MI

myocardial infarction

RISTRA

risk stratification

VF

ventricular fibrillation

Clinical Perspective

What Is New?

• Implementation of a graded coronary risk stratification algorithm in the emergency departments of an integrated healthcare delivery system safely resulted in less objective cardiac testing among patients with chest pain at low risk of major adverse cardiac events while increasing downstream testing among patients at non‐low risk.

What Are the Clinical Implications?

• Integration of graded coronary risk scores into clinical practice can help better match resource utilization to observed cardiac risk.

Patients presenting to the emergency department (ED) with chest pain have large variations in hospital admission rates, mostly driven by guideline recommendations to secure objective cardiac testing for possible acute coronary syndrome (ACS) prior to or within 72 hours of hospital discharge, despite a low overall incidence of acute coronary syndrome. ¹ , ² , ³ , ⁴ , ⁵ However, this practice has not been shown to be associated with improved near‐term outcomes, ⁶ , ⁷ , ⁸ , ⁹ , ¹⁰ with the notable exception of patients with elevated serum troponin levels. ¹¹ Accordingly, a recent clinical policy from the American College of Emergency Physicians recommended against routine objective cardiac testing for patients at low risk for ACS. ¹² Accurate identification of chest pain patients at low risk of ACS is essential to this recommendation.

Two well‐validated protocols for identifying patients at low risk of ACS are the History, Electrocardiogram, Age, Risk factors, and Troponin Pathway (HEART Pathway) and the Emergency Department Assessment of Chest Pain Score Accelerated Diagnostic Protocol (EDACS‐ADP). Both achieve negative predictive values above 99% for 30‐ to 45‐day major adverse cardiac events (MACE) and have specificities ranging between 40% and 60%. ¹³ , ¹⁴ , ¹⁵ However, we previously observed that there are subgroups of patients with discordant risk classifications between the 2 protocols (ie, low risk by one and non‐low risk by the other) and/or troponin values in the upper range of normal who have marginally higher risks of downstream MACE. ¹⁶

Based on these observations, we designed a risk stratification algorithm risk stratification for acute coronary syndrome (RISTRA‐ACS) using both the HEART pathway and EDACS‐ADP to predict 60‐day MACE risk among ED chest pain patients with possible ACS. We subsequently prospectively validated and evaluated the comparative performance of RISTRA‐ACS at 13 of 21 community EDs in an integrated healthcare system using electronic clinical decision support (eCDS) embedded within the electronic health record, finding that RISTRA‐ACS demonstrated the best overall performance with a negative likelihood ratio of 0.06 and an area under the receiver operating characteristic curve of 0.92 for 60‐day MACE. ¹⁷ We hypothesized that the combined availability of and education surrounding RISTRA‐ACS at these 13 EDs would lead to (1) a decrease in objective cardiac testing and hospital or observation unit admission among low risk patients and (2) an increase in objective cardiac testing among non‐low risk patients. Since the majority of patients presenting with chest pain are at low risk of adverse outcomes, ² we anticipated that decreases in utilization among low‐risk patients would outweigh increases in utilization among non‐low risk patients. Thus, an overall decrease in utilization represented our primary hypothesis.

Methods

Study Design and Setting

We did a controlled cohort study at the 21 EDs within Kaiser Permanente Northern California (KPNC), a private not‐for‐profit integrated health system of over 4 million members with ≈1.2 million ED visits annually. KPNC members include ≈34% of the region’s population and are representative of the demographic and socioeconomic diversity of the surrounding population. ¹⁸ All arenas of care (inpatient, outpatient, emergency) within KPNC utilize a single integrated electronic health record (Epic, Verona, WI). This study was approved by the KPNC Institutional Review Board with a waiver of informed consent. Author JH had full access to the data and takes responsibility for its integrity and data analysis. Because of the sensitive nature of the data collected for this study, requests to access the data set from qualified researchers trained in human subject confidentiality protocols may be sent to KPNC at kpnc.irb@kp.org.

The study examined the impact of RISTRA‐ACS availability for prospective validation (Clinicaltrials.gov NCT03286179) at 13 exposed EDs (RISTRA sites) versus 8 non‐exposed EDs (control sites) during the 24 months following RISTRA‐ACS implementation (January 1, 2018 to December 31, 2019) as compared to a 24‐month pre‐implementation period (January 1, 2016 to December 31, 2017). A time‐interrupted approach was planned to allow for a 12‐month run‐in period (January 1, 2018 to December 31, 2018) such that only the second post‐implementation year was used to assess the impact of RISTRA‐ACS availability. We chose a 12‐month run‐in period to maximize physician familiarity with RISTRA‐ACS, particularly among late‐adopters. Encounter inclusion criteria were age ≥18 years, chief complaint of chest pain or chest discomfort, serum troponin measurement within 6 hours of ED arrival, and active health plan membership defined as 9 out of 12 months prior and two continuous months following the encounter, except in cases of death, to ensure complete follow‐up for the outcome period. Encounters were excluded if there was an ED diagnosis of ST‐elevation myocardial infarction, a discharge from the ED against medical advice, or if the patient had an included encounter in the prior 60 days (owing to a 60‐day outcome period). We estimated a study cohort size of 150 000 encounters over the 36‐month analytic period based on the size of a prior retrospective study of coronary risk score performance in the same setting. ¹⁶

Serum troponin values at all sites were obtained using a fourth‐generation troponin I assay, the Access AccuTnI+3 (Beckman‐Coulter, Brea, California). The 99th percentile for this assay is 0.04 ng/mL per local institutional reporting guidelines and reference literature. ¹⁹ The coefficient of variation at the 99th percentile is <10%, and the limits of blank, detection and quantitation are <0.01 ng/mL, 0.01 ng/mL, and 0.02 ng/mL, respectively.

RISTRA‐ACS

RISTRA‐ACS was incorporated as a module in a web‐based eCDS interface referred to as RISTRA (risk stratification) which is nested within the electronic health record, as previously described for several other use cases. ²⁰ , ²¹ RISTRA‐ACS eCDS was made available at all RISTRA sites beginning on January 1, 2018. RISTRA‐ACS automatically imported relevant structured data from the electronic health record (eg, past medical history), which was modified and/or validated by the clinician, followed by user input of subjective elements from the clinical history. Details of RISTRA‐ACS data collection, troponin testing protocol, risk estimate algorithm and screen shots of the eCDS interface are available in Data S1, Figure S1, and Figure S2. Once no further serum troponin measurements were recommended, users were given one of four possible recommendations for disposition based on estimated risk, including an option for no further testing among patients with a low (2% or less) estimated 60‐day MACE risk (Figure 1).

Figure 1. Risk stratification for acute coronary syndrome‐ACS estimated risk prediction categories and accompanying recommendations.

MACE indicates major adverse cardiac event. RISTRA‐ACS, risk stratification for acute coronary syndrome.

Since a previous study employing the RISTRA eCDS platform suggested that availability of the eCDS without accompanying education was insufficient to influence practice change, ED physicians were educated about RISTRA‐ACS in advance of eCDS availability during the last quarter of 2017. ²⁰ Specific attention was given to internal and external findings regarding the predictive value of low versus high‐normal range troponin values for downstream MACE, ¹⁶ , ²² the validated test characteristics of the HEART pathway and EDACS‐ADP, ²³ , ²⁴ , ²⁵ and literature questioning the utility of routine non‐invasive cardiac testing and/or hospital admission among low‐risk patients. ²⁶ , ²⁷ , ²⁸ Real‐time prompts were available via automated text messages to ED physicians whenever serum troponin results became available for an adult patient under their care with a chief complaint of chest pain or discomfort. ²⁹ All ED physicians were able to place orders for objective cardiac testing, including outpatient appointments, though specific test availability varied by facility and day of the week.

Variables

Since this study concerned both RISTRA‐exposed and control sites, and included encounters from both pre‐ and post‐implementation periods irrespective of RISTRA‐ACS use or access, we ascertained component variables for RISTRA‐ACS risk determination for all study encounters using a standardized and previously validated automated retrospective methodology. ³⁰ In brief, variables needed to calculate HEART and EDACS scores were electronically extracted from the electronic health record using structured data (eg, past medical history from the problem list, smoking status, troponin values) supplemented with free text extraction and processing of both unstructured clinical notes (for presenting symptoms) and the finalized expert interpretation of electrocardiograms (ECGs) obtained during the index ED encounter. The history component of the HEART score was calculated in a standardized fashion by considering the net balance of any higher risk symptoms (eg, pain radiating to the arm) against any lower risk symptoms (eg, pain reproduced with inspiration). Only crescendo angina (an independent non‐low risk criterion in EDACS‐ADP) was not assessed retrospectively. Troponin values used for retrospective risk estimate determination were restricted to those obtained within 6 hours of ED arrival. Further details regarding retrospective risk score elements and determination are provided in Data S2, Table S1, and Table S2.

Outcomes

The primary study outcome was index visit resource utilization (defined as hospital or observation unit admission, or 7‐day objective cardiac testing). Objective cardiac testing included exercise electrocardiography, myocardial perfusion imaging, stress echocardiography, computed tomographic coronary angiography, or coronary catheterization. Secondary outcomes included 60‐day MACE (defined as the composite outcome of acute myocardial infarction, cardiac arrest, cardiogenic shock, coronary revascularization, or all‐cause mortality), 60‐day MACE excluding coronary revascularization (MACE‐CR), and 30‐day objective cardiac testing. MACE‐CR was included as a secondary outcome due to a lack of reliable methodology, specifically following an ED visit, ³¹ to categorize coronary revascularization procedures as either elective or non‐elective based on diagnostic and/or billing codes, and because inclusion of elective coronary revascularization procedures is inconsistent with consensus agreements on appropriate MACE endpoints. ³²

Acute myocardial infarction, cardiac arrest or cardiogenic shock was considered to have occurred if a corresponding International Classification of Disease, 10th revision (ICD‐10) code was the first or second diagnosis listed at an inpatient or ED encounter within the integrated healthcare system, or was used in a coded claim for services provided at facilities outside of the system (any coding position). For coronary revascularization, any corresponding ICD‐10 procedure or current procedural terminology (CPT) code during a hospitalization within or outside of the integrated healthcare system was counted. All‐cause mortality was determined using a composite death database comprised of KPNC mortality records, California Department of Public Health Vital Records, and Social Security Death Index data. Objective cardiac testing and hospital or observation admissions were tracked using internal procedure codes and patient care encounters, respectively. ICD‐10 and CPT codes used to define outcomes above are available in Data S3.

Data Analysis

Difference‐in‐differences analyses were used to compare changes in primary and secondary outcomes between RISTRA and control sites during the 12‐month post‐implementation period (beyond the 12‐month run‐in period) in comparison to the 24‐month pre‐implementation period. This analytic approach controls for secular trends and is not directly affected by imbalances in baseline variables between comparator groups, assuming those imbalances remain relatively constant over time within those groups. ³³ We analyzed all study eligible encounters from both pre‐ and post‐implementation periods, as opposed to focusing on encounters in which RISTRA‐ACS was employed, both because we anticipated that practice change would eventually develop independent of RISTRA‐ACS use (owing to intuitive familiarity with the algorithm) and due to concern for uncontrolled bias if we attempted to identify matched controls (due to unmeasured confounding of clinical concern for ACS and the physician’s perceived utility of clinical decision support for a given patient).

The difference‐in‐differences was determined from the coefficient of the interaction term between study site and implementation period in a mixed‐effects regression model, adjusted for patient‐level variables (age, sex, past medial history [diabetes, myocardial infarction, coronary revascularization], RISTRA‐ACS estimated 60‐day MACE risk, and peak troponin value within 6 hours of ED arrival) with random effects for initial treating ED physician and facility. These additional patient level variables were chosen de novo to account for underweighting or lack of representation within the risk scores, as well as to account for any key changes in patient risk within comparator groups over time. Stratified analyses for encounters with low (≤2%) and non‐low (>2%) estimated 60‐day MACE risk were performed to assess the hypotheses that RISTRA‐ACS implementation would be associated with (1) a decrease in objective cardiac testing and hospital or observation unit admission among low‐risk patients and (2) an increase in 30‐day objective cardiac testing among non‐low risk patients. This cut‐point was chosen given that RISTRA‐ACS recommended against (<0.5% estimated risk) or gave the option of deferring (1%–2% estimated risk) further objective cardiac testing for patients at 2% or lower estimated 60‐day MACE risk.

Sensitivity analyses included: (1) exclusion of patients with index visit diagnoses of MACE (to assess for impact of RISTRA‐ACS exposure on the incidence of delayed MACE diagnoses), (2) exclusion of patients seen by physicians in the lowest quartile of study eligible encounters (to enrich for physicians with greater patient contact during the study period), (3) exclusion of low‐adopting RISTRA sites (those sites with below median RISTRA‐ACS eCDS use among patients with a chief complaint of chest pain who underwent serum troponin testing, treating eCDS use as a proxy for facility‐level adoption of the RISTRA‐ACS algorithm), (4) use of a truncated 6‐month run‐in period with an 18‐month post‐implementation period, and (5) using fixed effects instead of random effects in the primary difference‐in‐differences model (to assess for bias from unmeasured variables). All hypotheses were two‐sided with significance set at α=0.05. Data analyses were performed using Stata 14.2 (StataCorp, College Station, Texas).

Results

There were 5 369 919 total ED encounters during the 48‐month study period, of which 154 914 were study eligible during the 36‐month analytic period (94 683 in the pre‐implementation period and 60 231 in the post‐implementation period, Figure 2). Of these eligible encounters, 109 583 (70.7%) presented to the 13 RISTRA sites. The overall median age was 60 years, 45.1% were male, 26.3% had a history of diabetes, 19.7% had coronary artery disease, and 12.5% had prior coronary revascularization, with similar prevalence of these risk factors and distribution of estimated 60‐day MACE risk between RISTRA and control sites (Table 1). RISTRA‐ACS was accessed during 14% of study eligible encounters at RISTRA sites in the post‐implementation period with interfacility variation ranging between 8% to 24%. Quarterly averages of RISTRA‐ACS use among study eligible encounters during the run‐in and post‐implementation periods are shown in Figure 3.

Figure 2. Study cohort selection and stratification.

ED, emergency department; KP, Kaiser Permanente, KPNC, Kaiser Permanente Northern California; MACE, major adverse cardiac event; RISTRA, risk stratification; STEMI, ST‐elevation myocardial infarction.

Table 1.

Patient Characteristics

	All sites	RISTRA sites (n=13)	Control sites (n=8)
Number of encounters	154 914	109 583	45 331
Age, y, median (IQR)	60 (48–72)	60 (48–72)	59 (47–71)
Male (%)	45.1	44.9	45.5
White (%)	51.1	50.8	51.6
Black (%)	12.3	13.4	9.7
Asian (%)	15.1	16.3	11.9
Hispanic (%)	19.8	17.6	24.9
Other (%)	1.9	1.9	1.9
Past medical history
Hypertension (%)	52.6	52.7	52.5
Hypercholesteremia (%)	52.0	52.2	51.8
Diabetes (%)	26.3	26.0	26.9
Coronary artery disease (%)	19.7	19.8	19.5
Coronary revascularization (%)	12.5	12.5	12.6
Myocardial infarction (%)	13.8	14.0	13.5
Stroke (%)	9.5	9.3	10.0
Peripheral artery disease (%)	3.7	3.7	3.7
Smoker (%)	9.5	9.4	9.8
Family history (%)*	4.6	4.4	5.2
Obesity (%)	42.1	41.0	44.6
Risk estimates
HEART score (median, IQR)	4 (3–5)	4 (3–5)	4 (3–5)
EDACS (median, IQR)	12 (7–17)	12 (8–18)	12 (6–16)
<0.5% 60‐d MACE risk (%)	36.1	35.5	37.7
1%–2% 60‐d MACE risk (%)	39.1	39.4	38.5
2%–3% 60‐d MACE risk (%)	7.4	7.4	7.4
5%–7% 60‐d MACE risk (%)	4.8	4.9	4.6
>7% 60‐d MACE risk (%)	12.5	12.8	11.8
Low (<2%) risk for 60‐d MACE (%) * , †	75.3	74.9	76.2
Non‐low (>2%) risk for 60‐d MACE (%) ‡	24.7	25.1	23.8

Data are presented for all sites and stratified by site designation (RISTRA versus control) and are inclusive of the pre‐implementation period (January 1, 2016 to December 31, 2017) and the post‐implementation period (January 1, 2019 to December 31, 2019). ECG indicates electrocardiogram; EDACS, Emergency Department Assessment of Chest pain Score; HEART, History, Electrocardiogram, Age, Risk factors, Troponin; IQR, interquartile ratio; MACE, major adverse cardiac event; RISTRA, risk stratification.

^*Family history of premature coronary artery disease in a first degree relative aged 55 or younger.

^† Estimated 60‐d MACE risk of 2% or less.

^‡ Estimated 60‐d MACE risk of >2%.

Figure 3. Quarterly averages of RISTRA‐ACS use among study eligible patient encounters (RISTRA sites only).

RISTRA‐ACS, risk stratification for acute coronary syndrome.

Frequencies of observed outcomes are reported in Table 2, with stratification by study period and site, and are summarized using unadjusted difference‐in‐differences statistics. Adjusted difference‐in‐differences analyses comparing post‐ and pre‐implementation periods at RISTRA versus control sites are shown in Table 3. For the primary outcome of index visit resource utilization, there were no statistically significant adjusted difference‐in‐differences at RISTRA sites overall (−0.3%, 95% CI −1.4% to +0.8%, P=0.62) or among patients with either low (−1.1%, 95% CI −2.3% to +0.2%, P=0.10) or non‐low estimated 60‐day MACE risks (+1.5%, 95% CI −0.6% to +3.5%, P=0.15). Likewise, there were no statistically significant changes in 60‐day MACE or 60‐day MACE‐CR outcomes overall in either risk strata. There was however a statistically significant decrease in 30‐day objective cardiac testing at RISTRA sites among patients at low estimated 60‐day MACE risk (−2.5%, 95% CI −3.7 to −1.2%, P<0.001) as well as an increase among patients at non‐low estimated risk (+2.8%, 95% CI +0.6 to +4.9%, P=0.014). Time‐trend graphs for the outcomes are shown in Figure 4 (index visit resource utilization), Figure 5 (30‐day objective cardiac testing), Figure 6 (60‐day MACE), and Figure 7 (60‐day MACE‐CR).

Table 2.

Study Outcomes and Index Encounter Findings

	All sites (n=21)		RISTRA sites (n=13)		Control sites (n=8)		RISTRA ‐ control
Implementation period	Pre	Post	Pre	Post	Pre	Post	Unadjusted difference in differences (95% CI)
Study subjects (n)	94 683	60 231	67 988	41 595	26 695	18 636
Index encounter
Multiple troponin tests within 6 h of ED arrival (%)	27.1	29.3	29.3	33.0	21.7	21.0	+4.4 (+3.5 to +5.2)
ED LOS, hours (mean, SD)	4.6 (2.9)	4.7 (3.3)	4.6 (2.9)	4.7 (3.4)	4.7 (2.9)	4.5 (3.1)	+0.3 (+0.2 to +0.3)
ED LOS, hours (median, IQR)	4 (3–6)	4 (3–6)	4 (3–6)	4 (3–6)	4 (3–6)	4 (3–5)	+0.3 (+0.2 to +0.3)
Total LOS, hours (mean, SD)	16.5 (39.3)	15.8 (37.4)	17.2 (41.2)	16.5 (37.8)	14.8 (34.0)	14.4 (36.5)	−0.3 (−1.2 to +0.6)
Total LOS, hours (median, IQR)	5 (3–14)	5 (3–9)	5 (3–17)	5 (3–12)	5 (3–8)	4 (3–7)	+0.1 (−0.1 to +0.2)
Initial troponin >99th percentile (%)	7.9	7.4	7.8	7.6	8.0	7.0	+0.8 (+0.2 to +1.4)
Troponin >99th percentile within 6 h of ED arrival (%)	8.9	8.4	8.9	8.7	9.0	7.7	+1.1 (+0.2 to +2.0)
MACE (%)	4.6	4.2	4.7	4.4	4.3	3.6	+0.4 (−0.1 to +0.9)
MACE‐CR (%)	3.9	3.6	3.9	3.8	3.9	3.2	+0.5 (0.0 to +0.9)
Acute MI (%)	3.7	3.3	3.7	3.5	3.6	3.0	+0.5 (0.0 to +0.9)
Index visit resource utilization
Admission (%)	10.2	9.4	10.4	9.5	9.8	9.1	−0.1 (−0.8 to +0.5)
Observation unit (%)	17.0	14.4	19.2	16.5	11.6	9.8	−0.9 (−1.7 to 0.0)
7‐d exercise electrocardiography (%)	24.3	15.4	26.2	16.1	19.3	14.0	−4.8 (−5.7 to −3.9)
7‐d myocardial perfusion imaging (%)	10.2	9.2	11.2	10.4	7.5	6.6	+0.2 (−0.5 to +0.8)
7‐d stress echocardiography (%)	0.3	0.1	0.2	0.2	0.4	0.1	+0.3 (+0.2 to +0.4)
7‐d CT coronary angiography (%)	0.2	0.3	0.2	0.3	0.4	0.3	+0.2 (0.0 to +0.3)
7‐d coronary catheterization (%)	7.4	6.4	7.5	6.6	7.2	6.1	+0.3 (−0.3 to +0.8)
Any 7‐d objective cardiac test (%)	35.6	27.7	37.5	29.8	30.6	23.1	−0.2 (−1.2 to +0.9)
Index visit resource utilization (%)	46.9	38.3	49.2	40.8	40.9	32.8	−0.3 (−1.4 to +0.8)
30‐d objective cardiac testing
Exercise electrocardiography (%)	28.4	19.9	30.2	20.0	23.9	19.7	−6.0 (−7.0 to −5.1)
Myocardial perfusion imaging (%)	12.3	11.8	13.4	12.9	9.4	9.3	−0.4 (−1.1 to +0.3)
Stress echocardiography (%)	0.8	0.5	0.7	0.5	0.9	0.3	+0.4 (+0.2 to +0.6)
CT coronary angiography (%)	0.4	0.6	0.4	0.6	0.6	0.7	+0.1 (−0.1 to +0.2)
Coronary catheterization (%)	8.4	7.3	8.4	7.4	8.2	7.1	+0.1 (−0.5 to +0.7)
Any objective cardiac test (%)	40.7	33.7	42.4	35.1	36.3	30.7	−1.8 (−2.8 to −0.7)
60‐d outcomes
MACE (%)	8.0	6.9	8.0	7.0	8.1	6.7	+0.5 (−0.1 to +1.1)
MACE‐CR (%)	7.0	6.3	6.9	6.4	7.1	6.1	+0.5 (0.0 to +1.1)
Acute MI (%)	5.9	5.1	5.8	5.2	6.0	4.8	+0.6 (+0.1 to +1.1)
Cardiac arrest/VF (%)	0.2	0.2	0.2	0.2	0.2	0.2	0.0 (−0.1 to +0.1)
Cardiogenic shock (%)	0.2	0.2	0.2	0.2	0.2	0.2	0.0 (−0.1 to +0.1)
Coronary revascularization (%)	3.6	2.8	3.5	2.7	3.9	3.0	+0.2 (−0.2 to +0.6)
All‐cause mortality (%)	1.3	1.3	1.3	1.3	1.3	1.4	0.0 (−0.3 to +0.2)

Unadjusted difference in differences represent the change in percentage of outcomes at RISTRA sites relative to control sites in the post‐implementation period (January 1, 2019 to December 31, 2019) as compared to the pre‐implementation period (January 1, 2016 to December 31, 2017). CT, computed tomography; ED, emergency department; LOS, length of stay; MACE, major adverse cardiac event; MACE‐CR, major adverse cardiac event excluding coronary revascularization; MI, myocardial infarction; RISTRA, risk stratification; VF, ventricular fibrillation.

Table 3.

Difference‐in‐Differences (DID) Analysis

	Unadjusted DID (95% CI)	P value	Adjusted DID (95% CI)	P value
Low risk subgroups
Index visit resource utilization*	−1.9% (−3.1 to −0.7)	0.003	−1.1% (−2.3 to +0.2)	0.10
30‐d objective cardiac testing	−3.8% (−5.0 to −2.6)	<0.001	−2.5% (−3.7 to −1.2)	<0.001
60‐d MACE	−0.3% (−0.7 to 0.0)	0.041	−0.3% (−0.6 to 0.0)	0.051
60‐d MACE‐CR	−0.2% (−0.4 to +0.1)	0.13	−0.2% (−0.4 to +0.1)	0.16
Non‐low risk subgroups
Index visit resource utilization*	+1.7% (−0.5 to +3.8)	0.13	+1.5% (−0.6 to +3.5)	0.15
30‐d objective cardiac testing	+3.4% (+1.1 to +5.7)	0.004	+2.8% (+0.6 to +4.9)	0.014
60‐d MACE	+1.2% (−0.8 to +3.2)	0.24	+0.4% (−1.2 to +2.1)	0.62
60‐d MACE‐CR	+1.1% (−0.9 to +3.0)	0.29	+0.2% (−1.4 to +1.8)	0.83
Overall
Index visit resource utilization*	−0.3% (−1.4 to +0.8)	0.56	−0.3% (−1.4 to +0.8)	0.62
30‐d objective cardiac testing	−1.8% (−2.8 to −0.7)	0.001	−1.0% (−2.1 to +0.1)	0.079
60‐d MACE	+0.5% (−0.1 to +1.1)	0.12	−0.2% (−0.6 to +0.3)	0.49
60‐d MACE‐CR	+0.5% (0.0 to +1.1)	0.062	−0.1% (−0.6 to +0.3)	0.52

Percentages represent the observed differences in outcomes at RISTRA sites relative to control sites in the post‐implementation period (January 1, 2019 to December 31, 2019) as compared to the pre‐implementation period (January 1, 2016 to December 31, 2017). Results are presented both with and without adjustment for age, sex, past medical history, estimated RISTRA risk, and troponin, with random effects at both facility and provider levels. The low‐risk subgroup includes patient encounters with an estimated 60‐day MACE risk of ≤2%, and the non‐low risk subgroup represents the remainder of encounters with >2% estimated 60‐day MACE risk. DID indicates difference‐in‐differences; MACE, major adverse cardiac event; MACE‐CR, major adverse cardiac event excluding coronary revascularization; RISTRA, risk stratification.

^*A composite of observation unit or hospital admission during the index ED visit, or objective cardiac testing within the following 7 days.

Figure 4. Time trends in index visit resource utilization.

Outcomes are stratified by RISTRA sites (blue lines) and control sites (orange lines). Error bars represent 95% confidence intervals. Figures are presented by estimated risk of 60‐day major adverse cardiac events: (A) patients with low (≤2%) estimated risk; (B) patients with non‐low (>2%) estimated risk; (C) overall (any risk).

Figure 5. Time trends in 30‐day objective cardiac testing.

Outcomes are stratified by RISTRA sites (blue lines) and control sites (orange lines). Error bars represent 95% confidence intervals. Figures are presented by estimated risk of 60‐day major adverse cardiac events: (A) patients with low (≤2%) estimated risk; (B) patients with non‐low (>2%) estimated risk; (C) overall (any risk).

Figure 6. Time trends in 60‐day major adverse cardiac events (MACE).

Outcomes are stratified by RISTRA sites (blue lines) and control sites (orange lines). Error bars represent 95% confidence intervals. Figures are presented by estimated risk of 60‐day major adverse cardiac events: (A) patients with low (≤2%) estimated risk; (B) patients with non‐low (>2%) estimated risk; (C) overall (any risk).

Figure 7. Time trends in 60‐day major cardiac adverse events except coronary revascularization (MACE‐CR).

Outcomes are stratified by RISTRA sites (blue lines) and control sites (orange lines). Error bars represent 95% confidence intervals Figures are presented by estimated risk of 60‐day major adverse cardiac events: (A) patients with low (≤2%) estimated risk; (B) patients with non‐low (>2%) estimated risk; (C) overall (any risk).

Sensitivity analyses (Tables S3, S4, S5, S6 and S7) were supportive of the primary analysis, with the additional statistically significant findings at RISTRA sites of (1) a decrease in 30‐day objective cardiac testing among all patients without an index encounter diagnosis of MACE (−1.3%, 95% CI −2.5% to −0.2%, P=0.026) and (2) a decrease in all outcomes among patients at low estimated risk of 60‐day MACE following exclusion of RISTRA sites with below median eCDS use during the post‐implementation period.

Discussion

In this controlled cohort study of the impact of RISTRA‐ACS availability, while there was no statistically significant adjusted difference‐in‐differences in the primary outcome of index visit resource utilization, there was a statistically significant redistribution of 30‐day objective cardiac testing at RISTRA sites during the post‐implementation period, with a 2.5% absolute decrease among patients at low (≤2%) estimated 60‐day MACE risk and a 2.8% absolute increase among patients at non‐low (>2%) estimated risk. While relatively modest, the decrease in objective cardiac testing among patients at low estimated risk of 60‐day MACE appeared safe in that there was no associated increase in 60‐day MACE or MACE‐CR outcome incidence (upper 95% CI for adjusted MACE and MACE‐CR difference‐in‐differences of 0.0% and +0.1%, respectively). These results remained robust in sensitivity analyses.

Prior studies have demonstrated decreases in objective cardiac testing following implementation of standardized coronary risk scores in the ED. In their stepped‐wedge multicenter randomized trial of a HEART score care pathway, Poldervaart et al reported an unadjusted 8% decrease (65% to 57%) in diagnostic procedures within 3 months following an ED chest pain visit. ³⁴ Mahler et al, analyzing a multicenter implementation of the HEART Pathway, observed a 6% decrease in 30‐day hospitalizations (61.6% to 55.6%) and a 3.8% decrease in 30‐day objective cardiac testing (34.5% to 30.7%). ³⁵ After implementation of the HEART score in 13 EDs of an integrated health system, Sharp et al noted a 4.4% adjusted decrease (accounting for pre‐implementation trends) in the composite of hospitalization or objective cardiac testing within 30 days. ³⁶ However, the latter two studies lacked concurrent controls to account for unanticipated time trends, and Poldervaart et al did not observe significant differences in utilization when controlling for clustering and time steps. This potential for confounding by time trends is apparent in the current study by the steady decrease in 30‐day objective cardiac testing at control sites noted in the time‐trend graphs (Figure 3). As such, the 2.5% absolute decrease in objective cardiac testing observed in the low‐risk subgroup arguably represents a more certain association between ED coronary risk score availability and utilization than previous reports, as does the 1.3% overall adjusted reduction in 30‐day objective cardiac testing among patients without an index encounter MACE diagnosis, who arguably better represent the population of interest (ie, patients without overt evidence of ACS).

We also observed a time‐dependence in the impact of RISTRA‐ACS availability among RISTRA sites, reflected by a gradual steepening of the downward slope for 30‐day cardiac testing at RISTRA sites during the post‐implementation period (Figure 3). One possible reason for this apparent incremental adoption of practice change at RISTRA sites relates to the notion that physicians are more comfortable engaging in risk‐averse behavior (eg, increased testing of patients identified as non‐low risk), as opposed to deferring objective testing for low risk patients. ³⁷ This is also suggested by the slight increase in 30‐day objective cardiac testing among non‐low risk patients at the end of the pre‐implementation period, corresponding to the beginning of physician education regarding the predictive value of low versus high‐normal range troponin values. It is thus conceivable that a longer observation period would have been revealing as physicians became increasingly comfortable with the notion of forgoing observation and deferring objective cardiac testing for low risk patients.

It is also notable that, on average, RISTRA‐ACS use among study eligible patients was relatively low (averaging 14% during the post‐implementation period). There was however a good deal of variability in RISTRA‐ACS use between facilities, with facility‐level averages ranging from 8% to 24%. While the drivers of this variability are likely complex (pre‐existing physician biases, perceived lack of proven utility and/or safety) and beyond the scope of this study, it is notable that sensitivity analysis following exclusion of RISTRA sites with below median RISTRA‐ACS eCDS use supported the study hypothesis in showing a statistically significant decrease in the primary outcome of index visit resource utilization among patients at low estimated risk of 60‐day MACE, in addition to the forementioned impacts on 30‐day objective cardiac testing. Thus, in treating the proportion of study eligible encounters with RISTRA‐ACS use as proxy for facility‐level adoption of the RISTRA‐ACS algorithm, this finding supports a potential for greater impact on resource utilization with higher overall adoption of the RISTRA‐ACS algorithm, such as might be realized in the wake of demonstrated safety and utility herein.

The potential for coronary risk stratification to decrease downstream resource utilization has important cost saving implications. Using a bottom‐up inventory‐based cost calculation methodology, investigators from the PROMISE trial placed the median cost of exercise electrocardiography at $174 and pharmacologic myocardial perfusion imaging at $1132. ³⁸ From that vantage, exercise electrocardiography appears to be a relatively low‐cost strategy with little potential for cost savings from resource stewardship. However, the total direct health care costs at 90 days were more similar between the two strategies, being $1770 for exercise electrocardiography and $2274 for myocardial perfusion imaging, likely owing to higher downstream costs due to false positive or indeterminate test results among patients undergoing exercise electrocardiography. Accordingly, an economic analysis of a randomized controlled trial of the HEART pathway in the ED estimated a median cost savings of $1785 at 30 days for every additional patient who did not undergo objective cardiac testing in the intervention arm. ³⁹ Thus, even a 1% absolute decrease in objective cardiac testing, as observed in this study, could yield savings upwards of 5 billion dollars annually when extrapolated to the 7.6 million patients evaluated annually for chest pain in U.S. EDs. ⁴⁰

In terms of associations between RISTRA‐ACS implementation and ED operations, we observed increases at RISTRA sites in the unadjusted difference‐in‐differences for ED length of stay (+0.3 hours) and the proportion of patients with multiple troponin tests within 6 hours of ED arrival (+4.4%). While some increase in troponin testing was anticipated due to RISTRA‐ACS’s emphasis on early repeat measurement of troponin values above the level of quantitation (to establish whether values were rising), it is more difficult to attribute increases in ED length of stay to RISTRA‐ACS availability given competing factors such as ED boarding times among admitted patients. Regardless, the increase in ED length of stay was relatively small.

Finally, we observed an increase in the proportion of patients at RISTRA sites diagnosed with acute myocardial infarction during the index encounter (unadjusted difference‐in‐differences of +0.5%, 95% CI 0.1% to 0.9%). A similar increase in index diagnoses of myocardial infarction was noted in the HEART Pathway implementation trial. ³⁵ However, as RISTRA sites also saw an increase in patients with initial troponin values over the 99th percentile (+0.8%, 95% CI +0.2% to +1.4%) and a nearly identical increase in 60‐day acute myocardial infarction diagnoses, inclusive of the index encounter (+0.6%, 95% CI +0.1% to 1.1%), it is less likely that the increase in index acute myocardial infarction diagnoses represented detection of otherwise “missed” events as opposed to variations in disease incidence among ED populations over time. This highlights the importance of using both risk‐adjustment and contemporaneous controls in measuring implementation‐associated impacts.

Limitations

Limitations include the retrospective determination of risk scores, outcome ascertainment using diagnostic and procedural codes, non‐randomized assignment of implementation (RISTRA) sites, the opt‐in nature of the RISTRA eCDS interface, and a study setting within an integrated healthcare system. Regarding the retrospective risk score methodology, given that we have previously shown it to have similar reliability as compared to prospective score calculations, ³⁰ and since the methodology was applied uniformly across all sites and study periods, we expect this carried minimal potential bias. However, the retrospective risk determination was dependent on troponin values, and we observed a 4.4% increase in patients undergoing repeat troponin testing at RISTRA sites. Thus, it is possible that risk was underestimated at control sites in the post‐implementation period. However, even assuming a 15% prevalence of acute myocardial infarction and a 15% gain in sensitivity with repeat troponin testing, any resulting bias would be expected to be exceedingly small (ie, <0.1% absolute difference). ⁴¹ , ⁴²

While outcomes were determined based on diagnostic and procedural codes, there was no expected outcome ascertainment bias between control and RISTRA sites as all are overseen by a centralized health information management department. Furthermore, by restricting the study population to patients with continuous health plan coverage during the follow‐up period we were able to obtain a full accounting of healthcare encounters both within and external to the integrated healthcare system. However, it is conceivable that the follow‐up window was too short to detect changes in outcomes from differing acute management strategies. For example, in the SCOT‐HEART randomized trial of computed tomographic coronary angiography, clear statistically supported differences in follow‐up MACE were not evident until nearly 2 years later. ⁴³ , ⁴⁴ Thus further study including long‐term follow‐up is thus warranted.

Since RISTRA site designation was not randomized but rather driven by the availability of local clinical champions and study investigators, it is possible that RISTRA sites were more amenable to practice change than control sites. However, the degree of variance in RISTRA‐ACS use amongst RISTRA sites (range 8% to 24% of possibly eligible encounters) at least demonstrates non‐uniform facility‐level uptake, though we cannot exclude unmeasured confounding from local practice initiatives during the study period (we are unaware of any). Additionally, though there was some occasional crossover of physicians from RISTRA to control sites due to intermittent staffing shortages, these instances represented <0.2% of study encounters. Regardless, such crossover would be expected to bias results towards the null.

From an eCDS implementation standpoint, the potential impact of RISTRA‐ACS may have been limited due to the opt‐in structure of the interface. However, there are a variety of potential clinical scenarios and diagnostic considerations encapsulated by a chief complaint of chest pain and/or the use of troponin testing. As such, an assistive, clinician‐selected portal within the electronic health record was deemed the most pragmatic solution to achieve the five “rights” of clinical decision support for a complex decision: the right information, to the right recipients, on the right platform, in the right format and at the right time. ⁴⁵ Finally, as the study was performed within an integrated healthcare system, external generalizability cannot be assumed.

Conclusion

Implementation of a coronary risk stratification algorithm in EDs of an integrated health system appeared safe in the short‐term. While RISTRA‐ACS availability was not associated with a change in index visit resource utilization, 30‐day objective cardiac testing did safely decrease among patients with a low estimated risk of 60‐day MACE, and appropriately increased among the remainder of patients with non‐low estimated risk.

Appendix

The Kaiser Permanente CREST Network Investigators are Dustin G Mark, MD; Jie Huang, PhD; Dustin W Ballard, MD, MBE; Mamata V Kene MD, MPH; Dana R Sax, MD, MPH; Uli K Chettipally, MD, MPH; James S Lin, MD; Sean C Bouvet, MD; Dale M Cotton, MD; Ian D McLachlan, MD, MPH; David R Vinson, MD; Adina S Rauchwerger, MPH; and Mary E Reed, DrPH.

Sources of Funding

This project is supported by The Permanente Medical Group (TPMG) Delivery Science Research Program.

Disclosures

None.

Supporting information

Data S1–S3

Tables S1–S7

Figures S1–S2