Appropriateness of Inpatient Stress Testing: Implications for Development of Clinical Decision Support Mechanisms and Future Criteria

Abstract

Background:

An upcoming national mandate will require consultation of appropriate use criteria (AUC) through a clinical decision support mechanism (CDSM) for advanced imaging. We aimed to evaluate our current ability to ascertain test appropriateness.

Methods:

We prospectively collected data on 288 consecutive stress tests and coronary computed tomography angiography (CTA) studies for medical inpatients. Study appropriateness was determined independently by two physicians using the 2013 Multimodality AUC.

Results:

The median age of the study population was 66 years (interquartile range (IQR) 56, 75), 40.8% were female, and 52.8% had a history of coronary artery disease (CAD). Review of the electronic health record (EHR) alone was sufficient to deem appropriateness for 87.2% of cases. The most common reason it was insufficient was inability to determine if the patient could exercise (59.5%). After reviewing the EHR and pilot CDSM data together, appropriateness could be determined for 95.8% of the cases. The most common reason appropriateness could not be determined was that the exam indication was not addressed by an AUC criterion (83.3%).

Conclusion:

In preparing for the mandate, it will be important for future CDSM to obtain information on the patient’s ability to exercise and for future AUC to include additional indications that are not currently addressed.

Introduction

The United States Congress passed the Protecting Access to Medicare Act (PAMA) in 2014. Under this legislation, provider ordering of and reimbursement for advanced imaging, including cardiovascular imaging, will require use of a clinical decision support mechanism (CDSM) that consults an approved Appropriate Use Criteria (AUC) starting on January 1, 2020.(1–3) As practices and hospitals prepare for implementing PAMA, it is important to understand both current ordering practices, as well as how often appropriateness can be determined by AUC using current electronic health record (EHR) systems.(4) This will allow imaging centers to prepare for effective use of CDSM to determine appropriateness of advanced imaging orders.

In order to improve allocation of health care resources in cardiovascular imaging and ensure patient-centered decision-making, the American College of Cardiology Foundation, along with key specialty and subspecialty societies, developed AUC for cardiovascular imaging for specific indications.(5) The 2013 Multimodality AUC for the Detection and Risk Assessment of Stable Ischemic Heart Disease(5) (2013 Multimodality AUC) were designed to allow for rating of tests side by side for the same indication, and are currently the most widely used criteria for advanced cardiovascular imaging.

An important concern regarding the PAMA legislation’s mandate to consult AUC is that it will place significant burden on imaging centers and ordering physicians. It has been suggested that use of the EHR may simplify the process of applying various AUC. However, it is unclear how data from the EHR can be combined with a CDSM in order to streamline the AUC consultation process. Therefore, our goal was to evaluate whether data from the EHR can be used to determine appropriateness of advanced cardiovascular imaging tests ordered in the inpatient clinical setting, where there is greater heterogeneity in types of studies ordered and physicians ordering the studies, and to evaluate whether a CDSM can enhance this determination.

Methods:

Study Population

From November 2014 through May 2015, data were collected on 308 consecutive patients who were admitted to a medical inpatient service at Brigham and Women’s Hospital (Boston, Massachusetts) and underwent stress testing or coronary computed tomography angiography (CTA). The final cohort consisted of 288 patients/studies after excluding 18 nuclear or cardiac magnetic resonance imaging (MRI) studies where only rest imaging was performed, and two stress echocardiograms performed to assess severity of aortic stenosis, not CAD (Supplemental Figure 1). Only the first study was included in the final cohort if a patient had multiple studies during the study period. Patient history, presentation, symptoms, indication for testing, and ability to exercise, along with other data (Figure 1), were collected prospectively at the time of order entry using a pilot CDSM. This pilot, self-developed CDSM was designed to help collect patient-specific data that may aid in selecting between different testing options. Other demographic data were collected prospectively at the time of cardiovascular testing. Dyslipidemia was defined as a history of dyslipidemia and/or current statin therapy. Family history of premature CAD was defined as a history of CAD in a first degree relative less than 55-years-old (male) or 65-years-old (female). Obesity was defined as a body mass index of ≥30 kg/m².

Figure 1. Questions Asked at Time of Order Entry via Pilot Clinical Decision Support Mechanism.

CAD = coronary artery disease. HR = heart rate.

Exam Appropriateness

Appropriateness of each study was determined independently by two physicians retrospectively using the 2013 Multimodality AUC.(5) The physicians ascertained if appropriateness could be determined (1) by the EHR alone, (2) by the data from the pilot CDSM alone, and (3) by both the EHR and data from the pilot CDSM combined. The AUC indication for each study was determined by the physician reviewers based on the data available to them. As per the 2013 Multimodality AUC,(5) pre-test probability of obstructive CAD by age, sex, and symptoms was determined by the Diamond and Forrester method.(6) If appropriateness could be determined, the physician reviewers noted if the study was rarely appropriate, maybe appropriate, or appropriate as per the 2013 Multimodality AUC. If appropriateness could not be determined, the physician reviewers recorded why, as well what necessary data were missing (if applicable). All reasons were recorded, not just the primary reason. Therefore, each case could have more than one reason why appropriateness could not be determined. In cases where there was disagreement between the two physician reviewers, a third physician reviewed the data and made the final assessment of appropriateness.

Statistical Analysis

Continuous variables were expressed as median with interquartile range (IQR), whereas categorical variables were expressed as frequency and percentages (%). Differences in proportions for dichotomous variables were assessed using the Fisher’s Exact test. A two-tailed p-value <0.05 was considered statistically significant. Analysis was performed using Stata IC version 14.2 (StataCorp LP, College Station, TX, USA).

Results

Baseline Characteristics, Symptoms, and Indications

The median age of the study population was 66 years (IQR 56, 75). 117 (40.8%) were female, and 152 (52.8%) had a history of CAD (Table 1). A majority of the patients had hypertension (248 (86.1%)) and dyslipidemia (228 (79.2%)).

Table 1.

Baseline Patient Characteristics.

Patient Characteristic	All Studies (n=288)
Age in years, median (IQR)	66 (56, 75)
Female	117 (40.8%)
CAD	152 (52.8%)
MI	85 (29.5%)
PCI	81 (28.1%)
CABG	49 (17.0%)
Hypertension	248 (86.1%)
Dyslipidemia	228 (79.2%)
Diabetes not on insulin	53 (18.4%)
Diabetes on insulin	64 (22.2%)
Active tobacco use	25 (8.7%)
Former tobacco use	114 (39.6%)
Family history of premature CAD	63 (21.9%)
Obesity	139 (48.3%)

CABG = coronary artery bypass grafting. CAD = coronary artery disease. IQR = interquartile range. MI = myocardial infarction. PCI = percutaneous coronary intervention.

The most common symptom, obtained via the pilot CDSM, that led to stress testing/coronary CTA was chest pain (161 (55.9%)): 71 (24.7%) patients had atypical angina, 56 (19.4%) had typical angina, and 34 (11.8%) had non-anginal chest pain (Table 2). Heart failure signs or symptoms, which led to testing for 43 patients (14.9%). The two most common indications, obtained by the pilot CDSM, for testing were “known CAD; evaluate presence/amount of ischemia” (145 (50.4%)) and “no known CAD; evaluate for symptoms which are suspicious for CAD” (117 (40.6%)).

Table 2.

Frequency of All Symptoms and Indications Obtained via Pilot Clinical Decision Support Mechanism.

	Number of Exams (n=288)
Symptom
Chest pain	161 (55.9%)
Atypical angina	71 (24.7%)
Typical angina	56 (19.4%)
Non-anginal chest pain	34 (11.8%)
Heart failure signs or symptoms	43 (14.9%)
Dyspnea	38 (13.2%)
Palpitations	16 (5.6%)
Syncope/Pre-syncope	16 (5.6%)
Asymptomatic	14 (4.9%)
Indication
Known CAD; evaluate presence/amount of ischemia	145 (50.4%)
No known CAD; evaluate for symptoms which are suspicious for CAD	117 (40.6%)
Evaluate cause of LV dysfunction of unknown etiology	18 (6.3%)
Pre-operative evaluation	5 (1.7%)
Asymptomatic patient; risk assessment	3 (1.0%)

CAD = coronary artery disease. LV = left ventricular.

Ordering Practices

Nuclear myocardial perfusion imaging (MPI) was the most common test ordered (232 (80.6%)): 124 (43.1%) pharmacologic stress SPECT, 53 (18.4%) exercise stress single-photon emission computed tomography (SPECT), and 55 (19.1%) stress positron emission tomography (PET) (Figure 2A). Comparing those without known CAD and known CAD, there was a statistically significant difference between the type of test ordered (p=0.020) (Figure 2B). Of the patients without known CAD, 103 (75.7%) underwent nuclear MPI compared with 129 (84.9%) patients with known CAD.

Figure 2. Study Type Frequency.

A. Study Type Frequency - All Patients. B. Study Type Frequency - By Absence or Presence of Known Coronary Artery Disease. CTA = computed tomography angiography. ECG = electrocardiogram. ETT = exercise treadmill testing. MRI = magnetic resonance imaging. PET = positron emission tomography. SPECT = single-photon emission computed tomography.

Appropriateness

Two physician reviewers independently reached the same appropriateness classification for 200 (69.4%) of the studies. When comparing cases of patients without known CAD and known CAD, there was a statistically significant difference in agreement between both physician reviewers; they agreed on 62.5% of cases without known CAD compared with 75.7% of cases with known CAD (p=0.021). Of the 88 studies where there was not agreement between the two physician reviewers, the third physician agreed with one of the other two physicians in 68 (77.3%) of the cases. The third physician chose an alternative AUC indication for the remaining 20 (22.7%) cases. Of the 88 studies where there was disagreement between the two physician reviewers, 39 (44.3%) of the disagreements were due to one of the two physicians not recognizing a history of revascularization or the date/results of prior testing and 19 (21.6%) of the disagreements were due to differing assessments of a patient’s pre-test probability of obstructive CAD.

Review of the EHR alone was insufficient to deem appropriateness for 37 (12.8%) cases (Figure 3A). The most common reason was inability to determine if the patient could exercise (22 (59.5%)) (Figure 4A). The physician reviewers were able to determine appropriateness using the data from the pilot CDSM alone in only 41 cases (14.2%). The reasons physician reviewers could not determine appropriateness from the pilot CDMS alone for the remaining cases (n=247 (85.8%)) included: inability to narrow to one indication (99 (34.4%)), needing access to prior coronary angiography results (88 (30.6%)), needing access to the patient’s ECG (60 (20.8%)), needing access to prior stress test results (49 (17.0%)), no matching indication in the multimodality AUC (6 (2.1%)), and other (17 (5.9%)). After reviewing the EHR and pilot CDSM data together, appropriateness could be determined for 276 of the 288 cases (95.8%): 251 (87.2%) cases were deemed appropriate, 22 (7.6%) were deemed maybe appropriate, and 3 (1.0%) were deemed rarely appropriate (Figure 3B). Among the cases for which appropriateness could be determined (n=276), the most common AUC indication for cases of no known CAD (n=133) was “3 – Symptomatic; intermediate pre-test probability of CAD; ECG interpretable AND able to exercise” (32 (11.6%)) and the most common AUC indication for cases of known CAD (n=143) was “64 – Post-revascularization (PCI or CABG); evaluation of ischemic equivalent” (101 (36.6%)) (Table 4). Clinical details of the three cases that were deemed rarely appropriate are listed in Table 5.

Figure 3. Appropriateness of Studies Ordered.

A. Appropriateness of Studies Ordered Using the Electronic Health Record Alone. B. Appropriateness of Studies Ordered Using Both the Electronic Health Record and Data from the Pilot Clinical Decision Support Mechanism.

Figure 4. Reasons Appropriateness Could Not Be Determined.

A. All Reasons Electronic Health Record Alone Was Unable to Deem Appropriateness (Total Cases n=37). B. All Reasons Electronic Health Record and Data from Pilot Clinical Decision Support Mechanism Together Were Unable to Deem Appropriateness (Total Cases n=12). AUC = appropriate use criteria.

Table 4.

Most Common Appropriate Use Criteria Indications Using Both the Electronic Health Record and the Pilot Clinical Decision Support Mechanism.

AUC Indication	Total Number of Exams for which Appropriateness Could be Determined (n=276)
No Known Coronary Artery Disease	133 (48.2%)
3 – Symptomatic; intermediate pre-test probability of CAD; ECG interpretable AND able to exercise	32 (11.6%)
58 – Follow-up testing: new or worsening symptoms; nonobstructive CAD on coronary angiography (invasive or noninvasive) OR normal prior stress imaging study	26 (9.4%)
12 – Other cardiovascular conditions; newly diagnosed systolic heart failure	21 (7.6%)
4 – Symptomatic; newly diagnosed systolic heart failure	11 (4.0%)
Known Coronary Artery Disease	143 (51.8%)
64 – Post-revascularization (PCI or CABG); evaluation of ischemic equivalent	101 (36.6%)
28 – Sequential testing (≤90 days): abnormal prior test/study); obstructive CAD on prior invasive coronary angiography	9 (3.3%)
58 – Follow-up testing: new or worsening symptoms; Nonobstructive CAD on coronary angiography (invasive or noninvasive) OR normal prior stress imaging study	9 (3.3%)
62 – Follow-up testing: new or worsening symptoms; obstructive CAD on invasive coronary angiography	8 (2.9%)

AUC = appropriate use criteria. CABG = coronary artery bypass grafting. CAD = coronary artery disease. ECG = electrocardiogram. PCI = percutaneous coronary intervention.

Table 5.

Summary of Studies Deemed Rarely Appropriate.

AUC Indication	Study Performed	Clinical Context
1 – Symptomatic; low pre-test probability of CAD; ECG interpretable AND able to exercise	Exercise Stress SPECT	41-year-old woman with hypertension. Study performed due to atypical chest pain.
28 – Sequential testing (≤90 Days): abnormal prior test/study); obstructive CAD on prior invasive coronary angiography	Coronary CTA	40-year-old man with a history of vasculitis who was found to have a 70% LAD stenosis on coronary angiography four days prior. Study performed to assess for suspected coronary vasculitis.
71 – Pre-operative evaluation for noncardiac surgery; moderate-to-good functional capacity (≥4 METs) OR no clinical risk factors; any surgery	Pharmacologic Stress SPECT	75-year-old man with CAD (history of MI and PCI), hypertension, diabetes, dyslipidemia, and prior tobacco use without symptoms of CAD. Study preformed as pre-operative assessment prior to thoracic surgery.

CAD = coronary artery disease. CTA = computed tomography angiography. ECG = electrocardiogram. LAD = left anterior descending coronary artery. METs = metabolic equivalents. MI = myocardial infarction. SPECT = single-photon emission computed tomography.

Appropriateness could not be determined for 12 (4.2%) cases after reviewing the EHR and pilot CDSM data together. The most common reason was there was no AUC indication matching the exam (n=10 (83.3%)) (Figure 4B). Such cases included coronary CTA prior to cardiac surgery (n=2 (20%)) and evaluation post-spontaneous coronary artery dissection (SCAD) (n=2 (20%)).

Discussion

We applied the 2013 Multimodality AUC to observed clinical practice in a medical inpatient cohort at a large tertiary care center with extensive experience in stress testing and coronary CTA. We found that review of the EHR alone was insufficient to deem appropriateness for 12.8% of cases. The most common reason was inability to determine if the patient could exercise (59.5%). After reviewing the EHR and pilot CDSM data together, appropriateness could not be determined for 4.2% of cases; the most common reason for indeterminate exams was no AUC indication matching the exam (83.3%). Such cases included coronary CTA prior to cardiac surgery (20%) and evaluation post-spontaneous coronary artery dissection (20%).

Challenges of Applying AUC

Similar to previously reported studies,(7) the two physician reviewers independently reached the same appropriateness classification for only 69.4% of the studies. This highlights one of the major challenges in applying AUC: subjectivity in interpretation of clinical scenarios. Even with the same available data, physician reviewers with similar experience did not agree on appropriateness classification for over 30% of the cases reviewed. This has important implications for the near future when the use of CDSM and AUC will be mandated. Methods to more consistently identify the pertinent data required to delineate appropriateness will need to be developed to prevent a detrimental effect to patient care, including financial and time costs to the patient, physician, and healthcare system.

Two other challenges of applying AUC include the availability of multiple AUC for many modalities and the inherent inability for AUC to be all-inclusive of every possible clinical scenario encountered in clinical practice. The former could create differences in appropriateness assessment depending on the AUC selected and the latter could result in the inability to deem appropriateness of a study regardless of the AUC chosen, as highlighted in our study.

Opportunities for Enhancing CDSM and AUC

Our study involved detailed review of the entire EHR by two independent physician reviewers for each study ordered. This level of time and resource expenditure is impractical for real-world practice. Thus, key goals for CDSM will be to: (1) integrate efficiently and effectively with existing EHR data to provide all necessary data in an automated manner, (2) accurately apply AUC, and (3) clearly communicate appropriateness to the ordering provider. The integration with existing EHR should provide accurate, up-to-date clinical information for each patient, including information that may be difficult or time-consuming to find. For example, a fully-integrated CDSM should be able to quickly obtain and display data on pre-test probability of obstructive CAD, as well as dates and results of prior stress tests. The accurate display and application of AUC will make the required clinical information clear to the provider, as well as the criteria by which appropriateness of the order will be assessed. Finally, appropriateness of the proposed study order should be clearly communicated by the CDSM so that the ordering provider can review the reasons why an order may be deemed inappropriate and can adjust the order accordingly.

Based on the data from our study, we outline in Figure 5 potential opportunities for enhancing effectiveness of future CDSMs and future AUC. It will be important for CDSMs to be designed so that information regarding a patient’s ability to exercise and prior noninvasive and invasive testing dates and results can be obtained at the time of order entry. Additionally, it may be helpful for CDSMs to be designed so a provider can choose the AUC indication that fits best with the clinical scenario at hand if the EHR or CDSM is unable to determine a specific AUC indication on its own. Given the very low rate of maybe or rarely appropriate tests in this cohort (we found that an overwhelming majority of studies performed were deemed appropriate, consistent with prior work evaluating the appropriateness of stress testing and coronary CTA studies(8–12)), reducing the rate of inability to determine appropriateness may be a more important goal for future CDSM than reducing inappropriate use.(13) It will also be important for future AUC to include indications for pre-cardiac surgery assessment. The 2013 Multimodality AUC document has a section on pre-operative testing for non-cardiac surgery, but not for cardiac surgery. Finally, the addition of indications for evaluation of SCAD cases should be considered for future AUC.

Figure 5. Opportunities for Enhancing Effectiveness of Clinical Decision Support Mechanisms and Future Appropriate Use Criteria.

AUC = appropriate use criteria. CDSM = clinical decision support mechanism.

Study Strengths

When compared to previously published AUC studies.(14, 15) unique features of this study include that all patients studied were inpatients, a majority of patients had a prior history of CAD (52.8%), and many had a history of revascularization (28.1% of patients had a history of PCI and 17.0% of patients had a history of CABG). These patients are often encountered in the inpatient setting at many medical centers and thus make our study generalizable to similar sites. Other unique features of this study include independent review of each case by two physicians, and a third reader to resolve any cases where there was no consensus.

Study Limitations

Our study evaluated patients admitted to a medical service at a single institution, which serves as a tertiary cardiovascular referral center. Therefore, the study cohort included a large percentage of patients with a known history of CAD. Given this, the findings from this study may be less generalizable to centers where the population of patients with prior history of known CAD may be lower. A related limitation is that studies ordered on patients with known CAD are also more likely to be classified as always or maybe appropriate by the Multimodality AUC. Therefore, the appropriateness frequencies from our study should not be generalized to populations with less prevalent CAD. A majority of studies performed were nuclear cardiology studies, which again is likely a reflection of the large percentage of patients with a known history of CAD in our cohort.

Patient-specific data were collected prospectively for the cohort, but appropriateness was determined retrospectively. Finally, the pilot CDSM used in this study was not designed specifically to help determine appropriateness, and as such the fact that information in the pilot CDSM alone was not useful for 247 (85.8%) of the cases is not surprising. Obtaining and presenting information regarding the patient’s ECG and prior testing (non-invasive and coronary angiography) results would have substantially increased the usefulness of the pilot CDSM in determining appropriateness. Despite these important limitations, the data presented in this study present opportunities for improvement of future CDSM and multimodality cardiovascular testing AUC.

Conclusions:

In a medical inpatient cohort at a large tertiary care center, review of the EHR alone was insufficient to deem appropriateness for 12.8% of stress tests or coronary CTAs using the 2013 Multimodality AUC. In preparing for PAMA, it will be important for future CDSM to obtain information on the patient’s ability to exercise and for future AUC to include additional indications that are not currently addressed.

New Knowledge Gained:

In a medical inpatient cohort, appropriateness could not be determined by the EHR alone for 12.8% of stress tests or coronary CTAs using the 2013 Multimodality AUC. In preparing for PAMA, it will be important for future CDSM to obtain information on the patient’s ability to exercise and for future AUC to include additional indications that are not currently addressed.

Table 3.

Summary of Correlation of Reviewers for Appropriateness Classification.

Agreement Between Both Physician Reviewers	All Cases (n=288)	No Known CAD (n=136)	Known CAD (n=152)	Fisher’s Exact Test p-value
Number (%)	200 (69.4%)	85 (62.5%)	115 (75.7%)	p=0.021

CAD = coronary artery disease.

Abbreviations

AUC

appropriate use criteria

CABG

coronary artery bypass grafting

CAD

coronary artery disease

CDSM

clinical decision support mechanism

CTA

computed tomography angiography

ECG

electrocardiogram

EHR

electronic health record

ETT

exercise treadmill testing

MI

myocardial infraction

MPI

myocardial perfusion imaging

PAMA

Protecting Access to Medicare Act

PCI

percutaneous coronary intervention

PET

positron emission tomography

SPECT

single-photon emission computed tomography