Effects of an Electronic Medical Record Intervention on Appropriateness of Transthoracic Echocardiograms: A Prospective Study

Abstract

Background:

Transthoracic echocardiograms (TTEs) account for approximately half of U.S. spending on cardiac imaging. We developed an electronic medical record (EMR)-based decision-support algorithm for TTE ordering and hypothesized that it would increase the appropriateness of TTE orders.

Methods:

This prospective observational study was performed at the Veterans Affairs Ann Arbor Healthcare System. From October to December 2016 (preintervention), consecutive TTEs ordered in the inpatient, outpatient, and emergency department settings were included. In May 2017, a decision-support algorithm was incorporated into the EMR, giving immediate feedback to providers. Chart review was performed for TTEs ordered from June to August 2017 (early intervention) and from June to August 2018 (late intervention). Appropriateness was determined based on the 2011 appropriate use criteria for echocardiography.

Results:

Appropriate TTE orders increased from 87.6% preintervention to 94.5% at early intervention (z = 0.00018) but decreased to 90.0% at late intervention (z = 0.51, compared with preintervention). Among patients with no previous TTEs in our system, 95.3% of TTEs were appropriate, compared with 87.7% of TTEs for patients with prior TTEs within 30 days prior (odds ratio = 2.85; 95% CI, 1.18–6.31; P = .005).

Conclusions:

The EMR algorithm initially increased the percentage of appropriate TTEs, but this effect decayed over time. Further study is needed to develop EMR-based interventions that will have lasting impacts on provider ordering patterns.

The transthoracic echocardiogram (TTE) is an important tool in the evaluation and management of a wide array of cardiovascular diseases. Given the variety of clinical indications for this safe, noninvasive test, and its widespread availability, TTE utilization has rapidly increased.¹ Several studies based on Medicare data have revealed that echocardiography accounts for roughly half of all spending on cardiac imaging, and over 50% of patients have a repeat TTE within 3 years.^2,3 Multiple prior retrospective studies performed in different clinical settings have reported that roughly 10%−15% of TTEs are done for inappropriate indications.^4–9 Aside from the financial and logistical burden that these studies place on our health care system, incidental diagnostic findings can lead to potentially harmful downstream testing and procedures and have profound psychological effects on patients. In an effort to reduce low-yield and unwarranted testing, the American College of Cardiology Foundation, along with the American Society of Echocardiography, developed the first set of appropriate use criteria (AUC) for echocardiography in 2007, which was updated in 2011.^10,11

Prior studies have examined various interventions, mostly education-based, to improve appropriate TTE ordering.^12–15 Computer-based decision-support tools for TTE ordering have been less common and limited in their scope.¹³ At the Veterans Affairs (VA) Ann Arbor Healthcare System, we have created a comprehensive decision-support algorithm based on the 2011 AUC for echocardiography, embedded within the electronic medical record (EMR). The algorithm requires the ordering provider to answer a series of simple questions and then informs the provider whether the ordered test is appropriate, inappropriate, or of uncertain appropriateness. We hypothesized that this intervention would reduce the percentage of inappropriate TTEs performed at our institution.

METHODS

Study Design

This prospective observational study was performed at the VA Ann Arbor Healthcare System. The Institutional Review Board reviewed the study protocol and granted a waiver, as the project was felt to be consistent with quality improvement work. Prior to the intervention period, consecutive TTEs ordered in inpatient, outpatient, and emergency department settings between October and December 2016 were reviewed for appropriateness based on in-depth chart review, guided by the 2011 AUC. In May 2017, a decision-support tool, requiring providers to answer questions regarding TTE indications and providing immediate feedback on TTE appropriateness, was incorporated into the EMR at the point of TTE ordering (Figure 1). One of three investigators reviewed each TTE ordered from June to August 2017 (early-intervention period) and from June to August 2018 (late-intervention period) and classified each TTE as appropriate, inappropriate, or uncertain based on the 98 TTE indications included in the 2011 AUC. The TTE orders that did not fall under the one of these indications were coded as unclassified. The TTEs ordered for research, compensation and pension, and interventional or procedural protocols (such as post-procedural pericardial effusion checks, pre– coronary artery bypass graft, and 30-day transcatheter aortic valve replacement follow-ups) were excluded, as were TTEs for which insufficient clinical data were available. However, TTEs that were ordered for evaluation of changes in status or to drive clinical decision-making, such as whether to remove pericardial drains, were included. For scenarios where the indication to code was considered complicated, the primary investigator served as a second reviewer and a consensus was reached on the ultimate indication to code. Ultimately, 190/1,633 (11.6%) of TTE orders were reviewed by two investigators. The time between each patient’s most recent TTE in the VA Ann Arbor system and the currently evaluated TTE was also noted. AUC indications were categorized into groups to help determine the most common general reasons for TTE orders and which of these reasons accounted for the most inappropriate TTE orders.

Figure 1.

Screenshots of decision-support tool. (A) Initial screen. (B) Example of output for an appropriate TTE. (C) Example of output for an inappropriate TTE.

Statistical Analysis

To determine postintervention appropriateness and effects of the intervention over time, proportions of appropriate and inappropriate TTEs were compared among preintervention, early-intervention, and late-intervention periods using z tests; Bonferroni correction was applied to z scores due to multiple comparisons. To evaluate the relationship between time since previous TTE and TTE appropriateness, odds ratios (ORs) were calculated. These analyses were performed with Stata (College Station, TX). Based on the effect size observed in the comparison of preintervention and early-intervention periods, we calculated the minimal sample size for the late-intervention period using G*Power 3.1 (Dusseldorf, Germany), with alpha 0.05 and power 0.80.

RESULTS

We reviewed 572 preintervention, 552 early-intervention, and 629 late-intervention TTE orders, of which we excluded 65, 26, and 29 orders, respectively. This resulted in a total of 1,633 analyzed TTEs, performed on 1,576 unique patients. The population was 97.0% male, and common chronic conditions included coronary artery disease (45.3%), obesity (44.9%), and diabetes mellitus (43.0%; Table 1).

Table 1.

Demographic data on the 1,576 unique patients whose TTEs were included in the analysis

Demographic characteristic	Value
Age, years ± SD	68.6 ± 10.5
Sex, male	1,529 (97.0)
Coronary artery disease	714 (45.3)
Myocardial infarction	170 (10.8)
Diabetes mellitus	677 (43.0)
Chronic kidney disease	542 (34.4)
Heart failure	549 (34.8)
Stroke/transient ischemic attack	135 (8.6)
Atrial fibrillation	503 (31.9)
Hypertension	359 (22.8)
Obesity (BMI $ 30)	708 (44.9)

BMI, Body mass index.

Data are n (%) unless otherwise specified.

Appropriate TTE orders increased from 444/507 (87.6%) preintervention to 497/526 (94.5%) during the early-intervention period (z = 0.00018). Using the difference between proportions of appropriate TTEs during the first two periods, we calculated a minimum sample size of 268 for the late-intervention period. In the late-intervention period, appropriate TTE orders decreased to 540/600 (90.0%; z = 0.51 for comparison with preintervention period). For inpatients, appropriate orders increased from 222/248 (89.5%) preintervention to 239/247 (96.8%; z = 0.0042) during the early-intervention period but decreased to 167/182 (91.8%) during the late-intervention period (z = 1.00 for comparison with preintervention period). For outpatients, appropriate orders increased from 194/231 (83.9%) preintervention to 244/265 (92.1%; z = 0.015) during the early-intervention period but decreased to 359/404 (88.9%) during the late-intervention period (z = 0.51 for comparison with preintervention period). For emergency department studies, there was no significant difference comparing preintervention to early (z = 1.00) or late (z = 1.00) intervention periods (Figure 2). Total inappropriate TTE orders decreased from 35/507 (6.9%) preintervention to 15/526 (2.9%; z = 0.0072) during the early-intervention period but increased to 28/600 (4.7%) during the late-intervention period (z = 0.33 for comparison with preintervention period; Table 2). Evaluation of TTEs by location, including all three study periods, showed that outpatient TTEs were least likely to be appropriate (88.6% vs 92.3% for inpatient, z = 0.005, and 98.2% for emergency department, z = 0.024 for comparison with outpatient).

Figure 2.

Changes in appropriate, inappropriate, and uncertain/unclassified TTEs by study period.

Table 2.

Effect of intervention on TTE appropriateness

	Preintervention	Early intervention	Late intervention	Z value (pre- vs early)*	Z value (pre- vs late)*	Z value (early vs late)*
Inpatient
Appropriate	222 (89.5)	239 (96.8)	167 (91.8)	0.0042	1.00	0.06
Inappropriate	16(6.5)	3(1.2)	8 (4.4)	0.0060	1.00	0.12
Uncertain	9 (3.6)	5 (2.0)	3(1.7)	0.84	0.66	1.00
Not classified	1 (0.4)	0 (0.0)	4 (2.2)	0.96	0.26	0.057
Outpatient
Appropriate	194(84.0)	244 (92.1)	359 (88.9)	0.015	0.23	0.51
Inappropriate	19(8.2)	12 (4.5)	20 (5.0)	0.27	0.30	1.00
Uncertain	13(5.6)	5(1.9)	21 (5.2)	0.078	1.00	0.090
Not classified	5 (2.2)	4(1.5)	4(1.0)	1.00	0.69	1.00
Emergency department
Appropriate	27 (96.4)	14 (100.0)	14(100.0)	1.00	1.00	N/A
Inappropriate	0(0)	0 (0.0)	0 (0.0)	N/A	N/A	N/A
Uncertain	1 (3.6)	0 (0.0)	0 (0.0)	1.00	1.00	N/A
Not classified	0(0)	0 (0.0)	0 (0.0)	N/A	N/A	N/A
Combined
Appropriate	443 (87.4)	497 (94.5)	540 (90.0)	0.00018	0.51	0.016
Inappropriate	35 (6.9)	15 (2.9)	28 (4.7)	0.0072	0.33	0.33
Uncertain	23 (4.5)	10 (1.9)	24 (4.0)	0.048	1.00	0.12
Not classified	6(1.2)	4 (0.8)	8(1.3)	1.00	1.00	1.00

Data are presented as n (%).

^*Calculated with Bonferroni correction.

Overall TTE volume increased over time, from 3,743 studies in 2016 to 3,816 in 2017 to 3,975 in 2018. Interestingly, in 2017, before the EMR intervention was implemented in May, an average of 356 TTEs were done per month, with a decrease to 291 TTEs per month afterward (Figure 3). However, by 2018, the average had increased to 331 per month.

Figure 3.

Total TTEs performed monthly before and after implementation of the EMR intervention. The pre-intervention data collection period was October-December 2016, and the early-intervention period was June-August 2017.

Among patients who had prior TTEs within 30 days of the index TTE, 87.7% of index TTEs were appropriate. For patients with prior TTEs 31–180 days before the index TTE, 93.8% of index TTEs were appropriate (OR = 2.11; 95% CI 0.76–5.74; P = .10, compared with those with prior TTEs within 30 days). Among all patients with prior TTEs in the system, 895/1,002 (89.3%) were appropriate. For patients with no previous TTEs in the system, 95.3% of index TTEs were appropriate (OR = 2.85; 95% CI, 1.18–6.31; P = .005, compared with within 30 days; OR = 2.42; 95% CI, 1.57–3.83; P < .0001, compared with all patients with prior TTEs; Table 3).

Table 3.

Patients with repeat TTEs: relationship between time since last TTE and TTE appropriateness

		OR (95% CI)
Time since last TTE	Number (%) of appropriate TTEs	Compared with 0–30 days	Compared with 31–180 days	Compared with 181–365 days	Compared with 365+ days
0–30 days	71/81 (87.7)	–
31–180 days	165/176(93.8)	2.11 (0.76–5.74)	–
181–365 days	126/144(87.5)	0.99 (0.38–2.40)	0.47(0.19–1.09)	–
365+ days	533/601 (88.7)	1.10(0.48–2.29)	0.52 (0.24–1.03)	1.12 (0.60–1.99)	–
No previous	586/615 (95.3)	2.85 (1.18–6.31)*	1.35(0.59–2.85)	2.89 (1.46–5.56)†	2.58 (1.62–4.20)‡

^*P = .005.

^†P = .0005.

^‡P < .0001.

Additionally, we evaluated appropriateness by TTE indication (Supplemental Table 1). The most common indications were heart failure/cardiomyopathy/ventricular function (group 3) and valves/murmurs/endocarditis (group 6), comprising 50.8% of TTEs ordered. Only 88.1% of these TTEs (88.0% of group 3 and 88.2% of group 6) were appropriate. Preintervention, 83.2% of the TTEs within these two groups were appropriate, which increased to 93.4% at early intervention (z = 0.00016, compared with preintervention) and decreased to 87.0% at late intervention (z = 0.20, compared with preintervention). The TTEs related to perioperative evaluation, cardiac trauma, and adult congenital disease (group 11) were the least common and accounted for 0.80% of all TTEs, of which 23.1% were appropriate. Perioperative evaluation comprised all the inappropriate TTEs in this group. The other eight groups each had appropriateness >91% (combined 96.6% appropriate). Preintervention, 93.4% of TTEs in these groups were appropriate, which increased to 99.2% at early intervention (z = 0.00096, compared with preintervention) and decreased to 96.9% at late intervention (z = 0.097, compared with preintervention). Comparing TTE appropriateness in different care settings throughout all three study periods, the largest difference in appropriateness was seen in the heart failure/cardiomyopathy/ventricular function indication group (group 3); 201/215 (93.5%) of inpatient/emergency department studies were appropriate, while 232/277 (83.8%) of outpatient studies were appropriate.

DISCUSSION

The results of this study demonstrate that a decision-support algorithm embedded within the EMR can significantly decrease the percentage of TTEs performed for inappropriate indications. This change was significant across the inpatient and outpatient settings but decayed from year 1 to year 2 while the algorithm was in place. Similarly, the total number of TTEs performed initially decreased following implementation of the decision-support algorithm, but this effect was not sustained.

Our study builds on prior important work evaluating interventions to improve appropriate TTE ordering. Bhatia and colleagues^12,14–16 have examined educational interventions promoting the AUC in both inpatient and outpatient settings. In these studies, intervention groups received lecture-based education, electronic pocket cards with tips on appropriate ordering, and individualized feedback. Compared with controls, the educational intervention groups had a greater proportion of appropriate and decreased proportion of inappropriate TTE orders. However, these effects were not sustained in the long term.^12,14–16 A more recent intervention by Clarke et al.¹⁷ targeted clinicians who most frequently ordered TTEs in the inpatient setting and provided education in three areas: rarely appropriate reasons for inpatient TTE, appropriate indications for stat orders, and common indications for repeat TTEs. This led to a decrease of 11.1% and 32.1% for inpatient TTE orders and duplicate TTE orders, respectively.¹⁷ Although educational interventions may be effective in some settings, they are relatively labor intensive to implement and require continued reeducation in the context of provider turnover. Furthermore, busy clinicians may not have the time or motivation to consult paper references before ordering tests in the EMR.

Evidence for point-of-care ordering interventions and decision-support tools to facilitate appropriate ordering of imaging tests has grown in recent years. An educational ordering intervention for stress echocardiography, in addition to a decision-support tool, resulted in a significant decrease in rarely appropriate studies, although it was paper-based.¹⁸ In a multicenter, prospective study, a decision-support tool for imaging in suspected coronary artery disease showed favorable results, but this was not integrated into the EMR.¹⁹ Bhave et al.²⁰ previously studied a web-based application that allowed rapid determination of appropriateness of TTE orders, although this application was not embedded in the EMR. Electronic medical record-based best-practice advisories alerting providers to recent TTEs performed might prevent inadvertent duplication of testing. In a West Virginia academic medical center, a best-practice advisory alerting providers to TTEs performed ≤ 12 months prior resulted in an 8.5% reduction in inpatient TTEs from March 2016 to March 2017 compared with the previous year.²¹ However, TTE appropriateness was not tracked in this study.

The radiology literature can provide guidance on designing and implementing computerized decision-support tools to improve appropriate ordering of studies. Multiple studies evaluating decision-support tools based on radiology AUC have shown substantial improvements in appropriate ordering for magnetic resonance, computed tomography, and ultrasound tests.^22–24 These interventions have required providers to choose study indications from lists and have provided immediate feedback on appropriateness, usually in the form of scores. A common approach has been to focus initially on the highest-volume inappropriate indications, minimizing extra clicks and extraneous information for providers to digest, and then to roll out a more comprehensive system addressing a broader range of indications. Such decision-support tools often require the provider to acknowledge the appropriate use score before the order is complete, potentially leading him or her to reconsider the order if the indication is deemed potentially inappropriate.

As the vast majority of hospital- and office-based practices in the United States currently employ EMRs,^25,26 a computer-based ordering intervention to support appropriate TTE ordering is appealing from a practical standpoint. We designed our decision-support tool to give immediate feedback to providers, with the intention of educating them at the point of ordering and limiting extra clicks. Anecdotally, our colleagues did not find the intervention overly time-consuming. In the interest of minimizing provider frustration and avoiding undue influence on patient care, we chose not to create hard stops for TTEs deemed inappropriate by the algorithm. We suspect that some providers may have gone through the algorithm quickly, especially after ordering several TTEs in the system, and dismissed the appropriateness recommendations. This might explain the decay in our intervention’s efficacy over time, similar to that seen in a study at the Durham, North Carolina, VA Medical Center, focusing on TTEs ordered for heart failure.¹³ Additionally, we suspect that some providers may have learned to “game the system” by providing answers that would lead to appropriate ratings. This phenomenon might become more problematic if hard stops were put in place or if negative incentives were imposed on clinicians with higher percentages of inappropriate orders.

For patients with no prior TTEs in our system, TTE orders were more likely to be appropriate than for patients with prior studies. This is not surprising, as new signs or symptoms of cardiovascular disease are generally associated with appropriate TTE indications. Interestingly, we found that TTEs ordered in the outpatient setting were significantly less likely to be appropriate compared with those ordered while in the inpatient setting or in the emergency department. The indication group comprising heart failure, cardiomyopathy, and ventricular function exhibited the largest gap in appropriateness between outpatient and inpatient/emergency department settings. This is likely due to a lower incidence of acute clinical status changes, especially heart failure exacerbations, in the outpatient setting.

Limitations

This was a single-center study. Appropriateness determinations were based on chart review, so it is possible that providers’ intentions in ordering TTEs were not always accurately captured. Given the demographics of the VA system, the study population was almost entirely male. Additionally, VA patients have poorer health status and more medical comorbidities compared with the general population,²⁷ possibly resulting in a lower rate of inappropriate TTEs than would be seen in other practice settings. Due to the time-intensive nature of chart reviews, each study period was relatively short at 3 months. We did not have the capability to perform automated tallies of appropriate and inappropriate studies in our system, although we suspect that this would be feasible in some EMRs with adequate information technology support.

After we built and implemented our EMR algorithm based on the 2011 AUC for echocardiography, two new multimodality AUC documents were published, one for valvular heart disease and the other for nonvalvular structural heart disease.^28,29 The new criteria provide appropriateness rankings for all cardiac imaging modalities, including cardiovascular magnetic resonance, cardiovascular computed tomography, and transesophageal echocardiography. In these documents, the older designation of “uncertain” has been replaced by “may be appropriate,” and “inappropriate” has been replaced by “rarely appropriate,” recognizing that not every particular set of clinical circumstances can be fully captured by the scenarios included. We felt that it was important to maintain a consistent approach to appropriateness determinations, so we did not reference the multimodality AUC while coding our findings. We recognize that appropriateness classifications based on the new AUC may have differed slightly from those based on the 2011 echocardiography AUC. However, it is worth noting that many common TTE indications previously deemed inappropriate (for instance, transient fever without other evidence of endocarditis and routine annual reevaluation of heart failure without a change in clinical status) would be deemed rarely appropriate by the current AUC.

Future Research

Given our finding that repeat TTEs were less likely to be appropriate, EMR-based interventions to alert providers to dates and major findings of prior TTEs could increase appropriateness by limiting redundant testing. Further studies could create and evaluate decision-support tools that focus on the “low-hanging fruit” of common inappropriate TTE indications (such as reassessment of left ventricular function in stable outpatients or annual reassessment of previously mild valvular disease). In our study, the lowest appropriateness occurred in the perioperative/cardiac trauma/congenital group, of which 23.1% were appropriate, and all inappropriate TTEs were perioperative evaluations. However, this group only comprised 0.8% of all total TTEs, so any intervention in this area would likely be low-yield. Additionally, our study showed the need for continued reinforcement to sustain the initial success of the primary intervention. Possible interventions to complement and reinforce EMR decision-support tools may include brief computer-based educational modules for all new hires with yearly refreshers, scheduled oral or poster presentations at provider staff meetings, periodic e-mail reminders, phone calls from echocardiographers to discuss orders labeled as inappropriate, and individual provider report cards. Although our EMR decision-support tool did not seem overly time-consuming or burdensome based on informal feedback, future studies should qualitatively or quantitatively assess for click fatigue in ordering providers.

Finally, while the bulk of the literature on imaging appropriateness has focused on limiting inappropriate testing, we must consider the potential inadvertent effects of any intervention, namely, denying imaging services that are needed for patient care. Keeping this in mind, we feel that a computer should never have the “last word.” That is, if a decision-support tool includes hard stops, a provider should still be able to obtain a TTE that is deemed rarely appropriate by the algorithm, perhaps after receiving approval from a laboratory manager or attending echocardiographer.

CONCLUSION

We have shown that a TTE decision-support algorithm incorporated into the EMR can improve appropriate TTE ordering in the short term, although the results of our intervention decayed after a year. Thus, our intervention would need revision and further testing before we could recommend use in other centers. If durably effective, such an intervention would have the potential to be implemented across the VA system, which includes over 1,200 health care facilities and provides care for over 9 million veterans. Ultimately, we remain hopeful that EMR-based decision support could provide a broader opportunity to reduce unnecessary testing for our patients, cut costs, and decrease wait times for appropriate TTEs.

HIGHLIGHTS.

• TTEs comprise roughly half of U.S. spending on cardiac imaging; utilization is increasing.

• An EMR decision-support algorithm significantly increased appropriate TTE orders.

• The efficacy of the EMR intervention decayed over time.

Abbreviations

AUC

Appropriate use criteria

EMR

Electronic medical record

OR

Odds ratio

TTE

Transthoracic echocardiogram

VA

Veterans Affairs