Implementation effort: Reducing the ordering of inappropriate echocardiograms through a point-of-care decision support tool

Abstract

Study objective

Reduce inappropriate transthoracic echocardiograms (TTEs) using a series of Plan-Do-Study-Act (PDSA) quality improvement cycles.

Design

Three PDSA cycles were designed with the first integrating a previously published decision support tool (DST) into the electronic TTE order, the second tailoring the DST to reflect the most common inappropriately ordered TTEs at our institution, and the third integrating direct clinician education.

Setting

Malcom Randall Veterans Administration Medical Center, Gainesville, Florida, USA.

Participants

Consecutive patients were studied using the database of all TTEs performed at our institution without regard for specific patient characteristics.

Interventions

Three PDSA Cycles as described above.

Main outcome measure

Reduction in inappropriate TTEs at our institution.

Results

After implementing our DST during the first cycle, no difference in inappropriate TTEs was observed (relative risk [RR] 0.71, p = 0.12, 95 % confidence interval [CI] 0.46–1.09). After the second cycle, we observed a reduction in the proportion of inappropriate TTEs (RR = 0.69, p = 0.014, 95 % CI 0.5–0.94), however two of the four inappropriate TTEs targeted by the DST increased. Feedback gathered from clinicians in the third cycle showed significant knowledge gaps regarding appropriate use criteria for TTE.

Conclusion(s)

At our facility, implementation of a DST failed to substantially reduce inappropriate TTEs, even when adapted to facility-specific ordering patterns. Gaps in clinician knowledge about TTEs may have contributed to the inefficacy of our DST.

1. Introduction

Transthoracic echocardiography (TTE) is commonly used to assess cardiovascular complaints and the volume of TTEs performed in the US has steadily grown for years [1]. Professional societies, seeking to ensure high value of care being delivered, have developed appropriate use criteria (AUC) for TTE and other tests [2]. Despite these AUC, an estimated 10–30 % of all cardiac testing performed is considered to be inappropriate [3]. Imaging tests are a significant contributor to the rising healthcare costs in the United States, encompassing up to 14 % of Medicare part B expenditure, therefore maximizing appropriate ordering of imaging tests is a priority [4]. Inappropriate use of TTE diverts resources from appropriate cases and increases overall cost burden which could be better invested otherwise.

A wide variety of potential solutions have been studied for reducing low value care, such as inappropriate TTEs. Decision support tools (DST) are one solution that have been developed to enhance value for a variety of tests and procedures, including TTE. The goal of this type of DST is to provide clinically relevant guidance at the point of care to help the ordering clinicians and echocardiography laboratory quickly screen for inappropriate TTEs [5]. We sought to implement a similar DST at our institution with the goal of reducing the ordering of inappropriate TTEs.

2. Materials and methods

As part of the Fellows Applied Quality Training curriculum, a team of cardiology fellows and internal medicine residents worked on reducing inappropriate TTEs at our VA Medical Center in Gainesville, Florida [6]. Using the Institute for Healthcare Improvement's Model for Improvement and serial Plan-Do-Study-Act (PDSA) cycles, the team initially sought to achieve a 50 % relative reduction of inappropriate TTEs. Because the order entry system at out facility does not distinguish between inpatients and outpatients, echocardiograms for both were included. In each cycle, the proportion of inappropriate TTEs was the selected outcome measure (Fig. 1). Balancing and process measures for each cycle were different and are described below. Appropriateness of testing was determined retrospectively by team members through review of medical records of patients for whom TTE had been performed and using the 2011 AUC for Echocardiography [2]. Data were stored in a custom database on a secure, internal shared hard drive. Statistical analysis was performed with SPSS version 21 (IBM, Armonk NY). In accordance with Veterans Affairs Handbook 1058.05, this project was performed with the purpose of improving the quality of care and was determined to not qualify as human subject research.

Fig. 1.

Gantt chart of improvement cycles.

Timelines of progress through the 3 cycles of improvement over 3 academic years.

Legend Q, quarter.

2.1. PDSA Cycle 1

In the first cycle, we incorporated a 4-question DST described by Fonseca [5] within the Computerized Patient Record System at our Veterans Affairs Medical Center (Fig. 2). When a clinician ordered a TTE, they were asked to answer the questions and their yes/no responses were recorded within the order for the TTE. The order screen informed the clinician that if they responded “yes” to 2 or more questions, that there was an 80 % chance that the order was inappropriate. At that point, the ordering clinician could choose to proceed with the order or cancel it. After implementation, we reviewed the appropriateness of each echocardiogram and compared the proportion to a sample from prior to implementation by chi-square. We calculated the likelihood of answering “yes” to 2 or more questions for predicting that an order was inappropriate. In the post-cohort, we reviewed the chart for each patient to ascertain if the clinician ordering the echocardiogram had accurately answered the 4 screening questions (compared by kappa statistic) [7]. In a prior project, the estimated rate of inappropriate echocardiograms at our facility was 25 %. We selected our sample size to detect a 10 % absolute reduction in inappropriate echocardiograms with an alpha of 0.05 and power of 0.80.

Fig. 2.

Screenshot of the DST from the Computerized Patient Record System which was used in Cycle 1.

2.2. Cycle 2

In the second cycle, the DST was modified with questions that specifically targeted the most common inappropriately ordered TTEs at our institution (Fig. 3). The most common inappropriately ordered TTEs were identified by analyzing the TTEs ordered during the first cycle. The DST was modified so that ordering clinicians were not required to respond yes/no for each of the question, but instead were provided an educational list of four points summarizing the most common inappropriate TTE indications with a free text input box for them to provide a clinical question to be answered by TTE. Sample size was selected to detect 10 % absolute reduction in inappropriate echocardiograms with an alpha of 0.05 and power of 0.80.

Fig. 3.

Screenshot of the DST from the Computerized Patient Record System which was used in Cycle 2.

2.3. Cycle 3

While the DST remained in place, we sought to better understand its function by shifting focus to direct education and engagement with the ordering clinicians, predominantly, the internal medicine residents and faculty medicine attendings. We interacted with the clinicians during their routine educational conferences with a combination lecture/feedback session to understand any knowledge gaps regarding TTE and AUC. The same sampling methods used in the prior 2 cycles were intended for use with this intervention. After a single session of education with internal medicine residents, this cycle was abbreviated due to the COVID-19 pandemic and the halting of educational conferences as well as practice changes. As a result of COVID, many QI projects at our facility were delayed or discontinued to focus on immediate risks of the pandemic.

3. Results

3.1. Cycle 1

We reviewed a total of 597 TTE orders, 294 before and 293 after the implementation of the DST. The proportion of inappropriate studies did not decrease significantly after implementation (16.3 % versus 11.6 %, RR 0.71, 95 % CI 0.46–1.09, p = 0.12). The sensitivity of the DST for predicting an inappropriate study was 24 % and the specificity was 89 %. We found significant disagreement between the ordering clinician responses to the DST questions and responses obtained during chart review. Q1 and Q4 showed least agreement between ordering clinician response and response obtained via chart review (ĸ values = 0.5 and 0.49 respectively) whereas Q2 and Q3 showed only moderate agreement between ordering clinician and chart review (ĸ values = 0.65 and 0.65). The most common inappropriate indications we observed are listed in Table 1 and notably only make up 22 % of all the inappropriate TTEs ordered.

Table 1.

A comparison of the most common indications for inappropriate echocardiograms.

	Cycle 2	Fonseca et al
Proportion of all inappropriate tests prior to intervention	Routine surveillance of ventricular function with known CAD and no change in clinical status or cardiac exam (17.2 %)	Initial evaluation when there are no other symptoms or signs of valvular or structural heart disease (36 %)
	Asymptomatic isolated sinus bradycardia (6.1 %)	Transient fever without evidence of bacteremia or a new murmur (26 %)
	Routine surveillance (<3 years) of mild valvular stenosis without a change in clinical status or cardiac exam (6.1 %)	Initial evaluation of ventricular function with no symptoms or signs of cardiovascular disease (9 %)
	Routine perioperative evaluation of ventricular function with no symptoms or signs of cardiovascular disease (4 %)	Suspected pulmonary embolism to establish diagnosis (6 %)

3.2. Cycle 2

We reviewed a total of 801 TTE orders, 450 before and 351 after the implantation of the modified DST. The proportion of inappropriate studies decreased significantly after the implementation (22 % versus 15.1 %, RR 0.686, 95 % CI 0.5–0.94 p = 0.014) (Fig. 4). Because this cycle of improvement was targeting the most common inappropriate indications at our facility, we specifically evaluated the change in frequency of those most common inappropriate indications. Despite an overall decrease in inappropriate TTE orders, we found that 2 of the 4 most common inappropriate indication had increased despite being specifically discouraged in the DST (Table 2).

Fig. 4.

Changes in appropriate and inappropriate ordering of TTEs after two PDSA cycles.

Table 2.

Changes within the four most common inappropriate reasons for ordering TTE.

	Pre-intervention inappropriate TTE (99 out of 450)	Post-intervention inappropriate TTE (53 out of 351)
Asymptomatic isolated sinus bradycardia	6 (6.1 %)	3 (5.7 %)
Routine surveillance of ventricular function with known CAD and no change in clinical status or cardiac exam	17 (17.2 %)	24 (45.3 %)
Routine perioperative evaluation of ventricular function with no symptoms or signs of cardiovascular disease	4 (4 %)	13 (24.5 %)
Routine surveillance (<3 years) of mild valvular stenosis without a change in clinical status or cardiac exam	6 (6.1 %)	0 (0 %)

3.3. Cycle 3

After a single education session during cycle 3 we were able to gather valuable information. There were several common misconceptions shared by many of the residents. There was an overall lack of knowledge regarding the parts that make up a complete TTE and the difference between a complete TTE and limited TTE. Many complete TTEs were being ordered for volume assessment where a limited TTE would suffice. Most residents were unaware of the value of TTE for evaluating presyncope versus true syncope. Residents frequently ordered TTEs for non-cardiac chest pain with negative biomarkers because they felt compelled to offer additional testing. There was lack of clarity regarding when to order a new TTE for patients with acute exacerbations of chronic heart failure with reduced ejection fraction who had a TTE completed within the past year. The residents offered creative solutions including an interactive menu that could use logical flow to guide the clinician towards the appropriateness of the order at point of entry.

4. Discussion

We conducted a multiple PDSA cycle quality improvement initiative to reduce inappropriate TTEs. When adopted directly as published, the clinician-facing DST we selected was insufficient to reduce inappropriate TTEs. When we later adapted the DST to reflect our local care patterns, we observed a reduction in the overall rate of inappropriate TTEs, however, there was an unexpected increase of inappropriate TTEs within two of the four groups targeted by the DST. Lastly, we hosted educational sessions that revealed substantial knowledge gaps amongst physicians who frequently order TTEs.

4.1. Why did Cycle 1 not work?

Our first cycle was likely unsuccessful for at least 2 major reasons. First, it failed to address our local practice patterns. The prevalence of TTE indications varies between hospitals and countries, limiting the applicability of directly adopting another facility's DST [8]. The DST created by Fonseca had an 80 % sensitivity for predicting an inappropriate TTE in their study but only a 24 % sensitivity in our study [5]. An external solution is more likely to be successful if adapted to the local environment using institutional evidence [9], [10]. The second reason was that physicians did not provide correct responses to the screening questions. Unfortunately, we could not ascertain if this was because the responses were simply incorrect, if the requested information was too difficult to find in the chart, if they were ignoring the questions, or if they knowingly providing inaccurate information to get the order they wanted. This may represent a form of reactance, where physicians resist use of clinical reminders and decision support tools that are perceived as a threat to autonomy [11]. While we offered all clinicians the opportunity to comment on our DST before it was adopted, we received no feedback. We interpreted this to mean the DST was acceptable, but we cannot be sure that clinicians silently opposed its use. Based on the poor agreement in the screening questions, we elected to make the second cycle more user friendly, rather than more punitive, such as with a hard-stop order entry system.

4.2. Why did Cycle 2 not work?

While we observed a significant reduction in inappropriate TTEs after the second cycle, the finding was spurious since our targeted indications were not reduced. If we had only evaluated the overall rate of inappropriate tests, we would have incorrectly rejected our null hypothesis and committed a Type 1 error. We think this is an important lesson to remind people engaged in QI activities that process and balancing measures can be equally important as outcome measures in order to fully comprehend the impact of a change. In making the transition from Cycle 1 to 2, we removed the requirement for clinicians to provide yes/no answers to our screening questions. This decision was made because our analysis suggested that clinicians were not providing accurate responses and postulated that making the DST less aggressive would cause clinicians to be more receptive. A previous systemic review found that overall hard stops often achieved desired process measure improvements and were superior to soft stops, but the use experience was heterogenous depending on the nature of the hard stop. Simple hard stops which were quick and did not impact management were well received but lengthier tasks or hard stops which impacted desired management were poorly received and providers would attempt to bypass them even if inaccurate or contradictory information was provided [12]. Several previous studies have shown providers often provide inaccurate responses to DSTs to justify their orders [13], [14], [15]. This change meant that we could not determine the degree to which they engaged with them in good faith. Absent these data, the most likely explanation is that even after revisions to our DST, it did not provide guidance to the clinicians on a majority of inappropriate indications for TTE.

4.3. Do DSTs work?

Prior systematic reviews and meta-analyses have shown mixed results amongst several studies which have attempted to use DSTs within electronic health records to reduce inappropriate test ordering. Winchester et al. performed a systemic review on the effect of AUC for reducing inappropriate cardiac imaging and found most studies which demonstrated a reduction in inappropriate care relied on multiple-component implantation strategies which include education, audit, and computerized DST. Nearly all studies which failed to find said reduction relied on a single-component implementation strategy [16]. Goldzweig et al. performed a systemic review on the effect of DSTs on diagnostic radiology test ordering and found that studies showed a moderate decrease in inappropriate studies but noted prominent heterogeneity (I² of 100 %) amongst all reviewed studies [17]. Chen et al. implemented a similar DST intervention at their institution and found the initial decrease in inappropriate TTEs decayed after one year [18].

AUC for echocardiography are complicated and may not be well suited to a reductionist DST. The AUC contains just over 200 indications, and it is an arduous undertaking to review the AUC against the clinical history of each patient who has a TTE ordered [2]. A DST for TTE must simplify this list, or it would be too difficult to use. Our experience suggests that even if a DST targets the most common inappropriate TTE indications, those indications still represent a minority of the overall volume of inappropriate TTEs and the most common indications may shift over time. Addressing this second problem would require both an active tracking of all inappropriate TTEs and regular revisions to the DST, two time and resource intensive activities that may not be a valuable effort. Another potential limitation, that was beyond the scope of our QI project is the perception or belief that AUC do not adequately capture the complexity of day-to-day clinical care resulting in clinicians practicing outside guidelines for certain “gray area” patients [14], [15]. An alternative approach for simplifying best practice used an advisory popup to discourage repeat TTE for inpatients and observed a 20 % reduction in overall volume, however individual appropriateness determinations were not made [19].

4.4. What might have worked instead?

Some behavior around ordering low-value tests may arise from a clinician's feeling that they need to “do something”. When a DST offers an appropriate alternative to an inappropriate test order, the DST may be more effective in reducing inappropriate imaging. Prior studies have found clinicians were more likely to change or cancel an initially inappropriate imaging order when provided with an appropriate alternative compared to no suggested alternative [20], [21].

Using “hard stops” has been shown to be significantly more effective at reducing inappropriate testing compared to simple pop-up alerts which can be easily bypassed [9], [12], [22]. In our interventions, we avoided this strategy out of concern that it would frustrate our network of hundreds of clinicians. Based on our data from cycle 1 on the incorrect responses to our screening questions, we feel this concern was valid. As discussed earlier, a multiple-component implementation strategy has been shown to reduce inappropriate imaging. We might have found more success if we implemented teaching and elicited feedback prior to, or in conjunction with, the DST in the first two cycles.

This study was a QI project using pre and post intervention cohort data without randomization or blinding. This study was limited to a single Veterans Health Administration facility, and while the Veteran population skews towards predominantly elderly males, we do not have any reason to think this would bias the impact of a DST for TTE. Our third QI cycle was prematurely abbreviated, and we were only able to gather feedback from a limited sample of trainees, and not physician faculty.

4.5. Limitations

Because the project was conducted as QI and not research, some data elements which may be preferable to report were not available for this manuscript. Because of the cyclical nature of the project, data on appropriateness were not collected continuously in order to show larger trends in ordering habits. Changing team members for each cycle may have introduced inconsistencies in the data collection and adjudication of appropriateness.

5. Conclusions

Our QI approach to reducing inappropriate ordering of inpatient TTEs has highlighted the significant gaps in clinician knowledge about TTEs and the challenging nature of changing practices to reduce inappropriate TTEs. In unstructured discussions with ordering clinicians, we observed that they had limited understanding of what a TTE can and cannot evaluate, when TTE may enhance patient management, and appropriateness in general.

Disclosures

This material is the result of work supported with resources and the use of facilities at the Malcom Randall Veterans Affairs Medical Center. All authors have no competing interests to disclose. Dr. Winchester is supported by a VA HSR&D Career Development Award-2 #13-023.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.