Impact of Clinician Engagement on Implementation of the Pediatric Echocardiography Appropriate Use Criteria

Abstract

We sought to determine whether awareness of the pediatric appropriate use criteria (AUC) affected transthoracic echocardiogram (TTE) ordering by cardiologists; evaluate for differences in classification of an encounter’s AUC clinical scenario by investigator chart review compared with clinicians at the time of the encounter; and assess TTE yield by appropriateness rating. AUC clinical scenario(s) were assigned to patients without prior TTE by investigator chart review of visits in 2014 (n = 209) and 2016 (n = 199), and by clinician classification in 2016 (n = 671). Investigators documented TTE utilization and findings. Between 2014 and 2016, TTE utilization decreased from 54 to 33% (p < 0.001) of “rarely appropriate” (R) encounters, resulting in a decrease in overall TTE utilization from 73 to 55% of encounters (p < 0.001). There was only moderate agreement of AUC appropriateness by investigator chart review and clinician classification (κ = 0.533, p < 0.001). Abnormal TTE findings were detected in 18.7% of R encounters, with 21 of 24 abnormalities in infants younger than 4 months presenting with murmur. A decrease in TTE utilization for R encounters may represent a change in practice from increased awareness of the AUC. AUC scenario classification by clinicians at the time of the encounter could be superior to chart review. TTE abnormalities may be missed in infants younger than 4 months of age when evaluated for murmurs with the AUC.

Introduction

Over the past decade, the American College of Cardiology, along with multiple partner organizations, has worked to develop “appropriate use criteria” (AUC) for various cardiovascular tests. These are mostly based on expert consensus due to a paucity of data on presentation-specific positive and negative predictive values. The intent of the AUC is to encourage clinicians to order tests only when they are cost-effective and likely to benefit patient care, while curbing the rapidly rising utilization of technologies such as echocardiography [1, 2]. In November 2014, the first pediatric AUC consensus statement for outpatient transthoracic echocardiography (TTE) was released [3]. These AUC outline 113 clinical indications for performing a pediatric patient’s first TTE, of which 53 are considered “appropriate” (A), 28 are considered “may be appropriate” (M), and 32 are considered “rarely appropriate” (R).

Several studies evaluating TTE utilization within the framework of the pediatric AUC scenarios have been published in recent years [4–11]. These studies have generally found mixed effects of the AUC on TTE prescribing practices; the outcomes appear largely dependent on the type of intervention performed, with active educational interventions more likely than passive interventions to result in a reduction in R echocardiograms. These studies have generally considered the proportion of TTEs performed that were classified by investigators as R.

Our objectives are to (1) determine the effect of the pediatric AUC on the proportion of A, M, and R encounters for which the pediatric cardiologist prescribed a TTE; (2) evaluate whether there is a difference in classification of the AUC scenarios by retrospective investigator chart review and prospective classification by the clinician at the time of the encounter; and (3) assess the yield of TTE by appropriateness rating, with special focus on evaluations of heart murmurs.

Methods

Ethical Consideration

The Vanderbilt Institutional Review Board approved the study.

Study Population

The target study population was patients for whom the pediatric AUC were intended. We included new patients who had no prior TTE evaluation and who were seen by a selected sample of four outpatient-focused pediatric cardiologists. These clinicians see 55–70% of the target study population in our group’s outpatient pediatric cardiology clinics. We included both patients who did and who did not have a TTE as part of their cardiology evaluation. Patients were excluded if they had a previous consultation by a pediatric cardiologist.

Data Management

A database of all new patient encounters from 2014, 2015, and 2016 was combined with a database of echocardiograms performed in the same time period. This combined dataset was merged with a Research Electronic Data Capture (REDCap) database hosted at Vanderbilt, which served as the repository for AUC ratings and TTE findings collected by investigators and participating clinicians [12].

2014 Investigator Chart Review

Pediatric AUC scenarios [3] were applied by retrospective chart review to randomly selected eligible patient visits between May 23 and August 22, 2014 (n = 209), prior to the publication of the AUC guidelines in November 2014. If a TTE was performed, a classification scheme of the findings (normal, incidental finding, or abnormal: mild; moderate; or severe) adapted from Sachdeva et al. [4] was also documented (Supplemental Table 1), as well as a brief description of any abnormal and incidental findings. To identify any delayed diagnoses resulting from the decision not to perform a TTE at the time of initial consultation, the findings of any echocardiograms on subsequent encounters were recorded.

In general, murmurs were considered to be “innocent” if they were systolic, no louder than 2/6, and described as “soft,” “musical,” or “vibratory.” In some instances, the documented murmur was too vague to confirm whether or not the cardiologist could rule out a pathologic murmur (e.g., “2/6 systolic murmur at left upper sternal border”); in these cases, we assumed the cardiologist was unable to rule out pathology if they ordered a TTE and thus designated the encounter as A. Two physicians (MSR and ANL) independently reviewed each chart, and any discrepancies in classification of the clinical scenario were arbitrated with a senior pediatric cardiologist (DAP).

2016 Clinician Classification

The four outpatient pediatric cardiologists received a formal general introduction to the AUC document in late 2015, via a brief presentation during a quarterly Echocardiography Quality Improvement Conference. They were then recruited to assign the most appropriate AUC scenarios to their eligible patient visits starting in 2016, at which time they received a more detailed orientation from the investigators, including instructions to classify cases based on the most relevant indication at the time of ordering the test. Tables of the numbered scenarios classified by chief complaint were provided to each clinician to aid in data entry. These tables did not include the appropriateness rating for each indication, such that the clinicians were blinded to this unless they specifically referenced the guidelines.

For each new patient meeting inclusion criteria, the clinicians entered into the REDCap database, the medical record number, the primary clinical scenario, and (optionally) any applicable secondary scenarios based on their encounter. Data were entered into the database within 24 h of finishing the documentation for each patient. The investigators confirmed every month that the providers were logging the patients who met inclusion criteria in a timely manner.

After the clinician specified the clinical scenario, the investigators then reviewed the electronic medical record to confirm appropriateness for study inclusion, and to document the same TTE information as collected in the retrospective review above. Data were collected during the same 13 consecutive weeks between May and August 2016 (n = 671) as collected in 2014. A follow-up window continued through December 2016, for identifying delayed diagnoses due to the decision not to perform a TTE at the time of initial consultation.

2016 Investigator Chart Review

To evaluate for possible misclassification by chart review, investigators reviewed a random subset of encounters included in the 2016 dataset. While blinded to the clinician classification, the same retrospective chart review process outlined above was employed (n = 199), enabling both a direct comparison between the two data collection methods and a comparison between 2014 and 2016, without confounding from different data collection methods.

Study Outcomes

The primary study outcomes were (1) TTE utilization, (2) appropriateness for TTE by AUC rating; and (3) TTE yield of abnormalities. Secondary outcomes included the same measures for murmur evaluations. Possible confounding variables were also compared between the datasets, including patient age, volume of patients seen by each clinician, and clinic location.

Statistical Analysis

Statistical analyses were performed using STATA version 14.1 software (College Station, Texas). Statistical significance was defined as p < 0.05. Differences in the 2014 and 2016 sample attributes and TTE utilization were evaluated with a series of Pearson Chi-square tests. Agreement in encounter appropriateness rating by the two 2016 data collection methods was measured using Cohen’s Kappa. When examining TTE yield, the risk of abnormal findings was reported as odds ratios (ORs) for A or M encounters as compared with R encounters and for infants from birth to three months of age as compared with children four months and older.

In an effort to observe temporal associations of TTE utilization with interventions, the percentage of new patient encounters with TTE from January 2014 through November 2016 was charted on a statistical process control p-chart. The initial baseline for TTE utilization was calculated using January 2014–October 2014. Either consecutive data points above or below the mean indicated special cause variation, resulting in a shift of the mean line.

Results

Out of 1024 new patients in the chosen timeframe in 2014, the four outpatient-focused cardiologists evaluated 563 (55%). Of these, 299 (53%) were randomly selected for review. Ninety (30.1%) patients were excluded, most commonly due to prior evaluation with TTE, and 209 (69.9%) were included for analysis.

Of 1250 new patients in the same timeframe in 2016, the four cardiologists evaluated 839 (67.1%) patients. Of these patients, 80.0% (n = 671) were included, 19.8% (n = 166) were reviewed but excluded, and 0.2% (n = 2) met inclusion criteria but were not included because the clinician had not yet been oriented to the study. Most of the patients (85%) were added to the database within two days of the clinic visit.

Comparing the sample population attributes between the two timeframes (Table 1), there were statistically significant changes in the clinician mix, with clinician D seeing a majority of new consultations in both time periods. There were also minor but statistically significant shifts in where patients were seen by these clinicians.

Table 1.

Sample population attributes

Dataset	A: 2014 investigator chart review (n = 209)	B: 2016 clinician classification (n = 671)	C: 2016 subset of investigator chart review (n = 199)	Significance
Patient age
Birth to 3 months	16.3% (n = 34)	14.9% (n = 100)	13.1% (n = 26)
4 to 11 months	6.7% (n = 14)	4.9% (n = 33)	3.0% (n = 6)
12 months to 18 years	77.0% (n = 161)	80.2% (n = 538)	83.9% (n = 167)
Clinician				*,†
A	6.7% (n = 14)	12.5% (n = 84)	14.1% (n = 28)
B	16.3% (n = 34)	15.2% (n = 102)	16.1% (n = 32)
C	15.3% (n = 32)	7.5% (n = 50)	9.1% (n = 18)
D	61.7% (n = 129)	64.8% (n = 435)	60.8% (n = 121)
Clinic location				*,†
Children’s hospital clinic	23.0% (n = 48)	30.6% (n = 205)	28.6% (n = 57)
Satellite clinic	77.0% (n = 161)	69.5% (n = 466)	71.4% (n = 142)

^*Statistically significant difference between column A and column B

^†Statistically significant difference between column A and column C

The utilization of TTE decreased significantly between 2014 and 2016 (Table 2; Fig. 1). This decrease was driven primarily by a change in TTE utilization for R scenarios when comparing the full datasets and when comparing the 2014 dataset with the 2016 subset in which the appropriateness classification method was kept constant. There was no significant change in TTE utilization for A and M encounters.

Table 2.

TTE utilization

Dataset	A: 2014 investigator chart review (n = 209)	B: 2016 clinician classification (n = 671)	C: 2016 subset: investigator chart review (n = 199)	Significance
Overall TTE utilization	72.7% (n = 152)	55.0% (n = 369)	55.8% (n = 111)	*,†
TTE utilization by AUC appropriateness for all encounters
Appropriate	88.0% (n = 81)	93.1% (n = 201)	94.6% (n = 70)
May be appropriate	68.2% (n = 15)	76.7% (n = 33)	66.7% (n = 6)
Rarely appropriate	54.0% (n = 41)	32.7% (n = 129)	29.5% (n = 33)	*,†
Unclassifiable	79.0% (n = 15)	33.3% (n = 6)	50.0% (n = 2)	*
TTE utilization by AUC appropriateness for murmur evaluations
Appropriate	95.2% (n = 40)	96.4% (n = 81)	95.7% (n = 22)
Rarely appropriate	60.4% (n = 29)	43.1% (n = 100)	41.0% (n = 25)	*,†

AUC appropriate use criteria, TTE transthoracic echocardiogram

^*Statistically significant difference between column A and column B

^†Statistically significant difference between column A and column C

Fig. 1.

Proportion of new patient encounters that included transthoracic echocardiography as part of the evaluation, presented as a statistical process control chart (p-chart), annotated with interventions that took place. These data include all new patient encounters at our institution spanning the study timeframe

There was a statistically significant difference in the appropriateness ratings between the time periods, with a greater proportion of new patient referrals classified as R in 2016 (Table 3, column A vs column B). This was particularly true for murmur evaluations, which comprised nearly half of new patient referrals. The absolute shift was attenuated but still statistically significant when holding the classification method constant (column A vs column C).

Table 3.

Appropriateness for transthoracic echocardiography

Dataset	A: 2014 investigator chart review (n = 209)	B: 2016 clinician classification (n = 671)	C: 2016 subset: investigator chart review (n = 199)	Significance
All encounters				*,†
Appropriate	44.0% (n = 92)	32.2% (n = 216)	41.2% (n = 82)
May be appropriate	10.5% (n = 22)	6.4% (n = 43)	4.0% (n = 8)
Rarely appropriate	36.4% (n = 76)	58.7% (n = 394)	42.7% (n = 85)
Unclassifiable	9.1% (n = 19)	2.7% (n = 18)	12.1% (n = 24)
Murmur evaluations				*,†
Appropriate	46.7% (n = 42)	26.6% (n = 84)	33.3% (n = 28)
Rarely appropriate	53.3% (n = 48)	73.4% (n = 232)	57.1% (n = 48)
Unclassifiable	–	–	10.7% (n = 8)

^*Statistically significant difference between column A and column B

^†Statistically significant difference between column A and column C

We found only moderate agreement between the two AUC appropriateness classification methods in the 2016 subset for all encounters (Fig. 2, Cohen’s Kappa = 0.533, p < 0.0001). Of 23 A to R discrepancies, murmur evaluations comprised 65%. The clinician rated 11 of these patients R and 4 of these patients A at the time of the encounter, while investigator chart review found the opposite in these cases (Supplemental Table 2).

Fig. 2.

Agreement of appropriateness rating between methods. We found only moderate agreement between investigator chart review and clinician classification, suggesting a need to engage clinicians in the classification process. *Denotes statistical significance

Using only the prospectively collected 2016 data, the overall yield of TTE abnormalities (minor and moderate) was 19.8% (Table 4). No TTEs revealed severe abnormalities during our study timeframe. The abnormality odds ratio for all A or M encounters was 1.2 (95% CI 0.7–2) when compared with R encounters (p = 0.6). The abnormality odds ratio for murmur evaluations for A or M encounters versus R encounters was 3.8 (95% CI 2–7.3) and was statistically significant (p < 0.0001).

Table 4.

Yield of TTE abnormalities

TTE classification	Normal	Incidental	Abnormal: minor	Abnormal: moderate	Odds ratio (95% CI)
Overall yield	68.6% (n = 253)	11.7% (n = 43)	14.9% (n = 55)	4.9% (n = 18)
TTE yield by AUC appropriateness for all encounters					1.2 (0.7–2.0)
Appropriate	73.1% (n = 147)	3.0% (n = 6)	17.9% (n = 36)	6.0% (n = 12)
May be appropriate	97.0% (n = 32)	0.0% (n = 0)	3.0% (n = 1)	0.0% (n = 0)
Rarely appropriate	52.7% (n = 68)	28.7% (n = 37)	14.0% (n = 18)	4.7% (n = 6)
Unclassifiable	100.0% (n = 6)
TTE yield by AUC appropriateness for murmur evaluations					3.8* (2.0–7.3)
Appropriate	45.7% (n = 37)	2.5% (n = 2)	40.7% (n = 33)	11.1% (n = 9)
Rarely appropriate	43.0% (n = 43)	35.0% (n = 35)	17.0% (n = 17)	5.0% (n = 5)
TTE yield by age group for rarely appropriate murmur evaluations					14.6* (1.9–114.2)
Birth to 3 months	25.4% (n = 17)	43.3% (n = 29)	23.9% (n = 16)	7.5% (n = 5)
4 to 11 months	71.4% (n = 15)	23.8% (n = 5)	4.8% (n = 1)
12 months to 18 years	91.7% (n = 11)	8.3% (n = 1)

AUC appropriate use criteria, TTE transthoracic echocardiogram

^*Statistically significant difference

The yield of abnormalities for the R scenario “presumptively innocent murmur with no symptoms, signs, or findings of cardiovascular disease and a benign family history” was significant, particularly in infants. Most patients with abnormalities were under 4 months with the exception of one 11-month-old patient diagnosed with a small atrial septal defect (Supplemental Table 3). The abnormality odds ratio for infants from birth to three months with presumably innocent murmurs was significant at 14.6 (95% CI 1.9–114.2) when compared to older children (p = 0.01).

Fourteen patients for whom a TTE was not part of the initial evaluation had TTEs ordered on a subsequent encounter, most commonly after an arrhythmia was detected on Holter monitoring. Three of these TTEs revealed incidental findings, and the remaining 11 revealed normal anatomy and function.

Discussion

Our key findings include that there was (1) a decrease in TTE utilization for R encounters since the publication of the pediatric AUC, (2) only moderate agreement on AUC appropriateness rating between retrospective chart review and prospective AUC classification by the clinicians, and (3) one or more abnormalities on nearly one-third of TTEs performed for R murmur evaluations in infants from birth to 3 months old.

The noted decrease in TTE utilization from 2014 to 2016 spanned the time in which the pediatric AUC were published. Importantly, there was a decrease in TTE utilization for R encounters, indicating a change in clinical practice, not simply a change in referral patterns. While this study detected a decline in utilization specific for murmur evaluations, it was insufficiently powered to detect declines for other categories of indications (e.g., chest pain, syncope).

We considered multiple potential change drivers for temporal association with reduced TTE utilization. First, the publication of the pediatric AUC document in late 2014 incited some discussion among academic clinicians about high-value care. Second, MSR and DAP presented an introduction to the pediatric AUC in mid-2015, to our division during its quarterly Echocardiography Quality Improvement Conference. Finally, we began involving clinicians in the data collection in mid-2016. Interestingly, the beginnings of the decrease in TTE utilization predated the publication of the AUC and recurred only when the clinicians began classifying patients as A, M, or R. The latter observation is consistent with prior work demonstrating the necessity of active interventions to effect change [10].

We engaged clinicians directly in assigning the AUC scenarios and compared this method to the retrospective chart review approach. While our clinic workflow necessitated introduction of possible bias by allowing the clinicians to classify the scenarios after knowledge of TTE findings, we found their involvement to mitigate much of the perceived ambiguity of the retrospective chart review.

We found only moderate agreement in the classification of the AUC clinical scenarios between chart review performed by investigators and clinicians at the time of the encounter, suggesting significant susceptibility of chart review to misclassification bias. For example, the clinicians occasionally correctly classified cases as A when they could not rule out a pathologic murmur, but then ultimately documented an innocent murmur on examination after the echocardiogram was normal and they were reassured that the auscultated murmur was indeed innocent in nature. Conversely, the clinicians sometimes classified cases as R when the documented murmur description may have lacked the details necessary to inform the investigator of whether the clinician thought the murmur was innocent or pathologic. In these cases when a TTE was ordered, the investigators generally incorrectly assumed that the auscultated murmur was pathologic. Though improvement in documentation quality could ameliorate some of this ambiguity, we hesitate to recommend an increase in documentation burden for busy outpatient pediatric cardiologists, without clear evidence that this added documentation would benefit patients.

Our pediatric AUC study includes all new patient encounters in which the clinician must decide for or against TTE, rather than exclusively evaluating TTEs that were performed. As the data presented above begin to inform changes in our clinical practice, we believe that the outcome measures we used stand to more effectively support quality improvement efforts than many prior studies because they possess greater clinical relevance than proportions of TTEs performed.

Our overall yield of TTE abnormalities (19.8%) was somewhat higher than the 10% reported by Sachdeva et al. [4]. Whether this is because of a more judicious practice style or a result of over-calling abnormalities will be the subject of further study. For example, “small atrial septal defect vs. patent foramen ovale” was considered by investigators to be abnormal, since an atrial septal defect has not been excluded and follow-up is generally thought to be indicated in these cases. However, one could argue that these abnormalities can be described as incidental findings.

Unlike the findings reported by Rose-Felker et al. [5], our yield of TTE abnormalities for R murmur encounters was significant. None of these abnormalities were life threatening, most were minor, and many were unrelated to the clinical indication for TTE; however, they all do require follow-up. The most common findings were small secundum atrial septal defects in infants less than four months of age. Perhaps this is to be expected, as one of the factors that increase overall utilization of TTE is young age [6, 11]. Though utilization of TTE for R indications was significantly higher than Stern et al. [11], we showed that 21/67 TTEs in the birth to 3-month age group revealed abnormalities when the cardiologist presumed the murmur to be innocent. This contrasts greatly with Rose-Felker et al. [5], who found that 0/39 pediatric patients with R TTEs had abnormal findings. Interestingly, we found no other prior studies that specifically evaluated the yield of TTEs in this age group. Further study is required to determine if the classification for innocent murmurs in the birth to 3-month age group would better fit in the M category.

Limitations

Our study was subject to some limitations. This was a single-center study that only included TTE ordering by pediatric cardiologists, which may somewhat limit generalizability. While misclassification bias was inherent in chart review, the clinicians were also subject to potential misclassification bias due to knowledge of relevant TTE findings.

This first iteration of the pediatric AUC is not perfect. We encountered many patients whose presentation fit into multiple clinical scenarios in the AUC. For example, encounters that could be classified as “probable neurocardiogenic (vasovagal) syncope” could almost always also be classified as “syncope with no other symptoms or signs of cardiovascular disease, a benign family history, and a normal [electrocardiogram].” Our evaluation was thus necessarily limited to overall appropriateness category (A, M, or R) rather than individual scenarios. Despite this limitation, we also demonstrated that the AUC scenarios can serve as a useful framework for performing quality improvement work.

In looking ahead to practice changes stemming from quality improvement initiatives, we must bear in mind that prospective data collection limits the timely detection of missed clinically significant diagnoses. Minor misses may not be captured by our methods. We assume that nearly all moderate to severe missed abnormalities would ultimately be evaluated at our facility as the regional tertiary care center for pediatric cardiology and cardiac surgery. It is quite reassuring that, under this assumption, in over 2 years of follow-up from the retrospective data in 2014, no significant congenital heart disease was apparently missed in this subset of patients.

Conclusions

There was a decrease in TTE utilization since the publication of the pediatric AUC in our center, driven by a reduction in TTE utilization for R encounters. There was only moderate agreement on AUC appropriateness rating between investigators performing chart review and clinicians specifying the AUC clinical scenario at the time of the visit, so future AUC quality improvement work should consider prospectively engaging clinicians in the process. The yield of TTE abnormalities for R murmur evaluations, as defined by the consulting pediatric cardiologist, in infants from birth to 3 months old was nearly one-third, mostly due to small atrial septal defects, suggesting that this age group may benefit from changing its indication from R to M.

Abbreviations

A

Appropriate

AUC

Appropriate use criteria

M

May be appropriate

OR

Odds ratio

R

Rarely appropriate

REDCap

Research electronic data capture

TTE

Transthoracic echocardiogram