Yield of CT Pulmonary Angiography in the Emergency Department When Providers Override Evidence-based Clinical Decision Support

When providers adhered to evidence presented in clinical decision support (CDS), the odds of an acute pulmonary embolism finding in the emergency department were nearly doubled compared with the odds when providers overrode CDS alerts.

Abstract

Purpose

To determine the frequency of, and yield after, provider overrides of evidence-based clinical decision support (CDS) for ordering computed tomographic (CT) pulmonary angiography in the emergency department (ED).

Materials and Methods

This HIPAA-compliant, institutional review board–approved study was performed at a tertiary care, academic medical center ED with approximately 60 000 annual visits and included all patients who were suspected of having pulmonary embolism (PE) and who underwent CT pulmonary angiography between January 1, 2011, and August 31, 2013. The requirement to obtain informed consent was waived. Each CT order for pulmonary angiography was exposed to CDS on the basis of the Wells criteria. For patients with a Wells score of 4 or less, CDS alerts suggested d-dimer testing because acute PE is highly unlikely in these patients if d-dimer levels are normal. The yield of CT pulmonary angiography (number of positive PE diagnoses/total number of CT pulmonary angiographic examinations) was compared in patients in whom providers overrode CDS alerts (by performing CT pulmonary angiography in patients with a Wells score ≤4 and a normal d-dimer level or no d-dimer testing) (override group) and those in whom providers followed Wells criteria (CT pulmonary angiography only in patients with Wells score >4 or ≤4 with elevated d-dimer level) (adherent group). A validated natural language processing tool identified positive PE diagnoses, with subsegmental and/or indeterminate diagnoses removed by means of chart review. Statistical analysis was performed with the χ² test, the Student t test, and logistic regression.

Results

Among 2993 CT pulmonary angiography studies in 2655 patients, 563 examinations had a Wells score of 4 or less but did not undergo d-dimer testing and 26 had a Wells score of 4 or less and had normal d-dimer levels. The yield of CT pulmonary angiography was 4.2% in the override group (25 of 589 studies, none with a normal d-dimer level) and 11.2% in the adherent group (270 of 2404 studies) (P < .001). After adjustment for the risk factor differences between the two groups, the odds of an acute PE finding were 51.3% lower when providers overrode alerts than when they followed CDS guidelines. Comparison of the two groups including only patients unlikely to have PE led to similar results.

Conclusion

The odds of an acute PE finding in the ED when providers adhered to evidence presented in CDS were nearly double those seen when providers overrode CDS alerts. Most overrides were due to the lack of d-dimer testing in patients unlikely to have PE.

^© RSNA, 2016

Introduction

The use of advanced medical imaging in the emergency department (ED) has increased substantially (1). This rapid growth has triggered concern about whether imaging is justified in certain conditions, as well as about the potential harm from imaging (2,3). Imaging overutilization may be associated with higher costs, radiation exposure, increased length of stay in the ED, and allergic reactions to intravenous contrast material (4,5).

Of particular interest is the ED use of computed tomographic (CT) pulmonary angiography for patients suspected of having pulmonary embolism (PE). A serious and potentially life-threatening condition, PE necessitates effective and timely treatment. Its diagnosis can be challenging because patients often present with nonspecific symptoms such as chest pain and shortness of breath. Clinical judgment and laboratory testing may effectively rule out PE without imaging in a subset of patients, whereas CT pulmonary angiography is an excellent tool with which to exclude PE (6). The increase in CT use has led to decreasing diagnostic yields among CT pulmonary angiographic examinations (7–9). Although the Wells criteria, a validated, evidence-based decision tool designed to help providers identify patients who require CT pulmonary angiography to exclude PE, have existed for more than a decade (10), their use in clinical practice remains sparse (11–13).

Clinical decision support (CDS) integrated with a computerized physician order entry system can help improve workflow efficiency and increase guideline adherence (14–16). Specifically, CDS based on high-quality evidence-based guidelines reduces the use of advanced imaging and decreases the rate of inappropriate imaging in certain outpatient and ED settings (17–19). An integrated computerized physician order entry and/or CDS system that is presented during the imaging ordering process and automatically calculates the Wells score helps increase guideline adherence, decrease use of CT pulmonary angiography, and increase the yield of this modality (17,18).

Typically, implementation of CDS does not include “hard stop” interventions except for the most serious circumstances (20). Therefore, physicians often override CDS recommendations, usually without providing any clinical justification. In one study, providers overrode 95% of repeated CT CDS alerts (21). It is not known whether this exercise of clinical judgment in overriding CDS is warranted. We sought to determine the frequency of, and yield after, provider overrides of evidence-based CDS for CT pulmonary angiography in the ED.

Materials and Methods

Study Setting, Design, and Participants

The institutional review board approved this Health Insurance Portability and Accountability Act–compliant study of data acquired between January 1, 2011, and August 31, 2013, at an urban level I adult trauma center ED with approximately 60 000 annual visits. The requirement to obtain informed consent was waived. The population of this retrospective cohort study included all patients suspected of having PE (including pregnant patients) who presented to the ED and underwent CT pulmonary angiography during the study period. All 2993 CT pulmonary angiographic studies (2655 unique patients) have been previously reported on (22). The previous study evaluated age-adjustment strategies for d-dimer values in the diagnosis of PE at CT pulmonary angiography; in contrast, the current study addresses the yield of CT pulmonary angiography when providers override evidence-based CDS alerts.

Intervention

The intervention consisted of a CDS tool based on validated decision rules (the Wells criteria) integrated into the ED radiology computerized physician order entry system (Percipio; Medicalis, San Francisco, Calif). During the period from January 2011 to August 2013, the two-tier Wells criteria were adopted because they are supported by the American College of Emergency Physicians’ 2011 clinical policy on PE (23). Our institution’s laboratory used STA-Liatest quantitative d-dimer latex-based automated assays with immunoturbidimetric readings (Stago, Parsippany, NJ), a high-sensitivity d-dimer test required for the two-tier model.

Each CT pulmonary angiography order was exposed to the CDS, which required data input in the form of a discrete “yes” or “no” response for each unique clinical attribute of the Wells criteria (clinical signs and symptoms of deep vein thrombosis [DVT], PE as the leading diagnosis, heart rate >100 beats per minute, immobilization for ≥3 days or surgery in the previous 4 weeks, previous PE or DVT, hemoptysis, and malignancy with treatment within the past 6 months) and a serum d-dimer level (normal, elevated, testing not ordered, or level not yet known) for the patient being imaged (Fig 1). The entered data were used to calculate the Wells score (range, 0–12.5) (Table 1), and a CDS alert was provided on the basis of the resulting score (Fig 2). For patients in whom PE was likely (“PE likely patients”) (Wells score >4), CT pulmonary angiography is recommended; thus, no CDS alert was presented to the ordering provider to alter the clinical decision to perform CT pulmonary angiography. For patients in whom PE was unlikely (“PE unlikely patients”) (Wells score ≤4), a CDS alert recommended d-dimer testing. For these patients, a normal d-dimer value effectively rules out the possibility of PE, whereas a positive d-dimer value suggests CT pulmonary angiography as the next diagnostic test. CT pulmonary angiography orders that adhered to the CDS guideline triggered no alerts. At each step of the process, the ordering provider could cancel the order or override the CDS recommendation and proceed with the CT pulmonary angiography order.

Figure 1:

Interface of computerized physician order entry system.

Table 1.

Components of the Wells Score

Figure 2:

Flowchart of CDS process. Decision support message 1: “Based on the information you have provided, CT may not be appropriate for your patient. Published guidelines suggest that a ‘PE unlikely’ patient should have a d-dimer measured to further guide decision making.” Decision support message 2: “Based on the information you have provided, CT may not be appropriate for your patient. Published guidelines suggest that a ‘PE unlikely’ patient with a normal d-dimer is unlikely to have a pulmonary embolism.” CTPA = CT pulmonary angiography.

Data Collection

For each completed CT pulmonary angiographic examination, we used a natural language processing algorithm based on General Architecture for Texting Engineering (University of Sheffield, South Yorkshire, United Kingdom) to classify it as having positive or negative results for acute PE on the basis of the binary outcome analysis of the radiology reports. The natural language processing algorithm was previously validated and published for CT pulmonary angiography reports in the ED population; examinations with indeterminate or limited findings were categorized as negative (18). Chart review was performed on all natural language processing–confirmed positive PE diagnoses to identify patients with small subsegmental and/or indeterminate PE who received no further anticoagulation treatment because the clinical relevance of these emboli is still controversial (24). For initial data analyses, these studies were considered to have negative results. We randomly sampled 50 patients among those unlikely to have PE who did not undergo d-dimer testing immediately before CT pulmonary angiography to confirm whether any underwent d-dimer testing 3 months after CT pulmonary angiography. For each CT pulmonary angiography order, the Wells score was calculated from the computerized physician order entry system database and the d-dimer value, if obtained, was acquired from the electronic health record.

Outcome Measures

The primary outcome was the diagnostic yield of CT pulmonary angiography, defined as the proportion of positive PE diagnoses among CT pulmonary angiographic examinations performed. We hypothesized that the yield of CT pulmonary angiography would be statistically significantly lower when CDS was overridden compared with when the Wells decision rule was followed.

Statistical Analysis

For our primary analysis, we compared the diagnostic yields in patients for whom the physician followed CDS recommendations (adherence group) and those for whom the physician did not follow recommendations (override group). The first comparison included all eligible patients: Patients in the adherence group had a Wells score of 4 or less with elevated d-dimer values or a Wells score greater than 4; those in the override group had a Wells score of 4 or less with normal d-dimer values or absent d-dimer testing. The second comparison of yield included only the subset of patients unlikely to have PE: Those in the adherence group had a Wells score of 4 or less with elevated d-dimer values, and those in the override group had a Wells score of 4 or less with normal d-dimer values or absent d-dimer testing. Furthermore, we performed analyses that initially considered subsegmental and/or indeterminate PEs that were not submitted to further anticoagulation treatment as studies with negative results; in an additional analysis, these were considered to be studies with positive results.

In a secondary analysis, we investigated whether malignancy, a common risk factor among those undergoing CT pulmonary angiography (18), could result in a higher yield in the override group, thus potentially justifying overriding of CDS. In this analysis, we assessed the yield of CT pulmonary angiography among patients in the override group who had undergone treatment for malignancy within 6 months. We treated studies in the same patients as independent studies because the average time between each set of repeat CT pulmonary angiographic examinations was 183 days. Repeated studies were not categorized into adherent and override groups because the same patient may fall into different groups.

The overall yield of PE in our institution’s ED after implementation of CDS was 10.4% (18). To detect a 7–percentage point absolute difference in the yield of PE between the two groups (assuming 12% in the adherent group and 5% in the override group), a sample size of 248 CT pulmonary angiographic studies in each group was needed to provide 80% power (α = .05, two-tailed test) for rejecting the null hypothesis. We used the χ² and Student t tests to analyze the differences between the two groups. Additionally, we performed regression analysis to account for the differences in risk factors between the two groups that could affect the probability of developing PE. All analyses were conducted by using software (JMP Pro 12; SAS Institute, Cary, NC).

Results

Patient Characteristics

During the study period, a total of 2993 CT pulmonary angiographic examinations were performed for suspected acute PE in the ED (Table 2). The 2993 examinations were ordered for 2655 unique patients. Comparison of the two groups showed no statistically significant difference with regard to sex, with both groups having more women than men (P = .2). However, the override group was, on average, 3.1 years older than the adherent group (P < .0001). The two groups also had different risk factors for PE: Compared with the adherent group, the override group included more patients with malignancy undergoing treatment (37.95% vs 56.5%, respectively; P < .0001), fewer patients with immobilization or recent surgery (31.5% vs 17.0%; P < .0001), and fewer patients with a previous diagnosis of PE or DVT (26.8% vs 10.1%; P < .0001). Among 338 repeat examinations, the mean time between each set was 183 days (range, 1–919 days; standard deviation, 180 days). Of 260 patients who underwent repeat examination, 203 had one repeat examination, 42 had two repeat examinations, and 15 had more than two repeat examinations.

Table 2.

Patient Characteristics during Study Period

Note.—CTPA = CT pulmonary angiography.

*Except where indicated, data are number of examinations, with percentages in parentheses.

^†Data are means ± standard deviations.

Yield of PE: Total CT Pulmonary Angiographic Examinations

Of the 2993 CT pulmonary angiographic examinations, 589 (19.7%) did not adhere to the evidence-based CDS. Among these examinations, 563 studies were from patients with a Wells score of 4 or less who did not undergo d-dimer testing and 26 were from patients with a Wells score of 4 or less and normal d-dimer values. For the 50 randomly selected studies with a Wells score of 4 or less and no d-dimer tests, no d-dimer testing was performed within 3 months after CT pulmonary angiography. Of 589 nonadherent orders, the natural language processing tool identified 30 studies positive for acute PE and chart review found five studies showing subsegmental and/or indeterminate results; patients in the latter group received no further anticoagulation treatment, resulting in a yield of 4.2% (25 of 589 examinations). For the remaining 2404 CT pulmonary angiographic examinations performed in adherence to CDS recommendations, the natural language processing tool identified 280 studies showing acute PE and chart review found 10 showing subsegmental and/or indeterminate results, for which patients received no further anticoagulation treatment; this led to a yield of 11.2% (270 of 2404 examinations) (Table 3).

Table 3.

Yield of CT Pulmonary Angiographic Examinations according to Adherence to CDS Recommendations

Note.—NA = not applicable, NLP = natural language processing, WS = Wells score.

*Subsegmental and/or indeterminate PEs without further anticoagulation treatment were counted as negative in this analysis.

The number of subsegmental and/or indeterminate studies for which patients received anticoagulation treatment was eight in the nonadherent group and 24 in the adherent group. Further analysis of the override group stratified according to whether patients were undergoing recent cancer treatment showed CT pulmonary angiographic yields of 4.78% (16 of 335 patients) in the group receiving cancer treatment and 3.52% (nine of 256 patients) in the group that did not receive cancer treatment (P = .54) (Table 4).

Table 4.

Override Group: Yield of CT Pulmonary Angiographic Orders in Patients with and Patients without Treatment for Malignancy within 6 Months

Results from the logistic regression analysis that took into account the differences in risk factors seen in Table 2 are shown in Table 5. The odds of an acute PE finding when CDS alerts were overridden were 51.3% lower than those seen with adherence to CDS (P < .001). The additional comparison that included patients with subsegmental and/or indeterminate results who received no further anticoagulation treatment resulted in a yield of 5.1% in the nonadherent group and 11.6% in the adherent group (P < .01 after logistic regression analysis). Reasons for no further treatment among the 15 subsegmental and/or indeterminate studies included (as noted in the electronic health record) “possible motion artifacts of CT pulmonary angiography” (four of 15 patients), “PE is unlikely the cause of symptoms” (four of 15 patients), “anticoagulation will not be therapeutic for chronic PE” (three of 15 patients), “risk of bleeding outweighs benefit” (two of 15 patients), and “patient refuses treatment” (two of 15 patients). The clinical characteristics of the 25 patients with positive PE diagnoses in the override group are summarized in Table 6.

Table 5.

Logistic Regression Analysis Controlling for Differences in Risk Factors for PE between Adherent and Override Groups

Note.—Age is a continuous parameter; sex = 1 for female, 0 for male; other parameters are 1 if positive, 0 if negative. Patients in the override group have override = 1, and those in adherent group have override = 0.

*Numbers in parentheses are the 95% confidence interval.

^†Statistically significant.

Table 6.

Characteristics of Patients with Positive PE Diagnoses in Override Group

Note.—CTPA = CT pulmonary angiography.

Yield of PE in Patients Unlikely to have PE

When we considered only patients unlikely to have PE, 476 studies were adherent to CDS compared with 587 that were nonadherent. When subsegmental and/or indeterminate studies for which patients did not receive anticoagulation treatment were considered to have negative results, the yield of studies performed in patients unlikely to have PE with adherence to CDS was 8.2%, compared with 4.2% when the physician overrode CDS (1.95-fold difference, P < .01). When subsegmental and/or indeterminate studies for which patients did not receive anticoagulation treatment were considered to have positive results, the yield of adherent studies performed in patients unlikely to have PE was 8.8%, compared with 5.1% in override studies (1.73-fold difference, P < .05) (Table 3).

Discussion

To improve quality of care and reduce waste, the Affordable Care Act encourages the adoption of electronic health records as well as computerized physician order entry systems with integrated CDS (14,25). Additionally, under the Protecting Access to Medicare Act of 2014, health care providers will be required to consult specified appropriate use criteria by using a qualified decision support mechanism when ordering advanced imaging for Medicare patients beginning in January 2017 (26). The current design of most CDS systems focuses only on real-time education, enabling physicians to easily ignore CDS alerts and allowing for autonomy in decision making and individual clinical judgment.

We found that the odds of an acute PE finding when providers adhered to well-accepted, validated decision rules embedded in CDS were nearly twice those seen when providers overrode CDS alerts. Most overrides occurred with providers not performing d-dimer testing in patients unlikely to have PE. Although more than half of these patients were undergoing treatment for known malignancy (a known cause of elevated d-dimer levels, even in the absence of acute PE) (27), it is possible that obtaining a normal d-dimer value in a subset of this PE unlikely group would have obviated the need for CT pulmonary angiography.

The overall yield of CT pulmonary angiography in our study was 9.9% (10.3% if subsegmental and/or indeterminate studies for which patients received no further anticoagulation treatment were considered to have positive results), which falls within the range of the reported yield of CT pulmonary angiography in the ED (5.7%–15%) (8,28,29) and is similar to the 10.4% overall yield previously reported in our ED (18).

Our study, which was performed from 2011 to 2013, used CDS based on the two-tier Wells criteria supported by American College of Emergency Physicians’ 2011 clinical policy on PE, wherein a patient unlikely to have PE should undergo d-dimer testing and a negative d-dimer test result (<500 ng/mL) can be safely used to exclude PE (10). Therefore, CT pulmonary angiography for these patients who did not undergo d-dimer testing or those with negative d-dimer results may be considered inappropriate. One study found that 32% of all CT pulmonary angiographic examinations are potentially inappropriate (12); however, that study did not investigate the CT pulmonary angiographic yield.

In our study, 19.7% of imaging studies (589 of 2993 examinations) were performed in patients unlikely to have PE with normal (26 of 2993 examinations [0.9%]) or absent (563 of 2993 examinations [18.8%]) d-dimer results. Our lower rate of potentially avoidable CT pulmonary angiography is likely due to long-standing CDS-enabled interventions at our institution under strong leadership. Although our experience supports the role of CDS in reducing inappropriate imaging, a roughly 20% rate of potentially avoidable CT pulmonary angiography implies opportunities for further performance improvement. The 2015 American College of Emergency Physicians’ guideline, which states that patients with a low pretest probability of PE (Wells score <2) should not undergo d-dimer testing (30), does not affect our results because our ED’s CDS was based on the two-tier system.

Our results support evidence showing overuse of CT pulmonary angiography and underuse of d-dimer testing. We found that the odds of an acute PE finding when providers followed CDS evidence were nearly double the odds when they overrode CDS recommendations. Analysis that included only patients unlikely to have PE, which yielded similar results, corroborated these findings. Admittedly, the Wells criteria are not effective for all patients suspected of having PE. In our study, patients who had examinations positive for PE (4.2%) in the override group often included those with nonspecific symptoms, such as exertional chest pain. Hence, imaging was necessary to diagnose PE in these patients. However, a portion of CT pulmonary angiographic examinations among the rest of the override group may have been unnecessary with previous d-dimer testing. A recent study showed that 28% of patients unlikely to have PE with a Wells score of 4 or less had a d-dimer level of less than 500 ng/mL, with a less than 0.5% chance of positivity for PE (31). In addition, patients unlikely to have PE with positive PE diagnoses would likely have positive d-dimer results because those with normal d-dimer results are very unlikely to have PE (10,31,32); none of these patients in our study had PE. Furthermore, our analysis did not reveal a statistically significant difference in the yield of CT pulmonary angiography in the override group for patients with or patients without recent cancer treatment. Therefore, a stronger enforcement of d-dimer testing before CT pulmonary angiography may safely obviate CT pulmonary angiography for an as-yet-unknown subset of patients unlikely to have PE (31,32).

Our findings are consistent with those of earlier studies that reported that CDS interventions that allow simple override are unlikely to optimize provider behavior during the imaging ordering process (33). Stronger interventions, such as requiring clinical justification for overriding CDS alerts (34), implementing physician performance feedback reports (35), or requiring real-time peer-to-peer consultation for providers who ignore evidence presented in CDS alerts (36), have increased adherence to evidence presented in CDS. Mandating d-dimer testing before CT pulmonary angiography for patients unlikely to have PE may be another potential strategy to further reduce the unnecessary use of CT pulmonary angiography.

This study had several limitations. First, it was performed at a single institution with more than 10 years of experience with imaging CDS; therefore, the results may not be generalizable to other institutions. Second, according to the Wells criteria, a d-dimer test should not be ordered in patients with a Wells score greater than 4 because CT pulmonary angiography is recommended. In our study, patients with a Wells score greater than 4 whose provider did not attempt to order CT pulmonary angiography were not captured by CDS because the Wells score is recorded only during computerized physician order entry. Similarly, physicians cannot be alerted when they opt for d-dimer testing only for patients with a Wells score greater than 4. However, we believe this behavior of underimaging is uncommon in our ED and are not aware of any reports of underimaging in the literature in patients suspected of having acute PE. Therefore, although a few instances may have been missed, the number was likely to be small and would probably not have affected our results. Moreover, the override group included CT pulmonary angiographic examinations from patients who had a Wells score of 4 or less with no d-dimer testing or normal d-dimer results, whereas the adherent group consisted of patients with a Wells score greater than 4 or a Wells score of 4 or less and elevated d-dimer levels. Therefore, patients in the adherent group had an inherently higher clinical risk of PE. Because a comparison that included only patients unlikely to have PE led to results similar to those in our primary analysis, we conclude that significant differences are seen even when other factors are considered. In addition, when subsegmental and/or indeterminate studies are categorized as negative, there may be bias because patients with lower Wells scores may be less likely to receive anticoagulation treatment. Our additional analysis that counted subsegmental and/or indeterminate studies as positive ensured that our results are robust. Last, our cohort included patients who underwent repeat CT pulmonary angiographic studies. We considered each CT pulmonary angiographic examination as an independent study because the mean time between each set of repeat CT pulmonary angiographic studies was 183 days. This could have affected the PE yield in both the adherent and override groups. However, considering the small proportion of repeated studies and the long mean interval between them, we do not believe the inclusion of repeat studies changed our conclusion.

In conclusion, the odds of CT pulmonary angiography showing positive results for acute PE nearly doubled when providers followed CDS based on Wells criteria compared with when providers overrode such CDS alerts. Our findings suggest that stronger interventions may be needed to further reduce the nearly 20% of CT pulmonary angiography requests that override education-only CDS alerts, primarily by ignoring d-dimer testing in patients unlikely to have PE. Further studies are needed to determine what subset of this patient population would benefit from d-dimer testing, which, if providing normal results, would obviate CT pulmonary angiography. Education-only CDS interventions, even if based on validated decision rules, are unlikely to optimize evidence-based decision making if providers can simply ignore CDS alerts and proceed with imaging.

Advances in Knowledge

• ■ Providers overrode 19.7% (589 of 2993 studies) of clinical decision support (CDS) alerts for emergency department CT pulmonary angiography orders from January 1, 2011, to August 31, 2013.

• ■ Most overrides (95.6% [563 of 589 studies]) were due to a lack of d-dimer testing in patients unlikely to have pulmonary embolism (PE).

• ■ The odds of a CT pulmonary angiographic result positive for acute PE (number of positive PE diagnoses/total number of CT pulmonary angiographic studies) were nearly doubled when providers followed decision support recommendations based on Wells criteria compared with the odds when providers overrode such CDS recommendations (P < .001).

Implications for Patient Care

• ■ Requiring d-dimer testing before proceeding to CT pulmonary angiography in patients unlikely to have PE may further reduce unnecessary imaging.

• ■ Education-only CDS alerts, even when based on validated, well-accepted decision rules, are unlikely to optimize practice; stronger CDS-enabled interventions may be needed to improve the return on the substantial national investments in health information technology.