Effect of Computerized Clinical Decision Support on the Use and Yield of CT Pulmonary Angiography in the Emergency Department

The implementation of evidence-based clinical decision support in the emergency department was associated with a significant (20.1%) decrease in the use, and a significant (69.0%) increase in the yield, of CT pulmonary angiography for the evaluation of acute pulmonary embolism during a 2-year period.

Abstract

Purpose:

To determine the effect of evidence-based clinical decision support (CDS) on the use and yield of computed tomographic (CT) pulmonary angiography for acute pulmonary embolism (PE) in the emergency department (ED).

Materials and Methods:

Institutional review board approval was obtained for this HIPAA-compliant study, which was performed between October 1, 2003, and September 30, 2009, at a 793-bed quaternary care institution with 60 000 annual ED visits. Use (number of examinations per 1000 ED visits) and yield (percentage of examinations positive for acute PE) of CT pulmonary angiography were compared before and after CDS implementation in August 2007. The authors included all adult patients presenting to the ED and developed and validated a natural language processing tool to identify acute PE diagnoses. Linear trend analysis was used to assess for variation in CT pulmonary angiography use. Logistic regression was used to determine variation in yield after controlling for patient demographic and clinical characteristics.

Results:

Of 338 230 patients presenting to the ED, 6838 (2.0%) underwent CT pulmonary angiography. Quarterly CT pulmonary angiography use increased 82.1% before CDS implementation, from 14.5 to 26.4 examinations per 1000 patients (P < .0001) between October 10, 2003, and July 31, 2007. After CDS implementation, quarterly use decreased 20.1%, from 26.4 to 21.1 examinations per 1000 patients between August 1, 2007, and September 30, 2009 (P = .0379). Overall, 686 (10.0%) of the CT pulmonary angiographic examinations performed during the 6-year period were positive for PE; subsequent to CDS implementation, yield by quarter increased 69.0%, from 5.8% to 9.8% (P = .0323).

Conclusion:

Implementation of evidence-based CDS in the ED was associated with a significant decrease in use, and increase in yield, of CT pulmonary angiography for the evaluation of acute PE.

© RSNA, 2011

Introduction

As part of the Health Information Technology for Economic and Clinical Health Act (1), the Department of Health and Human Services meaningful use criteria for the use of electronic health records require the implementation of at least one clinical decision support (CDS) tool (2). CDS has been shown to decrease the volume of imaging orders in certain clinical contexts (3). In addition, it has been proposed that implementation of CDS can increase the appropriateness of imaging (4).

Given that the use of emergency department (ED) computed tomographic (CT) pulmonary angiography for the diagnosis of pulmonary embolism (PE) is increasing (a recent study [5] found that the number of ED CT pulmonary angiograms of the chest at one large academic institution increased fivefold from 2001 to 2007) and that evidence-based guidelines for imaging patients suspected of having PE exist (6), it may be an excellent target for evaluating the impact of CDS. If suboptimal use of CT pulmonary angiography for the evaluation of PE in the ED exists, then integrating CDS for CT pulmonary angiography into a computerized physician order entry system has the potential to increase quality and appropriateness of ED imaging while decreasing potentially unnecessary examinations.

We performed this study to determine the impact of evidence-based CDS on the use and yield of CT pulmonary angiography for acute PE in the ED. We hypothesized that CDS, when integrated into computerized physician order entry workflow, would reduce use and increase yield.

Materials and Methods

Institutional review board approval was obtained for this Health Insurance Portability and Accountability Act–compliant study, which was performed between October 1, 2003, and September 30, 2009, at a 793-bed quaternary care institution with 60 000 annual ED visits.

Data Collection

Members of the study team (A.S.R. [an attending physician], I.K.I. [a research fellow], and C.F. [a research assistant]) reviewed the electronic medical record to determine which patients underwent CT pulmonary angiography during the study period. For each of these patients, we recorded seven variables: age, sex, date of study, history of neoplasm, history of thromboembolism, history of recent surgery, and d-dimer level. These characteristics were used to compare the pre- and postintervention populations. Age, sex, and date of study were obtained directly from the electronic medical record. Clinical data regarding a patient’s history of neoplasm and thromboembolism were obtained from the hospital billing system by using International Classification of Diseases, ninth revision, codes 140–239 (neoplasms); 451.0–451.9, 452, 453.0–453.9, and 557.0 (venous thrombosis); and 415.1–415.9 (PE). Diagnoses billed within the previous 12 months of an imaging examination were considered to be current. We defined recent surgical history as any procedures performed within the 4 weeks preceding the imaging examination, and these were determined with surgical reports in the electronic medical record. Exact d-dimer values were obtained by accessing the clinical laboratory database.

To capture whether a radiology report contained positive or negative findings for PE, we developed a natural language processing (NLP) algorithm using General Architecture for Text Engineering (University of Sheffield, South Yorkshire, England). The use of General Architecture for Text Engineering in the biomedical arena has been studied previously, with promising results (7,8). To determine yield, a binary outcome variable for imaging outcome was created from each report. A study was classified as either positive or negative for PE, with indeterminate or limited examinations categorized as negative and both chronic and acute PE as positive because a lack of diagnostic certainty often exists regarding chronicity. To validate the NLP algorithm, we randomly selected 179 de-identified reports from the full patient cohort. Each report was then manually categorized as positive or negative for PE by a physician-trained research assistant, and the findings from the NLP algorithm were then compared with this manual standard.

Intervention

We integrated CDS on the basis of validated decision rules (9) into our ED radiology computerized physician order entry system (Percipio; Medicalis, San Francisco, Calif) during August 2007. The CDS consisted of three rules. The first rule required that ordering clinicians choose both a d-dimer level (elevated, normal, or not evaluated) and the clinical suspicion of PE (high, intermediate, or low). The second rule displayed advice in patients with an intermediate or low level of suspicion in whom a d-dimer assay was not performed (“measuring a d-dimer value in patients with a low and/or intermediate clinical suspicion of PE is an appropriate first step in the work-up of acute PE and will exclude the need for CT pulmonary angiography in some patients”). The third rule displayed a second piece of advice in patients with a normal d-dimer level and intermediate or low suspicion for PE (“based on current evidence as well as our experience at Brigham and Women’s Hospital, diagnosing an acute PE with CT pulmonary angiography in low- or intermediate-risk patients with a normal d-dimer level is extremely unlikely”) (Fig 1). At each stage, clinicians could either cancel the imaging order or ignore the advice (Fig 2).

Figure 1:

d-Dimer level and clinical suspicion entry tools and decision support statements 1 and 2. CTPA = CT pulmonary angiography.

Figure 2:

Flowchart of decision support process. CTPA = CT pulmonary angiography.

In addition to the CDS itself, our intervention also included multidisciplinary discussions of the evidence basis for the CDS between emergency physicians and radiologists at two emergency medicine faculty meetings, as well as one emergency medicine residency grand rounds, over the course of the intervention.

Statistical Analysis

We descriptively analyzed the patient demographics. To validate the NLP PE detection algorithm, the total numbers of true-positive, true-negative, false-positive, and false-negative findings were determined and the accuracy, sensitivity, and specificity calculated by using a representative sample. Linear trend analysis according to quarter was used to assess for variations in the use of CT pulmonary angiography (number of examinations per 1000 patients), and logistic regression was used to determine variations in yield subsequent to implementation of CDS after controlling for patient demographic and clinical characteristics. P < .05 was considered indicative of a statistically significant difference, and all analyses were conducted by using JMP 9.0 (SAS Institute, Cary, NC).

Results

Use of CT Pulmonary Angiography

A total of 338 230 patients were seen in our ED between October 1, 2003, and September 30, 2009. Of these, 6838 patients (2.0%) underwent CT pulmonary angiography and were included in this study, for an average of 1140 CT pulmonary angiographic examinations per year. The characteristics of patients before and after CDS implementation were similar in all respects except that the postimplementation population was more likely to have a history of thromboembolism and less likely to have a history of recent surgery or d-dimer testing (Table 1).

Table 1.

Characteristics of Patients before and after Implementation of CDS

Note.—Except where indicated, data are numbers of patients, with percentages in parentheses.

^*Data were acquired between October 1, 2003, and July 31, 2007.

^†Data were acquired between August 1, 2007, and September 30, 2009.

^‡Numbers in parentheses are standard deviations.

The use of CT pulmonary angiography in the period before CDS implementation (from October 1, 2003, to July 31, 2007) increased 82.1% overall, from 14.5 to 26.4 examinations per 1000 patients (P < .0001). After implementation of the CDS, quarterly use decreased 20.1% overall, from 26.4 to 21.1 examinations per 1000 patients between August 1, 2007, and September 30, 2009 (P = .0379) (Fig 3).

Figure 3:

Graph shows CT pulmonary angiography (CTPA) use and yield before and after CDS implementation. CY = calendar year, Q1 = first quarter, Q2 = second quarter, Q3 = third quarter, Q4 = fourth quarter.

NLP Algorithm Validation

Of the 179 randomly selected reports chosen for validation of the NLP algorithm, 23 (12.8%) were positive for PE on the basis of manual detection. The algorithm’s detection of the presence or absence of PE had a sensitivity of 91.3% (95% confidence interval: 72.0%, 98.8%), a specificity of 98.7% (95% confidence interval: 95.2%, 99.9%), a positive predictive value of 91.3% (95% confidence interval: 72.0%, 98.8%), a negative predictive value of 98.7% (95% confidence interval: 95.2%, 99.9%), and an accuracy of 97.8% (95% confidence interval: 94.2%, 99.1%).

Yield of CT Pulmonary Angiography

Of the 6838 CT pulmonary angiographic examinations performed during the study period, 686 (10.0%) were positive for PE, as determined by the NLP application. Subsequent to the implementation of CDS, quarterly yield increased 69.0%, from 5.8% to 9.8% between July 31, 2007, and September 30, 2009 (P = .0323) (Table 2).

Table 2.

Quarterly Yield of CT Pulmonary Angiography

^*The yield of CT pulmonary angiography for all quarters was 10.0%.

Discussion

The implementation of evidence-based CDS was associated with a significant (20.1%) decrease in the use, as well as a significant (69.0%) increase in the yield, of CT pulmonary angiography for the evaluation of acute PE in the ED during a 2-year period.

The use of CT pulmonary angiography in our ED increased 82.1% from 2003 to 2007—the 4 years before implementation of CDS. Although this is a substantial increase, it is less than the increase noted in a recently published study at another large academic medical center (5), which found that overall use of chest CT increased fivefold from approximately 10 CT examinations per 1000 patients in 2003 to 60 examinations per 1000 patients in 2007. The less dramatic increase at our institution may have been due to a number of institution-specific factors, including the presence of fellowship-trained emergency radiologists in the ED around the clock—a factor that may have served to curb the otherwise unbridled increase in CT seen at other sites.

After implementation of CDS, the use of CT pulmonary angiography decreased over the subsequent eight quarters. We are unaware of any previous work in the ED with which to compare this decrease over time, but a similar study performed during a 7-year period in the outpatient setting has demonstrated the decreased use of imaging after implementation of CDS (3). In our study, however, the CDS focused on a specific clinical indication and specific diagnostic modality, which we believe better allows for integration and acceptance of targeted evidence-based guidelines. The continued reduction in use and increase in yield of CT pulmonary angiography in the ED 2 years after implementation of CDS is likely due to many factors. Although we had leadership support and multidisciplinary discussions in faculty meetings about the importance of using high-quality evidence-based guidelines for the evaluation of PE in the ED, we did not use more stringent processes shown to influence test-ordering behavior (eg, academic detailing or practice pattern variation reporting and/or benchmarking) (10,11). As such, our intervention may simply reflect the time it takes to bring meaningful change to our day-to-day practice (12).

One recent 4-month study of CDS in the ED failed to find a decrease in the use of CT pulmonary angiography after the implementation of CDS (13); a review of 439 CT pulmonary angiographic examinations resulted in the conclusion that the use of CT pulmonary angiography did not decrease significantly (from 14 to 12.8 examinations per 1000 patients) with the implementation of CDS. In addition, CDS had to be removed from their ED at the conclusion of the study owing to nonacceptance by the emergency physicians. Our findings may have differed from theirs because of our methods of integration. Our roll-out strategy involved targeted general multidisciplinary discussions at faculty meetings, emergency physician champions, and an education campaign around the evidence basis for our CDS strategy across our institution that, if reinforced over time, may account for the continued reduction in use and increase in yield of CT pulmonary angiography seen 2 years after implementation of CDS. We believe that all of these factors increased the acceptance of CDS implementation because a multifactorial approach to culture change has proved to be effective in other settings (14,15).

In addition to a decrease in use, there was a concomitant 69% increase in yield from 5.8% before CDS to 9.8% at the end of our study. This confirms the findings of a recent 2-year study (16), which included 261 patients (111 from the ED) and found that the yield of CT pulmonary angiography increased from 3.1% before the implementation of CDS to 16.5% after implementation. Our findings may differ slightly in magnitude owing to sample size or institutional variation. In addition, our yield of 9.8% is equal to the 9.8% yield found in another recent study that evaluated the utility of risk stratification in patients who underwent CT for suspected PE (17).

As evidence of the comparability of our study populations before and after implementation of CDS, the patients’ ages, sex proportion, and history of malignancy were similar, as noted in Table 1. However, whereas patient demographics stayed the same, the clinical factors of the patients who underwent CT pulmonary angiography before and after CDS varied. Patients undergoing CT pulmonary angiography after CDS implementation were more likely than those studied before implementation to have a history of thromboembolism. This is likely due to the implementation of a required pretest probability determination. Anecdotally, the most commonly used model for determining pretest probability at our institution is the Wells criteria, which rely on the knowledge of a history of thromboembolism (9). This requirement likely led to more patients being queried regarding a possible history of thromboembolism, which may have led to their being assigned a high-risk status and undergoing CT pulmonary angiography. This practice of risk stratification is necessary, and there is potential for inappropriate imaging utilization if it is not made routine (18).

Patients imaged after CDS implementation were also less likely to have recently undergone surgery. During our study period, the use of postoperative thromboprophylaxis increased significantly (19–21). Perhaps the number of postoperative patients who presented to our ED with symptoms suggestive of PE decreased because of the increased use of thromboprophylaxis, although these data are not readily available.

Patients who underwent CT pulmonary angiography after the implementation of CDS were also less likely to undergo d-dimer testing (57.2% before implementation vs 46.7% after implementation, P < .0001). This finding supports the results of the previously mentioned study by Drescher et al (13), who found that the use of d-dimer testing decreased after CDS implementation. It may be that the requirement for risk stratification mandated by our CDS resulted in a greater proportion of patients being appropriately deemed high risk and, correctly, the avoidance of d-dimer testing.

There were a number of limitations to our study. First, we did not assess the appropriateness of performing CT pulmonary angiography and were only able to determine its use and yield. Second, our results may not be generalizable; our institution is a quaternary care hospital and a large cancer center, our patients’ risk for and incidence of thromboembolic disease is likely higher than that of other hospitals, and our departments of radiology and emergency medicine have a strong and longstanding collaborative relationship. We did not have a concurrent control group, so external factors may have influenced the use of CT pulmonary angiography—especially the increase in attention in both the lay press and the medical literature regarding the risks of medical radiation. In addition, many variables may impact the effective adoption of decision support, including the health care institution’s leadership and culture of quality (4,22). It is possible that such a culture of quality reinforced through infrequent faculty meetings at our institution contributed to the impact of CDS in our study. Nonetheless, without the accountability afforded by the use of decision support, our results may not have been realized. Last, we cannot definitively state that there was no harm to patients with missed PEs because of our implementation of CDS. A review of our intradepartmental quality improvement records did not yield any cases of missed PE after CDS implementation; however, patients may have presented outside of our hospital system or may have yet to file a malpractice claim.

Despite these limitations, we believe that our data support the conclusion that the implementation of evidence-based CDS was associated with a significant decrease in the use, and a significant increase in the yield, of CT pulmonary angiography for acute PE in the ED and that such change likely takes many months to be optimized. Further studies to assess the impact of CDS on appropriateness of imaging, especially with regard to adherence to evidence-based guidelines for specific clinical populations or disease processes, may be useful. In addition, studies to assess the impact of more stringent processes (eg, academic detailing or practice pattern variation reporting) coupled with CDS to optimize the adoption of high-quality evidence-based guidelines in day-to-day practice may be helpful to devise optimum strategies for improving quality and reducing the inappropriate use of imaging (23).

Advances in Knowledge.

• Implementation of evidence-based computerized clinical decision support in the emergency department was associated with a 20.1% decrease (P = .0379) in the use of CT pulmonary angiography.

• Diagnostic yield of CT pulmonary angiography for pulmonary embolism increased 69.0% (P = .0323) in the emergency department after clinical decision support was implemented.

Implication for Patient Care.

• Computerized decision support may help decrease the number of unnecessary CT pulmonary angiographic examinations performed to evaluate for pulmonary embolism in the emergency department.

Disclosures of Potential Conflicts of Interest: A.S.R. Financial activities related to the present article: none to disclose. Financial activities not related to the present article: is employed by Diagnotion. Other relationships: none to disclose. I.K.I. No potential conflicts of interest to disclose. L.M.P. No potential conflicts of interest to disclose. A.D.S. No potential conflicts of interest to disclose. C.F. No potential conflicts of interest to disclose. R.D.Z. No potential conflicts of interest to disclose. R.H. No potential conflicts of interest to disclose. S.Z.G. Financial activities related to the present article: none to disclose. Financial activities not related to the present article: receives money for consultancy from Boehringer Ingelheim, Bristol Myers Squibb, Daiichi Sankyo, Eisai, EKOS, Medscape, Merck, Portola, and Sanofi Aventis; institution received a grant or grants are pending from Boehringer Ingelheim, Bristol Myers Squibb, Eisai, EKOS, Johnson & Johnson, and Sanofi Aventis. Other relationships: none to disclose. R.R.G. No potential conflicts of interest to disclose. R.K. Financial activities related to the present article: none to disclose. Financial activities not related to the present article: receives royalties from Medicalis; institution receives royalties from Medicalis; has stock or stock options from Medicalis; institution has stock or stock options from Medicalis. Other relationships: none to disclose.