Effect of Evidence-based Clinical Decision Support on the Use and Yield of CT Pulmonary Angiographic Imaging in Hospitalized Patients

Implementation of clinical decision support (CDS) on the basis of high-quality evidence was associated with a significant decrease in the use of CT pulmonary angiographic imaging for acute pulmonary embolism in hospitalized patients that occurred within the first 30 days after implementation of CDS.

Abstract

Purpose

To determine the effect of clinical decision support (CDS) on the use and yield of inpatient computed tomographic (CT) pulmonary angiography for acute pulmonary embolism (PE).

Materials and Methods

This HIPAA-compliant, institutional review board-approved study with waiver of informed consent included all adults admitted to a 793-bed teaching hospital from April 1, 2007, to June 30, 2012. The CDS intervention, implemented after a baseline observation period, informed providers who placed an order for CT pulmonary angiographic imaging about the pretest probability of the study based on a validated decision rule. Use of CT pulmonary angiographic and admission data from administrative databases was obtained for this study. By using a validated natural language processing algorithm on radiology reports, each CT pulmonary angiographic examination was classified as positive or negative for acute PE. Primary outcome measure was monthly use of CT pulmonary angiography per 1000 admissions. Secondary outcome was CT pulmonary angiography yield (percentage of CT pulmonary angiographic examinations that were positive for acute PE). Linear trend analysis was used to assess for effect and trend differences in use and yield of CT pulmonary angiographic imaging before and after CDS.

Results

In 272 374 admissions over the study period, 5287 patients underwent 5892 CT pulmonary angiographic examinations. A 12.3% decrease in monthly use of CT pulmonary angiography (26.0 to 22.8 CT pulmonary angiographic examinations per 1000 admissions before and after CDS, respectively; P = .008) observed 1 month after CDS implementation was sustained over the ensuing 32-month period. There was a nonsignificant 16.3% increase in monthly yield of CT pulmonary angiography or percentage of CT pulmonary angiographic examinations positive for acute PE after CDS (P = .65).

Conclusion

Implementation of evidence-based CDS for inpatients was associated with a 12.3% immediate and sustained decrease in use of CT pulmonary angiographic examinations in the evaluation of inpatients for acute PE.

^© RSNA, 2015

Online supplemental material is available for this article.

Introduction

Despite substantial benefits of medical imaging, concerns persist about inappropriate use that may result in waste, suboptimal quality of care, and, in the case of computed tomographic (CT) imaging, potentially harmful ionizing radiation (1,2). The Health Information Technology and Economic Health (HITECH) Act of 2009 aimed to encourage broad adoption of health information technology tools to improve quality and reduce waste, in part through the use of clinical decision support (CDS) that presents evidence in real time to providers at the point of care to improve clinical decision making. CDS is a fundamental component in stage 2 of the meaningful use criteria in the HITECH Act regulations (3,4).

From when it was initially proposed and studied (5,6), large-scale deployment of imaging CDS was reported (7). Imaging CDS and CDS-enabled interventions have been associated with reductions in use of imaging and improved appropriateness (8–13). Although condition-specific imaging CDS interventions for outpatients (9) and in the emergency department (ED) (10) have been reported, reports on inpatient interventions are lacking.

CT pulmonary angiographic imaging is the examination of choice to diagnose acute pulmonary embolism (PE) (14), and its use for this purpose is increasing in both ED and inpatient settings (10,15,16). Increase in use of CT pulmonary angiographic imaging has been associated with increased radiation exposure (17) and overdiagnosis of less clinically important PE (18). Evidence-based guidelines for imaging patients suspected of having PE have been derived and validated (14,19–20). Appropriate use of CT pulmonary angiography is also the focus of national initiatives, including the Choosing Wisely campaign, which is endorsed by a number of medical specialty societies (21). The National Quality Forum endorsed an imaging efficiency quality metric directed at CT pulmonary angiography appropriateness in ED patients with low pretest probability of PE (22). It was postulated (23) that compliance with this quality measure could reduce so-called avoidable imaging in low-risk ED patients by 32% by implementing evidence-based guidelines because CDS is associated with decreased use and increased yield (defined as the percentage of CT pulmonary angiographic examinations positive for acute PE) in the ED (10).

The purpose of our study was to determine the effect of CDS on use and yield of inpatient CT pulmonary angiographic imaging for acute PE.

Materials and Methods

Study Setting and Patients

This Health Insurance Portability and Accountability Act–compliant study was approved by the institutional review board with a waiver of informed consent. The study population included patients who were hospitalized between April 1, 2007, and June 30, 2012, at a 793-bed urban teaching hospital with over 50 000 annual admissions. We included all patients older than 16 years and we excluded those in hospice care.

Data Control and Collection

R.K. is a consultant to Medicalis Corporation. Control of data and information was solely the responsibility of authors who are not consultants to industry whose product was evaluated in our study (Percipio; Medicalis, San Francisco, Calif).

Monthly admission data were gathered from an administrative claims database. We reviewed our institutional radiology information system to determine which inpatients underwent CT pulmonary angiographic imaging during the study period and excluded studies ordered in the ED before admission. We recorded the following patient-related variables: age, sex, diagnosis of neoplasm within 1 year before CT examination, previous history of thromboembolism, and history of surgery within 4 weeks of the CT pulmonary angiographic examination. Age, sex, surgical history, and d-dimer status and level were obtained from the electronic medical record. Data regarding a patient’s history of neoplasm and thromboembolism were obtained from the hospital billing system by using the International Classification of Diseases, ninth revision, codes 140–239 (neoplasms); 451.0–451.9, 452, 453.0–453.9, and 557.0 (venous thrombosis); and 415.1–415.9 (PE).

For each completed CT pulmonary angiographic examination, we used a natural language processing algorithm based on General Architecture for Text Engineering (University of Sheffield, South Yorkshire, England) to classify the examination as either positive or negative for acute PE based on binary outcome analysis of the radiology reports. The natural language algorithm was previously validated and published for CT pulmonary angiographic reports in the ED population with indeterminate or limited examinations categorized as negative (10).

Intervention

The CDS intervention informed providers of the patient-specific pretest probability for PE based on a previously validated decision rule (9) when they submitted an order for a CT pulmonary angiographic examination. These decision rules were incorporated into CDS delivered by using the institution’s computerized physician order entry system (Percipio; Medicalis, San Francisco, Calif). Details of this system were previously reported (7). The CDS had three stages (Figs 1, 2). The first stage required the physician who submitted the order for the examination to select both the clinical suspicion of PE (low, intermediate, or high) and a d-dimer level. The second stage displayed a statement for patients with an intermediate or low level of clinical suspicion of PE in whom a d-dimer assay was not performed (Fig 1). The third stage displayed a statement for patients with a normal d-dimer level and intermediate or low clinical suspicion for PE (Fig 1). At each stage, the ordering physician had the option to cancel the imaging order or to ignore the advice. If the advice was ignored, no further action was required before placement of the order.

Figure 1:

Screen capture of d-dimer level and clinical suspicion data entry tools and CDS statements. CDS statements are shown. CTPA = CT pulmonary angiography.

Figure 2:

Flowchart shows the CDS process. Decision support statement 1: Measuring a d-dimer in patients with a low/intermediate clinical suspicion of pulmonary embolism is an appropriate first step in the workup of acute PE and will exclude the need for CTPA in some patients. Decision support statement 2: Based on current evidence as well as our experience at Brigham and Women’s Hospital, diagnosing an acute pulmonary embolism by CT pulmonary angiography in low- or intermediate-risk patients with a normal d-dimer is extremely unlikely. CTPA = CT pulmonary angiography.

Study Design and Outcomes

By using a before-and-after design, we compared outcomes before and after implementation of the CDS intervention on November 1, 2009. The primary outcome measure was monthly use intensity of inpatient CT pulmonary angiographic imaging, defined as the number of CT pulmonary angiographic examinations performed per month per 1000 inpatient admissions. The secondary outcome was the imaging yield. We also analyzed monthly CT pulmonary angiographic imaging use and yield before and after intervention, stratified by clinical specialty of the ordering providers. The clinical specialty of the ordering providers was categorized as follows: surgery, medicine, hematology and oncology, and other clinical specialty. Outcomes were analyzed on a monthly basis to allow evaluation of the immediate (within the first 30 days) effect of the CDS intervention. In addition, we performed a posthoc electronic medical record analysis of patients in our postintervention cohort with a normal d-dimer level (<500 ng/mL) whose subsequent CT pulmonary angiographic examination was positive for PE to determine pretest probability of PE based on previously validated evidence-based guidelines for imaging of patients suspected of having PE (19).

Statistical Analysis

A power calculation estimated the sample size required to detect a 5% effect size, or a relative difference of 5% in use of CT pulmonary angiography between the periods before and after implementation of CDS. Monthly use of CT pulmonary angiography was estimated to be 25.0 studies per 1000 inpatient admissions. A sample size of 675 patients in each period before and after CDS would provide 90% power (α = .05; two-tailed test) to reject the null hypothesis.

For the primary outcome, we performed linear trend analyses by using a multiple linear regression model by constructing two separate best-fit lines, one each for the preintervention and postintervention periods. This analysis allowed us to evaluate the immediate effect of CDS on use and yield of CT pulmonary angiography, and to compare trends in use and yield between the preimplementation and postimplementation periods. Our data set consisted of 63 sequentially numbered observations, and each represented a calendar month during the study period. For use of CT pulmonary angiography, the dependent or outcome variable was number of CT pulmonary angiographic examinations performed per month per 1000 inpatient admissions. For CT pulmonary angiography yield, the dependent variable was monthly percentage of positive CT pulmonary angiography studies. The independent or predictor variables included time, intervention, and an interaction term (intervention · time). We used the following model to perform our estimations: Y_i = α₁ + β₁T_i + β₂I_i + β₃O_i + ε_I, where Y_i is the estimated monthly CT pulmonary angiography use or yield, α₁ is intercept term (procedure volume or yield at month 0), T_i is month (n = 1–32)_, I_i is intervention (denoted as a binary variable: 0 or 1), O_i is interaction term (intervention · time) and ε_i is the error term. In terms of regression coefficients, β₁ represents the difference in predicted value of Y_i for each one unit difference in T_i; β₂ represents the difference in predicted value of Y_i for each one unit difference in I_i; and β₃ represents the difference in predicted value of Y_i for each one unit difference in O_i.

To determine the immediate effect of CDS, we calculated Y_i by using both the preintervention and postintervention best-fit lines before and after month 31 (when use of CDS began) of 63 months. To compare the trends in use, before and after intervention, we compared the slopes of the preintervention and postintervention best-fit lines before and after month 31.

Additional analysis was performed by using Student t test to compare continuous variables and χ² test was used to compare categorical variables between the preintervention and postintervention groups.

A two-tailed P value of less than .05 indicated statistical significance. All analyses were conducted by using statistical software (JMP 10.0; SAS institute, Cary, NC).

Results

Use of CT Pulmonary Angiography

In 227 374 eligible admissions during the complete study period, 5287 patients underwent 5892 CT pulmonary angiographic studies and an average of 1138 studies were performed per year. Patient-related characteristics before and after intervention were similar, other than a statistically significant difference in the history of previous venous thromboembolism (25% before CDS compared with 30% after CDS) (Table 1). d-dimer testing was not routinely performed before CT pulmonary angiography; 2604 of 3037 (85.7%) patients in the preintervention cohort and 2474 of 2825 (87.6%) patients in the postintervention cohort did not undergo d-dimer testing (Table 1). Among 73 patients with a normal d-dimer level (<500 ng/mL), subsequent CT pulmonary angiographic imaging showed PE in two (2.7%) patients in the postintervention cohort. When the medical records of the patients were reviewed, one patient had an intermediate pretest probability of PE based on previously validated evidence-based guidelines for imaging of patients suspected of having PE (19) and had an acute segmental PE based on CT pulmonary angiography report. The other patient underwent CT pulmonary angiographic imaging for evaluation of newly diagnosed pulmonary arterial hypertension. The CT pulmonary angiographic report was consistent with PE and likely chronic; however, subsequent invasive pulmonary angiographic imaging did not show PE.

Table 1.

Patient-related Characteristics per Study before and after Implementation of Evidence-based CDS

Note.—Data in parentheses are percentages except where otherwise indicated. There were 433 patients in whom d-dimer was obtained before CDS implementation and 351 patients in whom d-dimer was obtained after CDS implementation.

^*Data in parentheses are confidence intervals.

Mean monthly use of CT pulmonary angiographic imaging was 102.4 studies before intervention and 85.0 studies after intervention. When adjusted for monthly admissions, the mean monthly use of CT pulmonary angiographic imaging was 26.8 examinations per 1000 inpatient admissions before intervention and 22.6 examinations per 1000 inpatient admissions after intervention. This represented a relative reduction in use of 15.7% (26.8 − 22.6/26.8 · 100) (Table E1 [online]).

Based on the best-fit line from the preintervention period, use of CT pulmonary angiographic imaging in the month before intervention was estimated to be 26.0 examinations per 1000 admissions (Table 2; Table E2 [online]). By using a similar estimate based on postintervention period monthly use, estimated use of CT pulmonary angiographic imaging in the month immediately after intervention was 22.8 examinations per 1000 admissions (Table 2). This represented a significant relative reduction of 12.3% (P = .008) (Fig 3). There was no significant increase or decrease in use of CT pulmonary angiographic imaging in the periods before (April 1, 2007, to October 31, 2009) and after CDS implementation (November 1, 2009, to June 30, 2012 [P = .52]), which signified a sustained effect of CDS (Table 2). No significant difference in use of CT pulmonary angiographic imaging by clinician specialty was observed (Table 3).

Table 2.

Regression Model Results for Monthly Use and Yield of CT Pulmonary Angiography before and after Implementation of Evidence-based CDS

^*Found by the following equation: (Y_i) = α₁ + β₁T_i + β₂I_i + β₃O_i

^†Found by the following equation: β₁T_i + β₂I_i + β₃O_i

Figure 3:

Graph shows linear trend analysis for use of CT pulmonary angiographic imaging before (pre) and after (post) implementation of CDS.

Table 3.

Comparison of Monthly Use and Yield of CT Pulmonary Angiography before and after Implementation of Evidence-based CDS Stratified by Ordering Provider Subspecialty

Note.—Data are ± standard deviation where applicable; data in parentheses are percentages.

CT Pulmonary Angiography Yield

Of 5892 CT pulmonary angiographic examinations performed during the study period, 666 (11.3%) examinations were positive for PE. Based on the best-fit line from the preintervention period, the estimated monthly yield of CT pulmonary angiographic examinations or percentage of CT pulmonary angiographic examinations positive for acute PE in the month before implementation of CDS was 10.4% (Table 2; Table E2 [online]). Based on the best-fit line after intervention, the estimated monthly yield changed from 12.1% in the month after CDS implementation (Table 2; Table E2 [online]), a relative increase of 16.3% ([12.1 − 10.4/10.4] · 100); this difference was not statistically significant (P = .65) (Fig 4). There was no statistically significant increase or decrease in yield of CT pulmonary angiographic examinations in the periods before and after CDS implementation (P = .89). No significant difference in yield according to clinician specialty was observed (Table 3).

Figure 4:

Graph shows linear trend analysis for yield of CT pulmonary angiographic imaging before (pre) and after (post) implementation of CDS.

Discussion

We found that implementation of evidence-based CDS on the basis of a validated decision rule was associated with a significant immediate (12.3%) and sustained decrease in use of CT pulmonary angiographic examinations for evaluation of PE in hospitalized patients. An increase in yield of CT pulmonary angiographic imaging after intervention was not statistically significant. Our results suggest that the adoption of CDS on the basis of high-quality evidence can have an immediate significant effect on study ordering in the inpatient setting,

Our results differ from reported effect of CDS on use of CT pulmonary angiographic imaging in the ED. A study (10) reporting a 20.1% decrease in use of CT pulmonary angiographic imaging in the ED after CDS implementation with a concomitant educational campaign showed a gradual decline in the use of CT pulmonary angiographic imaging over a 2-year period after implementation of CDS. However, a 4-month study (15) did not find a decrease in the use of CT pulmonary angiographic imaging after implementation of CDS in the ED at another institution. Although our study does not allow for direct comparison, we believe this apparent variation in the effect of CDS in different care settings, even on the basis of the same validated decision rule, has important potential implications. It is likely that education-only CDS interventions (with no consequences to ordering providers who ignore the evidence presented in the CDS) will result in heterogeneous acceptance of CDS by providers depending on many factors, including care setting or level of training. Consequences for ignoring evidence may be needed to potentiate the effect of CDS in reducing unwarranted variations in care (9,12,24).

The observed 15.7% overall relative reduction in use of CT pulmonary angiographic imaging after CDS was greater than the estimated 12.3% reduction of use of CT pulmonary angiographic imaging within 30 days after CDS. Although this finding may represent a slight gradual decline in use of CT pulmonary angiographic imaging over the study period, the finding was not statistically significant. Our analysis showed a static trend in use of CT pulmonary angiographic imaging before CDS implementation without a significant change in the use trend before and after implementation of CDS. This is contrary to previous reports of increasing use of CT pulmonary angiographic imaging to evaluate patients suspected of having acute PE in both the ED (10,15,25) and inpatient settings (16). The static trend in use of CT pulmonary angiographic imaging before and after CDS at our institution may have been because of external and institution-specific factors, such as temporal changes in use of CT pulmonary angiographic imaging that were from increased emphasis within the medical literature and general media regarding the risks of medical radiation, increased knowledge of prediction rules to assess probability for acute PE, the effect of learning within our institution because of implementation of a similar CDS tool in our ED 27 months before the current study, and our institution’s established culture of quality. However, the immediate effect of our intervention observed within the first 30 days of CDS implementation argues against contribution of secular trends to our results.

While the yield of CT pulmonary angiographic imaging increased by 16.3%, this difference was not statistically significant, and our overall yield was lower than that of a previous study that reported a yield of 13.5% after implementation of risk stratification in 981 hospitalized patients who underwent CT pulmonary angiographic examinations because they were suspected of having PE (26). Our experience also differs from the findings in previous ED studies that reported increases in yield of CT pulmonary angiographic imaging for PE of between 50% and 81% after implementation of CDS (10,15,27). Our results may be explained in part by an institution-wide initiative that promoted venous thromboembolism prophylaxis in hospitalized patients, which was implemented during our study period. The effect of this concomitant intervention may have decreased the overall prevalence of venous thromboembolism in hospitalized patients at our institution and therefore masked a potentially greater increase in yield of CT pulmonary angiographic imaging after implementation of CDS than we observed.

Within our patient cohort, d-dimer testing was not routinely performed before CT pulmonary angiographic imaging. However, studies have shown that the use of d-dimer testing, in combination with a clinical decision rule that was incorporated in our intervention, may be used effectively to exclude PE in the inpatient population (28–30).

We observed a greater proportion of patients with a previous history of venous thromboembolism in our postintervention cohort. We suggest this observation is likely related to our intervention because patients without history of venous thromboembolic disease, a recognized risk factor for PE, were less likely to undergo CT pulmonary angiography than were those with known venous thromboembolic disease.

Our study had several limitations. First, it was performed at a single institution and so results may not be generalizable. Because our institution is a quaternary care hospital with a large oncology center, the risk for and incidence of thromboembolic disease in our patients may be higher than at other hospitals. Second, although our study demonstrated a sustained decline in use of CT pulmonary angiographic imaging after implementation of CDS, inclusion of control data from a similar institution where the CDS was not implemented would have better allowed us to account for potential secular changes in use of CT pulmonary angiographic imaging over the study period. In addition, the methodologic structure of our study precluded collection of additional patient characteristics, including heart rate, probability of an alternative diagnosis than PE, and direct evaluation of laboratory values including d-dimer level. Therefore, we were not able to independently apply established evidence-based criteria (19) to evaluate the appropriateness of performing CT pulmonary angiography and were only able to determine its use and yield. However, we believe the frequency of clinicians entering erroneous clinical information when ordering imaging studies to bypass perceived intrusive CDS interactions to be small (31). We were also unable to determine whether any patients had missed PE as a result of our CDS intervention because we did not have longitudinal data regarding a large portion of our cohort. These include patients with PE who did not undergo CT pulmonary angiographic imaging because the ordering physician accepted the evidence presented in CDS and cancelled the CT request, because the ordering physician decided not to obtain the CT after they learned about evidence from previous interaction with CDS in a different patient, or because the ordering physician conceivably avoided the computer interactions presented in CDS. The observed 16.3% increase in yield of CT pulmonary angiographic imaging was likely not statistically significant because our study was powered to detect a change in use, and therefore underpowered to detect a significant increase in yield of CT pulmonary angiography, given the overall low number of positive CT pulmonary angiographic studies during the study period. Similarly, our analysis of effect of CDS by ordering provider specialty was likely underpowered.

In conclusion, implementation of CDS on the basis of high-quality evidence was associated with a statistically significant decrease in the use of CT pulmonary angiographic imaging for acute PE in hospitalized patients that occurred within the first 30 days after implementation of CDS. This decrease in use was sustained over the ensuing 32 months. Imaging CDS and CDS-enabled interventions may provide an effective strategy to reduce waste, improve quality of care, and reduce unnecessary radiation exposure for hospitalized patients.

Advance in Knowledge

• ■ Implementation of evidence-based clinical decision support (CDS) was associated with a significant (12.3% decrease, from 26.0 to 22.8 CT pulmonary angiographic examinations per 1000 admissions before and after CDS, respectively; P = .008) immediate (within 1 month) and sustained (over the 32-month study period) decrease in use of CT pulmonary angiographic imaging for evaluation of acute pulmonary embolism in hospitalized patients.

Implication for Patient Care

• ■ Adoption of CDS on the basis of high-quality evidence can have an immediate and significant effect on the use of CT pulmonary angiographic imaging in the inpatient setting, with the potential to improve efficiency of care and reduce potentially unnecessary radiation exposure to patients.