CT Pulmonary Angiogram Clinical Pretest Probability Tool: Impact on Emergency Department Utilization

Abstract

Objective:

Currently, computed tomographic pulmonary angiogram (CTPA) for evaluating acute pulmonary embolism (PE) in Emergency Departments (EDs) is overused and with low yields. The goal of this study is to assess the impact of an evidence-based clinical decision support (CDS) tool, aimed at optimizing appropriate use of CTPA for evaluating PE.

Methods:

The study was performed at EDs in a large healthcare system and included 9 academic and community hospitals. The primary outcome was the percent difference in utilization (number of CTPA performed/number of ED visits) and secondary outcome was yield (percentage of CTPA positive for acute PE), comparing 12 months before (6/1/2021–5/31/2022) vs. 12 months after (6/1/2022–5/31/2023) a system-wide implementation of the CDS. Univariate and multivariable analyses using logistic regression were performed to assess factors associated with diagnosis of acute PE. Statistical process control (SPC) charts were used to assess monthly trends in utilization and yield.

Results:

Among 931,677 visits to Emergency Departments, 28,101 CTPAs were performed on 24,675 patients. 14,825 CTPAs were performed among 455,038 visits (3.26%) pre-intervention; 13,276 among 476,639 visits (2.79%) post-intervention, a 14.51% relative decrease in CTPA utilization (chi-square, p<0.001). CTPA yield remained unchanged (1371/14825=9.25% pre- vs. 1184/13276=8.92% post-intervention; chi-square, p=0.34). Patients with COVID diagnosis prior to CTPA had higher probability of acute PE. SPC charts demonstrated seasonal variation in utilization (Friedman test, p=0.047).

Discussion:

Implementing a CDS based on validated decision rules was associated with a significant reduction in CTPA utilization. The change was immediate and sustained for 12 months post-intervention.

BACKGROUND

Pulmonary embolism (PE) is a major cause of morbidity and mortality worldwide with an annual incidence of 3 in 1000 United States adults.^1,2 Accurate PE diagnosis is essential as untreated PE carries a mortality rate as high as 30%.³ However, PE can be difficult to diagnose, often presenting with differing or nonspecific signs and symptoms. Although testing for suspected PE has increased, recent data suggests that only 1–5% of all computed tomography pulmonary angiography (CTPA) examinations ordered are diagnostic of PE, signifying low yields when using this test in isolation.⁴ Identifying tools to increase CTPA yield and decrease unnecessary CTPAs is important to reduce potential safety risks associated with radiation exposure and use of intravenous (IV) contrast, and to reduce waste.

Global shortages of IV iodinated contrast due to vendor supply chain disruptions related to the COVID-19 pandemic further exacerbated the need to decrease unnecessary CTPA examinations.⁵ This shortage forced hospitals, and in particular radiology departments, to create strategies to mitigate this crisis.^6–8 At the Mass General Brigham (MGB), a large multi-hospital system, an important area of focus for mitigation strategies was the Emergency Department (ED), as there has been substantial growth in CT scan utilization in adult ED patients overall,^9,10 and a dramatic escalation in the utilization of CTPA to diagnose PE.^11,12

To decrease utilization and target appropriate testing, clinical pretest probability (CPTP) tools have been developed and validated to help clinicians better risk stratify patients.^13–19 Despite advocacy in numerous professional society guidelines,^20–25 the utilization of CPTP scores in practice remains low. Moreover, studies examining the effects of CDS have revealed mixed results. Many studies have demonstrated that when CPTP tools are embedded as clinical decision support (CDS) into the electronic health record (EHR), the number of imaging examinations ordered decreases while other studies have not.^26–30 Raja et al. demonstrated that after implementing a computerized evidence-based CDS tool in a single ED, quarterly CTPA utilization decreased by 20%.²⁶ In another study in the same institution, CTPA yield was twice as high for ED providers who adhered to CDS compared to those who overrode CDS alerts based on CPTP assessment.³¹ However, no multi-institutional studies have demonstrated the impact of CPTP tools implemented as CDS and embedded in the EHR to optimize utilization of CTPAs for acute PE.

Given the promising but inconsistent evidence and the additional impetus to respond to the IV contrast shortage, we rapidly designed and implemented CDS using CPTP tools based on high-quality evidence for the diagnosis of PE into our multi-hospital system-wide EHR in order to optimize the number of CTPAs performed for acute PE.

In this study, we aimed to assess the impact of an evidence-based CDS tool, aimed at optimizing appropriate use of CTPA for evaluating PE on both the utilization and yield of CTPAs, including impact over time.

METHODS

Study Setting and Population

This prospective cohort HIPAA-compliant study was approved by the Institutional Review Board. MGB, study site, is a large multisite healthcare system in the Northeast that includes 12 acute and specialty hospitals, with 9 EDs and >470,000 ED visits annually. We excluded 3 MGB specialty hospitals. All adult patients (age ≥18) who presented to one of the system’s EDs between June 1, 2021, and May 31, 2023 (12 months pre- and post-intervention), and for whom a CTPA was ordered in the system-wide EHR (Hyperspace, Epic Systems, Verona, WI), were included in analyses.

Intervention

A CDS tool for CTPA was designed over a 2-week period and implemented in the EHR on June 1, 2022, using Epic Best Practice Advisories (BPA), which are a central tool in the Epic decision support system. Radiology and Emergency Medicine have a single integrated governance structure across MGB institutions. A multidisciplinary expert panel from the departments of Emergency Medicine, Radiology, and Hematology as well as the EHR team was tasked with creating the CDS tool. Its clinical content utilized Clinical Evidence Logic Statements (CELS) created in the Harvard Medical School Library of Evidence (HLE),^32,33 representing CPTP tools derived from the three-tier model of the Well’s criteria³⁴ and the Pulmonary Embolism Rule Out Criteria (PERC),³⁵ both of which are based on several PE risk factors and endorsed by societies and societal guidelines.^20–25 These CELS are semi-structured artifacts given Level 2 grading in the HLE, based on the Oxford Center for Evidence-Based Medicine Levels of Evidence grading system for diagnosis on a scale from 1 to 5, with level 1 having the highest level of evidence. Level 2 evidence denotes “individual cross sectional studies with consistently applied reference standard and blinding.”³⁶ Expert consensus at that time decided not to use the YEARS algorithm for PE.³⁷

When providers attempted to order a CTPA, the CDS highlighted the required CPTP tool. In order to complete the CTPA order, providers were obligated to fill out the CDS tool, and prompted to use the results to determine the patient’s risk for PE (Supplemental Figure 1). For patients with a negative or null d-dimer result, the CTPA order displayed additional questions corresponding to the Well’s criteria. The first question offered the specific Well’s criteria; the second asked for the total number of points. Providers were then prompted to order the recommended next test. Options included: a) proceeding with an additional CPTP (PERC) if the CPTP score indicted low risk of PE, b) obtaining a highly sensitive plasma D-dimer test as additional information for risk stratification in intermediate risk patients, or c) obtaining CTPA in patients considered to be high risk (Supplemental Figure 2).

A sidebar in the EHR’s ordering workflow included a description of the CPTP tools with algorithms and links to references and the validation studies behind the CPTP tools (Supplemental Figure 3). After completing the CDS, ordering providers could opt to override the recommendations and were prompted to provide an explanation for their decision.

Health system leadership sent system-wide provider emails detailing the contrast media shortage and the rationale for implementing CPTP-based CDS. A ‘tip sheet’ including screenshots of the ordering system was included in the communication.

Data Collection

Data was obtained from our EHR’s data warehouse, including patient demographic variables (age, sex, race, insurance status, and number of days since COVID diagnosis to ED admission, if applicable) and the number of CTPAs and ED visits. We re-trained and validated a previously published natural language processing (NLP) tool used to extract information from radiology reports³⁸ to identify CTPA reports positive for acute PE. The number of CTPAs per ED visits and the percentage of CTPAs positive for acute PE were calculated subsequently. Data was collected and analyzed for each institution as well as the entire healthcare system; due to low numbers, data from the two smallest hospitals were combined.

Data Extraction

To identify radiology reports with acute PE, reports were analyzed using NLP search using Bidirectional Encoder Representations from Transformers (BERT),³⁸ a pretrained deep-learning model that was further trained to identify reports with acute PE. BERT for PE is publicly available in https://github.com/BWH-CEBI/CTPE_BERT.

BERT for PE was trained on a large body of radiology reports. Every CTPA performed from 4/2022–7/2022 (8 weeks before and 8 weeks after CPTP implementation) at the largest hospital in the system (Massachusetts General Hospital, 1131 reports) was reviewed independently by 2 reviewers (a radiology research assistant [MI] and radiology research fellow [NA]). Three additional independent expert attendings (Radiology [AS], Emergency Medicine [AR], and Hematology [RR]) reviewed an enriched sample of 100 CTPAs where half were positive for PE. Among all reviewers, kappa (Cohen’s kappa for MI and NA; Fleiss’ kappa for AS, AR and RR) and percent agreements were calculated. An additional 6732 reports from 8/2022–11/2022 were annotated by the research assistant [MI] to measure NLP accuracy (i.e., precision, recall, and F-score).

Statistical Analysis

The primary outcome was the relative percent difference in the total utilization of CTPAs (number of CTPAs per ED visits) comparing 12 months before and 12 months after system-wide implementation of the intervention. Secondary outcomes included the relative percent difference in total yield (percentage of CTPAs positive for acute PE) between the two study periods, as well as the utilization and yield of CTPA for each institution. Chi-square analysis was used to measure the significance of percent differences. We further assessed factors that were associated with being diagnosed with a PE. Univariate analysis was performed using logistic regression for categorical variables (sex, race, ethnicity, insurance, CDS intervention) and continuous variables (age). Days from ED admission to COVID diagnosis was evaluated using 5 categories, >180 days, 91–180 days, 31–90 days, 1–30 days from COVID diagnosis, and no COVID diagnosis. Logistic regression was employed for multivariable analysis. Factors with p<0.05 on univariate analysis were included in the multivariable logistic regression model as long as any one category was significant for the categorical variables. Wald test was used to assess model coefficients. No adjustments were made for multiple hypothesis testing. However, when analyzing the utilization and yield by site, given the numerous statical comparisons, the Bonferroni adjustment was used, and significance was a p value <0.006. For all other analyses, statistical significance was identified as p<0.05. Both kappa statistics and the percent agreement were used to measure interrater reliability. All analyses were performed using R version 3.4.1 (R Foundation for Statistical Computing, Vienna, Austria). Finally, a statistical process control (SPC) u-chart with 3-sigma control limits was used to assess relative changes in pre- and post-intervention utilization, and p-chart with 3-sigma control limits was used to assess relative changes in yield. We noted a seasonal trend in CTPA utilization, prompting inclusion of data from 12 months prior to the study period (June 1, 2020, to May 31, 2021) to assess seasonal trends in CTPA utilization in the SPC u-chart. In addition, Friedman test was used to evaluate the presence of seasonality. SPC analyses were performed using JMP (JMP, version 16.2, SAS Institute, Cary NC).

RESULTS

Study Cohort

Among 931,677 ED visits during the study period, 28,101 CTPAs were performed on 24,675 unique patients. Mean patient age was 59 years; 58% were female. Table 1 characterizes other patient demographics.

Table 1.

Patient Demographics

Patient Characteristics	Patients who underwent CTPA (N=24,675)
	Pre-Intervention (N=13,491)	Post-Intervention (N=11,184)
Age, mean (range, standard deviation)	Mean=59.1 (10–104, 18.5)	Mean=59.8 (5–104, 18.9)
Female (%)	7,727 (57.3)	6,605 (59.1)
Race
African American	1,430 (10.6)	1,186 (10.6)
Asian	402 (3.0)	401 (3.6)
Other	1,429 (10.6)	1,208 (10.8)
White	10,230 (75.8)	8,389 (75.0)
Ethnicity
Hispanic	1,458 (10.8)	1,180 (10.6)
Non-Hispanic	12,033 (89.2)	10,004 (89.4)
Insurance
Private/Others	7,163 (53.1)	6,093 (54.5)
Public	6,328 (46.9)	5,091 (45.5)
Days since COVID Diagnosis
Group 1 (180+ days)	7,145 (53.0)	7,532 (67.3)
Group 2 (91–180 days)	946 (7.0)	426 (3.8)
Group 3 (31–90 days)	746 (5.5)	390 (3.5)
Group 4 (1–30 days)	3,154 (23.4)	1,909 (17.1)
Group 5 (No COVID diagnosis)	1,500 (11.1)	927 (8.3)

Abbreviations:

CDS – clinical decision support; CTPA - computed tomographic; PE - pulmonary angiogram

Interrater Reliability and NLP Accuracy

The interrater reliability between the research assistant and research fellow for manual review for acute PE was 1 for Cohen’s kappa and 100% for percent agreement. The interrater reliability for the enriched sample between three expert attendings was 0.80 for Fleiss’ Kappa and 85% for percent agreement. Combining these two groups, kappa and percent agreements were 0.78 and 88%, respectively. The BERT for PE model has robust performance with precision of 0.98, recall of 0.96 and F-score of 0.97 in identifying reports containing PE within our health system.

CTPA Utilization and Yield

A total 14,825 CTPAs were performed among 455,038 ED patients (3.26%) pre-intervention and 13,276 CTPAs among 476,639 ED patients (2.79%) post-intervention, a statistically significant relative decrease of 14.51% in CTPA utilization (p<0.001; Figure 1A). The absolute decrease is 0.47%. Of the 14,825 pre-intervention CTPAs, 1,371 (9.25%) were positive for PE vs. 1,184 (8.92%) of the 13,276 post-intervention CTPAs, a relative decrease of 3.56% in the positive yield of CTPA, although this did not reach statistical significance (p=0.34) (Figure 1B). The absolute decrease is 0.33%.

Figure 1A. CTPA Utilization by Institution.

Computed tomographic pulmonary angiogram (CTPA) utilization (number of CTPAs per Emergency Department visits) by institution pre- and post-intervention, presented as percentages.

Abbreviations: BWF - Brigham and Women’s Faulkner; BWH – Brigham and Women’s Hospital; CDH – Coley Dickinson Hospital; MGH – Massachusetts General Hospital; MVH – Martha’s Vineyard Hospital; NCH – Nantucket Cottage Hospital; NWH – Newton Wellesley Hospital; SLM – Salem Hospital; WDH Wentworth Douglass Hospital; MGB – Mass General Brigham

Note: Brigham and Women’s Hospital and Massachusetts General Hospital are academic centers and the 7 community acute-care hospitals include Brigham and Women’s Faulkner, Cooley Dickinson Hospital, Martha’s Vineyard Hospital, Nantucket Cottage Hospital, Newton-Wellesley Hospital, Salem Hospital and Wentworth Douglass Hospital. Martha’s Vineyard Hospital and Nantucket Cottage Hospital, the two smallest hospitals, were combined for analysis. Brigham and Women’s Faulkner and Newton-Wellesley Hospital are community teaching hospitals, while Cooley Dickinson Hospital and Wentworth Douglass Hospital are medium-sized non-teaching hospitals. Salem Hospital is a large non-teaching hospital.

Figure 1B. CTPA Yield by Institution.

Computed tomographic pulmonary angiogram (CTPA) yield (percentage of CTPAs positive for acute pulmonary embolism) by institution pre-and post-intervention, presented as percentages.

Abbreviations: BWF - Brigham and Women’s Faulkner; BWH – Brigham and Women’s Hospital; CDH – Cooley Dickinson Hospital; MGH – Massachusetts General Hospital; MVH – Martha’s Vineyard Hospital; NCH – Nantucket Cottage Hospital; NWH – Newton Wellesley Hospital; SLM – Salem Hospital; WDH Wentworth Douglass Hospital; MGB – Mass General Brigham

Note: Brigham and Women’s Hospital and Massachusetts General Hospital are academic centers and the 7 community acute-care hospitals include Brigham and Women’s Faulkner, Cooley Dickinson Hospital, Martha’s Vineyard Hospital, Nantucket Cottage Hospital, Newton-Wellesley Hospital, Salem Hospital and Wentworth Douglass Hospital. Martha’s Vineyard Hospital and Nantucket Cottage Hospital, the two smallest hospitals, were combined for analysis. Brigham and Women’s Faulkner and Newton-Wellesley Hospital are community teaching hospitals, while Cooley Dickinson Hospital and Wentworth Douglass Hospital are medium-sized non-teaching hospitals. Salem Hospital is a large non-teaching hospital.

Institutional-level Findings

When assessed by site, the largest relative decrease in utilization, 26.89% (p<0.001), was in a medium-sized institution. The largest number of ED visits was at the Massachusetts General Hospital whose percent CTPA utilization decrease was 15.14% (p<0.001). The percent decrease in utilization of CTPAs was significant in all institutions except two (Supplemental Table 1).

One site had a statistically significant increase in yield. However, both academic medical centers had decreased yield post-intervention, although the differences were not statistically significant. There were some institutions where the yield increased post-intervention, also not significantly. Across all 9 institutions, the percent difference in yield was not statistically significant (Supplemental Table 2).

Univariate and Multivariable Analyses

In univariate analysis, the odds of being diagnosed with PE increased with age (Odds Ratio [OR]>1.01, p<0.001) (Table 2). The odds of being diagnosed with PE decreased for women vs. men (OR=0.77, p<0.001) and for patients of Asian race, compared to White race (OR=0.46, p<0.001) There was decreased odds of having PE in patients of Hispanic ethnicity, although this was not significant on multivariable analysis. There was also small increased PE diagnosis in patients who had COVID within 30 days (OR=1.30, p<0.001), 90 days (OR=1.57, p<0.001) and 180 days (OR=1.25, p=0.02) of undergoing CTPA. Insurance and CDS use were not associated with the risk of being diagnosed with PE.

Table 2.

Univariate and multivariable analysis using logistic regression to assess factors associated with the diagnosis of acute PE.

Patient Factors	Univariate		Multivariable
	Odds Ratio	p-value*	Odds Ratio	p-value*
Age, in years	1.01	<0.001	1.01	<0.001
Sex (male as reference)
Female	0.77	<0.001	0.80	<0.001
Race (white as reference)
African American	1.00	0.99	1.11	0.16
Asian	0.46	<0.001	0.46	<0.001
Other	0.76	<0.001	0.91	0.35
Ethnicity (Non-Hispanic as reference)
Hispanic	0.72	<0.001	0.87	0.18
Insurance (Private/Others as reference)
Public	0.93	0.08	n/a	n/a
Recent COVID Diagnosis (180+ days as reference)
Group 2 (91–180 days)	1.25	0.02	1.23	0.03
Group 3 (31–90 days)	1.57	<0.001	1.52	<0.001
Group 4 (1–30 days)	1.30	<0.001	1.31	<0.001
Group 5 (No COVID diagnosis)	1.49	<0.001	1.58	<0.001
Pre/Post CDS Intervention (pre-intervention as reference)
Post-Intervention	0.97	0.51	1.01	0.81

Abbreviations:

CDS – clinical decision support; PE – pulmonary embolism

^*Bold indicates statistical significance.

In the multivariable analysis, age, female sex, and Asian race remained significant (Table 2). COVID diagnosis also remained a significant predictor of being diagnosed with PE.

SPC Charts

The SPC chart for utilization demonstrated seasonal variation, with peaks in December to January, and troughs from June to September. Friedman test showed significant presence of seasonality (p=0.047). The SPC chart showed significant decrease in utilization during the time of the intervention, evidenced by a run of 7 data points below the 3-sigma limit (Figure 2A). The first point happened just before the intervention date, coinciding with the trough in seasonality. This impact was exaggerated and extended with decreased CTPA utilization at 3 standard deviations below the mean for an additional 12 months after the intervention. The SPC chart for CTPA yield (Figure 2B) did not show significant variation during the study period.

Figure 2A: Statistical Process Control U-chart of Pre- and Post-intervention CTPA Utilization.

Statistical process control to assess changes in pre- and post-intervention utilization.

Figure 2B: Statistical Process Control P-chart of Pre- and Post-intervention CTPA Yield.

Statistical process control to assess changes in pre- and post-intervention yield.

CTPA - computer tomography pulmonary angiogram; LCL - lower control limit; UCL - upper control limit, 3-sigma control limits

DISCUSSION

In this multisite healthcare system study, requiring interaction with CDS based on validated decision rules prior to ordering a CTPA for ED patients with suspicion of PE led to a 14.51% relative decrease in the utilization of CTPAs without a significant change in yield. Rapidly developed and implemented within the EHR, CDS was associated with an immediate, significant, and sustained impact across nine acute care hospitals, including two academic medical centers. Leveraging a multidisciplinary team to design the CDS tool and embedding it into a system-wide EHR enabled rapid implementation time. Enabled through a unified central clinical governance and access to a provider-led repository of vetted medical evidence, the CDS enabled clinicians to risk stratify their patients in real time and in one place.³²

Although the impetus behind our CDS tool for suspected PE was the global IV contrast media shortage, using an evidence-based prediction tool to reduce the number of CTPAs has the potential to decrease potential harm and costs associated with unnecessary CT imaging, including kidney injury and allergic reactions from IV contrast, as well as radiation exposure.^39,40 A repository of CDS-consumable knowledge artifacts enabled rapid development of a CDS tool, utilizing artifacts based on high-quality evidence.

Despite a significant decrease in utilization, there was no significant change in PE yield post-intervention, which is in line with some previous studies. Similar to our findings, Duanne et al reported a significant decrease in utilization but no significant change in yield of CTPA for PE after implementing an evidence-based CDS for inpatients.⁴¹ However, other studies such as the one from Mills et al showed an increase in yield after implementation of an evidence-based CDS in a multi-site prospective quality improvement study.⁴²

In our study, yield ranged from 7.02%−11.43% across all nine hospitals, and systemwide was 8.92%; physician and institutional preferences, as well as patient factors, likely contributed to this variation. The post-intervention yield likely did not increase because the incidence of PE decreased in patients the farther they were from their COVID diagnosis (Table 2). Specifically, we observed a statistically significant increase in the odds for PE when COVID diagnosis was between 1 and 180 days pre-CTPA, versus having COVID beyond 180 days. This is not surprising, as COVID-associated coagulopathy has been well described with reported incidence of venous thromboembolism ranging from 1–75% depending on the patient population.^43–45 Patients with COVID are particularly challenging to evaluate as shortness of breath is a common symptom for both COVID and PE. This finding highlights the need to carefully evaluate these patients in the setting of acute COVID.⁴⁶ In addition, a significant impact on yield might have been achieved with more educational and awareness efforts at the start of CDS implementation leading to more appropriate utilization. Concurrent interventions, including feedback reporting has been shown to improve adherence to CDS recommendations.^47,48 More importantly, CDS has been shown to be more effective when implemented as part of a multidisciplinary clinical program rather than simply an Information Technology initiative.^49,50 Another reason we did not observe an increase in CTPA yield is if patients with PE were not scanned due being excluded by the screening tool. The screening tool variables did not auto populate the CDS and rather, providers had to manually input them and calculate the score.

Our finding that women had a lower odds of PE diagnosis than men has been reported in prior studies using CPTP tools on patients with suspected PE.^51,52 When comparing the performance of three validated pre-imaging diagnostic algorithms in patients with suspected PE, van Mens, et al observed no sex differences in any of the separate algorithms, but the prevalence of PE was lower in women than men with all three.⁵³ An increase in PE incidence with age is well documented.^54,55

We noted a significant decrease in CTPA utilization during the succeeding 6 months after the intervention (Figure 2A). The SPC chart demonstrated special cause variation indicated by points outside the control limit. After that decrease, utilization increased as part of the seasonal trend, as reflected in the SPC chart, although it did not cross the three-sigma limit as it had in previous years. Seasonality may have contributed to the increase in utilization after the initial 6 months. Numerous studies have explored and demonstrated seasonal variation in the incidence of PE.⁵⁶ In a time-series analysis involving 162,032 diagnoses of PE gleaned from the Spanish National Health system over a 10-year period, there was a linear increase in the incidence of PE and a significant seasonal pattern with 17% more admissions in February and 12% fewer in June-July.⁵⁷ Another study using the French hospital discharge database found that compared with summer months, the winter peak was associated with 25% increasing rates in hospitalizations for and mortality due to PE.⁵⁸ No study has specifically evaluated the seasonal variation associated with ordering of CTPAs although it is not surprising that it would follow a similar seasonal pattern as that seen with the diagnosis of PE.

There are few limitations to our study worth noting. First, our study was that the CDS tool was only linked to the ordering of CTPAs. Clinicians could order any other chest CT scan for other clinical indications and bypass the CDS. Clinicians also did not have to mark whether each Well’s criterion was present or not; they only checked those they knew were present at the time of CPTP completion. There was no forcing function to ensure that each criterion was explicitly documented during the ordering process. Furthermore, once clinicians marked which Well’s criteria were present, they had to calculate the score (Supplemental Figure 1); it was not automatic. Due to a technical limitation in our EHR, a clinician could mark ‘high probability’ even if the calculated score based on Well’s criteria entered in the order did not match that assessment. These shortcomings may have contributed to the stable yield of CTPAs we observed. Second, our study involved only one health care system, which may limit the generalizability of our findings in terms of regional practice patterns and patient mix. However, our health system is broad including 9 institutions that vary in size, location, trauma designation. Third, a major confounding factor in our study is the influence of COVID on VTE disease. Moreover, the proximity of the study to COVID may have impacted the prevalence of PE and thus, the rates of PE positivity in the post intervention period. Lastly, although our study demonstrated a decrease in utilization, due to numerous variables involved in cost analysis, this topic was not explored. However, this question can be addressed in future studies.

Summary Statement:

In this multisite healthcare system study, requiring interaction with CDS based on validated decision rules prior to ordering a CTPA for ED patients with suspicion of PE led to a 14.51% decrease in the utilization of CTPAs without a significant change in yield.

Take-Home Points:

• The global contrast shortage provided our multisite healthcare system an occasion to change clinician practice patterns when evaluating an ED patient with suspected PE using a CDS tool based on validated decision rules.

• Leveraging a multidisciplinary team to design the CDS tool, a unified central clinical governance and access to a provider-led repository of vetted medical evidence, we embedded the CDS into a system-wide EHR with rapid implementation time.

• Using an evidence-based CDS tool to reduce the number of CTPAs has the potential to decrease potential harm and costs associated with unnecessary CT imaging.

• Implementing a CDS tool based on validated decision rules significantly decreased the utilization of CTPA without sacrificing the yield and sustained these changes over time.