Higher Imaging Yield when Clinical Decision Support is Used

SUMMARY SENTENCE

In this retrospective cohort study, after controlling for selection bias with propensity score matching diagnostic yield was 38% higher for computed tomography pulmonary angiography testing done to evaluate for pulmonary embolism when clinical decision support was used.

INTRODUCTION

Pulmonary embolism (PE) is a common condition with an untreated mortality rate of up to 34% [1–5]. Its clinical presentation is nonspecific and as a result providers demonstrate a low threshold to perform advanced diagnostic imaging [6–8]. Computed tomography pulmonary angiography (CTPA) use has quadrupled over the past decade since the landmark PIOPED II trial established it as both highly sensitive and specific for the diagnosis of PE.[9–11] Increased utilization however, has been associated with decreasing diagnostic yields and rising concerns about the harms of unnecessary testing. [4, 12–18] It is estimated that up to one third of exams ordered for PE may be inappropriate.[19] CTPA yield, the percentage of tests positive for PE, is increasingly being used by health care systems to measure appropriateness of use, with typical ranges of published yields varying from 6 to 25% [20, 21].

Clinical prediction rules have been developed to assist providers with imaging decisions.[22, 23] They classify patients into low, intermediate and/or high probability of PE based on key elements of the history and physical exam, establishing pre-test probability prior to imaging. The use of clinical prediction rules to assess pre-test probability increases specificity for evaluation of PE compared to gestalt assessment without decreasing sensitivity.[24, 25] Although these tools have been around for more than a decade they are not commonly used; at our institution pre-test probability is only formally calculated before CTPA about 3% of the time.[26]

Every year about 2.4 million CTPA scans are performed to evaluate for PE in EDs in the United States.[27] Each CTPA carries a 14% risk of contrast induced nephropathy[28] and a lifetime malignancy risk that can be as high as 2.76%[29]. Incidental findings requiring diagnostic follow up are found in 24% of tests, increasing both costs and harms from repeat imaging.[30] The use of clinical prediction rules to assess pre-test probability before CTPA reduces testing by 25% without any missed PEs.[31, 32] There routine use by providers would result in 600,000 fewer scans, 84,000 fewer cases of contrast induced nephropathy and prevent 3,000 malignancies as well as 2,000 cancer deaths in the United States every year[29].

Use of clinical decision support (CDS) incorporating clinical prediction rules has been shown to improve CTPA yield from 30% to 98% [33–35]. However, provider adoption of CDS as a whole tends to be low[36] with up to 96% of CDS being overridden[37]. We employed a user-centered development process to create an optional CDS tool that incorporates a well validated clinical prediction rule to assist providers with estimating pre-test probability of PE.[38–41] A prior study evaluating an optional CDS tool found higher yield for CDS users but the elevated yields may have been the result of selection bias.[42] The objective of this study was to characterize CDS tool users, determine whether optional CDS for CTPA was associated with increased yields and whether this association would remain after controlling for selection bias.

METHODS

This study was compliant with the Health Insurance and Portability and Accountability Act (HIPAA) and approved by the Institutional Review Board. We performed a retrospective cohort study in the Emergency Departments (ED) of two tertiary care hospitals. The study included all CTPAs performed at either institution between August 2015 and September 2018. The CDS tool was launched on April 28, 2015 at Hospital 1 and July 28, 2015 at Hospital 2. The tool was launched only in the EDs of the two institutions as the clinical prediction rule, Wells’ Criteria for Pulmonary Embolism, was originally validated in this setting.[43] The CDS tool was designed and launched in the Sunrise Emergency Care electronic health record, a product of Allscripts Healthcare Solutions. The tool was developed over a period of three months, at a cost of $54,000, by the Office of the Chief Information Officer at Northwell Health in collaboration with Allscripts Healthcare Solutions.

User Centered Design

The tool was developed using adaptive principles in web and health information technology design which are detailed in several previous publications.[38, 39, 44, 45] There were four design phases 1) focus-group feedback on ED workflow and CDS adoption barriers, 2) key-informant interviews using mocked up tool wireframes, 3) iterative “Think Aloud” usability testing of prototypes and 4) key-informant interviews after the launch of the tool in the real clinical environment. Based on phases 1-3 the tool was developed and launched on April 28, 2015 at Hospital 1 and July 28th, 2015 at Hospital 2.

Phase 4 was conducted in November of 2015 and the tool was revised at both sites in January of 2016. The tool was revised to allow CT testing for the intermediate-risk group and not to trigger for CT chest with intravenous contrast orders. A dialogue box, “Are you considering PE for this patient?” was added before the tool triggered to account for CTPAs ordered for aortic dissection evaluation. Before this dialogue box CTPA orders placed to evaluate for aortic dissection were removed from the calculation of CTPA yield by eliminating any with an order for CTA abdomen/pelvis during the same 24 hour period. This accounted for all CTPA orders not placed to evaluate for PE in our validation study. [46] The entire user centered design process, including tool revisions, were conducted with the expectation that they would increase use of the tool.[38]

CDS Tool

The tool was designed to reduce CTPA ordering and as a result, expected to increase CTPA yield. Providers entering any electronic order for the diagnosis of PE (d-dimer, V/Q scan or CTPA) were routed to the tool if they answer yes to a dialog box asking “Are you considering PE?”. The tool functions as an expanded order-set that allows providers to formally calculate pre-test probability of PE using the Wells’ Criteria for PE. (Figure 1) To calculate Wells’ Criteria for PE, two criteria are pre-populated, patient heart rate and history of PE or deep venous thrombosis and the remainder are manually entered by the user. After all criteria are populated patients are classified as low, intermediate or high risk. For low risk patients it forces providers to order d-dimer laboratory testing and for intermediate or high risk of PE it allows for d-dimer testing, V/Q scan or CTPA imaging. At any time the tool can be dismissed by providers and then any order can be placed. For the purposes of this study, a CTPA study with CDS use is one where the tool was not dismissed.

Figure 1.

CDS Tool

Data Collection

Data was collected by electronic health record reporting forms. Patient demographic data collected included age, gender, race, body mass index and comorbidities. Comorbidities were specified as cardiovascular diseases, dementia, chronic obstructive pulmonary disease, peptic ulcer disease, diabetes, chronic kidney disease, malignancy, human immunodeficiency disease, rheumatologic disease and liver disease. Total daily ED visits were also used to calculate a measure of ED crowdedness, where days with visits in the top quartile were considered crowded. Over 20 different definitions of ED crowding have been proposed in the literature.[47] Most however are unable to be queried by electronic health record report for a large observational study like ours. We choose this measure because we hypothesized that providers’ relative perception of a crowded ED might lead them to feel overwhelmed and be less likely to take the time to use the CDS tool. Ordering provider type (attending, resident or physician assistant) and date and time of CTPA order were also collected.

CTPA yield was calculated as the number of ED CTPA orders linked to an inpatient discharge diagnosis of PE divided by the total number CTPAs completed. CTPAs were removed if they occurred on the same day as a CT angiogram of the abdomen and pelvis as these were likely performed to evaluate for aortic pathology. This method has been validated at both of these hospitals.[48] PE diagnosis was measured based on a discharge diagnosis in accordance with International Classification of Diseases, Clinical Modification codes, versions 9 and 10 provided by the Centers for Medicare and Medicaid Services and the National Center for Health Statistics. The full range of PE diagnosis codes was used and consisted of the following: 415.0, 415.11, 415.12, 415.13, and 415.19 for ICD-9- CM; and I26.0, I26.01, I26.02, I26.09, I26.9, I26.90, I26.92, and I26.99 for ICD-10-CM.

Data Analysis

CTPA orders placed after the provider did not elect to dismiss the toolwere placed in the CDS-use group. CTPA orders placed after the provider elected to dismiss the tool were placed in the CDS-dismissal group. Orders completed by Fellows and Nurse Practitioners were excluded due to the small number, 20 and 27 respectively. Those cases with an abnormal d-dimer result before the CTPA order placement were removed as in these cases it would have been appropriate to dismiss the tool. CDS use rate was compared by provider type using the chi-squared test. Logistic regression was used to compare the proportions of CDS tool use among different provider types: attending, physician assistant and resident after controlling for patient variables: number of comorbidities, ethnicity, whether the ED was crowded or not on the same day, and quarter of visit and order time. Patient populations in the CDS-use and CDS-dismissal groups were compared using the t-test or Wilcoxon Rank Sum test for continuous variables, and chi-squared or Fisher’s exact test for categorical variables.

CTPA yield was calculated for tests done in the CDS-use and CDS-dismissal groups for all providers and stratified by provider type. The increase in CTPA yield between CTPA tests done in the CDS-use and CDS-dismissal groups was compared across provider type with the Cochran-Mantel-Haenszel test. As this was a retrospective cohort study, the probability of being in the CDS-use and CDS-dismissal groups could have been influenced by baseline patient, provider and environment factors. Propensity score matching was used to eliminate confounding due to measured baseline characteristics when comparing the CTPA yield of the two groups. The propensity scores for patients were estimated using logistic regression model with response variable as CDS-use and CDS-dismissal and predictor variables: patient age, gender, race, number of comorbidities and ED crowdedness or not on the same day, month of test, time of test and provider type. After propensity score matching, CTPA yield was compared for the CDS-use and CDS-dismissal groups for all providers and stratified by provider type with McNemar’s test.

RESULTS

A total of 9,108 CTPAs were ordered to evaluate for PE during the study period. Of those, 1,741 were removed because there was an abnormal d-dimer result before the CTPA order was placed. In those cases tool dismissal would be appropriate. The CTPA yield in those cases was 10.22%. Of the remaining 7,367 studies, providers used the CDS tool in 2,568 (35%) cases and did not use the tool in 4,799 (65%) of cases. The CTPA yield was 11.99% in the CDS-use group and 8.44% in the CDS-dismissal group (p < 0.001).

Tool Use

Patient variables, ED crowdedness, test time of day and season of the year and provider type for the CDS-use and CDS-dismissal groups are included as Table 1. The CDS-use group had a higher average number of comorbidities and a lower proportion of Asian patients. The CDS-use group had a higher proportion of tests ordered during the winter and spring months and during the daytime (8am-2pm). The CDS-use group had a higher percentage of residents as the ordering provider. Providers used the tool 39%, 33% and 22% of the time for residents, attendings and physician assistants respectively (p<0.001).

Table 1.

Baseline Characteristics CDSUse and CDS-Dismissal Groups

Variable	CDS-Use (N=2568)	CDS-Dismissal (N=4799)	P-value
Age, Mean±SD	59.03±18.47	58.55±18.80	0.2972
Total Comorbidities, Mean±SD	2.34±2.90	1.95±2.66	<0.0001
Gender Male, N (%)	850 (33.10%)	1590 (33.13%)	0.98
Race, N (%)			<0.0001
African American	693 (26.99%)	1294 (26.96%)
Asian	187 (7.28%)	504 (10.50%)
White	1270 (49.45%)	2226 (46.38%)
Other	418 (16.28%)	775 (16.15%)
Ethnicity, N (%)			0.8162
Hispanic or Latino	231 (9.00%)	426 (8.88%)
Not Hispanic or Latino	2269 (88.36%)	4257 (88.71%)
Unknown	68 (2.65%)	116 (2.42%)
Total ED Visit Crowded, N (%)	716 (27.88%)	1227 (25.57%)	0.0317
Visit Quarter, N (%)			<0.0001
Jan,Feb,March (Winter)	711 (27.69%)	1139 (23.73%)
Apr,May,June (Spring)	756 (29.44%)	1144 (23.84%)
July,Aug,Sep (Summer)	625 (24.34%)	1268 (26.42%)
Oct,Nov,Dec (Fall)	476 (18.54%)	1248 (26.01%)
Order Time, N (%)			0.0370
8pm-1:59am	657 (25.58%)	1262 (26.30%)
2am-7:59am	229 (8.92%)	491 (10.23%)
8am-1:59pm	705 (27.45%)	1185 (24.69%)
2pm-7:59pm	977 (38.05%)	1861 (38.78%)
Provider Type, N (%)			<0.0001
Attending	869 (33.84%)	1748 (36.42%)
Physician Assistant	195 (7.59%)	710 (14.79%)
Resident	1504 (58.57%)	2341 (48.78%)

A logistic regression model was created including patient age, gender, race, number of comorbidities, provider type, ED crowded or not and time and season of visit. (Table 2) The odds of tool use were lower with each additional year in patient age (OR 0.997, 95%CI 0.994-1.000, p=0.04), lower for Asian patients compared to white ones (OR 0.664, 95%CI 0.552-0.800, p <0.001), higher with each additional comorbidity (OR 1.056, 95%CI 1.036-1.075, p<0.001), lower for tests done in the summer (OR 0.775, 95%CI 0.676-0.887, p=0.0002) and fall months (OR 0.599, 95%CI 0.518-0.692, p<0.001) as compared to the winter months, lower in the late afternoon (OR 0.855, 95% CI 0.756, 0.968), evening (OR 0.843, 95% CI 0.736, 0.966) and overnight (OR 0.732, 95% CI 0.607-0.882, p=0.01) as compared to day time and lower for Attendings (OR 0.765, 95%CI 0.689-0.851, p<0.001) and physician assistants (OR 0.406, 95%CI 0.341-0.483, p<0.001) as compared to residents.

Table 2.

Odds of Tool Use by Baseline Characteristics

Variable	Level	Odds ratio (95% CI)	p-value
Age		0.997 (0.994, 1.000)	0.04
Gender	Female vs. Male	1.047 (0.943, 1.163)	0.39
Race	Asian vs White	0.664 (0.552, 0.800)	0.0001
	Black vs. White	0.934 (0.827, 1.055)	0.27
	Other vs. White	0.941 (0.808, 1.097)	0.44
Comorbidities		1.056 (1.036, 1.075)	0.0001
Ethnicity	Not Hispanic nor Latino vs Hispanic or Latino	1.006 (0.832, 1.216)	0.95
ED crowdedness	crowded vs not crowded	1.056 (0.945, 1.180)	0.34
BMI		0.934 (0.83, 1.05)	0.25
Quarter	Apr,May,June (Spring) vs Jan,Feb,March (Winter)	1.054 (0.922, 1.204)	0.44
	July,Aug,Sep (Summer) vs Jan,Feb,March (Winter)	0.775 (0.676, 0.887)	0.0002
	Oct,Nov,Dec (Fall) vs Jan,Feb,March (Winter)	0.599 (0.518, 0.692)	0.0001
Order time	2pm-7:59pm vs 8am-1:59pm	0.8554 (0.756, 0.968)	0.013
	8pm-1:59am vs 8am-1:59pm	0.8436 (0.736, 0.9669)	0.014
	2am-7:59am vs 8am-1:59pm	0.732 (0.607, 0.882)	0.001
Provider Type	Attending vs Resident	0.765 (0.689, 0.851)	0.0001
	Physician Assistant vs Resident	0.406 (0.341, 0.483)	0.0001

CTPA Yield After Propensity Score Matching

A total of 2552 matched pairs of CDS-use and CDS-dismissal testing events were formed. Standardized differences between matched pairs were less than 10%. CTPA yield was 11.99% in the CDS-use group and 8.70% in the CDS-dismissal group (p=0.0002). (Table 3) CTPA yield in the CDS-use group was higher for all provider types. CTPA yields were 7.2%, 9% and 9.42% for attendings, residents and physician assistants respectively which represented a 56.5% (p = 0.0046), 16.7% (p = 0.7283) and 38.7% (p=0.0029) increase when compared to the CDS-dismissal group. For the resident and attending provider types these represented statistically significant increases in CTPA yield as there was enough power to detect a difference (> 80% assuming a significant level of 0.05). However, because of the small number of physician assistants there was not enough power to detect a difference in CTPA yield by group…

Table 3.

CTPA Yield in CDS-Use vs. CDS-Dismissal Groups, Propensity Score Matched

	CTPA yield CDS-Dismissal	CTPA yield CDS-Use	CTPA yield Increase	p-value	# of pairs
All Users	8.70%	11.99%	37.8%	0.0002	2552
Attendings	7.20%	11.27%	56.5%	0.0046	861
Physician Assistants	9.42%	10.99%	16.7%	0.7283	191
Residents	9.00%	12.48%	38.7%	0.0029	1466

DISCUSSION

Our CTPA yields for the CDS-use and CDS-dismissal groups fall in range with recent studies with CTPA yields from 6-15%. [16, 42, 49, 50] In our propensity score matched sample we found a significant increase in CTPA yield for CTPA testing done with CDS use at 11.99% compared to when the tool was dismissed at 8.7% (38% increase). Meta-analysis of the impact of the use of Wells’ Criteria for PE found similar results with yields improving from 9% to 12% (33% increase).[31, 32]

A recent study found that users of a similar tool had a CTPA yield of 11.2% compared to 4.2% in tool non-users (260% increase).[42] That study controlled for patient age, sex and three elements of the Well’s Criteria (history of venous thromboembolism or malignancy and immobilization). In our current study we reproduced these findings to demonstrate that differences in CTPA yield found when CDS is used versus dismissed remain after propensity score matching for various additional patient factors including comorbidities and provider type, ED crowdedness and test time and season of the year.

Attendings showed the greatest increase in CTPA yield with tool use (7.20% to 11.27%, Δ 56.5%, p = 0.046) Attendings showed a larger increase in yield with tool use than residents. This demonstrates that although residents are more likely to use the tool than attendings, both of these groups potentially gain from tool use. These findings are somewhat counterintuitive as we would expect CDS to have the greatest effect on trainees who are still learning to develop their gestalt for pre-test probability of PE. The differences in yield for are additionally difficult to interpret as resident providers always work with an attending physician at these intuitions. Differences in yield may reflect differences in typical practice when attendings physicians work alone vs. with trainees.

We found as well that providers were less likely to use the tool with Asian patients. The reason for this is not clear. Previous direct comparison studies have found that Asians have lower rates of venous thromboembolism as compared to other races, independent of BMI.[51–54] This finding may represent a health disparity where Asian patients are being significantly over-tested for PE, with more unnecessary exposure to radiation and lower CTPA yields.[55] Lower use of CDS for these patients would likely contribute to this disparity.. Further research is needed to replicate these novel findings.

We found as well lower use in the summer and fall months and during the overnight shift. The reason for lower odds of tool use in the summer and fall months is also unclear but is likely related to the start of new trainees and attendings during the summer and fall of each year. The start of new trainees and attendings may add more stress for all providers making it more difficult to make time to use CDS at the start of the academic year. Lower odds of tool use during the overnight shift is likely explained by fatigue. These results fit into literature demonstrating that provider’s decisions are subject variability based on emotional and psychological factors. For example, providers are more likely to prescribe antibiotics during the later hours of their clinic sessions. [56]

Our study points to the potential benefit of CDS use for the formalized calculation of pre-test probability of PE before CTPA ordering. We estimate that 1,422 CTPA scans could have been avoided if all tests had been ordered with the use of the CDS tool. This is assuming a CTPA yield of 11.99% in the CDS-dismissal group and no missed PEs, which has been documented with the use of this clinical prediction rule.[31, 32] We are aware that in many cases clinicians dismiss CDS tools similar to this one with good reason,[57] however the balance has shifted too far towards dismissal. Increased CDS use would mean fewer instances of radiation induced malignancy, contrast induced nephropathy and the costly follow up of incidental findings. Further research is needed to discover successful strategies to increase provider use of these important tools.

LIMITATIONS

Our study was performed at two Emergency departments limiting generalizability. Our method for calculating CTPA yield in the ED depends on an inpatient diagnosis of PE. In the rare, but potentially growing, number of cases in which a patient is diagnosed with PE, but discharged directly from the ED, our methodology would lead us to interpret this situation as a negative CTPA, falsely decreasing CTPA yield. This was a rare occurrence however in our validation study. [46]

TAKE-HOME POINTS.

• A total of 7,376 studies were evaluated. Overall, the CDS tool was used 35% of the time.

• The odds of CDS tool use were lower for older patients and Asian patients and higher for patients with more comorbidities.

• Residents were more likely to use the tool than either physician assistants or attendings.

• After propensity score matching to address for selection bias in tool use, the diagnostic yield for testing was 11.99% when the CDS tool was used and 8.7% when the tool was dismissed.

• Diagnostic yield was higher for all provider types when the tool was used.