Does Clinical Decision Support Reduce Unwarranted Variation in Yield of CT Pulmonary Angiogram?

Abstract

OBJECTIVE

The study objective was to determine whether previously documented effects of clinical decision support on computed tomography for pulmonary embolism in the emergency department (ie, decreased use and increased yield) are due to a decrease in unwarranted variation. We evaluated clinical decision support effect on intra- and inter-physician variability in the yield of pulmonary embolism computed tomography (PE-CT) in this setting.

METHODS

The study was performed in an academic adult medical center emergency department with 60,000 annual visits. We enrolled all patients who had PE-CT performed 18 months pre- and post-clinical decision support implementation. Intra- and inter-physician variability in yield (% PE-CT positive for acute pulmonary embolism) were assessed. Yield variability was measured using logistic regression accounting for patient characteristics.

RESULTS

A total of 1542 PE-CT scans were performed before clinical decision support, and 1349 PE-CT scans were performed after clinical decision support. Use of PE-CT decreased from 26.5 to 24.3 computed tomography scans/1000 patient visits after clinical decision support (P < .02); yield increased from 9.2% to 12.6% (P < .01). Crude inter-physician variability in yield ranged from 2.6% to 20.5% before clinical decision support and from 0% to 38.1% after clinical decision support. After controlling for patient characteristics, the post-clinical decision support period showed significant inter-physician variability (P < .04). Intra-physician variability was significant in 3 of the 25 physicians (P < .04), all with increased yield post-clinical decision support.

CONCLUSIONS

Overall PE-CT yield increased after clinical decision support implementation despite significant heterogeneity among physicians. Increased inter-physician variability in yield after clinical decision support was not explained by patient characteristics alone and may be due to variable physician acceptance of clinical decision support. Clinical decision support alone is unlikely to eliminate unwarranted variability, and additional strategies and interventions may be needed to help optimize acceptance of clinical decision support to maximize returns on national investments in health information technology.

Medical imaging has helped transform the practice of medicine in part by helping to diagnose and treat disease. In parallel, the use of imaging, including computed tomography (CT), has increased considerably in the past several years.¹ Although in some clinical settings high-cost imaging can indeed improve measurable outcomes,² in other clinical presentations, increased use of high-cost imaging may not result in measurably improved outcomes and thus is likely overused.³ Implications of such potential overuse include both higher costs and the possible deleterious effects of ionizing radiation.^4-7 Computerized physician order entry with embedded decision support has been proposed as an important tool to help improve appropriateness of imaging,^8-10 and its use has been associated with a significant reduction in high-cost imaging.^10-13

Although clinical decision support has become more common, has gained some measure of acceptance, and has been included in federal policy via recent meaningful use regulations,^14-16 its effect on physician behavior is not yet fully understood.^10,17-19 Our previous study demonstrated that implementation of a clinical decision support tool based on high-quality evidence in the emergency department decreased overall use and increased overall yield of CT for acute pulmonary embolism.¹¹ We hypothesized that the evidence embedded in clinical decision support helped to standardize practice and reduce unwarranted inter-physician variability. The purpose of this study was to evaluate the effect of clinical decision support on inter-physician variability in the yield of CT for pulmonary embolism in the emergency department and to determine whether individual physicians changed practice after implementation of clinical decision support by measuring intra-physician variability.

MATERIALS AND METHODS

Study Design and Setting

Institutional review board approval was obtained for this Health Insurance Portability and Accountability Act–compliant, retrospective cohort study conducted from January 1, 2006, to March 31, 2009, in the emergency department of a 793-bed, quaternary-care academic hospital. The adult-only emergency department, a Level 1 Trauma and Burn Center, and Stroke and Cerebrovascular Disease Center, receives approximately 60,000 visits annually. It hosts a 4-year emergency medicine residency, and all emergency medicine attending physicians are board eligible and certified.

Study Cohort

The study population included all patients presenting to the emergency department who underwent pulmonary embolism computed tomography (PE-CT) during the 18-month periods before and after clinical decision support implementation (the intervention). Examinations requested by physicians who were not present throughout the entire study period were excluded from the analysis (n = 504).

Intervention

The intervention consisted of a clinical decision support tool for PE-CT (based on the criteria of Wells et al²⁰) within the institution's computerized physician order entry system (Percipio; Medicalis Corp, San Francisco, Calif) implemented between June and August 2007. The clinical decision support contained real-time, patient-specific physician education alerts based on clinical information entered at the time of ordering. The clinical decision support required information about the level of clinical suspicion for pulmonary embolism (low, intermediate, or high) and the serum D-dimer level (not done, unknown, normal, or elevated). If no information was provided for these variables, the application prompted the user to complete a form (Figure 1A). Imaging requests for patients with a normal D-dimer level and intermediate or low suspicion for pulmonary embolism received this message: “Based on current evidence, as well as our experience at Brigham and Women's Hospital, diagnosing an acute pulmonary embolism by CT pulmonary angiography in lowor intermediate-risk patients with a normal D-dimer level is extremely unlikely” (Figure 1B). Imaging requests for patients with an intermediate or low level of clinical suspicion without a D-dimer received the following advice: “Measuring a D-dimer value in patients with a low or intermediate clinical suspicion of pulmonary embolism is an appropriate first step in the work-up of acute pulmonary embolism and will exclude the need for CT pulmonary angiography in some patients” (Figure 1C). At each step of the ordering process, the advice could be ignored and the examination requested. No decision support was presented if the request was deemed appropriate (Figure 1D).

Figure 1.

The clinical decision support required information about the level of clinical suspicion for pulmonary embolism (low, intermediate, or high) and the serum D-dimer level (not done, unknown, normal, or elevated). (A) If no information was provided for these variables, the application presented the user with message A and prompted them to provide additional clinical information (add indication). (B) Imaging requests for patients with a normal D-dimer level and intermediate or low suspicion for pulmonary embolism received message B. (C) Imaging requests for patients with an intermediate or low level of clinical suspicion without a D-dimer received message C. (D) No decision support was presented if the request was deemed appropriate. CT = computed tomography; CTPA = computed tomography pulmonary angiography; PE = pulmonary embolism.

Methods and Measurement

All eligible radiology reports were reviewed for the presence of acute pulmonary embolism using a computerized algorithm incorporating Natural Language Processing techniques based on the General Architecture for Text Engineering framework. The engine recognizes negation and nuances in phrases within the radiology report to determine whether the diagnosis of pulmonary embolism was present or absent. For example, although the phrase “no evidence of filling defect to suggest pulmonary embolism” contains the words “filling defect” and “pulmonary embolism,” the engine recognizes that “no evidence of” excludes this possibility. Previously validated, the engine has shown an accuracy of 97.8% when compared with manual review.¹¹

Data Collection

The radiology report and patient age and gender were obtained from the Radiology Information System and combined with patient characteristics known to increase the risk of thromboembolism (history of malignancy, history of surgery, history of thrombosis, and evidence of D-dimer elevation) obtained from the clinical data repository.²¹ Each imaging request was assigned to an attending emergency physician, and a determination was made whether it was requested by an attending physician or a supervised clinician (trainee or physician assistant). In 86 of 2891 imaging requests (2.9%), this information could not be determined. Patient gender, presence of pulmonary embolism, history of malignancy, history of surgery, history of thrombosis, D-dimer elevation, and attending-generated imaging requests were recorded as categoric variables; patient age was continuous.

Statistical Analysis

Univariate analyses were performed to determine any underlying differences in patient characteristics between the pre- and post-intervention periods. Continuous variables were assessed using the Wilcoxon rank-sum test, and proportions from categoric data were assessed with chi-square statistics. Crude inter-physician variability in yield (percentage of examinations positive for acute pulmonary embolism per physician) was assessed before and after the intervention. Crude intra-physician variability (percent difference between pre- and post-intervention of PE-CT yield per physician) also was calculated. Adjusted inter-physician variability tested whether physicians had significantly different PE-CT yield between themselves pre- and post-clinical decision support, whereas adjusted intra-physician variability assessed differences in yield for the same physician pre- and post-clinical decision support. Adjusted yield variability analyses were made using logistic regression with PE-CT yield as the outcome variable, controlling for patient demographics and clinical characteristics. Analyses were performed using JMP Pro v10 software (SAS Institute, Inc, Cary, NC).

RESULTS

Of the 113,703 emergency department visits during the study period, 2891 (2.5%) generated a PE-CT. The characteristics of patients’ pre- and post-clinical decision support implementation were similar with the exception of a decreased proportion of patients with elevated D-dimers in the post-clinical decision support period (30.6% vs 24.4% P < .001) (Table 1). A total of 25 emergency physicians were responsible for 1542 examinations during the pre-intervention period and 1349 post-clinical decision support implementations. The overall use of PE-CT decreased from 26.5 to 24.3 CT scans/1000 patient visits post-clinical decision support, a decrease of 2.2 CT scans/1000 patient visits (P < .02), whereas the overall yield of these CT scans increased from 9.2% to 12.6%, a 3.4% increase (P < .01) post-clinical decision support.

Table 1.

Univariate Analyses Comparing Distribution of Each of the Studied Variables in Relation to the Pre- and Post-clinical Decision Support Implementation Periods

	Pre-CDS	Post-CDS	P Value
Total Patients	1542	1349
Patient age, y	55.23 ± 18.1	55.07 ± 17.7	.8 (NS)
Women	1006 (65.2%)	848 (62.9%)	.18 (NS)
History of malignancy	674 (43.7%)	624 (46.3%)	.17 (NS)
History of thrombosis	177 (11.5%)	184 (13.6%)	.08 (NS)
History of surgery	172 (11.15%)	154 (11.4%)	.82 (NS)
Elevated D-dimer	472 (30.6%)	329 (24.4%)	<.001
Imaging requests entered by attending physicians	246/1474 (16.7%)	197/1331 (14.8%)	.17 (NS)

CDS = clinical decision support;NS = not significant.

Overall, 15.8% of the imaging requests were entered by attending emergency physicians themselves with individual rates varying between 0% and 30%. No significant effect on PE-CT yield was observed between imaging studies requested by attending physicians in comparison with those requested by their supervised clinicians. Crude inter-physician variability in yield ranged from 2.6% to 20.5% in the pre-clinical decision support period and from 0% to 38.1% in the post-clinical decision support period. Crude intra-physician variability in yield ranged from a 9.1% decrease to a 21.0% increase in the yield of PE-CT (Table 2). After controlling for patient characteristics, only the post-clinical decision support period showed significant inter-physician variability (P < .04). Intra-physician variability was significant in 3 of the 25 physicians (P < .04) and related to an increase in yield post-clinical decision support implementation (Table 2).

Table 2.

Pre- and Post-clinical Decision Support Crude Pulmonary Embolism Computed Tomography Yield and Interval Change per Emergency Physician

Physician ID	Change in Crude PE-CT Yield Between Pre and Post-CDS Implementation	Pre-CDS Crude PE-CT Yield	No. of Ordered Examinations in the Pre-CDS Period	Post-CDS Crude PE-CT Yield	No. of Ordered Examinations in the Post-CDS Period	Statistical Significance of Adjusted Intra- and Inter-physician Variability (P Value)*
1	–9.09%	9.09%	22	0.00%	21	.22 (NS)
2	–8.70%	8.70%	23	0.00%	14	.37 (NS)
3	–6.63%	10.98%	82	4.35%	46	.08 (NS)
4	–3.60%	11.76%	51	8.16%	49	.70 (NS)
5	–1.62%	6.38%	47	4.76%	42	.99 (NS)
6	–1.56%	13.16%	76	11.59%	69	.53 (NS)
7	–0.30%	12.50%	64	12.20%	41	.75 (NS)
8	–0.27%	11.54%	52	11.27%	71	.91 (NS)
9	1.68%	11.19%	134	12.87%	101	.65 (NS)
10	1.83%	9.71%	103	11.54%	104	.95 (NS)
11	2.00%	8.60%	93	10.61%	66	.85 (NS)
12	2.39%	10.11%	89	12.50%	96	.90 (NS)
13	2.73%	10.00%	60	12.73%	55	.70 (NS)
14	3.85%	4.35%	46	8.20%	61	.63 (NS)
15	3.94%	10.34%	29	14.29%	28	.96 (NS)
16	5.52%	12.66%	79	18.18%	44	.60 (NS)
17	5.73%	5.26%	76	10.99%	91	.25 (NS)
18	6.15%	6.35%	63	12.50%	40	.17 (NS)
19	8.35%	8.05%	87	16.39%	61	.16 (NS)
20	9.63%	5.88%	17	15.52%	58	.33 (NS)
21	11.31%	4.76%	84	16.07%	56	.27 (NS)
22	12.38%	5.26%	38	17.65%	34	.15 (NS)
23	13.31%	4.08%	49	17.39%	46	.04
24	17.58%	20.51%	39	38.10%	21	.02
25	20.97%	2.56%	39	23.53%	34	.04
All physicians	3.39%	9.21%	1542	12.60%	1349	<.01

CDS = clinical decision support; ID = identification; NS = not significant; PE-CT = pulmonary embolism computed tomography.

The P value of the adjusted intra-physician variability measures the likelihood that differences between pre- and post-CDS yield could have occurred by chance after controlling for patient specific variables. Crude and adjusted inter-physician variability is shown in the last row for all physicians.

^*Accounting for patient characteristics.

DISCUSSION

An overall increase in the yield of PE-CT in the emergency department post-clinical decision support implementation occurred despite substantial heterogeneity in yield among individual ordering providers. Indeed, we found a significant increase in inter-physician variability in yield post-clinical decision support even after controlling for important patient variables that are known to be associated with pulmonary embolism. Our results suggest that despite overall improved performance, substantial—and likely unwarranted—variability in practice persisted post-clinical decision support.

In this study, we presented real-time, brief, actionable, unambiguous, context-specific (informed by patient-specific clinical information entered by ordering providers) alerts to providers at the time of order entry based on high-quality, validated evidence (criteria by Wells et al²⁰). However, our intervention was purely educational. Providers could elect to accept, ignore, or modify their decision as to whether or not to proceed with the PE-CT with a “mouse-click” at the time of ordering. The substantial inter-physician variability in yield of PE-CT post-clinical decision support reflects the heterogeneous impact of clinical decision support on intra-physician yield. Of the 25 emergency department physicians included in the study, we found significant improvement of yield in 3, with the yield not significantly changed for the remaining 22 physicians.

The heterogeneous impact of clinical decision support may reflect varied attitudes and acceptance of the evidence presented in clinical decision support by each physician. Passive physician education is known to be an ineffective method to change physician behavior and has demonstrated inconsistent results because although some physicians accept the changes proposed by clinical decision support, others consistently dismiss them.²² Our results support the notion that the purely educational component of clinical decision support, easily ignored by providers, is unlikely to reduce unwarranted variability in practice even if based on high-quality evidence. Consequences for ignoring high-quality evidence, and other strategies such as academic detailing or benchmarking,^19,20 may be needed to optimize the acceptance of evidence presented in clinical decision support to help optimize practice.^19,23,24

Patients who underwent PE-CT in the post-clinical decision support period were less likely to have elevated D-dimers (Table 1). This finding was likely related to a noted reduction in overall D-dimer requests as opposed to a reduction in its positivity rates. This was probably a result of better characterization of the patients’ pretest probabilities in light of the reinforcement by clinical decision support, because high-risk patients do not require D-dimer testing. This finding is in alignment with previously published data^11,25 that found decreased D-dimer testing with implementation of clinical decision support. The overall decreased use and increased yield after implementation of clinical decision support also suggests improvement in the pretest determination of patients’ risks for pulmonary embolism.

To our knowledge, no prior report has assessed intra- or inter-physician variability in the setting of clinical decision support implementation for pulmonary embolism. Our findings challenge the notion that clinical decision support alone can reduce variability in physicians’ ordering patterns. Although it has been suggested that clinical decision support may improve standardization of practice,²⁶ the effect on individual physicians has not been extensively examined. The observed heterogeneous effect of clinical decision support on yield suggests that acceptance of evidence presented in clinical decision support is not homogeneous, an observation with important health policy implications because clinical decision support is a major focus of stage 2 of federal regulations for meaningful use of health information technology.^27-29 Our findings suggest that technology-focused health information technology regulations are unlikely to reduce unwarranted variation in physician behavior. Other interventions, such as academic detailing and benchmarking, may be needed to help optimize adoption of evidence offered through clinical decision support to help maximize the return on the substantial national investment in health information technology.

To date, studies have not been able to clearly determine the variables associated with physicians’ ordering practices. Physicians’ experience, age, gender, workload, and risk tolerance have been studied in other clinical circumstances and have not been linked to ordering behavior.^30,31 Further investigation may be needed to better characterize these variables so that targeted interventions to improve clinical decision-making can be developed and implemented.

Some of the key elements to enhance clinical decision support adoption by clinicians include acceptance of the clinical content, speed of the ordering process, provision of real-time recommendations rather than just assessments, integration into physicians’ workflows, easy usability, execution via simple steps to minimize data entry, and allowance for change in the direction of the action as opposed to stopping its execution.^19-33,22 Although many if not all of these variables are essential for successful implementation of clinical decision support, some authors have suggested that physicians’ acceptance of the clinical content is the most crucial step for clinical decision support adoption.³³ Our clinical decision support deployment involved not only a high-quality evidence-based guideline but also support from committed leadership. In addition, our pre-implementation planning included multidisciplinary discussions jointly coordinated between radiology and emergency medicine leadership that likely increased physicians’ acceptance of its content. Our clinical decision support application also was carefully designed to fulfill the remaining previously described functional requirements.

Study Limitations

The study was performed at a single large academic center in which many other clinical decision support interventions exist; therefore, our results may not be generalizable to other institutions. In addition, although we included the most common clinical variables associated with pulmonary embolism in our analyses of patient characteristics, we were not able to account for all potential clinical factors associated with the disease, and some rare circumstances (eg, genetically acquired thrombophilic states) may not have been accounted for. Also, our study design did not include a control group, which would have allowed us to account for external factors that may have concurrently influenced ordering behaviors. Therefore, it is possible that the effect we observed was a combination of clinical decision support with other synchronous change in clinical practice. In addition, the emergency department is a dynamic environment, and many clinicians (including physician assistants, residents, and fellows) are allowed to request an imaging study for a patient under the direct supervision of an attending emergency physician. Although these supervised clinicians have a certain level of autonomy to request studies, they are expected to consult with the attending physician before or immediately after ordering the CT studies. Our emergency department leadership has focused significant attention on the use of imaging, and attending physicians are aware that they are ultimately responsible for all imaging ordered for their patients. Yet, no significant effect on PE-CT yield was observed between imaging studies requested by attending physicians and imaging studies requested by their supervised clinicians. Moreover, the shift assignments between attending emergency physicians and their supervised clinicians follow no predictable pattern. Therefore, if any bias had been introduced it would have favored the null hypothesis, decreasing variation. Despite these limitations, we believe that these data support our observation that increased variability in yield occurred after implementation of clinical decision support.

CONCLUSIONS

Although the overall yield of PE-CT improved in the emergency department, there was a significant increase in inter-physician variability in the yield of these studies after implementation of clinical decision support. This was likely due to variable physician acceptance of the evidence offered through clinical decision support. The purely educational effect of clinical decision support is unlikely to eliminate unwarranted variability. Further investigation is needed to determine whether augmenting clinical decision support with other strategies (eg, academic detailing, benchmarking) may optimize acceptance of evidence in clinical decision support to help reduce unwarranted variation. These strategies and interventions may be needed to help optimize adoption of evidence offered through clinical decision support to help maximize the return on the substantial national investment in health information technology.

CLINICAL SIGNIFICANCE.

• Overall pulmonary embolism computed tomography yield increased after clinical decision support implementation despite significant heterogeneity among physicians.

• Increased inter-physician variability may be due to variable physician acceptance of clinical decision support.

• Clinical decision support alone is unlikely to eliminate unwarranted variability, and additional strategies and interventions may be needed to help optimize acceptance of clinical decision support to maximize returns on national investments in health information technology.