Optimizing Clinical Decision Support in the Electronic Health Record

Summary

Objective

Adoption of clinical decision support (CDS) tools by clinicians is often limited by workflow barriers. We sought to assess characteristics associated with clinician use of an electronic health record-embedded clinical decision support system (CDSS).

Methods

In a prospective study on emergency department (ED) activation of a CDSS tool across 14 hospitals between 9/1/14 to 4/30/15, the CDSS was deployed at 10 active sites with an on-site champion, education sessions, iterative feedback, and up to 3 gift cards/clinician as an incentive. The tool was also deployed at 4 passive sites that received only an introductory educational session. Activation of the CDSS – which calculated the Pulmonary Embolism Severity Index (PESI) score and provided guidance – and associated clinical data were collected prospectively. We used multivariable logistic regression with random effects at provider/facility levels to assess the association between activation of the CDSS tool and characteristics at: 1) patient level (PESI score), 2) provider level (demographics and clinical load at time of activation opportunity), and 3) facility level (active vs. passive site, facility ED volume, and ED acuity at time of activation opportunity).

Results

Out of 662 eligible patient encounters, the CDSS was activated in 55%: active sites: 68% (346/512); passive sites 13% (20/150). In bivariate analysis, active sites had an increase in activation rates based on the number of prior gift cards the physician had received (96% if 3 prior cards versus 60% if 0, p<0.0001). At passive sites, physicians < age 40 had higher rates of activation (p=0.03). In multivariable analysis, active site status, low ED volume at the time of diagnosis and PESI scores I or II (compared to III or higher) were associated with higher likelihood of CDSS activation.

Conclusions

Performing on-site tool promotion significantly increased odds of CDSS activation. Optimizing CDSS adoption requires active education.

1. Background and Significance

The goal of knowledge translation (KT) is to close the gap between proven science and real-time care delivery. Historically, the adoption of new evidence into clinical practice has lagged a decade or more – with variable uptake across physicians and settings [1, 2]. While KT, and implementation science more generally, are relatively new fields of emphasis and study, previous work has identified barriers to effective KT as well as best practices for implementation [3].

For example, Davidoff has stressed the importance of concentrating not only on the diffusion of new validated techniques and treatments into practice but also on the “undiffusion” of previous practice – especially amongst those physicians who may be prone to suffering from inertia when it comes to changing their practice [4]. Michie has described a behavior change wheel framework for identifying well-suited interventions (e.g., education, environmental context, etc.) to combat barriers to implementation of knowledge translation tools [5]. Diner has described four key components to promote knowledge translation: acceptance, application, ability, and recall [6]. Recently, Carpenter and Milne further elucidated five “leaks” of effective KT at the provider level as well as two at the patient/family level (agreement and adherence) [6, 7].

Effective KT can be even more challenging in an unpredictable and fast-moving environment such as the Emergency Department (ED). Here, the pace of care delivery and other workflow barriers can slow the diffusion of new knowledge into practice – especially if implemented in an ad hoc fashion.

To combat these challenges, an emergency medicine (EM) consensus conference recommended the following: building collaborative networks of knowledge translation and emergency medicine researchers, undertaking a multidisciplinary research imperative, defining what knowledge is ready for translation, and building conceptual and theoretical frameworks for knowledge translation research and projects [8].

Others have suggested that knowledge translation may be best accomplished by integrating clinical decision support (CDS) into the clinician-based computerized workflow and by providing recommendations rather than assessments [9, 10]. However, prior attempts at integrating CDS into the electronic health record (EHR) have been met with variable results, including inconsistent degrees of physician receptiveness and uptake [11–13]. This study was motivated by the goal of evaluating specific predictors of emergency physician uptake of an EHR-based CDS system (CDSS).

2. Objectives

We set out to assess the effectiveness of clinician adoption of a CDSS for site-of-care decision making for ED patients with acute pulmonary embolism (PE) and to identify characteristics associated with increased likelihood of tool use. We hypothesized that CDSS activation would be significantly higher at active implementation sites (those with an on-site clinical champion and a structured incentives program) compared with passive sites (those with neither champion nor incentives), and that activation rates would be adversely affected by higher clinical loads – both on the individual provider and facility level.

3. Methods

3.1. Setting

Kaiser Permanente Northern California (KPNC) is a large integrated health care delivery system that provides comprehensive medical care for more than 3.9 million members at medical centers across the region. Medical center EDs are staffed by more than 500 salaried (board-certified or eligible) emergency physicians and have an annual ED visit census of approximately one million visits a year. Most EDs in our system employ an automated approach to assignment of patients to ED providers [14]. KPNC is supported by a comprehensive inpatient EHR (Epic, Verona, WI) fully deployed in 2010. This investigation utilized CDS provided by an external Web Service-based clinical decision support service and accessible via an ED Navigator button within each patient’s record. Our setting has been well described previously [15–18].

3.2. Interventions

We prospectively studied rates and characteristics of appropriate ED activation of our CDSS across 14 EDs within KPNC between 9/01/2014 and 4/30/2015. The CDSS, intended to be activated after the diagnosis of acute PE in ED patients age ≥ 18, calculated the eleven-variable Pulmonary Embolism Severity Index (PESI) score for assistance in the site-of-care disposition decision [19, 20]. The tool imported 10 of the 11 PESI variables via Web Services application from ED vital sign data and Problem List diagnoses in the EHR. The CDSS was launched by the user with one click from the Epic ED Navigator menu and loaded patient-specific data in less than three seconds. Clinicians were required to assess and record one of eleven PESI variables (altered mental status) and confirm the accuracy of the other data points. After submission, the tool provided a table with PESI stratification data and risk profiles adopted from the literature. Clinicians were also provided with a list of additional reasons to consider admission in patients with low PESI scores (Classes I & II). The tool was assistive rather than directive, and clinicians were not audited to discern compliance with recommendations. The development and technical details of the tool followed defined principles of effective CDS and practice feedback – such as minimizing cognitive load for end-users [7, 21]. Snapshots of the tool are shown in supplemetary online Appendices 1 and 2.

Our CDSS was deployed at 14 sites across KPNC, and in a pragmatic interventional study design, 10 sites were designated as “active” implementation sites and four were assigned to a “passive” arm. Site designation was on a convenience basis, dependent on the availability of an on-site champion (a staff emergency physician). Ongoing structured promotion included at least three brief educational reviews throughout the study period, monthly enrollment reports comparing each of the intervention sites with each other, iterative physician-specific feedback on every eligible case, and incentivization (a small gift card) for the first three uses of the CDSS. Passive implementation sites received only an introductory educational session and did not receive iterative feedback or gift cards.

Patient-level clinical data – including demographics, PESI variables and score, and ED disposition – were automatically collected from the EHR and transmitted to an internal data center prospectively. Eligibility and missed activations were assessed through weekly retrospective audits drawn from health system electronic data sources and reviewed by a study investigator (DRV). Supplemental patient-level clinical data were extracted from existing health system databases and used discharge diagnosis (ICD-9) codes as in prior work [18, 22, 23]. Electronic data sources are consistent in their definitions across medical centers. Chart validation followed published best practices with the exception that abstractors were not blinded to the larger study hypotheses (which were not based on chart review data) [24].

Inappropriate and missed activations were identified via 100% audit of enrollments and retrospective review of all ED patients meeting inclusion criteria above.

3.3. Outcomes

The primary outcome of this study was appropriate activation of the CDSS. Impact of the tool, in terms of disposition decisions, is being assessed separately. Appropriate activation was defined as completion of the tool in an eligible patient (age 18 or older with a diagnosis of acute PE) in the ED by a treating clinician. Inclusion criteria for retrospective auditing of appropriate activation were: KPNC patients with an ED or inpatient discharge diagnosis (primary or not) of non-gravid PE (ICD-9 codes: 415.11, 415.13, 415.19, 673.20, 673.21, 673.22, 673.24) who underwent a venous thromboembolism imaging study during the ED visit or in the 12 hours preceding ED admission (CPT Codes: 71275, 71260, 71270, 78579, 78580, 78582, 78584, 78585, 78586, 78587, 78588, 78591, 78593, 78594, 75746, 71555, 93970, 93971). Patients were considered ineligible if they had a deep vein thrombosis or PE diagnosis accompanied by radiographic imaging in the prior 30 days.

3.4. Methods and Measurements

In addition to patient level variables, we collected ED provider and facility variables as follows: 1) provider: ED CDSS activating provider age, gender, employment tenure with the medical group, and clinical load at the time of activation opportunity (defined as number of ED patients actively treated by or assigned to the provider at the time that PE imaging results were available), and 2) facility: active versus passive status of facility, ED facility volume at the time imaging results were available (based on comparison to average three-hour block ED census levels), and the presence or absence of an Emergency Severity Index level 1 acuity patient in the department at the time of PE imaging results [25].

At active sites we also collected data on the number of gift cards that the activating physician had received at the time of the activation opportunity (0 to a maximum of 3).

3.5. Analysis

We used bivariate comparisons and multivariable logistic regression with random effects at provider and facility levels to assess the association between appropriate activation of the CDSS and characteristics at the patient, provider and facility levels. For bivariate comparisons, chi-square tests were used and statistical significance was defined as a two-sided p-value <0.05.

Our multivariable logistic regression model included variables with p-value <0.2 in bivariate comparisons and not expected to be collinear (e.g., age and employment tenure). Incentive receipt history was excluded from the models as it was only applicable to active sites. We used Stata SE 13.1 (StataCorp LP, College Station, TX) for all statistical analyses.

4. Results

Study data sources and descriptive CDSS activation results are shown in ► Figure 1. Out of 662 eligible patient encounters, the CDSS activation rate was 55%. At active sites, the tool was activated 68% of the time (346/512) and at passive sites 13% (20/150).

Fig. 1.

Study Flow Diagram. Facility PE volume is defined as the relative percent of CDS-eligible cases at treating facility on day of enrollment. Facility Volume refers to the ED patient volume in the facility at the time of imaging result. Facility Acuity refers to the number of ED patients with an Emergency Severity Index of 1 present at the time of imaging result. PESI: Pulmonary Embolism Severity Index, PE: Pulmonary Embolism.

Comparison of active versus passive facility characteristics is shown in ► Table 1. Prior to CDSS deployment, mean PE hospitalization rates were 83.7% (range 62.2 to 100%) at active sites and 87.7% (range 79.3 to 96.0%) at passive sites. Detailed descriptive characteristics of activations versus non-activations are shown in ► Table 2.

Table 1.

Descriptive Characteristics of Facilities (Jan-Aug 2014)

Legend: ^*Rates prior to implementation of CDSS. PE: pulmonary embolism. ED: emergency department. ICU: intensive care unit. CT: computed tomography CDA; Clinical Decision Area. Association with a residency program refers to whether residents rotate through the ED. Hospitalization rate for PE refers to the percentage of PE patients diagnosed with PE in the ED between January 2014 to August 2014 who were then admitted to the hospital (this excludes patients admitted to a CDA and then discharged home).

Facility	#of ED beds	#of-Hosp. beds	CDA	Annual ED census	# of Full-time emergency physicians	# of ICU beds	Associated with a residency program	Trauma designation	#of CT	Acute PE Diagnosis/1000 adult ED visits (age>18)*	Hospitalization with Acute PE (% with admit)*
Active 1	48	323	Yes	56,460	29	20	Yes	No	2	1.6	83.0%
Active 2	18	50	No	46,207	23	8	Yes	No	1	0.8	100.0%
Active 3	61	166	Yes	89,275	45	20	No	No	2	2.0	62.2%
Active 4	36	288	No	89,691	44	32	Yes	No	3	1.0	84.3%
Active 5	18	102	No	26,145	15	12	No	No	2	1.8	85.7%
Active 6	41	161	Yes	103,752	42	20	Yes	Level II	2	1.1	86.4%
Active 7	19	120	No	35,689	21	20	No	No	2	1.5	76.7%
Active 8	46	327	Yes	67,132	45	30	Yes	No	3	1.9	78.3%
Active 9	16	116	Yes	32,937	28	10	No	No	1	1.6	88.4%
Active 10	40	264	Yes	30,294	27	22	No	No	2	1.2	91.7%
Passive 1	24	147	Yes	30,795	15	20	No	No	2	1.7	96.0%
Passive 2	26	224	No	36,209	18	32	Yes	No	2	1.9	85.0%
Passive 3	35	118	Yes	51,162	22	20	No	No	1	1.9	79.3%
Passive 4	28	190	Yes	57,107	15	20	No	No	2	1.0	90.6%

Table 2.

Descriptive Characteristics of Study Sample

Legend: Low ED Volume defined as lower or equal to average facility-specific ED census for three-hour time block during which the PE imaging results were obtained. Acuity1 patient is any ED patient with an Emergency Severity Index of 1 present at time of imaging result. KPNC Tenure is physician tenure at time of visit with the medical group. Clinical load is defined by the number of patients currently in the ED and electronically assigned to the enrolling provider. Incentive was a coffee gift card. PESI: Pulmonary Embolism Severity Index, KPNC: Kaiser Permanente Northern California, CDSS: Clinical decision support system, Pt(s): patient(s).

Characteristics			Total	CDSS Not Activated	CDSS Activated
			662 (100.0)	296 (100.0)	366 (100.0)
			Facility	Site	Passive	150 (22.7)	130 (43.9)	20 (5.5)
Active	512 (77.3)	166 (56.1)			346 (94.5)
Low ED Volume	No	334 (50.5)		166 (56.1)	168 (45.9)
	Yes	328 (49.6)		130 (43.9)	198 (54.1)
Presence Acuity1 Pt.	No	538 (81.3)		259 (87.5)	279 (76.2)
	Yes	124 (18.7)		37 (12.5)	87 (23.8)
Patient	Insurance	Commercial	268 (40.5)	114 (38.5)	154 (42.1)
		Medicaid	29 (4.4)	11 (3.7)	18 (4.9)
		Medicare	316 (47.7)	151 (51.0)	165 (45.1)
		Unknown	49 (7.4)	20 (6.8)	29 (7.9)
	PESI class	I, II	273 (41.2)	107 (36.2)	166 (45.4)
		III, I V, V	389 (58.8)	189 (63.9)	200 (54.6)
Provider	Gender	Female	252 (38.1)	104 (35.1)	148 (40.4)
		Male	410 (61.9)	192 (64.9)	218 (59.6)
	Age	<40	288 (43.5)	107 (36.2)	181 (49.5)
		40+	374 (56.5)	189 (63.9)	185 (50.6)
	KPNC Tenure	<5 yrs	451 (68.1)	189 (63.9)	262 (71.6)
		5+ yrs	211 (31.9)	107 (36.2)	104 (28.4)
	Clinical Load (Pts)	<5	362 (54.7)	151 (51.0)	211 (57.7)
		5+	300 (45.3)	145 (49.0)	155 (42.4)
	Incentives Received	0	325 (49.1)	131 (44.3)	194 (53.0)
		1	116 (17.5)	26 (8.8)	90 (24.6)
		2	48 (7.3)	8 (2.7)	40 (10.9)
		3	23 (3.5)	1 (0.3)	22 (6.0)
		N/A	150 (22.7)	130 (43.9)	20 (5.5)

4.1. Main results

► Table 3 demonstrates a comparison of descriptive characteristics of active versus passive site enrollments. In bivariate analysis, at active sites we observed an increase in activation rates based on the number of prior gift cards the treating physician had received (96% if 3 prior gift cards versus 60% if 0, p<0.0001). We also found an association at both active and passive sites between lower than average ED volume at the time of imaging result availability and rates of enrollment (active sites, p=0.05, passive sites, p=0.03). At passive sites, we observed that physicians < age 40 had higher rates of activation (p = 0.03).

Table 3.

Descriptive Characteristics of Active & Passive Sites

Legend: Low ED Volume defined as lower or equal to average facility-specific ED census for three-hour time block during which the PE imaging results were obtained. Acuity1 patient is any ED patient with an Emergency Severity Index of 1 present at time of imaging result. KPNC Tenure is physician employment tenure with KPNC at time of visit. Clinical load is defined by the number of patients currently in the ED and electronically assigned to the enrolling provider. Incentive was a coffee gift card. PESI: Pulmonary Embolism Severity Index, KPNC: Kaiser Permanente Northern California, CDSS: Clinical decision support system, Pt(s): patient(s).

Characteristics			Active Sites			Passive Sites
			Total	CDSS – no	CDSS – yes	Total	CDSS – no	CDSS – yes
		N	512	166	346	150	130	20
Facility	Low ED Volume	No	49.8%	56.0%	46.8%	52.7%	56.2%	30.0%
		Yes	50.2%	44.0%	53.2%	47.3%	43.9%	70.0%
	Presence Acuity1	No	76.8%	80.1%	75.1%	96.7%	96.9%	95.0%
		Yes	23.2%	19.9%	24.9%	3.3%	3.1%	5.0%
Patient	Insurance	Commercial	40.4%	37.4%	41.9%	40.7%	40.0%	45.0%
		Medicaid	4.9%	4.2%	5.2%	2.7%	3.1%	0.0%
		Medicare	46.7%	51.2%	44.5%	51.3%	50.8%	55.0%
		Unknown	8.0%	7.2%	8.4%	5.3%	6.2%	0.0%
	PESI Class	I, II	43.8%	40.4%	45.4%	32.7%	30.8%	45.0%
		III, I V, V	56.3%	59.6%	54.6%	67.3%	69.2%	55.0%
Provider	Gender	Female	39.1%	37.4%	39.9%	34.7%	32.3%	50.0%
		Male	60.9%	62.7%	60.1%	65.3%	67.7%	50.0%
	Age	<40	47.7%	44.0%	49.4%	29.3%	26.2%	50.0%
		40+	52.3%	56.0%	50.6%	70.7%	73.9%	50.0%
	KPNC Tenure	<5 yrs	71.5%	71.7%	71.4%	56.7%	53.9%	75.0%
		5+ yrs	28.5%	28.3%	28.6%	43.3%	46.2%	25.0%
	Clinical Load (Pts)	<5	56.6%	54.2%	57.8%	48.0%	46.9%	55.0%
		5+	43.4%	45.8%	42.2%	52.0%	53.1%	45.0%
	Incentives Received	0	63.5%	78.9%	56.1%	N/A	N/A	N/A
		1	22.7%	15.7%	26.0%	N/A	N/A	N/A
		2	9.4%	4.8%	11.6%	N/A	N/A	N/A
		3	4.5%	0.6%	6.4%	N/A	N/A	N/A

Our model results are shown in ► Table 4. Predictors of higher likelihood of CDSS activation were active site status (odds ratio [OR] 31.1, 95% CI 11.7, 83.0) and low ED volume at the time of PE diagnosis (OR 1.7, 95% CI 1.1, 2.7). Predictor of higher likelihood of CDSS activation included patient with PESI scores I or II – compared to III or higher (OR 1.7, 95% CI: 1.1, 2.5).

Table 4.

Multivariable analysis results: characteristics associated with CDSS activation

Legend: Model: logistic regression, with random effect at provider and facility levels. Low ED Volume defined as lower or equal to average facility-specific ED census for three-hour time block during which the PE imaging results were obtained. Acuity1 Patient is any ED patient with an Emergency Severity Index of 1 present at time of imaging result. Clinical load is defined by the number of patients currently in the ED and electronically assigned to the enrolling provider. PESI: Pulmonary Embolism Severity Index, OR: Odds Ratio, ED: Emergency Department

	Characteristics		Reference	OR	95% CI
Facility	Site	Active	Passive	31.1	11.7	83.0
	Low ED Volume	Yes	No	1.7	1.1	2.7
	Acuity 1 Patient	Yes	No	1.2	0.7	2.3
Provider	Sex	Female	Male	0.9	0.5	1.6
	Age	40+	<40	0.6	0.4	1.1
	Clinical Load (Pts)	5+	<5	0.9	0.6	1.4
Patient	PESI Class	I, II	III+	1.7	1.1	2.5

4.2. Limitations

Our study should be interpreted in the context of certain limitations. First, this was a pragmatic cluster interventional study. While there is a great deal of homogeneity across study settings (e.g., study EDs had similar staffing and system characteristics, shared the same EHR and had mostly similar baseline acute PE diagnosis and hospitalization rates), we cannot exclude facility-level differences that confounded our results. However, as shown in ► Table 1, we did not identify any a priori fundamental differences in active versus passive facilities. Our study utilized a CDSS meant to be activated downstream in the ED process – after the diagnosis of PE was established – and thus uptake rates of our particular tool may not translate to those designed for other temporal stages in the ED throughput process (e.g., prior to imaging). In the same vein, our CDSS did not employ an automatic EHR trigger (e.g., a Best Practice Alert) to drive clinicians to the CDSS as there was not a clean and time-appropriate trigger available for the specific clinical situation. However other ED CDSS studies with upstream targets may likely benefit from such triggers. Similarly, uptake of CDS tools will certainly depend on the nature of the support provided, the ease of manual calculation, and the availability of alternate means of assistance (e.g., MDCalc [MD Aware, LLC, San Francisco, CA]).

Finally, effectively recruiting and maintaining a structured on-site champion requires time and resource allocation to achieve optimum performance. For example, we know that chiefs at passive sites did not recruit and employ clinical champions to promote and audit use of the CDSS, even though there was nothing in our study design that prevented them from doing so. Preliminary analysis of our impact study of the disposition effect of our CDSS suggests that the activation of the tool resulted in increased rates of safe discharge home from the ED for low risk PE (unpublished data). However, we have not yet attempted to evaluate the cost-effectiveness of this approach to KT, though we suspect it performs favorably compared to other approaches such as the use of non-physician quality specialists to retrospectively review compliance with guidelines. It may be that there are particular pieces (e.g., iterative feedback on missed eligible cases) of the active site bundle that are more effective than others and these could be identified and implemented.

5. Discussion

In this study we found that performing active on-site tool promotion significantly increased odds of ED providers using a CDSS for site-of-care decision making for acute PE. We also observed that ED volume was inversely associated with likelihood of CDSS activation, supporting the intuitive notion that busier clinicians are more likely to make a gestalt decision rather than seeking out CDS tools. Another confirmation of an intuitive hypothesis in our multivariable analysis was that physicians were more likely to activate the tool when seeing patients with low PESI scores (I or II), than those with scores of III or above. We surmise that in patients with higher scores, clinicians would sometimes conclude that these are “slam-dunk” admissions without the need for decision support tools. Finally, in subgroup analysis, we found that at active sites, CDSS use increased steadily with prior incentive receipt and that at passive sites, younger physicians were more likely to use the tool than older ones.

Peer comparison and iterative feedback and audits have been shown to help drive knowledge translation and performance improvement and are acknowledged as best practices [26–28]. A Cochrane review on the topic reported that this type of feedback works best when it is provided repeatedly, through multiple modalities (e.g., verbally and in person), and by a person who is a supervisor or peer [27]. In this study, our active site study champion met each of these criteria, and we attribute the high uptake of the CDSS at active sites in large part to this on-site dynamic. Of course, this type of feedback loop can be difficult to sustain over time, so it is likely most effective in situations like these, where there is an opportunity to translate validated evidence into practice with the expectation that once translation has occurred there will be an opportunity for sustained practice change.

Conversely, there is peril associated with implementing CDSSs without proper education and feedback. Take for example, the evidence suggesting that adverse drug reaction alerts do little to change clinician-prescribing behavior, such as warnings about opioid prescriptions [29]. Consider as well the potentially deleterious effect of the number of “clicks” that EPs working in an EHR-enabled environment face each shift [30]. Other work suggests that CDSSs are more effective if adapted based on human factor analysis [31]. While there has been significant backlash regarding the benefits of the EHR, we know that certain aspects of it are effective – these include well-designed assistive order sets and condition-specific decision support tools [18, 32]. We also know that clinicians familiar with using the EHR are receptive to CDSSs through multiple means [33]. The results of this study suggest that active KT techniques may help amplify these beneficial attributes and support the contention that the EHR can be used to deliver evidence-based information at the point of care, thus improving workflow, “reducing care variability, and encouraging adherence to best practices” [34].

Our study also suggests that clinicians are less likely to use CDSSs when the departmental clinical volume is greater. This finding has face validity, as we know that there is a greater tendency to cut corners and/or practice via gestalt when busy. Other investigations have associated ED crowding with higher admission rates for transient ischemic attack and stroke [35]. As our CDSS addresses ED disposition decisions, we expect that metrics of ED crowding may also be associated with higher admission rates for patients with acute PE – low-risk or otherwise.

In the sub-analysis of active sites, we saw, as expected, a strong correlation of CDSS use with prior receipt of an enrollment incentive. This trend continued even once clinicians had reached their predefined maximum of three enrollment incentives, suggesting that an initial incentive, though finite, could be successful in establishing a sustainable behavior pattern. Prior work on the effects of financial incentives on physician performance is mixed [36]; however, our experience is that a small enrollment incentive can reliably and accurately facilitate initial CDSS uptake [23, 37]. Bolstered with a 100% audit process, we were able to police the incentive system to prevent clinicians from “gaming it” (e.g., enrolling ineligible patients in order to receive an incentive). By making the data collection simple with most variables pre-selected from the EHR, we were able to collect complete data without any missing data points. As such, we see a system of limited physician incentivization as an effective means of observational data collection, one that has a number of advantages over research assistant-driven methods.

Our study also identifies possible demographic barriers to CDSSs; at passive sites, physicians with longer ED practice tenure were less likely to activate the tool than younger physicians. These findings are consistent with previous survey studies demonstrating that younger physicians and those with fewer years since medical school graduation were more likely to be aware of and use specific radiograph-utilization based CDS [38, 39]. Further study should focus on how to offer decision support to clinicians with different characteristics in a manner that helps them make the best possible use of it. This concept of “personalized” CDSSs is not yet ready for clinical implementation, but the theoretical and technical underpinnings are being laid [40].

6. Conclusions

We found that lower facility ED volume status and performing active on-site tool promotion significantly increased odds of CDSS activation. Optimizing CDSS adoption requires active education and iterative feedback from an on-site clinical champion.

7. Clinical Relevance

There are numerous barriers that may prevent clinicians from changing their practice in the face of new and validated knowledge. CDSS tools that are supported by active and iterative feedback can help clinicians become more confident in the new best practice and assist to un-diffuse prior practice. Such support may be most important in situations where the clinician is the busiest – times when they are most likely to fall back on heuristics and gestalt – and this presents an opportunity to implement systems that help spin the behavior change wheel.

Funding Statement

Funding: This study was supported by a grant from The Permanente Federation’s Sidney R. Garfield Memorial Fund, Oakland, California