Table 4.

Approaches to overcome identified implementation barriers

Implementation barrier	Approach	Representative quote
Minimize alert firing	Threshold optimization	“We did a lot of testing to see at what threshold could we have the minimum number of alerts…we are very sensitive to alert fatigue”
	Heuristics to reduce redundant alerts	“If we alert the rapid response doctors, we won’t alert them again for the next 8 hours. Because we don’t want to be continuously sending the same alert”
	Different thresholds for different provider types	“We have an upper and lower kind of threshold, and at a lower threshold we alert the frontline team. So that would be the front line nurse and the front line provider. And at the upper threshold the plan we actually text, the platform will actually text our rapid response providers”
	Two-phase alerts	“We came up with a model that incorporates vital signs, past medical history, certain high risk factors, high risk neurological conditions, presence of a central line, sickle cell, some other things to develop an initial screening alert that’s targeting the inpatient nurse that is largely vital sign driven and then based on follow up assessments that they document and also presence or absence of some of those high risk conditions, a secondary alert would appear to the entire team”
Alert content	Concise alert messages	“Keep it as simple as possible… doctors and nurses are inundated by alerts all the time. If you expect them to read it, it is not going to happen. The alert needs to be very straightforward and specific”
	De-emphasize wordsmithing of alert	“The more clear you can be with that message the better but like changing the tense of a verb here or doing this or doing that doesn’t make any bit of a difference. I mean we’ve looked at the amount of time people spend in these alerts and it’s like a fraction of a second so it is not long enough to even notice a typo”
	Include explanations when possible	“I think it’s important for users to know why this alert went off. Now when we get an alert and it says… some indication for why this alert went off. I think that actually reduced the amount of negative feedback that we were getting”
Workflow integration	Avoid hard stops	“I think that having some acknowledgement reason [that] captures whether you agree or disagree with the alert is a bad thing”
	Ability to place orders that have not been placed	“I think at the time one of the draws was the ability to place orders…as a follow-up so if I was missing something [the tool] could say hey you are missing you know a second lactate and here is the order to place”
	Use different alerts for different locations	“Many hospitals decide to take two workflows. One for the ED and one for inpatients. This model requires that data is in place in order to make the prediction. Like, you know lab results, flowsheet values, medications… if there are no lab tests or medications you know for that patient, it’s not going to predict very well. So you know talking to Epic, they stated that many hospitals chose to take a two branch approach to the prediction”
Clinician buy-in	Garner support with data	“Just showing people data of how often it fires, who it fires for, where the false positives are, and giving them visual patterns of how is succeeding or failing is a powerful tool”
	Direct feedback to teams	“We have demonstrated that direct feedback to the clinicians certainly results in higher compliance with antibiotics and bundle elements”
	Point of care clinical support	“We created a resource through the virtual care team that allowed nursing staff, provider staff to call anytime 24/7…you tell them this is my number, what does that mean? And we would say it is just a number, let’s look at everything that went into it, let’s talk about it and then let’s talk about what that means for what we need to do for our patient”
	Emphasis on ongoing multimodal user education	“I think you need to approach education from a couple of angles, because there’s different folks who learn in different ways. You need a video, you need a PowerPoint, it needs to be referenceable, there needs to be frontline people who go out and support units”
	Use of metaphors and analogies to address intuitiveness of tool output	“[We created a video] comparing predictive models to a weather forecast. It doesn’t mean you’re going to put the rainboots on now because it’s not raining right now”
	Incorporating frontline practitioners onto implementation teams	“I think the fact that we as the clinical effectiveness team are clinicians, I think really helps”
	Managing expectations	“[We] have to manage expectations that we are not yet at a point where these rules are going to be able to define sepsis without help from humans…”