When an Alert is Not an Alert: A Pilot Study to Characterize Behavior and Cognition Associated with Medication Alerts

Abstract

Introduction. Preventable adverse drug events are a significant patient-safety concern, yet most medication alerts are disregarded. Pharmacists encounter the highest number of medication alerts and likely have developed behaviors to cope with alerting inefficiencies. The study objective was to better understand alert override behavior relating to a motivational construct framework. Methods. Mixed-methods study of 10 pharmacists (567 verifications) with eye-tracking observations and retrospective think aloud interviews. Results. Pharmacists spent on average 14 seconds longer verifying orders with alerts than orders without alerts (p<0.001). Verification occurred before alerts were triggered, and no order changes occurred after alerts. Pharmacists reported 62% of alerts as unhelpful and 21% as frustrating. Alert interactions took on average 3.9 seconds. Discussion. Pharmacists anticipate alerts by making appropriate checks and changes before alert prompts. Medication alerts seem to be useful. However, the observed pharmacists’ behavior suggests changes in the alert context are needed to match cognition.

Introduction

Implemented with the goal to improve patient safety, medication alerts are an integral part of modern pharmacotherapy. While medication alerts can help reduce errors, there is a real concern that clinicians are not appropriately addressing them given that up to 96% are disregarded¹. A high frequency of alert overrides is often attributed to “alert fatigue” that is characterized by a decline in clinician responsiveness to alerts^1,2. Reasons posited for the increases in alert overrides include desensitization, the establishment of habitual behavior, and a mismatch between the clinician’s goals and the alert information. Whatever the reason, a high volume of ineffective alerts places patients at risk³.

Various attempts and recommendations have been made to reduce medication alert overrides and prevent alert fatigue. Each of these interventions are based on implicit or explicit theories of alert psychology. In some cases, the most feasible approach has been to eliminate certain problematic alerts to reduce the overall alert frequency and therefore decrease cognitive load⁴. Such an approach, however, may result in preventable adverse drug events, missed learning opportunities, and increased liability for clinicians as well as healthcare systems. A seminal paper by Van Der Sijs et al. highlights issues such as low specificity of alerts and alert-workflow mismatches as opportunities to improve medication alerts. In this study, which focused on the prescriber’s perspective in the context of computerized provider order entry (CPOE), the authors attributed the act of ignoring alerts to clinician’s trust in the pharmacist’s order verification¹, thereby suggesting diffusion of responsibility. Payne et al.suggested that the usability of drug-drug interaction alerts can be improved by contextualizing alerts to increase their specificity, wherein only essential information is included in the display to reduce cognitive load and emphasize the learning opportunities provided in an alert. While contextualizing alerts is a promising approach, fully enabling this is not a trivial task⁵. Some have suggested human factors principles should be incorporated in designing medication alerts such as prioritizing information and reducing confusion in the presentation of alerts⁶. This approach emphasizes the role of cognitive load in alert processing. Phansalkar et al.proposed that acceptance could be improved by reducing false positive alerts and minimizing habituation, which is a consequence of frequent exposures to an inconsequential stimulus (e.g., low risk or inappropriate medication alert)⁶. Baysari et al.suggested the workflow and tasks associated with CPOE are ideal for creating a habit of overriding alerts. This is due to the predictable context when medication alerts are displayed, the overall frequency of alerts, and the enhanced self-control over one’s behavior because overrides allow the user to continue ordering the medication without any immediate consequences⁷. Clearly, based on the above literature, the psychological impact of alerts should be evaluated to improve understanding of clinicians’ behavior regarding medication alerts. To gain a deeper understanding of the behavior related to medication alerts, we chose to map study observations to psychological constructs.

The vast majority of medication alert literature does not directly assess the behavioral impact of these alerts. Moreover, prior work has primarily focused on issues and remedies from the prescriber’s perspective using a CPOE. This focus on prescribers is a limitation because it potentially ignores at least 50% of inpatient medication alerts. Once a provider places a medication order, a pharmacist is required to verify the order, in most cases, prior to administration. Medication alerts are frequently repeated at each phase of the pharmacotherapy pipeline (i.e., order entry, verification, and administration)^5,8. Thus, pharmacists are frequently exposed to the same CPOE alerts, and the frequency of these alerts is compounded by the high prescriber to pharmacist ratio and that medication alerts are often configured with a lower threshold for pharmacists than prescribers. Therefore, our study objective was to observe target users (pharmacists) resolving medication alerts to gain insights on the underlying motivation to override alerts. We chose to observe pharmacists in the verification phase for several reasons including 1) medication alerts are concentrated at this phase; 2) pharmacists are seen as medication experts and are relied on for pharmacotherapy decisions and confirming order appropriateness; 3) the verification environment is relatively structured and repetitive, which is acutely sensitive to technology inefficiencies and habit formation; and 4) observed behaviors can reasonably be conveyed to order entry, since CPOE alerts are often repeated at order verification.

Methods

Design. This was a mixed-methods study where qualitative data was collected from retrospective think aloud (RTA) interviews, and quantitative data was obtained from eye-tracking and screen recordings during actual medication verifications in the inpatient setting.

Theoretical framework. We proposed four theoretical constructs that might explain clinicians’ response to alerts. These constructs were derived from motivational theories that seek to explain the interface between cognition and behavior more broadly. The approach we are advocating is translational, driven by a desire to apply experimentally validated insights from psychology to the healthcare setting. This translational work requires integrating theoretical perspectives and constructs in order to be meaningful. Specifically, we examined four motivational constructs related to overriding medication alerts (Box 1).

Box 1.

Motivational construct explanations and a priori knowledge relevant to the behavior of overriding alerts.

Control	The motivation to gain and maintain control is a core and ubiquitous human motivation9,10. Threats to a loss of control result in work-arounds, frustration, and active strategies to regain control. Supporting autonomy and control improves workers’ performance and their emotional status. Behaviors that are motivated by threats to control include ignoring information, demeaning the messenger, and manipulating tools to identify causal factors of an outcome11. We expect that hardstop medication alerts interrupt order verification tasks and may cause frustration.
Self-efficacy	Self-efficacy refers to the belief about one’s ability to cope with work demands using the provided resources12. It is one of the most prominent factors in explaining choice, persistence, and resistance to negative feedback13. We expect that as pharmacists are exposed to medication alerts, they learn how and when to address triggering factors. Over time, pharmacists gain competence and may develop a mental model of the alert context, which gives them skills in managing attention and delegating effort when verifying medication orders.
Diffusion of responsibility	Diffusion of responsibility is defined as the tendency for people to take less responsibility when other individuals also contribute efforts to the same goal14. The result is a tendency to contribute less than equal effort or fail to fully commit to an assignment. Since similar if not the same medication alerts are seen during three different phases of the pharmacotherapy pipeline (i.e., order entry, order verification, and medication administration), we expect that pharmacists have mental models of who is receiving medication alerts, which may contribute to diffusion of responsibility.
Intrinsic motivation	Intrinsic motivation is the affective experience of enjoyment, involvement and concentration in an activity. We focused on the “flow” experience, which is described as being lost in an activity described as fun, enjoyable, and inherently interesting15. Intrinsic motivation at work is often described as a match between the task demands and the individual’s skill set and high “flow” states are those requiring substantial skills16. We expect that pharmacists may get a sense of satisfaction when overriding alerts because they have anticipated the work required which allows quick dismissal of the alert to resume work tasks.

Participants. We recruited 10 inpatient pharmacists from the University of Utah Hospital by convenience sampling over two consecutive days in October 2017. The recruited pharmacists were blinded to study goals and worked at their usual verification terminals during observations.

Procedures. Pharmacists were instructed to perform their work as usual for about 20 minutes. Tobii Pro X2-60 was used to track pharmacists’ eye fixations on displayed information and to record the screen. Following each recording, two authors (TR and TT) reviewed the observations to identify the time of medication alerts. Four or five alerts were randomly selected from each recording for a subsequent RTA interview. The RTA was a semi-structured interview where the pharmacist was shown an alert from their recording and asked to respond seven questions from researchers. The questions and code template were developed and iteratively refined by members of the Sociotechnical Core Group in the Department of Biomedical Informatics at the University of Utah (TR, HK, CW, TT). Coding of the interviews was done by a subject matter expert (TR) and confirmed by a second author (HK). This study was approved by the Institutional Review Board at the University of Utah.

Alert user interface. We focused on “hard-stop” alerts, which are presented in a single “pop-up” window that can be overridden by selecting a reason (e.g., “benefit outweighs risk”) from a pre-defined list; or canceled to gather additional information, modify the order, or clarify the order with the prescriber. Hard-stop alerts include a color-coded icon indicating the alert hazard level (e.g., red for “very high” hazard) and alert type (e.g., two doses crossed out for drug-drug interactions); an alert label for the type of alert text and offending medications (e.g., warfarin, TMP/SMX); and an additional message, which is displayed in a shaded region to the right of the alert icon label and shows triggering factors such as specific overlapping duplicate orders (Figure 1). Alert logic and the information displayed on the alert user interface are provided by drug knowledge base vendors.

Figure 1.

Pharmacist verification workflow and alert display components.

Measures. We measured the verification time and alert display time. The verification time was measured in seconds from when the medication order was visible to when the “verify” button was pressed (Figure 1). Medication alerts are displayed subsequent to order verification after pressing the verify button. Alert display time was measured from when the alert was visible to when it was accepted, overridden, or canceled. Pharmacists’ perceptions about alerts was collected through the RTA interviews, which included a validated a set of questions based on the above mentioned motivational factors. The questions and code templates were developed and iteratively refined by the Sociotechnical Group in the Department of Biomedical Informatics at the University of Utah. The coding taxonomy was initially constructed based on prior literature on alerts plus our stated psychological hypotheses. Refinement required an iterative process of coding, discussion of interrater coding differences, and refinement after consensus. The questions and protocol were piloted with two observations not included in the analysis. Coding of the interviews was done by a subject matter expert and confirmed by a second researcher. The seven interview questions covered alert importance, helpfulness, interference or frustration, triggering factors, and the concern for the patient/order and if the alert changed the plan. We also developed a coding protocol to evaluate the eye-tracking recordings. The protocol was extensively tested with multiple iterations to achieve consensus on variable definitions. From eye-tracking recordings, we obtained the amount of time pharmacists fixated on the different sections of the alert user interface, i.e., alert icons and label and alert message.

Analysis. First, we compared differences in mean verification time between orders with and without alerts. Additionally, we compared mean verification time and alert display time by alert hazard level. The significance of the differences was assessed with the Welch Two Sample t-test with a 95% confidence interval. Second, we used Risk Ratios (RRs) to evaluate four outcomes: 1) the probability that pharmacists changed medication orders with alerts compared to orders without alerts; 2) the probability of pharmacists reporting a specific override reason for alerts associated with antibiotics or anticoagulants; 3) the probability that fixating on alerts was associated with reports that alerts were helpful or important; and 4) the likelihood that pharmacists reported frustration when addressing “medium” or “low” hazard level alerts. We used the Wald Maximum Likelihood Estimation to calculate RRs. Last, we estimated the predictors of whether an alert was “important” or “helpful” by fitting a generalized linear model with multi-variate and simple regression. The analyses were conducted using R version 3.3.1 (2016).

Results

Results are presented as descriptive associations with univariate analysis for verification time comparisons, followed by descriptive and multivariate analyses to identify predictors of alert helpfulness and importance.

Participant observations. We observed pharmacists verifying 567 medication orders over 3 hours and 40 minutes. Of these verifications, 100 had hard-stop alerts. There was 1 hard-stop alert requiring pharmacist input for every 4.15 active medication orders. Table 1 depicts the average participant demographics and observation data.

Table 1.

Summary of participants and recorded observations. SD = Standard Deviation.

Pharmacists (N=10)	Mean (SD)	Range
Licensure Year	2007 (10.17)	(1982-2017)
Recording Duration (minutes)	21.83 (4.23)	(12.15-25.77)
Medication Verifications (N=567)
Medication verifications observed per participant	53.8 (26.14)	(17-92)
Medication verifications per minute	2.54 (1.07)	(0.88-4.05)
Patients with Orders Verified (N=203)
Patients reviewed per participant	20.3 (9.07)	(5-32)
Medication verifications per patient	2.8 (0.75)	(1.8-4.1)
Alerts Observed (N=100)
Alerts per participant	10 (6.48)	(3-19)
Alerts per verification	0.19 (0.09)	(0.08-0.35)
Verification Time (seconds)
Verification of order with no alert	9.02 (20.06)	(1-245)
Verification of order with alert	22.75 (24.95)	(1-120)
Alert display duration	3.92 (4.03)	(1-25)

Alert distributions. We conducted an RTA interview on 48 randomly selected alerts. The distribution of alert type, potential hazard level, and alert response were similar between alerts without an RTA and alerts with an RTA (Table 2). Moreover, the total distribution of study alerts was similar to the distribution of unobserved alerts recorded in the EHR’s audit log during the same month (October). Drug-drug and duplicate therapy alert types comprised 70% of the total alerts. Fifty-four percent of alerts had a “High” potential hazard level. Pharmacists selected “Benefit outweighs risk” 92% of the time to override alerts. Medication alerts were most frequently associated with antibiotic (21%) or anticoagulant (20%) orders.

Table 2.

Characteristics of medication alerts and responses with RTA and without RTA.

Alerts without RTA (N=52)	N	%	Alerts with RTA (N=48)	N	%	Total (N=100)
Alert Type
Drug-Drug	20	39	Drug-Drug	18	38	38
Duplicate Therapy	15	29	Duplicate Therapy	17	35	32
Duplicate Medication/Therapy	8	15	Duplicate Medication/Therapy	3	6	11
Allergy/Contraindication	6	11	Allergy/Contraindication	4	8	10
Duplicate Medication	2	4	Duplicate Medication	2	4	4
High Dose	1	2	High Dose	2	4	3
Dose	0	0	Dose	2	4	2
Potential Hazard Level
Very High	16	31	Very High	8	17	24
High	29	56	High	25	52	54
Medium	6	11	Medium	13	27	19
Low	1	2	Low	2	4	3
Alert Response
Benefit outweighs risk	50	96	Benefit Outweighs Risk	42	88	92
Per protocol	0	0	Per protocol	0	0	0
Does not apply to patient	0	0	Does not apply to patient	0	0	0
Inaccurate warning	0	0	Inaccurate Warning	2	4	2
Cancel	2	4	Cancel	4	8	4

Pharmacists’ responses to and perceptions of alerts. Pharmacists reported that 62% of medication alerts were unhelpful and 50% were unimportant; however, 58% of alerts did not interfere with their work or frustrate (Table 3). Alerts were overridden 44% of the time because they were reported as not clinically relevant. Pharmacists indicated that the prescriber and nurse in 75% and 52% of alerts respectively would share some responsibility in ensuring the patient was not at risk. Pharmacists took on average 3.9 seconds to resolve an alert. When the alert was displayed, they fixated 75% of the time on the alert icon and label and 25% of the time on the additional message. Nine of twenty-one order changes (43%) occurred with medication orders that had alerts. All of these changes, however, were completed during the verification time, before the hard-stop alert. In other words, no medication changes were made after hard-stop alerts.

Table 3.

Interview responses to RTA questions with eye-tracking and screen capture observations.

Retrospective Think Aloud (N=48)	N	%
Override Reason (self-reported)
Not clinically relevant	21	44%
Does not apply to this order/patient	18	38%
Alert error	4	8%
Appropriate alert	5	10%
Alert Important
No	24	50%
A little	15	31%
Yes	9	19%
Alert Helped
No	30	62%
A little	9	19%
Yes	9	19%
Alert Interfered or Frustrated
No	28	58%
A little	10	21%
Yes	10	21%
Other Clinicians Responsible for Alert
Provider	36	75%
Nurse	25	52%
Floor Pharmacist	11	23%
Alert Eye-tracking (N=100)
Fixated on Icon and Label	75	75%
Fixated on alert Message	25	25%
Medication Order Changes (N=30)
Order changes without alert (N=21)
Order changes with alert (N=9)
Order change before alert	9	100%
Order change after alert	0	0%

Association of alerts with observations. There was a significant difference in the mean verification time (seconds) for orders that had subsequent alerts versus no alert(μ =22.7 vs. 9.0; p<0.001) (Table 4). There was a significant difference in the mean verification time between orders with a subsequent “Very High” and “Medium” alerts =35.6 vs. 20.2; p<0.05). The average alert display time (seconds) trended down with decreasing potential hazard levels. The mean alert display time was significantly different with “Low” hazard level alerts having shorter display time than “Very High” “High”, and “Medium” severity alerts =1.7 vs. 5.0 vs. 3.5 vs. 4.2; p<0.05).

Table 4.

Order verification time (seconds) comparison with and without alerts. Pairwise comparisons of alert potential hazard level by verification time and alert display time (seconds). df = Degrees of freedom, SD = standard deviation.

Verification Time	Mean(SD)	Mean(SD)	t (df)	P
Medication Order
Alert vs No Alert	22.7(25)	9.0 (20)	5.2 (128)	<0.001
Potential Hazard Level
Very High vs High	35.6 (37)	20.2 (20)	1.8 (29)	0.08
Very High vs Medium	34.6 (37)	17.0 (16)	2.1 (33)	<0.05
Very High vs Low	34.6 (37)	11.0 (15)	2.1 (6)	0.08
High vs Medium	20.2 (20)	17.0 (16)	0.7 (37)	0.49
High vs Low	20.2 (20)	11.0 (15)	1.0 (2)	0.40
Medium vs Low	17.0 (16)	11.0 (15)	0.6 (3)	0.57
Alert Display Time	Mean(SD)	Mean(SD)	t (df)	P
Potential Hazard Level
Very High vs High	5.0 (6)	3.5 (2)	1.2 (26)	0.26
Very High vs Medium	5.0 (6)	4.2 (5)	0.5 (41)	0.61
Very High vs Low	5.0 (6)	1.7 (1)	2.5 (25)	<0.05
High vs Medium	3.5 (2)	4.2 (5)	–0.6 (25)	0.54
High vs Low	3.5 (2)	1.7 (1)	4.0 (7)	<0.05
Medium vs Low	4.2 (5)	1.7 (1)	2.3 (20)	<0.05

Probability of pharmacists’ alert responses. The probability for changing an order with a subsequent alert trended up but was not significantly increased (RR=1.1, 95% CI 1.0-1.1, p=0.08) (Table 5). Alerts associated with antibiotic and anticoagulant medication orders accounted for 41% of the total observed alerts. The most common reported reason for overriding antibiotic orders was that it was “not clinically relevant.” The probability of overriding an antibiotic alert because it was “not clinically relevant” was significantly greater than any other reason (RR=2.9, 95% CI 0.810.0, p<0.05). The probability for overriding an anticoagulant alert because it did “not apply to this order/patient” was significantly greater than any other reason (RR=2.4, 95% CI 0.9-6.2, p<0.05). Alerts where the pharmacist fixated on the icon/label or message did not increase the probability that they were helpful (RR=0.5, 95% CI 0.2-1.7, p=0.22) or important (RR=0.7, 95% CI 0.2-2.7, p=0.67). Results of fitting a multi-variate generalized linear model indicated there was not a collective statistically significant effect among alert display time, licensure, icon/label eye fixation, self-reported override reason, potential hazard level, and helpfulness or importance of the alert. The results of a single variable model indicate there was a significant association between potential hazard level and whether the alert was helpful (coefficient=0.9, 95% CI 0.1-2.0, intercept=-1.3, p<0.05) and important (coefficient=2.4, 95% CI 1.2-4.3, intercept=-6.9, p<0.05).

Table 5.

Probability of changing an order with an alert, overriding an antibiotic and anticoagulant alert for a specific self-reported reason, and a medium or low hazard level alert frustrating the pharmacist. RR = Risk Ratio, CI = Confidence interval.

Pharmacist Action	Alert (N=100)	No Alert (N=467)	RR (95% CI)	P
Changed order	9 (9%)	21 (5%)	1.1 (1.0-1.1)	0.08
Alert Override Reason
	Antibiotic (N=9)	Other (N=39)
Not Clinically Relevant	7 (78%)	14 (36%)	2.9 (0.8-10.0)	<0.05
	Anticoagulant (N=10)	Other (N=38)
Not Apply to Order/Patient	7 (70%)	11 (30%)	2.4 (0.9-6.2)	<0.05
Fixated on Alert Icon/Label	Yes (N=42)	No (N=6)
Helped with Work	16 (38%)	4 (67%)	0.5 (0.2-1.7)	0.22
Alert was Important	22 (52%)	4 (67%)	0.7 (0.2-2.7)	0.67
Alert Hazard Medium/Low	Yes (N=20)	No (N=28)
Interfere or Frustrate	5 (25%)	6 (21%)	1.1 (0.8-1.4)	1

Pharmacist explanatory statements for findings. In response to a duplicate therapy alert for heparin, a pharmacist stated, “My workflow is to ensure anticoagulants aren’t being duplicated, before the alert. But I wouldn’t take the alert away.” This supports our findings that verification time increases for certain orders that have subsequent alerts. The observed increase in alert display time by hazard level was characterized by a pharmacist stating, “Anytime there is a warning that says very high, I try to take extra time to consider it.” While this statement is supported by our aggregate findings, in this specific instance the “very high” alert was overridden almost immediately (1 second). This example illustrates that pharmacists do consider alerting information, but as it becomes known and anticipated the alert is no longer useful. In response to an allergy/contraindication with a ceftriaxone order, a pharmacist stated, “This information did not help me with my work since I knew it would be popping up from the allergies listed on the patient profile.” Again, the pharmacist seems to anticipate the alert by recognizing the relative frequency of beta-lactam allergies and used verification time to preemptively check the patient’s allergies before the alert. One pharmacist characterized his/her control over the work environment as, “None of the alerts really frustrate me.” Pharmacists seem to have adapted to alert management, and have developed heuristic rules for overriding the most frequent alerts (i.e., antibiotics and anticoagulants). According to the self-reported override reasons, pharmacists found antibiotic alerts to be “Not clinically relevant” (e.g., “two antibiotics are indicated for this infection”), and anticoagulant alerts to “Not apply to this patient/order” (e.g., “the warfarin will only overlap heparin for a short time”). This was an interesting delineation since the rationale seemed to overlap the two classifications in several instances. While pharmacists understood that medication alerts were primarily their responsibility, they knew other providers have and will see the same alerts at another point in the pharmacotherapy pipeline and use their clinical expertise to ensure the patient was not at risk (e.g., “Typically with fluid orders there is no concern; the nurse would not hang two bags.”).

Discussion

In this exploratory pilot study, we took a novel mixed-methods approach to examine pharmacists’ alert override behavior by leveraging a motivational framework. By using qualitative methods, eye-tracking and screen recording, we captured comprehensive data on how pharmacists have adapted and are coping with EHR inefficiencies and specifically medication alerts. The proposed mixed-method approach has the potential to generate deeper understanding of clinician-alert interaction and response behavior than purely quantitative methods. Pharmacists seem to preemptively address alert triggering factors by making order changes and checks prior to alerts; no order modifications were observed as a result of medication alerts. Medication alerts may improve pharmacists’ self-efficacy until the presented information is learned; at which point, quickly overriding repetitive alerts may become intrinsically motivating. Repeating the same medication alerts may lead to diffusing the responsibility of ensuring patients not at risk for preventable adverse drug events.

Pharmacists seem to have four motivational consequences relating to alert environment adaptation (i.e., control, self-efficacy, diffusion of responsibility, and intrinsic motivation). Pharmacists spend significantly more time verifying medication orders that had subsequent alerts; moreover, all order modifications occurred prior to alerts. This behavior may be driven by the ubiquitous motivation to regulate the use of cognitive resources (e.g., attention and effort) to support natural workflow. The implication is that alert interruptions may be anticipated and that attention to them can occur when ready instead of allowing alerts to interrupt. Certain medication alerts did not seem to frustrate the pharmacists, which may further support their ability to control the work environment by immediately overriding alerts. The substantial difference in verification time and alert display time also ties into intrinsic motivation to quickly override alerts. The verification tasks seem to match the alert expectations; this suggests that when clinicians immediately override alerts, they may be using them as feedback, which contributes to a sense of flow. The potential danger to quickly overriding alerts to maintain control and provide a sense of flow is that information can be missed or misinterpreted. While most of the observed alerts were unhelpful, pharmacists were reluctant to suggest removing alerts and stated that in certain situations these alerts would be very helpful. This finding aligns with providers’ perception that knowledge gaps among colleagues are a risk when removing alerts.⁴ It seems pharmacists have used the alert information to learn and cultivate self-efficacy, but over time the information becomes known and alert usefulness is diminished. When new knowledge is discovered and implemented as alerts, it is possible this information will not improve self-efficacy and proficiency, since it will be buried under known alerts and disregarded as such. In theory, displaying an alert at several points in the pharmacotherapy pipeline should increase the likelihood of catching an error. However, this can be problematic when clinicians ignore alerts with the assumption that someone else will take appropriate action to address it (i.e., diffusion of responsibility). While pharmacists seem to take ultimate responsibility of medication alerts, they are cognizant that other clinicians may have seen and acknowledged the same alerts and, at some level, may diffuse the responsibility.

Based on the examined motivational consequences, we propose certain behaviors can be improved by adhering to two overarching suggestions to make alerts 1) contextual and 2) adaptive (Figure 2). Contextualizing alerts towards increased personalization is not a new concept⁵; however, this study illuminated a couple additional opportunities that may modify behavior. It appears users receiving alerts learn to exert control over their environment. Leveraging contextual information (e.g., patient-specific risk factors, additional order information such as medication overlap) could allow alert logic and severity to be tailored to specific characteristics of the patient and other medications in use. To exert control, alerts could be modified such that users could interact with triggering factors to reduce or balance presented information. For example, adding a proton pump inhibitor to a warfarin order, for a patient with a history of gastrointestinal bleed, reduces the risk of a similar event; therefore, the alerting mechanism could be adjusted from a hard-stop alert to a passive information display. In addition, context could account for clinician roles and associated responsibilities. For example, pharmacists may want to see all drug-drug interactions at the ingredient level, whereas providers may want to see drug-indication information and nurses may want to see medication dosing limits. This type of contextualization could not only reduce the frequency of alerts but help delineate roles and prevent diffusion of responsibility.

Figure 2.

Proposed alert changes overlapped with motivational constructs and mapped to study findings.

We define alert adaptation as modifications in the presentation of an alert based on a history of user-alert interactions. We propose that as users become proficient at appropriately assessing patient factors related to alerts, the alerts could adapt by presenting new evidence or be filtered out to reduce the alert burden. A new user may have information/alerts that appear very frequently during, for example, a training period. As the user fully addresses alert triggering factors enough times to develop a habit, the alerts are automatically filtered. As known alerts are filtered, new alerts from the expanding drug knowledge base may increase proficiency and self-efficacy. Figure 2 depicts our interpretation of how these study finding could be mapped to motivational constructs and proposed alert changes.

While there are many instances in the literature where novel alert approaches are piloted and evaluated in simulation environments, prevalent commercial EHR environments are generally not amenable to substantive deviations from the current alerting framework. Currently, knowledge driving medication alerts are licensed from third-party vendors (e.g., Wolters Kluwer Health Medi-Span, First Databank), and it is currently not possible to customize the medication alerts in the manner proposed in Figure 2.

Future work. Promisingly, however, EHR vendors are beginning to support external CDS Web services in an approach known as CDS Hooks. An EHR could invoke this CDS at the time of medication order (using the medication-prescribe hook) or at the time of medication order review (using the order-review hook), and the CDS Web service could retrieve additional required data using the HL7 Fast Healthcare Interoperability Resources (FHIR) standard. This mechanism could pull in relevant clinical and contextual factors that allow the CDS logic to tailor alerts for a particular context. The University of Utah is already leveraging CDS Hooks, using the OpenCDS Web service, in conjunction with a middleware layer that converts an EHR-specific CDS Web service interface to the CDS Hooks interface. Thus, it is well-positioned to investigate in actual clinical care settings whether proposed changes in Figure 2 could enhance medication alert effectiveness.

Challenges remain to fully operationalizing the CDS Hooks approach. First, not all relevant contextual information is supported in EHR vendor FHIR interfaces. Second, until the medication alert logic provided by medication knowledge vendors is available through CDS Web services compliant with CDS Hooks, co-implementing the two approaches in parallel could lead to user disorientation and compounded alert fatigue. Finally, there is currently no consensus standard for sharing this type of medication knowledge among institutions⁵. To address this last challenge, there is ongoing effort to identify core context information that EHRs should make available for each high-priority potential drug-drug interactions (PDDIs) to contextualize high-priority medication alerts¹⁷. Work has begun within the Health Level 7 CDS and Pharmacy work groups to standardize the representation of PDDI knowledge and to specify standard CDS Web service capabilities using the emerging CDS Hooks standard.

Strengths and limitations. The primary strength of this study was the novel mixed-methods approach, combining traditional alert response data with eye-tracking, screening recordings, and RTA interviews. The analysis was guided by key psychological constructs that help explain broad human behavior. Applying these constructs to alert overriding behavior has provided unique insights to alert fatigue. We used synergistic techniques (e.g., eye-tracking, screen recording, interview) to capture comprehensive data on clinician behavior and support our hypotheses. The scope of this study was a limitation. Since this was a pilot study, it was restricted to a single institution, one clinician role, and a small sample size. Many of the inference analyses failed to reach significance (e.g., verification and alert display time by certain hazard levels), perhaps due to the relatively small sample size and low number of events observed (e.g., order changes). Additionally, we did not investigate the effect of differences in patient (e.g., complexity, acuity) or clinician characteristics (e.g., years of experience) in the analysis. While the additional equipment was minimal and participants were blinded to study goals, they were cognizant that we were gathering information which may have changed their behavior. The University of Utah has made a significant effort to refine logic and filter out non-clinically relevant alerts, so we were not able to observe pharmacists resolving previously filtered alerts. Finally, we observed pharmacists that were relatively experienced with the EHR; therefore, we cannot conclude how novice users would interact with alerts.

Conclusion

This study shows pharmacists adapting to the inefficiencies of medication alerts. In coping with medication alerts, pharmacist appear to have developed motivational consequences to overriding alerts. Medication alerts should support user performance by providing a sense of control, support self-efficacy, and limit diffusion of responsibility.