Assessing Medication CDS Usability: Pilot Results from 10 Outpatient Clinics

Abstract

Objectives

This study aimed to develop a human factors assessment for medication-related clinical decision support (CDS) based on a previously validated tool that assessed the integration of human factors principles in CDS, the instrument for evaluating human factors principles in medication-related decision support alerts (I-MeDeSA), and pilot it with 10 outpatient clinics across the United States.

Methods

The human factors assessment was developed based on past validations of I-MeDeSA. Examples included changing the wording of questions and reformatting answer choices to check-box options, allowing for multiple answer choices. We also added a section about how clinicians resolved alerts. Clinics received a percentage score based on how well their CDS adhered to human factors principles. To take the assessment, testing teams at each clinic triggered a high-severity drug–drug interaction (DDI) alert, and then took the human factors assessment. This assessment was piloted in 10 outpatient clinics, each of which used a different commercial electronic health record (EHR) system.

Results

The final assessment included five sections and twelve questions related to aspects like the timing, visual aspect, severity, content, and actions within the DDI alert. The mean overall percentage score was 62%. The sections regarding the timing and visual aspects of the alert were ones where clinics' EHRs performed the best. However, in the “actions” section, 40% of the clinics could bypass high severity alerts without any safeguards in place.

Conclusion

We found substantial variability in the integration of human factors principles in the design and delivery of DDI alerts among the outpatient clinics, and some lacked important medication safeguards. This assessment can be used by outpatient clinics for safety improvement initiatives.

Background and Significance

Fifteen years after the Health Information Technology for Economic and Clinical Health (HITECH) Act of 2009, almost 90% of outpatient practices have adopted electronic health records (EHRs). 1 In addition, as a result of stage 2 of the meaningful use program, part of HITECH, more than 95% of hospitals have adopted EHRs which include basic decision support functionalities such as drug–allergy and drug–drug interaction (DDI) checking. 2 3 Early studies of EHR implementation found that clinical decision support (CDS) delivered at the point of care can decrease the rate of preventable adverse drug events. 4 5 CDS can be categorized as passive or active. 6 Passive CDS does not require a clinician to acknowledge decision support. 6 In contrast, active CDS involves the delivery of patient-specific advice or information, and a clinician must perform an action to resolve it. 6 Active CDS is most often delivered to clinicians via interruptive alerts that are presented during the clinical workflow and require the clinicians to acknowledge the alert or provide an override reason if the clinician rejects the recommendation of the alert. These interruptive alerts can appear either during or after medication ordering. The degree to which clinics and hospitals can customize the delivery of CDS varies and is often dependent on the EHR vendor and the associated medication reference database vendor.

However, despite the potential for EHR customization, several studies have found that the design of EHRs and CDS often does not adhere to human factors design principles. 7 8 These principles revolve around understanding the interaction between users and the system they are using. 9 In designing CDS tools, these human factors design principles can be related to the use of color in signifying the severity of the CDS tool, the placement of actions related to resolving the alert, or the timing of the alert during clinical workflows. 7 10 The lack of integration of human factors principles into the design of CDS tools and other healthcare-related technologies has led to high rates of alert fatigue among clinicians. According to the Agency for Healthcare Research and Quality, alert fatigue is the progressive desensitization to alerts experienced by clinicians when too many decision support alerts are presented at once. 11 Indeed, studies have shown that alert fatigue has been shown to have negative effects on clinician well-being. 12 13 This poses serious threats to patient safety, as clinicians can miss important decision support alerts if they are presented together with other alerts, some of which are inconsequential. 7 14 Indeed, a study by Adams et al 15 that analyzed 2,700 patient safety event reports revealed that usability issues related to health IT use contributed to the medication errors identified in these reports.

In response to these issues, tools and surveys have been created to evaluate the usability of EHRs, specifically their relationship to medication-related CDS. One example is the instrument for evaluating human factors principles in medication-related decision support alerts (I-MeDeSA), 16 which measures the integration of human factors principles in designing and delivering medication-related decision support, initially developed by researchers at Brigham and Women's Hospital (United States). To develop this instrument, researchers performed a literature review that identified the use of human factors principles within alerting systems and, based on this review, provided 11 recommendations for designing and implementing alert systems. 7 These recommendations were related to the prioritization of alerts, interface design, presentation of clinical data, and formatting of the alerts. 7 In developing the I-MeDeSA, these recommendations were the basis of the content. Researchers then validated this content and piloted it by assessing DDI alerts across three different outpatient EHR systems, two of which were in the United States, and the other in Germany. 16 Based on these pilots, further refinements to the tool were made. The final version of the instrument consisted of nine sections and 26 questions that evaluated several aspects of CDS alert design, including placement and visibility, text-based information, and corrective action. This version was then piloted with 14 inpatient and outpatient EHR systems across the United States, and there were variable results in the integration of human factors principles in the design of DDI alerts. 14 The I-MeDeSA has also been externally validated by research groups in Australia 17 and South Korea. 18 These groups have highlighted several areas of potential improvement for the instrument, including ambiguous wording and scoring. 17 18 Regarding scoring, there were sections within the I-MeDeSA, like the prioritization section, that had conditional scoring, where if a feature within the EHR was unavailable, users of the instrument would lose points for that entire section. 17 Moreover, the instrument mostly focused on the delivery of DDI alerts and their visual aspects, and not much on the actions associated with how providers accept or reject a CDS alert. With the high override rate for CDS alerts and variability of the appropriateness of override reasons, 19 it is critical to measure the actions clinicians take to override alerts. Given this, we sought to develop a human factors assessment for CDS alerts that incorporated many of the original elements of I-MeDeSA as well as newer areas focused on the delivery and actions associated with resolving CDS alerts.

Objectives

To develop a human factors assessment for medication-related CDS based on the I-MeDeSA. We then piloted this assessment with 10 outpatient clinics across the United States, each using a different commercial EHR system, and present these results. This updated version was developed so that this tool could be scaled for use by hundreds of clinics every year as part of the new ambulatory version of the Leapfrog CPOE/EHR Inpatient Test.

Materials and Methods

The Ambulatory EHR Evaluation Tool

The assessment was administered through the Ambulatory EHR Evaluation Tool. 20 This electronic tool aims to provide outpatient clinics with an evaluation they can use to identify gaps in their medication decision support. The tool has three sections: (1) a medication safety test, (2) a medication reconciliation module, and (3) a human factors assessment focused on medication decision support. The tool's methodology closely mirrors that of the Leapfrog Group's inpatient CPOE evaluation tool, where clinics were provided with a set of test patients and associated medication test orders to enter into their live EHR system. As licensed prescribers entered these test orders, they recorded any advice or information they received from their EHR system and then receive immediate feedback on their performance. This tool was initially piloted with seven outpatient clinics, each using one of the leading outpatient EHR vendors, as identified by the 2017 Office of the National Coordinator (ONC) database. 1 The pilot results revealed that outpatient clinics have basic decision support features like drug allergy checking implemented, but advanced decision support features such as drug age, drug laboratory, and drug monitoring alerts were largely unavailable. 20 Based on this initial pilot, the tool underwent content changes, and the testing modules were further refined. This final version of the new Ambulatory EHR Evaluation Tool was then piloted with ten outpatient clinics, which included this new CDS human factors assessment.

Pilot Process for the Human Factors Assessment

The pilot process for the Ambulatory EHR Evaluation Tool has been previously published. 20 21 Briefly, the study team identified the seven leading outpatient EHR vendors using the 2017 ONC's database. From there, we contacted these vendors and requested recommendations for outpatient clinics that would be interested in participating in the initial piloting of the ambulatory EHR evaluation tool. We then sent these clinics recruitment emails and also asked for their recommendations for other outpatient clinics that would also be interested in participating in the pilot.

Once we finished recruitment, three video-conference calls were set up with each clinic. During the first conference call, the study team introduced the Ambulatory EHR Evaluation Tool and overviewed the testing process with each pilot clinic site. In addition, there was a registration process in which clinics were required to input demographic information and take a CDS Functionality Survey. 21 Clinic demographic information and results from the CDS Functionality Survey have been previously published 21 ; however, select results from these surveys that are relevant to the human factors assessment are presented. Next, during the second conference call, each clinic took the medication safety test and medication reconciliation module. Afterward, clinics were sent a link to the human factors assessment to complete on their own. For all the modules within the ambulatory EHR evaluation tool, each pilot site was asked to assemble a testing team. For this team, the minimum requirement was one clinician who had prescribing privileges and another individual familiar with the CDS system at each clinic (i.e., someone from health IT). After the second conference call and finishing the human factors assessment, a third conference call was held, which was a debrief meeting. During this meeting, results from all sections of the Ambulatory EHR Evaluation Tool were shared and discussed with each clinic. ( Supplementary Table S1 , available in the online version only). Note, all three of these conference calls were recorded with screen capture.

Process for Updating the I-MeDeSA

Assessment Methodology

The updated human factors assessment was delivered to clinics electronically through REDCap. 22 23 The methodology of the assessment mirrors that used in the original I-MeDeSA ( Fig. 1 ). We instructed clinics to generate a high-priority or “level 1” DDI alert using a preexisting test patient within their EHR system. To assist with this, we provided three high-severity DDI test orders for the clinics to choose from to trigger a DDI alert in their EHR system. These included an interaction between (1) simvastatin and verapamil, (2) clarithromycin and verapamil, and (3) simvastatin and clarithromycin. This list of DDI test orders was developed by the research pharmacist on our study team, who also developed the content for the medication safety test. 20 Next, once the DDI alert was generated, the testing team at each clinic answered the questions in the assessment, which were related to the availability and integration of human factors principles within the DDI alert. After finishing the assessment, the clinic's testing team received quantitative feedback through a percentage score of human factors principles to which their medication-related CDS adhered. Each clinic's percentage score was also associated with different levels of adherence to human factors principles, and this was done to provide clinics with a benchmark of their performance compared with other clinics in the study ( Table 1 ). After discussing the assessment results with each clinic, the research team sent each clinic a guidance document that provided qualitative feedback for the clinics.

Fig. 1.

Methodology for using and taking the human factors assessment.

Table 1. Association between the clinic's score on the human factors assessment and levels of adherence to the human factors assessment.

Level of adherence to human factors principles	Percentage score range (%)
Full adherence	≥75
Substantial adherence	50–75
Some adherence	25–50
Low adherence	<25

Assessment Content

Past reviews of I-MeDesA highlighted several areas in which it might be improved. These improvements included editing ambiguous wording used throughout the tool and changing the scoring of conditional items within the instrument. 17 18 Conditional items were ones where a negative score on one item affected the scoring of proceeding questions and, in some cases, an entire evaluation section. Given this, the format in which the questions were presented was edited so that clinics were provided with multiple options rather than open-ended or binary-style questions. For example, in the “text-based information” section of the I-MeDeSA, most of the questions were presented in a conditional fashion, where if specific text-based content were not available in the alert, the facility would score poorly for that entire section ( Fig. 2 ). As a result, this section was edited to “please select which of the following text-based information is included within the alert,” with four options presented in a check-box format so clinics could select which text-based information was included. Another example is in the “visibility” section of I-MeDeSA, where there were two questions that asked how an alert was made distinguishable from the rest of the screen. Like the previous example, we combined these two questions into one question: “How is the alert visually distinct from the rest of the screen?” with several check-box options that included the alert being a different color from the rest of the screen and the use of a larger and bolder font. These types of changes were made to several sections of the original instrument.

Fig. 2.

Example of the types of changes we made to the original I-MeDeSA 16 questions. This included changing the wording of questions and adding check-box options.

Next, we added a section to the assessment called “actions (how to resolve the alert),” which incorporated two questions from I-MeDeSA. These questions asked about the placement of the alert relative to the ordering window and the availability of corrective action. For the latter, instead of asking clinics whether their EHR system had this capability, we asked, “What happens when the alert is accepted?” The choices we provided were: (1) the prescriber must manually make changes to the order (i.e., discontinue the medication causing the reaction), and (2) the system automatically makes the recommended changes (corrective action). In this section, we also developed a question that asked users, “How, if at all, can the prescriber bypass the alert?” The two answer choices were “yes” or “no.” If users answered “yes,” they were presented with the follow-up questions about how, within the alert, they could bypass it. Possible options included: providing an override reason, clicking a button, such as “okay,” with justification, or a single click to dismiss. A full list of changes that were made to the original I-MeDeSA can be found in Supplementary Table S2 (available in the online version only).

This study was funded in part by a grant from the Gordon and Betty Moore Foundation to the University of Utah.

Results

The Human Factors Assessment

The final version of the human factors assessment consisted of five graded sections with twelve questions ( Supplementary Material. S1 , available in the online version only). These sections include (1) timing of the alert, (2) visual aspects of the alert, (3) severity of the alert, (4) content of the alert, and (5) actions (how to resolve the alert). The assessment also included an ungraded section named “irrelevant information in or surrounding the alert,” where clinicians could provide subjective information about any distracting information they encountered while entering the test order.

Results from Piloting the Human Factors Assessment

Clinic Demographics and Results from CDS Functionality Survey

The human factors assessment was piloted with 10 outpatient clinics, and each clinic's testing team took the assessment. Clinics that participated in this pilot were either standalone clinics (50%) or part of a healthcare system (50%). Results from the CDS Functionality Survey revealed variation in how these clinics receive IT support, provide feedback about alerts, and the extent to which these clinics can customize their CDS alerts. Most clinics (70%) have in-house IT support, while 30% received IT support from their EHR vendor ( Table 2 ). Next, for providing feedback about CDS alerts, most clinics used a combination of methods. Among those, the most common were submitting a help desk ticket to the EHR vendor (60%) or the clinic's IT department (50%), as well as providing feedback directly to the EHR vendor. The least common methods were periodic review of the alert content (20%), involving clinicians in the alert testing and design (30%), and usability testing (30%). In terms of customizing alerts, most clinics could customize alerts at the clinic level (60%), where any changes would affect all users. In contrast, fewer clinics could customize alerts based on clinical role (20%) or by the individual clinician (30%). Lastly, less than half (40%) of the clinics could customize the medication reference database used for their CDS alerts.

Table 2. Demographic table and select results from the CDS functionality survey.

Clinic demographics	n (%)
Region
Northeast	5 (50)
West	5 (50)
Healthcare system membership
Standalone clinic	5 (50)
Part of the healthcare system	5 (50)
Clinic services
Primary care	5 (50)
Multispecialty	5 (50)
Clinical decision support functionality survey	n (%)
How does your clinic receive IT support?
Physician organization	2 (20)
In-house IT support	7 (70)
EHR vendor	3 (30)
How does your clinic provide feedback about alerts?
EHR vendor involves providers in alert testing and design	5 (50)
Clinic involves providers in alert testing and design	3 (30)
Clinician provides feedback to the EHR vendor	3 (30)
Periodic review of alert content	2 (20)
Usability testing	3 (30)
Submit a “help desk” or “trouble ticket” to the EHR vendor	6 (60)
Submit a “help desk” or “trouble ticket” to the clinics' IS desk	5 (50)
Can alerts be customized in any of the following ways?
At the clinic level (impacting the alerts that all users see)	6 (60)
By role (i.e., resident physician, attending physician, and nurses)	2 (20)
By individual (i.e., a user may select which alerts they see by severity or suppress certain types of alerts)	3 (30)
By encounter type	1 (10)
Alerts cannot be customized by role	2 (20)
Can the medication reference database be customized or altered at your clinic?
Yes	4 (40)
No	6 (60)

Quantitative and Qualitative Results from the Pilot

The mean score for the human factors assessment was 62%. The range of scores was 56%, with a minimum score of 31% and a maximum score of 87% ( Fig. 3 ). Four clinics (40%) scored over 75% in the assessment, achieving “full adherence” to human factors principles, while two clinics (20%) achieved “substantial adherence,” and three clinics (30%) achieved “some adherence.” Most clinics (70%) triggered the DDI alert by ordering simvastatin and verapamil together, two clinics (20%) used clarithromycin and verapamil, and one clinic (10%) used simvastatin and clarithromycin.

Fig. 3.

The overall percentage score for each clinic in the human factors assessment and correlating level of adherence to human factors principles. The red dotted line represents the percentage score (75%) needed to be determined as full adherence to human factors principles.

For the first section of the assessment, “timing of the alert,” most clinics (60%) did not have multiple alerts appear in response to the high-severity DDI alert ( Table 3 ). For clinics that did have multiple alerts appear (40%), most had the most severe alert appear first ( Table 3 ), but the other lower priority alerts were not visually distinct compared with the highest severity alert ( Fig. 4 ).

Table 3. Pilot results from the human factors assessment.

Section 1: Timing of the alert
Question	Responses ( n , %)
Did multiple alerts appear for this medication order?	Yes			No
	4 (40)			6 (60)
	Does the most severe alert appear first?
	Yes 3; (75)		No; 1 (25)
When does the alert appear during the workflow?	During ordering		At signing	Both
	6 (60)		3 (30)	1 (14)
Section 2: Visual aspects of the alert
How is the alert visually distinct from the rest of the screen?	Alert appears as a pop-up	Different color	Font (i.e., larger and bolder)	Use of signal word	Use of a symbol or Icon
	9 (90)	6 (60)	3 (30)	7 (70)	5 (50)
When the alert is presented, is it located directly in the line of sight of the prescriber?	Yes			No
	10 (100)			0 (0)
Section 3: Severity of the alert
How is the severity of the alert represented?	Use of a color-coded system only		A symbol or icon only	Combination of colors and symbols	Other
	2 (20)		0 (0)	6 (60)	2 (20)
Section 4: Content of the alert
Are the generic names of medications spelled with all lowercase letters?	Yes			No
	4 (40)			6 (60)
Are the brand names of medications spelled with only the first letter capitalized?	Yes		No	NA (clinic only uses generic names for medications)
	3 (30)		3 (30)	4 (40)
Is Tallman lettering used when medications of similar names are placed adjacent to one another in drug lists?	Yes			No
	5 (50)			5 (50)
Are the two interacting drugs located at the beginning of the sentence?	Yes		No	NA (alert not in sentence form)
	5 (50)		2 (20)	3 (30)
Is it a signal word used to indicate the priority of the alert?	Yes			No
	8 (80)			2 (20)
Which of the following text-based information is included in the alert?	Drugs in question		Instruction statement	List of recommended tasks	Consequence statement
	9 (90)		4 (40)	1 (10)	9 (90)
Section 5: Action (how to resolve the alert)
Possible responses	Yes			No
Is the recommended action located within the same window of the alert?	7 (70)			3 (30)
				Where is the recommended action located?
				Different window (separate from the alert itself)	Different areas of the screen
				2 (67)	1 (33)
Can the prescriber bypass this alert?	Yes			No
	10 (100)			0 (0)
	If yes, how can the prescriber bypass the alert?
	Provide an override reason		Clicking a button, such as “okay,” with justification	Single-click to dismiss
	2 (20)		4 (40)	4 (40)
What happens when the prescriber accepts the recommendations of the alert?	Options for how to resolve the alerts are presented to the prescriber			The system automatically makes the recommended changes (Corrective Action)
	9 (90)			1 (10)

Note: A signal word is a term that represents the severity and priority of the alert. Examples include “warning,” “caution,” or “danger.” 16 Tallman lettering is mixed-case lettering that can aid licensed prescribers in selecting the correct medication.

Fig. 4.

Screenshot showing the similarity in the presentation of low severity alerts and high severity alerts. In this clinic, a food-related alert was presented in the same fashion as a high-severity drug–drug interaction alert (data presented in the figure are imaginary).

Next, for the “visual aspects of the alert” section, most clinics (90%) had the alert appear as a pop-up. The other clinic exclusively used noninterruptive alerts in their system. Common ways clinics made the alert visually distinct included using a signal word (70%) and different colors (60%). To represent the severity of the alert, most clinics (60%) used a combination of colors and icons ( Fig. 5 ).

Fig. 5.

Screenshot showcasing the use of colors and icons to illustrate the severity of the alert. In addition, this alert is an example of corrective action, where the prescriber can discontinue one of the interacting medications within the alert, instead of having to navigate back to the ordering screen (data presented in the figure are imaginary).

In the next section, “content of the alert,” most clinics (60%) did not differentiate between generic and brand names of medications by using only lowercase letters for generic medications. In addition, of the seven clinics that presented alerts in sentence format, 71% (five clinics) had the medications presented at the beginning of the sentence, while two clinics (29%) did not. Regarding the text-based information in the alert, almost all clinics (90%) included the interacting medications and a consequence statement that outlined the potential dangers of ignoring the alert ( Fig. 6 ). Less than half (40%) of the clinics included an instruction statement about avoiding ordering the medications again. Only one clinic (10%) listed recommended tasks for discontinuing the medication order.

Fig. 6.

Screenshot showing text-based information within an alert. The text-based information includes the interacting medications and a consequence statement outlining the dangers of administering these medications together.

Next, in the “actions” section, most clinics (70%) had the recommended action for resolving the alert located in the same alert window. Of the three clinics that did not, two (67%) had the actions in a different window that was separate from the alert itself, and the other clinic had the action located in a different area of the screen. Next, we asked clinics to attempt to bypass the alert, which all the clinics could do (100%). Most clinics (60%) could bypass the alert by providing an override reason or justification. The other four clinics (40%) could single-click to dismiss the alert and move on to signing the order. Next, we asked clinics to accept the alert, and only one clinic's (10%) EHR system had corrective action, which is when the system automatically makes the recommended changes to the order ( Fig. 5 ). The remaining clinics (90%) had to manually return to the ordering screen and change the medication order. Examples of these changes could be changing the dose of the medication or discontinuing the medication that is causing the interaction.

Discussion

In this study, we developed a human factors assessment for medication-related CDS alerts, as well as the results from a ten-clinic pilot that assessed DDI alerts and included the seven leading outpatient EHR vendors. Overall, clinics' DDI alerts adhered to most human factors principles, with a mean overall score of 62% in the assessment. However, there was substantial variability in the “actions” section of the assessment, which focused on the actions clinicians had to take to resolve or reject the recommendations of the DDI alert. In this section, 40% of the clinics could bypass the alert with a single click, while the other 60% required an override reason or justification. This poses a safety risk, as clinics were asked to trigger a high-severity DDI alert. In contrast, clinics performed well in the sections of the human factors assessment regarding the timing of the alert and the visual aspects of the alert. Most clinics delivered these alerts within the line of sight of the clinician, used symbols and icons to make the alert visually distinct from the rest of the screen, and delivered the alerts during the medication ordering process, rather than during signing. Given these results, the integration of human factors principles in the design of the delivery of DDI alerts was variable across the human factors assessment and revealed areas of improvement for improving patient safety and reducing clinician burnout.

Other surveys have been created to evaluate the usability and adherence to human factors of EHR systems. One example is the tool for evaluating medication alerting systems (TEMAS), an evidence-based self-assessment tool that evaluates CPOE systems' compliance with human factors principles. 24 The similarities in testing areas between our assessment and TEMAS include (1) the prioritization of alerts, (2) visual aspects of alerts, and (3) how the severity of alerts is represented. In contrast, the sections regarding actions in response to alerts differ. In our assessment, we specifically focus on what happens after an alert is accepted or rejected, while TEMAS focuses more on functionality (e.g., does the alert provide a function for the user to modify the order within the alert?). 24 Another example of an EHR usability survey has been developed by Pruitt et al, 25 and they used it to evaluate usability and safety within three domains of CPOE: laboratory, radiology, and medication ordering. There is an overlap in content areas of this survey, like the visual display of information. Pruitt et al included areas of assessment where we did not have system automation defaults and data availability. 25 However, our tool assessed specific aspects of the CDS alerts, which Pruitt et al 25 did not address, such as the actions clinicians need to take to resolve the alert. Since our assessment was part of a larger tool focusing on medication-related CDS, we wanted to understand how integrating human factors principles into the design and delivery of medication alerts can affect patient safety.

In the first section of the human factors assessment, “timing of the alert,” most clinics adhered to human factors principles. For example, most of the clinics had the DDI alert presented to the clinician once the interacting medication was selected. By doing this, clinicians can be alerted early in the ordering process and help prevent dangerous medication orders from reaching the signing stage. 16 26 Next, most clinics (60%) only had the DDI alert appear, while 40% of the clinics had multiple alerts appear. For example, in Fig. 4 , along with the DDI alert for clarithromycin and simvastatin, this clinic also received an alert about a “suspected major interaction with food” for simvastatin. During the debrief session, this clinic expressed that this was very common in their EHR system, and when there are several alerts presented at once, the clinician must read through all of them to ensure nothing is missed. In contrast, the DDI alert in Fig. 5 shows that this clinic only received one alert for the drug interaction between clarithromycin and simvastatin. It is important to note that another alert appeared for this clinic, but it was filtered, indicating that the other alert may be of lower priority. 27 In more recent studies, the process of filtering alerts has become more advanced with the use of machine learning algorithms. For example, Liu et al 28 developed a machine learning model to filter medication-related decision support alerts using contextual factors like provider and laboratory features, not commonly used by commercial medication reference databases. If fully implemented, this model could decrease alert volume by 54%. 28 Moreover, most of the clinics in this pilot could not customize their medication reference databases ( Table 2 ), so alternative methods to reduce alert volume could have positive implications for decreasing alert fatigue.

In the next two sections of the assessment tool, clinics used a variety of methods to make the DDI alert visually distinct, as well as represent the severity of the alert. These methods included using signal words, a more prominent and bolder font, and a combination of symbols and colors to make an alert visually distinct from the rest of the screen. Together, these factors made alerts more visible to prescribers, which can improve alert uptake by providers. 26 In addition, these results parallel the initial pilots of I-MeDeSA, where the visibility section of the instrument had the highest scores. 14 Next, the most common method to make the alert visually distinct was delivering the alert as a pop-up and within the direct line of sight of the provider. Given the severity of these DDIs, the delivery of the alert is critical in ensuring alert acceptance by providers. Indeed, Seidling et al 29 quantified the effects of human factors principles on DDI alert acceptance using a logistic regression model, and found that alert display was strongly correlated to alert acceptance. However, interruptive alerts should be used with caution as repeated interruptions to workflows can lead to alert fatigue. 30 If these are used, analysis of the downstream effect on patient outcomes should be performed by health systems. For example, Justice et al 31 determined that their tiered implementation of an active and passive CDS alert for acute kidney injury resulted in an increased discontinuation of nephrotoxins. Thus, monitoring of these outcomes as well as the impact of clinicians' workflows could ensure that CDS alerts are both effective and efficient.

The next section of the human factors assessment focused on the content of the alert. The first half of this section focused on how clinics differentiated between generic and brand names of medications, as well as medications with similar names. Half of the clinics used Tallman lettering when medications that are similar in spelling are placed together in lists during the medication ordering process. According to the Institute for Safe Medication Practices, Tallman lettering allows for medications with similar names to be spelled with a mix of lowercase and uppercase letters (i.e., chlorproMAZINE and chlorproPAMIDE). 32 In turn, this can be used as a method to prevent potential medication errors during the medication ordering process. For example, in a study by Waterson et al 33 that analyzed near-miss errors among lookalike-soundalike medications, found that initial cancellations and re-ordering of the correct medication were mainly due to mistakenly selecting the incorrect medication based on the beginning of the medication name. Thus, the use of Tallman lettering can increase the likelihood that the intended medication will be selected. Next, text-based information within the alert was also assessed. Less than half of the clinics (40%) included an instruction statement, while most (90%) had the interacting medications and consequence statements ( Fig. 6 ). According to the developers of I-MeDeSA, the instruction statement is one of the most critical components of the alert, as it details how to resolve the alert. 16 However, it is important for healthcare systems to be cognizant about the amount of text-based content within the alert, as having too information can also lead to alert fatigue, 19 34 while also ensuring that there is enough important information to allow clinicians to make a decision. 35

The assessment area with the most variability in responses was the section regarding the action clinics took to resolve alerts. When asked to bypass the alert, 40% of the clinics could do this by simply closing out of the window, with no justification or override reason required. This critical safety issue is inconsistent with best practices, as we asked clinics to trigger high-severity DDIs, which should rarely be overridden. It is also important to note that these test orders were taken to signing, indicating that these medications would have been prescribed to patients. Without the necessary safeguards in place in the system, these medication orders could have caused significant patient harm. This result is in parallel with a study by Ratwani et al, 36 where of the usability-related safety events they identified, 17.9% required additional monitoring or intervention to prevent harm. Moreover, in a recent study by Classen et al, 37 the authors found that for every one-point increase in the KLAS ARCH Collaborative's EHR Experience Score, there was a one-point increase in the overall percentage score in the Leapfrog Group's inpatient CPOE evaluation tool, with the DDI order category having the largest association. Next, only one clinic's EHR system had corrective action ( Fig. 5 ), which is the ability of the system to automatically make the recommended change to the medication order. 16 This functionality allows clinicians to make recommended changes within the alert window, without forcing them to navigate back to the beginning process to make individual changes to the medication details. Forcing clinicians to manually make changes with additional navigation could introduce additional opportunities for error since the clinician now has to remember what information needs to be changed. On top of this, if the placement of the actions to resolve the alert is outside the alert itself, it requires more “clicks” and further navigation in the EHR for the clinician, which can decrease the patient–clinician interaction. 38 Overall 30% of the clinics exhibited this, with two of the clinics having the recommended action located in a separate window from the alert.

Limitations

Our study has several limitations. First, we only evaluated this assessment with ten outpatient clinics, so the implementation of their EHR systems may not be representative of all clinics, or even of all the clinics using the same system within an organization. Next, we only tested for DDI alerts. Many other types of alerts clinicians encounter, such as allergy or dosing, can contribute to alert fatigue. Further use and refinement of this assessment could readily make it applicable to other types of alerts. Moreover, not all the testing teams at each outpatient clinic included those involved in informatics research. In turn, some information may have been misinterpreted or missed. In addition, we only assessed the integration of human factors principles in terms of the delivery of medication-related decision support. Other studies have identified several areas within EHRs to assess usability and have found that it could be better. 39 40 41

Conclusion

We developed and piloted an electronic human factors assessment for medication-related CDS alerts. The results from the pilot revealed substantial variation in how outpatient clinics resolve high-severity DDI alerts and how these alerts are delivered to clinicians. Most alarmingly, nearly half of the clinics in the study could bypass these types of alerts without any safeguards. The results also revealed that there were similarities in the delivery and visual aspects of less severe alerts compared to high-severity ones. The results from our study add to the current body of work that links the lack of integration of human factors principles in the design and delivery of CDS alerts, to clinician burnout and patient safety. However, most of these studies have been performed in the inpatient setting. As the Ambulatory EHR Evaluation Tool becomes more widely disseminated, this human factors assessment can be used and further refined for outpatient clinics in their quality and safety improvement initiatives.

Clinical Relevance Statement

Results from this study revealed gaps in the integration of human factors principles in the design and implementation of CDS alerts. Given that in this assessment, we asked clinics to trigger high-severity DDIs, it is alarming that almost half of the clinics could single-click to dismiss the alert without having to provide a reason. Thus, by using this human factor assessment, outpatient clinics can identify these gaps and use their results to improve the design and delivery of their CDS alerts.

Multiple-Choice Questions

1. Of the 10 outpatient clinics assessed in the pilot, what percentage of them could bypass the high-severity drug–drug interaction alert we asked them to trigger?

   1. 50%

   2. 75%

   3. 100%

   4. 0%

   Correct Answer: The correct answer is option c. In the pilot, all of the outpatient clinics could bypass the high-severity drug–drug interaction alert.

2. Of the 10 outpatient clinics assessed in the pilot, what percentage of them had multiple alerts appear for the high-severity drug–drug interaction alert we asked them to trigger?

   1. 0%

   2. 40%

   3. 50%

   4. 85%

   Correct Answer: The correct answer is option b. In the pilot, 40% of the outpatient clinics had multiple alerts appear for the high-severity drug–drug interaction alert we asked them to trigger.

Funding Statement

Funding None.