Table 2.

Summarized usability design principles, and descriptions of the main corresponding flaws. Principles and sub-principles are presented respectively as first and second indents. Principles that have been added or extended to complete the matching process are given in italics. The oblique bars indicate the absence of corresponding flaws

Usability design principles	Summary of corresponding flaws
Meta-principle A: #1 Improve the signal-to-noise ratio by improving the sensitivity and the specificity of the alerting system in order to decrease the number of irrelevant alerts [1, 4, 19, 20, 23] (e.g., system (non-medical) alerts, alerts with little evidence, low clinical relevance or redundant alerts, alerts that require no action).	There are too many alerts [8, 25–29] some are redundant [10, 25, 27, 30, 31] or irrelevant [11, 31], other do not need any action [29]. Potential events are over- or under-detected [10] due to sensitivity/specificity issues [32] or inappropriate triggering thresholds [11].
#2 Check the accuracy of the information retrieved from the CPOE (Computerized Physician Order Entry) /EHR (Electronic Health Record) [19], check whether they are outdated and / or reconciliated [1, 23].  #3 Use adequate evidence-based alert knowledge base [19]. It should be regularly up-dated/maintained [1, 2] (see #13).	Alerts are inconsistent with EHR data [10] especially with lists of patient’s actual medications [10, 12, 32, 33] or with patient’s diagnoses [34]. Medications interactions highlighted by the alerts are unknown in pharmaceutical reference books [27]. Knowledge supporting the alerts is not updated [34].
#4 Increase the variety of the sources of information used in the triggering model (e.g. several allergy bases [19]) and reconcile multiples entries [1]; when data are missing (degraded conditions), the system must continue to function [1].	/
#5 Consider temporal dimensions: interval between drugs’ administration [23]: distinguish “now”, “standing”, and “future” orders, evolution of the unsafe event: increase the severity of the unsafe event if it gets worse [21], time lab tests are overdue [1, 19], interval between the appearance of the unsafe event and the administration of drugs.	The alert is irrelevant because the adverse effect it presents happens too fast to be manageable [29]. The system does not distinguish orders specified as “now” and those specified as “future” or “standing” [31].
#6 Consider patient clinical context [1, 19, 23, 24]: besides the specific drug regimen(s) (e.g. dose, route, duration of therapy, sequence of initiating co-therapy, timing of co-administration), add patient and laboratory data into the expected interaction (e.g. age, gender, body weight, mitigating circumstances, predisposing risks factors, drug serum level, renal function, co-morbidity, and previous experiences). Consider the point during patient’s stay at which the alert is presented.	Medication order checking is not patient tailored [26, 35]: alerts may be valid but not applicable to patient clinical context [10, 31, 36]: e.g. pregnancy alerts for male patients and women of non-child-bearing age [32], no distinction between true allergies and side effects [10, 27]. An alert that is supposed to appear the last day of the stay (which is unforeseeable) appears every day [34].
#7 Consider actions already taken by the provider (e.g. dose adjustment) [21] and provider-specific data [1, 23] (e.g. clinical specialty: some drugs may be used off-label, others may be voluntarily duplicated triggering “duplicate orders” [19]).	Alerts appear while the corrective or monitoring actions have already been taken by the physicians [29]. Some corrective actions that are clinically relevant are not accepted by the system [36]. In some specialties, adverse events are intentional [29, 32] (e.g. psychiatry [32]); in other, drugs are used off-label (e.g. pediatrics [32]). Clinicians already know the alerts [25, 29].
#8 Include fuzzy logic-based algorithms, multi-attribute utility model and filters into the triggering model to change alerts’ activation when certain conditions apply [1, 19, 20]. Define appropriately thresholds to trigger the alerts and to prioritize the alerts according to the patient’s clinical context and the severity of the unsafe event [1, 4, 20, 23] (see #28).	Alerts triggering thresholds are too low [26]. Alerts are not ordered by severity level [22]. Non-significant, or low incidence, alerts are presented [11, 12, 29].
#9 Combine alerts [1]:	/
#10 Recommendations must be combined in a consistent way for patients with co morbidities [4].	/
#11 Aggregate low severity alerts in a single display to be reviewed all at once at a convenient point in the workflow [1, 20, 23].	Alerts are not grouped according to their severity [22].
#12 Support monitoring of the usage of alerts and their customization [1, 2, 19, 23, 24]	/
#13 Allow expert committees in each organization to adapt the knowledge-base and suggestions for action to local practices and guidelines and to remove potential errors from the base. Customizations should persist across version upgrades [19, 23, 24].	Alerts are in conflicts with local and common practices [10, 27, 36].
#14 Monitor alerts’ override reasons: alerts frequently overridden and of little value should be considered for removal or for a change of their presentation format (e.g. intrusiveness) [1, 2, 19, 23, 24]. However, do not eliminate or turn off relevant alerts even for specialists [24].	/
#15 Allow institutional flexibility in determining interruptive vs. non-interruptive alerts [23, 24].	/
Meta-principle B: #16 Support the collaborative work. Advocate a team approach [24], make the alerting system a team player [21].	The system does not inform physicians whether pharmacists review their justification of override and / or find them useful [10].
#17 Provide functions to support the team awareness of the alert management [21, 24]: (see #56).	/
#18 Signal the availability of information to all users [21] (even to non-prescribing clinicians [24]).	/
#19 Display override reasons entered by a physician to nurses and pharmacists in order to allow them to understand the rationale for overriding [1, 19, 21, 23, 24].	Justifications and comments are displayed to no one [37]
#20 Display alerts to concerned clinicians and then to non-prescribing clinicians as a second check [24]. Redirect alerts that do not concern physicians to support staff [1].	Pharmacists receive alerts that concern physicians (e.g. drug interaction [38]). Physicians receive alerts related to drugs administration that concern nurses [11, 29]. Physiotherapists and nurses receive alerts related to the ordering of drugs while they do not prescribe [26].
#21 Display consistently the basic alert content, i.e. the main elements of the alert, amongst all clinicians [21, 24]. Nonetheless, the detailed presentation may differ based on clinicians’ expertise (e.g. pharmacists may need more pharmacological data) [1, 23], on their role (privileges, responsibilities) and on the context of use [24]. Details may be presented upon request [21].	The way data are displayed is not adequate for all clinicians’ types [10].
#22 Share patient clinical information in the alert summary screen with all clinicians (e.g. with pharmacists) [1, 4].	/
Meta-principle C: #23 Fit the clinicians’ workflow and their mental model [1, 2, 4, 20].	Mental model implemented in the system does not fit users' one [12].
#24 Display the alert at the appropriate time during the decision making[1, 2, 24] or later during the medication management process [19].	Alerts appear out of the logical workflow [30], either too late in the ordering process [7, 8, 25, 31, 38, 39] or too early [7, 31].
#25 Alerts must be displayed and fully accessible quickly: screen transition time must be well under a second [1, 2], avoid scrolling and tabs [24].	Alerts appearance is delayed [12]: lags/down-times of 8 sec. [28] up to 15 sec. [10]. Clinicians must explore several parts of the alerts to get all relevant data [9]: they must scroll [10, 11, 22], explore several tabs [9, 40], or find information in tooltips [26] because short versions of alerts are not sufficiently informative [31].
#26 Display alerts over the CPOE/EHR screen in close proximity to the relevant controls and displays [20, 23].	Alerts are outside the region of the screen where clinicians are looking [26, 34].
#27 Streamline users’ workflow in response to alerts [22]. Make the resolution of alerts quick and easy (few steps) through screen operations [1, 22, 24]: cancel or reset alerts in response to the appropriate corrective action, do not require acknowledgment before a corrective action [20]. After the alert is resolved, resume the workflow [19] (see #49).	/
#28 Adapt the display of the alert to its type (medication alerts, system alerts) and its severity [19, 20, 23, 24].	Alerts of different severity levels and of different types are not distinguished [22].
#29 Adapt alert’s format (e.g. color, symbol) and location on the screen [20, 24]. More severe alerts must be placed within the focal region of the user’s visual field in order of importance while non severe alerts must be placed in side regions [1, 20, 23]. Distinguish system vs. medication alert messages [20].	Alerts are not distinguished by severity nor by type [10, 22, 32]. All alerts look the same [22]
#30 Adapt alert’s intrusiveness [1, 4, 19, 21, 23 24]. Interruptive alerts should be reserved for high severity warning and used judiciously: they should require an explicit response. Less important alerts must be displayed less intrusively (e.g. on-demand) as messages not requiring any actions. Do not use pop-up alerts for system messages.	High risk alerts are not seen when not intrusive [31]. On the contrary, low risk alerts that pop-up annoy users. Moreover, non-medication alerts are too intrusive [30] and contribute to desensitize the users [10, 27].
#31 Use a consistent structured alert template across the various systems used by the clinicians [23, 24].	Alerts combined but not structured cause visualization difficulties [10, 12, 27, 34]: users face difficulties to find specific data [11]. Thelack of guidance bother users and their understanding [30, 35, 41]. Useful ondemand information available in EHR is not available in the alerts [22].
#32 Make the alert concise and actionable; the description of the problem should be shorter than 10 words [1, 19, 24, 24]; labels of button must be concise too [1] (see #49).	Alerts contain too much text or extraneous information [10, 25, 30, 41].
#33 Present the most critical information on the top-level of the alert: the unsafe event, its causes and its severity [1, 23, 24]. Then display on-demand (linked) information on background and secondary considerations (contextual information, mechanism of interaction and evidence[19, 24]. The suggestion of action could be presented either at the top level or on-demand [20, 24] (see #36).	No data are highlighted within the alert [9, 10], the alert is on a paragraph form [41].
#34 Use consistent terms, phrases, classifications, colors and definitions (e.g. for the severity) [1, 21, 22].	/
#35 Terminology and messages should suit local conventions [23] and be understandable and non-ambiguous [1, 23].	The message conveyed by the alert is not understandable [10, 12, 40]: the text is ambiguous [41]. Icons used are misinterpreted [28, 41] as well as buttons labels [35].
Meta-principle D: #36 Display relevant data within the alert [1, 20, 23, 24] (see #33). For the relevant tools to propose, see meta-principles F.	/
#37 The cause of the unsafe event and its characteristics (e.g. dose) [1, 19, 20, 23]. Use the medication name as ordered as well as generic drug names when identifying the interaction [24], do not focus on pharmacological / therapeutic classes [24].	Alert does not provide information on why it is triggered [10, 12, 27].
#38 The unsafe event (potential or currently happening) [19, 20, 23, 24]. Do not use generic term (e.g. risk), prefer concrete description [24]. Present the frequency or incidence of the unsafe event [24].	Alert does not inform on the unsafe event [9, 10, 12, 27].
#39 The severity / priority of the unsafe event [1, 20, 23, 24]: use color code and a signal word to inform on the severity [20].	Alert does not inform on the severity of the unsafe event [9, 10, 27].
#40 The mechanism of the interaction (possibly by embedded links) [23, 24].	Alert does not provide information on how the causes of the unsafe event conduct to the unsafe event [9].
#41 Relevant data supporting the decision-making process and the suggestions of action [1, 4, 23, 24]: e.g. contextual information, modifying and predisposing factors (e.g. co-morbidity or lab-values). Provide a link to a summary of patient clinical data [23]. Display data necessary to interpret values provided (e.g. thresholds).	The alert does not provide essential patient information for the prescriber [10]. User can link to outside sources of information from elsewhere in the system, but there is no link within the alert [22]. Even when alerts provide patient biological results, the thresholds to interpret them are not presented [9].
#42 Suggest, do not impose. Make the system a clinician’s partner [21]: provide clinically appropriate suggestions of action for mitigating the potential harm, do not impose [1, 19, 20, 23, 24]. Present possible ancillary orders such as monitoring/surveillance actions, drug alternative (incl. Dose and frequency) and / or order modification or cancellation [1, 2, 23, 24]. Make the suggestions actionable [24] (see meta-principle F). In case of multiple suggestions, prioritize them and present their conditions of application [24]. Justify suggestions [21, 24]: check locally suggestions [24] and include link to institution-specific guidelines [19], make consensual suggestions [1, 19]. Monitor whether or not users followed through with the suggested action they started; if users fail notify them they did not finish the action they started [22].	Alerts do not provide suggestions of action [10, 22, 29, 41] nor alternative treatment options [9]. Alerts provide erroneous suggestions of action [35].
#43 Evidence supporting the alert (incl. Strength and source) using symbols/letter/numbers [24]. Include a link to a more complete documentation (monograph, evidence, extended information, context) [1, 2, 19, 23, 24].	Alerts do not provide existing evidence or the evidence that supports the alert is poor and contradict clinicians’ knowledge [10, 27, 29]. The alert does not present evidence references [9].
Meta-principle E: #44 Make the system transparent for the user. The system must not be a black box and its coverage must be accessible to its user [19]. Inform users when necessary data are missing (e.g. severity of the unsafe event) [24]. Make accessible:	The decision support is invisible to users [42]: capabilities and limitations of the system along with types of data that are checked are not shown to the users [10, 27, 42].
#45 The alerting algorithm / logic / formulas implemented within the system [1, 20, 23].	No alerts are appearing after ordering medications although clinicians expect one to come up for a patient [12]. The calculation formulas that the system applies are not understood by clinicians [7].
#46 A description of the characteristics of the tools included in the alert (e.g. duration of activation).	The system is not explicit about how to use and manage alerts effectively [31]: it does not make it clear that one can turn off some alerts [10] and for how much time [36].
#47 Explanations on the grading systems: levels of severity used by the alerting system (and their number by unsafe event) [20, 24], and explanations of their classification as unsafe events [20]).	The system does not explain the levels of severity that are used [22].
#48 The events that are checked by the alerting system [20] and the type and format of data (e.g., free-text, origin of the data (other hospital data), name of drugs vs. Anatomical Therapeutic Chemical-ATC codes)	The system does not explain which interactions are actually checked [10], which patients’ data are analyzed [31], whether orders in free text are checked [10] and whether checking is based on drugs’ names or on ATC codes [13].
Meta-principle F: #49 Include actionable tools within the alert to allow clinicians to take actions intuitively, easily and quickly [1, 2, 4, 19–24]; display those tools close to the suggestions they are related to [23] (see #27 and #32). The list of actionable tools should include:	There are dead ends in which clinicians face no reasonable options to proceed [28]; there are no useful actionable options [22].
#50 Modify the order being entered (or its dose) [1, 22–24]. For instance, propose formulary drugs lists [19] for formulary drug alerts. Allow ordering a drug suggested [23] or a new drug: in this case, clearly state that the existing drug will be discontinued if the new one is finalized [23], open a pre-populated ordering screen for the new drug [23].	The system does not guide users for switching a medication [41].
#51 Discontinue the preexisting active drug [22–24].	The system does not guide user for discontinuing a medication [41].
#52 Cancel the existing / new order [1, 22–24].	/
#53 Order lab tests or actions for monitoring as justified by the alert [1, 24].	/
#54 Provide patient education [24].	/
#55 Delay the alert for a predetermined amount of time (“snooze” function) [24], allow users to get the alert again.	Alerts cannot be pulled up later, hindering alert resolution [10, 36]. Moreover, it is not possible to get the alert again [11].
#56 Send the alert to another clinician [24].	The system does not support the transmission of alerts to others clinicians [28].
#57 Send the alert into the clinical note template	The system does not allow clinicians to send the alerts into a template for patient’s record [28, 41].
#58 Override the alert (meaning continue ordering, ignoring the alert) [22–24]. Most severe unsafe events must be more difficult to override (e.g. require a second confirmation, or even no possibility of overriding [23]) than less severe ones. Alerts must require the reason for override [24] (especially the most critical ones, optional otherwise [23]. Avoid text entries, propose a list of 3–4 (max 5 items) selectable coded reasons; reasons must be 1–2 word long [19, 23, 24].	The system does not provide appropriate options for justifying overrides [28]. The system is not explicit about the necessity to enter a justification [37]; moreover, free-text entries are not effective in the override justification logic [10] and entering data to justify overrides is seen as time burden [10, 36].
#59 Allow providers to remove redundant alerts for a patient who has previously tolerated a drugs combination (after more than one override) or when providers feel they have sufficient practice and knowledge about this alert or when the alert is outdated for a specific patient [1, 23].	The system does not provide users the possibility to remove irrelevant alerts that therefore continue to appear [28].
#60 Allow users access easily patient’s record from the alert screen to change erroneous data (e.g. allergy) or to add new data. Do not require entering additional data in the alert [1, 19].	The system asks the users to enter data in the alert and then in the patient record, leading to wasting time and double documentation [28].