A review of human factors principles for the design and implementation of medication safety alerts in clinical information systems

Abstract

The objective of this review is to describe the implementation of human factors principles for the design of alerts in clinical information systems. First, we conduct a review of alarm systems to identify human factors principles that are employed in the design and implementation of alerts. Second, we review the medical informatics literature to provide examples of the implementation of human factors principles in current clinical information systems using alerts to provide medication decision support. Last, we suggest actionable recommendations for delivering effective clinical decision support using alerts. A review of studies from the medical informatics literature suggests that many basic human factors principles are not followed, possibly contributing to the lack of acceptance of alerts in clinical information systems. We evaluate the limitations of current alerting philosophies and provide recommendations for improving acceptance of alerts by incorporating human factors principles in their design.

Computerized clinical decision support (CDS) systems have been defined as ‘computer programs that provide expert support for health professionals making clinical decisions’.¹ One type of CDS is the generation of alerts that warn healthcare staff about potential errors, enabling them to make better therapeutic decisions. Many types of clinical alerts are currently incorporated as part of complex CDS systems. Medication-related alerting forms a large part of the CDS in most clinical information systems. Examples of medication alerts in clinical information systems include an alert generated when two interacting drugs are prescribed together or a warning when the maximum dose for a drug is exceeded. Several mechanisms of providing medication-related decision support are available in a care provider order entry (CPOE) system but the specific types of alerts discussed here are visual alerts that may or may not be interactive in nature.

Medication-related alerting used in conjunction with a CPOE system has the ability to prevent dangerous adverse events and serves as an important decision support aid.^2–5 However, clinicians often override the recommendations provided in alerts, even clinically significant ones.^6–10 In particular, generation of an excessive number of alerts results in ‘alert fatigue’ and highlights the importance of reducing the utilization of this mechanism for providing CDS.^{7 11} The goal should be to reduce the number of alerts that are not useful to the clinician. A clinician may choose to override an alert because the data used to drive the alert may be incorrect or not current. However, it is also possible that a clinician might override a potentially useful alert due to alert fatigue. Our goal in this study is to discuss how the latter can be minimized by designing alerts that incorporate important human factors principles into alert design.

In a previous review, Kuperman et al¹² described the state of the art in medication-related decision support and also outlined recommendations for optimizing the effectiveness of medication-related alerting. Recommendations focused on: (1) bridging our gaps in understanding the impact of such alerting on clinician behavior; (2) optimizing alert presentation; and (3) considering the underlying reasons for alert fatigue.¹² An understanding of how to successfully prompt clinicians at the point of care is a complex problem, one that requires consideration of a combination of issues: technological, clinical, and sociotechnical. The technological and clinical aspects have received the most attention, while the sociotechnical aspect of alerting, which is related to the human aspect of the interaction between users and technology, has been relatively neglected.

Medical informatics is a multidisciplinary field that has, in the past, drawn from evidence in other domains.^13–18 In this paper, we begin by summarizing a wealth of research from the human factors literature focusing on computer-based alerting systems from a range of industry contexts. Our aim is primarily to delineate the effects of alert design and implementation parameters on task performance and alert acceptability. We then review the literature on computer-based alerting systems in health care, focusing on medication-related alerting. We conclude by providing actionable recommendations that need further consideration for the successful development and deployment of medication-related alerting in clinical information systems.

Methods

Literature review

We conducted a literature review to identify human factors principles that have been used in alerting systems. In this review, we also identified studies that described how these principles have been employed in the realm of clinical information systems. Two researchers (EH, JE) evaluated the literature from the last 20 years (January 1987 to December 2007) in the following databases: Web of Knowledge, PsychLit, Medline, PsychINFO and Google Scholar. The search terms used were selected to identify, first, all articles related to the design and implementation of alerts, and second, for computerized systems in health care. Search terms included: alarm, warning, alert design, handling, learning, compliance, use and medication-related alerting. These words were then combined with keywords used to describe computerized decision support systems such as electronic prescribing, e-prescribing, reminder systems, decision support systems, and drug alerts and prompts. Studies that appeared relevant were reviewed by two researchers (JE and EH). Furthermore, the bibliographies of extracted articles were reviewed to identify any relevant studies that were not identified through this search strategy. Reviewers examined studies to extract information regarding the design and implementation of the alerting system. In particular, in defining the underlying inclusion criteria for our review, we wanted to extract information surrounding the consideration of human factors principles in the design of the alerts themselves and on the way these alerts were implemented.

Our intent was not to conduct an exhaustive systematic review but instead to provide a literature-based discussion of how human factors principles have been implemented in clinical information systems, especially those related to medication ordering. We selected papers that reflected exemplary current implementations of medication-related alerting and used these to identify the shortcomings of current implementations and suggest actionable recommendations for future alert design and implementations.

Results

Review of human factors principles used in alert design

Alerting systems are implemented as safety mechanisms in a wide variety of industries.^19–24 The basic principles guiding the successful use of alerting systems are consistent across domains. These principles emerge from the discipline of human factors engineering, which applies knowledge about human capabilities and limitations to the design of products, processes, systems, and work environments. The application of these principles across domains improves ease of use and user satisfaction, while reducing potential errors and user fatigue.

An alerting system should be used when it is beneficial to indicate to a user that a particular action exceeds task safety thresholds and may undermine task accuracy, completion or safety. Alerting systems are implemented with the aim of improving task performance, for which careful consideration must be given to the design and implementation of the alerting system itself. While a well-designed alerting system will act as a decision aid and improve task performance, a poorly designed one may become an irritation or distraction to the user and can impede performance. Therefore, successful adoption requires careful consideration of not only the knowledge driving the alerting system but also the human factors principles in alert implementation.

We describe general guidance based on human factors principles that can be employed in the design and implementation of alerting systems. A majority of the alerts related to medication–decision support focus on safety, for example, drug–drug interaction alerts, therapeutic duplication alerts, etc. The guidelines discussed below are also applicable to other types of alerts available in clinical information systems, such as, blood product alerts or alerts related to the absence of allergy information, etc. Alerts in clinical information systems are largely visually displayed, hence the focus of this review is on the design and implementation of visually displayed computerized alerts in clinical information systems. While auditory alerts have been implemented in some clinical information systems, they are not discussed in this review.

Alarm philosophy

The logic that underpins the determination of an event as unsafe is termed ‘alarm philosophy’. A well-documented alarm philosophy is necessary to guide decision-making and ensure consistency in alerting. Documentation of alarm philosophy should include a catalog of unsafe events, an indication of the level of priority (based on the severity of the consequences) of an alert, a description of the logic underpinning the classification of an event as unsafe, and a description of the specific alerts indicating each unsafe event.

We discuss below important considerations that should be kept in mind when developing the alarm philosophy of an alerting system. First, the underlying philosophy should seek to minimize the overall number of alerts in the system and the frequency with which they activate. This can be achieved by ensuring that only safety-critical events are classified as unsafe and requiring an alert. This, in turn, would require an explicit definition of what is meant by a safety-critical event since this can vary based on the judgment of the user. Another important consideration is to assess what corrective actions are going to be required by an alert and how the alert is acknowledged and canceled. It is desirable that the users' response to an alert is the appropriate corrective action rather than an acknowledgment, that is alerts should cancel and reset in response to the appropriate corrective action rather than requiring an acknowledgment from the operator followed by the corrective action. Furthermore, the corrective actions should be easy to perform, facilitating the user's compliance with the alert.

False alarms

Consideration of false alarms is equally as important as the proper ergonomic design of alerts. A false alarm occurs when an alert activates but there is no need for the user to be aware of the event being signaled or to take corrective action. False alarms may be present either because the alerts in a system are too sensitively calibrated, that is they activate before a meaningful safety threshold is exceeded, or because alerts have been matched to events that are basically safe rather than unsafe. Another reason false alarms exist is because the data driving the alert may be incorrect or out of date.

False alarms increase workload and may even cause distraction and lower performance. There is evidence suggesting that people will decrease their response as the false alarm rate increases,^{25 26} and these might not necessarily be the most clinically significant alerts. Recommendations to curb false alarm rates include moving from ‘boundary based’ alarm strategies (whereby an alarm sounds when a given parameter exceeds pre-set limits) to intelligent alarm monitoring systems that monitor several parameters simultaneously and use fuzzy logic-based algorithms to initiate an alert.

Placement

The most important consideration with regard to the implementation of a visual alert is the likelihood that it will be seen by the user. In order to be reliably detected, visual alerts must be placed within an operator's visual field. The normal viewing angle is 15 degrees below the horizontal line of sight. However, this may be unnecessarily restrictive and the viewing angle can be anywhere from 0 to 50 degrees below the horizontal line of sight.²⁷ Once the viewing angle is established, the visual field can be defined. The visual field is usually defined vertically as 25% above and 30% below the normal viewing angle. Sanders²⁸ defined three areas in the visual field. In the stationary field (30 degrees horizontal from the normal viewing angle), targets can be detected without movement; in the eye field (30–80 degrees horizontal from the viewing angle), targets can be detected with eye movements; and in the head field (80–180 degrees horizontal from the normal viewing angle), targets can be detected with head movements. Variability in screen design, dimensions and the frequent use of additional monitors make it difficult to generalize the exact visual field of the user. However, these general guidelines may be useful in assessing what portions of the screen the user is most likely to focus on.

Visual alerts should be located in the visual field in order of importance, so that the highest priority alerts are located in the stationary field, with lower priority alerts in the eye field and head field. Besides being detected, it must also be clear what situation a visual alert is indicating. The proximity compatibility principle of Wickens and Carswell²⁹ suggests that in order to make clear what situation is being indicated by a visual alert, alerts should be grouped meaningfully and should be located in close proximity to the controls and displays relevant to the situation being indicated.

Visibility

Besides being detected, a visual alert must also be legible and visible, ie, bright enough for the user to perceive when it is presented on the screen. There are various factors that need to be considered to improve legibility and visibility. Target size, luminance, background contrast, and lettering characteristics are important considerations, but are context specific and relative to the expected viewing distance and potential exposure time. As a generalization, the size of the target should be increased as viewing distance increases or contrast decreases. Heglin³⁰ reports that a visual target should be twice as bright as the background against which it must be detected. Practical consideration must also be given to position alerts in order to avoid glare and reflection, as both will also reduce visibility.

For text, the same factors determine visibility, but here the size of the letters and the ratios between letter height and width and the stroke width also determine legibility. Letter heights should normally be larger when reading from a visual display unit. A mixture of upper and lower case lettering is easier to read than upper case only³¹; and dark text on a light background is easier to read than light text on a dark background.³²

Prioritization

Visual alerts should be prioritized, and prioritization goes hand in hand with hazard matching as a warning implementation strategy. In visual warning research, ‘hazard matching’ is used to describe the process of matching the appearance of the warning to the level of hazard implied by the situation that it indicates.³³ For visual warnings, priority is normally indicated by color. Colors such as red and orange increase the hazard, as well as the priority, associated with a visual alert when compared with colors such as green, blue and white.³⁴ Colorblind users must be considered when utilizing color to indicate the priority level of an alert.

In addition to color the use of a signal word is particularly relevant to text-based alerts. Providing additional information, signal words can enhance the user's ability to distinguish between alerts. Signal words are the header terms often found at the top of warning labels, terms such as ‘danger’, ‘warning’, and ‘caution’. A wealth of research has indicated that these signal words exist on a continuum of perceived hazard, with ‘deadly’, ‘danger’, and ‘lethal’ rated as implying more hazard than words such as ‘warning’ and ‘caution’, that in turn imply more hazard than words such as ‘note’ and ‘attention’.^34–39 It should be noted that although many standardization bodies recommend the use of the terms ‘danger’, followed by ‘warning’ and then ‘caution’ to denote decreasing levels of hazard, in fact most research suggests that the terms ‘warning’ and ‘caution’ are synonymous in terms of the level of hazard they imply. Riley et al⁴⁰ have indicated that it may be possible to prioritize visual alerts by shape, with angular and unstable shapes (ie, an inverted triangle) indicating higher levels of priority than regular shapes (ie, a circle). In this text, the use of the terms ‘low priority’ and ‘high priority’ alerts refers to how the principle of prioritization was adopted for a specific alert.

Color

Besides indicating priority level, color can be used in other ways to code visual alerts and make them less confusable. For example, visual alerts can also be color-coded by the type of unsafe event or category of risk, by response required, or by their function. The number of colors used to code in an environment should be kept to a minimum (fewer than 10) as incorrect color identifications increase with the absolute number of colors used.⁴¹ Furthermore, color affects the legibility of text-based alerts, with blue and green resulting in the poorest performance and red and yellow resulting in the best.⁴² This is important when considering general screen context, where a low contrast between the target and background screen may result in a textual alert not being easily detected.

Learnability and confusability

Color-coding is one means of making visual alerts more distinct from one another. Another means of reducing potential confusability between visual alerts is to minimize the number of visual features that they share with each other. The fewer features that alerts share, the more distinctive they appear. Sanders and McCormick⁴³ have shown that as the ratio of the number of shared features between alerts increases, so does the potential confusion. Color, shape, and size are the variables commonly manipulated to make visual alerts distinct from one another, although these manipulations must take into account any coding for prioritization.

Textual information

Visual alerts can contain textual information with specific details about the unsafe event. When text is being used in visual alerts, the first issue to resolve is what information should be provided by the text. Research on the information components of textual visual alerts has mainly focused on the context of warning labels.⁴⁴ Recommendations have been made that a warning label should, if possible, have four information components: a signal word to indicate the priority of the alerts (ie, ‘note’, ‘warning’, or ‘danger’), a statement of the nature of the hazard, an instruction statement (telling the user how to avoid the danger), and a consequence statement (telling the user what might happen if the instruction information is ignored). Research in this area has also suggested that when space considerations make it impossible to include all four information components, instruction and hazard statements are the most important to include.

Several other features of wording have been shown to influence the effectiveness or perceived effectiveness of visual alerts. While much of this research has been conducted in the context of warning labels, some general principles might be important in the context of general alert design. Features that increase the effectiveness of visual alerts include using explicit rather than non-explicit terms (eg, smoking causes lung cancer vs smoking is damaging to health) leads to higher ratings of risk seriousness.^{45 46} In addition, it has been shown that the effectiveness of labels can be increased by the use of definitive rather than probabilistic statements,^{46 47} and by including personal pronouns, ie, ‘You must (…)’, in the wording.⁴⁷ Previous research indicates that, for clarity, the order of words should reflect the order of required actions (ie, ‘turn off engine before opening emergency exit’ rather than ‘to open emergency exit first turn off engine’), that affirmative and active sentences are used, and that instructions are standardized and validated for clarity and understandability with the intended user population.⁴³ While these findings may not be completely generalizable to alerting in dynamic computer screens, some of the basic principles are useful in the determination of the text accompanying computerized visual alerts.

Habituation

Habituation refers to a reduction in the behavioral or physiological response to a stimulus that occurs with repeated exposure to that stimulus, if the stimulus does not have consequences for the recipient. If we think of an alert as a stimulus, then it is apparent that habituation has important implications for the design and implementation of alerts.

Habituation predicts that repeated exposure to an alert that does not require a meaningful response will result in a decrease, and eventual elimination of, responding to the alert. The phenomenon of habituation highlights the necessity to reduce false alarm rates in alerting systems. For example, when an alert activates erroneously and then requires no response, the likelihood of a future response to it will be reduced. Indeed, habituation probably explains the reduction in responses to alerts with a high false alarm rate that have been observed.²⁶ The existence of habituation also underscores the importance of an alarm philosophy that does not over-warn. Habituation may also imply that if the alerts that a user is exposed to all look or sound the same, then they will be perceived as the being the same stimulus and that habituation to them is more likely. Current research is limited in its understanding of how varying the appearance or sound of alerts, by priority or risk category, will likely help them to be perceived as different stimuli, reducing the likelihood of habituation.

Mental models

The belief and understanding that individuals have about a particular topic is called their ‘mental model’.⁴⁸ When people interact with technology, they form mental models of the objects with which they interact. Users' mental models of the task are likely to be derived from a variety of sources and contain inaccuracies or missing information. Given that mental models will be part of what drives subsequent behavior, it is essential that alerting systems adequately support users' mental models and correct them as appropriate. In order to achieve this, systems designers must elucidate the mental models that users already have of a given task.^49–52 This will allow systems to support users' assessment of the risks and processes associated with performing the tasks. Systems can be designed to reflect users' mental models by allowing users to express how they conceptualize a system to work.

Proximity of task components being displayed

Research into the design of visual displays distinguishes between different types of tasks being performed by the operator and the different displays that should be designed to support those tasks. Carswell et al⁵³ describe the ‘proximity compatibility principle’, whereby tasks can be conceptualized as being either high or low in proximity. A high proximity task requires the integration of information from different sources, dimensions, or categories for successful task completion, whereas a low proximity task requires information from different sources, dimensions or categories to be judged independently from each other.

The proximity compatibility principle suggests that high proximity tasks should be supported by high proximity displays and that low proximity tasks should be supported by low proximity displays. A high proximity display is achieved by ensuring that the components that are important for task decision-making are represented in the display in a perceptually similar manner. This can be achieved by presenting them as an integrated object and by presenting them close together spatially or temporally. For example, Robinson et al⁵⁴ found that a rectangle showing one task component as length and another task component as height was an advantageous way of displaying multidimensional data in a simulated flight task. While research such as this supports the use of high proximity displays to support high proximity tasks, it also makes the point that a high proximity display may result in poorer performance if the task is low proximity, ie, one that requires task components to be judged independently. The application of this principle is relevant in health care, where it has been shown that anesthesiologists' cognitive demand in interacting with a cardiovascular visualization system could be reduced by placing relevant pieces of information close together.⁵⁵

A number of relevant studies across domains have utilized these human factors principles to be able to improve the decision-making processes of the user. Computer-based alerts, irrespective of the domain for which they are constructed, need to consider the design principles discussed above. In the next section, we describe a literature review we conducted to understand how these principles have been employed in clinical information systems that utilized alerts for providing medication decision support.

Use of human factors principles in medication-related alerting

Development of a sound alarm philosophy and careful consideration of human factors principles form the underpinnings of effective alarm management. Despite their significance these factors have received less importance in the design of alerts in clinical information systems. We provide a literature-based discussion of how human factors principles have been implemented in clinical information systems. We have considered these human factors principles and have the following recommendations for future alert design.

Anton et al⁵⁶ described a computerized prescribing system used in the inpatient renal unit of a teaching hospital, where the alerting messages given to the provider were based on seven levels of urgency. An activity dimension was subsumed into the system, whereby depending on the level of urgency, the user was required to execute one or more different actions. Several limitations, both in terms of the nature as well as the structure, existed in the alarm philosophy. First, a single urgency metric was calculated regardless of the type of risk. Second, the logic for the calculation of this metric was not always apparent to the user. Third, the alerting system itself transgressed basic human factors principles. For example, the system described a category of alerts called ‘red information’. However, despite the use of the word ‘red’, the information itself was low priority, and no action was required. Fourth, missing data were categorized as a high priority alert by default, even though, in practice, missing data could be high or low priority. In addition to these limitations, some of the inferences drawn from the study were not based on empirical evidence. The authors inferred that providers with more experience using this system produced fewer alerts, and the decrease in alerts was interpreted as an improvement. Finally, there was no indication of how the alerting system was calibrated in terms of the relative severity of specific prescribing decisions. If the entire system was calibrated conservatively, then presumably acceptable but risky prescribing decisions would decrease as the propensity of the users to generate fewer alerts increased.

In another study, Kilbridge et al⁵⁷ evaluated inpatient CPOE systems at six medical centers in the USA. One hospital used a CPOE system that was developed in-house, while the remaining hospitals used commercial vendors. These systems were designed to provide alerts for 12 categories of prescribing errors, of which at least 50% were serious prescribing errors. The categories used in this system were: therapeutic duplication, single and cumulative dose events, allergies and cross allergies, contraindicated route of administration, drug–drug and drug–food interactions, four categories of contraindication (patient diagnosis, age/weight, dose-related laboratory studies and dose-related laboratory radiology studies), corollary, and cost of care and nuisance (adverse events with low consequence) alerts. A grading system was developed for these categories, the details of which were not reported in the paper.

Two studies described potentially problematic categories for which electronic prescribing systems should provide alerts. Luo⁵⁸ described nine categories of events which need to be signaled. These were medication withdrawal, missing data, drug–drug interaction, drug interactions with medications already being taken by the patient, rearranging of priorities by the family practitioner because of the system itself, knowledge gaps by the user, over-confidence in the efficacy of the system on the part of the user, data overload, and problems with the system. The study did not suggest that these situations should form part of the alert philosophy, but simply listed these as possible sources of problems. According to Goundry-Smith⁵⁹ medication alerts should be provided for the following categories: allergy checking; drug interactions; duplicate therapy/drug doubling; cautions/contraindications; dose checking; formulary/prescribing status; and monitoring warnings.

Taylor and Tamblyn⁶⁰ identified eight alert categories used by the Medical Office of the Twenty First Century (MOXXI–III) CDS system. In this intervention, CDS was provided to 30 outpatient primary care physicians through a personal digital assistant. The alert categories identified were age-related contraindications, allergies, adverse health contraindications, dosing errors, therapeutic duplication, known patient intolerance, medical interaction, and possible toxicity effects. All of these eight categories were directly related to the drugs themselves. The study showed that different categories of alerts were subject to higher or lower rates dismissal. Users disregard age alerts the least, while ignoring dosing errors and intolerance the most. Alternatively, users demonstrated greater compliance with interaction and adverse health contraindications categories.

van Mil et al⁶¹ illustrated 10 care activity codes assigned by community pharmacists to computer-generated alerts. These categories were: interaction; contraindication; allergy; drug duplication; unclear prescription; questionable strength; dosage different from a previous prescription; drug dispensed for the first time; incorrect patient data; and unusual quantity. Most of the categories corresponded to the alerts issued by the electronic system used in outpatient pharmacies.

Alert philosophies might be based on the most common causes of prescribing error. In a study that examined inpatient orders, Bobb et al⁶² identified the most common types of errors in prescribing: dose; frequency; nomenclature; drug allergy; incorrect medication; omission; duplication; route; and drug interactions. The authors also described a set of provider-related issues which may cause error, such as lack of knowledge of the patient, lack of specific medical knowledge, and mental slips. This study pointed to a general quandary in alert design: how to limit the scope of alert philosophy. For example, should the scope be limited to the patient and his/her drugs or should the scope be expanded to anticipate gaps in the provider's knowledge or compensate for a system's lack of information. Ideally, all of these issues should be considered when the alert philosophy is developed.

Nightingale et al⁸ described the impact on prescribing behavior of a rule-based CPOE system implemented in an inpatient renal unit. Although the details of the alerting system were not described, the alerts were produced in response to a range of the usual prescribing issues, such as drug–drug interactions, allergies, etc, as well as in response to non-prescribing activities, such as password-related alerts. The data also showed that low priority alerts were much more prevalent than alerts for high priority events. An example of a low priority event would be warning a nurse when he/she attempts to prematurely administer a drug, whereas a high priority alert would fire upon ordering penicillin for a patient who has a known penicillin allergy. When developing an alert philosophy, one must carefully consider the threshold of the alert. Previous studies have demonstrated that systems tend to have a low alerting threshold, resulting in physicians receiving a large number of clinically insignificant alerts. In studying an outpatient CPOE system in hospital-based clinics and community-based practices, Weingart et al¹⁰found that physicians justifiably overrode 91% of allergy alerts. Similar to the study by Nightingale et al,⁸ Weingart et al¹⁰ showed that high priority alerts were much less prevalent than lower priority alerts; however, providers overrode nearly 90% of high severity alerts.

In a study describing their experience with the development of the WizOrder system, an inpatient CPOE system developed and implemented at Vanderbilt Medical Center, Miller et al⁶³ described their alerting philosophy with respect to the timing of alerts. The type of alert determined the timing of its appearance in the workflow during CPOE. Rather than firing a drug–drug interaction or allergy alert after a physician has completed a medication order, an alert is fired as soon as the physician first indicates the name of the new medication to be prescribed, enhancing alerting efficiency. On the other hand, following an order for a blood thinner such as heparin, an alert could remind the clinician to order a partial thromboplastin time at the end of the order entry session. Holding such an alert until order completion facilitates a smooth workflow and promotes physician autonomy. Such delayed warnings or ‘exit checks’ give clinicians the opportunity to correct errors spontaneously and prevents overreliance on the clinical information system to make decisions. In addition, the WizOrder system prevents excessive alerting by generating general non-clinical alerts (ie, alerts advising users of new CPOE features) once and then spontaneously removing the user from the new users list.⁶³ Miller et al⁶³ also described the use of effective placement or ‘geographical consistency’ whereby the same types of clinical information are displayed in the same areas of the screen in order to draw attention to important warnings.

In general, the studies reviewed showed that there was a lack of uniformity in the logic for generating alerts. This was exemplified by a comparison done by Fernando et al,⁶⁴ who evaluated four alerting systems and found large discrepancies in alert firing criteria. The study evaluated the safety features of four commonly used primary care CPOE systems in the UK. The discrepancies in alerting arose because of differences in the underlying knowledge bases used in these CPOE systems.

The UK National Health Service Common User Interface Project effectively outlined the general design principles for alerting.⁶⁵ The study employed alerts for six categories: drug information; drug warning; drug critical; information; warning confirmation; and system critical. Only three of these categories (drug information, drug warning, and drug critical) were related to prescribing—the other three categories of alerts were informational. Informational alerts were used to inform the user that referral details had been sent, while warning confirmation alerts presented the user with information about his or her actions in relation to the system. System critical alerts informed the user of important system features. There are two issues that need consideration in this alerting schema. First, the three system alerts are very similar in design to drug-related alerts, potentially resulting in confusion. Also, this schema entails presentation of three different types of alerts (for allergy, drug–drug interactions, and drug contraindications), each with three levels of priority. This requires the alert to provide information about the category of risk being warned about, as well as the priority of the alert within that category. In this case, we recommend that system warnings be easily distinguishable from drug warnings.

Horsky et al⁶⁶ described a cognitive walk-though of a provider order entry system for heparin dosing, drawing attention to a number of deficits in the timing and manner of alert presentation. For example, delayed presentation of administrative guidelines decreased their usefulness, and the graphical representation of data was not well conceptualized. Also, a key piece of information, such as the calculated dose for a patient, was embedded in general text about the administration of heparin. This study highlighted the importance of the principles of proximity and prioritization by elucidating the effectiveness of appropriately timing information presentation and of distinguishing urgent information.

A review by van der Sijs et al⁷ in 2006 considered a number of papers which have looked at the efficacy of alerts. In this study, they found that physicians overrode alerts between 49% and 96% of the time, although the override rates for interactions and contraindications were lower (35% and 43%). Low-level alerts were overridden more often than high-level alerts. The most significant finding, however, was the lack of a systematic standardizing of alerts. This suggests a problem, both in terms of alert philosophy and the design and presentation of the alerts themselves. Van der Sijs et al⁷ point to a number of reasons for a high rate of alert override, all of which are well known in the alarms literature and have been indicated at various points in this report. The most significant contributor to a high override rate was high signal-to-noise ratios, ie, presenting too many low priority alerts. In addition, textual information lacked brevity. Reasons for a high override rate also included the lack of knowledge about the importance of the alert, suggesting that alerts had not been categorized by urgency and coded by color, size, and use of signal words. The authors describe an effective alerting system to be one that is tailored to either or both the user and the patient, has appropriately prioritized alerts which are designed in a way that can be easily understood by the user.

The content of medication alerts has been well studied within the domain of informatics. Nonetheless, very few studies have focused their attention on how these alerts should be communicated to the end-user. Following this review, we developed a set of actionable recommendations for future alerting systems, summarized in box 1. In the next section, we discuss the implementation of some recommendations and the consequent impact on the use of these clinical systems.

Box 1. Actionable recommendations for the design and implementation of future clinical information systems.

1. Alert philosophy: This should specify (as a minimum) which categories of problems should be included in the alerting system, and how many priorities there should be for each category of risk. A distinction should be made between alerts related to medications versus those that relate to system errors.

2. Prioritization of alerts: This should probably include three levels: low, medium, and high and should be coded using word, color, shape, position on screen, and other indicators known to influence urgency.

3. Low priority alerts: These should be avoided or classified as ‘information only’ indicators. Although from a safety point of view more alerts are seen as safer, in practice the reverse is true.

4. Information which is linked in forming a holistic judgement should be linked together perceptually; information which is disparate and needs to be considered separately should be readily differentiable.

5. Information which is contemporaneous should be presented at the same time in the system.

6. Interfaces which are tailored to the user and/or the patient are likely to lead to fewer perceived false alarms, and are likely to be less irritating, and less prone to error.

7. Alerts which require acknowledgement before the user moves on should be kept to a minimum.

8. Auditory alerts might have a very specific role in some circumstances, for example missing patient data, and should be considered for use in combination with visual alerts.

9. The presentation of alert information, and information more generally, should as far as possible match the mental models of the user.

10. The format of alerts should be chosen in order to avoid habituation. Alerts of the same level of severity should be perceived as equally urgent, but different from those of a lower severity or purpose.

Discussion

The medical informatics literature suggests that while there is general agreement about the major drug issues for which clinicians should be alerted, there is little agreement on how these alerts should be generated. In addition, studies evaluating the occurrence of alerts suggest that alerts for low severity events are much more prevalent than those for high severity events, and that users override the vast majority of low severity alerts. Few studies have examined the reason for frequently overriding low severity alerts, but it is most likely due to a conservatively calibrated system. Overly conservative calibrations would trigger an excessive amount of non-severe alerts that the user might consider insignificant. Over time, these excessive and clinically insignificant alerts could encourage users to lose faith in the reliability of the system, increasing the frequency of non-severe alert overrides. An alert philosophy which encourages low priority alerts will lead to many more alerts than one that signals for only high priority problems, since the prevalence of the latter is usually much lower. Given that low priority alerts are generally overridden, an important consideration is whether or not they should be included at all, or assigned to an ‘information only’ category. More research is needed to determine the effectiveness and possibility of causing harm when low priority alerts are not generated in a clinical information system. On the one hand, the removal of these alerts could focus the provider's attention on high priority alerts, while on the other, removing them completely may result in unintended harm in situations where these alerts were meaningful to the provider and the patient in context.

The design features outlined in the UK Common User Interface Project provide a valuable representation of employing human factors principles in alerting. In this project, drug alerts were prioritized into three levels based on severity. The study recommended the following design features: using green, yellow and red to indicate the three levels; presenting the lowest priority information in the periphery of the visual field, but the second and first priority information in the centre of the screen so as to be in the centre of the viewer's visual field; the appropriate use of urgent signal words (‘information’, ‘warning’, and ‘stop’) to differentiate between levels of priority. Finally, alert messages were designed to consist of four components: a signal word; the hazard itself; instructions; and consequences. The standardization of alerts along a common design framework and the use of human factors principles allow the system to alerts users effectively.

Alerts are meant to serve as a safety net for providers, but one key problem with current alarm philosophies is the large numbers of alerts, many of which disrupt workflow and are ignored. Robezieks⁶⁷ highlights the advantages of providing only high priority alerts in order to minimize interruption of workflow. That study also highlights the frustration caused by interrupting alerts which are not relevant to the specific prescribing. Over-alerting as well as lack of tailoring of the system to the specific requirements of the user, or the specific details relevant to the patient being prescribed for, will not only frustrate the user but will lead to the impression that of the system generates an excessive amount of false alerts, which will in turn degrade the integrity of the system. In addition, prescribing medications is an inherently complex task that requires taking into account the physiological condition of the patient along with characteristics of age, disease severity, etc, all of which might not be accounted for in the alert logic. Future alarm philosophies might take into consideration this complexity by assigning a ‘risk level’ rather than a simplistic ‘true/false’ for alerting clinicians.

In general, during alert design, alarm philosophies should consider the characteristics of specificity, information content, sensitivity, workflow, and safe and efficient handling. Alerts should be of clinical significance, denote urgency, and they should be accurate. These recommendations closely follow human factors principles about warning presentation. The studies reviewed determined their alert philosophies in a wide variety of ways. While some complied with basic design recommendations, others were not specific in informing the users about the underlying logic and the consequences of their actions. Designing an alert philosophy should undertake both tasks of outlining the nature as well as the structure of the alerts. Focusing on only one of these characteristics can undermine the effectiveness of alerts.

In this review, we limited our discussion to studies that implemented medication-related visual alerts as this is the most common alert modality used in clinical information systems. Several factors, beyond the human factors principles discussed here, need consideration in alert design. The modality of the alert, medium of presenting the alert, design of accompanying alerts, and the setting in which the clinical information system is implemented all play an important role and require further examination. Future work should explore how alert modality (visual or auditory) would impact acceptance among users because each category has its strengths and weaknesses. While we limited our review to medication alerting, the human factors principles discussed here are applicable to the design of alerts for other types of orders, such as laboratory results, radiology, etc. Also, the design of these non-medication alerts would consequently impact the design of medication alerts. Further exploration is needed to understand how a medium or a combination of mediums for presenting these alerts, ie, pagers, personal digital assistants, phones or other mobile and stationary devices or a combination of audio/visual modalities, may influence alert design. Determination of the optimal modality and medium would depend upon the specific setting in which the clinical information system would be used. Visual alerts may be most appropriate in a setting which is noisy or when noise is particularly undesirable. Also, user response to an alert would be determined by whether other forms of automated decision support are in place. Further research is needed to determine whether specific healthcare settings are better suited to visual, auditory or a combination of modalities, and how alert design can be tailored to suit the needs of the environment in which the clinical information system is implemented.

A number of studies suggest that computerized alerts and prompts can improve prescribing and medication error rates when carefully managed.^68–70 However, these alerts are not a panacea, and many implementations may actually be delivering little or no benefit if active management is not undertaken. This is of particular concern with alerts in CPOE systems that employ commercial drug frameworks to generate alert logic. In these systems users have little or no ability to manipulate alert priorities and content.

Studies thus far have focused mainly on understanding the override rates of alerts as a measure of the efficacy of the alert.^{7 10 71–73} Override rates are accessible and represent a valuable process measure. It will be helpful to assess the relationships between alerts and outcomes.

This review highlights how key human factors principles, such as learnability and confusability, placement, and visibility, etc, have not been adequately considered, implemented or described in the design of alerts in clinical information systems. Despite the usefulness of these human factors principles in other domains, their utility in clinical information systems has not been directly and comprehensively evaluated. While general design principles may be common, their impact on the complex domain of patient care needs further evaluation. Further research is also needed to examine how alert design changes can help or harm the provider's decision-making abilities.

The complex nature of clinical information systems requires careful consideration of human factors principles for future alert design and implementation. A deeper understanding of these design principles in the clinical context might reveal unique characteristics of clinical information systems that cannot be fulfilled by existing knowledge of these principles and might lead to the development of new human factors principles that are specific to the domain of medical informatics.

Conclusions

Despite the wealth of literature and experience available on alert design and implementation, we found that there were major deficiencies in many clinical systems, suggesting that current best practices are not being routinely implemented. Issues relating to what alerts are delivered, and how they are delivered are both extremely important. Addressing these issues requires additional research and stronger consensus by subject matter experts on the criteria that justify alert generation. Additional evidence supporting the ‘how’ of delivering alerts should be obtained to better inform design and implementation of future clinical alerting systems. Better use of human factors principles in the design of CDS systems demands more attention from the medical informatics community and from health information technology developers.