Evaluation of Clinical Decision Support to Reduce Sedative-Hypnotic Prescribing in Older Adults

Abstract

Objective  We sought to characterize the performance of inpatient and outpatient computerized clinical decision support (CDS) alerts aimed at reducing inappropriate benzodiazepine and nonbenzodiazepine sedative medication prescribing in older adults 18 months after implementation.

Methods  We reviewed the performance of two CDS alerts in the outpatient and inpatient settings in 2019. To examine the alerts' effectiveness, we analyzed metrics including overall alert adherence, provider-level adherence, and reasons for alert trigger and override.

Results  In 2019, we identified a total of 14,534 and 4,834 alerts triggered in the outpatient and inpatient settings, respectively. Providers followed only 1% of outpatient and 3% of inpatient alerts. Most alerts were ignored (68% outpatient and 60% inpatient), while providers selected to override the remaining alerts. In each setting, the top 2% of clinicians were responsible for approximately 25% of all ignored or overridden alerts. However, a small proportion of clinicians (2% outpatient and 4% inpatient) followed the alert at least half of the time and accounted for a disproportionally large fraction of the total followed alerts. Our analysis of the free-text comments revealed that many alerts were to continue outpatient prescriptions or for situational anxiety.

Conclusion  Our findings highlight the importance of evaluation of CDS performance after implementation. We found large variation in response to the inpatient and outpatient alerts, both with respect to follow and ignore rates. Reevaluating the alert design by providing decision support by indication may be more helpful and may reduce alert fatigue.

Background and Significance

Medication errors jeopardize patient safety, are costly to the health care system, and are a legal liability for providers and hospitals. 1 Medication errors are defined as “any preventable event that may cause or lead to inappropriate medication use or patient harm while the medication is in the control of a healthcare provider, patient, or consumer.” 1 Approximately 5% of deaths in the United States are thought to be attributed to adverse drug events (ADEs), 2 and ADEs often complicate hospital stays, leading to longer, costlier hospitalizations. 3 ADEs also drive up outpatient healthcare utilization: an analysis of 11 years of data from the National Center for Health Statistics found that there are approximately 4 million outpatient visits each year for which individuals sought medical treatment due to medication adverse events. 4

A seminal report released in 1999 by the Institute of Medicine (IOM), To Err is Human: Building a Safer Health System, noted that electronic health record (EHR) systems had the potential to reduce medication errors and increase patient safety. 5 Two decades later, EHRs and computerized provider order entry (CPOE) systems have found their way into nearly every hospital and feature computerized clinical decision support (CDS) as a standard. 6 7 A systematic review of medication-related CPOE systems with CDS identified five studies where CDS led to a statistically significant decrease in ADEs, and five studies where there was no significant reduction of ADEs after CDS implementation. 8 In clinical practice, CDS alerts are often overridden or ignored, and potential issues with medication-related CDS are not well understood. 9 Given the potential for ADEs in the inpatient and outpatient settings, evaluating the effectiveness of CDS alerts for medications with high risks of ADEs is critical. 10 11

Benzodiazepines and nonbenzodiazepine sedative hypnotics, such as zolpidem, eszopiclone, and zaleplon (often referred to as “z-drugs”), are thought to be high-risk medications in older adults, 12 as these medications can result in increased rates of cognitive impairment, falls, and motor vehicle accidents. 13 Older adults are at higher risk of ADEs due to age-related changes in drug metabolism, increased likelihood of polypharmacy, drug-drug interactions, and overall cognitive decline. 14 15 Since 1991, the American Geriatric Society (AGS) has maintained guidelines known as Beers Criteria which outline inappropriate medication use in older adults. 12 In recent years, the AGS has reiterated their recommendations against using benzodiazepines and nonbenzodiazepine sedatives in adults over age 65 years with specific recommendations that affirm the dangers of long-term use and support discontinuation/deprescription of these medications as treatment for numerous neuromuscular and psychiatric conditions. 12 Additionally, geriatric experts recommend medication tapering, treatment of dependency efforts, and the use of nonpharmacologic options and safer medications, such as selective serotonin reuptake inhibitors (SSRIs), as first-line options for anxiety and insomnia. 13

Given the risks and benefits of benzodiazepines and nonbenzodiazepine sedative hypnotics in older adults and their high rates of prescription, evaluating current CDS interventions aimed at reducing inappropriate prescribing is important. As these interventions are aimed at changing provider behavior, it is important to assess whether they may be effective, unnecessary, or potentially causing harm. For example, benzodiazepines may be appropriately prescribed in treating alcohol withdrawal, severe panic, seizure disorders, or end-of-life care, 16 and an alert would ideally not prevent their appropriate use. Moreover, abruptly withdrawing benzodiazepines in an older adult who has been taking these medications for a long time could cause severe withdrawal effects or even death. 17 The decision to prescribe or not prescribe a benzodiazepine or nonbenzodiazepine sedative to an older adult in the inpatient and outpatient settings is thus complex, as the prescriber must evaluate the indication, concurrent medications, history of use, and contraindications for prescription. 17 18 CDS aiming to improve prescribing would ideally incorporate these nuances. Thus, in this paper, we evaluated two CDS alerts focused on reducing the use of benzodiazepines and nonbenzodiazepine sedatives implemented at a tertiary care academic medical center with associated outpatient clinics more than a year after implementation.

Objectives

The objective of this study was to evaluate benzodiazepine and nonbenzodiazepine CDS by: (1) describing alert follow, ignore, and override rates; (2) examining the distribution of providers who followed or did not follow the alerts; and (3) and examining the reasons for override.

Methods

Study Design

We used a retrospective observational study design. The CDS evaluated in this study includes one inpatient and one outpatient alert. We analyzed all alert events triggered in 2019 for benzodiazepine and nonbenzodiazepine sedative orders for adults 65 and older in the outpatient and inpatient settings. The version of the inpatient alert evaluated was implemented on March 30, 2018, and the outpatient alert version was implemented on June 9, 2018. Both alerts provide clinicians with the same text ( Fig. 1 ) and are triggered when a provider signs an order for a benzodiazepine or nonbenzodiazepine sedative medication for a patient who is 65 years or older. The exclusion criteria for the alert includes the following: (1) the patient has a documented diagnosis of anxiety, seizure, alcohol withdrawal, or status epilepticus; (2) the patient received palliative measures in the previous 7 days; or (3) the provider's specialty is anesthesia, palliative medicine, or pharmacy. Examples of palliative chart elements might include the following: (1) Do Not Resuscitate (DNR)/Do Not Intubate (DNI) orders, (2) palliative care consult, and (3) instructions for comfort care only. Providers can choose to accept, override, or ignore the alert. If providers override the alert, they must select a discrete reason and/or leave comments in a free-text field. The discrete reasons available for overriding the alert are as follows: (1) failed nondrug tx (treatment) and first-line drugs, (2) severe/refractory generalized anxiety disorder, (3) rapid eye movement sleep disorders, (4) withdrawal/delirium tremens, (5) end-of-life care, (6) periprocedural anesthesia, and (7) other (please specify). After a provider takes an action on the alert, the alert is suppressed for that patient's chart for that particular provider for 24 hours.

Fig. 1.

( A ) Depiction of the alert user interface. Shown are the eight discrete reasons for override available to the user. A free-text box for optional comment is shown when the provider chooses a reason. Comments are recorded but are not attached to a provider/patient/order. ( B ) Flow of provider response to generated data. Use of the “Cancel” button at the bottom of the alert closes the alert window without further action required and is recorded as ignoring the alert. Choosing the “Keep” button prompts the user to provide a reason by clicking a discreet reason. A free-text box for optional comment is available. This is recorded as an override. The provider may choose to “Remove” an order or “Accept” the CDS, both these actions are recorded as a followed alert. CDS, clinical decision support.

We examined rates for accepted, overridden, and ignored alerts by clinician. Fig. 1 provides both a depiction of the alert interface and a flowchart that illustrates the categorization and collection of provider response data. We also examined the distribution of concurrent behavior (i.e., providers agreed with the recommendation and accepted the alert) and nonconcurrent behavior (i.e., providers disagreed with the recommendation and selected to override or ignore the alert) ( Fig. 2 ). Confidentiality was maintained by stripping clinician data of all identifiers and assigning each clinician a unique study ID before analysis.

Fig. 2.

This chart enumerates the combinations of possible alert appropriateness and provider response. We define clinically relevant to mean that the alert is raising reasonable concern, and concurrence as a provider following an alert.

Regarding the reasons for alert override, we aggregated data on how often each of the seven discrete options were selected and compiled the list of free-text comments inputted for each alert. We then conducted a multistage iterative card sort method to categorize the override reason. 19 20

Lastly, we examined which benzodiazepine and nonbenzodiazepine sedatives were most likely to trigger the alerts. The study was reviewed by the Institutional Review Board and deemed exempt from human subjects research.

Setting

The study was conducted at a regional health system, Cedars-Sinai Health System, with faculty, private, and salaried physicians who serve over 180,000 patients. The primary medical center has 886 beds and sees 90,000 emergency department visits, 30,000 surgeries, and 49,000 admissions per year. Three affiliated community hospitals within the metropolitan area add an additional 883 inpatient beds.

Results

Provider Alert Response in the Outpatient Setting

When an alert is triggered, the user response is recorded as “Ignore,” “Override,” or “Follow.” In the outpatient setting, of the 14,534 total alerts, 68% were ignored (8,911), 38% had an override (5,503), and 1% (120) were followed. We were not able to ascertain whether the provider changed the order after closing the alert given our limited dataset.

Only 1% of outpatient alerts were followed; most were ignored (68%) or overridden (31%). We found a wide variation in provider alert response and identified significant outliers. The majority of providers in the outpatient setting triggered the alert only a few times. In the outpatient setting, 563 prescribers placed orders that triggered the alert at least once, resulting in a total of 14,534 alerts. A large proportion of providers did not trigger the alert regularly: 75% of the provider population triggered and subsequently did not follow the alert fewer than 26 times in the 2019 calendar year. We also found a small group of providers who triggered the alerts frequently. The top 2% of most alerted providers were responsible for 28% of the nonconcurrent responses (i.e., triggered the alert and ignored or overrode the alert). A total of 56 providers were responsible for 59% of the total nonconcurrent response to outpatient alerts ( Fig. 3 ). The 12 most-alerted providers (top 2.5%) did not concur with the alert more than 200 times.

Fig. 3.

( A ) Outpatient: rate of concurrence as a function of number of alerts received. We defined highly responsive providers to be those who concur with the alerts at least 50% of the time, and high-intensity providers as those who are prescribing 1.5 times over the third quartile of prescribers. ( B ) Outpatient: distribution of clinicians by number of non-concurrent (ignored and overridden) alerts. Descriptive statistics for outpatient providers: Q1: 2 alerts/year Q3: 23 alerts/year, UB: 59.5 alerts/ year. In the outpatient setting, 57 (top 10.5%) of providers are statistical outliers accounting for 60% of total not-followed alerts (8,623/14,414). ( C ) Outpatient: rate of concurrence as a function of number of alerts received. We defined highly responsive providers to be those who concur with the alerts at least 50% of the time, and high-intensity providers as those who are prescribing 1.5 times over the third quartile of prescribers. ( D ) Inpatient: distribution of providers by number of non-concurrent (ignored and overridden) alerts. Descriptive statistics for inpatient providers; Q1: 1 alert/year, UB: 18.5 alerts/ year. In the inpatient setting, 62 (top 9.5%) of providers are statistical outliers accounting for 50% of total not-followed alerts.

We also identified a small group of providers who were highly responsive to alerts. This group of “highly responsive” clinicians were in concurrence with (i.e., agreed with the recommendation) the alerts at least 50% of the time. This group of 13 providers (2%) was responsible for 10% of all of the followed alerts in the calendar year. None of these providers was alerted more than three times.

Provider Alert Behavior in the Inpatient Setting

In the inpatient setting, of the 4,834 total alerts triggered, 60% (2,900) of alerts were ignored, 37% (1,804) were overridden, and 3% (130) were followed. We found that 655 prescribers placed orders that triggered the alert at least once, resulting in a total of 4,834 alerts. Similar to the outpatient provider behavior, 75% of the providers triggered and subsequently did not follow the alert fewer than nine times in the 2019 calendar year. The top 2% of most-alerted providers were responsible for approximately 23% of the total not-followed alerts and the top 63 (9.3%) providers were responsible for 44% of alerts ( Fig. 3 ). The 15 most-alerted providers (top 2.5%) were responsible for 23% of the total alerts. The maximum number of times one provider did not follow an alert was 914 times over the 1-year study period, an average of 2.5 times per day.

In the inpatient setting, the highly responsive provider group was comprised of 28 clinicians (4%) who were responsible for approximately 25% of all followed alerts during the calendar year. None of these providers received more than three alerts in the entire year.

Analysis of Discrete and Free-Text Reasons for Overriding Alerts

When overriding the alert, providers are given the choice to select an option for override. There are several discrete options built into the alert ( Fig. 1 , Table 1 ). The user also had the option of picking an “other (please specify)” descriptor, which chosen in 32% and 59% of all cases in the outpatient and inpatient settings, respectively.

Table 1. Frequency of discrete comments by category in the outpatient and inpatient settings for a benzodiazepine and nonbenzodiazepine sedative clinical decision support alert.

Comment category	Outpatient frequency	Inpatient frequency
Allergy/adverse reaction	5% (20)	10% (44)
Dependency/continuity-of-care	18% (65)	41% (179)
Disease-related (nononcology)	8% (31)	2% (10)
Generalized anxiety/ insomnia	9% (32)	14% (60)
Not available	9% (33)	12% (53)
Neurological/musculoskeletal	10% (36)	2% (9)
Oncology/chemotherapy	8% (29)	1% (5)
Situational anxiety	33% (121)	18% (77)
Totals	100% (367)	100% (437)

Using the iterative card-sort method, we categorized the free-text comments into seven themes ( Table 2 ). Using the same seven broad themes, we sorted the comments from the inpatient and outpatient alerts separately. The comments for both settings were sorted into eight categories: (1) allergy/adverse reaction, (2) dependency/continuity-of-care, (3) disease related (nononcology), (4) generalized anxiety/insomnia, (5) not applicable (N/A), (6) neurological/musculoskeletal, (7) oncology/chemotherapy, and (8) situational/acute use. The categories with the most comments in the inpatient setting were “Dependency/Continuity-of-Care,” while in the outpatient setting it was “Situational-anxiety.”

Table 2. Frequency and examples of free-text comments by category in the outpatient and inpatient settings for a benzodiazepine and nonbenzodiazepine sedative clinical decision support alert.

Comment category	Examples (outpatient)	Examples (inpatient)
Allergy/adverse reaction	“Allergic reaction prevention” “Prn for blood transfusion”	”Administer prior to platelet transfusion” “For iodine sensitivity”
Dependency/continuity-of-care	“Already on” “Continuity-long term use”	“Chronic use” “Home med, requested by patient”
Disease related (nononcology)	“Dizziness with Menier's disease” “MS”	“Acute MI” “Stiff man syndrome”
Generalized anxiety/insomnia	“Long time use for anxiety” “Ambien is not a benzodiazepine and has been tolerated”	“Takes every night” “Insomnia” “Anxious”
N/A	“!” “Not a benzodiazepine”	“MD order” “MD aware”
Neurological/musculoskeletal	“Cervical dystonia” “Spasms”	“Muscle spasm after breast recon”
Oncology/chemotherapy	“Chemo” “Nausea during chemo”	“Cancer dx with anxiety and difficulty sleeping” “Chemotherapy”
Situational anxiety	“Emergency med” “Rare use for anxiety” “Travel - just in case”	“Acute anxiety” “For MRI” “Postsurgery”

Abbreviations: MD, medical doctor; MI, myocardial infarction; MRI, magnetic resonance imaging; MS, multiple sclerosis; N/A, not available.

Description of Medications Triggering the Alerts

In both settings, the majority of medications triggering the alerts were benzodiazepines ( Table 3 ; Fig. 4 ). In the outpatient setting, there were 2.27 times more benzodiazepine orders than nonbenzodiazepine sedative orders associated with alerts. Lorazepam (33%) and alprazolam (26%), both short-acting benzodiazepines, comprised over half of all benzodiazepine prescriptions, and zolpidem (87%) comprised the majority of nonbenzodiazepine sedative medications. In the inpatient setting, there were 1.76 times more benzodiazepine medication orders triggering the alert than nonbenzodiazepine sedatives. Lorazepam (18%) and alprazolam (11%) comprised nearly a third of all benzodiazepine prescriptions, and zolpidem (44%) comprised a little less than half of all nonbenzodiazepine sedative medications ( Table 3 ).

Table 3. Medications triggering the outpatient and inpatient clinical decision support alerts.

Medication	Outpatient (%)	Inpatient (%)
Zolpidem	26.64	32.13
Lorazepam	22.77	23.00
Alprazolam	18.32	14.27
Clonazepam	10.70	8.68
Temazepam	7.68	10.31
Diazepam	7.45	6.79
Eszopiclone	2.41	0.67
Triazolam	1.64	0.62
Zaleplon	1.27	0.09
Flurazepam	0.34	0.02
Estazolam	0.31	0.04
Diphenhydramine HCL	0.21	3.32
Oxazepam	0.09	0.02
Chlordiazepoxide HCL	0.09	0.02
Zolpidem CR	0.04	0.0
Clorazepate dipotassium	0.02	0.02

Fig. 4.

The majority of the medication orders that triggered the alerts were classified as benzodiazepines. However, a noninsignificant portion are nonbenzodiazepine sedatives.

Discussion

Summary of Main Findings

In this descriptive, retrospective study examining two medication safety alerts implemented in an academic health system with associated outpatient clinics, we found a high rate of alert overrides and ignored alerts in the inpatient and outpatient settings. Over a 1-year period, we found that fewer than 3% of alerts in either setting were followed. These are high rates of provider-alert nonconcurrence, that is, where the provider selected a different action from the one recommended by the alert. While this high rate of disagreement could be indicative of inappropriate prescribing, it could also be indicative of an alert that is not useful to providers. For example, the alert could be overly sensitive, capturing a high number of appropriate orders, as well as inappropriate orders. Our analysis of current metrics illustrates a clear need to revise the current alert and provides insight into potential strategies to improve this alert and others to better capture and address the needs, experiences, and behaviors of providers.

Improving Sedative-Hypnotic Medication Clinical Decision Support Alert Design

Modifying existing CDS via more setting- and patient-specific criteria could reduce clinically irrelevant alerts. 21 For example, our findings suggest that there should be separate alerts for benzodiazepine and nonbenzodiazepine medications. Diphenhydramine, a nonbenzodiazepine sedative, is a trigger for the current alert. However, this antihistamine is most commonly used to suppress or treat allergic reactions. Our analysis of the free-text comments found that that a group of override comments are exclusively related to prevention or treatment of allergies and that many providers expressed annoyance at the appearance of irrelevant alerts. This not only increases alert fatigue but also erodes clinician trust in the alert system's accuracy. 21

Distinguishing between incident and prevalent prescription in the alert logic would also be helpful. 22 Instead of presenting the same alert to clinicians who are prescribing a one-time low dose for a sentinel event (e.g., managing anxiety during a long magnetic resonance imaging [MRI] scan) and to clinicians who are continuing a patient's long-term outpatient medications, indication-specific alerts might be more appropriate. For short-term use, the alert could suggest better alternatives. For long-term use, more sophisticated alerts might add functionality to allow the provider to send a note to the patient's primary care provider or the patient's prescriber, noting in the chart that the patient is at high risk for developing a dependence or is exhibiting signs of dependence (e.g., withdrawal symptoms). In the inpatient free-text comments, 179 free-text comments were categorized as continuation of care (i.e., the reason for the prescription was that only that the patient was already taking the medication at home regularly). We also found 16 comments that explicitly stated the patient had a benzodiazepine dependency, and five comments that addressed either a plan to taper or have the patient check in with the addiction medicine team. Interestingly, there were five additional comments that mention significantly reducing the outpatient medication dose in the inpatient setting. This suggests that the opportunity for introducing tapering for patients with long-term benzodiazepine/nonbenzodiazepine sedative use in the inpatient setting is being missed. Future alerts could facilitate involvement of a clinical pharmacist with expertise in tapering who could have a conversation with the patient about deprescribing.

Insights about Provider-Alert Response Behavior

Our results demonstrate that there are two outlier groups of providers as follows: (1) high-intensity providers who are responsible for a high proportion of all triggered alerts, overrides, and ignored alerts; and (2) highly responsive providers who are disproportionately responsible for followed alerts. Previous studies examining prescribing have characterized providers who are less likely to follow prescribing guidelines and have coined the term “high-intensity prescribers” to describe this phenomenon. 23 Furthermore, research has shown that high-intensity prescribers are less likely to follow guidelines across multiple classes of drugs, and more likely to keep their patients on these medications for inappropriately long periods. 24 In this study, we found that high-intensity providers disproportionately triggered the alert and were less likely than their peers to follow the alert. A thorough review of inpatient prescribing might identify providers who would benefit from pharmacist-led academic detailing or training in additional resources to manage anxiety and sleep in the hospital. 25 26

In direct contrast to high-intensity prescribers, researchers have identified low-intensity prescribers who prescribe at a lower volume than their peers. 27 In our study, we found that the majority of providers were not triggering the alert more than a few times per year, suggesting that the vast majority of clinicians are not prescribing benzodiazepines contrary to guidelines. We found a group of providers within the low-intensity prescribers that, while receiving an average number of alerts, was more likely than their peers to follow the recommendations cited in the alert. Given that the overall follow rate was 1% in the outpatient setting and 3% in the inpatient setting, we defined a “highly responsive” provider as a provider that followed the alert at least half of the time. Qualitative interviews with clinicians in the highly-responsive group could reveal valuable strategies for reducing the prescription of benzodiazepine/nonbenzodiazepine sedatives. 24 From the free-text comments, we found that providers are seeing many patients who are chronic users of these medications, and understanding how these highly responsive clinicians are able to successfully manage patient expectations and/or education around these medications would have value beyond simply reducing rates of sedatives in adults over 65 years.

Strategies for Improving Adherence of Provider Responses to Clinical Decision Support Alert

In the past decade, the recognition of high variation in the inappropriate prescription of opioids, antibiotics, and other high-risk medications has led to research in studying effective interventions for modifying prescribing behavior. 28 For example, Meeker et al evaluated several different behavioral interventions to examine the long-term impact of several behavioral interventions on inappropriate antibiotic prescribing. 29 30 This research suggests that consistent peer comparison reduces the rates of inappropriate prescribing a year after implementation. This work is directly applicable to improving CDS for benzodiazepine prescribing. The addition of peer comparison to CDS alerts, combined with provider and patient education, could increase rates of alert adherence and reduce inappropriate benzodiazepine prescribing.

Another effective intervention studied to reduce rates of inappropriate opiate prescription might be notifying doctors about ADEs. 31 In this context, a similar intervention could alert via CDS prescribers if a patient of theirs had fallen recently, noting that the patient fell and was on benzodiazepines. Hospitals often keep track of these patient-safety lapses or near misses, and it would be valuable to look at rates of in-hospital ADEs associated with benzodiazepine in patients over 65 years to see if the rate of these incidents is changing, as another measure of CDS effectiveness.

Postimplementation Evaluation of Alerts

This study adds to the literature by developing a process for evaluating CDS alerts, including evaluating provider behavior and response to the alert quantitatively and qualitatively.

Study Limitations

This study has several limitations. We did not have access to data about providers beyond their orders and alerts behavior. We also did not have patient data associated with orders and alerts. Thus, we could not control for patient volume, patient diagnoses, or comorbidities. Our inability to control for patient volume makes it difficult to draw strong conclusions about the high-intensity prescribers, as these providers could have significantly higher patient volume compared with other providers. We also could not register provider action outside the chart, so if a provider changed the prescription after the chart was closed during the alerted encounter, we were unable to track this action. Given the data limitations, we were not able to determine whether the provider response to the alert was appropriate.

Conclusion

Our findings indicate that CDS alerts in one health system for inappropriate benzodiazepine and nonbenzodiazepine sedative medication use in adults over 65 years are being ignored or overridden most of the time. By examining qualitative and quantitative data of provider response to this alert, we gained insight into specific areas of improvement for alert design and interventions to modify provider behavior. Consequently, we find that postimplementation evaluation of alert metrics is necessary to maximize alert effectiveness. Furthermore, these evaluations may reveal systematic problems in alert design and prescriber behavior that are unaddressed by current CDS implementation.

Clinical Relevance Statement

It is essential to evaluate the performance of clinical decision support (CDS) alerts postimplementation to make CDS clinically relevant and effective. A high degree of alert fatigue may result in missed opportunities for providers to reduce high-risk medication use.

Multiple Choice Questions

1. A “highly-receptive” provider is

   1. a provider that infrequently gets an alert

   2. a provider that has a positive view of clinical decision support (CDS)

   3. a provider that follow alert at least 50% of the time

   4. a provider that is willing to give feedback to CDS designers

   Correct Answer: The correct answer is option c. In this paper, we found that highly receptive providers were more likely than other providers to follow the alert recommendations.

2. Which of the following is true regarding implementation of clinical decision support (CDS) alerts?

   1. A majority of clinicians are “highly receptive” prescribers, and follow alert at least 50% of the time.

   2. A majority of clinicians receive the alert very frequently, while a minority of clinicians receive the alert only a few times a year.

   3. Once implemented, CDS alerts should always be followed by clinicians.

   4. Postimplementation evaluation is necessary to understand effectiveness of alert.

   Correct Answer: The correct answer is option d. In this study, by studying the free-text comments, we found that many providers did not find the alert useful as they were continuing medications that were prescribed in the outpatient setting or prescribing medications for indications such as allergies. Evaluating alerts postimplementation can help identify ways to improve the alert, such as making it more specific to reduce alert fatigue.

3. Alert fatigue can be safely reduced by

   1. Keep implementing new alerts, the more alerts the better.

   2. Designing alerts to be harder to ignore by making them a “hard-stop.”

   3. Reviewing alerts to remove non-clinically relevant alerts.

   4. Allowing the provider to turn off alerts completely.

   Correct Answer: The correct answer is option c. The Agency for Healthcare Research and Quality defines alert fatigue as “how busy workers (in the case of health care, clinicians) become desensitized to safety alerts, and as a result, ignore or fail to respond appropriately to such warnings.” Alert fatigue can be caused by alerts that reduce trust in the system (e.g., nonclinically relevant alerts).