Optimizing clinical decision support alerts in electronic medical records: a systematic review of reported strategies adopted by hospitals

Abstract

Objective

To identify and summarize the current internal governance processes adopted by hospitals, as reported in the literature, for selecting, optimizing, and evaluating clinical decision support (CDS) alerts in order to identify effective approaches.

Materials and methods

Databases (Medline, Embase, CINAHL, Scopus, Web of Science, IEEE Xplore Digital Library, CADTH, and WorldCat) were searched to identify relevant papers published from January 2010 to April 2020. All paper types published in English that reported governance processes for selecting and/or optimizing CDS alerts in hospitals were included.

Results

Eight papers were included in the review. Seven papers focused specifically on medication-related CDS alerts. All papers described the use of a multidisciplinary committee to optimize alerts. Other strategies included the use of clinician feedback, alert data, literature and drug references, and a visual dashboard. Six of the 8 papers reported evaluations of their CDS alert modifications following the adoption of optimization strategies, and of these, 5 reported a reduction in alert rate.

Conclusions

A multidisciplinary committee, often in combination with other approaches, was the most frequent strategy reported by hospitals to optimize their CDS alerts. Due to the limited number of published processes, variation in system changes, and evaluation results, we were unable to compare the effectiveness of different strategies, although employing multiple strategies appears to be an effective approach for reducing CDS alert numbers. We recommend hospitals report on descriptions and evaluations of governance processes to enable identification of effective strategies for optimization of CDS alerts in hospitals.

INTRODUCTION

Clinical decision support (CDS) in the form of computerized alerts has become an essential component of electronic medical record (EMR) and computerized provider order entry (CPOE) systems in the inpatient setting.^1,2 Alerts are triggered to warn clinicians of potential errors in orders or provide information to assist with decision-making. The majority of CDS alerts are interruptive and require the user to acknowledge the information before proceeding with their work.³ Studies evaluating the effectiveness of CDS alerts report mixed results^4–7 with alert fatigue, alert design, and usability issues contributing to poor user acceptance and uptake of alerts.^8–10

A review cited by 47 papers investigating alert overrides in hospitals found that drug safety alerts were overridden in 49%–96% of cases,⁹ and a systematic review published in 2019 identified the most common barrier to alert acceptance, as cited by prescribers, to be the large number of irrelevant alerts that are presented.¹⁰ Clinicians are more likely to override alerts as the volume of alerts increases and relevance of alerts decreases.^9–11 The high override rates reported in the literature suggest that alerts need to be improved to increase their effectiveness and acceptance and reduce problems associated with alert fatigue.

Developing CDS alerts for implementation into a hospital EMR/CPOE system is a challenging process, as there is a need to interpret, translate, and reach consensus on alert content and what type of alerts are required.^12–14 The CDS life cycle may differ depending on the system and hospital. One example provided by Yoshida et al¹⁵ starts with the submission of a request by individuals or leadership groups to add new CDS, which is reviewed by a CDS committee. Requests are prioritized by the committee and the CDS is designed using tools supplied by the EMR vendor. The CDS is then tested, implemented, and monitored by observing and tracking patterns of firing. Finally, the CDS is evaluated, which can result in revisions or its removal from the system.¹⁵

After CDS implementation, guidelines, regulations, and evidence are continually reviewed. Changing regulations, new evidence, and new guidelines require that CDS alerts be checked periodically and refined accordingly.¹² Monitoring CDS alerts post implementation and robust testing allow the identification of malfunctions and optimization opportunities, and these processes are considered to be essential in maintaining reliable and effective CDS alerts.^12,14,15 The organizational structure of hospitals is complex and varies depending on country and institution type, making governance difficult to measure and understand. However, a review identifying key components of successful transformation change for health information technology found that clear, consistent, and stable governance is needed for transformational change, and ongoing monitoring and evaluation of the established processes is needed.¹⁶ There have been a number of papers recommending governance processes for CDS^17,18 but limited guidance on how this is done—particularly with alerts—in practice.

Hospitals have reported a range of strategies to refine and optimize alerts after implementation. Increasing alert specificity and sensitivity, tailoring alerts to users, and only presenting severe alerts to users are some of the strategies that have been used to decrease the volume of irrelevant alerts presented.^2,19 However, how organizations operationalize these strategies, and make decisions about what alerts should be included in an EMR, is not well known. In this systematic review, we aimed to review the current internal governance processes for selecting, optimizing, and evaluating CDS alerts in hospitals in order to identify effective approaches. This information is useful for hospitals when selecting CDS alerts for implementation and for hospitals that have implemented CDS alerts and are embarking on the process of monitoring alert effectiveness or acceptance.

MATERIALS AND METHODS

Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines were followed for the methodology and reporting of this review.²⁰

Eligibility criteria

Eligible papers described internal governance processes for selecting CDS alerts for implementation or for optimizing existing CDS alerts in a hospital setting. We included papers that focused on any type of CDS alert (eg, drug-related, pathology-related, risk assessment alerts, etc.), provided alerts were embedded in the hospital’s EMR or CPOE system. We included English language papers, published from the January 1, 2010. There was no restriction on paper types (eg, trials, commentaries, case studies, etc, were all included).

Information sources

Online databases Medline, Embase, CINAHL, Scopus, Web of Science, IEEE Xplore Digital Library, CADTH (https://www.cadth.ca) and WorldCat (https://www.worldcat.org) were searched with the assistance of an academic liaison librarian. Two sets of keywords and subject headings relating to (1) CDS alerts and (2) Governance, were defined and searched with “or,” and the sets were combined with “and.” In consultation with the librarian, appropriate search terms were developed for each database to capture relevant papers. The database search terms are provided in the Supplementary Material. The original search was conducted on the December 6, 2019, and an updated search was conducted on the April 8, 2020.

Paper selection process

The paper selection process is depicted in Figure 1. The search results were imported into Endnote X9 referencing software, and papers prior to 2010 were removed. The remaining papers were imported into Covidence where duplicates were removed. Using Covidence, titles and abstracts were independently screened by 2 researchers. Eighty-two papers underwent full-text screening, and any disagreements were discussed until consensus was reached.

Figure 1.

Paper selection process (a = search conducted on December 6, 2019; b = search conducted on April 8, 2020).

Data extraction and analysis

Data were extracted independently by 2 researchers using an excel spreadsheet. General information, such as country, hospital, and alert type(s), were extracted along with data on how alerts were selected or optimized (ie, methods for monitoring and/or adding or removing alerts) and the stakeholders involved in the process. If the paper included an evaluation of changes made to CDS alerts, methods and results of this evaluation were also extracted. The researchers met and discussed any discrepancies until a consensus was reached.

RESULTS

Paper selection

The online database search in December 2019 returned 8687 papers, with an additional 282 papers found in April 2020. After screening, 8 papers met the inclusion criteria and were included for data extraction.

Paper characteristics

The paper characteristics are presented in Table 1. Seven of the 8 papers focused on medication-related alerts, with 4 specifically on drug–drug interaction (DDI) alerts. Most of the studies were conducted in the USA (n = 5). All hospitals used commercial EMR/CPOE systems. Two papers described the use of governance strategies to customize and select alerts prior to implementation as well as ongoing optimization after implementation.^23,28 The remaining papers only described optimizing alerts after implementation.

Table 1.

Paper characteristics

Author	Year	Country	Setting	Time frame when alert strategies were adopted	CDS Alert type(s)
Bhakta et al21	2019	USA	Academic, quaternary care institution	8 months after implementation	12 types of medication alerts
Chaparro et al22	2020	USA	Academic and free-standing children's hospital	13 years after implementation	Best practice advisory alerts (nonmedication)
Liberati et al23	2019	New Zealand	Public tertiary hospitals in a district health board (1300 beds)	Before and up to 2 years after implementation	4 types of medication alerts
Hatton et al24	2011	USA	Large teaching hospital	After implementationa	Contraindicated DDI alerts
Helmons et al25	2015	Netherlands	General hospital (341 beds)	After implementationa	DDI alerts
Parke et al26	2015	USA	Single site medical center	After implementationa	DDI alerts
Simpao et al27	2015	USA	Tertiary care children’s hospital (535 beds)	6–15 months after implementation	DDI alerts
Zenziper et al28	2014	Israel	Large tertiary care hospital	Before, during, and up to 1 year, 8 months after implementation	Dose alerts, Renal dose adjustments alerts, DDI alerts, Duplicate therapy alerts

Abbreviations: DDI, drug-drug interaction; ICU, intensive care unit.

^aSpecific implementation date not reported.

Strategies for optimizing CDS alerts

Strategies employed by hospitals to select and/or optimize alerts are presented in Table 2. All papers described the use of a committee, including expert panels, working groups, and multidisciplinary teams as part of their approach to alert optimization. All papers also used more than 1 method to optimize alerts (Table 2). In hospitals that used clinician feedback (n = 4), suggestions from staff were sent to committees to decide by consensus what alerts should be implemented or modified. Four papers described using alert data (ie, alert firing rates and alert override rates) extracted from the hospital’s EMR/CPOE to inform alert changes.^22,23,25,27 Visual dashboards were used by Chaparro et al²² and Simpao et al²⁷ to monitor and evaluate alert data and track outcomes such as alert and override rates after changes were made.

Table 2.

Strategies employed by hospitals to optimize alerts as reported in 8 included papers

Author	Committee	Clinician feedback	Alert data	Dashboard	Literature and drug references
Bhakta et al21	✓	✓
Chaparro et al22	✓	✓	✓	✓
Liberati et al23	✓		✓
Hatton et al24	✓				✓
Helmons et al25	✓	✓	✓
Parke et al26	✓				✓
Simpao et al27	✓		✓	✓	✓
Zenziper et al28	✓	✓			✓

Four papers^24,26–28 described the use of literature and drug references to inform their decisions on changing alerts. For example, Hatton et al²⁴ reviewed research evidence on each contraindicated DDI pair to determine if circumstances justified concomitant use of the drugs.

Committee members and groups involved in selecting and/or optimizing alerts

All papers reported that committees were involved in the selection/optimization of alerts in hospitals, but there was variability in the professional groups involved and the other groups that oversee this process (Table 3). Pharmacists and doctors were the most frequently reported members, with only Chaparro et al²² not including a pharmacist on their committee. This was also the only study that focused on optimization of nonmedication alerts. Two studies specifically mentioned involving clinical pharmacologists,^23,28 and only 1 mentioned involving a junior doctor.²³ Four papers specified involving informatics experts; Bhakta et al²¹ reported involvement of bioinformaticists, Chaparro et al²² reported a physician informaticist, Liberati et al²³ included a clinical informaticist, and Parke et al²⁶ mentioned the involvement of informatics pharmacists. Hospitals also reported the use of other committees, such as the therapeutics committee to provide oversight or final approval.^23,24,27

Table 3.

Committee members and groups reported to be involved in selecting and/or optimizing alerts in 8 included papers

Author	Committee members and groups
Bhakta et al21	Committee Physician Pharmacists Medication safety officers Bioinformaticists Other healthcare professions involved in medication use process from 8 hospitals Supported by chief quality officer, chief medical informatics officer, EMR analysts, and the system medication safety officer
Chaparro et al22	Committee Doctors: attending and resident physician Nurse practitioner EMR vendor analysts Interruptive alert team (received and prioritized requests) Physician informationist 2 EMR vendor analysts
Liberati et al23	Working group CDS pharmacist (expertise in EMR configuration) Clinical informaticist (expertise in data extraction) Doctors: general physician, clinical pharmacologist, junior doctor Ward nurse Ward pharmacists Medicines and Therapeutics Committee (provided oversight)
Hatton et al24	Committee Clinical pharmacists Pharmacy and Therapeutics Committee (provided final approval)
Helmons et al25	Committee Doctors: hematologist, nephrologist, geriatrician, cardiologist, rheumatologist, neurologist, pediatrician Pharmacist
Parke et al26	Committee 3 clinical pharmacists 2 informatics pharmacists 2 physicians
Simpao et al27	Committee 10 pediatric clinical pharmacists Physician group CDS Committee (provided approval) Therapeutic Standards Committee (provided final approval)
Zenziper et al28	Committee 2 clinical pharmacologists 1 pharmacist

Abbreviations: CDS, clinical decision support, EMR, electronic medical record.

Alert system changes and evaluation

Changes made to alerts following the adoption of optimization strategies are described in Table 4. Six of the 8 papers included in this review reported on the evaluation of changes to improve CDS alerts (see Table 4). Objectives of interventions varied between studies with some focusing on specific alerts and others targeting alerts more generally. For example, Liberati et al²³ evaluated the impact of interventions on a small number of alerts (eg Spironolactone high dose alerts), while Bhakta et al²¹ evaluated the impact of their intervention on all medication alerts.

Table 4.

Evaluation results of interventions

Author	System changes	Evaluation results
Bhakta et al21	Turned off 802 of the 875 moderate DDI alerts deemed unnecessary Reclassified the remaining 73 alerts (8.3%) to the severe category Filtered specific categories of pregnancy alerts Suppressed DDI alerts for medications ordered from electronic order sets and order panels Suppressed duplicate therapy alerts triggered by medications ordered across different phases of care and other medications not available until reconciliation	Reduced alerts from 68 900 to 50 300 per week (27% reduction) DDI alerts decreased; duplicate therapy alerts increased Alerts acknowledged increased from 11.8% to 13.7% Alerts that led to a modification in the medication order increased from 5% to 7.3%
Chaparro et al22	Redesigned alerts (Nielson’s usability heuristics) Reduced alerts by tailoring them to clinician type Modified alerts based on provider feedback	Reduced alerts from 7250 to 4400 per week (39% reduction)
Liberati et al23	Spironolactone high dose alerts set to fire for doses above 100mg daily Ceftriaxone shortage alerts implemented to reduce Ceftriaxone prescriptions Concomitant antithrombotic alerts implemented Fentanyl patch administration alerts revised to fire only when formulation is a patch and not when it is unspecified	Ceftriaxone prescriptions reduced 34 prescriptions that week, which was in the lowest 5th centile in relation to the previous 51 weeks 32% of concomitant antithrombotics alerts resulted in a prescription change False positive Fentanyl patch alerts reduced from 43% to 3%
Hatton et al24	Review of 20 most frequent DDI alerts resulted in 12 downgraded to a lower severity which no longer generated an alert Review of remaining DDI alerts resulted in 44.9% downgraded to a lower severity which no longer generated an alert	Not evaluated
Helmons et al25	Alerts were recategorized into: pop-up alerts for both pharmacists and prescribers pop up alerts for pharmacists but only a yellow exclamation mark on the medicine profile for the prescribers only a yellow exclamation mark on the medicine profile for both the pharmacist and prescriber	Reduced alerts by 55% Reduced time taken for pharmacists to check DDIs by 45%
Parke et al26	Severity ranking of 99 of 201 most frequent DDI alerts was reduced	Ratio of alerts to order decreased from 7:100 to 6:100 Pharmacists’ override responses changed: responses in the “previously tolerated” category increased from 344 to 649 (89% change) after recategorization; the number of “not clinically significant” responses declined from 3269 to 2554 (22% change). No significant difference was detected in the number of reported errors related to clinically significant DDI alerts
Simpao et al27	Deactivated 63 DDI alerts	Alert rate results For pharmacists: reduced from 58.74 alerts/100 orders to 25.11 alerts/100 orders For providers: reduced from 19.73 alerts/100 orders to 15.11 alerts/100 orders Override rate results For pharmacists: decreased from 95.14 overrides/100 alerts to 84.38 overrides/100 alerts (significant change) For providers: increased from 84.22 overrides/100 alerts to 84.91 overrides/100 alerts (no significant change)
Zenziper et al28	Silenced DDI alerts of minor clinical significance before implementation Silenced or modified further 3981 alerts (DDIs, duplicate therapy, dose, and renal adjustment alerts) after implementation	Not evaluated

Abbreviation: DDI, drug–drug interaction.

DISCUSSION

This study systematically reviewed existing literature to understand the approaches reported by hospitals to select and/or optimize their CDS alerts. We identified only a small number of papers, likely reflecting the fact that many internal processes are not described or reported in the literature. The majority of included papers were from the USA and focused on medication-related CDS alerts. Multidisciplinary committees were described in all papers, with doctors and pharmacists most frequently involved in alert optimization. The most frequent changes made to alerts were a reduction in number, reclassification of severity level and redesign of the alert interface. Six papers carried out evaluations of their system modifications, and 5 reported a reduction in alert rate following application of their optimization approach.

All but 1 paper reported on processes involved in optimizing medication alerts, with the majority focusing on DDI alerts. This may be due to the fact that medication alerts are a core feature of many CPOE/EMR systems²⁹ and, as a result, are a major contributor to alert fatigue. The literature also suggests that DDI alerts are often ignored with override rates as high as 95% reported.^9,30–32 Lack of relevance and specificity are frequently cited reasons for the high override rates observed.^10,32,33 Thus, it is not surprising that increasing relevance of alerts to the local context was a primary goal of hospitals adopting alert optimization strategies. By limiting alerts that are triggered based on local context and users, total alert numbers and alert fatigue may be reduced.

All hospitals in this review used multidisciplinary committees that involved end users, such as doctors and pharmacists. There is now little doubt that involving end users in the selection of CDS alerts is beneficial, as acceptance of CDS is strongly linked to user involvement in CDS development and implementation.^10,34–36 For example, a qualitative study investigating uptake of CDS systems found that involving clinicians in the alert selection process validated the CDS in the eyes of the users because the evidence was conceived by them.³⁵ A study comprising focus groups with doctors also found that CDS use was facilitated when CDS had a reliable knowledge base and when trusted peers were involved in its development.³⁶ It is interesting to note that only 1 study in this review specified the inclusion of a junior doctor on their alert-optimization committee, when research has shown that, in some hospital settings, junior doctors (1–3 years postprimary training) are the primary users of CPOE systems.^37–39 In particular, Australian and UK studies have shown that the majority of prescriptions are entered into CPOE systems by junior doctors.^37,38 This may not be the case in all countries and contexts, but it does highlight the potential absence of consultation with appropriate end users when optimizing CDS alerts.⁴⁰

Due to increasing implementation and use of health information technology, a growing number of health professionals—including doctors, pharmacists and nurses—are performing informatics roles,⁴¹ including designing, analyzing, implementing and evaluating information systems to improve patient care and health outcomes and strengthen the clinician–patient relationship.⁴² Health professional informaticians are considered to be key to the success of CDS knowledge management,¹⁸ yet only half of the papers in our review reported using informatics experts in the alert optimization process;^21–23,26 this suggests that health professionals with informatics experience appear to be underutilized in the management and governance of CDS. It is unclear if this is due to a lack of staff with the appropriate expertise; we therefore recommend future research focus on identifying and targeting barriers to clinical informatician involvement in governance processes.

Our review also highlighted that visual dashboards are an innovative way to monitor CDS alerts.^22,27 Due to the digitization of health information, there is increasing data available, including alert data, which can lead to information overload.⁴³ This has led to the emerging field of visual analytics, which allows large quantities of information, such as alert override rates, to be viewed in real time and understood by a broad range of users.^43,44 CDS evaluation methods including chart reviews, observations, user feedback, and statistical modeling are typically labor intensive.⁴⁵ The use of a dashboard allows CDS alert information to be filtered and examined easily and on an ongoing basis with limited resources. Papers in this review reported that the use of dashboards helped hospital CDS committees quickly identify alert types to target for optimization.^22,27

Limitations of this review include the small number of studies included, with most papers being descriptive case reports. Consequently, conducting a quality assessment of the papers was not feasible. Further, this review summarized published strategies only and may be impacted by publication bias. Therefore, internal governance processes identified in this review may not represent processes adopted by all hospitals. Depending on local context of the hospitals in this review, the system changes made were diverse. Unfortunately, this meant that we were unable to compare governance processes to ascertain which was the most effective in optimizing alerts. From information provided in papers, we were also unable to identify whether the strategies were ongoing processes or single instances of alert optimization. Further, some papers reported on the process to change CDS alerts but did not evaluate interventions resulting from the refinement process.^24,28 Without evaluation, it is difficult to know if the changes had the desired effect or resulted in unexpected consequences. For example, after evaluation, Bhakta et al²¹ found that their system modifications resulted in decreased DDI alerts but increased duplicate therapy alerts. Thus, monitoring the impact of system changes is critical for ensuring expected benefits are achieved and unintended consequences are identified and rectified.

CONCLUSION

This review summarized the current governance processes reported by hospitals to optimize CDS alerts. Multidisciplinary committees were the most frequently reported method but often in combination with other strategies, such as consulting literature and drug references. CDS governance committees comprised a range of health professionals with half of the papers specifying inclusion of an informatician. The use of visual dashboards was an innovative way of simplifying complex data to monitor CDS alert rates and impact. Due to variations in system changes and the availability of evaluation results, comparing the effectiveness of different strategies was not feasible. Our study has presented the current state of play as reported in the literature, but we recommend hospitals describe and report both successful and unsuccessful governance processes to enable identification of effective approaches or combinations of strategies for optimization of CDS alerts in hospitals.

FUNDING

None declared.

AUTHOR CONTRIBUTIONS

All authors contributed to the conception and design. BV, with assistance from an academic liaison librarian, conducted the database searches. BV, WYZ, and VS conducted title/abstract and full-text screening. BV and WYZ conducted data extraction. BV wrote the initial draft and WYZ, VS, and MB provided critical review. All authors approved the final version for publication.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.