Clinical Decision Support Alert Appropriateness: A Review and Proposal for Improvement

Abstract

Background

Many healthcare providers are adopting clinical decision support (CDS) systems to improve patient safety and meet meaningful use requirements. Computerized alerts that prompt clinicians about drug-allergy, drug-drug, and drug-disease warnings or provide dosing guidance are most commonly implemented. Alert overrides, which occur when clinicians do not follow the guidance presented by the alert, can hinder improved patient outcomes.

Methods

We present a review of CDS alerts and describe a proposal to develop novel methods for evaluating and improving CDS alerts that builds upon traditional informatics approaches. Our proposal incorporates previously described models for predicting alert overrides that utilize retrospective chart review to determine which alerts are clinically relevant and which overrides are justifiable.

Results

Despite increasing implementations of CDS alerts, detailed evaluations rarely occur because of the extensive labor involved in manual chart reviews to determine alert and response appropriateness. Further, most studies have solely evaluated alert overrides that are appropriate or justifiable. Our proposal expands the use of web-based monitoring tools with an interactive dashboard for evaluating CDS alert and response appropriateness that incorporates the predictive models. The dashboard provides 2 views, an alert detail view and a patient detail view, to provide a full history of alerts and help put the patient's events in context.

Conclusion

The proposed research introduces several innovations to address the challenges and gaps in alert evaluations. This research can transform alert evaluation processes across healthcare settings, leading to improved CDS, reduced alert fatigue, and increased patient safety.

INTRODUCTION

Many healthcare providers are adopting electronic health records (EHRs) that incorporate clinical decision support (CDS) to improve patient safety and meet Medicare and Medicaid Stage 1 meaningful use requirements.^1,2 Computerized alerts that prompt clinicians about drug-allergy, drug-drug, and drug-disease warnings or provide dosing guidance are most common.^3,4 Initial research reported that adverse drug events (ADEs) were potentially preventable by alerts and other CDS systems.⁵ Despite such promise, CDS implementations in diverse settings have not consistently improved patient outcomes.^6-9 Alert overrides occur when clinicians do not follow the guidance presented by the alert. For example, an alert may appear when a clinician orders amoxicillin, warning that the patient is allergic to penicillin-class medications. The clinician may accept the alert and cancel the order. Or the clinician may override the alert and order amoxicillin, either because he or she failed to read the alert or because the benefit to the patient outweighs the risk. In most organizations, the majority of daily alerts displayed to providers during the ordering process are overridden, and such overrides may be a barrier to improved patient and process outcomes.¹⁰ Both justifiable and nonjustifiable overrides occur, and detailed evaluation of the alerts and provider responses is necessary to determine appropriateness.^11,12 However, these evaluation methods are labor intensive and difficult to replicate for every alert implemented at individual institutions. More efficient approaches to effectively evaluate alert appropriateness are necessary for optimizing patient safety. This article reviews CDS alerts and proposes novel methods for evaluating and improving CDS alerts that build upon traditional informatics approaches.

CLINICAL DECISION SUPPORT ALERTS

Early reports of ADEs among hospitalized patients indicating that about 28% of ADEs were preventable have elicited substantial research into the use of CDS to prevent patient harm.⁵ Studies have since shown that medication errors, which occur in 4%-6% of orders, can be prevented by computerized provider order entry (CPOE) and CDS.^5,13-18 In one study, the use of CPOE and CDS decreased the rate of medication errors by 81%.¹⁸ Although improved patient safety is a leading motivation for CDS adoption, financial incentives also exist, as CDS has reportedly contributed to substantial savings.¹⁹ Finally, Stage 1 of meaningful use requires institutions to implement drug-drug and drug-allergy interaction checks, implement 1 high-priority condition CDS rule, and track CDS compliance.¹ Multiple CDS approaches exist, including alerts, simple guided-dosing algorithms, order sets, and complex ordering advisors.^20,21 Alerts are implemented in 61%-78% of hospitals and included in all major commercial EHRs to notify clinicians of interactions, changing laboratory values, or other information.^3,4,21,22 On average, as reported in 1 study, clinicians received 56 alerts per day and spent 49 minutes per day processing them, making the alerts a substantial component of the daily care workflow.²³

EVALUATION OF CLINICAL DECISION SUPPORT ALERTS

Alert Overrides

Despite initial reports of CDS success, evaluations of CDS systems have not always demonstrated improved patient outcomes.^6-9 Nonadherence to the alerts by clinicians, also referred to as alert overrides, occurs for 49%-96% of alerts and is a potential barrier to such success.^10,24-33 Although the CDS system may be designed to improve patient safety, it cannot be effective if the alerts are poorly implemented or the clinicians do not change their behavior in response to relevant alerts. Excess alerts, those that are repeated (eg, for each refill of a long-term medication) or not relevant, cause alert fatigue and contribute to alert overrides.¹⁰ Studies examining overrides have used chart review or user feedback to conclude that many overrides are clinically justifiable because of the clinical irrelevance of an alert, known patient tolerance for a drug, or documented clinician intention to monitor the patient, indicating a need for institutions to evaluate alerts to prevent alert fatigue.^{24,25,27,29,32-36} Researchers have used statistical modeling to evaluate possible predictors of alert overrides, including human factors (eg, workflow integration, prioritization), patient and clinician characteristics, triggering substance, alert frequency, response type required, and perceived severity and value.^37,38 Many of these factors significantly contributed to the alert acceptance rate in multivariable analysis, indicating that the modeling approach may be a viable alternative to extensive chart reviews. However, existing predictive models have not yet been shown to distinguish between inappropriate and justifiable overrides.

Alert and Response Appropriateness

Although alert overrides by providers have been the focus of many evaluations, some overrides are justifiable because of clinical irrelevance, patient tolerance, or the provider's documented intention to monitor the patient.^11,12 Likewise, some alerts are inappropriate, and adhering to the alert advice could cause harm to the patient.¹² Detailed evaluations of alert appropriateness are necessary to identify such undesirable, unintended consequences and to institute efforts to mitigate resulting errors.^39-41 In the evaluation framework from Ong and Coiera,⁴² signal detection theory is applied, classifying alerts as hits, misses, false alarms, and true negatives. In another report, Ancker et al described “The Triangle Model,” emphasizing simultaneous, interconnected evaluation of the patient, technology, and organization in conjunction with evaluation of providers' interactions.¹¹ A more relevant framework categorizes alerts as successes, justifiable overrides, provider nonadherence, and unintended adverse consequences through retrospective chart review based on alert and response appropriateness.¹² This approach is advantageous because it accounts for inappropriate alerts that result in justifiable overrides (ie, the clinician correctly disregards the alert advice) or unintended adverse consequences (ie, the clinician follows the alert advice and potentially harms the patient). Although these evaluation methods are necessary for determining the true effectiveness of alerts, they are labor intensive and difficult to replicate for every alert implemented at individual institutions. More efficient, semiautomated evaluation approaches are necessary to understand alert responses and overrides and ultimately to improve patient safety.

Surveillance Tools for Alert Evaluation

To facilitate alert evaluations, institutions have implemented CDS surveillance systems. Zimmerman et al⁴³ displayed retrospective CDS data in a spreadsheet-based dashboard, and Reynolds et al⁴⁴ developed a web-based, graphic dashboard to allow monitoring of order and alert volume by patient location, prescriber type, and alert type. In previous work by McCoy et al, review by an alerts committee or physician-led informatics group provided opportunities to identify poorly performing alerts and make system improvements. A real-time surveillance dashboard displayed lists of patients receiving high-risk medications, CDS interactions, and detailed patient views to clinical pharmacists to augment decision making.⁴⁵ The tool allowed informatics personnel to identify and correct inappropriate triggering criteria in existing alerts through aggregate evaluation of the appropriateness of responses by pharmacists during routine clinical duties.

These studies indicate that web-based surveillance tools can be increasingly useful in the evaluation and improvement of alerts. The surveillance tools may also be beneficial to clinicians, allowing them to review their alert and response histories and empowering them to change their behavior if necessary.

Methods for Improving Alerts

Several projects have attempted to improve CDS alerts and reduce override rates by turning off frequently overridden alerts.^46-49 Duke and Bolchini⁵⁰ developed a model for creating context-aware drug-drug interaction alerts that allowed tailoring alert displays based on relevant patient-specific information, resulting in improved acceptance of the alerts. However, alerts deemed inappropriate in some clinical scenarios (eg, increased international normalized ratio values following administration of warfarin, which may be acceptable for mechanical valve recipients) should also be suppressed. No consistent method exists to avoid false positive alerts (which divert clinician time and attention) and false negative alerts (which silently leave patients at risk) that is generalizable across systems and clinical domains.

A PROPOSAL TO EVALUATE AND IMPROVE THE APPROPRIATENESS OF ALERTS

Predicting Inappropriate Alerts and Responses

To better evaluate and improve CDS alert appropriateness, we first propose the use of the alert evaluation framework developed and evaluated in prior research that utilizes retrospective chart review to determine which alerts are clinically relevant and which overrides are justifiable.¹² The framework classifies alert and clinician response appropriateness, identifying successes, justifiable overrides, provider nonadherence, and unintended consequences (Table 1). This approach aims first to identify predictors of alert and response inappropriateness to eliminate the need for manual reviews, and second to validate our findings in both ambulatory and community hospital settings.

Table 1.

Alert Evaluation Framework

Multivariable binary logistic regression has been applied in several studies to evaluate the association between various clinician, patient, and alert characteristics and overrides.^{24,30,37,38,51,52} High-level characteristics frequently included as covariates in prior studies are listed in Table 2; other studies have evaluated provider-entered override reasons, but these explanations are not routinely collected across institutions. However, as demonstrated in the previously described evaluation framework, effective alert evaluations should assess alert and response appropriateness, not merely alert overrides. By identifying factors that predict inappropriate alerts and responses, informatics personnel can improve alert logic to account for these factors, increasing the specificity of the alerts. As a result of the improved specificity, clinicians may experience less alert fatigue, override fewer alerts, and provide better care for patients with conditions that warrant serious alerts.

Table 2.

Multivariable Alert Evaluation Covariates

Through independent chart review by 3 different clinicians (eg, physicians, pharmacists) using explicit and implicit review criteria and assessing inter-rater reliability using Cohen's kappa statistic, we plan to develop a gold standard for the appropriateness of each alert and clinician response (Table 1).¹² For each alert type (eg, drug-drug, drug-allergy), we then will assess its predictive power to identify inappropriate alerts and responses for each characteristic identified through literature review (Table 2), investigator experience, and collaboration with a human-factors expert. We will explore the use of different predictive models with variable selection, including multinomial logistic regression, using 10-fold cross-validation to split the data into training and test sets.

Novel Metrics for Predicting Inappropriate Alerts and Responses

Although variables traditionally included in the evaluation of alerts have been significantly associated with alert overrides, additional predictors of alert responses may improve the models. Substantial evidence demonstrates that integrating clinical context can increase alert appropriateness and improve alert acceptance.^10,50 The first variable that we will incorporate into the models is the indication of an alerted medication, whether entered manually by the clinician during e-prescribing or inferred from a medication indication knowledge base developed in our prior work.^53-55 The algorithms and back-end knowledge required to drive such integration or allow exceptions in simple rule-based logic are difficult to develop and maintain. We have previously explored and validated several complementary methods for developing this knowledge for use in patient summaries.^53-57 Additional variables derived from these knowledge bases will be included in the predictive models to determine if additional data improve detection of inappropriate alerts.

Prior work also has described methods for determining the reputation of users generating content, most often in the setting of e-commerce ratings, in which the reputation is computed as the proportion of ratings from a specific user that are the same as ratings submitted by other users.⁵⁸ In previous work, we developed a clinician reputation metric to evaluate crowdsourced knowledge about links between prescribed medications and indicated problems that we found to have a specificity of 99.5% and an improved sensitivity (66.3%) compared to alternative measures.⁵⁹ This method can be applied to alert override evaluations: a clinician's response to a specific alert is compared to other clinicians' responses to the same alert, given a similar patient scenario. By considering clinicians as users and alert responses as user-generated content, alert evaluators may adopt similar reputation metrics to identify inappropriate alerts that can be used in the previously developed predictive models.

Designing and Implementing an Interactive Alert Evaluation Dashboard

During previous research, we developed a condition-specific, web-based surveillance tool that allowed clinical pharmacists, informatics personnel, and clinicians to review CDS alert responses in the context of patients at high risk for ADEs.^45,60 Figure 1 depicts the surveillance workflow that is designed to improve patient safety. Although a randomized, controlled trial in which clinical pharmacists used the tool did not reduce ADEs in patients with acute kidney injury, the technology assisted informatics personnel in refining logic to improve the specificity of the CDS alerts.⁴⁵

Figure 1.

Surveillance workflow for improving patient safety. CPOE, computerized provider order entry; EHR, electronic health record.

We propose to develop and implement InSPECt (Interactive Surveillance Portal for Evaluating Clinical decision support), an open-source, EHR-independent dashboard that will incorporate the medication indication and reputation metrics developed in the first phase of the project and will permit further assessment of the use of surveillance in evaluating CDS implementations. InSPECt will consist of 2 view types: the alert detail and the patient detail. The alert detail view displays all logged alert instances and allows reviewers to identify inappropriate alerts at risk of harm, showing details such as alert time; triggering medication(s), laboratory value, or allergy; patient demographics; and clinician name and service. The display can be filtered or sorted on any column. This view will also display a graph of alert rates over time, including total and overridden alerts, and will report the estimated rate of inappropriate alerts using the metrics developed in the first phase of the project. Also within the alert detail view, users will be able to select alternate triggering criteria for the alerts, resulting in updated estimates for alert display, override, appropriateness, and response appropriateness rates. Figure 2 depicts a mockup of the alert detail view.

Figure 2.

Alert detail view mockup.

A mockup of the patient detail view is shown in Figure 3. This view displays a graph of events, such as relevant laboratory values and medications, and a detailed timeline for a patient in reverse chronological order to provide context for the alerts. The timeline will include all orders, problems, laboratory results, and alert interactions documented in the patient's EHR and can be sorted on any column. Reviewers can use the patient detail view to understand clinician actions and patient condition changes occurring in conjunction with alert overrides without having to search a patient's EHR independently.

Figure 3.

Patient detail view mockup.

After development and validation of InSPECt are complete, we will collaborate with CDS managers and clinicians at study sites to review alerts. We will take advantage of the InSPECt interactivity to identify poorly performing alerts and evaluate alternate-triggering criteria that may improve the rate of appropriateness and potentially reduce the rates of overrides and inappropriate responses. We will then work with other information technology staff, clinician leaders, and informatics investigators to design an intervention to improve the alerts and evaluate the effect of the improved alerts on patient, provider, and process outcomes.

CONCLUSION

Despite increasing implementations of CDS alerts, detailed evaluations rarely occur because of the extensive labor involved in manual chart reviews to determine alert and response appropriateness. Further, most studies have solely evaluated alert overrides that are appropriate or justifiable. Prior work is also limited by evaluations from single institutions with locally developed systems that restrict generalizability. Our proposed research introduces several innovations to address the challenges and gaps in alert evaluations; it builds upon the alert appropriateness framework developed previously, adopting predictive models and introducing metrics novel to the biomedical informatics domain that have proven successful in other domains. Expanding prior surveillance methods, we also aim to develop an EHR-independent application that is deployable by any institution using open-source, readily available technologies, validating the results in both ambulatory and community hospital settings utilizing commercial EHRs. Combined, this research can transform alert evaluation processes across healthcare settings, leading to improved CDS, reduced alert fatigue, and increased patient safety.

This article meets the Accreditation Council for Graduate Medical Education and the American Board of Medical Specialties Maintenance of Certification competencies for Patient Care, Medical Knowledge, and Systems-Based Practice.