What Evidence Supports the Use of Computerized Alerts and Prompts to Improve Clinicians' Prescribing Behavior?

Abstract

Alerts and prompts represent promising types of decision support in electronic prescribing to tackle inadequacies in prescribing. A systematic review was conducted to evaluate the efficacy of computerized drug alerts and prompts searching EMBASE, CINHAL, MEDLINE, and PsychINFO up to May 2007. Studies assessing the impact of electronic alerts and prompts on clinicians' prescribing behavior were selected and categorized by decision support type. Most alerts and prompts (23 out of 27) demonstrated benefit in improving prescribing behavior and/or reducing error rates. The impact appeared to vary based on the type of decision support. Some of these alerts (n = 5) reported a positive impact on clinical and health service management outcomes. For many categories of reminders, the number of studies was very small and few data were available from the outpatient setting. None of the studies evaluated features that might make alerts and prompts more effective. Details of an updated search run in Jan 2009 are included in the supplement section of this review.

Introduction

Prescribing medication represents an opportunity for providing appropriate and effective treatment, and yet medication errors are responsible for considerable levels of morbidity and mortality. Many errors which result in harm can potentially be prevented by the use of appropriately designed computerized clinical decision support (CDS), in particular computerized alerts and prompts.

Background

Medication errors are common and cause substantial harm; the injuries they cause are called preventable adverse drug events (ADEs). Studies have estimated that 2.4–3.6% of hospital admissions are caused by ADEs of which up to 69% would have been preventable. ¹ The incidence rate of ADEs in hospitalized patients is thought to be between 0.7 and 6.5% of which the preventable proportion reported has ranged from 17 to 56%. ² Since the main error in most preventable ADEs occurs at the stage of drug ordering ³ increasingly sophisticated clinical computer systems have been seen as a major opportunity to prevent inappropriate prescribing.

Clinical provider order entry (CPOE) systems are computer-assisted information systems that enable providers to write all orders, including prescribing medications. Computerized decision support systems (CDSS) are linked with CPOE to provide support for decision-making in patient care by integrating clinical and patient information. They range from simple to more complex decision support systems, ⁴ with all the more complex using patient-specific data and some of the most complex involving detailed clinical protocols and patient specific data. The category of decision support used and its timing within the clinical workflow are important questions to be addressed when CDSS are implemented. ⁵ Reminding and alerting represent an important category of providing decision support to the clinician at the point of care. In the area of prescribing, drug reminder or drug alerting systems generate information that warn healthcare staff against potential prescribing errors or support them in their prescribing decisions.

Several systematic and narrative reviews of aspects of CDSS in healthcare have shown general benefits, such as improved practitioner performance and reduced error rates. ^2,6–14 A Cochrane review on computerized dosing advice even showed moderate but significant reduction in hospital stay. ¹⁵ However, none of the reviews have specifically evaluated the impact of different alerting and reminding systems in computerized medication prescribing.

Therefore, we performed a systematic review to gather evidence of the types of alerting and reminding mechanisms that are in use. Our overall objective was to evaluate the impact of these systems on clinicians' prescribing behavior. In the literature, there are no clear distinctions made between “reminders”, “alerts”, and “prompts”. These terms will therefore be used interchangeably throughout the review.

Methods

Literature Search

We searched CINHAL (1982 to May 2007), EMBASE (1980 to May 2007), Inspec (1969 to May 2007), MEDLINE (1950 to May 2007) and PsychINFO (1967 to May 2007) for empiric studies of computerized prescribing alerts and prompts. Our search strategy combined several keywords around electronic prescribing, reminder systems and decision support systems. Articles from all languages were included. The bibliographies of obtained articles were used to identify further relevant studies following guidelines about study selection from the Cochrane Handbook of Systemic Reviews. ¹⁶ Previous literature reviews, in particular, were searched in full and any relevant references were added to the electronic selection process if considered appropriate. We rerun the search in Jan 2009 and found two recent studies that would qualify for inclusion in this review. Details concerning these studies, a summary of results and the updated search process can be found in the online appendix of this review (supplements 1–4).

Study Selection

Reference Manager (RefMan Version 10) was used to manage data and eliminate duplicate references. The identified references were distributed among four reviewers (AS, VP, CM, AA) who screened titles and abstracts for relevant studies. Relevant publications were obtained as full text articles and all were assessed by the four reviewers independently applying the following predetermined inclusion criteria:

• • Empiric trials

Studies were included if they were randomized controlled trials (RCTs), time series analysis and before-after studies.

• • Health care professionals prescribing drug treatment:

Studies had to target clinicians such as doctors, nurses and other health care professionals, who were actively involved in the prescribing and administration of medications for patients of any age receiving drug therapy for the treatment of a disease or for prevention of a medical condition.

• • Prompts at point of prescribing

Interventions of interest were computerized alerting or reminding systems consisting in electronic alerts, reminders or prompts related to electronically prescribed medications of a particular drug or class of drug, appearing on the computer screen at the point of prescribing.

• • Altered prescribing

Outcome measures were any changes in prescribing related to clinically relevant outcomes such as reduction in medication errors, reduction of risk or harm to patients, decrease of non-recommended medication, increase of recommended medication and increase/decrease of laboratory abnormalities with potential of adverse drug events.

An article was included if it met all the inclusion criteria listed above. Uncertainties or disagreements at any stage were resolved by discussion among reviewers.

Data Extraction

Study descriptions and outcomes were extracted using predefined collection tables that had been developed for the review. Data were extracted by four reviewers (AA, VP, AS, CM). Extracted data were verified by two reviewers independently (AS, CM).

Classification of Alert Types

We applied a classification system based on a modified version previously proposed by Kuperman, et al. ⁴ This allowed us to group different types of prompted CDS according to their characteristics and to compare effectiveness between the groups. There were three main categories of alerts and prompts: Basic drug alerts consisting of drug allergy warnings, drug–drug interaction warnings, duplication warnings and basic medication order guidance; advanced drug alerts including drug-laboratory alerts, drug-condition alerts, drug-formulary alerts and dosing guidelines; complex alert systems representing a set of CDSSs containing features of both basic and advanced alerts (see ▶ for more details).

Figure 1.

Categories of drug alerts.

Data Analysis

Alerts and prompts applied in the studies were grouped in tables and results were compared independently. On the basis of study designs, characteristics of population involved, medication prescribed, types of alerts used and outcome measures reported it was decided whether pooling of outcome data would be appropriate.

Quality Assessment

Studies were classed according to the appropriateness and quality of their design. Criteria such as study duration, similar before/after outcome assessment and before/after group comparison were applied to evaluate the quality of before-after studies. Randomized controlled trials were assessed considering their main sources of bias including selection bias, performance bias, attrition bias and detection bias. Please see tables in supplement 4 for more detail.

Results

Description of Studies

The search identified 14,137 publications of which 14,036 did not fulfill our inclusion criteria when assessing title and abstract (not an experimental trial, not concerned with alerts, not related to e-prescribing of a particular drug/class of drug and not concerned with change in clinician behavior). The 101 remaining full text articles were retrieved and 81 studies were excluded. The 20 remaining studies evaluating 27 different types of alerts addressed our objectives and were included in the review (see flow chart in ▶).

Figure 2.

Flow chart detailing process of identification and selection of relevant papers.

The geographic setting of most studies was the United States (18 studies). One study came from France and another from The Netherlands. ^17,18 Five of the studies took place in a primary care or outpatient setting. The remaining 15 took place in a hospital/inpatient setting; hence most of the providers were hospital doctors. In some studies hospital nurses and nurse practitioners contributed to the pool of providers. ^18–22

The medications prescribed in the studies varied widely. Some studies focused only on one class of drugs, such as sedatives, lipid lowering drugs, ^18,19 asthma treatment, ¹⁸ or antibiotic prescribing. ^18,20,23 Five studies evaluated more complex alert systems involving a wide range of prescribing situations. ^9,24–28

Four out of 20 studies evaluated clinical outcomes such as drug-induced renal impairment, ²⁹ serious adverse drug events, ²⁵ prescribing related falls ³⁰ and length of hospital stay. ³¹ Cost savings through modified prescribing were reported by two studies ^21,32 (see supplements 2 and 4).

Heterogeneity

Studies showed considerable variation in characteristics of providers, patients, and health care settings, types of alert interventions and time of observation. More importantly, there was a lack of consistency in the reporting of outcome measures. Studies looked at optimal, suboptimal, dangerous and harmful prescribing using outcome measures such as serious or nonserious medication errors, actual or potential adverse drug reactions, prescribing patterns as per guidelines, increased prescribing of recommended drugs, and decreased prescribing of nonrecommended drugs. As a consequence, pooling of the data or direct comparison between studies was inappropriate. Instead, we classified studies in groups to allow independent comparison.

Methodological Quality

An RCT represents the most robust method to evaluate the effects of an alert system. In these RCTs, some of the prescribing units are generally randomized to the newly implemented alerting system while the remainders are on the prealert system. Clustered randomized trials in particular have the ability to overcome potential for contamination but require large sample sizes. However, at times when a new and complex computer alert system is introduced in a health care setting, conducting an RCT is often not feasible. This may explain why only four out of 20 studies used RCT designs. 18,19,22,23 These varied in quality with no trials meeting all the criteria to avoid bias (see supplement 3). One study did not provide the appropriate data to evaluate the entire effect of computer prompts. ¹⁹

Other methods to compare pre- and post alert prescribing behavior are more open to bias. Quasi experimental designs used in the studies of this review were uncontrolled before-and-after-studies, time series analyses and interrupted time series analyses. Interrupted time series analysis (ITS) in particular is a practical and valuable tool to determine the impact of a complex intervention, both immediately and over time. Multiple measurements are taken before and after an intervention and are tested for their serial correlation. If sequential prescribing data confirm an immediate and long-lasting effect following the introduction of a new alert system, the prompt system is likely to be responsible for the change.

To account for temporal changes in prescribing patterns, such as seasonal prescribing changes, it is desirable to have long sequential baseline and follow up periods. Other important validity factors are similarity of control and intervention group and continuity of pre- and post alert outcome assessment. Four studies used interrupted time series analyses, 21,26,33,34 six studies used time series analyses 17,25,28–31 and six used before-after designs. 20,24,26,27,32,35 Four authors refined the time series analysis approach by alternating control periods and intervention periods to control for other influences. ^17,29–31 Important validity criteria were fulfilled by most studies (see supplement 3).

Change in Prescribing Outcomes

Some of the 20 studies included in this review evaluated more than one type of alert and were therefore classified in more than one category. ^24,28,31 In total, 27 types of alerts and prompts were assessed in terms of their ability to improve prescribing. Twenty-three of the 27 alerts resulted in statistically significant improved prescribing behavior and/or reductions in medication errors. Most of the changes noted were clinically relevant. The remaining four showed statistically nonsignificant improvements in prescribing ^18,24,33 or inappropriate study design ¹⁹ (see ▶ and supplement 2).

Table 1.

Table 1 Category of Alerts and Types of Alerts Showing a Beneficial Effect on Prescribing and/or Clinical Outcome

Category of Drug Alerts	Number of Alert Types	Number of Studies with Beneficial Effect on Prescribing	Number of Studies with No Effect on Prescribing	Number of Studies with Positive Effect on Clinical Outcomes
Basic drug alerts
1. Drug allergy warnings	1	1 24	—	—
2. Drug–drug interactions	1	—	1 24	—
3. Duplication alerts	0	—	—	—
4. Basic medication order guidance	2	2 24,28	—	—
Advanced drug alerts
1. Drug-laboratory alerts	4	4 26,27,31,36	—	1 31
2. Drug-condition alerts
2a Drug–disease contraindication	0	—	—	—
2b appropriate prescribing	4	2 17,28	2 18,19	—
2c drug-age alerts	4	3 30,34,35	1 33	1 30
3. Drug-formulary alerts	4	4 21,23,28,32	—	2 21,32
4. Dosing guidelines based on:
4a Renal Function	2	2 29,31	—	1 29
4b age	0	—	—	—
4c pregnancy/childbearing potential	0	—	—	—
4d Pediatric patients/weight based dosing	1	1 20	—	—
4e indications	0	—	—	—
Combination of basic and advanced features	4	4 22,24,25,28	—	—
Total	27	23	4	5

Alert types showing beneficial effect on prescribing (second column), no effect on prescribing (third column) and positive effect on clinical outcomes (fourth column) compared to total number of alerts in a certain alert category (first column).

Change in Clinical Outcomes

Of the four alert types that examined clinical outcomes, three had a positive and statistically significant impact leading to decreased prescribing-related renal impairment, ²⁹ fewer falls in elderly people ³⁰ and a reduced length of hospital stay. ³¹ One alert achieved a reduction in serious and life-threatening adverse drug events, but there was no p value reported. ²⁵ Financial implications of two alerts were evaluated and both achieved annual cost savings of up to US$36,500 ³² and US$6,400. ²¹

Types of Alerts and Changes in Outcomes

Applying the Kuperman framework ⁴ we classified 27 types of alerts and prompts used across the 20 studies according to their categories and subcategories of alerts as outlined above.

Two papers investigated the effects of four types of basic alerts of which three reported statistically significant beneficial effects on prescribing (table, Supplement 2):

• • Drug allergy warnings decreased allergy error events by 56% (p = 0.009).²⁴

• • Providing default dosing via basic medication order guidance alerts resulted in reduced dose errors in two studies of 23% (p = 0.02)²⁴ and 71% (p = 0.0013).²⁸

• • The 40% reduction in error events achieved by drug-drug interaction warnings did not reach statistical significance (p = 0.89).²⁴

All the 20 papers evaluated more advanced alert types and statistically significant effects were shown in 21 out of 23 types across five categories:

• • Four drug-laboratory alerts increased appropriate prescribing from 30 to 53% (p < 0.001),³¹ enhanced prescribing of electrolyte supplements from 6 to 35% and 5–49% (p < 0.01),²⁶ reduced the use of contraindicated medication in renal insufficiency from 89 to 47% (p < 0.0001)³⁶ and prevented hazardous prescribing by 50% (p = 0.03).²⁷ One study achieved a positive clinical outcome by an alert resulting in a small reduction in length of hospital stay from 4.5 to 4.3 days (p < 0.009).³¹

• • Five out of eight types of alerts investigating the effects in relation to patients' medical conditions or age showed beneficial effects that were statistically significant. Drug-condition alerts increased prescribing for venous thromboembolism prophylaxis in patients at risk from 83 to 95% (p < 0.001)¹⁷ and 24–47% (p < 0.001).²⁸ Three out of four drug-age alerts for older people showed a reduction of nonrecommended drugs by 18% (95% CI 24–13%),³⁵ 22% (p = 0.004)³⁴ and 30% (p < 0.001).³⁰ In addition, a reduction in falls from 6.4 to 2.8 per 1,000 patients was achieved by one alert type (p < 0.001).³⁰ Prescribing of lipid lowering medication was increased in a study using computer prompts in combination with a complex intervention package, but the study design did not allow evaluation of the alert effect on its own.¹⁹ Prescribing benefit did not reach the significance level in two studies. One was underpowered¹⁸ and the other was poorly designed comparing a refined age specific reminding system with a very similar, less sophisticated application.³³

• • The investigation of drug-formulary alerts was mostly based on cost implications. All four alert types had the intended effect on prescribing21,23,28,32 and reductions in costs were reported by two.^21,28 Each alert achieved changes in specific formulary items such as reductions in methylprednisolone prescriptions by 55% (p < 0.0001);³² increases of oral rather than intravenous quinolone orders by 13% (p < 0.001);²¹ reductions in unnecessary vancomycin orders by 29% (p = 0.03)²³ and an over five times increase in the use of a predefined H₂ blocker (p < 0.001).²⁸

• • Three studies investigated computerized drug-dosing prompts. Two alert types showed benefits for dosing guidance in renal impairment in terms of more appropriate dose prescribing from 54 to 67% (p < 0.001)³¹ and quicker response to high creatinine levels from 97.5 to 75.9 hours (p < 0.0001).²⁹ Another claimed a reduction to an error rate of zero in pediatric weight-based dosing for gentamicin, though proof of significance was not reported.²⁰ In addition to prescribing outcomes, a reduction in prescribing related renal impairment by 55% b (95% CI 6–78%) was achieved by the alert system used in one of the studies.²⁹

• • All the four alerting systems that combined basic and advanced alerts in a complex system of alerts and prompts demonstrated statistically significant benefits with respect to prescribing. Bates (1998) demonstrated a reduction in serious medication errors by 55% (p = 0.01) with the rate of potential ADEs falling by 84% (p = 0.002).²⁴ The preventable ADE rate fell by 17% (statistically not significant) and the non-preventable adverse drug event rate was unchanged. Another system applied by Bates (1999) demonstrated an 81% reduction in all medication errors (p < 0.0001) and an 86% reduction in serious medication errors after introduction of a system with multiple types of alert (p = 0.0003).²⁵ Teich et al demonstrated several marked improvements in the prescription of preferred agents from 16 to 81% (p < 0.001), in correct frequency prescribing from 6 to 75% (p < 0.001) and in reduced overdose prescribing from 2.1 to 0.6% (p < 0.001).²⁸ A modest impact with borderline statistical significance was demonstrated by an alert system aiming at more appropriate prescribing in 13 different areas improving overall prescribing from 28 to 31% (relative risk 1.11 with 95% CI 1.00–1.22).²²

For many alert categories, only a small number of alert types had been performed, and for some categories we found no studies (▶). Types with few or no published data included drug-allergy warnings (n = 1), drug–drug interaction alerts (n = 1), duplicate medication alerts (n = 0), drug-disease contraindication alerts (n = 0) and dosing guidelines based on age (n = 0), pregnancy/child bearing age (n = 0), pediatric patients (n = 1) and indications (n = 0).

Discussion

Limitations

Our study employed rigorous inclusion criteria for the identification of studies relevant to the objectives of the review. In particular, we used a wide variety of potentially relevant literature databases and we carefully selected relevant search terms and adapted these for each database. In addition, we carefully searched the bibliographies of studies and reviews that we obtained to identify other potentially relevant articles. Given the limitations of time and resources we split up the searches of publication titles and abstracts between three reviewers. We did however check three validation samples of 1,000 references and this showed no discrepancies in the selection of main outcome studies. Throughout the process of conducting the review any issues arising, including discrepancies, were resolved through discussion. The data concerning the main outcomes in Tables 1 and 2 were checked by two reviewers independently (AS and CM).

This study had several limitations. The area of computerized decision support systems has become a wide field. In this review, we were specifically interested in studies evaluating point-of-care decision support systems that confront the clinician during the prescribing process. Studies were only included if CDSS were concerned with e-prescribing of a particular drug or class of drugs, not considering systems concerned with improved screening or better responses to abnormal laboratory results. Some might view this intentional narrow focus as an important limitation of the current study.

It is possible that some citations were missed, although the search strategy was comprehensive. The diversity in study characteristics and outcome measures did not allow pooling of the data and even made comparison between categories difficult. The wide variation of alerts and prompts requires a framework to categorize different alert types when reviewing the literature. In this review we applied a classification system based on a version by Kuperman, et al, which is a complex framework of alerts and prompts. ⁴ Due to the limited number of studies, it was not possible to reveal superiority of any category of alerts. Other categorizations of alerts and prompts according to their functionality, purpose, timing and workflow interruptions have been developed and represent an alternative way of categorization. ⁵

Implementation for Practice and Further Research

We identified 20 studies and 27 types of alerts and prompts of which 23 achieved improved prescribing in terms of changes in prescribing behavior and/or reductions in medication errors. In many of the studies, the changes noted were clinically relevant. Positive effects were noted for a wide range of alerts and prompts. Three of the alert types with lacking benefit showed weaknesses in their methodology ¹⁸ or design. ^19,33 These findings provide support for the continued development and use of computerized alerts and prompts for prescribing. The alert types and categories presented in this review represent relevant information for developers of computerized prescribing systems when setting up of new systems or updating older systems.

The limited number of studies conducted did not cover all alert categories, and it will be useful to target these alerts and prompts in future research. Drug–drug interaction alerts, drug-disease contraindication alerts and dosing guidelines based on age are prescribing categories with great potential for affecting prescribing. However, the lack of study data are surprising. These types of alerts, in particular, need to be further evaluated in future experimental trials using appropriate study designs. Randomized controlled trials are the most valid and robust method to evaluate the effect of an alert systems and should be used whenever possible.

The main obstacle in comparing effects on prescribing was the lack of standardized outcome measures for measuring medication errors. The most common outcomes reported across the studies of this review were medication error (serious and nonserious) and ADEs (actual and potential; preventable and non preventable). A variety of efforts are underway now to build consensus around these and other safety measures. ³⁷ Other outcomes were the impact of alerts on prescribing patterns as per guidelines, prescribing fewer nonrecommended drugs and prescribing more recommended drugs. To improve validity of future studies, an effort has to be made to use more uniform definitions of outcome measures around suboptimal prescribing.

Also, few studies have assessed the impact of computerized alerts and prompts on clinical or health service management outcomes. Such studies need to have a large enough sample size to have sufficient power to assess the effects of these interventions. Evidence from these types of studies is likely to have the greatest impact on the decision making processes of clinical information system implementers.

Perhaps most notably, the current literature provides little evidence from experimental studies about the factors that are most essential to consider when designing computerized alerts and prompts for prescribing. To our surprise, we found no empiric studies assessing the effectiveness of different approaches to the design and display of alerts and prompts. Results from observational studies suggest that paying careful attention to the content of alerts and prompts is likely to improve adherence by clinicians. ³⁸ Regarding the ways in which computerized alerts are displayed, there was support from clinicians for these to be clear, precise, brief and noticeable and for important alerts to be made “intrusive”. ^39–41 In addition, the need to improve the signal-to-noise ratio of alerts and prompts appeared to be a clear priority. ^42,43 Clinicians expressed reasonably high levels of support for alerts aimed at improving the safety of prescribing and showed strong preference for receiving recommendations rather than just assessments. ⁴⁴ This suggests that, provided alerts and prompts are carefully designed and selected, clinicians are likely to be accepting of them and that adherence is likely to be improved if they are displayed clearly and the signal-to-noise ratio is reasonably high. It is clear from areas other than medicine that both design and wording of alerts can have a substantial impact on their effectiveness. ⁴⁵ This review has shown that there are serious gaps in the medical literature regarding the most effective methods for displaying alerts and prompts and that there is a need to investigate how to present and display information at the point of prescribing. These gaps are important, because many electronic records do not include substantial decision support. Models about benefits of electronic health records (EHRs) suggest that substantial benefits will be achieved only with advanced decision support. ⁴⁶ It also appears that apparently minor issues relating to human factors can have a very large impact on the success of specific alerts and prompts, ⁴⁷ yet best practices in this area have not been clearly delineated.

Conclusions

We conclude that most empiric studies evaluating the effects of computerized prompts and alerts on prescribing behavior show positive, and often substantial, effects. However, while a wide range of types of alerts and prompts have been evaluated, additional studies should be done to determine the design features that are most strongly associated with improved impact on prescribing and clinical outcomes.