Medical Alert Management: A Real-Time Adaptive Decision Support Tool to Reduce Alert Fatigue

Abstract

With the adoption of electronic medical records (EMRs), drug safety alerts are increasingly recognized as valuable tools for reducing adverse drug events and improving patient safety. However, even with proper tuning of the EMR alert parameters, the volume of unfiltered alerts can be overwhelming to users. In this paper, we design an adaptive decision support tool in which past cognitive overriding decisions of users are learned, adapted and used for filtering actions to be performed on current alerts. The filters are designed and learned based on a moving time window, number of alerts, overriding rates, and monthly overriding fluctuations. Using alerts from two separate years to derive filters and test performance, predictive accuracy rates of 91.3%–100% are achieved. The moving time window works better than a static training approach. It allows continuous learning and capturing of the most recent decision characteristics and seasonal variations in drug usage. The decision support system facilitates filtering of non-essential alerts and adaptively learns critical alerts and highlights them prominently to catch providers’ attention. The tool can be plugged into an existing EMR system as an add-on, allowing real-time decision support to users without interfering with existing EMR functionalities. By automatically filtering the alerts, the decision support tool mitigates alert fatigue and allows users to focus resources on potentially vital alerts, thus reducing the occurrence of adverse drug events.

Introduction

Much research has been performed on prescribers’ views on alerts and their reasons for overriding alerts, such as Hsieh et al. 2004 for drug allergy alerts, Grizzle et al. 2007 and Ko et al. 2007 for drug-drug interaction alerts, and Shah et al. 2006 and Van der Sijs et al. 2006 for multiple types of alerts.

Too many alerts and too many alert overrides could lead to alert fatigue, which could threaten patient safety. Some research focuses on improving the specificity of alerts to reduce inappropriate alerts. Hsieh et al. (2004) studied characteristics of drug allergy alert overrides and made specific recommendations for increasing the specificity of alerting. Shah et al. (2006) suggested making the least severe alerts non-interruptive, not requiring a user action. Van der Sijs et al. (2008) interviewed prescribers about turning off frequently overridden drug-drug interaction alerts, finding that most of them wanted to reduce alert overload but they did not agree on which alerts could be safely turned off. Seidling et al. (2009) deduced the upper dose limits for statins from pharmacokinetic studies, incorporating dosage checking into alerts for dose-dependent drug-drug interaction alerts. Lee et al. (2010) designed a decision-learning framework to consistently identify override alerts, which could be filtered automatically to reduce alert fatigue. Riedmann et al. (2011) comprehensively studied factors which could be used to prioritize alerts in order to present them adequately. Twenty factors were identified and grouped into three categories: organization unit, patient/case, and alert itself.

This paper extends the study from Lee et al. (2010). In the previous study we built filters from three criteria to decide which alerts could possibly be automatically turned off or become non-interruptive. The three criteria include (1) number of alerts, (2) override rate, and (3) range of monthly override rate. The effect of the filters was validated by applying them to real datasets. In this study we propose to use a moving time-window to build the filters for real-time adaptive decision support. We also analyze the effectiveness of the criteria via linear regression.

Methods

Based on the alert datasets from Children’s Healthcare of Atlanta (CHOA), we focus on three types of alerts: dose alerts, drug allergy alerts, and drug-drug interaction alerts. Drug allergy alerts consist of four severities: level 1, 2, 3, and 4. Drug-drug interaction alerts consist of three severities: moderate interaction, severe interaction, and contraindicated drug combination. We sort out drugs or interactions under each alert type and severity. The filter is built on four criteria: (1) number of previous months used for training (size of time window), (2) number of alerts, (3) override rate, and (4) range of monthly override rates. We implement moving time-windows for continuous learning of the cognitive overriding decision of users. Specifically, to filter the alerts of the current month, alert data and their associated user decision patterns from a specified number of previous months serve as the training set to establish the filter decision rule. Such a dynamic design is both intuitive and appealing as it captures users’ decision characteristics from previous months to help predict the alert action at the current time. For each drug/interaction, the automatic filter is established via the following criteria on the training set:

• the total number of alerts is greater than or equal to a threshold,

• the overall override rate is greater than or equal to a threshold, and

• the range of monthly override rates (i.e., the difference between the maximum and minimum monthly override rates) is less than or equal to a threshold.

The filtered drug/interaction is considered most likely to be overridden by users based on previous patterns and the nature of the drug alert.

To evaluate the predictive accuracy and effectiveness of each of the filtering criteria, we contrast the results against filters that are established via i) static time-window prediction; ii) a time gap between training set time to the blind prediction set; and iii) regression confidence interval analysis.

Medication order alert data and their override patterns by providers from CHOA are used to validate and test our methodology. The data cover January to September 2009 and January to July 2011, and include only unfiltered alerts output by the EMR system. These alerts require manual review and actions by providers. Using moving time-windows, we build the filter for each month based on previous months. Table 1 shows the total number of unfiltered drugs/interactions for different alert types and severities as obtained from the EMR system. Table 2 shows the total number of unfiltered alerts in different months based on alert types. The last row in Table 2 reports the percentage of unfiltered alerts that were eventually overridden by the providers.

Table 1:

Number of drugs/interactions in different alert types and severities

Alert type	Severity	Number of drugs/interactions
Dose alert	N/A	851
Drug allergy alert	Level 1	110
	Level 2	48
	Level 3	206
	Level 4	89
Drug-drug interaction alert	Moderate interaction	184
	Severe interaction	130
	Contraindicated drug combination	37
Total		1655

Table 2a:

Number of unfiltered alerts for the period January to September 2009

Alert Type	2009
	Jan	Feb	Mar	Apr	May	Jun	Jul	Aug	Sep
Dose	1330	1382	1396	1323	1301	1344	1334	924	1213
Drug Allergy	580	604	585	499	463	492	549	575	682
Drug-Drug	892	1021	1178	1075	1141	892	1091	877	969
Total	2802	3007	3159	2897	2905	2728	2974	2376	2864
% overridden	65.8%	63.2%	63.5%	59.6%	64.7%	65.4%	63.9%	73.6%	74.6%

There is a significant difference in the number of unfiltered alerts between these two periods as the hospital expanded the EMR alert system to cover more units. In addition, clinical staff performs regular review and adjusts EMR alert parameters to reflect current clinical practice guidelines, thus resulting in different levels of alert reporting. For those alerts being fired out (unfiltered), providers must review each of them manually and carefully to determine if the prescribed action is appropriate or if other actions should be taken. Those for which the current prescribed action is deemed appropriate result in alerts being overridden. Those that are not overridden require that providers take additional and/or alternative action. Reviewing all alerts can be a tedious and time-consuming task. Alert fatigue has been documented and reported on in numerous studies and can be detrimental to clinical outcome. Hence, a decision support system that can identify and appropriately handle non-essential alerts is critical because it enables providers to focus on vital ones and make proper decisions.

Results

In this section we describe a test implementation of our methodology at CHOA, a network of three pediatric hospitals in Atlanta, Georgia, which serves over half a million patients annually.

Predictive Performance of the Time Window Adaptive Filters

We consider the following criteria to construct filters:

1. Time-Window: Size of time window = 2, 3, 4, 5, 6 months

2. Number: Number of alerts ≥ 20, 30, 40, 50, 60

3. Rate: Override rate ≥ 90%, 92%, 94%, 96%, 98%

4. Range: Range of monthly override rates ≤ 10%, 12%, 14%, 16%, 18%

As an example, we use alert data from January and February 2009 for training (thus, size of time window is 2 months) and choose the number of alerts to be at least 20, with override rate of at least 90%, and override rate range to be within 10% to construct a filter, Filter1. Filter1 is then applied to the March 2009 alerts, resulting in ten drugs/interactions being filtered. Figure 3a shows the type of alerts being filtered versus the actual action by the providers. Those filtered by us and overridden by providers are true negatives (that is, these are non-essential alerts). Those not overridden by providers but filtered by us are false negatives. All unfiltered alerts are reviewed manually by providers.

Counts of alerts (and associated percentages) filtered by us versus counts of alerts overridden by providers are shown in Table 3b. Of those that are manually overridden by providers, we filter 22.5% of them. Such automatic filtering can save precious time, enabling providers to tend to other important patient care issues. Of those alerts that are not overridden by providers but are filtered by us, we commit a 3.5% error. There may be serious medical implications to these errors. We pursue two actions to correct such errors: a) adaptive learning to incorporate these alert contents within our filtering algorithm; b) second-layer filtering with inclusion of patient risk factors to enrich our filtering algorithm’s predictability. From Table 3b, we note that the predictive accuracy of this filter is 91.9% (452/(452+40)). This high value indicates that if an alert is filtered (by our algorithm), there is a high level of confidence that the filtering result is correct in predicting the actions of the providers.

Table 3b:

Filtered alerts and associated rates for Filter1 on March 2009 alerts

Filtering results		Counts		Percentage
Providers’ action		Filtered by us (negative)	Not filtered by us (positive)	Filtered by us	Not filtered by us
Counts	Manual Overridden (negative)	452	1555	22.5% (Specificity)	77.5% (False positive rate)
	Non-overridden (positive)	40	1112	3.5% (False negative rate, FNR)	96.5% (Sensitivity)
Percentage	Manual Overridden (negative)	91.9% = 452/(452+40) (Negative predictive value, NPV)	58.3%
	Non-overridden (positive)	8.1%	41.7%

To understand the tradeoffs among specificity, the false negative rate (FNR), and the negative predictive accuracy (NPV) with respect to different criteria, we derive 625 filters based on the four criteria, each taking on five potential values (5⁴ = 625). Each filter is applied to all potential training sets and is used to predict the override pattern of the following month. For example, a filter with time window of size 5 uses alert data and associated decisions from five consecutive months to predict the override pattern of the following month. So, January to May alert data and decisions are used as a training set to establish the rule, and the override pattern of June is predicted. Similarly, February to June alert data and provider decisions are used to develop a rule that is used to predict the override pattern for July. And so forth. Performing this on all alert data from January 2009 to June 2011 leads to 5000 training sets and prediction results. The predictive status of each alert is compared against the actual providers’ override action. These values are summarized via boxplots in Figures 1a–c, grouped with respect to specific values for each criterion. From these figures we observe that our filtering performance is most sensitive to Rate. As Rate increases (i.e., filters are set to be more stringent), the percentage of overridden alerts being filtered (specificity) decreases significantly (Figure 1a), fewer false negative errors are made by the filter (Figure 1b), and the predictive accuracy of the filter increases (Figure 1c). We also observe that Time-Window could have some effects on the filtering results, whereas Number and Range appear to have marginal impact.

Figure 1a:

Boxplots of % of overridden alerts being filtered (specificity) with respect to single-criterion value change.

Figure 1c:

Boxplots of predictive accuracy % among filtered alerts (overridden and filtered/total filtered) with respect to single-criterion value change.

Figure 1b:

Boxplots of % of non-override alerts being filtered (false negative rate) with respect to single-criterion value change.

Among the 625 filters, we present below the most conservative one. For each filter, the average values of Specificity, FNR, and NPV over different training/prediction months are taken as consideration to measure the overall performance. The goal is to have a high-specificity, low-FNR, and high-NPV filter. However, these metrics are somewhat conflicting, as reflected in the scatterplot matrix in Figure 2. Here we pick a filter with the best average rank over these three metrics:

1. Time-Window: Size of time window = 5 months

2. Number: Number of alerts ≥ 40

3. Rate: Override rate ≥ 94%

4. Range: Range of monthly override rate ≤ 10%

Using this filter the Specificity ranges from 6.6%~10.8%, FNR ranges from 0%~1%, and NPV ranges from 93.8%~100%, as shown in Table 4.

Table 4:

Predicted results with Time-Window=5, Number≥40, Rate≥94%, Range≤10%

Predicted month	Jun ’09	Jul ’09	Aug ’09	Sep ’09	Jun ’11	Jul ’11	Average
Metrics
Specificity (% of overridden alerts filtered by us)	9.1%	7.2%	8.2%	6.6%	10.6%	10.8%	8.8%
FNR (% of non-overridden alerts filtered by us)	0.0%	0.8%	0.6%	0.8%	1.0%	1.0%	0.7%
NPV (% of predictive accuracy)	100.0%	93.8%	97.3%	95.9%	97.3%	97.0%	96.9%

Effectiveness of the Filtering Criteria

We examine the effectiveness and significance of these criteria in more detail using regression analysis, focusing on understanding the importance of the moving time window, the allowance of time gap between the training months to the predictive months, and the degree of significance of each criterion. For brevity, we summarize the findings when the time window is set to 3.

Moving time-window versus static time-window

First we compare the results of using moving time-windows as training set versus employing static time-windows. We perform the filtering by maintaining the same for all but the time window criterion. Specifically, the filters will include the parameters: number of alerts ≥ 20, 30, 40, 50, 60, override rate ≥ 90%, 92%, 94%, 96%, 98%, range of monthly override rate ≤10%, 12%, 14%, 16%, 18%.

We apply paired t-test to see whether there is significant outcome difference between that of moving time-windows and those from static time-windows. There are 5*5*5=125 filters, and for each filter we predict 8 different months (May–Sep 2009 and May–Jul 2011), so in this analysis the sample size is 125*8=1000. The average net Specificity of moving time-windows is 13.6% versus 11.3% from those obtained via the static time-window. The resulting p-value is less than 0.0001, indicating that using moving time-windows is more effective than static ones.

Gap between training months and predicting month

Next we analyze how the filter performs if we allow a gap between the moving time-window and the month to be filtered (predicted). Under the same set of parameters in all criteria, we apply paired t-test to observe if there is any significant change in Specificity when using no gaps versus using gaps for the moving time-windows. The results are shown in Table 5. Regardless of the size of the gap (1, 2, or 3 months), the p-values are all less than 0.0001, indicating that using immediate previous months for training is more effective than using previous months with gaps. This reflects that the overriding patterns of immediate months reflect better the cognitive decision of the providers in the immediate next month.

Table 5:

Comparison of net Specificity outcome between no gaps and with gaps. The gap is measured by the number of months between the months for training and the month for prediction

Size of gap	Sample size	Average net Specificity		p-value
		No gaps	With gaps
1 month	1,000	13.6%	11.9%	<0.0001
2 months	750	13.9%	11.8%	<0.0001
3 months	500	13.9%	10.9%	<0.0001

Degree of Significance of Individual Criterion

We conduct regression analysis to gauge the relative effectiveness of each criterion in our filtering schema. Specifically, Specificity, FNR, and NPV are used as the response variables separately, while size of Time-Window, Number of alerts, override Rate, Range of monthly override rate, and their pairwise interactions are the predictor variables. In the model where the response variable is Specificity or FNR, almost all predictor variables including their interactions have significantly small p-values; thus there are no nonsignificant ones. In the model where the response variable is NPV, only Time-Window, Rate, and their interaction term are significant, thus we can rule out Number and Range. Below is the regression equation to predict NPV, the predictive accuracy of the filter:

$$\text{Predictive accuracy}\left(\mathit{NPV}\right)~0.053+0.265\mathit{Time}-\mathit{Window}+0.963\mathit{Rate}-0.287\mathit{Time}-\mathit{Window}*\mathit{Rate}$$

This finding validates the observations from the single-criterion boxplot results that Rate and Time-Window are the influencing criteria in determining the predictive accuracy (Figure 1c).

Discussion

With the adoption of electronic medical records (EMRs), drug safety alerts are increasingly recognized as valuable tools for reducing adverse drug events and improving patient safety. However, even with proper tuning of the EMR alert parameters, the volume of unfiltered alerts can be overwhelming to users. Studies have been performed to improve the specificity of alerts, to effectively prioritize and present the alerts for providers’ review, and to automate the filtering of some unnecessary or least severe alerts. This remains a challenging problem as there is a tradeoff and disagreement between reducing alert overload versus the safety of turning off some alerts.

In this paper, we design an adaptive decision support tool in which past cognitive overriding decisions of users are learned, adapted and used for filtering actions to be performed on current alerts. The filters are designed and learned based on four criteria: moving time window, number of alerts, override rates, and monthly override fluctuations. Using alerts from two separate years to derive filters and test performance, prediction accuracy rates of 91.3% – 100% are achieved. This associates to 0.0% – 6.1% non-overridden alerts being filtered by our system. To reduce this false negative rate, we test the concept of adaptive learning by mining the key contents of these false negative alerts and incorporating them within our filtering algorithm. This helps to bring down the false negative rate to 0.0% – 1.5%. Further reduction may be possible by importing clinical and patient risk factors into our analysis. Added clinical features may tag these to be non-filterable and thus correct the committed errors. Careful review of each such patient record is necessary. A follow-up study will have to be performd to validate its potential.

The moving time window works better than a static training approach. It allows continuous learning and capturing of the most recent overriding characteristics and seasonal variations in drug usage. Further, our study shows that the overriding patterns of immediate months reflect better the cognitive and clinical decision of the providers in the immediate next month. From regression analysis we conclude that the size of time window, number of alrts, override rate, range of monthly override rate, and their pairwise interaction are all critical factors in influencing specificity and false negative rate; whereas predictive accuracy is governed mostly by time window sizes and the override rates.

The clinical staff diligently reviews and fine-tunes EMR alert parameters to capture proper clinical practice guidelines. For those alerts being fired out, providers will manually review them to determine if the prescribed action is appropriate or if other actions should be taken. Those for which current prescribed action is deemed appropriate results in alerts being overridden. The non-overriding ones amount to important alerts that providers must take action on. Alerts fatigues have been reported by numerous studies and can be detrimental to clinical outcome. Our alert management decision support system helps to improve the specificity of alerts by reducing inappropriate alerts, thus enabling providers to focus their attention on important alerts and make proper decisions.

Among all the filters, some of the filtering levels are rather conservative, as roughly only 20% of overridden alerts are being filtered, with majority of the alerts remain unfiltered and require the attention of providers. Further, as the system “adapts” and ”learns” that certain drug interactions are critical and cannot be filtered and required specific actions, this knowledge will be transferred to the providers with alerts being tagged with knowledge of its importance. This provides an opportunity for training, in particular for those inexperienced providers who will benefit from the captured clinical knowledge.

From Table 2b, we observe that over 75% of alerts reviewed by providers are over-ridden. Using our system, about 20% of these are filtered. Using the estimate of 11,000 alerts per month and 1 minute per alert review, the estimated time savings is 27.8 provider-hours per month. More importantly, this enables providers to strategically allocate appropriate time to review vital alerts.

Table 2b:

Number of unfiltered alerts for the period January to July 2011

Alert Type	2011
	Jan	Feb	Mar	Apr	May	Jun	Jul
Dose	8516	8747	9878	8811	9174	8903	8736
Drug Allergy	1360	1349	1619	1281	1465	1403	1437
Drug-Drug	1293	1238	1453	1642	1305	1335	1260
Total	11169	11334	12950	11734	11944	11641	11433
% overridden	77.9%	79.1%	77.4%	77.1%	76.2%	76.8%	75.0%

The decision tool can be plugged into an existing EMR system as an add-on, allowing real-time decision support to users without interference with existing EMR functionalities. We have tested it on EPIC. Specifically, the alerts directed out of EPIC are captured and funneled into the decision support tool. The tool grabs the data, establishes the filters, and performs the filtering. The tool then reports the alerts with status “filtered” and “not-filtered”. In the latter, it also highlights those “learnt” critical alerts. Providers can review them accordingly. By automatically filtering the alerts, the tool mitigates alert fatigue and allows users to focus resources on potentially vital alerts that require clinical decisions, thus reducing the occurrence of adverse drug events. Further, the learnt knowledge of critical alerts will facilitate the decision process. Clinical trials must be performed to fully evaluate the impact of this decision support tool on alert fatigue, patient safety and quality of care.

Figure 2:

Scatterplot matrix of Specificity, FNR, and NPV

Table 3a:

Drugs/interactions from March 2009 alerts that are filtered by Filter1.

Type	Severity	Drug name	Providers’ actions
			Overridden #	Non-overridden #
Dose	(N/A)	CHLOROTHIAZIDE SODIUM	2	1
Dose	(N/A)	FOSPHENYTOIN	8	0
Dose	(N/A)	KCL	120	11
Dose	(N/A)	ROCURONIUM	2	0
Drug-Allergy	Level 3	INSULIN REGULAR HUMAN	0	0
Drug-Allergy	Level 3	MORPHINE SULFATE	16	6
Drug-Drug	Severe Interaction	CYCLOSPORINE/AZOLE ANTIFUNGAL AGENTS	71	1
Drug-Drug	Severe Interaction	KETOROLAC / ANTICOAGULANTS	204	12
Drug-Drug	Severe Interaction	LAMOTRIGINE / VALPROIC ACID	18	1
Drug-Drug	Severe Interaction	METHOTREXATE / PENICILLINS	11	8
Subtotal			452	40

Table 3c:

Predicted results of Filter 1 – Time-Window=2, Number≥20, Rate≥90%, Range≤10%

Predicted month	2009							2011					Average
Metrics	Mar	Apr	May	Jun	Jul	Aug	Sep	Mar	Apr	May	Jun	Jul
Specificity (% of overridden alerts filtered by us)	22.5%	30.4%	30.9%	33.1%	31.8%	30.3%	17.7%	14.5%	18.0%	20.6%	24.8%	26.3%	25.1%
FNR (% of non-overridden alerts filtered by us)	3.5%	4.4%	3.7%	2.3%	4.1%	7.5%	3.9%	3.3%	5.8%	5.3%	6.1%	6.9%	4.7%
NPV (% of predictive accuracy)	91.9%	91.0%	93.9%	96.4%	93.2%	91.9%	93.1%	93.7%	91.3%	92.6%	93.1%	91.9%	92.8%