Clinical Validity Assessment of Integrated Dose Range Checking Tool in a Tertiary Care Hospital Using an Electronic Health Information System

Abstract

Background/purpose: The electronic clinical decision support system (CDSS) is mainly used to assist health care providers in their decision-making process. CDSS includes the dose range checking (DRC) tool. This study aims to evaluate the clinical validity of the DRC tool and compare it to the institutional Formulary and Drug Therapy Guide powered by Lexi-Comp. Methods: This retrospective study analyzed DRC alerts in the inpatient setting. Alerts were assessed for their clinical validity when compared to recommendations of the institution’s formulary. Relevant data regarding patient demographics and characteristics were collected. A sample size of 3000 DRC alerts was needed to give a margin of error of 1% (using normal approximation to binomial distribution gives 30.26/3000 = 1%). Results: In our cohort, 1659 (55%) of the DRC alerts were generated for adult patients. A total of 1557 (52%) of all medication-related DRC alerts recommended renal dose adjustments, while 708 (24%) needed hepatic dose adjustments. Majority of alerts, 2844 (95%), were clinically invalid. A total of 2892 (96%) alerts were overridden by prescribers. In 997 (33%) cases, there was an overdose relative to the recommended dose, and in 1572 (52%) there was underdosing. Residents were more likely to accept the DRC alerts compared with other health provider categories (P < .001). Conclusion: Using DRC as a clinical decision support tool with minimal integration yielded serious clinically invalid recommendations. This could increase medication-prescribing errors and lead to alert fatigue in electronic health care systems.

Introduction

In an effort to make a difference in reducing drug dosage errors, the Institute of Medicine (IOM) recommended the use of electronic medication order entries with computerized decision support tools in a report titled “To Err is Human: Building a Safer Health System.”¹ Computerized provider order entry (CPOE) allows the verification of drug dosage with a documented reduction in medication errors.² The most common type of medication errors, which may include incorrect medication dosage and frequency, is errors at the prescriber’s level.³ In one study, 7% to 15% of hospital medication prescriptions have an error, and 30% to 50% of hospitalized patients are exposed to at least 1 prescription error during their hospitalization period.^4,5

Throughout history, various clinical decision support systems (CDSSs) have been introduced to help health care providers make decisions regarding diagnosis and/or treatment in a particular medical domain.⁶ Several CDSSs, including dose range checking (DRC), test selection, image recognition and interpretation, alerts and reminders for drug-drug interaction, drug duplication, lab levels, and others, have been integrated into our hospital’s clinical information system. Such tools that provide reminders or alerts within the clinical system have become more prevalent due to the increased use of CDSS in hospitals worldwide. This can contribute to the ever-increasing problem of alert fatigue. A published study showed that the most common complaint about using CDSS is the emerging issue of alert fatigue.⁷

The DRC is a clinical decision support tool that was developed to aid prescribers in selecting appropriate medication dosing. A dose triggers an alert if the dose of the prescribed drug exceeds the maximum dose limit (see Image 1). These alerts are generally generated twice, both during the prescribing phase by treating physicians and during pharmacist order verifications. The same process applies to subtherapeutic doses when the prescribed dose is below the minimum dose limit set by the Cerner Multum Middle East Drug Library database, which is part of Cerner Millennium Health Information System. In this health information system, the main activated DRC parameters are age and route of administration. Body weight is partially activated, whereas other dosing parameters (such as health condition and renal and hepatic profiles) are not activated. The dose limit variance percentage, which compares the dose variance with the upper and lower limits defined in Multum drug library and allows a percentage of the absolute calculated dose accordingly, is part of this system.

Image 1.

Dose range checking (DRC) alert example.

At our tertiary care hospital, health care providers often utilize DRC alerts as a clinical decision support tool in addition to our resourceful institutional online formulary and drug therapy guide. While both tools list medication dosing ranges, they differ in their database references. In our electronic health record (EHR), dose range alerts are generated from Cerner Multum Drug Library database, which is part of Cerner Millennium Health Information System. In contrast, our institutional formulary medication dose ranges are based on the Lexi-Comp database, requiring indication approval by the pharmacy and therapeutics (P&T) committee. The objective of this study was to evaluate the clinical validity of integrated DRC alerts as a part of the medication-prescribing process through comparing DRC alerts to the hospital formulary and drug therapy guide recommendations linked at the patient level.

Methodology

Study Design

This is a retrospective analysis of system-generated DRC alerts of each business day from May 8 to May 18, 2017 in the inpatient hospital settings. The study was conducted at a 1265-bed tertiary care and referral hospital based in Riyadh, Saudi Arabia.

Each alert was assessed for its clinical validity and relevance to the specific patient for whom the alert was generated. The appropriateness and validity of DRC alerts were based on the indication-specific dosing recommendations based on the hospital formulary, which is based on Lexi-Comp database, utilizing real and actual patient-specific data. For the sake of our study, a clinically valid alert was defined as an alert that follows the P&T committee (which is an evidence-based committee)-approved dose for specific indication as documented in the institutional formulary, whereas a significantly deviated alert was defined as an alert that is deviated from the recommended dose by 50%, either higher or lower. Table 1 lists standard definitions of common terms used in the study. The study was approved by the institutional review board (RAC no. 2171089).

Table 1.

Standard Definitions of Common Terms.

Term	Definition
Clinically valid alert	An alert that follows the P&T committee (which is an evidence-based committee)-approved dose for specific indication as documented in the institutional formulary (which is based on Lexi-Comp database)
Clinically invalid alert	An alert that does not follow the P&T committee (which is an evidence-based committee)-proved dose for specific indication as documented in the institutional formulary (which is based on Lexi-Comp database)
Implemented alert	An alert of a medication order that shows a documentation of medication administration in eMAR
Significantly deviated alert	An alert that is deviated from the recommended dose by 50%, either higher or lower
Accepted alert	An alert that has been accepted by the health care provider in the alert tool
Overridden alert	An alert that has been overridden by the health care provider in the alert tool
Fully activated tool	All dosing-related parameters (dependencies) such as lab results, diagnosis, weight, age, etc are defined and used by the DRC rule logic for more precise dosing recommendations
Partially activated tool	Only 2 parameters are defined and used (weight and age) by the DRC rule logic which gives less precise dosing recommendations
Inactivated tool	None of the dosing-related parameters are defined nor used and only generic dosing instructions/recommendations are displayed by the DRC rule logic

Note. P&T = pharmacy and therapeutics; eMAR = electronic medication administration record; DRC = dose range checking.

Data Collection

All data were collected from the EHR Cerner Millennium health information system which is internally called integrated clinical information system (ICIS). We obtained patient data and specific characteristics relevant to dosing adjustments, such as gender, age, weight, height, medical/hospital unit, diagnosis, specific indication, underlying comorbidities (liver or kidney dysfunction), drug-drug interactions, ordered medication(s), and ordered dose(s), including the unit, route, frequency, and duration. The position of the health care provider who received each alert and the type of DRC alert (overdose or underdose alert) were included in the data collection. Creatinine clearance was calculated for patients prescribed medications that required renal adjustments using Cockroft and Gault⁸ equation for adult patients and the Traub SL and Johnson CE⁹ method for pediatric patients. Child-Pugh scores were determined for patients’ medications that needed dose adjustments in the case of liver impairment.¹⁰

End-user satisfaction was assessed using a validated satisfaction questionnaire, computing satisfaction, that was sent to health care providers who received DRC alerts.¹¹ The survey questions addressed 5 components of the DRC alerts: (1) tool content, (2) accuracy, (3) format, (4) ease of use, and (5) timeliness. The results of the survey were summarized as percentages and weighted averages. A weighted average of 3 was considered neutral, any value that was more than 3 was considered satisfactory, and if the value was less than 3, it was considered as unsatisfactory.

Endpoints

Primary endpoint

The primary endpoint of this study was the percentage of clinically valid alerts generated by the DRC decision support tool.

Secondary endpoints

This study aimed to measure several secondary endpoints, including the total rate of overridden alerts, the percentage of accepted alerts, the percentage of implemented alerts (which are defined as alerted orders that reached the patients), the correlation between rate of alert acceptance by health care provider, and the quantification of the margin of dose deviation from the recommended dose. The deviation was considered to be significant when it deviated by more than 50% over or under the recommended dose. In addition, the study estimates the end-user computing satisfaction of the tool.

Statistical Analysis

This study intended to measure the percentage of clinically valid alerts generated by the DRC decision support tool out of the total generated alerts during the same time period. Descriptive statistics for the categorical variables were summarized as frequencies and percentages. In order to estimate the sample size, the prevalence of valid DRC alerts for the first 1000 DRC alerts was collected, which was 8.7%. Therefore, 3000 alerts collected over a 10-day period would provide an estimate for valid alerts with a margin of error of 1%. We analyzed the first consecutive 300 alerts for each day of the 10-day period. Seasonal effects were noticed in previous literature studies in which there was a variation among alerts in different seasons,¹² but, given the large volume of alerts with a daily average of 895 alerts and 9005 orders, this effect would be minimal. In addition, this study aimed to assess the clinical validity of the alerts. Pearson correlation was used to assess the correlation between the rate of DRC alert acceptance and the type of prescribers who received these alerts.

Results

Demographic data from patients subject to DRC and health care providers who received the alerts are outlined in Table 2. In our cohort, 1659 (55%) generated DRCs were for adult patients and 1586 (53%) were male. Of all alerted medications, 1557 (52%) and 708 (24%) needed renal and liver adjustments, respectively. The DRC alerts appeared for pharmacists in 1171 (39%) of our samples, and the rest were for physicians.

Table 2.

Baseline Characteristics.

Characteristic	N = 3000
Age category
Adult	1659 (55%)
Pediatric	1341 (45%)
Gender
Male	1586 (53%)
Female	1414 (47%)
Renal function
Alerted medications that needed renal adjustments	1557 (52%)
Patients who have any level of renal failure	398 (13%)
Liver function
Alerted medications that needed liver adjustments	708 (24%)
Patients who have any level of liver failure	89 (3%)
Nursing unit category
Medical units	2359 (79%)
Critical unit	641 (21%)
Position of the health care provider who received the alerts
Pharmacist	1171 (39%)
Medical resident	706 (24%)
Assistant physician	693 (23%)
Fellow	375 (12%)
Consultant	55 (2%)

A total of 1572 (52%) DRC alerts were underdose alerts, and a smaller proportion (33%) were overdose alerts (Figure 1). The majority of the generated DRC alerts were deemed clinically invalid, 2844 (95%), while the remaining 156 (5%) were valid alerts (Figure 2). The reasons for invalid alerts were analyzed. Most of the alerts had a dose that was within the correct range (such as alerts that should not have been generated [29%]). The second most frequent reason was due to doses that were not indication specific and were found in 425 (14%) alerts. The third most frequent reason was due to doses that did not follow the institutional formulary, found in 384 (13%) alerts. The remaining reasons for invalid alerts are outlined in Table 3.

Figure 1.

Type of dose range checking alerts generated during the study period (n = 3000).

Note. Other: Set messages by the system such as Not FDA approved and The safety of this drug is unknown. FDA = Food and Drug Administration.

Figure 2.

Clinical validity of dose range checking alerts generated during the study period (N = 3000).

Table 3.

Reason of Dosing Alert Invalidity.

Reason	N = 3000
The dose is within the correct range	876 (29%)
The dose recommended is not indication specific	425 (14%)
Does not follow institutional formulary dosing	384 (13%)
Does not consider drug-related lab parameter	317 (11%)
Not an FDA-approved use and not enough information to make recommendationsa	226 (8%)
Does not consider dosing that is based on drug level	199 (7%)
The safety of this drug is unknowna	137 (5%)
Not adjusted based on kidney function	83 (3%)
Inaccurate frequency	54 (2%)
Not adjusted based on age	33 (1%)
Inaccurate route of administration	24 (1%)
There is no suggested dose range for the medicationa	20 (1%)
Not adjusted based on weight	14 (0.5%)
Inaccurate infusion rate	2 (0.1%)
Not adjusted based on liver function	1 (0%)

Note. FDA = Food and Drug Administration.

^aStandard message.

Among the 3000 reviewed alerts, a total of 2892 (96%) alerts were overridden and 108 (4%) were accepted. Alert implementation was assessed for the accepted alerts, with only 11 (10%) alerts being implemented and 97 (90%) not implemented.

The margin of dose deviation from the recommended dose was assessed. The deviation was considered significant when it was more than 50% over or under the recommended dose. And 2569 (86%) deviated significantly from the recommended dose. The top 5 most frequent medications subjected to DRC were enoxaparin—243 (8%), acetaminophen—183 (6%), magnesium sulfate—136 (5%), magnesium oxide—119 (4%), and vancomycin—106 (4%). A list of the top 10 most frequently alerted medications during the study period is shown in Table 4.

Table 4.

Top 10 Most Frequently Alerted Medications During the Study Period.

Top 10 medications	No. (%)
Enoxaparin	243 (8)
Acetaminophen	183 (6)
Magnesium sulfate	136 (5)
Magnesium oxide	119 (4)
Vancomycin	106 (4)
Furosemide	105 (3.5)
Midazolam	82 (3)
Mycophenolate mofetil	76 (3)
Ondansetron	72 (2)
Morphine	68 (2)

The variation between the rate of alert acceptance and health care providers was also assessed. Residents were more likely to accept the recommendation, 6.1% versus other health professional categories (between 2.4% and 4.5%, (P < .001); Figure 3).

Figure 3.

Rate of alert acceptance by health care provider.

The end-user computing satisfaction survey was sent to 944 providers, with a response rate of 146 (15.4%). Approx-imately 37% of the responders were pharmacists and 63% were physicians. The performance of the tool was unsatisfactory with approximately 90% of the survey questions resulting in an average of less than 3 as shown in Table 5. However, for specific questions in the survey, if the responder stated “can’t assess,” this question was removed from the weighted average calculation.

Table 5.

Dose Range Checking Alert Satisfaction Survey Results as Completed by Health Care Practitioner.

Survey questions	Strongly disagree 1	Disagree 2	Neutral 3	Agree 4	Strongly agree 5	Weighted average
	No. (%)	No. (%)	No. (%)	No. (%)	No. (%)
Does the tool provide the precision you need to make a decision? (n = 143)	24 (16)	31 (21)	25 (17)	43 (29)	20 (14)	3.02
Does the information meet your needs as a clinician? (n = 141)	22 (15)	35 (24)	26 (18)	45 (31)	13 (9)	2.94
Does the system provide sufficient information to make a decision on dosing? (n = 142)	25 (17)	40 (27)	28 (19)	39 (27)	10 (7)	2.78
In your opinion, is the dose alert accurate? (n = 141)	20 (14)	45 (31)	36 (25)	33 (23)	7 (5)	2.73
What is your level of satisfaction with the accuracy of the tool? (n = 141)	21 (14)	35 (24)	38 (26)	37 (25)	10 (7)	2.85
Do you think the output is presented in a useful format? (n = 141)	18 (12)	36 (25)	36 (25)	40 (27)	11 (8)	2.92
Do you think the output information clear? (n = 143)	15 (10)	31 (21)	34 (23)	49 (34)	14 (10)	3.11
In your opinion, is the tool user-friendly? (n = 142)	27 (18)	28 (19)	30 (21)	46 (32)	11 (8)	2.90
Do you get the information you need on time? (n = 139)	16 (11)	36 (25)	26 (18)	48 (33)	13 (9)	3.04
Does the alert provide up-to-date information? (n = 138)	21 (14)	34 (23)	41 (28)	33 (23)	9 (6)	2.81

Discussion

Most organizations follow high-reliability organization (HRO) standards,¹³ in which they aim to diminish practice errors in the health care setting. This study suggests that having DRC as a tool can be more of a hindrance than a help. To some extent, our main finding was unexpected in which more than 90% of the DRC alerts were clinically invalid. This represents a serious challenge for teaching hospitals in particular, where interns and residents receive these alerts.

One of the interesting findings of this study was that enoxaparin, a high-alert medication, generated the most DRC alerts. This finding raises a major safety concern since enoxaparin dosed inappropriately can lead to serious consequences. The main issue with enoxaparin is that all of its generated alerts are not indication specific. For example, DRC alerts indicated prophylactic enoxaparin as the default recommended dose, in which treatment doses were not considered. Approximately 14% of the DRC alerts were invalid because they were not indication specific. This makes the lack of indication-specific DRC the second most frequent reason for clinical invalidity.

The variation between the rate of alert acceptance by health care provider was also analyzed. Residents were more likely to accept the alert recommendation compared to consultants and other health care providers. This can be alarming when considering that teaching hospitals have higher numbers of interns and residents who might tend to accept alerts more frequently and apply them in practice, thus causing potentially serious errors.

Ideally, dosing alerts should be integrated with laboratory values when generating a recommendation. For instance, magnesium replacement therapy dosing is based on laboratory levels. However, DRC alerts do not consider laboratory values when recommending a dose. In our study, both magnesium sulfate and oxide medications have high DRC alerts, and the fourth most frequent reason for DRC alert invalidity was the lack of consideration of drug-related laboratory values. This can contribute to alert fatigue.

The same concern applies when considering drugs that are dosed based on drug serum levels. An example of a drug adjusted based on its serum levels is vancomycin. Vancomycin is ranked as the fifth medication with the most DRC alerts (4% rate). One of vancomycin’s known undesirable effects is nephrotoxicity. However, DRC alerts did not consider the patient’s renal profile. The dosing alerts were also not customized in order to make a recommendation for a patient with liver dysfunction. Thus, these alerts were clinically invalid. Approximately 29% of all DRC alerts were due to doses that were within the correct range; hence, medication errors might occur and added to alert fatigue among physicians. Further customization could be done to solve such an issue.

One of the study limitations was the inability to detect medication errors caused by the accepted alerts. This was due to the lack of documentation in the patient EHR. In addition, this was not a controlled study in which we would look at results before and after the implementation of the CDSS and DRC alerts. However, conducting such a study can be challenging. Considering the observational nature of this study, it was difficult to establish causality, especially in the absence of appropriate documentation. The main objective of this study was to investigate the clinical validity of DRC alerts and that validity did not need preimplementation and postimplementation analysis. On the contrary, our study is one of the first studies that investigated the clinical validity of the DRC alert tool as a part of the CDSS with the largest sample size. A similar study aimed to evaluate dosing alert appropriateness in a pediatric hospital. Similarly, they compared alert doses with the hospital reference (Lexi-Comp) and institutional recommendations.¹⁴ This study utilized the Epic© EHR system, which is different from the system used in our study (Cerner Multum).

Future studies are needed to investigate medication errors caused by invalid DRC alerts and assess the clinical validity after tool redesigning with more focus on high-alert medications, such as enoxaparin. The DRC tool can be more useful when considering all contributing dosing- and patient-specific characteristics. Also, we believe that traditional clinical support systems will continue to add very limited benefits to our health care practitioners due to technical limitations. Further exploration of advanced clinical support tools using machine learning and artificial technologies is highly recommended and may solve many of the limitations of today’s clinical support tools.

Conclusion

The use of the standard traditional DRC function as an integrated clinical decision support tool yielded invalid clinical recommendations in the majority of the cases. The main reason for such low tool sensitivity was due to the use of the generic DRC function without any customization for patient-specific dosing factors. This can contribute to inappropriate recommendation adaptation and add to the ever-increasing alert fatigue among prescribers.