Analysis Of Electronic Medication Orders With Large Overdoses

Summary

Background

Users of electronic health record (EHR) systems frequently prescribe doses outside recommended dose ranges, and tend to ignore the alerts that result. Since some of these dosing errors are the result of system design flaws, analysis of large overdoses can lead to the discovery of needed system changes.

Objectives

To develop database techniques for detecting and extracting large overdose orders from our EHR. To identify and characterize users’ responses to these large overdoses. To identify possible causes of large-overdose errors and to mitigate them.

Methods

We constructed a data mart of medication-order and dosing-alert data from a quaternary pediatric hospital from June 2011 to May 2013. The data mart was used along with a test version of the EHR to explain how orders were processed and alerts were generated for large (>500%) and extreme (>10,000%) overdoses. User response was characterized by the dosing alert salience rate, which expresses the proportion of time users take corrective action.

Results

We constructed an advanced analytic framework based on workflow analysis and order simulation, and evaluated all 5,402,504 medication orders placed within the 2 year timeframe as well as 2,232,492 dose alerts associated with some of the orders. 8% of orders generated a visible alert, with ¼ of these related to overdosing. Alerts presented to trainees had higher salience rates than those presented to senior colleagues. Salience rates were low, varying between 4–10%, and were lower with larger overdoses. Extreme overdoses fell into eight causal categories, each with a system design mitigation.

Conclusions

Novel analytic systems are required to accurately understand prescriber behavior and interactions with medication-dosing CDS. We described a novel analytic system that can detect apparent large overdoses (≥500%) and explain the sociotechnical factors that drove the error. Some of these large overdoses can be mitigated by system changes. EHR design should prospectively mitigate these errors.

Background

Orders for medications written for infants and children are prone to error due to the increased complexity of weight-based dosing [1–3]. Despite decision support for dose calculations, electronic orders can contain dosing errors. While some reports indicate that these errors occur at a lower rate than comparable orders written on paper [4–6], others suggest that the error rate is similar [7, 8], but these are studies of electronic systems without dosing decision support. It is widely known that many EHR alerts are perceived as “noisy” (i.e., providing erroneous or irrelevant information) and are therefore overridden at high rates [9–11]. EHR implementers spend much energy on adjusting decision support rules of their systems to minimize this noise, with mixed results [12]. No one has identified an optimal signal-to-noise ratio that can be shown to be correlated with decreased prescribing error rates, but the general belief persists that reduction of noise will improve the effectiveness of prescribing decision support.

The challenge of reducing the noise in a set of dose-range decision rules is compounded by the very nature of pediatric practice. Many medications in common subspecialty use are not approved for use in children by the Federal Drug Administration (FDA), so they cannot have an approved dose range [3]. Moreover, standard dosing reference material used in pediatrics is inconsistent [13]. Add to that the observation that pediatric prescribers often use doses outside the range of published guidelines [7, 14] and it becomes clear that finding the optimal dose range for a given drug is not a straightforward task. Errors related to the amount of drug per dose are the most common type of error in pediatric prescribing [2, 8]. Of these errors, large overdoses are the greatest source of documented harm [15–17]. It is therefore considered a standard feature of EHRs used in pediatrics to provide support for dose calculation [18–21] and alerting for overdose errors.

One measure of dose-range-checking effectiveness is the extent to which large-overdose orders are detected and prevented. In our system, as in others [15], we noticed that prescribers were writing medication orders for large overdoses (as defined by the EHR’s dose-range rules) in both ambulatory and inpatient settings. While many of these so-called overdoses were surely the result of the aforementioned disagreement in the definition of dose range, a significant minority of these orders contained overdoses that were so high that they must have been the result of an error. Since the majority of alerts for these large overdoses were overridden by the user, we postulated that an analysis of these large overdoses would produce useful ideas for system changes that could prevent these overdose orders in the first place. We also wanted to define a potential metric that could indicate for which types of medications the alerts were more effective than in others.

Objectives

The objectives of this project were

1. to develop database techniques for detecting and extracting large overdose orders from our electronic health record system

2. to identify and characterize users’ responses to these large overdoses

3. to characterize User-CPOE (Clinical Provider Order Entry) actions associated with large overdose errors to find opportunities for preventive system configuration changes

Methods

Study Setting

Cincinnati Children’s Hospital Medical Center (CCHMC) is a 587-bed, freestanding children’s hospital with 13,000 employees and over 800 faculty members. It has over 30,000 admissions, 33,000 surgeries, 900,000 ambulatory encounters, and 125,000 emergency department visits per year. All ambulatory clinics are operated as part of the hospital. The institution brought the first ambulatory clinics live on an integrated electronic health record (EHR, EpicCare Inpatient®, Verona, WI) in late 2007. It brought the remainder of the system live in stages, including inpatient and perioperative areas in January 2010, and the final of 38 ambulatory departments in January 2012.

Drug-dosing Decision Support Configuration

Throughout this implementation, the EHR system has been configured to use the drug-dosing decision support rules database provided by Medi-Span (Wolters Kluwer Health, Philadelphia, PA). The publisher updates this database once per month; moreover, the pharmacy team at the hospital customizes these rules per local dosing guidance. ▶ Table 1 shows an example of the information contained in the single-dose rules for two forms of medications (methotrexate and acetaminophen). For the January 2010 inpatient go-live, the EHR pharmacy team converted a list of 3515 rules applying to 427 drugs from the legacy CPOE system to the new system. All were converted over a period from September 2009 to June 2010. Not all the existing rules fire in any given month, because not all dosing situations match all rules. In a typical month, only 36% (~2800) of all available generic product identifiers (GPIs) in the list of rules were associated with placed orders. These 36% of GPIs constituted about 60% of all dosing rules.

Table 1.

Information contained in rules for one form of parenteral methotrexate and a formulation of acetaminophen. The most specific rule that applies to the order is applied to generate the alert. Note that some rules for a specific amount (mg) to be given, and others are computed based on body surface area.

Medication	Route	Rule Conditions
		Patient age range (days)	Patient weight (kg)	Gestational age (days)	Dose range or maximum dose for single dose
Methotrexate 25 mg/mL injection	Intravenous	0 – 4,380			5–30 mg/m2
		4,381 – 36,135			5–50 mg/m2
	Intrathecal Oral	0 – 120			3 mg
		121 – 365			6 mg
		366 – 730			8 mg
		731 – 1,095			10 mg
		1,096 – 3,285			12 mg
		3,286 – 36,135			12–15 mg
Acetaminophen 160 mg/5 mL oral solution	0ral	0–9		196–230	12 mg/kg
				231–350	15 mg/kg
		10–29			15 mg/kg
		30–4379	0–9.999		15 mg/kg
		30–4379	10–40		15 mg/kg
		30–4379	40.001–100		1000 mg
		4380–99,999			1000 mg

Understanding Medication Order-Dosing Alert Workflows

While the EHR vendor provides a utility that creates a spreadsheet summarizing the medication alerts that fired over a given month, this report is meant to be a high-level overview. It does not allow trending across time or any detailed analysis involving the connection between the alert and the order. For this, a data mart designed specifically to answer certain questions about alert behavior and prescriber interaction is required. The EHR vendor makes all data available through a standard relational database. For any given research question, one must understand the clinical workflow that created the data and construct tables from the raw data. We used a test version of the system with its associated test database to walk through multiple ordering scenarios to determine how the data on alerts was recorded. ▶ Figure 1 illustrates the actions on the part of the user and the system that result in the logging of data elements related to alerts. Users can order multiple medications (and other orders) in one session. A single dialog box appears at the time the user indicates that the orders are ready to be signed. If one excludes the filtered alerts (alerts that are executed “behind the scenes” and normally hidden from the view of the prescriber) the user can cancel the signing process (presumably to make changes to one or more orders), remove individual medications from the batch (presumably to change something about an individual order), or override the alerts as a group. Users have the option to record reasons for overriding the alerts, but this is not required, and is seldom recorded. The option to record override reasons is not forced (is not a “hard-stop”) for several reasons, including the desire not to impede clinical workflows.

Fig. 1.

Clinical workflow and its effect on data recorded for each alert. Orders are typically placed in batches and signed in one operation. Alerts fire at the time of signing. This diagram indicates how user actions are reflected in the data about the alert. Alert status is recorded as one of five categories in the system; (1) filtered (user is not automatically shown the alert), (2) removed, (3) cancelled, (4) overridden, or (5) viewed (typically a nursing action and not part of the prescribing process).

Study Corpus

In this study, we focused on medications at doses that represented a ≥500% overdose (i.e., six times the amount of the upper range specified in the dose-range-checking rule, referred to as large overdoses) and a subset of these that represented ≥10,000% overdoses (extreme overdoses). Some 500% overdoses are merely proper dosing for medications given at high doses (e.g., chemotherapy) but we hypothesized that most 10,000% overdoses should reflect a fundamental, sociotechnical systemic error in how the medication order was constructed or handled by the system, and was not a valid order. For this study, we focused only on orders and the subsequent alerts that were generated during the prescriber ordering phase, and did not examine data from pharmacy verification or other staff viewing the orders.

Development Of Database Techniques For Detecting And Extracting Medication Overdose Information

The EHR vendor provides a relational database (data warehouse) into which data from the EHR is loaded on a daily basis (Oracle Corporation, Redwood Shores, CA). We developed routines for extracting data from this database into a data mart specifically designed to help answer questions about single-dose overdoses and their associated orders and alerts. To help simulate the behavior of the system, we included the dosing alert rules and basic patient information. This data mart contains these dimensions (▶ Figure 2):

Fig. 2.

Datamart schema. Multiple tables are joined, linking retrospective order and alert data to create a medication dosing alert analytic framework.

• Records of alerts (single-dose overdose) including data about the associated order (if not canceled), dose, medication, and an indication of which workflow contained the alert

• Medication orders, including patient weight at the time of the order

• Alert rules for dose ranges

• Patient demographics

• Diagnoses associated with the relevant encounters

• Encounter data (inpatient admission dates, office visit details)

• Medication metadata to indicate therapeutic and pharmaceutical class of each drug

Queries were performed in SQL Developer (Oracle Corporation, Redwood Shores, CA). The data spanned orders written from June 2011 through the end of May 2013. Some data points were extracted by simple text parsing routines from the stored text descriptions of the alerts.

To confirm our understanding of how data in the data mart reflected user actions and to ensure we understood how the proprietary EHR processed orders and alerts, we obtained access to a test version of the relational database that records EHR data and performed simulations of basic user actions that involved drug alerts. These tests were performed with the participation of our lead EHR project-team pharmacist [TM], with the aim of reproducing the alert output from the production system within our test system. If the output from both systems was reliably concordant, we assumed we had a complete understanding of how the proprietary software worked.

The rate of alerting in any system is highly dependent on the definition of the concept. In our system, we used the count of all non-filtered alerts that fired during the order-entry phase of order management (there are other phases, such as order verification, which did not contribute to the measure).

At the time this study was done, we had no access to data from the medication administration record or any data from the e-prescribing system on whether any of the ambulatory medication orders had been filled. In the case of inpatient or oncology day-hospital orders, all the medication orders are verified by a pediatric pharmacist before administration. We performed no temporal analysis to attempt to connect alerts to subsequent events in the medical record (e.g., to see if a non-overridden alert was associated with a dose change).

Characterizing User Response to Overdose Alerts

Given that user behavior could change only whether an alert was classified as removed, canceled, or overridden, we needed a metric that used only these data to characterize user responses. We formulated a measure of alert salience that approximated the degree to which a user reacted (in a corrective manner) to an overdose error by canceling the order. Since orders are processed in batches (i.e., one may write several medication orders at one time) and a batch of orders might generate several overdose alerts (one for each drug, potentially) it is not possible to know which alert caused the user to cancel the batch of orders, but we presumed that an alert that was more often associated with order cancellation reflected an alert that was being noticed more as a valid alert (was more salient) to users than other alerts. Users could also remove individual orders from a batch (▶ Figure 2) but this happened at such a low frequency (~0.25% of the time) that we excluded this data from the salience metric. We report our results related to user action in terms of salience, as defined by:

Salience rate descriptive statistics are presented in aggregate for >500% overdoses over the period of this study, in the context of prescriber role, and in relation to the magnitude of the medication order overdose. Salience rates for the large overdose alerts (>500%) were compared to the salience rates of all overdose alerts. Prescriber roles were based on EHR system user profiles, which were based on training and security levels. Finally, salience rates for all medications overdoses were plotted against medication overdose magnitude categories. Error bars for were computed using the Clopper-Pearson method based on the binomial distribution.

Extreme Overdose Categorization and Suggested Mitigation Actions

Lastly, we categorized the extreme overdose orders (as identified by their corresponding alert) into groups representing the probable underlying contributing prescribing system error, whether it was a prescriber error or the results of the CPOE system design. This was performed via manual inspection of the overdose details by the project team. Most overdose reasons were obvious (such as users substituting the wrong units, e.g., inhalers for puffs), while some categories required calculations to prove that users likely placed a medication order in the wrong scale (e.g., grams versus micrograms). We conclude by offering suggestions for system “fixes”, to prevent future recurrence of extreme overdoses. These proposed fixes were discussed, agreed upon, and reflect unanimous consensus opinion of all members of our project team.

Results

The swim lane diagram of our CPOE dosing rules evaluation system is shown in ▶ Figure 3. This system had three main components; the constructed order-alert data mart, the EHR test system, and the EHR production system order and alert reports. The evaluation system was used to gain a highly granular understanding of the user-EHR interactions and how this data is recorded in the production system. The interrogation allowed us to map detailed workflow diagrams (not published for proprietary reasons), with ▶ Figure 2 representing that workflow at a very abstracted level.

Fig. 3.

Swim lane diagram of the CPOE dosing rules evaluation system. The top lane represents the production system and its data warehouse, which provides data to the study data mart. The lower lane represents the test system. The data analyst interprets output from both the data mart and the test system after running medication-dosing scenarios.

The constructed order and alert data mart had 2,232,492 dose alerts and 5,402,504 medication orders from the study period. About 25–30% of all visible (unfiltered) alerts in our system were overdose alerts (▶ Figure 4). The ≥500% overdose alert counts decreased dramatically in November 2011 due to the identification and correction of a single medication dosing rule that was erroneous and pertained to a one frequently used medication, inhaled ipratropium (▶ Figure 5). Customization of that rule decreased the number of large overdoses. The ipratropium rule was inappropriately defined with parameters for the nasal formulation instead of the oral inhalation (nebulization) formulation. The typical dosing for these two formulations is significantly different – 84 mcg/dose for the nasal formulation versus 500 mcg/dose for the nebulization formulation. All orders placed using the inappropriate rule parameters resulted in large overdose alerts. The counts of monthly overdose alert totals and large overdoses (≥500% overdose) were relatively stable over the last year of the study.

Fig. 4.

Order-Alert rates. Rate chart of medication order alerts over the study period, by % visible alerts, % visible dosing alerts, and % visible overdose alerts in the Enter Orders phase of CPOE prescribing. Visible alerts are those alerts that are not filtered from view of the prescriber. The erratic behavior of this metric in the first 6 months can be attributed to discrepancies in order counts between datamart extracted data and the EHR utility-generated order counts. After some investigation, it was concluded that missing orders from the datamart are related to internal changes in how the EHR stores one category of the inpatient orders.

Fig. 5.

Total medication dose alerts and large overdose alerts (≥500% overdoses) over the study period. The total alert rate remained relatively stable over the study perior, while the large overdose alerts decreased after October 2011, then stabilized.

The salience rate was then calculated for the dosing alerts and trended over time (▶ Figure 6). Monthly salience rates varied mostly between 4–8%. Salience rates for each type of user are shown in ▶ Table 2. Trainees tended to have a slightly higher rate of reacting to the alerts compared to attending physicians. A nearly 3-fold increase of the salience rate in December of 2012 prompted us to investigate the data further. The increase in salience rate was the result of the cancellation of a large number of overdosed orders by a single provider, and occurred 1 month after the ipratropium rule fix.

Fig. 6.

Salience rate of overdose alerts in the “enter orders” phase by user action. Salience rates are shown for all single-dose overdose alerts and for ≥500% overdoses. A single spike in salience rate was attributed to one user.

Table 2.

Salience of overdose alerts by job role. Salience rates and alert counts by prescriber role. Salience (aggregate) was determined by aggregating all orders and alert data and calculating salience rates for each role as a whole. Average provider salience was determined by averaging the salience rates of each provider across the role (e.g., residents’ salience rates were averaged with equal weight, irrespective of the number of orders and alerts they were exposed to).

Prescriber Role	Alert Count	Salience (aggregate)	Average Provider Salience
Psychiatric Attending	6,596	2.2%	3%
Non-psychiatric Attending	40,583	7.4%	13%
Nurse Practitioner	25,483	7.9%	15%
Fellow (trainee)	11,722	9.6%	15%
Resident (trainee)	38,844	9.8%	17%
Anesthesiologist	6,128	13.5%	25%

▶ Figure 7 shows the salience rates for all dosing alerts, as well as the trends for several different therapeutic classes of medication. When considering all dosing alerts (dotted line), the salience rate was inversely proportional to the magnitude of the overdose. Investigation of the antineoplastic, as well as the analgesics & anesthetics therapeutic classes exhibited unusual trends.

Fig. 7.

Salience metric decreases with magnitude of overdose. As overdoses for a higher percentage over the upper dose range limit are ordered, users tend to override the alerts more often. This trend varies by therapeutic class, as shown by the contrasting patterns of salience for analgesics/anesthetics and anti-neoplastics (no variation in salience across the range of alerted high doses). Error bars were computed using the Clopper-Pearson method based on the binomial distribution.

Examination of extremely high overdoses (≥10,000%) led to a categorization of errors ▶ Table 3 shows the categorization and the relative frequency in our ≥10,000% sample. The most common reason for ≥10,000% overdose alerts was the system interpreting extended infusions as one-time doses. ▶ Table 4 displays the clinical factors related to the extreme overdose alerts.

Table 3.

Reasons for extreme overdose alerts, with relative frequency of their appearance in the database. The largest category of extreme overdose orders occurred as the result of how the alert logic interpreted a single infusion (occurring over several hours) as a single dose.

Extreme Overdose Alert-enabling Categories	Example	Description	Proposed Technical Fix	Count
Continuous med as onetime order	methotrexate infusion ordered with frequency of “once”, when it is actually infused over time	Dose is interpreted by the alert logic as a one-time dose (like an IV push) but it is delivered over several hours	Create a new “Infusion” frequency and dosing rules	209
Inappropriate unit selection	inhaler vs. puffs	User ordered # inhalers instead of by # puffs	Remove unit from pick-list or set hard stops for max doses	17
Unit scale issue	microgram vs. milligram vs. gram	User ordered wrong unit scale	Reduce ordering unit choices	14
Unit replacement issue*	milligram vs. milligram/ kilogram	Mass unit used instead of weight-based dosing	Application level changes (see detail below)	7
Value magnitude problem	power of 10 errors or „fat fingers”	User entered extra trailing digits	Create hard stop max dose limit	5
Mass/Volume unit substitution	milligram vs. milliliter	User substituted mass unit for volume unit, or vice-versa	Reduce ordering unit choices	4
Miscue errors	3350 gram of Polyethylene Glycol 3350 ordered	Prescriber used compound name as order mass value	User-centric issue, eliminate numbers from compound name, hard stop	2
Bad Alert Parameters	clindamycin rule found with erroneous maximum single-dose threshold	Electronic dosing rule parameters are erroneous	Adjust dosing rules to appropriate thresholds	1
Total				259

*If user enters a mg/kg dose order, the application provides a dosing calculator that displays total dose in mg. If the user replaces the mg/kg value with the total dose value, the unit of mg/kg remains, leading to a total mg value and mg/kg dosing unit (and potentially a large overdose). A corrective action would be to require the user to re-enter or confirm the unit after changing the dosing value.

Table 4.

Clinical factors related to extreme overdose alerts. Additional information regarding the clinical setting in which the extreme overdose alerts occurred.

Factor	Count (n= 259)
Inpatient vs. Ambulatory
Inpatient	253
Ambulatory	6
Hospital Unit
Unknown department	124
Hematology/Oncology	80
Acute care units	38
Critical Care	11
OR/ Post-Anesthesia Care Unit	5
Provider Role
Attending	202
Residents	22
Registered Nurse	11
Anesthesiologist	8
Fellow	8
Nurse Practitioner	6
Pharmacist	2

Discussion

Advantages of Creating an Advanced Analysis System

In this study, we describe the process and database techniques employed to gain an understanding of our CPOE/CDS (clinical decision support) system. The ultimate goal of gaining this insight was to more accurately describe the overdose alert burden and user behavior towards those alerts in our institution. Construction of our data mart was a resource-intensive effort, but we now have a platform tool for future research and operational studies in addition to the initial findings presented here. This analytic tool, as depicted in ▶ Figure 3, has allowed us to interpret more accurate medication ordering and dosing alert patterns than the vendor-supplied reports. Thorough evaluation of EHR medication ordering systems and associated clinical decision support systems is a difficult task, especially when some or all of the software is proprietary. Output from EHR reporting system utilities are often basic, too simplistic to translate in a clinically-relevant manner, and tend to reflect single transactional occurrences in databases (such as a single medication ordering occurrence), which does not allow for agile analysis of the temporal sequence that is typical of the ordering process. For example, attempts to order a single medication may create several orders and modifications, as well as CDS alerts, that are logged as individual transactions but really represent one continual prescriber experience. Typical reports of medication order and alert data may therefore be misleading and mis-representative of nature of the user-system interaction experience. Additionally, unless an analyst has “looked under the hood” of CPOE/CDS software or has access to an extremely granular data dictionary, the only way to understand how a proprietary system processes orders and alerts is by examination of the database schema and by experimenting with test scenarios and examining the resulting reporting output.

Counts and Trends of Orders and Alerts

The largest proportion of medication-related alerts seen by users at our institution are dosing alerts, particularly overdose alerts (▶ Figure 4 and ▶ Figure 5). These rates have remained relatively steady over the course of the study, although a single errant rule was found that, when corrected, dramatically decreased the rates of large overdoses (▶ Figure 5). This view revealed the effects of a single rule change for a medication (inhaled ipratropium) that was part of a very frequently used order set for asthma. Construction of our advanced analytics platform allowed us to detect this effect easily.

Salience Rate Findings

Salience rates were found to be remarkably low, indicating that prescribers were not often modifying the original medication order that triggered the overdose alert (▶ Figure 6). Even alerts corresponding to apparent overdoses of ≥500% (large overdoses) were rarely heeded. It was interesting to note the cancelation patterns of one user had on one month’s data point when he batch-cancelled a group of erroneous orders. It is unclear if alerts played a pivotal role in this recognition of large overdoses. The dramatic increase in large overdoses was even more apparent because the ipratropium rule customization had occurred the month prior, which lowered the number of large overdose alerts firing and, in turn, magnified the increase in the salience rate for that month. Without our analysis system these phenomena may have gone undetected, may have not been appreciated, and would not have helped identify system configuration changes to correct the issues.

Provider roles seem to influence salience rates (▶ Table 2), with learners and anesthesiologists heeding the alerts more than attending prescribers. The reason for this is yet unclear but is surely multifactorial in nature considering the level of training, medications classes prescribed, frequency of prescribing (attending physicians likely prescribe less than trainees in academic institutions) and other yet to be identified variables.

Salience rates demonstrated an inversely proportional response as the magnitude of the overdose increased, that is, users cancelled orders with overdose alerts less as the degree of the overdose increased (▶ Figure 7). This was partially due to one formulation (methotrexate) dominating the extreme overdoses group (methotrexate accounted for 194 of the 259 extreme overdoses). At CCHMC, the chemotherapeutic agents dosages are checked thoroughly by prescribers and pharmacists in an extra-CPOE system, before the doses are entered into our EHR. This careful dosing assessment likely instills confidence in the prescribers to the point that the CDS provided by the EHR is frankly ignored, which then lowers the salience rate. The higher salience rates at the lower levels of overdose magnitudes may result from prescribers paying more attention to alerts just beyond an accepted dosing threshold, since these alerts are less likely to be configured incorrectly or have bad parameters. Analysis of the salience rates of specific therapeutic classes of medications also produced interesting trends. The anti-neoplastic agents (including methotrexate) were reproducibly low across all overdose percentages, presumably for same reasons the extreme overdoses salience rate for all medications was dismal. The analgesics and anesthetics (AA) class salience rate trend is parabolic, with the outer overdose categories having the highest salience rates. Further investigation of the medication orders behind the 1,000–10,000% overdose AA class revealed that a significant portion of the orders (69 of 110) pertained to hydromorphone, a notoriously potent narcotic. Providers (at CCHMC, often anesthesiologists) are likely paying more attention to these alerts and cancelling them more often, increasing the salience rate. This behavior is also occurring when prescribing doses of analgesics that are just outside the upper limit of the dosing rule (<50% overdoses).

CPOE-CDS Configuration Correction Opportunities

Our analysis of the data so far show that extreme overdose orders occur despite the presence of alerts designed to prevent them. The reasons for these ≥10,000% overdoses vary, and the extent to which system changes can mitigate these errors also varies. Simple changes like removing an unlikely dosing unit (e.g., inhalers instead of puffs) are easy to do, but may be hard to detect within the mass of alert data produced by EHR systems in production. The output of our first two objectives informs and allows the third objective – correcting the prescriber-EHR system to prevent future recurrences and optimize CDS.

▶ Table 3 offers solutions to the extreme overdose alert-enabling issues. Some of the solutions are technical in nature, some are modifications in user behavior, and others require a combined socio-technical approach. In the first category (“Continuous med as one-time order”), a new ordering frequency should be created to accommodate infusions, that would distinguish the type of dosing rules to be applied from those that are applied to single doses. Reducing or limiting medication order unit picklist options and/or setting reasonable “hard stop” alert parameters would help mitigate five of the eight mechanisms that contribute to extreme overdoses (inappropriate unit selection, unit scale issue, value magnitude problem, mass/volume unit substitution, and miscue errors). Unit replacement issues represent a phenomenon that is largely technical in nature and a fix would require programmatic changes to the software code. Finally, in this corpus of large overdose alerts, only 1 instance of alerting occurred due to bad alert parameters. This is in contrast to previous work, which has shown that electronic dosing alert rules poorly match traditional dosing guidelines from non-electronic sources [13]. It is almost certain that investigation of less egregious overdose alerts will demonstrate an increased proportion of alerts due to bad rule parameters. Adjustment of the erroneous or suboptimal dosing rule thresholds themselves should be performed to prevent recurrence of overdose alerts due to this enabling factor.

Future Work

Moving forward we hope to make this advanced data analysis more accessible by providing a web-based business intelligence tool. We will also be expanding the catalog of analytic metrics, including more granular and temporal-based measures. Additional medication administration record (MAR) and audit trail information with timestamps will be added to aid in the temporal analysis of evolving orders, so that we can investigate the chain of CPOE-CDS events that occurs across multiple orders from multiple ordering sessions. We also plan to expand our analysis to cover actions of other providers in the medication ordering to administration cycle, namely pharmacists and nurses.

Limitations

Because we intentionally avoided analysis of medication administration or ambulatory medication order fills, we have no direct information to suggest that any of these large doses ever reached a patient. Additionally we have no reports through our voluntary safety reporting system of overdoses of these magnitudes reaching a patient. While we can speculate that this means that the doses were either intercepted or were in fact correct, our current methodology prevents any conclusion of this type. Additional data loads to our analysis system will allow us to determine the end result in future studies. Dosing alerts for one medication may fire simultaneously with other concomitantly ordered medication alerts. If users take action on a batch of all orders – such as canceling multiple medications in batch due to the recognition of a dosing error in one medication – the same status (canceled, in this example) will be applied to all alerts that fired in the same alert dialog box despite the appropriateness of the second alert. This may cause inaccurate assumptions of specific alert performance. However, in preliminary analyses by our team, users order only 1 medication during an ordering session a majority of the time, which mitigates this concern somewhat. Another limitation of this study is that we have not yet demonstrated a change to our system that decreases the alert overload. To get to this level, we must be able to make changes that affect a far larger percentage of the medication dosing rules. Our established framework and metrics will expedite this objective. Finally, the lack of sophisticated temporal analysis limits our understanding about whether a user’s familiarity with the system, or maturity as a clinician, affects the placing of large overdose orders. We would hypothesize that clinicians with more experience (e.g., attending oncologists) would be more comfortable with large, correct doses, and that clinicians with less experience would be more prone to writing large, incorrect doses.

Conclusions

Novel analytic systems are required to accurately understand prescriber behavior and interactions with medication-dosing CDS. We described a novel analytic system that can detect apparent large overdoses (≥500%) and explain the sociotechnical factors that activated the alerts. Some of these large overdose errors can be mitigated by system changes. EHR system design should take into account the possible pathways for the creation of large overdose medication errors.

List of Abbreviations

• EHR – electronic health record

• FDA – Federal Drug Administration

• CPOE – Clinical Provider Order Entry

• CCHMC – Cincinnati Children’s Hospital Medical Center

• GPI – generic product identifiers

• CDS – clinical decision support

• AA – analgesics and anesthetics

• MAR – medication administration record