Abstract
Objective
To determine the accuracy of vendor-supplied dosing eRules for pediatric medication orders. Inaccurate or absent dosing rules can lead to high numbers of false alerts or undetected prescribing errors and may potentially compromise safety in this already vulnerable population.
Materials and methods
7 months of medication orders and alerts from a large pediatric hospital were analyzed. 30 medications were selected for study across 5 age ranges and 5 dosing parameters. The resulting 750 dosing rules from a commercial system formed the study corpus and were examined for accuracy against a gold standard created from traditional clinical resources.
Results
Overall accuracy of the rules in the study corpus was 55.1% when the rules were transformed to fit a priori age ranges. Over a pediatric lifetime, the dosing rules were accurate an average of 57.6% of the days. Dosing rules pertaining to the newborn age range were as accurate as other age ranges on average, but exhibited more variability. Daily frequency dosing parameters showed more accuracy than total daily dose, single dose minimum, or single dose maximum.
Discussion
The accuracy of a vendor-supplied set of dosing eRules is suboptimal when compared with traditional dosing sources, exposing a gap between dosing rules in commercial products and actual prescribing practices by pediatric care providers. More research on vendor-supplied eRules is warranted in order to understand the effects of these products on safe prescribing in children.
Keywords: Electronic Health Record; Electronic Medical Record; Medical Order Entry System; CPOE; Decision Support Systems, Clinical; Medication Alert System
Introduction
Clinical decision support (CDS) for medication dosing often relies on vendor-supplied rules integrated within an electronic health record's (EHR's) computerized provider order entry (CPOE) system. These electronic rules (eRules) contain dosing parameter thresholds that can be used to initiate alerts on rule violation. eRules with thresholds inconsistent with common dosing practices can cause over-alerting and alert fatigue, as well as inappropriate dosing of medications.1–4 Alert fatigue undermines CDS because it leads users to ignore both accurate and inaccurate alerts. In addition, an absence of eRules can lead to undetected prescribing errors. Accurate drug dosing in CDS is particularly important in pediatric settings.
To date, no study has examined the accuracy of vendor-supplied eRules compared to traditional pediatric dosing guidelines found in commonly-used sources, such as authoritative textbooks and online references. These longstanding sources are accepted as the gold standard for pediatric dosing and reflect everyday prescribing behaviors. This study takes the first step towards understanding the magnitude of accuracy of eRules in the pediatric setting.
Background and significance
High numbers of medication-related errors and adverse drug events are known to occur in both inpatient and outpatient settings.5–9 In one adult study, most preventable adverse drug events occurred during drug ordering.10 In another large study, 6% of all medication orders had errors. Serious medication errors occurred in 10 of 100 admissions, over half of which were dosing or frequency errors.5 Incorrect dosing is also the most common cause of death in the FDA's Adverse Event Reporting System.11 Children are especially vulnerable to dosing-related adverse drug events. Folli found that the majority of medication errors related to dosing were in the most vulnerable children, those <2 years of age or intensive care unit patients.3
Drug dosing accuracy is especially difficult in pediatrics because of many age- and size-specific considerations that add to the complexity of prescribing medications to this population. Pediatric drug dosing is more complex than adult dosing due to weight-based dosing, varying drug metabolism and physiology during development, and the increased off-label use of medications in children.12–16 Online supplementary appendix 1 lists the more common factors that increase the complexity of prescribing medications to children.
EHRs with CPOE and effective CDS have shown the potential to reduce the risk of drug-related harm, including within the pediatric population.4 5 17 18 20–23 Alerts are one common form of CDS used to assist with dosing guidance in CPOE systems. However, the drug-dosing rules used in CDS to trigger alerts generally are not tailored for pediatric use.
We conducted a retrospective, quantitative, cross-sectional study to explore and characterize inaccuracies within a vendor-supplied set of dosing eRules compared to accepted dosing rules from traditional sources. The specific aims of the study were to determine match characteristics between vendor-based dosing eRules and dosing rules from traditional sources, and to determine which medications, age groups, and dosing parameter combinations most closely match traditional dosing guidelines. The Institutional Review Board of Cincinnati Children's Hospital Medical Center (CCHMC) deemed this research exempt from review.
Materials and methods
CCHMC is a 577-bed quaternary care medical center with over 1 million patient encounters per year. CCHMC has a fully-implemented EHR (Epic Systems Corporation, Verona, Wisconsin) with CPOE system that generates roughly 300 000 drug alerts per month. Fifty percent of those alerts are in the drug-dosing category. Medication order alerts are triggered by the CPOE system referencing eRules supplied by one of the industry-leading proprietary drug dosing database vendors.
Creation of the study corpus
All medication alerts and orders from the CCHMC clinical data repository (CDR) from June 1, 2011 through 31 December 2011 were obtained. During this period, there were 1 276 156 unique orders and 1 819 028 unique alerts. Inpatient orders comprised 63.3% of the corpus, with ambulatory orders comprising the remaining 36.7%. All prescribed orders and alerts were included, irrespective of clinical area, provider type, patient type, or location of administration, except as noted below. From these, three groups (the ‘medication groups’) were assembled: the most commonly prescribed medications (‘common medications’), the most frequently alerted medications (‘most alerted medications’), and the medications with the highest rates of alert override (‘medications with highest override rate’).
Orders for intravenous fluids and vaccines were excluded from consideration in the final corpus because the architecture of the CPOE system aggregates these orders into high-level groupings that do not permit deft analysis. Medications with less than 200 orders during the study period were also excluded to remove infrequently ordered medications from consideration.
The 10 most frequent medication formulations from each group were initially selected for study. Some medication formulations were found to belong to more than one group (eg, prednisolone 15 mg/5 mL oral solution in all three groups). Because of this duplication, unique medication formulations were added such that the final corpus had 30 unique medication formulations. By group, it contained the top 14 commonly prescribed medications, the top 13 most alerted medications, and the top 13 medications with most alert overrides (table 1).
Table 1.
Study corpus medication formulations by medication group
| Common medications | Most alerted medications | Medications with highest override rates | |
|---|---|---|---|
| Inclusion rationale | Users will require/interact with dosing rules for these drugs frequently | Highly-alerted drugs may predict poor underlying dosing rules | High override rates may indicate poor dosing rules; alerts did not change prescribing behavior |
| Selection criteria | Highest counts of distinct drug orders over the study period | Drugs with the highest counts of dosing-related alerts | Drugs with the highest override rates (and ordered a minimum of 200 instances during the study) |
| 1 | Acetaminophen* 80 mg/0.8 mL PO suspension | Acetaminophen* 80 mg/0.8 mL PO suspension | Paricalcitol IV solution 5 mcg/mL |
| 2 | Fentanyl citrate* 0.05 mg/mL injectable solution | Oxymetazoline HCl* 0.05% solution | Ipratropium bromide* 0.02% in solution |
| 3 | Ibuprofen* 100 mg/5 mL PO suspension | Acetaminophen* 325 mg PO tablets | Prednisolone sodium phosphate* 15 mg/5 mL PO solution |
| 4 | Morphine sulfate 1 mg/mL injectable solution | Fentanyl citrate* 0.05 mg/mL injectable solution | Pentobarbital sodium injection 50 mg/mL |
| 5 | Acetaminophen* 325 mg PO tablets | Lidocaine cream 4% | Epoetin alfa 2000 unit/mL injection solution |
| 6 | Oxymetazoline HCl* 0.05% solution | Polyethylene glycol 3350 oral powder | Antihemo factor-vwf 1000–2000 unit IV solution |
| 7 | Albuterol sulfate (2.5 mg/3 mL) 0.083% Nebulization | Ondansetron HCl* 4 mg/2 mL injectable solution | Fluoxetine HCl 20 mg PO capsules |
| 8 | Albuterol sulfate HFA 108 μg/act in aerosol | Prednisolone sodium phosphate* 15 mg/5 mL PO solution | Oxybutynin chloride 5 mg/5 mL PO syrup |
| 9 | Ondansetron HCl* 4 mg/2 mL injectable solution | Ciprofloxacin HCl 0.3% ophthalmic solution | Infliximab IV injection 100 mg |
| 10 | Bupivacaine HCl 0.25% injectable solution | Ibuprofen* 100 mg/5 mL PO suspension | Meropenem IV for solution 500 mg |
| 11 | Ibuprofen* 200 mg PO tabs | Ibuprofen* 200 mg po tabs | Rocuronium bromide IV solution 10 mg/mL |
| 12 | Ciprofloxacin-dexamethasone 0.3–0.1% otic suspension | Silver nitrate-potassium nitrate applicator 75–25% | Aspirin chew tablet 81 mg |
| 13 | Prednisolone Sodium* Phosphate 15 mg/5 mL PO solution | Ipratropium bromide* 0.02% in solution | Metronidazole tablet 500 mg |
| 14 | Cefazolin 100/mL injectable solution | ||
*Denotes medication belongs to more than one medication group.
IV, intravenous; PO, oral.
These 30 medication formulations were then expanded by both dosing parameters and age groups to generate medication/dosing parameter/age group triads. Five medication dosing parameters—total daily dose, single dose minimum, single dose maximum, daily frequency minimum, and daily frequency maximum—were combined with each medication formulation to create five new medication/dosing parameter dyads. Each dyad was further divided by five a priori age ranges, creating medication/dosing parameter/age group triads. The five age groups correlated to ages that are accepted as similar physiologically and developmentally: newborns (0–29 days), infants (30–364 days), toddlers/pre-school children (365–1824 days or 1–5 years), school-age children (1825–4379 days or 5–12 years), and adolescents (4380–6570 days or 12–18 years). This process resulted in 750 medication/dosing parameter/age group triads (a priori rules).
The eRules corresponding to each medication triad were then documented. At times, the age groups in the eRules did not match the a priori age ranges. When needed, additional triads were added to match the age specificity of the eRule. For instance, when the meropenem intravenous solution total daily dose eRule had dosing ranges of 0–7 days and 8–29 days, the dosing rule for the medication triad ‘meropenem total daily dose for newborns’ was replaced by two rules representing the two separate age groups in the eRule. Frequently, however, eRule dosing age ranges were large and spanned several of the a priori age ranges, for example a dosing rule that spanned 0–364 days and overlapped both newborn and infant ranges. Diagnosis-related dosing and rules modified based on creatinine clearance were not considered per se, although the widest possible dosing ranges from all rules available were recorded.
Creation of the gold standard
Five traditional and respected sources were selected for constructing the gold standard medication-dosing guidelines: Harriet Lane Handbook (19th edition),24 PDR.net (Physician's Desk Reference),25 Epocrates Online,26 Micromedex 2.0,27 and Lexi-Comp Online (CCHMC formulary).28 For each medication triad, information from all traditional sources was aggregated into a gold standard rule by finding the most common doses and units among the traditional guidelines. For example, if acetaminophen dosing was 75 mg/kg/day from two sources and 90 mg/kg/day from three sources, then 90 mg/kg/day was selected as the gold standard. If the majority of the sources had no rule available but a corresponding eRule existed, then available rules from the traditional sources were used as the gold standard to encourage matching (bias towards matching).
Dose rule matching
Analysis of the eRules began by comparison of each uncustomized (not locally modified) eRule against its gold standard (see online supplementary appendix 2). eRules were deemed to either match or not match (mismatch). Matches were instances where the gold standard and the eRule had exactly the same values and units (‘rule match’), or where no rule existed for either (‘no rules available’). Mismatches occurred when dosing rule units (mg, mL, mg/kg, etc) were identical but values were not (‘value mismatch’; such as 10 vs 20 mg), when the units of the eRules and gold-standard dosing rules were not equivalent and comparisons could not be made (‘unit mismatch’; such as 10 mg vs 10 mg/kg), when a gold-standard rule could not be constructed but an eRule was present (‘eRule only’), or when no eRules were available but a gold standard rule existed (‘absent eRule’). Value mismatches were further subdivided by their tendency to over-alert or under-alert. For instance, a single dose maximum eRule of 10 mg/kg would over-alert if the gold standard rule was 20 mg/kg and orders between 10–20 mg/kg were placed.
Primary analysis
To evaluate the accuracy of eRules and gold-standard dosing rules in a clinically relevant manner, matching was first analyzed by investigating the quality of matching across the five a priori age ranges. The first step was to map the matched/mismatched corpus dosing rules to the a priori age ranges. Any generated corpus rules that had an overlapping age range with the a priori age range of interest was assigned to, and considered part of, the a priori age group. For example, cefazolin 100 mg/mL injectable solution had two rules that overlapped the newborn period (rule 1 for 0–7 days, rule 2 for 8–364 days). Both rules had to match exactly for the dosing rule to be considered a match across the newborn period. If either one or both rules did not match, the eRule for that age range was considered a mismatch. In effect, this procedure transformed the dosing rules created in earlier steps into rules that fit the a priori age ranges (figure 1). Descriptive statistics of matching were calculated for the corpus by age range, dosing parameter, combinations of medication groups and age range, as well as combinations of medication groups and dosing parameters.
Figure 1.
An example of matching electronic dosing rule (eRule) age ranges to a priori age ranges. In this example, eRule age ranges were mapped to a priori age ranges for the primary analysis. Both the 0–7 day and 8–29 day eRules mapped to the newborn age range, while the 30–4380 day rule mapped to the infant, toddler, and school-age ranges. Examples of primary and secondary analyses are shown. New, newborn; M, match of eRule and gold standard; NM, mismatch of eRule and gold standard.
Secondary analysis
For the secondary analysis, the rate of rule matching across the entire typical pediatric age range was analyzed, without considering the a priori age ranges used in the primary analysis. In this analysis, the proportion of the pediatric lifetime, that is, the first 6570 days (0–18 years) of life, that a given eRule matched the gold standard was calculated (figure 1). For example, if the rules for a medication only matched during the 0–364 days range (but all other age ranges did not match), the match percentage was 364/6570 days, or 5.5%. This analysis was performed to give another sense of how frequently a match between eRules and gold standard rules would occur on any given day of a patient's first 18 years of life, irrespective of a priori age ranges. Rate of matching across all corpus rules, and by dosing parameter and medication group, was calculated.
Gold standard agreement analysis
To evaluate the strength of the constructed gold standard rules, the level of agreement among the five sources was assessed. Evaluation of the agreement between sources on each of the rules, as well as comparisons of rules with eRule matches versus those without eRule matches, was performed.
Dose rounding logic
Some CPOE and eRule vendors include software logic that permits acceptance of a range of dosing around an eRule (often set to ±5% or 10% the value of the eRules) to mitigate over-alerting that may occur due to rounding of doses. Such logic, if enabled, could lead to increased matching rates between the constructed gold standard rule and the eRule. Analysis was performed in this study to evaluate the matching effect of a ±10% allowance.
Results
Primary analysis
A total of 750 pairs of a priori dosing rules were compared. Table 2 displays the aggregate number of matches and proportion of rules that matched for age range and dosing parameter. Each age range and dosing parameter grouping consisted of 150 rules after transformation of the original rules to fit the a priori age ranges. There was a similar proportion of matching across all five age categories (range: 52–57.3%; table 2). The range of matching percent when analyzed by dosing parameter was wider (46–70%; table 2). Of the dosing parameters, daily frequency minimum and daily frequency maximum had the best fit. The mean match of all a priori adjusted dosing rules was 55.1%.
Table 2.
Dosing rule match rates based on a priori age ranges, by age range and dosing parameter
| # Matches | # Rules | Match rate (%) | |
|---|---|---|---|
| Age range | |||
| Newborns (0–29 days) | 81 | 150 | 54.0 |
| Infants (30–364 days) | 78 | 150 | 52.0 |
| Toddlers/pre-school (365–1824 days) | 83 | 150 | 55.3 |
| School-age children (1825–4379 days) | 86 | 150 | 57.3 |
| Adolescents (4380–6570 days) | 85 | 150 | 56.7 |
| Total | 413 | 750 | 55.1 |
| Dosing parameter | |||
| Total daily dose | 71 | 150 | 47.3 |
| Single dose minimum | 69 | 150 | 46.0 |
| Single dose maximum | 70 | 150 | 46.7 |
| Daily frequency minimum | 105 | 150 | 70.0 |
| Daily frequency maximum | 98 | 150 | 65.3 |
| Total | 413 | 750 | 55.1 |
When analyzed by a combination of medication group and age range, common medications had a higher match rate (66.3%) than the most alerted medications (61.2%) or the medications with the highest override rates (46.5%; table 3). The medications with the highest alert override rates group also showed the largest range, from 32.3% to 55.4%. Newborn common medication dosing rules had the best fit of all age ranges (78.6%), but the worst fit in the medications with the highest override rates group (32.3%).
Table 3.
Dosing rule match rates based on a priori age ranges, by medication group and age range, and medication group and dosing parameter
| # Matches | # Rules | Match rate (%) | |
|---|---|---|---|
| Medication group/age range | |||
| Common medications | |||
| Newborns (0–29 days) | 55 | 70 | 78.6 |
| Infants (30–364 days) | 48 | 70 | 68.6 |
| Toddlers/pre-school (365–1824 days) | 43 | 70 | 61.4 |
| School-age children (1825–4379 days) | 44 | 70 | 62.9 |
| Adolescents (4380–6570 days) | 42 | 70 | 60.0 |
| Total | 232 | 350 | 66.3 |
| Most alerted medications | |||
| Newborns (0–29 days) | 40 | 65 | 61.5 |
| Infants (30–364 days) | 35 | 65 | 53.9 |
| Toddlers/pre-school (365–1824 days) | 38 | 65 | 58.5 |
| School-age children (1825–4379 days) | 43 | 65 | 66.2 |
| Adolescents (4380–6570 days) | 43 | 65 | 66.2 |
| Total | 199 | 325 | 61.2 |
| Medications with highest override rates | |||
| Newborns (0–29 days) | 21 | 65 | 32.3 |
| Infants (30–364 days) | 27 | 65 | 41.5 |
| Toddlers/pre-school (365–1824 days) | 36 | 65 | 55.4 |
| School-age children (1825–4379 days) | 33 | 65 | 50.8 |
| Adolescents (4380–6570 days) | 34 | 65 | 52.3 |
| Total | 151 | 325 | 46.5 |
| Medication group/dosing parameter | |||
| Common medications | |||
| Total daily dose | 45 | 70 | 64.3 |
| Single dose minimum | 32 | 70 | 45.7 |
| Single dose maximum | 40 | 70 | 57.1 |
| Daily freq minimum | 55 | 70 | 78.6 |
| Daily freq maximum | 60 | 70 | 85.7 |
| Total | 232 | 350 | 66.3 |
| Most alerted medications | |||
| Total daily dose | 32 | 65 | 49.2 |
| Single dose minimum | 36 | 65 | 55.4 |
| Single dose maximum | 37 | 65 | 56.9 |
| Daily freq minimum | 50 | 65 | 76.9 |
| Daily freq maximum | 44 | 65 | 67.7 |
| Total | 199 | 325 | 61.2 |
| Medications with highest override rates | |||
| Total daily dose | 24 | 65 | 36.9 |
| Single dose minimum | 31 | 65 | 47.7 |
| Single dose maximum | 23 | 65 | 35.4 |
| Daily freq minimum | 39 | 65 | 60.0 |
| Daily freq maximum | 34 | 65 | 52.3 |
| Total | 151 | 325 | 46.5 |
Table 3 displays the match rates for dosing rules when analyzed by medication group and dosing parameter. In each medication group, the frequency dosing parameters had a higher match rate than total daily dose, single dose minimum, or single dose maximum.
Secondary analysis
Dosing rule match rates were calculated for the number of days that eRules matched the gold standard rules over the course of a pediatric lifetime. As shown in table 4, the daily frequency parameters had higher match rates (82.1% and 69.1%) than total daily dose, single dose minimum, or single dose maximum rules (50.0%, 45.2%, and 49.4%, respectively). The match rates of dosing rules by medication group and dosing parameter are shown in table 4. The number of matching days for a common medication, when the rule pertained to total daily dose, was 70%. In other words, 70% of the time the total daily dosing eRule for one of these drugs will match the gold standard and the most common traditional dosing guidelines. This translates to 4600 days in the course of a patient's pediatric lifetime that an eRule would be accurate.
Table 4.
Dosing rule match over the pediatric lifetime (0–18 years) by dosing parameter and medication group and dosing parameter
| # Matching/total days | Match rate (%) | |
|---|---|---|
| Dosing parameter | ||
| Total daily dose | 3283/6570 | 50.0 |
| Single dose minimum | 2972/6570 | 45.2 |
| Single dose maximum | 3248/6570 | 49.4 |
| Daily frequency minimum | 5393/6570 | 82.1 |
| Daily frequency maximum | 4541/6570 | 69.1 |
| Mean match rates of all dosing parameters | 59.2 | |
| Medication group/dosing parameter | ||
| Common medications | ||
| Total daily dose | 4600/6570 | 70.0 |
| Single dose minimum | 2460/6570 | 37.5 |
| Single dose maximum | 3600/6570 | 54.8 |
| Daily frequency minimum | 5184/6570 | 78.9 |
| Daily frequency maximum | 5482/6570 | 83.4 |
| Mean match rate of common medications | 64.9 | |
| Most alerted medications | ||
| Total daily dose | 3176/6570 | 48.4 |
| Single dose minimum | 3002/6570 | 45.7 |
| Single dose maximum | 3568/6570 | 54.3 |
| Daily frequency minimum | 5039/6570 | 76.7 |
| Daily frequency maximum | 4600/6570 | 70.0 |
| Mean match rate of most alerted medications | 59.0 | |
| Medications with highest override rates | ||
| Total daily dose | 2738/6570 | 41.7 |
| Single dose minimum | 3229/6570 | 49.2 |
| Single dose maximum | 2421/6570 | 36.9 |
| Daily frequency minimum | 4648/6570 | 70.8 |
| Daily frequency maximum | 3543/6570 | 53.9 |
| Mean match rate of medications with most alert overrides | 50.5 | |
Again, the highest match rates were found in the daily frequency parameters (table 4). The data also show a higher match rate for the common medications (64.9%) than for the most alerted medication (59.0%) or the medications with highest override rates (50.5%).
Figure 2 shows the proportion of matching and the various causes of mismatching in this analysis. eRule value mismatches that would lead to under-alerting providers to incorrect dosing comprised the largest portion of the mismatches (15.2%), followed by value mismatches that would lead to over-alerting (14.1%). Units of measure that could not be compared (eg, mg vs mg/kg) and scenarios lacking either an eRule or gold standard rule caused many fewer mismatches.
Figure 2.
The proportion of eRules that matched and mismatched the gold standard rules over the course of the pediatric lifetime (0–18 years). A pie chart demonstrating the proportion of matches, as well as mismatches by underlying etiology for the mismatch, for the secondary analysis results.
Gold standard analysis
Figure 3 displays the level of agreement between the gold standard references. A majority consensus (≥3 out of 5 gold standard sources agreed) existed in 76.2% of the rules in the study corpus.
Figure 3.
The agreement level of the gold standard reference rules. Bar graph representation of the amount of traditional dosing source rules that agreed with each other. The x-axis represents the level of agreement among the five gold standard sources while the y-axis represents the percentage of constructed gold standard (GS) rules that fit into that level of agreement. Analysis is subdivided by matching and non-matching GS eRule categories.
Dose rounding logic
Adjustment of the vendor eRules for a ±10% allowance to account for dose rounding logic only allowed one additional rule to match (<0.1% of the rule corpus).
Discussion
This study's primary aim was to evaluate how well one vendor-supplied set of dosing eRules matched common and accepted pediatric prescribing rules. The most notable finding from the study data was the clinically significant inaccuracy in the eRules in all groups. The aggregate match rate was 55.1% for all a priori dosing rules in the primary analysis, indicating that only about 1 out of every 2 eRules exactly matched the dosing rules in the gold standard traditional pediatric dosing rule sources. The secondary analysis of the match rates, without considering age, confirmed these findings, showing that only 57.6% of all pediatric days covered by the eRules were accurate. Low rates like these must be a strong contributor to the high number of false alerts reported in the literature.
The accuracy of eRules did not drastically change when evaluated by age group. Results for newborns, generally considered the most vulnerable age group, were similar to those of all other age groups although the newborn age range eRules accuracy did exhibit greater variability than other age groups in some parameters. Both the best (78.6%, common medications group) and the worst matching rate (32.3% medications with the highest override rates group) for all age ranges occurred in the newborn eRules. We believe several factors are responsible for these observations. Thirty-seven of the 55 newborn rule matches in the common medication group matched because no rules were available in either the eRules or the gold standard. The matching rate would be considerably lower (54.5%) if this subset of ‘matches’ were removed from the calculation. The opposite is true for the medications with the highest override rates group. Only 12 newborn rules in that group matched because no rules exist in either source. From a clinical perspective, it should be noted that most of the group of medications with the highest override rates are not commonly used in neonates, which likely limits the clinical impact of the high mismatch rates. However, other age ranges, where use of the medications is more prevalent, also demonstrated low matching rates for some groups. In these age ranges, low rates are of more clinical significance since they potentially increase the number of false dosing alerts.
Examination of the accuracy of eRules by dosing parameter showed that the daily frequency minimum and maximum eRules were consistently more accurate than total daily dose, single dose minimum, or single dose maximum eRules. This trend persisted when analysis was performed on medication groups and in both the primary and secondary analyses, likely due to the fact that dosing frequencies for many of the medications in the study corpus fall with a limited range, typically 1–4 doses per day. Other dosing group values are often much more variable, thereby increasing the chances of a mismatch.
Examination of the accuracy of eRules by medication group shows that the common medications were more like the gold standard rules than the most alerted medications rules and were much more accurate than the eRules from the medications with the highest override rates group. Explanations for this trend include that common medications are more likely to have formal established dosing rules than the other two medication subgroups. In addition, the medications with the highest override rates tended to be more specialized medications, compared to other groups. This group only required a relatively small number of orders (>200), thereby allowing for inclusion of less frequently prescribed, more-specialized medications. More-specialized medications are likely to have less formal or established dosing rules, which would lead to lower match rates and accuracy of the eRules for this subset.
Construction of the gold standard rules proved to be a difficult task. The analysis of the traditional sources used to create the gold standard reveals a relative heterogeneity of rules. While this may ‘tarnish’ the gold standard a bit, it also reflects the challenging task vendors have to create eRule databases that fit with real-world provider prescribing patterns while not putting patient safety at risk. However, over three-quarters of the gold standard rules created were the product of a majority consensus among the five sources. In cases where there was no clear majority, we made every reasonable attempt to select a gold standard rule that would match with the partner eRule. In short, we actually provided a bias towards matching, yet the study still demonstrated low matching rates.
The allowance for dose rounding logic, which increased the dosing parameter thresholds by ±10%, would theoretically improve matching rates as it expands the acceptable dosing ranges set by the vendor eRules. In this study, however, the matching rates were not appreciably improved by including this allowance, converting only one non-matching gold standard–eRule pair into a matching pair.
Regardless of approach, accuracy discrepancies of the magnitude found in this study highlight several important aspects of both pediatric dosing and use of vendor-supplied eRules to provide CDS for prescribing pediatric medications. First, these findings highlight that the eRules supplied by commercial vendors may not accurately reflect how pediatric providers are prescribing medications in everyday practice. It is possible that vendors of dosing eRules, for a variety of reasons, include only strictly accepted dosing guidelines, such as FDA-approved values, in their products while many medications are used for off-label indications and prior to rigorous pharmacologic study in pediatric populations.16 If this is true, then eRules should be modified by the institutions implementing them.
Second, the findings of the analysis underscore previously published literature that describes the inherent difficulties in dosing medications in children.3 5–9 17 29–31 Unique requirements such as weight-based dosing and other factors increase the heterogeneity of dosing rules seemingly exponentially, making it more difficult for vendors to create products that offer optimal pediatric CDS.
Finally, while the data presented in this report expose a deficiency in medication dosing support in pediatrics, the discrepancies discovered should be viewed as opportunities to improve current systems. Pediatric-specific rules and products should be designed and developed with the issues noted here in mind.
Strengths
Strengths of this study include that the study corpus was based on a large number of medication orders and alerts over 7 months, minimizing the potential for a sampling bias that may have resulted from data collected in a shorter time period. The medication orders and alerts were also derived from multiple practice settings, including inpatient units, outpatient/ambulatory settings, and the emergency department. Inclusion of orders from this variety of settings improves the generalizability of the results. Fourteen types of medication formulations (eg, injectables, suspensions, tablets, creams) were also included in the corpus medications, representing a wide array of formulations. Last, exceptions were made in the study design to encourage dose rule matching when feasible and logical. By doing so, the study attempted to err in a direction that would improve matching rates.
Limitations
The findings from this study were, however, based on data from only one institution and from interrogation of one vendor product. There is also a risk of selection bias mediated through the inclusion criteria chosen for the medication subgroups, with two of the subgroups chosen explicitly because of their potential to generate false alerts. It could be expected that the match rates for dosing rules from these groups would be lower than for commonly prescribed medications. However, those groups are responsible for a large proportion of the false alerts in CCHMC's system and represent targets for future dosing rule improvement. Including those medication groups highlights the disparities that exist and the need for rule reconfiguration.
Finally, intravenous fluids and vaccine dosing rules could not be evaluated due to the configuration of CCHMC's CPOE system.
Future studies
There are ample opportunities to further the work presented in this report. Repetition of the protocol in other healthcare organizations, with eRules from other vendors, using different medication groups, including intravenous fluids and/or vaccines, and examining matching rates based on ordering locations all might yield more information. Unless significant differences from this study are found, however, the next important work must be to determine whether or not quality of clinical care is truly affected by differences in dosing eRules and common ordering practice. Alert fatigue is postulated but not proven in this study, and, if truly present, the magnitude of its effect on clinical care is unknown.
Conclusions
This study evaluated the accuracy of medication dosing rules from a vendor-supplied set of eRules. When compared to a gold standard comprised of rules from traditional dosing sources, low rates of matching were found. The newborn age range was found to have the most variability in matching, demonstrating both high and low rates of congruence in different types of medications. Dosing parameters concerning daily frequencies (minimum and maximum doses per day) consistently matched at higher rates than total daily dose and single dose (minimum and maximum) parameters. Low levels of matching promote false alerts in CPOE systems and may compromise the utility of CDS. This study gives further evidence of the mismatch between eRules and traditional sources, which must be taken into account when CPOE systems are implemented in the pediatric setting. More evaluation of vendor-supplied eRules is required and identified deficiencies need to be addressed if this common form of CDS is intended to minimize prescribing errors and prevent adverse drug events.
Supplementary Material
Acknowledgments
The authors wish to acknowledge and thank Cecilia (Monifa) Mahdi for her data analysis assistance on this project and Pamela Schoettker for her editorial assistance on this manuscript.
Footnotes
Contributors: All of the listed authors are responsible for the reported research. We have all participated in the concept and design, analysis and interpretation of data, and drafting and revising of the manuscript. ESK and SAS contributed to the acquisition of the data. All authors contributed to the analysis and interpretation of the data and drafted the manuscript. We all approve the manuscript as submitted.
Competing interests: None.
Provenance and peer review: Not commissioned; externally peer reviewed.
Data sharing statement: All data from this research project is contained within the manuscript, except the proprietary third party drug dosing database rules.
References
- 1.van der Sijs H, Aarts J, van Gelder T, et al. Turning off frequently overridden drug alerts: limited opportunities for doing it safely. J Am Med Inform Assoc 2008;15:439–48 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.van der Sijs H, Aarts J, Vulto A, et al. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc USA 2006:138–47 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Folli HL, Poole RL, Benitz WE, et al. Medication error prevention by clinical pharmacists in two children's hospitals. Pediatrics 1987;79:718–22 [PubMed] [Google Scholar]
- 4.Raschke RA, Gollihare B, Wunderlich TA, et al. A computer alert system to prevent injury from adverse drug events: development and evaluation in a community teaching hospital. JAMA 1998;280:1317–20 [DOI] [PubMed] [Google Scholar]
- 5.Kaushal R, Bates DW, Landrigan C, et al. Medication errors and adverse drug events in pediatric inpatients. JAMA USA 2001:2114–20 [DOI] [PubMed] [Google Scholar]
- 6.Kaushal R, Goldmann DA, Keohane CA, et al. Medication errors in paediatric outpatients. Qual Saf Health Care England 2010:e30. [DOI] [PubMed] [Google Scholar]
- 7.Kaushal R, Goldmann DA, Keohane CA, et al. Adverse drug events in pediatric outpatients. Ambul Pediatr USA 2007:383–9 [DOI] [PubMed] [Google Scholar]
- 8.Kaushal R, Jaggi T, Walsh K, et al. Pediatric medication errors: what do we know? What gaps remain? Ambul Pediatr USA 2004:73–81 [DOI] [PubMed] [Google Scholar]
- 9.Zandieh SO, Goldmann DA, Keohane CA, et al. Risk factors in preventable adverse drug events in pediatric outpatients. J Pediatr USA 2008:225–31 [DOI] [PubMed] [Google Scholar]
- 10.Bates DW, Cullen DJ, Laird N, et al. Incidence of adverse drug events and potential adverse drug events. Implications for prevention. ADE Prevention Study Group. JAMA 1995;274:29–34 [PubMed] [Google Scholar]
- 11.Phillips J, Beam S, Brinker A, et al. Retrospective analysis of mortalities associated with medication errors. Am J Health Syst Pharm 2001;58:1835–41 [DOI] [PubMed] [Google Scholar]
- 12.Baiardi P, Ceci A, Felisi M, et al. In-label and off-label use of respiratory drugs in the Italian paediatric population. Acta Paediatr 2010;99:544–9 [DOI] [PubMed] [Google Scholar]
- 13.Choonara I, Conroy S. Unlicensed and off-label drug use in children: implications for safety. Drug Saf 2002;25:1–5 [DOI] [PubMed] [Google Scholar]
- 14.Eiland LS, Knight P. Evaluating the off-label use of medications in children. Am J Health Syst Pharm 2006;63:1062–5 [DOI] [PubMed] [Google Scholar]
- 15.Kumar P, Walker JK, Hurt KM, et al. Medication use in the neonatal intensive care unit: current patterns and off-label use of parenteral medications. J Pediatr 2008;152:412–15 [DOI] [PubMed] [Google Scholar]
- 16.McPhillips H, Stille C, Smith D, et al. Methodological challenges in describing medication dosing errors in children. In: Henriksen K, et al. eds. Advances in patient safety: from research to implementation (volume 2: concepts and methodology). Rockville, MD, 2005 [Google Scholar]
- 17.King WJ, Paice N, Rangrej J, et al. The effect of computerized physician order entry on medication errors and adverse drug events in pediatric inpatients. Pediatrics 2003;112(3 Pt 1):506–9 [DOI] [PubMed] [Google Scholar]
- 18.Schedlbauer A, Prasad V, Mulvaney C, et al. What evidence supports the use of computerized alerts and prompts to improve clinicians’ prescribing behavior? J Am Med Inform Assoc 2009;16:531–8 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Bates DW, Leape LL, Cullen DJ, et al. Effect of computerized physician order entry and a team intervention on prevention of serious medication errors. JAMA 1998;280:1311–16 [DOI] [PubMed] [Google Scholar]
- 20.Bates DW, Teich JM, Lee J, et al. The impact of computerized physician order entry on medication error prevention. J Am Med Inform Assoc 1999;6:313–21 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Wilson AL, Hill JJ, Wilson RG, et al. Computerized medication administration records decrease medication occurrences. Pharm Pract Manag Q 1997;17:17–29 [PubMed] [Google Scholar]
- 22.Kaushal R, Barker KN, Bates DW. How can information technology improve patient safety and reduce medication errors in children's health care? Arch Pediatr Adolesc Med USA 2001:1002–7 [DOI] [PubMed] [Google Scholar]
- 23.Potts AL, Barr FE, Gregory DF, et al. Computerized physician order entry and medication errors in a pediatric critical care unit. Pediatrics 2004;113(1 Pt 1):59–63 [DOI] [PubMed] [Google Scholar]
- 24.Tschudy MM, Arcara KM, Johns Hopkins Hospital. Children's Medical and Surgical Center The Harriet Lane handbook: a manual for pediatric house officers. 19th ed. Philadelphia, PA: Elsevier Mosby, 2012 [Google Scholar]
- 25.PDR.net. Secondary PDR.net. http://www.pdr.net/
- 26.Epocrates Online. Secondary Epocrates Online. http://www.online.epocrates.com.
- 27.Micromedex 2.0. Secondary Micromedex 2.0. http://www.thomsonhc.com/micromedex2/librarian/
- 28.Lexi-Comp Online. Secondary Lexi-Comp Online. http://www.lexi.com/institutions/products/online/
- 29.Koren G, Barzilay Z, Greenwald M. Tenfold errors in administration of drug doses: a neglected iatrogenic disease in pediatrics. Pediatrics 1986;77:848–9 [PubMed] [Google Scholar]
- 30.Perlstein PH, Callison C, White M, et al. Errors in drug computations during newborn intensive care. Am J Dis Child 1979;133:376–9 [DOI] [PubMed] [Google Scholar]
- 31.Rieder MJ, Goldstein D, Zinman H, et al. Tenfold errors in drug dosage. CMAJ 1988;139:12–13 [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.