Automated Detection of Medication Administration Errors in Neonatal Intensive Care

Abstract

Objective

To improve neonatal patient safety through automated detection of medication administration errors (MAEs) in high alert medications including narcotics, vasoactive medication, intravenous fluids, parenteral nutrition, and insulin using the electronic health record (EHR); to evaluate rates of MAEs in neonatal care; and to compare the performance of computerized algorithms to traditional incident reporting for error detection.

Methods

We developed novel computerized algorithms to identify MAEs within the EHR of all neonatal patients treated in a level four neonatal intensive care unit (NICU) in 2011 and 2012. We evaluated the rates and types of MAEs identified by the automated algorithms and compared their performance to incident reporting. Performance was evaluated by physician chart review.

Results

In the combined 2011 and 2012 NICU data sets, the automated algorithms identified MAEs at the following rates: fentanyl, 0.4% (4 errors/1005 fentanyl administration records); morphine, 0.3% (11/4009); dobutamine, 0 (0/10); and milrinone, 0.3% (5/1925). We found higher MAE rates for other vasoactive medications including: dopamine, 11.6% (5/43); epinephrine, 10.0% (289/2890); and vasopressin, 12.8% (54/421). Fluid administration error rates were similar: intravenous fluids, 3.2% (273/8567); parenteral nutrition, 3.2% (649/20124); and lipid administration, 1.3% (203/15227). We also found 13 insulin administration errors with a resulting rate of 2.9% (13/456). MAE rates were higher for medications that were adjusted frequently and fluids administered concurrently. The algorithms identified many previously unidentified errors, demonstrating significantly better sensitivity (82% vs. 5%) and precision (70% vs. 50%) than incident reporting for error recognition.

Conclusions

Automated detection of medication administration errors through the EHR is feasible and performs better than currently used incident reporting systems. Automated algorithms may be useful for real-time error identification and mitigation.

Graphical Abstract

1. Introduction

The Institute of Medicine report To Err is Human raised public and practitioner awareness about the frequency of errors in medical practice, while the 2006 report Preventing Medication Errors focused attention on the frequency and consequences of medication errors.[1,2] Error identification remains a significant issue because traditionally used systems like incident reporting and trigger tools are known to detect only a fraction of medication errors.[3–7] Although trigger tools have demonstrated improved ability to identify errors compared to incident reporting, application of trigger tools remains resource-intensive, requiring manual chart review and limiting review to a subset of patients.[8,9] Furthermore, retrospective identification of errors through manual review precludes timely mitigation of harm.[10]

Medication errors are the most common medical errors experienced by patients.[11–12] Studies have shown that medication errors with the potential to cause harm occur three times more frequently in the pediatric inpatient population than the adult population.[13] Neonates are even more susceptible to medication errors due to drug dosing that is influenced by weight, gestational age and postnatal age.[14–16] The broad range of patient weights in the neonatal population (500 grams to five kilograms) amplifies the severity of dosing errors.[14, 17–21]

Because incident reporting and trigger tools are suboptimal for identifying medication errors, new methods for error identification must be developed. The electronic health record (EHR) with computerized provider order entry (CPOE) and an electronic medication administration record (eMAR) offer a means to evaluate all medication orders and administrations efficiently and rapidly. The EHR can be used to evaluate larger cohorts with less resource utilization than that required by trigger tool analyses or manual chart review, making it a rich resource for medication error identification. [22–24] The EHR also has potential for enabling real-time identification of certain error types.

The medication cycle consists of five phases: 1) Ordering, 2) Transcribing, 3) Dispensing, 4) Administering and 5) Monitoring. Multiple studies have shown that the majority of errors occur during the ordering and administering phases of the medication cycle. [25–27] A significant effort has been made to reduce errors during the ordering phase through the use of computerized provider order entry and clinical decision support with success [28–33]. We chose to focus on developing a tool for identifying and ultimately mitigating errors that occur in the other harmful phase of the medication cycle, the medication administration phase.

The overall goal of our work is to automate error detection in the neonatal intensive care unit (NICU) and reduce harm using the EHR. The specific aims of this study are to 1) develop computerized algorithms for identification of medication administration errors (MAEs) in the EHR, 2) test the capacity of the computerized algorithms to detect MAEs for specific high-alert medications in a large NICU setting, and 3) compare the frequency and type of MAEs identified by the algorithms to those reported through incident reporting and trigger tool methods during the same time period.

2. Material and Methods

2.1 Setting

Cincinnati Children’s Hospital Medical Center (CCHMC) houses a level four NICU which provides the highest level of neonatal intensive care to complex and critically ill newborns requiring extracorporeal mechanical oxygenation (ECMO), surgical, and subspecialty care. The NICU has 59 registered beds and had 738 and 734 admissions in years 2011 and 2012 respectively. Most patients are admitted as transfers from other facilities and have an average gestational age of 35 weeks and an average length of stay of 26.6 days.

NICU safety interventions in place at the beginning of the study included the use of computerized provider order entry with clinical decision support, a barcode medication administration system, smart infusion pump technology which includes a customized neonatal library of 158 medications, daily rounding and prescription review by dedicated NICU pharmacists, and clinical guidelines for high-risk medications.

2.2 Data Sources

2.2.1 Electronic Health Record Data

We collected data from the EHR and voluntary incident reporting system for all CCHMC NICU patients treated in 2011 and 2012, representing a total of 16,388 patient days in 2011 and 16,685 patient days in 2012. This study protocol was approved by the CCHMC Institutional Review Board (IRB).

An EHR (Epic Systems, Verona, WI), introduced at CCHMC in March 2007, captures clinical data including medication orders entered via a CPOE module. All study data were extracted from the EHR data warehouse including information pertaining to medication order history, medication audit trail data, electronic medication administration record (eMAR) data, laboratory results, enteral feeding orders and nursing communication orders. The number of analyzed entries is described in Table 1.

Table 1.

Description and descriptive statistics of NICU EHR data evaluated

	Data Description	Number of NICU Entries/Objects Evaluated
		2011	2012
Medication Orders	Medications ordered by physicians. Extracted data include: medical record number medication order ID medication name ordered dose strength ordered dose frequency medication start date medication end date dosing weight time stamp	38,282	37,439
Medication Order Audit	Audit data describes changes to the medication order made by the physicians, including changes in dose, or frequency and the time stamp of the change	90,874	112,898
Electronic Medication Administration Record	Electronic Medication Administration Record (eMAR) includes: administered dose strength route time of administration for ordered medications	180,595	210,231
Laboratory Results	Laboratory result provides numerical values and units for each laboratory test, and reports the time the specimen was obtained.	333,014	543,092
Nursing Communication Orders	Free-text order for physician to nurse communication defines weaning rates and parameters for fluids and medications	18,694	17,762
Enteral Feeding Orders	Free-text order defines the rate of lipids and PN during rate weaning	15,466	17,697

2.2.2 Voluntary Incident Reporting Data Set

Incident reports are collected through voluntary reporting using Risk MonitorPro® (RL Solutions, Cambridge, MA).[34] Structured reports can be submitted by any employee, either anonymously or with identification, using an intranet link provided on all hospital computers or directly through an EHR user interface. Required report fields include the incident type, incident date, patient name and medical record number, clinical unit, contributing factors, immediate actions taken, harm assessment, and a brief description of the event. Training encourages participants to report a brief description of all potential events, including errors that reach the patient and cause harm, those that reach the patient but do not cause harm, and near misses that do not reach the patient. We analyzed NICU-specific reports submitted through the voluntary reporting system in 2011 and 2012.

2.2.3 Trigger Tool Data Set

CCHMC utilizes the Children’s Hospital Association pediatric trigger tool for measuring adverse drug events. Each month, 20 random charts from patients admitted to the institution for greater than 48 hours are evaluated using manual chart review. The identification of any trigger for 15 specific events within a reviewed chart spurs a more extensive chart review for errors (Appendix A). We analyzed NICU-specific trigger tool evaluations performed in 2011 and 2012.

2.3 Definitions

Medication administration: A single eMAR record, indicating a medication dose was administered.

Medication administration error (MAE): Dose or rate recorded in the eMAR does not match the medication dose or rate ordered in the medication order or medication order audit.

MAE rate: The total number of erroneous eMAR records /total number of eMAR records for a given medication. Given that we analyzed dosing errors only, and did not assess for timing or route errors, only one MAE could be identified per eMAR record.

2.4 Algorithm Development

Algorithm rules were specified by the neonatologist based on standard care practices, reviewed by the physicians on the research team, and implemented by the programmer. We developed the algorithms in the following research workflow:

1. Following initial team discussion, the neonatologist provided algorithm specifications in a written document. The algorithms compared the most recent medication orders, order audits, enteral feeding orders and nursing communication orders with eMAR entries and generated an error prediction output (Figure 1). The unit of analysis was a single eMAR record. The output consisted of matching ordered medication doses with those recorded on the eMAR in chronologic order with subsequent identification of correct matches or errors. Ordered medication doses were extracted from both structured data that resided in fixed data fields and unstructured data found in free-text order fields. Data was extracted from the unstructured free-text comments in orders using Natural Language Processing techniques (NLP). The algorithms were designed to detect a discrepancy of 0.1 difference between the ordered and administered doses, and to detect MAEs only, with the assumption of correct orders. Given that continuous medications may be weaned based on verbal orders in the NICU, we allowed a 30-minute time discrepancy between eMAR entries and orders for all medications. This allowed providers time to enter the verbal order electronically into CPOE, which is enforced by policy at CCHMC. Due to the risk of miscommunication and error with verbal orders, orders placed outside of 30 minutes were considered errors.

2. Based on the specifications, the programmer implemented the algorithm.

3. Patient data from 2011 was used to guide algorithm development. Prototype algorithms were tested and manually evaluated for compliance with physician specifications. This process was not only an engineering quality assurance step but aided in discovery of additional error scenarios missing from the initial algorithm. If new errors were identified, the algorithm was refined. The iterative development process repeated until adequate sensitivity was achieved.

4. The final algorithm was executed on the EHR data from 2011 and 2012. In the following, we provide details of the developed algorithms.

Figure 1.

General architecture of MAE detection algorithms

2.4.1 Narcotics

To assess narcotic MAEs, the algorithm compared the ordered dose with the administered dose for continuous morphine and fentanyl. The algorithm aligned the ordered and administered dose according to the action timestamp, as shown in Figure 2. If the ordered dose was not equal to the administered dose, an error was identified.

Figure 2.

Narcotic/Vasoactive MAE Detection Algorithm

2.4.2 Vasoactive medications

To assess for vasoactive MAEs, the algorithm evaluated for agreement between medication orders and administrations for dobutamine, dopamine, epinephrine, milrinone and vasopressin. In the CCHMC NICU, vasoactive medications can be weaned by nurses based on physician orders that include weaning dose and blood pressure parameters. For example, order comments may state “Please wean epinephrine by 0.01mcg/kg/min every hour for blood pressure means > 45 mmHg”. We evaluated orders to identify weaning doses but did not assess for time consistency or blood pressure correlation. (Figure 2)

2.4.3 Intravenous fluids (IVF), Parenteral Nutrition (PN) and Lipids

To assess for IVF, PN or lipid MAEs, the algorithm compared the ordered dose with the administered dose. The ordered dose was found in structured data fields like PN/lipid orders and order audit data, and in unstructured free-text fields including order comments (in medication orders), order administration instructions (in medication orders), enteral feeding order comments and nursing communication orders. The PN/lipid MAE detection algorithm is shown in Figure 3, and IVF MAE detection algorithm is shown in Figure 4. The algorithm aligned the ordered and administered dose according to the action timestamp. If the ordered dose was not equal to the administered dose, an error was identified. The case analysis for these MAE types is detailed in Appendix B.

Figure 3.

PN/Lipid Administration Error Detection Algorithm

Figure 4.

IVF Administration Error Detection Algorithm

We characterized the rate descriptions and rate changes by four methods: 1) absolute rate, where absolute fluid rates are defined (e.g. Decrease IVF to 9.2 ml/hr), 2) rate change, where a change value is defined (e.g. Decrease PN by 1 ml/hr), 3) parenteral fluid and enteral feed 1:1 changes, where orders specify rates of enteral feeding increase and PN decrease (e.g. Decrease PN 1:1 when advancing feeds by 1 ml/hr), and 4) total combined rate of parenteral fluids and enteral feeds, where orders specify a combined rate (e.g. PN + feeds = 28 ml/hr). The patterns used for characterization were identified by NLP-based regular expression and the identification performance for each pattern is listed in Appendix C. In order to evaluate the ability of the algorithm to extract study relevant information from the unstructured data comments of enteral feeding orders or nursing communication orders, two trained chart reviewers double annotated 1100 randomly selected comments (100 for the pilot study and 1000 for the evaluation) using a plug-in, Knowtator, for the annotation task with a graphic user interface. [35] The details of the NLP techniques used for information extraction, including gold standard development, annotator training and annotations have been previously described. [36–40] We report the positive predictive value (PPV) and sensitivity of the NLP algorithms for identifying unstructured entries in Appendix C.

2.4.4 Insulin

To assess for insulin administration errors, the algorithm evaluated for agreement between insulin orders and administrations. To standardize insulin treatment, a clinical guideline for insulin administration in the NICU was introduced at CCHMC in 2010. We also assessed for adherence to guideline recommendations. To assess for appropriate initiation of insulin, the algorithm evaluated whether infants met the guideline criteria of two consecutive blood glucose measurements > 180 mg/dL measured one hour apart while receiving a delivered glucose infusion rate of < 5 mg/kg/min. The algorithm also evaluated whether insulin was initiated at the guideline-specified dose and whether guideline parameters for discontinuation were followed (Table 2). The algorithm is presented in Appendix D.

Table 2.

Insulin administration clinical guideline

Initiation: Requires two consecutive blood glucose measurements > 180 mg/dL measured one hour apart while receiving a delivered glucose infusion rate of < 5 mg/kg/min
Glucose Level Parameters	Insulin Titration
Initial Dose
Glucose > 180 mg/dL and < 250 mg/dL	Start insulin at 0.01 units/kg/hr
Glucose > 250 mg/dL and < 300 mg/dL	Start insulin at 0.02 units/kg/hr
Glucose > 300 mg/dL	Start insulin at 0.03 units/kg/hr
Discontinuation
If glucose level < 100 mg/dL	Discontinue insulin
If glucose level is decreasing > 50 mg/dL/hr and blood glucose is < 200 mg/dL	Discontinue insulin

2.5 Error Validation by Chart Review

We evaluated the algorithms in the following research workflow:

1. To generate a gold standard or reference standard for performance evaluation of the algorithms, two independent reviewers, including one neonatologist, performed manual review of 1000 randomly selected orders and their associated eMAR entries from the 2012 data set for each studied medication. This included review of all administrations, not just those with errors. The 2012 data set was used as an unbiased data set that had not been used previously for algorithm development. If less than 1000 orders existed for a given medication, all orders and eMAR entries were reviewed. The sample size of 1000 medication orders was chosen because it provides the power to detect a sensitivity of 99%. The 2-sided 95% confidence interval for this sensitivity was expected to be 97.9–100%. Reviewers indicated an error was present when eMAR administration records differed from ordered medication doses. Consistent with algorithm specifications, reviewers evaluated administration errors based on an assumption of correct order entry. Reviewers were blinded to the algorithm error assessment at the time of review.

2. Incident reports were reviewed for the studied medications. Physician experts validated the incident reporting data at the time of analysis.

3. Algorithm output and errors identified by incident reporting were then compared to the reference standard to determine performance of each method. We report the sensitivity, specificity, and PPV of the MAE algorithm performance for each medication. We report administration error rates for both 2011 and 2012 according to the finalized algorithms.

4. The neonatologist assessed and classified factors contributing to MAE rates by reviewing all errors from the 2012 data set.

3. Results

3.1 Error Rates for Automated MAE Detection Algorithms

3.1.1 Narcotic medications

The combined average administration error rates in 2011 and 2012 identified by the algorithms were 0.4% (4/1005) for fentanyl and 0.3% (11/4009) for morphine. As shown in Table 3, the overall rates of MAEs were similar in 2011 and 2012 for both medications, with respective error rates for fentanyl of 0.3% (2/593) and 0.5% (2/412) and for morphine of 0.3% (5/1971) and 0.3% (6/2038). In comparison, incident reporting identified four MAEs in fentanyl administrations and six errors in morphine administrations. The majority of MAEs identified by the algorithms for both fentanyl and morphine were administration of doses that were too high or too low compared to ordered doses, while incident reporting most frequently identified pump programming errors that resulted in the wrong dose being administered. There were unique errors identified by each method, with only two errors identified by both the algorithm and incident reporting.

Table 3.

Medication Administration Errors 2011 and 2012

Medication	MAEs			Administrations			MAE Rate			MAEs identified by Incident Reporting
	2011	2012	Overall Totals	2011	2012	Overall Totals	2011	2012	Overall Totals
Narcotic Medications
Fentanyl	2	2	4	593	412	1005	0.3%	0.5%	0.4%	4
Morphine	5	6	11	1971	2038	4009	0.3%	0.3%	0.3%	6
Vasoactive Medications
Dobutamine	0	0	0	8	2	10	0%	0%	0%	0
Dopamine	3	2	5	32	11	43	9.4%	18.2%	11.6%	0
Epinephrine	184	105	289	1647	1243	2890	11.2%	8.4%	10%	7
Milrinone	4	1	5	1216	709	1925	0.33%	0.14%	0.3%	2
Vasopressin	11	43	54	128	293	421	8.6%	14.7%	12.8%	1
Fluids
IVF	137	136	273	4270	4297	8567	3.2%	3.2%	3.2%	13
PN	302	347	649	8613	11601	20124	3.5%	3.0%	3.2%	13
Lipid	103	100	203	6319	8908	15227	1.6%	1.1%	1.3%	11
Insulin
Insulin	9	4	13	284	172	456	3.2%	2.3%	2.9%	3

IVF- intravenous fluids; PN – parenteral nutrition

3.1.2 Vasoactive medications

Evaluating combined average MAE rates in 2011 and 2012, the algorithms detected no errors in dobutamine administrations (0/10), a low error rate of 0.3% in milrinone administrations (5/1925), and higher error rates for dopamine (5/43 = 11.6%), epinephrine (289/2890 = 10.0%), and vasopressin administrations (54/421 = 12.8%), as shown in Table 3. Of interest, the algorithm identified one epinephrine administration error that resulted in a ten-fold dosing error. In comparison, incident reporting identified seven epinephrine administration errors, two milrinone errors, and one vasopressin error (Table 3). The vast majority of errors identified by the algorithms were due to titration of the medication dose without a written order, while incident reporting identified administration of erroneous high doses and the presence of clamped lines during drug administration as the most common errors.

3.1.3 IVF, PN and Lipids

Similar combined average MAE rates in 2011 and 2012 identified by the algorithms are 3.2% (273/8567) for IVF, 3.2% (649/20124) for PN, and 1.3% (203/15227) for lipid administrations (Table 3). Administration errors identified by incident reporting were again low in comparison, with 13 IVF errors, 13 PN errors, and 11 lipid errors noted (Table 3). The most common error identified by the algorithms for all three parenteral fluids was delivery of a medication rate that was lower than ordered, with higher rates than ordered being the second most common error. Incident reports usually reflected administration of higher rates than ordered, and incident reporting often identified a second fluid administered incorrectly.

As part of our analysis, we assessed the ability of the algorithm to identify fluid rates and changes in enteral feeding order comments and nurse communication orders using NLP-based regular expression. Sensitivity rates for algorithm detection of rates ranged from 83–100% and PPV was 100% for all algorithms. (Appendix C)

3.1.4 Insulin

The combined average MAE rate for continuous insulin drip administrations identified by the algorithms on the 2011 and 2012 data set was 2.9% (13/456) with one ten-fold dosing error identified. The overall rates of MAEs were similar in 2011 and 2012 with error rates of 3.2% and 2.3% respectively. Comparatively, only three insulin errors were reported in incident reporting. The most common issue identified by the algorithms was the late entry of verbal orders, with the median time being 82 minutes and the maximum time being 190 minutes, followed by the administration of doses that were higher than ordered. Incident reporting also reflected doses that were higher than ordered and one delayed administration.

As part of our analysis, we used the algorithm to assess adherence to the clinical guideline for insulin administration in 25 patients who received continuous insulin infusions in 2011. The algorithm identified that 48% (12/25) of insulin initiations accurately followed the guideline, but 52% (13/25) of insulin initiations did not follow the guideline, with equal deficiencies in failing to decrease the glucose infusion rate to < 5 mg/kg/min (24%; 6/25) and failure to initiate the correct dose (24%; 6/25). One patient did not have two consecutive glucose values > 180 mg/dL (4%; 1/25) before initiating insulin therapy. For discontinuation of insulin, the algorithm found that 68% (17/25) of insulin infusions followed the guideline, while seven infusions continued when the blood glucose level was < 200 mg/dL and was decreasing > 50 mg/dL/hr (28%; 7/25). One infusion was continued with a glucose level < 100 mg/dL (4%; 1/25).

3.2 Algorithm Performance Evaluation on PPV, Sensitivity, and Specificity

As described in the study methods, algorithm performance compared to manual review is shown in Table 4. Overall, 259 administration errors were identified upon manual review, 235 of which were detected by the automated algorithms while incident reporting only identified seven errors. Although incident reporting has a slightly higher specificity (100% vs. 98.2%), the automated algorithms have a much higher sensitivity (82.1% vs. 5.5%) and PPV (70.2% vs. 50.0%). False positive results occurred when the algorithms failed to recognize weaning orders in the free-text comments, when medications being administered at the time of admission were documented prior to orders being entered, and when multiple order audits occurred together. False negative results occurred when the algorithm accepted the original order despite intervening order audits or missed free-text comments.

Table 4.

Error detection performance of the automated algorithm and incident reporting compared to manual review of administrations for each medication in the 2012 data subset

		Manual Review	Algorithms	Incident Reporting
Narcotic Medications
Fentanyl	No. Detected Sensitivity[CI] Specificity[CI] PPV[CI]	2	2 100.0% [19.3–100] 100.0% [99.6–100] 100.0% [12.3–100]	0 0[0–80.7] 100%[99.1–100]
Morphine	No. Detected Sensitivity[CI] Specificity[CI] PPV[CI]	3	4 100% [30.5–100.0] 99.9% [99.44–99.9] 75% [20.3.7–95.8]	1 33.3% [1.2–33.9] 100% [99.8–100] 100% [16.6–100]
Vasoactive Medications
Dobutamine	No. Detected Sensitivity[CI] Specificity[CI] PPV[CI]	0	0 100% [19.3–100]	0 100% [19.3–100]
Dopamine	No. Detected Sensitivity[CI] Specificity[CI] PPV[CI]	2	2 100% [19.3–100] 88.9% [51.7–98.2] 66.7% [11.6–94.5]	0 0% [0–80.7] 100% [66.2–100]
Epinephrine	No. Detected Sensitivity[CI] Specificity[CI] PPV[CI]	116	107 92.2% [85.8–96.4] 96.2% [94.9–97.2] 71.3% [63.4–78.4]	0 0% [0–3.2] 100% [99.7–100]
Milrinone	No. Detected Sensitivity[CI] Specificity[CI] PPV[CI]	1	1 100% [16.6–100] 99.6% [98.8–99.9] 25% [4.1–79.7]	0 0% [0–83.5] 100% [99.5–100]
Vasopressin	No. Detected Sensitivity[CI] Specificity[CI] PPV[CI]	48	48 100% [92.5–100] 98.4% [95.8–99.5] 92.3% [81.4–97.8]	0 0% [0–7.5] 100% [98.1–100]
Fluids
IVF	No. Detected Sensitivity[CI] Specificity[CI] PPV[CI]	27	26 96.3% [81–99.4] 99.5% [98.8–99.8] 83.9% [66.3–94.5]	1 3.7% [0.62–100] 100% [99.6–100] 100% [16.6–100]
PN	No. Detected Sensitivity[CI] Specificity[CI] PPV[CI]	28	24 85.7% [67.3–95.9] 99.6% [99–99.9] 85.7% [67.3–95.9]	2 7.1% [1.1–23.5] 100% [99.6–100] 100% [19.3–100]
Lipids	No. Detected Sensitivity[CI] Specificity[CI] PPV[CI]	17	14 82.3% [56.5–96] 99.8% [99.3–99.9] 87.5% [61.6–98.1]	2 11.8% [1.8–36.5] 100% [99.6–100] 100% [19.3–100]
Insulin
Insulin	No. Detected Sensitivity[CI] Specificity[CI] PPV[CI]	4	4 100% [40.2–100] 100% [97.9–100] 100% [40.2–100]	1 25% [4.1–79.7] 100% [97.8–100] 100% [16.6–100]
Overall
Overall	No. Detected Sensitivity Specificity PPV	259	234 82.1% 98.2% 70.2%	7 5.5% 100% 50%

No. Detected: Number of detected errors; CI: confidence interval; PPV: Positive Predictive Value

3.3 Trigger Tool Comparison

Results from the CHA trigger tool assessment, obtained over the same 2011–2012 time period, revealed five medication errors in 33 neonatal patients, none of which were from the medication categories evaluated by the algorithms. The five adverse drug events detected using the trigger tool were related to vancomycin, prednisolone, sirolimus and anesthetic agent use. In addition, our previous work demonstrated that the algorithms identified consistent error rates with those found using trigger tools when identical triggers were evaluated, but errors were identified in a more resource efficient manner. [40]

3.4 Overall MAE Rates

Our data accounted for the majority of continuous medications administered in the NICU, although we did not have information on continuous midazolam, or rarely used continuous medications like epoprostenol, nitroprusside, or prostaglandin. Overall rates of administration errors identified by the algorithms for the evaluated continuous medications were 46 errors per 1000 patient days and 103 errors per 100 NICU admissions in 2011 and 45 errors per 1000 patient days and 102 errors per 100 NICU admissions in 2012. In summary, there were 1506 errors for 54677 evaluated medication administration records (eMARs), for an administration error rate of 2.8%.

4. Discussion

The performance achieved by our automated algorithms for medication error detection was higher than that achieved through incident reporting and trigger tool methodology. Using incident reports and chart review, Kaushal found 91 medication errors per 100 NICU admissions. [14] Using incident reporting and direct observation of a select sample of continuous medication infusions, Larsen observed 3.5 errors per 1000 medication infusions.[20] Historically, measuring rates of medication errors in the NICU using incident reporting, chart review, or direct observation has been difficult and resource intensive. Our data directly compares the rates of errors identified electronically to those identified by incident reporting, and supports other studies that have found significant underreporting of errors based on incident reporting.[9] Our data also captures a much broader sample than that evaluated by trigger tool methodology. One of the significant strengths of our study is the ability to analyze all medication administration records for a given medication automatically, rather than a small sample, as well as the ability to analyze multiple years of data in a time and resource-efficient manner. Importantly, the algorithms identified rare but critical ten-fold dosing errors of high-risk medications that would benefit from early recognition. In an era when most institutions are adopting EHRs, the use of computerized algorithms holds the potential for widespread error identification. Although the algorithms have currently been tested only in a single EHR system, ongoing work includes assessment of generalizability to other institutions and other EHR products. Further, the potential to implement the algorithms as a real-time error monitor holds promise as a means to mitigate ongoing errors in continuous fluid and medication administrations. Our future work involves testing of the algorithms prospectively and in real-time to mitigate error and to prevent harm.

Improved MAE identification using the algorithms allowed us to observe factors that contributed to higher error rates, including bedside weaning, use of verbal orders, non-standardization of written weaning orders for both vasopressors and PN/fluids, and errors associated with syringe changes. For example, medications such as dopamine, epinephrine, or vasopressin, which are often weaned at the bedside based on ordered blood pressure parameters, had significantly higher error rates than medications started at a single dose and discontinued without weaning. Verbal orders were also more frequently used for these medications, creating higher risk situations, and we found several cases where written orders were placed hours after medication changes were initiated. We also found frequent administration errors when a new medication syringe was hung, as medications that had been weaned according to weaning parameters were temporarily returned to the previously ordered dose. As a unique error, we found that PN and lipids, often administered together, were frequently interchanged, resulting in an error for both medications. On the other hand, there are other factors, such as human factors, that may have influenced error rates that we may not have been able to observe. For example, providers may give less attention to familiar medications and more attention to less familiar medications.

The primary limitation to our study is that MAE identification is dependent on documentation in the EHR. There are situations where medication administrations are not recorded in the EHR, including during surgery and resuscitations. We did not review anesthesia records or resuscitation paperwork and may have missed medication errors during these periods. Second, to the study did not assess the level of harm associated with errors. Although some of the medication administration errors detected by the algorithms may not be harm causing, our goal was to create a tool that detected dosing discrepancies of 0.1 or greater between medication order and medication administration doses. Our thinking was that to effectively avoid errors that can cause patient harm, improvements must be made on the underlying, more common and less-harmful system problems that lead to near misses, as demonstrated in non-medical industries [41]. The algorithm parameters can be customized, however, to allow for a greater tolerance of minor errors. Third, some identified errors may be a result of erroneous documentation at the time of computer entry rather than errors in drug administration. Fourth, some medication administration dose errors may not be found in the EHR, such as those resulting from errors in medication pump programming. [42] We found that the algorithms did not identify several narcotic errors identified by incident reporting involving pump programming, especially programming of the wrong patient weight or wrong medication concentration. Identification of pump errors will require novel methods with evaluation of manually downloaded or wirelessly transmitted pump information for accurate identification. Fifth, we did not assess for all types of administration errors and did not attempt to identify delays in administrations or incorrect routes of administration. We also did not assess for the presence of all order specifics, such as blood pressure correlation for vasoactive medication weaning or the attainment of specific feeding volumes prior to PN weaning. Sixth, due to the use of non-standardized orders and the use of free-text comments for specifying rate changes of fluid and PN, the extraction of unstructured data was more difficult, and some errors may have been missed based on the lower sensitivity for detection. Most of these limitations would result in higher rates of MAE than we have reported. Finally, some discrepancies between ordered and administered medications may be due to errors in ordering; our methodology assumed that the order was correct and we did not attempt to look for incorrect dosing orders.

5. Conclusions

MAEs are common, but error reporting mechanisms such as incident reporting are insufficient to identify errors comprehensively. The EHR is a rich but underutilized source for MAE detection. Overall, our study demonstrates that automated detection of MAEs through the EHR is feasible and performs better, with higher sensitivity and precision, than currently used error detection systems such as incident reporting. Improved error identification allowed us to observe factors in the processes of medication ordering and administration that contribute to errors and are likely to be rectified through improvement work. Accurate error identification in continuously administered medications makes intervention possible, and future work will focus on real-time detection of ameliorable medication errors with physician and nurse notification. By systematically detecting and intercepting these errors, we will shift neonatal patient safety from passive reporting of errors to proactive identification and mitigation of unsafe care.

Highlights.

• We developed computerized algorithms to detect medication administration errors.

• We defined administration error rates for high-risk medications using the algorithms.

• Compared to incident reporting, the algorithms had better sensitivity and precision.

• Automated algorithms support real-time error identification and mitigation.