Designing and evaluating an automated system for real-time medication administration error detection in a neonatal intensive care unit

Abstract

Background

Timely identification of medication administration errors (MAEs) promises great benefits for mitigating medication errors and associated harm. Despite previous efforts utilizing computerized methods to monitor medication errors, sustaining effective and accurate detection of MAEs remains challenging. In this study, we developed a real-time MAE detection system and evaluated its performance prior to system integration into institutional workflows.

Methods

Our prospective observational study included automated MAE detection of 10 high-risk medications and fluids for patients admitted to the neonatal intensive care unit at Cincinnati Children’s Hospital Medical Center during a 4-month period. The automated system extracted real-time medication use information from the institutional electronic health records and identified MAEs using logic-based rules and natural language processing techniques. The MAE summary was delivered via a real-time messaging platform to promote reduction of patient exposure to potential harm. System performance was validated using a physician-generated gold standard of MAE events, and results were compared with those of current practice (incident reporting and trigger tools).

Results

Physicians identified 116 MAEs from 10 104 medication administrations during the study period. Compared to current practice, the sensitivity with automated MAE detection was improved significantly from 4.3% to 85.3% (P = .009), with a positive predictive value of 78.0%. Furthermore, the system showed potential to reduce patient exposure to harm, from 256 min to 35 min (P < .001).

Conclusions

The automated system demonstrated improved capacity for identifying MAEs while guarding against alert fatigue. It also showed promise for reducing patient exposure to potential harm following MAE events.

BACKGROUND AND SIGNIFICANCE

A medication error is a mistake in a treatment process that leads to, or has the potential to lead to, harm to a patient.¹ Medication errors can occur in one or multiple phases of the medication process: ordering, transcription, dispensing, administration, and monitoring.² Medication administration errors (MAEs), which are discrepancies that occur between the medication intended by a prescriber and the medication received by a patient, can directly harm patients. Despite a long-term effort to reduce medication errors by implementing state-of-the-art health information technologies such as electronic health records (EHRs), computerized physician order entry, smart infusion pumps, and barcode medication administration (BCMA) systems,^3–8 MAEs remain common in health care settings. As indicated by a recent systematic review of 91 studies, approximately 20% of hospital errors were MAEs, a significant proportion of which were associated with harmful effects such as adverse drug events.^9,10

Timely identification of MAEs and early mitigation promise great benefits for reducing the risk of patient exposure to potential harm. Several approaches have been developed to detect medication errors, including MAEs. These approaches either: (1) gather medication error information via reported incidents from institutional databases,^11,12 (2) use trigger tools to identify medication-related problems that have caused patient harm,^13–18 or (3) develop computerized algorithms to monitor the medication process and alert clinicians to potential medication errors.^{4–6,19–32} Although incident reporting and trigger tools are the most frequently implemented approaches for medication error detection, they are suboptimal for detecting all error types and events.^33,34 Incident reporting only captures a small fraction of errors, due to clinician reporting attitudes and behaviors.^35,36 Trigger tools have demonstrated improved sensitivity in identifying errors compared to incident reporting, but implementing them requires manual chart review and is therefore resource-intensive.¹⁶ In addition, the triggers usually have low positive predictive values (PPVs), which could cause alert fatigue, affecting clinicians’ responsiveness.^37,38 Incident reporting and trigger tools rely on the clinician’s awareness of specific symptoms, abnormal laboratory test results, or antidotes to medications, which could preclude timely mitigation of patient exposure to potential harm.³³

Taking advantage of recent advances in health information technologies, a variety of computerized algorithms have been developed to monitor the medication process and prevent medication errors.^{4–6,19–32} A significant effort has been made to reduce errors during the ordering phase using overdose alerts.^{19–21,28–32} Most of the alerts are implemented with a static formula based on a combination of ordered dose and patient variables (eg, weight).³⁹ The simple algorithms could not capture MAEs that require dynamic reconciliation between administrations, orders, and adjustments from clinician communication. By utilizing barcoding technology, BCMA systems verify details of in-hand medications (ie, correct patient, drug, dose, route, and scheduled time) with medication orders before administration.^4–6 The literature suggests a beneficial role for BCMA in reducing medication errors such as wrong patient and wrong dose errors.^40,41 Nevertheless, BCMA systems cannot eliminate all MAEs, particularly those that occur with continuous intravenous medications, where frequent dose adjustments based on laboratory results and patient status occur and do not involve barcode scanning.^41–43 Recent studies also report high rates of false positive alerts using a BCMA system, which could contribute to alert fatigue and increase workarounds during medication administrations.^44,45 Although BCMA has been widely adopted and a small number of dose alert systems have been integrated into clinical environments to facilitate real-time error prevention,^{4–6,19,21,28,29} few have measured the impact on harm mitigation subsequent to medication safety events.^8,46 As described in our earlier research,^25,26 rather than using static formulas and barcoding technology, we developed a set of computerized algorithms to analyze dynamic EHR content to identify MAEs. However, these technologies were tested on retrospective data to identify errors, preventing assessment of their impact on actual patient harm.

OBJECTIVE

To address the barriers and knowledge gaps pertaining to MAE detection, we augmented the algorithms from our earlier studies with advanced information technologies and developed a real-time MAE detection system.^25–27 The algorithms were refined for real-time assessment that could be integrated into clinical environments and used to prevent ongoing errors. This study sought to prospectively evaluate the system prior to its integration into institutional workflows. Our specific aims were: (1) to develop an automated system that utilizes comprehensive EHR information to detect dosing-related MAEs in real time, (2) to prospectively evaluate the system performance in an urban level 4 neonatal intensive care unit (NICU) prior to clinical integration, and (3) to estimate the system’s potential to mitigate MAE harm for neonatal patients. The study is the first known investigation of a real-time MAE detection system on mitigation of medication safety events.⁸ Our long-term objective is to develop an automated system that will achieve a more effective and generalizable framework for mitigating MAEs in health care institutions.

DATA AND METHODS

Setting and study population

The NICU is a complex, often chaotic environment, with frequent medication administrations and adjustments for critically ill patients.^47,48 In particular, NICU patients are more likely to be exposed to potential harm after MAE events.⁴⁹ For these reasons, we focused on automating MAE detection for neonatal patients admitted to the NICU at Cincinnati Children’s Hospital Medical Center (CCHMC). The study period was between January 1, 2017, and April 30, 2017, representing a total of 3462 patient days. Approval for this study was given by the CCHMC Institutional Review Board (study ID: 2013-4241), and a waiver of consent was authorized.

CCHMC houses a level 4 NICU that provides the highest level of neonatal intensive care for complex and critically ill newborns. The institution utilizes a fully computerized commercial EHR system (Epic Systems Corporation, Verona, WI, USA). Additional NICU safety interventions in place include the use of computerized provider order entry with embedded clinical decision support, a BCMA system, smart infusion pump technology with a customized neonatal library of medications, daily prescription review by dedicated NICU pharmacists, and clinical guidelines for high-risk medications.

We focused on reconciling 10 high-risk continuous intravenous infusions and medications prescribed to NICU inpatients: total parenteral nutrition (TPN), lipids, intravenous fluids (IVF), insulin, morphine, fentanyl, milrinone, vasopressin, dopamine, and epinephrine. Continuous intravenous infusion has a higher risk and severity of error than other medication administrations.^50,51 In particular, its administrations usually span multiple nursing shifts and involve complex dosage adjustments that are not captured by in-place interventions such as BCMA.

We planned to detect dosing-related errors, which are the most common MAEs in the NICU environment.^47,52,53 A dosing-related MAE was defined as any discrepancy between the medication dose or infusion rate administered to a patient and the dose/rate prescribed by physicians during patient care.

Study design

We performed a prospective observational study of an automated system designed to detect MAEs in the NICU and notify providers of MAE events in real time. The system consisted of 4 custom-developed software modules: detector, notifier, reporter, and learner. Figure 1 diagrams the overall processes, and the details of each process are provided below.

Figure 1.

The overall processes of the study.

Extraction of medication use information

The system first extracted medication use information in real time with a set of EHR-based application programming interfaces (step 1). The information included: (1) medication orders that documented medication doses (or infusion rates) prescribed to patients, (2) structured order modifications that adjusted the original doses/rates via computerized physician order entry, (3) medication administration records (MARs) that documented actual doses/rates administered to patients, and (4) free-text orders communicated from physicians to nurses that delivered complex dose/rate adjustments during patient care. The free-text communications were parsed with a set of regular expression-based natural language processing (NLP) algorithms to identify discrete dose/rate changes.^25,26

MAE detection

Given the extracted information, the detector module identified discrepant doses/rates between MARs and other data sources using a set of logic-based rules (step 2). The detector was built upon our earlier research on MAE detection, where the logic-based rules were abstracted from standard care practices, refined by neonatologists, and implemented by programmers.^25–27Figure 2 illustrates an example of the chronological ordering of extracted EHR data and an MAE identified by the detector. By analyzing the dynamic EHR information, the detector determined the latest dose/rate prescribed to a patient and matched it with an MAR dose/rate. If a discrepancy was identified, the module would trigger an MAE event with a summary and suggestion. For the purposes of our study, medication administrations based on verbal orders were considered errors if appropriate electronic orders were not placed within 30 min of medication changes. The 30-min time frame was deemed to be acceptable for clinicians to enter an order in the EHR to cover verbal adjustment. To accommodate the delayed electronic orders, we delayed the algorithm processing on each medication administration for 30 min.

Figure 2.

An example of chronological ordering of the medication use data and an error identified by the system.

Real-time notification and error prevention

The notifier module was implemented with the Skype for Business messaging platform (Microsoft Corporation, Redmond, WA, USA) to deliver MAE information (step 3).⁵⁴Figure 3 illustrates an example notification that was sent to a clinician investigator’s working iPhone in real time. The platform and device were chosen because they were widely available and familiar to the majority of clinicians. The institution developed adequate Health Insurance Portability and Accountability Act compliance programs, including a Business Associate Agreement with Microsoft, to assure the delivery of patients’ protected health information using Skype for Business. The mobile devices were under the institutional mobile device management procedures and policies that implemented industry standard best practices.

Figure 3.

An example of real-time notification sent by the system. The arrows highlight the MAE event, the latest medication order and adjustment details, and the corresponding medication administration shown in Figure 2.

The ultimate goal for the system is that clinicians, after investigating the message, will determine whether the event is a true MAE. If so, they will take appropriate action immediately (eg, communicate with bedside nurses and correct orders; step 4). They will also send decisions (eg, if an event is a clinical error, documentation issue, or false positive alert) to the reporter module (step 5), which will generate case reports for manual inspection and system improvement (step 6). In this preclinical study, notifications were sent to study investigators directly for evaluation. There was no direct interaction between study investigators and bedside nurses.

Periodic manual review and system improvement

Using the case reports, study investigators periodically inspected detected MAE events during which patients were potentially at risk for harm (step 7). Their feedback was fed into the learner module to adjust system parameters (eg, revising regular expressions to improve dose detection from free-text narratives; step 8).

Gold standard MAE events

During the study period, 2 board-certified pediatric physicians on the research team (including one neonatologist) chart reviewed the use of all 10 targeted medications/infusions to identify MAEs. Differences between the physicians’ decisions were resolved during adjudication sessions. Interrater reliability was calculated using Cohen’s κ to define the agreement.⁵⁵ After consensus sessions, the physicians analyzed and classified each MAE event into 1 of 2 categories: (1) documentation issue, in which the correct dose was very likely given but the clinicians did not follow standards of practice in documenting the medication use (eg, placing an electronic order after a prolonged period of time following the corresponding verbal order), or (2) clinical error, in which a wrong dose was actually administered. For each MAE event, the neonatologist also categorized the associated harm using the National Coordinating Council (NCC) for Medication Error Reporting and Prevention Index.⁵⁶ If the errors represented documentation issues, they were categorized as category A for their potential to cause harm. All other errors were considered to be at least category C, given that all medications reached patients. If medication doses administered were 2 times greater or lower than the prescribed doses, they were considered category D errors that required increased monitoring to assess for patient harm. The MAE events adjudicated by the physicians, along with their harm categorization, served as a gold standard set to evaluate system performance.

Baseline: standards of practice

The institution utilizes 2 approaches to monitor patient safety events: incident reporting and a trigger tool. Incident reports were collected through voluntary reporting using Risk MonitorPro (RL Solutions, Cambridge, MA, USA).⁵⁷ The reports were submitted by employees using an intranet link or directly through the EHR user interface. Each report described incident type, incident date, patient name and medical record number, clinical unit, contributing factors, immediate actions taken, harm assessment, and a brief description of the event. The trigger tool has been a stable program at CCHMC for >10 years.¹⁸ The patient charts containing any triggers of specific events in the trigger catalog (eg, naloxone administration for opioid overdose) were investigated manually to identify errors. In this study we used all MAE events documented in NICU-specific incident reports and the trigger tool evaluations as a baseline (denoted by BASELINE) to simulate the standards of practice used by the institution.

Experiments

Evaluation setup

After the development phase, the MAE detection system was migrated to a production environment. As part of our preclinical testing process, we performed an observational study without the clinically integrated intervention (ie, without step 4, intervention, in Figure 1). We also suspended the system improvement process (step 8) so that no manual customization was made to overfit the evaluation. The objective was to evaluate the system comprehensively before it was deployed in clinical practice. We compared system notifications against the gold standard MAE events to assess system performance. The primary outcome was to demonstrate that using the automated system would detect dosing-related MAEs more accurately and efficiently compared to the current standards of practice (BASELINE).

Evaluation metrics

We adopted 4 customary evaluation metrics to assess system performance: PPV, sensitivity (SEN), negative predictive value (NPV), and specificity (SPEC).^58,59 The metrics were calculated in aggregate and for each medication. We also evaluated expected mitigation of exposure to potential harm following a clinical MAE event with the automated system. In the current standard of care, the time of exposure is calculated between the MAR time of erroneous dose/rate administration and time of a documented clinician correction. In regard to the automated system, it was calculated between the MAR time and the time when an MAE notification occurred, assuming a clinician could respond immediately upon receiving an MAE message. An example calculation of time for exposure is presented in Figure 4. If an MAE was missed by the system (false negative), the time of exposure was identical to that without the system.

Figure 4.

An example calculation of exposure time with and without automated MAE detection. The calculation is based on the MAE event shown in Figures 2 and 3.

RESULTS

Descriptive statistics of the dataset

Table 1 presents descriptive statistics of the medication use data. The physicians reviewed 10 104 MARs for 5971 medication orders during the study period and identified 116 MAEs (1.15% MAE rate). The overall interrater reliability was 82.9%, indicating good agreement on the MAE decision. Among the targeted medications/infusions, epinephrine had the highest MAE rate, followed by TPN, IVF, morphine, and lipid. Five medications had no associated MAEs. The frequency of dose adjustments varied between medications/infusions during patient care. In particular, most adjustments for TPN, lipid, and IVF were delivered via free-text communication from physician to nurse. We observed a moderate positive correlation between error rate and number of dose adjustments (Pearson’s correlation = 0.65).⁶⁰

Table 1.

Descriptive statistics of the medication use data

Medication/infusion	No. of patients	No. of encounters	No. of orders	No. of MARs	No. of MAEs	MAE rate (%)	No. of dose adjustments per MAR, n (%)
Epinephrine	21	21	47	296	9	3.04	0.68 (13)
TPN	112	114	2543	3904	79	2.02	0.89 (100)
IVF	209	215	772	2140	23	1.07	0.91 (79)
Morphine	51	51	153	870	2	0.23	0.21 (0)
Lipid	112	114	2422	2723	3	0.11	0.02 (96)
Vasopressin	4	4	11	68	0	0.00	0.60 (5)
Milrinone	8	8	8	57	0	0.00	0.00 (0)
Insulin	3	3	5	7	0	0.00	0.43 (0)
Dopamine	2	2	4	9	0	0.00	0.33 (0)
Fentanyl	3	3	3	30	0	0.00	0.30 (0)
Total	213	219	5971	10 104	116	1.15

The number in parentheses represents the percentage of dose adjustments delivered via free-text clinician communication.

Performance of MAE detection

Table 2 shows the system performance in aggregate and for each medication. During the study period, the BASELINE system identified 111 medication safety events for NICU patients, 34 from incident reporting and 77 from the trigger tool. Only 5 events were related to dosing errors of the targeted medications, 4 for TPN and 1 for morphine. The overall performance of the BASELINE was 100.0%/4.3%/98.9%/100.0% (PPV/SEN/NPV/SPEC).

Table 2.

Performance of the BASELINE and the automated medication administration error detection system

	System performance (%)
Medication/infusion algorithm	BASELINE				Automated MAE detection
	PPV	SEN	NPV	SPEC	PPV	SEN	NPV	SPEC
Epinephrine	100.0	0.0	97.0	100.0	87.5	77.8	99.3	99.7
TPN	100.0	5.1	98.1	100.0	76.8	83.5	99.7	99.5
IVF	100.0	0.0	98.9	100.0	76.7	100.0	100.0	99.7
Morphine	100.0	50.0	99.9	100.0	100.0	50.0	99.9	100.0
Lipid	100.0	0.0	99.9	100.0	100.0	66.7	100.0	100.0
Vasopressin	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0
Milrinone	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0
Insulin	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0
Dopamine	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0
Fentanyl	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0
Total	100.0	4.3	98.9	100.0	78.0	85.3	99.8	99.7

Bold numbers indicate the best results.

The automated MAE detection system achieved an overall SEN of 85.3% and PPV of 78.0%. The SEN was >75% across all medications/infusions except lipid and morphine, where one lipid and one morphine MAE each was missed. The improvements of SEN over the BASELINE were statistically significant (P = 0.009 with paired t-test). The system achieved 100% PPV for the majority of the medications/infusions and >75% for those with frequent dose adjustments (epinephrine, TPN, and IVF). However, there was a very high negative correlation between PPV and number of free-text dose adjustments (Pearson’s correlation = −0.94).

Harm categorization and estimated mitigation

Table 3 presents harm categories of the gold standard MAEs and the associated causes. Approximately 72% of the MAEs were clinical errors that reached patients (categories C and D). According to physician chart review, none resulted in detectable clinical action. However, 15 MAEs (13%) involved substantial overdose or underdose, potentially necessitating monitoring for harm (category D). The automated MAE detection system identified 85% of category A, 84% of category C, and 100% of category D errors.

Table 3.

Medication administration error harm categorization comparing physician and automated MAE detection counts

	NCC Medication Error Index and causes
Medication/infusion	Category A	Category C		Category D
	Documentation issue	Overdose	Underdose	Substantial overdose	Substantial underdose
Epinephrine	7 (5)	0	0	2 (2)	0
TPN	19 (16)	22 (18)	32 (27)	6 (6)	0
IVF	7 (7)	3 (3)	6 (6)	3 (3)	4 (4)
Morphine	0	0	2 (1)	0	0
Lipid	0	1 (1)	2 (1)	0	0
Total	33 (28)	26 (22)	42 (35)	11 (11)	4 (4)

NCC Medication Error Index: Category A: circumstances or events that have the capacity to cause error; category C: an error occurred that reached the patient but did not cause patient harm; category D: an error occurred that reached the patient and required monitoring to confirm that it resulted in no patient harm.

The numbers outside the parentheses represent errors detected through physician review, and the numbers in parentheses represent errors captured by the automated MAE detection system.

Substantial overdose: the administered dose was 2 times great than the prescribed dose; substantial underdose: the administered dose was 2 times lower than prescribed dose.

Figure 5 illustrates the time of exposure to potential harm with and without automated MAE detection. The automated system could have potentially reduced time of exposure by 40% for IVF and >50% for the other medications/infusions. In aggregate, it could have potentially reduced the median exposure time from 256 to 35 min (P < .001 with paired t-test).

Figure 5.

Median time window for exposure to harm with and without automated MAE detection.

Classification of system errors

To identify challenges with detecting MAEs, we performed error analysis for the automated system by comparing all MAEs identified by the system to the gold standard evaluation. The system made 45 errors (28 false positives and 17 false negatives), which were grouped into 6 categories, shown in Table 4.

Table 4.

False positive/negative errors made by the automated MAE detection system

Error sources	Causes of errors identified by the chart review	Error
		FP	FN
Logic rules	The system matched an administered dose with one data source (eg, order dose or dose adjustment from clinician communication), while physicians considered it an error because it did not match dose adjustments from other sources that were filed more recently.	1	14
	Multiple dose adjustments were filed in a very short time window, but the system only reconciled with the one closest to the administration time.	4	1
EHR information	The system relied on enteral feeding rates documented by clinicians, which caused errors when the rates were not updated.	12	1
NLP component	The NLP component captured wrong information in free-text communication (eg, considered “7 mL” a dose adjustment in “Please check a bladder pressure with 7 mL’s of normal saline”).	5	0
	The system missed temporal information in free-text communication (eg, “please run new TPN at 7.9 mL/h” implied that the adjustment was for future administrations rather than the current one).	4	0
	The NLP component missed dose information in free-text communication (eg, missed “8 mL/h” from the clinician communication “IV + NG = 8 mL/h”).	2	1

FP: false positive; FN: false negative.

DISCUSSION

Although used in current practice, incident reporting and the trigger tool captured only 4.3% of MAE events in the gold standard evaluation. These technologies are therefore suboptimal for detecting medication errors related to doses administered, which is consistent with the findings of others in the literature.^33,34 Compared with the BASELINE, the automated MAE detection system showed good capacity for identifying MAEs and achieved statistically significant improvement in sensitivity. One strength of the system over existing technologies such as BCMA is its ability to analyze the large number of dose adjustments made during dynamic medication processes (eg, the frequent changes in TPN and IVF). The frequent adjustments represent times when clinicians are more likely to make errors, as evidenced by the positive correlation between dose adjustment and MAE rate. The 78.0% PPV achieved by the system suggests that for every 10 error notifications, 2 were false positive alarms. Compared to recent dose alert systems that reported <18% PPV in real-time settings,^28,29 the signal-to-noise ratio in our system holds promise for guarding against alert fatigue. Indeed, the system triggered approximately one error notification per day for all medications in aggregate during the study period, suggesting a minimal increase in staff workload for a potentially large safety benefit.

The automated system detected 86.7% of clinical errors that reached patients (Table 3). Importantly, it captured all rare but substantial dosing errors, for which early recognition is most critical. By leveraging real-time messaging technology, the system has the potential to reduce harm exposure significantly for all medications, and the most substantial reductions were realized for long-time intravenous medications/infusions such as TPN and lipid.

Error analysis, challenges, and future work

The error analysis uncovered several limitations of the automated system (Table 4). Most of the false negative errors were due to the lack of override rules between different data sources (ie, orders, structured order modifications, and clinician communications; category 1 in Table 4). If such override rules were implemented, meaning that a dose administered would be preferentially matched with the most recent dose adjustment from any source, the system could trigger a lot of false positive alarms, because an adjustment (eg, from clinician communication) could be placed in advance for future administrations. On the other hand, the system only implemented override rules within the same data source to reduce false positives, but it missed a number of real errors. In the next development phase, we will allow overriding between data sources but will also implement more granular rules to detect dose adjustments for future administrations. In some events, multiple dose adjustments were filed in a short time window, which confused the system and caused false positive alarms (category 2). To improve the robustness of the system, additional rules will be implemented to reconcile all information within a time window around an administration.

Another set of errors (29%) was caused by missing real-time feeding rates in the collected EHR information (category 3). Physicians usually specified total infusion rates that combined TPN (or IVF) with enteral feeding rates. This allowed a total fluid rate to be delivered as feedings were advanced. Because the current system relies on feeding rates documented by clinicians, it triggered several false positive alarms when the feeds were titrated over a long period of time without updated feeding information. To mitigate this documentation issue, project planning is in progress to integrate real-time feeding information directly from smart infusion pumps.

Finally, approximately 27% of the errors were caused by the system’s NLP component (categories 4–6). Although the component was equipped with a large set of regular expressions that have been validated in our earlier studies,^25,26 it failed to identify correct information when the free-text communication contained similar medications (category 4) or temporal expressions (category 5). The errors suggest the limitation of our regular expression-based NLP, which could impact the system’s scalability at other institutions with potentially different documentation workflow and style. However, this issue could be mitigated via the system’s learner module (step 8 in Figure 1), which is designed to accommodate new NLP expressions actively during periodic system improvement.

One limitation of our study is that we assumed error correction occurred at the time of notification and we did not add additional time for the health care team to assess and intervene. Consequently, Figure 5 provides a best-case scenario of the improvement our intervention could achieve in mitigating harm exposure. Because the study did not involve direct interaction with bedside nurses (step 4 in Figure 1), whether the harm would reach a patient at the time of notification or whether the health care team being notified would act in a timely manner remain to be seen. To take the next step, we will deploy the system in clinical practice, which will allow assessment of a more realistic performance of the proposed intervention. Second, our study only reported the system performance in a specific clinical environment at a single institution. To address this limitation, we have initiated evaluations on more diverse patient populations at our institution (eg, pediatric intensive care unit) and others. As a final limitation, we allowed a 30-min time delay in our algorithms and notifications to accommodate verbal orders. This delay limits a clinician’s response time and subsequently the system’s capability in harm mitigation. However, as demonstrated in Figure 5, the current time windows of harm exposure are long (between 50 and 1100 min for different medications). As such, significant gain can still be realized despite this limitation.

CONCLUSION

In this study we designed and evaluated an automated system for real-time MAE detection. In a gold standard–based prospective evaluation in a NICU environment, the system demonstrated good capacity for identifying MAEs while guarding against alert fatigue. In particular, the system could significantly reduce patients’ exposure to potential harm following MAE events. Consequently, we hypothesize that the automated MAE detection system, once fully deployed, holds great potential to significantly mitigate medication safety events among neonatal patients.

FUNDING

This work was supported by the National Institutes of Health (grant numbers: 1R01LM012230, 1U01HG008666, 5U18DP006134, 1R21HS024983). YN and TL were also supported by internal funds from Cincinnati Children’s Hospital Medical Center.

COMPETING INTERESTS

The authors have no competing interests to declare.

Contributors

YN conceptualized the study, coordinated the data extraction, developed the automated MAE detection system, analyzed the results, created the tables and figures, and wrote the manuscript. TL provided suggestions in system development, coordinated real-time system deployment, coordinated the results analysis, and contributed to the manuscript. EH coordinated the data extraction, consulted on data quality and cleaning, provided suggestions on system development and error analysis, and contributed to the manuscript. ML coordinated the data extraction, consulted on data quality and cleaning, and contributed to the manuscript. EK and KM conceptualized the study, supervised the work, chart reviewed the errors, provided suggestions on error analysis, and contributed to the manuscript. All authors read and approved the final manuscript.