Impact of Integrating a Neonatal Early-Onset Sepsis Risk Calculator into the Electronic Health Record

Introduction:

Investigators from Kaiser Permanente developed a risk-assessment calculator as a tool for evaluation of early-onset sepsis (EOS) to narrow antibiotic use for the treatment of EOS. The integration of the EOS risk calculator into an electronic health record will minimize manual calculations and data entry and improve compliance and accuracy through automation.

Methods:

We performed a retrospective chart review for neonates ≥34 weeks and 0 days gestational age. We collected data pre-integration and post-integration of the EOS risk calculator. The primary outcome measure is the accuracy of user input into the calculator. Secondary outcomes include compliance with using the EOS risk calculator, impact on clinical recommendation when incorrectly calculated, assessment of antibiotic utilization rate (AUR), and comparison of EOS risk calculator recommendations with Centers for Disease Control and American Academy of Pediatrics recommendations.

Results:

Miscalculations occurred in 52% of instances pre-integration and 19% of instances post-integration; P < 0.001. Compliance was 93% pre-integration and 98% post-integration; P = 0.138. Clinical recommendations were changed for 21% (13/62) of miscalculations pre-integration and 4% (1/23) of miscalculations post-integration; P = 0.099. The AUR for combined NICU and nursery patients was 47 pre-integration and 47 post-integration; P > 0.999. Six cases of culture-positive sepsis were identified, and all recommendations generated by the EOS risk calculator were in alignment with current Centers for Disease Control/American Academy of Pediatrics treatment guidelines.

Conclusions:

Integration of the EOS risk calculator into the electronic health record significantly increased calculator accuracy, although it did not show statistically significant differences with regards to compliance, clinical recommendations, or AUR.

INTRODUCTION

Neonatal early-onset sepsis (EOS) is defined as the onset of sepsis within the first 72 hours of life.¹ Group B Streptococci (GBS) or Escherichia coli that colonize the maternal gastrointestinal or genitourinary tract are the most common pathogens responsible for the development of EOS.² Until recently, guidelines published by the Centers for Disease Control and Prevention (CDC) and the American Academy of Pediatrics (AAP), have been the gold standard resource utilized by clinicians for the management of EOS.^1,3 These guidelines recommend empiric antibiotic initiation for all neonates who appear clinically ill at birth or who are born to a woman diagnosed with chorioamnionitis.

Consequences of early antibiotic exposure within the neonatal population include the development of necrotizing enterocolitis and invasive candidiasis as well as alterations in the microbiome, illustrating the importance of determining an objective method to risk-stratify neonates more narrowly.^4–6 A movement referred to as “Triple I,” intrauterine inflammation, infection, or both, has been one proposed method of reducing unnecessary antibiotic exposure within neonates by utilizing a more objective approach to diagnose maternal chorioamnionitis.⁷ A multivariate prediction model was developed to better assess the true risk of EOS development within the neonatal population.^8–10 This approach takes into account both maternal and neonatal objective data rather than relying exclusively on subjective findings, such as the presence or absence of neonatal clinical illness or maternal chorioamnionitis.

Studies utilizing this method have demonstrated a decrease in antibiotic exposure to uninfected neonates, blood culture draws with similar incidence rates of culture-positive sepsis, and no statistical difference in adverse events when compared with the CDC/AAP guidelines.^11–13 A previous study within our 2 healthcare centers identified a 69% reduction in recommended empiric antibiotics with the implementation of the EOS risk calculator.¹¹ In another study, implementation of the EOS risk calculator reduced empiric antibiotic use for EOS by 42%.¹³ These findings suggest that applying this risk calculator is a safe approach that can be utilized within this clinical setting to reduce the number of uninfected neonates administered empiric antibiotics.

The value of the integration of decision-making tools directly into the electronic health record (EHR) has been previously reported. A recent study integrated and then further improved integration of a decision-making tool into the EHR resulting in improved utilization of the decision-making tool and a decrease in frequency of antibiotic prescribing.¹⁴ McGinn et al successfully integrated a decision-making tool into an EHR, which resulted in improved compliance, a reduction in antibiotic utilization, and reduction in point-of-care testing.¹⁵ Another study evaluating the integration of a decision-making tool resulted in a significant decrease in unnecessary testing, with no associated increase in clinically important misdiagnoses or a decrease in physician satisfaction.¹⁶

Initial implementation of the EOS risk calculator within our healthcare system required manual data entry of patient information into the EOS risk calculator. Clinicians accessed the EOS risk calculator (https://neonatalsepsiscalculator.kaiserpermanente.org/) through a direct link incorporated within the EHR. Utilization of the EOS risk calculator resulted in increased workload, noncompliance, and miscalculations, leading to suboptimal utilization of the EOS risk calculator tool. Approximately 1 year following implementation, we integrated the EOS risk calculator directly into the EHR to automate the process and minimize the need for manual data collection and imputation.

The objective of this study is to evaluate the impact of the integration of the EOS risk calculator into the EHR. As above, many studies have validated the utility of implementation of this decision-making tool but have not compared the accuracy, compliance, the impact of imputation errors, or difference in antibiotic utilization rates (AURs) of the manual imputation of patient-specific data into the EOS risk calculator to an automated process. We hypothesize that the integration of the EOS risk calculator into the EHR will minimize manual data entry for calculation and improve compliance, accuracy, and antibiotic utilization through automation.

METHODS

Study Design

We conducted a retrospective observational review of neonates older than 34 weeks and 0 days gestational age within 2 large tertiary teaching hospitals. A total of 12 months of data were extracted with 6 months of data collected pre-EOS risk calculator integration from May 2017 to October 2017 and 6 months of data collected post-EOS risk calculator integration from December 2017 to May 2018. This study was compliant with the Guidelines for Human Experimentation from the US Department of Health and Human Services and received approval from the Institutional Review Board. All data collected were de-identified to maintain the confidentiality of all study subjects, and no informed consent was required due to the nature of the study design.

The patient population was generated through Epic and Tableau reporting systems, and we completed the chart review for each patient and the patient’s mother. Data were collected pre-integration and post-integration of the EOS risk calculator. A total of 20 patients were randomly selected and evaluated each month (10 from NICU, 10 from nursery).

To assess for the primary and secondary study objectives, we included in the neonatal data collection total admission days, gestational age, risk calculator score, clinical illness, and blood culture and susceptibilities. Maternal data collection included intrapartum temperature, duration of rupture of membranes, vaginal versus cesarean delivery, GBS status, and administration of perinatal GBS prophylaxis. Signs and symptoms of clinical illness were abstracted, and we classified patients as having a clinical illness, equivocal appearance, or well-appearing status according to documentation by medical providers and definitions described by risk calculator authors (available at https://neonatalsepsiscalculator.kaiserpermanente.org/classification.aspx). We calculated neonatal sepsis risk per 1,000 live births using a baseline incidence of 0.3 cases of sepsis per 1,000 live births, as this is closest to our healthcare system EOS rate of 0.26 cases of sepsis per 1,000 live births reported in 2015. EOS risk calculator recommendations for empiric antibiotics, blood cultures, and vital sign monitoring according to clinical status were recorded for each subject.

The primary outcome measure is the accuracy of the EOS risk calculator imputation. Secondary outcomes include compliance with the utilization of the EOS risk calculator, impact on clinical recommendation when incorrectly calculated, assessment of AUR (antibiotic days/1,000 patient admission days), and comparison of EOS risk calculator recommendations with CDC/AAP recommendations for all positive EOS cultures. Statistical analysis was completed using Fisher’s Exact Test. A resulting P value < 0.05 was considered statistically significant.

Implementation of the EOS Risk Calculator

The healthcare system’s Women and Children Clinical Performance Group assembled a Triple I and Early Onset Sepsis Workgroup in December 2015 to improve the identification of infants at risk for EOS. In May 2016, the Workgroup established the utilization of the EOS risk calculator and incorporated a link to the EOS risk calculator website into the EHR. Also, the EOS risk calculator infection screening parameters and nurse process instructions were added to the EHR newborn admission order set. A Triple I/Early Onset Sepsis Toolkit providing EOS risk calculator education was finalized in August 2016 and distributed in September 2016. Additional EOS risk calculator education was provided to Neonatal Nurse Practitioners at both Hospital A and Hospital B during September 2016. An EOS risk calculator “help card” was created and utilized by staff in Hospital B for calculator data imputation. Copies of the “help card” were placed at every computer workstation. Hospital B utilized a Nurse Clinical Educator to provide EOS risk calculator education to nursing staff within the nursery unit. EOS risk calculator utilization began in October 2016 in Hospital A, and November 2016 in Hospital B. An EOS risk calculation was to be performed following delivery on every neonate delivered older than 34 weeks and 0 days gestational age regardless of symptoms. Implementation of the EOS risk calculator required manual data collection and imputation of patient-specific data by the end-user (labor nurse or provider). Staff manually documented the EOS risk at birth score determined by the EOS risk calculator in the EHR newborn delivery summary. Providers were notified if the calculated EOS risk score was >0.65/1,000 births or if the EOS risk calculator recommended the acquisition of a blood culture.

Integration of the EOS Risk Calculator

In November 2017, the healthcare system informatics/application analyst team integrated the EOS risk calculator into the EHR. This integration provided automation and standardization to the EOS risk calculation. The Workgroup provided updated EOS risk calculator education to all staff. Hospital B continued to utilize a Nurse Clinical Educator to provide education to nursing staff within the nursery unit.

Following newborn delivery, the end-user calculates the EOS risk score using the EOS risk calculator located within the EHR newborn delivery summary, reviews the patient-specific data automatically collected for EOS risk calculation, and accepts the resulting EOS risk score into the patient EHR. The EOS risk score and patient-specific data are displayed in both the maternal and neonatal patient EHR. A smart link was developed to include the EOS risk score into progress note documentation. Nursing staff continued to notify providers if the calculated EOS risk score was >0.65/1,000 births or if the EOS risk calculator recommended the acquisition of a blood culture.

RESULTS

We identified a total of 252 patients during the 12-month study period. EOS Risk Calculator data were incorrectly input by users in 52% of instances pre-integration, and the EOS risk calculator data were either auto-populated incorrectly or adjusted by users incorrectly in 19% of instances post-integration (P < 0.001). When we characterized results further, there was not a statistically significant difference in miscalculations for Hospital A NICU (P = 0.2882). The EOS risk calculator was utilized in 93% of patients pre-integration and 98% of patients post-integration (P = 0.138). EOS risk calculator clinical recommendations were changed for 21% (13/62) of miscalculations pre-integration and 4% (1/23) of miscalculations post-integration (P = 0.099). One patient (1/13) pre-integration had a changed clinical recommendation due to the EOS risk calculator miscalculation and classification category (Table 1). The resulting miscalculation recommended a blood culture and CBC, whereas a correct calculation recommended no culture and no antibiotics with routine vitals monitoring. The remaining 12 patients had no change in clinical recommendation, despite a miscalculation. The most commonly associated reasons for score miscalculations pre-integration and post-integration included incorrect evaluation of highest maternal temperature (35%), duration of rupture of membranes (31%), timing/type of intrapartum antibiotics (15%), and maternal GBS status (14%) (Table 2).

Table 1.

Impact of Integration of Neonatal EOS Risk Calculator

Table 2.

Reasons for EOS Risk Calculator Score Miscalculations

We calculated the AUR from an identified 10,558 patient admission days with a total of 498 days of antibiotic therapy pre-integration and 10,119 patient admission days with a total of 477 antibiotic therapy days post-integration. The AUR for combined NICU and nursery patients was 47 pre-integration and 47 post-integration (P > 0.999). For NICU patients, the AUR was 164 pre-integration and 169 post-integration (P = 0.729). For nursery patients, the AUR was 13 pre-integration and 11 post-integration (P = 0.377, Table 3).

Table 3.

Antibiotic Utilization Rate

We identified 6 cases of confirmed culture-positive sepsis throughout the 12-month study period. All recommendations generated by the EOS risk calculator for each of the 6 cases of culture-positive sepsis identified were in alignment with recommendations from current CDC/AAP treatment guidelines for the initiation of empiric antibiotics (Table 4).

Table 4.

Incidence of Culture-Positive Sepsis

DISCUSSION

In this retrospective analysis, the EHR integration of the Kaiser Permanente neonatal EOS risk calculator significantly increased calculator accuracy. The integration of the EOS risk calculator resulted in an insignificant increase in calculator utilization and an insignificant decrease in the occurrence of changed clinical recommendations following miscalculations. Only 1 patient with a miscalculation was directly impacted by the changed clinical recommendation after categorization of the clinical presentation following examination. All other patients with a miscalculation, which resulted in a changed clinical recommendation, were not directly impacted. For these patients, the changed clinical recommendation was not for the specific clinical presentation classification that categorized these patients following clinical examination.

In a recent study, EOS risk calculator integration increased compliance from 59% to 85% and decreased the frequency of antibiotic prescribing from 7% to 1%.¹⁴ Within the study, the EOS risk calculator was integrated and then improved, but was not fully automated. We attribute our higher pre-integration utilization percentage to extensive education and guidance provided from a developed Triple I/Early Onset Sepsis Toolkit distributed throughout our entire healthcare system before EOS risk calculator implementation.

Accuracy in the utilization of the EOS risk calculator as a clinical decision-making tool in the setting of neonatal EOS is imperative. The utility of the tool decreases if not appropriately utilized, and inaccuracies may directly influence the management of a neonate with possible EOS. Incorrect utilization of this tool has the potential to inappropriately identify cases of EOS, leading to missed cases of true EOS as well as inappropriate antibiotic utilization and laboratory monitoring.

Hospital B provided staff education, a calculator “help card,” and utilized a Nurse Clinical Educator within the nursery unit during both pre- and post-EOS risk calculator integration phases. In the Hospital B nursery unit, a pharmacist reviewed identified calculator errors post-EOS risk calculator integration. A Nurse Clinical Educator provided education to individual staff. Select cases of identified errors were also reviewed by the healthcare system analyst team. We found that both Hospital A and Hospital B nursery units had similar rates of miscalculations pre-integration. Hospital A nursery unit showed a greater improvement in miscalculations post-integration, further confirming the necessity for EOS risk calculator integration to improve calculator accuracy.

We have identified several limitations of this study. First, the monthly evaluation of EOS risk calculator results was only from a small portion of neonates. Systematic evaluation in a prospective fashion of a larger population would help further identify the impact of EOS risk calculator miscalculations. There are also multiple limitations to the integration of the EOS risk calculator itself. With the integration, maternal and neonatal information is automatically collected and displayed for the user, but within our healthcare centers, there is still an opportunity for individual data manipulation by the user before acceptance of results. Providing education to healthcare providers on how to appropriately utilize any decision-making tool is paramount.

Within both Hospital A and Hospital B, the most common miscalculations were due to incorrect imputation of the highest maternal antepartum temperature and the duration of rupture of membranes. The EOS risk calculator requires the entry of 6 variables to perform a risk calculation. Without proper education, these variables can be incorrectly input, resulting in a miscalculation of the EOS risk score and possible incorrect clinical recommendation. The first variable, the incidence of EOS, has 12 possible incidence rates from which to select. This list allows a hospital or healthcare system to select an EOS incidence rate closely matching their own. Our healthcare system uses an EOS incidence rate of 0.3/1,000 live births, and following EOS risk calculator implementation, required the end-user to select the correct EOS incidence rate manually. The incidence rate of 0.5/1,000 live births listed within the EOS risk calculator selection menu states “CDC national incidence” and has led to confusion among some end-users who were unaware of our healthcare system EOS incidence rate. Following EOS risk calculator integration, the calculator automatically defaults the EOS incidence rate to 0.3/1,000 live births.

Gestational age is recorded in the EOS risk calculator as weeks and days, with a range of 34 to 43 weeks. The EOS risk calculator will not calculate an EOS risk score if clinicians enter a gestational age outside of this range. With EOS risk calculator integration, gestational age correctly auto-populates both fields and will not calculate if the gestational age is outside the defined range.

Implementation of the EOS risk calculator required the end-user to review numerous days of chart documentation to identify the highest recorded maternal antepartum temperature within an appropriate time frame. With EOS risk calculator integration, the highest maternal antepartum temperature within the previous 7 days before delivery is now auto-populated into the EOS risk calculator. Following integration, we have identified instances where the end-user overrode the auto-populated highest maternal antepartum temperature and selected a temperature recorded either beyond 7 days before delivery or during the postpartum period.

During the implementation phase of the EOS risk calculator, end-users would often enter or round the duration of rupture of membranes incorrectly. This problem was resolved with the integration of the EOS risk calculator as the automated EOS risk calculator auto-populates the duration of the rupture of membranes input field with information from the delivery summary.

With EOS calculator implementation, the end-user was required to determine maternal GBS status and correctly select negative, positive, or unknown within the EOS risk calculator. The EOS risk calculator “help card” used at Hospital B instructed the end-user to select maternal GBS status unknown if a negative GBS culture was drawn older than 5 weeks before the time of delivery. This recommendation is in agreement with revised CDC guidelines, which state the negative predictive value of GBS cultures declines if performed older than 5 weeks before delivery.³ Following EOS calculator integration, automation incorporates the 5-week GBS collection timeframe into the maternal GBS status decision-making process. We have identified instances post-integration where the maternal GBS status was adjusted by the end-user from unknown to negative based on progress note documentation or GBS culture result regardless of being reported as negative older than 5 weeks from time of delivery.

Education on the selection of intrapartum antibiotics was provided in the Triple I/Early Onset Sepsis Toolkit distributed across the entire healthcare system. Additional information was provided within the EOS risk calculator “help card” utilized at Hospital B. One challenge identified following the implementation of the EOS risk calculator was the proficiency of the end-user to appropriately classify antibiotics and/or antibiotic combinations as either GBS specific or broad-spectrum antibiotics. Also, the end-user had to consider the timing of antibiotic administration. With the integration of the EOS risk calculator, automation correctly identified intrapartum antibiotic classification based on the type of antibiotics and the timing of antibiotic administration. We identified instances post-integration where end-users adjusted the type of intrapartum antibiotics selected through the automation process.

There was no significant change in the AUR throughout the study period pre- and post-integration of the EOS risk calculator. Additionally, in all 6 cases of confirmed culture-positive sepsis, the EOS risk calculator recommended utilization of antibiotics consistent with the CDC/AAP guideline criteria, and correctly identified true cases of infection.

EOS of the newborn remains a challenging clinical dilemma due to its relative rarity, high mortality, and lack of highly specific biomarkers. The integration of the Kaiser Permanente neonatal EOS calculator as a clinical tool into the EHR significantly reduces the number of miscalculations.

ACKNOWLEDGMENTS

We appreciate the assistance of Nicole Tipping in identifying patients for this project, Ruyun Jin for statistical data analysis, and Melissa Han for providing nurse education.

DISCLOSURE

The authors have no financial interest to declare in relation to the content of this article.