Use of an electronic medical record to optimize a neonatal sepsis score for mortality prediction

Abstract

OBJECTIVE:

Late-onset sepsis (LOS) is a significant cause of mortality in preterm infants. The neonatal sequential organ failure assessment (nSOFA) provides an objective assessment of sepsis risk but requires manual calculation. We developed an EMR pipeline to automate nSOFA calculation for more granular analysis of score performance and to identify optimal alerting thresholds.

METHODS:

Infants born <33 weeks of gestation with LOS were included. A SQL-based pipeline calculated hourly nSOFA scores 48 h before/after sepsis evaluation. Sensitivity analysis identified the optimal timing and threshold of nSOFA for LOS mortality.

RESULTS:

Eighty episodes of LOS were identified (67 survivors, 13 non-survivor). Non-survivors had persistently elevated nSOFA scores, markedly increasing 12 h prior to culture. At sepsis evaluation, the AUC for nSOFA >2 was 0.744 (p = 0.0047); thresholds of >3 and >4 produced lower AUCs.

CONCLUSIONS:

nSOFA is persistently elevated for infants with LOS mortality compared to survivors with an optimal alert threshold >2.

INTRODUCTION

Neonatal sepsis is an important cause of morbidity and mortality, particularly in the preterm, very low birth weight (VLBW, defined as weight less than 1500 g) population as incidence of sepsis increases with decreasing gestational age and birth weight [1]. Late-onset sepsis (LOS), defined as sepsis occurring after the first 72 h of life, occurs in almost 25% of VLBW infants and is far more common than early-onset sepsis. The overall incidence of late-onset sepsis has decreased in the past 20 years, likely a reflection of improved hand hygiene, skin care, human milk feeding, uniform practices for central line care, and increased attention to discontinuation of invasive devices, such as central lines, after they are no longer necessary [2]. However, the overall risk of mortality for VLBW infants with LOS remains very high, as much as 18% [3, 4]. Despite being a common and potentially deadly condition, recognition and evaluation for sepsis continues to be driven in most centers by subjective clinical evaluation and lacks sensitivity and specificity. Diagnostic errors can occur in both directions—infants without sepsis may undergo unnecessary invasive procedures (e.g., venipuncture for blood culture) and exposure to antibiotics, while infants with sepsis may be unrecognized and untreated before rapidly progression to severe illness and death.

Sepsis is a syndrome, defined by life-threatening organ dysfunction due to a dysregulated host response to infection [5]. To improve the timeliness and specificity of sepsis risk recognition, standardized sepsis scores have been developed for adults (Sequential Organ Failure Assessment, SOFA) and children (Pediatric Sequential Organ Failure Assessment, pSOFA). These methods categorize the extent of organ dysfunction and are trended over time to identify patients with increased risk of mortality [6–8]. However, until recently, there was no equivalent measure to stratify organ dysfunction and related sepsis risk in neonates.

The Neonatal Sequential Organ Failure Assessment (nSOFA) was developed and validated as an objective way to define neonatal sepsis and was modeled on the existing SOFA and pSOFA scores [9, 10]. In a pilot study, clinical and laboratory information was evaluated for a cohort of preterm infants who died from LOS to determine which markers of organ failure were most indicative of fatal neonatal sepsis [11]. This work led to the development of the nSOFA score which, in a recent multi-center study, demonstrated generalizability to predict infection-related mortality in external validation [12] and early mortality after preterm birth in a separate study [13].

The early pilot evaluation and validation of the nSOFA score demonstrated the potential value of a quantitative and objective system for classifying sepsis risk in neonates at the time of clinical decompensation. The nSOFA score may also have a potential use case in the clinical environment as an early warning system for sepsis, especially if improved sensitivity/specificity over clinical evaluation can be demonstrated. However, additional investigation needs to be completed to determine early predictive power. A nontrivial barrier to the widespread adoption of nSOFA in routine clinical use is the labor-intensive nature of manual calculation of the nSOFA score on a periodic basis, a task that could be automated by computers.

The electronic medical record (EMR) is an ideal platform for automated risk score calculation—it contains an enormous amount of structured data and is in constant use by providers. EMRs provide the source of data and the means to display it. The value of such tools has been demonstrated in other contexts to aid in early detection of patient deterioration [14–17]. Tools that evaluate severity of illness require large sample sizes for model development and frequent reevaluation with up-to-date data for score additions or readjustments [17]. Incorporation of the score into the EMR allows for the opportunity to develop an algorithm with a larger amount of training data and easily adaptable number of included features.

We hypothesized that calculating the nSOFA score at more frequent time intervals could identify high-risk patients earlier than the time of sepsis evaluation. To accomplish this, we aimed to automate the nSOFA data extraction and score calculation within the EMR to perform high-volume data collection and increase the speed of analysis. We also aimed to evaluate a set of thresholds to identify the optimal nSOFA threshold and time interval to best identify those at a higher risk of LOS-related mortality.

METHODS

Cohort selection

This single-center retrospective cohort study was performed using data from preterm infants admitted to Saint Louis Children’s Hospital, a level IV neonatal intensive care unit (NICU), a unit serving urban, suburban, and rural populations. The electronic medical record (Epic, Epic Systems Corp, Madison, Wisconsin) was used to build a data query to identify eligible infants admitted between June 1, 2018 and November 30, 2020. Infants were included if they were born before 33 completed weeks of gestation and had late-onset sepsis (defined as a positive blood culture after the first 72 h of life). Baseline demographics and clinical information including gestational age at birth, sex, birth weight, ethnicity, delivery mode, and age at time of blood culture were extracted.

Sepsis and outcome definitions

Each positive culture was considered a separate late-onset sepsis episode. The onset of sepsis episode was defined as the date/time a blood culture was drawn; this point was set as time zero, t₀. The primary outcome of interest was mortality, defined as death within 7 days of positive blood culture and/or during sepsis treatment. Provider documentation was reviewed for each infant to confirm that a) the positive blood culture was considered pathologic (as opposed to a contaminant), b) the infant was actively being treated with antimicrobials, and c) infection was the medical concern for cause of death. Based on this outcome, sepsis episodes were classified as “survivor” or “non-survivor” episodes. Non-survivor episodes were further divided into those that died within the first 48 h after positive blood culture, and those that died after 48 h.

According to our typical clinical practice, all infants with positive blood cultures have scheduled daily repeat cultures until negative. In our results, we excluded those cultures that were repeated within 24 h after the first positive culture and grew the same organism, as these should be considered part of the same discrete sepsis episode. Contaminated cultures were also excluded, identified by the growth of Staphylococcus Epidermidis/Hominis or Micrococcus, an antibiotic course of 48 h or less, and/or documentation of contamination in the provider note.

nSOFA score calculation

nSOFA score elements consist of a respiratory component (intubation, fraction of inspired oxygen requirement (FiO₂) and pulse oximetry oxygen saturation (SpO₂)), a cardiovascular component (use of inotropic medications and corticosteroids) and a platelet component (Table 1). We developed an automated pipeline for extraction of score elements from the EMR to avoid the manual calculation normally used to generate nSOFA scores. The data were extracted using SQL, a programming language commonly used for data stored in a relational database such as the structure used within the Epic EMR.

Table 1.

Neonatal Sequential Organ Failure Assessment (nSOFA) Components and Scoring.

Respiratory Score	0 Not intubated OR intubated + SpO2/FiO2 ≥ 300	2 Intubated + SpO2/FiO2 < 300	4 Intubated + SpO2/FiO2 < 200	6 Intubated + SpO2/FiO2 < 150	8 Intubated + SpO2/FiO2 < 100
Cardiovascular Scorea	0 No inotropes AND no systemic steroids	1 No inotropes AND systemic steroid treatment	2 One inotrope AND no systemic steroids	3 Two or more inotropes OR systemic steroid treatment	4 Two or more inotropes AND systemic steroid treatment
Hematologic Scoreb	0 Platelet count ≥150 × 103	1 Platelet count 100–149 × 103	2 Platelet count <100 × 103	3 Platelet count <50 × 103	NA

This table is adapted from Wynn and Polin [10], scores range from 0 (best) to 15 (worst).

^aMedications considered as inotropic: dopamine, dobutamine, epinephrine, norepinephrine, vasopressin, milrinone, and phenylephrine.

^bMost recent platelet count available to the clinician.

Each element of the nSOFA score was mapped to the corresponding data field within the SQL database. FiO₂ and SpO₂ data were mapped to flowsheet rows, using the last value recorded at the time of interest. Upon review, we identified that no single field was available to determine whether the patient was intubated at a particular time. Instead, we developed a combination of flowsheet data, including use of oxygen and delivery device used, to extrapolate whether an infant was intubated. Inotrope and corticosteroid information was collected from a medication database (as opposed to the order entry database) to ensure data was included only if medications were given. There were instances in which inotropes or steroids were ordered but not given; these were excluded. Platelet information was mapped to lab results and designed to look back for the last recorded value up to 7 days in the past. If no value was found, a score of 0 was assigned for the hematologic score. After data extraction, processing and calculation of each nSOFA component score as well as the total score were performed using Python (Python Software Foundation, Python Language Reference, version 3.8).

Starting at t₀, all nSOFA score elements were extracted in one-hour intervals, spanning 48 h before (t₋₄₈) and after (t₊₄₈) septic evaluation. For infants that died within 48 h from t₀, no further data was included past the time of death. The total nSOFA score was calculated for each hour interval. After initial data extraction, cycles of manual validation were completed, and appropriate adjustments made to query code to ensure data accuracy. Manual validation of the final extraction and calculations confirmed accuracy of data pipeline.

Statistical approach and ethics approval

The Mann-Whitney test was used for comparison of nonparametric continuous data and are summarized as medians with quartiles (25th and 75th percentiles). The Fisher exact test was used for categorical data which are summarized as percentages. Tests were considered statistically significant where p < 0.05.

A sensitivity analysis was conducted to identify the optimal timeframe and accuracy of nSOFA to predict mortality from late-onset sepsis using serial evaluation of the area under the receiver operating characteristic curves (AUC). Prior to initiation of sepsis evaluation, the average total nSOFA scores between the survivor and non-survivor groups ranged between 1 and 5. This led to the decision to evaluate the total score thresholds of >2, >3, and >4. The AUC for mortality was calculated using each of these thresholds at every hour from t₋₄₈ to t₊₄₈. The AUC values were graphed over this time to show how each threshold performed prior to and after the time of sepsis evaluation.

All statistical calculations were performed in R Studio (RStudio PBC, Boston, MA, Version 1.3.1093). Figures were generated using the ggplot package for R [18].

The study design was reviewed and approved by the Washington University Institutional Review Board under a waiver of informed consent.

RESULTS

Cohort description

A total of 67 unique patients met all inclusion criteria. These infants had a median gestational age of 26.2 weeks (interquartile range [IQR] 24–28 weeks) and median birth weight of 790 g (IQR 648–978 g). Within this cohort, we identified 80 distinct sepsis episodes (median of 1.2 per patient) which were considered individually and analyzed by episode (Table 2). Patients with repeat episodes of sepsis had a median of 40 days (IQR 15–55 days) between episodes. The most common organisms identified were Coagulase Negative Staphylococcus Species (n = 17), Staphylococcus epidermidis (n = 12), and Staphylococcus aureus (n = 10).

Table 2.

Cohort demographics.

	Survivors (n = 67)	Non-survivors (n = 13)	P value
Gestational age, weeks	26.4 (24.6, 28.3)	24.3 (24.1, 26.4)	0.019a
Birth weight, g	820 (700, 1010)	560 (480, 760)	0.005a
Male	38 (57)	7 (54)	1b
Ethnicity			0.029b
AA	33 (49)	9 (69)
White	29 (43)	2 (15)
Hispanic	4 (6)	0 (0)
Pacific Islander	1 (1)	0 (0)
Other	0 (0)	2 (15)
C-section	50 (75)	9 (69)	0.735b
Day of life of septic evaluation, days	23 (10, 49)	16 (10, 31)	0.430a

Data are presented as median (IQR) or n(%).

^aMann-Whitney.

^bFisher exact Test.

Sepsis episodes and outcomes

A total of 80 episodes of late-onset sepsis were identified. Infants survived 67 of the episodes (83.8%) and 13 (16.2%) culminated in death. Non-survivors had a significantly smaller birth weight (median 560 g [IQR 480–760 g] vs 820 g [IQR 700–1010 g], p = 0.005) and were born at an earlier gestational age (median 24.3 weeks [IQR 24.1–26.4 weeks] vs 26.4 weeks [IQR 24.6–28.3 weeks], p = 0.019). On average, non-survivors had infections earlier after birth than survivors (median 16 days [IQR 10–31 days] vs 23 days [IQR 10–49 days], p = 0.43), however, this difference was not statistically significant. There was also no statistical difference seen between groups based on sex or mode of delivery.

nSOFA score comparison

From the 80 sepsis episodes, a total of 7760 hourly nSOFA score calculations were made and compared between survivors and non-survivors. The average total nSOFA scores for these groups over time are shown in Fig. 1. Survivors were noted to have a lower average initial nSOFA score of 1.5 at t₋₄₈, which underwent minimal change during the study window and had a lower average peak score of 2.7 at t₊₁. Non-survivors had a higher average nSOFA score than survivors at all time points.

Fig. 1. Total nSOFA scores before and after sepsis evaluation.

Total nSOFA scores for all episodes (n = 80) separated into three groups: survivors (n = 67, C), non-survivors who died within 48 h after positive blood culture (n = 7, A), and non-survivors who died after 48 h from culture (n = 6, B). Data are graphed hourly from 48 h before (t₋₄₈) to 48 h after (t₊₄₈) the blood culture. Nonlinear regression lines with 95% CI are shown. No deaths occurred between t₊₃₀ and t₊₄₈.

Non-survivors were divided into two groups, those that died within the first 48 h after sepsis evaluation, and those that died after 48 h. This separation better illustrates the change in nSOFA score for those with the most severe illness. The average total scores for the two groups of non-survivors are initially similar. Both groups show a higher baseline score than the survivor group. The non-survivors with later mortality have a consistent gradual rise in total nSOFA score to a maximum average score of 8 at t₊₄₃. In contrast, the non-survivors that died earlier had a more pronounced rise to a maximum nSOFA score of 15 at t₊₃₀. This acceleration of the rise in nSOFA score begins 12 h prior to the sepsis evaluation (t₋₁₂).

Outliers can be found within survivor and non-survivor groups. Several cases within the survivor group were found to have high initial nSOFA scores, however, they remained unchanged throughout the sepsis evaluation. On the other hand, non-survivors had widely varied initial nSOFA scores (0–11), but in all cases, their scores gradually increased over time.

There were 23,280 component scores (separate respiratory, cardiovascular, and hematologic scores) calculated for all sepsis episodes to generate total nSOFA scores. For the two groups of non-survivors, each component score was graphed separately over time to determine which components contributed most to the rise in total nSOFA score (Supplementary Fig. 1). Each component increases over time, however, the largest contributions for the non-survivors who die early were from the respiratory (increase in average from 2 to 8) and cardiovascular (increase in average from 0 to 3) components.

AUC analysis

The AUC values for mortality for nSOFA thresholds of >2, >3, and >4 are summarized in Fig. 2. The AUC at the time of sepsis evaluation (t₀) is highest for a threshold of nSOFA >2 with a value of 0.744 (p = 0.005). This threshold also has the best performance prior to the culture with an AUC of 0.78 (p = 0.005) at t₋₄₈. For the threshold of nSOFA > 3, the AUC at t₀ is 0.682 (p = 0.02). This threshold has the best performance after the time of culture with an AUC of 0.827 (p = 0.001) at t₊₄₈. A threshold of nSOFA > 4 does not perform better than the other two thresholds. The AUC for t₀ is 0.666 (p = 0.027).

Fig. 2. Mortality prediction AUC by threshold and time.

AUC values calculated for total nSOFA score thresholds of 2 (circle), 3 (triangles), and 4 (squares). AUC values are shown hourly from 48 h before (t₋₄₈) to 48 h after (t₊₄₈) the positive blood culture. Nonlinear regression lines with 95%CI are shown.

For almost all the time points evaluated, the confidence intervals are overlapping, indicating that each of the thresholds selected may be an appropriate choice in identifying patients with higher mortality risk. The performance of all the thresholds decreases slightly from t₋₄₈ to t₋₁₂, and then has a steady rise for the remainder of the timepoints.

DISCUSSION

Using automated, large-volume data extraction from the EMR, we were able to increase the granularity of nSOFA score calculations and show that infants at highest risk of early mortality have a distinct early rise in total nSOFA score. The automation of score calculation was found to be accurate, and easily modifiable to allow for detailed data analysis. In this study, nSOFA score elements were extracted hourly, however, this could be changed to any time interval of interest. Increased granularity also allowed for detailed evaluation of various score thresholds, clarifying that no single threshold can be used to identify high-risk patients.

While our overall results are consistent with previously published data that show higher total nSOFA scores correlate with increased risk of mortality [10, 13], we noted a distinct separation in nSOFA scores between those with early and late mortality, a difference not previously identified. Those who die most quickly have the earliest and sharpest rise in nSOFA score, beginning 12 h prior to the sepsis evaluation. When examining the range of possible thresholds to use for identification of survival or non-survival, we noted overlapping performance between each threshold. The increased granularity of our data, compared to earlier publications, allowed us to identify that the change in a patient’s baseline score and a sustained rise in the nSOFA score are key factors for predicting mortality and not crossing any discrete threshold. Survivors tended to have a low, unchanging baseline score that remained unchanged throughout the sepsis evaluation period while non-survivors had early, high baseline scores with further increases. It is therefore crucial to develop a system for close monitoring of changes in a patient’s baseline nSOFA score.

The total nSOFA score is higher in non-survivors at all data points, even as early as 48 h prior to the culture time. This may indicate that there is an even earlier time point where scores diverge between survivors and non-survivors. This could be evaluated with further data collection earlier than t₋₄₈. It is also possible that non-survivors have a higher baseline nSOFA score since the time of birth, implying that they have some underlying physiologic predispositions to higher mortality. The non-survivor group is of smaller size and younger gestational age, which are likely contributors and may generate chronically high nSOFA scores. Further prospective investigation will help determine if earlier sepsis evaluation and initiation of antibiotics in patients with higher baseline nSOFA scores can reduce mortality.

Incorporating nSOFA into the EMR as a routine clinical decision support tool is a nontrivial challenge along both technical and clinical domains. For technical challenges, although the components of the nSOFA score are far from obscure, reliable automated abstraction—mapping values from multiple flowsheets or lab records—can be deceptively difficult. Data within the EMR is documented and stored in different locations with different naming conventions between institutions. There are numerous vendors of EMR software worldwide, each of which will require unique, custom software engineering for nSOFA deployment. Furthermore, even within software from the same vendor, there are unique idiosyncrasies of local deployment which prevent simple “plug and play” installation. To apply the score consistently, data used for each component would need to be mapped to a universal code.

In terms of clinical challenges, we anticipate that each center adopting nSOFA will need to calibrate their model prior to use. Variation in laboratory systems, pharmacy conventions, and respiratory support charting may influence score values and calculation. In our experience in utilizing an Epic EMR for calculation of the nSOFA score, repeated testing and validation against manually calculated scores were required in data mapping to ensure accuracy. These challenges highlight the importance of standardization of medical record data, to simplify and improve interoperability [19]. While EMR systems have repeatedly shown improvement in quality of data, reduction in errors, and greater adherence to guidelines [20], significant barriers arise in population health and clinical research due to wide variations of data collection and storage [21–23]. National health policies or health information exchanges may help to assure standardization to allow for effective collaboration and exchange of critical patient data [19].

The nSOFA score differs from the other scores that have been previously developed for assessing neonatal mortality risk. Clinical Risk Index for Babies-II (CRIB-II) and Score for Neonatal Acute Physiology-Perinatal Extension-II (SNAPPE-II) assess illness severity and overall risk of mortality based on information limited to the first 12 h of life [24, 25]. These scores are not designed for ongoing monitoring of patients. The heart rate characteristics (HRC) index (HeRO score) was developed as an early warning tool for LOS using a monitoring algorithm of a patient’s heart rate variability and patterns. In a recent comparison to nSOFA, HRC was shown to similarly increase prior to blood culture, and be higher in non-survivors. The predictive performance increased when both scores were used together, demonstrating their potential for combined use [26]. However, the HRC index requires the purchase of additional equipment/software to analyze a patient’s heart rate data and calculate the score. The cost and infrastructure to acquire and maintain this system hinder accessibility. In contrast, nSOFA is calculated using data already captured within the EMR and could be automated directly in the EMR without additional software requirements.

Although nSOFA is a linear composite score derived from clinical variables that clinicians may be aware of, the use of nSOFA as a scoring tool translates the escalation of care into predicted risk. To further improve this risk prediction, the future investigations could leverage artificial intelligence or machine learning for further optimization or exploration of other variables to increase performance. The application of machine learning technologies to medicine has rapidly expanded in the last decade with numerous examples of expert systems to identify need for ICU admission [27], microscopic cell classification or cancer diagnosis [28, 29], and prediction of heart failure progression [30]. Such algorithms could be applied to further develop the complexity and precision of a neonatal sepsis early warning score. In addition to using data from the current score paradigm, numerous other features including lab results, imaging, and medications could be evaluated and included in the determination of a patient’s illness severity. As practices and available therapies evolve, machine learning algorithms can adapt and continue to improve their precision, although regulatory hurdles are not insignificant [31].

We acknowledge several limitations in our study. Our single-center cohort included a moderate total number of patients, although our results, including the optimal threshold of >2, were in line with previous publications [13] suggesting that our sample performed as anticipated. Several infants in this cohort had repeated episodes of sepsis. We cannot rule out that these infants have distinct physiology which increases their predisposition for infection and may impact nSOFA scores. Given the lack of autopsy data, it is not possible to say with complete certainty that the infants in this study died due to infection. However, as we limited the cohort to those with known positive blood cultures and death within proximity to infection, we felt this conclusion was justified. Finally, platelet values are frequently missing, as this component of the score requires active clinical investigation for updated values (as opposed to the respiratory or medication components, which always have a current value instead of being performed episodically). Although we used a standardized approach to this missing data, it undoubtedly added uncertainty.

CONCLUSIONS

The nSOFA score can predict mortality in high-risk infants with sepsis. This objective, data-driven indicator of worsening clinical status may have the potential to be used as an early warning system for those infants on the path to severe illness. Ongoing prospective analysis will be needed to further assess the utility of the nSOFA score. Integration into the EMR is the first step in speeding up the ability to gather more granular data and may lead to future innovations by leveraging advances in computer science. With precise monitoring of changes in nSOFA score, this tool may lead to earlier, specific interventions and reduced neonatal mortality.

DATA AVAILABILITY

The datasets generated during and/or analyzed during the current study are not publicly available due to patient privacy restrictions. A limited and de-identified dataset may be available from the corresponding author on reasonable request.