Artificial and Human Intelligence for Early Identification of Neonatal Sepsis

Abstract

Artificial intelligence has a role in the early detection of sepsis in neonates. Machine learning can identify patterns that predict high or increasing risk for clinical deterioration from a sepsis-like illness with careful consideration of the data and methods used to develop, validate, and evaluate prediction models. Sepsis AI systems can alert clinicians to a change in a patient’s condition that warrants a bedside evaluation before signs and symptoms of sepsis become obvious. At this point, human intelligence and experience come into play to determine an appropriate course of action: evaluate and treat or wait and watch closely. With intelligently developed, validated, and implemented AI sepsis systems, both clinicians and patients stand to benefit.

Scenario 1:

An experienced neonatologist is rounding on 15 very low birth weight (VLBW) preterm infants in a level IV NICU. Nurses are concerned about five infants; one with more apnea than usual, one with feeding intolerance, one with a low platelet count on routine testing, one requiring more incubator heat to maintain temperature, and one who, according to her mother, is less active than usual. The neonatologist examines each of the five infants and reviews the vital signs, laboratory tests, and clinical data from the past several days. A sepsis workup is performed, antibiotics are started for one infant, and the others undergo close observation. One of these infants has clinical deterioration requiring intubation the next day, at which point antibiotics are started.

Scenario 2:

An artificial intelligence (AI) system for early detection of neonatal sepsis has been implemented in a level IV NICU. An automated algorithm incorporates analysis of heart rate and oxygen saturation patterns to give a score that is the fold increased chance that the infant’s condition will deteriorate due to a sepsis-like illness in the next 24 hours. A clinician is rounding on 15 VLBW infants, and the Sepsis Score is elevated for 7 of them. On further review, some infants have a chronically elevated score. After examining the infants and their available data, a sepsis workup is performed, and antibiotics are started for two infants. A point-of-care biomarker blood test is ordered for two other infants, and when the test result is abnormal for one a sepsis workup is performed, and antibiotics are initiated.

Which scenario is better?

As AI becomes a reality in healthcare, a more appropriate question is “How can computer intelligence combine with human intelligence to improve patient outcomes?” In this narrative review, we will address a number of questions related to AI for sepsis prediction and detection in NICU patients. First, we address aspects of neonatal sepsis that make it a tractable problem for machine learning (ML) predictive models. Next, we cover technical aspects of ML model development and validation, including variable selection using both static and dynamic data. We will then review some existing early warning and ML systems. And finally, we will discuss the benefits of and barriers to implementing sepsis prediction systems in the NICU, with the goal of “right timing” antibiotics to improve patient outcomes.

Q1: How suitable is the problem of neonatal sepsis for AI solutions?

Premature infants in the NICU are an ideal population for AI-based sepsis monitoring. They are a relatively homogeneous group with the same reason for needing ICU-level care: they were born too early and need medical support while they grow and develop. They are by default immune-compromised, with immaturity of both innate and adaptive arms of host defense¹, and they require life-sustaining invasive devices that create a high risk for infections that can lead to sepsis. In contrast to adult and pediatric ICU patients, many preterm infants are relatively healthy for most of their weeks spent in the NICU. Therefore it is possible to derive models that detect the transition from “well” to “ill.” Moreover, neonatal sepsis AI models have a greater return on investment compared to those developed for adults due to the potential for improving health for a long life expectancy. The ultimate goal is for clinicians and data scientists to collaborate to develop and implement AI models that facilitate earlier detection and treatment of true sepsis. Ideally, models will be equitable (free of bias) and explainable (understood by clinicians). An important goal when implementing sepsis AI is to limit antibiotic overuse which disrupts the developing microbiome and negatively impacts health outcomes^2–6.

Timing and gestational age impact the use of predictive models for neonatal sepsis. Early-onset sepsis (EOS) is generally transmitted perinatally and diagnosis occurs shortly after birth whereas late-onset sepsis is typically due to hospital-acquired pathogens and usually occurs beyond the first week of age^7,8. Risk of both conditions increases with decreasing gestational age. For EOS, a static prediction model can be applied for term and late preterm infants not undergoing continuous monitoring⁹. In contrast, prediction of LOS for preterm NICU patients can incorporate continuous risk prediction from streaming vital sign data¹⁰.

Prediction models will perform best if the targeted outcome is well-defined and validated. For sepsis, this requires careful medical record review rather than simply relying on ICD codes since many studies have shown that diagnostic codes for sepsis are inaccurate 11^11,12. A challenge with regard to neonatal sepsis is the lack of a consensus definition^13,14. The diagnosis of sepsis in adults and older children is based on standard criteria that include measures of organ dysfunction. Because of the lack of consensus definitions for neonatal sepsis, studies vary widely in how they define sepsis as an outcome, making it difficult to compare and interpret results across studies^15,16. Some prediction models train only on culture-positive sepsis, while many include cases of “clinical sepsis” in which an infant has significant signs of illness and clinicians opt to prolong antibiotic treatment despite negative cultures. Experts argue that in the setting of modern laboratory equipment and sufficient inoculation volume, the likelihood of a false negative blood culture is extremely low^17,18. Nonetheless, as discussed later in this review, many prediction models have been developed using both clinical and culture-positive sepsis cases, and it is therefore important for clinicians to use judgment to decide on duration of therapy in the face of a high or rising risk score, since misuse of antibiotics can lead to adverse outcomes^{19, 6}.

Another challenge to consider in developing neonatal sepsis AI is variability in illness severity. The clinical spectrum of culture-positive neonatal sepsis is broad, from a minimally symptomatic infant with rapid recovery and no apparent sequelae to an infant with fatal or life-threatening multi-organ dysfunction and permanent neurologic impairment. Consensus definitions not only of sepsis but of its severity would be useful both for clinical care and for research, including predictive model development^13,14. The neonatal Sequential Organ Failure Assessment (nSOFA) assigns points for increasing signs of organ dysfunction and was built to discriminate sepsis survivors from non-survivors ²⁴. Multicenter analysis of nSOFA showed that it predicts fatal sepsis with high discrimination accross centers and gestational age ranges²⁵, making it a promising metric for defining sepsis severity.

A final consideration in model development is whether to include significant, physiologically overlapping conditions such as necrotizing enterocolitis (NEC). One approach is to develop models specific for NEC^20,21, though developing robust models would require a very large population of preterm infants since NEC is an uncommon event. Another approach is to include cases of both NEC and sepsis as outcomes in model development, and then evaluate the model’s ability to predict each condition separately²². And finally, models may be trained on cases of sepsis and then tested for their ability to predict NEC, as was done for the HRC index discussed later in this review²³.

Q2: What are the important AI and Machine Learning model development concepts?

Figures 1 and 2 provide a conceptual overview of aspects of AI relevant to healthcare applications, from algorithm development through clinical implementation and integration. The term “artificial intelligence” refers to many methodologies that use computers to “think” like humans. In some circumstances, properly developed AI may be superior to human intelligence because it is not susceptible to fatigue, distractions, inexperience, and understaffing that may lead to human errors.

Figure 1:

Schema of artificial intelligence (AI), machine learning (ML), and clinical prediction models.

Figure 2:

A conceptual diagram illustrating the process and key steps of developing sepsis AI technology.

Machine learning (ML) is a type of AI that involves “teaching” computers to predict or detect specific outcomes by finding patterns in data²⁶. ML methods can be thought of in several general categories. Supervised methods include classification and regression, using algorithms to find structure in labeled data, while unsupervised ML involves clustering and dimension reduction of unlabeled data. Reinforcement ML involves an iterative trial-and-error process whereby penalties are assessed for model errors and rewards given for accurate predictions. Generally, sepsis prediction models use supervised ML with various modeling methods, including regression, tree-based methods, neural networks, and others. In some studies, a variety of modeling methods were shown to have similar predictive performance^27,28, while in other studies, a specific method is found to have better performance than others ²⁹.

Many methods exist to assess model performance when developing and reporting a predictive model, the most common being the area under the receiver operating characteristics curve (AUC). The AUC summarizes the model’s ability to discriminate cases from controls over all possible thresholds³². In models designed for clinical use, the AUC value alone is insufficient to evaluate model performance since it does not consider prior probability, does not provide information about the distribution of errors, and weights omission and commission errors equally^33,34. Moreover, even a model with good discrimination may provide risk estimates that are unreliable³⁵.

Another way to evaluate model performance is by calculating sensitivity, specificity, and negative and positive predictive values (NPV, PPV)³⁶. Although late-onset sepsis occurs in approximately 15% of very preterm infants, the chance of an infant developing sepsis on any particular day is quite low. Thus, the PPV of non-static models (continuously evaluating the risk of imminent sepsis as vital sign patterns and clinical variables change) will be low in order to have acceptably high sensitivity. For a high morbidity and mortality condition such as sepsis, and one for which an action (antibiotics) can be life-saving, this is a fair tradeoff.

In addition to AUC and measures like sensitivity and PPV, ML model performance should be evaluated using qualitative methods. Figure 3 illustrates a ROC curve as well as two qualitative analyses, a calibration plot and a time-to-event plot. A model can have good discrimination (i.e., a high AUC) yet still have estimated risk predictions that are unreliable if it is poorly calibrated³⁷. The calibration of a model’s risk predictions can be visualized by plotting the observed risk as a function of the predicted risk³⁸. Figure 3b shows an example of a well-calibrated model in red, for which the predicted risk plotted in deciles on the x-axis is very close to the estimated event rate on the y-axis and models that over- or under-predict risk in blue and green, respectively. The reliability of these plots depends on a sufficiently large sample size and the number of events in the data³⁹.

Figure 3:

Examples of Model Performance Metrics. The left panel displays an area under the receiver operating characteristic curve for a population of patients. The AUC is created by plotting the sensitivity against the specificity across all thresholds of the model output. The middle panel is an example of three model calibration curves for a population of patients. Predicted risk relative to average is on the abscissa and observed risk relative to average is on the ordinate. Each point represents one decile of predicted risk. The line of identity is shown as a dashed line and represents a well-calibrated model. The other two lines represent models that either over- or under-predict risk. The right panel is an example of a “Time to Event” plot for an individual patient. In this example, the model output (risk of sepsis score) is rising steeply 4 hours before a patient had a blood culture sent that diagnosed sepsis at time zero. Theoretically, if clinicians could see and interpret the rising score, antibiotics might have been given 4 hours earlier in this example.

Another important way to evaluate model performance is with time-to-event plots, as shown in Figure 3c. These show the average model output in a cohort relative to the time of the event and illustrate the horizon or lead time for sepsis prediction. This qualitative model assessment provides valuable information about its clinical utility since a score without a rise before clinicians recognize illness is not likely to benefit patients. Evaluating models based on how much time in advance of sepsis the model predicts a rising risk of sepsis will improve clinical utility by facilitating the administration of antibiotics hours earlier than with conventional monitoring and clinical observation.

Once a model is developed or trained, testing or validation is an essential next step. A validation data set can be internal (a subset of the original dataset) or external, from a new cohort at a different center. Validation in external datasets with similar patient characteristics and practice patterns compared to the test data provides evidence for reproducibility of model performance, while external validation using data from cohorts with different characteristics (for example, different centers, patient demographics, level of illness, or clinical practices), provides evidence of model transportability⁴⁰. In an example from our prior work, we showed that differences in invasive versus non-invasive respiratory support across NICUs impacted the performance of a sepsis prediction model that incorporated features to detect apnea²².

In addition to external validation, ongoing evaluation of ML models is essential to ensure adequate performance after implementation. Data shift or drift may occur over time as practices, hospital systems, and patient populations change⁴¹. Examples that could impact NICU sepsis model performance include a change in bedside monitors with differences in HR or SpO2 averaging times, change in practices for obtaining specific laboratory tests that serve as model inputs, or changes in the use of medications or respiratory support that may impact vital sign patterns.

The ultimate step in model evaluation is conducting randomized clinical trials to determine whether displaying the output of an AI algorithm leads to meaningfully improved outcomes for patients or for care providers. Only through well-designed large RCTs will sepsis AI systems be trusted, implemented, and routinely used for patient care. And finally, since many research groups are developing sepsis AI, it is important that results and algorithms be shared with other researchers. In order to better interpret results across studies, models should be reported using a standardized format such as TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis)³⁰ or a subsequent format specific for AI (TRIPOD-AI)³¹.

Q3: What data can be used in neonatal sepsis prediction models?

A first step in ML model development is to determine which data variables to use for modeling. This is done through various methods, foremost of which is using domain knowledge of clinical associations with neonatal sepsis. Other factors often considered involve the accessibility of the data and the feasibility of its use in prospective implementation. When predicting imminent late-onset sepsis, other decisions include whether to use physiologic data derived from high-resolution data (e.g., the electrocardiogram waveform signal sampled at 250 Hz), low-resolution data (e.g., demographics, clinical risks or signs, intermittently sampled vital signs, or laboratory tests), or a combination of both.

Vital sign data for sepsis AI algorithms:

Physiological monitoring data for sepsis prediction models can be continuously or intermittently sampled. Continuous bedside monitoring data available for NICU patients include heart rate or inter-heartbeat intervals from the electrocardiogram, SpO2 and pulse rate from the pulse oximeter, and respiratory rate from chest impedance signals. Neonates have irregular respirations, so the displayed breathing rate on the monitor, averaged over a short time interval, often oscillates from high to low when the actual respiratory rate counted over 30–60 seconds is somewhere in between. NICU monitors are also notoriously inaccurate for detecting apnea⁴². For these reasons, respiratory rate from current NICU bedside monitors may not be particularly useful in sepsis detection algorithms.

Hypotension is an important marker of sepsis but often presents later in the course of illness. Many infants do not have an invasive arterial catheter at the time they develop sepsis, and cuff measurements can be unreliable^43,44, but new technology for continuous non-invasive blood pressure monitoring is under development⁴⁵. With regard to temperature, premature infants are cared for in incubators, thus considering both patient and incubator temperature would be needed to detect temperature instability (hypothermia or fever) as a sign of sepsis. Detecting a difference between core and peripheral temperature has been proposed as a warning sign of poor perfusion related to sepsis, and research is ongoing to determine how to incorporate this into early warning systems^46–48. Other new technologies for continuous monitoring of ICU patients are under development and may enable greater depth and breadth of vital signs and other physiologic signals that can be incorporated into sepsis AI models. Wireless sensors are being developed for PICU and NICU patients that can measure not only vital signs but also other clinical variables such as motion and vocalization that might provide clues to changes in patient status⁴⁹.

With existing monitoring technology, a pathologic change in cardiorespiratory patterns can serve as a physiomarker of sepsis in the early stages as the body mounts a systemic inflammatory response^50,51. Low variability of HR⁵² accompanied by HR decelerations was recognized as a signature of illness due to neonatal sepsis in premature infants ⁵³. The mechanism of abnormal heart rate characteristics in sepsis is not completely understood but likely involves both cytokine effects and autonomic nervous system activation with increased vagus nerve firing^54–57. The heart rate characteristics (HRC) index, a continuous sepsis prediction model, was developed to capture these abnormal patterns and is discussed later in this review ¹⁰.

In searching beyond HR patterns for other physiomarkers of neonatal sepsis, a logical place to look is in the respiratory data. While respiratory rate has limitations due to inaccuracies in measurement and sampling, SpO2 data contain information that may add to HR analysis. An increase in central apnea is one of the major signs of sepsis in preterm NICU patients, ^56–60 due at least in part to cytokine-triggered release of endogenous prostaglandins ^56–58. Apnea detection through chest impedance waveform analysis is complicated, while detection of a decline in HR and SpO2 that often accompany apnea is simpler. One analytic that serves this purpose is cross-correlation of HR and SpO2 which measures the degree to which the two signals co-trend within a set lag time. An increase in this metric captures deceleration-desaturation events, which have been shown to correlate with increased central apnea or exaggerated pathologic periodic breathing in preterm infants^22,59.

Although continuous physiologic monitoring data are important for sepsis prediction, there are limitations. First, differences in equipment sampling rate, averaging time, or artifact filtering may impact predictive model performance when applied at different centers or in different epochs. Second, clinical practices that impact vital signs vary widely among NICUs, including medications, respiratory support, and monitor alarm settings. Our group conducted a three-NICU comparison of vital signs measured every two seconds from VLBW infants and found small differences in SpO2 patterns and in number of bradycardia episodes between sites. These were likely due to differences in equipment and in site-specific clinical practices such as use of mechanical ventilation ⁶¹. Another important consideration is that cardiorespiratory instability in preterm infants is not a specific sign of sepsis. For example we have shown that infants with severe intraventricular hemorrhage and severe acute or chronic lung disease have abnormal heart rate characteristics in absence of sepsis ^62,63.

Demographic, Laboratory, and Clinical Data for sepsis AI algorithms:

Gestational age and birth weight are inversely proportional to the rate of neonatal sepsis and can be incorporated into prediction models ^10,77, ^7,8,78. Postnatal and postmenstrual age also provide information on sepsis risk, since late-onset sepsis incidence peaks at 1–3 weeks of age and decreases with maturation as immunity becomes more robust and invasive lines and tubes are no longer needed^79,80. Male sex has been associated with a higher risk of sepsis, with the hypothesis that sex hormones impact immune system development ^81,82. Finally, race and ethnicity may be associated with increased risk, and there is some evidence this may be due not to biological but rather to socioeconomic factors ^83,84.

Laboratory tests that measure components of the host response to infection, such as immature neutrophils or serial c-reactive protein (CRP) values, are the most commonly used tests for sepsis screening^64–68. These tests do not have adequate sensitivity to stand on their own, but may serve as decision support for either starting or withholding antibiotics in conjunction with other clinical variables. Cytokines, such as interleukin-6, are likely to be more useful than CRP for sepsis prediction^69–72 or sepsis severity and mortality risk prediction⁷³, but are not yet readily available in most clinical laboratories, and research into other biomarkers is ongoing^74,75. Salivary diagnostics are increasingly used for non-invasive screening and might lead to testing strategies to identify infants with sepsis prior to clinical deterioration ⁷⁶.

Clinical risk factors for LOS include the presence of a central vascular catheter or the need for mechanical ventilation. Certain medications may be associated with increased sepsis risk, either because they weaken the immune system or because they were prescribed due to physiologic instability. For example, hydrocortisone may be given to preterm infants with severe chronic lung disease and might slightly increase risk of late-onset sepsis by immune suppression, but hydrocortisone is also commonly prescribed for hypotension ⁸⁵ and therefore may indicate presence of (and not simply risk for) sepsis. Variables known to decrease sepsis risk include an exclusive human milk diet and probiotics. While such variables might improve the predictive performance of a sepsis ML model, they generally report on things that clinicians already know and are not particularly useful in developing early warning systems.

Clinical signs of sepsis are the main way that humans (in contrast to machines) assess whether a preterm infant should be evaluated and treated. Incorporating clinical signs into sepsis algorithms generally requires user input. For example, the EOS calculator, described later in this review, incorporates risk factors but also allows users to input their clinical assessment of the infant as asymptomatic, equivocal, or having clinical illness ⁹. With regard to LOS, common signs include increased apnea, respiratory distress, feeding intolerance (ileus), poor perfusion (delayed capillary refill, oliguria), temperature instability, hypotension, and lethargy. Several studies have developed logistic regression models to assess the sensitivity or predictive value of clinical signs documented as prompting sepsis evaluation^60,86,87. These signs may be captured from EHR documentation, but often not reliably or in a time frame that could go into an AI algorithm and help a patient. For example, text mining from clinician notes might extract specific words or word combinations like “lethargic” or “poor perfusion”, but once an ICU clinician writes this in the EHR they typically have already ordered the blood culture and antibiotics. One solution to automatically measure lethargy is monitoring for decreased movements, either with an actimeter ⁴⁹ or by detecting changes in motion artifact in waveforms from standard bedside monitors.⁸⁸

Q4: What advanced warning systems for sepsis exist, and what lies in the future?

Before discussing developed and future AI systems for sepsis, consideration should be given to “Early Warning Scores” (EWS). EWS and ML models are both designed to alert the medical team to concerning vital signs, laboratory values, or other clinical changes that might otherwise go unnoticed. Both can be integrated into the EHR or displayed at the bedside, and both can incorporate information from a mix of static and dynamic clinical variables. EWS employ a “track and trigger” approach, whereas AI models use math and the data to learn temporal trends and correlation among parameters ⁸⁹. For example, the Pediatric Early Warning Score (PEWS) is calculated based on periodic observations of multiple physiological parameters and designed to predict clinical deterioration (including but not limited to sepsis) in hospitalized children ^90,91. Thresholds are set and points assigned for behavioral, respiratory, and cardiovascular compromise. An RCT of PEWS compared to standard care, performed across multiple centers in Canada, found no mortality reduction but fewer significant clinical deterioration events⁹². In a retrospective single-center review of 72 hospitalized children that required emergent transfer to the ICU (mostly for respiratory distress), a barrier to earlier identification of patient deterioration was lack of continuous telemetry or pulse oximetry monitoring in about one-third of cases.⁹³ Furthermore, in that study PEWS was calculated incorrectly in nearly half of cases in the 24h period before ICU transfer. These studies highlight the potential benefits of continuous rather than intermittent vital sign monitoring for high risk non-ICU patients, as well as the importance of having a system for correct input of data into EWS algorithms.

Several EWS scores for adult patients have been implemented, including the National Early Warning Score (NEWS)⁹⁴ in the United Kingdom and the Modified Early Warning Score (MEWS)⁹⁵. Like PEWS, these scores track patient data and trigger an alert based on points assigned by thresholds. In contrast, a recently described Targeted Real-time Early Warning System (TREWS) uses a machine learning model, rather than threshold-based points, to predict septic shock⁹⁶. TREWS is calculated using 27 vital signs and laboratory inputs from the EHR. A comparison of TREWS versus MEWS for predicting septic shock in hospitalized adults showed improved performance, demonstrated by AUC of 0.83 compared to 0.73. In a recent report, adoption of TREWS was reported to be high, with 89% of alerts linked to a clinician response, and about one-third of alerts associated with septic shock⁹⁷. AI models would be expected to perform better than EWS because they can use the data as continuous rather than categorical values determined by thresholds. Additionally, modeling, rather than empirically derived cutoffs, can detect more subtle and complex patterns in the data associated with the target outcome.

One ML model for adults, the Advance Alert Monitor⁹⁸, uses EHR data to predict clinical deterioration (not limited to sepsis) in hospitalized patients. This system sends an alert to a centralized team of specially trained nurses and was tested using a staggered implementation clinical trial design in 21 hospitals⁹⁹. For about 15,000 patients at hospitals using the system, mortality was 16% lower compared to about 30,000 similar patients at hospitals not using the system. Another consideration is that ML models that display illness trajectories over time are useful for determining whether a patient’s condition is improving or worsening. One such system, Continuous Monitoring of Event Trajectories (CoMET), analyzes vital sign data to predict clinical deterioration and is undergoing a cluster-randomized clinical trial to determine impact on important clinical outcomes¹⁰⁰.

ML models have been developed and implemented for pediatric sepsis. In a study of children 2–17 years of age presenting to an Emergency Department, a model developed using boosted ensemble decision trees incorporated hourly vital signs and (if available) platelet count, white blood cell count, and Glascow Coma Scale. The model was trained on data from nearly 10,000 patient encounters to detect severe sepsis which occurred in 101 patients, and had AUC of 0.916 at the time of diagnosis and (more importantly) AUC 0.718 four hours prior to clinical diagnosis¹⁰¹. Another group used data from 1,425 PICU patients with 187 cases of sepsis to develop a sepsis ML model incorporating continuous vital sign data, 11 laboratory values, and 4 clinical variables (age, sex, central line and mechanical ventilation)¹⁰². Both logistic regression and random forest modeling were tested and had similar AUC, but the latter captured a relationship between age and sepsis better.

Neonatal sepsis AI

A model for predicting EOS has now been widely implemented as a web-based calculator ^9,103,104. The model was developed from a large dataset generated by a network of newborn hospitals, the Northern California Kaiser Permanente System and deployed as an online calculator. It uses perinatal risk factors known at the time of birth in a logistic regression model to derive prior probability and then incorporates the clinicians’ assessment (asymptomatic, equivocal, or clinically ill) using Bayes’ theorem. The risk per 1,000 live births is displayed for each of three categories of illness^9,103. Clinical decision support is provided, allowing for clinical judgment to guide the application of the AI technology, which is likely a factor in the widespread adoption of this model. Studies of the impact of the EOS calculator have shown it reduces the number of asymptomatic or equivocal infants with sepsis risk factors undergoing laboratory evaluations and exposure to antibiotics.¹⁰³

A number of models for prediction of LOS in NICU patients before they are obviously sick have been published¹¹³. Table 1 summarizes models using continuous or intermittently sampled data to make predictions about imminent sepsis, with the goal of early detection and treatment which may improve outcomes^{10,88,107,111,112}. Several other studies have used data at the time of blood culture to predict whether sepsis will be ruled in or out (positive versus negative culture) which may be useful for determining when to start and stop antibiotics^114,115. And finally, several studies have used vital sign data shortly after birth to predict risk of developing sepsis later in the NICU course, which might identify highest risk infants in need of enhanced vigilance or preventive strategies not suitable for the entire preterm population^116,117.

Table 1.

A summary of select studies reporting the development and performance of machine learning (ML) models to predict imminent late-onset sepsis.

First author (Model name)	Population	Late-onset sepsis cases used for modeling	Data Input	Sepsis training window (hours)	External validation	Modeling method
Griffin et al. (HRC index) 7,74,89	VLBW	culture positive and clinical	ECG heart rate characteristics (low variability, decels)	−24 to 0	Yes	Logistic regression
Gur et al. (RALIS) 67,90,91	VLBW	culture positive and clinical	HR, RR, temp, weight, desaturation, bradycardia episodes	Not reported	Not reported	Not specified
Joshi, et al.60; Cabrera-Quiros, et al.92	<32 weeks GA	culture positive	4 feature subsets: (1) HRV, (2) movement, (3) respiration, and (4) combination of above	−24 to 0 and −3 to 0	No	Logistic regression, naive Bayes, nearest mean classifier
Mani et al.69	All NICU patients	culture positive and clinical	EHR data: 71 laboratory 30 demographic + clinical variables	−48 to +12	No	9 ML models
Masino et al. 93	All NICU patients	culture positive and clinical	36 features from EHR, including VS and clinical status	−48 to −4	No	8 ML models
Song et al.68	All NICU patients	culture positive and clinical	Vital signs, blood gas data, CBC	−24 to 0	No	8 ML models

To date, the only commercially available system for NICU predictive monitoring using continuous bedside monitor data is the HRC index, or HeRO Score. This is also the only ML model for neonatal sepsis that has been tested in a randomized clinical trial and shown to improve important clinical outcomes¹⁰⁵. The HRC algorithm uses electrocardiogram data from standard NICU bedside monitors to calculate the fold-increased risk of a clinical deterioration due to sepsis (culture-proven) or a sepsis-like illness (clinical sepsis) in the next 24 hours. The algorithm uses mathematical calculations that report on decreased HR variability and transient HR decelerations, patterns shown in pre-clinical models to reflect pathogen-induced inflammatory cytokine release and vagus nerve firing^55,118. In a randomized clinical trial of 3003 VLBW infants at 9 NICUs¹⁰⁵, display of this risk score was associated with significantly lower sepsis-associated mortality(12% versus 20%), presumably due to earlier treatment¹¹⁹. Importantly, display of the score was associated with only a small increase in the number of blood cultures and antibiotic days, perhaps indicating that clinicians also used the score for its negative predictive value to decide not to start antibiotics or to discontinue antibiotics in some patients with non-specific, mild clinical signs.

The HRC index provides an example of AI development and translation from the bench to bedside, and it uses relatively limited input data. Adding respiratory analytics to heart rate characteristics might lead to better prediction of LOS during the pre-clinical (non-obvious) phase of infection. The major sign of sepsis in preterm infants is apnea, which may manifest as spells of bradycardia and oxygen desaturation. High or increasing cross-correlation of HR and SpO2, reflecting concurrent transient decelerations and desaturations, was shown to be the best predictor of impending sepsis or necrotizing enterocolitis in a 2-NICU study of more than 1,000 VLBW infants²². In a 3-NICU study of LOS compared with sepsis ruled out, cross-correlation HR-SpO2 increased more for LOS cases, and combining vital sign data with clinical data improved discrimination and classification of events as sepsis ruled-in or ruled-out⁸⁶. Another group developed a ML model incorporating HR, SpO2, and RR features as well as ECG waveform analysis to detect decreased movement that may occur in sepsis.⁸⁸ These examples represent the power of combining domain expertise of neonatologists, mathematicians, engineers, and data scientists to optimize sepsis AI.

Q5: What are some benefits and barriers for sepsis ML model implementation and clinical integration?

Much has been written recently about the potential benefits of AI implementation in healthcare¹²⁰. On the cautionary side, hype about “AI solutions” implies there is a relatively simple way to implement AI and improve outcomes of complex medical problems, and this is not the case. Even optimal AI models will not replace the hard work of clinicians deciding which tests to order and which therapies to administer. ¹²¹ That said, with regard to sepsis the obvious benefit of implementing an automated prediction system is earlier antibiotic treatment and supportive care leading to improved outcomes. The 20% reduction in sepsis-associated mortality with continuous HRC index display in the HeRO RCT is an example. For survivors of neonatal sepsis, earlier treatment may not improve neurodevelopmental outcomes ¹²² but might reduce NICU length of stay ¹²³.

Beyond direct patient benefits, other potential benefits of using AI risk models for sepsis include resource allocation and risk stratification, which can be useful for cost-effectiveness analyses, classification for research, and benchmarking across hospitals. Also, AI might reduce bias in healthcare, but only if care is taken to avoid introducing biased data into algorithms. An example of racial bias in a NICU algorithm was recently discovered in a risk calculator for bronchopulmonary dysplasia. The algorithm did not account for the competing outcome of death, which was higher among Black infants leading to a lower risk of being diagnosed with BPD. This might create inequity in clinical care, since the calculator has been promoted as a decision support tool for administering glucocorticoids to reduce the severity of chronic lung disease. A first step in addressing bias in AI is to develop and test algorithms for performance across the spectrum of patient sex, race, ethnicity, and socioeconomic status. With regard to sepsis, a potential advantage of physiology-based algorithms is that heart rate patterns of neonates tend to be similar across the spectrum of patient diversity. Adding pulse oximetry data to heart rate characteristics will need close scrutiny since racial differences in accuracy of pulse oximetry data have recently been described in adults¹²⁴, children¹²⁵, and neonates¹²⁶. Regardless of what sources of data serve as model input, AI algorithms should be developed and validated in large, diverse patient populations with efforts made to minimize bias of all types.

Although there are many potential benefits of AI, there are also many barriers. We developed the acronym “BARRIERS” to summarize some major challenges in this field: Babies, Analytics, Reactors, Reassurance, Integration, Equipment, Reeducation, and Space¹²⁷. Babies themselves can complicate the development and deployment of early warning systems for sepsis since they cannot announce that they feel sick, and their signs of sepsis are non-specific and overlap with normal preterm physiology. This creates the problem of false alarms in a unit already prone to alarm fatigue^128,129. Another barrier, “Analytics,” refers to the difficulty in creating models due to heterogeneity of event identification, variable selection, and modeling techniques, as previously discussed in this review. “Reactors” are the model users, NICU clinicians with varied education, experience, and responsibilities. The barrier, in this case, is the difficulty in displaying data and clinical decision support in a way that is effective for a broad range of clinicians. “Reassurance” can be a problem with AI models if a low-risk score falsely reassures the clinical team faced with an infant with significant signs of illness, leading to a delay in treatment. “Integration” refers to the challenge of introducing an AI model without creating too many distracting false alarms. One way to mitigate alarm fatigue yet assure that critical information is transmitted to the right person is to have a centralized clinical team that reviews alerts and determines which ones should be transmitted to the care team⁹⁹, an approach that may not be broadly feasible. The “E” in BARRIERS is equipment that must be integrated into the clinical workflow. Once the system is integrated, education and “Re-education” for users are critically important to assure proper implementation. And finally, “Space” can be a barrier since the NICU bedside may already be crowded with equipment and monitors. A new sepsis prediction system needs to be positioned in such a way to be noticeable but not overwhelming.

In the case of an AI system that is shown to improve patient outcomes, implementation relies on hospital administrators and clinicians “buying in.” A survey-based study of continuous predictive monitoring reported that users had positive engagement with the system if they trusted the data used in the model and if they understood the science behind the model outputs¹³⁰. This is the basis for the term “explainable AI” which some view as essential for clinicians to utilize the system, although others argue that methods to make models explainable sacrifice their performance¹³¹. A final consideration is that AI systems may introduce unintended consequences such as inappropriate testing and therapies. Other sources of harm if AI algorithms are not appropriately developed and utilized is the creation of distractions which can lead to medical errors and contribute to burnout which is an increasing problem among healthcare workers.

CONCLUSION

Sepsis AI is a way to analyze and present data to clinicians for earlier detection and treatment leading to improved patient outcomes. If properly developed and implemented, AI systems can alert clinicians to a change in a patient’s condition that warrants a bedside evaluation. At that point, human intelligence (and experience) kick in. Initiating a workup and antibiotics for all patients with a bad score would constitute malpractice. Rather, clinicians should consider the AI result and then incorporate what they see and what they know to make the best decisions for individual patients.

Impact Statement:

• This narrative review highlights the application of AI in neonatal sepsis prediction. It describes issues in clinical prediction model development specific to this population.

• This article reviews the methods, considerations, and literature on neonatal sepsis model development and validation.

• Challenges of AI technology and potential barriers to using sepsis AI systems in the NICU are discussed.