Sensitivity of the Kaiser Permanente early-onset sepsis calculator: A systematic review and meta-analysis

Summary

Background

Determining which babies should receive antibiotics for potential early onset sepsis (EOS) is challenging. We performed a meta-analysis quantifying how many EOS cases might be ‘missed’ using the Kaiser Permanente electronic calculator, compared with National Institute for Health and Care Excellence (NICE) guidelines.

Methods

A systematic literature search was carried out for studies citing the article in which the calculator was publicised. Studies were eligible if they presented data evaluating the calculator, either by retrospective case review or prospective cohort study. The primary outcome measure was numbers of culture positive EOS cases where the calculator did not recommend empirical antibiotics, but NICE guidelines would have. Data were pooled using a random effect meta-analysis. A subgroup analysis was performed using data from studies of babies exposed to chorioamnionitis.

Findings

Eleven studies were included. There were a total of 75 EOS cases across the studies and a minimum of 14 (best case scenario), and a maximum of 22 (worst case scenario) cases where use of the calculator would have resulted in delayed or missed treatment, compared to if NICE guidelines had been followed. The probability of missed/delayed treatment for an EOS case were best case 0.19 [95% confidence intervals 0.11 – 0.29], worst case 0.31 [95% CI 0.17 – 0.49]. The probability of missing cases was significantly more in babies exposed to chorioamnionitis

Interpretation

A large proportion of EOS cases were ‘missed’ by the calculator. Further evaluation of the calculator is recommended before it is introduced into UK clinical practice.

Funding

None.

Research in context.

Evidence before this study

Early onset sepsis (EOS) is a rare but significant cause of mortality and morbidity in neonates. Current guidelines lead to large numbers of well babies receiving antibiotics. The Kaiser-Permanente calculator has been shown to reduce antibiotic usage in several international settings.

We carried out a systematic review using a modified cluster technique, snowballing from studies citing the 2017 JAMA pediatrics paper describing the calculator and its implementation. Citing articles were sought on Ovid MEDLINE(R) and Epub Ahead of Print, In-Process & Other Non-Indexed Citations, Daily and Versions(R) 1946 to July 23, 2019, Embase 1996 to 2019 Week 29 and Maternity & Infant Care Database (MIDIRS) 1971 to May 2019, and Google Scholar with the “Since 2017” filter on 24/07/2019. Risk of bias was assessed using QUADAS-2 and was low/moderate for all included articles.

Added value of this study

The probability of missed or delayed treatment for an EOS case (in addition to cases missed by NICE guidelines) were at best 0.19 [95% confidence intervals 0.11–0.29, I² 0%], worst case 0.31 [95% confidence intervals 0.17–0.49, I² 37%]. Amongst a subset of babies exposed to chorioamnionitis, the calculator appears more likely to miss cases. The number of cases of EOS on which the calculator has been tested has not reached the threshold for effective external validation.

Implications of all the available evidence

Whilst antibiotic use was reduced in all included studies, a large proportion of EOS cases are ‘missed’ by the calculator, additional to cases which would be missed by current guidelines.

Alt-text: Unlabelled box

1. Introduction

Early onset neonatal sepsis (EOS) is defined as infection occurring within 72 h of birth. Babies with EOS may be asymptomatic initially, whilst non-infective pathology can mimic EOS; determining who should receive antibiotics is challenging. In the UK, babies are treated based on risk factors and clinical indicators (see Tables 1 and 2) as defined by the National Institute for Health and Care Excellence (NICE) [1].

Table 1.

Risk factors for EOS defined by NICE (1).

•Invasive group B streptococcal (GBS) infection in a previous baby •Maternal GBS colonisation, bacteriuria or infection in current pregnancy •Prelabour rupture of membranes •Preterm birth following spontaneous labour •Rupture of membranes for > 18 h in a preterm birth •Intrapartum fever > 38 °C, or confirmed or suspected chorioamnionitis •Parenteral antibiotic treatment given to the woman (red flag) •Suspected or confirmed infection in another baby in the case of a multiple pregnancy (red flag)

Table 2.

Clinical indicators of possible EOS described by NICE (1).

•Altered behaviour or responsiveness •Altered muscle tone (e.g. floppiness) •Feeding difficulties / intolerance •Abnormal heart rate •Signs of respiratory distress •Respiratory distress starting > 4 h after birth (red flag) •Hypoxia •Jaundice within 24 hrs after birth •Apnoea •Encephalopathy •Seizures (red flag) •Need for CPR •Mechanical ventilation in a term baby (red flag) •Persistent pulmonary hypertension •Temperature abnormality •Signs of shock (red flag) •Bleeding, thrombocytopenia, or abnormal coagulation •Oliguria •Altered glucose homeostasis •Metabolic acidosis •Local signs of infection

Neonatal antibiotic administration has been reported to be associated with pain, parental anxiety and separation, and in childhood: asthma, allergy and autoimmune disease [2], as well as presenting significant workload and financial cost.

An electronic risk calculator has been developed by Kaiser Permanente (KP), for babies born ≥ 34 weeks gestation, to aid decision making when considering EOS. It is available online: https://neonatalsepsiscalculator.kaiserpermanente.org/InfectionProbabilityCalculator.aspx.

In a high-profile paper published in JAMA Pediatrics it was shown to reduce empirical antibiotic use substantially [3]. The calculator contains the following fields: background EOS incidence, gestational age, highest maternal antepartum temperature, rupture of membranes in hours, maternal GBS status and type/timing of intrapartum antibiotics. Clinical presentation is factored in, which adjusts the ‘at birth’ recommendation given by the calculator. Empirical antibiotics are recommended when the risk is ≥ 3/1000 births, or a blood culture and observation if the risk is ≥ 1/1000 live births.

The use of a calculator as part of a strategy of managing neonatal EOS adds an objective element to the decision to give or withhold antibiotics. It is vital that the calculator is thoroughly evaluated; it provides an exciting opportunity to provide tailored treatment, but because EOS is rare, it could take substantial time before any additional ‘missed’ cases of EOS became apparent. In addition, we should be cautious when extrapolating results between countries, because of differences in the EOS incidence and in healthcare practices (e.g. national screening programmes for GBS). Given the impressive reduction in antibiotic usage which has been demonstrated [3], we wanted to investigate whether this might be at the expense of delaying or missing treatment for some babies with true sepsis.

The aim of this meta-analysis was to pool all the data for available studies of the calculator, to determine the proportion of babies with culture proven EOS which would have been treated using NICE guidelines but were not identified as requiring empiric antibiotics by the calculator.

2. Methods

2.1. Search strategy and selection criteria

We carried out a systematic review and meta-analysis of the sensitivity of the KP EOS calculator, in studies where it was either implemented or compared theoretically to standard practice.

2.1.1. Eligibility criteria

Any studies using the calculator in babies which reported the number of cases of EOS in the study period were eligible for inclusion. No studies prior to 2011 (when the calculator was first described [4]) were included. Papers where there were no cases of EOS in the study period were excluded, as uninformative to the sensitivity calculation. Individual patient level data was sought. Review articles and commentaries, where no new data were presented, were excluded. Only one report of each cohort was included. No language limits were applied.

2.1.2. Outcome measures

The proportion of cases of EOS which would have been ‘missed’, or where treatment would have been delayed by using the calculator, were compared to NICE guidelines. We considered any baby with positive blood cultures and an EOS risk score of <3/1000 as being a ‘miss’ only if the NICE guidelines would have recommended treatment (as per Tables 1 and 2). Any cases where treatment was not recommended by the calculator, but this was concordant with NICE guidelines, were not classified as a ‘miss’.

2.1.3. Search

The search was carried out by two authors independently (KJP/KM), using a modified cluster technique, snowballing from studies citing the 2017 JAMA pediatrics paper [3]. Citing articles were sought on Ovid MEDLINE(R) and Epub Ahead of Print, In-Process & Other Non-Indexed Citations, Daily and Versions(R) 1946 to July 23, 2019, Embase 1996 to 2019 Week 29 and Maternity & Infant Care Database (MIDIRS) 1971 to May 2019, and Google scholar with the “Since 2017” filter on 24/07/2019. Reference lists of included articles were reviewed, and relevant articles included. To ensure a complete dataset, we searched for any papers that might have been published by the developers of the tool without citing Kuzniewicz et al.’s 2017 JAMA pediatrics paper. The calculator's origins are described by Puopolo et al. in a 2011 paper [4]. We searched Medline, MEDIRS and Embase for each author's publications since 2011 for any additional papers. Posters presented at conferences were included where possible. Clinicians known to be using the calculator were contacted and asked to share data.

2.1.4. Study selection

Once duplicates were removed, the abstracts of all the articles from the search were reviewed independently by two authors (KP and KM), discussing any differences. A third was available for resolution, if required.

2.1.5. Data analysis

The number of EOS cases in each study period and the number of those which were ‘missed’ or where treatment was delayed when compared to NICE guidelines (as per Tables 1 and 2) was recorded, using the study authors’ calculation of risk based on their local baseline rate of infection (note that this varied between studies). Where there was uncertainty about whether a case would be ‘missed’ or not, the authors were contacted for further details and clarification. A sensitivity analysis was undertaken.

2.1.6. Data items

Where possible, we collected the following data from included studies:

• •Number of babies in the sample
• •Location
• •Date of data collection
• •Organism grown
• •Clinical condition at birth (well, equivocal or ill by KP standards)
• •Gestational age
• •Highest maternal temperature
• •Maternal GBS status
• •Duration of membrane rupture
• •Maternal antibiotics
• •EOS risk at birth and after examinationRisk of bias across studies

The risk of bias across studies (“publication bias”) was not statistically assessed.

2.1.7. Risk of bias in individual studies

The risk of bias was assessed for each included study using QUADAS-2 risk of bias assessment tool, see Table 3 and Supplementary table S2.

Table 3.

Summary of included articles.

	Location	Study type	Dates	Population	EOS data	Antibiotic usage	Risk of bias as per QUADAS-2
Kuzniewicz et al. (2017)	California, USA	Prospective single-centre cohort	2010 – 2015	Baseline period: 95,543 babies Calculator period: 56,261 babies ≥35 weeks gestation	Epoch 1: 24 cases of EOS 4 missed by calculator. Epoch 2: 15 cases of EOS 2 missed by the calculator, plus 4 additional possible mises Epoch 3: 12 cases of EOS 2 missed by calculator as confirmed following d/w Dr Kuzniewicz	Antibiotic use reduced from 5.0% to 2.6%	low
Dhudasia et al. (2018)	Philadelphia, USA	‘retrospective’ single-centre cohort	Epoch 1: March 2014-May 2015 Epoch 2: July 2015 - October 2016	Epoch 1: 5692 babies Epoch 2: 6090 babies ≥36 weeks gestation	4 cases of EOS: 1 × GBS, 3 × E. coli maximum 3 misses, minimum 1 miss Case 1 E. coli - maternal temp 38.8C, on parenteral antibiotics, initial risk score: 1.54, after examination: 7.66 - could have been a delay in treatment, depending on how old baby was when symptoms developed F = possible miss Case 2 GBS – score at birth 0.3. Tachypnoeic at 36hours which is a NICE red flag for antibiotics, therefore – miss. Case 4 E. coli - unclear if mother was treated for chorioamnionitis – maternal temp 39C. Score at birth: 3.41 but was ‘well’; score after examination: 1.40 = possible miss note the authors used different thresholds: we used the thresholds in Kuzniewicz 2017 for consistency	Antibiotic use reduced from 6.3% to 3.7%	low
Strunk et al. (2018)	Subiaco, Western Australia	Prospective, single-centre cohort	Epoch1: Oct 2014 - January 2015 Epoch 2: July -December 2016	Epoch1: 1731 babies Epoch 2: 2502 babies ≥35 weeks gestation	2 cases of EOS 1 case in each epoch, both were clinically unwell requiring admission to the neonatal unit. The calculator recommended empirical treatment in both cases, therefore, no missed cases	Antibiotic use reduced from 12.0% to 7.6%	low
Goel et al. (2019)	Wales, UK	Prospective cohort (EOS calculator results not acted upon – NICE guidelines followed)	February–April 2018	3593 babies ≥34 weeks gestation	7 positive blood cultures, 2 contaminants (Staph. aureus and alpha-haemolytic streptococci) 5 cases of EOS 3 × cases picked up by both NICE and calculator 2 × missed by both, therefore no misses compared to NICE	Antibiotic use would have been reduced from 16% to 4.3%, 74% relative reduction	low
Arora et al. (2019)	Illinois, USA	Prospective, single-centre cohort	August 2016 –September 2017	276 babies ≥34 weeks gestation admitted to neonatal unit	1 case of EOS Treated with empirical antibiotics according to calculator recommendations, therefore no misses	Antibiotic use reduced from 70.3% to 49.6%	low
Stipelman et al. (2019)	Utah, USA	Single centre, prospective quality improvement programme	June 2014 – December 2017	11,924 babies admitted to a newborn nursery unclear gestation 187 infants were excluded because their treatment was decided by NICU clinicians who had not received the quality improvement training.	3 cases of EOS, 1 × GBS, 1 × Enterococcus faecalis, 1 × E. coli 2 misses: Case 1: GBS score at birth: 0.09; well: 0.04, equivocal: 0.43, ill: 1.83 Treated because became clinically ill at 6 h. This is a miss –it fits NICE criteria for treatment based on clinical indicators but scored <3. Case 2: Enterococcus faecalis score at birth: 0.25; Maternal chorioamnionitis and prolonged rupture of membranes. Would most likely have been treated from birth using NICE guidelines.	Antibiotic use reduced from 7% to 1%	low
Shakib et al. (2015)	Utah, USA	Single centre, retrospective record review	2006 −2013	698 babies ≥34 weeks gestation born to mothers with chorioamnionitis	6 positive blood cultures, 5 contaminants (4x CoNS, 1 × micrococcus) 1 case of EOS: GBS No misses - EOS risk score for the GBS case was 7.85 per 1000	Antibiotic use would have been reduced to 12%	Low-moderate
Money et al. (2017)	Staten Island, NY, USA	Single centre, retrospective record review	2009 – 2016	362 babies “term” babies, unwell babies excluded born to mothers with chorioamnionitis	3 positive blood cultures, 2 contaminants (1 × CoNS, 1 × Actinomyces odontolyticus) 1 case of EOS: Enterococcus Enterococcus: highest maternal temp 38.1 °C, 20 h ROM, chorioamnionitis exposed. EOS risk score was 0.77/1000 = 1 × miss	Antibiotic use would have been reduced from 99.7% to 2.5%	Low-moderate
Carola et al. (2018)	Philadelphia, USA	Single centre, retrospective record review	2006 – 2017	896 babies ≥35 weeks gestation born to mothers with chorioamnionitis	5 cases of EOS 1 × GBS, 1 × E. coli, 3 × ‘other’ Streptococci minimum 1 miss, maximum 3. E. coli: at birth score: 2.39, after examination: 0.98, therefore miss GBS: at birth score: 4.02, after examination: 1.65, possible miss, depending which score was used Strep. sanguinis: at birth score 2.03, after examination 41.34, possible miss, depending which score was used	Antibiotic use would have been reduced to 23.5%	Low-moderate
Sharma et al. (2019)	Minneapolis, USA	Single-centre prospective cohort study	Pre implementation January - December 2015 post implementation: April 2016 - March 2018	epoch 1: 109 babies epoch 2: 180 babies ≥36 weeks gestation born to mothers with chorioamnionitis	1 case of EOS Treated with empirical antibiotics according to calculator recommendations, therefore no misses	Antibiotic use reduced from 40% to 23%	low
Joshi et al. (2019)	Stanford, USA	Single centre, retrospective record/chart review, as part of a separate quality improvement project	2015–2017	339 babies ≥34 weeks’ gestation born to mothers with chorioamnionitis	1 case of EOS: GBS Highest maternal temp 38.3 °C, known maternal GBS carriage but untreated, chorioamnionitis exposed. EOS score at birth was 0.93 per 1000, therefore= 1 × miss	Antibiotic use reduced by 12.3% to 5.1%, but this was not achieved by using the calculator	Low-moderate

2.1.8. Subgroup analyses

There has been discussion in the literature about the use of the calculator amongst higher-risk groups of babies, specifically, babies exposed to chorioamnionitis [5,6]. We therefore performed a subgroup analysis of all papers focussed on chorioamnionitis exposed babies.

2.1.9. Sensitivity analyses

Determining which cases were missed by the calculator compared to NICE guidelines, presented some challenges. Maternal fever may prompt a clinical team to commence parenteral antibiotics for possible maternal sepsis/chorioamnionitis. Parenteral antibiotics given up to 24 h after birth are a red flag for neonatal sepsis, but maternal fever in isolation is not an indicator for neonatal antibiotics [1]. If papers did not specify whether a mother was being treated for possible sepsis or chorioamnionitis, it was not possible to say whether the baby would have been treated according to NICE guidelines. Dr Kuzniewicz advised that when in doubt, they would classify a high maternal temperature as potential sepsis, in order to give a conservative evaluation of the calculator.

For completion, we performed two meta-analyses, a ‘worst case’ scenario where uncertain cases were recorded as a ‘miss’, and a ‘best case’ scenario where the uncertain cases were treated as if they were not missed.

2.1.10. Summary measures

The principle summary measure is the proportion of delayed/missed treatment of EOS using the calculator compared to NICE guidelines.

2.1.11. Synthesis of results

The pooled proportion of delayed/missed cases was estimated by a binomial-normal model (rma.glmm) of logit transformed proportions in the metafor package of R. The effect of being in a whole-population cohort vs. a high-infection risk cohort was evaluated as a moderator variable. Heterogeneity between studies was assessed describing I².

3. Results

3.1. Study selection

The initial literature search produced 144 papers, see Fig. 1. Duplicates were removed leaving 90 articles. The Kuzniewicz 2017 article, two articles found on reviewing reference lists and a conference poster were added resulting in 94 papers which were reviewed in detail. No relevant additional papers were found by searching for publications from the authors of Puopolo 2011 [4]. 15 studies were reviewed in depth. One study was excluded because there were no cases of EOS [7]; a further was excluded as no information on the number of EOS cases was provided [8]. One study was excluded as it was not possible to extract individual patient level data [9]. A further study was excluded as it was not possible to determine whether the participants were started on antibiotics due to strict adherence to guidelines [10]. 11 studies met the inclusion criteria.

Fig. 1.

Study selection.

3.2. Study characteristics

The studies are summarised in Table 3. The largest is Kuzniewicz 2017 [3], where the calculator was introduced into practice in place of national guidance. Dhudasia 2018 [11], Strunk 2018 [12] and Arora 2019 [13] are cohort studies where the calculator was formally introduced as standard clinical practice and antibiotic use before and after introduction compared. Sharma 2019 [14] is similar; the calculator was introduced prospectively into clinical practice, but the paper relates specifically to babies exposed to chorioamnionitis. Due to insufficient data availability, we had to exclude the pre-intervention group in Arora 2019 [13] because we were unable to determine which cases would have been missed.

Carola 2018 [15], Shakib 2015 [16] and Money 2017 [17] were comparing retrospectively and theoretically how many babies would have been recommended treatment in a subset of babies born to mothers with confirmed or suspected chorioamnionitis.

Joshi 2019 was a quality improvement project mainly relating to serial physical examinations as an alternative method to manage babies exposed to chorioamnionitis [18]. It was eligible for inclusion because the authors had compared their results to those which might have been achieved using the calculator and presented data on the number of EOS cases.

Goel 2019 was a prospective cohort study comparing the calculator against NICE guidelines. All babies were managed according to NICE guidelines, and the calculator recommendations were not acted upon [19].

Stipelman 2019 was a 2 phased quality improvement project, designed to improve uptake of the calculator in their centre [20]. There was sufficient data to incorporate it into the meta-analysis.

3.3. Results of individual studies

See and Supplementary Table S1 – data extracted from articles, excluding contaminants.

3.4. Risk of bias across studies

When blood culture results are known, i.e. in retrospective studies, it is possible that the clinical condition of the baby at birth may be recorded differently in the light of this, introducing bias. This is a potential issue for Joshi 2019, Shakib 2015, Money 2017 and Carola 2018 which evaluated the calculator retrospectively (using contemporaneous notes). Kuzniewicz 2017, Dhudasia 2018, Strunk 2018, Sharma 2019 and Arora 2019 had retrospective comparison groups which could also have been affected.

Kuzniewicz 2017, Dhudasia 2018, Strunk 2018, Sharma 2019, and Arora 2019 were using the calculator prospectively and were at risk of a different form of bias, that is, their retrospective interpretation of the national guidelines. For example, a baby born to a febrile mother, who has a low EOS score, but goes on to become septic. According to the calculator the baby would not receive immediate antibiotics. However, a clinician using national guidelines could have been more conservative and treated the baby empirically in case of possible maternal sepsis. The interpretation of such findings is not clearly described and may be a source of potential bias.

Dhudasia 2018 is the only paper which explains exactly how the “at birth” and “after examination” scores were acted upon; nursing staff calculated the risk at birth for every baby, and if it was > 0.7/1000 then the paediatrician was alerted, and the baby examined.

No clear evidence of publication bias was noted.

3.5. Synthesis of results

Across the studies there were 75 cases of culture proven EOS. There was a minimum of 14 (best case scenario), and a maximum of 22 (worst case scenario), cases of EOS where the calculator did not recommend empirical antibiotics, i.e. they would have been/were initially ‘missed’ compared to NICE guidelines, see supplementary table S3.

The pooled probability of missing a case of EOS, which would not have been missed by the NICE guidelines, were best case 0.19 [95% confidence interval (CI) 0.11–0.29, I² 0%] (see Fig. 2), worst case 0.31 [95% CI 0.17–0.49, I² 37%] (see Fig. 3). Exclusion of the papers focused on chorioamnionitis exposed babies [14], [15], [16], [17], [18] did not alter these estimates meaningfully (0.17 vs 0.19, and 0.26 vs 0.31 for the ‘best’ and ‘worst’ case analyses, see Fig. 4, Fig. 5). Studies which retrospectively evaluated the calculator in babies exposed to chorioamnionitis showed the calculator performed less well: best case scenario 0.33 [95% CI 0.11 – 0.67, I² 0%] (see Fig. 6), worst case scenario 0.56 [95% CI 0.25 – 0.82, I² 0%] (see Fig. 7) (test for subgroup difference, p = 0.03).

Fig. 2.

Proportion of cases missed by the calculator (additional to any cases missed by NICE), all studies − best case scenario.

Fig. 3.

Proportion of cases missed by the calculator (additional to any cases missed by NICE), all studies − worst case scenario.

Fig. 4.

Proportion of cases missed by the calculator (additional to any cases missed by NICE), whole cohort studies − best case scenario.

Fig. 5.

Proportion of cases missed by the calculator (additional to any cases missed by NICE), whole cohort studies − worst case scenario.

Fig. 6.

Proportion of cases missed by the calculator (additional to any cases missed by NICE), chorioamnionitis exposed babies − best case scenario.

Fig. 7.

Proportion of cases missed by the calculator (additional to any cases missed by NICE), chorioamnionitis exposed babies − worst case scenario.

4. Discussion

This systematic review and meta-analysis has demonstrated that, compared to the NICE guidelines the probability of the calculator missing a case of EOS were best case 0.19 [0.11 – 0.29], worst case: 0.31 [0.17 – 0.49]. Amongst a subset of babies exposed to chorioamnionitis, the calculator appears more likely to miss cases; the probability of missing cases was best case 0.33 [0.11 – 0.67], worst case 0.56 [0.25 – 0.82]. The studies were relatively homogeneous; the patient groups were similar (all neonates <72hours old, all >34 weeks gestation).

Some studies were retrospective, but since they were using contemporaneous notes, there is a limit to the effect this could have had. The intervention, the KP EOS calculator, was the same across the studies. The outcome was well defined and consistent: was the EOS score greater or less than 3/1000 births.

Compared to current guidance, the calculator failed to recommend treatment in at least 14/75 babies with EOS (and possibly up to 22 babies) who would have been treated using NICE guidelines. This is in addition to any babies who will inevitably be missed by the NICE guidelines. The baby with E.coli, for example, in Stipelman 2019 [20] had an isolated temperature of >38 °C, which would not trigger treatment based on NICE guidelines, however the clinician decided to commence antibiotics. The EOS score was <3/1000 and so the calculator would also not have recommended treatment and would have missed this case.

The impressive reduction in the use of antibiotics which can be achieved with the calculator can be seen in Table 3, e.g. from 5% to 2.6% in Kuzniewicz et al. [3], or in babies exposed to chorioamnionitis one paper demonstrated a potential reduction from 99.7% to 2.5% [17]. However, given the low sensitivity we have demonstrated, we believe that the risks of introducing this tool do not outweigh the benefits. Potential delays in antibiotic administration by using the calculator have been highlighted previously, in a letter to JAMA pediatrics by Rajbhandari et al. [21]. This was rebutted by Kuzniewicz and colleagues, explaining that any potential delays in treatment are far outweighed by the dramatic reduction in antibiotics which they have achieved [22].

Difference in microbiology and in healthcare practices between the UK and the USA are significant in this context. In the UK for example, GBS screening is not routine, observation nurseries (as described by Money et al.) are not common, prolonged stays on post-natal wards for babies not receiving antibiotics are unusual, and paediatricians are unlikely to ever encounter the majority of babies, as most care is delivered by midwives. When the EOS calculation is low risk, the calculator recommends “routine vitals”. However, the majority of UK babies would not have routine observations carried out at all, unless there was clinical concern.

Kuzniewicz et al., point out that “if adopting out approach, individual centers must assess local care structures” [3]. If the calculator were to be adopted in the UK, current postnatal care would have to be adapted. For example, if a baby who would have qualified for observations due to one risk factor as per NICE has a low EOS risk score, would they be allowed to go home after 6 h? Clearly, robust local policies would need to be drawn up on exactly how the calculator were to be used.

It is striking that in all the available evidence there were only 75 cases of EOS. Collins et al. demonstrate that effective evaluation of a prediction model would require at least 100 (and possibly 200) external cases, to validate the model [23]. We would suggest this be carried out retrospectively by accumulating a large series of cases of babies evaluated for sepsis (whilst continuing to use NICE guidelines) rather than rolling out the calculator and potentially missing further cases of EOS.

We have not carried out a corresponding meta-analysis to determine the number of misses that NICE guidelines might have compared to the calculator. Given the broad recommendations for antibiotic use in the NICE guidelines, this was felt unlikely to be beneficial. This study evaluates the sensitivity of the calculator compared to NICE guidelines. We have not made any assessment of its specificity, or its safety when implemented with a whole-nursery system of newborn care. This study does not make a comprehensive assessment of the overall performance of the calculator, rather we have assessed its immediate ability to accurately detect babies who go on to have positive blood cultures.

Whether the benefits of reducing antibiotics use outweigh the occasional ‘miss’ or delay is hard to quantify, since it is difficult to estimate the effects of widespread (over)use of antibiotics to individuals and populations. This meta-analysis has demonstrated that a large proportion of true cases of EOS are ‘missed’ by the calculator, in addition to those which would be missed by NICE guidelines. The probability of missing additional cases compared to the NICE guidelines is significantly higher amongst babies exposed to chorioamnionitis.

Given these concerns, further analyses of known cases of EOS is required to determine what proportion would be initially missed or result in delayed treatment. No method for predicting EOS is perfect, and there is no substitute for clinical monitoring, since there will almost inevitably be some babies without risk factors for infection who nevertheless go onto to become septic. Effective evaluation will need approximately 100 cases of EOS to have occurred, a threshold not yet reached in the available literature.

Declaration of competing interest

We declare no competing interests.