Multicentre external validation of the Neonatal Healthcare-associated infectiOn Prediction (NeoHoP) score: a retrospective case–control study

Abstract

Background and objectives

Neonatal mortality due to severe bacterial infections is a pressing global issue, especially in low-middle-income countries (LMICs) with constrained healthcare resources. This study aims to validate the Neonatal Healthcare-associated infectiOn Prediction (NeoHoP) score, designed for LMICs, across diverse neonatal populations.

Methods

Prospective data from three South African neonatal units in the Neonatal Sepsis Observational (NeoOBS) study were analysed. The NeoHoP score, initially developed and validated internally in a South African hospital, was assessed using an external cohort of 573 sepsis episodes in 346 infants, focusing on different birth weight categories. Diagnostic metrics were evaluated, including sensitivity, specificity, positive predictive value and area under the receiver operating characteristic curve.

Results

The external validation cohort displayed higher median birth weight and gestational age compared with the internal validation cohort. A significant proportion were born before reaching healthcare facilities, resulting in increased sepsis evaluation, and diagnosed healthcare-associated infections (HAIs). Gram-negative infections predominated, with fungal infections more common in the external validation cohort.

The NeoHoP score demonstrated robust diagnostic performance, with 92% specificity, 65% sensitivity and a positive likelihood ratio of 7.73. Subgroup analysis for very low birth weight infants produced similar results. The score’s generalisability across diverse neonatal populations was evident, showing comparable performance across different birth weight categories.

Conclusion

This multicentre validation confirms the NeoHoP score as a reliable 'rule-in' test for HAI in neonates, regardless of birth weight. Its potential as a valuable diagnostic tool in LMIC neonatal units addresses a critical gap in neonatal care in low-resource settings.

WHAT IS ALREADY KNOWN ON THIS TOPIC

• Severe bacterial infections contribute significantly to neonatal mortality globally, particularly in low-middle-income countries (LMICs), however, it presents a diagnostic dilemma to healthcare providers.

• Existing research highlights the need for effective tools to predict healthcare-associated infections (HAIs) in neonates, especially in low-resource settings.

WHAT THIS STUDY ADDS

• This study validates the Neonatal Healthcare-associated infectiOn Prediction (NeoHoP) score, originally designed for LMICs, across diverse neonatal populations, demonstrating its robust diagnostic performance.

• By demonstrating the NeoHoP score’s effectiveness in different birth weight categories and diverse settings, this study reinforces its potential as a valuable diagnostic tool for identifying neonatal sepsis in low-resource settings.

HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

• The validation of the NeoHoP score in diverse neonatal populations underscores its utility as a reliable ‘rule-in’ test for HAIs in neonates, regardless of birth weight, potentially influencing clinical practice by aiding in the early identification and management of neonatal sepsis.

• This study may inform policy discussions surrounding neonatal care in LMICs, emphasising the importance of implementing effective diagnostic tools like the NeoHoP score to address the significant burden of neonatal infections and improve outcomes in low-resource settings.

Introduction

Globally, severe bacterial infection is an important cause of neonatal mortality, contributing to up to a quarter of newborn deaths in low-middle-income countries (LMICs).¹ Available data on the impact of neonatal sepsis in sub-Saharan Africa confirm the substantial risk of mortality and serious morbidity, ranging from 9% to 29% in neonatal sepsis episodes.2,5 Furthermore, inadequate healthcare resources in LMICs negatively impact general newborn care, resulting in an increased risk of newborns developing healthcare-associated infections (HAIs).^{6 7} Preterm and very low birthweight (VLBW; <1500 g) infants are especially vulnerable to HAI, owing to underdeveloped immunity, prolonged hospital stay, and frequent use of antibiotics and indwelling devices.⁸

Despite the substantial neonatal sepsis disease burden in LMIC neonatal units, the availability and accuracy of clinical and laboratory diagnostic tests for neonatal HAIs are limited.⁹ Blood culture remains the gold standard for diagnosing neonatal sepsis, but its utility is limited by low diagnostic yield (5%–10%) and a propensity for contamination by skin commensals.^{10 11} Ongoing research is exploring the use of complete blood count indices, procalcitonin, C reactive protein (CRP), interleukin-6 and others, however, the ideal sepsis biomarker has not yet been identified.12,14 Owing to challenges in the laboratory confirmation of neonatal sepsis, infection prediction scores have been developed to predict infection likelihood using clinical signs and symptoms. However, in a performance comparison of existing infection prediction scores at a South African neonatal unit, none achieved sufficient diagnostic accuracy to recommend use in LMICs.¹⁵

The Neonatal Healthcare-associated infectiOn Prediction (NeoHoP) score was subsequently developed and internally validated at a large South African hospital. It is intended for use in hospitalised VLBW infants.¹⁶ This simple score uses four clinical and one laboratory variable with a score of ≥2 being highly specific for the presence of neonatal sepsis, and can be used by the attending physician to guide the decision to perform further special investigations and inform antibiotic decision-making.¹⁶ We present the results of an external validation of the NeoHoP score, using multicentre data collected from the South African study sites included in a global neonatal sepsis observational cohort study (NeoOBS-SA)¹⁷ to determine its predictive performance in other LMIC neonatal units, and in neonates of all birth weight categories.

Materials and methods

Study design and participants

To validate the NeoHoP score, we used prospectively collected clinical and laboratory data from three large South African neonatal units that participated in a global neonatal sepsis observational cohort study (NeoOBS),¹⁷ including 200 neonates hospitalised in each centre (Chris Hani Baragwanath Hospital, Charlotte Maxeke Johannesburg Academic Hospital and Tygerberg Hospital; NeoOBS-SA) between August 2018 and February 2020.

The NeoOBS study enrolled hospitalised infants of any birth weight or gestation, with postnatal age of <60 days, with suspected sepsis episodes, collecting daily observations of clinical signs and symptoms of possible infection, vital signs, supportive care, antibiotic treatment, laboratory indices (complete blood count, CRP and microbiology specimens) and 28-day mortality.¹⁷ Infants were eligible for inclusion if the local physician commenced antibiotic treatment for a new episode of neonatal sepsis meeting the enrolment criteria. These criteria included a minimum of two clinical, or one clinical and one laboratory sepsis criteria based on the WHO¹⁸ and EMA¹⁹ criteria.¹⁷ If the same pathogen was isolated on a repeat culture while the patient was still receiving appropriate antibiotic treatment, it was considered a single infection episode. Multiple sepsis episodes were included if a new antibiotic regimen was started after a blood culture for a new episode of sepsis occurring during the 28-day follow-up period. Infants were excluded if an alternative primary diagnosis was suspected by the treating physician. Data were collected by research and clinical staff based on clinical observation and routine source documentation and entered using REDCap electronic data capture tools.^{20 21}

To derive the patient population for the external validation of the NeoHoP score, the NeoOBS-SA data were interrogated, and the following exclusion criteria were applied to identify patients to be included in the external validation cohort. The exclusion criteria were sepsis investigations performed <72 hours after birth/hospital admission; positive cultures from sites other than blood, cerebrospinal fluid (CSF); positive cultures from the tip of a central venous catheter; repeat investigations within a single infection episode; positive cultures (blood and/or CSF) classified as contaminants by the site clinicians and microbiologists (using site-specific definitions); absence of a CRP result; and incomplete clinical notes and/or absence of any of the score parameters.

Study definitions

The eligible NeoOBS-SA HAI episodes, occurring after 72 hours of life/admission, were classified into two categories based on definitions adapted from those used in the NeoHoP study (table 1).¹⁶ Central line-associated bloodstream infection was defined as a primary bloodstream infection in a patient who had a central line within 48 hours before the development of infection; infection must not be related to an alternative cause.²² Laboratory investigations were considered as related to a single episode of infection if they were performed within 48–72 hours of the original investigation.

Table 1. Definitions used to categorise NeoOBS-SA HAI episodes.

1. AnyHAI
Infections occurring after 72 hours of admission, including the following
a. Proven HAI	Infants with a positive culture isolating a pathogen from a sterile site (blood, cerebrospinal fluid), together with clinical signs and/or symptoms of infection.
b. Central line-associated bloodstream infection (CLABSI)	Laboratory-confirmed bloodstream infection, not related to an infection from another site, that develops 48 hours after the placement of a central line or within 48 hours of its removal.39
c. Presumed HAI	Infants with clinical signs and/or symptoms of infection in the presence of a CRP ≥10 mg/L and negative blood cultures, where antimicrobial treatment was continued for ≥5 days.
2. No HAI
Infants with signs and/or symptoms of possible infection but no objective confirmation of sepsis, ie, negative microbiological cultures, CRP <10 mg/L and discontinuation of antimicrobial therapy within 48–72 hours of initiation.

HAIhealthcare-associated infectionNeoOBSNeonatal Sepsis Observational study

NeoHoP score

The NeoHoP score is a novel infection prediction score developed for the evaluation of suspected neonatal HAI episodes using clinical, management and laboratory variables in a retrospective cohort of VLBW infants in a low-resource setting.¹⁶ It is an easy-to-use score, consisting of five variables (capillary refill time >3 s, lethargy, abdominal distention, presence of a central venous catheter currently or in the preceding 48 hours and laboratory CRP ≥10 mg/L. A score of ≥2 is well positioned to be used as a ‘rule-in’ test.

Statistical analysis and evaluation of NeoHop score

Statistical analysis was performed using IBM SPSS Statistics for MacIntosh, V.27.0, using an α level of 0.05 with a corresponding 95% CI for descriptive statistics. For normally distributed continuous variables, means and SD were calculated. Medians and IQRs were used for non-normally distributed continuous data.

We used a standard 2×2 contingency table to classify the disease status of all cases. Using the reference standard of any HAI, we classified the disease status as ‘true positive’ when the NeoHoP score correctly identified the presence of any HAI (table 1). ‘True negative’ indicates a neonate where the score correctly identified the absence of any HAI. ‘False positive’ refers to neonates where the score falsely identified the presence of any HAI, when the neonate did not have any HAI. ‘False negative’ indicates neonates where the score failed to identify the presence of any HAI.

Diagnostic test evaluation was performed using MedCalc Software, V.20.0.5. The sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV) and likelihood ratios (positive likelihood ratio, PLR; negative likelihood ratio, NLR) were calculated for any HAI episode in the external validation cohort (NeoOBS-SA), as well as for a subset of the NeoOBS-SA data that included all VLBW infants (NeoOBS-SA-VLBW). A good screening test should have a low false-negative rate and thus a high sensitivity.²³ A test with a PLR of >10 or conversely NLR <0.1, is considered a good screening test.^{23 24} The discriminative performance of the score was evaluated by assessing the receiver operating characteristic (ROC) curves and area under ROC curves. The Youden index (J), a summary measure of the ROC curve, was also calculated. J=1 represents a perfect diagnostic test, and J=0 indicates a test that is not effective in determining disease status.²⁵

All findings were reported in accordance with the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE-NI) criteria.²⁶

Patient and public involvement

Patients and/or the public were not involved in this research’s design, conduct, reporting or dissemination plans.

Results

Derivation of the study population for the external validation of the NeoHoP score

In the NeoOBS-SA cohort (n=600 patients) at three tertiary hospital neonatal units, 1766 possible sepsis episodes were investigated (figure 1). A total of 1193 (68.6%) episodes were excluded, leaving 573 sepsis investigation episodes performed in 346 infants suitable for external validation of the NeoHoP score (figure 1). Of the 573 suspected HAI episodes, the majority were confirmed as any HAI (468, 81.7%) including presumed HAI (282, 49.2%) and proven HAI (186, 32.5 %).

Figure 1. Flow diagram of HAI episodes investigated in the NeoOBS study with participants enrolled from three South African neonatal units. CRP, C reactive protein; HAI, healthcare-associated infection; NeoOBS, Neonatal Sepsis Observational study.

Neonatal demographic profile

Neonates in the external validation cohort (NeoOBS-SA) had higher median birth weight (1270 g vs 1010 g) and gestational age (30 weeks vs 28 weeks) than those in the internal validation cohort, with both differences being statistically significant (p<0.001) (see table 2). Neonates in the external validation cohort (NeoOBS-SA) were more likely to be born before arrival at the healthcare facility when compared with the internal validation cohort. The proportion of neonates undergoing sepsis evaluation, where any HAI was diagnosed, was significantly higher in the external validation cohort than the internal validation cohort. Pathogen distribution was similar in the cohorts with Gram-negative infections predominating; fungal infections were more common in the external validation cohort (online supplemental table 1).

Table 2. Characteristics of the external validation cohort (NeoOBS-SA study) compared with the internal validation cohort (NeoHoP study)¹⁶.

	External validation cohort		Internal validation cohort		Univariate p value
Baseline characteristics
n	346		406
Gestational age (weeks), median (IQR)	30	(28–34)	28	(27–30)	<0.001
Birth weight (g), median (IQR)	1270	(990–1845)	1010	(850–1178)	<0.001
VLBW infants, <1500 g, n (%)	222	(64.2)	406	(100.0)	<0.001
Male, n (%)	188	(54.3)	193	(47.5)	0.063
Born in a healthcare facility, n (%)	272	(78.6)	369	(90.9)	<0.001
Vaginal birth, n (%)	142/331	(42.9)	197	(35.6)	0.128
Born to mother living with HIV, n (%)	115/344	(33.4)	120	(29.6)	0.255
Characteristics of HAI episodes
n	573		552
No HAI, n (%)	105	(18.3)	337	(61.1)	<0.001
Any HAI, n (%)	468	(81.7)	215	(38.9)	<0.001
Presumed HAI	282	(60.3)	106	(49.3)	0.008
Proven HAI	186	(39.7)	109	(50.7)	0.008
Gram-negative	100	(53.8)	56	(51.4)	0.157
Gram-positive	41	(22.0)	35	(32.1)	0.056
Fungi	23	(12.4)	2	(1.8)	<0.001
Polymicrobial	22	(11.8)	16	(14.7)	0.481

HAIhealthcare-associated infectionNeoHoPNeonatal Healthcare-associated infectiOn PredictionNeoOBSNeonatal Sepsis Observational studyVLBWvery low birth weight

Performance of the NeoHoP score

When applied to the external validation cohort (NeoOBS-SA), the NeoHoP score achieved a sensitivity of 65%, specificity of 92%, a PLR of 7.73 and a Youden index of 0.57 for a score of ≥2. The area under the ROC curve was 0.874 (95% CI: 0.838 to 0.910) for the prediction of HAI (table 3). There were 1.6% (9/573) false positive results and 28% (163/573) false negative results. Similar results were achieved when the NeoHoP score was applied to a subgroup of the external validation cohort, which included only VLBW infants (NeoOBS-SA-VLBW). In this subgroup, a NeoHoP score of ≥2 achieved diagnostic performance of sensitivity (66%), specificity (90%), PLR (6.44) and Youden index (0.56), with an area under the ROC curve of 0.861 (95% CI: 0.815 to 0.907) (table 3). When applying the score to only proven HAI, the area under the curve was 0.627 (95% CI: 0.579 to 0.627) (figure 2).

Table 3. Performance of the NeoHoP score ≥2 for the diagnosis of any HAI in the NeoHoP validation cohort, the NeoOBS-SA cohort and the NeoOBS-SA-VLBW cohort.

	External validation cohort*		External validation cohort: VLBW only†		Internal validation cohort‡
		95% CI		95% CI		95% CI
n	573		373		552
Sensitivity (%)	65.0	60.5–69.4	66.1	60.2–71.3	54.2	47.3–60.9
Specificity (%)	91.6	84.6–96.1	89.7	80.8–95.5	96.4	93.9–98.2
PLR	7.7	4.1–14.5	6.4	3.3–12.5	15.2	8.6–26.9
NLR	0.38	0.33–0.44	0.38	0.32–0.45	0.48	0.41–0.55
PPV (%)	97.1	94.7–98.4	96.1	92.6–97.9	90.7	84.7–94.5
NPV (%)	37.6	34.4–40.8	41.2	36.8–45.2	76.7	73.9–79.2
Diagnostic accuracy (%)	70.0	66.1–73.7	71.2	66.0–75.4	79.9	76.3–83.2
Youden index	0.57		0.56		0.51

^*External validation cohort includes 346 infants with 573 episodes of suspected HAI selected from the NeoOBS-SA study.17

^†External validation cohort: VLBW only is a subset of 222 VLBW infants within the NeoOBS-SA cohort (NeoOBS-SA-VLBW).17

^‡Internal validation cohort: Tthe NeoHoP study included 406 VLBW infants with 552 episodes of suspected HAI that were used for internal validation of the NeoHoP score.16

HAIhealthcare-associated infectionNeoOBSNeonatal Sepsis Observational studyNLRnegative likelihood ratioNPVnegative predictive valuePLRpositive likelihood ratioPPVpositive predictive valueVLBWvery low birth weight

Figure 2. Receiver operating characteristic (ROC) curves for the diagnostic accuracy of the NeoHoP score for the diagnosis of any HAIs in the external validation cohort (area under curve: 0.874; 95% CI: 0.838 to 0.910); external validation cohort: VLBW only (area under curve 0.861; 95% CI: 0.815 to 0.907); external validation cohort: proven HAI only (area under curve: 0.627; 95% CI: 0.579 to 0.627); and the internal validation cohort (area under curve: 0.868; 95% CI: 0.837 to 0.900). HAI, healthcare-associated infection; VLBW, very low birth weight.

Discussion

We conducted a multicentre external validation of the NeoHoP score in a prospectively enrolled cohort of 600 South African neonates participating in a global neonatal sepsis observational (NeoOBS) study. The NeoHoP infection prediction score demonstrated comparable diagnostic accuracy with high specificity and high PLRs (>85%) across all weight categories. This confirms that a NeoHoP score ≥2 is effectively positioned as a ‘rule-in’ test for HAI in settings where CRP is used as a marker for neonatal sepsis.

The NeoHoP score comprises five variables making it easy to use and incorporating parameters available in most low-resource settings. Compared with the original study the score achieved higher sensitivity, slightly lower specificity, higher PPV and a higher Youden index in both birth weight categories. Although the PLRs of 7.74 and 6.44 for all birth weights and VLBW infants, respectively, were lower than the ideal PLR of >10 required to ‘rule-in’ disease, the high specificity of the NeoHoP score still retains valuable clinical utility.^{23 24}

Existing infection prediction scores, such as the NOSEP1,²⁷ NOSEP-NEW1,²⁸ Singh et al,²⁹ Rosenberg et al³⁰ and Bekhof et al³¹ scores, were all developed in settings outside of Africa and did not perform well when applied to a South African cohort of VLBW infants. In an evaluation of these scores in an LMIC setting, the ROC curves for diagnostic accuracy for the prediction of proven and/or presumed infection were 0.898, 0.820, 0.550, 0.566 and 0.620, respectively.¹⁵ The present external validation of the NeoHoP score achieved diagnostic accuracy similar to the NOSEP1 score (fever ≥38.2°C, CRP ≥14 mg/L, neutrophil percentage >50%, platelets <1 50 000/mm³, total parenteral nutrition ≥14 days),²⁷ with an area under ROC curve of 0.874 and 0.861 for all birthweights and VLBW infants, respectively.

Infection prediction scores developed for use in LMIC should ideally not include parameters such as blood gas analysis, blood pressure monitoring and continuous heart rate monitoring as these are not frequently available in resource-limited settings. The NeoHoP score was developed specifically for such settings, and purposively included parameters that are readily available in most LMIC neonatal units. Lethargy and abdominal distention are subjective clinical parameters included in the NeoHoP score, and this may be concerning as it is subject to individual interpretation. However, it should be highlighted that the presence of two or more features is required for the NeoHoP score to achieve clinical relevance. The NeoOBS-SA data were collected prospectively for a global neonatal sepsis study, thus recording the parameters required for the NeoHoP score was not mandated. Due to this, 19% of HAI episodes were excluded due to incomplete clinical records, and 28% were excluded as no CRP was done. Despite the exclusion of 46% of the HAI episodes investigated, the NeoHoP score performed well in the NeoOBS-SA cohort. Of note, the NeoOBS study targeted infants with highly suspected sepsis, in comparison to the internal validation cohort of the NeoHoP score which included any baby that received a sepsis work-up. This may explain the higher proportion of confirmed HAI in the external validation cohort (NeoOBS-SA), as well as the difference in the specificity and NPV between the internal validation cohort and the NeoOBS-SA cohort.

The inclusion of CRP in the score may be considered controversial. The purpose of an infection prediction score is to assist clinicians at the bedside in their decision-making process. A laboratory CRP is not always immediately available in many resource-limited neonatal unit settings. Using a point-of-care (POC) CRP test can mitigate the delay in results, thus enhancing the utility of the score as a bedside diagnostic aid.¹³ POC CRP has been evaluated in another South African neonatal unit and was found to be a quick and reliable method to determine CRP, that can be used to rationalise antibiotic use and reduce hospital expenditure.³² In a recent systematic review, it was demonstrated that POC tests, including POC CRP, are cost-effective in reducing antimicrobial prescribing in LMIC settings,³³ highlighting the need for further studies evaluating the use of POC CRP, especially looking at cost-effectiveness and feasibility in low-resource settings. Additionally, there is no consensus in the literature regarding the appropriate time to perform a CRP. Performing a CRP at the onset of symptoms may result in a false-negative result as serum CRP concentrations rise within 10–12 hours, and peak after 36–48 hours.³⁴ In this study, conducted in three large tertiary hospitals in South Africa, CRP testing was not universally performed in all investigations, despite all of these facilities having access to an on-site laboratory. This may be due to several reasons, including the uncertainty regarding the correct time to perform the test and the interpretation of these test results, as well as internal institutional cost-saving measures that may limit laboratory investigations.

A strength of this study is that the external validation cohort included a more diverse neonatal population, as evidenced by the statistically significant differences in the demographics of the study population when compared with the internal validation cohort. Although the NeoHoP score was initially developed for use in VLBW infants, it demonstrated strong performance across all weight categories, thus increasing its generalisability for diagnosing HAI in all newborns.

The predominance of Gram-negative organisms in the external validation cohort is in keeping with reports from other LMICs.^{7 35} In a recent South African national study on neonatal bloodstream infections, 57% of the identified isolates were Gram-negative bacteria, and 7% were fungal.³⁶ The study also found that these infections were more likely to occur in national central and provincial tertiary facilities, with the highest incidence occurring in the Gauteng province, where two of the study sites are located.³⁶ In the external validation cohort, Gram-negative bacteria were isolated in 53.8% and fungal infections in 12.4% of cases, confirming that the study population is representative of the broader South African neonatal population. It is concerning that the diagnostic performance of the score was less optimal when only including proven HAI, however, this may be indicative of the low yield of blood cultures in paucibacillary bacteraemia, as well as the important role of presumed sepsis and non-bacterial infections in the neonate.³⁷ It must be noted that in the Newborn Essential Solutions and Technologies (NEST360) study, it was demonstrated that in some African countries blood cultures are under-used, with 40% of the hospitals evaluated not performing any blood cultures for newborns.³⁸

The major strength of this study is the use of a large, well-curated, prospectively collected dataset (NeoOBS-SA) including all birth weight categories, demonstrating the utility of the NeoHoP score in neonatal populations other than VLBW infants. As the score performed well in this expanded patient cohort, it speaks to its generalisability in a wider neonatal population where CRP testing is included in the management of neonatal sepsis.

A major limitation of this study was the retrospective study design and the application of the score to neonates from central academic hospitals in a single country. The NeoOBS study targeted infants with highly suspected sepsis, which may have introduced some patient selection bias.

Conclusion

The NeoHoP score has demonstrated remarkable performance in a multicentre validation, establishing its efficacy as a rule-in test for HAI across all birth weight categories. This signifies a significant advancement, suggesting that healthcare practitioners in LMIC settings could use the NeoHoP score as a valuable sepsis diagnostic tool to ‘rule-in’ HAI, complementing existing laboratory evaluations. Future research efforts should focus on prospectively evaluating the NeoHoP score as a bedside test in non-tertiary low-resource settings, possibly combined with POC CRP testing to improve the practicality, and to evaluate the time point at which the use of the score would be most valuable. This synergistic approach holds considerable potential to enhance the diagnostic accuracy and practical application of the NeoHoP score within LMIC neonatal units.

Data availability statement

Data are available upon reasonable request.