Predicting sepsis-related mortality and ICU admissions from telephone triage information of patients presenting to out-of-hours GP cooperatives with acute infections: A cohort study of linked routine care databases

Feike J Loots; Marleen Smits; Kevin Jenniskens; Artuur M Leeuwenberg; Paul H J Giesen; Lotte Ramerman; Robert Verheij; Arthur R H van Zanten; Roderick P Venekamp

doi:10.1371/journal.pone.0294557

. 2023 Dec 13;18(12):e0294557. doi: 10.1371/journal.pone.0294557

Predicting sepsis-related mortality and ICU admissions from telephone triage information of patients presenting to out-of-hours GP cooperatives with acute infections: A cohort study of linked routine care databases

Feike J Loots ^1,^*,^#, Marleen Smits ^2,^#, Kevin Jenniskens ¹, Artuur M Leeuwenberg ¹, Paul H J Giesen ², Lotte Ramerman ³, Robert Verheij ³, Arthur R H van Zanten ^4,⁵, Roderick P Venekamp ¹

Editor: Marcelo Arruda Nakazone⁶

PMCID: PMC10718413 PMID: 38091283

Abstract

Background

General practitioners (GPs) often assess patients with acute infections. It is challenging for GPs to recognize patients needing immediate hospital referral for sepsis while avoiding unnecessary referrals. This study aimed to predict adverse sepsis-related outcomes from telephone triage information of patients presenting to out-of-hours GP cooperatives.

Methods

A retrospective cohort study using linked routine care databases from out-of-hours GP cooperatives, general practices, hospitals and mortality registration. We included adult patients with complaints possibly related to an acute infection, who were assessed (clinic consultation or home visit) by a GP from a GP cooperative between 2017–2019. We used telephone triage information to derive a risk prediction model for sepsis-related adverse outcome (infection-related ICU admission within seven days or infection-related death within 30 days) using logistic regression, random forest, and neural network machine learning techniques. Data from 2017 and 2018 were used for derivation and from 2019 for validation.

Results

We included 155,486 patients (median age of 51 years; 59% females) in the analyses. The strongest predictors for sepsis-related adverse outcome were age, type of contact (home visit or clinic consultation), patients considered ABCD unstable during triage, and the entry complaints”general malaise”, “shortness of breath” and “fever”. The multivariable logistic regression model resulted in a C-statistic of 0.89 (95% CI 0.88–0.90) with good calibration. Machine learning models performed similarly to the logistic regression model. A “sepsis alert” based on a predicted probability >1% resulted in a sensitivity of 82% and a positive predictive value of 4.5%. However, most events occurred in patients receiving home visits, and model performance was substantially worse in this subgroup (C-statistic 0.70).

Conclusion

Several patient characteristics identified during telephone triage of patients presenting to out-of-hours GP cooperatives were associated with sepsis-related adverse outcomes. Still, on a patient level, predictions were not sufficiently accurate for clinical purposes.

Background

There is an increasing awareness of the importance of sepsis worldwide [1], with sepsis being a leading cause of death globally [2], and survivors often suffering from long-term cognitive and physical impairments [3]. Early recognition of sepsis is crucial for the prognosis, as early stages of sepsis-related organ dysfunction are easily reversible with timely and adequate treatment [4, 5]. Delay of such treatment is associated with increased mortality [6, 7]. However, reducing mortality and morbidity from sepsis poses a significant challenge for healthcare providers [8]. General practitioners (GPs) are usually the first healthcare providers to assess patients with (possible) sepsis. Therefore, the decision of the GP whether or not to refer a patient to the hospital is essential for patients’ prognosis. In the Netherlands, GPs act as gatekeepers: they handle more than 90% of medical problems presented, and a referral is needed for visits to medical specialists in hospitals. Out-of-hours (OOH) primary care is provided by 51 organisations of GP cooperatives, each of which has 50 to 250 GPs who provide care to 100,000 to 500,000 citizens. The cooperatives serve 99% of the Dutch population of 17 million and are available daily from 5 p.m. to 8 a.m. on weekdays and all hours on weekends [9].

Previous research from our project group has shown that about half of the patients admitted to an Intensive Care Unit (ICU) due to community-acquired sepsis had contacted an OOH GP cooperative prior to admission [10]. In almost half of these patients, infection was not suspected by GPs who had assessed the patient. Mortality in patients where there was no suspicion of an infection was substantially higher than in patients where the GP did suspect an infection(42% versus 16%). About one in three patients were not referred to the hospital after the first contact. This observation indicates that the OOH GP cooperative is a setting where there is a relatively high-risk for sepsis, and the focus should not only be on patients with obvious signs of an infection.

In Dutch OOH GP cooperatives, each patient contact is preceded by telephone triage by a trained triage nurse who is supervised by a GP. The triage nurse uses a computer-aided triage system to determine the urgency and required follow-up care (S1 Appendix). Follow-up care can include telephone advice, clinic consultation, home visits or ambulance deployment [9, 11]. To prevent delay in treatment of patients with sepsis, triage nurses should identify patients at risk for sepsis and allocate them a high urgency level. Currently, no specific tools are available for the recognition of possible sepsis during telephonic triage in the primary care setting. This study aimed to predict sepsis-related mortality and ICU admissions from telephone triage information of patients presenting to OOH GP cooperatives with possible infections. Subsequently, we aimed to assess the possible clinically utility of such predictions.

Methods

Design

This retrospective cohort study used linked routine care databases from out-of-hours (OOH) GP cooperatives, general practices, hospitals, and mortality registration.

Population and setting

We included contacts from adult patients (≥18 years) who had a face-to-face medical assessment by a GP (clinic consultation or home visit) from one of the 28 OOH GP cooperatives in the Netherlands participating in Nivel Primary Care Database (PCD) between 1 January 2017 and 30 November 2019 (application number NZR-00321.010). The follow-up period was 30 days. The calendar year 2019 was the most recent data available during the conduct of the study, but the data from December 2019 could not be used as follow-up data was not available for this month. We did not select earlier years as the amount of data was sufficient using the selected years, and older data is less representative for the current situation. We included only adult patients as clinical presentation and predictors of sepsis in children differ substantially from adults. The Nivel PCD contains pseudonymized data on patient level from electronic medical records from both OOH GP cooperatives and individual GPs. The total catchment area of the participating OOH GP cooperatives consists of approximately 10.5 million Dutch inhabitants. For approximately 10% of these patients, data from contacts with their daytime general practice are also available in Nivel PCD. Contacts were only included in our study if there was data from both the OOH GP cooperative and the regular GP. Therefore, the population at risk consist of a ~ 1 million inhabitants of the Netherlands, and can be considered a representative sample of the country. The following contacts were excluded:

Patients who contacted the OOH GP cooperative with complaints unlikely associated with severe infections (such as trauma, eye or ear complaints)
Terminally ill patients (because acute hospital referral is not indicated to prevent mortality in these patients)

See S2 Appendix. for criteria for terminal illness and a list of in- and excluded entry complaints.

Data collection

We used data from OOH GP cooperatives, regular general practices, hospitals and the mortality registration (see Fig 1). To link the data, patient identification numbers were pseudonymized by a Trusted Third Party (ZorgTTP). All data were stored and analyzed in a secure environment, facilitated by the Central Bureau for Statistics (CBS), to guarantee data safety.

First, we extracted data from contacts with the OOH GP cooperative from Nivel PCD: age, gender, date and time of contact, contact type (telephone advice, clinic consultation or home visit), ICPC code (International Classification of Primary Care), antibiotics prescribed during contact, urgency at triage (six levels, ranging from U0: Resuscitation to U5: Advice), and entry complaint at triage. Contacts were labelled as “index contacts” if there was a time interval of at least seven days between two face-to-face contacts. Only the index contacts were included.

Second, data from the patients’ general practices were retrieved from Nivel PCD. Candidate predictors were selected based om literature, expert knowledge and data availability: date and time of contact, ICPC code, antibiotics prescribed, chronic disorders (COPD, DM, heart and vascular disease, neurological disease, renal impairment, active malignancy), and use of immunosuppressive and benzodiazepine medication. These data were linked to the OOH GP cooperative-contacts based on pseudonyms to protect the patients’ privacy. Only general practice contacts within 72 hours before and after OOH GP cooperative-contact were linked with the OOH GP cooperative contacts.

Third, data from hospital admissions were extracted from Dutch Hospital Data (DHD): date of hospital admission, admission department (including ICU), date and time of discharge, and discharge diagnosis. Only hospital admissions within seven days after the contact with the OOH GP cooperative were linked with the OOH GP cooperative contacts.

Finally, mortality data were extracted from the Personal Records Database (BRP): date and cause of death. Only deaths within 30 days after the contact with the OOH GP cooperative were linked with the OOH GP cooperative contacts.

Outcome measures

Primary outcome

We used “sepsis-related adverse outcome” as our primary outcome, defined as 1) hospital admission due to infection within seven days, with at least one day at ICU or 2) death of a patient due to infection within 30 days after index contact with OOH GP cooperative (S3 Appendix). Criteria when to label patients as having an infectious condition as part of the primary outcome definition are shown in S3 Appendix.

Secondary outcome

The predefined primary outcome of interest only concerns patients with a very severe disease course. Since many more patients have an indication for immediate hospital referral to treat serious acute infections, we included “hospital admission due to infection within 72 hours after contact” as secondary outcome, defined as: i) hospital admission of patient with infectious condition within 72 hours after index contact with OOH GP cooperative and ii) length of stay of ≥ 4 days.

Predictors

We selected the following variables as candidate predictors for development of the prediction model: age (years); sex (male/female); time of contact (day/evening/night); entry complaint at triage; urgency category (U0-U5); type of consultation (clinic consultation/home visit); earlier contact with own GP < 72 hours (yes/no); earlier antibiotics prescribed by own GP < 72 hours; earlier telephone contact with OOH GP cooperative < 72 hours (yes/no); presence of comorbidities (COPD, diabetes, heart and vascular disease, neurological disease, kidney disease, malignancy); number of comorbidities; use of immunosuppressive medication (yes/no). All variables were selected that may be associated with sepsis based on literature and expert knowledge. No statistical methods were used for preselection of candidate predictors.

Ethics approval

The use of electronic health records for research purposes is allowed under certain conditions. When these conditions are fulfilled, neither obtaining informed consent from patients nor formal approval by a medical ethics committee is obligatory for this type of observational studies containing no directly identifiable data (art. 24 GDPR Implementation Act jo art. 9.2 sub j GDPR). The Ethical Research Committee of the Radboud University Medical Center Nijmegen stated that this study does not fall within the remit of the Dutch Medical Research Involving Human Subjects Act [Wet Mensgebonden Onderzoek] (file number 2021–7325).

Analyses

Index contacts were the unit of analysis. We used descriptive analyses on population characteristics, healthcare processes such as urgency and type of consultation, and the primary outcome measure, sepsis-related adverse outcome. We did not impute missing data, as we observed almost complete data with only two missing values for sex. These two patients were excluded from the analyses.

The four entry complaints Airway, Breathing, Circulation and Disability are used for documentation of (potential) instability of vital functions, and were recoded into one entry complaint “ABCD unstable”. We calculated relative risks (RR) with corresponding 95% confidence intervals (CI) to examine the relation between all included entry complaints and sepsis-related adverse outcomes.

For model development, we primarily performed multivariable logistic regression analysis including all predictors. The complete model was reduced by removing variables based on Akaike’s information criterion (AIC) change [12]. The final logistic regression model was established when no more variables could be removed from the model without inducing a significant rise in AIC. Besides the logistic regression model, we explored whether machine learning techniques could lead to an improved prediction model. Therefore, a gradient boosted random forest model, and a fully connected feed-forward neural network were developed. Both models were fit by minimizing the cross-entropy (i.e., maximizing likelihood). Tuning of important hyperparameters was performed via grid search based on 10-fold cross-validated C-statistic (performed in the training data). To fit the random forest model, the XGBoost library was used [13], and the tuned parameters were the maximum tree depth, the percentage of total predictors sampled for the development of each tree, and gamma. For the neural network model, the Torch library was used [14], and the tuned parameters were the learning rate, the dropout regularization rate, the number of hidden layers and batch size. Details about the hyperparameters used for model development are provided in S4 Appendix.

Logistic and machine learning model performance was assessed in terms of discrimination and calibration. Discrimination was analyzed using the C-statistic, which equals the area under the receiver operator characteristic curve. Calibration was assessed by visual inspection of the calibration plot, as well as calibration in the large calibration slope, Brier score, and O/E ratio.

For the derivation of all models, we used the data of the first two calendar years (2017 and 2018) as train data. The data of 2019 was used as test data to validate the derived models.

The best performing model was used for developing a sepsis alert, which uses a cut-off for referral of patients suspected of sepsis. In case of similar performance, the logistic model was preferred over the machine learning models due to its transparent nature.

Subgroup and sensitivity analyses were performed using the 2019 data. Predictions of the best performing model were applied in the subgroups of clinical consultations and home visits. Additionally, the model was assessed for prediction of an alternative composite outcome, defined as either of the primary outcome, and the secondary outcome (hospital admission due to infection < 72 hours after contact). Finally, a new model was developed in the subgroup of patients receiving home visits, to assess if predictions would improve in this subgroup with the highest prevalence of sepsis.

All analyses were performed with IBM SPSS 22 and R version 4.1.3 Statistical Software.

Results

Patient characteristics

In total, 155,486 index contacts from 114,917 individual patients were included in the analyses (Fig 1); 120,684 (78%) concerned clinical consultations and 34,802 (22%) home visits. (Table 1). The median age of the study population was 51 years (IQR 33–71), and 41% were males. The overall 30-day mortality rate was 1.8% and the 30-day infection-related mortality rate was 0.7%. The total number of index contacts with sepsis-related adverse outcomes in the study population was 1,363 (0.9%).

Table 1. Patient characteristics stratified by sepsis-related adverse outcome (yes vs no), and for the total study population.

Patient characteristic	Sepsis-related adverse outcome		Total
	Yes	No	N = 155,486
	N = 1,363	N = 154,123	N = 155,486
Age, median years (IQR)	81 (70–88)	51 (33–70)	51 (33–71)
Sex N (%)
Male	693 (51)	63,290 (41)	63,983 (41)
Female	670 (49)	90,831 (59)	91,501 (59)
Time of contact N (%)
Mo-Fr 8.00–23.59h	353 (26)	44,685 (29)	45,038 (29)
Sat-Sun / holiday 8.00–23.59h	727 (53)	82,524 (54)	83,251 (54)
Night 0.00–7.59h	283 (21)	26,914 (17)	27,197 (17)
Type contact N (%)
Clinic consultation	227 (17)	120,458 (78)	120,684 (78)
Home visit	1,137 (83)	33,665 (22)	34,802 (22)
Urgency category, N (%)
U0-U1	32 (2.3)	1,526 (1.0)	1,556 (1.0)
U2	778 (57)	54,660 (35)	55,438 (36)
U3	527(39)	74,642 (48)	75,169 (48)
U4	23 (1.7)	20,524 (13)	20,547 (13)
U5	4 (0.3)	2771 (1.8)	2775 (1.8)
Comorbidities
COPD	342 (25)	13,081 (8.5)	13,423 (8.6)
Diabetes	423 (31)	21,129 (14)	21,552 (14)
Neurological disease	114 (8.4)	6,005 (3.9)	6,119 (3.9)
Kidney disease	350 (26)	11,490 (7.5)	11,840 (7.6)
Malignancy	137 (10)	6,716 (4.4)	6,853 (4.4)
Cardiovascular	742 (54)	31,000 (20)	31,742 (20)
Number of comorbidities
0	264 (19)	98,543 (64)	98,807 (64)
1	446 (33)	31,626 (21)	32,072 (21)
2	381 (28)	15,888 (10)	16,269 (10)
> 2	263 (19)	7,938 (5.2)	8,201 (5.3)
Immunosuppressive medication use	262 (19)	9,825 (6.4)	10,087 (6.5)
Antibiotics prescribed during contact	277 (20)	27,702 (18)	27,979 (18)
Antibiotics prescribed <72h before contact	144 (13)	4,103 (5.3)	4,217 (5.4)
Hospitalization due to infection <72h after contact	514 (38)	3,107 (2.0)	3,621 (2.3)
Duration of hospitalization, median (IQR)	7.0 (4.0–14.0)	4.0 (2.0–7.0)	4.0 (2.0–7.0)
30-day all-cause mortality	1,028 (75)	1,805 (1.2)	2,833 (1.8)
30-day infection-related mortality	1,028 (75)	-	1,028 (0.7)
ICU admission during hospitalization	412 (30)	-	412 (0.3)

Open in a new tab

IQR, interquartile range; ICU, intensive care unit

Of the 27 entry complaints selected for inclusion in the study population (S1 Appendix), “ABCD unstable” showed the highest increase in the risk of a sepsis-related adverse outcome (RR 7.65 [95% CI 5.67–10.34]), followed by "general malaise" (RR 4.0 [95% CI 3.53–4.53]) and "shortness of breath” (RR 3.01 [95% CI 2.78–3.26]). The results for all entry complaints are shown in S1 Table.

Multivariable logistic regression showed for "ABCD unstable" an adjusted odds ratio (OR) of 2.23 (95% CI 1.49–3.32); for "general malaise" OR 2.82 (95% CI 2.25–3.52) and for "shortness of breath” OR 1.81 (95% CI 1.48–2.20). For the entry complaint “fever”, an adjusted OR of 2.24 (95% CI 1.7–2.94) was found. The multivariable logistic regression showed age (OR per year 1.03 [95% CI 1.03–1.04], and type of contact (clinic consultation or home visit) (OR 4.38 [95% CI 3.51–5.47]) to be other important predictors of the primary outcome (S2 Table). Similar to the logistic regression analysis, the type of contact and age were the most important predictors in the random forest model (S1 Fig).

Prediction models

The derivation data consisted of 104,552 contacts, with sepsis-related adverse outcomes occurring in 0.82%. The validation data consisted of 50,932 contacts, of which 505 (0.99%) met the primary outcome. The ROC curves and precision-recall curves of the three derived prediction models are shown in Figs 2 and 3, respectively.

The multivariable logistic regression model resulted in a C-statistic of 0.892 (95% CI 0.880–0.904). The random forest and neural network models resulted in nearly identical C-statistics (Table 2). The logistic regression model showed a Brier score of 0.0094, a calibration slope of 0.97 and a calibration intercept of 0.12. Calibration plots of the three different models in the validation data are presented in S2 Fig. As the predictions of the random forest model and neural network model were not superior to the logistic regression model, we applied the logistic regression model in further analyses.

Table 2. Discrimination and calibration of the different models.

The C-statistic is presented for both the derivation and validation data. The remaining outcome measures are reported only for the test data.

Model	C-statistic		Brier score	Slope	E/O ratio
Model	Derivation data	Validation data (95% CI)	Validation data	Validation data	Validation data
Logistic regression	0.894	0.892 (0.880–0.904)	0.0094	0.97	1.22
Random forest (XGBoost)	0.914	0.894 (0.882–0.905)	0.0094	1.18	1.19
Neural network	0.901	0.887 (0.875–0.899)	0.0095	1.50	1.03

Open in a new tab

E/O, expected/observed; CI, confidence interval

To assess the clinical usefulness of a “sepsis alert” during triage, we calculated relevant diagnostic accuracy measures at different thresholds of the predicted probability (see Figs 2 and 3). The optimal cut-off point based on the ROC curve was calculated at a predicted probability > 0.59%. However, this is below the prevalence of the outcome (0.99% in the validation data), resulting in a large number of false positive results. Table 3 shows the results of various different thresholds of the predicted probability. We consider a threshold >1% most appropriate. With higher cut-off values, the sensitivity decreases below 80%, making it inefficient during an initial screening tool for sepsis-related adverse outcomes using telephone triage information.

Table 3. Specificity, positive predictive value (PPV) and negative predictive value (NPV) at different threshold of predicted probability.

Threshold of predicted probability	Sensitivity	Specificity	PPV	NPV	LR+	LR-
>0.5%	90	76	3.6	99.9	3.7	0.14
>0.59%	89	78	3.8	99.9	4.0	0.15
>1%	82	83	4.5	99.8	4.7	0.22
>2%	68	89	5.7	99.6	6.1	0.36
>3%	56	92	6.7	99.5	7.2	0.48
>4%	43	95	7.3	99.4	7.8	0.60
>5%	33	96	7.9	99.3	8.5	0.70
Clinic consultations
>0.5%	35	93.2	5.4	99.2	5.1	0.70
>1%	16	98.3	2.0	99.8	9.3	0.85
>2%	2	99.8	11	98.9	11.3	0.98
Home visits
>1%	94	25	12	98	1.2	0.24
>3%	65	64	16	95	1.8	0.54
>5%	38	83	19	93	2.2	0.75

Open in a new tab

PPV, positive predictive value; NPV, negative predictive value; LR+, positive likelihood ratio; LR-, negative likelihood ratio

Analyses of subgroups based on type of contact (clinic consultations versus home visits)

Most outcomes were observed in patients who were visited at home (3.9% versus 0.2% in clinical consultations), while this subgroup only represented 22% of the total population. We, therefore, examined the predictive value of a sepsis alert by type of contact. In Table 3, the performance measures at different thresholds of the predicted probabilities of the logistic regression model are presented for both clinic consultations and home visits. In case a sepsis alert would be implemented at a threshold of a predicted probability >1%, the sensitivity in patients receiving a clinic consultation would be 16%. This implies that 84% of the patients with sepsis-related adverse outcomes in this subgroup would not be identified by the sepsis alert. The sensitivity of the subset of home visits is 95%, but the positive likelihood ratio is 1.24. This implies the probability that a patient receives a sepsis alert when visited at home is only slightly above the baseline risk, making this alert highly inefficient. The C-statistic of the logistic regression model after stratification for the type of contact was 0.84 for the clinic consultations and 0.70 for the home visits. A logistic regression model fitted on only the data from the patients receiving home visits did not improve predictions in this subgroup, as the C-statistic remained 0.70.

Sensitivity analysis

In sensitivity analysis, predictions for the combined primary and secondary endpoint resulted in C-statistics of 0.853 for the logistic regression model, 0.849 for the random forest model, and 0.851 for the neural networks model, respectively. For a sepsis alarm at a threshold of a predicted probability >1%, we found a sensitivity of 70%, specificity of 84% and PPV of 11% (see S3 and S4 Tables for futher details).

Discussion

Summary of findings

In this large cohort study using linked routine care databases, we identified patient characteristics associated with sepsis-related adverse outcomes during telephone triage among patients presenting to OOH GP cooperatives for possible infections. The variables associated with the highest risk of sepsis-related adverse outcomes were age, type of contact (home visit or clinic consultation), patients considered ABCD unstable during triage, and the entry complaints, “general malaise”, “shortness of breath” and “fever”. A multivariable logistic regression model resulted in a C-statistic of 0.89 for the prediction of sepsis-related adverse outcomes. Prediction models based on machine learning techniques did not lead to an improved model. A sepsis alert based on a predicted probability >1% resulted in a sensitivity of 82%, PPV of 4.6% and a positive likelihood ratio of 4.8. However, about 80% of the outcomes were observed among home visit patients, and in this subgroup, the model showed a C-statistic of 0.70. The sepsis alert in this subgroup would only result in a slightly increased risk above baseline (positive likelihood ratio of 1.2), making this alert highly inefficient.

Comparison with previous research

The observation that most patients with adverse sepsis-related outcomes were found in the relatively small group of patients receiving a home visits, was in line with previous research of our study group. In a retrospective study of patients admitted to the ICU due to sepsis, the majority of patients (59%) who contacted the GP cooperative were assessed during a home visit [10]. Patients receiving home visits are at particular risk for sepsis as these patients are not able to visit the GP cooperative themselves due to illness severity and are mainly frail elderly. We did not identify any prediction models for sepsis or sepsis-related outcomes during triage in the primary care setting. All published models are developed in the hospital setting and include vital signs and sometimes laboratory results which are not available during telephone triage. A 2020 systematic review identified 130 models published in 24 papers for predicting sepsis using machine learning [15]. Most models were intended for the ICU setting, but also models for the diagnosis and prediction of sepsis at emergency department (ED) admission were published. We also identified several further models for the ED setting published between 2020 and 2022. The reported C-statistics of the models for the ED setting ranged between 0.87 and 0.97 and were found to be superior to the commonly used systemic inflammatory response syndrome (SIRS) criteria and quick Sequential Organ Failure Assessment (qSOFA) score. Goh and colleagues published machine learning models to predict sepsis during hospital admission or time frames varying from 4 to 48 hours afterwards [16] and compared performance to physicians’ predictions. The model based on unstructured clinical notes showed to potentially increase the early detection of sepsis by 32% and reduce false positives by up to 17%.

Strengths and limitations

We were able to compose an extensive database with very few missing data by combining electronic medical records of GPs, OOH GP cooperatives, hospitals, and mortality registrations. This is a significant strength, as this provided sufficient power for multivariable analyses of all relevant variables.

Several limitations should be addressed. First, not all patients requiring hospital treatment for (impending) sepsis were included in our primary outcome "sepsis-related adverse outcome". We could not use the outcome “sepsis”, because detailed medical information to determine the presence of organ dysfunction was not available in the data. Furthermore, in the Netherlands, the coding of sepsis using the available ICD-10 codes is not accurate. The diagnosis of sepsis requires expert knowledge, and coding is sometimes performed by administrative personnel. Accuracy of coding varies between hospitals, and, in general, diagnosis of sepsis based on ICD codes is known to be an underrepresentation of the true incidence [17]. Moreover, ICU admission and death resulting from infection are important adverse outcomes for patients that should be avoided as much as possible. By focusing on these outcomes, patient characteristics can be identified in patients who benefit the most from immediate hospital referral.

Based on previous research [18], we estimate the number of patients requiring immediate hospital treatment to be about 3–5 times greater than the number of sepsis-related adverse outcomes but do not expect the predictors to differ between both groups. Another limitation is that detailed triage information was not available in the data. For example, patients presenting with the entry complaints "Shortness of breath", "General malaise", or "Strange or suicidal behavior” are asked during triage whether fever is present. Adding this important information to the prediction model may substantially improve the predictions. Finally, the performance of the models might be slightly optimistic due to the fact that some patients had multiple contacts during the study. Contacts within one week were excluded, but new contacts after that time period were analyzed as new index contacts. We did not exclude these additional contacts, as this would lead to an underestimation of the importance of characteristics of patients with multiple contacts. We do not believe this resulted in relevant bias. Firstly, the vast majority of patients were included only once in the study, and secondly, a slightly lower predictive performance of the model would not have changed the overall conclusion of the study.

Implications for practice and further research

Although the C-statistics of the developed models were promising, we considered predictions insufficient for implementing a sepsis alert during telephone triage. The numbers of false positives and false negatives were unacceptably high depending on the predicted probability threshold. Patients receiving a home visit are about 20 times more likely to meet the endpoint sepsis-related adverse outcome in our study compared to patients assessed during clinic consultations. Entry complaints not specific for infections such as “General malaise” and “Shortness of breath” are common in sepsis. This has two important implications. First, GPs should be vigilant for possible sepsis during OOH home visits for nonspecific complaints. Secondly, triage systems should be designed to allocate appropriate urgency levels to patients with high risk of complications from sepsis in a broad range of entry complaints.

Machine learning algorithms to predict sepsis are increasingly developed in the hospital setting and potentially improve the early detection of sepsis. In our study, the machine learning models had no added value over the logistic regression model. This finding is observed in most studies developing clinical prediction models [19]. However, machine learning can potentially improve predictions when the quantity of data or model development increases, especially when the data is unstructured [19]. Free text or voice recordings during telephone triage may be used in the future. These raw data were unavailable in our databases, and are currently not stored centrally. Adding unstructured data may improve the predictions, making it potentially feasible to implement a sepsis alert after telephone triage in primary care. Such sepsis alerts should not directly guide further treatment, but support GPs in their clinical decision-making process. GPs should carefully assess these patients including the measurement of the patients’ vital signs and consider the diagnosis sepsis. Recently our study group published a new sepsis prediction tool for GPs based on age, temperature, blood pressure, heart rate, oxygen saturation and mental status [18]. A sepsis alert during triage may be a valuable reminder for GPs to use this model during the face-to-face assessment.

In conclusion, sepsis-related adverse outcomes can be predicted based on telephone triage information of patients presenting to out-of-hours GP cooperativesbut, based on the currently available data, not sufficiently accurate to be of added value in clinical practice.

Supporting information

S1 Appendix. Urgency and entry complaints.

(DOCX)

Click here for additional data file.^{(21.8KB, docx)}

S2 Appendix. Criteria for terminal patients.

(DOCX)

Click here for additional data file.^{(18.7KB, docx)}

S3 Appendix. Criteria for infectious condition.

(DOCX)

Click here for additional data file.^{(19.5KB, docx)}

S4 Appendix. Details of machine learning models.

(DOCX)

Click here for additional data file.^{(18.6KB, docx)}

S1 Fig. Importance matrix of the variables in the random forest (XGBoost) model).

(DOCX)

Click here for additional data file.^{(36.1KB, docx)}

S2 Fig. Calibration plots of the developed models in the test data.

(DOCX)

Click here for additional data file.^{(113.2KB, docx)}

S1 Table. Relative risk (RR) of sepsis-related adverse outcomes for the 27 included entry complaints.

(DOCX)

Click here for additional data file.^{(20.2KB, docx)}

S2 Table. Crude and adjusted odds ratio (OR) of all variables included in the multivariable logistic regression model.

(DOCX)

Click here for additional data file.^{(21.7KB, docx)}

S3 Table. C-statistic, Brier score and slope of the predicted probabilities of the different models for the composite primary and secondary outcome in the test data.

(DOCX)

Click here for additional data file.^{(18.7KB, docx)}

S4 Table. Diagnostic performance measures at different thresholds of the predicted probabilities of the logistic regression model for the composite primary and secondary outcome in the test data.

(DOCX)

Click here for additional data file.^{(19.8KB, docx)}

Data Availability

Study results were derived from non-public microdata from the Central Bureau for Statistics (CBS). Under certain conditions, these microdata are accessible for statistical and scientific research. For further information: microdata@cbs.nl.

Funding Statement

This study was funded by ZonMw (grant number 10060011910005). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

References

1.Cecconi M, Evans L, Levy M, Rhodes A. Sepsis and septic shock. Lancet. 2018. Jul 7;392(10141):75–87. doi: 10.1016/S0140-6736(18)30696-2 Epub 2018 Jun 21. . [DOI] [PubMed] [Google Scholar]
2.Rudd KE, Johnson SC, Agesa KM, Shackelford KA, Tsoi D, Kievlan DR, et al. Global, regional, and national sepsis incidence and mortality, 1990–2017: analysis for the Global Burden of Disease Study. Lancet. 2020. Jan 18;395(10219):200–211. doi: 10.1016/S0140-6736(19)32989-7 . [DOI] [PMC free article] [PubMed] [Google Scholar]
3.Prescott HC, Angus DC. Enhancing Recovery From Sepsis: A Review. JAMA. 2018. Jan 2;319(1):62–75. doi: 10.1001/jama.2017.17687 . [DOI] [PMC free article] [PubMed] [Google Scholar]
4.Levy MM, Evans LE, Rhodes A. The Surviving Sepsis Campaign Bundle: 2018 Update. Crit Care Med. 2018. Jun;46(6):997–1000. doi: 10.1097/CCM.0000000000003119 . [DOI] [PubMed] [Google Scholar]
5.Angus DC, Bindman AB. Achieving Diagnostic Excellence for Sepsis. JAMA. 2022. Jan 11;327(2):117–118. doi: 10.1001/jama.2021.23916 . [DOI] [PubMed] [Google Scholar]
6.Kumar A, Haery C, Paladugu B, Kumar A, Symeoneides S, Taiberg L, et al. The duration of hypotension before the initiation of antibiotic treatment is a critical determinant of survival in a murine model of Escherichia coli septic shock: association with serum lactate and inflammatory cytokine levels. J Infect Dis. 2006. Jan 15;193(2):251–8. doi: 10.1086/498909 Epub 2005 Dec 13. . [DOI] [PubMed] [Google Scholar]
7.Seymour CW, Gesten F, Prescott HC, Friedrich ME, Iwashyna TJ, Phillips GS, et al. Time to Treatment and Mortality during Mandated Emergency Care for Sepsis. N Engl J Med. 2017. Jun 8;376(23):2235–2244. doi: 10.1056/NEJMoa1703058 Epub 2017 May 21. . [DOI] [PMC free article] [PubMed] [Google Scholar]
8.Machado FR, Azevedo LCP. Sepsis: A Threat That Needs a Global Solution. Crit Care Med. 2018. Mar;46(3):454–459. doi: 10.1097/CCM.0000000000002899 . [DOI] [PubMed] [Google Scholar]
9.Smits M, Rutten M, Keizer E, Wensing M, Westert G, Giesen P. The Development and Performance of After-Hours Primary Care in the Netherlands: A Narrative Review. Ann Intern Med. 2017. May 16;166(10):737–742. doi: 10.7326/M16-2776 Epub 2017 Apr 18. . [DOI] [PubMed] [Google Scholar]
10.Loots FJ, Smits M, van Steensel C, Giesen P, Hopstaken RM, van Zanten ARH. Management of sepsis in out-of-hours primary care: a retrospective study of patients admitted to the intensive care unit. BMJ Open. 2018. Sep 17;8(9):e022832. doi: 10.1136/bmjopen-2018-022832 . [DOI] [PMC free article] [PubMed] [Google Scholar]
11.Smits M, Plat E, Alink E, de Vries M, Apotheker M, van Overdijk S, et al. Reliability and validity of the Netherlands Triage Standard in emergency care settings: a case scenario study. Emerg Med J. 2022. Aug;39(8):623–627. doi: 10.1136/emermed-2021-211359 Epub 2022 Feb 8. . [DOI] [PubMed] [Google Scholar]
12.Akaike, H. (1973), "Information theory and an extension of the maximum likelihood principle", in Petrov, B. N.; Csáki, F. (eds.), 2nd International Symposium on Information Theory, Tsahkadsor, Armenia, USSR, September 2–8, 1971, Budapest: Akadémiai Kiadó, pp. 267–281. Republished in Kotz, S.; Johnson, N. L., eds. (1992), Breakthroughs in Statistics, vol. I, Springer-Verlag, pp. 610–624.
13.Chen T, Guestrin C. Xgboost: A scalable tree boosting system. InProceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining 2016 13 August (pp. 785–794).
14.Falbel D, Luraschi J (2022). torch: Tensors and Neural Networks with ’GPU’ Acceleration. https://torch.mlverse.org/docs, https://github.com/mlverse/torch.
15.Fleuren LM, Klausch TLT, Zwager CL, Schoonmade LJ, Guo T, Roggeveen LF, et al. Machine learning for the prediction of sepsis: a systematic review and meta-analysis of diagnostic test accuracy. Intensive Care Med. 2020. Mar;46(3):383–400. doi: 10.1007/s00134-019-05872-y Epub 2020 Jan 21. . [DOI] [PMC free article] [PubMed] [Google Scholar]
16.Goh KH, Wang L, Yeow AYK, Poh H, Li K, Yeow JJL, et al. Artificial intelligence in sepsis early prediction and diagnosis using unstructured data in healthcare. Nat Commun. 2021. Jan 29;12(1):711. doi: 10.1038/s41467-021-20910-4 . [DOI] [PMC free article] [PubMed] [Google Scholar]
17.Jolley RJ, Quan H, Jetté N, Sawka KJ, Diep L, Goliath J, et al. Validation and optimisation of an ICD-10-coded case definition for sepsis using administrative health data. BMJ Open. 2015. Dec 23;5(12):e009487. doi: 10.1136/bmjopen-2015-009487 . [DOI] [PMC free article] [PubMed] [Google Scholar]
18.Loots FJ, Smits M, Hopstaken RM, Jenniskens K, Schroeten FH, van den Bruel A, et al. New clinical prediction model for early recognition of sepsis in adult primary care patients: a prospective diagnostic cohort study of development and external validation. Br J Gen Pract. 2022. May 26;72(719):e437–e445. doi: 10.3399/BJGP.2021.0520 . [DOI] [PMC free article] [PubMed] [Google Scholar]
19.Christodoulou E, Ma J, Collins GS, Steyerberg EW, Verbakel JY, Van Calster B. A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models. J Clin Epidemiol. 2019. Jun;110:12–22. doi: 10.1016/j.jclinepi.2019.02.004 Epub 2019 Feb 11. . [DOI] [PubMed] [Google Scholar]

PLoS One. doi: 10.1371/journal.pone.0294557.r001

Decision Letter 0

Marcelo Arruda Nakazone

14 Aug 2023

PONE-D-22-30322Predicting sepsis-related mortality and ICU admissions from telephone triage information of patients presenting to out-of-hours GP cooperatives with acute infections: a cohort study of linked routine care databasesPLOS ONE

Dear Dr. Loots,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

==============================

ACADEMIC EDITOR:

The manuscript is interesting but will require further reworking and a major revision.<o:p></o:p>

While they recognize the potential interest of the subject studied, the reviewers raised a number of important issues that need to be properly addressed.

==============================

Please submit your revised manuscript by Sep 28 2023 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Marcelo Arruda Nakazone, M.D., Ph.D.

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf.

2. Please provide additional details regarding participant consent. In the ethics statement in the Methods and online submission information, please ensure that you have specified (1) whether consent was informed and (2) what type you obtained (for instance, written or verbal, and if verbal, how it was documented and witnessed). If your study included minors, state whether you obtained consent from parents or guardians. If the need for consent was waived by the ethics committee, please include this information.

If you are reporting a retrospective study of medical records or archived samples, please ensure that you have discussed whether all data were fully anonymized before you accessed them and/or whether the IRB or ethics committee waived the requirement for informed consent. If patients provided informed written consent to have data from their medical records used in research, please include this information.

3. Please include captions for your Supporting Information files at the end of your manuscript, and update any in-text citations to match accordingly. Please see our Supporting Information guidelines for more information: http://journals.plos.org/plosone/s/supporting-information.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #4: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #4: No

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: No

Reviewer #4: No

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #4: No

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Title is good but the objective or aim is mixed up with the development of model.

Need to explain more about the time frame of data. Why up to 30th Nov, 2019 should have a better explanation.

If it's retrospective cohort study then unexposed group of population at risk were not mentioned clearly. Is it survey or cohort study should be justified.

Discussion is too short related to findings, should be more explanatory with other study findings.

Confusing conclusion as the title was not focused in development of any tool related to clinical practice.

Development of an accurate prediction model based on telephone triage information and whether this model can be used to implement a “sepsis alert” was not in title.

Reviewer #2: In this paper, the authors conduct a retrospective cohort study to predict sepsis-related mortality and ICU admissions from telephone triage information of patients with acute infections presenting to out-of-hours GP cooperatives in the Netherlands. The paper reports that age, type of contact, patients considered ABCD unstable, and other entry complaints (general malaise, shortness of breath and fever) were the strongest predictors for sepsis-related adverse outcome. The authors also aimed to develop a prediction model based on triage information. However, they conclude that sepsis-related adverse outcomes couldn't be predicted sufficiently accurately for clinical purposes.

The research questions explored in this study – if (established and) repeated in a number of robust studies – can be of significance for the early treatment and management of patients suffering from critical illnesses like sepsis. They are key to improving healthcare delivery and clinical decision-making.

I wish the authors the very best with their research!

Reviewer #3: The authors submit a large, retrospective cohort study of sepsis-related outcomes among patients presenting to out-of-hours general practitioner cooperatives for possible infections. High risk predictors of outcomes were identified including unstable ABCD findings, age, home visits, and specific chief complaints. However, the majority of outcomes occurred in home visits, where a predictive model demonstrated a C-statistic of 0.7 and a positive predictive value of 12% with a threshold of probability >1%. In this study the authors sought to use available data from contacts with a GP cooperative and patient level data to answer an important question: can the risk of adverse sepsis outcomes be predicted using these variables so as to improve triage decisions, and thereby mitigate the risk of these outcomes occurring. Although the study did not yield a useful prediction model, it certainly adds to the expanding body of knowledge regarding tele-triage. Some specific comments and questions:

- The study includes data from GP contacts between 2017-2019. As the data is now >4 years old, can the authors provide some rationale as to why more recent data was not collected and consider including this as a limitation?

- A limited set of chronic disorders and prescription medications were selected for collection of patient level data. How were these diagnoses and, in particular, medications selected?

- In secondary outcomes, patients admitted to the hospital with infection within 72h of contact were included only if their length of stay was >= 4 days. Why was a minimum admission of 4 days chosen? Would not any hospitalization due to infection be an important outcome?

- The authors comment that ICD codes often underrepresent the true incidence of sepsis. If that is the case, why was a predicted probability of 1% selected as most appropriate? Should additional sensitivity analysis be considered for the model with a higher predicted probability?

- Why do the authors believe that home visits were so much more likely to be associated with adverse outcomes? Some discussion of this finding would be a valuable addition to the manuscript.

- Given the limited value of the prediction model, do the authors have proposed suggestions for improvement in triage data collection that could inform future analysis?

Reviewer #4: The authors of this study have conducted an interesting study with a potential to make a difference in patient care. The authors aim to identify predictors and develop a prediction model for adverse sepsis-related outcomes for patients who are telephone-triaged and presented to out-of-hours GP cooperatives.

Overall: nice paper but could be improved with respect to language and methodology.

BACKGROUND:

- Provide a strong rationale for the derivation of this model. You need to provide rationale that such a tool is needed and that there are no other tools already in the introduction/rationale for study.

- The following sentence does not read well: "Early recognition of sepsis is crucial for the prognosis, as early stages of

sepsis-related organ dysfunction are easily reversible with timely and adequate treatment [4,5],

and delay of adequate treatment resulting in increased mortality [6,7]." Perhaps break into 2-3 sentences?

- The following sentence also does not read well and might need rewording and possibly splitting into two: "In

almost half of these patients, infection was not suspected by GPs who had assessed the patient

and mortality in patients without suspected infection was substantially higher (42% versus 16%)."

- For the sentence "About one in three patients was not referred to the hospital after the first contact.", you probably meant "About one in three patients were..."

- The sentence "This observation indicates that the OOH GP cooperative is a relatively high-risk setting for sepsis..." might need to be reworded to indicate that the OOH GP is a setting where there is a relatively high risk for sepsis as the setting itself is not high risk.

- "This study aims to identify predictors for adverse sepsis-related outcomes based on telephone triage information among patients presenting to OOH GP cooperatives with possible infections. We also aimed to develop an accurate prediction model based on telephone triage information, and whether this model can be used to implement a “sepsis alert”. Your study objective could be just one objective: to derive a prediction model. Developing a prediction model by itself identifies predictors.

- If you do consider your objective(s) to derive a prediction model, you should follow the TRIPOD Statement: Collins, G.S., Reitsma, J.B., Altman, D.G. et al. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD Statement. BMC Med 13, 1 (2015). https://doi.org/10.1186/s12916-014-0241-z

METHODS:

Population: Please summarize the healthcare system in Netherlands and especially the GP. The authors indicate that 28 OOH GP cooperatives in the Netherlands participated in Nivel Primary Care Database (PCD) and these were included. What is the total number of GP cooperatives in the area of study? Why wouldn't other participate? Any bias in this case?

Please provide a rationale why you selected such years of data, i.e. from Jan 2017 to November 2019. Why didn't you select earlier years? Was it due to quality or availability of the data? Why not include more recent data? Was it because of COVID that you stopped at November 2019? You need to provide the rationale.

Data collection: Were you able to link all the records? If not please provide the % of records that were not linked and excluded.

Predictors: How did you decide on these predictors? Were they based on literature review, statistical variable selection methods, availability, etc?

Statistical analyses: The authors used a split-sample method for the derivation and validation of the study. Please note that this can only be considered as internal validation. External validation is still needed prior to using the model in clinics. In addition, I would suggest you use all the available data for the derivation of the study and use bootstrap methodology for the internal validation -- this way you will have a more robust model.

- A significant portion of your patients had multiple contacts and so were included in your models more than onces. Did you account for such patients in your models? It is likely that those patients had same/similar characteristics in each of those visits so your models are biased. Perhaps do a sensitivity analysis by using unique patients?

- When you say "Subgroup and sensitivity analyses were performed", do you mean for the internal validation?

- Which R package(s) did you use for the modelling?

RESULTS:

- Figure 1 plot is not clear.

DISCUSSION:

- Could be more concise

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Muhammed Ashraful Alam

Reviewer #2: No

Reviewer #3: No

Reviewer #4: No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

Attachment

Submitted filename: Predicting sepsis_Review.docx

Click here for additional data file.^{(13.2KB, docx)}

PLoS One. 2023 Dec 13;18(12):e0294557. doi: 10.1371/journal.pone.0294557.r002

Author response to Decision Letter 0

13 Oct 2023

To: Editorial Board of PLoS ONE

9 October 2023

Subject: PONE-D-22-30322.R1

Dear editor,

Thank you for your letter of 15 August, in which you offered us the opportunity to submit a revised version of our manuscript.

We would like to thank the editor and reviewers for their constructive comments which enabled us to further improve the manuscript. Please find below our response to each of the points raised.

We hope that our revised manuscript meets your expectations and look forward to hearing from you.

Please do not hesitate to contact us, should you have any questions or remarks.

Yours sincerely,

on behalf of all authors,

Dr. Feike Loots

f.j.loots@umcutrecht.nl

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #4: Yes

________________________________________

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #4: No

________________________________________

3. Have the authors made all data underlying the findings in their manuscript fully available?

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: No

Reviewer #4: No

Answer authors: All data were stored and analyzed in a secure environment, facilitated by the Central Bureau for Statistics (CBS), to guarantee data safety. Therefore, we cannot publicly share the data. However, we have written a guide for others who want to gain access to the data.

________________________________________

4. Is the manuscript presented in an intelligible fashion and written in standard English?

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #4: No

________________________________________

5. Review Comments to the Author

Reviewer #1:

1.1 Title is good but the objective or aim is mixed up with the development of model.

Answer authors: We rephrased the aim of the study in line with the title. We did not aim to identify predictors of sepsis or to derive a prediction model, but rather to investigate whether sepsis can predicted accurately based of triage information.

1.2 Need to explain more about the time frame of data. Why up to 30th Nov, 2019 should have a better explanation.

Answer authors: The data collection of the study was performed in 2021. At this time data from 2020 was not yet available as the hospital data from 2020 is processed during 2021 and ready for research purposes in the beginning of 2022. Therefore we used the most recent data available at the time.

We added the following lines in the text:” The follow-up period was 30 days. The calendar year 2019 was the most recent data available during the conduct of the study, but the data from December 2019 could not be used, as follow-up data was not available for this month.”

1.3 If it's retrospective cohort study then unexposed group of population at risk were not mentioned clearly. Is it survey or cohort study should be justified.

Answer authors: As mentioned in the Methods section, we performed a retrospective cohort study. The population at risk for the study has now been more clearly described in the Methods section: “Therefore the population at risk consist of a ~ 1 million inhabitants of the Netherlands, which can be considered a representative sample of the country.”

1.4 Discussion is too short related to findings, should be more explanatory with other study findings.

Answer authors: We have added the following paragraph to the Discussion section:

“The observation that most patients with adverse sepsis-related outcomes were found in the relatively small group of patients receiving a home visit, was in line with previous research of our study group. In a retrospective study of patients admitted to the ICU due to sepsis, the majority of patients (59%) who contacted the GP cooperative were assessed during a home visit. Patients receiving home visits are at particular risk for sepsis as these patients are not able to visit the GP cooperative themselves due to illness severity and are mainly frail elderly..”

1.5 Confusing conclusion as the title was not focused in development of any tool related to clinical practice.

Development of an accurate prediction model based on telephone triage information and whether this model can be used to implement a “sepsis alert” was not in title.

Answer authors: Prompted by the reviewer’s comment, we have rephrased our conclusion in both the abstract and the main text. The title, objective and conclusions are now aligned.

I wish the authors the very best with their research!

Answer authors: We thank the reviewer for the positive remarks.

3.1 - The study includes data from GP contacts between 2017-2019. As the data is now >4 years old, can the authors provide some rationale as to why more recent data was not collected and consider including this as a limitation?

We added the following line in the text:” The calendar year 2019 was the most recent data available during the conduct of the study, but the data from December 2019 could not be used, as follow-up data was not available for this month.”

3.2 - A limited set of chronic disorders and prescription medications were selected for collection of patient level data. How were these diagnoses and, in particular, medications selected?

Answer authors: We have added in the Methods section: “Candidate predictors were selected based om literature, expert knowledge and data availability”.

3.3 - In secondary outcomes, patients admitted to the hospital with infection within 72h of contact were included only if their length of stay was >= 4 days. Why was a minimum admission of 4 days chosen? Would not any hospitalization due to infection be an important outcome?

Answer authors: The reason we chose a minimum length of hospital stay of 4 day, was to select patients more likely to have sepsis and not only less serious infections. For example, when a patient with pneumonia is referred to the hospital and the length of hospital stay is only 1-3 days, immediate referral after contact with the GP cooperative may not have been crucial. In case of longer hospitalizations it is more likely sepsis was present and immediate referral could potentially prevent further clinical deterioration and adverse outcomes. In case we would have chosen every hospitalization as the secondary outcome, this outcome is largely influenced by the decision of the GP to refer the patient, and not by the fact that this referral is needed to prevent adverse outcomes resulting from sepsis.

3.4 - The authors comment that ICD codes often underrepresent the true incidence of sepsis. If that is the case, why was a predicted probability of 1% selected as most appropriate? Should additional sensitivity analysis be considered for the model with a higher predicted probability?

Answer authors: As ICD codes are known to underrepresent the incidence of sepsis, we chose mortality and ICU admission resulting from sepsis as a more unequivocal primary endpoint. As not all patients with sepsis requiring immediate hospital treatment will meet this outcome, we performed a secondary analysis by including patients admitted >= 4 days due to infection as explained above (point 3.3). In case we would use a predicted probability >1% for the primary outcome, even more patients with sepsis will be missed by the model (false negatives).

3.5 - Why do the authors believe that home visits were so much more likely to be associated with adverse outcomes? Some discussion of this finding would be a valuable addition to the manuscript.

Answer authors: The statement that home visits are more often associated with adverse outcomes is not an assumption, but based on the data we collected. This observation is in line with previous research of our study group. In a retrospective study of patients admitted to the ICU due to sepsis, the majority of patients (59%) who contacted the GP cooperative were assessed during a home visit (BMJ Open 2018 doi: 10.1136/bmjopen-2018-022832). We have added this information to the Discussion section of the revised manuscript.

3.6 - Given the limited value of the prediction model, do the authors have proposed suggestions for improvement in triage data collection that could inform future analysis?

Answer authors: The data available in the study did not include the answers to triage questions during the contact with the GP cooperative. In the Discussion section we suggested to use the free text or voice recordings during telephone triage to further improve the prediction model.

Overall: nice paper but could be improved with respect to language and methodology.

BACKGROUND:

4.1 - Provide a strong rationale for the derivation of this model. You need to provide rationale that such a tool is needed and that there are no other tools already in the introduction/rationale for study.

Answer authors: The rationale for the recognition of sepsis during telephone triage of patients presenting to out-of-hours GP cooperatives is explained in the Background section. We added the following line:

“Currently, no specific tools are available for the recognition of possible sepsis during telephone triage in the primary care setting.”

4.2 - The following sentence does not read well: "Early recognition of sepsis is crucial for the prognosis, as early stages of sepsis-related organ dysfunction are easily reversible with timely and adequate treatment [4,5], and delay of adequate treatment resulting in increased mortality [6,7]." Perhaps break into 2-3 sentences?

Answer authors: As suggested, we split the sentences into:

Early recognition of sepsis is crucial for the prognosis, as early stages of sepsis-related organ dysfunction are easily reversible with timely and adequate treatment [4,5].

Delay of such treatment is associated with increased mortality [6,7].

4.3 - The following sentence also does not read well and might need rewording and possibly splitting into two: "In almost half of these patients, infection was not suspected by GPs who had assessed the patient and mortality in patients without suspected infection was substantially higher (42% versus 16%)."

Answer authors: Thank you for the suggestion, we changed this sentence into:

“In almost half of these patients, infection was not suspected by GPs who had assessed the patient. Mortality in patients where there was no suspicion of an infection was substantially higher than in patients where the GP did suspect an infection (42% versus 16%)”.

4.4 - For the sentence "About one in three patients was not referred to the hospital after the first contact.", you probably meant "About one in three patients were..."

Answer authors: We corrected the grammatical mistake accordingly.

4.5 - The sentence "This observation indicates that the OOH GP cooperative is a relatively high-risk setting for sepsis..." might need to be reworded to indicate that the OOH GP is a setting where there is a relatively high risk for sepsis as the setting itself is not high risk.

Answer authors: We have changed the sentence into “This observation indicates that the OOH GP cooperative is a setting where there is a relatively high-risk for sepsis.”

4.6 - "This study aims to identify predictors for adverse sepsis-related outcomes based on telephone triage information among patients presenting to OOH GP cooperatives with possible infections. We also aimed to develop an accurate prediction model based on telephone triage information, and whether this model can be used to implement a “sepsis alert”. Your study objective could be just one objective: to derive a prediction model. Developing a prediction model by itself identifies predictors.

Answer authors: This comment closely resembles those from the other reviewer (comment 1.1). As elaborated upon above, we have changed this paragraph. The objectives specified in the revised manuscript now also better align with the title and conclusion.

4.7 - If you do consider your objective(s) to derive a prediction model, you should follow the TRIPOD Statement: Collins, G.S., Reitsma, J.B., Altman, D.G. et al. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD Statement. BMC Med 13, 1 (2015). https://doi.org/10.1186/s12916-014-0241-z

Answer authors: We followed the TRIPOD statement, and uploaded the completed form.

METHODS:

4.8 Population: Please summarize the healthcare system in Netherlands and especially the GP. The authors indicate that 28 OOH GP cooperatives in the Netherlands participated in Nivel Primary Care Database (PCD) and these were included. What is the total number of GP cooperatives in the area of study? Why wouldn't other participate? Any bias in this case?

Answer authors: We added a short summary of relevant information concerning the Dutch healthcare system to the Background section.

Regarding the data available in the PCD, we cannot determine the exact reasons for OOH GP cooperatives to participate, but do not anticipate relevant bias. The age and gender of patients in the database are representative for the Netherlands, with only a slight overrepresentation of highly urban areas (https://www.nivel.nl/sites/default/files/bestanden/1003915.pdf). We have added this to the Methods section.

4.9 Please provide a rationale why you selected such years of data, i.e. from Jan 2017 to November 2019. Why didn't you select earlier years? Was it due to quality or availability of the data? Why not include more recent data? Was it because of COVID that you stopped at November 2019? You need to provide the rationale.

Answer authors: This comment closely resembles those from the other reviewers. We therefore would like to refer to our replies above.

4.10 Data collection: Were you able to link all the records? If not please provide the % of records that were not linked and excluded.

Answer authors: We linked records in the database of GP OOH contacts with records in the other databases (GP practice contacts, hospital admissions or mortality registration). If there was no link, we did not exclude the GP OOH record, but assumed there was no earlier or later contact with GP practice or hospital, nor death. In short, we did not exclude any data based on the absence of link with other databases.

4.11 Predictors: How did you decide on these predictors? Were they based on literature review, statistical variable selection methods, availability, etc?

Answer authors: We used all variables that are available in the data form the GP cooperative. From the regular GP practice, all variables were selected that may be associated with sepsis based om literature and expert knowledge. No statistical methods were used for preselection of candidate predictors. We have added this information to the Methods section.

4.12 Statistical analyses: The authors used a split-sample method for the derivation and validation of the study. Please note that this can only be considered as internal validation. External validation is still needed prior to using the model in clinics. In addition, I would suggest you use all the available data for the derivation of the study and use bootstrap methodology for the internal validation -- this way you will have a more robust model.

Answer authors: We chose to use a split sample method for internal validation based on the year of inclusion. By using the data from 2019 as validation data, the results are most representative of an external validation. With sufficient available data - which was the case in our study - a split sample validation is an alternative for using bootstrap methods for internal validation. During the analyses we compared the difference of the model based on the data from 2017 with data from both 2017 and 2018, and found no clinically relevant improvement. Therefore, adding data from 2019 to the model development step was deemed to not meaningfully affect model performance. Our approach provides a temporal validation, which is more robust.

External validation is required before use in clinical practice,(https://doi.org/10.1186/s12916-023-02779-w). However, external validation is not appropriate in our situation since our findings do not support the use of the prediction model in clinical practice.

4.13- A significant portion of your patients had multiple contacts and so were included in your models more than onces. Did you account for such patients in your models? It is likely that those patients had same/similar characteristics in each of those visits so your models are biased. Perhaps do a sensitivity analysis by using unique patients?

Answer authors: We agree with the reviewer that the fact the some patient presented more than once at the GP cooperative during the study period, may have resulted in slightly optimistic predictions of the models. Correction for multiple contacts using a mixed-model approach was deemed not feasible given the complexity. Calculation times of running the model were already up to one week in the current analyses. Using only one contact of every unique patient would introduce more bias, and we do not believe a sensitivity analysis with this approach is appropriate. Furthermore, we conclude that the predictions of the models are not sufficiently accurate for use in clinical practice. Therefore, the possible slightly optimistic predictions as a result of multiple contacts do not impact on the final conclusion of the study. Prompted by the reviewer’s comment, we added the following paragraph to the limitation section of the Discussion:

“The performance of the models might be slightly optimistic due to the fact that some patients had multiple contacts during the study. Contacts within one week were excluded, but new contacts after that time period were analyzed as new index contacts. We did not exclude these additional contacts, as this would lead to an underestimation of the importance of characteristics of patients with multiple contacts. We do not believe this resulted in relevant bias. Firstly, the vast majority of patients were included only once in the study, and secondly, a slightly lower predictive performance of the model would not have changed the overall conclusion of the study.”

4.14 - When you say "Subgroup and sensitivity analyses were performed", do you mean for the internal validation?

Answer authors: The Subgroup and sensitivity analyses were performed during internal validation in the data from 2019.

4.15- Which R package(s) did you use for the modelling?

Answer authors: We used the XGBoost package for the random forest model and Torch package for the neural network model. This has been mentioned in the manuscript under the subheading Analyses.

RESULTS:

4.16 - Figure 1 plot is not clear.

Answer authors: We uploaded Figure 1 in a higher resolution and made some changes to make more clear that four different data sources were linked.

DISCUSSION:

4.17 - Could be more concise

Answer authors: We did not shorten the Discussion section since other reviewers asked for a more detailed Discussion section. Also, addition of some further paragraphs was required to sufficiently address the comments of all reviewers.

PLoS One. doi: 10.1371/journal.pone.0294557.r003

Decision Letter 1

Marcelo Arruda Nakazone

6 Nov 2023

Predicting sepsis-related mortality and ICU admissions from telephone triage information of patients presenting to out-of-hours GP cooperatives with acute infections: a cohort study of linked routine care databases

PONE-D-22-30322R1

Dear Dr. Loots,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Marcelo Arruda Nakazone, M.D., Ph.D.

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #3: All comments have been addressed

Reviewer #4: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

Reviewer #3: Yes

Reviewer #4: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #3: Yes

Reviewer #4: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

Reviewer #3: Yes

Reviewer #4: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

Reviewer #3: Yes

Reviewer #4: Yes

**********

6. Review Comments to the Author

Reviewer #3: (No Response)

Reviewer #4: Thank you for addressing all my comments. The paper reads much better now. Good luck with your paper.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #3: No

Reviewer #4: No

**********

PLoS One. doi: 10.1371/journal.pone.0294557.r004

Acceptance letter

Marcelo Arruda Nakazone

1 Dec 2023

PONE-D-22-30322R1

Dear Dr. Loots:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at customercare@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Professor Marcelo Arruda Nakazone

Academic Editor

PLOS ONE

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

S1 Appendix. Urgency and entry complaints.

(DOCX)

Click here for additional data file.^{(21.8KB, docx)}

S2 Appendix. Criteria for terminal patients.

(DOCX)

Click here for additional data file.^{(18.7KB, docx)}

S3 Appendix. Criteria for infectious condition.

(DOCX)

Click here for additional data file.^{(19.5KB, docx)}

S4 Appendix. Details of machine learning models.

(DOCX)

Click here for additional data file.^{(18.6KB, docx)}

S1 Fig. Importance matrix of the variables in the random forest (XGBoost) model).

(DOCX)

Click here for additional data file.^{(36.1KB, docx)}

S2 Fig. Calibration plots of the developed models in the test data.

(DOCX)

Click here for additional data file.^{(113.2KB, docx)}

S1 Table. Relative risk (RR) of sepsis-related adverse outcomes for the 27 included entry complaints.

(DOCX)

Click here for additional data file.^{(20.2KB, docx)}

S2 Table. Crude and adjusted odds ratio (OR) of all variables included in the multivariable logistic regression model.

(DOCX)

Click here for additional data file.^{(21.7KB, docx)}

S3 Table. C-statistic, Brier score and slope of the predicted probabilities of the different models for the composite primary and secondary outcome in the test data.

(DOCX)

Click here for additional data file.^{(18.7KB, docx)}

S4 Table. Diagnostic performance measures at different thresholds of the predicted probabilities of the logistic regression model for the composite primary and secondary outcome in the test data.

(DOCX)

Click here for additional data file.^{(19.8KB, docx)}

Attachment

Submitted filename: Predicting sepsis_Review.docx

Click here for additional data file.^{(13.2KB, docx)}

Data Availability Statement

[pone.0294557.ref001] 1.Cecconi M, Evans L, Levy M, Rhodes A. Sepsis and septic shock. Lancet. 2018. Jul 7;392(10141):75–87. doi: 10.1016/S0140-6736(18)30696-2 Epub 2018 Jun 21. . [DOI] [PubMed] [Google Scholar]

[pone.0294557.ref002] 2.Rudd KE, Johnson SC, Agesa KM, Shackelford KA, Tsoi D, Kievlan DR, et al. Global, regional, and national sepsis incidence and mortality, 1990–2017: analysis for the Global Burden of Disease Study. Lancet. 2020. Jan 18;395(10219):200–211. doi: 10.1016/S0140-6736(19)32989-7 . [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0294557.ref003] 3.Prescott HC, Angus DC. Enhancing Recovery From Sepsis: A Review. JAMA. 2018. Jan 2;319(1):62–75. doi: 10.1001/jama.2017.17687 . [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0294557.ref004] 4.Levy MM, Evans LE, Rhodes A. The Surviving Sepsis Campaign Bundle: 2018 Update. Crit Care Med. 2018. Jun;46(6):997–1000. doi: 10.1097/CCM.0000000000003119 . [DOI] [PubMed] [Google Scholar]

[pone.0294557.ref005] 5.Angus DC, Bindman AB. Achieving Diagnostic Excellence for Sepsis. JAMA. 2022. Jan 11;327(2):117–118. doi: 10.1001/jama.2021.23916 . [DOI] [PubMed] [Google Scholar]

[pone.0294557.ref006] 6.Kumar A, Haery C, Paladugu B, Kumar A, Symeoneides S, Taiberg L, et al. The duration of hypotension before the initiation of antibiotic treatment is a critical determinant of survival in a murine model of Escherichia coli septic shock: association with serum lactate and inflammatory cytokine levels. J Infect Dis. 2006. Jan 15;193(2):251–8. doi: 10.1086/498909 Epub 2005 Dec 13. . [DOI] [PubMed] [Google Scholar]

[pone.0294557.ref007] 7.Seymour CW, Gesten F, Prescott HC, Friedrich ME, Iwashyna TJ, Phillips GS, et al. Time to Treatment and Mortality during Mandated Emergency Care for Sepsis. N Engl J Med. 2017. Jun 8;376(23):2235–2244. doi: 10.1056/NEJMoa1703058 Epub 2017 May 21. . [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0294557.ref008] 8.Machado FR, Azevedo LCP. Sepsis: A Threat That Needs a Global Solution. Crit Care Med. 2018. Mar;46(3):454–459. doi: 10.1097/CCM.0000000000002899 . [DOI] [PubMed] [Google Scholar]

[pone.0294557.ref009] 9.Smits M, Rutten M, Keizer E, Wensing M, Westert G, Giesen P. The Development and Performance of After-Hours Primary Care in the Netherlands: A Narrative Review. Ann Intern Med. 2017. May 16;166(10):737–742. doi: 10.7326/M16-2776 Epub 2017 Apr 18. . [DOI] [PubMed] [Google Scholar]

[pone.0294557.ref010] 10.Loots FJ, Smits M, van Steensel C, Giesen P, Hopstaken RM, van Zanten ARH. Management of sepsis in out-of-hours primary care: a retrospective study of patients admitted to the intensive care unit. BMJ Open. 2018. Sep 17;8(9):e022832. doi: 10.1136/bmjopen-2018-022832 . [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0294557.ref011] 11.Smits M, Plat E, Alink E, de Vries M, Apotheker M, van Overdijk S, et al. Reliability and validity of the Netherlands Triage Standard in emergency care settings: a case scenario study. Emerg Med J. 2022. Aug;39(8):623–627. doi: 10.1136/emermed-2021-211359 Epub 2022 Feb 8. . [DOI] [PubMed] [Google Scholar]

[pone.0294557.ref012] 12.Akaike, H. (1973), "Information theory and an extension of the maximum likelihood principle", in Petrov, B. N.; Csáki, F. (eds.), 2nd International Symposium on Information Theory, Tsahkadsor, Armenia, USSR, September 2–8, 1971, Budapest: Akadémiai Kiadó, pp. 267–281. Republished in Kotz, S.; Johnson, N. L., eds. (1992), Breakthroughs in Statistics, vol. I, Springer-Verlag, pp. 610–624.

[pone.0294557.ref013] 13.Chen T, Guestrin C. Xgboost: A scalable tree boosting system. InProceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining 2016 13 August (pp. 785–794).

[pone.0294557.ref014] 14.Falbel D, Luraschi J (2022). torch: Tensors and Neural Networks with ’GPU’ Acceleration. https://torch.mlverse.org/docs, https://github.com/mlverse/torch.

[pone.0294557.ref015] 15.Fleuren LM, Klausch TLT, Zwager CL, Schoonmade LJ, Guo T, Roggeveen LF, et al. Machine learning for the prediction of sepsis: a systematic review and meta-analysis of diagnostic test accuracy. Intensive Care Med. 2020. Mar;46(3):383–400. doi: 10.1007/s00134-019-05872-y Epub 2020 Jan 21. . [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0294557.ref016] 16.Goh KH, Wang L, Yeow AYK, Poh H, Li K, Yeow JJL, et al. Artificial intelligence in sepsis early prediction and diagnosis using unstructured data in healthcare. Nat Commun. 2021. Jan 29;12(1):711. doi: 10.1038/s41467-021-20910-4 . [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0294557.ref017] 17.Jolley RJ, Quan H, Jetté N, Sawka KJ, Diep L, Goliath J, et al. Validation and optimisation of an ICD-10-coded case definition for sepsis using administrative health data. BMJ Open. 2015. Dec 23;5(12):e009487. doi: 10.1136/bmjopen-2015-009487 . [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0294557.ref018] 18.Loots FJ, Smits M, Hopstaken RM, Jenniskens K, Schroeten FH, van den Bruel A, et al. New clinical prediction model for early recognition of sepsis in adult primary care patients: a prospective diagnostic cohort study of development and external validation. Br J Gen Pract. 2022. May 26;72(719):e437–e445. doi: 10.3399/BJGP.2021.0520 . [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0294557.ref019] 19.Christodoulou E, Ma J, Collins GS, Steyerberg EW, Verbakel JY, Van Calster B. A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models. J Clin Epidemiol. 2019. Jun;110:12–22. doi: 10.1016/j.jclinepi.2019.02.004 Epub 2019 Feb 11. . [DOI] [PubMed] [Google Scholar]

PERMALINK

Predicting sepsis-related mortality and ICU admissions from telephone triage information of patients presenting to out-of-hours GP cooperatives with acute infections: A cohort study of linked routine care databases

Feike J Loots

Marleen Smits

Kevin Jenniskens

Artuur M Leeuwenberg

Paul H J Giesen

Lotte Ramerman

Robert Verheij

Arthur R H van Zanten

Roderick P Venekamp

Roles

Abstract

Background

Methods

Results

Conclusion

Background

Methods

Design

Population and setting

Data collection

Fig 1. Flowchart of included patients and linkage of data from different sources.

Outcome measures

Primary outcome

Secondary outcome

Predictors

Ethics approval

Analyses

Results

Patient characteristics

Table 1. Patient characteristics stratified by sepsis-related adverse outcome (yes vs no), and for the total study population.

Prediction models

Fig 2. Receiver operating characteristic (ROC) curves of the different models.

Fig 3. Precision-recall curves of the different models.

Table 2. Discrimination and calibration of the different models.

Table 3. Specificity, positive predictive value (PPV) and negative predictive value (NPV) at different threshold of predicted probability.

Analyses of subgroups based on type of contact (clinic consultations versus home visits)

Sensitivity analysis

Discussion

Summary of findings

Comparison with previous research

Strengths and limitations

Implications for practice and further research

Supporting information

Data Availability

Funding Statement

References

Decision Letter 0

Marcelo Arruda Nakazone

Roles

Author response to Decision Letter 0

Decision Letter 1

Marcelo Arruda Nakazone

Roles

Acceptance letter

Marcelo Arruda Nakazone

Roles

Associated Data

Supplementary Materials

Data Availability Statement

ACTIONS

PERMALINK

RESOURCES

Similar articles

Cited by other articles

Links to NCBI Databases