Skip to main content
NCBI home page
PLOS One logoLink to PLOS One
. 2021 Feb 22;16(2):e0247624. doi: 10.1371/journal.pone.0247624

Added value of clinical prediction rules for bacteremia in hemodialysis patients: An external validation study

Sho Sasaki 1,2,3,*,#, Yoshihiko Raita 4,5,#, Minoru Murakami 6, Shungo Yamamoto 3,7, Kentaro Tochitani 3,7, Takeshi Hasegawa 8,9,10, Kiichiro Fujisaki 1, Shunichi Fukuhara 10,11,12
Editor: Tatsuo Shimosawa13
PMCID: PMC7899347  PMID: 33617601

Abstract

Introduction

Having developed a clinical prediction rule (CPR) for bacteremia among hemodialysis (HD) outpatients (BAC-HD score), we performed external validation.

Materials & methods

Data were collected on maintenance HD patients at two Japanese tertiary-care hospitals from January 2013 to December 2015. We enrolled 429 consecutive patients (aged ≥ 18 y) on maintenance HD who had had two sets of blood cultures drawn on admission to assess for bacteremia. We validated the predictive ability of the CPR using two validation cohorts. Index tests were the BAC-HD score and a CPR developed by Shapiro et al. The outcome was bacteremia, based on the results of the admission blood cultures. For added value, we also measured changes in the area under the receiver operating characteristic curve (AUC) using logistic regression and Net Reclassification Improvement (NRI), in which each CPR was added to the basic model.

Results

In Validation cohort 1 (360 subjects), compared to a Model 1 (Basic Model) AUC of 0.69 (95% confidence interval [95% CI]: 0.59–0.80), the AUC of Model 2 (Basic model + BAC-HD score) and Model 3 (Basic model + Shapiro’s score) increased to 0.8 (95% CI: 0.71–0.88) and 0.73 (95% CI: 0.63–0.83), respectively. In validation cohort 2 (96 subjects), compared to a Model 1 AUC of 0.81 (95% CI: 0.68–0.94), the AUCs of Model 2 and Model 3 increased to 0.83 (95% CI: 0.72–0.95) and 0.85 (95% CI: 0.76–0.94), respectively. NRIs on addition of the BAC-HD score and Shapiro’s score were 0.3 and 0.06 in Validation cohort 1, and 0.27 and 0.13, respectively, in Validation cohort 2.

Conclusion

Either the BAC-HD score or Shapiro’s score may improve the ability to diagnose bacteremia in HD patients. Reclassification was better with the BAC-HD score.

Introduction

Bacteremia is a disease with a high mortality rate [1–4]. Early diagnosis and treatment are keys to improving prognosis. However, due to the variety of clinical presentations of bacteremia, it is not always first on the list of possible diagnoses. In this context, several clinical predictive models (CPRs) of bacteremia in the general population have been developed [5–8]. Among these models, those developed by Shapiro et al. (Shapiro’s model) [9] have been widely validated and internationally recognized [10–12].

Patients on hemodialysis (HD) are known to have a higher morbidity and mortality rate from bacteremia compared to the general population. Previous cohort studies have shown that the incidence of bacteremia in patients on maintenance HD is 10.40–18.98 per 100 person-years [13–16], which is much higher than the incidence in the general population of 0.22 per 100 person-years [1]. The annual mortality due to sepsis, a severe complication of bacteremia, in HD patients is 50–100 times higher than that of the general population [17, 18].

The most frequent cause of bacteremia in the general population is urinary tract infections caused by Escherichia coli [19–22], whereas the most frequent cause in HD is Staphylococcus aureus [23–25]. Since many blood test findings in HD patients are often affected by dialysis, the accuracy of items included in existing CPRs developed in the general population, such as blood counts and serum creatinine levels, may be greatly affected.

[12]. In order to address the peculiarities of bacteremia among patients with HD, we developed a CPR specific to bacteremia in HD patients, the BAC-HD score [26].

Since the external validities of Shapiro’s model [12] and the BAC-HD score in patients on maintenance HD have not been verified, we assessed the diagnostic accuracy of these two prediction models for bacteremia.

Materials and methods

The present study was approved by the ethics committees of Iizuka Hospital (17167–436), Okinawa Prefectural Chubu Hospital (H28-51) and Saku General Hospital (R201701-01). The study was conducted in accordance with the ethical standards of the Declaration of Helsinki. Since all patient information analyzed in this study was retrospective, participants’ written informed consent was not required by the ethics committee. All data were fully anonymized before authors accessed them. We accessed the medical records to obtain data at Okinawa Prefectural Chubu Hospital from February 9th to February 13th, 2017, and at Saku General Hospital from February 24th to 26th, 2017. The study results are reported in accordance with the Standards for Reporting Diagnostic Accuracy (STARD) statement [27].

Study design and participants

We conducted a cross-sectional study of maintenance HD patients at two tertiary-care teaching hospitals.

Data were collected from medical records from January 2013 to December 2015 in each facility. We enrolled consecutive participants on maintenance HD who were aged ≥ 18 y with two sets of blood cultures drawn at admission because of suspicion of bacteremia. Exclusion criteria were participants who met any of the following items: 1) inpatients transferred from another hospital, 2) patients with a vintage of dialysis < 2 months, 3) patients also receiving peritoneal dialysis, and 4) patients receiving HD less than once a week.

Index tests

Two clinical prediction rules (CPR) for bacteremia were adopted as index tests.

The first CPR, the BAC-HD score, consists of the following 5 items: 1) body temperature ≥ 38.3°C, 2) heart rate ≥ 125, 3) C-reactive protein ≥ 10 × 104μg/L, 4) alkaline phosphatase > 6 μkat IU/L, and 5) no prior antibiotic use within the past week. Each item is regarded as 1 point, for a maximum total of 5 points.

Shapiro’s score is composed of “major criteria” defined as: 1) temperature > 39.5°C (3 points), 2) indwelling vascular catheter (2 points) or clinical suspicion of endocarditis (3 points); plus “minor criteria” (1 point each) defined as: 1) temperature 38.3–39.4°C (101–102.9°F), 2) age > 65 y, 3) chills, 4) vomiting, 5) hypotension (systolic blood pressure < 90 mmHg), 6) white blood cell count > 18 × 109/L, 7) bands > 5%, 8) platelets < 150 × 109/L, and 9) creatinine > 176.8 μmol/L.

Reference standard

The reference standard was bacteremia, as per the results of the admission blood cultures. Contamination was defined as: one of the two sets of culture bottles was positive, or cases where certain species of bacteria known to be contaminants, such as diphtheroids, Bacillus spp., Propionibacterium spp., Micrococci, Corynebacterium spp., and coagulase-negative Staphylococci were detected. Finally, an external consensus panel of infectious disease physicians with > 10 y clinical experience and Japanese Board of Infectious Disease certification who were blinded to the present study design determined whether a culture was contaminated or not based on the above definitions and their clinical expertise.

Statistical analysis

Validation cohorts

Since there are no standard criteria for obtaining blood cultures, it was suspected that the selection of subjects may have differed depending on the facility. Therefore, two validation cohorts were set based on the logic that it is desirable to verify the validity at multiple facilities. Validation cohort 1 (360 subjects) and validation cohort 2 (96 subjects) were defined as patients at Okinawa Prefectural Chubu Hospital and Saku General Hospital, respectively.

Descriptive statistics

We analyzed each item with the two CPRs and used proven bacteremia as a reference standard as well as other clinical information, including sex, blood pressure, respiratory rate, hemodialysis vintage, and presence of diabetes mellitus. Continuous and categorical variables are presented as the median (interquartile range: IQR) and number (percentage), respectively (Table 1).

Table 1. Baseline characteristics.
Validation cohort 1 Validation cohort 2
n = 360 n = 96
Bacteremia (-) Bacteremia (+) Missing (n) Bacteremia (-) Bacteremia (+) Missing (n)
n = 323 n = 37 n = 80 n = 16
Age (years), median (IQR) 72 (61–79) 73 (62–79) 0 72 (65–81) 71.5 (68.5–82.5) 0
sex 0
    male 185 (42.7) 16 (43.2) 56 (70) 12 (75)
    female 138 (57.3) 21 (56.8) 24 (30) 4 (25)
vital signs, median (IQR)
    systolic blood pressure (mmHg) 137 (110–153) 140 (109–150) 12 141 (129–160) 134 (118–155) 2
    diastolic blood pressure (mmHg) 70 (60–80) 64 (60–70) 36 76 (63–87) 72 (57–84) 2
    pulse rate (/min) 88 (78–100) 93.5 (83–110) 17 83 (72–94) 86 (74–95) 5
    respiratory rate (/min) 20 (18–24) 20 (18–24) 22 20 (16–25) 18 (16–24) 76
    body temperature (°C) 37.1 (36.5–37.7) 37.6 (36.7–38.9) 12 37.4 (37–38) 38.1 (37.8–38.8) 17
laboratory findings, median (IQR)
    white blood cell count (× 109/L) 8.1 (6–11.3) 9.9 (5.9–13.2) 5 7.5 (5.4–9.4) 8.1 (5.9–14) 14
    platelet count (× 109/L) 168 (128–219) 137 (109–195) 6 148 (116–189) 89 (70–168) 14
    alkaline phosphatase (μkat/L) 4.6 (3.7–5.9) 6.1 (4.0–8.2) 182 4.3 (3.5–5.4) 5.1 (3.2–8.1) 19
    creatinine (μg/L) 539.2 (389.0–751.4) 495.0 (371.3–654.2) 5 610.0 (503.9–751.4) 539.2 (415.5–830.96) 23
    C-reactive protein (× 104μmol/L) 4.2 (1.5–9.9) 6.7 (1.6–15) 45 4.2 (1.3–10.1) 6.0 (2.6–11.6) 17
hemodialysis vintage (months), median (IQR) 48.8 (16.6–116) 49 (18.7–92.8) 17 87 (44.1–207.2) 49.9 (9.1–164) 4
diabetes mellitus 159 (49.2) 19 (51.4) 1 28 (35) 7 (43.8) 1
antibiotic use within 1 week 18 (5.6) 2 (5.4) 13 9 (11.3) 3 (18.8) 1
indwelling venous catheters 27 (8.4) 8 (21.6) 2 19 (23.8) 10 (62.5) 1
clinically suspected bacterial endocarditis 28 (8.7) 8 (21.6) 2 3 (3.8) 4 (25) 1
symptoms
    shaking chills 76 (23.5) 15 (40.5) 2 14 (17.5) 5 (31.3) 1
    vomiting 45 (13.9) 3 (8.1) 2 7 (8.8) 2 (12.5) 1
Open in a new tab

IQR: interquartile range

Basic model

The basic model to assess the value of reclassification of CPRs was conducted using a logistic regression model with explanatory variables of sex (0: female, 1: male), age (y), mean arterial pressure (mmHg), heart rate, body temperature (°C), presence of diabetes mellitus, HD vintage (months), and white blood cell count (× 109/L). These items were selected by clinicians as those typically used when evaluating patients for bacteremia in their daily medical practice. In validation cohort 2, it was clear at the planning stage that the respiratory rate was often missing, so the respiratory rate was excluded from the basic model.

Added value of CPRs

The discriminatory abilities of the basic model (Model 1), the model that added BAC-HD score (Model 2), or the model that added Shapiro’s score (Model 3) to the basic model were assessed by calculating the area under the receiver operating characteristic curve (AUC) using a logistic regression model. Calibration of each model was performed based on the slope and intercept of the calibration plot [28, 29].

The number of patients correctly reclassified by adding the CPR to the basic model is shown using Net Reclassification Improvement (NRI) (Table 2). The prediction probabilities in the three models were stratified based on the tertile of the prediction probabilities of the basic model, as low (< 0.08), intermediate (0.08–0.2), or high (> 0.2).

Table 2. Reclassification table comparing the probability of bacteremia predicted by Model 2 (basic model + BAC-HD score) and Model 3 (basic model + Shapiro’s score).
Validation cohort 1 Model 2: basic modela + BAC-HD score Model 3: basic modela + Shapiro’s score
Model 1: basic modela <8% 8–20% >20% total <8% 8–20% >20% total
Probability group for bacteremia
Participants with bacteremia
< 8% 2 2 0 4 4 0 0 4
8–20% 5 8 7 20 4 12 4 20
> 20% 0 1 12 13 0 3 10 13
Total 7 11 19 37 8 15 14 37
Participants without bacteremia
< 8% 127 11 0 138 114 24 0 138
8–20% 79 63 16 158 55 89 14 158
> 20% 2 15 10 27 0 19 8 27
total 208 89 26 323 169 132 22 323
Validation cohort2 Model2: basic modela + BAC-HD score Model3: basic modela + Shapiro’s score
Model1: basic modela < 8% 8–20% > 20% total < 8% 8–20% > 20% total
Probability group for bacteremia
Participants with bacteremia
< 8% 1 0 0 1 0 1 0 1
8–20% 1 1 2 4 0 4 0 4
> 20% 0 1 10 11 0 2 9 11
total 2 2 12 16 0 7 9 16
Participants without bacteremia
< 8% 30 0 2 32 29 0 3 32
8–20% 17 7 3 27 10 17 0 27
> 20% 0 9 12 21 4 5 12 21
total 47 16 17 80 43 22 15 80
Open in a new tab

Validation cohort 1:net reclassification improvement (NRI) [Model 2] 0.24−0.16+0.30−0.08 = 0.3, NRI [Model 3] 0.11− 0.16 + 0.23 − 0.12 = 0.06

Validation cohort 2: NRI [Model2] 0.13 − 0.13 + 0.33 − 0.06 = −0.27, NRI [Model 3] 0.06 − 0.13 + 0.24 − 0.04 = −0.13

basic model: logistic regression model with explanatory variables of sex (0: female, 1: male), age (y), mean arterial pressure (mmHg), pulse rate (per minute), body temperature (°C), presence of diabetes mellitus, HD vintage, white blood cell count (× 109/L).

Assessment of performance

To evaluate potential cut-off scores, we computed the sensitivity, specificity, likelihood ratio, positive predictive value, and negative predictive value for the CPRs. For brevity, only the values in validation cohort 1 are summarized.

Handling of missing values

All missing values were addressed using multiple imputations by chained equation treated as missing at random; ten imputed datasets were created. Three logistic regression models were conducted on each of 10 datasets and combined with Rubin’s rule.

All statistical analyses were performed using Stata version 15.0 (Stata Corp., College Station, TX, USA).

Results

Study participants

As the final analytic cohort, there were 360 participants and 96 participants in validation cohort 1 and validation cohort 2, respectively, as shown in Fig 1.

Fig 1. Study flow.

Fig 1

Open in a new tab

Added diagnostic value of BAC-HD score and Shapiro’s score

In validation cohort 1, compared to the AUC of 0.69 (95% confidence interval [95% CI]: 0.59−0.80) of Model 1 (Basic model), the AUCs of Model 2 (Basic model + BAC-HD score) and Model 3 (Basic model + Shapiro’s score) increased to 0.8 (95% CI: 0.71−0.88) and 0.73 (95% CI: 0.63−0.83), respectively, as shown in Fig 2. In validation cohort 2, compared to the AUC of 0.81 (95% confidence interval [95% CI]: 0.68−0.94) of Model 1 (Basic model), the AUCs of Model 2 (Basic model + BAC-HD score) and Model 3 (Basic model + Shapiro’s score) increased to 0.83 (95% CI: 0.72–0.95) and 0.85 (95% CI: 0.76−0.94), respectively, as shown in Fig 2.

Fig 2. Receiver operating characteristic curve.

Fig 2

Open in a new tab

A: results for validation cohort 1, B: results for validation cohort 2. Model 1 = basic model, Model 2 = basic model + BAC-HD score, Model 3 = basic model + Shapiro’s score, AUC: area under the receiver operating characteristic curve, 95% CI: 95% confidence interval.

In validation cohort 1, the slopes (intercept) of the calibration plots of Models 1, 2, and 3 were 1.19 (−0.20), 1.09 (−0.01) and 1.15 (−0.01), respectively. In validation cohort 2, the slopes (intercept) of the calibration plots of Models 1, 2 and 3 were 1.03 (−0.01), 1.00 (−0.001) and 0.86 (0.03), respectively. Calibration plots are shown in Fig 3.

Fig 3. Calibration plot.

Fig 3

Open in a new tab

Model 1 = basic model, Model 2 = basic model + BAC-HD score, Model 3 = basic model + Shapiro’s score.

Reclassification by BAC-HD score and Shapiro’s score

In validation cohort 1 (N = 360), 138 of 142 patients predicted to have a low probability of bacteremia as a result of the basic model (Model 1) were not bacteremic (negative predictive value [NPV] = 97.1%). Thirteen of the 40 patients predicted to have a high probability of bacteremia had bacteremia (positive predictive value; PPV 32.5). Adding the BAC-HD score (Model 2) increased the number of patients predicted to be low-probability from 142 to 215, of whom 208 were not bacteremic (NPV 96.7%). The number of patients predicted to have a high probability of bacteremia was increased from 40 to 45, of whom 19 were bacteremic (PPV 42.2%). Adding Shapiro’s score (Model 3) increased the number of patients predicted to be low-probability from 142 to 177, of whom 169 were not bacteremic (NPV 95.4%). The number of patients predicted to be high probability was reduced from 40 to 36, of whom 14 were bacteremic (PPV 38.9%). The NRI values for addition of BAC-HD score and Shapiro’s score were 0.3 and 0.06, respectively, in validation cohort 1. The results for validation cohort 2 are shown in Table 2 and the NRIs for addition of BAC-HD score and Shapiro’s score were 0.27 and 0.13, respectively.

Assessment of performance

The sensitivities, specificities, likelihood ratios and predictive values for possible cut-off scores in CPRs in validation cohort 1 are shown in Table 3.

Table 3. Assessment of performance of BAC-HD and Shapiro’s score in validation cohort 1.

BAC–HD score
Cut–off Sensitivity 95%CI Specificity 95%CI LR+ 95%CI LR– 95%CI PPV 95%CI NPV 95%CI
≥ 4 5.2 (3.2–7.8) 94.6 (93.8–95.3) 1 (0.6–1.5) 1 (1.0–1.0) 9.8 (6.2–14.6) 89.7 (88.7–90.7)
≥ 3 21.1 (17.3–25.4) 92.5 (91.5–93.3) 2.8 (2.3–3.5) 0.9 (0.8–0.9) 24.3 (19.9–29.1) 91.1 (90.1–92)
≥ 2 63.6 (58.8–68.3) 78.2 (76.8–79.6) 2.9 (2.7–3.2) 0.5 (0.4–0.5) 25.1 (22.5–27.8) 94.9 (94.1–95.7)
≥ 1 85.7 (82–89) 35.2 (33.6–36.8) 1.3 (1.3–1.4) 0.4 (0.3–0.5) 13.2 (11.9–14.5) 95.6 (94.3–96.6)
Shapiro’s score
Cut–off Sensitivity 95%CI Specificity 95%CI LR+ 95%CI LR– 95%CI PPV 95%CI NPV 95%CI
≥ 9 5.7 (3.6–8.4) 99.2 (98.8–99.5) 6.9 (4.0–11.9) 1 (0.9–1.0) 44.2 (30.5–58.7) 90.2 (89.2–91.1)
≥ 8 11.3 (8.4–14.8) 97 (96.4–97.6) 3.8 (2.7–5.3) 0.9 (0.9–1.0) 30.3 (23.1–38.2) 90.5 (89.5–91.4)
≥ 7 24.8 (20.7–29.3) 94.7 (93.9–95.4) 4.6 (3.7–5.8) 0.8 (0.8–0.8) 34.7 (29.2–40.5) 91.7 (90.7–92.5)
≥ 6 39.6 (34.8–44.5) 90.9 (89.9–91.9) 4.4 (3.7–5.1) 0.7 (0.6–0.7) 33.3 (29.1–37.7) 92.9 (92.0–93.8)
≥ 5 45.9 (41.0–50.9) 85 (83.8–86.1) 3.1 (2.7–3.5) 0.6 (0.6–0.7) 25.9 (22.8–29.3) 93.2 (92.3–94.1)
≥ 4 54.1 (49.1–59.0) 71.4 (69.9–72.9) 1.9 (1.7–2.1) 0.6 (0.6–0.7) 17.8 (15.7–20.0) 93.1 (92.1–94.1)
≥ 3 86.5 (82.8–89.7) 41.6 (40–43.2) 1.5 (1.4–1.6) 0.3 (0.3–0.4) 14.5 (13.1–16.0) 96.4 (95.4–97.3)
≥ 2 89.2 (85.8–92) 11.7 (10.6–12.8) 1 (1.0–1.0) 0.9 (0.7–1.2) 10.4 (9.4–11.4) 90.4 (87.3–92.9)
Open in a new tab

95%CI: 95% confidence interval, LR+: positive likelihood ration, LR−: negative likelihood ratio, PPV: positive predictive value, NPV: negative predictive value

Discussion

This is the first study to validate and compare the external validity of the prediction rules for bacteremia in HD patients. BAC-HD showed an excellent added value, suggesting that it can be a useful tool for improving the diagnostic ability of bacteremia in dialysis patients. In addition, BAC-HD is a simple CPR consisting of five items, and has the advantage of being highly versatile in clinical settings.

On the other hand, although Shapiro’s score has many components, its added value was inferior to the BAC-HD score. The reason for this was considered to be the effects of three characteristics of patients on maintenance HD. The first is the difference in the bacteremia etiology. Bacteremia in the general population is often due to gram-negative rods (GNR) [19–22], usually due to urinary tract infections (UTI) [1, 30], while patients on HD often have gram-positive cocci (GPC) bacteremia due to cutaneous infections [23–25]. In our experience, UTI and BSI often have different clinical presentations. Therefore, it is considered that Shapiro’s score, a CPR developed in a population with a high rate of UTI, could not predict bacteremia in maintenance HD patients with a high rate of cutaneous infections.

This is consistent with our previous studies showing that the systemic inflammatory response syndrome (SIRS) criteria [31] and the quick Sequential (Sepsis-Related) Organ Failure Assessment (qSOFA) score [32, 33] were not useful in predicting bacteremia in HD patients. Second, the clinical information of patients on maintenance HD such as body weight, vital signs [34], electrolytes, blood urea nitrogen, and creatinine vary greatly between dialysis and non-dialysis days. Since bacteremia is assumed to develop regardless of the timing of dialysis, the predictive ability of these values may be impaired. Third, patients with end-stage renal failure are often immunocompromised and may have different clinical presentations [35].

This study has two strengths. First, since we verified the added value in two validation cohorts, the robustness of the results is likely increased. Second, the basic model was used to show the added value of CPRs. Although some external validation studies only showed discrimination and calibration of the CPR itself, this is not enough to evaluate the degree of improvement in predictive ability [36].

This study also has some limitations. First, since only Japanese patients with HD were included, its validity in other ethnic groups is unknown. On the other hand, AV-fistula (AVF) is used as vascular access in more than 93% [37] of Japanese on HD, satisfying the 65% goal of the fistula first initiative [38]. Since AVF is associated with a lower risk of infection compared to other vascular access methods, especially central venous catheters [39], it seems significant that added value was shown in a population with a high proportion of AVFs, which is the desired result. Second, since this study analyzed retrospective medical data, there are risks of bias caused by missing values or lower measurement accuracy. However, we performed multiple imputations for missing data to minimize such a bias [40]. Prospective studies are needed for better verification. Third, since bands, which were one of the components of Shapiro’s score, were not evaluated, the score may have been underestimated. Since a number of facilities cannot always measure the proportion of bands, such as facilities included in a previous validation study of Shapiro’s score [12], this is considered acceptable in terms of versatility. Since there are studies (including this present study) that have modified the items included in Shapiro’s score, future validation of these is awaited [41]. Fourth, since there were no standard criteria for when to obtain blood cultures, the possibility that there was some degree of arbitrariness in the decision to draw blood cultures cannot be denied. However, since we used data before the development of the BAC-HD score, it is unlikely that the items in the BAC-HD score influenced the decision on whether or not to draw blood cultures. Furthermore, it is possible that some of the patients whose blood cultures were not drawn included cases of bacteremia. Fifth, it is unclear whether blood cultures were collected from a central venous catheter (CVC) that was the site of vascular access. However, since it is unlikely that both sets of blood cultures were collected from the CVC, and the number of CVCs was small, we believe that the effect of this was not significant.

Conclusions

We verified the added value of the BAC-HD score and Shapiro’s score to the usual criteria for predicting bacteremia in HD patients. We suggest that either the BAC-HD score or Shapiro’s score may improve the accuracy of predicting bacteremia in patients on HD. Reclassification was better with the BAC-HD score.

Improving the diagnostic ability is expected to contribute to the early initiation of appropriate treatment and improve the prognosis of bacteremia.

Supporting information

S1 Table. Pathogens involved in bacteremia in validation cohorts 1 and 2.

(DOCX)

Click here for additional data file. (20.9KB, docx)
S1 Data. Validation cohort 1.

(XLS)

Click here for additional data file. (243KB, xls)
S2 Data. Validation cohort 2.

(XLS)

Click here for additional data file. (83.5KB, xls)

Acknowledgments

The authors thank the JOINT-KD collaborators, Ryo Nishioka and Yasunori Suzuki, Department of Rheumatology, Kanazawa University Hospital, Kanazawa and Naomi Ako, Division of Nephrology and Hypertension, Department of Internal Medicine, St. Marianna University School of Medicine, Kawasaki for their intellectual support in the management of this study. We also thank Libby Cone, MD, MA, of DMC Corp. (www.dmed.co.jp) for editing drafts of this manuscript.

Data Availability

All relevant data are within the manuscript and its Supporting Information files.

Funding Statement

No, the authors have not received a specific grant for this study from any funding agency in the public, commercial, or not-for-profit sectors.

References

  • 1.Nielsen SL, Pedersen C, Jensen TG, et al. ; Decreasing incidence rates of bacteremia: a 9-year population-based study. J Infect 2014; 69(1):51–9. 10.1016/j.jinf.2014.01.014 [DOI] [PubMed] [Google Scholar]
  • 2.Ferrer R, Martin-Loeches I, Phillips G, et al. ; Empiric antibiotic treatment reduces mortality in severe sepsis and septic shock from the first hour: results from a guideline-based performance improvement program. Critical care medicine 2014; 42(8):1749–1755. 10.1097/CCM.0000000000000330 [DOI] [PubMed] [Google Scholar]
  • 3.Dellinger RP, Levy MM, Rhodes A, et al. ; Surviving Sepsis Campaign: International Guidelines for Management of Severe Sepsis and Septic Shock: 2012. Critical Care Medicine 2013; 41(2):580–637. 10.1097/CCM.0b013e31827e83af [DOI] [PubMed] [Google Scholar]
  • 4.Levy MM, Artigas A, Phillips GS, et al. ; Outcomes of the Surviving Sepsis Campaign in intensive care units in the USA and Europe: a prospective cohort study. The Lancet infectious diseases 2012; 12(12):919–924. 10.1016/S1473-3099(12)70239-6 [DOI] [PubMed] [Google Scholar]
  • 5.Ratzinger F, Dedeyan M, Rammerstorfer M, et al. ; A risk prediction model for screening bacteremic patients: a cross sectional study. PLoS One 2014; 9(9):e106765 10.1371/journal.pone.0106765 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6.van Werkhoven CH, Huijts SM, Postma DF, et al. ; Predictors of Bacteraemia in Patients with Suspected Community-Acquired Pneumonia. PLoS One 2015; 10(11):e0143817 10.1371/journal.pone.0143817 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Eliakim-Raz N, Bates DW, Leibovici L; Predicting bacteraemia in validated models—a systematic review. Clin Microbiol Infect 2015; 21(4):295–301. 10.1016/j.cmi.2015.01.023 [DOI] [PubMed] [Google Scholar]
  • 8.Takeshima T, Yamamoto Y, Noguchi Y, et al. ; Identifying Patients with Bacteremia in Community-Hospital Emergency Rooms: A Retrospective Cohort Study. PLoS One 2016; 11(3):e0148078 10.1371/journal.pone.0148078 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Shapiro NI, Wolfe RE, Wright SB, et al. ; Who needs a blood culture? A prospectively derived and validated prediction rule. J Emerg Med 2008; 35(3):255–64. 10.1016/j.jemermed.2008.04.001 [DOI] [PubMed] [Google Scholar]
  • 10.Laukemann S, Kasper N, Kulkarni P, et al. ; Can We Reduce Negative Blood Cultures With Clinical Scores and Blood Markers? Results From an Observational Cohort Study. Medicine (Baltimore) 2015; 94(49):e2264 10.1097/MD.0000000000002264 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Jessen MK, Mackenhauer J, Hvass AM, et al. ; Prediction of bacteremia in the emergency department: an external validation of a clinical decision rule. Eur J Emerg Med 2016; 23(1):44–9. 10.1097/MEJ.0000000000000203 [DOI] [PubMed] [Google Scholar]
  • 12.Hodgson LE, Dragolea N, Venn R, et al. ; An external validation study of a clinical prediction rule for medical patients with suspected bacteraemia. Emerg Med J 2016; 33(2):124–9. 10.1136/emermed-2015-204926 [DOI] [PubMed] [Google Scholar]
  • 13.Hoen B, Paul-Dauphin A, Hestin D, et al. ; EPIBACDIAL: a multicenter prospective study of risk factors for bacteremia in chronic hemodialysis patients. J Am Soc Nephrol 1998; 9(5):869–76. [DOI] [PubMed] [Google Scholar]
  • 14.Dopirak M, Hill C, Oleksiw M, et al. ; Surveillance of hemodialysis-associated primary bloodstream infections: the experience of ten hospital-based centers. Infect Control Hosp Epidemiol 2002; 23(12):721–4. 10.1086/502000 [DOI] [PubMed] [Google Scholar]
  • 15.Ishani A, Collins AJ, Herzog CA, et al. ; Septicemia, access and cardiovascular disease in dialysis patients: the USRDS Wave 2 study. Kidney Int 2005; 68(1):311–8. 10.1111/j.1523-1755.2005.00414.x [DOI] [PubMed] [Google Scholar]
  • 16.Fysaraki M, Samonis G, Valachis A, et al. ; Incidence, clinical, microbiological features and outcome of bloodstream infections in patients undergoing hemodialysis. Int J Med Sci 2013; 10(12):1632–8. 10.7150/ijms.6710 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Foley RN, Guo H, Snyder JJ, et al. ; Septicemia in the United States dialysis population, 1991 to 1999. J Am Soc Nephrol 2004; 15(4):1038–45. 10.1097/01.asn.0000119144.95922.c4 [DOI] [PubMed] [Google Scholar]
  • 18.Sarnak MJ, Jaber BL; Mortality caused by sepsis in patients with end-stage renal disease compared with the general population. Kidney Int 2000; 58(4):1758–64. 10.1111/j.1523-1755.2000.00337.x [DOI] [PubMed] [Google Scholar]
  • 19.Fontanarosa PB, Kaeberlein FJ, Gerson LW, et al. ; Difficulty in predicting bacteremia in elderly emergency patients. Ann Emerg Med 1992; 21(7):842–8. 10.1016/s0196-0644(05)81032-7 [DOI] [PubMed] [Google Scholar]
  • 20.Meyers BR, Sherman E, Mendelson MH, et al. ; Bloodstream infections in the elderly. Am J Med 1989; 86(4):379–84. 10.1016/0002-9343(89)90333-1 [DOI] [PubMed] [Google Scholar]
  • 21.Elhanan G, Raz R, Pitlik SD, et al. ; Bacteraemia in a community and a university hospital. J Antimicrob Chemother 1995; 36(4):681–95. 10.1093/jac/36.4.681 [DOI] [PubMed] [Google Scholar]
  • 22.Khayr WF, CarMichael MJ, Dubanowich CS, et al. ; Epidemiology of bacteremia in the geriatric population. Am J Ther 2003; 10(2):127–31. 10.1097/00045391-200303000-00008 [DOI] [PubMed] [Google Scholar]
  • 23.Nielsen LH, Jensen-Fangel S, Benfield T, et al. ; Risk and prognosis of Staphylococcus aureus bacteremia among individuals with and without end-stage renal disease: a Danish, population-based cohort study. BMC Infect Dis 2015; 15:6 10.1186/s12879-014-0740-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Girndt M; Bacteraemia in haemodialysis patients—not always Staphylococcus aureus. Nephrol Dial Transplant 2015; 30(7):1055–7. 10.1093/ndt/gfv223 [DOI] [PubMed] [Google Scholar]
  • 25.Vandecasteele SJ, Boelaert JR, De Vriese AS ; Staphylococcus aureus infections in hemodialysis: what a nephrologist should know. Clin J Am Soc Nephrol 2009; 4(8):1388–400. 10.2215/CJN.01590309 [DOI] [PubMed] [Google Scholar]
  • 26.Sasaki S, Hasegawa T, Kawarazaki H, et al. ; Development and Validation of a Clinical Prediction Rule for Bacteremia among Maintenance Hemodialysis Patients in Outpatient Settings. PLoS One 2017; 12(1):e0169975 10.1371/journal.pone.0169975 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27.Bossuyt PM, Reitsma JB, Bruns DE, et al. ; STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies. Bmj 2015; 351:h5527 10.1136/bmj.h5527 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28.Hosmer DW Jr, Lemeshow S, Sturdivant RX. Applied logistic regression. John Wiley & Sons, Hoboken, NJ: 2013;chapter 5: 144–167 [Google Scholar]
  • 29.Steyerberg EW, Vickers AJ, Cook NR, et al. ; Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology 2010; 21(1):128–38. 10.1097/EDE.0b013e3181c30fb2 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30.Arpi M, Renneberg J, Andersen HK, et al. ; Bacteremia at a Danish university hospital during a twenty-five-year period (1968–1992). Scand J Infect Dis 1995; 27(3):245–51. 10.3109/00365549509019017 [DOI] [PubMed] [Google Scholar]
  • 31.Uchida D, Sasaki S, Kawarazaki H, et al. ; Systemic Inflammatory Response Syndrome Is Not an Indicator of Bacteremia in Hemodialysis Patients With Native Accesses: A Multicenter Study. Asaio j 2017; 63(4):501–506. 10.1097/MAT.0000000000000493 [DOI] [PubMed] [Google Scholar]
  • 32.Freund Y, Lemachatti N, Krastinova E, et al. ; Prognostic Accuracy of Sepsis-3 Criteria for In-Hospital Mortality Among Patients With Suspected Infection Presenting to the Emergency Department. Jama 2017; 317(3):301–308. 10.1001/jama.2016.20329 [DOI] [PubMed] [Google Scholar]
  • 33.Nishiwaki H, Sasaki S, Hasegawa T, et al. ; External validation of the quick Sequential Organ Failure Assessment score for mortality and bacteraemia risk evaluation in Japanese patients undergoing haemodialysis: a retrospective multicentre cohort study. BMJ Open 2019; 9(7):e028856 10.1136/bmjopen-2018-028856 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34.Oosugi K, Fujimoto N, Dohi K, et al. ; Hemodynamic and pathophysiological characteristics of intradialytic blood pressure elevation in patients with end-stage renal disease. Hypertens Res 2014; 37(2):158–65. 10.1038/hr.2013.123 [DOI] [PubMed] [Google Scholar]
  • 35.Hauser AB, Stinghen AE, Kato S, et al. ; Characteristics and causes of immune dysfunction related to uremia and dialysis. Perit Dial Int 2008; 28 Suppl 3:S183–7. [PubMed] [Google Scholar]
  • 36.Moons KG, de Groot JA, Linnet K, et al. ; Quantifying the added value of a diagnostic test or marker. Clin Chem 2012; 58(10):1408–17. 10.1373/clinchem.2012.182550 [DOI] [PubMed] [Google Scholar]
  • 37.Pisoni RL, Zepel L, Fluck R, et al. ; International Differences in the Location and Use of Arteriovenous Accesses Created for Hemodialysis: Results From the Dialysis Outcomes and Practice Patterns Study (DOPPS). American Journal of Kidney Diseases 2018; 71(4):469–478. 10.1053/j.ajkd.2017.09.012 [DOI] [PubMed] [Google Scholar]
  • 38.Gold JA, Hoffman K; Fistula First: the National Vascular Access Improvement Initiative. Wmj 2006; 105(3):71–3. [PubMed] [Google Scholar]
  • 39.Ravani P, Gillespie BW, Quinn RR, et al. ; Temporal risk profile for infectious and noninfectious complications of hemodialysis access. J Am Soc Nephrol 2013; 24(10):1668–77. 10.1681/ASN.2012121234 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 40.van der Heijden GJ, Donders AR, Stijnen T, et al. ; Imputation of missing values is superior to complete case analysis and the missing-indicator method in multivariable diagnostic research: a clinical example. J Clin Epidemiol 2006; 59(10):1102–9. 10.1016/j.jclinepi.2006.01.015 [DOI] [PubMed] [Google Scholar]
  • 41.Wyss G, Berger S, Haubitz S, et al. ; The Shapiro–Procalcitonin algorithm (SPA) as a decision tool for blood culture sampling: validation in a prospective cohort study. Infection 2020; 48: 523–533 10.1007/s15010-020-01423-6 [DOI] [PubMed] [Google Scholar]

Decision Letter 0

Tatsuo Shimosawa

9 Dec 2020

PONE-D-20-33423

Added value of clinical prediction rules for bacteremia in hemodialysis patients: An external validation study

PLOS ONE

Dear Dr. Sasaki,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

The topic is potentially interesting and important issue.  The three experts raised some concerns and points to be clearly described.

Please submit your revised manuscript by Jan 23 2021 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

  • A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

  • A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

  • An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

We look forward to receiving your revised manuscript.

Kind regards,

Tatsuo Shimosawa, M.D., Ph.D.

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Thank you for stating in the text of your manuscript The present study was approved by the ethics committee of Iizuka Hospital (17167-436), Okinawa Prefectural Chubu Hospital (H28-51) and Saku General Hospital (R201701-01). The study was conducted in accordance with the ethical standards of the Declaration of Helsinki." Please also add this information to your ethics statement in the online submission form.

3. Please confirm whether ethical approval was obtained for the original cohort and for the validation cohorts, or whether there was a different ethical approval for cohorts 1&2.

4. Thank you for stating in your manuscript "Since all patient information analyzed in this study was retrospective, participants’ written informed consent was not obtained." In your ethics statement in the Methods section and in the online submission form, please clarify whether all data were fully anonymized before or after you accessed them and/or whether the IRB or ethics committee waived the requirement for informed consent.

5. Please include the date(s) on which you accessed the databases or records to obtain the data used in your study.

6. We note that you have indicated that data from this study are available upon request. PLOS only allows data to be available upon request if there are legal or ethical restrictions on sharing data publicly. For information on unacceptable data access restrictions, please see http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions.

In your revised cover letter, please address the following prompts:

a) If there are ethical or legal restrictions on sharing a de-identified data set, please explain them in detail (e.g., data contain potentially identifying or sensitive patient information) and who has imposed them (e.g., an ethics committee). Please also provide contact information for a data access committee, ethics committee, or other institutional body to which data requests may be sent.

b) If there are no restrictions, please upload the minimal anonymized data set necessary to replicate your study findings as either Supporting Information files or to a stable, public repository and provide us with the relevant URLs, DOIs, or accession numbers. Please see http://www.bmj.com/content/340/bmj.c181.long for guidelines on how to de-identify and prepare clinical data for publication. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories.

We will update your Data Availability statement on your behalf to reflect the information you provide.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

Reviewer #2: No

Reviewer #3: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Dear authors,

Predicting bacteria in in HD patients is important clinical tool to care for this patient population. The methodology is standard and statistical analysis is robust. Minor comments are as follows.

1- Why was the study design as a basic model and added models? Why not the comparison between the 2 final models? Was the basic model validated to start with? (To assess the value of reclassification)

2- A typo in second line of introduction (Key should be keys).

3- How does the lack of standard criteria for blood cultures make the 2 cohorts different in characteristics?

4- How did the expert panel exclude the contamination from retrospective data?

5- Please explain why difference in etiology (Gram-negative vs Gram-positive) between genera population and HD patient will render Shapro's score ineffective in predicting bacteremia.

6- Please explain the calibration plot.

Regards

Reviewer #2: This is an interesting study addressing the external validation of a novel clinical prediction rule (CPR) for bacteremia (BAC-HD score) which the authors have previously developed specifically for hemodialysis (HD) patients. They compared the diagnostic accuracy of BAC-HD score and the Shapiro’s criteria and reported that the former was better in the discrimination ability to predict bacteremia than the latter. The article is well drafted and certainly provides valuable information to supplement clinical judgment and treatment decision in the dialysis unit. The reviewer has a few comments as follows:

1) The indications for obtaining blood cultures are not presented, and the patient selection for the two cohorts cannot escape a certain degree of arbitrariness. There may have been patients with bacteremia who did not have a blood culture obtained and therefore were not included I the patient population. This point should be clearly stated as limitation in the manuscript.

2) The authors refer to the uniqueness of bacteria etiology of HD patients. Can the authors present data of the frequency of the organisms actually cultured from the blood culture of the patients? Can the authors present data concerning the suspected infectious foci of the patients as well?

3) Concerning the ‘false negative’ cases (i.e. score did not suggest a culture but the patient was found to be bacteremic), there seems to be seven patients in Model 2 and eight in Model 2. Do these patients of the two Models show considerable overlapping? Can the authors describe the clinical circumstances of these patients and comment on the reason why the patients were missed by the prediction rule?

Reviewer #3: This study intends to provide exteral validation to a recently described prediction model for diagnosing bacteremia in hemodialysis patients. Further, the diagnostic yield of the new model is compared to an established model by Shapiro et al.

Since bacteremia is an important problem and the diagnosis is not straight forward, this approach is relevant and interesting. The paper is well written.

The study is retrospective in design, two validation cohorts were formed at two different hospitals. Patients were included if they were on maintenance hemodialysis and two blood cultures were drawn at hospital admission. The prediction models were compared with the results of the blood cultures.

Both additive models improved prediction of positive blood cultures. The authors claim that their new score performed better than Shapiro's rule, however, the data suggest that both rules add similarly to the basic model. The authors should check if they really can assume superiority.

Some suggestions to the authors:

1) line 61: fortunately bacteremia is not identical with sepsis, although it may lead to this severe complication. The mortality rates cited in #18 and #19 are for Sepsis.

2) lines 65 f: please consider rewording, this sentence is hard to understand. Further, at least with CVC the time point of dialysis is most likely irrelevant for the diagnostic yield of blood cultures when drawn from the catheter.

3) Would the addition of the condition "CVC present" further improve the BAC-HD score? This seems most likely given the numbers in Tbl. 1

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

Reviewer #3: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

PLoS One. 2021 Feb 22;16(2):e0247624. doi: 10.1371/journal.pone.0247624.r002

Author response to Decision Letter 0

22 Jan 2021

RESPONSE TO EDITOR

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

RESPONSE: Thank you for your very clear guidance. We have referred to the guide and revised the manuscript accordingly.

2. Thank you for stating in the text of your manuscript The present study was approved by the ethics committee of Iizuka Hospital (17167-436), Okinawa Prefectural Chubu Hospital (H28-51) and Saku General Hospital (R201701-01). The study was conducted in accordance with the ethical standards of the Declaration of Helsinki." Please also add this information to your ethics statement in the online submission form.

RESPONSE: Thank you for reminding us of this; we will include this on the submission form,.

3. Please confirm whether ethical approval was obtained for the original cohort and for the validation cohorts, or whether there was a different ethical approval for cohorts 1&2.

RESPONSE: The validation cohort and the original cohort were separately approved by the ethics committees of the relevant institutions. In this article, we have provided the approval numbers for the validation cohort.

4. Thank you for stating in your manuscript "Since all patient information analyzed in this study was retrospective, participants’ written informed consent was not obtained." In your ethics statement in the Methods section and in the online submission form, please clarify whether all data were fully anonymized before or after you accessed them and/or whether the IRB or ethics committee waived the requirement for informed consent. Please include the date(s) on which you accessed the databases or records to obtain the data used in your study.

RESPONSE: Thank you for your suggestion. We have revised the sentence in the Methods section. We will also include additional information on the submission form.

Revised sentences (lines 83–87):

Since all patient information analyzed in this study was retrospective, participants’ written informed consent was not required by the ethics committee. All data were fully anonymized before authors accessed them. We accessed the medical records at Okinawa Prefectural Chubu Hospital from February 9th to February 13th, 2017, and at Saku General Hospital from February 24th to 26th, 2017.

5. We note that you have indicated that data from this study are available upon request. PLOS only allows data to be available upon request if there are legal or ethical restrictions on sharing data publicly. For information on unacceptable data access restrictions, please see http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions.

RESPONSE:

We have obtained permission to provide the minimal anonymized datasets from the ethical committees of participating facilities.

REVIEWER 1:

1- Why was the study design as a basic model and added models? Why not the comparison between the 2 final models? Was the basic model validated to start with? (To assess the value of reclassification)

REPONSE: We wish to express our appreciation to Reviewer 1 for his or her insightful comments, which have helped us significantly improve the paper.

We appreciate your raising this important point. As you pointed out, we created a basic model to assess the added value of clinical prediction rules (CPRs). We referred to previous studies (Moons, 2012 #12226), (Takada, 2020 #12228) as a basis for considering that validation of the basic model is not necessary. We have revised line 141 as below:

Revised sentence (lines 142–145):

The basic model to assess the value of reclassification of CPRs was conducted using a logistic regression model with explanatory variables of sex (0: female, 1: male), age (y), mean arterial pressure (mmHg), heart rate, body temperature (°C), presence of diabetes mellitus, HD vintage (months), and white blood cell count (× 109/L).

2- A typo in second line of introduction (Key should be keys).

RESPONSE:

We apologize for our mistake and have corrected it (Line 55).

3- How does the lack of standard criteria for blood cultures make the 2 cohorts different in characteristics?

RESPONSE:

We appreciate your raising this important point. Unfortunately, it is difficult to provide data on the standard procedures for the drawing of blood cultures at different facilities. Clinically, the threshold for blood cultures being drawn varies among institutions. We have modified line 123 as follows:

Revised sentence (line 128):

Since there are no standard criteria for obtaining blood cultures, it was suspected that the selection of subjects may have differed depending on the facility.

4- How did the expert panel exclude the contamination from retrospective data?

RESPONSE:

We appreciate your important point. We apologize for the lack of explanation in the text. The expert panel, which was blinded to the study design, used the results of the blood cultures, as well as basic patient information and blood test results, to determine contamination. We have revised lines 117–121 accordingly:

Revised sentences (lines 119–123):

Finally, an external consensus panel of infectious disease physicians with > 10 y clinical experience and Japanese Board of Infectious Disease certification who were blinded to the present study design determined whether a culture was contaminated or not based on the above definitions and their clinical expertise.

5- Please explain why difference in etiology (Gram-negative vs Gram-positive) between genera population and HD patient will render Shapiro's score ineffective in predicting bacteremia.

RESPONSE:

We appreciate Reviewer 1’s pointing out this important issue. Unfortunately, we have not found adequate evidence that different species or foci of infection have different clinical presentations. We have revised accordingly:

Revised text (lines 227–230):

In our experience, UTIs and BSIs often have different clinical presentations. We therefore considered that Shapiro’s score, a CPR developed in a population with a high rate of UTIs, would be unable to predict bacteremia in maintenance HD patients with a high rate of cutaneous infections.

6- Please explain the calibration plot.

We apologize to reviewer 1 for the lack of explanation. The calibration plot is a diagram for evaluating the calibration of a logistic regression model, checking the difference between the observed probability and the probability estimated by the regression model. Therefore, the closer the slope of the plot is to 1 and the closer the intercept is to 0, the better the calibration is.

REVIEWER 2:

1) The indications for obtaining blood cultures are not presented, and the patient selection for the two cohorts cannot escape a certain degree of arbitrariness. There may have been patients with bacteremia who did not have a blood culture obtained and therefore were not included I the patient population. This point should be clearly stated as limitation in the manuscript.

We wish to express our appreciation to Reviewer 2 for his or her insightful comments, which have helped us significantly improve the paper.

RESPONSE TO REVIEWER 2:

We appreciate your important point. We added the following to the limitations (line 261–267) as suggested by reviewer 2:

Fourth, since there were no standard criteria for when to obtain blood cultures, the possibility that there was some degree of arbitrariness to the decision to draw blood cultures cannot be denied. However, since we used data before the development of the BAC-HD score, the items in the BAC-HD score could not have influenced the decision on whether or not to draw blood cultures. Furthermore, it is possible that some of the patients whose blood cultures were not drawn included cases of bacteremia.

2) The authors refer to the uniqueness of bacteria etiology of HD patients. Can the authors present data of the frequency of the organisms actually cultured from the blood culture of the patients? Can the authors present data concerning the suspected infectious foci of the patients as well?

RESPONSE:

We thank reviewer 2 for these important points. We have described the frequency of the causative organism in S1 Table. Unfortunately, we do not have reliable data on suspected infectious foci due to the fact that this is a retrospective study, so we are unable to provide data on this.

New table:

S1 Table. Pathogens involved in bacteremia in validation cohorts 1 and 2

Validation cohort 1

n = 37 Validation cohort 2

n = 16 Total

n = 53

Staphylococcus aureus (S. aureus) 9 (24.4) 9 (56.2) 18 (34.0)

[methicillin-resistant S. aureus] 5 2 7

coagulase negative Staphylococci 1 (2.7) 2 (12.5) 3 (5.7)

Streptococcus spp. 2 (5.4) 2 (12.5) 4 (7.5)

Enterococcus spp. 2 (5.4) 2 (12.5) 4 (7.5)

Escherichia coli 11 (29.7) 0 11 (20.8)

Klebsiella pneumoniae 4 (10.8) 1 (6.3) 5 (9.4)

Others 8 (21.6) 0 8 (15.1)

3) Concerning the ‘false negative’ cases (i.e. score did not suggest a culture but the patient was found to be bacteremic), there seems to be seven patients in Model 2 and eight in Model 2. Do these patients of the two Models show considerable overlapping? Can the authors describe the clinical circumstances of these patients and comment on the reason why the patients were missed by the prediction rule?

RESPONSE:

We thank Reviewer 2 for these important questions. Two of the false-negative cases overlapped. Unfortunately, we could not find any similarities between these cases, and the reason why the predictive scores showed false negatives is unknown.

REVIEWER 3:

This study intends to provide external validation to a recently described prediction model for diagnosing bacteremia in hemodialysis patients. Further, the diagnostic yield of the new model is compared to an established model by Shapiro et al.

Since bacteremia is an important problem and the diagnosis is not straight forward, this approach is relevant and interesting. The paper is well written.

The study is retrospective in design, two validation cohorts were formed at two different hospitals. Patients were included if they were on maintenance hemodialysis and two blood cultures were drawn at hospital admission. The prediction models were compared with the results of the blood cultures.

Both additive models improved prediction of positive blood cultures. The authors claim that their new score performed better than Shapiro's rule, however, the data suggest that both rules add similarly to the basic model. The authors should check if they really can assume superiority.

We wish to express our appreciation to Reviewer 3 for his or her insightful comments, which have helped us significantly improve the paper.

We appreciate your important point. As you noted, the interpretation of the results was incorrect. The following corrections have been made:

Revised sentence (lines 47–49):

Either the BAC-HD score or Shapiro’s score may improve the ability to diagnose bacteremia in HD patients. Reclassification was better with the BAC-HD score.

Revised sentence (lines 274–276):

We suggest that either the BAC-HD score or Shapiro’s score may improve the accuracy of predicting bacteremia in patients on HD. Reclassification was better with the BAC-HD score.

Some suggestions to the authors:

1) line 61: fortunately bacteremia is not identical with sepsis, although it may lead to this severe complication. The mortality rates cited in #18 and #19 are for Sepsis.

RESPONSE:

We appreciate the important point made by reviewer 3. We apologize for our inaccurate description. We have revised line 65 as follows:

The annual mortality due to sepsis, a severe complication of bacteremia, in HD patients is 50–100 times higher than that of the general population

2) lines 65 f: please consider rewording, this sentence is hard to understand. Further, at least with CVC the time point of dialysis is most likely irrelevant for the diagnostic yield of blood cultures when drawn from the catheter.

RESPONSE:

We appreciate your important point. We have revised lines 69–72. In addition, we added sentences to the limitations (line 257) that blood cultures may have been taken from the CVC. However, in our opinion, since only a small number of Japanese dialysis patients use CVC, the impact of CVC on the present study was small.

Revised sentence (lines 69–72):

Since many blood test findings in HD patients are often affected by dialysis, the accuracy of items included in existing CPRs developed in the general population, such as blood counts and serum creatinine levels, may be greatly affected

New sentence (lines 267–271):

Fifth, it is unclear whether blood cultures were collected from a central venous catheter (CVC) that was the site of vascular access. However, since it is unlikely that both sets of blood cultures were collected from the CVC, and the number of CVCs was small, we believe that the effect of this was not significant.

3) Would the addition of the condition "CVC present" further improve the BAC-HD score? This seems most likely given the numbers in Tbl. 1

RESPONSE:

Thank you for pointing out this very important issue. In the BAC-HD development paper, we used “CVC use” as a candidate predictor, but it did not remain in the final model. Since the purpose of this study was to test external validity, we did not test the effect of CVC use on improving predictive ability. There are two reasons why CVC use was not included in the final model even though it is an important predictor, the first being that it was incompatible with other variables in multivariate models, and the second that the number of CVCs used was too small.

Decision Letter 1

Tatsuo Shimosawa

10 Feb 2021

Added value of clinical prediction rules for bacteremia in hemodialysis patients: An external validation study

PONE-D-20-33423R1

Dear Dr. Sasaki,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Tatsuo Shimosawa, M.D., Ph.D.

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Dear authors,

Thank you for addressing my previous comments. I do not have any other comments at this point.

Best,

Reviewer #2: (No Response)

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Islam M. Ghazi, PharmD

Reviewer #2: No

Acceptance letter

Tatsuo Shimosawa

12 Feb 2021

PONE-D-20-33423R1

Added value of clinical prediction rules for bacteremia in hemodialysis patients: An external validation study

Dear Dr. Sasaki:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Prof. Tatsuo Shimosawa

Academic Editor

PLOS ONE

Associated Data

    This section collects any data citations, data availability statements, or supplementary materials included in this article.

    Supplementary Materials

    S1 Table. Pathogens involved in bacteremia in validation cohorts 1 and 2.

    (DOCX)

    Click here for additional data file. (20.9KB, docx)
    S1 Data. Validation cohort 1.

    (XLS)

    Click here for additional data file. (243KB, xls)
    S2 Data. Validation cohort 2.

    (XLS)

    Click here for additional data file. (83.5KB, xls)

    Data Availability Statement

    All relevant data are within the manuscript and its Supporting Information files.

    Articles from PLoS ONE are provided here courtesy of PLOS

    RESOURCES