In patients with Staphylococcus aureus bacteremia, VIRSTA can be used to rule out endocarditis, but at the expense of a high number of patients classified as high risk and thus requiring TEE. The POSITIVE and PREDICT scores require adjustment.
Keywords: Staphylococcus aureus bacteremia, endocarditis, risk stratification, echocardiography
Abstract
Background
Staphylococcus aureus bacteremia (SAB) is in 10% to 20% of cases complicated by infective endocarditis. Clinical prediction scores may select patients with SAB at highest risk for endocarditis, improving the diagnostic process of endocarditis. We compared the accuracy of the Prediction Of Staphylococcus aureus Infective endocarditiseTime to positivity, Iv drug use, Vascular phenomena, preExisting heart condition (POSITIVE), Predicting Risk of Endocarditis Using a Clinical Tool (PREDICT), and VIRSTA scores for classifying the likelihood of endocarditis in patients with SAB.
Methods
Between August 2017 and September 2019, we enrolled consecutive adult patients with SAB in a prospective cohort study in 7 hospitals in the Netherlands. Using the modified Duke Criteria for definite endocarditis as reference standard, sensitivity, specificity, negative predictive (NPV), and positive predictive values were determined for the POSITIVE, PREDICT, and VIRSTA scores. An NPV of at least 98% was considered safe for excluding endocarditis.
Results
Of 477 SAB patients enrolled, 33% had community-acquired SAB, 8% had a prosthetic valve, and 11% a cardiac implantable electronic device. Echocardiography was performed in 87% of patients, and 42% received transesophageal echocardiography (TEE). Eighty-seven (18.2%) had definite endocarditis. Sensitivity was 77.6% (65.8%–86.9%), 85.1% (75.8%–91.8%), and 98.9% (95.7%–100%) for the POSITIVE (n = 362), PREDICT, and VIRSTA scores, respectively. NPVs were 92.5% (87.9%–95.8%), 94.5% (90.7%–97.0%), and 99.3% (94.9%–100%). For the POSITIVE, PREDICT, and VIRSTA scores, 44.5%, 50.7%, and 70.9% of patients with SAB, respectively, were classified as at high risk for endocarditis.
Conclusions
Only the VIRSTA score had an NPV of at least 98%, but at the expense of a high number of patients classified as high risk and thus requiring TEE.
Clinical Trials Registration
Netherlands Trial Register code 6669.
Staphylococcus aureus is a common cause of bacteremia [1] and is associated with myriad clinical syndromes, ranging from an uncomplicated central line-associated bacteremia to fulminant endocarditis. In prospective cohort studies, 10%–20% of S aureus bacteremia (SAB) episodes is complicated by infective endocarditis (IE) [2, 3]. Recognition of IE can be difficult, as up to 40% of S aureus infective endocarditis (SA-IE) occurs in patients without known predisposing risk factors for IE [4], and typical signs of endocarditis (eg, heart murmur, embolic event) may be absent on presentation [2–4].
Echocardiography is the preferred imaging modality screen for patients with a clinical suspicion of IE, with transesophageal echocardiography (TEE) having a higher sensitivity than transthoracic echocardiography [5]. However, TEE is an invasive and expensive procedure and not feasible for all patients with SAB in most healthcare settings [6, 7].
There are a number of well-recognized risk factors for IE, and several sets of clinical criteria have been developed to help clinicians decide which patients should undergo TEE [6]. In the past 5 years, 3 multivariate prediction rules have been proposed to classify patients with SAB as being at either high or low risk for SA-IE. The Predicting Risk of Endocarditis Using a Clinical Tool (PREDICT) score, based on a cohort of 678 patients from a single center in the United States, provides risk scores for day 1 and day 5 of SAB [8]. The VIRSTA score, developed using a multicenter prospective cohort of 2008 patients with SAB in France, provides a risk score for day 4 of SAB [9]. The Prediction Of Staphylococcus aureus Infective endocarditiseTime to positivity, Iv drug use, Vascular phenomena, preExisting heart condition POSITIVE score is based on a cohort of 465 patients with SAB from Sweden [10]. All scores use different combinations of known risk factors and PREDICT and VIRSTA also use results of follow-up blood cultures, whereas POSITVE ignores follow-up cultures but uses time to positivity (TTP) of blood cultures as a predictor (Table 1).
Table 1.
Overview of POSITIVE, PREDICT, and VIRSTA Scores
| POSITIVE Cutoff: >4 |
PREDICT Cutoff: ≥2 (for Day 5 Score) |
VIRSTA Cutoff: ≥3 |
|||
|---|---|---|---|---|---|
| Item | Points Assigned | Item | Points Assigned | Item | Points Assigned |
| TTP <9 h | 5 | ICD | 2 | Cerebral or peripheral emboli | 5 |
| TTP 9–11 h | 3 | Permanent pacemaker | 3 | Meningitis | 5 |
| TTP 11–13 h | 2 | Community acquisition | 2 | Permanent intracardiac device or previous IE | 4 |
| IV drug use | 3 | Healthcare acquisition | 1 | Preexisting native valve diseasea | 3 |
| Vascular phenomenab | 6 | Positive culture after 72 h | 2 | IV drug use | 4 |
| Predisposing heart diseasec | 5 | Positive culture after 48 h | 3 | ||
| Community or healthcare-associated bacteremia | 2 | ||||
| Severe sepsis or septic shock | 1 | ||||
| C-reactive protein >190 mg/L | 1 |
Abbreviations: ICD, implantable cardioverter defibrillator; IV, intravenous; TTP, time to positivity.
aAny condition classified as medium or high risk by Dajani et al [25].
bDefined as arterial embolus, septic pulmonary embolus, mycotic aneurysm, intracranial bleeding, conjunctival hemorrhage, or Janeway lesions.
cPrevious endocarditis, prosthetic heart valve, or any condition classified as medium or high risk [25].
VIRSTA and PREDICT reported excellent negative predictive values (NPV) for endocarditis in the development cohorts (98.5% for PREDICT, 98.8% for VIRSTA), and NPVs were 99.5% and 95.1%, for VIRSTA and PREDICT, respectively, in a retrospective cohort study [11]. The derivation study for the POSITIVE score did not report an NPV, but reported a sensitivity of 93%. This implies all 3 scores may improve the diagnostic process in patients with SAB, leading to more accurate diagnosis of SA-IE and more appropriate use of TEE.
External validation and comparison of these prediction rules in different clinical settings is necessary for future implementation in clinical practice [12]. In this study, we evaluate the diagnostic performance of the POSITIVE, PREDICT, and VIRSTA scores in a cohort of 477 patients with community-, hospital-, and healthcare-associated SAB. We aimed to determine whether these scores can be used to accurately classify a subset of patients with SAB as low risk for endocarditis, in whom TEE could be safely withheld without missing a diagnosis of SA-IE.
METHODS
Study Design
From August 2017 to September 2019, we conducted the Improved Diagnostic Strategies in Staphylococcus aureus study, a prospective, multicenter cohort study among hospitalized patients with SAB in 2 university hospitals and 5 teaching hospitals in the Netherlands. We follow the Standards for Reporting of Diagnostic Accuracy Studies guidelines for reporting the results [13].
Participants
All consecutive patients aged 18 years and older with 1 or more blood cultures positive for S aureus were eligible for inclusion. In patients with multiple episodes of SAB, we included only the first episode. For the current analysis, patients who died within 48 hours after blood culture collection were excluded because risk stratification is meaningless for patients who die before the prediction rule can be applied.
Patients were identified through each hospital’s microbiology service. A dedicated member of the study team (T.v.d.V.) then approached the patient or his or her legal representatives for written informed consent. Patients who died before informed consent could be obtained were included in the study as appropriate under Dutch law. The Medical Ethics Committee of the Academic Medical Centre Amsterdam approved this study (METC2017_094). This study is registered in the Netherlands Trial Register under trial code 6669. The study was originally intended to evaluate VIRSTA and PREDICT; POSITIVE was added to the analysis after its publication [10].
Procedures
Information needed to calculate the POSITIVE, PREDICT, and VIRSTA scores; signs and symptoms of SAB and SA-IE; and laboratory, microbiology, and imaging data were extracted from the electronic health record (EHR). We used the results from echocardiography, radiology, and nuclear medicine studies performed for regular care and as reported in the EHR. No additional imaging or microbiology studies were required for the study. Patients were followed for up to 90 days after bacteremia. Follow-up was conducted by telephone interview after day 90. For patients who could not be reached by telephone, we determined postdischarge health status through contact with the patient’s primary care physician, the hospital EHR, and the municipal death records. Data extraction was performed by the study research physician and trained junior researchers. The study research physician checked all data entered by the junior researchers. Data were stored in an electronic case record file (Research Online).
Definitions
Time of first positive blood culture was defined as the time the first blood culture that grew S aureus had been collected. Blood cultures were collected in BactTec (Becton Dickinson) and BacT/ALERT (Biomerieux) bottles and incubated in the appropriate automated culture systems. TTP was the interval between placement of the culture vial in the culture system and signal of positivity by the incubator. If more than 1 positive blood culture had been drawn on the first day of bacteremia, we used the shortest TTP from that day for the calculation of the POSITIVE score. Severe sepsis was defined according to the 2001 International Sepsis definition criteria [14]. Comorbidities were scored using the Charlson Comorbidity index and the McCabe score [15, 16].
Analysis
The prespecified cutoffs suggested by the authors of the POSITIVE, PREDICT, and VIRSTA scores were used to classify patients as either being at low or high risk of SA-IE [8–10]. There were no missing data needed to calculate the risk scores, except for values of C-reactive protein on the day of blood culture (required for VIRSTA). These missing C-reactive protein values were imputed using multiple imputation using the mice R-package. For follow-up blood culture results (required for PREDICT and VIRSTA), we assumed that if no blood cultures were drawn at the required timepoint, but later blood cultures were positive, the missed earlier blood cultures would have been positive (ie, if culture results at 48 hours were missing, but the blood culture at 72 hours was positive, we assumed the culture at 48 hours to be positive). If no further blood cultures had been collected, we assumed the follow-up cultures to be negative.
The reference standard was the presence of definite infective endocarditis according to the modified Duke Criteria [17]. The presence of endocarditis was used to calculate the diagnostic accuracy measures for the risk scores. The Duke Criteria were applied blinded to the results of the risk scores and vice versa.
The primary endpoint was the diagnostic accuracy of the POSITIVE, PREDICT, and VIRSTA scores for the diagnosis SA-IE. Sensitivity, specificity, NPV, and positive predictive value (PPV) were calculated, and 95% confidence intervals (CIs) were calculated using exact binary confidence intervals. We constructed receiver-operating curve (ROC) plots and calibration plots. All statistical analysis was done in R version 4.0.0 (R Core Team, 2019. R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. https://www.R-project.org/).
We conducted 2 sensitivity analyses, 1 restricted to patients who received echocardiography, and the second to patients that had at least 1 follow-up culture drawn in the first 96 hours.
The potential impact of implementing the scores was determined by comparing the number of TEEs actually performed to the number of TEEs recommended by each risk score. In the impact analysis, patients who were unable to undergo TEE because of a contraindication or who had died before a TEE would have been possible were excluded.
We prespecified that a risk score would need a 98% NPV to be of clinical use. With a 1-sided difference of at most 2.5%, a power of 80%, an alpha of 5%, and an SA-IE prevalence of 15%, 419 patients with SAB had to be included in the study. To account for potential loss to follow-up and lower endocarditis prevalence than expected, we aimed to include 450 patients in this study.
RESULTS
Patients
Between August 2017 and September 2019, 637 patients with SAB were screened and informed consent was obtained for 491 (77%). Main reasons for noninclusion were refusal to provide informed consent (n = 29), discharged home before consent was possible (n = 46), and incapacitated patients without a legal representative (n = 22) (supplementary Table S1). For the analysis of the risk scores, 14 patients who died within 48 hours of blood culture collection were excluded, leaving a total of 477 patients. Detailed postdischarge information was available for 99% of patients (supplementary materials).
Demographics and Clinical Characteristics of Patients
The demographic and clinical characteristics of included patients are shown in Table 2. Predisposing conditions for IE were found in 30%, and 33% had community-acquired bacteremia. Echocardiography was performed in 87% of patients, and 91% of patients had at least 1 follow-up blood culture drawn after the index culture.
Table 2.
Demographic and Clinical Characteristics
| All Patients | No Endocarditis | Definite Endocarditis | |
|---|---|---|---|
| N = 477 | N = 390 | N = 87 | |
| Demographics | |||
| Sex | |||
| Female | 157 (33%) | 124 (32%) | 32 (37%) |
| Male | 320 (67%) | 265 (68%) | 55 (63%) |
| Age, y | 68 (57–76) | 68 (57–76) | 68 (57–75) |
| Diabetes mellitus | 153 (32%) | 132 (34%) | 21 (24%) |
| Immunosuppressant use | 86 (18%) | 76 (19%) | 10 (11%) |
| HIV-AIDS | 3 (1%) | 3 (1%) | 0 (0%) |
| Chronic renal failure | 136 (28%) | 114 (29%) | 18 (21%) |
| Hemodialysis | 28 (6%) | 26 (7%) | 2 (2%) |
| MRSA bacteremia | 10 (2%) | 9 (2%) | 1 (1%) |
| Intravenous drug use | 5 (1%) | 1 (0%) | 4 (5%) |
| Charlson Comorbidity Index | 3 (2–5) | 3 (2- 5) | 2 (1–4) |
| McCabe score | |||
| Nonfatal | 202 (43%) | 163 (41%) | 39 (45%) |
| Ultimately fatal | 216 (45%) | 170 (44%) | 46 (53%) |
| Rapidly fatal | 59 (12%) | 57 (15%) | 2 (2%) |
| Any predisposing cardiac condition | 143 (30%) | 94 (24%) | 49 (56%) |
| History of endocarditis | 13 (3%) | 6 (2%) | 7 (8%) |
| Native valve disease | 86 (18%) | 62 (16%) | 24 (28%) |
| Prosthetic valve | 39 (8%) | 19 (5%) | 20 (23%) |
| CRT-D | 12 (3%) | 5 (1%) | 7 (8%) |
| ICD | 10 (2%) | 8 (2%) | 2 (2%) |
| Pacemaker | 28 (6%) | 14 (4%) | 14 (16%) |
| Place of acquisition | |||
| Community-acquired | 165 (35%) | 114 (29%) | 51 (59%) |
| Healthcare-associated | 155 (32%) | 133 (34%) | 22 (25%) |
| Hospital-acquired | 157 (33%) | 143 (37%) | 14 (16%) |
| Signs and symptoms present within 72 h of first positive blood culture | |||
| Severe sepsis | 186 (39%) | 132 (34%) | 54 (62%) |
| Septic shock | 43 (9%) | 29 (7%) | 14 (16%) |
| C-reactive protein >190 mg/La | 167 (40%) | 115 (35%) | 52 (63%) |
| Meningitis | 5 (1%) | 1 (0%) | 4 (5%) |
| Vertebral osteomyelitis | 24 (5%) | 16 (4%) | 8 (9%) |
| Intensive care as admitting specialty | 38 (8%) | 26 (7%) | 12 (14%) |
| Positive follow-up culture at 48 h | 142 (30%) | 83 (22%) | 59 (67%) |
| Positive follow-up culture at 72 h | 103 (22%) | 57 (15%) | 46 (52%) |
| Time to positivityb | 12.6 (8.8–16.9) | 13.3 (9.7–17.9) | 9.00 (6.6–11.9) |
| Management and outcomes of SAB | |||
| Follow-up culture taken within 96 h | 434 (91%) | 355 (91%) | 79 (91%) |
| TTE performed | 404 (85%) | 320 (82%) | 84 (97%) |
| TEE performed | 201 (42%) | 128 (33%) | 73 (84%) |
| 18-FDG PET/CT performed | 179 (38%) | 128 (33%) | 51 (59%) |
| Infectious diseases consultation performed | 383 (80%) | 308 (79%) | 75 (86%) |
| 30-day mortality | 101 (21%) | 74 (19%) | 27 (31%) |
| 90-day mortality | 146 (31%) | 109 (28%) | 37 (43%) |
| 90-day relapse rate | 13 (3%) | 10 (3%) | 3 (3%) |
Data are n (%) or median + interquartile range unless otherwise indicated.
Abbreviations: CRT-D, Cardiac resynchronization device; HIV, human immunodeficiency virus; ICD, implantable cardioverter-defibrillator; MRSA, methicillin resistant Staphylococcus aureus; TEE, transesophageal echocardiography; TTE, transthoracic echocardiography; 18-FDG PET/CT, 18-fluorodeoxyglucose positron emission tomography/computed tomography.
aC-reactive protein at day of blood culture was missing for 63 patients: 4 patients with SA-IE and 59 patients without SA-IE.
bTime to positivity was available for 362/477 patients.
Prevalence of Endocarditis
Definite SA-IE was diagnosed in 87 patients (18.2%), in 63 (72.4%) patients based on 2 major criteria, and in 24 (27.6%) on 1 major with 3 or more minor criteria. Endocarditis was surgically or pathology proven in 17 (19.5%) of these patients, 14 of whom met the criteria for definite endocarditis before surgery or death. SA-IE involved native valves in 53 (60.9%) patients, prosthetic valves in 20 (23.0%), and a cardiac implantable electronic device (CIED) in 14 (16.1%).
Diagnostic Accuracy of POSITIVE, PREDICT, and VIRSTA Scores
TTP data were available for 6 of 7 hospitals (362 patients); diagnostic accuracy estimates of the POSITIVE score are reported for these patients. The VIRSTA score met the prespecified NPV threshold of 98% with an NPV of 99.3 (95% CI, 94.9–100) and also had the highest discriminatory ability (area under the ROC curve: 88.9 [95% CI, 85.3–92.5] (Table 3). Figure 1 shows the ROC curves of the risk scores. Visual inspection of the calibration curves (Figure 2) shows that VIRSTA systematically overestimates the risk of endocarditis, whereas PREDICT and POSITVE underestimate the risk. Findings from primary analyses were robust in both the echocardiography and the follow-up blood culture sensitivity analyses (supplementary Tables S2 and S3).
Table 3.
Diagnostic Accuracies of POSITVE, PREDICT, and VIRSTA Scores
| Score | Sensitivity (% + 95% CI) | Specificity (% + 95% CI) | Negative Predictive Value (% + 95% CI) | Positive Predictive Value (% + 95% CI) | AUC |
|---|---|---|---|---|---|
| POSITIVEa | 77.6 (65.8–86.9) | 63.1 (57.3–68.6) | 92.5 (87.9–95.8) | 32.3 (25.1–40.1) | 77.8 (71.9–83.7) |
| PREDICT day 1 | 22.9 (14.6–33.5) | 97.4 (95.3–98.8) | 85.0 (81.4–88.2) | 66.7 (47.2–82.7) | 71.4 (65.2–77.5) |
| PREDICT day 5 | 85.1 (75.8–91.8) | 56.9 (51.8–61.9) | 94.5 (90.7–97.0) | 30.5 (24.7–36.8) | 79.7 (73.9–85.4) |
| VIRSTA | 98.9 (95.7–100) | 35.7 (30.8–40.6) | 99.3 (94.9–100) | 25.5 (20.7–30.3) | 88.9 (85.3–92.5) |
Abbreviations: AUC, area under the receiver operating characteristic curve; CI, confidence interval.
aPOSITIVE score was calculated on the TTP cohort of 362 patients.
Figure 1.
Receiver operating characteristic curves of POSITIVE, PREDICT, and VIRSTA scores.
Figure 2.
Calibration curves of the POSITIVE, PREDICT, and VIRSTA scores. The points show the relation between predicted probability of Staphylococcus aureus infective endocarditis for each decile of predicted probability of endocarditis as calculated using the respective scores. The solid line is the Loess fit through the points. The dashed line represents perfect calibration.
Hypothetical Effect on Patient Care
A high-risk classification was given to 44.5% (161/362) by the POSITIVE score, 50.7% (242/477) by the PREDICT day 5 score, and 70.9% (338/477) by the VIRSTA score. In our cohort, 42.1% (201/477) of patients actually received a TEE. When excluding patients with a contraindication to TEE and patients who died before a TEE was possible, the TEE rate in our cohort increased to 47.6% (201/422). Had the risk scores been applied to this cohort, POSITIVE would have required fewer TEEs, PREDICT approximately the same, and implementing VIRSTA would mean an increase of 45.3% in TEEs necessary (Table 4). Of note, many TEEs were performed in patients classified as low risk by the risk scores.
Table 4.
Effect of Applying Risk Scores on TEE Usage
| n (%) | Change Compared With Baseline | |
|---|---|---|
| Actually done (full cohorta) | 201/422 (47.6) | Reference |
| PREDICT d 5: high risk | 210/422 (49.8) | +4.5% |
| VIRSTA: high risk | 292/422 (69.2) | +45.3% |
| Separate Cohort for POSITIVE Score | ||
| Actually done (TTP cohortb) | 165/316 (52.2) | Reference |
| POSITIVE: high risk | 137/316 (43.4) | –16.9% |
Abbreviations: TEE, transesophageal echocardiography; TTP, time to positivity.
aExcluded were 55 patients who could not undergo TEE because of death before TEE was possible or had a contraindication for TEE.
bExcluded were 45 patients who could not undergo TEE because of death before TEE was possible or had a contraindication for TEE.
Of the patients who underwent a TEE, 52% (85/165) were classified as low risk by POSITIVE, 38% (76/201) were low risk according to day 5 PREDICT, and 20% (40/201) were low risk according to VIRSTA.
Patients Misclassified as Low Risk
Supplementary Table S4 gives an overview of the patients with SA-IE misclassified as low risk by the scores and their respective risk factors. Nearly all patients misclassified as low risk for SA-IE by POSITIVE had either community-acquired SAB, a CIED, or persistent SAB, whereas patients misclassified as low risk by PREDICT often had either preexisting native valve disease or a prosthetic valve. One patient with native aortic valve SA-IE was incorrectly classified as low risk by all 3 scores.
Discussion
In this multicenter cohort study, we found that only the VIRSTA score met our predefined threshold of a NPV for the diagnosis SA-IE above 98%. Although both the POSITIVE and PREDICT (day 5) scores had acceptable sensitivity and NPV, both had a NPV below the preset threshold (92.5% and 94.5%, respectively).
The primary goal of these risk scores is to classify patients with SAB as low or high risk for SA-IE to avoid TEE in those with low risk. All 3 scores had a high NPV, but the VIRSTA score had the highest discriminatory ability (area under ROC curve 88.9%). PPV of the scores was relatively low (with the exception of the day 1 PREDICT score), with PPVs ranging from 25.5% for VIRSTA to 32.3% for POSITIVE. Yet, it is equally important to consider the effects of implementing the risk score on clinical practice. Although VIRSTA had the highest sensitivity and NPV, it also classified 70% of SAB patients as high risk, and thus having an indication for TEE. In most studies on SAB, TEE rates are between 30% and 50% [9, 10, 18, 19]. Therefore, VIRSTA would allow a clinician to safely exclude SA-IE, but would also increase the number of patients undergoing TEE. Naturally, patients with high risk scores may not undergo TEE for other reasons (such as clinical instability, refusal, or an alternative diagnosis). Therefore, the clinical consequences of implementation of any of these scores remains to be determined. These risk scores only provide additional information to the clinician and should not replace clinical judgment in individual patients.
The PREDICT day 1 score was designed for early recognition of the patients with the highest risk of SA-IE, who should undergo immediate TEE [8]. Day 1 PREDICT performs well in this regard, with high specificity (97%) and PPV (67%).
Both POSITIVE and PREDICT use fewer parameters than VIRSTA, and both lack some traditional risk factors that are associated with the risk of SA-IE. POSITIVE does not take into account the setting where the SAB was acquired, the presence of a CIED, or the results of follow-up blood cultures, whereas PREDICT does not include preexisting native valve disease or a prosthetic valve as a risk factor. This probably results from the statistical approach that was taken to avoid overfitting while developing the risk scores [20]. Although a methodologically sound method to develop a diagnostic model, lack of these traditional risk factors in POSITIVE and PREDICT results in decreased sensitivity. Fourteen of the 15 patients incorrectly classified as low risk by POSITIVE had either persistent SAB, a CIED, or had community-acquired SAB, and 10 of 12 patients incorrectly classified by PREDICT had a prosthetic valve or preexisting native valve disease.
PREDICT was previously validated in 199 patients from the same center that developed the score, and this validation study yielded 100% sensitivity and NPV [18]. These results contrasted those from another cohort with 205 patients with SAB in which day 5 PREDICT had 87% sensitivity and a 93% NPV, which is comparable to our results [21]. Peinado-Acevedo et al validated both PREDICT and VIRSTA in a cohort of 922 SAB patients from Colombia. In this study, PREDICT had lower sensitivity and NPV than VIRSTA (51.6% and 95.1% vs 99.6% and 99.5%) [11]. Because of the low prevalence of traditional risk factors for IE and the high proportion of catheter-related SAB, their results may not be generalizable to other settings.
The sensitivity of POSITIVE is lower in our cohort (77.6%) than in the development cohort and the NPV (92.5%) did not meet our predefined threshold of 98%. POSITIVE is the only score that uses TTP as a predictor of endocarditis. In the development cohort, all patients with SA-IE had a TTP shorter than 15 hours. This is in contrast to other studies that reported patients with SA-IE and TTP well above 15 hours [22, 23]. The strong weight assigned to TTP in the POSITIVE score may have led to overreliance of the statistical model on this predictor, resulting in elimination of other important predictors from the model, explaining the lower accuracy in our cohort. Adjustment of the cutoff times for TTP would likely improve the POSITIVE score, but this adjustment would require additional validation.
Strengths of our study include the prospective enrollment with nearly complete data, the multicenter design allowing for an unselected cohort from both general and university hospitals, and the high percentages of TTE, TEE, and follow-up blood cultures.
Study limitations include that not all patients with SAB were enrolled because of the need for informed consent. That patients who refused or were discharged before consent was possible may have resulted in selection bias; in particular, the latter group may comprise the least ill patients, with a lower likelihood of SA-IE. Yet, because most would likely have been classified as low risk by the prediction scores, this would not influence sensitivity and could further increase NPV. Furthermore, our study population is comparable in demographics, rates of echocardiography, prevalence of endocarditis, and mortality to other cohort studies that included all patients without the need for informed consent [3, 4, 8, 19, 24]. As in nearly all studies on SAB, blood culture collection was not standardized, leading to some uncertainty on duration of bacteremia. Despite this, at least 1 follow-up blood culture was taken in 91% of patients. The diagnostic accuracy results were robust in the sensitivity analysis including only patients that had at least 1 follow-up blood culture. Finally, because of a recent change in laboratory software, TTP data were missing from 1 hospital. This smaller population for validation of POSITIVE may have resulted in wider CIs, but is unlikely to bias the diagnostic accuracy point estimates.
In summary, we found that of the 3 proposed prediction sets for risk of IE in patients with SAB, VIRSTA has the highest NPV but at the expense of a high number of patients classified as high risk and thus requiring TEE. Risk scores should therefore not only be evaluated with regard to diagnostic accuracy, but also on their clinical implications. The POSITIVE and PREDICT scores need adjustment before they can be used to rule out endocarditis. Until further adjustment of the POSITVE and PREDICT scores, clinicians can use the VIRSTA score to safely rule out SA-IE without performing a TEE.
Supplementary Data
Supplementary materials are available at Clinical Infectious Diseases online. Consisting of data provided by the authors to benefit the reader, the posted materials are not copyedited and are the sole responsibility of the authors, so questions or comments should be addressed to the corresponding author.
Supplementary Material
Notes
Financial support. This work was supported by the University Medical Center Utrecht and the Academic Medical Center Amsterdam.
Potential conflicts of interest. The authors: No reported conflicts of interest. All authors have submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest.
References
- 1. van Cleef BA, van Benthem BH, Haenen AP, Bosch T, Monen J, Kluytmans JA. Low incidence of livestock-associated methicillin-resistant Staphylococcus aureus bacteraemia in The Netherlands in 2009. PloS One 2013; 8:e73096. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2. Fowler VG Jr, Li J, Corey GR, et al. Role of echocardiography in evaluation of patients with Staphylococcus aureus bacteremia: experience in 103 patients. J Am Coll Cardiol 1997; 30:1072–8. [DOI] [PubMed] [Google Scholar]
- 3. Rasmussen RV, Høst U, Arpi M, et al. Prevalence of infective endocarditis in patients with Staphylococcus aureus bacteraemia: the value of screening with echocardiography. Eur J Echocardiogr 2011; 12:414–20. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4. Le Moing V, Alla F, Doco-Lecompte T, et al. ; VIRSTA study group. Staphylococcus aureus Bloodstream Infection and Endocarditis–a Prospective Cohort Study. PLoS One 2015; 10:e0127385. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5. Cecchi E, Chirillo F, Faggiano P, et al. ; Italian Registry on Infective Endocarditis (RIEI) Investigators. The diagnostic utility of transthoracic echocardiography for the diagnosis of infective endocarditis in the real world of the Italian Registry on Infective Endocarditis. Echocardiography 2013; 30:871–9. [DOI] [PubMed] [Google Scholar]
- 6. Bai AD, Agarwal A, Steinberg M, et al. Clinical predictors and clinical prediction rules to estimate initial patient risk for infective endocarditis in Staphylococcus aureus bacteraemia: a systematic review and meta-analysis. Clin Microbiol Infect 2017; 23:900–6. [DOI] [PubMed] [Google Scholar]
- 7. Blyth CC, Darragh H, Whelan A, O’Shea JP, Beaman MH, McCarthy JS. Evaluation of clinical guidelines for the management of Staphylococcus aureus bacteraemia. Intern Med J 2002; 32:224–32. [DOI] [PubMed] [Google Scholar]
- 8. Palraj BR, Baddour LM, Hess EP, et al. Predicting Risk of Endocarditis Using a Clinical Tool (PREDICT): scoring system to guide use of echocardiography in the management of Staphylococcus aureus bacteremia. Clin Infect Dis 2015; 61:18–28. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9. Tubiana S, Duval X, Alla F, et al. ; VIRSTA/AEPEI Study Group. The VIRSTA score, a prediction score to estimate risk of infective endocarditis and determine priority for echocardiography in patients with Staphylococcus aureus bacteremia. J Infect 2016; 72:544–53. [DOI] [PubMed] [Google Scholar]
- 10. Kahn F, Resman F, Bergmark S, et al. T ime to blood culture positivity in Staphylococcus aureus bacteraemia to determine risk of infective endocarditis. Clin Microbiol Infect 2020; 10.1016/j.cmi.2020.11.007. [DOI] [PubMed] [Google Scholar]
- 11. Peinado-Acevedo JS, Hurtado-Guerra JJ, Hincapie-Osorno C, et al. Validation of VIRSTA and PREDICT scores to determine the priority of echocardiography in patients with Staphylococcus aureus bacteremia. Clin Infect Dis 2021; 73:e1151–7. [DOI] [PubMed] [Google Scholar]
- 12. Fida M, Saleh OA, Baddour LM, Sohail MR. Relationship between time-to-Positivity and infective endocarditis in patients with Staphylococcus aureus bacteremia. Clin Microbiol Infect 2021. [DOI] [PubMed] [Google Scholar]
- 13. Bossuyt PM, Reitsma JB, Bruns DE, et al. ; STARD Group. STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies. BMJ 2015; 351:h5527. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14. Levy MM, Fink MP, Marshall JC, et al. 2001 SCCM/ESICM/ACCP/ATS/SIS International Sepsis Definitions Conference. Crit Care Med 2003; 31: 1250–6. [DOI] [PubMed] [Google Scholar]
- 15. Charlson ME, Pompei P, Ales KL, MacKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chronic Dis 1987; 40:373–83. [DOI] [PubMed] [Google Scholar]
- 16. McCabe WR, Jackson GG. Gram negative bacteremia: I. Etiology and ecology. Arch Intern Med 1962:845–7. [Google Scholar]
- 17. Li JS, Sexton DJ, Mick N, et al. Proposed modifications to the Duke criteria for the diagnosis of infective endocarditis. Clin Infect Dis 2000; 30:633–8. [DOI] [PubMed] [Google Scholar]
- 18. Abu Saleh O, Fida M, Asbury K, et al. Prospective validation of PREDICT and its impact on the transesophageal echocardiography use in management of Staphylococcus aureus bacteremia. Clin Infect Dis 2020: ciaa844. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19. Kuehl R, Morata L, Boeing C, et al. ; International Staphylococcus aureus collaboration study group and the ESCMID Study Group for Bloodstream Infections, Endocarditis and Sepsis. Defining persistent Staphylococcus aureus bacteraemia: secondary analysis of a prospective cohort study. Lancet Infect Dis 2020; 20:1409–17. [DOI] [PubMed] [Google Scholar]
- 20. Moons KG, Kengne AP, Woodward M, et al. Risk prediction models: I. Development, internal validation, and assessing the incremental value of a new (bio)marker. Heart 2012; 98:683–90. [DOI] [PubMed] [Google Scholar]
- 21. Longobardo L, Klemm S, Cook M, et al. Risk assessment for infected endocarditis in Staphylococcus aureus bacteremia patients: when is transesophageal echocardiography needed? Eur Heart J Acute Cardiovasc Care 2019; 8:476–84. [DOI] [PubMed] [Google Scholar]
- 22. Siméon S, Le Moing V, Tubiana S, et al. ; VIRSTA/AEPEI Study Group. Time to blood culture positivity: an independent predictor of infective endocarditis and mortality in patients with Staphylococcus aureus bacteraemia. Clin Microbiol Infect 2019; 25:481–8. [DOI] [PubMed] [Google Scholar]
- 23. Khatib R, Riederer K, Saeed S, et al. Time to positivity in Staphylococcus aureus bacteremia: possible correlation with the source and outcome of infection. Clin Infect Dis 2005; 41:594–8. [DOI] [PubMed] [Google Scholar]
- 24. Ariaans MBPA, Roovers EA, Claassen MAA, Hassing RJ, Swanink CMA, Gisolf EH. Increased overall survival after introduction of structured bedside consultation in Staphylococcus aureus bacteraemia. Eur J Clin Microbiol Infect Dis 2018; 37:1187–93. [DOI] [PubMed] [Google Scholar]
- 25. Dajani AS, Taubert KA, Wilson W, et al. Prevention of bacterial endocarditis: recommendations by the American Heart Association. Clin Infect Dis 1997; 25:1448–58. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.