External validation of risk indices and the development of a multivariable risk prediction model for surgical site infection in adults: a prospective observational study

Abstract

Background

To enable proper benchmarking of rates of surgical site infections (SSIs), it is important to consider the variability in case mix and risk factors in the data analysis. SSI risk indices have been used to make the data more comparable. However, different risk indices exist, and studies comparing these indices head-to-head are limited. Thus, the purpose of this study was to compare and externally validate six indices of SSI risk prediction.

Methods

This study was conducted with data from ASPIRE-SSI, a prospective cohort study conducted at 33 sites in ten European countries. The following risk indices were assessed: the National Nosocomial Infections Surveillance System (NNIS) risk index and NNIS index improved for cardiac patients, the Australian clinical risk index, the Infection risk index in cardiac surgery, the risk index A, and risk index B (range of area under the receiver operating characteristic curves in the derivation studies: 0.62–0.67). Comparison was done in two cohorts of patients; an overall cohort, consisting of 9657 patients who underwent 11 different types of surgical procedures, and a sub-cohort, consisting of the 1772 patients who underwent open cardiac surgery. The main endpoint was SSI of any cause up to 90 days after surgery. Model discrimination was assessed with and without accounting for clustering , and model calibration was assessed only in the overall cohort. Furthermore, we attempted to improve the predictive ability of the risk indices by developing a new model consisting of predictor variables from the assessed risk indices.

Results

5.2% (502/9657) of patients in the overall cohort, and 8.9% (157/1772) of patients in the sub-cohort developed an SSI within 90 days after surgery. When clustering was not accounted for, the risk indices exhibited low discriminative power in both the overall cohort (highest C-statistic 0.60) and sub-cohort (highest C-statistic 0.58), and overestimated the risk of SSI, especially for patients in higher SSI risk categories. The C-statistic estimates were slightly higher in both cohorts (range C-statistic: 0.63–0.65) when clustering was taken into account. The newly developed prediction model (without correction for overfitting) had poor discrimination (C-statistic 0.67, 95% CI 0.64–0.69), but a good agreement between the observed and predicted SSI risks.

Conclusion

The SSI risk indices had comparable discrimination when clustering was taken into account, but suboptimal calibration in our cohorts compared with their derivation cohorts.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13756-026-01712-z.

Background

Surgical site infections (SSIs) are among the most important healthcare-associated infections, and are associated with prolonged hospital stay, increased healthcare costs, and mortality [1–4]. Although the incidence of SSIs varies depending on anatomical region, surgical type, study design, and definitions used to classify the infection (e.g., superficial vs. complex), SSIs can have major consequences for the individual and the community as a whole [5–7]. For this reason, prevention of this complication is of major importance.

Surveillance has become an essential element of SSI prevention and quality improvement strategies, including the feedback of SSI rates into improvement strategies aimed at SSI reduction [8]. For instance, when patients are stratified according to their risk of developing an SSI, these risk indices can be used to monitor trends in SSI rates in specific high-risk patient groups. In addition, individual hospitals and national healthcare planners can utilize these data to set up infection control strategies and evaluate the effectiveness of these efforts [9–11].

Yet, hospitals treating patients with different SSI risk factors, would expect different SSI rates. To allow for more meaningful comparisons between different hospitals, it is not only essential that the reported data take into account the variability in case mix, but also that it is adjusted for risk factors, and is based on a consistent case definition. This is especially true when summary data are used as a performance metric [12, 13].

A widely known risk adjustment tool for SSI is the one applied by the National Healthcare Safety Network (NHSN). The NHSN reports on adjusted SSI rates using a risk index consisting of three equally weighted factors: the American Society of Anesthesiologists (ASA) score, wound classification, and procedure duration [14]. However, it has been reported that this risk index performs poorly among surgical patients with similar ASA scores and wound classification, such as patients undergoing cardiac surgery [15]. Other, more specific risk adjustment tools have been investigated for patients undergoing cardiac surgery. However, studies that have directly assessed and compared the performance of these risk indices with each other, are limited.

The objective of this study was to assess the performance of the NHSN risk index and five risk indices that were specifically developed in cardiac surgery patients (the Australian clinical risk index [ACRI] [16], the Infection risk index in cardiac surgery [IRIC] [17], the Risk indices A [RIA] and B [RIB] [18], and the NNIS risk index improved for cardiac surgery patients [19]), in an independent population of surgical cardiac patients who participated in the ASPIRE-SSI study. In addition, as proof of concept, we further investigated the performance of these risk indices in the broader surgical population of the ASPIRE-SSI study, to assess the transportability of these risk indices to a more diverse patient population. Finally, we aimed to develop an improved prediction model composed of the predictor variables included in these risk indices.

Methods

Study design and setting

Existing data from the ASPIRE-SSI study (Advanced understanding of Staphylococcus aureus [SA] infections in Europe – SSI, clinicaltrial.gov identifier NCT02935244) were used for this study. In short, ASPIRE-SSI was a prospective cohort study of adult surgical patients recruited at 33 regional, university-affiliated, or university hospitals in ten European countries between December 2016 and November 2019. The study design and rationale of the ASPIRE-SSI study [20], as well as the main study results [21], are published elsewhere. This paper follows the “Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD)” guidelines [22].

Participants

Adult patients undergoing 11 different types of surgical procedures consisting of (semi-)elective surgery (further classified as open heart surgery, implantable cardioverter defibrillator implantation, peripheral artery bypass surgery, central artery reconstructive surgery, hip prosthesis surgery, knee prosthesis surgery, laminectomy, spinal fusion, craniotomy, mastectomy) and emergency surgery (trauma surgery or unplanned cardiac, vascular, neuro-, spinal, or orthopedic surgery) were screened for SA carriage in the nose, throat, and perineum within 30 days prior to surgery. These patients comprised the source population. Based on their preoperative SA colonization status, SA carriers and non-carriers were subsequently enrolled in a 2:1 ratio in the study cohort (main study population). Exclusion criteria included active SSI as the reason for surgery, and parallel participation in any antistaphylococcal interventional study [20].

Primary outcome

The primary outcome was the occurrence of an SSI (composite of superficial, deep, or organ/space SSI) of any microbial etiology up to 90 days post-surgery. This was determined through medical chart review and telephone interviews with the patients or the patient’s next of kin at fixed pre-determined time-points during follow-up (at days 7, 14, 21, 28, 60, and 90 after surgery (± 3 days each)). If an SSI was suspected, the patient was advised to seek medical attention for clinical assessment and collection of microbiological cultures. The SSI diagnosis was ascertained by study investigators using diagnostic criteria developed by the Centers of Disease Control and Prevention (CDC) [23]. Assessors were not blinded to the clinical course of the patient.

Risk indices

Table 1 summarizes the six risk indices that were validated in this study. Five of these risk indices were developed specifically for cardiac surgery and were intended to be used in this specific population. Their validity has not yet been assessed for other surgical procedures. However, they include general predictor variables that could also have predictive value in other surgical population. For this reason, we assessed the transportability of these risk indices to other surgical populations. The derivation study of all models only provided the predicted risk per risk category.

Table 1.

Description of the Risk Indices

Parameter	Risk indices
	NNIS surgical wound infection risk index	Australian clinical risk index (ACRI)1	Infection risk index in cardiac surgery (IRIC)	Risk index A (RIA)	Risk index B (RIB)
Development setting	115 hospitals providing general medical-surgical inpatient services to acute care patients	6 hospitals providing surveillance data to the Victorian Hospital-Acquired Infection Surveillance System (VICNISS) database and hospitals providing data to the Australasian Society of Cardiac and Thoracic Surgeons (ASCTS) database	1 Hospital (level 3 university-affiliated teaching hospital)	1 Hospital (350-bed, university-affiliated, tertiary-care referral center)	See description of RIA
Type of patients	Patients undergoing various types of surgical procedures, including cardiac surgery, spinal fusion, joint prosthesis, craniotomy	4633 patients undergoing coronary artery bypass graft (CABG) procedures	Consecutive patients undergoing major cardiac surgery. Exclusion criteria: mini thoracotomy; patients < 18 years; operation without extracorporeal circulation. 1298 procedures were included	2345 Consecutive patients undergoing CABG	See description of RIA
Outcome	SSIs ascertained according to NNIS criteria by CDC guidelines	286 SSIs (259 patients); 149 SSIs were sternal SSIs SSIs were ascertained according to NNIS criteria by CDC guidelines	60 SSIs were ascertained according to NNIS criteria by CDC guidelines	199 SSIs; 50 SSIs were sternal SSI. SSIs were ascertained according to NNIS criteria by CDC guidelines	See description of RIA
Data collection period	Ongoing since 1970	1 January 2013 through 31 March 2015; No routine post-discharge surveillance was conducted, but patients who were readmitted with an SSI were identified	Training sample: 1 January 2010 through 31 December 2014; Test sample: January 1, 2015, through 31 December 2017	1 December 1996 through 29 September 2000	See description of RIA
Elements of the risk index	1. Wound classification: < 3: 0 points, ≥ 3: 1 point 2. ASA score: < 3: 0 points; ≥ 3: 1 point 3. Length surgical procedure: lasting < 75th percentile of surgical duration: 0 points; lasting T > 75th percentile of the surgical duration: 1 point (different duration cut-off points, depending on the type of procedure; rounded to the nearest hour)	1. Diabetes mellitus: No:0 points; Yes:1 point 2. BMI: < 30:0 points; 30–34.9:1 point; ≥ 35:2 points	1. Diabetes mellitus: No:0 points; Yes:1 point 2. BMI: ≤ 30:0 points; > 30:1 point	1. Obesity: No: 0 points; Yes: 1 point 2. Peripheral/cerebrovascular disease: No: 0 points; Yes: 1 point 3. Insulin-dependent diabetes mellitus: No: 0 points; Yes: 1 point	1. Obesity: No: 0 points; Yes: 1 point 2. Peripheral/cerebrovascular disease: No: 0 points; Yes: 1 point 3. Insulin-dependent diabetes mellitus: No: 0 points; Yes: 1 point 4. Length surgical procedure > 5 h: No: 0 points, Yes: 1 point
Risk scores	Ranges from 0 to 3 by adding the number of the risk factors present	Ranges from 0 to 3 by adding the number of the risk factors present	Ranges from 0 to 2 by adding the number of the risk factors present	Ranges from 0 to 3 by adding the number of the risk factors present	Ranges from 0 to 4 by adding the number of the risk factors present
Predicted SSI risk per risk category	Based on REF	0: 0.5% 1: 2.3% 2: 13.3% 3: 36.1% (based on the ORs of the sternal infection risk scores: OR 1.62, 95% CI 1.1–2.5) for category 1; OR 3.51, 95% CI 2.3–5.4) for category 2; and OR 4.81, 95% CI 2.7–8.7) for category 3)	0: 2.5% 1: 8.4% 2: 28.6%	0: 6.8% 1: 11.8% 2: 17.5% 3: 40%	0: 6.3% 1: 11.0% 2: 15.4% 3: 35.3% 4: not provided

Parameter	Risk indices
	NNIS Surgical Wound Infection Risk Index (Improved adjustment for coronary artery bypass graft Surgical Site Infections)2
Development setting	293 hospitals that reported CABG procedures
Type of patients	Patients who underwent a total of 133,503 CABG procedures (these included those procedures with sternal and harvest site incisions, as well as only sternal site incisions)
Outcome	SSIs ascertained according to NNIS criteria by CDC guidelines. Only complex (deep and organ/space), sternal, admission or readmission detected SSIs were included
Data collection period	January 1, 2006, through December 31, 2008
Elements of the risk index	1. ASA score: 1/2 and 3, vs. 4/5 (OR 1.47, 95% CI 1.31–1.65) 2. Procedure duration 10: for every additional 10-minute increase in procedure duration (OR 1.03, 95% CI 1.02–1.03) 3. Medical school affiliation: yes vs no (OR 1.21, 95% CI 1.08–1.36) 4. Interaction between Age 10 (for every 10-year age increments) and Sex: Male (OR 0.95, 95%CI 0.90–0.98) and Female (OR 0.82, 95% CI 0.79–0.86)
Risk scores	Not provided
Predicted SSI risk per risk category	Range from 0 to 100%

¹For ACRI, the predicted risks were not provided in the paper nor by the author. However, the odds ratios (ORs) for SSI of the different risk categories (OR category 1: 1.6 [1.1–2.5]; category 2: 3.5 [2.3–5.4]; category 3: 4.8 [2.7–8.7]) were provided. We used these ORs as input for the linear predictor (LP) and included the LP as an offset value in a logistic regression model, in order to calculate the intercept based on the ASPIRE-SSI dataset (which was -5.4). We then calculated the predicted SSI risks of the ACRI categories with the following formula: Predicted risk of category n = , where n = 0, 1, 2, or 3, and category 1,2, and 3 were binary variables with values 0 and 1, depending on whether a patient was in one of the categories (then that category took the value of 1)

²For the improved NNIS risk index, the predicted risks were not provided in the paper nor by the author. However, the odds ratios (ORs) for SSI of the different variables were provided. We used these ORs as input for the linear predictor (LP) and included the LP as an offset value in a logistic regression model, in order to calculate the intercept based on the ASPIRE-SSI dataset (which was -45.35). We then calculated the predicted SSI risks of the improved risk index with the following formula: Predicted risk of category n = 

Missing data

We assumed that data were missing at random. We used default multiple imputation by chained equation procedures to acquire ten imputed datasets for these analyses [24]. Main reasons for missing data in the original data set were that the data could not be retrieved from the medical records, or that samples were not collected. Missing data of predictor variables were predicted based on other predictors in the dataset. The variables that were used in the imputation model, are listed in supplemental methods 1. Outcome data were not imputed as there were no missing outcome data. No interaction terms were included in the imputation model.

Statistical analysis

External validation of the risk indices

The predictive performance of the risk indices was assessed by examining model calibration and discrimination. The study cohort of the ASPIRE-SSI study was sampled from a source population of surgical patients, and the source population was a sample from the broader surgical population. Therefore, to conduct a valid external validation study, we used weighting methods to recreate the source population from the study cohort. A detailed description of the applied weighting methods can be found in the supplemental material (Supplemental methods 2). We assessed the predictive performance of the risk indices in the complete weighted surgical cohort (henceforth referred to as overall cohort), and in the sub-cohort of subjects who underwent open cardiac surgery (henceforth referred to as the sub-cohort). Ten imputed datasets were created from both the overall cohort and sub-cohort of open cardiac surgery patients, and in each imputed dataset we calculated for each subject the risk score according to each risk index, based on the definitions reported in the original papers (Table 1). Subsequently, 5,000 bootstrap samples were drawn from each imputed dataset to obtain confidence intervals.

Sample size justification

This study was conducted using existing data from the ASPIRE-SSI study, so the sample size was limited to the number of patients recruited and events reported in that study. For this reason we did not carry out a sample size calculation. However, with over 500 outcome events in the overall cohort, and 150 in the sub-cohort, we are confident that the sample size is large enough for the validation and derivation purposes described in this paper [25].

Assessment of discrimination (without adjustment for clustering)

Model discrimination is the ability of a model to differentiate between patients who do and who do not experience an event during the study period. A widely reported measure of discrimination is the C-statistic; a value of 0.5 represents no discrimination and 1 represents perfect discrimination [22]. In logistic regression modelling, the C-statistic is equal to the area under the curve (AUC) of the receiver operating characteristic (ROC) curve.

For each risk index, the following steps were repeated:

Each of the 5,000 bootstrap samples created from each imputed dataset were inflated using the weights, and then the C-statistic of the risk index was calculated. This created, for each imputed dataset, a sequence of 5,000 C-statistics for the risk index. This sequence was used to derive the point estimate (50th percentile) with 95% confidence interval (CI, the 2.5th and 97.5th percentile) of each imputed dataset. We pooled the logit-transformed C-statistic estimates of the ten imputed datasets using Rubin’s rule [25]. Then, using inverse logit-transformation, we derived the pooled C-statistic.

Assessment of discrimination (with adjustment for clustering)

For this analysis, we considered all patients from one hospital as a cluster. We recalculated the pooled C-statistic estimates (with 95% CI) for each risk index for both the overall cohort and sub-cohort. This was done by calculating hospital-specific C-statistic estimates based on the methods described above, and pooling these C-statistic estimates into one overarching pooled C-statistic using inverse variance weighted meta-analysis on the log-odds scale, with adjustment for hospital size [26].

Assessment of calibration

Model calibration refers to the agreement between the predictions of the model, and the observed outcomes [22]. To assess model calibration, we extracted the predicted risks of the different risk categories from the original papers.

For each risk index, the following steps were repeated:

In each of the 5,000 weighted bootstrap samples created from each imputed dataset, we calculated the observed risk of SSI per risk category of the risk index. This observed risk was calculated by dividing the number of SSIs in that particular risk category by the total number of patients in that risk category. Using this method, we derived, for each imputed dataset, a sequence of 5,000 observed SSI risk estimates for each category of the risk index. The observed SSI risk (50th percentile) with 95% CI (the 2.5th and 97.5th percentile) for each category of the risk index was pooled across the ten imputed datasets using the Rubin’s rules. Finally, we plotted the pooled observed SSI risks with 95% CI of each category of the risk index against the predicted risk estimates. Pooled estimates for the calibration intercepts and slopes were estimated “at the large” using univariable logistic regression modeling, where the predicted SSI risk per risk category according to the different risk indices, was used as the only predictor to estimate the observed SSI risk.

Model building

To improve the predictive accuracy, we combined the predictor variables that were used in at least one of the risk indices into a single model. As these variables were already part of established risk indices, we assumed that they had predictive value and did not perform any additional variable selection.

The assessed risk indices had nine mutually exclusive predictor variables: wound classification, body mass index (BMI), duration of surgery, presence of cerebrovascular or peripheral vascular disease, ASA score, diabetes, age, sex, and medical school affiliation. All predictor variables were defined as categorical variables (Table 1). For our model, we used the same functional forms as the derivation studies for the following predictor variables: wound classification, presence of cerebrovascular or peripheral vascular disease, ASA score, age, sex, and medical school. Diabetes and BMI were included as binary variables in the model. Lastly, we included the natural logarithm of duration of surgery in our model instead of the original value, as the natural logarithm had a better linear relationship with the outcome. In conclusion, we included the following forms of the variables in the new model: natural logarithm-transformed duration of surgery, ASA-score as a 4-category variable (combining ASA-score 4 and 5), age as a number between 1 and 9 (1 being ages 10–19, 2 being ages 20–29 etc.), and the remaining variables as binary variables. The variable medical school affiliation was omitted from the analysis, as all participating hospitals in this study were affiliated to a medical school.

We fitted a multivariable logistic regression model with these variables and computed robust standard errors. No interaction terms were included. Overall model performance was assessed by measuring the explained variation as described by Nagelkerke’s R². Calibration was assessed by plotting the observed proportion of observed SSI events against the predicted risks. To assess model discrimination, an ROC curve was computed, and the AUC was calculated. The predictive performance of this model was only assessed for the overall cohort. All analyses were done on the ten imputed datasets acquired from this cohort, and only pooled results are reported.

In all analysis (discrimination, calibration, and derivation), the primary outcome was treated as a binary outcome to align with the original derivation studies. We considered this justified because follow-up duration was relatively short, the follow-up was complete for all subjects, and predictors were time-fixed, avoiding time-dependent bias.

All statistical analyses were performed using R version 4.0.2 and the following packages: boot v1.3–28, pROC v1.10, mice v3.14, psfmi v1.0.0, ggplot2 v3.3.5, tidyverse v2.0.0, tidymodels v1.4.1, prediction v1.2.0 and sandwich v3.0–1 [27].

Results

A total of 5,004 patients, including 969 open cardiac surgery patients, were enrolled in the ASPIRE-SSI study cohort. 354 patients, including 107 open cardiac surgery patients, developed an SSI. After weighting, the overall cohort consisted of 9,657 patients, of which 1,772 were cardiac surgery patients. A total of 502 patients in the overall validation cohort, and 157 patients in the sub-cohort, developed an SSI within 90 days after surgery (Fig. 1). Most SSIs in the overall cohort (72.3%) were superficial SSIs. Tables 2 and supplemental Table 1 show the patient characteristics of both populations. Substantial differences in sex, ASA scores, BMI, and diabetes were observed between our cohorts and the original derivation cohorts of ACRI, IRIC, RIA/RIB, and the NNIS index (improved for cardiac surgery patients) (Supplemental Table 2). These data were not available for the original NNIS risk index.

Fig. 1.

Selection of patients for the validation study

Table 2.

Baseline characteristics of the overall cohort

Characteristics	Subjects with SSI		Subjects without SSI		Total
	No (%) or Median value (IQR)	Missing	No (%) or Median value (IQR)	Missing	No (%) or Median value (IQR)	Missing
Number of patients	502	0 (0.0%)	9155	0 (0.0%)	9657	0 (0.0%)
Age (median, in years)	65 (56;73)	0 (0.0%)	66 (56;73)	0 (0.0%)	66 (56;73)	0 (0.0%)
Sex at birth
Male	312 (62.2%)	0 (0.0%)	4232 (46.2%)	0 (0.0%)	4544 (47.1%)	0 (0.0%)
Female	190 (37.8%)		4923 (53.8%)		5113 (52.9%)
Type of surgery
Craniotomy	29 (5.8%)	0 (0.0%)	490 (5.4%)	0 (0.0%)	519 (5.4%)	0 (0.0%)
Laminectomy	25 (5.0%)		900 (9.8%)		925 (9.6%)
Spinal fusion	9 (1.8%)		437 (4.8%)		446 (4.6%)
Central artery reconstructive surgery	31 (6.2%)		158 (1.7%)		189 (2.0%)
Peripheral artery bypass surgery	52 (10.4%)		386 (4.2%)		438 (4.5%)
Mastectomy	66 (13.1%)		965 (10.5%)		1031 (10.7%)
Open cardiac surgery	157 (31.3%)		1615 (17.6%)		1772 (18.3%)
Implantable cardioverter defibrillator implantation	7 (1.4%)		212 (2.3%)		219 (2.3%)
Emergency surgery*	25 (5.0%)		675 (7.4%)		700 (7.2%)
Hip prosthesis surgery	38 (7.6%)		1485 (16.2%)		1523 (15.8%)
Knee prosthesis surgery	63 (12.5%)		1832 (20.0%)		1895 (19.6%)
BMI	29 (25; 32)	3 (0.6%)	28 (25; 31)	169 (1.8%)	28 (25; 31)	172 (1.8%)
BMI > 30 kg/m2	202 (40.5%)	3 (0.6%)	2984 (33.2%)	169 (1.8%)	3186 (33.6%)	172 (1.8%)
ASA score
Class I	27 (5.6%)	16 (3.2%)	928 (10.7%)	468 (5.1%)	955 (10.4%)	484 (5.0%)
Class II	160 (32.9%)		4158 (47.9%)		4318 (47.1%)
Class III	247 (50.8%)		3153 (36.3%)		3400 (37.1%)
Class IV	52 (10.7%)		445 (5.1%)		497 (5.4%)
Class V	0 (0.0%)		3 (0.0%)		3 (0.0%)
History of SA colonization or infection	35 (7.0%)	0 (0.0%)	412 (4.5%)	25 (0.3%)	447 (4.6%)	25 (0.3%)
SA colonization status prior to surgery
SA-colonized	261 (52.0%)		3108 (33.9%)		3369 (34.9%)
Non-colonized	241 (48.0%)		6047 (66.1%)		6288 (65.1%)
Comorbidities
Peripheral vascular disease	93 (18.5%)	0 (0.0%)	1151 (12.6%)	1 (0.0%)	1244 (12.9%)	1 (0.0%)
Cerebrovascular disease	43 (8.6%)	0 (0.0%)	472 (5.2%)	1 (0.0%)	515 (5.3%)	1 (0.0%)
Diabetes mellitus	127 (25.3%)	0 (0.0%)	1612 (17.6%)	1 (0.0%)	1739 (18.0%)	1 (0.0%)
Insulin-dependent diabetes mellitus	53 (10.4%)	0 (0.0%)	413 (4.5%)	1 (0.0%)	465 (4.8%)	1 (0.0%)
Congestive heart failure	62 (12.4%)	0 (0.0%)	927 (10.1%)	1 (0.0%)	989 (10.2%)	1 (0.0%)
Chronic pulmonary disease	43 (8.6%)	0 (0.0%)	692 (7.6%)	1 (0.0%)	735 (7.6%)	1 (0.0%)
Renal disease	22 (4.4%)	0 (0.0%)	417 (4.6%)	1 (0.0%)	439 (4.5%)	1 (0.0%)
Solid tumor (any)	98 (19.5%)	0 (0.0%)	1123 (12.3%)	1 (0.0%)	1221 (12.6%)	1 (0.0%)
Wound classification
Clean (class 1)	498 (99.2%)	0 (0.0%)	9047 (98.8%)	2 (0.0%)	9545 (98.8%)	2 (0.0%)
Clean-contaminated (class 2)	3 (0.6%)		85 (0.9%)		88 (0.9%)
Contaminated (class 3)	0 (0.0%)		5 (0.1%)		5 (0.1%)
Dirty (class 4)	1 (0.2%)		16 (0.2%)		17 (0.2%)
Non-removal implant in the body prior to surgery	135 (26.9%)	0 (0.0%)	2263 (24.8%)	17 (0.2%)	2398 (24.9%)	17 (0.2%)
Use of immunosuppressive medication within 2 weeks prior to surgery	37 (7.4%)	1 (0.2%)	369 (4.0%)	7 (0.1%)	406 (4.2%)	8 (0.1%)
90-day mortality	16 (3.2%)		107 (1.2%)		123 (1.3%)

Abbreviations: ASA, American Society of Anesthesiology Physical classification status; BMI, body mass index; IQR, interquartile range; No, number; SA, S. aureus; SSI, surgical site infection

^* Emergency surgery included surgical patients who underwent trauma surgery or unplanned cardiac surgery, vascular surgery, neurosurgery, spinal surgery, or orthopedic surgery in the acute setting

Performance of the risk indices

When not accounting for clustering, the C-statistic in the overall cohort was highest for the NNIS risk index (0.60, 95% CI 0.56–0.63), whereas IRIC had the highest C-statistic (0.58, 95%CI 0.50–0.66) in the sub-cohort of open cardiac surgery patients (Table 3). When clustering was taken into account, the discriminatory ability of all risk indices improved marginally in both the overall cohort (Table 4) and sub-cohort (Table 5) (range C-statistic: 0.63–0.65). It is important to note that not all hospitals were included in the cluster-adjusted analysis, as it was not possible to calculate a hospital-specific C-statistic estimate for sites that were too small and/or had too few SSI events. When calibration was assessed, most risk indices overestimated the risk of developing SSI for higher risk categories in both the overall cohort (Supplemental Fig. 1 and supplemental Table 3) and sub-cohort (Supplemental Fig. 2 and supplemental Table 4). Estimates for the pooled calibration intercepts and slopes (at the large) of the calibration plots of the different risk indices are provided in the supplemental material for the overall cohort (Supplemental Table 5) and the sub-cohort (Supplemental Table 6).

Table 3.

Pooled median C-statistic estimate of the risk indices in the overall cohort and sub-cohort

Risk index	Median C-statistic (95% CI) in the overall cohort (N = 9657)	Median C-statistic (95% CI) in the sub-cohort (N = 1772)
NNIS risk index	0,60 (0,56–0,63)	–
NNIS risk indexa	–	0,47 (0,33–0,63)
ACRI	0,54 (0,51–0,58)	0,57 (0,49–0,64)
IRIC	0,55 (0,52–0,59)	0,58 (0,50–0,66)
RIA	0,58 (0,54–0,61)	0,55 (0,48–0,61)
RIB	0,58 (0,54–0,61)	0,52 (0,37–0,67)

Abbreviations: ACRI, Australian Clinical Risk Index; CI, confidence interval; IRIC, Infection risk index in cardiac surgery; NNIS, National Nosocomial Infections Surveillance; RIA, Risk index A; RIB, Risk index B

^a improved for cardiac surgery

Table 4.

Pooled cluster-adjusted median C-statistic estimate of the risk indices in the overall cohort

Risk index	Number of hospitals	Total number of patients	Number of SSI events	Median C-statistic (95% CI) according to the fixed effects model	Median C-statistic (95% CI) according to the random effects model	I2 (95%CI)
NNIS risk index	11	5385	398	0.64 (0.62–0.68)	0.64 (0.59–0.68)	0.0% (0.0–60.2%)
ACRI	10	4804	368	0.63 (0.59–0.67)	0.64 (0.59–0.68)	0.0% (0.0–62.4%)
IRIC	10	4804	368	0.63 (0.60–0.67)	0.64 (0.59–0.68)	0.0% (0.0–62.4%)
RIA	10	4804	368	0.64 (0.61–0.67)	0.64 (0.60–0.69)	0.0% (0.0–62.4%)
RIB	9	4362	356	0.64 (0.61–0.67)	0.64 (0.60–0.68)	0.0% (0.0–64.8%)

Abbreviations: ACRI, Australian Clinical Risk Index; CI, confidence interval; IRIC, Infection risk index in cardiac surgery; NNIS, National Nosocomial Infections Surveillance; RIA, Risk index A; RIB, Risk index B; SSI, surgical site infections. I²: a measure of the proportion of variability in site-specific estimates that is due to between-cluster heterogeneity rather than chance

Table 5.

Pooled cluster-adjusted median C-statistic estimate of the risk indices in the sub-cohort

Risk index	Median C-statistic (95% CI) according to the fixed effects model	Median C-statistic (95% CI) according to the random effects model	I2 (95%CI)
NNIS risk index	0.62 (0.60–0.64)	0.63 (0.57–0.68)	0.0% (0.0–74.6%)
ACRI	0.65 (0.58–0.71)	0.65 (0.58–0.72)	0.0% (0.0–74.6%)
IRIC	0.64 (0.58–0.70)	0.65 (0.57–0.72)	0.0% (0.0–74.6%)
RIA	0.64 (0.58–0.70)	0.65 (0.58–0.71)	0.0% (0.0–74.6%)
RIB	0.64 (0.58–0.70)	0.65 (0.57–0.71)	0.0% (0.0–74.6%)

Based on the results from 6 sites, with a total number of 1222 patients and 138 SSI events

Abbreviations: ACRI, Australian Clinical Risk Index; CI, confidence interval; IRIC, Infection risk index in cardiac surgery; NNIS, National Nosocomial Infections Surveillance; RIA, Risk index A; RIB, Risk index B; SSI, surgical site infections. I²: a measure of the proportion of variability in site-specific estimates that is due to between-cluster heterogeneity rather than chance

New model

Based on the slightly better performance of the risk indices in the overall cohort compared to the sub-cohort, we decided to develop and assess the performance of the new SSI model only in the overall cohort. The results of the multivariable analysis are shown in supplemental Table 7. The variables natural logarithm (ln)-transformed duration of surgery (odds ratio [OR] 1.60 per 1 log-unit increase, 95% CI 1.24–2.06), ASA score 3 (OR 2.02, 95% CI 1.00–4.05), and female (OR 0.67, 95% CI 0.49–0.92) reached statistical significance.

Model performance

The pooled median predicted risk of developing an SSI within 90 days following surgery was 0.026 (range 0.0019–0.18 in all imputed datasets), indicating that most patients were categorized as having a low risk of SSI. The patients who developed an SSI had on average a higher predicted risk than those who did not develop an SSI (Supplemental Fig. 3). The calibration plot with the average agreement between the observed SSI risk and the predicted risks, computed over the imputed datasets, is shown in supplemental Fig. 4 (Nagelkerke R2 0.05). The pooled C-statistic of the model was 0.67 (95% CI 0.64–0.69), indicating poor discrimination. No further internal validation was done.

Discussion

An independent evaluation of six different risk indices for predicting SSI risk demonstrated that all six risk indices performed poorly in our weighted overall validation cohort and sub-cohort. They all exhibited poor discriminatory ability, although their discriminatory ability improved slightly after accounting for clustering. Also, most risk indices overestimated the SSI risk, especially for patients in higher risk categories. In addition, our attempt to improve the predictive accuracy of the risk indices yielded only a minimal improvement in performance.

A possible explanation for the poor performance of the risk indices in our validation study could be that the composition of our overall cohort and sub-cohort was too distinct compared to the original cohorts used to derive the risk indices. Except for the NNIS index, all other risk indices were specifically developed to stratify the risk of SSI after cardiac surgery (predominantly coronary artery bypass graft [CABG]). When studying transportability of prediction models in related populations, some differences in patient characteristics between the derivation cohorts of the different risk indices and the validation study cohorts are expected. However, all risk indices performed equally poor in both overall cohort and sub-cohort, though we expected better model performance in the sub-cohort, as it more closely resembled the derivation cohorts of the risk indices. Interestingly, the estimates of the C-statistics of the risk indices improved and reached values comparable to those reported in the respective derivation studies, when we accounted for clustering. This emphasizes the importance of adequate adjustment when dealing with clustered data.

Another explanation for the poor performance could be that while some derivation studies focused solely on complex SSIs or employed a shorter (or no) post-discharge SSI surveillance, our analyses included both superficial and deep SSIs occurring over a period of 90 days following surgery. This might have increased the number of events but decreased the sensitivity of the risk indices in this study. Also, improvements in surgical procedures, preventive measures, and perioperative management of care, may have led to improvements in patient outcomes over time. This may explain why the models primarily overestimate the SSI risks. However, IRIC was developed more recently and has yielded no better performance. Lastly, regarding the original NNIS index, as most of the patients underwent clean surgery and had an ASA score of 2 or 3, most patients were concentrated in these two NNIS categories, decreasing the overall discriminative power of this risk index.

A possible explanation for the lack of predictive performance of our developed model is that the model only included general predicator variables, and although these variables are readily available, they probably lack predictive accuracy. In addition, we optimized the form of the variables BMI and duration of surgery for our model, which could potentially lead to overfitting of the model. Because of the poor performance of the derived model, we did not conduct internal validation to correct for potential overfitting. Our model could potentially be improved by adding predictor variables with higher prediction value to the model, and by applying a better variable selection. We did not do this, as we aimed to only include variables in our model from at least one of the risk indices.

Comparison with relevant findings from literature

Figuerola-Tejerina and colleagues conducted an independent study in which they validated the ACRI for predicting 30-day SSI in patients undergoing coronary artery surgery or valvular surgery. They also compared the predictive accuracy of ACRI against that of the original NNIS index. In both cohorts, the ACRI performed marginally better than the NNIS (ACRI: C-statistic 0.64, 95% CI 0.5–0.7 and NNIS: 0.62, 95% CI 0.5–0.7 for valvular surgery; ACRI: 0.70, 95% CI 0.5–0.8 and NNIS: 0.60, 95% CI 0.4–0.7 for coronary artery surgery), suggesting that ACRI was better at predicting SSI risk compared to the NNIS index. However, as the model performance of ACRI was still suboptimal, the authors concluded that ACRI is still not a good enough model for SSI prediction and that continued efforts are needed to identify better predictors of SSI risk [28]. Chen and colleagues also investigated the ability of ACRI, compared to the original NNIS index, for predicting SSI after CABG. They found that the NNIS index was unable to adequately predict SSI after CABG, whereas an increase in ACRI category was associated with an increased SSI risk (category 2: OR 2.39, 95% CI 1.33–4.29; category 3: OR 4.46, 95% CI 1.83–10.85) [29]. In the study by Bustamante-Munguira and colleagues, in which the IRIC was derived and validated against the original NNIS index and ACRI, the IRIC (C-statistic: 0.66, 95% CI 0.55–0.76) performed slightly better than NNIS (0.61, 95% CI 0.50–0.71) or ACRI (0.61, 95% CI 0.50–0.72) when predicting SSI after cardiac surgery [17]. Lastly, the study by Paul and colleagues showed that the original NNIS index exhibited poor discriminatory abilities (C- statistic 0.64, 95% CI 0.53–0.76) and a limited ability to stratify patients undergoing CABG into infection risk groups [30]. When the performance of the original NNIS index was assessed in an elderly population undergoing elective surgery, the discrimination was also poor (C-statistic 0.59), suggesting that the index was suboptimal at stratifying patients into risk categories and predicting SSI risk [31]. Regarding RIA/RIB, this was the first study assessing their external validity.

Compared to the studies mentioned above, the current study is the first assessing the predictive performance of the ARIC, ACRI, RIA, and RIB, in a broad, international surgical population.

Limitations

A limitation of this study was that we had to use weighting methods to correctly account for the sampling procedure employed in the ASPIRE-SSI study. This may have had an impact on the generalizability of the study for non-carriers of SA. Also, we used data from an existing cohort which was not designed initially for this purpose. This means that the way patients were selected and data were collected in this study, may have differed from the way this was done in the original studies. We are confident that the definitions of the explanatory and outcome variables in this study are similar to those of the original studies. For instance, all studies used the same CDC criteria for SSI. However, not all studies used active post-discharge surveillance for SSI like the ASPIRE-SSI study (e.g., absence of routine post-discharge surveillance in the ACRI and RIA/RIB derivation studies). This is probably the reason why we observed a higher SSI incidence in our overall cohort and sub-cohort compared to the development studies. Despite this, the SSI risk for patients with higher risk indices were still overestimated in our study, perhaps due to reasons explained earlier. In addition, our study probably captured more superficial SSIs than the derivation studies because of the active surveillance with standardized follow-up post-discharge. Lastly, the patients included in the study underwent primarily planned and clean surgical procedures. For this reason, our newly developed model is probably not fit to predict SSI risk after unplanned or contaminated/dirty surgical procedures.

Conclusion

The assessed risk indices performed poorly in our validation study, although their discriminatory ability improved after accounting for clustering. Risk indices consisting of readily available but rather unspecific predictors lack predictive accuracy.

Abbreviations

ACRI

Australian clinical risk index

ASA

American society of anesthesiologist’s

ASPIRE-SSI

Advanced understanding of Staphylococcus aureus infections in Europe surgical site infections

BMI

Body mass index

C-statistic

Concordance statistic

CABG

Coronary artery bypass graft

CDC

Centers of disease control and prevention

IRIC

Infection risk index in cardiac surgery

Ln

Natural logarithm

NNIS

National nosocomial infection surveillance system

OR

Odds ratio

RIA

Risk index A

RIB

Risk index B

SA

Staphylococcus aureus

SSI

Surgical site infection

TRIPOD

Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis

Author contributions

Concept and design: Troeman, Van Werkhoven, Kluytmans, Harbarth. Statistical analysis: Troeman, Van Werkhoven. Interpretation of data: Troeman, Van Werkhoven, Kluytmans, Harbarth. Drafting of the manuscript: Troeman. Critical revision of the manuscript for important intellectual content: Troeman, Van Werkhoven, Kluytmans, Harbarth.

Funding

This research project received support from the Innovative Medicines Initiative Joint Undertaking under grant agreement 115523, which are composed of financial contributions from the European Union Seventh Framework Programme (FP7/2007‐2013) and European Federation of Pharmaceutical Industries and Associations companies in-kind contribution.

Data availability

The data sets generated and analyzed during the current study are not publicly available for confidentiality reasons but are available from the corresponding author upon scientific review and approval of the request by the Study’s Scientific Committee of the ASPIRE-SSI study.

Declarations

Ethics approval and consent to participate

For the original ASPIRE-SSI study, institutional or ethics review boards at each participating hospital or country approved the study protocol prior to patient enrolment at any site. All included patients gave written informed consent for participation in the ASPIRE-SSI study.

Consent for publication

Not applicable.

Competing interests

C.H.W. reported receiving research grants/in-kind contribution from DaVolterra, BioMerieux, and Limmatech and consultancy fees from Merck/MSD and Sanofi-Pasteur. All other authors have no competing interests to declare.