The ACS NSQIP Risk Calculator Is a Fair Predictor of Acute Periprosthetic Joint Infection

Abstract

Background

Periprosthetic joint infection (PJI) is a severe complication from the patient’s perspective and an expensive one in a value-driven healthcare model. Risk stratification can help identify those patients who may have risk factors for complications that can be mitigated in advance of elective surgery. Although numerous surgical risk calculators have been created, their accuracy in predicting outcomes, specifically PJI, has not been tested.

Questions/Purposes

(1) How accurate is the American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) Surgical Site Infection Calculator in predicting 30-day postoperative infection? (2) How accurate is the calculator in predicting 90-day postoperative infection?

Methods

We isolated 1536 patients who underwent 1620 primary THAs and TKAs at our institution during 2011 to 2013. Minimum followup was 90 days. The ACS NSQIP Surgical Risk Calculator was assessed in its ability to predict acute PJI within 30 and 90 days postoperatively. Patients who underwent a repeat surgical procedure within 90 days of the index arthroplasty and in whom at least one positive intraoperative culture was obtained at time of reoperation were considered to have PJI. A total of 19 cases of PJI were identified, including 11 at 30 days and an additional eight instances by 90 days postoperatively. Patient-specific risk probabilities for PJI based on demographics and comorbidities were recorded from the ACS NSQIP Surgical Risk Calculator website. The area under the curve (AUC) for receiver operating characteristic (ROC) curves was calculated to determine the predictability of the risk probability for PJI. The AUC is an effective method for quantifying the discriminatory capacity of a diagnostic test to correctly classify patients with and without infection in which it is defined as excellent (AUC 0.9–1), good (AUC 0.8–0.89), fair (AUC 0.7–0.79), poor (AUC 0.6–0.69), or fail/no discriminatory capacity (AUC 0.5–0.59). A p value of < 0.05 was considered to be statistically significant.

Results

The ACS NSQIP Surgical Risk Calculator showed only fair accuracy in predicting 30-day PJI (AUC: 74.3% [confidence interval {CI}, 59.6%–89.0%]. For 90-day PJI, the risk calculator was also only fair in accuracy (AUC: 71.3% [CI, 59.9%–82.6%]). Conclusions The ACS NSQIP Surgical Risk Calculator is a fair predictor of acute PJI at the 30- and 90-day intervals after primary THA and TKA. Practitioners should exercise caution in using this tool as a predictive aid for PJI, because it demonstrates only fair value in this application. Existing predictive tools for PJI could potentially be made more robust by incorporating preoperative risk factors and including operative and early postoperative variables.

Level of Evidence

Level III, diagnostic study.

Introduction

Individual risk factors for development of prosthetic joint infection (PJI) have been identified, including age, sex, and the presence of various medical comorbidities [2, 10, 11]. Several clinical risk stratification systems generate a patient-specific risk assessment for morbidity and mortality across multiple surgical procedures. These include Charlson Comorbidity Index [3] and American Society of Anesthesiologists Classification (ASA Class) [6] and the All Patient Refined-Diagnosis Related Groups scoring system [12]. In 2013, the Universal American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) Surgical Risk Calculator was developed for use across all surgical subspecialties and covering 2500 Current Procedural Terminology (CPT) codes [1]. By entering patient characteristics and a CPT code into a publically available software program, the surgeon calculates a patient-specific risk estimate for 11 postoperative complications [7]. Included within the scope of this clinical prediction tool is a capacity to calculate a patient’s individual risk for development of PJI after TKA and THA. Considering the devastating consequences of PJI, including individual morbidity and healthcare system resource utilization, a validated patient-specific risk assessment for development of PJI would be of great value during the patient and surgeon decision-making process.

The Centers for Medicare & Medicaid Services may soon begin financially incentivizing surgeons to document use of preoperative risk stratification tools during their discussion with the patient about surgical risk and obtaining surgical consent [9, 13]. At present, the ACS NSQIP Surgical Risk Calculator is available online for the orthopaedic surgeon to use in the preoperative clinic setting as a decision-making tool for prediction of PJI and patient counseling. However, the ACS Calculator has not been validated for prediction of early PJI. The impetus of our investigation was to determine if the ACS NSQIP Surgical Risk Calculator could accurately predict the risk of PJI after elective THA and TKA.

We therefore asked: (1) How accurate is the ACS NSQIP Surgical Risk Calculator in predicting 30-day postoperative infection? (2) How accurate is the calculator in predicting 90-day postoperative infection?

Patients and Materials

We conducted a review of all patients who had a primary THA and TKA performed at our institution between January 1, 2011, and December 31, 2013, using our electronic medical record. Institutional review board approval was obtained before initiating the study. We collected relevant demographic information, medical comorbidities, and postoperative surgical outcomes within 90 days of surgery, specifically including reoperations for infection, hematoma, or drainage. The minimum followup was 90 days.

For purposes of this study, we defined patients who developed PJI as those who underwent a repeat surgical procedure within 90 days of the index arthroplasty and in whom at least one positive intraoperative culture on solid media was obtained at the time reoperation. At our institution, aerobic cultures are observed for at least 2 days until they are deemed no growth, whereas the anaerobic portion is held for at least 7 days until it is considered to be negative for growth. Patients underwent primary TKA or THA for the following etiologies: primary osteoarthritis (94% [1523 of 1620]), avascular necrosis (3.1% [50 of 1620]), posttraumatic arthritis (2.8% [45 of 1620]), and rheumatoid arthritis (0.1% [two of 1620]). We excluded patients whose index arthroplasty was performed for postinfectious arthritis of their native joint. We also excluded 10 patients who underwent reoperation for mechanical complications within 90 days such as periprosthetic fracture (four patients), dislocation (three patients), extensor mechanism rupture (two patients), and cup loosening (one patient). These patients were excluded because they have an elevated risk for developing PJI (33%) after unavoidable reoperations during the acute recovery phase and hence would bias our study [6]. All patients received intraoperative antibiotics within 1 hour of incision. The posterior approach and medial parapatellar approach were used to perform THA and TKA, respectively. General anesthesia was used in the majority of patients (66%), whereas the remaining received neuraxial anesthesia.

Our study population consisted of 1536 patients who underwent 1620 THAs and TKAs with a mean age of 66 years (range, 21–94 years) and body mass index of 32.7 kg/m² (range, 15.4–68.6 kg/m²). Women constituted 58% of the population, whereas knees were more prevalent (63%) than hips. A total of 24 patients underwent reoperation within 90 days for either a hematoma or wound complications. One patient underwent superficial irrigation and débridement for a stitch abscess and was removed from the study group. At 1-year followup, the patient did not require further surgical intervention or develop clinical signs of PJI. Three patients had irrigation and débridement with polyethylene liner exchange for a hematoma from which intraoperative cultures failed to isolate an organism. A single patient underwent irrigation and débridement with polyexchange for persistent wound drainage, but intraoperative cultures were not taken at that time. We excluded these five patients because we were unable to confirm or exclude the presence of early PJI. Hence, a total of 19 patients were left who underwent reoperation within 90 days and had at least one positive intraoperative culture, three of whom had a resection arthroplasty with insertion of antibiotic cement spacer. The organism profile consisted of six patients with methicillin-sensitive Staphylococcus aureus, five patients with Staphylococcus species other than S aureus, two patients with methicillin-resistant S aureus, two patients with Streptococcus species, one patient with Escherichia coli, one patient with Klebsiella oxytoca, and two patients with polymicrobials. Of the 19 patients with positive cultures, 11 underwent reoperation within 30 days, whereas the remaining eight patients underwent reoperation between the 30- and 90-day period. For each study patient, 21 preoperative factors (Table 1) were retrospectively entered into the web-based ACS NSQIP Surgical Risk Calculator interface [7], which generated a unique patient-specific ACS NSQIP surgical site infection (SSI) risk score representing the 30-day estimated risk for PJI. A research assistant (JG) not involved in the patients’ care populated the 21 preoperative variables into the web-based application. The data were directly abstracted from the institutions electronic medical record (EMR). For each preoperative risk factor, the appropriate section of the patient EMR was scoured for presence or absence of the given condition. There were a total of 99 patients who had missing data elements required by the ACS NSQIP interface, and hence an SSI risk score could not be generated and analyzed. Furthermore, three patients died postoperatively and did not have 90-day followup available.

Table 1.

List of the 21 variables required by the ACS NSQIP Surgical Risk Calculator interface to generate a risk score for PJI

Age

Sex

Functional status

Emergent case

ASA score

Wound class

Steroid use for chronic condition

Ascites within 30 days before surgery

Systemic sepsis within 48 hours before surgery

Ventilator-dependent

Disseminated cancer

Diabetes

Hypertension requiring medication

Previous cardiac event

CHF in 30 days before surgery

Dyspnea

Current smoker within 1 year

History of severe COPD

Dialysis

Acute renal failure

BMI

ACS NSQIP = American College of Surgeons National Surgical Quality Improvement Program; PJI = periprosthetic joint infection; ASA = American Society of Anesthesiologists; CHF = congestive heart failure; COPD = chronic obstructive pulmonary disease; BMI = body mass index.

Statistical Analysis

The means of the ACS NSQIP SSI risk scores were compared between the patients who developed PJI and those were infection-free at the 30-day postoperative followup using the Wilcoxon rank-sum test. The ability of the ACS NSQIP SSI risk scores to predict PJI occurrence was determined by generating receiver operating characteristic (ROC) curves and calculating the area under the curve (AUC). An ROC curve graphically depicts the performance of a diagnostic test, plotting true-positive rate against the false-positive rate. The AUC numerically evaluates the discriminatory capacity of a diagnostic test to correctly classify patients with and without infection with a value approaching one representing ideal behavior. The postoperative followup was expanded to 90 days and a similar analysis was conducted to assess the ability of the same ACS NSQIP SSI risk scores to predict PJI at that later followup. Discriminatory value of ROC curves was interpreted as excellent (AUC 0.9–1), good (0.8–0.89), fair (0.7–0.79), poor (0.6–0.69), or fail/no discriminatory capacity (0.5–0.59) [8]. A p value of < 0.05 was considered to be significant. All statistical analyses were performed using SAS 9.3 (SAS Institute Inc, Cary, NC, USA).

Results

The ACS NSQIP Risk Calculator showed only fair accuracy in predicting 30-day PJI (AUC: 74.3% [confidence interval {CI}, 59.6%–89.0%]; Fig. 1). However, patients who developed SSI within 30 days had very slightly higher ACS NSQIP SSI risk scores (1.2 ± 0.3% versus 0.9 ± 0.3%, mean difference: −0.26 and 95% CI −0.44 to −0.05, p = 0.004; Fig. 2); it is unclear whether this small difference is clinically meaningful.

Fig. 1A–B.

ROC curves were constructed to determine how accurate the SSI risk estimates were in predicting PJI at 30 days postoperatively (A) and 90 days postoperatively (B).

Fig. 2A–B.

The Wilcoxon rank-sum test was used to compare the mean distribution of the SSI risk estimates for infected and noninfected patients at 30 days postoperatively (A) and 90 days postoperatively (B).

For 90-day PJI, the accuracy of the risk calculator likewise was also only fair (AUC: 71.3% [CI, 59.9%–82.6%]; Fig. 1). The calculated ACS NSQIP SSI risk score was higher in patients who developed PJI within 90 days of their primary joint replacement as compared with those patients who were infection-free (1.19% ± 0.35 SD versus 0.94% ± 0.29 SD; mean difference: −0.25 and 95% CI −0.39 to −0.12, p = 0.001; Fig. 2).

Discussion

A clinical prediction tool capable of estimating a patient-specific risk for PJI after primary THA and TKA would be of great value to guide patient and surgeon decision-making in the clinic setting. The ACS NSQIP Surgical Risk Calculator estimates surgical risks for morbidity and mortality after nearly all surgical procedures, including TKA and THA [1]. The ACS NSQIP Surgical Risk Calculator has not been validated for prediction of PJI after joint replacement. Therefore, we aimed to determine the accuracy of the ACS NSQIP Surgical Risk Calculator in predicting 30-day and 90-day postoperative infection.

Our investigation is marked by a number of limitations. Although our study included approximately 1600 THAs and TKAs, a larger study population could have yielded more robust ROC curves and may have ultimately reached different conclusions. True statistical independence of subjects was not realized in our statistical analysis, because patients who underwent more than one arthroplasty were represented in our data set with more than one data point entry (one data point for each arthroplasty event). At present, there is no perfect test to determine the presence or absence of joint infection and this fact also tempers the potential value of our findings. The Musculoskeletal Infection Society (MSIS) definition for PJI, although still imperfect, provides a universally agreed-on definition for chronic PJI [14]. Our investigation analyzed acute infections, however, and used a different working definition of infection. Serological tests in the acute postoperative period are usually elevated and are not routinely obtained at our center, whereas frozen section in the early postoperative period has yet to be defined. Furthermore, it is not routine practice in our institution to aspirate hip and knee replacements in the acute recovery period even when presenting with signs of PJI. Hence, numerous data elements required to satisfy the MSIS criteria were not available with the potential of misdiagnosing culture negative infections, which may have affected our analysis. Our study was conducted among a rural patient population at a large tertiary care teaching hospital. Findings generated from this population and hospital system may not be generalizable across all other settings for total joint arthroplasty in the United States. Finally, our study findings can only be as accurate as the data captured from our EMR with the possibility of incomplete or otherwise inaccurate data that pose as a potential critique against our conclusions.

Our investigation suggests that the ACS NSQIP Surgical Risk Calculator exhibits only fair test performance for prediction of 30-day postoperative infection. Recently, Edelstein et al. found the ACS NSQIP Calculator to have low discriminatory capacity in identifying Medicare patients who developed PJI after their elective total joint arthroplasty with an AUC of 54.7%[7]. In contrast, our study yielded an AUC of 74.3% for prediction of PJI within 30 days of THA or TKA. These differences observed between the two studies may be explained by their investigation’s criteria for diagnosing infection that were not specified, demographic differences (urban academic center), smaller sample size, quality of EMR data, and collection. Our investigation adds to this previous work an increased methodologic rigor with specific focus on the complication of PJI and a larger sample size in a rural academic center. However, both studies revealed that the ACS NSQIP Surgical Risk Calculator does fall short of the desired strong c-statistic; performance was less than “good” (0.8–0.89) or “excellent” (0.9–1). It should be noted that within the original data set from which it was formulated, the Universal ACS NSQIP Surgical Risk Calculator demonstrated good performance in predicting complications including SSI (AUC 0.817) [4], but this has not been reproduced. Although the mean ACS NSQIP SSI risk score in patients with PJI was higher than in noninfected patients at 30 days, this difference was a very small numeric value, one unlikely to be clinically significant or useful to the patient and physician in choosing whether to proceed with surgical intervention.

For prediction of 90-day PJI, the ACS NSQIP Surgical Risk Calculator also demonstrated only fair performance. The ACS NSQIP Surgical Risk Calculator was formulated for prediction of 30-day complications and outcomes. However, 90-day complication rates, including PJI, are also an important metric through which Centers for Medicare & Medicaid Services assesses hospital performance and an ideal predictive calculator would have the capacity to extend beyond 30 days [5]. As stated previously, the mean difference of the ACS risk score between infected and noninfected patients was too minute, which poses a dilemma for the surgeon when counseling the patient on risk-stratifying.

Our investigation finds the ACS NSQIP Surgical Risk Calculator is not a good predictor for acute PJI at 30-day and 90-day postoperative intervals after TKA and THA. As such, compulsory changes to current surgeon practice are not warranted. The ACS NSQIP provides some utility as a general risk assessment tool for PJI and builds on predictive value of certain medical conditions and other existing prediction tools such as ASA score, Charlson Comorbidity Index, and others [2, 11, 12]. We suggest that the ACS NSQIP Surgical Risk Calculator may be best used as an optional tool to complement preoperative surgeon-patient discussions of PJI risk. Given the fair performance of the ACS NSQIP Surgical Risk Calculator, mandated use is not warranted, and numerical results by themselves cannot justifiably supersede a traditional shared decision-making process between doctor and patient. Further research is required in generating PJI risk calculators for elective THA and TKA that incorporate more robust patient-specific risk factors and pertinent operative and postoperative variables or simply improving on the ones we currently have at our disposal [2].